In problems where the possible outcomes are “Conservative, Labour or Liberal-Democrat” or “Red, Blue, Green” there is no apparent order to the possible outcomes. When the outcomes are “Small, Medium, Large” or “City, State, Country” or “Strongly Disagree, Disagree, Agree, Strongly Agree” there is an intrinsic order. We now address the case of multinomial logistic regression where the outcomes for the dependent variable can be ordered.
Suppose the possible outcomes for the dependent variable are 1, …, r. Let pih = P(yi ≤ h), i.e. the cumulative probabilities. Thus 0 = pi0 < pi1 ⋯ < pir = 1 (thereby capturing the order of the outcomes), where pi0 = 0 for notational convenience. Then for h = 1, …, r
This model can be viewed as r binary models with events y ≤ h vs. h < y. The logit models for h = 1, …, r–1 are therefore
where for convenience we set xi0 = 1. Thus
The likelihood and log-likelihood statistics are as follows:
Example 1: A study was conducted based on a sample of 420 people to determine how satisfied people are with their mobile device based on a Likert scale (1 to 4 with 1 = not very satisfied and 4 = very satisfied). People in the sample were characterized by gender (female = 1 and male = 0) and age (0 = under 18, 1 = 18-24, 2 = 25-30, 3 = 31-40, 4 = over 40). Create an ordered logistic regression for this study based on the data in Figure 1.
Figure 1 – Data for Example 1
We now present different approaches for creating the ordinal logistic regression models, especially for finding the coefficients. We also compare the results obtained with those obtained using a multinomial logistics regression model.
Using binary logistic regression models
We begin by developing three cumulative binary regression models as shown in Figure 2.
Figure 2 – Cumulative binary logistic regression models
We now find the coefficients for each of these models using the Logistic Regression data analysis tool or the LogitCoeff function. E.g. the coefficients for the 1 vs. 2+3+4 model in range F16:F18 can be calculated by the array formula =LogitCoeff(A16:D23).
We now build the ordinal logistic regression model as shown in Figure 3 and 4
Figure 3 – Ordinal logistic regression model (part 1)
Figure 4 – Ordinal logistic regression model (part 2)
Representative formulas used in Figures 3 and 4 are shown in Figure 5.
Figure 5 – Representative formulas from Figure 3 and 4
Note: The formula for cell AL9 in Figure 5 should be =COUNT(AG6:AI7).
As we did for multinomial logistic regression models we can improve on the model we created above by using Solver. As before, our objective is to find the coefficients (i.e. range AG5:AI7 in Figure 4) that maximize LL (i.e. cell AD13 in Figure 3 or AL6 in Figure 4). The result is shown in Figure 6.
Figure 6 – Revised ordinal logistic regression model
We see that the new value of LL is -50.5323, a slight improvement over the previously calculated value of -51.0753.
Observation: We can’t initialize the coefficient values with zeros since this would result in taking the log of zero. We therefore choose to initialize the coefficients with the values from the three binary models.
Real Statistics Function: The Real Statistics Resource Pack provides the following supplemental array function
OLogitPredC(R0, R1) – outputs a 1 × r row vector which lists the probabilities of outcomes 1, …, r (in that order), where r = 1 + the number of columns in R1, for the values of the independent variables contained in the range R0 (in the form of either a row or column vector) based on the ordinal logistic regression coefficients contained in R1. Note that if R0 is a 1 × k row vector or k × 1 column vector, then R1 is a (k+1) × (r – 1) range.
Figure 7 shows the forecast for a female (gender = 1) 25-30 (age = 2).
Figure 7 – Forecasting using the model
Here the values in range C19:F19 are calculated using the formula =OLogitPredC(A19:B19, AG5:AI7) where the coefficients in range AG5:AI7 are shown in Figure 6.
The results show that the probability that a 25-30 year-old woman will be very unsatisfied is 15.1%, unsatisfied 25.7%, satisfied 31.1% and very satisfied 28.0%. These values agree with the data shown in range V11:Y11 of Figure 6.
We can also use the OLogitPredC function for forecasts corresponding to data not in our sample. E.g. for women in age group 2.5 (presumably halfway between 2 and 3, say aged approximately 21-28) we can use the formula =OLogitPredC(A20:B20, AG5:AI7) to obtain the results shown in range C20:F20 of Figure 7.
Using proportional odds model
A common approach used to create ordinal logistic regression models is to assume that the binary logistic regression models corresponding to the cumulative probabilities have the same slopes, i.e. bj1 = bj2 = ⋯ = bjr-1 for all j ≠ 0. This is the proportional odds assumption.
E.g. for Example 1 we can create a chart of the observed y values for each of the three binary logistic regression models (after sorting them) as shown in Figure 8.
Figure 8 – Testing the proportional odds assumption
As you can see these graphs are roughly parallel, indicating that the proportional odds assumption holds. While this assumption doesn’t always hold, this type of model is commonly used since it reduces the number of coefficients needed. In fact, defining bj = bj1 = bj2 = ⋯ = bjr-1 for all j ≠ 0 and defining ah = b0h for all h, we only require the full set of intercept coefficients but only one set of slope coefficients.
Figure 9 shows this model.
Figure 9 – Proportional odds model
Solver is used to maximize the value of LL (i.e. maximize cell AD13 while changing the values of AG9:AG13). We see that the value of LL is -50.9083, which is better than the value obtained from the first model (binary only) but not as good as the second model (binary + Solver). In any case, the result obtained from all three models are similar.
Observation: As in the previous model we can’t initialize the coefficient values with zeros since this would result in taking the log of zero. We therefore choose to initialize the coefficients with the intercepts from the three binary models and the slope coefficients from the first binary model.
Using multinomial logistic regression
We could of course ignore the order in Example 1 and simply use a multinomial logistic regression model. The results are shown in Figure 10.
Figure 10 – Multinomial logistic regression model
Here we are using the following functions
to calculate the coefficients, LL0, LL and other values. We also use
to obtain the forecasted values. The values are similar to those we have seen earlier, but in general when the independent variables are ordered it is best to use an ordinal logistic regression model.