Regression Analysis

The goal of regression analysis is to describe the relationship between two variables based on observed data and to predict the value of the dependent variable based on the value of the independent variable. Even though we can make such predictions, this doesn’t imply that we can claim any causal relationship between the independent and dependent variables.

Definition 1: If y is a dependent variable and x is an independent variable, then the linear regression model provides a prediction of y from x of the form

Regression model formula

where α + βx is the deterministic portion of the model and ε is the random error. We further assume that for any given value of x the random error ε is normally and independently distributed with mean zero.

Observation: In practice we will build the linear regression model from the sample data using the least squares method. Thus we seek coefficients a and b such that

Equation straight line

For the data in our sample we will have

image1688

where ŷi is the y value predicted by the model at xi. Thus the error term for the model is given by

image1689

Example 1: For each x value in the sample data from Example 1 of One Sample Hypothesis Testing for Correlation, find the predicted value ŷ corresponding to x, i.e. the value of y on the regression line corresponding to x. Also find the predicted life expectancy of men who smoke 4, 24 and 44 cigarettes based on the regression model.

Forecast prediction Excel

Figure 1 – Obtaining predicted values for data in Example 1

The predicted values can be obtained using the fact that for any i, the point (xi, ŷi) lies on the regression line and so ŷi = a + bxi. E.g. cell K5 in Figure 1 contains the formula =I5*E4+E5, where I5 contains the first x value 5, E4 contains the slope b and E5 contains the y intercept (referring to the worksheet in Figure 1 of Method of Least Squares). Alternatively this value can be obtained by using the formula =FORECAST(I5,J5:J19, I5:I19). In fact, the predicted y values can be obtained, as a single unit, by using the array formula TREND. This is done by highlighting the range K5:K19 and entering the array formula =TREND(J5:J19, I5:I19) followed by pressing Ctrl-Shft-Enter.

The predicted values for x = 4, 24 and 44 can be obtained in a similar manner using any of the three methods defined above. The second form of the TREND formula can be used. E.g. to obtain the predicted values of 4, 24 and 44 (stored in N19:N21), highlight range O19:O21, enter the array formula =TREND(N19:N21,J5:J19,I5:I19) and then press Ctrl-Shft-Enter. Note that these approaches yield predicted values even for values of x that are not in the sample (such as 24 and 44). The predicted life expectancy for men who smoke 4, 24 and 44 cigarettes is 83.2, 70.6 and 58.1 years respectively.

Definition 2: We use the following terminology:

Picture101

The Residual is the error term of Definition 1. We also define the degrees of freedom dfT, dfReg, dfRes, the sum of squares SST, SSReg, SSRes and the mean squares MST, MSReg, MSRes as follows:

Picture100

Property 1:

image1711 image1712 image1713

Observation: SST is the total variability of y (e.g. the variability of life expectancy in Example 1 of One Sample Hypothesis Testing for Correlation). SSReg represents the variability of y that can be explained by the regression model (i.e. the variability in life expectancy that can be explained by the number of cigarettes smoked), and so by Property 1, SSRes expresses the variability of y that can’t be explained by the regression model.

Thus SSReg/SST represents the percentage of the variability of y that can be explained by the regression model. It turns out that this is equal to the coefficient of determination.

Property 2:
image1720

Property 3:
image1721

Observation: Note that for a sample size of 100, a correlation coefficient as low as .197 will result in the null hypothesis that the population correlation coefficient is 0 being rejected (per Theorem 1 of One Sample Hypothesis Testing for Correlation). But when the correlation coefficient r = .197, then r2 = .039, which means that model variance SSReg is less than 4% of the total variance SST which is quite a small association indeed. Whereas this effect is “significant”, it certainly isn’t very “large”.

Observation: From Property 2, we see that the coefficient of determination r2 is a measure of the accuracy of the predication of the linear regression model. r2 has a value between 0 and 1, with 1 indicating a perfect fit between the linear regression model and the data.

Property 4:
image1724 image1725

Definition 3:  The standard error of the estimate is defined as

Standard error estimate

Observation: The second assertion in Property 4 can be restated as

image1728

For large samples \frac{n-1}{n-2} ≈ 1 and so

image1730

Note that if r = .5, then

image5045

which indicates that the standard error of the estimate is still 86.6% of the standard error that doesn’t factor in any information about x; i.e. having information about x only reduces the error by 13.4%. Even if r = .9, then sy.x = .436·sy, which indicates that information about x reduces the standard error (with no information about x) by only a little over 50%.

Property 5:

a)  The sums of the y values is equal to the sum of the ŷ values; i.e. \sum_i{y_i}\sum_i{\hat{y_i}}

b)  The mean of the y values and ŷ values are equal; i.e. ȳ = the mean of the ŷi

c)  The sums of the error terms is 0; i.e. \sum_i{e_i} = 0

d)  The correlation coefficient of x with ŷ is sign(b); i.e. rxŷ = sign(rxy)

e)  The correlation coefficient of y with ŷ is the absolute value of the correlation coefficient of x with y; i.e. r_{y\hat{y}} = |r_{xy}|

f)  The coefficient of determination of y with ŷ is the same as the correlation coefficient of x with y; i.e. r_{y\hat{y}}^2 = r_{xy}^2

Observation: Click here for the proofs of the various properties described above.

16 Responses to Regression Analysis

  1. Amanda Steele says:

    Hi Charles,

    I would appreciate some guidance- I am helping with a study designed to evaluate whether a preoperative score on a certain scale (continuous scores from 1-100) can predict postoperative outcomes (also scored on continuous scales from 1-100). This seems like a fairly straightforward regression situation to me, but using the realstats pack I seem to be running into issues since I have 1 independent variable (pre-op score) and multiple dependent variables (post-op scores). Can you suggest an analysis strategy?

    Thanks!

    • Charles says:

      Amanda,
      You can multiple regressions, one for each dependent variable. This should be sufficient if there is little correlation between the dependent variables.
      Alternatively, you can use multivariate multiple linear regression. The initial output is identical to that from separate multiple regressions, but additional analyses are performed. The Real Statistics Resource Pack doesn’t support this second step yet.
      Charles

  2. yask says:

    Hi Charles,
    Is there a way to determine order of regression analysis (if data requires first order regression or second order in other words linear regression or polynomial regression) using realstats?
    I have multiple variables and xl does not have a correct way to judge if data requires either linear regression or polynomial regression other than looking at p-value, which in my case 0.05.

    • Charles says:

      Yask,
      I don’t know of a simple way to determine the optimum order of polynomial regression model. I will look into this for a future enhancement.
      To make matters worse you could have interactions: y = x^3 + z^3 + x^2 + x^3*z^2 + xz + x + z (with two independent variables x and z).
      Charles

  3. Amy says:

    Hi Charles,

    I am running a regression test involving life expectancy as the dependent variable and am examining the independent variables: income, pharamacuetical spending… as variables affecting life exp. Now how do I run the regression, do I also compile data on life expectancy and put that as the dependent variable in Stata?

    Amy

    • Charles says:

      Amy,
      Sorry, but I don’t use Stata and so can’t answer your question. You can run the regression data analysis from Excel or from the Real Statistics Resource Pack.
      Charles

    • Ms. Z says:

      Hi Amy,

      have you tried SPSS Statistic? Its a software available online. You can use it to run your data. you can also check youtube on how to handle SPSS statistic.

  4. Sampath Jayasekara says:

    Hello Mr. Charles,

    I have figures only for independent variable. So how can I take dependent variable values in order to conduct correlation and regression test.

    • Charles says:

      You can’t. You need a dependent variable to perform regression. Perhaps one of the independent variables is really a dependent variable.
      Charles

  5. Nadee says:

    Dear Charles,
    I have a sample of 30 to measure the factors constraining to the adoption of technology. I am thinking to run regression analysis to adoption rate ( if it is more that 50% considered as 1 and less than 50 is 0 taking 50 percent adoption as threshold limit).
    to measure the constraining factors , I used the 5 point likert scale. (highly significant to least significant) and already extracted important variables using principal component analysis.
    Now what kind of regression analysis should I use to measure the relative importance of each factors ?. Linear or multiple ?

    Nadee

    • Charles says:

      Dear Nadee,

      When you say “multiple” I assume that you mean “multiple linear regression”, which just means that you have more than one independent variable. When you have only one independent variable often the term “linear regression” or “simple linear regression” is used. Since you say that you have multiple factors, you would often use multiple linear regression.

      Since your outcome (dependent variable) could be viewed as dichotomous (0 or 1), you might find that logistic regression gives a better fit for the data. You can compare AIC values for this.

      Charles

  6. Dennis says:

    hi Charles,

    I had a set of data to do the regression, but the result I have got from this addin is different to EXCEL. Could you pls let me know why is it?

    Thanks.

  7. Wilfred says:

    Hello Charles,

    I must commend your work, I have been following your materials here and I must confess, they are awesomely helpful in my ongoing project research. However I humbly ask for a more tailored explanation on the following.

    I am doing a study on the impact of some sectors on the Economic Growth of Nigeria using the GDP as a measure. I collected sample on GDP (response variable) and 5 independent variables over a period of n = 32 years. So far I have been able to run my analysis using SAS, my problem however is how to manually explain the following computations.

    1. The coefficients of regression Bo, B1, B2,…, B5.
    2. Formular to calculate the correlation coefficients.
    3. How to calculate my R^2 manually.

    I need these because in my school, students are required to be able to explain how values can manually be computed using calculator (other than software). I await your kind support.

    Thanks for the great work.
    Wilfred Okwudili,
    Department of Statistics,
    Federal University of Technology, Owerri, Imo State, Nigeria.

Leave a Reply

Your email address will not be published. Required fields are marked *