The goal of regression analysis is to describe the relationship between two variables based on observed data and to predict the value of the dependent variable based on the value of the independent variable. Even though we can make such predictions, this doesn’t imply that we can claim any causal relationship between the independent and dependent variables.

**Definition 1**: If y is a dependent variable and *x* is an independent variable, then the** linear regression model** provides a prediction of y from *x* of the form

where *α + βx* is the deterministic portion of the model and ε is the random error. We further assume that for any given value of *x* the random error *ε* is normally and independently distributed with mean zero.

**Observation**: In practice we will build the linear regression model from the sample data using the least squares method. Thus we seek coefficients *a* and *b* such that

For the data in our sample we will have

where ŷ* _{i}* is the y value predicted by the model at

*x*. Thus the error term for the model is given by

_{i}**Example 1**: For each *x* value in the sample data from Example 1 of One Sample Hypothesis Testing for Correlation, find the predicted value ŷ corresponding to *x*, i.e. the value of y on the regression line corresponding to *x*. Also find the predicted life expectancy of men who smoke 4, 24 and 44 cigarettes based on the regression model.

**Figure 1 – Obtaining predicted values for data in Example 1**

The predicted values can be obtained using the fact that for any *i*, the point (*x _{i}*, ŷ

*) lies on the regression line and so ŷ*

_{i}*=*

_{i}*a + bx*. E.g. cell K5 in Figure 1 contains the formula =I5*E4+E5, where I5 contains the first

_{i}*x*value 5, E4 contains the slope

*b*and E5 contains the y intercept (referring to the worksheet in Figure 1 of Method of Least Squares). Alternatively this value can be obtained by using the formula =FORECAST(I5,J5:J19, I5:I19). In fact, the predicted y values can be obtained, as a single unit, by using the array formula TREND. This is done by highlighting the range K5:K19 and entering the array formula =TREND(J5:J19, I5:I19) followed by pressing Ctrl-Shft-Enter.

The predicted values for *x* = 4, 24 and 44 can be obtained in a similar manner using any of the three methods defined above. The second form of the TREND formula can be used. E.g. to obtain the predicted values of 4, 24 and 44 (stored in N19:N21), highlight range O19:O21, enter the array formula =TREND(N19:N21,J5:J19,I5:I19) and then press Ctrl-Shft-Enter. Note that these approaches yield predicted values even for values of *x* that are not in the sample (such as 24 and 44). The predicted life expectancy for men who smoke 4, 24 and 44 cigarettes is 83.2, 70.6 and 58.1 years respectively.

**Definition 2**: We use the following terminology:

The Residual is the error term of Definition 1. We also define the degrees of freedom *df _{T}*,

*df*, the sum of squares

_{Reg}, df_{Res}*SS*and the mean squares

_{T}, SS_{Reg}, SS_{Res}*MS*

_{T}, MS_{Reg}, MS_{Res}as follows:

**Observation**: *SS _{T} *is the total variability of y (e.g. the variability of life expectancy in Example 1 of One Sample Hypothesis Testing for Correlation).

*SS*represents the variability of y that can be explained by the regression model (i.e. the variability in life expectancy that can be explained by the number of cigarettes smoked), and so by Property 1,

_{Reg}*SS*expresses the variability of y that can’t be explained by the regression model.

_{Res}Thus *SS _{Reg}*/

*SS*represents the percentage of the variability of y that can be explained by the regression model. It turns out that this is equal to the coefficient of determination.

_{T}**Observation: **Note that for a sample size of 100, a correlation coefficient as low as .197 will result in the null hypothesis that the population correlation coefficient is 0 being rejected (per Theorem 1 of One Sample Hypothesis Testing for Correlation). But when the correlation coefficient *r* = .197, then *r ^{2}* = .039, which means that model variance

*SS*is less than 4% of the total variance

_{Reg}*SS*which is quite a small association indeed. Whereas this effect is “significant”, it certainly isn’t very “large”.

_{T}**Observation**: From Property 2, we see that the coefficient of determination *r ^{2}* is a measure of the accuracy of the predication of the linear regression model.

*r*has a value between 0 and 1, with 1 indicating a perfect fit between the linear regression model and the data.

^{2}**Definition 3**: The **standard error of the estimate **is defined as

**Observation**: The second assertion in Property 4 can be restated as

For large samples ≈ 1 and so

Note that if *r* = .5, then

which indicates that the standard error of the estimate is still 86.6% of the standard error that doesn’t factor in any information about *x*; i.e. having information about *x* only reduces the error by 13.4%. Even if *r* = .9, then *s*_{y.x} = .436·s_{y}, which indicates that information about *x* reduces the standard error (with no information about *x*) by only a little over 50%.

a) The sums of the y values is equal to the sum of the ŷ values; i.e. =

b) The mean of the y values and ŷ values are equal; i.e. ȳ = the mean of the ŷ_{i}

c) The sums of the error terms is 0; i.e. = 0

d) The correlation coefficient of *x* with ŷ is sign(b); i.e. *r*_{xŷ} = sign(*r*_{xy})

e) The correlation coefficient of y with ŷ is the absolute value of the correlation coefficient of *x* with y; i.e. = ||

f) The coefficient of determination of y with ŷ is the same as the correlation coefficient of *x* with y; i.e. =

**Observation**: Click here for the proofs of the various properties described above.

Hi Charles,

I would appreciate some guidance- I am helping with a study designed to evaluate whether a preoperative score on a certain scale (continuous scores from 1-100) can predict postoperative outcomes (also scored on continuous scales from 1-100). This seems like a fairly straightforward regression situation to me, but using the realstats pack I seem to be running into issues since I have 1 independent variable (pre-op score) and multiple dependent variables (post-op scores). Can you suggest an analysis strategy?

Thanks!

Amanda,

You can multiple regressions, one for each dependent variable. This should be sufficient if there is little correlation between the dependent variables.

Alternatively, you can use multivariate multiple linear regression. The initial output is identical to that from separate multiple regressions, but additional analyses are performed. The Real Statistics Resource Pack doesn’t support this second step yet.

Charles

Thank you!

Hi Charles,

Is there a way to determine order of regression analysis (if data requires first order regression or second order in other words linear regression or polynomial regression) using realstats?

I have multiple variables and xl does not have a correct way to judge if data requires either linear regression or polynomial regression other than looking at p-value, which in my case 0.05.

Yask,

I don’t know of a simple way to determine the optimum order of polynomial regression model. I will look into this for a future enhancement.

To make matters worse you could have interactions: y = x^3 + z^3 + x^2 + x^3*z^2 + xz + x + z (with two independent variables x and z).

Charles

Hi Charles,

I am running a regression test involving life expectancy as the dependent variable and am examining the independent variables: income, pharamacuetical spending… as variables affecting life exp. Now how do I run the regression, do I also compile data on life expectancy and put that as the dependent variable in Stata?

Amy

Amy,

Sorry, but I don’t use Stata and so can’t answer your question. You can run the regression data analysis from Excel or from the Real Statistics Resource Pack.

Charles

Hi Amy,

have you tried SPSS Statistic? Its a software available online. You can use it to run your data. you can also check youtube on how to handle SPSS statistic.

Hello Mr. Charles,

I have figures only for independent variable. So how can I take dependent variable values in order to conduct correlation and regression test.

You can’t. You need a dependent variable to perform regression. Perhaps one of the independent variables is really a dependent variable.

Charles

Dear Charles,

I have a sample of 30 to measure the factors constraining to the adoption of technology. I am thinking to run regression analysis to adoption rate ( if it is more that 50% considered as 1 and less than 50 is 0 taking 50 percent adoption as threshold limit).

to measure the constraining factors , I used the 5 point likert scale. (highly significant to least significant) and already extracted important variables using principal component analysis.

Now what kind of regression analysis should I use to measure the relative importance of each factors ?. Linear or multiple ?

Nadee

Dear Nadee,

When you say “multiple” I assume that you mean “multiple linear regression”, which just means that you have more than one independent variable. When you have only one independent variable often the term “linear regression” or “simple linear regression” is used. Since you say that you have multiple factors, you would often use multiple linear regression.

Since your outcome (dependent variable) could be viewed as dichotomous (0 or 1), you might find that logistic regression gives a better fit for the data. You can compare AIC values for this.

Charles

hi Charles,

I had a set of data to do the regression, but the result I have got from this addin is different to EXCEL. Could you pls let me know why is it?

Thanks.

Hi Dennis,

Can you send me the data you used so that I can find out what went wrong? My email address is czaiontz@gmail.com.

Charles

Hello Charles,

I must commend your work, I have been following your materials here and I must confess, they are awesomely helpful in my ongoing project research. However I humbly ask for a more tailored explanation on the following.

I am doing a study on the impact of some sectors on the Economic Growth of Nigeria using the GDP as a measure. I collected sample on GDP (response variable) and 5 independent variables over a period of n = 32 years. So far I have been able to run my analysis using SAS, my problem however is how to manually explain the following computations.

1. The coefficients of regression Bo, B1, B2,…, B5.

2. Formular to calculate the correlation coefficients.

3. How to calculate my R^2 manually.

I need these because in my school, students are required to be able to explain how values can manually be computed using calculator (other than software). I await your kind support.

Thanks for the great work.

Wilfred Okwudili,

Department of Statistics,

Federal University of Technology, Owerri, Imo State, Nigeria.

Hello Wilfred,

1. The B coefficients are calculated using matrix operations as described in Property 1 of http://www.real-statistics.com/multiple-regression/least-squares-method-multiple-regression/.

2. The correlation coefficient for any pair of data sets is given by the formula in Definition 2 of http://www.real-statistics.com/correlation/basic-concepts-correlation/.

The correlation matrix (for all the pairwise correlation coefficients) is calculated using Property 0 of http://www.real-statistics.com/multiple-regression/least-squares-method-multiple-regression/.

3. R^2 can be calculated in many ways; e.g. using Property 2 in http://www.real-statistics.com/multiple-regression/multiple-regression-analysis/, where Y-hat is calculated as in Property 1 of http://www.real-statistics.com/multiple-regression/least-squares-method-multiple-regression/ and r_yy-hat is calculated as described in item 2 above.

Charles