Simple Linear Regression
Previously we’ve discussed the notion of inference – using sample statistics to find
out information about a population. This was done in the context of the mean of a
random variable. In addition we talked early on about correlation between two
variables. With the correlation coefficient and Chi-Square tests we were able to see if
there were relationships between random variables.
Here we want to take this a step further and formalize the relationship between two
variables, and then extend this to multivariate analysis. This is done through the
concept of regression analysis.
Suppose we are interested in the relationship between two variables: length of stay and
hospital costs. We think that LOS causes costs: a longer LOS results in higher costs.
Note that with correlation there are no claims made about what causes what – just that
they move together. Here we are taking it further by “modeling” the direction of the
relationship. How would we go about testing this? If we take a sample of individual
stays and measure their LOS and cost, we might get the following:
Stay
LOS
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Cost
3
5
2
3
3
5
5
3
1
2
2
5
1
3
4
2
1
1
1
6
2614
4307
2449
2569
1936
7231
5343
4108
1597
4061
1762
4779
2078
4714
3947
2903
1439
820
3309
5476
1
Just looking at the data there looks to be a positive relationship, individuals with more
LOS seem to have a higher cost; but not always.
Another way of looking at the data would be with a scatter diagram:
Scatter Diagram: LOS vs Cost
8000
7000
$Cost
6000
5000
4000
3000
2000
1000
0
0
1
2
3
4
5
6
7
LOS
Ignore the trend line for now.
Cost is on the Y axis, LOS is on the X. It looks like there is a positive relationship
between cost and LOS. Simply eyeballing the data, it looks like the dots go up and to the
right.
The trendline basically connects the dots as best as possible. Note that the slope of this
line will tell you the marginal impact of LOS on charges: what happens to costs if LOS
increases by 1 unit? This line intersects the Y axis just above zero. This would be the
predicted cost if LOS was zero.
If the correlation coefficient between LOS and costs was 1 then all the dots would be on
this line. Note that for some observations the line is real close to the dot, while for
others it is pretty far away. Let the distance between any given dot and the line be the
error in the trend line. The trend line is drawn so that these errors are minimized. Since
errors above the line will be positive, while errors below the line will be negative we
have to be careful – positive errors will tend to wash out negative errors. Thus a strategy
in estimating this line would be to draw the line such that we minimize the sum of the
squared error term.
This is known as The Least Squares Method.
2
I.
The Logic
The idea of Least Squares is as follows:
In theory our relationship is:
Y =  o +  1X + 
Y is the dependent variable – the thing we are trying to explain.
X is the independent variable – what is doing the explaining.
o and 1 are population parameters that we are interested in knowing
In our case Y is the charge, X is LOS. o is the intercept (where the line crosses the Y
axis), and 1 is the slope of the line. This coefficient 1 is the marginal impact of X on Y.
These are population parameters that we do not know. From our sample we estimate the
following:
Y = bo + b1X + e
Note I’ve switched to non-Greek letters since we are now dealing with the sample. So bo
is an estimator for o and b1 is an estimator for 1. e is the error term reflecting the fact
that we will not always be exactly on our line. If we wanted to get a predicted value for
Y (costs) we would use:
^
Y i  bo  b1 Xi {the ^ means predicted value)
Note the error term is gone. So this is the equation for the estimated line. So suppose
that bo=3 and b1 = 2, then someone with a LOS of 5 days would be predicted to have
3+2*5=$13 in charges, etc.
Least squares finds bo and b1 by minimizing the sum of the squared difference between
the actual and predicted value for Y:
II.
Specifics
n
Sum of squared difference =
 (Y  Yˆ )
i
i
2
i 1
Substituting:
n
 (Y  Yˆ )
i
i 1
i
n
2
  [Yi  (bo  b1 Xi )] 2
i 1
3
Thus least squares finds bo and b1 to minimize this expression. We are not going to go
into the details here of how this is done, but we will focus on the intuition of what is
going on.
The easiest way to think about it is to go back to the scatter diagram: least squares draws
the trend line to connect the dots the best way possible. We choose the parameters to
minimize the size of our mistakes or errors.
III.
How do we do this in Excel?
Excel can do both simple (one independent variable) and multiple (more than one)
regression.
You need the Data Analysis Toolpack to do it.
Load the Analysis ToolPak
1. Click the File tab, click Options, and then click the Add-Ins category.
2. In the Manage box, select Excel Add-ins and then click Go.
3. In the Add-Ins available box, select the Analysis ToolPak check box, and then click OK.
For the Mac you have to use a different add-in called Statplus:
http://www.analystsoft.com/en/products/statplusmacle/
4
SUMMARY OUTPUT
Regression Statistics
Multiple R
0.807337
R Square
0.651794
Adjusted R Square 0.632449
Standard Error
995.4983
Observations
20
ANOVA
df
Regression
Residual
Total
Intercept
LOS
1
18
19
SS
MS
F
Significance F
33390795 33390795 33.69347
1.69E-05
17838305 991016.9
51229100
CoefficientsStandard Error t Stat
P-value Lower 95% Upper 95%
997.4659
465.7353 2.141701 0.046147
18.99164 1975.94
818.8394
141.0671 5.804607 1.69E-05
522.4681 1115.211
What does all this mess mean?
Skip the first two sections for now and just look at the bottom part. The numbers under
Coefficients are our coefficient estimates:
Costi = 997.5 + 818.8*LOSi + ei
So we would predict that a patient starts with a cost of 997.5 and each day adds 818.8 to
the cost.
The least squares estimate of the effect of LOS on cost is $818.8 per day. So someone
with a LOS of 5 days is predicted to have: 997.5 + 818.8*5 = $5091.5 in costs.
The statistics behind all this get pretty complicated, but the interpretation is easy. And
note that we can now add as many variables as we want and the coefficient estimate for
each variable is calculated holding constant the other right-hand-side variables.
Science vs. Art in Regression
Causation
Omitted variable bias
IV.
Measures of Variation:
We now want to talk about how well the model predicted the dependent variable. Is it a
good model or not? This will then allow us to make inferences about our model.
5
The Sum of Squares
The total sum of squares (SST) is a measure of variation of the Y values around their
mean. This is a measure of how much variation there is in our dependent variable.
n
_
SST   (Yi  Y ) 2
i 1
[Note that if we divide by n-1 we would get the sample variance for Y]
The sum of squares can be divided into two components
Explained variation, or the Regression Sum of Squares (SSR) – that which is attributable
to the relationship between X and Y, and unexplained variation, or Error Sum of Squares
(SSE) – that which is attributable to factors other than the relationship between X and Y.
^
Y
Yi
(Yi  Y i ) 2  ei
^
Y i  bo  b1 Xi
_
(Yi  Y ) 2
^
(Y i  Y ) 2
_
Y
Xi
X
The dot, represents one particular actual observation (Xi,Yi), the horizontal line
_
represents the mean of Y ( Y ), the upward sloping line represents the estimated
_
regression line. The distance between Yi and Y is the total variation (SST). This is
broken into two parts, that explained by X and that not explained. The distance between
6
the predicted value of Y and the mean of Y is that part of the variation that is explained
by X. This is SSR. The distance between the predicted value of Y and the actual value
of Y is the unexplained portion of the variation. This is SSE.
Suppose that X had no effect whatsoever on Y. Then the best regression line would
simply be the mean of Y. So the predicted value of Y would always be the Mean of Y no
matter what X is. So X is doing nothing in helping us to explain Y. Then all the
variation in Y will be unexplained.
Suppose, alternatively, that the predicted value was exactly correct – the dot is on the
regression line. Then notice that all the variation in Y is being explained by the variation
in X. In other words, if you know X you know Y exactly.
^
As shown above, some of the variation in Y is due to variation in X ( (Y i  Y ) 2 ) and some
^
of the variation is not explained by variation in X ( (Yi  Y i ) 2 ).
n
^
So to get the SSR we calculate SSR   (Y i  Y ) 2
i 1
n
And to get the SSE we calculate:
^
 (Y  Y
i
i
)2
i 1
Referring back to our first regression output notice the middle table looks as follows:
ANOVA
df
Regression
Residual
Total
SS
MS
F
Significance F
1 33390795 33390795 33.69347
1.69E-05
18 17838305 991016.9
19 51229100
The third column is labeled SS (or sum of squares), then the first row is regression So
SSR = 33390795. Residual is another word for error (or leftover) so SSE = 17838305,
and the SST = 51229100. Notice that: 51229100=33390795+17838305
How do we use this information? In general, the method of Least Squares chooses the
coefficients so to minimize SSE. So we want that to be as small as possible – or
equivalently, we want SSR to be as big as possible.
Notice that the closer SSR is to SST the better our regression is doing. In a perfect world
SSR = SST: or our model explains ALL the variation in Y. So if we look at the ratio of
SSR to SST, this will tell us how our model is doing. This is known as the Coefficient of
Determination or R2.
7
R2 = SSR/SST
for our example: R2= 33390795/51229100= .652
Is this good? It depends.
Thus, 65% of the variation in charges can be explained by variation in LOS. Note that
this is pretty low since there are many other things that determine charges.
Standard Error of the Estimate
Note that for just about any regression all the data points will not be exactly on the
regression line. We want to be able to measure the variability of the actual Y from the
predicted Y. This is similar to the standard deviation as a measure of variability around a
mean. This is called The Standard Error of the Estimate
n
SYX 
SSE

n2
 (Y  Yˆ )
i
i
2
i 1
n2
Notice that this looks very much like the standard deviation for a random variable. But
here we’re looking at variation of actual values around a prediction.
For our example SYX =
17838305
=995.5
18
Note that the top table in the Excel output has the R-squared and the Standard Error
listed, among other things.
This is a measure of the variation around the fitted regression line – a loose interpretation
would be that on average the data points are about $995 off of the regression line. We
will use this in the next section to make inferences about our coefficients.
V.
Inference
We made our estimates above for the regression line based on our sample information.
These are estimates of the (unknown) population parameters. In this section we want to
make inferences about the population using our sample information.
t-test for the slope
Again, our estimate of  is b. We can show that under certain assumptions (to come in a
bit) that b is an unbiased estimator for . But as discussed above there will still be some
sampling error associated with this estimate. So we can’t conclude that =b every time,
only on average. Thus we need to take this sampling variability into account.
Suppose we have the following null and alternative hypothesis:
8
Ho: 1=0 (there is no relationship between X and Y)
H1: 1 0 (there is a relationship)
This can also be one tailed if you have some prior information to make it so.
Our test statistic will be:
t = b1-1/Sb1, where Sb1 is the standard error of the coefficient.
Sb1 = SYX/SSX
Where SSX = (Xi-Xb)2
This follows a t-distribution with n-2 degrees of freedom.
[NOTE: in general this test has n-k-1 degrees of freedom, where k is the number of righthand-side variables. In this case k=1 so it is just n-2]
The Standard error of the coefficient is the standard error of the estimate divided by the
squared deviation in X.
Again note the bottom part of the Excel output:
Intercept
LOS
Coefficient Standard
Lower
Upper
s
Error
t Stat
P-value
95%
95%
997.4659 465.7353 2.141701 0.046147 18.99164 1975.94
818.8394 141.0671 5.804607 1.69E-05 522.4681 1115.211
So our LOS coefficient is 818.8. Is this statistically different from zero?
Our test statistic is: t=(818.8-0)/.141.1 = 5.8. We can use the reported p-value: .0000169
to conclude that we would reject the null hypothesis and say that there is evidence that 1
is not zero. That is, LOS has a significant effect on charges.
The t-test can be used to test each individual coefficient for significance in a multiple
regression framework. The logic is just the same.
One could also test other hypothesis: Suppose it used to be the case that each day in the
hospital resulted in $1000 charge, is there evidence that it has changed?
Ho: 1=1000
Ha: 11000
t=(818.8-1000)/141.06 = -1.28 the pvalue associated with this is .215 – so there is a
21.5% chance we could get a coefficient of 818 or further away from 1000 if the null is
true. Thus, we would fail to reject the null and conclude that there is no evidence that the
slope is different from 1000.
9
We could also estimate a confidence interval for the slope:
b1  tn-2Sb1
Where tn-2 is the appropriate critical value for t. You can get excel to spit this out for you
as well. Just click the confidence interval box and type in the level of confidence and it
will include the upper and lower limits in the output.
For my example we are 95% confident that the population parameter 1 is between 522
and 1115.
You can also predict confidence interval. Suppose a patient was going to stay 3 days in
the hospital, what do you predict costs to be?
Point estimate: cost = 997+3*818=$3,451
Or a 95% confidence interval of the expected cost would be:
18.99 + 3*(522)=1586
1975 +3*(1115)=5321
Or we are 95% confident that the cost would be between $1,586 and $5,321
10
Multiple Regression
VI.
Introduction
In the last section we looked at the simple linear regression model where we have only
one explanatory (or independent) variable. This can be easily expanded to a multivariate
setting. Our model can be written as:
Yi = o + 1X1i +2X2i + … + kXki + I
So we would have k explanatory variables. The interpretation of the ’s is the same as in
the simple regression framework. For example 1 is the marginal influence of X1 on the
dependent variable Y, holding all the other explanatory variables constant.
This is easy to do in Excel. It is similar to simple regression except that one needs to
have all the X variables side by side in the spreadsheet.
Inference about individual coefficients is exactly the same as in simple regression.
Suppose we have the following data for 10 hospitals:
Y
Cost
X1
Size
2750
2400
2920
1800
3520
2270
3100
1980
2680
2720
X2
CEO IQ
225
200
300
350
200
250
175
400
350
275
6
37
14
33
11
21
21
22
20
16
Cost is the cost per case for each hospital, Size is the size of the hospital in the number of
beds, and CEO IQ is a scale that measures how much the administrator knows about
competitor hospitals. In this case we might expect larger hospitals to have lower costs per
case, and when the administrator has more knowledge about his competition costs will be
lower as well.
SUMMARY OUTPUT
Regression Statistics
Multiple R
0.834034
R Square
0.695612
Adjusted R Square 0.608644
Standard Error
323.9537
Observations
10
ANOVA
11
df
Regression
Residual
Total
Intercept
Size
CEO IQ
2
7
9
SS
1678818
734622
2413440
MS
F
Significance F
839409 7.998485
0.01556
104946
Coefficients Standard Error t Stat
P-value Lower 95% Upper 95%
4240.131
435.6084 9.733813 2.56E-05
3210.081 5270.181
-3.76232
1.442784 -2.60768 0.035032
-7.17395 -0.35068
-29.8955
11.66298 -2.56328 0.037372
-57.4741 -2.31699
So in this case we’d say that each bed lowers the per case cost of the hospital by $3.76,
and every one unit increase in the CEO IQ scale lowers costs by 29.90. Note that these
are not the same results we would get if we did two simple regressions:
If we only included Size:
SUMMARY OUTPUT
Regression Statistics
Multiple R
0.640237
R Square
0.409904
Adjusted R Square 0.336142
Standard Error
421.9245
Observations
10
ANOVA
df
Regression
Residual
Total
Intercept
Size
1
8
9
SS
MS
F
Significance F
989277.9 989277.9 5.557108
0.046149
1424162 178020.3
2413440
Coefficients Standard Error t Stat P-value Lower 95% Upper 95% Lower 95.0% Upper 95.0%
3804.717
522.4326 7.282694 8.53E-05
2599.984 5009.449
2599.984
5009.44
-4.3696
1.853606 -2.35735 0.046149
-8.64403 -0.09518
-8.64403
-0.0951
While if we only included CEO IQ:
SUMMARY OUTPUT
Regression Statistics
Multiple R
0.632394
R Square
0.399922
Adjusted R Square 0.324912
Standard Error
425.4781
Observations
10
12
ANOVA
df
SS
965187.2
1448253
2413440
MS
965187.2
181031.6
Coefficients Standard Error
3315.282
332.1822
-34.8896
15.11012
t Stat
9.980311
-2.30902
Regression
Residual
Total
1
8
9
Intercept
CEO IQ
F
Significance F
5.331595
0.049765
P-value
8.61E-06
0.049765
Why the changes in the coefficients?
Note the R-squared do not add up. The multiple Regression R2 is .695, and the two
simple R2’s are .40 and .41.
VII. Testing for the Significance of the Multiple Regression Model
F-test
Another general summary measure for the regression model is the F-test for overall
significance. This is testing whether or not any of our explanatory variables are
important determinants of the dependent variable. This is a type of ANOVA test.
Our null and alternative hypotheses are:
Ho: 1=2= … = k =0 (none of the variables are significant)
H1: At least one j  0
Here the F statistic is:
SSR
F=
SSE
k
n  k 1
Notice that this statistic is the ratio of the regression sum of squares to the error sum of
squares. If our regression is doing a lot towards explaining the variation in Y then SSR
will be large relative to SSE and this will be a “big” number. Whereas if the variables are
not doing much to explain Y, then SSR will be small relative to SSE and this will be a
“small” number.
This ratio follows the F distribution with k and n-k-1 degrees of freedom.
The middle portion of the Excel output contains this information (this is the model with
school and experience, not shoe size):
13
ANOVA
df
Regression
Residual
Total
SS
2 1678818
7 734622
9 2413440
MS
F
Significance F
839409 7.998485
0.01556
104946
F = (1678818/2)/(734622/(10-2-1)) = 839409/104946 = 7.998
The “Significance F” is the p-value. So we’d reject Ho and conclude there is evidence
that at least one of the explanatory variables is contributing to the model. Note that this is
a pretty weak test: it could be only one of the variables or it could be all of them that
matter, or something in between. It just tells us that something in our model matters.
VIII. Dummy Variables in Regression
Up to this point we’ve assumed that all the explanatory variables are numerical. But
suppose we think that, say, earnings might differ between males and females. How
would we incorporate this into our regression?
The simplest way to do this is to assume that the only difference between men and
women is in the intercept (that is the coefficients on all the other variables are equal for
men and women).
Wage
Men
Women
male
female
Education
14
Assume for now the only other variable that matters is education. The idea is that we
think men make more than women independent of education. That is the male intercept
(male) is greater than the female intercept (female). We can incorporate this into our
regression by creating a dummy variable for gender. Suppose we let the variable Male
=1 if the individual is a male, and 0 otherwise. Then our equation becomes:
Wagei = o + 1Educationi + 2Malei + i
So if the individual is male the variable Male is “on” and if she is female Male is “off”.
The coefficient 3 indicates how much extra the wage of males is than female (this can be
positive or negative in theory). In terms of our graph, female = o, and male = o+3.
So the dummy variable indicates how much the intercept shifts up or down for that group.
This can be done for more than two categories. Suppose we think that earnings also
differ by race then we can write:
Wagei = o + 1Educationi + 2Malei +3Blacki + 4Asiani + 5Otheri + i
Where Black is a dummy variable equal to 1 if the individual is black, Asian =1 if the
individual is Asian, other =1 for other nonwhite race. Note that white is omitted from
this group. Just like female is omitted. Thus the coefficients 3, 4, and 5 indicate how
wages differ for blacks, Asian, and other, relative to whites:
Note that if there are x different categories, we include x-1 dummy variables in our
model. The omitted group is always the comparison.
Suppose we estimate this and get:
.^
Wage = -20 + 3.12*School + 4*Male –5*black –3*Asian –3*other.
So males make $4/hour more than females
Blacks make $5 less than whites
Asians make $3 less than whites
Other races make $3 less than whites.
Note that this is controlling for differences in other characteristics. That is, a male with
the same level of schooling, experience and race will earn $4 more than the identical
female. Similarly for race.
IX.
Interaction Effects
Suppose we’re interested in explaining total costs and we think LOS and gender are the
explanatory variables. We could estimate our model and get something like:
15
SUMMARY OUTPUT
Regression Statistics
Multiple R
0.961293
R Square
0.924085
Adjusted R
Square
0.918011
Standard Error
619.0914
Observations
28
ANOVA
df
Regression
Residual
Total
Intercept
LOS
Female
2
25
27
Coefficients
638.4373
1113.62
-244.657
Significance
SS
MS
F
F
1.17E+08 58317790 152.1569
1.01E-14
9581854 383274.2
1.26E+08
Standard
Error
257.5843
65.47048
246.0593
t Stat
P-value
2.478557 0.020295
17.0095 2.97E-15
-0.9943 0.329603
Interpret this.
Now what if we think the effect of LOS on cost is different for males than females. How
might we deal with this? Note that the idea is that not only is there an intercept
difference, but there is a slope difference as well. To get at this we can interact LOS and
Female – that is create a new variable that multiplies the two together, then we would get
something like the following:
SUMMARY OUTPUT
Regression Statistics
Multiple R
0.998708
R Square
0.997417
Adjusted R
Square
0.997095
Standard Error
116.5416
Observations
28
16
ANOVA
df
Regression
Residual
Total
Intercept
LOS
Female
LOS*Female
3
24
27
Coefficients
2075.093
606.5649
-2472.39
711.2028
SS
MS
F
1.26E+08 41963822 3089.677
325966.7 13581.95
1.26E+08
Standard
Error
73.34754
23.00362
97.09693
27.24366
t Stat
28.29125
26.36823
-25.4631
26.10526
Significance
F
3.54E-31
P-value
6.06E-20
3.12E-19
7.01E-19
3.93E-19
Note the adjusted R2 increases which suggests that adding this new variable is “worth it”.
How do we interpret?
Females have costs that are $2,472 lower than males holding constant LOS. A one unit
increase in LOS increases costs for males by $606 for males, while the effect for females
is $711 LARGER. That is each day in the hospital increases costs for females by
606.5+711.2 = $1,317.7. So females start at a lower point, but increase faster with LOS
than do males.
17