Stat 328 Regression Summary
Simple Linear Regression Model
The basic (normal) "simple linear regression" model says that a response/output variable
` depends on an explanatory/input/system variable _ in a "noisy but linear" way. That is,
one supposes that there is a linear relationship between _ and mean ` ,
K`Š_ º ?> ?? _
and that (for fixed _) there is around that mean a distribution of ` that is normal. Further,
the model assumption is that the standard deviation of the response distribution is
constant in _. In symbols it is standard to write
` º ?> ?? _ B
where B is normal with mean > and standard deviation R . This describes one ` . Where
several observations `P with corresponding values _P are under consideration, the
assumption is that the `P (the BP ) are independent. (The BP are conceptually equivalent to
unrelated random draws from the same fixed normal continuous distribution.) The model
statement in its full glory is then
`P º ?> ?? _P BP for P º ? @ U
BP for P º ? @ U are independent normal î> R @ ï random variables
The model statement above is a perfectly theoretical matter. One can begin with it, and
for specific choices of ?> ?? and R find probabilities for ` at given values of _. In
applications, the real mode of operation is instead to take U data pairs
î_? î`? ïîî_@ î`@ ïîîî_U î`U ï and use them to make inferences about the parameters
?> î ?? and R and to make predictions based on the estimates (based on the empirically
fitted model).
Descriptive Analysis of Approximately Linear î_ `ï Data
After plotting î_î`ï data to determine that the "linear in _ mean of `" model makes some
sense, it is reasonable to try to quantify "how linear" the data look and to find a line of
"best fit" to the scatterplot of the data. The sample correlation between ` and _
U
bî_P Ÿ _ïî`P Ÿ `ï
Pº?
Yº
U
U
Pº?
Pº?
bî_P Ÿ _ï@ ¤ b î`P Ÿ `ï@
is a measure of strength of linear relationship between _ and ` .
1
Calculus can be invoked to find a slope ?? and intercept ?> minimizing the sum of
U
squared vertical distances from data points to a fitted line, b î`P Ÿ î?> ?? _P ïï@ . These
Pº?
"least squares" values are
U
bî_P Ÿ _ïî`P Ÿ `ï
I? º
Pº?
U
b î_P Ÿ _ï@
Pº?
and
I> º ` Ÿ I? _
It is further common to refer to the value of ` on the "least squares line" corresponding to
_P as a fitted or predicted value
‘` P º I> I? _P
One might take the difference between what is observed (`P ) and what is "predicted" or
"explained" (‘`P ) as a kind of leftover part or "residual" corresponding to a data value
LP º `P Ÿ ‘` P
U
The sum bî`P Ÿ `ï@ is most of the sample variance of the U values `P . It is a measure of
Pº?
raw variation in the response variable. People often call it the "total sum of squares"
and write
U
tt[V[ º cî`P Ÿ `ï@
Pº?
U
The sum of squared residuals bL@P is a measure of variation in response remaining
Pº?
unaccounted for after fitting a line to the data. People often call it the "error sum of
squares" and write
U
U
Pº?
Pº?
ttf º cL@P º cî`P Ÿ ‘` P ï@
One is guaranteed that ttu V[ ¾ ttf . So the difference ttu V[ Ÿ ttf is a nonnegative measure of variation accounted for in fitting a line to î_î`ï data. People often
call it the "regression sum of squares" and write
tts º tt[V[ Ÿ ttf
2
The coefficient of determination expresses tts as a fraction of ttu V[ and is
s@ º
tts
ttu V[
which is interpreted as "the fraction of raw variation in ` accounted for in the model
fitting process."
Parameter Estimates for SLR
The descriptive statistics for î_î`ï data can be used to provide "single number estimates"
of the (typically unknown) parameters of the simple linear regression model. That is, the
slope of the least squares line can serve as an estimate of ?? ,
‘ ? º I?
?
and the intercept of the least squares line can serve as an estimate of ? > ,
‘ > º I>
?
The variance of ` for a given _ can be estimated by a kind of average of squared residuals
Z@L º
U
?
ttf
cL@P º
U Ÿ @ Pº?
UŸ@
Of course, the square root of this "regression sample variance" is ZL º Z@L and serves as
a single number estimate of R .
Interval-Based Inference Methods for SLR
The normal simple linear regression model provides inference formulas for model
parameters. Confidence limits for ?? (the slope of the line relating mean ` to _ ... the
rate of change of average ` with respect to _) are
ZL
I? ¢ [
U
bî_P Ÿ _ï@
Pº?
where [ is a quantile of the [ distribution with L º U Ÿ @ degrees of freedom. Dielman
U
calls the ratio ZL ì bî_P Ÿ _ï@ the standard error of I? and uses the symbol ZI? for it. In
Pº?
these terms, the confidence limits for ?? are
I? ¢ [ZI?
3
Confidence limits for K`Š_ º ?> € ?? _ (the mean value of ` at a given value _) are
?
î_ Ÿ _ï@
îI> I? _ï ¢ [ZL U
U
bî_P Ÿ _ï@
Pº?
One can abbreviate îI> I? _ï as ‘` , and Dielman calls ZL
?
U
î_Ÿ_ï@
U
b î_P Ÿ_ï@
the standard
Pº?
error of the mean and uses the symbol ZT for it. In these terms, the confidence limits for
K`Š_ are
‘` ¢ [ZT
(Note that by choosing _ º >, this formula provides confidence limits for ?> , though this
parameter is rarely of independent practical interest.)
Prediction limits for an additional observation ` at a given value _ are
?
î_ Ÿ _ï@
îI> I? _ï ¢ [ZL ?
U
U
bî_P Ÿ _ï@
Pº?
Dielman calls ZL
?
?
U
î_Ÿ_ï@
U
b î_P Ÿ_ï@
the standard error of prediction and uses the symbol
Pº?
ZW for it. It is on occasion useful to notice that
ZW º Z@L Z@T
Using the ZW notation, prediction limits for an additional ` at _ are
‘` ¢ [ZW
Hypothesis Tests and SLR
The normal simple linear regression model supports hypothesis testing. H0 :?? º # can
be tested using the test statistic
u º
I? Ÿ #
ZL
U
b î_P Ÿ_ï@
º
I? Ÿ #
ZI?
Pº?
4
and a [UŸ@ reference distribution. H0 :K`Š_ º # can be tested using the test statistic
u º
îI> I? _ï Ÿ #
ZL
?
U
º
î_Ÿ_ï@
U
‘` Ÿ #
ZT
b î_P Ÿ_ï@
Pº?
and a [UŸ@ reference distribution.
ANOVA and SLR
The breaking down of ttu V[ into tts and ttf can be thought of as a kind of
"analysis of variance" in `. That enterprise is often summarized in a special kind of
table. The general form is as below.
Source
Regression
Error
Total
SS
tts
ttf
tt[V[
ANOVA Table (for SLR)
df
MS
?
n ts º ttsì?
U Ÿ @ n tf º ttfìîU Ÿ @ï
UŸ?
F
g º n tsìn tf
In this table the ratios of sums of squares to degrees of freedom are called "mean
squares." The mean square for error is, in fact, the estimate of R@ (i.e. n tf º Z@L ).
As it turns out, the ratio in the "F" column can be used as a test statistic for the hypothesis
H0 :?? º 0. The reference distribution appropriate is the g? UŸ@ distribution. As it turns
out, the value g º n tsìn tf is the square of the [ statistic for testing this hypothesis,
and the g test produces exactly the same W-values as a two-sided [ test.
Standardized Residuals and SLR
The theoretical variances of the residuals turn out to depend upon their corresponding _
values. As a means of putting these residuals all on the same footing, it is common to
"standardize" them by dividing by an estimated standard deviation for each. This
produces standardized residuals
LP
L¥P º
@
ZL ? Ÿ U? Ÿ Uî_P Ÿ_ï
b î_P Ÿ_ï@
Pº?
These (if the normal simple linear regression model is a good one) "ought" to look as if
they are approximately normal with mean > and standard deviation ?. Various kinds of
plotting with these standardized residuals (or with the raw residuals) are used as means of
"model checking" or "model diagnostics."
5
Multiple Linear Regression Model
The basic (normal) "multiple linear regression" model says that a response/output
variable ` depends on explanatory/input/system variables _? _@ _R in a "noisy but
linear" way. That is, one supposes that there is a linear relationship between
_? _@ _R and mean ` ,
K`Š_? _@ , _R
º ?> ?? _? ?@ _@ ?R _R
and that (for fixed _? _@ _R ) there is around that mean a distribution of ` that is
normal. Further, the standard assumption is that the standard deviation of the response
distribution is constant in _? _@ _R . In symbols it is standard to write
` º ?> ?? _? ?@ _@ ?R _R B
where B is normal with mean > and standard deviation R . This describes one ` . Where
several observations `P with corresponding values _?P _@P _RP are under
consideration, the assumption is that the `P (the BP) are independent. (The BP are
conceptually equivalent to unrelated random draws from the same fixed normal
continuous distribution.) The model statement in its full glory is then
`P º ?> ?? _?P ?@ _@P ?R _RP BP for P º ? @ U
BP for P º ? @ U are independent normal î> R @ ï random variables
The model statement above is a perfectly theoretical matter. One can begin with it, and
for specific choices of ?> ?? ?@ ?R and R find probabilities for ` at given values of
_? _@ _R . In applications, the real mode of operation is instead to take U data
vectors î_?? _@? _R? `? ï î_?@ _@@ _R@ `@ ï î_?U _@U _RU `U ï and use
them to make inferences about the parameters ?> ?? ,?@ ?R and R and to make
predictions based on the estimates (based on the empirically fitted model).
Descriptive Analysis of Approximately Linear î_? ß _@ ß
ß _R ß `ï Data
Calculus can be invoked to find coefficients ?> ?? ?@ ?R minimizing the sum of
squared vertical distances from data points in îR ?ï dimensional space to a fitted
U
surface, bî`P Ÿ î?> ?? _?P ?@ _@P Pº?
?R _RP ïï@ . These "least squares" values
DO NOT have simple formulas (unless one is willing to use matrix notation). And in
particular, one can NOT simply somehow use the formulas from simple linear regression
in this more complicated context. We will call these minimizing coefficients
I> I? I@ IR and need to rely upon JMP to produce them for us.
6
It is further common to refer to the value of ` on the "least squares surface"
corresponding to _?P _@P _R P as a fitted or predicted value
‘`P º I> I? _?P I@ _@P IR _RP
Exactly as in SLR, one takes the difference between what is observed (`P ) and what is
"predicted" or "explained" (‘`P ) as a kind of leftover part or "residual" corresponding to a
data value
LP º `P Ÿ ‘` P
U
The total sum of squares, ttu V[ º b î`P Ÿ `ï@ , is (still) most of the sample variance
Pº?
of the U values `P and measures raw variation in the response variable. Just as in SLR,
U
the sum of squared residuals bL@P is a measure of variation in response remaining
Pº?
unaccounted for after fitting the equation to the data. As in SLR, people call it the error
sum of squares and write
U
U
Pº?
Pº?
ttf º cL@P º cî`P Ÿ ‘` P ï@
(The formula looks exactly like the one for SLR. It is simply the case that now ‘`P is
computed using all R inputs, not just a single _.) One is still guaranteed that
ttu V[ ¾ ttf . So the difference ttu V[ Ÿ ttf is a non-negative measure of
variation accounted for in fitting the linear equation to the data. As in SLR, people call it
the regression sum of squares and write
tts º tt[V[ Ÿ ttf
The coefficient of (multiple) determination expresses tts as a fraction of ttu V[ and
is
s@ º
tts
ttu V[
which is interpreted as "the fraction of raw variation in ` accounted for in the model
fitting process." This quantity can also be interpreted in terms of a correlation, as it turns
out to be the square of the sample linear correlation between the observations `P and the
fitted or predicted values ‘`P .
Parameter Estimates for MLR
The descriptive statistics for î_? _@ _R `ï data can be used to provide "single
number estimates" of the (typically unknown) parameters of the multiple linear regression
model. That is, the least squares coefficients I> I? I@ IR serve as estimates of the
parameters ?> ?? ?@ ?R . The first of these is a kind of high-dimensional "intercept"
7
and in the case where the predictors are not functionally related, the others serve as rates
of change of average ` with respect to a single _, provided the other _'s are held fixed.
The variance of ` for a fixed values _? _@ _R can be estimated by a kind of average
of squared residuals
Z@L º
U
?
ttf
cL@P º
U Ÿ R Ÿ ? Pº?
UŸRŸ?
The square root of this "regression sample variance" is ZL º Z@L and serves as a single
number estimate of R .
Interval-Based Inference Methods for MLR
The normal multiple linear regression model provides inference formulas for model
parameters. Confidence limits for ?Q (the rate of change of average ` with respect to _Q )
are
IQ ¢ [ZIQ
where [ is a quantile of the [ distribution with L º U Ÿ R Ÿ ? degrees of freedom and ZIQ
is a standard error of IQ . There is no simple formula for ZIQ , and in particular, one can
NOT simply somehow use the formula from simple linear regression in this more
complicated context. It IS the case that this standard error is a multiple of ZL , but we will
have to rely upon JMP to provide it for us.
Confidence limits for K`Š_? _@ ,_R º ?> € ?? _? € ?@ _@ €
value of ` at a particular choice of the _? _@ _R ) are
€ ?R _R (the mean
‘` ¢ [ZT
where ZT is a standard error for ‘` and depends upon which values of _? _@ _R are
under consideration. There is no simple formula for ZT and in particular one can NOT
simply somehow use the formula from simple linear regression in this more complicated
context. It IS the case that this standard error is a multiple of ZL , but we will have to rely
upon JMP to provide it for us.
Prediction limits for an additional observation ` at a given vector î_? ß _@ ß á ß _R ï
are
‘` ¢ [ZW
where ZW is a standard error of prediction. It remains true (as in SLR) that
ZW º Z@L Z@T but otherwise there are no simple formulas for it, and in particular one
can NOT simply somehow use the formula from simple linear regression in this more
complicated context.
8
Hypothesis Tests and MLR
The normal multiple linear regression model supports hypothesis testing. H0 :?Q º # can
be tested using the test statistic
u º
IQ Ÿ #
ZIQ
and a [UŸRŸ? reference distribution. H0 :K`Š_? _@ statistic
u º
, _R
º # can be tested using the test
‘` Ÿ #
ZT
and a [UŸRŸ? reference distribution.
ANOVA and MLR
As in SLR, the breaking down of ttu V[ into tts and ttf can be thought of as a kind
of "analysis of variance" in `, and summarized in a special kind of table. The general
form for MLR is as below.
Source
Regression
Error
Total
ANOVA Table (for MLR Overall F Test)
SS
df
MS
F
tts R
n ts º ttsìR
g º n tsìn tf
ttf
U Ÿ R Ÿ ? n tf º ttfìîU Ÿ R Ÿ ?ï
tt[V[ U Ÿ ?
(Note that as in SLR, the mean square for error is, in fact, the estimate of R @ (i.e.
n tf º Z@L ).)
As it turns out, the ratio in the "F" column can be used as a test statistic for the hypothesis
º ?R º 0. The reference distribution appropriate is the gRUŸRŸ?
H0 :?? º ?@ º
distribution.
"Partial F Tests" in MLR
It is possible to use ANOVA ideas to invent F tests for investigating whether some whole
group of ? 's (short of the entire set) are all >. For example one might want to test the
hypothesis
H0 :?S? º ?S@ º
º ?R º 0
9
(This is the hypothesis that only the first S of the R input variables _P have any impact on
the mean system response ... the hypothesis that the first S of the _'s are adequate to
predict ` ... the hypothesis that after accounting for the first S of the _'s, the others do not
contribute "significantly" to one's ability to explain or predict `.)
If we call the model for ` in terms of all R of the predictors the "full model" and the
model for ` involving only _? through _S the "reduced model" then an g test of the
above hypothesis can be made using the statistic
g º
îttsFull Ÿ ttsReduced ïìîR Ÿ Sï
n tfFull
and an gRŸSUŸRŸ? reference distribution. ttsFull ¾ ttsReduced so the numerator here is
non-negative.
Finding a W-value for this kind of test is a means of judging whether s @ for the full model
is "significantly"/detectably larger than s @ for the reduced model. (Caution here,
statistical significant is not the same as practical importance. With a big enough data set,
essentially any increase in s@ will produce a small W-value.) It is reasonably common to
expand the basic MLR ANOVA table to organize calculations for this test statistic. This
is
(Expanded) ANOVA Table (for MLR)
Source
Regression
_? ÞÞÞ _S
SS
ttsFull
ttsRed
df
R
W
MS
n tsFull º ttsìR
F
g º n tsFull ìn tfFull
_S? ÞÞÞ _R Š_S ÞÞÞ _W
Error
Total
ttsFull Ÿ ttsRed
ttfFull
tt[V[
RŸS
UŸRŸ?
UŸ?
îttsFull Ÿ ttsRed ïìîR Ÿ Sï
n tfFull º ttfFull ìîU Ÿ R Ÿ ?ï
îttsFull ŸttsRed ïìîRŸSï
n tfFull
Standardized Residuals in MLR
As in SLR, people sometimes wish to standardize residuals before using them to do
model checking/diagnostics. While it is not possible to give a simple formula for the
"standard error of LP " without using matrix notation, most MLR programs will compute
these values. The standardized residual for data point P is then (as in SLR)
LP
L¥P º
standard error of LP
If the normal multiple linear regression model is a good one these "ought" to look as if
they are approximately normal with mean > and standard deviation ?.
10
Intervals and Tests for Linear Combinations of ? 's in MLR
It is sometimes important to do inference for a linear combination of MLR model
coefficients
m º J> ?> J? ?? J@ ?@ J JR ?R
(where J> J? JR are known constants). Note, for example, that K`Š_? _@ ,_R is of this
form for J> º ? J? º _? J@ º _@ and JR º _R . Note too that a difference in mean
responses at two sets of predictors, say î_? _@ _R ï and î_•? _•@ _•R ï is of this
form for J> º > J? º _? Ÿ _•? J@ º _@ Ÿ _•@ and JR º _R Ÿ _•R .
An obvious estimate of m is
‘ º J> I> J? I? J@ I@ J JR IR
m
Confidence limits for m are
‘ ¢ [îstandard error of mï
‘
m
‘" This standard error is a multiple of
There is no simple formula for "standard error of m
‘ and its
Z, but we will have to rely upon JMP to provide it for us. (Computation of m
standard error is under the "Custom Test" option in JMP.)
H0 :m º # can be tested using the test statistic
u º
‘Ÿ#
m
‘
standard error of m
and a [UŸRŸ? reference distribution. Or, if one thinks about it for a while, it is possible to
find a reduced model that corresponds to the restriction that the null hypothesis places on
the MLR model and to use a "S º R Ÿ ?" partial g test (with ? and U Ÿ R Ÿ ? degrees of
freedom) equivalent to the [ test for this purpose.
11