1
Ordinary Least Squares Using Linear Algebra
The objective of regression is to explain the
behavior of one variable (dependent variable),
Y, by using the values of some other variable(s)
(explanatory or independent variable(s)). This
example has only one independent variable, X.
You collect a sample of 10 observations:
Y
X
20
1
24
2
23
3
27
4
30
5
29
6
35
7
36
8
38
9
40
10
Each observation is denoted as i and your
sample size is n  10. The following figure
represents the relationship between X and Y
from your sample. There appears to be a
positive relationship between the two variables,
i. e., as X increases, Y also increases.
50
40
30
Y
20
0
= Y'Y  Y ' X   Y ' X   ( X  ) ' X 
= Y'Y  2Y ' X    ' X'X 
The first-order necessary condition for a
minimum and respective solution is derived as
follows:
ε'ε
= 2 X'Y  2 X'Xβ = 0

  X'X  X'Xβ =  X'X  X'Y
-1
1
2
3
4
5
6
7
8
9 10
X
After looking at the above picture, you decide
to estimate Y from X using a linear regression
model like the following:
Yi   0  1 X i   i , i  1, 2,...,10
 i ~ N (0,  2 )
Using matrix notation allows us to simplify the
way the above model is written:
(2)
The sum of the squared errors in matrix algebra
is denoted by ε'ε . But what is ε ? From
equation (2), ε = Y  X  . Thus to minimize
the sum of the squared errors we proceed as
follows:
min ε'ε = Y  X   ' Y  X  
 X'Xβ = X'Y
10
(1)
The objective of OLS is to find the value of β
that minimizes the sum of the squared errors.
Y  Xβ  ε,
ε ~ N (0,  2 I )
where Y is a 10 by 1 vector of observations on
Y, X is a 10 by 2 matrix of observations on X
(in the example X is full rank, thus the inverse
of X’X exists; OLS can be used when X is not
full rank but that is beyond the scope of your
class), β is a 2 by 1 vector of parameters that
you wish to estimate, and ε is a 10 by 1 vector
of errors. In matrix algebra your data set now
looks like this:
 1 
 20 
1 1 
 
 24 
1 2 
 , β =  0  , ε =   2 
Y =  , X = 
 
 
 


 1
 
 


 40 
1 10 
10 
Created by Rita Carreira
-1
 Iβ =  X'X  X'Y
-1
-1
 βˆOLS = bOLS =  X'X  X'Y
where I is the identity matrix. The secondorder sufficient condition for a minimum is a
non-negative definite Hessian matrix (the
Hessian is the matrix of second derivatives).
Now that we have the OLS estimator we can
use matrix algebra to compute the estimate of
the parameters for our model:
10 55 
 0.4667 0.0667 
X'X  
, (X'X ) 1  


55 385
 0.0667 0.0121 
 302  ˆ
 18 
X'Y  
, βOLS  (X'X ) 1 X'Y  


1844
 2.22
OLS estimation using matrix algebra can be
computed in SAS using PROC IML. The
following code was used in the estimation of
the parameters for our model (note that most of
the code contains comments, which are
optional):
2
Ordinary Least Squares Using Linear Algebra
dm 'log;clearoutput; clear;';
options ps=50 ls=70 pageno=1;
run;
***********************************
* Author: Rita Carreira
*
* Date:
November 6,2002
*
* Purpose: OLS using Proc IML
*
***********************************
*Read data from text file;
data OLS;
infile "a:\OLS.txt";
input Yi Xi;
run;
*Initiate Proc IML;
Proc IML; use OLS;
read all var{Yi} into Y;
read all var{Xi} into X1;
/*matrix X1 only contains the Xis,
we need to get the intercept in it, so
we'll create a vector of 1s called J,
which has 10 rows and 1 column*/;
J=repeat(1,10,1);
/*Create a new X matrix by sticking
J and X1 together*/;
X=J||X1;
and the option “I” to get the inverse of X’X,
parameter estimates, and SSE:
SAS CODE:
proc reg data=OLS;
model Yi=Xi/XPX I ;
run;
SAS output:
The REG Procedure
Model: MODEL1
Model Crossproducts X'X X'Y Y'Y
Variable
Intercept
Xi
Yi
*Compute X'Y, which I will call XPY;
XPY=T(X)*Y;
*Compute estimate of parameters;
betahat=IXPX*XPY;
/*If you don't tell SAS to print it,
you will not get any output*/;
print betahat;
run;
quit;
The output from PROC IML is:
The SAS System
1
15:50 Wednesday, November 6, 2002
BETAHAT
18
2.2181818
To confirm your results run PROC REG on the
data set OLS and use the option “XPX” to get
the model cross-products (X’X, X’Y, and Y’Y)
Created by Rita Carreira
Xi
55
385
1844
Yi
302
1844
9540
Dependent Variable: Yi
X'X Inverse, Parameter Estimates, and SSE
Variable Intercept
Intercept 0.4666667
Xi
-0.0666667
Yi
18
*Compute X'X, which I will call XPX;
XPX=T(X)*X;
/*Compute the inverse of X'X, which I will
call IXPX*/;
IXPX=inv(XPX);
Intercept
10
55
302
Xi
-0.066666667
0.01212121
2.2181818182
Yi
18
2.2181818
13.6727273
Analysis of Variance
Source
Model
Error
Corrected
Total
Sum of
Mean
DF Squares
Square F Value
Pr>F
1 405.92727 405.927 237.51 <.0001
8 13.67273 1.70909
9
419.60000
Root MSE
1.30732
Dependent Mean 30.20000
Var
4.32888
R-Square
Adj R-Sq
0.9674
0.963 Coeff
Parameter Estimates
Variable
Intercept
Xi
Parameter Standard
DF Estimate
Error
1 18.00000 0.89307
1 2.21818
0.14393
t Value Pr>|t|
20.16 <.0001
15.41 <.0001