Statistical Analysis
Professor Lynne Stokes
Department of Statistical Science
Lecture 5QF
Introduction to Vector and Matrix
Operations Needed for the Theory
of Quadratic Forms
1
Linear Statistical Models
Regression Model
yi   0  x i1 1  x i2  2  ...  x ik  k  ei , i  1,2,..., n
One Factor (Fixed Effects) General Linear Model
yij     i  eij , i  1,2,..., a j  1,2,..., r
Common Matrix Form
y  Xβ  e
Regression: X full column rank
GLM: X less than full column rank
2
Notation
Response Vector
 y1 
 
 y2 
y  
...
 
 yn 
Design / Regressor Matrix
 x11
x
X   21
 ...
x
 n1
x12
x 22
...
xn2
x1k 
... x 2 k 

... ... 
... x nk 
...
Error Vector
 e1 
 
 e2 
e 
...
 
 en 
General Matrices : A, B, …
3
Matrix Rank
Linear Independence
x1, x 2 ,...x r are linearly independen t if and only if
r
the only constants for which  c jx j  0 are c1  c 2  ...c r  0
j1
Can’t Express any of the Vectors as a Linear
Combination of the Other Vectors
Rank of a Matrix
Maximum Number of Linearly Independent Columns
(Row Rank = Column Rank)
Note: A square matrix with a nonzero determinant is full rank, or nonsingular.4
Special Matrices
Diagonal Matrix
d1 0
0 d
2
D  diag (d1 , d 2 ,..., d n )  
 ... ...
0 0

Identity Matrix
1
0
I  diag (1,1,...,1)  
...
0

0
... 0 

... ... 
... d n 
...
0 ... 0 
1 ... 0 

... ... ...
0 ... 1 
5
Special Matrices
Matrix of Ones
Null Matrix
1
1
J
...
1

0
0

...
0

1 ... 1 
1 ... 1 

... ... ...
1 ... 1 
0 ... 0 
0 ... 0 

... ... ...
0 ... 0 
1
 
1
1 
...
 
1
(any dimensions)
6
Matrix Operations
Addition
 
 
A  a ij
, B  bij

, A  B  a ij  bij

A and B must have the same
dimensions
Vector Multiplication
n
x1x 2   x i1x i 2
i 1
n
e.g., 1 x   x i
i 1
Matrix Multiplication
a  
 1 
a
A   2  , B  b1 b 2 ... b k  , C  AB  cij , cij  aib j
 ... 
 
7
A must have the same number of columns
a n 
as B has rows: A (n x s), B (s x k)
 
Matrix Operations
Transpose
 
A  a ij
 
, A  a ji
Interchange rows and columns
Symmetric Matrix: A (n x n) with A = A’
i.e, aij = aji
Inverse
Matrix A has an inverse, denoted A-1 if and only if (a) A is a square (n x n)
matrix and (b) A is of full (row, column) rank. Then AA-1 = A-1A = I.
A matrix inverse is unique.
8
Special Vector and Matrix
Properties
Orthogonal Vectors
Normalized Vectors
a’b = 0
xx  1  xx
Orthonormal Matrix
Symmetric Idempotent Matrix
AA  I
A  A , A 2  A
Note : then A-1 = A’
Only Full-Rank Symmetric Idempotent Matrix:
I
Note: A matrix all of whose columns are mutually orthogonal is called an
orthogonal matrix. Often “orthogonal” is used in place of “orthonormal.”
9
Eigenvalues and Eigenvectors
A is square (n x n) and symmetric: All eigenvalues and eigenvectors are real-valued.
Eigenvalues: l1, l2, …, ln
A  l jI  0 , j  1,2,..., n
(solve an nth degree polynomial
equation in l)
Eigenvectors: v1, v2, …, vn
A  l jI  v j  0
, j  1,2,..., n
Note: If all eigenvalues are distinct, all eigenvectors are mutually orthogonal and
can, without loss of generality, be normalized. If some eigenvalues have
multiplicities greater than 1, the corresponding eigenvectors can be made
10
to be orthogonal. Eigenvectors are unique up to a multiple of –1.
Eigenvalues and Rank




The rank of a symmetric matrix equals the number of
nonzero eigenvalues
All the eigenvalues of an idempotent matrix are 0 or 1
 It’s rank equals the number of eigenvalues that are
1
 The sum of its diagonal elements equals its rank
A diagonal matrix has its eigenvalues equal to the
diagonal elements of the matrix
The identity matrix has all its eigenvalues equal to 1
 Any set of mutually orthonormal vectors can be
used as eigenvectors
11
Quadratic Forms
q  xAx
A can always be assumed to be symmetric:
For any B, x’Bx = x’Ax with aij = (bij + bji)/2
n
xx  xIx   x i2 " sum of squares"
i 1
12
Assignment 3
5 0 1 
2 1 
 .5 - .5
0 6 2 
A
,
B

,
C


- .5 .5 


1
2




1 2 4
1.
2.
3.
4.
5.
Determine the rank of each of these matrices.
For each full-rank matrix, find its inverse.
Determine whether any of these matrices are orthogonal
Determine whether any of these matrices are idempotent.
Find the eigenvalues and eigenvectors of A and B.
13