1
Chapter 2 Matrix Algebra and Random Vectors
2.2 Some basics of matrix and vector algebra
(i) Definition of
r c
matrix:
An r  c matrix A is a rectangular array of rc real numbers arranged in r
horizontal rows and c vertical columns:
 a11
a
A   21
 

 ar 1
a12

a22



ar 2

a1c 
a2 c 
 .

arc 
The i’th row of A is
rowi ( A)  ai1 ai 2  aic , i  1,2,, r, ,
and the j’th column of A is
 a1 j 
a 
2j
col j ( A)   , j  1,2,, c.
  
 
 arj 
We often write A as
 
A  aij  Ar c .
(ii) Matrix addition:
Let
A  Arc
 a11 a12
a
a
 [aij ]   21 22
 


 ar1 ar 2
 a1c 
 a2 c 
  ,

 arc 
2
B  Bcs
b11 b12
b b
 [bij ]   21 22



bc1 bc 2
 b1s 
 b2 s 
  ,

 bcs 
D  Drc
 d11 d12
d
d 22
 [d ij ]   21
 


 d r1 d r 2
 d1c 
 d 2 c 
  .

 d rc 
Then,
 (a11  d11 ) (a12  d12 )
( a  d ) ( a  d )
21
22
22
A  D  [aij  d ij ]   21




 ( a r1  d r1 ) ( a r 2  d r 2 )
 pa11
 pa
pA  [ paij ]   21
 

 par1
 (a1c  d1c ) 
 (a2 c  d 2 c )
,



 (arc  d rc ) 
pa1c 
 pa2 c 
, p  R.

 

 parc 
pa12 
pa22

par 2
and the transpose of A is denoted as
At  Actr
 a11 a21
a
a22
 [a ji ]   12
 


 a1c a2 c
 a r1 
 ar 2 
  

 arc 
Example 1:
Let
 1
A
 4
3
5
3
1
B

and
8
0

7
1
0
.
1
3
Then,
 1 3
A B  
 4  8
37
5 1
 1 2
2A  
 4  2
3 2
5 2
1  0  4

0  1
 4
1 2   2

0  2
  8
4
6
6
10
and
1  4
At   3 5  .
 1 0 
(iii) Matrix multiplication:
We first define the dot product or inner product of n-vectors.
Definition of dot product:
The dot product or inner product of the n-vectors
a  a1 a2  ac 
and
 b1 
b 
b   2
  ,
 
bc 
are
c
a  b  a1b1  a2b2    ac bc   ai bi .
i 1
Example 2:
4
Let a  1 2 3 and b  5  . Then, a  b  1 4  2  5  3  6  32 .
6 
Definition of matrix multiplication:
E  E rs
 e11
e
 eij   21


 er1
 
e12
e22

er 2
 e1s 
 e2 s 
  

 ers 
1
,
1

2
0

4
 row1 ( A)  col1 ( B) row1 ( A)  col 2 ( B)
row ( A)  col ( B) row ( A)  col ( B)
1
2
2
 2




 rowr ( A)  col1 ( B) rowr ( A)  col 2 ( B)
 row1 ( A) 
row ( A)
2
col ( B) col ( B)

2
 
 1


row
(
A
)
r


 a11 a12  a1c  b11 b12
a
 b
a

a
21
22
2
c
  21 b22

 
    



 ar1 ar 2  arc  bc1 bc 2
 row1 ( A)  col s ( B) 
 row2 ( A)  col s ( B)




 rowr ( A)  col s ( B) 
 cols ( B)
 b1s 
 b2 s 
 Arc Bcs
  

 bcs 
That is,
eij  rowi ( A)  col j ( B)  ai1b1 j  ai 2 b2 j    aicbcj , i  1,, r , j  1,, s.
Example 3:
1 2 
0 1 3 
A22  
,
B

 23  1 0  2 .
3  1


Then,
 row ( A)  col1 ( B) row1 ( A)  col 2 ( B) row1 ( A)  col3 ( B)   2 1  1
E23   1


row2 ( A)  col1 ( B) row2 ( A)  col 2 ( B) row2 ( A)  col3 ( B)  1 3 11 
since
0
0
row1 ( A)  col1 ( B)  1 2   2 , row2 ( A)  col1 ( B)  3  1   1
 1
 1
1
row1 ( A)  col2 ( B )  1 2   1 , row2 ( A)  col 2 ( B)  3  11  3
0 
0 
 3 
row1 ( A)  col3 ( B)  1 2   1 , row2 ( A)  col3 ( B)  3
  2
 3 
 1   11 .
  2
5
Example 4
a31
1
1
 4 5 




 2, b12  4 5  a 31b12  24 5   8 10 
3
3
12 15
Another expression of matrix multiplication:
Arc Bcs
 row1 ( B) 
row ( B)
 col1 ( A) col 2 ( A)  colc ( A) 2 
  


rowc ( B) 
c
 col1 ( A)row1 ( B)  col 2 ( A)row2 ( B)    colc ( A)rowc ( B)   coli ( A)rowi ( B)
i 1
where coli ( A)rowi ( B) are r  s matrices.
Example 3 (continue):
 row ( B) 
A22 B23  col1 ( A) col 2 ( A) 1   col1 ( A)row1 ( B)  col 2 ( A)row2 ( B)
row2 ( B)
1
2
 0 1 3   2 0  4  2 1  1
  0 1 3    1 0  2  



3
 1
0 3 9  1 0 2   1 3 11 
Note:
 row1 ( A) 
row ( A) 
2
 and
Heuristically, the matrices A and B, 





 rowr ( A) 
col1 ( B)
col 2 ( B)  col s ( B) , can be thought as r  1 and 1 s vectors.
Thus,
Arc Bcs
 row1 ( A) 
row ( A)
2
col ( B) col ( B)  col ( B)

2
s
   1


row
(
A
)
r


6
can be thought as the multiplication of r  1 and 1 s vectors. Similarly,
Arc Bcs
 row1 ( B) 
row ( B)
2

 col1 ( A) col2 ( A)  colc ( A)
 



 rowc ( B) 
can be thought as the multiplication of 1 c and c  1 vectors.
Note:
I.
AB
BA . For instance,
3
2  1
and
B

0 2 
 1


is not necessarily equal to
1
A
2

II.
AC  BC
 2 5  0 7 
AB  

 BA .


 4  4  4  2
 A
might be not equal to
1 3
2
A , B
0 1
2
 2
 AC  
1
III.

IV.
4
 1  2
and
C

 1 2 
3


4
 BC but A  B

2
AB  0 , it is not necessary that A  0
1
A
1
0
AB  
0
B . For instance,
or
B  0 . For instance,
1
 1  1
and
B

 1 1 
1


0
 BA but A  0, B  0.

0
A p  A  A A , A p  Aq  A pq , ( A p ) q  A pq
7
p factors
Also, ( AB) p is not necessarily equal to A p B p .
 AB
t
V.
 B t At .
(iv) Inverse matrix:
Definition of inverse matrix:
An n  n matrix A is called nonsingular or invertible if there exists an n  n
matrix B such that
AB  BA  I n ,
where
In
is a n  n identity matrix. The matrix B is called an inverse of A.
If there exists no such matrix B, then A is called singular or noninvertible.
Theorem:
If A is an invertible matrix, then its inverse is unique.
[proof:]
Suppose B and C are inverses of A. Then,
BA  AC  I n  B  BI n  B( AC)  ( BA)C  I nC  C .
Note: Since the inverse of a nonsingular matrix A is unique, we denoted
the inverse of A as
A 1 .
Note: There are two methods to find the inverse matrix of A:
(a) Using Gauss-Jordan reduction:
The procedure for computing the inverse of a n  n matrix A:
1. Form the n  2n augmented matrix
 a11 a12
a
a
A  I n    21 22
 


a n1 a n 2
 a1n  1 0  0
 a 2 n  0 1  0
      

 a nn  0 0  1
8
and transform the augmented matrix to the matrix
C
D

in reduced row echelon form via elementary row operations.
2. If
(a) C  I n , then A1  D .
(b) C  I n , then A is singular and A 1 does not exist.
Example 5:
1
 1
To find the inverse of A   2


 1
 2
, we can employ the procedure
 5

5 

3
3
introduced above.
1.
1
2

 1
(3)(3)(1)
( 2)( 2)2*(1)

( 2 )1*( 2 )

(1)(1)( 2)
(3)(3)2*( 2)

(1)(1)(3)
( 2)( 2)(3)

1
2  1
0
3
5 
0
1

0
0
3
5
1
0

0
1
2 
1
1
2
3
1
0

0
1  2  1
1
0

0
1
0

0
1
0

2
1

1
0
1
1

2
3
 1
0
1 
3
1

2
0
1

3
0
0
1
0
 1 0
0 1
0
2
1
0
0 .
1
 1 0
 1 0
2 1
0
0 
0
1
1
0 
5
3
0
1 
3
2
1
 1
1 
9
2.
The inverse of A is
 0
 5


 3
(b) Using the adjoint
1
3
2
adj ( A)
1 
 1
.
1 

of a matrix:
Definition of cofactor:
Let
  be n  n matrix. The cofactor of
A  aij
Aij
where
M ij
a ij
is defined as
i j


  1 det( M ij )
,
is the (n  1)  (n  1) submatrix of A by deleting the i’th row of
j’th column o.f A
Example 6:
Let
2
A
 1

1
0
4
3
3 
 2

5 

Then,
 4  2
  1  2
 1 4 
M 11  
,
M

,
M

 12  1 5  13  1  3 ,
 3 5 




 0 3
2 3
2 0 
M 21  
,
M

,
M

22
23

1 5
1  3 ,
 3 5




0 3 
2 3
 2 0
M 31  
,
M

,
M

 32  1  2 33  1 4
 4  2




Thus,
10
A11   1
det( M 11 )  1  4  5  (2)  (3)  14,
A12   1
det( M 12 )   1
11
(1)  5  (2)  1  3
1 3
1 3
A13   1 det( M 13 )   1 (1)  (3)  4  1  1
2 1
2 1
A21   1 det( M 21 )   1 0  5  (3)  3  9
2 2
2 2
A22   1 det( M 22 )   1 2  5  3  1  7
23
23
A23   1 det( M 23 )   1 2  (3)  0  1  6
31
31
A31   1 det( M 31 )   1 0  (2)  3  4  12
3 2
3 2
A32   1 det( M 32 )   1 2  (2)  3  (1)  1
3 3
3 3
A33   1 det( M 33 )   1 2  4  0  (1)  8
1 2
1 2
Important result:
Let
  be an n  n matrix. Then,
A  aij
det( A)  ai1 Ai1  ai 2 Ai 2    ain Ain , i  1,2,, n
 a1 j A1 j  a 2 j A2 j    a nj Anj , j  1,2,, n
Example 6(continue):
A11  14, A12  3, A13  1, A21  9, A22  7, A23  6, A31  12, A32  1, A33  8
Thus,
det( A)  a11 A11  a12 A12  a13 A13  2  14  0  3  3  1  25
 a 21 A21  a 22 A22  a 23 A23  (1)  (9)  4  7  (2)  6  25
 a31 A31  a32 A32  a33 A33  1  (12)  (3)  1  5  8  25
Also,
det( A)  a11 A11  a21 A21  a31 A31  2  14  (1)  (9)  1  (12)  25
 a12 A12  a22 A22  a32 A32  0  3  4  7  (3)  1  25
 a13 A13  a23 A23  a33 A33  3  (1)  (2)  6  5  8  25
Definition of adjoint:
The n  n matrix
adj (A) , called the adjoint of A, is
11
 A11
A
adj ( A)   12
 

 A1n
An1   A11
 An 2   A21

    
 
 Ann   An1
A21 
A12
A22
A22

A2 n

An 2
A1n 
 A2 n 
  

 Ann 

T
Example 6 (continue):
 A11
adj ( A)   A12
 A13
A21
A22
A23
A31  14  9  12
A32    3
7
1 
A33   1 6
8 
and
14  9  12
adj ( A)
1 

A1 

3
7
1
.
det( A) 25 
 1 6
8 
Important Result:
As
det( A)  0 , then
A 1 
adj ( A)
.
det( A)
Important Result:
Let A be an
(a)
nn
matrix.
rank  A  n
 A is nonsingula r
 det  A  0
 Ax  b has a unique solution.
 The column vec tors of A are linearly indpendent .
 The row vectors of A are linearly indpendent .
.
12
(b) rank
A  n
 Ax  0 has a nontrivial solution.
(v) Symmetric Matrices:
Definition of symmetric matrix:
A r  r matrix Arr is defined as symmetric if
 a11
a
A   12
 

 a1r
a12
a22

a2 r
A  At . That is,
a1r 
 a2 r 
, aij  a ji
.
  

 arr 

Example 7:
1 2 5 
A  2 3 6  is symmetric since
5 6 4
A  At .
(vi) Orthogonal Matrices:
Definition of orthogonality:
Two n  1 vectors u and v are said to be orthogonal if
u t v  vtu  0
A set of n  1 vectors
x1 , x2 ,, xn 
is said to be orthonormal if
xit xi  1, xit x j  0, i  j, i, j  1,2,, n.
Definition of orthogonal matrix:
A n n square matrix P is said to be orthogonal if
PP t  P t P  I nn .
Note:
13
 row1 ( P)row1t ( P) row1 ( P)row2t ( P)

row2 ( P)row1t ( P) row2 ( P)row2t ( P)
t

PP 




t
t
rown ( P)row1 ( P) rown ( P)row2 ( P)
1 0  0
0 1  0 


   


0 0  1 
col1t ( P)col1 ( P) col1t ( P)col 2 ( P)
 t
col 2 ( P)col1 ( P) col 2t ( P)col 2 ( P)





 t
t
col n ( P)col1 ( P) col n ( P)col 2 ( P)
 Pt P
 row1 ( P)rownt ( P) 

 row2 ( P)rownt ( P) 




 rown ( P)rownt ( P)
 col1t ( P)col n ( P) 

 col 2t ( P)col n ( P)




 col nt ( P)col n ( P)
rowi ( P)rowit ( P)  1, rowi ( P)row tj ( P)  0

colit ( P)coli ( P)  1, colit ( P)col j ( P)  0
Thus,
row ( P), row ( P),, row ( P) and col1 (P), col2 (P),, coln (P)
t
1
t
2
t
n
are both orthonormal sets!!
(vii) Eigen-analysis:
Definition:
Let A be an n  n matrix. The real number  is called an eigenvalue
of A if there exists a nonzero vector x in R n such that
Ax  x .
The nonzero vector x is called an eigenvector of A associated with the
eigenvalue  .
Procedure of finding the eigenvalues and eigenvectors of A:
1. Solve for the real roots of the characteristic equation
f ( )  0 .
14
These real roots 1 , 2 , are the eigenvalues of A.
2.
Solve
for
i I  Ax  0 ,
the
homogeneous
system
 A  i I x  0
or
i  1,2,  . The nontrivial (nonzero) solutions are the
eigenvectors associated with the eigenvalues
i .
Example 8:
Find the eigenvalues and eigenvectors of the matrix
5
A   4
 2
4
5
2
2
2 
2 
.
[solution:]
 5
f ( )  det( I  A)   4
4
 5
2
2
2
 2    1   10  0
2
 2
   1, 1, and 10.
1. As
 1,
 4
1  I  Ax   4
 2
4
4
2
 2  x1 
 2  x 2   0 .
 1  x3 
 x1   s  t   1  1
 x1  s  t , x2  s, x3  2t  x   x2    s   s  1   t  0 , s, t  R.
 x3   2t   0   2 
Thus,
 1  1
s  1   t  0 , s, t  R, s  0 or t  0 ,
 0   2 
are the eigenvectors associated with eigenvalue
 1.
15
2. As
  10 ,
 5
10  I  Ax   4
 2
4
5
2
 2  x1 
 2  x2   0 .
8   x3 
 x1  2r  2
 x1  2r , x2  2r , x3  r  x   x2   2r   r 2, r  R.
 x3   r  1
Thus,
 2
r 2, r  R, r  0 ,
1 
are the eigenvectors associated with eigenvalue
  10 .
Very Important Result:
If A is an n  n symmetric matrix, then there exists an orthogonal
matrix P such that
D  P 1 AP  Pt AP ,
where
col1 ( P), col 2 ( P), , col n ( P)
are n orthonormal eigenvectors of A and the diagonal elements of D
are the eigenvalues of A associated with these eigenvectors.
(viii) Positive Definite Matrices:
Definition of positive definite matrix:
A symmetric n  n matrix A satisfying
x1tn Ann xn1  0 for all x  0 ,
is referred to as a positive definite (p.d.) matrix.
Intuition:
If ax 2  0 for all real numbers x, x  0 , then the real number a is
positive. Similarly, as x is a n 1 vector, A is a n  n matrix and
x t Ax  0 , then the matrix A is “positive”.
16
Note:
A symmetric n  n matrix A satisfying
x1tn Ann xn1  0 for all x  0 ,
is referred to as a positive semidefinite (p.d.) matrix.
Example 9:
Let
1
 x1 
1
x 
2 
 .

and
l

x 

  


 
 xn 
1
Thus,
n
 x
i 1
i
n
 x    xi2  nx 2
2
i 1
 x1
x2
 x1 
x 
 xn  2   nx1

 
 xn 
x2
 x1 
1 / n
x 
1 / n


1 / n 1 / n  1 / n 2 
 xn 

  
 


1 / n
 xn 
 ll t
1
 1
 x t Ix  x t  n ll t  x  x t Ix  x t 
n
 n
 n

 x


ll t 
x
 x t  I 
n 

Let
ll t
A I 
n
. Then, A is positive semidefinite since for x  0,
x Ax 
t
n
 x
i 1
i
 x  0.
2
Spectral Decomposition of A Symmetric Matrix:
Let A be an n  n symmetric matrix. Then,
A  1w1w1t  2 w2 w2t    n wn wnt ,
where
1 , , n are eigenvalues of A and w1 ,, wn are
associated orthonormal eigenvectors.
17
Example 8 (continue):
(i) Find the orthogonal matrix P and diagonal matrix D such that P t AP  D .
(ii) Find the spectral decomposition of A.
[solution:]
(i) In this example, the eigenvalues are   1, 1, and 10.
  1
 1


  1  v1   1  and v 2   0  are two eigenvectors of A. However, the two
 2 
 0 
eigenvectors are not orthogonal. We can obtain two orthonormal eigenvectors via
Gram-Schmidt process. The orthogonal eigenvectors are
 1
  1 / 2
v2  v1  


*
v  v1   1 , v2  v2    v1   1 / 2
.
v1  v1


0

 2 

*
1
Standardizing these two eigenvectors results in
 1 / 2 
 1 / 18 




v1*
v2*
w1 

1
/
2
,
w



1
/
18
2



.
*
v1*
v
2


 4 / 18 
0




 2
  10  v3  2 is an eigenvectors of A. Standardizing the eigenvector results in
1
 2 / 3
v3
w3 
  2 / 3 .
v3
1 / 3 
Thus,
P  w1
w2
 1 / 2


w3    1 / 2

 0
(ii) The spectral decomposition is
1
1
3 2
3 2
2 2
3
2 / 3
1


2 / 3 , D  0

0
1/ 3 

0
1 0  .
0 10
0
18
A  1 w1 w1t   2 w2 w2t  3 w3 w3t
 1
2

 1
 1  1
2 

 2
0


 1

1   1
 4

18 
 1
18  
 18
18 


0 

1
2
1
18
 2 3
2
1
 2
10  
 2 3 
3
3
3



1
3


1
 1
 1
0
2
2
 18


1
 1  1
0  1 1
2
 18
 02

2
0
0





9

4
2 
4
9
9
 9
4
2 
 10  4
9
9
2 9
2
1

9
9
 9



18 
4
1
18
1
18
2
9
2

9
2 
9
8
9 
