Linear Algebra
Textbook:
Reference:
Website:
Homework submission:
J. Hefferon, “Linear Algebra”
T.M.Apostol, “Linear Algebra”
http://ckw.phys.ncku.edu.tw
class@ckw.phys.ncku.edu.tw
Preface
Evolution of Linear Algebra:
• Linear simultaneous algebraic equations.
• Matrix formulation → matrix algebra.
• Vector space.
• Linear system of differential equations
→ Inner product space (Hilbert space).
Applications of Linear Algebra:
• Minimax (optimization) problems.
• Linear operators (eigen-problems).
• Linear approximations to non-linear problems.
Computerization
• Procedural programming ( FORTRAN, C, … )
– Routine libraries: EISPACK, LINPACK, IMSL, …
• Symbolic programming
– Mathematica, Maple, Reduce, …
• Dedicated packages:
– MATLAB, …
Being Computer Aided
• Leave drudgery to the C.
• Post-C learning process:
– Understand the subject:
• Reasoning.
• Applicability.
• Limitations / Error check.
• Comparison / relation with other subjects
– Apply by hand concept to simple cases.
– C.
– Make knowledge C. searchable.
Goals of this Course
Concepts:
• Linear Systems of Equations
• Linear (Vector) Spaces
• Linear Transformations / Operators
• Matrix Algebra & Determinants
• Inner Product (Hilbert) Spaces
Techniques:
• Mathematical Proofs
• Computational Skills
Hefferon
1.
2.
3.
4.
5.
Linear Systems
Vector Spaces
Maps Between Spaces
Determinants
Similarity
Chapter One: Linear Systems
I. Solving Linear Systems
II. Linear Geometry of n-Space
III. Reduced Echelon Form
Topic:
Topic:
Topic:
Topic:
Computer Algebra Systems
Input-Output Analysis
Accuracy of Computations
Analyzing Networks
I. Solving Linear Systems
I.1. Gauss’ Method
I.2. Describing the Solution Set
I.3. General = Particular + Homogeneous
Example of a Linear System: Balances
Find the unknown masses h & c balanced as follows:
→
40h  15c  50  2
25c  25  2  50h
Example of a Linear System: Chemical Reaction
Toluene + Nitric acid → Trinitrotoluene (TNT)
甲苯 + 硝酸
+ Water
→ 三硝基甲苯 (黄色炸藥) + 水
x C7 H8  y HNO3  z C7 H 5O6 N 3  w H 2O
C : 7x  7z
N : y  3z
H : 8 x  y  5z  2 w
O : 3 y  6z  w
C6 H 2  NO2 3 CH3
Definition: An ordered set of n real numbers is called an n-tuple.
 a1 , a2 ,
, an 
ai  R
Definition: The set of all n-tuples is called Rn.
Rn 
 a
1
, a2 ,
, an  ai  R

Definition 1.1a: A linear equation in variables x1, x2, …, xn has the form
a1 x1  a2 x2 
 an xn  d
(1)
where a1, a2, …, an R are the equation’s coefficients, and dR its constant.
Definition 1.1b: An n-tuple ( s1, s2, …, sn ) R is a solution of, or satisfies, eq(1) if
a1s1  a2 s2 
 an sn  d
Definition 1.1c: ( s1, s2, …, sn ) is a solution of the system of m linear equations
a11 x1  a12 x2 
 a1n xn  d1
a21 x1  a22 x2 
 a 2 n xn  d 2
(2)
am1 x1  am 2 x2 
 amn xn  d m
if it is a solution of all m equations in eq(2).
Example 1.2: (1, 5 ) is a solution of
(5, 1) is not a solution because
3x1  2 x2  7
 x1  x2  6
since
3  5  2  1  13  7
  5   1  6  6
3  1  2  5  7
  1  5  6
I.1. Gauss’ Method
Gauss’method  Systematic elimination of variables.
Example 1.3:
x1 5 x2
1
x1 2 x2
3
3x3
2 x3
9
2


3
Forward elimination ( upper triangularization ) :
row 1  row 3 :
1
x1
3
x1
 2 x2
5 x2
 row 1 + row 2:
3 row 1 :
3
2 x3
2
3x3
9
x1
x1
 6 x2
5 x2
2 x3
3x3
9
2
9
x1
 6 x2
 x2
2 x3
3x3
9
 7
9
Backward substitution (diagonalization) with ρi = row i :
1/3ρ3 ;
x1
 6 x2
x2
ρ2 :
2 x3
x3
2ρ3 + ρ2 :
9
7
3
x1
 6 x2
x2
x3
6ρ2 +ρ1 :
9
1
3
x1
x2
x3
3
1
3
Theorem 1.4 (Gauss’ Method) :
A linear system is invariant under the following operations :
1. ρi  ρj , where i  j.
2. ρi → c ρi , where c  0.
3. ρi → ρi + c ρj , where i  j and c  0.
Definition 1.5 (Elementary reduction / row / Gaussian operations):
1. Swapping.
2. Multiplying by a scalar.
3. Rescaling / pivoting.
Example 1.6:
x y
0
2 x  y 3z  3
x 2 y  z  3
21   2


 1   3
x
y
0
3 y 3z  3
3 y  z  3
1
x y
 3
4


3 y
3 3   2
0
3
z 0
 2  3


x
1
x
 3
3


  2  1
y
3 y
0
3z  3
4 z  0
1
 1
y
z
0
Definition 1.9:
Leading variable = 1st variable with non-zero coefficient in a row.
A system is in echelon form if each leading variable is to the right of the one above it.
A linear system needs not have the same number of equations as unknowns.





Example 1.6:
x 3 y  1
2 x  y  3
2 x 2 y  3
x 3 y  1
2 1   2


5 y  5
2 1   3
4 y  4
1
x 3 y  1
 2
5


y
1
42  3
0 0




A linear system can have exactly one, none, or infinitely many solutions.
Example 1.12 ( System with no solution ) :


x 3 y  1
2 x  y  3
2 x 2 y  0
x 3 y  1
2 1   2


5 y  5
2 1   3
4 y  2


1
x 3 y  1
 2
5


y
1
42  3
0 2
Inconsistent.
Example 1.14 ( System with infinitely many solutions ) :
x y  4
2 x 2 y  8
x y  4
2 1   2


0 0






Unique solution:
(Triangular)





No solution:
(Echelon)




Many solution:
(Echelon)




Exercise 1:
Four positive integers are given. Select any three of the integers, find their
arithmetic average, and add this result to the fourth integer. Thus the numbers
29, 23, 21, and 17 are obtained. One of the original integers is:
(a) 19 (b) 21 (c) 23 (d) 29 (e) 17
Exercise 2:
Laugh at this: AHAHA + TEHE = TEHAW.
It resulted from substituting a code letter for each digit of a simple example in
addition, and it is required to identify the letters and prove the solution unique.
I.2. Describing the Solution Set
Example 2.1: System with many solutions.
2x
z  3
x  y z  1
3x  y
4
2x
z
3
1
 1   2
3
1
2


y  z  
3
2
2
 1   3
2
3
1
y  z  
2
2
x
1
1 ; 2
2


2  3
Solution set =
1
3
 z 
2
2
3
1
y  z 
2
2
0
0

1 3
3 1

  x, y , z     z ,  z , z 
2 2
2 2



z R 

z = free variable
z = parameter
Definition 2.2: The non-leading variables in an echelon form are free variables.
Variables used to describe a solution set are parameters of the solution set.
Example 2.3: System with 2 free variables.
 z w
1
 z  w  1
3x
6 z 6w  6
 y  z w  1
x
y
y
3 2   3


2  4
x
31   3


x  y  z w  1
y  z  w  1
0
0
0
0
x
  3  1


Solution set =
 y  z w  1
y
 z  w  1
3 y 3z 3w  3
 y  z w  1
2 z 2 w  2
y  z  w  1
0
0
0
0
  x, y , z, w    2  2 z  2 w ,
 1  z  w , z , w
z, w = free variables
z, w  R

z, w = parameters
Matrix
Definition 2.6 : Matrix
An mn matrix is a rectangular array of numbers with m rows and n columns.
Each number in the matrix is an entry.
Example: A 23 matrix
 1 2.2 5 
 a11 a12
A

a
 i j   a a

3
4

7


 21 22
a13 
a23 
i = row index
j = column index
Example 2.7:
x1 2 x2
x2
x1
 x3
2 x3
Linear system
4
0
4
1 2 0 4
 13

  0 1 1 0 
 0 2 2 0 


can be abbreviated as
1 2 0 4
 0 1 1 0 


1 0 2 4


1 2 0 4
2 2 3
 0 1 1 0 



0 0 0 0


S    x1 , x2 , x3    4  2 z , z , z  z  R 








1 0 2 4
2 2 1

  0 1 1 0 
0 0 0 0


4
 2 
 0  z  1  z  R
 
 
 0
 1
 
 








Definition 2.8: Vector
An m-dimensional (column) vector is an m1 matrix.
An n-dimensional row vector is an 1n matrix.
The entries of a vector are its components.
 v1   1 
v   2 
v v  2  
 v3   7 
   
 v4   1
Definition 2.9: Solution
A linear equation
a1x1+ a2x2+…+ an xn = d
of n unknowns x1 , x2 , …, xn is satisfied by a vector s

a1s1+ a2s2+…+ an sn = d
A vector satisfies a linear system if it satisfies each equation in the system.
Definition 2.10: Vector Addition (Sum)
 u1  v1 
 u1   v1 

u  v        

u  v 
u  v 
 n n
 n  n
 u  v i  ui  vi
Definition 2.10a: Matrix Addition (Sum)
 a11
A  B  
a
 m1
a1n   b11

 
amn   bm1
 A  B i j  ai j  bi j
b1n   a11  b11
 
 
bmn   am1  bm1
i  1,
,m
a1n  b1n 


amn  bmn 
j  1,
,n
Definition 2.11: Scalar Multiplication
 a11
cA  c 
a
 m1
a1n 
 ca11
 


 ca

amn 
 m1
 v1   cv1 

c v  c    

 v   cv 
 m  m
Example 2.12:
ca1n 


camn 
 cv i  cvi
 2  3   5
 3    1    2 
     
 1  4   5
     
 cA i j  cai j
i  1,
,m
1  7 
 4   28 

7 
 1   7 
  

 3   21
i  1,
,m
j  1,
,n
Example 2.13:
2x  y
w
4
y
 w u  4
x
 z 2w
0
 2 1 0 1 0 4 
0 1 0 1 1 4


 1 0 1 2 0 0 





1
1
0 1 0 4 
2
 1 3
2

  0
1
0
1 1 4 
 0 1/ 2 1 5/ 2 0 2 


1
 2 1 0 1 0 4 
 2  3
2

  0 1 0 1
1 4 
 0 0 1 3 1/ 2 0 


1
 1 0 0 1 1/ 2 0 
1

2  0 1 0 1

1
4


 0 0 1 3 1/ 2 0 





 
1
1 
S    w  u ,  w  u  4 , 3w  u  w, u  R   
2
2 

 


 2 0 0 2 1 0 
3

  0 1 0 1
1
4 
 2  1
 0 0 1 3 1/ 2 0 





x
1
 1/ 2   0 
 y
 1 
 1   4 
 
 

  
 z   w  3   u  1/ 2    0 
 
 

  
w
1
0
 
 

  0
u
0
 1   0
 
 

  




w, u  R 



Particular solution
Possible scenarios:













































Exercise:
Solve the following using matrix notations &
express the solution sets in vector notations.
x
 z w  4
2x  y
w  2
3x  y  z
7
 y z
0
y
w  0
3x 2 y 3z  w  0
y
w  0
x
Exercise:
1. Describe all functions f(x) = a x2 + b x + c such that f(1) = 2 & f(1) = 6.
2. Describe all functions f(x) = a x2 + b x + c such that f(1) = 2
Exercise:
Consider the linear system { a x + y = a2 , x + a y = 1 }.
Find values of a so that it has
1) No solutions.
2) Infinitely many solutions.
I.3. General = Particular + Homogeneous
Definition 3.2: Homogeneous Equations
A linear equation is homogeneous if it can be put in the form
a1x1 + a2x2 + … + anxn = 0
A linear system of equations is homogeneous if it can be put in the form
a11 x1  a12 x2 
 a1n xn  0
a21 x1  a22 x2 
 a 2 n xn  0
am1 x1  am 2 x2 
 amn xn  0
Every linear system has an associated homogeneous system obtained by
setting all constant terms to zero.
Definition 3.4: Zero vector = 0 with 0i = 0  i.
A homogeneous system is always consistent since 0 is always a solution.
If s is a solution to a homogeneous system, so is c s  cR .
From Example 2.13:
2x  y
w
4
y
 w u  4
x
 z 2w
0
1
 1 
 
3
 
1
0
 
 1/ 2 
 1 


&  1/ 2 
 0 
 1 







S





 x
1
 1/ 2   0 
 y
 1
 1   4 
 
 

  
 z   w  3   u  1/ 2    0 
 
 

  
w
1
0
 
 

  0
u
0
 1   0
 
 

  
Homogeneous Particular
solution
solution
satisfy the assoicated 2 1   1
 1
homogeneous system:
1
 1
  3
 0
4
 
 0
 
 0
 0
 
1/ 2 
2  0   4 
satisfies the
inhomogeneous system:
 0
 4
  0
 1/ 2 
  0
  0
0
 1   0  0
2 1
0
2 1/ 2    1
 1





w, u  R 





0
  0  1  0
2  0 
0
4
  0   0  4
2  0 
0
Example 3.5: Homogeneous system with unique solution 0.
3x 2 y  z  0
6 x 4 y
0
y
z  0
 3 2 1 0
 6 4 0 0


 0 1 1 0


 3 2 1 0
2 12

  0 0 2 0 
 0 1 1 0


 3 2 0 0
 3 2
 0 1 0 0



 3 1
 0 0 1 0


 3 2 1 0
3   2 

  0 1 1 0 
1
 0 0 1 0
 3


2
1 0 0 0
2 2  1

  0 1 0 0 
1
 0 0 1 0
1


3
Lemma 3.7:
For any homogeneous linear system, there exists vectors β1 , …, βk s.t.

S  c1 β 1 
 ck βk
c1 ,
, ck  R

where k is the number of free variables in an echelon form of the system.
Proof :
Let the system be of m equations in n unknowns with m  n.
By means of the Gauss method, it can be reduced to an echelon form.
Next, we make sure the leading variable of every row is on the diagonal position
by moving any offending columns to the rightmost position.
Let there be p equations of the form 0 = 0 at the bottom:
 a11 a12

 0 a22

0




0



a13
a1n
a23
a2 n
0
am  p , m  p
0
am  p , m  p 1
0
am  p , n
0
0
0

0


0
0


0 
Moving columns
means re-ordering
the unknowns.
By meaning of backward substitution, one gets
1
0










0
1
0
0
b1, m  p 1
b2, m  p 1
b1n
b2 n
0
0
1 bm  p , m  p 1
0
0
bm  p , n
0
0
0
0
0 



0

0


0
With suitable re-labeling of the unknowns, we get
xi  bi , m p 1 xm p 1 
 ai , n xn
i  1,
where xj for j = mp+1, …, n are free variables.
By definition:
nm p  k
,m  p
Proof is completed by setting βj to be
the jth column of the nk matrix
 β1
 b1, m p 1

 b2, m  p 1


βk   
 bm  p , m  p 1

1



0

b1n 

b2 n 



bm  p , n 

0 


1 
z
The non-b part is a kk unit matrix.
Particular solution:
Lemma 3.8:
S   p  h p = particular solution, h = homogeneous solution 
Proof:
1. Every solution s can be be written as s = p + h :
Let s be a solution, then sp satisfies the associated homogeneous system since
a i1  s1  p1  
 ai n  sn  pn    ai1s1 
 ai n sn    ai1 p1 
 ai n pn   di  di
0
Setting h = s  p completes the proof.
2. Every s = p + h is a solution :
a i1  p1  h1  
Theorem 3.1:

 ai n  pn  hn    ai1 p1 
 ai n pn    ai1h1 
 ai n hn   di  0
For any linear system, there exists vectors β1 , …, βk s.t.
S  p  c1 β 1 
 ck βk
c1 ,
, ck  R

where p is a particular solution and k is the number of free variables.
 di
Example 3.10: System with empty solution set
x
 z w  1
2x  y
w  3
x  y 3z 2w  1
1 0 1 1 0


2

1
0
1
0


1 1 3 2 0


1 0 1 1 0
2 1  2



  0 1 2 1 0 
 1  3
0 1 2 1 0


Homogeneous solution set =
 1 0 1 1 1


2

1
0
1
3


1 1 3 2 1 









 1 
 1 
 
 

2
1 



z
w
1
0
 
 
0
 
1
1 0 1 1 0
 2



 0 1 2 1 0
 2  3
0 0 0 0 0





z, w  R 



 1 0 1 1 1
 2
2 1  2




  0 1 2 1 5  
 2  3
 1  3
0 1 2 1 2 


→ no particular solution
→ no general solution
 1 0 1 1 1 


0
1
2
1

5


0 0 0 0 7 


Corollary 3.11:
Solution sets of linear systems are either
• empty (system is inconsistent / example 3.10)
• have one element (solution is unique / example 3.5)
• have infinitely many elements (system is indeterminate / example 2.13)
Dim( h) = 1
Dim( h) > 1
p exists
Unique
( 1 general solution )
indeterminate
(  general solutions )
p does not exist
inconsistent
( 0 general solutions )
inconsistent
( 0 general solutions )
Definition 3.12 : Singular and Nonsingular Matrices
A square matrix is nonsingular if it is the matrix of coefficients of a homogeneous
system with a unique solution.
It is singular otherwise, that is, if it is the matrix of coefficients of a homogeneous
system with infinitely many solutions.
Definition: Rank of a Matrix
The rank of a matrix is equal to the number of non-zero rows in its echelon form.
Let A be an mn matrix, then Rank A  min( m, n ).
Theorem
A square matrix is nonsingular iff its rank is equal to its dimension.
Exercises