Lecture 10
Dimensions, Independence, Basis and
Complete Solution of Linear Systems
Shang-Hua Teng
Linear Independence
Linear Combination
of a set of vectors v1,v2 ,...,vn  is

n
i 1
 i vi

Linear Independence
A set of vectors v1,v2 ,...,vn 
is linearly independent if only if none of them can
be expressed as a linear combination of the others
Examples
1 1 
  ,  
0 1 
2 1 
  ,  
2 1 
 1   2   0  
      
  0  ,  2  , 1  
 1   4  1  
      
Linear Independence and Null Space
Theorem/Definition
A set of vectors v1,v2 ,...,vn 
is linearly independent if and only
1v1+2v2+…+nvn=0 only happens when all  ’s
are zero
The columns of a matrix A are linearly
independent when only solution to Ax=0
is x = 0
2D and 3D
v
w
u
v
How do we determine a set of
vectors are independent?
Make them the columns of a matrix
Elimination
Computing their null space
Permute Rows and Continuing
Elimination (permute columns)
1
1
A
1

1
 1

0
 1

2 1 1 1 0 
1 0 0
1 1 1
1 1 1
1
2
3
Theorem
If Ax = 0 has more have more unknown than
equations (m > n: more columns than rows),
then it has nonzero solutions.
There must be free variables.
Echelon Matrices
*
0
A
0

0
* * * * * *

* * * * * *
0 0 0 * * *

0 0 0 0 0 *
Free variables
Reduced Row Echelon Matrix R
1
0
A
0

0
0 * * 0 * 0

1 * * 0 * 0
0 0 0 1 * 0

0 0 0 0 0 1
Free variables
Computing the Reduced Row
Echelon Matrix
• Elimination to Echelon Matrix
E1PA = U
• Divide the row of pivots by the pivots
• Upward Elimination
E2E1PA = R
Example: Gauss-Jordan Method
for Matrix Inverse
• [A I]
• E1[A I] = [U, I]
• In its reduced Echelon Matrix
• A-1 [A I] = [I A-1]
A Close Look at Reduced
Echelon Matrix
1 3 0 2  1


R  0 0 1 4  3
0 0 0 0 0 
• The last equation of R x = 0 is redundant
0=0
• Rank of A is the number of pivots rank(A).
What is the Rank of Outer Product
 u1 
  v  v 
n
  1
un 
Rank and Reduced Row Echelon Matrix
1
0

A  0

0
0
0
1
0
0
0
*
*
0
0
0
*
*
0
0
0
0
0
1
0
0
*
*
*
0
0
0
0 
0

1
0
Free variables
• Theorem/Definition
• Rank(A) = number of independent rows
• Rank(A) = number of independent columns
Dimension of the Column Space
and Null Space
• The dimensions of the column space of A is
equal to Rank(A).
• The dimension of the null space of A is
equal to the number of free variables which
is n – Rank(A)
• A is an m by n matrix
Rank and Reduced Row Echelon Matrix
1
0

A  0

The Pivot
0
columns are not 
0
combinations
0
1
0
0
0
*
*
0
0
0
*
*
0
0
0
0
0
1
0
0
*
*
*
0
0
of earlier
columns
Pivot columns
Free variables
Free Columns
0

0
0

1
0
Reduced Echelon and Null Space Matrix
 x1 
x 
1
3
0
2

1

  2  0 






Rx  0 0 1 4  3 x3  0
 
0 0 0 0 0   x4  0
 x5 
• Nullspace Matrix
• Special Solutions
 3  2 1
1

0
0


N   0  4 3


0
1
0


 0
0 1
 3  2 1
1
 1 0 0
0
0



 0  4 3  0 1 0 



1 0 0 0 1
0
 0
0 1
Null Space Matrix
• Ax=0 has n-Rank(A) free variables and special
solutions
• The Nullspace matrix has n-Rank(A) columns
• The columns of the nullspace matrix are
independent
• The dimension of the Null space is n – rank(A)
Complete Solution of Ax = 0
• After column permutation, we can write
I F 
R

0
0


r pivot columns
r pivot rows
m-r zeros rows
n-r free columns
• Nullspace matrix
 F 
N 

 I 
• Moreover: RN = [0]
Pivot variables
r
m-r Free variables
Complete Solution to Ax = b
• A is an m by n matrix, and b is an n-place vector
– Unique solution
– Infinitely many solution
– No solution
Suppose Ax = b has more then one solution, say x1, x2 then
A x1 = b
A x2 = b
So
A (x1 - x2 ) = 0
(x1 - x2 ) is in nullspace(A)
Complete Solution to Ax = b
Suppose we found a particular solution xp to Ax = b i.e,
A xp = b
Let F be the indexes of free variables of Ax = 0
Let xF be the column vector of free variables
Let N be the nullspace matrix of A
Then
xp  NxF
defines the complete set of solutions to Ax = b
Example: Complete Solution to
Ax = b
 x1 
1 3 0 2   1 
 0 0 1 4   x2    6 

x   
3
1 3 1 6   7 
 x4 
Augmented matrix [A b]
1 3 0 2 1
0 0 1 4 6

 
1 3 1 6 7
Elimination to obtain [R d]
1 3 0 2 1
0 0 1 4 6

 
0 0 0 0 0
Set free variables to 0 to find a particular solution
(1,0,6,0)T
Compute the nullspace matrix
 3  2
 1

0


 0  4


1
 0
Complete solution is
  2
1  3  2
1
 3
 
0   1
 x
0 
 1
0
 2  
0







x


 x2
 x4


  4
 6   0  4   x4   6 
 0
 
  

 
 
1
 1
0   0
0 
 0
Full Rank Matrix
• Suppose A is an m by n matrix. Then
rank ( A)  min( m, n)
• A is full column if rank(A) = n
– columns of A are independent
• A is full row rank if rank(A) = m
– Rows of A are independent
Full Column Rank Matrix
•
•
•
•
Columns are independent
All columns of A are pivot columns
There are non free variables or special solutions
The nullspace N(A) contains only the zero
vector
• If Ax=b has a solution (it might not) then it has
only one solution
I 
R 
0 
n by n
m-n rows of zeros
Full Row Rank Matrix
•
•
•
•
•
Rows are independent
All rows of A have pivots, R has no zero rows
Ax=b has a solution for every right hand side b
The column space is the whole space Rm
There are n-m special solutions in the null space
of A
R  I F 
The Whole Picture
• Rank(A) = m = n
Ax=b has unique solution
R   I
• Rank(A) = m < n
Ax=b has n-m dimensional solution
R  I F 
I 
R 
0 
I F 
• Rank(A) < n, Rank(A) < m
R

0
0
Ax=b has 0 or n-rank(A) dimensions


• Rank(A) = n < m
Ax=b has 0 or 1 solution
Basis and Dimension of a Vector Space
• A basis for a vector space is a sequence of
vectors that
– The vectors are linearly independent
– The vectors span the space: every vector in the
vector can be expressed as a linear combination
of these vectors
Basis for 2D and n-D
• (1,0), (0,1)
• (1 1), (-1 –2)
• The vectors v1,v2,…vn are basis for Rn if and
only if they are columns of an n by n
invertible matrix
Column and Row Subspace
• C(A): the space spanned by columns of A
– Subspace in m dimensions
– The pivot columns of A are a basis for its column space
• Row space: the space spanned by rows of A
–
–
–
–
Subspace in n dimensions
The row space of A is the same as the column space of AT, C(AT)
The pivot rows of A are a basis for its row space
The pivot rows of its Echolon matrix R are a basis for its row
space
Important Property I: Uniqueness
of Combination
• The vectors v1,v2,…vn are basis for a vector
space V, then for every vector v in V, there is a
unique way to write v as a combination of
v1,v2,…vn .
v = a1 v1+ a2 v2+…+ an vn
v = b1 v1+ b2 v2+…+ bn vn
• So: 0=(a1 - b1) v1 + (a2 -b2 )v2+…+ (an -bn )vn
Important Property II: Dimension
and Size of Basis
• If a vector space V has two set of bases
– v1,v2,…vm . V = [v1,v2,…vm ]
– w1,w2,…wn . W= [w1,w2,…wn ].
• then m = n
– Proof: assume n > m, write W = VA
– A is m by n, so Ax = 0 has a non-zero solution
– So VAx = 0 and Wx = 0
• The dimension of a vector space is the number of
vectors in every basis
– Dimension of a vector space is well defined
Dimensions of the Four Subspaces
Fundamental Theorem of Linear
Algebra, Part I
•
•
•
•
•
Row space: C(AT) – dimension = rank(A)
Column space: C(A)– dimension = rank(A)
Nullspace: N(A) – dimension = n-rank(A)
Left Nullspace: N(AT) – dimension = m –rank(A)