Chapter 6
The Column and the Null Space
6.1
The column space
Each matrix (square or rectangular) has four spaces associated with it: its column space, its row space, its null
space, and its left null space. Knowing the dimensions of
these spaces tell us much about which type of solutions (one
unique solution, infinitely many solutions, or none) we might
expect of a system with this matrix as coefficient matrix.
We shall commence by introducing the column space of A.
Consider the following set
C = {b ∈ R3 |b = λ[1 1 0]T + µ[0 1 1]T , λ, µ ∈ R}
C is a vector space because it is a set of all linear combinations of two vectors. However, C can also be expressed as
follows:
#
1 0 "
λ
3
C = {b ∈ R |b = 1 1
, where λ, µ ∈ R}
µ
0 1
Or we may express it as follows:
1 0
C = {b ∈ R3 |b = Ax, with A = 1 1 , and x ∈ R2 }
0 1
C is called the column space of A.
1
2
Definition: Column space
The column space of A is the set of all linear combinations of the columns of A.
Let A be m × n.
pressed as
Then the column space of A is ex-
C(A) = {b ∈ Rm |b = Ax, x ∈ Rn }
Every matrix has a column space.
C(A) is a vector space:
Let us confirm that the column space of any matrix A is a
vector space indeed.
Check Rule 1: Let
b1 = Ax1 ,
and
b2 = Ax2 .
Add the two elements,
v = Ax1 + Ax2 = A(x1 + x2 ) = Ax,
where x = x1 + x2 ∈ Rn . Therefore Rule 1 is satisfied.
Check Rule 2: Let
b1 = Ax1 .
Multiply b1 by a scalar,
w = λ(Ax1 ) = A(λx1 ) = Ax,
where x = λx1 ∈ Rn , therefore Rule 2 is also satisfied, and
a column space is a vector space indeed.
6.2
The null space
The null space of a matrix A, is the set of all solutions of
Ax = 0. Let us first look at three examples.
Examples:
Find the null spaces of A, B, and C.
"
A=
2 0
0 5
#
"
,
B=
2 2
4 4
#
"
,
C=
0 0
0 0
#
.
In order to find the null space of a matrix A, one must solve
Ax = 0.
3
Solving Ax = 0, (using backward substitution) gives, 5x2 =
0, i.e. x2 = 0, and then 2x1 = 0, i.e. x1 = 0. The only
possible solution is,
"
#
0
x=
.
0
This is he smallest null space possible, the set {0}. In general
any invertible square matrix has the null vector as null space.
For matrix B we have,
2x1 + 2x2 = 0
4x1 + 4x2 = 0
It should be quite clear that if x2 = −x1 , then both equations are satisfied. (First decomposing B as LU , and then
solving U x = 0 is the recommended way to find the solution.) Therefore all solutions of the form
"
x=
λ
−λ
#
"
=λ
1
−1
#
solves Bx = 0. The null space of B is a line through the
origin.
For matrix C, it is clear that any x will solve Cx = 0. The
null space of C is R2 .
Formally:
Definition: Null space
The null space of A is the set of all solutions of Ax = 0.
Let A be m × n. Then the null space of A is expressed as
N (A) = {x ∈ Rn |Ax = 0, 0 ∈ Rm }
N (A) is a vector space:
Let us confirm that the null space of any matrix A is indeed
a vector space.
Check Rule 1: Let
Ax1 = 0,
and
Ax2 = 0.
Add the two equations,
Ax1 + Ax2 = A(x1 + x2 ) = 0,
therefore x = x1 + x2 also solves Ax = 0. Rule 1 is satisfied.
4
Check Rule 2: Let
Ax1 = 0.
Multiply the equation by a scalar λ,
λ0 = λ(Ax1 ) = A(λx1 ) = 0,
therefore Rule 2 is also satisfied, and a null space is indeed
a vector space.
6.3
Introduction to Rectangular Systems
In the previous chapter on LU decomposition, the systems
of equations that were considered always had N equations
and N variables. We shall now consider systems with too
few equations, so called under determined systems and also
systems with too many equations, so called over determined
systems. Here are some examples.
An under-determined system:
Consider
x +y +z = 3
y +z = 2.
This is an under-determined system — it has 3 variables but
only 2 equations. One may express it in matrix form as
"
1 1 1
0 1 1
#
#
"
x
3
y =
2
z
or Ax = b, where A is now rectangular. For an underdetermined system, A is a landscape matrix. Such a system
has either (1) infinitely many solutions, or (2) none. The
system above has,
x = 1,
and
y + z = 2.
Any y and z may be chosen, as long as their sum is 2. One
may choose z to be a free variable and express y as
y =2−z
The solution is then,
x
1
y = 2 − z ,
z
z
for any z ∈ R.
This is a case where an under-determined system has infinitely many solutions.
5
Here is an example of an under determined system that has
no solution:
x +y +z = 3
2x +2y +2z = 7.
If x + y + z = 3, then 2(x + y + z) must be 6 — it cannot
be 7. Even though we have too few equations, we have no
solution.
An over-determined system:
Consider
x + y = 2
2x + 2y = 4
y = 1.
This is an over determined system — it has 3 equations but
only 2 variables. One may express it in matrix form as
1 1 " #
2
x
= 4
2 2
y
0 1
1
or Ax = b, where A is rectangular again. For an overdetermined system, A is a portrait matrix. Such a system
has either (1) infinitely many solutions, or (2) one unique
solution, or (3) none. The system above has,
x = 1,
y=1
as only solution.
Here is an example with infinitely many solutions:
x +y = 2
2x +2y = 4
3x +3y = 6.
If y is chosen as free variable, then x = 2 − y.
Here is an example with no solution:
x
=1
y =1
x +y = 3.
We would like to have a method to solve these systems,
which will also tell us if there is no solution. The way to do
it is to LU-decompose the matrix A, and then to use forward
substitution followed by backward substitution (if possible).
Row operations on rectangular matrices:
A
=
A
=
U
L
L
U
LU decomposition of a rectangular matrix is done in the
same way as for a square matrix — the only difference is
Figure 6.1. Appropriate sizes for LU decomposition for a rectangular A.
6
that the pivot elements need not lie on the diagonal. Row
operations are performed on A to create zero elements in the
lower triangular part of the matrix. Since a row operation
is equivalent to pre multiplying A by a suitable elementary
matrix, one may view the process of obtaining more and
more zero elements in the lower triangular part as a series
of pre-multiplications by various elementary matrices. For
a (3 × n) matrix, we may have
E32 E31 E21 A = U.
or
L−1 A = U,
with
L = (E32 E31 E21 )−1 .
L must therefore be a square and invertible matrix. This
means that, if A is (m × n) then L is (m × m) and U is
(m×n). Figure 4.3 illustrates this set-up for both landscape
as well as portrait matrices.
6.4
LU-decomposition of a landscape matrix
An example will be the best way to explain this.
Example:
Do LU-decomposition of the following (3×4)-matrix A using
three elementary matrices and check that the final decomposition is correct by multiplying out again.
1
3
2 −5
A = −2 −6 −2
11
3
9 14 −11
The first pivot
is boxed and the elementary matrix
element
−2
must have − 1 in the (2, 1)-position. Hence
1 0 0
E21 = 2 1 0 .
0 0 1
Then
1 3 2 −5
1 .
E21 A = 0 0 2
3 9 14 −11
7
In order to get a zero in the (3, 1)-position, we take the
multiplier as − 31 , therefore
2 −5
2
1 .
8
4
1 0 0
E31 = 0 1 0
−3 0 1
and
1 3
E31 E21 A = 0 0
0 0
Note that the second row has two zeros – and it is impossible
to create more zeros in that row, without destroying the
zeros we have obtained already.
But we may try to produce more zeros in the third row.
Take the first non-zero element in the second row (i.e. the
’2’ in the (2, 3)-position) as the next pivot element. It is OK
if it does not lie on the diagonal. The multiplier is − 28
and the elementary matrix is
1
0 0
1 0 .
E32 = 0
0 −4 1
Note that the multiplier goes into the (3, 2)-position of E32 ,
regardless of where the zeros is produced. The 32 signifies
that E32 will be doing a row operation on the third row (the
first index) by subtracting some multiple of the second row
(the second index) and will put the result in the third row
(the first index). Where the zero will be produced in the
result depends on which multiplier was used. (In fact, you
can produce a zero in any position of your choice in the third
row by choosing an appropriate multiplier. However, it may
destroy some other zeros that you have created already.)
Therefore
1 3 2 −5
1 =U
E32 E31 E21 A = 0 0 2
0 0 0
0
Confirm that
1 0 0
−1 −1 −1
L = (E32 E31 E21 )−1 = E21
E31 E32 = −2 1 0
3 4 1
Of course, we could have have inserted the negatives of the
multipliers in the correct positions in L without explicitly
creating elementary matrices.[Do Checkpoint 6.4.4]
Now confirm that LU = A.
Checkpoint 6.4.4:
Do the
LU decomposition of
3
2 1
6 −2
4 3 13 −3
A= 6
−3 −2 2 −3
5
8
6.5
Null space of a landscape matrix
As mentioned before, the null space of A is the space of all
solutions of Ax = 0. We shall now attempt to find the null
space of the matrix A in the previous section.
Solving Ax = 0, is equivalent to solving
LU x = 0.
Since L is always invertible, we may pre multiply by L−1 ,
to find
U x = L−1 0 = 0.
The same x that solves U x = 0, also solves Ax = 0. In
other words, A and U have the same null space.
Let us write out the system,
x
1 3 2 −5 1
0
x
1 2 = 0
0 0 2
x3
0 0 0
0
0
x4
and express it as equations
1x1 + 3x2 + 2x3 − 5x4 = 0
2x3 + 1x4 = 0
0 = 0
..... (1)
..... (2)
..... (3)
Equation (3) is actually
0x1 + 0x2 + 0x3 + 0x4 = 0.
We have in effect lost one equation. Although we might have
thought we had three equations, we have in effect only two.
With 2 equations and 4 variables, we should choose 2 free
variables, and solve the other two in terms of these.
The effective number of equations in a system of equations
is called the rank of the matrix. Therefore the rank is the
number of pivot elements.
Although one often has a choice of which variables one wants
to solve, it is good practice to solve those variables associated
with pivots in terms of the others. Since the pivots lie in
columns 1 and 3, x1 and x3 will be solved here in terms of the
rest of the variables. It is also good practice to replace the
free variables with new symbols — we shall employ Greek
letter to denote free variables.
9
Therefore, in the current case, x1 and x3 will be solved in
terms of x2 (relabeled as λ) and x4 (relabeled as µ). The
system to be solved is as follows (pivot elements are boxed),
1 3
0 0
0 0
or
x3 = −
x
0
2 −5 1
λ
= 0
2
1
x3
0
0
0
µ
We use backward substitution.
The second row is
2x3 + µ = 0,
µ
2
The first row is
µ
x1 + 3λ + 2(− ) − 5µ = 0,
2
x1 = −3λ + 6µ
or
A general solution is
6
−3
−3λ + 6µ
0
1
λ
x=
+ µ 1 .
= λ
−2
0
− µ2
0
µ
1
Check that Ax = 0, for all choices of λ and µ. Formally the
null space of A can be expressed as
6
−3
0
1
N (A) = {x ∈ R4 |x = λ
+ µ 1 , λ, µ ∈ R}.
−2
0
0
1
6.6
A Basis for the Column Space
The column space of a matrix is spanned by all the columns
of the matrix. For landscape matrices there are always more
columns than the minimum that are independent. We therefore need a way to select an independent set amongst the
columns in order to find a basis.
Note that it is easy to find a basis for the columns space
of U — pick those columns with a pivot element. This also
shows that the dimension of the column space is equal to the
number of pivots, which is equal to the rank of the matrix.
However, A has a different column space from U (although
the two column spaces have the same dimension).
10
Independence is determined by the null space:
In order to know the independence structure of a set of
columns, one must look at which coefficients in the linear
combination will result in a zero vector.
We shall not prove this, but only illustrate the concept by
means of an example. Let
|
|
|
A = a1 a2 a3
|
|
|
and let
3
A −1 = 0.
0
This means that [3 − 1 0]T is a basis vector of the null
space of A. We then have that
3a1 − a2 + 0a3 = 0
and a1 and a2 are multiples of each other while a3 is independent of either. Then a basis for the column space of A
could be
either
{a1 , a3 },
or
{a2 , a3 }.
It is the coefficients in the combination, [3, −1, 0] in this case,
that determine the interdependence of the vectors. Let us
rephrase it: It is the basis vector of the null space that
determines the interdependence of the column vectors of A.
In general, every null space basis vector groups a number of
columns of A together such that the particular linear combination forms 0. All those columns then form a subset that
is dependent. In choosing a basis for the column space, one
of those vectors in this subset should be omitted.
For each free variable, there will be one basis vector of the
null space, and each such null space basis vector creates
another dependent subset. The way in which we find the null
space introduces the free variables one after the other, so it
makes sense to always omit the vector associated with that
free variable. Therefore, omit all the columns associated
with free variables, and the remaining columns (they are
pivot columns) will form an independent set, spanning the
column space.
This result may also be expressed in the following way: If A
and B share the same null space, then the interdependency
11
of their columns is similar. In other words, if you know
which columns to pick to find a suitable basis for A, you
should just pick the same columns of B to form a basis for
B.
A Basis for A:
If A = LU then U and A share the same null space. We
know that choosing only the pivot columns in U forms a
suitable basis for U . Then choosing the same columns of A
will form a suitable basis for A.
Example: Find a suitable basis for the columns space of A,
1
3
2 −5
11
A = −2 −6 −2
3
9 14 −11
After LU-decomposition, we find
2 −5
1
2
0
0
1 3
U = 0 0
0 0
and there are pivot elements in columns 1 and 3 of U . Then
columns 1 and 3 of A is a suitable basis for C(A),
1
2
basis(C(A)) = { −2 , −2 }.
3
14
6.7
Solving an under-determined system
We now know how to solve Ax = 0 for a landscape matrix
A. We shall now try to solve Ax = b for the same landscape
matrix.
Under-determined systems have either infinitely many solutions (containing one or more free variables), or no solutions
at all. We shall illustrate this by means of an example.
Example:
Solve Ax = b, with
1
3
2 −5
11 ,
A = −2 −6 −2
3
9 14 −11
and
2
b = −5 .
2
12
The first step is to LU-decompose A (done in Section 4.9),
1 3 2 −5
1 0 0
1 .
A = LU = −2 1 0 0 0 2
0 0 0
0
3 4 1
Then solve Lc = b by forward substitution,
2
c1
1 0 0
−5
c
−2
1
0
=
.
2
2
c3
3 4 1
From the first equation, we find c1 = 2. From the second
equation, −2(2) + c2 = −5, therefore c2 = −1. From the
last equation, 3(2) + 4(−1) + c3 = 2, therefore c3 = 0. Then
2
c = −1 .
0
Now, solve U x = c and choose non-pivot columns to be
associated with free variables. We shall relabel x2 as λ and
x4 as µ.
1 3
0 0
0 0
x
2
2 −5 1
λ
= −1 .
1
2
x3
0
0
0
µ
The last equation is 0 = 0. This equation (obtained from
the row with all zeros) determines whether there is a solution
at all or none. For each row of zeros in U , c must have a
corresponding zero element. If this is not the case, then
there is no solution.
The second last equation is 2x3 + µ = −1, therefore x3 =
− 12 − µ2 .
The first
equation,
with x3 = − 21 − µ2 , substituted, is x1 +
3λ + 2 − 21 − µ2 − 5µ = 2, therefore x1 = 3 − 3λ + 6µ. Then
3 − 3λ + 6µ
3
−3
6
λ
0
1
0
x=
= 1 + λ
+ µ 1 .
µ
1
−2
0
−2
−2 − 2
µ
0
0
1
Notice that the solution contains the null space (previously
found in Section 4.9) plus an extra constant vector. This
constant vector is called the particular solution. Let us denote it by xp and let us denote the null space part by xn .
13
The solution to such a system is always
x = xp + xn ,
where Axp = b and Axn = 0.
If one has previously found the null space of A, then it is
only necessary to find the particular solution as well. This
can be done by setting λ = 0 and µ = 0, and then to solve
1 3
0 0
0 0
6.8
x
2
2 −5 1
0
= −1 .
2
1
x3
0
0
0
0
Solving an over-determined system
We shall illustrate this case with an example.
Solve the following over-determined system:
3x −4y =
6x −6y =
−3x +14y =
9x −10y =
2
6
.
8
8
Then we must solve Ax = b, with
3 −4
6
−6
A=
,
−3
14
9 −10
2
6
b = .
8
8
Do LU-decomposition of A. L has the following form
1
?
L=
?
?
0
1
?
?
0
0
1
?
0
0
0
1
and the first pivot in A is at position (1, 1).
3
−4
−6
6
.
−3
14
9 −10
The three multipliers which will render three zeros in the
first columns, are
`21 =
6
= 2,
3
`31 =
−3
= −1,
3
`41 =
9
= 3.
3
14
Then L becomes
1
2
L=
−1
3
0
1
?
?
0
0
1
?
0
0
0
1
and A0 = E41 E31 E21 A is
3 −4
2
0
A0 =
0 10
0
2
We have already boxed the next pivot element.
The next two multipliers are
`32 =
10
= 5,
2
`42 =
2
= 1.
2
0
1
5
1
0
0
0
1
Then L becomes
1
2
L=
−1
3
0
0
1
?
and U = E42 E321 A0 is
3 −4
0
2
U =
.
0
0
0
0
Note that U is already in upper triangular form, even though
we have not yet done a row operation on row 4 using row 3.
This can be seen by the still unassigned position (4, 3) in L.
Let us put an α there.
1
2
L=
−1
3
0 0
1 0
5 1
1 α
0
0
.
0
1
Check that LU = A, regardless of the value of α. The action
of the multiplier α is as follows: it adds α times the third
row of U (a zero row) to the fourth row of U (a zero row) and
puts it in the place of the fourth row (already a zero row). It
therefore leaves the last row of U unchanged. We normally
just put a zero in the place of such ’free’ multipliers, since
it does not matter what we put there.
15
Next solve Lc = b by forward substitution,
1
2
−1
3
0
1
5
1
0
0
1
0
2
c1
0
0 c2 6
.
=
0 c3 8
8
c4
1
Then
2
2
c=
0
0
Then solve U x = c by backward substitution.
2
3 −4 "
#
2
x
0
2
1
= .
0
0
0 x2
0
0
0
Notice that c has zeros where U has zero rows. This means
we have a solution. Then
"
x=
2
1
#
.
In this case, we have a unique solution, because A has full
rank (as high as can be) and c had zeros in the correct
places. If A is rank deficient and c has zeros in the correct
places then we need to introduce free variables and we have
infinitely many solutions, even though the system is overdetermined.
The most common situation is, however, that A is a full rank
matrix with a c that has non-zero elements corresponding
to zeros rows in U — i.e. an over-determined system usually
has no solution.
6.9
The row and left-null-space
There are two other spaces associated with a matrix, its row
space and its left null space. Let us define them here:
Definition: Row space
The row space of A is the set of all linear combinations of the rows of A.
Let A be m × n. Then the row space of A is expressed as
R(A) = {c ∈ Rn |cT = yT A, y ∈ Rm }
16
Some properties of the row space are listed here. Prove them
all yourself.
(1) R(A) is a vector space,
(2) R(A) = C(AT ),
(3) dim(R(A)) = rank(A),
(4) When A = LU , then U and A have the same row
space. (Remember, this was not the case for the column
space and the null space — they have different column
spaces.)
Definition: Left null space
The left null space of A is the set of all solutions of
y T A = 0T .
Let A be m × n.
expressed as
Then the left null space of A is
L(A) = {y ∈ Rm |yT A = 0T , 0 ∈ Rn }
Some properties of the left null space are listed here. Prove
them by yourself.
(1) L(A) is a vector space,
(2) L(A) = N (AT ),
(3) dim(L(A)) = m − rank(A),
(4) When A = LU , then the rows of L−1 corresponding
to zero rows in U form a basis for the left null space
of A.
Although one may use the current LU-decomposition of A
to find the left null space (it requires inverting L) it is often
better to find the null space of AT from scratch.
6.10
Summary of the four fundamental
spaces
The Rank:
The rank of a matrix is the number of (non zero) pivot
elements in U . It is the dimension of the column space as
17
well as the row space. It is also the effective number of
equations one has in a system.
It is difficult to simple ’see’ the effective number of equations
in a normal system, but once the system is rewritten using U ,
one can clearly see the zero rows of U . If the corresponding
elements in c are zero too, then those equations drop out of
the system. Therefore, the number of non-zero rows in U
are the only effective equations that remain.
Dimensions of the fundamental spaces:
Let us review the dimensions of the various fundamental
spaces of a matrix. We shall first do this for a specific example.
Let A be a 4×5 matrix. Suppose that, after LU-decomposition,
we find that U has the following form
U =
×
0
0
0
×
0
0
0
×
×
0
0
× ×
× ×
0 0
0 0
Then rank(A) = 2. A basis for the columns space of A
consists of columns 1 and 3. The dimension of C(A) is 2.
In the process of solving U x = 0, we will have to introduce
three free variables (one for each non-pivot column). The
dimension of N (A) is 3. It is obtained in general by (the
number of columns) minus (the rank), or 5 − 2 = 3 in the
case of this example.
Since U and A have the same row space, and U has only
two independent rows (the number of pivot elements), A
would also have only two independent rows. The dimension
of R(A) is 2 (the rank).
The left null space is a little more tricky. First LU-decompose
AT , i.e. let AT = L0 U 0 . The number of independent rows in
A (i.e. 2) is the same as the number of independent columns
in AT . In fact, a matrix and its transpose always have the
same rank. Then U 0 will be of the following form (perhaps
requiring row exchanges)
U =
0
×
0
0
0
0
×
×
0
0
0
× ×
× ×
0 0
0 0
0 0
Solving U 0 y = 0 will require introducing two free variables.
18
The dimension of the left null space is therefore the number of rows
minus the rank. Or, in the case of the example, 4 − 2 = 2.
To summarize:
Dimensions of the fundamental spaces
Let A be (m × n) with rank r. Then
C(A) ∈ Rm and has dimension r,
L(A) ∈ Rm and has dimension m − r,
R(A) ∈ Rn and has dimension r,
N (A) ∈ Rn and has dimension n − r.
Orthogonal complement spaces:
If x ∈ N (A), then Ax = 0. Each of the zeros in the null
vector, came from a dot product between a row of A and
x. Every row of A is orthogonal to every vector in the null
space of A. This means that R(A) and N (A) are orthogonal
spaces.
Also note that R(A) and N (A) both lie in Rn . Their dimensions are r and n − r. The sum of their dimensions is
n. This means that R(A) and N (A) are orthogonal complement subspaces of Rn .
Similarly, if y ∈ L(A), then yT A = 0T . Each of the zeros
in the null vector, came from a dot product between y and
a column of A. Every column of A is orthogonal to every
vector in the left null space of A. This means that C(A) and
L(A) are orthogonal spaces.
Similarly, their dimensions are r and m − r respectively, so
that they form orthogonal complement spaces in Rm .
0
advertisement
Download
advertisement
Add this document to collection(s)
You can add this document to your study collection(s)
Sign in Available only to authorized usersAdd this document to saved
You can add this document to your saved list
Sign in Available only to authorized users