Matrix
Analysis
Matrix Analysis
for Scientists
for
Scientists &
& Engineers
Engineers
This
page intentionally
intentionally left
left blank
blank
This page
Matrix
Matrix Analysis
Analysis
for Scientists
Engineers
for
Scientists &
& Engineers
Alan J.
J. Laub
Alan
Laub
University of California
Davis, California
slam.
Copyright © 2005
by the
the Society
Society for
Industrial and
and Applied
Mathematics.
Copyright
2005 by
for Industrial
Applied Mathematics.
10987654321
10987654321
All
America. No
this book
All rights
rights reserved.
reserved. Printed
Printed in
in the
the United
United States
States of
of America.
No part
part of
of this
book
may be
be reproduced,
reproduced, stored,
stored, or
or transmitted
transmitted in
in any
any manner
manner without
the written
may
without the
written permission
permission
of the
publisher. For
For information,
information, write
to the
the Society
Society for
Industrial and
Applied
of
the publisher.
write to
for Industrial
and Applied
Mathematics,
Mathematics, 3600
3600 University
University City
City Science
Science Center,
Center, Philadelphia,
Philadelphia, PA
PA 19104-2688.
19104-2688.
MATLAB® is
is a
a registered
registered trademark
trademark of
The MathWorks,
MathWorks, Inc.
Inc. For
For MATLAB
MATLAB product
product information,
information,
MATLAB®
of The
please
contact The
Apple Hill
01760-2098 USA,
USA,
please contact
The MathWorks,
MathWorks, Inc.,
Inc., 3
3 Apple
Hill Drive,
Drive, Natick,
Natick, MA
MA 01760-2098
508-647-7000, Fax:
Fax: 508-647-7101,
508-647-7101, info@mathworks.com,
www.mathworks.com
508-647-7000,
info@mathworks.com, wwwmathworks.com
Mathematica is
is a
a registered
registered trademark
trademark of
of Wolfram
Wolfram Research,
Research, Inc.
Mathematica
Inc.
Mathcad is
is a
a registered
registered trademark
of Mathsoft
Mathsoft Engineering
Engineering &
& Education,
Education, Inc.
Mathcad
trademark of
Inc.
Library of
of Congress
Congress Cataloging-in-Publication
Cataloging-in-Publication Data
Data
Library
Laub, Alan
J., 19481948Laub,
Alan J.,
Matrix analysis
scientists and
and engineers
engineers // Alan
Matrix
analysis for
for scientists
Alan J.
J. Laub.
Laub.
p. cm.
cm.
p.
Includes bibliographical
bibliographical references
references and
and index.
Includes
index.
ISBN 0-89871-576-8
0-89871-576-8 (pbk.)
(pbk.)
ISBN
1. Matrices.
Matrices. 2.
2. Mathematical
Mathematical analysis.
analysis. I.I. Title.
Title.
1.
QA188138
2005
QA
188.L38 2005
512.9'434—dc22
512.9'434-dc22
2004059962
2004059962
About
the cover:
cover: The
The original
original artwork
artwork featured
on the
cover was
created by
by freelance
About the
featured on
the cover
was created
freelance
permission .
artist
Aaron Tallon
artist Aaron
Tallon of
of Philadelphia,
Philadelphia, PA.
PA. Used
Used by
by permission.
•
slam
5.lam...
is a
a registered
registered trademark.
is
trademark.
To
To my
my wife,
wife, Beverley
Beverley
(who captivated
captivated me in the UBC
UBC math library
nearly
forty years ago)
nearly forty
This
page intentionally
intentionally left
left blank
blank
This page
Contents
Contents
Preface
Preface
xi
xi
11
Introduction
Introduction and
and Review
Review
1.1
Notation and
1.1 Some
Some Notation
and Terminology
Terminology
1.2 Matrix
Matrix Arithmetic
1.2
Arithmetic . . . . . . . .
1.3 Inner
Inner Products
and Orthogonality
1.3
Products and
Orthogonality .
1.4
Determinants
1.4 Determinants
11
11
33
4
44
2
2
Vector
Vector Spaces
Spaces
2.1 Definitions
Examples .
2.1
Definitions and
and Examples
2.2 Subspaces.........
2.2
Subspaces
2.3
2.3 Linear
Linear Independence
Independence . . .
2.4 Sums
and Intersections
Intersections of
2.4
Sums and
of Subspaces
Subspaces
77
77
99
10
10
13
13
33
Linear
Linear Transformations
Transformations
3.1 Definition
Definition and
Examples . . . . . . . . . . . . .
3.1
and Examples
3.2
Matrix Representation
of Linear
3.2 Matrix
Representation of
Linear Transformations
Transformations
3.3 Composition
Transformations . .
3.3
Composition of
of Transformations
3.4 Structure
of Linear
Linear Transformations
Transformations
3.4
Structure of
3.5
3.5 Four
Four Fundamental
Fundamental Subspaces
Subspaces . . . .
17
17
17
17
18
18
19
19
20
20
22
22
4
4
Introduction
Introduction to
to the
the Moore-Penrose
Moore-Penrose Pseudoinverse
Pseudoinverse
4.1
Definitions
and Characterizations
Characterizations.
4.1
Definitions and
4.2
Examples..........
4.2 Examples
4.3
Properties and
and Applications
Applications . . . .
4.3 Properties
29
29
30
30
31
31
55
Introduction
Introduction to
to the
the Singular
Singular Value
Value Decomposition
Decomposition
5.1
5.1 The
The Fundamental
Fundamental Theorem
Theorem . . .
5.2 Some
Basic Properties
Properties . . . . .
5.2
Some Basic
5.3
Row and Column
Compressions
5.3 Rowand
Column Compressions
35
35
35
35
38
40
6
6
Linear
Linear Equations
Equations
6.1 Vector
Vector Linear
Linear Equations
Equations . . . . . . . . .
6.1
6.2 Matrix
Linear Equations
Equations . . . . . . . .
6.2
Matrix Linear
6.3
6.3 A
A More
More General
General Matrix
Matrix Linear
Linear Equation
Equation
6.4 Some
Useful and
and Interesting
Inverses.
6.4
Some Useful
Interesting Inverses
43
43
43
43
vii
44
47
47
47
47
viii
viii
Contents
Contents
7
Projections, Inner Product Spaces, and Norms
7.1
Projections . . . . . . . . . . . . . . . . . . . . . .
7.1
Projections
7.1.1
The
fundamental orthogonal
orthogonal projections
projections
7.1.1
The four
four fundamental
7.2
Inner Product
Product Spaces
Spaces
7.2 Inner
7.3
7.3 Vector
Vector Norms
Norms
7.4
Matrix Norms
Norms . . . .
7.4 Matrix
51
51
51
51
52
52
54
54
57
57
59
59
8
Linear Least Squares Problems
8.1
Linear Least
Least Squares
Problem . . . . . . . . . . . . . .
8.1 The
The Linear
Squares Problem
8.2
8.2 Geometric
Geometric Solution
Solution . . . . . . . . . . . . . . . . . . . . . .
8.3
Linear Regression
Regression and
and Other
8.3 Linear
Other Linear
Linear Least
Least Squares
Squares Problems
Problems
8.3.1
Linear regression
8.3.1 Example:
Example: Linear
regression . . . . . . .
8.3.2
problems . . . . . . .
8.3.2 Other
Other least
least squares
squares problems
8.4
Least Squares
8.4 Least
Squares and
and Singular
Singular Value
Value Decomposition
Decomposition
8.5 Least
Squares and
QR Factorization
Factorization . . . . . . .
8.5
Least Squares
and QR
65
65
65
65
67
67
67
67
67
67
69
70
70
71
71
9
Eigenvalues and Eigenvectors
9.1
Fundamental Definitions
Definitions and
Properties
9.1
Fundamental
and Properties
9.2 Jordan
Jordan Canonical
Canonical Form
Form . . . . .
9.2
the JCF
9.3
Determination of
9.3 Determination
of the
JCF . . . . .
9.3.1
Theoretical
computation .
9.3.1
Theoretical computation
l's in
in JCF
blocks
9.3.2 On
the +
9.3.2
On the
+1's
JCF blocks
9.4 Geometric
Aspects of
JCF
of the
the JCF
9.4
Geometric Aspects
9.5 The
The Matrix
Sign Function
Function.
9.5
Matrix Sign
75
75
75
82
82
85
85
86
86
88
88
89
89
91
91
10 Canonical Forms
10.1
Basic Canonical
10.1 Some
Some Basic
Canonical Forms
Forms .
10.2 Definite
10.2
Definite Matrices
Matrices . . . . . . .
10.3
Equivalence Transformations
Transformations and
10.3 Equivalence
and Congruence
Congruence
10.3.1
matrices and
10.3.1 Block
Block matrices
and definiteness
definiteness
10.4
Rational Canonical
10.4 Rational
Canonical Form
Form . . . . . . . . .
95
95
95
95
99
102
102
104
104
104
104
11 Linear
Differential and
and Difference
Difference Equations
Equations
11
Linear Differential
11.1 Differential
ILl
Differential Equations
Equations . . . . . . . . . . . . . . . .
11.1.1
matrix exponential
11.1.1 Properties
Properties ofthe
of the matrix
exponential . . . .
11.1.2
11.1.2 Homogeneous
Homogeneous linear
linear differential
differential equations
equations
11.1.3
11.1.3 Inhomogeneous
Inhomogeneous linear
linear differential
differential equations
equations
11.1.4
Linear matrix
differential equations
11.1.4 Linear
matrix differential
equations . .
11.1.5
decompositions . . . . . . . . .
11.1.5 Modal
Modal decompositions
matrix exponential
11.1.6
11.1.6 Computation
Computation of
of the
the matrix
exponential
11.2 Difference
Equations . . . . . . . . . . . . . .
11.2
Difference Equations
11.2.1
linear difference
difference equations
11.2.1 Homogeneous
Homogeneous linear
equations
11.2.2
Inhomogeneous
linear
difference
equations
11.2.2 Inhomogeneous linear difference equations
11.2.3
powers .
11.2.3 Computation
Computation of
of matrix
matrix powers
Equations. . . . . . . . . . . . . . .
11.3
Higher-Order Equations
11.3 Higher-Order
109
109
109
109
109
109
112
112
112
112
113
113
114
114
114
114
118
118
118
118
118
118
119
119
120
120
Contents
Contents
ix
ix
12
Generalized Eigenvalue
Eigenvalue Problems
Problems
12 Generalized
12.1
The
Generalized
EigenvaluelEigenvector
12.1 The Generalized Eigenvalue/Eigenvector Problem
Problem
12.2
Forms . . . . . . . . . . . . . . . . .
12.2 Canonical
Canonical Forms
12.3
Application to
to the
the Computation
of System
Zeros .
12.3 Application
Computation of
System Zeros
12.4
Generalized Eigenvalue
Eigenvalue Problems
12.4 Symmetric
Symmetric Generalized
Problems .
12.5 Simultaneous
Simultaneous Diagonalization
12.5
Diagonalization . . . . . . . . .
12.5.1 Simultaneous
Simultaneous diagonalization
12.5.1
diagonalization via
via SVD
SVD
12.6 Higher-Order
Higher-Order Eigenvalue
Problems ..
12.6
Eigenvalue Problems
12.6.1 Conversion
Conversion to
first-order form
form
12.6.1
to first-order
125
125
125
127
127
130
131
131
133
133
133
135
135
135
13 Kronecker
13
Kronecker Products
Products
13.1 Definition
and Examples
Examples . . . . . . . . . . . . .
13.1
Definition and
13.2 Properties
Properties of
of the
the Kronecker
Kronecker Product
Product . . . . . . .
13.2
13.3
Application to
to Sylvester
and Lyapunov
Lyapunov Equations
Equations
13.3 Application
Sylvester and
139
139
139
139
140
144
144
Bibliography
Bibliography
151
Index
Index
153
This
page intentionally
intentionally left
left blank
blank
This page
Preface
Preface
This
intended to
for beginning
(or even
even senior-level)
This book
book is
is intended
to be
be used
used as
as aa text
text for
beginning graduate-level
graduate-level (or
senior-level)
students in
the sciences,
sciences, mathematics,
computer science,
science, or
students
in engineering,
engineering, the
mathematics, computer
or computational
computational
science who wish to be familar with enough
prepared to
science
enough matrix analysis
analysis that they
they are
are prepared
to use its
tools and
ideas comfortably
in aa variety
variety of
applications. By
By matrix
matrix analysis
analysis II mean
mean linear
tools
and ideas
comfortably in
of applications.
linear
algebra and
and matrix
application to
algebra
matrix theory
theory together
together with
with their
their intrinsic
intrinsic interaction
interaction with
with and
and application
to
linear
linear differential
text
linear dynamical
dynamical systems
systems (systems
(systems of
of linear
differential or
or difference
difference equations).
equations). The
The text
can
be used
used in
one-quarter or
or one-semester
one-semester course
course to
to provide
provide aa compact
compact overview
of
can be
in aa one-quarter
overview of
much
important and
and useful
useful mathematics
mathematics that,
that, in
many cases,
cases, students
meant to
to learn
learn
much of
of the
the important
in many
students meant
thoroughly
somehow didn't
manage to
topics
thoroughly as
as undergraduates,
undergraduates, but
but somehow
didn't quite
quite manage
to do.
do. Certain
Certain topics
that may
may have
have been
been treated
treated cursorily
cursorily in
in undergraduate
undergraduate courses
courses are
treated in
more depth
that
are treated
in more
depth
and more
more advanced
is introduced.
only the
and
advanced material
material is
introduced. II have
have tried
tried throughout
throughout to
to emphasize
emphasize only
the
more important and "useful" tools, methods, and mathematical structures. Instructors are
encouraged
to supplement
the book
book with
with specific
specific application
from their
their own
own
encouraged to
supplement the
application examples
examples from
particular
area.
particular subject
subject area.
The
choice of
algebra and
and matrix
matrix theory
theory is
is motivated
motivated both
both by
by
The choice
of topics
topics covered
covered in
in linear
linear algebra
applications and
computational utility
relevance. The
The concept
of matrix
applications
and by
by computational
utility and
and relevance.
concept of
matrix factorization
factorization
is
is emphasized
emphasized throughout
throughout to
to provide
provide aa foundation
foundation for
for aa later
later course
course in
in numerical
numerical linear
linear
algebra.
are stressed
than abstract
vector spaces,
spaces, although
although Chapters
and 3
3
algebra. Matrices
Matrices are
stressed more
more than
abstract vector
Chapters 22 and
do cover
cover some
geometric (i.e.,
subspace) aspects
aspects of
fundamental
do
some geometric
(i.e., basis-free
basis-free or
or subspace)
of many
many of
of the
the fundamental
notions. The books by Meyer [18], Noble and Daniel [20], Ortega
Ortega [21], and Strang [24]
are
excellent companion
companion texts
for this
book. Upon
course based
based on
on this
this
are excellent
texts for
this book.
Upon completion
completion of
of aa course
text,
the student
is then
then well-equipped
to pursue,
pursue, either
via formal
formal courses
through selftext, the
student is
well-equipped to
either via
courses or
or through
selfstudy, follow-on topics on the computational side (at the level of [7], [II],
[11], [23], or [25], for
example) or
or on
on the
side (at
level of
[12], [13],
[13], or
[16], for
example).
of [12],
or [16],
for example).
example)
the theoretical
theoretical side
(at the
the level
essentially just an understanding
Prerequisites for
for using this
this text are quite modest: essentially
understanding
of
and definitely
some previous
previous exposure
to matrices
matrices and
linear algebra.
Basic
of calculus
calculus and
definitely some
exposure to
and linear
algebra. Basic
concepts such
such as
determinants, singularity
singularity of
eigenvalues and
concepts
as determinants,
of matrices,
matrices, eigenvalues
and eigenvectors,
eigenvectors, and
and
positive definite matrices
matrices should have been covered at least
least once, even though their recollection
may occasionally
occasionally be
be "hazy."
However, requiring
requiring such
material as
as prerequisite
prerequisite permits
tion may
"hazy." However,
such material
permits
the early
"out-of-order" by
standards) introduction
of topics
the
early (but
(but "out-of-order"
by conventional
conventional standards)
introduction of
topics such
such as
as pseupseudoinverses and
and the
singular value
decomposition (SVD).
tools
doinverses
the singular
value decomposition
(SVD). These
These powerful
powerful and
and versatile
versatile tools
can
can then be exploited
exploited to
to provide a unifying foundation
foundation upon which to base subsequent
subsequent toptopics.
Because tools
tools such
the SVD
are not
not generally
generally amenable
to "hand
"hand computation,"
computation," this
this
ics. Because
such as
as the
SVD are
amenable to
approach necessarily
availability of
of appropriate
mathematical software
software on
appropriate mathematical
on
approach
necessarily presupposes
presupposes the
the availability
aa digital
digital computer.
computer. For
For this,
this, II highly
highly recommend
recommend MAlLAB®
MATLAB® although
although other
other software
software such
such as
as
xi
xi
xii
xii
Preface
Preface
Mathcad® is also excellent. Since this text is not intended for a course in
Mathematica® or Mathcad®
numerical linear algebra per
per se,
se, the details of most of the numerical aspects of linear algebra
are
deferred to
are deferred
to such
such aa course.
course.
The presentation of the material in this book is
is strongly influenced
influenced by
by computacomputational issues for two principal reasons. First, "real-life"
"real-life" problems seldom yield to simple
closed-form
closed-form formulas or solutions. They must generally be solved computationally and
it is important to know which types of algorithms can be relied upon and which cannot.
Some of
of the
numerical linear
linear algebra,
form the
Some
the key
key algorithms
algorithms of
of numerical
algebra, in
in particular,
particular, form
the foundation
foundation
virtually all of modern
modem scientific and engineering computation. A second
upon which rests virtually
motivation for a computational emphasis is that it provides many of the essential tools for
what I call "qualitative mathematics."
mathematics." For example, in an elementary linear algebra course,
a set of vectors is either linearly independent or it is not. This is an absolutely fundamental
fundamental
concept. But in most engineering or scientific contexts we want to know more than that.
If
linearly independent,
independent, how "nearly dependent" are the vectors? If
If a set of vectors is linearly
If they
are linearly dependent, are there "best" linearly independent subsets? These tum
turn out to
be
more difficult
difficult problems
frequently involve
involve research-level
research-level questions
questions when
be much
much more
problems and
and frequently
when set
set
in the context of
of the finite-precision, finite-range floating-point arithmetic environment of
of
most modem
modern computing platforms.
Some of
of the
the applications
applications of
of matrix
matrix analysis
analysis mentioned
mentioned briefly
briefly in
in this
this book
book derive
modem state-space
from the modern
state-space approach to dynamical systems. State-space
State-space methods are
modem engineering where, for example, control systems with
now standard
standard in much of modern
large numbers
numbers of interacting inputs, outputs, and states often give rise to models
models of very
high order that must be analyzed, simulated, and evaluated. The "language" in which such
described involves vectors and matrices. It is thus crucial to acquire
models are conveniently described
knowledge of the vocabulary
vocabulary and grammar of this language. The tools of matrix
a working knowledge
analysis are also applied
applied on a daily basis to problems in biology, chemistry, econometrics,
physics, statistics, and a wide variety of other fields, and thus the text can serve a rather
diverse audience.
audience. Mastery of the material in this text should enable the student to read and
diverse
understand the modern
modem language of matrices used throughout mathematics, science, and
engineering.
prerequisites for this text are modest, and while most material is developed
developed from
While prerequisites
basic ideas in the book, the student does require a certain amount of what is conventionally
referred to as "mathematical maturity." Proofs
Proofs are given for many theorems. When they are
referred
not
given explicitly,
obvious or
or easily
easily found
found in
literature. This
This is
is ideal
ideal
not given
explicitly, they
they are
are either
either obvious
in the
the literature.
material from which to learn a bit about mathematical proofs and the mathematical maturity
and insight gained thereby. It is my firm conviction
conviction that such maturity is neither
neither encouraged
nor nurtured by relegating the mathematical aspects of applications (for example, linear
algebra for elementary state-space theory) to
introducing it "on-the-f1y"
"on-the-fly" when
algebra
to an appendix or introducing
foundation upon
necessary. Rather,
Rather, one must
must lay
lay a firm
firm foundation
upon which
which subsequent applications and
and
perspectives can be built in a logical, consistent, and coherent fashion.
perspectives
I have taught this material for many years, many times at UCSB and twice at UC
Davis,
course has
successful at
enabling students
students from
from
Davis, and
and the
the course
has proven
proven to
to be
be remarkably
remarkably successful
at enabling
disparate backgrounds to acquire a quite acceptable
acceptable level of mathematical maturity and
graduate studies in a variety of disciplines. Indeed, many students who
rigor for subsequent graduate
completed the course, especially
especially the first few times it was offered,
offered, remarked afterward that
completed
if only they had had this course before they took linear systems, or signal processing.
processing,
if
Preface
Preface
xiii
XIII
or estimation theory, etc., they would have been able to concentrate on the new ideas
deficiencies in their
they wanted to learn, rather than having to spend time making up for deficiencies
background in matrices and linear algebra. My fellow instructors, too, realized that by
background
requiring this course as a prerequisite, they no longer had to provide as much time for
"review" and could focus instead on the subject at hand. The concept seems to work.
-AJL,
— AJL, June 2004
This
page intentionally
intentionally left
left blank
blank
This page
Chapter 1
Chapter
1
Introduction and
and Review
Introduction
Review
1.1
1.1
Some Notation
Notation and
and Terminology
Terminology
Some
We
begin with
with aa brief
brief introduction
notation and
used
We begin
introduction to
to some
some standard
standard notation
and terminology
terminology to
to be
be used
throughout the
text. This
This is
review of
of some
some basic
notions in
throughout
the text.
is followed
followed by
by aa review
basic notions
in matrix
matrix analysis
analysis
and linear
linear algebra.
algebra.
and
The
The following
following sets
sets appear
appear frequently
frequently throughout
throughout subsequent
subsequent chapters:
chapters:
1.
Rnn== the
the set
set of
of n-tuples
n-tuples of
of real
real numbers
as column
column vectors.
vectors. Thus,
Thus, xx Ee Rn
I. IR
numbers represented
represented as
IR n
means
means
where Xi
xi Ee R
for ii Ee !!.
n.
IR for
where
Henceforth,
the notation!!
notation n denotes
denotes the
the set
set {I,
{1, ...
..., , nn}.
Henceforth, the
}.
Note: Vectors
Vectors are
vectors. A
vector is
where
Note:
are always
always column
column vectors.
A row
row vector
is denoted
denoted by
by y~
yT, where
yy G
E Rn
IR n and
and the
the superscript
superscript T
T is
is the
the transpose
transpose operation.
operation. That
That aa vector
vector is
is always
always aa
column vector
vector rather
rather than
row vector
vector is
entirely arbitrary,
arbitrary, but
this convention
convention makes
makes
column
than aa row
is entirely
but this
it
text that,
x TTyy is
while
it easy
easy to
to recognize
recognize immediately
immediately throughout
throughout the
the text
that, e.g.,
e.g., X
is aa scalar
scalar while
T
xy
is an
an nn xx nn matrix.
xyT is
matrix.
en
2. Cn = the
the set
set of
of n-tuples
n-tuples of
of complex
complex numbers
numbers represented
represented as
as column
column vectors.
vectors.
2.
3. IR
xn =
Rrnmxn
= the
the set
set of
of real
real (or
(or real-valued)
real-valued) m
m xx nn matrices.
matrices.
4. 1R;n
xn
Rmxnr
=
xn denotes
= the set
set of
of real
real m x n matrices of
of rank
rank r. Thus,
Thus, IR~
Rnxnn
denotes the
the set
set of
of real
real
nonsingular
matrices.
nonsingular n
n xx nn matrices.
e
mxn
5.
=
5. Crnxn
= the
the set
set of
of complex
complex (or
(or complex-valued)
complex-valued) m xx nn matrices.
matrices.
6. e;n
xn
Cmxn
=
n matrices
= the
the set
set of
of complex
complex m
m xx n
matrices of
of rank
rank r.
r.
1
Chapter 1.
1. Introduction
Introduction and
and Review
Review
Chapter
22
We now classify some of the more familiar "shaped" matrices. A matrix A Ee IRn xn
x
(or A
A E
enxn
) is
eC"
")is
diagonal if
if aij
a,7 == 00 for
forii i=
^ }.j.
•• diagonal
upper triangular
triangular if
if aij
a,; == 00 for
forii >> }.j.
•• upper
lower triangular
triangular if
if aij
a,7 == 00 for
for i/ << }.j.
•• lower
tridiagonal if
if aij
a(y =
= 00 for
for Ii|z -—JI
j\ >
> 1.
•• tridiagonal
1.
pentadiagonal if
if aij
ai; =
= 00 for
for Ii|/ -—J
j\I >> 2.
•• pentadiagonal
2.
upper Hessenberg
Hessenberg if
if aij
afj == 00 for
for ii -— jj >> 1.
•• upper
1.
lower Hessenberg
Hessenberg if
if aij
a,; == 00 for
for }j -—ii >> 1.
•• lower
1.
Each of the above also has a "block" analogue obtained by replacing scalar components in
nxn
mxn
the respective definitions
definitions by block
block submatrices.
submatrices. For
For example,
example, if
if A Ee IR
Rnxn
, , B Ee IR
R nxm
,, and
C Ee jRmxm,
Rmxm, then
then the
the (m
(m + n)
n) xx (m
(m + n)
n) matrix
matrix [~
[A0Bc
block upper
upper triangular.
triangular.
~]] isisblock
C
T
A is
AT and
is the
matrix whose
entry
The transpose of
The
of aa matrix
matrix A
is denoted
denoted by
by A
and is
the matrix
whose (i, j)th
j)th entry
7
mx
A, that is, (AT)ij
A E
jRmxn,
AT7" e
E jRnxm.
is the (j,
(7, i)th
Oth entry of A,
(A ),, = aji.
a,,. Note that if A
e R
", then A
E" xm .
If
A Ee em
If A
C mxxn,
", then its Hermitian
Hermitian transpose (or conjugate
conjugate transpose) is denoted by AHH (or
H
sometimes
A*) and
j)th entry
is (AH)ij
the bar
bar indicates
sometimes A*)
and its
its (i, j)\h
entry is
(A ), 7 =
= (aji),
(«77), where
where the
indicates complex
complex
= a
IX + jf$
jfJ (j
= ii =
jfJ. A
A is
conjugation; i.e.,
i.e., if z =
(j =
= R),
v^T), then z =
= IX
a -— jfi.
A matrix A
is symmetric
T
H
if
A =
A T and Hermitian
A =
A H.
We henceforth
if A
= A
Hermitian if A
= A
. We
henceforth adopt the convention that,
that, unless
otherwise noted,
an equation
equation like
= A
ATT implies
implies that
that A
is real-valued
real-valued while
while aa statement
A =
A is
statement
otherwise
noted, an
like A
H
like A
A =
AH implies that A
A is complex-valued.
= A
complex-valued.
z
Remark
While \/—\
most commonly
commonly denoted
denoted by
in mathematics
mathematics texts,
Remark 1.1. While
R isis most
by ii in
texts, }j is
is
the
common notation
notation in
in electrical
and system
system theory.
is some
some
the more
more common
electrical engineering
engineering and
theory. There
There is
advantage to being conversant with both notations. The notation j is used throughout the
text but
but reminders
reminders are
text
are placed
placed at
at strategic
strategic locations.
locations.
Example 1.2.
1.2.
Example
~
1. A = [ ;
2. A
5
= [ 7+}
3 · A -- [ 7 -5 j
is symmetric
symmetric (and
Hermitian).
] is
(and Hermitian).
7+
is complex-valued
symmetric but
Hermitian.
2 j ] is
complex-valued symmetric
but not
not Hermitian.
7+}
is Hermitian
Hermitian (but
symmetric).
2 ] is
(but not
not symmetric).
Transposes
block matrices
be defined
defined in
obvious way.
is
Transposes of
of block
matrices can
can be
in an
an obvious
way. For
For example,
example, it
it is
easy to
to see
see that
that if
if A,,
are appropriately
appropriately dimensioned
dimensioned subblocks,
subblocks, then
easy
Aij are
then
r
= [
1.2. Matrix Arithmetic
3
11.2
.2 Matrix Arithmetic
Arithmetic
It is assumed that the reader is familiar with the fundamental notions of matrix addition,
multiplication of a matrix by a scalar, and multiplication of matrices.
A special case of matrix multiplication
multiplication occurs when the second
second matrix is a column
i.e., the matrix-vector product Ax.
Ax. A very important way to view this product is
vector x, i.e.,
interpret it as a weighted
weighted sum (linear
combination) of the columns of A. That is, suppose
to interpret
(linear combination)
suppose
A =
la' ....• a"1
E
m
JR " with a,
Then
Ax =
Xjal
E
JRm and x =
+ ... + Xnan
Il ;xn~
]
E jRm.
The importance
importance of this interpretation
interpretation cannot be overemphasized. As a numerical example,
take
= [96
take A
A =
[~ 85 74]x
~], x ==
!
2 . Then
can quickly
quickly calculate
dot products
rows of
[~].
Then we
we can
calculate dot
products of
of the
the rows
of A
A
column x to find Ax
Ax =
= [50[;~],
matrix-vector product
product can also be computed
with the column
32]' but this matrix-vector
computed
via
v1a
3.[ ~ J+2.[ ~ J+l.[ ~ l
For large arrays of numbers, there can be important computer-architecture-related
computer-architecture-related advantages to preferring the latter calculation method.
mxn
nxp
multiplication, suppose
A e
E R
jRmxn and
and B = [bi,...,b
[hI,.'" hpp]] e
E R
jRnxp with
For matrix multiplication,
suppose A
1
hi E
jRn.. Then the matrix product A
AB
bi
e W
B can be thought of as above, applied p times:
There is also an alternative, but equivalent, formulation of matrix multiplication that appears
frequently in the text and is presented below as a theorem. Again, its importance cannot be
overemphasized. It
It is deceptively simple and its full understanding is well rewarded.
pxn
Theorem 1.3.
[Uj, ....
Theorem
1.3. Let U
U = [MI,
. . ,, un]
un]Ee jRmxn
Rmxn with
withUiut Ee jRm
Rm and
andVV == [VI,
[v{.•.
,...,, Vn]
vn]Ee lRRPxn
p
jRP.
with Vi
vt eE R
. Then
n
UV T
=
LUiVr E jRmxp.
i=I
If
(C D)TT =
If matrices C and D are compatible for multiplication, recall that (CD)
= DT
DT C TT
H
H H
(or (CD}
(C D)H =— DH
C H).). This gives a dual to the matrix-vector
matrix-vector result above. Namely, if
if
D C
mxn
jRmxn has
C EeR
has row
row vectors cJ
cj Ee jRlxn,
E l x ", and
and is
is premultiplied
premultiplied by
by aa row
row vector yT
yTeE jRlxm,
Rlxm,
then the product can be written as a weighted linear sum of the rows of C as follows:
follows:
yTC=YICf +"'+Ymc~
EjRlxn.
Theorem 1.3 can then also be generalized to its "row
reader.
Theorem
"row dual." The details are left
left to the readei
4
4
1.3
1.3
Chapter
Review
Chapter 1.
1. Introduction
Introduction and
and Review
Inner
Inner Products
Products and
and Orthogonality
Orthogonality
For
IRn, the
Euclidean inner
inner product
For vectors
vectors x, yy E
e R",
the Euclidean
product (or inner
inner product, for
for short)
short) of x and
is given
given by
by
yy is
n
T
(x, y) := x y = Lx;y;.
;=1
Note that
that the
inner product
product is
is aa scalar.
Note
the inner
scalar.
If
we define
complex Euclidean
inner product
product (or
(or inner
inner product,
product,
If x, y Ee <en,
C", we
define their
their complex
Euclidean inner
for short)
short) by
for
by
n
(x'Y}c :=xHy
= Lx;y;.
;=1
y)c
x}c,
Note that (x,
(x, y)
= (y,
(y, x)
i.e., the order
order in
in which
which xx and yy appear
appear in
in the complex inner
c =
c, i.e.,
product is
is important.
important. The
The more
more conventional
conventional definition
definition of
of the
the complex
inner product
product is
is
product
complex inner
((x,
x , yy)c
)c =
yHxx =
Eni=1 x;y;
xiyi but
the text
text we
with the
= yH
= L:7=1
but throughout
throughout the
we prefer
prefer the
the symmetry
symmetry with
the real
real
case.
case.
Example
1.4. Let
[1j]] and
and yy == [~].
[1/2]. Then
Then
Example 1.4.
Let xx =
= [}
(x, Y}c = [ }
JH [ ~ ] =
[I
- j] [
~
] = 1 - 2j
while
while
and we
see that,
indeed, (x,
(x, Y}c
y)c =
= {y,
(y, x)c'
x)c.
and
we see
that, indeed,
Note that
that xx TTxx =
= 0
0 if
if and
and only
only if
if xx =
= 00 when
when xx eE Rn
IRn but
but that
that this
this is
is not
not true
true if
ifxx eE Cn.
en.
Note
HH
What
is true
complex case
and only
if x = 0.
illustrate, consider
consider
What is
true in
in the
the complex
case is
is that
that X
x x = 00 if
if and
only if
O. To
To illustrate,
T
H
the
nonzero vector
=0
the nonzero
vector xx above.
above. Then
Then X
x TXx =
0 but
but X
x HXX =
= 2.2.
n
Two nonzero
nonzero vectors
vectors x,
x, y eE IR
to be
be orthogonal
if their
their inner
product is
is
Two
R are
are said
said to
orthogonal if
inner product
H
zero,
i.e., xxTTyy =
= 0.
if X
0. If xx and
zero, i.e.,
O. Nonzero
Nonzero complex
complex vectors
vectors are
are orthogonal
orthogonal if
x Hyy =
= O.
and yy are
are
T
T
orthogonal and
and X
x TXx =
and yyT
= 1,1, then
then we
we say
say that
that xx and
are orthonormal.
orthonormal. A
A
orthogonal
= 11 and
yy =
and yy are
nxn
T
T
nxn
matrix A eE IR
is an
orthogonal matrix
matrix if
if A
AT
AAT
=
I, where
where /I is
is the
the n
n x
x nn
matrix
R
is
an orthogonal
AA =
= AA
= /,
nx
identity matrix.
matrix. The notation /„
In is sometimes
identity
sometimes used
used to denote
denote the identity matrix in
in IRRnxn
"
x
nxn
H
H
(or en xn).
A eE en
= I. Clearly
(orC"
"). Similarly,
Similarly, a matrix A
C xn is said
said to be unitary if A H A =
= AA H =
an orthogonal
orthogonal or
or unitary
unitary matrix
rows and
is
an
matrix has
has orthonormal
orthonormal rows
and orthonormal
orthonormal columns.
columns. There
There is
mxn
no special
name attached
attached to
to aa nonsquare
nonsquare matrix
matrix A
A e
E ]Rrn"n
(or €
E e
))with
no
special name
R mxn (or
Cmxn
with orthonormal
orthonormal
rows
columns.
rows or
or columns.
1.4
1.4
Determinants
Determinants
It
A E
IRnnxn
xn
It is assumed
assumed that the reader is familiar with the basic theory of
of determinants.
determinants. For A
eR
nxn
(or A
A 6
E en
we use
use the
the notation
det A
A for
determinant of
of A.
A. We
We list
list below
below some
some of
of
(or
C xn)
) we
notation det
for the
the determinant
1.4. Determinants
1.4.
Determinants
5
properties of determinants. Note that this is
the more
more useful properties
is not aa minimal set, i.e., several
of
one
or
more
of
the
others.
properties
are
consequences
properties are consequences of one or more of the others.
1. If
If A
A has a zero row or if any two rows of A
A are equal, then det A
A =
= 0.o.
= 0.
2. If
If A
A has
has aa zero
zero column
column or
or if
if any
any two
two columns
columns of
of A
A are
are equal,
equal, then
then det
det A
A =
O.
3. Interchanging
of A
sign of
3.
Interchanging two
two rows
rows of
A changes
changes only
only the
the sign
of the
the determinant.
determinant.
4. Interchanging two columns of A changes only the sign of
of the determinant.
5.
scalar a
5. Multiplying
Multiplying aa row
row of
of A
A by
by aa scalar
ex results
results in
in aa new
new matrix
matrix whose
whose determinant
determinant is
is
a det A.
exdetA.
Multiplying a column of A
A by a scalar
6. Multiplying
scalar ex
a results in a new matrix whose determinant
determinant
is
a det
is ex
det A.
A.
7. Multiplying
of A
scalar and
and then
then adding
adding it
it to
7.
Multiplying aa row
row of
A by
by aa scalar
to another
another row
row does
does not
not change
change
the
the determinant.
determinant.
8. Multiplying aa column
8.
column of
of A by a scalar
scalar and then adding it to another column
column does
does not
change the
the determinant.
change
determinant.
nxn
9. det
detAT
= det
detA
= detA
A eE C
C"X").
AT =
A (detA
(det AHH =
det A if A
).
10.
If A is diagonal,
diagonal, then det A =
=a11a22
alla22 ...
10. If
• • • ann,
ann, i.e.,
i.e., det
det AA isis the
the product
product of
of its
its diagonal
diagonal
elements.
a22 ...
11.
11. If
If A is upper triangular, then det
det A =
= all
a11a22
• • • a"n.
ann.
12. If
triangular, then
= a11a22
• • • ann.
ann.
12.
If A
A is
is lower
lower triangUlar,
then det
det A
A=
alla22 ...
13.
A is block
block diagonal
block upper triangular or block lower triangular), with
13. If A
diagonal (or
(or block
A 11, A22,
A 22 , ...
An"
A ==
square diagonal blocks A11,
• • •,, A
(of possibly different
different sizes), then det A
nn (of
det
A 11 det
det A22
A22 ...
det Ann.
det A11
• • • det
Ann.
xn
14. If
eRIRnnxn
,thendet(AB)
= det
5.
14.
If A,
A, B
B E
, then det(AB) =
det A
A det
det B.
1
15. If
If A
Rnxn, then
=1det
15.
A €
E lR~xn,
then det(Adet(A- 1)) =
de: AA.
.
nxn
xm
mxm
16.
A eE R
lR~xn
and D
DE
IR m
detA
det(D –- CA–
CA-l 1 B).
B).
16. If
If A
and
eR
,, then det
det [~
[Ac B~]
A det(D
D] = del
Proof:
from the
LU factorization
Proof" This
This follows
follows easily
easily from
the block
block LU
factorization
[~ ~J=[
~ ][ ~
xn
mxm
17. If
If A
and D
D eE RM
, then
then det
det [~
[Ac B~]
BD
– 11C
).
17.
A Ee R
IRnnxn
and
lR~xm,
det D
D det(A
det(A -– B
DC).
D] = det
Proof" This follows easily from the block UL factorization
Proof:
BD- 1
I
][
Chapter 1.
1. Introduction
Introduction and
and Review
Chapter
Review
6
6
Remark
1.5. The
factorization of
of aa matrix
into the
of aa unit
lower triangular
Remark 1.5.
The factorization
matrix A
A into
the product
product of
unit lower
triangular
matrix L
L (i.e., lower triangular with all l's
1's on the diagonal) and an
an upper triangular matrix
V
U is
is called an
an LV
LU factorization;
factorization; see,
see, for example,
example, [24].
[24]. Another
Another such
such factorization
factorization is
is VL
UL
where V
U is unit upper triangular and L is lower triangular.
triangular. The factorizations used above
are block analogues of these.
Remark
[~ BD].
~ ].
Remark 1.6. The matrix D -— e
C A –-I1 BB is called the Schur complement of A in[AC
l
D – l C is the Schur complement of
in [~
[AC B~D ].
Similarly, A -– B
BD-Ie
of D
Din
EXERCISES
EXERCISES
1. If A eE jRnxn
a is a scalar, what is det(aA)? What is det(–A)?
det(-A)?
Rnxn and or
A is orthogonal, what is det A?
A? If
A is unitary,
unitary, what is det A?
A?
2. If
If A
If A
3. Let
Letx,y
jRn. Show
Showthatdet(l-xyT)
x, y eE Rn.
that det(I – xyT) = 11 – yTx.
yTx.
4. Let U1,
VI, V2,
E jRn
xn be orthogonal matrices. Show that the product V
U2, ...
. . .,,Vk
Uk €
Rnxn
U =
=
VI
U1 V2
U2 ...
• • •V
Ukk is
is an
an orthogonal matrix.
5. Let A
A E
of A, denoted
denoted TrA,
Tr A, is defined as the sum of its diagonal
e jRNxn.
R n x n . The trace of
aii.
elements,
Eni=1 au·
elements, i.e.,
i.e., TrA
TrA =
= L~=I
linear function; i.e., if A, B eE JRn
xn and a, ft
f3 eE R,
JR, then
(a) Show that the trace is a linear
Rnxn
Tr(aA + f3B)
fiB)=
+ fiTrB.
Tr(aA
= aTrA
aTrA +
f3TrB.
(b) Show that Tr(AB)
= Tr(BA),
AB i=
BA.
Tr(Afl) =
Tr(£A), even though in general AB
^ B
A.
nxn
(c) Let S €
E R
jRnxn be skew-symmetric,
skew-symmetric, i.e., S
STT =
= -So
TrS = 0.
O. Then
-S. Show that TrS
either prove the converse or provide a counterexample.
x
6. A matrix A
A eE W
jRnxn
A22 = A.
" is said to be idempotent if A
22
/ x™
.
,
• ,
2cos<9
0
(a) Show that the matrix A
_..
A =
= --2!I [T|_ 2cos
.
2f)
sin 2^
sm
0
J. .
sin 20 1 . .d_,
..lor all
II
_sin. 20
is idempotent
for
2sin
aII #.
o.
r
2z 0
2sm2rt
# J IS I empotent
X
(b) Suppose
A eE IR"
jRn xn"isisidempotent
Suppose A
idempotentand
andAAi=^ I.I. Show
Showthat
thatAAmust
mustbe
besingular.
singular.