Matrix
1)
2)
3
Conditions
Invertibility
Square
matrix
Columns
and
A
has
and
equal
rows
both
left
of A
and
are
right
linearly independent
inverse
,
they
are
unique
Inverse
A
Matrix
and
its
inverse
AX
=
and
OR
factorization
If
A
is
square
Or
&
and
+
Orthogonal
R
A"
both
=
(OR)"
are
related
as
I
Enverse
A
X
and
invertible
QR
matrix
R
,
=
upper
mateix
are
=
invertible
R
&
"
=
triangular
G
R OT
=
&T
Solving
linear
If
is
A
Ax
X =
Ab
equations
invertible
=
b
,
then
has
# b
a
unique
solution
Solving
Using
linear
equations
of
structure
triangular matrix
upper
j
i
=
[0
J
RilR--- Rn
....
Run
nxn
solving
the
RX
system
=
b
of
Rij
Ric
o
isj
fo
-i
=
(diagonalenteresa
equations
x =
[2]b =
[ -] (
Ril
=
This
leads
R ,, X1
to
+
equations
R1zXz +
R22X2 +
:
. . .
-
..
+
RinXn
+
RanYn
b,
=
=
ba
i
RG 1)( 1) Xn
-
-
Run
Xn
-
+
1
--
.
+
Ry k)()Xn
=
=
be
=
bu
-
1
Using
method
start
2)
back
of
Run Xn
*
->
Xn
=
by
=
Substitute
2)
equation
last
with
substitution
Run #o
last
second
in
*
RG 1)( 1) Xn
-
-
-
xn +
=
1
+
--
Evaluate
.
+
equation
Ry k)()Xn
=
=
DRM
=
-
!
bu
-
Xn
Each
1
equation
depends
an
previous solved
Variable
3)
(2ntox1)
Substituting upward
Continue
For
Xth
the
+
RKk X k
Solve
for
*
variable
in
Raj :
Y
XK
=
R y
X
upper
bk
malix
-
Rkj
=
No
solve
for
we
need
all
Xi
,
i >K
triangular
0
XK
kxj
,
Algorithm
of
back
1)
Initialize
2)
For
k
Compute
substitution
Xz
=
n
-
1
x
=
to
2
=
[
here
↑=
R
3)
Return
the
Solution
Vector
x
=
[
For
RX
=
lower
b
triangular
Rij
matrices
is
0
=
[=I
Ril
Equations
R , Xi
=
R2
+
X1
b
R22 Xz
=
bz
!
RnX1
+
Ruz X2 + ... RunYn
=
3
by
a)
start
X
b)
with
=
substitute
find
X2
Forward
X1
and
in next
Continue
equation
Substitution
Complexity
1)
2)
of
solving triangular Systems
Initialize
For th
compute
Range of "j'
starts
=
x=
at
Xz
=
I division
>
-
variable
I
XR
K+ 1
[
and
ends
at
N
>
division
I
subtraction
-
[
Total
=
if k
flop
[n ( + )+T] products
>
-
-
1
2
>
-
-
-
I
-
(k +1 + 1)
n
-
k +
n
2(n k) + 1
-
-
-
1]
k
-
additions
1 + 1 + 1
3)
Total
flops
M
&
2(n 1) +
=
flops
+
22
-
k =/
total
calculating
for
=
2n
-
2
-
2
[1
'n
all
variables
+
k=1
+ 2
...
+
n]
t
u times
[
1+ 1 +
-.
+ 1
un
J
n times
using a
Total flops
=
2n
2
-
n(n + 1) + n
N
-N
2
0(x2)
>
>
-
>
-
Input
Dimension
of
b
and
R
R (rxn)
Vector'x' of
Size
Initialize Solution
>
-
Reverse
n-1 ,
range
Initialize X[i]
b[i]
>
-
:
=
n-2
...
w
n
>
-
>
Generates
values
-
Random
vector
O and I
b
UT mateix
between
with
4
O
(4x4)
,
andI
elements
between
Solving
linear
Systems
Ax
#
A
can
be
>
-
2)
A
factorized
in
Square
->
to
two
I
Invertible
components
&
matrix
triangular
upper
A
matrix
R
Ax
b
:
=
QR
AX = b
A
Substitute
ORX
2)
Factorization
Or
b
orthogonal
>
-
Solve
=
Lia
Multiply
both
=
OR
=
in
=
b
sides
by
&T
(use
(IR
OTD
=
R)
=
1)
STORX
3)
solve
=
Ab
the
RX =Tb
->
resulting
System
a)
Compute
y
b)
solve
the
Substitution
=
following steps
Via
&Tb
system
RX
=
Y
via
back
:
Complexity
Step
1
to
Compute
:
a)
A
b)
Compute
orthogonalize
↑
=
find
Solve
rest
[a
=
,
the
92
Columns
each
<
9i's
Linear
OR
. . .
of
Column
systems
via
A
Factorization
an]
0
=
of A
OR factorization
OR
=
matrix
uxu
[9192
to
=
ParSell
find
...
[n]
columns
of O
Note
Pq(9j) (aj) 9
=
Projection of a; along
Gi
We
previously (See
:
the
calculated
have
lecture
Complexity of
4
Complexity
of
GS
algorithm
notes)
GS
Lnk
~
for
u-vectors
algorithm
For
a
linear
system
A nxn
-
&
Square
w
Complexity
'kinput
Luxu
=
In 3
-nx
.
R
Find
GTA
:
using
=TOR
Rij = aj
Complexity
A
ofR
is
QR
IR
=
for
=
(j
included
>,
R
=
i)
in
upper triangular
GS
Evaluating
part
algorithm
these
of
GS
inner
products are
algorithm
.
Step
Compute
2
be
ux1
vector
y
T
G
=
,
b
-
Orthogonal mateix
uxn
T
Yi
= Dij bj
>
-
=
with
ith
product of
inner
b
of $
vector
Complexity
~
Total
row
of each
n
inner
+
inner
( 1)
=
products
product
=2n
=
Total flops- In
u
3
Step
:
=
Back
Substitution
Complexity
u2
Complexity
of
Complexity
Solving
of Step
Complexity
+
Total
flops
=
Linear
+
1
Systems
Complexity
Via
OR
factorization
of Step 2
of step 3
243
+
W
dominantStep
2x3
2n2 + 42 Y
two
Compare
>
-
>
-
upl. Solve
upl
upl
Solve
.
-
inv
upl inv
methods
:
upl Solve
.
T (x
=
1000) o
faster
than
32 8 ms
.
upl.inv
T(n
=
1000)
c
=
=
32 8
.
ms
=
3 28 x108 ms
.
c
.
u3
C =
execution time
per flop
T(n 2000)
=
T(x 2000)
=
=
N
c
.
271 2
.
n3
ms
=
3
.
28x158x(2000)
ms
Factor-Solve
Solve
methods
multiple
A
for
system
X1
linear
of
equations
linear
equations
b
,
=
X > X2
AX2
-..
XI
>
-
-Vector
bz
=
by
i
AXx
,
=
bk
,
be
,
...
bea
n-vector
Factor-Solve
Solve
#
2)
2)
AXi
A
is
invertible
A
is
factorized
all
K
.
Evaluate
back
for
methods
bi
=
via
(complexity
Yi
=
OTbi
for
substitution
#
Complexity
Y:
#
Complexity
RXi
for
=
linear
equations
i
OR
~
=
1, 2
,
factorization
...
K
only
for
once
2n3)
i = 1, 2
,
all
each
RXi = Tbi
solve
and
i
QT bi
-
22
...
via
k
In
for each
i
Complexity
for
evaluating
Total
Complexity
If
kn
Then
Complexity
=
2n
Y : (i
=
1
,
2
,
..
k) 242k
RXi =Tbi
Complexity for solving
3
all
+
202k
+ wik
=
2 + 3rk
Computational
I
2n3
-
u2k
cost
is
similar
to
solving
2
of
linear
set
Equations
↑
x
=
[x I
X =
XII
- - - -
X1K
X2k
Xa ...
Xu
-
Xnk
↑
↑
XI
XK
I
nXk
[]Jax
Ik
=
b
Anxn) Xqx
=
B(xk)
>
-
A (2000 x 2000)
>
-
>
-
Row form
two
b,
and b2
>
(2xn)
-
stacks
the two
arrays
row
B T
.
>
-
[x2)
Column form
X- (nx2)
form
in
form
Comparable
timings
.
of equations
to
solve
I
set
of equations
or
2
sets
Solving
for
of
coefficients
Vander monde
Cubic
polynomial
a
matrix
Polynomial
2)
P(x)
Find
P (11)
Find
-
G
+
that
b,
,
C
=
coefficients
=
and
C
Gx +
,
P(0 4)
G
(x2 + (4x3
polynomial
satisfy
.
,
Cubic
,
C
=
?
be
,
P(02)
=
by
,
ploo
=
b4
This
1)
equivalent
is
C
a
2)
3)
C
4)
C
G(
+
=
1)
+
following
to
((y 1) + (( 1 1)3
-
-
+
2( 0 4) + G( 0 4)
+
G(0 2)
+
G(0 8)
+
+
.
-
.
.
AC
=
equations
b
+
((0 2) +
.
.
((
-
0
.
=
4)3
((0 2)3
.
b(0 8) + G(0 8)3
.
.
b
=
=
=
b
by
34
A
[i
=
-
-
!
c
X
=
n
=
=
[
=
1
1
0
0
.
1
4
(
0
2
(0 2)
-
.
-
=
,
0
.
4)2
2
,
0
.
2
,
(
0
0
-
(0
.
1)
-
.
8)
I
3
3
4)
.
.
4
1
.
(0 2)
(0 8)
0 8
1
.
(
-
.
.
.
1)
2
C
I
.
.
3
3
8]
3
[]
,
b
=
Vander monde
matrix
V(X
,
n +
1)
Solution
C
-
#
Once
can
be
"C
New
#
points
Construct
known
is
evaluated
P(x)
=
where
at
C
+
V(y
=
,
,
any
Gx +
want
we
P(y)
A- b
=
n+
to
then
-X
the
via
(x2 + (4x3
evaluate
13x4
V(y
,
n+
polynomial
the
here
1)3x44x1
Y1 Y2 Y
polynomial um
,
n =3
,
Vander monde
Special
from
type
matrix
matrix
of
the
of
powers
,
u+
)
=
each
now
[ , xm I
X ,
X
&
x-xz
!
is
X-values.
corresponding
I
V(x
where
=
-
Xi
.
xi
. . .
!
--
.
xi
derived
P(y)
=
V(y
,
n+
1)3x4
*
4x
P(y)
Y
Pseudo Inverse
(Non-square
Suppose
a
A
Gram
It's
is
matrix
G
al
·
AT A
=
an
n
[Ana I
a All
at A12
T
matrix
mxn
is
matrices)
ata ata
A21
-
:
mu
uxm
AT
A
ma
[
,
T
aa ,
...
↓
Tan
-- an
!
ana , an a- anan
-.
G
nxn
statement
Proof
:
X a,
In
+
linearly independent
has
Gram
its
if
only
Amyn
:
matrix
G
is
Linearly
columns
of
X292
+
Xn9n
=
X , Xz
matrix
Ax
=
=
representation
0
iff
X
=
,
0
0
this
can
where
if and
invertible
independent
+....
columns
be
x =
A
,
implies
-..
Xn
written
[
=
0
as
We
#
prove this
has
Assume
A
can
#
only
solution
of
both
sides
multiply
>
-
(ATA)X
=>
This
of
:
G
=
A
=
ATA
,
independent
Ax
AT (AX)
0
is
AT
(left)
=
by
ATAX
implies
also
matrix
linearly
=
=
ATA
has
is
invertible.
columns
ATo
0
linearly
X =0
=
which
0
implies
independent
Gram
columns
Counter
Statement
:
must
be
Columns
Proof :
-
If
AX
=
has
is
then
invertible
its
independent
.
if
0
[linearindependenc ea3
X= 0
a
(left)
=
11 AXIK
=>
A
=
XT
XT(ATA)X
=>
linearly
(ATA) X
multiply
ATA
=
(xTAT) (AX)
0
=>
0
Recall
11 x 11
=
0
0
=
X
=
0
(positive definiteness)
0
linearly
=
independent
columns
Pseudo Inverse of Tall matrices
Recall :
If
A
linearly independent Columns (91
has
mxn
9,
A
[
All
=
A2l
An
92
A12
--
Ain
Azu
Azz
.
Amz
Ami
.
"
,
92 ,
-- an)
Independence-dimension
&
relation
-
Amn
No .
of
Vectors
linearly independent
1
Size
of the
Vector
=>
:
it
This
has
should
left
be
inverse .
a
tall
n(n
or
square matrix
and
Consider
tall
A
G
=>
e
=
Pseudo
A
linearly independent
inverse
Dim
Dim
of
At
is
At
A
(ATA)"
AT
DimAt
.
columns
linearly independent
ATA
Dimension
%
(m > n)
wide
=
=
=
=
Columns
"AT
(ATA)
rxu
[ATA
[A
nxm
(nxn)(nxm)
(nxm)
(Invertible)
=
&nXm
A is tall
(m > n)
is
is
nxm
square
uxn]
mxn]
[ AT
same
]
as
Is
At
left
or
right
inverse
of
A
?
Dim
A
Dim
#A
.
Check :
AT
=
=
3
nxm
mxn
(A) ATA
=
At
08
At
the
is
Pseudo
inverse
using
ADA)
=
left
inverse
should
==
inverse
left
is
It
=>
be
of
=
inverse
(AB)"
.
E
A
when
AEA
property
inverse
=
=
A
A
i E
Square.
is
=
A
Pseudo
inverses
for
matrices .
wide
Recall :
-
If
Amen
has
linearly
independent
rows
dependence-dimension
T
[A
>
relation
-
A
A
=
=
[
-
No .
of
Vectors
linearly independent
1
Size
of the
Vector
=>
I 2n
C Wide
or
Square)
A
(m (n)
wide matrix
-
A
If
At
has
has
linearly independent
linearly independent
Gram matrix
of AT
also
has
Pseudo
Inverse
linearly
G
=
columns
(AT) AT
independent
of A (wide)
AT
=
then
rows
=
AAT
columns.
"
AT (AAT)
and it
At
of
Dimension
Dim
AAT
Dim
At
=
=
(mxn)(nxm)
Dim
Since
Amynis
0
=
A
is
mym
wide)
=>
Dim
(AT)
=
mxm
nxm
=>
.
(When
Atexm
Pin
[ I
of
At
=
wide
is
(nx m) (mxm)
(m < n)
tall
=
nXm
At
ALT
Check
the
is
:
=
A
When
right
inverse
AT (AAT)"
A
(AAT)"
is
reduces
inverse
a
A
(AAT) (AAT)"
square
=
IA"
um
of
a
square
=
I
matrix
At (AT)"A "
=
~
it
=
of
matrix
=
At
Moore
1)
A
is
-
Penrose
tall
pseudo
(m n)
>
inverse
A
(AA)
=
wide
dim
At
=
AT
-
2)
A
is
wide
(n(n)
== A (AAT)
tall
dim At
=
AT
1
Pseudo
Left
invertible
Let
A
factorization
matrix
linearly
has
A
OR
Via
inverses
=
independent
Columns
& - orthogonal
QR
G
A
=
(ORJOR
=
RT T Or
R
is
-
and
=>
ATA
=
RTR
(tall)
%
=
QT
upper
invertible
triangular
:
At
(ATA)"AT
=
At
=>
=
"
=
(RTR)" (OR)
T
R (RT)
RIOT
*
=
R
&T
for
tall
u
:
At
Left
=
R
*
&T
inverse
A
# Steps
At
=
How
#
compute
to
R &T
to
we
solved
#
solve
have
solving
multiple
pseudo
RAT
=>
for
At
previously
,
=
solved
for
At
b's
(Coloumns
is
&T
RAT
'X'given
for
At (A
inverse
this
-
=
AX
is
S
OT
=
b
is
tall)
a
linear
dim R
=
nxn
=
nxne
dimat
dim &T
=
nxm
where
we
b
equivalent
of
to
GT)
system
solving for
3
Solving
for
Inverse
Left
R
I R
RII
:
Rzz
Cana
...
an]
=
29.
Run
nXn
9.
,
91
92 .
,
92
#Solve
...
An e
...
for
are
columns
of
In t
are
columns
of
each
Column
Via
back
At
OT
substitution .
>
-
A3x2 (tall)
At => (2x3)
Pseudo
Right
A
inverses
Via
invertible
matrix
has
pseudo
linearly
inverse
At
=>
A
+
R
OR
factorization
independent
can
be
-
=
=
rows
found
(wide)
using
T
GR
&
>
-
solve
this
linear
equations for
system of
At