Linear Algebra
• Vectors, Matrices, Determinants
• Span, Linear Independence, Basis
• Linear equations, LU-factorization
• Eigenvalues
• Norms, Scalar Products
Projection, Least Squares
The projection of a vector v on a subspace W is the vector p ∈ W
such that kp − vk2 becomes minimal.
In other words, if
v =p+z
with p ∈ W and z ⊥ W
we want that kzk2 is minimal.
If B = (b1 , . . . , bn ) is a basis of V , we can express p as a linear
combination of elements in B and get:
p=
n
X
i=1
x i bi = x 1 b1 + x 2 b2 + · · · + x n bn .
Substituting yields:
v=
n
X
!
x i bi
+ z.
i=1
Now we form scalar products with all vectors in B: The product
with bj yields
!
!
n
n
X
X
hbj , vi = hbj ,
xi bi + zi =
xi hbj , bi i + hbj , zi
i=1
i=1
The condition z ⊥ W , gives hbj , zi = 0 and thus:
!
n
X
hbj , vi =
xi hbj , bi i = x1 hbj , b1 i+x2 hbj , b2 i+· · ·+xn hbj , bn i.
i=1
In matrix form:

hb1 , b1 i hb1 , b2 i · · · hb1 , bn i

 hb , b i hb , b i · · · hb , b i
2
2
2
n
 2 1

..
..
..
..

.
.
.
.

hbn , b1 i hbn , b2 i · · · hbn , bn i
 
 
 
 
·
 
 
x1

hb1 , vi

x2
..
.
 
  hb , vi
  2
=
..
 
.
 
hbn , vi






xn

Since the bi and v are given, we can evaluate all scalar products and
solve for the xi .
Special Case: Column spaces
The scalar product can be written as v1 T v2 , thus the equation
simplifies to

bT1 b1 bT1 b2 · · · bT1 bn
 

 bT b bT b · · · bT b
 2 1
2 2
2 n
 .
..
..
..
 ..
.
.
.

bTn b1 bTn b2 · · · bTn bn
 
 
 
·
 
 
x1

x2
..
.
 
  bT v
  2
= .
  ..
 
bTn v
xn

bT1 v




.


If we write the elements of B as columns in a matrix B, we can write
this in the form

T
T
(B · B) · x = B v
with



x=


x1

x2
..
.



,


xn
If B is linear independent (and thus a basis of its span), the matrix
B T · B is invertible, thus we can solve for x and obtain:
p = B(B T B)−1 B T · v.
We call P = B(B T B)−1 B T the projection matrix for orthogonal
projection onto W . The orthogonal projection of v onto the column
space of B is P · v.
Applications
Solving a system of linear equations Ax = b can be interpreted as
expressing b in terms of the columns of A. If an overdetermined
system has no solution due to measurement/rounding errors, the
projection is often the “best” solution.
When searching a polynomial p(x) to approximate experimental
results (xi , yi ) the approximation rule is usually that
X
(yi − p(xi ))2 is minimal (least squares)
i
Consider a basis 1, x, x2 , x3 , . . . of the space of polynomials and
corresponding column vectors of the form bi = (xe1 , xe2 , . . . , xen )T . The
least squares problem then means: Project (y1 , y2 , . . . , yn ) on this
subspace.
Matrix norm
If k · k is a vector norm on Rn , and A a matrix, we define the matrix
norm of A as
kAxk
kAk = max
= max kAxk.
x6=0 kxk
kxk=1
One can show:
kAk∞ = max
1≤i≤n
n
X
|ai,j |,
j=1
(maximum row sum, column sum).
kAk1 = max
1≤i≤n
n
X
j=1
|aj,i |
Suppose A is nonsingular and we solve Ax = b. We obtain (because
of rounding errors) the solution x̂ with Ax̂ = b̂.
Then x − x̂ = A−1 (b − b̂) and thus
kx − x̂k ≤ kA−1 k · kb − b̂k
The relative error fulfills (kbk ≤ kAk · kxk):
kx − x̂k
kb − b̂k
−1
≤ kAk · kA k ·
kxk
kbk
kb − b̂k
= κ(A) ·
kbk
with κ(A) := kAk · kA−1 k the condition number of A.
Triangular systems
We want to find a numerically reliable way to solve a system of
equations.
Consider a system of the form:
a1,1 x1
= b1
a2,1 x1 + a2,2 x2
= b2
a3,1 x1 + a3,2 x2 + a3,3 x3
..
.
= b3
an,1 x1 + an,2 x2 + · · · + an,m xm = bn
We can solve this system by forward substitution: Solve for x1 ,
substitute x1 in 2nd equation, solve for x2 and so on.
Similarly we use back substitution to solve a system of the form
a1,1 x1 + a1,2 x2 + · · · + a1,m xm = b1
a2,2 x2 + · · · + a2,m xm = b2
..
.
an−1,1 x1 + an−1,2 x2 = bn−1
an,1 x1 = bn
Now suppose we want to solve a general system of the form Ax = b
and that we can factor A = L · U with L lower triangular and U upper
triangular.
Then we have to solve LU x = b, which we solve (setting U x = y) first
as Ly = b and then U x = y.
This approach requires fewer operations than solving directly by
diagonalizing A.
Gaussian Algorithm
To find L and U , transform A into upper triagonal form (this will
become the U ) while simultaneously doing inverse transforms to an
identity matrix (this matrix will become L:
Transform U
Transform L
Add λ· row i
Subtract λ column j
to row j
from column i
Multiply row i by λ
Multiply column i by
Swap row i and j
Swap column i and j
1
λ
If no swaps are done, and we transform A systematically (normalize
leading entry, then clear column, iterate) the transformations on L
amount only to changing single entries in the identity matrix.
Stability Problem
Solve (for small )


1
1 1


·x=
1
0


We factorize (in floating point, rounding the red 1 away)


 
 
1
1
1 0


=
·
0 (1)−1/
1 1
1/ 1

Forward solving Ly = b yields y = 
1
−1/

.

Backward solving U x = y yields x = 
0
1


which is badly wrong!
The problem is that we cleared out a coulumn with and thus had to
multiply by the enormous factor 1/.
It would have been better to swap the rows and clean out with 1.
Pivoting
At the k-th step of the Gaussian Algorithm (cleaning out column k) we
permit to reorder rows k to n to bring the (absolute) largest element in
position (k, k).
We then clean out with this element, avoiding the need to multiply by
a large number.
(This is called partial pivoting. It is also possible to interchange columns
as well, this is called full pivoting. Full pivoting is numerically more
stable, but comes at the cost of many more comparisons.)
We keep track of the row swaps in the rows of a permutation matrix
P , and do the same swap in the rows of L in the “previous” columns
and get the factorization
P A = LU or A = P T LU
For example
Let

1 2
3




A= 4 5 6 

7 8 10
Swap rows 1 and 3, subtract 4/7 times row 1 from row 2, −1/7
times row 1 from row 3 gives


L=

1 0 0
4
7
1
7


1 0 
,
0 1

7 8 10

U =
 0
0
3
7
6
7
2
7
11
7


,


0 0 1



.
P =
0
1
0


1 0 0
Now swap row 2 and 3 gives


L=

1 0 0
1
7
4
7


1 0 
,
0 1

7 8 10

U =
 0
0
6
7
3
7
11
7
2
7



,

0 0 1




P =
1
0
0


0 1 0
and finally subtracting 1/2 times row 2 from row 3 gives:


L=

1
1
7
4
7
0 0


1 0 
,
1/2 1
with P A = LU .

7 8
10



6
11
,

U = 0 7
7 
0 0 −1/2

0 0 1




P = 1 0 0 

0 1 0
We now can solve Ax = b as:
1. Factorize A = P T LU .
2. Let b̂ = P b.
3. Solve Ly = b̂.
4. Solve U x = y.
BLAS and LAPack
Scientists have developed a large library of numerical routines for linear
algebra. These routines comprise the LAPack package that can be
obtained from the www.netlib.org software depository on the
web.
The LAPack routines build on (lower level) routines for basic linear
algebra, these are denoted by BLAS (basic linear algebra subroutines).
These routines comprise the time-critical parts of a calculation and
should be provided in a machine-specific version (using particularities
of the architecture – for example parallelization). This way the LAPack
code itself remains highly portable without losing on performance.
LAPack comes with a default set of BLAS routines, typically a
compiler or hardware manufacturer will provide optimized
replacement routines.
Arrays
Since LAPack stems from old Fortran times, its conventions are
Fortran-like: Arrays (and thus matrices) are stored column by column
in memory. In contrast, C by default stores arrays row by row.
To use the LAPack routines, from C, matrices have to be
stored in “transposed” form.
We also need interface code that provides a C interface to the Fortran
library routines. For SUN workstations thsi is provided by the “SUN
performance library”.
Function Names
Each LAPack routine has a 6-letter name of the form XYYZZZ with
X indicating the data type (see the link to the LAPack documentation
on the links web page):
s single
d double
c complex
z double complex
YY indicates the type of matrix. ge means general, there are many
more for banded, symmetric or hermitian matrices.
ZZZ determines the actual task performed.
trf factorize
trs use a factorization for substitution
con estimate condition number κ(A).
tri use factorization to compute inverse
sv simple driver that solves system of equations
svx expert driver (also checks condition number, computes error
bounds, requires more time and storage)
Example
We want to solve a system of equations with the double precision
general simple driver – dgesv.
The manual page for this (man dgesv or
man -M /opt/SUNWspro/SC5.0/man 3p dgesv) tells us:
#include <sunperf.h>
void dgesv(int n, int nrhs, double *da, int lda,
int *ipivot, double *db,int ldb, int *info);
with
n Number of equations.
nrhs Number of right hand sides (here 1).
a double precision array for matrix A, entries in adjacent columns are
with a distance of lda.
b double precision array for right hand side array b, stride ldb
info integer variable to hold result code.
This gives us the following function:
#include <sunperf.h>
#define MAX 10
int main(){
int n;
int nrhs =1;
double a[MAX][MAX]; double b[1][MAX];
int lda = MAX; int ldb = MAX; int ipiv[MAX];
int info,i,j;
n=3; /* nr
a[0][0]=1;
a[0][1]=4;
a[0][2]=7;
of equations */
a[1][0]=2; a[2][0]=3;
a[1][1]=5; a[2][1]=6;
a[1][2]=8; a[2][2]=10;
b[0][0]=3; b[0][1]=6; b[0][2]=0;
dgesv(n, nrhs, &a[0][0], lda, ipiv,
&b[0][0], ldb, &info);
if (info == 0) {
for(i = 0; i < n; i++)
printf("b[%d]=%f\n",i,b[0][i]);
} else
printf("dgesv returned error %d\n",info);
return info;
}
To compile, we have to link in the sun math/performance libraries and
Fortran link support:
cc solv.c -lsunperf -lF77 -lsunmat
Note: On other architectures/compilers a slightly different interface to
Lapack routines might be provided. See the respective documentation.