Solving Systems of Linear Equations with
Matrices
Computers are especially good for this
much of High Performance Computing (HPC)
Rubin H Landau
With
Sally Haerer
Computational Physics for Undergraduates
BS Degree Program: Oregon State University
“Engaging People in Cyber Infrastructure”
Support by EPICS/NSF & OSU
Introductory Computational Science
1
© Rubin Landau
Recall 2 Weights on a String Problem
L
θ
T1
θ!
L1
θ
L3
W1
L2
⎡
⎢
⎢
⎢
⎢
⎢
⎢
[x] = ⎢
⎢
⎢
⎢
⎢
⎢
⎣
x1
x2
x3
x4
x5
x6
x7
x8
x9
⎤
T2
W2
⎡
⎥ ⎢
⎥ ⎢
⎥ ⎢
⎥ ⎢
⎥ ⎢
⎥ ⎢
⎥=⎢
⎥ ⎢
⎥ ⎢
⎥ ⎢
⎥ ⎢
⎥ ⎢
⎦ ⎣
sin θ1
sin θ2
sin θ3
cos θ1
cos θ2
cos θ3
T1
T2
T3
Introductory Computational Science
⎤
⎥
⎥
⎥
⎥
⎥
⎥
⎥,
⎥
⎥
⎥
⎥
⎥
⎦
T3
θ!
f1 (x)
f2 (x)
f3 (x)
f4 (x)
f5 (x)
f6 (x)
f7 (x)
f8 (x)
f9 (x)
2
=
=
=
=
=
=
=
=
=
3x4 + 4x5 + 4x6 − 8 = 0
3x1 + 4x2 − 4x3 = 0
x7 x1 − x8 x2 − 10 = 0
x7 x4 − x8 x5 = 0
x8 x2 + x9 x3 − 20 = 0
x8 x5 − x9 x6 = 0
x21 + x24 − 1 = 0
x22 + x25 − 1 = 0
x23 + x26 − 1 = 0
© Rubin Landau
Systems of Equations via Matrices
⎡
∂f1 /∂x1
⎢ ∂f2 /∂x1
⎢
⎢
⎣
···
···
···
..
.
···
⎤⎡
⎤
⎡
∆x1
∂f1 /∂xN
⎢
⎥
⎢
∂f2 /∂x9 ⎥
⎥ ⎢ ∆x2 ⎥
⎢
=
−
⎥ ⎢ .. ⎥
⎢
⎦⎣ . ⎦
⎣
∂f9 /∂x9
∆x9
[?]
f1
f2
..
.
f9
⎤
⎥
⎥
⎥
⎦
(1
o
[A]~x = ~b
(2
• Many physical models  simultaneous equations
• Place in matrix form, easier math (more abstract)
• More realistic models  larger matrices
• Computer = excellent tool (same steps many times)
Introductory Computational Science
3
© Rubin Landau
Scientific Subroutine Libraries
• Industrial strength, matrix subroutines
• > 10X faster than elementary methods
• Minimize roundoff error, failure
• Robust: high chance of success, broad class of problems
• Recommend: do not write your own matrix subroutines
• Also auto scales: desktop 
parallel cluster
What's the cost?
1. Must find them (not installed)
2. Must find names of all subroutines
3. May be Fortran only, C only
Introductory Computational Science
4
© Rubin Landau
Classes of Matrix Problems (Math)
1. Rules of math still apply!
2. N unknowns > N equations (unique)?
3. Equations not linearly independent?
4. N equations > N unknowns (fitting)?
5. Basic problem: system linear equations (2 masses)
[A]~x
=
~b
(1
[A]N×N × ~xN×1
=
~bN×1
(2
• [A] = known N x N matrix
• x = unknown length N vector
• b = known length N vector
5
© Rubin Landau
Solution Linear Equations
[A]~x
=
~b
(1
[A]N×N × ~xN×1
=
~bN×1
(2
[?]
• "Best" solution: Gaussian elimination
• Triangular decomposition: no [A]-1
• Slower, less robust: compute [A]-1
[A]−1 [A]~x
=
[A]−1~b
~x
=
[A]−1~b
[A]-1 (1
• Both methods in libes
6
© Rubin Landau
Classes of Matrix Problems (cont)
[A]~x = λ~x
(1
• Different matrix equation, not [A]~x = ~b
• x (vector),  (number) = unknowns RHS
• No direct solution,  for some 
• When  ?
Trivial solution
([A] − λ[I]) ~x
× ([A] − λ[I])
Nontrivial solution
−1
=
0
(3
⇒
~x = 0
(4
−1
6 ∃ ([A] − λ[I])
Secular Equation (Cramer’s Rule)
(2
det[A − λI] = 0
(5
(6
Evaluate det[ ] & Search
7
© Rubin Landau
Practical Aspects of Matrix Computing
• Scientific programming bugs: often arrays
• Even vector V[N] = “array” (1-D)
• Rules of thumb
⎡
⎢ V1
⎢
⎢ V2
⎢
⎢ ..
⎣ .
VN
⎤
⎥
⎥
⎥
⎥
⎥
⎦
• Computers are finite: size matters
• Physical dimension of 100: A[100][100][100][100] ≈ 1GB
• Processing time: ~ N3 steps for A[N][N]
• Double N  8X time
• Avoid page faults: 1 word  entire page
8
© Rubin Landau
Practical Aspects: Memory
• Matrix storage: we think blocks, computer stores linear
Java, C
Fortran
• Avoid large
“strides”
• Don't have too many indices:
•

V[L, Nre, Nspin, k, kp, Z, A]
V1[k, kp],
V2[k, kp],
V3[k, kp]
(1
(2
• Subscript 0: math must match (count from 1 or 0?)
(3
(l + 1)Pl+1 − (2l + 1)xPl + lPl−1 = 0
• Physical vs logical dimensions
• declared a[3][3], defined (') up to a[2][2]
a[1][1]’ a[1][2]' a[1][3] a[2][1]' a[2][2]’ a[2][3] a[3][1] a[3][2] a[3][3]
9
(4
© Rubin Landau
Implementation: Scientific Libraries, WWW
NETLIB
WWW metalib of free
math libraries
LAPACK
Linear Algebra pack
JLAPACK
LAPACK library in
Java
SLATEC
Comprehensive Math
& Stats
ESSL
Engr & Sci Lib (IBM)
IMSL
Intl Math & Stats
CERNLIB
European Cntr
Nuclear Res
BLAS
Basic Linear Algebra
Subs
JAMA
Java Matrix Lib
NAG
Numerical Algorithms
Group (UK)
Lapack++
Linear Algebra pack
in C++
ScaLAPACK
Distributed Memory
LAPACK
TNT
C++ Template
Numerical Toolkit
GNU GSL
Full Scientific Libe
in C & C++
Linear algebra
Matrix operations
Interpolation, fitting
Eigensystem analysis
Signal processing
Sorting and searching
Solution of linear eqns
Differential equations
Roots, zeros & extrema
Random-number ops
Statistical functions
Numerical quadrature
10
© Rubin Landau
JAMA: Java Matrix Library
• JAMA = basic linear algebra package for Java
• Works well, natural, non-expert, free
• Jampack: complex matrices
• True Matrix objects; linear algebra, aligned elements
• e.g.
[A] x = b
1
double[][] array = { {1.,2.,3}, {4.,5.,6.}, {7.,8.,10.} };
2
Matrix A = new Matrix(array);
3
Matrix b = Matrix.random(3,1);
4
Matrix x = A.solve(b);
5
Matrix Residual = A.times(x).minus(b);
6
Matrix Itest = A.inverse().times(A);
11
// Test inverse
© Rubin Landau
JamaEigen.java: Eigenvalue Problem
import Jama.*;
1
import java.io.*;
2
public class JamaEigen {
3
public static void main(String[] argv) {
4
double[][] I = { {2./3,-1./4,-1./4}, {-1./4,2./3,-1./4}, {-1./4,-
5
1./4,2./3}};
6
Matrix MatI = new Matrix(I);
// Array matrix 7
System.out.print( "Input Matrix" );
8
MatI.print (10, 5);
// Print matrix 9
EigenvalueDecomposition E = new EigenvalueDecomposition(MatI);
10
double[] lambdaRe = E.getRealEigenvalues();
//
Eigens
System.out.println("Eigenvalues: \t lambda.Re[]="+ lambdaRe[0]);
Matrix V = E.getV();
11
12
// Vectors 13
System.out.print("\n Matrix with column eigenvectors ");
14
V.print (10, 5);
15
}}
16
12
© Rubin Landau
Run Program for Output
13
© Rubin Landau
JamaFit.java: fit y(x) = b00 + b11 x + b22 x22
• Look at now, will describe math and use latter
• Let’s take a test drive before purchase
14
© Rubin Landau