Solving large linear systems
with multiple right–hand sides
Julien Langou
C ERFACS – Toulouse, France.
Solving large linear systems with multiple right–hand sides – p. 1/12
Motivations
In electromagnetism design, it is necessary to be able to predict
the currents induced by an incident plane wave on a given
object.
Using the boundary element method on the Maxwell equation,
this amounts to solve the linear system
where
represents the impedance of the object ( Ohm )
represents the incident field ( Volt )
represents the current on the object ( Ampere )
Solving large linear systems with multiple right–hand sides – p. 2/12
Antenna design
Representation of the electric current due to an antenna on the Citroën C5 car.
Solving large linear systems with multiple right–hand sides – p. 3/12
Monostatic radar cross section
− Radar cross section − airbus 23676 − phi = 30o − theta = 0o:90o − vertical polarization − nofmm direct solver −
15
10
5
0
RCS
−5
−10
−15
−20
−25
−30
−35
0
10
20
30
40
50
60
70
80
90
theta
Monostatic Radar Cross Section (RCS) for the Airbus 23676.
Solving large linear systems with multiple right–hand sides – p. 4/12
Motivation
This class of problem naturally leads to
that is to say
solve a linear system with multiple right–hand sides.
Solving large linear systems with multiple right–hand sides – p. 5/12
Outline
Study of the Gram–Schmidt algorithm and its
variants
Implementation of iterative methods
The electromagnetism applications
Solving large linear systems with multiple right–hand sides – p. 6/12
Study of the Gram–Schmidt
algorithm and its variants
Solving large linear systems with multiple right–hand sides – p. 7/12
The GMRES method
The GMRES method (Saad & Schultz ’86) gives,
at each setp , the approximate solution
that
solves the least–squares problem
where
is the Krylov space
Span
Solving large linear systems with multiple right–hand sides – p. 8/12
Link with QR–factorization
To summarize the GMRES methods needs
a matrix–vector product operation (and also a
preconditioner operation)
an orthogonalization scheme
Solving large linear systems with multiple right–hand sides – p. 9/12
Gram–Schmidt algorithm
Starting from
with full
rank, the Gram–Schmidt algorithm computes
and so as
is upper triangular with positive
elements on the diagonal
Solving large linear systems with multiple right–hand sides – p. 10/12
Gram–Schmidt algorithm
Jörgen P. Gram Erhard Schmidt
for
do
end for
Solving large linear systems with multiple right–hand sides – p. 11/12
Outline of the first part
When modified Gram-Schmidt generates a
well conditioned set of vectors
Reorthogonalization issues: twice is enough
Reorthogonalization issues: about selective
reorthogonalization criterion
Reorthogonalization issues: a–posteriori
reorthogonalization algorithm in the modified
Gram–Schmidt algorithm
Solving large linear systems with multiple right–hand sides – p. 12/12
Classical/Modified
Classical Gram–Schmidt (CGS)
for
do
end for
Modified Gram–Schmidt (MGS)
for
do
end for
Solving large linear systems with multiple right–hand sides – p. 13/12
An experimental fact
Let take
is the 12–by–12 Hilbert matrix with elements
hilb
..
.
..
.
.
It is an example of a ill–conditioned matrix, we have
is far from being orthogonal.
Solving large linear systems with multiple right–hand sides – p. 14/12
Round–off errors
Computation in floating–point arithmetic implies
in general round–off error.
Standard exists to control these errors, we work
here with the IEEE 754 standard
fl
op
op
Solving large linear systems with multiple right–hand sides – p. 15/12
Björck (’67)
, with singular values :
,
where
with full rank
,
MGS computes so as
and
is the unit round–off. These results hold
under the assumptions :
Solving large linear systems with multiple right–hand sides – p. 16/12
An experimental fact
Let take is the 12–by–12 Hilbert matrix with
elements
It is an example of a ill–conditioned matrix, we
.
have
is far from being orthogonal.
However,
is well conditioned.
Solving large linear systems with multiple right–hand sides – p. 17/12
Giraud Langou (’02) – IMAJNA
Let us define the polar factor of (
)
then Schönemann (’66), Higham (’94) showed
Björck (’67) showed that
This implies that
Solving large linear systems with multiple right–hand sides – p. 18/12
Giraud Langou (’02) – IMAJNA
14
10
12
10
10
10
8
10
6
10
4
10
2
10
0
10
0
10
2
10
4
10
6
10
8
10
10
10
12
10
14
10
16
10
18
10
Solving large linear systems with multiple right–hand sides – p. 19/12
Two MGS loops
MGS
MGS
Solving large linear systems with multiple right–hand sides – p. 20/12
TMGS / MGS2
for
do
do
do
do
do
end for
end for
end for
end for
end for
end for
do
for
for
for
for
for
Solving large linear systems with multiple right–hand sides – p. 21/12
MGS2( ),
do
for
for
for
do
do
end for
if
end for
then reorthogonalize endif
end for
Solving large linear systems with multiple right–hand sides – p. 22/12
Giraud Langou Rozložník (’01)
twice is enough for CGS2 (and MGS2 as well) if
is not too ill conditioned
Sketch of the proof
For all , we have
and then by induction on
.
Solving large linear systems with multiple right–hand sides – p. 23/12
Giraud and Langou (’03) – SISC
!
$#"
!
no proof exists ...
A rather comonly used criterion (Rutishauser (’67), Gander (’77),
Ruhe(’83), Hoffmann (’89), Dax (’00))
!
# !%
Giraud–Langou (’03)
!
# !&
Abdelmaleck (’72) & Kiełbasiński (’74)
.
where
'
(
Daniel, Gragg, Kaufman and Stewart (’76)
Solving large linear systems with multiple right–hand sides – p. 24/12
Giraud Langou (’03) – SISC
+
+
*
)
matrix
*
*
*
*
MGS2( )
MGS2
*
*
*
*
CGS2( )
CGS2
Solving large linear systems with multiple right–hand sides – p. 25/12
Giraud Langou (’03) – SISC
+
+
*
)
matrix
*
*
*
*
MGS2( )
MGS2
*
*
*
*
CGS2( )
CGS2
Solving large linear systems with multiple right–hand sides – p. 26/12
Giraud Langou (’03) – SISC
+
+
*
)
matrix
*
*
*
*
MGS2( )
MGS2
*
*
*
*
CGS2( )
CGS2
Solving large linear systems with multiple right–hand sides – p. 27/12
Blindy MGS
gives
and
Solving large linear systems with multiple right–hand sides – p. 28/12
,
-
Blindy MGS
given by blindy MGS on
.
Solving large linear systems with multiple right–hand sides – p. 29/12
Giraud Gratton Langou (’03)
Singular values study.
under the assumption
, for all
,
01 2
3 .
.
/
2
43.
2
01
.
/
2
E.g.
Solving large linear systems with multiple right–hand sides – p. 30/12
Proof
Follow Björck and Paige (’92) and instead of
doing
do
Solving large linear systems with multiple right–hand sides – p. 31/12
Interpretation in term of rank.
|| A ||
2
c u K(A)
s (A)
cu
n
Singular values of
Singular values of
Solving large linear systems with multiple right–hand sides – p. 32/12
How to find
Run MGS to have
? - the rank–1 case
and
.
Form and find its singular value
decomposition such that
Compute
Form
.
and
.
:
This algorithm needs
flops!
For comparison, we
recall that a complete reorthogonalization loop costs
flops.
Solving large linear systems with multiple right–hand sides – p. 33/12
Application to iterative methods
*
# iter
5391
31
Airbus
23676
104
Sphere
40368
59
*
*
*
Cetaf
(a) cetaf
(b) airbus
Almond 104973
71
Characteristics of
(c) sphere
(d) almond
Solving large linear systems with multiple right–hand sides – p. 34/12
Application to iterative methods
a–posteriori reorth.
6
)
6
MGS2(
5
MGS
6
6
5
Cetaf
6
6
6
5
5
Airbus
6
Sphere
Residual errors –
Almond
Solving large linear systems with multiple right–hand sides – p. 35/12
Application to iterative methods
a–posteriori reorth.
8
)
6
MGS2(
7
MGS
8
6
7
Cetaf
6
6
6
7
7
Airbus
6
Sphere
Orthogonality –
Almond
Solving large linear systems with multiple right–hand sides – p. 36/12
Summary
MGS generates a well conditioned set of
vectors
twice is enough
a new selective reorthogonalization criterion
has been exhibited
counter example matrices for the
are given
–criterion
a new a–posteriori reorthogonalization
schemes is provided
Solving large linear systems with multiple right–hand sides – p. 37/12
Future work
the singular case has not been treated
(RRQR, SVD,...)
currently, we are interested in CGS
as well as the study of Gram–Schmidt
algorithm with non Euclidean scalar product
Solving large linear systems with multiple right–hand sides – p. 38/12
Implementation of iterative
methods
Solving large linear systems with multiple right–hand sides – p. 39/12
The electromagnetism
applications
Solving large linear systems with multiple right–hand sides – p. 40/12
Motivations
,
9
9
, where
9
Boundary element method in the Maxwell equations that leads to
complex, dense and huge linear systems to solve,
is accessed via matrix–vector products using the fast multipole
method, a Krylov solver is used,
Previous work:
Frobenius norm minimizer preconditioner,
flexible GMRES solver (inner GMRES preconditioner),
spectral low rank update preconditioner,
still problematic to have a complete monostatic radar cross section.
Solving large linear systems with multiple right–hand sides – p. 41/12
Outline of the third part
techniques to improve one right–hand side solvers for multiple
right–hand sides problem,
seed–GMRES algorithm: two cases, two problems and two
remedies,
linear dependencies of the right–hand sides in the
electromagnetism context.
Solving large linear systems with multiple right–hand sides – p. 42/12
Collaborators
This presentation uses results and codes provided by Guillaume Alléon
EADS, Guillaume Sylvand (INRIA(Sophia–Antipolis)–CERMICS),
Bruno Carpentieri and Émeric Martin.
Iain S. Duff (CERFACS–RAL) and Luc Giraud (CERFACS) supervise
the work at CERFACS.
Francis Collino is also working with us.
Solving large linear systems with multiple right–hand sides – p. 43/12
The single right–hand side solvers
increase the amount of work in the preconditioner in order to
increase its efficiency. In our case, for example, we can
1.a. increase the density of the Frobenius norm minimizer
preconditioner (Bruno)
1.b. add a spectral low rank update that shifts the smallest
eigenvalue close to one (Émeric)
Solving large linear systems with multiple right–hand sides – p. 44/12
The single right–hand side solvers
increase the amount of work in the preconditioner in order to
increase its efficiency
:
:
find an initial approximate solution for the current system from the
previous systems solved
2.a. An obvious strategy is to take
?
@
:
then take the initial guess
:
@
;
= <
>
:
:
2.b. Another strategy is to solve the least squares problem
Solving large linear systems with multiple right–hand sides – p. 45/12
The single right–hand side solvers
convergence of full GMRES on the Airbus 23676 (30o,30o)
0
10
without initial guess
−1
10
−2
||rn||/||b||
10
−3
10
436
−4
10
−5
10
0
100
200
300
400
iterations
500
600
700
Solving large linear systems with multiple right–hand sides – p. 46/12
The single right–hand side solvers
convergence of full GMRES on the Airbus 23676 (30o,30o)
0
10
without initial guess
with initial guess
−1
10
−2
||rn||/||b||
10
−3
10
71
436
−4
10
−5
10
0
100
200
300
400
iterations
500
600
700
Solving large linear systems with multiple right–hand sides – p. 47/12
The single right–hand side solvers
increase the amount of work in the preconditioner in order to
increase its efficiency
find an initial approximate solution for the current system from the
previous systems solved
gathered multiple GMRES iterations
Solving large linear systems with multiple right–hand sides – p. 48/12
9
9
A first test case: the almond 104793
formulation (JINA 2002)
9
The almond case
and CFIE
Solving large linear systems with multiple right–hand sides – p. 49/12
The almond case
iterations
elapse time (s)
seed–GMRES
GMRES with initial guess strategy 2
iterations and elapsed time (s) on
(Compaq–C ERFACS).
processors
Solving large linear systems with multiple right–hand sides – p. 50/12
The almond case
iterations
elapse time (s)
seed–GMRES
GMRES with initial guess strategy 2
iterations and elapsed time (s) on
(Compaq–C ERFACS).
processors
Solving large linear systems with multiple right–hand sides – p. 51/12
The almond case
14000
elapsed time (s)
number of iterations
4000
12000
8000
iterations
elapsed time (s)
10000
6000
2000
4000
2000
0
0
50
100
150
200
250
300
350
w
Elapsed time and number iterations versus the size of the restart
for restart seed–GMRES.
Solving large linear systems with multiple right–hand sides – p. 52/12
The almond case
iterations
elapse time (s)
seed–GMRES
GMRES with initial guess strategy 2
seed–GMRES (
)
iterations and elapsed time (s) on
(Compaq–C ERFACS).
processors
Solving large linear systems with multiple right–hand sides – p. 53/12
9
9
A second test case: the cobra 60695
formulation
9
The cobra case
and EFIE
Solving large linear systems with multiple right–hand sides – p. 54/12
The cobra case
strat. 2
seed–GMRES
(0.0,0.0)
1.000
1.000
(0.5,0.0)
1.000
0.230
(1.0,0.0)
0.071
0.074
(1.5,0.0)
0.048
0.024
(2.0,0.0)
0.039
0.007
(2.5,0.0)
0.037
0.004
(3.0,0.0)
0.028
0.004
(3.5,0.0)
0.028
0.004
(4.0,0.0)
0.024
0.004
(4.5,0.0)
0.022
0.004
(5.0,0.0)
0.027
0.004
(5.5,0.0)
0.022
0.004
HG
iter
F
E
/
iter
F
iter
E
E
D
B A
C
G
default guess
Solving large linear systems with multiple right–hand sides – p. 55/12
The cobra case
strat. 2
seed–GMRES
(0.0,0.0)
338
338
1.000
338
1.000
(0.5,0.0)
339
339
1.000
190
0.230
(1.0,0.0)
340
63
0.071
210
0.074
(1.5,0.0)
341
56
0.048
130
0.024
(2.0,0.0)
341
81
0.039
197
0.007
(2.5,0.0)
341
116
0.037
215
0.004
(3.0,0.0)
342
139
0.028
237
0.004
(3.5,0.0)
342
151
0.028
226
0.004
(4.0,0.0)
343
159
0.024
254
0.004
(4.5,0.0)
344
180
0.022
258
0.004
(5.0,0.0)
345
179
0.027
265
0.004
(5.5,0.0)
346
192
0.022
299
0.004
HG
iter
F
E
/
iter
F
iter
E
E
D
B A
C
G
default guess
Solving large linear systems with multiple right–hand sides – p. 56/12
The cobra case
0
10
seed GMRES
GMRES with initial guess strategy 2
−1
10
−2
10
−3
10
0
500
1000
1500
2000
2500
3000
3500
Comparison between the seed–GMRES method and the GMRES
method with initial guess strategy 2.
Solving large linear systems with multiple right–hand sides – p. 57/12
The cobra case
Seed moral in le Cid
Ô rage ! Ô désespoir ! Ô vieillesse ennemie !
N’ai-je donc tant tant vécu que pour cette infamie ?
Ô Anger! Ô despair! Ô ennemy of age!
Should I have lived so much for such infamy?
Corneile (1682), Le Cid.
Solving large linear systems with multiple right–hand sides – p. 58/12
Seed GMRES + Spectral LRU
0
10
seed GMRES
seed GMRES with spectral low rank update (15)
−1
10
−2
10
−3
10
60
61
62
63
64
65
66
right−hand sides
67
68
69
70
71
Solving large linear systems with multiple right–hand sides – p. 59/12
Seed GMRES + Spectral LRU
0
10
seed GMRES
seed GMRES with spectral low rank update (15)
−1
10
−2
10
−3
10
0
100
200
300
400
500
600
700
iterations
Solving large linear systems with multiple right–hand sides – p. 60/12
Seed GMRES + Spectral LRU
in a race starting close from the arrival
and running fast ensures to finish first!
Solving large linear systems with multiple right–hand sides – p. 61/12
Seed GMRES + Spectral LRU
in a race starting close from the arrival
and running fast ensures to finish first!
Luc Giraud ( my thesis advisor).
Solving large linear systems with multiple right–hand sides – p. 62/12
Linear dependencies RHS
sphere of 1 meter lightened by different plane waves
Airbus lightened by different plane waves
8.993775e+08 Hz
1.199170e+09 Hz
1.798755e+09 Hz
2.398340e+09 Hz
0
10
−2
−2
10
10
−4
−4
10
10
−6
−6
10
10
−8
−8
10
10
−10
−10
10
10
−12
−12
10
10
−14
−14
100
150
200
250
300
350
10
Singular values distribution of
50
100
150
200
250
9
50
9
10
2.286e+09 Hz
4.572e+09 Hz
6.858e+09 Hz
0
10
300
350
.
Solving large linear systems with multiple right–hand sides – p. 63/12
J
6
6
M
6
cm
6
cm
L
7
,
L C
8
7
cm
-
,
7
L
cm
-
cm
L ,
C
cm
-
cm
7
cm
K
7
,
7
7
5
8
,
8
cm
-M
L
6
L
L
L
6
L
8
,
L
L ,
C
cm
GHz
cm
,
7
7
5
6
GHz
cm
L
7
L ,
C
K
M
-6
K
6
7
7
L
L
GHz
GHz
L
8
K
GHz
GHz
L
,
L
-
7
L
-
7
-5
M,
,
76
GHz
GHz
L
6
5
,6
GHz
GHz
L
7
7
,
M
M
GHz
GHz
L
5-
6
L
-
M
5
L
7
,K
7
,
almond
K
almond
-5
cobra
M
cobra
7
Airbus
K
Airbus
Airbus
6
sphere
M
sphere
sphere
-
sphere
,6M
8
cetaf
/
dof
I
, the size in wavelengths
Solving large linear systems with multiple right–hand sides – p. 64/12
Harmonic spheric decomposition
N
where
is such that the errors done by
representing the right–hand sides in the basis of
.
the spherical harmonic is
This is joint work with Francis Collino.
Solving large linear systems with multiple right–hand sides – p. 65/12
P
O
J
6
6
M
-
6
-
K
7
,
L
cm
-
cm
-
7
,
7
7
-
cm
L
cm
L
,
cm
7
,
7
7
,
L
L ,
C
8
L
5
6
L
8
8
L
6
cm
L
cm
L ,
C
cm
6
5
7
L
GHz
cm
7
7
GHz
cm
-M
L
-6
K
6
7
GHz
GHz
L
GHz
cm
L
7
L ,
C
K
,
M
L
GHz
L
,
8
L
6
L
GHz
cm
, we compare with , the number of singular values
greater than
.
7
U
P
O
,
GHz
GHz
L
-
7
8
K
K
GHz
GHz
L
-
7
5M,
,
5-
7
-
T
R Q
S
almond
We take
and compute
GHz
L C
7
-5
7
6
5
6,
,
L
-
M
L
7
,K
7
5
6
M
M
almond
cobra
67
cobra
Airbus
M
Airbus
K
Airbus
sphere
6
sphere
M
sphere
-
sphere
,6M
8
cetaf
/
dof
and
I
Comparison of
Solving large linear systems with multiple right–hand sides – p. 66/12
7
6
7
6
7
,
,
5
,
6
5M
76
,
-
6
L
7
L
P
O
J
6
6
-
M
cm
-
,
7
cm
-
cm
7
cm
6
5
8
7
8
cm
,
8
L
6
L
L
,
L
6
L
cm
cm
-M
GHz
cm
K
7
7
,
7
L
L ,
C
cm
,
7
7
cm
L
,
L
-6
K
6
7
L
5
7
L
GHz
GHz
L
,
L
-
8
6
L
,
GHz
cm
, we compare with , the number of singular values
greater than
.
L
7
8
K
GHz
GHz
cm
L ,
C
7
L ,
C
K
M
GHz
L
-
7
P
U
7
-5
M,
,
-5
GHz
GHz
L
7
7
,
6
5
,6
GHz
GHz
L
-5
L
-
M
L
7
,K
7
5
6
M
M
7
-
T
R Q
S
almond
We take
and compute
K
scriptsize almond
GHz
O
cobra
76
cobra
Airbus
M
Airbus
K
Airbus
sphere
6
sphere
M
sphere
-
sphere
,6M
8
cetaf
/
dof
and
I
Comparison of
Solving large linear systems with multiple right–hand sides – p. 67/12
7
7
7
-
6
7
,
,K
M
5
KM
6
6M
,
5
6
P
O
,
,
7
7
,
67
,
5
,
,
M
6
6
-
5M
6
6
J
6
6
M
6
K
7
,
L
cm
-
cm
-
7
,
7
7
-
cm
L
cm
L
,
cm
7
,
7
7
,
L
L ,
C
8
L
5
6
L
8
8
L
6
cm
L
cm
L ,
C
cm
6
5
7
L
GHz
cm
7
7
GHz
cm
-M
L
-6
K
6
7
GHz
GHz
L
GHz
cm
L
7
L ,
C
K
,
M
L
GHz
L
,
5
6
,
,
8
L
6
L
GHz
cm
, we compare with , the number of singular values
greater than
.
7
U
P
O
,
GHz
GHz
L
-
7
8
K
K
GHz
GHz
L
-
7
5M,
,
5-
7
-
T
R Q
S
almond
We take
and compute
GHz
L C
7
-5
7
6
5
6,
,
L
-
M
L
7
,K
7
5
6
M
M
almond
cobra
67
cobra
Airbus
M
Airbus
K
Airbus
sphere
6
sphere
M
sphere
-
sphere
,6M
8
cetaf
/
dof
and
I
Comparison of
Solving large linear systems with multiple right–hand sides – p. 68/12
Linear dependency of RHS
To solve the
systems
1. we solve the system with the
W
V
with
since we can write
right–hand sides
and then update the solutions with
.
2. with block–GMRES algorithm, we minimize the right–hand sides
on the block–Krylov space constructed with .
Solving large linear systems with multiple right–hand sides – p. 69/12
On the Airbus 23676
θ
600
0
20
40
60
80
100
120
140
160
180
GMRES
GMRES with SpU
Block GMRES − SVD (49)
Block GMRES − SVD (49) − SpU
500
400
300
200
100
0
9
9
9
For each right–hand sides we plot the number of iterations (matrix–vector product) required to converge. The
.
test example is the Airbus 23676
Solving large linear systems with multiple right–hand sides – p. 70/12
Results
DOF
# RHS
# SVD
# it
avr. # it
avr. # it
SVD
classic
coated cone sphere
77604
181
15
285
1.5
20
almond
104793
361
63
532
1.4
10
cobra LRU(15)
60695
91
35
1712
18.8
185
Airbus (5e-2)
94704
181
70
1430
7.9
130
Solving large linear systems with multiple right–hand sides – p. 71/12
Summary
adapting one right hand–side solver to the multiple right–hand
sides case is worthly to be considered
seed–GMRES algorithm may greatly be improved by restart or
spectral low rank update
linear dependencies of the right–hand sides is systematic in the
electromagnetism context
SVD preprocessing and use with the block–GMRES algorithm is
the best method at that moment
Solving large linear systems with multiple right–hand sides – p. 72/12
Prospectives (1) – block/seed
set the vectors in–core to gain in the orthogonalization process
restart in block–GMRES has the same catastrophic effect (in our
case) than restart in GMRES and prevents from convergence on hard
cases, block–GMRES with restart and spectral low rank update is
therefore tested on big test cases where full block–GMRES is not
affordable anymore. Also we are thinking on how implementing
block–GMRES–DR algorithm in the code since GMRES–DR works
fine.
Solving large linear systems with multiple right–hand sides – p. 73/12
Prospectives (2) – relax
Relaxing the accuracy of the matrix–vector product operation during the convergence of
GMRES – Bouras, Frayssé and Giraud (’00) strategy
0
10
prec−1
prec−2
prec−3
−1
10
−2
10
−3
10
50
100
150
200
250
Cetaf 5391
300
350
400
Solving large linear systems with multiple right–hand sides – p. 74/12
Jury
Guillaume Alléon
Director of project,
Åke Björck
Professor, Linköping University
Iain S. Duff
Project leader CERFACS et
Luc Giraud
Senior researcher, CERFACS
Gene H. Golub
Professor, Stanford University
Gérard Meurant
Research director, CEA
Chris C. Paige
Professor, McGill University
Yousef Saad
Professor, University of Minnesota
EADS – CCR
RAL
Solving large linear systems with multiple right–hand sides – p. 75/12
Study of the Gram–Schmidt
algorithm and its variants
Solving large linear systems with multiple right–hand sides – p. 76/12
Classical/Modified
Modified Gram-Schmidt (MGS)
for
do
Classical Gram-Schmidt (CGS)
for
do
for
do
end for
end for
end for
do
for
end for
Solving large linear systems with multiple right–hand sides – p. 77/12
The Arnoldi algorithm
The Arnoldi method provides, at each step , an
orthonormal basis
for the Krylov space
that verifies
So
find
find
and set
that minimizes
that minimizes
Solving large linear systems with multiple right–hand sides – p. 78/12
Björck (’67)
, with singular values :
,
with full rank
,
MGS computes so as
where are constants depending on ,
arithmetic, and is the unit round–off.
and the details of the
Solving large linear systems with multiple right–hand sides – p. 79/12
+
and
+
Björck (’67)
These results hold under the assumptions :
and
Solving large linear systems with multiple right–hand sides – p. 80/12
Gram–Schmidt algorithm
for
for
do
do
cost:
flops
end for
end for
Solving large linear systems with multiple right–hand sides – p. 81/12
Unitary transformations
G RAM
S CHMIDT
VS
G IVENS H OUSEHOLDER
Solving large linear systems with multiple right–hand sides – p. 82/12
L
BL
L
such that for all
Idea : find a series of unitary matrix
, the –th first columns of
B T Householder & Givens
where
for
,
,
Householder reflections:
Cost :
for ,
Givens rotations:
are upper triangular.
Choice : typically we use
and
.
.
Solving large linear systems with multiple right–hand sides – p. 83/12
Advantages & drawbacks
Advantages :
possibility to exploit the structure of (e.g. Hessenberg
Givens rotations)
if only the factor is needed then the number of flops is less
than Gram–Schmidt algorithm
the factor has columns orthornormal up to machine
precision (Wilkinson, ’63)
Drawbacks :
to obtain doubles the cost of the method
the fact that unitary transformations implies a factor with
orthonormal columns highly depends on the dot product used
Solving large linear systems with multiple right–hand sides – p. 84/12
Björck and Paige (’92)
with full rank
values :
MGS computes
,
such that
, with singular
,
+
and
where
and is the unit round–off.
This results holds under the assumptions
and
Solving large linear systems with multiple right–hand sides – p. 85/12
Daniel and al. (’76)
+
such that
(
X
Let take
, we consider the serie of vectors
X
and
is defined via
,
where
Then Daniel and al. showed that
(e.g.)
.
Solving large linear systems with multiple right–hand sides – p. 86/12
+
Giraud, Langou and Rozložník (’01
,
ζ0
0
10
ζ1
−2
10
−4
ζ2
10
−6
10
ζ3
−8
10
ζ
−10
10
4
−12
10
ζ5
−14
10
−16
−14
−12
−10
10
−6
10
−4
−2
10
10
0
10
−8
10
10
10
−16
10
,–
,
,–
10
Solving large linear systems with multiple right–hand sides – p. 87/12
How to find
?
We have expressed as a sum of to low rank
matrix of rank . The choice of is a
compromise between accuracy and efficiency.
Solving large linear systems with multiple right–hand sides – p. 88/12
2D case
G OLDORAK (
,
).
Solving large linear systems with multiple right–hand sides – p. 89/12
2D case
9
We run full GMRES with MGS without preconditionner and a
right-hand side corresponding to an incident wave at .
The tolerance is fixed to
.
0
10
−2
10
−4
10
−6
Y
-
HG
10
−8
10
−10
10
−12
10
20
40
60
80
100
120
Solving large linear systems with multiple right–hand sides – p. 90/12
2D example
X
X
X
From Greenbaum, Rozložník and Strakoš (’97) we know that
0
10
−2
10
-
GH
−4
Y
10
−6
.
2
! G
2[
3
Y
B
3
Y
.
Y
-
Z !F 10
−8
10
−10
10
−12
10
20
40
60
80
100
120
Solving large linear systems with multiple right–hand sides – p. 91/12
2D example
X
so from Björck (1967) we can check that the loss of orthogonality is
proportionnal to
.
0
10
−2
−4
10
Y
-
HG
10
−6
.
2
! G
2[
3
Y
B
3
Y
.
Y
-
Z !F 10
−8
Y
[(
Y
[
-
\
10
−10
10
−12
10
−14
10
20
40
60
80
100
120
Solving large linear systems with multiple right–hand sides – p. 92/12
2D example
X
X
X
X
X
As
,
and are well-conditionned, we deduced that
the second smallest singular value of
is reasonnably big.
0
10
-
−2
Y
GH
10
−4
.
3
B
Y
Y
Y
2
! G
2[
]
.
-
ZF 10
−6
h
_
fg
edc $ba
h
f
10
lkj
a
−8
i_^
-
`_^
10
−10
10
−12
10
20
40
60
80
100
120
Solving large linear systems with multiple right–hand sides – p. 93/12
LLS problems with multiple RHS
X
with MGS
X
X
..
.
when using an orthonormal basis
..
.
Solving large linear systems with multiple right–hand sides – p. 94/12
(
T
T
T
,
7
75
n L
L
L
T
T
7
n 7F
M
M
7
n 7
F
K
7
7
L ,
C
L
T
n m
L
L
L
L
L
T
T
T
5
M
n 7
7
K
n 7
,
6
7
7
n 7
M
o
m
m
8
MGS2(
n U
K
-
8
o
L
L
T
n 7
8
5
7F
L
L
7
7
7F
8
76
n 7
F
5
L
T
n m
7
L
MGS2( )
L
L
L
L
7
7
n m
7
5
L ,
C
)
'p
'p
3 \
7
7
,
MGS2(
L
T
Almond
L
Sphere
Airbus
M
Cetaf
L
Application to iterative methods
)
.
Solving large linear systems with multiple right–hand sides – p. 95/12
Implementation of iterative
methods
Solving large linear systems with multiple right–hand sides – p. 96/12
Outline
GMRES–DR
seed–GMRES
block–GMRES
Solving large linear systems with multiple right–hand sides – p. 97/12
GMRES–DR
new
need
vectors of size . whereas if we
and
first perform the LU decomposition of
then do
new
; we only need
vectors.
Solving large linear systems with multiple right–hand sides – p. 98/12
The electromagnetism
applications
Solving large linear systems with multiple right–hand sides – p. 99/12
One right–hand side solvers
spectrum
0.8
0.6
0.4
0.2
0
−0.2
−0.4
−0.6
−0.8
−0.2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
Cetaf preconditioned with “default” Frobenius norm minimizer preconditioner
Cetaf preconditioned with “default” Frobenius norm minimizer preconditioner with a spectral rank– update.
Solving large linear systems with multiple right–hand sides – p. 100/12
assembly
application
E
One right–hand side solvers
procs
FMM
Frob
SLRU
FMM
Frob
SLRU
9.6
631.8
3.7 (20)
0.25
0.11
0.02 (20)
4
Airbus 23676
114.8
184.8
36.7 (20)
1.86
0.29
0.03 (20)
4
cobra 60695
33.3
120.2
30.6 (15)
1.93
0.22
0.03 (15)
8
almond104793
53.1
345.8
182.8 (60)
2.92
0.41
0.10 (60)
8
cetaf 5391
almond104793
53.1 345.8
59.3 (20) 2.92 0.41 0.05 (20)
8
Time of assembly and application of the FMM (prec-3), the Frobenius preconditioner and
the spectral update (the number of shifted eigenvalues is given in bracket).
Solving large linear systems with multiple right–hand sides – p. 101/12
One right–hand side solvers
600
Frob
SpU (20)
550
500
# full gmres iterations
450
400
350
300
250
200
150
0
20
40
60
80
100
120
140
160
180
illuminating angle
Airbus 23676
Solving large linear systems with multiple right–hand sides – p. 102/12
total
78
-
L
7
-
5
5
L
7
M
87
7
L
FMM
&Precond
5
7
L
5
7
Precond (s)
L
FMM (s)
M
GMRES with zero
unitary
L
)
7
gathered GMRES (
unitary
E
Gathered multiple GMRES
total
E
7
L
L ,
C
7
5
5
6
8
M
-
7
L
-M
,
,
&Precond
K
,
7
FMM
6
5
L
6
Precond (s)
FMM (s)
L
GMRES with strategy 2
unitary
L
)
.
unitary
5
K
GMRES gathered (
6,
(a) Airbus
as initial guess
for the initial guess
(b) coated cone sphere 77604.
Solving large linear systems with multiple right–hand sides – p. 103/12
F D
T
/
r
F D
. This amounts to
B
q
r
F D
/
c xw
v us t
y
We first run GMRES to solve the linear system
solving the linear least squares problem
q
The seed GMRES algorithm (1)
h
h
az
|
T
{
[
h
rF
a
F
a
F D
/
F
q
where
is the Krylov space of size built with an Arnoldi process on using the
.
starting vector
In most applications, the computational burden lies in the matrix–vector products and the
scalar products required to solve this linear least squares problem.
In seed–GMRES, the subsequent right–hand sides benefit from this work. An effective
initial guess
for the system is obtained by solving the same linear
least squares problem but with another right–hand side, namely
dot products and
c vx
y
L Dz
q
r
/
sut
3
Dz
The additional cost is
zaxpy operations per right–hand sides.
Solving large linear systems with multiple right–hand sides – p. 104/12
The seed GMRES algorithm (2)
0
10
K1
K2
−1
K3
10
K4
K5
K6
−2
10
K7
K8
K9
K10
4.5
5
−3
10
0
0.5
1
1.5
2
2.5
θ
3
3.5
4
5.5
}
}
}
Norm of the residuals after each minimization on the Krylov
spaces. The test example is the cobra 60695,
.
Solving large linear systems with multiple right–hand sides – p. 105/12
The almond case
7
A
~
6
~
8
,
L
8
~
8
~
,
,
L
-8
~
-
~
8
,
L ,
C
~
,
~
8
,
,
L
~
~
8
,
8
L
~
~
8
7
5
iter
E
L
7
A
~
8
T
/
~
Starting the seed–GMRES method from the right–hand side
, we obtain the
following sequence for the number of iterations for the first
right–hand sides:
Solving large linear systems with multiple right–hand sides – p. 106/12
The almond case
7
A
,
,
8
8
6
L
5
6
L
5
6
L
5
6
L C
5
6
L
5
6
L
5
6
7
A
A
T
/
~
L C
5
6
L
5
6
L
8
/
5
E
iter
This law describes the number of iterations of the seed–GMRES method on 15
right–hand sides starting from
. We assume that this behaviour holds for any
.
starting right–hand side
L
8
~
8
~
L
-8
~
-
~
8
L ,
C
,
~
8
,
5
~
8
L
~
~
8
L
~
~
8
7
L
7
A
~
,
,
,
8
5
E
iter
We define the transient phase by the sequence , and ; the stationnary phase is the
remaining where the number of iterations does not change much. It is well defined by a
plateau at a value equal to the average number of iterations for the right–hand sides
solved in this stationnary phase. Finally we obtain the following model law:
,
L
8
~
8
~
L
-8
~
-
~
8
L ,
C
~
,
~
8
L
~
~
8
L
~
~
8
7
L
7
A
~
8
T
/
~
Starting the seed–GMRES method from the right–hand side
, we obtain the
following sequence for the number of iterations for the first
right-hand sides:
Solving large linear systems with multiple right–hand sides – p. 107/12
K
L
L
,
K
7
7
-
7
7
7
L
L
L
L
8
7
,
7
,
7
7
7
7
7
7
7
7
7
7
7
7
7
7
7
L
L
L
L
L
L
L
L
L
L
L
L
M
,
-6
6
7
7KM
7
7
,
7K
7
7
-
-
M
-
8
7
76
7
7
L
L
L
-
6
K
7
L
M
,
5M
76
,6
7
L
L
L
M
-6
7
7
M
78
7
7
-6
7
78
7
,
76
7
7
L
L
L
L
-
,
M
5
7
-
7
7
7
K
7
7
7
6
7
7
7
5
6
7
7
7
L
L
7
76
7
7
L
L
L
7
7
-
7
7
-
M
7
7
7
L
L
L
L
L
7
7
7
7
,
7
K
57
7
-
7
7
7
7
L
L
-
-
7
7
7
7
7
L ,
C
L
L
L
K
K
78
7
7
7
7
L
6
5
FMM
M
8
M
7K
7
-
7
7
,
-
7
7
-
K
7
7
K
,
7
75
7
7
L
L
7
7
7
7
-
7
6,
precond
L
-
7
L
-
7
7
7
7
L
L
-
7
zdot
L
-
76
7
7
L
,
7
7
7
7
8-
7
7
7
,6M
zcopy
L
75
7
7
7
L
L
7
-
M
M,
5-
K
zdscal
L
76
7
7
7
7
M7
7
7
-5
,
7
-
zaxpy
L
L
almond
8
Elsapsed time for each basic operation needed by the iterative solvers.
,
8
,K
sphere
K
cobra
K
6
Airbus
76
5
sphere
M
Airbus
5-
cetaf
8
sphere
7
M
sphere
6
Airbus
Kernel operations
# proc.
4
8
Solving large linear systems with multiple right–hand sides – p. 108/12
7
7
7
7
7
7
7
7
7
7
M
-
7
7
-
7
7
L
L
L
M
7
7
K
K
8
8
K
,
-
K
6
5
L C
7
,
L ,
C
L
L
L
M
7
7
,8
5
76
KM
K
5
6
,
-
L ,
C
L
L
-
7
7
8
-5
,
M
7
-
8
-5
L ,
C
L
L
L
FMM
M
7
7
-
76
5
L ,
C
-
L
5
6
6
L ,
C
L ,
C
L
K
M
L ,
C
K
M
8
5
8
7
-
-
K
,5
7
5
L
L
,6
L
L
precond
-
7
5
K
6
7
6
7K
L
L
L
8
M
5
8
7
5
,
,
5
L
L
K
zdot
-
L
L
6
5
zcopy
L
L
8
L ,
C
-
-M
-
7
,6
zdscal
8
7
7
M
8
7
-
7
-
L
M
-
7
zaxpy
7
L
L
L
L
7
7
,6M
100
K
7
M
M
6
6
L
-M
7
-
M
-5
100
-
L
L
L
-5
L ,
C
7
-5
,
M,
100
K
K
-5
K
M
78
M
,
8
,K
7
K
100
L
L
L
L
L
almond
K
sphere
K
cobra
-
Airbus
76
6
Airbus
-5
5
sphere
M
M
sphere
7
cetaf
8
sphere
6
Airbus
Kernel operations
# proc.
4
8
8
Percentage of time required by 100 calls to each basic
operations with respect to one call to the FMM
Solving large linear systems with multiple right–hand sides – p. 109/12
The cobra case
iterations
elapsed time (s)
iter
4102
2005
2819
15565.6
6765.6
9863.7
F
E
/
E
iter
F
D
E
iter
GH
E
B A
C
seed–GMRES
G
strat. 2
default guess
Solving large linear systems with multiple right–hand sides – p. 110/12
The cobra case
rien ne sert de courir, il faut partir à point
Jean de la Fontaine.
Solving large linear systems with multiple right–hand sides – p. 111/12
Special feature RHS
†
„ $ƒ‚

$€
The –st entry of the right–hand side
corresponding to a plane
wave coming from the direction and wave number on the object
is
Solving large linear systems with multiple right–hand sides – p. 112/12
Seed–GMRES
SVD preprocess.
Linear dependency of RHS Lötstedt and Nilsson
?
:
@
;
= <
>
:
:
(’02)use initial guess strategy 2.b. :
solve the least squares problem
@
.
:
then take the initial guess
?

:
Seed GMRES

:
Solving large linear systems with multiple right–hand sides – p. 113/12
Control of the error
# SVD
15
16
17
18
19
20
21
22
# it
–
299
303
316
322
334
341
350
# verif
2.285445132223185E-003
9.430404985428857E-004
3.785604066482061E-004
1.416418780336473E-004
5.349504623890520E-005
1.892546027913748E-005
6.750489170848254E-006
2.264388221336797E-006
603
243
196
184
182
181
181
Solving large linear systems with multiple right–hand sides – p. 114/12
Cost of the SVD preprocessing
M
M
1649
,
7
-
,
,
214
67
# procs
550
,6
Elapsed time to compute the SVD for
Elapsed time GMRES
388
7
K
K
K
7
7
-K
,6M
-
7
Elapsed time SVD
M
-5
,
M
M
6
5
-
Size of the problem
right–hand sides the sphere. For comparison,
we give the average elapsed time for the solution of one right–hand side using full
to
7
GMRES (assembly of the preconditioner phase not included). The backward error is set
for the GMRES solve.
Solving large linear systems with multiple right–hand sides – p. 115/12
6
7
-
K
,
5
M,
K
7
7K
M
K
,
)
K
block–GMRES SVD(
,
)
6
seed–GMRES SVD(
6
5M
seed–GMRES
elapsed time (s)
M
K
,5
GMRES with second strategy initial guess
5
iterations
-
E
On the coated–cone–sphere
}
}
}
Four competitive methods on the coated cone sphere,
.
Solving large linear systems with multiple right–hand sides – p. 116/12
Prospectives (3/4)
Toward an adapted stopping criterion
0
0
10
10
bwd error : || A ( x − x ) || / || A x ||
n
ref
ref
rcs error : | rcs n − rcs ref(fmm) | / | rcs ref(fmm) |
rcs error : | rcs n − rcs ref(nofmm) | / | rcs ref(nofmm) |
bwd error : || A ( x − x ) || / || A x ||
n
ref
ref
rcs error : | rcs n − rcs ref(fmm) | / | rcs ref(fmm) |
−1
10
−1
10
−2
10
−2
10
−3
10
−3
10
−4
10
−4
10
50
100
150
200
iterations
Airbus 23676
250
10
20
30
iterations
40
50
60
coated cone 77604
Solving large linear systems with multiple right–hand sides – p. 117/12
Prospectives (4/4)
o
o
Cetaf 5391 − CFIE 1 − (0 ,90 ) − symmetric FROB precond.
0
10
GMRES FMM prec−3
SQMR no fmm
SQMR FMM prec−1
SQMR FMM prec−2
SQMR FMM prec−3
SQMR FMM prec−1 double
SQMR FMM prec−2 double
SQMR FMM prec−3 double
−1
10
−2
10
−3
10
−4
10
−5
10
50
100
150
200
250
300
Solving large linear systems with multiple right–hand sides – p. 118/12
GMRES–DR
0
10
full GMRES
GMRES(30)
GMRES−DR(30,20)
GMRES−DR(20,10),SLRU(10)
GMRES(10),SLRU(20)
−1
10
−2
10
−3
10
−4
10
0
50
100
150
iterations
200
250
300
Solving large linear systems with multiple right–hand sides – p. 119/12
GMRES–DR
0
10
Eigenvalue from Arpack
Harmonic Ritz value from GMRESDR(30,20)
−1
10
−2
10
−3
10
40
60
80
100
120
140
160
180
Solving large linear systems with multiple right–hand sides – p. 120/12
GMRES–DR
Eigenvalue from Arpack
Harmonic Ritz value from GMRESDR(30,20) − it 170
Path of the first Harmonic Ritz value
0.2
0.15
0.1
0.05
0
−0.05
−0.1
−0.15
−0.2
−0.25
−0.1
−0.05
0
0.05
0.1
0.15
0.2
0.25
Solving large linear systems with multiple right–hand sides – p. 121/12