ETNA
Electronic Transactions on Numerical Analysis.
Volume 33, pp. 207-220, 2009.
Copyright  2009, Kent State University.
ISSN 1068-9613.
Kent State University
http://etna.math.kent.edu
ALGEBRAIC PROPERTIES OF THE BLOCK GMRES
AND BLOCK ARNOLDI METHODS∗
L. ELBOUYAHYAOUI†, A. MESSAOUDI‡, AND H. SADOK§
Abstract. The solution of linear systems of equations with several right-hand sides is considered. Approximate
solutions are conveniently computed by block GMRES methods. We describe and study three variants of block
GMRES. These methods are based on three implementations of the block Arnoldi method, which differ in their
choice of inner product.
Key words. block method, GMRES method, Arnoldi method, matrix polynomial, multiple right-hand sides,
block Krylov subspace, Schur complement, characteristic polynomial.
AMS subject classifications. 65F10.
1. Introduction. Many problems in science and engineering require the solution of
large linear systems of equations with multiple right-hand sides
AX = B,
A ∈ Cn×n ,
B ∈ Cn×s ,
X ∈ Cn×s ,
1 ≤ s ≪ n.
(1.1)
Instead of applying a standard iterative method to the solution of each one of the linear systems of equations
Ax(i) = b(i)
for
i = 1, . . . , s
(1.2)
independently, it is often more efficient to apply a block method to (1.1). The first block
method, a block conjugate gradient method, was introduced by O’Leary [14] for the solution
of a linear system of equations with multiple right-hand sides (1.1) and a symmetric positive
definite matrix. For systems (1.1) with a nonsymmetric matrix, a block version of GMRES
was introduced in [29] and studied in [26, 27]. This method is based on a block version of
the standard Arnoldi process [1]; see, for example, [17, 28].
The purpose of the present paper is to compare three variants of GMRES for multiple
right-hand sides, including the block GMRES method considered in [26, 27, 29]. These
schemes are based on block Arnoldi-type methods and differ in the choice of inner product.
We provide a unified description of the methods discussed, and derive new expressions and
bounds for the residual errors.
The paper is organized as follows. Section 2 defines block GMRES iterates with the aid
of Schur complements, and presents a connection with matrix-valued polynomials. We use
these polynomials to derive some new relations in Section 3. A few examples that illustrate
the theory are provided in Section 4.
We conclude this section by introducing notation used in the remainder of this paper. We
first recall the definition of the Schur complement. [24].
D EFINITION 1.1. Let M be a matrix partitioned into four blocks
C D
M=
,
E F
∗ Received April 27, 2009. Accepted August 22, 2009. Published online on December 16, 2009. Recommended
by Lothar Reichel.
† Département de Mathématiques, Faculté des Sciences et Techniques, Université de Mohammadia, Mohammedia, Morocco. lakhdar.elbouyahyaoui@gmail.com.
‡ Département
de
Mathématiques,
Ecole
Normale
Supérieure,
Rabat,
Morocco.
abderrahim.messaoudi@gmail.com.
§ L.M.P.A, Université du Littoral, 50 rue F. Buisson BP699, F-62228 Calais Cedex, France.
sadok@lmpa.univ-littoral.fr.
207
ETNA
Kent State University
http://etna.math.kent.edu
208
L. ELBOUYAHYAOUI, A. MESSAOUDI, AND H. SADOK
where the submatrix F is assumed to be square and nonsingular. The Schur complement of
F in M , denoted by (M/F ), is defined by
(M/F ) = C − DF −1 E.
Throughout this paper, I and Is denote identity matrices and ek their kth column. For two
matrices Y and Z in Cn×s , we define the inner product hY, ZiF = trace(Y H Z), (where Y H
denotes the conjugate transpose of Y ). The associated norm is the Frobenius norm k · kF .
The 2-norm of a matrix X ∈ Cn×s is denoted by kXk2. The Kronecker product of the
matrices C = [ci,j ] and D is given by C ⊗ D = [ci,j D]. If X is an n × s matrix, x = vec(X)
is the ns vector obtained by stacking the s columns of the matrix X.
Finally, the roots of a matrix-valued polynomial P, which is a square matrix whose entries
are ordinary polynomials, are defined to be the roots of the ordinary polynomial det(P(t)).
2. Block minimal residual-type methods. The nonsingular linear system with multiple
right-hand sides (1.1) can be solved by Krylov subspace methods in two distinct ways. The
first approach is to apply classical GMRES [16] for linear systems of equations with singlevector right-hand sides to the s linear systems separately. The second approach is to treat all
the right-hand sides simultaneously.
Before investigating the two approaches, we need some notation. Let the initial approxi(1)
(2)
(s)
(1)
(s)
mation of the solution of (1.1) be X0 = [x0 , x0 , . . . , x0 ], and let R0 = [r0 , . . . , r0 ] =
(i)
(i)
B − AX0 be the corresponding residual, with r0 = b(i) − A x0 and B = [b(1) , . . . , b(s) ].
In what follows, we let Kk denote the block Krylov matrix
Kk = [R0 , AR0 , . . . , Ak−1 R0 ],
and Ki,k the Krylov matrix
(i)
(i)
(i)
Ki,k = [r0 , A r0 . . . , Ak−1 r0 ]
for i = 1, . . . , s. We also introduce the matrix
Wk = AKk .
2.1. Standard GMRES applied to systems with multiple right-hand sides. In this
section we apply the standard GMRES method to each one of the s linear systems of equations
(i)
(1.2). Define the ith classical Krylov subspace Kk (A, r0 ) by
(i)
(i)
(i)
(i)
Kk (A, r0 ) = span{r0 , A r0 , . . . Ak−1 r0 } ⊂ Cn .
(2.1)
(i)
It is well-known that the kth approximation xk,S of GMRES applied to the ith linear system
(1.2) satisfies
(i)
(i)
(i)
(i)
(i)
xk,S − x0 ∈ Kk (A, r0 ) and (Aj r0 )H rk,S = 0 for j = 1, . . . k,
(i)
(i)
(i)
(2.2)
where rk,S = b(i) − A xk,S . It follows that the residual vector rk,S can be written as a linear
(i)
combination of the vectors Aj r0 , j = 0, 1, . . . , k, i.e.,
(i)
(i)
(i)
rk,S = pk,S (A)r0 ,
ETNA
Kent State University
http://etna.math.kent.edu
ALGEBRAIC PROPERTIES OF THE BLOCK GMRES AND BLOCK ARNOLDI METHODS
209
where
det
(i)
pk,S (t)
=
"
1
H H (i)
Ki,k
A r0
t
...
H H
Ki,k
A AKi,k
tk
#!
H AH AK )
det(Ki,k
i,k
.
The residual error for each one of the s linear systems satisfies
(i)
krk,S k22 =
H
eT1 (Ki,k+1
1
,
Ki,k+1 )−1 e1
i = 1, . . . , s;
(2.3)
see [21, 22]. Therefore, the Frobenius norm of the residual
(1)
(2)
(1)
(1)
(s)
(s)
Rk,S = [rk,S , . . . , rk,S ] = [pk,S (A)r0 , . . . , pk,S (A)r0 ]
(2.4)
can be written as
kRk,S k2F =
s
X
i=1
1
.
H
eT1 (Ki,k+1
Ki,k+1 )−1 e1
(2.5)
Similarly to the situation for standard GMRES, the residual Rk,S can be expressed in terms
of a polynomial in A. We deduce from (2.4) that
vec(Rk,S ) = PG
k,S (A) vec(R0 ),
(1)
(s)
where PG
k,S (t) = diag(pk,S (t), . . . , pk,S (t)).
(2.6)
2.2. The global GMRES method. Instead of using standard GMRES to solve each
linear system (1.2) separately, we may apply GMRES to a block diagonal matrix. The s linear
systems (1.2) can be rewritten in a compact form as (A ⊗ Is )x = vec(B), with x = vec(X).
This gives the following linear system with a single right-hand side

 (1) 

b
A



. 
.
..
(2.7)
 x =  ..  .

A
b(s)
Application of standard GMRES to (2.7) yields the global GMRES method, which also can
be defined as follows. Let
k−1
KG
U } ⊂ Cn×s
k (A, U ) = span{U, A U, . . . , A
denote the matrix Krylov subspace spanned by the matrices U, AU, . . . , Ak−1 U , where U is
an n × s matrix. Note that Z ∈ KG
k (A, U ) implies that
Z=
k
X
αj Aj−1 U ,
j=1
αj ∈ C,
j = 1, . . . , k.
At step k, the global GMRES method constructs the approximation Xk,G , which satisfies
the relations
j
Xk,G − X0 ∈ KG
k (A, R0 ) and hA R0 , Rk,G iF = 0,
j = 1, . . . , k.
The residual Rk,G = B − AXk,G satisfies the minimization property
kRk,G kF =
min
kR0 − AZkF .
Z ∈ KG
k (A, R0 )
(2.8)
ETNA
Kent State University
http://etna.math.kent.edu
210
L. ELBOUYAHYAOUI, A. MESSAOUDI, AND H. SADOK
The problem (2.8) is solved by applying the global Arnoldi process [8].
Global GMRES is a generalization of the global MR method proposed by Saad for approximating the inverse of a matrix [18, p. 300]. The global method also is effective, compared to block Krylov subspace methods, when applied to the solution of large and sparse
Lyapunov and Sylvester matrix equations with right-hand sides of low rank; see [9, 10, 23].
Applications of the global Arnoldi method in control theory, model reduction, and quadratic
matrix equations are given in [4–6, 30].
It is convenient to introduce the matrix product ⋄. Let Y = [Y1 , Y2 , . . . , Yp ] and
Z = [Z1 , Z2 , . . . , Zl ] be matrices of dimension n × ps and n × ls, respectively, where Yi and
Zj (i = 1, . . . , p; j = 1, . . . , l) are n × s matrices. Then ⋄ is defined by


hY1 , Z1 iF hY1 , Z2 iF . . . hY1 , Zl iF
hY2 , Z1 iF hY2 , Z2 iF . . . hY2 , Zl iF 


YH ⋄Z =
 ⊂ Cp×l .
..
..
..
..


.
.
.
.
hYp , Z1 iF
hYp , Z2 iF
. . . hYp , Zl iF
T HEOREM 2.1 ([3]). Let the matrix (AKk )H ⋄ (AKk ) be nonsingular. Then
−1
Rk,G = R0 − AKk (AKk )H ⋄ (AKk )
((AKk )H ⋄ R0 )
= PG
k,G (A)R0 ,
where
det
PG
k,G (t) =
1
t
...
(AKk )H ⋄ R0
(AKk )H ⋄ (AKk )
det((AKk )H ⋄ (AKk ))
tk
.
(2.9)
Moreover,
kRk,G k2F =
eT1
1
1
.
= T Ps
H
−1
⋄ Kk+1 ) e1
e1 ( i=1 Ki,k+1 Ki,k+1 )−1 e1
i
h
(1)
(s)
G
or equivalently
= PG
k,G (A)r0 , . . . , Pk,G (A)r0
H
(Kk+1
We also can write Rk,G
G
vec(Rk,G ) = diag PG
k,G (A), . . . , Pk,G (A) vec(R0 ).
The matrix-valued polynomial involved in the preceding two studied methods are both diagonal. In the next section, we consider a general matrix-valued polynomial.
2.3. The block GMRES method. Another approach to solving (1.1) is to consider all
the s right-hand side vectors b(i) , i = 1, . . . , s, as an n × s matrix. This leads to the block
GMRES method (BGMRES). This method determines at step k an approximate solution
Xk,B of (1.1) from the requirements
Xk,B − X0 ∈ KB
k (A, R0 ),
Rk,B = B − AXk,B ⊥ KB
k (A, AR0 ),
and
(2.10)
where
k−1
KB
U },
k (A, U ) = block span{U, AU, . . . , A
and “block span” is defined by
(
KB
k (A, U )
=
X∈C
n×s
|X =
k−1
X
i=0
i
A U Ω i ; Ωi ∈ C
s×s
for i = 0, . . . , k − 1
)
⊂ Cn×s .
ETNA
Kent State University
http://etna.math.kent.edu
ALGEBRAIC PROPERTIES OF THE BLOCK GMRES AND BLOCK ARNOLDI METHODS
211
Alternatively, BGMRES can be defined by considering the approximate solution of the
ith system (1.2), which is determined by
(i)
(i)
(i)
xk,B − x0 ∈ Kk (A, R0 ) and (Aj R0 )H rk,B = 0,
j = 1, . . . , k; i = 1, . . . , s,
(2.11)
where
Kk (A, R0 ) = Range([R0 , AR0 , . . . , Ak−1 R0 ]) ⊂ Cn .
(2.12)
Note that the Krylov subspace Kk (A, R0 ) is a sum of s classical Krylov subspaces
Kk (A, R0 ) =
s
X
(i)
Kk (A, r0 ).
i=1
Each column of the residual matrix Rk,B is obtained by projecting orthogonally the corresponding column of R0 onto the block Krylov subspace A Kk (A, R0 ). Therefore, BGMRES
is a minimization method
kRk,B kF =
min
Z∈KB
k (A,R0 )
kR0 − AZkF .
The following result will be used in the sequel.
T HEOREM 2.2. ([2]) Let the matrix Wk = AKk be of full rank. Then
−1 H
Wk R0
Rk,B = R0 − AKk WkH Wk
R0
Wk
H
/
W
W
=
k .
k
WkH R0 WkH Wk
Following Vital [29], we introduce the operator
PG
k,B (A) ◦ R0 =
k
X
Ai R0 Ωi ,
i=0
−1
where Ω0 = Is , [Ω1 , . . . , Ωk ] = − WkH Wk
WkH R0 , and PG
k,B is the matrix-valued
polynomial defined by



k
Is
tIs
...
tk Is
X
 /WkH Wk  .
ti Ω i =  
(2.13)
PG
k,B (t) =
H
H
i=1
Wk R0
Wk Wk
Then the residual Rk,B can be expressed as
Rk,B = PG
k,B (A) ◦ R0 .
Theorem 2.2 helps us compare the residuals of standard GMRES applied to (1.2) and of
BGMRES applied to (1.1). The relation (2.3) is the key to developing convergence results for
GMRES [21, 22]. We have the following expression for the norm of the residuals determined
by BGMRES.
T HEOREM 2.3. Assume that the matrix Wk is of full rank. Then
(i)
krk,B k2 =
" H
(i)
(i)
T
0
e1 r0 r(i)
WkH r0
1
(i) H
Wk
r0
WkH Wk
#−1
for i = 1, . . . , s.
e1
(2.14)
ETNA
Kent State University
http://etna.math.kent.edu
212
L. ELBOUYAHYAOUI, A. MESSAOUDI, AND H. SADOK
Proof. From the first expression in Theorem 2.2, we deduce that
(i)
(i)
rk,B = r0 − Wk WkH Wk
Consequently,
(i) H (i)
rk,B
(i)
−1
(i)
krk,B k2 = r0
−1
(i)
WkH r0 = (I − Wk WkH Wk
(i)
(i)
= (r0 )H r0 − (r0 )H Wk WkH Wk
and so we obtain
(i)
krk,B k2
=
"
(i)
WkH )r0 .
−1
(2.15)
(i)
WkH r0 ,
!
#
(i)
(i)
(i)
(r0 )H (r0 ) (r0 )H Wk
H
/ Wk Wk .
(i)
WkH r0
WkH Wk
(2.16)
(i)
Hence, krk,B k2 is the Schur complement of WkH Wk in the matrix
"
(i)
(i)
(r0 )H r0
(i)
WkH r0
#
(i)
(r0 )H Wk
,
WkH Wk
which can be factored into a product of a block upper and a block lower triangular matrix
(UL factorization)
#
# "
−1 " (i) 2
(i)
(i)
(i)
(i)
krk,B k
0
(r0 )H r0
(r0 )H Wk
1 (r0 )H Wk WkH Wk
.
=
(i)
(i)
0
I
WkH r0
WkH Wk
WkH Wk
WkH r0
This factorization yields
eT1
" H
(i)
(i)
r0 r0
(i)
WkH r0
(i) H
Wk
r0
H
Wk Wk
#−1
e1 =
1
(i)
krk,B k2
,
(2.17)
which proves the theorem.
The above theorem allows us to improve the well-known result
min
Z∈KB
k (A,R0 )
kR0 − AZkψ ≤ max
min
i=1...s z ∈K (A,r (i) )
i
k
0
(i)
kr0 − zi k,
which was stated in [26, 27] with kZkψ = max (kzi k), and was shown by Vital in her
i=1,...,s
thesis [29]. It shows that the residual obtained by BGMRES is bounded by the maximum of
the norm of the residuals obtained by applying standard GMRES to each one of the s systems
(1.2).
T HEOREM 2.4. Let the matrix Wk be of full rank. Then
(i)
(i)
krk,B k ≤ krk,S k
for
i = 1, . . . , s
and
kRk,B kF ≤ kRk,S kF ≤ kRk,G kF .
ETNA
Kent State University
http://etna.math.kent.edu
ALGEBRAIC PROPERTIES OF THE BLOCK GMRES AND BLOCK ARNOLDI METHODS
213
Proof. It suffices to show the first part of the theorem for i = 1. Let us first remark that
there exists a permutation matrix P , such that Wk = A[K1,k , . . . , Ks,k ] P . Therefore, we
(1)
can rewrite the expression of kRk,B k2 as
1
,
eT1 Fk−1 e1
(1)
krk,B k2 =
where

(1) H (1)
r0
r
 0
(AK1,k )H r0(1)
Fk = 
..


.
H (1)
(AKs,k ) r0

(1) H
AKs,k
...
r0

H
. . . (AK1,k ) AKs,k 
.
..


...
.
(1) H
AK1,k
r0
H
(AK1,k ) AK1,k
..
.
H
(AKs,k ) AK1,k
H
...
(AKs,k ) AKs,k
H
By noticing that the (k + 1) × (k + 1) principal submatrix of Fk is K1,k+1
K1,k+1 , using
Theorem 6.2 of [31, p. 177] and (2.3), we deduce that
1
(1)
krk,B k2
H
= eT1 Fk−1 e1 ≥ eT1 (K1,k+1
K1,k+1 )
−1
e1 =
1
(1)
krk,S k2
.
To prove the last inequality, we apply (2.5) and Theorem 7.2 of [15], and obtain
kRk,S kF =
s
X
eT
i=1 1
H
(Ki,k+1
1
1
= kRk,G k2F ,
≤ T Ps
H
−1
Ki,k+1 ) e1
e1 ( i=1 Ki,k+1 Ki,k+1 )−1 e1
which completes the proof.
We now examine the zeros of the matrix-valued polynomial PG
k,B .
T HEOREM 2.5. Let the matrix Wk be of full rank. Then
det(PG
k,B (t)) =
ks
(k)
Y
(α − t)
i
(k)
i=1
αi
,
(k)
where the αi , for i = 1, ..., ks, are the generalized eigenvalues of the matrix pair
{WkH Wk , WkH Kk }.
Proof. Let α be a root of det(PG
k,B ). Then from Theorem 2.2, we deduce that

det 
Is
WkH R0
αIs
...
WkH Wk
αk Is

 = 0.
(2.18)
Let us denote the ith block column of this determinant by Ci . Then by replacing the block
column Ci by Ci − αCi−1 for i = 2, . . . , k, we obtain
det(WkH Wk − αWkH Kk ) = 0,
(2.19)
which shows that α is a generalized eigenvalue of the matrix pair {WkH Wk , WkH Kk }. The
proof is completed by noticing that det(PG
k,B (0)) = 1.
ETNA
Kent State University
http://etna.math.kent.edu
214
L. ELBOUYAHYAOUI, A. MESSAOUDI, AND H. SADOK
3. Block Arnoldi-type algorithms for multiple starting vectors. This section provides the framework for block Arnoldi-type algorithms. These algorithms are used for determining multiple or clustered eigenvalues. They also are applied in implementations of block
and global GMRES methods.
We give a unified presentation of Arnoldi-type algorithms, which include the standard
Arnoldi algorithm applied to each column of the starting block vector, the global Arnoldi
method, and the block Arnoldi method. Let U be an n × s matrix. The Arnoldi-type algorithms construct a basis {V1• , . . . , Vk• } of a subspace of KB
k (A, U ). The basis satisfies an
orthogonality property and Hk• = (Vk• )H AVk• is upper block Hessenberg.
We examine three possibly choices of orthogonality. Let Φ• : Cn×s × Cn×s → Cs×s be
defined for • ∈ {B, S, G} by
 B
 Φ (X, Y ) = X H Y,
ΦS (X, Y ) = the diagonal of the matrix X H Y,
 G
Φ (X, Y ) = trace(X H Y )Is = hX, Y iF Is ,
for all X ∈ Cn×s and for all Y Cn×s .
If ΦB (X, Y ) = X H Y = 0, then the block-vectors X, Y are said to be block-orthogonal;
see Gutknecht [11]. Moreover, X is said to be block-normalized if X H X = Is . Of course,
the vector space of block vectors is a finite-dimensional inner product space with inner product hX, Y iF = trace(X H Y ). If hX, Y iF = 0, then X and Y are said to be F-orthogonal. If
ΦS (X, Y ) = diag(X H Y ) = 0, then we say that X and Y are diagonally orthogonal.
Using the map Φ• , we will show how the matrices Vk• and Hk• are computed.
Block Arnoldi-type algorithms
1. Let U be an n × s matrix.
•
2. Compute V1• ∈ Cn×s by determining the factorization of U : U = V1• H1,0
,
•
s×s
•
•
•
•
•
•
H1,0 ∈ C , such that H1,0 = Φ (V1 , U ) and Φ (V1 , V1 ) = Is .
3. for i = 1, . . . , k do
• Compute W = AVi• .
• for j = 1, . . . , i do
•
= Φ• (Vj• , W )
(a) Hj,i
•
(b) W = W − Vj• Hj,i
• End
•
•
•
• Compute Hi+1,i
by determining the decomposition of W : W = Vi+1
Hi+1,i
,
•
•
•
•
•
•
such that Hi+1,i = Φ (Vi+1 , W ) and Φ (Vi+1 , Vi+1 ) = Is .
4. End
We now consider two particular choices.
3.1. The block Arnoldi algorithm. For Φ• (X, Y ) = ΦB (X, Y ) = X H Y , the preceding algorithm reduces to block Arnoldi algorithm [11, 18–20, 25–27, 29], which builds an
orthonormal basis {V1B , . . . , VkB } such that the block matrix VkB = [V1B , . . . , VkB ] satisfies
H
(VkB ) VkB = Iks . It is well known that
B
B
AVkB = VkB HkB + Vk+1
Hk+1,k
EkT ,
(3.1)
where EkT = [0s , . . . , 0s , Is ] ∈ Rs×ms . Multiplying equation (3.1) by Ek , we deduce that
H
B
B
Vk+1
Hk+1,k
= AVkB − VkB VkB AVkB .
ETNA
Kent State University
http://etna.math.kent.edu
ALGEBRAIC PROPERTIES OF THE BLOCK GMRES AND BLOCK ARNOLDI METHODS
215
H
B
We also have AV1B H1,0 = AU and V2B H2,1
= AV1B − V1B V1B AV1B , which imply that
H
B
V2B H2,1
H1,0 = AU − V1B V1B AU . Thus, by induction, we deduce that
H
B
B
B
B
Vk+1
Hk+1,k
Hk,k−1
· · · H1,0
= Ak U − VkB VkB Ak U.
Furthermore, if Kk = VkB RB
k is the QR decomposition of the full-rank matrix Kk , then
Vk Vk B = Kk (KkH Kk )−1 KkH . Hence, we have
B
B
B
B
Vk+1
Hk+1,k
Hk,k−1
· · · H1,0
= Ak U − Kk (KkH Kk )−1 KkH Ak U.
(3.2)
B
A
Consider the representation Vk+1
= PA
k,B (A) ◦ U . Since it is not easy to express Pk,B in
terms of Krylov matrices, we consider a monic matrix-valued polynomial, which, apart from
eA
a multiplicative matrix, is the polynomial PA
k,B . Thus, let Pk,B denote the matrix-valued
polynomial
B
e A (t) = PA (t)H B
P
k,B
k,B
k+1,k · · · H1,0 ,
and let {Zk } be the block vectors defined by Z1 = U and
B
B
B
Zk+1 = Vk+1
Hk+1,k
· · · H1,0
.
Then
A
(A) ◦ U
Zk+1 = I − Kk (KkH Kk )−1 KkH Ak U = Pek,B
e A can be expressed as
The matrix-valued polynomial P
k,B

tk Is
eA (t) = 
P
k,B
KkH Ak U
Is
...
tk−1 Is
KkH Kk
k ≥ 1.
for


 /KkH Kk  .
Applying the determinant function to this Schur complement, we obtain

 k
t Is
Is
...
tk−1 Is

det 
H
H k
K
A
U
K
K
k
k
k
e A (t)) =
.
det(P
k,B
det(KkH Kk )
eA .
The following result examines the zeros of P
k,B
T HEOREM 3.1. Let the matrix Kk be of full rank. Then
e A (t)) =
det(P
k,B
(k)
ks
Y
(k)
(t − θi ),
i=1
where θi , i = 1, ..., ks, are the eigenvalues of the matrix (KkH Kk )−1 (KkH AKk ).
e A (t)). It follows from (3.5) that
Proof. Let θ be a root of det(P
k,B

Is
det 
...
KkH Kk
θk−1 Is
θk Is
KkH Ak R0

 = 0.
(3.3)
(3.4)
(3.5)
ETNA
Kent State University
http://etna.math.kent.edu
216
L. ELBOUYAHYAOUI, A. MESSAOUDI, AND H. SADOK
Let Ci denote the ith block column of this determinant. Then replacing Ci by Ci − θCi−1
for i = 2, . . . , k, we obtain
det(KkH AKk − θKkH Kk ) = 0.
Since the matrix KkH Kk is nonsingular, θ is an eigenvalue of (KkH Kk )−1 KkH AKk . This result,
for the special case s = 1, is shown in [21].
Using the QR decomposition of the full-rank matrix Kk = VkB RB
k , we deduce that
−1 B B
Hk Rk .
(KkH Kk )−1 KkH AKk = RB
k
B
Consequently the roots of PA
k,B (t) are the eigenvalues of Hk .
3.2. The global Arnoldi algorithm. We have
Φ• (X, Y ) = ΦG (X, Y ) = hX, Y iF Is .
Hence, the global Arnoldi process builds an F-orthonormal basis {V1G , . . . , VkG } of KB
k (A, U ),
such that the matrix VkG = [V1G , . . . , VkG ] satisfies
G
G
AVkG = VkG HkG + Vk+1
Hk+1,k
EkT ,
where HkG = HkG ⊗ Is and the matrix HkG is a k × k Hessenberg matrix whose nonzero
entries (hG
i,j ) are defined by the following algorithm.
Global Arnoldi algorithm
1. Let U be an n × s matrix.
2. Compute V1G ∈ Cn×s by V1G = U/kU kF ,
3. for i = 1, . . . , k do
• Compute W = AViG .
• for j = 1, . . . , i do
G
(a) hG
j,i = hVj , W iF
G
(b) W = W − hG
j,i Vj
• End
G
• Compute hG
i+1,i = kW kF and set Vi+1 = W/hi+1,i .
4. End
G
G
A
It is easy to see that Hk+1,k
= hG
k+1,k Is and that Vk+1 = Pk,G (A)U . Moreover, if we
set
G
A
e A (t) = hG
P
k,G
k+1,k · · · h2,1 kU kF Pk,G (t)
eA ,
and use the explicit form of P
k,G

eA (t) =
P
k,G
det 
tk
KkH
1
k
...
tk−1
(KkH
⋄ (A U )
⋄ Kk )
det(KkH ⋄ Kk )
we can characterize the roots.
T HEOREM 3.2. Let the matrix (KkH ⋄ Kk ) be nonsingular. Then
e A (t)) =
P
k,G
s
Y
i=1
(k)
(t − θei ),


,
(3.6)
ETNA
Kent State University
http://etna.math.kent.edu
217
ALGEBRAIC PROPERTIES OF THE BLOCK GMRES AND BLOCK ARNOLDI METHODS
(k)
where θei , for i = 1, ..., s, are the eigenvalues of the matrix (KkH ⋄ Kk )−1 (KkH ⋄ (AKk )).
(k)
The eigenvalues θei can be called the F-Ritz values, since they also are the eigenvalues
H
G
of the Hessenberg matrix HkG = (VkG ⋄ (AV k )).
When we apply the global or the block Arnoldi processes with s = 1 and with the ith
columns of U , we obtain the standard Arnoldi process. Hence, the standard Arnoldi vectors
(i)
(i)
A
obtained with the ith columns of U can be written as vk = pk,S (A)U ei . Let Vk,S
be the
(1)
(s)
vector whose columns are vk , . . . , vk . We have
(1)
(s)
A
Vk,S
= [pk,S (A)U e1 , . . . , pk,S (A)U es ].
Consequently,
A
vec(Vk,S
) = PA
k,S (A) vec(U ),
(1)
(3.7)
(s)
where PA
k,S (t) = diag(pk,S (t), . . . , pk,S (t)).
4. Examples. We illustrate the theory developed in this paper with two examples. The
first one involves a diagonalizable matrix; the matrix of the second example is defective. In
these examples, we set X0 = 0 and U = B.
E XAMPLE 4.1. Consider the matrix and right-hand sides




1 1
−1 0 −1 1
 0 0
 0 2 0 −1



A=
 0 0 1 −1 and B =  1 1 .
−1 2
0 0 0 −2
Results obtained by the block Arnoldi and BGMRES√ methods are reported in Table 4.1.
2
A
Moreover, we have for block Arnoldi, det(PA
1,B (t)) = 18 (2t + 1)(t + 2) and det(P2,B (t)) =
√
2 2
9 (t
− 1)(t2 − 4). Hence, the eigenvalues of the matrix A are the roots of PA
2,B . We also
1
remark that the roots of PA
and
−2.
are
−
1,B
2
k
1)
2 2
+ 23 t −
3 (t
√
2 t
3 ( 3 + 1)
2)
3
3
1
2
3
3
PA
k,B (t)
2
3 (t√+
− 62
√
√
"√
"√
#
1
3 (t − 2)
t + 34
#
− 10
t
+
4)
3
t − 59 t − 38
1 2
3 (t
2
PG
k,B (t)
3
26 t
+1
1
− 13
t
(− 34 t2 − 16 t + 1
1
− 41 t2 − 12
t
5
26 t
7
13 t
+1
− 21 t2 + 13 t
− 21 t2 − 16 t + 1
TABLE 4.1
Polynomials obtained by the block Arnoldi and BGMRES methods.
On the other hand, the upper block Hessenberg matrix H2B determined by block Arnoldi
algorithm is
√
√


5 2
1
2
−1
2
18
√
√9
 2
2 
−3
5
 2
2
18
18√ 
H2B =  3√
,
11
 2 2 − 12
− 5182 
18
0
√
2
2
√
−5 182
17
9
ETNA
Kent State University
http://etna.math.kent.edu
218
L. ELBOUYAHYAOUI, A. MESSAOUDI, AND H. SADOK
A
Pk,G
(t)
1
√ (3t + 4)
23
k
1
√ 1
(69t2
15042
2
3
1
3
3037830 (981t
3
4
q
G
Pk,G
(t)
4
( 13 t + 1)
+ 1482t2 − 2066t − 2182)
327
2
9290 (t
88
1467 t
2
( −21
113 t +
+ 64t − 91)
3
( −2182
11809 t −
1 2
4 (t
− 1)(t2 − 4)
4589 2
11809 t
+
+ 1)
716
1687 t
+ 1)
− 1)(t2 − 4)
TABLE 4.2
Polynomials obtained by the global Arnoldi and global GMRES methods.
k
√
5
5
1
"√
6
2
6 (3t
2
3
"√
30
3
120 (15t
A
Pk,S
(t)
t+1
0
0
2t + 3
+ 2t − 5)
0
#
0
√
870
2
870 (15t + 22t − 7)
#
0
174
3
2
870 (87t + 108t − 323t − 352)
#
"√
30
0
√
(t2 − 1)(t2 − 4) 8
174
0
10
+ 15t2 − 28t − 8)
0
4
√
TABLE 4.3
Polynomials obtained by the standard Arnoldi method.
G
Pk,S
(t)
0
6t + 1
3
0
7t + 1
k
1
1
2
3
44
2
0
( −75
161 t − 161 t + 1)
178
2
0
( −3
71 t + 497 t + 1)
1 2
3
( −1
22 t − 2 t −
0
2
11 t
+ 1)
1 2
4 (t
4
0
443 2
3
( −352
t
−
1567
1567 t +
1358
1567 t
+ 1)
− 1)(t2 − 4)I2
TABLE 4.4
Polynomials obtained using the Standard GMRES method.
with the characteristic polynomial
PH2 (t) = det(tI4 − H2B ) = (t2 − 1)(t2 − 4).
The polynomials determined by global Arnoldi and global GMRES are displayed in Table 4.2. The standard Arnoldi methods yields the polynomials of Table 4.3, and the polynomial determined by standard GMRES are shown in Table 4.4.
ETNA
Kent State University
http://etna.math.kent.edu
ALGEBRAIC PROPERTIES OF THE BLOCK GMRES AND BLOCK ARNOLDI METHODS
k
A
Pk,B
(t)
(4t
−
9)
4
1
1
√
10
5
2
t−1
√
5 10
3
5
4t − 7
− 41
3
5t − 2
t
4 −1
3
4 (t − 3)
5t−7
4
219
G
Pk,B
(t)
7
t
− 18 t + 1
18
13
29
− 54
t
− 54
t+1
−3t
t−1 5t − 8
8
−t
7t − 8
TABLE 4.5
Polynomials obtained by the block Arnoldi and BGMRES methods.
E XAMPLE 4.2. Define the defective matrix

1 2
0 1
A=
0 0
0 0
and let

1
1
B=U =
0
0
1
0
1
0

0
1

0
2

2
0
.
1
1
Table 4.5 shows the results obtained by the block algorithms. The upper block Hessenberg
matrix determined by the block Arnoldi algorithm is given by
√ 
√

45
5
−3√ 10 4 10
1 
25
5 10
0 
.
 −15
√
√
H2B =

−2
−4 
20 5 10 −3√ 10
0
4 10
16
32
It has the characteristic polynomial
PH2 (t) = (t − 1)3 (t − 2).
We also have
√
2 19
A
P3,G
(t) = √ (t − 1)2 (t − 2),
67
1
G
P3,G (t) = − (t − 1)2 (t − 2),
2
"√
38
A
2
P3,S (t) = (t − 1) (t − 2)
0
0
√
66
9
#
,
1
G
P3,S
(t) = − (t − 1)2 (t − 2).
2
We remark that for all iterations except for the last one, the roots of the BGMRES polynomials are, in general, different from those of the corresponding Arnoldi polynomials. Moreover, apart from a multiplicative scalar, the determinant of the Arnoldi polynomial is the characteristic polynomial of the Hessenberg matrix obtained from the Arnoldi-type algorithms.
ETNA
Kent State University
http://etna.math.kent.edu
220
L. ELBOUYAHYAOUI, A. MESSAOUDI, AND H. SADOK
REFERENCES
[1] W. E. A RNOLDI , The principle of minimized iterations in the solution of the matrix eigenvalue problem,
Quart. Appl. Math., 9 (1951), pp. 17–29.
[2] R. B OUYOULI , K. J BILOU , A. M ESSAOUDI , AND H. S ADOK, On block minimal residual methods, Appl.
Math. Lett., 20 (2007), pp. 284–289.
[3] R. B OUYOULI , K. J BILOU , R. S ADAKA , AND H. S ADOK, Convergence properties of some block Krylov
subspace methods for multiple linear systems, J. Comput. Appl. Math., 196 (2006), pp. 498–511.
[4] C. C. C HU , M. H. L AI , AND W.S. F ENG, MIMO interconnects order reductions by using the multiple point
adaptive-order rational global Arnoldi algorithm, IECE Trans. Elect., E89-C (2006), pp 792–808.
[5] C.C. C HU , M.H. L AI , AND W.S. F ENG, The multiple point global Lanczos method for multiple-inputs
multiple-outputs interconnect order reduction, IECE Trans. Elect., E89-A (2006), pp. 2706–2716.
[6] M. H EYOUNI AND K. J BILOU, Matrix Krylov subspace methods for large scale model reduction problems,
Appl. Math. Comput., 181 (2006), pp. 1215–1228.
[7] K. J BILOU AND A. M ESSAOUDI , Matrix recursive interpolation algorithm for block linear systems. Direct
methods, Linear Algebra Appl. 294 (1999), pp. 137–154 .
[8] K. J BILOU , A. M ESSAOUDI , AND H.S ADOK, Global FOM and GMRES algorithms for matrix equations,
Appl. Numer. Math., 31 (1999), pp. 49–43.
[9] K. J BILOU AND A.J. R IQUET, Projection methods for large Lyapunov matrix equations, Linear Algebra
Appl., 31 (2006), pp. 49–43.
[10] Y. Q. L IN, Implicitly restarted global FOM and GMRES for nonsymmetric matrix equations and Sylvester
equations, Appl. Math. Comput., 167 (2005), pp. 1004–1025.
[11] M. H. G UTKNECHT, Block Krylov space methods for linear systems with multiple right-hand sides: an
introduction, in Modern Mathematical Models, Methods and Algorithms for Real World Systems, A. H.
Siddiqi, I. S. Duff, and O. Christensen, eds., Anamaya Publishers, New Delhi, 2005, pp. 420–447.
[12] A. M ESSAOUDI , Recursive interpolation algorithm: a formalism for linear equations I. Direct methods, J.
Comput. Appl. Math., 76 (1996), pp. 13–30
[13]
, Recursive interpolation algorithm: a formalism for linear equations II. Iterative methods, J. Comput.
Appl. Math., 76 (1996), pp. 31–53.
[14] D. O’L EARY, The block conjugate gradient algorithm and related methods, Linear Algebra Appl., 29 (1980),
pp. 293–322.
[15] C. R. R AO, Statistical proofs of some matrix inequalities, Linear Algebra Appl., 321 (2000), pp. 307–320.
[16] Y. S AAD AND M. S CHULTZ, GMRES: A Generalized minimal residual algorithm for solving nonsymmetric
linear systems, SIAM J. Sci. Statist. Comput., 7 (1986), pp. 856–869.
[17] Y. S AAD, Numerical Methods for Large Eigenvalue Problems, Halstead, New York, 1992.
, Iterative Methods for Sparse Linear Systems, PWS, Boston, 1996.
[18]
[19] M. S ADKANE, Block Arnoldi and Davidson methods for unsymmetric large eigenvalue problems, Numer.
Math., 64 (1993), pp. 687–706.
[20] M. S ADKANE AND M. ROBB É, Exact and inexact breakdowns in the block GMRES method, Linear Algebra
Appl., 419 (2006), pp. 265–285.
[21] H. S ADOK, Méthodes de projections pour les systèmes linéaires et non linéaires. Habilitation thesis, University of Lille 1, Lille, France, 1994.
[22]
, Analysis of the convergence of the minimal and the orthogonal residual methods, Numer. Algorithms,
40 (2005), pp. 101–115.
[23] D. K. S ALKUYEH AND F. T OUTOUNIAN, New approaches for solving large Sylvester equations, Appl. Math.
Comput., 173 (2006), pp. 9–18
[24] I. S CHUR, Potenzreihen im Innern des Einheitskreises, J. Reine Angew. Math., 147 (1917), pp. 205–232.
[25] V. S IMONCINI , Ritz and Pseudo-Ritz values using matrix polynomials, Linear Algebra Appl., 241-243 (1996),
pp. 787–801.
[26] V. S IMONCINI AND E. G ALLOPOULOS , Convergence properties of block GMRES and matrix polynomials,
Linear Algebra Appl., 247 (1996), pp. 97–119.
, An iterative method for nonsymmetric systems with multiple right-hand sides, SIAM J. Sci. Comput.,
[27]
16 (1995), pp. 917–933.
[28] G. W. S TEWART, Matrix Algorithms II: Eigensystems, SIAM, Philadelphia, 2001.
[29] B. V ITAL, Etude de quelques méthodes de résolution de problèmes linéaires de grande taille sur multiprocesseur, Ph.D. thesis, Université de Rennes, Rennes, France, 1990.
[30] N. WAGNER, A Krylov based method for quadratic matrix equations, in Angewandte und Experimentelle
Mechanik – Ein Querschnitt, K. Willner and M. Hanss, eds., Der Andere Verlag, Tönning, 2006, pp. 283–
303 .
[31] F. Z HANG, Matrix Theory, Springer, New York, 1999.