AN ALTERNATING GRADIENi METHOD
FOR SOLVING LINEAR SYSTES
by
EVERETT CRAìER
RI&LE
A THESIS
submitted to
OREGON STATE COLLEGE
in partial fulfillment ol
the requireients for the
degree of
MASTEE OF SCIENCE
June 1959
APPROVED:
Associafle Professor of Mathematics
In Charge of Major
Chairnian of Department of Mathematics
Chairman oT School of Science Graduate Committee
Dean of Graduate School
Date thesis is presented
Typed by Everett Rig1e
Auust
11,
l8
TABLE OF CONTENTS
PA(E
I.
II.
III.
IV.
V.
VI
VII.
VIII.
INTRODUCTION
i
DEVELOPMENT OF FORMULAS
PROOF
01?
BASIC PROPERTIES
9
ITERATIVE ROUTINE
16
EXAITIPLES
18
CORRECTION OF ROU1D-0FF ERh3R8
21
TABLES
2i-
BIBLIOGRAPHY
38
AN ALTERNATING GRADIENT METHOD
POR SOLVING LINEAR SYSTEMS
INTRODUCTION
Various gradient methods for solving
linear equations have bee
systenis
presented. by Temple
of
(,
p. 476-500), Stein (k,p. 407-413), Forsythe and Motzkiri
(l,p. 304-305), Hestenes
and
and Stiefel
40936),
(2,p.
However, at present, there is no best
others.
method for the solution of systeas of linear equations,
as each method depends to some extent upon the particu-
lar system to be solved.
In the present paper, a modified gradient method
is exploited to
In
obtin the solution, if
linear equations in n unknowns.
not exist,
ari
it exists,
of
If the solution does
indication is obtained by the process of
solution.
Let the given system of equations be denoted in
matrix notation by Ax-b=O, where A is any
b
and
O
are
colwnn vectors with
in
zu
x n matiix,
components,
and x
is
a column vector with n components.
Define a function of z as the sum of the absolute
values of the components of Ax-b.
Ø(x) =
I(Ax-b)11 + KAX_b)21
That is,
K)mI
This positive definite function has the value zero if
and only
if x is the solution to the given system.
2
constant, represents a f aimily of suriaces in n
=
dimensions, with x as its center.
Given any arbi-
trary point, determined by the position vector X0,
that point lies on soue surface of
$=c.
The problem
euations now
of £indin; a solution to the systea of
becomes one of finding the center of the surfaces.
This will be accomplished in the following
manner:
at x0
The negative of the gradient of the function
will
be used as a
first direction vector
y0.
Proceeding in the direction of y0, a point x1 is
reached such that the distance,
minimum
Ix-x11
,
is a relative
with respect to this direction.
The point
lies in a hyperplane which is orthoonal to V0
and passes through the center x.
some
suriaco of
Ø=c2, hence,
It also lies on
the gradient of the func-
tion at this point can be determined.
direction vector y1 jg maie
a linear combination of grad
The second
orthogonal to
i
and V0,
v0by taking
As before,
the direction of the vector y1 is followed until a
point x2 is reached such that the distance, Ix_X21
a
,
is
relative minimum with respect to this direction,
This procedure is continued, obtaining a set o
mut-
ually orthogonal directiri. vectors (y0, V1, ...) and.
successive new estimates x1, x2, .... It will be
proved later, that ii
imate, XM
M
a
solution exists, the M
will be the solution x, tor
rnin(rn,n).
sorne
th.
est-
value of
4.
DEVELOPMENT OF FORMULkS
develop the ïorntüa for grad
4(x) is written as follows:
To
4,
the function
4(x)= 1a11x1+a12x2+.. .+a1x-b1I +.
The gradient of this function
.
f
ax-b1.
is the column vector
Xi
grad
4 =
;
e..
n
iisl(a1ixi+. e+Sinxn_bi)+*
I
'
e e e
s
.
For
any vector
e I
e
s s e e
I e
e
e
.+a1x-b1)+.
Ax_b=r1.
Xj,
+a1sn(a1x+.
e s
e
. .
s
s
e s
e. e.
e
e
ee.II
. . .
That is,
+axfl_b)=r2
,.eS.e.
sesee..,...
aalxjl+a2xj2+ ' ' +axjn_bm=rn
.
s .
.+amnxn_bm)
i
s
i e e e
e e
1s(a1x1+. e+amnxn_b)
a1ix1+a12x12+. .
a21x1+a22x12
.
r1i+...agx1
Then, grad
la sn
$=
...a
r.
i
int
sn r1
are sn
The notation
components
The
direction
v0=.rad $
v=ßrad
r
r..
represents the vector whose
(j=1,2,...,m).
vectors are developed by defining
and V1 as a linear combination of
and the previous direction vectors.
2)
e e e e
r1+...asgzi
grad $=ATsgn
1)
sgn r
le.a
I...
r
rad $
That is,
j+e1,1_1vj1+ej,j_2vj_2+.. .+ej,ovo.
The direction vectors are to be mutually orthogonal,
hence,
vl.vk=O
Then, v±evk=rad
(i,k).
$.vkel,_lv_l.vk+.. .+el,OvO.vk=O.
Each term of this equation is zero except for the
first term
and.
the term containing
grad
3)
e1,k=
.v1+e.
-grad $.eVk
VkeVk
V
Thus,
,k'k .vO;
-grad
=
v'v.
4
V,
(k=O,1,...,i-l).
Vk
As previously stated, the initial estiiaate X0 01
the solution x, is to be arbitrary.
cessive new estimates
x,
X2, ...
formula,
LI-)
xi+l=xi_-IiVi,
Then, the sucare obtained by the
6
is picked such that the distance,
vthere
x-x1t,
v.
x11x.
will be a minimum with respect to the direction
That is, v
will be orthogonal to the vector
Geometrically, this can be represented as follows:
fv±
xi
xi+1. ----Let
cl
Ixj+ijI ,
represent the distance
them,
(x1_x) .v
Ivi'
jVj
The desired vector
diagram as
xx1,
which is represented in the
can be written as
v.
by the normalized vector
-
(x1_x).v
(x1-x).v1
vi,
=
lvii
lvii
(x.
and
A1=
1
-x).v.
V.
Introduce now a vector
vi=
)
i
.v.
u,
such that,
Au.
_(x_xi)TATu±
en,
A i:-
r
vivi
But,
and 101 any vector
multiplied.
That is,
v
rtivi=
cl
-
T
vivi
Ax-b=O,
x,
Ax-b=r;
7
hence,
A(x_x1)=_r.
6)
Making this substitution, we now have,
ru
I'i= vivi
'T
However, this
be written in a more useful form
can.
as follows:
x=xj_
-"±_2v1_2;
X1_1 =X1_2
......... .. .........
x.=x
i
Hence,
o
-
x=x-
v1-. .
.-
(x1-x0) v1=O;
(x1-xxx0) .v1=O;
r..u.=r
i i o .u..
i
'ihus,
r'i=
(1
r
T
O
i
u.
'T
vivi
It is necessary to have u1 to determine
However, it is possible to deterraine u1 recursively
as fo1lois:
Since
v0=grad
$0ATsn
we have for u0, the vector sgn r0.
r0,
II
u
and hence
roi
are determined up through the value
v1= grad $1+e1
i-1,
we have,
,_1v1_1e ,_2v_2+e1 ,0v0;
v1=ATsgn
v1=AT(sgn
r1+e1,_iu±_i+e,1_.2u_2+...+e,0u0).
Thus the necessary recurrence relation for u1 is defined by:
8)
u1=sg n r.+e.
u j_l+ei,j_2uj_2 +.
.e.
u
PROOF OF IìA3IO PROPERTIES
Proof of the basic properties of this method are
given in the following theorems.
Theorem I.
...
The set of vectors y0, v1
form a mutually
orthogonal set.
The proof will be by induction.
v1=grad $1+e10v0;
By formula 2),
V1 .v0=grad
but, by formula
3),
,0v0 .v;
l
-grad $1.v0
e1,0=
vo vo
hence v1.v0=O, which implies y1 is orthogonal to y0.
Assume that the direction vector
is orthogonal
to all preceding direction vectors, that is,
VklVjO
(j=O,l,...,k-2).
By formula 2), vk=grad
k+ek klvk_1+...+ek 0v0;
v.vi=grad. $k.vk_lek,k_lvk_1 Vkl+I . .e!,OvO.vl,
v.v_1=grad $k.vk_l+ek,k_lvkl .v_1;
but,
by formula 3),
hence,
e11=
-grad $kVk_l;
VksVl=O.
lo
Similarly, for case
vv=rad
j
O
k-1,
$k.vJek,k_lvk_l.v+. ..+ek,ovovl;
v.vJ=rad $ksvekJv.v;
-grad
ek,=
v..vj
VksVO
hence,
Therefore, it is proved by induction that for all
v1
k,
is orthogonal to all preceding direetfon vecturs,
and y0,
v1
...
form a mutually orthogonal set.
Corollary:
Since they are orthogonal, if none
V0
v1
..., v
is the zero vector,
of the vectors
they are
linearly
independent.
Theorem II.
If
The
a
solution exists, vO when
proof will be by contradiction.
exists when soie
ti3fl
v=O
positive integer such that
By formula
and
v=O
rO
when
Let
for
i
R-l.
Assume a solus
be the first
r5O.
5),
vs= ATu5= O.
This leads to
us=
O
to
and ATu5=
cases to be considered; ATu5= o otien
when u
O.
li
Case
By
I.
when u5=O.
formula 8),
u9= sgn rs+es,s....lu.s_l +...+e s,o u o
Tacing the inner product of both sides of the equation
r,
sn rs rs
with the vector
u s 'r s =
9)
+e
However,
u ,.r s +...+e s,00
u rs
s,s-1s-.
Ax5-b=r5,
and by formula 'i.),
Xs = X
s-1 -A5_1v5_i.
Substituting,
we
have,
51Av51b=r5.
s-1
But,
hence,
r5=r5_1-
Similarly,
r5_1=r5_2-
5_2Av5_2;
r5 2=r5
5_v5-;
.
.
5_1Av5_1.
-
0sSI ø
.
s
s...
s
r1=r0- ?0Av0.
Then,
r5=r0- \0Av0- ?1AV1-.
.
.-
taking the inner product or both sides of this
equation with the vector u1,
Ui .r5=uj r0
0u 'Av0- ?1u .Av1-. .
_111j Av5_1.
Now,
Since v=ATu
this
Ui 'r5= u .r0-
can be written as follows,
0v .v0-
\1v
sv1-.
,
,_
'v51.
12
For j=O,l,...,s-1 every terni
on.
the ri,ht
equation is zero except for the first terni
containing v..v.
By formula
and.
of
the
the term
u.r0-Av.v.
Hence,
7),
sid.e
A-°3
i_
r .u.
vi.vi,
and.
u.r3=O for j=O,1,...,s-1.
tion into equation
9),
from
Making this substitu-
the previous pase,
sgu
By hypothesis
But if
r5r5.
hence, sgn r8.r8 > O.
u=O, u.r5=O.
This is a contradiction, thus,
u5jO.
Âu=O
Case II.
when
uÁO.
This implies that the vector u
each of the coiumnn vectors of A.
there exists R column vectors
ol'
which are linearly independent.
2'
'R' u)
u5=
If A is of raa& R,
A,
Sg;n
us...l=
(o,oc-, ...,c),
i
Therefore, the vectors
form a linearly independent set.
3),
By formula
is orthogonal to
r5+e5,5_1u5_1. .
sgn r5_1+e5_1,5_2u5_2. . .+e5_1,0 u;
... .... .. . . .. . . . a...
u0= S3fl r0.
13
Upon subtitution,
s11 r5+e5,_ici1 r5_1+...-4-(e530...e1,0)sgn
at least one of the constants is not equal
Since
to zero and the vectors (u5,
r51,...,sgn r0)
r,
s
form a linearly dependent set.
Now
consider the equation r=Ax-b. The
r
ooiaponents of
may be
r1=
If any
r=O
sgi:ì
r11
written,
Ir±I
sn r=O
r1= sn
then
as follows
:
(i=l,2,...,m).
and we may re-define
Jrj, where
andIrj=1, if r.=O.
if
row of this system by
1i1
Dividing the
=
IiiI
i
th
IrI, i=1,2,...,m respectively,
the system becines:
a11
Irt:
i
a21
sgn
r=
r'.
j2
e..
a12
I
b1
k1I
IriI
Ji
a22
a
b2
Ir2I
e..
2
am2
r'
jm
ImI
xjn
F
2I
e..
a
b
rjm
vector sgn r is a linear coubination of these
column vectors and the iva,rix composed of them is of
the same rank as the matrix (Ab) since the rows have
merely been divided by constants.
By the assumption
14
that a solution exists, rank (Ab)
=
rank A.
for (j=O,1,...,R-1), the set of vectors
sn
r2, ...,
sn rRl)
(s
Hence,
r0, srn
lie in a subspace of the space
spanned by the R independent column vectrs of A.
(u5,1,
Thereiore, the vectors
linearly dependent set.
2'
.*R)
form a
This is a contradiction,
thus, if a solution exists, ATu5
O for s
R-l.
Theorem III.
The method presented gives an exact solution, if
it exists, in M steps, where M
sion of
t;he
R.
(R is the
dimen-
smallest space spanned by the column vec-
tors of A).
Let M be the first natural number, if it exists,
such that x-x0 is in the subspace spanned by
vo, vi, ...,
V91.
Then,
x-x =k y +...+kMlvÌl
o
oo
(
(x-x0) .v1=k1v .v.;
C
x-x ).v.
i=
vi.vi
But, from formula 6),
-(x-x ).v.
(¼]
o
-
vi.vi
hence,
i
i
not all zero);
15
x=x0-
ana
From formula
X0v0-...-\1v_1.
x1=x0-
L1),
v0;
lS
?V
x2=x1-\1v1=x0.
..
y 00
........
X:X0_
Therefore,
x
=
.
.
s
..
x.
Theorem IV.
If the direction vectors
then
.., VR1 exist,
is always edual to zero.
Since
tlie
direction vectors (y0, y1, ...) are
linearly independent, the vectors (y0, y1,
.., vR_i)
span the space and VR must equal zero.
Theorem V.
If
v=O
For i
when
then no solution exists.
R-1 this theorem is just the contraposi-
tive of Theorem II.
For i=h, Theorem IV states that
Vp always equals zero ii V0, V1,
no solution exLsts,
r()
and.
..
VR_i exist.
the theorem is proved.
If
16
ITERATIVE huUTINU
The Íornu1as deve1opd can now be presented in
the Xollowing routine:
O)
Select an arbitrary vector x0.
Compute:
1)
r1=Ax1-b
2)
grad
(i=O,1,...,M-1).
=
ATs
r.
-erad
3)
e,k=
eO,k=O,
(k=O,1,... ,i-1).
n
)
u1=
s
)
v1=
ATu.
r
'.J)
f¼
o
T
u.
i
V. V.
i
i
7)
xi+1
=
4
v
17
A summary of the important attributes of this
method are listed be1ow:
1)
No restrictions of
any kind
are placed
upon the matrix A.
2)
If a solution exists, it is obtained in
M
)
R iterations.
If no solution exists, it is indicated in
Rl).
the i th iteration (i
¿)
matrix
unaltered during
the
given
the vector b are
process, so that a
A ano.
the
maximum of the oriina1 data is used.
advantage oÍ havin; many zeros in
the
The
matrix
is preserved.
5)
At the end of each iteration an estimate of
the solution is given, which is an improve-
ment over the one given in the preceding
iteration.
6)
At the end of any iteration one can start
anew, keeping the estimate last obtained as
the initial estinate.
EXAMPLES
In
this section
demonstrate the
speed computer.
'ivexi
various eamp1es will 0e
to
method. and exploit its use on a high
Special attention will
round-oU errors and
be given
comparison will be
the Conjugate Gradient Method
o
Hestenes
ith the exception of the first
iaade
and.
two, the
to
with
Stiefel.
results
of ail exaaples were obtained by use of the ALWAC III-O
Computer, using a single length floating point routine.
I.
Demonstration of Process.
The process of solution for a 3 x 3 system is
shown in Table I.
The exact solution is obtained in
three iterations.
exhibits the process involved
the inconsistency of this set o
The second example,
in demonstrating
three equations in two unknowns.
Comparison Vith the Conjugate Gradient
II.
of Hestenes and Stiefel
etLod
(2, p. k09_A1.36).
formulas presented by Hestenes and Stiefel in
the non-symmetric case were coded using the same reiThe
ative methods employed in coding the method presented
in this
codin
paper.
of either
No atten.pt was made to optLnize the
method and no correetion
of rcuiii-
off errors was made.
Table II, gives the number of nain memory channels
19
running tinie of various
The running tinie was calculated
systexas of equations.
fron the time of the last input until the last output
used for storage and the
was completed.
This includes the
ti3le
consumed in
typing out each succesive estimate of the solution.
Due
to
round-off
errors the exact solution
was
not
obtained in many cases of ill-conditioned matrices.
The relative accuracy of the solution of a few of the
:ìany
and.
systems tried are given by Tables III, IV, V,
VI.
A summary of the results o1 the comparison are
stated below:
1.
The Alternatin
Gradient Method requires more stor-
age space in a computer.
That is,
2ni
channels are
needed in excess of the requirement for the Method of
Conjugate Gradients.
2.
The time ìnvolved in the use of the Alternating
Gradient Method becomes increasingly jreater over the
tine involved in the Conjugate Gradient
Iethod, as the
dimension of the system increases.
3.
The Alternating iradient Method is more accurate,
epecilly
tions.
in large systems of ill-conditioned equa-
However, the Conjuate Gradient Method allows
continuance past the n th step for greater accuracy.
Thìs cannot be done with the Method of Alternatin;
Gradients.
Both methods allow startin
over
at any
20
time using the last estim-ite calculated as the initial
vector.
III.
Inconsistent Systems.
The inconsistency of a systeni oV equations is
exhibited by the direction vector v
O when
O.
Due to round-oif errors this condition is not usually
satisfied exactly.
muer product
However, in ail
01 this vector
cass
trieci.,
the
with itself gave a zero
divisor which caused the alarm to sound.
Furthermore,
in all cases tried where a solution did exist, none of
the direction vectors was near enough to zero to Sound
the alarm.
is
4ven
Usin
in.
An example of the type of results obtained
Table VII.
the correction formulas that are developed
in the next section, the direction vector did equal
zero exactly in every inconsistent system tried.
21
CORRECTION OF ROUND-OFF ERRORS
The formula for the successive approximations of
tue solution was given as
x1=x-v.
Tbi
recur-
sive í'orxaula presents two possible errors, namely:
1.
The
direction vector vi
does not
;ive
the correct
direction due to round-off errors.
2.
The distance, governed by
approximation Xj3
from
to the new
may be incorrect due to round-off
errors.
To correct the first of these, note that
v0=
ATsn
r0 must be very accurate, as the only proc-
esses involved are the multiplying of each component
of the columns of A by
t].
and.
adding.
Hence, by
insuring that every y1 that is generated after
forms a mutually orthoonal set with the previously
derived direction vectors, their directions are
guaranteed,
A method for this type of correction is
given by Cornelius Lanezos
Let
(,
p.
271).
v1= theoretically correct vector;
vi= actual computed vector;
v:t=
corrected vector;
i-1
V. .V
then,
v=v!a.
j
k=O
i
A.
V.
VkVk
That this method actually corrects the generated error
22
is proved
follows.
as
Let
(kO,l,...,í-1),
e,k+ Lk
=
represent the accumulation of the round-oil error
t.
Prom formula 2),
v=grad
1+e1,_iv_i+.. .+e,0v0;
v=g'rad
+ej,_iv_i+. . .+e,0v0+
v1=rad
Vj=
£kvk;
Vi+kVk.
k=O
Substituting this into the Lanczos formula,
i-1
i-1
=
v
i-1
Ek'crk)k
(v1+
L kVk
V.
VkSVk
Since the v's fornì a mutually orthogonal set,
v= V+
i-1
i-].
CkVkVk
Vk
Vk
vi A
vi
r
formula for the correction of round-off errors
of the second type can best be approached in the follow-
Ing geometrical way.
vi
X.
li \
\
x!.#'
1+1
Ii.
I
I
x_
.,
-
23
If
x and
. is
the hyperplane which contains the solution
is orthogonal to the direction vector
then
approxiatiun of the
is theoretically the next
solution x.
v,
Due to round-off errors assume that x!i1
is the actual computed approximation, then the error,
C, is the distance between x!
can be represented by the
This distance
and. x.
inner
product of the vector
(x1+i-x) with the normalized vector
v.
That is,
Iv±I
but by formula 6), this may be writtei,
i
.
1+.L
Ivi'
The correction formula may now be written as:
(rï.u)
x+l=
+l=
XI+i-.
1i
Ivi,
(r1i.u)
-,
vi.
vi
vi SV1
Examples of the corrections instituted by these
formulas are given in Table VIII.
2
TABLE I
DEMONSTRAHON OF PROCESS
Example 1.
x1x2-2x3=-1
1
1
-2
-xl+x2+ x5=-2
A = -1
1
1
xl+x2+ x3= 2
1
1
1
00)
X
o
=
r
o
=-1
i -2
i
li
i
li
i
grade o =1
(.2
e0,0
-i)
(i
J
2J
i
i
i
i
i
Ii
Ii
f].
13
=
(,l
O
fi
14)
u
o
=
fi
15)
vo = 13
lo
16)
=
Ii
Il
(i
3
i)
(1
3
0) 13
I.L_
I
=
lo
i7)
Xi
21 )
=
ri s
fi
Ii
Iii
-ti 210J li
134
151
ri
i
1.1
1
J-
=
f 34
l-34
f-i' f-i
1-341-1-21=
iJ11J
2J
-2
2
1--2J=
Ii-i
12)
13)
b =
Ii
fi
11)
-1
I
2
(h-i
25
1-1
22)
grad$1=
-2
-1
1
i
1
i
1
i
1-3
i
=
-1
23)
3
=-(-3-1
e
1,0
2)
O)
(1
=
.2
i
3
o
24)
u
-i)
i
-ii
S i
25)
3.
V
i
-2/5
8/5
-2/5
iI+i
=
-1
i
=
-2
-12/5
415
10/5
1-2/51
ii
i
1
i
1
18/51=
-2/5j
26)
-2/
(1
?
i
8/5
-2/5
1)
3
(-12/5
415
10/5)
13
1-12/5
415
10/5
I
1.
27)
X2
31)
i/il
-1/21 2/2J
1
J,,
1
1
i
1 -21
i
i
i
i
1-12/51
grad.
i
2
J
t-2
37/261
t_
1
37/26
1-i1
r
=1
l-21/26 - J-21
6/26
L
L
0
i
J
J
0
Io
i'ìfi
i
i
15/13
L-11
2J
Ii-i
32)
1
4151=1-21/26
6/26
L1o/5J
.I
O
=J
1)L-1
37)
L-3
1-12/5
e
-(0
2,1
0 -3)
(-12/5
4./5
10/5)
I
4./5
t
10/5
12
1-12/51
26
10/5
[
26
13
e
0-3)
-(0
2,0
(1
0)
3
2-=o
2--=
lo
1i1
Is
1.0
3L)
u2
i)
O
=
-
1 i -1
1
1
=
1.-2
1
1
=
o)
1._2/5J
1
1
1
I
8/51+
I
26
35)
O
01
1-2/51
+
10/131
12/13
_16/l3J
I
=
36)
I
10/13
12/13
1.-16/13
1-18/13
6/13
I
1 10/151
I
0 -15/13)
(15/15
(-18/15
6/13 -24/13)
12/131
-16/13J
12
1-i8/i1
6/13
I
37)
x
=
37/261
-21/26 - 12
6/26J
-18/131
6/131
-24/13)
I
=
1 2
I-i
1.
i
This is the exact solution.
Example 2.
x1-x2=-4
1 -1
l21
A=
2
00)
b=
1
1 -1
1i-ir
r
=
2
grad
:
1
12)
e0,0
=
O
(4
-g-
=
i
2
X0 _Ii
il)
13)
2
-4
2
27
1)
u
°
=
i
i
-1
_12
15)
o
16)
i
2-2)--1
(2
1)r21
(L1
-
-
1.
17)
Xl
21)
=
r*
1
(-
(
J-11/)
.
f-iiìs
j
1 -1
-
1)
8 (2)
11
1i1
iJ
8
.
-
I
-
1 -1
22)
grad
2
ii
-i
i
iJ
25)
-
-18/5
2
Ii
12/5
1
=
12
_-(-2-1)
1
(21)12
1¼1
2L)
1
u. =
1
25)
i =
-1
-1
+
1-21
i-1f +
1
i
2
o
=
-2
-1
(2)
Ill
Io
fO
This indicates that no solution exists.
28
TABLE
II
COMPARISON OF METHODS
Storage Space in Main Memory oi
Computer:
Alternating Gradient
control routine
4
sub-routines
(fioatiní pt.)
Conjugate Gradient
channels
channels
1+
channels
channels
general st orage
3n6
channels
n5
channels
totals
n+l5 channels
n+114-
channels
Runnin
Time:
Dimensions
of System
2
x 2
3
x
Alternating Gradient
Conjugate Gradient
31 sec.
50 sec.
3
1 min.
14 sec.
i rain.
4 x 4
2 min.
22 sec.
2 min. 12 sec.
6 x 6
6 min. 26 sec.
8 x 8
16 x 16
1 hr.
15 min.
4-O
27 min.
,2
sec.
sec. i hr.
10 sec.
5 min.
26 sec.
11 min.
6 sec.
6 min.
sec.
29
TABLE III
Ç0ÌlPARISON 0F METHODS
Two Equations in Two Unknowns:
Example 1.
(well-conditioned)
i.3tio of largest to
Initial vector
=
(5
smallest eigenvalues is 2.
[i
2
tl
OJ
x2
8).
Alternating
Gradient
Exact
Solution
O on j ugat e
Gradient
1.0000000
1.0000000
0.
1.0000000
1. 0000000
0.9999991
Example 2.
999999
(ill-conditioned)
Ratio of largest to smallest eienva1ues is 20,000.
200
256
x1
a-56
]
155.5625
Initial vector
Exact
Solution
=
(5
199.08
X2!
1sL.62sJ
8).
Alternating
Gradieiìt
Conjugate
Gradient
xl= 1.0000000
l.0018l
0.8772738
x2= 1.0000000
0.998500Ll
1.7127669
30
TABLE IV
COMPARISON
Oi
METHODS
Four Equations in Four Unknowns:
Example 1.
(well-conditioned)
Ratio of 1arest to s[nallest eigenvalues is 9.47.
-2
0
X1
-4
1-2 1 0
o
l-2 i
0
0
1-2
x2
5
x7
i
1
Initial vector
0
=
(1
)
-6
1
1
Exact
Solution
x1= 1.0000000
x2=-2 .0000000
x3= 0.0000000
X4= 5.0000000
Example 2.
1).
Alternating
Gradient
C onjugat e
0.9999999
-1. 99)9999
0.0000000
0. 9999978
Gradient
-1.9999988
o. 0000012
2.999998
2.
999977
(ill-conditioned)
Ratio of largest to smallest eigenvalues is 2984.
lo 7 8 7
x1
52
5
8
610 9
910
5
7
Initial vector
Exact
Solution
6
7
=
(0
5
0
0
x2
x
23
x4
51
33
0).
Alternating
Gradient
Conjugate
Gradient
xl= 1.0000000
x2= 1.0000000
0.9999721
1.0953595
1.0000532
0,8401629
x3= 1.0000000
x4= 1.0000000
0.9999796
1.0585410
1.0000264
0.9757126
31
TABLE V
COMPARISON OF METHuDS
Eight Equations in Eight Untnowns:
Example 1.
(well-conditioned)
Ratio oÏ 1ar:est to smallest ei:enva1ues is
xo= (5
lo
7
8
7
6
8
5
4
X1
55
7
5
6
5
7
7
k
(9
X2
50
8
610
9
8
5
8
2
x
56
7
5
910
5
9
6
8
x
59
6
7
8
5
5
7
6
X5
4-8
3
7
5
9
410
9
5
x6
57
5
14.
8
6
7
910
1
x7
50
L4.
9
2
8
6
5
1
5
x3
II-O
2 -1
-
-3
Exact
Solution
1.5
3O.39.
2 -k).
Alternatin,
Gradient
Conju'ate
Gradient
xl= 1.0000000
0.9999989
1.O05k4-
x2= 1.0000000
1.0000005
l.00u4616
X3::
1.0000000
0.9999998
0.9980522
XL1.=
lQ0000OO
0.9999j88
0.9957286
x5= 1.0000000
1.0000008
1.0026373
1.0000000
1.0000002
1.0021600
x7= 1.0000000
0.9999993
0.9993067
1.0000000
0.9999996
0.9988578
TABLE V (coNT.)
(ill-conditioned)
Example 2.
Ratio oI 1arest to smallest eigenvalues is 667.69.
1
1
1
1
1
1
1
1
X1
8.0
i
i
i
i
i
i
1
.9
x2
7.9
i
i
i
1
1
1
.9 .9
x
7.8
i
i
1
i
1
.9
.9 .9
XL
77
i
1
1
i
.9
.9
.9 .9
X5
7.6
i
i
1
.9 .9 .9
.9 .9
X6
1
1
.9 .9 .9 .9 .9 .9
X7
7.4
.9 .9 .9
x5
7.3
i
x0= (5
.
2 -i
Exact
Solution
.9
.9
4 -3
.9
1.5
2 -4).
A1ternatin
Gradient
Conjuate
1.0000000
1.0000303
0.9862018
x2= 1.0000000
0.9999988
1.0203175
X3= 1.0000000
1.0000105
0.9771331
x4= 1.0000000
0.9999942
1.0148934
x5= i.0000000
1.0000108
0.9934853
1.0000000
0.9999983
1.0067546
x7= 1.0000000
0.9999836
0.9880737
1.0000000
1.0000003
1.0132430
X1::
Gradient
33
TABLE VI
COMPARISON OF METHODS
Sixteen Equations in Sixteen
Urilcnowns:
Example 1.
1
2-7 4-1
4
2
0-2
1
4
3-2
0-1
2
1
0
3-1 4-2
2
1
2-1
0-4
1-1
0
1-13113-311102120-1
1 -1
0 -2
4
0 -1 -2
0
3
2
0--1-2-1-1
-4
1
1
0
1
1-1-2
1-1
0
1
0-1
1
1
1
0-1
-2
O
1
1
1
0
1
7
2
2
1
3
3
1
3
2-1
1
1
1
7-1
2
0
1
2
2
1
4-1
7
1
8
1
1-1-2-2 1-1
-
1-1-1 1
2
4-1-1
2
1-1-2
2
2
4 -1 -2
0
1
1-1
4
4
2
4
0-4
0-1
3-1--1
1
2
3
1-6
0
0
2-1
4
1
5-1-3
2
0-
1
O-2-1 1-1 1 0
i
i
1
2
1-1 1 0-1-1 3 4 0-2 0 1
4
3-2 3 1 4-1-3 1 0-2-1 0 1 0-2
2-2 1 1-1 0 0 2 0-1 1 1 3 2-1-1
1
1-1 4 0-1 0 1 1 1 1 3-1 O-1-2
3
1
1
2
0
1
3
b= (21 10 6 14 0 -2 14 8
16 10 6 11 6 -7 13).
x0= (1 1 1 1 1 1 1
1
1
1
1 1
1 1
1
1).
TABLE VI (CONT.)
Solution
Alternating
Gradient
Conjugate
Gradient
xl= -1.0000000
-0.9999985
-0.8892401
2.0000000
2.0000001
1.8488161
x3= -2.0000000
-2.0000007
-1.824722
xL1_=
1.0000000
0.9999993
1.1011682
x5=
2.0000000
1.9999999
2.0104537
x6
10000000
1.0000019
0.7997739
x7=
10000000
-1.0000017
-0.8470299
x8=
2.0000000
2.0000011+
1.6985474
-1.0000000
-0.9999993
-1.0599308
1.0000000
0.9999978
1.5231497
0.0000000
0.0000050
-0.6304432
2.0000000
1.9999982
2.2931508
xlS= 0.0000000
-0.0000036
0.1149432
x14=-1 .0000000
-1,0000004
-0.9168063
0000024
0.5898514
-0.0000009
0.4474895
Exact
x2=
X10
xl5= 1,0000000
x16= 0.0000000
1
35
TABLE VII
INOuNSIBtIENT SYSTE1iS
Example 1.
L_3
i
9
xl
11
11
-2
7
1
C)
X2
-1
3-2
1
x
ï
8-618
x
O
¿
The ratio Oí the 1arest to smallest eißenva1ue
excluding the zero elgenvalue is 7.9269.
X0
=
(O
;ko
=
0.0508474
O
O
O).
Ai0.639i669
r3= (-5.5OOOO4
v3= (0.0000006
10.9999942
-0.0000005
20)295198
-5.5000019
0.0000012
10.9999890).
0.0000013).
Example 2.
i
.5
.33333333
.25
.71428571
The ratio
.25771k236
the
o.E
=
1
1
.2
.1O4-7619
1arest to
exc1udin.; the zero
.25
x
i
.2
x.
O
.166666667
x
O
.1428714)
X
O
.25
.333333333
.5
X0 = (1
.33;333333
sLualiest eigenvalues
eienva1ue is 10.2273959.
1).
0.L19084837
A1
=
37.O5562O
r2= (-0.0000000
-0.1815374
-0.333)11
0.5714285).
v2= (-0.0000000
-0.0000000
-0.0000000
-0. 0000000).
1j
TABLE
VIII
CORREC: ION OF ROUND-OFF ERhORS
Example 1.
Equations in Two Unknowns:
The system given in Table III, Example 2, is used
as the lirst example. This system was i1l-conIitioned.
Two
=
(5
8).
Exact
Method oi
Alternating Gradients
With Correction
Without Correction
Solution
xl= 1.0000000
1.00183k1
1.0018177
x2= 1.0000000
0. 9985OO'4
O. 9985353
Example 2.
Four Equations in Four Unknowns:
This system is given in Table IV, Example 2.
The system is ill-conditioned.
X0
=
(0
0
0
Exact
Solution
0).
Method ot Alternating Gradients
Without Correction
'ith Correction
xl= 1.0000000
0. 9999721
1.0000072
x2= 1.0000000
1.0000532
0. 9999938
x3= 1.0000000
0. 9999796
i
x4= 1.0000000
1.00002&1-
O. 9999975
0000041
TABLE VIII (CONT.)
Example
.
Eight Equations in Eight Unknowns:
This system is given in Table V, Example 2.
It was not too badly conditioned, the ratio of its
largest to smallest eigenvalues being 667.69.
=
(5
2 -1
Exact
Solution
4 -5
1.5
2 -4).
Method of Alternating Gradients
Without Correction
With Correction
xl= 1.0000000
1. 000003
1.0000036
1.0000000
0.9999988
0.9999996
x3= 1.0000000
1.0000105
0.9999992
1.0000000
0. 9999942
0.
x5= 1.0000000
1 0000108
0 . 9999993
x6= 1.0000000
0. 9999983
0. 9999995
x7= 1.0000000
0.9999886
0.9)997
x3= 1.0000000
1
0000003
0 . 9999998
.
9999986
BIBLIOGRAPHY
1.
Forsythe, G. E. and. T. S. Motzkin. Acceleration of
the optimum gradient method. Preliminary report.
Bulletin 01' the American Matheivatical Society
57:5O.L3O5.
191. (Abstract).
2.
ilestenes, Magnus
3.
1aiic:os, Cornelius.
An iteration method for the
solution of the eigenvalue problem of linear differential and integral operators. U. S. National
Bureau of Standards Journal of h eseareh '+5:255-282.
R. and. Eduard Stie1e1. Method
ol: conjugate gradients Lor solving linear systems.
U. S. National Bureau of Standaris Journal of
Research '49:4-O9-436. 1952.
1950.
4.
Stein, arvin L. Gradient methods in the solution
of systems of linear ejuations. U. S. National
Bureau of Standards Journal of Research 48:407-413.
1952.
5.
Temple, G. The general theory of relaxation methods
applied to linear systems. Proceedings of the Royal
Society of London. Series A 169:4.76-500. 1939.