Chapter 2
Iterative Methods for Solving Sets of Equations
2.4.2 The Steepest Descent Method
If you move in the opposite direction of the steepest ascent you follow the steepest descent
path. You may find the minimum of a function f(x,y) starting at point 0 as shown in Figure
2.4-4 using the steepest descent method. In this method you can move in the opposite
direction of the initial gradient until f(x,y) stops decreasing, that is, becomes level along your
direction of travel. This stopping point (1) becomes the starting point where f is reevaluated
and a new direction is followed. The process is repeated until the bottom is reached.
y
a minimum
2
0
1
x
Figure 2.4-4 A graphical depiction of the method of steepest descent.
The steepest descent method to find the minimum can be applied to solve a system of
nonlinear equations of the form
f1(x1, x2, , xn) = 0,
f2(x1, x2, , xn) = 0,


fn(x1, x2, , xn) = 0.
This is due to the fact that the system of nonlinear equations has a solution at x = (x1, x2, ,
xn) precisely when the function g defined by
g(x1, x2, , xn) =
n

[ fi(x1, x2, , xn)]2
i 1
2-17
has a minimum value of zero. g can also be expressed as a product of two vectors
g = ff T
where f = [f1 f2  fn] and f T is the transpose of f. The method of steepest descent will
generally converge linearly to the solution. The method will converge even with a poor initial
guess therefore it is often used to find starting approximation for other techniques that can
converge faster. A strategy for the method of steepest descent can be outlined as follows:
1. Evaluate g at an initial approximation x(0).
2. Determine a direction form x(0) using the gradient of g.
3. Move to a new appropriate position x(1) so that g(x(1)) < g(x(0)).
4. Repeat steps 2-4 with x(0) replaced by x(1).
The details procedure for the method of steepest descent is discussed in the next example.
Example 2.4-2
Use the method of steepest descent with the initial guess x = [0 0 0] to obtain the solutions to
the following equations2
1
=0
2
f2(x1, x2, x3) = x12  81(x2 + 0.1)2 + sin x3 + 1.06 = 0
10  3
f2(x1, x2, x3) = e  x1x2 + 20x3 +
=0
3
f1(x1, x2, x3) = 3x1  cos(x2 x3) 
Solution
1. Evaluate g at an initial approximation x(0).
x(0) = [0 0 0]; f = [f1 f2 f3]
 f1 
g = ff T = [f1 f2 f3]  f 2  = f12 + f22 + f32 = 111.975
 
 f 3 
2. Determine a direction form x(0) using the gradient of g.
g(x) =
g 
g 
g 
i +
j +
k
x1
x2
x 3
Note: The variable used in Matlab program will be within the bracket {} for example g(x) {delg}.
2
Numerical Analysis by Burden and Faires
2-18
f 1
f
f
+ 2f2 2 + 2f3 3 ;
x1
x1
x1
f
f
f
2f1 1 + 2f2 2 + 2f3 3 ;
x2
x2
x2
f
f
f
2f1 1 + 2f2 2 + 2f3 3 ]
x 3
x 3
x 3
g(x) = [2f1
g(x){delg} = 2J  f
T
T
 f1
 x
 1
f
= 2 2
 x1
 f
 3
 x1
f1
x2
f 2
x2
f 3
x2
f1 
x3 
f 2 
x3 
f 3 

x3 
T
 f1  g1 
 f  = g 
 2  2
 f 3  g 3 
The magnitude of the gradient at x(0) = [0 0 0] is
|g(x)|{zo} = { g12 + g22 + g32}1/2 = {g(x)Tg(x)}1/2
Evaluate the unit vector in the direction of steepest ascent (descent)
z = g(x)/ |g(x)|
3. Move to a new appropriate position x(1) so that g(x(1)) < g(x(0)).
The direction of greatest decrease in the value of g at x is the direction given by  g(x),
therefore x(1) is given by
x(1) = x(0)  g(x)
g
g1
x(0)
g3
3
2

1=0
Figure 2.4-5 A typical behavior of g in the direction of steepest descent.
2-19
where  is to be determined. The procedure is to fit a quadratic to three points (1, g1), (2,
g2), and (3, g3) then choose  so that g is a minimum in the direction of steepest descent.
The steps are
a) Let 1 = 0 at x(0), therefore g1 = g(x(0)).
b) Let 3 = 1 and evaluate g3 = g(x(0)  3z)
c) If g3 > g1, let 3 = 3/2 and repeat step (b)
d) Let 2 = 3/2 and evaluate g2 = g(x(0)  2z)
The quadratic through three points (1, g1), (2, g2), and (3, g3) has the form
P() = g1 + h1 + h3(  2)
where h1 =
g 2  g1
2
, h2 =
At the location where
g3  g2
h h
, and h3 = 2 1
3  2
3
dP
h
= h1 + 2 h3  h32 = 0  0 = 0.5(2  1 )
d
h3
e) Evaluate g0 = g(x(0)  0z)
Since a quadratic through three points can have a minimum or a maximum as shown in
Figure 2.4-6,  is chosen so that g is the lowest value between g0 and g3 as follows:
f) If g0 < g3 then  = 0 else  = 3
g
g
g1 g0
g2
g3
g1
g2
1=0
2
g3
g0
3
0

0
2
3

Figure 2.4-6 A quadratic through three points can have a minimum or a maximum.
4. Repeat steps 2-4 with x(0) replaced by x(1).
Table 2.4-2 lists the Matlab program with values of x for 10 iterations.
2-20
Table 2.4-2 Matlab program for the steepest descent method ------------%
% Steepest Descent Method
% Example 1, pg.420 Faires and Burden
%
f1='3*x(1)-cos(x(2)*x(3))-.5';
f2='x(1)*x(1)-81*(x(2)+.1)^2+sin(x(3))+1.06';
f3= 'exp(-x(1)*x(2))+20*x(3)+10*pi/3-1' ;
% Initial guess
%
x=[0 0 0];
f=[eval(f1) eval(f2) eval(f3)];
g1=f*f';
for i=1:10
Jt=[3
2*x(1)
-x(2)*exp(-x(1)*x(2))
x(3)*sin(x(2)*x(3)) -162*(x(2)+.1) -x(1)*exp(-x(1)*x(2))
x(2)*sin(x(2)*x(3)) cos(x(3))
20];
% Gradient vector
%
delg=2*Jt*f';
zo=sqrt(delg'*delg);
% Unit vector in the steepest descent
%
z=delg/zo;
xs=x;a3=1;x=x-a3*z';
f=[eval(f1) eval(f2) eval(f3)];
g3=f*f';
while g3>g1
a3=a3/2;
x=xs-a3*z';
f=[eval(f1) eval(f2) eval(f3)];
g3=f*f';
end
a2=a3/2;x=xs-a2*z';
fsave3=f;
f=[eval(f1) eval(f2) eval(f3)];
g2=f*f';h1=(g2-g1)/a2;h2=(g3-g2)/(a3-a2);h3=(h2-h1)/a3;
%
% Choose a0 so that g will be minimum in z direction
%
a0=.5*(a2-h1/h3);
x=xs-a0*z';
f=[eval(f1) eval(f2) eval(f3)];
g0=f*f';fsave0=f;
if g0<g3
alfa=a0;g=g0;f=fsave0;
else
alfa=a3;g=g3;f=fsave3;
end
2-21
x=xs-alfa*z';
fprintf('x = %9.6f %9.6f %9.6f',x(1),x(2),x(3))
fprintf(' , g = %9.6f\n',g)
if abs(g)<1e-2, break, end
g1=g;
end
x=
x=
x=
x=
x=
x=
x=
x=
x=
x=
0.011218
0.137860
0.266959
0.272734
0.308689
0.314308
0.324267
0.330809
0.339809
0.345746
0.010096
-0.205453
0.005511
-0.008118
-0.020403
-0.014705
-0.008525
-0.009678
-0.008592
-0.009034
-0.522741 , g =
-0.522059 , g =
-0.558494 , g =
-0.522006 , g =
-0.533112 , g =
-0.520923 , g =
-0.528431 , g =
-0.520662 , g =
-0.528080 , g =
-0.520941 , g =
2.327617
1.274058
1.068131
0.468309
0.381087
0.318837
0.287024
0.261579
0.238486
0.217440
2-22