..
:,'
Afrika Matematika
Series 2 Vol. 4 (1991)
. -.
.
o\~'
, {
\
.
\
,
M.A. IBIEJUGBA, T.A. ADEWALE and O.M. DAMIGDOLA
Abstract
It was generally believed, that because of the extensive work involved in coding
and evaluating Hessian matrices, the most reasonable approach to unconstrained optimization that involves twice
differentiable factorable functions is one which completely avoids these calculations. In this paper, by using the wellknown quasi-Newton method and the ingenuity of computer programming technique wielded to carry out direct
inversion of the associated non-singular Hessian matrices of this class of problems, we obtain some numerical results
which compare favourably with known results.
1.
Introduction
Like other ~econd deri~ative methods, a quasi-Newton method can contrive a
quadratic approximation of the objectiverfu~ction ~nd .(uFthermore, the method has the
distinct advantage or'quadr~tic, q:myergep~ejn, the vicinity of the minimum. Many experts in
nonlinear ~ptl~i~atiori 'belfe~~ that second derivative inethods are more reliable because they
usually require significantly fewer number of iterations and function evaluations than such
methods which use only function or gradient informati~n: Hence this' explains the extreme
popularity pf NE(~t~~. and quasi-Newtoll. methods in the, fielp, of unconstr~in~d nonlinear
progra.mnung.
One thing that has impeded nu~erical prpgre~s in n?nlinear programming is
lack of a convenient method for representing and coding nonlinear functions of several
variables, its gradient vector and. Hessian at a point that would make computer solutions easily
available. McCormick [1] developed a factora.ble progra.rnm!ng c<;>de that has been helpful
in this regard. For our subsequent development, we recall those definitions of factorable
functions as defined in Refs. 2 and 4 as follows.
19
~,',' ',"
20
M.A. IBIEJUGBA et al.
Definition 1.1: A function f( x 1, X2, . . . ,xn) is a factorable function of several variables if it can
be represented as the last in a finite sequence of functions Uj(x)} which are composed as
follows:
(i) for j =
1, 2, .. ., n, fj(x) is given by fi(x) = xi;
(ii) for j> n, fj(x) is of the form Ji,(x) + ft(x), k, e < j,
or of the form Ji,(x)'. ft(x), k, e < j
or of the form T[h (x)] , k < j, where T (f) is a function variable.
of a single
A quadratic approximation of the objective function, f( i), can be made by neglecting third-andhigher-order terms in the Taylor series expansion represented as follows:
f(~)
= f(~) + V'T(~)(~ - ~) + 1/2~(~ - ~fV'2 f(~)(~ - ~) ,. ...
(1.1)
where V'2 f(~) is the Hessian matrix, H(~), that is, the square matrix of second partial derivatives
of the objective function, f(f..), evaluated at ~ and defined by
o2f
ox2
1
V'2 f(k) = H() =
(~ - !.k)
by
-6x
OXIOXn
I
I
o2f
OXnOXl
Replacing
o2f
...
...
where 6Xk - !.k.,-l - !.k>
(1.2)
o2f
ox
then Equation (1.1)
yields:
f(~+J) = f(~) + V'T f(f..k)6xk ' 1/2'(6xkfV'2 f(~)6xk
(1.3)
Let the transition at kth iterative step from a point Xk to the point !.k+ 1 be defined by:
!.k+ 1
~+1
=
!.k
+ 6Xk
= !.k + >"k'!)k-= !.k + >"kPk'
where,
!.Ie = ycct;)[ :,'0;:1 ;;'ic ~O ;1:.\;+ l'
P..k -" a unit, vector ir: the direction of l:-,Xk.
(1.4 )
(1.5)
MINIMIZING FACTORABLE FUNCTIONS
1!.1c
21
= any vector in the direction of ~Xk, ~k are chosen scalars such that,
L::lxlc = .\Ic~ = >'IcEJc
(1.6)
The minimum of I(;!) in the direction of L::lxk is obtained by differt:ntiating I(;!) with respect to
each of the components of L::lx and equati.ng the resulting expressions to zero to obtain:
L::lxk
= - [V2 1(~)r1 V I(~k),
(1.7)
where [.V2 I(xlc)] -1 is the inverse of the Hessian matrix of the objective function
I(~, with respect to ~ at~. Putting Equation (1.7) into Equation (1.4) the transition from
~ to ~k+l for Newton's method becomes:
~+1
= ~ - V I(~)[V2I(xlc) ,1
-1
(1.8)
It should be noted that both the direction and steplength are specified. If f(~) is actually
quadratic, only one step is required to reach the minimum of I(i). However, for a general
nonlinear objective function the minimum of I(~) will not be reached in one step, so
that Equation (1.8) is usually modified to conform to the following equation:
1~+1 = ~ - ~/(~),
(1.9)
of the steepest descent method by introducing the parameter ..\ for the steplength into Equation
(1.8) to give:
Ak [V2 1(~k)r1 LV(~)]
~+1 = ~ - II [V2/(!.k)] 1 [V
The ratio:
(1.10)
f(~)] II
Ak
"[I
['\72 1(~k)]=1 [Vl(;k)] II
is just some scalar quantity which can be denoted by ~A:,
hence, Equation (1.10)
~+1
or
= ~kbecomes:
- ~k [V2 1(::) r1 V I(Xk),
- -1
~+1 = -~k - AkH (~k)V I(~k)
The search direction!!. is now given by
Ek = -H-1(Xk)'\7/(~)
(1.Ua)
(1.Ub)
(1.12)
22
~tA\ IBIEtiUdBAet 8.1.
2:'q .Qua~iLN:~:wtbir!,~~t'hjj~ ja~g~ri~hm.' ,~,,:. , .""
",,,,,, , " ,,'l," ",' )";'.'H~F,'n I'; .,' ',',( '~", n ,,'" : ""c',>"'" ," . 'r, > "".e,
A ,p~!Ilber ,Qf,stJ.gg~t~C?JFh~~Y~ beep:off~red by many autho~s '(~ee, e.g: [2 ~ 10])
for 'the implementation 'Of Equation (1.11). Such suggestion§ include (i) ways of choOsing the
de$ired steplength; (ii) suitable choice.of search directions; (iii) computational ways of avoiding
the cumbersome task of coding and inverting
th~ 'HeSsian matrix whei-e th~f iilverseexist'sand' (iv)1exploitation-strategies of expressing the
6uter product of the Hessian matrix. Most of the methods proposed can be classified into the following two categories: (i) 'negative curvature direction
methods and (ii) descent-direction methods. However, several methods now combine features
from both types. It is generally believed [2] that even when the Hessian matrix of the objective
function is not positive definite, a lme search can be conducted &long a direction of negative
curvature to reduce the value 'of the objective function. The fonowing steps are recommended by
Polak [6; pp. 71 - 75] for solving such problems as min{f(~I~ E rn.n}, for !(~ that is twice
continuously differentiable.
'
Step 0: Select,; ~ E rn.nj select f3 E (0.5,0,8) and set k = O.
Step 1: Compute, V!(?:,k), the gradient at ~.
Step 2: If ~f(~k) = O,stop; else, go to step 3.
Step 3: Compute H(%k) = a2 f(~)/ ail. the Hessian matrix at ~k' Step 4: If H1(Xk) exists, compute E(~k) by solving the equation
H(!.AJE(fk) = - V !(~k),
and go to step 5; else set
Set>. = 1.
Step 5:
Step 6:
Step 7:
step 6.
Step 8:
3.
Compute 6 =
EC~k) = -:- V !(~k) and go to step 5.
>.
f(~ + >',\:E(fk» - 2(Vf(~'\:)'E(~))'
If 6::; 0, set >',\: = >. and go to step 8; else set >',\: = >'13 and go to
Set ~+l = ~ + >'kE(~'\:); set k = k + 1 and go to step 1.
Modifications of Newton's method
Fiacco and McCormick [11] were the first to introduce the idea of finding a direction of
negative curvature. Gill and Murray [10] developed a descent-direction method which avoids
the eigen-vector analysis by performing an LDLT factorization of the Hessian matrix of the
objective function. Emami and McCormick [12] developed a negative-curvature direction
method which made use of the outer
product form of the Hessian of a factorable function. Sisser [2] developed a
23.
MINIMIZING FACTORABLE FUNCTIONS
descent-direction method which also exploits the dyadic nature of the Hessian. Sisser perturbed
the Hessian matrices by a quantity p added to the elements in th~ main diagonal where p. is
determined to be a dyad. The resulting matrix is positive definite. A search parameter .A is
included, so that the recursive formula for Newton's method then becomes:
~+1
= ~ - .Ak [H(~k) + Pk~-I ~i:)
(3.1)
Equation (3.1) can be rewritten as given below:
v2 f = H == D + pI,
where D is the diagonal,
(3.2)
2m
p. =
L ISilJ Qt,
(3.3)
i=l+ I
where the ai are n x 1 vectors and Si are scalars. The resulting Hessian matrix is inverted by the
use of the Sherman-Morrison formula in [15] and generalized by Woodbury [13]:
(A + asaT) -I = A-I - A-Ia (8-1 + aT A-Ia) -I aT A-I
(3.4)
In this study, a computer programme is written, to test for positive definiteness and to
perform the task of inverting the Hessian matrix at each stage of the iteration. The experimental
results of the authors of this paper on some test functions are given in Tables 1 - 12. More
specifically, Table 1 is the minimization
of Rosenbrock's Banana-shaped valley function, Table 2 is the minimization of Sisser's
function, Table 3 is the minimization of Ros~nbrock's Cliff function, Table 4 shows the
minimization of a power function while Table 5 is a summary of the results of the authors of
this paper 'compared with those of Sisser's, etc.
4.
A coordinate transformation
It is usually profitable to scale the nonlinear programming problem whose solution is desired
by optimization technique as this often has significant influence on the performance of the
algorithm of interest. A well-scaled problem is one in which the contours of the objective
function are approximately hyperspherical or
elongated parallel to most search directions [3, pp. 30 - 34, 73 - 96, 190 - 217]. A
good scaling device ensures speedy convergence of optimization routines and can lead to
accurate solutions. Several effective scaling methods exist (see, e.g. Ref. 2) and could be used
to computational advantages. One of such methods that involves the transformation of the
variables from their physical nature to variables having properties that are desired in terms of
optimization is employed here and
24
M.A. IBIEJUGBA et al.
the computational experience of the authors of thi$ paper is clearly tabulated. For examplefj,
Table 6 is the minimization.of transformed Rosenbrock's Bananashaped valley function, Table
7 is the minimization of Powell's quartic function, Table 8 shows the minimi~tion of
hyperbola-circle function, while Table 9 is transformed Sisser's function, etc.
Let p( Xl, X2, . . . , xn) be a point in a certain coordinate system. Consider a
different coordinate system which can be generated from the original system by a simple
rotation. Let the coordinates of the point p with respect to the new coordinate system be (Xl,
X2, ..., Xn). Let the angle between the Xi-axis and the x;-axis be denoted by (Xi, X;); i, j = 1,
2,...2, n. Let f.i; be the direction cosine of the Xi-axis relative to the x;-axis. That is,
f.u = cos(Xl, Xl), 62 = cos(Xl, X2), ..., 6n = cos(Xl, Xn)
(4.1)
Therefore,
n
(4.2)
Xi = Lf.i;X;, i=I,2,...,n
;=1
The desired inverse transformation can be represented as (see [14]; pp. 1 - 16).
n
(4.3)
xi=LE;iX;, i=I,2,...,n
;=1
It is convenient to arrange the Ei; into a square array called a transformation matrix J, as
follows:
Eu
J
= I El
f.nl
where,
E12
...
6n
62
...
E2n
En2
...
Enn
( 4.4)
n
L Ei;Ek; =
bik
= {o, i= k
(4.5)
J=l 1, I = k
are the orthogonality conditions obeyed and where of course, bik is the Kronecker delta.
In this study, the test functions are transformed by suitable changes of variables and it is
c:liscovered that in most cases the tranformation matrices (the Hessian matrices) of the resulting
objective functions are diagonal matrices with positive real eigenvalues that guarantee positive
definiteness. This is because the
25
MINIMIZING FACTORABLE FUNCTIONS
transformations remove the cross-product terms of the quadratic approximation and the
objective functions are transformed to the "Canonical forms" :3;. A val
ley lies in the direction of the eigenvector associated with a small eigenvalue of the Hessian
matrix of the objective function. The direction, - H( x) -1 V f( x), evaluated at different points in
the space of the new variable always points toward the minimum of the objective function under
investigation.
5.
Numerical problems
The following test problems are used:
Example 5.1:
Rosenbrock's Banana-shaped valley function, n = 2
f(XI,X2) = (1 - x.)2 + l00(X2 - xi)2,
Starting
point: (-1.2,1).
Example 5.2:
Powell's quartic function, n = 4,
f(Xl, X2, X3, x.) = (Xl + lOx2)2 + 5(X3 -- x.)2 +- (X2 - 2X3).
Starting
- IO(XI - x.)',
point: (3, -1,0,1).
Example 5.3:
Powell's badly scaled function, n = 2,
f(Xl, X2) = (10.XIX2 - 1)2 +- [exp( -X.) + exp( -X2) - 1.000f2,
Starting
point: (0, 1).
Example 5.4:
Wood's function, n = 4,
f(Xl,:Z:2,X3,X4) = l00(x2 - xi)2 + (1 - x.)2 -r- 9O(x4 - x5)2
+1O[(x2 - 1)2 + (x. - 1)2] + 19.8(x2 - l)(x. - 1),
Starting point: (-3, -1, -3, -1).
Example 5.5:
Brown's badly scaled function, n = 2,
f(Xl,X2) = (Xl - 106)2 + (X2 - 2.10-6)2 + (XIX2 - 2)2,
Starting
point: (1, 1).
Beale's function, n = 2,
Example 5.6:
f(XI,X2) = [1.5 - Xl(1- X2W + [2.25 - xl(1 - x~W -r- [2.625 - xl(1 - x~W
I
Starting point: (1, 1).
'-
26
M.A. IBIEJUG£A et al.
Example 5.7:
-
Resenbrock's Cliff function, n = 2,
l(xl,x2) = «XI - 3)/100)2 - (Xl - %2) + exp[20(xl - X2)],
Starting point: (0, -"1).
Example 5.8: Gottfried function, n = 2,
I(xl, X2) = (xl - O.l136(XI + 3X2)(1 - Xl)]2 + [X2 + 7.5(2xI - x2)(1 x2)12, Starting point: (0.5, 0.5).
Example 5.9:
Hyperbola-circle function, n = 2,
I(xl, X2) = (XIX2 - 1)2 + (x~ + x~ - 4)2,
Starting
point: (0, 1).
Example 5.10:
Sisser's function, n = 2,
l(xl,x2) = 3x1- 2x~x~ + 3x~,
point: (1, 0.1).
Power function, n = 3t
Example 5.11:
Starting
/(X1,X2, xs) '" It. ;xl
Starting
r
'
point: (1, 1, 1).
Example 5.12:
A quadratic case, n = 3,
I(Xl,X2,X3) = 1 + (XI + X2 + X3) + (x~ + 2x~ -j- 4x; - 2XIX2),
Starting point: (0, 0, 0).
The following transformations are suggested to scale some of the problems:
Example 5.1a:
Transformed Rosenbrock's function, n = 2,
F(ZI,Z2) = zi + l00z~,
where
=) - XI, Z2 = X2 - XI'
Starting point:
(2.2, -0.44).
2
ZI
27
MINIMIZING FACTORABLE FUNCTIONS
Powell's quartic function, n = 4,
Example 5.2a:
F(Zl,Z2,Z3,Z4) = zi -+ 5z~ -+ z~ + lOz:,
where
Zl
Starting
= xl + 10X2, Z2 = X3 - X4, Z3 = (X2 - 2X3)2, Z4 = (Xl - X4)2,
point: (-7, -1, -1,4).
Example 5.3a:
Transformed Powell's badly scaled function, n = 2,
F(Zl,Z2) = zi - z~,
where
Zl
= 104x1X2 - 1, Z2 = [exp( -xd +
exp( -X2) - 1.0001J ' Starting
point: (-1, e-1 - 1.00(1).
Example 5.5a:
Transformed Brown's badly scaled problem, n
- 22
F( Z2 - Zl +) Z2 + Zl Z2 + 10 Z2 - 2 . 1 Zl ,
[
6
= 2,
0 -6
2 Zl,
]
where
Zl
Starting
= xl - 106, z2 = x2 - 2. 10-6, (1 -
point: 106, 1 - 2.10-6).
Example 5.6a:
Transformed Beale's function, n = 3,
F(zl, Z2, Z3) = zi + z~ + z;,
where
Zl = 1.5 - x1(1 - X2); Z2 = 2.25 - X1(1- xn; Z3 = 2.625 - x1(1 - x~);
point: (1.5, 2.25, 2.625).
Transformed Rosenbrock's Cliff function, n = 2,
Example 5.7 a:
Starting
F(Zl, Z2) = 1O-4zi - Z2 + exp(20Z2),
where,
Zl = Xl - 3, Z2 = Xl - %2,
Starting point: (-3, 1).
Example 5.8a:
Transformed Gottfried function, n = 2,
28
M. A. IBIEJUGBA et al.
F( Zl, Z2) is obtainable by the substitution
Starting point: (2, 0.5)
Zl = Xl + 3X2, Z2 = 2X1 - X2.
Example 5.9a: Transformed hyperbola-circle function, n = 2, F(Zl,Z2) is obtainable
by the substitution Zl = X1X2 + 2, Z2 = Xl + X2, Starting point: (2, 1).
Example 5.10a: Transformed Sisser's function, n = 2, F(Zl,Z2) is
obtainable by the substitution Zl = xi, Zz = x~, Starting point: (1,
0.01).
Example 5.11a: Transformed Power's function, n = 2, Consider the
transformation Zi =
x;,
Starting point: (1, 1, 1).
6. Discussion of computatio~~esults
{<
The computer programme used for-the untransformed problems is slightly modified to suit the
transformed version and run on model 128k Amstrad personal computer in single precision. The
study has demonstrated that:
(i) an algorithm cannot be expected to solve all problems but the robustness or reliability of an
algorithm in obtaining an optimal solution for a wide range of problems can be tested
experimentally by applying it to many test problems and from there a general comment can be
drawn;
(ii) a simple change in the starting point or initial step can have an impact on the search
trajectory in minimizing an objective function [3];
(iii) when possible, a transformation of coordinates can be effected, thus spherising the contours
of the objective functions and thus pointing the search direction toward the minimum. The
algorithm failed for test Problem (5.2) but the result of the transformation suggested
demonstrates that something can, at times, be done to get out of circumstances impeding the
success of an algorithm on a test problem or a class. of problems. The failure of the algorithm on
test Problem (5.2) can be ascribed to the fact that it is a quartic function with a Hessian matrix of
rank 2. The function cannot therefore be approximated by a quadratic in the vicinity of the
minimum as quasi-Newton method is understood to contrive. When transformed, the Hessian is
reduced to a diagonal matrix with positive eigenvalues. However, the algorithm went through 30
iterations to bring the function under minimization to an accuracy of order 10-8 when gT g is of
order 10-7. The stopping criterion is Ilgll < 10-6. This is an improvement over Sisser's result [1]
reported as 10-5 for the same stopping criterion when the termination criterion is changed, the
algorithm went through 158 iterations to bring
MINIMIZING FACTORABLE FUNCTIONS
29
the function value to its minimum.
One expects that after transformation the Rosenbrock's banana-shaped valley function would
attain its minimum in much fewer iterations than (5.2) but geometrically the function represents
a deep parabolic valley thereby necessitating more frequent changes of search directions;
(iv) an algorithm can be rendered impotent by the way a problem is posed. Problems (5.3) and
(5.5) demonstrate this. A total failure is met with transformed version of Gottfried function.
Three different starting values are considered for Problem (5.7). The third starting point requires
only six iterations to take the function to the minimum and with a better accuracy than the first
two. For Problem (5.8) the accuracy is improved from the" order of 10-7 to 10-17 in twelve
iterations.
An unusual thing happened during the minimization of the power function as a result of
Syivesters theorem that is coded into the programme to test for positive definiteness. There is a
sudden change in the column indicating "positive definite" or "not positive definite". At the
42nd iteration, there is an indication
of "not positive definite" after the minimum has been attained to the order of
10-27 in 40 iterations when gT 9 is zero.
.
Since some of the test problems are nonlinear functions that can be transformed into
quadratic or an approximation to quadratic by changes of variables, it is therefore evident that
conjugate gradient method can be employed to such cases. This will be the subject of the next
paper.
30
M.A. IBIEJUGBA et al.
TABLE 1
MbdmbaUon of Rosenbrock's banana-shaped valley function (untransformed)
Iterative
Step
Function Value
0
2
4
6
8
24.2
4.10704765
2.75739953
1. 74224204
1.01702471
5.4 X 10-1
2.2 x 10-1
6.60 X 10-2
1.0 x 10-2
2.46 x 10-<1
2.57 x 10-0
8.08 x 10-16
0.0
10
12
14
16
18
20
21
*22
,T,
Xl
54227.3601
-1.2
911.214472
-9.2 x 10-1
364.647807
-5.7 10-1
127.668726
-2.4 x 10-1
43.7392975
-4.78 x 10-2
18.4492845
2.9 x 10-1
13.6092977
5.5 x 10-1
6.8860487
7.6 x 10-1
1.38608476
9.05 x 10-1
3.84 X 10-2
9.85 X 10-1
4.66 X 10-7
9.9 X 10-1
1.02 X 10-12
9.99 X 10-1
0.0
1.0
* Convergence in 22 iterations.
Iterative
Step Function Value
X2
1.0
7.9 X 10-1
2.6 X 10-1
1.3 X 10-2
-3.09 X 10-2
6.5 X 10-2
2.9 X 10-1
5.6 X 10-1
8.2 X 10-1
9.7 X 10-1
9.9 X 10-1
9.99 X 10-1
1.0
TABLE 2 Minbnization of
Sisser's function
gTg
Xl
X2
0
2.9803
143.192144
1
6.1
2
1.16 x 10-1
1.10363149
4.4 x 10-1
4.4 X 10-2
4
4.5 X 10-3
8.5 X 103
1.9 X 10-1
1.9 X 10-2
1.77 X 10-<1
6
6.6 X 105
8.77x 10-2
8.77 x 10-3
10 2.69 X 10-7
3.89 X 10-9
1.7 X 10-2
1.7 X 10-3
12 1.05 x 10-8
3.0 X 10-11
7.7 X 103
7.7 X 10-4
14 4.1 x 10-10
2.3 X 10-13
3.4 X 10-3
3.4 X 104
16 1.6 x 10-11
1.78 x 10-15
1.5 x 10-3
1.5 X 10-4
18 6.2 x 10-13
1.37 X 10-17
6.7 X 10-4
6.7 X 10-5
.1.06 X 10-10
3.0 X 104
3.0 X 10-5
2.4 X 10-1<1
20
22
9.5 x 10-16
8.2 X 1022
1.3 X 104
1.3 X 10-5
24
3.7x 10-17
6.3 x 10-24
5.9 X 10-5
5.9 X 10-6
26
1.4 x 10-18
4.85 X 10-26
2.6 X iO-5
2.6 X 106
28
5.6 x 10-20
3.7 X 10-28
1.17 x 10-5
2.6 x 10-6
30
2.2 X 10-21
2.88 X 10-30
5.2 X 10-6
5.2 X 10-7
32
8.6 x 10-23
2.2 X 10-32
2.3 X 10-6
2.3 X 10-7
34
3.3 x 10-2<1
1.7 x 10-34
1.0 x 10-6
1.0 X 10-7
. Convergence in 40 iterations.
36
1.3 x 10-25
g - gradient
1.3 X 10-36
4.6 X 10-7
4.6 X 10-8
gT - transpose of the gradient
vector.
38
5.1 x 10-27
(
1.0 X 10-38
2.0 X 10-7
2.0 X 10-8
.
40
1.99 X 10-28
0.0
9.0 X 10-8
9.0 x 10-0
42
7.78 x 10-30
0.0
4.0 X 10-8
4.0 x 10-0
44
3.0 x 10-31
0.0
1.8 X 10-8
1.8 x 10-0
46
1.18 x 1032
0.0
7.94 x 10-0
7.94 X 10-10
48
4.6 x 10-3<1
0.0
3.5 x 10-0
3.5 X 10-10
50
1.8 X 10-35.
0.0
1.6 X 10-9
1.6 X 10-10
31
MINIMIZING FACTORABLE FUNCTIONS
Iterative
Step
0
1
2
3
4
6
6
7
8
9
10
11
12
TABLE 3
Mlnlmis.don 01 Roaenbrock 'a CUB' Function
Function Value
1.0009
4.2 X 10-1
2'.6 X 10-1
2.1 X 10-1
2.0 X 10-1
2.0 X 10-1
2.0 X 10-1
2.0 x 10':"1
2.0 X 10-1
2.0 X 10-1
2.0 X 10-1
2.0 X 10-1
2.0 x 10-1
gTg
1.9999984
1.98 x 10°
1.43 x 10°
3.1 x 10- 1
5.2 X 10-2
1.18 X 10-'1.
2.8 X 10-3
6.9 x 10-.
1.7 x 10-.
4.3 x 1O1.1 x 1O2.6 X 10-6
9.0 X 10-13
XI
0
0
0
0
0
0
0
0
0
0
0
0
0
X2
1
4.2 X 10-1
2.4 X 10-1
1.75 x 10-1
1.6 X 10-1
1.5 x 10-1
1.5 X 10-1
1.5 X 10-1
1.6 X 10-1
1.5 x 10-1
1.49 X 10-1
1.49 X 10-1
1.49 X 10-1
TABLE 4
.._- -. - . --. ---..--
..
Iterative
Step
0
KEYS:
Function Value
7.11111111
gT,
707 .960618
XI
1
X2
1
3
2.8 x 10-1
6.46642091
2.9 x 10-1
2.9 X 10-1
6
2.14 X 10-3
3.69 X 10-3
8.77 X 10-'1.
.
9
1.66 X 10-6
2.49 X 10-6
2.6 X 10-'1.
2.6 X 10-'1.
12
1.27 X 10-'7
1.69 X 10-11
7.7 x 10-3
7.7' x 103
16
9.79 x 10-10
1.14 X 10-1'1.
2.28 X 10-3
2.28 X 10-3
+ve
7.6 x P08itive
10-1'1. Definite
7.7 X 10-16
18 DEF
6.76 x 10.
NOT +ve DEF Heaaian Not Positive Definite.
6.76 x 10-.
21
5.8 x 10-1.
6.2 X 10-111
2.0 x 10-.
6..76 x 10-.
24
4.48 x 10-16
3.54 x 10-'1.'1.
5.9 X 10-6
5.9 X 10-6
.
27
3.46 x 10-18
2.39 x 10-'1.6
1.76 X l(J6
1.76 X 10-5
.
30
2.66 X 10-'1.0
1.6 x 10-'1.8
5. X 10-6
5.2 X 10-6
S3
2.05 X 10-'1.2
1.09 x 10-31
1.5 X 10-6
1.6 X 10-6
S6
1.58 X 10-2.
7.4 X 10-36
4.6 X 10-'7
4.6 X 10-'7
39
1.2 x 10-26
5.0 X 10-38
1.3 X 10-'7
1.3 X 10-'7
42
1.27 x 10-28
0.0
4.3 x 10-8
4.3 x 10-8
-
-
Remarks
+ve DEF
.
.
.
.
.
.
H
.
.
.
NOT +ve DEF.
M. A. IBIEJUGBA et al.
32
TABLE 5
c
.'h 5i
om parlSon 01 our resuns wu_- on-g'1"g
Test Problem
Author
Number of Iterations
6.1
S
6'x.10 .-,;,
32
I
0.0
22
S
5.2 x 10 .",
31
I
4.0 x 10 ."
15
S
2 x 10 .,
I
1.7 x 10 .,,,
S
1.0 x 10 ."
I
1.3 (F)
6.5
S
2 x 10 '4'>
I
1.0 (F)
6.6
S
2 x 10 .-,;,
I
0.0
1
6.7
S
F
F
I
9 x 10
6.8
S
4 x 10 ." (F)
6.2
6.3
6.4
141
3
32
F
12
F
13
12
13
F
I
."
S - Sisser; I - Ibiejugba
et1.5
al.;xF10-Algorithm
failed
6.9
S
9 x 10 .;>u
6.10
12
9
I
5.5 x 10 "
12
S
-TABLE 6
22
I
0.0 valley function (transformed)
Minimisation of Rosenbrock's banana-shaped
39
Iterative!
0
1
3
6
9
12
15
18
21
24
27
30
33
36
39
42
45
48
51
(
Step I Function Value
24.2
6.05
3.78 x 10-1
5.90 x 10-3
9.2 X 10-5
1.44 x 10-6
2.25 x 10-8
3.5 x 10-10
5.5 x 10-12
8.6 x 10-14
1.3 x 10-15
2.09 X 10-17
3.3xlO-19
5.1 x 10-21
8.0 x 10-23
1.25 x 10-24
1.9 x 10-26
3.0 x 10-28
4.7 x 10-30
gTg
7763.36
1940.84
121-.3025
1.89535156
2.96 X 10-2
4.60 X 10-4
7.2 X 10-6
1.13 X 10-7
1.76 X 10-9
2.76 X 10-11
4.3 X 10-13
6.7 x 10-15
1.05 x 10-16
1.6 X 10-18
2.57 X 10-20
4.0 X 10-22
6.27 X 10-24
9.79 X 10-26
1.53 X 10-27
Xl
2.2
1.1
2.75 x 10-1
3.4 x 10-2
4.29 X 10-3
5.37 x.1O-4
6.7 X 10-5
8.39 X 10-6
1.05 X 10-6
1.3 X 1C,-7
1.64 X 10-8
2.0 X 10-9
2.6 X 10-10
3.2 X 10-11
4.0 X 10-12
5.0xlO-13
6.25 X 10-14
7.8 X 10-15
9.7xlO-16
X2
-0.44
-0.22
-0.055
-6.9 X 10-3
-8.59 X 10-4
-1.07 x 10-4
-1.34 X 10-5
-1.67 X 10-6
-2.09 X 10-7
.-2.6 X 10-8
-3.28 x 10-0
-4.09 X 10-10 I
-5.1 X 10-11
-6.4 X 10-12
-8.0 X 10-13
-1.0 X 10-13
-1.25 X 10-14
-1.56 X 10-15
-1.95 X 10-16
MINIMIZING FACTORABLE FUNCTIONS
Iterative
Step
0
6
12
18
24
30
36
42
48
54
60
66
72
78
84
90
96
102
108
114
120
126
132
138
144
150
156
162
Function
Value
33
TABLE 7
MiniInization of Powell's quartic function (transformed)
g7.'g
Xl
X2
X3
X4
215
5.026035
6700
21.4811359
-7
-2.2 x 10-0
-1
6.8 X 10-2
1
3.15 X 10-2
4
-5.3 X 10-2
9.89 X 10-2
7.13 X 10-1
-2.97 X 10-1
-3.7 X 10-3
4.2 X 10-2
2.7 x 10-3
6.4 x 10-5
1.6 X 10-6
4.2 x 10-8
1.8 x 10-0
5.08 x 10-11
9.9 x 10-13
2.7 x 10-14
6.15 x 10-:-16
1.56 x 10-17
6.47 x 10-19
1.8 x 10-20
3.6 X 10-22
9.9 x 10-24
2.3 x 10-25
5.9 x 10-27
2.3 x 10-28
6.6 x 10-30
1.3 x 10-31
3.6 x 10-33
8.5 x 10-35
2.2 x 10-36
9.96 x 10-38
9.3 x 10-30
0.0
1.7 X 10-2
3.1 x 10-4
2.4 x 10-5
5.1 X 10-7
7.6 x 10-0
2.1 X 1010
6.6 X 10-12
1.6 X 10-13
9.87 X 10-15
2.09 X 10-16
2.77 X 10-18
7.7 X 10-20
2.6 X 10-21
6.2 X 10-23
4.1 X 10-24
8.6 X 10-26
1.0 X 10-27
2.78 X 10-20
1.0 X 10-30
2.39 X 10-32
4.2 X 10-34
3.5 X 10-35
6.8 X 10-37
1.55 X 10-37
8.9 x 10-38
-4.9 X 10-2
-7.8 X 10-3
-1.06 X 10-3
-1.77 x 10-4
-4.2 x 10-5
-7.0 X 10-6
-9.5 X 10-7
-1.6 X 10-7
-2.0 X 10-8
-3.4 x 10-0
-7.9 X 10-10
-1.3 X 10-10
-1.8 X 10-11
-3.0 X 10-12
-3.8 X 10-13
-6.4 x 10- I'll
-1.5 x 10-14
-2.5 X 10-15
-3.4 X 10-16
-5.7 X 10-17
-8.9 X 1Qt8
-1.2 X 10-18
-3.16 X 10-19
-9.6 X 10-20
-2.3 X 10-20
-5.57 x 10-5
-4.5 x 10-5
2.47 X 10-6
3.69 X 10-7
5.0 X 10-9
7.5 X 10-10
-4.1 X 10-11
-6.1 X 10-12
1.8 X 10-13
2.7 X 10-14
3.7 X 10-16
5.5 X 10-17
-2.9 X 10-18
-4.5 X 10-19
1.3 X 10-20
1.9 X 10-21
2.7 X 10-23
4.0 X 10-24
-2.2 X 10-25
-3.3 X 10-26
-2.7 X 10-27
1.45 xlO-28
4.46 X 10-30
4.3 X 10-31
-1.6 X 10-32
2.
2.9 x.lO4.2 X 10-3
-3.9 x 10-4
2.2 x 10-4
3.1 x 10-5
-1.1 X 10-6
-1.5 X 10-6
8.6 X 10-8
1.2 X 10-8
4.5 x 10-0
6.4 X 10-10
-2.2 X 10-11
-3.1 X 10-12
1.7 X 10-12
2.5 X 10-13
9.3 X 10-14
1.3 X 10-14
-4.6 X 10-16
-6.5 X 10-17
3.6 X 10-17
5.14 X 10-18
-4.8 X 10-19
2.7 X 10-19
2.6 X 1O-2C1
-1.7 X 10-20
1.49 X 10-20
. 7.1 X 10-3
1.12 X 10-3
1.5 x 10-4
2.5 x 10-5
5.97 X 10-6
1.0 X 10-6
1.35 X 10-7
2.28 X 10-8
2.86 X 10-9
4.8 X 10-10
1.1 x 1O-1C1
1.9 X 10-11
2.6xlO-12
4.3 X 10-13
5.4 X 10-14
9.1 X 10-15
2.15 X 10-15
3.6 X 10-16
4.86 X 10-17
8.18 X 10-18
1.28 X 10-18
1.7 x 10-19
4.5 X 10-20
1.4 X 1O-2C1
3.3 X 1021
TABLE 8
Minimization of Hyperboia-circle function (transformed)
Function Value
g'g
Xl
X2
0
10
244
2
1
3
2.1 X 10-1
3.15 x 100
2.5
2.3
6
3.06 x 10-3
7.3 X 10-2
2.9
2.4
Iterative Step
9
4.7 x 10-5
1.19 X 10-3
2.99
2.447
12
7.38 x 10-7
1.87 x 10-5
2.999
2.449
15
1.15 x 10-8
2.9 X 10-7
2.9999
2.4494
18
1.18 x 10-10
4.6 X 10-9
2.99998
2.44948
21
2.8 x 10-12
7.15 X lOll
2.999984
2.449489
24
4.4 x 10-14
1.12 X 10-12
2.9999998
2.4494896
27
7.11 x 10-16
2.06 X 10-14
2.99999998
2.4494897
30
7.8 x 10-18
3.1 X 10-17
3
I
31
8.67 x 10-19
3.46 X 10-18
3
2.44948974
.
Convergence in 31 iterations.
2.44948974
.
i
- ~---
34
M.A. IBIEJUGBA et al.
TABLE 9
Minimisation of transformed Slaser's funed
Iterative
Step
0
* 1
2
Iterative
Step
0
1
2
3
4
5
6
7
8
9
10"
11
12
..
* Starting
Xl
u..
X2
2.98
40.004
1
0.01
3.57 X 1O-2
4.76 X 10-21
0
-1.09 X 10-11
0
0
0
0
* There is convergence in one iteration.
Function Value
5.78992996
2.06310453
1.23835341
1.18578523
1.17372824
1.16980327
1.1698031
1.16980306
1.1698036
1.1698036
1.16980306
1.16980306
1.16980306
TABLE 10 Minimisation of
Gottfried function
g'g
2295.81781
258.38057
25.45483
7.42725042
1.94669061
2.84660 x lO-.
7.25414 x 10-f>
1.85 x 10-5 0.603005063
4.71998 x lO-6
1.20479 x 10-6
9.25225 x 10-7
9.25225 x 10-7
9.25225 x 10-7
Xl
X2
0.5*
0.516210901
0.544568287
0.567457805
0.584203534
0.602809815
0.602938802
1.04023337
0.603039116
0.603056619
0.603058869
0.603058869
0.603058869
0.5*
0.71657717
0.895810376
0.958387374
0.997670211
1.03983364
1.04009894
1.04030151
1.04033607
LO034045
1.04034045
1.04034045
point given.
Convergence in 10 iterations.
Iterative
Step
0
1
2
3
4
5
6
7
8
9
..
g"-g
Function Value
TABLE 11 Minimisation of
Gottfried function
Function
Value
1.09072144
1.16985437
1.1698154
1.16980601
1.16980376
1.16980322
1.1698031
1.16980306
1.16980306
1.16980306
10 1.16980306
11 1.16980306
* 12 1.16980306
9:9
Xl
X2
4.2523169
5.94087 x 10-3
1.46699 x 10-3
3.62374 x 10-4
8.95341 x 10-5
2.21258 x 10-"1>
5.46878 x 10-6
1.35197 x 10-6
3.34306 x 10-7
1.20414 x 10-10
9.33451 x 10-14
2.01716 x 10-16
1.48205 x 10-17
0.5**
0.604701673
6.0 X lO-1
6.0 X 10-1
6.0 X lO-1
6.0 X lO-1
6.0 X lO-1
6.0 X 10-1
6.0 X 10-1
6.03075573
6.0307515
6.03075138
6.03075138
1**
1.03961707
1.0
1.0
1.0
1.0
1.0
1.04036848
1.040368
1.04037207
1.04037165
1.04037
1.04037164
* Convergence in l~ iterations.
Starting point changed: improved accuracy.
35
MINlw.:3ING FACTORABLE FUNCTIONS
TABLE 12
.Jmparison ~f results with transformed problems versus original problems
Test problem
6.1a
6.2a
6-3a
6.5a
6.6a
T
0
T
0
T
0
T
0
T
Min. Fund. Value
0.0
0.0
0.0
F
1.2 x 10 -..
2.0
0.0
0.0
0.0
g.1"g
0.0
0.0
2.7 x 10 ..>1
F
4.9 x 10.
1.7 x 10 -,,,
0.0
0.0
0.0
No. of Iterations
52
22
158
F
4(F)
.2
1
2
1
0
1.4 x 10'
0.0
1
T
1.9 x 10 -,
2 x 10 .,
2.3 x 10.w
2.7 x 10 .,,,
26
24
6.7a
_.
2 x 10-1
9 X 10-13
2xl0-10
1 X 10-10
6
T
F
1.17
F
9.2 x 10
F
10
0
1.17
1.5 x 10-17
12
0
.
6.8a
T - Tn..::::>rmed Problem;
..
0-
12
Orical Problem;
T 8.7 x 10 .1"
3.5 x 10 .1"
31
5.5 x 10 .'1
12
F - Th, ."orithm failed. 6.9a 0 1.3 x 10'0
T
0.0
0.0
. Chao..:>f starting point improves both minimum function value and gT2g.
6.lOa 0gT g.0.0
Ch"-=e of starting point improves
T 1.26 x 10 -....
Refe:-'!1ces
6.l1a 0
2.4 x 10-"0
0.0
0.0
0.0
..
40
17
40
1. _? McCormick, (1974): A minimanual for the lJ6e of the SUMT Computer Program :;d the
Factorable Programming Language, Standford University, Systems Optimization _:Joratory,
Technical Report No. SOL - 74 - 15, 1974.
2. - .. Sisser, (1982): A modified Newton'lS method for minimizing factorable functionlS, Jour
:.. of Optimization Theory and Applications, Vol. 38, No.4, Dec. 1982, pp. 461
.~
3. - .-'1. David, (1972) Applied nonlinear programming, McGrawhill Book Company, New
-. ~k, 1972.
4. - 3. Gerald, (1980): Applied numerical analYlSilS, Addison-Wesley Publishing Company,
::,ading, Massachusetts, 1980.
5. . -::;. Singh and A. Titli, (1978) SYlSterru optimization and decompolSition, Pergamon Press,
- 78.
6. : Polak, (1971): Computational methods in optimization, Academic Press, New York,
-71.
36
M.A. IBIEJUGBA et al.
7. D. RU88ell, (1970): Optimization theorl/, W.A. Benjamin, Inc., New York, 1970.
8. P.E. Gill and W. Murray, (1978): Algorithml lor the .olution of lea.t .quaru probleml,
SIAM Journal of Numerical Analysis, Vol. 15, No.5, Oct. 1978, pp. 977 - 991.
9. J.C. Nash, (1979): Compact numerical metholblor compute", linear algebra and function
minimization, Adam Hilger Limited, Bristol, 1979.
10. P.E. Gill and W. Murray, (1974): Newton tl/pe method. lor unco"'trained linearly con
.tructed optimization, Mathematical Programming, Vol. 7, 1974, pp. 311 - 350.
11. A.Vi. Fiacco and G.P. McCormick, (1968): Nonlinear programming: sequential uncon
.trained minimization technique., John Wiley and Sons, New York, 1968.
12. G. Emami and G.P. McCormick, (1978): U.e 01 a .table generalized inver3e algorithm to evaluate
Newton method "rategie., The George Washington University, Institute for Management Science
and Engineering, Program in Logistics, Technical Report Serial No. T-384, 1978.
13. M. Woodbury,(1950): Inverting modified matrice., Princeton University, Princeton, New
Jersey, Statistical Research Group, Memorandum No. 42, 1950.
14. J.B. Marion, (1970): Cl~.ical dl/namic. of particle. and .y&tem8, Academic Press, New
York, 1970.
15. J. Sherman and W.J. Morrison, (1949): Adjwtment of an inverse matrix corresponding to
change. in the element, of a given column or a given row of the original matrix, Annals of
Mathematical Statistics, Vol. 20, pp. 621, 1949.
M.A. IBIEJUGBA, T.A. ADEWALE
Department of Mathematics,
University of Ilorin,
Borin, Nigeria
and
O.M. BAMIGBOLA,