DAMTP 2004/NA08
The NEWUOA software for unconstrained
optimization without derivatives1
M.J.D. Powell
Abstract: The NEWUOA software seeks the least value of a function F (x),
x ∈ Rn , when F (x) can be calculated for any vector of variables x. The algorithm
is iterative, a quadratic model Q ≈ F being required at the beginning of each
iteration, which is used in a trust region procedure for adjusting the variables.
When Q is revised, the new Q interpolates F at m points, the value m = 2n+1 being
recommended. The remaining freedom in the new Q is taken up by minimizing the
Frobenius norm of the change to ∇2 Q. Only one interpolation point is altered on
each iteration. Thus, except for occasional origin shifts, the amount of work per
iteration is only of order (m+n)2 , which allows n to be quite large. Many questions
were addressed during the development of NEWUOA, for the achievement of good
accuracy and robustness. They include the choice of the initial quadratic model,
the need to maintain enough linear independence in the interpolation conditions
in the presence of computer rounding errors, and the stability of the updating
of certain matrices that allow the fast revision of Q. Details are given of the
techniques that answer all the questions that occurred. The software was tried
on several test problems. Numerical results for nine of them are reported and
discussed, in order to demonstrate the performance of the software for up to 160
variables.
Department of Applied Mathematics and Theoretical Physics,
Centre for Mathematical Sciences,
Wilberforce Road,
Cambridge CB3 0WA,
England.
November, 2004.
1
Presented at The 40th Workshop on Large Scale Nonlinear Optimization (Erice, Italy, 2004).
1. Introduction
Quadratic approximations to the objective function are highly useful for obtaining
a fast rate of convergence in iterative algorithms for unconstrained optimization,
because usually some attention has to be given to the curvature of the objective
function. On the other hand, each quadratic model has 12 (n+1)(n+2) independent
parameters, and this number of calculations of values of the objective function
is prohibitively expensive in many applications with large n. Therefore the new
algorithm tries to construct suitable quadratic models from fewer data. The model
Q(x), x ∈ Rn , at the beginning of a typical iteration, has to satisfy only m
interpolation conditions
Q(xi ) = F (xi ),
i = 1, 2, . . . , m,
(1.1)
where F (x), x ∈ Rn , is the objective function, where the number m is prescribed
by the user, and where the positions of the different points xi , i = 1, 2, . . . , m,
are generated automatically. We require m ≥ n+2, in order that the equations
(1.1) always provide some conditions on the second derivative matrix ∇2 Q, and
we require m ≤ 21 (n+1)(n+2), because otherwise no quadratic model Q can satisfy
all the equations (1.1) for general right hand sides. The numerical results in the
last section of this paper give excellent support for the choice m = 2n+1.
The success of the new algorithm is due to a technique that is suggested by
the symmetric Broyden method for updating ∇2 Q when first derivatives of F are
available (see pages 195–198 of Dennis and Schnabel, 1983, for instance). Let an
old model Qold be present, and let the new model Qnew be required to satisfy some
conditions that are compatible and that leave some freedom in the parameters of
Qnew . The technique takes up this freedom by minimizing k∇2 Qnew −∇2 Qold kF ,
where the subscript “F ” denotes the Frobenius norm
kAkF =
n
n X
nX
A2ij
i=1 j=1
o1/2
,
A ∈ Rn×n .
(1.2)
Our conditions on the new model Q = Qnew are the interpolation equations (1.1).
Thus ∇2 Qnew is defined uniquely, and Qnew itself is also unique, because the
automatic choice of the points xi excludes the possibility that a nonzero linear
polynomial p(x), x ∈ Rn , has the property p(xi ) = 0, i = 1, 2, . . . , m. In other
words, the algorithm ensures that the rows of the (n+1)×m matrix
X =
1
1
···
1
x1 −x0 x2 −x0 · · · xm −x0
!
(1.3)
are linearly independent, where x0 is any fixed vector.
The strength of this updating technique can be explained by considering the
case when the objective function F is quadratic. Guided by the model Q = Qold at
the beginning of the current iteration, a new vector of variables xnew = xopt +d is
chosen, where xopt is such that F (xopt ) is the least calculated value of F so far. If
2
the error |F (xnew )−Qold (xnew )| is relatively small, then the model has done well in
predicting the new value of F , even if the errors of the approximation ∇2 Q ≈ ∇2 F
are substantial. On the other hand, if |F (xnew ) − Qold (xnew )| is relatively large,
then, by satisfying Qnew (xnew ) = F (xnew ), the updating technique should improve
the accuracy of the model significantly, which is a win/win situation. Numerical
results show that these welcome alternatives provide excellent convergence in the
vectors of variables that are generated by the algorithm, although usually the
second derivative error k∇2 Q − ∇2 F kF is big for every Q that occurs. Thus
the algorithm seems to achieve automatically the features of the quadratic model
that give suitable changes to the variables, without paying much attention to other
features of the approximation Q ≈ F . This suggestion is made with hindsight, after
discovering experimentally that the number of calculations of F is only O(n) in
many cases that allow n to be varied. Further discussion of the efficiency of the
updating technique can be found in Powell (2004b).
The first discovery of this kind, made in January 2002, is mentioned in Powell
(2003). Specifically, by employing the least Frobenius norm updating method,
an unconstrained minimization problem with 160 variables was solved to high
accuracy, using only 9688 values of F , although quadratic models have 13122 independent parameters in the case n = 160. Then the author began to develop
a Fortran implementation of the new procedure for general use, but that task
was not completed until December, 2003, because, throughout the first 18 months
of the development, computer rounding errors caused unacceptable loss of accuracy in a few difficult test problems. A progress report on that work, with some
highly promising numerical results, was presented at the conference in Hangzhou,
China, that celebrated the tenth anniversary of the journal Optimization Methods
and Software (Powell, 2004b). The author resisted pressure from the editor and
referees of that paper to include a detailed description of the algorithm that calculated the given results, because of the occasional numerical instabilities. The
loss of accuracy occurred in the part of the Fortran software that derives Qnew
from Qold in only O(m2 ) computer operations, the change to Q being defined by
an (m+n+1)×(m+n+1) system of linear equations. Let W be the matrix of this
system. The inverse matrix H = W −1 was stored and updated explicitly. In theory
the rank of Ω, which is the leading m×m submatrix of H, is only m−n−1, but this
property was lost in practice. Now, however, a factorization of Ω is stored instead
of Ω itself, which gives the correct rank in a way that is not damaged by computer
rounding errors. This device corrected the unacceptable loss of accuracy (Powell,
2004c), and then the remaining development of the final version of NEWUOA
became straightforward. The purpose of the present paper is to provide details
and some numerical results of the new algorithm.
An outline of the method of NEWUOA is given in Section 2, but m (the
number of interpolation conditions) and the way of updating Q are not mentioned,
so most of the outline applies also to the UOBYQA software of the author (Powell,
2002), where each quadratic model is defined by interpolation to 21 (n+1)(n+2)
values of F . The selection of the initial interpolation points and the construction
3
of the first quadratic model are described in Section 3, with formulae for the initial
matrix H and the factorization of Ω, as introduced in the previous paragraph. Not
only Q but also H and the factorization of Ω are updated when the positions of
the interpolation points are revised, which is the subject of Section 4. On most
iterations, the change in variables d is an approximate solution to the trust region
subproblem
Minimize Q(xopt +d) subject to kdk ≤ ∆,
(1.4)
which receives attention in Section 5, the parameter ∆ > 0 being available with Q.
Section 6 addresses an alternative way of choosing d, which may be invoked when
trust region steps fail to yield good reductions in F . Other details of the algorithm
are considered in Section 7, including shifts of the origin of Rn , which are necessary
to avoid huge losses of accuracy when H is revised. Several numerical results are
presented and discussed in Section 8. The first of these experiments suggests a
modification to the procedure for updating the quadratic model, which was made
to NEWUOA before the calculation of the other results. It seems that the new
algorithm is suitable for a wide range of unconstrained minimization calculations.
Proofs of some of the assertions of Section 3 are given in an appendix.
2. An outline of the method
The user of the NEWUOA software has to define the objective function by a
Fortran subroutine that computes F (x) for any vector of variables x ∈ Rn . An
initial vector x0 ∈ Rn , the number m of interpolation conditions (1.1), and the
initial and final values of a trust region radius, namely ρbeg and ρend , are required
too. It is mentioned in Section 1 that m is a fixed integer from the interval
n+2 ≤ m ≤
1
2
(n+1) (n+2),
(2.1)
and that often the choice m = 2n+1 is good for efficiency. The initial interpolation
points xi , i = 1, 2, . . . , m, include x0 , while the other points have the property
kxi −x0 k∞ = ρbeg , as specified in Section 3. The choice of ρbeg should be such that
the computed values of F at these points provide useful information about the
behaviour of the true objective function near x0 , especially when the computations
may include some spurious contributions that are larger than rounding errors. The
parameter ρend , which has to satisfy ρend ≤ ρbeg , should have the magnitude of the
required accuracy in the final values of the variables.
An outline of the method is given in Figure 1. The details of the operations
of Box 1 are addressed in Section 3. The parameter ρ is a lower bound on the
trust region radius ∆ from the interval [ρend , ρbeg ]. The value of ∆ is revised on
most iterations, but the purpose of ρ is to maintain enough distance between
the interpolation points xi , i = 1, 2, . . . , m, in order to restrict the damage to Q
from the interpolation conditions (1.1) when there are substantial errors in each
computation of F . Therefore ρ is altered only when the constraint ∆ ≥ ρ seems
to be preventing futher reductions in the objective function. Each change to ρ is
4
Pick the initial interpolation points, letting xopt be
an initial point where F is least. Construct the first
BEGIN
quadratic model Q ≈ F . Set ρ = ρbeg and ∆ = ρbeg .
1
-
2?
Subroutine TRSAPP calculates
d by minimizing Q(xopt + d)
approximately, subject to the
bound kdk ≤ ∆. If kdk < ∆,
then CRVMIN is set to the least
curvature of Q that is found.
'
$
Using CRVMIN,
test if three reY cent values of
kdk and F −Q
Y
are “small”.
&
%
N
4
?
Calculate F (xopt + d), and set
3
14
1
kdk ≥ 2 ρ?
N
If MOVE > 0, then Q is mod)−F (x
+d)
F (x
ified by subroutine UPDATE,
RATIO= Q(xopt )−Q(xopt +d) . Thus
opt
opt
so that Q interpolates F at 5 ∆ is revised, subject to ∆ ≥ ρ.
xopt + d instead of at xMOVE .
Then MOVE is set to zero or to
If F (xopt +d) < F (xopt ), then
the index of the interpolation
xopt is overwritten by xopt+d.
point that will be dropped next.
6?
#
xMOVE is going to be replaced by xopt+d,
where d is chosen here by subroutine
Y RATIO
BIGLAG or BIGDEN, in a way that helps
≥ 0.1?
the conditioning of the linear system
" !
N
that defines Q. Set RATIO= 1.
96
#
Y
N
10
max
[kdk,
∆]
≤
ρ
DIST≥ 2∆?
and RATIO≤ 0?
N
"
Y
!
8
15 ?
Reduce ∆ by
a factor of 10
or to its lower
bound ρ. Set
RATIO= −1.
7?
Let xMOVE be the current
interpolation point that
maximizes the distance
DIST= kxMOVE −xopt k.
Reduce ρ by about a
11 ? Termination, after cal
factor of 10 subject to culating F (xopt +d), if END
13
12
ρ = ρend ?
ρ ≥ ρend , and reduce
this has not been done
N
Y
∆ to max[ 21 ρold , ρnew ].
due to kdk < 21 ρ.
Figure 1: An outline of the method, where Y = Yes and N = No
5
a decrease by about a factor of ten, except that the iterations are terminated in
the case ρ = ρend , as shown in Boxes 11-13 of the figure.
Boxes 2-6 of Figure 1 are followed in sequence when the algorithm performs
a trust region iteration that calculates a new value of F . The step d from xopt is
derived from the subproblem (1.4) in Box 2 by the truncated conjugate gradient
procedure of Section 5. If kdk < ∆ occurs here, then Q has positive curvature
along every search direction of that procedure, and CRVMIN is set to the least of
those curvatures, for use in Box 14 when the N branch is taken from Box 3, which
receives attention later. Box 4 is reached in the present case, however, where ∆
is revised in a way that depends on the ratio
RATIO = {F (xopt ) − F (xopt +d)} / {Q(xopt ) − Q(xopt +d)},
(2.2)
as described in Section 7. The other task of Box 4 is to pick the m interpolation
points of the next quadratic model. Usually one of the current points xi , i =
1, 2, . . . , m, is replaced by xopt +d, and all the other points are retained. In this
case the integer MOVE is set in Box 4 to the index of the interpolation point
that is dropped. The only other possibility is no change to the interpolation
equations, and then MOVE is set to zero. Details of the choice of MOVE are also
given in Section 7, the case MOVE > 0 being mandatory when the strict reduction
F (xopt +d) < F (xopt ) is achieved, in order that the best calculated value of F so
far is among the new interpolation conditions. The updating operations of Box
5 are the subject of Section 4. Box 6 branches back to Box 2 for another trust
region iteration if the ratio (2.2) is sufficiently large.
The N branch is taken from Box 6 of the figure when Box 4 has provided a
change F (xopt )−F (xopt +d) in the objective function that compares unfavourably
with the predicted reduction Q(xopt )−Q(xopt +d). Usually this happens because
the positions of the points xi in the interpolation equations (1.1) are unsuitable for
maintaining a good quadratic model, especially when the trust region iterations
have caused some of the distances kxi −xopt k, i = 1, 2, . . . , m, to be much greater
than ∆. Therefore the purpose of Box 7 is to identify the current interpolation
point, xMOVE say, that is furthest from xopt . We take the view that, if kxMOVE−xopt k ≥
2∆ holds, then Q can be improved substantially by replacing the interpolation
condition Q(xMOVE ) = F (xMOVE ) by Q(xopt + d) = F (xopt + d), for some step d that
satisfies kdk ≤ ∆. We see in the figure that the actual choice of d is made in Box
9, details being given in Section 6, because they depend on the updating formulae
of Section 4. Then Box 5 is reached from Box 9, in order to update Q as before,
after the calculation of the new function value F (xopt+d). In this case the branch
from Box 6 to Box 2 is always followed, due to the setting of the artificial value
RATIO= 1 at the end of Box 9. Thus the algorithm makes use immediately of the
new information in the quadratic model.
The N branch is taken from Box 8 when the positions of the current points xi ,
i = 1, 2, . . . , m, are under consideration, and when they have the property
kxi −xopt k < 2∆,
i = 1, 2, . . . , m.
6
(2.3)
Then the tests in Box 10 determine whether the work with the current value of
ρ is complete. We see that the work continues if and only if one or more of the
conditions kdk > ρ, ∆ > ρ or RATIO > 0 holds. Another trust region iteration is
performed with the same ρ in the first two cases, because ρ has not restricted the
most recent choice of d. In the third case, RATIO> 0 implies F (xopt +d) < F (xopt )
in Box 4, and we prefer to retain the old ρ while strict reductions in the objective
function are being obtained. Thus an infinite loop with ρ fixed may happen
in theory. In practice, however, the finite precision of the computer arithmetic
provides an upper bound on the number of different values of F that can occur.
Finally, we consider the operations of Figure 1 when the step d of Box 2
satisfies kdk < 21 ρ. Then Box 14 is reached from Box 3, and often F (xopt+d) is not
going to be calculated, because, as mentioned already, the computed difference
F (xopt )−F (xopt+d) tends to give misleading information about the true objective
function when kdk becomes small. If Box 14 branches to Box 15, a big reduction
is made in ∆ if allowed by ∆ ≥ ρ, and then, beginning at Box 7, there is a choice
as before between replacing the interpolation point xMOVE , or performing a trust
region iteration with the new ∆, or going to Box 11 because the work with the
current ρ is complete. Alternatively, we see that Box 14 can branch directly to
Box 11, the reason being as follows.
Let x̂opt and x̌opt be the first and last values of xopt during all the work with
the current ρ, and let x̂i , i = 1, 2, . . . , m, be the interpolation points at the start of
this part of the computation. When ρ is less than ρbeg , the current ρ was selected
in Box 12, and, because it is much smaller than its previous value, we expect
the points x̂i to satisfy kx̂i − x̂opt k ≥ 2ρ, i 6= opt. On the other hand, because of
Boxes 7 and 8 in the figure, Box 11 can be reached from Box 10 only in the case
kxi−x̌opt k < 2ρ, i = 1, 2, . . . , m. These remarks suggest that at least m−1 new values
of the objective function may be calculated for the current ρ. It is important to
efficiency, however, to include a less laborious route to Box 11, especially when m
is large and ρend is tiny. Details of the tests that pick the Y branch from Box 14
are given in Section 7. They are based on the assumption that there is no need for
further improvements to the model Q, if the differences |F (xopt +d)−Q(xopt +d)|
of recent iterations compare favourably with the current second derivative term
1 2
ρ CRVMIN.
8
When the Y branch is taken from Box 14, we let dold be the vector d that has
satisfied kdk < 21 ρ in Box 3 of the current iteration. Often dold is an excellent step to
take from xopt in the space of the variables, so we wish to allow its use after leaving
Box 11. If Box 2 is reached from Box 11 via Box 12, then d = dold is generated
again, because the quadratic model is the same as before, and the change to ∆ in
Box 12 preserves the property ∆ ≥ 21 ρold > kdold k. Alternatively, if the Y branches
are taken from Boxes 14 and 11, we see in Box 13 that F (xopt+dold ) is computed.
The NEWUOA software returns to the user the first vector of variables that gives
the least of the calculated values of the objective function.
7
3. The initial calculations
We write the quadratic model of the first iteration in the form
Q(x0 +d) = Q(x0 ) + d T ∇Q(x0 ) + 21 d T ∇2 Q d,
d ∈ Rn ,
(3.1)
x0 being the initial vector of variables that is provided by the user. When the
number of interpolation conditions (1.1) satisfies m ≥ 2n+1, the first 2n+1 of the
points xi , i = 1, 2, . . . , m, are chosen to be the vectors
x1 = x0
and
xi+1 = x0 + ρbeg ei
xi+n+1 = x0 − ρbeg ei
)
,
i = 1, 2, . . . , n,
(3.2)
where ρbeg is also provided by the user as mentioned already, and where ei is the
i-th coordinate vector in Rn . Thus Q(x0 ), ∇Q(x0 ) and the diagonal elements
(∇2 Q)ii , i = 1, 2, . . . , n, are given uniquely by the first 2n+1 of the equations (1.1).
Alternatively, when m satisfies n+2 ≤ m ≤ 2n, the initial interpolation points are
the first m of the vectors (3.2). It follows that Q(x0 ), the first m−n−1 components
of ∇Q(x0 ) and (∇2 Q)ii , i = 1, 2, . . . , m−n−1, are defined as before. The other
diagonal elements of ∇2 Q are set to zero, so the other components of ∇Q(x0 ) take
the values {F (x0 +ρbeg ei )−F (x0 )}/ρbeg , m−n ≤ i ≤ n.
In the case m > 2n+1, the initial points xi , i = 1, 2, . . . , m, are chosen so that
the conditions (1.1) also provide 2(m−2n−1) off-diagonal elements of ∇2 Q, the
factor 2 being due to symmetry. Specifically, for i ∈ [2n+2, m], the point xi has
the form
(3.3)
xi = x0 + σp ρbeg ep + σq ρbeg eq ,
where p and q are different integers from [1, n], and where σp and σq are included
in the definitions
σj =
(
−1, F (x0 −ρbeg ej ) < F (x0 +ρbeg ej )
+1, F (x0 −ρbeg ej ) ≥ F (x0 +ρbeg ej ),
j = 1, 2, . . . , n,
(3.4)
which biases the choice (3.3) towards smaller values of the objective function.
Thus the element (∇2 Q)pq = (∇2 Q)qp is given by the equations (1.1), since every
quadratic function Q(x), x ∈ Rn , has the property
n
ρ−2
beg Q(x0 ) − Q(x0 +σp ρbeg ep ) − Q(x0 +σq ρbeg eq )
o
+ Q(x0 +σp ρbeg ep +σq ρbeg eq )
= σp σq (∇2 Q)pq .
(3.5)
For simplicity, we pick p and q in formula (3.3) in the following way. We let j be
the integer part of the quotient (i−n−2)/n, which satisfies j ≥ 1 due to i ≥ 2n+2,
we set p = i−n−1−jn, which is in the interval [1, n], and we let q have the value
p+j or p+j−n, the latter choice being made in the case p+j > n. Hence, if n = 5
and m = 20, for example, there are 9 pairs {p, q}, generated in the order {1, 2},
{2, 3}, {3, 4}, {4, 5}, {5, 1}, {1, 3}, {2, 4}, {3, 5} and {4, 1}. All the off-diagonal
8
elements of ∇2 Q that are not provided by the method of this paragraph are set
to zero, which completes the specification of the initial quadratic model (3.1).
The preliminary work of NEWUOA includes also the setting of the initial
matrix H = W −1 , where W occurs in the linear system of equations that defines
the change to the quadratic model. We recall from Section 1 that, when Q is
updated from Qold to Qnew = Qold+D, say, the quadratic function D is constructed
so that k∇2Dk2F is least subject to the constraints
D(xi ) = F (xi ) − Qold (xi ),
i = 1, 2, . . . , m,
(3.6)
these constraints being equivalent to Qnew (xi ) = F (xi ), i = 1, 2, . . . , m. We see
that the calculation of D is a quadratic programming problem, and we let λj ,
j = 1, 2, . . . , m, be the Lagrange multipliers of its KKT conditions. They have the
properties
m
X
λj = 0
m
X
and
j=1
j=1
λj (xj −x0 ) = 0,
(3.7)
and the second derivative matrix of D takes the form
∇2D =
m
X
λj xj xTj =
j=1
m
X
j=1
λj (xj −x0 ) (xj −x0 )T
(3.8)
(Powell, 2004a), the last part of expression (3.8) being a consequence of the equations (3.7). This form of ∇2D allows D to be the function
D(x) = c + (x−x0 )T g +
1
2
m
X
j=1
λj {(x−x0 )T (xj −x0 )}2 ,
x ∈ Rn ,
(3.9)
and we seek the values of the parameters c ∈ R, g ∈ Rn and λ ∈ Rm . The conditions
(3.6) and (3.7) give the square system of linear equations
A
X
X
0
where A has the elements
Aij =
1
2
T
!


λ
r



 c  = 
0
g


{(xi −x0 )T (xj −x0 )}2 ,
l m
(3.10)
l n+1 ,
1 ≤ i, j ≤ m,
(3.11)
where X is the matrix (1.3), and where r has the components F (xi )−Qold (xi ),
i = 1, 2, . . . , m. Therefore W and H are the matrices
W =
A
XT
X
0
!
and
H = W
−1
=
Ω
ΞT
Ξ
Υ
!
,
(3.12)
say. It is straightforward to derive the elements of W from the vectors xi ,
i = 1, 2, . . . , m, but we require the elements of Ξ and Υ explicitly, with a factorization of Ω. Fortunately, the chosen positions of the initial interpolation points
9
provide convenient formulae for all of these terms, as stated below. Proofs of the
correctness of the formulae are given in the appendix.
The first row of the initial (n+1)×m matrix Ξ has the very simple form
Ξ1j = δ1j ,
j = 1, 2, . . . , m.
(3.13)
Further, for integers i that satisfy 2 ≤ i ≤ min[n+1, m−n], the i-th row of Ξ has
the nonzero elements
Ξii = (2 ρbeg )−1
Ξi i+n = −(2 ρbeg )−1 ,
and
(3.14)
all the other entries being zero, which defines the initial Ξ in the cases m ≥ 2n+1.
Otherwise, when m−n+1 ≤ i ≤ n+1 holds, the i-th row of the initial Ξ also has
just two nonzero elements, namely the values
Ξi1 = −(ρbeg )−1
and
Ξii = (ρbeg )−1 ,
(3.15)
which completes the definition of Ξ for the given interpolation points. Moreover,
the initial (n+1)×(n+1) matrix Υ is amazingly sparse, being identically zero in
the cases m ≥ 2n+1. Otherwise, its only nonzero elements are the last 2n−m+1
diagonal entries, which take the values
Υii = − 21 ρ2beg ,
m−n+1 ≤ i ≤ n+1.
(3.16)
The factorization of Ω, mentioned in Section 1, guarantees that the rank of Ω
is at most m−n−1, by having the form
Ω =
m−n−1
X
sk z k z Tk =
k=1
m−n−1
X
z k z Tk = Z Z T ,
(3.17)
k=1
the second equation being valid because each sk is set to one initially. When
1 ≤ k ≤ min[n, m−n−1], the components of the initial vector z k ∈ Rm , which is
the k-th column of Z, are given the values
√ −2 
√
1
Z1k = − 2 ρ−2
,
Z
=
2 ρbeg , 
k+1
k
beg
2
√
(3.18)
Zk+n+1 k = 21 2 ρ−2
Zjk = 0 otherwise, 
beg ,
so each of these columns has just three nonzero elements. Alternatively, when
m > 2n+1 and n+1 ≤ k ≤ m−n−1, the initial z k depends on the choice (3.3) of xi
in the case i = k+n+1. We let p, q, σp and σq be as before, and we define p̂ and
q̂ by the equations
xp̂ = x0 + σp ρbeg ep
and
xq̂ = x0 + σq ρbeg eq .
(3.19)
It follows from the positions of the interpolation points that p̂ is either p+1 or
p+n+1, while q̂ is either q+1 or q+n+1. Now there are four nonzero elements in
the k-th column of Z, the initial z k being given the components
Zp̂k = Zq̂k = −ρ−2
beg ,
Zjk = 0 otherwise.
Z1k = ρ−2
beg ,
Zk+n+1 k = ρ−2
beg ,
10
)
(3.20)
All the given formulae for the nonzero elements of H = W −1 are applied in only
O(m) operations, due to the convenient choice of the initial interpolation points,
but the work of setting the zero elements of Ξ, Υ and Z is O(m2 ). The description
of the preliminary work of NEWUOA is complete.
4. The updating procedures
In this section we consider the change that is made to the quadratic model Q on
each iteration of NEWUOA that alters the set of interpolation points. We let the
new points have the positions
+
say,
x+
t = xopt + d = x ,
+
xi = xi ,
i ∈ {1, 2, . . . , m}\{t},
)
(4.1)
which agrees with the outline of the method in Figure 1, because now we write
t instead of MOVE. The change D = Qnew −Qold has to satisfy the analogue of the
conditions (3.6) for the new points, and Qold interpolates F at the old interpolation
points. Thus D is the quadratic function that minimizes k∇2DkF subject to the
constraints
n
o
+
+
D(x+
i ) = F (x ) − Qold (x ) δit ,
Let W + and H + be the matrices
+
W =
A+
(X + )T
X+
0
!
+
+ −1
and H = (W )
i = 1, 2, . . . , m.
=
Ω+
(Ξ+ )T
Ξ+
Υ+
(4.2)
!
,
(4.3)
where A+ and X + are defined by replacing the old interpolation points by the new
ones in equations (1.3) and (3.11). It follows from the derivation of the system
(3.10) and from the conditions (4.2) that D is now the function
D(x) = c+ + (x−x0 )T g + +
1
2
m
X
j=1
+
T
2
λ+
j {(x−x0 ) (xj −x0 )} ,
x ∈ Rn ,
(4.4)
the parameters being the components of the vector
λ+
o
n
 c+  = F (x+ ) − Qold (x+ ) H + e ,
t
g+


(4.5)
where et is now in Rm+n+1 . Expressions (4.5) and (4.4) are used by the NEWUOA
software to generate the function D for the updating formula
Qnew (x) = Qold (x) + D(x),
x ∈ Rn .
(4.6)
The matrix H = W −1 is available at the beginning of the current iteration, the
P
submatrices Ξ and Υ being stored explicitly, with the factorization m−n−1
sk z k z Tk
k=1
11
of Ω that has been mentioned, but H + occurs in equation (4.5). Therefore Ξ and
Υ are overwritten by the submatrices Ξ+ and Υ+ of expression (4.3), and also the
new factorization
Ω+ =
m−n−1
X
+ + T
s+
k z k (z k )
(4.7)
k=1
is required. Fortunately, the amount of work of these tasks is only O(m2 ) operations, by taking advantage of the simple change (4.1) to the interpolation points.
Indeed, we deduce from equations (4.1), (1.3), (3.11), (3.12) and (4.3) that all
differences between the elements of W and W + are confined to the t-th row and
column. Thus W + −W is a matrix of rank two, which implies that the rank of
H + −H is also two. Therefore Ξ+ , Υ+ and the factorization (4.7) are constructed
from H by an extension of the Sherman–Morrison formula. Details and some
relevant analysis are given in Powell (2004c), so only a brief outline of these calculations is presented below, before considering the implementation of formula (4.6).
The updating of H in O(m2 ) operations is highly important to the efficiency of
the NEWUOA software, since an ab initio calculation of the change (4.4) to the
quadratic model would require O(m3 ) computer operations.
In theory, H + is the inverse of the matrix W + that has the elements
Wit+ = Wti+ = (W + et )i , i = 1, 2, . . . , m+n+1,
Wij+ = Wij = Hij−1 , otherwise, 1 ≤ i, j ≤ m+n+1.
)
(4.8)
It follows from the right hand sides of this expression that H and the t-th column
of W + provide enough information for the derivation of H + . The definitions (1.3)
and (3.11) show that W + et has the components
T
+
2
Wit+ = 12 {(x+
i −x0 ) (x −x0 )} ,
+
+
+
Wm+1
t = 1 and Wi+m+1 t = (x −x0 )i ,
i = 1, 2, . . . , m
i = 1, 2, . . . , n
)
,
(4.9)
+
the notation x+ being used instead of x+
t , because x = xopt +d is available before
t = MOVE is picked in Box 4 of Figure 1. Of course t must have the property
that W + is nonsingular, which holds if no divisions by zero occur when H + is
calculated. Therefore we employ a formula for H + that gives conveniently the
dependence of H + on t. Let the components of w ∈ Rm+n+1 take the values
wi = 21 {(xi −x0 )T (x+ −x0 )}2 ,
wm+1 = 1 and wi+m+1 = (x+ −x0 )i ,
i = 1, 2, . . . , m
i = 1, 2, . . . , n
)
,
(4.10)
so w is independent of t. Equations (4.1), (4.9) and (4.10) imply that W + et differs
from w only in its t-th component, which allows H + to be written in terms of H,
w and et . Specifically, Powell (2004a) derives the formula
h
H + = H + σ −1 α (et − H w) (et − H w)T − β H et eTt H
n
+ τ H et (et − H w)T + (et − H w) eTt H
12
oi
,
(4.11)
the parameters being the expressions
α = eTt H et ,
τ = eTt H w

β = 12 kx+ − x0 k4 − wT H w, 
and
σ = α β + τ 2.
(4.12)

We see that Hw and β can be calculated before t is chosen, so it is inexpensive
in practice to investigate the dependence of the denominator σ on t, in order
to ensure that |σ| is sufficiently large. The actual selection of t is addressed in
Section 7.
Formula (4.11) was applied by an early version of NEWUOA, before the introduction of the factorization of Ω. The bottom left (n+1)×m and bottom right
(n+1)×(n+1) submatrices of this formula are still used to construct Ξ+ and Υ+
from Ξ and Υ, respectively, the calculation of Hw and Het being straightforward
when the terms sk and z k , k = 1, 2, . . . , m−n−1, of the factorization (3.17) are
stored instead of Ω.
The purpose of the factorization is to reduce the damage from rounding errors
to the identity W = H −1 , which holds in theory at the beginning of each iteration.
It became obvious from numerical experiments, however, that huge errors may
occur in H in practice, including a few negative values of Hii , 1 ≤ i ≤ m, although
Ω should be positive semi-definite. Therefore we consider the updating of H
when H is very different from W −1 , assuming that the calculations of the current
iteration are exact. Then H + is the inverse of the matrix that has the elements
on the right hand side of expression (4.8), which gives the identities
+
+
i = 1, 2, . . . , m+n+1,
and (H + )−1
(H + )−1
ti = Wti ,
it = Wit
−1
+
+ −1
Wij − (H )ij = Wij − Hij , otherwise, 1 ≤ i, j ≤ m+n+1.
)
(4.13)
In other words, the overwriting of W and H by W + and H + makes no difference
to the elements of W − H −1 , except that the t-th row and column of this error
matrix become zero. It follows that, when all the current interpolation points have
been discarded by future iterations, then all the current errors in the first m rows
and columns of W − H −1 will have been annihilated. Equation (4.13) suggests,
however, that any errors in the bottom right (n+1)×(n+1) submatrix of H −1 are
retained. The factorization (3.17) provides the perfect remedy to this situation.
Indeed, if H is any nonsingular (m+n+1)×(m+n+1) matrix of the form (3.12),
and if the rank of the leading m×m submatrix Ω is m−n−1, then the bottom
right (n+1)×(n+1) submatrix of H −1 is zero, which can be proved by expressing
the elements of H −1 as cofactors of H divided by det H (Powell, 2004c). Thus the
very welcome property
+
(H + )−1
ij = Wij = 0,
m+1 ≤ i, j ≤ m+n+1,
(4.14)
is guaranteed by the factorization (4.7), even in the presence of computer rounding
errors.
13
The updating of the factorization of Ω by NEWUOA depends on the fact that
the values
k ∈ K,
(4.15)
s+
and z +
k = sk
k = zk,
are suitable in expression (4.7), where k is in K if and only if the t-th component
of z k is zero. Before taking advantage of this fact, an elementary change is made
if necessary to the terms of the sum
Ω =
m−n−1
X
sk z k z Tk ,
(4.16)
k=1
which forces the number of integers in K to be at least m − n − 3. Specifically,
NEWUOA employs the remark that, when si = sj holds in equation (4.16), then
the equation remains true if z i and z j are replaced by the vectors
cos θ z i + sin θ z j
and
− sin θ z i + cos θ z j ,
(4.17)
respectively, for any θ ∈ [0, 2π]. The choice of θ allows either i or j to be added
to K if both i and j were not in K previously. Thus, because sk = ±1 holds for
each k, only one or two of the new vectors z +
k , k = 1, 2, . . . , m−n−1, have to be
calculated after retaining the values (4.15). When |K| = m−n−2, we let z +
m−n−1
be the required new vector, which is the usual situation as the theoretical positive
definiteness of Ω should exclude negative values of sk . Then the last term of the
new factorization (4.7) is defined by the equations


s+
m−n−1 = sign (σ) sm−n−1
,
−1/2

z+
+
Z
chop
(e
}
=
|σ|
{
τ
z
−H
w)
t
m−n−1
m−n−1
t
m−n−1
(4.18)
where τ , σ and et−H w are taken from the updating formula (4.11), where Zt m−n−1
is the t-th component of z m−n−1 , and where chop (et −H w) is the vector in Rm
whose components are the first m components of et −H w. These assertions and
those of the next paragraph are justified in Powell (2003c).
In the alternative case |K| = m−n−3, we simplify the notation by assuming that
+
only z +
1 and z 2 are not provided by equation (4.15), and that the signs s1 = +1 and
s2 = −1 occur. Then the t-th components of z 1 and z 2 , namely Zt1 and Zt2 , are
+
nonzero. Many choices of the required new vectors z +
1 and z 2 are possible, because
of the freedom that corresponds to the orthogonal rotation (4.17). We make two
of them available to NEWUOA, in order to avoid cancellation. Specifically, if
2
the parameter β of expression (4.12) is nonnegative, we define ζ = τ 2 +β Zt1
, and
NEWUOA applies the formulae
s+
1 = s1 = +1,
s+
2 = sign (σ) s2 = −sign (σ),
−1/2
{ τ z 1 + Zt1 chop (et −H w) },
z+
1 = |ζ|
−1/2
{ −β Zt1 Zt2 z 1 + ζ z 2 + τ Zt2 chop (et −H w) }.
z+
2 = |ζ σ|
14









(4.19)
2
Otherwise, when β < 0, we define ζ = τ 2 −β Zt2
, and NEWUOA sets the values
s+
1 = sign (σ) s1 = sign (σ),
s+
2 = s2 = −1,
−1/2
z+
{ ζ z 1 + β Zt1 Zt2 z 2 + τ Zt1 chop (et −H w) },
1 = |ζ σ|
−1/2
z+
{ τ z 2 + Zt2 chop (et −H w) }.
2 = |ζ |





(4.20)




The technique of the previous paragraph is employed only if at least one of
the signs sk , k = 1, 2, . . . , m−n−1, is negative, and then σ < 0 must have occurred
in equation (4.18) on an earlier iteration, because every sk is set to +1 initially.
Moreover, any failure in the conditions α ≥ 0 and β ≥ 0 is due to computer rounding
errors. Therefore Powell (2004a) suggests that the parameter σ in formula (4.11)
be given the value
σnew = max[0, α] max[0, β] + τ 2 ,
(4.21)
instead of αβ + τ 2 . If the new value is different from before, however, then the
new matrix (4.11) may not satisfy any of the conditions (4.8), except that the
factorizations (4.16) and (4.7) ensure that the bottom right (n+1)×(n+1) submatrices of H −1 and (H + )−1 are zero. Another way of keeping σ positive is to
retain α = eTtHet , τ = eTtHw and σ = αβ +τ 2 from expression (4.12), and to define
β by the formula
h
βnew = max 0,
1
2
i
kx+ − x0 k4 − wTH w .
(4.22)
In this case any change to β alters the element (H + )−1
tt , but every other stability
property (4.13) is preserved, as proved in Lemma 2.3 of Powell (2004c). Therefore
equation (4.21) was abandoned, and the usefulness of the value (4.22) instead of
the definition (4.12) of β was investigated experimentally. Substantial differences
in the numerical results were found only when the damage from rounding errors
was huge, and then the recovery that is provided by all of the conditions (4.13)
is important to efficiency. Therefore the procedures that have been described
already for updating Ξ, Υ and the factorization of Ω are preferred, although in
practice α, β, σ and some of the signs sk may become negative occasionally. Such
errors are usually corrected automatically by a few more iterations of NEWUOA.
Another feature of the storage and updating of H by NEWUOA takes advantage of the remark that, when d is calculated in Box 2 of Figure 1, the constant
term of Q is irrelevant. Moreover, the constant term of Qold is not required in
equation (4.5), because the identities Qold (xopt ) = F (xopt ) and x+ = xopt +d allow
this equation to be written in the form
λ+
n
o
 c+  = [ F (x +d) − F (x ) ] − [ Qold (x +d) − Qold (x ) ] H + e .
opt
opt
opt
opt
t
g+
(4.23)
Therefore NEWUOA does not store the constant term of any quadratic model.
It follows that c+ in expression (4.23) is ignored, which makes the (m+1)-th row


15
of H + unnecessary for the revision of Q by formula (4.6). Equation (4.23) shows
that the (m+1)-th column of H + is also unnecessary, t being in the interval [1, m].
Actually, the (m + 1)-th row and column of every H matrix are suppressed by
NEWUOA, which is equivalent to removing the first row of every submatrix Ξ
and the first row and column of every submatrix Υ, but the other elements of these
submatrices are retained. Usually this device gains some accuracy by diverting
attention from actual values of F and x ∈ Rn to the changes that occur in the
objective function and the variables, as shown on the right hand side of equation
(4.23) for example. The following procedure is used by NEWUOA to update H
without its (m+1)-th row and column.
Let “opt” be the integer in [1, m] such that i = opt gives the best of the
interpolation points xi , i = 1, 2, . . . , m, which agrees with the notation in Sections
1 and 2, and let v ∈ Rm+n+1 have the components
vi = 21 {(xi −x0 )T (xopt −x0 )}2 ,
vm+1 = 1 and vi+m+1 = (xopt −x0 )i ,
i = 1, 2, . . . , m
i = 1, 2, . . . , n
)
.
(4.24)
Therefore v is the opt-th column of the matrix W , so expression (3.12) implies
Hv = eopt in theory, where eopt is the opt-th coordinate vector in Rm+n+1 . Thus
the terms Hw and wTHw of equations (4.11) and (4.12) take the values
H w = H (w − v) + eopt
(4.25)
wT H w = (w − v)TH (w − v) + 2 wT eopt − v T eopt .
(4.26)
and
These formulae allow the parameters (4.12) to be calculated without the (m+1)th row and column of H, because the (m + 1)-th component of w − v is zero.
Similarly, the first m and last n components of Hw are given by formula (4.25),
and these components of Het are known. Thus all the terms of expression (4.11)
are available for generating the required parts of Ξ+ and Υ+ . Moreover, after
constructing chop (et −Hw), the updating of the factorization of Ω is unchanged.
It is proved in Lemma 3 of Powell (2004a) that, when this version of the updating
procedure is applied, and when H has been damaged by rounding errors, then the
new H + enjoys stability properties that are analogous to the conditions (4.13).
We see that the given procedures for updating H require only O(m2 ) computer
operations, which is highly favourable in the recommended case m = 2n+1. On
the other hand, the function (4.4) has the second derivative matrix
∇2D =
m
X
j=1
+
+
T
λ+
j (xj − x0 ) (xj − x0 ) ,
(4.27)
so the calculation of its elements would take O(mn2 ) operations. Therefore
∇2 Qnew is not derived explicitly from formula (4.6). Instead, as suggested at
the end of Section 3 of Powell (2004a), the NEWUOA software employs the forms
∇2 Qold = Γ +
Pm
∇2 Qnew = Γ+ +
j=1
Pm
γj (xj − x0 ) (xj − x0 )T
j=1


,
+
T 
γj+ (x+
j − x0 ) (xj − x0 )
16
(4.28)
overwriting the symmetric matrix Γ and the real coefficients γj , j = 1, 2, . . . , m,
by Γ+ and γj+ , j = 1, 2, . . . , m, respectively. At the beginning of the first iteration,
each γj is set to zero, and we let Γ be the second derivative matrix of the initial
quadratic model, its elements being specified in the first two paragraphs of Section
3. When the change (4.6) is made to the quadratic model, conditions (4.1), (4.27)
and (4.28) allow the choices
Γ+ = Γ + γt (xt − x0 ) (xt −x0 )T ,
and
γt+ = λ+
t 

,
j ∈ {1, 2, . . . , m}\{t} 
γj+ = γj + λ+
j ,
(4.29)
which are included in NEWUOA, because they can be implemented in only O(n2 )
operations. Finally, the gradient of the quadratic model (3.1) is revised by the
formula
(4.30)
∇Qnew (x0 ) = ∇Qold (x0 ) + g + ,
in accordance with expressions (4.4) and (4.6), where g + is taken from equation
(4.23). The description of the updating of Q, without the unnecessary constant
term Q(x0 ), is complete, except that some of the numerical results of Section 8
suggested a recent modification that is described there.
5. The trust region subproblem
We recall from Box 2 of Figure 1 that subroutine TRSAPP generates a step d from
xopt that is an approximate solution of the subproblem
Minimize Q(xopt +d) subject to kdk ≤ ∆.
(5.1)
The method of the subroutine is explained below. Figure 1 shows that the trust
region radius ∆ and the quadratic model Q are available when the subroutine is
called, but, as mentioned at the end of Section 4, the matrix ∇2 Q is stored in the
form
m
∇2 Q = Γ +
X
k=1
γk (xk − x0 ) (xk − x0 )T ,
(5.2)
because it would be too onerous to work with all the elements of ∇2 Q explicitly
when n is large. Expression (5.2) implies the identity
∇2 Q u = Γ u +
m
X
k=1
ηk (xk − x0 ),
(5.3)
where ηk = γk (xk −x0 )Tu, k = 1, 2, . . . , m, and where u is a general vector in Rn .
Thus the product ∇2 Q u can be calculated in O(mn) operations for any choice of
u. Therefore it is suitable to generate d by a version of the truncated conjugate
gradient method (see Conn, Gould and Toint, 2000, for instance).
17
This method produces a piecewise linear path in Rn , starting at xopt = xopt+d0 ,
where d0 = 0. For j ≥ 1, we let xopt +dj be the point on the path at the end of the
j-th line segment. It has the form
xopt + dj = xopt + dj−1 + αj sj ,
j ≥ 1,
(5.4)
where sj is the direction of the line segment and αj is now a steplength. We
do not include any preconditioning, because the norm of the bound kdk ≤ ∆ in
expression (5.1) is Euclidean. Moreover, the path is truncated at xopt + dj−1 if
k∇Q(xopt +dj−1 )k is sufficiently small, if kdj−1 k = ∆ holds, or if some other test
is satisfied, as specified later. The complete path has the property that, if one
moves along it from xopt , then the Euclidean distance from xopt in Rn increases
monotonically.
When the j-th line segment of the path is constructed, its direction is defined
by the formula
sj =
(
−∇Q(xopt ),
−∇Q(xopt +dj−1 ) + βj sj−1 ,
j = 1,
j ≥ 2,
(5.5)
where βj is the ratio k∇Q(xopt +dj−1 )k2/k∇Q(xopt +dj−2 )k2 , this convenient value
being taken from Fletcher and Reeves (1964). Then the steplength αj of equation
(5.4) is chosen to minimize Q(xopt +dj ) subject to αj ≥ 0 and kdj k ≤ ∆ for each j.
Formula (5.5) provides the well-known descent condition
sTj ∇Q(xopt +dj−1 ) = −k∇Q(xopt +dj−1 )k2 < 0,
j ≥ 1,
(5.6)
which depends on the choice of αj−1 when j ≥ 2. It follows from kdj−1 k < ∆ that
αj is positive.
The form (5.3) of the product ∇2 Q u assists the calculation of the gradients
∇Q(xopt + dj ), j ≥ 0, and the steplengths αj , j ≥ 1. The initial vector u is the
difference xopt −x0 , in order to obtain from expression (3.1) the gradient
∇Q(xopt ) = ∇Q(x0 ) + ∇2 Q {xopt − x0 }.
(5.7)
The other choices of u are just all the vectors (5.5) that occur. The availability of
∇Q(xopt +dj−1 ) and ∇2 Q sj allows αj to be found cheaply, because it is the value
of α in the interval [0, α̂j ] that minimizes the function
Q(xopt+ dj−1+ α sj ) = Q(xopt+ dj−1 ) + α sTj ∇Q(xopt+ dj−1 ) + 21 α2 sTj ∇2 Q sj , (5.8)
where α̂j is the positive root of the equation kxopt+dj−1+α̂j sj k = ∆. Therefore we
ask whether Q(xopt +dj−1 +αsj ), 0 ≤ α ≤ α̂j , decreases monotonically. Equations
(5.6) and (5.8) imply that the answer is affirmative in the case
−k∇Q(xopt + dj−1 )k2 + α̂j sTj ∇2 Q sj ≤ 0,
18
(5.9)
and then αj = α̂j is selected. Otherwise, sTj ∇2 Q sj is positive, and the subroutine
picks the value
αj = k∇Q(xopt + dj−1 )k2
.
sTj ∇2 Q sj < α̂j .
(5.10)
After finding αj , the gradient ∇Q(xopt +dj ) is constructed by the formula
∇Q(xopt + dj ) = ∇Q(xopt +dj−1 ) + αj ∇2 Q sj ,
(5.11)
which is derived from the relation (5.4), the product ∇2 Q sj being employed again.
The techniques of this paragraph are applied for each line segment of the path.
The path is truncated at xopt +dj in the case αj = α̂j , because then d = dj is on
the boundary of the trust region kdk ≤ ∆. Moreover, it is truncated at its starting
point xopt + d0 = xopt in the unusual case when the initial gradient ∇Q(xopt ) is
identically zero. Otherwise, we try to truncate the path when the ratio
h
Q(xopt ) − Q(xopt +dj )
i.h
n
oi
Q(xopt ) − min Q(xopt +d) : kdk ≤ ∆
(5.12)
is sufficiently close to one, in order to avoid conjugate gradient iterations that
improve only slightly the reduction in the objective function that is predicted by
the quadratic model. The implementation of this aim is empirical. Specifically,
the iterations are terminated if at least one of the conditions
h
k∇Q(xopt + dj )k ≤ 10−2 k∇Q(xopt )k
Q(xopt + dj−1 ) − Q(xopt + dj )
i
i
h


≤ 10−2 Q(xopt ) − Q(xopt + dj ) 
(5.13)
is satisfied, the change in Q for each line segment being derived from expression
(5.8), and Q(xopt )−Q(xopt +dj ) is the sum of the changes so far. The path is also
truncated if j reaches the theoretical upper bound on the number of iterations,
namely n, but we expect this test to be redundant for n ≥ 10.
Let xopt + dj be the final point of the path. The step d = dj is returned by
subroutine TRSAPP in the case kdj k < ∆, because then there is no interference
with the conjugate gradient iterations from the trust region boundary. Further,
the parameter CRVMIN, introduced in Box 2 of Figure 1, is given the value
n
o
CRVMIN = min sTi ∇2 Q si / ksi k2 : i = 1, 2, . . . , j .
(5.14)
Otherwise, CRVMIN is set to zero, and, because of the possibility that the ratio
(5.12) may be substantially less than one, the following iterative procedure is
applied. It also calculates dj from dj−1 , the initial point xopt +dj−1 being the final
point of the truncated piecewise linear path, so ∇Q(xopt +dj−1 ) is available. The
conditions kdj k = kdj−1 k = ∆ are going to hold on every iteration of the additional
procedure.
At the beginning of an iteration, we decide, using only dj−1 and ∇Q(xopt+dj−1 ),
whether d = dj−1 is acceptable as an approximate solution of the subproblem (5.1).
If dj−1 were the true solution, then, by the KKT conditions of the subproblem,
19
∇Q(xopt+dj−1 ) would be a nonpositive multiple of dj−1 , and we also give attention
to the first of the conditions (5.13). Indeed, subroutine TRSAPP picks d = dj−1 if
one or both of the inequalities
k∇Q(xopt + dj−1 )k ≤ 10−2 k∇Q(xopt )k
T
dj−1
∇Q(xopt + dj−1 ) ≤ −0.99 kdj−1 k k∇Q(xopt + dj−1 )k
)
(5.15)
is achieved. Otherwise, dj−1 and ∇Q(xopt+dj−1 ) span a two dimensional subspace
of Rn , and dj is calculated to be the vector in this subspace that minimizes
Q(xopt +dj ) subject to kdj k = ∆. Therefore dj has the form
dj = d(θ) = cos θ dj−1 + sin θ sj ,
θ ∈ [0, 2π],
(5.16)
where now the search direction sj is chosen to be a vector in the two dimensional
subspace that has the properties
sTj dj−1 = 0
and
ksj k = ∆.
(5.17)
Equation (5.16) implies that Q(xopt +d(θ)) is the expression
Q(xopt ) + cos θ dj−1 + sin θ sj
n
T
∇Q(xopt ) +
o
1
2
cos2 θ dj−1 + cos θ sin θ sj
∇Q(xopt + dj−1 ) − ∇Q(xopt ) + 21 sin2 θ sTj ∇2 Q sj ,
0 ≤ θ ≤ 2π,
T
(5.18)
because the last term in braces is the product ∇2 Q dj−1 . Again ∇2 Q sj is constructed by formula (5.3), after which the minimization of the function (5.18)
takes only O(n) operations. Thus dj is determined, and the subroutine returns
d = dj if the second of the conditions (5.13) holds, or if j is at least n. Alternatively, ∇Q(xopt +dj ) is calculated for the next iteration, by applying the remark
that equation (5.16) gives the gradient
∇Q(xopt+ dj ) = (1−cos θ) ∇Q(xopt ) + cos θ ∇Q(xopt+dj−1 ) + sin θ ∇2 Q sj . (5.19)
Then j is increased by one, in order that the procedure of this paragraph can be
applied recursively until termination occurs.
6. Subroutines BIGLAG and BIGDEN
We recall from Section 2 that, if Box 9 of Figure 1 is reached, then the condition
(1.1) with index i = MOVE is going to be replaced by the interpolation condition
Q(xopt +d) = F (xopt +d), where d is calculated by the procedure of this section.
In theory, given the index MOVE, the choice of d is derived from the positions xi ,
i = 1, 2, . . . , m, of the current interpolation points, but in practice it depends also
on the errors that occur in the matrices that are stored and updated, namely
the submatrices Ξ and Υ of expression (3.12) and the factorization (4.16). We
20
write t instead of MOVE, in order to retain the notation of Section 4. In particular,
equation (4.1) shows the new positions of the interpolation points.
The t-th Lagrange function of the current interpolation points is important. It
is the quadratic polynomial `t (x), x ∈ Rn , that satisfies the Lagrange conditions
`t (xi ) = δit ,
i = 1, 2, . . . , m,
(6.1)
where the remaining freedom in the usual case m < 21 (n+1)(n+2) is taken up by
minimizing the Frobenius norm k∇2 `t kF . Therefore `t is the function
`t (x) = c + (x−x0 )T g +
1
2
m
X
k=1
λk {(x−x0 )T (xk −x0 )}2 ,
x ∈ Rn ,
(6.2)
the parameters c, g and λk , k = 1, 2, . . . , m, being defined by the linear system
of equations (3.10), where the right hand side is now the coordinate vector et ∈
Rm+n+1 . Thus the parameters are the elements of the t-th column of the matrix
H of expression (3.12). For each x ∈ Rn , we let w(x) be the vector in Rm+n+1
that has the components
w(x)k = 21 {(x−x0 )T (xk −x0 )}2 ,
w(x)m+1 = 1 and w(x)i+m+1 = (x−x0 )i ,
k = 1, 2, . . . , m
i = 1, 2, . . . , n
)
.
(6.3)
It follows that expression (6.2) can be written in the form
`t (x) =
m
X
k=1
λk w(x)k + c w(x)m+1 +
n
X
gi w(x)i+m+1 = (H et )T w(x).
(6.4)
i=1
Therefore, when the symmetric matrix H is updated by formula (4.11), because of
the change (4.1) to the interpolation points, expression (4.12) includes the value
τ = eTt H w = eTt H w(x+ ) = (H et )T w(xopt +d) = `t (xopt +d).
(6.5)
Thus the Lagrange function (6.2) gives the dependence of τ on the choice of d.
As mentioned in Section 4, we expect a relatively large modulus of the denominator σ = αβ +τ 2 to be beneficial when formula (4.11) is applied. Usually
σ > τ 2 holds in practice, because in theory both α and β are positive. Thus we
deduce from the previous paragraph that it may be advantageous to let d be an
approximate solution of the subproblem
Maximize |`t (xopt +d)| subject to kdk ≤ ∆,
(6.6)
where ∆ > 0 is prescribed. This calculation is performed by subroutine BIGLAG,
details being given in the next paragraph. There is an excellent reason for a
large value of |`t (xopt + d)| in the case m = 21 (n + 1)(n + 2). Specifically, one
picks a convenient basis of the space of quadratic polynomials, in order that the
construction of Q from the interpolation conditions (1.1) reduces to the solution
of an m×m system of linear equations. Let B and B + be the old and new matrices
21
of the system when the change (4.1) is made to the interpolation points. Then,
as shown in Powell (2001), the dependence of the ratio det B + / det B on d ∈ Rn
is just a quadratic polynomial, which is exactly `t (xopt +d), d ∈ Rn , because of the
Lagrange conditions (6.1). In this case, therefore, the subproblem (6.6) is highly
suitable for promoting the nonsingularity of B + .
The method of BIGLAG is iterative, and is like the procedure of the last paragraph of Section 5. As in equation (5.16), the j-th iteration generates the vector
dj = d(θ) = cos θ dj−1 + sin θ sj ,
(6.7)
where dj−1 is the best estimate of the required d at the beginning of the current
iteration, where dj−1 and sj have the properties
kdj−1 k = ksj k = ∆
sTj dj−1 = 0,
and
(6.8)
and where the angle θ of equation (6.7) is calculated to maximize |`t (xopt +dj )|.
The choice
(6.9)
d0 = ± ∆ (xt − xopt ) / kxt − xopt k
is made for the first iteration, with the sign that provides the larger value of
|`t (xopt+d0 )|, which implies ∇`t (xopt+d0 ) 6= 0, because `t is a quadratic polynomial
that satisfies the Lagrange conditions `t (xopt ) = 0 and `t (xt ) = 1. The vector s1 of
the first iteration is taken from the two dimensional subspace that is spanned by
d0 and ∇`t (xopt ), provided that both the inequalities
|d0T ∇`t (xopt )|2
2
≤ 0.99 ∆ k∇`t (xopt )k2 

and k∇`t (xopt )k ≥ 0.1 |`t (xopt + d0 )| / ∆ 
(6.10)
hold, because this use of ∇`t (xopt ) is unattractive if the subspace is nearly degenerate, or if the bound ∆ k∇`t (xopt )k on the first order term of the identity
`t (xopt + d) = d T ∇`t (xopt ) + 12 d T ∇2`t d,
kdk ≤ ∆,
(6.11)
compares unfavourably with |`t (xopt + d0 )|. Alternatively, if at least one of the
conditions (6.10) fails, then s1 is defined by the technique that gives sj for j ≥ 2.
Specifically, sj is a linear combination of dj−1 and ∇`t (xopt +dj−1 ) that has the
properties (6.8), except that the subroutine returns the vector d = dj−1 in the
unlikely situation
2
T
| dj−1
∇`t (xopt + dj−1 ) |2 ≥ (1−10−8 ) ∆ k∇`t (xopt + dj−1 )k2 .
(6.12)
The usual test for termination is the condition
|`t (xopt + dj )| ≤ 1.1 |`t (xopt + dj−1 )|,
(6.13)
because the iteration has not improved very much the objective function of the
subproblem (6.6). Then d = dj is returned, which happens too if j reaches the
value n. Otherwise, as in equation (5.19), the gradient
∇`t (xopt + dj ) = (1−cos θ) ∇`t (xopt ) + cos θ ∇`t (xopt +dj−1 ) + sin θ ∇2 `t sj (6.14)
22
is calculated, and then j is increased by one for the next iteration. Because the
second derivative matrix of the function (6.2) is not constructed explicitly, the
remarks on ∇2 Q in Section 5 apply also to ∇2 `t , including the use of the formula
∇2 `t u =
m
nX
k=1
o
λk (xk − x0 ) (xk − x0 )T u =
m
X
k=1
ηk (xk − x0 ),
(6.15)
where ηk = λk (xk − x0 )T u, k = 1, 2, . . . , m. Now the vectors u that occur are just
xopt −x0 , d0 and each sj , so the amount of work of BIGLAG is similar to that of
subroutine TRSAPP.
The parameter ∆ of the subproblem (6.6) is set automatically to a value that
depends on three considerations. Firstly, because of the purpose of ρ, as described
in the second paragraph of Section 2, the bound ∆ ≥ ρ is imposed. Secondly,
the Y-branch has been taken from Box 8 of Figure 1 because DIST = kxt −xopt k
is unacceptably large, so the condition ∆ ≤ 0.1 DIST is reasonable. Thirdly, ∆
should be no greater than the current ∆ of the trust region subproblem of Section
5, and we anticipate that ∆ may be halved. These remarks provide the choice
∆ = max [ min{0.1 DIST, 0.5 ∆}, ρ ] ,
(6.16)
which seems to be suitable in practice, even if the given ρbeg causes ρ to be much
less than the required changes to the variables.
After the construction of d by subroutine BIGLAG, the parameters (4.12) are
calculated, x+ being the vector xopt +d. It has been mentioned already that in
theory α and β are positive, but that negative values of σ = αβ +τ 2 may occur
occasionally, due to computer rounding errors. We recall also that formula (4.11)
is applied even if σ is negative, but the updating would be unhelpful if σ were too
close to zero. Therefore the d from BIGLAG is rejected if and only if the current
parameters have the property
| σ | = | α β + τ 2 | ≤ 0.8 τ 2 .
(6.17)
The alternative choice of d is made by calling subroutine BIGDEN, which seeks a
big value of the denominator |σ| instead of a big value of |τ |. The dependence of
σ on x = xopt +d is obtained by substituting x+ = x and w = w(x) into expression
(4.12), using the definition (6.3). Then BIGDEN sets d to an approximation to the
solution of the subproblem
Maximize |σ(xopt +d)| subject to kdk ≤ ∆,
(6.18)
where ∆ still has the value (6.16). This task is much more laborious than the
calculation of BIGLAG, because σ(x), x ∈ Rn , is a quartic polynomial. Fortunately,
numerical experiments show that the situation (6.17) is very rare in practice.
The methods of subroutines BIGLAG and BIGDEN are similar, except for obvious
changes due to the differences between their objective functions. Indeed, BIGDEN
also picks initial vectors d0 and s1 that satisfy the equations (6.8), in order to
23
begin an iterative procedure. Again the j-th iteration lets dj have the form (6.7),
but now θ is calculated to maximize |σ(xopt +dj )|. When j ≥ 2, the vector d = dj
is returned by BIGDEN if it has the property
|σ(xopt + dj )| ≤ 1.1 |σ(xopt + dj−1 )|,
(6.19)
or if j has reached the value n, the test (6.19) being analogous to condition (6.13).
Otherwise, after increasing j by one, the gradient ∇σ(xopt +dj−1 ) is constructed,
using some numbers that are known already, as described at the end of this section.
If the inequality
2
T
∇σ(xopt + dj−1 ) |2 < (1−10−8 ) ∆ k∇σ(xopt + dj−1 )k2
| dj−1
(6.20)
holds, then Sj = span{dj−1 , ∇σ(xopt + dj−1 )} is a well-defined two dimensional
subspace of Rn . Then another iteration is performed, sj being set to a vector
in Sj with the properties (6.8). If the test (6.20) fails, however, the first order
conditions for the solution of the subproblem (6.18) are nearly achieved, so BIGDEN
returns the vector d = dj−1 .
The choice of d0 in BIGDEN is the d that has just been picked by BIGLAG,
because we expect |σ(xopt + d)| to be large when |`t (xopt + d)| is large, although
rounding errors have caused the unwelcome situation (6.17). The direction s1 is
taken from the space S1 = span{d0 , u}, where u is the step xk −xopt from xopt to
one of the other interpolation points. The value of k depends on the ratios
ωi =
| (xi − xopt )T d0 |2
,
kxi − xopt k2 kd0 k2
i ∈ {1, 2, . . . , m}\{opt}.
(6.21)
Priority is given to k = t, this selection being made in the case ωt ≤ 0.99. Otherwise,
k is such that ωk is the least of the ratios (6.21). A criticism of this procedure is
that it ignores the objective function σ, which is why the test (6.19) for termination
is not tried on the first iteration. The possibility u = ∇σ(xopt ) is unattractive,
because ∇σ(xopt ) is zero in exact arithmetic, and it would be inconvenient to
pick u = ∇σ(xopt +d0 ), because the numbers that assist the construction of this
gradient, mentioned in the previous paragraph, are not yet available.
Let σ̂(θ), θ ∈ R, be the value of σ(xopt + d), when d = d(θ) is the vector
(6.7). The main task of an iteration of BIGDEN is to assemble the coefficients σ̌` ,
` = 1, 2, . . . , 9, such that σ̂ is the function
σ̂(θ) = σ̌1 +
4
X
k=1
{σ̌2k cos(kθ) + σ̌2k+1 sin(kθ)} ,
θ ∈ R.
(6.22)
Because the right hand side of equation (4.25) is used in the calculation of σ,
matrices U and V of dimension (m + n) × 5 are constructed, that provide the
dependence of the relevant parts of w −v and H(w −v), respectively, on θ. We
define w by putting the vector
x = xopt + d(θ) = xopt + cos θ dj−1 + sin θ sj ,
24
θ ∈ R,
(6.23)
into expression (6.3), but the definition (4.24) of v is independent of θ. Thus we
find the components
(w − v)i =
=
1
2
1
2
{(x − x0 )T (xi − x0 )}2 − 21 {(xopt − x0 )T (xi − x0 )}2
{(x − xopt )T (xi − x0 )} {(x + xopt − 2 x0 )T (xi − x0 )}
= {v̂i cos θ + ŵi sin θ} {ûi + 21 v̂i cos θ + 12 ŵi sin θ},
1 ≤ i ≤ m, (6.24)
and
(w−v)i+m+1 = cos θ (dj−1 )i + sin θ (sj )i ,
i = 1, 2, . . . , n,
(6.25)
T
(xi − x0 )
where ûi , v̂i and ŵi are the scalar products (xopt − x0 )T (xi − x0 ), dj−1
T
and sj (xi − x0 ), respectively. We construct the rows of U by regarding these
components of w−v as functions of θ, writing them in sequence in the form
Ui1 + Ui2 cos θ + Ui3 sin θ + Ui4 cos(2θ) + Ui5 sin(2θ),
i = 1, 2, . . . , m+n. (6.26)
Then we define V by the property that the terms
Vi1 + Vi2 cos θ + Vi3 sin θ + Vi4 cos(2θ) + Vi5 sin(2θ),
i = 1, 2, . . . , m+n, (6.27)
are the first m and last n components of H(w − v). In other words, because
(w−v)m+1 is zero, V is the product Hred U , where Hred is the matrix H without its
(m+1)-th row and column, which receives attention in the paragraph that includes
equation (4.23). The product of the displays (6.26) and (6.27) is expressed as a
constant plus a linear combination of cos(kθ) and sin(kθ), k = 1, 2, 3, 4, and the
results are summed over i. Thus we find the coefficients β̌` , ` = 1, 2, . . . , 9, of the
function
T
(w−v) H (w−v) = β̌1 +
4
X
k=1
{β̌2k cos(kθ) + β̌2k+1 sin(kθ)},
θ ∈ R.
(6.28)
The contribution from these coefficients to expression (6.22) is explained below.
The definitions (6.3) and (4.24) provide wT eopt = 12 {(xopt −x0 )T (x−x0 )}2 and
v T eopt = 12 kxopt−x0 k4 in formula (4.26). Hence equations (4.12), (4.26) and (4.25),
with t 6= opt, allow σ̂ to be written in the form
σ̂(θ) = α
h
1
2
kx−x0 k4 − {(xopt −x0 )T (x−x0 )}2 + 21 kxopt −x0 k4
h
− α (w−v)T H (w−v) + eTt H (w−v)
i2
.
i
(6.29)
Therefore, because α = eTt H et is independent of x = xopt +d(θ), subroutine BIGDEN
sets the required coefficients of the function (6.22) to σ̌` = −αβ̌` , ` = 1, 2, . . . , 9,
initially, and then it makes the adjustments that provide the square bracket terms
of equation (6.29).
25
The adjustment for the last term of this equation begins with the remark that
is the function (6.27) of θ in the case i = t. Therefore BIGDEN expresses
the square of this function as a constant plus a linear combination of cos(kθ) and
sin(kθ), k = 1, 2, 3, 4, and it adds the resultant coefficients to the corresponding
values of σ̌` , ` = 1, 2, . . . , 9. Moreover, one can deduce from the conditions (6.23)
and (6.8) that the first square brackets of equation (6.29) contain the function
eTt H(w−v)
2
∆ + v̂opt cos θ + ŵopt sin θ
2
+∆
2
ûopt − 21 ∆
2
θ ∈ R,
,
(6.30)
where ûopt , v̂opt and ŵopt are taken from expression (6.24). It follows that the final
adjustment of the σ̌` coefficients is elementary. Next, BIGDEN computes the values
σ̂(2πi/50), i = 0, 1, . . . , 49, directly from equation (6.22), identifying the integer
î ∈ [0, 49] that maximizes |σ̂(2π î/50)|. Then the quadratic polynomial q̂(θ), θ ∈ R,
is constructed by interpolation to σ̂ at the points θ = 2πi/50, i = î−1, î, î+1. The
choice of θ for the definition (6.7) of dj is completed by giving it the value that
maximizes |q̂(θ)| within the range of its interpolation points.
After calculating dj , and then increasing j by one if the test (6.19) fails, the
gradient ∇σ(xopt+dj−1 ) is required, as mentioned already. We are going to derive
it from expression (6.29), the right hand side being the function σ(x), x ∈ Rn ,
where w depends on x through equation (6.3). We consider the equivalent task of
finding ∇σ(xopt +dj ) for the old value of j, in order to retain the notation of the
previous three paragraphs.
The gradient of the first line of the function (6.29) at x = xopt +dj is the vector
h
n
o
2α kx−x0 k2 (x−x0 ) − (xopt −x0 )T (x−x0 (xopt −x0 )
h
n
o
i
i
= 2α kx−x0 k2 dj + djT (x−x0 ) (xopt −x0 ) ,
(6.31)
the right hand side being given by the relation (x − x0 ) = dj + (xopt − x0 ). It is
employed by BIGDEN, in order to avoid some cancellation when kdj k is relatively
small. The remainder of the gradient of the function (6.29) is the sum
−2α
m+n+1
X
i=1
{H (w−v)}i ∇{w(x)i } + 2 {eTt H (w−v)}
m+n+1
X
i=1
Hti ∇{w(x)i }. (6.32)
An advantage of the work so far is that the terms (6.27) for the chosen θ are the
first m and last n components of H(w −v). Thus expression (6.27) provides the
numbers η̂i = {H(w−v)}i , i = 1, 2, . . . , m, and η̌i = {H(w−v)}i+m+1 , i = 1, 2, . . . , n.
We recall from equations (4.12) and (4.25), with t 6= opt, that eTt H (w −v) is the
current value of τ . Therefore, because the definition (6.3) shows that w(x)m+1 is
constant, the sum (6.32) can be written in the form
2
m
X
i=1
(τ Hti −α η̂i ) ∇{w(x)i } + 2
n
X
i=1
(τ Ht i+m+1 −α η̌i ) ∇{w(x)i+m+1 }.
26
(6.33)
Equation (6.3) gives ∇{w(x)i } = {(x−x0 )T (xi −x0 )}(xi −x0 ), i = 1, 2, . . . , m, and
∇{w(x)i+m+1 } = ei , i = 1, 2, . . . , n. It follows that the required gradient of σ(x) is
the sum of three vectors, namely expression (6.31), the sum
2
m n
X
i=1
o
(τ Hti −α η̂i ) (x−x0 )T (xi −x0 ) (xi −x0 ),
(6.34)
and the vector in Rn with the components 2 (τ Ht i+m+1− α η̌i ), i = 1, 2, . . . , n. The
description of the method of BIGDEN is complete.
7. Other details of NEWUOA
We see in Figure 1 of Section 2 that ∆ is revised and MOVE is set in Box 4, that
ρ is reduced in Box 12, and that a test is made in Box 14. We recall also from
the end of Section 1 that shifts of origin are important to the accuracy of the H
matrix. The details of these operations are addressed in this section.
Let ∆old and ∆new be the old and new values of ∆ that occur in Box 4. As
mentioned already, the choice of ∆new depends on the ratio (2.2), and also the
Euclidean length of the step d receives attention. Possible values of ∆new are
1
kdk, kdk and 2kdk in the cases RATIO ≤ 0.1, 0.1 < RATIO ≤ 0.7 and RATIO > 0.7,
2
respectively, but we take the view that, if RATIO > 0.1, then a large reduction in
∆ may be too restrictive on the next iteration. Moreover, we observe the bound
∆ ≥ ρ, and we prefer to sharpen the test in Box 10 by avoiding trust region radii
that are close to ρ. Therefore NEWUOA sets ∆new to ρ or to ∆int in the cases
∆int ≤ 1.5ρ or ∆int > 1.5ρ, respectively, where ∆int is the intermediate value
∆int




1
2
kdk,
=  max { kdk, 21 ∆old },


max { 2 kdk, 21 ∆old },
RATIO ≤ 0.1,
0.1 < RATIO ≤ 0.7,
RATIO > 0.7.
(7.1)
The selection of MOVE in Box 4 provides a relatively large denominator for the
updating formula (4.11), as stated after expression (4.12). We recall that Hw and
β in this expression are independent of t. Let T be the set {1, 2, . . . , m}, except
that the integer “opt” is excluded from T in the case F (xopt+d) ≥ F (xopt ), in order
to prevent the removal of xopt from the set of interpolation points. The numbers
σt = (eTt H et ) β + (eTt H w)2 ,
t∈T ,
(7.2)
are calculated, σt being the denominator that would result from choosing MOVE= t.
There is a strong disadvantage in making |σMOVE | as large as possible, however, as
we prefer to retain interpolation points that are close to xopt . The disadvantage
occurs, for instance, when at least n+1 of the points xi , i = 1, 2, . . . , m, are within
distance ∆ of xopt , but xt is much further away. Then the Lagrange conditions
(6.1) suggest that `t may be not unlike the function kx−xopt k2 /kxt−xopt k2 , x ∈ Rn ,
which, because of the bound kdk ≤ ∆, would imply the property
|`t (xopt + d)| = O ∆2 / kxt − xopt k2 .
27
(7.3)
Now the equations (6.5) include eTt Hw = `t (xopt +d), and it is usual for (eTt Het )β
and (eTt Hw)2 to be positive numbers of similar magnitudes in expression (7.2).
Thus, for general t ∈ T , it may happen that |σt | is O(1) or O(∆4 /kxt −xopt k4 ) in
the case kxt −xopt k ≤ ∆ or kxt −xopt k > ∆, respectively. Therefore NEWUOA sets
MOVE either to zero or to the integer t∗ ∈ T that satisfies the equation
wt∗ |σt∗ | = max {wt |σt | : t ∈ T } ,
(7.4)
where wt is a weighting factor that is necessary for the automatic removal of
interpolation points that are far from xopt . This removal is encouraged by using
a sixth power of kxt −xopt k instead of the fourth power that is indicated above.
Another consideration is that interpolation points tend to cluster near xopt only
when ∆ is either being reduced or is at its lower bound ρ, so the weights are given
the values
h
n
wt = max 1, kxt − x∗ k
.
max [ 0.1 ∆, ρ ]
o6 i
,
t∈T ,
(7.5)
where x∗ is the xopt that is going to be selected in Box 5 of Figure 1. The MOVE= 0
alternative preserves the old interpolation points, so it is available only in the
case F (xopt +d) ≥ F (xopt ). We wish to avoid applications of formula (4.11) that
cause abnormal growth in the elements of H, taking into consideration that some
growth is usual when a remote interpolation point is dropped. Therefore MOVE is
set to zero instead of to t∗ if and only if both the conditions F (xopt +d) ≥ F (xopt )
and wt∗ |σt∗ | ≤ 1 hold.
The value of ρ is decreased from ρold to ρnew in Box 12 of Figure 1. The
reduction is by a factor of 10, unless only one or two changes to ρ are going to
attain the final value ρ = ρend . The equation ρold /ρnew = ρnew /ρend gives a balance
between the two reductions in the latter case. These remarks and some choices of
parameters provide the formula
ρnew




ρend ,
=
(ρold ρend )1/2 ,



0.1 ρold ,
ρold ≤ 16 ρend ,
16 ρend < ρold ≤ 250 ρend ,
ρold > 250 ρend ,
(7.6)
for the adjustment of ρ by NEWUOA.
The reason for Box 14 in Figure 1 is explained in the penultimate paragraph
of Section 2, the calculations with the current value of ρ being complete if the “Y”
branch is taken. We see that Box 14 is reached when the trust region subproblem
of Box 2 yields a step d that has the property kdk < 12 ρ, which suggests that the
current quadratic model Q is convex. Therefore, assuming that CRVMIN is a useful
estimate of the least eigenvalue of ∇2 Q, we prefer not to calculate F (xopt+d) when
the predicted reduction in F , namely Q(xopt )−Q(xopt+d), is less than 81 ρ2 CRVMIN.
Further, if the values of the error |Q(xopt +d)−F (xopt +d)| on recent iterations
are also less than this amount, then we take the view that trying to improve the
accuracy of the model would be a waste of effort. Specifically, the test in Box 14
28
is satisfied if at least 3 new values of F have been computed for the current ρ,
and if all the conditions
(j)
(j)
kd(j) k ≤ ρ and |Qj (xopt + d(j) ) − F (xopt + d(j) )| ≤ 18 ρ2 CRVMIN,
j ∈ J , (7.7)
(j)
hold, where Qj , d(j) and xopt are Q, d and xopt at the beginning of Box 5 on the
j-th iteration, where CRVMIN is generated on the current iteration, and where J
contains 3 integers, namely the iteration numbers of the 3 most recent visits to
Box 5 before the current iteration. Thus the work of NEWUOA with the current
ρ is terminated often, although some of the distances kxi −xopt k, i = 1, 2, . . . , m,
may exceed 2ρ.
In order to show the importance of x0 in practice to the rounding errors of the
updating formula (4.11), we assume that all the distances kxi −xj k, 1 ≤ i < j ≤ m,
between interpolation points are of magnitude one, that kdk = kx+−xopt k is also of
magnitude one, but that kxopt −x0 k = M , say, is large. In theory, the parameters
α, β, τ and σ of expression (4.12), and also the leading m×m submatrix of H,
are independent of x0 (Powell, 2004a), but the definition (4.10) implies that each
of the first m components of w is approximately 21 M 4 . Thus much cancellation
occurs in the formula
(7.8)
β = 12 kx+ − x0 k4 − wT H w.
Further, if there were an error of ε in H11 , and if there were no other errors on
the right hand side of equation (7.8), then β would include an error of magnitude
M 8 ε, this power of M being so large that M > 100 could be disastrous. The
substitution of expression (4.26) into formula (7.8) is less unfavourable, because
H11 is multiplied by −(w1 −v1 )2 , and the middle line of equation (6.24) provides
the value
w1 − v1 =
1
2
{(x+ − xopt )T (x1 − x0 )} {(x+ + xopt − 2 x0 )T (x1 − x0 )}.
(7.9)
Thus the error in β is now of magnitude M 6 ε cos2 θ, where θ is the angle between
x1 −x0 and d = x+ −xopt . The factorization (4.16) also helps the attainment of
adequate accuracy. Nevertheless, we found from numerical experiments in REAL*8
arithmetic, using some difficult objective functions, that sequences of iterations
may cause unacceptable errors if kxopt −x0 k ≥ 102.5 kdk is allowed in the updating
calculations of Section 4. Therefore NEWUOA tests the condition
kdk2 ≤ 10−3 kxopt − x0 k2
(7.10)
before replacing xMOVE by xopt+d in Box 5 of Figure 1. If this condition holds, then
x0 is overwritten by the xopt that occurs at the beginning of Box 5, which alters the
last n rows of the matrix (1.3) and all the elements (3.11). In practice, however,
the matrix H = W −1 of expression (3.12) is stored instead of W . Therefore H is
revised in the way that is implied by the change to W , except that the (m+1)-th
row and column of H are not required. Details of this task are considered in
Section 5 of Powell (2004a), so only a brief outline is given below of the changes
that are made to H when x0 is shifted.
29
Let xav and s be the vectors 12 (x0 +xopt ) and xopt −x0 , respectively, before x0
is overwritten by xopt , let Y be the n×m matrix that has the columns
y j = {sT (xj − xav )} (xj − xav ) + 41 ksk2 s,
j = 1, 2, . . . , m,
(7.11)
and let Θold and Θnew be the old and new H matrices without their (m+1)-th rows
and columns. Then, according to equations (5.11) and (5.12) of Powell (2004a),
Θnew is defined by the formula
Θnew =
I
Y
0
I
!
Θold
I
0
YT
I
!
.
(7.12)
Thus, as mentioned already, the submatrix Ω of expression (3.12) is undisturbed,
and we keep its factorization (4.16). It follows also from expressions (3.12) and
T
(7.12) that the product Y Ω and the sum Y Ξred
+Ξred Y T +Y Ω Y T are added to
the last n rows of Ξ and to the trailing n×n submatrix of Υ, respectively, where
Ξred is the original matrix Ξ without its first row.
When x0 is overwritten by xopt , the gradient ∇Q(x0 ) has to be revised too.
Specifically, because the function (3.1) can be written in the form
Q(xopt + d) = Q(xopt ) + d T ∇Q(xopt ) + 12 d T ∇2 Q d,
d ∈ Rn ,
(7.13)
and because ∇Q(xopt ) = ∇Q(x0 ) + ∇2 Q s follows from s = xopt − x0 , the vector
∇2 Q s is added to ∇Q(x0 ). The constant term of Q is unnecessary, as stated at
the end of Section 4, and ∇2 Q is independent of x0 , except that, as in equation
(4.28), it is expressed as the sum
∇2 Q = Γ +
= Γ+
Pm
j=1
Pm
j=1
γj (xj − x0 ) (xj − x0 )T
γj (xj − xopt + s) (xj − xopt + s)T
= Γ + v sT + s v T +
Pm
j=1
γj (xj − xopt ) (xj − xopt )T ,
(7.14)
m
1
where v = m
j=1 γj (xj − xopt + 2 s) =
j=1 γj (xj − xav ). Therefore the shift in x0
T
T
requires v s +s v to be added to Γ, although the parameters γj , j = 1, 2, . . . , m,
are unchanged.
The amount of work in the previous paragraph is only O(mn), but the implementation of the product (7.12) takes O(m2 n) operations. Therefore we hope that
condition (7.10) holds on only a small fraction of the total number of iterations,
especially when n is large. Rough answers to this question are provided by the
running times of the numerical experiments of the next section. They suggest
that usually the average work per iteration of NEWUOA is close to O(mn).
P
P
8. Numerical results
In December, 2003, the author released the Fortran software of the version of
NEWUOA that has been described, having tested it on a range of problems with
30
up to 200 variables. Then, at the conference in Erice of these proceedings, he
discussed with Nick Gould some other problems that might be tried, which led to
more experiments. It became clear from one of them that a further modification
would be advantageous occasionally. It has now been made, and is the first subject
of this section, because the numerical results that follow were calculated by the
new version of NEWUOA.
The experiment that suggested the modification is the VARDIM test problem
on page 98 of Buckley (1989). The objective function is the quartic polynomial
F (x) =
n
X
`=1
(x` −1)2 +
n
nX
`=1
` (x` −1)
o2
+
n
nX
`=1
` (x` −1)
o4
,
x ∈ Rn ,
(8.1)
which takes its least value of zero at x = e, the vector of ones. Analytic differentiation gives the second derivative matrix
h
∇2 F (x) = 2 I + 2 + 12 {
Pn
`=1
i
` (x` −1)}2 Θ,
x ∈ Rn ,
(8.2)
where I is the n×n unit matrix and where Θ is the rank one matrix that has the
elements θij = ij, 1 ≤ i, j ≤ n. Thus ∇2 F (x) has n−1 eigenvalues of 2 and one
P
of 2+[ 13 +2{ n`=1 `(x` −1)}2 ] n(n+1)(2n+1). When NEWUOA is employed with
m = 2n+1, however, the initial quadratic model has a diagonal second derivative
matrix, the diagonal elements of ∇2 Q being approximately those of ∇2 F (x0 ),
where x0 is the given starting vector of variables, which has the components 1−i/n,
i = 1, 2, . . . , n, in the VARDIM test problem. Thus initially the eigenvalues of ∇2 Q
P
are about 2+[ 2+12{ n`=1 `(x`−1)}2 ] i2 , i = 1, 2, . . . , n, the term in square brackets
being 2+ 13 (n+1)2 (2n+1)2 . It follows that, at the start of the calculation, ∇2 Q
is a very bad estimate of ∇2 F . Further, if n = 80 for instance, the range of
eigenvalues of ∇2 Q initially is from about 5.7 × 107 to 3.6 × 1011 , but the large
eigenvalue of ∇2 F at the solution x = e is only 347762. Therefore NEWUOA
cannot perform satisfactorily unless huge improvements are made to ∇2 Q by the
updating formulae of the sequence of iterations.
Unfortunately, however, each application of the least Frobenius norm updating
method makes the smallest change to ∇2 Q that is allowed by the new interpolation
conditions, so the basic method of NEWUOA is not suitable for the VARDIM test
problem. Therefore the recent modification tries to recognise when the elements
of ∇2 Q are much too large, and, if there is strong evidence for this possibility, then
Q is replaced by Qint , which is the quadratic model that minimizes k∇2 Qint kF ,
instead of the Frobenius norm of the change to ∇2 Q, subject to the conditions
Qint (xi ) = F (xi ), i = 1, 2, . . . , m, the interpolation points xi being the updated ones
at the exit from Box 5 of Figure 1. When Qint is preferred, the gradient ∇Qint (x0 )
and the parameters γj , j = 1, 2, . . . , m, of the expression
∇2 Qint =
Pm
j=1
γj (xj − x0 ) (xj − x0 )T
(8.3)
are required. It follows from the definition of Qint that they are the vector g and the
components of λ in the system (3.10), where r has the components ri = F (xi )−φ,
31
Original NEWUOA
n
Modified NEWUOA
F (xfin )
#F
F (xfin )
#F
20 12018 : 11517 2×10−11 : 8×10−11
40 45510 : 56698 7×10−10 : 3×10−10
80 196135 : 234804 7×10−9 : 3×10−9
5447 : 4610 4×10−11 : 3×10−11
17106 : 17853 1×10−10 : 8×10−11
60305 : 55051 1×10−10 : 3×10−10
Table 1: Two versions of NEWUOA applied to VARDIM with m = 2n+1
i = 1, 2, . . . , m, for any φ ∈ R. Some damage from rounding errors is avoided by
the choice φ = F (xopt ). We deduce from the notation (3.12) that g and λ are the
products Ξred r and Ω r, respectively, where Ξred is still the matrix Ξ without its
first row. Thus NEWUOA constructs a useful form of Qint in O(m2 ) operations.
When the elements of ∇2 Q are much too large, the interpolation equations
(1.1) imply that k∇Q(x)k is also much too large for most vectors of variables.
Usually a huge value of k∇Q(xopt )k causes the ratio (2.2) to be tiny. Moreover,
because ∇Q(x0 ) is available, and because we have found that ∇Qint (x0 ) is the
product Ξred r, it is easy to compare k∇Qint (x0 )k with k∇Q(x0 )k. On the iterations of the new version of NEWUOA that reach Box 5 of Figure 1 from Box 4,
a flag is set to YES or NO, the YES being chosen when the conditions
RATIO ≤ 0.01
and
k∇Qint (x0 )k ≤ 0.1 k∇Q(x0 )k
(8.4)
hold at the end of Box 5. Then Q is replaced by Qint if and only if three consecutive
settings of the flag are all YES.
The VARDIM test problem with 80 variables can be solved by the older version
of NEWUOA, in spite of the deficiencies in ∇2 Q that have been noted. Results
for the unmodified and modified versions, using ρbeg = (2n)−1 and ρend = 10−6 , are
displayed on the left and right hand sides, respectively, of Table 1. The heading
#F denotes the total number of calculations of the objective function, and xfin is
the vector of variables that is returned by NEWUOA, because it gives the least
calculated value of F . In theory, a reordering of the variables makes no difference,
the initial set of interpolation points being unchanged for m = 2n+1, so this device
can be used to investigate some effects of computer rounding errors. The entries
to the left and right of the colons in Table 1 were obtained with different orderings.
We see that rounding errors are highly influential, that the values of F (xfin ) are
satisfactory, and that the modification is successful in reducing #F .
During the development of NEWUOA, the objective function that was used
most is the trigonometric sum of squares
F (x) =
2n n
X
i=1
bi −
n X
Sij sin(θj xj ) + Cij cos(θj xj )
j=1
32
o2
,
x ∈ Rn ,
(8.5)
n
20
40
80
160
m = m(av)
m = 2n+1
m = 21 (n+1)(n+2)
#F
kxfin −x∗ k∞
#F
kxfin −x∗ k∞
#F
kxfin −x∗ k∞
931
1809
3159
6013
1.4×10−6
4.2×10−6
3.8×10−6
5.8×10−6
833
1716
3471
—
6.9×10−7
1.3×10−6
2.1×10−6
—
649
2061
—
—
2.0×10−7
5.5×10−7
—
—
Table 2: Averages for NEWUOA applied to 5 versions of TRIGSSQS
namely TRIGSSQS. The elements of the matrices S and C are random integers
from [−100, 100], each scaling factor θj is sampled from the logarithmic distribution on [0.1, 1], and the parameters bi , i = 1, 2, . . . , 2n, are defined by F (x∗ ) = 0,
where x∗ has the components x∗j = x̂∗j /θj , j = 1, 2, . . . , n, each x̂∗j being picked
from the uniform distribution on [−π, π]. The initial vector x0 has the components (x̂∗j +0.1ŷj∗ )/θj , j = 1, 2, . . . , n, where every ŷj∗ is also taken at random from
[−π, π]. The function (8.5) has saddle points and maxima, due to periodicity,
and the values of the scaling factors θj provide a tougher problem than the case
θj = 1, j = 1, 2, . . . , n. For each n, we generate five different objective functions
and starting points by choosing different random numbers. We let the number
of interpolation conditions, namely m, be 2n+1, m(av) or 21 (n+1)(n+2), where
m(av) is the integer that is nearest to {(n + 21 )(n + 1)(n + 2)}1/2 . Results of the
NEWUOA software for some of these cases, with four values of n and the parameters ρbeg = 10−1 and ρend = 10−6 , are reported in Table 2, the entries in the main
part of the table being averages for the five different test problems that have been
mentioned. Both #F and xfin have been defined already. Again the results are
sensitive to the effects of computer rounding errors. The dashes in the table indicate that the problems were not tried, because of the running times that would be
required on a Sun Ultra 10 workstation. The values of #F in the m = 2n+1 part
of the table are much smaller than the author had expected originally, because
they become less than the number of degrees of freedom in a quadratic model
when n is large. This highly welcome situation provides excellent support for
the least Frobenius norm updating technique. The accuracy of the calculations is
satisfactory, the kxfin −x∗ k∞ entries in the table being comparable to ρend .
The method of NEWUOA, in particular the use of the bound ∆ ≥ ρ in Figure
1, is intended to be suitable for the minimization of functions that have first
derivative discontinuities. Therefore Table 3 gives some results for the objective
function TRIGSABS, which has the form
F (x) =
2n X
i=1
bi −
n X
j=1
Sij sin xj + Cij cos xj ,
x ∈ Rn .
(8.6)
The parameters bi , Sij and Cij , and the initial vector x0 , are generated randomly
33
n
20
40
80
160
m = 21 (n+1)(n+2)
m = m(av)
m = 2n+1
#F
kxfin −x∗ k∞
#F
kxfin −x∗ k∞
#F
kxfin −x∗ k∞
1454
3447
7626
16496
1.0×10−8
1.6×10−8
1.2×10−8
2.2×10−8
2172
6232
16504
—
6.6×10−9
7.7×10−9
7.2×10−9
—
4947
24039
—
—
4.8×10−9
5.9×10−9
—
—
Table 3: Averages for NEWUOA applied to 5 versions of TRIGSABS
as in the previous paragraph, except that we employ the scaling factors θj = 1,
j = 1, 2, . . . , n. Different random numbers provide five test problems for each n as
before. We retain ρbeg = 0.1, but we set ρend = 10−8 , in order to take advantage
of the sharpness of the minimum of F at x = x∗ . The entries in Table 3 are
analogous to those of Table 2. We see that, for each n, the least value of #F
occurs in the m = 2n+1 column, those results being very encouraging. If ρend is
reduced to 10−6 , the figures for m = 2n+1 and n = 160 become #F = 12007 and
kxfin−x∗ k∞ = 1.6×10−6 , so again #F is less than the number of degrees of freedom
in a quadratic model.
We consider next a test problem that was invented by the author recently,
namely SPHRPTS. Here n is even, and n/2 points have to be placed on the
surface of the unit sphere in three dimensions at positions that are far apart. We
let the k-th point pk ∈ R3 have the coordinates


cos x2k−1 cos x2k

pk = 
 sin x2k−1 cos x2k  ,
sin x2k
k = 1, 2, . . . , n/2,
(8.7)
where x ∈ Rn is still the vector of variables. The problem is to minimize the
function
Pn/2 P
−2
F (x) = k=2 k−1
(8.8)
x ∈ Rn ,
`=1 kp` − pk k ,
where initially the points pk are equally spaced on the equator of the sphere, the
vector x0 having the components (x0 )2k−1 = 4πk/n and (x0 )2k = 0, k = 1, 2, . . . , n/2.
The NEWUOA software was applied to this problem, taking m and n from Tables
2 and 3, with ρbeg = n−1 and ρend = 10−6 . The resultant values of #F are shown
to the left of the colons in Table 4. We found also that F (xfin ) agrees with the
minimum value of F (x), x ∈ Rn , to more than 10 decimal places, although there
is much freedom in the optimal vector of variables, because permutations of the
points and rotations of the unit sphere do not alter the value of the double sum
(8.8). Therefore many adjustments of the variables in practice cause only a tiny
reduction in the objective function. Indeed, after computing each F (xfin ), we
inspected the sequence of values of F calculated by NEWUOA, in order to note
34
n
m = 2n+1
m = m(av)
m = 12 (n+1)(n+2)
20
40
80
160
2077 : 351
7245 : 1620
9043 : 3644
24031 : 8193
1285 : 513
4775 : 2884
18679 : 13898
—
1161 : 627
6636 : 2924
—
—
Table 4: Values of #F for the SPHRPTS problem
the position in the sequence of the first value that satisfies F (x) ≤ 1.001 F (xfin ).
These positions are given to the right of the colons in Table 4. We see that, for the
SPHRPTS problem, most of the work is spent on marginal improvements to F ,
especially during the calculations of the m = 2n+1 column.
The NEWUOA software has also been tested on several problems that have
been proposed by other authors. The final table presents results in the following
five cases using m = 2n+1. The ARWHEAD problem (see the Appendix of Conn
et al, 1994) has the objective function
F (x) =
Pn−1 n
o
(x2i + x2n )2 − 4 xi + 3 ,
i=1
x ∈ Rn ,
(8.9)
and the starting point x0 is e ∈ Rn , which is still the vector of ones. In the
CHROSEN problem (see page 45 of Buckley, 1989), we let F be the function
o
Pn−1 n
F (x) =
4 (xi − x2i+1 )2 + (1 − xi+1 )2 ,
i=1
x ∈ Rn ,
(8.10)
and the starting point x0 is −e ∈ Rn . The PENALTY1 problem (see page 79 of
Buckley, 1989) includes two parameters, and we pick the objective function
F (x) = 10−5
n
X
i=1
(xi − 1)2 +
1
4
−
n
X
x2i
i=1
2
,
x ∈ Rn ,
(8.11)
with the starting point (x0 )i = i, i = 1, 2, . . . , n. Our choice of parameters for the
PENALTY2 problem (see page 80 of Buckley, 1989) gives the function
F (x) =
n n
X
i=2
(exi−1 /10 + exi /10 − e(i−1)/10 − ei/10 )2 + (exi /10 − e−1/10 )2
n
+ 1−
n
X
(n−i+1) x2i
i=1
o2
+ (x1 − 51 )2 ,
x ∈ Rn ,
o
(8.12)
and the starting point x0 is 21 e ∈ Rn . The PENALTY3 problem (see page 81 of
Buckley, 1989) has the objective function
F (x) = 10−3 1 + R exn + S exn−1 + R S
+
n
nX
i=1
(x2i − n)
o2
35
+
n/2
X
i=1
(xi − 1)2 ,
x ∈ Rn ,
(8.13)
n
ARWHEAD CHROSEN PENALTY1 PENALTY2 PENALTY3
20
40
80
160
404
1497
3287
8504
845
1876
4314
9875
7476
14370
32390
72519
2443
2455
5703
?
3219
16589
136902
?
Table 5: Values of #F for 5 problems with m = 2n+1
where R and S are the sums
R =
n−2
X
i=1
(xi + 2 xi+1 + 10 xi+2 − 1)2
and S =
n−2
X
i=1
(2 xi + xi+1 − 3)2 ,
(8.14)
and we let the starting point x0 ∈ Rn be the zero vector. We set ρend = 10−6 in
every case, while ρbeg is given the value 0.5, 0.5, 1.0, 0.1 and 0.1 for ARWHEAD,
CHROSEN, PENALTY1, PENALTY2 and PENALTY3, respectively. Table 5
shows the numbers of function evaluations that occurred when NEWUOA was
applied to these problems with our usual choices of n, except that ? indicates that
#F exceeded 500,000.
All the ARWHEAD, CHROSEN and PENALTY1 calculations were completed
successfully, the greatest distance kxfin−x∗ k∞ being 6.1×10−6 , where xfin and x∗ are
still the final and optimal vectors of variables. Good accuracy was also achieved
in the PENALTY2 calculations with n ≤ 80, the values of F (xfin ) agreeing to
13 decimal places with other values that were obtained for permutations of the
variables and other choices of m. When n = 160 is selected, however, the constants
ei/10 , i = 1, 2, . . . , n, vary from 1.1 to 9×106 , so the magnitudes of the terms under
the first summation sign of expression (8.12) vary from 1 to 1013 , which causes
the PENALTY2 problem to be too difficult in REAL*8 arithmetic. We compared
the given results of the PENALTY3 calculations with those that occurred after
permuting the variables. The #F entries became 4336, 18209 and 125884 for
n = 20, n = 40 and n = 80, respectively, which agrees well with the last column of
Table 5. Further, for each n, the two values of F (xfin ) were slightly less than n2 ,
and they agreed to about 11 decimal places. A feature of PENALTY3, however,
is that the minimum value of the objective function is close to 10−3 and is hard to
find. This magnitude is exposed by picking the variables xi = 1, i = 1, 2, . . . , n−1,
and xn = −(n2 − n + 1)1/2 , because then exn is tiny and both S and the second
line of expression (8.13) are zero, which provides F (x) = 10−3 (1+R exn ) ≈ 10−3 .
When NEWUOA was applied to PENALTY3 with n = 160, the original ordering
of the variables yielded #F = 629582 and F (xfin ) = 25447.688, while the new
ordering yielded #F = 16844 and F (xfin ) = 0.001002. We had not expected the
new ordering to be so favourable, because the differences in the results are due
entirely to computer rounding errors.
36
The average amount of work per iteration is mentioned at the end of Section
7, being at best O(n2 ) in the case m = 2n+1. We tested this possibility in the
ARWHEAD and PENALTY1 experiments of Table 5. The total time in seconds
of each calculation on a Sun Ultra 10 workstation was divided by the product of n2
and #F . The resultant quotients for ARWHEAD are 8.4×10−6 , 8.0×10−6 , 8.5×10−6
and 8.8×10−6 in the cases n = 20, n = 40, n = 80 and n = 160, respectively, and
the corresponding quotients for PENALTY1 are 9.2×10−6 , 8.5×10−6 , 8.6×10−6
and 9.3 × 10−6, the running time in the last case being nearly 5 hours, while
ARWHEAD with n = 20 was solved in only 1.36 seconds. These findings suggest
that the average complexity of each iteration is proportional to n2 , which is most
welcome.
The development of NEWUOA has taken nearly three years. The work was
very frustrating, due to severe damage from computer rounding errors in difficult
cases, before the factorization (4.16) of Ω was introduced. Therefore the author
has had doubts about the use of the explicit inverse matrix H = W −1 , instead
of using a factored form of W that allows the system (3.10) to be solved in
O(m2 ) operations. The numerical results are still highly sensitive to computer
rounding errors, but the experiments of this section show that good accuracy is
achieved eventually, which confirms the stability of the given techniques. Thus we
conclude that the least Frobenius norm method for updating quadratic models is
highly successful in unconstrained minimization calculations without derivatives.
Readers are invited to request a free copy of the NEWUOA Fortran software by
sending an e-mail to mjdp@cam.ac.uk.
Appendix: Proofs for Section 3
The assertions of the last two paragraphs of Section 3 are presented below as lemmas with proofs. The positions of the relevant interpolation points are described
at the beginning of Section 3, followed by the definitions of the matrices (3.12).
Lemma 1: The first row of the initial matrix Ξ has the elements (3.13), and,
for every integer i that satisfies 2 ≤ i ≤ min[n+1, m−n], the i-th row includes the
elements (3.14). When m ≤ 2n, the nonzero elements of the remaining rows of Ξ
take the values (3.15), where i is any integer from the interval [m−n+1, n+1].
All other elements of the initial matrix Ξ are zero.
Proof: For each integer j in [1, m], we let the quadratic polynomial
`j (x) = `j (x0 ) + (x−x0 )T ∇`j (x0 ) + 21 (x−x0 )T ∇2 `j (x−x0 ),
x ∈ Rn ,
(A.1)
be the j-th Lagrange function of the initial interpolation points, which means that
k∇2 `j kF is as small as possible subject to the conditions
`j (xi ) = δij ,
i = 1, 2, . . . , m,
(A.2)
as stated in the second paragraph of Section 6. The construction of `j is the same
as the construction of D in Section 3, if the constraints (3.6) have the right hand
37
sides F (xi )−Qold (xi ) = δij , i = 1, 2, . . . , m. Therefore `j (x0 ) and ∇`j (x0 ) are the
same as c and g, respectively, in the system (3.10), when r is the coordinate vector
ej ∈ Rm . In this case, the partitioned vector on the left hand side of equation (3.10)
is the j-th column of W −1 . It follows from the notation (3.12) that `j (x0 ) and
∇`j (x0 ) provide the j-th column of Ξ, as shown in the expression
Ξ =
`1 (x0 )
`2 (x0 ) · · · `m (x0 )
∇`1 (x0 ) ∇`2 (x0 ) · · · ∇`m (x0 )
!
.
(A.3)
The remainder of the proof depends on the positions of the initial interpolation
points. In particular, because of the choice x1 = x0 with the first of the conditions
(A.2) for each j, the first row of the matrix (A.3) has the elements (3.13). Moreover, when k satisfies 1 ≤ k ≤ min[n, m−n−1], the points xk+1 = x0 +ρbeg ek and
xk+n+1 = x0 −ρbeg ek have been chosen, so the k-th component of ∇`j (x0 ) is the
divided difference
∇`j (x0 )
k
= (2 ρbeg )−1 `j (xk+1 ) − `j (xk+n+1 )
= (2 ρbeg )−1 (δk+1 j − δk+n+1 j ),
j = 1, 2, . . . , m,
(A.4)
because `j is a quadratic that takes the values (A.2). We replace k +1 by i, and
then expression (A.3) gives (∇`j (x0 ))k = Ξk+1 j = Ξij . It follows from equation
(A.4) that formula (3.14) does provide all the nonzero elements of the i-th row of
Ξ for 2 ≤ i ≤ min[n+1, m−n]. Finally, if k satisfies m−n ≤ k ≤ n, then only the
first two of the vectors x1 = x0 , xk+1 = x0 +ρbeg ek and x0 −ρbeg ek are interpolation
points. Further, the minimization of k∇2 `j kF subject to the conditions (A.2)
yields (∇2 `j )kk = 0, j = 1, 2, . . . , m, so the univariate function `j (x0 +αek ), α ∈ R,
is a linear polynomial for each j. Therefore the (k+1)-th row of the matrix (A.3)
contains the divided differences
Ξk+1 j = ∇`j (x0 )
k
= (ρbeg )−1 `j (xk+1 ) − `j (x1 )
= (ρbeg )−1 (δk+1 j − δ1j ),
j = 1, 2, . . . , m.
(A.5)
Again we replace k+1 by i, so equation (A.5) establishes that the nonzero elements
of the i-th row of Ξ have the values (3.15) when i satisfies m−n+1 ≤ i ≤ n+1.
The proof of the lemma is complete.
2
Lemma 2: When m ≥ 2n + 1 holds, the initial matrix Υ is identically zero.
Otherwise, Υ is a diagonal matrix, and expression (3.16) gives all the elements of
Υ that are nonzero.
Proof: Let m̃ be the integer min[m, 2n+1], and let Ξ̃, Ã and X̃ be the leading
(n+1)×m̃, m̃×(n+1) and (n+1)×(n+1) submatrices of Ξ, A and X, respectively.
The definitions (3.12) provide the matrix equation ΞA+ΥX = 0, and its first n+1
columns give the identity Ξ̃Ã+ΥX̃ = 0, which depends on the property in Lemma
1 that, if m > 2n+1, then the last m−2n−1 columns of Ξ are zero. We deduce
38
from equations (3.2) and (3.11) that à has the elements
4
Ãii = Aii = 21 ρbeg
,
4
,
Ãi+n i = Ai+n i = 12 ρbeg
Ãij = Aij = 0,

i = 2, 3, . . . , n+1 


i = 1, 2, . . . , m̃,
j = 1, 2, . . . , n+1.
i = 2, 3, . . . , m̃−n  ,


otherwise
(A.6)
We seek the elements of the product Ξ̃Ã, which is a square matrix. For each
integer j in [1, n+1], equations (3.13), (3.14) and (3.15) give the formula




Ã1j
i = 1,
(Ξ̃Ã)ij =  (2 ρbeg )−1 (Ãij − Ãi+n j ),


(ρbeg )−1 (Ãij − Ã1j ),
2 ≤ i ≤ min[n+1, m−n],
(A.7)
m−n+1 ≤ i ≤ n+1,
the last line being void in the case m ≥ 2n + 1. It follows from equation (A.6)
that Ξ̃Ã is a diagonal matrix, and that its first row and column are zero. Further,
because min[n+1, m−n] is the same as m̃−n, we find the diagonal elements
(Ξ̃Ã)ii = 0,
3
,
(Ξ̃Ã)ii = 21 ρbeg
1 ≤ i ≤ m̃−n


.
m−n+1 ≤ i ≤ n+1 
(A.8)
We now consider the identity Ξ̃Ã+ΥX̃ = 0. The definition (1.3) of X with x1 = x0
imply Ξ̃e1 = e1 , where e1 is the first coordinate vector in Rn+1 , and we recall
Ξ̃Ãe1 = 0. It follows from (Ξ̃Ã + ΥX̃)e1 = 0 that the first column of Υ is also
zero. Thus Ξ̃Ã+ΥX̃ = 0 remains true if any change is made to the first row of X̃.
Expressions (1.3) and (3.2) allow the new X̃ to be ρbeg times the (n+1)×(n+1) unit
matrix. Hence Υ is the matrix −ρ−1
beg Ξ̃Ã, which we know is diagonal. Further,
we deduce from equation (A.8) that Υ is the zero matrix in the cases m ≥ 2n+1,
and that otherwise the nonzero elements of Υ take the values (3.16). Therefore
the lemma is true.
2
Lemma 3: The initial matrix Ω has the factorization
Ω =
m−n−1
X
z k z Tk = Z Z T ,
(A.9)
k=1
where the vectors z k ∈ Rm , k = 1, 2, . . . , m−n−1, are the columns of Z. Further,
the first min[n, m − n − 1] of these vectors have the components (3.18), and, if
m > 2n+1, the remaining vectors have the components (3.20), the subscripts p̂
and q̂ being introduced in the last paragraph of Section 3.
Proof: Let Q(x), x ∈ Rn , be the initial quadratic model, given at the beginning
of Section 3. Each element of ∇2 Q is either defined by the equations (1.1) or is set
to zero. Therefore the choice of Q minimizes k∇2 QkF subject to the interpolation
conditions. It follows from the derivation of the system (3.10) that, if we let r
have the components ri = F (xi ), i = 1, 2, . . . , m, and if we set λ = Ωr, where Ω is
39
taken from expression (3.12), then λ is the unique vector satisfying the constraints
(3.7), such that ∇2 Q is the matrix (3.8). These remarks characterise Ω uniquely,
because they are valid for all right hand sides ri = F (xi ), i = 1, 2, . . . , m. Hence it
is sufficient to verify that, if we put the matrix (A.9) into the equation λ = Ωr for
general r, then λ has the properties that have been mentioned.
The first of the constraints (3.7) is λT e = 0, where e ∈ Rm is the vector of ones.
Substituting λ = Ωr and Ω = ZZ T , this condition becomes rT ZZ T e = 0, which is
achieved, because each column z k of Z has the components (3.18) or (3.20), and
both sets of components provide z Tk e = 0. Similarly, the relation λ = ZZ T r implies
that the other constraint (3.7) also holds if Z satisfies the equations
m
X
i=1
Zik (xi − x0 ) = 0,
k = 1, 2, . . . , m−n−1.
(A.10)
For 1 ≤ k ≤ min[n, m−n−1], the values (3.18) and (3.2) imply that the left hand side
of this expression is a multiple of the difference ρbeg ek −ρbeg ek = 0. Alternatively,
for n+1 ≤ k ≤ m−n−1, the values (3.20), (3.19) and (3.3), with i = k+n+1 and
x1 = x0 , give the condition
m
X
i=1
Zik (xi− x0 ) = ρ−2
beg (−σp ρbeg ep − σq ρbeg eq + σp ρbeg ep + σq ρbeg eq ) = 0. (A.11)
Thus, for general r ∈ Rm , the vector λ = Ωr does obey the constraints (3.7).
By substituting λ = ZZ T r, we write the matrix (3.8) in the form
2
∇D =
m−n−1
X
(z Tk r)
k=1
m
nX
j=1
o
Zjk (xj −x0 ) (xj −x0 )T ,
(A.12)
and we complete the proof by establishing ∇2 Q = ∇2D. For 1 ≤ k ≤ min[n, m−n−1],
the components (3.18) provide the equations
o 
√ −2 n
1
1
T

z k r = 2 ρbeg −F (x0 ) + 2 F (x0 +ρbeg ek ) + 2 F (x0 −ρbeg ek ) 
o
n
. (A.13)
√
√ −2
Pm
T
T 
2
T

e
=
e
Z
(x
2
ρ
ρ
2
e
−x
)
(x
−x
)
=
e
jk
j
0
j
0
k k
j=1
beg k k
beg
Moreover, the construction in the first paragraph of this section employs the
divided difference
n
o
(∇2 Q)kk = ρ−2
beg F (x0 −ρbeg ek ) − 2 F (x0 ) + F (x0 +ρbeg ek ) .
(A.14)
It follows that the first min[n, m − n − 1] terms of the sum over k in expression
(A.12) provide a diagonal matrix, whose diagonal elements are the same as those
of ∇2 Q. Thus ∇2 Q = ∇2D is achieved in the cases m ≤ 2n+1. It remains to show
that, if m > 2n+1, then the last m−2n−1 values of k in expression (A.12) generate
the off-diagonal elements of ∇2 Q without disturbing the diagonal elements.
For each k in the interval [n+1, m−n−1], the interpolation points (3.3) and
(3.19) are relevant with i = k +n+1. Indeed, the components (3.20) imply that
40
z Tk r is just the left hand side of equation (3.5), while the term in the braces of
expression (A.12) is the matrix
−ep eTp − eq eTq + (σp ep +σq eq ) (σp ep +σq eq )T = σp σq (ep eTq + eq eTp ).
(A.15)
Therefore the k-th term of the sum (A.12) contributes to (∇2D)pq and (∇2D)qp
the amount that is required by equation (3.5), and it does not alter any other
element of ∇2D. Thus all the different elements of ∇2 Q that can be nonzero are
provided by the different values of k in expression (A.12). The justification of the
initial choice of Z is complete.
2
Acknowledgements
The author is very grateful for the facilities he has enjoyed, throughout the development of NEWUOA, as an Emeritus Professor at the Centre for Mathematical
Sciences of the University of Cambridge. He has also received excellent support
for this research from the City University of Hong Kong and from the University
of Minnesota. The first numerical experiments on the given method for updating
Q were run during a two month stay in Hong Kong, and the investigations of
several auxiliary techniques were helped greatly by discussions with other visitors
during the IMA Program on Optimization in Minneapolis.
References
A.G. Buckley (1989), “Test functions for unconstrained minimization”, Technical
Report 1989 CS-3, Dalhousie University, Canada.
A.R. Conn, N.I.M. Gould, M. Lescrenier and Ph.L. Toint (1994), “Performance of
a multifrontal scheme for partially separable optimization”, in Advances in
Optimization and Numerical Analysis, eds. Susana Gomez and Jean-Pierre
Hennart, Kluwer Academic (Dordrecht), pp. 79–96.
A.R. Conn, N.I.M. Gould and Ph.L. Toint (2000), Trust-Region Methods, MPS–
SIAM Series on Optimization (Philadelphia).
J.E. Dennis and R.B. Schnabel (1983), Numerical Methods for Unconstrained
Optimization and Nonlinear Equations, Prentice Hall (Englewood Cliffs).
R. Fletcher and C.M. Reeves (1964), “Function minimization by conjugate
gradients”, Computer J., Vol. 7, pp. 149–154.
M.J.D. Powell (2001), “On the Lagrange functions of quadratic models that are
defined by interpolation”, Optim. Meth. Software, Vol. 16, pp. 289–309.
M.J.D. Powell (2002), “UOBYQA: unconstrained optimization by quadratic
approximation”, Math. Programming, Vol. 92, pp. 555–582.
41
M.J.D. Powell (2003), “On trust region methods for unconstrained minimization
without derivatives”, Math. Programming, Vol. 97, pp. 605–623.
M.J.D. Powell (2004a), “Least Frobenius norm updating of quadratic models that
satisfy interpolation conditions”, Math. Programming, Vol. 100, pp. 183–215.
M.J.D. Powell (2004b), “On the use of quadratic models in unconstrained minimization without derivatives”, Optim. Meth. Software, Vol. 19, pp. 399–411.
M.J.D. Powell (2004c), “On updating the inverse of a KKT matrix”, in
Numerical Linear Algebra and Optimization, ed. Ya-xiang Yuan, Science
Press (Beijing), pp. 56–78.
42