MATH 565X
Spring 2013
Midterm Exam
1. (20 points) Let f (x) = x41 + x1 x2 + (1 + x2 )2 .
(a) Carry out one step of the steepest descent method with exact line search, using
x0 = 0 as the starting guess.
(b) Find x1 using Newton’s method, using x0 = 0 as starting guess.
(c) Is the result of taking the Newton step satisfactory? Before deciding, check the
values of f (x0 ) and f (x1 ). If the result is not satisfactory, explain the difficulty.
Solution: The steepest descent direction is p0 = −∇f (0) = [0, −2]T , so in the line
search we minimize φ(α) = f (x0 + αp0 ) = f (0, −2α) = (1 − 2α)2 . The minimum
therefore occurs for α = 1/2, that is, x1 = (0, −1). The Hessian is
12x21 1
2
∇ f (x) =
1
2
so the Newton step is
p0 = −∇2 f (0)−1 ∇f (0) = [−2, 0]T
giving x1 = x0 + p0 = (−2, 0). The Newton step is not a good outcome since
f (x1 ) > f (x0 ), i.e. p0 is not a descent direction. This√happens because the Hessian
is not positive definite at x0 (the eigenvalues are 1 ± 2).
2. (20 points) Let f (x) = x41 + x21 + x1 x2 + (1 + x2 )2 .
(a) Show that f has at least one, but no more than three, critical points.
(b) Show that any critical point must be a local minimum of f .
Solution: Setting ∇f (x) = 0 we get the pair of equations
4x31 + 2x1 + x2 = 0
x1 + 2 + 2x2 = 0
Eliminating x2 leaves the equation for x1
8x31 + 3x1 − 2 = 0
Since this is a cubic with real coefficients, there are either 3 real roots or one real root
and one complex conjugate pair (drawing a reasonably careful graph will convince
you that there is only one real root). Since x2 is uniquely determined by x1 , part a)
follows. The Hessian is
12x21 + 2 1
2
∇ f (x) =
1
2
whose eigenvalues must be positive for any x (for example, since it has a positive
determinant and a positive trace). Thus ∇2 f (x) > 0 for any x, so in particular any
critical point must be a local minimum.
MATH 565X
Spring 2013
Midterm Exam
3. (10 points) In the steepest descent method with exact line search, show that successive
search directions are always orthogonal.
Solution: If αk is the result of an exact line search along direction pk from xk then
d
f (xk + αpk )|α=αk = 0
dα
that is,
∇f (xk+1 )T pk = ∇f (xk + αk pk )T pk = 0
But pk+1 = −∇f (xk+1) , so pTk+1 pk = 0, meaning that pk , pk+1 are orthogonal.
4. (20 points) The PSB (Powell-symmetric-Broyden) Hessian update is
Bk+1 = Bk +
sTk (yk − Bk sk ) T
(yk − Bk sk )sTk + sk (yk − Bk sk )T
−
sk sk
sTk sk
(sTk sk )2
where sk = xk+1 − xk , yk = ∇f (xk+1 ) − ∇f (xk ) as usual. Show that the secant equation
is satisfied, and that Bk+1 is symmetric, if Bk is.
Solution: First we must verify that Bk+1 sk = yk :
(yk − Bk sk )(sTk sk ) + sk (yk − Bk sk )T sk [sTk (yk − Bk sk )]sk T
(sk sk )
−
sTk sk
(sTk sk )2
sk [(yk − Bk sk )T sk ] [sTk (yk − Bk sk )]sk
= yk +
−
sTk sk
sTk sk
= yk
Bk+1 sk = Bk sk +
since (yk − Bk sk )T sk = sTk (yk − Bk sk ) is a scalar.
The second term on the right in the update equation is in the form of a matrix plus
its transpose, so is symmetric, and the third term is of the form αuuT for a scalar α,
which is also symmetric. Thus if Bk is symmetric, the same is true for Bk+1 .
5. (15 points) Let A be a symmetric n × n matrix.
(a) Show that unit vectors p1 , p2 , . . . pn are conjugate directions for A if and only if
P T AP = I where P = [p1 p2 . . . pn ] is the matrix whose j’th column is pj .
(b) If Q is an orthogonal matrix (QT Q = I) and P = [p1 p2 . . . pn ] where p1 , p2 , . . . pn
are conjugate directions for A, show that the columns of P Q are also conjugate
directions for A.
Page 2
MATH 565X
Spring 2013
Midterm Exam
Solution: The i, j entry in P T AP is the dot product of the i’th row of P T with the
j’th column of AP , in other words, the dot product of the i’th column of P with A
times the j’th column of P . This, in turn, is the same as pi · Apj = pTi Apj which
for the case of conjugate directions is 1 if i = j, 0, otherwise. Thus P T AP is the
identity matrix. Similarly for the converse.
If Q is orthogonal then (P Q)T A(P Q) = QT (P T AP )Q = QT IQ = I, so by the first
part the columns of P Q are conjugate directions for A.
6. (15 points) Let f (x) = x2 in R1 . Show directly
that a line search for minimizing f (x)
1
k
leading to the sequence xk = (−1) 1 + 2k would violate the Wolfe conditions.
Solution: The first of the two Wolfe conditions, which amounts in this case to
x2k+1 − x2k ≤ 2c1 xk (xk+1 − xk )
for all k and some c1 > 0, cannot be satisfied here. It is enough to consider the case
of k even, so that it is equivalent to the inequality
xk+1 + xk ≥ 2c1 xk
But xk+1 + xk = 1/2k+1 → 0 as k → ∞, while the right hand side tends to 2c1 , so
the inequality can’t be satisfied for any c1 > 0.
Page 3