EE 572 OPTIMIZATION THEORY
FINAL EXAM
Date: 17 Jan 2008
1-) Apply the steepest descent algorithm with analytical line search to find the minimum point of
f ( x1 , x2 )  x12  x1 x2  x22  x2
Take x0   0.5 0.5 as the initial point. Check your result using the first order necessary conditions.
2-) Show that the diagonally-loaded Newton method can be obtained by solving the following constrained
minimization problem
min f (xk  p)
subject to
p
p  r2
2
where f (x k  p) can be approximated by the quadratic function of p
1
f (xk  p) f (xk )  gk p  pF(xk )p
2
where gk  f (xk ) and F(xk )  2 f (xk ) .
p r
The diagonally-loaded Newton method:
xk 1  xk  k  k I  F(xk ) f (xk )
1
where  k minimizes f (xk   dk ), dk   k I  F(xk ) f (xk ) .
1
3-) For the unconstrained minimization of the function
f (x)  x

where x  x12  x22  ...  xn2

1/ 2
3
is the Euclidean norm, estimate the convergence rate of the modified
Newton method in terms of the parameter  :
xk 1  xk   F(xk ) f (xk )
Hint: Use the matrix inversion lemma
( I  CBC)1  I  C(B1  CC)1 C
where I is the identity matrix and C : n  m, B : m  m
1
4-) Solve the following equality constrained optimization problem:
n
, xn )   ai xi2
min f ( x1 ,
xi
n
subject to
i 1
x
i 1
i
c
where ai  0, i  1,..., n, and c is a constant.
5-) Solve the following inequality constrained optimization problem
min f (x)  x  x  a
3
2
x
and find the optimum values of f (x) for the cases
(i) a   4 5 3
(ii) a  1 1 1 .
subject to x  1
SOLUTION
1-)
Steepest descent:
x k 1  x k   k g k
where  k minimizes f (x k   g k ).
g  f ( x)   2 x1  x2
x 0  [0.5 0.5]
g 0 =[ 0.5 0.5]

f (x 0   g 0 )  0.25(3 2  2  1) 


x1  2 x2  1
x1 =  -1/3 2/3

 0.5  0.5 
x0   g 0 = 

 0.5  0.5 
f ( )  0   0  1/ 3

f ( x1 )  0

x* = x1
2-)
The KKT conditions:
p f (x k  p )   k  p g c (p )  0
g c (p)  p  r 2
2
k ( p  r 2 )  0
2
k  0
g k  F (x k )p  2k p  0  p    F (x k )   k I  g k
1


Assume that the constraint is active
p  r,
where  k  2k
k > 0
p  pp  gk  F (x k )   k I  g k  r 2  0
2
2
  k > 0 is such that  F (x k )   k I  is positive definite.

x k 1  x k  p  x k   F (x k )   k I  g k
1
If the constraint is inactive,
k = 0

p    F (x k ) g k
1
may not exist if F (x k ) is singular.
3-)
f (x)  x   x12 
xn2 
3
f 3 2
  x1 
xi 2
3/ 2
 f
 f ( x )  
 x1
xn2  .2 xi  3 x .xi
1/ 2
f
x2
 f (x)  3 x . x1

 x
xi
xi2 
2 f

3
x

3
x

3
x

3
x

3
x



i
i

xi2
xi
x
x 

xj
2 f
 3xi
i j
x j xi
x


xx 
xx 
 F ( x)  3  x I 
3 x I  2 



x 
x 


f 

xn 
x2
xn   3 x .x
Modified Newton method:
x k 1  x k   F(x k )  f (x k )
1
Using the matrix inversion lemma,
F(x k )
1





x x 
 I  k k2 

2 x k 

1
3 xk

x x 
x ( x x ) 1
 I  k k 2  .3 x k .x k  x k  k k 2k  x k

2
2 x k 
2 xk

1
 1 
x k 1  x k   x k  1    x k
0   2
2
 2 
x k 1
1
1
Convergence rate,
1 
linear with rate 1   .
xk
2
2
F(x k )
1
f ( x k ) 
1
3 xk
4-)
Lagrangian:
n
where h(x)   xi  c
l ( x,  )  f ( x )   h ( x )
i 1
 x l ( x,  )  0,
Necessary conditions
f ( x )    h ( x )  0
h(x)  0

 f
f ( x )  
 x1
h(x)  1 1
(1)
  l ( x,  )  0

2a j x j    0
Constraint equation (2)
1
n
 
i 1

xj 
1
aj
c
n
1
a
i 1
i
f 
   2a1 x1
xn 
f
x2
j  1,

(1)
(2)
,n
1
c
2ai

xj  
2a2 x2

2a j
 
2c
1

i 1 ai
n
2an xn 
5-)
f ( x)  x  x  a
3
2
g (x)  x  1  xx  1
2
KKT conditions:
f (x)  g (x)  0
 g (x)  0,
 0
f (x)  3 x x  2(x  a)
 g ( x )  2 x
3 x x  2(x  a)   2x  0

(1)
 ( x  1)  0,
 0
2

x
2a
5  2
 0 
Suppose constraint is inactive


x
2 x 3 x = 2 a
2a
5  2
1   
2 a 5
2
 3 x x  2x  2a

x 
2a
23 x
Constraint equation 
2
This solution is feasible if
x 
a  2.5
  0
2a
23 x
(5  2  )x  2a

 Constraint 
 This solution is feasible if

x 1
3x  2  x  2x  2a

(1)
 0

Suppose constraint is active
2 x 3 x 5
2
a  2.5
The optimum solution is found by solving the quadratic equation above for x when a is given.
(i) a   4 5 3


a  50  3
 Constraint is active.
a
1

 4 5 3
a
50
x* 
2
2
a
f (x )  1 
 a  1   a  1  37.8578
a
*
(ii) a  1 1 1


a  33
2 x 3 x = 2 3
2
 Constraint is inactive.
Let r  x
The positive solution of this equation is r  0.79175
2
f (x* )  0.4963  x*  a =1.38046


3r 2  2 r  2 3  0
x* 
2a
 0.45712a
2  3 x*