Math 408
Homework Set 8
(1) Locate all of the KKT points for the following problems. Can you show that these points are local
solutions? Global solutions?
(a)
minimize e(x1 −x2 )
subject to ex1 + ex2 ≤ 20
0 ≤ x1
First Solution: This is convex problem so any local solution is a global solution. Obviously,
we wish to make x1 as small as possible and x2 as big as possible. Hence, we must have
x1 = 0 which gives the solution (x1 , x2 ) = (0, ln(19)). By plugging this solution into the KKT
conditions, we obtain the multipliers (y1 , y2 ) = (1, 20)/(19)2 . The details of how to obtain the
KKT multipliers associated with the point (x1 , x2 ) = (0, ln(20 − 1)) follow:
(i) Declare dual variables for each constraint:
associate y1 with ex1 + ex2 − 20 ≤ 0 and y2 with −x1 ≤ 0.
(ii) Write the Lagrangian:
L(x1 , x2 , y1 , y2 ) := e(x1 −x2 ) + y1 (ex1 + ex2 − 20) − y2 x1 .
(iii) To obtain values for (y1 , y2 ) associated with our guess at the solution (x1 , x2 ) = (0, ln(19)),
we need to solve the equation
exp(x1 − x2 ) + y1 exp(x1 ) − y2
0
= ∇x L(x1 , x2 , y1 , y2 ) =
− exp(x1 − x2 ) + y1 exp(x2 )
0
with (x1 , x2 )T = (0, ln(19))T , which becomes
1
0=
+ y1 − y2
19
1
0 = − + y1 19 − y2 ,
19
2
yielding y1 = (1/19) and y2 = 20/(19)2 . Since (y1 , y2 ) = (1, 20)/(19)2 is dual feasible,
we have obtained KKT multipliers for the point (x1 , x2 ) = (0, ln(19)).
Second Solution: It is not necessary to guess the solution to this problem in order to obtain
a KKT pair associated with a global solution. But it is necessary to observe that the problem
is convex. It is convexity that tells us that any KKT point yields a global solution.
To obtain a KKT point without guessing the values for (x1 , x2 ) simply write the KKT conditions for the the problem and run through the various cases for activities in the constraints,
or equivalently, values for the dual variables. The steps are as follows:
Step 1: Declare dual variables for each constraint:
associate y1 with ex1 + ex2 − 20 ≤ 0 and y2 with −x1 ≤ 0.
Step 2: Write the Lagrangian:
L(x1 , x2 , y1 , y2 ) := e(x1 −x2 ) + y1 (ex1 + ex2 − 20) − y2 x1 .
Step 3: Write the KKT conditions:
(i) (primal feasibility) ex1 + ex2 ≤ 20 and 0 ≤ x1
(ii) (dual feasibility) 0 ≤ y1 and 0 ≤ y2
(iii) (complementarity) 0 = y1 (ex1 + ex2 − 20) and 0 = y2 x1
1
(iv) (stationarity of the Lagrangian)
0
exp(x1 − x2 ) + y1 exp(x1 ) − y2
= ∇x L(x1 , x2 , y1 , y2 ) =
0
− exp(x1 − x2 ) + y1 exp(x2 )
Step 4: Examine possible cases until a KKT point is found, or it is shown that none exists.
Case 1: Niether constraint is active so that y1 = y2 = 0 due to complementarity.
In this case the stationarity condition for the Lagrangian implies that 0 = exp(x1 −
x2 ), which cannot happen. Hence, any KKT point, if one exists, must have one or
more of the constraints active.
Case 2: The constraint ex1 + ex2 ≤ 20 is inactive and the constraint 0 ≤ x1 is active.
In this case, x1 = 0 and the complementarity conditions imply that y1 = 0. The
stationarity of the Lagrangian again implies that 0 = exp(x1 − x2 ) which cannot
happen. Hence the first constraint must be active.
Case 3: The constraint ex1 + ex2 ≤ 20 is active and the constraint 0 ≤ x1 is inactive.
In this case, exp(x1 ) + exp(x2 ) = 20 and y2 = 0 by the compilmentarity conditions.
Stationarity of the Lagrangian becomes
0
exp(x1 − x2 ) + y1 exp(x1 )
=
.
0
− exp(x1 − x2 ) + y1 exp(x2 )
By adding the two rows together, we find that 0 = y1 (exp(x1 ) + exp(x2 )) = 20y1
so that y1 = 0. But then exp(x1 − x2 ) = 0 which cannot happen. Hence, both
constraints must be active.
Case 4: Both constraints are active. In this case we find that x1 = 0 so that x2 =
ln(19), and we are back where we started giving multipliers (y1 , y2 ) = (1, 20)/(19)2
and establishing (x1 , x2 ) = (0, ln(19)) as a KKT point.
(b)
minimize e(−x1 +x2 )
subject to ex1 + ex2 ≤ 20
0 ≤ x1
Solution: Here we want to make x1 as big as possible and x2 as small as possible. By fixing
x + 1 at zero and sending x2 to −∞, the constraints are satisfied and the objective goes to
zero. Hence, no solution exists and the optimal value is 0.
(c)
minimize x21 + x22 − 4x1 − 4x2
subject to x21 ≤ x2
x1 + x2 ≤ 2
Solution: This is a convex optimization problem and so any KKT point will give global
optimality. Check that (x1 , x2 ) = (1, 1) and (y1 , y2 ) = (0, 2) is a KKT pair for this problem.
To obtain this solution first observe that this is a convex problem with strictly convex objective
function. Hence, if a solution exists, it is the unique global solution. We search for the solution
by locating all possible first-order stationary points, i.e. KKT points. By convexity, all such
KKT points will identify the unique solution if it exists. The steps in the derivation of the
solution from the KKT conditions is given below.
Step 1: Declare dual variables for each constraint:
associate y1 with x21 ≤ x2 and y2 with x1 + x2 ≤ 2.
Step 2: Write the Lagrangian:
L(x1 , x2 , y1 , y2 ) = x21 + x22 − 4x1 − 4x2 + y1 (x21 − x2 ) + y2 (x1 + x2 − 2).
Step 3: Write the KKT conditions:
(i) (primal feasibility) x21 ≤ x2 and x1 + x2 ≤ 2
(ii) (dual feasibility) 0 ≤ y1 and 0 ≤ y2
(iii) (complementarity) 0 = y1 (x21 − x2 ) and 0 = y2 (x1 + x2 − 2)
(iv) (stationarity of the Lagrangian)
0
2x1 − 4 + 2y1 x1 + y2
= ∇x L(x1 , x2 , y1 , y2 ) =
0
2x2 − 4 − y1 + y2
Step 4: Examine possible cases until a KKT point is found, or it is shown that none exists.
Case 1: Niether constraint is active so that y1 = y2 = 0 due to complementarity. In
this case, stationarity of the Lagrangian tells us that the solution must be the point
(x1 , x2 ) = (2, 2) which is infeasible. Hence, at least one constraint must be active.
Case 2: The constraint x21 ≤ x2 is active and the constraint x1 + x2 ≤ 2 is inactive.
In this case, x21 = x2 and y2 = 0. Stationarity of the Lagrangian becomes
0
2x1 − 4 + 2y1 x1
=
.
0
2x2 − 4 − y1
√
Hence, 0 ≤ y1 = 2x2 − 4 = 2x21 − 4. Since 0 ≤ y1 , it must be the case that x1 ≥ 2.
Plugging y1 = 2x21 − 4 into the equation 2x1 − 4 + 2y1 x1 gives the cubic equation
2x31√− 3x1 − 2 = 0, so we need
√ a root of this equation that is greater than or equal
to 2. But√if we plug x1 = 2 into this equation we find that it is a solution, i.e.
2
2 · 23/2 − 3 2 − 2 = 0. In addition, the √
derivative of this
√ polynomial is 3(2x − 1)
which is positive for √all x1 ≥ 1. Since 2 > 1, x1 = 2 is the only root of the
polynomial
for x1 ≥ 2 (since the polynomial is
√ increasing for x1 ≥ 1). Therefore,
√
2
x1 = 2, so x2 = x1 = 2. But then x1 + x2 = 2 + 2 > 2 a contradiction. Hence,
the constraint x1 + x2 ≤ 2 must be active with y2 > 0.
Case 3: The constraint x21 ≤ x2 is inactive and the constraint x1 + x2 ≤ 2 is active.
In this case, x1 + x2 = 2 and y1 = 0. Stationarity of the Lagrangian becomes
0
2x1 − 4 + y2
=
.
0
2x2 − 4 + y2
Hence, x1 = x2 which, together with x1 + x2 = 2 implies that x1 = x2 = 1 and
y2 = 2. Since (x1 , x2 ) = (1, 1) and (y1 , y2 ) = (0, 2) satisfy the KKT conditions,
(x1 , x2 ) = (1, 1) is the unique global solution.
Note the that at (x1 , x2 ) = (1, 1), the constraint x21 ≤ x2 is active! This
violates our working hypothesis that this constraint is inactive. But this doesn’t
matter since we found a KKT point regardless. This is a degenerate problem in
the sense that the multiplier of an active constraint is zero. Recall that in linear
programming this kind of degeneracy lead to the possibility of cycling in the simplex
algorithm. It can also lead to numerical difficulties in nonlinear optimization.
(d)
minimize 21 kxk22
subject to Ax = b
where b ∈ Rm and A ∈ Rm×n satisfies Nul (AT ) = {0}.
Solution: This is a convex problem so x̄ is a solution if and only if there is a ȳ such that
(x̄, ȳ) is a KKT pair for this problem. The Lagrangian is L(x, y) = 12 kxk22 + y T (b − Ax). The
KKT conditions are Ax̄ = b and x̄ = AT ȳ. Hence b = Ax̄ = AAT ȳ. Since Nul (AT ) = {0},
Nul (AAT ) = {0} so that the matrix AAT is invertible. Consequently, ȳ = (AAT )−1 b and
x̄ = AT ȳ = AT (AAT )−1 b.
(2) Show that the set
Ω := {x ∈ R2 | − x31 ≤ x2 ≤ x31 }
is not regular at the origin. Graph the set Ω.
Solution: You will not be tested on regularity on the final. TΩ (0, 0) = {0} × R+ while
d ∇f1 (0, 0)T d ≤ 0 and ∇f1 (0, 0)T d ≤ 0 = {0} × R ,
where f1 (x1 , x2 ) = −(x31 + x2 ) and f2 (x1 , x2 ) = x2 − x31 .
(3) Construct an example of a constraint region of the form
{x |f1 (x) ≤ 0 i = 1, . . . , s, fi (x) = 0 i = s + 1, . . . , m }
at which the MFCQ is satisfied, but the LI condition is not satisfied.
Solution: You will not be tested on constraint qualifications on the final.
(x1 , x2 ) x31 ≤ x2 and 0 ≤ x2 .
(4) Suppose Ω = {x ; Ax ≤ b, Ex = h} where A ∈ Rm× , E ∈ Rk×n , b ∈ Rm , and h ∈ Rk .
(a) Given x ∈ Ω, show that
TΩ (x) = {d : Ai· d ≤ 0 for i ∈ I(x), Ed = 0},
where Ai· denotes the ith row of the matrix A and I(x) = {i Ai· x = bi }.
Solution: It was shown in class (see page 3 of the course notes Optimality Conditions for
Constrained Problems) that
(♠)
TΩ (x) ⊂ {d : Ai· d ≤ 0 for i ∈ I(x), Ed = 0},
so we need only show the reverse inclusion. Let x ∈ Ω and d be and element of the set of the
right hand side of (♠). Note that by continuity there is a t̄ > 0 such that x + td ∈ Ω for all
0 ≤ t ≤ t̄. Hence d ∈ TΩ (x).
(b) Given x ∈ Ω, show that every d ∈ TΩ (x) is a feasible direction for Ω at x.
Solution: This is what we showed in the answer to the previous question.
(c) Note that parts (a) and (b) above show that
[
TΩ (x) =
λ(Ω − x)
λ>0
whenever Ω is a convex polyhedral set. Why?
Solution: Because every polyhedral convex set canb be given a representation
Ω = {x ; Ax ≤ b, Ex = h} .
(5) Let C ⊂ Rn be non-empty, closed and convex. For any x ∈ Rn consider the problem of finding the
closest point in C to x using the 2-norm:
D
minimize 12 kx − zk22
.
subject to x ∈ C
Show that z̄ ∈ C solves this problem if and only if
hx − z̄, z − z̄i ≤ 0 for all z ∈ C.
(♣)
Solution: Our basic theorem on first-order optimality conditions for constrained problems states
that for a convex optimization problem min {f (x) |x ∈ C } (i.e. where f is a convex function and C
is a convex set) the point x̄ is a global optimal solution if and only if f 0 (x; d) ≥ 0 for all d ∈ TΩ (x).
But this is precisely the condition specified in (♣) for this problem since for a convex set we have
!
[
TΩ (x) = cl
λ(Ω − x) .
λ>0
(6) Let Ω be a non-empty closed convex subset of Rn . The geometric object dual to the tangent cone
is called the normal cone:
NΩ (x) = {z ; hz, di ≤ 0, for all d ∈ TΩ (x)}.
(a) Show that if x̄ solves the problem min{f (x) : x ∈ Ω}, then
−∇f (x̄) ∈ NΩ (x̄).
Solution: If x̄ solves min{f (x) : x ∈ Ω}, then
∇f (x̄)T (x − x̄) ≥ 0 ∀ x ∈ Ω,
or equivalently,
(−∇f (x̄))T (t(x − x̄)) ≤ 0 ∀ x ∈ Ω and t > 0,
or equivalently,
(−∇f (x̄))T d ≤ 0 ∀ d ∈
[
t(Ω − x̄),
t>0
which, by continuity, implies that
(−∇f (x̄))T d ≤ 0 ∀ d ∈ TΩ (x),
or equivalently,
−∇f (x̄) ∈ NΩ (x̄).
(b) Show that
NΩ (x̄) = {z : hz, x − x̄i ≤ 0, for all x ∈ Ω}.
Solution: Replace −∇f (x̄) by z in the argument given in Part (a).
(c) Let x̄ ∈ Ω. Show that x̄ solves the problem min{ 21 kx − yk22 : x ∈ Ω} for every y ∈ x̄ + NΩ (x̄).
Solution: Let f (x) := 21 kx − yk22 so that f is convex and ∇f (x) = (x − y). Hence, x̄ solves
min{ 12 kx−yk22 : x ∈ Ω} if and only if y − x̄ = −∇f (x̄) ∈ NΩ (x̄), or equivalently, y ∈ x̄+NΩ (x̄).
(7) Consider the functions
1
f (x) = xT Qx − cT x
2
and
1
ft (x) = xT Qx − cT x + tφ(x),
2
where t > 0, Q ∈ Rn×n is positive semi-definite, c ∈ Rn , and φ : Rn → R ∪ {+∞} is given by
Pn
− i=1 ln xi , if xi > 0, i = 1, 2, . . . , n,
φ(x) =
+∞
, otherwise.
(a) Show that φ is a convex function.
Solution: φ is twice continuously differentiable on its convex domain with
−2
∇φ(x) = diag(x−2
1 , . . . , xn )
which is positive definite on this domain.
(b) Show that both f and ft are convex functions.
Solution: It is the sum of two convex functions.
(c) If Q is positive definite, how that the solution to the problem min ft (x) always exists and is
unique.
Solution: f is coercive and strictly convex which implies that ft is as well for t > 0.
(d) Let {ti } be a decreasing sequence of positive real scalars with ti ↓ 0, and let xi be the solution
to the problem min fti (x). Show that if the sequence {xi } has a cluster point x̄, then x̄ must
be a solution to the problem min{f (x) : 0 ≤ x}.
Solution: For t > 0, the unique solution x(t) to min ft (x) satisfies
Qx(t) − c = w(t) > 0 where x(t)j w(t)j = t j = 1, . . . , n ,
−1 T
i.e., w(t) = (tx(t)−1
1 , . . . , tx(t)n ) .
Let ti ↓ 0 and set xi = x(ti ) and wi = w(ti ). If the sequence {xi } has a cluster point
x̄, then there is a subsequence J ⊂ N such that xi →J x̄, wi →J w̄ = Qx̄ − c ≥ 0, and
x̄j w̄j = 0 j = 1, . . . , n. But then (x̄, w̄) is a KKT pair for the problem min {f (x) |0 ≤ x }.