Optimization Methods: HW1
Instructor: Lei Li, INS, Shanghai Jiao Tong University;
Email: leili2010@sjtu.edu.cn
All “Numerical optimization” means the second edition unless explicitly
stated otherwise
1
Theory
1.
• Find the tangent plane of the level set of the surface F (x, y, z) =
14 at P (5, −2, 3) where F (x, y, z) = xyz + x2 − 2y 2 + z 3 .
• Suppose f (x, y) = ln(x2 + y 2 ). Compute the tangent plane of the
graph at (1, 1, ln 2). Can you find a normal vector of this plane?
Compute the tangent line of the level set f (x, y) = ln(2). Can
you find a normal vector of this line?
2. The cone z 2 = x2 + y 2 and the plane 2x + 3y + 4z + 2 = 0 intersect
in an ellipse. Find the tangent line of the ellipse at P (3, 4, −5). Find
the plane that is normal to the ellipse at P (3, 4, −5).
3. Explain for which direction ⃗v (a unit vector),
d
ϕ(x + ⃗v ∆s) − ϕ(x)
ϕ(x + s⃗v )|s=0 := lim
ds
∆s
∆s→0+
is maximized. If this is maximized, what is the maximum value? Using
this idea, justify formally (no need to be rigorous) why
Z
Z
f (x)δ(ϕ(x) − c)|∇ϕ(x)| dx =
f (x)dS(x),
Rd
S
where δ is the one-dimensional Dirac delta, S is the level set ϕ(x) = c
and dS is the surface element of S (the Hausdorff measure). If you are
interested in this, you may compare this with the coarea formula.
4. Let A ∈ Rn×n be a square real matrix. Compute the Hessian of the
function f (x) = xT Ax.
5. (a). Prove the Taylor theorems in lecture notes using the Taylor theorems for one-variable functions. (b). Give an example that the claim
∇f (x + p) = ∇f (x) + ∇2 f (x + tp)p in general is not true.
1
6. Problems 2.13, 2.14, 2.16 in the book “Numerical optimization”.
7. Problem 2.6, 2.7 in the book “Numerical optimization”.
8. Suppose that f : (a, b) → R is increasing and convex. Is the inverse
function convex or concave?
9. Suppose D is open, if f is convex and differentiable in D, show that
f (y) ≥ f (x) + ∇f (x) · (y − x). This means that the function is above
the tangent plane.
10. Suppose that f is a convex function. Then, show that the Jensen’s
inequality holds
X
X
X
f(
λi xi ) ≤
λi f (xi ), λi ≥ 0,
λi = 1
i
i
i
As a remark, more generally, one has
f (EX) ≤ Ef (X)
Using this, one can show that for two probability densities p and q,
the KL divergence satisfies
Z
p(x)
p(x) log
dx ≥ 0.
q(x)
11. Suppose that f : Rn → R is continuous. Show that it is convex if and
only if
Z 1
1
f (x + λ(y − x))dλ ≤ [f (x) + f (y)], ∀x, y.
2
0
12. Suppose that f : Rn → R is a smooth convex function, and suppose
that the set of global minimizers S is nonempty. Show that S is a
closed and convex set.
13. Problem 2.9 in the book “Numerical optimization” (section 2.2).
14. Let S be the set of n×n symmetric matrices. Show that X 7→ σ1 (X) :=
max{|λ(X)|}, the largest singular value, is a convex function for X ∈
S. [Hint: show that this is a norm.] Extra to think: what if we
consider the largest eigenvalue (with sign considered)? To answer this,
one property is needed: the pointwise maximum of affine functions is
again convex.
2
Q
15. Show the set {x ∈ Rn+ : ni=1 xi ≥ 1} is a convex set. (You may use
the inequality aθ b1−θ ≤ θa + (1 − θ)b for θ ∈ (0, 1). )
16. Let S ⊂ Rn be a subset of Rn . Let ∥·∥ be a norm. Let a > 0. Consider
the set Sa := {x : d(x, S) ≤ a} and S−a := {x : B(x, a) ⊂ S}. Show
that both sets are convex if S is convex.
Bonus problem
This is a bonus problem. The credits here will be added to your total homework scores (the total HW scores of the whole semester), without
exceeding the maximum points.
set. Let xi ∈ C. Suppose θi ≥ 0 and
P∞Suppose that C is a convex
P∞
θ
=
1.
Show
that
if
x
=
i=1 i
i=1 θi xi converges, then x ∈ C. (Note that
C may not be closed.) Does this fact hold for general Banach spaces with
infinite dimension?
3