Sheet 6 – Optimization for data science
Date: 20.04.2026
SS 2026
Goal: Exploring Steepest Descent accelerations.
Reminders
1. Heavy Ball Method (f L-smooth, µ-convex)
xk+1 = xk − α∇f (xk ) + β(xk − xk−1 ), α, β > 0, x0 = x−1 ∈ Rn .
2. 
Nesterov optimal method (f L-smooth, convex)
k+1

= y k− L1 ∇f (y k ), 
x
p
τk+1 = 12 1 + 4τk2 + 1 ,
τ0 = 1 and x0 = y0 ∈ Rn .


y k+1 = xk+1 + τk −1 (xk+1 − xk ).
τk+1
3. Nesterov optimal method for strongly convex functions (f L-smooth, µconvex)
(
xk+1 = y k − L1 ∇f√(y k ),
x0 = y0 ∈ Rn .
√k (xk+1 − xk ).
y k+1 = xk+1 + 1−
1+ k
Exercise 6.1
We consider the function
f : R2 → R,
f (x) := (x1 − 3)2 + k(x0 − 2)2
for k = 10 and k = 0.01.
1. Implement
√
√
L− µ
• Heavy ball method with α = (√L+4√µ)2 , β = √L+√µ ,
• Nesterov’s optimal method,
• Nesterov’s optimal method for strongly convex functions,
where L denotes the Lipschitz constant of the gradient of f , and µ the modulus
of convexity of f . Use (0, 0)⊤ as initial iterate. Terminate the loop as soon as k ≥
maxiter or ∥∇f (xk )∥2 ≤ epsilon, with maxiter = 1000 and epsilon = 10−5 .
2. Draw a plot of the convergence behavior, plotting iteration number against log10 (f (xk )−
f (x∗ )). (Use the same figure, with different colors for each algorithm. Also add the
gradient descent method using steplength 1/L and 2/(µ + L).)
3. Discuss the results.
1
Exercise 6.2
Let f : Rn → R be L-smooth and convex and let x0 ∈ Rn be arbitrary. Assume that f
admits a minimizer x∗ ∈ Rn . We consider the iteration
y k = xk + βk (z k − xk )
xk+1 = y k − αk ∇f (y k )
z k+1 = z k − γk ∇f (y k ),
k
with αk = L1 , βk = 1 − ωωk+1
, γk = αβkk for ωk+1 = ωk + k2 and ω0 = 0.
2
1. Show that βk = k+1
for k > 1.
2. Show that the following equality holds:
∥z k+1 − x∗ ∥22 − ∥z k − x∗ ∥22
(1)
2αk
2αk (βk − 1)
α2
⟨∇f (y k ), x∗ − xk ⟩.
= − k2 ∥∇f (y k )∥22 + 2 ⟨∇f (y k ), x∗ − xk+1 ⟩ +
2
βk
βk
βk
3. Show that the sequence {φk }k∈N , φk := ( k2 )2 (f (xk ) − f (x∗ )) + L2 ∥z k − x∗ ∥22 is monotonically decreasing.
Hint: Use the following inequality verified for η1 > 0, η2 > 0 and η2 − η1 > 0 in a
substraction with (1).
0 ≥ η2 (f (xk+1 ) − f (x∗ )) − η1 (f (xk ) − f (x∗ )) − η2
+ η1 ⟨∇f (y k ), xk − x∗ ⟩ + η2 ⟨∇f (y k ), x∗ − xk+1 ⟩,
Lαk2
∥∇f (y k )∥22
2
(2)
4. Show that the iterates {xk }k∈N satisfy
f (xk ) − f (x∗ ) ≤
2L 0
∥x − x∗ ∥22
2
k
for all k ∈ N.
5. Show that it yields the same sequences (xk )k∈N and (y k )k∈N as Nesterov’s accelerated
gradient method
y k = xk + β̃k (xk − xk−1 )
xk+1 = y k − α̃k ∇f (y k ),
with α̃k = L1 and β̃k = k−1
for all k ∈ N.
k+2
2