Convex Optimization
CSE 251A Winter 2025
Instructor: Taylor Berg-Kirkpatrick
Based on notes by Sanjoy Dasgupta & Kamalika Chaudhuri
Optimization
d
• Given f : ℝ → ℝ, nd:
min
f(w)
•
w∈ℝd
• w* = argminw∈ℝ f(w)
d
fi
• f : continuous, differentiable, etc.
Optimization
• Example: Linear Regression
(i) (i)
• Given training data (x , y ), i = 1,…, n
w* = argminw∈ℝd ∥y − Xw∥22
• Example: Logistic Regression
(i) (i)
(x
, y ), i = 1,…, n
Given
training
data
•
w* = argminw∈ℝd
n
log(1 + exp( − y (i)(w ⊤x (i) + b)))
∑
i=1
Gradients & Hessians
d
Suppose
we
have
f
:
ℝ
→ℝ
•
• Analogue of 1st derivative is the gradient
‣ Gradient: ∇f(x) =
• Analogue of 2nd derivative is the Hessian
2
Hessian:
∇
f(x) is a d × d matrix
‣
2
∂
f(x)
2
∇ f(x)ii =
‣
∂xi2
2
∂
f(x)
2
where i ≠ j
∇ f(x)ij =
‣
∂xi∂xj
Optimization
• Suppose we have a solution w*
• What makes w* a (locally) optimal solution?
• In 1-dimension:
• Necessary conditions: f′(w*) = 0, f′′(w*) ≥ 0
• Suf cient conditions: f′(w*) = 0, f′′(w*) > 0
• In d-dimensions:
2
∇f(w*)
=
0,
∇
f(w*) is positive
Necessary
conditions:
•
semi-de nite (PSD)




fi
fi
fi

fi

2
∇f(w*)
=
0,
∇
f(w*) is positive
Suf
cient
conditions:
•
de nite (PD)
Positive Semi-De niteness
• Remember (will review in discussion this week):
• A d × d matrix A is PSD iff for all d × 1 vectors z:
⊤
z
‣ Az ≥ 0
• A d × d matrix A is PD iff for all d × 1 vectors z ≠ 0:
fi
⊤
z
‣ Az > 0
Convexity Review
d
A
set
is convex iff for any x, y ∈ S and any
S
⊆
ℝ
•
0 ≤ λ ≤ 1:
• (λx + (1 − λ)y) ∈ S as well
d
d
f
:
ℝ
→
ℝ
x,
y
∈
ℝ
A
function
is
convex
iff
for
any
and
•
any 0 ≤ λ ≤ 1:
• f(λx + (1 − λ)y) ≤ λf(x) + (1 − λ)f(y)
Properties of Convex Functions
1. If f is differentiable, then for any x, y:
⊤
f(y) − f(x) ≥ ∇f(x) (y − x)
"f lies above the gradient"
2. If f is doubly differentiable at x, then:
2
∇ f(x) is PSD
Properties of Convex Functions
• Why are convex functions special?
1. Local minima or global minima
2. Optimal solutions are all connected
3. You can always go "downhill" to optimal solution
• Examples:
2
⊤
f(w) = ∥y − Xw∥2 = (y − Xw) (y − Xw)
f(w) =
n
log(1 + exp( − y (i)(w ⊤x (i))))
∑
i=1
Gradient Descent
• Simplest algorithm: gradient descent
Go "downwards" along
the gradient direction
• Algebraically:
d
arbitrary
point
in
w
=
ℝ
• 0
• Iterate: wt+1 = wt − η ∇f(wt)
Here: η is called the "learning rate" or "step size"
Gradient Descent
• How to set the step size η? (Important to get right)
• η too small → slow convergence
• η too big → solution will bounce around
• In practice:
• Set η to be a small constant, trial and error
• Backtracking line search
Backtracking Line Search
• Backtracking line search:
1.Pick ᾱ > 0, c ∈ (0,1), ρ ∈ (0,1). Descent direction dt
2. Set α = ᾱ
3. Repeat until f(xt + αdt) ≤ f(xt) + cα ∇f(xt)⊤dt:
α ← ρα
4. Terminate with ηt = α
• Notes:
1. For gradient descent, dt = − ∇f(xt), but the algorithm applies even if we
are not going exactly opposite to the gradient -- any direction in which f
decreases will work.
2. We stop when the decrease in f is suf cient (hyper-param c).
fi
fi
fi
3. In general, better to spend more resources nding better/more
directions to descend along, rather than re ning step size.
Subgradients
• What if f is not differentiable everywhere? (e.g. L1 reg)
• The Subgradient Method:
• Use a "subgradient" to descend along instead of the
gradient
• Like gradient descent, otherwise, but with small
caveats
• We'll use subderivative as warmup
Subgradients
• Subderivative (warmup):
f : a convex function on ℝ
A subderivative of f at x0 is a real number c s.t.:
f(x) − f(x0) ≥ c(x − x0)
‣ If f is differentiable at x0, then the subderivative is f′(x)
‣ Otherwise, there are a set of subderivatives [a, b] where

f(x) − f(x0)
f(x) − f(x0)
and b = lim
a = lim−
x→x0
x − x0
x→x0+
x − x0
‣ Subdifferential ∂f(x0) → set of all subderivatives at x0
Subgradients
• Subgradient:
d
d
f
:
ℝ
→
ℝ
v
∈
ℝ
For
a
real-valued
convex
function
,
is a
•
subgradient iff
⊤
f(x)
−
f(x
)
≥
v
(x − x0)
‣
0
• ∂f(x0) : set of all subgradients at x0
• Convergence:
• Guaranteed convergence for small enough constant step
size (scaled by gradient norm)
•
(t)
(t−1)
But must maintain fbest = min (fbest , f(xt)), i.e. best
solution so far
• Slow!
GD Examples
• Linear Regression:
f(w) = ∥y − Xw∥22
∇f(w) = − 2X(y − Xw)
Gradient descent update:
wt+1 = wt + 2ηX(y − Xwt)
GD Examples
• Logistic Regression:
f(w) =
n
log(1 + exp( − y (i)w ⊤x (i)))
∑
i=1
∇f(w) =
=
n
1
n
−y (i)x (i)
∑ 1 + exp − y (i)w ⊤x (i)
(
)
i=1
⋅ exp( − y (i)w ⊤x (i)) ⋅ (−y (i)x (i))
∑ 1 + exp y (i)w ⊤x (i)
(
)
i=1
Gradient descent update:
wt+1 = wt + η
n
y (i)x (i)
∑ 1 + exp y (i)w ⊤x (i)
(
)
i=1
What about large data?
Many ML problems: f(w) =
n
∑
g(w, x (i), y (i))
i=1
• Computing ∇f(wt) requires an entire pass over the
training data and is therefore computationally expensive
when the training set is large (e.g. logistic regression).
• Stochastic Gradient Method:
(1) (1)
(n) (n)
Input:
Data
points
(x
,
y
),
…,
(x
,y )
•
• At optimization step t:
‣ Pick i ∈ {1,…, n} uniformly at random
(i) (i)
w
=
w
−
η
∇g(w
,
x
,y )
‣ t+1 t t
t
Stochastic Gradient Descent
• Notes:
1. f(wt+1) may not be ≤ f(wt). But on average, the
process will lower f(w)
2. Typically needs multiple passes over data (epochs)
3. ηt has to be diminishing for the variance to go to 0 as
t→∞
• Improvement: SGD with mini-batches:
• Instead of picking a single i, pick a random subset of
indices to form each update.
• Lower variance than SGD.
Stochastic Gradient Descent
• Improvement: SGD with mini-batches:
• Instead of picking a single i, pick a random subset of
indices to form each update.
• Lower variance than SGD.
• Speci cally, at timestep t, let Bt be a random subset of
{1,…, n}
1
wt+1 = w + t − ηt
∇g(wt, x (i), y (i))
∑ | Bt |
i∈B
fi
t
Strong Convexity
d
• A function f is m-strongly convex iff for all x, y ∈ ℝ
and 0 ≤ λ ≤ 1 we have:
f(λx + (1 − λ)y) ≤ λf(x) + (1 − λ)f(y)
m
−
λ(1 − λ)∥x − y∥22
2
• Note: m = 0 gives just regular convexity
Strong Convexity
• Properties:
1. If f is differentiable, then the de nition is equivalent to:
m
f(y) ≥ f(x) + ∇f(x) (y − x) + ∥y − x∥22
2
⊤
and ( ∇f(y) − ∇f(x))⊤(y − x) ≥ m∥x − y∥22
2. If f is doubly differentiable, then the de nition is
equivalent to:
fi
is PSD for all x
fi
∇2 f(x) − mI
Strong Convexity
• Examples:
λ
2
f(x) = ∥x∥2
2
λ
f(w) = ∥w∥22 + ∥Xw − y∥22
2
λ
1
f(w) = ∥w∥22 +
log(1 + exp(−y (i)w ⊤x (i)))
2
n∑
i
Strong Convexity
• What is good about strongly convex optimization?
• Usually faster convergence (for SGD, GD, etc.)
• Stability:
‣ For strongly convex functions, all directions are steep. So if f
changes a little, minimum does not move too much.
Strong Convexity
• Theorem: If f is m-strongly convex and
w1 = argminw f(w)
w2 = argminw f(w) + g(w)
maxw ∥∇g(w)∥
Then: ∥w1 − w2∥ ≤
m
• This means that L2 regularization implies stability.
Changing a few data points does not change the
solution much.
Constrained Optimization
minn f(x)
x∈ℝ
s.t.
ci(x) = 0, i ∈ E (equality constraints)
ci(x) ≥ 0, i ∈ I
(inequality constraints)
• Feasible Set Ω = {x | ∀i∈E ci(x) = 0, ∀i∈I ci(x) ≥ 0}
• How do we know if an x is a (local) solution?
Constrained Optimization
• Simple Case 1: Single equality constraint
minn f(x)
x∈ℝ
s . t . c1(x) = 0
• x is a local solution if for small s, either
f(x + s) > f(x)
or
x+s∉Ω
• Suppose there exists a small step s s.t. (i.e. not opt x)
c1(x + s) = c1(x) = 0 and
f(x + s) < f(x)
⊤
• Then: c1(x + s) ≈ c1(x) + ∇c1(x) s = 0
⊤
f(x
+
s)
<
f(x)
and:
implies
0
>
f(x
+
s)
−
f(x)
≈
∇f(x)
s
•
Constrained Optimization
• Simple Case 1: Single equality constraint (continued)
⊤
c
(x
+
s)
≈
c
(x)
+
∇c
(x)
s=0
Then:
•
1
1
1
⊤
and:
implies
f(x
+
s)
<
f(x)
0
>
f(x
+
s)
−
f(x)
≈
∇f(x)
s
•
• Therefore, non (local) optimality implies:
⊤
⊤
for
which
and
∃s
∇c
(x)
s
=
0
∇f(x)
s<0
‣
1
‣ Therefore, if x is a local minimum, then there is no such s
which can only happen when:
‣ ∇c1(s) = λ1 ∇f(x) for some λ1 ∈ ℝ
‣ Optimality conditions are:
‣ ∇f(x) = λ1 ∇c1(x) for some scalar λ1
Constrained Optimization
• Simple Case 2: Single inequality constraint
minn f(x)
x∈ℝ
s . t . c1(x) ≥ 0
• Suppose there exists a small step s s.t. (i.e. not opt x)
c1(x + s) ≥ 0 and
• Then:
and
f(x + s) < f(x)
c1(x + s) ≈ c1(x) + ∇c1(x)⊤s ≥ 0 (a)
⊤
f(x + s) ≈ f(x) + ∇f(x) s < f(x)
(b)
Constrained Optimization
⊤
• Then: c1(x + s) ≈ c1(x) + ∇c1(x) s ≥ 0 (a)
and
f(x + s) ≈ f(x) + ∇f(x)⊤s < f(x)
(b)
• Case 1: x is in the interior of Ω
• Any s satis es (a), and there exists an s that satis es
(b), unless ∇f(x) = 0
• Therefore, optimality condition is:
fi
fi
‣ ∇f(x) = 0 and c1(x) > 0
Constrained Optimization
• Then:
and
c1(x + s) ≈ c1(x) + ∇c1(x)⊤s ≥ 0 (a)
f(x + s) ≈ f(x) + ∇f(x)⊤s < f(x)
(b)
• Case 2: x lies on the boundary of Ω
• c1(x) = 0, therefore:
⊤
(a)
implies
∇c
(x)
s≥0
‣
1
⊤
∇f(x)
s<0
(b)
implies
‣
• There optimality condition is exists no s that satis es both (a)
and (b), which happens iff:
fi
‣ ∇f(x) = λ1 ∇c1(x) for λ1 ≥ 0
Constrained Optimization
• A more general condition:
• The Lagrangian function:
‣
ℒ(x, λ) = f(x) −
∑
i∈E∪I
λici(x)
• Karush-Kuhn-Tucker (KKT) conditions:
‣ At a local solution x*, under some conditions, there exist Lagrange
multipliers λ* s.t.:
∇x ℒ(x*, λ*) = 0
ci(x*) = 0 ∀i ∈ E
ci(x*) ≥ 0 ∀i ∈ I
λ*
≥ 0 ∀u ∈ I
i
λ*
c (x*) = 0 ∀i ∈ E ∪ I ("Complementary slackness": either
i i
λ*
= 0 or ci(x*) > 0)
i
Constrained Optimization
• Example: