Introduction to Convex Optimization
Chee Wei Tan
CS 8292 : Advanced Topics in Convex Optimization and its Applications
Fall 2010
Outline
• Optimization
• Examples
• Overview of Syllabus
• An Example: The Illumination Problem
• References
Acknowledgement: Thanks to Mung Chiang (Princeton),
Stephen Boyd (Stanford) and Steven Low (Caltech) for the course
materials in this class.
1
Optimization Mentality
OPT
Problems
Solutions
OPT
OPT
Theories
Optimization as a language, as an engineering system, as a unifying
principle
2
Optimization
minimize f (x)
subject to x ∈ C
Optimization variables: x. Constant parameters describe objective
function f and constraint set C
3
Questions
• How to describe the constraint set?
• Can the problem be solved globally and uniquely?
• What kind of properties does it have?
another optimization problem?
How does it relate to
• Can we numerically solve it in an efficient, robust, and distributed
way?
• Can we optimize over a sequence of time instances?
• Can we find the problem for a given solution?
4
Solving Optimization Problems
• general nonlinear optimization problem
often difficult to solve, e.g., if nonconvex
methods involve some compromise, e.g., very long computational time
• local optimization
find a suboptimal solution
computationally fast but initial point dependent
• global optimization
find a global optimal solution
computationally slow
In convex optimization, the art and challenge is in problem formulation. In
nonconvex optimization, the art and challenge is in problem structure. Convex
optimization plays an important role in exploiting this problem structure
5
Perspective
Widely known: linear programming is powerful and easy to solve
Modified view: ... the great watershed in optimization
isn’t between linearity and nonlinearity, but
convexity and nonconvexity.– SIAM Review 1993, R.
Rockafellar
• Local optimality is also global optimality
• Lagrange duality theory well developed
• Know a lot about the problem and solution structures
• Efficiently compute the solutions numerically
So we’ll start with convex optimization introduction, then specialize into different
applications involving convex problems (and nonconvex problems)
6
Examples
• Least-Squares:
minimizex kAx − bk2
Analytical solution: x? = (AT A)−1AT b; reliable and efficient algorithms and
software; computational time proportional to n2k (A ∈ Rk×n); less with
structure; a mature technology
• Using Least-Squares: several standard techniques increase the flexibility of least
squares, e.g., adding weights, regularization
• Linear Programming:
minimize cT x
subject to aTi x ≤ bi, i = 1, . . . , m
No analytical formula for solution; reliable and efficient algorithms and software;
computational time proportional to n2m if m ≥ n; less with structure; a mature
7
technology
Not as easy to recognize as least squares
• Convex Optimization:
minimize f0(x)
subject to fi(x) ≤ bi, i = 1, . . . , m
No analytical formula for solution; reliable and efficient algorithms and software;
computational time (roughly) proportional to max{n3, n2m, F } where F is the
cost of evaluating the objective and constraint functions fi and their first and
second order derivatives; almost a technology
• Using convex optimization
Often difficult to recognize, many tricks for transforming problems, surprisingly
many problems can be solved (or be tackled elegantly) via convex optimization
Once the formulation is done, solving the problem is, like least-squares or linear
programming, (almost) technology. Learn by examples and applications
8
Why Applications?
լᐬլඔΔ
լ↾լ࿇Ζ
ᜰԫၻլ‫א‬
Կၻ֘Δ
ঞլ༚ՈΖ
πᓵ፿ ૪ۖ
รԮρ
Draw inferences about other cases from one instance
Need to know how to recognize and formulate convex optimization, and use
optimization tools to solve your problem (an objective of this course)
9
Convex Sets and Convex Functions
f (y)
f (x) + ∇f (x)T (y − x)
(x, f (x))
10
Convex Optimization and Lagrange Duality
11
Linear Programming and Quadratic
Programming
−c
x∗
P
12
Geometric Programming
(a + b)/2
√
ab
13
Semidefinite Programming
y
K
K∗
x
14
Decomposition
Master Problem
prices / resources
Secondary Master
Problem
prices / resources
...
Subproblem 1
Subproblem N
Subproblem
First Level
Decomposition
Second Level
Decomposition
15
Internet Congestion Control &
Network Utility Maximization
92%
2G
48%
Average
utilization
95%
1G
27%
16%
19%
txq= 100
Linux TCP
txq= 10000
Linux TCP
FAST
txq= 100
txq= 10000
Linux TCP
Linux TCP
FAST
16
Power Control in Wireless
17
Uncovering the Perron-Frobenius Root
Rate region (units in nats/symbol)
5
log(1 + 1/ρ(F + (1/p̄i)veTi ))(1, 1)T
4
3
r2
w = x(F + (1/p̄i)veTi ) ◦ y(F + (1/p̄i)veTi )
2
90 degrees
1
0
0
1
2
r1
3
4
5
18
Combinatorial Problems
Maximize the number of edges between a subset and its
complementary subset
19
Convex-Cardinality Problems
Sparsity, Compressed Sensing, Single-Path Routing
20
Data Mining
Face ≈ eyes + nose + lip + chin + etc
21
Illumination Problem
m lamps illuminating n (small, flat) patches
Intensity Ik at patch k depends linearly on lamp powers pj :
Ik =
m
X
akj pj ,
−2
akj = rkj
max{cos θkj , 0}
j=1
problem: achieve desired illumination Ides with bounded lamp powers
minimize
max | log Ik − log Ides|
k=1,...,n
subject to 0 ≤ pj ≤ pmax, j = 1, . . . , m
22
How to Solve Illumination Problem
1. use uniform power: pj = p, vary p
2. use least-squares:
minimize
n
X
2
(Ik − Ides)
k=1
round pj if pj > pmax or pj < 0
3. use weighted least-squares:
minimize
n
X
k=1
2
(Ik − Ides) +
m
X
2
wj (pj − pmax)
j=1
iteratively adjust weights wj until 0 ≤ pj ≤ pmax
23
4. use linear programming:
minimize
max |Ik − Ides|
k=1,...,n
subject to 0 ≤ pj ≤ pmax, j = 1, . . . , m
which can be solved via linear programming
of course these are suboptimal ‘solutions’ (to the original problem)
5. use convex optimization: problem is equivalent to
minimize
f0(p) = max h (Ik /Ides)
k=1,...,n
subject to 0 ≤ pj ≤ pmax, j = 1, . . . , m
with h(u) = max{u, 1/u}
24
5
4
h(u)
3
2
1
0
0
1
2
u
3
4
f0(p) is convex because maximum of convex functions is convex
exact solution obtained with effort ≈ modest factor × least-squares effort
additional constraints: does adding (a) or (b) below complicate the problem?
(a) no more than half of total power is in any 10 lamps
(b) no more than half of the lamps are on (pj > 0)
25
• answer: with (a), still easy to solve; with (b), extremely difficult
• moral: (untrained) intuition doesn’t always work; without the proper
background very easy problems can appear quite similar to very difficult
problems
6. Can you think of yet another way to solve the Illumination Problem exactly?
26
References
• Primary Textbook: Convex Optimization by Stephen Boyd and
Lieven Vandenberghe, Cambridge University Press, 2004
http://www.stanford.edu/∼boyd/cvxbook
• Lectures on Modern Convex Optimization: Analysis, Algorithms,
and Engineering, by A. Ben-Tal and A. Nemirovski, SIAM, 2001
• Parallel and Distributed Computation: Numerical Methods by D.
Bertsekas and J. N. Tsitsiklis, 1st Edition, Prentice Hall, 1989
27