Optimization - Lecture 4, Part 1
M. Pawan Kumar
http://www.robots.ox.ac.uk/~oval/
Slides available online http://mpawankumar.info
“Recap” of Linear Programming
Polyhedron
Ax ≤ b
A : m x n matrix
b: m x 1 vector
Bounded Polyhedron = Polytope
Ax ≤ b
A : m x n matrix
b: m x 1 vector
Vertex
z is a vertex of P = {x, Ax ≤ b}
z is not a convex combination of two points in P
There does not exist x, y ∈ P and 0 < λ < 1
x ≠ z and y ≠ z
such that z = λ x + (1-λ) y
Vertex
z is a vertex of P = {x, Ax ≤ b}
Recall A is an m x n matrix
Az is a submatrix of A
Contains all rows of A such that aiTz = bi
Vertex
⟺
z is a vertex of P
Proof?
Rank of Az = n
See “hidden” slides
Outline
• Linear Programming
• Duality
• Solving the LP
Linear Program
Maximize a linear function
maxx cTx
Objective function
s.t. A x ≤ b
Constraints
Over a polyhedral feasible region
A: m x n matrix
b: m x 1 vector
c: n x 1 vector
x: n x 1 vector
Example
maxx x1 + x2
s.t.
x1 ≥ 0
x2 ≥ 0
4x1 – x2 ≤ 8
2x1 + x2 ≤ 10
5x1 - 2x2 ≥ -2
What is c? A? b?
Example
x1 ≥ 0
x2 ≥ 0
Example
x1 ≥ 0
x2 ≥ 0
4x1 – x2 = 8
Example
x1 ≥ 0
x2 ≥ 0
4x1 – x2 ≤ 8
Example
x1 ≥ 0
x2 ≥ 0
4x1 – x2 ≤ 8
2x1 + x2 ≤ 10
Example
x1 ≥ 0
x2 ≥ 0
4x1 – x2 ≤ 8
2x1 + x2 ≤ 10
5x1 - 2x2 ≥ -2
Example
x1 ≥ 0
x2 ≥ 0
4x1 – x2 ≤ 8
2x1 + x2 ≤ 10
5x1 - 2x2 ≥ -2
x 1 + x2 = 0
maxx x1 + x2
Example
x 1 + x2 = 8
Optimal solution
x1 ≥ 0
x2 ≥ 0
4x1 – x2 ≤ 8
2x1 + x2 ≤ 10
5x1 - 2x2 ≥ -2
maxx x1 + x2
Outline
• Linear Programming
• Duality
• Solving the LP
Example
maxx 3x1 + x2 + 2x3
7x
2x
s.t. -x1 ≤ 0, -x2 ≤ 0, -x3 ≤ 0
3 x x1 + x2 + 3x3 ≤ 30
2x1 + 2x2 + 5x3 ≤ 24
4x1 + x2 + 2x3 ≤ 36
Scale the constraints, add them up
3x1 + x2 + 2x3 ≤ 90
Upper bound on solution
Example
maxx 3x1 + x2 + 2x3
1x
s.t. -x1 ≤ 0, -x2 ≤ 0, -x3 ≤ 0
x1 + x2 + 3x3 ≤ 30
2x1 + 2x2 + 5x3 ≤ 24
1 x 4x1 + x2 + 2x3 ≤ 36
Scale the constraints, add them up
3x1 + x2 + 2x3 ≤ 36
Upper bound on solution
Example
maxx 3x1 + x2 + 2x3
1x
s.t. -x1 ≤ 0, -x2 ≤ 0, -x3 ≤ 0
x1 + x2 + 3x3 ≤ 30
2x1 + 2x2 + 5x3 ≤ 24
1 x 4x1 + x2 + 2x3 ≤ 36
Scale the constraints, add them up
3x1 + x2 + 2x3 ≤ 36
Tightest upper bound?
Example
maxx 3x1 + x2 + 2x3
y1
y2
y3
s.t. -x1 ≤ 0, -x2 ≤ 0, -x3 ≤ 0
y4 x1 + x2 + 3x3 ≤ 30
y5 2x1 + 2x2 + 5x3 ≤ 24
y6 4x1 + x2 + 2x3 ≤ 36
We should be able to add up the inequalities
y 1, y 2, y 3, y 4, y 5, y 6 ≥ 0
Example
maxx 3x1 + x2 + 2x3
y1
y2
y3
s.t. -x1 ≤ 0, -x2 ≤ 0, -x3 ≤ 0
y4 x1 + x2 + 3x3 ≤ 30
y5 2x1 + 2x2 + 5x3 ≤ 24
y6 4x1 + x2 + 2x3 ≤ 36
Coefficient of x1 should be 3
-y1 + y4 + 2y5 + 4y6 = 3
Example
maxx 3x1 + x2 + 2x3
y1
y2
y3
s.t. -x1 ≤ 0, -x2 ≤ 0, -x3 ≤ 0
y4 x1 + x2 + 3x3 ≤ 30
y5 2x1 + 2x2 + 5x3 ≤ 24
y6 4x1 + x2 + 2x3 ≤ 36
Coefficient of x2 should be 1
-y2 + y4 + 2y5 + y6 = 1
Example
maxx 3x1 + x2 + 2x3
y1
y2
y3
s.t. -x1 ≤ 0, -x2 ≤ 0, -x3 ≤ 0
y4 x1 + x2 + 3x3 ≤ 30
y5 2x1 + 2x2 + 5x3 ≤ 24
y6 4x1 + x2 + 2x3 ≤ 36
Coefficient of x3 should be 2
-y3 + 3y4 + 5y5 + 2y6 = 2
Example
maxx 3x1 + x2 + 2x3
y1
y2
y3
s.t. -x1 ≤ 0, -x2 ≤ 0, -x3 ≤ 0
y4 x1 + x2 + 3x3 ≤ 30
y5 2x1 + 2x2 + 5x3 ≤ 24
y6 4x1 + x2 + 2x3 ≤ 36
Upper bound should be tightest
miny 30y4 + 24y5 + 36y6
Dual
miny 30y4 + 24y5 + 36y6
s.t.
y 1, y 2, y 3, y 4 , y 5, y 6 ≥ 0
-y1 + y4 + 2y5 + 4y6 = 3
-y2 + y4 + 2y5 + y6 = 1
-y3 + 3y4 + 5y5 + 2y6 = 2
Original problem is called primal
Dual of dual is primal
Dual
maxx cTx
s.t. A x ≤ b
Dual
miny≥0 maxx cTx - yT(A x – b)
KKT Condition? ATy = c
miny≥0 bTy
s.t.
ATy = c
maxx cTx
Primal
s.t. A x ≤ b
miny≥0 bTy
s.t.
ATy = c
Dual
Strong Duality
p=
maxx cTx
Primal
s.t. A x ≤ b
If p ≠ ∞ or d ≠ ∞, then p = d.
Think back to the intuition of dual
d=
miny≥0 bTy
s.t.
ATy = c
Skipping the proof
Dual
Question
maxx cTx
s.t. A1 x ≤ b1
A2 x ≥ b2
A3 x = b3
Dual?
Outline
• Linear Programming
• Duality
• Solving the LP
Graphical Solution
x 1 + x2 = 8
x1 ≥ 0
x2 ≥ 0
4x1 – x2 ≤ 8
2x1 + x2 ≤ 10
5x1 - 2x2 ≥ -2
Optimal solution at a vertex
maxx x1 + x2
Graphical Solution
x1 = 3
x1 ≥ 0
x2 ≥ 0
4x1 – x2 ≤ 8
2x1 + x2 ≤ 10
5x1 - 2x2 ≥ -2
Optimal solution at a vertex
maxx x1
Graphical Solution
x2 = 6
x1 ≥ 0
x2 ≥ 0
4x1 – x2 ≤ 8
2x1 + x2 ≤ 10
5x1 - 2x2 ≥ -2
Optimal solution at a vertex
maxx x2
Graphical Solution
x1 ≥ 0
x2 ≥ 0
4x1 – x2 ≤ 8
2x1 + x2 ≤ 10
5x1 - 2x2 ≥ -2
An optimal solution always at a vertex
Proof?
maxx cTx
Solving the LP
• Dantzig (1951): Simplex Method
– Search over vertices of the polyhedra
– Worst-case complexity is exponential
– Smoothed complexity is polynomial
• Khachiyan (1979, 1980): Ellipsoid Method
– Polynomial time complexity
– LP is a P optimization problem
• Karmarkar (1984): Interior-point Method
– Polynomial time complexity
– Competitive with Simplex Method
Solving the LP
• Plenty of standard software available
• Mosek (http://www.mosek.com)
– C++ API
– Matlab API
– Python API
– Free academic license
Questions?