Convex Relaxations
Ben Recht
May 2, 2004
Outline
•
•
•
•
Lagrangian Duality
Linear Programming and Combinatorics
Non-convex quadratic programming
Positivstellensatz and Polynomial Programming
Lagrangian Duality
• General Problem
Lagrangian Duality
• General Problem
• Lagrangian:
Lagrangian Duality
• From Calculus: search for
• Lagrangian:
Lagrangian Duality
• Equivalent Optimization
• sup is infinite unless constraints satisfied
• Lagrangian:
Lagrangian Duality
• Equivalent optimization:
• Consider
• This is the dual problem
Visualization
f(x)
(g(x), h(x))
optimum
Search over half
spaces containing
the epigraph of the
function
(m,l,1)
Visualization
f(x*)
(m*,l*,1)
Visualization
f(x*)
(m*,l*,1)
Visualization
duality gap
Linear Programming Duality
• Lagrangian
Linear Programming Duality
• Minimize with respect to x
Linear Programming Duality
• Minimize with respect to x
Linear Programming Duality
• Minimize with respect to x
Either 0 or -1
Linear Programming Duality
• Minimize with respect to x
Either 0 or -1
Independent of
x
Linear Programming Duality
• Form the Dual
Linear Programming Duality
• Primal
• Dual
Integer Programming
• Primal
Integer Programming
• Primal
• Dual
Integer Programming
• Primal
• Dual
the same dual – dual dual is just the LP without integer constraints
Integer Programming and
Combinatorics
• Primal Dual Methods (shortest path, network flows)
• Total Unimodularity
– Guarantee integer solutions
• Total Dual Integrality and Min-max Relations
– Prove problem in NPÅcoNP
• Branch and Bound, Branch and Cut
Quadratic Programming
• Problem is convex only when the Ai are positive
semidefinite.
Nonconvex Quadratically
Constrained Quadratic Programs
• Consider the general problem: no assumption on
definiteness.
NCQ2P
• Consider the general problem: no assumption on
definiteness.
NCQ2P
• Simplified presentation
NCQ2P
• Form the Lagrangian
NCQ2P
• Form the Lagrangian
NCQ2P
• Form the Lagrangian
Taking inf over x gives 0 or -1
NCQ2P
• Dual
• This is a semidefinite program
NCQ2P
• Dual
• Dual Dual
NCQ2P
SDP
Relaxation
Dual Dual
Dual Dual
Dual Dual
Dual Dual
Dual Dual
• Recovered same
relaxation
• This technique doesn’t
generalize (duality
does!)
Bounding the gap
• For Aº0
Bounding the gap
• For Aº0
Bounding the gap
• For Aº0
Bounding the gap
• For Aº0
Bounding the gap
• For Aº0
• Take x=sign(y), y~N (0,Z*). Then
The MAX-CUT Relaxation
• Invented by Goemans and Williamson
• Guarantees accuracy of 88% for the MAX-CUT
problem. An algorithm with accuracy of 95%
would prove P=NP.
• Specific instance of the “A0” matrix in the relaxation
we discussed.
• Generalizes to MAX-2-SAT, MAX-SAT, graph
coloring, MAX-DICUT, etc.
MAX-CUT
• Let G=(V,E) be a graph and let w:E! R be an
arbitrary function. A cut in the graph is a partition
of the vertices into two disjoint sets V1 and V2 such
that V1 [ V2 = V. Let F(V1) denote the set of edges
which have exactly one node in V1.
• The weight of the cut is defined w(F) = f2 F w(f)
• Problem: find the partition which maximizes w.
• Graph: G=(V,E)
• Maximum-Cut
• Graph: G=(V,E)
• Maximum-Cut
• Graph: G=(V,E)
• Maximum-Cut
• Graph: G=(V,E)
• Maximum-Cut
Easy for bipartite graphs. In general, NP-Hard
Petersen Graph
Classic Counterexample
Maximum-Cut = 12
Petersen Graph
Classic Counterexample
Maximum-Cut = 12
Algorithm 1: Erdős
• Expected Error is 50%
Algorithm 1: Erdős
• Expected Error is 50%
• Flip a coin for each node
Algorithm 1: Erdős
• Expected Error is 50%
• Flip a coin for each node
• Probability edge is cut=1/2
Algorithm 1: Erdős
• Expected Error is 50%
25% error
• Flip a coin for each node
• Probability edge is cut=1/2
• State of the art until 1993
Graph Laplacians
• The Laplacian is the |V|£ |V| matrix defined by
• where Adj(v) is the set of vertices adjacent to v.
MAX-CUT as an IQP
• We can write the MAX-CUT problem as
• Or, using the Laplacian, we can write this as
and use the SDP techniques to find an approximate
answer
Analysis
• FACT 1: For -1 · x · 1,
2/ arccos(x) ¸ (1-x)
with  ¸ 0.87856
• FACT 2: If y is drawn randomly from a Gaussian
with zero mean and covariance Z
Pr[sign(yi)  sign(yj)] = 1/ arccos(Zij)
Analysis (continued)
• For any edge e2 E, let e denote the indicator function for e
in the cut. Then the expected value of a cut is
• So we have E[cut] ¸  (1/4)Tr(LZ) ¸ MAX CUT(G)
LPs and emptiness
A2x > b2
• To prove there is no
intersection, must find
positive l1 and l2 such that
for all x
A1x < b1
l1(b1 - A1x) + l2(A2x – b2) < 0
LPs and emptiness
A2x > b2
• To prove there is no
intersection, must find
positive l1 and l2 such that
for all x
A1x < b1
l1(b1 - A1x) + l2(A2x – b2) < 0
Main Idea: Positive combinations of
positive terms cannot be negative!
Functions and Emptiness
g2(x) > 0
• To prove there is no
intersection, find positive
functions l1(x) and l2(x) such
that for all x
g1(x) < 0
l1(x)(-g1(x)) + l2(x)g2(x) < 0
Main Idea: Positive combinations of
positive terms cannot be negative!
Generalizing Lagrangian Duality
• Recall
• where
Positivstellensatz
• Let f1,…,fn, g1,…,gm be functions in a class F
• Then W=; if and only if there exist mI(x)¸0 and lj(x)
in F such that
Positivstellensatz
• True for smooth functions
• True for polynomials
• Here, search for multipliers can be posed as a
semidefinite program
• Hierarchies of approximations
Polynomials and Emptiness
g2(x) > 0
• To prove there is no
intersection, find positive
functions l1(x) and l2(x) such
that for all x
g1(x) < 0
g(x) = a4 x4+ a3 x3 + a2 x2
+ a1 x + a0
l1(x)(-g1(x)) + l2(x)g2(x) < 0
• When g1 and g2 are
polynomials, we can assume l1
and l2 are polynomials
Cones and Ideals
• The cone generated by polynomials f1,…,fn is the set
of polynomials
where the ab(x) are all positive
• N.B. If all of the fi are nonnegative everywhere,
then any polynomial in the cone is nonnegative
everywhere
Cones and Ideals
• The ideal generated by polynomials g1,…,gn is the set
of polynomials
where the q(x) are arbitrary
• N.B. If all of the gi are zero everywhere, then any
polynomial in the ideal is zero everywhere
Cones and Ideals
• The cone generated by f1,…,fn is (ab(x)>0)
• The ideal generated by g1,…,gn is
Positivstellensatz
Let f1,…,fn, g1,…,gm be polynomials and
Then W=; if and only if there exists a F(x) in the cone
of the fi’s and a G(x) in the ideal of the gi’s such that
Same Interpretation
• If ai > 0 and bj = 0 and
with all li positive, then either one of the ai is negative
or one of the bj is not zero.
Main Idea: Positive combinations of
positive terms cannot be negative!
SOS/SDP Theorem
• One can search for certificates to prove that W=;
using semidefinite programming (Parillo 2000).
• Let x be a vector of all monomials in L variables
of degree less than or equal to N/2. Write
Then if Qº0, p is a positive polynomial and hence
W=;.
Two kinds of complexity
• Degree of polynomial multipliers
• Conditioning of numerical search
• The interplay between the two can be quite intricate
Looks hard is easy
f(x) - y < 0
• Not convex, not
separable by a line
y - f(x) + c > 0
Looks hard is easy
f(x) - y < 0
• Not convex, not
separable by a line
• Proof:
(-f(x) + y) + (y - f(x) - c) =
-c
• Fragility is the size of c
y - f(x) - c > 0
More involved
(x-1)2 + (y-1)2 < 1
x3-8x-2y = 0
More involved
(x-1)2 +
(y-1)2 <
1
• Explicitly: find a and b such
that
(x-1)2 + (y-1)2 – 1 + (ax + b)(x3 – 8x – 2y) < 0
x3-8x-2y = 0
More involved
(x-1)2 +
(y-1)2 <
1
• Explicitly: find a and b such
that
(x-1)2 + (y-1)2 – 1 + (ax + b)(x3 – 8x – 2y) < 0
• Using SDP we find a
certificate:
b = 0.31625 and a = -0.1517
x3-8x-2y = 0
Pretty Darned Hard
x>0, y>0
x2y+xy2-3xy+1<0
• 2nd order
multipliers
• Ill conditioning
Looks easy is hard
• Does the square leave the
ellipse?
Square: x2 < 1, y2 < 1
Ellipse: x2+y2 - xy –x –y > 4.5
Looks easy is hard
• Does the square leave the
ellipse?
• Need quadratic
multipliers for proof
• 3 corners equidistant
from the ellipse boundary
Square: x2 < 1, y2 < 1
Ellipse: x2+y2 - xy –x –y > 4.5
Is there a crash?
• Given: a line in the plane
a0 + a1x + a2y = 0
• Question: does it hit a
corner of the square?
x2=1, y2=1
Is there a crash?
• Given: a line in the plane
a0 + a1x + a2y = 0
• Question: does it hit a
corner of the square?
x2=1, y2=1
• If it hits a corner, then the
problem is fragile
Is there a crash?
• Given: a line in the plane
a0 + a1x + a2y = 0
• Question: does it hit a
corner of the square?
x2=1, y2=1
• Inside the square, quadratic
multipliers are needed.
• This is the first nontrivial
duality gap for NCQ^2P
Software
•
•
•
•
SOS tools
Yalmip
SeDuMi
Write your own (see Boyd and Vandenberghe)
Problem 1 – Spin Glasses
• Try to solve problem 13.1 in NMM using a
semidefinite relaxation. Estimate the duality gap
using your primal and dual solutions. Compare your
answer to the one returned by simulated annealing.
• Now assume that the spins are coupled together on
an 10x10 square lattice. Estimate the lowest energy
state using simulated annealing and the semidefinite
relaxation and compare the results.
Problem 2 – MAX 2 SAT
• A clause is the OR of two Boolean variables or their
negation. (e.g. xÇ~y)
• Given a Boolean expression which is the AND of a
bunch of clauses with two variables, the MAX 2
SAT problem is to determine how the maximum
number of clauses that can be simultaneously
satisfied.
Problem 2 – Continued
•
•
•
•
•
Let xi be true if xi = zi where zi=§ 1
Let v(x) = (1+zx)/2 and v(~x)=(1-zx)/2
Show that v(x)=1 if x is TRUE and 0 if x is FALSE
Let
Given a clause, show that C(x,y) is TRUE if and
only if v(x,y)=1. Show that C(x,y) is false if and
only if C(x,y)=0.
• Combine these facts to write MAX 2 SAT as an
integer quadratic program.
Problem 2 – Continued
• Use the analysis from above to show that the
semidefinite relaxation of the quadratic program
returns an -approximation of the MAX 2 SAT
problem.