Semidefinite Programming and Approximation
Algorithms for NP-hard Problems: A Survey
Sanjeev Arora
Princeton University
Arora: SDP + Approx Survey
NP-completeness
Thousands of problems are NP-complete
(TSP, Scheduling, Circuit layout, Machine Learning,..)
Pragmatic Researcher
“Why the fuss? I am perfectly content with approximately
optimal solutions.” (e.g., cost within 10% of optimum)
Good news: Possible for a few problems. (“Approximation
Algorithms”)
Bad News: NP-hard for many problems. (“PCPs”)
Arora: SDP + Approx Survey
Talk Outline
• Defn of approximation and example
• SDP and its use in approximation
• Understanding SDPs <-> high dimensional geometry
• Faster algorithms (multiplicative update rule)
• Limitations of SDPs: local vs global issues
• Connections (a) metric spaces (b) avg case complexity
(c) unique games conjecture
• Open problems
Arora: SDP + Approx Survey
Approximation Algorithms
MAX-3SAT: Given 3-CNF formula !, find assignment
maximizing the number of satisfied clauses.
An "-approximation algorithm is one that for every
formula, produces in polynomial time an assignment that
satisfies at least OPT/" clauses. (" >= 1).
Good News: [KZ’97] An 8/7-approximation algorithm exists.
Bad News: [Hastad’97] If P ! NP then for every # > 0, an
(8/7 -#)-approximation algorithm does not exist.
(Similar results for many other problems…)
Arora: SDP + Approx Survey
Good news (for me)
Status of many basic problems is still unresolved:
• Vertex Cover
• Sparsest Cut and most graph partitioning problems
• Graph coloring
• Random instances of 3SAT
My feeling: Interesting algorithms remain undiscovered;
semidefinite programming (SDP) may be helpful.
SDP = Generalization of linear programming
Graph
Vector Representation
Arora: SDP + Approx Survey
Example: 2-approximation for Min Vertex Cover
G= (V, E)
Vertex Cover = Set of vertices that
touches every edge
“LP Relaxation”
most
Claim: Value at least OPT/2
Proof: On Complete Graph Kn,
Proof: “Rounding”
OPT = n-1
but setting all xi = 1/2
gives feasible LP soln
Arora: SDP + Approx Survey
General Philosophy…
Interested in: NP-hard
Minimization Problem
Value = OPT
Write tractable
relaxation
value=
Round to get a solution of cost
= Approximation ratio
= Integrality gap
Arora: SDP + Approx Survey
Main Idea in SDP: “Simulate” nonlinear
programming
Nonlinear program for Vertex Cover
SDP relaxation:
New variable
intended to stand for
Arora: SDP + Approx Survey
Homogenized
How do you understand these
vector programs?
Ans. Interesting geometric analysis
Arora: SDP + Approx Survey
Understanding SDPs <--> Understanding
phenomena in high-dimensional geometry
Vertex Cover SDP
computes c-approximation
for c < 2 iff following is true
Vertices: n unit vectors
Edges: almost-antipodal pairs
Rn
Every graph in this family has an
independent set of size
Thm [Frankl-Rodl’87] False.
Arora: SDP + Approx Survey
SDP rounding: The two generations
First generation: *Uses random hyperplane as in [GW];
* Edge-by-edge analysis
Max-2SAT and Max-CUT [GW’94] ;Graph coloring
[KMS’95]; MAX-3SAT [KZ’97]; Algorithms for
Unique Games;..
Second generation: Global rounding and analysis
Graph partitioning problems [ARV’04],
Graph deletion and directed partitioning problems [ACMM’05],
New analysis of graph coloring [ACC’06]
Disproof of UGC for expanding constraints [AKKSTV’08]
(Similarly, two generations of results showing limits
on performance of SDPs)
Arora: SDP + Approx Survey
1st Generation Rounding: Ratio 1.13.. for MAX-CUT
[GoemansWilliamson’93]
G = (V,E) Find
that maximizes capacity
Quadratic Programming Formulation
Semidefinite Relaxation [DP ’91, GW ’93]
Arora: SDP + Approx Survey
.
Randomized Rounding
Rn
v2
v1
v6
v3
v5
[GW ’93]
Form a cut by partitioning v1,v2,...,vn
around a random hyperplane.
SDPOPT
vi
$ij
vj
Old math rides to the rescue...
Arora: SDP + Approx Survey
Fact 1: No rounding algorithm can produce a better
solution out of this SDP [Feige-Schectman]
“Edges between all pairs of vectors
making an angle 138 degrees.”
Fact 2: If P NP then impossible to get 1.09-approximation
by any efficient algorithm [Hastad’97]
Fact 3: If “unique games conjecture” is true, it is impossible
to get a better than 1.13-approximation.[KKMO’05]
(i.e., algorithm on prev. slide is optimal)
Arora: SDP + Approx Survey
2nd Generation:
for c-balanced separator
G= (V, E); constant c >0
1
-1
Goal: Find cut
s.t. each side contains at
least c fraction of nodes and
minimized
SDP:
“Triangle
inequality”
Angle subtended by the line joining
two of them on the third is non-obtuse;
“
“ condition.
Arora: SDP + Approx Survey
Rounding algorithm for
-approximation
[ARV’04]
1. Pick random hyperplane
S
T
2. Remove points in “slab” of width
3. Remove any pair (i, j) that lie on opp.
sides of slab but
4. Call remaining sets S, T. Do BFS from
S to T according to distance
S
T
Arora: SDP + Approx Survey
5. Output level of BFS tree with least # of edges.
Geometric fact underlying the analysis
(restatement of [ARV04] “Structure Theorem” by [AL06])
Vertices: unit vectors
satisfying “triangle inequality”
If
then no graph
in this family is an “expander.”
(“expander” : |%(S)|! &(|S|) )
Edges:
Proof is delicate and difficult
Arora: SDP + Approx Survey
Issue of Running Time
Solving SDPs with m constraints takes
time.
m =n3 in some of these SDPs!
Next few slides: Often, can reduce running time: O(n2) or O(n3).
[AHK’05], [AK’07]
Main idea: “Primal-dual schema.”Solve to approximate optimality;
using insights from the rounding algorithms.
“Multiplicative Weight-Update Rule for psd matrices”
Arora: SDP + Approx Survey
Classical MW update rule
(Example: predicting the market)
1$ for correct prediction
0$ for incorrect
• N “experts” on TV
• Can we perform as good as the best expert ?
Thm[Going back to Hannan, 1950s] Yes.
Arora: SDP + Approx Survey
Weighted Majority Algorithm [LW’94]
“Predict according to the weighted majority”
• Maintain a weight for each expert. Initially
• At step t, if expert i’s prediction was incorrect,
Claim: Expected Payoff of our algorithm
Similar algorithms discovered in a variety of areas:
decision theory, learning theory (“boosting”),
cryptography (“hardcore sets”), approx soln of LPs,..
(see survey [A, Hazan, Kale])
Arora: SDP + Approx Survey
Primal-dual approach for SDP relaxations
[A., Kale’07]
At step t:
Primal player: PSD matrix Pt; candidate primal
Let me run the rounding
algorithm on Pt, get a primal
integer candidate and point out
how pitiful it is.
Dual player:
“Feedback”
matrix Mt
Primal player: Pt+1 = exp(-# "t Mt)
(Analysis uses formal analogy between real #s and
symmetric matrics:
[Other ingredients: flow computations,
eigenvalues, dimension reduction tricks, etc.]
Arora: SDP + Approx Survey
Implications for geometric embeddings of
metric spaces
(X, d): metric space
d(x, y)
y
f(y)
f
x
f(x)
C = distortion
Thm (Bourgain’85) For every X, there is f s.t. C= O(log n).
Open qs since then: is it possible to achieve smaller C for
concrete X, say X =
?
[CGR’05,ALN05]: Yes, C
[KV06]: Cannot reduce C below
Arora: SDP + Approx Survey
possible for X =
Unique Games
Given: Number p, and m equations in n vars of the form:
Promise: Either there is a solution that satisfies
fraction
of constraints or no solution satisfies even
fraction.
UGC [Khot’02]: Deciding which case holds is intractable.
Seems to capture our current limitations of thinking about
SDPs; basis of many recent “hardness” results.
Arora: SDP + Approx Survey
Anatomy of a UGC-based hardness result
Variables
Equations
Interpret as a graph
Prove using harmonic
analysis that near-optimum solns
correspond to good Equations
Variables
Solution to the
unique game
Replace edges/vertices
with hypercube-like gadgets
Arora: SDP + Approx Survey
Limitations of SDPs
For many problems, we know neither an NP-hardness
result (via PCPs) nor a good SDP-based approach.
Can we show that known SDPs don’t work??
1st generation results: Specific SDPs don’t work
2nd Generation results: Large families of LPs or
SDPs don’t work
[ABL’02], [ABLT’06]: “Proving integrality gaps without
knowing the LP.”
Much subsequent work, especially on families obtained
from “lift and project” ideas)
Arora: SDP + Approx Survey
“Lifted” SDP relaxations
Recall: SDP tries to “simulate” nonlinear programming;
Variable for
Why not take it to the next level? Variables
for products of up to k variables.
This is the main idea of Lovasz-Schrijver’91, Sherali-Adams,
Lasserre etc.
“SDP as a proof system”: Integrality gaps proved in 2nd
generation results.
Arora: SDP + Approx Survey
Main issue: Local versus Global
Example: [Erdos] There are graphs on n vertices that
cannot be colored with 100 colors yet
every subgraph on 0.01 n vertices is 3-colorable.
LP relaxations or SDP relaxations concern local
conditions.
How well do such local conditions capture global property
in question?
Results for MAX-k-SAT [], [AAT’05], Vertex Cover[ABLT’06],
[STT’07a+b] MAX-CUT, Vertex Cover
etc. [CMM’08]
“Lifted SDPs.” Connections to Proof Complexity.
Arora: SDP + Approx Survey
Connections to Avg. Case Complexity
(SDP used in reductions)
Problems like 3SAT seem difficult not only in the worst
case but also “on average.” (Needs careful definition!)
Theory of Avg Case complexity exists, but doesn’t usually
apply to problems of practical interest.
Recent development: Interreducibility among some
“average case” problems of interest. [Feige’01][Alekh.03]
SDP is used in the reduction!
Arora: SDP + Approx Survey
Open problems
• Techniques for proving lowerbounds on lifted SDPs.
(difficult local-global results)
• New rounding algorithms
• Clarify nature of connection to average case
complexity.
• Resolve UGC (recently, disproof of UGC when the
constraint graph is an expander.[AKKSTV08]
• SDP as a proof technique---apply to open problems of
circuit complexity, communication complexity etc.
Looking forward to many developments
THANK YOU!
Arora: SDP + Approx Survey
SDPs and MW Updates: Primal-dual algorithm
Known: MW Update rule --> Approx. solutions to LPs
[PST’91, Y’95, GK’97,..etc.]
“experts” <-> constraints
“payoffs” <-> “slack in constraint”
[AK’07] Matrix MW update rule that uses formal analogy
between psd matrices and nonnegative real #s.
[Golden-Thompson]
(Spl. Case: LPs=
SDPs with 0’s on offdiagonals)
[Other ingredients: flow computations,
eigenvalues, dimension reduction tricks, etc.]
Arora: SDP + Approx Survey
Embeddings and Cuts
Thm[LLR94, AR94]: Integrality gap for SDP for
Nonuniform Sparsest Cut = Min distortion of any
embedding of
into
Rounding algorithm of [ARV04] gives insight into structure
of
; basis of new embeddings
Hardness results for sparsest cut yielded insights at the
heart of the embedding impossibility results.
Arora: SDP + Approx Survey