Semidefinite Programming and Approximation
Algorithms for NP-hard Problems: A Survey
Sanjeev Arora
Princeton University
http://intractability.princeton.edu
Arora: SDP + Approx Survey
NP-completeness
Thousands of problems are NP-complete
(TSP, Scheduling, Circuit layout, Machine Learning,..)
Pragmatic Researcher
“Why the fuss? I am perfectly content with approximately
optimal solutions.” (e.g., cost within 10% of optimum)
Good news: Possible for a few problems. (“Approximation
Algorithms”)
Bad News: NP-hard for many problems. (“PCPs”)
Arora: SDP + Approx Survey
Approximation Algorithms
MAX-3SAT: Given 3-CNF formula , find assignment
maximizing the number of satisfied clauses.
An -approximation algorithm is one that for every
formula, produces in polynomial time an assignment that
satisfies at least OPT/ clauses. ( > 1).
Good News: [KarloffZwick’97] 8/7-approximation algorithm.
Bad News: [Hastad’97] If P  NP then for every  > 0, an
(8/7 -)-approximation algorithm does not exist.
Many similar results...
Arora: SDP + Approx Survey
Example: 2-approximation for Min Vertex Cover
G= (V, E)
Vertex Cover = Set of vertices that
touches every edge
“LP Relaxation”
0 · xi · 1
most
Claim: Value at least OPT/2
Proof: On Complete Graph Kn,
Proof: “Rounding”
OPT = n-1
but setting all xi = 1/2
gives feasible LP soln
Arora: SDP + Approx Survey
General Philosophy…
Interested in: NP-hard
Minimization Problem
Value = OPT
Write tractable
relaxation
value=
Round to get a solution of cost
= Approximation ratio
Arora: SDP + Approx Survey
= Integrality gap
SDP = Generalization of linear programming;
“vector programming”
Graph
Vector Representation.
(Inner products satisfy
some linear constraints)
Developed in 1970s as one of many flavors of
nonlinear optimization.
Can be solved in poly time (GLS’81).
Has many applications in operations research, control theory,
approximation algorithms for NP-hard problems.
Arora: SDP + Approx Survey
Main Idea in SDP: “Simulate” nonlinear
programming by convex program
Nonlinear program for Vertex Cover
SDP relaxation:
New variable
intended to stand for
“Vector Programs.”
Arora: SDP + Approx Survey
Homogenized
Take home message…
SDP gives best approximation known for host of
NP-hard problems (and algorithms can be made
highly efficient):
• Vertex Cover
• Sparsest Cut and most graph partitioning problems
• Graph coloring
•Max-cut, and every “Constraint Satisfaction Problem”….
Analysis of these algorithms used interesting geometric
ideas, which have had other applications.
Compelling evidence from complexity theory that no poly-time
algorithm can do better than many of these SDP-based
algorithms.(Novel interplay between SDP, reductions,
high-dimension geometry….)
Arora: SDP + Approx Survey
Outline: SDPs & Approximation
• SDP and its use in approximation: Generations 1 & 2
• Understanding SDPs <-> high dimensional geometry
• Faster algorithms (multiplicative update rule)
• Limitations of SDPs, Unique Games Conjecture
• Future directions
• Open problems
Arora: SDP + Approx Survey
How do you understand these
vector programs?
Ans. Interesting geometric analysis
Arora: SDP + Approx Survey
Understanding SDPs <--> Understanding
phenomena in high-dimensional geometry
Vertex Cover SDP
computes c-approximation
for c < 2 iff following is true
[GK96]
Vertices: n unit vectors
Edges: almost-antipodal pairs
Rn
Every graph in this family has an
independent set of size
Thm [Frankl-Rodl’87] False.
Arora: SDP + Approx Survey
SDP rounding: The two generations
Generation 1: *Uses random hyperplane as in [GW];
* Edge-by-edge analysis
Max-2SAT and Max-CUT [GW’94] ;Graph coloring
[KMS’95]; MAX-3SAT [KZ’97]; Algorithms for
Unique Games;..
Generation 2: Global rounding and analysis
Graph partitioning problems [ARV’04],
Graph deletion and directed partitioning problems [ACMM’05],
New analysis of graph coloring [ACC’06]
Disproof of UGC for expanding constraints [AKKSTV’08]
Recently, generation 1.5: “Squish ‘n solve” rounding.
k-CSPs
[RS’09]
Arora: SDP + Approx Survey
1st Generation Rounding: Ratio 1.13.. for MAX-CUT
[GoemansWilliamson’93]
G = (V,E) Find
that maximizes capacity
Quadratic Programming Formulation
Semidefinite Relaxation [DP ’91, GW ’93]
Arora: SDP + Approx Survey
.
Randomized Rounding (1st Gen)
Rn
v2
v1
v6
v3
v5
[GW ’93]
Form a cut by partitioning v1,v2,...,vn
around a random hyperplane.
SDPOPT
vi
ij
vj
Old math rides to the rescue...
Arora: SDP + Approx Survey
Surely this bizarre algorithm is not
the “right” way to solve max cut??
Arora: SDP + Approx Survey
Fact 1: No rounding algorithm can produce a better
solution out of this SDP [Feige-Schechtman]
“Edges between all pairs of vectors
making an angle 138 degrees.”
Fact 2: If P NP then impossible to get 1.06-approximation
in poly time [Hastad’97]
Fact 3: If “unique games conjecture” is true, no better than
1.13-approximation possible in poly time.[KKMO’05]
(i.e., algorithm on prev. slide is optimal)
Arora: SDP + Approx Survey
2nd Generation:
for c-balanced separator
G= (V, E); constant c >0
1
-1
Goal: Find cut
s.t. each side contains at
least c fraction of nodes and
minimized
SDP:
“Triangle
inequality”
Arora: SDP + Approx Survey
Angle subtended by the line joining
two of them on the third is non-obtuse;
“
“ condition.
Rounding algorithm for
-approximation
[ARV’04]
1. Pick random hyperplane
S
T
2. Remove points in “slab” of width
3. Remove any pair (i, j) that lie on opp.
sides of slab but
4. Call remaining sets S, T. Do BFS from
S to T according to distance
S
T
5. Output level of BFS tree with least # of edges.
Heart of analysis: Shows |S|, |T| = W(n);
“Large well-separated sets”
Arora: SDP + Approx Survey
Geometric fact underlying the analysis
(restatement of [ARV04] “Structure Theorem” by [AL06])
Vertices: unit vectors
satisfying “triangle inequality”
If
then no graph
in this family is an “expander.”
(“expander” : |(S)| =W(|S|) )
Edges:
Proof: Difficult “chaining” argument.
(Aside: Has been used to prove that l1 embeds into l2 with distortion
[CGR’05,ALN’06])
Arora: SDP + Approx Survey
Next few slides: Results showing Approximation is
hard assuming Unique-Games-Conjecture (UGC)
Recall:
Integrality gap of an SDP = min c st
<= c
Let “tough instance” = problem instance with
integrality gap c
These play crucial role in above reductions!!
Arora: SDP + Approx Survey
Unique Games
Given: Number p, and m equations in n vars of the form:
Promise: Either there is a solution that satisfies
fraction
of constraints or no solution satisfies even
fraction.
UGC [Khot’02]: Deciding which case holds is NP-hard.
[Raghavendra’08; building upon [KKMO’04][MOO’05]]
UGC  For every MAX-CSP, the simplest SDP relaxation
is the best possible poly-time approximation.
Arora: SDP + Approx Survey
Anatomy of a UGC-based hardness result
(eg Khot-Regev, KKMO, Raghavendra08)
Variables
Equations
Interpret as a graph
Prove using harmonic
analysis that near-optimum solns
correspond to good Equations
solution to the
unique game
Variables
Replace edges/vertices with gadgets
involving “integrality gap instance”
Arora: SDP + Approx Survey
Generation 1.5 rounding: Squish-n-solve
(Raghavendra-Steurer’09; provably optimal approximation for all
k-CSPs if UGC is true)
Project to random t-dim
subspace; t =O(1)
Merge nodes
whose vectors
are close
together; get
instance of
size exp(t) and
solve it optimally.
If g= approx. ratio for this algorithm then it is also the
integrality gap for SDP, and also the best possible
approx. ratio (assuming UGC).
Arora: SDP + Approx Survey
Issue of Running Time
Solving SDPs with m constraints takes
time.
m =n3 in some of these SDPs!
Next slide: Often, can reduce running time: O(n2) or O(n3)
[AHK’05], [AK’07] even O(n) for CSPs! [St’10]; O(n1.5 +m) for sparsest
cut [S’09]
Main idea: “Primal-dual schema.” Solve to approximate optimality;
using insights from the rounding algorithms.
“Multiplicative Weight-Update Rule for psd matrices”
Arora: SDP + Approx Survey
Primal-dual approach for SDP relaxations (contd.)
[A., Kale’07]
At step t:
Primal player: PSD matrix Xt; candidate primal
Dual player:
“Candidate slack matrix”
Mt
Let me run the rounding
algorithm on Pt, get a primal
integer candidate and point out
how pitiful it is.
Primal player: Xt+1 = exp(- t Mt)
(Analysis uses formal analogy between real #s and
symmetric matrics:
[Other ingredients: flow computations, eigenvalues,
dimension reduction tricks, etc.; simplified by [KRV07], [KRVV08]]
Arora: SDP + Approx Survey
Future directions….
Arora: SDP + Approx Survey
Direction 1: How to soup up the relaxation
• Other forms of convex relaxations instead of SDP?
(e.g. geometric programming, convex programming)
For a start, re-derive existing approximation ratios
(eg 0.878.. For MAX-CUT) using the above.
Arora: SDP + Approx Survey
Direction 2: Understand “Lifted” SDP relaxations
Recall: SDP tries to “simulate” nonlinear programming;
Variable for
Why not take it to the next level? Variables
for products of up to k variables.
This is the main idea of Lovasz-Schrijver’91, Sherali-Adams,
Lasserre etc. Yields better approx for hypergraph I.S. [CS’08]
Don’t seem to help for some problems, even for k =n/100:
MAX-k-SAT [BGHMP’06], [AAT’05], Vertex Cover[ABLT’06],[STT’07a+b]
MAX-CUT, Vertex Cover etc. [CMM’08] MAX-3SAT (Sch’09)
(v. subtle and beautiful ideas!)
Arora: SDP + Approx Survey
Direction 3: graph expansion
[ARV]; SDP w/ triangle ineq.
Approx Ratio
O(1) approx is UGC hard; as is improving
Cheeger for constt 
[Cheeger’72]; eigenvalue

Expansion

Lots of space for improvement even if UGC holds
Also, “small set expansion” problem is v. close to UG and
progress on it would possibly give progress on UG. [RS’10]
Arora: SDP + Approx Survey
Direction 4: Subexponential algorithms
• Inspiration: Unique Games with completeness 1- can
be approximately solved in exp(n) time [ABS’10]
(For problems like MAX3SAT, no such algorithms exist
if 3SAT has no subexp. algorithms)
Intriguing possibility: Many of the UGC-hard problems
have subexponential algorithms.
Another interesting idea: derive [ABS’10] algorithm for UG
using SDPs or Lasserre relaxations.
Arora: SDP + Approx Survey
Direction 5: SDP in Avg. Case Complexity
Problems like 3SAT seem difficult not only in the worst
case but also “on average.” (Needs careful definition!)
Theory of Avg Case complexity [Levin’84] doesn’t usually
apply to problems of practical interest (e.g., random 3SAT).
Recent development: Interreducibility among some
“average case” problems of interest. [Feige’01]; e.g
Easy “optimal” 7/8-hardness of MAX-3SAT.
SDP is used in the reduction! (Used to weed out
“uninteresting” cases)
Arora: SDP + Approx Survey
Direction 6: Applications to quantum computing
• PSD matrix of trace 1 = “density matrix”, a way to
describe a mixed quantum state
Recent result QIP=PSPACE (JUW’10) uses the [AK’07]
Primal-dual framework. (“Fast NC computation of
near-optimum quantum state”)
Expertise in designing specialized matrices for SDP
integrality gaps may prove useful in QC…
Arora: SDP + Approx Survey
Open problems
• Can Lasserre relaxations compute nontrivial
approximations to Vertex Cover, MAX-CUT, etc?
(ruled out already for MAX-3SAT [Sch’08])
• Generation 3 rounding?
• Iterative rounding for SDPs?
• Resolve UGC (eg disprove by giving truly subexp.
algorithms)
• SDP as a proof technique---apply to open problems of
circuit complexity, communication complexity etc.
Looking forward to many developments
THANK YOU!
Arora: SDP + Approx Survey
Classical MW update rule
(Example: predicting the market)
1$ won for correct prediction
1$ lost for incorrect prediction
• N “experts” on TV
• Can we perform as good as the best expert in hindsight?
Thm[Going back to Hannan, 1950s] Yes.
Arora: SDP + Approx Survey
Weighted Majority Algorithm (LW’94)
Losses
M1t
M2
t
• For each expert, weight wi. Initially wi 1
• Follow expert i w/ prob. proportional to wit
•Update weights according to
M3t Claim: Expected per-round loss of our algorithm
…
Arora: SDP + Approx Survey
Lagrangian method to approximately solve LPs
(PST’91, many others)
Losses
M1t
M2
t
M3t
…
Arora: SDP + Approx Survey
Experts
= Dual
constraints
; maintain
weighting
• For
eachLP
expert,
weight w
i. Initially wi 1
Loss vector
= expert
Slack vector
of candidate
dualtosoln
• Follow
i w/ prob.
proportional
w it
•Update weights according to
Claim:
a few rounds;
the average
of all
loss
Claim:After
Expected
per round
loss of
ourthealgorithm
vectors is an approximately feasible dual soln.
Lagrangian method to approximately solve LPs
(PST’91, many others since)
x = weighting of n experts; updated via multiplicative update
Loss vector
Expected loss
Only 1 constraint !
Claim: Expected per round loss of our algorithm
Average per-round loss of expert i = i’th coordinate of
Arora: SDP + Approx Survey
!
Lagrangian method to approximately solve SDPs
(A.,Kale ‘07)
psd
x =x=weighting
PSD matrix;
of n updated
experts; according
updated via
to multiplicative update
Loss vector
Expected loss
Claim: Expected per round loss of our algorithm
Arora: SDP + Approx Survey
!
SDPs and MW Updates: Primal-dual algorithm
Known: MW Update rule --> Approx. solutions to LPs
[PST’91, Y’95, GK’97,..etc.]
“experts” <-> constraints
“payoffs” <-> “slack in constraint”
[AK’07] Matrix MW update rule that uses formal analogy
between psd matrices and nonnegative real #s.
[Golden-Thompson]
(Spl. Case: LPs=
SDPs with 0’s on offdiagonals)
[Other ingredients: flow computations,
eigenvalues, dimension reduction tricks, etc.]
Arora: SDP + Approx Survey
Embeddings and Cuts
Thm[LLR94, AR94]: Integrality gap for SDP for
Nonuniform Sparsest Cut = Min distortion of any
embedding of
into
Rounding algorithm of [ARV04] gives insight into structure
of
; basis of new embeddings
Hardness results for sparsest cut yielded insights at the
heart of the embedding impossibility results.
Arora: SDP + Approx Survey
Limitations of SDPs
For many problems, we know neither an NP-hardness
result (via PCPs) nor a good SDP-based approach.
Can we show that known SDPs don’t work??
1st generation results: Specific SDPs don’t work
2nd Generation results: Large families of LPs or
SDPs don’t work
[ABL’02], [ABLT’06]: “Proving integrality gaps without
knowing the LP.”
Much subsequent work, especially on families obtained
from “lift and project” ideas)
Arora: SDP + Approx Survey
Example: 2-approximation for Min Vertex Cover
G= (V, E)
Vertex Cover = Set of vertices that
touches every edge
“LP Relaxation”
most
Claim: Value at least OPT/2
Proof: On Complete Graph Kn,
Proof: “Rounding”
OPT = n-1
but setting all xi = 1/2
gives feasible LP soln
Arora: SDP + Approx Survey
Next, briefly
Connection between analysis of SDPs and
Geometric Embedding of Metric Spaces
Arora: SDP + Approx Survey
Geometric embeddings of metric spaces
(X, d): metric space
y
d(x, y)
f(y)
f
x
f(x)
C = distortion
Thm (Bourgain’85) For every X, there is f s.t. C= O(log n).
Open qs since then: is it possible to achieve smaller C for
concrete X, say X =
?
[CGR’05,ALN05]: Yes, C
Via [LLR94,AR94] implies
Arora: SDP + Approx Survey
possible for X =
approx for general
sparsest cut
Main issue: Local versus Global
Example: [Erdos] There are graphs on n vertices that
cannot be colored with 100 colors yet
every subgraph on 0.01 n vertices is 3-colorable.
LP relaxations or SDP relaxations concern local
conditions.
How well do such local conditions capture global property
in question?
Results for MAX-k-SAT [], [AAT’05], Vertex Cover[ABLT’06],
[STT’07a+b] MAX-CUT, Vertex Cover
etc. [CMM’08]
“Lifted SDPs.” Connections to Proof Complexity.
Arora: SDP + Approx Survey