Approximations for Isoperimetric and Spectral Profile and Related Parameters Prasad Raghavendra MSR New England S joint work with David Steurer Prasad Tetali Princeton University Georgia Tech Graph Expansion d-regular graph G with n vertices expansion(S) = # edges leaving S d |S| d vertex set S A random neighbor of a random vertex Conductance of Graph G in S is outside of S with probability expansion(S) ФG = minimum expansion(S) |S| ≤ n/2 Extremely well-studied, many different contexts pseudo-randomness, group theory, online routing, Markov chains, metric embeddings, … Uniform Sparsest Cut Problem Given a graph G compute ФG and find the set S achieving it. S Measuring Graph Expansion d-regular graph G expansion(S) = with n vertices # edges leaving S d |S| d Conductance of Graph G vertex set S ФG = minimum expansion(S) |S| ≤ n/2 Complete Graph Path Complete Graph Complete graphs with a perfect matching Typically, small sets expand to a greater extent. Isoperimetric Profile expansion(S) = S # edges leaving S d |S| Isoperimetric Profile of Graph G Ф(δ) = minimum |S| ≤ δn expansion(S) 1 • Conductance function – defined by [Lovasz-Kannan] used to obtain better mixing time bounds for Markov chains. Ф(δ) Set Size δ 0.5 •Decreasing function of δ Approximating Isoperimetric Profile 1 expansion(S) = # edges leaving S d |S| Isoperimetric Profile of Graph G Ф(δ) = Ф(δ) minimum |S| ≤ δn expansion(S) S Set Size δ 0.5 What is the value of Ф(δ) for a given graph G and a constant Uniform Sparsest Cut: Determine the lowest point on the curve. δ > 0? Gap Small Set Expansion Problem (GapSSE)(η, δ) Given a graph G and constants δ, η > 0, Is Ф(δ) < η OR Ф(δ) > 1- η? ----Closely tied to Unique Games Conjecture (Last talk of the day “Graph Expansion and the Unique Games Conjecture” [R-Steurer 10]) Algorithm Theorem For every constant δ > 0, there exists a poly-time algorithm that finds a set S of size O(δ) such that expansion (S) ≤ O( ( ) log( 1 / ) ) -- A (Ф(δ) vs ( ) log( 1 / ) ) approximation • For small enough δ, the algorithm cannot distinguish between Ф(δ) < η OR Ф(δ) > 1- η Theorem [R-Steurer-Tulsiani 10] An improvement over above algorithm by better than constant factor, would help distinguish Ф(δ) < η OR Ф(δ) > 1- η A Spectral Relaxation 0 Let x = (x1 , x2 , .. xn) be the indicator function of the unknown least expanding set S SC 2 ( x x ) i j Number of edges leaving S = 1S ( i , j )E Relaxing 0,1 to real numbers ( ) min n 2 ( x x ) i j ( i , j )E xx {0 ,1}n d xi |Support(x)| < δn i 2 |S| = xi2 i x T Lx minn T x {0 , 1} d x x x n Why is it spectral? Spectral Profile: ( ) minn [Goel-Montenegro-Tetali] 2 ( x x ) i j ( i , j )E x |Support(x)| < δn d xi 2 i x T Lx min T x n dx x |Support(x)| < δn Smallest Eigen Value of Laplacian: min n x x 1 (x i ( i , j )E xj) d xi i 2 2 x T Lx min T n x d x x x 1 Observation: Λ(δ) is the smallest possible eigenvalue of a submatrix L(S,S) of size at most < δn of the Laplacian L . Rounding Eigenvectors Cheeger’s inequality There is a sparse cut of value at most 2 Smallest Eigen Value of Laplacian Rounding x x T Lx minn x d xT x x 1 ( ) minn x |Support(x)| < δn (x i ( i , j )E xj) d xi i 2 2 x T Lx min x n d xT x |Support(x)| < δn 0 Lemma: There exists a set S of volume at most δ whose expansion is at most 2( ) Spectral Profile [Goel-Montenegro-Tetali] ( ) minn x |Support(x)| < δn 2 ( x x ) i j ( i , j )E d xi i 2 x T Lx min T x n dx x |Support(x)| < δn Unlike eigen values, Λ(δ) is not the optimum of a convex program. We show an efficiently computable SDP that gives good guarantee. Theorem: There exists an efficiently computable SDP relaxation Λ* (δ) of Λ(δ) such that for every graph G Λ* (δ) ≤ Λ(δ) ≤ Λ* (δ/2)∙O(log 1/δ) Recap Theorem: (Approximating Spectral Profile) There exists an efficiently computable SDP relaxation Λ* (δ) of Λ(δ) such that for every graph G Λ* (δ) ≤ Λ(δ) ≤ Λ* (δ/2)∙O(log 1/δ) Lemma: (Cheeger Style Rounding) There exists a set S of volume at most δ whose expansion is at most 2( ) Theorem (Approximating Isoperimetric Profile) For every constant δ > 0, there exists a poly-time algorithm that finds a set S of size O(δ) such that expansion (S) ≤ O( ( ) log( 1 / ) ) Restricted Eigenvalue Problem Given a matrix A, find a submatrix A[S,S] of size at most δn X δn matrix with the least eigenvalue. -- Our algorithm is applicable to diagonally dominant matrices (yields a log(1/δ) approximation). Approximating Spectral Profile ( ) minn (x i ( i , j )E x |Support(x)| < δn xj) d xi i 2 2 SDP Relaxation for ( ) minn 2 ( x x ) i j ( i , j )E d xi x |Support(x)| < δn i Replace each xi by a vector vi: Numerator | v ( i , j )E i Denominator v j |2 | v | ( i , j )E 2 i =n Without loss of generality, xi can be assumed positive. This yields the constraint: vi∙vj ≥ 0 Enforcing Sparsity: 2 2 xi | Support ( x) | xi i i By Cauchy-Schwartz inequality: v i i 2 n | vi |2 n 2 i 2 SDP Relaxation for ( ) minn 2 ( x x ) i j ( i , j )E x |Support(x)| < δn d xi i 2 | v v | i j Minimize (Sum of Squared Edge Lengths) ( i , j )E Subject to Positive Inner Products: Average squared length vi∙vj ≥ 0 =1 Average Pairwise Correlation < δ 2 | v | i n ( i , j )E 2 v v n i j i, j 2 Rounding Two Phase Rounding: • Transform SDP vectors in to a SDP solution with only nonnegative coordinates. • Use thresholding to convert non-negative vectors in to sparse vectors. Making SDP solution nonnegative Let v be a n-dimensional real vector. Let v* denote the unit vector along direction v. Map the vector v to the following function over Rn : fv = ||v||∙ (Square Root of Probability Density Function of n-dimensional Gaussian centered at v*) Formally, f v ( x) v ( x v* ) Where: Ф(x) = probability density function of a mean 0, variance σ spherical Gaussian in n-dimensions. Properties f v ( x) v ( x v* ) Where: Ф(x) = probability density function of a mean 0, variance σ spherical Gaussian in n-dimensions. SDP Constraint Average Pairwise Correlation < δ 2 v v n i j i, j Lemma: (Pairwise correlation remains low if σ is small) 2 2 | f i | (e 1/ 4 )n 2 (i , j )E Pick σ = 1/sqrt{log(1/δ)} Properties f v ( x) v ( x v* ) Where: Ф(x) = probability density function of a mean 0, variance σ spherical Gaussian in n-dimensions. Lemma: (Squared Distances get stretched by at most 1/σ2) |f1 – f2 |2 ≤ O(1/σ2) |v1 – v2 |2 For our choice of σ, squared distances are stretched by log(1/δ) . With a log(1/δ) factor loss, we obtaining a non-negative SDP solution. Rounding a positive vector solution Let us pretend the vectors vi are non-negative. i.e., vi(t) ≥ 0 for all t Rounding Non-Negative Vectors 1. Sample t 2. Compute threshold θ = (average of vi(t) over i) * (2/ δ ) 3. Set xi = max{ vi(t) – θ, 0 } for all i Observation: |Support(x)| < δ/2 ∙ n Observation: E[ (x x ) ( i , j )E i j 2 ] vi v j 2 Open Problem Minimize (Sum of Squared Edge Lengths) | v ( i , j )E Subject to Positive Inner Products: Average squared length i v j |2 vi∙vj ≥ 0 =1 Average Pairwise Correlation < δ 2 | v | i n ( i , j )E v i v j n 2 i, j Find integrality gaps for the SDP relaxation: Current best have δ = 1/poly(logn), Do there exist integrality gaps with δ = 1/poly(n)? Thank You Spectral Profile Second Eigen Value: Spectral Profile ( ) minn (x i ( i , j )E x |Support(x)| < δn xj) d xi 2 2 x T Lx min x n d xT x i Remarks: • Λ(δ) ≤ minimum of Ф(δ0) over all δ0 ≤ δ • Unlike second eigen value, Λ(δ) is not the optimum of a convex program. • Λ(δ) is the smallest possible eigenvalue of a submatrix of size at most < δn of the matrix L • xi can be assumed to be all positive. Small Sets via Spectral Profile Using an analysis along the lines of analysis of Cheeger’s inequality, it yields: Theorem [Raghavendra-Steurer-Tetali 10] There exists a poly-time algorithm that finds a set S of size O(δ) such that expansion (S) ≤ sqrt (Λ* (δ)∙O(log 1/δ)) So if there is a set S with expansion(S) = ε, then the algorithm finds a cut of size O( log 1 / ) Similar behaviour as the Gaussian expansion profile Spectral Profile Second Eigen Value: Spectral Profile ( ) minn (x ( i , j )E x |Support(x)| < δn xj) i d xi i Replace xi by vectors vi * ( ) min vi 2 x T Lx min x n d xT x 2 2 | v v | i j ( i , j )E d | vi |2 i Subject to v i i 2 n | vi |2 i vi∙vj ≥ 0 Graph Expansion d-regular graph G d vertex set S Approximation Algorithms: expansion(S) = # edges leaving S d |S| A random neighbor of a random vertexGin S Conductance of Graph is outside of S with probability expansion(S) ФG = minimum expansion(S) |S| ≤ n/2 Extremely well-studied, many different Uniform Sparsest Cut Problem contexts Given a graph G compute ФG pseudo-randomness, group theory, androuting, find the set S achieving it. online Markov chains, metric embeddings, … •Cheeger’s Inequality [Alon][Alon-Milman] Given a graph G, if the normalized adjacency matrix has second eigen value λ2 then, (1 2 ) G 2(1 2 ) 2 •A log n approximation algorithm [Leighton-Rao]. •A sqrt(log n) approximation algorithm using semidefinite programming [Arora-Rao-Vazirani]. Limitations of Eigenvalues • The best lower bound that Cheeger’s inequality gives on expansion is (1-λ2)/2 < ½, while Ф(δ) can be close to 1. • Consider graph G Connect pairs of points on {0,1}n that are εn Hamming distance away. Then Second eigenvalue ≈ 1- ε and ф(1/2) ≈ ε yet Ф(δ) ≈ 1 (small sets have near-perfect expansion) • A SIMPLE SDP RELAXATION cannot distinguish between - all small sets expand almost completely - exists small set with almost no expansion A Conjecture Small-Set Expansion Conjecture: 8η>0, 9 ± >0 such that GapSSE(η, ± ) is NP-hard, i.e., Given a graph G, it is NP-hard to distinguish YES: expansion(S) < η for some S of volume ¼ ± NO: expansion(S) > 1- η for all S of volume ¼ ± Road Map • Algorithm: – Spectral Profile. • Reductions within Expansion • Relationship with Unique Games Conjecture Gaussian Curve Gaussian Graph Vertices: all points in Rd (d dimensional real space) Edges: Weights according to the following sampling procedure: • Sample a random Gaussian variable x in Rd • Perturb x as follows to get y in Rd y (1 ) x 2 2 z Γε (δ) = Gaussian noise sensitivity of a set of measure δ = least expanding sets are caps/thresholds of measure δ 1 = ( log 1 / ) Ф(δ) Add an edge between x an y Set Size δ 0.5 Approximating Expansion Profile Reductions within Expansion Reductions within Expansion Theorem [Raghavendra-Steurer-Tulsiani 10] For every positive integer q, and constants ε,δ,γ, given a graph it is SSE-hard to distinguish between: • There exists q disjoint small sets S1 , S2 , .. Sq of size close to 1/q, such that expansion(Si) ≤ ε • No set of size μ> δ has expansion less than size Γε/2 (μ) -expansion of set of size μ in Gaussian graph with parameter ε/2 expansion (S) < sqrt (Λ* (δ)∙O(log 1/ μ)) Informal Statement 1 1 Ф(δ) 0.5 Set Size δ Set Size δ Quantitative Statement Qualitative Assumption Given a graph G , it is SSE-hard to distinguish whether, •There is a small set of size whose expansion is ε. GapSSE is NPhard •Every small set of size μ (in a certain range) expands at least as much as the corresponding Gaussian graph with noise ε O( log 1 / ) Corollaries Corollary: The algorithm in [Raghavendra-Steurer Tetali 10] has near-optimal guarantee assuming SSE conjecture. Corollary: Assuming GapSSE, there is no constant factor approximation for Balanced Separator or Uniform Sparsest Cut. Relation with Unique Games Conjecture Unique Games Unique game ¡ : label set A L(A) Referee graph of size n ¼ B Each edge (A,B) has an associated map ¼ ¼ is bijection from L(A) to L(B) A labelling satisfies (A,B) if ¼ (label of A) = label of B Goal: Find an assignment of labels to vertices such that maximum number of edges are satisfied. sample (A,B,¼) A B Player 1 Player 2 pick a in L(A) pick b in L(B) a b Referee players win if ¼(a) = b value( ¡ ): no communication between players maximum success probability over all strategies of the players Unique Games Conjecture [Khot02] Unique Games Conjecture: [Khot ‘02] 8²>0, 9 q >0: NP-hard to distinguish for ¡ with label set size q YES: value( ¡ ) > 1-² NO: value( ¡ ) < ² Implications of UGC BASIC SDP is optimal for … UGC Constraint Satisfaction Problems [Raghavendra`08] MAX CUT, MAX 2SAT Metric Labeling Problems [MNRS`08] MULTIWAY CUT, 0-EXTENSION Ordering CSPs [GMR`08] MAX ACYCLIC SUBGRAPH, BETWEENESS Strict Monotone CSPs [KMTV`10] VERTEX COVER, HYPERGRAPH VERTEX COVER Kernel Clustering Problems [KN`08,10] Grothendieck Problems [KNS`08, RS`09] … “Reverse Reductions” BASIC SDP optimal for PROBLEM X ? * UGC BASIC SDP is optimal for lots of optimization problems, e.g.: MAX CUT and VERTEX COVER Win-Win Situation If we could show * , then a refutation of UGC would imply an improved algorithm for PROBLEM X PROBLEM X = MAX CUT Parallel Repetition is natural candidate reduction for * [FeigeKinderO’Donnell’07] Bad news: this reduction cannot work [Raz’08, BHHRRS’08] Small Set Expansion and Unique Games • Solving Unique Games Finding a small nonexpanding set in the “label extended graph” Theorem [Raghavendra-Steurer 10] Small Set Expansion Conjecture Unique Games Conjecture Establishes a reverse connection from a natural problem. Implications of UGC BASIC SDP is optimal for … UGC UGC With expansion Gap SSE Constraint Satisfaction Problems [Raghavendra`08] MAX CUT, MAX 2SAT Metric Labeling Problems [MNRS`08] MULTIWAY CUT, 0-EXTENSION Ordering CSPs [GMR`08] MAX ACYCLIC SUBGRAPH, BETWEENESS Strict Monotone CSPs [KMTV`10] VERTEX COVER, HYPERGRAPH VERTEX COVER Kernel Clustering Problems [KN`08,10] Grothendieck Problems [KNS`08, RS`09] … Uniform Sparsest Cut [KNS`08, RS`09] Minimum Linear Arrangement[KNS`08, RS`09] Most known SDP integrality gap instances for problems like MaxCut, Vertex Cover, Unique games have graphs that are “small set expanders” Theorem [Raghavendra-Steurer-Tulsiani 10] Small Set Expansion Conjecture MaxCut or Unique Games on Small Set Expanders is hard. Reverse Connections? Approximating Spectral Profile Roadmap Introduction Graph Expansion: Cheeger’s Inequality, Leighton Rao, ARV Expansion Profile: Small Sets expand more than large ones. Cheeger’s inequality and SDPs fail GapSSE Problem Relation to Unique Games Conjecture: Unique Games definition, Applications, lack of reverse reductions. Label extended graph Small sets Small Set Expansion Conjecture UGC UGC with SSE is easy Algorithm for SSE: Spectral Profile, SDP for Spectral Profile, Rounding algorithm Relations within expansion: GapSSE Balanced Separator Hardness NP-hard Optimization Example: MAX CUT: partition vertices of a graph into two sets so as to maximize number of cut edges fundamental graph partitioning problem benchmark for algorithmic techniques Approximation MAX CUT Trivial approximation Random assignment, cut ½ of the edges First non-trivial approximation approx-ratio 0.878 Goemans–Williamson algorithm (‘95) based on a semidefinite relaxation (BASIC SDP) Beyond Max Cut: Analogous BASIC SDP relaxation for many other problems Almost always, BASIC SDP gives best known approximation (often strictly better than non-SDP methods) Approximation MAX CUT Trivial approximation Random assignment, cut ½ of the edges First non-trivial approximation approx-ratio 0.878 Goemans–Williamson algorithm (‘95) based on a semidefinite relaxation (BASIC SDP) Can we beat the approximation guarantee of BASIC SDP? Unique Games Conjecture Can we beat the approximation guarantee of BASIC SDP for Max Cut? No, assuming Khot’s Unique Games Conjecture! [KKMO`05, MOO`05, OW’08] Unique Games Conjecture [Khot’02] (roughly): it is NP-hard to approximate the value of Unique Games certain optimization problem: given equations of the form xi – xj = cij mod q satisfy as many as possible UGC: Is it true? What are hard instances? Any non-trivial consequences if UGC is false? UGC for certain SDP hierarchies UGC ? [RaghavendraS’09,KhotSaket’09] UGC on expanding instances [AKKSTV’08, AIMS’10] UGC on product instances [BHHRRS’08,Raz’08] “Reverse Reductions” BASIC SDP optimal for PROBLEM X UGC BASIC SDP is optimal for lots of optimization problems, e.g.: MAX CUT and VERTEX COVER SMALL-SET EXPANSION Win-Win Situation A refutation of UGC implies an improved algorithm for SMALL-SET EXPANSION (better than BASIC SDP!) First reduction from natural combinatorial problem to UNIQUE GAMES Approximating Small-Set Expansion Expansion profile of G at ±: minimum expansion(S) over all S with volume < ± How well can we approximate the expansion profile of G for small ± ? BASIC SDP cannot distinguish between - all small sets expand almost completely - exists small set with almost no expansion Small-Set Expansion Conjecture: 8²>0, 9 ± >0: NP-hard to distinguish YES: expansion(S) < ² for some S of volume ¼ ± NO: expansion(S) > 1-² for all S of volume ¼ ± S Unique 2-Prover Games Unique game ¡ : label set A L(A) Referee sample (A,B,¼) from D universe of size n ¼ B A label set L(B) distribution D over triples (A,B,¼) - A and B are from U - ¼ is bijection from L(A) to L(B) value( ¡ ): maximum success probability over all strategies of the players B Player 1 Player 2 pick a in L(A) pick b in L(B) a b Referee players win if ¼(a) = b no communication between players Approximating Unique Games How well can we approximate the value of a unique game for large label sets? BASIC SDP cannot distinguish between games with value ¼ 1 and ¼ 0 for large label sets Unique Games Conjecture: [Khot ‘02] 8²>0, 9 q >0: NP-hard to distinguish for ¡ with label set size q YES: value( ¡ ) > 1-² NO: value( ¡ ) < ² Approximating Unique Games Unique Games Conjecture: 8²>0, 9 q >0: NP-hard to distinguish for ¡ with label set size q YES: value( ¡ ) > 1-² NO: value( ¡ ) < ² Our main theorem: Small-Set Expansion Conjecture ) Unique Games Conjecture Reduction: Small-Set Expansion Unique Games Task: find non-expanding set of volume ¼ ± graph G B A Referee sample R = 1/± random edges M A = one half of each edge B = other half of each edge A B Player 1 Player 2 pick a 2 A pick b 2 B a b Referee players win if (a,b) 2 M Completeness Small Non-Expanding Set (Partial) Strategy graph G S referee allows players to refuse for few queries B A Suppose expansion(S) < ² With constant probability, |A Å S |= {a} Conditioned on this event: P( other half of a’s edge outside of S )<² partial game value > 1 - ² Strategy for Player 1: pick a 2 A if a is unique intersection with S otherwise, refuse to answer Soundness Strategy Small Non-Expanding Set standard trick: can assume both players have same strategy Idea: strategy distribution over sets - sample R-1 vertices U - output S = { x | Player 1 picks x if A=U+x } Easy to show: E volume(S) = 1/R = ± E # edges leaving S Suppose: players win with prob > 1-² E d |S| <² Are we done? No! Problem: volume(S) might not be concentrated around ± Soundness Strategy Small Non-Expanding Set ² R random vertices + ²-noise + ²-noise Idea: Referee adds ²-noise to A and B New distribution over sets: - sample R-1 vertices U - S = { x | players pick x if A = U+x+noise with probability > ½ } Can show: Suppose: players win with prob > 1-² 1 8 U. volume(S) < # noise vertices =±/² Intuition: players cannot distinguish x and noise Summary BASIC SDP optimal for SMALL-SET EXPANSION UGC BASIC SDP is optimal for lots of optimization problems, e.g.: MAX CUT and VERTEX COVER Open Questions (+ Subsequent Work) more reverse reductions? noise here vs. noise in other hardness reductions? Hardness results based on Small-Set Expansion Conjecture? Reductions between Expansion Problems [Raghavendra S Tulsiani’10] Better algorithms for UNIQUE GAMES via SMALL-SET EXPANSION? poly(²) 2n algorithm for (1-², ½)-UG via SSE [Arora Barak S’10] Thanks! Questions? APPENDIX Rounding a UG strategy to a small non-expanding set Unique Games Unique Games instance ¡ : random edge (u,v) decorated bipartite graph u u v label set L(u) Referee label set L(v) bijection ¼uv: L(u) L(v) value( ¡ ): maximum success probability over all strategies of the players v Player 1 Player 2 pick i in L(u) pick j in L(v) i j Referee players win if ¼uv(i) = j each player knows only half of referee’s edge Approximation MAX CUT Trivial approximation Random assignment, cut ½ of the edges First non-trivial approximation approx-ratio 0.878 Goemans–Williamson algorithm (‘95) based on a semidefinite relaxation (BASIC SDP) General constraint satisfaction problems (CSPs): simple generic approximation algorithm [RaghavendraS’09] for every CSP: approximation matches integrality gap of certain SDP near-linear running time [S’10] Can we beat the approximation guarantee of BASIC SDP (for Max Cut)? Combinatorial Optimization Example: MAX CUT: partition vertices of a graph into two sets so as to maximize number of cut edges A concrete practical application: ¼ MAX CUT CAIDA at UCSD analyzes business relationships among Autonomous Systems in the internet using MAX 2SAT