Algorithm Design Using
Spectral Graph Theory
Richard Peng
M.I.T.
OUTLINE
Laplacian paradigm
Division using multiplication
How are graphs similar?
Operator view of algorithms
LARGE GRAPHS
Images, road networks,
meshes, web-graphs…
Algorithmic challenges:
store, process, optimize…
LAPLACIAN PARADIGM ([TENG `10])
[DS`84]:
• random walks
• electrical flows
Linear system in the graph Laplacian
GRAPH LAPLACIANS
• Symmetric
• Row/column sum to 0
• Non-positive off-diagonals
KEY SUBROUTINE
Linear System Solve
• Given LG, vector b
• Find vector x such that LGx=b
Size of problem:
• n-by-n matrix  n vertices
• m non-zeros  m edges
GENERAL LINEAR SYSTEM SOLVES
Oldest studied algorithmic problem
• [Liu 179]…[Newton `1710]
[Gauss 1810]: O(n3)
• [HS `52]: conjugate gradient, O(nm)*
• [Strassen `69] O(n2.8)
• [CW `90] O(n2.3755)
• [Stothers `10] O(n2.3737)
• [Vassilevska Williams`11] O(n2.3727)
FAST SOLVERS FOR LAPLACIANS
Input: graph Laplacian L, vector b
Output: x ≈ε L+b
Note:
• Approximate solution
• L+: pseudo-inverse
• Omitting log(1/ε)
Cost:
[Vaidya`91]: O(m7/4)
[BH`01] O(mn)
[ST`03, `04] O(mlogcn)
[KMP `10, `11]: O(mlogn) work, O(m1/3) depth
[KOSZ `12, LS `13]: O(mlog1/2n)
[PS `14]: O(mlogcn) work, O(mlogcn) depth
CLOSELY RELATED SYSTEMS
• [GM`96] SDD matrices
• [DS`08] M-matrices, factor-width 2
• [BHV`04] elliptic finite element PDEs
• [CFMNPW`14]: Helmholtz on 3D mesh
DATA ANALYSIS
• [ZGL `03], [ZHS `05]: inference
• [KMST`09], [CMMP `13]: image
segmentation / denoising
GRAPHS
• [Tutte `62] Planar embeddings.
• [KM`09] random spanning tree
• [DS`08][LS`13] max/mincost flow
WHY ANY GRAPH PROBLEM
• [Karmarkar, Ye, Renegar, Nesterov,
Nemirovski …]: convex optimization
via. solving O(m1/2) linear systems
• [DS `08]: efficient solvers on graphs
GRAPHS, FASTER
• [CKMST `11][LRS `12] approximate
undirected maxflow
• [KMP `12]: multi-commodity flow
• [OSV `12]: balanced cuts, heat kernel
• [Madry `13] bipartite matching
FOCUS OF THIS TALK
• Given LG, vector b
• Find vector x such that LGx=b
Size of problem:
• n-by-n matrix  n vertices
• m non-zeros  m edges
OUTLINE
Laplacian paradigm
Division using multiplication
How are graphs similar?
Operator view of algorithms
N = 1 CASE
Given: scalar L, b = 1
Goal: compute 1 / L
Allowed operation: multiply
Why? Underlying routine in most
numerical algoirthms: b = Lx
DIVISION WITH MULTIPLICATION:
L-1 = 1 + (1 – L) + (1 - L)2 + (1 - L)3 …
Error with first N terms:
(1 – L) N + (1 – L) N+1 + (1 – L) N+2…
= (1 – L) NL-1
CONVERGENCE
Error with N terms:
(1 – L) N + (1 – L) N+1 + (1 – L) N+2…
= (1 – L) NL-1
Convergence if |1 - L| < 1:
1/κ ≤ L ≤ 1  κ terms reduce
error by constant factor
WHAT IF L IS FAR FROM 1?
Solve LGx = b
x(t+1) = b + (1 – LG)x(t)
LG-1 = 1 + (1 – LG) + (1 – LG)2…
Preconditioned solve:
Solve LH-1LGx = LH-1b
x(t+1) = b + (1 – LH-1LG)x(t)
PRECONDITIONER
(LH-1LG)-1 = 1 + (1 - LH-1LG)+ (1 - LH-1LG)2 …
• x(t+1) = LH-1b + (1 – LH-1LG)x(t)
• Need: LH-1LG close to 1
HOW MANY TERMS?
(LH-1LG)-1 = 1 + (1 - LH-1LG)+ (1 - LH-1LG)2 …
Relative error with N terms:
(1 – LH-1LG)N
• 1/κ ≤ LH-1LG ≤ 1  κ terms
• Equivalent to: LG ≤ LH ≤ κLG
[TURING `48]: CONDITION NUMBER
Generalize LG ≤ LH ≤ κLG
to n-by-n matrices
xTLGx ≤ xTLHx ≤ κxTLGx
for all vector x
 LG ≼ LH ≼ κLG x,
[DS `84] G and H are electrical
networks with similar energy output
COMPUTATION AS ERROR REDUCTION
Solve problem in G using
(many) problems in H
LG ≼ LH ≼ κLG
• Richarson: O(κ) iterations
• Chebyshev: O(κ1/2) iterations
OUTLINE
Laplacian paradigm
Division using multiplication
How are graphs similar?
Operator view of algorithms
GOAL
A
B
Given G, find H such that:
• LG ≼ LH ≼ κLG
• H ‘easier’ than G to solve
‘easier’  smaller  fewer edges
SIMILAR, SMALLER GRAPHS
A
B
• [Peleg `89]: spanners
• [BK `96]: cut sparsifiers
• [ST `04]: spectral sparsifiers
SPECTRAL SPARSIFICATION
G complete graph:
• H expander, e.g. O(nlogn) random edges
• [LPS `88][Margulis `88][Morgenstern `94]:
nearly-optimal constructions
SPECTRAL SPARSIFICATION
[Batson-Spielman-Srivastava `09]
For any LG, there is sparse LH with O(nε2) entries s.t. L ≼ L ≼ 2L
G
H
G
EVEN EASIER H
`
`
• [Vaidya `91]: tree plus edges
• [ST`04]: for any G, exists H with
O(n – 1 + mlogcn/κ) edges
such that LG ≼ LH ≼ κLG
ULTRA-SPARSIFIERS
Recall: Chebyshev: O(κ1/2) iterations
T(m) = κ1/2 (m + T(mlogcn /κ))
•
•
•
•
[KMST `10]: c ≤ 1 in poly-time
[KMP `10]: c ≤ 2, efficient
[KMP `11]: ‘spine heavy graphs’
[CKPPR `14]: analyze randomness
with iterative method, akin to c ≤ 1
Combine these: T(m) ≈ O(mlog1/2n)
KEY SUBROUTINE
Approximating graphs by trees
OUTLINE
Laplacian paradigm
Division using multiplication
How are graphs similar?
Operator view of algorithms
ALGORITHMIC PICTURE
T(m) = κ1/2 (m + T(mlogcn /κ))
W-cycle, recursive, algorithm
RELATED TO
`
`
Multigrid / Multilevel methods
(e.g. [Fedorenko `61, `64]
[Brandt `73, `77] [Hackbush `77] )
Difference: lack of fractal structure,
hard analyze using Fourier moments
ISSUE WITH ERROR ANALYSIS
Solve in LH recursively
• SolveH ≠ LH+
• Approximating LH+ with code
ARITHMETIC CIRCUITS:
Input: b
output: x = ZHb
Z: linear operator, symmetric,
positive semidefinite matrix
Can write ZH ≈ LH+ as LH ≼ ZH+ ≼
2LH
Compose with LG ≼ LH ≼ κLG:
LG ≼ ZH+ ≼ 2κLG
HOW DOES THIS HELP?
LG
LH
• Running code =
compose linear operators
• View the operator, ZH, as a
preconditioner for LG
ALGORITHM DESIGN
 OPERATOR DESIGN
LG
LH
• [ST `04]: recursive preconditioning
• [Sherman`13, KLOS `13]: oblivious
routing scheme, undirected maxflow
• [PS `14]: first parallel Laplacian solver
OPERATOR DESIGN
Factorize power method
1 + a + a2 + a3 + a4 + a5 + a6 + a7 + a8…
≈ (1 + a) (1 + a2) (1 + a4)…
[PS `14]: symmetrized version:
zi  ½ [1+(1 + ai)zi+1(1 + ai)]
Allows (repeated) sparsification
APPROXIMATE INVERSE CHAIN
1 - a1 ≈ε 1 – a2
1 – a2 ≈ε 1 – a12
…
1 – ai ≈ε 1 – ai-12
1 - aO(logκ) ≈ 1
• Matrix setting: 1  D, a  D-1A
FUTURE WORK
• Improve these algorithms
• Extensions to non-linear?
• Other notions of similarity?
THANK YOU!
Questions?