The Role of Communication
Complexity in Distributed Computing
Rotem Oshman
Background: Distributed Computing
Distributed Computing
• Typical model:
– Local computation for free
– Charge for “communication”
Distributed Lower Bounds
• “All or nothing”:
– If 𝑝, 𝑞 have not communicated, they know nothing
about each other
– If 𝑝, 𝑞 communicated, they know everything
about each other
• Recently: interest in quantifying the amount
of communication
• Natural to use communication complexity
Shared Memory
• Processes communicate by accessing objects
in shared memory
– Read/write registers
– Read-modify-write: CAS, T&S, …
• Typically asynchronous:
– Schedule = sequence of process IDs
– Adversarially chosen
– Sometimes processes may crash
Shared Memory Lower Bounds
• “Covering arguments”
• Examples:
– Deterministic consensus impossible if one process
may fail [FLP’79]
– Mutual exclusion requires 𝑛 shared registers
[BL’90]
• Can we introduce a cell-probe-like model for
shared memory lower bounds?
Message Passing
• Processes communicate by sending messages
– Over some network graph, often complete
• Fully synchronous, fully asynchronous, or
anywhere in between
• Processes can crash, recover, cheat, lie,…
• Many successful applications of CC
Some differences…
Distributed Computing
Complexity #rounds (limited bandwidth)
measure
Comm. Complexity
total #bits
Problems
Search
Decision
Input
Number-In-Hand
Number-onForehead (usually)
Message-Passing Models
#rounds
total CC
LOCAL
SHARED BLACKBOARD
CONGEST
MESSAGE-PASSING
Talk Overview
I. Lower bound techniques
a. CONGEST (#rounds): reductions from 2-party
communication complexity
b. Total CC with private channels
II. Shared blackboard
a. Number-in-hand
b. “Not-quite-number-in-hand”
The CONGEST Model
• 𝑛 nodes communicate over a
network graph (small diameter)
• Computation proceeds in
synchronous rounds
• Each message = 𝐵 bits
• Several variants:
– Communication by unicast vs. local broadcast
– Arbitrary graph vs. complete graph
The CONGEST Model
• Arbitrary graph topology: strong lower bounds
– [Distributed Verification and Hardness of Distributed
Approximation, by Das Sarma, Holzer, Kor, Korman,
Nanongkai, Pandurangan, Peleg, Wattenhofer]
• Nearly-tight lower bound on MST approximation
• Non-tight lower bounds on all-pairs shortest path,
minimum cut, shortest s-t path, …
• Almost all lower bounds ≤ Ω
𝑛/𝐵
– Very few Ω 𝑛/𝐵 bounds known, no super-linear
bounds
The CONGEST Model
• Complete graph topology, unicast:
– Known to be extremely powerful, e.g.,
• Sort 𝑛2 values in 𝑂 log log 𝑛 rounds
• MST construction in 𝑂 log ∗ 𝑛 rounds
– No explicit super-constant lower bound known
– [Drucker, Kuhn, O. ‘14]: even slightly superconstant lower bound ⇒ new ACC lower bound
• Complete graph topology, broadcast = shared
blackboard
CONGEST Lower Bounds for
Arbitrary Graphs
… by reduction from 2-party
disjointness
2-Party Reductions
• Textbook reduction [Kushilevitz Nisan]:
Given algorithm 𝐴 for solving task 𝑇…
𝑋
𝑌
Simulate 𝐴
Based
on 𝑋
𝑛 bits
Based
on 𝑌
Solution for 𝑇 ⇨ answer for, e.g., Disjointness
2-Party Reductions
• More generally:
Given algorithm 𝐴 for solving task 𝑇…
𝑌
𝑋
Based
on 𝑋
Based
on 𝑌
Solution for 𝑇 ⇨ answer for, e.g., Disjointness
Multi-Player NIH Communication
with Private Channels
The Message-Passing Model
•
•
•
•
𝑘 players
Private channels
Private 𝑛-bit inputs 𝑋1 , … , 𝑋𝑘
Private randomness
• Goal: compute 𝑓 𝑋1 , … , 𝑋𝑘
• Cost: total communication
The Coordinator Model
• 𝑘 players, one coordinator
• The coordinator has no input
Message-Passing vs. Coordinator
≈
Secure multi-party
computation!
Message-Passing Lower Bounds
For 𝑘 players with 𝑛-bit inputs…
• Woodruff, Zhang ‘12: 𝐹𝑝 estimation
• Phillips, Verbin, Zhang ’12:
– Ω 𝑘𝑛 for bitwise problems (AND/OR, MAJ, …)
• Woodruff, Zhang ‘13:
– Ω 𝑘𝑛 for threshold and graph problems
• Braverman, Ellen, O., Pitassi, Vaikuntanathan
‘13: Ω 𝑘𝑛 for disjointness
Symmetrization [Phillips, Verbin, Zhang ’12]
Lower bound for 𝑘-player problem 𝑃𝑘 :
• Choose hard 2-player problem 𝑃2
• Fix hard distribution 𝜇2 for 𝑃2 , let 𝑋, 𝑌 ∼ 𝜇2
Bob: 𝑌
“Smear” 𝑌 across 𝑘 − 1 players Alice: 𝑋
Symmetrization [Phillips, Verbin, Zhang ’12]
Lower bound for 𝑘-player problem 𝑃𝑘 :
Bob: 𝑌
“Smear” 𝑌 across 𝑘 − 1 players
Alice: 𝑋
– 𝑘-player distribution must be symmetric
– Answer to 𝑃𝑘 ⇒ answer to 𝑃2
2
𝑘
𝔼[ cost for 𝑃2 ] = ⋅ cost for 𝑃𝑘
Symmetrization Example: Bitwise-XOR
• 2-party problem:
– Input: uniform independent 𝑋, 𝑌 ∈ 0,1
– Goal: Alice outputs Bob’s input, 𝑌
𝑛
• Reduction:
– Sample uniform 𝑖 ∈ 𝑘 , Alice sets 𝑋𝑖 = 𝑋
– Bob chooses 𝑋−𝑖 uniformly s.t. ⊕𝑗≠𝑖 𝑋𝑗 = 𝑌
– From ⊕𝑘ℓ=1 𝑋ℓ , Alice can reconstruct 𝑌
Bob: 𝑌
Alice: 𝑋
Set Disjointness
𝑋2
𝑋1
𝑋3
?
𝑋5
𝑋4
𝑛
𝑘
𝑗
𝑋𝑖
DIS J 𝑛,𝑘 =
𝑖=1 𝑗=1
Symmetrization vs. Disjointness
• Consider any symmetric distribution…
Bob: 𝑌
Alice: 𝑋
• How many 0s in coord. 𝑗, given
𝑗
𝑘
𝑖=1 𝑋𝑖
= 0?
– More than one ⇒ Bob probably sees a zero
• Ignore this coordinate
– Only one ⇒ Pr[ Alice got it ] ≈
– 2-party CC ≤ 𝑂
𝑛 log 𝑛
𝑘
1
𝑘
[BEOPV’13] Lower Bound Outline
𝑛
𝑘
𝑗
𝑋𝑖
DIS J 𝑛,𝑘 =
𝑖=1 𝑗=1
1. Direct sum: DIS J 𝑛,𝑘 ≥ 𝑛 ⋅ AND𝑘
2. One-bit lower bound: AND𝑘 ≥ Ω 𝑘
Reduction from DISJ to
graph connectivity [Based on WZ’13]
(Players)
𝑝1
𝑋𝑖
1
(Elements)
2
𝑝2
3
4
𝑝𝑘
input graph
connected
⇔
⋃𝑋𝑖 = 𝑛
⇔
⋂𝑋𝑖 = ∅
5
6
𝑛 ∖ ⋃𝑋𝑖
Number-In-Hand Shared
Blackboard
Why Should We Care?
• Some fundamental question still open
• Natural model for distributed computing
– Single-hop wireless network
– More generally, abstracts away network topology
– Related to MapReduce, etc. [Hegeman and
Pemmaraju’14]
Example: NIH Multi-Party Disjointness
• Trivial upper bound: 𝑂 𝑛𝑘
– Also easy to get 𝑂 𝑛 log 𝑛 + 𝑘
• Simultaneous CC: Ω 𝑛𝑘
[Braverman,Ellen,O.,Pitassi,Vaikuntanathan’13]
– In 𝑟 rounds?
– Looks like for 𝑘 = 𝑛, Ω
𝑟
1+1/2
𝑛
[current work with Ran Raz]
• Unbounded rounds: Θ 𝑛 log 𝑘
Braverman]
[with Mark
“Not-Quite Number in Hand”
• In undirected networks, each edge is known to
both endpoints
• Distributed graph property testing:
– Players 1, … , 𝑛, input graph 𝐺 = 𝑛 , 𝐸
– Input to player 𝑖: its neighbors in 𝐺
– Goal: test if 𝐺 satisfies property 𝑃
• Example: subgraph detection [Drucker, Kuhn, O. ‘14]
Example: Lower Bound for 𝐶5
• Claim: 𝑅-round algorithm for 𝐶5 detection ⇒
solve DISJ𝑛2 in 𝑂 𝑅 ⋅ 𝑛 ⋅ 𝐵 bits
• Reduction outline:
– Alice and Bob get inputs 𝑋, 𝑌 ⊆ 𝑛 × 𝑛
– Construct input graph 𝐺 on 5𝑛 nodes, such that
𝐺 contains 𝐶5 ⇔ 𝑋 ∩ 𝑌 ≠ ∅
– Simulate the run of 𝐶5 -detection algorithm on 𝐺
Construction of 𝐺 from 𝑋, 𝑌
1
1
2
2
3
3
4
4
1
1
2
2
3
3
4
4
Construction of 𝐺 from 𝑋, 𝑌
1
1
2
2
3
3
4
4
Top-bottom:
𝒊, 𝒋 ∈ 𝑿
Bottom-top:
𝒋, 𝒊 ∈ 𝒀
1
1
2
2
3
3
4
4
Construction of 𝐺 from 𝑋, 𝑌
1
1
2
2
3
3
4
4
𝟐, 𝟒 ∈ 𝑿 ∩ 𝒀
1
1
2
2
3
3
4
4
Simulating the Algorithm
• Alice simulates 3𝑛 nodes, Bob simulates 2𝑛
• To simulate one round, each player:
– Locally computes message broadcast by each
node it simulates
– Sends all messages to the other player
– Cost: 𝑂 𝑛 ⋅ 𝐵 per round
• Total cost: 𝑂 𝑅𝑛𝐵
⇒𝑅=Ω
𝑛
𝐵
What About 𝐶4 ?
• More complicated….
Can contain 𝐶4 regardless of 𝑋
Solution: use
extremal 𝐶4 -free
graph 𝐹
Elements of DISJ
= edges of 𝐹
Upper Bound on Subgraph Detection
• Turán number: 𝑒𝑥 𝐻, 𝑛 = max # edges in 𝐻free graph on 𝑛 vertices
• Upper bound: solve 𝐻-subgraph detection in
𝑂
𝑒𝑥 𝐻,𝑛
𝑛𝐵
rounds
– Example: 𝑒𝑥 𝐶5 , 𝑛 = 𝑛2 /4, nearly tight
– Open problem: is this tight for all subgraphs?
Detecting Triangles
• Trivial upper bound: 𝑂
𝑛
𝐵
rounds
• Lower bound?
– 2-party (black box) reduction cannot prove it
– For each triangle, one player knows all 3 edges
Triangles to 3-Party NOF Disjointness
• [Ruzsa and Szemerédi ’76]: there is a tripartite
graph 𝐺 = (𝐴 ∪ 𝐵 ∪ 𝐶, 𝐸) where
– 𝐴 = 𝐵 = 𝑛, 𝐶 = 𝑛/2
– 𝐺 contains T = 𝑛2 /𝑒 𝑂 log 𝑛 triangles
– Each edge in 𝐸 belongs to exactly one triangle
• Reduce from 3-party NOF Disjointness on 𝑇
elements, each representing one triangle in 𝐺
Triangles to 3-Party NOF Disjointness
• Input: sets of triangles 𝑋, 𝑌, 𝑍 ⊆ 𝑇
𝑍
• Let 𝑡(𝑒) be the unique triangle
edge 𝑒 belongs to
• Construct 𝐺 ′ ⊆ 𝐺, including:
𝐶
– 𝑒 ∈ 𝐴 × 𝐵 iff 𝑡 𝑒 ∈ 𝑍
– 𝑒 ∈ 𝐵 × 𝐶 iff 𝑡 𝑒 ∈ X
– 𝑒 ∈ 𝐴 × 𝐶 iff 𝑡 𝑒 ∈ 𝑌
• Note: endpoints of each
edge agree on its inclusion!
𝑋
𝑌
𝐴
𝐵
Triangles to 3-Party NOF Disjointness
• Input: sets of triangles 𝑋, 𝑌, 𝑍 ⊆ 𝑇
• Triangle appears in 𝐺 ′ ⇔ 𝑋 ∩ 𝑌 ∩ 𝑍 ≠ ∅
• Cost of simulation:
𝑍
𝐶
𝑋
– 𝑂(𝑛𝐵) bits per round
⇒ Round complexity of triangle detection
≥ 𝐶𝐶(DIS J 𝑇 )/(𝑛𝐵) for 3-party NOF
𝑌
𝐴
𝐵
3-Party NOF Disjointness
• Randomized CC:
– Sherstov’13: Ω
– We get nothing:
#elements
𝑇
𝑛𝑏
<1
• Deterministic CC:
– Rao and Yehudayoff’14: Ω #elements
⇒Ω
𝑛
𝐵⋅𝑒 𝑂
detection
log 𝑛
for deterministic triangle
Conclusion
#rounds
total CC
LOCAL
SHARED BLACKBOARD
CONGEST
MESSAGE-PASSING
Directions for Future Research
• Exploiting asynchrony and faults to get
stronger communication lower bounds
Example 1: Dynamic Networks
• Abstract model for dynamic networks:
– In each round 𝑟 we get a different graph 𝐺 𝑟
• [Kuhn, Lynch, O. ‘10]:
– Assume each 𝐺 𝑟 is connected
– Any function can be deterministically computed in
𝑂 𝑛2 rounds using 𝑂 log 𝑛 -bit messages
– Lower bounds?
Example 1: Dynamic Networks
• For exchanging all inputs:
– Determistic, “routing-based” algorithms: Ω
[Haeupler, Kuhn’12]
– Randomized??
– Non-routing based??
𝑛2
log 𝑛
Example 2: Byzantine Consensus
• 𝑛 processes, synchronous message-passing
• Each process receives a bit
• Goal:
– Everyone outputs the same bit
– If everyone received 𝑏, the output is 𝑏
𝑛
3
• Byzantine faults: up to − 𝜖 processes may
behave arbitrarily
Example 2: Byzantine Consensus
• [King, Saia ‘10]: can solve with 𝑂(𝑛3/2 ) total
bits
• No general lower bound better than Ω 𝑛