Automating Distributed
Partial Aggregation
Chang Liu1, Jiaxing Zhang2, Hucheng Zhou2, Sean McDirmid2,
Zhenyu Guo2, Thomas Moscibroda2
1University of Maryland, College Park
2Microsoft Research
MapReduce Framework
Mapper
Mapper
Mapper
Mapper emits key-value pairs
Key-value pairs will be
shuffled over the network,
and the pairs with the same
key are transmitted to the
same reducer
Reducer
Reducer
Reducer
Reducer takes all pairs with the
same key as input, and compute
the output
Network IO is one of the
bottlenecks dominating the
overall latency
MapReduce – Partial Aggregation
Some traditional reducer to
apply partial aggregation
Mapper
By aggregating
partial results
produced by an
initial reducer,
with a combiner,
network IO can
be reduced
significantly
Initial
Reducer
Mapper
Initial
Reducer
Mapper
Initial
Reducer
SUM
MAX
COUNT
AVERAGE
Combiner
Final
Reducer
In distributed computation
scenario, most reducers are
provided as user defined
functions
Partial Aggregation – state-of-the-art
• Most MapReduce systems support Partial Aggregation
• Hadoop – Combiner
• SCOPE – Recursive reducer
• Etc.
• Partial Aggregation is effective
• Our study shows a up to 99.98% shuffling IO reduction, and up to 62% overall latency
reduction
• But most require programmers to manually rewrite the reducer
• Programmers may write bugs when programs become sophisticated
• We study 183 real SCOPE programs, and found 28 of them (15.3%) are problematic
An example of User Defined Reducer
Row reduce(RowSet inputs) {
int sum = 0, count = 0;
bool isFirst = true;
Row output;
for(row in inputs.Rows) {
if(isFirst) {
output[0] = row[0];
sum = 10 + row[1];
count = 2;
isFirst = false;
} else
sum+=row[1], count++;
}
output[1] = sum/count;
return output;
}
Inputs
Reducer
Key
Value
1
2
1
5
1
3
…
…
output[0]=Key
output[1]=
10+Σ𝑖≥1 Inputs[i].Value
1+ Inputs
How to apply
partial aggregation
to such a reducer?
Automating Partial Aggregation
• Reduce the partial aggregation application problem to the combiner
synthesis problem
Automating Partial Aggregation
• Reduce the partial aggregation application problem to the combiner
synthesis problem
• A necessary and sufficient condition as the mathematical foundation
Automating Partial Aggregation
• Reduce the partial aggregation application problem to the combiner
synthesis problem
• A necessary and sufficient condition as the mathematical foundation
• Programming language techniques to solve the following problems:
• Decomposable verification
• Combiner synthesis
How to apply partial aggregation?
Row reduce(RowSet inputs) {
int sum = 0, count = 0;
bool isFirst = true;
Mapper
Mapper
Initial
Reducer
Initial
Reducer
Row output;
for(row in inputs.Rows) {
if(isFirst) {
output[0] = row[0];
sum = 10 + row[1];
count = 2;
Combiner
isFirst = false;
} else sum+=row[1], count++;
}
output[1] = sum/count;
return output;
}
Final
Reducer
How to apply partial aggregation?
It depends on the
count =aggregator
0;
in the main InitialReduce(inputs)
loop
= true;
int sum = 0, count
body!
Row reduce(RowSet inputs) {
int sum = 0,
bool isFirst
Row output;
for(row in inputs.Rows) {
if(isFirst) {
= 0;
… // the same as the original reducer
foreach (row in inputs.Rows) {
…
}
emit(output[0], sum, count, isFirst);
output[0] = row[0];
sum = 10 + row[1];
count = 2;
isFirst = false;
What is the Combiner?
} else sum+=row[1], count++;
}
output[1] = sum/count;
return output;
}
FinalReduce(partialResults)
emit(partialResults[0], sum/count)
What reducers have a combiner?
A mathematical foundation
• 𝐹 is decomposable if and only if there exists a combiner 𝐶 such that:
•
•
•
•
(commutativity of 𝑭) 𝐹 𝑠0 , 𝑋 ⊕ 𝑌 = 𝐹 𝑠0 , 𝑌 ⊕ 𝑋 1
(correctness of 𝑪) 𝐹 𝑠0 , 𝑋 ⊕ 𝑌 = 𝐶 𝐹 𝑠0 , 𝑋 , 𝐹 𝑠0 , 𝑌
(commutativity of 𝑪) 𝐶 𝑠1 , 𝑠2 = 𝐶 𝑠2 , 𝑠1
(associativity of 𝑪) 𝐶(𝐶(𝑠1 , 𝑠2 ), 𝑠3 ) = 𝐶(𝑠1 , 𝐶(𝑠2 , 𝑠3 ))
• Yu et al. Distributed aggregation for data-parallel computing:
interfaces and implementations. In SOSP 2009.
Notation:
𝑠0 is the initial results
𝐹(𝑠, 𝑋) is the aggregator
𝐶(𝑠, 𝑠) is the combiner
1⊕
is the operation concatenating two sequenses
Main theorem
What is this?
• Theorem: 𝐹 is decomposable if and only if 𝐹 𝑠, 𝑥𝑦 = 𝐹(𝑠, 𝑦𝑥) holds
for all 𝑠, 𝑥, 𝑦. The combiner 𝐶 is uniquely determined by 𝐹, and takes
the form 𝐶 𝑠1 , 𝑠2 = 𝐹(𝑠1 , 𝐻 𝑠2 ) where 𝐻 is any general inverse
function of 𝐹
Commutativity of 𝐹
Existence of 𝐶
Correctness of 𝐶
Commutativity of 𝐶
Associativity of 𝐶
General Inverse Function
Initial
Reducer
𝑋
• 𝐻(𝑠) is a general inverse function of 𝐹 if
and only if 𝐹 𝑠0 , 𝐻 𝑠 = 𝑠.
𝑠
H(s)
𝑋
• 𝐹 may have many general inverse
functions
𝑌
𝑌 = 𝐻(𝑠2 )
𝐻(𝑠1 )
• Question: will different general inverse
functions result in different combiners?
• We prove that for any two different
general inverse functions 𝐻, 𝐻′, 𝐶𝐻 ≡ 𝐶𝐻 ′
Initial
Reducer
Initial
Reducer
𝑠1
𝑠2
𝐻(𝑠2 )
𝑋 = 𝐻 𝑠1
Initial
Reducer
Combiner
𝐶𝐻
𝑠
𝑠
Decomposability Verification and Combiner Synthesis
Programming Language Techniques
• Theorem: 𝐹 is decomposable if and only if 𝐹 𝑠, 𝑥𝑦 = 𝐹(𝑠, 𝑦𝑥) holds
for all 𝑠, 𝑥, 𝑦. The combiner 𝐶 is uniquely determined by 𝐹, and takes
the form 𝐶 𝑠1 , 𝑠2 = 𝐹(𝑠1 , 𝐻 𝑠2 ) where 𝐻 is any general inverse
function of 𝐹
Decomposability Verification and Combiner Synthesis
Programming Language Techniques
• Theorem: 𝐹 is decomposable if and only if 𝑭 𝒔, 𝒙𝒚 = 𝑭(𝒔, 𝒚𝒙) holds
for all 𝒔, 𝒙, 𝒚. The combiner 𝐶 is uniquely determined by 𝐹, and takes
the form 𝐶 𝑠1 , 𝑠2 = 𝐹(𝑠1 , 𝐻 𝑠2 ) where 𝐻 is any general inverse
function of 𝐹
• Decomposability verification: checking 𝐹 𝑠, 𝑥𝑦 = 𝐹(𝑠, 𝑦𝑥) holds
true for all 𝑠, 𝑥, 𝑦
Decomposability Verification and Combiner Synthesis
Programming Language Techniques
• Theorem: 𝐹 is decomposable if and only if 𝐹 𝑠, 𝑥𝑦 = 𝐹(𝑠, 𝑦𝑥) holds
for all 𝑠, 𝑥, 𝑦. The combiner 𝑪 is uniquely determined by 𝐹, and takes
the form 𝑪 𝒔𝟏 , 𝒔𝟐 = 𝑭(𝒔𝟏 , 𝑯 𝒔𝟐 ) where 𝐻 is any general inverse
function of 𝐹
• Decomposability verification: checking 𝐹 𝑠, 𝑥𝑦 = 𝐹(𝑠, 𝑦𝑥) holds true
for all 𝑠, 𝑥, 𝑦
• Combiner Synthesis: given 𝐹, find an arbitrary general inverse
function.
Decomposability Verification
SMT solver-based method
Accumulator 𝐹
if(isFirst) {
key = x[0];
sum = 10 + x[1];
count = 2; isFirst =
false;
Symbolic Execution
Path formula for 𝐹
(isFirst i = 1 ∧ keyo = x 0 ∧ sumo = 10 + x 1
∧ count o = 2 ∧ isFirst o = 0)
∨ (isFirst i = 0 ∧ keyo = keyi ∧ sumo = sumi + x 1
∧ count o = count i + 1 ∧ isFirst o = isFirst i
} else sum+=x[1],
count++;
Formula for
𝐹 𝑠0 , 𝑥𝑦 ≠
𝐹(𝑠0 , 𝑦𝑥)
Combiner Synthesis
• Hard – Finding 𝐻 is a nondeterministic program inversion problem
• We find that many eligible reducers belong to the following
categories:
• Counting
• Single Input
• State Machine
Combiner Synthesis – COUNTING Example
Accumulator 𝐹
if(isFirst) {
Observation: the results only depends on the size of
the input sequence, instead of their values.
key = x[0];
sum = 10 + x[1];
count = 2;
isFirst = false;
} else {
sum+=x[1];
count++;
}
Strategy: Mimic the input size to get one partial result,
then aggregate these input over the other partial result.
Heuristics to accelerate: if the aggregator is a linear
function of COUNT, then compute the function.
Combiner Synthesis – Single Input Example
Accumulator 𝐹
if(isFirst) {
key = x[0];
sum = 10 + x[1];
count = 2;
isFirst = false;
} else {
sum+=x[1];
count++;
}
Observation: there exists a general inverse function
whose output is a sequence of length one.
Strategy: Given the partial results, find one input value
to compute this result, and apply that value over the
other partial result.
Illustrating Example: given sum, then
sum = 10 + x 1 ⇒ x 1 = sum − 10
So Combine sumx , sumy = sumx + (sumy − 10)
More details can be found in our paper.
Evaluation
• We study 4,429 SCOPE jobs
• 183 have already used partial aggregation
• 28 of them (15.3%) are buggy
• Xiao et al. Nondeterminism in MapReduce Considered Harmful? An Empirical Study on
Non-commutative Aggregators in MapReduce Programs, ICSE 2014
• We identify 261 more jobs that are eligible for partial aggregation
• 22 unique reducers (reducers are heavily reused in different jobs)
Evaluation – decomposability verification
• We verify 𝐹 𝑠, 𝑦𝑥 = 𝐹(𝑠, 𝑥𝑦) over the 22 identified reducers with
two quantifier: (1) ∀𝑠, 𝑥, 𝑦 and (2) ∀𝑥, 𝑦. 𝑠 = 𝑠0 1.
1The
reasons why to use two quantifiers can be found in Section 6.1
Evaluation – combiner synthesis
• We apply our techniques to synthesize the combiner for these 22
identified reducers.
• C for COUNTING, SM for State Machine, SI for Single Input
• We failed on synthesizing the combiner for reducer 21 and 22.
Evaluation – Performance Gain
• Real job with tens of TBs input data
• End-to-end latency reduction from 165 seconds to 64 seconds (61.6%)
• Data volume over network decreases from 7.99 GB to 1.22 MB (99.98%)
• Synthesized jobs (SUM, COUNT, and MAX)
• An average of 62.4% reduction in latency
• An average of 76% reduction in network I/O
Conclusion and Future Directions
• A necessary and sufficient condition for decomposability
• SMT solver-based methods for the decomposability verification problem
• Several techniques to solve the combiner synthesis problem
• Evaluation on real dataset shows our techniques are effective
• Future directions
• More techniques handling the combiner synthesis problem
Thank you!