Logic and Lattices for
Distributed Programming
Neil Conway, William R. Marczak,
Peter Alvaro, Joseph M. Hellerstein
UC Berkeley
David Maier
Portland State University
Distributed
Programming:
Key Challenges
Asynchrony
Partial
Failure
Output
f(x)
Input
x
Network
Behavior
Output
f(x)
Output
f(x)
Dealing with Disorder
Enforce global order
– Paxos, Two-Phase Commit, GCS, …
– “Strong Consistency”
Tolerate disorder
– Programmer must ensure correct behavior for
many possible network orders
– “Eventual Consistency”
• Typical goal: replicas converge to same final state
Dealing with Disorder
Enforce global order
– Paxos, Two-Phase Commit, GCS, …
– “Strong Consistency”
Tolerate disorder
– Programmer must ensure correct behavior for
many possible network orders
– “Eventual Consistency”
• Typical goal: replicas converge to same final state
Goal:
Make it easier to write
programs on top of
eventual consistency
This Talk
1. Prior Work
– Convergent Modules (CRDTs)
– Monotonic Logic (CALM)
2. BloomL
3. Case Study
Write:
Read:
{Alice,
{Alice,
Bob,Bob}
Carol}
Client0
Students
{Alice,
Bob,
Carol}
{Alice,
Bob}
How to resolve?
Write:
Read:
{Alice,
{Alice,
Bob,Bob}
Dave}
Client1
Students
{Alice,
{Alice,
Bob,
Bob}
Dave}
Proble
m
Replicas perceive different
event orders
Goal
Same final state at all replicas
Solutio Use commutative operations
n
(“merge functions”)
Client0
Students
{Alice, Bob, Carol, Dave}
Merge = Set Union
Client1
Students
{Alice, Bob, Carol, Dave}
Commutative Operations
• Common design pattern
• Formalized as CRDTs:
Convergent and Commutative
Replicated Data Types
– Shapiro et al., INRIA (20092012)
– Based on join semilattices
Lattices
hS,t,?i is a bounded join semilattice iff:
– S is a set
– t is a binary operator (“least upper bound”)
• Associative, commutative, and idempotent
• Induces a partial order on S: x ·S y if x t y = y
• Informally, “merge function” for elements of S
– ? is the “least” element in S
• 8x 2 S: ? t x = x
12
Time
Set
(LUB = Union)
Increasing
Integer
(LUB = Max)
Boolean
(LUB = Or)
Read: {Alice, Bob, Carol, Dave}
Students
{Alice,
{Alice,
Bob,
Bob,
Carol,
Carol}
Dave}
Client0
Read: {<Alice,Bob>}
Write: {<Alice,Bob>, <Carol,Dave>}
Teams
Teams
{<Alice, Bob>,
{<Alice, Bob>}
<Carol, Dave>}
Replica Synchronization
Remove: {Dave}
Client1
Students
{Alice,
Bob,
Carol}
{Alice,
Bob,
Carol,
Dave}
Teams
Teams
{<Alice, Bob>,
{<Alice, Bob>}
<Carol, Dave>}
Read: {Alice, Bob, Carol}
Students
{Alice,
Bob,
Carol,
Dave}
{Alice,
Bob,
Carol}
Client0
Read: {<Alice,Bob>}
Teams
{<Alice, Bob>}
{<Alice,
Bob>}
Replica
Nondeterministic
Synchronization
Outcome!
Remove: {Dave}
Students
{Alice,
Bob,
Carol}
{Alice,
Bob,
Carol,
Dave}
Client1
Teams
{<Alice, Bob>}
{<Alice,
Bob>}
Problem:
Composition of CRDTs can
result in non-determinism
Possible Solution:
Encapsulate all distributed
state in a single CRDT
Hard to design,
verify, and test
Doesn’t scale with
application size
Goal:
Design a language that allows
safe composition of CRDTs
Solution: … Datalog?
• Concurrent work: distributed
programming using Datalog
– P2 (2006-2010)
– Bloom (2010-2012)
• Monotonic logic: building
block for convergent
distributed programs
Monotonic Logic
• As input set grows,
output set does not
shrink
– “Retraction-free”
• Order independent
• e.g., map, filter, join,
union, intersection
Non-Monotonic Logic
• New inputs might
retract previous
outputs
• Order sensitive
• e.g., aggregation,
negation
Monotonicity and Determinism
Agents learn strictly more
knowledge over time
Different learning order,
same final outcome
Result:
Program is deterministic!
Consistency
As
Logical
Monotonicity
CALM Analysis
1. All monotone programs
are deterministic
2. Simple syntactic test for
monotonicity
Result: Whole-program
static analysis for
eventual consistency
Problem:
CALM only applies to
programs over growing sets
Version Numbers
Timestamps
Threshold Tests
Quorum Vote
• A coordinator accepts
votes from agents
• Count # of votes
– When Count(Votes) > k,
send “success” message
Quorum Vote
• A coordinator accepts
votes from agents
• Count # of votes
– When Count(Votes) > k,
send “success” message
Aggregation is
non-monotonic!
BloomL
Whole program
analysis
Flexible types
(any lattice)
CRDTs
Limited scope
(single object)
Flexible types
(any lattice)
CALM
Whole program
analysis
Limited types
(only sets)
BloomL Constructs
Organization
Communication
State
Computation
Collection of agents
Message passing
Lattices
Functions over lattices
Monotone Functions
f : ST is a monotone function iff
8a,b 2 S : a ·S b ) f(a) ·T f(b)
28
Time
Monotone function from
set  increase-int
size()
Set
(LUB = Union)
Monotone function from
increase-int  boolean
>= 5
Increasing
Integer
(LUB = Max)
Boolean
(LUB = Or)
Quorum Vote in BloomL
QUORUM_SIZE = 5
RESULT_ADDR = "example.org"
class QuorumVote
include Bud
Annotated Ruby class
Communication interfaces
state do
channel :vote_chn, [:@addr, :voter_id]
Program state
channel :result_chn, [:@addr]
Lattice state declarations
lset :votes
lmax :vote_cnt
lbool :got_quorum
Accumulate votes
end
Monotonic  CALM
into set
Monotone function: set ! max
Monotone function: max ! bool
bloom do
votes
<= vote_chn {|v| v.voter_id}
Program
vote_cnt <= votes.size
got_quorum <= vote_cnt.gt_eq(QUORUM_SIZE)
result_chn
<~ got_quorum.when_true
{ [RESULT_ADDR] }
Merge function
for set lattice
end
Threshold test on bool (monotone)
end
logic
30
BloomL Features
• Generalizes logic programming to lattices
– Integration of relational-style queries and
functions over lattices
– Efficient incremental evaluation scheme
• Library of built-in lattices
– Booleans, increasing/decreasing integers,
sets, multisets, maps, …
• API for defining custom lattices
Case Studies
Key-Value Store
– Object versioning via
vector clocks
– Quorum replication
Replicated Shopping Cart
– Using custom lattice types
to encode domain-specific
knowledge
Case Studies
Key-Value Store
– Object versioning via
vector clocks
– Quorum replication
Replicated Shopping Cart
– Using custom lattice types
to encode domain-specific
knowledge
Case Study: Shopping Carts
CartReplica
Item Add/ Remove
CartClient
Checkout
Request
Checkout Response:
Cart Summary
CartReplica
CartReplica
34
Case Study: Shopping Carts
CartReplica
ADD: {beer: 2}
ADD: {coffee: 1}
CartClient
{beer: 2}
{coffee: 1}
CartReplica
CartReplica
35
Case Study: Shopping Carts
CartReplica
CartClient
ADD: {tea: 3}
REMOVE: {coffee: 1}
{beer: 2}
{coffee: 1}
CartReplica
CartReplica
{tea: 3}
{coffee: -1}
36
Case Study: Shopping Carts
CartReplica
{beer: 2}
{coffee: 1}
???
CartClient
CartReplica
CHECKOUT
CartReplica
{tea: 3}
{coffee: -1}
37
Perspectives on Shopping
• CRDTs
– Individual server replicas converge
• Bloom
– Checkout is non-monotonic  requires
distributed coordination
• Built-in BloomL lattice types
– Checkout is not a monotone function of any
of the built-in lattices
Observation:
Once a checkout
occurs, no more
shopping actions
can be performed
Observation:
Each client knows
when a checkout can be
processed “safely”
Monotone Checkout
OPS =
[1,2,3]
Complete
OPS = [2,3]
Incomplete
OPS = [1,2]
Incomplete
OPS = [1]
Incomplete
OPS = [2]
Incomplete
OPS = [3]
Incomplete
41
Monotone Checkout
OP #1
ADD: {beer: 2}
ADD: {coffee: 1}
CartClient
CartReplica
OP #1:
{beer: 2}
{coffee: 1}
COMPLETE = false
CartReplica
CartReplica
42
Monotone Checkout
CartReplica
CartClient
OP #2
ADD: {tea: 3}
REMOVE: {coffee: 1}
{ops: [1]}
{beer: 2}
{coffee: 1}
COMPLETE = false
CartReplica
CartReplica
{ops: [2]}
{tea: 3}
{coffee: -1}
COMPLETE = false
43
Monotone Checkout
CartReplica
{ops: [1]}
{beer: 2}
{coffee: 1}
COMPLETE = false
COMPLETE = false
CartClient
CartReplica
{ops: [3]}
NEED OPS: {1,2,3}
COMPLETE = false
CartReplica
{ops: [2]}
{tea: 3}
{coffee: -1}
COMPLETE = false
OP #3
CHECKOUT: [1--3]
44
Monotone Checkout
CartReplica
CartClient
{beer: 2}
{tea: 3}
COMPLETE = true
{ops: [1]}
{beer: 2}
{coffee: 1}
COMPLETE = false
CartReplica
{ops: [3]}
NEED OPS: {1,2,3}
COMPLETE = false
CartReplica
{ops: [1,2,3]}
{beer: 2}
{tea: 3}
NEED OPS: {1,2,3}
COMPLETE = true
45
Shopping Takeaways
• Checkout summary is a monotone function
of client’s activities
• Custom lattice type captures applicationspecific notion of “forward progress”
– “Unsafe” state hidden behind ADT interface
Recap
1. How to build eventually consistent systems
– Write disorderly programs
2. Disorderly state
– Lattices
3. Disorderly computation
– Monotone functions over lattices
4. BloomL
– Type system for deterministic behavior
– Support for custom lattice types
Thank You!
http://www.bloom-lang.net
Backup Slides
Strong Consistency in Industry
“… there was a single overarching theme within
the keynote talks… strong synchronization of
the sort provided by a locking service must be
avoided like the plague… [the key] challenge is
to find ways of transforming services that might
seem to need locking into versions that … can
operate correctly without locking.”
-- Birman et al.,
“Toward a Cloud Computing Research Agenda”
(LADIS, 2009)
50
Bloom Operational Model
Fixpoint
State
Update
Local Updates
Bloom Rules
System Events
Inbound Network
atomic, local,
deterministic
Outbound
Network
51
Quorum Vote in Bloom
QUORUM_SIZE = 5
RESULT_ADDR = "example.org"
class QuorumVote
include Bud
Annotated Ruby class
Communication
state do
channel :vote_chn, [:@addr, :voter_id]
channel :result_chn, [:@addr]
table :votes, [:voter_id]
scratch :cnt, [] => [:cnt]
end
Program state
Persistent Storage
Transient Storage
Not
(set) monotonic!
Count
votes
Accumulate votes
bloom do
votes
<= vote_chn {|v| [v.voter_id]}
Program
cnt
<= votes.group(nil, count(:voter_id))
result_chn <~ cnt {|c| [RESULT_ADDR] if c >= QUORUM_SIZE}
end
end
Send message when quorum reached
logic
52
Built-in Lattices
Name
Description
?
atb
lbool
Threshold test
lmax
Sample Monotone Functions
false
a∨ b
Increasing
number
1
max(a,b
)
gt(n) ! lbool
+(n) ! lmax
-(n) ! lmax
lmin
Decreasing
number
−1
min(a,b)
lt(n) ! lbool
lset
Set of values
;
a[b
intersect(lset) ! lset
product(lset) ! lset
contains?(v) ! lbool
size() ! lmax
lpset
Non-negative set
;
a[b
sum() ! lmax
lbag
Multiset of values
;
a[b
mult(v) ! lmax
+(lbag) ! lbag
lmap
Map from keys to
lattice values
empty
map
when_true() ! v
at(v) ! any-lat
intersect(lmap) ! lmap
53
Failure Handling
Great question! 
1. Monotone programs handle transient faults
very well
– Deterministic  simple logging
– Commutative, idempotent  simple recovery
2. Future work: “controlled non-determinism”
– Timeout code is fundamentally non-deterministic
– But we still want mostly deterministic programs
Handling Non-Monotonicity
… is not the focus of this talk 
Basic alternatives:
1. Nodes agree on an event order using
distributed coordination (e.g., Paxos)
2. Allow non-deterministic outcomes
•
If needed, compensate and apologize