Global States in a Distributed
System
By John Kor and Yvonne Cheng
Initial Problem Example
Garbage Collector
 Free’s up memory which is no longer in use
 Check’s if a reference to memory still exists
What about in a distributed system
Initial Problem Example (cont’d)
A distributed system consists of multiple
processes
Each process is located on a different
computer
 No sharing of processor or memory
Initial Problem Example (cont’d)
Each process can only determine its
own “state”
Problem: How do we determine when to
garbage collect in a distributed system?
 How do we check whether a reference to
memory still exists?
System Model
 A distributed system consists of multiple
processes
 Each process is located on a different
computer
 Each process consists of “events”
 An event is either sending a message,
receiving a message, or changing the value of
some variable
 Each process has a communication channel in
and out
Our Garbage Collection Problem
In order to test whether a certain
property of our system is true, we cannot
just look at each process individually
A “snapshot” of the entire system must
be taken to test whether a certain
property of the system is true
This “snapshot” is called a Global State
Definition
The global state of a distributed system is the
set of local states of each individual
processes involved in the system plus the
state of the communication channels.
Determinism
Deterministic Computation
 At any point in computation there is at most
one event that can happen next.
Non-Deterministic Computation
 At any point in computation there can be
more than one event that can happen next.
Deterministic Computation
Non-Deterministic Computation
Determinism
Deterministic computation
 A local event would reveal everything about
the global state!
 The process will know other process’ state
Non-Deterministic computation
 Because of branching, a local event cannot
reveal what the next step will be
Simple Algorithm
Create a new process that collects the
states of every other process
Every process will save their state at an
arbitrary time and send it to this new
process
Advantages
Very simple
Easy to implement
Problems?
Based on the assumption that all
processes work on a synchronized
global clock
Wrong assumption!
Problems (cont’d)
State recorded by p
p
m
q
Problems (cont’d)
p
q
m
Problems (cont’d)
State recorded by q
p
q
m
Problems (cont’d)
Global state recorded
p
q
m
m
Another view
p
m
q
Another view
Process p has no record of sending m
Process q HAS record of receiving m
 Problem?
 Global state does not show p sending m,
therefore there is confusion as to where m
came from
 Breaks the Consistency concept
Consistency
A global state is consistent if it could
have been observed by an external
observer
If e  e` , then both e and e` must
reside within the same state
For a successful Global State, all states
must be consistent
Solution
Need to develop an asynchronous
algorithm
Cannot depend on a clock
Must ensure consistency in all global
states
Assumptions
 Distributed system: Finite set of processes
and channels; described by graph
 Processes
 Set of states, initial state, set of events
 Channels
 FIFO, error-free, infinite buffers, arbitrary but finite
delay
PART 2
Presented By: Yvonne
Idea of a global state recording
algorithm
- each process records its own state
- the two processes incident by one
channel cooperate in recording the
channel state
Challenge
- No global clock
- Need a meaningful result
- Superimposed on underlying computation
Meaningful:
The notion of Consistency
- it could have been observed by an
external observer
- All feasible states are consistent
An Example
q
p
p
Sp0
Sp1
Sp2
Sp3
m2
m1
q
Sq0
m3
Sq1
Sq2
Sq3
A Consistent State?
p
Sp0
p
q
Sp 1
Sq1
Sp1
Sp2
Sp3
m2
m1
q
Sq0
m3
Sq1
Sq2
Sq3
Yes
p
Sp0
p
q
Sp 1
Sq1
Sp1
Sp2
Sp3
m2
m1
q
Sq0
m3
Sq1
Sq2
Sq3
A Consistent State?
p
Sp0
p
q
Sp 2
Sq3
Sp1
m3
Sp2
Sp3
m2
m3
m1
q
Sq0
Sq1
Sq2
Sq3
Yes
p
Sp0
p
q
Sp 2
Sq3
Sp1
m3
Sp2
Sp3
m2
m3
m1
q
Sq0
Sq1
Sq2
Sq3
An inconsistent State
p
Sp0
p
q
Sp 1
Sq3
Sp1
Sp2
Sp3
m2
m1
q
Sq0
m3
Sq1
Sq2
Sq3
Conducting algorithm:
Using An Example
- Processes: p and q
- Channels: c and c’
- Token: t
p
q
c
c’
An Example
- p records its state
p
q
c
t
c’
An Example
- q, c, and c’ record their states
p
q
c
t
c’
An Example
- The composite global state!
p
q
c
t
t
c’
An Example
- n: number of messages sent along c
before p’s state is recorded
- n’: number of message sent along c
before c’s state is recorded
p
q
c
c’
An Example
- Reason of inconsistency: n<n’
p
q
c
t
n=0
c’
p
q
c
t
n’ = 1
c’
Similar scenario
c is recorded when the token is at process p.
p sends the token through channel c, and the
states of c’, p, and q are recorded.
The recorded global state : no tokens in the
system.
The reason of inconsistency : n>n’
Conclusion from the example
A consistent global state
requires
n = n’
Similar Conclusion
m : number of messages received along c
before q’s state is recorded
m’ : number of messages received along c
before c’s state is recorded
To be consistency:
m=m’
Some other equations
n’ >= m’
 m’ : number of messages received along c before c’s state is
recorded
 n’ : number of messages sent along c before c’s state is
recorded
 m : number of messages received along c before p’s state is
recorded
 n : number of messages sent along c before p’s state is
recorded
n >= m
 n = n’
 m = m’
Other Fact
 The state of channel c that is recorded must be the
sequence of messages sent along the channel before
the sender’s state is recorded, excluding the sequence
of messages received along the channel before the
receiver’s state is recorded.
 Two cases:
 n’=m’ : c is empty
 n’>m’: c must be the (m’+1)st…n’th messages sent by p
along c
Put All Together:
A brief sketch of the algorithm
 p sends a marker message along all its outgoing
channels after it records its state and before it
sends any other messages.
 On receipt of a marker message from channel c
else
state ( c ) = messages received on c since it
had recorded its state excluding the marker.
if p has not recorded its state
record the state
state ( c ) = EMPTY
Chandy and Lamport Algorithm
Features:
 Does not promise us to give us exactly what is
there
 But gives us consistent state!!
Algorithm in Action
Sp0
p
q
Sp1
m1
Sq0
Sp2
m2
Sq1
Sp3
m3
Sq2
Sq 3
Algorithm in Action
q records state as Sq1 , sends marker to p
Sp0
p
q
Sp1
m1
Sq0
Sp2
m2
Sq1
Sp3
m3
Sq2
Sq 3
Algorithm in Action
p records state as Sp2, channel state as empty
Sp0
p
q
Sp1
m1
Sq0
Sp2
m2
Sq1
Sp3
m3
Sq2
Sq 3
Algorithm in Action
q records channel state as m3
Sp0
p
q
Sp1
m1
Sq0
Sp2
m2
Sq1
Sp3
m3
Sq2
Sq 3
Algorithm in Action
Recorded Global State = ((Sp2, Sq1), (0,m3) )
Sp0
p
q
Sp1
m1
Sq0
Sp2
m2
Sq1
Sp3
m3
Sq2
Sq 3
Algorithm in Action
Recorded Global State = ((Sp2, Sq1), (0,m3) )
Sp0
p
q
Sp1
m1
Sq0
Sp2
m2
Sq1
Sp3
m3
Sq2
Computation may not even have
passed through the state recorded!
Sq3
What have we recorded
 The recorded consistent state can be anything!
Properties of the recorded global state
Si : global state when the algorithm starts
Sj : global state when the algorithm finishs
S*: state recorded by the algorithm
Then
 S* is reachable from Si
 Sj is reachable from S*
S* Is reachable from Si
Si
Sj
Sj Is reachable from S*
Si
Sj
Still what good is it?
Stable Properties
A property Y is called a stable property
iff for all states S` reachable from S
Y(S) -> Y(S’)
Detection of Stable Properties
Outcome = false;
while ( outcome == false )
{
determine Global State S;
outcome = Y (S);
}
Checkpoint
 S* serves as a checkpoint
 On a failure, restart the
computation from S*
 Problem!
 Not able to restore to Sj
Si
S*
Sj
Solution: Publishing
A Broadcast medium
A central recorder process records all the
messages received by each process
Processes record their states at their own
time and send it to the recorder
Determining Global State
Recorder can construct global state from
 Checkpointed States of all processes
Plus
 Messages recorded since last checkpoint
Problems
Publishing keeps track of all messages
received by each process
Expensive!
Solution
 recorder takes checkpoint of process p at
time t
 deletes all messages recd by p before t.
Comparison
SNAPSHOT PUBLISHING
Network
Strongly
connected
Need not be
Mode
Distributed
Centralized
Scalability
Yes
No
Restorability
No
Yes
Conclusion
Global State detection difficult in
Distributed Systems
Snapshot algorithm may not give an
actual state but is very helpful in
detecting Stable Properties
Publishing gives an asynchronous way
of determining global states but is
unscalable