Distributed Markov Chains
P S Thiagarajan
School of Computing,
National University of Singapore
Joint work with
Madhavan Mukund, Sumit K Jha and Ratul Saha
Probabilistic dynamical systems
• Rich variety and theories of probabilistic dynamical
systems
– Markov chains, Markov Decision Processes (MDPs), Dynamic
Bayesian networks
• Many applications
• Size of the model is a bottleneck
– Can we exploit concurrency theory?
• We explore this in the setting of Markov chains.
Our proposal
• A set of interacting sequential systems.
– Synchronize on common actions.
Our proposal
• A set of interacting sequential systems.
– Synchronize on common actions.
a
Our proposal
• A set of interacting sequential systems.
– Synchronize on common actions.
a
Our proposal
• A set of interacting sequential systems.
– Synchronize on common actions.
– This leads a joint probabilistic move by the participating
agents.
a, 0.2
a, 0.8
Our proposal
• A set of interacting sequential systems.
– Synchronize on common actions.
– This leads a joint probabilistic move by the participating
agents.
a, 0.2
a, 0.8
Our proposal
• A set of interacting sequential systems.
– Synchronize on common actions.
– This leads a joint probabilistic move by the participating
agents.
a, 0.2
a, 0.8
Our proposal
• A set of interacting sequential systems.
– Synchronize on common actions.
– This leads a joint probabilistic move by the participating
agents.
a, 0.2
a, 0.8
Our proposal
• A set of interacting sequential systems.
– Synchronize on common actions.
– This leads a joint probabilistic move by the participating
agents.
– More than two agents can take part in a synchronization.
– More than two probabilistic outcomes possible.
– There can also be just one agent taking part in a
synchronization.
• Viewed as an internal probabilistic move (like in a Markov chain) by the
agent.
Our proposal
• This type of a system has been explored by Pighizzini
et.al (“Probabilistic asynchronous automata”; 1996)
– Language-theoretic study.
• Our key idea:
– impose a “determinacy of communications”
restriction.
– Study formal verification problems using partial
order based methods.
• We study here just one simple verification method.
Some notations
Some notations
Determinacy of communications.
s’’
{a}
s {a}
s’
i
Determinacy of communications.
s’’
{a}
s
s’
i
j
Determinacy of communications.
s’’
a
{a}
s a
a s’
i
j
loc(a) = {i , j}
(s, s’), (s, s’’)  en a
Not allowed!
s’’
{a}
s
s’
k
i
act(s) will have more than
one action.
j
Some notations
Some notations
Example
– Two players each toss a fair coin
– If the outcome is the same, they toss again
– If the outcomes are different, the one who tosses Heads wins
Example
Two component DMC
Interleaved semantics.
Coin tosses are local actions, deciding a winner is
synchronized action
Goal
• We wish to analyze the behavior of a DMC in
terms of its interleaved semantics.
• Follow the Markov chain route.
– Construct the path space .
• The set of infinite paths from the initial state.
• Basic cylinder: a set of infinite paths with a common
finite prefix.
• Close under countable unions and complements.
The transition system view
3
1
3
1
1
1
2
4
3/5
2/5
1
2/5
1
4
1
Pr(B) = 1  2/5  1  1
= 2/5
2
1
1
B – The set of all paths that
have the prefix 3 4 1 3 4
3/5
3
3
1
1
4
B
4
Concurrency
• Events can occur independent of each other.
• Interleaved runs can be (concurrency)
equivalent.
• We use Mazurkiewicz trace theory to group
together equivalent runs: trace paths.
• Infinite trace paths do not suffice.
• We work with maximal infinite trace paths.
(in1, in 2)
t1, 0.5
h1, 0.5
t2, 0.5
(T1, in2)
(H1, in2)
(in1, T2)
(T1, T2)
(H1, T2)
(T1, H2)
W1, L2
L1, W2
w1
l2
W1, L2
l2
W1, L2
W1, L2
w1
h2, 0.5
(in1, H2)
(H1, H2)
The trace space
• A basic trace cylinder is the one generated by
a finite trace
• Construct the -algebra by closing under
countable unions and complements.
• We must construct a probability measure over
this -algebra.
• For a basic trace cylinder we want its
probability to be the product of the
probabilities of all the events in the trace.
(in1, in 2)
t1, 0.5
h1, 0.5
t2, 0.5
(T1, in2)
(H1, in2)
(in1, T2)
(T1, T2)
(H1, T2)
(T1, H2)
Pr(B) = 0.5  0.5 = 0.25
B
W1, L2
L1, W2
w1
l2
W1, L2
l2
W1, L2
W1, L2
w1
h2, 0.5
(in1, H2)
(H1, H2)
The probability measure over the
trace space.
• But proving that this extends to a unique
probability measure over the whole -algebra is
hard.
• To solve this problem :
– Define a Markov chain semantics for a DMC.
– Construct a bijection between the maximal traces of
the interleaved semantics and the infinite paths of
the Markov chain semantics.
• Using Foata normal form
– Transport the probability measure over the path space
to the trace space.
The Markov chain semantics.
The Markov chain semantics.
Markov chain semantics
What if there were 𝑘 players?
𝑘 parallel probabilistic moves generate 2𝑘 global moves
This has a bearing simulation time.
Probabilistic Product Bounded LTL
Local Bounded LTL
• Each component 𝑖has a local set of atomic propositions
𝐴𝑃𝑖
– Interpreted over Si
• Formula of type 𝑖 are atomic propositions 𝑎𝑝 ∈
𝐴𝑃𝑖 , ¬𝜑, 𝜑 ∨ 𝜓 and 𝜑 𝑈 𝑡 𝜓
– 𝜑 𝑈 𝑡 𝜓 ∶ Until holds within t (local) moves of component i
Probabilistic Product Bounded LTL
Local Bounded LTL
• Each component 𝑖has a local set of atomic propositions
𝐴𝑃𝑖
• Formula of type 𝑖 are atomic propositions 𝑎𝑝 ∈
𝐴𝑃𝑖 , ¬𝜑, 𝜑 ∨ 𝜓 and 𝜑 𝑈 𝑡 𝜓
– 𝜑 𝑈 𝑡 𝜓 ∶ Until holds within t (local) moves of component 𝑖
Product Bounded LTL
• Boolean combinations of Local Bounded LTL formulas
Probabilistic Product Bounded LTL
• 𝑃𝑟≥𝛾 (𝜙) where 𝜙 is a Product Bounded LTL formula
• Close under boolean combinations
PBLTL over interleaved runs
• Define 𝑖–projections for interleaved runs .
• Define
for local BLTL formulas and ρ, 𝑗 𝜙
for product BLTL formulas
• Use the measure on traces to define
Statistical model checking…
SPRT based model checking
• In our setting, each local BLTL formula for component 𝑖
fixes a bound on the number of steps that 𝑖 needs to
make ; by then one will be able to decide if the formula
is satisfied or not.
• Product BLTL formula induces a vector of bounds
• Simulate the system till each component meets its bound
– A little tricky we can not try to achieve this bound greedily.
Case study
Distributed leader election protocol [Itai-Rodeh]
• 𝑁 identical processes in a unidirectional ring
• Each process randomly chooses an id in [1. . 𝐾] and
propagates
• When a process receives an id
– If it is smaller than its own, suppress the message
– If it is larger than its own, drop out and forward
– If it is equal to its own, mark collision and forward
• If you get your own message back (message hop count is
𝑁, 𝑁 is known to all processes)
– If no collision was recorded, you are the leader
– If a collision occurred these nodes go to the next round.
Case study…
• In the Markov chain semantics:
– Initial choice of identity: probabilistic move, 𝐾 𝑁 alternatives
– Building the global Markov to analyze system is expensive
– Asynchronous semantics allows interleaved exploration
Case study…
Distributed leader election protocol [Itai-Rodeh]
Case study
Dining Philosophers Problem
• 𝑁 philosophers (processes) in a round table
• Each process tried to eat when hungry, and needs both the
forks to his right and left
• The steps for a process are
– move from thinking to hungry
– when hungry, randomly choose to try and pick up the left or right fork;
– wait until the fork is down and then pick it up;
– if the other fork is free, pick it up; otherwise, put the original
fork down (and return to step 1);
– eat (since in possession of both forks);
– when finished eating, put both forks down in any order and
return to thinking.
Case study…
Dining Philosophers Problem
Other examples
• Other PRISM case studies of randomized distributed
algorithms
– consensus protocols, gossip protocols…
– Need to “translate" shared variables using a protocol
• Probabilistic choices in typical randomized protocols are local
• DMC model allows communication to influence probabilistic
choices
– We have not exploited this yet!
– Not represented in standard PRISM benchmarks
Summary and future work
• The interplay between concurrency and probabilistic
dynamics is subtle and challenging.
• But concurrency theory may offer new tools for factorizing
stochastic dynamics.
– Earlier work on probabilistic event structures [Katoen et al,
Abbes et al, Varacca et al] also attempt to impose
probabilities on concurrent structures.
– Our work shows that formal verification as the goal offers
valuable guidelines
• Need to develop other model checking methods for DMCs.
– Finite unfoldings
– Stubborn sets for PCTL like specifications.