STOCHASTIC MODELLING – WEEK 5 TUTORIAL
STATIONARY PROBABILITIES, FIRST PASSAGES
(1) (easy) Find the stationary probabilities of a the simple weather chain with S = {0, 1}
(sunny, rainy) and
0.6 0.4
P =
0.2 0.8
Solution: We solve the global balance equations
0.6 0.4
(π0 , π1 ) = (π0 , π1 )
0.2 0.8
with π0 + π1 = 1 to obtain π0 = 1/3 and π1 = 2/3.
(2) (easy) A chain is called doubly
P stochastic if all of its columns also sum to one (the rows
already sum to one), that is i∈S pij = 1 for all j. Show that for an irreducible doubly
stochastic chain with N states the stationary probabilities are πi = 1/N for all i.
Solution: Just substitute πi = 1/N into the global balance equations
X
πi pij =
i
and check that
P
1 X
1
pij =
= πj
N
N
i
πi = 1. For an irreducible chain this is the unique stationary solution.
(3) (easy) The detailed balance condition means that πi pij = πj pji for any i, P
j ∈ S, that is the
probability flow between any pair of states balances out (and additionally i πi = 1). Show
that if the detailed balance condition is satisfied then the global balance condition is also
satisfied.
Solution: Sum both sides of the detailed balance condition over j to obtain the global
balance condition
X
X
πi pij = πi =
πj pji
j
and
P
j
πi = 1 was assumed.
(4) (easy) Consider the following DTMC:


0.5 0.5 0
P = 0.3 0.1 0.6
0.2 0.4 0.4
(a) Calculate the stationary probabilities. (Hint: can you use your previous results?)
(b) Show that it doesn’t satisfy detailed balance.
Solution:
(a) This chain is doubly stochastic, hence π = (1/3, 1/3, 1/3).
(b) There cannot be detailed balance since p13 = 0, and p31 6= 0, which implies π3 = 0,
and not 1/3.
(5) (standard) Birth and death chains on a state space {`, ` + 1, . . . , r − 1, r} are defined by the
property that they cannot move by more than one site in a single step (describing a single
birth or death event or that nothing happened), that is
pi,i+1 = pi ,
l≤i<r
pi,i−1 = qi ,
`<i≤r
pi,i, = 1 − pi − qi ` ≤ i ≤ r
2
SMO – WEEK 5 TUTORIAL
with all pi , qi > 0, and all other transition probabilities are zero. Show that the stationary
probabilities are
p` p`+1 · · · p`+i−1
π`+i = π`
q`+1 q`+2 · · · q`+i
P
where π` can be determined from the normalization ri=` πi = 1. (Hint: this chain has to
satisfy detailed balance (why?), but you can also use the global balance equations).
Solution: For any i < r state we can write down the detailed balance equation πi pi =
πi+1 qi+1 , from which
pi
πi
πi+1 =
qi+1
Using this repeatedly starting from i = ` we get
π`+1 =
π`+2 =
..
.
p`
q`+1 π`
p` p`+1
q`+1 q`+2 π`
which leads to the result. To prove it just use induction.
(6) (standard) Random walk on graphs: A graph is a set of vertexes where some of them
are connected by edges. The adjacency matrix A describes which vertexes are connected:
aij = 1 if i and j vertexes are connected and zero otherwise. We assume P
that aii = 0 for all
i. The degree δi of vertex i is the number of edges connected to it, δi = j aij . Consider a
random walk Xn which takes values from the vertexes, and in each time step it jumps to a
randomly chosen connected vertex.
(a) Write down the one-step transition probabilities pij .
(b) Show that for a connected graph (why do we need this?) the stationary distribution
is πi = cδi , and give the value of c. (Hint: use either detailed or global balance.)
Solution:
(a)
aij
δi
(b) πi satisfies the detailed balance condition
pij =
πi pij = caij = caji = πj pji
P
and from i πi = 1 we get c = 1/ i δi .
Alternatively, we can use the global balance equations
X
X aij
X
X
πj =
πi pij =
cδi
=c
aij = c
aji = cδj = πj
δi
P
i
i
i
i
where we used that aij = aji be definition.
(7) (standard) There are two rooms and there is a spider in the second room. A fly flies around
every morning and lands in a room according to the one-step transition matrix
0.5 0.5
P =
0.6 0.4
If the fly lands in the room with the spider, then the spider immediately catches it with
probability 0.7.
(a) Formulate this problem as a DTMC. (Hint: we need an extra state for the fly getting
caught.)
(b) What is the expected time until the fly is caught by the spider, if the fly is initially in
one of the rooms? (hint: condition on first step.)
Solution:
(a) We define the state space as S = {1, 2, D}, where 1 and 2 means the room where the
fly is if it’s alive, and D corresponds to the state where the fly is dead. Then, the one
SMO – WEEK 5 TUTORIAL
3
step transition probability can be written as


0.5 0.15 0.35
P = 0.6 0.12 0.28
0
0
1
(b) Let T̂ = min{n ≥ 0 : Xn = D}, and gi = E(T̂ |X0 = i), the expected number of steps
starting from state i until the fly is caught. We know that gD = 0 and for the other
states
g1 = 1 + 0.5g1 + 0.15g2 + 0.35gD
g2 = 1 + 0.6g1 + 0.12g2 + 0.28gD
Solving these two equations we find that g1 = 2.94286 and g2 = 3.14286. So it’s better
to start from room 2 if you are a fly.
(8) (hard) A spider is hunting a fly and moves randomly between two rooms according to the
following one-step transition matrix
0.3 0.7
P =
0.6 0.4
The fly, unaware of the spider, also moves according to the one-step transition matrix
0.5 0.5
P =
0.6 0.4
If spider and fly arrive into the same room, then the spider immediately catches the fly
with probability 0.7.
(a) Formulate this problem as a DTMC. (Hint: to describe the location of both animals
we need a size four state space, but we need an extra state for the fly getting caught.)
(b) What is the expected time until the fly is caught by the spider?
Solution:
(a) We define the state space as S = {(1, 1), (1, 2), (2, 1), (2, 2), D}. For example, Xn =
(2, 1) means that the spider is in room 2 and the fly is in room 1 and alive. D
corresponds to the state where the fly is dead. Then, the one step transition probability
can be written as


0.045 0.15 0.35 0.105 0.35
 0.054 0.12 0.42 0.084 0.332 



P =
 0.09 0.3 0.2 0.06 0.35 
 0.108 0.24 0.24 0.048 0.364 
0
0
0
0
1
(b) Let again T̂ = min{n ≥ 0 : Xn = D}, and gi = E(T̂ |X0 = i), the expected number of
steps starting from state i until the fly is caught. We know that gD = 0 and for the
other states
g11 = 1 + 0.045g11 + 0.15g12 + 0.35g21 + 0.105g22 + 0.35gD
g12 = 1 + 0.054g11 + 0.12g12 + 0.42g21 + 0.084g22 + 0.35gD
g21 = 1 + 0.09g11 + 0.3g12 + 0.2g21 + 0.06g22 + 0.35gD
g22 = 1 + 0.108g11 + 0.24g12 + 0.24g21 + 0.048g22 + 0.35gD
Solving these four equations we find that g11 = 2.89339, g12 = 2.97351, g21 = 2.90512,
and g22 = 2.86067. Flies do not live long.