ST3236: Stochastic Process
Tutorial 4
TA: Mar Choong Hock
Email: g0301492@nus.edu.sg
Exercises: 5
Question 1
An urn initially contains a single red and single
green ball. A ball is drawn at random, removed and
replaced by a ball of the opposite color and this
procedure repeats so that there are always two balls
in the urn.
Let Xn be the number of red balls in the urn after n
draws, with X0 = 1.
Specify the transitions probabilities for MC {X}.
Question 1
Case (Xn=0): Both balls are green. One ball will
certainly be replaced with red after a ball is drawn.
P(Xn+1 = 1 | Xn = 0) = 1, P(Xn+1 = 0 | Xn = 0) = 0,
P(Xn+1 = 2 | Xn = 0) = 0
Case (Xn=2): Both balls are red. One ball will
certainly be replaced with green after a ball is
drawn.
P(Xn+1 = 1 | Xn = 2) = 1, P(Xn+1 = 0 | Xn = 2) = 0,
P(Xn+1 = 2 | Xn = 2) = 0
Question 1
Case (Xn = 1): One ball is red. Outcome dependent
on the colour of the ball drawn.
Note: P(Green is drawn|Xn=1) =
P (Red is drawn|Xn=1) = 0.5
P(Xn+1 = 0 | Xn = 1) = P(Red is drawn|Xn = 1) = 0.5
P(Xn+1 = 2 | Xn = 1) = P(Green is drawn|Xn = 1) = 0.5
P(Xn+1 = 1 | Xn = 1) = 0
Question 1
The transition matrix is given as follows:
Question 2
Find the mean time to reach state 3 starting from
state 0 for the MC whose transition probability
matrix is
Question 2
Let T = min{n : Xn = 3} and vi = E(T | X0 = i). The
mean time to reach state 3 starting from state 0 is
v0. We apply first step analysis.
Recursive Formula for average time step
required:
vo  1   p0i vi
i
0
i
1 step
3
vi
Question 2
Therefore,
v0 = 1 + 0.4v0 + 0.3v1 + 0.2v2 + 0.1v3
v1 = 1 + 0v0 + 0.7v1 + 0.2v2 + 0.1v3
v2 = 1 + 0v0 + 0v1 + 0.9v2 + 0.1v3
v3 = 0
Solving the equations, we have
v0 = 10.
Question 3
Consider the MC with transition probability matrix
(a) Starting in state 1, determine the probability that
the MC ends in state 0
(b) Determine the mean time to absorption.
Question 3a
Let T = min{n : Xn = 0 and Xn = 2},
ui = P(XT = 0|X0 = i) and vi = E(T|X0 = i).
u0 = 1
u1 = 0.1u0 + 0.6u1 + 0.3u2
u2 = 0
we have u1 = 0.25.
Question 3b
Let T = min{n : Xn = 0 and Xn = 2},
ui = P(XT = 0|X0 = i) and vi = E(T|X0 = i).
v0 = 0
v1 = 1 + 0.1v0 + 0.6v1 + 0.3v2
v2 = 0
we have v1 = 2.5.
Question 4
Consider the MC with transition probability matrix
(a) Starting in state 1, determine the probability that
the MC ends in state 0
(b) Determine the mean time to absorption.
Question 4a
Let T = min{n : Xn = 0 and Xn = 2},
ui = P(XT = 0|X0 = i) and vi = E(T|X0 = i).
u0 = 1
u1 = 0.1u0 + 0.6u1 + 0.1u2 + 0.2u3
u2 = 0.2u0 + 0.3u1 + 0.4u2 + 0.1u3
u3 = 0
we have,
u1 = 0.3810, u2 = 0.5238
Starting in state 1, the probability that the MC ends
in state 0 is u1 = 0.3810
Question 4b
Let T = min {n : Xn = 0 and Xn = 2},
ui = P(XT = 0|X0 = i) and vi = E(T|X0 = i).
v0 = 0
v1 = 1 + 0.1v0 + 0.6v1 + 0.1v2 + 0.2v3
v2 = 1 + 0.2v0 + 0.3v1 + 0.4v2 + 0.1v3
v3 = 0
we have v1 = v2 = 3.33.
Question 5
A coin is tossed repeatedly until two successive
heads appear. Find the mean number of tosses
required.
[Hint: Let Xn be the cumulative number of
successive heads. Then the state space is 0,1,2
before stop]
Question 5 – Method 1
Let Yn be the outcome {H, T} of each toss and
(Yn-1, Yn) denotes the sample point for the sucessive
tosses. There are 4 possible sample points.
Denote :
0
1

Zn  
2
3
T , T 
T , H 
H ,T 
H , H 
Denote: States = {(Yn-1, Yn)}
Where Yn = H if head at nth toss and T if otherwise.
0
(T,T)
P(Yn=H)=0.5
1 P(Yn=H)=0.5 3
(T,H)
(H,H)
P(Yn=T)=0.5
P(Yn=H)=0.5
P(Yn=T)=0.5
P(Yn=T)=0.5
2
(H,T)
1
Question 5 – Method 1
The transition probability matrix is
0
0.5 0.5 0
0
0 0.5 0.5

P
0
0.5 0.5 0
0

0
0
1


Question 5 – Method 1
Let vi denote the mean time to each state 3 starting
from state i.
By the first step analysis, we have the following
equations:
v0 = 1 + 0.5v0 + 0.5v1
v1 = 1 + 0.5v2 + 0.5v3
v2 = 1 + 0.5v0 + 0.5v1
v3 = 0
Therefore, v0 = 6 (v1 = 4, v2 = 6)
Question 5 – Method 2
Let Xn be the cumulative number of successive
heads. The 3-state state space now is {0,1,2}.
Example:
n
Yn
0
T
1
H
2
T
3
H
4
H
Xn
0
1
0
1
2
Question 5 – Method 2
Case (Xn = 0) :Previous two tosses are tails Or a
head followed by tail. Note: P(Head) = P(Tail) = 0.5
P(Xn+1 = 1 | Xn = 0) = P(Head) = 0.5,
P(Xn+1 = 0 | Xn = 0) = P(Tail) = 0.5,
P(Xn+1 = 2 | Xn = 0) = 0
Case (Xn = 1): A tail is followed by a head
P(Xn+1 = 2 | Xn = 1) = P(Head) = 0.5
P(Xn+1 = 0 | Xn = 1) = P(Tail) = 0.5
P(Xn+1 = 1 | Xn = 1) = 0
Question 5 – Method 2
Case (Xn = 2): Previous two tosses are heads. We
make this state absorbing.
P(Xn+1 = 1 | Xn = 2) = 0
P(Xn+1 = 2 | Xn = 2) = 1,
P(Xn+1 = 0 | Xn = 2) = 0
Question 5 – Method 2
The transition probability matrix is
Question 5 – Method 2
Let vi denote the mean time to each state 2 starting
from state i.
By the first step analysis, we have the following
equations:
v0 = 1 + 0.5v0 + 0.5v1
v1 = 1 + 0.5v0 + 0.5v2
v2 = 0
Thus, we have,
v0 = 6, v1 = 4