MATH 5040/6810: Homework 2
Due on Thursday, Sep. 22, by the end of lecture.
Problem 1
a) Consider the following transition matrix.
are transient and which are recurrent? What

.1 0 0
 .1 .2 .2

 0 .1 .3

 .1 0 0

 0 0 0
0 0 0
Identify the communication classes. Which
is the period of each state?

.4 .5 0
0 .5 0 

0 0 .6 

.9 0 0 

.4 0 .6 
0 .5 .5
b) Decide whether the following matrix is irreducible and/or aperiodic and then decide if
it has an invariant distribution. If yes, find it. If not, explain why.
1 2 3
1 .4 .6 0
2 .2 .4 .4
3 0 .3 .7
Problem 2
Suppose that the probability it rains today is 0.3 if neither of the last two days was rainy,
but 0.6 if at least one of the last two days was rainy. Let the weather on day n , Wn , be R
for rain, or S for sunny.
(1) Show that Mn = (Wn+1 , Wn ) is a four-state, irreducible, aperiodic Markov chain
by finding and analyzing the transition matrix.
(2) If there wasn’t a single drop of rain during the weekend, when can we expect two
days of consecutive sunshine again?
(3) A farmer planted seeds during that sunny weekend and now he really wants two
consecutive days of rain. What is the expected number of days he needs to wait for
that?
Problem 3
What is the expected number of throws of a fair coin until we see 3 heads in a row?
Hint: Find a suitable four-state Markov chain that keeps track of consecutive number
of heads until you see three of them in succession.
Problem 4
You may want to use a computer. Look at the figure before you answer the questions.
Consider an ant performing a random walk on the graph displayed in the picture. The ant
can move to adjacent vertices of the graph and it is choosing its next position with equal
probability among all available. The ant started at vertex B.
1
2
D
A
C
H
G
E
F
B
(1) You have been watching this ant go for a long time. What is the probability that
is on vertex A?
(2) What is the expected number of steps before returning to B?
(3) What is the expected number of steps before reaching A?
(4) What is the probability that it reaches H before reaching A?
(5) What is the probability that it reaches the inner square before reaching A?
Problem 5 Consider the two state Markov chain with transition matrix
a
1−a
1−b
b
1 the time of first return to state i if we started the chain from state i as defined
and let Ti,i
in lecture.
1 explicitly, i.e. find P{T 1 = k} for i = 0, 1.
(1) Find the distribution of Ti,i
i,i
1 ).
(2) Find E(Ti,i
Problem 6
Suppose Xn and Yn are independent Markov chains on finite state spaces R and S
respectively. We showed that Zn = (Xn , Yn ) is a Markov chain on R × S. Assume further
that Xn , Yn are irreducible and aperiodic, with corresponding invariant distributions π X ,
πY .
(1) Show that Zn is irreducible and aperiodic.
(2) What is the invariant distribution of Zn = (Xn , Yn )?
If this turns out to be too complicated you may submit for half the credit the same
problem with Xn , Yn being independent copies of the two-state Markov chain with transition
matrix as in Problem 5.