Math 554
Introduction to Stochastic Processes
Instructor: Alex Roitershtein
Iowa State University
Department of Mathematics
Fall 2015
Solutions to Homework #2
1.7
In what follows, the state j is fixed throughout. We want to show that the probability of
avoiding a subset of the state space (a certain state j in particular) for a long time decays
exponentially with the time. Informally speaking, the idea of the proof is as follows. Find
Ki ∈ N so large such that the probability of visiting to j during the first Ki steps is positive
when X0 = i. This is possible since Xn is an irreducible Markov chain. Since the state space
is finite, the probability of visiting to j during the first K := max1≤i≤N Ki steps is bounded
away from zero by a strictly positive constant ρ ∈ (0, 1), no matter where we started. Then,
using the Markov property, argue that avoiding j during the first nK steps can be compared
to the “worst-case scenario” of getting n “failures” in a row in a sequence of n i.i.d. cointossing trials with the probability of “success” in a trial equals 1 − ρ > 0. In order to make
this argument rigorous, one needs to carefully verify that the Markov property indeed makes
the very last step of the proof working. This is not obvious straight from the definitions
because an application of the Markov property per se requires conditioning on a particular
current state. The problem is that just by indicating that a state j has not been visited for
a while, one does not specify the current location of the Markov chain. Thus a strong form
of the Markov property is required here. A sufficient for our purpose version of the Markov
property is provided below in Lemma 1.
Let Ω := {1, . . . , N } denote the state space of Xn . We will use without a further notice
the Markov property in the following form. For any integer times i, m ∈ N, sets Ak (including
the possibility of being empty sets, singletons, or the whole finite state space), and a state i,
P Xk ∈ Ak : n + 1 ≤ k ≤ n + mXn = i, Xk ∈ Ak : 0 ≤ k ≤ n − 1
= P Xk ∈ Ak : n + 1 ≤ k ≤ n + mXn = i
= P Xk ∈ Ak : 1 ≤ k ≤ mX0 = i
(1)
We will also need the following consequence of the Markov property:
Lemma 1. Let E ⊂ [0, 1] be a subset of the closed interval [0, 1]. Consider an integer m ∈ N
and a sequence of events Ak ⊂ Ω, k ≥ 0. If
P Xk ∈ Ak : 1 ≤ k ≤ mX0 = i ∈ E
(2)
for all i ∈ Ω, then
P Xk ∈ Ak : n + 1 ≤ k ≤ n + mXk ∈ Ak : 0 ≤ k ≤ n ∈ E,
for any integer n ∈ N.
1
Proof of the lemma. By virtue of the Markov property, we have:
P Xk ∈ Ak : n + 1 ≤ k ≤ n + mXk ∈ Ak : 0 ≤ k ≤ n
X
=
P Xn = iXk ∈ Ak : 0 ≤ k ≤ n
i∈Ω
×P Xk ∈ Ak : n + 1 ≤ k ≤ n + mXn = i, Xk ∈ Ak : 0 ≤ k ≤ n
X
=
P Xn = iXk ∈ Ak : 0 ≤ k ≤ n · P Xk ∈ Ak : n + 1 ≤ k ≤ n + mXn = i .
i∈Ω
This completes
P the proof of the lemma in view of the fact that condition (2) holds for any
i ∈ Ω while i∈Ω P Xn = iXk ∈ Ak : 0 ≤ k ≤ n = 1.
The following remark elucidates the result in lemma but its content is beyond the scope.
Ignoring this remark completely or partially is of absolutely no harm :) If you want to
read more about conditional expectation, you can consult, for instance, the introductory
section in the Chapter about martingales (Chapter 5 in two recent editions) of Durrett’s
text Probability Theory and Examples. We will discuss conditional expectation later in this
course while studying martingales.
Remark for an “overqualified fellow”. Using the machinery of conditional expectation,
the lemma can be proved as follows. Let Fn = σ(X0 , . . . , Xn ) be a σ-algebra of the past events
and F n = σ(Xk : k > n) be a σ-algebra of the future events. Then for any A ∈ F n , with
probability one,
P(A|Fn ) = E P(A|Xn , Fn ) = E P(A|Xn ) ,
from which the lemma follows. The first identity in the display is a general property of
conditional probabilities, the second one is the Markov property.
Back to the solution of Exercise 1.7. Since Xn is irreducible, for each state i there exist
an integer Ki ∈ N and a positive real δi > 0 such that
P(XKi = j|X0 = i) = δi .
Let K = max1≤i≤N Ki and δ = min1≤i≤N δi Then, for any i ∈ Ω
P Xm 6= j, m = 0, . . . , K X0 = i ≤ P Xm 6= j, m = 0, . . . , Ki X0 = i
≤ P XKi 6= j X0 = i = 1 − P XKi = j X0 = i = 1 − δi ≤ 1 − δ < 1.
For simplicity of notations, for an integer t ≥ 0 and a state i ∈ Ω, introduce the following
events:
At := Xm 6= j, K(t − 1) ≤ m ≤ Kt
Bt (i) := X0 = i and Xm 6= j, 0 ≤ m ≤ Kt .
2
For a real number x ∈ R let bxc denote the integer part of x, that is
bxc := max{n ∈ N : n ≤ x}.
By Lemma 1, for any n ≥ K and any state i ∈ Ω we have
P Xm 6= j, m = 0, . . . , nX0 = i ≤ P Xm 6= j, m = 0, . . . , Kbn/KcX0 = i
=P
∩ts=1 As X0
= i0
bn/Kc
Y
P At X0 = i, ∩t−1
A
= P A1 X0 = i0 ×
s
s=1
t=2
bn/Kc
≤ (1 − δ)
where C0 := (1 − δ)−
set
C1 :=
K−1
K
n
≤ C0 ρ ,
and ρ := (1 − δ)1/K . To cover the case n < K, one can for instance
max max ρ−n · P Xm 6= j, m = 0, . . . , nX0 = i ,
0≤n≤K−1 i∈Ω
and C = max{C0 , C1 }.
1.9 (a)
Then chain is irreducible. For instance, the following loop path has a positive probability
and visits at every state:
1 → 2 → 4 → 1 → 3 → 5 → 1.
1.9 (b)
Period of the chain equals to the period of any of its state. Consider, for instance, the state
5. From 5 one can go only to 1. From 1 the chain moves to either 2 or 3. From either 2 or
3 the chain can only move into {4, 5}. If it moves to 5, then 5 has been reached in 3 steps.
If it moves to 4, then in three steps (overall, in six steps from the beginning), the chain will
be again in {4, 5}. Using the induction, one can conclude that period os the state 5 is three.
Thus d = 3.
1.9 (d)
From 1 the chain moves into {2, 3}. From {2, 3} the chain can only move into {4, 5}. Finally,
from {4, 5} the chain can only move to 1. Thus
P(T = 3|X0 = 1) = 1
and hence
E(T |X0 = 1) = 3.
It follows from the law of large numbers stated on p. 25 of the textbook that
n
1
1 X
P(Xm = 1|X0 = 1) = .
π(1) = lim
n→∞ n + 1
3
m=0
1.11
3
Let Zn ∈ {0, 1 . . . , 7} be the reminder of Sn after division by 8. Clearly (verify this though!),
Zn is a Markov chain with P(Z0 = 0) = 1 and transition matrix given by


0 1/6 1/6 1/6 1/6 1/6 1/6 0
 0
0 1/6 1/6 1/6 1/6 1/6 1/6 


 1/6 0

0
1/6
1/6
1/6
1/6
1/6


 1/6 1/6 0

0
1/6
1/6
1/6
1/6

P=
 1/6 1/6 1/6 0
0 1/6 1/6 1/6 


 1/6 1/6 1/6 1/6 0

0
1/6
1/6


 1/6 1/6 1/6 1/6 1/6 0
0 1/6 
1/6 1/6 1/6 1/6 1/6 1/6 0
0
For i ∈ S, let Ti = inf{k ≥ 1 : Xk = i and ui = E(Ti |Z0 = 0). Using the result of Exercise 1.15
(stated also as formula (1.1) on p. 25), we can conclude that
E(T0 |Z0 = 0) = u0 =
1
.
π(0)
Since
π(i) =
1
for any state i,
8
(3)
we obtain that
u0 = 8.
In order to obtain (3), one can first produce a guess based on a heuristic symmetry argument
(for instance, putting the states on a circle, observing that P looks then the same from
every state, and using the interpretation of π as the vector of asymptotic frequencies) and
then make the argument rigorous by a simple checking that the proposed π is the (unique)
probability vector solving πP = π,
For i ∈ S, let vi = E(T1 |Z0 = i). It is computationally considerably more complicated to
compute without help of a calculator/ computer/ friend
E(T1 |Z0 = 0) = v0 .
Let v = (v0 , v2 , v3 , v4 , v5 , v6 , v7 ) ∈ R7 and denote e = (1, 1, 1, 1, 1, 1, 1) ∈ R7 . Then, conditioning on the first step of the Markov chain,
v = e + Qv,
where Q is the matrix obtained from P by removing the second row and second column
(corresponding to the state 1) from matrix P. Thus v = (I − Q)−1 e (we actually have
recovered the result obtained on the bottom of p. 28 in the textbook). You can use MATLAB
to find the solution of this linear system and hence to determine v0 . I believe, v0 ≈ 6.8.
4