Math 554
Introduction to Stochastic Processes
Instructor: Alex Roitershtein
Iowa State University
Department of Mathematics
Fall 2015
Test #1 (solutions to the sample)
1 [20 Points].
(a) Solve Exercise 1.5 in the textbook.
(b) A fair die is thrown repeatedly. Let Xn denote the sum of the first n throws. Find
lim P(Xn is a multiple of 13).
n→∞
Solution:
(a) There are three communication classes. S1 = {0, 1} and S2 = {2, 4} are recurrent,
and S3 = {3, 5} is transient. Let π = {π0 , π1 } be the stationary distribution of the
chain within the recurrent communication class. Then π0 = 0.5π + 0.7π1 , and hence
π0 = 3/5π1 . Thus π0 = 3/8 and π1 = 5/8. Since P restricted to S1 is aperiodic, it
follows that
lim P n (0, 0) =
n→∞
8
1
= .
π0
3
Furthermore, using the notation of Section 1.5 (see pp. 29-30), we obtain
0.25 0.25 0 0.25
0 0.25
1
−0.25
S=
, Q=
, I −Q=
,
0
0.2 0 0.2
0.2 0.4
−0.2 0.6
1 12 5
1 3 4 0 4
−1
M = (I − Q) =
, A = MS =
,
11 4 20
11 1 5 0 5
and
lim P n (5, 0) = α(5, 0) + α(5, 1) = A(2, 1) + A(2, 2) =
n→∞
6
.
11
(b) Let Yn be the reminder of Xn after dividing by 13. That is Xn − Yn is a multiple of
13 and 0 ≤ Yn ≤ 12. Then Yn is a Markov chain with transition kernel P defined as
follows:
P (i, j) = 1/6 if i + 1 ≤ j ≤ i + 6 or i + 1 − 13 ≤ j ≤ i + 6 − 13,
and P (i, j) = 0 otherwise. In words, if the states from 0 to 12 are arranged around a
circle in the natural order clockwise, then P (i, j) = 1/6 as long as j is located within
the distance between 1 to 6 from i, moving clockwise. It is not hard to check that the
chain is an aperiodic and irreducible and that the uniform distribution on {0, . . . , 12}
1.
is invariant for it. Hence, the limit is equal to 13
1
2 [20 Points].
(a) Solve Exercise 2.6 (a) in the textbook.
(b) Solve Exercise 2.6 (b) in the textbook.
(b) Solve Exercise 2.6 (c) in the textbook.
Solution:
(a) p(i, i + 1) = p and p(i, 0) = 1 − p.
(b) We have:
π0 = (1 − p)
∞
X
πi = 1 − p,
i=0
and, for i ∈ N,
πi = pπi−1 .
Using induction, πi = pi (1 − p).
(c) Since the chain is positive recurrent, E(Tk ) < ∞. Once the state k − 1 reached, in the
next step the chain moves to k with probability p and returns to zero with probability
q. Therefore,
E(Tk ) = E(Tk−1 ) + E(Tk − Tk−1 ) = E(Tk−1 ) + p · 1 + q · 1 + E(Tk )
= E(Tk−1 ) + 1 + qE(Tk ).
Hence,
E(Tk ) =
1 1
+ E(Tk−1 ).
p p
Iterating the last identity, we obtain
1
1 1
1 11 1
1
1
+ E(Tk−1 ) = +
+ E(Tk−2 ) = + 2 + 2 E(Tk−2 )
p p
p p p p
p p
p
k−1
X
1
= ... =
p−j + k−1 E(T1 ).
p
j=1
E(Tk ) =
Since T1 is a geometric random variable with parameter p (i. e., P(T1 = m) = q m−1 p
for m ≥ 1, we have E(T1 ) = p−1 . Thus
E(Tk ) =
k
X
j=1
p−j =
1 1 − p−k
1 −k
=
p
−
1
.
p 1 − p−1
q
2
2 [20 Points]. [20 points]
(a) Solve Exercise 2.7 (a) in the textbook.
(b) Solve Exercise 2.7 (c) in the textbook.
Solution:
To establish whether or not the underlying Markov chain is transient, it is convenient to
use the criterion stated as the Fact on p. 50. Let z = 0 and assume that α(x) defined by
Equations (2.2)–(2.4) on p. 49 exists. In particular, for x > 0 we have:
α(x) =
∞
X
p(x, y)α(y) = p(x, 0)α(0) + p(x, x + 1)α(x + 1)
y=0
= p(x, 0) + p(x, x + 1)α(x + 1),
and hence
α(x + 1) = α(x) ·
p(x, 0)
1
−
.
p(x, x + 1) p(x, x + 1)
Using induction (iterating the above inequality), it is not hard to verify that
α(2) =
α(1)
p(1, 0)
−
p(1, 2) p(1, 2)
and, for x > 1,
x−1
x
x
X
Y
Y
p(x, 0)
p(y, 0)
1
1
α(x + 1) = −
−
+ α(1) ·
p(x, x + 1) y=1 p(y, y + 1) t=y+1 p(t, t + 1)
p(t, t + 1)
t=1
Using the probabilistic meaning of α(x) = Px (Xn = 0 for some n ≥ 0), one conclude that
α(x) is a non-increasing function of x given by
α(x) = 1 −
∞
Y
p(t, t + 1).
(1)
t=x
(a) It follows from (1) that α(x) ≡ 1 and hence the chain is recurrent. To see whether the
chain is positive- or null-recurrent, let T = inf{k > 0 : Xk = 0} and compute
E0 (T ) =
=
∞
X
n=2
∞
X
n=2
∞
n−2
X
Y
nP0 (T = n) =
n
p(t, t + 1) · p(n − 1, 0)
n=2
n
n−2
Y
t=0
t=0
∞
X
n
t+1
·
=
t+2 n+1
Thus the chain is null-recurrent.
3
n=2
n
= +∞.
n+1
(c) It follows from (1) that
α(x) = 1 −
∞ Y
1−
t=x
1 ,
t2 + 2
and hence limx→∞ α(x) = 0. Thus the chain is transient.
4 [20 Points].
(a) Solve Exercise 2.8 (a) in the textbook.
(b) Solve Exercise 2.8 (c) in the textbook.
Solution:
(a) a is the minimal solution of φ(a) = a in (0, 1]. We have:
a = p0 + p1 a + p2 a2 ⇐⇒ 35a2 − 60a + 25 = 0
Hence a = 5/7.
(c) We have, µ = 0.05 + 0.01(2 + 3 + 6 + 13) = 0.29 < 1. Thus the extinction is certain.
5 [20 Points]. Solve Exercise 2.16 in the textbook.
Solution:
(a) By the definition of the invariant measure,
π(n) =
∞
X
π(m)p(m, n) =
m=0
∞
X
π(m)pn−m ,
n ≥ 1.
(2)
m=0
Since pn−m = 0 if n − m > 1, it follows that π(n) =
P∞
m=n−1
π(m)pn−m .
(b) Let Y be a random variable valued in the set of non-negative integers, such that
P(Y = n) = qn , n ≥ 0. Then
E(Y ) =
∞
X
qn n =
n=0
= −
∞
X
p1−n n = −
n=0
n=0
X
∞
X
p1−n (1 − n) +
∞
X
p1−n
n=0
pk k + 1 = −µ + 1 > 0,
k
since it is given that µ < 0. To conclude that the equation α = E(αY ) has a root within
the interval (0, 1), one can use the same argument as for the super-critical branching
processes.
(c) This involves a guess. The correct answer isPπ(n) = cαn which can be validated
directly using (2) together with the fact that ∞
n=0 π(n) = 1. The latter also implies
c = (1 − α)−1 .
4