Solutions
Markov Chains
1
16.4-1) Given the following one-step transition matrices of a Markov chain,
determine the classes of the Markov chain and whether they are
recurrent.
a.
1 2
3 3
P 1 0 0 0
0 1 0 0
0 0
0 1 0
0
1
3
2
0
All states communicate, all states recurrent
b.
1 0
0
1
2
1
2
1
2
1
2
1
0
2
0
0
P
0
0
0
0
0
1
2
Communicate {0}, {1, 2}, {3}
states 0, 1, 2 recurrent
state 3 transient
1
2
3
Solutions
Markov Chains
2
16.4-3) Given the following one-step transition matrix of a Markov chain,
determine the classes of the Markov chain and whether they are
recurrent.
1
4
3
4
1
P
3
3
4
1
4
1
3
1
3
0 0
0
0 0
0
0
0 0
0
0 0
0 0
3
4
1
4
0
1
2
3
1
4
3
4
Drawing the transition diagram above, we see that states 1 and 2 communicate. Once
you leave state 2, there is a finite probability you will never return. Therefore, state 2 is
transient. States 3 and 4 communicate with each other but not with states 0, 1, or 2.
Consequently,
{ 0, 1 } is recurrent
{ 2 } is transient
{ 3, 4 } is recurrent
4
Solutions
Markov Chains
3
16.5-4) The leading brewery on the West Coast (A) has hired a TM specialist
to analyze its market position. It is particularly concerned about its
major competitor (B). The analyst believes that brand switching can
be modeled as a Markov chain using 3 states, with states A and B
representing customers drinking beer produced from the
aforementioned breweries and state C representing all other brands.
Data are taken monthly, and the analyst has constructed the following
one-step transition probability matrix.
A
B
C
0.70 0.20 010
.
P  0.20 0.75 0.05
010
.
010
.
0.80
What are the steady-state market shares for the two major breweries?
Soln:
Let,
 A  0.7 A  0.20 B  01
. C
 B  0.2 A  0.75 B  01
. C
 C  01
.  A  0.05 B  0.8 C
 A 1
(1)
(2)
(3)
Then, from (1),
0.3  0.2 B  0.1 C
 C  3  2 B
(4)
Solutions
Markov Chains
4
16.5-4) (cont.)
 A  0.7 A  0.20 B  01
. C
 B  0.2 A  0.75 B  01
. C
 C  01
.  A  0.05 B  0.8 C
 C  3  2 B
from (2),
 B  .2(.1)  .75 B  .1(3  2 B )
.25 B  .2  .3  .2 B
.45 B  .5
 B  1111
.
from (4)
 C  3  2 (1111
. )
 0.777
(1)
(2)
(3)
(4)
Solutions
Markov Chains
16.5-4) (cont.)
 A  1000
.
 B  1111
.
 C  0.777
  2.888
But,
 A   B   C 1
.
 A  1000
2.888  0.346
.
 B  1111
2.888  0.385
 C  0.777 2.888  0.269
Note: This could also be checked by Pn for some large n.
5
Solutions
Markov Chains
6
16.6-1) A computer is inspected at the end of every hour. It is found to be
either working (up) or failed (down). If the computer is found to be
up, the probability of its remaining up for the next hour is 0.90. It it is
down, the computer is repaired, which may require more than one
hour. Whenever, the computer is down (regardlewss of how long it
has been down), the probability of its still being down 1 hour later is
0.35.
a. Construct the one-step transition probability matrix.
b. Find the expected first passage time from i to j for all i, j.
Soln:
Let,
S = 0 if computer is down
= 1 if computer is up
Then,
P
0.35 0.65
010
.
0.90
b. Find expected first passage times
 01  1  P00  01
 01 (1  P01 )  1
 01 
1
1

 154
.
1  P00 1  .35
Solutions
Markov Chains
7
16.6-1) (cont.)
b. Find expected first passage times
P
0.35 0.65
010
.
0.90
 01  1  P00  01
 01 (1  P01 )  1
 01 
1
1

 154
.
1  P00 1  .35
 00  1  P01 10
 1  .65(10)
 7.5
 10  1  P11 10
 10 (1  P11 )  1
 10 
1
1

 10.0
1  P11 1  .9
 11  1  P10  01
 1  .1(154
. )
 1154
.
Solutions
Markov Chains
8
16.6-1) (cont.) Alternative solution to b.
P
0.35 0.65
010
.
0.90
 0  .35  0  .1 1
 1  .65  0  .9  1
1  0   1
 1  1  0
Substituting (4) into (1) gives
.1 1  .65  0
.1(1   0 )  .65  0
 0  0.13
And from (4),
 1  0.87
(1)
(2)
(3)
(4)
Solutions
Markov Chains
16.6-1) (cont.) Alternative solution to b.
P
0.35 0.65
010
.
0.90
 0  013
.
 1  0.87
 00 
 11 
1
0
1
1

1
 7.5
.13

1
 1154
.
.87
9
Solutions
Markov Chains
10
16.6-2) A manufacturer has a machine that, when operational at the beginning
of a day, has a probability of 0.1 of breaking down sometime during
the day. When this happens, the repair is done the next day and
completed at the end of that day.
a. Formulate the evolution of the status of the machine as a 3 state
Markov Chain.
b. Fine the expected first passage times from i to j.
c. Suppose the machine has gone 20 full days without a breakdown
since the last repair was completed. How many days do we
expect until the next breakdown/repair?
Soln:
a. Let,
S = 0 if machine running at day’s end | running at start
= 1 if machine down at day’s end | running at start
= 2 if machine running at day’s end | down at start
P00  P{ X 1  0 | X 0  0}
 P{ run at start and end day 1| run at start and end day 0}
 0.9
P10  P{ X 1  0 | X 0  1}
 P{ run at start and end day 1| run at start and down at end day 0 }
 not possible  0.0
Solutions
Markov Chains
11
16.6-2) (cont.)
a. S = 0 if machine running at day’s end | running at start
= 1 if machine down at day’s end | running at start
= 2 if machine running at day’s end | down at start
Continuing in this fashion gives the one-step transition prob.
0.9 01
. 0.0
P  0.0 0.0 10
.
0.9 01
. 0.0
b.
 ij  1   Pik  kj
k j
 01  1  P00  01  P02  21
 01 
0
1
1

 10
1  P00 1  .1
Note: This makes intuitive sense. If the machine has a 10% chance
of failing on any given day, then the expected number of days between
failures is 10, (01 = 10).
Solutions
Markov Chains
12
16.6-2) (cont.)
0.9 01
. 0.0
P  0.0 0.0 10
.
0.9 01
. 0.0
 02  1  P00  02  P01 12
.1 02  1  .1 12
 01  1  P00  01  P02  21
 01 
1
1

 10
1  P00 1  .1
0
 00  1  P01 10  P02  20
 1  .1(10)
 10  1  P11 10  P12  20
 1   20
 00  2
0
0
 12  1  P10  02  P11 12
1
 11  1  P10  01  P12  21
 1   21
 21  1  P20  01  P22  21
 1  .9  01
 1  .9(10)  10
Solutions
Markov Chains
16.6-2) (cont.)
0.9 01
. 0.0
P  0.0 0.0 10
.
0.9 01
. 0.0
 20  1  P21 10  P22  20
 1  .1 10
 1  .1(1   20 )
 20  1.222
 22  1  P20  02  P21 12
 1  .9(11)  .1(1)
 11
Back substituting for 02, 10, 11
 02  11.0
 10  2.22
 11  11.0
13
Solutions
Markov Chains
14
16.6-2) (cont.)
 00  2.0
 10  2.22
 20  122
.
 01  10.0
 11  110
.
 21  10.0
 02  110
.
 12  10
.
 22  110
.
c. Suppose the machine has gone 20 full days without a breakdown
since the last repair was completed. How many days do we
expect until the next breakdown/repair?
If we read this as the expected number of days to breakdown since
the last repair, we are asking for 21 in which case,
 21  10.0
If we read this as the expected number of days to breakdown
and subsequent repair since the last repair, we are asking for 22.
 22  110
.
Again, this should make intuitive sense. A machine has a 10% chance
of breaking down. Therefore, the expected time between failures is
10 days. Since it takes 1 day to repair, the time from repair to repair is
10+1 = 11 days.