Regular Markov
Chains — steadystate probability
distributions
COPYRIGHT © 2006 by LAVON B. PAGE
Example
!.8 .2$
&
T= #
#".4 .6&%
!.72 .28$
&
T = #
#".56 .44&%
2
COPYRIGHT © 2006 by LAVON B. PAGE
Example
!.8 .2$
&
T= #
#".4 .6&%
!.668 .332$
&
T = #
#".664 .336&%
6
!.72 .28$
&
T = #
#".56 .44&%
2
"2 / 3 1 / 3%
'
T !$
$#2 / 3 1 / 3'&
6
COPYRIGHT © 2006 by LAVON B. PAGE
Example
!.8 .2$
&
T= #
#".4 .6&%
!.668 .332$
&
T = #
#".664 .336&%
6
!.72 .28$
&
T = #
#".56 .44&%
2
"2 / 3 1 / 3%
'
T !$
$#2 / 3 1 / 3'&
6
n
As n gets larger and larger, T gets
closer and closer to the matrix
!2 / 3 1 / 3$
#
&
#"2 / 3 1 / 3&%
COPYRIGHT © 2006 by LAVON B. PAGE
Notice
!2 / 3 1 / 3$
& = 2/3 1/3
1 0#
#"2 / 3 1 / 3&%
[
]
[
]
COPYRIGHT © 2006 by LAVON B. PAGE
Notice
!2 / 3
1 0#
#"2 / 3
!2 / 3
0 1#
#"2 / 3
[
[
]
1 / 3$
& = 2/3 1/3
1 / 3&%
]
1 / 3$
& = 2/3 1/3
1 / 3&%
[
[
]
]
COPYRIGHT © 2006 by LAVON B. PAGE
!2 / 3
1 0#
#"2 / 3
!2 / 3
0 1#
#"2 / 3
!2 / 3
1/2 1/2 #
#"2 / 3
Notice
[
[
[
]
1 / 3$
& = 2/3 1/3
1 / 3&%
]
1 / 3$
& = 2/3 1/3
1 / 3&%
]
1 / 3$
& = 2/3 1/3
1 / 3&%
[
[
[
]
]
]
COPYRIGHT © 2006 by LAVON B. PAGE
!2 / 3
1 0#
#"2 / 3
!2 / 3
0 1#
#"2 / 3
!2 / 3
1/2 1/2 #
#"2 / 3
!2 / 3
a b#
#"2 / 3
Notice
[
[
[
[
]
1 / 3$
& = 2/3 1/3
1 / 3&%
]
1 / 3$
& = 2/3 1/3
1 / 3&%
]
1 / 3$
& = 2/3 1/3
1 / 3&%
]
1 / 3$
& = 2/3 1/3
1 / 3&%
[
]
[
]
[
[
]
]
no matter what a and b are.
COPYRIGHT © 2006 by LAVON B. PAGE
Remember T
n
!2 / 3 1 / 3$
&
!#
#"2 / 3 1 / 3&%
when n is large
Moral of the story: No matter what
assumptions you make about the initial
probability distribution, after a large
number of steps have been taken the
probability distribution is approximately
(2/3 1/3)
COPYRIGHT © 2006 by LAVON B. PAGE
Moral of the story: No matter what
assumptions you make about the initial
probability distribution, after a large
number of steps have been taken the
probability distribution is approximately
(2/3 1/3)
Question: How could we determine
this without computing large powers
of T and estimating the limiting
matrix?
COPYRIGHT © 2006 by LAVON B. PAGE
Clue: Notice that the probability
distribution (2/3 1/3) has the property
that
!.8 .2$
& = 2/3 1/3
2/3 1/3 #
#".4 .6&%
[
]
[
]
COPYRIGHT © 2006 by LAVON B. PAGE
Clue: Notice that the probability
distribution (2/3 1/3) has the property
that
!.8 .2$
& = 2/3 1/3
2/3 1/3 #
#".4 .6&%
[
]
[
]
In other words, a solution of the matrix
equation
!.8 .2$
&= x y
x y#
#".4 .6&%
[
]
[
]
is x = 2/3, y = 1/3
COPYRIGHT © 2006 by LAVON B. PAGE
Idea!!!
We can find the steady state probability
distribution [ (2/3 1/3) in this example] by
solving for x and y below:
!.8 .2$
&= x y
x y#
#".4 .6&%
[
]
[
]
(Remember also that x + y =1)
COPYRIGHT © 2006 by LAVON B. PAGE
We can find the steady state probability
distribution [ (2/3 1/3) in this example] by
solving for x and y below:
!.8 .2$
&= x y
x y#
#".4 .6&%
[
]
[
]
(Remember also that x + y =1)
.8x + .4y = x
.2x + .6y = y
x+ y =1
COPYRIGHT © 2006 by LAVON B. PAGE
We can find the steady state probability
distribution [ (2/3 1/3) in this example] by
solving for x and y below:
!.8 .2$
&= x y
x y#
#".4 .6&%
[
]
[
]
(Remember also that x + y =1)
.8x + .4y = x
.4y = .2x
.2x + .6y = y
x+ y =1
.2x = .4y
x = 2y
COPYRIGHT © 2006 by LAVON B. PAGE
We can find the steady state probability
distribution [ (2/3 1/3) in this example] by
solving for x and y below:
!.8 .2$
&= x y
x y#
#".4 .6&%
[
]
[
]
(Remember also that x + y =1)
.8x + .4y = x
.4y = .2x
.2x + .6y = y
x+ y =1
.2x = .4y
x = 2y
2y + y = 1
COPYRIGHT © 2006 by LAVON B. PAGE
We can find the steady state probability
distribution [ (2/3 1/3) in this example] by
solving for x and y below:
!.8 .2$
&= x y
x y#
#".4 .6&%
[
]
[
]
(Remember also that x + y =1)
.8x + .4y = x
.2x + .6y = y
x+ y =1
.4y = .2x
x = 2y
.2x = .4y
y = 1/3
2y + y = 1
x = 2/3
COPYRIGHT © 2006 by LAVON B. PAGE
Example: Bob, Alice and Carol are playing
Frisbee. Bob always throws to Alice and
Alice always throws to Carol. Carol throws
to Bob 2/3 of the time and to Alice 1/3 of
the time. In the long run what percentage
of the time do each of the players have the
Frisbee?
COPYRIGHT © 2006 by LAVON B. PAGE
Example: Bob, Alice and Carol are playing
Frisbee. Bob always throws to Alice and
Alice always throws to Carol. Carol throws
to Bob 2/3 of the time and to Alice 1/3 of
the time. In the long run what percentage
of the time do each of the players have the
Frisbee?
T=
A
B
A! 0
0
#
B# 1
0
C ##"1 / 3 2 / 3
C
1$
&
0&
&
0&%
COPYRIGHT © 2006 by LAVON B. PAGE
T=
A
B
A! 0
0
#
B# 1
0
C ##"1 / 3 2 / 3
C
1$
&
0&
&
0&%
We must solve the matrix equation
! 0
0
1$
#
&
x y z# 1
0
0& = x y z
#
&
#"1 / 3 2 / 3 0&%
[
]
[
]
and use the fact that x + y + z = 1
COPYRIGHT © 2006 by LAVON B. PAGE
! 0
0
1$
#
&
x y z# 1
0
0& = x y z
#
&
1
/
3
2
/
3
0
#"
&%
[
]
[
]
We multiply out the left side and equate
to the right side:
COPYRIGHT © 2006 by LAVON B. PAGE
! 0
0
1$
#
&
x y z# 1
0
0& = x y z
#
&
1
/
3
2
/
3
0
#"
&%
[
]
[
]
We multiply out the left side and equate
to the right side:
y + 1/3 z = x
2/3 z = y
x=z
COPYRIGHT © 2006 by LAVON B. PAGE
! 0
0
1$
#
&
x y z# 1
0
0& = x y z
#
&
1
/
3
2
/
3
0
#"
&%
[
]
[
]
We multiply out the left side and equate
to the right side:
y + 1/3 z = x
2/3 z = y
x=z
Remember also: x + y + z = 1
COPYRIGHT © 2006 by LAVON B. PAGE
Equations to solve:
x+y+z=1
y + 1/3 z = x
2/3 z = y
x=z
COPYRIGHT © 2006 by LAVON B. PAGE
Equations to solve:
x+y+z=1
y + 1/3 z = x
2/3 z = y
x=z
Solution:
x + y + z = 1 can be rewritten as
z + 2/3 z + z = 1
COPYRIGHT © 2006 by LAVON B. PAGE
Equations to solve:
x+y+z=1
y + 1/3 z = x
2/3 z = y
x=z
Solution:
x + y + z = 1 can be rewritten as
z + 2/3 z + z = 1
so z = 3/8
Therefore x = 3/8 and y = 1/4
COPYRIGHT © 2006 by LAVON B. PAGE
x = 3/8, y = 1/4, z = 3/8
Alice — 3/8 probability
Bob — 1/4 probability
Carol — 3/8 probability
COPYRIGHT © 2006 by LAVON B. PAGE
In the truck rental problem we had the
following transition matrix:
NC SC VA
NC !.5 .2 .3$
#
&
SC #.4 .4 .2&
#
&
VA #.4 .1 .5&
"
%
What fraction of the time does a truck
spend in each of the 3 states?
COPYRIGHT © 2006 by LAVON B. PAGE
We have to solve for
!.5 .2 .3$
#
&
[x y z]#.4 .4 .2& = [x y z]
#
&
#".4 .1 .5&%
remembering that x + y + z = 1.
COPYRIGHT © 2006 by LAVON B. PAGE
We have to solve for
!.5 .2 .3$
#
&
[x y z]#.4 .4 .2& = [x y z]
#
&
#".4 .1 .5&%
remembering that x + y + z = 1.
So the system of equations is
.5x + .4y + .4z = x
.2x + .4y + .1z = y
.3x + .2y + .5z = z
x + y + z =1
COPYRIGHT © 2006 by LAVON B. PAGE
.5x + .4y + .4z = x
.2x + .4y + .1z = y
.3x + .2y + .5z = z
x + y + z =1
Rewrite the equations in standard form:
x + y+ z=1
!.5x + .4y + .4z = 0
.2x + !.6y + .1z = 0
.3x + .2y + !.5z = 0
COPYRIGHT © 2006 by LAVON B. PAGE
.5x + .4y + .4z = x
.2x + .4y + .1z = y
.3x + .2y + .5z = z
x + y + z =1
Rewrite the equations in standard form:
x + y+ z=1
!.5x + .4y + .4z = 0
.2x + !.6y + .1z = 0
.3x + .2y + !.5z = 0
This system of
equations could be
solved using the
augmented matrix
method of Chapter 1.
COPYRIGHT © 2006 by LAVON B. PAGE
The methods we have learned in this
section work only with regular Markov
chains.
COPYRIGHT © 2006 by LAVON B. PAGE
The methods we have learned in this
section work only with regular Markov
chains.
A regular Markov Chain is one where for
n
some positive integer n, the matrix T
has no 0 entries.
COPYRIGHT © 2006 by LAVON B. PAGE
"$
$
$
T = $$
$
$#
T
3
0
2
3
1
2
0
1
4
1 %'
'
1'
3 ''
1'
'
4&
T
"$ 1
$ 6
$
$
= $ 13
$ 24
$
$ 43
$# 96
3
4
3
16
21
64
1 %'
12 ''
13 '
'
48 '
'
43 '
192'&
0
2
"$ 2
$ 3
$
$ 1
= $$ 6
$
$ 7
$# 24
0
3
4
9
16
1 %'
3 ''
1 ''
12'
'
7 '
48'&
COPYRIGHT © 2006 by LAVON B. PAGE
"$
$
$
T = $$
$
$#
T
3
0
2
3
1
2
0
1
4
1 %'
'
1'
3 ''
1'
'
4&
T
"$ 1
$ 6
$
$
= $ 13
$ 24
$
$ 43
$# 96
3
4
3
16
21
64
1 %'
12 ''
13 '
'
48 '
'
43 '
192'&
0
2
"$ 2
$ 3
$
$ 1
= $$ 6
$
$ 7
$# 24
0
3
4
9
16
1 %'
3 ''
1 ''
12'
'
7 '
48'&
T is regular
because T 3
contains no 0
entries.
COPYRIGHT © 2006 by LAVON B. PAGE