2.3 Steady State Probabilities
2.3.1 Finding Steady State Probabilities
In section 2.2.1 we saw how to compute the powers Pn of the transition matrix P. We saw that each element
of Pn was a constant plus a sum of multiples of powers of numbers i whose absolute value is no greater
than one. In fact, in most cases the i have absolute value strictly less than one, so their powers go to zero
as n  . So Pn approaches a limiting matrix, which we shall denote by P, as n tends to . The rows of
this limiting matrix contain the probabilities of being in the various states as time gets large. These
probabilities are called steady state probabilities. Let's look at the examples of the previous section.
Example 1. If it is sunny one day there is a 90% chance that it will be sunny the next day and a 10%
chance that it will be cloudy. If it cloudy on a particular day, there is a 60% chance that it will be sunny the
next day and a 40% chance that it will be cloudy. We model this by a Markov chain with two states with
state 1 corresponding to being sunny on a particular day and state 2 to being cloudy. We let n count the
days with n = 0 being today so that Xn is the state of day n, either sunny or cloudy. The transition matrix is
(1)
P =
 0.9 0.1 
 0.6 0.4 
In the previous section we saw that
 7 + 7(0.3)
 6 - 6(0.3)
7 7
6
Pn =
1
n
n



1 1
- (0.3)n
7 7
1 6
+ (0.3)n
7 7
As n   one has
 7 + 7(0.3)
 6 - 6(0.3)
7 7
6
Pn =
1
n
n
1 1
- (0.3)n
7 7
1 6
+ (0.3)n
7 7



7
6
7
6
 P =
1
7
1
7



since (0.3)n  0. In fact within a week (7 days) one has Pn quite close to P since (0.3)7  0.0002. So for a
day more than a week in the future one has
Pr{sunny} 
Pr{cloudy} 
6
7
1
7
no matter whether it is sunny or cloudy today. In the long run it is sunny six days out of seven. These
probabilities
2.3.1 - 1
1 =
2 =
lim Pr{ Xn = 1 | X0 = 1} =
n
lim Pr{ Xn = 2 | X0 = 1} =
n
lim Pr{ Xn = 1 | X0 = 2} =
6
7
lim Pr{ Xn = 2 | X0 = 2} =
1
7
n
n
are called the steady state probabilities of being in states 1 and 2. In many applications of Markov chains
the steady state probabilities are the main items of interest since one is interested in the long run behavior of
the system. For the time being we shall assume the following about the transition matrix P.
(1)
The eigenvalue one is not repeated.
(2)
The other eigenvalues have absolute value strictly less than one.
Then
Pn = TDnT -1
0
 
0
 1 t12
1
t22
=  

 1 tn2
t1n
t2n
1
0
(2)n
tnn
0

T
( ) 
0
0
n
-1
n
In general each entry of Pn is a constant plus a sum of multiples of powers of numbers of absolute value less
than one. As n approaches infinity Pn approaches the matrix P of the constants. Thus as n   we get
P
 1 t12
1
t22
=  

 1 tn2
=
0
1
1
0

 
1 0
t1n  1
t2n  0
 
tnn  0
 
 
0  
0
0
0
0
0
0
0

0
T -1
21
2
22
n
2n
n1
n2
nn
1



0
1
1
0

=
 
1 0
 



0
T -1
1
2
2
n
n
1
2
n
1
=
0
0



where we let
 


21
2
22
n
2n
n1
n2
nn
 


1
2
2
n
n
1
2
n
1
T -1 =



So
1
P =



So the rows of P are all equal to the vector which is the first row of T-1. If we take the equation P = TDT -1
and multiply both sides on the left by T-1 we get T -1P = DT -1. The ith row of T -1P is the ith row of T-1 times
2.3.1 - 2
P. The ith row of DT -1 is i times the ith row of T-1. Thus the ith row of T -1P is i times the ith row of T-1. So
the ith row of T -1 is a left eigenvector of P with eigenvalue i. In particular, the first row of T -1 is a left
eigenvector of P with eigenvalue 1. In other words
(3)
P = 
where
 = (1, 2, …, n)
So the rows of P are vectors  which are solutions of the equation P =  and which are probability
vectors, i.e. the sum of the entries of  equals 1. The entries of  are called steady state probabilities.
Another way to see that the rows of P satisfy (7) is to start with the relationship Pn+1 = PnP and let n  .
Since Pn  P and Pn+1  P one gets P = PP. Since the ith row of PP is the ith row of P times P we
see the rows of P satisfy (3).
If the eigenvalue one is not repeated then there is only one such vector  satisfying (3). In this case this is
enough information to find it without computing Pn and taking the limit as n  .
Example 1 (continued). Find the steady state vector  = (1, 2) in Example 1.
Since P =  we get
0.9 0.1
(1, 2) 0.6 0.4  = (1, 2)
Therefore
0.91 + 0.62 = 1
0.11 + 0.42 = 2
Therefore
- 0.11 + 0.62 = 0
0.11 - 0.62 = 0
Both of these equations are equivalent to 1 = 62. In order to find 1 and 2 we need to use the fact that
1 + 2 = 1. Combining this with 1 = 62 gives 62 + 2 = 1 or 72 = 1 or 2 = 1/7 and 1 = 6/7.
Example 2. (see Problem 1 of section 2.1.1) At the start of each day an office copier is checked and its
condition is classified as either good (1), poor (2) or broken (3). Suppose the transition matrix from one
state to another in the course of day is
2.3.1 - 3
P =
 0.8
 0
 0.32
0.06
0.4
0.08
0.14 
0.6 
0.6 
In the previous section we saw that
0.55 + (0.52)(0.51) - (0.078)(0.28)
P =  0.55 - (1.74)(0.51)n + (1.19)(0.28)n
 0.55 - (0.33)(0.51)n - (0.22)(0.28)n
n
n
n
0.1 - (0.05)(0.51)n - (0.05)(0.28)n 0.35 - (0.47)(0.51)n + (0.12)(0.28)n

0.1 + (0.18)(0.51)n + (0.72)(0.28)n 0.35 + (1.56)(0.51)n - (1.91)(0.28)n 
n
n
n
n
0.1 + (0.03)(0.51) - (0.14)(0.28) 0.35 + (0.29)(0.51) + (0.36)(0.28) 
As n   one has
Pn  P =
0.55
0.55
0.55
0.1 0.35

0.1 0.35
0.1 0.35
So if n is bigger than 10 or so
Pr{ the copier is in good condition }  0.55
Pr{ the copier is in poor condition }  0.10
Pr{ the copier is broken }  0.35
So the vector of steady state probabilities is  = (0.55, 0.10, 0,35). As in the previous example, this vector
can be found by solving P =  along with the fact that its components sum to one. The equation P =  is
 0.8
(1, 2, 3) 0
 0.32
0.06
0.4
0.08
0.14 
0.6  = (1, 2, 3)
0.6 
Therefore
0.81
+ 0.323 = 1
0.061 + 0.42 + 0.83 = 2
0.141 + 0.62 + 0.63 = 3
Therefore
- 0.21
+ 0.323 = 0
0.061 - 0.62 + 0.083 = 0
0.141 + 0.62 -
0.43 = 0
The third equation is just the negative of the sum of the first and the second, so we can solve the first two
alone. Multiplying them by 100 gives
2.3.1 - 4
- 201
+ 323 = 0
61 - 602 + 83 = 0
Solving the first equation for 1 gives 1 =
8
 . Substituting into the second equation we get
5 3
48
 - 602 + 83 = 0
5 1
22
8
22
 . Since 1 + 2 + 3 = 1 we have 5 3 + 75 3 + 3 = 1. Solving for 3 gives
75 3
75
120
22
3 = 217. So 1 = 217 and 2 = 217. Thus
Solving for 2 gives 2 =
120 22
75
 = (1, 2, 3) = ( 217, 217, 217 )  (0.55, 0.10, 0.35)
Problem 1. Let P =
1 - p p . Show that the steady state vector is  = ( q , p )
 q 1 - q
p+q p+q
2.3.1 - 5