Section M.3
MATH 166:503
April 28, 2015
Topics from last notes: regular Markov processes, regular stochatic matrix, steady state
1
MARKOV CHAINS
1.2
Absorbing Markov Processes
In a Markov process, a state in which, once entered, it is impossible to leave is called absorbing.
A Markov process is called absorbing if
1. there is at least one absorbing state
2. it is possible to move from any non-absorbing state to one of the absorbing states in a finite
number of stages
The transition matrix for an absorbing Markov process is said to be an absorbing stochastic
matrix.
ex. A drunk is walking down the street towards his home. Somehow he ended up on a street lined
with police stations on either side of the road. If he wanders into any police station, he will get
booked for public intoxication and must stay the night at the station. While he’s swaying down
the road, he has a 50% chance of moving to the left or right. Draw a tree diagram that describes
this situation.
1
Write the transition matrix that corresponds to this situation.
ex. Is
0
1
0
.1
.8
.1
1!
0
0
a transition matrix for an absorbing Markov process?
ex. Is
.25
.75
0
1
a transition matrix for an absorbing Markov process?
2
What is the limiting matrix, L, in the case of the drunken walk?
In an absorbing Markov process, the long term probability of going from any non-absorbing state
to some absorbing state is 1.
Finding L for an absorbing Markov process: Let T be the transition matrix of an absorbing
Markov process with n absorbing states and m non-absorbing states. Reorder the rows and columns
so that
Inxn Anxm
T = 0mxn Bmxm
Then
A(Imxm − B)−1
0mxm
Inxn
T n → L = 0mxn
as n → ∞.
Use this technique in the drunken walk example.
3
F = (I − B)−1 captures interesting information:
1. the fi,j entry of F gives the expected number of times that the system will be in the ith
non-absorbing state if it is initially in the jth non-absorbing state.
2. the sum of the entries in the jth column of F is the expected number of stages before
absorption in initially in the jth non-absorbing state.
ex. A survey indicates that in a certain area, people either take their summer vacations at the
beach, lake, or the mountains. Of the people who went to the beach one summer, 15% will return
to the beach next summer and 50% will visit the lake. Of the people who went to the lake, 20% will
return and 45% will visit the beach. The mountains are so beautiful that once visits the mountains,
they will always return next summer. Last year, the lake was contaminated so no one visited. Half
of the people went to the beach and half went to the mountains. Find the limiting matrix L for
this example.
4