LECTURE 17
Review
Markov Processes – II
• Discrete state, discrete time, time-homogeneous
• Readings: Section 7.3
– Transition probabilities pij
– Markov property
Lecture outline
• rij (n) = P(Xn = j | X0 = i)
• Review
• Key recursion:
!
rij (n) =
rik (n − 1)pkj
• Steady-State behavior
k
– Steady-state convergence theorem
– Balance equations
• Birth-death processes
Warmup
9
3
Periodic states
6
5
1
• The states in a recurrent class are
periodic if they can be grouped into
d > 1 groups so that all transitions from
one group lead to the next group
4
2
7
8
P(X1 = 2, X2 = 6, X3 = 7 | X0 = 1) =
P(X4 = 7 | X0 = 2) =
Recurrent and transient states
• State i is recurrent if:
starting from i,
and from wherever you can go,
there is a way of returning to i
9
5
4
8
6
3
• If not recurrent, called transient
• Recurrent class:
collection of recurrent states that
“communicate” to each other
and to no other state
2
1
1
7
Expected Frequency of a Particular Transition
Steady-State Probabilities
• Do the rij (n) converge to some πj ?
(independent of the initial state i)
Consider n transitions of a Markov chain with a single class which is aperiodic, starting from a given initial state. Let qjk (n) be the expected number
of such transitions that take the state from j to k. Then, regardless of the
Visit frequency interpretation
initial state, we have
qjk (n)
= πj pjk .
lim
n→∞
n!
πj =
πk pkj
k
• Yes, if:
– recurrent states are all in a single class,
and
Given the frequency interpretation of πj and πk pkj , the balance equation
m
! of being in j: πj
• (Long run) frequency
πj =
– single recurrent class is not periodic
• Assuming “yes,” start from key recursion
!
k
rik (n − 1)pkj
• Frequency
of transitions
j: the
πk pexpected
kj
has an intuitive meaning.
It expresses
the factk →
that
frequency π
of visits to j is equal to the sum of the expected frequencies πk pkj of transition
!
that lead to j; see• Fig.
7.13.
Frequency of transitions into j:
π p
k kj
k
1
– take the limit as n → ∞
πj =
!
π j pjj
for all j
πk pkj ,
2
k
π 2p2j
j
. . .
– Additional equation:
!
π 1 p1j
. . .
rij (n) =
πk pkj
k=1
πj = 1
j
π m pmj
m
Figure 7.13: Interpretation of the balance equations in terms of frequencies. In
a very large number of transitions, we expect a fraction πk pkj that bring the state
from k to j. (This also applies to transitions from j to itself, which occur with
frequency πj pjj .) The sum of the expected frequencies of such transitions is the
expected frequency πj of being at state j.
Example
0.5
0.8
0.5
Birth-death processes
† In fact, some stronger statements are also true, such as the following. Wheneve
1experiment
- p1- q1
1 - q m of the Markov chai
1- p0
we carry out a probabilistic
and generate a trajectory
over an infinite time horizon, the observed long-term frequency with which state j
p
p
visited will be exactly equal0 to πj , 1and the observed long-term frequency of transition
3
m
1
0
from j to k will be exactly equal to πj p2 jk . Even
though the trajectory
is random, thes
q
q 1 certainty,
q2
m
equalities hold with essential
that is, with probability
1.
.. .
2
1
pi
0.2
i+1
i
πipi = πi+1qi+1
q i+1
• Special case: pi = p and qi = q for all i
ρ = p/q =load factor
p
πi+1 = πi = πiρ
q
π i = π 0 ρi ,
i = 0, 1, . . . , m
• Assume p < q and m ≈ ∞
π0 = 1 − ρ
E[Xn] =
2
ρ
1−ρ
(in steady-state)
MIT OpenCourseWare
http://ocw.mit.edu
6.041SC Probabilistic Systems Analysis and Applied Probability
Fall 2013
For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.