Markov Chains
Discrete-Time Markov Chains
A discrete-time Markov chain is a discrete-time , discrete-value
random sequence such that the next random variable X ⎡⎣ n + 1⎤⎦
depends only on X ⎡⎣ n ⎤⎦ through the transition probability
(
)
= P ( X ⎡⎣ n + 1⎤⎦ = j | X ⎡⎣ n ⎤⎦ = i,X ⎡⎣ n − 1⎤⎦ = i
Pij = P X ⎡⎣ n + 1⎤⎦ = j | X ⎡⎣ n ⎤⎦ = i
n−1
,,X ⎡⎣0 ⎤⎦ = i0
X ⎡⎣ n ⎤⎦ is called the state of the system which produces the Markov
chain and the sample space of X ⎡⎣ n ⎤⎦ is called the state space. The
transition probabilities of a Markov chain satisfy Pij ≥ 0 ,
∞
∑P
ij
j=0
The sum of all the probabilities of going from state i to any of
the other states in the state space is one.
= 1.
)
State Diagrams
Transition Probability
Directed Arc
Directed Arc
Node
Node
State 0
Transition Probability
State 1
Directed Arc
State-Transition Matrix
A Markov chain can have a finite number of states or a countable
infinity of states. In a system with a state space {0,1, 2,L , N }
there are ( N + 1) transition probabilities and they can be represented
2
by a state - transition matrix of the form
⎡ P00 P01 L P0 N ⎤
⎢P
⎥
P
L
P
11
1N ⎥
P = ⎢ 10
M⎥
⎢ M M O
⎢
⎥
P
P
L
P
N1
NN ⎦
⎣ N0
The elements in each row must sum to one.
Two-State Example
Find the probability that if the initial state is 0, the state after
two steps is 1.
There are two ways this can happen 0 → 0 → 1 and 0 → 1 → 1.
P ( 0 → 0 → 1) = P00 P01 = 0.6 × 0.4 = 0.24
P ( 0 → 1 → 1) = P01 P11 = 0.4 × 0.3 = 0.12
Two-State Example
The two transition sequences are mutually exclusive so the
probability of transitioning from 0 to 1 in two steps is the sum
of their probabilities.
P ( 0 to 1 in two steps ) = P00 P01 + P01 P11 = 0.24 + 0.12 = 0.36
The state-transition matrix for this Markov chain is
⎡ 0.6 0.4 ⎤
P=⎢
⎥
0.7
0.3
⎣
⎦
Notice that if we square this matrix we get
⎡0.64 0.36 ⎤
P =⎢
⎥
0.6
3
0.37
⎣
⎦
2
Two-State Example
We can see why by looking at the details of the matrix-squaring
process.
P ( 0 → 0 in 2 steps ) P ( 0 → 1 in 2 steps )
⎡ P00
P =⎢
⎣ P10
2
P01 ⎤ ⎡ P00
P11 ⎥⎦ ⎢⎣ P10
P01 ⎤ ⎡ P00 P00 + P01 P10
=⎢
⎥
P11 ⎦ ⎣ P10 P00 + P11 P10
P00 P01 + P01 P11 ⎤
P10 P01 + P11 P11 ⎥⎦
P (1 → 0 in 2 steps ) P (1 → 1 in 2 steps )
Discrete-Time Markov Chain
Dynamics
In a Markov chain with a state space {0,1, 2,L , N } the statetransition matrix for n steps is P [ n ] = P n . It then follows that
P [n + m] = P [n] P [m]
Discrete-Time Markov Chain
Dynamics
Example:
Given this Markov chain find the state-transition matrix for 3 steps.
⎡0.2 0.8 0 ⎤
P = ⎢0.2 0.3 0.5⎥
⎢
⎥
⎢⎣0.4 0.6 0 ⎥⎦
3
0.2 ⎤
⎡0.2 0.8 0 ⎤ ⎡ 0.28 0.52
P [3] = P 3 = ⎢0.2 0.3 0.5⎥ = ⎢ 0.23 0.495 0.275⎥
⎢
⎥ ⎢
⎥
⎢⎣0.4 0.6 0 ⎥⎦ ⎢⎣0.26 0.49 0.25 ⎥⎦
Discrete-Time Markov Chain
Dynamics
Raising a matrix to a power can be done most efficiently using
eigenvalues and eigenvectors. A square matrix A can be
diagonalized into A = SΛS −1 where S is a matrix consisting of
the eigenvectors of A as its columns and Λ is a diagonal matrix
whose non-zero elements are the eigenvalues of A. Then the
nth power of A is A n = SΛ nS −1. Raising Λ to the nth power is
much simpler than raising A to the nth power because it is diagonal.
Its nth power is the diagonal matrix in which each eigenvalue is
individually raised to the nth power.
Discrete-Time Markov Chain
Dynamics
Consider a the two-state Markov chain with probability p for
the 0-1 transition and probability q for the 1-0 transition. The
state-transition matrix is
p ⎤
⎡1 − p
P=⎢
⎥
q
1
−
q
⎣
⎦
The eigenvalues are found by solving
det ( P − λ I ) = 0 ⇒ (1 − p − λ )(1 − q − λ ) − pq = 0
The two solutions are λ1,2 = 1,1 − ( p + q ) .
Discrete-Time Markov Chain
Dynamics
The eigenvectors are found by solving
Pxi = λi xi ⇒ ( P − λi I ) xi = 0.
⎛ ⎡1 − p
p ⎤
⎡ 1 0 ⎤ ⎞ ⎡ x1 ⎤
−λ ⎢
⎜⎢
⎟⎢ ⎥ = 0
⎥
⎥
1− q⎦
⎣0 1 ⎦ ⎠ ⎣ x 2 ⎦
⎝⎣ q
From the top equation,
p + λ −1
1
−
p
−
λ
x
+
p
x
=
0
⇒
x
=
x1
(
) 1
2
2
p
Discrete-Time Markov Chain
Dynamics
For the two eigenvalues λ1,2 = 1,1 − ( p + q ) we get the two
eigenvectors,
⎡1⎤
⎡ 1 ⎤
x1 = ⎢ ⎥ and x 2 = ⎢
⎥
1
−
q
/
p
⎣⎦
⎣
⎦
Then
1 ⎤ ⎡λ
⎡1
P = SΛ S = ⎢
⎥⎢ 0
1
−
q
/
p
⎣
⎦⎣
n
n
−1
n
1
1 ⎤
0 ⎤ ⎡1
⎥
λ2n ⎦ ⎢⎣1 −q / p ⎥⎦
−1
Discrete-Time Markov Chain
Dynamics
Multiplying out the matrices,
n
n
n
n
⎡
q
λ
+
p
λ
λ
−
λ
p⎤
(
)
1
2
1
2
1
⎢
⎥
Pn =
p + q ⎢( λ1n − λ2n ) q pλ1n + qλ2n ⎥
⎣
⎦
Since one eigenvalue is one (which will always be true for a
state-transition matrix),
n
⎡
q
+
p
λ
2
1
n
⎢
P =
p + q ⎢(1 − λ2n ) q
⎣
n
1
−
λ
( 2 ) p ⎤⎥
p + qλ2n ⎥⎦
Discrete-Time Markov Chain
Dynamics
n
n
⎡
q
+
p
λ
1
−
λ
p⎤
(
)
2
2
1
⎢
⎥
Pn =
n
n
p + q ⎢(1 − λ2 ) q p + qλ2 ⎥
⎣
⎦
If the second eigenvalue has a magnitude less than one, as n
approaches infinity, the elements in P n approach finite limits.
The second eigenvalue is 1 − ( p + q ) . If 0 < p < 1 or 0 < q < 1,
then this second eigenvalue is less than one in magnitude. If
p = q = 1, the second eigenvalue is − 1 and
n
n
⎡
1
+
−
1
1
−
−
1
( )
( )⎤
1
n
⎥
P = ⎢
n
n
2 ⎢1 − ( −1) 1 + ( −1) ⎥
⎣
⎦
The elements of P n oscillate between 1 and − 1.
Discrete-Time Markov Chain
Dynamics
n
⎡
q
+
p
λ
2
1
n
⎢
P =
p + q ⎢(1 − λ2n ) q
⎣
n
1
−
λ
( 2 ) p ⎤⎥
p + qλ2n ⎥⎦
If p = q = 0, the second eigenvalue is +1 and
⎡1 0 ⎤
P =⎢
⎥
0
1
⎣
⎦
n
and neither state ever transitions to the other state.
Discrete-Time Markov Chain
Dynamics
A state probability vector p [ n ] is a column vector of probabilities
that the Markov chain is in each allowable state at time n. Given a
starting probability vector p [0] it is possible to compute p [ n ]
using P [ n ].
pT [ n ] = pT [ 0 ] P n
More generally, pT [ n + m ] = pT [ n ] P m .
Discrete-Time Markov Chain
Dynamics
For this Markov chain find the state-probability
vector p [ n ] for times, n = 1, 2,10,100,1000
given that the initial probability vector is
p [0] = [0.3 0.5 0.2] .
T
pT [1] = pT [0] P = [0.24 0.51 0.25]
pT [ 2] = pT [0] P 2 = [0.25 0.495 0.255]
pT [10] = pT [0] P10 = [0.25 0.5 0.25]
pT [100] = pT [0] P100 = [0.25 0.5 0.25]
pT [1000] = pT [0] P1000 = [0.25 0.5 0.25]
Discrete-Time Markov Chain
Dynamics
Graph the state-probability vector vs time for three different
starting probability vectors pT [0] = [1 0 0] , pT [0] = [0 1 0]
and pT [0] = [0 0 1] over the
time range, 0 ≤ n ≤ 10.
Limiting State Probabilities for a
Finite Markov Chain
For a finite Markov chain with an initial state-probability vector
p [0] the limiting state probabilities, if they exist, are the elements of
the vector π = lim p [ n ]. There are three possible cases.
n →∞
1. The limit exists and is independent of the initial state-probability
vector,
2. The limit exists but it depends on the initial state-probability vector,
3. The limit does not exist.
If a finite Markov chain with state-transition matrix P and initial
state-probability vector p [0 ] has a limiting state-probability vector
π = lim p [ n ] then π Τ = π Τ P and π is said to be stationary.
n →∞
Limiting State Probabilities for a
Finite Markov Chain
If a finite Markov chain with a state-transition matrix P is
initialized with a stationary probability vector p [0] = π then
p [ n ] = π for all n and the stochastic process X [ n ] is stationary.
If, in a Markov chain, the state probabilities are stationary, it is
said to be in steady - state.
Limiting State Probabilities for a
Finite Markov Chain
Consider again the two-state example.
n
⎡
q
+
p
λ
2
1
n
⎢
P =
p + q ⎢(1 − λ2n ) q
⎣
n
1
−
λ
( 2 ) p ⎤⎥
p + qλ2n ⎥⎦
Case 1. 0 < p + q < 2 In this case λ2 < 1 and
n
⎡
q
+
p
λ
2
1
n
⎢
lim P = lim
n →∞
n →∞ p + q ⎢
(⎣ 1 − λ2n ) q
(1 − λ ) p ⎤⎥
n
2
1 ⎡q
=
⎢q
n
p
+
q
p + qλ2 ⎥⎦
⎣
p⎤
p ⎥⎦
Limiting State Probabilities for a
Finite Markov Chain
Case 2.
p=q=0
In this case λ2 = 1 and
1 ⎡q + p
P =
p + q ⎢⎣ 0
n
Case 3. p = q = 1
0 ⎤ ⎡1 0 ⎤
=⎢
⎥
p + q ⎦ ⎣0 1 ⎥⎦
In this case λ2 = −1 and
n
n
⎡
1
+
−
1
1
−
−
1
( )
( )⎤
1
n
⎥
P = ⎢
n
n
2 ⎢1 − ( −1) 1 + ( −1) ⎥
⎣
⎦
Limiting State Probabilities for a
Finite Markov Chain
State probabilities
approach limit
independent of
initial state
State probabilities
approach limit
dependent on
initial state
State probabilities
do not approach
a limit
State Classification
State j is accessible from state i (indicated by the notation,
i → j) if Pij ⎡⎣ n ⎤⎦ > 0 for any n > 0. When state j is not
accessible from state i, that is indicated by i → j.
States i and j communicate (indicated by the notation,
i ↔ j) if i → j and j → i.
A communicating class is a nonempty subset of states
C such that if i ∈C then j ∈C if and only if i ↔ j.
State i has a period d if d is the largest integer such that
Pii ⎡⎣ n ⎤⎦ = 0 whenever n is not evenly divisible by d and if
d > 1. If d = 1 then state i is aperiodic.
State Classification
In a finite Markov chain, a state i is transient if there exists a
state j such that i → j but j → i. If no such state j exists then
state i is recurrent. If i is transient, then N i , the number of visits
to state i over all time, has a finite expected value E ( N i ) < B
where B is a finite upper bound.
A Markov chain is irreducible if there is only one communicating
class.
State Classification
Are states 0 and 1 periodic? It can be shown that
⎧ p n / 2 , n even ⎫
P00 [ n ] = P11 [ n ] = ⎨
⎬
⎩0 , n odd ⎭
States 0 and 1 are both transient and periodic. A signal in an LTI
system cannot be both transient and periodic. State 2 is recurrent
and aperiodic.
State Classification
The communicating classes are C1 = {0,1, 2,3} and C2 = {4,5, 6}.
Class C1 is aperiodic. Class C2 is periodic with period d = 3.
States 0, 1, 2 and 3 are transient. States 4, 5 and 6 are recurrent.
This Markov chain is reducible.
State Classification
There is one communicating class C1 = {0,1, 2,3, 4}. Class C1
is periodic with period d = 2. All states are recurrent.
This Markov chain is irreducible.
State Classification
The communicating classes are C1 = {0,1, 2,3, 4} , C2 = {5} and
C3 = {6, 7,8,9}. Class C1 is periodic with period d = 2. The other
classes are aperiodic. States 0, 1, 2, 3, 4 and 5 are transient. States
6, 7, 8 and 9 are recurrent. This Markov chain is reducible.
Limit Theorems
For an irreducible, aperiodic, finite Markov chain with states
{0,1, 2,L
, N } the limiting n-step state-transition matrix is
⎡π 0 π 1
⎢π π
1
lim P n = 1πT = ⎢ 0
n →∞
⎢M M
⎢
⎣π 0 π 1
where 1 = [1 1 L
L
L
O
L
1] and π = [π 0 L
T
πN ⎤
πN ⎥
⎥
M⎥
π N ⎥⎦
π N ] is the unique
T
vector satisfying πT = πT P , πT 1 = 1. For an irreducible, aperiodic,
finite Markov chain with state-probability transition matrix P and
initial state-probability vector p [0 ] , lim p [ n ] = π.
n →∞
Limit Theorems
Example
Calculate the stationary state-probability vector π.
In steady state,
π 0 = 0.2π 0 + 0.2π 1 + 0.4π 2
π 1 = 0.8π 0 + 0.3π 1 + 0.6π 2
π 2 = 0.5π 1
⎡ 0.8 −0.2 −0.4 ⎤ ⎡π 0 ⎤ ⎡0 ⎤
⎢ −0.8 0.7 −0.6 ⎥ ⎢ π ⎥ = ⎢0 ⎥
⎢
⎥⎢ 1⎥ ⎢ ⎥
−0.5
1 ⎥⎦ ⎢⎣π 2 ⎥⎦ ⎢⎣0 ⎥⎦
⎢⎣ 0
These three equations are not linearly independent.
Limit Theorems
The other equation needed is π 0 + π 1 + π 2 = 1
Then we can write
⎡ 0.8 −0.2 −0.4 ⎤ ⎡π 0 ⎤ ⎡0 ⎤
⎢ −0.8 0.7 −0.6 ⎥ ⎢ π ⎥ = ⎢0 ⎥
⎢
⎥⎢ 1⎥ ⎢ ⎥
1
1 ⎥⎦ ⎢⎣π 2 ⎥⎦ ⎢⎣1 ⎥⎦
⎢⎣ 1
The solution is πT = [0.25 0.5 0.25].
Limit Theorems
Alternatively we could use
⎡π 0 π 1
⎢π π
1
lim P n = 1πT = ⎢ 0
n →∞
⎢M M
⎢
⎣π 0 π 1
L
L
O
L
where
⎡0.2 0.8 0 ⎤
P = ⎢0.2 0.3 0.5⎥
⎢
⎥
⎢⎣0.4 0.6 0 ⎥⎦
πN ⎤
πN ⎥
⎥
M⎥
π N ⎥⎦
Limit Theorems
The diagonalized form of P n is
0.78
0.78
⎡0.58
⎤
P n = ⎢0.58 −0.44 + j 0.19 −0.44 + j 0.19 ⎥
⎢
⎥
0.1 + 0.38 ⎥⎦
⎢⎣0.58 0.1 − j 0.38
n
0
0
⎡1
⎤
⎥
× ⎢0 −0.25 + j 0.19
0
⎢
⎥
0
−0.25 − j 0.19 ⎥⎦
⎢⎣0
0.43
0.87
0.43
⎡
⎤
× ⎢ 0.48 − j 0.45 −0.32 − j 0.58 −0.16 + j1.03⎥
⎢
⎥
⎢⎣0.48 + j 0.45 −0.32 + j 0.58 −0.16 − j1.03⎥⎦
Two eigenvalues have a magnitude less than one.
Limit Theorems
Finding the limit as n → ∞,
0.78
0.78
⎡0.58
⎤
lim P n = ⎢0.58 −0.44 + j 0.19 −0.44 + j 0.19 ⎥
⎢
⎥
n →∞
0.1 + 0.38 ⎥⎦
⎢⎣0.58 0.1 − j 0.38
0.43
0.87
0.43
⎡1 0 0 ⎤ ⎡
⎤
× ⎢0 0 0 ⎥ ⎢ 0.48 − j 0.45 −0.32 − j 0.58 −0.16 + j1.03⎥
⎢
⎥⎢
⎥
⎢⎣0 0 0 ⎥⎦ ⎢⎣0.48 + j 0.45 −0.32 + j 0.58 −0.16 − j1.03⎥⎦
Multiplying out the matrices,
⎡0.25 0.5 0.25⎤
1πT = lim P n = ⎢0.25 0.5 0.25⎥ ⇒ πT = [0.25 0.5 0.25]
⎢
⎥
n →∞
⎢⎣0.25 0.5 0.25⎥⎦
Partitioning
It can be shown that for an irreducible, aperiodic, finite Markov
chain with state-transition matrix P and stationary probability
vector π partitioned into two disjoint state-space subsets S
and S ′ that
∑ ∑ π P = ∑ ∑π
i∈S j∈S ′
i ij
j∈S ′ i∈S
j
Pji .
Partitioning
Example
Router with buffer size c. Using
∑ ∑ π P = ∑ ∑π
i∈S j∈S ′
i ij
we can write
p
π i p = π i +1 (1 − p ) ⇒ π i +1 = π i
1− p
j∈S ′ i∈S
j
Pji
Partitioning
Generalizing,
⎛ p ⎞
p
p
π1 = π 0
, π 2 = π1
= π0 ⎜
⎟
1− p
1− p
1
−
p
⎝
⎠
2
⎛ p ⎞
, L , πi = π0 ⎜
⎟
1
−
p
⎝
⎠
The state probabilities must sum to one. Therefore,
1 − ( p / (1 − p ) )
⎛ p ⎞
⎛ p ⎞
πi = ∑π 0 ⎜
= π0∑⎜
= π0
=1
∑
⎟
⎟
1 − ( p / (1 − p ) )
i =0
i =0
i =0 ⎝ 1 − p ⎠
⎝ 1− p ⎠
c
π0 =
i
c
1 − ( p / (1 − p ) )
1 − ( p / (1 − p ) )
c +1
c
i
c +1
1 − ( p / (1 − p ) ) ⎛ p ⎞
πi =
⎟
c +1 ⎜
1
−
p
⎠
1 − ( p / (1 − p ) ) ⎝
i
i
Periodic States and Multiple
Communicating Classes
For an irreducible, recurrent, periodic, finite Markov chain
with state-transition matrix P the stationary probability vector
π is the unique non-negative solution of πT = πT P , πT 1 = 1 .
This is the same formula used to compute the limiting state
probabilities for an irreducible, aperiodic finite Markov chain,
but here they are called “stationary” instead of “limiting”
because in a recurrent, periodic chain the probabilities don’t
actually converge to a limit but instead oscillate.
Periodic States and Multiple
Communicating Classes
⎡0
⎢0
Example P = ⎢
⎢0
⎢
⎣1
1
0
0
0
0
1
0
0
0⎤
0⎥
⎥
1⎥
⎥
0⎦
[π 0
π 1 π 2 π 3 ] = [π 0 π 1 π 2
[π 0
⎡1⎤
⎢1⎥
π3 ]⎢ ⎥ = 1
⎢1⎥
⎢⎥
⎣1⎦
π1 π 2
⎡0
⎢0
π3 ]⎢
⎢0
⎢
⎣1
1
0
0
0
0
1
0
0
0⎤
0⎥
⎥
1⎥
⎥
0⎦
Periodic States and Multiple
Communicating Classes
Combining equations,
⎡1 −1 0 0 ⎤ ⎡π 0 ⎤ ⎡0 ⎤
⎢0 1 −1 0 ⎥ ⎢ π ⎥ ⎢0 ⎥
⎢
⎥⎢ 1⎥ = ⎢ ⎥
⎢0 0 1 −1⎥ ⎢π 2 ⎥ ⎢0 ⎥
⎢
⎥ ⎢π ⎥ ⎢ ⎥
1
1
1
1
⎣
⎦ ⎣ 3 ⎦ ⎣1 ⎦
The solution is πT = [0.25 0.25 0.25 0.25]. These are the
probabilities of being in each state at a randomly chosen time.
Periodic States and Multiple
Communicating Classes
Let the initial state be state 0. Then
⎡0
⎢0
pT ( n ) = pT ( 0 ) P n = [1 0 0 0] ⎢
⎢0
⎢
⎣1
1
0
0
0
0
1
0
0
0⎤
0⎥
⎥
1⎥
⎥
0⎦
n
The diagonalized form of P n is
1 ⎤ ⎡1 0 0 0 ⎤
⎡1 1 1
⎢1 −1 j − j ⎥ ⎢0 −1 0 0 ⎥
⎥⎢
⎥
P n = SΛS −1 = ⎢
⎢1 1 −1 −1⎥ ⎢0 0 j 0 ⎥
⎢
⎥⎢
⎥
1
−
1
−
j
j
0
0
0
−
j
⎣
⎦⎣
⎦
n
1⎤
⎡1 1 1
⎢1 −1 j − j ⎥
⎢
⎥
⎢1 1 −1 −1⎥
⎢
⎥
1
−
1
−
j
j
⎣
⎦
−1
Periodic States and Multiple
Communicating Classes
⎡0
⎢0
P2 = ⎢
⎢1
⎢
⎣0
0
0
0
1
1
0
0
0
pT [0] = [1
pT [1] = [0
and pT [ 2] = [0
pT [3] = [0
M M
0⎤
⎡0 0 0 1 ⎤
⎡1 0
1 ⎥ 3 ⎢1 0 0 0 ⎥ 4 ⎢ 0 1
⎥ P =⎢
⎥ P =⎢
0⎥
⎢0 1 0 0 ⎥
⎢0 0
⎥
⎢
⎥
⎢
0⎦
0
0
1
0
⎣
⎦
⎣0 0
0 0 0] So the state probabilities
1 0 0] are periodic and do not
0 1 0]. converge to a limiting
value, but they are
0 0 1]
stationary.
M
0
0
1
0
0⎤
0⎥
⎥
0⎥
⎥
1⎦
Periodic States and Multiple
Communicating Classes
For a Markov chain with recurrent communicating classes C1 ,L , Cm
let π ( ) indicate the limiting state probabilities (for all states in the
k
Markov chain) associated with entering class Ck . Given that the
system starts in a transient state i the limiting probability of
state j is
lim Pij [ n ] = π (j ) P [ Bi1 ] + L + π (j ) P [ Bim ]
1
m
n →∞
where P [ Bik ] is the conditional probability that the system enters
class Ck . (The condition is that it started in state i.)
Periodic States and Multiple
Communicating Classes
Example
For each possible starting state i ∈ {0,1, 2,3, 4,5, 6} find
the limiting state probabilities.
Periodic States and Multiple
Communicating Classes
The communicating classes are C1 = {0,1} , C2 = {3} , C3 = {4, 5, 6}
and C4 = {2}. Classes C1 , C2 and C3 are recurrent and C4 is
transient. If the initial state is either 0 or 1, the limiting state
probabilities for states 2-6 are all zero. The analysis proceeds as
though the states 2-6 did not exist.
So π
(1)
(1)
= ⎡⎣π 0
(1)
π1
T
0 0 0 0 0 ⎤⎦ and we can find the vector
⎡π 0(1) π 1(1) ⎤ from ⎡π 0(1) π 1(1) ⎤ = ⎡π 0(1) π 1(1) ⎤ P (1) , ⎡π 0(1) π 1(1) ⎤ 1 = 1
⎣
⎦
⎣
⎦ ⎣
⎦
⎣
⎦
where P ( ) is the state-transition matrix for class C1 alone.
1
Periodic States and Multiple
Communicating Classes
Solving,
π
(1)
= [0.5882 0.4118 0 0 0 0 0]
T
π ( ) = [0 0 0 1 0 0 0]
T
2
π
( 3)
= [0 0 0 0 0.2941 0.4706 0.2353]
T
For the case of starting in state 2
π = π ( ) P ( B21 ) + π ( ) P ( B22 ) + π ( ) P ( B23 )
123
123
123
1
2
= 0.5
3
= 0.2
= 0.3
πT = [0.2941 0.2059 0 0.2 0.0882 0.1412 0.0706]
Countably Infinite Chains: State
Classification
The state space for a countably-infinite Markov chain is {0,1, 2,L }.
The state-transition matrix and state-probability vector still have
the same notation, but now they both have infinite dimension.
The basic relationships hold but cannot now be computed by
matrix manipulation. They are
∞
Pij [ n + m ] = ∑ Pik [ n ] Pkj [ m ]
k =0
∞
∞
i =0
i =0
p j [ n ] = ∑ pi [0] Pij [ n ] = ∑ pi [ n − 1] Pij
π j = lim p j [ n ]
n →∞
This last limit may or may not exist.
Countably Infinite Chains: State
Classification
For this countably-infinite Markov chain, whether or not a state
is recurrent depends on the parameter p.
Countably Infinite Chains: State
Classification
Given that a Markov chain is in state i at some arbitrary time,
and
1.
Vii is the event, the system eventually returns to state i,
2.
Tii is the number of transitions until the system first returns
3.
to state i,
N ii is the number of times in the future that the system
returns to state i.
Countably Infinite Chains: State
Classification
If a system starts in state i it will return if the number of
transitions Tii is finite.
P (Vii ) = P ⎡⎣Tii < B ⎤⎦ = lim FT ⎡⎣ n ⎤⎦
ii
n→∞
where B is a finite upper bound and FT ⎡⎣ n ⎤⎦ is the cumulative
ii
distribution function (CDF) for Tii as a function of discrete time n.
For a countably-infinite Markov chain, state i is recurrent if
P (Vii ) = 1, otherwise it is transient.
Countably Infinite Chains: State
Classification
Example
Is state 0 transient or recurrent?
T00 will be greater than n if the system reaches state n before
returning to state zero The probability that T00 is greater than n is
1 2
n − 1 1 ⋅ 2 ⋅ 3 L ( n − 1) 1
P [T00 > n ] = 1× × × L ×
=
=
2 3
n
2 ⋅ 3 L ( n − 1) ⋅ n n
Countably Infinite Chains: State
Classification
The CDF for Tii is FTii [ n ] = 1 − 1/ n and the probability that a
return to state 0 eventually happens is
P (Vii ) = lim FTii [ n ] = lim (1 − 1/ n ) = 1
n →∞
n →∞
An eventual return to state 0 is guaranteed and state 0 is recurrent.
Countably Infinite Chains: State
Classification
If a state i is recurrent then over an infinite time the expected
number of returns to that state E ( N ii ) is infinite. If a state is
transient the expected number of returns to the state is finite.
Therefore, a state i is recurrent if, and only if, E ( N ii ) is infinite.
The expected number of visits to state i over all time is
∞
E ( N ii ) = ∑ Pii [ n ]
n =1
(not 0)
Countably Infinite Chains: State
Classification
Example
Is state 0 recurrent?
Random Walk
Countably Infinite Chains: State
Classification
If we start at state 0, in order to return to state 0 after n steps, n
must be even. Half the steps are to the right and half the steps are
to the left. So we are looking for the probability in n = 2m steps
(m an integer) that we take exactly m to the right and m to the left.
This is 2m trials of an experiment (one step), exactly m of them
being of a certain type so the probability is
⎛ 2m ⎞ m
m
P00 [ n ] = ⎜ ⎟ p (1 − p )
⎝m⎠
Countably Infinite Chains: State
Classification
∞
E ( N 00 ) = ∑ P00 [ n ] =
n =1
∞
∞
∑ P [ n ] = ∑ P [ 2m ]
n=2
n even
00
m =1
00
⎛ 2m ⎞ m
m
E ( N 00 ) = ∑ ⎜ ⎟ p (1 − p )
m =1 ⎝ m ⎠
We can use Stirling’s approximation to help sum this series.
∞
n ! ≅ 2nπ ( n / e ) for large n where e is the base of the natural
n
logarithm.
Countably Infinite Chains: State
Classification
⎛n⎞
( 2m )!
n!
⎜ m ⎟ = m ! n − m ! = m !m !
(
)
⎝ ⎠
P00 [ n ] ≅
4mπ ( 2m / e )
2m
2mπ ( m / e ) × 2mπ ( m / e )
m
E ( N 00 ) =
1
π
∞
∑
p (1 − p )
m
m
m
⎡⎣ 4 p (1 − p )⎤⎦
=
mπ
αm
m
where α = 4 p (1 − p ) . If p = 1/2, the series diverges, the expected
m =1
number of returns to state 0 is infinite and state 0 is recurrent.
Otherwise the series converges, the expected number of returns to
state 0 is finite and state 0 is transient.
m
Countably Infinite Chains: State
Classification
If a state is recurrent it may be positive recurrent or null recurrent.
Positive recurrent means that the expected time of return to the state
is finite. Null recurrent means that the expected time to return
to the state is infinite.
Probability of
Expected Number Expected Time
Eventual Return
of Returns
of First Return
Transient
Null Recurrent
<1
1
Finite
Infinite
Infinite
Infinite
Positive Recurrent
1
Infinite
Finite
Countably Infinite Chains: State
Classification
Example
We have already seen that state 0 is recurrent. Is it positive recurrent
or null recurrent?
The probability that the time of return equals n is
1
1
1
P (T00 = n ) = P (T00 > n − 1) − P (T00 > n ) =
− =
, n >1
n − 1 n n ( n − 1)
Countably Infinite Chains: State
Classification
The expected time of first return is
∞
∞
∞
1
1
E (T00 ) = ∑ n P (T00 = n ) = ∑
=∑
n =0
n=2 n − 1
n =1 n
This is a harmonic series and it diverges, indicating that the
expected time of first return to state 0 is infinite and therefore
that state 0 is null recurrent.
Countably Infinite Chains: State
Classification
In a communicating class all states have the same recurrence
qualities. They are either
or
1.
All transient,
2.
All null recurrent,
3.
All positive recurrent.
Countably Infinite Chains:
Stationary Probabilities
For an irreducible, aperiodic, positive-recurrent Markov chain with
states {0,1, 2,L } , the limiting n-step transition probabilities are,
lim Pij [ n ] = π j where {π j | j = 0,1, 2,L
n →∞
∞
} are the unique state
probabilities satisfying π j = ∑ π i Pij , j = 0,1, 2,L
i =0
and
∞
∑π
j =0
j
= 1.
Countably Infinite Chains:
Stationary Probabilities
Example
Find the stationary probabilities and specify for which values of p
these probabilities exist.
p
π i p = π i+1 1− p ⇒ π i+1 =
πi
1− p
(
)
Countably Infinite Chains:
Stationary Probabilities
2
i
⎛ p ⎞
⎛ p ⎞
p
π1 =
π0 ⇒ π2 = ⎜
π0 ⇒ πi = ⎜
π0
⎟
⎟
1− p
⎝ 1− p ⎠
⎝ 1− p ⎠
The sum of the stationary probabilities must be one (if they exist).
i
⎛ p ⎞
πi = π0 ∑⎜
=1
∑
⎟
i =0
i =0 ⎝ 1 − p ⎠
This summation converges if p < 1 / 2.
∞
∞
Continuous-Time Markov Chains
A continuous-time Markov chain { X ( t ) | t ≥ 0} is a continuous-time,
discrete-value random process such that for an infinitesimal
time step Δ
Probability of an i-j
P ⎡⎣ X ( t + Δ ) = j | X ( t ) = i ⎤⎦ = qij Δ
transition in time Δ
P ⎡⎣ X ( t + Δ ) = i | X ( t ) = i ⎤⎦ = 1 − ∑ qij Δ
j ≠i
Probability of an i-i
transition in time Δ
This implies that
P ⎡⎣ X ( t + Δ ) ≠ i | X ( t ) = i ⎤⎦ = ∑ qij Δ
j ≠i
Probability of a
transition out of
state i in time Δ
Continuous-Time Markov Chains
Since Δ is very small, the product qij Δ is also small making the
probability of a transition in any one time slot Δ small. Thus for
most of these small time slots nothing happens. Only rarely
does a transition occur.
Continuous-Time Markov Chains
For a Markov chain in state i the time until the next transition
is an exponential random variable with parameter, vi = ∑ qij
j ≠i
called the departure rate of state i. For an exponential random
variable, the waiting time until the next transition is independent
of all previous waiting times. That is, no matter how long we have
waited without a transition, the expected time of the next transition
is still the same 1/ vi . The time between transitions is random with
an exponential PDF.
Continuous-Time Markov Chains
For a continuous-time Markov chain, when the system enters state i
a Poisson process N ik ( t ) with rate parameter qik starts for every
other state k . If the process N ij ( t ) happens to be the first to have
an arrival, then the system transitions to state j. When the system
is in state i the time until departure is an exponential random variable
with expected time of transition 1/ vi . Given the event Di that the
system departs state i in the time interval t0 < t ≤ t0 + Δ the
conditional probability of the event Dij that the system transitioned
to state j is P ( Dij | Di ) =
P ( Dij )
P ( Di )
=
qij Δ
vi Δ
=
qij
vi
. If we ignore the time
spent in each state, the transition probabilities can be viewed as the
transition probabilities of a discrete-time Markov chain.
Continuous-Time Markov Chains
For a continuous-time Markov chain with transition rates qij and
state-i departure rates ν i the embedded discrete - time Markov chain
has transition probabilities Pij = qij / vi for states i with ν i > 0 and
Pii = 1 for states i with ν i = 0. The communicating classes of a
continuous-time Markov chain are given by the communicating
classes of its embedded discrete-time Markov chain. A continuoustime Markov chain is irreducible if its embedded discrete-time Markov
chain is irreducible. An irreducible continuous-time Markov chain is
positive recurrent if, for all states i the time Tii to return to state i
satisfies E (Tii ) < B where B is a finite upper bound.
Continuous-Time Markov Chains
Example
In the summer, an air conditioner is in one of three states: (0) off,
(1) low and (2) high. While in the off state, transitions to the low
state occur after an exponential time with an expected value of 3
minutes. While in the low state, transitions to the off state or the
high state are equally likely and transitions from the low state occur
at a rate of 0.5 per minute. When the system is in its high state, it
makes a transition to the low state with a probability of 2/3 or to the
off state with a probability of 1/3. The time spent in the high state is
an exponential random variable with an expected value of 2 minutes.
Model this air conditioning system using a continuous-time Markov
chain.
Continuous-Time Markov Chains
The transition rate from state 0 to state 1 q01 is 1/3 per minute and
the transition rate from state 0 to state 2 q02 is zero. The transition
rate out of state 1 ν 1 is 0.5 per minute. The fact that transitions
from 1 to 0 and 1 to 2 are equally likely means that
q10 /ν 1 = q12 /ν 1 = 0.5. Therefore q10 = q12 = 0.5 × 0.5 = 0.25.
q21 /ν 2 = 2 / 3
q20 /ν 2 = 1/ 3
q20
1
1
ν2 =
⇒ q20 = ×ν 2 =
1/ 3
3
6
ν 2 = 1/ 2
q21
2
1
ν2 =
⇒ q21 = ×ν 2 =
2/3
3
3
Continuous-Time Markov Chains
In a continuous-time Markov chain the process is characterized by
the transition rates rather than the transition probabilities. The
transition rate from a state to itself is taken as zero because in that
transition that nothing has actually changed. The state transition
matrix is
⎡ 0 q01
⎢q
0
10
⎢
Q=
⎢ M M
⎢
⎣ qN 1 qN 2
L
L
O
L
q0 N ⎤
q1N ⎥
⎥
M⎥
⎥
0 ⎦
Continuous-Time Markov Chains
Another matrix, the rate matrix R is also useful.
⎡ −ν 0
⎢q
R = ⎢ 10
⎢ M
⎢
⎣ qN 1
q01
L
−ν 1 L
M O
qN 2 L
q0 N ⎤
q1N ⎥
⎥
M⎥
⎥
−ν N ⎦
Since ν i = ∑ j ≠i qij , each row of R must sum to zero.
Continuous-Time Markov Chains
The probability that the system is in state i is pi ( t ) = P ⎡⎣ X ( t ) = i ⎤⎦ .
If the number of states is finite the state probabilities can be written
T
in matrix form as p ( t ) = ⎡⎢ p0 ( t ) p1 ( t )  p N ( t ) ⎤⎥ .The system is
⎣
⎦
characterized by a system of differential equations and, in the case of
a finite number of states, by one vector differential equation in p ( t ) .
The evolution of state probabilities over time is governed by
d
d T
p j ( t ) = ∑ rij pi ( t ) , j = 0,1,2, or
p ( t ) = pT ( t ) R
dt
dt
all i
(
)
(
)
Continuous-Time Markov Chains
Rate of increase of
the probability of
being in state j
Probability of
being in state i
(
) ∑ r p (t )
d
p j (t ) =
dt
ij
i
all i
Rate of transitions
from state i to state j
(one element of the
rate matrix R)
Continuous-Time Markov Chains
The vector form of the equations can be solved and the solution
form is p ( t ) = p ( 0 ) e
T
T
Rt
∞
where e = ∑
Rt
k=0
( Rt )
k!
k
is known as the
matrix exponential. Usually, in practice, the desired solution is
the steady-state solution for t → ∞.
Continuous-Time Markov Chains
For an irreducible, positive-recurrent, continuous-time Markov
chain, the state probabilities satisfy
T
T
r
p
=
0
or
p
R
=
0
∑ ij i
all i
and
T
p
=
1
or
p
1 = 1.
∑ j
all j
In steady state,
p ν = ∑ pi qij .
{j j
rate of
1i ≠ j 2 3
transitions
out of state j
rate of
transitions
into state j
Continuous-Time Markov Chains
Example
In a continuous-time ON-OFF process, alternating OFF and ON
(states 0 and 1) periods have independent exponential durations.
The average ON period lasts 1/ µ seconds while the average OFF
period lasts 1/ λ seconds. Draw the diagram and find the limiting
state probabilities.
Transition rate,
not transition
probability
Continuous-Time Markov Chains
p R = 0 ⇒ [ p0
T
T
⎡ −λ
p1 ] ⎢
⎣µ
p 1 = 1 ⇒ [ p0
T
λ ⎤ ⎡0⎤
=⎢ ⎥
⎥
− µ ⎦ ⎣0⎦
⎡1⎤
p1 ] ⎢ ⎥ = 1
⎣1⎦
Combining equations,
⎡ −λ
⎢1
⎣
µ ⎤ ⎡ p0 ⎤ ⎡0 ⎤
=⎢ ⎥
⎢
⎥
⎥
1 ⎦ ⎣ p1 ⎦ ⎣1 ⎦
Solving,
−λ 0
1 0 µ
µ
1
λ
p0 =
=
, p1 =
=
−λ − µ 1 1 λ + µ
−λ − µ 1 1 λ + µ
Continuous-Time Markov Chains
Example
Find the stationary distribution for the Markov chain describing
the air conditioning system.
Continuous-Time Markov Chains
0 ⎤ ⎡0 ⎤
⎡ −1/ 3 1/ 3
⎡001 ⎤
T
T
T
⎢
⎥
⎢
⎥
p R = 0 ⇒ [ p0 p1 p2 ] 1/ 4 −1/ 2 1/ 4 = 0 ⇒ p [ R 01 R 2 ] ⎢ ⎥
⎢
⎥ ⎢ ⎥
0⎦
⎣
⎢⎣ 1/ 6 1/ 3 −1/ 2 ⎥⎦ ⎢⎣0 ⎥⎦
⎡1⎤
pT 1 = 1 ⇒ [ p0 p1 p2 ] ⎢1⎥ = 1
⎢⎥
⎢⎣1⎥⎦
Combining equations and solving,
⎡ −1/ 3 1/ 4 1/ 6 ⎤ ⎡ p0 ⎤ ⎡0 ⎤
⎡R ⎤
⎡001 ⎤ ⎢
⎥ ⎢ p ⎥ = ⎢0 ⎥
p
=
⇒
1/
3
−
1/
2
1/
3
⎢
⎥
1⎥
⎢1⎥ ⎢
⎥
⎢
⎢ ⎥
1
⎣ ⎦
⎣
⎦
1
1 ⎥⎦ ⎢⎣ p2 ⎥⎦ ⎢⎣1 ⎥⎦
⎢⎣ 1
pT = [ 2 / 5 2 / 5 1/ 5]
T
01
T
Birth-Death Processes and
Queueing Systems
A continuous-time Markov chain is a birth-death process if the
transition rates satisfy qij = 0 for i − j > 1.
The transition rate qi ,i −1 is called the service rate and the
transition rate qi ,i +1 is called the arrival rate. We will assume
that µi > 0 for all states i that are reachable from state 0. This
ensures that the chain is irreducible.
Birth-Death Processes and
Queueing Systems
For a birth-death queue with arrival rates λi and service rates µi
the stationary probabilities satisfy pi −1λi −1 = pi µi and
∞
∑p
i =0
i
= 1.
λi
Define the load as ρi =
. The limiting state probabilities,
µi +1
ρ
∏
=
1+ ∑ ∏
i −1
if they exist, are pi
converges.
j =0
j
∞
k −1
k =1
j =0
ρj
and they exist if
∑ ∏
∞
k −1
k =1
j =0
ρj
Birth-Death Processes and
Queueing Systems
Product of loads
on states 0 through
i-1
∏
i −1
j =0
ρj
Load on state j
Product of loads
on states 0 through
k-1
Sum of products
of loads on states
0 through k - 1,
for all k > 0
∑ ∏
∞
k −1
k =1
j =0
ρj
Load on state j
Birth-Death Processes and
Queueing Systems
Naming convention for queues A / S / n / m
A for arrival process, S for service process, n for number of
servers and m for number of customers
For positions A and S in the queue name,
M means the process is a Poisson process and memoryless
D means the process is deterministic with a uniform rate
G means the process is a general process
When the number of customers in the system is less than the
number of servers n then an arriving customer is immediately
assigned to a server. When m is finite and the queue has m
customers in it, new arrivals are blocked (discarded). If m is
not specified, it is assumed to be infinite.
Birth-Death Processes and
Queueing Systems
The M / M /1 Queue
The arrivals are a Poisson process with rate parameter λ
independent of the service requirements of the customers. The
service rate is an exponential random variable (because the
exponential random variable is memoryless) independent of the
service rate of the system. There is only one server in the system
so the departure rate from any state i > 0 is µi = µ . The limiting
state probabilities are pn = (1 − ρ ) ρ n , n = 0,1, 2,L
Birth-Death Processes and
Queueing Systems
Example
Cars arrive at a toll booth as a Poisson process at a rate of 0.6
cars per minute. The rate at which they are serviced is an
exponential random variable with an expected value of 0.3 minutes.
What are the limiting state probabilities for N, the number of cars
at the toll booth? What is the probability that there are no cars at
the toll booth at any randomly-chosen time in the future?
λ
0.6
ρ= =
= 0.18 pn = (1 − 0.18 ) 0.18n = 0.82 × 0.18n
µ (1/ 0.3)
The probability that there are no cars at the toll booth is
p0 = 0.82 × 0.180 = 0.82
Birth-Death Processes and
Queueing Systems
The M / M / ∞ Queue
With infinitely many servers, each customer is served immediately
upon arrival. When n customers are in the system the system
departure rate is nµ although the service rate (which applies to
individual customers) is still µ .
Birth-Death Processes and
Queueing Systems
The M / M / ∞ queue with arrival rate λ and service rate µ has
limiting state probabilities
⎧ ρ ne− ρ
, n = 0,1, 2,L
⎪
pn = ⎨ n !
⎪0
, otherwise
⎩
Birth-Death Processes and
Queueing Systems
Example
At the beach in the summer, swimmers enter the ocean as a
Poisson process with a rate parameter of 300 swimmers per hour.
The time spent in the ocean by a randomly-chosen swimmer is
an exponential random variable with an expected value of 20
minutes. Find the limiting state probabilities of the number of
swimmers in the ocean.
λ 300 / hour
ρ= =
= 100
µ
3 / hour
⎧100n e −100
, n = 0,1, 2,L
⎪
pn = ⎨
n!
⎪0
, otherwise
⎩
Birth-Death Processes and
Queueing Systems
The average number of swimmers in the ocean is the expected
value of n,
n
n
∞
∞
100 n e−100
100
100
E ( n ) = ∑ npn = ∑ n
= e−100 ∑ n
= e−100 ∑ n
n!
n!
n!
n=0
n=0
n=0
n=1
∞
∞
Using the infinite-series definition of the exponential function,
(
)
e = ∑ n=0 x n / n! , we get
x
∞
∞
n−1
n
∞
100
100
E ( n ) = 100e−100 ∑
= 100e−100 ∑
= 100e−100 e100 = 100
n=1 ( n − 1)!
n=0 n!
Birth-Death Processes and
Queueing Systems
The M / M / c / c Queue
This queue has c servers and a capacity for c customers in the
system. Customers arrive as a Poisson process with an arrival rate
λ. If there are c − 1 or fewer customers being served, an arriving
customer is immediately served. If there are c customers being
served an arriving customer is turned away and denied service
(it never enters the system). The service time of a customer
admitted to the system is an exponential random variable with
an expected value of µ.
Birth-Death Processes and
Queueing Systems
The limiting state probabilities satisfy
⎧ ρ n / n!
, j = 0,1, 2,L
⎪ c
j
pn = ⎨ ∑ j =0 ρ / j !
⎪
, otherwise
⎩0
Since this queue can only accommodate c customers, the probability
that a customer will not be served is the same as the probability
that there are c customers currently being served which is
ρ c / c!
pc = c
j
ρ
∑ j =0 / j !
Birth-Death Processes and
Queueing
Systems
Example
A telephone exchange can handle up to 100 calls simultaneously.
Call duration is an exponential random variable with an expected
value of 2 minutes. If requests for connection occur as a Poisson
random variable with a rate of 40 calls per minute, what is the
probability that a caller will get a busy signal?
ρ=
λ
40
=
= 80
µ (1/ 2 )
c = 100
The probability of a busy signal is p100 =
80100 /100!
∑
100
j =0
j
80 / j !
.
nnnnnnnnnnnnnn
nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn
The answer is 0.004.