Embedded Markov Chain
Limit Probabilities
Limit Distribution of BD Process
Chapter 5: Continuous Time
Markov Chain
Part 3
Kurnia Susvitasari
ksusvita@sci.ui.ac.id
Department of Mathematics
University of Indonesia
Kurnia Susvitasari
SCST602204
1 / 22
Embedded Markov Chain
Limit Probabilities
Limit Distribution of BD Process
Outlines
2 Limit Probabilities
1 Embedded Markov Chain
Jump Process
Class Properties of a
CTMC
Kurnia Susvitasari
The Existence Conditions
Balanced Equations
3 Limit Distribution of BD Process
SCST602204
2 / 22
Embedded Markov Chain
Limit Probabilities
Limit Distribution of BD Process
Jump Process
Definition 5.6
Given a CTMC {Y (t) : t ≥ 0} defined in a state space S. The jump
process is a discrete-time process {Xn = Y (Tn ) : n ∈ Z+ } where
Tn is the sojourn time between the (n − 1)-th and n-th events.
Kurnia Susvitasari
SCST602204
3 / 22
Embedded Markov Chain
Limit Probabilities
Limit Distribution of BD Process
Interpretation of Def’n 5.6
Jump process is derived from a CTMC by sampling this
process at the jump times.
Sometimes, jump process is also called embedded chain of a
CTMC.
The corresponding jump transition probabilities are, for i, j ∈ S
Pij = Pr{Xn+1 = j|Xn = i} = Pr{Y (Tn+1 ) = j|Y (Tn ) = i}
The jump process is a homogeneous Markov chain with
countable state space.
Kurnia Susvitasari
SCST602204
4 / 22
Embedded Markov Chain
Limit Probabilities
Limit Distribution of BD Process
Example: A Poisson Process
Given a Poisson process {N(t) : t ≥ 0} with rate λ for all state
i ∈ {0, 1, . . .}.
We know that the inter-arrival time is exponentially
distributed and independent.
The process can only jump from i to i + 1; we call it pure birth
process.
The embedded Markov chain has the transition probabilities
P0,1 = Pr{N(T1 ) = 1, N(0) = 0}
Pi,i+1 = Pr{N(Ti+1 ) = i + 1, N(Ti ) = i}
Kurnia Susvitasari
SCST602204
5 / 22
Embedded Markov Chain
Limit Probabilities
Limit Distribution of BD Process
Recurrent and Transient States of a CTMC
Given the embedded DTMC {Xn } associated with any CTMC. Then,
The transition probabilities of {Xn } form the P-matrix of
CTMC.
For any state i ∈ S, Pii = 0, i.e. states do not transition to
themselves.
Periodicity is not possible in CTMC.
Kurnia Susvitasari
SCST602204
6 / 22
Embedded Markov Chain
Limit Probabilities
Limit Distribution of BD Process
Class Properties of CTMC
Let i, j ∈ S, where i ̸= j. Denote “↔” be communication property.
Lemma 5.2
States i ↔ j in a CTMC if i ↔ j in the embedded MC.
Lemma 5.3
Since communication of states partitions embedded MC into
classes, it also partitions the associated CTMC.
Kurnia Susvitasari
SCST602204
7 / 22
Embedded Markov Chain
Limit Probabilities
Limit Distribution of BD Process
Class Properties of CTMC (Con’d)
Lemma 5.4
A CTMC is irreducible if the embedded MC is irreducible.
Lemma 5.5
State i is recurrent in a CTMC if i is recurrent in the embedded MC.
Lemma 5.6
State i is transient in a CTMC if i is transient in the embedded MC.
Kurnia Susvitasari
SCST602204
8 / 22
Embedded Markov Chain
Limit Probabilities
Limit Distribution of BD Process
Class Properties of CTMC (Con’d)
Lemma 5.7
State i is positive recurrent in a CTMC if i is positive recurrent in the
embedded MC.
Lemma 5.8
Transient and recurrent are class properties of a CTMC.
Kurnia Susvitasari
SCST602204
9 / 22
Embedded Markov Chain
Limit Probabilities
Limit Distribution of BD Process
Limit Probabilities
Definition 5.7
Given a CTMC {X (t) : t ≥ 0} with TPF P(t). Let
πj = lim Pij (t),
t→∞
i.e. the MC will be in state j at t no matter where it initially comes
from. If such a limit exists, we call it limit probabilities of {X (t)}.
Sometimes, it is also called long-run proportion of time that the
process is in state j.
Kurnia Susvitasari
SCST602204
10 / 22
Embedded Markov Chain
Limit Probabilities
Limit Distribution of BD Process
Conditions for Existence
Theorem 5.5
If a CTMC {X (t) : t ≥ 0} defined in state space S is
(a) irreducible, and
(b) positive recurrent MC,
then, for all j ∈ S,
πj = lim Pij (t)
t→∞
exists and independent of initial state i.
If limit distribution exists, we call such process ergodic.
Kurnia Susvitasari
SCST602204
11 / 22
Embedded Markov Chain
Limit Probabilities
Limit Distribution of BD Process
Implication of Thm 5.5
Suppose {X (t) : t ≥ 0} has transition rates qij for all k , j ∈ S.
The steady state (or limit) probabilities πj ≥ 0 are the unique
solution of the system of linear equations
rate at which the process enters j
rate at which the process leaves j
z}|{
νj πj
=
zX}| {
qkj πk
(5.1)
k ̸=j
X
πk = 1
k ∈S
P
We can show that πj = νj /( i∈S νi ).
We call eq. (5.1) a balance equation.
Kurnia Susvitasari
SCST602204
12 / 22
Embedded Markov Chain
Limit Probabilities
Limit Distribution of BD Process
Kolmogorov’s Equation to Limit Probabilities
We will show the algebraic relations of (5.1) obtained from Kolmogorov’s forward equation.
The Kolmogorov’s eq. shows the instantaneous change in
probabilities P(t) in an infinitesimal time h.
When the process is steady (stationary), the change is zero,
i.e. if the limit distribution exists
=πj
=πk
z }| {
X z }| {
d
Pij (t) = − lim Pij (t) νj +
lim Pik (t) qkj
lim
t→∞
t→∞
t→∞ dt
k ̸=j
X
⇔ 0 = −πj νj +
πk qkj .
k ̸=j
That is, we obtained our desired equation.
Kurnia Susvitasari
SCST602204
13 / 22
Embedded Markov Chain
Limit Probabilities
Limit Distribution of BD Process
Balanced Equation in Matrix Form
Given that the limit distribution exists,
P ≡ lim P(t) = 1π,
t→∞
where 1 is the column vector of 1’s and π is the row vector of
limit probabilities πj ’s.
Then,
=1
z }| {
πP = (π1) π = π
As well,
πQ = 0.
Kurnia Susvitasari
SCST602204
14 / 22
Embedded Markov Chain
Limit Probabilities
Limit Distribution of BD Process
Recall a BD Process
define BD process with birth rate λi and death rate µi
draw a transition scheme
solve balance equation by recursive
show the necessary and sufficient condition for which the limit
distribution exists.
Kurnia Susvitasari
SCST602204
15 / 22
Embedded Markov Chain
Limit Probabilities
Limit Distribution of BD Process
Recall a BD Process
Recall a BD process. When X (t) = i, let birth and death occur
exponentially with rate λi and µi , respectively.
The transition scheme is given as follows
Kurnia Susvitasari
SCST602204
16 / 22
Embedded Markov Chain
Limit Probabilities
Limit Distribution of BD Process
Balance Eq. in BD Process
The balance eq. can be expressed as follows
Kurnia Susvitasari
State
0
1
2
..
.
Balance Eq.
λ0 π0 = µ1 πi
(λ1 + µ1 )π1 = µ2 π2 + λ0 π0
(λ2 + µ2 )π2 = µ3 π3 + λ1 π1
..
.
n≥1
(λn + µn )πn = µn+1 πn+1 + λn−1 πn−1
SCST602204
17 / 22
Embedded Markov Chain
Limit Probabilities
Limit Distribution of BD Process
Solving Balance Equation
Using recursive method, we can solve the balance equation in
terms of π0 as follows
λ0
π0
µ1
λ0
λ1 λ0
π2 =
π1 =
π0
µ1
µ2 µ1
..
.
λn−1 . . . λ0
λn−1
πn =
πn−1 =
π0
µn
µn . . . µ1
π1 =
Kurnia Susvitasari
SCST602204
18 / 22
Embedded Markov Chain
Limit Probabilities
Limit Distribution of BD Process
Solving Balance Eq. (Cont’d)
Since
P
n πn = 1, we obtain
1 = π0 + π0
X λn−1 . . . λ0
n
π0 =
µn . . . µ1
1
P λn−1 . . . λ0
1+ n
µn . . . µ1
Thereby, for n ≥ 1
πn =
µn . . . µ1
Kurnia Susvitasari
λn−1 . . . λ0
!
X λn−1 . . . λ0
1+
µn . . . µ1
n
SCST602204
19 / 22
Embedded Markov Chain
Limit Probabilities
Limit Distribution of BD Process
Solving Balance Eq. (Cont’d)
The necessary condition for πn to exist is such that
X λn−1 . . . λ0
n
µn . . . µ1
<∞
The condition above can be shown to be sufficient.
Kurnia Susvitasari
SCST602204
20 / 22
Embedded Markov Chain
Limit Probabilities
Limit Distribution of BD Process
Some Notes
From the transition scheme, the chain is irreducible (every
state communicates to each other). We can argue that the
embedded MC is positive recurrent chain, hence the limit
probabilities exist.
Fact, it is very difficult to show a chain with infinitely many
states to be positive recurrent. In practice, we solve the
balance eq. and then determine the condition in which
these probabilities exist.
Kurnia Susvitasari
SCST602204
21 / 22
Embedded Markov Chain
Limit Probabilities
Limit Distribution of BD Process
“If you have knowledge, let others light their candles at it.”
– Margaret Fuller
Best wishes on your study!
Kurnia Susvitasari
SCST602204
22 / 22