Chapter 6
Continuous-Time Markov Chains

Introduction
o Definition: It is a stochastic process that moves from state to state in
accordance with a discrete-time Markov chain, but the amount of time spent
in each state is exponentially distributed.
o Memoryless
o Transition probability (pij) is the probability that the process will make a
transition from state i to state j.
o Pii = 0,
all i
o  Pij = 1,
j

all i
o Problem 6.2 and 6.12(a)
Birth and death processes: A continuous-time Markov chain for which transitions
may take place only between neighboring states (i.e. n to n + 1 or n – 1 only).
n
N(t) = n
n
o n – arrival rate
o n – departure rate
o n and n are independent of time.
o n and n may be dependent on n.
o State transition rate vi (the rate at which the process leaves the state i):

v0 = 0
 vi = i + i
o State transition probability:
 p01 = 1
 Birth before death:
i
pi ,i 1 
i   i
 Death before birth:
i
pi ,i 1 
i   i
o Problem 6.4
o Special cases:
 Pure birth process:  = 0. If  = constant, it is a Poisson process (Ex. 6-2)
 Pure death process:  = 0. If  = constant, it is a Poisson process (Problem
6-9)
o Queueing process basics:
 M/M/1:
n = 
n0
;
n = 
n1
 M/M/1 with blocking:
n is the probability that a customer will enter the system with n
customers already in the system.
n = n 
n0
;
n = 
n1
 M/M/s:
s = number of severs
n
=
n0
n
1ns
n
=
s
n>s
o Transition time Ti: The time required to go from state i to state i+1.
1
1 i
E[T0 ] 
E[Ti ] 

E[Ti 1 ]
{
0
Var (T0 ) 
i
i
1
0 2

i
1
 i Var (Ti 1 ) 
( E[Ti 1 ]  E[Ti ]) 2
i (i  i ) i
(i  i )
Transition probability function:
o Pij is the one-step transition probability.
Var (Ti ) 

o Pij(t) is the transition probability for a time period equal to t, during which
any number of steps including zero step can happen.
o As a result, although Pii must equal to zero, Pii(t) is in general greater than 0.
o Let vi be the transition rate when the process is in state i and qij be the
transition rate from state i to state j.
qij
qij
qij  pij vi  vi   vi Pij   qij and Pij 

vi  qij
j
j
j
o From the above relationships we can derive a set of differential equations for
the transition probability function Pij(t).
 Kolmogorov’s backward equations:
P'ij (t )   qik Pkj (t )  vi Pij (t ), t  0
k i

Kolmogorov’s forward equations:
P'ij (t )   qkj Pik (t )  v j Pij (t ), t  0
k j
o The general equation for Pii(t) is too messy to deal with. We will only discuss
the birth and death process here (Example 6.11).

Limiting Probability: Pj  lim Pij (t )
t 
o General formulation:
v j Pj

Departure rate

 qkjPk
k j


and
 Pj  1
j
Arrival rate
o For a birth and death process:
0 P0  1P1
(1  1 ) P1   2 P2  0 P0  1P1  1P1   2 P2  0 P0   1P1   2 P2
(2   2 ) P2   3 P3  1P1  2 P2   2 P2   3 P3  1P1  2 P2   3 P3
.
.
(n   n ) Pn   n 1Pn 1  n 1Pn 1

0 P0  1P1
 1P1   2 P2
.
.
n Pn   n 1Pn 1

P1  0 P0
1


P2  1 P1  1 0 P0
2
 2 1
.
.
.
n 1
 i
Pn  i  0
n
 i
P0 
i 1
P0 
0 1...n 1
 
 
 
   ... 
1 2 ... n 1  0  0 1  0 1 2  ...  0 1 2 n 1 
1 1 2 1 2 3
1 2 3 ... n 

1
j 1





  k

k 0
1   j

 j 1  
 k

k 1


Problem 6.13

1

 
 
   ...
1  0  0 1  0 1 2  ...  0 1 2 n 1
1 1 2 1 2 3
1 2 3 ... n
o Limiting probability may not exist. For example, an M/M/1 process with 
greater than  would have a queue that expands indefinitely. For a birth and
death process the condition is:
j 1



 
   ... 
  0  0 1  0 1 2  ...  0 1 2 n1   
 
 
1 2  3 ... n 
j 1
 k  1 1 2 1 2 3


k
k 0
j
k 1

o Ex. 6.14 (Use infinite series)
Introduction to queueing theory (Chapter 8)
o Definition: It is a class of models in which customers arrive randomly at a
service facility. Upon arrival they will wait for their turns and then will be
served. The time it takes to serve a customer is also a random function. Once
served they are generally assumed to leave immediately.
o Cost functions:
 L – The average number of customers in the system
 LQ – The average number of customers waiting in the queue
 W – The average amount of time a customer spends in the system
 WQ – The average amount of time a customer spends in the queue
o Cost identity: Average earning rate (system) = a  (average cost / customer)
N (t )
a  lim
(a is the average arrival rate of entering customers)
t 
t
o Little’s formula: Average number of customers in the system = the average
arrival rate of entering customers  the average amount of time a customer
spends in service.
L = a  W
LQ = a  WQ

o Steady-state probabilities: Pn is the limiting probability that there will be n
customers in the system. It also represents the proportion of time that n
customers in the system.
M/M/1 queue
o M/M/1 with infinite capacity:
P0 = 1 –  / ,
Pn = ( / )n (1 –  / )
L =  n Pn =  / ( – ),
W = L / a = 1 / ( – )
WQ = W – E[S] =  /  ( – ) LQ = a WQ = 2 /  ( – )
Example: On average a single-processor CPU completes 10 jobs per second and a job
arrives every 0.4 seconds. Jobs are executed immediately if the CPU is idle.
Otherwise they will be placed in the hard disk to wait for their turns according to the
order of arrivals. If arrival and completion times of jobs are exponentially distributed,
a) estimate the average CPU time for each job.
1 / 10 = 0.1 second
b) estimate the average total wait time for each job.
 = 1/0.4 = 2.5 / seconds,  = 10 / seconds, WQ =  /  ( – ) = 0.033 seconds


o M/M/1 with finite capacity (N): The condition / < 1 is unnecessary in this
case since the size of the queue is limited to N.
N 1
N


 
 1  N  
 ( N  1)  



   

L
,
    N 1 

(    ) 1   
 



L
W
a
For arrival : a  
For entering : a   (1  PN )
M/M/k queue
o Erlang loss system (no queue):
   n 
    j 
    
    
    
k     
Pn  
n = 0, 1, 2, …k


j  0  j!  
 n! 










o For a M/G/k queue (Erlang loss formula):

[E ( S )]n 
  k 
[E ( S )] j 

n = 0, 1, 2, …k
Pn  
  j  0 

n
!
j
!







 


where E[S] = 1/ is the mean service time.
o Infinite queue:
n



    j    k
    


    
    
 k 1        
k 


 j  0 
, n  k


n
!
j
!
k
!
k















Pn  







n
  
   k k
  k 
P0 , n  k

k!
Summary:
o This is a basic presentation of the theory of queueing systems.
o Other types of queueing systems include the G-types, which refer to general
distributions.
o Other complications include network of queues, open/closed queues, priority
queues, busy and idle periods, …
o Queueing theory itself is a graduate-level course.