Lecture 13 – ContinuousTime Markov Chains
Topics
• Markovian property
• Exponential distribution
• Rate matrix
• ATM Example
• Birth and death processes
• Queuing systems
Markovian Property for CTMCs
Stochastic process: {Yt : t  0 }, where Yt is a nonnegative integer
CTMC is similar to that of a DTMC except “one-step” has no
meaning in continuous time so the Markovian property must
hold for all future times instead of just for one step.
Definition 1: The process Y = {Yt : t ≥ 0 } with state space S is a
CTMC if the following condition holds for all j  S, and t, s ≥ 0
Pr{Yt+s = j | Yu , u ≤ s } = Pr{Yt+s = j | Ys }.
In addition, the chain is said to have stationary transitions if
Pr{Yt+s = j | Ys = i } = Pr{Yt = j | Y0 = i }.
Interpretation: First equation says that the conditional
distribution of the future Yt+s given the present Ys and the past
Yu, 0 ≤ u ≤ s, depends only on the present and is independent of
the past. Second equation says that Pr{Yt+s = j | Ys = i } is
independent of s.
Example of Definition 1
• Problem: Suppose that a CTMC enters state i at, say, time 0 and
does not leave during the next 15 minutes; i.e., a transition does
not occur.
• Question: What is the probability that a transition will not occur
in the next 5 minutes?
• Approach: Markovian property tells us that the probability that
the process will remain in state i during the interval [15, 20] is
just the unconditional probability that it stays in state i for at
least 5 minutes.
• Solution: Let Ti denote the amount of time that the process stays
in state i before making a transition into a different state. Then
Pr{ Ti > 20 | Ti > 15 } = Pr{ Ti > 5 }
or, in general,
Pr{Ti > s + t | Ti > s } = Pr{ Ti > t }
for all s, t ≥ 0. Hence, the random variable Ti is memoryless and
so is exponentially distributed.
Generalization of Example
The Markovian property gives
Pr{ Ti > s + t | Ti > s } = Pr{ Ti > t }
for all s, t ≥ 0.
Implication: The random variable Ti is memoryless and thus
is exponentially distributed.
Alternative definition of CTMC: A stochastic process having the
properties that each time it enters state i,
(i) the amount of time it spends in that state before making a
transition into a different state is exponentially distributed
with mean, say, 1/li , and
(ii) when the process leaves state i it next enters state j with
some probability, say, pij , where pij must satisfy
pii = 0, for all i  S
Sj pij = 1, for all i  S
ATM Example (max of 5 in system)
Statistics
• Average time between arrivals = 30 sec (0.5 min): l = 2/min
• Average service time = 24 sec (0.4 min): m = 2.5/min
Design questions
• How many ATMs should there be?
• Should the foyer be expanded?
State-transition network
a
0
a
1
d
3
2
d
d
a
a
a
5
4
d
d
Exponential Distribution
pdf: f (t ) = le–lt for t ≥ 0
Parameters: Mean = 1/l
Var = (1/l )2
CDF: F (t ) = 1 – e–lt for t ≥ 0
f(t)
0.6
0.5
0.4
0.3
0.2
0.1
0
0
0.5
1
1.5
2
2.5
3
3.5
4
t
Exponential distribution with Mean = 0.5 (l = 2)
Poisson Process
When the duration of the time between events is
exponentially distributed, the number of occurrences
of the event in a given time interval has a Poisson
distribution.
 lt
l
t
e
 
k
Pr{ k arrivals in time t } =
for k = 0, 1,…
k!
For the ATM example with l = 2, the expected
number of arrivals in the interval [0, t ] is 2t.
Rate Diagram
• For CTMCs, activities are better represented
by their rate of occurrence, so rather than
using a state-transition network we use a
rate diagram or rate network.
• This network is easily constructed from the
state-transition network by replacing the
activity designation by the activity rate.
l
ATM
Network
0
l
1
µ
µ
µ
5
4
3
2
l
l
l
µ
µ
Transient analysis 
Rate Matrix
A computationally more convenient alternative to the
rate diagram is the rate matrix R whose element rij is
the transition rate from state i to j. In general,
rij = lpij or rij = mpij
General rate
matrix
 0
 r
10
R


 rm1,0
r01
0
r02
r12
rm1,1 rm1,2
Rate matrix for
ATM example
r0,m1 
r1,m1 


0 
0
m

0
R
0
0

 0
l
0
0
0
0
l
0
0
m
0
l
0
0
m
0
l
0
0
m
0
0
0
0
m
0
0 
0

0
l

0 
Transient Analysis
• Determine the probability that the system will
be in a particular state at time t.
• The transient probabilities are a function of the
initial state.
• Unconditional probability vector:
q(t ) = (q0(t ), q1(t ), q2(t ),…,qm-1(t ))
• Requirement:
m 1
 q (t )  1
i 0
i
Transient Analysis (cont’d)
• For some small interval of time ∆, let n = t/∆ be
the number of steps or increments required to
represent t.
• The transient solution of the process can be
approximated at time t = n∆ with a DTMC by
solving the following equation:
q(n∆) = q(0)P(n) or q(n∆+∆) = q(n∆)P
where P is a state-transition matrix determined
from the rate matrix R.
Transition Matrix for Transient Analysis
Let ai be the sum of all transition rates out
of state i ; that is,
m 1
ai   rij
and let pij  rij. Then
j 0
r01
r02
1  a 0
 r
1


a

r
10
1
12

P


 rm1,0 rm1,1 rm1,2


r0,m1 
r1,m1 



 1  a m1 
Transition Matrix for ATM Example
 Rate network
l
0
0
0
0 
1  l
 m 1  (l  m )


l
0
0
0


 0
m
1   (l  m )
l
0
0 
P

0
0

m
1


(
l

m
)

l
0


 0
0
0
m
1   (l  m )
l 


0
0
0
0

m
1


m


2
0
0
0
0 
1   2
  2.5 1   4.5


2
0
0
0


 0
 2.5 1   4.5
2
0
0 


0
0

2.5
1


4.5

2
0


 0
0
0
 2.5 1   4.5
2 


0
0
0
0

2.5
1


2.5


Transient Analysis for ATM Example
• Assume system is empty at t = 0.
• We wish to approximate the transient probabilities
at t = 1 min.
• Initial probability vector: q(0) = (1, 0, 0, 0, 0, 0)
• Use equation q(n∆) = q(0)P(n)
• Number of steps: n = t/∆ = 1/∆
– Case 1: ∆ = 0.05  n = 20 steps (1 min)
q(20∆) = q(1) = (0.433, 0.291, 0.162, 0.075, 0.029, 0.011)
– Case 2: ∆ = 0.025  n = 40 steps (1 min)
q(40∆) = q(1) = (0.435, 0.291, 0.160, 0.073, 0.029, 0.011)
(almost identical)
Transient Solution for ATM Example
(∆ = 0.05, 0  t  1)
1
0.8
Sta te 0
Sta te 1
Sta te 2
Sta te 3
Sta te 4
Sta te 5
0.7
0.6
0.5
0.4
0.3
0.2
Steps
20
18
16
14
12
10
8
6
4
0
2
0.1
0
Transient probabilities
0.9
Steady-State Solutions
• Definition: The probability that the system is in state
i is constant (independent of initial conditions).
• Steady-state probability for state i : iP = limtq(t )
• Vector:  P   1P ,  2P ,...,  mP 1 
• Calculations in Chapter 15: must solve m
simultaneous linear equations in m unknowns.
• ATM example:
– After 1 min with ∆ = 0.25
q(1) = (0.435, 0.291, 0.160, 0.073, 0.029, 0.011)
– In the limit
P = (0.271, 0.217, 0.173, 0.139, 0.111, 0.089)
Transient Computations for ATM
Example with ∆ = 0.025
Steps, n
Tim e
(mi n)
q0
q1
q2
q3
q4
q5
0
0
1
0
0
0
0
0
40
1
0.435
0.291
0.160
0.073
0.029
0.011
80
2
0.348
0.258
0.175
0.110
0.066
0.042
120
3
0.311
0.239
0.175
0.124
0.087
0.063
160
4
0.292
0.228
0.175
0.131
0.098
0.075
200
5
0.282
0.223
0.174
0.135
0.104
0.082
240
6
0.277
0.220
0.174
0.137
0.107
0.085
M
M
M
M
M
M
0.271
0.217
0.173
0.139
0.111
0.089
Steady
state

System Statistics
• Provide managerial insights
• Evaluate system performance and quality
of service
• Evaluate design options
• ATM Example:
– Proportion of time ATM is idle:  0P  0.271
– Efficiency (proportion of time busy): 1   0P  0.729
P
– Proportion of customers rejected:  5  0.089
P
P
– Proportion of customers who wait: 1   0   5  0.64
– Expected number in system:
P
i

i0 i  1.868
5
ATM Example (cont’d)
– Expected number in queue:
P
i

1



i1
i  1.139
5
– Throughput rate (average number passing


through the system): l 1   5P  1.822
– Balking rate (average number of customers
lost): l 5P  0.178
– Average time in system (given by Little’s law):
average number in system 1.868

 1.025 min
throughput rate
1.822
ATM Design Alternatives
• Performance summary (contradictory?)
– Busy 73% of time
– Space in foyer less than 40% utilized; that is,
(average no. in systems / 5)  100% = 37.36%
– 9% of customers lost
– Average wait in queue = 60(1.139/1.822) = 37 sec
• Options
– Add machines
– Expand size of foyer
– Add human teller
Add More ATMs
Rate diagram
for 3 ATMs:
l = 2, m = 2.5
Comparative
analysis
l
0
l
1
µ
2µ
5
4
3
2
l
l
l
3µ
3µ
3µ
Alternative
System measures
ATM1
ATM2
ATM3
Number of machines
1
2
3
Capacity of foyer
5
5
5
Average number in system
1.8683
0.9187
0.8134
Average time in system
1.0252
0.4635
0.4078
Average number in queue
1.1394
0.1258
0.0156
Average time in queue
0.6252
0.0635
0.0078
Throughput rate
1.8224
1.9823
1.9946
Efficiency (utilization)
0.7289
0.3965
0.2659
Proportion who must wait
0.7289
0.224
0.0511
Proportion of customers lost
0.0888
0.0088
0.0027
Add Human Teller
• Performance
– Average service rate for teller: m1 = 1/min
– Average service rate for ATM: m2 = 2.5/min
– Arrival rate: l = 2/min
• Two-server queuing system
– Indices: teller = 1; ATM = 2
• State variables: s = (s1, s2, s3)
0 if server i is idle
si  
for i = 1,2
1 if server i is busy
s3  number in queue
Add Human Teller (cont’d)
• Events
– Arrival = a
– Service completion for teller = d1
– Service completion for ATM = d2
• State-transition network
(100)
d1
0
(000)
2
d2
d1
a
d2
a
1
a
d1, d 2
3
(110)
d1, d 2
4
a
(111)
d1, d 2
5
a
(112)
6
a
(113)
(010)
• Explanation: s = (110); teller and ATM are
busy, no customers are waiting.
Add Human Teller (cont’d)
• Event rates
– Arrival: l = 2/min
– Service completion for teller: m1 = 1
– Service completion for ATM: m2 = 2.5
• Rate diagram
(100)
µ
0
(000)
l
2
1
µ2
l
µ2
3
µ1
1
(010)
µ1 + µ2
l
(110)
µ1 +µ2
4
l
(111)
µ1 + µ2
5
l
(112) l
6
(113)
Add Human Teller (cont’d)
Rate matrix R = (rij)
where rij = transition rate from state i to state j
(000) (010) (100) (110) (111) (112) (113)
R=
(000)
0
0
2
0
0
0
0
(010)
2.5
0
0
2
0
0
0
(100)
1
0
0
2
0
0
0
(110)
0
1
2.5
0
2
0
0
(111)
0
0
0
3.5
0
2
0
(112)
0
0
0
0
3.5
0
2
(113)
0
0
0
0
0
3.5
2
Explanation: r43 = m1 + m2 = 1 + 2.5 = 3.5
where state 4 = (111) and state 3 = (110)
Comparisons For ATM Example
Alternative
System measures
ATM1
ATM and teller
ATM2
Average number in system
1.8683
1.580
0.9187
Average time in system
1.0252
0.8213
0.4635
Average number in queue
1.1394
0.3659
0.1258
Average time in queue
0.6252
0.1902
0.0635
Throughput rate
1.8224
1.9234
1.9823
ATM Efficiency (utilization)
0.7289
0.4731
0.3965
Teller efficiency (utilization)
––
0.7408
––
Proportion who must wait
0.7289
0.4275
0.224
Proportion of customers lost
0.0888
0.0383
0.0088
Steady-state solution for human teller:
s =
(000)
(010) (100)
(110)
(111) (112) (113)
πP = (0.214, 0.046, 0.313, 0.205, 0.117, 0.067, 0.038)
Pure Birth Processes
Example: Hurricanes
0
1
a0
a1
2
a2
3
4
a3
…
Rate matrix
Properties
(let li be arrival
rate for state i )
• Markov process if time between arrivals
has exponential distribution
0 l0
0 0

R  0 0

0 0

0
l1
0
0
0
0
l2
0






• No steady state [transient probabilities
are governed by Poisson distribution:
pk(t ) = (lt )ke-lt/k !, k = 0, 1, 2, … ]
• Probability of N (t ) arrivals in time t is  n:
Pr{ N (t )  n } =
k 0  lt 
n
k
e  lt k !
Pure Death Processes
Examples
• Delivery of packages
• Completion of 10 course study units
0
d1
1
d2
2
d3
Rate matrix
• Let mi be completion rate for state i
• State space S = (0,1,…,10}
Steady state probability vector:
πP = (1,0,…,0)
State 0 is an absorbing state
3
1
m
 1
0
R
0


 0
d4
4
…
0
0

0
0
0

0
m2
0

0
0
m3 
0
0
0
 m10
0
0 
0

0


0 
Pure Death Process Example
• Assume all units have the same completion rate:
rk,k–1 = µk = µ, k = 1,…,10
• Then transient probabilities are:
p10–k(t ) = (mt )ke-mt/k !, 0  k < 10, and
p0(t ) = 1 – p1(t ) – · · · – p10(t )
• Let m = 1 completions per week
• Probability of completing k units in t = 14 weeks:
Pure Death Process Example (cont’d)
• Transient probabilities for k units remaining:
pk(t ) = (mt )10–ke-mt/(10–k) !, 0  k < 10,
and
p0(t ) = 1 – p1(t ) – · · · – p10(t )
• Let m = 1 completions per week
• Probability of k units remaining in t = 14 weeks:
Incomplete units, k
0
1
2
3
4
5
6
7
Probabili ty, pk(14)
0.891
0.047
0.03
0.017
0.009
0.004
0.001
0.0005
General Birth and Death Processes
Examples
• Repair shop for a taxi company
• Intensive care unit in hospital (turnover of nurses)
d2
d1
0
a1
Rate matrix
• Assume 7 states
• Typically, m and l
depend on state
• Steady state
probabilities, P,
will exist
d4
2
1
a0
d3
3
4
a3
a2
0
m
 1
0

R0
0

0
0

l0
0
0
l1
0
0
m2
0
l2
0
0
0
0
0
0
0
m3
0
l3
0
0
0
0
0
0
0
m4
0
l4
0
0
m5
0
0
m6
0
0 
0

0
0

l5 
0 
…
Queuing Systems
Input
source
Customers
Queue
Service
mechanism
Departures
Queue Discipline: Order in which customers are served; FIFO,
LIFO, Random, Priority
Five Field Notation:
Arrival distribution / Service distribution / Number of servers /
Maximum number in the system / Number in the calling population
Queuing Notation
Distributions (interarrival and service times)
M = Exponential
D = Constant time
Ek = Erlang
GI = General independent (arrivals only)
G = General
Parameters
s = number of servers
K = Maximum number in system
N = Size of calling population
Characteristics of Queues
Infinite queue: e.g., Mail order company (GI/G/s)
d
0
2d
1
a
2
…
sd
sd
s –1
a
s
a
s +1
…
a
Finite queue: e.g., Airline reservation system (M/M/s/K)
…
sd
K–1
K
a
a. Customer arrives but then leaves
a
…
sd
K–1
K
a
b. No more arrivals after K
Characteristics of Queues (continued)
Finite input source: e.g., Repair shop for taxi company (N vehicles)
with s service bays and limited capacity parking lot (K – s spaces).
Each repair takes 1 day (GI/D/s/K/N).
d
0
2d
1
Na
…
sd
s+1
s
(N–s+1)a
…
(N–1)a
sd
s–1
2
(N–s)a
…
sd
N
N–1
a
In this diagram N = K so we have GI/D/s/K/K system.
Single Channel Queue – Two Kinds of Service
Bank teller: normal service (d ), travelers checks (c ), idle (i )
Let p = portion of customers who buy travelers checks after
normal service
s1 = number in system, where s1  { 0, 1, 2, . . . }
s2 = status of teller, where s2  {i, d, c }
s = (s1, s2)
Statetransition
network
(1,c)
c
d, p
a
(3,c)
d, p
d, 1– p
d, 1– p
(2,d)
a
…
c
d, p
(1,d)
a
(2,c)
c
d, 1– p
(0,i)
a
(3,d)
a
…
Single Channel Queue for Bank (cont’d)
• State transitions w.r.t. customer departures from teller
– Current state: s = ( j, d ), j = 1, 2,… (teller busy)
– Next state: either
s' = ( j –1, d ), departure with probability 1 – p, or
s' = ( j, c ), get checks with probability p
• State transitions w.r.t. customer departures after
purchasing travelers checks
– Current state: s = ( j, c ), j = 1, 2,… (customer buying checks)
– Next state: s' = ( j –1, d ), departure with probability 1
• State transitions w.r.t. customer arrivals
– Current state: s = ( j, d or c), j = 1, 2,… (teller or checks busy)
– Next state: s' = ( j +1, d or c), arrival with probability 1
Single Channel Queue for Bank (cont’d)
• Rate of transitions
–
–
–
–
Event: x (arrival or departure)
Rate of event x : gx (where ga = l, gd = m1, gc = m2)
Conditional probability: p(s,s' | x ) or p(i, j|x )
Computations: rij = gxp(i, j |x )
• States -- assume limited no. customers at teller: K = 2
s0 = (0,i ), s1 = (1,d ), s2 = (1,c ), s3 = (2,d ), s4 = (2,c )
• Rate matrix
0
l

(1  p) m
0
1

R   m2
0

0
(1  p) m1


0
0
0
0
pm1 l
0
0
0 l
m2 0
0  (0, i)
0  (1, d )
l  (1, c)

pm1  (2, d )
l  (2, c)
Part Processing with Rework
• Consider a machining operation in which there is a
0.4 probability that upon completion, a processed
part will not be within tolerance.
• Machine is in one of 3 states [ s = { (0), (1), (2) } ]:
0 = idle
1 = working on part for first time
2 = reworking part
Events
State-transition network
a = arrival
d1 = service completion
from state 1
d2 = service completion
from state 2
d1, 0.4
a
(0)
(1)
d1, 0.6
d2
(2)
Classification of States
Accessible: Possible to go from state i to state j (path exists in
the network from i to j).
d2
d1
0
2
1
a1
a0
0
a0
d3
1
a1
d4
3
a2
…
4
…
a3
a2
2
4
3
a3
Two states communicate if both are accessible from each other. A
system is irreducible if all states communicate.
State i is recurrent if the system will return to it some time in the
future after leaving it.
If a state is not recurrent, it is transient.
What You Should Know
About Markov Chains
• Definition of a CTMC.
• What the difference is between a DTMC and
a CTMC.
• What the rate matrix and rate diagram are.
• What is meant by a transient solution
• What is meant by a steady-state solution.
• What a birth-death process is.
• Classification of the various types of
queuing systems.