Limiting probabilities
Pj  lim Pij (t )
t 
 lim Pij' (t )  lim   k  j qkj Pik (t )  v j Pij (t ) 
t  

t 
  k  j qkj Pk  v j Pj
  k  j qkj Pk  v j Pj  0  v j Pj   k  j qkj Pk
The limiting probabilities can be obtained by solving the following
system of equations:

k j
qkj Pk  v j Pj  0

j
Pj  1
v j Pj  rate at which the process leaves state j

k j
qkj Pk  rate at which the process enters state j
v j Pj   k  j qkj Pk
 rate out of state j  rate into state j
When do the limiting probabilities exist?
The limiting probabilities Pj exist if
(a) all states of the Markov chain communicate (i.e., starting in
state i, there is a positive probability of ever being in state j, for
all i, j and
(b) the Markov is positive recurrent (i.e, starting in any state, the
mean time to return to that state is finite).
The M/M/1 queue
l
0
1
m
State
0
1
2
n 1
l
l
2
m
3
m
rate out of state j  rate into state j
l P0  m P1
(l  m ) P1  m P2  l P0
(l  m ) P2  m P3  l P1
(l  m ) Pn  m Pn 1  l Pn 1
The M/M/1 queue
By adding to each equation the equation preceding it, we obtain
2
l P0  m P1
l P1  m P2
l P2  m P3
n 1
l Pn  m Pn 1
0
1
The M/M/1 queue
Solving in terms of P0 yields
0
P1  (l / m ) P0
1
P2  (l / m )2 P0
2
P3  (l / m )3 P3
n 1
Pn  (l / m ) n P0
The M/M/1 queue
Using the fact that


n0
Pn  1, we obtain
P0  P0  n 1 (l / m ) n  1

 P0 
1
1   n 1 (l / m ) n

1
l
m
 Pn   l / m  (1  l / m )
n
Note that we must have l / m  1.
The expected number in the system
E (n)   n 0 nPn   n 0 n  l / m  (1  l / m ) 

(use the fact

n
nx
 n 0 

n
x
)
2
(1  x)
l
m l
The birth and death process
l0
0
1
m1
State
0
1
2
n 1
l1
l2
2
m2
3
m3
rate out of state j  rate into state j
l0 P0  m1 P1
(l1  m1 )P1  m2 P2  l0 P0
(l2  m2 ) P2  m3 P3  l1 P1
(ln  mn )Pn  mn 1 Pn 1  ln 1 Pn 1
By adding to each equation the equation preceding it, we obtain
0
1
2
n 1
l0 P0  m1 P1
l1 P1  m2 P2
l2 P2  m3 P3
ln Pn  mn 1 Pn 1
Solving in terms of P0 yields
0
P1  (l0 / m1 ) P0
1
P2  (l1 / m2 ) P1  (l1l0 / m 2 m1 ) P0
2
P3  (l2 / m3 ) P2  (l2 l1l0 / m3 m2 m1 ) P0
n 1
Pn  (ln 1 / mn ) Pn 1  (ln 1ln  2 ...l1l0 / m n m n 1...m 2 m1 ) P0
Using the fact that


n0
Pn  1, we obtain
ln 1ln  2 ...l1l0
P0  P0  n 1
1
mn mn 1 ...m 2 m1

 P0 
1
ln 1ln  2 ...l1l0
1   n 1
mn mn 1 ...m 2 m1
ln 1ln  2 ...l1l0
 Pn 

 ln 1ln  2 ...l1l0 
mn mn 1 ...m 2 m 1   n 1

m
m
...
m
m
n n 1
2 1 


Note that we must have
ln 1ln  2 ...l1l0
 n1 m m ...m m  .
n n 1
2 1

The M/M/m queue
The M / M / k queue is a birth and death process with
ln  l
 nm
mn  
 km
if n  k
if n  k
 P0 
1
l l ...l l
1   n 1 n 1 n  2 1 0
mn mn 1...m2 m1


1

ln
ln
1   n 1
  nk nk
n
n!m
k k !m n
k 1
The M/M/m queue
 (l / m ) n
P0 ,

 n!
 Pn  
n
(
l
/
m
)

P0
nk

 k k!
nk
nk
 The stability condition is l / k m  1 .
The M/M/m queue
The expected number of customers in a M / M / k queue
(l / m ) k 
 E (n)   n  0 nPn  l / m 
P where   l / k m .
2 0
k !(1   )

A machine repair model
A system with M machines and one repairman.
The time between machine is exponentially
distributed with mean 1/l. Repair times are also
exponentially distributed with mean 1/m. What is
the average number of working machines? What is
the fraction of time each machine is in use?
The machine repairman problem
The system is in state n if there are n failed machines
 n  0,1, 2,..., M
The machine repairman problem
The system is in state n if there are n failed machines
 n  0,1, 2,..., M
The corresponding process is a birth and death process with
if n  M
( M  n )l
ln  
if n  M
0
mn  m ,
n 1
The machine repairman problem
The system is in state n if there are n failed machines
 n  0,1, 2,..., M
The corresponding process is a birth and death process with
if n  M
( M  n )l
ln  
if n  M
0
mn  m ,
n 1
 P0 
1
ln 1ln  2 ...l1l0
1   n 1
mn mn 1...m2 m1

M

1   n 1
1
1   n 1 (l / m ) n M !/( M  n)!
M
M
1
M l ( M  1)l...( M  n  1)l
mn
The machine repairman problem
ln 1ln  2 ...l1l0
 Pn 
P0
mn mn 1 ...m2 m1
=
(l / m ) n M !/( M  n)!
1   n 1 (l / m ) M !/( M  n)!
M
n
,
n  0,1,..., M
The machine repairman problem
ln 1ln  2 ...l1l0
 Pn 
P0
mn m n 1 ...m 2 m1
=
(l / m ) n M !/( M  n)!
1   n 1 (l / m ) M !/( M  n)!
M
n
,
n  0,1,..., M
n
n
(
l
/
m
)
M !/( M  n)!
 n1
M
 E (n)   n 1 nPn 
M
1   n 1 (l / m ) n M !/( M  n)!
M
The machine repairman problem
P{machine is working}   n  0 P{machine is working | n not working}Pn
M
The machine repairman problem
P{machine is working}   n  0 P{machine is working | n not working}Pn
M
M n
=  n0
Pn
M
M

1
M
n0
M
nPn
 1  E ( n) / M
The automated teller machine (ATM)
problem
Customers arrive to an ATM according to a Poisson
process with rate l. If the customer finds more than
N other customers at the machine, he/she does not
wait and goes away. Machine transaction times are
exponentially distributed with mean 1/m. What is
the probability that a customer goes away? What is
the average number of customers at the ATM? If the
machine earns $h per customer served, what is the
average revenue the machine generates per unit
time?
The M/M/1/N queue
State
0
1  n  N -1
N 1
rate out of state j  rate into state j
l0 P0  m1 P1
(l  m ) Pn  m Pn 1  l Pn 1
m PN  l PN 1
The M/M/1/N queue
Pn  (l / m ) n P0
 n0 Pn  1 P0 
N
1 l / m
1  (l / m ) N 1
(l / m ) n (1  l / m )
 Pn 
1  (l / m ) N 1
The M/M/1/N queue
Pn  (l / m ) n P0
1 l / m
 n0 Pn  1 P0  1  (l / m ) N 1
N
(l / m ) n (1  l / m )
 Pn 
1  (l / m ) N 1
1 l / m


n
 E (n)   n  0 nPn 
n
(
l
/
m
)

1  (l / m ) N 1 n  0
l[1  N (l / m ) N 1  ( N  1)(l / m ) N ]

( m  l )[1  (l / m ) N 1 ]
The M/M/1/N queue
(l / m ) N (1  l / m )
The probability that a customer goes away is PN 
1  (l / m ) N 1
The average number of customers at the ATM is E (n)
The average revenue per unit time (revenue rate) is l (1  PN )h
The production inventory problem
Consider a production system that manufacturers a single
product. Production times are exponentially distributed
with mean 1/m. The production facility can produce ahead
of demand by holding finished goods inventory. Orders
from customers arrives according to a Poisson process
with rate l. If there is inventory on-hand, the order is
satisfied immediately. Otherwise, the order is
backordered. The production system incurs a holding
cost $h per unit of held inventory per unit time and a
backorder cost $b per unit backordered per unit time. The
production system manages its finished goods inventory
using a base-stock policy with base-stock level s.
The production inventory problem
• What is the expected inventory level?
• What is expected backorder level?
• What is the expected total cost?
• What is the optimal base-stock level?
Three basic processes
I: level of finished goods inventory
B: number of backorders (backorder level)
IO: inventory on order.
Three basic processes
Under a base-stock policy, the arrival of each
customer order triggers the placement of an
order with the production system
 s = I + IO – B
 s = E[I] + E[IO] – E[B]
Three basic processes
I and B cannot be positive at the same time
 I = max(0, s - IO) = (s – IO)+
 E[I] = E[(s – IO)+]
 B = max(0, IO - s) = (IO - s)+
 E[B] = E [(IO - s)+]
The production system behaves like an M/M/1 queue, with
IO corresponding to the number of customers in the system.
 Pr( IO  n)   n (1   )
 E[ IO] 

1 
Expected backorder level
E[ B ]  E[max(0, IO  s )]
  n  s ( n  s) n (1   )

  n  0 n n  s (1   )


s
n
n

 n 0 (1   )
 s 1

1 

Expected inventory level

 s 1
E[ I ]  s  E ( IO )  E[ B ]  s 

1  1 
Expected cost
z ( s ) : expected cost (holding cost + backorder cost)

 s 1
 s 1
 z ( s )  hE[ I ]  bE[ B]  h( s 

)b
1  1 
1 
Optimal base-stock level

r
r s2
r s2  
r
r s 1
r s 1 
z ( s  1)  z ( s )   h( s  1 

)b

)b
   h( s 
0
1 r 1 r
1 r  
1 r 1 r
1 r 

r s  2 r s 1
r s  2 r s 1
 h(1 

)  b(

)0
1 r 1 r
1 r 1 r
r s 1
h
(1  r )(h  b)  0
1 r
 h  r s 1 (h  b)  0
 r s 1 
h
hb
Optimal base-stock level
h
hb
h
ln
h  b 1
s
ln r
h
h

 
ln
ln

 
h

b
hb
 s*  
 1  
 ln r
  ln r

 
 r s 1 





If we ignore the integrality of s*, then s* 
ln
h
hb .
ln r