LECTURE 16
Checkout counter model
• Discrete time n = 0, 1, . . .
Markov Processes – I
• Customer arrivals: Bernoulli(p)
• Readings: Sections 7.1–7.2
– geometric interarrival times
• Customer service times: geometric(q)
Lecture outline
• Checkout counter example
• “State” Xn:
time n
• Markov process definition
number of customers at
• n-step transition probabilities
• Classification of states
1
0
Finite state Markov chains
2
.. .
3
9
10
n-step transition probabilities
• State occupancy probabilities,
given initial state i:
• Xn: state after n transitions
– belongs to a finite set, e.g., {1, . . . , m}
rij (n) = P(Xn = j | X0 = i)
– X0 is either given or random
• Markov property/assumption:
(given current state, the past does not
matter)
Time 0
Time n-1
Time n
1
pij = P(Xn+1 = j | Xn = i)
i
p 1j
k
r ik(n-1)
p kj
j
...
= P(Xn+1 = j | Xn = i, Xn−1, . . . , X0)
...
r i1(n-1)
r im(n-1)
p mj
• Model specification:
m
– identify the possible states
– Key recursion:
– identify the possible transitions
rij (n) =
– identify the transition probabilities
m
!
k=1
rik (n − 1)pkj
– With random initial state:
P(Xn = j) =
m
!
i=1
1
P(X0 = i)rij (n)
Example
0.5
Generic convergence questions:
• Does rij (n) converge to something?
0.8
0.5
0.5
2
1
n=1
1
n odd: r2 2(n)=
n=2
3
1
0.2
n=0
0.5
2
1
n = 100
n = 101
n even: r2 2(n)=
• Does the limit depend on initial state?
r11(n)
r12(n)
0.4
r21(n)
1
r22(n)
0.3
r1 1(n)=
r3 1(n)=
r2 1(n)=
Recurrent and transient states
• State i is recurrent if:
starting from i,
and from wherever you can go,
there is a way of returning to i
• If not recurrent, called transient
3
4
6
5
1
2
7
8
– i transient:
P(Xn = i) → 0,
i visited finite number of times
• Recurrent class:
collection of recurrent states that
“communicate” with each other
and with no other state
2
2
0.3
3
4
MIT OpenCourseWare
http://ocw.mit.edu
6.041 / 6.431 Probabilistic Systems Analysis and Applied Probability
Fall 2010
For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.