196 4 : Markov chains in continuous time
rir_._;_,(ii),,•When Ai = AJ• = Ao, j > 0, and /to = 0, show that
ei •-•=-1-j(l +1)X0-1.
(This is a random walk, with reflection at O.)
(iii) When Ai =
=
Agi + 1), j > 0, and AO = 0, show that
ej + 1 - ( j - 1 ) +
2 )
+
+
(This is simple birth—death, with reflection at O.)
Exercise 1
,k•
(b)* Visits. Let X(t) be a continuous-time simple random walk on the
integers, which is to say that pn,n+i(h) = Ah + o(h)/5 nr n,n--1(h)
p,h + o(h ). Find the distribution of the time V spent visiting r 0
on an excursion from O.
om
webtaiwwomakiuilimileAusimis,..hgiii3>k,,
• •••A;,,
4.6 Stationary distributions and the long run
We have seen several examples in which transition probabilities p jk(t) converge as t —› oo, of which the simplest is the flip—flop process. Here, from
(4.2.11), go = A, ql = p,, and as t —› oo, for j E 10, 11,
(1)
Pil
P
j
A-Fau
o
•
A + p ,
We consider the general consequences of convergence, but for simplicity we
first confine ourselves to a finite state space.
(2) Equilibrium. Suppose that, for all j and k in the finite set S,
lim pjk(t)
t-3oo
= Zic•
Then It has a number of key properties.
(i) Allowing t +> oo in the Chapman—Kolmogorov identity P(t + s) =
P(t)P(s) we find from any row (all being the same) that
(3)
Jr = irP(s),
for all s. That is to say, It is an equilibrium or stationary distribution.
4.6 Stationary distributions and the long run 2 0 1
In this case the queue is non-null recurrent, and as t o o ,
P(X(t) n ) 7 r n , n > O.
If A = p, then we glance at the jump chain to see that X(t) is null-recurrent,
and if A > w e find likewise that X(t) is transient. In either case, as t -± co,
P(X(t) n ) —)• 0, n > O.
Because we know that go -,- A and qn ( A + p,), n > 1, (14) also
gives us the mean recurrence times AO = 1/(7rogo) p l ( g p , —A)), and
= (tti(P,2 A2))(PIA)n, when A < a .
Of course the jump chain has a different stationary distribution, v, given
for A < /2, by
tin =
VO '
att 2
n
-
1
2,p,
—
A natural question is to ask how long an arriving customer can expect to
wait, when the queue is in equilibrium. It is left as an exercise for you to
find the distribution of the time such a customer spends in the queue, and
to show that the expected time spent in the queue is (,u - A ) l . Of course
the time spent waiting to be served is therefore,
1
1
A
A a u au(11, — A)'
because the arriving customer's expected service time is A-1.
,•••7,-.
•
1Exercises
(a)* Let X(t) be a simple birth-death process, with A j A , =
for j > 1, and A.0 A . Find the stationary distribution of X(t),
Exercise 2 and also of the jump chain, when they exist. Explain the difference
in your answers.
(b) Erlang's loss formula. Suppose there are s servers, each of whose
service times are independent Exponential (p,). Customers arrive
at the instants of a Poisson process of intensity A, but leave imme-A
diately i f all servers are busy. Show that, in equilibrium, t e
204 4 : Markov chains in continuous time
(8) Departures from WM/s. If X(t) is the number of customers present at
time t for the MIMIs queue in equilibrium, then the departures up to t form
a Poisson process of rate A independent of X(t).
This has some even more interesting consequences that we sketch in the
next section.
We conclude here with another example of the amenability of reversible
chains.
(9) Censored state space. Suppose a reversible chain X(t) with equilibrium
distribution Jr is running on a state space S. Now it is decided that the chain
is forbidden to enter a subset C of S. That is to say, any attempted move
to k E Cc is allowed to succeed, but if the chain attempts to move to a
state in C, then it simply remains where it is. The effect of this is that for
E Cc S \ C ,
ik = 0,
k
E C.
Then the new chain 17 (t) is still reversible, with stationary distribution ijj 0(
E C . In detail we have
(10)
q
f
=7J/ 7jEs\cr i ) •
Proof. It is trivial to check that 7/ satisfies the detailed balance equations, so
the result follows from (5).
0
(11) Shared waiting room. Suppose that two independent queues are forced
to share a waiting room with capacity w, where the first queue is MJM/s
and the second is M/M/r. A customer who arrives to find the waiting room
full (containing w customers waiting for service) leaves immediately never
to return.
By Exercise (4.7(d)) the queues are jointly reversible in equilibrium, with
stationary distributions ir and v, say. The finite shared waiting room truncates the state space to the set R of states such that x < s + w, y < r + w,
a n d x + y < s + r + w.
By the previous example, the equilibrium distribution is
7rfk = n-jVk I EI 0
kExercises
Exercise 3
,k)ECr t
jVk .
0
At.
(a)* Let X(t) be an irreducible non-null recurrent birth—death process:'
Show that X (t) is reversible in equilibrium..-?,:-
4,8 Queues 2 0 5
Exercise 3
r ( b ) Kolmogorov's criterion. Show that X(t) is reversible if and only if
for any sequence • • • in, j5 we have
= q(f,i„), • • • D •
(c)* Show that an irreducible Markov chain X(t) with two states is
reversible in equilibrium.
, (d)* I f X(t) and Y (t) are independent reversible processes, prove that
(X(t),Y (t)) is a reversible process.
1111111111101111.1111116
4.8 Q u e u e s
We have seen several individual queues in previous sections, but in the
real world we are often faced with the prospect of moving through several
queues in succession. For example, if you telephone a faceless (or indeed,
any) corporation seeking to speak to someone who has some relevant knowledge, you will first spend a random time conversing with a computer, then
join a queue on hold to speak to a switchboard operator, and then at least
one further queue on hold before succeeding in your quest.
We first consider a simple version of such nightmares.
(1) Tandem queue. There are two single-server queues, whose service times
are independent and exponentially distributed with parameters p,i and A2,
respectively. Customers arrive at the first queue as a Poisson process of
rate A, where A < Al A pl. On completing that service, they move immediately to the second queue. On completing the second service, they leave
the system forever.
Let the total number in each queue be Xi (t) and X2(t), respectively. When
the system is in equilibrium, it is remarkable that
(i) X i (t) and X2(t) are independent at any time t.
(ii) For any given customer, the total time spent in the first queue is
independent of the total time spent in the second.
(iii) For any given customer, the total times spent waiting before service in
each queue are not independent.
Proof.
(i) In the previous section (4.7.8), we showed that the departure process
from the first queue up to time t is a Poisson process of rate A, and
1
310 6 : Hints and solutions
4.4(a)
(i) I f X(t) spends only a finite amount o f time visiting j , then
P(X(nh). j) < oo, and j is transient for Xn.
(ii) The event that Xn = j includes the possibility that the process X(t) I
arrives in j during ((n — 1)h,nh) and remains there until time nh. I
Hence,
P(Xn I
Xo= l) > Pii(t)e—qjh,
(n — 1)h < t < nh.
Hence, since all terms are positive,
00
00
p i i ( t ) d t h e x p ( h q i ) p q n = i t xo = l),
and the result follows.
4.5(b) The distribution of V is a mixture of an exponential density and 1
an atom of probability at zero. To be specific, let 0 be the probability
that the walk visits r before returning to 0, and 0 the probability that
the walk, started at r, revisits r ever without a visit to O. Then the
number of visits to r before any visit to 0 is geometric with parameter
(1 — 0), and by Exercise 4.2(e) the total time spent in r is Exponential
t(A + p,)(1— 0)), because each holding time is Exponential (X +A). This
outcome occurs at all with probability 9; otherwise P(V = 0) = 1—O.
Elementary calculations from Section (2.2) yield
Solution exercise 1
—P
(q p ) r — 1 + p A q
I
and
=
q
t
(
q
/
p
)
r
i
1
)
/
1
(
q
/
p
)
r
1
1
+
p
A
q
.
where p = AAA. + p , / (A + p,), p A q m i n f p , q ) , and p • . .
The case p = - 1 is left for you.
4.6(a) We find the stationary distribution of X(t) by solving irQ = 0,
which yields the set of equations
Solution exercise 2
-
+1),turi+1 —
+
+
1)AJT1---1 = 0, 1 2 , *
,
Hints and solution!
with 211,72—( A + 11)7r1 +
to show that
= 0 = n'l p, A 7 0 , and E r i = 1. It is easy
7ro
••
JfLJ
which yields a stationary distribution if E T 1(Xiau)j(1/f) < oo, that is,
if < tu•
The jump chain is simply a random walk reflected at the origin, and
it is easy to find that when A < p, there is a stationary distribution of
I., the form
+
=
>1
ItJ
whence
x2
— 1
j 1,
— 2/12
;To =
itt
•
2,u
The difference arises naturally because the holding times of X(t) depend
['on the state of the process. From (4.6.6) we know
A
j 0(
=
( j )--(- X + 1,07
1
)
and
fro oc 7rogo = 7r0ANow we set Eft,/ = 1, yielding fro = (u,—A,)/2p, as above. Incidentally)
no-1 = 1 — log (1 — (A/g)).
' 4.7(a) The equations r Q = 0 take the form
rcise 3
—irA = gituf —
1
with 7r1p,1 = r0A0. Therefore, n-j p j = 71-j_lAj_15 and m exists,
so X(t) is reversible.
(c) I f gol = a and gm = 6, it is easy to check that 7t0 =
+
0)
and 7r1 = a/(a + fi). So n-Ogol = 71,710, and the result follows.
(d) Because the detailed balance equations hold for X and Y) and
these are independent, i t is straightforward to see that the
detailed balance equations for (X, Y) are satisfied. In an obvious
notation
a d i b i b b o u s . . 71-x(i)n-Y(m)qx(i, ( m , n) = x ( D g Y (n)qx(i O q y
m
310 6 : Hints and solutions
4.4(a)
(i) I f X(t) spends only a finite amount o f time visiting j , then
P(X(nh). j) < oo, and j is transient for Xn.
(ii) The event that Xn = j includes the possibility that the process X(t) I
arrives in j during ((n — 1)h,nh) and remains there until time nh. I
Hence,
P(Xn I
Xo= l) > Pii(t)e—qjh,
(n — 1)h < t < nh.
Hence, since all terms are positive,
00
00
p i i ( t ) d t h e x p ( h q i ) p q n = i t xo = l),
and the result follows.
4.5(b) The distribution of V is a mixture of an exponential density and 1
an atom of probability at zero. To be specific, let 0 be the probability
that the walk visits r before returning to 0, and 0 the probability that
the walk, started at r, revisits r ever without a visit to O. Then the
number of visits to r before any visit to 0 is geometric with parameter
(1 — 0), and by Exercise 4.2(e) the total time spent in r is Exponential
t(A + p,)(1— 0)), because each holding time is Exponential (X +A). This
outcome occurs at all with probability 9; otherwise P(V = 0) = 1—O.
Elementary calculations from Section (2.2) yield
—P
(q p ) r — 1 + p A q
I
and
=
q
t
(
q
/
p
)
r
i
1
)
/
1
(
q
/
p
)
r
1
1
+
p
A
q
.
where p = AAA. + p , / (A + p,), p A q m i n f p , q ) , and p • . .
The case p = - 1 is left for you.
4.6(a) We find the stationary distribution of X(t) by solving irQ = 0,
which yields the set of equations
-
+1),turi+1 —
+
+
1)AJT1---1 = 0, 1 2 , *
,
Hints and solution!
with 211,72—( A + 11)7r1 +
to show that
= 0 = n'l p, A 7 0 , and E r i = 1. It is easy
7ro
••
JfLJ
which yields a stationary distribution if E T 1(Xiau)j(1/f) < oo, that is,
if < tu•
The jump chain is simply a random walk reflected at the origin, and
it is easy to find that when A < p, there is a stationary distribution of
I., the form
+
=
>1
ItJ
whence
x2
— 1
j 1,
— 2/12
;To =
itt
•
2,u
The difference arises naturally because the holding times of X(t) depend
['on the state of the process. From (4.6.6) we know
A
j 0(
=
( j )--(- X + 1,07
1
)
and
fro oc 7rogo = 7r0ANow we set Eft,/ = 1, yielding fro = (u,—A,)/2p, as above. Incidentally)
no-1 = 1 — log (1 — (A/g)).
' 4.7(a) The equations r Q = 0 take the form
—irA = gituf —
1
with 7r1p,1 = r0A0. Therefore, n-j p j = 71-j_lAj_15 and m exists,
so X(t) is reversible.
(c) I f gol = a and gm = 6, it is easy to check that 7t0 =
+
0)
and 7r1 = a/(a + fi). So n-Ogol = 71,710, and the result follows.
(d) Because the detailed balance equations hold for X and Y) and
these are independent, i t is straightforward to see that the
detailed balance equations for (X, Y) are satisfied. In an obvious
notation
a d i b i b b o u s . . 71-x(i)n-Y(m)qx(i, ( m , n) = x ( D g Y (n)qx(i O q y
m