Markov chains
In a certain city, it is found that if it rains on a particular day, there is 75% chance
that the next day is sunny.
If it is sunny, then the chance that the next day is sunny is 40%.
What percentage of the days are rainy/sunny? (Assume those are the only two
possibilities).
We have to find the long range percentage of time of sun/rain.
Consider the following joint probability density: P (x, y) = P r(x&y). Sum over all
possible values of y
X
all
P (x = X, y = Y ) = P (x = X, anything else ) =
Y
X
P (x = X|y = Y )P (y = Y )
We have two states:
Sun = S = 0, Rain = R = 1, then we get
Pn (0) = P (0, day n) =
1
X
P (0, day n, i on n − 1)
i=0
=
1
X
P (0, day n|i on n − 1)P (i on n − 1)
i=0
Pn (1) = P (1, day n) =
1
X
P (1, day n, i on n − 1)
i=0
=
1
X
P (1, day n|i on n − 1)P (i on n − 1)
i=0
So the condition on previous events = Σ over all possible previous events. This
is useful since we have a Markov chain: set of discrete events (states), with the
probability of changing states depending only on the previous state.
Where
|
Pn (0)
Pn (1)
!
{z
}
Πn
T
=
Pn−1 (0)
Pn−1 (1)
!
{z
}
|
p0|0 p1|0
p0|1 p1|1
Πn−1
!
p0|0 p1|0
p0|1 p1|1
= (pij ) = (pr (j|i))
78
!
Note:
X
pij = 1 =
j
X
P (j|i) = P r(something happens after i )
j
In general if we have k states


pn (1)


pn (2)   .  = pn−1 (1) pn−1 (2) · · ·
 . 
 . 
pn (k)

p
 11
.
pn−1 (k) 
 ..
p12 · · ·
pk1 pk2 · · ·
|
{z
P
p1k
pkk




}
(FIGURE)
Limiting distribution
Πn+1 = Πn P = Πn−1 P 2 =⇒ Πn+k = Πn P k . As k → ∞ we get the limiting distribution

π(1) π(2) · · ·
 .
.
P −→ P̂ = 
 .
π(1) π(2) · · ·
where
π(j) =
k
X
π(i)pij
,
k
X
pij = π(j) = π(j)
k
X
i=1
The steady state equation is
Π = ΠP
79
π(k)
π(i) = 1
i=1
i=1
π(k)
π(i)




For our example:
P =
0.4 0.6
0.75 0.25
!
0.56
,Π =
0.44
!
=
P (0)
P (1)
!
As p0|1 ↑=⇒ P (0) ↑ and P (1) ↓
As p0|0 ↓=⇒ P (0) ↓ and P (1) ↑
Queueing Problem
In general the number of people in the queue could be infinite.
We define the states in terms of the number of people in the queue.
(FIGURE)
Jumping from one state depends on people arriving/ leaving the queue.
For example
P (1|2) =
X
X
P (all possible ways to go from 1 to 2)
j k,j−k=1
= P (1 arrives|0 served) + P (2 arrive|1 served) + P (3 arrive|2 served) + etc.
= P (1 arrives)P (0 served) + P (2 arrive)P (1 served) + P (3 arrive)P (2 served) + etc.
We take time interval small enough so P (n arrive)≈ 0. If we a queue that empties,
then entries for large n will be 0.
Suppose we have a queue with no more than 2 people. Therefore we have 3 states:
0, 1, 2.
(FIGURE)
80


P (0|0) P (1|0) P (2|0)


P = P (0|1) P (1|1) P (2|1) 
P (0|2) P (1|2) P (2|2)


p(0) + p(1)α p(1)(1 − α) + p(2)α
p(2)(1 − α)


=  p(0)α
p(0)(1 − α) + p(1)α
p(1)(1 − α) + p(2)

0
p(0)α
p(0)(1 − α) + p(2) + p(1)
λ
µ
Stability: It is proven for a single server when the traffic intensity is < 1 =⇒ the
queu will empty at a finite time.
Traffic intensity−→ ρ =
Other questions:
• Minimize the waiting time.
• Minimize the idle time.
• Minimize the cost of waiting if C = c 1 x1 +c2 x2 where ci are the costs for each type.
Fluid models (use expected arrival/ service times as rates)
(FIGURE)
ẋ1 = λ − µ
Can we simulate this? - Expensive!
Now we consider an example counter-intuitive.
For the fluid model is easy to compute the behavior of the system over the time
(continuous time).
81
Let consider a network with 2 stations and the traffic intensity for each station.
Initially xA (0) = 2 and xB (0) = 0
(FIGURE)
Let A,B,C,D be the job types
λ
λ
+
<1
µA µD
λ
λ
ρ2 =
+
<1
µB
µC
ρ1 =
There are 5 stochastic processes, λ - arrivals and four types of jobs. In what order
do they occur?
This is important!
Let’s consider a case where the traffic intensities are < 1 and let λ = 2, µ A = 8, µB =
1 2
1 2
4, µc = 6, µD = 3. Then λ1 = + < 1 and λ2 = + < 1
4 3
2 3
Idea: Start with the unit 2 in A only and then see if the system empties.
Serve A: fluid flows into B, C, until A empties. We have:
ẋA = 2 − 8 = −6 until t =
1
then A empties
3
ẋA = xA (0) − 6t
ẋB = 8 − 4 =⇒ ẋB = −4t
ẋC = 4 =⇒ ẋC = 4t serve B before C
ẋD = 0
Next serve 1 to keep xA = 0. Then
ẋA = 2 − 2 = 0
ẋB = 2 − 4 = −2
ẋC = 4
82
(FIGURES)
Next, B is empty so we could serve C. As soon as the fluid goes to D, serve D not
A. So, no fluid goes from A to B. Just serve C and D.
(FIGURE)
Then
ẋA = 2
ẋB = 0
2
3
ẋD = 6 − 3 = 3 =⇒ ẋD = (6 − 3)t = 2
ẋC = −6 =⇒ ẋC = 4 − 6t until t =
Now empty D. So serve only D, C, B, empty.
(FIGURE)
83
Then
ẋA = 2 =⇒ xA =
8
=⇒ longer than original
2
ẋB = 0
ẋC = 0
ẋD = −3 =⇒ D empties at t = 2
84
1
3