MA22S6 Homework 7 Solutions
ˆ Question 1: Suppose we have a system that evolves in time by making transitions at
points in time t = 1, 2, 3, ... between n distinct states χ1 , . . . , χn . Denote by ψ(t) the state
occupied at time t. If the probability that ψ(t + 1) = χi , conditional on the previous
history of states satisfies,
P(ψ(t + 1) = χi ∣ψ(t) = χj , ψ(t − 1) = χk , . . .) = P(ψ(t + 1) = χi ∣ψ(t) = χj ),
the time evolution of the system is called a Markov process. In other words, the probability
of the next state is conditional only on the present state of the system, regardless of the
previous history. The Markov matrix, M , gives the probability of transitioning between
the possible states, with Mij the probability of a direct transition from χj to χi , i.e.
Mij = P(ψ(t + 1) = χi ∣ψ(t) = χj ) .
A Markov matrix M has the property
n
∑ Mij = 1 for j = 1, . . . , n,
(1)
i=1
because the probability to have a transition from some state j to any of the states i,
i = 1, . . . , n must be 1. Since all matrix elements of M are probabilities these have to take
values between zero and one, i.e. 0 ≤ Mij ≤ 1 for all i, j = 1, . . . , n.
Generically (due to theorems by Perron and Frobenius), Markov matrices have a single
eigenvector v to eigenvalue 1, with all other eigenvalues being smaller (in modulus). This
implies that there is a “fixed point” equation:
Mv = v .
(2)
One may describe the time evolution of the system by a sequence of vectors v(t) such
1
that
v(t) = M v(t − 1) = M 2 v(t − 2) = ⋯ = M t v(0),
(3)
and normalized such that ∑ni=1 vi (t) = 1 for all times t = 0, 1, . . .. The vector component
vi (t) is then interpretated as the probability that the system at time t is in state χi . These
probabilities do depend on the choice of the initial vector v(0), however, this dependence
dies out exponentially fast due to the above properties of the Markov matrix. One finds
that
lim v(t) = v,
(4)
t→∞
where the approach to the limit distribution v occurs with an exponential rate of convergence which is determined by the gap between 1 and the next-largest eigenvalue of
M.
Suppose we denote States 1,2 and 3 by cloudy, rainy and sunny respectively, i.e., this is our
choice of basis. Given the information in the statement of the question (and completing
the missing entries by recalling that columns sum to 1) our Markov matrix M is
⎛ 0.35 0.20 0.35
⎜
⎜
M =⎜
⎜ 0.45 0.70 0.15
⎜
⎜
⎝ 0.20 0.10 0.50
⎞
⎟
⎟
⎟
⎟
⎟
⎟
⎠
What is the probability that it is sunny tomorrow, given that it is cloudy today? Our
initial vector is v(t = 0) = (1, 0, 0)T , i.e. we choose the initial probability distribution that
today it is cloudy with probability 1. We then use the formula v(t + 1) = M v(t) for t = 0
to find the distribution for tomorrow’s weather (t = 1), and then recursively for the day
after tomorrow (t = 2) and the following day (t = 3):
2
⎛ 0.35 0.20 0.35 ⎞ ⎛ 1 ⎞ ⎛ 0.35 ⎞
⎟
⎜
⎟⎜ ⎟ ⎜
⎜
⎟
⎟⎜ ⎟ ⎜
⎜
⎜
⎟
⎜
⎟
v(1) = ⎜ 0.45 0.70 0.15 ⎟ ⎜ 0 ⎟ = ⎜ 0.45 ⎟
⎟
⎜
⎟
⎟⎜ ⎟ ⎜
⎜
⎟
⎟⎜ ⎟ ⎜
⎝ 0.20 0.10 0.50 ⎠ ⎝ 0 ⎠ ⎝ 0.20 ⎠
⎛ 0.35 0.20 0.35
⎜
⎜
v(2) = ⎜
⎜ 0.45 0.70 0.15
⎜
⎜
⎝ 0.20 0.10 0.50
⎞ ⎛ 0.35 ⎞ ⎛ 0.2825
⎟⎜
⎟ ⎜
⎟⎜
⎟ ⎜
⎟ ⎜ 0.45 ⎟ = ⎜ 0.5025
⎟⎜
⎟ ⎜
⎟⎜
⎟ ⎜
⎟⎜
⎟ ⎜
⎠ ⎝ 0.20 ⎠ ⎝ 0.2150
⎞
⎟
⎟
⎟
⎟
⎟
⎟
⎠
⎛ 0.35 0.20 0.35
⎜
⎜
v(3) = ⎜
⎜ 0.45 0.70 0.15
⎜
⎜
⎝ 0.20 0.10 0.50
⎞ ⎛ 0.2825 ⎞ ⎛ 0.274625
⎟⎜
⎟ ⎜
⎟⎜
⎟ ⎜
⎟ ⎜ 0.5025 ⎟ = ⎜ 0.511125
⎟⎜
⎟ ⎜
⎟⎜
⎟ ⎜
⎟⎜
⎟ ⎜
⎠ ⎝ 0.2150 ⎠ ⎝ 0.214250
⎞
⎟
⎟
⎟
⎟
⎟
⎟
⎠
Hence, given that today is cloudy, the probabilities that tomorrow, the day after, and the
following day are sunny are 0.2, 0.215 and 0.214250, respectively.
To find the average/long term probabilities we must find the fixed point distribution
ie. solve the system M v = v:
⎛ 0.35 0.20 0.35 ⎞ ⎛ v ⎞ ⎛ v ⎞
⎜
⎟⎜ 1 ⎟ ⎜ 1 ⎟
⎜
⎟⎜
⎟ ⎜
⎟
⎜ 0.45 0.70 0.15 ⎟ ⎜ v ⎟ = ⎜ v ⎟
⎜
⎟⎜ 2 ⎟ ⎜ 2 ⎟
⎜
⎟⎜
⎟ ⎜
⎟
⎟
⎜
⎟⎜
⎟ ⎜
⎝ 0.20 0.10 0.50 ⎠ ⎝ v3 ⎠ ⎝ v3 ⎠
We obtain v1 = 79 ⋅ v3 and v2 =
17
7
⋅ v3 .
Using the constraint v1 + v2 + v3 = 1 gives us v3 =
7
33
The resulting fixed point distribution is:
⎛v ⎞
⎜ 1⎟
⎜ ⎟
⎜v ⎟ =
⎜ 2⎟
⎜ ⎟
⎜ ⎟
⎝v3 ⎠
⎛9⎞
⎜ ⎟
⎜ ⎟
1 ⎜ ⎟
33 ⎜17⎟
⎜ ⎟
⎜ ⎟
⎝7⎠
3
⎛0.2̇7̇⎞
⎜
⎟
⎜
⎟
⎜
= ⎜0.5̇1̇⎟
⎟
⎜
⎟
⎜
⎟
⎝0.2̇1̇⎠
Thus the long term probability of a cloudy day is 27.28%, of sunny day it is 21.21% and
for a rainy day it is 51.51%.
d) Changing the basis by permuting the labels 1,2,3 for rainy, cloudy and sunny will correspondingly permute the rows and columns of the matrix M . For instance, if one chooses
the basis where the states 1,2 and 3 correspond to cloudy, sunny and rainy respectively
(i.e. sunny and rainy are interchanged w.r.t. the previous choice). The Markov matrix
M is then given by
⎛ 0.35 0.35 0.20 ⎞
⎜
⎟
⎜
⎟
⎜
M = ⎜ 0.20 0.50 0.10 ⎟
⎟
⎜
⎟
⎜
⎟
⎝ 0.45 0.15 0.70 ⎠
which means that the second and third columns as well as the second and third rows
have been interchanged w.r.t. the previous choice. The fixed point distribution is then
vT = (v1 , v3 , v2 ) with v1,2,3 from the previous case.
e) The detailed balance equations read Mij vj = Mji vi , for fixed indices i, j. For i = j the
equations are trivially satisfied, so we need to check whether for all i ≠ j
Mij
Mji
=
vi
vj .
Given
the fixed point distribution v above and the Markov matrix M we compare left and right
hand sides:
– lhs =
M12
M21
=
2
10
4
⋅ 100
45 = 9 ,
rhs =
v1
v2
=
3
11
33
⋅ 17
=
– lhs =
M13
M31
=
35
100
7
⋅ 10
2 = 4,
rhs =
v1
v3
=
3
11
9
⋅ 33
7 = 7,
– lhs =
M23
M32
=
15
100
3
⋅ 10
1 = 2,
rhs =
v2
v3
=
17
33
⋅ 33
7 =
9
17 ,
17
7 .
Clearly the detailed balance equations are not satisfied by the Markov matrix M .
f) Now we turn things around: given the fixed point distribution v from above we want
to find another Markov matrix M̃ that satisfies the detailed balance equation for this
distribution, i.e. such that
M̃ij vj = M̃ji vi .
(5)
4
Summing this equation over j and using the Markov matrix property (columns sum to 1)
one obtains M̃ v = v. In other words, if one finds a Markov matrix M̃ which satisfies the
detailed balance conditions this matrix is guaranteed to have v as fixed point distribution
(a.k.a. eigenvector to eigenvalue 1). On the other hand we have seen above that M
does not satisfy detailed balance but has the fixed point distribution v. Hence, requiring
detailed balance for M̃ is only a sufficient but not necessary condition to ensure the fixed
point equation. In practice detailed balance can be easily implemented and verified and
therefore provides an easy way to find a Markov process which is guaranteed to converge
to the desired fixed point distribution.
Note that the requirement that a matrix M̃ satisfies detailed balance does not specify
this matrix uniquely (there is actually an infinite number of solutions). We will use the
following solution from class:
M̃ij
⎧
⎪
⎪
1
⎪
⎪
⎪ n Aij ,
= ⎨
⎪
⎪
1
⎪
⎪
A + 1 −
⎪
⎩ n ij
where Aij = min(1,
i≠j
1
n
∑k=1 Akj
n
i=j
vi
vj ).
So we have:
⎛
1
min(1, vv12 ) min(1, vv13 )⎞
⎟
⎜
⎟
⎜
v
v
⎜
2 ⎟
=
A = ⎜min(1, v2 )
1
min(1,
)
v3 ⎟
1
⎟
⎜
⎟
⎜
v
v
1
⎝min(1, v13 ) min(1, v32 )
⎠
⎛1
⎜
⎜
⎜1
⎜
⎜
⎜7
⎝9
We then find the diagonal entries of M̃ ,
M̃11 =
M̃22 =
M̃33 =
× 1 + 1 − 31 (1 + 1 + 97 ) =
11
27
9
1
× 1 + 1 − 13 ( 17
+ 1 + 17
)=
35
51
× 1 + 1 − 31 (1 + 1 + 1) =
1
3
1
3
1
3
1
3
and for i ≠ j we have M̃ij = 13 Aij , so that we find
⎛ 11
⎜ 27
⎜
M̃ = ⎜
⎜ 31
⎜
⎜7
⎝ 27
5
3
17
1⎞
3⎟
⎟
35
51
1 ⎟.
3⎟
⎟
7
51
1
3⎠
⎟
9
17
1
7
17
1⎞
⎟
⎟
1⎟
⎟.
⎟
⎟
1⎠
One may check that the columns add up to 1 and that all matrix elements are between
0 and 1 as must be the case for any Markov matrix.
6