22S6 - Numerical and data analysis
techniques
Mike Peardon
School of Mathematics
Trinity College Dublin
Hilary Term 2012
Mike Peardon (TCD)
22S6 - Data analysis
Hilary Term 2012
1 / 11
Introduction to
Markov processes
Mike Peardon (TCD)
22S6 - Data analysis
Hilary Term 2012
2 / 11
Markov chains
In 1906, Markov was interested in demonstrating that
independence was not necessary to prove the (weak) law
of large numbers.
He analysed the alternating patterns of vowels and
consonants in Pushkin’s novel “Eugene Onegin”.
In a Markov process, a system makes stochastic
transitions such that the probability of a transition
occuring depends only on the start and end states. The
system retains no memory of how it came to be in the
current state. The resulting sequence of states of the
system is called a Markov chain.
Mike Peardon (TCD)
22S6 - Data analysis
Hilary Term 2012
3 / 11
Markov chains
A Markov chain
A system can be in any one of n distinct states, denoted
{χ1 , χ2 , . . . χn }.
If ψ(0), ψ(1), ψ(2), . . . is a sequence of these states observed
at time t = 0, 1, 2, . . . and generated by the system making
random jumps between states so that the conditional
probability
P(ψ(t) = χi |ψ(t − 1) = χj , ψ(t − 2) = χk , . . . ψ(0) = χz )
= P(ψ(t) = χi |ψ(t − 1) = χj )
then the sequence {ψ} is called a Markov Chain
If P(ψ(t) = χi |ψ(t − 1) = χj ) does not depend on t, then the
sequence is called a homogenous Markov chain - we will
consider only homogenous Markov chains.
Mike Peardon (TCD)
22S6 - Data analysis
Hilary Term 2012
4 / 11
Markov matrix
The conditional probabilities describe the probability of
the system jumping from state χj to state χi . There are
n × n possible jumps and these probabilities can be
packed into a matrix, with elements Mij being the
probability of jumping from j to i.
The Markov matrix
A matrix containing the n × n probabilities of the system
making a random jump from j → i is called a Markov matrix
Mij = P(ψ(t + 1) = χi |ψ(t) = χj )
Since the entries are probabilities, and the system is always in
a well-defined state, a couple of properties follow. . .
0 ≤ Mij ≤ 1
Pn
M =1
i=1 ij
Mike Peardon (TCD)
22S6 - Data analysis
Hilary Term 2012
5 / 11
Markov Processes (4)
Dublin’s weather
An example: On a rainy day in Dublin, the probability
tomorrow is rainy is 80%. Similarly, on a sunny day the
probability tomorrow is sunny is 40%.
This suggests Dublin’s weather can be described by a
(homogenous) Markov process. Can we compute the
probability any given day is sunny or rainy?
For this system, the Markov matrix is
Sunny Rainy
Sunny
0.4
0.2
Rainy
0.6
0.8
Mike Peardon (TCD)
22S6 - Data analysis
Hilary Term 2012
6 / 11
Dublin’s weather (2)
1
0
If today is sunny we can write the state as ψ(0) =
,
0.4
and the state tomorrow is then ψ(1) =
, and
0.6
0.28
0.256
ψ(2) =
, ψ(3) =
, ...
0.72
0.744
0
If today is rainy we can write the state as ψ(0) =
,
1
0.2
, and
and the state tomorrow is then ψ(1) =
0.8
0.24
0.248
ψ(2) =
, ψ(3) =
,
0.76
0.752
The vector ψ quickly collapses to a fixed-point, which
must be π, the eigenvector of M with eigenvalue 1,
P2
normalised such that i=1 πi = 1.
Mike Peardon (TCD)
22S6 - Data analysis
Hilary Term 2012
7 / 11
Dublin’s weather (3)
Finding the probability of sun or rain a long time in the
future is equivalent to solving
0.4 0.2
π1
π1
=
0.6 0.8
π2
π2
with the normalising condition for the probabilities;
π1 + π2 = 1
0.25
We find π =
. This is the invariant probability
0.75
distribution of the process; with no prior information these
are the probabilities any given day is sunny (25%) or rainy
(75%).
Mike Peardon (TCD)
22S6 - Data analysis
Hilary Term 2012
8 / 11
Population migrations
In a year, the fraction of the population of three provinces A, B
and C who move between provinces is given by
From/ To
A
B
C
A
3%
7%
B
1%
C
1%
2%
7%
Show the stable populations of the three provinces are in the
proportions 8 : 3 : 1
Mike Peardon (TCD)
22S6 - Data analysis
Hilary Term 2012
9 / 11
Winning a Tennis game at Deuce
Two tennis players, Alice and Bob have reached Deuce. The
probability Alice wins a point is p while the probability Bob
wins is q = 1 − p. Write a Markov matrix describing transitions
this system can make.
Answer:


1 p 0 0 0


 0 0 p 0 0 


M= 0 q 0 p 0 


 0 0 q 0 0 
0 0 0 q 1
with states given by χ = {A wins, Adv A, Deuce, Adv B, B wins}
Mike Peardon (TCD)
22S6 - Data analysis
Hilary Term 2012
10 / 11
Winning a Tennis game at Deuce (2)
Remember: entry Mij = P(ψ(t + 1) = χi |ψ(t) = χi )
Some transitions are forbidden (like Adv A → Adv B)
Some states are “absorbing” - once in that state, the
system never moves away.
With a bit of work, it is possible to see the long-time
average after starting in state χ3 ≡ Deuce is


p2



π3 = 



1−2pq
0
0
0
q2
1−2pq







1
The tennis game ends with probability 1
2
Alice wins with probability
Mike Peardon (TCD)
p2
1−2pq
22S6 - Data analysis
Hilary Term 2012
11 / 11