Tutorial: Markov Chains
Steve Gu
Feb 28, 2008
Outline
• Markov chain
• Applications
– Weather forecasting
– Enrollment assessment
– Sequence generation
– Rank the web page
– Life cycle analysis
• Summary
History
• The origin of Markov chains is due to Markov,
a Russian mathematician who first published
in the Imperial Academy of Sciences in St.
Petersburg in 1907, a paper studying the
statistical behavior of the letters in Onegin, a
well known poem of Pushkin.
A Markov Chain
P01
P00
"1"
"0"
P10
P 11
Transition Probability Table
 P11

P   P21
 P31
P12
P22
P32
P13 

P23 
P33 
Pij  0, i = 1,..., n; j = 1,..., n and
n
 Pij
j =1
P11 = 0.7
P21 = 0.
P12 = 0.2 P13 = 0.1
P22 = 0.6 P23 = 0.4
P31 = 0.3 P32 = 0.5 P33 = 0.2
1
Example 1:
Weather Forecasting[1]
Weather Forecasting
• Weather forecasting example:
– Suppose tomorrow’s weather depends on today’s weather only.
– We call it an Order-1 Markov Chain, as the transition function depends on the
current state only.
– Given today is sunny, what is the probability that the coming days are sunny,
rainy, cloudy, cloudy, sunny ?
– Obviously, the answer is : (0.5)(0.4)(0.3)(0.5) (0.2) = 0.0054
0.1
0.5
0.4
0.4
sunny
0.3
0.3
rainy
cloudy
0.3
0.2
0.5
Weather Forecasting
• Weather forecasting example:
– Given today is sunny, what is the probability that it will be rainy 4 days later?
– We only knows the start state, the final state and the input length = 4
– There are a number of possible combinations of states in between.
0.1
0.5
0.4
0.4
sunny
0.3
0.3
rainy
cloudy
0.3
0.2
0.5
Weather Forecasting
• Weather forecasting example:
– Chapman-Kolmogorov Equation:
– Transition Matrix:
s
( nm)
ij
P
r

  Pik( n ) Pkj( m )
c
k 0
s  0.5

r  0.3
0.4 0.1

0.4 0.3 
c  0.2 0.3 0.5 


0.1
0.5
0.4
0.4
sunny
0.3
0.3
rainy
cloudy
0.3
0.2
0.5
Weather Forecasting
• Weather forecasting example:
(00 x 01) + (01 x 11) + (02 x 21)  01
– Two days:
 0.5 0.4 0.1  0.5 0.4 0.1  0.39 0.39 0.22 


 

 0.3 0.4 0.3   0.3 0.4 0.3    0.33 0.37 0.30 
 0.2 0.3 0.5   0.2 0.3 0.5   0.29 0.35 0.36 


 

– Four days:
 0.5 0.4 0.1  0.5 0.4 0.1  0.3446 0.3734 0.2820 

 

 
 0.3 0.4 0.3   0.3 0.4 0.3    0.3378 0.3706 0.2916 
 0.2 0.3 0.5   0.2 0.3 0.5   0.3330 0.3686 0.2984 
 


 
2
2
0.1
0.5
0.4
0.4
sunny
0.3
0.3
rainy
cloudy
0.3
0.2
0.5
Weather Forecasting
• Weather forecasting example:
–
–
–
–
What is the probability that today is cloudy?
There are infinite number of days before today.
It is equivalent to ask the probability after infinite number of days.
We do not care the “start state” as it brings little effect when there are infinite
number of states.
– We call it the “Limiting probability” when the machine becomes steady.
0.1
0.5
0.4
0.4
sunny
0.3
0.3
rainy
cloudy
0.3
0.2
0.5
Weather Forecasting
• Weather forecasting example:
– Since the start state is “don’t care”, instead of forming a 2-D matrix, the limiting
probability is express a a single row matrix :
 0 , 1 , 2 
– Since the machine is steady, the limiting probability does not change even it
goes one more step.
0.1
0.5
0.4
0.4
sunny
0.3
0.3
rainy
cloudy
0.3
0.2
0.5
Weather Forecasting
• Weather forecasting example:
– So the limiting probability can be computed by:
 0 , 1 , 2 
 0.5 0.4 0.1


 0.3 0.4 0.3 
 0.2 0.3 0.5 


  0 ,  1 ,  2 
18
– We have  0 ,  1 ,  2   ( 21 , 23 , 18 )  probability that today is cloudy =
62
62 62 62
0.1
0.5
0.4
0.4
sunny
0.3
0.3
rainy
cloudy
0.3
0.2
0.5
Example 2:
Enrollment Assessment [1]
Undergraduate Enrollment Model
Stop Out
Freshmen
Sophomore
Graduate
Junior
Senior
State Transition Probabilities
TP
Fr
So
Jr
Sr
S/O
Gr
Fr
.2
.65
0
0
.14
.01
So
0
.25
.6
0
.13
.02
= Jr
0
0
.3
.55 .12
.03
Sr
0
0
0
.4
.55
S/O 0.1
0.1
0.4
0.1 0.3
0
Gr
0
0
0
1
0
.05
0
Enrollment Assessment
Stop Out
Freshmen
Sophomore
Junior
Senior
Graduate
TP
=
Fr
So
Jr
Fr
.2
0
0
So
.65
.25
0
Jr
0
.6
.3
Sr
0
0
.55
S/O
.14
.13
.12
Gr
.01
.02
.03
Sr
S/O
Gr
0
0.1
0
0
0.1
0
0
0.4
0
.4
0.1
0
.05
0.3
0
.55
0
1
Given:
Transition probability table &
Initial enrollment estimation,
we can estimate the number
of students at each time point
Example 3:
Sequence Generation[3]
Sequence Generation
Markov Chains as Models of
Sequence Generation
s  sttacggt
1s2 s3s4 
•
0th-order
N
N
i 1
i 1
g 
P0 s   pts1 ppts2 p aps3p
c  p
psi   psi 
•
•
1st-order
1th-order
N
N
s1ps
P1 s   pts1 ppts| 2t | sp1 a p|ts3 |ps2c
| a p
psip|si 1| si 1 
1  
•
•
2
2nd-order
i 2 i 2
N N
 pps1s21s2 
PP22ss  pptts1s2pap| stt3 |sp1sc2 |ta
ps4p|sg2 s| 3ac

ppsi s| is|i s2i s2i s1i1 
i 3i 3
A Fifth Order Markov Chain
Example 4:
Rank the web page
How to rank the
importance of web
pages?
PageRank
PageRank
http://en.wikipedia.org/wiki/Image:PageRanks-Example.svg
PageRank: Markov Chain
For N pages, say p1,…,pN
Write the Equation to compute PageRank as:
where l(i,j) is define to be:
PageRank: Markov Chain
• Written in Matrix Form:
 PR(p1,n + 1) 
 PR(p1,n) 
l(1,N)  

  l(1,1) l(1,2)

PR(p
,n
+
1)
PR(p
,n)


  l(2,1) l(2,2)

2
2
l(2,N)






 

  PR(pN-1,n) 
 PR(pN-1,n + 1)  
l(N,N - 1) l(N,N)  
 PR(p ,n + 1)   l(N,1)


N

 PR(pN,n) 
Example 5:
Life Cycle Analysis[4]
How to model life cycles of Whales?
http://www.specieslist.com/images/external/Humpback_Whale_underwater.jpg
Life cycle analysis
In real application, we need to specify
or learn the transition probability table
calf
immature
mature
dead
mom
Post-mom
Application: The North Atlantic right whale
(Eubalaena glacialis)
June 2006
Hal Caswell -- Markov Anniversary Meeting
30
Endangered, by any
standard
N < 300 individuals
Minimal recovery since
1935
feeding
Ship strikes
Entanglement with fishing
gear
calving
June 2006
Hal Caswell -- Markov Anniversary Meeting
31
Mortality and serious injury
due to entanglement and ship strikes
2030: died October 1999
entanglement
June 2006
1014 “Staccato” died April 1999 ship strike
Hal Caswell -- Markov Anniversary Meeting
32
0.96
time trend
best model
0.94
Calf survival
0.92
0.9
0.88
0.86
0.84
0.82
1980
June 2006
1984
1988
Year
1992
Hal Caswell -- Markov Anniversary Meeting
1996
33
1
time trend
best model
0.95
Mother survival
0.9
0.85
0.8
0.75
0.7
0.65
0.6
1980
June 2006
1984
1988
Year
1992
Hal Caswell -- Markov Anniversary Meeting
1996
34
0.5
time trend
best model
0.45
Birth probability
0.4
0.35
0.3
0.25
0.2
0.15
0.1
1980
June 2006
1984
1988
Year
1992
Hal Caswell -- Markov Anniversary Meeting
1996
35
70
period
Life expectancy
60
50
Things don’t look good for the right whale!
40
30
20
10
1980
June 2006
1982
1984
1986
1988 1990 1992
Hal Caswell -- Markov
Anniversary Meeting
Year
1994
1996
1998
36
Summary
• Markov Chains: state transition model
• Some applications
– Natural Language Modeling
– Weather forecasting
– Enrollment assessment
– Sequence generation
– Rank the web page
– Life cycle analysis
– etc (Hopefully you will find more  )
Thank you
Q&A
Reference
[1] http://adammikeal.org/courses/compute/presentations/Markov_model.ppt
[2] http://uaps.ucf.edu/doc/AIR2006MarkovChain051806.ppt
[3]http://germain.umemat.maine.edu/faculty/khalil/courses/MAT500/JGraber/genes2007.ppt
[4] http://www.csc2.ncsu.edu/conferences/nsmc/MAM2006/caswell.ppt