18.440 Lecture 33
Scott Sheffield
MIT
1

Markov chains
Examples
Ergodicity and stationarity
18.440 Lecture 33
2

Markov chains
Examples
Ergodicity and stationarity
18.440 Lecture 33
3

I Consider a sequence of random variables X
0
, X
1
, X
2
, . . .
each taking values in the same state space, which for now we take to be a finite set that we label by { 0 , 1 , . . . , M } .
I Interpret X n as state of the system at time n .
I Sequence is called a Markov chain if we have a fixed collection of numbers P ij
(one for each pair i , j ∈ { 0 , 1 , . . . , M } ) such that whenever the system is in state i , there is probability P ij that system will next be in state j .
I Precisely,
P { X n +1
= j | X n
= i , X n − 1
= i n − 1
, . . . , X
1
= i
1
, X
0
= i
0
} = P ij
.
I Kind of an “almost memoryless” property. Probability distribution for next state depends only on the current state
(and not on the rest of the state history).
18.440 Lecture 33
4

I For example, imagine a simple weather model with two states: rainy and sunny.
I If it’s rainy one day, there’s a .
5 chance it will be rainy the next day, a .
5 chance it will be sunny.
I If it’s sunny one day, there’s a .
8 chance it will be sunny the next day, a .
2 chance it will be rainy.
I In this climate, sun tends to last longer than rain.
I Given that it is rainy today, how many days to I expect to have to wait to see a sunny day?
I Given that it is sunny today, how many days to I expect to have to wait to see a rainy day?
I Over the long haul, what fraction of days are sunny?
18.440 Lecture 33
5

I To describe a Markov chain, we need to define P ij for any i , j ∈ { 0 , 1 , . . . , M } .
I It is convenient to represent the collection of transition probabilities P ij as a matrix:
 P
00
P
01
. . .
P
0 M



P
10
P
11
. . .
P
1 M

 ·

A =





·
· 



P
M 0
P
M 1
. . .
P
MM
I For this to make sense, we require P ij
P
M j =0
P ij
= 1 for each i
≥ 0 for all i , j and
. That is, the rows sum to one.
18.440 Lecture 33
6

I Suppose that p i is the probability that system is in state i at time zero.
I What does the following product represent?
p
0 p
1
. . .
p
M
 P
00
P
01
. . .
P
0 M

10
P
11
. . .
P
1 M

P

·




·
·
P
M 0
P
M 1
. . .
P
MM








I Answer: the probability distribution at time one.
I How about the following product?
p
0 p
1
. . .
p
M
A n
I Answer: the probability distribution at time n .
18.440 Lecture 33
7

I We write P ij
( n ) over n steps.
for the probability to go from state i to state j
I From the matrix point of view

( n ) ( n )
P
00
( n )
 P


10
·
P
01
( n )
P
11

 ·



·
P
( n )
M 0
P
( n )
M 1
( n )
. . .
P
0 M
( n )
. . .
P
1 M
. . .
P
( n )
MM













=




P
00
P
01
. . .
P
0 M
P
10
P
11
. . .
P
1 M
·
·
·
P
M 0
P
M 1
. . .
P
MM
I If A is the one-step transition matrix, then A n transition matrix.
is the n -step
 n







18.440 Lecture 33
8

I What does it mean if all of the rows are identical?
I Answer: state sequence X i consists of i.i.d. random variables.
I What if matrix is the identity?
I Answer: states never change.
I What if each P ij is either one or zero?
I Answer: state evolution is deterministic.
18.440 Lecture 33
9

Markov chains
Examples
Ergodicity and stationarity
18.440 Lecture 33
10

Markov chains
Examples
Ergodicity and stationarity
18.440 Lecture 33
11

I Consider the simple weather example: If it’s rainy one day, there’s a .
5 chance it will be rainy the next day, a .
5 chance it will be sunny. If it’s sunny one day, there’s a .
8 chance it will be sunny the next day, a .
2 chance it will be rainy.
I Let rainy be state zero, sunny state one, and write the transition matrix by
A =
.
5 .
5
.
2 .
8
I Note that
I Can compute A
10
=
A
2
=
.
64 .
35
.
26 .
74
.
285719 .
714281
.
285713 .
714287
18.440 Lecture 33
12

In a relationship
Single
It’s complicated
Married
Engaged
I Can we assign a probability to each arrow?
I Markov model implies time spent in any state (e.g., a marriage) before leaving is a geometric random variable.
I Not true... Can we make a better model with more states?
18.440 Lecture 33
13

Markov chains
Examples
Ergodicity and stationarity
18.440 Lecture 33
14

Markov chains
Examples
Ergodicity and stationarity
18.440 Lecture 33
15

I Say Markov chain is ergodic if some power of the transition matrix has all non-zero entries.
I
I
Turns out that if chain has this property, then
π j
:= lim n →∞
P ij
( n ) exists and the π j are the unique non-negative solutions of π j
= P
M k =0
π k
P kj that sum to one.
This means that the row vector
π = π
0
π
1
. . .
π
M is a left eigenvector of A with eigenvalue 1, i.e., π A = π .
I We call π the stationary distribution of the Markov chain.
I One can
π j
= P
M solve the system of linear equations k =0
π k
P kj to compute the values π j
. Equivalent to considering A fixed and solving π A = π . Or solving
( A − I ) π = 0. This determ ines π up to a multiplicative constant, and fact that P π j
= 1 determines the constant.
18.440 Lecture 33
16

I If A =
.
5 .
5
.
2 .
8
, then we know
π A = π
0
π
1
I Recall that
.
285719 .
714281
A 10 =
.
285713 .
714287
18.440 Lecture 33
.
5 .
5
.
2 .
8
I This means that .
5 π we also know that π
0
0
+ .
2 π
+ π
1
1
= π
0 and .
5 π
0
+ .
8 π
1
= π
1 and
= 1. Solving these equations gives
π
0
= 2 / 7 and π
1
= 5 / 7, so π = 2 / 7 5 / 7 .
I Indeed,
π A = 2 / 7 5 / 7
.
5 .
5
.
2 .
8
= 2 / 7 5 / 7 = π.
≈
=
2
2
/
/
7
7
π
5
5
0
/
/
7
7
π
1
=
= π.
π
π
17

MIT OpenCourseWare http://ocw.mit.edu
18.440
Probability and Random Variables
Spring 2014
For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms .