Mathematical Methods
in Economics
Markov Chains
Larson 2.5
I
Stochastic Modelling
• How should we model things like
 Stock market movement?
 Weather of a particular day?
 Market share of different companies in the same
market?
• How to describe what is happening at a given
moment in time?
• How to quantify the effect of the past on the
future?
State Matrix
• A state matrix 𝑋𝑋𝑡𝑡 is a matrix or a vector that
represents the state of the situation in a
particular time period 𝑡𝑡
 In stock market analysis, 𝑋𝑋𝑡𝑡 might represent
whether the stock market is up or down in day 𝑡𝑡
 If stock market is up:
𝑋𝑋𝑡𝑡 =
1
0
 If stock market is down: 𝑋𝑋𝑡𝑡 = 0
1
State Matrix
• A state matrix 𝑋𝑋𝑡𝑡 is a matrix or a vector that
represents the state of the situation in a
particular time period 𝑡𝑡
 In weather forecasting, 𝑋𝑋𝑡𝑡 might represent the
weather condition at day 𝑡𝑡
 Each row of 𝑋𝑋𝑡𝑡 represents one particular weather
condition
Sunny
Cloudy
Rainy
1
𝑋𝑋𝑡𝑡 = 0
0
0
𝑋𝑋𝑡𝑡 = 1
0
0
𝑋𝑋𝑡𝑡 = 0
1
State Matrix
• A state matrix 𝑋𝑋𝑡𝑡 is a matrix or a vector that
represents the state of the situation in a
particular time period 𝑡𝑡
 In operation research, 𝑋𝑋𝑡𝑡 might represent the
market share each company has in a particular
period of time 𝑡𝑡.
 Row 𝑖𝑖 of 𝑋𝑋𝑡𝑡 is the market share of company i
0.15
𝑋𝑋𝑡𝑡 = 0.2
0.65
Company 1 has 15%
market share
Company 2 has 20%
market share
Stochastic Matrix
• A stochastic matrix 𝑃𝑃 contains the probabilities of
transiting from one state to another
The probability that the
𝑝𝑝11 ⋯ 𝑝𝑝1𝑛𝑛
next state is 1 when the
⋮
⋱
⋮
current state is 1
𝑝𝑝𝑛𝑛𝑛 ⋯ 𝑝𝑝𝑛𝑛𝑛𝑛
The probability that the
next state is n when the
current state is 1
next state
current state
• Properties of a stochastic matrix:
• 𝒑𝒑𝒊𝒊𝒊𝒊 must be between 0 and 1. 0 means no chance of
happening; 1 means 100% chance of happening
• Each column must sums up to 1. The current state must
transit to some future state. Something must happen.
Markov Chains
• Markov Chain is a sequence of state matrices that are
related by the following equation:
𝑋𝑋𝑡𝑡+1 = 𝑃𝑃𝑋𝑋𝑡𝑡
• Interpretation:
 What happens next only depends on what is happening now
 The past does not affect the future beyond its effect on
today
Markov Chains
• Markov Chain is a sequence of state matrices that are
related by the following equation:
𝑋𝑋𝑡𝑡+1 = 𝑃𝑃𝑋𝑋𝑡𝑡
• What about next next?
• Next next next?
• n-periods after?
𝑋𝑋𝑡𝑡+2 = 𝑃𝑃𝑋𝑋𝑡𝑡+1 = 𝑃𝑃2 𝑋𝑋𝑡𝑡
𝑋𝑋𝑡𝑡+3 = 𝑃𝑃3 𝑋𝑋𝑡𝑡
𝑿𝑿𝒕𝒕+𝒏𝒏 = 𝑷𝑷𝒏𝒏 𝑿𝑿𝒕𝒕
Chapman-Kolmogorov Equations
What happens from now to tomorrow morning
depends on: 1. what happens from now to
midnight and 2. what happens from midnight to
tomorrow morning.
Today
Midnight
Tomorrow
Chapman-Kolmogorov Equations
• More generally, what happens from 𝑡𝑡 to 𝑡𝑡 + 𝑛𝑛
depends on 1. what happens from 𝑡𝑡 to 𝑚𝑚 and
2. what happens from 𝑚𝑚 to 𝑛𝑛, where 𝑚𝑚 is any
integer smaller than 𝑛𝑛.
Time 𝑡𝑡
Time 𝑡𝑡 + 𝑚𝑚
Time 𝑡𝑡 + 𝑛𝑛
Chapman-Kolmogorov Equations
• More generally, what happens from 𝑡𝑡 to 𝑡𝑡 + 𝑛𝑛
depends on 1. what happens from 𝑡𝑡 to 𝑚𝑚 and
2. what happens from 𝑚𝑚 to 𝑛𝑛, where 𝑚𝑚 is any
integer smaller than 𝑛𝑛.
Time 𝑡𝑡
𝑋𝑋𝑡𝑡
Time 𝑡𝑡 + 𝑚𝑚
𝑃𝑃𝑛𝑛
Time 𝑡𝑡 + 𝑛𝑛
𝑃𝑃𝑛𝑛 𝑋𝑋𝑡𝑡
Chapman-Kolmogorov Equations
• More generally, what happens from 𝑡𝑡 to 𝑡𝑡 + 𝑛𝑛
depends on 1. what happens from 𝑡𝑡 to 𝑚𝑚 and
2. what happens from 𝑚𝑚 to 𝑛𝑛, where 𝑚𝑚 is any
integer smaller than 𝑛𝑛.
Time 𝑡𝑡
𝑋𝑋𝑡𝑡
𝑃𝑃𝑚𝑚
Time 𝑡𝑡 + 𝑚𝑚
𝑃𝑃𝑚𝑚 𝑋𝑋𝑡𝑡
𝑃𝑃𝑛𝑛−𝑚𝑚
Time 𝑡𝑡 + 𝑛𝑛
𝑃𝑃𝑛𝑛 𝑋𝑋𝑡𝑡
Chapman-Kolmogorov Equations
• What happens from 𝑡𝑡 to 𝑡𝑡 + 𝑛𝑛 depends on 1.
what happens from 𝑡𝑡 to 𝑚𝑚 and 2. what
happens from 𝑚𝑚 to 𝑛𝑛, where 𝑚𝑚 is any integer
smaller than 𝑛𝑛.
𝑃𝑃𝑛𝑛 = 𝑃𝑃𝑛𝑛−𝑚𝑚 𝑃𝑃𝑚𝑚
• The above equations—since there are many
possible choices of 𝑚𝑚—are called
Chapman-Kolmogorov Equations.
Example
• Larson 2.5 Example 2-3
Two companies compete in a
city with 100,000 potential
subscribers. The figure on
the right shows the changes
in subscription each year.
Company A now has 15,000
subscribers and Company B
has 20,000. How many
subscribers will each
company have in one year?
Example
• Current state:
• Company A:
15,000
• Company B:
20,000
• None:
65,000
• State matrix (1 = 100%):
0.15
𝑋𝑋0 = 0.2
0.65
Example
• Stochastic matrix:
Example
• Stochastic matrix:
0.7 0.15 0.15
𝑃𝑃 = 0.2 0.8 0.15
0.1 0.05 0.7
Where Company A’s
customers in this
period are going to
Where Company A’s customers in
the next period are coming from
Where Company B’s
customers in this
period are going to
Example
• What happens in one year?
𝑋𝑋1 = 𝑃𝑃𝑋𝑋0
0.7 0.15 0.15 0.15
= 0.2 0.8 0.15 0.2
0.1 0.05 0.7 0.65
0.2325
= 0.2875
0.48
• Company A will have 23,250
subscribers while Company B
will have 28,750 subscribers.
Example
• What happens in two years?
𝑋𝑋2 = 𝑃𝑃2 𝑋𝑋0
𝑋𝑋2 = 𝑃𝑃𝑋𝑋1
0.7 0.15 0.15 0.2325
= 0.2 0.8 0.15 0.2875
0.1 0.05 0.7
0.48
0.2779
= 0.3485
0.3736
• Company A will have 27,790
subscribers while Company B
will have 34,850 subscribers.
Steady State Matrix
• The state matrix 𝑋𝑋� is a steady state if
𝑋𝑋� = 𝑃𝑃𝑋𝑋�
• What’s happening today is what’s happening
tomorrow. E.g.:
• 𝑋𝑋𝑡𝑡 = 0.7 stock market has a 70% chance of going
0.3
up today
• 𝑋𝑋𝑡𝑡+1 = 0.7 stock market also has a 70% chance of
0.3
going up tomorrow
• The market is in a steady state
Steady State Matrix
�
• How to find 𝑋𝑋?
𝑋𝑋� = 𝑃𝑃𝑋𝑋�
𝑃𝑃𝑋𝑋� − 𝑋𝑋� = 𝑂𝑂
𝑃𝑃 − 𝐼𝐼 𝑋𝑋� = 𝑂𝑂
Solve this linear system to get 𝑋𝑋�
Steady State Matrix
�
• How to find 𝑋𝑋?
𝑃𝑃 − 𝐼𝐼 𝑋𝑋� = 𝑂𝑂
Solve this linear system to get 𝑋𝑋�
• Notice that 𝑋𝑋� = 𝑂𝑂 is always a solution to this
system, but that’s not what we want!
• Stock market either goes up or goes down
• Customers are either buying or not buying
• More generally, states cannot be all zeros
• One more requirement: 𝑥𝑥̅1 + ⋯ + 𝑥𝑥̅𝑛𝑛 = 1
• Something has to have happened
Example
• Larson 2.5 Example 5
0.7 0.15 0.15
𝑃𝑃 = 0.2 0.8 0.15
0.1 0.05 0.7
𝑃𝑃 − 𝐼𝐼 𝑋𝑋� = 𝑂𝑂
0.7
𝑃𝑃 − 𝐼𝐼 = 0.2
0.1
0.15
0.8
0.05
1
0.15
0.15 − 0
0
0.7
0
1
0
0
0
1
Example
• Add in the requirement that 𝑥𝑥̅1 + 𝑥𝑥̅2 + 𝑥𝑥̅3 =
1, we have the augmented matrix:
−0.3 0.15 0.15 0
Augmented
Augmented
0.2 −0.2 0.15 0 matrix of:
matrix of:
𝑃𝑃 − 𝐼𝐼 𝑋𝑋� = 𝑂𝑂
0.1 0.05 −0.3 0 𝑥𝑥1̅ + ⋯ + 𝑥𝑥̅𝑛𝑛 = 1
1
1
1
1
• Solve this system with Gaussian Elimination
Example
−0.3 0.15
0.2 −0.2
0.1 0.05
1
1
R2 = R2 – 2R3:
−0.3 0.15
0
−0.3
0.1 0.05
1
1
R3 = R3 – R4/10:
−0.3 0.15
0
−0.3
0
−0.05
1
1
0.15 0
0.15 0
−0.3 0
1
1
0.15
0.75
−0.3
1
0
0
0
1
0.15
0
0.75
0
−0.4 −0.1
1
1
R2 = R2 - 6R3:
−0.3 0.15 0.15
0
0
0
3.15 0.6
0
−0.05 −0.4 −0.1
1
1
1
1
R2 = R2/3.15, R3 = 20R3:
−0.3 0.15 0.15
0
0
0
1
0.19
0
1
8
2
1
1
1
1
R3 = R3 – 8R2:
−0.3 0.15 0.15
0
0
0
1
0.19
0
1
0
0.48
1
1
1
1
𝑥𝑥2 = 0.48, 𝑥𝑥3 = 0.19, 𝑥𝑥1 = 0.33