Unit - V
Markov Chain Analysis
What is Markov Model?
• If the future states of a process are independent of the past and depend
only on the present, the process is called Markov Process.
• A Markov chain is “ a stochastic model describing a sequence of
possible events in which the probability of each event depends only on
the state attained in the previous event.”
• A Markov Chain is a random process with the property that “ the next
stage depends only on the current state.
• Markov Chains are used to analyze trends and predict the future such as
Weather, stock market, genetics, games, customer loyal towards the
brands etc.
• Some Examples are:
– Snake & ladder game
– Weather system
.
Assumptions for Markov model
• A fixed set of states,
• Fixed transition probabilities, and the possibility of getting
from any state to another through a series of transitions.
• A Markov process converges to a unique distribution over
states. This means that what happens in the long run
won’t depend on where the process started or on what
happened along the way.
• What happens in the long run will be completely
determined by the transition probabilities – the
likelihoods of moving between the various states.
Lets try to understand Markov chain from very
simple example
•
Weather:
•
raining today
40% rain tomorrow
60% no rain tomorrow
•
•
not raining today
20% rain tomorrow
80% no rain tomorrow
•
Stochastic Finite State
Machine:
0.6
0.8
0.4
rain
no rain
0.2
Markov Process
Simple Example
Weather:
• raining today
40% rain tomorrow
60% no rain tomorrow
20% rain tomorrow
• not raining today
80% no rain tomorrow
The transition matrix:
Rain
Rain
No rain
0.4 0.6 
P 


No rain
 0.2 0.8 
• Stochastic matrix:
Rows sum up to 1
Elements in the matrix >= 0
7
Problem
There are 2 products Brand X & Brand Y respectively. The 2 products have 50% of the
total market share and the market is assumed to be of fixed size. The Transation
matrix as applied to these 2 products is given below:
To
From
Determine the market share
(1) One period later
(2) on a long run
X
Y
X
0.9
0.1
Y
0.5
0.5
Solution:
States : Brand X & Brand Y
Initial Market Shares: Brand X = 50% Brand Y = 50%
Transition Matrix =
x
X
Y
X
P 11 = 0.9
P12 = 0.1
Y
P21 = 0.5
P22 = 0.5
y
0.5 0.5
x
y

 x 0.9 0.1


 y 0.5 0.5
Market Share for second period:
x
y
0.5 0.5
x
y

 x 0.9 0.1


 y 0.5 0.5
(i) Market share for Brand X = (0.5*0.9) + (0.5*0.5) = 0.70
(ii) Market share for Brand Y = (0.5*0.1) + (0.5*0.5) = 0.30
The market share at second period are :
x
0.7
y
0.3
Market Share for third period:
x
y
0.7 0.3
x
y

 x 0.9 0.1


 y 0.5 0.5
(i) Market share for Brand X = (0.7*0.9) + (0.3*0.5) = 0.78
(ii) Market share for Brand Y = (0.7*0.1) + (0.3*0.5) = 0.22
The market share at Third period are :
x
0.78
y
0.22
Market Share for fourth period:
x
y
x
y

 x 0.9 0.1


 y 0.5 0.5
0.78 0.22
(i) Market share for Brand X = (0.78*0.9) + (0.22*0.5) = 0.81
(ii) Market share for Brand Y = (0.78*0.1) + (0.22*0.5) = 0.19
The market share at fourth period are :
x
0.81
y
0.19
Similarly determine market share for
5th , 6th , 7th , 8th and 9th Period:
Market Share at 7th , 8th and 9th Period
X = 0.833 and Y = 0.167
We observe that Market shares for Brand X
& Brand Y remains same from 7th, 8th and
9th Period, Hence we conclude that the
system has reached steady state
( Equlibrium condition or on Long run)
Alternatively:
n


Vi Vi P
n 1
Market Share for second period:
V V P
2
i
2 1
i
1
0.9 0.1
 0.7 0.3
V  0.5 0.5

0.5 0.5
2
i
Market Share for third period:
V i  V i P
3
V  0.5
2
i
31
0.9
0.5
0.5

 0.5
0.9
0.5
0.5

 0.78
0.22
0.1
0.5

2
0.1 0.9



0.5 0.5
0.1

0.5
For Steady State condition / Equilibrium / Long
run Condition


V V P
*
Where:
 V 1
*
0.9 0.1
X Y  


X

 0 .5 0 . 5 
Where X  Y  1
0.9 X  0.5 Y  X
(eqn.1)
0.1X  0.5 Y  Y
(eqn.2)
X  Y  1 (eqn.3)
Y
Solving equations, Considering :
X = 1 – Y (eqn. 4)
Substituting Eqn.4 in Eqn.1 & Eqn. 2, we get:
0.9(1  Y )  0.5 Y  1  Y
0.9  0.9Y  0.5Y  Y  1
0.6Y  0.1
0 .1
Y 
 0.167
0 .6
Therefore X = 1- 0.167 = 0.833
Problem No: 2
A market survey is made of 3 brands of food items x, y and
z. each time a customer purchases a new package, he may
use the same brand or switch to another, the following
estimates are obtained expressed in decimal fractions:
x 0.7 0.2 0.1
P  y  0.3 0.4 0.3
z  0.3 0.3 0.4
At this time, it is estimated that 30% of the people buy
brand x, 20% brand y and 50% of brand z.
What would be the distribution of customers:
a) 2 period later
b) 4 period later
c) What will be the distribution of customers in the long
run?
Solution:
0.3 0.2 0.5
 0.7 0.2 0.1
 0 .3 0 .4 0 .3 


 0.3 0.3 0.4
0.42 0.29 0.29
Distribution of customers two period later:
V V P
3
i
31
i
0.7 0.2 0.1 0.7 0.2 0.1
3




V i  0.3 0.2 0.50.3 0.4 0.3  0.3 0.4 0.3
0.3 0.3 0.4 0.3 0.3 0.4
0.58 0.25 0.17
3


V i  0.3 0.2 0.50.42 0.31 0.27
0.42 0.30 0.28
V  0.468 0.287 0.245
3
i
Distribution of customers two period later:
V V P
4
i
4 1
i
0.7 0.2 0.1 0.7 0.2 0.1 0.7 0.2 0.1
3
 0 .3 0 .4 0 .3    0 .3 0 .4 0 .3    0 .3 0 .4 0 .3 



0
.
3
0
.
2
0
.
5
Vi

 
 

0.3 0.3 0.4 0.3 0.3 0.4 0.3 0.3 0.4
0.58 0.25 0.17 0.7 0.2 0.1
3
0.42 0.31 0.27  0.3 0.4 0.3



0
.
3
0
.
2
0
.
5
Vi

 

0.42 0.30 0.28 0.3 0.3 0.4
V  0.487 0.281 0.243
3
i
Distribution of customers in long run:


V V P
*
x
y
z
where
x  y  z 1
*
 0.7 0.2 0.1
 0.3 0.4 0.3  x


 0.3 0.3 0.4
y
z
0 . 7 x  0 .3 y  0 .3 z  x
eqn.1
0 . 2 x  0 .4 y  0 . 3 z  y
eqn.2
0.1x  0.3 y  0.4 z  z
eqn.3
x  y  z 1
eqn.4
Considering eqn.1 & eqn 2 :
 0 . 3 x  0 .3 y  0 . 3 z  0
0.2 x  0.6 y  0.3 z 0
and let
x 1  y  z
0 .6 y  0 .6 z  0 .3
 0.8 y  0.1z   0.2
x  0.501 y  0.278 z  0.222
Regular Markov Chains:
A transition matrix P is regular if some power of P has only
positive entries. A markov chain is regular if its transition matrix
is regular.
ex,. Which of the following matrices are regular.
 0.3
a) 
 0.1
0.7 
0
b) 

0.9 
1
 0.2
d) 
0


 1
0.3
0
0
0.5 
1 

0 

1
 0.2
c) 

0
 1
0.8
0 

Problem No: 3
A major bank evaluates the credit rating of its credit card customers
on a monthly basis. Customers credit card ratings are evaluated as
Poor, Good and Excellent depending on there payment history. The
following matrix of transition probabilities reflect the probabilities
that a customers classified in one credit category during one month
will be evaluated in a given category on the following month.
0.80 0.18 0.02
P  Good  0.2 0.75 0.05
Excellent  0
0.16 0.84
Poor
Given an instating pool of 1 lakh customers, in the long run how
many customers accounts will be classified in each category?


Vi Vi P
n
0 0 1
Poor
 0.80 0.18 0.02
 0.2 0.75 0.05


 0
0.16 0.84
0.076
Good  0.3107
Excellent 0.6127
4 13
n 1
 
V
4


V V P
*
P G
*
 0.80 0.18 0.02
 0.2 0.75 0.05  P G


 0
0.16 0.84
E
E
where
P G  E 1
Poor
0.410
0.410x100000 = 41000
Good
0.410
0.410x100000 = 41000
Excellent
0.1798
0.1798x100000 = 17980
Problem No: 4
A salesman territory consists of 3 cities A, B and C. the Salesman
never sells in the same city on successive days. If he sells in city A,
then the next day he sells in city B. However if he sells in either
city B or city C, then the next day he is twice as likely to sell in city
A as in the other city. Determine how in the long run the salesman
sells in the each cities.
Problem No: 4
A salesman territory consists of 3 cities A, B and C. the Salesman
never sells in the same city on successive days. If he sells in city A,
then the next day he sells in city B. However if he sells in either
city B or city C, then the next day he is twice as likely to sell in city
A as in the other city. Determine how in the long run the salesman
sells in the each cities.
A 0 1 0 
2
1
P  B 3 0 3
C  23 13 0
A
B
C
0.4% 0.45% 0.15%
Problem No: 5
A housewife buys 3 types of cereals A, B and C. She never buys the
same cereals on successive weeks. If she buys cereal A then the
next week she buys cereal C. however, if she buys B or C, then the
next week she is 3 times as likely to buy type A as the other brand.
Find the transition matrix. In the long run how does she buy each of
the 3 brands?
Problem No: 5
A housewife buys 3 types of cereals A, B and C. She never buys the
same cereals on successive weeks. If she buys cereal A then the
next week she buys cereal C. however, if she buys B or C, then the
next week she is 3 times as likely to buy type A as the other brand.
Find the transition matrix. In the long run how does she buy each of
the 3 brands?
A 0 0 1 

3
1
P  B 4 0 4
C  34 14 0
A
0.429
B
0.114
C
0.457
Problem No: 6
The following transition matrix shows the record survey on the
toothpastes:
Babool 0.7 0.1 0.1
Co lg ate  0 0.7 0.2
P
Pepsodent  0
0 0 .9

Himalaya 0.2 0.1 0.1
0.1
0.1
0.1

0 .6 
a) If the customer now using Babool Paste, What are the
probability that are associated with the 4th purchase.
b) Whatis the market share of each brand during long run?
Problem No: 7
A market survey has been conducted to determine the
movements of people b/w type of residences. The results of
survey are tabulated as follows:
Previous Residence
Pr esent Re sidence 
APT 100 20
40
40
200
TH 150 40
0
10
200
20 120 10
200
OH 50
RH 100 20
20
60
200
The data are believed to be representation of the behavior of
the population at large. Develop a model & Answer the
following:
(i) What is the probability that someone who is now in an
apartment will own his own home after 2 moves.
(ii) what is the probability that same person will own home
after 3 moves, but no sooner.
(iii) what can you say about the long term demand for town
houses relative to the other residence type.
APT 0.25 0.375 0.125 0.25
TH  0.2
0.4
0 .2
0.2 
Pr evious 
OH 0.22
0
0.66 0.11


RH 0.33 0.083 0.083 0.5 
APT 0.25 0.375 0.125 0.25
TH  0.2
0.4
0 .2
0.2 
Pr evious 
OH 0.22
0
0.66 0.11


RH 0.33 0.083 0.083 0.5 
apt
th
0.2475
APT
0.2454
oh
rh
0.2645
TH
OH
0.1414
0.209
0.276
RH
0.3887
0.2232
AT STEADY STATE CONDITION
APT
0.2535
TH
OH
0.1959
RH
0.2795
0.2710
Problem No: 8
A certain piece of equipment is inspected at the end of each day
and classified as just overhauled (1), good (2), fair (3) or
inoperative (4). If the item is inoperative it is overhauled, a
procedure takes one day. Assume that the working condition of
the equipment follows a Markov process with the following
state-transition matrix:
1  0 3 / 4 1/ 4 0 
2  0 1 / 2 1 / 2 0 
Today 
3 0
0 1 / 2 1 / 2


4 1.0 0
0
0 
It costs Rs. 125 to overhaul a machine on the avergae, and
Rs 75 is lost in production, if a machine is found
inoperative. Use steady state probabilities to compute the
expected per day cost of maintenance.
Problem No: 8
A unit is placed in different states 0,1,2 & 3 depending on the
quantity of products, when it is at 3, the company is replacing it
by a new machine with cost of Rs. 5000. When it is working at
states 1 or 2, reworking can be done at a cost Rs 1000 & 2000
respectively. What is the cost of the present policy. The
following transition matrix was found to hold good.
0 0.5 0.3 0.1 0.1
1  0 0.6 0.2 0.2
Pr esent 
2 0
0 0.5 0.5


3 1.0 0
0
0
Suppose that the company adopts the policy of replacing the
machines when it is in states 2 or 3. what is the cost of this
policy?
Problem No: 9
Two players A and B play a game with coins. Initially A has got
3 coins and B has got 2 coins. The game consists of A & B
showing one coin each. If the coin matches, then A wins both
the coins, otherwise B wins both. The game ends when one of
them do not have anymore coin to play. Identify the game as
markov chains and draw the transition diagram.
Absorbing Markov Chains:
A state in the Markov chain is called an absorbing state if once the
state is entered, it is impossible to leave.
Like regular markov chains, absorbing markov chains have the
property that the powers of the transition matrix approach a limiting
matrix.
Additionally,
A markov chain is an absorbing chain if
(a) There is at least an one absorbing state.
(b) It is possible to go from each non-absorbing state to at least one
absorbing state in a finite number of steps.
Identify any absorbing states for the following transition
matrix:
A 1
0
0
A 0 0 1
A) P  B 0.3 0.7 0  B) B 0 1 0
C  0 0.2 0.8 C 1 0 0
A 1
0
0
P  B 0.3 0.7 0 
C  0 0.2 0.8
We see that A = Absorbing state
B & C = are non absorbing state
1.0
A
0.3
B
0.7
C
0.8
From the transition diagram we infer that from state C we can go to State B, From
State B to State A. conversely from figure we see that we cannot we cannot move
from sate A to State B and from State B to State C. However we can move from State
B to State A, But cannot move to State C.
Explanation of how a stochastic process
(markov chain) can be applied for random walking
Case 1: Random walk with reflecting barrier.
a0
a1
a2
a3
a0  0
1
0
0
0
0

a1 0.5 0 0.5 0
0
0


a 2  0 0 .5 0 0 .5 0
0
P


a3  0
0 0 .5 0 0 .5 0 
a4  0
0
0 . 5 0 0 .5 


a5  0
0
0
0
1
0 
a4
a5
One can walk from a0 to a1
With a probability of 1.
Similarly on can walk back
from a1to a0 with a probability
of 0.5, similarly one can walk
from a1 to a2
with a probability of 0.5.
there is no other route from a1
to a3,a4 and a5.
Similarly other states can be
derived as shown in the
transition matrix.
Explanation of how a stochastic process
(markov chain) can be applied for random walking
Case 2: Random walk with absorbing barrier.
a0
a1
a2
a3
a4
a5
We assume that a man remains at the endpoints whenever he
reaches the states. This is also called markov chain with
absorbing barriers and the transition matrix is given by
a0  1
0
0
0
0
0

a1 0.5 0 0.5 0
0
0


a 2  0 0 .5 0 0 .5 0
0
P


a3  0
0 0 .5 0 0 .5 0 
a4  0
0
0 0 . 5 0 0 .5 


a5  0
0
0
0
0
1 
END
Markov Process
Let Xi be the weather of day i, 1 <= i <= t. We may
decide the probability of Xt+1 from Xi, 1 <= i <= t.
• Markov Property: Xt+1, the state of the system at time t+1 depends
only on the state of the system at time t
PrX t 1  x t 1 | X 1 …… X t  x1 ……x t   PrX t 1  x t 1 | X t  x t 
X1
X2
X3
X4
X5
•Stationary Assumption: Transition probabilities are independent of
time (t)
Pr X t1  b| X t  a   pab
Markov Process
Gambler’s Example
– Gambler starts with $10 (the initial state)
- At each play we have one of the following:
• Gambler wins $1 with probability p
• Gambler looses $1 with probability 1-p
– Game ends when gambler goes broke, or gains a fortune of $100
(Both 0 and 100 are absorbing states)
p
0
1
1-p
p
p
99
2
1-p
p
1-p
1-p
100
1-p
Start
(10$)
51
Markov Process
• Markov process - described by a stochastic FSM
• Markov chain - a random walk on this graph
(distribution over paths)
• Edge-weights give us
Pr X t1  b| X t  a   pab
• We can ask more complex questions, like PrX t 2  a | X t  b   ?
p
0
1
1-p
p
p
99
2
1-p
p
1-p
100
1-p
Start
(10$)
52
Markov Process
Coke vs. Pepsi Example
•Given that a person’s last cola purchase was Coke,
there is a 90% chance that his next cola purchase will
also be Coke.
•If a person’s last cola purchase was Pepsi, there is
an 80% chance that his next cola purchase will also be
Pepsi.
transition matrix:
coke
pepsi
0.9 0.1
P  0.2 0.8
pepsi


coke
0.1
0.9
coke
0.8
pepsi
0.2
53
Markov Process
Coke vs. Pepsi Example (cont)
Given that a person is currently a Pepsi purchaser,
what is the probability that he will purchase Coke two
purchases from now?
Pr[ Pepsi?Coke ] =
Pr[ PepsiCokeCoke ] + Pr[ Pepsi Pepsi Coke ] =
0.2 *
0.9
+
0.8 *
0.2
= 0.34
0.9
0.9 0.1
0.10.9 0.1 0.83 0.17
P 0.2 0.80.2 0.8  0.34 0.66
0.2 0.8
 

2
Pepsi  ?
?  Coke
54
Markov Process
Coke vs. Pepsi Example (cont)
Given that a person is currently a Coke purchaser,
what is the probability that he will buy Pepsi at the
third purchase from now?
0.9 0.10.83 0.17  0.781 0.219
P  0.2 0.80.34 0.66  0.438 0.562


 

3
55
Markov Process
Coke vs. Pepsi Example (cont)
• Assume each person makes one cola purchase per week
• Suppose 60% of all people now drink Coke, and 40% drink Pepsi
• What fraction of people will be drinking Coke three weeks from now?
0.9 0.1
P  0.2 0.8


 0.781 0.219
P 3  0.438 0.562


Pr[X3=Coke] = 0.6 * 0.781 + 0.4 * 0.438 = 0.6438
Qi - the distribution in week i
Q0= (0.6,0.4) - initial distribution
Q3= Q0 * P3 =(0.6438,0.3562)
56
Markov Process
Coke vs. Pepsi Example (cont)
Simulation:
2/3
3
Pr[Xi = Coke]
2
1
0.9 0.1 2

3 
3
1
0.2 0.8
3

stationary distribution
0.1
0.9
coke
0.8
pepsi
0.2
week - i
57
Supervised vs Unsupervised
 Decision tree learning is “supervised
learning” as we know the correct output of
each example.
 Learning based on Markov chains is
“unsupervised learning” as we don’t know
which is the correct output of “next letter”.
58
Implementation Using R
 msm (Jackson 2011) :handles Multi-State Models
for panel data;
 mcmcR (Geyer and Johnson 2013) implements
Monte Carlo Markov Chain approach;
 hmm (Himmelmann and www.linhi.com 2010) fits
hidden Markov models with covariates;
 mstate fits Multi-State Models based on Markov
chains for survival analysis (de Wreede, Fiocco,
and Putter 2011).
 markovchain
59
Implementaion using R
 Example1:Weather Prediction:
The Land of Oz is acknowledged not to have ideal
weather conditions at all: the weather is snowy or rainy
very often and, once more, there are never two nice days
in a row. Consider three weather states: rainy, nice and
snowy, Given that today it is a nice day, the corresponding
stochastic row vector is w0 = (0 , 1 , 0) and the forecast
after 1, 2 and 3 days.
Solution: please refer solution.R attached.
60
Source
 Slideshare
 Wikipedia
 Google
61