Chapter 10
Markov Chains
Section 10.1
Basic Properties
of Markov Chains
Some Background Information
 Mathematical
models that evolve
over time in a probabilistic manner
are called stochastic processes.
A
special kind of stochastic process is
a Markov Chain, where the outcome
of an experiment depends only on
the outcome of the previous
experiment.
Why Study Markov Chains?
 Markov
chains are used to analyze
trends and predict the future.
(Weather, stock market, genetics,
product success, etc.
A Sociology Example
Sociologists classify people by income as lower-class,
middle-class, and upper-class. They have found that
the strongest determinant of the income class of an
individual is the income class of that person’s parents.
Transition Diagrams
Transition Matrices
Characteristics of a
Transition Matrix
 It
is a square matrix.
 All entries are between 0 and 1
(because all the entries are
probabilities).
 The sum of the entries in any row
must be 1.
 Denoted by P .
Key Features of Markov Chains
A
sequence of trials of an experiment is
a Markov chain if
1.) the outcome of each experiment
is one of a set of discrete states;
2.) the outcome of an experiment
depends only on the present state,
and not on any past states;
3.) the transition probabilities remain
constant from one transition to the
next.
Transition Probabilities
from One State to Another
 The
transition matrix shows the
probabilities of moving from state-tostate from the current generation to
the next.
k
P
gives the probabilities of a
transition from one state to another
in k repetitions of an experiment,
provided the transition probabilities
remain constant from one repetition
to the next.
Example 1

Use the transition matrix from the
Sociology example to find the probabilities
of change for the grandchildren of the
current generation.
Example 2

Write a transition matrix for the following
situation. Then find the probabilities associated
with a third and fourth purchase from the
bookstores.
In a study of the market share of the three
bookstores in a university town, it was found that
75% of those who had bought from University
Bookstore would buy from it again, 15% from
Campus Bookstore and 10% from Bookmart.
Of those who bought from Campus Bookstore,
90% would buy from it again and 5% each would
buy from University Bookstore and Bookmart.
85% of those who bought from Bookmart would
buy from it again, 5% from University Bookstore
and 10% from Campus Bookstore.
Probability Vector (Matrix)
 When
a study is first begun, the
probabilities of the states are called
the initial distribution.
 This distribution is written as a
matrix of only one row.
 Denoted by X0 .
Characteristics of a
Probability Vector
 It
is a row matrix.
 Each entry must be between 0 and 1
inclusive.
 The sum of the entries of the row
must be 1.
Making Predictions about the
Population Proportion with Markov Chains
1.) Create a probability vector, X0 .
The entries are the initial probabilities of the states.
2.) Create the transition matrix, P .
The entries are the probabilities of passing from
current states (rows) to the first following states
(columns).
n
3.) Calculate X0 P
This is the probability vector after n repetitions of
the experiment. In other words, it is a prediction
for the proportion of the population after n
repetitions.
Note: The columns and rows of the probability vector and the
transition matrix must be labeled the same.
Example 3

A marketing analysis shows that KickKola
currently commands 14% of the cola
market.
Further analysis indicates that 12% of
the consumers who do not currently
drink KickKola will purchase KickKola the
next time they buy a cola (in response to
a new advertising campaign) and that
63% of the consumers who currently
drink KickKola will purchase it the next
time they buy a cola.
Predict KickKola’s market share at:
a.) the next following purchase
b.) the second following purchase
Example 4

It has been noted that George the golfer tends to repeat
himself. In his first shot, he will hit his ball in the fairway
(F), in the rough (R), or out of bounds (B).
On his first shot he hits in the fairway 60% of the time, in
the rough 30% of the time, and out of bounds 10% of the
time.
However, on subsequent shots, if he hit in the fairway the
first time, he will hit in the fairway next time 90% of the
time and in the rough 10% of the time.
If he hit in the rough the first time, he will hit the rough
next time 50% of the time and out of bounds 5% of the
time.
If he hits out of bounds the first time, he will hit out of
bounds next time 70% of the time and in the fairway 15%
of the time.
Find the probability that his next following shot is in:
a.) the fairway
b.) the rough
c.) out of bounds