Markov Chains
Brian Carrico
The Mathematical Markovs
 Vladimir Andreyevich Markov (1871-1897)

 Andrey Markov’s younger brother
 With Andrey, developed the Markov brothers’
inequality
Andrey Andreyevich Markov Jr (1903-1979)
 Andrey Markov’s son
 One of the key founders of the Russian school
of constructive mathematics and logic
 Also made contributions to differential
equations,topology, mathematical logic and
the foundations of mathematics
 Which brings us to:
Andrey Andreyevich Markov
Андрей Андреевич Марков
June 14, 1856 – July 20, 1922
 Born in Ryazan
 (roughly 170 miles Southeast of
Moscow)
 Began Grammar School




in 1866
Started at St Petersburg
University in 1874
Defended his Masters
Thesis in 1880
Doctoral Thesis in 1885
Excommunicated from
the Russian Orthodox
Church
Precursors to Markov Chains
 Bernoulli Series
 Brownian Motion
 Random Walks
Bernoulli Series
 Jakob Bernoulli (1654-1705)
 Sequence independent random variables
X1, X2,X3,... such that
 For every i, Xi is either 0 or 1
 For every i, P(Xi)=1 is the same
 Markov’s first discussions of chains, a
1906 paper, considers only chains with
two states
 Closely related to Random Walks
Brownian Motion
 Described as early as 60 BC by Roman
poet Lucretius
 Formalized and officially discovered by
botanist Robert Brown in 1827
 The seemingly random movement of
particles suspended in a fluid
Random Walks
 Formalized in 1905 by Karl Pearson
 The formalization of a trajectory that consists of
taking successive random steps
 The results of random walk analysis have been
applied to computer
science, physics, ecology, economics, and a
number of other fields as a
fundamental model for random processes in
time
 Turns out to be a specific Markov chain
So what is a Markov Chain?
 A random process where all information
about the future is contained in the present
state
 Or less formally: a process where future
states depend only on the present state,
and are independent of past states
 Mathematically:
Applications of Markov Chains
 Science
 Statistics
 Economics and Finance
 Gambling and games of chance
 Baseball
 Monte Carlo
Science
 Physics
 Thermodynamic systems generally have timeinvariant dynamics
 All relevant information is in the state
description
 Chemistry
 An algorithm based on a Markov chain was
used to focus the fragment-based growth of
chemicals in silico towards a desired class of
compounds such as drugs or natural products
Economics and Finance
 Markov Chains are used model a variety
of different phenomena, including asset
prices and market crashes.
 Regime-switching model of James D.
Hamilton
 Markov Switching Multifractal asset pricing
model
 Dynamic macroeconomics
Gambling and Games of Chance
 In most card
games each
hand is
independent
 Board games
like Snakes and
Ladders
Baseball
 Use of Markov chain models in baseball
analysis began in 1960
 Each at bat can be taken as a Markov
chain
Monte Carlo
 A Markov chain with a large number of
steps is used to create the algorithm for
the basis of the Monte Carlo simulation
Statistics
 Many important statistics measure
independent trials, which can be
represented by Markov chains
An Example from Statistics
 A thief is in a dungeon with three identical doors. Once
the thief chooses a door and passes through it, the door
locks behind him. The three doors lead to:
 A 6 hour tunnel leading to freedom
 A 3 hour tunnel that returns to the dungeon
 A 9 hour tunnel that returns to the dungeon
 Each door is chosen with equal probability. When he is
dropped back into the dungeon by the second and third
doors there is a memoryless choice of doors. He isn’t
able to mark the doors in any way. What is his expected
time of escape?
 Note:
Example (cont)
 We plug the values in for xi and p(xi) to get:
 E(X)=6*(1/3)+x2*(1/3)+x3*(1/3)
 But what are x2 and x3?
 Because the decision is memoryless, the
expected time after returning from tunnels 2 or 3
doesn’t change from the initial expected time.
So, x2=x3=E(X).
 So,
 E(X)=6*(1/3)+E(X)*(1/3)+E(X)*(1/3)
 Now we’re back in Algebra 1
Sources
 Wikipedia
 The Life and Work of A.A. Markov.
Basharin, Gely P. et al.
http://decision.csl.illinois.edu/~meyn/pages
/Markov-Work-and-life.pdf
 Leemis (2009), Probability