Stochastic Processes
Nancy Griffeth
January 10, 2014
Funding for this workshop was provided by the program “Computational Modeling and Analysis of Complex Systems,” an
NSF Expedition in Computing (Award Number 0926200).
Motivation

Continuous solutions depend on large
numbers (e.g., ODE’s)

Stochastic solutions are better for small
numbers

Strange and surprising things happen!
Game


Two players each have a pile of pennies
Taking turns:




Player flips a penny
Heads, he gets a penny from the other
player
Tails, he gives the penny to the other
player
Until one player runs out of pennies
Questions




What each player’s chance of winning?
What is the expected value of the game
to a player?
How long will the game go on before a
player runs out of pennies?
What is the probability after m moves
that player A has a pennies and player
B has b pennies?

Or the probability distribution over (a,b)?
Chance of winning


If A and B have the same number of
coins, they’re equally likely to win.
With m coins to player A and n coins to
player B



A wins with probability m/(m+n)
B wins with probability n/(m+n)
The player with more coins is more
likely to win.
Expected Value

Definition:




Exp(winnings)=Prob(winning)*value of winning
+Prob(losing)*value of losing
Value of winning is m+n
Value of losing is
0
So, if A starts with m coins and B with n:


E(A winnings) = m/(m+n)*(m+n) + n/(m+n)*0
=m
E(B winnings) = n/(m+n)*(m+n) + m/(m+n)*0
=n
Markov Chain
3,3
2,4
1,5
2,4
1,5
0,6
0,6
The probability of each transition is ½,
except 0,6->0,6 and 6,0->6,0, where
the probability is 1
How long



Relatively hard from the UCLA tutorial
Sum over n of
n*[p((5,1);n)*0.5+p((1,5);n)*0.5] = Sum
over n of n*p((5,1;n)
Starting from (3,3), about 8 turns
Probability Matrix
0,6 1,5 2,4 3,3 4,2 5,1 6,0
0,6
1
0
0
0
0
0
0
1,5
0.5
0
0.5
0
0
0
0
2,4
0
0.5
0
0.5
0
0
0
3,3
0
0
0.5
0
0.5
0
0
4,2
0
0
0
0.5
0
0.5
0
5,1
0
0
0
0
0.5
0
0.5
6,0
0
0
0
0
0
0
1
Two Steps
0,6 1,5 2,4 3,3 4,2 5,1 6,0
0,6
1
0
0
0
0
0
0
1,5
0.5
0.25
0
0.25
0
0
0
2,4
0.25
0
0.5
0
0.25
0
0
3,3
0
0.25
0
0.5
0
0.25
0
4,2
0
0
0.25
0
0.5
0
0.25
5,1
0
0
0
0.25
0
0.25
0.5
6,0
0
0
0
0
0
0
1
Three Steps
0,6 1,5 2,4 3,3 4,2 5,1 6,0
0,6
1
0
0
0
0
0
0
1,5
0.625
0
0.25
0
0.125
0
0
2,4
0.25
0.25
0
0.375
0
0.125
0
3,3
0.125
0
0.375
0
0.375
0
0.125
4,2
0
0.125
0
0.375
0
0.25
0.25
5,1
0
0
0.125
0
0.25
0
0.625
6,0
0
0
0
0
0
0
1
64 Steps
0,6 1,5 2,4 3,3 4,2 5,1 6,0
0,6
1
0
0
0
0
0
0
1,5
0.833
0
0
0
0
0
0.167
2,4
0.667
0
0
0
0
0
0.333
3,3
0.5
0
0
0
0
0
0.5
4,2
0.333
0
0
0
0
0
0.667
5,1
0.167
0
0
0
0
0
0.833
6,0
0
0
0
0
0
0
1
Steady States

Sooner or later, somebody wins (with
probability=1)

Either party can win

The system has two stable steady
states (bistable)
The “Game” with Molecules
L
2
L
1
L
0
L
0
R
10
R
9
R
8
R
8
L.R
0
L.R
1
L.R
2
L.R
0
L.R.R.L
0
L.R.R.L
0
L.R.R.L
0
L.R.R.L
1
Chemical Master Equation

A set of first-order differential equations
describing the change in the probability
of states with time t.

With time-independent reaction rates,
the process is Markovian.
The Chemical Master Equation
We are interested in p(x; t), the probability that
the chemical system will be in state x at time
t.
The time evolution of p(x; t) is described by the
Chemical Master Equation:
Where sμ is the stoichiometric vector for reaction μ,
giving the changes in each type of molecule as a
result of μ, and aμ is the propensity for reaction μ
Bistability

A steady state is a state that a system
tends to stay in once it reaches it

A steady state can be stable or unstable

A system that has two stable steady
states is called bistable.