The parallel replica method for Markov chains
David Aristoff (joint work with T. Lelièvre and G. Simpson)
Colorado State University
March 2015
D. Aristoff (Colorado State University)
March 2015
1 / 29
Introduction
Definition.
Let (Xn ) be a time homogeneous Markov chain:
Z
P(Xn+1 ∈ A | X0 , . . . , Xn−1 , {Xn = x}) =
K (x, dy ).
A
Our goal.
To efficiently simulate (Xn ) when it is metastable.
State space can be discrete:
random walk in random environment, Markov State Models
or continuous:
time discretization of molecular dynamics (MD)
(Xn ) can be reversible:
MCMC, time discretization of Brownian/Langevin MD
or non-reversible:
stochastic MD with flow/external forces
We are interested in both dynamics and equilibrium!
D. Aristoff (Colorado State University)
March 2015
2 / 29
Introduction
Metastability can arise from energetic barriers or entropic barriers.
entropic barrier
S
V = 0 in blue region
V = ∞ in red region
Example: Energy (left) and entropic (right) barriers for Brownian/Langevin
dynamics. Here, S is a metastable subset of state space.
D. Aristoff (Colorado State University)
March 2015
3 / 29
The QSD
Definition.
If ν is a probability measure supported in S with
ν(·) = P(Xn ∈ · | X0 ∼ ν, X1 , . . . , Xn ∈ S),
n ≥ 1,
then ν is a quasi-stationary distribution (QSD) in S.
If for any µ supported in S and any A ⊂ S,
ν(A) = lim P(Xn ∈ A | X0 ∼ µ, X1 , . . . , Xn ∈ S),
n→∞
then ν is the unique QSD in S.
Intuitively: (Xn ) reaches ν after spending a “long” time in S without leaving.
Our setting: Time scale to reach ν time scale to leave S.
We need to understand exit events from S, starting from ν.
D. Aristoff (Colorado State University)
March 2015
4 / 29
The QSD
Let ν be the QSD in S and τ the first exit time from S.
Theorem.
If X0 ∼ ν, then τ is geometrically distributed, and τ , Xτ are independent.
The geometric distribution is
P(τ > n) = (1 − p)n ,
p = P(X1 ∈
/ S | X0 ∼ ν).
It is memoryless:
P(τ > n + k | τ > k) = P(τ > n).
D. Aristoff (Colorado State University)
March 2015
5 / 29
The QSD
S
ν
Figure: Serial simulation of an exit event from S, starting from ν.
D. Aristoff (Colorado State University)
March 2015
6 / 29
The QSD
S
ν
ν
ν
Figure: Parallel simulation of an exit event from S, starting from ν.
D. Aristoff (Colorado State University)
March 2015
7 / 29
The QSD
S
ν
ν
ν
Figure: Parallel simulation of an exit event from S, with a tie.
D. Aristoff (Colorado State University)
March 2015
8 / 29
The QSD
S
ν
ν
ν
Figure: Parallel simulation of an exit event from S, with a tiebreaker.
D. Aristoff (Colorado State University)
March 2015
9 / 29
The Parallel Step
Parallel Step. Choose a polling time Tpoll , set M = 1 and iterate:
1. Start N i.i.d. replicas of (Xn ) at independent samples of ν.
2. Evolve the replicas from time (M − 1)Tpoll to time MTpoll .
3. If no replica leaves S during this time, update M = M + 1 and return to 2.
Else, stop and compute an exit time, τacc , and exit point, Xacc .
D. Aristoff (Colorado State University)
March 2015
10 / 29
The Parallel Step
Tpol l
2Tpol l
...
(M − 1)Tpoll
MTpol l
replica 1
replica 2
..
.
replica K − 1
replica K
replica K + 1
..
.
replica N
time
0
D. Aristoff (Colorado State University)
NTpol l
2NTpol l
...
(M − 1)NTpoll
τacc
March 2015
11 / 29
The Parallel Step
Tpol l
2Tpol l
...
(M − 1)Tpoll
MTpol l
replica 1
replica 2
..
.
replica K − 1
replica K
replica K + 1
..
.
replica N
time
0
D. Aristoff (Colorado State University)
NTpol l
2NTpol l
...
(M − 1)NTpoll
τacc
March 2015
12 / 29
The Parallel Step
Tpol l
2Tpol l
...
(M − 1)Tpoll
MTpol l
replica 1
replica 2
..
.
replica K − 1
replica K
replica K + 1
..
.
replica N
time
0
D. Aristoff (Colorado State University)
NTpol l
2NTpol l
...
(M − 1)NTpoll
τacc
March 2015
13 / 29
The Parallel Step
Tpol l
2Tpol l
...
(M − 1)Tpoll
MTpol l
replica 1
replica 2
..
.
replica K − 1
replica K
replica K + 1
..
.
replica N
time
0
D. Aristoff (Colorado State University)
NTpol l
2NTpol l
...
(M − 1)NTpoll
τacc
March 2015
14 / 29
The Parallel Step
Tpol l
2Tpol l
...
(M − 1)Tpoll
MTpol l
replica 1
replica 2
..
.
replica K − 1
replica K
replica K + 1
..
.
replica N
time
0
D. Aristoff (Colorado State University)
NTpol l
2NTpol l
...
(M − 1)NTpoll
τacc
March 2015
15 / 29
The Parallel Step
Tpol l
2Tpol l
...
(M − 1)Tpoll
MTpol l
τK
replica 1
replica 2
..
.
replica K − 1
replica K
replica K + 1
..
.
replica N
time
0
NTpol l
2NTpol l
...
(M − 1)NTpoll
τacc
τacc = sum of lengths of blue lines = (M − 1)NTpoll + (K − 1)Tpol l + [τ K − (M − 1)Tpoll]
Xacc = position of K-th replica at red cross (time τ K )
D. Aristoff (Colorado State University)
March 2015
16 / 29
The Parallel Step
Parallel Step. Choose a polling time Tpoll , set M = 1 and iterate the following.
1. Start N i.i.d. replicas of (Xn ) at independent samples of ν.
2. Evolve the replicas from time (M − 1)Tpoll to time MTpoll .
3. If no replica leaves S during this time, update M = M + 1 and return to 2.
Else, record the exit event (τacc , Xacc ).
Theorem.
The Parallel Step is exact:
(τacc , Xacc ) ∼ (τ, Xτ ),
with τ and Xτ the first exit time and exit point from S, starting at ν.
D. Aristoff (Colorado State University)
March 2015
17 / 29
The Main Algorithm
Let S be a collection of disjoint subsets of state space.
Let Tcorr = Tcorr (S) be ≈ the time it takes to reach the QSD in S.
Main Algorithm.
1. Evolve (Xn ) until it spends time Tcorr in some set S ∈ S without leaving.
2. Run the Parallel Step in S. Then, return to 1.
Let Π be the quotient map which mods out state space by S.
Coarse dynamics.
The Main Algorithm produces an approximation (X̃n ) of (Π(Xn )) via
X̃n = Π(Xn ) in Step 1,
D. Aristoff (Colorado State University)
X̃n = Π(S) in Step 2.
March 2015
18 / 29
The Main Algorithm
Main Algorithm.
1. Evolve (Xn ) until it spends time Tcorr in some set S ∈ S without leaving.
2. Run the Parallel Step in S. Then, return to 1.
Let f be a function on state space and π the equilibrium distribution of (Xn ).
Equilibrium averages.
The Main Algorithm produces an approximation
fsim = fsim + f (Xn ) in Step 1,
fsim
Tsim
of
R
f dπ via
fsim = fsim + facc in Step 2,
where facc is the sum of values of f over replicas in the Parallel Step. In Step 1,
Tsim increases by 1 for each time step; in Step 2, Tsim is increased by τacc .
D. Aristoff (Colorado State University)
March 2015
19 / 29
The Main Algorithm
Tpol l
2Tpol l
...
(M − 1)Tpoll
MTpol l
τK
replica 1
replica 2
..
.
replica K − 1
replica K
replica K + 1
..
.
replica N
time
τacc = sum of lengths of blue lines
facc = sum of values of f over positions of replicas along blue lines
D. Aristoff (Colorado State University)
March 2015
20 / 29
The Main Algorithm
Idealization.
After spending Tcorr consecutive time steps in S, (Xn ) exactly reaches the QSD.
Assumption.
(Xn ) is uniformly ergodic:
lim sup ||P(Xn ∈ · | X0 = x) − π||TV = 0.
n→∞ x
Theorem.
Under the Idealization, the Main Algorithm generates exact coarse dynamics:
(X̃n ) ∼ (Π(Xn )).
Under the Idealization and Assumption, the Main Algorithm generates exact
equilibrium averages:
Z
fsim a.s.
lim
= f dπ.
Tsim →∞ Tsim
D. Aristoff (Colorado State University)
March 2015
21 / 29
Numerics
200
180
160
140
S2
120
100
80
S1
60
pathways
barrier
40
20
0
−100
−50
0
50
100
150
200
Figure: A random walk with entropic barriers. At each step it moves up, down, left or
right at random, unless that results in crossing a barrier.
D. Aristoff (Colorado State University)
March 2015
22 / 29
Numerics
75
92
91
70
90
89
hyi
hxi
65
60
88
87
55
86
50
45
85
0
2000
4000
84
6000
0
2000
4000
6000
Tcorr (S1 )
Tcorr (S1 )
0.41
25
time sp eedup
0.4
hfi
0.39
0.38
0.37
20
15
0.36
0.35
0
2000
4000
Tcorr (S1 )
6000
10
0
2000
4000
6000
Tcorr (S1 )
Figure: Equilibrium averages vs. Tcorr when N = 100. Here f = 1(x,y )∈upper half of right box .
D. Aristoff (Colorado State University)
March 2015
23 / 29
Numerics
6
decorrelation steps
time sp eedup
10
8
6
4
2
x 10
10
12
0
20
40
60
80
9.5
9
8.5
100
0
20
40
N
60
80
100
60
80
100
N
7
580
2
x 10
560
parallel lo ops
parallel steps
570
550
540
530
520
1.5
1
0.5
510
500
0
20
40
60
N
80
100
0
0
20
40
N
Figure: Parallel step stats when Tcorr = largest value from last slide and Tsim = 5 × 109 .
D. Aristoff (Colorado State University)
March 2015
24 / 29
Numerics
1.6
1.4
1.2
slope = 0.10
1
slope = 0.05
0.8
0.6
0.4
0.2
0
0
10
S1
20
30
S2
40
50
60
S3
Figure: A random walk with energetic barriers. At each step, it moves to the left with
probability 1/2 + slope, and to the right with probability 1/2 − slope.
D. Aristoff (Colorado State University)
March 2015
25 / 29
Numerics
30
0.25
28
0.2
26
hfi
hxi
0.15
24
0.1
22
0.05
20
18
0
20
40
0
60
0
20
Tcorr (S3 )
40
60
Tcorr (S3 )
time sp eedup
70
65
60
55
50
0
20
40
60
Tcorr (S3 )
Figure: Equilibrium averages vs. Tcorr when N = 100. Here, f = 1y ∈right half of state space .
D. Aristoff (Colorado State University)
March 2015
26 / 29
Numerics
5
time sp eedup
80
60
40
20
0
x 10
10.2
decorrelation steps
100
0
100
200
300
400
500
10
9.8
9.6
9.4
9.2
0
100
200
N
300
400
500
400
500
N
6
x 10
10200
5
10000
parallel lo ops
parallel steps
10100
9900
9800
9700
9600
4
3
2
1
9500
9400
0
100
200
300
N
400
500
0
0
100
200
300
N
Figure: Parallel step stats when Tcorr = largest value from last slide and Tsim = 109 .
D. Aristoff (Colorado State University)
March 2015
27 / 29
Conclusion
Remarks.
Our algorithm applies to any Markov chain.
It is efficient when for S ∈ S,
Time scale to reach QSD in S Time scale to leave S.
S need not be known a priori. Sets in S can be identified on the fly.
Future work.
(Xn ) only approximately reaches ν. How does this error propagate?
When can S be efficiently identified on the fly?
D. Aristoff (Colorado State University)
March 2015
28 / 29
Conclusion
Acknowledgements
Collaborators T. Lelièvre (École des Ponts ParisTech), G. Simpson (Drexel
University)
Funding DOE Award de-sc0002085 (Thanks to Mitch Luskin for the
support)
http://www.math.colostate.edu/~aristoff/
D. Aristoff (Colorado State University)
March 2015
29 / 29