(Non)Equilibrium computation of free energy differences using Langevin dynamics Gabriel STOLTZ CERMICS & MICMAC project team, Ecole des Ponts ParisTech http://cermics.enpc.fr/∼stoltz/ Edinburgh, june 2010 – p. 1/33 Outline of the talk A brief presentation of methods to compute free energy differences Thermodynamic integration using Langevin dynamics Nonequilibrium Langevin dynamics Edinburgh, june 2010 – p. 2/33 Computing free energy differences Edinburgh, june 2010 – p. 3/33 Microscopic description of a classical system Positions q (configuration), momenta p = M q̇ (M diagonal mass matrix) Microscopic description of a classical system (N particles): (q, p) = (q1 , . . . , qN , p1 , . . . , pN ) ∈ E For instance, E = T ∗ D = D × R3N with D = R3N or T3N More complicated situations can be considered... (constraints defining submanifolds of the phase space) N X p2i + V (q1 , . . . , qN ) Hamiltonian H(q, p) = 2m i i=1 All the physics is contained in V Canonical probability measure: µ(dq dp) = Z −1 −βH(q,p) e dq dp, 1 β= kB T Edinburgh, june 2010 – p. 4/33 Sampling the canonical measure The aim is to compute an approximation of the high dimensional integral Z hAi = A(q, p) µ(dq dp) T ∗D Restated as a one-dimensional integral using ergodic properties of an irreducible dynamics for which the canonical measure is invariant: 1 lim T →+∞ T Z 0 T A(qt , pt ) dt = Z A(q, p) µ(dq dp) a.s. T ∗D Overdamped Langevin dynamics (momenta trivial to sample) dqt = −∇V (qt ) dt + r 2 dWt β Zero mass limit of the Langevin dynamics or the limit of the Langevin dynamics when the friction goes to infinity (with suitable time rescaling) Edinburgh, june 2010 – p. 5/33 Langevin dynamics Stochastic perturbation of the Hamiltonian dynamics dq = M −1 p dt t t dpt = −∇V (qt ) dt−γ(qt )M −1 pt dt + σ(qt ) dWt 2 Fluctuation/dissipation relation σσ = γ β T Invariance of the canonical measure when it is a stationary solution of the Fokker-Planck equation ∂t ψ = L∗ ψ with eβH −βH L = {·, H} + ∇p · divp γe β T T and {A1 , A2 } = (∇A1 )T J∇A2 = (∇q A1 ) ∇p A2 − (∇p A1 ) ∇q A2 Irreducibility amounts to controllability (Hörmander condition) Numerical schemes obtained by a splitting strategy for instance (Verlet scheme + partial randomization of momenta) Edinburgh, june 2010 – p. 6/33 Metastability (1) Numerical discretization of the overdamped Langevin dynamics: q n+1 n n = q − ∆t∇V (q ) + s 2∆t n G β 1.5 1.5 1 1.0 x coordinate y coordinate where G n ∼ N (0, IddN ) i.i.d. 0.5 0 −0.5 −1 0.5 0.0 −0.5 −1.0 −1.5 −1.5 −1 −0.5 0 0.5 X coordinate 1 1.5 −1.5 0.0 2000 4000 6000 8000 10000 Time Projected trajectory in the x variable for ∆t = 0.01, β = 8. Edinburgh, june 2010 – p. 7/33 Metastability (2) Although the trajectory average converges to the phase-space average, the convergence may be slow... Slowly evolving macroscopic function of the microscopic degrees of freedom: reaction coordinate ξ(q) ∈ Rm with m ≪ N Two origins : energetic or entropic barriers (in fact, free energy barriers) y coordinate x coordinate 8 4 0 −4 x coordinate (a) Entropic barrier. −8 0.0 5000 10000 15000 20000 Time (b) Associated trajectory. Edinburgh, june 2010 – p. 8/33 Metastability (3) 1.5 1.5 1 1.0 x coordinate y coordinate Assume the free energy F associated with the slow direction x has been computed, and sample the modified potential V(x, y) = V (x, y) − F (x). 0.5 0 −0.5 0.0 −0.5 −1.0 −1 −1.5 −1.5 −1.5 0.5 −1 −0.5 0 0.5 x coordinate 1 1.5 0.0 2000 4000 6000 8000 10000 Time Projected trajectory in the x variable for ∆t = 0.01, β = 8. Many more transitions! The variable x is uniformly distributed. Reweighting with weights e−βF (x) to compute canonical averages Compute efficiently the free energy? Edinburgh, june 2010 – p. 9/33 Computation of free energy differences Alchemical transition: indexed by an external parameter λ (force field parameter, magnetic field,...) Z −βH1 (q,p) e dq dp ; ZT ∗ D ∆F = −β −1 ln −βH0 (q,p) e dq dp T ∗D Typically, Hλ = (1 − λ)H0 + λH1 . Other parametrizations possible (see Gelman and Meng, Stat. Sci., 1998) (given) reaction coordinate ξ : R3N → Rm (angle, length,. . . ): Z −βH(q,p) e δξ(q)−z1 (dq) dp Σ(z)×R3N −1 . Z ∆F = −β ln −βH(q,p) e δξ(q)−z0 (dq) dp Σ(z)×R3N n with Σ(z) = q ∈ D o ξ(q) = z Edinburgh, june 2010 – p. 10/33 Cartoon comparison of the methods (reaction coordinate case) (a) (c) Histogram method Nonequilibrium switching dynamics (b) Thermodynamic integration (d) Adaptive dynamics Edinburgh, june 2010 – p. 11/33 Some elements on the scientific landscape We focus on the reaction coordinate case Histogram methods: WHAM (Kumar et al.), MBAR (Chodera/Shirts) Thermodynamic integration in the Hamiltonian case (Carter et al., den Otter/Briels, Sprik/Ciccotti) and HMC (Hartmann/Schütte) or for overdamped Langevin dynamics (Ciccotti/Lelièvre/Vanden-Eijnden) Nonequilibrium methods: overdamped case (Lelièvre/Rousset/Stoltz) or steered versions (potentials Vλ (q) = V (q) + K(ξ(q) − λ)2 ) Adpative methods: adaptive biasing force (Darve/Pohorille, Chipot/Hénin), nonequilibrium metadynamics (Bussi/Laio/Parrinello), Wang-Landau, self-healing umbrella sampling (Marsili et al., Dickson et al.), etc Aims of this work: Thermodynamic integration with Langevin dynamics Nonequilibrium Langevin dynamics Edinburgh, june 2010 – p. 12/33 Thermodynamic integration with Langevin dynamics Edinburgh, june 2010 – p. 13/33 Constrained Langevin dynamics (1) Consider the following Langevin process: −1 dq = M pt dt, t dpt = −∇V (qt ) dt − γ(qt )M −1 pt dt + σ(qt ) dWt + ∇ξ(qt ) dλt , ξ(qt ) = z 2 Standard fluctuation/dissipation relation σσ = γ β T dξ(qt ) Hidden velocity constraint: = vξ (qt , pt ) = ∇ξ(qt )T M −1 pt = 0 dt The corresponding phase-space is Σξ,vξ (z, 0) where n o 6N Σξ,vξ (z, vz ) = (q, p) ∈ R ξ(q) = z, vξ (q, p) = vz An explicit expression of the Lagrange multiplier can be found by computing the second derivative in time of the constraint Edinburgh, june 2010 – p. 14/33 Constrained Langevin dynamics (2) Reformulation of the constrained Langevin dynamics as dqt = M −1 pt dt, dpt = −∇V (qt ) dt + ∇ξ(qt )f M (qt , pt ) dt − γP (qt )M −1 pt dt + σP (qt ) dWt , rgd Constraining force (projection of the conservative force + centrifugal term) −1 M T −1 −1 −1 frgd (q, p) = G−1 (q)∇ξ(q) M ∇V (q) − G (q)Hess (ξ)(M p, M p) q M M where GM (q) = ∇ξ(q)T M −1 ∇ξ(q) Projected fluctuation/dissipation matrices T (σP , γP ) := (PM σ, PM γ PM ) T −1 where PM (q) = Id − ∇ξ(q)G−1 (q)∇ξ(q) M M Edinburgh, june 2010 – p. 15/33 Properties of the projected Langevin dynamics Generator of the dynamics: L = Lham + Lthm with 1 Lham = { · , H}Ξ , Lthm = eβH divp e−βH γP ∇p · β Invariant measure −1 −βH(q,p) µΣξ,vξ (z,0) (dq dp) = Zz,0 e σΣξ,vξ (z,0) (dq dp), where σΣξ,vξ (z,vz ) (dq dp) is the phase space Liouville measure of Σξ,vξ (z, vz ) induced by the symplectic matrix J Reversibility: Law(qt , pt ; 0 ≤ t ≤ T ) = Law(qT −t , −pT −t ; 0 ≤ t ≤ T ) Detailed balance condition (momentum flip S(q, p) = (q, −p)) Z Z ϕ1 LΞ (ϕ2 ) dµΣξ,vξ (z,0) = (ϕ2 ◦ S) LΞ (ϕ1 ◦ S) dµΣξ,vξ (z,0) Σξ,vξ (z,0) Σξ,vξ (z,0) Ergodicity (convergence of the longtime trajectorial averages) The normalizing constant of this canonical distribution defines a rigid free energy (more on this later) Edinburgh, june 2010 – p. 16/33 Numerical discretization Splitting into Hamiltonian part + constrained Ornstein-Uhlenbeck process Midpoint scheme for the momenta (reversible for the canonical measure with constraints) r ∆t ∆t γ M −1 (pn + pn+1/4 ) + σ G n + ∇ξ(q n ) λn+1/4 , pn+1/4 = pn − 4 2 with the constraint ∇ξ(q n )T M −1 pn+1/4 = 0 RATTLE scheme (symplectic) ∆t n n n+1/2 n+1/4 n+1/2 ∇V (q ) + ∇ξ(q ) λ , p = p − 2 q n+1 = q n + ∆t M −1 pn+1/2 , pn+3/4 = pn+1/2 − ∆t ∇V (q n+1 ) + ∇ξ(q n+1 ) λn+3/4 , 2 with ξ(q n+1 ) = z and ∇ξ(q n+1 )T M −1 pn+3/4 = 0 ∆t Overdamped limit obtained when γ = M ∝ Id 4 Metropolization of the RATTLE part to eliminate the time-step error Edinburgh, june 2010 – p. 17/33 Thermodynamic integration (1) The free energy can be estimated from constrained samplings as Z 1 e−βH(q,p) δξ(q)−z (dq) dp F (z) = − ln β Σ(z)×R3N Z 1 M (z) − ln = Frgd (det GM )−1/2 dµΣξ,vξ (z,0) + C β Σξ,v (z,0) ξ with the rigid free energy (constraints on both q and p) Z 1 M e−βH(q,p) dµΣξ,vξ (z,0) Frgd (z) = − ln β Σξ,v (z,0) ξ Extension to the case of molecular constraints Thermodynamic integration through the computation of the mean force Z M M (q, p) µΣξ,vξ (z,0) (dq dp) frgd (z) = ∇z Frgd Σξ,vξ (z,0) Edinburgh, june 2010 – p. 18/33 Thermodynamic integration (2) Longtime (a.s.) convergence 1 lim T →+∞ T Z 0 T M dλt = ∇z Frgd (z) No second order derivatives of ξ needed! Variance reduction: keep only the Hamiltonian part of λt Numerical discretization: approximate the mean force using only the Lagrange multipliers from the RATTLE part: N−1 N−1 X X 1 1 M n n M frgd (q , p ) ≃ (λn+1/2 + λn+3/4 ) ∇z Frgd (z) ≃ N n=0 N∆t n=0 Consistency result λn+1/2 + λn+3/4 ∆t M n n+1/4 M = frgd (q , p ) + frgd (q n+1 , pn+3/4 ) + O(∆t3 ) 2 Edinburgh, june 2010 – p. 19/33 Application: Solvatation effects on conformational changes (1) 2 2 (r − r0 − w) Two particules (q1 ,q2 ) interacting through VS (r) = h 1 − w2 Solvent: particules through the purely repulsive potential interacting σ 12 σ 6 VWCA (r) = 4ǫ − + ǫ if r ≤ r0 , 0 if r > r0 r r |q1 − q2 | − r0 Reaction coordinate ξ(q) = , compact state ξ −1 (0), 2w stretched state ξ −1 (1) 111 000 000 111 000 111 000 111 000 111 000 111 000 111 1111 0000 0000 1111 0000 1111 0000 1111 0000 1111 0000 1111 111 000 000 111 000 111 000 111 000 111 000 111 111 000 000 111 000 111 000 111 000 111 000 111 000 111 111 000 000 111 000 111 000 111 000 111 000 111 1111 0000 0000 1111 0000 1111 0000 1111 0000 1111 0000 1111 111 000 000 111 000 111 000 111 000 111 000 111 000 111 111 000 000 111 000 111 000 111 000 111 000 111 111 000 000 111 000 111 000 111 000 111 000 111 111 000 000 111 000 111 000 111 000 111 000 111 1111 0000 0000 1111 0000 1111 0000 1111 0000 1111 0000 1111 111 000 000 111 000 111 000 111 000 111 000 111 111 000 000 111 000 111 000 111 000 111 000 111 111 000 000 111 000 111 000 111 000 111 000 111 1111 0000 0000 1111 0000 1111 0000 1111 0000 1111 0000 1111 0000 1111 111 000 000 111 000 111 000 111 000 111 000 111 1111 0000 0000 1111 0000 1111 0000 1111 0000 1111 0000 1111 111 000 000 111 000 111 000 111 000 111 000 111 111 000 000 111 000 111 000 111 000 111 000 111 111 000 000 111 000 111 000 111 000 111 000 111 000 111 111 000 000 111 000 111 000 111 000 111 000 111 111 000 000 111 000 111 000 111 000 111 000 111 000 111 1111 0000 0000 1111 0000 1111 0000 1111 0000 1111 0000 1111 1111 0000 0000 1111 0000 1111 0000 1111 0000 1111 0000 1111 1111 0000 0000 1111 0000 1111 0000 1111 0000 1111 0000 1111 0000 1111 111 000 000 111 000 111 000 111 000 111 000 111 000 111 Edinburgh, june 2010 – p. 20/33 Application: Solvatation effects on conformational changes (2) 0.0 20 −0.5 Free energy Mean force 10 0 −10 −1.5 −2.0 −20 −30 −0.2 −1.0 −2.5 0.0 0.2 0.4 0.6 Reaction coordinate 0.8 1.0 1.2 −0.2 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Reaction coordinate Left: Estimated mean force. Right: Corresponding potential of mean force. Parameters: β = 1, N = 100 particles, solvent density ρ = 0.436, WCA interactions σ = 1 and ε = 1, dimer w = 2 and h = 2. Mean force estimated at the values zi = zmin + i∆z, with zmin = −0.2, zmax = 1.2 and ∆z = 0.014, by ergodic averages obtained with the projected dynamics with Metropolis correction (time T = 2 × 104 , step size ∆t = 0.02, scalar friction γ = 1). Edinburgh, june 2010 – p. 21/33 Nonequilibrium Langevin dynamics Edinburgh, june 2010 – p. 22/33 Presentation of the dynamics (1) Idea: start at equilibrium and perform a switching from the initial to the final state in a finite time T Schedule z(t) for t ∈ [0, T ] and nonequilibrium dynamics −1 dq = M pt dt, t dpt = −∇V (qt ) dt − γP (qt )M −1 pt dt + σP (qt ) dWt + ∇ξ(qt ) dλt , ξ(q ) = z(t), (Cq (t)) t with equilibrium initial conditions (q0 , p0 ) ∼ µΣξ,vξ (z(0),ż(0)) (dq dp) T Projected fluctuation/dissipation relation (σP , γP ) := (PM σ, PM γ PM ) so that the noise act only in the direction orthogonal to ∇ξ Hidden constraint on the reaction coordinate velocity vξ (q, p) = ż(t) M A computation shows dλt = frgd (qt , pt ) dt + G−1 M (qt )z̈(t) dt Not the same dynamics as in Latorre/Hartmann/Schütte Edinburgh, june 2010 – p. 23/33 Presentation of the dynamics (2) Backward dynamics: start at equilibrium for the final value of the schedule, switching with a time-reversed schedule t′ 7→ z(T − t′ ) Initial conditions (q0b , pb0 ) ∼ µΣξ,vξ (z(T ),ż(T )) (dq dp) and evolution b −1 b ′ dq p ′ = −M ′ dt , t t dpbt′ = ∇V (qtb′ ) dt′ − γP (qtb′ )M −1 pbt′ dt′ + σP (qtb′ ) dWtb′ + ∇ξ(qtb′ ) dλbt′ , ξ(qtb′ ) = z(T − t′ ) The generator of the forward dynamics is (Gram matrix Γ = {Ξ, Ξ}, ζ = (z, ż)) Lft = Lham + Lthm + {·, Ξ} Γ−1 ζ̇(t) Ξ Ξ while the generator of the backward dynamics is Lbt′ = R LfT −t′ R (where R : φ 7→ φ ◦ S is the momentum flip operator with S(q, p) = (q, −p)) Edinburgh, june 2010 – p. 24/33 Generalized free energy and work M Rigid free energy Frgd (z, vz ) = − 1 ln β Z Σξ,vξ (z,vz ) e−βH(q,p) dµΣξ,vξ (z,vz ) ξ,v Actual free energy recovered from the difference F (z) − Frgdξ (z, vz ), which equals, up to an unimportant additive constant: Z β T −1 1 (det GM (q))−1/2 exp vz GM (q)vz µΣξ,vξ (z,vz ) (dq dp) − ln β 2 Σξ,v (z,vz ) ξ Work performed during the switching: several expressions Z T ż(t)T dλt Force times displacement: W0,T {qt , pt }0≤t≤T = 0 Z T Energy variations: W0,T {qt , pt }0≤t≤T = w(t, qt , pt ) dt where 0 d T −1 w(t, q, p) = ζ̇(t) Γ {Ξ, H} (q, p) = (q, p) with Φ H ◦ Φt,t+h dh h=0 the flow of the switched Hamiltonian dynamics Edinburgh, june 2010 – p. 25/33 Jarzynski-Crooks relation For any bounded path functional ϕ[0,T ] , −βW0,T ({qt ,pt }t∈[0,T ] ) E ϕ[0,T ] ({qt , pt }0≤t≤T ) e Zz(T ),ż(T ) = Zz(0),ż(0) {q b′ , pb′ } ′ E ϕr [0,T ] t t 0≤t ≤T where ( · )r denotes the composition with the operation of time reversal of paths: ϕr[0,T ] {qtb′ , pbt′ }0≤t′ ≤T = ϕ[0,T ] {qTb −t , pbT −t }0≤t≤T This leads in particular to the following free energy estimator −β [W0,T ({qt ,pt }t∈[0,T ] )+C(T,qT )] 1 E e F (z(T )) − F (z(0)) = − ln −βC(0,q ) 0 β E e 1 1 ln det GM (q) − ż(t)T G−1 with the corrector C(t, q) = M (q)ż(t) 2β 2 Standard methods can then be used (bridge estimators, etc) Edinburgh, june 2010 – p. 26/33 Some elements on the proof The proof relies on the following nonequilibrium detailed balance condition Z −βH b f dσΣξ,vξ (z(t),ż(t)) ϕ1 Lt (ϕ2 ) − ϕ2 LT −t (ϕ1 ) e Σξ,vξ (z(t),ż(t)) = Z Σξ,vξ (z(t),ż(t)) + d dt Z βw(t, ·)ϕ1 ϕ2 e−βH dσΣξ,vξ (z(t),ż(t)) Σξ,vξ (z(t),ż(t)) ϕ1 ϕ2 e−βH dσΣξ,vξ (z(t),ż(t)) ! Left-hand side: standard (equilibrium) detailed balance contribution Right-hand side: evolution of the measure and work correction Edinburgh, june 2010 – p. 27/33 Numerical schemes: splitting strategy Fluctuation/dissipation part (no Lagrange multiplier needed) r ∆t ∆t γP (q n )M −1 (pn+1/4 + pn ) + σP (q n )G n pn+1/4 = pn − 4 2 Hamiltonian part for the forward evolution ∆t n+1/2 n+1/4 n n n+1/2 p = p − ∇V (q ) + ∇ξ(q )λ , 2 n+1 n −1 n+1/2 q = q + ∆t M p , ξ(q n+1 ) = z(tn+1 ), (Cq ) ∆t n+3/4 n+1/2 n+1 n+1 n+3/4 p = p − ∇V (q ) + ∇ξ(q )λ , 2 ∇ξ(q n+1 )T M −1 pn+3/4 = z(tn+2 ) − z(tn+1 ) , (Cp ) ∆t which defines a symplectic map z(tn+1 ) − z(tn ) z(tn+2 ) − z(tn+1 ) Φn : Σξ,vξ z(tn ), → Σξ,vξ z(tn+1 ), ∆t ∆t Edinburgh, june 2010 – p. 28/33 Discrete Jarzynski-Crooks equality: Alchemical case Idea in the alchemical case Discrete schedule {λ(0), . . . , λ(tNT )} with NT ∆t = T 0 0 Initial conditions (q , p ) ∼ µ0 (dq dp) = Z0−1 exp −βHλ(0) (q, p) dq dp Successive updates of the parameter and the configuration: Change the parameter from λ(n∆t) to λ((n + 1)∆t) Update work: W n+1/2 = W n + Hλ((n+1)∆t) (q n , pn ) − Hλ(n∆t) (q n , pn ) Update the configuration using the Verlet map Φn+1 associated with the Hamiltonian Hλ((n+1)∆t) W n+1 = W n+1/2 + Hλ((n+1)∆t) (q n+1 , pn+1 ) − Hλ((n+1)∆t) (q n , pn ) Total work: W n = Hλ(n∆t) (q n , pn ) − Hλ(0) (q 0 , p0 ) Z n −βW n = It can be checked that E e for any value of ∆t such that the Z0 Verlet scheme is stable Extension to path functionals Edinburgh, june 2010 – p. 29/33 Discrete Jarzynski-Crooks equality: The reaction coordinate case Discrete schedule {z(0), . . . , z(tNT )} Initial conditions (q 0 , p0 ) ∼ µΣ (q b,0 , pb,0 ) ∼ µ ξ,vξ “ z(t )−z(t ) z(t0 ), 1 ∆t 0 ” (dq dp) and „ « (dq dp) z(tN +1 )−z(tN ) T T Σξ,vξ z(tNT ), ∆t Initial work W 0 = 0, and work update W n+1 = W n + H(q n+1 , pn+3/4 ) − H(q n , pn+1/4 ) Time discretization error free estimator of the free energy difference: Z z(NT ), Zz(t z(tN +1 )−z(tN ) T T ∆t 0 ), z(t1 )−z(t0 ) ∆t n n −βW NT E ϕ[0,NT ] ({q , p }0≤n≤NT ) e = ′ ′ E ϕr[0,NT ] ({q b,n , pb,n }0≤n′ ≤NT ) ∆t ∆t Overdamped limit γ=M = Id: no bias due to the finite time-step in 4 2 the estimator (compare to Lelièvre/Rousset/Stoltz, 2007) Edinburgh, june 2010 – p. 30/33 Application: Solvatation effects on conformational changes 4 Free energy 3 2 1 0 −1 0.0 0.2 0.4 0.6 0.8 1.0 Reaction coordinate Estimated free energy profiles for T = 1 with K = 105 realizations (top curve), T = 10 with K = 104 and T = 100 with K = 103 (smoothest curve). Same parameters as before, except ∆t = 0.01. Schedule z(t) = zmin + (zmax − zmin ) Tt with zmin = −0.1 and zmax = 1.1. Edinburgh, june 2010 – p. 31/33 References Edinburgh, june 2010 – p. 32/33 References The main references for this work: T. L ELIÈVRE , M. R OUSSET AND G. S TOLTZ, Langevin dynamics with constraints and computation of free energy differences, arXiv preprint 1006.4914 (2010) T. L ELIÈVRE , M. R OUSSET AND G. S TOLTZ Free energy computations: A Mathematical Perspective, Imperial College Press. Other recent works on adaptive computation of free energy differences: Free energy methods for efficient exploration of mixture posterior densities, HAL preprint 00460914 (2010) N. C HOPIN , T. L ELIÈVRE AND G. S TOLTZ, Free energy calculations: An efficient adaptive biasing potential method, J. Phys. Chem. B 114(17), 5823-5830 (2010) B. D ICKSON , F. L EGOLL , T. L ELIÈVRE , G. S TOLTZ AND P. F LEURAT-L ESSARD, Workshop “Simulation of hybrid dynamical systems and applications to molecular dynamics” (IHP, Paris, September 27-30th) Edinburgh, june 2010 – p. 33/33