Quantifying Uncertainty in Differential Equation Models:
Manifolds, Metrics and Russian Roulette
Mark Girolami
Department of Statistical Science
University College London
October 9, 2013
Talk Outline
• Why statistical inference for mechanistic models is hugely challenging
Talk Outline
• Why statistical inference for mechanistic models is hugely challenging
• One approach is to attack the problem exploiting intrinsic geometry of
mechanistic dynamics
Talk Outline
• Why statistical inference for mechanistic models is hugely challenging
• One approach is to attack the problem exploiting intrinsic geometry of
mechanistic dynamics
• Mechanistic dynamics described via PDE and ODE systems with
non-analytic solutions
Talk Outline
• Why statistical inference for mechanistic models is hugely challenging
• One approach is to attack the problem exploiting intrinsic geometry of
mechanistic dynamics
• Mechanistic dynamics described via PDE and ODE systems with
non-analytic solutions
• How to account for uncertainty induced by implicit definition of dynamics
via PDE and ODE representation
Talk Outline
• Why statistical inference for mechanistic models is hugely challenging
• One approach is to attack the problem exploiting intrinsic geometry of
mechanistic dynamics
• Mechanistic dynamics described via PDE and ODE systems with
non-analytic solutions
• How to account for uncertainty induced by implicit definition of dynamics
via PDE and ODE representation
• Oftentimes under fine spatial mesh refinement computing a likelihood
exactly may be infeasible
Talk Outline
• Why statistical inference for mechanistic models is hugely challenging
• One approach is to attack the problem exploiting intrinsic geometry of
mechanistic dynamics
• Mechanistic dynamics described via PDE and ODE systems with
non-analytic solutions
• How to account for uncertainty induced by implicit definition of dynamics
via PDE and ODE representation
• Oftentimes under fine spatial mesh refinement computing a likelihood
exactly may be infeasible
• Forward and Inverse inference can still progress exploiting
pseudo-marginal constructions in general form of Russian Roulette
Simple Dynamics
sim.
data
1.5
1
=
θ1 yz
0.5
=
−xz
=
−θ2 xy
z
dx
dt
dy
dt
dz
dt
0
ï0.5
ï1
ï1.5
1.5
ï1
ï0.5
1
0
0.5
1
0.5
1.5
2
0
y
x
Simple Dynamics
Simple Dynamics
10
9
8
7
e2
6
5
4
3
2
1
1
2
3
4
5
e1
6
7
8
9
10
• Simple nonlinear model - protein autoregulation.
dx
dt
dy
dt
=
72
−α
36 + y
=
βx − 1
• Simple nonlinear model - protein autoregulation.
dx
dt
dy
dt
=
72
−α
36 + y
=
βx − 1
• 120 equally spaced measurements of system from t = 0...60seconds
with Normal errors having known variance 0.5, α = 3, β = 1.
• Simple nonlinear model - protein autoregulation.
dx
dt
dy
dt
=
72
−α
36 + y
=
βx − 1
• 120 equally spaced measurements of system from t = 0...60seconds
with Normal errors having known variance 0.5, α = 3, β = 1.
• Induces a data density posing many challenges for simulation based
inference
Systems Identification - Posterior Inference
Mixing of Markov Chains
5
4
3
2
1
1
2
3
4
5
Geometric Concepts in MCMC
Geometric Concepts in MCMC
• Rao, 1945 to first order
χ2 (δθ) =
Z
|p(y; θ + δθ) − p(y; θ)|2
dy ≈ δθ T G(θ)δθ
p(y; θ)
Geometric Concepts in MCMC
• Rao, 1945 to first order
χ2 (δθ) =
Z
|p(y; θ + δθ) − p(y; θ)|2
dy ≈ δθ T G(θ)δθ
p(y; θ)
• Expected Fisher Information G(θ) is metric tensor of a Riemann manifold
Geometric Concepts in MCMC
• Rao, 1945 to first order
χ2 (δθ) =
Z
|p(y; θ + δθ) − p(y; θ)|2
dy ≈ δθ T G(θ)δθ
p(y; θ)
• Expected Fisher Information G(θ) is metric tensor of a Riemann manifold
• For nonlinear DE’s ẋ(t) = f(x(t), θ) observed under Normal additive
error with covariance , then FR metric tensor
Geometric Concepts in MCMC
• Rao, 1945 to first order
χ2 (δθ) =
Z
|p(y; θ + δθ) − p(y; θ)|2
dy ≈ δθ T G(θ)δθ
p(y; θ)
• Expected Fisher Information G(θ) is metric tensor of a Riemann manifold
• For nonlinear DE’s ẋ(t) = f(x(t), θ) observed under Normal additive
error with covariance , then FR metric tensor
G(θ) = ∂θ xC−1 ∂θ xT
Geometric Concepts in MCMC
• Rao, 1945 to first order
χ2 (δθ) =
Z
|p(y; θ + δθ) − p(y; θ)|2
dy ≈ δθ T G(θ)δθ
p(y; θ)
• Expected Fisher Information G(θ) is metric tensor of a Riemann manifold
• For nonlinear DE’s ẋ(t) = f(x(t), θ) observed under Normal additive
error with covariance , then FR metric tensor
G(θ) = ∂θ xC−1 ∂θ xT
• Sensitivity equations define metric can be exploited to locally transform
coordinate system for sampling
Geometric Concepts in MCMC
• Rao, 1945 to first order
χ2 (δθ) =
Z
|p(y; θ + δθ) − p(y; θ)|2
dy ≈ δθ T G(θ)δθ
p(y; θ)
• Expected Fisher Information G(θ) is metric tensor of a Riemann manifold
• For nonlinear DE’s ẋ(t) = f(x(t), θ) observed under Normal additive
error with covariance , then FR metric tensor
G(θ) = ∂θ xC−1 ∂θ xT
• Sensitivity equations define metric can be exploited to locally transform
coordinate system for sampling
• Discretised Langevin diffusion (MALA) on manifold defines efficient
proposal mechanism
θd0 = θd +
D
p
X
2 −1
d
G (θ)∇θ L(θ) − 2
G(θ)−1
G−1 (θ)z
ij Γij + 2
d
d
i,j
Geometric Concepts in MCMC
• Rao, 1945 to first order
χ2 (δθ) =
Z
|p(y; θ + δθ) − p(y; θ)|2
dy ≈ δθ T G(θ)δθ
p(y; θ)
• Expected Fisher Information G(θ) is metric tensor of a Riemann manifold
• For nonlinear DE’s ẋ(t) = f(x(t), θ) observed under Normal additive
error with covariance , then FR metric tensor
G(θ) = ∂θ xC−1 ∂θ xT
• Sensitivity equations define metric can be exploited to locally transform
coordinate system for sampling
• Discretised Langevin diffusion (MALA) on manifold defines efficient
proposal mechanism
θd0 = θd +
D
p
X
2 −1
d
G (θ)∇θ L(θ) − 2
G(θ)−1
G−1 (θ)z
ij Γij + 2
d
d
i,j
• Non-Euclidean geometry can exploit geodesics equations to devise
sampling schemes (RMHMC)
Infinite and finite dimensional mismatch
• Uncertainty induced by discrete mesh
Infinite and finite dimensional mismatch
• Uncertainty induced by discrete mesh
1
0.8
0.6
0.4
0.2
0
−0.2
−0.4
−0.6
−0.8
−1
−1
−0.5
0
0.5
1
Infinite and finite dimensional mismatch
• Models relate ẋ(t, θ) ∈ RP to states x(t, θ) ∈ RP by vector field
fθ (t, ·) : RP → RP indexed by parameter vector, θ ∈ Θ
Infinite and finite dimensional mismatch
• Models relate ẋ(t, θ) ∈ RP to states x(t, θ) ∈ RP by vector field
fθ (t, ·) : RP → RP indexed by parameter vector, θ ∈ Θ
• Observations, y(t) ∈ RR , via transformation, G : RP → RR , of the true or
exact solution, x(t, θ), of system at T time points.
Infinite and finite dimensional mismatch
• Models relate ẋ(t, θ) ∈ RP to states x(t, θ) ∈ RP by vector field
fθ (t, ·) : RP → RP indexed by parameter vector, θ ∈ Θ
• Observations, y(t) ∈ RR , via transformation, G : RP → RR , of the true or
exact solution, x(t, θ), of system at T time points.
y (r ) (t) = G (r ) x(t, θ) + (r ) (t),
(r ) (t) ∼ NT (0, Σ(r ) ),
where G (r ) is the r th output of the observation model.
Infinite and finite dimensional mismatch
• Models relate ẋ(t, θ) ∈ RP to states x(t, θ) ∈ RP by vector field
fθ (t, ·) : RP → RP indexed by parameter vector, θ ∈ Θ
• Observations, y(t) ∈ RR , via transformation, G : RP → RR , of the true or
exact solution, x(t, θ), of system at T time points.
y (r ) (t) = G (r ) x(t, θ) + (r ) (t),
(r ) (t) ∼ NT (0, Σ(r ) ),
where G (r ) is the r th output of the observation model.
• If unique analytic, x(t, θ), at each observation time (and spatial position
in the case of PDEs), given data y(t) posterior takes the form
Infinite and finite dimensional mismatch
• Models relate ẋ(t, θ) ∈ RP to states x(t, θ) ∈ RP by vector field
fθ (t, ·) : RP → RP indexed by parameter vector, θ ∈ Θ
• Observations, y(t) ∈ RR , via transformation, G : RP → RR , of the true or
exact solution, x(t, θ), of system at T time points.
y (r ) (t) = G (r ) x(t, θ) + (r ) (t),
(r ) (t) ∼ NT (0, Σ(r ) ),
where G (r ) is the r th output of the observation model.
• If unique analytic, x(t, θ), at each observation time (and spatial position
in the case of PDEs), given data y(t) posterior takes the form
p θ, x0 |y(t) ∝ p y(t)|x(t, θ), x0 × π θ, x0 .
Infinite and finite dimensional mismatch
• Due to non-analytic x(t, θ), likelihood p y(t)|x(t, θ), x0 unavailable
Infinite and finite dimensional mismatch
• Due to non-analytic x(t, θ), likelihood p y(t)|x(t, θ), x0 unavailable
• Replace x(t, θ) with finite-dimensional representation xN (t, θ) - Finite
spatial and temporal mesh, i.e. numerically integrated solution
Infinite and finite dimensional mismatch
• Due to non-analytic x(t, θ), likelihood p y(t)|x(t, θ), x0 unavailable
• Replace x(t, θ) with finite-dimensional representation xN (t, θ) - Finite
spatial and temporal mesh, i.e. numerically integrated solution
• Disregard the additional uncertainty induced in the discrete solution
y (r ) (t) = G (r ) xN (t, θ) + (r ) (t),
(r ) (t) ∼ NT (0, Σ(r ) ),
Infinite and finite dimensional mismatch
• Due to non-analytic x(t, θ), likelihood p y(t)|x(t, θ), x0 unavailable
• Replace x(t, θ) with finite-dimensional representation xN (t, θ) - Finite
spatial and temporal mesh, i.e. numerically integrated solution
• Disregard the additional uncertainty induced in the discrete solution
y (r ) (t) = G (r ) xN (t, θ) + (r ) (t),
(r ) (t) ∼ NT (0, Σ(r ) ),
Yielding posterior measure
pN θ, x0 |y(t) ∝ p y(t)|xN (t, θ), x0 × π θ, x0 .
Infinite and finite dimensional mismatch
• Due to non-analytic x(t, θ), likelihood p y(t)|x(t, θ), x0 unavailable
• Replace x(t, θ) with finite-dimensional representation xN (t, θ) - Finite
spatial and temporal mesh, i.e. numerically integrated solution
• Disregard the additional uncertainty induced in the discrete solution
y (r ) (t) = G (r ) xN (t, θ) + (r ) (t),
(r ) (t) ∼ NT (0, Σ(r ) ),
Yielding posterior measure
pN θ, x0 |y(t) ∝ p y(t)|xN (t, θ), x0 × π θ, x0 .
• Seemingly innocuous implicit assumption may lead to serious statistical
bias and misleading inferences as unaccounted errors accumulate in the
discretisation
Infinite and finite dimensional mismatch
• Due to non-analytic x(t, θ), likelihood p y(t)|x(t, θ), x0 unavailable
• Replace x(t, θ) with finite-dimensional representation xN (t, θ) - Finite
spatial and temporal mesh, i.e. numerically integrated solution
• Disregard the additional uncertainty induced in the discrete solution
y (r ) (t) = G (r ) xN (t, θ) + (r ) (t),
(r ) (t) ∼ NT (0, Σ(r ) ),
Yielding posterior measure
pN θ, x0 |y(t) ∝ p y(t)|xN (t, θ), x0 × π θ, x0 .
• Seemingly innocuous implicit assumption may lead to serious statistical
bias and misleading inferences as unaccounted errors accumulate in the
discretisation
y (r ) (t) = G (r ) x(t) + (r ) (t)
= G (r ) xN (t) + δ (r ) (t) + (r ) (t)
where δ (r ) (t) = G (r ) x(t) − G (r ) xN (t) .
Full Bayesian posterior measure for model uncertainty
• Define a probability measure over functions
p x(t, θ), f|θ, x0 , Ψ
Full Bayesian posterior measure for model uncertainty
• Define a probability measure over functions
p x(t, θ), f|θ, x0 , Ψ
where Ψ are parameters of error model for x(t, θ), and f evaluations of
vector field at discretised approximate solution
Full Bayesian posterior measure for model uncertainty
• Define a probability measure over functions
p x(t, θ), f|θ, x0 , Ψ
where Ψ are parameters of error model for x(t, θ), and f evaluations of
vector field at discretised approximate solution
• The full posterior distribution therefore follows as
p θ, x0 , x(t, θ), f, Ψ|y(t) ∝ p y(t)|x(t, θ) × p x(t, θ), f|θ, x0 , Ψ × π θ, x0 , Ψ
|
{z
} |
{z
} |
{z
}
Likelihood
Probabilistic Solution
Prior
Full Bayesian posterior measure for model uncertainty
• Define a probability measure over functions
p x(t, θ), f|θ, x0 , Ψ
where Ψ are parameters of error model for x(t, θ), and f evaluations of
vector field at discretised approximate solution
• The full posterior distribution therefore follows as
p θ, x0 , x(t, θ), f, Ψ|y(t) ∝ p y(t)|x(t, θ) × p x(t, θ), f|θ, x0 , Ψ × π θ, x0 , Ψ
|
{z
} |
{z
} |
{z
}
Likelihood
Probabilistic Solution
Prior
• Uncertainty in the probabilistic solution x(t, θ) is made explicit taking into
account the mismatch between state solution and a finite approximation
Full Bayesian posterior measure for model uncertainty
• Approximate solutions at N knots, x̃(s1 ), · · · , x̃(sN )
Full Bayesian posterior measure for model uncertainty
• Approximate solutions at N knots, x̃(s1 ), · · · , x̃(sN )
• From these obtain values of N approximate vector field evaluations as,
f1:N = fθ (s1 , x̃(s1 )), · · · , fθ (sN , x̃(sN ))
Full Bayesian posterior measure for model uncertainty
• Approximate solutions at N knots, x̃(s1 ), · · · , x̃(sN )
• From these obtain values of N approximate vector field evaluations as,
f1:N = fθ (s1 , x̃(s1 )), · · · , fθ (sN , x̃(sN ))
• Error between f1:N and ẋ(s) = [ẋ(s1 ), · · · , ẋ(sN )] define probabilistically
Full Bayesian posterior measure for model uncertainty
• Approximate solutions at N knots, x̃(s1 ), · · · , x̃(sN )
• From these obtain values of N approximate vector field evaluations as,
f1:N = fθ (s1 , x̃(s1 )), · · · , fθ (sN , x̃(sN ))
• Error between f1:N and ẋ(s) = [ẋ(s1 ), · · · , ẋ(sN )] define probabilistically
• Infinite dimension function space, no Lebesgue measure
Full Bayesian posterior measure for model uncertainty
• Approximate solutions at N knots, x̃(s1 ), · · · , x̃(sN )
• From these obtain values of N approximate vector field evaluations as,
f1:N = fθ (s1 , x̃(s1 )), · · · , fθ (sN , x̃(sN ))
• Error between f1:N and ẋ(s) = [ẋ(s1 ), · · · , ẋ(sN )] define probabilistically
• Infinite dimension function space, no Lebesgue measure
• Radon-Nikodym derivative of posterior measure with respect to GP prior
dµf
1
2
ẋ(s)
∝
exp
−
||
ẋ(s)
−
f
||
1:N
Λ
N
2
dµf0
Full Bayesian posterior measure for model uncertainty
• Gaussian measure, µf0 = N (m0f , C0f ), on a Hilbert space, H, mean
function m0f , covariance operator C0f well defined
Full Bayesian posterior measure for model uncertainty
• Gaussian measure, µf0 = N (m0f , C0f ), on a Hilbert space, H, mean
function m0f , covariance operator C0f well defined
• Eigenfunctions of covariance operator form basis for derivative space
Full Bayesian posterior measure for model uncertainty
• Gaussian measure, µf0 = N (m0f , C0f ), on a Hilbert space, H, mean
function m0f , covariance operator C0f well defined
• Eigenfunctions of covariance operator form basis for derivative space
• m0f (t1 ) = `(t1 ),
C0f (t1 , t2 ) = α−1 ∫R Rλ (t1 , z)Rλ (t2 , z)dz = RR(t1 , t2 ).
Full Bayesian posterior measure for model uncertainty
• Gaussian measure, µf0 = N (m0f , C0f ), on a Hilbert space, H, mean
function m0f , covariance operator C0f well defined
• Eigenfunctions of covariance operator form basis for derivative space
C0f (t1 , t2 ) = α−1 ∫R Rλ (t1 , z)Rλ (t2 , z)dz = RR(t1 , t2 ).
• Posterior measure for ẋ(t) denoted by µfn = N mnf (t), Cnf (t, t)
• m0f (t1 ) = `(t1 ),
Full Bayesian posterior measure for model uncertainty
• Gaussian measure, µf0 = N (m0f , C0f ), on a Hilbert space, H, mean
function m0f , covariance operator C0f well defined
• Eigenfunctions of covariance operator form basis for derivative space
C0f (t1 , t2 ) = α−1 ∫R Rλ (t1 , z)Rλ (t2 , z)dz = RR(t1 , t2 ).
• Posterior measure for ẋ(t) denoted by µfn = N mnf (t), Cnf (t, t)
• m0f (t1 ) = `(t1 ),
mnf (t) = m0f (t) + RR(t, s) Λn + RR(s, s)
−1
f1:n − m0f (s)
−1
Cnf (t, t) = RR(t, t) − RR(t, s) Λn + RR(s, s)
RR(s, t)
Full Bayesian posterior measure for model uncertainty
• Gaussian measure, µf0 = N (m0f , C0f ), on a Hilbert space, H, mean
function m0f , covariance operator C0f well defined
• Eigenfunctions of covariance operator form basis for derivative space
C0f (t1 , t2 ) = α−1 ∫R Rλ (t1 , z)Rλ (t2 , z)dz = RR(t1 , t2 ).
• Posterior measure for ẋ(t) denoted by µfn = N mnf (t), Cnf (t, t)
• m0f (t1 ) = `(t1 ),
mnf (t) = m0f (t) + RR(t, s) Λn + RR(s, s)
−1
f1:n − m0f (s)
−1
Cnf (t, t) = RR(t, t) − RR(t, s) Λn + RR(s, s)
RR(s, t)
• Linearity of integral operator provides Gaussian prior measure for x(t)
Full Bayesian posterior measure for model uncertainty
• Gaussian measure, µf0 = N (m0f , C0f ), on a Hilbert space, H, mean
function m0f , covariance operator C0f well defined
• Eigenfunctions of covariance operator form basis for derivative space
C0f (t1 , t2 ) = α−1 ∫R Rλ (t1 , z)Rλ (t2 , z)dz = RR(t1 , t2 ).
• Posterior measure for ẋ(t) denoted by µfn = N mnf (t), Cnf (t, t)
• m0f (t1 ) = `(t1 ),
mnf (t) = m0f (t) + RR(t, s) Λn + RR(s, s)
−1
f1:n − m0f (s)
−1
Cnf (t, t) = RR(t, t) − RR(t, s) Λn + RR(s, s)
RR(s, t)
• Linearity of integral operator provides Gaussian prior measure for x(t)
• Posterior measure follows as p x(t)f1:N , θ, x0 , Ψ = NT mN (t), CN (t, t)
Full Bayesian posterior measure for model uncertainty
• Gaussian measure, µf0 = N (m0f , C0f ), on a Hilbert space, H, mean
function m0f , covariance operator C0f well defined
• Eigenfunctions of covariance operator form basis for derivative space
C0f (t1 , t2 ) = α−1 ∫R Rλ (t1 , z)Rλ (t2 , z)dz = RR(t1 , t2 ).
• Posterior measure for ẋ(t) denoted by µfn = N mnf (t), Cnf (t, t)
• m0f (t1 ) = `(t1 ),
mnf (t) = m0f (t) + RR(t, s) Λn + RR(s, s)
−1
f1:n − m0f (s)
−1
Cnf (t, t) = RR(t, t) − RR(t, s) Λn + RR(s, s)
RR(s, t)
• Linearity of integral operator provides Gaussian prior measure for x(t)
• Posterior measure follows as p x(t)f1:N , θ, x0 , Ψ = NT mN (t), CN (t, t)
mn (t) = m0 (t) + QR(t, s) Λn + RR(s, s)
−1
f1:n − m0f (s)
−1
Cn (t, t) = QQ(t, t) − QR(t, s) Λn + RR(s, s)
RQ(s, t),
Probabilistic Integration
• Posterior measure of functional solutions in H is Gaussian Process
Probabilistic Integration
• Posterior measure of functional solutions in H is Gaussian Process
• Approximate solution at knots x̃(s1 ), · · · , x̃(sN ) probability measure for
solution values at any t follows in Gaussian conditional
Probabilistic Integration
• Posterior measure of functional solutions in H is Gaussian Process
• Approximate solution at knots x̃(s1 ), · · · , x̃(sN ) probability measure for
solution values at any t follows in Gaussian conditional
• Provides calibrated probability measure of uncertainty due to finite
solution and infinite model mismatch
Probabilistic Integration
• Posterior measure of functional solutions in H is Gaussian Process
• Approximate solution at knots x̃(s1 ), · · · , x̃(sN ) probability measure for
solution values at any t follows in Gaussian conditional
• Provides calibrated probability measure of uncertainty due to finite
solution and infinite model mismatch
p θ, x0 , x(t, θ), f, Ψ|y(t) ∝ p y(t)|x(t, θ) × p x(t, θ), f|θ, x0 , Ψ × π θ, x0 , Ψ
|
{z
} |
{z
} |
{z
}
Likelihood
Probabilistic Solution
Prior
Probabilistic Integration
• Posterior measure of functional solutions in H is Gaussian Process
• Approximate solution at knots x̃(s1 ), · · · , x̃(sN ) probability measure for
solution values at any t follows in Gaussian conditional
• Provides calibrated probability measure of uncertainty due to finite
solution and infinite model mismatch
p θ, x0 , x(t, θ), f, Ψ|y(t) ∝ p y(t)|x(t, θ) × p x(t, θ), f|θ, x0 , Ψ × π θ, x0 , Ψ
|
{z
} |
{z
} |
{z
}
Likelihood
Probabilistic Solution
Prior
• Important in quantifying uncertainty (or uncertainty reduction) in moving
from coarse to fine meshing
Probabilistic Integration
• Posterior measure of functional solutions in H is Gaussian Process
• Approximate solution at knots x̃(s1 ), · · · , x̃(sN ) probability measure for
solution values at any t follows in Gaussian conditional
• Provides calibrated probability measure of uncertainty due to finite
solution and infinite model mismatch
p θ, x0 , x(t, θ), f, Ψ|y(t) ∝ p y(t)|x(t, θ) × p x(t, θ), f|θ, x0 , Ψ × π θ, x0 , Ψ
|
{z
} |
{z
} |
{z
}
Likelihood
Probabilistic Solution
Prior
• Important in quantifying uncertainty (or uncertainty reduction) in moving
from coarse to fine meshing
• Suggests probabilistic construction (integration), sampling of solutions
Kuramoto-Sivashinsky model of reaction-diffusion
• Kuramoto-Sivashinsky model of reaction-diffusion systems
Kuramoto-Sivashinsky model of reaction-diffusion
• Kuramoto-Sivashinsky model of reaction-diffusion systems
∂4
∂
∂
∂2
u−
u,
u = −u
u−
2
∂t
∂x
∂x
∂x 4
Kuramoto-Sivashinsky model of reaction-diffusion
• Kuramoto-Sivashinsky model of reaction-diffusion systems
∂4
∂
∂
∂2
u−
u,
u = −u
u−
2
∂t
∂x
∂x
∂x 4
• Domain x ∈ [0, 32π], t ∈ [0, 150]
Kuramoto-Sivashinsky model of reaction-diffusion
• Kuramoto-Sivashinsky model of reaction-diffusion systems
∂4
∂
∂
∂2
u−
u,
u = −u
u−
2
∂t
∂x
∂x
∂x 4
• Domain x ∈ [0, 32π], t ∈ [0, 150]
• Initial function u(0, x) = cos (x/16) 1 + sin (x/16)
Kuramoto-Sivashinsky model of reaction-diffusion
• Kuramoto-Sivashinsky model of reaction-diffusion systems
∂4
∂
∂
∂2
u−
u,
u = −u
u−
2
∂t
∂x
∂x
∂x 4
• Domain x ∈ [0, 32π], t ∈ [0, 150]
• Initial function u(0, x) = cos (x/16) 1 + sin (x/16)
• Discretize spatial domain, obtaining a high-dimensional (128
dimensions) system of stiff ODEs
Kuramoto-Sivashinsky model of reaction-diffusion
• Kuramoto-Sivashinsky model of reaction-diffusion systems
∂4
∂
∂
∂2
u−
u,
u = −u
u−
2
∂t
∂x
∂x
∂x 4
• Domain x ∈ [0, 32π], t ∈ [0, 150]
• Initial function u(0, x) = cos (x/16) 1 + sin (x/16)
• Discretize spatial domain, obtaining a high-dimensional (128
dimensions) system of stiff ODEs
• Use the integrating factor method to transform the system to one of
purely nonlinear ODEs
Kuramoto-Sivashinsky model of reaction-diffusion
• Kuramoto-Sivashinsky model of reaction-diffusion systems
∂4
∂
∂
∂2
u−
u,
u = −u
u−
2
∂t
∂x
∂x
∂x 4
• Domain x ∈ [0, 32π], t ∈ [0, 150]
• Initial function u(0, x) = cos (x/16) 1 + sin (x/16)
• Discretize spatial domain, obtaining a high-dimensional (128
dimensions) system of stiff ODEs
• Use the integrating factor method to transform the system to one of
purely nonlinear ODEs
• Probabilistic IVP solutions sampled using 2K uniform solver knots
Kuramoto-Sivashinsky model of reaction-diffusion
• Kuramoto-Sivashinsky model of reaction-diffusion systems
∂4
∂
∂
∂2
u−
u,
u = −u
u−
2
∂t
∂x
∂x
∂x 4
• Domain x ∈ [0, 32π], t ∈ [0, 150]
• Initial function u(0, x) = cos (x/16) 1 + sin (x/16)
• Discretize spatial domain, obtaining a high-dimensional (128
dimensions) system of stiff ODEs
• Use the integrating factor method to transform the system to one of
purely nonlinear ODEs
• Probabilistic IVP solutions sampled using 2K uniform solver knots
• Fifteen solution samples illustrate uncertainty over domain propagates
through system resulting in noticeably distinct dynamics, not captured by
deterministic numerical solvers.
Kuramoto-Sivashinsky model of reaction-diffusion
Figure: Side view and top view of a probabilistic solution realization of
the
Kuramoto-Sivashinsky PDE with initial function u(0, x) = cos (x/16) 1 + sin (x/16)
and domain x ∈ [0, 32π], t ∈ [0, 150].
Kuramoto-Sivashinsky model of reaction-diffusion
Figure: Fifteen realizations of the probabilistic solution of the Kuramoto-Sivashinsky
PDE using a fixed initial function. The solution is known to exhibit temporal chaos.
Deterministic numerical solutions only capture one type of behaviour given a fixed initial
function, which can lead to bias when used in conjunction with data-based inference.
Full Bayesian Uncertainty Quantification
• Consider now joint parameter and solution inference
Full Bayesian Uncertainty Quantification
• Consider now joint parameter and solution inference
• Draw K samples from the posterior distribution p θ, x(t), f1:N |y(t), x0 , Ψ
Full Bayesian Uncertainty Quantification
• Consider now joint parameter and solution inference
• Draw K samples from the posterior distribution p θ, x(t), f1:N |y(t), x0 , Ψ
Initialize θ;
for k = 1 : K do
Propose θ ? ∼ q(θ ? |θ);
?
Conditionally
simulate
a solution realisation x (t) from
p x(t), f1:N θ, x0 , Ψ
Compute:
ρ(θ, x(t) → θ ? , x? (t)) =
?
q(θ|θ ? ) p(θ ? ) p y(t)|G x (t) , Σ
;
q(θ ? |θ) p(θ) p y(t)|G x(t) , Σ
if min[1, ρ(θ → θ ? )] > U[0, 1] then
Update θ, x(t) = θ ? , x? (t);
end if
Return θ, x(t).
end for
Inference for model of cellular signal transduction
Figure: Experimental data and sample paths of the observation processes obtained by
transforming a sample from marginal posterior state distribution by observation function
Inference for model of cellular signal transduction
Figure: Marginal parameter posterior based on sample of size 100K generated by a
parallel tempering algorithm utilizing seven chains, with the first 10K samples removed.
Prior densities are shown in red.
Intractable Likelihoods under Mesh Refinement
What can we do?
Large Scale GMRF Ozone Column Model
1
80
0.8
Latitude
60
40
0.6
20
0.4
0
0.2
−20
0
−40
−0.2
−60
−80
−0.4
0
50
100
150
200
Longitude
250
300
350
Large Scale GMRF Ozone Column Model
• Cressie, (2008) comprised of 173,405 ozone measurements
Large Scale GMRF Ozone Column Model
• Cressie, (2008) comprised of 173,405 ozone measurements
• Data and spatial extent has precluded full Bayesian analysis to date
Large Scale GMRF Ozone Column Model
• Cressie, (2008) comprised of 173,405 ozone measurements
• Data and spatial extent has precluded full Bayesian analysis to date
• Matern covariance function triangulated over 196,002 vertices on sphere
Large Scale GMRF Ozone Column Model
• Cressie, (2008) comprised of 173,405 ozone measurements
• Data and spatial extent has precluded full Bayesian analysis to date
• Matern covariance function triangulated over 196,002 vertices on sphere
p(x|θ)
=
1
N
(2π) 2
i
1h
T −1
exp − log |Cθ | + x Cθ x
2
Large Scale GMRF Ozone Column Model
• Cressie, (2008) comprised of 173,405 ozone measurements
• Data and spatial extent has precluded full Bayesian analysis to date
• Matern covariance function triangulated over 196,002 vertices on sphere
p(x|θ)
=
1
N
(2π) 2
=
1
N
(2π) 2
i
1h
T −1
exp − log |Cθ | + x Cθ x
2
( ∞
)
∞
1X T
1X1
T
n
m
exp
E{z (I − Cθ ) z} −
x (I − Cθ ) x
2
n
2
n=1
m=0
Large Scale GMRF Ozone Column Model
• Cressie, (2008) comprised of 173,405 ozone measurements
• Data and spatial extent has precluded full Bayesian analysis to date
• Matern covariance function triangulated over 196,002 vertices on sphere
p(x|θ)
=
1
N
(2π) 2
=
1
N
(2π) 2
i
1h
T −1
exp − log |Cθ | + x Cθ x
2
( ∞
)
∞
1X T
1X1
T
n
m
exp
E{z (I − Cθ ) z} −
x (I − Cθ ) x
2
n
2
n=1
m=0
p(x | θ) =
1
exp
(2π)N/2
1
exp
(2π)N/2
=
1
1
log |Qθ | − xT Qθ x
2
2
1
1
Ez {zT log(Qθ )z} − xT Qθ x
2
2
Large Scale GMRF Ozone Column Model
• Cressie, (2008) comprised of 173,405 ozone measurements
• Data and spatial extent has precluded full Bayesian analysis to date
• Matern covariance function triangulated over 196,002 vertices on sphere
p(x|θ)
=
1
N
(2π) 2
=
1
N
(2π) 2
i
1h
T −1
exp − log |Cθ | + x Cθ x
2
( ∞
)
∞
1X T
1X1
T
n
m
exp
E{z (I − Cθ ) z} −
x (I − Cθ ) x
2
n
2
n=1
m=0
p(x | θ) =
1
exp
(2π)N/2
1
exp
(2π)N/2
=
1
1
log |Qθ | − xT Qθ x
2
2
1
1
Ez {zT log(Qθ )z} − xT Qθ x
2
2
• Exploit Pseudo-Marginal construction - Andrieu & Roberts, 2009 -
Russian Roulette unbiased truncation of infinite series - MCMC based
inference can proceed..... in principle ;-)
Conclusions and Discussion
• EQUIP presents some of the most exciting open research problems at
the leading edge of computational statistics
Conclusions and Discussion
• EQUIP presents some of the most exciting open research problems at
the leading edge of computational statistics
• Geometric approaches to MCMC
Conclusions and Discussion
• EQUIP presents some of the most exciting open research problems at
the leading edge of computational statistics
• Geometric approaches to MCMC
• Probabilistic solution of DE’s
Conclusions and Discussion
• EQUIP presents some of the most exciting open research problems at
the leading edge of computational statistics
• Geometric approaches to MCMC
• Probabilistic solution of DE’s
• Addressing intractable nature of likelihoods under mesh refinement
Conclusions and Discussion
• EQUIP presents some of the most exciting open research problems at
the leading edge of computational statistics
• Geometric approaches to MCMC
• Probabilistic solution of DE’s
• Addressing intractable nature of likelihoods under mesh refinement
• EQUIP is awesome
Conclusions and Discussion
• EQUIP presents some of the most exciting open research problems at
the leading edge of computational statistics
• Geometric approaches to MCMC
• Probabilistic solution of DE’s
• Addressing intractable nature of likelihoods under mesh refinement
• EQUIP is awesome
• EQUIP is awesome
Conclusions and Discussion
• EQUIP presents some of the most exciting open research problems at
the leading edge of computational statistics
• Geometric approaches to MCMC
• Probabilistic solution of DE’s
• Addressing intractable nature of likelihoods under mesh refinement
• EQUIP is awesome
• EQUIP is awesome
• EQUIP is awesome
Conclusions and Discussion
• EQUIP presents some of the most exciting open research problems at
the leading edge of computational statistics
• Geometric approaches to MCMC
• Probabilistic solution of DE’s
• Addressing intractable nature of likelihoods under mesh refinement
• EQUIP is awesome
• EQUIP is awesome
• EQUIP is awesome