Probability Measures on Numerical Solutions of ODEs
and PDEs for Uncertainty Quantification and Inference
Patrick R. Conrad (Warwick Statistics)
Mark Girolami (Warwick Statistics)
Andrew Stuart (Warwick Mathematics)
Simo Särkkä (Aalto BECS)
Konstantinos Zygalakis (Southampton Mathematics)
June 8, 2015
This work is partly supported by the Engineering and Physical Sciences Research
Council (EPSRC) of the United Kingdom (grants EP/K034154/1 and
EP/J016934/2)
Statistics with deterministic numerical simulations
If we have a physical model u(θ) requiring the solution of differential
equations, we form an approximation,
ku(θ) − Uh (θ)k ≤ ψ(h) → 0 as h → 0.
Applied in forward UQ:
Eθ [Φ(u(θ))] ≈ Eθ [Φ(Uh (θ))],
or a Bayesian inverse problem:
µ(θ) ∝ L(u(θ)|d)p(θ) ≈ L(Uh (θ)|d)p(θ).
This neglects uncertainty in the solver Uh (θ)! Our analysis will be
over-confident about the solution u.
Conrad, et al.
Measures on Numeric Solvers
2/42
Bayesian posterior with deterministic solver
Posterior is over-confident at finite h values
Conrad, et al.
1000
800
6
600
5
4
2.0
3
1.8
1.0
1.6
0.5
2.0
1.8
1.6
6
5
4
3
1000
800
600
h=.1
h=.05
h=.02
h=.01
h=.005
Measures on Numeric Solvers
3/42
Statistics with random numerical simulations
Instead, form a randomized solver,
Eω ku(θ) − Uh,ω (θ)k ≤ ψ(h) → 0 as h → 0.
Applied in forward uncertainty quantification:
Eθ [Φ(u(θ))] ≈ Eθ,ω [Φ(Uh,ω (θ))],
or a Bayesian inverse problem:
µ(θ) ∝ L(u(θ)|d)p(θ) ≈
Z
L(Uh,ω (θ)|d)p(θ)dω.
Allow consistent statistical analysis across multiple resolutions
Conrad, et al.
Measures on Numeric Solvers
4/42
Lorenz with Classical 4th order Runge-Kutta
20
15
10
5
0
5
10
15
20
30
20
10
0
10
20
30
45
40
35
30
25
20
15
10
50
5
Conrad, et al.
10
15
20
Measures on Numeric Solvers
5/42
Lorenz with Randomized 4th order Runge-Kutta
20
15
10
5
0
5
10
15
20
30
20
10
0
10
20
30
50
40
30
20
10
00
5
Conrad, et al.
10
15
20
Measures on Numeric Solvers
6/42
Lorenz movie
Conrad, et al.
Measures on Numeric Solvers
7/42
Overview
I
Our aim is to quantify uncertainty in existing solvers for
combination with statistical methods
I
Describe uncertainty as a measure over solutions that contracts to
the true solution
I
Construct Monte Carlo samples by perturbing the discretization
with random Gaussian fields
I
Developed for both ODE and PDE solvers
Disclaimers:
I
Not a route to faster converging solvers or eliminating bias
I
Assume that analytic solution u(θ) is our objective
Conrad, et al.
Measures on Numeric Solvers
8/42
Existing approaches to statistical error models
Deterministic error indicators are well developed, e.g., an h refinement
indicator
e(t) = U h (t) − U h/2 (t)
suggest measure
µ(t) = N (U h (t), e(t)2 )
Pointwise i.i.d. Gaussian error is too simplistic; correlations impact later
analysis
Related work on statistical treatment of discretization error
I
O’Hagan (1992); Skilling (1991); Diaconis (1988)
I
Chkrebtii, Campbell, Girolami, Calderhead (2014)
I
Schober, Duvenaud, Hennig (2014)
Conrad, et al.
Measures on Numeric Solvers
9/42
Randomized ODE solvers
Consider the ODE:
du
= f (u),
dt
u(0) = u0 .
Integral equation
Choose a fixed step size h. For uk = u(kh). For t ∈ [tk , tk+1 ]:
Z t
u(t) = uk +
f u(s) ds
tk
Some approximation is required to create a numeric method.
Conrad, et al.
Measures on Numeric Solvers
10/42
Randomized ODE solvers (cont)
One-step numerical method
For Uk ≈ u(kh):
Uk+1 = Ψh (Uk ),
U0 = u0 .
Continuous approximation:
U(t) ≈ Ψt−tk (Uk )
Randomized numerical method
Assume the flow map is perturbed by a Gaussian process, ξk (·) defined on
[0, h], where ξk (·) = 0. This gives approximation U(t) for t ∈ [tk , tk+1 ]:
U(t) = Ψt−tk (Uk ) + ξk (t − tk ),
Uk+1 = Ψh (Uk ) + ξk (h).
Conrad, et al.
Measures on Numeric Solvers
11/42
Illustration of randomized ODE step
Standard basis
Randomized basis
2.0
1.0
u
u
1.5
0.5
0.00.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0
t
2.5
2.0
1.5
1.0
Standard Euler
0.5
Perturbed Euler
0.00.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0
t
Randomized solver is locally Gaussian, but globally non-Gaussian
Conrad, et al.
Measures on Numeric Solvers
12/42
Assumptions
Assumption 1
Let there exist K > 0, p ≥ 1 such that, for all t ∈ [0, h],
2
E ξ(t)ξ(t)T ≤ Kt 2p+1 .
F
Furthermore, assume there is a constant σ, independent of h, such that
E[ξ(h)ξ(h)T ] = σh2p+1 I.
Assumption 2
The function f and a sufficient number of its derivatives are bounded
uniformly in Rn in order to ensure that f is globally Lipschitz and that the
numerical flow-map Ψh has uniform local truncation error of order q + 1
with respect to the true flow-map Φh :
sup |Ψt (u) − Φt (u)| ≤ Kt q+1 .
u∈Rn
Conrad, et al.
Measures on Numeric Solvers
13/42
Convergence result
Theorem
Under Assumptions 1 and 2 it follows that there is K > 0 such that
sup E|uk − Uk |2 ≤ Kh2 min{p,q} .
0≤kh≤T
Furthermore
sup E|u(t) − U(t)| ≤ Khmin{p,q} .
0≤t≤T
Scaling of Noise
I
Optimal scaling of noise is p = q.
I
Then deterministic rate of convergence is unaffected.
I
But maximal noise is added to the system.
I
Fit constant σ to an error estimator.
Conrad, et al.
Measures on Numeric Solvers
14/42
Convergence of random solutions
Draws from the random solver for fixed σ
4
2
0
2
4
4
2
0
2
40
= .
= .
5 10 15 20 0
Conrad, et al.
= .
= .
= .
= . ×
5 10 15 20 0
5 10 15 20
Measures on Numeric Solvers
15/42
Backward error analysis
Modified (Stochastic Differential) Equation
q
√
X
du h
dW
= f (u h ) + hq
h` f` (u h ) + σh2q
,
dt
dt
l=0
u h (0) = u0
Theorem
Under Assumptions 1 and 2, for Φ a C ∞ function with all derivatives
bounded uniformly on Rn , there is a choice of {f` }q`=0 such that
Φ u(T ) − EΦ Uk ) ≤ Khq ,
and
kh = T .
EΦ u h (T ) − EΦ Uk ) ≤ Kh2q+1 ,
Conrad, et al.
kh = T .
Measures on Numeric Solvers
16/42
Impact of the scale parameter σ
Draws from the random solver for fixed h = 0.1
4
2
0
2
4
4
2
0
2
40
=
=
= .
= .
= .
= .
5 10 15 20 0
Conrad, et al.
5 10 15 20 0
5 10 15 20
Measures on Numeric Solvers
17/42
Choosing scale of random perturbations
The scale σ is problem dependent, choose it to match the classical error
indicator, by sampling
h
i
p(σ) ∝ exp −d N (E[Uσh ], V[Uσh ]), N (U h (t), e(t)2 )
,
or optimizing
min d N (E[Uσh ], V[Uσh ]), N (U h (t), e(t)2 ) .
σ
The Bhattacharyya distance works well
1
1
ln
d N (µp , σp2 ), N (µq , σq2 ) =
4
4
Conrad, et al.
σp2 σq2
+
+2
σq2 σp2
!!
1
+
4
(µp − µq )2
σp2 + σq2
Measures on Numeric Solvers
!
18/42
Density of scale parameter in FitzHugh-Nagumo
0.2
0.4
0.6
h=.1
h=.05
h=.02
h=.01
h=.005
logσ
Conrad, et al.
Measures on Numeric Solvers
19/42
Calibrated difference from deterministic solution
=.
1010
10
10 12
10 3
10 4
10 5
10
100
10 12
10 3
10 4
10 5
10 6
10
100
10 12
10 3
10 4
10 5
10 6
10 7
10 0
Random
Indicator
=.
=.
5
Conrad, et al.
10
15
Measures on Numeric Solvers
20
20/42
Bayesian posterior
For a true ODE problem,
du
= f (u, θ),
dt
u(0) = u0 ,
construct the true posterior,
P(θ|d) ∝ π(θ)L(u(t, θ)|d).
Given a numerical approximation, U h (t), construct approximate posterior,
≈ Ph (θ|d) ∝ π(θ)L(U h (t, θ)|d).
Typically convergent, in the sense that [Cotter, Dashti, Stuart]
dHell (Ph (θ|d), P(θ|d)) → 0 as h → 0
Conrad, et al.
Measures on Numeric Solvers
21/42
Modifying the inference problem
Deterministic solver
Ph (θ|d) ∝ π(θ)L(U h (t, θ)|d).
Deterministic error indicator
Ph (θ|d) ∝ π(θ)
Z
L(U h (t) + ξ(t)|d)dξ(t)
ξ(t) ∼ GP(0, e(t)2 ), either i.i.d. or AR(1)
Randomized solver
Ph (θ|d) ∝ π(θ)
Z
L(Uσh (t|ξ)|d)dξ
Apply noisy pseudomarginal MCMC to sample integrals
Conrad, et al.
Measures on Numeric Solvers
22/42
Repressilator inference
Protein concentration
80
60
40
20
0
800
60
40
20
0
800
60
40
20
00
60
40
20
10 20 30 40 50 60 00
60
40
20
10 20 30 40 50 60 00
60
40
20
10 20 30 40 50 60 00
mRNA
10 20 30 40 50 60
10 20 30 40 50 60
10 20 30 40 50 60
Elowitz and Leibler, 2000
Conrad, et al.
Measures on Numeric Solvers
23/42
Repressilator random integrals
Inference uses 2nd order Runge-Kutta
80
60
40
20
0
= .
= .
= .
= .
80
60
40
20
00 10 20 30 40 50 60 0 10203040 5060
Conrad, et al.
= .
0 10203040 5060
Measures on Numeric Solvers
24/42
Repressilator posterior with deterministic solver
Posterior is over-confident at finite h values
Conrad, et al.
1000
800
6
600
5
4
2.0
3
1.8
1.0
1.6
0.5
2.0
1.8
1.6
6
5
4
3
1000
800
600
h=.1
h=.05
h=.02
h=.01
h=.005
Measures on Numeric Solvers
25/42
Repressilator posterior with error indicator and i.i.d.
Uncorrelated model error has little impact
Conrad, et al.
1000
800
6
600
5
4
2.0
3
1.8
1.0
1.6
0.5
2.0
1.8
1.6
6
5
4
3
1000
800
600
h=.1
h=.05
h=.02
h=.01
h=.005
Measures on Numeric Solvers
26/42
Repressilator posterior with error indicator and AR(1)
Simple correlation model has little impact
Conrad, et al.
1000
800
6
600
5
4
2.0
3
1.8
1.0
1.6
0.5
2.0
1.8
1.6
6
5
4
3
1000
800
600
h=.1
h=.05
h=.02
h=.01
h=.005
Measures on Numeric Solvers
27/42
Repressilator posterior with random solver
Posterior still contains bias, but posterior width reflects error
Conrad, et al.
1000
800
6
600
5
4
2.0
3
1.8
1.0
1.6
0.5
2.0
1.8
1.6
6
5
4
3
1000
800
600
h=.1
h=.05
h=.02
h=.01
h=.005
Measures on Numeric Solvers
28/42
Summary of random ODE solvers
1. Insert uncertainty into discretisation with local Gaussian processes
2. Prove convergence of random solver and backwards error analysis
3. Scale noise in practice by matching error indicators
4. Demonstrate improved results on inference
Conrad, et al.
Measures on Numeric Solvers
29/42
Randomizing standard PDE solvers
Weak Form
u ∈ V : a(u, v ) = r (v ),
∀v ∈ V.
Galerkin Method
u h ∈ V h : a(u h , v ) = r (v ),
∀v ∈ V h .
Then
V h = span{Φj = Φsj }Jj=1 .
Randomized Galerkin Method
V h comprises small randomized perturbations of the standard Galerkin
method:
V h = span{Φj = Φsj + Φrj }Jj=1 .
Conrad, et al.
Measures on Numeric Solvers
30/42
Illustration of randomized PDE basis
2.0
1.5
1.0
0.5
0.0
0.50.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0
x
Randomized basis
u
u
Standard basis
2.0
Standard hat basis
1.5
Randomized hat basis
1.0
0.5
0.0
0.50.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0
x
I
For nodal points xk , Φrj (xk ) = 0.
I
Choose supp Φsj = supp Φrj to maintain sparsity
I
Greater flexibility in choosing properties of random field
I
Randomness generated in advance, solver step is unaffected
Conrad, et al.
Measures on Numeric Solvers
31/42
Assumptions
Assumption 1
The {Φrj }Jj=1 are independent, mean zero, Gaussian random fields, with
the same support as the {Φsj }, and satisfying
Φrj (xk ) = 0,
J
X
EkΦrj k2a ≤ Ch2q .
j=1
Assumption 2
The true solution u of problem (30) is in L∞ (D). Furthermore the
standard deterministic interpolant of the true solution, defined by
v s :=
J
X
u(xj )Φsj ,
j=1
satisfies ku − v s ka ≤ Chp .
Conrad, et al.
Measures on Numeric Solvers
32/42
Convergence result
Theorem
Under Assumptions 1 and 2 it follows that the random approximation U h
satisfies
Eku − U h k2a ≤ Ch2 min{p,q} .
Corollary
Consider the Poisson equation with Dirichlet boundary conditions and a
random perturbation of the piecewise linear FEM approximation, with
p = q = 1. Under Assumptions 1 and 2 it follows that the random
approximation U h satisfies
Eku − U h kL2 ≤ Ch2 .
Conrad, et al.
Measures on Numeric Solvers
33/42
Elliptic PDE inverse problem
Standard elliptic inversion problem:
∇ · (κ(x )∇u(x )) = 4x
u(0) = 0, u(1) = 2
2.0
1.5
1.0
0.5
0.0
0.5
1.0
1.5
2.00.0
u
log
Data with small i.i.d. Gaussian error
0.2
0.4
x
Conrad, et al.
0.6
0.8
1.0
2.5
2.0
1.5
1.0
0.5
0.0
0.50.0
0.2
0.4
x
0.6
0.8
Measures on Numeric Solvers
1.0
34/42
Elliptic inference with standard solver
Conrad, et al.
2
0
4
2
0
4
2
0
2
3
2
1
0
61
2
0
2
4
46
2
0
2
0
2
2
h=1/10
h=1/20
h=1/40
h=1/60
h=1/80
Measures on Numeric Solvers
35/42
Elliptic inference with random solver
Conrad, et al.
2
0
4
2
0
4
2
0
2
3
2
1
0
61
2
0
2
4
46
2
0
2
0
2
2
h=1/10
h=1/20
h=1/40
h=1/60
h=1/80
Measures on Numeric Solvers
36/42
2D Elliptic problem
Standard 2D elliptic equation:
∇ · (κ(x , y )∇u(x , y )) = f (x , y )
Solved on a 30 × 30 grid. Perturbation fields are N (0, (−4)−2 )
κ(x , y )
x
Conrad, et al.
u(x , y )
6.0
4.5
3.0
1.5
0.0
1.5
3.0
4.5
y
5.0
4.5
4.0
3.5
3.0
2.5
2.0
1.5
1.0
0.5
0.0
y
y
f (x , y )
x
x
Measures on Numeric Solvers
37/42
Perturbations in random solutions
Top PCA modes of perturbation are shown
Conrad, et al.
Measures on Numeric Solvers
38/42
Shallow water equation solver
I
I
I
Vortex shedding on a global shallow water model
Advanced mimetic finite element scheme ≈ 10,000 DOF
Introduced perturbations to ODE step and roughly calibrated scale
Solver due to Thuburn, Cotter (2014)
Conrad, et al.
Measures on Numeric Solvers
39/42
Perturbations to shallow water solver
Top PCA modes of perturbation are shown
Conrad, et al.
Measures on Numeric Solvers
40/42
Contributions
I
Our aim is to quantify uncertainty in existing solvers for
combination with statistical methods
I
Describe uncertainty as a measure over solutions that contracts to
the true solution
I
Construct Monte Carlo samples by perturbing the discretization
with random Gaussian fields
I
Developed for both ODE and PDE solvers
I
Simple construction can be adapted to many useful solvers
I
Demonstrated more consistent statistical analysis using randomized
solvers
Conrad, et al.
Measures on Numeric Solvers
41/42
Future work
I
Study other classes of PDEs and backwards error analysis
I
Extend to other types of model error (e.g., dimension reduction)
I
Apply to other problem classes, e.g., Stochastic Differential Equations
I
Study rates of convergence of forward UQ and Bayesian posteriors
I
Computational issues surrounding pseudomarginal posteriors
I
Leverage efficient statistical methods, e.g., multilevel sampling
I
Extensions to Bayesian inference on infinite dimensional spaces
I
Apply to large, real-world applications, such as shallow water
equations
Conrad, et al.
Measures on Numeric Solvers
42/42