The automatic solution of PDEs
using a global spectral method
2014 CBMS-NSF Conference, 29th June 2014
Alex Townsend
PhD student
University of Oxford
(with Sheehan Olver)
Supervised by Nick Trefethen, University of Oxford.
Supported by EPSRC grant EP/P505666/1.
Introduction
Chebfun
Piecewise smooth
(2010)
Endpoint
singularities
(2010*)
Blow up
functions
(2011)
Reducing regularity
Chebfun
Linear ODEs
(2008)
Nonlinear
ODEs
(2012)
Chebgui
(2011*)
(2004)
Ordinary differential equations
Chebfun2
(2013)
Piecewise
smooth
Arbitrary
domains
Linear PDEs
Nonlinear
PDEs
Two dimensions
Alex Townsend @ Oxford
1/21
Introduction
Chebop: Spectral collocation for ODEs
In 2008: Overload the MATLAB backslash command \ for operators [Driscoll,
Bornemann, & Trefethen 2008].
L = chebop(@(x,u) diff(u,2)-x.*u,[-30 30]); % Airy equation
L.lbc = 1; L.rbc = 0;
% Set boundary conditions
u = L \ 0; plot(u)
% Solve and plot
5
0
−5
−10
−30
Alex Townsend @ Oxford
−20
−10
0
10
20
30
2/21
Introduction
Spectral collocation basics
Given values on a grid, what are the values of the derivative on that same grid?:
u’ u’
u’
u’
u1 u2
u5
u9
x1 x2
x5
x9
1
2
   0
u1  u1 
   
Dn  ...  =  ...  ,
   0
un
un
5
9
Dn = diffmat(n).
For example, u0 (x) + cos(x)u(x) is represented as
Ln = Dn + diag (cos(x1 ), . . . , cos(xn )) ∈ Rn×n .
Alex Townsend @ Oxford
3/21
Introduction
Why do spectral methods get a bad press?
1. Dense matrices.
2. Ill-conditioned matrices.
3. When has it converged? Tricky.
See, for example: [Canuto et al. 07], , [Fornberg 98], [Trefethen 00].
Alex Townsend @ Oxford
4/21
Introduction
Why do spectral methods get a bad press?
1. Dense matrices.
2. Ill-conditioned matrices.
3. When has it converged? Tricky.
See, for example: [Canuto et al. 07], , [Fornberg 98], [Trefethen 00].
Alex Townsend @ Oxford
4/21
Introduction
Why do spectral methods get a bad press?
1. Dense matrices.
2. Ill-conditioned matrices.
3. When has it converged? Tricky.
Condition number
15
10
10
10
5
10
0
10
0
500
1000
See, for example: [Canuto et al. 07], , [Fornberg 98], [Trefethen 00].
Alex Townsend @ Oxford
4/21
Introduction
Why do spectral methods get a bad press?
1. Dense matrices.
2. Ill-conditioned matrices.
3. When has it converged? Tricky.
Condition number
Error in solution
15
10
0
10
10
10
−5
10
5
10
−10
10
0
10
0
500
1000
200
400
600
800
1000
See, for example: [Canuto et al. 07], , [Fornberg 98], [Trefethen 00].
Alex Townsend @ Oxford
4/21
A fast and well-conditioned spectral method
Differentiation operator
Work with coefficients: Spectral methods do not have to result in dense, illconditioned matrices. (Just don’t discretize the differentiation operator faithfully.)
The idea is to use simple relations between Chebyshev polynomials:

1


(Uk − Uk −2 ) , k ≥ 2,



2


dTk
1
kUk −1 , k ≥ 1,
Tk = 
=
k = 1,


2 U1 ,

0,
dx
k = 0,

U0 ,
k = 0.


0 1



2


 ,
D = 

3



. . . 


1
1 0 − 2


 1 0 − 1

 2
2
 .
S = 
1
1

0 −2

2


. . . . . . . . . 
Olver & T., A fast and well-conditioned spectral method, SIAM Review, 2013.
Alex Townsend @ Oxford
5/21
A fast and well-conditioned spectral method
Multiplication operator
Tj Tk =
1
T|j−k |
2

2a0 a1 a2 a3

 a
 1 2a0 a1 a2
1 
M[a] =  a2 a1 2a0 a1
2 
 a3 a2 a1 2a0

 .
... ... ...
..
|
{z
Toeplitz
+


 0
. . .



. . . 
a1



. . .  1 
 + a2
. . .  2 

a3


. . . 
 .
..
}
|
1
Tj+k
2
0 0

. . .
.
a2 a3 a4 . . 

.
.
a3 a4 a5 . 

.
.
a4 a5 a6 . 

.
.
.

.. .. .. ...
{z
}
0
Hankel + rank-1
Multiplication is not a dense operator in finite precision. It is m-banded:
∞
m
X
X
a(x) =
ak Tk (x) =
ãk Tk (x) + O(),
k =0
Alex Townsend @ Oxford
k =0
6/21
A fast and well-conditioned spectral method
What about this new spectral method?
1. Almost banded matrices.
2. Well-conditioned matrices.
3. When has it converged? Trivial.
Other approaches: [Clenshaw 57], [Greengard 91], [Shen 03].
Alex Townsend @ Oxford
7/21
A fast and well-conditioned spectral method
What about this new spectral method?
1. Almost banded matrices.
2. Well-conditioned matrices.
3. When has it converged? Trivial.
Other approaches: [Clenshaw 57], [Greengard 91], [Shen 03].
Alex Townsend @ Oxford
7/21
A fast and well-conditioned spectral method
What about this new spectral method?
1. Almost banded matrices.
2. Well-conditioned matrices.
3. When has it converged? Trivial.
Condition number
2
10
1
10
0
10
0
500
1000
Other approaches: [Clenshaw 57], [Greengard 91], [Shen 03].
Alex Townsend @ Oxford
7/21
A fast and well-conditioned spectral method
What about this new spectral method?
1. Almost banded matrices.
2. Well-conditioned matrices.
3. When has it converged? Trivial.
Condition number
Error in solution
2
10
0
10
1
10
−10
10
0
10
0
500
1000
200
400
600
800
1000
Other approaches: [Clenshaw 57], [Greengard 91], [Shen 03].
Alex Townsend @ Oxford
7/21
A fast and well-conditioned spectral method
First example
u0 (x) + x 3 u(x) = 100 sin(20,000x 2 ),
u(−1) = 0.
The exact solution is
!
Z x
t4
x4
100e 4 sin(20,000t 2 )dt .
u(x) = e − 4
−1
1.2
N = chebop(@(x,u) · · · );
N.lbc = 0; u = N \ f;
Adaptively selects the
discretisation size.
Forms a chebfun object
[Chebfun V4.2].
kũ − uk∞ = 1.5 × 10−15 .
1
u(x)
0.8
degree(u) = 20391
0.6
0.4
time = 15.5s
0.2
0
−0.2
−1
Alex Townsend @ Oxford
−0.5
0
x
0.5
1
8/21
A fast and well-conditioned spectral method
Another example
1
u(x) = 0,
u(−1) = 1.
1 + 50,000x 2
The exact solution with a = 50,000 is
√
√ !
tan−1 ( ax) + tan−1 ( a)
u(x) = exp −
.
√
a
u0 (x) +
−5
10
1.005
1
−10
Absolute error
u(x)
degree(u) = 5093
0.995
10
−15
10
0.99
0.985
−1
Alex Townsend @ Oxford
−20
−0.5
0
x
0.5
1
10
−1
Old chebop
Preconditioned collocation
US method
−0.5
0
0.5
1
x
9/21
A fast and well-conditioned spectral method
A high-order example
u(10) (x) + cosh(x)u(8) (x) + cos(x)u(2) (x) + x 2 u(x) = 0
u(±1) = 0, u0 (±1) = 1, u(k ) (±1) = 0, k = 2, 3, 4.
0
0.5
10
0.4
−5
0.3
10
||un+1−un||2
0.2
u(x)
0.1
0
−0.1
−10
Z
10
−15
10
(ũ(x) + ũ(−x))2
−1
−0.2
= 1.3 × 10−14 .
−20
−0.3
! 12
1
10
−0.4
−0.5
−1
−25
−0.5
0
x
Alex Townsend @ Oxford
0.5
1
10
20
40
60
80
100
120
140
n
10/21
Chebop and Chebop2
Convenience for the user
L = chebop(@(x,u) diff(u,2)-x.*u,[-30 30]); % Airy equation
L.lbc = 1; L.rbc = 0;
% Set boundary conditions
u = L \ 0;
% u = chebfun
Convert
handle into
discretisation
instructions
Impose
bcs and
solve
à x = b
Construct
discretisation
A ∈ Rn×n
Converged
to the
solution?
yes
Construct
a chebfun
no, increase n
L = chebop2(@(x,y,u) laplacian(u)+(1000+y)*u);% Helmholtz with gravity
L.lbc = 1; L.rbc = 1; L.ubc = 1; L.dbc = 1;% Set boundary conditions
u = L \ 0;
% u = chebfun2
Convert
handle into
discretisation
instructions
Construct a
matrix equation
with an ny × nx
solution matrix
Impose
bcs and
solve
matrix
equation
Converged
to the
solution?
yes
Construct
a chebfun2
no, increase nx or ny or both
Alex Townsend @ Oxford
11/21
Chebop and Chebop2
Convenience for the user
L = chebop(@(x,u) diff(u,2)-x.*u,[-30 30]); % Airy equation
L.lbc = 1; L.rbc = 0;
% Set boundary conditions
u = L \ 0;
% u = chebfun
Convert
handle into
discretisation
instructions
Impose
bcs and
solve
à x = b
Construct
discretisation
A ∈ Rn×n
Converged
to the
solution?
yes
Construct
a chebfun
no, increase n
L = chebop2(@(x,y,u) laplacian(u)+(1000+y)*u);% Helmholtz with gravity
L.lbc = 1; L.rbc = 1; L.ubc = 1; L.dbc = 1;% Set boundary conditions
u = L \ 0;
% u = chebfun2
Convert
handle into
discretisation
instructions
Construct a
matrix equation
with an ny × nx
solution matrix
Impose
bcs and
solve
matrix
equation
Converged
to the
solution?
yes
Construct
a chebfun2
no, increase nx or ny or both
Alex Townsend @ Oxford
11/21
Interpreting user-defined input
Automatic differentiation
Implemented by
forward-mode operator
overloading
Interpret anonymous
function as a sequence
of elementary
operations
Can also calculate
Fréchet derivatives
Key people:
Ásgeir Birkisson and
Toby Driscoll
Alex Townsend @ Oxford
uxx + uyy + 50u + yu
+
.*
+
+
uxx
u
*
uyy
y
u
u
u
12/21
Low rank approximation
Numerical rank
For A ∈ Cm×n , SVD gives best rank k wrt 2-norm [Eckart & Young 1936]
A=
min(m,n)
X
σj uj vj∗
≈
j=1
k
X
σj uj vj∗ ,
σk +1 < tol.
j=1
For Lipschitz smooth bivariate functions [Schmidt 1909, Smithies 1937]
f (x, y) =
∞
X
σj uj (y)vj (x) ≈
j=1
k
X
σj uj (y)vj (x).
j=1
For compact linear operators acting on functions of two variables,
!
L=
∞
X
j=1
Alex Townsend @ Oxford
y
σj Lj
⊗
Lxj
≈
k
X
y
σj Lj ⊗ Lxj .
j=1
13/21
Low rank approximation
Numerical rank
For A ∈ Cm×n , SVD gives best rank k wrt 2-norm [Eckart & Young 1936]
A=
min(m,n)
X
σj uj vj∗
≈
j=1
k
X
σj uj vj∗ ,
σk +1 < tol.
j=1
For Lipschitz smooth bivariate functions [Schmidt 1909, Smithies 1937]
f (x, y) =
∞
X
σj uj (y)vj (x) ≈
j=1
k
X
σj uj (y)vj (x).
j=1
For compact linear operators acting on functions of two variables,
!
L=
∞
X
j=1
Alex Townsend @ Oxford
y
σj Lj
⊗
Lxj
≈
k
X
y
σj Lj ⊗ Lxj .
j=1
13/21
Low rank approximation
Numerical rank
For A ∈ Cm×n , SVD gives best rank k wrt 2-norm [Eckart & Young 1936]
A=
min(m,n)
X
σj uj vj∗
≈
j=1
k
X
σj uj vj∗ ,
σk +1 < tol.
j=1
For Lipschitz smooth bivariate functions [Schmidt 1909, Smithies 1937]
f (x, y) =
∞
X
σj uj (y)vj (x) ≈
j=1
k
X
σj uj (y)vj (x).
j=1
For compact linear operators acting on functions of two variables,
!
L=
∞
X
j=1
Alex Townsend @ Oxford
y
σj Lj
⊗
Lxj
≈
k
X
y
σj Lj ⊗ Lxj .
j=1
13/21
Low rank approximation
Do the low rank stuff before discretization
Low rank-then-discretize: Instead of low rank techniques after discretization,
do them before.
For example, Helmholtz is of rank 2
∇2 u + K 2 u = (uxx +
K2
K2
K2
K2
u) + (uyy +
u) = (D2 +
I) ⊗ I + I ⊗ (D2 +
I).
2
2
2
2
Let A be your favourite ODE discretization of D2 +
K2
2 I,
then (typically)
AXI + IXA T .
In general, if L is of rank k we have
k
X
Aj XBjT = F
j=1
Alex Townsend @ Oxford
14/21
Low rank approximation
Do the low rank stuff before discretization
Low rank-then-discretize: Instead of low rank techniques after discretization,
do them before.
For example, Helmholtz is of rank 2
∇2 u + K 2 u = (uxx +
K2
K2
K2
K2
I) ⊗ I + I ⊗ (D2 +
I).
u) + (uyy +
u) = (D2 +
2
2
2
2
Let A be your favourite ODE discretization of D2 +
K2
2 I,
then (typically)
AXI + IXA T .
In general, if L is of rank k we have
k
X
Aj XBjT = F
j=1
Alex Townsend @ Oxford
14/21
Low rank approximation
Do the low rank stuff before discretization
Low rank-then-discretize: Instead of low rank techniques after discretization,
do them before.
For example, Helmholtz is of rank 2
∇2 u + K 2 u = (uxx +
K2
K2
K2
K2
I) ⊗ I + I ⊗ (D2 +
I).
u) + (uyy +
u) = (D2 +
2
2
2
2
Let A be your favourite ODE discretization of D2 +
K2
2 I,
then (typically)
AXI + IXA T .
In general, if L is of rank k we have
k
X
Aj XBjT = F
j=1
Alex Townsend @ Oxford
14/21
Low rank approximation
Computing the rank of a partial differential operator
Recast differential operators as polynomials: Once you have polynomials
computing the rank is easy.
The rank of
L=
Ny X
Nx
X
aij (x, y)
i=0 j=0
∂i ∂j
∂y i ∂x j
equals a TT-rank [Oseledets 2011] (between {x, s} and {y, t}) of
h(x, s, y, t) =
Ny X
Nx
X
i j
aij (s, t)y x =
i=0 j=0
Rank 1:
ODEs
Trivial PDEs
Alex Townsend @ Oxford
Rank 2:
Laplace, Helmholtz
Transport, Heat, Wave
Black-Scholes
k
X
cj (t, y)rj (s, x).
j=1
Rank 3:
Biharmonic
Lots here.
15/21
Low rank approximation
Computing the rank of a partial differential operator
Recast differential operators as polynomials: Once you have polynomials
computing the rank is easy.
The rank of
L=
Ny X
Nx
X
aij (x, y)
i=0 j=0
∂i ∂j
∂y i ∂x j
equals a TT-rank [Oseledets 2011] (between {x, s} and {y, t}) of
h(x, s, y, t) =
Ny X
Nx
X
i j
aij (s, t)y x =
i=0 j=0
Rank 1:
ODEs
Trivial PDEs
Alex Townsend @ Oxford
Rank 2:
Laplace, Helmholtz
Transport, Heat, Wave
Black-Scholes
k
X
cj (t, y)rj (s, x).
j=1
Rank 3:
Biharmonic
Lots here.
15/21
Low rank approximation
Computing the rank of a partial differential operator
Recast differential operators as polynomials: Once you have polynomials
computing the rank is easy.
The rank of
L=
Ny X
Nx
X
aij (x, y)
i=0 j=0
∂i ∂j
∂y i ∂x j
equals a TT-rank [Oseledets 2011] (between {x, s} and {y, t}) of
h(x, s, y, t) =
Ny X
Nx
X
i j
aij (s, t)y x =
i=0 j=0
Rank 1:
ODEs
Trivial PDEs
Alex Townsend @ Oxford
Rank 2:
Laplace, Helmholtz
Transport, Heat, Wave
Black-Scholes
k
X
cj (t, y)rj (s, x).
j=1
Rank 3:
Biharmonic
Lots here.
15/21
Low rank approximation
Construct a nx by ny generalised Sylvester matrix equation
If the PDE is Lu = f , where L is of rank-k then we solve for X ∈ Cny ×nx in,
k
X
σj Aj XBjT = F,
Aj ∈ Cny ×ny ,
Bj ∈ Cnx ×nx .
j=1
X = solution’s coefficients


Aj = 
Alex Townsend @ Oxford
y
Aj , Bj = 1D spectral discretization of Lj , Lxj




16/21
Low rank approximation
Matrix equation solvers
Rank 1: A1 XB1T = F. Solve A1 Y = F, then B1 X T = Y T .
Rank 2: A1 XB1T + A2 XB2T = F. Generalised Sylvester solver (RECSY)
[Jonsson & Kågström, 2002].
Rank k, k ≥ 3: Solve N × N system using almost banded structure.
2
/2
O(
N3
Execution time
1
10
)
O(
N2
)
10
0
10
N)
(
O
−1
10
blue = rank 1
green = rank 2
red = rank 3
−2
10
−3
10
1
10
Alex Townsend @ Oxford
2
√
N
10
3
10
17/21
Examples
Helmholtz equation
∇2 u + 2ω2 u = 0,
u(x, ±1) = f (x, ±1),
u(±1, y) = f (±1, y),
where f (x, y) = cos(ωx) cos(ωy).
Wave numbers vs. polynomial degree
ω = 50
600
1
500
0.5
0
n
y
400
300
200
−0.5
100
−1
−1
Alex Townsend @ Oxford
−0.5
0
x
0.5
1
0
0
100
200
300
400
500
ω
18/21
Examples
Variable helmholtz equation
N = chebop2(@(x,y,u) laplacian(u) + 10000(1/2+sin(x)ˆ2).*cos(y)ˆ2.*u);
N.lbc = 1; N.rbc = 1; N.ubc = 1; N.dbc = 1;
u = N \ chebfun2(@(x,y) cos(x.*y));
0
Absolute maximum error
10
−5
10
−10
10
−15
10
200
N = 1,050,625,
Alex Townsend @ Oxford
error ≈ 1.47 × 10−13 ,
400
√ 600
N
800
1000
time = 44.2s.
19/21
Examples
Wave and Klein–Gordon equation
N = chebop2(@(u) diff(u,2,1) - diff(u,2,2) + 5*u); % u_tt - u_xx + 5u
N.dbc = @(x,u) [u-exp(-10*x) diff(u)]; N.lbc = 0; N.rbc = 0;
u = N \ 0;
Alex Townsend @ Oxford
20/21
Conclusion
Spectral methods do not have to be ill-conditioned. (Don’t discretize
differentiation faithfully.)
Spectral methods are extremely convenient and flexible.
As of 2014, global spectral methods are heavily restricted to a few
geometries.
Thank you for listening
Alex Townsend @ Oxford
21/21