Ordinary Differential Equations
Jyun-Ming Chen
1
Contents
• Review
• Euler’s method
• 2nd order methods
– Midpoint
– Heun’s
•
•
•
•
Systems of ODE
Stability Issue
Implicit Methods
Adaptive Stepsize
• Runge-Kutta Method
2
Review
• DE (Differential Equation)
– An equation specifying the relations among the
rate change (derivatives) of variables
• ODE (Ordinary DE) vs. PDE (Partial DE)
– The number of independent variables involved
2
d x
dt
2
 x
x (t )
f
x

f
y
1
f ( x, y )
3
Review (cont)
• Solution of DE vs. Solution of Equation
• Solution of an
equation:
f ( x)  x  1  0
2
Solution
• Geometrically,
f(x)
: x   1  f (  1)  0
x
4
Review (cont)
• Solution of an
differential equation:
dx
Need additional
conditions to
specify a solution
• Geometrically:
x
 x
dt
Solution
x  c e
:
t

 c e
t


 c  e
 That is,

t
t
x

e
,
x


2
e
,

 are all valid solutions






t
t
5
Review (cont)
• Order of an ODE
– The highest derivative in the equation
• nth order ODE requires n conditions to
specify the solution
– IVP (initial value problem): All conditions
specified at the same (initial) point
– BVP (boundary value problem): otherwise
6
y
IVP VS. BVP
Revisit Shooting Problem
y  f (t )
y   g (let g  10 )
g
2
d y
dt
t
dy
2
  10
  10 t  c1
(1)
y   5 t  c1 t  c 2
(2)
dt
2
7
IVP vs. BVP
IVP :
BVP :
y (t  0 )  0
y (t  0 )  0
y ( t  0 )  5
y ( t  10 )  100
c2  0
y (0)  c2  0
c1  5
y (10 )   500  10 c1  100  c1  60
 y (t )   5t  5t
2
Physical
meaning
 y   5 t  60 t
2
8
Ode2: solves 1st and 2nd order ODE
Ic1, ic2, bc: setting conditions
‘ do not evaluate
Maxima on ODE
dy
dx
2
  y , y (0)  2
d x
dt
2

dx
 x  sin t , x ( 0 )  1, x ( 0 )  0
dt
9
Linear ODE
• Linearity:
– Involves no product nor nonlinear functions of
y and its derivatives
• nth order linear ODE
n
an ( x)
d y
dx
n
 a n 1 ( x )
d
n 1
dx
y
n 1
   a1 ( x )
dy
dx
 a0 ( x) y  b( x)
10
Focus of This Chapter
• Solve IVP of nth order ODE numerically
• e.g.,
2
d y
dx
2
  x , initial
conditions
 y (0)  1
 dy
 dx ( 0 )  0
Solve y ( x ), x  0
11
ODE (IVP)
• First order ODE (canonical form)
dy
dx
 f ( x , y ), y ( x  0 )  y 0
• Every nth order ODE can be converted to n
first order ODEs in the following method:
12
n
an ( x)
d y
dx
n
 a n 1 ( x )
dx
dy 1
y1 ( x )  y ( x )
dx
y2 ( x) 
dy 1
dx
dx
y3 ( x) 
dy 2
d y

dy
dy 2
dx
2

dx
dx
dx

d
n 1
dx
n 1
   a1 ( x )
dy
dx
 a0 ( x) y  b( x)
 y2
 y3
dy n 1
dy n 1
y

2

yn ( x) 
d
n 1
y
n 1
dx
an ( x)
 yn
dy n
dx
 a n 1 ( x ) y n    a 1 ( x ) y 2  a 0 ( x ) y 1  b ( x )
13
Example
3
d y
dx
3
2
2
d y
dx

y1  y

dy
dy
Let  y 2  dx1  dx
 y  dy 2  d 2 y
2
dx
dx
 3
2
x
2
dy
 5x
dx
 dy 3
2

2
y

x
y2  5 x
3
 dx

 dy 2
 y3

 dx
 dy 1
 y2

 dx
y2
 y1 


 


   
Let y   y 2  , y   f ( y )  
y3

y 
 x2 y  2 y  5x
2
3
 3


14
End of Review
15
The Canonical Problem
dy
dx
y
x
 f ( x , y ), y ( x 0 )  y 0

dy
 f ( x, y )
dx
 x  0,
y
x

dy
dx
y  f ( x, y )x
y ( x  x)  y ( x)  y  f ( x, y )x
 y ( x  x)  y ( x)  f ( x, y )x
This is
Euler’s
method
16
Example
dy
  y , y (0)  1
dx
Compare with exact sol:
f ( x , y )   y , f ( 0 ,1)   1
Choose  x  0 . 1
y ( 0 . 1)  y ( 0 )  (  1)  0 . 1  1  0 . 1  0 . 9
y ( 0 . 2 )  y ( 0 . 1)  (  0 . 9 )  0 . 1  0 . 9  0 . 09  0 . 81
y ( 0 . 3 )  y ( 0 . 2 )  (  0 . 81 )  0 . 1  0 . 81  0 . 081  0 . 729

17
Example (cont)
y
1
Exact Solution
dy

:
  dx
y

ln y   x  c

y=e–x
x
y  c e
 y e
y (0)  c   1
x
x
0 .1
18
Error Analysis
(Geometric Interpretation)
dy
 f ( x , y ), y ( x 0 )  y 0
dx
y
x

dy
 f ( x, y )
dx
y  f ( x, y )x
y1  y 0   y  f ( x 0 , y 0 )  x
 y1  y 0  f ( x 0 , y 0 )  x
Think in
terms of
Taylor’s
expansion
If the true solution were a straight line, then
19
Euler is exact
Error Analysis
(From Taylor’s Expansion)
y ( x   x )  y ( x )  y  ( x )  x  y  ( x )
 y ( x)  f ( x, y )x 
Euler’s
d
dx
x
2
2
 y ( x )
f ( x, y )
x
2
2
x
6
3


Euler’s truncation error
O(x2) per step
1st order method
20
y
Cumulative Error



x
x=0
x=T
Remark:
x Error 
But computation time
Number of steps = T/x
Cumulative Err. = (T/x)  O(x2) = O(x)
21
Example (Euler’s)
dy
dx
  y, y0  1
exact : y ( x )  e
x
f ( x, y )   y ( x)
y ( x  x)  y ( x)  y ( x)x
Let  x  0 . 1
y ( x   x )  y ( x )( 1   x )  0 . 9 y ( x )
y ( 0 . 1)  0 . 9 , y ( 0 . 2 )  0 . 81 , y ( 0 . 3 )  0 . 729 ,...
22
Methods Improving Euler
Motivated by
Geometric Interpretation
23
Midpoint Method
y(x 
x
2
)  y ( x )  f ( x , y ( x ))
y( x  x)  y( x)  f ( x 
x
2
x
2
, y(x 
x
2
))  x
24
Example (Midpoint)
dy
dx
  y, y0  1
exact : y ( x )  e
x
f ( x, y )   y
y(x 
x
2
)  y ( x)  f ( x, y )
 y( x)  y( x)
x
2
y( x  x)  y( x)  f ( x 
 y( x) 
x
2
 y ( x ) 1 
x
2
, y(x 
x
2
x
2

))  x
  y ( x ) 1  2x  x
y ( x   x )  y ( x )   y ( x ) 1 
x
2
 x
Let  x  0 . 1
y ( x  0 . 1)  y ( x )  y ( x )( 1  0 . 05 )( 0 . 1)  0 . 905 y ( x )
y ( 0 . 1)  0 . 905 , y ( 0 . 2 )  0 . 819 , y ( 0 . 3 )  0 . 741 ,...
25
Heun’s Method
y ( x   x )  y ( x )  f ( x , y ( x ))  x
0
y( x  x)  y( x) 
1
2
 f ( x, y ) 
(Predictor )

f ( x   x , y ( x   x ))  x
0
26
Example (Heun’s)
dy
dx
  y, y0  1
exact : y ( x )  e
x
f ( x, y )   y
y ( x  x)  y ( x)  f ( x, y )x
0
 y ( x )  y ( x )  x  y ( x ) 1   x 
y(x  x)  y(x) 
1
2
 y(x) 
1
2
 f ( x , y )  f ( x   x , y ( x   x ))   x
 y ( x )  y ( x   x )  x
 y(x) 
1
2

0
0
y ( x )  y ( x ) 1   x  x
Let  x  0 . 1
y ( x  0 . 1)  y ( x ) 1 
1
2
  1  0 . 9 0 . 1  0 . 905
Note the result
is the same as
Midpoint!?
y( x)
y ( 0 . 1)  0 . 905 , y ( 0 . 2 )  0 . 819 , y ( 0 . 3 )  0 . 741 ,...
27
Remark
• Comparison of Euler,
Heun, midpoint
– 1st order: Euler
– 2nd order: Heun,
midpoint
• “order”:
• All are special cases of
RK (Runge-Kutta)
methods
Exact
Euler
(error)
Midpoint
(error)
y(0.1)
0.905
0.9
(0.005)
0.905
(0)
y(0.2)
0.819
0.81
(0.009)
0.819
(0)
y(0.3)
0.741
0.729
(0.012)
0.741
(0)
28
RK Methods
y i 1  y i   ( x i , y i , h ) h
where
y i  y ( x i ),
y i 1  y ( x i  h )
 ( x i , y i , h ) : incrementa
(estimate
l function
slope from i to i  1)
29
RK Methods (cont)
  a1 k 1  a 2 k 2    a n k n
a ' s : constant
k1  f ( x i , y i )
k 2  f ( x i  p 1 h , y i  q 11 k 1 h )
k 3  f ( x i  p 2 h , y i  q 21 k 1 h  q 22 k 2 h )

k n  f ( x i  p i h , y i  q n  1 ,1 k 1 n    q n  1 , n  1 k n  1 n )
p i , q ij : constants
 Constants
( a i , p i , q ij ) are chosen to
match Taylor' s expansion
30
Taylor’s Expansion
y ( x i  h )  y ( x i )  y ( x i ) h 
d
dx
y ( x i )
f
2
 y ( x i )  f ( x i , y i ) h  [ x 
f
i
f
 y i  f i h  [ x 
i
f
y
i
fi ]
2
h
2
 O (h )
3
h
2
y
i
fi ]
2
h
2
 O (h )
3
 O (h )
3
d
f ( x, y ) 
dx

 f dx
 x dx
f
x


f
y
 f dy
 y dx

f
x

f
y
f
31
y'
f
y ( x i  h )  y i  f i h  [ x 
i
RK
st
1
f
y
2
i
f i ] h2  O ( h )
3
Order
n 1
  a1 k 1  a1 f ( x i , y i )
y i 1  y i  a1 f ( x i , y i ) h
Compare with Taylor' s expantion
a1  1
 y i 1  y i  f ( x i , y i ) h
Euler' s method
32
f
y ( x i  h )  y i  f i h  [ x 
i
RK
nd
2
f
y
2
i
f i ] h2  O ( h )
3
Order
n  2   a1 k 1  a 2 k 2
k 1  f ( x1 , y i )
k 2  f ( x i  p 1 h , y i  q 11 k 1 h )
f
x
y i 1  y i  ( a1 k 1  a 2 k 2 ) h
Expand
a1  a 2  1
fih :
f
y
a 2 p1 
2
h :
1
2
f i h : a 2 q 11 
2
1
k2 :
k 2  f ( xi , yi ) 
f
x
( p1 h ) 
i
y i 1  y i  a1 k 1 h  a 2 [ f i 
f
y
f
x
 y i  a1 f i h  a 2 f i h  a 2
( q 11 k 1 h )  O ( h )
2
i
p1 h 
f
x
f
y
q 11 k 1 h ] h  O ( h )
p1 h  a 2
2
3
f
y
q 11 f i h  O ( h33 )
2
3
2
RK
nd
2
Order (cont)
a1  a 2  1 
3 eqns,

2
1
h : a 2 p1  2 
4 unknowns
2
1 
f i h : a 2 q11  2

fih :
f
x
f
y
a 2  1  a1  0 , p1 
1
2
, q 11 
1
2
y i 1  y i  ( a1 k 1  a 2 k 2 ) h
 y i  k 2 h  y i  f ( x i  p1 h , y i  q11 k 1 h ) h
y i 1  y i  f ( x i 
1
2
h, yi 
1
2
f ( x i , y i )) h
y i 1  y i  ( a1 k 1  a 2 k 2 ) h  y i 
1
2
( k1  k 2 ) h
k 1  f ( x i , y i ), k 2  f ( x i  h , y i  f i h )
34
th
4
RK
Order
• Mostly commonly used one
• Higher order … more evaluation, but less
gain on accuracy
y n 1  y n 
1
6
( k1  2 k 2  2 k 3  k 4 ) h
where
Classical
4th order
RK
k 1  f ( x n ,y n )
k 2  f ( xn 
1
2
h,y n 
1
2
k1h )
k3  f ( xn 
1
2
h,y n 
1
2
k2h)
k 4  f ( x n  h,y n  k 3 h )
35
Classical
th
4
order RK
36
System of ODE
• Convert higher order ODE to 1st order
ODEs
• All methods equally apply, in vector form
37
Example (Mass-Spring-Damper System)
• Governing Equation
m x  c x  kx  0
• After setting the initial conditions x(0) and x’(0),
compute the position and velocity of the mass for
any t > 0
Initial Condition
c
k
m
x
38
Example (cont)
x1  x , x 2  x
then
 x1  x 2

m x 2  cx 2  kx 1  0  
1
(  kx 1  cx 2 )
 x 2 
m

 x1 ( 0 )  1
IC : 
 x2 (0)  0
n  Dim Euler






y ( x   x )  y ( x )  f ( x, y ) x, y (0)  y0
39
Example (cont)
 x2

  x1   




x    f (x,t)   1

 x2 
 m (  kx 1  cx 2 ) 
Assume m=1,c=1, k=1
(for ease of computation)
 
x2


f  
  x1  x 2 
set t=0.1
 
1


x ( 0 . 1)  x ( 0 . 0 )  f ( x ( 0 . 0 ), 0 . 0 )  0 . 1    
0
 0 

 0 . 1 
1 0
 
 1 


 
x ( 0 . 2 )  x ( 0 . 1)  f ( x ( 0 . 1), 0 . 1)  0 . 1  

0
.
1


 1 


  0 .1 
 0 .1



 0 . 1 

1

(

0
.
1
)


 
 0 . 99 


 
x ( 0 . 3 )  x ( 0 . 2 )  f ( x ( 0 . 2 ), 0 . 2 )  0 . 1  
  0 . 19 
 0 . 99 



0
.
19


 0 . 19



 0 . 1 
  0 . 99  (  0 . 19 ) 
 0 . 971 


  0 . 27 
40
Stability: Symptom
dy
  10 y , y ( 0 )  1
dx
exact solution
: y(x)  e
 10 x
41
Symptom: Unstable Spring System
Become unstable instantly …
Start with this …
Cause by stiff (k=4000) springs
42
Stability (cont)
• Example Problem:
dy
  y , y ( 0 )  1,   0
Euler :
dx
y i  1  y i   y i h  (1   h ) y i
exact solution
: y( x)  e
 x
stable if 1   h  1
 unstable if 1   h  1



 That is,
  stability limit h *  2
 1   h  1, or 1   h   1



h  0
 h  2 

Conditionally stable
exact solution
: y( x)  e
 x
43
Discussion
• Different algorithm different stability limit
– Check Midpoint Method
• Different problem different stability limit
– use the previous problem as benchmark
44
Review: Numerical Differentiation
Taylor’s expansion:
Forward difference
Backward difference
45
Numerical Difference (cont)
Central difference
46
Implicit Method (Backward Euler)
dy
dx
 f ( x, y )
Euler : ( i  1, i )
y i 1  y i
h
Forward difference
 O ( h )  f ( xi , yi )
 y i 1  y i  f ( x i , y i ) h  O ( h )
2
Backward
y i  y i 1
h
Euler : ( i , i  1)
Backward difference
 O ( h )  f ( xi , yi )
 y i  f ( x i , y i ) h  y i 1  O ( h )
2
i 
   y i 1  f ( x i 1 , y i 1 ) h  y i  O ( h )
2
47
Example
f ( x , y )   y
 y i 1  f ( x i 1 , y i 1 ) h  y i 1   y i 1 h  y i  O ( h )
2
 (1   h ) y i  1  y i  O ( h )
2
y i 1 
1
1  h
yi
• Remark:
– Always stable (for this problem)
– Truncation error the same as Euler (only
improve the stability)
48
Stiff Set of ODE
Use the change of variable
u  2 y  z
v   y  z
u   v  y  u  v   y
u   2 v   z    1000 u  2 v    1000 z
Get the following solution:
y  c1 e
x
z  c2e
 1000 x
u ( 0 )  1  2 c1  c 2  1
v ( 0 )  1   c1  c 2  0  c1  c 2  1
Stability limit
A stiff equation is a differential equation for which
certain numerical methods for solving the equation
are numerically unstable, unless the step size is
49
taken to be extremely small
50
Linear System of ODE with
Constant Coefficients
y    Cy
Explicit
Implicit
Euler
Euler
y i  1  y i   hCy i  O ( h )
y i  y i 1   hCy i  O ( h )
y i  1  ( I  hC ) y i  O ( h )
 I  hC  y i
2
2
y i  1   I  hC  y i
2
y i 1
 y i 1  O ( h )
2
1


 I  hC
yi
Do not really use (..)-1.
Solve linear system
instead
51
Analysis
Explicit and implicit Euler both in the form: y   Ay
x
( i 1)
 Ax
(i )
x
(0)
 c1 x1  c 2 x 2
x
(1 )
 Ax
(0)

x
(2)
 Ax
(1 )
 A c1  1 x1  c 2  2 x 2   c1  1 Ax 1  c 2  2 Ax 2  c1  1 x1  c 2  2 x 2
x
(n)
 c1  1 x1  c 2  2 x 2
n
represent x(0) in eigen basis
A c1 x1  c 2 x 2   c1 Ax 1  c 2 Ax 2  c1  1 x1  c 2  2 x 2
2
2
n
x will converge if |i|  1
52
Analysis (cont)
Explicit
Euler
y i 1   I  hC  y i
i
: e - values of C
hi
: e - values of hC
1  hi
: e - values of I  hC
Implicit
Euler
y i  1   I  hC

1
1  h  i  1 : e - values
yi
of  I  hC
 1
• Stiff equations have large eigenvalues
• Explicit Euler requires small h to converge
• Implicit Euler always converges (in this problem)
53
Example

 
u 
  998
 u  998 u  1998 v
     


v
 999
 v   999 u  1999 v
Explicit
stable if 1  1000 h  1
  1  1  1000 h  1
 1998   u 
  
1999   v 
Implicit
stable if
1
1  1000 h
1
1000 h  2  h  0 . 002
54
Solving implicit methods by linearization is called
a “semi-implicit” method
Semi-Implicit Euler
• Not guaranteed to be stable, but usually is
Jacobian
y n  1  y n  hf  y n   h
f
y
y n 1  h
f
y
yn


f 
f 
I

h
y

I

h

 n 1 
 y n  hf  y n 
y 
y 


55
About Jacobian
Taylor’s expansion:
f
 x1  h , x 2  h   f 1  x1 , x 2  
1
f
 x1  h , x 2  h   f 2  x1 , x 2  
2
f 2
f
 x1  h , x 2  h   f 3  x1 , x 2  
3
f 3
f

f

 f
 x1  h , x 2  h  
1

 x 1  h , x 2  h  
2
 x1  h , x 2  h 
3
f

f

 f
 f1
 x1
 x1
 x1

h
h
 f1
x 2
f 2
x 2
f 3
x 2
f



x
,
x

1
2
x
1


f


x
,
x

1
2 
 x
2
 f



x
,
x
1
2 
3
 x
  
 
  f1 , f 2 , f 3  
f x  h  f x  
h
  x1 , x 2 

h
1
1
2
1
3
1
h  O (h )
2
h  O (h )
2
h  O (h )
2
 f1

h
f 2   
2

O
(
h
)
x 2   
h
f 3   
x 2 
x 2
Jacobian
56
damping
force
 c x
Initial Condition
c
m
k
Equation

x
of Motion :
f  m x   c x  kx  m x
m x  c x  kx  0
Implicit Euler
y    Cy
y i  1   I  hC

1
yi
Let m  c  k  1
x1  x 2
 x1  x

 

 x 2  x
 x 2   x1  x 2
0
y   
1
y n 1
1
 
h
 1
y
1 
h 

1  h
1
yn
57
fake damping
force
2
 c x
Initial Condition
c
k
Equation

of Motion :
2
f  m x   c x  kx  m x
m x  c x  kx  0
2
m
x
Let m  c  k  1
x1  x 2
 x1  x

 

2


x

x
x


x

x
1
2
 2
 2
y  f ( y)
 0
 
y  1
f
Semi-Implicit Euler
y n 1


 2 x2 
1
1
 yn  h 
h
h
1
x2
 

 
2
1  2 x 2 h    x1  58
x2 
Ex: Semi-Implicit Euler
 y 1  y 2
y   g , y ( 0 )  0 , y ( 0 )  3  
, y1 ( 0 )  0 , y 2 ( 0 )  3
 y 2   g
J 
y n 1
f
y

  f1 , f 2 
  y1 , y 2 
1
 yn  h 
0
0
 
0
 h

1 
1
 y1 ( t  h )   y1 ( t ) 

  
 
 y 2 (t  h )   y 2 (t ) 
1

0
1
fn  yn  h 
0
h

0
2
h   y 2 (t ) 

 
h   g 
h
 fn
1
Compare
Euler
y n  1  y n  hf n
 y1 ( t  h )   y1 ( t )   y 2 ( t ) 

  
  
 h
y
(
t

h
)
y
(
t
)

g

 2
  2  
59
Amazingly, this translates to…
 y1 ( t  h )   y1 ( t ) 

  
 
 y 2 (t  h )   y 2 (t ) 
h

0
2
h   y 2 (t ) 

 
h   g 
y ( t   t )  y ( t )  v ( t )  t  g  t 
2
v (t   t )  v (t )  g  t
Very similar to Verlet
integration formula… no
wonder Verlet is pretty stable
60
Adaptive Stepsize
• Solving ODE
numerically … tracing
the integral curve y(x)
• what’s wrong with
uniform step size
– Uniformly small: waste
effort
– Uniformly large: might
miss details
61
Step Doubling
• Idea:
– Estimate the truncation error by taking each
step twice: one full step, two half steps
– control the step size such that the estimated
error is not too big.
1(2h1)
Desired h0
2(h1)
62
Ex: RK
nd
2
Order
y ( x  2 h )  y1  c ( 2 h )  O ( h )
3
4
y ( x  2h)  y2  2c(h)  O (h )
3
4
y1  c ( 2 h )  O ( h )  y 2  2 c ( h )  O ( h )
3
4
3
4
  y 2  y 1  6 ch    h (the truncatio
3
3
3
1
3
 h0 
 0 
  h0  
 h1
 



 1  h1 
 1 
0
 1 : the estimated
h 0 : the stepsize
error by step h1
n error)
Overhead:
# of f(x,y) evaluations
24–2 = 6
to produce the desired error  0
63
Adaptive Step with RK4 (NR)
64
GSL
x¢¢ + m x¢ ( x -1) + x = 0, x(0) =1, x¢(0) = 0
2
find x (100)
x¢ = y
y¢ = -x + m y (1- x 2 )
65
Initialization
66
Iteration
67