What is Numerical Analysis?
I
Numerical analysis uses algorithms (a set of rules) to
approximate solutions to mathematical problems, which cannot
be solved analytically, or fit a mathematical model to
experimental data.
I
The method of computation uses only simple operations of
arithmetic.
“ARITHMETIC” = The Algebra of Numbers, An “Algebra” uses Equality
(=), Addition (+), Subtraction (-), Multiplication (× or .) and Division (÷).
What functions can be approximated?
When trying to find an approximative solution to a problem we
generally require a function to
I
have no gaps, ie to be continuous, and
I
be comparatively smooth (differentiable).
Questions associated with numerical problems
I
Will this be the only solution?
I
What other information about the function can we use to choose
an appropriate method, for example about its smoothness.
I
I
Is the used method stable? Does it work for any initial values?
How fast does it converge?
I
What can we say about the error bounds?
I
How will the error propagate when we iterate a large number of
steps.
I
What stopping criterion do we apply?
I
How does computer arithmetics (limited number of digits)
influence our results?
I
How much RAM will I need? Will I need a parallel compute
cluster to perform the computations? Are there optimized
compilers, numerical libraries that should be used?
I
How do I visualise the results?
Computer Arithmetics
I
Why are the following statements not generally true when
computing with machine numbers?
(a + b) c is a c + b
a ⇤ (b c) is a ⇤ b a ⇤ c
What are round-off errors
and what has the representation of numbers
in the computer to do with it?
I
The origin of round-off errors is finite digit arithmetic
How is a typical 64bit machine number represented?
I
64bit Machine numbers are composed of
sign (1 bit) s + characteristic (11 bit exponent) e + mantissa m
(52 bit binary fraction): e.g.
1 ⇤ 20 + 1 ⇤ 2
1
+0⇤2
2
+1⇤2
3
+ ···
The base for exponent is 2.
I
52 binary digits correspond to between 15 and 16 decimal digits,
therefore we have at least 15 decimal digits of precision.
I
eps= 2^(-52) is the smallest number that can be stored in the
mantissa f . It is the distance from 1 to the next larger
floating-point number and is related to the number of significant
digits.
Which different errors are used to specify the accuracy
of a solution?
For the real value p and its approximation p⇤ we define:
I
I
I
I
Absolute error: |p
Relative error: |p
p⇤ |
p⇤ |/|p|
Approximated error (in iterations) |pt⇤ pt⇤ 1 |
Relative approximated error |pt⇤ pt⇤ 1 |/|pt⇤ |
A bound for the relative error when using k-digit
rounding arithmetic is 5 ⇥ 10 k . What consequences
does this have for relative and absolute errors involved
in rounding of machine numbers?
I
For machine numbers the bounds for the relative error are
independent of the actual number being represented. The size of
the relative error does not depend on whether the number is
small or big. Vice versa the absolute precision of a floating point
number depends on its magnitude.
What is the relation between relative error and
significant digits?
Definition
The number p⇤ is said to approximate p to t significant digits (or
figures) if t is the largest nonnegative integer for which
|p
p⇤ |
 5 ⇥ 10
|p|
t
Which operation results in the worst loss of significant
digits?
I
Subtraction of nearly equal numbers results in a cancellation of
significant digits and is the most common error-producing
calculation.
Let p = 0.54617 and q = 0.54601 . The exact value of r = p q is
r = 0.00016 . Let’s use 4 digit rounding arithmetic so that
p⇤ = 0.5462 and q ⇤ = 0.5460 . Worryingly the result
r ⇤ = p⇤ q ⇤ = 0.0002 leaves us with only 1 significant digit and
results in the large relative error of 0.25 .
How can we avoid round-off errors, when for example
evaluating polynomials?
I
Nesting reduces the number of arithmetic calculations and
polynomials should always be formulated in a nested form.
Uses nesting requires n multiplications and n additions to
evaluate any n ’th degree polynomial
Taylor’s Theorem
I
What are the conditions?
I
What is the truncation error and why are we interested in finding
realistic bounds for it?
I
What values can the function ⇠(x) take which appears in the
truncation error?
How is the order of a numerical method related to
Taylor polynomials?
Definition
A numerical method is of order n, where n is a positive integer, if the
method is exact for polynomials of degree n or less.
If the true solution is a polynomial of degree n, then the approximated
solution obtained by a method of order n will be identical.
This implies, that the higher the order, the more accurate the method.
What does “big Oh” stand for?
A function f (x) approximated arounjd x0 by the n’th Taylor polynomial
in the form of f (x) = Pn (x) + Rn (x)
can be written as:
f (x) = Pn (x) + O(hn+1 )
with h = x
x0
n+1
I
O(h ) here stands for omitting terms starting with the power
hn+1 .
I
As x ! x0 the truncation error converges to zero with the same
rapidity that hn+1 converges to zero
Total error
I
The total error is the sum of truncation and round-off errors
What is a stable algorithm?
Definition
E0 denotes the magnitude of an error introduced at some stage in the
calculations
En is the error after n calculations
I
If En ⇡ CnE0 the growth of error is said to be linear (C is a
constant independent of n ): stable algorithms if C and E0 are
small
I
En ⇡ C n E0 for some C > 0 , then the growth of the error is
exponential and the algorithm is unstable
Root finding problems
The Bisection Method
Figure : Bisection Method
From BURDEN/FAIRES. Numerical Analysis, International Edition, 8E.
© 2005 Brooks/Cole, a part of Cengage Learning, Inc.
Reproduced by permission. www.cengage.com/permissions
I
I
I
What requirements must f meet?
Which theorem tells you that there exists a number p in [a, b]
with f (p) = 0 (p is a root)?
What can we say about the convergence of the bisection
method?
What is a fixed-point problem
I
We call p a fixed point for a given function if g(p) = p
Figure : Fixed points for g(x) = x 2
From BURDEN/FAIRES. Numerical Analysis, International Edition, 8E.
© 2005 Brooks/Cole, a part of Cengage Learning, Inc.
2
What is the connection between fixed-point and root
finding problems?
I
Root-finding and fixed-point problems are equivalent
I
For a problem f (p) = 0 we can construct a function g in various
ways, e.g.
g(x) = x
I
f (x)
Consequently, if g(p) = p is a fixed-point then f (p) must be zero.
How can we iteratively approximate a fixed-point?
Figure : Fixed-point iteration
From BURDEN/FAIRES. Numerical Analysis, International Edition, 8E.
© 2005 Brooks/Cole, a part of Cengage Learning, Inc.
Reproduced by permission. www.cengage.com/permissions
I
I
Choose an initial approximation p0 and generate the sequence
1
{pn }n=0 by letting pn = g(pn 1 ) , for each n 1 .
|g 0 (x)|  k with k < 1 is required for convergence (see the
Fixed-point Theorem).
Equivalence of Newton’s method and fixed point
iterations
I
The fixed-point theorem states that under certain conditions the
sequence
pn = g(pn 1 )
converges to the unique fixed-point p .
I
Newton’s method
pn = g(pn
1)
= pn
1
f (pn 1 )
f 0 (pn 1 )
is equivalent to a fixed-point iteration, and converges provided
we start with a good enough initial approximation.
I
Newton’s method is quadratically convergent. What does that
mean?
Order of convergence for iterative methods
Definition
1
Suppose {pn }n=0 is a sequence that converges to p , with pn 6= p for
all pn . If positive constants and ↵ exist with
lim
n!1
1
|pn+1 p|
↵ =
|pn p|
then {pn }n=0 converges to p of order ↵ , with asymptotic error
constant .
An iterative technique of the form pn = g(pn 1 ) is said to be of order
1
↵ if the sequence {pn }n=0 converges to the solution p = g(p) of order
↵.
We are mainly interested in the order ↵ . A higher ↵ means more
rapid convergence of the iterative method.
I
If ↵ = 1 , the sequence is linearly convergent
I
If ↵ = 2 , the sequence is quadratically convergent
solving ODEs
Initial Value Problems for Ordinary Differential
Equations
We speak of an initial value problem (IVP), if we want to solve an
ODE
dy
= f (t, y ) , for a  t  b
(1)
dt
with initial conditions y (a) = ↵ .
I
I
We have to introduce a more rigorous definition of continuity
(Lipschitz condition) to make sure that we obtain a unique
solution and have a well posed problem. The error of a perturbed
problem then stays bounded.
Euler’s method
I
We are looking for an approximation to the well-posed initial
value problem
dy
= f (t, y ) , for a  t  b
dt
(2)
I
We discretize the problem and approximate the solution at mesh
points that are a step size h apart. We might want to interpolate
afterwards to find intermediate approximations.
I
Using Taylor series we can derive the following difference
equation for Euler’s method
wi+1 = wi + hf (ti , wi ) , with i = 0, 1, 2, . . . , N
and starting value w0 = ↵ .
1
I
The error bound for the Euler method depends linearly on the
step size h .
I
Is there a lower limit for h ?
I
The Euler method has a local truncation error of O(h) . Using
higher order Taylor polynomials one can achieve higher order
local truncation errors O(hn ) , but in practise the computation of
the higher order derivatives often turns out to be too complicated.
In what sense is the Improved Euler method
improved?
I
The improved Euler method uses the mean of the derivative at
both ends of the interval h to compute our next approximation.
1
wi+1 = wi + h [f (ti , wi ) + f (ti , ŵi+1 )]
2
I
The truncation error for the improved Euler method isO(h2 ) .
I
Why is this method called a predictor-corrector method?
What are very common higher order methods for
solving ODEs?
I
Runge-Kutta methods, eg for the fourth order RK method the
local truncation for error is O(h4 ) . Four evaluations per step are
needed.
What are adaptive methods?
What are multistep methods?
Multivariable ODEs
An m ’th order system of first-order initial value problems has the form
du1
(t)
dt
du2
(t)
dt
dum
(t)
dt
=
f1 (t, u1 , u2 , . . . , um )
=
f2 (t, u1 , u2 , . . . , um )
..
.
=
fm (t, u1 , u2 , . . . , um )
for a  t  b , with initial conditions
u1 (a) = ↵1 , u2 (a) = ↵2 , . . . , um (a) = ↵m
The problem is now to solve for all variables u1 (t) , . . . , um (t).
I
The methods for single variable systems can be easily extended
to multivariable systems
Convergence of difference equations
I
A one step method is convergent when the solution to the
difference equation approaches the solution to the differential
equation as the step size goes to zero
lim max |wi
h!0 1iN
y (ti )| = 0
(Rate of convergence for Euler’s method is for example O(h) .)
Numerical solutions of diffusion problems
I
Fick’s second law:
@c
@2c
=D 2
@t
@x
is the prototype of a parabolic PDE.
Reaction-diffusion problems
I
A generic 2-variable reaction-diffusion system can be written as
@u
@t
@v
@t
= f (u, v )+
= g(u, v )+
@2u
@x 2
@2v
Dv 2
@x
Du
where the functions f (u, v ) and g(u, v ) express the local reaction
kinetics of the two variables u and v and the terms containing Du
and Dv describe diffusion of u and v .
Boundary conditions
I
Typical boundary conditions for initial value problems of
reaction-diffusion models are:
I
I
Dirichlet boundary conditions: u | = ↵
Neumann boundary conditions: @u
| = ↵ , eg no-flux boundary
@x
conditions with @u
| =0
@x
What are finite difference methods?
In finite difference methods derivatives are replaced with
difference quotients.
⇣ 2 ⌘
@ c
For example, using difference equations for @c
we
@t i,j and @x 2
I
i,j
obtain the following central difference equation for Fick’s 2nd law:
ci,j+1 ci,j
ci
=D
4t
1,j
2ci,j + ci+1,j
(4x)2
and obtain the recursive equation:
ci,j+1 = ci,j + 4t D
ci
1,j
2ci,j + ci+1,j
(4x)2
which is of order O(4t, (4x)2 )
I What can we say about the stability of this explicit scheme?
What is the method of lines?