What is Numerical Analysis? I Numerical analysis uses algorithms (a set of rules) to approximate solutions to mathematical problems, which cannot be solved analytically, or fit a mathematical model to experimental data. I The method of computation uses only simple operations of arithmetic. “ARITHMETIC” = The Algebra of Numbers, An “Algebra” uses Equality (=), Addition (+), Subtraction (-), Multiplication (× or .) and Division (÷). What functions can be approximated? When trying to find an approximative solution to a problem we generally require a function to I have no gaps, ie to be continuous, and I be comparatively smooth (differentiable). Questions associated with numerical problems I Will this be the only solution? I What other information about the function can we use to choose an appropriate method, for example about its smoothness. I I Is the used method stable? Does it work for any initial values? How fast does it converge? I What can we say about the error bounds? I How will the error propagate when we iterate a large number of steps. I What stopping criterion do we apply? I How does computer arithmetics (limited number of digits) influence our results? I How much RAM will I need? Will I need a parallel compute cluster to perform the computations? Are there optimized compilers, numerical libraries that should be used? I How do I visualise the results? Computer Arithmetics I Why are the following statements not generally true when computing with machine numbers? (a + b) c is a c + b a ⇤ (b c) is a ⇤ b a ⇤ c What are round-off errors and what has the representation of numbers in the computer to do with it? I The origin of round-off errors is finite digit arithmetic How is a typical 64bit machine number represented? I 64bit Machine numbers are composed of sign (1 bit) s + characteristic (11 bit exponent) e + mantissa m (52 bit binary fraction): e.g. 1 ⇤ 20 + 1 ⇤ 2 1 +0⇤2 2 +1⇤2 3 + ··· The base for exponent is 2. I 52 binary digits correspond to between 15 and 16 decimal digits, therefore we have at least 15 decimal digits of precision. I eps= 2^(-52) is the smallest number that can be stored in the mantissa f . It is the distance from 1 to the next larger floating-point number and is related to the number of significant digits. Which different errors are used to specify the accuracy of a solution? For the real value p and its approximation p⇤ we define: I I I I Absolute error: |p Relative error: |p p⇤ | p⇤ |/|p| Approximated error (in iterations) |pt⇤ pt⇤ 1 | Relative approximated error |pt⇤ pt⇤ 1 |/|pt⇤ | A bound for the relative error when using k-digit rounding arithmetic is 5 ⇥ 10 k . What consequences does this have for relative and absolute errors involved in rounding of machine numbers? I For machine numbers the bounds for the relative error are independent of the actual number being represented. The size of the relative error does not depend on whether the number is small or big. Vice versa the absolute precision of a floating point number depends on its magnitude. What is the relation between relative error and significant digits? Definition The number p⇤ is said to approximate p to t significant digits (or figures) if t is the largest nonnegative integer for which |p p⇤ | 5 ⇥ 10 |p| t Which operation results in the worst loss of significant digits? I Subtraction of nearly equal numbers results in a cancellation of significant digits and is the most common error-producing calculation. Let p = 0.54617 and q = 0.54601 . The exact value of r = p q is r = 0.00016 . Let’s use 4 digit rounding arithmetic so that p⇤ = 0.5462 and q ⇤ = 0.5460 . Worryingly the result r ⇤ = p⇤ q ⇤ = 0.0002 leaves us with only 1 significant digit and results in the large relative error of 0.25 . How can we avoid round-off errors, when for example evaluating polynomials? I Nesting reduces the number of arithmetic calculations and polynomials should always be formulated in a nested form. Uses nesting requires n multiplications and n additions to evaluate any n ’th degree polynomial Taylor’s Theorem I What are the conditions? I What is the truncation error and why are we interested in finding realistic bounds for it? I What values can the function ⇠(x) take which appears in the truncation error? How is the order of a numerical method related to Taylor polynomials? Definition A numerical method is of order n, where n is a positive integer, if the method is exact for polynomials of degree n or less. If the true solution is a polynomial of degree n, then the approximated solution obtained by a method of order n will be identical. This implies, that the higher the order, the more accurate the method. What does “big Oh” stand for? A function f (x) approximated arounjd x0 by the n’th Taylor polynomial in the form of f (x) = Pn (x) + Rn (x) can be written as: f (x) = Pn (x) + O(hn+1 ) with h = x x0 n+1 I O(h ) here stands for omitting terms starting with the power hn+1 . I As x ! x0 the truncation error converges to zero with the same rapidity that hn+1 converges to zero Total error I The total error is the sum of truncation and round-off errors What is a stable algorithm? Definition E0 denotes the magnitude of an error introduced at some stage in the calculations En is the error after n calculations I If En ⇡ CnE0 the growth of error is said to be linear (C is a constant independent of n ): stable algorithms if C and E0 are small I En ⇡ C n E0 for some C > 0 , then the growth of the error is exponential and the algorithm is unstable Root finding problems The Bisection Method Figure : Bisection Method From BURDEN/FAIRES. Numerical Analysis, International Edition, 8E. © 2005 Brooks/Cole, a part of Cengage Learning, Inc. Reproduced by permission. www.cengage.com/permissions I I I What requirements must f meet? Which theorem tells you that there exists a number p in [a, b] with f (p) = 0 (p is a root)? What can we say about the convergence of the bisection method? What is a fixed-point problem I We call p a fixed point for a given function if g(p) = p Figure : Fixed points for g(x) = x 2 From BURDEN/FAIRES. Numerical Analysis, International Edition, 8E. © 2005 Brooks/Cole, a part of Cengage Learning, Inc. 2 What is the connection between fixed-point and root finding problems? I Root-finding and fixed-point problems are equivalent I For a problem f (p) = 0 we can construct a function g in various ways, e.g. g(x) = x I f (x) Consequently, if g(p) = p is a fixed-point then f (p) must be zero. How can we iteratively approximate a fixed-point? Figure : Fixed-point iteration From BURDEN/FAIRES. Numerical Analysis, International Edition, 8E. © 2005 Brooks/Cole, a part of Cengage Learning, Inc. Reproduced by permission. www.cengage.com/permissions I I Choose an initial approximation p0 and generate the sequence 1 {pn }n=0 by letting pn = g(pn 1 ) , for each n 1 . |g 0 (x)| k with k < 1 is required for convergence (see the Fixed-point Theorem). Equivalence of Newton’s method and fixed point iterations I The fixed-point theorem states that under certain conditions the sequence pn = g(pn 1 ) converges to the unique fixed-point p . I Newton’s method pn = g(pn 1) = pn 1 f (pn 1 ) f 0 (pn 1 ) is equivalent to a fixed-point iteration, and converges provided we start with a good enough initial approximation. I Newton’s method is quadratically convergent. What does that mean? Order of convergence for iterative methods Definition 1 Suppose {pn }n=0 is a sequence that converges to p , with pn 6= p for all pn . If positive constants and ↵ exist with lim n!1 1 |pn+1 p| ↵ = |pn p| then {pn }n=0 converges to p of order ↵ , with asymptotic error constant . An iterative technique of the form pn = g(pn 1 ) is said to be of order 1 ↵ if the sequence {pn }n=0 converges to the solution p = g(p) of order ↵. We are mainly interested in the order ↵ . A higher ↵ means more rapid convergence of the iterative method. I If ↵ = 1 , the sequence is linearly convergent I If ↵ = 2 , the sequence is quadratically convergent solving ODEs Initial Value Problems for Ordinary Differential Equations We speak of an initial value problem (IVP), if we want to solve an ODE dy = f (t, y ) , for a t b (1) dt with initial conditions y (a) = ↵ . I I We have to introduce a more rigorous definition of continuity (Lipschitz condition) to make sure that we obtain a unique solution and have a well posed problem. The error of a perturbed problem then stays bounded. Euler’s method I We are looking for an approximation to the well-posed initial value problem dy = f (t, y ) , for a t b dt (2) I We discretize the problem and approximate the solution at mesh points that are a step size h apart. We might want to interpolate afterwards to find intermediate approximations. I Using Taylor series we can derive the following difference equation for Euler’s method wi+1 = wi + hf (ti , wi ) , with i = 0, 1, 2, . . . , N and starting value w0 = ↵ . 1 I The error bound for the Euler method depends linearly on the step size h . I Is there a lower limit for h ? I The Euler method has a local truncation error of O(h) . Using higher order Taylor polynomials one can achieve higher order local truncation errors O(hn ) , but in practise the computation of the higher order derivatives often turns out to be too complicated. In what sense is the Improved Euler method improved? I The improved Euler method uses the mean of the derivative at both ends of the interval h to compute our next approximation. 1 wi+1 = wi + h [f (ti , wi ) + f (ti , ŵi+1 )] 2 I The truncation error for the improved Euler method isO(h2 ) . I Why is this method called a predictor-corrector method? What are very common higher order methods for solving ODEs? I Runge-Kutta methods, eg for the fourth order RK method the local truncation for error is O(h4 ) . Four evaluations per step are needed. What are adaptive methods? What are multistep methods? Multivariable ODEs An m ’th order system of first-order initial value problems has the form du1 (t) dt du2 (t) dt dum (t) dt = f1 (t, u1 , u2 , . . . , um ) = f2 (t, u1 , u2 , . . . , um ) .. . = fm (t, u1 , u2 , . . . , um ) for a t b , with initial conditions u1 (a) = ↵1 , u2 (a) = ↵2 , . . . , um (a) = ↵m The problem is now to solve for all variables u1 (t) , . . . , um (t). I The methods for single variable systems can be easily extended to multivariable systems Convergence of difference equations I A one step method is convergent when the solution to the difference equation approaches the solution to the differential equation as the step size goes to zero lim max |wi h!0 1iN y (ti )| = 0 (Rate of convergence for Euler’s method is for example O(h) .) Numerical solutions of diffusion problems I Fick’s second law: @c @2c =D 2 @t @x is the prototype of a parabolic PDE. Reaction-diffusion problems I A generic 2-variable reaction-diffusion system can be written as @u @t @v @t = f (u, v )+ = g(u, v )+ @2u @x 2 @2v Dv 2 @x Du where the functions f (u, v ) and g(u, v ) express the local reaction kinetics of the two variables u and v and the terms containing Du and Dv describe diffusion of u and v . Boundary conditions I Typical boundary conditions for initial value problems of reaction-diffusion models are: I I Dirichlet boundary conditions: u | = ↵ Neumann boundary conditions: @u | = ↵ , eg no-flux boundary @x conditions with @u | =0 @x What are finite difference methods? In finite difference methods derivatives are replaced with difference quotients. ⇣ 2 ⌘ @ c For example, using difference equations for @c we @t i,j and @x 2 I i,j obtain the following central difference equation for Fick’s 2nd law: ci,j+1 ci,j ci =D 4t 1,j 2ci,j + ci+1,j (4x)2 and obtain the recursive equation: ci,j+1 = ci,j + 4t D ci 1,j 2ci,j + ci+1,j (4x)2 which is of order O(4t, (4x)2 ) I What can we say about the stability of this explicit scheme? What is the method of lines?