Math 150 Ali Behzadan Lecture 5 Tuesday (2/9/2021), Lecture 6 Thursday (2/11/2021) 1 1) Review, Two fundamental questions 1-1) Analysis of errors 1-2) Analysis of the speed of the algorithm 2) Plan for the next several lectures Item 1: As we discussed, numerical analysis involves the design and "study" of algorithms for obtaining (approximate) solutions to various problems of continuous mathematics. In particular, our plan for Math 150 is to hopefully study the following topics: 1. Approximation of functions 2. Numerical differentiation and integration 3. Iterative methods for linear systems (Ax = b) Iterative methods for nonlinear equations (equations such as ex = x2 ) 4. Numerical methods for ordinary differential equations Also recall that there are two fundamental questions that we seek to answer in the study of algorithms of numerical analysis: (1) How accurate is the algorithm? (analysis of errors and stability) (2) How fast is the algorithm? (analysis of the speed of an algorithm) More generally, how expensive is the algorithm? How much time and space does it require to solve the problem? Item 1-1: (Analysis of errors) Here we will consider three sources of errors: (i) roundoff errors (ii) errors in the input data (iii) errors caused by the algorithm itself, due to approximations made within the algorithm (truncation errors, discretization errors, · · · ) (i) Roundoff errors: • Fact 1: Algorithms of numerical analysis are destined to be implemented on computers (a computer program or code must be written to communicate the algorithm to the computer. This is, of course, a nontrivial matter; there are so many choices of computers and computer languages.) • Fact 2: Computers have a finite amount of memory (they use a finite number of bits to represent a real number). =⇒ they can represent only a bounded (and in fact finite) subset of the set of real numbers (so there must be gaps between numbers that can be represented by a computer). (Not all numbers can be represented exactly. Which numbers are represented exactly? How are numbers stored in a computer? For now it is enough to know the following :) There are various protocols. For example, by default, MATLAB uses a number system called double precision floating point number system which we denote by F. Math 150 Ali Behzadan 2 F has only finitely many elements. For example, among all the infinitely many numbers in the interval [1, 2], only the following numbers belong to F: 1, 1 + 2−52 , 1 + 2 × 2−52 , 1 + 3 × 2−52 , · · · , 2 The largest element in F (in absolute value) is approximately 10308 . The smallest nonzero element in F (in absolute value) is approximately 10−308 . Roughly speaking, MATLAB rounds off the input data and the results of arithmetic operations in our algorithms to the nearest number that exists in F. Fact 1 and Fact 2 =⇒ there will be roundoff errors (and the errors can clearly accumulate). As you will see, particularly in your numerical linear algebra classes, for many good algorithms the effect of roundoff errors can be studied by using a property which is called backward stability. (ii) Errors in the input data: There may be errors in the input data caused by prior calculations or perhaps measurement errors. (Note: what do we mean by "input data"? For example, in the linear system problem Ax = b, "input data" refers to the entries in A and b.) As you will see in your numerical analysis courses, in order to investigate the effects of this type of error, it is useful to study the following related question: Is the problem well-conditioned? If we slightly change the input, how does this affect the exact solution? For example, consider the problem of solving a linear system Ax = b. • Problem I: Ax = b • Problem II: A + ∆A x = b + ∆b. Here the matrix ∆A represents a change in A; the vector ∆b represents a change in b. Can we establish a relationship between the (exact) solution to (I) and the exact solution to (II)? If ∆A and ∆b are "small", will the solution of (II) be "close" to the solution of (I)? (if the answer is "yes", then the problem is well-conditioned) How "close" is the solution of (II) to the solution of (I)? Remark Generally, algorithms of numerical analysis can be divided into two groups: 1) Algorithms that in principle (i.e. in the absence of roundoff errors and errors in the input data) should produce the exact answer in a finite number of steps. Such algorithms are usually referred to as direct methods. Example: Row reduction algorithm (Gaussian elimination) for solving Ax = b. 2) Algorithms that in principle (even in the absence of rounding errors) produce an approximate answer. Such an algorithm usually gives us a way to construct an infinite sequence u1 , u2 , · · · of approximate solutions that converge to the true solution in some sense. Such algorithms are usually referred to as iterative methods. Each step or iteration of the algorithm produces a new approximate solution. Of course, in practice we cannot run the algorithm forever. We stop once we have produced an approximate solution that is "sufficiently good" (so there will be an error. Math 150 3 Ali Behzadan Item 1-2: (Analysis of the speed of an algorithm) For simple algorithms, specially direct algorithms in numerical linear algebra, it is common to consider the number of arithmetic operations (addition, subtraction, multiplication, division) required by the algorithm/program as the main indicator of the speed of the algorithm. That is, the following simplifying assumptions are made: speed of an algorithm ∼ ∼ computational cost of the algorithm number of arithmetic operations Remark 1) For reasons that hopefully will be discussed in problem sessions, each addition, subtraction, multiplication, division is referred to as a floating point operation or flop. (So it is common to count the number of flops that the algorithm requires as a measure of the speed of the algorithm.) 2) Of course, there is much more to the speed of an algorithm/program than flop count: (i) In flop count, we ignore the evaluation of sine, cosine, etc. (ii) An algorithm may have a smaller flop count but require much more memory. Depending on how much memory is on the system, an algorithm with a larger flop count but less memory use may run faster. (iii) The execution time can also be affected by the movement of data between elements of the memory hierarchy and by competing jobs running on the same processor. (iv) A better written implementation of algorithm A may run faster than algorithm B, even if algorithm A has a larger flop count. For example, algorithm B may not reuse variables already allocated in memory, slowing it down. Example: (dot product) a1 b1 .. .. Let a = . and b = . be two vectors in Rn . Recall that the dot product (inner product) of a and b an is defined as follows: bn a · b = a1 b1 + · · · + an bn 1 2 (for example, if a = and b = , then a · b = (1)(2) + (3)(8) = 26.) 3 8 Let’s write a simple MATLAB function that takes as input two vectors, and outputs their dot product. We will also count the flops. function y= dotprodMath150 (a ,b) y =0; for i =1: length (a) y=y+a(i)*b(i); end Let’s go through the program step-by-step (for the case where length(a) = 2): • The word function on the first line tells MATLAB that you are about to define a function. • function y |{z} output variable = dotprodMath150 ( | {z } name of the function a |{z} , b |{z} ) input variable input variable Math 150 Ali Behzadan 4 • We initialize the variable y to 0. • i is called the loop-variable. • MATLAB enters the i-loop and sets i to 1. • It takes the current value of y (y = 0) and adds a1 b1 to y, then assigns this new value to y (so now y = a1 b1 ). (Warning: y = y + a(i) ∗ b(i) is NOT solving an equation.) • It reaches the end-statement of the loop. • Since it has not completed the loop yet, MATLAB jumps back to the for-statement of the loop and sets i to 2. • It takes the current value of y (y = a1 b1 ) and adds a2 b2 to y, then assigns this new value to y (so now y = a1 b1 + a2 b2 ). • It reaches the end-statement of the loop. • It realizes that it has completely carried out the loop and it stops. Question: If the vectors are in Rn (that is , each vector has n entries), what will be the computational cost (#flops) of our program? f u n c t i o n y=dotprodMath150 ( a , b ) y=0; f o r i = 1 : length ( a ) y=y+a ( i ) * b ( i ) ; (†) end 2 flops each time (†) is executed (†) is executed n times =⇒ 2n flops Note: Doubling the size, doubles the work. size (n) 10 work (2n) 20 20 40 40 80 We say that our algorithm (program) for computing dot product is of order n (our algorithm is O(n)). Definition: An algorithm (a process) is O(na ) ⇐⇒ for large n # flops ≈ some constant times na More precisely, an algorithm is of order na if # flops <∞ n→∞ na 0 < lim Math 150 Ali Behzadan 5 For example, • if an algorithm requires 2n3 − n flops • if an algorithm requires n2 + 40n flops the algorithm is O(n3 ) =⇒ =⇒ the algorithm is O(n2 ) Note that, • for an O(n2 ) algorithm, doubling n roughly quadruples the work size (n) 10 ∼ work for large n (a constant times) 100 20 (a constant times) 400 40 (a constant times) 1600 • for an O(n3 ) algorithm, doubling n roughly multiplies the work by 8 size (n) 10 ∼ work for large n (a constant times) 1000 20 (a constant times) 8000 40 (a constant times) 64000 Item 2: Question: What are the nicest functions that we know? Which functions are easiest to work with? Answer: Polynomials; they are easy to integrate, differentiate, etc. One of our goals is to study the following situation: Suppose we have information about a function f only at a finite set of points. Our goal is to find an approximation for f (x) by another function p(x) from a pre-specified class of functions (usually polynomials) using only information about f at a finite set of points. Specifically, we will study the following topics: (1) Taylor polynomials (2) (Lagrange) interpolation (3) Hermite interpolation (4) Piecewise polynomial interpolation and spline functions In what follows we will begin our discussion with Taylor polynomials. Our plan for the next several lectures is to introduce and study the properties of Taylor polynomials.