Math 150
Ali Behzadan
Lecture 5 Tuesday (2/9/2021), Lecture 6 Thursday (2/11/2021)
1
1) Review, Two fundamental questions
1-1) Analysis of errors
1-2) Analysis of the speed of the algorithm
2) Plan for the next several lectures
Item 1:
As we discussed, numerical analysis involves the design and "study" of algorithms for obtaining (approximate) solutions to various problems of continuous mathematics. In particular, our plan for Math
150 is to hopefully study the following topics:
1. Approximation of functions
2. Numerical differentiation and integration
3. Iterative methods for linear systems (Ax = b)
Iterative methods for nonlinear equations (equations such as ex = x2 )
4. Numerical methods for ordinary differential equations
Also recall that there are two fundamental questions that we seek to answer in the study of algorithms
of numerical analysis:
(1) How accurate is the algorithm? (analysis of errors and stability)
(2) How fast is the algorithm? (analysis of the speed of an algorithm)
More generally, how expensive is the algorithm? How much time and space does it require to solve
the problem?
Item 1-1: (Analysis of errors)
Here we will consider three sources of errors:
(i) roundoff errors
(ii) errors in the input data
(iii) errors caused by the algorithm itself, due to approximations made within the algorithm (truncation
errors, discretization errors, · · · )
(i) Roundoff errors:
• Fact 1: Algorithms of numerical analysis are destined to be implemented on computers (a computer program or code must be written to communicate the algorithm to the computer. This is, of
course, a nontrivial matter; there are so many choices of computers and computer languages.)
• Fact 2: Computers have a finite amount of memory (they use a finite number of bits to represent a
real number).
=⇒ they can represent only a bounded (and in fact finite) subset of the set of real numbers (so
there must be gaps between numbers that can be represented by a computer).
(Not all numbers can be represented exactly. Which numbers are represented exactly? How are
numbers stored in a computer? For now it is enough to know the following :)
There are various protocols. For example, by default, MATLAB uses a number system called
double precision floating point number system which we denote by F.
Math 150
Ali Behzadan
2
F has only finitely many elements. For example, among all the infinitely many numbers in the
interval [1, 2], only the following numbers belong to F:
1, 1 + 2−52 , 1 + 2 × 2−52 , 1 + 3 × 2−52 , · · · , 2
The largest element in F (in absolute value) is approximately 10308 .
The smallest nonzero element in F (in absolute value) is approximately 10−308 .
Roughly speaking, MATLAB rounds off the input data and the results of arithmetic operations in
our algorithms to the nearest number that exists in F.
Fact 1 and Fact 2 =⇒ there will be roundoff errors (and the errors can clearly accumulate).
As you will see, particularly in your numerical linear algebra classes, for many good algorithms
the effect of roundoff errors can be studied by using a property which is called backward stability.
(ii) Errors in the input data:
There may be errors in the input data caused by prior calculations or perhaps measurement errors.
(Note: what do we mean by "input data"? For example, in the linear system problem Ax = b, "input
data" refers to the entries in A and b.)
As you will see in your numerical analysis courses, in order to investigate the effects of this type of
error, it is useful to study the following related question:
Is the problem well-conditioned? If we slightly change the input, how does this affect the exact
solution?
For example, consider the problem of solving a linear system Ax = b.
• Problem I: Ax = b
• Problem II: A + ∆A x = b + ∆b. Here the matrix ∆A represents a change in A; the vector ∆b
represents a change in b.
Can we establish a relationship between the (exact) solution to (I) and the exact solution to (II)?
If ∆A and ∆b are "small", will the solution of (II) be "close" to the solution of (I)? (if the answer is
"yes", then the problem is well-conditioned)
How "close" is the solution of (II) to the solution of (I)?
Remark
Generally, algorithms of numerical analysis can be divided into two groups:
1) Algorithms that in principle (i.e. in the absence of roundoff errors and errors in the input data) should
produce the exact answer in a finite number of steps. Such algorithms are usually referred to as direct
methods.
Example: Row reduction algorithm (Gaussian elimination) for solving Ax = b.
2) Algorithms that in principle (even in the absence of rounding errors) produce an approximate answer.
Such an algorithm usually gives us a way to construct an infinite sequence u1 , u2 , · · · of approximate
solutions that converge to the true solution in some sense. Such algorithms are usually referred to
as iterative methods. Each step or iteration of the algorithm produces a new approximate solution.
Of course, in practice we cannot run the algorithm forever. We stop once we have produced an
approximate solution that is "sufficiently good" (so there will be an error.
Math 150
3
Ali Behzadan
Item 1-2: (Analysis of the speed of an algorithm)
For simple algorithms, specially direct algorithms in numerical linear algebra, it is common to consider
the number of arithmetic operations (addition, subtraction, multiplication, division) required by the algorithm/program as the main indicator of the speed of the algorithm. That is, the following simplifying
assumptions are made:
speed of an algorithm
∼
∼
computational cost of the algorithm
number of arithmetic operations
Remark
1) For reasons that hopefully will be discussed in problem sessions, each addition, subtraction, multiplication, division is referred to as a floating point operation or flop. (So it is common to count the
number of flops that the algorithm requires as a measure of the speed of the algorithm.)
2) Of course, there is much more to the speed of an algorithm/program than flop count:
(i) In flop count, we ignore the evaluation of sine, cosine, etc.
(ii) An algorithm may have a smaller flop count but require much more memory. Depending on
how much memory is on the system, an algorithm with a larger flop count but less memory use
may run faster.
(iii) The execution time can also be affected by the movement of data between elements of the memory hierarchy and by competing jobs running on the same processor.
(iv) A better written implementation of algorithm A may run faster than algorithm B, even if algorithm A has a larger flop count. For example, algorithm B may not reuse variables already
allocated in memory, slowing it down.
Example: (dot product)
 
 
a1
b1
 .. 
 .. 
Let a =  .  and b =  .  be two vectors in Rn . Recall that the dot product (inner product) of a and b
an
is defined as follows:
bn
a · b = a1 b1 + · · · + an bn
1
2
(for example, if a =
and b =
, then a · b = (1)(2) + (3)(8) = 26.)
3
8
Let’s write a simple MATLAB function that takes as input two vectors, and outputs their dot product.
We will also count the flops.
function y= dotprodMath150 (a ,b)
y =0;
for i =1: length (a)
y=y+a(i)*b(i);
end
Let’s go through the program step-by-step (for the case where length(a) = 2):
• The word function on the first line tells MATLAB that you are about to define a function.
• function
y
|{z}
output variable
=
dotprodMath150 (
|
{z
}
name of the function
a
|{z}
,
b
|{z}
)
input variable input variable
Math 150
Ali Behzadan
4
• We initialize the variable y to 0.
• i is called the loop-variable.
• MATLAB enters the i-loop and sets i to 1.
• It takes the current value of y (y = 0) and adds a1 b1 to y, then assigns this new value to y (so now
y = a1 b1 ). (Warning: y = y + a(i) ∗ b(i) is NOT solving an equation.)
• It reaches the end-statement of the loop.
• Since it has not completed the loop yet, MATLAB jumps back to the for-statement of the loop and
sets i to 2.
• It takes the current value of y (y = a1 b1 ) and adds a2 b2 to y, then assigns this new value to y (so now
y = a1 b1 + a2 b2 ).
• It reaches the end-statement of the loop.
• It realizes that it has completely carried out the loop and it stops.
Question:
If the vectors are in Rn (that is , each vector has n entries), what will be the computational cost (#flops) of
our program?
f u n c t i o n y=dotprodMath150 ( a , b )
y=0;
f o r i = 1 : length ( a )
y=y+a ( i ) * b ( i ) ;
(†)
end

2 flops each time (†) is executed

(†) is executed n times
=⇒ 2n flops


Note: Doubling the size, doubles the work.
size (n)
10
work (2n)
20
20
40
40
80
We say that our algorithm (program) for computing dot product is of order n (our algorithm is O(n)).
Definition:
An algorithm (a process) is O(na )
⇐⇒
for large n
# flops ≈ some constant times na
More precisely, an algorithm is of order na if
# flops
<∞
n→∞
na
0 < lim
Math 150
Ali Behzadan
5
For example,
• if an algorithm requires 2n3 − n flops
• if an algorithm requires n2 + 40n flops
the algorithm is O(n3 )
=⇒
=⇒
the algorithm is O(n2 )
Note that,
• for an O(n2 ) algorithm, doubling n roughly quadruples the work
size (n)
10
∼ work for large n
(a constant times) 100
20
(a constant times) 400
40
(a constant times) 1600
• for an O(n3 ) algorithm, doubling n roughly multiplies the work by 8
size (n)
10
∼ work for large n
(a constant times) 1000
20
(a constant times) 8000
40
(a constant times) 64000
Item 2:
Question: What are the nicest functions that we know? Which functions are easiest to work with?
Answer: Polynomials; they are easy to integrate, differentiate, etc.
One of our goals is to study the following situation:
Suppose we have information about a function f only at a finite set of points. Our goal is to find an
approximation for f (x) by another function p(x) from a pre-specified class of functions (usually polynomials) using only information about f at a finite set of points.
Specifically, we will study the following topics:
(1) Taylor polynomials
(2) (Lagrange) interpolation
(3) Hermite interpolation
(4) Piecewise polynomial interpolation and spline functions
In what follows we will begin our discussion with Taylor polynomials. Our plan for the next several
lectures is to introduce and study the properties of Taylor polynomials.