CISE-301: Numerical Methods
Topic 1:
Introduction to Numerical Methods and Taylor Series
Lectures 1-4:
KFUPM
CISE301_Topic1
1
Lecture 1
Introduction to Numerical Methods



What are NUMERICAL METHODS?
Why do we need them?
Topics covered in CISE301.
Reading Assignment: Pages 3-10 of textbook
CISE301_Topic1
2
Numerical Methods
Numerical Methods:
Algorithms that are used to obtain numerical
solutions of a mathematical problem.
Why do we need them?
1. No analytical solution exists,
2. An analytical solution is difficult to obtain
or not practical.
CISE301_Topic1
3
What do we need?
Basic Needs in the Numerical Methods:
 Practical:
Can be computed in a reasonable amount of time.

Accurate:


CISE301_Topic1
Good approximate to the true value,
Information about the approximation error
(Bounds, error order,… ).
4
Outlines of the Course






Taylor Theorem
Number
Representation
Solution of nonlinear
Equations
Interpolation
Numerical
Differentiation
Numerical Integration
CISE301_Topic1




Solution of linear
Equations
Least Squares curve
fitting
Solution of ordinary
differential equations
Solution of Partial
differential equations
5
Solution of Nonlinear Equations

Some simple equations can be solved analytically:
x2  4x  3  0
Analytic solution roots 
4
4 2  4(1)(3)
2(1)
x  1 and x  3

Many other equations have no analytical solution:
x 9  2 x 2  5  0

 No analytic solution
x

xe

CISE301_Topic1
6
Methods for Solving Nonlinear Equations
o
Bisection Method
o
Newton-Raphson Method
o
Secant Method
CISE301_Topic1
7
Solution of Systems of Linear Equations
x1  x2  3
x1  2 x2  5
We can solve it as :
x1  3  x2 ,
3  x2  2 x2  5
 x2  2, x1  3  2  1
What to do if we have
1000 equations in 1000 unknowns.
CISE301_Topic1
8
Cramer’s Rule is Not Practical
Cramer' s Rule can be used to solve the system :
3 1
5 2
 1,
x1 
1 1
1 3
1 5
2
x2 
1 1
1 2
1 2
But Cramer' s Rule is not practical for large problems.
To solve N equations with N unknowns, we need (N  1)(N  1)N!
multiplica tions.
To solve a 30 by 30 system, 2.3 1035 multiplica tions are needed.
A super computer needs more than 10 20 years to compute this.
CISE301_Topic1
9
Methods for Solving Systems of Linear
Equations
o
Naive Gaussian Elimination
o
Gaussian Elimination with Scaled
Partial Pivoting
o
Algorithm for Tri-diagonal
Equations
CISE301_Topic1
10
Curve Fitting


Given a set of data:
x
0
1
2
y
0.5
10.3
21.3
Select a curve that best fits the data. One
choice is to find the curve so that the sum
of the square of the error is minimized.
CISE301_Topic1
11
Interpolation


Given a set of data:
xi
0
1
2
yi
0.5
10.3
15.3
Find a polynomial P(x) whose graph
passes through all tabulated points.
yi  P( xi ) if xi is in the table
CISE301_Topic1
12
Methods for Curve Fitting
o
Least Squares
o
o
o
Linear Regression
Nonlinear Least Squares Problems
Interpolation
o
o
Newton Polynomial Interpolation
Lagrange Interpolation
CISE301_Topic1
13
Integration

Some functions can be integrated
analytically:
3
3
1 2
9 1
1 xdx  2 x 1  2  2  4
But many functions have no analytical solutions :
a
e
 x2
dx  ?
0
CISE301_Topic1
14
Methods for Numerical Integration
o
Upper and Lower Sums
o
Trapezoid Method
o
Romberg Method
o
Gauss Quadrature
CISE301_Topic1
15
Solution of Ordinary Differential Equations
A solution t o the differenti al equation :
x(t )  3 x (t )  3 x(t )  0
x (0)  1; x(0)  0
is a function x(t) that satisfies the equations.
* Analytical solutions are available for
special cases only.
CISE301_Topic1
16
Solution of Partial Differential Equations
Partial Differential Equations are more
difficult to solve than ordinary differential
equations:
 u
2

 u
2
20
x
t
u (0, t )  u (1, t )  0, u ( x,0)  sin( x)
2
CISE301_Topic1
2
17
Summary


Numerical Methods:
Algorithms that are
used to obtain
numerical solution of a
mathematical problem.
We need them when
No analytical solution
exists or it is difficult
to obtain it.
Topics Covered in the Course









CISE301_Topic1
Solution of Nonlinear Equations
Solution of Linear Equations
Curve Fitting
Least Squares
Interpolation
Numerical Integration
Numerical Differentiation
Solution of Ordinary Differential
Equations
Solution of Partial Differential
Equations
18
Lecture 2
Number Representation and Accuracy





Number Representation
Normalized Floating Point Representation
Significant Digits
Accuracy and Precision
Rounding and Chopping
Reading Assignment: Chapter 3
CISE301_Topic1
19
Representing Real Numbers

You are familiar with the decimal system:
312.45  3 10 2  1101  2 100  4 10 1  5 10 2

Decimal System:

Standard Representations:

sign
CISE301_Topic1
Base = 10 , Digits (0,1,…,9)
3 1 2 . 4 5
integral
fraction
part
part
20
Normalized Floating Point Representation

Normalized Floating Point Representation:
 0. d1 d 2 d 3 d 4  10 n
sign
d1  0,
mantissa
exponent
n : integer

No integral part,

Advantage: Efficient in representing very small or very
large numbers.
CISE301_Topic1
21
Calculator Example

Suppose you want to compute:
3.578 * 2.139
using a calculator with two-digit fractions
3.57
*
2.13 = 7.60
True answer:
CISE301_Topic1
7.653342
22
Binary System

Binary System:
Base = 2, Digits {0,1}
 0. 1 b2 b3 b4  2 n
sign
mantissa
exponent
b1  0  b1  1
1
2
3
(0.101) 2  (1 2  0  2  1 2 )10  (0.625)10
CISE301_Topic1
23
7-Bit Representation
(sign: 1 bit, mantissa: 3bits, exponent: 3 bits)
CISE301_Topic1
24
Fact

Numbers that have a finite expansion in one numbering
system may have an infinite expansion in another
numbering system:
(0.1)10  (0.000110011001100...) 2

You can never represent 0.1 exactly in any computer.
CISE301_Topic1
25
Representation
Hypothetical Machine (real computers use ≥ 23 bit mantissa)
Mantissa: 3 bits
Exponent: 2 bits
Sign: 1 bit
Possible positive machine numbers:
.25 .3125 .375 .4375 .5 .625 .75 .875
1
CISE301_Topic1
1.25 1.5
1.75
26
1
Representation
0.8
Gap near zero
0.6
0.4
0.2
0
-0.2
-0.4
-0.6
-0.8
-1
0
0.2
CISE301_Topic1
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
27
Remarks

Numbers that can be exactly represented are called
machine numbers.

Difference between machine numbers is not uniform

Sum of machine numbers is not necessarily a machine
number:
0.25 + .3125 = 0.5625 (not a machine number)
CISE301_Topic1
28
Significant Digits

Significant digits are those digits that can be
used with confidence.
CISE301_Topic1
29
Significant Digits - Example
48.9
CISE301_Topic1
30
Accuracy and Precision

Accuracy is related to the closeness to the true
value.

Precision is related to the closeness to other
estimated values.

The degree of precision in a measurement is
shown by using significant digits.
CISE301_Topic1
31
CISE301_Topic1
32
How Many Significant Digits?



By convention, a measurement is recorded by
writing all exactly known numbers and 1 number
which is uncertain, together with a unit label.
Example:
Blue line is 2.73 cm long. This measurement has 3
significant digits.


First 2 digits (2.7 cm) are exactly known
Third digit (0.03 cm) is uncertain because it was
estimated 1 digit beyond the smallest graduation.
CISE301_Topic1
33
How Many Significant Digits? (Contd.)



Rule: all non-zero digits are significant
Rule: zeros between non-zero digits are significant

Example: 205 m  3 SD
Rule: zeros to the left of the first non-zero digit
are NOT significant

Example: 0.00345 s  3 SD

Example: 2.00345 s  6 SD
CISE301_Topic1
34
How Many Significant Digits? (Contd.)

Rule: if a number has a decimal point, then
trailing zeros are significant

Example: 0.275000 m  6 SD

Example: 267.000 m  6 SD

Rule: if a number has NO decimal point, then
trailing zeros are NOT significant

Example: 275 000 m  3 SD

Example: 275 000. m  6 SD
CISE301_Topic1
35
How Many Significant Digits? (Contd.)

Rule: for addition and subtraction, the answer has
the same precision as the measurement with the
LEAST precision

Example: 375.4 m + 2.54 m = ?  377.9 m

Rule: for multiplication and division, the answer
has the same number of significant figures as the
measurement with the FEWEST significant digits

Example: (20.0 m)(3.0 m) = ?
 60. m or 6.0 x 101 m
CISE301_Topic1
36
SDs – Addition and Subtraction
• When adding or subtracting do NOT extend the result beyond
the first column with a doubtful figure. For example, …
SDs – Addition and Subtraction
• What is 16.874 + 2.6?
• What is 16.874 - 2.6?
SDs – Multiplication and Division
• When multiplying or dividing the answer will have the same
number of significant digits as the least accurate number used
to get the answer. For example, …
2.005 g / 4.95 mL = 0.405 g/mL
• What is 16.874 x 2.6?
• What is 16.874 / 2.6?
SDs and Calculations that Require Multiple Steps
• An average is the best estimate of the true value of a
parameter.
• A standard deviation is a measure of precision.
• Averages and standard deviations require several steps to
calculate. You must keep track of the number of significant
figures during each step. Do NOT discard or round any figures
until the final number is reported.
SDs and Calculations that Require Multiple Steps
2 Significant
Figures
∞ Significant Figures
1 Significant Figure
1 Significant Figure
2 Significant Figures
1 Significant Figure
0 Significant Figures
• What is average and standard deviation for the following 3
measurements of the same sample?
Rounding and Chopping

Rounding: Replace the number by the nearest
machine number.

Chopping: Throw all extra digits.
CISE301_Topic1
43
Rounding and Chopping - Example
CISE301_Topic1
44
Error Definitions – True Error
Can be computed if the true value is known:
Absolute True Error
Et  true value  approximat ion
Absolute Percent Relative Error
true value  approximat ion
t 
*100
true value
CISE301_Topic1
45
Error Definitions – Estimated Error
When the true value is not known:
Estimated Absolute Error
Ea  current estimate  previous estimate
Estimated Absolute Percent Relative Error
current estimate  previous estimate
a 
*100
current estimate
CISE301_Topic1
46
Notation
We say that the estimate is correct to n
decimal digits if:
Error  10
n
We say that the estimate is correct to n
decimal digits rounded if:
1
n
Error   10
2
CISE301_Topic1
47
Summary

Number Representation
Numbers that have a finite expansion in one numbering system
may have an infinite expansion in another numbering system.

Normalized Floating Point Representation



Efficient in representing very small or very large numbers,
Difference between machine numbers is not uniform,
Representation error depends on the number of bits used in
the mantissa.
CISE301_Topic1
48
Lectures 3-4
Taylor Theorem



Motivation
Taylor Theorem
Examples
Reading assignment: Chapter 4
CISE301_Topic1
49
Motivation

We can easily compute expressions like:
3 10 2
2( x  4)
But, How do you compute
4.1, sin( 0.6) ?
We can use the graphical definition to compute
b
sin(0.6) !? is this a practical way?
a
0.6

Taylor series is used to express function in an
approximate fashion
CISE301_Topic1
50
Taylor Series

Taylor series predicts func. value at one point x in terms
of the func. value & its derivatives at another point x0.
The Taylor series expansion of f ( x) about x0 :
( 2)
( 3)
f
(
x
)
f
( x0 )
0
f ( x0 )  f ' ( x0 ) ( x  x0 ) 
( x  x0 ) 2 
( x  x0 ) 3  ...
2!
3!
or
1st-order approx.

1 (k )
f ( x0 ) ( x  x0 ) k
k  0 k!
If the series converge, we can write :
Taylor Series  
zero-order approx.
∞
1
f ( x)  ∑ f ( k ) ( x0 ) ( x  x0 ) k
k  0 k!
CISE301_Topic1
51
Taylor Series – Example 1
Obtain Taylor series expansion of f ( x)  e x about x0  0:
f ( x)  e x
f (0)  1
f ' ( x)  e x
f ' (0)  1
f ( 2 ) ( x)  e x
f ( 2 ) (0)  1
f ( k ) ( x)  e x
f ( k ) (0)  1 for k  1
∞
∞
k
1
x
e x  ∑ f ( k ) ( x0 ) ( x  x0 ) k  ∑
k  0 k!
k  0 k!
The series converges for x  ∞.
CISE301_Topic1
52
Taylor Series
3
Example 1
2.5
exp(x)
1+x+0.5x 2
2
1+x
1.5
1
1
0.5
0
-1
CISE301_Topic1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
53
Taylor Series – Example 2
Obtain Taylor series expansion of f ( x)  sin( x) about x0  0:
f ( x)  sin( x)
f ' ( x)  cos( x)
f (0)  0
f ' (0)  1
f ( 2 ) ( x)   sin( x)
f ( 2) (0)  0
f ( 3) ( x)   cos( x)
f ( 3) (0)  1
∞
3
5
7
f ( k ) ( x0 )
x
x
x
sin( x)  ∑
( x  x0 ) k  x     ....
k!
3! 5! 7!
k 0
The series converges for x  ∞.
CISE301_Topic1
54
4
3
x
2
1
x-x 3/3!+x 5/5!
0
sin(x)
-1
x-x 3/3!
-2
-3
-4
-4
CISE301_Topic1
-3
-2
-1
0
1
2
3
4
55
Convergence of Taylor Series
(Observations, Example 1)

The Taylor series converges fast (few terms
are needed) when x is near the point of
expansion. If |x-c| is large then more terms
are needed to get a good approximation.
CISE301_Topic1
56
Taylor Series – Example 3
1
Obtain Taylor series expansion of f(x) 
about x0  0 :
1 x
1
f ( x) 
f (0)  1
1 x
1
f '( x) 
f '(0)  1
2
1  x 
f (2) ( x) 
f
(3)
( x) 
2
3
f (2) (0)  2
4
f (3) (0)  6
1  x 
6
1  x 
Taylor Series Expansion of:
CISE301_Topic1
1
 1  x  x 2  x 3  ....
1 x
57
Example 3 – Remarks

Can we apply Taylor series for x>1??

How many terms are needed to get a good
approximation???
These questions will be answered using
Taylor’s Theorem.
CISE301_Topic1
58
Taylor’s Theorem
If a function f(x) possesses continuous derivative s
of orders 1, 2, ..., (n  1) in a closed interval [a, b],
then for any c ∈[a, b] :
n
f ( k ) (c )
( x  c) k
f ( x)  ∑
k!
k 0
where :
(n+1) terms Truncated
Taylor Series
 En 1
Remainder
( or approx. error)
f ( n 1) ( )
( x  c) n 1 and  is between c and x.
En 1 
(n  1)!
CISE301_Topic1
59
Taylor’s Theorem
We can apply Taylor' s theorem for :
1
f(x) 
with the point of expansion c  0 if | x |  1.
1 x
If [a, b] includes x  1, then the function and its
derivative s are not defined.
 Taylor Theorem is not applicable .
CISE301_Topic1
60
Error Term
To get an idea about the approximat ion error,
we can derive an upper bound on :
f ( n 1) ( )
n 1
En 1 
( x  c)
(n  1)!
for all values of  between c and x.
CISE301_Topic1
61
Error Term – Example 4
How large is the error if we replaced f ( x)  e x by
the first 4 terms (n  3) of its Taylor series expansion
about x0  0 when x  0.2?
f ( k ) ( x)  e x
f ( k ) ( )  e0.2 for k  1
f ( n 1) ( )
En 1 
( x  c) n 1
(n  1)!
En 1
CISE301_Topic1
Remember that   [c  0, x  0.2]
 f ( k ) ( )  e0.2
e0.2
n 1

 0.2   E4  8.14268 105
(n  1)!
62
Alternative form of Taylor’s Theorem
Let f ( x) have continuous derivatives of
orders 1, 2, ..., (n  1) on an interval [a,b],
and x  [a,b] and x  h  [a,b], then:
n
f ( x  h)  
k 0
f ( k ) ( x) k
h  En 1
k!
f ( n 1) ( ) n 1
En 1 
h where  is between x and x  h
(n  1)!
CISE301_Topic1
63
Taylor’s Theorem – Alternative forms
( n 1)
f ( k ) (c )
f
( )
k
f ( x)  
( x  c) 
( x  c) n 1
k!
(n  1)!
k 0
where  is between c and x.
n
x  x  h, c  x
f ( k ) ( x) k f ( n 1) ( ) n 1
f ( x  h)  
h 
h
k!
(n  1)!
k 0
where  is between x and x  h.
n
CISE301_Topic1
64
Mean Value Theorem
If f (x) is a continuous function on a closed interval [a,b]
and its derivative is defined on the open interval (a,b)
then there exists ξ  [ a, b]
df(ξ ) f(b)  f(a)

dx
(b - a)
Proof : Use Taylor's Theorem for n  0, x  a, x  h  b
df(ξ )
f(b)  f(a) 
(b - a)
dx
CISE301_Topic1
65
Alternating Series Theorem
Consider the alternating series:
S  a1  a2  a3  a4 
 a a a a 
2
3
4
 1
If  and
 lim a  0
 n n
The series converges

then 
and

S  S n  an 1

S n : Partial sum (sum of the first n terms)
an 1 : First omitted term
CISE301_Topic1
66
Alternating Series – Example 5
1 1 1
sin(1) can be computed using : sin (1)  1     
3! 5! 7!
This is a convergent alternatin g series since :
a1  a2  a3  a4   and lim an  0
n 
Then :
 1 1
sin (1)  1   
 3!  5!
 1 1 1
sin (1)  1    
 3! 5!  7!
CISE301_Topic1
67
Example 6
1. Obtain the Taylor series expansion of
f (x)  e 2x 1 about c  0.5 (the center of expansion)
2. How large can the error be when (n  1) terms are used
to approximate e 2x 1 with x  1 ?
CISE301_Topic1
68
Example 6 – Taylor Series
1. Obtain Taylor series expansion of f ( x)  e 2 x 1 , c  0.5
f ( x)  e 2 x 1
f (0.5)  e 2
f '( x)  2e 2 x 1
f '(0.5)  2e 2
f (2) ( x)  4e 2 x 1
f (2) (0.5)  4e 2
f ( k ) ( x)  2k e 2 x 1
f ( k ) (0.5)  2k e 2
e
2 x 1


k 0
f ( k ) (0.5)
( x  0.5) k
k!
2
k
(
x

0.5)
(
x

0.5)
 e 2  2e 2 ( x  0.5)  4e 2
 ...  2k e 2
 ...
2!
k!
CISE301_Topic1
69
Example 6 – Error Term
2. f ( k ) ( x)  2k e 2 x 1
f ( n 1) ( )
( x  0.5) n 1
Error 
(n  1)!
Error  2n 1 e 2 1
Error  2
n 1
(1  0.5) n 1
(n  1)!
(1 2) n 1
max e 2 1
(n  1)!  [0.5,1]
e3
Error 
(n  1)!
CISE301_Topic1
70
Remark

In this course, all angles are assumed to
be in radian unless you are told otherwise.
CISE301_Topic1
71
Maclaurin Series

Find Maclaurin series expansion of cos (x).

Maclaurin series is a special case of Taylor
series with the center of expansion c = 0.
CISE301_Topic1
72
Maclaurin Series – Example 7
Obtain Maclaurin series expansion of : f ( x)  cos( x)
f ( x)  cos( x)
f ' ( x)   sin( x)
f ( 0)  1
f ' ( 0)  0
f ( 2 ) ( x)   cos( x)
f ( 2 ) ( 0 )  1
f (3) ( x)  sin( x)
f ( 3 ) ( 0)  0
∞
2
4
6
f ( k ) ( 0)
x
x
x
cos( x)  ∑
( x) k  1     ....
k!
2! 4! 6!
k 0
The series converges for x  ∞.
CISE301_Topic1
73