Engineering
Computation
Lecture 2
E. T. S. I. Caminos, Canales y Puertos
1
Round-off Error due to Arithmetic Operations
Smearing
Occurs when individual terms are larger than summation
itself. Consider the exponential series with x = -10
x 2 x3 x 4 x5




Consider formulas such as: e  1  x 
2! 3! 4! 5!
x
With 7-decimal-digit accuracy:
exact answer
= 4.54 10-05
computed answer = – 6.26 10-05
(45 terms)
wrong sign !
Largest intermediate terms are:
9th = –2,755.732 & 10th = 2,755.732
E. T. S. I. Caminos, Canales y Puertos
2
Truncation Error
Error caused by the nature of the numerical technique
employed to approximate the solution.
Example:
Maclaurin series expansion of ex
2
3
4
5
x
x
x
x
ex  1  x 




2! 3! 4! 5!
2
x
x
e
 1 x 
If we use a truncated version of the series:
2!
Then the Truncation Error is:
E. T. S. I. Caminos, Canales y Puertos
x3 x 4 x5



3! 4! 5!
3
Approximations and Rounding Errors
• Precautions:
– Sums of large and small numbers: due to equaling the
exponent. They are common in sums of infinite series
where the individual terms are very small when
compared with the accumulated sum. This error can be
reduced by summing first the small terms and using
double precision.
– Cancellation of the subtraction: The subtraction of very
similar numbers.
– Smearing: The individual terms are larger than the total
sum.
– Inner products: They are prone to rounding errors. Thus,
it is convenient to use double precision in this type of
calculations.
n
 xi y i  x1y1  x2y 2    xn yn
i 1
E. T. S. I. Caminos, Canales y Puertos
4
Error Propagation
Errors which appear because we are basing current
calculations on previous calculations which also incurred
some form of error
Stability and Condition Number
Numerically Unstable: Computations which are so
sensitive to round-off errors that errors grow
uncontrollably during calculations.
Condition: sensitivity to such uncertainty; "well
conditioned" vs. "ill conditioned"
Condition Number: measure of the condition; i.e.,
extent to which uncertainty in x is amplified by ƒ(x)
C.N.  1
===> "well-conditioned"
C.N. >> 1 ===> "ill-conditioned"
E. T. S. I. Caminos, Canales y Puertos
5
Taylor Series Expansion
Basic Idea:
Predict the value of a function, ƒ, at a point xi+1
based on the value of the function and all of its
derivatives, ƒ, ƒ', ƒ",… at a neighboring point xi
Given xi, ƒ(xi), ƒ'(xi), ƒ"(xi), ... ƒn+1(xi),
we can predict or approximate ƒ(xi+1)
E. T. S. I. Caminos, Canales y Puertos
6
Taylor Series Expansion
General Form:
h2
h3
f (x i 1 )  f (x i ) hf (x i )  f (x i )  f (x i ) 
2!
3!
hn n

f (x i )  R n
n!
h = "step size" = xi+1 – xi
Rn = remainder to account for all other terms
h n 1 n 1

f ( )
(n  1)!
with xi    xi+1
= O (hn+1) with x not exactly known "on the order of hn+1
"
Note: f(x) must be a function with n+1 continuous derivatives
E. T. S. I. Caminos, Canales y Puertos
7
Taylor Series Expansion
h2
f (x i1 )  f (x i )  h f (x i )  f (x i ) 
2!
hn n
 f (x i )  O(h n 1)
n!
 0th order T.S. approx. (n = 0):
f(xi+1) = f(xi) + O (h1)
 1st order T.S. approx. (n = 1):
f(xi+1) = f(xi) + hf '(xi) + O (h2)
2
h
 2nd order T.S. approx. (n = 2):f (x i 1 )  f (x i )  h f (x i )  f (x i )  O (h n 1 )
2!
 nth order T.S. approximation will be exact for an nth order
polynomial
E. T. S. I. Caminos, Canales y Puertos
8
Taylor Series Expansion
f(x )
f(xi )
Zero order
f(xi+1 ) f(xi )
f(xi+1 ) f(xi )+f '(xi )h
f(xi+1 ) f(xi )+f '(xi )h+ )+f "(xi )h2/2!
f(xi+1 )
xi
xi+1
x
h
E. T. S. I. Caminos, Canales y Puertos
9
Numerical Differentiation from Taylor Series Expansion
Objective:
Evaluate the derivatives of function, ƒ(xi), without
doing it analytically.
When would we want to do this?
1. function is too complicated to differentiate
analytically:
2  cos(1  x ) 0.5x
e
1  0.5x
2. function is not defined by an equation,
i.e., given a set of data points (xi, ƒ(xi)), i=1,…,n
i
0
1
2
3
4
xi
1.0
3.0
5.0
7.0
9.0
ƒ(xi) 2.3
4.1
5.5
5.7
5.9
E. T. S. I. Caminos, Canales y Puertos
10
Numerical Differentiation from Taylor Series Expansion
– First derivative with backward difference.
f ( xi 1 )  f ( xi )  f ' ( xi )( xi  xi 1 )
f ' ( xi ) 
f ( xi )  f ( xi 1 )
( xi  xi 1 )
E. T. S. I. Caminos, Canales y Puertos
11
Numerical Differentiation from Taylor Series Expansion
Backward Difference Approx.:
First Derivative:
h2
f (x i1 )  f (x i )  (x i1  x i )f '(x i )  f "()
2
Letting h = xi - xi-1
h2
f (xi1 )  f (xi )  hf '(x i )  f "()
2
h2
hf '(xi )  f (x i )  f (x i1)  f "()
2
f (x i )  f (x i 1 )
f '(x) 
 O(h)
h
first backward difference
E. T. S. I. Caminos, Canales y Puertos
12
Example of 1st Backward FDD
Using data below calculate ƒ'(x1) :
i
0
1
2
3
xi
1.0 3.0 5.0 7.0
ƒ(xi) 2.3 4.1 5.5 5.7
4
9.0
5.9
First Backward Finite-Divided-Difference at x1:
f (x1 )  f (x 0 )
f '(x) 
 O(h)
h
f '(x1 ) 
f ' (x1) 
4.1  2.3
 O(h)
2
0.9
E. T. S. I. Caminos, Canales y Puertos
{ + O (h) }
13
Backward Difference Approximation
Second Derivative:
f (x i2 )  f (x i )   x i2  x i  f '(x) 
 x i 2  x i 
2
f "(x i )
2!
+ O([xi-2– xi]3)
with
h = xi– xi-1
and
2h = xi – xi-2
The 2nd order approximation to ƒ(xi-2) becomes:
ƒ(xi-2) = ƒ(xi) – 2hƒ'(xi) + 2h2 ƒ"(xi) +O (h3)
[1]
2nd order approximation to ƒ(xi-1):
h2
f (x i1 )  f (x i )  hf '(x) 
f "(x i )  O(h 3 )
2!
E. T. S. I. Caminos, Canales y Puertos
[2]
14
Backward Difference Approximation
Subtracting 2*[2] from [1] yields:
f(xi-2) – 2f(xi-1) = –f(xi) + h2f"(xi) + O (h3)
Rearranging:
h2ƒ"(xi) = f(xi) – 2f(xi-1) + f(xi-2) + O (h3)
f "(x i ) 
f "(x i ) 
f (x i )  2f (x i1 )  f (x i2 )  O(h 3 )
h2
f (x i )  2f (x i1 )  f (x i2 )
h
2
 O(h)
Second backward difference
E. T. S. I. Caminos, Canales y Puertos
15
Example of 2nd Backward FDD
Using data below calculate ƒ"(x2) :
i
0
1
2
3
xi
1.0 3.0 5.0 7.0
ƒ(xi) 2.3 4.1 5.5 5.7
4
9.0
5.9
Second Backward Finite-Divided-Difference at x2:
f "(x 2 ) 
f "(x 2 ) 
f (x 2 )  2f (x1 )  f (x 0 )
h
2
5.5  2*4.1  2.3
f " (5.0)  - 0.1
E. T. S. I. Caminos, Canales y Puertos
2
2
 O(h)
 O(h)
{ + O (h) }
16
Other Forms of Numerical Differentiation
What points are used for each form?
Backward:
…, ƒ(xi-2), ƒ(xi-1), ƒ(xi), ƒ(xi+1), ƒ(xi+2), …
Forward:
…, ƒ(xi-2), ƒ(xi-1), ƒ(xi), ƒ(xi+1), ƒ(xi+2), …
Centered:
…, ƒ(xi-2), ƒ(xi-1), ƒ(xi), ƒ(xi+1), ƒ(xi+2), …
E. T. S. I. Caminos, Canales y Puertos
17
Taylor Series and Truncation errors
- Higher order divided differences.
f ' ' ( xi ) 2
h 
2!
f '' ( xi ) 2
f ( xi 1 )  f ( xi )  f ' ( xi )h 
h 
2!
f ( xi 1 )  f ( xi )  f ' ( xi )h 
- Second finite central divided difference
E. T. S. I. Caminos, Canales y Puertos
18
Other Forms of Numerical Differentiation
Forward:
 (x i1 )   (x i )
f '(x i ) 
 O(h)
h
 (x i2 )  2 (x i1 )   (x i )
f "(x i ) 
 O(h)
2
h
- f ( xi  2 )  4 f ( xi 1 ) -3 f ( xi )
f '(x i ) =
+ O(h 2 )
2h
Centered:
 (x i1 )   (x i1 )
f '(x i ) 
 O(h 2 )
2h
f "(x i ) 
 (x i1 )  2 (x i )   (x i1 )
E. T. S. I. Caminos, Canales y Puertos
h2
 O(h 2 )
19
Taylor Series and Truncation errors
• Use of the Taylor series to calculate derivatives.
– First derivative with forward difference.
f ( xi 1 )  f ( xi )  f ' ( xi )( xi 1  xi )
f ' ( xi ) 
f ( x i 1 )  f ( x i )
( x i 1  x i )
E. T. S. I. Caminos, Canales y Puertos
20
Taylor Series and Truncation errors
– First derivative with central differences.
f ' ' ( xi ) 2
h 
2!
f '' ( xi ) 2
f ( xi 1 )  f ( xi )  f ' ( xi )h 
h 
2!
f ( xi 1 )  f ( xi )  f ' ( xi )h 
f ' ( xi ) 
f ( 3) ( xi ) 3
f ( xi 1 )  f ( xi 1 )  2f ' ( xi )h 
h 
3!
f ( xi 1 )  f ( xi 1 )
2h
E. T. S. I. Caminos, Canales y Puertos
21
Taylor Series and Truncation errors
Questions:
• Which is a better approximation?
Forward, Centered, or Backward?
• Why?
• When would you use which?
Note:
We also can get higher order forward, centered, and
backward difference derivative approximations
[C&C Chapter 23, tabulated in Figs. 23.1-3]
E. T. S. I. Caminos, Canales y Puertos
22
Example Combining Roundoff and Truncation Error
Determine h to minimize the total error of a forward finitedivided difference approximation for:
f (x i1 )  f (x i )
f '(x) 
h
• Truncation Error:
f (x i1 )  f (x i ) h
f '(x) 
 f "()
h
2
• Round-off Error:
f  x   fˆ  x    f  x 
xi    xi+1
with  = machine epsilon.
.
.

(x

h)
(1


)


(x
)
i
i (1  ) h
ˆ
  "()
As a result: f ' =
h
2
Roundoff   (x i  h)    (x i )   2  (x i )
Error
h
h
E. T. S. I. Caminos, Canales y Puertos
23
Example Combining Roundoff and Truncation Error
 Total error  = truncation error + roundoff error 
h
f "() +
E = | Total Error | 
2
2 f (x i )
h
NOTE: Truncation error decreases as h decreases
Round-off error increases as h decreases
E. T. S. I. Caminos, Canales y Puertos
24
Example Combining Roundoff and Truncation Error
E. T. S. I. Caminos, Canales y Puertos
25
Example Combining Roundoff and Truncation Error
To minimize total error E with respect to h, set the first
derivative to zero:
dE f "() 2 f (x i )


0
2
dh
2
h
Solve for h and approximate f "() as f
"(xi):
4 f (x i )
h
f "(x i )
E. T. S. I. Caminos, Canales y Puertos
26
Example Combining Roundoff and Truncation Error
Linear Application:
Determine h that will minimize total error for calculating f’(x)
for
f(x) =  x
at
x=1
Using the first forward-divided-difference approximation with
error O(h) and a 5-decimal-digit machine:
e = b1-t = 101-5 = 10-4 = 0.0001
f '(x) = ;
h
f "(x) = 0
4 f (x i )
E. T. S. I. Caminos, Canales y Puertos
f "(x i )
 infinity
27
Example Combining Roundoff and Truncation Error
h
0
0.000001
0.00001
0.0001
0.001
0.01
0.1
1
f(x+h)=
 (x+h)
3.1415
3.1415
3.1416
3.1419
3.1447
3.1730
3. 4557
6.2831
f(x+h)-f(x)
{exact: 3.1415}
[f(x+h)-f(x)]/h
0
0.0001
0.0004
0.0032
0.0315
0.3142
3.1416
0
10
4.0
3.2
3.15
3.142
3.146
Underlined digits are subject to round-off error. They
are likely to be in error by ± one or two units. This does
not cause much problem when h = 1, but causes large
errors in the final result when h < 10-4.
E. T. S. I. Caminos, Canales y Puertos
28
Example Combining Roundoff and Truncation Error
Nonlinear Application:
Determine h for minimizing the total error for computing f’(x) for
ƒ(x) = ex
at
x=3
Using the first forward-divided-difference approximation with
error O(h) and a 5-decimal-digit machine:
e = b1-t = 101-5 = 10-4 = 0.0001
f(x) = f '(x) = f "(x) = ex = 20.0855;
h
4  (x i )
 "(x i )
E. T. S. I. Caminos, Canales y Puertos
 0.02
or about 0.01
29
Example Combining Roundoff and Truncation Error
h
f(x+h)=ex+h
0
0.00001
0.0001
0.001
0.01
0.1
1
20.085
20.085
20.087
20.105
20.287
22.198
54.598
full precision
f(x+h)-f(x) [f(x+h)-f(x)] [f(x+h)-f(x)]
h
h
{exact = 20.085}
0.0
0
20.086
0.002
20
20.086
0.020
20
20.096
0.202
20.2
20.18
2.113
21.13
21.12
34.513
34.523
34.512
Roundoff
Truncation
Underlined digits subject to roundoff error.
Bold digits in error due to truncation.
E. T. S. I. Caminos, Canales y Puertos
30