1
Numerical
Computation
Prepared by:
Richard Mitchell
Humber College
CASE STUDY
1.1-THE REAL NUMBERS
1.1-PLACE VALUE-Page 3
Whole Numbers
213 - Nearest Whole Number
51 700 - Nearest Hundreds
710 010 - Nearest Tens
52 - Nearest Whole Number
517 000 - Nearest Thousands
Decimal Numbers
0.105 - 3 DP’s/Nearest Thousandths
0.21 - 2 DP’s/Nearest Hundredths
11.051 - 3 DP’s/Nearest Thousandths
20.005 - 3 DP’s/Nearest Thousandths
7.00 - 2 DP’s/Nearest Hundredths
1.1-EXACT NUMBERS-Page 4
Counted Quantities have no uncertainty.
4 wheels (exactly counted)
501 roses (exactly counted)
17 letters (exactly counted)
2 010 washers (exactly counted)
81 700 cars (exactly counted) 24 hours in a day (exactly counted)
Whole Numbers and Fractions (i.e., not measured and not in Decimal Form)
have no uncertainty.
0
1
2
1
2
2
3
3
10
516
21 019
3/ 4
Defined Numbers have no uncertainty.
1 inch = 25.4 mm (exactly measured by definition)
20 000
1.1-APPROXIMATE NUMBERS-Page 4
Measured Quantities have some degree of uncertainty.
(Last SD and/or Decimal Place is often estimated visually on a scale or meter).
217 cm
1.75 inches
13.0 m
1 000 m/s
50 112 Hz
2.02 lbs
0.00051 km
0.0020 mph
Decimal Numbers (Measured and Non-Measured) have some degree of
uncertainty.
0.217
2 157.0 cm
3.00
12.05 ft
Decimal Form of Fractions and Irrational Numbers
2/3 (exact form) equals 0.6667 (approximate decimal form)
∏ (exact form) equals 3.1428571 (approximate decimal form)
3 (exact form) equals 1.732 (approximate decimal form)
Estimates
approximately 3 500 people
about 2 710 cars were built
1.1-SIGNIFICANT DIGITS-Page 5
Whole Numbers
- 3 SD’s
210 015 - 6 SD’s
1 000 201 - 7 SD’s
24 900 - 3 SD’s
310 400 - 4 SD’s
217
274 Ō00
31 Ō00
- 4 SD’s
- 3 SD’s
- 4 SD’s
5 101 110 - 6 SD’s
8 003
Decimal Numbers
0.152 - 3 SD’s
0.0005
- 1 SD
11.25 - 4 SD’s
2.0500
- 5 SD’s
- 3 SD’s
0.000583 - 3 SD’s
2.008 - 4 SD’s
15.05 - 4 SD’s
0.0177
1.1-ACCURACY vs PRECISION-Page 5
Accuracy is Determined by Significant Digits
1.255
-
4 SD’s of Accuracy
- 3 SD’s of Accuracy
0.0050 - 2 SD’s of Accuracy
125 - 3 SD’s of Accuracy
310.03 - 5 SD’s of Accuracy
23 800
Precision is Determined by Place Value
1.255 - 3 DP’s of Precision
(Nearest Thousandths)
- Nearest Hundred’s of Precision
0.0050 - 4 DP’s of Precision (Nearest Ten Thousandths)
125 - Nearest Whole Number of Precision
310.03 - 2 DP’s of Precision (Nearest Hundredths)
23 800
1.2-ADDITION AND SUBTRACTION
1.2-EXAMPLE 19-Page 10
A d d th e f o llo w in g a p p r o x im a te n u m b e r s .
3 2 .4 c m + 5 .8 2 5 c m
1 Decimal Place
(Approximate)
3 Decimal Places
(Approximate)
= 3 8 .2 2 5 c m
= 3 8 .2 c m
USE: The Rule of PRECISION (1 Decimal Place)
A N S : 3 8 .2 c m
(1 D e c im a l P la c e )
1.2-EXAMPLE 21-Page 10
A certain stadium contains (about ) 3500 people. It starts to rain,
and 372 people (exactly) leave. How many are left in the stadium ?
(Subtract the follow ing approximate and exact numbers).
3500 - 372
Nearest Hundreds
(Approximate)
Nearest Ones
(Exact)
= 3 1 2 8 p e o p le
= 3 1 0 0 p e o p le
A N S : 3 1 0 0 p e o p le
USE: The Rule of PRECISION (Nearest Hundreds)
(N e a re st H u n d re d s)
1.3-MULTIPLICATION
1.3-EXAMPLE 31-Page 15
M u ltip ly th e f o llo w in g a p p r o x im a te n u m b e r s .
1 2 3 .5 6 x 2 .2 1
5 Significant Digits
(Approximate)
3 Significant Digits
(Approximate)
= 2 7 3 .0 6 7 6
= 273
A N S: 273
USE: The Rule of ACCURACY (3 Significant Digits)
(3 S ig n ific a n t D ig its )
1.3-EXAMPLE 32-Page 15
If a certain tire weighs 32.2 kg (approximately) when m ounted,
how much will four (exactly counte d ) such tires weigh ?
(M ultiply the following approximate and exact numbers).
3 2 .2 k g x 4
3 Significant Digits
(Approximate)
Do not count these as
Significant Digits.
= 1 2 8 .8 k g
= 129 kg
USE: The Rule of ACCURACY (3 Significant Digits)
A N S: 129 kg
(3 S ig n ific a n t D ig its )
1.4-DIVISION
1.4-EXAMPLE 35-Page 17
D iv id e th e f o llo w in g a p p r o x im a te n u m b e r s .
8 4 6 .2  4 .7 5
4 Significant Digits
(Approximate)
3 Significant Digits
(Approximate)
= 1 7 8 .1 4 7 3 6 8 4
= 178
A N S: 178
USE: The Rule of ACCURACY (3 Significant Digits)
(3 S ig n ific a n t D ig its )
1.4-EXAMPLE 36-Page 17
D ivid e 8 4 6 .2 in to th ree eq u a l p arts.
(D ivid e th e fo llo w in g ap p ro x im ate an d ex act n u m b ers) .
8 4 6 .2  3
4 Significant Digits
(Approximate)
Do not count these as
Significant Digits.
= 2 8 2 .0 6 6 6 6 6 6
= 2 8 2 .1
USE: The Rule of ACCURACY (4 Significant Digits)
A N S : 2 8 2 .1
(4 S ig n ific a n t D ig its )
1.5-POWERS AND ROOTS
1.5-EXAMPLES-Pages 19 to 21
Powers
(1.45)5 ( approx .number )
 6.409 734 063
 6.41 ( 3 SD ' s )
(3.85) 3 ( approx .number )
 0.017 523 377
 0.0175 ( 3 SD ' s )
80.67 ( approx .number )
 4.027 822 200
 4.0 ( 2 SD ' s )
2
83
Roots
3
8 ( exact number )
2
Because 23  8
4
81 ( exact number )
3
Because 34  81
8 ( exact number )
 2
Because (-2)3  8
3
5
( exact number )
 4 ( exact answer )
28.4 ( approx . number )
 1.952 826 537
 1.95 ( 3 SD ' s )
1.6-COMBINED OPERATIONS
1.6-EXAMPLE extra
S olve the follow ing form ula ( exact num bers ) and round off your
answ er according to the rules of A ccuracy and/or P recision.
 118  423 


136 

3
541




 11.66190379 
3
  46.39036728 
3
 99 835.14033 (exactly)
A n s : 9 9 8 3 5 .1 4 0 3 3
( e x a c t ly )
1.6-EXAMPLE 57-Page 23
S olve the follow ing form ula ( approxim ate num ber s) and round o ff
your answ er according to the rules of A ccuracy and/or P recision.
2 3
 118.8 cm  4.23 cm


2
136 cm


2
 (10.54973546 cm)3
 1 174.153 047 cm3
2 3
 123.03 cm


2
 136 cm 
 1 170 cm3
(3 Significant Digits)
USE: The Rule of ACCURACY (3 Significant Digits)
 123.03 cm



11.66190379
cm


2
3
A ns: 1 170 cm 3
( 3 S .D .'s )
1.7-SCIENTIFIC AND
ENGINEERING NOTATION
1.7-EXAMPLES-Pages 25 to 28
Large Numbers
346 = 3.46 x 102 (3 SD’s)
2 700 = 2.7 x 103 (2 SD’s)
5 101 000 = 5.101 x 106 (4 SD’s)
31Ō 000 = 3.10 x 105 (3 SD’s)
Small Numbers
0.0000931 = 9.31 x 10-5
(3 SD’s)
0.008300 = 8.300 x 10-3
(4 SD’s)
0.00000950 = 9.50 x 10-6
(3 SD’s)
1.7-CALCULATOR SKILLS-Pages 25 to 28
Normal Mode
(30 000) x (215 000) = 6 450 000 000
(30 000) + (215 000) = 245 000
Exponential Mode (EXP or EE key)
(3.00 x 104) x (2.15 x 105) = 6 450 000 000
(3.00 x 104) + (2.15 x 105) = 245 000
Scientific Mode (Mode or FSE key)
(3.00 x 104) x (2.15 x 105) = 6.45 x 109
(3.00 x 104) + (2.15 x 105) = 2.45 x 105
USE: The Rule of ACCURACY when using
approximate numbers (Significant Digits) .
1.7-EXAMPLE 69 and 70-Page 27
C o n v e r t th e f o llo w in g n u m b e r s in to E n g in e e r in g N o ta tio n .
2 1 8 4 0  2 1 .8 4 0 x 1 0 3
RULE: Digits from the decimal part of the
number are grouped into sets of three.
5 4 8 0 0 0  5 4 8 .0 0 0 x 1 0 3
7 2 5 6 0 0 0 0  7 2 .5 6 0 0 0 0 x 1 0 6
0 .8 7 2 1 7 = 0 .8 7 2 1 7 = 8 7 2 .1 7 x 1 0  3
RULE: For numbers less than 1, separate the digits
following the decimal point into groups of three.
0 .0 0 0 7 3 6 4 9 2  0 .0 0 0 7 3 6 4 9 2 = 7 3 6 .4 9 2 x 1 0  6
0 .0 0 0 0 0 0 0 4 7 2  0 .0 0 0 0 0 0 0 4 7 2  4 7 .2 x 1 0  9
1.8-UNITS OF MEASUREMENT
1.8-EXAMPLE 77-Page 31
C o n v e r t 6 5 4 .5 f e e t
ft
=
ft
(ft.)
m
m
to m e te rs
(m ).
OPTIONAL FORMULA
Multiply by the conversion factor that will
cancel the units you wish to eliminate.
654.5 ft. x (0.3048 m / 1 ft.)
Conversion Factor: 1 ft. = 0.3048 m
654.5 ft.
x m
=
1 ft. 0.3048 m
(654.5)  (0.3048)
x
(1)
(1)•(x)=(654.5)•(0.3048)
x  199.4916 m
A n s : 1 9 9 .5 m
( 4 S . D . 's )
1.8-EXAMPLE 78-Page 31
C o n v e rt 1 3 4 a c re s to h e c ta re s
acres
=
acres
134 acres
2.471 acres
ha
ha
x ha
=
1 ha
(134)•(1)=(2.471)•(x)
(h a ).
Conversion Factor: 1 ha = 2.471 acres
(134)  (1)
x
( 2.471)
x  54.22905706 ha
A n s : 5 4 .2 h a
( 3 S .D .'s )
1.8-EXAMPLE 81-Page 33
C o n v e rt 7 2 9 2 5 m e tre s
metres
=
metres
(m )
t o k i lo m e t r e s
km
km
72 925 metres
x km
=
1 000 metres
1 km
(72 925)•(1)=(1 000)•(x)
(k m ).
Conversion Factor: 1 km = 1000 m
(72 925)•(1)
x
(1 000)
x  72.925 km
A n s : 7 2 .9 2 5 k m
( 5 S .D .'s )
1.8-EXAMPLE 82-Page 33
C o n v e rt 2 .7 5 x 1 0 5 d y n e s to n e w to n s
dynes
=
dynes
N
N
(N ) .
Conversion Factor: 1 N = 1.0 x 105 dynes
2.75 x 105 dynes
xN
=
5
1N
1.0 x 10 dynes
(2.75 x 105 )•(1)=(1.0 x 105 )•(x)
(2.75 x 105 )•(1)
x
(1.0 x 105 )
x  2.75 N
A n s : 2 .7 5 N
( 3 S .D .'s )
1.8-EXAMPLE 83-Page 34
C o n v e r t 2 .8 4 c u b i c f e e t
cu.ft.
gal.
=
cu.ft.
gal.
( c u .f t . )
t o U .S . g a llo n s
Conversion Factor: 1 cu.ft. = 7.481 gal.(U.S.)
2.84 cu.ft.
x gal.(U.S.)
=
1 cu.ft.
7.481 gal.(U.S.)
(2.84)•(7.481)=(1)•(x)
( g a l. ) .
(2.84)•(7.481)
x
(1)
x  21.24604
A n s : 2 1 .2 g a l. ( U .S .)
( 3 S .D .'s )
1.9-SUBSTITUTING INTO
EQUATIONS AND FORMULAS
1.9-EXAMPLE 87-Page 37
S u b s t i t u t e t h e v a lu e s o f a = 5 , b = 3 a n d c = 6 i n t o t h e e q u a t i o n .
3(a )  (b)
x
(c )
3(5)  (3)

(6)
18

6
A ns: 3
1.9-EXAMPLE 88-Page 37
A tensile load of 4500 lb. is applied to a bar that is 5.2 yd. long and has
cross-sectional area of 1 1.6 cm 2 (Fig.1-19). The elongation is 0.38 mm.
Using Eq.A54 , find the modulus of elasticit y E in pounds per square inch.
PL
E
ae
842400 lb.
E
0.02689759873 sq.in.
4500 lb x 5.2 yd.
E
11.6 cm2 x 0.38 mm
Convert all units to
AEpounds
ns:31318780.85
3 1 and
0 0 0inches.
0 0 0 lblb./sq.in.
./sq .in . (2 S D 's)
4500 lb. x 187.2 in.
E
1.797892126 in.2 x 0.01496062992 in.
1.10-PERCENTAGE
1.10-EXAMPLES-Pages 39 to 40
C o n v e r t in g F r a c t io n s t o D e c im a l s t o P e r c e n t s a n d b a c k a g a in .
1
 0.25  25%
4
75 3
75%  0.75 

100 4
1
1  1.25  125%
4
75
3
575%  5.75  5
5
100
4
3
27  27.6  2760%
5
5
1
0.05%  0.0005 

10000 2000
1
 0.04  4%
25
1
225
9
2 %  0.0225 

4
10000 400
1.10-STRATEGY-Pages 40 to 41
S t r a t e g y f o r s o lv i n g s i m p le p e r c e n t w o r d p r o b le m s .
RATE (%)
(Change)
PART
(Change)
FORMULA
RATE (%) PART (Change)

100%
of BASE
BASE
(Original)
OPTIONAL FORMULA
Amount = Rate x Base
A = PB
(where P is expressed as a decimal)
1.10-EXAMPLE 96-Page 40
W hat
(h o w m u c h )
OPTIONAL FORMULA
Amount = Rate x Base
A = PB
i s 3 5 .0 p e r c e n t o f 8 0 .0 ?
PART (Change)
(35.0)  (80.0)
(100)
RATE (%) PART (Change)

100%
of BASE
x
35.0 %
x

100% of 80.0
x  28
(100)  ( x)  (35.0)  (80.0)
A n s : x  2 8 .0
( 3 S . D . 's )
1.10-EXAMPLE 97-Page 41
F in d
(h o w m u c h is )
OPTIONAL FORMULA
Amount = Rate x Base
A = PB
3 .7 4 % o f 5 7 1 0 .
PART (Change)
(3.74)  (5710)
(100)
RATE (%) PART (Change)

100%
of BASE
x
3.74 %
x

100% of 5710
x  213.554
(100)  ( x)  (3.74)  (5710)
A ns: x  214
( 3 S . D . 's )
1.10-EXAMPLE 100-Page 41
OPTIONAL FORMULA
Amount = Rate x Base
A = PB
4 2 .0 i s w h a t p e r c e n t o f 4 0 5 ?
RATE (%)
RATE (%) PART (Change)

100%
of BASE
x%
42.0

100% of 405
( x)  (405)  (100)  (42.0)
x
(100)  (42.0)
(405)
______
x  10. 370 %
A n s : x  1 0 .4 %
( 3 S . D . 's )
1.10-EXAMPLE 101-Page 41
W h a t p e r c e n t o f 1 .4 5 i s 0 .3 5 7 ?
OPTIONAL FORMULA
Amount = Rate x Base
A = PB
RATE (%)
(100)  (0.357)
(1.45)
RATE (%) PART (Change)

100%
of BASE
x
x%
0.357

100% of 1.45
x  24.62068966 %
( x)  (1.45)  (100)  (0.357)
A n s : x  2 4 .6 %
( 3 S . D . 's )
1.10-EXAMPLE 98-Page 41
OPTIONAL FORMULA
Amount = Rate x Base
A = PB
1 2 % o f w h a t n u m b e r is 7 8 ?
BASE (Original)
(100)  (78)
(12)
RATE (%) PART (Change)

100%
of BASE
x
12 %
78

100% of x
x  650
(12)  ( x)  (100)  (78)
A ns: x  650
( 2 S . D . 's )
1.10-EXAMPLE 99-Page 41
OPTIONAL FORMULA
Amount = Rate x Base
A = PB
1 4 0 is 2 5 % o f w h a t n u m b e r ?
BASE (Original)
(100)  (140)
(25)
RATE (%) PART (Change)

100%
of BASE
x
25 % 140

100% of x
x  560
(25)  ( x)  (100)  (140)
A ns: x  560
( 2 S . D . 's )
1.10-STRATEGY-Pages 41 to 43
S t r a t e g y f o r s o lv i n g m o r e c o m p le x p e r c e n t w o r d p r o b le m s .
Type I: Percent Change
Percent Change=
New Value-Original Value
x 100
Original Value
Type II: Percent Efficiency
Percent Efficiency=
Output
x 100
Input
Type III: Percent Error
Percent Error=
Measured Value-Correct Value
x 100
Correct Value
Type IV: Percent Concentration
Percent Concentration=
Amount of Ingredient A
x 100
Total Amount of Mixture
1.10-EXAMPLE 102-Page 42
A certain price rose from $1.55 to $1.75 . Find the percentage
change in price.
Type I: Percent Change
Percent Change=
New Value-Original Value
x 100
Original Value
($1.75-$1.55)
% Change=
x 100
($1.55)
x  12.9032258%
($0.20)
% Change=
x 100
($1.55)
% Change=0.129032258 x 100
A n s : x  1 2 .9 %
( 3 S . D . 's )
1.10-EXAMPLE 103-Page 42
Find the cost of a $156.00 suit after th e price increases by 2
Type I: Percent Change
1
%.
2
 Percent Change

New Value=(Original Value)+ 
•Original Value 
100


 2.5

New Value=($156.00)+ 
•$156.00 
 100

New Value=($156.00)+  0.025•$156.00 
New Value=($156.00)+  $3.90 
New Value=$159.90
A n s : x  $ 1 5 9 .9 0
(E x a c t)
1.10-EXAMPLE 104-Page 42
A certain electric m otor consum es 865 W and has an output
of 1.12 hp. Find the efficiency of the m otor (1 hp  746 W ) .
Type II: Percent Efficiency
Percent Efficiency=
Output
x 100
Input
836 W
Percent Efficiency=
x 100
865 W
 746 W
NB: Output=1.12 hp• 
 1 hp

 =836 W

Percent Efficiency=0.966473988 x 100
Percent Efficiency=96.6473988 %
A n s : x  9 6 .6 %
( 3 S .D .'s )
1.10-EXAMPLE 105-Page 43
A laboratory w eight that is certified to be 500.0 g is placed on
a scale. T he scale reading is 507.0 g. W hat is the percent error ?
Type III: Percent Error
Percent Error=
Measured Value-Correct Value
x 100
Correct Value
(507.0 g-500.0 g)
Percent Error=
x 100
(500.0 g)
x=1.4 %
(7.0 g)
Percent Error=
x 100
(500.0 g)
Percent Error=0.014 x 100
A n s : x  1 .4 0 0 %
H i g h ( 4 S .D .'s )
1.10-EXAMPLE 106-Page 43
A certain fuel m ixture contains 18.9 litres of alcohol and 84.7
of gasoline. Find the percentage of gasoline in the m ixture.
litres
Type IV: Percent Concentration
Percent Concentration=
Amount of Ingredient A
x 100
Total Amount of Mixture
84.7litres
Percent Concentration=
x 100
(18.9litres+84.7litres)
84.7litres
Percent Concentration=
x 100
103.6litres
Percent Concentration=0.8175676 x 100
x=81.75676 %
A n s : x  8 1 .8 %
H i g h ( 3 S .D .'s )
Copyright
Copyright © 2012 John Wiley & Sons Canada, Ltd.
All rights reserved. Reproduction or translation of this work beyond that
permitted by Access Copyright (The Canadian Copyright Licensing Agency) is
unlawful. Requests for further information should be addressed to the
Permissions Department, John Wiley & Sons Canada, Ltd. The purchaser may
make back-up copies for his or her own use only and not for distribution or resale.
The author and the publisher assume no responsibility for errors, omissions, or
damages caused by the use of these programs or from the use of the information
contained herein.