Chapter 1: Introduction and Mathematical Background
1.1 What is Numerical Analysis?
• Definition: Development and analysis of algorithms for obtaining numerical solutions to mathematical
problems.
• Analytical vs. Numerical Methods:
– Analytical: Exact solutions, efficient for simple problems.
– Numerical: Approximate solutions, useful for complex problems, requires iterations.
• Applications: Solving differential equations, optimization, interpolation, and more.
1.2 Representation of Numbers on a Computer
• Base-β Number Representation:
– Any integer digit β > 1 can be used as the base.
– Common bases: Decimal (β = 10), Binary (β = 2), Octal (β = 8), Hexadecimal (β = 16).
– Conversion between bases:
∗ Procedure 1: Express the number in nested form and replace digits.
∗ Procedure 2: Separate integer and fractional parts, convert each part separately.
• Floating Point Representation:
– Used to represent very large or very small numbers.
– IEEE-754 Standard: Single precision (32 bits), Double precision (64 bits).
– Overflow and Underflow: Errors caused by numbers outside the representable range.
– Normalization: Represent numbers in the form ±d0 .d1 d2 d3 · · · × β e .
1.3 Errors in Numerical Solutions
• Types of Errors:
– Round-Off Errors: Due to limited precision in representing numbers.
– Truncation Errors: Due to approximating infinite processes with finite steps.
– Total Error: Sum of round-off and truncation errors.
• Error Measures:
– True Error: Ex = x − x̃
– True Relative Error: ExR =
x−x̃
x
– Estimated Relative Error: ExER =
x̃n −x̃n−1
x̃n
• Error Propagation: Errors can accumulate during calculations, especially in iterative methods.
1.4 Mathematical Background
• Taylor Series Expansion:
– Approximates a function near a point using derivatives.
– Taylor Series for One Variable:
f (x) = f (x0 ) + f ′ (x0 )(x − x0 ) +
f ′′ (x0 )
(x − x0 )2 + . . .
2!
– Taylor Series for Two Variables:
f (x + h, y + k) = f (x, y) + h
1
∂f
∂f
+k
+ ...
∂x
∂y
• Jacobian Matrix:
– Used in solving systems of nonlinear equations.
– For functions f1 , f2 , . . . , fn , the Jacobian matrix J is:
 ∂f1
∂x1
 ∂f2
J =  ∂x1
..
.
∂f1
∂x2
∂f2
∂x2
..
.

...
. . .

..
.
• Partial Derivatives: Used in multivariable functions to find rates of change.
• Chain Rule: For functions of multiple variables, the chain rule helps in differentiating composite
functions.
Chapter 2: Solutions of Nonlinear Equations
2.1 Bracketing Methods
• Bisection Method:
– Principle: Halve the interval containing the root until the solution is found.
– Algorithm:
1. Start with interval [a, b] where f (a)f (b) < 0.
2. Compute midpoint c = a+b
2 .
3. If f (c) = 0, stop. Else, replace a or b with c based on the sign of f (c).
– Convergence: Linear, guaranteed if f is continuous.
– Error: |xT S − xN S | ≤ b−a
2n
– Advantages: Simple and robust.
– Disadvantages: Slow convergence.
• Regula Falsi Method:
– Principle: Similar to bisection but uses linear interpolation to find the next approximation.
(bi )−bi f (ai )
– Equation: xN Si = aiff (b
i )−f (ai )
– Convergence: Faster than bisection but still linear.
– Advantages: Faster convergence than bisection.
– Disadvantages: Can be slower for certain functions.
2.2 Open Methods
• Newton’s Method:
– Principle: Use tangent lines to approximate the root.
i)
– Equation: xi+1 = xi − ff′(x
(xi )
– Convergence: Quadratic, but requires a good initial guess.
– Advantages: Fast convergence.
– Disadvantages: Requires derivative, may diverge if initial guess is poor.
• Secant Method:
– Principle: Similar to Newton’s method but uses a secant line instead of a tangent.
i )(xi−1 −xi )
– Equation: xi+1 = xi − ff(x
(xi−1 )−f (xi )
– Convergence: Superlinear, slower than Newton’s method.
– Advantages: No need for derivative.
2
– Disadvantages: Slower convergence than Newton’s method.
• Fixed-Point Iteration:
– Principle: Rewrite f (x) = 0 as x = g(x) and iterate.
– Convergence: Guaranteed if |g ′ (x)| < 1 near the root.
– Advantages: Simple to implement.
– Disadvantages: Convergence depends on the choice of g(x).
2.3 Systems of Nonlinear Equations
• Newton-Raphson Method:
– Principle: Extend Newton’s method to multiple dimensions.
– Equation: xi+1 = xi − J −1 f (xi ), where J is the Jacobian matrix.
– Convergence: Quadratic, but requires a good initial guess and J must be invertible.
– Advantages: Fast convergence for well-behaved systems.
– Disadvantages: Computationally expensive for large systems.
• Fixed-Point Iteration:
– Principle: Extend fixed-point iteration to multiple dimensions.
– Convergence: Guaranteed if the spectral radius of the Jacobian is less than 1.
– Advantages: Simple to implement.
– Disadvantages: Convergence depends on the choice of iteration function.
Chapter 3: Systems of Linear Equations
3.1 Direct Methods
• Gauss Elimination:
– Principle: Convert the system to upper-triangular form and solve using back substitution.
– Algorithm:
1. Forward elimination: Eliminate variables below the pivot.
2. Back substitution: Solve for variables starting from the last equation.
– Pivoting: Avoid division by zero by rearranging rows.
– Advantages: Direct solution, no iterations.
– Disadvantages: Sensitive to round-off errors.
• Gauss-Jordan Elimination:
– Principle: Convert the system to diagonal form and solve directly.
– Algorithm:
1. Normalize the pivot row.
2. Eliminate variables above and below the pivot.
– Advantages: Direct solution, no back substitution.
– Disadvantages: More computationally expensive than Gauss elimination.
• LU Decomposition:
– Principle: Factorize A into lower (L) and upper (U ) triangular matrices.
– Algorithm:
1. Decompose A into L and U .
2. Solve Ly = b for y.
3
3. Solve U x = y for x.
– Crout’s Method: Diagonal elements of U are 1.
– Cholesky’s Method: For symmetric, positive-definite matrices.
– Advantages: Efficient for multiple right-hand sides.
– Disadvantages: Requires factorization step.
3.2 Iterative Methods
• Jacobi Iterative Method:
– Principle: Update all variables simultaneously using previous values.
P
– Equation: xi,k+1 = a1ii (bi − j̸=i aij xj,k )
– Advantages: Simple to implement.
– Disadvantages: Slow convergence.
• Gauss-Seidel Iterative Method:
– Principle: Update variables sequentially using the most recent values.
P
P
– Equation: xi,k+1 = a1ii (bi − j<i aij xj,k+1 − j>i aij xj,k )
– Advantages: Faster convergence than Jacobi.
– Disadvantages: Still slower than direct methods for some systems.
3.3 Matrix Inversion
• Principle: Solve AX = I for X, where X = A−1 .
• Gauss-Jordan Method:
– Append I to A and perform row operations to convert A to I.
• LU Decomposition:
– Solve Ly = I for y, then solve U x = y for x.
• Advantages: Useful for solving multiple systems with the same coefficient matrix.
• Disadvantages: Computationally expensive for large matrices.
3.4 Eigenvalues and Eigenvectors
• Definition: For a square matrix A, if Ax = λx, then λ is an eigenvalue and x is an eigenvector.
• Characteristic Polynomial: det(A − λI) = 0.
• Power Method: Iterative method to find the dominant eigenvalue.
• Inverse Power Method: Iterative method to find the smallest eigenvalue.
• Shifted Power Method: Iterative method to find eigenvalues near a given value.
• Advantages: Useful in stability analysis, vibration analysis, and more.
• Disadvantages: Computationally expensive for large matrices.
Chapter 4: Interpolation
4.1 Introduction to Interpolation
• Objective: Given a set of n data points (xi , yi ), where i = 1, 2, . . . , n, the goal is to approximate the
value of y for any x that lies between the provided data points.
• Curve Fitting: This involves determining a function y = f (x) (e.g., a polynomial) that passes as
closely as possible through the given data points. The function is then used to estimate y values for
x values that lie between two known points.
4
4.2 Key Concepts in Interpolation
• Errorless Data: In simpler cases, the data is assumed to be free of errors, and f (x) passes exactly
through all given points.
• Data with Errors: When data contains measurement errors, f (x) does not pass exactly through the
data points. Instead, it is chosen to minimize discrepancies, often using methods like least-squares
estimation.
4.3 Polynomial Interpolation
• Objective: Given n data points (xi , yi ), find a polynomial f (x) of the lowest possible degree such
that f (xi ) = yi for all i = 1, 2, . . . , n.
• Theorem: If x1 , x2 , . . . , xn are distinct real numbers, then for any values y1 , y2 , . . . , yn , there exists
a unique polynomial f (x) of degree at most m = n − 1 such that f (xi ) = yi for 1 ≤ i ≤ n. This
polynomial accurately reproduces the data and can estimate y-values for any x-values.
• Solving a System of Linear Equations:
– Construct an equation for each point (xi , yi ) in the dataset. Solve the resulting n × n system of
linear equations to find the unknown coefficients a1 , a2 , . . . , an .
– While these systems can be solved using methods from Chapter 3, this approach is not recommended for high-degree polynomials because the matrix can become ill-conditioned, especially
when data points are close.
4.4 Lagrange Interpolating Polynomials
• Definition: Interpolating polynomials f (x) are expressed as:
y = f (x) =
n
X
yi Li (x),
i=1
where Li (x) are Lagrange (cardinal) functions dependent on x1 , x2 , . . . , xn .
– Li (xj ) = δij , where δij is the Kronecker delta (1 if i = j, 0 if i ̸= j).
• Key Points:
– Explicitly determining the polynomial’s coefficients is unnecessary.
– The polynomial can be written immediately using the data points and used to interpolate the
value of y at any point.
4.5 Development of the Lagrange Method
• Linear Interpolation: Given two points (x1 , y1 ) and (x2 , y2 ), the value of y = f (x) for an x between
x1 and x2 is:
x − x1
x − x2
+ y2
.
y = y1
x1 − x2
x2 − x1
• Quadratic Interpolation: Given three points (x1 , y1 ), (x2 , y2 ), and (x3 , y3 ), the value of y = f (x)
for an x between x1 and x3 is:
y = y1
(x − x2 )(x − x3 )
(x − x1 )(x − x3 )
(x − x1 )(x − x2 )
+ y2
+ y3
.
(x1 − x2 )(x1 − x3 )
(x2 − x1 )(x2 − x3 )
(x3 − x1 )(x3 − x2 )
• Generalization: As more data points and higher-degree polynomials are used, the procedure can be
generalized to derive the general formula for Lagrange interpolating polynomials. Adding more data
requires recomputing interpolated values.
5
4.6 Newton’s Interpolating Polynomials
• Motivation: Lagrange polynomials require recomputation when new data points are added.
• Principle: Newton’s interpolating polynomials are constructed so that higher-degree polynomials
build upon lower-degree ones.
– Linear: f (x) = a1 + a2 (x − x1 ).
– Quadratic: f (x) = a1 + a2 (x − x1 ) + a3 (x − x1 )(x − x2 ).
• General Expression: The general expression for Newton’s polynomials is:
fk (x) = fk−1 (x) + ak+1 (x − x1 )(x − x2 ) . . . (x − xk ).
• Divided Differences: The coefficients ak are computed using divided differences:
ak = f [xk , xk−1 , . . . , x1 ].
– Computation: The coefficients are computed sequentially and systematically.
4.7 Spline (Piecewise) Interpolation
• Motivation: Accuracy can sometimes be improved by using higher-degree polynomials, but it can
also deteriorate near the endpoints. Spline interpolation uses low-degree polynomials fitted piecewise
to subsets of the data points.
• Definition: Splines are piecewise polynomials defined on subintervals of the data set and joined
together by certain continuity conditions.
• 1st Degree or Linear Splines S1 (x):
– They are straight lines connecting adjacent points.
– First-degree splines have discontinuous derivatives at the data points.
• 2nd Degree or Quadratic Splines S2 (x):
– They are piecewise quadratic polynomials joined together at the knots, resulting in a spline
function that is continuous and has continuous first derivatives.
– Given n points, there are n − 2 interior knots and n − 1 equations, requiring solving for 3n − 3
coefficients.
Key Formulas and Concepts
Taylor Series Expansion
• One Variable:
f (x) = f (x0 ) + f ′ (x0 )(x − x0 ) +
f ′′ (x0 )
(x − x0 )2 + . . .
2!
• Two Variables:
f (x + h, y + k) = f (x, y) + h
Error Measures
• True Error: Ex = x − x̃
• True Relative Error: ExR =
x−x̃
x
• Estimated Relative Error: ExER =
x̃n −x̃n−1
x̃n
6
∂f
∂f
+k
+ ...
∂x
∂y
Bisection Method
• Error Bound: |xT S − xN S | ≤ b−a
2n
• Number of Iterations: n > ln(b−a)−ln(δ)
ln(2)
Newton’s Method
i)
• Iteration Formula: xi+1 = xi − ff′(x
(xi )
Secant Method
i )(xi−1 −xi )
• Iteration Formula: xi+1 = xi − ff(x
(xi−1 )−f (xi )
Fixed-Point Iteration
• Iteration Formula: xi+1 = g(xi )
• Convergence Condition: |g ′ (x)| < 1
Gauss Elimination
• Forward Elimination: Eliminate variables below the pivot.
• Back Substitution: Solve for variables starting from the last equation.
LU Decomposition
• Decomposition: A = LU
• Solution: Solve Ly = b for y, then solve U x = y for x.
Jacobi Iterative Method
• Iteration Formula: xi,k+1 = a1ii (bi −
P
j̸=i aij xj,k )
Gauss-Seidel Iterative Method
• Iteration Formula: xi,k+1 = a1ii (bi −
P
j<i aij xj,k+1 −
P
j>i aij xj,k )
Eigenvalues and Eigenvectors
• Characteristic Polynomial: det(A − λI) = 0
• Power Method: Iterative method to find the dominant eigenvalue.
7