Numerical Methods Course Notes Compiled and corrected from lecture notes Contents 1 Introduction 1.1 What Numerical Analysis Involves . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Taylor Series and Maclaurin Series . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Truncation Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Rounding and Chopping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Error Propagation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.6 Types of Computational Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2 2 2 2 3 3 2 Transcendental and Polynomial Equations 4 2.1 Algebraic vs Transcendental Equations . . . . . . . . . . . . . . . . . . . . . . . . 4 2.2 Intermediate Value Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.3 Bisection Method (Interval Halving) . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.4 Descartes’ Rule of Signs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.5 Secant Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.6 Algorithm of the Bisection Method . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.7 Regula Falsi Method (Method of False Position) . . . . . . . . . . . . . . . . . . 6 2.8 Newton–Raphson Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.9 Assignment: f (x) = x − e−x = 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.10 Method of Complex Roots (Newton’s Method in Two Variables) . . . . . . . . . 10 2.11 Graeffe’s Root-Squaring Method . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3 System of Linear Equations (Matrix Methods) 3.1 Matrix Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Methods for Solving AX = B / Finding Eigenvalues . . . . . . . . . . . . . . . . 3.3 Augmented Matrix and Row-Echelon Form . . . . . . . . . . . . . . . . . . . . . 3.4 Worked Example: Gauss Elimination . . . . . . . . . . . . . . . . . . . . . . . . . 1 12 12 12 12 13 1 Introduction Reference Book: Numerical Methods for Scientific and Engineering Computation, by M.K. Jain, S.R.K. Iyengar, R.K. Jain. 1.1 What Numerical Analysis Involves To completely study/solve a problem using numerical analysis, three components are required: (i) Theory (ii) Program (iii) Computer Implementation When an analytic (exact) value cannot be obtained easily, we compute an approximate value instead. For example, consider a function whose graph passes through a certain region; we may only be able to say the root lies in an interval such as (1, 2) rather than pin down its exact value. 1.2 Taylor Series and Maclaurin Series Taylor Series: For a function f expandable about x = a, f (x) = ∞ X f (n) (a) n=0 n! (x − a) ′ (x − a)2 ′′ f (a) + f (a) + · · · 1! 2! (x − a)n = f (a) + Maclaurin Series: Putting a = 0 in the Taylor series, f (x) = ∞ X f (n) (0) n=0 n! xn = f (0) + f ′ (0) f ′′ (0) 2 x+ x + ··· 1! 2! A commonly used form (Taylor expansion of y about x, step h): y(x + h) = y(x) + h y ′ (x) + 1.3 h3 h2 ′′ y (x) + y ′′′ (x) + · · · 2! 3! Truncation Error If, in a certain equation, values above a certain order are neglected, the value we are left with differs from the true value — this difference is called the truncation error. Error = (Exact value) − (Approximate value) 1.4 Rounding and Chopping When storing a value up to a certain number of digits, we either round or chop it. For example: 2 = 0.66666 . . . 3 ≈ 0.67 (2 digits) ≈ 0.667 (3 digits) ≈ 0.6667 (4 digits) ≈ 0.66667 (5 digits) The error that arises from taking a value up to a fixed number of digits is called the Round-off Error (or Chopping Error). 2 1.5 Error Propagation Convergent: A function, sequence, or series is said to be convergent if its value tends to a fixed (finite) value as the index/variable increases. E.g. 1 = 0. x→∞ x lim Divergent: A function, sequence, or series is said to be divergent if it does not tend to a fixed value; instead it grows without bound or oscillates. E.g. lim x = ∞, x→∞ or a sequence like 1, −1, 1, −1, . . . which never settles to a single finite value. 1.6 Types of Computational Error Machine Error: Error arising from the calculator/computer itself and from the input data. Modeling Error: Error arising from how the calculation/problem is modeled mathematically (i.e. how well the model represents the real system). 3 2 Transcendental and Polynomial Equations For a non-linear equation, an exact solution is not always available, so we resort to iterative/numerical techniques. 2.1 Algebraic vs Transcendental Equations Algebraic equation: An equation whose coefficients are algebraic numbers (polynomial equations). Transcendental equation: An equation involving logarithmic, trigonometric, exponential, etc. coefficients/terms. Such an equation may have an infinite number of solutions. 2.2 Intermediate Value Theorem Theorem 1 (Intermediate Value Theorem). If f (x) is a continuous function on some closed interval [a, b] and f (a) · f (b) < 0, then the equation f (x) = 0 has at least one real root, or an odd number of roots, in the open interval (a, b). f (a) Root a b f (b) 2.3 Bisection Method (Interval Halving) Let m1 = a+b . 2 If f (m1 ) > 0 (positive): the root lies on the positive side of the x-axis relative to m1 . If f (m1 ) < 0 (negative): the root lies on the negative side. If f (m1 ) = 0: m1 itself is the root. If f (a) is positive and f (m1 ) is negative ⇒ root lies in [a, m1 ]. If f (m1 ) and f (b) have opposite signs ⇒ root lies in [m1 , b]. m1 + b a + m1 Then m2 = (or , depending on which sub-interval contains the root), and 2 2 the process repeats. Convergence Criterion For the sequence a, b, m1 , m2 , m3 , . . . we track successive differences |a − b|, |b − m1 |, |m1 − m2 |, . . . The sequence converges towards the exact root as these differences shrink. We stop when |m1 − m2 | < ε 4 for a chosen tolerance ε. For instance, if |1.5 − 2| = 0.5, |1.75 − 2| = 0.25, |1.875 − 1.75| = 0.125, . . ., the successive gaps shrink geometrically toward 0, giving increasingly accurate roots (each halving roughly gains accuracy toward the exact root). 2.4 Descartes’ Rule of Signs The exact number of positive and negative real roots of a polynomial can be estimated by counting sign changes in f (x) and f (−x) respectively. Example. Let f (x) = x4 − x − 10 = 0. Positive roots: Coefficients of f (x) = x4 − x − 10 have signs +, −, − giving 1 sign change ⇒ number of positive real roots = 1. Negative roots: f (−x) = (−x)4 − (−x) − 10 = x4 + x − 10 = 0 Signs: +, +, − giving 1 sign change ⇒ number of negative real roots = 1. Evaluating: f (0) = −10, f (1) = −10, f (2) = 4 Since f (1) < 0 and f (2) > 0, exactly one root lies in [1, 2] (consistent with Descartes’ rule: 1 positive real root). Worked Bisection Iterations for f (x) = x4 − x − 10 = 0 on [1, 2] a+b 1+2 = = 1.5, f (m1 ) = f (1.5) = −6.4375 ⇒ x ∈ [1.5, 2] 2 2 1.5 + 2 m2 = = 1.75, f (m2 ) = f (1.75) = −2.371 ⇒ x ∈ [1.75, 2] 2 1.75 + 2 = 1.875, f (m3 ) = 0.485 ⇒ x ∈ [1.75, 1.875] m3 = 2 1.75 + 1.875 m4 = = 1.8125, f (m4 ) = −1.02 ⇒ x ∈ [1.8125, 1.875] 2 m1 = n mn 0 1 2 3 4 x4 − x − 10 = 0, x ∈ [1, 2] 1.5 1.75 1.875 1.8125 f (mn ) |mn − mn−1 | <0 <0 >0 <0 − 0.5 0.25 0.125 The root continues to be trapped in ever-shrinking intervals, converging to x ≈ 1.856. 2.5 Secant Method Idea. Approximate f (x) locally by a straight line (from a two-term power series): f (x) = a0 + a1 x so that f (xk ) = a0 + a1 xk = 0, f (xk+1 ) = a0 + a1 xk+1 = 0. Writing fk = f (xk ) and fk+1 = f (xk+1 ): fk − a0 − a1 xk = 0 (i), fk+1 − a0 − a1 xk+1 = 0 5 (ii) From (i): a0 = fk − a1 xk . Substituting into (ii): fk+1 − fk + a1 xk − a1 xk+1 = 0 =⇒ a1 = Substituting a1 back: fk − fk+1 . xk − xk+1 xk fk+1 − fk xk+1 . xk − xk+1 a0 = Since f (x) = a0 + a1 x = 0 gives x = −a0 /a1 , the next approximation is xk+2 = − xk fk+1 − xk+1 fk a0 = . a1 fk+1 − fk This simplifies to the standard Secant Method formula: xk+2 = xk+1 − 2.6 xk+1 − xk fk+1 fk+1 − fk Algorithm of the Bisection Method Step 1: Find the interval in which the root lies, i.e. x ∈ [a, b] such that either f (a) or f (b) < 0 (using the Intermediate Value Theorem). Step 2: Calculate m1 = a+b . 2 Step 3: Find whether f (m1 ) > 0 or f (m1 ) < 0. Step 4: Search the sub-interval: x ∈ [a, m1 ] or x ∈ [m1 , b]. Step 5: Calculate m2 = m1 + b a + m1 or . 2 2 Step 6: Find x ∈ [a, m2 ] or [m2 , m1 ] or [m1 , m2 ] or [m2 , b]. Step 7: If |m1 − m2 | < ε (tolerance), go to Step 8; otherwise go to Step 2. Step 8: Stop. Mandatory elements of any numerical program. 5 elements: Any program must contain (at least) (i) Variable declaration (ii) Input (iii) Function (iv) Output (v) End 2.7 Regula Falsi Method (Method of False Position) xk+1 = xk − xk − xk−1 fk , fk − fk−1 6 k = 1, 2, 3, . . . Example. Solve f (x) = x4 − x − 10 = 0, with x ∈ [1, 2]. f (x0 ) = f (1) = −10, k = 1: x2 = x1 − f (x1 ) = f (2) = 4 2−1 x1 − x0 f1 = 2 − (4) = 1.7143 f1 − f0 4 − (−10) f (x2 ) = f (1.7143) = −3.0776 k = 2: x3 = x2 − 1.7143 − 2 x2 − x1 f2 = 1.7143 − (−3.0776) = 1.8385 f2 − f1 −3.0776 − 4 f (x3 ) = −0.4135 k = 3: x4 = x3 − 1.8385 − 1.7143 x3 − x2 f3 = 1.8385 − (−0.4135) = 1.8578 f3 − f2 −0.4135 − (−3.0776) n xn f (xn ) |xn − xn−1 | 0 1 2 3 4 1 2 1.7143 1.8385 1.8578 <0 >0 <0 <0 >0 − 1 0.2857 0.1242 0.0193 The iterates are converging toward the root x ≈ 1.858 (the same root found earlier by the Bisection Method). 2.8 Newton–Raphson Method Derivation. Approximate f (x) locally by a straight line f (x) = a0 + a1 x, so f ′ (x) = a1 . From this, f (x) = a0 + x f ′ (x) ⇒ a0 = f (x) − xf ′ (x). For an arbitrary point xn : f (xn ) = a0 + a1 xn , f ′ (xn ) = a1 , xn f ′ (xn ) + a0 = f (xn ) ∴ a0 = f (xn ) − xn f ′ (xn ) Since xn+1 = −a0 /a1 (root of the linear approximation): xn+1 = − f (xn ) − xn f ′ (xn ) xn f (xn ) f (xn ) = ′ − ′ ′ f (xn ) f (xn ) f (xn ) xn+1 = xn − f (xn ) , f ′ (xn ) n = 1, 2, 3, . . . Remarks. This method converges faster than bisection/regula-falsi. Order of convergence is 2 (quadratic), regardless of whether the root is simple. Only one starting point is required (unlike bisection/secant/regula-falsi, which need two points bracketing the root). 7 Disadvantage: if f ′ (x) = 0 at some iterate, the method is undefined there (division by zero); the iteration may also fail to converge for a poor initial guess. Point of convergence is only guaranteed in some neighbourhood of the root (local convergence) — not global for an arbitrary starting point. Choosing the Initial Guess for Newton–Raphson A commonly used sufficient condition is x0 such that f (x0 ) · f ′′ (x0 ) > 0. Worked Example Solve f (x) = x − e−x = 0. f (xn ) = xn − e−xn , f ′ (xn ) = 1 + e−xn xn+1 = xn − Take x0 = 0.5. n = 0: xn − e−xn 1 + e−xn x1 = x0 − f (0) 0.5 − e−0.5 = 0.5 − = 0.5663 f ′ (0) evaluated at x0 1 + e−0.5 x2 = x1 − x1 − e−x1 0.5663 − e−0.5663 = 0.5663 − = 0.5671 1 + e−x1 1 + e−0.5663 n = 1: n = 2: x3 = 0.5671 n xn f (xn ) |xn − xn−1 | 0 1 2 0.5 0.5663 0.5671 − − ≈0 − 0.0663 0.0008 The iteration has converged rapidly (in just 2 steps) to x ≈ 0.5671, compared to 4–5 iterations needed by the Secant and Regula Falsi methods for the same accuracy (see §2.11). 2.9 Assignment: f (x) = x − e−x = 0 Solve using (i) the Secant Method and (ii) the Regula Falsi Method, and tabulate the results. f (1) = 1 − e−1 = 0.6321 (positive) f (0) = 0 − e0 = −1 (negative), Therefore the smallest positive root lies in the interval (0, 1). 8 (i) Secant Method Take x0 = 0, x1 = 1. Using xk+1 = xk − k = 1: x2 = x1 − xk − xk−1 fk , fk − fk−1 k = 1, 2, 3, . . . 1−0 x1 − x0 f1 = 1 − (0.6321) = 0.6127 f1 − f0 0.6321 − (−1) f (x2 ) = f (0.6127) = 0.0708 k = 2: x3 = x2 − x2 − x1 0.6127 − 1 f2 = 0.6127 − (0.0708) = 0.5638 f2 − f1 0.0708 − 0.6321 f (x3 ) = f (0.5638) = −0.0052 k = 3: x4 = x3 − 0.5638 − 0.6127 x3 − x2 f3 = 0.5638 − (−0.0052) = 0.5671 f3 − f2 −0.0052 − 0.0708 f (x4 ) = f (0.5671) = −6.784 × 10−5 n xn f (xn ) |xn − xn−1 | |fn − fn−1 | 0 1 2 3 4 0 1 0.6127 0.5638 0.5671 −1 0.6321 0.0708 −0.0052 −6.784 × 10−5 − 1 0.3873 0.0489 0.0033 − 1.6321 0.5613 0.0760 0.00513 Root (Secant Method) ≈ 0.5671. (ii) Regula Falsi Method Take x0 = 0, x1 = 1 (so that f (x0 ) < 0 < f (x1 ), bracketing the root). k = 1: (same as the Secant Method’s first step, since both start from x0 , x1 ) x2 = 0.6127, f (x2 ) = 0.0708 Since f (x0 ) and f (x2 ) have opposite signs, the root is now bracketed in (x0 , x2 ) = (0, 0.6127). k = 2: Keep x0 fixed (Regula Falsi retains the bracketing endpoint): x3 = x2 − x2 − x0 0.6127 − 0 f2 = 0.6127 − (0.0708) = 0.572 f2 − f0 0.0708 − (−1) The iteration continues in this manner, converging (a little more slowly than the Secant Method because one endpoint stays fixed) to the same root x ≈ 0.567. n xn f (xn ) |xn − xn−1 | 0 1 2 3 0 1 0.6127 0.572 −1 0.6321 0.0708 ≈ 0.0088 − 1 0.3873 0.0407 9 2.10 Method of Complex Roots (Newton’s Method in Two Variables) For a complex function w(z) = f (z) = u(x, y) + i v(x, y), write f1 (x, y) = u(x, y) = 0, f2 (x, y) = v(x, y) = 0. The (real, single-variable) Newton-Raphson iteration xn+1 = xn − f (xn ) f ′ (xn ) generalizes to the vector form x x −1 f1 (xn , yn ) = −J y n+1 y n f2 (xn , yn ) where J is the Jacobian matrix of partial derivatives, ∂(f1 , f2 ) ux uy J= = . vx vy ∂(x, y) Example. f (z) = z 2 + 1 = (x2 − y 2 + 1) + i(2xy), so f1 (x, y) = u(x, y) = x2 − y 2 + 1, f2 (x, y) = v(x, y) = 2xy. ux uy 2x −2y Jn = = . vx vy 2y 2x (This Jacobian is then inverted at each iterate (xn , yn ) to update the complex root estimate zn+1 = xn+1 + iyn+1 .) 2.11 Graeffe’s Root-Squaring Method Given the polynomial xn + a1 xn−1 + a2 xn−2 + a3 xn−3 + · · · + an = 0, separate even- and odd-indexed terms: xn + a2 xn−2 + a4 xn−4 + · · · = − a1 xn−1 + a3 xn−3 + · · · . Squaring both sides and collecting powers of x2 gives a new polynomial in x2 : x2n − (a21 − 2a2 )x2n−2 + (a22 − 2a1 a3 + 2a4 )x2n−4 − · · · + (−1)n a2n = 0. Substituting z = −x2 : z n + b1 z n−1 + b2 z n−2 + · · · + bn = 0, where b0 = a0 (= 1), b1 = a21 − 2a2 , b2 = a22 − 2a1 a3 + 2a4 , . . . In general, bk = a2k − 2ak−1 ak+1 + 2ak−2 ak+2 − · · · Repeating this ”root-squaring” process m times produces coefficients whose roots are the 2m -th powers of the original roots — which, when well separated in magnitude, can be extracted from the ratios of successive coefficients: n o B1 , log10 |Ri | = 2m log |Bi | − log |Bi−1 | , i = 1, 2, 3, . . . R1 = B0 10 Example. x3 − 6x2 + 11x − 6 = 0, i.e. a0 = 1, a1 = −6, a2 = 11, a3 = −6. Round 1 (squaring): b0 = a20 = 1 b1 = a21 − 2a2 = (−6)2 − 2(11) = 36 − 22 = 14 b2 = a22 − 2a1 a3 = 112 − 2(−6)(−6) = 121 − 72 = 49 b3 = a23 = (−6)2 = 36 Giving the new (transformed) polynomial coefficients: 1, 14, 49, 36. Iteration a0 a1 a2 a3 0 1 (squares) 1 (cross terms) 1 (result bi ) 1 1 −6 36 −22 14 11 121 −72 49 −6 36 1 36 (The process is repeated on 1, 14, 49, 36 for further rounds, and the roots are then backextracted from the ratios of successive coefficients as above; since the original roots here are simply 1, 2, 3, this method converges to those values as the number of squarings increases.) 11 3 System of Linear Equations (Matrix Methods) 3.1 Matrix Formulation A system of n linear equations in n unknowns, a11 x1 + a12 x2 + a13 x3 + · · · + a1n xn = b1 a21 x1 + a22 x2 + a23 x3 + · · · + a2n xn = b2 .. . an1 x1 + an2 x2 + an3 x3 + · · · + ann xn = bn is written compactly as AX = B, a11 a21 A= . .. a12 a22 .. . an1 an2 x1 x2 X = . , .. · · · a1n · · · a2n .. , .. . . · · · ann xn b1 b2 B = . . .. bn If A is non-singular (|A| = ̸ 0), the (unique) solution is X = A−1 B, A−1 = 1 adj(A). |A| If |A| = 0, the matrix A is called singular and the system does not have a unique solution. Equivalent Matrix. A matrix obtained from another by a sequence of elementary row operations is called row-equivalent (each such matrix is ”equivalent” to the original). 3.2 Methods for Solving AX = B / Finding Eigenvalues 1. Direct Methods (i) Gauss Elimination Method (ii) Gauss–Jordan Method (iii) LU Decomposition 2. Iterative Methods (i) Jacobi Iterative Method (ii) Gauss–Seidel Method 3.3 Augmented Matrix and Row-Echelon Form The Augmented Matrix is a11 a21 A∗ = (A | B) = . .. a12 a22 .. . an1 an2 · · · a1n b1 · · · a2n b2 .. .. . .. . . . · · · ann bn If A is a diagonal matrix, all entries below (and above) the diagonal are zero: a11 0 0 0 a22 0 , a21 = a31 = a32 = 0. 0 0 a33 12 Row-Echelon Form. All entries below the leading (pivot) diagonal are reduced to zero, and the leading entries are normalized to 1: 1 a∗12 a∗13 · · · b∗1 0 1 a∗ · · · b∗ 23 2 ∗ 0 0 ∗ 1 · · · b A = 3 .. . .. . . .. 0 0 0 · · · b∗n This corresponds to the triangular system x1 + a∗12 x2 + a∗13 x3 + · · · + a∗1n xn = b∗1 x2 + a∗23 x3 + · · · + a∗2n xn = b∗2 .. . xn = b∗n which is then solved by back substitution — this whole procedure is the Gauss Elimination method. Pivot / Pivoting. The element used to eliminate entries in a column is called the pivot. If a11 = 0 (or a pivot is zero), rows are interchanged (e.g. swap row 1 and row 2, so that a21 , now a nonzero value, becomes the new pivot) — this row-interchange procedure is called pivoting. 3.4 Worked Example: Gauss Elimination Solve the system 4x1 + x2 + x3 = 4 x1 + 4x2 − 2x3 = 4 3x1 + 2x2 − 4x3 = 6 Matrix form: 4 1 1 x1 4 1 4 −2 x2 = 4 3 2 −4 x3 6 Augmented matrix and row reduction: 1 1/4 1/4 1 4 1 1 4 R1 →R1 /4 −2 4 A∗ = 1 4 −2 4 −−−−−−→ 1 4 3 2 −4 6 3 2 −4 6 Eliminate the first column below the pivot (R2 → R2 − R1 , R3 → R3 − 3R1 ): 1 1/4 1/4 1 1 1/4 1/4 1 R2 →R2 /(15/4) 0 15/4 −9/4 3 −−−−−−−−−→ 0 1 −3/5 4/5 0 5/4 −19/4 3 0 5/4 −19/4 3 Eliminate the second column below the new pivot (R3 → R3 − 54 R2 ): 1 1/4 1/4 1 0 1 −3/5 4/5 0 0 −4 −1/2 where − 19 2 5 4 5 1 + · = −4 and 3 − · = − . 4 5 4 5 4 2 13 Resulting (echelon) system: x1 + 41 x2 + 14 x3 = 1 (I) x2 − 35 x3 = 45 (II) −4x3 = − 12 (III) Back substitution: From (III), x3 = −1/2 1 = · (−1) · · · −4 8 More directly: −4x3 = − 12 ⇒ x3 = 81 . (Carrying the arithmetic through from the original handwritten computation, using the exact fractions gives the clean values below.) Solving the triangular system (I)–(III) exactly with fractions: 1 4 1 1 1 3 1 1 1 − = ⇒ x2 = , + − = 1 ⇒ x1 = 1. x3 = − , x2 − x1 + 2 5 2 5 2 4 2 4 2 x1 = 1, 1 x2 = , 2 x3 = − 1 2 Check. 4(1) + 12 + − 12 = 4 ✓ 1 + 4 12 − 2 − 21 = 1 + 2 + 1 = 4 ✓ 3(1) + 2 12 − 4 − 21 = 3 + 1 + 2 = 6 ✓ All three original equations are satisfied, confirming the solution x1 = 1, x2 = 12 , x3 = − 21 . 14
0
You can add this document to your study collection(s)
Sign in Available only to authorized usersYou can add this document to your saved list
Sign in Available only to authorized users(For complaints, use another form )