CONTENT
I. NUMERICAL METHODS FOR NONLINEAR EQUATIONS AND SYSTEMS
OF EQUATIONS
I.1. NEWTON'S METHOD FOR EQUATIONS
2-4
I.2. NEWTON'S METHOD FOR SYSTEMS OF EQUATIONS
4-8
II. NEWTON SEARCH FOR A MINIMUM
III. THE GOLDEN RATIO SEARCH FOR A MINIMUM
IV. NELDER-MEAD SEARCH FOR A MINIMUM
8-9
9-11
11-13
V. NUMERICAL METHODS FOR DIFFERENTIAL EQUATIONS AND SYSTEM OF
EQUATIONS
V. 1 RUNGE-KUTTA METHODS
13-18
V. 2 FINITE DIFFERENCE METHODS
18-21
VI. MATHEMATICAL NOTIONS
21-27
I. NUMERICAL METHODS FOR NONLINEAR EQUATIONS AND SYSTEMS
OF EQUATIONS
I.1. NEWTON'S METHOD FOR EQUATIONS
Here we try to approximate a root z of the equation f x   0 in such a way that we use
the tangential straight line of at different points. Starting from x 0  the left figure shows
a successful trial and the right figure shows an unsuccessful one:
So, x  k 1 approximation of z can be got by Taylor's series of the function f x  used
only the constant and linear term of it. Hence x  k 1 is the solution of the linear equation
 
 

f x k   f ' x k  x  x k   0 .
If this equation has unique solution for k  0, 1, 2,... , then
x k 1  x k  
 
 
f x k 
,
f ' x k 
and it is called the formula of Newton's method for the equation f x   0 .
THEOREM.
Let the function f x   0 be twice continuously differentiable on a, b . Assume that
there is a root of the equation f x   0 on the interval a, b and f ' x   0 f " x   0
x  a, b . Then Newton's iteration converges (starting from x 0   a , if
f ' x  * f " x   0 and from x 0   b , if f ' x  * f " x   0 ) to the unique solution on a, b
and
2
M k 1
x k 1  z 
x
 x k  ,
2m
where
0  m  f '  x  , M  f " x  , x  a, b .
NUMERICAL EXAMPLE
Determine the root of the equation
ln 3x  
1
0
x
by Newton's method and with error bound   5 * 10 5 .
(1) We can get a suitable interval (for which the conditions of our theorem are
satisfied) by graphical way and (or) tabling. If we write our equation into the form
ln 3 x  
1
x
and represent the function of left and right sides (roughly), then we get. a (rough)
estimate about the root:
We can see the equation has unique root (z) and it is on the interval (1/3, 2). Moreover,
by the table
z is on the interval (2/3, 1), too.
(2) Now let us examine the conditions concerning the convergence of the method
a) f '  x  
x 1
x2
2 
 0, f '"  x    3  0, x   , 1.
2
x
x
3 
2 
b) Since f ' x   0 and f '" x   0 x   , 1 , so on a small neighbourhood of z the
3 
graph of f x  is the following:
Hence x 0   a 
2
the right choice.
3
2 
 3 , 1 ,
f x  f 1  2  m. Furthermore, f x   0 and it is strictly monotone increasing on
c) Since
f x   0
and
f x 
is strictly monotone
decreasing on
2 
 3 , 1 (because of f'''(x)>0). Therefore, f x  is strictly monotone decreasing and
2
f x   f    9  M .
3
(3) The computation of the iterations and the estimation of the errors:
x 0  
2
3
(1)
x (1)  0.881827 , | x  z | 0.104161
( 2)
x ( 2 )  0.948421 , | x  z | 0.009978
( 3)
x ( 3)  0.952451 , | x  z | 0.000037
As | x (3)  z |  is fulfilled, x 3  can be considered a suitable approximation of the root.
(If we used more rought bounds for m and M then the value M/2m would be larger and
we might need more iterations to guarantee the same accuracy.)
I.2. NEWTON'S METHOD FOR SYSTEMS OF EQUATIONS
Let the nonlinear system of equations
f1 x1 , x 2 ,..., x n   0
f 2 x1 , x 2 ,..., x n   0

f n x1 , x 2 ,..., x n   0
be given. In shortly form:
f ( x)  0,
f : n  n .
Starting from the initial vector x ( 0 )  {x1( 0) , x 2( 0) ,, x n( 0) } make x (1) on such a way that the
functions f1 , f 2 ,, f n are made „linear" by the first (n + 1) terms of their Taylor series
at x ( 0 ) It means the surfaces are replaced by their tangential planes concerning x ( 0 ) .
Hence, we can get x (1) from the system of linear equations
 
 
f1 x 0   1 f1 x 0  x1  x1
 
 
0 
 ...   f x   x
n
 xn
n
 xn
0
n 1
0 

0 
f n x0   1 f n x0  x1  x1
 ...   f x  x
0
n n
0 
 0
 0
if there exists unique solution of it. Using matrices we can write
 f1




 f n
x  
0
.
.
x  
0
 
 1 f1 x 0 
 

 
 
0 
 1 f n x
 
 
...  n f1 x 0    x1  x10   0
  

.
.
   . .

 .

.
.
0    
0   
...  n f n x   xn  xn  0
 
Shortly
f ( x ( 0) )  J ( x ( 0) ) ( x  x ( 0) )  0 ,
where J is called Jacobian matrix of the function system f1 , f 2 , , f n . If there exists
 
unique solution (det J x0   0 ) then we can express x  x (1) from this equation with
using the inverse matrix of J x 0 
 
x 1 x 0J 1 
x 0f
x 0
.
So, the formula of Newton's method (written x ( k ) and x ( k 1) into the places of x ( 0 ) and
x (1) respectively)
x k1 x kJ 1 
x kf
x k
,
k 0, 1, 2, 
The following theorem gives a fundamental characterization of the method.
THEOREM
Suppose that each of the partial derivatives  i f j , i, j  1, 2,, n
is continuous on a
neighbourhood of a solution z and the matrix J (z ) is invertable. Then Newton's method is
convergent starting from sufficiently short distance. And if the partial derivates
 i  j f k i, j , k  1, 2,  , n are also continuous on the above mentioned neighbourhood then
the error decreases at least quadratically, that is
x k1zc x kz2 ,
where c is a positive constant.
REMARK
It is not practical to compute x k 1
by the formula
x( k 1)  x( k )  J 1 ( x( k ) ) f ( x( k ) )
(here we have to compute the inverse of a matrix of order n in every step), but it is
practical to solve the original system of linear equations for the unknowns
x k 1  x k  , and then we can get the values x k 1 immediately. Namely, solving
system of linear equations requires smaller work than the computing of the inverse
matrix.
NUMERICAL EXAMPLE.
Let a circle and a paraboloid be given by the formulas
x u
x  5 2 cos t
y  2 sin t
z 0
t    ,   , and
yv
z  u 2  v2 .
respectively. Determine the distance of the given curve and surface.
Geometrically we have to find the shortest way starting from a circle point ending at a
paraboloid point:
The square of the distance
For the extremum points it is a necessary condition that
D
 4 sin t (u  5)  4v cos t  0
t
D
 10  4 cos t  2u  4u 3  4uv 2  0
u
D
 4 sin t  2v  4vu2  4v 3  0.
v
This way we have a nonlinear system of equations. Now introduce the notations
x  {x1 , x2 , x3 } , where x1  t , x2  u, x3  v and try to solve the nonlinear system by Newton's
method starting from x ( 0)  {2, 2, 2} . We determine the vector e  x ( k )  x ( k 1) from the
linear system of equations
J ( x ( k ) ) ( x ( k )  x ( k 1) )  f ( x ( k ) ) k  0,1, 2, ,
again and again. For k  0 the solution of the linear system is
e  {0.895051, 0.613877 , 0.700131}
and
x 1  x 0   e  2.895051, 1.386123, 1.299869.
The next approximations are
x 2 2. 954787 , 1. 112591 , 0. 742890,
x 4 3. 131045 , 1. 019094 , 0. 042924,
x 3 3. 066840 , 1. 047144 , 0. 295795,
x(5)  {3.141308 , 1.000788 , 0.001324}, -
x( 6)  {3.141592 , 1.000001, 0.000002}.
In the practice the following stop-criteria are usually used:
a)
the iteration is stopped if | fi ( x1 , x2 ,, xn ) |   i , where   0 is a given positive
b)
number;
the iteration is stopped if the distance of the last two approximating vectors (in
some norm) is small enough.
If we use the criteria a) and   5 10 4 , then x 6  is the last iteration, since
 
 
 
f 1 x 6   2.2  107   , f 2 x 6   7.1  106   , f 3 x 6   4.1  106   .
II. NEWTON SEARCH FOR A MINIMUM
Definition (Gradient). Assume that
and has partial derivatives
, is the vector function
is a function of two variables,
and
. The gradient of
, denoted by
.
Definition (Jacobian Matrix). Assume that
variables,
, their Jacobian matrix
are functions of two
is
.
Definition (Hessian Matrix). Assume that
is a function of two variables,
, and has partial derivatives up to the order two. The Hessian matrix
is
defined as follows:
.
Lemma 1. For
functions
the Hessian matrix
, i. e.
is the Jacobian matrix for the two
.
Lemma 2. If the second order partial derivatives of
matrix
is symmetric.
are continuous then the Hessian
,
Outline of the Newton Method for Finding a Minimum
Start with the approximation
to the minimum point . Set
(i) Evaluate the gradient vector
.
and Hessian matrix
(ii) Compute the next point
.
(iii) Perform the termination test for minimization. Set
.
Repeat the process.
Example 3. Use the Newton search method to find the minimum of
.
Solution 3.
III. THE GOLDEN RATIO SEARCH FOR A MINIMUM
Definition (Unimodal Function) The function
exists a unique number
such that
is decreasing on
,
is increasing on
.
is unimodal on
, if there
and
Golden Ratio Search
If
is known to be unimodal on
, then it is possible to replace the interval
with a subinterval on which
takes on its minimum value. One approach is to select
two interior points
. This results in
. The condition that
is unimodal
guarantees that the function values
and
are less than
.
If
, then the minimum must occur in the subinterval
, and we replace b
with d and continue the search in the new subinterval
. If
, then the
minimum must occur in the subinterval
, and we replace a with c and continue the
search in the new subinterval
. These choices are shown in Figure 1 below.
If
, then squeeze from the right and use the
squeeze from the left and use the
new interval
and the four points
.
four points
.
If
new interval
, then
and the
Figure 1. The decision process for the golden ratio search.
The interior points c and d of the original interval
resulting intervals
, and
are symmetrical in
that
, and produces the two equations
(1)
and
(2)
where
, must be constructed so that the
. This requires
,
,
(to preserve the ordering
).
We want the value of r to remain constant on each subinterval. If r is chosen judicially
then only one new point e (shown in green in Figure 1) needs to be constructed for the next
iteration. Additionally, one of the old interior points (either c or d) will be used as an interior
point of the next subinterval, while the other interior point (d or c) will become an endpoint of
the next subinterval in the iteration process. Thus, for each iteration only one new point e will
have to be constructed and only one new function evaluation
, will have to be made. As
a consequence, this means that the value r must be chosen carefully to split the interval of
into subintervals which have the following ratios:
(3)
and
(4)
and
,
and
.
If
and only one new function evaluation is to be made in the interval
we must have
, then
.
Use the facts in (3) and (4) to rewrite this equation and then simplify.
,
,
,
.
Now the quadratic equation can be applied and we get
.
The value we seek is
if
and it is often referred to as the "golden ratio." Similarly,
, then it can be shown that
.
Example 1. Find the minimum of the unimodal function
interval
.
Solution 1.
on the
IV. NELDER-MEAD SEARCH FOR A MINIMUM
Nelder-Mead Method
The Nelder-Mead method is a simplex method for finding a local minimum of a function of
several variables. It's discovery is attributed to J. A. Nelder and R. Mead. For two variables, a
simplex is a triangle, and the method is a pattern search that compares function values at the
three vertices of a triangle. The worst vertex, where
is largest, is rejected and replaced
with a new vertex. A new triangle is formed and the search is continued. The process
generates a sequence of triangles (which might have different shapes), for which the function
values at the vertices get smaller and smaller. The size of the triangles is reduced and the
coordinates of the minimum point are found.
The algorithm is stated using the term simplex (a generalized triangle in n dimensions) and
will find the minimum of a function of n variables. It is effective and computationally
compact.
Initial Triangle
Let
be the function that is to be minimized. To start, we are given three vertices of a
triangle:
, for
. The function
is then evaluated at each of the
three points:
, for
. The subscripts are then reordered so that
. We use the notation
(1)
,
to help remember that
, and
.
is the best vertex,
is good (next to best), and
is the worst vertex.
Midpoint of the Good Side
The construction process uses the midpoint
by averaging the coordinates:
(2)
of the line segment joining
and
. It is found
.
Reflection Using the Point
The function decreases as we move along the side of the triangle from to , and it
decreases as we move along the side from to . Hence it is feasible that
takes on
smaller values at points that lie away from on the opposite side of the line between
and . We choose a test point that is obtained by “reflecting” the triangle through the side
. To determine , we first find the midpoint of the side
. Then draw the line segment
from to and call its length d. This last segment is extended a distance d through to
locate the point . The vector formula for is
(3)
.
Expansion Using the Point
If the function value at is smaller than the function value at , then we have moved in the
correct direction toward the minimum. Perhaps the minimum is just a bit farther than the point
. So we extend the line segment through and to the point . This forms an expanded
triangle
. The point is found by moving an additional distance d along the line joining
and . If the function value at is less than the function value at , then we have found a
better vertex than . The vector formula for is
(4)
.
Contraction Using the Point
If the function values at
and
are the same, another point must be tested. Perhaps the
function is smaller at , but we cannot replace with because we must have a triangle.
Consider the two midpoints and of the line segments
and
, respectively. The
point with the smaller function value is called , and the new triangle is
.
Note: The choice between and might seem inappropriate for the two-dimensional case,
but it is important in higher dimensions.
Shrink Toward
If the function value at is not less than the value at , the points and must be shrunk
toward . The point is replaced with , and is replaced with , which is the midpoint of
the line segment joining with .
Logical Decisions for Each Step
A computationally efficient algorithm should perform function evaluations only if needed. In
each step, a new vertex is found, which replaces . As soon as it is found, further
investigation is not needed, and the iteration step is completed. The logical details for twodimensional cases are given in the proof.
Example 1. Use the Nelder-Mead method to find the minimum of
.
Solution 1.
V. NUMERICAL METHODS FOR DIFFERENTIAL EQUATIONS AND SYSTEM OF
EQUATIONS
V. 1 RUNGE-KUTTA METHODS
Let the initial value problem
y  f ( x y) 
y (a)  A
be given and suppose that we would like to determine (approximating) values of the solution
function in the finite interval [ a b ] . (It is well known that this problem is correctly given if
f ( x y ) and f y ( x y ) are continuous in a large enough neighbourhood of the point (a A) .
The basic idea of the so-called one-step methods (including Runge-Kutta methods) is the
following: if we have a formula for determining a good approximating value to y (u  h) by
using the value u v  y (u ) , then starting from u  a and using the previous (approximating)
value at every step, we can give approximating values to the solution function y ( x) at the
points a  h a  2h … a  nh  b .
From the equality
y(u  h)  y(u )  y (u )h  O(h 2 )  v  hf (u v)  O(h 2 )
it seems reasonable to try with a formula
y(u  h)  v  hD(u v h)  O(h p )
which is the general form of the so-called one-step explicit formulas. (Of course we would
like to choose such a function D(u v h) that p is sufficiently large.) The speciality of
Runge-Kutta methods is that the function value hD (u v h) is given by several values of
hf (……) originated recursively.
In detail, determine members
k1  hf (u v)
k2  hf (u   2 h v   21k1 )
k3  hf (u  3h v   31k1   32 k2 )
k4  hf (u   4 h v   41k1   42 k2   43k3 )
ks  hf (u   s h v   s1k1  …  s  s 1k s 1 )
and using these values give an approximating value in form
y(u  h) v  1k1   2 k2  …  s ks .
We have a concrete formula after choosing the number s of members k j and determining the
coefficients 1   2   2   21 …  s  s 1 . When determining the coefficients we harmonize the
Taylor series belonging to u of the left side accurate value y (u  h) and the Taylor series
belonging to (u v) of the right side approximating value v  1k1  …  s ks to the largest
power of the step length h .
Look at three different cases:
a) Let s  1 . Then
k 1  hf (u v)
v  1k1 
y (u  h)
that is,
y(u  h)
v  1hf (u v)
The Taylor series belonging to u of the left side is:
y (u  h)  y (u )  y  (u )h 
1 
1
y ( )h 2  v  hf (u v)  y  ( )h 2 
2
2
the Taylor series belonging to (u v) of the right side is itself, so the Taylor series are
harmonized concerning h to power 1, if 1  1 . The obtained formula
k 1  hf (u v)
y (u  h)
v  k1 
is called Euler's formula. The so-called local error concerning u  h of this formula (which
gives the error after one step starting from an accurate value) is 12 y  ( )h 2 (where
  [ u u  h ] ) or in short form this local error is O(h2 )
b) Let s  2 . Then
k1  hf (u v)
k2  hf  u   2 h, v   21k1 
y(u  h) v  1k1   2 k2 
that is
y(u  h) v  1hf (u v)   2hf (u  2h v   21hf (u v)) 
The Taylor series belonging to u of the left side is
1
1
y (u  h)  v  hf (u v)  h 2  f x'  u , v   f y'  u , v   f  u , v    y    h 3 ,
2
6
where it was used that
y ( x)  f ( x y)  y ( x)  f x ( x y) 1  f y ( x y) y ( x)
The Taylor series belonging to (u v) of the right side is
v  1hf (u v)   2 h  f (u v)  f x (u v )  2 h  f y (u v ) 21hf (u v )  O (h 2 )  
The two Taylor series are harmonized concerning h to power 2, if
1   2  1
1
2
1
 2 21  .
2
 22 
So, we have three equations for four unknowns, and we can get infinitely many solutions. E.g.
1   2  1  2  2   21  1 is a possible choise and the formula is
k1  hf (u v)
k2  hf  u  h, v  k1 
y (u  h)
v
1
 k1  k2 
2
which is called Heun's formula (or trapezoidal
y(u  h)  (v  12 (k1  k2 )) is characterized by O(h3 ) here.
formula).
The
local
error
c) Let s  4 . Then we can originate 11 equations to the 13 unknowns by harmonizing the
Taylor series concerning h to power 4. (For harmonizing to power 5 the 13 parameters are
not enough.) A possible choise is
1   4 
1
1
1
  2  3 
  2  3 
 4  1 
6
3
2
 21   32 
1
  31   41   42  0   43  1
2
and the formula is
y (u  h)
1
v  (k1  2k2  2k3  k4 ) 
6
which
is
called
the
classical
Runge-Kutta
formula.
The
5
1
y(u  h)  (v  6 (k1  2k2  2k3  k4 )) can be characterized by O(h ) now.
We have already mentioned that each Runge-Kutta type formula has the form
y (u  h)
v  hD(u v h)
local
error
after substituting the expressions k1  k2 … ks into the last "equation" (see the general form of
one-step explicit methods). Hence, using such a formula, first we determine the
approximating value
z1  z0  hD( x0  z0  h)
concerning y( x0  h) starting from the accurate initial couple ( x0  z0 )  (a A) . Then we
determine the approximating value
z2  z1  hD( x1 z1 h)
concerning y ( x0  2h) starting from the approximating couple ( x1 z1 ) , etc.
So, we can give the process by the formula
zk 1  zk  hD( xk  zk  h) 
( x0  z0 )  (a A) 
xk 1  b .
DEFINITION
The quantity
y (u  h)  (v  hD(u v h))
where v  yu  , is called local error belonging to the place u  h . The quantity
y( xk  h)  zk 1
(where zk 1 is originated by the above mentioned formula) is called accumulated error
belonging to the place xk  h .
THEOREM
If f ( x y ) is smooth enough in a suitable neighbourhood of the point (a A) in the case of the
problem y  f ( x y)  y(a)  A , then for the Euler, Heun, and Runge-Kutta methods (which
have O(h 2 )  O(h3 )  O(h5 ) local errors, respectively) the accumulated error can be given by
O(h)  O(h2 )  O(h 4 ) in the finite interval [ a b ] .
NUMERICAL EXAMPLE
1. Given the differential equation
1 1 1
y  (
 )
4 x y
Determine the approximating values of that particular solution which undergoes the point
(1  1) by the classical Runge-Kutta formula to the interval 1  x  2 if h  025 .
The results:
For x  125
 
1
1
k  025  14 

 0117749
10125 10125 2 
1
1
k  025  14 

 0117950
10125 10117749 2 
1
k  025  14  1  10117950
  0111808
1025
k1  025  14 1  11  0125
1
2
3
4
1
y (125) 1  (0125  2  0117749  2  0117950  0111808)  1118035
6
y (15) 1224746
y (175) 1322876
y (2) 1414214
V. 2 FINITE DIFFERENCE METHODS
The basic idea of these methods is the following: we substitute the derivatives in the (ordinary
or partial) differential equation and in the initial or (and) boundary conditions by expressions
of approximating derivation formulas. In this way we can get an "approximating system of
equations" for the required function values. We illustrate the possibilities on two well-known
problems.
a) Let the boundary value problem
 y  q( x) y  r ( x) 
y (a)  A 
y(b)  B
be given, where a  x  b , q( x)  0  x  [ a b ] and q ( x) r ( x) are at least continuous on
[ a b ] . Take an equidistant partition on the interval [ a b ]
°
by the mesh-points xi  a  ih  i  01 2… n  1 .
If the solution function y ( x) is four times continuously differentiable on [ a b ] then by using
the values y( xi 1 ) y ( xi ) y ( xi 1 ) belonging to the places xi 1 xi  xi 1 we can get
y ( xi ) 
y ( xi 1 )  2 y ( xi )  y ( xi 1 ) h 2 ( IV )
 y (i ) 
h2
12
where i [ xi 1 xi 1 ]  i  1 2… n . Introducing the notation yi  y ( xi ) , substituting the
previous expression into the differential equation for i  1 2… n and arranging the equations
with respect to y1…yn , we can obtain the system of linear equations
for the required function values y1  y2 … yn . Introducing the notations








2 







2  q( x1 )h 2
1
1
2  q( x2 )h 2
1
1
2  q( x3 )h 2
1
C
h
1
1 2  q( xn 1 )h 2
1
y






















n 1 

n 
y1
y2
y
y

k














r ( x1 )  A  h 2 


r ( x2 )

r ( xn 1 )
r ( xn )  B  h
our linear system can be written in form






2



1
2  q( xn )h 2
 y ( IV ) (1 ) 
 ( IV )

y ( 2 ) 
2 
h

d  
12  ( IV )

 y ( n 1 ) 
 y ( IV ) ( ) 
n 

















Cy  k  d .
During the practical computation we cannot determine the elements of the vector d , and
therefore we omit this member (the linear system is truncated) and we determine the solution
vector z of the linear system
Cz  x ,
hoping that z lies near enough to y . (Both linear system have unique solution because the
matrix C is diagonally dominant due to q( x)  0  x  [ a b ] ).
THEOREM
If  y
( IV )
M (b  a)2 2
h.
( x)  M   x  [ a b ] then max  yi  zi 
1i  n
96
NUMERICAL EXAMPLES
1. Give approximating values to the solution function of the boundary value problem
y  xy  2  x3
y (1)  1  y(3)  9
If we use the former approximating formula concerning y  ( xi ) at the inner meshpoints
x2  15  x3  2  x4  25 , then
Respecting the boundary values z1  1  z5  9 and arranging the equations, we get the linear
system
2375 z2  z3  134375
z2  25 z3  z4  15
z3  2625 z4  1240625
This has the unique solution
z2  225
z3  4 
So , the solution:
z4  625 
y2
225
y3
4
y4
625 
Since the solution function is y  x 2 , therefore the used approximating formula is accurate,
i.e. y( xi ) = zi in this problem.
VI. MATHEMATICAL NOTIONS
1. DEFINITION
The set L  x, y, z,... is called linear vector space if the addition and the multiplication by
real number are defined on the set L and these operations have the following properties:
The elements of such a set L are called vectors.
Examples
a) Let
where ai , bi , ci , ... . are real numbers, that is, L is the set of such "objects" which have n
"coordinates". If the addition and multiplication by number are defined by the traditional
rules, that is,
then the mentioned properties can be proved easily. This linear space is usually denoted by
n .
b) Let the interval  a, b be given and denote L the set of the continuous functions on  a, b .
If the addition f  x   g  x  and the multiplication by number  f  x  are defined by the
traditional rules, then the required properties can be proved easily. This linear space is usually
denoted by  a, b .
2. DEFINITION
The elements x, y,..., w  L (where L is a vector space and the number of the given vectors is
finite) form a linearly independent system if
Examples
a) In the vector space  4 the vectors a  1,0,0,0  and b   0,0,1,0  from linearly
independent system, since the zero element
 0,0,0,0
can only be obtained in form
0 a  0b .
b) In the vector space
 a, b the vectors
x, x 2 x3 form linearly independent system, since the
zero element (the identically zero function) can only be obtained in form 0  x  0  x 2  0  x3 .
3. DEFINITION
The linearly independent system formed by the vectors x, y,..., w  L is a basis of L , if
arbitrary element of L can be obtained as a linear combination of the elements x, y,..., w ,
that is, in form
 x   y  ...   w .
Examples
a) In the linear space  3 the vectors 1,0,0 ,  0,1,0 ,  0,0,1 form a well-known basis.
Another possible basis is given e.g. by vectors  2,0,0 ,  0,1,0  ,  0,0,1 .
b) Denote P the set of the polynomials of degree at most 2. The traditional addition and
multiplication by number do not lead out from the set, P is a vector space.
The elements (functions, vectors) 1, x, x 2 form a basis of P , because they form a linearly
independent system and arbitrary at most quadratic polynomial can be obtained in form
 1    x   x 2 .
Another basis is given e.g. by elements 1, x  1, x 2 .
4. DEFINITION
If we can order real numbers
respectively in such a way that
x , y ,... to all elements x, y,... . of the linear space L
x  0,
x 0x 0
xy  x  y
x    x
then the vector space L is called normed space and the real number x is called the norm
of the vector x .
Examples
a) Define the quantities
for each element a  a1 , a 2 , ..., a n  of the linear space  n . These quantities define three
different norms which are called norm number 1, norm number 2 (or Euclidean norm) and
norm number infinity.
b) Denote M n the set of the real (square) matrices of order n . M n form a linear space with
the traditional addition and multiplication by number. The quantities
where   is the largest eigenvalue of the matrix AT A and
define three different norms. The norms belonging to the same notation (the same name) in a)
and b) are called compatible, because such pairs have also the properties
for any x   n , A  M n beside the properties appearing in the definition. By the latter
property we could have defined the above mentioned norms of the matrix A as the smallest
upper bound of the suitable vector norm Ax under the condition x  1 . In our lecture notes
we supposed compatible norms at using Ax every time.
c) In the vector space C a, b we can define norms e.g. by equalities
Here we mention that the distance of the elements x, y of a normed space (just like at
traditional case) we can measure by the quantity
x   y   x  y .
5. DEFINITION
If we can order a real number c to arbitrary two elements x, y of the linear space L in such a
way that
then the vector space L is called Euclidean space and the real number x, y  is called the
scalar product of the elements (vectors) x and y .
Examples
a) Let a  a1 , a 2 , ..., a n 
and b  b1 , b2 , ..., bn  be two arbitrary elements of  n Then the
quantity
ab  a1b1  a2 b2  ...  an bn
defines the traditional scalar product on  n .
b) If f x  and g x are two arbitrary elements of C a, b , then the quantity
defines a scalar product
6. DEFINITION
The matrix A of order n is called positive definite matrix if
x T Ax  0, x   n , x  0 .
7. DEFINITION
The matrix A of order n is called negative definite matrix if
x T Ax  0, x   n , x  0 .
8. DEFINITION
The matrix A of order n is called positive semidefinite matrix if
x T Ax  0, x   n .
9. DEFINITION
The matrix A of order n is called negative semidefinite matrix if
x T Ax  0, x   n .
10. Definition
Denote z the limit of the convergent sequence of real numbers x1 , x2 , x3 ,... .If there exist
numbers C1 ,C 2 such that for arbitrary k
where p  1 , then we say: the sequence x k converges to z in order p . If only the left
(right) side inequality is fulfilled then we say: the order of convergence is at most (at least)
p.
Examples
a) For p  2 (quadratic convergence), which often appears at numerical methods, we
illustrate the rapidity of convergence on a simple example. Suppose that C2  1 and the
distance of xk and z is 0.1. Then (by using the right side inequality):
which means a surprising rapidity comparing the traditional sequences.
b) If we use the vector sequence x1 , x 2 , x3 ,... xi   n  instead of the number sequence
x1 , x2 , x3 ,... xi   and we use a norm instead of absolute value, then the previous
definition can be repeat with the inequality
11. DEFINITION
Let f x, hx : a, b     and f x, hx  0, x  a, b. Then the equality
f x   hx
(where  can be say capital ordo ) means there exists positive constant C such that
f x  C hx ,
x  a, b .
Examples
a) Let z be the limit of the sequence x1 , x2 , x3 ,...
xi   . Then the equality
 
xk  z   q k , where 0  q  1 ,
means that the error of xk  Cq k , where C is independent on k . In other words, it means that
"the error is proportional to q k ". Such a type of decreasing the error we can see e.g. at powermethod for eigenvalues.
b) Let x k be the previous sequence. Then the equality (supposed it is true)
 
xk  z   h s
(where s  1 is a constant and h  hk   0 if k   ) means that "the error is proportional
to h s ”. Such a type of decreasing the error we can see e.g. at Runge-Kutta methods.
c) Sometimes we use the sign  for truncation of a polynomial. E.g. the equality
 
px   2 x 4   x 3 , where x  0 ,
means that
 
px   2 x 4  C x 3 ,
where C  0 is independent on x . Such an application of sign  can be found e.g. at
elimination methods .