Nonlinear Algebraic Equations Example
Continuous Stirred Tank Reactor (CSTR).
Look for steady state concentrations & temperature.
In: N s spieces with concentrations ci
(in )
,
)
(in)
heat capacities c(in
and
temperature
T
p,i
Inside: N r reactions with stoichiometric coefficients σ α ,i
and reaction constants rα .
Out: N s spieces with concentrations ci
(c-s may be equal to zero),
heat capacities cp,i and temperature T.
November 2001
10.001 Introduction to Computer
Methods
(in )
Nonlinear Algebraic Equations Example
Nr
(F/ V) ci(in) − ci  + ∑σα ,i rα = 0
i = 1, 2,..., Ns
α =1
Mass balancefor spieces1, 2,..., Ns
Ns
Nr
i =1
α =1
(in)
(F/ V)∑ ci(in)c(in)
T
− ci cp,i T − ∑ ∆Hα rα = 0
p,i
Energy balance
Column of unknown variables :x = [c1 ,c2 ,...,cNs ,T]
T
November 2001
10.001 Introduction to Computer
Methods
Nonlinear Algebraic Equations Example
Each of the above equations may be written in the general form:
f i (x1 , x 2 ,..., x N ) = 0 :
In vector form:
f(x)=0
f1 (x1 , x 2 ,..., x N ) = 0
f1 (x) 
f (x) 

f(x)=  2
...



f
(x)
 N 
f 2 (x1 , x 2 ,..., x N ) = 0
...
f N (x1 , x 2 ,..., x N ) = 0
∧
Let x be the solution staisfying f(x)=0.
∧
We do not know x and take x[0] as initial guess.
November 2001
10.001 Introduction to Computer
Methods
Nonlinear Algebraic Equations
We need to form a sequence of estimates to the solution:
[1]
[2]
∧
[3]
x , x , x ,... that will hopehully converge to x .
Thus we want: lim x
m →∞
[m]
∧
=x
∧
lim x − x[m] = 0
m →∞
Unlike with linear equations, we can’t say much about
existence or uniqueness of solutions, even for
a single equation.
November 2001
10.001 Introduction to Computer
Methods
Single Nonlinear Equation
∧
We assume f(x) is infinitely differentiable at the solution x .
∧
Then f(x) may be Taylor expanded around x :
∧
∧
∧
1
1
2 ’’
f (x) = f (x) + (x − x)f (x) + (x − x) f (x) + (x − x)3 f ’’’(x) + ...
2
3!
Assume x
∧
[0]
f (x ) ≈ (x
[0]
’
∧
being close to x so that the series may be truncated:
[0]
∧
− x)f ’(x[0] ),
or
f (x[0] ) ≈ (x[0] − x[1] )f ’(x[0] )
[0]
f
(x
)
[1]
[0]
x = x − ’ [0] −
f (x )
− Newton ’s method
Example : f (x) = (x − 3)(x − 2)(x − 1) = x 3 − 6x 2 + 11x − 6
November 2001
10.001 Introduction to Computer
Methods
Single Nonlinear Equation
November 2001
10.001 Introduction to Computer
Methods
Single Nonlinear Equation
November 2001
10.001 Introduction to Computer
Methods
Single Nonlinear Equation
For f(x)=(x-3)(x-2)(x-1)=x3-6x2+11x+6=0, we se that
Newton’s method converges to the root at x=2 only if
1.6<x[0]<2.4.
We may look at the direction of the first step and see
why:
x[1]- x[0]=-f(x[0])/ f’(x[0]).
So, we have 2 “easy” roots: x=1, x=3 and a more
difficult one.
We may “factorize out” the roots we already know.
November 2001
10.001 Introduction to Computer
Methods
Single Nonlinear Equation
f (x)
Introduce g(x) =
(x − 3)(x −1)
 1
f (x)
f (x)
1 
g (x) =
−
+


(x − 3)(x −1) (x − 3)(x −1)  (x − 3) (x −1) 
’
’
We now use Newton’s method to find the roots of g(x):
[i+1]
x
g(x)
=x − ’
→ get x = 2.
g (x)
November 2001
[i]
10.001 Introduction to Computer
Methods
Systems of Nonlinear Equations
Let’s extend the method to multiple equations:
f1(x1,x2,x,…,xN)=0
f2(x1,x2,x,…,xN)=0 =>
f(x)=0
…..
Start from initial guess x[0]
fN(x1,x2,x,…,xN)=0
∧
∧
As before expand each equation at the solution x with f( x )=0:
∧
∧
∧
∂f i (x) 1
∂ fi (x)
+ ∑∑ (x j − x j )
fi (x) = fi (x) + ∑ (x j − x j )
(x k − x k ) + ...
∂x j
∂x j∂x k
2 j=1 k =1
j=1
∧
Assume x
[0]
N
N
November 2001
N
N
∧
2
∧
is close to x and discard quadratic terms:
∧
fi (x) ≈ ∑ (x j − x j )
j=1
∧
∧
∂f i (x)
∂x j
10.001 Introduction to Computer
Methods
Systems of Nonlinear Equations
∧
Let ’s define the Jacobian matrix J( x ) with the elements:
∧
∧
J ij (x) =
∂f i (x)
∂x j
Then our approximate expansion may be written as:
∧
 ∧

f i (x) ≈ ∑ J ij (x)(x j − x j ) =  J(x)(x − x) 

i
j=1
This gives us the linear system:
N
∧
∧
∧
∧
f (x) ≈ J(x)(x − x)
November 2001
10.001 Introduction to Computer
Methods
Systems of Nonlinear Equations
∧
 ∧

f (x ) = J(x )(x − x ) =  J(x)(x − x) 

i
Note, Jacobian is evaluated at the position of
[i]
[i]
[i +1]
[i]
an old iteration, not at an unknown solution
Defining ∆x [i] = x [i +1] − x [i] , we rewrite the equation as
J(x [i] ) ∆x [i] = −f (x [i] )
or just
J[i] ∆x [i] = −f [i]
The iterations are continued untill some convergence
criteria are met:
relative error f [i] ≤ δ rel f [0]
absolute error f [i] ≤ δ abs
November 2001
10.001 Introduction to Computer
Methods
Systems of Nonlinear Equations
Newton’s method does not always converge. If it does,
can we estimate the error?
Let our function f(x) be continuously differentiable in the
vicinity of x and the vector connecting x and x +p all lies
inside this vicinity.
∂f
J(x + sp) = T
∂x
x+sp
x+p
x + sp
1
f (x + p) = f (x) + ∫ J(x + sp) p ds
x
0
− use path integral along the line x - x+p, parametrized by s.
November 2001
10.001 Introduction to Computer
Methods
Systems of Nonlinear Equations
Add and subtract J(x)p to RHS:
1
f (x + p) = f (x) + J(x) p + ∫ [ J(x + sp) − J(x)] p ds
0
In Newton’s method we ignore the integral term and
choose p to estimate f(x+p). Thus the error in this case is:
1
∫ [ J(x + sp) − J(x)] p ds = f (x + p)
0
What is the upper bound on this error?
f(x) and J(x) are continuous:
J(x+sp)-J(x) → 0 as p → 0 for all 0 ≤ s ≤ 1.
November 2001
10.001 Introduction to Computer
Methods
Systems of Nonlinear Equations
Av
The norm of the matrix is defined as: A = max
,
v≠0
v
Ay
so for any y A ≥
, or Ay ≤ A y , therefore
y
1
∫ [ J(x + sp) − J(x) ] p ds
0
1
≤
∫ [ J(x + sp) − J(x) ] ds
p
0
The error goes down at least as fast as p because for
a continuous Jacobian J(x+sp)-J(x) → 0 as p → 0.
November 2001
10.001 Introduction to Computer
Methods
Systems of Nonlinear Equations
If we suggest that there exist some L>0 such that
J(y)-J(z) ≤ L y − z
or there is some upper bound on the "stretching" effect of J.
1
f (x + sp) =
∫ [ J(x + sp) − J(x) ]p ds
0
1
≤
∫ [ J(x + sp) − J(x) ] ds
0
1
and
∫ [ J(x + sp) − J(x) ] ds
≤ L s p , so in this case
0
1
f (x + sp) =
∫ [ J(x + sp) − J(x) ] pds ≤ ( L s p ) p
0
( )
≤ (L s ) p = O p
2
November 2001
10.001 Introduction to Computer
Methods
2
p
Systems of Nonlinear Equations
Thus if we are at distance p from the solution, our error
scales as ||p||2.
What about convergence? How the error scales with
the number of iterations?
The answer is:
∧ 2

x[i +1] − x = O  x[i] − x  - local quadratic convergence,


a very fast one! Works when you are close enough to
the solution.
∧
November 2001
10.001 Introduction to Computer
Methods
Systems of Nonlinear Equations
∧
x − x : 0.1 = 10−1
[i]
x
x
x
[i +1]
[i + 2]
[i + 3]
∧
− x : 0.01 = 10−2
∧
− x : 0.00001 = 10−4
∧
− x : 0.000000001 = 10−8
Works if we are close enough to an isolated
 ∧ 
(non-singular) solution: det  J( x )  ≠ 0.


November 2001
10.001 Introduction to Computer
Methods
Systems of Nonlinear Equations, Example
Let’s examine how Newton’s method
works for a simple system :
f1 = 3x + 4 x 2 − 145 = 0 ∧ 3 
x= 
2
3
f 2 = 4 x1 − x 2 + 28 = 0
4
3
1
2
The Jacobian is :
 ∂f1
 ∂x
J= 1
 ∂f 2
 ∂x1
November 2001
∂f1 
2


9 x1
∂x 2
=
∂f 2 
 8x 1
∂x 2 
8x 2 
2
− 3x 2 
10.001 Introduction to Computer
Methods
Systems of Nonlinear Equations, Example
At each step we have the following system
to solve :
( )
[i ] 2
[i ]


9 x1
8x 2
[i]
J =

[i ]
[i ] 2
− 3 x 2 
 8x1
3 x [i ] 3 + 4 x [i ] 2 − 145
1
2
[i ]


f =
4 x [i ] 2 − x [i ] 3 + 28 
2
 1

[i]
[i ]
[i ]
[ i +1]
[i ]
[i ]
= x + ∆x
J ∆x = −f
x
( )
( ) ( )
( ) ( )
November 2001
10.001 Introduction to Computer
Methods
Systems of Nonlinear Equations, Example
Let us examine performance of
Newton’s method with the convergence
criterion f = max{f1 , f 2 } < δ abs
δ bs = 10
November 2001
−10
10.001 Introduction to Computer
Methods
Newton’s Method
Newton’s method works well close to the solution,
but otherwise takes large erratic steps, shows poor
performance and reliability.
Let’s try to avoid such large steps: employ reduced
step Newton’s algorithm.
“Full” Newton’s step gives x[i+1]=x[i]+p[i]. We’ll use
only fraction of of the step: x[i+1]=x[i]+λi p[i], 0<λi<1.
November 2001
10.001 Introduction to Computer
Methods
Newton’s Method
How do we choose λi ?
Simplest way - weak line search:
- start with λi =2-m for m=0,1,2,…
As we reduce the value of λi bya a factor of 2 at
each step, we accept the first one that satisfies a
descent criterion:
[i ]
[i ]
[i ]
f x + λi p < f x
(
)
( )
It can be proven that if the Jacobian is not singular,
the correct solution will be found by the reduced
step Newton’s algorithm.
November 2001
10.001 Introduction to Computer
Methods