Introduction to Simulation - Lecture 9
Multidimensional Newton Methods
Jacob White
Thanks to Deepak Ramaswamy, Jaime Peraire, Michal
Rewienski, and Karen Veroy
Outline
• Quick Review of 1-D Newton
– Convergence Testing
• Multidimensonal Newton Method
– Basic Algorithm
– Description of the Jacobian.
– Equation formulation.
• Multidimensional Convergence Properties
– Prove local convergence
– Improving convergence
Newton Idea
1-D Reminder
( )
Problem: Find x such that f x = 0
*
*
Use a Taylor Series Expansion
0
*
( )
f x = f ( x) +
∂f ( x )
∂x
(x
*
)
−x +
∂ f ( x)
If x is close to the exact solution
∂f ( x )
∂x
SMA-HPC ©2003 MIT
(
)
x* − x ≈ − f ( x )
2
∂x
2
(x
*
−x
)
2
Newton Algorithm
1-D Reminder
x 0 = Initial Guess, k = 0
Repeat {
( )
∂f x k
∂x
(x
k +1
)
( )
− xk = − f xk
k = k +1
} Until ?
x k +1 − x k < threshold ?
SMA-HPC ©2003 MIT
(
)
f x k +1 < threshold ?
1-D Reminder
Newton Algorithm
Algorithm Picture
SMA-HPC ©2003 MIT
Newton Algorithm
1-D Reminder
Convergence Checks
Need a "delta-x" check to avoid false convergence
f(x)
x
k +1
x
x
k +1
− x > ε xa + ε xr x
k
k
x
(
f x
SMA-HPC ©2003 MIT
k +1
)
< ε fa
k +1
*
X
Newton Algorithm
1-D Reminder
Convergence Checks
Also need an "f ( x ) " check to avoid false convergence
f(x)
(
f x
x*
x
SMA-HPC ©2003 MIT
− x < ε xa + ε xr x
k
)
> ε fa
X
x k +1 x k
k +1
k +1
k +1
Newton Algorithm
1-D Reminder
Local Convergence
Convergence Depends on a Good Initial Guess
f(x)
x1
x1
x
SMA-HPC ©2003 MIT
2
x
0
x
0
X
Multidimensional
Newton Method
F
l= x +y
2
FL
2
(lo − l )
F = EAc
= ε (lo − l )
lo
x
x
f x = F = ε (lo − l )
l
l
y
y
f y = F = ε (lo − l )
l
l
SMA-HPC ©2003 MIT
Example Problem
Strut and Joint
F (x) =
f x + FLx = 0
f y + FLy = 0
OR
x
ε (lo − l ) + FLx = 0
l
y
ε (lo − l ) + FLy = 0
l
Multidimensional
Newton Method
v1
i2 + v2b -
Example Problem
Nonlinear Resistors
v2
Nodal Analysis
i3
+
v3b
i1
+
v1b
-
Nonlinear
Resistors
i = g (v)
SMA-HPC ©2003 MIT
At Node 1: i1 + i2 = 0
⇒ g ( v1 ) + g ( v1 − v2 ) = 0
At Node 2: i3 − i2 = 0
⇒ g ( v3 ) − g ( v1 − v2 ) = 0
Two coupled
nonlinear equations
in two unknowns
Multidimensional
Newton Method
General Setting
( )
Problem: Find x such that F x = 0
*
x ∈
*
N
*
N
and F :
→
N
Use a Taylor Series Expansion
0
( )
F x = F ( x) + JF ( x)
*
(
Jacobian
Matrix
If x is close to the exact solution
(
)
J F ( x ) x* − x ≈ − F ( x )
SMA-HPC ©2003 MIT
)
x* − x + H .O.T .
Nodal Analysis
Multidimensional
Newton Method
x ∈
*
2
Strut and Joint
and F :
2
F
x
ε (lo − l ) + FLx = 0
l
y
ε (lo − l ) + FLy = 0
l
SMA-HPC ©2003 MIT
→
2
FL
⎡? ? ⎤
JF ( x) = ⎢
⎥
⎣? ? ⎦
Multidimensional
Newton Method
v1
i1
x* ∈
i2
b
v
+2
-
+
v2
i3
+
v1b
v3b
-
-
SMA-HPC ©2003 MIT
2
Nodal Analysis
Nonlinear Resistor
and F :
2
→
2
At Node 1: i1 + i2 = 0
⇒ F1 ( v ) = g ( v1 ) + g ( v1 − v2 ) = 0
At Node 2: i3 − i2 = 0
⇒ F2 ( v ) = g ( v3 ) − g ( v1 − v2 ) = 0
⎡? ? ⎤
JF ( x) = ⎢
⎥
⎣? ? ⎦
Multidimensional
Newton Method
Jacobian Matrix
J F ( x ) ∆x ≈ F ( x + ∆ x ) − F ( x )
⎡ ∂F1 ( x )
⎢
⎢ ∂x1
J F ( x ) ∆x ≡ ⎢
⎢
⎢ ∂FN ( x )
⎢ ∂x
1
⎣
SMA-HPC ©2003 MIT
∂F1 ( x ) ⎤
⎥
∂xN ⎥ ⎡ ∆x1 ⎤
⎥
⎥⎢
⎥
⎥⎢
∂FN ( x ) ⎥ ⎢⎣ ∆xN ⎥⎦
∂xN ⎥⎦
Multidimensional
Newton Method
Jacobian Matrix
Singular Case
Suppose J F ( x ) is singular?
⎡ ∂F1 ( x )
⎢
⎢ ∂x1
J F ( x ) ∆x = ⎢
⎢
⎢ ∂FN ( x )
⎢ ∂x
1
⎣
∂F1 ( x ) ⎤
⎥
∂xN ⎥ ⎡ ∆x1 ⎤
⎥=0
⎥⎢
⎥
⎥⎢
∂FN ( x ) ⎥ ⎢⎣ ∆xN ⎥⎦
∂xN ⎥⎦
What does it mean?
SMA-HPC ©2003 MIT
Multidimensional
Newton Method
Newton Algorithm
x 0 = Initial Guess, k = 0
Repeat {
( ) ( )
( x )( x − x ) = − F ( x )
Compute F x k , J F x k
Solve J F
k
k +1
k
k
for x k +1
k = k +1
} Until
SMA-HPC ©2003 MIT
x k +1 − x k ,
(
f x k +1
)
small enough
Computing the Jacobian
and the Function
Multidimensional
Newton Method
Consider the contribution of one nonlinear resistor
Connected between nodes n1 and n2
b
i + vb i b = g vb
n1
n2
( )
(
)
Summing currents at Node n1: Fn1 ( v ) = g vn1 − vn2 +…
(
)
Summing currents at Node n 2 : Fn2 ( v ) = − g vn1 − vn2 +…
Differenting at Node n1:
∂Fn1 ( v )
∂vn1
=
(
∂g vn1 − vn2
∂vn1
∂g
∂v
SMA-HPC ©2003 MIT
) +…
∂Fn1 ( v )
∂vn2
=
(
∂g vn1 − vn2
∂vn2
−
∂g
∂v
) +…
Computing the Jacobian
and the Function
Multidimensional
Newton Method
i
Stamping a
Resistor
n1
n2
⎡
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎣⎢
n1
n1
(
∂g vn1 − vn2
n2
)
−
∂v
−
(
∂g vn1 − vn2
)
(
∂g vn1 − vn2
∂v
(
∂g vn1 − vn2
∂v
SMA-HPC ©2003 MIT
vb
+
-
∂v
JF (v )
)
)
⎤
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎦⎥
n2
⎡
⎢
⎢ g vn1 − vn2
⎢
⎢
⎢− g v − v
n1
n2
⎢
⎢⎣
F (v )
(
)
(
)
⎤
⎥
⎥ n1
⎥
⎥
⎥n
⎥ 2
⎥⎦
Multidimensional More Complete Newton
Algorithm
Newton Method
x 0 = Initial Guess, k = 0
Repeat {
( )
( )
Compute F x k , J F x k
Zero J F and F
for each element
Compute element currents and derivatives
Sum currents to F , sum derivatives to J F
( )( x
Solve J F x k
k +1
)
( )
− xk = − F xk
for x k +1
k = k +1
} Until
SMA-HPC ©2003 MIT
x k +1 − x k ,
(
f x k +1
)
small enough
Multidimensional
Newton’s Method
Example: Heat Flow in
leaky bar
T (1)
T (0)
T1
vs = T(0)
TN
+
-
+
-
What is the Jacobian?
SMA-HPC ©2003 MIT
ih = k1T + k2T
vs = T(1)
3
Multidimensional
Newton Method
Multidimensional
Convergence Theorem
Theorem Statement
Main Theorem
If
( )
≤β
( Inverse is bounded )
a)
J F−1 x k
b)
JF ( x) − JF ( y) ≤
x− y
( Derivative is Lipschitz Cont )
Then Newton’s method converges given a sufficiently
close initial guess
SMA-HPC ©2003 MIT
Multidimensional
Newton Method
If J F ( x ) − J F ( y ) ≤
x− y
Multidimensional
Convergence Theorem
Key Lemma
( Derivative is Lipschitz Cont )
Then F ( x ) − F ( y ) − J F ( y )( x − y ) ≤
2
x− y
2
There is no multidimensional mean value theorem.
SMA-HPC ©2003 MIT
Multidimensional
Convergence Theorem
Multidimensional
Newton Method
Theorem Proof
By definition of the Newton Iteration and the assumed
bound on the inverse of the Jacobian
( ) ( )
x k +1 − x k = J F−1 x k F x k
( )
≤ β F xk
Again applying the Newton iteration definition
( )
(
)
(
x k +1 − x k ≤ β F x k − F x k −1 − J F x k −1
0
Finally using the Lemma
x
SMA-HPC ©2003 MIT
k +1
−x ≤
k
β
2
x −x
k
k −1 2
)(
x k − x k −1
)
Multidimensional
Convergence Theorem
Multidimensional
Newton Method
Theorem Proof Continued
Reorganizing the equation
x
k +1
⎛β
k
k −1 ⎞
x − x ⎟ x k − x k −1
−x ≤⎜
⎝ 2
⎠
k
⎛β
1
0 ⎞
If ⎜
x − x ⎟ ≤ γ <1
⎝ 2
⎠
∞
(
)
x k +1 − x k ≤ γ k ⇒ ∑ x k +1 − x k + x 0 converges
k =0
SMA-HPC ©2003 MIT
Non-converging
Case
1-D Picture
f(x)
x1
x
0
Must Somehow Limit the changes in X
SMA-HPC ©2003 MIT
X
Newton Method
with Limiting
Newton Algorithm
Newton Algorithm for Solving F ( x ) = 0
x = Initial Guess, k = 0
0
Repeat {
( ) ( )
( x ) ∆x = − F ( x )
+ limited ( ∆x )
Compute F x k , J F x k
Solve J F
x k +1 = x k
k = k +1
} Until
SMA-HPC ©2003 MIT
k
k +1
k
for ∆x k +1
k +1
(
∆x k +1 , F x k +1
)
small enough
Newton Method
with Limiting
Limiting Methods
• Direction Corrupting
(
)
limited ∆x k +1 =
i
∆xik +1 if ∆xik +1 < γ
(
limited ∆x k +1
γ sign ( ∆xik +1 ) otherwise
)
∆x k +1
γ
• NonCorrupting
(
)
limited ∆x k +1 = α∆x k +1
⎧⎪
γ
α = min ⎨1,
k +1
∆
x
⎪⎩
⎫⎪
⎬
⎪⎭
(
limited ∆x
k +1
γ
Heuristics, No Guarantee of Global Convergence
SMA-HPC ©2003 MIT
)
∆x k +1
Damped Newton
Scheme
Newton Method
with Limiting
General Damping Scheme
( )
( )
Solve J F x k ∆x k +1 = − F x k
for ∆x k +1
x k +1 = x k + α k ∆x k +1
Key Idea: Line Search
(
Pick α to minimize F x + α ∆x
k
(
F x + α ∆x
k
k
k +1
)
2
2
k
k
(
k +1
)
≡ F x + α ∆x
k
k
2
2
k +1
) F (x
T
k
+ α k ∆x k +1
Method Performs a one-dimensional search in
Newton Direction
SMA-HPC ©2003 MIT
)
Damped Newton
Newton Method
with Limiting
Convergence Theorem
If
( )
≤β
( Inverse is bounded )
a)
J F−1 x k
b)
JF ( x) − JF ( y) ≤
x− y
( Derivative is Lipschitz Cont )
Then
There exists a set of α k ' s ∈ ( 0,1] such that
(
)
(
F x k +1 = F x k + α k ∆x k +1
)
( )
< γ F xk
with γ <1
Every Step reduces F-- Global Convergence!
SMA-HPC ©2003 MIT
Damped Newton
Newton Method
with Limiting
Nested Iteration
x 0 = Initial Guess, k = 0
Repeat {
( ) ( )
Solve J ( x ) ∆x = − F ( x ) for ∆x
Find α ∈ ( 0,1] such that F ( x + α ∆x )
Compute F x k , J F x k
k
k +1
k +1
k
F
k
k
x k +1 = x k + α k ∆x k +1
k = k +1
} Until
(
∆x k +1 , F x k +1
)
k
k +1
is minimized
small enough
How can one find the damping coefficients?
SMA-HPC ©2003 MIT
Newton Method
with Limiting
Damped Newton
Singular Jacobian Problem
f(x)
x2
1
x
1
D
x
x
0
X
Damped Newton Methods “push” iterates to local minimums
Finds the points where Jacobian is Singular
SMA-HPC ©2003 MIT
Summary
• Quick Review of 1-D Newton
– Convergence Testing
• Multidimensonal Newton Method
–
–
–
–
Basic Algorithm
Description of the Jacobian.
Jacobian Construction.
Local Convergence Theorem
• Damped Newton Method
– Nested Algorithm with line search
– Global convergence IF Jacobian nonsingular