Introduction to Simulation - Lecture 11
Newton-Method Case Study – Simulating
An Image Smoother
Jacob White
Thanks to Deepak Ramaswamy, Andrew Lumsdaine,
Jaime Peraire, Michal Rewienski, and Karen Veroy
Outline
• Image Segmentation Example
– Large nonlinear system of equations
– Formulation? Continuation? Linear Solver?
• Newton Iterative Methods
– Accuracy Theorem
– Matrix-free idea
• Gershgorin Circle Theorem
– Lends insight on iterative method convergence
• Arc-Length Continuation
SMA-HPC ©2003 MIT
Simple Smoother
Smoothed
Output
SMA-HPC ©2003 MIT
Circuit Diagram
Image
Input
Nonlinear
Smoother
SMA-HPC ©2003 MIT
Circuit Diagram
Nonlinear
Smoother
i v
SMA-HPC ©2003 MIT
Nonlinear Resistor
Constitutive Equation
1 e
Dv
E J D v 2
Nonlinear
Smoother
Nonlinear Resistor
Constitutive Equation
Current
Voltage
SMA-HPC ©2003 MIT
Nonlinear
Smoother
SMA-HPC ©2003 MIT
Nonlinear Resistor
Constitutive Equation
Varying Beta
Questions
• What Equation Formulation?
– Node-Branch or Nodal?
• What Newton Method?
– Standard, Damped, or Continuation?
– What kind of Continuation?
• What Linear Solver?
– Sparse Gaussian Elimination or Krylov?
– Will Krylov converge rapidly?
– How will formulation, Newton choices
interact?
SMA-HPC ©2003 MIT
Basic Algorithm
Newton-Iterative
Method
Nested Iteration
x 0 = Initial Guess, k 0
Repeat {
Compute F x k , J F x k
Solve (Using GCR)
J F x k 'x k 1 F x k
x k 1
k
for 'x k 1
x k 'x k 1
k 1
} Until
'x k 1 , F x k 1
small enough
How Accurately Should We Solve with GCR?
SMA-HPC ©2003 MIT
Newton-Iterative
Method
Basic Algorithm
Solve Accuracy Required
After l steps of GCR
k 1,l
'
x
Newton
delta from
l GCR Steps
dE
J F xk
If
a)
b)
c)
J F1 x k
F xk k ,l
rN
GCR
Residual
Inverse is bounded JF x JF y d A x y
Derivative is Lipschitz Cont 2
k ,l
k
r d C F x More accurate near convergence Then
The Newton-Iterative Method Converges Quadratically
SMA-HPC ©2003 MIT
Newton-Iterative
Method
Basic Algorithm
Convergence Proof
By definition of the Newton-Iterative Method
1
JF x
F x k r k ,l
Approximate Newton Direction
Multidimensional Mean Value Lemma
A
2
F x F y J F y x y d x y
2
x
k 1
k
k
x Combining
F x
k 1
F x J x k
SMA-HPC ©2003 MIT
k
F
1
F x r ªJ x
¬« F
k
k
k ,l
º d A J xk
»¼ 2 F
1
F x r k
k ,l
2
Newton-Iterative
Method
Basic Algorithm
Convergence Proof Cont.
Canceling the Jacobian and its inverse on the previous slide
F x
k 1
F x F x r
k
k
k ,l
A
d J F xk
2
1
F x r k
Combining terms and using the triangle inequality
F x k 1
A
d J F xk
2
1
F x r k
k ,l
2
r k ,l
Using the Jacobian Bound and the triangle inequality
F x
SMA-HPC ©2003 MIT
k 1
d
E 2A
2
F x
k
2
§ E 2 A r k ,l
¨1 ¨
2
©
·
¸ r k ,l
¸
¹
k ,l
2
Newton-Iterative
Method
Basic Algorithm
Convergence Proof Cont. II
Using the bound on the iterative solver error
F x
k 1
2
d
E A
2
F x
k
§ E 2A F x k
2
¨¨1 2
¨
©
2
·
¸ C F xk
¸
¸
¹
2
And combining terms yields
F x
k 1
SMA-HPC ©2003 MIT
2
§ 2 §
2
k
· ·
E
F
x
A
¨E A ¨
¸ C ¸ F xk
d¨
¨1 ¸ ¸
2
2
¨
¸ ¸
¨
©
¹ ¹
©
Easily Bounded
2
Newton-Iterative
Method
Matrix-Free Idea
Consider Applying GCR to The Newton Iterate Equation
J F x k 'x k 1
F xk
At each iteration GCR forms a matrix-vector product
JF x
k
l
p |
1
H
F x k H pl F x k
It is possible to use Newton-GCR without Jacobians!
Need to Select a good H
SMA-HPC ©2003 MIT
Gerschgorin Circle
Theorem
Given a matrix
M
Theorem Statement
ª m1,1 " m1, N º
«
»
%
# »
« #
« mN ,1 " mN , N »
¬
¼
For each eigenvalue of M there exists an i, 1< i < N such that
O mi ,i d ¦ mi , j
j zi
We say that the eigenvalues are contained in the
union of the Gerschgorin circles
SMA-HPC ©2003 MIT
Gerschgorin Circle
Theorem
Theorem Statement
Picture of Gerschgorin
Im O ith circle
radius
¦m
Eigenvalues are in the
union of all the disks
i, j
j zi
Re O ith circle
center
SMA-HPC ©2003 MIT
mi ,i
Gerschgorin Circle
Theorem
1
3
2
N
SMA-HPC ©2003 MIT
Grounded Resistor Line
Nodal Matix
4
ª2.1 1
º
« 1 2.1
»
«
»
«
% 1»
«
»
1 2.1¼
¬
M
N 1
Nodal
Equation
Form
N
Gerschgorin Circle
Theorem
1
2
N
SMA-HPC ©2003 MIT
3
Resistor Line
Nodal Matix
4
N 1
ª 2 1
º
«1 2
»
«
» Nodal
«
% 1» Equation
«
»
Form
1 2 ¼
¬
M
N
Basic Concepts
Continuation Schemes
General Setting
Solve F x O , O 0 where:
a) F x 0 , 0 0 is easy to solve
b) F x 1 ,1
F x
Starts the continuation
Ends the continuation
c) x O is sufficiently smooth
Hard to insure!
x O Dissallowed
0
SMA-HPC ©2003 MIT
1
O
Continuation Schemes
Solve F x 0 , 0 , x O prev GO =0.01, O GO
Basic Concepts
Template Algorithm
x 0
While O 1 {
x 0 O x O prev Try to Solve F x O , O 0 with Newton
}
If Newton Converged
x O prev x O , O O GO , GO
Else
1
GO
GO , O O prev GO
2
SMA-HPC ©2003 MIT
2GO
Jacobian Altering Scheme
Continuation Schemes
Description
F x O , O O F x O 1 O x O Observations
O =0 F x 0 , 0 x 0 0
wF x 0 , 0 wx
I
O =1 F x 1 ,1 F x 1 wF x 0 , 0 wF x 1 wx
wx
SMA-HPC ©2003 MIT
Problem is easy to solve and
Jacobian definitely nonsingular.
Back to the original problem
and original Jacobian
Continuation Schemes
Solve F x 0 , 0 , x O prev GO =0.01, O GO
Jacobian Altering Scheme
Basic Algorithm
x 0
While O 1 {
x 0 O x O prev ?
Try to Solve F x O , O 0 with Newton
}
If Newton Converged
x O prev x O , O O GO , GO
Else
1
GO
GO , O O prev GO
2
SMA-HPC ©2003 MIT
2GO
Continuation Schemes
Jacobian Altering Scheme
Still can have problems
xO
Must switch back to
increasing lambda
Arc-length
steps
0
1
lambda steps
SMA-HPC ©2003 MIT
Must switch from
increasing to
decreasing lambda
O
Continuation Schemes
Jacobian Altering Scheme
Arc-length Steps?
xO
arc-length |
Arc-length
steps
0
1
2
'x GO O
Must Solve For Lambda
F x, O 0
2
2
2
O
O
x
x
O
arc
prev prev 2
SMA-HPC ©2003 MIT
0
2
Jacobian Altering Scheme
Continuation Schemes
Arc-length steps by Newton
ª wF x k , O k
«
wx
«
«
k
2
x
x O prev «¬
T
ª
«
« Ok O
prev
«¬
SMA-HPC ©2003 MIT
k
k
wF x , O º
» ª x k 1 x k º
wO
» « k 1
k»
» ¬O O ¼
k
2 O O prev »
¼
x O k
k
F x ,O
2
x
k
prev
2
2
º
»
arc 2 »»
¼
Jacobian Altering Scheme
Continuation Schemes
Arc-length Turning point
x O
What happens here?
0
Upper left-hand
Block is singular
1
O
SMA-HPC ©2003 MIT
ª wF x k , O k
«
wx
«
«
k
x O prev 2
x
«¬
T
wF x k , O k º
»
wO
»
»
k
2 O O prev »
¼
Summary
• Image Segmentation Example
– Large nonlinear system of equations
– Examined issues in selecting numerical
methods
• Newton Iterative Methods
– Do not need to solve iteration equations exactly
• Gershgorin Circle Theorem
– Sometimes gives useful bounds on eigenvalues
• Arc-Length Continuation
SMA-HPC ©2003 MIT