Look Closer to Inverse
Problem
Qianqian Fang
Thanks to: Paul M. Meaney, Keith D. Paulsen, Dun Li,
Margaret Fanning, Sarah A. Pendergrass
and all other friends
RIP 2003
Research In Progress Presentation 2003
Outline
Linearization
Numerical Methods
Ax = y
Singular
Matrices
What is MATRIX?
Inverse problem
Solving inverse problem
Singular
Value
Decomposition
Multi-Freq Recon.
Improve the solutions
Conclusions
Research In Progress Presentation 2003
Time-Domain Recon.
Numerical Methods and
linearization
 Modern Numerical Techniques
Modern Numerical
Techniques
Nonlinear methods
NN, GA, SA, Monte-Calo
Numerical
Linear Relation
Ax=b
Model
Diff. Equ./Integral Equ.
Mathematical
Reality
Infinitely Complicated,
Accuracy 
Efficiency 
Dynamically Changing,
Research In Progress Presentation 2003
Noisy and Interrelated
What is MATRIX
Unfortunatel
y no one can
be told what
the matrix is,
you have to
see it for
yourself
from movie The Matrix,
WarnerBros,1999,
Research In Progress Presentation 2003
What is MATRIX
 Linear Transform
 Map from one space to another
 Stretch, Rotations, Projections
 Structural Information- on grid
æa11 K
çç
çç M O
çç
çça
è m1 L
a1n ÷
ö
÷
÷
M÷
÷
÷
÷
amn ÷
÷
øm ´ n
÷
X Î Rn
Y Î Rm
 Simple data structure (comparing with
list/tree/object etc)
 But not that simple (comparing with single
variable)
40
30
20
Research In Progress Presentation 2003
10
10
20
30
40
Geometric Interpretations
 2X2 matrix->Map 2D image to 2D image
æ2 1 ÷
öæx i ö
çç
÷çç ÷
=
÷
çç 3 - 1 ÷
÷
y
ç
÷
è
øè i ø
2
æxˆ i ÷
ö
çç ÷
ççyˆ ÷
è i÷
ø
1
0
-4
-3
-2
-1
0
1
2
3
-1
-2
-3
Research In Progress Presentation 2003
æ2 1 ö
æx i ö
÷
çç
÷çç ÷
=
÷
çç 3 1.5 ÷
÷
y
ç
÷è i ø
è
ø
æxˆ i ö
çç ÷
÷
ççyˆ ÷
÷
è iø
eig(A)= {3.5, 0}
Geometric Interpretations 2
1. Stretching
 3D matrix
2. Rotation
3. Projection
æ2
öæx ö
1
3 ÷
çç
çç i ÷
÷
÷
÷
çç
÷
֍
ççy i ÷
=
÷
çç- 1 2 - 2 ÷
÷
÷
÷
ç
÷
÷ç ÷
çç
÷
ø
çè 0 - 1 2 ÷
÷
øçè z i ÷
÷
æx i¢ö
çç ÷
÷
çç ÷
÷
ççy i¢÷
÷
÷
çç ÷
÷
ççè z i¢÷
÷
ø
÷
æ2 1 3 ÷
ö
çç
÷
÷
çç
÷
÷
1
2
2
çç
÷
÷
÷
çç
÷
çè- 1 2 - 2 ÷
ø
÷
Research In Progress Presentation 2003
•
•
•
Diagonal Matrix
Orthogonal Matrix
Projection Matrix
Geometric Interpretations 3
 N-Dimensional matrix-> Hyper-ellipsoid
if $s i = 0
Orthogonal Basis
s N uˆ N
Singular Matrix
s 1û1
s 4û4
s 2û2
s 3û3
Ellipsoid will collapse
To a “thin” hyperplane
Information along
“Singular” direction
Will be wiped out
After the transform
Information losing
Research In Progress Presentation 2003
Inverse Problem
 Which is inverse? Which is forward?
The latter discovered?
Forward?
X domain
Transformation
The more difficult one?
Y domain
Inverse?
 Information
 Sensitivity
Integration operator has a smoothing nature
òWinput ´
system
d
W=
output
1442 443
kernel
Research In Progress Presentation 2003
Inversion: Information Perspective
 From damaged information to get all.
 From limited # of projected images to recover
the full object
Multi-view scheme:
?
-- From the website of
"PHOTOGRAPHY
CLUBS in Singapore"
 Projections -> Related to singular matrix
Research In Progress Presentation 2003
SVD-the way to degeneration
 Singular Value Decomposition
Am ´ n = U m ´ m S m ´ nV nT´ n
Am ´ n =
U m ´ n S n ´ nV nT´ n
A
 What this means
 Good/Bad, how good/how bad
A
2 miles
4 miles
Research In Progress Presentation 2003

U
Am ´ n = U m ´ m S m ´ mV mT ´ n
U
VT

VT
Thin SVD
economy
One step further…
 SVE- Singular Value Expansion
Am ´ n = [u1, u 2 , L , u n ]diag(s 1, s 2 , L , s n ) [v1, v2 , L , vn ]T
å
=
s i u i vTi
i
Principal Planes
 Solving Ax=y
x =
å
i
ui , y T
vi
si
x = A - 1y
 Given the knowledge of SVD and noise, we
master the fate of the inverse problem
Research In Progress Presentation 2003
Principal Planes of a matrix
A
s 1u1v1T
s 2u 2vT2
s 3u 3vT3
s 4u 4vT4
Research In Progress Presentation 2003
Singular Values
 -Diagonal Matrix {i}
és 1
ê
ê
ê
ê
ê
ê
êêë 0
s2
 Ranking of importance,
 Ranking of ill-posedness
 How linearly dependent for equations
O
0 ù
ú
ú
ú
ú
ú
ú
s n úú
û
[A] is an ill-posed matrix
-> very thin hyper-ellipsoid
-> decreasing spectrum
[A] is an orthogonal matrix
-> Hyper-sphere
-> Perfectly linearly independent
Research In Progress Presentation 2003
[A] is a singular matrix
-> degenerated ellipsoid
-> 0 singular value
Regularization, the saver
 Eliminating the bad effect of small
singular values, keep major information
 A filter, filter out high frequency noise
AND high freq. useful information
Truncated SVD(T-SVD)
M
x =
å
i= 1
Tikhonov regularization (standard)
solve Al+ x I ,l = y
ui , y T
vi
si
M is t he t runcat ion level
Truncation level
s i2
si
s i2 + l 2
Al+ = [(AT A + l 2I )- 1 AT
singular values becomes:

Research In Progress Presentation 2003
- 1
]
L-curve: A useful tool
Under-smoothed
solution
log x
“best solution”
2
: Regularization parameter increasing
log A x - b 2
Research In Progress Presentation 2003
Over-smoothed
solution
† See reference [1]
Can we do better?
 Adding more linearly independent
measurement
 More antenna/more receivers
 Same antenna, but more frequency points
Research In Progress Presentation 2003
Multiple-Frequency Reconstruction
Project the object with different
Wavelength microwave
æ 2
¶ E (w )
¶ E (w ) ö
çç w2 mgQ gS e ' ( w1 )g 2R 1
w22 mgS e "( w1 )g R2 1 ÷
÷
çç
¶ kR ( w1 )
¶ kI ( w1 ) ÷
÷
÷
çç
÷
¶ E I ( w1 )
¶ E I ( w1 ) ÷
÷
çç 2
2
÷
w2 mgS e "( w1 )g 2
÷
çç w2 mgQ gS e ' ( w1 )g 2
¶ kR ( w1 )
¶ kI ( w1 ) ÷
÷
çç
÷
÷
çç
÷
÷
¶
E
(
w
)
¶
E
(
w
)
R
2
R
2
2
2
÷
çç w mgQ gS ( w )g
w
m
g
S
(
w
)
g
÷
e'
2
2
e" 2
2
2
çç 2
÷ æ1
¶ kR ( w2 )
¶ kI ( w2 ) ÷
ö÷
÷
çç
÷
÷gçççQ D e ' ÷
÷
çç 2
¶ E I ( w2 )
¶ E I ( w2 ) ÷
÷=
֍
2
÷
÷
çç w2 mgQ gS e ' ( w2 )g 2
w2 mgS e "( w2 )g 2
÷ çç D e '' ÷
÷
¶ kR ( w2 )
¶ kI ( w2 ) ÷
çç
è
ø
÷
÷
÷
çç
÷
......
......
÷
çç
÷
÷
÷
çç
÷
çç w2 mgQ gS ( w )g¶ E R ( wM ) w2 mgS ( w )g¶ E R ( wM ) ÷
÷
÷
e'
M
2
e" M
2
2
çç M
÷
¶
k
(
w
)
¶
k
(
w
)
÷
R
M
I
M
çç
÷
÷
÷
ççç w2 mgQ gS ( w )g¶ E I ( wM ) w2 mgS ( w )g¶ E I ( wM ) ÷
÷
÷
e'
M
2
e" M
2
2
ççè M
¶ kR ( wM )
¶ kI ( wM ) ÷
ø
÷
çç
÷
÷
÷
çç
÷
÷
ç
÷
Low frequency component stabilize the reconstruction
High frequency component brings up details
Research In Progress Presentation 2003
æD E R ( w1 ) ö÷
çç
÷
÷
ççD E ( w ) ÷
÷
I
1
çç
÷
÷
÷
çç
÷
D
E
(
w
)
R
2 ÷
÷
ççç
÷
÷
çç D E ( w ) ÷
÷
I
2
÷
÷
ççç
÷
÷
çç...
÷
÷
ççç D E ( w ) ÷
÷
÷
R
M
çç
÷
÷
÷
çç
÷
÷
ø
çèç D E I ( wM ) ÷
÷
ç
÷
÷
35.71 35.71
0.892857
28.57 28.57
0.714286
21.43 21.43
0.535714
14.29 14.29
0.357143
7.14 7.14
0.178571
0 0.00 0.00
0.892857
35.71
0.892857
0.714286
28.57
0.714286
0.535714
21.43
0.535714
0.357143
14.29
0.357143
0.178571
0.1785717.14
0
0.00
0
0.892857
0.714286
0.535714
0.357143
0.178571
0
I
100.00100.00
2.5
92.86 92.86
2.32143
85.71 85.71
2.14286
78.57 78.57
1.96429
71.43 71.43
1.78571
64.29 64.29
1.60714
57.14 57.14
1.42857
50.00 50.00
1.25
42.86 42.86
1.07143
35.71 35.71
0.892857
28.57 28.57
0.714286
21.43 21.43
0.535714
14.29 14.29
0.357143
7.14 7.14
0.178571
0 0.00 0.00
I
I
2.5
100.00
2.5
2.32143
92.86
2.32143
2.14286
85.71
2.14286
1.96429
78.57
1.96429
1.78571
71.43
1.78571
1.60714
64.29
1.60714
1.42857
1.4285757.14
50.00
1.25 1.25
1.07143
42.86
1.07143
0.892857
35.71
0.892857
0.714286
28.57
0.714286
0.535714
21.43
0.535714
0.357143
14.29
0.357143
0.178571
0.1785717.14
0 0.00
0
I
2.5
2.32143
2.14286
1.96429
1.78571
1.60714
1.42857
1.25
1.07143
0.892857
0.714286
0.535714
0.357143
0.178571
0
Reconstruction results I:
Simulations
 High contrast(1:6)/Large object
True object
Result from
single freq. recon
Result from
3 freq. recon
Cross cut of
reconstruction
Reconstructed Permitivity using
Multi-Frequency-Point Method
90
80
70
I
100.00100.00
2.5
92.86 92.86
2.32143
85.71 85.71
2.14286
78.57 78.57
1.96429
71.43 71.43
1.78571
64.29 64.29
1.60714
57.14 57.14
1.42857
50.00 50.00
1.25
42.86 42.86
1.07143
35.71 35.71
0.892857
28.57 28.57
0.714286
21.43 21.43
0.535714
14.29 14.29
0.357143
7.14 7.14
0.178571
0 0.00 0.00
I
I
2.5
100.00
2.5
2.32143
92.86
2.32143
2.14286
2.1428685.71
1.96429
78.57
1.96429
1.78571
71.43
1.78571
1.60714
64.29
1.60714
1.42857
57.14
1.42857
50.00
1.25 1.25
1.07143
42.86
1.07143
0.892857
35.71
0.892857
0.714286
28.57
0.714286
0.535714
21.43
0.535714
0.357143
14.29
0.357143
0.178571
0.1785717.14
0 0.00
0
I
2.5
60
2.32143
2.14286
1.96429 50
1.78571
1.60714
1.42857 40
1.25
1.07143
0.89285730
0.714286
0.535714
0.35714320
0.178571
0
10
0
Large object
Background
inclusion
0
10
20
30
40
50
60
70
80
90
100
60
70
80
90
100
Reconstructed Permitivity using
Multi-Frequency-Point Method
1.8
1.6
100.00100.00
92.86 92.86
85.71 85.71
78.57 78.57
71.43 71.43
64.29 64.29
57.14 57.14
50.00 50.00
42.86 42.86
35.71 35.71
28.57 28.57
21.43 21.43
14.29 14.29
7.14 7.14
0.00 0.00
I
I
2.5
2.5
2.32143
2.32143
2.14286
2.14286
1.96429
1.96429
1.78571
1.78571
1.60714
1.60714
1.42857
1.42857
1.25 1.25
1.07143
1.07143
0.892857
0.892857
0.714286
0.714286
0.535714
0.535714
0.357143
0.357143
0.178571
0.178571
0
0
1.4
1.2
1
0.8
0.6
0.4
0.2
0
Research In Progress Presentation 2003
0
10
20
30
40
50
14.29
7.14
0.00
0.357143
14.29 14.29
0.178571
7.14
7.14
0.00
0 0.00
0.357143
0.35714314.29
0.1785710.178571
7.14
00.00
0
0.357143
0.178571
0
I
2.5
100.00 100.00
2.32143
92.86 92.86
2.14286
85.71 85.71
1.96429
78.57 78.57
1.78571
71.43 71.43
1.60714
64.29 64.29
1.42857
57.14 57.14
1.25
50.00 50.00
1.07143
42.86 42.86
0.892857
35.71 35.71
0.714286
28.57 28.57
0.535714
21.43 21.43
0.357143
14.29 14.29
0.178571
7.14 7.14
0 0.00 0.00
I
I
2.5
2.5
100.00
2.321432.32143
92.86
2.142862.14286
85.71
1.964291.96429
78.57
1.785711.78571
71.43
1.607141.60714
64.29
1.428571.42857
57.14
1.25
1.25
50.00
1.071431.07143
42.86
0.892857
0.892857
35.71
0.714286
0.714286
28.57
0.535714
0.535714
21.43
0.357143
0.357143
14.29
0.178571
0.178571
7.14
0 0.00
0
I
2.5
2.32143
2.14286
1.96429
1.78571
1.60714
1.42857
1.25
1.07143
0.892857
0.714286
0.535714
0.357143
0.178571
0
100.00
92.86
85.71
78.57
71.43
64.29
57.14
50.00
42.86
35.71
28.57
21.43
14.29
7.14
0.00
I
2.5
100.00 100.00
2.32143
92.86 92.86
2.14286
85.71 85.71
1.96429
78.57 78.57
1.78571
71.43 71.43
1.60714
64.29 64.29
1.42857
57.14 57.14
1.25
50.00 50.00
1.07143
42.86 42.86
0.892857
35.71 35.71
0.714286
28.57 28.57
0.535714
21.43 21.43
0.357143
14.29 14.29
0.178571
7.14
7.14
0 0.00
0.00
I
I
2.5
2.5
100.00
2.321432.32143
92.86
2.142862.14286
85.71
1.964291.96429
78.57
1.785711.78571
71.43
1.607141.60714
64.29
1.428571.42857
57.14
1.25
1.25
50.00
1.071431.07143
42.86
0.892857
0.892857
35.71
0.714286
0.714286
28.57
0.535714
0.53571421.43
0.357143
0.357143
14.29
0.178571
0.178571
7.14
0 0.00
0
I
2.5
2.32143
2.14286
1.96429
1.78571
1.60714
1.42857
1.25
1.07143
0.892857
0.714286
0.535714
0.357143
0.178571
0
100.00
92.86
85.71
78.57
71.43
64.29
57.14
50.00
42.86
35.71
28.57
21.43
14.29
7.14
0.00
I
2.5
100.00
2.32143 92.86
2.14286 85.71
1.96429 78.57
1.78571 71.43
1.60714 64.29
1.42857 57.14
1.25
50.00
1.07143 42.86
0.89285735.71
0.71428628.57
0.53571421.43
0.35714314.29
0.178571 7.14
0
0.00
I
2.5
100.00
2.32143
92.86
2.14286
85.71
1.96429
78.57
1.78571
71.43
1.60714
64.29
1.42857
57.14
1.25
50.00
1.07143
42.86
0.892857
35.71
0.714286
28.57
0.535714
21.43
0.357143
14.29
0.178571
7.14
0 0.00
I
2.5
2.32143
2.14286
1.96429
1.78571
1.60714
1.42857
1.25
1.07143
0.892857
0.714286
0.535714
0.357143
0.178571
0
100.00
92.86
85.71
78.57
71.43
64.29
57.14
50.00
42.86
35.71
28.57
21.43
14.29
7.14
0.00
Reconstruction results I:
Phantom
 Saline Background/Agar Phantom with
inclusion
Results from
Single frequency
Reconstructor
At 900MHz
Research In Progress Presentation 2003
100.00
92.86
85.71
78.57
I
2.5
2.32143
2.14286
1.96429
Results from
Multi-frequency
Reconstructor
500/700/900MHz
Time-Domain solver
 A vehicle to get full-spectrum by one-run
A pulse signal is transmitted
From source
A distorted pulse is received
At receivers
Interacting with
inhomogeneity
FFT
0.6
Full Spectrum
Response
retrieved
0.5
0.4
0.3
0.2
0.1
Research In Progress Presentation 2003
0.5
1
1.5
2
2.5
3
Animations
Microwave scattered by object
Object
Research In Progress Presentation 2003
Source:
Diff Gaussian Pulse
Conclusions
 SVD gives us a scale to measure the Difficulties




for solving inverse problem
SVD gives us a microscope that shows the very
details of how each components affects the
inversion
Incorporate noise and a priori information, SVD
provide the complete information (in linear sense)
Regularization is necessary to by suppressing
noise
Difficulties can be released by adding more
linearly independent measurements
Research In Progress Presentation 2003
Key Ideas
 Decomposing a complex problem into some
building blocks, they are simple, invariant to
input, but addable, which can create certain
degree of complexity, but manageable.
 Find out the unchanged part from changing,
that are the rules we are looking for
 It is impossible to get something from nothing
Research In Progress Presentation 2003
References
 Rank-Deficient and Discrete IllPosed Problems, Per Christian Hansen,
SIAM 1998
 Regularization Methods for Ill-Posed
Problems, Morozov
 Matrix Computations, G. Golub, 1989
 Linear Algebra and it’s applications,
G. Strang
Research In Progress Presentation 2003
x =
å
i
ui , y + dy T
vi
si
Questions?
A
U

VT
Research In Progress Presentation 2003
Eigen-values vs. Singular value
Eigen-vectors
Directions:
Invariant of
rotations
2
1
0
-4
-3
-2
-1
0
1
2
3
-1
-2
-3
Research In Progress Presentation 2003
Singular-vectors
Directions:
Maximum span
Outline details








Numerical Methods and linearization
What is MATRIX? Geometric interpretations
Inverse Problem
Singular value decomposition and
implementations in inverse problem
Solving inverse problem
Improve the solution, can we?
Multiple-Frequency Reconstruction & TimeDomain Reconstruction
Conclusions
Research In Progress Presentation 2003
Right Singular Vectors
 Eigen-modes for solution
1.25
1
0.75
0.5
0.25
1
0.5
1
2
3
4
5
6
0.5
1
1
0.25
1
2
3
4
5
6
 Building blocks for solutions,
 if the solution is a image, vi are components of the image
 Less variant respect to different y=> a property of the
system
Research In Progress Presentation 2003
Left Singular Vectors
 A group of “basic RHS’s”-> source mode
 Arbitrary RHS y can be decomposed with
this basis
Research In Progress Presentation 2003
Noise
 Always Noise
 Small perturbation for RHS
 Ax=y
ui , y + dy T
 y=y+y
x =
å
i
si
vi
† Modified from coca-cola’s patch
Research In Progress Presentation 2003