Lecture 20
18.086
Presentations
•
5 presentations: Thomas, James, Kathleen,
Mukarram, Matthew
•
all presentations on Thursday, May 12 during lecture
•
Presentations: 10 minutes + 2 mins Q&A
•
project grade (50% of total grade): based on
presentation and report
SVD for matrices with noise
see Mathematica
t=
Example
Regularized Linear Least Squares Problems.
% Compute A
t = 10.^(0:-1:-10)’;
A = [ ones(size(t)) t t.^2 t.^3 t.^4 t.^5];
1.0000
0.1000
0.0100
0.0010
0.0001
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
% compute exact data
xex = ones(6,1); bex = A*xex;
% data perturbation of 0.1%
deltab = 0.001*(0.5-rand(size(bex))).*bex;
b
= bex+deltab;
% compute SVD of A
[U,S,V] = svd(A); sigma = diag(S);
for i = 0:7 % solve regularized LLS for different lambda
lambda(i+1) = 10^(-i)
xlambda = V * (sigma.*(U’*b) ./ (sigma.^2 + lambda(i+1)))
err(i+1) = norm(xlambda - xex);
end
loglog(lambda,err,’*’);
- 11x_{\lambda}||_2’); xlabel(’\l
D. Leykekhman
- MARN 5898 Parameter estimation in marine ylabel(’||x^{ex}
sciences
Linear Least Squares –
Linear Least Squares Problem.
larized Error
Linear Least
Squaresinverse
Problem.
of SVD
rIkx
x
k
for
di↵erent
values
of
(loglog-scale)
ex
2
The error kxex x k2 for di↵erent values of (loglog-scale):
Here is optimal λ!
But how to
choose if
xex is
unknown?
Let’s see now this works for the previous Matlab example.
Optimal choice of λ
The error kxex x k2 for di↵erent
values of (log-log-scale)
kekhman - MARN 5898 Parameter estimation in marine sciences
The residual kAx
bk2 and
k bk2 for di↵erent values of
(log-log- scale)
Linear Least Squares
–
14
Motivation: Why Inverse Problems?
A large-scale example, coming from a collaboration with
Università degli Studi di Napoli “Federico II” in Naples.
Inverse problems
From measurements of the magnetic field above Vesuvius, we
• Examples: Image denoising/deblurring, scattering/Xray
determine the activity inside the volcano.
imaging (reconstruction) etc.
Measurements
on the surface
)
Reconstruction
inside the volcano
C. Hansen, Discrete
Inverse
Problems: Insight and Algorithms, SIAM2
02906 – DIP – Chapter 1 and 2:P.Introduction
and Integral
Equations
some “interior” properties using “exterior” measurements.
Inverse problems
Problem
Inverse
Known
One of these is known
'
Input
&
™
°
°
$
)
but with
errors
@
R
@
System
%
'
) Output
&
02906 – DIP – Chapter 1 and 2: Introduction and Integral Equations
$
%
4
Inverse Problems: Examples
A quite generic formulation:
Z
input £ system d≠ = output
P. ≠C. Hansen, Discrete Inverse Problems: Insight and Algorithms, SIAM
02906 – DIP – Chapter 1 and 2: Introduction and Integral Equations
4
Inverse problems
Inverse Problems: Examples
A quite generic formulation:
Z
input £ system d≠ = output
≠
Image restoration
scenery ! lens ! image
Tomography
X-ray source ! object ! damping
Seismology
seismic wave ! layers ! reflections
P. C. Hansen, Discrete Inverse Problems: Insight and Algorithms, SIAM
Put P0 dollars at time t = 0 into a 401K with instantane
Outline
rate r (t).Computing Derivatives
The Interest Rate Problem
Put
P
dollars
at
time
t
=
0
into
a
401K
with
instantaneous
r
0
Discrete
Data
Why Derivatives Are Hard (And What To Do About It)
Failure
The Gravity Problem
rate r (t).
Forward Problem: Compute P(t) fromOutline
P0 and r (t). Th
The Interest Rate Pro
A More Interesting Problem
Computing Derivatives
Discrete Data
solving
the
DE
Why Derivatives Are Hard (And What To Do About It)
Failure
•
The
Gravity
Problem
Interest
rate
problem:
Put
P0
USD
into
account
at
Forward Problem: Compute P(t)
0 from P0 and r (t). This m
P
(t)
=
r
(t)P(t)
variable
interest rate
r(t).
Estimating
the Interest Rate
solving
the DE
0
The
solution
is
Forward problem:
P
(t)
=
r
(t)P(t)
Outline
The Interest Rate Problem
Computing Derivatives
Inverse
Problem:
Estimate
r
(t)
from
P(t).
This
means
finding
Discrete Data
✓
◆
Z
atives Are Hard (And What To Do About It)
t
Failure
The Gravity Problem
rThe
(t) from
the DE
solution
is
Suppose
P(t) =wePknow
(s) dstk =. k t, k =
solution:
0 expP(t) at rtimes
0
ting the Interest Rate P (t)
= r (t)P(t).
✓Znearest
◆ of course. We can
0penny
rounded
to the
t
P(t)
=
P
exp
r
(s)
ds
.
0
P(t
P(tk 1 )
Inverse problem: Estimate r(t) from P(t)
k+1 )
Example: Differentiation
0
P
(t
)
⇡
0
k
The solution is just
ose we know P(t) at times tk r=(t)
k =
t, P
k 0=
0,
1,
2,
.
.
.,
(t)/P(t).
Solution:
0 (t)/P(t) we get
From
r
(t)
=
P
ed to the nearest penny of course. We can estimate
Kurt Bryan
2 t
Inverse Problems 3: Why Di↵erentia
P(tk+1 ) P(tk
P(tk+1 ) P(tk 1 )
But it’s
as it looks... we get r (tk ) ⇡
withnot
P (tas
⇡
k ) simple
2 tP(tk )
2 t
0
Kurt Bryan
1)
.
Inverse Problems 3: Why Di↵erentiation is H
rounded to the nearest penny of course. We can e
Outline
Computing Derivatives
Why Derivatives Are Hard (And What To Do About It)
The Gravity Problem
P(tk+1 ) P(tk
P (tk ) ⇡
The Interest Rate Problem
2 t
Discrete Data
0
1)
Example:
Differentiation
Example
Failure
From r (t) = P 0 (t)/P(t) we get
Ex:
P(t
P(tk 1 )
k+1 )
Suppose r (t) = 0.04(3 2 cos(2t) + t/3) on r0(t )t ⇡
 5,
with
.
r(t)=0.04 (3-2cos(2t)+t/3)
k
P(tk+1 ) P(tk 1 )
2thetP(tk )
P(0) = 100. If we use r (tk ) ⇡
with
t
=
0.5
2 tP(tk )
result is
Δt=0.5:
Kurt Bryan
Kurt Bryan
Inverse Problems 3: Why
Inverse Problems 3: Why Di↵erentiation is Harder than Integration
rounded to the nearest penny of course. We can e
Outline
Computing Derivatives
Why Derivatives Are Hard (And What To Do About It)
The Gravity Problem
P(tk+1 ) P(tk
P (tk ) ⇡
The Interest Rate Problem
2 t
Discrete Data
0
1)
Example:
Differentiation
Example
Failure
From r (t) = P 0 (t)/P(t) we get
Ex: r(t)=0.04 (3-2cos(2t)+t/3)
With
t = 0.05 the result is better:
P(tk+1 ) P(tk
r (tk ) ⇡
2 tP(tk )
1)
.
Δt=0.05:
Kurt Bryan
Inverse Problems 3: Why
rounded to the nearest penny of course. We can e
P(tk+1 ) P(tk
P (tk ) ⇡
2 t
0
1)
Example: Differentiation
Outline
Computing Derivatives
Why Derivatives Are Hard (And What To Do About It)
The Gravity Problem
The Interest Rate Problem
Discrete0Data
Failure
From r (t) = P (t)/P(t) we get
Example
Ex: r(t)=0.04 (3-2cos(2t)+t/3)
But with
P(tk+1 ) P(tk
r (tk ) ⇡
2 tP(tk )
1)
.
t = 0.005 we get
Δt=0.005:
Kurt Bryan
Inverse Problems 3: Why
rounded to the nearest penny of course. We can e
P(tk+1 ) P(tk
P (tk ) ⇡
2 t
0
1)
Example: Differentiation
Outline
Computing Derivatives
Why Derivatives Are Hard (And What To Do About It)
The Gravity Problem
The Interest Rate Problem
0 Data
Discrete
Failure
From r (t) = P (t)/P(t) we get
Example
Ex: r(t)=0.04 (3-2cos(2t)+t/3)
And
t = 0.0005 yields
P(tk+1 ) P(tk
r (tk ) ⇡
2 tP(tk )
1)
.
Δt=0.0005:
Kurt Bryan
Kurt Bryan
Inverse Problems 3: Why
Inverse Problems 3: Why Di↵erentiation is Harder than Integration
The Interest Rate Problem
Discrete Data
Failure
Computing Derivatives
Why Derivatives Are Hard (And What To Do About It)
The Gravity Problem
We don’t
really know P(tk ), but P(tk ) rounded to the nearest
What’s
Wrong?
Example: Differentiation
cent, i.e., we know
P̃(tk ) = P(tk ) + ✏k
We don’t really know P(tk ), but P(tk ) rounded to the nearest
where
|✏k | we
 0.005
Wedollars.
don’t know P(t), but have noise in
cent, Problem:
i.e.,
know
0 (t ) is then
Our estimate
of
P
measurementk P̃(tk ) = P(tk ) + ✏k
where |✏k |  0.005 dollars.
P̃(tk+1 )
0
P̃(tk
P (tk ) ⇡
2 t
P(tk+1 ) P(tk
=
2{zt
|
better as
t!0
Kurt Bryan
Kurt Bryan
1)
1)
}
+
✏k+1 ✏k 1
| 2{zt }
may blow up as
.
t!0
Inverse Problems 3: Why Di↵erentiation is Harder than Integration
Inverse Problems 3: Why Di↵erentiation is Harder than Integration
The Gravity Problem
An Improvement
Example: Differentiation
Good
Idea: Add to Q a term that (from the minimization
Outline
Solution:Optimization
Add Approach
regularization! Search for di, i=0,..,n-1
Computing Derivatives
And What To perspective)
Do About It)
Tikhonov
Regularization
penalizes
highly Outline
oscillatory values for the dk :
The Gravity
Problem that
such
Computing Derivatives
Why Derivatives Are Hard (And What To Do About It)
The Gravity Problem
Q(d0Example
, . . . , dn 1 ) =
n 1✓
X
dk
Optimization Approach
Tikhonov Regularization
Outline
Computing Derivatives
Why Derivatives Are Hard (And What To Do About It)
The Gravity Problem
Pfk+1 Pfk
◆2
Example
t
f (t) = t + sin(3t) on 0  t  1 sampled at 100
k=0
+
n 2✓
X
↵
dk+1
2
|
k=0
◆2
Optimization Approach
Tikhonov Regularization
dk
t
{z
magnitude 0.01 added to each sample. Here’s the
regularization
term
estimate of f 0 . The result with ↵ = 10 4
2
The result with ↵ = 10
is minimized!
TheP(t)
parameter
↵ is called
regularization
parameter.
Weget:
can
For
= t + sin(3t)
and the
noise
of magnitude
0.001, we
adjust it.
Kurt Bryan
.
}
Inverse Problems 3: Why Di↵erentiation is Harder than Integr
Hubble telescope
•
When first brought to space, Hubble’s mirror had serious malfunctions:
•
=> Huge interest in image restoration…!
Hubble telescope
•
This is what one can get out of a blurry picture:
A(X) = B,
m×n
X, B ∈ R
(real matrices)
otation, a reflection, a shift operator, etc (all of th
tible, in principle), but there are “ugly” operators,
Image
de-blurring
⎛
⎞
⎜
⎜
⎜
A⎜
⎜
⎝
⎟
⎟
⎟
⎟=
⎟
⎠
Naive solution of an equation with the blurring operator
Think of image
as aoperator
matrix with
pixelweentries.
Blurring
described
a linear
Since the
is linear
can rewrite
theisproblem
as aby
system
g operator.
operator A acting on image: Af = b
(f: orig. image, b: blurred image
of linear algebraic equations
g operator
is
linear
and
in
purely
mathematical
sen
Supposeand
we solve
knowit:
A. True
Naivex solution:
Invert
A to and
find naive
original
imageAf:−1b:
and blured
b image,
solution
ut there are fundamental difficulties with the real pr
ns of this inverse problem.
Ax = b,
x = true image
A ∈ Rnm×nm,
x, b ∈ Rnm
b = blurred, noisy image
,
x = inverse solution
,
,
Image blurring
Image blurring is described by a convolution:
b(~x) = (a ⇤ f )(~x) :=
b = Af
Z
a(~s)f (~x
⌦
2
~s)d s
kernel
blurring is typically described by a Gaussian kernel:
!
s2y
s2x
a(~s) = C exp
2 x2
2 y2
In the discrete (image) case, the integral becomes a double sum:
Af =
wx
X
wy
X
adiscr (sx , sy )f (sx
sx = wx sy = w y
adiscr: discretization of a(s), finite support 2wx * 2wy
w x , sy
wy )
Back to the naive solution
Image de-blurring
There are fundamental difficulties with solving the linear problem:
• Action of the blurring operator A is realized by convolution with very
smooth 2D Gauß function, the operator has a smoothing property.
• The (example) right-hand side B is represented by smooth function.
So why is image deblurring by inverting A not working? −1
• While solving the linear system, i.e. evaluation A (B), we invert a
smoothing operator,
it onkernel.
a smooth function, and we want to
Blurring smoothens
image by apply
a smooth
obtain an “Peaking”:
image X which
typically
discontiuous
function.
Recall
Inverse operation:
Turn is
smooth
function
into (almost)
discontinuous
edges
verse problems ≡ Troubles!
The problem is typically very sensitive to small perturbations (e.g.,
ooth B, smooth A, nonsmooth solution).
But B is always corrupted by rounding errors (noise). Thus we have
,
X=
.
tem of equations B =
b = blurred, noisy image
Ax = b + eexact,
x = true image
where
e ∈ Rmn
is unknown,
d we want to find xtrue ≡ A−1b. The naive solution ilustrates the
Reason: Errors in measured image (b) magnify by naive inversion:
astophical impact of noise
x = inverse solution
Af = b + ✏
1
Aexact
(b + ✏) =
X naive ≡ A−1(B
+ E) =
.
tead of the solution we see the amplified noise only.
Condition number of A is very large, e.g. κ(A) ≈ 10100, i.e. A is
Image de-blurring
e effect of inverted noise — SVD
0
10
−10
10
−20
10
−30
10
Use regularization to cut-off small singular
values
Don’t panic!
We have a plan B!
σj
|u*jb |
e
*
|uj b |
s
10
Using
“clever” methods (regularization, spectral filtering) survive the
0
10
20
30
40
50
60
problem
j
−40
b = blurred, noisy image
B=
659 iterations
−→
X approx =
.
Main idea of regularization methods: Filter out the components cor-
sperm head (the direction of rolling however cannot be inferred from the head rotation). Although rolling is characterized by 3D beat dynamics, we find that most of the
times, the flagellar beat remains nearly planar, with a
plane of changing rotation as explained in the Main Text.
The 3D flagellum dynamics for 8 representative samples
of left- and right-turning sperm cells is shown in Fig. S4
(head-on view from the front).
A
B
Woolley [10] reports characteristic flagelloid curves for
• 2D microscopy:
mouse sperm in the rolling beat mode. These curves were
30
ntowards and be1.6
found for head-fixated sperm pointingn1.2
s S3 (R)
n is planar S1 (R)
1.2 S2 (L)
α
ing oriented perpendicular towards the coverslip. By fo20
1.4
s
d and studs
*
1the sperm head,
cussing at a focal plane
⇠ 15 µm behind
1
–9]. To this
1.2
y˜
Woolley could directly observe
the out-of-plane motion
10
jected tan-1
0.8
0.8
of a single point along
the flagellum (Fig.
S3A). Tracking
rresponding
x˜
0
0.8
0.6
0.6
lum length
0.6
en the ma0.4
0.4
z˜
the first 0.4
is
d from the
0.2
0.2
y˜
0.2
ote that by
0
0
tive values,0
0
10
20
0
10
20
0
10
20
me periodic
s (µm)
s (µm)
s (µm)
e frequency
are periodic
Time (s)
0
0.26
its a spuri1.6
1.6
1.6
disappears
S5 (L)
S6 (R)
S7 (L)
(Fig. S2C).
1.4
1.4
Fig. S3: (A) Flagelloid
curve for mouse 1.4
sperm, reproduced
3D flagellar beat reconstruction
of human sperm cells
Inten
0
S2
-0.8
S4 (L)
1.4
-0
S1
0
1
-5
0.6
0
5
10
20
n (μm)
0
-0
0.8
0.4
-0.8
0.2
0
0
s (µm)
0.3
0.2
0.1
0
1.6
S8 (R)
1.4
Helicity h
1.2
Time (s)
B
Time (s)
A
Time (s)
Time (s)
s (μm)
1.6
a)
b)
zp
c)
d)
x
Rayleigh-Sommerfeld
other relevant parameters
were λ = 0.505µ m, a = 0.1µ m, the pa
and nm = 1.33
respectively, and t
ium refractive
indices n p = 1.55
backscattering
(spherical
scatterer)
c)
µ m.f)d)Based on
e x, yb)plane, 10e) pixels/
g) this simulated data, we rec
W
(c) using
150determines position above
Guy phase
shift:
ciated(a)with the particle
the Rayleigh-Sommerfeld
scheme.
(f)
or
below
plane
The other
were
λ=
0.505focal
µ m,briefly,
a = 0.1µ m,
particle and
ribed
in relevant
detail parameters
elsewhere
[11,
12],
but
wetherecover
thesurround
electr
respectively, and the sampling freque
medium refractive indices n p = 1.55 and nm = 1.33100
of the10Rayleigh-Sommerfeld
propagator
data, we reconstruct the light fi
inthe
theuse
x, y plane,
pixels/µ m. Based on this simulated
y
-z'
-z'
x'
x'
x
x
Gouy phase shift
zp
y
y
x
x
-z'
-z'-z'
-z'
x'
x'
x' x'
Intensity [Arb.]
y focal
associated
with the particle
using
theg)Rayleigh-Sommerfeld
scheme. This′ method has b
f) particle.
Fig.e)
3.plane
Example data fromfocal
a single
Scale bars
represent 2 µ m in all 50
cases. (a)
exp(ikr
)
∂
1
′
′
′
Vertical slice
through
the
center
of
an
image
stack
created
by
physically
translating
the
described
in
detail
elsewhere
[11,
12],
but
briefly,
we
recover
the
electric
field
at a height
plane
x
h(x
,
y
,
z
)
=
sample (see text). (b) Image of a particle located at z ≈ 9µ m (‘downstream’ of the focal
′ field:r ′
with plane
the inuse
of the Rayleigh-Sommerfeld
propagator
of scattered
the illumination
path). (c) Optical fieldPropagation
reconstructed from
the previous
panel.
The
2
π
∂
z
0
(d) below the bottom of the image. (d) Intensity
(b) plane (z′ = 0) would be located
hologram
focal
central spot is azimuthally symmetric about the z′ -axis and
gradient g(x′ , z′ ) < 0. The dark
′ ′ ′
plane dimensions. (e,f)
givesy the particle location in all three
The companion images to (b,c), for
-z'
-z'
′ -50
′
gradient
a particle located at z ≈ −9µ m (‘upstream’ in the illumination path). (g) Intensity
x'
x'
x
g(x′ , z′ ) > 0. The particle location is specified by a maximum of g(x′ , z′ ) for those scatterers
′2 (c)1/2
with z′ < 0. The′2intensity′2in Panels
and (f) have been rescaled for clarity, but the shape
le data of
from
a single
particle.
Scale bars represent 2 µ m in all cases. (a)
the optical
field
is unchanged.
-100
′)
exp(ikr
∂
1
′2
′2
′2
1/2
,y ,z ) =
(x + y + z ) h(x. Note
the
2πuse
∂ z ofrprimed
′
r
re =
coordinates to in
nstructed
(as
opposed
to
physical)
volume.
We
can
reconstruct
the
where r = (x + y + z ) . Note
the use light
of primed
coordinates
to indicate
a position in
Scattered
at any
position x’,y’,z’
as convolution
mage)
at ofany
height
above
the
hologram
plane
by
the
convolution
hrough the center
an
stack created
by
physically
translating
the We can
reconstructed
(asimage
opposed
to
physical)
volume.
reconstruct
the
field (and from th
with scattered wave at the plane z’=0:
reconstructed
xt). (b) Image of a particle located at z ≈ 9µ m (‘downstream’ of the focal
scanned
the
image)
atOptical
anydiameter
height
the
by
the particles
convolution
umination
path). (c)
field
reconstructed
from
thehologram
previous
panel. plane
The
manufacturer-specified
535above
nm). This
sample
was
allowed
to
dry,
leaving
′ was
′Intensity
′ refilled-150
′ ′
(e)ofThe
be located
below
the chamber.
bottom
the chamber
image. (d)
eadhered
(z′ = 0)towould
the bottom
surface
of the
then
with
immersion
-0.8
0
0.4
s ′ -0.4
′ sabout
′ ′ the z′ -axis and
′ ′
◦
)
<
0.
The
dark
central
spot
is
azimuthally
symmetric
was, y
done
they,
effects
aberrations
oil (Nikon ‘type A’, nd = 1.515 at 23 C). This
, zto) minimize
= Es (x,
0) ⊗ofh(x
, y , z ). z [µm]
Es (x
le locationwhen
in allimaging
three dimensions.
(e,f) The
toany
(b,c),
for reflections between
incurred
through water
intocompanion
glass
and images
to S.
limit
multiple
images:
Lee,
D.G.
Grier, Optics Express 15, 1505-1512 (2007)
m (‘upstream’
thesample
illumination
path).
(g)stepper
Intensity
gradient
ed
z ≈ −9µand
theatparticles
the wall ofinthe
chamber.
The
motor
used to position the objec-
z
x
′
E (x , y , z ) = E (x, y, 0) ⊗ h(x , y , z ).0.8