Estimation Theoretical Image Restoration
by
Jean J. Dolne
Ph. D., Electrical Engineering
City University of New York, 1996
Submitted to the System Design and Management Program
In Partial Fulfillment of the Requirements for the Degree of
ARCHIVES
Master of Science in Engineering and Management
at the
MASSACHUSETTS
INSTITUTE
OF TECHNOLOGY
Massachusetts Institute of Technology
JUN 0 3 2009
September 2008
LIBRARIES
C 2008 Jean J. Dolne
All rights reserved
The author hereby grants to MIT permission to reproduce and to distribute publicly paper and electronic copies of
this thesis document in whole or in part in any medium now known or hereafter created.
Signature of Author .....
....
......................
o
System Design and Management Program
August 08, 2008
Certified by ....................
-i
..........
William T. Freeman
Profssor of Electrical Engineering & Computer Science
S
^
Thesis Advisor
Accepted by..............
.................
L/ --
Patrick Hale
Director
System Design and Management Program
Acknowledgement
This thesis is possible due to the sponsorship by The Boeing Company. I truly appreciate the
help of my thesis advisor, Prof. Freeman, in directing this work. The assistance of the
Engineering System Design department personnel has been truly wonderful.
Estimation Theoretical Image Restoration
by
Jean J. Dolne
Submitted to the System Design and Management Program on August 8, 2008
In Partial Fulfillment of the Requirements for the Degree of Master of Science in
Engineering and Management
Abstract
In this thesis, we have developed an extensive study to evaluate image restoration from a single
image, colored or monochromatic. Using a mixture of Gaussian and Poisson noise process, we
derived an objective function to estimate the unknown object and point spread function (psf)
parameters. We have found that, without constraint enforcement, this blind deconvolution
algorithm tended to converge to the trivial solution: delta function as the estimated psf and the
detected image as the estimated object. We were able to avoid this solution set by enforcing a
priori knowledge about the characteristics of the solution, which included the constraints on
object sharpness, energy conservation, impulse response point spread function solution, and
object gradient statistics. Applying theses constraints resulted in significantly improved
solutions, as evaluated visually and quantitatively using the distance of the estimated to the true
function. We have found that the distance of the estimated psf was correlated better with visual
observation than the distance metric using the estimated object. Further research needs to be
done in this area.
To better pose the problem, we expressed the point spread function as a series of Gaussian basis
functions, instead of the pixel basis function formalism used above. This procedure has reduced
the dimensionality of the parameter space and has resulted in improved results, as expected.
We determined a set of weights that yielded optimum algorithm performance. Additional
research needs to be done to include the weight set as optimization parameters. This will free the
user from having to adjust the weights manually. Of course, if certain knowledge of a weight is
available, then it may be better to start with that as an initial guess and optimize from there.
With the knowledge that the gradient of the object obeys long-tailed distribution, we have
incorporated a constraint using the first two moments, mean and variance, of the gradient of the
object in the objective function. Additional research should be done to incorporate the entire
distribution in the objective and gradient functions and evaluate the performance.
Thesis Advisor: William T. Freeman
Title: Professor of Electrical Engineering & Computer Science
Table of Contents
A cknow ledgem ent .............................................................................................................
Ab stract ........................................................................................................................................
Figure C aption ....................................................................................................................
2
3
5
Chapter 1: Thesis organization ................................................................................................
Chapter 2: Image Resolution Techniques ....................................................................
2-1
2-2
2-3
2-4
............
8
. 10
Introduction ........................................................................................................................
A lgorithm ..........................................................................................................................
Image Quality Characterization................................... ................................................
References ..........................................................................................................................
10
11
14
19
Chapter 3: Im age restoration......................................................................................................... 22
3-1 Introduction ........................................................................................................................
3-2 A lgorithm ......................................................................................................................... .....
3-3 M odel Verification .............................................................................................................
3-4 Inference model description ............................................................ ............................
3-5 Psf mode characterization...................................................................................................
3-6 References ..........................................................................................................................
Chapter 4 Effect of active constraints on image restoration .......................................
22
22
26
34
35
48
.............. 50
4-1 Introduction .. ..........................................................................................................
50
4-2 Constraints based on the mean and variance of the object gradients ................................. 50
4-3 A lgorithm ................................... ..................................................................................... 51
4-3-1 Derivative with respect to pixels in row 1 and any column (a = 1, f = f) ......... 52
4-3-2 Derivative with respect to pixels in last row and any column (a = X , f
=P
)... 53
4-3-3 Derivative with respect to pixels in any row and column 1 (a - a,8 - 1) ......... 53
4-3-4 Derivative with respect to pixels in any row and column Y (a = a,
4-3-5 Derivative with respect to pixels in row 1 and any column (a
Y) ....... 53
=1, /3= /)
4-3-6 Derivative with respect to pixels in last row and any column (a = X ,
4-3-7 Derivative with respect to pixels in any row and column 1 (a
=
=
,
......... 54
= P ) ... 54
= 1)
........ 55
4-3-8 Derivative with to pixels in any row and column Y (a = , = Y) ................... 55
4-4 References ... ................................................................................................................
88
Chapter 5 Future work ...........................................................................................................
89
Figure Caption
Figure 1: Phase Diversity image restoration. Image quality of estimated object is much better
than that of in-focus and defocus images. The in-focus designation means that this image is
placed at the position of best focus. It is blurry because of the system's aberrations. ................. 18
Figure 2: Phase Diversity image restoration. The image quality of the estimated object is
assumed to have a NIIRS difference of 3 over that of the in-focus image ............................... 18
Figure 3: Listing of NIIRS levels and their associated taskings .......................................
19
Figure 4 Sample object and point spread function (PSF) used to compare the objective function
gradients obtained by analytical and finite difference methods..................................
....... 26
Figure 5 Figure showing the derivatives obtained analytically and finite difference method. The
values are very close to each other for the psf and the three components of the object. ............ 27
Figure 6 Latent R, G, B images and PSF in the first row; Enhanced R, G, and B images, and
estimated PSF in the second row. Due to non-uniqueness, the estimated and true functions are
different .............................................................................................
..................................... 2 8
Figure 7 Values of the objective function are in blue. Those of the maximum value of the
gradient of the objective function are in green. The algorithm has reached a local minimum..... 29
Figure 8 Latent R, G, B images and PSF in the first row; Enhanced R, G, and B images, and
estimated PSF in the second row. The estimated psf is the dirac delta function; the enhanced
latent components are the same as the detected image components .......................................... 30
Figure 9 The percent difference of enhanced vs. detected image. The percent difference is on the
order of 1e-4 and is considered to be zero .......................................................
31
Figure 10 Image restoration using the likelihood function in eq. 23. The delta function solution
for the point spread function is avoided with the constraints. .........................................
34
Figure 11 Image restoration using the inference model of ref. 1...............................................
35
Figure 12 Comparison of the analytical with the finite difference model of the gradient. Models
match as indicated by the ratios of the derivatives .................................................................... 37
Figure 13 Results when the distance between the initial guess and the truth is 0.043186 and
0.13457 for the object and psf, respectively ................................................................. ....
39
Figure 14 Results when the distance between the initial guess and the truth is 0.34578 and
0.25775 for the object and psf, respectively. .........................................................................
40
Figure 15 Results when the distance between the initial guess and the truth is 0.23506 and
0.32978 for the object and psf, respectively. .........................................................................
41
Figure 16 Results when the distance between the initial guess and the truth is 0.087996 and
0.037915 for the object and psf, respectively. .......................................................................
42
Figure 17 Plot of the distance of the estimated to truth object vs. that of the initial object guess to
the pristine object ..................................................................................................................
. 43
Figure 18 Results obtained with an RGB image of Mount Rushmore .................................... 44
Figure 19 The distance of the initial and the truth object and psf more than doubled as compared
to the values in fig. 18 .. ................................................................................................ . 45
Figure 20 Greater distance from the truth values (0.15872 and 0.18488 for the psf and object,
respectively) results in a slight performance degradation............................................................. 46
Figure 21 Difference between the starting guesses and pristine values of the psf and objects
increased significantly ......................................................................................................
. . 47
Figure 22 Plot of the distance of the estimated and initial object guess from the truth object..... 48
Figure 23 The case numbers over which the constraints weights were varied one at a time ....... 56
Figure 24 Results when all the constraints are set to zero .....................
....... 57
Figure 25 Results when the constraint PSm e=10. .........................................
............ 58
Figure 26 Results when the constraint PSm e=100. .........................................
........... 59
Figure 27 Results when the constraint PSse=l ....................................................................
60
Figure 28 Results when the constraint PSs e=10 .........................................
............. 61
Figure 29 Results when the constraint PSs e=100 ..........................................
............ 62
Figure 30 Results when the constraint PSs e=500 ..........................................
............ 63
Figure 31 Results when the constraint P=0.5. .......................................................................
64
Figure 32 Results when the constraint P=. ..........................................................................
65
Figure 33 Results when the constraint P=5. ..........................................................................
66
Figure 34 Results when the constraint P=10. ........................................................................
67
Figure 35 Results when the constraint P=1040. .........................................................................
68
Figure 36 Results when the constraint P=2080. .........................................................................
69
Figure 37 Results when the constraint PO=0.001............................................................
70
Figure 38 Results when the constraint PO=0.01..............................................................
71
Figure 39 Results when the constraint PO=0.1 .......................................................................
72
Figure 40 Results when the constraint PO= ...........................................................................
73
Figure 41 Results when the constraint PO=10.................................................................
74
Figure 42 Results when the constraint PO=240...............................................................
75
Figure 43 Results when the constraint ZZm=l e-5.........................................
........................ 76
Figure 44 Results when the constraint ZZm= .....................................................................
l
77
Figure 45 Results when the constraint ZZm= 10 ...................................................................
77
Figure 46 Results when the constraint ZZm=100 .........................................
............. 78
Figure 47 Results when the constraint ZZm=1000 ........................................
............. 78
Figure 48 Results when the constraint ZO=5e-12. ................................................................
80
Figure 49 Results when the constraint ZO=5e-9 .......................................................................
80
Figure 50 Results when the constraint ZO=5e-6. .......................................................................
81
Figure 51 Results when the constraint ZO=5e-3. .......................................................................
81
Figure 52 Results when the constraint ZO=0.5. .....................................
82
..............
Figure 53 Table of cases run by changing the weights many at a time. .................................... 83
Figure 54 Results for the constraints [P PO ZZm ZO PSm_e PSs_e] =[5 10 10 5e-12 10 1]...... 83
Figure 55 Results for the constraints
PO ZZm ZO PSm_e PSse] =[5 1 10 5e-12 10 1]........ 84
Figure 56 Results for the constraints
PO ZZm ZO PSm_e PSs_e] =[10 1 1 5e-12 10 1]........ 84
Figure 57 Results for the constraints
PO ZZm ZO PSm_e PSs_e] =[10 1 1 5e-12 1 1] .......... 85
Figure 58 Results for the constraints
PO ZZm ZO PSm_e PSs_e] =[1 1 1 5e-12 1 1] ............ 85
Figure 59 Results for the constraints
PO ZZm ZO PSm_e PSs_e] =[1 10 1 5e-12 1 1].......... 86
Figure 60 Results for the constraints
PO ZZm ZO PSm_e PSs_e] =[1 10 10 5e-12 1 1]........ 86
Figure 61 Results for the constraints
PO ZZm ZO PSm_e PSse] =[1 10 1 5e-12 10 1]........ 87
Figure 62 Results for the constraints
PO ZZm ZO PSm_e PSs_e] =[1 1 1 5e-12 10 1] ....... 87
Figure 63 Results for the constraints
PO ZZm ZO PSm_e PSs e] =[1 110101] ..............
88
Chapter 1: Thesis organization
This thesis will attempt to restore an object from a single degraded image that may be either
color or monochromatic, or detected at two different image planes. It will use estimation
theoretic approaches to derive optimal estimators of the parameters of interest. These methods
can be easily extended to accommodate multiple detected images. The thesis is organized as
follows.
*
Chapter 2 will discuss image restoration of images detected at two different planes,
separated by a known longitudinal distance. It will show tremendous improvement of the
"after" comparer to the "before" image. It will discuss image quality metrics to quantify
the utility of an image. This metric will be used to characterize the improvement of the
"after" image, as mentioned previously.
* Chapter 3 will derive an algorithm to estimate a latent or pristine object given one
colored image. This algorithm will use a mixture of Gaussian and Poisson noise model
for the detected image. To simplify the model, we will modify the Gaussian part of the
random process such that the mean and the variance become equal. That effectively
transforms the Gaussian process into a Poisson process and the noise mixture becomes
the sum of two Poisson random variable. This procedure, valid for large variances,
results in a more easily derived likelihood function. To minimize instead of maximizing,
we will use the objective functions which is defined as the negative of the likelihood
function. Since we will be using multi-dimensional parameters, we will use analytic
gradient instead of finite difference gradient, which is extremely processing intensive.
We will evaluate performance of our technique on images of various sizes.
* Chapter 4 will evaluate the performance of the algorithm when multiple constraints are
incorporated in the objective function. The effect of the constraints weights of object
sharpness, energy conservation, impulse response point spread function solution, object
gradient statistics is evaluated in this procedure. These weights will be varied
individually and together.
*
Chapter 5 will discuss potential future research.
Chapter 2: Image Resolution Techniques
2-1 Introduction
It is often said that a picture is worth a thousand words. Many may testify, however, that certain
pictures are not worth even a single letter of a word. This is because the quality of the image may
be so bad that barely any information can be gleaned from it. Many techniques1-8, heuristic and
physics-based, have been devised to restore those images to their original quality. The heuristic
methods considered include contrast enhancement, histogram equalization, thresholding, and
template matching. The physics-based algorithms can be separated into three categories:
*
The deconvolution algorithms assume that the point spread function (psf) is known a
priori and attempt to estimate an object which convolves with the psf to produce the
detected image. Examples include the Richardson-Lucy deconvolution and the Wiener
filtering.
*
The blind deconvolution algorithms estimate both the psf and the object and convolve
these two functions to generate the detected image. We invoke the [blind extension of
the] Richardson-Lucy deconvolution as one example.
*
The PD technique not only estimates the psf but also the wave front that can be due to a
combination of many factors: optics, thermal, and/or atmosphere. From wave optics
propagation theory, we know that many wave fronts can yield the same psf (as an
example, a wave front O(U, V) and its rotated version O(-U,-V) yield exactly the
same psf). We extend more on the PD technique in the following paragraphs.
The phase diversity(PD)9-13 is commonly implemented by collecting simultaneously (or within
the same aberration realization and same object aspect) two or more images separated by a
certain defocus distance usually assumed to be known. In many cases, however, the diversity
defocus is not known exactly, or if known, can change due to the operating environment. Certain
factors affecting this uncertainty include measurement errors, and variation in the hardware
configuration. In certain cases, for instance an optical system in space, these mishaps can not be
corrected easily. One study14 that considered erroneous defocus effect was performed to retrieve
the wave front using a Gerchberg-Saxton algorithm on the image of a point target. However, if
the optical system is expected to image extended targets, one needs to evaluate the performance
with similar scenes since such performance can be substantially different from that of a point
target. Some possible factors affecting performance include target size and contrast.
The phase diversity algorithm exploits the contrast between the "in-focus" and diversity images.
We expect little contrast or too much difference between the images when they are too close or
too far apart, respectively. Therefore the PD algorithm should yield marginal performance for
these extreme cases. Some studies"' 13 have determined the optimal defocus distance using
fundamental bound metrics.
2-2 Algorithm
The imaging equation within an isoplanatic region is given by'5-16
i(x, y) = o(x, y) & h(x, y) + n(x, y),
(1)
where 0 stands for the convolution operation, i(x, y) is the blurred and noisy image captured
by the detector, o(x, y) is the latent object (commonly referred to simply as object) or scene,
h(x, y) is
the point spread function (psf) of the atmosphere-optics combination hereforth
referred to as the optical system point spread function, and n(x, y) is a Gaussian distributed noise
term of mean zero and standard deviation a. Similarly, the defocused image is given by:
id (x, y) = o(x, y) ® hd (x, y)+ n
(x,y).
(2)
When uncorrelated Gaussian statistics 17-18
p(iD (X, Y) o,h)=
S1__I
(id(x, y) - o ® hd (x,y)) 2
exp -
20
(3)
2o7
2La
is assumed, the likelihood of obtaining the object and point spread function given D diversity
images of size X x Y is given by:
D,X,Y
1(o, hI i1( 1, 1) , . . . , iD (X, Y))
-
(x, y) -o
exp - (id
H
d=1,x=1,y=1
2i-
2
2U
2
hd (x,
y))
.(4)
In this paper, we will consider two diversity images, the first diversity image and psf will be
denoted by i(x, y) and h(x, y), respectively and will be referred to as the "in-focus" image
and psf, respectively. The other diversity image i2 (x, y) and psf h 2 (x, y) will be denoted by
id (x, y) and hd (x, y), and will be referred to as diversity image and psf, respectively. The
designation of "in-focus" only means that the image is placed at the best focus position. This
image is often blurred due to the system's aberrations.
Applying Parseval's theorem to the negative of the log likelihood function (neglecting an
inconsequential constant) given by:
2
DX
L = - ln(l(o, h i (1,1),..., iD(X, Y)))
Y
(id (x, y)- o
=
hd(, y)) 2
d=1 x= y=l
,(4)
yields the metric to be minimized. The maximum likelihood object function is given by 9 -1 3
L *t".
v)
.
T".
. X
H(u,(uv)I(uv)+Hd
t
L
2
A(u,
*t".
A
.
T
t
.
."
(5)
(u,v) 22
H (u, v) + Hd
The expressions for the Fourier transform of the "in-focus" and diversity images are
I(u,v) = O(u,v)H(u,v) + N(u ,v), and Id (u, v) = O(u, v)Hd (u, v) + Nd (u, v), respectively,
where, with 3 representing the Fourier transform operator, we have:
I(u,v) = 3(i(x, y)),
d (u, v) =
(id(X, y))
O(u,v) = 3(o(x, y)),
H(u,v) = 3(h(x,y)),
Hd (u, v) = 3(hd (, y)),
N(u,v) = 3(n(x,y)), and Nd (u, v) =
d(nd
(x,
(6)
y)) .
The OTF terms H(u,v) and Hd(u, v) are expressed as autocorrelation of the generalized pupil
function as follows 15-16
H(u, v) = P(u, v)ei2 6 (uv) 0 P(u, v)ei 27(uv)
Hd(u, v) = P(u, v)e i2, ( ( uv)+
("uv))
0 P(u, v)ei27 ( (
u v)+ D(u
v))
(7)
where P(u,v) is a binary function with values of 1 inside the pupil and 0 outside, and to reduce
the
optimization
parameter
space,
the
OPD
(Optical
Path
Difference)
map
a,Z,,(u,v) is expressed as a weighted sum of basis polynomials Z,(u,v). The
O(u,v)=
n
Zernike basis polynomials are often used due to their orthogonality over the interior of a unit
circle and the rotational symmetry of the optical system. More details on properties of Zernike
polynomials may be found in [19].
An example of this PD processing technique is shown in figs. 1 and 2. The in-focus and defocus
images are shown in the leftmost and middle of fig. 1, respectively. The in-focus image is not an
image without aberrations. It has the same aberration as the defocus image. However, the
defocus image has an extra aberration since it is detected at a plane displaced from that of the infocus image. That is why the defocus image is blurrier than the in-focus image. The estimated
object is displayed at the right of fig. 1. It is clear that the quality of the estimated image is much
better than that of both the in-focus and defocus images.
The example shown in fig. 2 is even more dramatic. Only the in-focus and estimated images are
shown in fig. 2 ( the defocus image is much blurrier than the in-focus image in this case also) .
As can be seen, it is impossible to identify any features in the in-focus image. After applying the
PD algorithm, the features are now easily identifiable in the estimated object shown at the right
of ig. 2. This improvement is quantified by a NIIRS (National Imagery Interpretability Rating
Scale) increase of- 3. We expand on the NIIRS formalism in the following.
2-3 Image Quality Characterization
The NIIRS metric is used to quantify image quality. This measure:
*
Makes and communicates quantitative judgments about the potential interpretability of an
image.
*
Describes the information that can be extracted from a given image.
* Relates quality of an image to the interpretation tasks for which it may be used.
The NIIRS metric is obtained from using the generalized image quality metric (GIQE). The
GIQE is a regression based technique to estimate image quality. Human observers are presented
with images of different quality and evaluate their utilities. One of the latest versions available is
the GIQE 4 given b
20
-22:
NIIRS = 10.251 - a log10 (GSD) +b log10 (RER) - 0.656H - 0.344G / SNR,
(8)
where:
RER
: Relative edge response
GSD
: Ground sample distance
H
: Edge height overshoot
G
: Noise gain due to MTF boost
SNR
: Radiometric Signal to Noise ratio (SNR)
a=
3.32 RER > 0.9
3.16 RER < 0.9
and
1.559 RER
0.9
- 2 .8 17 RER < 0.9
The GSD represents the projection of the detector pixel on the target. If a charge coupled device
(CCD), of horizontal pixel size ccdpix,H and vertical pixel size ccdpix,v, is used, then for a zenith
angle of zero (the sensor looks straight down on the scene), the geometric mean GSD is given
by:
GSDGM = GSDH GSDV,
with
GSDV
ccdpi
-
xR
f
and
GSDH
ccd
dpix,H
(9)
xR
f
R: Range from sensor to scene
f: focal length of imaging system
If the last three terms of the NIIRS equation are neglected, we can see that NIIRS changes by
one when GSD doubles.
The geometric RER is given by:
RERGM
=
ERH (0.5) - ERH (-0.5)XERv (0.5) - ERv (-0.5)
(10)
where the horizontal edge response is:
00
ER H (a) = 0.5 + 2a JMTF(f x ) sin c(2afx)dfx
(11)
0
and the vertical edge response is:
ERv (a) = 0.5 + 2a MTF(fy) sin c(2afy)dfy.
0
(12)
sin c(x) -=sin(ax)
For the horizontal edge response ERH, the edge is oriented along the vertical direction.
Similarly for the vertical edge response ERV, the edge is oriented along the horizontal
direction.
The height overshoot H represents the maximum edge response. This overshoot term penalizes
the application of a boost to the OTF.
The NIIRS metric, shown in fig. 3, encompasses the range from 0 to 9, with 9 being the best. A
NIIRS of 0 indicates no utility to the image. The in-focus in fig. 3 can be evaluated to be of
NIIRS 0. This rating may be justified since no interpretability can be inferred from that image.
The NIIRS of the estimated image is conservatively estimated at 3, but can be easily rated at a
much higher NIIRS.
Image
In-focus Image
Defocus Image
Defocus Imaze
Estimated Object
Estimated Object
Figure 1: Phase Diversity image restoration. Image quality of estimated object is much better than that of infocus and defocus images. The in-focus designation means that this image is placed at the position of best
focus. It is blurry because of the system's aberrations.
In-focus Image
Estimated Object
Figure 2: Phase Diversity image restoration. The image quality of the estimated object is assumed to have a
NIIRS difference of 3 over that of the in-focus image.
NURS 0 - Ground Resolved Distance N/A
No Interpretation of Imagery
Obscurations, Degradation, Poor Resolution
NIIRS 5 - Ground Resolved Distance 0.75 to 1.2 meters
Identify Radaras Vehicle or Trailer Mounted
Identify Deployed Tactical SSM Systems
Identify Individual Rail Cars
NIRS1 - Ground Resolved Distance > 9 meters
NIIRS 6 - Ground Resolved Distance 40 to 75 cm
Detect Medium Port Facility
Distinguish Between Small/Large Helicopters
Distinguish Taxiways and Runways (Large Airport)
Distinguish Between SA Missile Airframes
Distinguish Between Sedans and Station Wagons
N1RS 2 - Ground Resolved Distance 4.5 to 9 meters
Detect Large Hangars
Detect Military Training Areas
Detect Large Buildings
NIIRS 7 - Ground Resolved Distance 20 to 40 cm
Identify Fittings and Fairings on Fighter Aircraft
Identify Ports, Ladders, Vents on Electronics Vans
Identify Individual Rail Ties
NIIRS3 - Ground Resolved Distance 2.5 to 4.5 meters NIIRS 8 - Ground Resolved Distance 10 to 20 cm
Identify Large Aircraft
Identify Rivet Lines on Bomber Aircraft
Identify a Large Surface Ship in Port
Identify a Hand-Held SAM
Detect Trains on Railroad Tracks
Identify Windshield Wipers on a Vehicle
NIIRS4 - Ground Resolved Distance 1.2 to 2.5 meters NIRS 9 - Ground Resolved Distance < 10 cm
Identify Large Fighter Aircraft
Differentiate Cross/Straight-Slot Aircraft Fasteners
Identify Tracked Vehicles
Identify Missile Screws and Bolts on Components
Determine the Shape ofa Submarine Bow
Identify Vehicle Registration Numbers on Trucks
Figure 3: Listing of NIIRS levels and their associated taskings
2-4 References
1. R. D. Fiete, T. A. Tantalo, J. R. Calus, and J. A. Mooney, "Image quality of sparse-aperture
designs for remote sensing," Opt. Eng. 41, 1957-1969 (2002).
2. J. R. Fienup, "MTF and integration time versus fill factor for sparse-aperture imaging
systems," in Imaging Technology and Telescopes, James W. Bilbro, James B. Breckinridge,
Richard A. Carreras, Stanley R. Czyzak, Mark J. Eckart, Robert D. Fiete, Paul S. Idell; Eds.,
Proc. SPIE 4091, 43-47 (2000).
3. W. T. Freeman, T. R. Jones, and E. C. Pasztor, "Example-Based Super-Resolution," IEEE
Computer Graphics and Applications, 56-65 (2002).
4. W. T. Freeman, E. C. Pasztor, and 0. T. Carmichael, "Learning low-level vision," Int '
Journalon Compter Vision, 40, 25-47 (2000).
5. F. Tsumuraya, N. Miura, and N. Baba, "Iterative blind deconvolution method using Lucy's
algorithm," Astron. Astrophys. 282, 699-708 (1994).
6. G. R. Ayers and J. C. Dainty, "Iterative blind deconvolution method and its applications,"
Opt. Lett. 13, 547-549 (1988).
7. A. Labeyrie, "Attainment of diffraction limited resolution in large telescopes by Fourier
analyzing speckle patterns in star images," Astron. Astrophys. 6, 85-87 (1970).
8. S. M. Jefferies, and J. C. Christou, "Restoration of astronomical images by iterative blind
deconvolution," Astron. J. 415, 862-874 (1993).
9. R. A. Gonsalves, "Phase retrieval and diversity in adaptive optics," Opt. Eng. 21, 829-832,
(1982).
10. R. G. Paxman, T. J. Schultz, and J. R. Fienup, "Joint estimation of object and aberrations by
using phase diversity," J. Opt. Soc. A. 9, 1072-1085 (1992).
11. J. J. Dolne, R. T. Tansey, K. A. Black, Jana H. Deville, P. R. Cunningham, K. C. Widen, and
P. S. Idell, "Practical Issues in Wave-front Sensing by Use of Phase Diversity," Appl. Opt.
IP, 42, 5284-5289 (2003).
12. M. G. Lofdahl, G. B. Scharmer, "Wave-front sensing and image restoration from focused and
defocused solar images," Astron. Astrophys. Suppl. Ser. 107, 243-264, (1994).
13. D. J. Lee, M. Roggeman, B. M. Welsh, "Cramer-Rao analysis of phase-diverse wave-front
sensing," J. Opt. Soc. Am. A 16, 1005-1015, (1999).
14. N. Baba, E. Kenmochi, "Wavefront retrieval with use of defocused PSF data," Optik 84, 7072, (1990).
15. J. W. Goodman, Introduction to Fourier Optics (McGraw-Hill, New York, 1968).
16. M. C. Roggemann and Byron Welsh, Imaging Through Turbulence (CRC Press LLC,
Florida, 1996).
17. S. M. Kay, Fundamentals of Statistical Signal Processing: Estimation Theory (Prentice Hall,
New Jersey, 1993).
18. A. Papoulis, Probability, Random variables, and Stochastic Processes (McGraw-Hill, New
York, 1991).
19. R. Noll, "Zemike polynomials and atmospheric turbulence," J. Opt. Soc Am. 66, 207-211,
(1976).
20. R. G. Driggers, M. Kruer, D. Scribner, P. Warren, and J. Leachtenauer, "Sensor Performance
Conversions for Infrared Target Acquisition and Intelligence-Surveillance- Reconnaissance
Imaging Sensors," Appl. Opt. 38, 5936-5943 (1999).
21. J. Leachtenauer, W. Malila, J. Irvine, L. Colbum, and N. Salvaggio, "General Image-Quality
Equation: GIQE," Appl. Opt. 36, 8322-8328, (1997).
22. Gerard Holst, Sampling, Aliasing, and Data Fidelity (SPIE Press, Washington, 1998).
Chapter 3: Image restoration
3-1 Introduction
Nowadays, imaging equipments are widely available over the market. Consumer electronics such
as video recorders and cell phone imagers are routinely used. However, the resulting images
have significant blur due to a combination of factors as mentioned above. Furthermore, motion
blunrring is common in these images. One peculiarity of these images is that, sometimes, only a
single frame is available. Therefore, it may not be applicable to use the techniques mentioned in
the previous chapter. In this chapter we will assume a single RGB image is available. We will
address a procedure to mitigate these blurring effects and estimate the latent (enhanced) object
and point spread function from the detected noisy and blurred image. Our blur is 2-D.
3-2 Algorithm
Assuming an isoplanatic optical system, the detected image ic (x, y) is given by' 3 :
i,(x, y)= n,c(x, y) + np,(x, y),
(1)
where
lG,c (X,
y)
is a Gaussian noise process of zero mean and variance
np,c (x, y), a Poisson noise process of mean Oc 0
o c (x, y) is the latent color image and C E {R,
h(x, y),
h(x, y);
G, B);
the point spread function, common to all the colors; and
(x, y), the spatial coordinates.
0"
;
To make the likelihood function manageable, we use an approximation, valid for large variance
value of the signal, similar to what has been used in the literature. We will modify the Gaussian
noise process as follows 1-6
n
y) =Gc (X, ) +Y)
G(x,
This way, the Gaussian noise process, fn
(x, y),
(2)
is similar to a Poisson process of mean and
2
variance U . The modified detected image, representing the sum of the Poisson processes, is
itself Poisson and is given by:
i (x, y)
=
n
y)
,(x,
np, c (x, y)
(3)
The mean and variance of this Poisson process is specified as:
o ® h(x,y)+o.
(4)
Accordingly, the likelihood function may be written as:
)
(i c (x,y)+c 2
2
L h(x, y)+ 2 )
fl
xy
O
c=R,G,B
r.
,
e-(oc ®h(x,y)+a )
(5)
)
( (x, Y) + oc
The log-likelihood, given by the logarithm of the above expression, is (neglecting the constant
term):
LL =
(ic(xy) +
xy
c=R,G,B
)In(oc
lUf 0 h(x,y) + O )- (oc
(6)
h(x, y)+
.
To minimize, instead of maximizing, we take the negative of the previous expression, and adding
some constant terms to make sure that the minimum is zero, we obtain:
c=R,G,B
xy
}i
+ic(x, y) +o2 11n(ic (x, y) + C )- In(oc 0 h(x, y) +o2
I+[Oc ®h(x,y) + )- (ic (x, ) + )
LL =
(7)
This negative of the log-likelihood is referred to as the objective function.
The goal is to find the functions o c (X, y) and h(x, y) that minimize the objective function
given in equation 7. Since an exhaustive approach is not possible, we will use a gradient search
technique to arrive at the solution. Attempting to calculate the gradient using finite difference is
computationally expensive and takes extremely long. Therefore, we resort to computing the
gradient analytically.
The derivative of the objective function with respect to the object pixels is given by:
aocL,= I ( h(x - a,y
aoc(a,P)
h(x-a,y - f)
oc0 h(x, y) + o
2
_'fl) - (ic(x,
xY
(8)
which may be written as:
8LL
=C h(x - a,Y - P)1
0Oc(a,p)
(ic(x,y) + 0 2 )
oc h(x, y)
2
(9)
Similarly, the derivative with respect to the point spread function is given by:
8LL
ah(a,f)
=L
xy c=R,G,B
(ic(x, y) + 72)
Oc(x- a,y
81(1
Oc
h(x,y)+ c2
(10)
This imaging system, being incoherent, will yield positive valued object values. To ensure that
these values are positive, we will perform the estimation with respect to the square root of the
object pixels, Oc,sr (X, y), and express the object pixel as:
c(x, y) = Oc,s, (X,y)2
(11)
The derivative of the objective function, with respect to the square root of the object pixels, is:
LL
oc,s,r (a, 8)
where
Boc (a,p)
8LL
aoc (a, l)
(12)
o c (a, fi) oc,s,r (a, 8)'
= 2 , (a, f).
oc,sr (a, fl)
Similarly, in order to use unconstrained optimization techniques for the psf, the estimation of the
point spread function is expressed in terms of its square root, with:
h(x, y) = h,,(X, y) 2 .
(13)
Using the chain rule, we write the derivative of the objective function with respect to hr,(x, y)
as:
aLL
ahs, (a, /)
where
8h(a,/)
=f-=
2hs, (a, f).
8hs,r (a, 6)
aLL ah(a,/)
ah(a, ) ahs,(a, )'
(14)
3-3 Model Verification
To assure us that the model works, we checked the analytical gradient with that computed from
finite difference. As an example, we used an 8x8x3 pristine object and the psf shown in fig. 4,
right-hand side, to compare the gradients from both methods.
aject prfst G
2
4
6
8
2
4
6
8
CectQrparstB
2
4
6
8
PSF
2
4
6
8
Figure 4 Sample object and point spread function (PSF) used to compare the objective function gradients
obtained by analytical and finite difference methods.
The following figure, fig. 5, shows the values obtained for the derivatives. The numbers are very
close to each other. The ratio is approximately one for all the object and PSF pixel values as
shown by the columns labeled Analytic/Finite Difference, signifying the ratio of the derivatives
obtained by analytic method to those by finite difference.
....
:i.- -. . .
-I
I
Figure 5 Figure showing the derivatives obtained analytically and finite difference method. The values are
very close to each other for the psf and the three components of the object.
Applying this algorithm to a color image yields the results shown in fig. 6. It seems that the
enhanced components do not resemble the original functions. This is due to the fact that we have
four unknowns, the R, G, B images, and the PSF, but only three measurements. The problem is
27
ill-posed. This example is a local solution of the problem. We note that in many cases, the
algorithm converges to a point solution for the estimated psf and yields the detected image as
solution to the latent (true) object.
Object Component R
Object Component G
Enhanced Object
Component R
Enhanced Object
Component G
Object Component B
Enhanced Object
Component B
PSF
Estimated PSF
Figure 6 Latent R, G, B images and PSF in the first row; Enhanced R, G, and B images, and estimated PSF in
the second row. Due to non-uniqueness, the estimated and true functions are different.
We say that the solution given in fig. 6 is a local solution by examining the metric in fig. 7. It
shows the value and the maximum absolute of the gradient of the objective function at each
iteration. The objective function decreases from 7.5 to 1.5e-8. Furthermore, the maximum value
of the gradient reaches - 1.4e-5, small enough to be considered negligible. However, this is not
the global solution since, when substituting the true color latent image and the true PSF, the
value of the objective function is - 5e-16.
MU
0
> 10
._E
(3
C
; -010,11
10 o
100
1
Iteration Number
)
Figure 7 Values of the objective function are in blue. Those of the maximum value of the gradient of the
objective function are in green. The algorithm has reached a local minimum.
One of the solutions often obtained is that where the psf is a delta function as shown in Fig. 8.
Although the objective function value is 1.085954e-008 (objective function value when true
pristine and psf are substituted), we see that the latent image components are very blurry
compared to the "truth" latent object components.
Component R
R
Object Component
Object
Enhanced Object
Component R
G
Object Component
Component G
Object
Enhanced Object
Component G
Object Component B
Object Component B
Enhanced Object
Component B
PSF
PSF
Estimated PSF
I
Figure 8 Latent R, G, B images and PSF in the first row; Enhanced R, G, and B images, and estimated PSF in
the second row. The estimated psf is the dirac delta function; the enhanced latent components are the same as
the detected image components.
The enhanced image components in fig. 8 are very close to the detected image components. This
is further emphasized in fig. 9 where we plotted the percent difference between the detected and
enhanced images. The percent difference is on the order of le-4 and is, therefore, considered to
be zero.
0x
0x10
0.
10
B
R
-0.5
-0.5
-1
-1
-1.5
-1.5
1
-20
50
0
100
Pixel Number
Pixel Number
50
100
Pixel Number
Figure 9 The percent difference of enhanced vs. detected image. The percent difference is on the order of le-4
and is considered to be zero.
To avoid that solution, we impose the constraint with a weighting of PO:
PO *
log(1 - h(x, y)).
(15)
This term (eq. 15) penalizes the objective function as the psf approaches fewer and fewer pixels.
In order to have a high-fidelity object, we include a sharpness metric as follows:
- ZZ *Y cO,
(x,y) ,
(16)
cxy
where ZZ designates the level of importance of sharpness in the objective function. By
controlling ZZ, the level of sharpness in the recovered object can be manipulated. This metric
0oc (x, y)
is based on the fact that, given a positive function distribution, - ZZ *
cxy
2
will be
smaller if the object pixels are concentrated in a few pixels than if they are distributed in many
pixels. An other way of enforcing the sharpness constraint is through:
ZZm / 1 oc (x, y) 2 .
(17)
cxy
More information will be given below on the expression 17.
Making sure that the psf sums to 1 was enforced through:
2
P*
h(x, y)
-1
,
(18)
xy
with a weight P.
The conservation of energy, with a weighting of zo, was enforced as follows:
ZO*
(oc
(x, y) - ic(cxy
cxy
y))
(19)
Eq. 18 is a "mild" form of ensuring that the energy is conserved from object to image. It can be
seen in the following.
Ignoring noise in eq. 1, we write the detected image as:
- o(x', y')h(x'-x, y'-y)dx'dy'.
ic (x, y)=
(20)
The total energy Id over the detector plane is found as:
I d = i(x,y)dxdy= f Jo(x',y')h(x'-x,y'-y)dx'dy'dxdy, (21)
xy
xyx'y'
which becomes:
Id
(22)
= o(x', y')dx'dy'.
fi
c(x, y)dxdy=
x'y'
xy
This implies that having the psf sum to one helps achieve energy conservation. The complete
objective function is:
+ I(oc 0 h
LL =
xy
c=R,G,B
y)++(x, )-
h(x, y)j
+P*
+ ZO*
1
h(, y)+
- Oc
2n(i
c
+2
(,) )ic+
1 -ZZ *
hy
oc(x, y) 2 +ZZm/
(c (x, y) - ic (x, y))2 + PO*
o, (x, Y) 2
log(1 - h(x, y))
xy
(23)cxy
(23)
The derivative of this new objective function is computed accordingly (remark that the sharpness
oc (x, y)
term constraint - ZZ *
2
was not used because it tended to yield images that
cxy
were very often two-tone). We mention that with this new objective function, we were able to
avoid the delta function solution for the point spread function. However, the solution did not
seem to converge to the true object-psf pair when the object was more than a few pixels in
extent. An example of the restoration is shown in fig. 10. The quality of the enhanced object is
clearly superior to that of the detected image. We note that the estimated psf is not a point, but
occupies many pixels in extent as is the case for the "truth" psf.
These results are compared to those obtained with inference model described below.
Truth Obiect
Detected Imaae
Tr uth Pef
Estimated Obiect
Estimated Psf
Figure 10 Image restoration using the likelihood function in eq. 23. The delta function solution for the point
spread function is avoided with the constraints.
3-4 Inference model description
In ref. 4, Fergus, Singh, Hertzmann, Roweis, and Freeman devised a method for removing blur
from camera shake using an inference scheme. That approach consists of approximating the full
posterior distribution using the variational Bayesian formalism. This minimizes the distance
between the approximating and the true posterior distribution. An example of the algorithm
result is shown in fig. 11. The estimated object is clearly much better than the detected image.
However, the estimated psf does not look very closely to the true psf. More tweaking may need
to be done in the parameters used.
Truth Object
Detected Image
Estimated Object
Estimated Psf
Figure 11 Image restoration using the inference model of ref. 1.
3-5 Psf mode characterization
The problem of image restoration is ill-posed. For our case, we are solving for four 128 x 128
values, however, we only have three 128 x 128 measurements. To better pose the problem, we
express the psf as a series of Gaussian basis functions. This way, the dimensionality of the
estimation space is reduced. Each Gaussian function is characterized by a mean value in the
vertical, /Iyj , one in the horizontal direction, pj, and standard deviation, Oj, as follows:
h(x,y) =
C,7
1 exp(
2go j~
Px
1 )2+(
-
-
y
(24)
Y 2,2)
The derivative of the objective function with respect to the psf parameters is given, using the
chain rule, by:
8LL
SLL
a/ ahs, (a, p)
itxa
where
8hsr (a, /)
(25)
xax
h(x, y) = hs,, (x, ) 2 and
hs,,j(x, Y) = (212
1
(2l2
Ye
exp(_((x
x )Y+( - py
40- . (26)
aLL
) derived above. If only one basis function is
Eq. 25 allows us to use the results for h (a,
used to characterize the psf, the derivatives with respect to the psf parameters are:(a,
used to characterize the psf, the derivatives with respect to the psf parameters are:
ahsr (a, fl)
8hs,(a,
(a, ) and
(27)
ayxa
ah(a,f)
apxa
Similar expressions are given for
(a,, ) ah(a,f)
-hsr
ahsr,(a, /)
apya
The expanded expressions for the derivatives are given by:
ah(x, y)
(X Ua
-
ah(x, y)
7_
a
xa
+
0 -u
a
_
(Y - PUya) exp
2
and
_
xa ) exp(- ((X -
2
a
((x - Pxa)2+
- tya Y
2
ya
/20
a ),(28)
), (29)
ah(x, y)
x
__ (X - /"ya)
dor
)2i
+ (Y - /yay2
)2
(X -
xa
2
a
2o.2J
(30)
To make sure that our analytical model works, we compared the analytical with the finite
difference model of the gradient in fig. 12. The two models match very closely as shown by the
ratio of approximately 1 of the analytic to the finite difference gradient elements.
s
•G
Object pixeI R
Analytic
Fnt
Dlforenco
Object
Finit
n
c/li*Anayi
Diforence
pixel GGradientOjetamet
Dlference*
AnltcFnt
Dtrn.
e
Anlyi
Fiit
Difrec
AR~lileA
Diftrence
An
4.4"U*
.0.4984
1.00000
-1.39700
-1.r700
1.o00000Q
-. 42851
442851
1.00000
-1.97040
-1.97030
1.00005
-2.47640
-2.47070
1.00004
-1.13930
-1.13930
1.00000
-1.50180
0.9993
4.aess -1.50,190
.o~os
t o.ss0
-0.835
-0.410937
0.993S
.0.63550
-0.63551
0.99"S8
o.os4
-0.os3ea
1.00003
0.4310
-0.94312
0.99M8I
-1.53170
-1.53160
1.00007
-1.28380
-1.230
1.00000
-0.93885
-0.93884
1.00001
-. 29530
-12950
1.00000
4.71875
-0.71875
1.00000
-. 72475
-0.72474
1.00001
-. 91440
4.91439
1.00001
-2.1840
-2.18830
1.00005
-. 7479
-1.749W
0.99994
-. 90210
-1.90200
1.00005
-2.480
-2.48850
1.00000
-1.3300
-1.a33o
1.00000
4.51735
-0.51736
1.00000
4.50185
4.50486
0.99998
0.73140
-4.73142
0.99M97
-1.17160
-1.17150
1.00009
-0.82470
4.82469
1.00001
-. 70135
.. 70137
0.99"7
-1.39120
-1.39120
1.00000
-0.43446
0.4344
1.00000
-- 1.8290
-1.1290
1.00000
4 As514
-0.4s51
1.00002
-0.46729
0."72"
1.00000
-0.79160
4.79158
1.00003
4.49112
-. 49112
1.00000
-0.33661
4.33661
1.00000
4-0.908s0
0.999
.0.90
40.38311
-0.38311
1.00000
-1.134640
-1.13640
1.00000
-0.27290
4.27289
1.00004
-0.30695
-0.300
1.00000
-0.57625
-O.5023
1.00003
40.47346
0.4734
1.00000
04.735
0.39734
1.00003
-0.75560
-. 75560
1.00000
-0.33819
-0.33819
1.00000
-0.56790
4."788
1.00004
0.09667
0.0967
0.99997
0.0216
4.02164
0.99995
4-0.15388
-. 15387
1.00006
-0.25511
1.25511
0.12812
0.12812
1.00000
-0.24161
.24161
1.00000
0.36128
-0.36128
1.00000
-. 45306
-0.40306
1.00000
0.13293
0.13293
1.00000
0.051 7
0.05167
0.
a***
Tori -0.07196
0..*0*9
47."1100
iie
AnaltcIFhn
Dlference
Diforene
-14.8se0
.11
47."7400
tooooo
.9
1.00001
I
-
-
_
I
4.00000
-0.20578
4.20578
1.00000
.0.24310
4.0908"
-. 235451
-0.033141
-0.24310
.0.09085
4.23545
0.0334
1.00000
1.00002
1.00000
0.99997
Figure 12 Comparison of the analytical with the finite difference model of the gradient. Models match as
indicated by the ratios of the derivatives.
We then evaluated the estimation method using small RGB images of pixel size 8 x 8. These
images were generated using the pristine objects shown in fig. 13. The first guess of the
estimated object is a noisy version of the pristine object generated as follows:
* Add a noisy term of a certain standard deviation to the pristine object
* Zero out the negative components
* Take the square root of the resulting non-negative values and use as first guess of the
square root of the estimated object
*
Iterate until algorithm converges or the maximum number of iterations is reached
To characterize how far apart the functions are from their true values, we adopted a distance
measure where for two functions f(x,y) and g(x,y), it is given by:
j
(f(x, y) - g(x, y)) 2
xy(31)
product(size(f (x, y))) * product(size(g(x,y)))
Fig. 13 shows the results when the distance between the initial guess and the truth is 0.043186
and 0.13457 for the object and psf, respectively. The distances between the estimated and the
true values were found to be, 0.00021558 and 0.00047069, for the psf and object, respectively.
The small distance value indicates how close the two functions are. This is further reinforced by
the visual comparison of the estimated and pristine objects; they look almost indistinguishable
from each other.
First guess of Object
R
r,
R
Detected Image
G
Estimated Image
Distance of estimated
from truth psf
0.00021558
Distance of first guessed
from truth psf
0.13457
Distance of estimated
from truth object
0.00047069
Distance of guessed
from truth object
0.043186
Figure 13 Results when the distance between the initial guess and the truth is 0.043186 and 0.13457 for the
object and psf, respectively
As mentioned above, a metric of performance is the distance between the estimated and pristine
object and psf parameters. Another metric used, although to a lesser extent, is the visual
comparison of the estimated to the true detected image, where the estimated image is given by
the convolution of the estimated object with the estimated point spread function. In this example,
the estimated and detected images are very similar visually.
Figs. 14 to 16 show that distance (between estimated and true values) increases as the initial
guess of the object deviated significantly from the pristine object.
First guess of Object
R
G
B
Detected Image
Pri.stin OC)hipd
Estimated Image
Estimated Obiect
Distance of estimated
from truth psf
0.24634
Distance of first guessed
from truth psf
0.25775
Distance of estimated
from truth object
0.25514
Distance of guessed
from truth object
0.34578
Figure 14 Results when the distance between the initial guess and the truth is 0.34578 and 0.25775 for the
object and psf, respectively.
First guess of Object
Detected Image
G
Pristine Obiect
Estimated Image
Estimated Obiect
Distance of estimated
from truth psf
0.022912
Distance of first guessed
from truth psf
0.32978
Distance of estimated
from truth object
0.13163
Distance of guessed
from truth object
0.23506
Figure 15 Results when the distance between the initial guess and the truth is 0.23506 and 0.32978 for the
object and psf, respectively.
First guess of Object
R
G
B
Detected Image
G
Pristine Obiect
Estimated Image
Distance of estimated
from truth psf
0.00097104
Distance of first guessed
from truth psf
0.037915
Distance of estimated
from truth object
0.0017361
Distance of guessed
from truth object
0.087996
Figure 16 Results when the distance between the initial guess and the truth is 0.087996 and 0.037915 for the
object and psf, respectively.
To visualize the performance of the constrained maximum likelihood algorithm, we plotted the
distance of the estimated to truth object vs. that of the initial object guess to the pristine object in
fig. 17. As noticed in the above images, the performance decreases as the initial estimates get
further and further from the true values.
7r
0
0Io -'4--
n- - -
-
------
II
R
First guess of Object
G
B
Detected Image
G
Pristine Obiect
Estimated Image
F.timqtpd Chipnt
Distance of estimated
from truth psf
0.01303
Distance of first guessed
from truth psf
0.020347
Distance of estimated
from truth object
0.04117
Distance of guessed
from truth object
0.048188
Figure 18 Results obtained with an RGB image of Mount Rushmore
Using a worse starting guess did not seem to affect the results significantly, as shown in fig. 19.
In this case, the distance of the initial and the truth object and psf more than doubled as
compared to the values in fig. 18. However, the final result of the objects resembled closely the
truth object. The distance between the estimated and truth values is only 0.081681 for the object
and 0.013236 for the psf. The distance for the psf does not change much from that in fig. 18.
First guess of Object
R
r
R
Detected Image
G
Estimated Image
Estimated Obiect
Distance of estimated
from truth psf
0.013236
Distance of first guessed
from truth psf
0.096511
Distance of estimated
from truth object
0.081681
Distance of guessed
from truth object
0.096023
Figure 19 The distance of the initial and the truth object and psf more than doubled as compared to the
values in fig. 18
Starting with a greater distance from the truth values (0.15872 and 0.18488 for the psf and
object, respectively) results in a slight performance degradation as shown in fig. 20. The
estimated and the pristine objects are very close visually. Numerically, they are only separated
by a distance of 0.01547 and 0.15746 for psf and object, respectively.
R
First guess of Object
G
B
Detected Image
R
G
B
Estimated Image
F timtar
Distance of estimated
from truth psf
0.01547
hiaret
Distance of first guessed
from truth psf
0.15872
Distance of estimated
from truth object
0.15746
Distance of guessed
from truth object
0.18488
Figure 20 Greater distance from the truth values (0.15872 and 0.18488 for the psf and object, respectively)
results in a slight performance degradation.
Fig. 21 shows significant difference between the starting guesses and pristine values of the psf
and objects. After applying our constrained maximum likelihood algorithm, the object and psf
are nicely recovered. The estimated objects bring out features otherwise unrecognizable in both
the initial guess and the detected images. The distance between the estimated and the truth values
are 0.052744 and 0.31862 for the psf and object, respectively.
First guess of Object
R
G
B
Detected Image
R
G
B
Estimated Image
Distance of estimated
from truth psf
0.052744
Distance of first guessed
from truth psf
0.52175
Distance of estimated
from truth object
0.31862
Distance of guessed
from truth object
0.41729
Figure 21 Difference between the starting guesses and pristine values of the psf and objects increased
significantly.
A plot of the results cited above is shown in fig. 22. It shows the distance of the estimated and
initial object guess from the truth object. The slope, being less than one, indicated that the
estimated object is closer to the truth object than the initial guess is. This indicates that our
algorithm performs as expected and always points towards the best solution. However, as the
initial guess becomes worse and worse, the ML algorithm is more prone to getting stuck in a
local minimum.
0
Q00
Q1
Q15
Q2
DamdFirst G
Q
Q3
_ssjedt
fronThih je
Q05
Q4
045
Figure 22 Plot of the distance of the estimated and initial object guess from the truth object
3-6 References
1. D. L. Snyder, C. W. Helstrom, A. D. Lanterman, M. Faisal, and R. L. White, "Compensation
for readout noise in CCD images," JOSA, 12, 272-283 (1995).
2. D. L. Snyder and A. M. Hammoud, "Image recovery from data acquired with a chargecoupled-device camera," 10, 1014-1023 (1993).
3. L. A. Shepp and Y. Vardi, "Maximum-likelihood reconstruction for emission tomography,"
IEEE Trans. Med. Imaging 1, 113-122 (1982).
4. R. Fergus, B. Singh, A. Hertzmann, S. Roweis, and W. T. Freeman, "Removing camera
shakefrom a single image, " SIGGRAPH 2006.
5. W. T. Freeman, E. C. Pasztor, and 0. T. Carmichael, "Learning low-level vision," Int'l
Journalon Compter Vision, 40, 25-47 (2000).
6. M. J. Black, Y. Yacoob, A. D. Jepson, and D. J. Fleet, "Learning parameterized models of
image motion," Proceedings IEEE CVPR, 561-567 (1997).
7. R. Fletcher, Practical Methods of Optimization, (Wiley-Interscience, Britain, 2001).
8. Jorge Nocedal and Stephen Wright, Numerical Optimization (Springer Series, New York,
1999).
Chapter 4 Effect of active constraints on image restoration
4-1 Introduction
In this chapter, we will study the effects of various constraints on image enhancement
performance. As noted earlier, the gradient of a natural scene follows long-tailed distribution' 2.
It seems logical that inclusion of such information may improve the algorithm performance. This
chapter will evaluate results obtained by including constraints based on the first two moments
(the mean and standard deviation) of the estimated object gradients.
4-2 Constraints based on the mean and variance of the object
gradients
The constraint on the mean of the object function gradient is enforced with this term:
P Sm e (vo,
where
-
J,
vo )
(1)
vo,~ is the mean of the gradient of the estimated object color component c given by:
(x, y -
vo1
=(o
2
XYXy
( -1, y) + o c (x, y) - oc (x, y - 1)), (2)
which becomes:
Y
vo=2XY
(X, y) - o,(1, y))+
x
(o (x, Y) - O(X,1)).
(3)
We enforce the variance constraints through:
PSse(62voc -
-2vo
(4)
The variance of the gradient of the object function is the sum of the variances of the gradients in
the x and y directions and is defined as:
XY1 +oC
_
vo -XY-1 XY
2
2(x',y')
xY ,
Y)
o
(x
1,
y)
-I(o, (x', y') x,
(,
+ oC (x, y)-
1 xy
,(x, y -I) XY
0, Wx-
1, y')))
2
y') - Oc (X',
(o(x,
x'y'
(5)
which becomes:
1
o (x,y)- o, (x - 1,y)
X
&2
o
XY-1
xy
+ oc(x,y)
-
(o0 (X, y') - o0(1,y'))
XY
--
o (X,y- 1)
.(6)
1
X
-Z(oc
XY
xt
(x',M)
-
oc (x',1))
4-3 Algorithm
The modified objective function is given by:
LL =
I
xy
Iic (X,y) +
+ 2)_-Ino 0 h(x,y)+ 2
2In (ic(x, y)
2
cc
c=R,G,B
cn
0 h x y) +o~c
12
+[ oc0 h(x, y) + o)-
(ic(x, y) + C+Pp
Zh(x,y)-
1)
xy
-zz *
oC
(x, y) 2
+ ZZm / 1 oc
cxy
- PO *
(x, y) 2
+ ZO *
cxy
log(1 - h(x,y)) + PSm_ e
+ PSs _eL (2Vo
C
-
2
-- O-Vo
(oC (x, y) - ic(X,y))
Lcxy
(tvo
c
12
- UPVo
,(7)
whose derivative with respect to the object pixel Oc (a,
aLL
- 2ZZm
y-fl)
oc (a, fl)
P) is:
S(ic(x,Y)+
)
-2ZZ*
Oc ®h(x,y)+c
+ 2ZO * Z (oc (x', y')
- ic(X', y') )
.(8)
oC (X,y)
+2PSm_e AVO - /Vo
)
afvc
^2
aoc (a, f)
+ 2PSs_
2
e (2o ,
(
aoc,(a, P
^2
The derivative expressions for the mean,
ao, (a, #)
, and the variance,
Voc
aoc (a, P)
, are
considered in the following paragraphs.
As can be seen from eq. 3, the derivative of the mean value,
c
aoc (a, 8)
, is zero except for the
border pixels. We compute the derivatives, understanding that the derivatives of the mean of the
gradient of the estimated image color component c with respect to any other object pixel of a
color component different than c are zero, in four steps.
4-3-1 Derivative with respect to pixels in row 1 and any column
(a = 1,I = P)
In this case, the derivative is given by:
c(aop (a,/l)
v
1 + 6(
2XY
- Y) -
(# -1)).
(9)
4-3-2 Derivative with respect to pixels in last row and any column
The derivative is expressed as:
aVO
ao, (a, /)
(1+ ( - Y)- S(p -1)).
-
2XY
(10)
4-3-3 Derivative with respect to pixels in any row and column 1
(a = a, fl= 1)
The derivative is:
ao, (a, 8)
1 (-1+ (a - X) - 6(a -1)).
2XY
(11)
4-3-4 Derivative with respect to pixels in any row and column Y
The derivative is:
o(a,)
ao, (a,,8)
1
1 (1+ (a - X) - (a - 1)).
2XY
(12)
The derivatives of the variance of the gradient of the estimated image color component c with
respect to the inner object pixels of the same color component c (i.e. those that do not touch the
borders) where a
# 1, a # X, 8
1, 8 # Y are (of course the derivatives of the variance of
the gradient of the estimated image color component c with respect to any other object pixel of a
color component different than c are zero):
07Vo,
ao,(a,,)
-2
F(oc(a+l,P)-
(a, 8)) - (o, (a,fl)- oc(a- 1,p))
XY- 1 _+(o(a,p+ 1)- 0, (a,,8))- (o, (a,,8)- oc (a, f - 1))9,
.(13)
The derivatives of the border pixels are evaluated in the following manner, noting these term
designations:
Xd (X, y) = (OcX,
(
Y
1, ))- AT
)- Oc (X -
(oc (x
oc (1, y)) (14)
y=l
and
X
Yd (X, y) = (Oc (X,
)- Oc (X, y-1))
1
XY x=1
(Oc(X, Y)-o,(x,i)).
(15)
4-3-5 Derivative with respect to pixels in row 1 and any column
(a = i,,8 = P)
2
(x- 1,y- f) + XY - 8(x- 2,y-
Xd (x,y)
,-2
UVo
2
ao, (a,f) XY - 1,
(x - 1,y -)
+Yd (X,Y)r
+
7-
-
(x - 1,y -
(g( - 1) - ( f-
,)
.(16)
- 1)
Y))
I
4-3-6 Derivative with respect to pixels in last row and any column
(a =X,5 = P)
Xd(X, y)((X-X, y-,8)- 6(x-X- 1,y -,8)
,,a2
ao^Vo c
Coc (a, f)
2
XY -1,
(4(x - X, y -)
+ Yd (X Y)
1(8
;YY1
- 6(x - X, y -/8 - 1)
1)-
fiY)
Y
(17)
4-3-7 Derivative with respect to pixels in any row and column 1
(a = a, fl = 1)
-a,y-1)-8(x-a
-y-lj
Xd
a-2
XY
ao (a,8) XY - 1 y
+ Yd
(o(a - 1) - 6(a ,j
)(5(x a,y -1)- 8(x- a,y - fl -1)
1
Y
(18)
4-3-8 Derivative with to pixels in any row and column Y (a = a, P = Y)
-a,y-Y)-3(x-a-1,y- -Y)
,2
SVo
_
2
XY
ao, (a,p ) xY-1
+Yd (, y)
(8(a -1)-
- a, y - Y) -
8(a - X))
(x - a, y -
(19)
The results using the above expressions are discussed below.
An extensive experiment is designed to study the effect of the various constraints as shown in
fig. 23. The constraints weights are varied over considerable range to gauge their true effect on
the algorithm performance.
0
1
o
I
0.5
I
Figure 23 The case numbers over which the constraints weights were varied one at a time.
Fig. 24 shows the initial object guess (first three columns of the first row on the left side), the
initial psf guess (fourth column of the first row on the left side). As before, the pristine objects
and truth psf are displayed in the second row on the left side. The estimated object and psf are
shown in the third row on the left side. On the right side of the fig. 24, are shown the detected
and estimated images. As stated before, the detected image is obtained through convolving the
pristine object with the truth psf, whereas the estimated image is obtained by convolving the
estimated object and estimated psf. All the constraints were turned off in this case. The psf is
characterized by its basis function parameters.
First guess of Object
Detected Image
G
Pristine Object
True nsf
Estimated Image
Estimated Object
Distance of estimated
from truth psf
0.00042836
Estimated osf
Distance of first guessed
from truth psf
0.0015291
Figure 24 Results when all the constraints are set to zero.
Distance of estimated
from truth object
0.32402
Distance of first
guessed from truth
object
0.41394
Figs. 24 to 26 show the results from changing the weight PSme of the mean of the gradient
constraint from 0 to 100. Looking at the sizes of the estimated and truth psf in figs. 24, 25 and
26, we may conclude that PSm_e=100 is the optimum value, of those considered. The distance
from the estimated to the truth psf, 0.00038594, is the lowest of these three cases. The estimated
object looks better than that of the other two cases, even though the distance metric indicates that
the PSm_e=100 yields a slightly worse image than that of the two other cases. We would argue
that the small difference in the distance metric, less than 0.3%, can not by itself determine the
best looking image. This factor must be correlated with visual confirmation, which was carefully
done in this study. The detected and estimated images look very similar, indicating that our
results are self consistent and that our algorithms work!
First guess of Object
R
Pristine Object
Detected Image
G
True psf
Estimated Image
Estimated psf
Estimated Object
RI
Distance of first
Distance of estimated
from truth psf
0.00042223
Distance of first guessed
from truth psf
0.0015291
Figure 25 Results when the constraint PSm_e=10.
Distance of estimated
from truth object
0.32363
guessed from truth
object
0.41394
First guess of Object
P
r,
P
Firet nliwoef
nef
Detected Image
G
Pristine Object
True nsf
Estimated Imaqe
Estimated Object
Distance of estimated
from truth psf
0.00038594
Estimated psf
Distance of first guessed
from truth psf
0.0015291
Distance of estimated
from truth object
0.32489
Distance of first
guessed from truth
object
0.41394
Figure 26 Results when the constraint PSm e=100.
Figs. 27 to 30 present the results for varying PSs_e from 1 to 500. Again, looking at the
estimated images quality seems to indicate that the case PSs_e=l performed the best. In this
scenario, the distance between the estimated and truth psf was found to be 0.00040925. It is
worth emphasizing that that the distance between the estimated and pristine object can not be the
only factor in image evaluation. It is well understood that sometimes an optimum amount of
noise may make an image look more pleasant to a human. This topic is further discussed in the
framework of stochastic resonance 3, which is not covered in this thesis.
First guess of Object
B
First auess of osf
Detected Image
Pristine Object
True osf
Estimated Obiect
Estimated nsf
P
Distance of estimated
from truth psf
0.00040925
Distance of first guessed
from truth psf
0.0015291
Figure 27 Results when the constraint PSs e=1.
Distance of estimated
from truth object
0.32066
e
PO ZZm ZO PSm e PSs
Distance of first
guessed from truth
object
0.41394
First guess of Object
Detected Image
Pristine Object
Estimated Image
Estimated Object
Estimated psf
P PO ZZm
0 0 0
Distance of estimated
from truth psf
0.000533
Distance of first guessed
from truth psf
0.0015291
Figure 28 Results when the constraint PSs e=10.
Distance of estimated
from truth object
0.28692
PSm e PSs e
0
10
Distance of first
guessed from truth
object
0.41394
R
First guess of Object
G
B
First auess of osf
Detected Image
G
Pristine Object
True nsf
Estimated Object
7.
Estimated psf
P PO ZZm ZO PSm e PSs e
0 0 0 0
Distance of estimated
from truth psf
0.00079616
Distance of first guessed
from truth psf
0.0015291
Figure 29 Results when the constraint PSs_e=100.
Distance of estimated
from truth object
0.21903
0
100
Distance of first
guessed from truth
object
0.41394
R
First guess of Object
G
B
First auess of osf
Pristine Object
Estimated Object
Distance of estimated
from truth psf
0.0010833
Distance of first guessed
from truth psf
0.0015291
Distance of estimated
from truth object
0.19844
Distance of first
guessed from truth
object
0.41394
Figure 30 Results when the constraint PSse=500.
Figs. 31 to 36 depict results for varying P from 0.5 to 2080. As before, P is the coefficient of the
term constraining the sum of the psf pixels to equal to 1. Again, evaluating visually the estimated
objects, the best case seems to be produced when P=5. Although, the initial guesses differ
significantly from the pristine object, the algorithm was able to yield an answer that is very close
to the truth. The distance of the estimated and truth psf, 0.00028154, is the lowest of the six
cases for P=[0.5 1 5 10 1040 2080]. This quantitative determination is verified with a visual
evaluation of the psfs; the estimated and truth psf look very similar as shown in fig. 33.
Furthermore, consistency between the detected and estimated images is maintained, as shown in
the right side of figs. 31 to 36.
Figure 31 Results when the constraint P=0.5.
First guess of Object
R
R
Firzt ni aCnf nzf
Detected Image
Pristine Object
True nsf
Estimated Image
Estimated Object
Estimated psf
U
Distance of estimated
from truth psf
0.00034004
Distance of first guessed
from truth psf
0.0015291
Figure 32 Results when the constraint P=1.
PPO ZZm ZO PSm e PSs e
Distance of estimated
from truth object
0.32752
Distance of first
guessed from truth
object
0.41394
Figure 33 Results when the constraint P=5.
First guess of Object
R
G
B
First guess of psf
Detected Image
Pristine Object
True Dsf
Estimated Imaqe
Estimated Object
Distance of estimated
from truth psf
0.00052674
Estimated psf
Distance of first guessed
from truth psf
0.0015291
Figure 34 Results when the constraint P=10.
Distance of estimated
from truth object
0.36238
Distance of first
guessed from truth
object
0.41394
First guess of Object
R
G
B
First quess of psf
Detected Image
G
Pristine Object
True psf
Estimated Image
Estimated Obiect
Distance of estimated
from truth psf
0.0042761
Estimated Dsf
Distance of first guessed
from truth psf
0.0015291
Figure 35 Results when the constraint P=1040.
Distance of estimated
from truth object
0.95856
Distance of first
guessed from truth
object
0.41394
First guess of Object
G
B
First auess of osf
Detected Image
G
Pristine Object
True osf
Estimated Image
Estimated Object
Distance of estimated
from truth psf
0.0043563
Estimated psf
Distance of first guessed
from truth psf
0.0015291
Distance of estimated
from truth object
1.1815
Distance of first
guessed from truth
object
0.41394
Figure 36 Results when the constraint P=2080.
Figs. 37 to 42 display the results when PO varied from 0.001 to 240. Visual evaluation reveals
superior estimated image quality for PO=10. Additional confirmation is provided in the distance
between the estimated and truth psf, 0.0003083, being the lowest for the six values,
[0.001, 0.01,
0.1, 1, 10, 240], of PO studied. This parameter, enforcing the constraint that the energy in the
images and estimated object be equal is working very well in the algorithm.
First guess of Object
R
BR
First niips onfnsf
Detected Image
Pristine Object
True psf
Estimated Image
Estimated Object
Distance of estimated
from truth psf
0.00040174
Estimated psf
Distance of first guessed
from truth psf
0.0015291
Figure 37 Results when the constraint PO=0.001.
Distance of estimated
from truth object
0.32436
Distance of first
guessed from truth
object
0.41394
First guess of Object
R
G
B
First auess of osf
Detected Image
Pristine Object
True Dsf
Estimated Image
Estimated Object
Distance of estimated
from truth psf
0.00044527
Estimated psf
Distance of first guessed
from truth psf
0.0015291
Figure 38 Results when the constraint PO=0.01.
Distance of estimated
from truth object
0.32365
Distance of first
guessed from truth
object
0.41394
First guess of Object
Detected Image
Pristine Object
True osf
Estimated Image
Estimated Object
Distance of estimated
from truth psf
0.0004403
Estimated psf
Distance of first guessed
from truth psf
0.0015291
Figure 39 Results when the constraint PO=0.1.
Distance of estimated
from truth object
0.32398
Distance of first
guessed from truth
object
0.41394
First guess of Object
pR
R
Firft
rIIe of nf
Detected Image
Pristine Object
Estimated Image
Estimated Obiect
Distance of estimated
from truth psf
0.00037594
Estimated psf
Distance of first guessed
from truth psf
0.0015291
Figure 40 Results when the constraint PO=1
Distance of estimated
from truth object
0.32402
Distance of first
guessed from truth
object
0.41394
First guess of Object
G
B
First nip.ss of nsf
Detected Image
G
Pristine Object
True psf
Estimated Image
Estimated Object
Distance of estimated
from truth psf
0.0003083
Estimated psf
Distance of first guessed
from truth psf
0.0015291
Figure 41 Results when the constraint PO=10.
Distance of estimated
from truth object
0.33171
Distance of first
guessed from truth
object
0.41394
First guess of Object
G
B
First auess of Dsf
Detected Image
Pristine Object
True osf
Estimated Image
Estimated psf
Estimated Obiect
Distance of estimated
from truth psf
0.0021391
Distance of first guessed
from truth psf
Distance of estimated
from truth object
0.0015291
0.53287
Distance of first
guessed from truth
object
0.41394
Figure 42 Results when the constraint PO=240.
Figs. 43 to 47 show results from varying the weight ZZm for the object sharpness constraint that,
as mentioned above, is given in the form of
1
. This implies that the sharper the
(As
ofexample assume that an object
an
less
number
object
is,the
object is, the less number of pixels it occupies. As an example assume that an object
1 1 1
i1 1
1
0 1-1
0
00
02- =0
9 0
00
xy
and another object has all theses values concentrated in one pixel as
o02(x, y)2
. It is clear that the sharper object 02 will have a greater integrated energy
0
8 ithan that of object o1 where
01 (X, y)
2
=
9.
This manifests in the
xy
objective function as the inverse of the sharpness values, as given above. Therefore, the higher
the sharpness values, the lower their inverse is, and the lower their contribution to the objective
function. We recall that we are minimizing the objective function and the object pixel values are
positive.
Detected Image
R
Pristine Object
G
True Dsf
Estimated Imaae
Estimated Obiect
Distance of estimated
from truth psf
0.00042767
Estimated nf
Distance of first guessed
from truth psf
0.0015291
Figure 43 Results when the constraint ZZm=le-5.
Distance of estimated
from truth object
0.32401
Distance of first
guessed from truth
object
0.41394
First guess of Object
G
B
First auess of Dsf
Detected Image
R
Pristine Obiect
Tri
nf
Estimaterd nf
Estimated Obiect
Distance of estimated
from truth psf
G
Distance of estimated
from truth object
Distance of first guessed
from truth psf
0.0004115
0.0015291
0.32401
Distance of first
guessed from truth
object
0.41394
Figure 44 Results when the constraint ZZm=1.
First guess of Object
Detected Image
guess of Object
First
Estimated Imaae
Fstimatpd Chinr.t
Distance of estimated
from truth psf
0.00040862
Ftimntri ncf
Distance of first guessed
from truth psf
0.0015291
Figure 45 Results when the constraint ZZm=10.
Distance of estimated
from truth object
0.3236
Distance of first
guessed from truth
object
0.41394
First guess of Object
Detected Image
G
Pristine Object
B
True osf
Estimated Image
Estimated Object
Distance of estimated
from truth psf
0.00044963
Estimated psf
Distance of first guessed
from truth psf
Distance of first
guessed from truth
object
Distance of estimated
from truth object
0.0015291
0.32374
0.41394
Figure 46 Results when the constraint ZZm=100.
R
First guess of Object
G
B
First uess off
Detected Image
R
Pristine Object
G
B
True psf
Estimated Image
Estimated Object
Estimated psf
P
Distance of estimated
from truth psf
0.00045756
Distance of first guessed
from truth psf
0.0015291
Figure 47 Results when the constraint ZZm=1000.
PO 1ZZm ZO PSme PSs e
Distance of estimated
from truth object
0.32445
Distance of first
guessed from truth
object
0.41394
A careful visual evaluation of the estimated images in figs. 43 to 47 reveals that the value of
ZZm=10 produced the best result. The estimated and pristine objects look closer together than
for the other ZZm values. This is confirmed with the quantitative metric of distance between the
estimated and truth psfs of 0.00040862, the lowest value produced by the ZZm parameters
considered.
Varying the weight ZO yields results in figs. 48 to 52. We increased ZO from 5e-12 to 0.5. The
distances from the estimated and truth psf were computed and were found to be lowest for
ZO=5e-9. The quality of the estimated object was found to be the best, visually, for this value of
ZO also. Some other values of ZO yield significantly degraded results. For ZO=5e-6 and 5e-3,
the psf is extremely different from the truth psf, yielding estimated objects of low quality. This
happens because this constraint attempts to match the estimated and detected image energies. As
ZO becomes large, this constraint heavily weighs on the objective function resulting in energy
being more consistent while object distribution deviates significantly from the pristine object.
R
First guess of Object
G
B
First uess of
Detected
G Image
R
Pristine Object
True psf
Estimated Image
Estimated Object
Distance of estimated
from truth psf
0.00043501
Estimated psf
Distance of first guessed
from truth psf
0.0015291
Distance of first
guessed from truth
object
Distance of estimated
from truth object
0.32361
0.41394
Figure 48 Results when the constraint ZO=5e-12.
First guess of Object
Detected Image
R
G
B
Pristine Object
Estimated Image
Distance of estimated
from truth psf
0.0004049
Distance of first guessed
from truth psf
0.0015291
Figure 49 Results when the constraint ZO=5e-9.
Distance of estimated
from truth object
0.32631
Distance of first
guessed from truth
object
0.41394
First guess of Object
p
r,
p
Fir tAIIc Af r-f
Detected Image
G
Pristine Obiect
Tn ip nef
Estimated Image
Ftimtpd Ohipnt
Distance of estimated
from truth psf
0.0043231
F-timntrd nef
Distance of first guessed
from truth psf
Distance of first
guessed from truth
object
Distance of estimated
from truth object
0.0015291
1.6314
0.41394
Figure 50 Results when the constraint ZO=5e-6.
First guess of Object
Detected Image
R
Pristine Obiect
G
Tri ua nf
Estimated Imaae
F-timqtpd OChinr.t
Distance of estimated
from truth psf
0.0046361
Fqtimatd nqf
Distance of first guessed
from truth psf
0.0015291
Figure 51 Results when the constraint ZO=5e-3.
Distance of estimated
from truth object
1.8327
Distance of first
guessed from truth
object
0.41394
Detected Image
R
G
B
Pristine Object
Estimated Image
I~B~g~l~a4
Estimated Object
Estimated psf
P PO ZZm ZO PSm e PSs e
Distance of estimated
from truth psf
0.0012296
Distance of first guessed
from truth psf
0.0015291
Distance of estimated
from truth object
1.6084
Distance of first
guessed from truth
object
0.41394
Figure 52 Results when the constraint ZO=0.5.
To further evaluate the effect of the various weights, we changed more than one factor at a time.
The cases run are displayed in fig. 53 for various values of the weights. We can see clearly that
the best result is obtained in fig. 61 for [P PO ZZm ZO PSm_e PSs_e] =[1 10 1 5e-12 10 1]. The
estimated and true psf distance is also the lowest at 0.00027408. This value is the lowest of all
the cases considered in this chapter. The maximum likelihood algorithm that we have derived
worked!
Figure 53 Table of cases run by changing the weights many at a time.
Detected Image
G
Pristine Object
True osf
Estimated Imane
Estimated Obiect
Distance of estimated
from truth psf
0.00042553
Fstimated nf
Distance of first guessed
from truth psf
0.0015291
Distance of estimated
from truth object
Distance of first
guessed from truth
object
0.35096
Figure 54 Results for the constraints [P PO ZZm ZO PSm_e PSs_e] =[5 10 10 5e-12 10 11.
0.41394
First guess of Object
Detected Image
G
B
Pristine Object
Estimated Image
F
timntrl rhi
at
Distance of estimated
from truth psf
0.00050904
Estimated psf
Distance of first guessed
from truth psf
0.0015291
Distance of estimated
from truth object
0.36015
Distance of first
guessed from truth
object
0.41394
Figure 55 Results for the constraints [P PO ZZm ZO PSm_e PSs_e] =[5 1 10 5e-12 10 1].
First guess of Object
Pristine Object
Distance of estimated
from truth psf
0.00067455
Distance of first guessed
from truth psf
0.0015291
Distance of estimated
from truth object
0.37307
Figure 56 Results for the constraints [P PO ZZm ZO PSm_e PSs_el =[10 1 1 5e-12 10 1].
Distance of first
guessed from truth
object
0.41394
First guess of Object
Detected Image
R
Pristine Object
G
B
True psf
Estimated Image
Estimated Object
Distance of estimated
from truth psf
0.00055465
Estimated psf
Distance of first guessed
from truth psf
Distance of first
guessed from truth
object
Distance of estimated
from truth object
0.0015291
0.36109
0.41394
Figure 57 Results for the constraints [P PO ZZm ZO PSm_e PSs_e =[10 1 1 5e-12 1 1].
First guess of Object
Detected Image
G
Pristine Object
Estimated nsf
Distance of estimated
from truth psf
0.00036719
Distance of first
Distance of first guessed
from truth psf
0.0015291
Distance of estimated
from truth object
0.32342
Figure 58 Results for the constraints [P PO ZZm ZO PSm_e PSs_e] =[1 1 1 5e-12 1 11.
guessed from truth
object
0.41394
First nifC nf nzf
Detected Image
Pristine Object
True psf
Estimated Imaae
Estimated Obiect
Distance of estimated
from truth psf
0.00028961
Estimated Dsf
Distance of first guessed
from truth psf
Distance of first
guessed from truth
object
Distance of estimated
from truth object
0.0015291
0.33044
0.41394
Figure 59 Results for the constraints [P PO ZZm ZO PSm_e PSs_e] =[1 10 1 5e-12 1 1].
First guess of Object
R
Pristine Object
Detected Image
G
B
True psf
Estimated Image
Estimated Object
Distance of estimated
from truth psf
0.00028428
Estimated psf
Distance of first guessed
from truth psf
0.0015291
Distance of estimated
from truth object
0.3308
Figure 60 Results for the constraints [P PO ZZm ZO PSm_e PSs_e] =[1 10 10 5e-12 1 11.
Distance of first
guessed from truth
object
0.41394
R
First guess of Object
G
B
First auess of nsf
Detected Image
Pristine Object
True nsf
Estimated Image
Estimated psf
Distance of estimated
from truth psf
Distance of first guessed
from truth psf
0.00027408
Distance of first
guessed from truth
object
Distance of estimated
from truth object
0.0015291
0.33092
0.41394
Figure 61 Results for the constraints [P PO ZZm ZO PSm_e PSs_e] =[1 10 1 5e-12 10 1].
First guess of Object
R
G
B
First uess of
Detected Image
Pristine Object
True nsf
Estimated Imane
IPctimat
Distance of estimated
from truth psf
0.00035602
l (hiort
Distance of first guessed
from truth psf
0.0015291
Distance of estimated
from truth object
0.32386
Figure 62 Results for the constraints [P PO ZZm ZO PSm_e PSs_e =[1 1 1 5e-12 10 1].
Distance of first
guessed from truth
object
041394
First guess of Object
G
B
First uess of
Detected Image
Pristine Obiect
TnrIp n-.f
Estimated Image
Estimated Object
Distance of estimated
from truth psf
0.0003647
Estimated psf
Distance of first guessed
from truth psf
0.0015291
Distance of estimated
from truth object
0.32391
Distance of first
guessed from truth
object
0.41394
Figure 63 Results for the constraints [P PO ZZm ZO PSm_e PSs_e] =[1 1 0 10 1].
4-4 References
1. R. Fergus, B. Singh, A. Hertzmann, S. Roweis, and W. T. Freeman, "Removing camera shake
from a single image, " SIGGRAPH 2006.
2. D. Field, "What is the goal of sensory coding?" Neural Computation, 6, 559-601 (1994).
3. S. Mitaim and B. Kosko, "Neural fuzzy stochastic resonance," IEEE International Conference
on Systems, Man, and Cybernetics, 3, 2237 - 2242 (1998)
Chapter 5 Future work
Many issues have been resolved in this paper. We have designed a rather comprehensive
experiment to address the problem of single image enhancement. More remains to be done. One
topic that needs to be investigated is the added value, if any, additional constraints using the
higher order moments of the gradient of the object would bring. Another topic that needs to be
investigated is whether adding another detection channel would better constrain the problem and
yield a better solution. This additional channel should be such that no additional unknowns are
introduced. One way to accomplish that is to use another detector that detects the integrated
intensity, i.e. sums the three colored images. This measurement architecture would serve to
constrain the sum of the estimated images to add up to that integrated measured intensity. Since
the optical system did not change, the psf remained the same and no additional unknowns would
be introduced.
To increase convergence rate, it may be advantageous to not only express the psf but also the
object as a set of basis functions. The object expansion may be more challenging since there is
many spatial frequency information that this expansion would need to capture. Finding an
appropriate basis function set for feature rich scenes would be a great contribution towards high
resolution image enhancement.
Another possible research are includes hybrid image enhancement. When looking at images, part
of the image may be recognized while other parts may not be. If features are available for the
recognized parts, then it may be useful to isolate those parts and enhance only the unrecognized
portions. The final enhanced image would then be a combination of the separate pieces, stitched
together such that edge boundaries are avoided. Some smoothing at the blocks boundaries may
be required.