Super-Resolution
Authors: Wagner, Waagen and Cassabaum
Presented By: Mukul Apte
Introduction


Definition: Create High Resolution visual output from
Low Resolution visual input.
Mathematical assistance to features viz. motion
detection, face recognition, person detection.





Action Packed Sports Images
Astronomy
Medical Imaging
Surveillance
Range of resources: low (single camera, polaroid lenses)
to high (super-resolution chips, androids-ASIMO)
Basic idea
Given: A set of degraded
(warped, blurred, decimated,
noised) images:
Required: Fusion of the
measurements into a higher
resolution image:
Types
Static Super-Resolution (SSR) - The
creation of a single improved image, from
the finite measured sequence of images
 Dynamic Super-Resolution (SSR) - Low
Quality Movie In - High Quality Movie Out

Simple Example
Concepts I: Raster, CCD, Image in
mathematical domain, Matlab
 Concepts II: Bresenham’s line algorithm,
non-uniform interpolation, frequency
domain

Simple Example
For a given band-limited
image, the Nyquist
sampling theorem states
that if a uniform sampling
is fine enough (pixel
density = twice highest
static frequency), perfect
reconstruction is
possible.
D
D
Simple Example
Due to our limited
camera resolution,
we sample using
an insufficient 2D
grid
2D
2D
Simple Example
However, we are
allowed to take a
second picture and
so, shifting the
camera ‘slightly to
the right’ we obtain
2D
2D
Simple Example
Similarly, by
shifting down we
get a third image
2D
2D
Simple Example
And finally, by
shifting down
and to the right
we get the fourth
image
2D
2D
Simple Example - Conclusion
It is trivial to see that
interlacing the four
images, we get that the
desired resolution is
obtained, and thus
perfect reconstruction is
guaranteed.
This is SuperResolution in its
simplest form
Uncontrolled Displacements
In the previous example
we counted on exact
movement of the camera
by D in each direction.
What if the camera
displacement is
uncontrolled?
Uncontrolled Displacements
It turns out that there is a
sampling theorem due to
Yen (1956) and Papoulis
(1977) covering this case,
guaranteeing perfect
reconstruction for periodic
uniform sampling if the
sampling density is high
enough (1 sample per
each D-by-D square).
Uncontrolled Rotation/Scale/Disp.
In the previous examples
we restricted the camera
to move
horizontally/vertically
parallel to the photograph
object.
What if the camera
rotates? Gets closer to
the object (zoom)?
Uncontrolled Rotation/Scale/Disp.
There is no
sampling
theorem
covering this
case
Further Complications



Sampling is not a point
operation – there is a blur
Motion may include
perspective warp, local motion,
etc.
Samples may be noisy – any
reconstruction process must
take that into account.
Static Super-Resolution
Y1
YN
Y2 Y3
Low Resolution Measurements
Static
Super-Resolution
Algorithm
X̂  f Y1, Y 2 , Y 3 ,  , Y N 
X̂
High Resolution
Reconstructed Image
t
Dynamic Super-Resolution
Low Resolution Measurements
Yt t
t


X̂t   f Yt , X̂t  1
High Resolution Reconstructed Images
Dynamic
Super-Resolution
Algorithm
t
X̂t 
t
Approach



Image Registration
Motion Estimation
Projection onto High-Resolution Grid
 Non-Uniform
Interpolation
 Frequency domain – alias correction
Projection
Registration
Low-res Images
Registration
(sub-pixel grid)
High Res Grid
1.1 Registration (angle)
•Rotation Calculation
•Correlate 1st LR image with all LR images at all angles
OR
•Calculate energy at all angles for all LR images. Correlate
energy vector to find the rotation angle
Anglei = max index(correlation(I1(θ), Ii (θ)))
LR image 1
LR image 2
i = 2,3,..,N (number of LR
images)
Energy at angle Ii(θ)
Energy at angle I2(θ)
1.1 Registration (shift)

Shift Calculated using Frequency Domain
Method
Fi (uT) = ej2πuΔsF1(uT)
Δs  [Δx Δy]T
Δs = angle( Fi
(uT)
/ F1
(uT)
u  [fx fy]
)
2πu
• Used only 6% lower u (high freq could be aliased)
• Used least square to calculate Δs
2.1 Frequency Domain



Input: Down-sampled aliased images
Goal I: Correct the low-freq aliased data
Goal II: Predict the lost high freq values
-π
π
Original High-Res
-π
π
Down-sampled
Aliased (fix it)
Lost (find it)
-π/2
π/2
Up-sampled
π
-π
π
Desired High-Res
2.2 Projection onto High-res grid

Papoulis-Gerchberg Algorithm
 Projection
onto convex sets
Known pixel values
 Known Cut-off freq in the HR image

 Algorithm:
I (known pixel positions) = Known Values
I_fft = fft2(I)
I_fft(higher Freq) = 0
I= ifft2 (I_fft)
SSR – The Model
Geometric Warp Blur
HighResolution
Image
F1=I
H1
Decimation
Y1
D1
V1
X
LowResolution
Images
Additive Noise
FN
HN
YN
DN
VN
N

1 
 Y k  Dk H k Fk X  V k , V k ~ N0, Wk  

k 1
Warp – Linear Operation
Z
X
Per every
point in X
find a
matching
point in Z
 x1  
   
  
xj   
  
   
x N  
1
0
1
F[j,i]=1
0
  z1 
  
 
 z j 
 




1  z N 
Model Assumptions
We assume that the images Yk and the operators Hk,
Dk, Fk,& Wk are known to us, and we use them for
the recovery of X.
Yk – The measured images (noisy, blurry, down-sampled ..)
Hk – The blur can be extracted from the camera characteristics
Dk – The decimation is dictated by the required resolution ratio
Fk – The warp can be estimated using motion estimation
Wk – The noise covariance can be extracted from the camera
characteristics
Special Condition - Noiseless
 Y1   D1H1F1 
Y   D H F 
 2    2 2 2 X
   


  

Y
D
H
F
 N  N N N
Clearly, this linear system of equations should have
more equations than unknowns in order to make it
possible to have a unique solution.
SSR – Handling Problems
Single image de-noising
 Y  X  V
Single image restoration
 Y  HX  V
Single image scaling
 Y  DX  V
Motion compensation average


 Yk  Fk X  V 
T
Using AX  X ST WS X

1
X̂  H H  S WS HT Y

T
T

1
X̂  DT D  ST WS DT Y


X̂   FkT Fk  ST WS 
 k 1

N
N
k k 1

1
X̂  I  S WS Y
T
1 N
T
F
 k Yk
k 1
SSR Standard Equation

N

X̂ j1  X̂ j   FkT HTk DTk Wk Y k  Dk H k Fk X̂ j  ST WS X̂ j
k 1
Back
projection
Simulated
error
Weighted
edges
All the above operations can be interpreted as
iterative operations performed on images.
(Typically 15-20 iterations are performed.)
Thumb Rule on Desired Resolution

Assume that we have N images of M-by-M
pixels, and we would like to produce an
image X of size L-by-L. Then –
L  N M
Papoulis – Gerchberg Algorithm
Initial Setup
FFT (Initial image)
Taj Mahal – Low-res image I
Papoulis – Gerchberg Algorithm
Known Pixel
Values
Image after 1st iteration
Image at iteration 0
FFT
I(high freq) =0
Papoulis – Gerchberg Algorithm
Known Pixel
Values
Image at iteration 1
Image after 10 iterations
FFT
I(high freq) =0
Papoulis – Gerchberg Algorithm
After 50 iterations
Bilinear Interpolation
Bicubic Interpolation
SR Reconstructed image
Results (Images - I)
• Input: 4 snaps using a high-res digital camera
• Cropped the same part of each image
• SR algorithm compared with bicubic interpolation
Original Low-res images
(Courtesy: Patrick Vandewalle)
Results (Images - I)
Bicubic Interpolation
Results (Images - I)
Super-resolution
Results (Images - II)
Low-Res Image I
Low-Res Image II
• SR Algorithm didn’t work as expected !!!
• Reason:
• Motion was not restricted to shifts & rotation
• Images had affine mapping
• Rule I - Need Correct Registration
Results (Frames - I)
Original
Bicubic
SR
Why didn’t SR work???
• Low-res images were created by forcing shifts at
critical velocities
• Rule II - If low-res frames are at critical velocities,
can’t create good high-res frame
Results (Frames - II)
Original
Bicubic
SR
Why did SR work so well???
• Low-res images were created by forcing shifts at
non-critical velocities
• Rule III  If low-res images have all the info
about high-res then HR image can be perfectly
constructed
Future Work
• Superresolution with multiple motions between
frames - create high res video
• Predict the high-res frequency components using
wavelet methods
Predict
Predict
Predict
Acknowledgements
Prof John Apostolopoulos
 Prof Susie Wee
 Patrick Vandewalle

THANK YOU!!!
QUESTIONS?
COMMENTS?