Remote Engineered Super
Resolved Imaging
Zeev Zalevsky
Faculty of Engineering,
Bar-Ilan University, 52900 Ramat-Gan, Israel
1
Outline
•Introduction
•The “SW Adaptation” Process
•Diffractive type Super Resolution
•Geometrical type Super Resolution
•Hearing with light
•Conclusions
2
Outline
•Introduction
•The “SW Adaptation” Process
•Diffractive type Super Resolution
•Geometrical type Super Resolution
•Hearing with light
•Conclusions
3
Introduction-Diffraction Limitation
What is Resolution?

Resolution is finest spatial feature that an
imaging system can resolve.

Resolution of optical systems is restricted by
diffraction (Lord Rayleigh, Abbe), by the
geometry of the detector and by the noise
equivalence of its pixels.
4
Introduction-Diffraction Limitation
Diffraction limitation of resolution is proportional to
the F number of the imaging optics.
xRES  MIN {X }  1.22f #  1.22
f
B
5
Introduction- Geometrical Limitation
Geometrical resolution is limited by the number
of detector’s pixels and their size.
6
IntroductionNoise Equivalent Resolution
Noise equivalent resolution is originated by the
internal noises existing within each pixel of the
detector (electronic noises, shot noises etc).
7
Outline
•Introduction
•The “SW Adaptation” Process
•Diffractive type Super Resolution
•Geometrical type Super Resolution
•Hearing with light
•Conclusions
8
SW Adaptation Process
If not resolved, is it hopeless?
No, if A Priori information
on the object is available!!!
Types of a priori information:
• A single dimensional object
• Polarization restricted information
• Temporally restricted signal
• Wavelength restricted signal
• Object shape
9
SW Adaptation Process- cont.
The Suggested Solution:
Having a priori knowledge of the signal may lead
to super resolution using an SW (space-bandwidth)
adaptation process:
Adapt the SW of the signal to the
acceptance SW of the system
1 for all W( x, )  Wtresh
SW( x, )  
0 otherwise

 SW(x, )dxd  N
10
Outline
•Introduction
•The “SW Adaptation” Process
•Diffractive type Super Resolution
•Geometrical type Super Resolution
•Hearing with light
•Conclusions
11
Diff. SR- Time Multiplexing
Time Multiplexing:
Conversion of temporal degrees of freedom to
spatial domain (diffraction)
Ap.
Obj.
Img.
 ... dt
G1
G2
Synchron. moving gratings
The structure of the rotated
grating (for 2-D S.R. effect)
Recent improvements:
•Automatic synchronization (one grating, transmitted twice)
•2-D objects, 2-D gratings
•Dammann gratings
12
Diff. SR- Time Multiplexing
Diff. SR- Time Multiplexing
With clear
aperture
Without time
multiplexing
With time
multiplexing
14
Remote Diff. SR
Open aperture
Reconstruction
Projected grating
Closed aperture
15
Diff. SR- Speckle Projection
Closed aperture
Reconstruction
Coherent
Incoherent
16
Remote Diff. SR- via Background
Open aperture
Closed aperture
Reconstruction
17
Remote Diff. SR- via background,
cont.
Open aperture
Background
Reconstruction
Closed aperture
Closed aperture sequence
Reconstruction
18
Remote Diff. SR- from satellite
Numerical simulations
Experimental results
19
Remote Diff. SR- via rain/droplets
Open aperture
Closed aperture
Reconstruction
20
Spatial DLP based SR
CAMERA
DMD1
Two possible
experimental setups.
DMD2
FOV
FOV SR.
21
FOV improvement 3X3
Raw images (different DMD positions)
y o1
Sensor size image
20
40
60
80
y o2
20
40
60
80
20 40 60 80 100
Sensor FOV
20
20
40
60
80
20 40 60 80 100
y o4
10
y o3
20 40 60 80 100
y o5
y o6
30
40
20
40
60
80
50
60
70
80
90
10
20
30
40
50
60
70
80
90
100
20
40
60
80
20
40
60
80
110
20 40 60 80 100
20 40 60 80 100
y o7
20
40
60
80
20
40
60
80
20 40 60 80 100
20 40 60 80 100
y o8
y o9
20
40
60
80
20 40 60 80 100
20 40 60 80 100
22
FOV improvement 3X3
Two types of algorithms (LSQR, L1)
LSQR N=8
N = 8 DMDs positions
L1 N=8
Original FOV = middle square only
23
Spatial DLP based SR
The experimental setup.
(a).
(c).
(b).
Experimental results: (a). Image captured with open iris. (b).
Image captured with semi closed iris. (c). Reconstructed super
resolved image (demonstrating optical zooming of 3x).
24
Optical SAR- I
Focus
plane 1
Sufficiently far
distance to
justify far field
approximation
Random phase 1
Imaging
lens
Detector
Focus
plane 2
A1 amplitude of
image with scale 1
A1 exp(i1 )
Scale transformation (based on
zero padding in Fourier domain)
from 1 to 2
B2 exp(i2 )
Movement
direction
Remote scene
A2 amplitude of
image with scale 2
If MSE reaches minima
then stop
A2 exp(i2 )
Scale transformation (based
on zero padding in Fourier
domain) from 2 to 1
B1 exp(i1 )
Left: Schematic sketch of the proposed configuration. Right: The proposed iterative algorithm.
25
Optical SAR- II,
Simulations
50
50
50
100
100
100
150
150
150
200
200
200
(a).
(b).
250
50
100
150
250
250
200
(c).
250
50
100
150
200
250
50
100
150
200
250
Numerical simulations. (a). Original image. (b). Its reconstruction. (c). The type of reconstruction that is
obtained when the phase is wrongly reconstructed.
350
300
250
200
150
100
50
0
0
50
100
150
200
250
300
The unwrapped phase of the Fourier of the
original image and the reconstructed phase.
Left: Original object. Middle: Original object Fourier
transform. Right: Blurred image.
26
Optical SAR- III,
Experimental results
27
Outline
•Introduction
•The “SW Adaptation” Process
•Diffractive type Super Resolution
•Geometrical type Super Resolution
•Hearing with light
•Conclusions
28
Geometrical SRIntermediate plane mask
(a).
(b).
(a). High resolution ref.
(b). Low resolution image
without SR
(a).
(b).
(c).
Reconstruction: (a). Field of view border condition. (b). High resolution mask in the
intermediate image plane. (c). Low resolution mask in the intermediate image plane.
Mask+sensor are shifted for the micro scanning process
29
Geometrical SR- Improved Technique
Intermediate
Image Plane
Object
Plane
(a).
(b).
Image Plane
Moving Mask
Lens
CCD
(a) A random mask with size of . (b) Optical configuration including the binary mask.
The mask has to be in an intermediate imaging position. The mask can be moved only in
one direction to get different images each time.
30
Geometrical SR- Improved Technique
Improvements:
•The mask is the only part being shifted (instead of
mask+sensor).
•The SR is achieved, without any spatial loss of information.
•The reconstruction has a reduced sensitivity to noise.
•Although the movement of the mask is in 1-D, the
obtainable SR is 2-D.
•The movement of the mask does not have to be in sub pixel
steps.
•The recovery time is improved (the reconstruction process
each pixel can be treated separately & simultaneously).
31
Geometrical SR- Simulations vs. SR factor
(a).
(a).
(b).
(b).
(d).
(d).
(d).
Super Resolution Factor (Blocked Area 50%, VarNoise 0.001)
(c).
(c).
(e).
(e).
(e).
(f).
(f).
(g).
(g).
(h).
(h).
(i).
(i).
(a).
Simulation results
depicting the algorithm
dependence on the super
resolution factor. Image
Size: 256256, Number of
images(b).taken during the (c).
process = 2×SR factor,
percentage of the image
covered by the random
mask = 50%, Noise
variance = 0.001. (a).
Reference image.
(d).
(e).
Low resolution images
((b), (d), (f), (h)) and their
corresponding high
resolution reconstructions
((c), (e), (g), (i)) for SR
factor of 4X4, 8X8, 12X12
and 16x16
respectively. (g).
(f).
Dependence of the algorithm and the
runtime on the super resolution factor.
Number of images taken during the process
= 2× SR factor, the size of the ring image is
256×256 pixels. It has been obtained by
performing matrix inversions. The size of
each matrix is 2×SR factor rows, and SR
factor columns.
32
Geometrical SR- Experimental results
Spherica
l Mirror
Auto
Collimator
Auto
Collimator
Detector
Relay
Lens
Folding
Mirror
Apertur
e Stop
Manual
micromete
r
Binary
Mask
The experimental setup
33
Geometrical SR- Experimental results
Upper row: The left image: Central part of a low
resolution image. The right image: Resulting
reconstructed higher resolution image.
Lower row: The cross sections.
34
Outline
•Introduction
•The “SW Adaptation” Process
•Diffractive type Super Resolution
•Geometrical type Super Resolution
•Hearing with light
•Conclusions
35
Opto-Phone: Hearing with Light
Hearing with Light: Features
•The ultimate voice recognition system compatible to
“hear” human speech from any point of view (even from
behind).
•There is no restriction on the position of the system in
regards to the position of the sound source.
•Capable of hearing heart beats and knowing physical
conditions without physical contact for measuring.
36
Opto-Phone: Hearing with Light
Features- cont.
•Works clearly in noisy surroundings and even through
vacuum.
•Allows separation between plurality of speakers and
sounds sources.
•Works through glass window.
•Simple and robust system ( does not include
interferometer in the detection phase).
37
Opto-Phone: Hearing with Light
Imaging
module
Camera
Sensor
Invisible L
aser
projection
Laser
Any visible distance
Camera
Imaging
lens
Laser
38
Let’s listen…from 80m
Cell phone
Back part of neck
Counting…1,2,3,4,5,6
Counting…5,6,7
X - movement
1
0.5
0
-0.5
Face (profile)
Counting…5,6
-1
120
140
160
180
200
220
240
260
280
300
320
Heart beat pulse
taken from a throat
All recordings were done in a very noisy constriction site at distance of more
than 80m.
39
Results: Detection of
occluded objects I
(a).
(b).
(c).
(a). Camouflaged object. (b). Camouflage without the object. (c). The object (upper left part) and the low
resolution camouflaged scenery.
Spectrogram
Spectrogram
Spectrogram
0
0
0
2000
50
900
50
1800
100
50
12
800
100
100
1600
700
200
1200
600
200
200
250
500
300
400
600
350
300
350
400
400
400
200
400
450
200
450
100
450
250
1000
300
800
350
10
150
Frequency [Hz]
150
1400
Frequency [Hz]
Frequency [Hz]
150
8
250
6
300
4
2
500
(d).
0
0.1
0.2
0.3
0.4
Time [sec]
0.5
0.6
0.7
500
(e).
500
0
0.1
0.2
0.3
0.4
Time [sec]
0.5
0.6
0.7
(f).
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Time [sec]
(d). The spectrogram of the camouflaged object with its engine turned on. (e). The spectrogram
of the object with its engine turned on and without the camouflage. (f). The spectrogram of the
camouflaged object without turning on its engine.
40
Results: Detection of
occluded objects II
Y - pos
10
(a).
(b).
5
0
-5
Y - pos
-10
10
5
-15
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
Sample
0
-5
-10
-15
0
-20
0
1
1000
2
3
2000
4
3000
5
4000
6
5000
7 [sec]
Sample
(a). The scenario of the experiment. (b). Experimental results:
upper recording is of the
camouflaged subject. Lower recording is the same subject without the camouflage.
6000
41
Outline
•Introduction
•The “SW Adaptation” Process
•Diffractive type Super Resolution
•Geometrical type Super Resolution
•Hearing with light
•Conclusions
42
Conclusions:
•
•
•
•
•
Resolution of optical system is restricted by various
terms.
SW Adaptation process is a useful tool for designing
super-resolution systems.
A generalization for handling more types of resolution
restrictions was introduced for large variety of
applications.
Examples of achieving super resolution effects were
viewed.
New approach for “hearing” with light was
demonstrated.
43