National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
Depth Estimation and
Focus Recovery
(景深估計與聚焦重建)
Speaker: Yu-Che Lin ( 林于哲 )
Adviser : Prof. Jian-Jiun Ding (丁建均 教授)
Digital Image and Signal Processing
Laboratory (DISP) MD-531
Institute of Communication Engineering
Outlines
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
 Motivations
 Overview on previous works
Feb. 2008
• Structure of camera lens / Geometric optics
• Introduction to Fourier optics
• Blurring function / Equal-focal assumption
 Binocular / Stereo vision
• Vergence
 Monocular
• Depth from focus
 Estimator of degree on focus / Sum of Laplacian
 Interpolation
• Depth from defocus
 Arbitrary changing camera parameters with large variation
 Trace amount on changing camera parameters
Digital Image and Signal Processing
Laboratory (DISP) MD-531
2
Institute of Communication Engineering
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
 Linear canonical transform (LCT) upon the optical system
Feb. 2008
• Linear canonical transform (LCT)
• Approximation on the optical system by LCTs
 Focus recovery: common method, alternative method
• The common method
• Alternative method: one point focus recovery
• Simulation on simple pattern
 Conclusions
 References
Digital Image and Signal Processing
Laboratory (DISP) MD-531
3
Institute of Communication Engineering
Motivations
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
 Depth is an important information for robot and the 3D reconstruction.
 Image depth recovery is a long-term subject for other applications such as
robot vision and the restorations.
 Most of depth recovery methods based on simply camera focus and defocus.
 Focus recovery can help users to understand more details for the original
defocus images.
Digital Image and Signal Processing
Laboratory (DISP) MD-531
4
Institute of Communication Engineering
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
Overview on previous works
Digital Image and Signal Processing
Laboratory (DISP) MD-531
Institute of Communication Engineering
Structure of camera
lens (1)
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
 Physical lens.
cited on :
http://en.wikipedia.org/wiki/Concave_lens#Types_
of_lenses
Feb. 2008
Higher wavelength,
lower refractive
index
 Structure of camera lens against the aberration (像差)
• Two of aberrations : Chromatic (色像) , Spherical aberration.
convex
concave
cited on :
Digital Image and Signal Processing cited on :
6
http://en.wikipedia.org/wiki/Concave_lens#Types_ Laboratory (DISP) MD-531
http://en.wikipedia.org/wiki/Index_of_refraction
of_lenses
Institute of Communication Engineering
1 1 1
 
u v F
Structure of camera
lens (2)
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
 Common solutions for aberrations:
• “Asperical” lens and complementary lenses
(Groups).
• One shot always has multiple lenses.
Feb. 2008
Complementary convex
and concave lenses
 Geometric on imaging.
cited on :
F : focal length u: object dist.
http://www.schneideroptics.com/info/photography.
1 1 1
htm
 
v: imaging dist. D: lens diameter
u v F
R: blurring radius s: dis. between lens and screen (CCD)
Digital Image and Signal Processing
Laboratory (DISP) MD-531
7
Institute of Communication Engineering
Structure of camera
lens (3)
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
 Combination by lenses of the real camera.
• The effective focal length :
1 1 1
L
  
F F1 F2 F1 F2
• Due to the above effective value, we
can now just ignore the complicated
combinations.
Digital Image and Signal Processing
Laboratory (DISP) MD-531
L1
L2
F
8
Institute of Communication Engineering
Introduction to Fourier
optics (1)
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
 Aperture effect.
When the wave incident through an aperture,
the observed field is the combination:
1. The unperturbed incident wave by geometric optics.
2. A diffractive wave originating from the rim of the aperture.
 Diffraction.
•
•
Fresnel principle ( near-field diffraction )
... . ..
h  x0 , y0  U  x0 , y0 
 
 k 
e jk z
2
2 
U  x0 , y0  
U  x1 , y1  exp  j
 x0  x1    y0  y1    dx1dy1


j z  
 2z

Fraunhofer principle ( far-field diffraction )
U  x0 , y0  
e
jk z
e
j

k
x02  y02
2z
j z

F U  x1 , y1  
Digital Image and Signal Processing
Laboratory (DISP) MD-531
strict on field
distance, z
9
Institute of Communication Engineering
Introduction to Fourier
optics (2)
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
 The Huygens-Fresnel transform.
• Considering a square wave :
x-axis [0,200]
x-axis [0,200]
Square wave
After Huygens-Fresnel Transform
z=20
y-axis [0,200]
Square wave
After Huygens-Fresnel Transform
z=14
y-axis [0,200]
Square wave
After Huygens-Fresnel Transform
z=1
y-axis [0,200]
y-axis [0,200]
Square wave
Before Huygens-Fresnel Transform
x-axis [0,200]
Digital Image and Signal Processing
Laboratory (DISP) MD-531
x-axis [0,200]
10
Institute of Communication Engineering
Introduction to Fourier
optics (3)
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
 The field intensity through the circular aperture (ex: camera aperture )
by unit amplitude plane wave under Fraunhofer diffraction theory
is actually a sinc function .
 The structure inside the camera shot should more like a near-field condition
, so the intensity pattern acts more like a Gaussian function.
y1
y0
x1
r01
P1(x1,y1)
x0
P0(x0,y0)
Digital Image and Signal Processing
Laboratory (DISP) MD-531
11
Institute of Communication Engineering
Blurring function /
Equal-focal assumption
(1)
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
screen
F
F
Blurring radius: R<0
D/2
s
u
2R : R<0
Biconvex
v
F
F
Blurring radius: R>0
D/2
s
u
2R : R>0
v
screen
Digital Image and Signal Processing
Laboratory (DISP) MD-531
12
Institute of Communication Engineering
Blurring function /
Equal-focal assumption
(2)
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
 Due to geometric optics, the intensity inside the blur circle should
be constant.
 Considering of aberration and diffraction and so on, we easily assume
a blurring function:
•

: diffusion parameter
Feb. 2008
K: calibrated by each specific camera
 x2  y 2 
h  x, y  
exp  

2
2
2
2



 Diffusion parameter is related to blur radius:
1
  kR
• Derived from triangularity in geometric optics
 For easy computation, we always assume that
foreground has equal-diffusion, background has
equal-diffusion and so on
 However, this equal-focal Digital
assumption
Image and Signal Processing
Laboratory (DISP) MD-531
will be a problem

Ds  1 1 1 
R


 F  s  u 
2


FsD
u 

sD   2 R  D  F
13
Institute of Communication Engineering
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
Binocular / Stereo vision
Digital Image and Signal Processing
Laboratory (DISP) MD-531
Institute of Communication Engineering
Vergence
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
 Vergence movement :
•
is some kind of slow eye movement that two eyes move in different directions.
 Disadvantage :
 sin 2 l   r  cos 2 r cos 2 l 
• Correspondence problem ( trouble ). z  B 


2
2
4sin



sin







l
r
l
r 

2
2
Digital Image and Signal Processing
Laboratory (DISP) MD-531
15
Institute of Communication Engineering
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
Depth from focus
Digital Image and Signal Processing
Laboratory (DISP) MD-531
Institute of Communication Engineering
Estimator of degree on
focus / Sum of
Laplacian
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
 Actively taking pictures at different observer distance or object distance .
 Estimator of degree on focus.
• we need an operator to abstract how “ focused ” the region is
 Since the blur model is a low pass filter, the estimator can be a Laplacian
 2 Ib  2 Ib
 Ib  2  2  ML  x, y 
x
y
2
 Such operator point to a measurement on a single pixel influence, a sum of
Laplacian operator is needed:
n  i, j  
j k
ik
  ML u, v 
v  j  k u i  k
Digital Image and Signal Processing
Laboratory (DISP) MD-531
17
Institute of Communication Engineering
Interpolation (1)
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
 We use Gaussian interpolation to form a set of approximations.
Focus measure
[SML]
NP
Feb. 2008
Measured curve
Nk
Ideal condition
Nk-1
Nk+1
dk- dp d
k
1
dk+1
displacement
 We have dp that is the camera displacement performing perfect focused :
2

 1  d  d p  

N

N

exp

 
 ,
p
• 
2

 

 

  k  1, k , k  1
Digital Image and Signal Processing
Laboratory (DISP) MD-531
18
Institute of Communication Engineering
Interpolation (2)
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
 The depth solution dp from above Gaussian :
Feb. 2008
 ln N k  ln N k 1   d k 2  d k 12 
dp 
2  d k 1  d k   ln N k  ln N k 1    ln N k  ln N k 1 
 ln N k  ln N k 1   d k 2  d k 12 

2  d k 1  d k   ln N k  ln N k 1    ln N k  ln N k 1 
Digital Image and Signal Processing
Laboratory (DISP) MD-531
19
Institute of Communication Engineering
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
Depth from defocus
Digital Image and Signal Processing
Laboratory (DISP) MD-531
Institute of Communication Engineering
Arbitrary changing
camera parameters
with large variation
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
 Two diffusion parameters are considering.

kDs1  1 1 1 


 1
   
I f x , f y  H f x , f y ; i  R f x , f y , i  i | i  different depth region
2

 F s1 u 

  kDs2  1  1  1 
 H f x , f y ; i  exp 2 i 2 2 f x 2  f y 2


 2
2  F s2 u 

Replace to solve u
• Intuitively, we can get two pictures by changing one of camera parameters
and solve the triangularity problem. On the spatial domain or frequency domain.

 


2
1i
  2i
2
1

 A
w
  
 


 I1  f x , f y  
1
w 4 2  f x 2  f y 2  ln  I 2  f x , f y   df x  df y


i : equal focal subimage
Digital Image and Signal Processing
Laboratory (DISP) MD-531
21
Institute of Communication Engineering
Trace amount on
changing camera
parameters (1)
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
 More accurate by changing camera parameters with trace amount.
Feb. 2008
• We use the power spectral density :
P  fx , f y   e

4 i 2 2 f x 2  f y 2

 R  f x , f y  R*  f x , f y 
 P1  f x , f y  4 2  i12  i 22  f x2  f y 2 

e
 P2  f x , f y 
 Utilizing the fact that differential on the Gaussian function is still a Gaussian.
I  f x , f y 
 i

 H  f x , f y ;  i  R  f x , f y   4 i 2  f x 2  f y 2 

 I  f x , f y  4 i 2  f x 2  f y 2 


Digital Image and Signal Processing
Laboratory (DISP) MD-531
22
Institute of Communication Engineering
Trace amount on
changing camera
parameters (2)
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
 We have no idea on diffusion parameters, but we can replace it by camera
parameters by differential factor.
 i kD  1 1 

 

s
2 F u
I  f x , f y 
s


kD  1 1 
  i
2  F u 
kDs  1 1 1 
 replace  i 
 
2  F s u 
 I  f x , f y   4 2  f x 2  f y 2  
Digital Image and Signal Processing
Laboratory (DISP) MD-531
23
Institute of Communication Engineering
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
Linear canonical transform
(LCT) upon the optical
system
Digital Image and Signal Processing
Laboratory (DISP) MD-531
Institute of Communication Engineering
Linear canonical
transform (LCT) (1)
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
 Linear canonical transform (LCT) :
Feb. 2008
• The LCT gives a scalable kernel to describe wave propagation such as the
fractional Fourier transform and the Fresnel transform and etc.
• Definition (normalize as four parameters).

LM  f  u     LM  u, u ' f  u ' du ' :
 LM  u, u ' 

 D 2
1
A 2 

1/
B
exp

j

/
4
exp
j

u

2
uu
'

u '   for b  0,

  

B
B


 B

 LM  f  u   

 D exp  j CDu 2   f  D  u  for b=0



2

Digital Image and Signal Processing
25
Laboratory (DISP) MD-531
Institute of Communication Engineering
Linear canonical
transform (LCT) (2)
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
 Through a brief derivation, we can see that a combination of the LCTs is still
a LCT.
 The reason for the parameters mapping is for its convenience coordinates
transformation on the time-frequency distribution.
•
t '   A B   t 
 f '  C D    f 
  
  
 Some important properties connected by the LCT :
• Scaling, phase delay (chirp multiplication), modulation, chirp convolution and the
fractional Fourier transform.
Digital Image and Signal Processing
Laboratory (DISP) MD-531
26
Institute of Communication Engineering
Linear canonical
transform (LCT) (3)
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
 Sinc value in the frequency domain of a rectangle signal in the time domain.
Its parameters of LCT (A,B,C,D) = (0,1,-1,0) : the Fourier transform.
Digital Image and Signal Processing
Laboratory (DISP) MD-531
27
Institute of Communication Engineering
Approximation on the
optical system by LCTs
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
 We now consider a simple and common optical system.
•
z
Phase delay
(chirp multiplication)
Uo
s
Ul
Ul’
 The effective LCT parameters :
 1
 A B  1  s  
C D    0 1    1

 
 f

Ui
Free space diffraction
(chirp convolution)
s

1

0

f
 1  z  
1   0 1   1

  f

Digital Image and Signal Processing
Laboratory (DISP) MD-531
 zs

 s
f

z

1

f

 z
28
Institute of Communication Engineering
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
Focus recovery: common
method, alternative method
Digital Image and Signal Processing
Laboratory (DISP) MD-531
Institute of Communication Engineering
The common method (1) N
ational Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
 The most common focus recovery method :
• Based on the assumption that a simply constructed image scene has two layers,
foreground and background.
 Two input images, one focus on foreground ( f1 ) and the other focus on
background ( f2 ). Using adjustable values R1 and R2 to generate images.
• Where hi ( i = 1, 2, a, b ) indicates point spread functions, note that a and b are
adjusted parameters
 g1  f 1  h2  f 2

 g 2  h1  f 1  f 2
g1
K1
R1
f  ha  f 1  hb  f 2
g2
• -- Design filters K1 and K2 !!
Digital Image and Signal Processing
Laboratory (DISP) MD-531
f
K2
R2
30
Institute of Communication Engineering
The common method (2) N
ational Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
 The matrix form.
G1 
1 H 2 
 F1 
G  HF G    H  
 F  F 
G
H
1
 2
 1 
 2
 F  KG
K  H

1 T
1
W
1  H1 H 2
Ha 
F WT F W   
 Hb 
 H 2  H a 
 1
 H1
1   H b 

T
 Considering the existence of the inverse matrix H 1 (singular or nonsingular).
•
2
2
2


R

R

R
1
b
a
1
lim k  2
u,v 0
R1  R2 2  R2 2  Ra 2  Rb 2 
Digital Image and Signal Processing
Laboratory (DISP) MD-531
31
Institute of Communication Engineering
The common method (3) N
ational Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
 Filters result.
  R12  Rb 2  Ra 2

, at DC component of H1 or H 2 

2
2
  R1  R2

  H a  H b H1

, elsewhere


1

H
H
 K  
1 2

K  1

2
2
2
 K 2    R2  Ra  Rb
, at DC component of H1 or H 2 

2
2
  R1  R2



H

H
H
a
2
 b

, elsewhere
  1  H1 H 2

Digital Image and Signal Processing
Laboratory (DISP) MD-531
32
Institute of Communication Engineering
Alternative method :
one point focus
recovery (1)
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
 Depth of field (DOF).
• The ideal case (larger aperture).
Larger Aperture
These blurred
areas are too
large for the
HVS and result
in two blurring
areas.
Position of object
。。。
。。。
。。。
F
thin lens
sensor
Digital Image and Signal Processing
Laboratory (DISP) MD-531
33
Institute of Communication Engineering
Alternative method :
one point focus
recovery (2)
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
• The effective focused interval (smaller aperture).
Smaller Aperture
Position of object
。 。
。
These blurred
areas are too
small for the HVS
and results in an
effective focused
plane.
F
thin lens
sensor
Effective “depth of field” interval
Digital Image and Signal Processing
Laboratory (DISP) MD-531
34
Institute of Communication Engineering
Alternative method :
one point focus
recovery (3)
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
 Approximation by LCTs.
• Paraxial approximation (phase delay).
 The original Fresnel transform :
2
2 1/ 2 


 x0  x1   y0  y1   
1

U  x0 , y0  
U  x1 , y1  exp  jk z1  
 
   dx1dy1


j z  
  z   z   


 
 

  h  x , y , x , y U  x , y  dx dy
0
0
1
1
1
1
1
1
 
1/ 2
  x  x 2  y  y 2 
 1   0 1    0 1  
  z   z  
2
1  x0  x1  1  y0  y1 
 1 
  

2 z  2 z 
2
 
 k 
e jk z
2
2 
 U  x0 , y0  
U
x
,
y
exp
j
x

x

y

y
 1 1    0 1   0 1    dx1dy1


j z  
 2z

Digital Image and Signal Processing
Laboratory (DISP) MD-531
35
Institute of Communication Engineering
Alternative method :
one point focus
recovery (4)
 Flow chart for the
alternative
method :
one point focus
recovery.
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
Defocused
image pair
SML measurement
Maximum value searching
focal point
Depth measurement of a
point
Using the specific depth to
retrieve imaging distance
Small aperture construction
Linear canonical transform
based on constructed optical
system
Digital Image and Signal Processing
Laboratory (DISP) MD-531
Full focused
image
36
Institute of Communication Engineering
Simulation on simple
pattern
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
 Considering the Gaussian point light source.
• For simplicity, we assumes the parameters :
   1,   550 nm

 z  3 m, f  28 mm




(a) – the input Gaussian pattern.
(b) – LCT for s = 27 mm.
(c) – LCT for s = 30 mm.
(d) – Inverse LCT for s = 27 mm.
-5
-5
0
0
5
-5
0
5
5
-5
-5
-5
0
0
0
5
a b
c d
5
Digital Image and Signal Processing
-5
Laboratory (DISP) MD-531
0
5
5
-5
0
37
5
Institute of Communication Engineering
Conclusions
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
 Most of the literature discussed on the depth or the depth recovery fall in the
equal focal problems (DFD, DFF) or the correspondent problems (stereo
vision).
 Relying on the LCTs by the paraxial approximation system can avoid such
problems.
 Using LCTs is more like a deblurring procedure. Such action can keep the
original realities of the images from disturbance.
Digital Image and Signal Processing
Laboratory (DISP) MD-531
38
Institute of Communication Engineering
References
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
 Y. Xiong and S. A. Shafer, “Depth from focusing and defocusing,” IEEE
Conference on Computer Vision and Pattern Recognition, pp. 68-73, 1993.
 M. Subbarao, “Parallel depth recovery by changing camera parameters,”
Second International Conference on Computer Vision, 1988, pp. 68-73, 1988.
 K. S. Pradeep and A. N. Rajagopalan, “Improving shape from focus using
defocus information,” 18th International Conference on Pattern Recognition,
2006, vol. 1, p.p. 731-734, Sept. 2006.
 M. Asif and A. S. Malik, T. S. Choi “3D shape recovery from image defocus
using wavelet analysis,” IEEE International Conference on Image Processing,
2005, vol. 1, pp. 11-14, Sept. 2005.
 K. Nayar and Y. Nakagawa, “Shape from focus,” IEEE Transactions on
Pattern Analysis and Machine Intelligence, vol. 16, issue 8, pp. 824-831, Aug,
1994.
Digital Image and Signal Processing
Laboratory (DISP) MD-531
39
Institute of Communication Engineering
National Taiwan University
Taipei, Taiwan (R.O.C.)
r95942115@ntu.edu.tw
Feb. 2008
 Y. Y. Schechner and N. Kiryati, “Depth from defocus vs. stereo: how different
really are they?,” in ICPR 1998, vol. 2, pp. 1784-1786, Aug. 1998.
 M. Haldun Ozaktas, Zeev Zalevsky and M. Alper Kutay, “The fractional
Fourier transform with applications in optics and signal processing,” JOHN
WILEY & SONS, LTD, New York, 2001.
 A. Kubota and K. Aizawa, “Inverse filters for reconstruction of arbitrarily
focused images from two differently focused images,” IEEE Conferences on
Image Processing 2000, vol.1, pp.101-104, Sept. 2000.
 A. P. Pentland, “A new sense for depth of field”, IEEE Transaction on Pattern
Analysis and Machine Intelligence, vol. 9, no. 4, pp. 523-531, 1987.
 M. Hansen and G. Sommer, “Active depth estimation with gaze and vergence
control using gabor filters,”, Proceedings of the 13th International Conference
on Pattern Recognition 1996, vol. 1, pp. 287-291, Aug. 1996.
Digital Image and Signal Processing
Laboratory (DISP) MD-531
40
Institute of Communication Engineering