Depth Estimation and Focus
Recovery
Reporter: Wade Chang
Advisor: Jian-Jiun Ding
1
Outline
 Motivation
 Overview




Blurring model and geometric optics
Blurring function
Fourier optics
Linear canonical transform (LCT)
 Depth estimation methods
 Binocular vision system
 Monocular vision system
 Focus recovery methods
 Reference
2
Outline
 Motivation
 Overview




Blurring model and geometric optics
Blurring function
Fourier optics
Linear canonical transform (LCT)
 Depth estimation methods
 Binocular vision system
 Monocular vision system
 Focus recovery methods
 Reference
3
Motivation
 Focus recovery is important, it can help users to know the
detail of original defocused image.
 Depth is a important information for focus restoration.

4
Outline
 Motivation
 Overview




Blurring model and geometric optics
Fourier optics
Linear canonical transform (LCT)
Blurring function
 Depth estimation methods
 Binocular vision system
 Monocular vision system
 Focus recovery methods
 Reference
5
Blurring Model and Geometric
Optics(1)
 What is the perfect focus distance?
1 1 1
 
f s z
 Why does the blurring image generate?
This area is too small for the HVS
and results in an effective focused
plane
z
s
Position of object
。 。 。
。 。 。
。 。 。
F
lens
Effective “depth of field” interval
6
。
。 。
sensor
。 。
。
Blurring Model and Geometric
Optics(2)
 Ideal and real spherical convex lens
Ideal spherical convex lens
Aspherical convex lens
7
Real spherical convex lens
Blurring Model and Geometric
Optics(3)
 combination of the convex lens and the concave lens
Convex lens
F2
Incident rays
F1
8
Concave lens
Blurring Model and Geometric
Optics(4)
 Effective focal length of the combination of lenses
l2
l1
Combination of the
thin convex lenses
F4
F1
9
F2
F3
Outline
 Motivation
 Overview




Blurring model and geometric optics
Blurring function
Fourier optics
Linear canonical transform (LCT)
 Depth estimation methods
 Binocular vision system
 Monocular vision system
 Focus recovery methods
 Reference
10
Blurring Function (1)
screen
F
F
Blurring radius: R<0
D/2
s
u
2R : R<0
Biconvex
v
F
F
Blurring radius: R>0
D/2
s
u
2R : R>0
screen
11
v
Blurring Function (2)
 Blurring radius relates to depth value:

Ds  1 1 1 
R

  


2 F s u


FsD
u 

sD   2 R  D  F

 Considering of diffraction, we may suppose a blurring
function as:
I  x, y   h  x, y   r  x, y 
 x2  y 2 
h  x, y  
exp  
2
2 
2
2



1
 :diffusion parameter
12
h  x, y  : blurring function

r  x, y  : radiance

 I  x, y  : defocus image
Outline
 Motivation
 Overview




Blurring model and geometric optics
Blurring function
Fourier optics
Linear canonical transform (LCT)
 Depth estimation methods
 Binocular vision system
 Monocular vision system
 Focus recovery methods
 Reference
13
Fourier Optics(1)
 Aperture effect(Huygens-Fresnel transform)
 When a plane wave progress through aperture, the observed
field is a diffractive wave generated from the rim of aperture.
... . ..
14
Fourier Optics(2)
 Where the examples are through Huygens-Fresnel transform
at z=1 meter, z=14 meters and z=20 meters respectively.
Square wave
After Huygens-Fresnel Transform
z=1
x-axis [0,200]
x-axis [0,200]
x-axis [0,200]
15
Square wave
After Huygens-Fresnel Transform
z=20
y-axis [0,200]
Square wave
After Huygens-Fresnel Transform
z=14
y-axis [0,200]
y-axis [0,200]
y-axis [0,200]
Square wave
Before Huygens-Fresnel Transform
x-axis [0,200]
Outline
 Motivation
 Overview




Blurring model and geometric optics
Blurring function
Fourier optics
Linear canonical transform (LCT)
 Depth estimation methods
 Binocular vision system
 Monocular vision system
 Focus recovery methods
 Reference
16
Linear Canonical Transform (1)
 Why we use Linear canonical transform?
 Definition
LM  f u    LM u, u ' f u ' du ' :

 D 2
1
A 2 
L
u
,
u
'

1
/
B
exp

j

/
4
exp
j

u

2
uu
'

u '   for B  0,




 
 M
B
B
B


 

 L f u  D exp  j CDu 2   f D  u for B=0
 

M 

 

2



17
Linear Canonical Transform (2)
 Effects on time frequency analysis can help us realize most
properties by changing those four parameters.
 Let us consider one of time frequency analysis-Gabor
transform:
  t   2 

 g   e j 2 f  d
G  t , f    exp  


2 


 After g ( ) is substituted as a LCT signal, the result in a new
coordinate on time and frequency is as follows.
   At  Bf   '2  
   g  '  exp   j 2  Ct  Df  ' d
G  t ', f '   exp   

 
2


 

18
t '   A B   t 
 f '   C D    f 
  
  
Linear Canonical Transform (3)
Characteristics
Function representation
Chirp multiplication
Chirp convolution


exp  j qu 2   u  u '
exp  j / 4 1 / r

 exp j  u  u '  / r
Fourier transform
Scaling
19

1  i  cot   icot   2 /2
e
2
Fractional Fourier
transform
2
e
Transforming parameters
 1 0
 q 1 


1 r 
0 1


 cos  / 2  sin  / 2  


  sin  / 2  cos  / 2  
 icsc t  icot  t 2 /2
exp  j / 4
 exp  j 2 uu '
S   u  Su '
 0 1
 1 0


0 
S
 0 1/ S 


Linear Canonical Transform (4)
 Consider a simple optical system.
z
Uo
s
Ul
Ul’
Ui
 The equivalent LCT parameter:
 1
 A B  1  s  
C D    0 1    1

 
 f

20
 s
1
0

f
 1  z  
1  0 1   1

  f

 zs

 s
f

z

1

f

 z
Linear Canonical Transform (4)
 Special case of an optical system
f
f
f : focal length
Uo
Ul
Ul’
 1
 A B  1  f  
C D   0 1    1

 
 f

 0
A B 

   1
C
D


  f
0
 1  f 
1 0 1 

 f   f

0   0
 
Ui
0
1
f
scaling
21

  0 1
  1 0

Fourier transform
Outline
 Motivation
 Overview




Blurring model and geometric optics
Blurring function
Fourier optics
Linear canonical transform (LCT)
 Depth estimation methods
 Binocular vision system
 Monocular vision system
 Focus recovery methods
 Reference
22
Binocular Vision System(1)
23
Binocular Vision System(2)
 Binocular vision at a gazing point.
 sin 2  L   R  cos 2  L cos 2  R
u B 

 4sin 2     sin 2    
L
R
L
R

2
2
Gazing point
(Corresponding point)
g
L Depth (u)
B/2
B/2
Baseline (B)
24
R



Outline
 Motivation
 Overview




Blurring model and geometric optics
Blurring function
Fourier optics
Linear canonical transform (LCT)
 Depth estimation methods
 Binocular vision system
 Monocular vision system
 Focus recovery methods
 Reference
25
Monocular Vision System(1)
 Method 1:
 Utilizing diffusion parameter to calculate depth value.

kDs1  1 1 1 


 1
   
2  F s1 u 


  kDs2  1  1  1 


 2
2
F
s
u

2


F
F
u
D
/
2
s
2R :
R>0
scr
een
v
26
Blurring radius:
R>0
Monocular Vision System(2)
 Using power spectral density to calculate depth value.



 

;    exp 2
I f x , f y  H f x , f y ;  i  R f x , f y , i  i | i  different depth region

 H fx , f y


P fx , f y  e



i
4 i 2 2 f x 2  f y 2

 2  f x2  f y2 
2
i
  R f , f R* f , f
 x y  x y


 P1 f x , f y
4 2  i12  i 22  f x 2  f y 2 

e
P
f
,
f
 2 x y

27
2
1i
  2i
2

1

Aw

w


 P1 f x , f y
4 2
ln 
f x 2  f y 2  P2 f x , f y

  df
 
x
 df y
Monocular Vision System(3)
 Method 2:
 Take differentiation on equation I which respect to  i .

I f x , f y
 i
H
f ,f
x

 I fx , f y
 i kD  1 1 


s
2  F u 
28
y;

 i  R  f x , f y   4 i 2  f x 2  f y 2 
  4   f
2
i
2
x
 f y2




I f x , f y
s
I
 f , f    4  f
2
x
y
replace  i 
2
x

 f y2 
kDs  1 1 1 
 
2  F s u 
kD  1 1 
  i
2  F u 
Monocular Vision System(4)
 Method 3:
 Using LCT blurring models
z
Uo
 1
A
B
1

s

 


C D   0 1    1

 
 f

1
A B
D

C D 
 C



29
B
A 
s
Ul Ul’ Ui
 s
1
0

1

z
f




1  0 1   1

  f

 zs

 s
f

z

1

f

 z
Outline
 Motivation
 Overview
 Fourier optics
 Linear canonical transform (LCT)
 Blurring function
 Depth estimation methods
 Binocular vision system
 Monocular vision system
 Focus recovery methods
 Reference
30
Focus Recovery Methods(1)
 Derive MMSE filter
31
Focus Recovery Methods(2)
 Derive MMSE filter

32
Focus Recovery Methods(3)
 Derive Wiener Filter:
 s( x, y) : ideal image

sˆ( x, y) : final image
e( x, y) : noise
h( x, y) : po int spread function
r ( x, y) : sharpening filter
e( x, y) : noise
33
Reference
[1] M. Robinson, D. Stork, “Joint Design Lens Systems and
Digital Image Processing”
[2] P. C. Chen, C. H. Liu, ”Digital Decoding Design for Phase
Coded Imaging”
[3] Y. C. Lin, “Depth Estimation and Focus Recovery”
34
Thank You for Listening
35