Motion from normal flow
Optical flow difficulties
• The aperture problem
• Depth discontinuities
Translational Normal Flow
un 
u tr
n
Z
• In the case of translation each
normal flow vector constrains
the location of the FOE to a
half-plane.
• Intersection of half-planes
provides FOE.
Egoestimation from normal flow
• Idea: choose particular directions:
patterns defined on the sign of normal flow
along particular orientation fields
• positive depth constraint
• 2 classes of orientation fields: copoint
vectors and coaxis vectors
Optical flow and normal flow
Optical flow and normal flow
Coaxis vectors
with respect to axis (A,B,C)
Coaxis vectors
Translational coaxis vectors
Translational coaxis vectors
h passes through FOE and (Af/C, Bf/C), defined by 2 parameters
Rotational coaxis vectors
Rotational coaxis vectors
Rotational coaxis vectors
g passes through AOR and (Af/C, Bf/C), defined by 1 parameter
Combine translation and rotation
Positive + positive
 positive
Negative + negative
negative
Positive + negative
don’t know (depends on structure)
Coaxis pattern
translational
rotational
combined
g-vectors: Translation
g-vectors: Rotation
u rot
 r2 
 r2 
    sin      cos 
 f 
 f 
g-vectors: Translation and Rotation
U
W
V
W


Three coaxis vector fields
alpha
beta
gamma
Copoint vectors
copoint vectors
O
Copoint vectors
defined by point (r,s)
Translational copoint vectors
AOR
AOR
FOE

FOE
FOE

g
AOR
: Negative
: Positive
: Don't know
AOR
FOE
Translational copoint vectors
FOE
(r, s)
k passes through FOE and (r,s) defined by 1 parameter
Rotational copoint vectors
Rotational copoint vectors
AOR
(r, s)
l passes through AOR and (r,s), is defined by 2 parameters
translational component
rotational component
FOE
AOR
(r, s)
(r, s)
FOE
AOR
(r, s)
Three coaxis vector fields
(a)
(b)
(c)
a,b,c : positive and negative
,,g vectors
c,d,e: Fitting of ,,g patterns
g: Separation of (,,g coaxis pattern
h: Separation of (x0, y0) copoint pattern
AOR
AOR
FOE

FOE
FOE

g
AOR
: Negative
: Positive
: Don't know
AOR
FOE
Optical
illusion
What is the Problem?
• Flow can be accurately estimated in an image patch
corresponding to a smooth scene patch,
• But erroneous flow estimates are obtained for image
patches corresponding to scene patches containing
discontinuities
Image Flow
3D Motion
Scene structure
Discontinuities
Depth variability constraint
• Errors in motion estimates lead to distortion
of the scene estimates.
• The distortion is such that the correct
motion gives the “smoothest” (least
varying) scene structure.
Depth estimation
• Scene depth can be estimated from normal
flow measurements:
1
un  u  n  u tr  n  u rot  n
Z
1 un  u rot ω̂  n

u tr (tˆ)  n
Zˆ
Visual Space Distortion

ˆ
u
t
n
tr
ˆ
Z  Z  D, D 
u tr t   u rot δω  n
• Wrong 3D motion gives rise to a rugged
(unsmooth) depth function (surface).
• The correct 3D motion leads to the
“smoothest” estimated depth.
Inverse depth estimates
correct motion
incorrect motion
The error function
• A normal flow measurement:
1
un  u tr  n  u rot  n
Z
1
ˆ)  n  u rot (ω
ˆ ) n
u

u
(
t
n
tr
ˆ
For an estimate t, ωˆ
Z
• The error function to be minimized:
1


ˆ
ˆ )  ni  ( )(u tr (t )  ni ) 
  Wi  un i  u rot (ω
Zˆ


i
2
The error function
• Estimated normal flow
1
ˆ ) n
uˆn  u tr (tˆ)  n  u rot (ω
Zˆ
• The error function to be minimized:
  Wi uˆn  un 
2
R
i
• Global parameters: tˆ, ω
ˆ
• Local parameter: Ẑ
1
locally planar patches: ˆ  ax  by  c
Z
Error function evaluation
• Given a translation candidate t̂ , each local
depth can be computed as a linear function
of the rotation ω̂
.
• We obtain a second order function of the
rotation; its minimization provides both the
rotation and the value of the error function.
Is derived from image gradients only
Brightness consistency:
Flow:
Planar patch:
Handling depth discontinuities
• Given a candidate motion, the scene depth
can be estimated and further processed to
find depth discontinuities.
• Split a region if it corresponds to two depth
values separated in space.
The algorithm
• Compute spatio-temporal image derivatives
and normal flow.
• Find the direction of translation that
minimizes the depth-variability criterion.
– Hierarchical search of the 2D space.
– Iterative minimization.
– Utilize continuity of the solution in time;
usually the motion changes slowly over time.
The Algorithm
Divide image into small
patches
Depth
variation
small?
Search in the 2D space of
translations
YES
Use the error
NO
For each candidate 3D
motion, using normal flow
measurements in each
patch, compute depth of the
scene.
Distinguish between two
cases
wrong 3D motion
existence of a
discontinuity at
the patch
For each image patch
Use the error
Split the patch
and repeat the
process
3D reconstruction
• Comparison of the original sequence and
the re-projection of the 3D reconstruction.
3D model construction