3D Motion Estimation
3D model construction
3D model construction
Video Manipulation
Visual Motion
• Allows us to compute useful properties of
the 3D world, with very little knowledge.
• Example: Time to collision
Time to Collision
v
f
L
l(t)
t
Do
D(t)
L
t=0
An object of height L moves with
constant velocity v:
•At time t=0 the object is at:
• D(0) = Do
•At time t it is at
•D(t) = Do – vt
•It will crash with the camera at
time:
• D(t) = Do – vt = 0
• t = Do/v
Time to Collision
The image of the object has size l(t):
v
f
L
l(t)
t
Do
D(t)
L
t=0
Taking derivative wrt time:
Time to Collision
v
f
L
l(t)
t
Do
D(t)
L
t=0
And their ratio is:
Time to Collision
Can be
directly
measured
from image
v
f
L
l(t)
t
Do
L
t=0
And time to collision:
D(t)
Can be found, without knowing L or Do or v !!
Structure from Motion
Z
r
u  u tr  u rot
u
u tr
t, 
u rot

1 

 t  r  
ˆ
z


Z 
1
 zˆ  r   ω r 
F
Passive Navigation and Structure
from Motion
The system moves with a rigid motion wit h translat ional velocity
t  t x,t y , t z  and rotational velocity   ωx  ωy  ωz  .
T
T
Scene points R   X , Y , Z  project onto image points r  x, y, f 
T
  V ,V ,V  of a scene point is observed in the image
and the 3D velocity R
x
y
z
as velocity r  u, v.0.
Consider a 3D point P and its image:
P
p
f
Z
z
Using pinhole camera equation:
V
Let things move:
P
v
p
f
Z
The relative velocity of P wrt camera:
z
Translation
velocity
Rotation
angular
velocity
3D
Relative
Velocity:
V
P
v
p
f
Z
The relative velocity of P wrt camera:
z
Motion
Field:
the
velocity
of
p
V
P
Taking derivative wrt time:
v
p
f
Z
z
Motion
Field:
the
velocity
of
p
V
P
v
p
f
Z
z
Motion
Field:
the
velocity
of
p
V
P
v
p
f
Z
z
Motion
Field:
the
velocity
of
p
V
Translational
component
P
v
p
f
Z
z
Scaling ambiguity
(t and Z can only
be derived up to a scale
Factor)
Motion
Field:
the
velocity
of
p
V
Rotational
component
P
v
p
f
Z
z
NOTE: The rotational component is independent of depth Z !
Translational flow field
ty 
 tx
where p o  x0 , y0     f ,  f  is the focus of expansion (FOE)
tz 
 tz
or focus of contractio n (FOC).
Rotational flow field
 x  y 

f,
f  is the point wher e the rotation axis pierces the image plane (AOR).
 z z 
Pure Translation
What if tz  0 ?
All motion field vectors are parallel to each other and
inversely proportional to depth !
Pure Translation:
Properties of the MF
• If tz 0 the MF is RADIAL with all vectors pointing
towards (or away from) a single point po. If tz = 0
the MF is PARALLEL.
• The length of the MF vectors is inversely
proportional to depth Z. If tz  0 it is also directly
proportional to the distance between p and po.
Pure Translation:
Properties of the MF
• po is the vanishing point of the direction of
translation.
• po is the intersection of the ray parallel to
the translation vector and the image plane.
Special Case: Moving Plane
V
P
v
p
f
Z
n
Planar surfaces are common
in man-made environments
z
Question: How does the MF of a moving plane look like?
Special Case: Moving Plane
Points on the plane must satisfy the
equation describing the plane.
X
n
d
Y
O
Let
P
Z
•n be the unit vector normal to the plane.
•d be the distance from the plane to the
origin.
•NOTE: If the plane is moving wrt
camera, n and d are functions of time.
Then:
where
Special Case: Moving Plane
X
Let
be the image of P
n
d
Y
Using the pinhole
projection equation:
P
p
O
Z
Using the plane equation:
Solving for Z:
Special Case: Moving Plane
X
Now consider the MF equations:
n
d
Y
p
P
O
And Plug in:
Z
Special Case: Moving Plane
X
The MF equations become:
n
d
Y
p
O
where
P
Z
Special Case: Moving Plane
X
MF equations:
n
d
Y
O
p
P
Z
Q: What is the significance of this?
A: The MF vectors are given by low order (second)
polynomials.
•Their coeffs. a1 to a8 (only 8 !) are functions of n, d, t and .
•The same coeffs. can be obtained with a different plane and
relative velocity.
Moving Plane:
Properties of the MF
• The MF of a planar surface is at any time a
quadratic function in the image coordinates.
• A plane nTP=d moving with velocity V=-t-£ P
has the same MF than a plane with normal n’=t/|t| ,
distance d and moving with velocity
V=|t|n –( + n£ t/d)£ P
Classical Structure from Motion
• Established approach is the epipolar
minimization: The “derotated flow” should
be parallel to the translational flow.
E
utr
-urot
E
u
-urot
utr
u
t
The Translational Case
(a least squares formulation)
yt z  t y 
xtz  t x  

  u  Z    v  Z  dx dy  min
2
2
|| f ( x, y ) || 
 ( f ( x, y))
Substitute   t x  xtz
2
dx dy
  t y  yt z
  

u


  Z    v  Z   min
2
2
We minimize d 2
Step 1 : Minimize with respect to Z . (Find the length of u tr for which d
would be minimized. )
 
   
u   2  v   2  0
ZZ 
ZZ

Substitute back

u  v 2 dx dy  min
2  2
2  2
Z
u  v
Step 2: Differentiate with respect to tx, ty, tz, set expression to zero.

u  v u  v 
Let K 

2


2 2
I   t y  yt z K dx dy  0
II   t x  xtz K dx dy  0
III   yt x  xty K dx dy  0
3 linearly dependent equations t x  I  t y  II  t z  III  0
Equations are nonlinear in tx, ty, tz
Using a different norm
yt z  t y 
xtz  t x  

 ( 2   2 ) dx dy  min
u 
   v 
Z  
Z 
 
2
2
|| f ( x, y ) || ,    f ( x, y ) 2 ( 2   2 ) dx dy
First, differentiate integrand with respect to Z and set to zero
a  
 

u   2  v   2  0 
ZZ 
ZZ

2


u


v

dx dy  min

 g (t x , t y , t z )
2  2
Z
u  v
g (t x , t y , t z )  at x2  bt y2  ct z2  2dt xt y  2et yt z  2 ftxt z
where
a   v 2 dx dy,
b   u 2 dx dy,
c   ( xv  yu ) dx dy,
d   uv dx dy,
e   u ( xv  yu ) dx dy,
f   v( xv  yu ) dx dy,
Differentiate g( tx, ty, tz ) with respect to tx, ty, tz and set to zero
Gt  0
a

G d
f

with ||t||  1
d
b
e
f

e
c 
Solution is singular vector corresponding to
smallest singular value
The Rotational Case
2
2




u

u

v

v
 min
rot
 rot
Let f=1
u   x xy   y x 2  1   z y  0
v   x  y 2  1   y xy   z x  0
 xy
 2
y  f

In matrix form
x  f
2
 xy
 x 
y    u 
  y    
 x    v 
z 
A  u
  A A AT u
T
1
The General Case
Minimization of epipolar distance
2
  u  u rot    v tr  
   v  vrot    u tr   dx dy  min
or, in vector notation
2






t

r
r



r
dr  min

Motion Parallax
The difference in motion between two very
close points does not depend on rotation.
Can be used at depth discontinuities to obtain
the direction of translation.
FOE
Motion Parallax
u  (u, v)  u tr  u rot
At points p1 and p 2  ( x, y ) we have
u1 , rot  u 2 , rot
utr  u1,tr  u2,tr
1
1
 ( x  xo )(  )
Z1 Z 2
vtr  v1,tr  v2,tr
1
1
 ( y  yo )(  )
Z1 Z 2
v y  y0

u x  x0
Vectors perpendicular to
translational component
u
u u

tr

tr

( y  y 0 , x  x0 )
||( y  y0 , x  x0 )||
 ||( y  y01, x  x0 )|| ( y  y0 )urot  ( x  x0 )vrot
Vector component perpendicular to translational component
is only due to rotation
rotation can be estimated from it.
Motion Estimation Techniques
•
Prazdny (1981), Burger Bhanu (1990), Nelson Aloimonos (1988), Heeger
Jepson (1992):
Decomposition of flow field into translational and rotational
components. Translational flow field is radial (all vectors are
emanating from (or pouring into) one point), rotational flow field is
quadratic in image coordinates.
Either search in the space of rotations: remainig flow field
should be translational. Translational flow field
is evaluated by minimizing deviation from radial field:
.
(v, u)  ( x  x0 , y  y0 )  0
Or search in the space of directions of translation: Vectors
perpendicular to translation are due to rotation only
Motion Estimation Techniques
• Longuet-Higgins Prazdny (1980), Waxman (1987):
Parametric model for local surface patches (planes or
quadrics) solve locally for motion parameters and
structure, because flow is linear in the motion parameters
(quadratic or higher order in the image coordinates)