CRV2010 Tutorial
May 30, 2010
3D-2D registration
Kazunori Umeda
Chuo Univ., Japan
umeda@mech.chuo-u.ac.jp
http://www.mech.chuo-u.ac.jp/umedalab/
Registration
of range image and color image
Necessary for texture mapping
Range image
(3D model)
Color image
When 3D-2D registration is given,
Texture mapping
Parameters to obtain
for 3D-2D registration
Object
(range image)
Sensor
Z
coordinate system
Intensity
Projection of
range intensity image
Image plane
image
X
Range image
sensor Y
Extrinsic parameters
Color camera
Intrinsic parameters
(Distortion parameters)
Parameters to obtain
for 3D-2D registration
Extrinsic parameters
Object’ rotation R and translation t
or camera’s orientation Rc and position tc
 r11 r12 r13 
t x 
R  r21 r22 r23 , t  t y 
 r31 r32 r33 
t z 
Sensor
Z
coordinate system
Rc  RT , t c  RT t
X
Range image
sensor Y
R, t (Rc, tc)
Color camera
Parameters to obtain
for 3D-2D registration
Intrinsic parameters
camera
coordinate system
3D space
(X,Y,Z)
X  x u
( X ,Y , Z )
Image plane
(u,v)
u
v
(u , v)
Image plane
 u X  sY

 u0  ,  , s, u , v
Color camera
u 
u
v
0 0
Z

u, v: focal length/pixel size
v   vY  v
0
s: skew, u0, v0: principal point coordinates

Z
Parameters to obtain
for 3D-2D registration
X w
Xw
u 
Y 
Y 
w 
w 




s  v   AR t 
P
 Zw 
 Zw 
1 
 
 
 1 
 1 
 u s

A   0 v
 0 0
u0 

v0 
1 
P: 34 matrix
Homogenous coordinates
11 unknown parameters
(6 extrinsic + 5 intrinsic)
2 constraints
When correspondences between range image
and color image are given,
3D
2D
Parameters can be calculated.
Equivalent to camera calibration problem.
For extrinsic parameter estimation,
n6
n3
Equivalent to PnP (Perspective n-Point) problem
…It is hard to obtain correspondences even
manually.
Range image
Range intensity image
(reflectance image)
By using range intensity image, obtaining
correspondences becomes easier!
e.g., corners, edges,
SIFT [Böhm 2007]
Our approach: gradient-based method
(not explicitly using correspondences)
Initial camera parameters
Produce a 2D image
from a range image
Two 2D
images are matched
No
Gradient-based method
Update camera parameters
End
Yes
Optical flow constraint
I (u  u, v  v, t  t )  I (u, v, t )
u
v
Tailor expansion
I u  I v v   I t
Projection of
range intensity image u
I
I
I
Iu 
, I v  , It 
u
v
t
Intensity image
(u , v)
(u , v)
It: difference
between intensity
image and
projected range
intensity image
(1) Constraints for extrinsic parameters
 u X  sY

 u0
u 
Z

v   vY  v
0

Z
When intrinsic parameters are constant,
u  s  u X  sY 


u

X Y
Z

2
Z
Z

Z

v   v Y   vY Z

Z
Z2

Substituting for the optical flow constraint
I u u  I vv   I t
Iu
u   s
    X  sY
Y
X   Iu  Iv v Y   Iu u 2
 Iv v2 Z  It
Z

Z
Z 

Z 
Z
( X ,Y , Z )
u
v
Camera motion: v0, w
  v  ω  X
X
0

v0  v0 x v0 y
X  X

Y

Z T
v0 z , ω  wx wy wz
T

T
(u , v)
Digital camera
av0 x  bv0 y  cv0 z  (bZ  cY )w x  (cX  aZ )w y  (aY  bX )w z   I t ,
u
s
v
 vY 
 u X  sY
a  Iu
, b  Iu  I v
, c   I u
 Iv 2 
2
Z
Z
Z
Z
Z 

Linear equation for 6 motion parameters v0, w
cf.
[Yamamoto 1985]
[Horn IJCV1988]
v0, w can be solved with 6 or more points
by linear least square method.
v0, w
R(33 rotation matrix) and t (3D translation vector)
Motion parameters are supposed to be small
Iteration is necessary
(2) Constraints for intrinsic parameters
 u X  sY

u

 u0

Z

v   vY  v
0

Z
When intrinsic parameters are also variables,
u  s  u X  sY  X
Y



X Y
Z  u  s  u0
u 
2
Z
Z

Z

v   v Y   vY Z  Y   v
v
0
2

Z
Z
Z

Z
Z
Substituting for the optical flow constraint
I u u  I vv   I t
av0 x  bv0 y  cv0 z  (bZ  cY )w x  (cX  aZ )w y  (aY  bX )w z
X
Y
Y


 I u  u  I v  v  I u s  I u u0  I vv0   I t
Z
Z
Z
a,b,c: same as previous equation
Linear equation for 6 motion parameters v0, w and
5 intrinsic parameters
v0, w and intrinsic parameters can be solved with 11 or
more points by linear least square method.
(3) Constraints for distortion
Distortion model (the simplest)
X  x  xd  u
Distortion
 u X  sY

 u0
u 
Z

v   vY  v
0

Z
2

x

x
(
1

k
'
r
)
 d
1

2
y

y
(
1

k
'
r
)

1
 d
r  x2  y 2

 u X  sY 
X 2  Y 2 
u 
1  k1
 u0
2


Z

Z




 vY 
X 2  Y 2 
1  k1
 v0
v 
2


Z 
Z


av0 x  bv0 y  cv0 z  (bZ  cY )wx  (cX  aZ )w y  (aY  bX )wz
 X k1 X ( X 2  Y 2 ) 
 Y k1Y ( X 2  Y 2 ) 
 Y k1Y ( X 2  Y 2 ) 
u  I v  
 v  Iu  
s
 Iu  
3
3
3
Z





Z
Z
Z


Z

Z


(u X  sY )( X 2  Y 2 )
 vY ( X 2  Y 2 )  

 I uu0  I vv0  I u
 Iv
k1   It
3
3
Z
Z




2
2
 u

(
3
X

Y
)  2sXY 
2 v XY
u

a  Iu
 k1

I
k
,
v 1
3
3
 Z

Z
Z


2
2 

 v
s( X 2  3Y 2 )  2 u XY 

(
X

3
Y
)
s

v

b  I u   k1

I

k
,

v
1
3
3

Z
Z
Z
Z








3(u X  sY )( X 2  Y 2 ) 
3 vY ( X 2  Y 2 ) 
 u X  sY

  vY

c   Iu 

k

I

k
 v 2

1
1
2
4
4
Z
Z
Z
Z








Linear equation for 6 motion parameters v0, w ,
5 intrinsic parameters and a distortion parameter
The parameters can be solved with 12 or more points
by linear least square method.
Implementation
[Irani ICCV1998]
So as to absorb the differences between a range
intensity image and an intensity image
2 images: horizontal, vertical. Prewitt operator.
Differential images
-1 0 1
-1 0 1
-1 0 1
Coarse to fine
Control of resolution and s of Gaussian
Extrinsic onlyv0+Intrinsicall
-1 -1 -1
0 0 0
1 1 1
Experimental results
Range image sensor: ShapeGrabber PLM300
（Slit laser, triangulation，wavelength 670nm)
Digital camera: Nikon COOLPIX 5000
(5M, 25601920 pixels,
2/3” CCD, pixel dimension 3.4mm?, f=7.1-21.4mm)
R-channel，RAW format
256019201280960 at registration
Measurement of a range image
312730 points
Summary
３D-2D registration (for texture mapping, etc.)
• Projective geometry
• Obtaining camera’s
parameters
extrinsic
and
intrinsic
• Range intensity (reflectance) image is useful
• With correspondences
• Equivalent to {camera calibration / PnP} problems
• Using optical flow constraint
• Explicit correspondences are not necessary
• Linear equation for motion parameters