Facoltà di Ingegneria
Reprojection of 3D points of Superquadrics
Curvature caught by Kinect IR-depth sensor
to CCD of RGB camera
Mariolino De Cecco, Nicolo Biasi, Ilya Afanasyev
Trento, 2012
21/11/2011
1/20
Content
1.
2.
3.
4.
5.
6.
7.
8.
9.
21/11/2011
Kinect installation.
Kinect calibration.
Kinect outputs.
MATLAB Preprocessing Kinect data.
MATLAB Inputs.
SQ Curvature Software.
Tests.
Results.
Links.
2/20
Kinect installation
1st attempt
1. OpenNI backend:
1.
2.
3.
4.
5.
21/11/2011
Nicolas Burrus software [1]: Kinect RGBDemo v0.6.1. -> supports
OpenNI/Nite backends [2] and has the experimental infrared
support with OpenNI (still buggy) -> I tried to check this work for
OpenNI backend under Windows 64bit.
Install Sensor Kinect drivers under Windows 32 bit [3] and
OpenNI/Nite modules (Win32) [2] according to the instruction [4]
and verified that they work properly.
Use RGB-D Capture [1] to grab Kinect RGB and IR images with
intensity.raw and depth.raw files. -> the software doesn’t grab IR
images!! -> only RGB image, intensity.raw, depth.raw and
calibration file (calibration.yml without distortion parameters)!!
“calibrate-openni-intrinsics --pattern-size 0.0405 grab1
calibration.yml” -> gives “openni_calibration.yml” with Intrinsic
matrix and distortion coefficients for Kinect RGB camera !!
“calibrate-openni-depth.exe --pattern-size 0.0405 grab1” gives
partly processed figures with a message about errors.
3/20
The comparison of Kinect calibration results for
- Libfreenect backend (under OS Linux / Ubuntu 10.10 - 32 bit)
- OpenNI backend (under OS Windows 32bit)
Libfreenect backend
23/11/2011
The same figures of Intrinsic
Matrixes for Kinect RGB and
Depth Cameras
OpenNI backend
The Depth Camera Calibration
for OpenNI backend is
NOT available!!!
-
The comparison of Kinect calibration results for
- Libfreenect backend (under OS Linux / Ubuntu 10.10 - 32 bit)
- OpenNI backend (under OS Windows 32bit)
Libfreenect backend
23/11/2011
There is NO extrinsic mapping
between Kinect Depth and RGB
cameras for OpenNI backend
OpenNI backend
The reprojection of 3D points
from Kinect Depth Camera to
RGB image for OpenNI is NOT
possible!!
Kinect installation
2nd attempt
1. Libfreenect backend (Windows):
1.
2.
3.
4.
5.
21/11/2011
Nicolas Burrus software [1]: Kinect RGBDemo v0.6.1. -> supports
Libfreenect backend [2] and has RGB and infrared support -> I
tried to check the work under Windows 64bit.
Install Xbox NUI Motor drivers under Windows 32 bit [3] and
OpenNI/Nite modules (Win32) [2] according to the instruction [4]
and verified that they work properly.
Use RGB-D Capture [1] to grab Kinect RGB and IR images with
intensity.raw and depth.raw files. -> the software doesn’t grab IR
images!! -> only RGB image, intensity.raw, depth.raw and
calibration file (calibration.yml without distortion paraeters)!!
“calibrate-openni-intrinsics --pattern-size 0.025 grab1
calibration.yml” -> gives “openni_calibration.yml” with Intrinsic
matrix and distortion coefficients for Kinect RGB camera !!
“calibrate-openni-depth.exe --pattern-size 0.025 grab1” gives
partly processed figures with a message about errors.
6/20
Kinect installation
New Idea
1. OpenNI backend:
1.
2.
3.
4.
5.
21/11/2011
Nicolas Burrus software [1]: Kinect RGBDemo v0.6.1. -> supports
OpenNI/Nite backends [2] and has the experimental infrared
support with OpenNI (still buggy) -> I tried to check this work for
OpenNI backend under Windows 64bit.
Install Sensor Kinect drivers under Windows 32 bit [3] and
OpenNI/Nite modules (Win32) [2] according to the instruction [4]
and verified that they work properly.
Use RGB-D Capture [1] to grab Kinect RGB and IR images with
intensity.raw and depth.raw files. -> the software doesn’t grab IR
images!! -> only RGB image, intensity.raw, depth.raw and
calibration file (calibration.yml without distortion paraeters)!!
“calibrate-openni-intrinsics --pattern-size 0.025 grab1
calibration.yml” -> gives “openni_calibration.yml” with Intrinsic
matrix and distortion coefficients for Kinect RGB camera !!
“calibrate-openni-depth.exe --pattern-size 0.025 grab1” gives
partly processed figures with a message about errors.
7/20
5. MATLAB Inputs.
Cube_Curvatura_20110925\
main_reproject.m
-> color.png (480x640 pixels)
-> calibData.mat:
- matrixes of intrinsic
parameters for IR
camera (K_ir) and RGB
camera (K_rgb)
- extrinsic mapping between
ir (depth) and rgb
Kinect camera (R,T)
- distortion coefficients
(kc_ir, kc_rgb)
-> Points.mat (N x 6), where
N – number of points
with (x,y,z,R,G,B) info
21/11/2011
8/20
6. MATLAB Software.
6.1 Elimination of the ground.
RANSAC search of the ground by using the plane equation.
21/11/2011
9/20
6.2 3D Points Reprojection.
Transformation from IR (Depth) to RGB camera reference system:
P_rgb = R * P_ir + T, where R,T - extrinsic mapping between ir and rgb Kinect
cameras; P_rgb, P_ir – points in RGB and IR camera reference systems.
Cube Reprojection without considering distortion (up figures).
The distance from
Kinect to the cube
z = 0.5 m
???
Cube Reprojection with considering distortion (bottom figures).
21/11/2011
Bad
calibration?
10/2
0
6.2 3D Points Reprojection.
Cube and Cylinder Reprojection without considering distortion (up figures).
The distance from
Kinect to the objects
z = 0.6 m
???
Bad
calibration?!
23/11/2011
Cube and Cylinder Reprojection with considering distortion (bottom figures).
11/2
0
6.2 3D Points Reprojection.
Transformation from IR (Depth) to RGB camera reference system:
P_rgb = R * P_ir + T, where R,T - extrinsic mapping between ir and rgb Kinect
cameras; P_rgb, P_ir – points in RGB and IR camera reference systems.
Cube Reprojection with calibration parameters from Nicolo (upper figures).
The distance from
Kinect to the cube
z = 0.5 m
Bad
calibration?!
Cube Reprojection with calibration parameters from Alberto (bottom figures).
21/11/2011
12/2
0
6.2 3D Points Reprojection.
Camera Calibration and 3d Reconstruction [9] for so-called pinhole camera model.
The scene view is formed by projecting 3D points into the image plane using a
perspective transformation.
H
A = K_rgb
R(T) = [R, T]
where
A
R(T)
A
R(T)
- are the coordinates of a 3D point in the world coordinate space.
- are the coordinates of the projection point in pixels.
- matrix of intrinsic parameters (does not depend on the scene viewed).
- a principal point (that is usually at the image center)
- are the focal lengths expressed in pixel-related units.
- the matrix of extrinsic parameters.
Cube Reprojection with calibration parameters from Alberto (bottom figures).
21/11/2011
13/2
0
6.3 RANSAC fitting SQ to 3D data points.
With using Levenberg-Marquardt algorithm of distance minimization from SQ to 3D
points.
Red points – outliers.
Blue points – inliers.
Green points – SQ model.
21/11/2011
14/2
0
6.3 Object structure creation and reprojection on image.
Reprojection of the lines between the 3D points of cube vertexes to CCD of Kinect RGB
camera (figure a) and to the image with reprojected 3D points (figure b). The information
about vertex position was obtained in the previous stage of RANSAC cube pose
estimation.
Red lines – the cube frameworks
Figure a
Figure b
21/11/2011
15/2
0
9.
1.
2.
3.
4.
5.
6.
21/11/2011
References
Nicolas Burrus. Kinect. RGBDemo, calibrate and visualize Kinect
output. // http://nicolas.burrus.name/, 2011.
OpenNI Modules. // www.openni.org,
SensorKinect drivers. // github social coding.
https://github.com/avin2/SensorKinect.
How-to: Successfully install Kinect on Windows (OpenNI and
NITE). // Vangos Pterneas blog:
http://studentguru.gr/b/vangos/archive/2011/01/20/how-tosuccessfully-install-kinect-windows-openni-nite.aspx
Install OpenKinect for Windows 7 and XP. //
http://kinect.dashhacks.com/kinect-guides/2011/01/09/installopenkinect-windows-7-and-xp
Camera Calibration and 3d Reconstruction. // OpenCV (Open
Source Computer Vision) v2.1 documentation:
http://opencv.willowgarage.com/documentation/cpp/camera_calibr
ation_and_3d_reconstruction.html
16/20