Reconstruction with Depth and
Color cameras for 3D
Autostereoscopic Consumer Displays
SAIT – INRIA collaboration
Period: 15 July 2012 / 15 January 2013
Date: 3-4 December 2012
INRIA team
•
•
•
•
•
Georgios Evangelidis, postdoc, 100%
Michel Amat, development engineer, 100%
Soraya Arias, senior development engineer, 20%
Jan Cech, posdoc, 20%
Radu Horaud, 10%
Past achievements
• A method and software for aligning TOF data
with a stereoscopic camera pair
• Extension to the calibration of several TOFstereo units
• 3D texture-based rendering of the TOF data
using the color-image information
Publications
• One CVPR 2011 paper
• A tutorial at ICIP 2011
• One Springer Briefs book
just published
• The two teams published
several other papers
Current achievements
• Finalization of the calibration & rectification
methods/software
• TOF to stereo-pair mapping with filtering
• TOF + texture in live mode
• Disparity map initialization
• Stereo correspondence based on seed-growing
• Final high-resolution depth map with gap filling
• A paper submitted to CVPR’13
Improved Calibration
• New calibration board
– mat sticker glued to a
rigid plane
– plane attached to a tripod
• Refined Calibration algorithm
– TOF-Stereo Calibration error: <1.5 pixel
• Improved Rectification
– Rectification error:
<0.25 pixel
Given a calibrated TOF-Stereo system
• Each TOF point PT defines a correspondence
between PL and PR
Correspondences (samples) obtained
by using the calibration parameters
• each correspondence comes from a TOF point
• different color -> different depth
Correspondences (samples) obtained
by using the calibration parameters
• each correspondence comes from a TOF point
• different color -> different depth
TOF-to-Left Mapping
• We use the left image as reference
TOF-to-Left Mapping is not perfect
Resolution mismatch
TOF-to-Left Mapping is not perfect
Left-to-Tof Occlusions
Left-to-Tof Occlusions: the depth decreases from left to right
TOF-to-Left Mapping is not perfect
Tof-to-Left Occlusions
Tof-to-Left Occlusions: the depth increases from left to right
Point Cloud filtering
• We reject points in left-to-tof occluded area
• We keep the minimum-depth points in case of
overlap (due to Tof-to-left occlusions)
Disparity Map: Initialization
• Run Delauney-Triangulation on low-resolution point
cloud
Disparity Map: Initialization
• Run Delauney-Triangulation on low-resolution point
cloud…
It looks good,
• …and initialize the stereo disparity map
but it’s noisy and
non-accurate!
Seed-Growing Idea
• Start from points with known disparities
(seeds) and propagate the disparity to
neighboring points (video?)
• Main issues:
– What are our seeds?
– What is the visiting order of seeds?
– How do I propagate the message?
– How the stereo and depth data are fused within
this framework?
Depth-Color Fusion
• Built on the seed-growing idea
– A:Depth data, S: Stereo data, dN : neighbor of d
– For each pixel (node), find its disparity value that
maximizes the posterior probability (MAP)
A
dN
S
d
Input data
Range-search
constraint
Penalize the choice
wrt to depth
information
Penalize the choice
wrt to color
information
Pixel with unknown disparity
Pixel with known disparity
A represents the initial estimation of d (obtained by the previous interpolation)
S represents the color matching cost that corresponds to d
Depth-Color Fusion
A
• Bayes rule translates each posterior into a
likelihood
S
d
N
Because of the
Bayes rule
d
Input data
Because of the
uniform distribution
• If likelihood terms are chosen from the
exponential family, the “-log”-ness translates MAP
into an energy minimization scheme We are currently
Pixel with unknown disparity
Pixel with known disparity
working on these
terms!
Seed-Growing Idea (revisited)
• For each pixel, an energy function is defined and we
look for its minimizer (disparity)
• Main issues:
– What are our seeds?
• the points from Tof-to-Left mapping after refinement
– What is the visiting order of seeds?
• First visit reliable seeds (points with low energy value)
– How do we propagate the message?
• Given the disparity of a seed, bound the disparity-range for its
neighbor
– How the stereo and depth data are fused within this
framework?
• Described above
Examples
White areas: unreliable matches
Black areas: Occlusions
Examples (with gap filling)
Examples (with gap filling)
Examples (with gap filling)
Examples (with gap filling)
Paper Submission
• Stereo-Depth Fusion for High-Resolution
Disparity Maps. G. Evangelidis, R. Horaud, M.
Amat, and S. Lee – submitted to CVPR 2013.
• An extended version of the CVPR submission
is under preparation and it will be submitted
to IEEE TPAMI in January/February 2013.
Work during the remaining month
• Improve the accuracy of the matching by
better exploiting the color/texture information
• Currently the software implementation runs in
offline-mode: We will provide a live-mode
version at approximatively 1-2 frames/second
• An updated version will be available at the
end of the period (~15 January 2013)
Prospects for the next collaboration
(1 February 2013 – 31 January 2014)
• Finalize the TOF-stereo seed-growing algorithm, in
particular improve the performance in non-textured
areas
• Depth disambiguation using TOF-TOF and TOF-stereo
• Combine depth disambiguation with the seed-growing
algorithm
• Perform full 3D realistic rendering with four TOF-stereo
units
• Perform continuous 3D reconstruction with a moving
TOF-stereo unit