Stereo Vision using
PatchMatch Algorithm
Junkyung Kim
Class of 2014
1. Introduction
Vision Problem Revisited
• Loss of information
– 3-D physical world projected onto a 2-D surface
Vision Problem Revisited
– Therefore, an inherently Ill-posed problem
Vision Problem Revisited
• Possible solutions
– 1. Directly exploit the Well-behavedness of the
physical world
• Shading
• Occlusion
• Textural transformation & alignment …
Vision Problem Revisited
• Possible solutions
– 2. Rely on different sources of information
• Ecolocation
• “Kinect” …
Vision Problem Revisited
• Possible solutions
– 2. Use multiple 2-D images
• Motion
• Stereo
1. Description of the problem
What is Stereo Vision?
• A constrained version of motion parallax
– When the observer moves, closer objects appear
to move faster
– Constraint 1 : only two frames (binocular)
– Constraint 2 : observer moves only laterally
– Constraint 3 : all the objects are stationary
What is Stereo Vision?
• Binocular disparity for depth computation
What is Stereo Vision?
• How to get depth? = How to get delta-X ?
– Must find the correspondence pairing first
www.consortium.ri.cmu.edu
What is Stereo Vision?
• How to get depth? = How to get delta-X ?
Correspondence Problem
• Ambiguity, again
– Exhaustive search, even though reduced only within
the epipolar line, is highly prone to false pairing
– 1. inherent imbalance : search space >> sample space
– 2. noise
Correspondence Problem
• Ambiguity, again
– Worst-case scenario : binary RDS
Correspondence Problem
• Solutions
– 1. Use more evidence (neighboring pixels)
• Equivalent to increasing sample space
• Better-posed than ill-posed
– 2. Enforce well-behavedness of the world
• smoothness
Correspondence Problem
• Implementation
– 1. Representational
• Some feature map rather than raw image
– 2. Computational
• PatchMatch
3. Method : PatchMatch
PatchMatch
• A Randomized Correspondence Algorithm for
Structural Image Editing
• Barnes, et al
• Patten Analysis & Recognition, 2009
PatchMatch
• Proposed Applications
– Image reconstruction (reshuffling)
Barnes et al, 2009
PatchMatch
• Proposed Applications
– Image completion (inpainting)
Barnes et al, 2009
PatchMatch
• Proposed Applications
– Image retargeting (transformations)
Barnes et al, 2009
PatchMatch
• Why is it suitable for stereo correspondence?
– 1. patch-based match
• Nearest-Neighbor Field (NNF)
• large sample space
> uniqueness constraint better satisfied
PatchMatch
• Why is it suitable for stereo correspondence?
– 2. computational efficiency :
• randomized search followed by propagation
– Not just faster, but easier to implement smoothness
enforcement
• Important in application (e.g. navigation)
4. Milestones
Milestones
• Week 1~2
– Study PatchMatch and source code
– Gather dataset to work on
• Preferrably with one with ground-truth disparity map
Milestones
• Week 3~5 (mid-project presentation)
– Preliminary implementation
– Generate first-round of outputs
Milestones
• Week 6~9 (final presentation)
– Evaluate the algorithm with full outputs
– (if time allows) Draw comparison among several
different implementations