1
Window-based Approach For
Fast Stereo Correspondence
Raj Kumar Gupta, Siu-Yeung Cho
IET Computer Vision, 2013
2
Outline
• Introduction
• Related Work
• Proposed Method
• Experimental Results
• Conclusion
3
Introduction
• Using two correlation windows to improve the
performance of the algorithm
• 3*3 and 9*9
• Real-time suitability
• more than 10 frame/s on CPU in case of 320 × 240-sized
image pair with disparity value 16
4
Related Work
• Local methods are usually base on correlation.
• Area-based (NCC, SAD, SSD)
• Feature-based: rely on feature extraction and match local
cues (BF, GF)
• Bigger window size, more information, more blurred.
5
Outline
• Introduction
• Related Work
• Proposed Method
• Experimental Results
• Conclusion
6
Flow Chart
7
Flow Chart
8
Initial Matching
• Matching cost computation: SAD
d
Left
Right
9
Problem in disparity selection
a. Determine disparity easily for unique minimum value
b. Ambiguous disparity in case of multiple minima
c. Matching cost calculated at point (205, 230) of
Tsukuba image
10
Initial Matching: large correlation window
• Matching cost computation: SAD + penalty
• Penalty term
• Disparity computation
11
Problem in disparity selection
12
Initial Matching: small correlation window
• Only those disparity values that are carried by
neighbouring pixels.
• Matching cost computation without penalty
• N: the disparity values of the neighbouring pixels.
• Avoid local minima and speed up
13
Flow Chart
14
Unreliable pixel detection
• left–right cross-checking
15
Disparity Interpolation
• Search for pixels with reliable disparity value in its
eight neighbouring pixels.
• Compute similarity of unreliable pixel and its reliable
neighbor.
16
Flow Chart
17
Disparity Refinement
18
Outline
• Introduction
• Related Work
• Proposed Method
• Experimental Results
• Conclusion
19
Experimental Results
• Computation time of the proposed algorithm for
different window sizes on Tsukuba image.
(image size 384 × 288 with 16 disparity labels)
20
• Percentage error in non-occluded (nocc), whole image (all) and
near depth discontinuities (disc) for different window sizes for all
four images (Tsukuba, Venus, Teddy and Cones)
21
Experimental Results
• a. Without using small correlation window Ws and the disparity refinement step
• b. Without using the disparity refinement step
• c. Without using small correlation window Ws
• d. With all four steps on Tsukuba image
22
Experimental Results
23
24
25
Experimental Results
• Comparison the performance of the proposed
algorithm with other correlation-based algorithms.
26
Experimental Results
27
Reference
• [24] Gupta, R., Cho, S.-Y.: ‘Real-time stereo matching using adaptive binary window’
•
•
•
•
•
(3D Data Processing, Visualization and Transmission, 2010)
[25] Zhang, K., Lu, J., Lafruit, G., Lauwereins, R., Gool, L.V.: ‘Real-time accurate
stereo with bitwise fast voting on Cuda’. Int. Conf. Computer Vision Workshops,
2009, pp. 540–547
[26] Humenberger, M., Zinner, C., Weber, M., Kubinger, W., Vincze, M.:‘A fast stereo
matching algorithm suitable for embedded real-time systems’, Comput. Vis. Image
Underst., 2010, 114, (11),pp. 1180–1202
[27] Gong, M., Yang, Y.: ‘Near real-time reliable stereo matching using
programmable graphics hardware’. IEEE Conf. Computer Vision and Pattern
Recognition, 2005, pp. 924–931
[28] Richardt, C., Orr, D., Davies, I., Criminisi, A., Dodgson, N.: ‘Real-time
spatiotemporal stereo matching using the dual-cross-bilateral grid’. European Conf.
Computer Vision, 2010, vol. 6313, pp. 510–523
[29] Ambrosch, K., Kubinger, W.: ‘Accurate hardware-based stereo vision’,Comput.
Vis. Image Underst., 2010, 114, (11), pp. 1303–1316
28
29
30
31
32
33
34
Conclusion
• A new correlation-based stereo-matching approach.
• Large window improves at non-textured image regions
• Small window improves at depth discontinuities
• The CPU implementation computes at a speed of more than 10
frame/s.
• Easily implemented on GPU.
• The proposed method can be used in real-time applications to
reconstruct the 3D structures with great accuracy at object
boundaries.
35
Codebook based Stereo
Matching for Natural User
Interface
Sung-il Kang and Hyunki Hong
2013 IEEE International Conference on Consumer
Electronics (ICCE)
36
Outline
• Introduction
• Proposed Method
• Experimental Results
• Conclusion
37
Introduction
• Interactive user interface has been one of the major
topics in consumer electronics.
• Gesture based user interface
• Interactive smart TV, Nintendo Wii, Sony PlayStation3 Move,
and Microsoft Kinect.
• Propose a stereo system implemented on GPGPU for
real-time performance.
• Employ codebook to solve occlusion.
38
Flow chart
39
Proposed Method
• Pre-processing
• Laplace od Gaussian (LoG) filter for alleviating the lighting effects.
• Cost initialization
• AD+Census[6]
[6] X. Mei, X. Sun, M. Zhou, H. Wang, and X. Zhang, “On building an accurate stereo matchng
system on graphics hardware,”
Proc. of GPUCV, pp. 467-474, 2011. http://www.camdemy.com/media/4724
40
Proposed Method
• Cost aggregation[6]
• Cross-based aggregation
• Color similarity and the length constraint
• Refinement[6,7]
• Left-right consistency check
• Iterative region voting
• Sub-pixel enhancement
d
[7] Q. Yang, C. Engels, R. Yang, H. Stewenius, and D. Nister, “Stereo matching with colorweighted correlation, hierarchical belief propagation and occlusion handling,” IEEE
Transactions on PAMI, 2009.
41
Proposed Method
Yes
Occlusion?
Find codeword
No
Yes
Codeword?
No
Add a new
codeword
Update
codeword
42
Experimental Results
• Device: Intel Quad 2.66GHz with Nvidia GTX460.
• Stereo images are captured by a Bumblebee 3 from Point Grey Inc.
• Time: 80~110ms/frame
• Stereo matching is implemented on GPU.
• The codebook generation and its evaluation is on CPU.
43
Experimental Results
44
Experimental Results
[8] K. J. Yoon and I. S. Kweon, “Adaptive support-weight approach for correspondence search,” IEEE Transactions
on Pattern Analysis and Machine Intelligence, vol. 28, pp. 650-656, 2005.
[9] C. Richardt, D Orr, I Davies, and A Criminisi, “Real-time spatiotemporal stereo matching using the dual-crossbilateral grid,” Proc. of ECCV, 2010.
45
Conclusion
• Propose a stereo system implemented on GPGPU for
real-time performance.
• Good performance at static background Only.