High Quality Novel View Synthesis
Based on Low Resolution Depth Image
and High Resolution Color Image
Jui-Chiu Chiang, Zheng-Feng Liu, and Wen-Nung Lie
National Chung Cheng University
Taiwan
2012/11/11
1
Outline





2012/11/11
Introduction
Our hybrid camera system
The proposed depth processing algorithm
Experimental results
Conclusion
2
Color + Depth in 3DTV/FTV

Depth image, in combination with DIBR (Depth
Image-Based Rendering) technique, is important
in multi-view 3DTV/FTV applications.
Color image
Depth image
DIBR
view9
2012/11/11
view8
view7
view6
view5
view4
view3
view2
view1
3
Depth acquisition

From stereo matching via disparity estimation



From active photo-electrical sensing


Kinect
2012/11/11
Unreliable for textureless regions
Computationally intensive
Lower resolution
Higher acquisition speed and accuracy
SR4000
ToF
4
Hybrid camera systems



High-resolution color cameras + Low-resolution depth
camera (mis-alignment between their optical axis)
[12]: Use warped data of depth camera to refine ROI
disparity of the dual-eye camera
Proposed: one color image is used to refine the warped (to
both sides) and up-sampled depth images
SR4000
Camera L
2012/11/11
Camera R
5
Processing diagram of our system
left color image
Left preliminary
Left high-quality
high-resolution depth
high-resolution depth
image processing
image processing
Right preliminary
Right high-quality
high-resolution depth
image processing
high-resolution depth
Left depth image
central depth image
right color image
2012/11/11
Right depth image
image processing
6
Generation of preliminary highresolution depth image

Purpose: to generate high-resolution depth image
at the left and right sides
Depth image from
depth camera
Pre-processing depths to
remove noise
Calibrated camera
parameters
Warping and up-sampling
depth image to the
reference views
Preliminary high
resolution depth images
Post-processing warped
depths to remove incorrect
depth values
2012/11/11
7

Pre-processing of the depth map


Low resolution to high resolution depth warping


Forward warping with a block size of 55 pixels
Post-processing by morphological operation

2012/11/11
33 median filter
Morphological opening
8
Generation of high quality highresolution depth image

Color-image-guided depth map refinement
Segment n and Db(x,y)
1
Sn
depth value
is correct?
Yes
depth value is
unchanged
0
No
depth value D(x,y) with
Db(x,y)=0 inside the
segment n is averaged
and used as the updated
depth value
depth value
is correct?
No
Yes
depth value D(x,y) with
Db(x,y)=1 inside the
segment n is averaged
and used as the updated
depth value
depth value is
unchanged
2012/11/11
9


Binarizing the warped depth image to have
foreground/background classification
Color segmentation by mean-shift algorithm



2012/11/11
Label each segment as FG or BG
Refine the corresponding depth values according
to the segment categorization
Bilateral filtering is performed for the edge parts
of the modified depth image
10
Experiments – system
configurations



2012/11/11
Color image: 640x480
Depth image: 176x144 (SR4000)
Intel Core 2 Duo Q6600 2.33 GHz, and DDR2
800 4GB
11
Preliminary high-resolution depth
image
After depth
warping
2012/11/11
(a)
(c)
(e)
(b)
(d)
(f)
Traditional depth
warping
After morphological
operation
12
High-quality high-resolution
depth image
After
correction
Meanshift
(a)
(c)
(e)
Incorrect
pixels
(b)
2012/11/11
(d)
(f)
Bilateral
filtering
13
A series of generated depth maps
2012/11/11
14
Virtual view synthesis based on
various depth maps
(a)
(c)
(e)
(b)
(d)
(f)
Hole
filled
2012/11/11
Bilateral
filtering
Bilateral filtering
on edges
Bilateral filtering on
15
edges + Gaussian
Speed performance
Pre-processing
Post-processing
Procedure
Time (sec)
Depth Warping
0.117
Color segmentation
0.742
Bilateral filtering on edges
0.352
Gaussian filtering
0.104
Novel view synthesis by DIBR (including hole filling)
2012/11/11
0.144
16
Conclusion



A hybrid camera system consisting of highresolution color camera and low-resolution depth
camera is proposed
A high-quality high-resolution depth images can
be obtained for the left and right eyes – multiview video plus depth (N=2 channels).
Future work

2012/11/11
Combine Kinect device and stereo camera to get better
depth image and cheaper cost
17