Light Field Video Stabilization
ICCV 2009, Kyoto
Presentation for CS 534: Computational Photography
Friday, April 22, 2011
Brandon M. Smith
Li Zhang
University of Wisconsin, Madison
Hailin Jin Aseem Agarwala
Adobe Systems Incorporated
Motivation
Feng Liu et al., SIGGRAPH 2009
© 2011 Brandon M. Smith
Video Stabilization: Professional Solutions
camera crane
steadicam
camera dolly
Use special hardware to avoid camera shake
• Expensive
• Cumbersome
© 2011 Brandon M. Smith
Video Stabilization: Software Solutions
2D-transformation, software based methods
Burt & Anandan, Image Stabilization by Registration to
a Reference Mosaic. DARPA Image Understanding
Workshop, 1994
Hansen et al., Real-Time Scene Stabilization and
Mosaic Construction. DARPA Image Understanding
Workshop, 1994
…
Lee et al., Video Stabilization Using Robust Feature
Trajectories. ICCV 2009
• Distant, planar scenes
• Rotational camera motion
© 2011 Brandon M. Smith
Video Stabilization: Very Simple Algorithm
© 2011 Brandon M. Smith
Video Stabilization: Camera Solutions
Vibration Reduction (VR)
sensor stabilization
optical stabilization
• Limited DOF
• Small baseline
© 2011 Brandon M. Smith
Video Stabilization: State-of-the-Art
Demos taken from Feng Liu et al., ACM Transactions on Graphics 2011
http://web.cecs.pdx.edu/~fliu/project/subspace_stabilization/index.htm
Video Stabilization: State-of-the-Art
Works well unless
•
•
•
•
Background has few“trackable” visual features - demo
There are large dynamic targets - demo
Too much rolling shutter and motion blur – demo 1, demo 2
Parallax is too significant
Demos taken from Feng Liu et al., ACM Transactions on Graphics 2011
http://web.cecs.pdx.edu/~fliu/project/subspace_stabilization/index.htm
Video Stabilization Challenges
• Violent shake
• Nearby dynamic targets
• Few “trackable” background visual features
© 2011 Brandon M. Smith
New Approach
Panasonic HD Stereo Camcorder
LOREO 3D Lens in a Cap 9005
Viewplus Profusion 25C
© 2011 Brandon M. Smith
New Approach
Panasonic HD Stereo Camcorder
LOREO 3D Lens in a Cap 9005
Viewplus Profusion 25C
Existing applications
• New-view synthesis [Levoy & Hanrahan SIGGRAPH ‘96, Gortler et al. SIGGRAPH ‘96]
• Synthetic aperture [Wilburn et al., SIGGRAPH 2005] - demo
• Noise Removal [Zhang et al., CVPR 2009]
New application
• Video Stabilization
Left Input
Synthetic Viewpoint
Right Input
© 2011 Brandon M. Smith
Why Does a Camera Array Help?
Stabilization as image based rendering [Buehler et al. CVPR 2001]
Virtual Camera Path
Actual Camera Path
Synthesize a video along a virtual smooth camera path
More input views at each time instant
• Easier to work with dynamic scenes
• Better handling of parallax
© 2011 Brandon M. Smith
How to Avoid Structure from Motion?
Insight: Only Need Relative Transformation Rf, tf
Virtual Camera Path
Actual Camera Path
Spacetime optimization:
Maximize smoothness of virtual video as function of {Rf, tf}f=1…F
Advantage:
Do not need to compute 3D input camera path
© 2011 Brandon M. Smith
How to Define the Smoothness of a Video?
pf-1 , Zf-1
pf+1 , Zf+1
pf , Zf
Original Camera
reproj(pf-1, Zf-1, Rf-1,tf-1) reproj(pf, Zf, Rf,tf) reproj(pf+1, Zf+1, Rf+1,tf+1)
qf-1
qf
qf+1
Virtual Camera
F-1
 w || q
E ( {R,t}f=1,…,F ) =
f=2 jf
f
f,j
f,j
- 12
( qf-1,jprev+ qf+1,jnext ) ||
2
+ Ereg
© 2011 Brandon M. Smith
Salient Features
Canny
Edge
Maps
Original
Camera
Aside: you can generate these in Matlab:
>> img = imread(‘image.png’);
>> edgemap = edge(img, ‘canny’);
© 2011 Brandon M. Smith
Matching Features
Canny Edge Maps
Optical Flow [Bruhn et al. 2005]
© 2011 Brandon M. Smith
Algorithm Outline
1. Compute depth map for each time instant [Smith et al. 2009]
2. Compute optical flow for each time instant [Bruhn et al. 2005]
3. Detect Canny edges, use flow to match edges over time
4. Run spacetime optimization to find {R,t}f=1,…,F
E ({R,t}f=1,…,F ) =
F-1
w || q

f
f,j
f=2 j
f,j
( qf-1,jprev+ qf+1,jnext ) ||
2
1
--
2
+ Ereg
f
5. New view synthesis
© 2011 Brandon M. Smith
New View Synthesis
Relative
Transformation
R,t
Input Images
Depth Maps
New View
Synthesis
Synthesized Image
© 2011 Brandon M. Smith
Results
http://pages.cs.wisc.edu/~lizhang/projects/lfstable/
© 2011 Brandon M. Smith
Summary of Contributions
• Use an array for stabilization
• Stabilization without structure from motion
• Can handle challenging cases:
– Nearby, dynamic targets
– Large scene depth variation
– Violent camera shake
http://pages.cs.wisc.edu/~lizhang/projects/lfstable/
© 2011 Brandon M. Smith
Limitations and Future Work
• Increase algorithm efficiency
• Use fewer cameras (two instead of five)
• Motion deblurring with camera arrays
• Better handle image periphery problems
• Evaluate a range of camera baselines
http://pages.cs.wisc.edu/~lizhang/projects/lfstable/
© 2011 Brandon M. Smith
Results and Paper
http://pages.cs.wisc.edu/~lizhang/projects/lfstable/