Fast and Accurate Optical Flow
Estimation
Primal-Dual Schemes and
Second Order Priors
Thomas Pock and Daniel Cremers
CVPR Group, University of Bonn
Collaborators: Christopher Zach, Markus Unger, Werner Trobin, and Horst Bischof
Variational Optical Flow – Short History
1981
Horn and Schunck
1993
Black and Anadan,
Cohen
2000
2004
Aubert
Brox et al.
2006
Bruhn et al.
Outline
•
•
•
•
•
Model of Horn and Schunck
TV-L1 Model
Fast Numerical Scheme
Parallel Implementation
2nd order Prior
The Model of Horn and Schunck [1]
Regularization Term
Data Term (OFC)
+ Convex
+ Easy to solve
- Does not allow for sharp edges in the solution
- Sensitive to outliers violating the OFC
[1] Horn and Schunck. Determinig Optical Flow. Artificial Intelligence, 1981
Can we do better?
• Replace quadratic functions by L1 – norms
• Done by Cohen, Aubert, Brox, Bruhn, ...
+Allows for discontinuities in the flow field
+Robust to some extent to outliers in the OFC
+Still convex
- Much harder to solve
How can we minimize this functional ?
• Compute Euler-Lagrange Equations
• Non-linear, non-smooth, ...
Standard Approach
• Replace L1 – norm by regularized variants
(Charbonnier function)
• Example:
• Small epsilon: Nearly degenerated
• Large espilon: Smears edges
Our Approach(1)
• Introduce auxiliary variables and constraints
• Quadratic penalty
Our Approach(2)
• What do we gain?
• We solve a sequence of simpler problems
1D Problem
ROF Model [2]
Algorithm[3]:
1. For fixed (u´,v´), solve for(u,v) using Chambolle‘s algorithm[4]
2. For fixed (u,v), solve for (u´,v´) using a 1D shrinkage formula
3. Goto 1. until convergence
[2] Rudin, Osher and Fatemi. Nonlinear Total Variation Based Noise Removal Algorithms, 1992
[3] Zach, Pock and Bischof. A Duality Based Algorithm for Realtime TV-L1 Optical Flow, DAGM 2007
[4] Chambolle. An Algorithm for Total Variation Minimization, 2004.
Implementation
• Numerical scheme can be easily parallelized
• We use state-of-the-art GPUs
Performance Evaluation
• TV-L1 Optical Flow Implemented in CUDA 2.0
• Computed on Nvidia GeForce GTX 280
• 25 Overall Iterations (5 Chambolle Iterations)
Image Size
128x128
256x256
512x512
Frames per Second
192
108
36
Results for TV-L1
Input Image:
Ground Truth:
Our Results:
2nd order Prior
• TV regularization favors piecewise constant flow
fields (frontoparallel motion)
• Extension to piecewise affine flow fields?
• Approach of Cremers et al. [5]
– Fixed number of regions
• Approach of Nir et al. [6]
– Over-parametrized optical flow
• Our approach [7]
– 2nd order derivatives to regularize flow field
[5] Cremers and Soatto, Motion Competition: A Variational Framework for Piecwise Parametric Motion
Segmentation.
[6] Nir, Bruckstein and Kimmel, Over-Parameterized Variational Optical Flow, IJCV 2007
[7] Trobin, Pock, Cremers and Bischof, An Unbiased Second-Order prPior for High-Accuracy
Motion Estimation, DAGM 2008
2nd-L1 Optical Flow
• 2nd order derivatives are not orthogonal
• We use a transformation due to Danielsson [8]
• Optimization
– Similar strategy to TV-L1
– 4th order PDE
[8] Danielsson and Lin, Efficient Detection of Second-Degree Variations in 2D and 3D Images, 2001.
Comparison
Ground truth
TV-L1
2nd -L1
Results for 2nd-L1
Ground Truth:
Our Results:
Conclusion
• TV-L1 Optical Flow
– Fast Numerical Scheme
• Parallel Implementation
– Realtime Performance
• 2nd order prior
– Piecewise affine motion
Recent Application: Tracking