Contours and Optical Flow:
Cues for Capturing Human Motion in Videos
Thomas Brox
Computer Vision and Pattern Recognition Group
University of Bonn
Research partially funded by the German Research Foundation (DFG)
Human pose tracking from video
1. Tracking of markers attached to the body
+ Designed to be easy to track
 Reliable and fast tracking
Thomas Brox
University of Bonn
– Accuracy limited by number of markers
– People may feel uncomfortable
• Introduction
• Segmentation
• Optic Flow
• Summary
2. Tracking features that naturally appear in the images
• Patches (e.g. KLT, SIFT, etc.)
• Contour/Silhouette
• Optic flow
How to extract these features reliably from the images
2
Contour and optic flow based human tracking
Thomas Brox
University of Bonn
• Introduction
• Segmentation
• Optic Flow
• Summary
Joint work with Bodo Rosenhahn
3
Part I
Object Contour Extraction
Object contour extraction
• Find two regions:
object & background
Thomas Brox
University of Bonn
• Introduction
• Segmentation
• Often: Static background
 background subtraction
• Optimality criteria here:
– Strong similarity within regions
– Small boundary
• Optic Flow
• Summary
5
• Bayesian approach:
Level set representation of contours
(Dervieux-Thomasset 1979, Osher-Sethian 1988)
• Introduce embedding function
• Contour C represented as zero-level line of
Thomas Brox
University of Bonn
• Introduction
• Segmentation
• Optic Flow
• Summary
6
Courtesy of Daniel Cremers
Region-based active contours
(Chan-Vese 2001, Paragios-Deriche 2002)
• Minimize negative logarithm:
Thomas Brox
University of Bonn
H(x)
• Introduction
• Segmentation
• Optic Flow
• Gradient descent:
• Summary
H’(x)
plus update of p1 and p2
7
Region statistics
• 7 channels:
Thomas Brox
– 3 color channels (CIELAB)
– 4 texture channels
University of Bonn
• Channels assumed to be independent
• Introduction
• Segmentation
• Optic Flow
• Summary
8
• Probability densities pij approximated by Gaussians
Texture
• Usually modeled by Gabor filters
(Gabor 1946)
Thomas Brox
University of Bonn
• Includes
1. Magnitude
2. Orientation
3. Scale
• Introduction
• Segmentation
• Optic Flow
• Summary
• High redundancy
• Sparse alternative representation feasible
• Nonlinear structure tensor
(Brox et al. 2006)
• Region based local scale measure
(Brox-Weickert 2004)
9
Sparse texture features
Thomas Brox
University of Bonn

• Introduction
• Segmentation
• Optic Flow
• Summary
Gabor filter bank
10
Sparse representation
Examples for contour extraction
Thomas Brox
University of Bonn
• Introduction
• Segmentation
• Optic Flow
• Summary
11
Local region statistics
• Object and background usually
not homogeneous
Thomas Brox
University of Bonn
• Idea: assume them to be
locally homogeneous
• Introduction
• Segmentation
• Optic Flow
• Summary
12
• Probability densities estimated by local Gaussians
Introducing a shape prior
• Idea: object model can serve as 3-D shape prior
Thomas Brox
 Constrains the segmentation, unwanted solutions
discarded
University of Bonn
• Bayesian formula:
• Introduction
• Segmentation
• Optic Flow
• Summary
• Pose parameters of model unknown
 Two variables: contour and pose
13
Joint optimization
• Simultaneously optimize contour and pose:
Thomas Brox
University of Bonn
conventional segmentation part
shape+pose constraint
• Introduction
• Segmentation
• Optic Flow
• Summary
• Iterative alternating scheme:
– Update contour
– Update pose parameters
• Related works: 2-D shape priors
(Leventon et al. 2000, Cremers et al. 2002, Rousson-Paragios 2002)
14
Part II
Optic Flow
Optic flow based tracking
Thomas Brox
University of Bonn
• Introduction
• Segmentation
Image 1 and 2, estimate flow in between
Given pose at Image 1
Pose change due to optic flow
Estimated pose at Image 2
• Optic Flow
• Summary
16
Tracking example
Thomas Brox
University of Bonn
• Introduction
• Segmentation
• Optic Flow
• Summary
17
How to compute the optic flow?
Thomas Brox
University of Bonn
• Introduction
•
Given: two images I(x,y,t) and I(x,y,t+1) in a sequence
•
Goal: displacement vector field (u,v) between these images
•
Variational approach:
• Segmentation
• Optic Flow
• Summary
18
(Horn-Schunck 1981)
Enhanced model
(Brox et al. 2004, Papenberg et al. 2006)
Original Horn-Schunck:
Thomas Brox
University of Bonn
Robust smoothness term (Cohen 1993, Schnörr 1994)
Robust data term (Black-Anandan 1996, Mémin-Pérez 1996)
Gradient constancy
• Introduction
(Brox et al. 2004)
Non-linearized constancy
(Nagel-Enkelmann 1986, Alvarez et al. 2000)
• Segmentation
• Optic Flow
• Summary
19
Spatiotemporal smoothness
Final optic flow model:
(Nagel 1990)
Impact of each improvement
Thomas Brox
University of Bonn
• Introduction
Correct result
• Segmentation
• Optic Flow
• Summary
Nonlinear
Gradient
Horn-Schunck
constancy
constancy
Spatiotemporal
Robust
data
smoothness
term
Robust
smoothness
8
Horn-Schunck
Robust smoothness
6
Robust data term
4
20
1.78
2.44
3.5
5.97
6.36
0
Nonlinear constancy
7.17
2
Gradient constancy
Spatio-temporal smoothness
Accurate and robust optic flow computation
Technique
Nagel
Thomas Brox
University of Bonn
• Introduction
• Segmentation
• Optic Flow
• Summary
21
AAE
10.22°
Uras et al.
8.94°
Alvarez et al.
5.53°
Mémin-Pérez
4.69°
Brox et al. (Noisy)
4.49°
Bruhn et al.
4.17°
Brox et al.
1.78°
Contour and optic flow based human tracking
Thomas Brox
University of Bonn
• Introduction
• Segmentation
• Optic Flow
• Summary
Joint work with Bodo Rosenhahn
22
Summary
• Contours and optic flow can be reliable features for
pose tracking
Thomas Brox
University of Bonn
• Texture, local statistics, and a shape prior are
important for general contour based human motion
tracking
• High-end optic flow helps in case of fast motion
• Introduction
• Segmentation
• Optic Flow
What’s next?
• Summary
• Real-time performance
• Automatic pose initialization
• Prior knowledge about joint angle configurations
23
Outlook
Thomas Brox
University of Bonn
• Introduction
• Segmentation
• Optic Flow
• Summary
Joint work with Bodo Rosenhahn
24
Backup: nonlinear structure tensor
• Texture orientation can be measured with the
structure tensor (second moment matrix)
(Förstner-Gülch 1987, Rao-Schunck 1991, Bigün et al. 1991)
Thomas Brox
University of Bonn
• Introduction
• Gaussian smoothing  nonlinear diffusion
• Segmentation
• Optic Flow
• Summary
Input image
25
Linear
structure tensor
Nonlinear
structure tensor
Backup: region based local scale measure
• Estimate regions, measure
their size
Thomas Brox
University of Bonn
• Nonlinear diffusion: TV flow
(Andreu et al. 2001)
Input image
• Introduction
• Segmentation
• Optic Flow
• Summary
• Tends to yield piecewise
constant images  regions
• Local evolution speed inversely
proportional to size of region
(Steidl et al. 2004)
 local scale measure
26
Local scale