Spatiotemporal Regularity Flow
(SPREF)
Mubarak Shah
Computer Vision Lab
School of Electrical Engineering & Computer Science
University of Central Florida
Orlando, FL 32765
What are good features?




Color Histograms
Eigen vectors
Wavelet Coefficients
Edges





Spatiotemporal Surfaces of edges
XY, XT, YT slices
Spatial/spatiotemporal tensors
SIFT
Optical Flow
SPREF

New Spatiotemporal feature for VACE
Generalization of Isophotes, Optical flow,…
Can be computed when gradient is zero
It analyzes whole region instead of a single pixel

Applications







Image and Video In-painting
Object removal
Video Compression
Tracking, Segmentation, …
Spatiotemporal Regularity




Definition: A spatiotemporal volume is regular
along the directions, in which the pixels
change the least.
SPatiotemporal REgularity Flow (SPREF)
 3D vector field ζ
 models the directions of regularity
No motion (Spatial Regularity)
 Depends on the regularity of a single frame
Presence of motion (Temporal Regularity)
 Global motion


Single regularity model
Local motion

Multiple regularity models
Estimating SPREF

…gives the directions, along which the sum
of the gradients is minimum:
2
E  
( F  H )( x, y, t )
dxdydt
 ( x, y, t )
where F is the spatiotemporal volume, and H
is a regularizing filter (Gaussian)
The SPREF Energy Functions

The energy function is modified according to
the flow type:

x-y Parallel: ζ(c1'[t], c2'[t],1)
E   c1 '[t ] f x  c2 '[t ] f y  f t  dxdydt
2

y-t Parallel: ζ(1,c2'[x], c3'[x])
E    f x  c2 '[ x] f y  c3 '[ x] f t  dxdydt
2

x-t Parallel: ζ(c1'[y],1, c3'[y])
E   c1 '[ y ] f x  f y  c3 '[ y ] f t  dxdydt
2
Solving for the SPREF

Approximate each flow component, cm'[p],
with a 1D spline

Incorporates multiple frames in the solution.
cm '[ p]   i b(2 p  i)
l
i
• Quadratic minimization of the energy functions
• Solve for the spline parameters
Solving T-SPREF Equation
The original synthetic sequence (8 frames)
There are three types of planar parallelism constraints.
x-y Parallelism: ζ(c1'[t], c2'[t],1)
y-t Parallelism: ζ(1,c2'[x], c3'[x])
x-t Parallelism: ζ(c1'[y],1, c3'[y])
The SPREF Curves

… define the actual paths, along which the
GOF is regular.
p
cm [ p]   cm '[i]
i 1
m {1,2,3}
T-SPREF - An Overview

Demo
x-y Parallel SPREF
y-t Parallel SPREF
y
y
t
x
ζ(1,c2'[x], c3'[x])
x
x-t Parallel SPREF
y
y
t
x
ζ(c1'[y],1, c3'[y])
x
T-SPREF Results (Flower Sequence)
Oblique View
Top View
Side View
T-SPREF Results (Alex Sequence)
Oblique View
Top View
x
y
x
Side View
y
t
t
t
The Affine SPREF (A-SPREF)

When the directions of regularity depend on
multiple axes (zooming, rotation and etc.)



Precision of T-SPREF goes down
Translational flow model to Affine flow model
Affine (A-)SPREF
Flow energy equation:
H '
H '
H
E   ( F  )c1[ x, y, t ]  ( F  )c2 [ x, y, t ]  F *
x
y
t
Vi
c1' [ x, y, t ]  a11[t ]x  a12[t ] y  a13[t ]
c2' [ x, y, t ]  a21[t ]x  a22[t ] y  a23[t ]
2
Comparison of T- and A- SPREF
1st row: A synthetic sequence from the Lena image.
2nd row: T-SPREF approximation to the underlying directions of
regularity.
3rd row: A-SPREF approximation of the directions of regularity.
More examples
T-SPREF
T-SPREF
A-SPREF
A-SPREF
Comparison of T- and A- SPREF
Optical Flow Vs SPREF

SPREF carries similar but not necessarily the
same information as the optical flow.




SPREF captures both the spatial and temporal
regularity
Optical flow only cares about motion information
in temporal direction.
When motion exists, the directions of xy parallel
SPREF depend on direction of motion.
If the motion is globally translational, then xyparallel SPREF converges to the optical flow.
Optical Flow Vs SPREF



Optical Flow is not well-defined where the
spatiotemporal gradients are insignificant.
Spline-based formulation of SPREF minimizes
over multiple frames.
The true optical flow usually lacks plane
parallelism.
Optical Flow Vs SPREF
Applications of SPREF
Inpainting

Filling in the regions of missing data

Image Inpainting



Missing regions create spatial holes
Inpainting the missing region in the SPREF direction
Video Inpainting


Missing regions create spatiotemporal holes
Inpainting these holes require using the information
from temporal neighbors.
Image Inpainting
Video Inpainting

Requires understanding the temporal behavior of
the pixels.

The temporal behavior of the undamaged pixels
gives clues about the behavior of the damaged
pixels

Temporal behavior

Modeled explicitly by x-y Parallel SPREF
Video Inpainting

The algorithm (cont’d)
1.
2.
3.
Estimate the x-y Parallel SPREF curves using the nonmissing regions.

The pixels along the SPREF curves vary smoothly
Fit a spline to the non-missing pixels along each flow
curve.
Interpolate the values of the missing pixels from the
splines
Results

Big Bounce (Before)
Results

Big Bounce (Flow)
Results

Big Bounce (After)
Supervised Removal of Objects
from Videos
Motivation

Object removal from videos




Preceding step to video inpainting
Manual selection of the object from each frame is
required.
Time consuming
Use x-y Parallel SPREF to decrease the
amount of manual work

Removal along the SPREF curves
Algorithm

Given a group of frames (GOF):
1)
2)
3)
Compute the x-y Parallel SPREF, and the
SPREF curves
Remove the object from the first and the last
frames of the GOF
Remove the pixels along the curves, whose first
and last pixels have been removed.
Results

Golden Eye (Final)

86% reduction in manual work!
Video Compression Using SPREF
3D Wavelet Decomposition

Problem


The spatiotemporal regularity of the GOF is not
taken into account
Solution


Decompose the GOF along the SPREF directions
Entropy along these directions is lower:

Higher compression rate
SPREF-based Video Compression

Warping the wavelet basis along the flow
curves



x-y Parallel : G(x,y,t) = (x+c1[t], y+c2[t], t)
y-t Parallel : G(x,y,t) = (x,y+c2[x],t+c3[x])
x-t Parallel : G(x,y,t) = (x+c1[y], y, t+c3[t])
Choosing the correct SPREF type

The correct SPREF type is the one that
minimizes the compression cost : Di + λRi

Di: Reconstruction error

λ: Lagrange multiplier

Ri: Bit cost of the bandelet and flow coefficients
Segmentation for Optimal Compression

Find the segmentation of the GOF (F) into
subGOFs (Fi), such that the total
compression cost is minimized:
D  R   Di  Ri
i
Fi
Oct-tree Segmentation



Recursively partition the GOF (F) into
rectangular prisms (cuboids), Fi.
Compute the best flow and the compression
cost for each cuboid.
Use split/merge algorithm to achieve the final
segmentation.

Merge the child nodes if:
Di  Ri   D j  R j
i
Compression results for frames 98-105 of the
Alex sequence at 1000kbps
Compression results for frames 11-18 of the Akiyo
sequence at 480kbps
Compression results for frames 99-106 of the Mobile
sequence at 350kbps
Compression results for frames 26-33 of the
Foreman sequence at 500kbps
Compression Results
(a)
(b)
The bit-rate vs PSNR plots of (a) Alex, (b) Akiyo. Both SPREF-based
compression and LIMAT framework are shown in the results.
LIMAT framework, Secker and Taubman, IEEE TIP, 2004.
Compression Results (cond.)
(a)
(b)
The bit-rate vs PSNR plots of (a) Foreman, (b) Mobile. Both SPREFbased compression and LIMAT framework are shown in the results.
Summary

SPREF






New Spatiotemporal Feature
Computes direction of regularity simultaneously in
space & Time
Similar to optical flow, edge direction..
SPREF is plane parallel (xy, xt, yt)
SPREF is computed using region/image
information instead of a single pixel
SPREF is defined even when gradient is zero
Summary

Applications





Image and Video In-painting
Object Removal
Video Compression
Tracking
Segmentation
Orkun Alatas
August 16th, 1977 - September 3rd, 2005
Publications

Orkun Alatas, Omar Javed, and Mubarak Shah, “Video
Compression Using Structural Flow", International Conference on
Image Processing, Genova, Italy, September 11-14, 2005.

Orkun Alatas, Omar Javed, and Mubarak Shah, “Video
Compression Using Spatiotemporal Regularity Flow, IEEE
Transactions on Image Processing, December 2006.

Orkun Alatas, Pingkun Yan, and Mubarak Shah, “Spatiotemporal
Regularity Flow, (SPREF): Its Estimation and Applications”,
IEEE Transactions on Circuit & Systems Video Technology
(accepted).
Computer Vision Lab
shah@cs.ucf.edu
http://www.cs.ucf.edu/~vision
Group