Background Subtraction
1
Purpose of Background Subtraction

Reduce problem set for further processing



Only process part of picture that contains the
relevant information
Segment the image into foreground and
background
Add a virtual background
2
Encountered Problems

Lighting


Shadows
Gradual/Sudden illumination changes

Camouflage

Moving objects


Foreground aperture
Foreground object becomes motionless
3
Lighting and Shadows

Weight the luminance with other
characteristics



Depth of object
Region/Frame information
Adjust the background model with time

Store a history of previous backgrounds
4
5
Widely Used
6
Simple Approach
7
Frame Differencing
The accuracy of this approach is dependent on speed of
movement in the scene. Faster movements may require higher
8
thresholds
Frame Differencing
This approach will only work for cases where all
foreground pixels are moving and all background
pixels are static.
9
Mean Filter
where N is the number of preceding images taken for averaging. This
averaging refers to averaging corresponding pixels in the given images.
N would depend on the video speed (number of images per second in10the
video) and the amount of movement in the video.
Mean Filter
11
Median Filter
Nathan Johnson
12
Median Filter
Nathan Johnson
13
Advantages & Shortcomings
14
Advantages & Shortcomings
15
Adaptive Background Mixture Models
16
Normal Gaussian Distribution
As a result, a pixel, once it has become foreground, can only become background
again when the intensity value gets close to what it was before turning foreground.
This method, however, has several issues: It only works if all pixels are initially
17
background pixels (or foreground pixels are annotated as such).
Running Gaussian Average



The pdf of every pixel is characterized by
mean and variance.
In order to initialize variance, we can, for
example, use the variance in x and y from a
small window around each pixel.
Note that background may change over time
(e.g. due to illumination changes or non-static
background objects). To accommodate for
that change, at every frame , every pixel's
mean and variance must be updated
.18
Algorithm Overview
If a pixel is categorized as foreground for a too long period of time, the
background intensity in that location might have changed (because illumination
has changed etc.). As a result, once the foreground object is gone, the19 new
background intensity might not be recognized as such anymore.
Mixture Model
20
Mixture Model
Nathan Johnson
21
Model Adaptation
Nathan Johnson
22
Background Model Estimation
23
Background Model Estimation
Nathan Johnson
24
Advantages Vs Shortcomings
25
Relevant papers
Nathan Johnson
26
Summary
27
Selectivity
28
Selectivity
29
Limitations (Selectivity)
Nathan Johnson
30
Kernel Density Estimators
Kernel density estimation is a fundamental data smoothing problem
where inferences about the population are made, based on a finite
samples.
Kernel density estimates are closely related to histograms, but can
be endowed with properties such as smoothness or continuity 31by
using a suitable kernell.
Mean- Shift Based Estimation
Mean Shift Based Estimation
Mean shift is a procedure for locating the maxima of density function from given
discrete data sampled density function.
The simplest such algorithm would create a confidence map in the new image
based on the color histogram of the object in the previous image, and use mean
shift to find the peak of a confidence map near the object's old position.
The confidence map is a probability density function on the new image, assigning
each pixel of the new image a probability, which is the probability of the 33pixel
color occurring in the object in the previous image.
Problems & Solutions
34
Eigen Backgrounds



The method uses the difference between input
image and the reconstructed background image
to detect foreground objects based on
eigenvalue decomposition.
Foreground regions are represented in the
reconstructed image using eigenbackground.
As the principle components of a scene are its
background in general, the eigenbackgrounds
that are used to reconstruct the background
depict the characteristics of background.
35
Eigen- backgrounds
36
Spatial Correlation
Nathan Johnson
37
Binary Morphology



Binary Morphological techniques probe an
image with a small shape or template called a
structuring element.
The structuring element is positioned at all
possible locations in the image and it is
compared
with
the
corresponding
neighbourhood of pixels.
Some operations test whether the element
"fits" within the neighbourhood, while others
test whether it "hits" or intersects the
neighbourhood.
38
Summary
Nathan Johnson
39
Summary
Nathan Johnson
40
Definition of Motion Detection

Action of sensing physical movement in a give
area

Motion can be detected by measuring change in
speed or vector of an object
41
Motion Detection
Goals of motion detection
• Identify moving objects
• Detection of unusual activity patterns
• Computing trajectories of moving objects
Applications of motion detection
• Indoor/outdoor security
• Real time crime detection
• Traffic monitoring
Many intelligent video analysis systems are based on
motion detection.
42
Two Approaches to Motion
Detection

Optical Flow


Compute motion within region or the frame as a
whole
Change detection


Detect objects within a scene
Track object across a number of frames
43
Background Subtraction



Uses a reference background image for
comparison purposes.
Current image (containing target object) is
compared to reference image pixel by pixel.
Places where there are differences are detected
and classified as moving objects.
Motivation: simple difference of two images
shows moving objects
44
a. Original scene
Subtraction of scene a from scene b
b. Same scene later
Subtracted image with threshold of 100
45
Static Scene Object Detection and
Tracking




Model the background and subtract to obtain
object mask
Filter to remove noise
Group adjacent pixels to obtain objects
Track objects between frames to develop
trajectories
46
Background Modelling
47
Background Model
48
After Background Filtering…
49
Approaches to Background Modeling

Background Subtraction

Statistical Methods
(e.g., Gaussian Mixture Model, Stauffer and Grimson 2000)
Background Subtraction:
1. Construct a background image B as average of few images
2. For each actual frame I, classify individual pixels as
foreground if |B-I| > T (threshold)
3. Clean noisy pixels
50
Background Subtraction
Background Image
Current Image
51
Statistical Methods

Pixel statistics: average and standard deviation of
color and gray level values
(e.g., W4 by Haritaoglu, Harwood, and Davis 2000)

Gaussian Mixture Model (e.g., Stauffer and
Grimson 2000)
52
Proposed Approach
Measuring Texture Change



Classical approaches to motion detection are
based on background subtraction, i.e., a model of
background image is computed, e.g., Stauffer and
Grimson (2000)
Our approach does not model any background
image.
We estimate the speed of texture change.
53
In our system we divide video plane in disjoint blocks
(4x4 pixels), and compute motion measure for each block.
mm(x,y,t) for a given block location (x,y) is a function of t
54
8x8 Blocks
55
Block size relative to image size
Block 24x28
1728 blocks
per frame
Image Size:
36x48 blocks
56
Motion Measure Computation





We use spatial-temporal blocks to represent videos
Each block consists of NBLOCK x NBLOCK pixels
from 3 consecutive frames
Those pixel values are reduced to K principal
components using PCA (Kahrunen-Loeve trans.)
In our applications, NBLOCK=4, K=10
Thus, we project 48 gray level values to a texture
vector with 10 PCA components
57
Motion Measure Computation
3D Block Projection with PCA (Kahrunen-Loeve
trans.)
t+1
t
t-1
4*4*3 spatial-temporal block
Location I=24, J=28,
time t-1, t, t+1
48-component block
vector (4*4*3)
10 principal components
-0.5221 -0.0624 -0.1734 -0.2221 -0.2621 -0.4739 -0.4201 -0.4224 -0.0734 -0.1386
58
Texture of spatiotemporal blocks works
better than color pixel values


More robust
Faster
We illustrate this with texture trajectories.
59
499
624
863
1477
60
Detection of Moving Objects Based
on Local Variation
For each block location (x,y) in the video plane
•
•
•
Consider texture vectors in a symmetric window [tW, t+W] at time t
Compute the covariance matrix
Motion measure is defined as
the largest eigenvalue of the covariance matrix
61
Background Subtraction and
Matting
Image Stack
time
255
0

t
As can look at video data as a spatiotemporal volume



If camera is stationary, each line through time
corresponds to a single ray in space
We can look at how each ray behaves
What are interesting things to ask?
Background Subtraction

A largely unsolved problem…
One video
frame
Estimated
background
Difference
Image
Thresholded
Foreground
on blue
How does Superman fly?


Super-human powers?
OR
Image Matting and Compositing?
Compositing Procedure
1. Extract Sprites (e.g using Intelligent Scissors in Photoshop)
2. Blend them into the composite (in the right order)
Multiple Alpha Blending




So far we assumed that one image
(background) is opaque.
If blending semi-transparent sprites (the “A
over B” operation):
Icomp = aaIa + (1-aa)abIb
acomp = aa + (1-aa)ab

Note: sometimes alpha is
premultiplied:
im(aR,aG,aB,a):

Icomp = Ia + (1-aa)Ib

(same for alpha!)
“Pulling a Matte”

Problem Definition:

The separation of an image C into





A foreground object image Co,
a background image Cb,
and an alpha matte a
Co and a can then be used to composite the
foreground object into a different image
Hard problem



Even if alpha is binary, this is hard to do
automatically (background subtraction problem)
For movies/TV, manual segmentation of each frame
is infeasible
Need to make a simplifying assumption…
Blue Screen
Blue Screen matting


Most common form of matting in TV studios &
movies
A form of background subtraction:


Need a known background
Compute alpha as SSD(C,Cb) > threshold


Hope that foreground object doesn’t look like
background



Or use Vlahos’ formula: a = 1-p1(B-p2G)
no blue ties!
Why blue?
Why uniform?
The Ultimatte
p1 and p2
Blue screen for superman?
Semi-transparent mattes

What we really want is to obtain a true
alpha matte, which involves semitransparency

Alpha between 0 and 1
Matting Problem: Mathematical
Definition
Why is general matting hard?
Solution #1: No Blue!
More examples
Removing Shadows (Weiss,
2001)

How does one detect (subtract away)
shadows?
Averaging Derivatives
Recovering Shadows
Compositing with Shadows
Outline
•
•
•
•
•
Applications of segmentation to video
Motion and perceptual organization
Motion field
Optical flow
Motion segmentation with layers
Video
•
•
A video is a sequence of frames captured
over time
Now our image data is a function of space
(x, y) and time (t)
Applications of segmentation to
video
•
Background subtraction


A static camera is observing a scene
Goal: separate the static background from the
moving foreground
Applications of segmentation
to video
•
Background subtraction


Form an initial background estimate
For each frame:




Update estimate using a moving average
Subtract the background estimate from the frame
Label as foreground each pixel where the magnitude
of the difference is greater than some threshold
Use median filtering to “clean up” the results
Applications of segmentation to video
•
•
Background subtraction
Shot boundary detection



Commercial video is usually composed of shots or
sequences showing the same objects or scene
Goal: segment video into shots for summarization
and browsing (each shot can be represented by a
single keyframe in a user interface)
Difference from background subtraction: the
camera is not necessarily stationary
Motion and perceptual
organization
•
Sometimes, motion is the only cue
Motion estimation techniques
•
Direct methods



•
Directly recover image motion at each pixel from
spatio-temporal image brightness variations
Dense motion fields, but sensitive to appearance
variations
Suitable for video and when image motion is small
Feature-based methods



Extract visual features (corners, textured areas)
and track them over multiple frames
Sparse motion fields, but more robust tracking
Suitable when image motion is large (10s of pixels)