Video
Introduction
Background Models
o Static Background, Frame difference, Running Average,
Selectivity, Median, Running Gaussian Average, GMM
o Shadow Detection
Tracking
o Exhaustive Search, Mean Shift, Optical Flow, Feature
Point Tracking
Performance
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 1
Introduction – Objects of interest
Reliably detecting moving objects of interest in a scene.
1.
2.
3.
Motion detection
Moving object detection & location
Derivation of 3D object properties
When is an object of interest?
Size
Max and min velocity and acceleration
Assumptions:
Mutual correspondence
Common motion
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 2
Introduction – Common problems
Illumination & appearance changes
Gradual (e.g. time of day)
Sudden (e.g. clouds, lights)
Shadows
Weather (e.g. rain, snow)
Background changes
Objects becoming part of the background
Objects leaving the background
Background objects oscillating slightly
Setup
Camera motion
Frame rate
Field of view
Distance to objects
Location of camera
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 3
Introduction – Difference images
Image subtraction
..
..
Colour image version:
Process each channel separately?
Just process hue?
absdiff( frame, background, difference );
© Reproduced by permission of
Dr. James Ferryman, University of Reading
False Positives
False Negatives
The threshold (ε) may be…
Too low or too high
Just right?
High contrast
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 4
Background Models – Static Background
| fn(i,j) - b(i,j) | > T
Simplest approach
Sensitive to the Threshold T
absdiff( current_frame, first_frame, difference );
cvtColor( difference, moving_points, CV_BGR2GRAY );
threshold( moving_points, moving_points, 30, 255,
THRESH_BINARY );
Mat display_image = Mat::zeros( moving_points.size(),
CV_8UC3 );
current_frame.copyTo( display_image, moving_points );
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 5
Background Models – Static Background
| fn(i,j) - b(i,j) | > T
Background b(i,j) = first frame?
© Reproduced by permission of Dr. James Ferryman, University of Reading
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 6
Background Models – Frame Difference
| fn(i,j) – fn-1(i,j) | > Threshold
Depends on the speed of the objects and the frame rate
Sensitive to the Threshold
© Reproduced by permission of Dr. James Ferryman, University of Reading
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 7
Background Models – Running Average
Adapts to changing scene
Incorporates moving objects…
© Reproduced by permission of Dr. James Ferryman, University of Reading
accumulateWeighted( current_frame_gray,
running_average_background, 0.01 );
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 8
Background Models – Selectivity
Selectivity (with running average)
Avoids incorporating moving objects
Adapts to scene changes?
© Reproduced by permission of Dr. James Ferryman, University of Reading
accumulateWeighted( current_frame_gray,
running_average_background, 0.01, foreground_mask );
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 9
Background Models – Colour
Running Average
Selectivity
Selectivity (with
running average)
© Reproduced by permission of Dr. James Ferryman, University of Reading
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 10
Background Models – Colour
vector<Mat> input_channels(3);
split(current_frame,input_channels);
vector<Mat> channels(3);
background.convertTo(temp_background,CV_8U);
absdiff(temp_background,current_frame,difference);
split(difference,channels);
temp_sum = (channels[0] + channels[1] + channels[2])/3;
threshold(temp_sum,foreground,30,255,THRESH_BINARY_INV);
split(background,channels);
accumulateWeighted(input_channels[0], channels[0], alpha, foreground);
accumulateWeighted(input_channels[1], channels[1], alpha, foreground);
accumulateWeighted(input_channels[2], channels[2], alpha, foreground);
invertImage(foreground,foreground);
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 11
Background Models – Colour
if (update_foregrond)
{
accumulateWeighted(input_channels[0], channels[0], alpha /3.0,foreground);
accumulateWeighted(input_channels[1], channels[1], alpha /3.0,foreground);
accumulateWeighted(input_channels[2], channels[2], alpha /3.0,foreground);
}
merge(channels,background);
if (clean_binary_images)
{
morphologyEx(foreground, closed_image, MORPH_CLOSE, Mat());
morphologyEx(closed_image,f oreground, MORPH_OPEN, Mat());
}
foreground_image.setTo(Scalar(0,0,0));
current_frame.copyTo(foreground_image, foreground);
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 12
Background Models – Selectivity
Selectivity (with running average)
Reduce the incorporation of moving objects
© Reproduced by permission of Dr. James Ferryman, University of Reading
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 13
Background Models – Median
Median: Middle value (from an ordered list)
No. of frames (m)
Histogram quantisation?
Computational expense
Adding, storing and removing frames
Change in median can be tracked inexpensively from frame to frame
Can be approximated using aging
Can also use selective update
Could use the Mode instead… (Most common value)
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 14
Background Models – Median Results
Learning Rate = 1.001
Learning Rate = 1.005
Learning Rate = 1.02
© Original Video reproduced by permission of Dr. James Ferryman, University of Reading
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 15
Background Models – Median Algorithm
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 16
Background Models – Running Gaussian Average
Fit a Gaussian distribution over the histogram
Background PDF (μ, σ)
Update using running average
Foreground test
Threshold is a multiple of the standard deviation
Can also use selective update
e.g. Pfinder (Wren et al. 1997)
Problems?
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 17
Background Models – Gaussian Mixture Model
How to deal with multi-modal background pixels
e.g. from trees, water
Stauffer & Grimson, 2000
Algorithm presented is based on that in Sonka (3rd edition) pp.777-780
Model multiple values (3-5) at each point.
Unsupervised learning…
Most popular method for background modelling
© Reproduced by permission of Dr. James Ferryman, University of Reading
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 18
Background Models – GMM Model details
Model each pixel fn(i,j) using k Gaussian distributions Nk = N(μk, Σk)
πn
μn
Σn
Covariance between two channels a and b for pixel (i,j) at frame t:
Σt ((X t(i,j,a) - μk(i,j,a)) * ((Xt(i,j,b) - μk(i,j,b))
Approximate using σ2k
Assume different components independent and of equal variance
Set a learning constant α
in the range 0.01 … 0.1
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 19
Background Models – GMM Model Update
For a new sample fn(i,j)
Select the best close Gaussian distribution
close = within 2.5 σn (i,j,m) of μn (i,j,m)
If there is a best close Gaussian l
If there is no close Gaussian (replace one…)
x = argmink(πn(i,j,k))
μn+1(i,j,x) = fn(i,j)
σ 2n+1(i,j,x) = 2.maxk σ 2n(i,j,k)
πn+1(i,j,x) = ½.mink πn(i,j,k)
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 20
Background Models – GMM Moving Points
Identifying background distributions
Define T, a proportion of the frames in which background pixels should be
visible.
Order the Gaussians by πn+1(i,j,k) / σ n+1(i,j,k)
Gaussians 1..B are considered background where
B = argminb( (Σb=1..k πn+1(i,j,k) ) > T )
Just check if best close Gaussian (or the new Gaussian distribution) is a
background distribution
Finally use morphological dilations and erosions to remove small regions and fill
in holes.
© Reproduced by permission of Dr. James Ferryman, University of Reading
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 21
Background Models – Shadow Detection
Compare current frame with background image…
(Prati 2003)
Intensity / luminance / value drops
Intensity / luminance / value has not dropped too
much (below λ)
Saturation does not increase too much
Hue does not change too much
Hue unpredictable & Change can decrease a little
(Tattersall 2003)
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 22
Background Models – Shadow Detection
© Reproduced by permission of Dr. James Ferryman, University of Reading
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 23
Background Models – Shadow Detection – Code
BackgroundSubtractorMOG2 gmm;
gmm( current_frame, foreground_mask );
threshold( foreground_mask, moving_points, 150, 255,
THRESH_BINARY );
…
threshold( foreground_mask, changing_points, 50, 255,
THRESH_BINARY );
absdiff( moving_points, changing_points, shadow_points );
…
Mat mean_background_image;
gmm.getBackgroundImage( mean_background_image );
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 24
Tracking – Introduction
Used in video surveillance, sports video analysis,
vehicle guidance systems, etc.
A hard task because objects
may be undergoing complex motion
may change shape
may be occluded
may change appearance due to lighting/weather
may physically change appearance
Approaches considered:
Exhaustive search
Mean Shift
Optical Flow
Feature based tracking
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 25
Tracking – Exhaustive Search
Extract object to be tracked from frame
Compare in all possible positions in future frame(s)
Use a similarity metric
E.g. normalised cross correlation
Pick the best match
Need extra degrees of freedom for scale and
orientation.
May fail if object motion is too complex.
Template matching and chamfer matching support
this type of tracking.
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 26
Tracking – Mean Shift
Back-projects a histogram of the object into the
current frame
Searches for a region of the same size within the back
projection looking for the highest (weighted) sum.
Uses hill climbing to iteratively look for the (local)
maximum
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 27
Tracking – Mean Shift
Rect position(starting_position);
TermCriteria criteria( cv::TermCriteria::MAX_ITER, 5, 0.01);
meanShift( back_projection_probabilities, position, criteria);
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 28
Tracking – Dense Optical Flow
Compute a motion field (known as optical flow) for
the entire image
Direction & Magnitude
© Reproduced by permission of Dr. James Ferryman, University of Reading
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 29
Tracking – Dense Optical Flow
Based on the brightness constancy constraint
Object points will have the same brightness over a short
period of time
Need to find the displacement
minimise the residual error
Video
which will
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 30
Tracking – Dense Optical Flow
To compute the optical flow
the displacement is small…
, assuming that
Hence (given our previous equation
So
And reorganising
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 31
Tracking – Dense Optical Flow – Code
calcOpticalFlowFarneback (previous_gray_frame, gray_frame,
optical_flow, 0.5, 3, 15, 3, 5, 1.2, 0);
cvtColor (previous_gray_frame, display, CV_GRAY2BGR);
for (int row = 4; row < display.rows; row+=8)
for(int column = 4; column < display.cols; column+8)
{
Point2f& flow = optical_flow.at<Point2f>(row,column);
line (display, Point(column,row), Point(
cvRound(column+flow.x), cvRound(row+flow.y)),
passed_colour);
}
gmm.getBackgroundImage( mean_background_image );
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 32
Tracking – Feature-based Optical Flow
We cannot accurately compute optical flow for
constant regions or along edges.
Often better to compute optical flow just for
features… (e.g. Lucas Kanade feature tracker)…
© Reproduced by permission of Dr. James Ferryman, University of Reading
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 33
Performance – Ground Truth Formats
1.
Label pixel masks
•
Label all pixels corresponding to objects of interest (e.g. moving people).
•
Very time consuming.
•
Very hard to get agreement particularly near boundaries.
2.
Bounding Boxes
•
Put a box around all objects of interest.
•
Much faster to do.
•
Much more inaccurate (e.g. size of bounding box vs. area of object).
3.
Labelled events
•
Simply label a frame or range of frames to indicate when some event has
taken place (e.g. car has stopped).
•
Much easier to compute.
•
May need a bounding box or labelled pixels as well for the frame.
•
Quite hard to agree on what frame an event occurs in.
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 34
Performance – Metrics
For labelled pixel masks or labelled events
we can use metrics like Precision, Recall, F1 measure, etc.
For bounding boxes we must use other metrics such as
1.
Dice coefficient
2.
Overlap
3.
Percentage of Lost Tracks
•
Lost Track = Overlap falls below a threshold
•
Threshold = 10% or 20% ???
Video
Based on A Practical Introduction to Computer Vision with
OpenCV by Kenneth Dawson-Howe © Wiley & Sons Inc. 2014
Slide 35