Users Guide:Showframes
Showframes is a comprehensive program that makes use of relatively simple image processing
techniques to track, classify, and count objects in videos of various roadways. The main idea behind the
program is to exploit object motion and utilize the large number of frames available. To classify and
count the objects, 4 main variables are required, 1 of which is calculated from 2 others.
First, once an object is detected in a frame, 5 feature vectors are created and updated for as
long as that object is in successive frames (Nx1 where N is the number of frames). These features are
area, aspect ratio, density, and the (x,y) center coordinates which are used to calculate the speed.
Second, for every detected object the variable seg will tell you the starting frame, ending frame, object,
direction, and a scaled version of the object frame number. Third, with the 5 feature vectors and the seg
variable, the averaged feature matrix is created by dividing the sum of each feature vector by the
number of frames (Nx4 matrix where N is the number of objects). The fourth and final main variable is
the mean feature matrix (3x4 – 3 different objects with 4 different features) which is created by running
the program on a “training” folder explained later. The program compares this mean feature matrix to
every row of the averaged feature matrix to classify what each object is. Next the menu options will be
explained followed by a more detailed look at folders and MAT files used.
Menu options (MAT files explained in more detail after):
1. Toggle mode: Just use this if you really need to single step through a video file instead of run
through the entire file (more for debugging).
2. Crop image: You need to crop one image from each folder before you analyze that folder which
creates and saves “crop_settings.mat “. This just has to be done once per folder, but it can be
redone anytime. A new “crop_settings.mat “ file should be created for each different roadway
location as different roadways use different camera angles and distances, and thus have
different areas that need to be cropped out. The cropping is used to remove the time stamp as
well as any other things that don’t interfere with the road and the objects traveling on it or
beside it. This speeds things up by using less memory and also removes some noise on the
image border (waving branches, moving clouds, etc).
3. Change settings: Each moving object has its own “segment” which is a range of frames where it
appears. There are several filter parameters that try to filter out noise, sometimes split
segments (if more than one object is present) and analyze the segment to compute features.
The default settings seem to work reasonably well, but more testing and new data should help
discover optimal settings. If changed, the settings will be saved as “settings.mat”.
4. Analyze a clip: This allows you to analyze one video clip to see how many segments there are
and what they contain. The results are saved as MAT files by the same name.
5. View features of a clip: Once a clip has been analyzed, the results are stored in “filename.mat”.
With this option you can go back and look at the results for any clip that has been analyzed.
6. Analyze a folder: This analyzes all the clips in a folder, not just one, and saves their results in
MAT files by the same name. For small test folders (5-10 videos) it does not take too long,
however, folders with hundreds of video clips will take a very long time. It only has to be done
once if you don’t change the settings. You must analyze a folder before you can use any of the
remaining options.
7. Find means: This should be applied to a “training” folder that has several good examples of
each type of object (vehicle, bike, pedestrian). You can reject examples that don’t look like
representative samples. The result is saved in “mean_features.mat”.
8. Nearest mean classification: This classifies the objects by using the mean feature matrix
(”mean_features.mat”) from the “training” folder to analyze objects in a larger folder. Of course
you can use the same folder to train and classify, but this is not a good test to see if it
extrapolates well to new unseen cases. The results are saved as “classify_summary.mat”.
9. Test classification accuracy: This allows to you to manually check to compute the percentage
accuracy for classifying the three types of objects. It does not take into account possible extra
objects when segments are split or perhaps missing objects because it was too dark. The results
are saved as “test_summary.mat”.
Folders and MAT files:
Showframesv5 is a standalone program that does not require calls to other programs. However,
showframesv5 does require a simple folder structure as well as certain MAT files to be close at hand,
optimally in the same directory. This folder structure is essentially a “training” folder and a “test” folder.
The “test” folder should obviously be the 24-hour video files at any one roadway. Videos from
different roadways should not be combined. This is because different camera distances and angles can
affect the object features, especially size and speed. The “training” folder should be a much smaller
folder with several good examples of each type of object desired to be classified. For more improved
results, more training examples of each object should be used. It is important there are several
examples of each object; otherwise the calculated means will not be representative. Showframesv5 uses
this “training” folder to create a 3x4 mean feature matrix (Area, Aspect Ratio, Speed, Density) for the 3
different objects (Vehicle, Pedestrian, Bicyclists). This mean feature matrix is saved as
”mean_features.mat” in the same directory and is used by the program later to classify objects.
There are several important MAT files that program menu choices will create besides
“mean_features.mat” above. The next such file is “settings.mat” which the program will always search
for upon startup, load it if the file is found, or use default settings if the file is not found. To create and
save a new “settings.mat” file if more optimal settings are discovered, the third menu option “Change
Settings” can be used. The next MAT file is “crop_settings.mat” which is required to analyze any videos.
This file is simply a 2x5 matrix of the (x,y) coordinates of the corners of the requested cropped image
(the first coordinate is repeated at the end to complete the square) and is created by the second menu
option “Crop image”.
The three remaining MAT files are “filename.mat”, “classify_summary.mat”, and
“test_summary.mat” which are created when you actually analyze a video, attempt to classify it, and
then test those classifications respectively. Many variables are saved in the “filename.mat” files and
one is created for each individual video clip. The main variables of interest are the 5 feature vectors
mentioned in the first paragraph (area, aspect ratio, density, (x,y) coordinates) which are eventually
manipulated and averaged over the segment length. To perform this averaging, the saved variable seg
is used which contains the beginning and end frame number for each segment for each object in the
video(s). The file “classify_summary.mat” also contains many variables including the averaged feature
matrix (Nx4, N being the number of objects), the mean feature matrix (3x4), and the number and
direction of vehicles, pedestrians, and bicyclists. Lastly, the file “test_summary.mat” saves the results of
the manual check on the classification accuracy. Variables saved in this file include the number and
direction of classified objects, the actual number of objects after checking, and the percentage correct
of each object.
As stated all of these MAT files are created by the program itself with different menu options. If
you do not have these MAT files already available you will have to run through menu options 2, 6, 7, 8,
and 9 to fully analyze a video folder from scratch.