Automated video surveillance: challenges and solutions.
ACE Surveillance (Annotated Critical Evidence) case study.
Dmitry Gorodnichy and Tony Mungham
Laboratory & Scientific Services Directorate
Canada Border Services Agency www.videorecognition.com/ACE

2
 Problems with status-quo Video Surveillance
 Real-time and archival problems
 Operational considerations


Next generation solution - Video Analytics based


“Motion detection” myth and problem
“Object detection” as example of real intelligence
ACE Surveillance – first fully-functional objectdetection-based prototype
 Year long tests with different levels of complexity
 What that means for future of Video Surveillance
 Conclusions

 In the context of enhancing security, Video
Technology (VT) is one of the most demanded technologies of the 21st century
 It is publicly acceptable
 It provides rich in content data
3
 Multi-million funding in Canada and worldwide:
 CBSA Port Runner project invested 10s of Millions in
CCTV upgrades
 Transport Canada opens $35M of funding towards procurement of CCTV

4
 CBSA is a major user of CCTV systems at POEs
 Most major CCTV installations start to leverage VT
 Current task: to lead applied R&D to push VT to help
CBSA apply S&T innovative approaches to border management:
 Event detection and notification to provide effective response to events
 Traffic trends analysis to assist with border management
 Video storage management to manage the cost of storage and meet obligations under the privacy act
 Data integration/fusion of contextualised video information

5
Modes of operation:
1.
2.
Active - personnel watch video at all times
Passive - in conjunction with other duties
3.
Archival - for post-event analysis
Current systems and protocols are not efficient in either mode!
Problem in real-time modes: an event may easily pass unnoticed .

 due to false or simultaneous alarms, lack of time needed to rewind and analyse all video streams.

6
Due to temporal nature of data:
1.
•
Storage space consumption problem
Typical assignment:
2-16 cameras, 7 or 30 days of recording, 2-10 Mb / min.
 1.5 GB per day per camera / 20 - 700 GB total !
2.
•
Data management and retrieval problem
London bombing video backtracking experience:
“Manual browsing of millions of hours of digitized video from thousands of cameras proved impossible within timesensed period”
[by the Scotland Yard trying to back-track the suspects]

7
 Lots of CCTV infrastructure: Many local initiatives, not coordinated
 Most video technology decisions are influenced by vendors - short-term solutions
 Over 30 different video systems within the same dept. (at
RCMP)
 A national program with proper benchmark-based planning and evaluation of VT is required
 Leveraging advances recently made in S&T
 Technical standards for capturing /saving video data.
 Policies in when, where and how VT should be used.

8
Video Analytics (Video Recognition)
21 st century
Wireless, Network Connected (IP)
Digital
Analog
First video recording
20 th century

9
also identified as:
 Video Recognition,
 Intelligent Video ,
 Smart Video / Smart Camera
 Video Analysis & Content Extraction
 Perceptual Vision
 is not much about capturing better data (better lenses, grabbers, coders, transmitters)
 but about understanding captured data (better theory)

10
Transport Canada CCTV Reference Manual for
Security Application .
Australian Government National code of practice for
CCTV applications in urban transport
USA Government :recommended security Guidelines for Airport Planning, Design and Construction.
…. refer to “Motion-based” capture as Intelligent
Surveillance Technology, and make their recommendations based on thereon .



Term “Motion-based” is coined to make people believe that video recognition is happening, which is not!
It’s actually illumination-change-based, as it uses simple point brightness comparison :
11
 Which often happens not because of motion!
 Changing light / weather (esp. in 24/7 monitoring)
 Against sun/light, out of focus, blurred, thru glass
 Reflections, diffraction, optical interferences
 Image transmission, compression losses

… but few can do it, since necessary advances in video recognition theory became possible only recently (>2002).
In 2002 National Research Council of Canada (NRC) starts developing Video Recognition Systems to leverage its scientific Video Recognition expertise for the industry.
In 2005, it develops ACE Surveillance: an object-detection-based A utomated surveillan CE prototype capable of automatically extracting
Annotated Critical Evidence from live video.
12
NRC becomes also the organizer of the first Canadian academic workshops dedicated to Video Processing for Security (since 2004)



A Windows software that performs real-time video analytics by integrating best object detection and tracking algorithms.
Replaces video clips with annotated still images:
 Compresses 1 Gb of video into 2 Mb of easy to browse and analyze still images
ACE Surveillance output:
A 7-hour activity from day to night (17:00 - 24:00) is summarized into 2 minutes
(600Kb) of Annotated Critical
Evidence snapshots.
Note illumination changes! - Watch tree shadows and sun light.
13

14


Works with ordinary USB cameras or CCTV cameras with USB video converters.
Adds on top of existing infrastructure using an ordinary desktop computer.
ALARM!
..
.
Video clips (Tb)
C.E.S. (Gb)
Last captured
CES
Real-time mode of operation
ACE Capture
Archival mode of operation
ACE Browser

15

16
(Courtesy: NRC-IIT Video Recognition Systems project)
1. Many captured snapshots are useless: either noise or redundant
2. Without visual annotation, motion information is lost.
3. Hourly distribution of snapshots is not useful

17
(Courtesy: NRC Video Recognition Systems project)
1. Each captured shot is useful.
2. Object location and velocity shown augmentent.
3. Hourly distribution of shots is indicative of what happened in each hour, provides good summarization of activities over long period of time.

18
Tested in different levels of complexity:
 lighting conditions,

 object motion patterns, camera location
 environmental constraints. most difficult - outdoors in unconstrained environments with little or no object motion consistency (as around a private house in a regular neighbourhood). most easy - in controlled indoor environment where minimal direct sunlight is present and where all objects are of approximately the same size and exhibit similar motion pattern (as at access gate inside the business building).

Outdoor, wireless, eye-level Outdoor, webcam, overview Indoor with sunlight, CCTV Indoor w/o sunlight, CCTV
VT within CBSA 19

Back Door Entry Delivery Entry
20
More than usual

21
In real-time mode: alarm sounds & last captured evidence
(time-stamped) is shown.
In archival mode: “Zoom on the evidence” browsing of captured evidences – zoom on a day, on hour, then on event - point and click (for high res as needed)
Made Commissioners much more aware of activities.

22
 Affordable automated (intelligent) video surveillance
(AVS) is possible !


To replace traditional DVR
OR to supplement them: DVR for 1 month + AVS for 1 year
However:
 Requires extra training from security officers.


Requires new protocols to handle automatically extracted evidence.
 - From forensic prospective, data that are not original and have been processed by a computer can not be considered as evidence.
Requires new privacy policies.
 - Surveillance data are normally not kept for a long period of time
(<1 month), due to their size. AVS allows to store on local machine many months (even years) of evidence data.

 ACE Surveillance (which is developed by a research lab) provides a reference standard against which can be measured solutions coming from industry.
 It deals with common misconceptions related to deploying intelligent video surveillance systems (IVS):


“motion detection” myth vs real object detection and tracking.
The “one-fit-all” myth. - Extra video analytics expertise is required to set and operate IVS.
 better video data (better resolution or compression) do not imply better video intelligence. - ACE Surveillance is shown to work with regular TV quality data (320 by 240 pixels).
 However better quality of video image is needed for forensic purposes as evidence
 Due to closing of the project by NRC, CBSA takes lead on it.
23