Thrust Technical Review
March 2012
Distributed Sensor Exploitation 6.1
Rapid Deployment of Smart
Camera Networks
Dr. B.S. Manjunath
University of California, Santa Barbara
manj@ece.ucsb.edu
805-893-7112
Distributed Sensor Exploitation 6.1
Rapid Deployment of Smart Camera Networks
2012 Thrust Area Technical Review
UAV: Provides intelligence
from overhead
RESEARCH QUESTIONS:
• How can we geo-calibrate a stationary network using mobile sensors and
associated GPS and other information?
• What are the issues in discovering the geometric/spatial and temporal
connections and constraints within a camera network with overlapping and
non-overlapping views?
• What information needs to be computed at the smart sensor nodes for
activity analysis without distributing the raw video?
• When and how to fuse multi-modal information, e.g., mobile, airborne, and
fixed cameras, for robust object tracking and activity analysis?
MILITARY RELEVANCE/OPERATIONAL IMPACT:

Rapid deployment of a network of smart cameras and UAVs can help
monitor any potential area, especially in high risk and remote sites.
Rapidly Deployed Smart Camera:
Provides ground-based views and
distributed image analysis
TECHNICAL APPROACH:




Geo-calibration of fixed cameras using GPS and other sensors on
smart phones.
Auto-calibration and network topology discovery using visual analysis
Detecting and tracking objects in a camera network through fusion of
multiple views and geometry.
Analyzing events over non-overlapping camera views, with emphasis
on distributed tracking and fusion of tracked events spatially and
temporally.
NAVAL S&T FOCUS AREAs ADDRESSED:

Asymmetric & irregular warfare
SCHEDULE:
TASKS
FY10
FY11
Camera network setup
Auto-calibration and topology
Object detection & 2D tracking
Object tracking in 3D
Event & behavior detection
PERFORMER:
University of California at Santa Barbara
B. S. Manjunath, Professor
Demos on campus testbed
TTA: N/A
TECH TRANSITION PATH: DYNAMIC
WIKI/RESOURCE MANAGER (D&I)
FY12
Top Level POA&M (Summary Task)
2012 Thrust Area Technical Review
Principal Events & Activities
FY2010
FY2011
FY2012
O N D J F M A M J J A S
O N D J F M A M J J A S
O N D J F M A M J J A S
Camera Network Setup
Auto Calibration & Toplology
Geo-calibration
Object Detection and 2D tracking
Object Tracking in 3D
Event/Behavior analysis
Demonstration on DURIP Testbed
Field deployment
Peer Review presentation
Final demonstration
3
Project Work Breakdown Structure
2012 Thrust Area Technical Review
Detailed Technical Approach
2012 Thrust Area Technical Review
• Objective
– Development of a rapidly deployable network of smart camera
sensors to gather mission-specific information for surveillance and
human activity analysis
• Challenges
– Minimize human interaction
– Wireless communication constraints: limited bandwidth,
unpredictable latency, dropped packets, and time synchronization
– Few examples to train the system
5
Detailed Technical Approach
2012 Thrust Area Technical Review
Project Roadmap
Events in FY 10
 Setup of Outdoor Camera Network
 Quick Chessboard Calibration
 Preliminary network topology discovery
Events in FY11





Network topology discovery
Automatic calibration without patterns
Beyond 2D tracking
Continued multi-camera tracking with overlapping and non-overlapping views.
Preliminary distributed processing with existing work
Events in FY12
• Geo-calibration with mobile sensors
• Multi-camera multi-object tracking in a wide-area network
• Browsing and search in a distributed network
6
Detailed Technical Approach
2012 Thrust Area Technical Review
• Project Introduction
• Overview of the UCSB Campus Camera Network
– Infrastructure funded by a ONR DURIP award
– Implementation by students on this project
• Research Accomplishments
–
–
–
–
Distributed Geo-calibration
Distributed tracking and fusion
Topology discovery and network data summarization
Browsing and search in a distributed network
• Summary and Conclusions
7
Research Emphasis
2012 Thrust Area Technical Review
Geo-calibration
Topology
discovery
Wide-area
activity analysis
Network
summarization
Camera
placement
N+1 camera
calibration
Object
searching
Distributed
object tracking
UCSB Camera Network
2012 Thrust Area Technical Review
• Examined challenges through manual network setup
• Developing optimal camera placement algorithm based on
GPS trajectories
11
Project Technical Assesment
2012 Thrust Area Technical Review
Recent Progress





New Geo-calibration formulation with mobile sensors
Distributed Object Tracking on a Camera Network
Video Summarization in a Multi-camera setting
Distributed browsing and search
Collection of ground based and UAV (quadcopter) data for
activity analysis
12
Geo-calibration
2012 Thrust Area Technical Review
How do you calibrate cameras with non-overlapping views?
15
Geo-Calibration
2012 Thrust Area Technical Review
• Geo-Calibration is the process of finding the projective
geometry of a camera in a global coordinate system.
• Traditional, image-based methods are in a local coordinate
system and cannot be shared across different calibrations1
• Existing methods for integrating global coordinates involve
GPS tagged objects in the scene or aligning with a map2
1. Zhang, “A flexible new technique for camera calibration,” PAMI 2000
2. Kaminsky et al. “Alignment of 3D Point Clouds to Overhead Images.” W on Internet Vision 2009.
16
Geo-Calibration Method
2012 Thrust Area Technical Review
• Take pictures of the camera’s scene using smartphones
and collect location and orientation metadata
– Location sensor: GPS
– Orientation sensor: compass + accelerometer or gyroscope
• No objects need to be placed in the scene
• Data can be collected rapidly
17
Geo-Calibration Method
2012 Thrust Area Technical Review
• Geo-calibration: Given image(s) from a camera
1. Collect, using smartphone, calibration images with GPS and
orientation metadata
2. Calibrate cameras in a local coordinate system using images
3. Transform from local to global GPS coordinates
4. Refine position and orientation estimates
• Two approaches: centralized and distributed
18
Simulation Results
2012 Thrust Area Technical Review
• 2 quarter circle rows of cameras every 5 degrees
• Middle camera used as fixed camera
• Random set of points around origin
Estimated location from
Distributed Method
Real World Results
2012 Thrust Area Technical Review
• Smartphone: Data collected using HTC Evo smartphone
• Accurate Sensors: Data collected using CyberQuad UAV
sensors and camcorder
• Error Distance compared against GPS/Sensor from data
Dataset
Error Distance
Batch
Distributed
Smartphone
Location
2.83 m
4.21 m
Orientation
0.2399
0.0017
Location
1.80 m
2.001 m
Orientation
5.1490
0.0002
Accurate Sensors
Smartphone Results
2012 Thrust Area Technical Review
Smartphone Estimate
Estimation from distributed method
17
34
1
2
16
22
3
35
4
24
20
5
25
19
6
17
27
Estimation from
centralized method
H
e
ig
h
t(
m
e
te
r
s
)
15
14
16
9
10
1
.5
0
−
.
5
−
1
−
1
.
5
−
2
−
5
31
14
11
12
28
34
35
15
30
31
13
5
1
0
32
1
5
1
0
Y
[N
o
r
th
−
S
o
u
th
]
(
m
e
te
r
s
)
5
29
33
0
0
26
18
7
8
23
21
X
[E
a
s
t−
W
e
s
t]
(
m
e
te
r
s
)
1
5
23
Accurate Sensor Results
2012 Thrust Area Technical Review
UAV Estimate
Estimation from
distributed method
Estimation from
centralized method
6
5
8
9
4
6
5
7
1
0
H
e
ig
h
t
(
m
e
te
r
s
)
8
.7
1
9
4
1
2
9
3
8
0
−
.7
1
0
7
6
1
2
5
3
4
4
5
3
6
7
X
[E
a
s
t−
W
e
s
t]
(
m
e
te
r
s
)
2
8
9
1
1
0 0
Y
[N
o
r
th
−
S
o
u
th
]
(
m
e
te
r
s
)
Geo-calibration: MLE
2012 Thrust Area Technical Review
• New formulation as a maximum likelihood
• Maximize the orientation and position probability estimates
of the fixed camera given:
– Smartphone orientation position and location
– Pairwise relationship based on the image measurements
25
Consensus Framework
2012 Thrust Area Technical Review
• This estimate can be solved using a consensus algorithm
• In a consensus algorithm, each smartphone has a state of
the fixed camera’s orientation and location. The states are
iteratively updated to a consensus state.
29
Object Tracking and Search
2012 Thrust Area Technical Review
In a smart camera network setup,
• Distributed multi-camera object tracking
– Given a target, how do the network of cameras with
overlapping views robustly track the target with
collaboration despite limited network bandwidth?
• Efficient object searching in a large network
– How to search for objects of interest efficiently without
significant visual processing at search time?
33
Browsing and Searching
2012 Thrust Area Technical Review
Problem:
• A camera network deployed
over a large area
• No live streaming of any video
• Local camera nodes have
storage to archive video and
limited processing power for
simple video analysis
?
• How a human image
analyst at a distance
central node interacts with
the remote cameras?
34
Browsing and Searching
2012 Thrust Area Technical Review
• Envision the following application scenarios:
– A user instantiates the interaction with the network by
specifying regions on the image plane (cameras, time
intervals) of interest. E.g., “FIND object instances related to region A
FROM camera 1 OR region B FROM camera 4 between time 9:30am and
9:35am”
– With the results from the previous scenario, the user could then identify
one specific object of interest to initiate further searching for the same or
related objects. E.g., “FIND all objects related to the object instance at
region C FROM camera 1 at time 9:32:41.3am”.
35
Conventional Approaches
2012 Thrust Area Technical Review
• Try to provide human users with an high-level interface, such
as dynamic global scene visualization and aim to detect/track
all observed objects across the entire camera network
• To deal with appearance variations across views, much prior
work focused on finding the best matching criterion (Javed
ICCV’03,Javed CVPR’05, Farenzena CVPR’10, Zhen CVPR’11, Rios-Cabrera CVPR’11.)
• What if we cannot have reliable object detection and tracking?
Any possible strategy without pair-wise matching or global
trajectory finding?
36
Proposed System
2012 Thrust Area Technical Review
• Instead of trying to find global trajectories for every object
visible in the network, model camera observation directly with
a graph model
• Act as an intermediate agent between distributed cameras
and human image analysts and provide recommendations to
the user with a concise and representative set of video
snapshots captured by the camera network
• Help the image analysts to browse, search and identify
objects of interest by giving a canonical overview of the entire
set of visual observations in the network
37
Proposed System
2012 Thrust Area Technical Review
38
Real-time detection & tracking
2012 Thrust Area Technical Review
• Real time object detection with
background modeling and
tracking with mean-shift algorithm
• An observation record is
generated for each detected and
tracked object by the camera and
sent to the central node over the
network
39
Modeling camera observations
2012 Thrust Area Technical Review
• Given a user query, we need to find video frames
with following properties:
– Centrality, representative ones which are closely related to the query
and many other observations and hence considered important.
– Diversity, covering as many distinct groups as possible
• A graph G(V, W) to model relationship among camera
observations and perform unified graph ranking for different
queries
– Individual camera observations (i.e., frames with detected objects) form the
vertices set V
– Weight matrix W defines the strength of connectivity between camera
observations
– G(V, W) built at the central server incrementally as the new records are
received in real time from the cameras.
40
Spatial Temporal Topology
2012 Thrust Area Technical Review
• Divide image plane
into 8x6 blocks
• Model the time delay
for an object to
travel between any
two blocks across
cameras with a
Gaussian model with
known mean and
variance
41
Demonstration – Test bed
2012 Thrust Area Technical Review
•
•
•
11 camera nodes (Cisco
wireless-G WVC2300)
observing bike path
Each camera streams
video to a dedicated
computer to simulate a
smart camera node
Approx. 600 meters in
width/length
42
Browsing Demonstration
2012 Thrust Area Technical Review
•
•
Top 10 ranked frames (Decreasing order: left to right, top to down)
A total of 10 distinct objects satisfying the criterion. All of them have been
identified (labeled in yellow). The 8th ranked frame is a “false positive” (it
has not passed the queried regions within the specified time interval).
43
Mapping trajectories
2012 Thrust Area Technical Review
9:44:07 - 9:44:08
9:43:13 - 9:43:16
Missed
9:44:10 - 9:44:13
9:43:07 - 9:43:08
Searching Demonstration
2012 Thrust Area Technical Review
• Results when searching for object P6 (2nd -11th ranked
frame)
Forecasted Key Events
2012 Thrust Area Technical Review
UAV: Provides
intelligence from
overhead
Events in FY 10
 Setup of Outdoor Camera Network
 Quick Chessboard Calibration
 Preliminary network topology discovery
Events in FY11




Network topology discovery
Automatic calibration without patterns
Beyond 2D tracking
Continued multi-camera tracking with
overlapping and non-overlapping
views.
 Preliminary distributed processing with
existing work
Rapidly Deployed
Smart Camera:
Provides groundbased views and
distributed image
analysis
Soldier image courtesy of US Marines. Map image courtesy of Google Earth.
Events in FY12
• Geo-calibration with mobile sensors
• Multi-camera multi-object tracking in a
wide-area network
• Browsing and search in a distributed
network
46
Project Technical Risk Assessment
2012 Thrust Area Technical Review
No.
Risk Area and Description
Risk Rating
Risk Reduction Actions
Due Date
(Date
Completed)
1) Improve antenna and camera
placements
2) Robust algorithms
1) 12/2010
(L,M,H)
General
1
Challenging datasets due to wireless
communication constraints
L
2) 09/2012
Calibration and Topology
2
Lack of distinctive environmental
features for autocalibration
M
1) Weak scene calibration
2) Explore mobile cameras
1) 05/2011
2) 05/2011
3
Current topology approach may not
scale to large separations
M
1) Explore tracking-based approach
2) Explore mobile sensors
1) 11/2010
2) 6/2012
M
Explicit occlusion handling
09/2011
Object Detection/Tracking
4
Algorithm fails in severe occlusion
Event/Behavior Analysis
5
Complexity of the algorithms
M
Distribute the algorithm to multiple
computing nodes
03/2012
6
UAV data exploration: FAA and
university regulations
H
Collect data in remote areas (nonurban)
6/2012
47
Summary
2012 Thrust Area Technical Review
• On target with proposed schedule
•
•
•
•
•
Setup of campus-wide camera network
Quick chessboard calibration
Camera network topology discovery
Wide-area human mobility patterns
Single camera activity discovery
• Ongoing
• Geo-Calibration: automatic camera calibration and camera
placement
• Online learning and multi-camera fusion for distributed tracking
• Multi-camera activity discovery
48
Publications/Reports
2012 Thrust Area Technical Review
–
–
–
–
–
–
–
–
–
–
T. Kuo, Z. Ni, S. Sunderrajan, and B.S. Manjunath. Map Me: Camera Geo-calibration using Mobile Devices,
2012 (under preparation)
T. Kuo, S. Sunderrajan, B.S. Manjunath. “Trajectory Matching for Camera Calibration and Synchronization ”,
2012 (under preparation)
Jiejun Xu, Vishwakarma Singh, Ziyu Guan, B.S. Manjunath, “Unified Hypergraph for Image Ranking in a
Multimodal Context”, Intl. Conf. on Acoustics, Speech and Signal Processing(ICASSP), Kyoto, Japan, March
2012
S. Sunderrajan, Z. Ni, B. S. Manjunath, “Robust Object Tracking in a Distributed Camera Network,” Technical
Report, Nov 2011 (to be submitted).
Z. Ni, J.-J Xu, B. S. Manjunath, “Object Browsing and Searching in A Camera Network using Graph Models”,
Technical Report, Nov 2011.
C. De Leo, B.S. Manjunath. Multicamera Video Summarization from Optimal Reconstruction, Tenth International
Workshop on Visual Surveillance, Nov. 2010.
J. Xu, Z. Ni, C. De Leo, T. Kuo, B.S. Manjunath, “Spatial-temporal understanding of Urban Scenes through
Large Camera Network”, ACM Multimedia Workshop on Multimodal Pervasive Video Analysis, Oct. 2010.
Z. Ni, S. Sunderrajan, A. Rahimi, B. S. Manjunath. "Distributed Particle Filter Tracking With Online Multiple
Instance Learning In a Camera Sensor Network,” IEEE Int. Conf. on Image Processing, Sep. 2010.
Z. Ni, S. Sunderrajan, A. Rahimi, B.S. Manjunath. "Particle Filter Tracking With Online Multiple Instance
Learning", Int. Conf. Pattern Recognition, Aug. 2010.
T. Kuo, Z. Ni, C. De Leo, B.S. Manjunath. "Design and Implementation of a Wide Area, Large-Scale Camera
Network", The First IEEE Workshop on Camera Networks, Jun. 2010.
49
Students/Acknowledgement
2012 Thrust Area Technical Review
• Zefeng Ni (graduated Fall 2011)
– Distributed Tracking, activity analysis
• Thomas Kuo (expected to graduate in 2012)
– Calibration/Geo-calibration
• Carter De Leo
– Video summarization in a camera network, Topology discovery
• JieJun Xu (expected to graduate in 2012)
– Searching and browsing in large image/video datasets
• Thanks to ONR/Martin Kruger for the support and
encouragement!
50