Virtual Vision:
A Simulation Framework for
Camera Sensor Networks Research
Demetri Terzopoulos
UCLA
2007 PhD Thesis of Faisal Qureshi
University of Toronto
Publications:
•
•
•
•
•
2008 Proceedings of the IEEE
2008 11th Communications and Networking Simulation Symposium (CNS)
2008 ACM Symposium on Virtual Reality Software and Technology (VRST)
2007 First IEEE/ACM Intl. Conf. on Distributed Smart Cameras (ICDCS)
2007 IEEE/ACM Intl. Conf. on Distributed Computing in Sensor Systems
•
•
•
•
•
2007 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)
2006 ACM Intl. Wksp. on Distributed Smart Cameras (DSC)
2006 ACM Multimedia Systems Journal
2005 ACM Wksp. on Video Surveillance and Sensor Networks (VSSN)
2005 IEEE Intl. Wksp. on Visual Surveillance (VS-PETS)
(DCOSS)
1
Camera Networks and Smart Cameras
•
Visual surveillance is becoming ubiquitous
– London has roughly 4,000,000 cameras
•
Effective visual coverage of large spaces
require multi-camera systems
•
Operator monitoring is infeasible for large
networks
•
Need networks of smart cameras capable of autonomous operation
•
Smart cameras = visual sensor nodes
– Local on-board processing
– Communication with neighbors
Difficulty of Doing Large-Scale Visual Sensor Networks
Research
• Deploying large-scale camera networks in extensive public spaces
for research purposes:
– Hardware related technical challenges
– Privacy and legal issues
– Prohibitive cost
• Infeasible for most computer vision researchers
2
Virtual Camera Sensor Networks
Virtual Vision
Visually and behaviorally realistic simulators for designing and
evaluating machine vision systems
Camera Model:
Virtual
Camera
Environment Models:
geometry, texture,
illumination
Pedestrian Models:
appearance, movement,
behavior
Reality Emulator
Virtual Penn Station
pan, tilt, zoom,
camera jitter,
color response,
lens distortions,
etc.
High-Level Control
camera control,
assignment,
handover,
etc.
Virtual
Video
Machine Vision
tracking,
recognition,
etc.
3
Virtual Vision for Camera Networks Research
•
Emulates the characteristics of a physical vision system
•
Flexibility during system design and evolution
•
Readily available ground truth
•
Offline operation and testing
•
No legal impediments
•
No special hardware
•
Repeatability
•
Low cost
QuickTime™ and a
decompressor
are needed to see this picture.
Environmental Model of
Original Penn Station in NYC
[Shao & Terzopoulos, 2005]
Platforms
Concourses
Main
MainWaiting
WaitingRoom
Room
Concourses
Arcade
Arcade
Platforms
4
Architecture of the
Autonomous Pedestrian Model
Environment
& Interaction
Cognition
Behavior
Motion
Geometry
Pedestrian Simulation
5
Pedestrian Simulation
Human Activity in the Station
6
Virtual Vision
• Testbed for multi-camera sensor networks
Computer Vision Emulation Using Synthetic Video
• Pedestrian detection
– Background subtraction using a learned background model
QuickTime™ and a
decompressor
are needed to see this picture.
7
Low-Level Computer Vision
Emulation Using Synthetic Video
•
Pedestrian tracker
– Appearance based
Low-Level Computer Vision Emulation
Using Synthetic Video
• Appearance based pedestrian tracker
– [Swain & Ballard 91]
– Zoom invariant
Video frame
Signature
Backprojected image:
image
High intensity suggests
presence of target
8
Anatomy of a Camera Node
Decision
Decision logic
logic
Communication
Communication subsystem:
subsystem:
message
message passing
passing to
to
neighboring
neighboring nodes
nodes
Tracking
Task
Task “relevance”
“relevance” computation
computation
framework
framework
region of
interest
failure
reacquire
Vision
Vision routines:
routines: pedestrian
pedestrian tracking
tracking
Lost
Free
state management
timeout
y
Searching
timeout
left right
zoom
z
x
Image
Image driven
driven reactive
reactive behaviors:
behaviors:
fixation
fixation and
and zooming
zooming PD
PD controllers
controllers
up down
active pan-tilt-zoom camera
Virtual Vision:
A Tool for Visual Sensor Network Research
High-Level Camera Control
Visual Sensing
Visual Sensing
Synthetic Video
Real Video
Virtual Camera Network
Physical Camera Network
Synthetic World (Reality Emulator)
Real World
Virtual Vision
Real Vision
9
Camera Sensor Network #1
Active PTZ Camera Scheduling
Scheduling Active PTZ Cameras
• Number of Pedestrians
>
Number of active cameras
• Task active cameras to observe pedestrians in the scene
Active PTZ camera
Passive wide-FOV camera
10
Scheduling Active PTZ Cameras
• Scheduling problem
– Given n PTZ cameras, and m pedestrians, persistently observe
every pedestrian using one PTZ camera at a time
• Goal
– Observe as many pedestrians for as long as possible
Virtual Active Camera Scheduling Setup
• Passive wide-FOV cameras
– Calibrated
– Pedestrian localization through
triangulation
• Active PTZ cameras
– Un-calibrated
– Learn a coarse mapping between 3D
locations and internal pan-tilt settings
• Reliable pedestrian identification in
different cameras via appearance
based signatures
Calibrated passive cameras
at the four corners of
the waiting room in
the virtual train station
11
Distinguishing Features of Our Active Camera
Scheduling Strategy
•
Previous attempts
–
Costello et al., 2004
Costello et al., 2006
Lim et al., 2006
Huang and Trivedi, 2003
•
Exit times are not known a priori
•
Recording duration is not known a priori
•
Allow multiple observations of the same pedestrian
•
Multiple pan/tilt/zoom cameras
•
Tracking failures due to occlusions
Active Camera Scheduling Strategy
• Camera assignment via weighted round robin
• First come, first served tie breaking
• Multiple observation
• Preemption
Close-up snapshots captured
by active PTZ cameras
12
Active Camera Scheduling Results
Preemption (P)
No preemption (NP)
Single observation (SO)
Multiple observations (MO)
Single-class model (SC)
Multi-class model (MC)
Multiple observations, multi-class,
preemption scheduler outperforms other
variants
Up to 4 Cameras; 10, 20 Pedestrians
Camera Sensor Network #2
Active PTZ Camera Assignment and Grouping
13
Smart Camera Network
•
Cameras self-organize to observe pedestrians during their presence in the
region
– Smart, PTZ active cameras
– Un-calibrated
– Ad hoc deployment
– Camera additions & removal
– Handle camera failures
– Deal with imperfect communication
•
Previous work
– Simmons et al., 2006
Mallet, 2006
Park et al., 2006
Kulkarni et al., 2005
Javed et al., 2000;
Devarajan and Radke, 2006
Vision and Goal
• Ad hoc deployment
14
Vision and Goal
• Cameras work towards common sensing goals
Task assignment
Vision and Goal
• Cameras work towards common sensing goals
Group formation
15
Vision and Goal
• Cameras work towards common sensing goals
Group evolution
Vision and Goal
• Cameras work towards multiple common sensing goals
Group 1
Group 2
16
Vision and Goal
• Cameras work towards multiple common sensing goals
Camera failure
Vision and Goal
• Cameras work towards multiple common sensing goals
Task assignment
17
Vision and Goal
• Cameras work towards multiple common sensing goals
Conflict resolution
Camera Grouping and Reassignment
18
Active PTZ Camera Assignment and
Grouping
• Camera selection, grouping, and handoff via an auction model
– Announcement/Bidding/Selection
– ContractNet
• Smith, 1983
• Conflict resolution within a Constraint Satisfaction Problem
framework
– Partially distributed
• A Camera can only perform a single task at any given time
Camera Grouping: Announcement
•
Start with a single camera that is tasked to observe a person
Leader
19
Camera Grouping: Announcement
•
Seeks out other cameras in the vicinity to form a group to help
it with the observation task
Leader
Camera Grouping: Bidding
•
One or more cameras that receive the task announcement respond
with their relevance values
Never Received
the Message
Leader
Response: high
relevance
Response:
low relevance
Did not
respond
20
Camera Grouping: Bidding
•
One or more cameras that receive the task announcement respond
with their relevance values
•
Relevance encodes how successful a camera will be at an
observation task
Never Received
the Message
Leader
Response: high
relevance
Response:
low relevance
Did not
respond
Camera Grouping: Selection
•
After the leader gets relevance messages from neighboring
cameras, it selects suitable cameras to join the group
Join
Group
Leader
21
Conflict Detection
•
A conflict is detected when multiple tasks require the same
camera node to proceed successfully
Conflict Detection
•
A Red group member receives a recruit query from Green group
22
Conflict Detection
•
A Red group member receives a recruit query from Green group
Conflict Detection
•
A Red group member receives a recruit query from Green group
Conflict!
23
Conflict Detection
•
Nodes belonging to both groups send information to one of the
leaders
•
Leader selection
Centralization
Conflict Detection
•
The resulting node (camera) assignment is sent to the individual
nodes
Solution
24
Conflict Detection
•
The resulting node (camera) assignment is sent to the individual
nodes
Persistent Observation:
Virtual Active PTZ Camera Assignment and Grouping
25
Network Model
•
It does not require camera calibration or network calibration
– Can take advantage of calibration information, if available
– Ad hoc deployment
•
Camera grouping is a strictly local negotiation
– Typically camera groups are spatially local arrangements
•
Camera groups are dynamic arrangements
•
Camera handoffs occur naturally during negotiations
•
It can gracefully handle node and message failures
Network Model
• Camera nodes can be added or removed during the lifetime of a
task
• Even assuming perfect sensing, the proposed model can still fail if
– a significant number of messages are lost
– catastrophic node failure
– group evolution can’t keep up with a fast-changing observation task
• Scalability
– Small group sizes
– Conflict resolution is viable as long as the number of relevant sensors
for each task remains low ( < 10 )
– Optimal sensor assignment
26
Future Work
• Camera sensor network
– Consistent labeling of pedestrians
– Who, What, Where, When?
– Cognitive modeling
• Physical sensor networks
– Environment monitoring, urban sensing, intelligent environments, …
• Bigger, better reality emulators
– High-fidelity synthetic video
– An entire city …
Virtual LA
Urban Simulation Lab
Architecture Dept
UCLA
27
Virtual LA
• Building level of detail
Virtual LA
• Interior spaces
28
Strategy Looking Forward
• Collaborative engagement of the CV and CG communities
– Open source framework to develop and test state-of-the-art
vision algorithms and systems
– Modular www-accessible environment
• e.g., SRI’s Open Agent Architecture (OAA)
– “Any-world” emulator
– More realism (“The Matrix”)
Thank You !
29