Watchdog
Confident Event Detection in Heterogeneous Sensor Networks
Matthew Keally1, Gang Zhou1, Guoliang Xing2
1College of William and Mary, 2Michigan State University
Overview

Problem Statement

Challenges

Related Work

Contributions

Design

Evaluation
2
Confident Event Detection


Many applications for event detection have stringent accuracy
requirements and demand long system lifetimes

Vehicular traffic monitoring

Falls in elderly patients

Military/intrusion detection
Perform confident event detection


Meet user-defined false positive and false negative rates in the
presence of in-situ sensing reality
Reduce energy usage to extend system lifetime
3
Challenges of Confident Event Detection

How to cluster the right sensors to meet user accuracy
requirements?




Learn the detection capabilities of individual sensors and clusters
Use part of the detection capability to meet user requirements and
save energy
How to efficiently perform collaboration between heterogeneous
sensors to meet user requirements?

Difficult for modality-specific models and data fusion

Need a generic solution
How to adapt detection capability to runtime observations?

Easier observations and harder observations need different
detection capabilities
4
Related Work



Sensing Coverage

Do not address user accuracy requirements

Do not explore detection capability of deployment
Modality-specific Sensing Models and Data Fusion

User requirements not met in reality

Difficult to perform heterogeneous sensor fusion

Do not cluster the right sensors to meet user requirements
Machine Learning

Do not address user accuracy requirements

Do not adapt sensing capability to runtime observations
5
Motivation: Related Work Shortfalls

Vehicle Detection: sensing irregularity

Same distance, different accuracies

Accuracy can increase with distance

Sensing Coverage may overdetect or underdetect events

Theoretical sensing models assume all sensors are identical
6
Motivation: Related Work Shortfalls

Different clusters (C1,C2,C3) have the same accuracy, 100%,
better than individual sensors

Difficult to capture for existing works: Due to lack of knowledge of
detection capability of different sensors and clusters
7
Watchdog Contributions



A confident and energy efficient event detection framework

Choose the right sensors to meet user requirements

Generic framework that provides heterogeneous sensor fusion
Adapt detection capability to runtime observations

Easy observations: low-power sentinel sensors

Hard observations: higher-power reinforcement sensors
Performance evaluation: two scenarios

Monitor traffic entering and leaving computer science building

Vehicle detection using Wisconsin trace data

Compare against sensing coverage and signal attenuation model
8
Watchdog Design Overview
Node
Local Aggregation
Cluster Generation
Sentinel & Rein. Selection
Runtime Event Detection
Aggregator
Sensor
Cluster
Generation
Local
Aggregation
Request
Reinforcement
Data
Sentinel and Reinforcement
Selection
Observations
Runtime Event
Detection

Efficient heterogeneous collaboration

Explore detection capability of a deployment

Cluster the right sensors to meet user requirements

Adapt detection capability to runtime observations
Training
Results
9
Cluster Generation


Local Aggregation
Cluster Generation
Sentinel & Rein. Selection
Runtime Event Detection
Goal: determine detection capability of

Individual sensors and sensor clusters

A specific deployment
Method

Randomly generate up to M clusters for each cluster size

For each generated cluster

Step 1: Train a Hidden Markov Model for the cluster
HMM is good for heterogeneous sensor fusion
 HMM captures time and space correlation of sensor data
Step 2: Determine cluster FP/FN based on the HMM decision and
ground truth at each time interval


10
Step 2: Determine cluster FP/FN based
on the HMM decision and ground truth


Local Aggregation
Cluster Generation
Sentinel & Rein. Selection
Runtime Event Detection
At each aggregation interval:

Determine event detection decision with trained HMM

Compare cluster detection decision with ground truth
Get the cluster FP/FN (accuracy)

Determine FP/FN for each possible event probability
11
Sentinel and Reinforcement
Selection

Local Aggregation
Cluster Generation
Sentinel & Rein. Selection
Runtime Event Detection
Choose sentinel cluster: low detection capability
– Meets user's FN requirement
– Makes easy detection decisions

Choose reinforcement cluster: higher detection capability
– Meets both FP and FN requirements
– Used to make more difficult detection decisions

All other sensors go to sleep
12
Runtime Event Detection

Local Aggregation
Cluster Generation
Sentinel & Rein. Selection
Runtime Event Detection
Goal: adapt detection capability to runtime observations
– Easier observations and harder observations need different
detection capabilities

Method:
– Sentinels and reinforcements form local observations at each
aggregation interval
– Sentinels report non-default observations to the aggregator to
make detection decisions
– Reinforcements requested when sentinel event probability false
positive rate exceeds user requirements
– Reinforcements return non-default observation data and
aggregator makes a confident decision
13
Runtime Event Detection
User requirements: u.FN = u.FP = 0.05
Acoustic
56
Seismic
Aggregator
52
Sentinels
0
1
2
3
Reinforcements
54
60
Time interval
Local Aggregation
Cluster Generation
Sentinel & Rein. Selection
Runtime Event Detection
4
5
t=1: No Event, s.FN = .01 < u.FN
t=2: Event, s.FP = .02 < u.FP
t=3: No Event, s.FN = .01 < u.FN
t=4 :Undecided, s.FP = .45 > u.FP
t=4 :Event, r.FP = 0.3 < u.FP
t=5: No Event, s.FP = 0.2 < u.FP
14
Evaluation

App1: Wisconsin SensIT trace data

– Vehicle detection at a fixed location
– Distance-based signal attenuation
– 75 nodes with acoustic, seismic, and
infrared sensors
– 100ms aggregation interval

App2: Computer Science Building
Traffic Monitor
– Five IRIS motes mounted on main
entrance door
– MTS 310: 2-axis accelerometer, 2-axis
magnetometer, acoustic, and light
sensors
Compare with a modality-specific
sensing model
– Data fusion for event decisions

Compare with V-SAM, a state of the
art protocol for handling sensing
irregularity
– Measure data similarity between
sensors
– Keep awake only sensors with
dissimilar readings
– Define event as when someone opens
the door and walks through
– 4s aggregation interval
15
Exploring Detection Capability &
Meeting Requirements



Only a limited & discrete number of FP/FN rates supported by the
deployment
For a specific FP/FN rate, a large number of clusters may be available
During runtime detection, Watchdog meets FP/FN explored during
training
16
Compare with V-SAM: Accuracy

V-SAM with k-coverage and similarity coverage

Watchdog outperforms all with near perfect accuracy
17
Compare with Modality-Specific
Sensing Model: Accuracy

Vehicle detection with acoustic sensors
– Select clusters with two different ranges to target location: near (<25m) and far
(>40m)

Watchdog always meets user requirements

Modality-specific model ignores in-situ sensing reality
18
Compare with Modality-Specific
Sensing Model: Energy

Watchdog clusters the right sensors to meet user requirements
– Meets requirements with reduced energy

Watchdog adapts its capability to runtime observations to save energy

Modality-specific sensing model uses all sensors in the cluster
19
Adapting Detection Capability to
Runtime Observations

Experimental setting
– Vehicle trace data and sensors from <25m
– User requires 0% false positives and false negatives


Watchdog clusters the right sensors to meet user requirements
Sentinel FP/FN
(%)
Reinforcement
FP/FN (%)
Reinforcement
Requests (%)
9.5/0.0
0.0/0.0
21.0
Neither V-SAM nor the modality-specific sensing model adapts
detection capability to runtime observations
20
Conclusions and Future Work

Existing works do not provide event detection with confidence,
we need to
– Cluster the right sensors to meet user requirements
– Provide a generic approach for heterogeneous deployments
– Adapt detection capability to runtime observations

Watchdog: a confident event detection framework
– Meets user accuracy requirements
– Exceeds accuracy of existing approaches
– Uses knowledge of detection capability to save energy

Future Work
– Online and distributed detection
21
Thanks to NSF grants ECCS-0901437 and
CNS-0916994
22
Compare with V-SAM: Training Length

Watchdog achieves maximum performance with a short training

V-SAM requires little training, but is less accurate
23
Local Aggregation
Local Aggregation
Cluster Generation
Sentinel & Rein. Selection
Runtime Event Detection

Allows for heterogeneous sensor fusion

Raw data is combined to form a single observation
–

Use a common aggregation technique
Discrete, finite number of possible observations
–
Same number for each sensor and modality
–
Allow for comparison between sensors of all modalities
–
We use two discrete observations
24
Event Probability Discussion

Local Aggregation
Cluster Generation
Sentinel & Rein. Selection
Runtime Event Detection
Differentiate the accuracy between different event probabilities
– Some observations are more reliable than others
– Probabilities near 0.5 are more inaccurate

Determine FP and FN for each of p probability ranges (p=10)
– Probability between .1 and .2 has zero false negatives
– Probability between .9 and 1.0 has 6% false positive rate
– Ranges with no events have 100% false positive or false negative rates
25