Active Inference for Retrieval in
Camera Networks
Daozheng Chen1, Mustafa Bilgic2, Lise Getoor1, David
Jacobs1, Lilyana Mihalkova1, Tom Yeh1
1 Department
of Computer Science, University of Maryland, College Park
2 Department of Computer Science, Illinois Institute of Technology
Problem
• Search camera network videos to retrieve frames
containing specified individuals.
query
Time
query
Time
Related Work
• Person re-identification [Wang et al. ’07]
• Graphical Models for camera networks [Loy et
al. ’09]
• Tracking over camera networks [Song et al. ’07]
• Active Learning [Settles ’09]
Our Contributions
• Map video frames in a camera network onto a
graphical model and use a collective
classification algorithm to predict frame states
and perform frame retrieval.
• Apply active inference to direct human attention
to portions of the videos which are most likely to
have the biggest performance improvement.
Graphical
Structures
Collective
Classification
Active
Inference
Active
Inference
Outline
•
•
•
•
•
Graphical model construction
Iterative classification algorithm
Active inference
Experiment
Conclusion
Graphical Model Construction
• Temporal neighbors (TN)
• Frames from the previous and next k time steps
within the same camera.
• Positively correlated spatial neighbors (PSN)
• Correlation of the labels of two camera is greater
than some threshold.
• Negatively correlated spatial neighbors (NSN)
• Correlation of the labels of two camera is less than
some threshold.
Temporal
Neighbors
(k = 1)
Positively
correlated
spatial
neighbors
Negatively
correlated
spatial
neighbors
Graphical
Structures
Outline
•
•
•
•
•
Graphical model construction
Iterative Classification Algorithm
Active Inference
Experiment
Conclusion
The Iterative Classification Algorithm
(ICA)
• Local Models (LM).
• The label of a frame is only dependent on its features.
• Relational Models (RM).
• The label of a frame is dependent on its features and its
neighbors’ current labels
• First apply the local model for initialization, and then use
the relational model iteratively until predicted labels
converge. [Sen et al. ’08]
The Iterative Classification Algorithm
(ICA)
• Local Models (LM).
• Logistic regression as the classifier.
• Cosine similarity based on signatures using bag-of-feature model as features
Fq = [fq1,fq2,…,fqn]
F = [f1,f2,…,fn]
COS(Fq,F)
The Iterative Classification Algorithm
(ICA)
• Relational Models (RM).
• Logistic regression as the classifier.
• Use aggregation function to construct a feature vector encoding neighbors’
information.
F = [f21,f22,…,f2n]
The Iterative Classification Algorithm
(ICA)
• Relational Models (RM).
• Logistic regression as the classifier.
• Use aggregation function to construct a feature vector encoding neighbors’
information.
F = [f21,f22,…,f2n]
FTN = [fTN1,fTN2]
The Iterative Classification Algorithm
(ICA)
• Relational Models (RM).
• Logistic regression as the classifier.
• Use aggregation function to construct a feature vector encoding neighbors’
information.
F = [f21,f22,…,f2n]
FTN = [fTN1,fTN2]
FPSN = [fPSN1,fPSN2]
The Iterative Classification Algorithm
(ICA)
• Relational Models (RM).
• Logistic regression as the classifier.
• Use aggregation function to construct a feature vector encoding neighbors’
information.
F = [f21,f22,…,f2n]
FTN = [fTN1,fTN2]
FPSN = [fPSN1,fPSN2]
FNSN = [fNSN1,fNSN2]
The Iterative Classification Algorithm
(ICA)
• Relational Models (RM).
• Logistic regression as the classifier.
• Use aggregation function to construct a feature vector encoding neighbors’
information.
F = [f21,f22,…,f2n]
FTN = [fTN1,fTN2]
FPSN = [fPSN1,fPSN2]
FNSN = [fNSN1,fNSN2]
FRM
Outline
•
•
•
•
•
Graphical model construction
Iterative Classification Algorithm
Active Inference
Experiment
Conclusion
Active Inference
• The retrieval algorithm can request the correct labels
for some frames at inference time. [Rattigan et al. ’07]
• Subsequent inference using ICA is based on these
corrected labels.
• Common methods for selecting frames to label:
• Random (RND).
• Uniform (UNI).
• Most certain to be relevant (MR).
• Most uncertain (UNC)
• Reflect and Correct. [Bilgic and Getoor. ’09]
Reflect and Correct (RAC)
[Bilgic and Getoor. ’TKDD09]
Adaptive RAC (MLI)
We adapt RAC with the following the 10 features to train a classifier to determine
whether a frame, f, is misclassified – Most Likely Incorrect (MLI)
Entropy(f)
Entropy of the probability estimate of f based on RM
Entropy(TN(f))
Average entropy of the probability estimate of TN of f based on RM
Entropy(PSN(f))
Average entropy of the probability estimate PSN of f based on RM
Entropy(NSN(f))
Average entropy of the probability estimate of NSN of f based on RM
KL(f)
KL divergence between probability estimates of LM and RM of f
KL(TN(f))
KL divergence between probability estimates of LM and RM of TN of f
KL(PSN(f))
KL divergence between probability estimates of LM and RM of PSN of f
KL(NSN(f))
KL divergence between probability estimates of LM and RM of NSN of f
IsRelevant(f)
Whether is f is predicted to be relevant by RM
%Agree(f)
the percentage of TN and PSN whose predicted or corrected labels agree
with the predicted label of f.
Outline
•
•
•
•
•
Graphical model construction
Iterative Classification Algorithm
Active Inference
Experimental Evaluation
Conclusion
Dataset
[Ding et al. ’10]
Queries
Region of Interests
• Background subtraction to determine region of interest in the frame.
• Densely sample key points in the regions
• Use color histogram in RGB space to describe the region spanned by a key point
• Quantized the descriptor according to learned 500 code words.
• Produce a single signature for a video frame.
Spatial Topology
Methods for Comparison
• Active inference based on LM using RND, UNI, MR,
UNC, MLI
• Active inference based on RM using RND, UNI, MR,
UNC, MLI
• Average accuracy and Average 11-average precision
as measurement
Results
• UNC-LM has the best performance when results are based on LM.
• RM always perform better than LM does under the same sampling method.
• UNC-RM and MLI always perform better.
• MLI never perform worse than MLI does.
Outline
•
•
•
•
•
Graphical model construction
Iterative Classification Algorithm
Active Inference
Experiment
Conclusion
Conclusion
• Using a graphical model provides significant
performance improvements in frame retrieval.
• A simple method that captures the frame
uncertainty has an advantage over other baseline
methods.
• Our adaptation of RAC has overall better
performance.
Questions?