TMSG
NuActiv: Recognizing Unseen New
Activities Using Semantic AttributeBased Learning
實驗室: 先進網路技術與服務實驗室
報告者: 黃福銘 (Angus F.M. Huang)
2013.07.10
Publication
• MobiSys 2013
• Session 8: Behavior and Activity Recognition
• NuActiv: Recognizing Unseen New Activities Using Semantic AttributeBased Learning
• SocioPhone: Everyday Face-To-Face Interaction Monitoring Platform
Using Multi-Phone Sensor Fusion
• MoodScope: Building a Mood Sensor from Smartphone Usage Patterns
• Auditeur: A Mobile-Cloud Service Platform for Acoustic Event Detection
on Smartphones
Angus F.M. Huang
2
Authors
• Heng-Tze Cheng
• Feng-Tso Sun
• Martin Griss
– Electrical and Computer Engineering
– Carnegie Mellon University
• Paul Davis
• Jianguo Li
• Di You
– Applied Research Center
– Motorola Mobility
Angus F.M. Huang
3
• To recognize a new human activity
– No such training example
• Essential for
– User-centric & Context-aware applications
• Most methods
– Only recognize known activities
• NuActiv
– Semantic attributes for representation
– Two-layer zero-shot learning algorithm
– User feedback for reinforcing accuracy
Abstract
Angus F.M. Huang
• Evaluation
– 10-exercise-activity dataset collected
– Public dataset of 34 daily life activities
– Achieved up to 79% accuracy
4
Outline
• INTRODUCTION
• EVALUATION
– System Implementation
• NUACTIV SYSTEM OVERVIEW
– Scenarios and Design
Considerations
– System Architecture of NuActiv
• SYSTEM DESIGN AND
ALGORITHMS
– Feature Extraction
•
– Semantic Attribute-Based Activity
Recognition
– Active Learning: Reinforcing
Activity Recognition Using Minimal
User Feedback
Angus F.M. Huang
–
–
–
–
–
Datasets
Evaluation Methodology
Case Study I: Exercise Activities
Case Study II: Daily-Life Activities
Active Learning Experiments
CONCLUSION
5
Introduction
• Diversified people daily activities
– 462 different activities
– American Time Use Survey
• Two steps of existing approaches
– (1) Collect and label
– (2) Classify
Angus F.M. Huang
6
Two Research Questions
• Q1. How to recognize a previously unseen new
activity class when we have no training data from
users?
• Q2. If we have the opportunity to ask users for
labeled training data, how to reinforce the
recognition accuracy using minimal help from users?
Angus F.M. Huang
7
Two Inspirations
• Many human activities and context types share the
same underlying semantic attributes
– Sitting vs...
• Having lunch in the cafeteria
• Working at a desk
• The limit of supervised learning can be overcome by
incorporating human knowledge
– Office working vs...
• Motion-related attributes: Sitting, HandsOnTable,
• Sound-related attributes: PrinterSound, KeyboardSound,
Conversations
Angus F.M. Huang
8
Challenges – Q1
• Zero-shot learning
– To learn a classifier that can recognize new classes that have never
appeared in the training dataset
• Which attributes are useful
• From Static image data to Sequential sensor data
• Decompose high-level activities into combinations of
semantic attributes
– Human readable term
– Two-layer attribute-based learning algorithm
Angus F.M. Huang
9
Challenges – Q2
• Accuracy reinforcement by user feedback
• Outlier-aware active learning algorithm
• Hybrid stream/pool-based sampling scheme
Angus F.M. Huang
10
Activity
Recognition
Angus F.M. Huang
11
NuActiv System Overview
• Two scenarios of activity domain
• Daily life activities
– ex. ReadingAtHome & Driving
– → ReadingOnTrain
– It is also arguably of much larger variation because different people do
the same things differently (time, situation,…)
• Exercise activities
– Various same underlying attributes
Angus F.M. Huang
12
System Architecture of NuActiv
Angus F.M. Huang
13
System Design and Algorithms
• Feature Extraction
• Semantic Attribute-Based Activity Recognition
• Active Learning: Reinforcing Activity Recognition
using Minimal User Feedback
Angus F.M. Huang
14
Feature Extraction
• The mean and standard deviation of sensor data in
dimension x, y, and z
• Pairwise correlation between each pair of
dimensions x, y, and z
• Local slope of sensor data in dimension x, y, and z
in using 1st-order linear regression
• Zero-crossing rate in dimension x, y, and z
Angus F.M. Huang
15
Examples of features extracted
from acceleration data for each
exercise activity
Angus F.M. Huang
16
Semantic Attribute-Based
Activity Recognition
• Activity class space, y = {y1,y2,…,yk}
• Feature space, x = [X1,X2,…,Xd]
• Classifier function f: x→y
• y = {{y1,y2,…,ys},{ys+1,…ys+u}} = yS U yU
– yS, seen classes
– yU, unseen classes
• Problem: How to recognize an unseen class y ∈ yU ?
• Semantic attribute space, a = [A1,A2,…,Am]
Angus F.M. Huang
17
Graphical representation of
semantic attribute-based
activity recognition
Angus F.M. Huang
18
Activity-Attribute Matrix
• Encode the human knowledge
• M x N matrix; M activities, N attributes
• Common-sense knowledge, domain knowledge, web text
mining, crowdsourcing platforms…
Angus F.M. Huang
19
Attribute Detection
• To train a set of attribute detectors so that we are able to infer
the presence/absence of an attribute from the sensor data
features
• Training data
– what we need is one set of positive samples and another
set of negative samples
• Classifier
– Support Vector Machine (SVM)
• After training phase, we have a trained attribute detector for
each attribute specified in the activity-attribute matrix
Angus F.M. Huang
20
Attribute-Based Activity Classification
• A nearest-neighbor classifier is used to recognize
the high-level activity
– given an attribute vector generated from attribute detectors
– in the attribute space
• The activity recognizer
– takes an attribute vector a = [A1,A2,…,Am] as input
– returns the closest high-level activity y*
Angus F.M. Huang
21
Hybrid Feature/Attribute-Based
Activity Recognition
• Transforming low-level features to mid-level attributes has the benefit for
unseen class recognition
– to keep the advantages of both feature-based and attribute-based
• How do we know if a sample belongs to a seen class or an unseen class?
– a sample from seen class -> similar to others
– a sample from unseen class -> like anomaly
• We first train an unseen class detector using the one-class SVM classifier
– where only the positive samples are given to the classifier
• After using the unseen class detector, we then do a hybrid
feature/attribute-based activity recognition using the Algorithm-1
Angus F.M. Huang
22
Angus F.M. Huang
23
Active Learning: Reinforcing Activity
Recognition using Minimal User Feedback
• Our idea is simple:
– We ask a user for labels only when we are highly
uncertain about our recognition result
• To achieve this:
– We used the idea of uncertainty sampling in the field of
active learning
• The idea of active learning algorithms
– a machine learning algorithm can perform better with less
training data if it is allowed to choose the data from which
it learns
Angus F.M. Huang
24
• Hybrid Sampling Scheme
– Stream-based sampling
– Pool-based sampling
• Uncertainty Sampling Metrics
– Least Confident
– Minimum Margin
– Maximum Entropy
• But!
– Outliers get higher uncertainty scores
– Outliers do not help training a classifier
• Outlier-Aware Uncertainty Sampling
– To select samples that are uncertain but not outliers
– Mean similarity between this sample and all the other samples
– Algorithm-2
Angus F.M. Huang
25
Angus F.M. Huang
26
Evaluation
• System Implementation
– Nexus S 4G phones & MotoACTV wristwatches
• Accelerometer and gyroscope
– Android app
• JAVA, SVM classifier, LibSVM library
Angus F.M. Huang
27
Angus F.M. Huang
28
• Datasets
– Exercise activity dataset
• 20 subjects, 10 exercise activities, 10 iterations, dumbbell
• phone-arm, watch-wrist, watch-hip
• 30 Hz sampling rate, 1 second time window size with 50% overlap
– Public dataset on daily-life activities
• A published dataset, Technische Universität Darmstadt
• 34 classes, one subject, 7 days
• Wearable sensor-wrist&hip, 100Hz, 30 seconds with 50% overlap
• 17 attributes
Angus F.M. Huang
29
Evaluation Methodology
• Leave-two-class-out validation
– Each time we first train our system on (N-2) classes, and then test the
discriminative capability of the classifier on the remaining 2 classes
that were "unseen" by the system during the training process
– True positive (TP), True negative (TN), False positive (FP), False negative (FN)
• Precision
– the percentage of times that a recognition result made by the system is correct
• Recall
– the percentage of times that an activity performed by a user is detected by the
system
• F1-score
– a integrated measure that combines both
Angus F.M. Huang
30
Unseen Activity Recognition Result
• What is the overall precision/recall of unseen activity recognition using
NuActiv? How does the performance vary among classes?
• Average accuracy
– 79%, for overall activities
– 80-90%, for five activities
Angus F.M. Huang
31
The Impact of Number of Unseen
Classes And Comparison with Baseline
• How does the recognition accuracy change with the number
of unseen classes?
• Baseline approach
– Random-guess prediction
Angus F.M. Huang
32
Comparison of Different Attribute Detectors
• How does the performance vary with the use of different
classification algorithms for attribute detectors?
– Decision Tree classifier, Naive Bayes classifier, and k-Nearest
Neighbor (k-NN) classifier
– k-NN is comparable to SVM
• but requires the storage and access to all the training data
Angus F.M. Huang
33
Evaluation of The Importance of Attributes
• How to select attributes based on their importance to unseen
activity recognition?
– Discriminability
• how well can an attribute discriminate between different high-level classes
• ArmUp, ArmDown, ArmFwd
– Detectability
• how accurately can we detect the presence or absence of an attribute
Angus F.M. Huang
34
Cross-User Activity Recognition Results
• What is the cross-user performance, i.e. when the users in
the training set are different from those in the testing set? Is
the system able to generalize from one or a few users to
many new users?
– 70-80% for all
– 5 seen users
– 65% for 1 seen
Angus F.M. Huang
35
Impact of Device Position on Attribute
Detection Accuracy
• An attribute is often inherently associated with a
characteristic of a human activity or a motion of a specific
part of human body
– to understand how the position or the set of positions at which the
sensors are placed affects the attribute detection accuracy
• The upper arm sensor
– achieves better and stable
Angus F.M. Huang
36
Recognizing Unseen New Daily Life Activity
• How does NuActiv perform on recognizing unseen daily life
activities?
– 60-70% precision and recall rate
– Some classes have High recall and Low precision, and vice versa
– Some classes do not have clear difference
• Sitting-desk-activities vs. Sitting-talking-on-phone
• Low precision rate
Angus F.M. Huang
37
Comparison of Active Learning Algorithms
• How efficiently can the system reinforce its performance
using active learning? How does the performance vary with
different active learning algorithms?
Angus F.M. Huang
38
Outlier-Aware Uncertainty Sampling Results
• What is the effect of outlier-aware uncertainty sampling on
active learning algorithms?
Angus F.M. Huang
39
Conclusion
• NuActiv
– Uses semantic attribute-based learning to recognize
unseen new activity classes by reusing and generalizing
the attribute model learned for other seen activities
• Outlier-aware active learning algorithm
– To efficiently improves the recognition accuracy of the
system using minimal user feedback
• 79% recognition accuracy on the unseen activity recognition
Angus F.M. Huang
40
Angus Comments
• Very depend on domain knowledge
• Pity: it is unseen, not unknown
• Can apply it to PLASH..
– Unseen activities recognition
– Unseen trip recommendation
– Composite mobility-activity recognition
• car-driver / car-passenger, ...
• Important for adaptive service provision
• Important attributes evaluation for PLASH’s Urban Routing & Friend
Matchmaking
Angus F.M. Huang
41
Dumbbell Fly
Dumbbell Curl
Angus F.M. Huang
42
Bent-Over Row
Chest Press
Angus F.M. Huang
43
Daily Life Activities
Angus F.M. Huang
44
Angus F.M. Huang
45