Physical Activity Recognition from Acceleration
Data under Semi-Naturalistic Conditions
by
Ling Bao
Submitted to the Department of Electrical Engineering and Computer
Science
in partial fulfillment of the requirements for the degree of
Master of Engineering in Electrical Engineering and Computer Science
at the
MASSACHUSETTS INSTITUTE OF TECHNOLOGY
August 2003
@ Ling Bao, MMIII. All rights reserved.
The author hereby grants to MIT permission to reproduce and
distribute publicly paper and electronic copies of this thesis document
in whole or in part.
MASSACH SETTSC 1MNST1TU1TE
OF TECHNOLOGY
JUL 3 0 2003
LIBRARIES
Author.
I epa 7 ment of Electrical Engineering and Computer Science
August 21, 2003
Certified by...
Stephen S. Intille
Research Scientist, Changing Places Consortium
Thesis Supervisor
Accepted by .....
Arthur C. Smith
Chairman, Department Committee on Graduate Students
BARKER
2
Physical Activity Recognition from Acceleration Data under
Semi-Naturalistic Conditions
by
Ling Bao
Submitted to the Department of Electrical Engineering and Computer Science
on August 21, 2003, in partial fulfillment of the
requirements for the degree of
Master of Engineering in Electrical Engineering and Computer Science
Abstract
Achieving context-aware computer systems requires that computers can automatically recognize what people are doing. In this work, algorithms are developed and
evaluated to detect physical activities from data acquired using five small accelerometers worn simultaneously on different parts of the body. Acceleration data was collected from twenty subjects in both laboratory and semi-naturalistic environments.
For semi-naturalistic data, subjects were asked to perform a sequence of everyday
tasks outside of the laboratory. Mean, energy, frequency-domain entropy, and correlation of acceleration data was calculated over 6.71 s sliding windows. Decision table,
nearest neighbor, decision tree, and Naive Bayesian classifiers were tested on these
features. Classification results using individual training and leave-one-subject-out
validation were compared. Leave-one-subject-out validation with decision tree classifiers showed the best performance recognizing everyday activities such as walking,
watching TV, and vacuuming with an overall accuracy rate of 89%. The classifier
captures conjunctions in acceleration feature values that effectively discriminate activities. This is the first work to investigate performance of recognition algorithms
on 20 activities using semi-naturalistic data. These results demonstrate that some
activities of daily living can be accurately detected from acceleration data without
the need for individual training. However, preliminary findings also indicate that individual training can lead to higher recognition rates given sufficiently large training
sets.
Thesis Supervisor: Stephen S. Intille
Title: Research Scientist, Changing Places Consortium
3
4
Acknowledgments
This thesis would not have been possible without my advisor, Stephen Intille, who
kindly prodded me along throughout the entire project; my parents, who supported
me during my arduous years at this Institute; my brother, who entertained me tirelessly, knowingly or otherwise; and my friends, who distracted me from working too
hard. I am deeply grateful to all these people.
Thanks also to the National Science Foundation for funding this project (#0112900
ITR/PE: Using context recognition for preventive medicine in the home) and my research over the past year.
5
6
Contents
1
Introduction
15
2
Background
17
3
Theory
21
4
Design
25
5
. . . . . . . . . . . . .
25
. . . . . . . . . . . . . . . . . . .
25
..........
4.1
Overview.
4.2
Accelerometers
4.2.1
Hardware.....
25
4.2.2
Placement . . . . .
27
4.3
Activity Labels
. . . . . .
29
4.4
Semi-Naturalistic Protocol
30
4.5
Activity Protocol . . . . .
31
4.6
Activity Annotation
. . .
32
4.7
Feature Extraction
. . . .
33
4.8
Recognition Algorithms.
37
39
Evaluation
5.1
Collection Protocol . . . .
39
5.2
Data Collected
. . . . . .
39
5.3
Results . . . . . . . . . . .
40
5.4
Analysis . . . . . . . . . .
47
5.5
Future Work . . . . . . . .
48
7
6
Conclusion
51
A Subject Protocol
53
A.1
Semi-Naturalistic Session Instructions . . . . . . . . . . . . . . . . . .
54
A.2
Laboratory Session Instructions . . . . . . . . . . . . . . . . . . . . .
55
B Activity List
57
C Prior Work Details
59
D Data Collection Worksheets
67
D.1
Obstacle Course Worksheet
. . . . . . . . . . . . . . . . . . . . . . .
67
D.2
Activity Worksheet . . . . . . . . . . . . . . . . . . . . . . . . . . . .
67
E Hoarder Clock Synchronization
71
F Result Details
73
G Related Results
81
H Data Availability
83
8
List of Figures
4-1
Top view of a hoarder board with accelerometer (a). Hoarders secured
to a subject (b). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4-2
26
Acceleration from five biaxial accelerometers for walking, running, and
tooth brushing. Signals from both axes of each biaxial accelerometer
are shown.........
..................................
28
4-3
A room with a television, vacuum, sofa, and reading materials. .....
4-4
Five minutes of acceleration data annotated with subject self-report
31
activity labels. Data within 10 s of self-report labels is discarded as
indicated by m asking . . . . . . . . . . . . . . . . . . . . . . . . . . .
34
4-5
Differences in feature values aide in discriminating different activities.
36
5-1
Aggregate confusion matrix for C4.5 classifier based on leave-one-subjectout validation for 20 subjects using laboratory and obstacle course data. 46
C-1
Confusion matrix for laboratory data results in [19]. Confusion matrix
is not available for naturalistic results.
. . . . . . . . . . . . . . . . .
62
Obstacle course worksheet, page 1 of 3. . . . . . . . . . . . . . . . . .
68
D-2 Obstacle course worksheet, page 2 of 3. . . . . . . . . . . . . . . . . .
69
D-3 Obstacle course worksheet, page 3 of 3. . . . . . . . . . . . . . . . . .
70
D-4 Activity sequence worksheet. . . . . . . . . . . . . . . . . . . . . . . .
70
E-i
72
D-1
Visually aligned synchronization signals across five accelerometers.
9
.
.
F-i
Aggregate confusion matrix for C4.5 classifier based on leave hip accelerometer in validation for 20 subjects using laboratory and obstacle
course data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F-2
74
Aggregate confusion matrix for C4.5 classifier based on leave wrist accelerometer in validation for 20 subjects using laboratory and obstacle
course data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F-3
75
Aggregate confusion matrix for C4.5 classifier based on leave arm accelerometer in validation for 20 subjects using laboratory and obstacle
course data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F-4
76
Aggregate confusion matrix for C4.5 classifier based on leave ankle accelerometer in validation for 20 subjects using laboratory and obstacle
course data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F-5
77
Aggregate confusion matrix for C4.5 classifier based on leave thigh accelerometer in validation for 20 subjects using laboratory and obstacle
course data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F-6
78
Aggregate confusion matrix for C4.5 classifier based on thigh and wrist
accelerometer data for 20 subjects using laboratory and obstacle course
d ata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F-7
79
Aggregate confusion matrix for C4.5 classifier based on hip and wrist
accelerometer data for 20 subjects using laboratory and obstacle course
d a ta . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10
80
List of Tables
3.1
Summary of past work on activity recognition using acceleration. The
"No. Subj." column specifies the number of subjects who participated
in each study, and the "Data Type" column specifies whether data was
collected under laboratory (L) or naturalistic (N) settings. The "No.
Sensors" column specifies the number of uniaxial accelerometers used
per subject. .........
5.1
................................
22
Summary of classifier results (mean ± standard deviation) using individual training and leave-one-subject-out training. Classifiers were
trained on laboratory data and tested on obstacle course data. ....
5.2
41
Summary of preliminary classifier results (mean ± standard deviation)
using individual training and leave-one-subject-out training where both
training data sets are equivalent to five laboratory data sessions. . . .
5.3
Aggregate recognition rates (%) for activities studied using leave-onesubject-out validation over 20 subjects. . . . . . . . . . . . . . . . . .
5.4
42
44
Difference in recognition accuracy (mean ± standard deviation) due to
leaving only one or two accelerometers in. Accuracy rates are aggregated for 20 subjects using leave-one-subject-out validation.
C.1
. . . . .
Recognition ratios (%) for walking (Level), ascending stairs (Up), descending stairs (Down) in [30]. . . . . . . . . . . . . . . . . . . . . . .
C.2
60
Recognition ratios (%) for standing, walking (Level), ascending stairs
(Up), descending stairs (Down) in [35].
C.3
45
. . . . . . . . . . . . . . . . .
Confusion matrix for three martial arts moves recognized in [12].
11
.
.
.
61
64
C.4
C.5
Recognition ratios (%) for walking (Level), ascending stairs (Up), de-
scending stairs (Down) in [29]. . . . . . . . . . . . . . . . . . . . . . .
65
Recognition rates (%) for activities studied in [52] . . . . . . . . . . .
66
G.1 Summary of classifier results (mean ± standard deviation) using individual training and leave-one-subject-out training for 2 G data. Classifiers were trained on laboratory data and tested on semi-naturalistic
data. ..........
....................................
82
G.2 Summary of classifier results (mean ± standard deviation) using individual training and leave-one-subject-out training. Classifiers were
trained on laboratory data and tested on naturalistic data. . . . . . .
12
82
Chapter 1
Introduction
Context-aware computing can improve human-computer interactions by leveraging
situational information about the user [1]. One facet of the user's context is his physical activity. This work focuses on recognition of everyday activities using acceleration
data that could be collected by a mobile computer.
Although existing works discuss physical activity recognition using acceleration
[30, 12, 10, 33, 35, 52, 19, 5, 22, 51, 6, 32] or a fusion of acceleration and other data
modalities [31, 7, 34], it is unclear how these systems will perform under real-world
conditions.
Specifically, most of these works rely on data collected from subjects under artificially constrained laboratory settings to validate recognition results [30, 12, 35, 52,
5, 22, 6]. Some works also evaluate recognition performance on data collected in natural, out-of-lab settings but only use limited data sets collected from one individual
[51, 13].
A number of works use naturalistic data but do not quantify recognition
accuracy [31, 7, 33, 34, 32].
Lastly, research using naturalistic data collected from
multiple subjects has focussed on recognition of a limited subset of nine or fewer
everyday activities consisting largely of ambulatory motions and basic postures such
as sitting and standing [19, 10]. Consequently, it is uncertain how any of these systems will perform in recognizing a variety of everyday activities for a diverse sample
population under real-world conditions.
In particular, the possibility that large-scale real-world implementation of physical
13
activity recognition systems may require training on individuals' specific patterns
must be explored. Training on individual patterns to maximize recognition accuracy is
required for effective speech dictation recognition systems [46]. If the same individual
training is required for accurate activity recognition, then current systems trained
purely on data collected from supervised laboratory settings [30, 12, 35, 52, 5, 22, 6]
are impractical for real-world application.
Given such a scenario, researchers must
explore activity recognition systems using unsupervised data collection and training
techniques for individual customization.
Although individual training is needed in speech dictation recognition to customize
conditional inter-word probabilities based on a user's vocabulary and dictation style,
accurate word recognition for automated telephone systems does not require individual training. These systems limit the context of the user's speech and train on large
example sets of specific words, simplifying word recognition. For instance, when using
a movie theater's automated telephone system, a user will utter a very limited subset
of words such as "times," "purchase tickets," or the name of a movie that is playing.
Context is exploited to resolve recognition ambiguities. Similarly, the need for individual training of physical activity recognition systems may vary depending upon the
activities or "words" to be recognized, and the difficulty of recognizing activities may
increase substantially in less constrained, non-laboratory settings.
This paper explores the need for individual training of activity recognition systems
and examines the feasibility of an unsupervised approach to such training. The performance of activity recognition algorithms under conditions akin to those found in
real-world settings is assessed. Activity recognition results are based on acceleration
data collected from five biaxial accelerometers placed on 20 subjects under laboratory and semi-naturalistic conditions. These results show that an overall recognition
accuracy of 84.26% can be achieved for 20 everyday activities using five accelerometers without training on individual data. Acceleration of the thigh is the most useful
single sensor location for discriminating activities, and acceleration of the thigh and
wrist can be used to recognize activities with 80.99% overall accuracy.
14
Chapter 2
Background
Technological obstacles to large-scale real-world activity recognition systems are diminishing as mobile computing power improves and becomes more prevalent. Mobile computing devices such as personal digital assistants (PDAs) are increasingly
widespread with estimates of over 12 million handheld units shipped in 2002 [37].
Watches with GPS receivers are currently available, and researchers have prototyped
watches that run Linux [41]. As these technologies continue to develop, it may be possible to create inconspicuous mobile activity recognition systems in the near future.
Researchers have already prototyped wearable computer systems that use acceleration, audio, video, and other sensors to recognize user activity [40, 14, 36, 47, 17].
With further advances in miniaturization, it may be possible to develop a mobile activity recognition system in the convenient form factor of a watch. The system could
include a built-in computer and radio transceiver. Accelerometers embedded within
wrist bands [12], bracelets, adhesive patches, or belts could be placed on various parts
of the user's body, relaying acceleration data using integrated transceivers. The system's onboard computer could collect acceleration data from its sensors, enabling it
to recognize user activities and run real-time context-aware applications.
However, one major unresolved issue with mobile activity recognition systems is
customization. High accuracy recognition of some activities may require an individual training phase as is the case with speech dictation recognition [46]. In dictation
recognition, intra-word and inter-word level relations across individuals exhibit strong
15
similarities due to the words and rules shared through spoken language [46]. Nonetheless, accurate recognition relies upon training on individual speech to account for individual variation in word pronunciation and to reduce uncertainty in sentence-level
recognition [46]. A comparable parallel may apply to activity recognition.
For instance, ambulation may exhibit strong commonalities in body acceleration
between many individuals. However, differences in gait from person to person may
still necessitate customization on an individual basis. These differences may be analogous to differences in word pronunciation between individuals in speech recognition.
Extending the analogy, if activities such as standing, walking, running, jumping, and
sitting are words, then sequences of these activities can be thought of as sentences.
Playing basketball could be a sentence level activity involving sequences of running
and jumping with certain transition probabilities between the word level activities.
Similarly, a sequence of walking, standing, and sitting with certain transition probabilities could be a sentence level representation for commuting on a bus. Paragraph
level activities such as working a job could be considered sequences of sentence level
activities such as commuting on a bus, working at a desk, and sitting through a
meeting with appropriate transition probabilities. Activity recognition systems could
train for individual variation in sentence level activities by modelling the individual's
transition probabilities for the sequence of word level activities composing that sentence. The same could be done for the transition probabilities between sentence level
activities in paragraph level activities.
To determine the feasibility of wide-scale mobile activity recognition systems, this
work assesses the need for individual training of recognition algorithms.
The goal
is to determine whether individual training is required for accurate recognition of
activities such as walking, climbing stairs, eating, and vacuuming. Should the results
support individual training of activity algorithms, this work will explore unsupervised
training techniques for recognition systems.
Note that recognition of sentence and paragraph level activities consisting of sequences of more basic activities such as those listed above is beyond the scope of this
work. Although algorithms such as hidden Markov models may work well for recog-
16
nizing higher level activities [46], collecting sufficient data to train the Markov models
is time consuming. Whereas sentences in speech take seconds to utter, sentence-level
activities may take minutes or hours. Furthermore, understanding word-level activity recognition is a prerequisite to sentence- and paragraph-level activity recognition.
Thus, this work must focuses on lower level activity recognition.
A data set collected from a diverse sample population is critical in assessing the
need for individual training and evaluating the robustness of any proposed recognition
algorithms. For example, a small sample set of tall male test subjects exhibit fairly
identical walking gaits. Consequently, such a limited data set would not be helpful
in assessing the need for individual training of recognition algorithms.
Moreover, if individual variations in activity are significant, an activity recognition
algorithm must be tested on data from many subjects to ensure it performs well for
a range of individual patterns.
Beyond the size of the data set, the quality of the data is also critical. Specifically,
it is essential to verify activity recognition systems on data collected under naturalistic
circumstances because laboratory environments may artificially constrict, simplify, or
influence subject activity patterns.
For instance, laboratory acceleration data of
walking displays distinct phases of a consistent gait cycle which can aide recognition
of pace and incline [6].
However, acceleration data from the same subject outside
of the laboratory may display marked fluctuation in the relation of gait phases and
total gait length. In a naturalistic environment, the subject is less aware of the fact
that his walking is monitored and may assume a more relaxed, unconventional gait.
Furthermore, traffic patterns and mood may frequently alter the subject's walking
pace and style. Consequently, a highly accurate activity recognition algorithm trained
on laboratory data may rely too heavily on distinct phases and periodicity of the gait
cycle for detecting walking. The accuracy of such a system may suffer when tested
on naturalistic data, where there is greater variation in gait pattern.
Due to the above factors, this work compares recognition results for laboratory
and semi-naturalistic data collected from twenty test subjects who were aged 17 to
48 and lived in the Boston-Cambridge area.
17
18
Chapter 3
Theory
Many past works have demonstrated 85% to 95% recognition rates for ambulation,
posture, and other activities using acceleration data [30, 10, 33, 35, 19, 5, 22, 51, 11,
6, 32]. Activity recognition has been performed on acceleration data collected from
the hip [30, 35, 51, 11, 6] and from multiple locations on the body [10, 33, 19, 5, 22].
Table 3 summarizes activity recognition results using acceleration from recent works.
Additional details about these works are available in Appendix C.
Related work in the field of activity counts and computer vision also support
the potential for activity recognition using acceleration.
The energy of a subject's
acceleration can discriminate sedentary activities such as sitting or sleeping from
moderate intensity activities such as walking or typing and vigorous activities such
as running [44, 53, 21, 42, 38, 43].
Body trajectory, which can be derived from
acceleration, is used in computer vision to recognize gestures and gait [9, 26, 25].
Although the literature supports the use of acceleration for physical activity recognition, little work has been done to validate the idea under real-world circumstances.
Most works on activity recognition using acceleration rely on data collected in controlled laboratory settings [30, 12, 35, 52, 5, 22, 6]. Additionally, all of the literature
focuses on recognizing a special subset of physical activities such as ambulation with
the exception of [19] which examines nine everyday activities.
Interestingly, [19]
demonstrated 95.8% recognition rates for data collected in the laboratory as opposed
to 66.7% recognition rates for data collected outside the laboratory in naturalistic set-
19
Ref. Recognition Activities Recognized
Accuracy
[30]
[35]
No.
Subj.
92.85% ambulation
to 95.91%
83% ambulation, posture
Data
No.
Type Sensors
8
L
2
6
L
6
to 90%
[19]
Sensor
Placement
2 thigh
3 left hip,
3 right hip
24
L
4
chest, thigh,
wrist, forearm
24
N
4
[5]
95.8% ambulation, posture,
typing, talking, bicycling
66.7% ambulation, posture,
typing, talking, bicycling
89.30% ambulation, posture
5
L
2
chest, thigh,
wrist, forearm
chest, thigh
[22]
N/A walking speed, incline
20
L
4
3 lower back
[19]
1 ankle
[51]
[12]
[52]
86% ambulation, posture,
1
to 93% play
96.67% 3 Kung Fu
arm movements
42% ambulation, posture,
N
3
2 waist,
1
L
2
1 thigh
2 wrist
1
L
2
2 lower back
10
L
2
2 knee
to 96% bicycling
[47]
85% ambulation, posture
to 90%
Table 3.1: Summary of past work on activity recognition using acceleration. The
"No. Subj." column specifies the number of subjects who participated in each study,
and the "Data Type" column specifies whether data was collected under laboratory
(L) or naturalistic (N) settings. The "No. Sensors" column specifies the number of
uniaxial accelerometers used per subject.
20
tings. These results underline uncertainty in the performance of current algorithms
for recognizing a broad array of everyday activities in naturalistic, uncontrolled environments.
To address these issues, this work will test activity recognition algorithms on
acceleration data collected from 20 subjects performing 20 everyday activities under
both laboratory and semi-naturalistic conditions.
Another concern with physical activity recognition is the need for individual training.
Recognition accuracy rates of 80% to 95% can be achieved for postures and
ambulatory activities using accelerometer data without individual training [30, 35,
19, 5, 51, 47].
However, this may not be the case for all everyday activities.
Al-
though comparisons of activity recognition performance with and without the use
of individual training is lacking, recognition accuracy for certain activities such as
ascending stairs has been shown to improve through use of individual training [29].
Additionally, for an activity such as tooth brushing, individuals may display significant variations in brushing vigor, duration, and posture. In such cases, individual
training would improve discrimination of tooth brushing from other similar motions
such as window scrubbing. Thus, this work will compare recognition accuracy for a
variety of activities with and without the use of individualized training.
Prior literature demonstrates that forms of locomotion such as walking, running,
and climbing stairs and postures such as sitting, standing, and lying down can be
recognized at 83% to 95% accuracy rates using hip, thigh, and ankle acceleration
[30, 35, 19, 5, 22, 51].
Acceleration data of the wrist and arm improve recognition
rates of upper body activities [12, 19] such as typing and martial arts movements. All
past works have used accelerometers with wires, which may restrict subject movement.
Based on these results, this work will use data collected from five wire-free biaxial
accelerometers placed on each subject's right hip, dominant wrist, non-dominant upper arm, dominant ankle, and non-dominant thigh to recognize ambulation, posture,
and other everyday activities. Although each of the above five locations have been
used for sensor placement in past work, no work addresses which of the accelerometer
locations provide the best data for recognizing activities. Consequently, this work will
21
also determine which sensor locations result in the best recognition accuracy. Note
that five biaxial accelerometers is equivalent to ten uniaxial accelerometers. This exceeds the maximum number of simultaneous sensors used in past work, which is six
uniaxial accelerometers [35]. The increased number of simultaneous sensors should
aide in determining which locations are most useful in activity recognition.
22
Chapter 4
Design
4.1
Overview
Subjects wore 5 accelerometers as they performed a variety of activities under two
different protocols for semi-naturalistic and laboratory data collection. Subjects first
participated in a semi-naturalistic data collection session and then participated in a
laboratory data collection session. Details about the accelerometers, activities, and
protocols are provided below.
4.2
4.2.1
Accelerometers
Hardware
Subject acceleration was collected using ADXL210E accelerometers from Analog Devices. These two-axis accelerometers are accurate to ±10 G with tolerances within
2%. Accelerometers were mounted to hoarder boards [20], which sampled at 76.25 Hz
with minor variations based on onboard clock accuracy and stored acceleration data
on compact flash memory. This sampling frequency is more than sufficient compared
to the 20 Hz frequency required to assess daily physical activity [8].
The hoarder
board time stamped one out of every 100 acceleration samples, or one every 1.31
seconds. The time stamp consisted of a year, month, date, hour, minute, and second
23
(b)
(a)
Figure 4-1: Top view of a hoarder board with accelerometer (a). Hoarders secured to
a subject (b).
field. Continuous running time of the hoarder board was limited by battery life. Four
AAA batteries can power the hoarder board for roughly 24 hours. This is more than
sufficient for the 90 minute data collection sessions used in this study. A hoarder
board is shown in Figure 4-la.
Previous work shows promising activity recognition results from +2 G acceleration
data [15] even though typical body acceleration amplitude can range up to 12 G
[8].
Ideally, this study would only use +12 G accelerometers to minimize signal
saturation. However, due to limitations in availability of ±12 G accelerometers, ±10
G acceleration data was used. Moreover, although body limbs and extremities can
exhibit a 12 G range in acceleration, points near the torso and hip experience a 6 G
range in acceleration [8].
The hoarder boards were not electronically synchronized to each other and relied
on independent quartz clocks to time stamp data. Electronic synchronization would
have required wiring between the boards which would restrict subject movements,
especially during whole body activities such as bicycling or running. Additionally, a
24
wire-free sensor system more closely models the long term vision of a mobile activity
recognition system that collects data from wireless "bracelet" sensors that do not
encumber the wearer.
To achieve synchronization without wires, hoarder board clocks were synchronized
with subjects' watch times at the beginning of each data collection session. Due to
clock skew, hoarder clocks and the watch clock drifted between 1 and 3 seconds every
24 hours. To minimize the effects of clock skew, hoarders were shaken together in
a fixed sinusoidal pattern in two axes of acceleration at the beginning and end of
each data collection session.
Watch times were manually recorded for the periods
of shaking. The peaks of the distinct sinusoidal patterns at the beginning and end
of each acceleration signal were visually aligned between the hoarder boards. Time
stamps during the shaking period were also shifted to be consistent with the recorded
watch times for shaking. Acceleration time stamps were linearly scaled between these
manually aligned start and end points. Further details on the synchronization process
is available in Appendix E.
To characterize the accuracy of the synchronization process, three hoarder boards
were synchronized with each other and a digital watch using the above protocol. The
boards were then shaken together several times during a full day to produce matching
sinusoidal patterns on all boards. Visually comparing the peaks of these matching
sinusoids across the three boards showed mean skew of 4.3 samples with a standard
deviation of 1.8 samples between the boards. At a sampling frequency of 76.25 Hz,
the skew between boards is equivalent to .0564 ± .0236 s.
4.2.2
Placement
A T-Mobile Sidekick phone pouch was used as a carrying case for all hoarders. The
carrying case was light, durable, and provided protection for the electronics. A carrying case was secured to the subject's belt on the right hip. Elastic medical bandages
were used to wrap and secure carrying cases at sites other than the hip.
Typical
placement of hoarders is shown in Figure 4-1b. Figure 4-2 shows acceleration data
collected for walking, running, and tooth brushing from the five accelerometers.
25
Walking
10
0
-10
.
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
0.5
1
1
1.5
1
2
2.5
3
3.5
4
4.5
5
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
10
0CI:
10
0
-10
1
1
1
1
1
1
1
1
1
10
-101
0
101
0
-10
0
time (s)
Running
Brushing teeth
10
2-
0
0.
-2
-
0
-
1A
1
2
3
4
0-0
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
20
CII
-10
0
in
-2
1
2
3
4
00
5
2-
&
0
E0
CU
-2
-in
i)
0
1
-
2
3
4
5
0
A
&
2:
0-
A?
0
C:
C
CU
c -2-
-10
0
in
1
2
3
4
0
5
-2
0)
0
0
-E
0
1
2
3
time (s)
4
-2
0
5
time (s)
Figure 4-2: Acceleration from five biaxial accelerometers for walking, running, and
tooth brushing. Signals from both axes of each biaxial accelerometer are shown.
26
No wires were used to connect the hoarders to each other or any other devices.
Each hoarder in its carrying case weighed less than 120 g. Lack of wiring and the
lightness of accelerometers minimized restrictions in subject movement.
4.3
Activity Labels
Twenty activities were studied. These activities were walking, walking while carrying
items, sitting and relaxing, working on computer, standing still, eating or drinking,
watching tv, reading, running, bicycling, stretching, strength-training, scrubbing,
vacuuming, folding laundry, lying down and relaxing, brushing teeth, climbing stairs,
riding an elevator, and riding an escalator. To address ambiguities in activity labels,
subjects were provided short sentence descriptions of each activity (see Appendix B).
For example, walking was described as "walking without carrying any items in you
hand or on your back heavier than a pound" and scrubbing is described as "using a
sponge, towel, or paper towel to wipe a window."
The 20 activities were selected from a comprehensive list of everyday activities
[4] and designed to include a range of common everyday activities that involve different parts of the body and range in level of intensity. Whole body activities such as
walking, predominantly arm-based activities such as brushing of teeth, and predominantly leg-based activities such as bicycling were included. Sedentary activities such
as sitting, light intensity activities such as eating, moderate intensity activities such
as window scrubbing, and vigorous activities such as running were recognized.
In selecting activity labels, one major concern was distinguishing style from content [50]. For instance, an algorithm may recognize running but not the exact speed
of the running action or window scrubbing but not the vigor of the scrubbing action.
In this work, activity labels were chosen to reflect the content of the actions but do
not specify the style. Consideration of the style of each of the 20 activities would
multiply the number of activity labels, requiring greater amounts of training data
and possibly specialized algorithms for distinguishing style. Thus, recognizing style
is beyond the scope of this paper.
27
However, even limiting activity labels to reflect content is challenging. In speech
recognition, words are defined by a specific sequence of letters. However, physical
activities are not so clearly defined.
The activity "scrubbing" can be interpreted
as window scrubbing, dish scrubbing, or car scrubbing. To mitigate ambiguity in
performance of activities, subjects were given short definitions of the 20 activity labels
(see Appendix B). These short definitions resolved major ambiguities in the activity
labels while leaving room for interpretation so that subjects could show natural,
individual variations in how they performed activities.
4.4
Semi-Naturalistic Protocol
Semi-naturalistic data collection is an intermediate between laboratory and naturalistic data collection. For semi-naturalistic data collection, subjects ran an obstacle
course consisting of a series of activities listed on a worksheet (see Appendix D.2).
These activities were disguised as goals in an obstacle course to minimize subject
awareness of data collection. For instance, subjects were asked to "use the web to
find out what the world's largest city in terms of population is." Subjects recorded the
time they began each obstacle and the time they completed each obstacle. Subjects
completed each obstacle on the course ensuring capture of all 20 activities being studied. There was no researcher supervision of subjects while they collected data under
the semi-naturalistic collection protocol. Many activities were performed outside of
the lab, but some activities such as watching TV, vacuuming, lying down and relaxing, and reading were performed in a "common room" within the lab equipped with a
television, vacuum, sofa, and reading materials. This room is pictured in Figure 4-3.
No researchers or cameras monitored this room.
Although naturalistic data is most desirable, it requires direct observation of subjects by researchers, subject self-report of activities, or use of the experience sampling
method [54] to label subject activities for algorithm training and testing. However,
direct observation can be costly and scales poorly for the study of large subject populations, subject self-report is prone to recall errors [28], and the experience sampling
28
Figure 4-3: A room with a television, vacuum, sofa, and reading materials.
method requires frequent interruption of subject activity, which agitates subjects
over an extended period of time. Furthermore, some activities such as folding laundry, riding escalators, and scrubbing windows may not occur on a daily basis. A
purely naturalistic protocol would not capture sufficient samples of these activities
for thorough testing of recognition systems.
Due to these deficiencies, a semi-naturalistic collection protocol was used in place
of a purely naturalistic protocol to overcome its shortcomings while minimizing laboratory control on subjects.
4.5
Activity Protocol
Activity recognition results on laboratory data is designed to evaluate algorithm performance under ideal circumstances and to act as a baseline for comparison with
results on semi-naturalistic data. Subjects were requested to perform random sequences of activities defined on a worksheet (see Appendix D.1) during laboratory
data collection. Subjects performed the sequence of activities given at their own pace
and labelled the start and end times of each activity in this case. Although this data
collected under this protocol is considered laboratory data, the protocol allowed sub29
jects to perform their activities anywhere including outside of the laboratory. Also,
there was no researcher supervision during the data collection session.
Requiring
subjects to perform certain activities under this protocol ensured that activities that
may appear rarely in everyday life could be captured. These activities include folding
laundry, riding escalators, and scrubbing windows. In assessing the robustness of a
recognition system, it is important to test on these types of everyday activities since
they may not appear frequently enough during naturalistic collection for sufficient
sample sizes to be gathered. Subjects were free to perform activities anywhere, but
they often performed activities such as watching TV, vacuuming, lying down and
relaxing, and reading in the "common room" shown in Figure 4-3.
4.6
Activity Annotation
Under the laboratory data collection protocol, subjects were asked to perform a randomized sequence of activities on a worksheet (see Appendix D.1). For example, the
first 3 activities listed on the worksheet might be "bicycling," "riding elevator," and
"standing still." As subjects performed each of these activity in the order given on
their worksheet, they labelled the start and stop times for that activity and made any
relevant notes about that activity such as "I climbed the stairs instead of using the
elevator since the elevator was out of service." Acceleration data collected between
the start and stop times were labelled with the name of that activity.
Under the semi-naturalistic protocol, subjects were asked to perform an obstacle
course in the form of a worksheet (see Appendix D.2). As subjects performed each
of these obstacles in the order given on their worksheet, they labelled the start and
stop times for that activity and made any relevant notes about that activity such as
"I couldn't eat any of the snacks provided because I am on a strict diet." Acceleration data collected between the start and stop times were labelled with the name of
that activity. Subjects were free to rest between obstacles and proceed through the
worksheet at their own pace as long as they performed obstacles in the order given.
Furthermore, subjects had freedom in how they performed each obstacle. For exam-
30
ple, one obstacle was to "read the newspaper in the House-n common room. Read
the entirety of at least one non-frontpage article." The subject could choose which
and exactly how many articles to read.
Start and stop times for labels were hand annotated by the subject, making them
imprecise. To minimize mislabelling, data within 10 s of the start and stop times was
discarded. For example, if a subject reported that he started running at 11:05:10 and
stopped running at 11:07:13, data from 11:05:10 to 11:05:20 and data from 11:07:03 to
11:07:13 was discarded. Only data from 11:05:20 to 11:07:03 was labelled as running.
Since the subject is probably standing still or sitting while he records the start and
stop times, the data immediately around these times may not correspond to the
activity label. Figure 4-4 shows acceleration data annotated with subject self-report
labels.
4.7
Feature Extraction
Features were computed on 512 sample windows of acceleration data with 256 samples
overlapping between consecutive windows. At a sampling frequency of 76.25 Hz, each
window represents 6.7 seconds. Mean, energy, frequency-domain entropy, and correlation features were extracted from the sliding windows signals for activity recognition.
Feature extraction on sliding windows with 50% overlap has demonstrated success
in past works [15, 52]. A window of several seconds was used to sufficiently capture
cycles in activities such as walking, window scrubbing, or vacuuming. The 512 sample
window size enabled fast computation of FFTs used for some of the features.
The DC feature is the mean acceleration value of the signal over the window. The
energy feature was calculated as the sum of the squared discrete FFT component
magnitudes of the signal. The sum was divided by the window length for normalization. Additionally, the DC component of the FFT was excluded in this sum since
the DC characteristic of the signal is already measured by another feature. Note that
the FFT algorithm used produced 512 components for each 512 sample window. Use
of mean [19, 5] and energy [49] of acceleration features has been shown to result in
31
10
rir
strength-training
vin
0
-10
0
0.5
1
1011111
0
1.5
riding r
2.5
2
walkin
3
3.5
4
4.5
5
4
4.5
5
4
4.5
5
strength-training
0
*
-10,
0
0.5
1
10
-10
&
10
1.5
ridingr
0.5
0
1
1.5
2
2.5
walkin
3
3.5
strength-training
2
2.5
3
3.5
=:PII
r0 eletr
in
st ngth-training
-a10,
10
0
0.5
1
0
-10,
0
1.5
2
eldvntor
0.5
1
1.5
2.5
3
3.5
4
4.5
5
3
3.5
4
4.5
5
instrength-training
2
2.5
time (minutes)
Figure 4-4: Five minutes of acceleration data annotated with subject self-report activity labels. Data within 10 s of self-report labels is discarded as indicated by masking.
32
accurate recognition of certain postures and activities (see Table 3).
Frequency-domain entropy is calculated as the normalized information entropy of
the discrete FFT component magnitudes of the signal. Again, the DC component of
the FFT was excluded in this calculation. This feature may support discrimination of
activities with similar energy values. For instance, biking and running may result in
roughly the same amounts of energy in the hip acceleration data. However, because
biking involves a nearly uniform circular movement of the legs, a discrete FFT of hip
acceleration in the vertical direction may show a single dominant frequency component at 1 Hz and very low magnitude for all other frequencies. This would result in
a low frequency-domain entropy. Running on the other hand may result in complex
hip acceleration and many major FFT frequency components between 0.5 Hz and 2
Hz. This would result in a higher frequency-domain entropy.
Correlation is a feature computed between two different acceleration axes.
It
is calculated as the dot product of two acceleration axes divided by the window
length of 512. Features that measure correlation or acceleration between axes can
improve recognition of activities involving movements of multiple body parts [22, 6].
Correlation is calculated between the two axes of each accelerometer hoarder board
and between all pairwise combinations of axes on different hoarder boards.
Figure 4-5 shows the use of the above features in discriminating different activities.
It is anticipated that certain activities will be difficult to discriminate using these
features.
For example, "watching TV" and "sitting" should exhibit very similar if
not identical body acceleration. In fact, it may be impossible to distinguish these two
activities purely from acceleration data. Additionally, activities such as "stretching"
may show marked variation from person to person and for the same person at different
times. Stretching could involve light or moderate energy acceleration in the upper
body, torso, or lower body. The number of possible stretches a subject could engage
in makes recognition difficult. Nonetheless, recognition will be performed on these
types of activities to test the limitations of algorithms presented.
33
standing: mean vertical acceleration = 1.05 G
0.
CL
sifting: mean vertical acceleration = 0.54 G
2
2
1
1
0
-
0
-1
-2
-2
0
1
2
3
4
5
walking: vertical acceleration energy = 68.4
10
10
5
0
0
1
2
3
4
5
running: vertical acceleration energy = 685.2
5
0
0.
4
0
-c
-5-
-5
10
-10'
0
1
2
3
4
5
walking: forward acceleration FFT entropy = 0.7
10'
1
2
3
4
5
10l
1
5
5
0
0
C
Cz
0
bicycling: forward acceleration FFT entropy = 0.9
0
224K~
CO
-5
-10'
-5
0
'
1
2
3
time (s)
4
-101
0
5
1
2
3
4
5
time (s)
Figure 4-5: Differences in feature values aide in discriminating different activities.
34
4.8
Recognition Algorithms
Conventional decision table [24], nearest neighbor [3], decision tree [45], and Naive
Bayesian [23] classifiers were tested for activity recognition using the feature vector.
Decision based [5] and nearest neighbors [19, 29] algorithms have been used in past
work to recognize activities. Naive Bayes is a computationally efficient algorithm that
has been used for pattern classification in a variety of applications. The 20 activities
we are recognizing can be considered "word level" activities since they do not involve
sequences of other more atomic activities. Using "word level" activity recognition,
systems could detect "sentence level" activities by evaluating transition probabilities
between "word level" activities. Transition modelling could be achieved using hidden
Markov models, which have been used to recognize sign language words from video
by modelling transition probabilities between signed letters [48].
35
36
Chapter 5
Evaluation
5.1
Collection Protocol
Subjects were recruited through flyers seeking research study participants for compensation. Flyers were posted around the MIT campus and were also emailed to the
MIT student population.
Twenty subjects from the MIT community volunteered.
Details on subject recruitment and instructions are given in Appendix A.
Each subject participated in two sessions of study. In the first session, subjects
wore five accelerometers and a digital watch. Subjects completed an obstacle course
worksheet, noting the start and end times of each obstacle on the worksheet.
In the second session, subjects wore the same set of sensors. Subjects performed
the sequence of activities listed on their activity worksheet, noting the start and end
times of these activities. This session lasted around 90 minutes.
5.2
Data Collected
Data was collected from 13 males and 7 females. Subjects ranged in age from 17 to
48 with a mean of 21.8 and standard deviation of 6.59. For laboratory data, each
subject collected between 54 and 131 minutes of data with a mean of 96 minutes and
a standard deviation of 16.7. Eight subjects skipped between one to four activities
during laboratory data collection due to factors such as inclement weather, time
37
constraints, or problems with equipment including the television, vacuum, computer,
and bicycle. Each subject collected between 82 and 160 minutes of semi-naturalistic
data with a mean of 104 minutes and a standard deviation of 13.4. Subjects performed
each activity on their obstacle course for an average of 156 seconds with a standard
deviation of 50 seconds. Six subjects skipped between one to two obstacles during
semi-naturalistic data collection due to factors listed earlier.
5.3
Results
Mean, energy, entropy, and correlation features were extracted from acceleration
data. Activity recognition on these features was performed using decision table [39],
instance-based learning (IBL or nearest neighbor) [18, 2], C4.5 decision tree [45],
and naive Bayes [16, 27] classifiers found in the Weka Machine Learning Algorithms
Toolkit [55].
Classifiers were trained and tested using two protocols. Under the first protocol,
classifiers were trained on each subject's activity sequence data and tested on that
subject's obstacle course data. This individual training protocol was repeated for
all twenty subjects. Under the second protocol, classifiers were trained on activity
sequence and obstacle course data for all subjects except one. The classifiers were
then tested on obstacle course data for the only subject left out of the training data
set. This leave-one-subject-out validation process was repeated for all twenty subjects. Mean and standard deviation for classification accuracy under both protocols
is summarized in Table G.
Overall, recognition accuracy was highest for decision tree classifiers whereas
Bayesian approaches performed poorly. The strong performance of decision tree classifiers is consistent with past work where decision based algorithms recognized lying,
sitting, standing and locomotion with 89.30% accuracy [5]. Nearest neighbor is the
second most accurate algorithm and its strong relative performance is also supported
by past works where nearest neighbor algorithms recognized ambulation and postures
with over 90% accuracy [29, 19]. Rule-based activity recognition captures conjunc38
Classifier
Individual Training Leave-one-subject-out Training
Decision Table
IBL
C4.5
Naive Bayes
36.32 ± 14.501
69.21 ± 6.822
71.58 7.438
34.94 ± 5.818
46.75
82.70
84.26
52.35
± 9.296
± 6.416
± 5.178
± 1.690
Table 5.1: Summary of classifier results (mean ± standard deviation) using individual
training and leave-one-subject-out training. Classifiers were trained on laboratory
data and tested on obstacle course data.
tions in feature values that may lead to higher recognition accuracy. For instance,
the C4.5 decision tree classified sitting as an activity having nearly 1 G downward
acceleration and low energy at both hip and arm. The tree classified bicycling as an
activity involving moderate energy levels and low frequency-domain entropy at the
hip and low energy levels at the arm. The tree distinguishes "window scrubbing"
from "brushing teeth" because the first activity involves more energy in hip acceleration even though both activities show high energy in arm acceleration. The fitting
of probability distributions to acceleration features under a Naive Bayesian approach
may be unable to adequately model such rules due to the assumptions of conditional
independence between features and normal distribution of feature values. Furthermore, Bayesian algorithms may require more data to accurately model feature value
distributions.
Recognition accuracy was significantly higher for all algorithms under the leaveone-subject-out validation process. This indicates that the effects of individual variation in body acceleration may be dominated by strong commonalities between people
in activity pattern.
Additionally, because leave-one-subject-out validation resulted
in larger training sets consisting of data from 19 subjects, this protocol may have
resulted in more generalized and robust activity classifiers.
The markedly smaller
training sets used for the individual training protocol may have limited the accuracy
of classifiers.
To control for the effects of sample size in comparing leave-one-subject-out and
individual training, preliminary results were gathered using a larger training data
39
Classifier Individual Training Leave-one-subject-out Training
C4.5
77.31 + 4.328
72.99 ± 8.482
Table 5.2: Summary of preliminary classifier results (mean ± standard deviation)
using individual training and leave-one-subject-out training where both training data
sets are equivalent to five laboratory data sessions.
set collected for three subjects. These subjects were researcher affiliates and are not
included in the official 20 subject count. Each of these subjects participated in one
semi-naturalistic and five laboratory data collection sessions. The C4.5 decision tree
algorithm was trained for each individual using data collected from all five of his
laboratory sessions and tested on the semi-naturalistic data. The algorithm was also
trained on five laboratory data sets from five random subjects other than the individual and tested on the individual's semi-naturalistic data. The results are compared in
Table 5.3. In this case, individual training resulted in an increase in recognition accuracy of above 4.32% over recognition rates for leave-one-subject-out-training. This
difference shows that given equal amounts of training data, individual training can
recognize activities more accurately than leave-one-subject-out training. However,
the certainty of these conclusions is limited by the low number of subjects used for
this comparison and the fact that the three individuals studied were all researcher
affiliates.
Nonetheless, these initial results support the need for further study of
individual training for physical activity recognition.
The above results suggest that real-world activity recognition systems can rely
on classifiers that are pre-trained on large activity data sets. Although preliminary
results show that individual training can lead to more accurate activity recognition
given large training sets, pre-trained systems offer greater convenience. Pre-trained
systems could recognize many activities accurately without requiring training on data
from their user, simplifying the deployment of these systems. Furthermore, since the
activity recognition system needs to be trained only once before deployment, the slow
running time for decision tree training is not an obstacle. Nonetheless, there may be
limitations to a pre-trained algorithm. Although "word-level" activities were accu40
rately recognized, higher level "sentence-level" activities may show greater variation
between individuals. Ambulation and posture are similar across individuals due to
shared physiology, but higher level activities such as playing football or walking the
dog are more subject to personal behavioral patterns. For instance, playing football
may involve sequences of running and dominant arm acceleration for a quarterback
and crouching and sudden bursts of body acceleration due to collision for a lineman.
However, leave-one-subject-out validation still shows deficiencies in recognizing
certain activities. Figure 5-1 shows an aggregate confusion matrix for the C4.5 classifier based on all 20 trials of leave-one-subject-out validation. Recognition accuracies
for stretching and riding an elevator were below 50%.
Recognition accuracies for
"watching TV" and "riding escalator" were 77.29% and 70.56%, respectively. These
activities do not have simple characteristics and are easily confused with other activities. For instance, "stretching" is often misclassified as "folding laundry" because
both may involve the subject moving the arms at a moderate rate. Similarly, "riding
elevator" is misclassified as "riding escalator" since both involve the subject standing
still. "Watching TV" is confused with "sitting and relaxing" and "reading" because
all the activities involve sitting. "Riding escalator" is confused with "riding elevator"
since the subject may experience similar vertical acceleration in both cases. "Riding
escalator" is also confused with "climbing stairs" since the subject sometimes climbs
the escalator stairs.
Recognition accuracy using the decision tree classifier was also computed under a
leave-one-accelerometer-in protocol. Specifically, recognition results were computed
five times, each time using data from only one of the five accelerometers for the
training and testing of the algorithm. The differences in recognition accuracy rates
using this protocol from accuracy rates obtained from all five accelerometers are
summarized in Table 5.3. These results show the accelerometer placed on the subject's
thigh is the most powerful for recognizing activities. Acceleration of the dominant
wrist is more useful in discriminating activities than acceleration of the non-dominant
arm.
Acceleration of the hip is the second best for activity discrimination.
This
suggests that an accelerometer attached to a subject's cell phone, which is often
41
Activity
Recognition Accuracy
Walking
89.71
82.10
94.78
97.49
95.67
88.67
77.29
91.79
87.68
96.29
41.42
82.51
81.09
96.41
95.14
94.96
85.27
85.61
43.58
70.56
Walking while carrying items
Sitting and relaxing
Working on computer
Standing still
Eating or drinking
Watching TV
Reading
Running
Bicycling
Stretching
Strength-training
Scrubbing
Vacuuming
Folding laundry
Lying down and relaxing
Brushing teeth
Climbing stairs
Riding elevator
Riding escalator
Table 5.3: Aggregate recognition rates (%) for activities studied using leave-onesubject-out validation over 20 subjects.
42
Accelerometer(s) Left In Difference in Recognition Accuracy
Hip
Wrist
Arm
Ankle
Thigh
Thigh and Wrist
Hip and Wrist
-34.12
-51.99
-63.65
-37.08
-29.47
-3.27
-4.78
± 7.115
± 12.194
± 13.143
± 7.601
± 4.855
± 1.062
± 1.331
Table 5.4: Difference in recognition accuracy (mean ± standard deviation) due to
leaving only one or two accelerometers in. Accuracy rates are aggregated for 20
subjects using leave-one-subject-out validation.
placed at a fixed location such as on a belt clip, may enable recognition of certain
activities.
Confusion matrices resulting from leave-one-accelerometer-in testing (see Appendix
F) show that data collected from lower body accelerometers placed on the thigh, hip,
and ankle is generally best at recognizing forms of ambulation and posture.
Ac-
celerometer data collected from the wrist and arm is better at discriminating activities involving characteristic upper body movements such as reading from watching
TV or sitting and strength-training (push ups) from stretching. To explore the power
of combining upper and lower body accelerometer data, a leave-two-accelerometers-in
protocol was used. This protocol is analogous to the leave-one-accelerometer-in protocol, except that data from two accelerometers is used. Data from thigh and wrist
accelerometers and hip and wrist accelerometers were used. These results are also
listed in Table 5.3. Note that recognition rates improved over 25% for the leave-twoaccelerometers-in results as compared to the best leave-one-accelerometer-in results.
Of the two pairs tested, thigh and wrist acceleration data resulted in the highest
recognition accuracy. However, both thigh and wrist and hip and wrist pairs showed
less than a 5% decrease in recognition rate from results using all five accelerometer
signals. This suggests that effective recognition of certain everyday activities can be
achieved using two accelerometers placed on the wrist and thigh or wrist and hip.
43
Hi
NJ M
C 0 00
0
PJ
0
U'
W
JEn
-u00
NJ
0
OH
M
4P
P
J
J
ED
0P
ED
J
0 A 0
D
PNJ
nn
--
H
0
"".l"O
H
n
N0
w
En
.
n-'*
dt-
o
r
C>
-0 n
t
-
a-i
w
n-
-W -- P-
0
r r
-- '
.
0GO kA
to
-'i -
~on-r n
a-'*
LE
0 p 0+a
a-
A
0
i El EE IE EE
N
rtx~nCru
on-
w
Wfl
0 OV4
0
Aj
En '
44
out validation for 20 subjects using laboratory and obstacle course data.
Figure 5-i: Aggregate confusion matrix for C4.5 classifier based on leave-one-subject-
c
n
-'*c c
II
-hm
wooo-
EE EE El EE El
=
to
0 0
EEE1l El
a
x
N
H F,0
3n
-A-I
n
nJ
HC -
0
'tJ
n3t7C--i f.-J
c
atcd
OnLAtlnp
P
W H 0 H 000
ED
Pu
P
PP0 H J 00 o
H
a PU Go P~ WO P" 0 110 0W-0n
wM Ki
EEEE EEEE El
M Ncn-
El
1U'
to
4 0000
oFA w
U -J
PED
N
H En AP0
0
0
0 H n OH
Co
W
0
-4 0 -J HOC FA WHOH
oo 0 En A P00
0PQ0
00
N
EP
w
PA0-O-sJU'NHOMPPOHO0PN3n
N
M
000000 F
COP
OW J00
NOP
ow
-
En
0 -11
P00 aI
wPM
wNJ
PHH
-J W
00
HA -NJ
PPH 0
0Wo oDpHO"Oof
ED
wD A- P n Pf o Ci ED W
WPHOOPaCOJOHOOJOOonH-
O
0wP
W NJ 0
.0 0000
O
FPo
-4
H~ J 00 NJ 00 0 wD wM 0
H
AA
D0-0 En E
En PC 00
0 En 00000
PuOOONOOONtDOHHPnOPWOO-n
N0
0FJ
IQ
0 -400
0000
0 000
oH
0000
Ow
-J 0
c
OP H
Ili
-J
N
1J 0 A
n
5.4
Analysis
This work shows that individual training is not necessary to achieve recognition rates
of over 80% for 20 everyday activities. Using features from previous literature such as
mean [19, 5], energy [49], and correlation [22, 61 of acceleration along with frequencydomain entropy of acceleration, this work demonstrates successful recognition of activities under laboratory and semi-naturalistic conditions.
Classification accuracy rates of between 80% to 95% for walking, running, climbing
stairs, standing still, sitting, lying down, working on a computer, bicycling, and
vacuuming are on par with recognition results using laboratory data from previous
works [30, 35, 29, 52, 47, 19, 5, 51].
However, all cited works used data collected
under controlled laboratory conditions to achieve their recognition accuracy rates.
Considering that subjects could move about freely outside the lab without researcher
supervision while collecting laboratory and semi-naturalistic data, the 84.26% overall
recognition rate achieved in this work is significant in understanding physical activity
recognition under more realistic conditions.
The C4.5 classifier used mean acceleration to recognize postures such as sitting,
standing still, and lying down. Ambulatory activities and bicycling were recognized
by the level of hip acceleration energy. Frequency-domain entropy and correlation
between arm and hip acceleration strongly distinguished bicycling, which showed low
entropy hip acceleration and low arm-hip correlation, from running, which displayed
higher entropy in hip acceleration and higher arm-hip movement correlation. Both
activities showed similar levels of hip acceleration mean and energy. Working on a
computer, eating or drinking, reading, strength-training as defined by a combination of situps and pushups, window scrubbing, vacuuming, and brushing teeth were
recognized by arm posture and movement as measured by mean acceleration and
energy.
"Watching TV," "stretching," "riding elevator," and "riding escalator" were recognized at between 40% and 80% accuracy. The C4.5 decision tree has difficulty in
distinguishishing the activities from others with similar feature values such as sitting,
45
folding laundry, standing, and climbing stairs. Higher accuracy rates for recognition
of these types of activities may require "sentence level" activity recognition systems.
For instance, stretching is usually followed by vigorous exercise. Riding an escalator
is usually preceded by standing still while riding an escalator is preceded by walking.
Accurate discrimination of activities such as watching TV from others such as sitting
may be impossible without audio or visual sensory data.
The stronger performance of leave-one-subject-out training over individual training demonstrated in this work does not invalidate individual training of activity recognition algorithms. Because leave-one-subject-out validation produced training sets 19
times larger than those encountered with individual training, it may be that the individual training data sets collected were just too small or homogeneous. Preliminary
experiments with individual training using more training data per individual shows
improved performance.
However, more data from a greater number of subjects is
required to substantiate these initial findings.
Furthermore, it may be possible to augment the power of individual training
data by requesting subjects to perform activities in a variety of manners to decrease
homogeneity of the training set. This could reduce overfitting and improve recognition
rates achieved with individual training. For instance, subjects could be requested to
scrub windows, slowly, vigorously, and carefully to collect training data for a number
of possible scrubbing motions encountered in real-life situations. However, increasing
training data requires more training time, which is a barrier to widespread real-world
adoption of activity recognition systems. Nonetheless, combining individual training
data with population training data may improve activity recognition rates further
without greatly increasing training time.
5.5
Future Work
Although this work demonstrates that training classifiers exclusively on an individual
basis may lead to poor activity recognition accuracy, combining individual training
with a pre-trained classifier may yield improved activity recognition. In the case of
46
speech, recognition systems integrate pre-compiled general information about language with individual training to maximize recognition accuracy [46].
If the same
paradigm applies for activity recognition, researchers must determine how best to integrate individual training with pre-trained activity models. An effective scheme for
weighting and incorporating the individual training data into a pre-trained classifier
must be determined.
If real-world systems will train on individual activity patterns, the details of unsupervised training for a mobile recognition system must be addressed.
A PDA
connected wirelessly to sensors could collect acceleration and other types of data.
The PDA could ask the user to perform certain activities to initially train the system
for the individual. Once more, this paradigm is akin to that of speech recognition,
where users are requested to utter specific words for the system to train on [46].
Lower recognition accuracies for activities such as stretching, scrubbing, riding
an elevator, and riding an escalator suggest that higher level analysis is required to
improve classification of these activities. Temporal modelling may enable better detection of these and other more complex activities such as "walking a pet" or "playing
basketball." These activities are characterized by a sequence of activities such as walking, standing still, walking, or running, jumping, running. Hidden Markov models
exploit the temporal relation of words in sentences to improve speech recognition [46].
Similarly, low-level activities such as walking, sitting, or standing still can be thought
of as words and higher-level activities can be thought of as sentences. Then hidden
Markov models can exploit the temporal relation of low-level activities in higher-level
activities to improve activity recognition.
Temporal information in the form of duration and time and day of activities could
also be used to detect activities. For instance, standing still and riding an elevator
are similar in terms of body posture. However, riding an elevator usually lasts for
a minute or less whereas standing still can last for a much longer duration.
By
considering the duration of a particular posture or type of body acceleration, these
activities could be distinguished from each other with greater accuracy. Similarly,
adults may be more likely to watch TV at night than at other times on a weekday.
47
Thus, date and time can be used to improve discrimination of watching TV from
sitting and relaxing. However, because daily activity patterns may vary dramatically
across individuals, individual training may be required to effectively use date and
time information for activity recognition.
The decision tree algorithm used in this work can recognize the content of activities, but may not readily recognize activity style. Although a decision tree algorithm could recognize activity style using a greater number of labels such as "walking
slowly," "walking briskly," "scrubbing softly," or "scrubbing vigorously," the extensibility of this technique is limited. For example, the exact pace of walking cannot
be recognized using any number of labels. Other techniques such as neural nets or
regression could be used to recognize parameterize activity style.
Use of other sensor data modalities may further improve activity recognition.
Heart rate data could be used to augment acceleration data to detect intensity of
physical activities. GPS location data could be used to infer whether an individual
is at home or at work and affect the probability of activities such as working on the
computer or lying down and relaxing. Decision trees have been used to recognize
activities such as working, shopping, attending class from GPS location data with
over 90% accuracy [13]. However, use of GPS data may require greater individual
training since individuals can work, reside, and shop in totally different locations.
48
Chapter 6
Conclusion
Using decision tree classifiers, recognition accuracy of over 80% on a variety of 20
everyday activities was achieved using leave-one-subject-out-validation on both laboratory and semi-naturalistic data from 20 subjects. These results are competitive with
activity recognition that only used laboratory data [30, 35, 29, 52, 47, 19, 5, 511. The
success of leave-one-subject-out validation across 20 subjects suggests that accurate
mobile activity recognition systems using acceleration data do not require training on
individuals. This simplifies the wide deployment of these systems in the real-world.
Furthermore, this work shows acceleration can be used to recognize activities such
as vacuuming, window scrubbing, and working on the computer. This extends previous work on recognizing ambulation and posture using acceleration [30, 35, 29, 52,
47, 5, 51]. This work indicates that a mobile system consisting of a mobile computer
and small wireless accelerometers placed on an individual's thigh and wrist may be
able to detect everyday activities in naturalistic settings. The computer could compute mean, acceleration, frequency-domain entropy, and correlation of acceleration
features and run a pre-trained decision tree algorithm on these features. The algorithm could be pre-trained on a large data set of everyday activities. Because decision
trees are slow to train but quick to run, a pre-trained decision tree should be able to
classify user activities in real-time.
Such a mobile activity recognition system would be applicable to many innovative computer applications for mobile computing devices. In the field of preventive
49
health, such systems could recognize changes in physical activity patterns of the sick
or elderly, helping detect the onset of diseases or physical conditions. In the entertainment industry, activity recognition systems could enable portable video game
units to solicit user interaction while the user is idle. PDAs using activity recognition
could encourage users to engage in healthier behavior by suggesting that they take
the stairs the next time they are about to ride an elevator or that they walk to lunch
instead of driving.
To encourage further research in the area of physical activity recognition, all data
collected for this study is made publicly available. See Appendix H for details on
acquiring this data.
50
Appendix A
Subject Protocol
The following outlines the subject recruitment and data collection process.
This
protocol was approved by the MIT Committee On the Use of Humans as Experimental
Subjects.
1. Flyers publicized the need for research subjects. These flyers were distributed
around MIT. Additionally, emails were sent to the MIT community soliciting
volunteer research subjects. Gift incentives of $5 certificates to a local ice cream
shop were used to attract potential subjects.
2. Potential subject volunteers were informed that any data collected from them
during the study would be disassociated with their actual identities and would
be used strictly for training and testing pattern recognition algorithms. Participating subjects signed an informed consent form to document their agreement
to participate in the study.
3. Subjects participated in two data collection sessions lasting between 60 minutes
and 2 hours.
During each session, subjects were given instructions found in
Appendices A.1 and A.2 and a definition of activity labels shown in Appendix
B.
4. Subjects followed given instructions and wore five accelerometers. Acceleration
data was collected during these sessions.
51
5. Subjects who experienced difficulty with their devices or had questions or concerns during the experiment could talk to the research investigator.
6. At the conclusion of the experiment, subjects could submit informal written or
oral feedback about the study. Feedback was not required.
7. Each subject received a $5 gift certificate to a local ice cream parlor at the end
of the study. Subjects were free to decline this offer.
The following sections detail instructions given to subjects for each of the two
data collection sessions.
A.1
Semi-Naturalistic Session Instructions
General Instructions - Session 1
We would like you to complete a series of obstacles in the form of an obstacle
course. Completing all the obstacles should take you around 90 minutes and involves
walking to certain locations and reaching objectives at those locations.
Here are
detailed instructions for this session.
1. You will meet with a lab person to receive a digital watch and 5 accelerometers. Wear the watch on your non-dominant wrist. A lab person will secure
accelerometers to your hip, dominant wrist, non-dominant upper arm, dominant
ankle, and non-dominant thigh. The lab person will give you a worksheet titled
"Obstacle Course Worksheet" (See Appendix D.1). The worksheet will consist
of a table with numbered "obstacles" for you to perform. These obstacles will
usually involve going to a certain destination or performing a specific task.
2. Complete each item on the obstacle course on the worksheet in the order that
they are numbered. Once you are ready to complete an obstacle, write down the
time on your watch (to the second) next to the activity entry on the worksheet
under the "Start Time" column.
52
3. When you are done with an obstacle on the worksheet, write down the time on
your watch (to the second) next to the obstacle entry on the worksheet under
the "End Time" column. You can also make notes about any unusual or special
circumstances under the notes column.
4. Repeat Steps 2 and 3 until you have completed all obstacles on the worksheet
to the best of your abilities. This may require you to go to several different
locations and sometimes to move back and forth between locations.
5. When you have completed all obstacles on the worksheet, return to the lab
person. He will retrieve your watch and remove any accelerometers from your
body. You may ask him any questions or give him comments during this time.
If you have any problems or questions during the study, ask the researcher investigator Ling Bao.
You can also call him if you are outside of the building at
617-452-5642.
A.2
Laboratory Session Instructions
General Instructions - Session 2
We would like you to complete a sequence of activities in a particular order given
on an activity worksheet. Completing all the activities on the worksheet should take
you around 90 minutes and involves such tasks as walking, running, and eating. Here
are detailed instructions for this session.
1. You will meet with the lab person to receive a digital watch and 5 accelerometers. Wear the watch on your non-dominant wrist. A lab person will secure
accelerometers to your hip, dominant wrist, non-dominant upper arm, dominant ankle, and non-dominant thigh. The lab person will give you a worksheet
titled "Activity Worksheet" (see Appendix D.2). The worksheet will consist of
a table with numbered activities for you to perform. These activities will usually involving doing such things as watching TV, doing push ups, and standing
still.
53
2. Complete each activity on the activity worksheet in the order that they are
numbered. Once you are ready to perform an activity, write down the time on
your watch (to the second) next to the activity entry on the worksheet under
the "Start Time" column.
3. Perform each activity for at least 1 minute or to the best of your abilities unless
otherwise specified on the worksheet. When you are done with an activity on
the worksheet, write down the time on your watch (to the second) next to the
obstacle entry on the worksheet under the "End Time" column and the location
at which you performed the activity under the "Location" column. You can also
make notes about any unusual or special circumstances under the notes column.
4. Repeat Steps 2 and 3 until you have completed all activities on the worksheet.
This may require you to go to certain locations.
For instance, the "riding
escalator" activity may require you to find a building with an escalator in it.
You are free to choose exactly where you perform each of the activities.
5. When you have completed all obstacles on the worksheet to the best of your
ability, return to the lab person. He will retrieve your watch and remove any
accelerometers from your body. You may ask him any questions or give him
comments during this time.
If you have any problems or questions during the study, ask the researcher investigator Ling Bao.
You an also call him if you are outside of the building at
617-452-5642.
54
Appendix B
Activity List
The following list describes the 20 activities studied.
The same list was given to
subjects so they were clear on the definition of each activity type.
Walking Walking without carrying any items in your hand or on your back heavier
than a pound.
Walking while carrying items Walking while carrying a backpack.
Sitting and relaxing Sitting down inactively. Does not include working on computer or reading while sitting.
Working on computer Using a computer while seated.
Standing still Standing without moving of legs.
Eating or drinking Consuming foods or beverages.
Watching TV Watching television from a seated position.
Reading Reading a book or texts while seated.
Running Jogging at a moderate clip or higher. Only one foot is contacting the
ground at a time.
Bicycling Riding a bicycle or using a cycling machine in a gym.
Stretching Doing stretching exercises for the arms and legs.
Strength-training Doing push-ups and sit-ups.
Scrubbing Using a sponge, towel, or paper towel to wipe a window.
Vacuuming Vacuuming a carpet.
Folding laundry Folding clothes neatly.
Lying down and relaxing Lying on your back and being inactive.
Brushing teeth Brushing your teeth with a tooth brush.
Climbing stairs Ascending or descending stairs.
Riding elevator Ascending or descending floors via an elevator.
Riding escalator Ascending or descending floors via an escalator.
55
56
Appendix C
Prior Work Details
Details for prior literature relating to physical activity recognition is provided below.
Reference [30]
Activities Recognized walking, ascending stairs, descending stairs
Recognition Accuracy 92.85% to 95.91%
Features standard deviation over 50 samples of forward acceleration, upward acceleration, and thigh angle along with integral of thigh angular velocity between
the last 4 zero-crossings of angular velocity; acceleration low-pass filtered with
2.5 Hz cutoff
Algorithm created fuzzy sets of Gaussian distributions based on features for each
activity; samples classified as activity with highest likelihood given activity
distributions
Subjects 8 (6 males, 2 females)
Data Collection Methodology subjects performed a scripted sequence of walking
motions; data collected with microcontroller worn by subject
Sensor Placement 1 biaxial accelerometer and 1 angular velocity sensor placed
on outside of thigh; accelerometer axes faced forward and upward relative to
subject; axis of angular velocity sensor points forward relative to subject
Sensor Equipment ±2 G accelerometer and angular velocity sensor for detecting
thigh angle sampled at 50 Hz; sensors wired to microcontroller and PDA for
data storage
Figures and Tables Table C.1
57
Level
Up
Down Unclassified
Level 95.91 0.51 0.67
Up
0
94.35
0
Down 0.51
0
92.85
2.92
5.65
6.63
Table C.1: Recognition ratios (%) for walking (Level), ascending stairs (Up), descending stairs (Down) in [30].
Reference [35]
Activities Recognized standing, walking, climbing stairs
Recognition Accuracy 83% to 90%
Features 6 axes of acceleration data decimated by 2 with anti-aliasing filtering; ICA
is used on filtered data to generate 6 components; components are normalized to
have mean of 0 and variance of 1; wavelet coefficients for levels 5 to 8 composed
from 256 sample sliding windows with 64 sample shifts to generate 24 channels;
power of each channel calculated for 24 feature vector
Algorithm 3 multilayer perception neural networks trained for 3 activities using
back propagation; 10-fold cross validation used to test classifiers
Subjects 6
Data Collection Methodology subjects walked a predefined route through an office environment; data collected with laptop computer carried by researcher
with subject
Sensor Placement 2 triaxial accelerators placed on left and right side of hip
Sensor Equipment ±2 G accelerometers sampled at 256 Hz; accelerometers wired
to laptop PC, which collected data
Figures and Tables Table C.2
Reference [19]
Activities Recognized walking, ascending stairs, descending stairs, sitting, standing, lying supine, talking, bicycling, typing
Recognition Accuracy 95.8% for data collected from subjects performing scripted
sequence of activities, 66.7% for naturalistic data
58
Standing Level Down Up
Standing
Level
Down
Up
90
1
0
3
7
85
16
14
0
4
84
0
3
10
0
83
Table C.2: Recognition ratios (%) for standing, walking (Level), ascending stairs
(Up), descending stairs (Down) in [35].
Features DC and AC components of raw signals separated using simulated resistancecapacitance circuit; mean of DC and AC components for signals over 20 second
durations used as features
Algorithm classification used weighted sum of absolute differences between feature
values and activity reference patterns; samples classified as nearest activity
reference using this distance metric
Subjects 24
Data Collection Methodology each subject performed sequence of activities for
fixed durations around the laboratory; 50 minutes of naturalistic data collected
from each subject as the subject actly freely; researchers following subjects
annotated naturalistic data; signals recorded with portable Vitaport recorder
Sensor Placement 4 uniaxial accelerometers placed perpendicular to body surface
on sternum, dorsum of the wrist distal from m. extensor carpi ulnaris, frontal
aspect of thigh distal from m. rectus femoris, frontal aspect of lower leg, 1
uniaxial accelerometer placed between left ear and mastoid to measure vertical
head movement, microphone attached to throat, electrocardiogram with modified Nehb anterior leads
Sensor Equipment ±2 G accelerometers, microphone, electrocardiogram; sensors
wired to microcontroller for data storage
Figures and Tables Figure C-1
Reference [5]
Activities Recognized lying, sitting, standing, locomotion
Recognition Accuracy 89.30%
59
Lying
Sitting
Talking
Typing
Standing
Walking
Descending
Stairs
Ascending
Stairs
Cycling
Lying
16
0
0
0
0
0
0
0
0
Sitting
2
6
2
0
2
0
0
0
0
Talking
0
3
2
0
1
0
0
0
0
Typing
0
16
3
2
1
0
0
0
0
Standing
0
0
0
0
114
3
1
0
0
Walking
0
0
0
0
6
107
4
4
0
Descending
Stairs
Ascending
Stairs
0
0
0
0
2
80
26
1
0
0
0
0
0
3
20
1
25
0
Cycling
0
0
0
0
0
0
0
0
Figure C-1: Confusion matrix for laboratory data results in [19].
is not available for naturalistic results.
13
Confusion matrix
Features forward and vertical acceleration, lowpass filtered acceleration with 0.5 Hz
cutoff, median of acceleration, mean absolution deviation of acceleration, mean
absolution deviation of sum of acceleration channels all over 1 second windows
formed features
Algorithm decision tree using feature set
Subjects 5 (4 males, 1 female)
Data Collection Methodology
subjects acted freely for 1 hour in studio room
except they were requested to perform each type of activity studied at least
once for several minutes; video footage used to annotate data; Physilog system
recorded signals
Sensor Placement uniaxial accelerometer placed at chest to measure vertical acceleration, uniaxial accelerometer perpendicular to front of thigh measured forward
acceleration
Sensor Equipment accelerometers with unknown sensitivity wired to Physilog recording device
Reference [22]
Activities Recognized
walking speed and incline
Recognition Accuracy root mean squared error (rmse) of .12 m/s for speed, 0.014
rad rmse for incline
60
Features 4 acceleration axes, mean, median, variance, skewness, and kurtosis of
acceleration channels, covariance between pairs of channels, gait cycle time,
and maximal negative acceleration of the heel used as feature vector
Algorithm stepwise regression analysis determines features that are most associated
with variability of subject speed, incline; these features are used to train 2 neural
networks for prediciting speed and incline
Subjects 20 (10 m, 10 f); subjects were volunteers aged between 19 and 29
Data Collection Methodology each subject ran 18 laps around an 80-100 m track
at varying speeds; 12 laps of data per subject was used for training, the rest
of data was used for testing; data collected with an electronic device carried
around waist
Sensor Placement triaxial accelerometer placed at lower back, uniaxial accelerometer measuring frontal acceleration placed on Achilles tendon above the ankle
Sensor Equipment ±5 G accelerometers with exception of one ±10 G accelerometer placed at heel; accelerometers wired to microcontroller and sampled at 200
Hz
Reference [51]
Activities Recognized locomotion, standing, sitting, lying down, playing
Recognition Accuracy 86% to 93%
Features unknown due to proprietary nature of DynaPort Monitor
Algorithm proprietary DynaPort ADL Monitor software
Subjects 1, a maintenance worker at city hall
Data Collection Methodology the subject worked regularly while the DynaPort
Monitor recorded acceleration data on 2 days; video footage was used for activity
annotation
Sensor Placement 2 uniaxial accelerometers at the waist measure forward and upward movement; 1 uniaxial acceleratomer on the left thigh measure frontal
acceleration; all accelerometers are attached to DynaPort Monitor
Sensor Equipment ±14 G accelerometers wired to microcontroller
Reference [12]
Activities Recognized 3 types of Kung Fu martial arts movements (cuts, elbows,
punch blocks)
Recognition Accuracy 96.67%
61
Cuts Elbows Punch Blocks
Cuts
Elbows
Punch Blocks
10
1
0
0
9
0
0
0
10
Table C.3: Confusion matrix for three martial arts moves recognized in [12].
Features zero crossing rate of first and second derivatives, root mean square, and
mean of acceleration channels were computed over sliding windows of 32 samples
with 8 samples overlap
Algorithm 9 hidden markov models were used for 9 sub-gestures such as "wood cut",
"grass cut", "throat cut", "side cut"; 3 markov model trained on sequence of
sub-gestures were used to for each of the 3 major gestures
Subjects 1, a martial arts instructor
Data Collection Methodology martial arts instructor performed 10 examples of
gestures for training and 30 sequences of moves, 10 of each type of gesture, for
testing; gestures were performed with only the arm that was monitored with
accelerometers
Sensor Placement 2 orthogonal uniaxial accelerometers secured at wrist measuring
acceleration in the direction of the extended thumb and index finger; accelerometer data recorded via computer
Sensor Equipment ±2 G accelerometer sampled at 96 Hz; accelerometer wired via
PC serial port
Figures and Tables Table C.3
Reference [29]
Activities Recognized walking, ascending stairs, descending stairs
Recognition Accuracy 83.3% to 96.3%
Features AC components of forward and vertical acceleration lowpass filtered with
5 Hz cutoff extracted; time of last positive and negative peaks in filtered acceleration signals over 0.5 s sliding windows comprise 4 features; lag in crosscorrelation between forward and vertical signals is fifth featrue
62
Level
Up
Down
Level
Up
96.3
11.1
0
1.7
83.3
2.8
Down Unclassified
0
0
95.8
2.0
5.6
1.4
Table C.4: Recognition ratios (%) for walking (Level), ascending stairs (Up), descending stairs (Down) in [29].
Algorithm nearest neighbor classification using Euclidean distance between feature
vector and personalized reference feature vectors for each activity; reference
vectors calculated as average of training sample feature vectors for the individual
Subjects 6
Data Collection Methodology subjects performed a scripted sequence of walking
motions; data collected with microcontroller worn by subject
Sensor Placement 2 orthogonal uniaxial accelerometers secured at lower back measuring acceleration in the forward and vertical directions
Sensor Equipment ±2 G accelerometer sampled at 50 Hz, digital compass, infrared
light detector; sensors wired to microcontroller and PDA for data storage
Figures and Tables Table C.4
Reference [52]
Activities Recognized walking, running, ascending stairs, descending stairs, sitting, standing, bicycling
Recognition Accuracy 42% to 96%
Features maximum acceleration, standard deviation of acceleration, number of acceleration zero crossings, mean of standard deviation of acceleration over 50
samples composed the feature set
Algorithm 2-dimensional Kohonen map of 20 by 20 neurons trained on feature set
Subjects 1, researcher
Data Collection Methodology subject carried a laptop which requested the subject to perform certain activities; the subject was asked to press certain buttons
depending on what activity they were doing; this information was used to annotate the data
63
Activity
Recognition Accuracy
Sitting
Standing
Walking
Running
Climbing stairs
96
94
75
78
42
Descending stairs
64
Riding bicycle
91
Table C.5: Recognition rates (%) for activities studied in [52].
Sensor Placement 2 orthogonal uniaxial accelerometers secured above the knee
measuring acceleration in the forward and vertical directions
Sensor Equipment ±5 G accelerometers wired to laptop PC
Figures and Tables Table C.5
Reference [47]
Activities Recognized walking, running, ascending stairs, descending stairs, sitting, standing
Recognition Accuracy 85% to 90%
Features 4 features: root mean square and integration of 2 acceleration axes over 2
s windows
Algorithm single layer neural network trained on set of 4 feature
Subjects 10
Data Collection Methodology data collection protocol not described in paper
Sensor Placement 2 orthogonal uniaxial accelerometers placed in the pocket measuring acceleration in the forward and vertical directions
Sensor Equipment ±2 G accelerometers wired to onHandPC
64
Appendix D
Data Collection Worksheets
Sample obstacle course and activity sequence worksheets are shown below.
D.1
Obstacle Course Worksheet
The obstacle course worksheet lists a fixed order of everyday activities for subjects to
perform during the semi-naturalistic session. A scanned obstacle course worksheet is
shown below.
D.2
Activity Worksheet
The activity sequence worksheet lists a random order of 20 activities subjects to
perform during the laboratory session. Each of the 20 activities is repeated twice on
the worksheet. A scanned activity sequence worksheet is shown below.
65
Everyday Tasks Worksheet
Name:
Name Removed
for Anonymity
Date:
6 - ) -- v?
Start Time (24 hr time End Time (24 hr
HH:MM:SS)
time HH:MM:SS)
Tasks
Turn on the Housen TV and VCR and watch
the videotape. Watch the tape until Raymond
starts kissing his wvife, then
and VCR,
turn off
Special notes
9
the TV
Read the newspaper in the House-n
common room. Read the entirety of at least
one non-frontpagearticle.
Sit in the House-n common room and relax.
Breathe in deeply. Inhale and exhale as
slowly as you can 12 times.
Look at the painting of Boston in the House-n
common room. Count the number of graves
in the cemetery on the left side of the
painting,. Write that number down in the
notes column.
Z2Z
2
Neatly fold the House-n laundry in the
common room. Don't worry, these are clean.
-
-
2-71$--7S
'
-
2-2
1 -2
O
Vacuum the Housen common room. Try to
get all the paper scraps littering the floor.
Turn the vacuum off when you're done.
Use the Windex and paper towels on the
Housejn shelves to clean the windows near
the entrance of the lab. Be sure to clean all
the stains off the windows. Wash your hands
in the kitchen afterwards.
Browse the web on the House-n computer.
Use the web to find out what the world's
largest city In terms of population is. Also find
a major landmark or tourist attraction of this
city. Write down the cQty's name and the
nameef the landmar under the notes
column. Try your best; it it takes too long,
move on.
Reward yourself by eating some snacks and
drinking some water in the common room.
Snacks, cups, and water from the sink can
found in the Housejn kitchen.
a
2
bA
2
L
be
Go to the bathroom and brush your teeth
after that meal. Remember to brush for at
least a few minutes to clean your gums
thoroughly.
0
C
L;
(
(l
6151A
I
I
Figure D-1: Obstacle course worksheet, page 1 of 3.
66
I
Put the House-n backpack on and walk from
the Housen lab ttheelevator banks
Take the elevator to the first floor and stop in
the lobby next o the couches.
Take the stairs down the Kendall T station
and browse the T map. Find the
2
2
2D
-
northernmoststop on the blue line and write
down Its name under the notes column.
£2
3
the T st 6
Ascendhe stairsoutfheTttin
Walk to Dewey Library and enter the library.
44
-2 7
S
2
-2
Go to the elevator banks
1
2
_
V
level.
Walk to the abstract statue in front of the
green building. Write down the name of the
L4 4: 5
statue (Its on a plaque) in the notes column.
Jog at a comfortable pace back to 1
Cambridge Center.
Take the elevator up to the House-n
4.
o
Return the book to its shelf and go back to
the I Cambridge Center lobby.
Take the elevator to Housejn and put down
your backpack in the common room.
Its tIme for
a workout, start stretching!
Stretch your arms, tegs, and back in the
common rloom.
Do 3 sets of 10 sit ups or as many as you can
you're done.
Head down the stairs of the Kendall T. This
time, write down the name of the
westemmost T stop on the MBTA map In the
notes column.
Ride the escalator of the T back to ground
3
57
\
Find any book that looks interesting. Skim
the book jacket and write a onesentence
synopsis of it in the notes column.
fRide the elevator to the lobby
Go outside and unlock the House-n bike (6 12 -6). Bike on the side walks. Staying on
the side walks, bike to the quantum
bookstore and back, Lock up the bike when
-,
33
j
j
5
~
if
_
_____
common room.
Do 3 sets of 10 push ups or as many as you
can and then drink some water from the
fountain on the floor
Figure D-2: Obstacle course worksheet, page 2 of 3.
67
Stretch your arms, legs, and back in the
common room to warm down
Lie down on the sofa and relax for at least a
minute after all that exercise
You're done! Talk to.the researcher to
conclude this session.
I L-(
- z
-A, k A 4
L
Figure D-3: Obstacle course worksheet, page 3 of 3.
Activity Worksheet
Name.
Name Removed
for
Anonymity
-
Date:
Start Time (ipaq 24
Brushing teeth
Walkin while carrying items
Sitting and relaxing
t-
3
if
%
--
45
5SA
-117-
S
,
Starndingistill ,
Riding escalator(best effort)
I-Sol
Brushing teeth
0 W
Vacuuming
Walking
Runnirig (best effort)
Folding landry (2 minutes)_
Watching TV (2minu"tes)
Ciimbingstairs (best effoit)
j
2
-2S
I
-z F'lk*5
5
3
a-
5-1
4vt
a
&
IQ og
q G Zi
71 4
q
33
1'
s
Climbing stairs (best effort)
Standing still
scrubbing
Workingon oSmputer (2 minutes)
0
-3
_
cu
______
"?
(
3
c
_L
_
4
3
.to
115
_
GI
1 1_t__7-_
_
Pi *
3 1 '3
I
11_
____U____,
3 4l
3
W
_
ZA. uf,.
c/ -
3-3
3
IA VG.tb
minutes)
V
m/
%IVd+
w 4tz-I
Vacuuming
Riding escalator(best effort)
_____
7
3
z
?
___
A
37 1 "3
7
ti
k
5
1 1.
Strength-training
tc
i
114
tj
Riding elevator (best effort)
Running (best effort)
Walking
Lying down and relaxing
Lying down and relaxing
Walkingwile carrylng Items
Working on computera(2minutes)
Sitting and relaxing
_
tV vb
0
f
Stretching
Reading
_
_____
A%
Riding elevatorj!bt-eff rt)
or drinking (1.5
1-
ZIS %0_
-z-s r L.
kS
Eating or drinking (1.5 minute)
Scrubbing.
__s
_
4
2V Vw,A
StrOtoising
N_
L__
7--
2
Strengthraining
Bicycling
Folding laundry (2 minutes)
Watchng TV (2 minutes)
Notes
Locat__
r.V~~
4
Reading
Eating
Location
C_
'2-
Bioc1ing
End Time (ipaq 24 hr
HH:MM:SS) time HH:MM:SS)
L
)Z
J
(,
>6
c'
I
1.
2
Activ_____________y___hr time
0 _\
I
-4t
1
__C___
_
_
:t 3
'Io_1_______
YL
-4
A
I____t
Figure D-4: Activity sequence worksheet.
68
t
_
_
f
Appendix E
Hoarder Clock Synchronization
Five hoarder boards were shaken together in a sinusoidal fashion along their two axes
before and after a data collection session. The sinusoidal signals at the beginning and
end of each set of collected data were visually aligned. Time stamps for the aligned
sinusoidal signals were adjusted to match across the five hoarder boards. This was
done at the beginning and end of the signal. Time stamps in between the beginning
and end of the signal were linearly adjusted to be consistent with the aligned time
stamps at the beginning and end. This process mitigates the effects of independent
clock skew between the hoarder boards. Figure E-i shows the sinusoidal signal across
the five hoarder boards after visual alignment.
69
600&500
C)
0
6000
-500-
S400C,,
I
II
2700
2800
2900
I
I
2600
2700
2800
2900
3000
2600
2700
2800
2900
3000
2500
2600
2700
2800
samples (sampling frequency = 76.25 Hz)
2900
3000
2400
2500
2600
2300
2400
2500
2300
2400
2500
2300
2400
D2300
3000
o 600C\j
0 5000400o
III
o600S500E
a,
CZ'
: 6005004001-II
Figure E- 1: Visually aligned synchronization signals across five accelerometers.
70
Appendix F
Result Details
Confusion matrices for leave-one-accelerometer-in and leave-two-accelerometers-in results are shown below.
71
H
ON
AJ r'JA
NJH
w
wLAN
H
A -N
WNJN
Ho
A
0O
NH
PLHHwN r
N
NH
0
l
O O
4 -A,
AA
H OH H A
a,,
NJ FA
H
P)NNLANA
NJNJwJC)@
LANJ
N
H
-
I|
I
O-Na,
0 0l
0
ds
rj
rj
--
'-Jw
H
II
O
It
It
H
It
II
OLA
Anndr.CmP-10jptALA
O
W j . H"
O
NJH
A
I
ww
0
NJA
H
NJNJ
A
I
0
.
II
-hf
O
X
J
cj
--
rtLC
Z _5
-
--
DO
717-
.3:
.3
Lo
7 "
.
a
LD
n
.CL-J-
J :3 :rCLA LtW
D'a
O.LQ
0L LO, O CIL
-t2.
< CL
La
3
r-
L 0
t
0
J C
II
(D 1t
lI
-
C>
a
It
I
fl
72
Figure F-1: Aggregate confusion matrix for C4.5 classifier based on leave hip accelerometer in validation for 20 subjects using laboratory and obstacle course data.
LQ
A
0
,1
rID
S<
-
C3 LAA() 3~:33
=
()1 M rt :3ItO31
C
La LC L
MMLO3
I
t
0 0
tAO -1--0 03
a -J
-hrD -C- n rr I
LxAf-na~cmmanrznus-tx-x
P--J-JA
a
,
,
.- t-C
cAN'n O M.
Dw J
NMw -rf-C
0 PJDLn.1
r-afb C rfCU rtO
It
H
1..-
NHH
HNNJL'tNNNJHH
-N P'
A
on
00 wHa
O~
JNwoMSO
NL
W
r-a
NwONw
A
N
AN
HoNJ c-
000 C)NHO
FA 0 JO pAOHWt
Hj
AH
H -C
NJ
CL0
HHH
J A N HNwc0NLcONJWOOOLNNWJO-O
NL H 00 HN -J N N ON C) Hj Fa' wO
HC)NJ@
ONww, -4 (Ij w J L
CD
03 NJ
.PA0N
@HcO NJH@NJH
N J
@ NC
H0
o MOO ND
@0H c oA A-
OM %0 O
0%
NJNNJ@H
H
L0NJN
HaNHN
0
AC)
Hj
w
0
Nw MJ C) 0 -O PAOoW N N NJ N0 JO N0 or
C)JNH.,NN
HJ @HN
HH
WJ rJ M'~o NJ H H F1 MO -4 -C) A NJ A 0 O F0-A
NJ
P
JO-
A
OH
A N C)
~ww
NMHAO AANNPj H
HO)r'
O
H0
OHJ
OWNJ
Po F4
f
ON Pm
HA
JOP
MH
r-a
OwcA MJ to C% NJNJww
NNJNJ
r a' HPM
o A% C)P HC)
00NJ00
~
H
HH
AA0H
F4 C HrjArujawM
-
H DwH
OOH
AJ i
0 rJ 1-i
J
NJ
NJ
Ni
DL H
iiH00
NJ
N
LiNJ
0
'-i
W
PAO
03
NJ A
IJ -4
H-4
WO Ao @M0
Ni
NJ
N
NJ
i
N
i
W 000
in
H
H
NJ P
koW tPopw
NJNON0
0 H
HN
H Q -,PG
A H 00Li A
W0
NJ
HNJ
W
(3N r
FA
A
NJ
A i JO
0inW
o
Zp - rip
'
WM
w- I-4-0 4NJ NW W
W
PA
pN
W 0 oA
0A
J-
-P
m
WJ
3
A JOH J
f
ii,
:3
i
3
1303
in
ID13
HL,
A
-
11f11
1
1U
:3
in
,t
-3
-(D
.
J
a
0
7 A
-n L
i
3 -
to
and
--
A
b
WJ NJ -J
0 W iE
.D I
CL00-P
asi)e
-Ni.ill
r+-x
rrii-
Cr)
" 3:
O1:
Loraory
:3
.
,O -4.i
3
in H
J
AH
LuHoAx
NM
NH
WJ NJ Wn -4
W
H
D
3
M
br
Ht
n OH
I
n
eave-wrista-
73
Figure F-2: Aggregate confusion matrix for C4.5 classifier based on leave wrist accelerometer in validation for 20 subjects using laboratory and obstacle course data.
0
:3C
L
:-IfE n
:3:3t innM-HO -1-aLi
< <
-,
J
ao H A '
3aiatiO :
inL 0 A
.C
O HH -4NiH
F
x11
l1
1><3 - -0
oo:2i--.-cntha
-0C
F-ir:3LA 0t
-rC0 0f
OH JO
Li-
4
X 0O
OA
O
"
NJ N H Ho
pJ HNJ
-T -' J 0W -J 0 NJ iW
..
-NJNJNJ-
S
0 FJ iiNNJ
--
HANJ-
NJNJo,
00
NJH
A-ON JO W3---..-NJAHN
H.J.-HOOAsNJNJN
H
--
H-uHNJ0NJNJONJHHH
L'iHHAHHNJ
A JHN JOAHi
A9 H 0WJNJN 0M qJO
H o ANJH J J @0
JO inP i-u
NJJNJJ
N i
N 0 H JA NJ
-PQCc
w ij
0J wH PrG
ON ci
00
A
0
s ZL
NJNJNJWJAAA00NJNWW00N0rW
NJ
H
pJ H D A 0 H HH
Lu
W
()3 M wJ A A L A a 0 0 n OH NJ WJ A9 WJ AC
PoN
F1
H1 PcP
00WJ
N
WNNWNJ@NJN
A,
A oH aN0P
JON
0
WNm i
MJ-4 -4
J WOH H ko MoH 00ON31
OW J Ho p n a 0 n ON
J H~
J
NJNJwHN N OWNPWJ
9L ON
0 O 0 O A H9 p
N
J
NHNPANHH
W NJ J W XL 0 H
A O NJ li riO %D
NJNNiJAA@ NJNMPi
in A @000 to in NJ M JO NJ -A 00 ONW JO 00 MJ -t_
NJ
Jo JO 1% M@0
NJN
NJ
AW 00
OAA
M
O NJ
0
W
H
-
-NM
A
H
04
0
A
0
3N
'JW
3
-1
Z 031010
nDtA-t:3
r)
fsI'n mari
X
I'0a33
:S
N
~
WH
14 Q M
D
0 03-
W
Lur
rt:
z3
-2
l
tL
La
3
clsife
riX-
LO M
wP 03-
-03-'.
dx3D:3"a
L
tr
base kAlae
3 r C
U
74
celeroer in3 validgatio cofori20 sujetsi usin labrcaoryie ande obstaleaure
for< C4.
03
t"
MM-NC
D :3
f U-
A
PF 4M
A A0 F3 'H
3 IJ a7
H 0D
-h
Nk40003 '
-H0O
03 000
- I
-LO--M:3
f):
WH k
n 3L03).:3:3:3:3.
H
-J
wFA
-
H oH
luFW
300
.'-0
H
003
lIli0 10
3A L
NID-ID
0 HM0 W0HW-03
,i A 0 o -30 0-0
- '*'c- Cr-:3
a fO0
LCCM
I'D
-a.
LLaO
no
:3
03
mt
,n-3
s.I'Didaion3Co
0
A
l3
00---'*L
ac -3m - 4
oA
-r'r 3 -"r)
W
3A033
P 0030
H AHHHNJH-A
A WC
3 HWA
NJ
O
P
0 A
0 -- WW3
H
HH - r
00 J A OH N
HD
W
a.
MWt~
-
W
MP
-onr
w MJw 0 U
H A 0
WH 0-a
03
w P
H0
-l
tJF
WN
Q J-N %DG a-D N
FA Il ' to
0
N'
-'H
H
W
AH
WM
M'
03
-
H W
-
W
W4-W-A
O 14 0 M
NM W 33 oNJ 03 0-
W
M
W
AP Ho 'H
FA
H H0
HMrm
HW
-4,0
H
WAM 0. A
H
MW
003M
03 MO
F-3 Agreat
W-
AW01Z MOO
HA K)
PQ -4
to
A
030030
,Pe
NM
NQ
03H1a'AOm3M
"'J
P
4 03NwwO3MHa
'- W0M03WH
4 wNJ
r
NJ tWNJ
HA
HAMMrJwAP6jwHFAH
NJO 0 J0
- AP ,3
W
W
0-n
A
WMWWN
O
NJ
:3 :3
'a'
w
W
MW
00
WH
- 03H'ZP"
-3w. H
- WOA4a'0303
" DLw,
H)
o m
P03HJ
W
00 M
00o3
WWN
HH
AA
H0 -J W -J NJ H-
HH-J
-4
H 0 NI OH W
A
ZAHHHHNJ
H
-
AAAAWWWM
WHWNJNJ03
0
H NJ At .
W
NJ 0
HH
H
W
N
HW
03 W
PJww
H
030
am a
030 w
w-AWW
03-.JHAMMw03'aWAom0H OPA 0033 4 A 03000
NJ
W
AH H A W 03
HHA
W
HAA
H
Wa
H
MW,
AN
N
M3 A oW
ANHMNJ
N
NJ 3 W'-JW 0P
Figure
-3AgrgtcofsomarxfrC.lasiirbsdoDev
Figure~~~~~~
ONJ
o '
HNJ
ow HA
HNJ
HF
HOPMWHHWW
3 "J00 WP wM
c
r
drmaa.
c
r
H
P
w
MN
-4W
NJWAW
O AA W
'
wJ0
N
AA
NNJA
a
OJP
NNP'HWU'P
00O3W O At-w-4W
W NNO
N~
f,
NNPP
N
W
ap
j
N A~L
o
NP
o
U
N P
HP
H
'
P
0
PON A
PN
oP N'H
NJ
03 A ON A
P
-DO Nt P0P P
N PW
HFAUN
WA
N
n
j0
w
0H
Wo--s
NP
p'
P N 03 J N
JO
ON
ND
pL
-- Jc
Cv0j=
03
-3Ia,+mm
-nl
(
C3 Man-W
0zrnx-7c--P,
tnaIf--
in
NWWW
W0
-(
t
LO
LO-.
a X
LA
C.
LDa
4L
: .
AL
A
-:MD
:
3:
I
II
2 EC
N00
z n --
I I I I
2E M n
A
El
D,"
D
II
II
11
I
75
Figure F-4: Aggregate confusion matrix for C4.5 classifier based on leave ankle accelerometer in validation for 20 subjects using laboratory and obstacle course data.
Li
:3
x
-0
wF
PN 00-
A 03 N -J NO JO
H II
H II
H1
n
I
'n <LW LnfA
Ln CD~ P X i X
P
C>
A
J n-L C m.C M n n -L n -_
ii, mti o-3 :3 -:-- ' -LO
-3 -= n 5 'n oD LD
t
QM L LO
n O
3
- LAfl'O
LA<n
LO1:
3IA 0
) Mn- -LA t 3
LO
--<an
3 a-W
:3): Y
n < ri MC
-3
< n-L
C.
,- C
M
3
w
-,'
-)
rf 0 -3a
C -.
-3
.
-i (D
O.
P0
N LP
3G
11 II
I
;a oflcar
AN
NJU
FA Fj FA
w
J O,
ON
0w
rpiMw
PZ Ch t
M J
PJ
N C k JO N3 Ah POPA JO OP JO C3 -3
O j 0 j
0
PNl
O
-4
0 03N Jt
PP
A rJ0O
H P 1F
PC>HGo%
W AN
H
PWNi
wNH0C3PPH-L
H
w
A
NP A-
PP WP
NNWPNAPA
A O04 NP
W-4 NPo
U
APAPNNP
UAL
NM
NP
Pj
P NP WON3
to 43%.C
AN
4P
00303
4
N A 4 W 00
MP P
0%AF W
W 03
OANHWJO
PA CA CC
jw
P
W N 030
PAPNJNNWNMPU'U'CNUWWPo-J
Aj fA
W
W
P
Mw
w Wjp
WN
Air
w p P 0J
N 0W N
NJ A r 3 4O WN NJ
0000
HWiPNIN
A A NJ 0
w
COW
-4la %J Fj
N
NJPP
N, JO NWW
NN
ANJANW
0 A
0P
PJ0 ANNWW WOWA W N03JwO
N1 NLW 00
O4
"J LA A%0
N A W
a
-O NJ W WD0
WNWPW
N NJ PPWNPp
A N 3A 03PP AA H
N
AW
NJNHW
NJO
P
00
N',, P~
WN
HP
JOAOWmoOOCUN0PlAW
p
0 03 NJ
-
MOO
wN
F 00 H A N
w
ANJ
NJWN
FA
H
H
wANwM
j AL
''
wN
0
D
W
NHPHo
O4p
(3om 0 ) 00
ja
ap
W
--
a
3
LO
0
FA r FAcf FA
rQ 0 -AL 0 N MON
;C
xH
300
D
N
A
O
N-
A0C M
w MMpH
)
1
A.-h
a0:
>--. 0w
Q
A
H u r
3Fn ONr
WH
-Qin
HP
C
H3
00-
WHOM NJ-.
77
kA
WH
I
-CL-01j
FA0 W LA
0
-"-7
N
w w 0CM H H W B
oFA -
a
- 1E-0 UWO
1J
-C -3 tD w
OW NFA a
NJ.a
NHH
05 A
3 C
celerometer
76
in validation for 20 subjects using laboratory and obstacle course data.
Figure F-5: Aggregate confusion matrix for C4.5 classifier based on leave thigh ac-
wo-i
W
M
o MW N H a OH
0
w
Hi
HUN1a0
H WOW
D LOWWa toN
O
(ML"H
ko"C
A
M
0 D 0
HH0
W-- 0
H
wP
OH FA 00 NJ
W HU NJ
W-Ln 4aO NJ H4
W H J:
C
N
WO
CMJ a
o wH0,
o
O
O W
N H a w o)a a WHO MOW oH HL CO 4 M
M
"-h
A w F4O l
a H Ho a=
AocL
o~ 0A0
00 w -LJ
mpoNa
-4
o >(-
NJHW~OONJA0UkoWWPP-O
ak) N
Pa
A
OH
WJ 00Wr MWC
H N0 O OWN
W CNJ OIf
W
w
FAW
PQA
%AA
O Ln
HJF
0 H4 000wW
O
r
towoA* Oo
00 jwo" P~oor,
HFA
APWI
WHF
NJ
NJHA MM
H H
H
WNJi J
N
to @to -4
H Wa ON H tjKo U, WO
H
F1
H
0 HO1OF
O J 00
0
H
HHWH
00aHF,
NW
WO
N
HA H
oP
M P'D
H N
ic
00
NJ
P
NJ
a1
A
--
W
w
03
4
L
N a
4 NH
cf
H~
HA
H
p
--
NO
oP N1-
J
n
P F-6 Agreat cfuio
3
W,
-in
Ii
n
mo
-'-'n
I li
c -
-'
a'-'--**-
1 -0I
I
H t
o-c
i
I
I
I
an - r3 H
1
Ip~
N~ -4
'-"P-WO
OW
Z W--*m
I
0
C
t
mari for C4.5O -sife baeIntigDn
3: T- 3 L
c-n
Din nn-3'-
I' D LO
(Dt
9U
n - -n-
a) MI
n n-n-T I -'3
I
"1
-h
ri FJ000 37
F
H
pP WOO 0
~
O
HP
NJO
F,
H
o~ 0 a 1L
P
HWN M
FA rO
OP
J pa0-Ho
-0
H03H
PP
JD
DLO rD 0 03 p 4 "
MJP
W" 03H
II
PA
o
D WM Kr.J
W
a P00
0JH
HP H--
N o a
'- " 0
0 4M NJ NJ,
J P .n
WJ HPP WO
NJ
N NW
r
P 0
N 00 0 0
N4 03 0NrJ w HO
.c
H
OPHHH
HD 00 N aC
W
00
NW 0
'- i
NPp
40H
H
OD
-
1J 41 W~O-- C> -H -1C> D " - p o r
0 0 003
-
H
W H
N
NJ
W - HJ
0
H1 H 0 0 0>
0
H -D a P
00
=r
H
I-D w D
aoo
PP
0 oo aL
1414
0
@6
NPNW NNOONNFA
0D
Go
0 0 N4 W 0
F, j oW
P w AP
W
00
a01
p
P
o NJ 0 0 N 0 a 0 a @0 0NN 00 '-" "00D
H
POP~
O
NJ
'-"P
o
H,
HN
p oF
H N J 0 NJ 0 00 0 @0 W0 '-P" P OW %00 0
M A
HHJ
w -P at
OOIl ON O 03
@0
@0 NJ NJ W
WNJ
cN
0 C >0 M 'po
NJ
WJPH
H 00 HA -
00
H,
o HOM Ni
rs
77
Figure F-6: Aggregate confusion matrix for C4.5 classifier based on thigh and wrist
accelerometer data for 20 subjects using laboratory and obstacle course data.
Figure~~~~~~
J J
0 00
NNJ
HQ
0ON 0 HO
0
INJ
P03
IVN
0
P000000
NJ
--J
NJL
H
P
0
0 O NJ -J
NJw"o o0 NJoo
ON
J
NJ
H0
030
P
NJ UJ "J
GONJ 0 -4
H,
0
L
0O-
-
0
3
NJ 0
J J 1 11 JJ
.
H
F1
H H -J
-
00 W-0
-
NJ kA
wFJoM~
0 0 N)
H
aA
A
11
a
trOJ
I
i
f
Co
-. 4=3C
3 -L3
a,~ 9U fD oa :3
3
5 --4m L a
LD
c
,3
LD
J)
D
-
U3
rt X-3
3 D
"9
" JIX
< CLd rt
aa30k
a CL
,3 U3:r i-3
OL0
_r
1W
3.-
N
-
1 1 JJ
JJ
JJ
n
78
Figure F-7: Aggregate confusion matrix for C4.5 classifier based on hip and wrist
accelerometer data for 20 subjects using laboratory and obstacle course data.
0)
() rb 0 e
vu < rt r
LA -
M 0
OH PNJ 0 0 H H r+
-hD
iJ
J
00
03 H 00
0>0 H 0 00 -405
NJ 00 '4
NJ -C
JJ
-.
N
HH
woN
OH wJo -
W
.XnI( MO 2
3
r M<0
2-O--*-t'Ywr+-aJfJOmN
J3C-J -3
A
W &. & WJDO-0JO
-J S t s-3 t
D C-JC nO M<
3 - Z 7 f 777D 3m
=no
c--r n- - ::3
L - --- -- J3 JD DOO
3 DJ-3
LD
C.
1 JJ
.
J NUJWOO NJ W
w
NJ
FA
N-h,
H
030 H H JO 0 77
H 0 NJ 0 0
HA
N
H 0W
w-t
0 WJ 0
Jn
0 0JJ
NJ NJ 0 00 -
NJi
H
N H
%O
JO JO 0 NJ Z, WO N 0 -D
cow P
ON -
NJi 03W JO
'-J '
-4
0
H
0 NJ 0 H %o000M000 Cco
r
03
Ho
-
JH
o
0
-t-P -P ZL
W0 P
PJ OHfJ
H
P (1HNH)J0NJOP
ri 03 W aW
NIJ
F
H OH
W NJ M PHJO
00M M
NJ
P DWw
H
0NJ
P
P0
rJ
PH
H H NJ 4 -D H,
H J
H
0
PN
~-J
HN-J-4PO OH
NJ0H
-J -J
HJ0NJOJ
P
H
P ON P P
No
- .
00W-PJO
H
NJi
N
-PJ
w
0 NJJ
00
00
O JO 0 N
P
H
PH
OJO 00 W
H
MJ WJ~ 0HH H Z,
NJ
41000
OH M 00
Iv
0JO
NwN
WJNJ 0
rP 31F
w
4
r- i
-4WDJ
00N
P
J
NJO
JO H 0
H NJ W00
NJ i
0
0NJi NJ
H3
WN0 -J
w
H
H
ONJOOPO4
W
H
00
N
0 W
030OO
0
HNJ
P
N
Ji H W3 C: 0 Ni P
00
H
JOM
0
N
Mmzboowrliwrli
mw WwWOOw00o W
Appendix G
Related Results
A preliminary experiment was conducted prior to this work using ADXL202JE accelerometers from Analog Devices.
Two hoarder boarders mounted with the ac-
celerometers were placed in each subject's pockets and around the subject's arm.
Laboratory and semi-naturalistic data was collected for twenty subjects using protocol similar to those used for the 10 G accelerometer study. Additionally, five subjects
participated in naturalistic data collection. For naturalistic data collection, subjects
carried a PDA equipped with experience sampling software that interrupted them
periodically and asked them for their activity. During naturalistic sessions, subjects
acted freely outside the laboratory. For instance, subjects went to work wearing the
accelerometers and carrying the PDA.
There were several problems with this preliminary experiment.
Because typical
body acceleration amplitude can range up to 12 G [8], data collected with the 2 G
accelerometers was often saturated. Accelerometers placed in the pocket were not
secured firmly and may have collected dampened acceleration of the hip due to loose
pockets.
Furthermore, many subjects participating in the preliminary experiment
were affiliated with the researcher or with the research group. Nonetheless, results
for this preliminary study are shown below.
79
Classifier
Decision Table
IBL
C4.5
Naive Bayes
Individual Training Leave-one-subject-out Training
49.95 +
70.33
73.69 i
32.22 t
11.858
7.333
4.082
8.593
61.93
77.59
88.26
45.71
± 7.492
± 6.913
± 3.941
± 6.188
Table G. 1: Summary of classifier results (mean ± standard deviation) using individual
training and leave-one-subject-out training for 2 G data. Classifiers were trained on
laboratory data and tested on semi-naturalistic data.
Classifier
Decision Table
IBL
C4.5
Naive Bayes
Individual Training Leave-one-subject-out Training
33.18 ± 13.607
62.99 7.579
68.42 ± 5.997
22.58 ± 10.366
56.48 ± 9.523
72.79 ± 7.828
83.31 ± 4.093
37.54 ± .8.571
Table G.2: Summary of classifier results (mean ± standard deviation) using individual
training and leave-one-subject-out training. Classifiers were trained on laboratory
data and tested on naturalistic data.
80
Appendix H
Data Availability
All data collected for this work is freely available to the public for research purposes.
Contact Stephen Intille at intilleLmit.edu for information on acquiring data. The
entire data set is 3.2 GB.
81
82
Bibliography
[1] G.D. Abowd and A.K. Dey. Panel: Towards a better understanding of context and context-awareness. Proceedings of the CHI 2000 Conference on Human
Factors in Computing Systems, pages 304-7, 1999.
[2] D. Aha. Tolerating noisy, irrelevant, and novel attributes in instance-based learning algorithms. International Journal of Man-Machine Studies, 36(2):267-287,
1992.
[3] D. Aha and D. Kibler. Instance-based learning algorithms. Machine Learning,
6:37-66, 1991.
[4] B. E. Ainsworth, W. L. Haskell, M. C. Whitt, M. L. Irwin, A. M. Swartz, S. J.
Strath, W. L. O'Brien, Jr. Bassett, D. R., K. H. Schmitz, P. 0. Emplaincourt, Jr.
Jacobs, D. R., and A. S. Leon. Compendium of physical activities: An update
of activity codes and met intensities. Medicine & Science in Sports & Exercise,
32(9 Suppl):S498-504, 2000.
[5] K. Aminian, P. Robert, E. E. Buchser, B. Rutschmann, D. Hayoz, and M. Depairon.
Physical activity monitoring based on accelerometry:
validation and
comparison with video observation. Medical & Biological Engineering & Com-
puting, 37(3):304-8, 1999.
[6] K. Aminian, P. Robert, E. Jequier, and Y. Schutz.
Estimation of speed and
incline of walking using neural network. IEEE Transactions on Instrumentation
and Measurement, 44(3):743-746, 1995.
83
[7] G. Bieber and M. Korten. User tracking by sensor fusion for situation aware
systems. Proceedings of the 22nd Asian Conference on Remote Sensing, 2:125863, 2001.
[8] C. V. Bouten, K. T. Koekkoek, M. Verduin, R. Kodde, and J. D. Janssen. A triaxial accelerometer and portable data processing unit for the assessment of daily
physical activity. IEEE Transactions on Bio-Medical Engineering., 44(3):136-47,
1997.
[9] M. Brand, N. Oliver, and A. Pentland.
Coupled hidden Markov models for
complex action recognition. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, pages 994-999. Los Alamitos, CA, 1997.
[10] J. B. Bussmann, W. L. Martens, J. H. Tulen, F. C. Schasfoort, H. J. van den
Berg-Emons, and H. J. Stam. Measuring daily behavior using ambulatory accelerometry: the activity monitor. Behavior Research Methods, Instruments, L
Computers, 33(3):349-56, 2001.
[11] J. B. Bussmann, J. H. Tulen, E. C. van Herel, and H. J. Stam. Quantification
of physical activities by means of ambulatory accelerometry: A validation study.
Psychophysiology, 35(5):488-96, 1998.
[12] G. S. Chambers, S. Venkatesh, G. A. W. West, and H. H. Bui. Hierarchical recognition of intentional human gestures for sports video annotation. In International
Conference on Pattern Recognition, pages 1082-5, Quebec City, 2002.
[13] N. Chungfat.
Context-Aware Activity Recognition using TAN Classifiers. M.
Eng. thesis, Massachusetts Institute of Technology, 2002.
[14] B. Clarkson and A. Pentland. Predicting daily behavior via wearable sensors.
Technical report, Massachusetts Institute of Technology Media Laboratory, July
2001. http://web.media.mit.edu/ clarkson/isensed/isensed.Dec.01.2001.pdf.
[15] R.W. DeVaul and S. Dunn. Real-time motion classification for wearable computing applications. Technical report, MIT Media Laboratory, 2001.
84
[16] R.O. Duda and P.E. Hart. Pattern Classification.John Wiley & Sons, New York,
2nd edition, 2000.
[17] J. Farringdon, A. Moore, N. Tilbury, and J. Church P.D. Biemond. Wearable
sensor badge & sensor jacket for context awareness. In 3rd InternationalSymposium on Wearable Computers, pages 107-13. San Francisco, CA, 1999.
[18] E. Fix and J. L. Hodges. Discriminatory analysis, non-parametric discrimination:
consistency properties. Technical report, USAF School of Aviation Medicine,
Randolph Field, Texas, November 1951.
[19] F. Foerster, M. Smeja, and J. Fahrenberg. Detection of posture and motion by
accelerometry: a validation in ambulatory monitoring. Computers in Human
Behavior, 15:571-583, 1999.
[20] Vadim
port,
Gerasimov.
Massachusetts
Hoarder
Institute
board
specifications.
of Technology
Media
Technical
Laboratory,
re2002.
http://vadim.www.media.mit.edu/Hoarder/Hoarder.htm.
[21] D. Hendelman, K. Miller, C. Baggett, E. Debold, and P. Freedson. Validity of
accelerometry for the assessment of moderate intensity physical activity in the
field. Medicine & Science in Sports & Exercise, 32(9 Suppl):S442-9, 2000.
[22] R. Herren, A. Sparti, K. Aminian, and Y. Schutz. The prediction of speed and
incline in outdoor running in humans using accelerometry. Medicine and Science
in Sports and Exercise., 31(7):1053-9, 1999.
[23] G.H. John and P. Langley. Estimating continuous distributions in bayesian classifiers. In Proceedings of the Eleventh Conference on Uncertainty in Artificial
Intelligence, pages 338-45. Morgan Kaufmann, San Mateo, 1995.
[24] R. Kohavi. A study of cross-validation and boostrap for accuracy estimation
and model selection. In Fourteenth International Joint Conference on Artifical
Intelligence, pages 1137-43, Montreal, Canada, 1995. Morgan Kaufmann.
85
[25] Y. Kuniyoshi and H. Inoue. Qualitative recognition of ongoing human action
sequences.
In International Joint Conference on Artificial Intelligence, pages
1600-9, 1993.
[26] H. M. Lakany, A. Birbilis, and G. M. Hayes.
moving light displays.
Recognizing walkers using
Technical report, The University of Edinburgh, 1996.
http://citeseer.nj.nec.com/lakany96recognising.html.
[27] P. Langley and S. Sage. Induction of selective Bayesian classifiers. In Proceedings
of the Tenth Conference on Uncertainty in Artificial Intelligence, pages 399-406.
Seattle, WA, 1994.
[28] R. Larson and M. Csikszentmihalyi. The experience sampling method. In Naturalistic Approaches to Studying Social Interaction: New Directionsfor Methodology of Social and Behavioral Science. Jossey-Bass, San Francisco, CA, 1983.
[29] Seon-Woo Lee and Kenji Mase. Recognition of walking behaviors for pedestrian
navigation.
In Proceedings of 2001 IEEE Conference on Control Applications
(CCA01), pages 1152-5. Mexico City, Mexico, 2001.
[30] Seon-Woo Lee and Kenji Mase. Activity and location recognition using wearable
sensors. IEEE Pervasive Computing, 1(3):24-32, 2002.
[31] P. Lukowicz, H. Junker, M. Stager, T. Von Buren, and G. Troster. Wearnet:
A distributed multi-sensor system for context aware wearables. Proceedings of
UbiComp 2002, pages 361-70, 2002.
[32] M. Makikawa and H. lizumi. Development of an ambulatory physical activity
monitoring device and its application for categorization of actions in daily life.
In MEDINFO, pages 747-750. 1995.
[33] M. Makikawa, S. Kurata, Y. Higa, Y. Araki, and R. Tokue. Ambulatory monitoring of behavior in daily life by accelerometers set at both-near-sides of the
joint. In Proceedings of Medinfo, volume 10(Pt) 1, pages 840-3. 2001.
86
[34] M. Makikawa, S. Kurata, M. Kawato, H. Kobayashi, A. Takahashi, and R. Tokue.
Ambulatory physical activity monitoring system.
In Proceedings of Medinfo,
volume 9 Pt 1, pages 277-81. 1998.
[35] J. Mantyjarvi, J. Himberg, and T. Seppanen. Recognizing human motion with
multiple acceleration sensors. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, pages 747-52. IEEE Press, 2001.
[36] K. Mase, B. Clarkson, and A. Pentland. Recognizing user's context from wearable sensors: Baseline system. Technical report, Massachusetts Institute of Technology Media Laboratory and Advanced Telecommunications Research, Kyoto,
Japan, March 2000.
[37] Tom
wide
McCall.
PDA
market
Press
suffers
release:
through
Gartner
a
dismal
dataquest
year
in
says
world-
2002,
2003.
http://www.gartner.com/5-about/press.releases/pr27jan2003a.jsp.
[38] E. L. Melanson Jr. and P. S. Freedson. Validity of the Computer Science and
Applications, Inc. (CSA) activity monitor.
Medicine & Science in Sports &
Exercise, 27(6):934-40, 1995.
[39] J. Metzner and B. Barnes. Decision Table Languages and Systems. Academic
Press, 1977.
[40] S.J. Morris and J.A. Paradiso. A compact wearable sensor package for clinical
gait monitoring. Motorola Journal,pages 7-15, 2002.
[41] C. Narayanaswami and M. T. Raghunath. Application design for a smart watch
with a high resolution display. In Proceedings of the Fourth International Symposium on Wearable Computers (ISWC '00), pages 7-14. 2000.
[42] J. F. Nichols, C. G. Morgan, L. E. Chabot, J. F. Sallis, and K. J. Calfas. Assessment of physical activity with the Computer Science and Applications, Inc.,
accelerometer: laboratory versus field validation. Research Quarterlyfor Exer-
cise and Sport., 71(1):36-43, 2000.
87
[43] S. M. Patterson, D. S. Krantz, L. C. Montgomery, P. A. Deuster, S. M.
Hedges, and L. E. Nebel. Automated physical activity monitoring: validation
and comparison with physiological and self-report measures. Psychophysiology,
30(3):296-305, 1993.
[44] M. R. Puyau, A. L. Adolph, F. A. Vohra, and N. F. Butte. Validation and
calibration of physical activity monitors in children. Obesity Research, 10(3):150-
7, 2002.
[45] J. Quinlan. C4.5: Programsfor machine learning. Morgan Kaufmann, 1993.
[46] L.R. Rabiner and B.-H. Juang. Fundamentals Of Speech Recognition. Prentice
Hall, Englewood Cliffs, NJ, 1993.
[47] C. Randell and H. Muller. Context awareness by analyzing accelerometer data. In
Blair MacIntyre and Bob Iannucci, editors, The Fourth InternationalSymposium
on Wearable Computers, IEEE Computer Society, pages 175-176. IEEE Press,
2000.
[48] T. Starner and A. Pentland. Visual recognition of American sign language using
hidden Markov models. In Proceedings of the International Workshop on Automatic Face- and Gesture-Recognition, pages 189-94, Zurich, Germany, 1995.
[49] A. Sugimoto, Y. Hara, T. W. Findley, and K. Yoncmoto. A useful method for
measuring daily physical activity by a three-direction monitor.
Scandinavian
Journal of Rehabilitation Medicine, 29(1):37-42, 1997.
[50] J.B. Tenenbaum and W.T. Freeman. Separating style and content. Advances in
Neural Information Processing Systems, 9:662-8, 1997.
[51] M. Uiterwaal, E. B. Glerum, H. J. Busser, and R. C. van Lummel. Ambulatory
monitoring of physical activity in working situations, a validation study. Journal
of Medical Engineering & Technology., 22(4):168-72, 1998.
88
[52] K. Van Laerhoven and 0. Cakmakci. What shall we teach our pants? In The
Fourth International Symposium on Wearable Computers, pages 77-83. IEEE
Press, 2000.
[53] G. Welk and J. Differding. The utility of the digi-walker step counter to assess daily physical activity patterns. Medicine & Science in Sports & Exercise,
32(9):S481-S488, 2000.
[54] L. Wheeler and H.T. Reis. Self-recording of everyday life events: Origins, types,
and uses. Journal of Personality,59:339-354, 1991.
[55]
I.H. Witten and E. Frank. Data Mining: PracticalMachine Learning Tools and
Techniques with Java Implementations. Morgan Kaufmann, 1999.
89