Wockets 2010
Posture & Ambulation
version April/23/10
Data Quality Assessment Check List
Please go through the list in order to assess the quality of a data set. Please try to do this check no later than one day
after the data collection session (Stanford) or one day after the data has been sent to you to double check (MIT).
Initial and date when each test is completed. In the notes file in the “Special Observations” section write down any
anomalies you encounter.
Data set ID (type ID)
Stanford checker (type name and email)
MIT checker (type name and email)
Please conduct each set of tests. Type your initials in the completed column for each when it is done.
Test
Completed?
Special Notes

If you were present during data collection, think of anything unusual that
happened. Check the notes file and make sure these details are present in as
much detail as possible (e.g., “Wocket ABCD dropped off the ankle roughly
between 11:45 and 12:53 during the treadmill activities.”)
Directory Structure & Files

Check the data collected from all sensors are located in the appropriate folder
according to the directory structure described in the wiki here:
http://wockets.wikispaces.com/Files+and+naming+conventions .

If RTI sensor data was collected, upload the corresponding files (name-rti.csv
and name-rti.xls) to the server under the“MIT Cheked\rti-original\” folder.

If Columbia sensor data was collected, upload the corresponding file (namecolumbia.csv) to the server under the“MIT Cheked\columbia-original\” folder.

If the data collection was done before February 24 of 2010, do the following
steps:
1.
Rename the existing “ActivityLabelsRealtime.xml” file located under the
wockets folder to “Previous_ActivityLabelsRealtime_DATE.xml”
(where “DATE” is the date when you made the copy, e.g. Mar0110).
2.
Copy the new XML “ActivityLabelsRealtime.xml” (new protocol) to the
wockets folder (e.g. copy to “…/Jan1310/wockets” folder). The new
protocol can be downloaded from the wiki at:
http://wockets.wikispaces.com/ActivityLabelsRealtime
Wockets 2010
Posture & Ambulation
version April/23/10
Copy the folder named “audioannotation” located within the annotation
directory (e.g. “Session/annotation/audioannotation”). Rename the copy
to “Previous_audioannotation_DATE” (where “DATE” is the date when
you made the copy).
Sensor Placement
3.

Check that the sensors placement used in the experiment is consistent with
what is recorded in the SensorData.Xml and “notes.doc” files. This check has
to be done for Wockets and MITes sensors, if both were used.
 If any changes were made, correct the files.
Annotations (Part 1)

Open the Wockets Annotator Software and load the annotated activities.
Check that there are not inconsistencies with the assignment of labels or
anything unusual. Some errors will be highlighted red. These may require
manual
adjustment
(e.g.,
if
the
old
version
of
the
“ActivityLabelsRealTime.xml” had misidentified or misspelled activities
without names identical to those in the current protocol, the software will
require that a valid annotation label be reselected).

Check that the “running vigorously in one place” markers are included at the
beginning and end of session and, on newer datasets, that the “Moving one
sensor only” and 5 Wocket position checks are included.

Check that the last labels have appropriate start and end times and date. Be
particularly careful if there has just been a change in daylight savings time.

If any corrections are made, save the session and generate a new XML file.
Annotations (Part 2)

Using the annotator summary window created after clicking the “generate
Xml” button, confirm that the annotation time for both the postures and the
activities is normal given what is expected. For instance, all major activities
that were done for multiple minutes should be present and have approximately
the right number of minutes.

Using the annotator summary window, compare the list of “annotated
postures and activities” and “no annotated postures and activities.” Make
sure that there are not missing or incorrect labels in the list (for example,
activities that were definitely performed, but they don’t appear on the
corresponding list). If there is a problem, it may be necessary to edit the
annotations or replace the “ActivityLabelsRealtime.xml” file with the file
containing the latest version of the protocol.

If there are any important comments written in the session notes, write them
in the notes file.
Merged Data

Remerge the session data using the Wockets Merger Software.
Wockets 2010
Posture & Ambulation
version April/23/10
Quality Assessment Results (Part 1: non-Wockets/MITes devices)

Open the quality assessment report generated by the merger (“results.html”).

Within the quality assessment document, check which devices were merged.
If any devices are missing, indicate them in the notes file. If devices are
legitimately missing because of equipment broke down, check that the
explanation appears in the notes file. If not, indicate the reason is unknown.

Check the percentage of samples and time collected for each type of sensor
(Zephyr, Actigraph, Oxycon, RTI, Columbia, GPS). Indicate in the notes file
if there is anything suspicious or unusual.

Check the start time and start date for each of the devices and make sure
there is not a discrepancy, especially a 1-3 hour shift from what you would
expect to see.
Quality Assessment Results (Part 2: Wockets and MITes data)

If MITes were used, check that the MITes sampling rate for each placement is
more than 45%. If not, indicate which are below 45% in the notes file. This
check has to be done only for the old data collections in which MITes were
included.

Check that the Wockets data loss is NOT more than 5%. If there are Wockets
with more that 5% data loss, indicate their IDs in the notes file.

Go the Wockets data lost per activity table and confirm that there are not
activities completely missing or that have an unusual small amount of seconds
of data. If there are, please indicate this in the notes file. Either explain or
indicate that there is no know explanation.

Data Visualization (Part 1: Overall check for missing data and gaps)

Visualize the data collected during the session using the Wockets Viewer
Software. (Make sure the software uses the full screen!)

Verify that all successfully merged devices appear in the visualization graphs
with data points showing. Completely missing devices should be indicated in
the notes file. (Note: data that supposed to be present but not plotting often
indicates there is a timestamp problem, causing the data to not appear.)

Get a global view of data and visually confirm that there are not large
unexpected data gaps or write a note if there are sensors having a quality issue
that has not already been identified. Add all identified problems to the notes
file.

Verify that the annotation labels are correct by checking that the start and end
time of the whole annotation stream. Subsequently, check that the annotation
stream contains “running vigorously in one place” labels at the start and at the
end of the session. For later datasets, check that the “Moving [sensor] only”
activities are present.
Wockets 2010
Posture & Ambulation
version April/23/10
Data Visualization (Part 2: Overall check for annotation label timing)

Uncheck the visualizer check boxes for the MITES (if collected in the
session), Colombia, RTI, SenseWear and HeartRate sensors. This will make it
easier to see the remaining plots of the sensor streams.

Go back to the global view of all the sensors. Verify that the patterns of data
signals cluster within activities (e.g. walking/running on treadmill, bicycling
and lying on back). Check that the start and end annotation times look
accurate, i.e., that there are not activities where the label is clearly too short or
two long based on what you can see in the signal about the activity transitions.
For example, running on treadmill should maintain a steady pattern of high
values in all sensors. In contrast, lying on back should maintain a steady
pattern of low values in all sensors. The transitions between different activities
should line up with the annotation transitions. If you see anything suspicious,
add an entry to the notes file.
Data Visualization (Part 3: Confirm sensor location labeling)

Verify that the Wockets and MITes sensor placement labels make sense given
specific activities.
For newer datasets with the “Moving [sensor location] only” annotated
segments, highlight these segments and zoom in. The AUC (Area under the
curve) red line should have the greatest value for the segment that is marked
as moving. Confirm that each of the five sensors was in the location where it
was labeled. If there is a discrepancy, make a highlighted entry in the notes
file about which locations may be swapped based on this test (this is a serious
problem that will require some manual editing of files to fix!).
For older datasets without the “Moving [sensor location] only” annotated
segments, do the checks described below.

Thigh and ankle check. Go to the “biking” activity and check that the
wrist, hip and upper arm are not showing a significant amount of
movement by looking at the AUC metric (red dots). In this case, thigh
and ankle should register higher movement than the wrist, upper arm and
hip. The ankle should be moving more than the thigh. This check will
confirm that the thigh and ankle placement is what is expected to be.
Note that you must check the values by pointing at some of the dots,
because the relative scales of the axes are different between different
sensors.

Hip check. Go to the “painting with brush” activity and check that the
wrist and upper arm are moving considerably more than the hip using the
AUC curve.

Wrist and upper arm check. Go to the “painting with brush” activity
and check that the wrist is moving more than the upper arm. If it is not
clear one is more than the other, check the “sweeping” activity.
If any of these checks fail, make a note to the right and also in the notes
file about which locations may be swapped based on this test. This is a
serious problem that will require some manual editing of files to fix!
Wockets 2010
Posture & Ambulation
version April/23/10
Data Visualization (Part 4: Confirm sensor alignment and identify missing
transitions)

Verify within the “running vigorously in one place” data segment that the
various sensors are aligned. To do this, first select (zoom in) the segment
labeled as “running vigorously in one place” and, then, click on the “raw
view” button located at the right lower corner of the Wockets Viewer. (This
check confirms that the annotation computer clock was synced.) If you see
any problem with the sensors synchronization, add an entry to the notes file.

Double check the sensors synchronization. Pick an activity and check if the
different devices (zephyr, columbia, RTI, Actigraphs) are aligned with the
Wockets. Add an entry in the notes file if you see an offset of more than +/- 2
seconds.

Look for missing transitions. If there is any, you need to add them using the
“Wockets Annotation” Software. To add/modify the missing transitions with
the annotation software, you need to add the “start and end rows” and edit
their “Label Time” and “Time MS” fields. After your editions, save the
annotation session and re-merge using the “Wockets Merge” Software.
Finally, verify that the transitions were added using the “Wockets Viewer”
Software. Indicate in the notes file which transitions you have added. For
example:
“Unknown added @ time: 10:42:50 am – 10:43:10am”.
Dataset Checking Hand Off
For Stanford:
 Upload the data to the MIT server in the StanfordChecked directory.
 Send an email, attaching this file (with saved initial info) to the MIT team
indicating the data is ready for checking.
 If you don’t get a final confirmation from MIT that the data is checked within
TWO days, send another email asking about the data check status.
 Once you get confirmation the data has been checked from MIT, download it
from the MIT server’s “Stanford Checked” directory. Store it in a convenient
location.
For MIT:
 Add this file (with saved initial info) to the data directory.
 Zip the session folder verifying that the zipped file has the appropriate folder
structure (e.g. “Session/Session/wockets”).
 Move the data to the “MIT Checked” directory on the server.
 Make a backup of the entire dataset to a DVD. Label it and give it to Stephen.
 Send an email to the Stanford team confirming the data collection for this
session is complete.