End-to-End Application
Considerations
Daniel P. Siewiorek
Carnegie Mellon
University
February 2013
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
1
Outline


Introduction
Individual
» Wearable Computers
» Augmented Reality

Infrastructure
» Museums
» Hospital
» e-Display

Ad Hoc
» Wildfire
» Volcano

Concluding Lessons
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
2
Outline


Introduction
Individual
» Wearable Computers
» Augmented Reality

Infrastructure
» Museums
» Hospital
» e-Display

Ad Hoc
» Wildfire
» Volcano

Concluding Lessons
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
3
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
4
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
5
Wearable Applications and
Architecture



Procedures - upload at completion
Work Orders - incremental updates
Collaboration - real time interaction
» Client-Server
– Thin Client Legacy Systems
– Interactive Electronic Technical Manuals (IETMs)
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
6
Time Rate of Change of Data
Taxonomy


Procedures. Maintenance and plant operation
applications are characterized by a large volume
of information that varies slowly over time.
A typical request consists of approximately ten
pages of text and schematic drawings. Changes
to the centralized information base can occur on
a weekly basis.
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
7
Savings Using Tactical Information Assistants in
Marine Heavy Vehicle Maintenance
Current Practice
SAVINGS FACTOR
VuMan 3 Field Trials
Personnel
2:1
Current Practice
SAVINGS FACTOR
VuMan 3 Field Trials
Inspection
time
40%
less
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
8
Time Rate of Change of Data
Taxonomy (continued)


Work Orders. The trend is towards more
customization in systems.
Manufacturing or maintenance personnel
receive a job list that describes the tasks
and includes text and schematic
documentation. This information can
change on a daily or even hourly basis.
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
9
Time Rate of Change of Data
Taxonomy (continued)


Collaboration. An individual often
requires assistance. In a “Help Desk” an
experienced person is contacted for
audio and visual assistance. The Help
Desk can service many people
simultaneously.
Information can change on a minute-byminute and sometimes even a second-bysecond basis.
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
10
C-130 Help Desk
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
11
Lessons Learned From Usage






Maximum Size, weight, energy consumption
before change user behavior
No fixed relationship between
input/output/display
User less patient, expect instant response
Intuitive to use, no user’s manual
Information overload, user may focus on
computer rather than physical world
Potential to lose initiative, user does exactly
what the computer tells them to do
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
12
Based on a Lecture
Dan Siewiorek, Thad Starner, Asim Smailagic
Morgan & Claypool


U - User
C - Corporal
» Interface physically without discomfort or distraction

A - Attention
» Divided between physical and virtual world

M - Manipulation
» Controls quick to find, easy to operate

P - Power
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
13
Major Factors in Portable Electronic Systems
and Their Relationship to Design Disciplines
Power
Corporal
Attention
Manipulation
User
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
14
UCAMP
User
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
15
Sensor-based models of
human activity and situation
Example:
» Using just the sensors in a laptop we have predictive
model that determines
“now is a bad time to interrupt”
more accurately
than human
Better than
observers
human observers
90%
with just sensors
85%
already in laptop!
Interns
5Only a 6
Researchers
4
Managers
2
Researchers
1
Managers
65%
Interns
70%
Researchers
76.9%
Human 75%
Observers
Managers
80%
All Subjects

60%
© 2010-2013 Daniel P. Siewiorek
Full Sensor Sets
3
Laptop
7 No Cameras
8
9
or Microphones
Mobil Computing
16
Cognitive Load
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
17
fMRI Experiment Configuration
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
18
JUL 31, 2001
Car Calls May Leave
Brain Short-Handed
By SANDRA BLAKESLEE
The study, published in the Aug. 1, 2001 issue
of the journal NeuroImage, was led by Dr.
Marcel Just, a psychology professor and codirector of the Center for Cognitive Brain
Imaging at Carnegie Mellon University in
Pittsburgh...
Scientists have bad news for
people who think they can deftly
drive a car while gabbing on a cell
phone.
The first study using magnetic
resonance images of brain activity
to compare what happens in
people's heads when they do one
complex task, as opposed to two
tasks at a time, reveals a
disquieting fact: the brain appears
to have a finite amount of space for
tasks
requiring attention.
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
19
Capacity constraint in association areas:
Activation volume is less in dual task compared to single tasks, even
for tasks without cortical overlap (auditory comprehension and mental
rotation)
90.0
80.0
70.0
Right Temporal
60.0
Left Temporal
50.0
Right Parietal
Left Parietal
40.0
30.0
20.0
10.0
0.0
Sentences Alone
© 2010-2013 Daniel P. Siewiorek
Rotations Alone
Sum of Single Tasks
Dual Task
Mobil Computing
20
Assessing measures of cognitive load
while performing tasks in divided attention
•
•
•
•
Cognitive load can be a measure for whether, when
and how to present information while performing a
number of different tasks
Need to manage human attention and understand
when users can attend to new information
Necessary while driving, learning new skills
Particularly an issue with elders
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
21
Assessing measures of cognitive load
•
We are working to develop a reliable, realtime and objective measure of cognitive load
for contexts of divided attention
•
Current techniques not appropriate
• Post-hoc, so not real time
• Not accurate
• Subjective measures
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
22
Study of cognitive load
• Six ECTs: visual perception
and cognitive speed
• Four sensors
• Gaze tracking
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
23
Assessing Cognitive Load
•
20 young adults
•
•
•
•
•
Compared sensors for determining cognitive load
Best feature: > 74%
Sensors/features: median heat flux, ECG
> 81% across all participants
Older adults
• Same sensors useful, although heat flux also seems important
•
•
•
BodyMedia device seems to be sufficient to capture
cognitive load with elementary tasks
Need to see how this transfers to real settings
Then, use information about cognitive load in virtual
coaches
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
24
User Lessons

“The user is not I”
» Built two dozen systems to identify repeated patterns (e.g.
procedures, work orders, team collaboration)

Mobile Users are more impatient than desktop users
» Must operate more like a flashlight than a computer rebooting

Institutional Review Board
» “But madam, they are Marines and they do what they are told”

Be prepared to reduce functionality
» “It’s a fine goal”

Instant Creditability
» “I served with you on the Vinson’s first world cruise”
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
25
UCAMP
Corporal
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
26
“If it looks good,
it will fly well”

MoCCA and VuMan 3 received the prestigious
Industrial Design Excellence Awards (IDEA)
from award co-sponsors Business Week
magazine and the Industrial Designers
Society of America (IDSA).
VuMan 3
MoCCA
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
27
NATO Soldier Using CMU TIA-P Language
Translator in Bosnia
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
28
Corporal Lessons

Air Logistics Command, Sacramento CA
» The vapor barrier on a hot August day

Fort Gorden Battle Command
» “Does this bend?!”

Six units to Bosnia ….. none returned
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
29
UCAMP
Attention
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
30
But, I’ve Always Done it This Way!
On
A KC-135 Aircraft was being pressurized at ground level. The
outflow valves which are used to regulate the pressure of the
aircraft were capped off during a 5 year overhaul and never
opened back up. The post-investigation revealed: that a civilian
depot technician who, "had always done it that way," was using
a homemade gauge, and no procedure.
The technician's gauge didn't even have a max "peg" for the
needle and so it was no surprise he missed it when the needle
went around the gauge the first time. As the technician
continued to pressurize the aircraft, and as the needle was on
its second trip around the gauge the aircraft went "boom" - the
rear hatch was blown over 70 yards away, behind a blast fence!
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
31
KC-135 Pressure Test Results
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
32
Attention Lessons


The more information and more engaging the
virtual world, the less interaction with the
physical world
Use systems for reference (e.g. information
dashboard) then focus on physical world
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
33
UCAMP
Manipulation
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
34
Selection of “hot links” with CMU’s
Wheel/Pointer
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
35
Manipulation Lessons

Strong user interaction mental model
» Not only trained in the field in less than ten minutes but
also able to train the next user
» No users manual
» Training without access to what the user is seeing

Without stationary reference, users easily
become disoriented and confused with
traditional input devices
» “Which is the left button”

Control devices have to be flexible to mount
on different parts of the body
» Boeing – shoulder holster
» Sailors on aircraft carrier – shoulders used to carry tiedown chains
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
36
UCAMP
Power
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
37
Power Lessons

It’s the peak, not the average
» Muffled pops


Warning labels are no substitute for good
design
More to be gained on reducing demand than
increasing source
» Current batteries have half the energy density of
dynamite
» Current user interactions demand all resources to reduce
response time

Land Warrior
» Army standard operating system
» Nine different battery types
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
38
Speech Enabled Augmented
Reality for Maintenance [Goose]



SEAR – Speech Enabled Augmented Reality
3 D Augmented Reality graphical view with
location-sensitive 3 D speech-driven interface
3 D component specific vocabulary triggered by
proximity sensors
» Display component specific commands when in vicinity
» Key word activated

Simultaneous 3 D “parameterized” synthesized
speech streams
» For current values from physical plant

Coded visual markers for tracking, location
» 4 X 4 matrix with dots for over 10,000 combinations
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
39
Augmented Reality - Maintenance of
Nuclear Power Plant Components
Room “augmented” with the CAD drawing of a floorplan
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
40
Speech Enabled Augmented
Reality for Maintenance [Goose]

System Architecture and User Interface
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
41
Outline


Introduction
Individual
» Wearable Computers
» Augmented Reality

Infrastructure
» Museums
» Hospital
» e-Display

Ad Hoc
» Wildfire
» Volcano

Concluding Lessons
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
42
Interactive Museums [Fleck]

San Francisco Exploratorium
» Several hundred interactive exhibits
» Frequently rotated off the floor

Prior work
» PDA, Acoustic Guides, IR beacons for pointing
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
43
Interactive Museums - Services





Informer – detailed information on exhibit
Suggester – what to try
Remember – build record of experiences,
selects what to record
Guider – suggests an order of exhibits*
Communicator – instant messaging, leave
notes at exhibits*
* Not implemented
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
44
Interactive Museums – Lessons
Learned









Overall positive responses
Not enough hands to hold PDA and operate exhibit
interfering with exploration
Undesirable demand on user attention
Lost in hyper reality – focus on device rather than
exhibit
Wow factor – part of positive feedback
Beacons OK but sometimes picked up other beacons
Browser interface – people unfamiliar with stylus
Content design – did not know what was clickable
Forgetting to use Remember – application complexity
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
45
Pervasive Technologies in a
Hospital [Hansen]

Hospital scheduling, coordination, and
awareness system
» Location tracking, context awareness, large interactive
displays, mobile phones

Awareness Media
»
»
»
»
»
»
Status of work in Operating Rooms
Video stream for accessing current state
Progress bar
Chat area
Schedule
Location tracking of who is in Operating Rooms
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
46
Pervasive Technologies in a
Hospital - Scenario






Acute patient
Find/schedule Operating Room
Find surgeon not in Operating Room
Send message
Notify patient ward
Scheduled patient notified of their surgery
postponement
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
47
Pervasive Technologies in a
Hospital – Hardware Lessons
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
48
Pervasive Technologies in a
Hospital – Software Lessons
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
49
Pervasive Technologies in a
Hospital – User Lessons
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
50
e-Campus Display [Storz]

Three deployments
Art Center
Underground Bus Station
Conference Signage
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
51
e-Campus Display [Storz]

Conference Signage
» Dynamic schedule of time/location of conference events

Art Center Exhibition
» 60 year celebration of VE day
» Three displays, video diary, web based diary, interactive
artifact exhibit
» News footage, images of artifacts, visitors record own
memories

Underground Bus Station
» Three screens
» Artistic material, textual information, videos
» Context triggered
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
52
e-Campus Display – Lessons Learned
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
53
e-Campus Display – Lessons
Learned Details – Part 1

Deployments are costly
» Time, money, is there a cheaper way to get the same results

Environmental Challenges can be significant
» Fire alarm tests and power cycling, diesel fumes clogging
projector air filter

After deployment comes maintenance
» Underpass projector failure required road closure, insure
accessibility. Shadow system in lab

Follow the rules
» Accessibility for people with disabilities. Work with local site
manager

See what public sees
» Remotely monitor what public perceives. Add cameras to
view projector output
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
54
e-Campus Display – Lessons
Learned Details – Part 2

Build white boxes, not black
» Be able to monitor whole system state (what displayed
where, when) rather than just observing projected images

Content is king
» Content is the system. Considered late in cycle led to
mismatch between content support requirements and what
was provided

Content is expensive
» Generating compelling content is nontrivial requiring
specialist skills and domain knowledge

Manage your assets
» Difficulties in managing content once it is created including
encodings, archiving, previewing, approval
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
55
e-Campus Display – Lessons
Learned Details – Part 3

Define the user experience
» Precisely define user experiences – carefully coordinated
performances

Provide transactions
» Transaction level atomicity – because multiple components
involved there should be no partial transitions

Manage expectations
» Testing confused the viewing public. Communicate access
policies. Blank displays create expectation of content to
come

Be accountable
» Entering public domain makes work subject to increased
levels of scrutiny
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
56
Outline


Introduction
Individual
» Wearable Computers
» Augmented Reality

Infrastructure
» Museums
» Hospital
» e-Display

Ad Hoc
» Wildfire
» Volcano

Concluding Lessons
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
57
Monitoring Weather Conditions
in Wildland Fires [Hartung]


Predictions based on current conditions,
previous weather, and predicted weather
Thermal belts, temperature inversions
» Cold air moves into valleys forcing war air to rise which is
trapped by continuously moving air above ridge tops
» Fire stays more active in warmer air with lower relative
humidity

Current approach
» Manual Kit – 5 to 10 minutes to collect at current location
each hour
» Remote Automated Weather Stations – measure temperature,
wind speed/direction, relative humidity, precipitation,
barometric pressure, fuel moisture/temperature, soil
moisture
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
58
Monitoring Weather Conditions
in Wildland Fires -Background
Inversion Layer
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
Manual Measurement
59
Monitoring Weather Conditions
in Wildland Fires -Architecture
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
60
Monitoring Weather Conditions in
Wildland Fires – Lessons Learned 1


Fire activity changes rapidly with even small
changes at low Relative Humidity
Conform to existing infrastructure/logistics
» Powered by AA batteries – quickly accepted since nearly all
electronics devices operate by crews used AA batteries

Battery temperature has significant effect on
battery power available
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
61
Monitoring Weather Conditions in
Wildland Fires – Lessons Learned 2

Housing vents and open bottom
»
»
»
»

Allows airflow but protects from rain
Prevent heat trapping
Painted white to protect from radiation heating
Mount 1.8 meters above ground to protect from heat damage
caused by fire
Large change in elevation increased radio range by
4 to 10 times
» Further from “ground” less interference
» Fresnel Zones - measure of phase difference between reflection of
waves between transmitter and receiver that cause cancelling
effect
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
62
Air Dropped Sensors for
Volcano Monitoring [Song]

Goals
» Synchronized sampling – time synchronized within 1 msec
» Real Time continuous raw data
» One year robust operation – harsh conditions including
heavy rain, snow, ice, wind gusts to 120 mph
» On-line configurable
» Fast deployment – air drop sensors (70 pounds), network
self starting and organizing

Sensors
» Seismic
» Infrasonic
» Lightning – RF pulse detector
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
63
System for Volcano Monitoring
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
64
Air Dropped Sensors for
Volcano Monitoring - Lessons







Signal strength does not reflect signal quality,
rather signal to noise ratio
Problems with connectors
System data delivery 91.7%
Station tipped over, damaged voltage regulator
Infrasonic – pressure sensor also recorded
heavy wind gusts
Outage for 20 hours since data base could not
handle daylight savings time
Large data loss due to low battery voltage
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
65
Outline


Introduction
Individual
» Wearable Computers
» Augmented Reality

Infrastructure
» Museums
» Hospital
» e-Display

Ad Hoc
» Wildfire
» Volcano

Concluding Lessons
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
66
Common Lessons Learned







Demands on User attention
Risk of user becoming engaged in virtual world
and become less aware of physical context
Mobile users impatient
Strong interaction mental model
Fit into current procedures/logistics
Environmental influences
Power especially battery characteristics
© 2010-2013 Daniel P. Siewiorek
Mobil Computing
67