Studying The Use of Handhelds To
Control Smart Appliances
Jeffrey Nichols
Carnegie Mellon University
May 19, 2003
Jeffrey Nichols • 0 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
The Problem
Appliances are too complex
Jeffrey Nichols • 1 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
The Problem, cont.
 Each complex appliance
has its own idiosyncratic
interface!
• Home and Car Stereos
• Car Navigation Systems
• Answering Machines
•…
 Increasingly
Computerized
 Low Usability
Jeffrey Nichols • 2 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Our Solution
Separate the interface from the appliance!
Specifications
Control
Feedback
Key Features
 User interface-independent appliance specification
 Automatic generation of GUI and speech interfaces
Jeffrey Nichols • 3 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Benefits of Our Approach
 Handheld has richer interface technology than
appliance can afford
 Color LCD screen, touch screen, text entry technology
 More effort can be put into interface design
technology
 Appliance manufacturer’s must weigh trade-offs
between usability, cost, time-to-market, etc.
 Two-way communication channel
 Better feedback can be provided to the user regarding
the appliance’s state.
Jeffrey Nichols • 4 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Automatic Generation of UIs
Benefits
 All interfaces consistent for the user
 With conventions of handheld
Other applications and UI guidelines
 Even from multiple manufacturers
Addresses idiosyncracy problem!
 Multiple modalities (GUI + Speech UI)
 Can take into account user preferences
 Will work on special purpose devices (for
disabled)
Jeffrey Nichols • 5 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Outline
 A First Step
 User Studies
 Current Work
Jeffrey Nichols • 6 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
A First Step
 Build Reference Interfaces
 Remote control interfaces for various
appliances that we design manually.
 Verify that better interfaces can be
created on a handheld
 Used for understanding what
functional knowledge is necessary to
make a good interface.
Jeffrey Nichols • 7 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Reference Interfaces
 Interfaces were hand-designed
for two appliances and two
handhelds
 Appliances
 AIWA CX-NMT70 Shelf Stereo
 AT&T 1825 Telephone/Answering
Machine
 Handhelds
 Palm
 Microsoft PocketPC
Jeffrey Nichols • 8 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Palm Interfaces
We initially designed paper-prototype interfaces
for Palm
telephone
stereo
Jeffrey Nichols • 9 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
PocketPC Interfaces
We later implemented interfaces for Microsoft’s
PocketPC (simulated remote control).
telephone
stereo
Jeffrey Nichols • 10 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Interface Quality?
 We iteratively improved the interfaces
using heuristic analysis techniques.
 We conducted a think-aloud study
with several Carnegie Mellon students
to find problems in the interfaces.
 Lastly, we conducted a user study that
compared our reference interfaces
with the manufacturer’s interfaces.
Jeffrey Nichols • 11 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Outline
 A First Step
 User Studies
 Current Work
Jeffrey Nichols • 12 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
User Studies
 Two Studies
 Study #1:
Paper-Prototype Palm vs. Actual
Appliance
 Study #2:
Functional PocketPC vs. Actual
Appliance
Jeffrey Nichols • 13 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
User Studies, cont.
 Procedure
 We did a between-subjects study.
 Each subject worked on two sets of
tasks.
 In order to minimize subjects, each worked
on both the stereo and the phone.
 We controlled for order and interface
type.
Jeffrey Nichols • 14 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Evaluation of Task Performance
 Three Metrics:
 Time to complete all tasks
 Number of times help was requested
 How often did the subject need the manual
or online help?
 Number of missteps
 Misstep = the pressing of a button that does
not advance the progress on the current
task
 No missteps were counted after the user
requested help.
Jeffrey Nichols • 15 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
User Study #1: PalmOS
 Compared paper prototype
interfaces with the interfaces of
the actual appliances
 Experimenter changed paper
as subjects tapped
 Control of stereo and phone
simulated verbally
 When the stereo started playing music,
the experimenter said “you now hear
music from the stereo”
Jeffrey Nichols • 16 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
User Study #1, cont.
 Participants
 13 Carnegie Mellon Graduate Students






Five female, Eight male
Enrolled in School of Computer Science
Volunteers (unpaid)
Seven owned a Palm device
One had no Palm experience
Four owned Aiwa-brand stereo systems
Jeffrey Nichols • 17 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
User Study #1 Results
Users made five
times the errors
and needed help
twice as often with
the actual
appliances!
All results
significant
(p < 0.001 for all)
Jeffrey Nichols • 18 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
User Study #2
 We implemented the interfaces
on a handheld and simulated
remote control of an actual
appliance.
 Remote control applications built
in Visual Basic on a PocketPC
 Control of stereo and phone
simulated in software
 Feedback appeared to come from the
actual appliance
Jeffrey Nichols • 19 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
User Study #2, cont.
 Participants
 Twelve students from Carnegie Mellon
 Four female, Eight male
 Volunteered in response to a newsgroup
advertisement
 Paid $7 for their participation (30-45 minutes)
 All had limited handheld experience
 Half (6) owned Aiwa-brand stereos
 Two had AT&T digital answering machines
Jeffrey Nichols • 20 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
User Study #2 Results
All differences are significant (p < 0.05)
About ½ the time and ½ the errors!
Jeffrey Nichols • 21 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Qualitative Results
 Why were the reference interfaces
better?
 Clear feedback and explanation of the
current state was possible.
 Elements could be disabled on the
screen (graying out)
 Functions were separated across
multiple screens.
Jeffrey Nichols • 22 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Outline
 A First Step
 User Studies
 Current Work
Jeffrey Nichols • 23 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Current Work
Designed a XML-based Specification Lang
 Functions of Device
State Variables and Commands
 Labeling
Multiple labels are necessary
 Grouping
Hierarchical groups
 Dependency Information
For enabling and structure
Jeffrey Nichols • 24 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Current Work, cont.
 Built multiple
automatic interface
generators
 PocketPC
 SmartPhone
 Tablet PC
(desktop)
 Speech
Jeffrey Nichols • 25 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Control of Real Appliances
Jeffrey Nichols • 26 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Acknowledgements
Funding
 National Science
Foundation
 Microsoft
 General Motors
 Pittsburgh Digital
Greenhouse
Equipment Grants
 Mitsubishi (MERL)
 VividLogic
 Hewlett-Packard
PUC Project Members











Brad A. Myers
Thomas K. Harris
Roni Rosenfeld
Michael Higgins
Joseph Hughes
Kevin Litwack
Rajesh Seenichamy
Mathilde Pignol
Stefanie Shriver
Jeffrey Stylos
Peter Lucas
Jeffrey Nichols • 27 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Thanks!
 For more information see…
 http://www.cs.cmu.edu/~jeffreyn/
 http://www.cs.cmu.edu/~pebbles/puc/
 Or e-mail me at…
 jeffreyn@cs.cmu.edu
Jeffrey Nichols • 28 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Jeffrey Nichols • 29 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Actual Appliance Interfaces
 Lots of Problems
 Poorly labeled and overloaded buttons
 Insufficient feedback
 Timer example
 Programming the speed-dial
 Phone has technical separation between
phone and answering machine
Jeffrey Nichols • 30 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Qualitative Results
 Grouping controls is important
 Groups define which elements are
placed adjacent to each other and
how elements are separated onto
pages.
 Groupings vary between devices
and interface styles.
Jeffrey Nichols • 31 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Qualitative Results, cont.
 Dual-associated functions are
hard to make obvious for users
 The record button is associated
with both tapes (record onto) and
each of the other modes (recorded
from).
 Some users expected the first
mapping to used, whereas the
controller used the second
mapping.
Jeffrey Nichols • 32 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Qualitative Results, cont.
 Tree-based structures are not
sufficient for fully describing an
interface
 Some interface concepts, especially dualassociated functions, break the tree
because they may interact with the
children of several different elements
within the tree
 The record button breaks the stereo’s tree
structure because it is globally accessible
but has different local effects.
Jeffrey Nichols • 33 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Qualitative Results, cont.
 A single function may map to multiple
interface widgets (and vice versa)
 Example: One state variable could be used
to represent all of the playback states of a
tape player. The play, stop, fast-forward,
and rewind buttons all act on this single
variable.
Jeffrey Nichols • 34 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Applying These Results
 We are actively applying these results
to the design of the specification
language
 A tree-grouping structure is augmented
with a dependency graph to help describe
dual-mapped functions
 Ranking relationships within groups using
“priorities”
 We will also apply them in the design
of the automatic layout engine
Jeffrey Nichols • 35 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Future Work
 Build the specification language
and automatic generation engine
Jeffrey Nichols • 36 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
A Hard Problem…
 Automatically generating a good user
interface is hard, but we think we can
do it for several reasons:
 Remote controls are a special class of user
interface that use relatively simple
interaction techniques.
 Buttons, text fields, and other standard
widgets.
 Our approach differs from earlier work…
Jeffrey Nichols • 37 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
The Approach
 Study Interfaces
 Functional knowledge of the appliance
 What must the appliance tell the handheld
about itself so that a “good” interface can
be constructed.

Design and Layout
 How do we turn the knowledge about the
appliance into a usable interface?
 Design a specification language
 Build an automatic interface generator
Jeffrey Nichols • 38 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Our Progress…
 Study Interfaces
 Functional knowledge of the appliance
 What must the appliance tell the handheld
about itself so that a “good” interface can
be constructed.

Design and Layout
 How do we turn the knowledge about the
appliance into a usable interface?
 Design a specification language
(in progress)
 Build an automatic interface generator
Jeffrey Nichols • 39 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003
Problems with User Study #1
 Paper-prototype study introduced a
high possibility of experimenter
interference.
 Solution
 Create an environment that completely
simulates what one might experience using
a personal universal controller
 Interfaces running on an actual handheld
 Interfaces should appear to control an
actual appliance
Jeffrey Nichols • 40 • International Workshop on Smart Appliances and Wearable Computing • May 19, 2003