Describing Appliance User Interfaces
Abstractly with XML
Jeffrey Nichols
Workshop on Developing User Interfaces with XML
Advanced Visual Interfaces
May 25, 2004
Gallipoli, Italy
Problem
Appliances are complex and their user interfaces are often hard to use!
1
Solution
Separate the user interface from the appliance
Specifications
Control
Feedback
Appliances
Interface Devices
We call this approach the “Personal Universal Controller” (PUC)
2
Language Goals
Inform with hand-designed interfaces
•
What functional information is needed to
create interfaces?
Additional Requirements
•
•
•
•
Support complete functionality of
appliance
No specific presentation information
Short and concise
Easy to author
“only one way to specify anything”
•
Sufficient detail for generating high
quality interfaces without assistance
Full documentation and XML Schema available at: http://www.cs.cmu.edu/~pebbles/puc/
3
Language Elements
Elements
•
•
•
•
•
State variables & commands
Labels
Group tree
Dependency information
Smart Templates
Example media player specification
•
•
Play, stop, pause, next track,
previous track
Play list
4
Smart Templates
A means to specify high-level information
Solution
•
Mark groups with tags that identify high-level
information
media-controls, phone-dialpad, time, date,
etc.
•
Restrict the contents of groups so that
interface generators can interpret the highlevel meaning
•
Standardize the tags and restrictions in
advance, so that designers know what
interface generators expect
•
Parameterize the tags so one Smart
Template can be used for multiple
appliances and situations
5
Smart Templates
A means to specify high-level information
Solution
•
Mark groups with tags that identify high-level
information
media-controls, phone-dialpad, time, date,
etc.
•
Restrict the contents of groups so that
interface generators can interpret the highlevel meaning
•
Standardize the tags and restrictions in
advance, so that designers know what
interface generators expect
•
Parameterize the tags so one Smart
Template can be used for multiple
appliances and situations
6
Actual Appliances
We have written specifications for many actual appliances
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Sony Digital Camcorder
Windows Media Player
Axis UPnP Pan & Tilt Camera
Lutron Lighting
X10 Lighting
Audiophase Shelf Stereo
AudioReQuest MP3 player
GM Vehicle Information System
GM Vehicle Climate Control
Elevator
Telephone/Answering Machine
Several Alarm Clocks
Task Manager (To Do List)
GM Navigation System (in progress)
Mitsubishi DVCR (in progress)
Used to build apps for Intel’s “Personal Server”
Used by group at TU Vienna for controlling AR apps
7
Interface Generators
Four Interface Generators for
Two Modalities
Graphical
•
Desktop, PocketPC, and
Microsoft Smartphone
Speech
•
•
•
Ph.D. work of Thomas Harris
Built on top of the PUC
framework
Implemented using Universal
Speech Interface (USI)
techniques [Rosenfeld 2001]
8
Language Analysis
Pros
•
Useful for generating both graphical and speech interfaces
•
Concise
•
Easily learned by multiple undergraduates
•
Capable of describing many existing appliances
•
Has enough detail to generate usable interfaces without designer
intervention
Cons
•
No task information
Though some “flow” can be specified using dependencies
•
Describes appliance functions, not appliance user interface
•
Designed for appliance descriptions, not desktop applications
9
Acknowledgements
Funding
•
•
•
•
National Science Foundation
Microsoft
General Motors
Pittsburgh Digital Greenhouse
Equipment Grants
•
•
•
•
•
•
•
Mitsubishi (MERL)
VividLogic
Symbol Technologies
Hewlett-Packard
Lucent
Lutron
Lantronix
PUC Project Members
•
•
•
•
•
•
•
•
•
•
•
•
•
Brad A. Myers
Kevin Litwack
Thomas K. Harris
Roni Rosenfeld
Michael Higgins
Joseph Hughes
Rajesh Seenichamy
Mathilde Pignol
Stefanie Shriver
Pegeen Shen
Jeffrey Stylos
Suporn Pongnumkul
Peter Lucas
10
Thanks for listening!
For more information…
http://www.cs.cmu.edu/~pebbles/puc/
http://www.cs.cmu.edu/~jeffreyn/
Describing Appliance User Interfaces
Abstractly with XML
Jeffrey Nichols
Workshop on Developing User Interfaces with XML
Advanced Visual Interfaces
May 25, 2004
Gallipoli, Italy
More Screenshots
13
Future Work
Further develop Smart Templates idea
Add task information
•
•
Focus on “distributed task information” among multiple appliances
Generate one combined user interface for multiple connected appliances
(e.g. home theaters)
Ensure interface consistency for similar appliances
Evaluate generated interfaces and specification language
•
User studies, etc.
14
Research Approach
1. Hand-design remote control interfaces
2. Determine functional information needed
from appliances to design user interfaces
3. Design language for describing appliance
functions
4. Build interface generators for multiple
platforms
5. Perform user studies to evaluate the
interface generators
15
Benefits of Automatic Generation
•
Devices not pre-programmed with appliance details
•
Multiple modalities (GUI + Speech UI)
•
All interfaces consistent for a user
•
•
•
With conventions of the handheld
Even from multiple manufacturers
Create user interfaces that control multiple connected
appliances
e.g. a home theater
16
Initial Approach
What information is needed about the
appliance to automatically generate
remote control interfaces?
Investigate via a design process
Create interfaces by hand
AIWA Shelf Stereo
• AT&T Telephone/Answering Machine
•
Improve quality with heuristic analysis and
think-aloud studies with several users
Compare interfaces with actual appliance
interfaces to validate PUC concept
Analyze interfaces for functional information
17
Palm Interfaces
Initially designed paper-prototype interfaces for Palm
telephone
stereo
18
Hand-Designed Interfaces
Interfaces for Microsoft’s PocketPC (simulated remote control)
telephone
stereo
19
Comparison Study
Compared performance of first-time users (not experts)
Appliance
Hand-design
Phone
Stereo
Procedure
•
Each subject worked two sets of tasks
both stereo and phone
controlled for order and interface
Performance Metrics
•
Time to complete all tasks
•
Number of times a user manual was needed
•
Number of missteps
20
Comparison Study Results
Using our interfaces, users were twice as fast
and made half as many errors
All differences are significant (p < 0.05)
21
Architecture
APPLIANCES
(Stereo, Alarm Clock, etc.)
PUC DEVICES
(automatic interface generation)
ADAPTOR
(publishes description +
appliance state + controls appliance)
PROTOCOL
(two-way communication
of specification & state)
device specification &
state feedback
COMMUNICATION
(802.11, Bluetooth, Zigbee, etc.)
PROTOCOL
(two-way communication
of specification & state)
COMMUNICATION
(802.11, Bluetooth, Zigbee, etc.)
control
22
Language Elements, cont.
State Variables and Commands
•
Represent functions of
appliance
•
State variables have types
•
•
Boolean, Enumeration, Integer,
String, etc.
Variables sufficient for most
functions but not all
•
e.g. “seek” button on a Radio
23
Language Elements, cont.
Label Information
One label not suitable everywhere
•
•
The optimal label length changes
with screen size
Speech interfaces may benefit from
pronunciation and
text-to-speech information
“Label Dictionary”
•
•
•
A group of semantically similar labels
Different lengths
Information for different modalities
24
Language Elements, cont.
Label Information
One label not suitable everywhere
•
•
The optimal label length changes
with screen size
Speech interfaces may benefit from
pronunciation and
text-to-speech information
“Label Dictionary”
•
•
•
A group of semantically similar labels
Different lengths
Information for different modalities
25
Language Elements, cont.
Group Tree
•
Specify organization of
functions
•
We use n-ary tree with
variables or commands
at leaves
•
Also used for specifying
complex types
Lists
Unions
26
Language Elements, cont.
Group Tree
•
Specify organization of
functions
•
We use n-ary tree with
variables or commands
at leaves
•
Also used for specifying
complex types
Lists
Unions
27
Language Elements, cont.
Dependency Information
• Formulas
that specify when a
variable or command is active
in terms of other state
variables
Equals, Greater Than, Less Than,
Is Defined
Linked with logical operators
(AND, OR)
• Allows
feedback to user when a
function is not available
28
Smart Templates Example
29
Graphical Interface Generator
Rule-based approach
•
Multiple phases that iteratively
transform a specification into a
user interface
Focuses on panel structure of
user interface
•
•
•
Small groups of controls have
basic layouts
Complexity comes from structure
of groups
Structure can be inferred from
dependency info!
30
Generation Process
1. Determine conceptual layout
•
Infer panel structure from
dependencies using “mutual
exclusion” property
•
Choose controls (decision tree)
•
Choose row layout
(one column, two column, etc.)
2. Allocate space
•
Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
31
Generation Process
1. Determine conceptual layout
•
Infer panel structure from
dependencies using “mutual
exclusion” property
•
Choose controls (decision tree)
•
Choose row layout
(one column, two column, etc.)
2. Allocate space
•
Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
32
Generation Process
1. Determine conceptual layout
•
Infer panel structure from
dependencies using “mutual
exclusion” property
•
Choose controls (decision tree)
•
Choose row layout
(one column, two column, etc.)
2. Allocate space
•
Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
33
Generation Process
1. Determine conceptual layout
•
Infer panel structure from
dependencies using “mutual
exclusion” property
•
Choose controls (decision tree)
•
Choose row layout
(one column, two column, etc.)
2. Allocate space
•
Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
34
Generation Process
1. Determine conceptual layout
•
Infer panel structure from
dependencies using “mutual
exclusion” property
•
Choose controls (decision tree)
•
Choose row layout
(one column, two column, etc.)
2. Allocate space
•
Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
35
Generation Process
1. Determine conceptual layout
•
Infer panel structure from
dependencies using “mutual
exclusion” property
•
Choose controls (decision tree)
•
Choose row layout
(one column, two column, etc.)
Without layout fixing rules
2. Allocate space
•
Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
With layout fixing rules
36
High-Level Conventions
Problem
•
Human designers
rely partly on
conventions when
making an
interface
•
Users expect their
appliances to use
conventions they
know about
37
Smart Templates
Need a way to specify high-level
information to interface generators
Solution
•
Mark groups with tags that identify highlevel information
media-controls, phone-dialpad, time,
date, etc.
•
Restrict the contents of groups so that
interface generators can interpret the
high-level meaning
•
Standardize the tags and restrictions in
advance, so that designers know what
interface generators expect
38
Smart Templates, cont.
Features
Parameterized
• Specified using primitive
elements of specification
language
•
39
Continuing Work
Define and implement Smart Templates
•
•
date, mute, power, time-absolute, volume, etc.
Develop more as more appliances are specified
Combinations of templates
•
•
Less implementation cost than a new template
e.g. date and time-absolute
Use of templates with data already on controller device
•
•
e.g. calendar and address information
Might allow user to enter address from contact list into
navigation system
40
Smart Templates, cont.
One template can be used across multiple appliances and can be
generated on multiple platforms
41