Lecture 13:
Continuing Work in
Model-Based User Interfaces
Brad Myers
Slides originally authored by Jeffrey Nichols, 2004
05-830: Advanced User Interface Software
Last time…

Model-based User Interfaces





Automatic generation of the user interface so the
programmer won’t do a bad job.
Dialog boxes are relatively easy to generate
The full application interface is hard to generate
Abstract descriptions of the interface can be
longer and harder to generate than implementing
the interface itself.
Interface builders turned out to be easier…
Research continued…

Multiple models were integrated



Relational models were developed to explicitly link
other kinds of models
Data models became more detailed
Task models became very important
Research continued…
Fragmented into two different approaches

Software engineering approach (early 90’s-)




Very detailed models to improve overall design process
Intelligent design assistants instead of automatic generation
Significant use of task models
Ubiquitous computing approach (2000-)



Tons of “invisible” processors that perform tasks for us
UIs for these processors are presented on other devices
(mobile phone, PDA, speech, etc.)
Impossible to manually build user interfaces for every
combination
What are task models, anyway?



Key part of many current HCI approaches
Description of the process a user takes to reach a
goal in a specific domain
Typically have hierarchical structure


Introduced by GOMS
Number of different task modeling languages



GOMS
UAN
ConcurTaskTrees
ConcurTaskTrees

Developed by Fabio Paterno et al.
for the design of user interfaces

Goals



Graphical for easy interpretation
Concurrent model for representing UI
tasks
Different task types


Represent all tasks, including those
performed by the system
Used almost universally by new
model-based systems
Task Building Process

Three phases



Hierarchically decompose
the tasks
Identify the temporal
relationships among tasks
at same level
Identify what objects are
manipulated and what
actions can be performed
on them, and assign these
to the tasks as appropriate.

Temporal Relationships










T1 [] T2 - Choice
T1 ||| T2 - Interleaving
T1 |[]| T2 - Synchronization
T1 >> T2 - Enabling
T1 []>> T2 - Enabling with
Information Passing
T1 [> T2 - Deactivation
T1* - Iteration
T1(n) - Finite Iteration
[T1] - Optional
T – Recursion
Example

Note: First example is ambiguous


T1 [] T2 - Choice
T1 ||| T2 - Interleaving
Another Example




T1 [] T2 – Choice
T1 >> T2 - Enabling
T1 []>> T2 - Enabling with
Information Passing
T1 [> T2 – Deactivation
Building/Editing Task Models

Tools are available

ConcurTaskTrees
Environment
http://giove.cnuce.cnr.it/ctte.html or Google for “ConcurTaskTrees”
Software Engineering Approach

Lots and lots of systems!



Commercial work
MasterMind
Angel Puerta’s work at Stanford



UIML
Cameleon Project



Mecano, Mobi-D, XIML
Teresa
USIXML
etc…
Software Engineering Approach



Commercial work
“Model-based design”
Example: BPMN “business
process modeling notation”



Business experts should be
able to author models
Converted into code to
support the process (requires people)
Keynote at ICSE’08: Herbert Hanselmann: Challenges in
Automotive Software Engineering

“Model-based design (MBD) of functional behaviour has been a
big help in the recent past”
MasterMind
Neches, et.al., IUI’93
 One of the first system to integrate multiple models
together

Mobi-D





Angel Puerta, IUI’97
Set of tools to support
a clearly defined
development cycle
Uses a series of
different models
Explicit relationships
that specify how the
models are related to
each other
Explicit interaction
between end users
and developers
Mobi-D Models

User-task Model


Domain Model


Describes the human-computer conversation at a low level
Presentation Model


Models of the entities that are manipulated in an interface and their
properties
Dialog Model


Describes tasks the user performs and what interaction capabilities are
needed
Specifies how the interaction objects appear in all of the different states
of the interface.
Relations


Describes how each of the models relate to each other
Tasks/Domain, Dialog/Presentation, Tasks/Dialog, etc.
Mobi-D Process
1. Elicit user tasks


Start with informal description and then convert to outline (basis for task models in
Mobi-D)
More formal task analysis methods could probably be used
2. User-task and domain modeling



Skeleton domain model is built from task outline
Developer refines model
Explicit methods for generalizing pieces of model for reuse in other interface designs
3. Presentation and Dialog design



Decision support tools provide recommendations from model and interface guidelines
(if available)
System steps through task model and helps designer build final interface
By carrying the task model through the process, Mobi-D’s designers believe
users can provide more useful feedback
Mobi-D Discussion

Provides assistance rather than automating design

Recommendations do not limit flexibility, but
organize the design process

For usability engineers, not everyday users



Benefits come from reuse among small projects or for
managing interaction data from a large project
Models can be large and appear to require significant effort
to develop
Spawned a profitable company http://www.redwhale.com/
that does UI work
XIML
eXtensible Interface Markup Language
 XIML.org
 Based on Mobi-D work
 Supports full development lifecycle
 Used by RedWhale Software to drive
their interface consultant business

They have developed many tools



move interaction data to/from XIML
Leverage data in XIML to better
understand various interfaces
Automate parts of the interface design
process
Example Use for XIML
Multi-platform interface development
Other Systems

UIML





Originally a research project at Virginia Tech, now being developed commercially
by Harmonia
Goal is platform independent language for describing UI
Early versions were not very platform independent
Recent versions using task models to automatically generate parts of the old
language that were not platform independent
Teresa



(http://www.harmonia.com/)
(http://giove.cnuce.cnr.it/teresa.html)
Transformation Environment for inteRacti
Tool for taking ConcurTaskTrees models, building an abstract interface, and then
building a concrete interface on multiple platforms.
USIXML


(http://www.usixml.org)
Many of the same features of XIML
Novel aspect is the use of graph structure for modeling relations (seems very
complex)
Ubiquitous Computing
Approach
“Pervasive computing cannot succeed if every device
must be accompanied by its own interactive software
and hardware…What is needed is a universal
interactive service protocol to which any compliant
interactive client can connect and access any service.”
-Dan Olsen (Xweb paper)
The web comes close to solving this problem, but is
interactively insufficient.
Ubiquitous Computing
Approach

There are two problems here:

Infrastructure issues



How do devices communicate?
How do devices discover each other?
User Interfaces issues



Are devices described sufficiently to build a good UI?
How are interfaces generated?
How can one interface be created for controlling
combinations of related devices?
Infrastructure Issues

Possible to investigate these issues without
automatically generating UIs

Being addressed by lots of systems

Commercially


Research


UPnP, JINI, Salutation, HAVi
Speakeasy (PARC), many others…
Most systems that address the UI issues also have
some infrastructure component
Systems addressing UI issues

XWeb


ICrafter


A system for integrating user interfaces from multiple
devices
Supple


Now known as ICE – Interactive Computing Everywhere
A system for automatically generating interfaces with a
focus on customization/personalization.
Personal Universal Controller

Jeff Nichol’s research…
XWeb

Work by Dan Olsen and group at BYU
E.g. UIST’2000, pp.191 - 200

Premise:



Apply the web metaphor to services in general
Support higher levels of interactivity
XWeb Protocols

Based upon the architecture of the web




XTP Interaction Protocol
Server-side data has a tree structure
Structured Data in XML
URLs for location of objects



xweb://my.site/games/chess/3/@winner
xweb://automate.home/lights/livingroom/
xweb://automate.home/lights/familyroom/-1
XWeb & XTP

CHANGE message (similar to GET in HTTP)

Sequence of editing operations to apply to a sub-tree






Set an attribute’s value
Delete an attribute
Change some child object to a new value
Insert a new child object
Move a subtree to a new location
Copy a subtree to a new location
Platform Independent
Interfaces

Two models are specified


DataView – The attributes of the service
XView – A mapping of the attributes into high-level “interactors”

Atomic







Numeric
Time
Date
Enumeration
Text
Links
Aggregation


Group
List
XWeb Example
DataView
Xweb
Example
XView
XWeb Example
Interface
Other XWeb Details

Has simple approach for
adjusting to different screen
sizes



Shrink portions of the
interface
Add additional columns of
widgets
Also capable of generating
speech interfaces

Based on a tree traversal
approach like Universal
Speech Interfaces
ICrafter



Part of the Interactive Workspaces research project
at Stanford
Ponnekanti, et. al. Ubicomp’2001
Main objective:


“to allow users of interactive workspaces to flexibly interact
with services”
Contribution


An intelligent infrastructure to find services, aggregate
them into a single interface, and generate an interface for
the aggregate service.
In practice, much of the interface generation is done by
hand though automatic generation is supported.
ICrafter Architecture
How is aggregation
accomplished?

High-level service interfaces (programmatic)



Data Producer
Data Consumer
The Interface Manager has pattern
generators


Recognize patterns in the services used
Generate interfaces for these patterns

This means that unique functionality will not be
available in the aggregate interface
Automatic Generation in
ICrafter
Manual Generation in ICrafter
Supple




Eventual goal is to support automatic
personalization of user interfaces
Treats generation of interfaces as an
optimization problem
Can take into account usage patterns in
generation
Krzysztof Gajos and Daniel S. Weld, “SUPPLE: Automatically Generating User Interfaces” in
Proceedings of Intelligent User Interfaces 2004, Funchal, Portugal.
Modeling Users with Traces

Supple uses traces to keep a usage model

Sequences of events:



<interface element, old value, new value>
Interfaces are rendered taking the traces into
account (though traces are not required)
Trails are segmented at interface close or reset
Generating with Optimization

Uses a branch-andbound search to
explore space of
alternatives

Guaranteed to find
an optimal solution
Optimization Equation
Cost of navigation between
subsequent controls in a trace
Device-specific measure
of how appropriate a
control is for manipulating
a variable of a given type
Total cost for one user trace
Total cost of all traces
Screenshots
Personal Universal Controller


Jeff Nichol’s PhD work
Problem:


Appliance interfaces are too complex and too
idiosyncratic.
Solution:

Separate the interface from the appliance and use
a device with a richer interface to control the
appliance:

PDA, mobile phone, etc.
Approach
Use mobile devices to control all appliances in the environment
Specifications
Control
Feedback
Appliances
Mobile Devices
Key Features
Two-way communication, Abstract Descriptions, Multiple Platforms,
Automatic Interface Generation
The PUC System Architecture
APPLIANCES
(Stereo, Alarm Clock, etc.)
PUC DEVICES
(automatic interface generation)
ADAPTOR
(publishes description +
appliance state + controls appliance)
PROTOCOL
(two-way communication
of specification & state)
device specification &
state feedback
COMMUNICATION
(802.11, Bluetooth, Zigbee, etc.)
PROTOCOL
(two-way communication
of specification & state)
COMMUNICATION
(802.11, Bluetooth, Zigbee, etc.)
control
Automatic Generation of UIs

Benefits

All interfaces consistent for a user


With conventions of the handheld
Even from multiple manufacturers




Addresses hotel alarm clock problem
Can take into account user preferences
Multiple modalities (GUI + Speech UI)
A Hard Problem

Previous automatic systems have not
generated high quality interfaces
PUC Specification Language


XML
Full documentation for the
specification language and
protocol:
http://www.pebbles.hcii.cmu.edu/puc/

Contains sample
specification for a stereo
Properties of PUC Language

State variables & commands

Each can have multiple labels


Typed variables



Useful when not enough room
Base types: Boolean, string,
enumerated, integers,
fixed-point, floating-point, etc.
Optional labels for values
Hierarchical Structure

Groups
Dependency Information


Crucial for high-quality interfaces
Expressed as <active-if> clauses

Operations:


Combined Logically


Equals, Less-Than,
Greater-Than
AND, OR
Used for:



Dynamic graying out
Layout
Widget selection
Specifications

Have working specifications for:









Audiophase stereo
X-10 lights control
Sony CamCorder
Windows Media Player
Audio ReQuest hardware MP3 player
WinAmp Media Player
Elevator
Parts of GMC Yukon Denali SUV
Etc.
Controller Generators

iPaq PocketPC

SmartPhone

No touchscreen

Desktop (TabletPC)

Speech
Generating Speech Interfaces

“Universal Speech Interface” (USI) project



Creates grammar, language model and
pronunciation dictionary from PUC specification




Prof. Roni Rosenfeld of CMU
http://www.cs.cmu.edu/~usi
Pronunciation from labels using phonetic rules
Can provide other pronunciations as labels for fine-tuning
Will use dependency information to help with
disambiguation and explanation
Supports queries and spoken feedback

Paraphrases as confirmation
Adaptors

“Adaptors” provide the interface to existing
(and future) appliances
If do not support specification language directly
Custom hardware
 AV/C (standard protocol)
 Sony CamCorder
Custom software
 Lutron
 HAVi
 Windows Media Player
 UPnP
X-10
 Axis Camera





Light switches, etc.
Generating Consistent UIs



Personally consistent
Reduce learning time
Add unique functions
Generating Combined UIs



For multiple appliances, such as home
theaters
Specify content flow
Combined controls
Summative Study



Compared PUC to manufacturer’s
interfaces for HP and Canon
printer/fax/copiers
PUC twice as fast, 1/3 the errors
Consistent: another factor of 2 faster
18
16
1200
Number of Failures
Average Time (sec)
1400
1000
800
600
400
14
12
10
8
6
4
200
2
0
0
HP
HP-PUC
HPConsistent
Canon
Canon-PUC
CanonConsistent
HP
HP-PUC
HPConsistent
Canon
Canon-PUC
CanonConsistent
Details of the Language Design
Informed by hand-designed interfaces
 What functional information was
needed to create interfaces?
Additional Requirements
 Support complete functionality of
appliance
 No specific layout information
 Only one way to specify anything
Full documentation available at: http://www.cs.cmu.edu/~pebbles/puc/
Language Elements
Elements




State variables & commands
Labels
Group tree
Dependency information
Example media player
specification


Play, stop, pause, next track,
previous track
Play list
Language Elements,
cont.
State Variables and
Commands

Represent functions of
appliance

State variables have types


Boolean, Enumeration,
Integer, String, etc.
Variables sufficient for most
functions but not all

e.g. “seek” button on a Radio
Language Elements, cont.
Label Information
One label not suitable everywhere


The optimal label length
changes with screen size
Speech interfaces may benefit
from pronunciation and
text-to-speech information
“Label Dictionary”



A group of semantically similar
labels
Different lengths
Information for different
modalities
Language Elements,
cont.
Label Information
One label not suitable everywhere


The optimal label length
changes with screen size
Speech interfaces may benefit
from pronunciation and
text-to-speech information
“Label Dictionary”



A group of semantically similar
labels
Different lengths
Information for different
modalities
Language Elements, cont.
Group Tree

Specify organization
of functions

We use n-ary tree
with variables or
commands at leaves

Also used for
specifying complex
types
Lists
Unions
Language Elements,
cont.
Group Tree

Specify organization
of functions

We use n-ary tree
with variables or
commands at leaves

Also used for
specifying complex
types
Lists
Unions
Language Elements,
cont.
Dependency Information

Formulas that specify when a
variable or command is active
in terms of other state
variables
Equals, Greater Than, Less
Than, Is Defined
Linked with logical operators
(AND, OR)

Allows feedback to user when
a function is not available
Graphical Interface Generator
Rule-based approach
 Multiple phases that
iteratively transform a
specification into a
user interface
Focuses on panel structure of
user interface
 Small groups of controls have
basic layouts
 Complexity comes from
structure of groups
 Structure can be inferred
from dependency info!
Generation Process
1. Determine conceptual layout

Infer panel structure from
dependencies using “mutual
exclusion” property

Choose controls (decision tree)

Choose row layout
(one column, two column, etc.)
2. Allocate space
 Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
Generation Process
1. Determine conceptual layout

Infer panel structure from
dependencies using “mutual
exclusion” property

Choose controls (decision tree)

Choose row layout
Without layout fixing rules
(one column, two column, etc.)
2. Allocate space
 Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
With layout fixing rules