Continuing Work in
Model-Based User Interfaces
Jeffrey Nichols
05-830: Advanced User Interface Software
Last time…

Model-based User Interfaces





Automatic generation of the user interface so the
programmer won’t do a bad job.
Dialog boxes are relatively easy to generate
The full application interface is hard to generate
Abstract descriptions of the interface can be
longer and harder to generate than implementing
the interface itself.
Interface builders turned out to be easier…
Research continued…

Multiple models were integrated



Relational models were developed to explicitly link
other kinds of models
Data models became more detailed
Task models became very important
Research continued…
Fragmented into two different approaches

Software engineering approach (early 90’s-)




Very detailed models to improve overall design process
Intelligent design assistants instead of automatic generation
Significant use of task models
Ubiquitous computing approach (2000-)



Tons of “invisible” processors that perform tasks for us
UIs for these processors are presented on other devices
(mobile phone, PDA, speech, etc.)
Impossible to manually build user interfaces for every
combination
What are task models, anyway?


Description of the process a user takes to reach a
goal in a specific domain
Typically have hierarchical structure


Introduced by GOMS
Number of different task modeling languages



GOMS
UAN
ConcurTaskTrees
ConcurTaskTrees

Developed by Fabio Paterno et al.
for the design of user interfaces

Goals



Graphical for easy interpretation
Concurrent model for representing UI
tasks
Different task types


Represent all tasks, including those
performed by the system
Used almost universally by new
model-based systems
Task Building Process

Three phases



Hierarchically decompose
the tasks
Identify the temporal
relationships among tasks
at same level
Identify what objects are
manipulated and what
actions can be performed
on them, and assign these
to the tasks as appropriate.

Temporal Relationships










T1 [] T2 - Choice
T1 ||| T2 - Interleaving
T1 |[]| T2 - Synchronization
T1 >> T2 - Enabling
T1 []>> T2 - Enabling with
Information Passing
T1 [> T2 - Deactivation
T1* - Iteration
T1(n) - Finite Iteration
[T1] - Optional
T – Recursion
Example

Note: First example is ambiguous
Another Example
Building/Editing Task Models

Tools are available

ConcurTaskTrees
Environment
http://giove.cnuce.cnr.it/ctte.html or Google for “ConcurTaskTrees”
Software Engineering Approach

Lots and lots of systems!


MasterMind
Angel Puerta’s work at Stanford



UIML
Cameleon Project



Mecano, Mobi-D, XIML
Teresa
USIXML
etc…
MasterMind

One of the first system to integrate multiple models together
Mobi-D




Set of tools to support
a clearly defined
development cycle
Uses a series of
different models
Explicit relationships
that specify how the
models are related to
each other
Explicit interaction
between end users
and developers
Mobi-D Models

User-task Model


Domain Model


Describes the human-computer conversation at a low level
Presentation Model


Models of the entities that are manipulated in an interface and their
properties
Dialog Model


Describes tasks the user performs and what interaction capabilities are
needed
Specifies how the interaction objects appear in all of the different states
of the interface.
Relations


Describes how each of the models relate to each other
Tasks/Domain, Dialog/Presentation, Tasks/Dialog, etc.
Mobi-D Process
1. Elicit user tasks


Start with informal description and then convert to outline (basis for task models in
Mobi-D)
More formal task analysis methods could probably be used
2. User-task and domain modeling



Skeleton domain model is built from task outline
Developer refines model
Explicit methods for generalizing pieces of model for reuse in other interface designs
3. Presentation and Dialog design



Decision support tools provide recommendations from model and interface guidelines
(if available)
System steps through task model and helps designer build final interface
By carrying the task model through the process, Mobi-D’s designers believe
users can provide more useful feedback
Mobi-D Discussion

Provides assistance rather than automating design

Recommendations do not limit flexibility, but
organize the design process

For usability engineers, not everyday users



Benefits come from reuse among small projects or for
managing interaction data from a large project
Models can be large and appear to require significant effort
to develop
Spawned a profitable company that does UI work
XIML
eXtensible Interface Markup Language
 Based on Mobi-D work
 Supports full development lifecycle
 Used by RedWhale Software to drive
their interface consultant business

They have developed many tools



move interaction data to/from XIML
Leverage data in XIML to better
understand various interfaces
Automate parts of the interface design
process
Example Use for XIML
Multi-platform interface development
Other Software Engineering Systems

UIML





Originally a research project at Virginia Tech, now being developed commercially
by Harmonia
Goal is platform independent language for describing UI
Early versions were not very platform independent
Recent versions using task models to automatically generate parts of the old
language that were not platform independent
Teresa



(http://www.harmonia.com/)
(http://giove.cnuce.cnr.it/teresa.html)
Transformation Environment for inteRacti
Tool for taking ConcurTaskTrees models, building an abstract interface, and then
building a concrete interface on multiple platforms.
USIXML


(http://www.usixml.org)
Many of the same features of XIML
Novel aspect is the use of graph structure for modeling relations (seems very
complex)
Ubiquitous Computing Approach
“Pervasive computing cannot succeed if every device
must be accompanied by its own interactive software
and hardware…What is needed is a universal
interactive service protocol to which any compliant
interactive client can connect and access any service.”
-Dan Olsen (Xweb paper)
The web comes close to solving this problem, but is
interactively insufficient.
Ubiquitous Computing Approach

There are two problems here:

Infrastructure issues



How do devices communicate?
How do devices discover each other?
User Interfaces issues



Are devices described sufficiently to build a good UI?
How are interfaces generated?
How can one interface be created for controlling
combinations of related devices?
Infrastructure Issues

Possible to investigate these issues without
automatically generating UIs

Being addressed by lots of systems

Commercially


Research


UPnP, JINI, Salutation, HAVi
Speakeasy (PARC), many others…
Most systems that address the UI issues also have
some infrastructure component
Systems addressing UI issues

XWeb


ICrafter


A system for integrating user interfaces from multiple
devices
Supple


Now known as ICE – Interactive Computing Everywhere
A system for automatically generating interfaces with a
focus on customization/personalization.
Personal Universal Controller

My research…
XWeb

Work by Dan Olsen and group at BYU

Premise:


Apply the web metaphor to services in general
Support higher levels of interactivity
XWeb Protocols

Based upon the architecture of the web




XTP Interaction Protocol
Server-side data has a tree structure
Structured Data in XML
URLs for location of objects



xweb://my.site/games/chess/3/@winner
xweb://automate.home/lights/livingroom/
xweb://automate.home/lights/familyroom/-1
XWeb & XTP

CHANGE message (similar to GET in HTTP)

Sequence of editing operations to apply to a sub-tree






Set an attribute’s value
Delete an attribute
Change some child object to a new value
Insert a new child object
Move a subtree to a new location
Copy a subtree to a new location
Platform Independent Interfaces

Two models are specified


DataView – The attributes of the service
XView – A mapping of the attributes into high-level “interactors”

Atomic







Numeric
Time
Date
Enumeration
Text
Links
Aggregation


Group
List
XWeb Example
DataView
XWeb Example
XView
XWeb Example
Interface
Other XWeb Details

Has simple approach for
adjusting to different screen
sizes



Shrink portions of the
interface
Add additional columns of
widgets
Also capable of generating
speech interfaces

Based on a tree traversal
approach like Universal
Speech Interfaces
ICrafter


Part of the Interactive Workspaces research project
at Stanford
Main objective:


“to allow users of interactive workspaces to flexibly interact
with services”
Contribution


An intelligent infrastructure to find services, aggregate them
into a single interface, and generate an interface for the
aggregate service.
In practice, much of the interface generation is done by
hand though automatic generation is supported.
ICrafter Architecture
How is aggregation accomplished?

High-level service interfaces (programmatic)



Data Producer
Data Consumer
The Interface Manager has pattern
generators


Recognize patterns in the services used
Generate interfaces for these patterns

This means that unique functionality will not be available
in the aggregate interface
Automatic Generation in ICrafter
Manual Generation in ICrafter
Supple




Eventual goal is to support automatic
personalization of user interfaces
Treats generation of interfaces as an
optimization problem
Can take into account usage patterns in
generation
Krzysztof Gajos and Daniel S. Weld, “SUPPLE: Automatically Generating User Interfaces” in
Proceedings of Intelligent User Interfaces 2004, Funchal, Portugal.
Modeling Users with Traces

Supple uses traces to keep a usage model

Sequences of events:



<interface element, old value, new value>
Interfaces are rendered taking the traces into
account (though traces are not required)
Trails are segmented at interface close or reset
Generating with Optimization

Uses a branch-andbound search to
explore space of
alternatives

Guaranteed to find
an optimal solution
Optimization Equation
Cost of navigation between
subsequent controls in a trace
Device-specific measure
of how appropriate a
control is for manipulating
a variable of a given type
Total cost for one user trace
Total cost of all traces
Screenshots
Personal Universal Controller


My work with Brad
Problem:


Appliance interfaces are too complex and too
idiosyncratic.
Solution:

Separate the interface from the appliance and use
a device with a richer interface to control the
appliance:

PDA, mobile phone, etc.
Approach
Use mobile devices to control all appliances in the environment
Specifications
Control
Feedback
Appliances
Mobile Devices
Key Features
Two-way communication, Abstract Descriptions, Multiple Platforms,
Automatic Interface Generation
The PUC System Architecture
APPLIANCES
(Stereo, Alarm Clock, etc.)
PUC DEVICES
(automatic interface generation)
ADAPTOR
(publishes description +
appliance state + controls appliance)
PROTOCOL
(two-way communication
of specification & state)
device specification &
state feedback
COMMUNICATION
(802.11, Bluetooth, Zigbee, etc.)
PROTOCOL
(two-way communication
of specification & state)
COMMUNICATION
(802.11, Bluetooth, Zigbee, etc.)
control
Research Approach
1. Hand-design remote control interfaces
2. Determine functional information needed
from appliances to design user interfaces
3. Design language for describing appliance
functions
4. Build interface generators for multiple
platforms
5. Perform user studies to evaluate the
interface generators
Language Design
Informed by hand-designed interfaces
 What functional information was
needed to create interfaces?
Additional Requirements
 Support complete functionality of
appliance
 No specific layout information
 Only one way to specify anything
Full documentation available at: http://www.cs.cmu.edu/~pebbles/puc/
Language Elements
Elements




State variables & commands
Labels
Group tree
Dependency information
Example media player
specification


Play, stop, pause, next track,
previous track
Play list
Language Elements, cont.
State Variables and
Commands

Represent functions of
appliance

State variables have types


Boolean, Enumeration,
Integer, String, etc.
Variables sufficient for most
functions but not all

e.g. “seek” button on a Radio
Language Elements, cont.
Label Information
One label not suitable everywhere


The optimal label length
changes with screen size
Speech interfaces may benefit
from pronunciation and
text-to-speech information
“Label Dictionary”



A group of semantically similar
labels
Different lengths
Information for different
modalities
Language Elements, cont.
Label Information
One label not suitable everywhere


The optimal label length
changes with screen size
Speech interfaces may benefit
from pronunciation and
text-to-speech information
“Label Dictionary”



A group of semantically similar
labels
Different lengths
Information for different
modalities
Language Elements, cont.
Group Tree

Specify organization
of functions

We use n-ary tree
with variables or
commands at leaves

Also used for
specifying complex
types
Lists
Unions
Language Elements, cont.
Group Tree

Specify organization
of functions

We use n-ary tree
with variables or
commands at leaves

Also used for
specifying complex
types
Lists
Unions
Language Elements, cont.
Dependency Information

Formulas that specify when a
variable or command is active
in terms of other state
variables
Equals, Greater Than, Less
Than, Is Defined
Linked with logical operators
(AND, OR)

Allows feedback to user when
a function is not available
Graphical Interface Generator
Rule-based approach
 Multiple phases that
iteratively transform a
specification into a
user interface
Focuses on panel structure of
user interface
 Small groups of controls have
basic layouts
 Complexity comes from
structure of groups
 Structure can be inferred
from dependency info!
Generation Process
1. Determine conceptual layout

Infer panel structure from
dependencies using “mutual
exclusion” property

Choose controls (decision tree)

Choose row layout
(one column, two column, etc.)
2. Allocate space
 Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
Generation Process
1. Determine conceptual layout

Infer panel structure from
dependencies using “mutual
exclusion” property

Choose controls (decision tree)

Choose row layout
(one column, two column, etc.)
2. Allocate space
 Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
Generation Process
1. Determine conceptual layout

Infer panel structure from
dependencies using “mutual
exclusion” property

Choose controls (decision tree)

Choose row layout
(one column, two column, etc.)
2. Allocate space
 Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
Generation Process
1. Determine conceptual layout

Infer panel structure from
dependencies using “mutual
exclusion” property

Choose controls (decision tree)

Choose row layout
(one column, two column, etc.)
2. Allocate space
 Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
Generation Process
1. Determine conceptual layout

Infer panel structure from
dependencies using “mutual
exclusion” property

Choose controls (decision tree)

Choose row layout
(one column, two column, etc.)
2. Allocate space
 Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
Generation Process
1. Determine conceptual layout

Infer panel structure from
dependencies using “mutual
exclusion” property

Choose controls (decision tree)

Choose row layout
Without layout fixing rules
(one column, two column, etc.)
2. Allocate space
 Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
With layout fixing rules
Controlling Appliances
We have built adaptors for many
actual appliances









Sony Digital Camcorder
Windows Media Player
Axis UPnP Pan & Tilt Camera
Lutron Lighting
X10 Lighting
Audiophase Shelf Stereo
AudioReQuest MP3 player
GM Vehicle Information System
GM Vehicle Climate Control
Written specifications for others




Elevator
Telephone/Answering Machine
GM Navigation System
Several Alarm Clocks