Recent Work in Model-Based User Interfaces
Jeffrey Nichols
Lecture #13
05-830: Advanced User Interface Software
Last time…

Model-based User Interfaces





Automatic generation of the user interface so the
programmer won’t do a bad job.
Dialog boxes are relatively easy to generate
The full application interface is hard to generate
Abstract descriptions of the interface can be
longer and harder to generate than implementing
the interface itself.
Interface builders turned out to be easier…
But work continued…

Focus Changed



Task models were leveraged more
Design assistant aspect emphasized
A Couple Projects of Interest:




Trident
Mecano & Mobi-D
FUSE
AIDE
TRIDENT

Vanderdonckt, J., Knowledge-Based Systems for Automated User Interface Generation: the
TRIDENT Experience, Technical Report RP-95-010, Facultes Universitaires Notre-Dame de la
Paix, Institut d’Informatique, Namur, 1995.

An interface design assistant
Interesting features:


Knowledge-based approach (i.e. expert system)



Choosing Widgets
Doing Layout
Use of Task Models

Decides where separate windows are needed
Choosing Widgets

Used a decision tree

Chose abstract
interaction objects
(AIO)


Similar to Brad’s
Interactor Model
Lots of parameters



Continuous?
Capacity
Etc.
Choosing Layout
Uses Right/Bottom Strategy


Next component is placed to the right or below
the current component
Decision made by heuristics or designer
Right/Bottom Strategy
Windows from Task Models

Basically used for constructing wizard-like interfaces

What information should be on the first screen, etc.
What are task models, anyway?


Description of the process a user takes to reach a
goal in a specific domain
Typically have hierarchical structure


Introduced by GOMS
Number of different task modeling languages



GOMS
UAN
ConcurTaskTrees
ConcurTaskTrees


Developed by Fabio Paterno et
al. for the design of user
interfaces
Goals



Graphical for easy interpretation
Concurrent model for
representing UI tasks
Different task types

Represent all tasks, including
those performed by the system
Task Building Process

Three phases



Hierarchically decompose
the tasks
Identify the temporal
relationships among tasks
at same level
Identify what objects are
manipulated and what
actions can be performed
on them, and assign these
to the tasks as appropriate.

Temporal Relationships










T1 [] T2 - Choice
T1 ||| T2 - Interleaving
T1 |[]| T2 - Synchronization
T1 >> T2 - Enabling
T1 []>> T2 - Enabling with
Information Passing
T1 [> T2 - Deactivation
T1* - Iteration
T1(n) - Finite Iteration
[T1] - Optional
T – Recursion
Example

Note: First example is ambiguous
Another Example
Building/Editing Task Models

Tools are available

ConcurTaskTrees
Environment
http://giove.cnuce.cnr.it/ctte.html or Google for “ConcurTaskTrees”
Recent Systems

XIML – eXtensible Interface Markup Language



XWeb


Now known as ICE – Interactive Computing Everywhere
ICrafter


Developed by the makers of Mecano/Mobi-D and Trident
Kitchen-sink language for modeling any part of the
interface design process
A system for integrating user interfaces from multiple
devices
Personal Universal Controller

My research…
XIML
eXtensible Interface Markup Language


Designed by RedWhale Software
Intended to support the
full lifecycle of interface
building
XIML Requirements

Central Repository of Data





For one user interface or many
Comprehensive Lifecycle Support
Abstract and Concrete Elements
Relational Support
Underlying Technology

XIML must be independent of particular tools
Models in XIML

An XIML document can contain any type of
model





Task
Domain
User
Presentation
Dialog
Example Use for XIML
Multi-platform interface development
Status of XIML

Used by RedWhale Software to drive their
interface consultant business

They have developed many tools



move interaction data to/from XIML
Leverage data in XIML to better understand various
interfaces
Automate parts of the interface design process
Model-Based Interfaces for Control



XWeb
ICrafter
PUC
XWeb


Work by Dan Olsen and group at BYU
Premise:


“Pervasive computing cannot succeed if every device must
be accompanied by its own interactive software and
hardware…What is needed is a universal interactive
service protocol to which any compliant interactive client
can connect and access any service.”
The web comes close to solving this problem, but is
interactively insufficient.
XWeb Protocols

Based upon the architecture of the web




XTP Interaction Protocol
Server-side data has a tree structure
Structured Data in XML
URLs for location of objects



xweb://my.site/games/chess/3/@winner
xweb://automate.home/lights/livingroom/
xweb://automate.home/lights/familyroom/-1
XWeb & XTP

CHANGE message (similar to GET in HTTP)

Sequence of editing operations to apply to a sub-tree






Set an attribute’s value
Delete an attribute
Change some child object to a new value
Insert a new child object
Move a subtree to a new location
Copy a subtree to a new location
Platform Independent Interfaces

Two models are specified



DataView – The attributes of the service
XView – A mapping of the attributes into high-level “interactors”
Interactors are somewhat like abstract interaction objects

Atomic







Numeric
Time
Date
Enumeration
Text
Links
Aggregation


Group
List
XWeb Example
DataView
XWeb Example
XView
XWeb Example
Interface
Other XWeb Details

Has simple approach for
adjusting to different screen
sizes



Shrink portions of the
interface
Add additional columns of
widgets
Also capable of generating
speech interfaces

Based on a tree traversal
approach like Universal
Speech Interfaces
ICrafter


Part of the Interactive Workspaces research project
at Stanford
Main objective:


“to allow users of interactive workspaces to flexibly interact
with services”
Contribution


An intelligent infrastructure to find services, aggregate them
into a single interface, and generate an interface for the
aggregate service.
In practice, much of the interface generation is done by
hand though automatic generation is supported.
ICrafter Architecture
How is aggregation accomplished?

High-level service interfaces (programmatic)



Data Producer
Data Consumer
The Interface Manager has pattern
generators


Recognize patterns in the services used
Generate interfaces for these patterns

This means that unique functionality will not be available
in the aggregate interface
Automatic Generation in ICrafter
Manual Generation in ICrafter
Personal Universal Controller


My work with Brad
Problem:


Appliance interfaces are too complex and too
idiosyncratic.
Solution:

Separate the interface from the appliance and use
a device with a richer interface to control the
appliance:

PDA, mobile phone, etc.
Idea
Specifications
Control
Feedback


Control existing appliances
Generate multi-modal interfaces
Architecture - Appliance
Comm. Protocol
Interface
Specification
Generators
Adaptors
Lang.
XML-based
Language Design
Approach

Create reference interfaces



Test interfaces with subjects


AIWA Shelf Stereo
AT&T Telephone/Answering
Machine
Users twice as fast and made half
the errors with reference interfaces
as compared to manufacturers’
interfaces
Analyze interfaces for functional
information
Language Elements
State Variables and Commands

Represent functions of appliance

State variables have types


Boolean, Enumeration, Integer, String, etc.
Variables sufficient for most functions but not all

“seek” button on a Radio
Label Information

One label not suitable everywhere


The optimal label length changes with screen size
Speech interfaces may benefit from pronunciation and
text-to-speech information
Language Elements, cont.
Group Tree

Specify organization
of functions

We use n-ary tree
with variables or
commands at
leaves
Language Elements, cont.
Dependency Information

Formulas that specify when a variable or
command is active in terms of other state
variables



Equals, Greater Than, Less Than
Linked with logical operators (AND, OR)
For example,
<and>
<equals state=“PowerState”>true</equals>
<equals state=“RadioBand”>AM</equals>
</and>
Interface Generators
Generators for Two Modalities

Graphical



Implemented for PocketPC in Java 1.1
Uses dependency information to generate panel structure of
interface
Speech


Implemented using Universal Speech Interface (USI)
techniques [Rosenfeld 2001]
Uses dependency information to disambiguate shortcut
words (e.g. “play”) and resolve pre-conditions for a
requested function (e.g. “play CD”)
Graphical Interface Generator
Focuses on panel structure
of user interface

Small groups of controls
have basic layouts

Complexity comes from
structure of groups

Structure can be inferred
from dependency info!
Inferring Structure
Find sets of variables that are
“mutually exclusive”

Every variable in a set will
never be active at the same
time as a variable in another set
Create structure with sets, using
overlapping panels
Choosing Panel Types
a)
b)
c)
full screen
tabbed
partial screen
Making the Interface Concrete
Finish conceptual layout


Choose controls (decision
tree)
Choose row layouts
(one column, two column, etc.)
Allocate space

Examine panel contents and
choose sizes
Instantiate and place controls
Generating Speech Interfaces
Automatically build USI tree from dependencies

Allows verbal navigation of functional groups
Automatically generate grammar for parser

Phrases for query and control
“What is playmode?”
“Set playmode to play”
“play”
Automatically generate language model and
pronunciation for recognizer