Lecture 14:
Model-based tools:
Creating the UI Automatically
Brad Myers
05-830
Advanced User Interface Software
© 2013 - Brad Myers
1
Schedule of Lectures



Everyone OK?
See also instructions off the homeworks page
Right away: Check to make sure there are 23 articles for each person
© 2013 - Brad Myers
2
Model-Based Tools

Overview

Programmer describes the operation of the
system or the user interface in a specification
language = the "model".



model is a high-level description
usually declarative (listing parts and relationships)
System automatically creates the interface

Uses a low-level toolkit for the widgets
© 2013 - Brad Myers
3
Goals:




High-level description of an interface is easier to write than low-level
toolkit code
Automatic generation may produce better UIs than programmers
Allow separation of UI design (embodied in rules) from UI contents
(supplied by the programmer)
Support dynamic creation of objects



define templates or prototypes
Increase re-use since design rules shared by multiple applications
Tools can reason over the specification to produce extra stuff:




Automatic generation of help, undo, etc.
Transform interface into different but functionally equivalent interface
Enabling and disabling of widgets
Enforcement or checking of design guidelines- consistency,
completeness



Enforces consistency since rules will pick similar objects for similar situations
Automatic adjustment to different screen sizes, etc., since rules can take
this into account
Automatic analysis for quality

NGOMSL analysis : GLEAN (Kieras, UIST'95)
© 2013 - Brad Myers
4
Overview, cont.

Related to the "Declarative" approach discussed
in previous lecture


but here system has some intelligence or knowledge
so less has to be specified by the programmer.
Different types:



Dialog box creators: Mickey, DON, Jade (lots of
others)
Representations of the full UI: ITS, UIDE, Humanoid,
MasterMind,
Newer: (XML-based) ConcurTaskTrees, Xweb,
ICrafter, Supple, PUC
© 2013 - Brad Myers
5
Dialog Box Creators


Easiest part of the UI to create
Given a list of the contents, automatically:
1. choose widgets:

specify type of desired input:
string = text input field
number = slider
one-of-many = radio buttons or pop-up options
many-of-many = check boxes or checks in a menu
commands = menu
© 2013 - Brad Myers
6
Dialog Box Creators, cont.
2. arrange widgets

based on look-and-feel guidelines



where OK goes
which commands go in which menus
based on good graphic design principles.
3. set variables

to reduce the number of callbacks necessary
© 2013 - Brad Myers
7
Example: Mickey




Programmer specifies UI by
putting special comments in a
Pascal file.
Uses the Apple Macintosh
guidelines
Pre-processor to parse the
Pascal code and generate the
Macintosh resources.
Maps Procedures into Menu
items.

If parameter is one of a standard
set, pops up appropriate dialog
box or waits for input


File to be read, file to be written
New point, line or rectangle
8
© 2013 - Brad Myers
Mickey, cont.

Variables:



Enumerated types mapped to
check lists separated by lines.
Sets the variables when
changed.
Enumerated types with 2
choices mapped to name
changes
Booleans: single checked items
© 2013 - Brad Myers
9
Jade



Brad Vander Zanden and Brad A. Myers, "Automatic, Look-and-Feel
Independent Dialog Creation for Graphical User Interfaces,"
Proceedings SIGCHI'90: Human Factors in Computing
Systems. Seattle, WA, April 1-5, 1990. pp. 27-34. ACM DL Reference
"Judgment-based Automatic Dialog Editor"
Given a textual specification of just the
contents and their types, creates a dialog box



Separately specify which look-and-feel
(not part of the specification)
Defines mapping from types to widget
selection
Graphic design rules for "nice" layout
© 2013 - Brad Myers
10
DON

(Won Chul Kim & Foley, InterCHI'93, pp. 430-437)





Ultimate in dialog box layout
Kim's PhD thesis
Works with OpenLook and devGuide
Allows interactive designer guidance (preferences) on
sizes, layout, widget choice, etc.


ACM DL Reference
Can also choose among proposed layouts
Sophisticated 2-D layout




Tries to balance dialog box
Groupings of related items
Effective use of white space (even margins, minimize wasted
space)
Generates multiple designs and uses an evaluation metric to
choose.
© 2013 - Brad Myers
11
Don, pictures

PDF, with other pictures
© 2013 - Brad Myers
12
Generating Full UI



These next tools require a specification of the
full UI
Usually have rule-based components
Specifications are in a special language
© 2013 - Brad Myers
13
Interactive Transaction System
(ITS)




Bennett, et.al., UIST'89 pp. 67-75
Wiecha, et.al. CHI'89, pp. 277-282
Wiecha, et.al., ACM TOIS, 8(3), Jul'90, pp. 204-236
Goal: capture designers knowledge as style rules


So unlike other systems, designer is required to edit the rules, not
just the specification
All of UI must be created by editing the rules


no interactive editing of generated interface (since then the knowledge
about why the generated interface wasn't good enough would be lost)
Like dialog-box systems, separate specification of content
and style



Style-independent tags associated with content
"Style expert" programs the style for each tag
Styles include both output (display) and input (interaction techniques)
14
specifications
© 2013 - Brad Myers
ITS, cont.



Can handle dialog boxes, forms, node-link diagrams,
kiosk frames, etc.
Used for a number of internal IBM applications
Used for all the information services at Expo'90 in Spain



Evaluation




Information, maps, restaurant reservations, etc.
IBM researchers and content experts were in Spain for months
+ Full representation of design may increase re-use
- Design specification ends up containing many specific "hacks"
used to achieve specific effects in single interfaces
- Complex formal language for specification and rules
Pictures from ITS, TOIS, 8(3), pp. 213, 215, 217
© 2013 - Brad Myers
15
ITS, pictures

PDF
© 2013 - Brad Myers

16

The User Interface Design
Environment (UIDE)




Foley, et. al. CHI'88, pp. 67-72
Foley, et. al. IEEE Software, Jan'89, 25-32;
Sukaviriya, et. al. InterCHI'93, pp. 375-382
Long-term project of Foley's at George
Washington and Georgia Tech


Ended about 1994 when Foley left
A loose collection of separate implementations:





IDL's transformations
DON dialog boxes (described above)
Sukaviriya's animated help
Martin Frank's work (EET in Event-Based lecture)
© 2013 - Brad Myers
- etc.
17
UIDE

Programmer defines Knowledge-base
"schemas" describing parts of the interface:



Pre-conditions and post-conditions are in a very
limited language



Objects: in a class, sub-class inheritance hierarchy
Actions: what can be done in the interface
counting, booleans, simple tests
used for testing enabled and explaining why
Transformations change among equivalent UIs:


e.g. Currently-selected obj <=> currently-selected cmd
performed based on pre-, post-conditions
© 2013 - Brad Myers
18
Pictures from UIDE
© 2013 - Brad Myers
19
UIDE, cont.

Automatic generate help for why commands are
not available


Sukaviriya, et. al. AVI’94, Pages: 44 - 52
Animated help provides animations as a tutorial



Determines what needs to be done to demonstrate action
Sequence of actions
E.g. bring windows to the front, create an object, etc.
© 2013 - Brad Myers
20
UIDE

Evaluation


+ Support for more than dialog boxes
- Pre and post condition language is weak


can't express the test "if the selected object is a
polygon..."
- Model language is a different, difficult language
to learn
© 2013 - Brad Myers
21
Humanoid






Szekely, et. al. UIST'90, pp. 1-9
Szekely, et. al. CHI'92, pp. 507-514
Szekely, et. al. InterCHI'93, pp. 383-390
High-level UIMS for Manufacturing
Applications Needing Organized Iterative
Development
Model application data and interaction similar
to UIDE
Model whole application: semantics +
interface
© 2013 - Brad Myers
22
Humanoid, cont.


System picks generic interaction techniques
immediately using "templates"
Designer can refine interface iteratively by
creating more specific sub-classes:




Single-Command-Input-With-Alternatives
Single-Command-Input-With-Few-Alternatives
Allows exploration with incomplete designs
Evaluation


+ Much richer specification language than UIDE
- More complex to define interfaces (more to learn)

but interactive tools help
© 2013 - Brad Myers
23
Humanoid Pictures
Pictures from Humanoid,
CHI'93 pp. 384
© 2013 - Brad Myers
24
MasterMind




Neches, et. al. ACM 1993 Intelligent User Interfaces Workshop, pp.
63-70
Models Allowing Shared Tools and Explicit Representations to Make
Interfaces Natural to Develop
Idea: combine UIDE and Humanoid
Support entire life-cycle: early conceptual design through
maintenance






Knowledge base is shared among all tools
Knowledge base serves as an integrating framework for various
tools at design time and run time.
Spent a lot of time negotiating on how to combine models
Lots of different parts to the model
Personnel and coordination problems in doing the research
Used Amulet!
© 2013 - Brad Myers
25
MasterMind

One of the first system to integrate multiple models
together
© 2013 - Brad Myers
26
Mastermind
Pictures
© 2013 - Brad Myers
27
Research continued…
Fragmented into two different approaches

Software engineering approach (early 90’s-)




Very detailed models to improve overall design process
Intelligent design assistants instead of automatic generation
Significant use of task models
Ubiquitous computing approach (2000-)



Tons of “invisible” processors that perform tasks for us
UIs for these processors are presented on other devices
(mobile phone, PDA, speech, etc.)
Impossible to manually build user interfaces for every
combination
© 2013 - Brad Myers
28
What are task models, anyway?



Key part of many current HCI approaches
Description of the process a user takes to reach a
goal in a specific domain
Typically have hierarchical structure


Introduced by GOMS
Number of different task modeling languages



GOMS, CogTool
UAN
ConcurTaskTrees
© 2013 - Brad Myers
29
ConcurTaskTrees

Developed by Fabio Paterno et al.
for the design of user interfaces

Goals



Graphical for easy interpretation
Concurrent model for representing UI
tasks
Different task types

Represent all tasks, including those
performed by the system
© 2013 - Brad Myers
30
Building/Editing Task Models
Tools are available


ConcurTaskTrees
Environment
http://giove.cnuce.cnr.it/ctte.html or Google for “ConcurTaskTrees”

© 2013 - Brad Myers
31
Software Engineering Approach



Commercial work
“Model-based design”
Example: BPMN “business
process modeling notation”



Business experts should be
able to author models
Converted into code to
support the process (requires people)
Keynote at ICSE’08: Herbert Hanselmann: Challenges in
Automotive Software Engineering

“Model-based design (MBD) of functional behaviour has been a
big help in the recent past”
© 2013 - Brad Myers
32
Mobi-D





Angel Puerta, IUI’97
Set of tools to support
a clearly defined
development cycle
Uses a series of
different models
Explicit relationships
that specify how the
models are related to
each other
Explicit interaction
between end users
and developers
© 2013 - Brad Myers
33
XIML
eXtensible Interface Markup Language
 XIML.org
 Based on Mobi-D work
 Supports full development lifecycle
 Used by RedWhale Software to drive
their interface consultant business

They have developed many tools



move interaction data to/from XIML
Leverage data in XIML to better
understand various interfaces
Automate parts of the interface design
process
© 2013 - Brad Myers
34
Other Systems

UIML





Originally a research project at Virginia Tech, now being developed commercially
by Harmonia
Goal is platform independent language for describing UI
Early versions were not very platform independent
Recent versions using task models to automatically generate parts of the old
language that were not platform independent
Teresa



(http://www.harmonia.com/)
(http://giove.cnuce.cnr.it/teresa.html)
Transformation Environment for inteRacti
Tool for taking ConcurTaskTrees models, building an abstract interface, and then
building a concrete interface on multiple platforms.
USIXML


(http://www.usixml.org)
Many of the same features of XIML
Novel aspect is the use of graph structure for modeling relations (seems very
complex)
© 2013 - Brad Myers
35
Ubiquitous Computing
Approach
“Pervasive computing cannot succeed if every device
must be accompanied by its own interactive software
and hardware…What is needed is a universal
interactive service protocol to which any compliant
interactive client can connect and access any service.”
-Dan Olsen (Xweb paper)
The web comes close to solving this problem, but is
interactively insufficient.
© 2013 - Brad Myers
36
Systems addressing UI issues

XWeb


ICrafter


A system for integrating user interfaces from multiple
devices
Supple


Now known as ICE – Interactive Computing Everywhere
A system for automatically generating interfaces with a
focus on customization/personalization.
Personal Universal Controller

Jeff Nichol’s research…
© 2013 - Brad Myers
37
XWeb

Work by Dan Olsen and group at BYU
E.g. UIST’2000, pp.191 - 200

Premise:



Apply the web metaphor to services in general
Support higher levels of interactivity
© 2013 - Brad Myers
38
XWeb Example
Interface
© 2013 - Brad Myers
39
ICrafter



Part of the Interactive Workspaces research project
at Stanford
Ponnekanti, et. al. Ubicomp’2001
Main objective:


“to allow users of interactive workspaces to flexibly interact
with services”
Contribution


An intelligent infrastructure to find services, aggregate
them into a single interface, and generate an interface for
the aggregate service.
In practice, much of the interface generation is done by
hand though automatic generation is supported.
© 2013 - Brad Myers
40
Supple




Eventual goal is to support automatic
personalization of user interfaces
Treats generation of interfaces as an
optimization problem
Can take into account usage patterns in
generation
Krzysztof Gajos and Daniel S. Weld, “SUPPLE: Automatically Generating User Interfaces” in
Proceedings of Intelligent User Interfaces 2004, Funchal, Portugal.
© 2013 - Brad Myers
41
Modeling Users with Traces

Supple uses traces to keep a usage model

Sequences of events:



<interface element, old value, new value>
Interfaces are rendered taking the traces into
account (though traces are not required)
Trails are segmented at interface close or reset
© 2013 - Brad Myers
42
Generating with Optimization

Uses a branch-andbound search to
explore space of
alternatives

Guaranteed to find
an optimal solution
© 2013 - Brad Myers
43
Screenshots
© 2013 - Brad Myers
44
Personal Universal Controller


Jeff Nichol’s PhD work
Problem:


Appliance interfaces are too complex and too
idiosyncratic.
Solution:

Separate the interface from the appliance and use
a device with a richer interface to control the
appliance:


PDA, mobile phone, etc.
Goal: Generate high-quality UIs
© 2013 - Brad Myers
45
Approach
Use mobile devices to control all appliances in the environment


Specifications

Control

Feedback
Appliances


Mobile Devices
Key Features
Two-way communication, Abstract Descriptions, Multiple Platforms,
Automatic Interface Generation
© 2013 - Brad Myers
46
Properties of PUC Language

State variables & commands

Each can have multiple labels


Typed variables



Useful when not enough room
Base types: Boolean, string,
enumerated, integers,
fixed-point, floating-point, etc.
Optional labels for values
Hierarchical Structure

Groups
© 2013 - Brad Myers
47
Dependency Information


Crucial for high-quality interfaces
Expressed as <active-if> clauses

Operations:


Combined Logically


Equals, Less-Than,
Greater-Than
AND, OR
Used for:



Dynamic graying out
Layout
Widget selection
© 2013 - Brad Myers
48
Generating Consistent UIs



Personally consistent
Reduce learning time
Add unique functions
© 2013 - Brad Myers
49
Generating Combined UIs



For multiple appliances, such as home
theaters
Specify content flow
Combined controls
© 2013 - Brad Myers
50
Summative Study



Compared PUC to manufacturer’s
interfaces for HP and Canon
printer/fax/copiers
PUC twice as fast, 1/3 the errors
Consistent: another factor of 2 faster
18
16
1200
Number of Failures
Average Time (sec)
1400
1000
800
600
400
14
12
10
200
8
6
4
2
0
HP
HP-PUC
HPConsistent
Canon
© 2013 - Brad Myers
Canon-PUC
CanonConsistent
0
HP
HP-PUC
HPConsistent
Canon
Canon-PUC
51
CanonConsistent
Model-based systems
advantages/disadvantages



+ Separate specification of UI from content
+ Automatic reformatting, retargeting for different
platforms, customization to users
+ May allow programmers (non-experts) to write
specification and have a good UI automatically created





But this didn’t really work out
– Result is often unpredictable
– Often UI can be worse UI than hand-drawn
– Sometimes model is larger than the code it would
replace
– Model often in a different language that must be
learned
© 2013 - Brad Myers
52