Past, Present and Future of
User Interface Software
Tools
Brad A. Myers, Scott E. Hudson, and Randy Pausch
Developed for HCIC’99 and TOCHI
Updated 2009
1

Introduction
User Interface Software Tools



Help developers design and implement user
interfaces
Focus on Tools, but influenced by future UIs
Today’s tools are highly successful



Window Managers, Toolkits, Interface Builders
ubiquitous
Most software built using them
Are based on HCI research
Brad A. Myers. “A Brief History of Human Computer Interaction Technology.”
ACM interactions. Vol. 5, no. 2, March, 1998. pp. 44-54.
http://www.cs.cmu.edu/~amulet/papers/uihistory.tr.html
2
Talk Outline

Historical Perspective





What worked
What didn’t catch on
Why
Lessons Learned
Future Prospects and Visions


UI Trends that will require new tools
Important issues
3
Historical Perspective

Themes

Address the useful & important aspects of UIs


Threshold / Ceiling



Tools influence user interfaces created
Predictability


Threshold = How hard to get started
Ceiling = how much can be achieved
Path of Least Resistance


Tools that succeeded helped (just) where needed
If not predictable, then not accepted by programmers
Moving Targets

Changing user interface styles makes tools obsolete
4
What Worked








Window Managers and Toolkits
Event Languages
Graphical, Interactive Tools
Component Architectures
Scripting Languages
Hypertext
Object Oriented Programming
Constraints
5
Window Managers




Multiple (tiled) windows in research systems of
1960’s: NLS, etc.
Overlapping introduced in Alan Kay’s thesis
(1969)
Smalltalk, 1974 at Xerox PARC
Successful because multiple windows help
users manage scarce resources:



Screen space and input devices
Attention of users
Affordances for reminding and finding other work
6
Toolkits

A collection of widgets



Menus, scroll bars, text entry fields, buttons, etc.
Toolkits help with programming
Help maintain consistency among UIs

Key insight of Macintosh toolbox
 Path

of least resistance translates into getting
programmers to do the right thing
Successful partially because address
common, low-level features for all UIs

Address the useful & important aspects of UIs
7
Event Languages


Create programs by writing event handlers
Many UIMSs used this style


Now used by HyperCard, Visual Basic, Lingo, etc.


Toolkits with call-backs or action methods are related
Advantages:



Univ. of Alberta (1985), Sassafras (1986), etc.
Natural for GUIs since generate discrete events
Flow of control in user’s hands rather than programmer’s
 Discourages moded UIs
May not work well in future
8
Graphical Interactive Tools

Create parts of user interface by laying out
widgets with a mouse



Examples: Menulay (1983), Trillium (1986), JeanMarie Hullot from INRIA to NeXT
Now: Interface Builders, Visual Basic’s layout
editor, resource editors, “constructors”
Advantages:

Graphical parts done in an appropriate, graphical
way


Address the useful & important aspects of UIs
Accessible to non-programmers

Low threshold
9
Component Architectures

Create applications out of components which are
separately developed and compiled


In UI software, each component controls an area of the screen
Example: drawing component handles picture inside a
document
Invented by Andrew research project at CMU (1988)
 1999: OLE, OpenDoc, ActiveX, Java Beans
 Now: SOA
 Address the useful & important aspects of UIs


Just the “glue” to hold together components
10
Scripting Languages

First GUIs used interpreted languages

Smalltalk, InterLisp
Rapid development, supports prototyping
 Low threshold



Then C and C++ became popular
Now, bringing back advantages in scripting
languages



tcl/tk, Python, perl
Visual Basic, Javascript, Ruby, …
But language must contain general-purpose
control structures
11
Hypertext



Ted Nelson named it in 1965 and developed
Hypertext system at Brown University
Important systems: NLS (1967), Hyperties
(1986)
World-Wide Web

Phenomenal success due to:




Ease of use of Mosaic browser
Support for embedded graphics
Support for easy authoring
Low threshold both for authoring and viewing
12
Object Oriented Programming



Success of OO owes much to UI software
field
Popularized by Smalltalk
GUI elements (widgets) seem like objects


Have state, accept events (messages)
Rise parallels GUIs



C++ with Windows 3.1
Java for behaviors in WWW
2009: Flash, etc.
13
Constraints



Declare a relationship and system maintains it
Sketchpad (1963), ThingLab (1979), Higgens (85), Garnet
(1990), Amulet (1997), SubArctic (1996)
1999: hadn’t caught on


Now: Flash data bindings



Connect data to graphics
Address the useful & important aspects of UIs
Predictability



We thought would be mostly used for graphics
Constraint networks can be hard to debug
Especially in multi-way constraints
High threshold


Programmer must specify (or deduce) solving order
Constraints require thinking differently
14
What Hasn’t Caught On



User Interface Management Systems
Formal Language-Based Tools
Model-Based and Automatic Techniques
15
User Interface Management
Systems


Original goal: like databases, provide highlevel language that abstracts details of input
and output devices
This separation has not worked in practice


Good user interfaces must take into account the
pragmatics and detailed behavior of all objects
Standardization of GUI input and output
devices has made goal somewhat moot

Doesn’t address the useful & important aspects of
UIs
16
Formal Language Based Tools
Early UIMSs used grammars and statetransition diagrams
 Focus on dialog management
 Moving Targets


Direct manipulation made dialog management
less important
 Path

State diagrams afford worse user interfaces
 High

of Least Resistance
threshold
Formal languages are often hard to learn
17
Model-Based and Automatic
Techniques



Automatic techniques for generating UIs from a
model or declarative specification of contents
Cousin (1985), Mike (1986), UIDE (1993),
MasterMind (1993)
Try to separate specification of UI from content

May provide automatic reformating, retargeting,
customization to users, etc.
 Result


is often unpredictable
Often can be worse UI than hand-drawn
Sometimes model is larger than the code it
would replace
18
Discussion of Themes
 Address
the useful & important aspects of
UIs


Narrower tools have been more successful than
ones that try to do “everything”
Do one thing well
 Threshold



/ Ceiling
Research systems often aim for high ceiling
Successful systems seem to instead aim for a low
threshold
Impossible to have both?
19
Discussion of Themes, cont.

Path of Least Resistance



Predictability




Programmers do not seem willing to release control
Especially when system may do sub-optimal things
Moving Targets


Tools should guide implementers into better user interfaces
Goal for the future: do this more?
Long stability of Macintosh Desktop paradigm has enabled
maturing of tools
1999: We predicted a change soon
2009?
20
Future Prospects and Visions

Important Trends





Ubiquitous Computing
Move to recognition-based interfaces
3-D interfaces
End-user customization and scripting
Violate assumptions of today’s tools



Assumptions limit what designers can do
Often unrecognized
Implications for future tools
21
Ubiquitous Computing



Computation embedded in many kinds of devices
Digital pagers and cell phones, Palm Pilots,
CrossPads, laptops, wall-size displays, “smart”
rooms
Next wave: easy communication with radio


E.g., BlueTooth: www.bluetooth.com
Significant Implications for tools

Tools for coordinating multiple, distributed,
communicating devices


“Multi-computer” user interfaces
Moving target problem
22
Varying Input and Output


Today’s Desktop screens vary by a factor of 2.5 in
size and a factor of 4 in pixels
Tomorrow’s screen will vary by factors of 100 in size
and a factor of 625 in pixels

Cell phone to Stanford’s wall (3796 x 1436 pixels)
23
Need New Interaction
Techniques

Interaction techniques for desktop will not
work




No room on small devices
Can’t reach menubar on wall-size devices
2009: iPhone doesn’t use desktop metaphors
Want to run same application on different
devices
24
Need for Prototyping Devices



User interface will be in hardware
Rapid design and prototyping needed for
hardware
Pragmatics and usability cannot be evaluated
from a simulation on a screen
25
Multiple, Distributed,
Communicating

Computers more for communication, not for
computation


Computers as intermediaries between people



CSCW
But can’t assume have similar systems
Single person with multiple devices



Already true for WWW, email, digital pagers, cell-phones
Room-area networks like BlueTooth or HomeRF
People communicating with themselves
Tools will need to help with data sharing and
synchronization
26
Limitations of Today’s Tools
for UbiComp


Tools assume a Pointing Device
Hidden reliance on specific characteristics of
common devices




Size of display
Many tools cannot handle a different number of
mouse buttons
Change to a stylus on a touchpad requires
different techniques
Assumptions about the setting


Assume user is sitting and looking at UI
Assume has user’s full attention
27
Move to Recognition-Based
Interfaces


Speech, gestures, camera-based vision
Multimodal interaction



User will pick which modality to use
Use multiple modalities at same time
Today, programming these requires knowing
about Hidden-Markov Models, grammars,
feature vectors, etc.

Need tools to hide these complexities
28
Fundamental Differences of
Recognition-based UIs

Input is uncertain



Recognition can make errors
Requires monitoring, feedback, correction
Interpreting input requires deep knowledge of
data


Context of the application
“Move the red truck to here”
29
Implications of Recognition-based UIs

GUI event model no longer works



Do not produce discrete events
Separation of UI from application no longer
works
Need a architecture based on accessible
application data structures

“Reflection”, “Open Data Model”
30
3-D Interfaces

Difficult to design the right abstractions for
tools





Demise of VRML for Web
Need to settle on the 3-D widgets and
interaction techniques that will be standard
Requirement for near-real-time interactivity
Need to hide the mathematics
2009: but what useful for?
31
End-user Customization and
Scripting





Spreadsheet enables end users to specify
their own computation
Visual Basic, other “scripting” languages
Needed in all applications
Threshold for programming is too high
Need “gentle slope systems”
32
Gentle Slope Systems
Programming in C
MFC
Visual Basic
Flash
HyperCard
C Programming
C Programming
Difficulty
of
Use
xCmds
ActionScript
HyperTalk
Basic
Goal
Sophistication of what can be created
33
More Assumptions of Today’s
Tools

Skill and Dexterity of users



Non-overlapping and opaque components


Preclude translucency, magic lens interactions
Fixed libraries of components (widgets)



Older users
Makes single, fixed library of widgets untenable
Creating new widgets is very difficult
New devices will require new interaction
techniques
Interactive tools provide freedom of design

Aim for “Mechanism not Policy”
34
Operating Systems
Considerations

What is in the OS?



Need ever increasing services for
applications
Need more access to low-level information


Window Manager? Toolkit? Communication?
Scripting facilities?
E.g., hardware buttons, whether on network
Ideally, API to support competition and
research into these components
35
Some Design Guidelines for
Future Tools


Many things require further research
Organize around providing rich context




Of application and device state
To inquire about data & methods; “reflection”
Enables EUP, Recognition-Based UIs, data
sharing for UbiComp
Rather than event-based
36
More Design Guidelines

Replaceable User Interfaces




Aim for low threshold, rather than high ceiling



Ability to have multiple UIs
Enabled by procedural interface to everything in
UI
Enables UbiComp devices, EUP
But cover the right parts of the interface
Predictable for programmers rather than
“smart” or automatic
Need for support for evaluation
37
Conclusions




Research in tools necessarily trails innovation
in UI design
Due to consolidation on desktop metaphor,
significant progress in tools
UI design poised for radical changes
New opportunities and challenges for tools
38