Past, Present and Future of
User Interface Software Tools
Brad A. Myers, Scott E. Hudson, and Randy Pausch
Human-Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
http://www.cs.cmu.edu/~bam
bam@cs.cmu.edu
1
Introduction

User Interface Software Tools
 Help
developers design and implement user interfaces
Focus on Tools, but influenced by future UIs
 Today’s tools are highly successful

 Window
Managers, Toolkits, Interface Builders ubiquitous
 Most software built using them
 Are based on HCI research
Brad A. Myers. “A Brief History of Human Computer Interaction Technology.”
ACM interactions. Vol. 5, no. 2, March, 1998. pp. 44-54.
http://www.cs.cmu.edu/~amulet/papers/uihistory.tr.html

Future tools must be different
HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
2
Talk Outline

Historical Perspective
 What
worked
 What didn’t catch on
 Why
 Lessons Learned

Future Prospects and Visions
 UI
Trends that will require new tools
 Important issues
HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
3
Historical Perspective

Themes
Address
the useful & important aspects of UIs
 Tools that succeeded helped (just) where needed
Threshold / Ceiling
 Threshold = How hard to get started
 Ceiling = how much can be achieved
Path of Least Resistance
 Tools influence user interfaces created
Predictability
 If not predictable, then not accepted by programmers
Moving Targets
 Changing user interface styles makes tools obsolete
HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
4
What Worked
Window Managers and Toolkits
 Event Languages
 Graphical, Interactive Tools
 Component Architectures
 Scripting Languages
 Hypertext
 Object Oriented Programming

HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
5
Window Managers
Multiple (tiled) windows in research systems of
1960’s: NLS, etc.
 Overlapping introduced in Alan Kay’s thesis (1969)
 Smalltalk, 1974 at Xerox PARC
 Successful because multiple windows help users
manage scarce resources:

 Screen
space and input devices
 Attention of users
 Affordances for reminding and finding other work
HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
6
Toolkits

A collection of widgets
 Menus,
scroll bars, text entry fields, buttons, etc.
Toolkits help with programming
 Help maintain consistency among UIs

 Key
insight of Macintosh toolbox
Path
of least resistance translates into getting
programmers to do the right thing
 Successful partially because address common,
low-level features for all UIs
Address
HCIC’99
the useful & important aspects of UIs
Myers, Hudson & Pausch
CMU - HCI Institute
7
Event Languages
Create programs by writing event handlers
 Many UIMSs used this style

 Univ.

of Alberta (1985), Sassafras (1986), etc.
Now used by HyperCard, Visual Basic, Lingo, etc.
 Toolkits

with call-backs or action methods are related
Advantages:
 Natural
for GUIs since generate discrete events
 Flow of control in user’s hands rather than programmer’s
 Discourages moded UIs

Won’t work well in future
HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
8
Graphical Interactive Tools

Create parts of user interface by laying out widgets
with a mouse
 Examples:
Menulay (1983), Trillium (1986), Jean-Marie
Hullot from INRIA to NeXT
 Now: Interface Builders, Visual Basic’s layout editor,
resource editors, “constructors”

Advantages:
 Graphical
parts done in an appropriate, graphical way
 Address the useful & important aspects of UIs
 Accessible to non-programmers
 Low threshold
HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
9
Component Architectures

Create applications out of components which are
separately developed and compiled
 In
UI software, each component controls an area of the
screen
 Example: drawing component handles picture inside a
document
Invented by Andrew research project at CMU (1988)
 Now: OLE, OpenDoc, ActiveX, Java Beans
Address the useful & important aspects of UIs

 Just
HCIC’99
the “glue” to hold together components
Myers, Hudson & Pausch
CMU - HCI Institute
10
Scripting Languages

First GUIs used interpreted languages
 Smalltalk,
InterLisp
 Rapid development, supports prototyping
 Low threshold
Then C and C++ became popular
 Now, bringing back advantages in scripting
languages

 tcl/tk,
Python, perl
 Visual Basic, Javascript

But language must contain general-purpose
control structures
HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
11
Hypertext
Ted Nelson named it in 1965 and developed
Hypertext system at Brown University
 Important systems: NLS (1967), Hyperties (1986)
 World-Wide Web

 Phenomenal
success due to:
 Ease of use of Mosaic browser
 Support for embedded graphics
 Support for easy authoring
Low threshold both for authoring and viewing
HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
12
Object Oriented Programming
Success of OO owes much to UI software field
 Popularized by Smalltalk
 GUI elements (widgets) seem like objects

 Have

state, accept events (messages)
Rise parallels GUIs
 C++
with Windows 3.1
 Java for behaviors in WWW
HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
13
What Hasn’t Caught On
User Interface Management Systems
 Formal Language-Based Tools
 Constraints
 Model-Based and Automatic Techniques

HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
14
User Interface Management Systems
Original goal: like databases, provide high-level
language that abstracts details of input and output
devices
 This separation has not worked in practice

 Good
user interfaces must take into account the
pragmatics and detailed behavior of all objects

Standardization of GUI input and output devices
has made goal somewhat moot
Doesn’t
HCIC’99
address the useful & important aspects of UIs
Myers, Hudson & Pausch
CMU - HCI Institute
15
Formal Language Based Tools
Early UIMSs used grammars and state-transition
diagrams
 Focus on dialog management
Moving Targets

 Direct
manipulation made dialog management less
important
Path
of Least Resistance
 State
High
diagrams afford worse user interfaces
threshold
 Formal
HCIC’99
languages are often hard to learn
Myers, Hudson & Pausch
CMU - HCI Institute
16
Constraints
Declare a relationship and system maintains it
 Sketchpad (1963), ThingLab (1979), Higgens (85),
Garnet (1990), Amulet (1997), SubArctic (1996)
Predictability

 Constraint
networks can be hard to debug
 Especially in multi-way constraints
 Programmer must specify (or deduce) solving order
High
threshold
 Constraints

require thinking differently
May be appropriate for graphical layout
Address
HCIC’99
the useful & important aspects of UIs
Myers, Hudson & Pausch
CMU - HCI Institute
17
Model-Based and Automatic Techniques
Automatic techniques for generating UIs from a
model or declarative specification of contents
 Cousin (1985), Mike (1986), UIDE (1993),
MasterMind (1993)
 Try to separate specification of UI from content

 May
provide automatic reformating, retargeting,
customization to users, etc.
Result
is often unpredictable
 Often can be worse UI than hand-drawn
 Sometimes model is larger than the code it would
replace
HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
18
Discussion of Themes
Address
the useful & important aspects of UIs
 Narrower
tools have been more successful than ones
that try to do “everything”
 Do one thing well
Threshold
/ Ceiling
 Research
systems often aim for high ceiling
 Successful systems seem to instead aim for a low
threshold
 Impossible to have both?
HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
19
Discussion of Themes, cont.
Path
of Least Resistance
 Tools
should guide implementers into better user
interfaces
 Goal for the future: do this more?
Predictability
 Programmers
do not seem willing to release control
 Especially when system may do sub-optimal things
Moving
Targets
 Long
stability of Macintosh Desktop paradigm has
enabled maturing of tools
 We predict a change soon
HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
20
Future Prospects and Visions

Important Trends
 Computers
becoming a commodity
 Ubiquitous Computing
 Move to recognition-based interfaces
 3-D interfaces
 End-user customization and scripting

Violate assumptions of today’s tools
 Assumptions
limit what designers can do
 Often unrecognized
 Implications for future tools
HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
21
Computers Becoming a Commodity
There are no longer “high-end” computers
 Computer Science research’s trick of buying faster
computers to “time travel” may no longer be
possible
 Moore’s law continues to operate

 New
opportunities
 Quantitative change makes qualitative change possible
 Enables the diversity of platforms
 UIs more cinematic
 Smooth transitions, animation, visual
HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
22
Ubiquitous Computing
Computation embedded in many kinds of devices
 Digital pagers and cell phones, Palm Pilots,
CrossPads, laptops, wall-size displays, “smart” rooms
 Next wave: easy communication with radio

 E.g.,

BlueTooth: www.bluetooth.com
Significant Implications for tools
 Tools
for coordinating multiple, distributed,
communicating devices
 “Multi-computer” user interfaces
 Moving target problem
HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
23
Varying Input and Output
Today’s Desktop screens vary by a factor of 2.5 in
size and a factor of 4 in pixels
 Tomorrow’s screen will vary by factors of 100 in
size and a factor of 625 in pixels

 Cell
HCIC’99
phone to Stanford’s wall (3796 x 1436 pixels)
Myers, Hudson & Pausch
CMU - HCI Institute
24
Need New Interaction Techniques

Interaction techniques for desktop will not work
 No
room on small devices
 Can’t reach menubar on wall-size devices

Want to run same application on different devices
HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
25
Need for Prototyping Devices
User interface will be in hardware
 Rapid design and prototyping needed for hardware
 Pragmatics and usability cannot be evaluated from
a simulation on a screen

HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
26
Multiple, Distributed, Communicating

Computers more for communication, not for
computation
 Already

true for WWW, email, digital pagers, cell-phones
Computers as intermediaries between people
 CSCW
 But

can’t assume have similar systems
Single person with multiple devices
 Room-area
networks like BlueTooth or HomeRF
 People communicating with themselves

Tools will need to help with data sharing and
synchronization
HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
27
Limitations of Today’s Tools for UbiComp
Tools assume a Pointing Device
 Hidden reliance on specific characteristics of
common devices

 Size
of display
 Many tools cannot handle a different number of mouse
buttons
 Change to a stylus on a touchpad requires different
techniques

Assumptions about the setting
 Assume
user is sitting and looking at UI
 Assume has user’s full attention
HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
28
Move to Recognition-Based Interfaces
Speech, gestures, camera-based vision
 Multimodal interaction

 User
will pick which modality to use
 Use multiple modalities at same time

Today, programming these requires knowing about
Hidden-Markov Models, grammars, feature
vectors, etc.
 Need
HCIC’99
tools to hide these complexities
Myers, Hudson & Pausch
CMU - HCI Institute
29
Fundamental Differences of
Recognition-based UIs

Input is uncertain
 Recognition
can make errors
 Requires monitoring, feedback, correction

Interpreting input requires deep knowledge of data
 Context
of the application
 “Move the red truck to here”
HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
30
Implications of Recognition-based UIs

GUI event model no longer works
 Do
not produce discrete events
Separation of UI from application no longer works
 Need a architecture based on accessible
application data structures

 “Reflection”,
HCIC’99
“Open Data Model”
Myers, Hudson & Pausch
CMU - HCI Institute
31
3-D Interfaces

Difficult to design the right abstractions for tools
 Demise
of VRML for Web
Need to settle on the 3-D widgets and interaction
techniques that will be standard
 Requirement for near-real-time interactivity
 Need to hide the mathematics

HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
32
End-user Customization and Scripting
Spreadsheet enables end users to specify their
own computation
 Visual Basic, other “scripting” languages
 Needed in all applications
 Threshold for programming is too high
 Need “gentle slope systems”

HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
33
Gentle Slope Systems
Programming in C
MFC
Visual Basic
Director (v6)
HyperCard
C Programming
C Programming
Difficulty
of
Use
xCmds
Lingo
HyperTalk
Basic
Goal
Sophistication of what can be created
HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
34
More Assumptions of Today’s Tools

Skill and Dexterity of users
 Older
users
 Makes single, fixed library of widgets untenable

Non-overlapping and opaque components
 Preclude

translucency, magic lens interactions
Fixed libraries of components (widgets)
 Creating
new widgets is very difficult
 New devices will require new interaction techniques

Interactive tools provide freedom of design
 Aim
HCIC’99
for “Mechanism not Policy”
Myers, Hudson & Pausch
CMU - HCI Institute
35
Operating Systems Considerations

What is in the OS?
 Window
Manager? Toolkit? Communication? Scripting
facilities?
Need ever increasing services for applications
 Need more access to low-level information

 E.g.,

hardware buttons, whether on network
Ideally, API to support competition and research
into these components
HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
36
Some Design Guidelines for Future Tools
Many things require further research
 Organize around providing rich context

 Of
application and device state
 To inquire about data & methods; “reflection”
 Enables EUP, Recognition-Based UIs, data sharing for
UbiComp
 Rather than event-based
HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
37
More Design Guidelines

Replaceable User Interfaces
 Ability
to have multiple UIs
 Enabled by procedural interface to everything in UI
 Enables UbiComp devices, EUP

Aim for low threshold, rather than high ceiling
 But
cover the right parts of the interface
Predictable for programmers rather than “smart” or
automatic
 Need for support for evaluation

HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
38
Conclusions
Research in tools necessarily trails innovation in
UI design
 Due to consolidation on desktop metaphor,
significant progress in tools
 UI design poised for radical changes
 New opportunities and challenges for tools

HCIC’99
Myers, Hudson & Pausch
CMU - HCI Institute
39