User Interface Toolkits and End User
Programming for UIs
Vishal Dwivedi
Institute for Software Research
Carnegie Mellon University
vdwivedi@cs.cmu.edu
Advanced User Interface Software
1
Carnegie Mellon University, School of Computer Science
“End-user programming enables end users to create
their own programs. Today, millions of end users
create numerous programs [Scaffidi et al, 2005]
And they make a mess out of it 
Christopher Scaffidi, Mary Shaw, Brad A. Myers: Estimating the Numbers of End Users and End User
Programmers. VL/HCC 2005: 207-214
2
Carnegie Mellon University, School of Computer Science
Agenda


Motivation/Definitions
Examples (of EUP toolkits)



Historical trends
Key techniques


Alice, SUIT, Garnet, HANDS, SWiFT, …
VPL, PBD, Natural Programming, Application
specific languages,…
Challenges
3
Carnegie Mellon University, School of Computer Science
Definitions

“Programming”
‘‘The process of transforming a mental plan of desired actions for
a computer into a representation that can be understood by the
computer’’
– Jean-Michel Hoc and Anh Nguyen-Xuan


“End-User Programming”

Programming by people who write programs, but not as their
primary job function.

Instead, they must write programs in support of achieving their
main goal, which is something else.

Covers a wide range of programming expertise

Business analysts, Neuroscientists, Physicists, Teachers, Accountants,
etc.
4
Carnegie Mellon University, School of Computer Science
Definitions

“End User Programming toolkits”


An assembly of tools; set of basic building units that help end
users do end user programming, mostly by manipulating or
creating graphical user interfaces.
Examples:

Toolkits for:

Learning to write code

Creating web pages
Simulations
Email filtering
Other user-created functionality.



5
Carnegie Mellon University, School of Computer Science
Major Techniques

“Visual Programming”



“Programming by Demonstration”


“Programming in which more than one dimension* is used to
convey semantics.”
- Myers, 1990
A programming language that lets users create programs by
manipulating program elements graphically rather than by
specifying them textually
- Golin , 1990
An end user development technique for teaching a computer or a
robot new behaviors by demonstrating the task to transfer directly
instead of programming it through machine commands.
Dan Halbert, 1984
“Natural Programming”

Set of techniques to make making programming languages and
environments easier to learn, more effective, and less error
prone.
- Myers, 19906
* Multi dimensional graphical objects, spatial relationships, time relationships (before/after)
Example Toolkits
Alice
Carnegie Mellon University, School of Computer Science
[Randy Pausch, 2000]
7
Youtube Video
Example Toolkits
Garnet

One of the earliest GUI
development toolkit

Aimed at tools to help
designers create, modify and
maintain highly interactive,
graphical, direct manipulation
interfaces.
Carnegie Mellon University, School of Computer Science
[Myers, 1988]
8
Example Toolkits
SUIT

[Randy Pausch, 1992]
Provides a collection of screen objects described by its:




Carnegie Mellon University, School of Computer Science
state
a C procedure that displays it
a C procedure that handles input and updates the object state
Similar toolkits : UIMX, Interviews, Motif widgets
Pausch, R., Conway, M., & DeLine, R. (1992). Lesson Learned from SUIT, the Simple User Interface
Toolkit. ACM Transactions on Information Systems, 10(4), 320-344.
9
Example Toolkits
HANDS
Carnegie Mellon University, School of Computer Science
[J.F. Pane, 2002]
10
J.F. Pane, "A Programming System for Children that is Designed for Usability," Ph.D. Thesis, Carnegie Mellon
Example Toolkits
Carnegie Mellon University, School of Computer Science
Ozone – a publish-subscribe based widget composition and creation toolkit

A popular framework that enables rapid creation,
assembly and configuration of rich, lightweight widgets.

Promoted to create a common widget repository for the US
military – similar to ‘Android Market’.
11
Link
Example Toolkits
Carnegie Mellon University, School of Computer Science
Rule based programming (Agent Sheet, MOSL, VISPATCH, SAM, VXT, …)

Programming Paradigm for
Programming with Pictures,
particularly visual Simulations and
rules.

Rule based design to:






animate agents
play sounds and MIDI instruments
speak text
react to mouse and keyboard events
visualize values as colors and plots
query web pages, control web
browsers
Agent Sheet: http://www.agentsheets.com
12
Example Toolkits
Carnegie Mellon University, School of Computer Science
Dataflow based (PROGRAPH, ObjectFlow, The Cube Language, SWiFT,…)
Method case



General-purpose Data-flow
driven languages
Information flows through
components
Data is transformed at
certain points in the program
Operations
13
Example Toolkits
Carnegie Mellon University, School of Computer Science
SWiFT– a web-based toolkit web service composition (Neuroscience example)
/usr/local/fsl/bin/flirt
ALIGN
-ref standard
-in example_func
-out example_func2standard
-omat example_func2standard.mat
-cost corratio -dof 12
-searchrx -90 90
-searchry -90 90
-searchrz -90 90
-interp trilinear
A large
SPATIAL
script file
FILTERING
that
contains
program
calls
TEMPORAL
Program
(a large number of binaries
that
FILTERING
perform one or more functions)
Parameters (numbers range from 5 to 25)
14
Historical trends
o AMBIT/G/L
o Grail
o GAL
o Graphical Program
Editor
o Query by Example
o Pygmalion
o I/O Pairs
1960
Carnegie Mellon University, School of Computer Science
o Action
Graphics
o Forms
o FORMAL
o Editing by
Example
o ThingLab
o PICT
o Hi-Visual
o Lotus 1-2-3
o LabView
o PROGRAPH o SIL-ICON
o VisiCalc
o PIGS
o HiGraphs
o Pict
o Rehearsal o Miro
o StateMaster
o SmallStar
1980
Techniques
Techniques
o Graphs
o Flowcharts
o Flowchart derivatives
o FORMS
o Demonstrational
o Graphs
o Flowcharts
o Flowchart
derivatives
o FORMS
o Demonstrational
o Data Flows
o Spreadsheets
o Matrices
o Jigsaw Puzzles
o Petri nets
o Flowchart
derivatives
o Cube
o AVS
o Cantata
o Mondrian
o SchemePaint o ChemTrains
o CODE 2.0
o Vampire
o Iconicode
o VIPR
o MViews
o SPE
1990
o LOFI/HIPI
o FOXQ
o VMQL
o GXL
o Euler View
o Yahoo Pipes
o Popfly
2000
Techniques/Goals
Techniques/Goals
o 3D Rendering
o Visual Hierarchy
o Procedures
o Control Structures
o Programmable Graphics
o Animations
o Video Imagery Exploitation
o General purpose, declarative language
o Audio, video and image processing
o Graphical models from behavioral models
o Learning and Cognitive abilities in vision
processes
o Handling Scalability, typing, and
imperative design
o Collaborative Software Development
o Child Learning
o Xquery by FORMS
o Spreadsheet Analysis
o Visual Model Query
o Layouts
o Specification and Interchange
o Mashups
o Web-based design
o Programming for end-users
(non-Professionals
15
Carnegie Mellon University, School of Computer Science
Historical trends
o Let users program in
Visual Languages
o Support the
cognition
aspect of
Programming
o Strive for improvements
in programming
language design
o Make programming
more accessible
o Natural programming
o Support domain specific designs
o (Almost) Make textual
languages redundant
1960
1980
1990
2000
16
Key Techniques
Carnegie Mellon University, School of Computer Science
Programming by demonstration

“Programming by Demonstration”


[Dan Halbert, 1984]
An end user development technique for teaching a computer or a
robot new behaviors by demonstrating the task to transfer directly
instead of programming it through machine commands.
Many toolkits based on this technique:

COCOA, HyperCard, Grammax, Mail Filtering
17
Halbert, Dan (November 1984). Programming by Example. U.C. Berkeley (PhD dissertation)
Key Techniques
Carnegie Mellon University, School of Computer Science
Visual Programming by demonstration

“Visual Programming”


“Programming in which more than one dimension* is used to
convey semantics.”
- Myers, 1990
Many, many toolkits based on this technique.


Myers’ Taxonomy
Burnett’s Taxonomy
- Myers, 1990
- Burnett, 1994
Brad Myers, Taxonomies of Visual Programming and Program Visualization, 1990
Margaret Burnett, A Classification System for Visual Programming Languages, 1994.
18
Key Techniques
Carnegie Mellon University, School of Computer Science
Natural Programming

“Natural Programming”


Set of techniques to make making programming languages and
environments easier to learn, more effective, and less error
prone.
- Myers, 1990 - Present
“Brad Myers Natural Programming project”

http://www.cs.cmu.edu/~NatProg/index.html
19
Key Techniques
Carnegie Mellon University, School of Computer Science
Direct Manipulation

“Direct Manipulation”


A technique that involves users to directly manipulate objects
presented to them, using actions that correspond at least loosely
to the physical world.
“Central Ideas”




Visual Model of the world
Visual objects that can be operated on
Results of actions are reflected in the objects immediately.
Objects, once operated on, can be further operated on.
- Shneiderman, 1983
20
Shneiderman, Ben. "Direct Manipulation. A Step Beyond Programming Languages" (HTML). IEEE Transactions on Computers
Key Techniques
Carnegie Mellon University, School of Computer Science
Application/Domain specific Languages

“Application/Domain specific Languages”



A popular trend is to provide toolkits that allow end users to
create domain/application specific platforms.
Many of them are VPL platforms
Examples:





Dataflow based: Yahoo Pipes
Pub-sub based (widget composition): Ozone, Synapse
Service composition based: Loni Pipeline, Taverna
Ontology based: WINGS
Scripting based: TCL-TK, various domain-specific scripts
21
Challenges
Carnegie Mellon University, School of Computer Science
Understanding the end user ecosystem
Scientists,
Physicists,
Astronomists
Accountants,
Moms and Pops
People with Professional end
user software developer role
[Segal, 07]
Domains that
involve writing a
lot of code


Domains that involve
adaptations of software, and
change in configurations
People with a
(technically)
novice role
Domains that
involve using or
adapting turnkey software
The goal of End user programming is to support their professional task.
But it is not always that they perform all these roles themselves.
22
Challenges
Carnegie Mellon University, School of Computer Science
Understanding the end user ecosystem…

“Multiple roles in
Component based software
development”



Framework builders,
Component developers,
Application assemblers, and
end users
Additional role: end user
developers – mainly involved
with tailoring the applications.
Successful tools like Taverna,
WINGS, LONI-pipeline are
built on such ecosystems.
Morch, A., Stevens, G., et al. (2004) Component-based technologies for end-user development.
Communications of the ACM, 47(9), pp. 59-62
23
Challenges
Carnegie Mellon University, School of Computer Science
Reuse is hard

Supporting reuse of existing end user code requires
efficient packaging and search.

Current work*:


Package new capability as plugins
[Scaffidi 2008, Sestoff 2002]
Finding resources through hierarchy of repositories from personal to
local to global
[Scaffidi, 2009]
*mostly in context of spreadsheets
24
Challenges
Carnegie Mellon University, School of Computer Science
Testing, verification and analysis is even harder (and hardly available)

Most toolkits generate or select code as a result of user
interaction.


For example in the SUIT toolkit:



Users typically have limited abilities to make changes in that code.
Creating custom widgets was hard
Teaching callbacks was not possible
Studies have demonstrated that programs produced
using various end user programming toolkits are
erroneous up to the level of 90% [Leventhal et al. 1994,
Stolee et al. 2010]

There is limited support to analyze and fix such errors.
25
Challenges
Carnegie Mellon University, School of Computer Science
High Cost and low quality end user programming tools

Many end user programming tools are developed from
scratch, leading to:



High cost of development, and
Low quality
Instead of many tools, what is needed is frameworks to
support:




Component level reuse
Testing and analysis
Execution support, and finally
Packaging and sharing
26
Next…
Carnegie Mellon University, School of Computer Science
We discuss an approach called End User
Architecting that is aimed to address some of
these challenges…
27
Carnegie Mellon University, School of Computer Science
References









Agent Sheet: http://www.agentsheets.com
Ben Shneiderman. "Direct Manipulation. A Step Beyond Programming Languages"
(HTML). IEEE Transactions on Computers
Brad A. Myers, John F. Pane and Andy Ko, "Natural Programming Languages and
Environments". Communications of the ACM. 47(9), pp. 47-52.
Brad A. Myers. "Taxonomies of Visual Programming and Program Visualization," Journal of
Visual Languages and Computing. vol. 1, no. 1. March, 1990. pp. 97-123.
Christopher Scaffidi, Christopher Bogart, Margaret M. Burnett, Allen Cypher, Brad A.
Myers, Mary Shaw: Predicting reuse of end-user web macro scripts. VL/HCC 2009: 93-100
Conway, M., Audia, S., Burnette, T., Cosgrove, D., Christiansen, K., Deline, R., et al.
(2000, Apr 1-6). Alice: Lessons Learned from Building a 3D System For Novices. Paper
presented at the Proceedings of
CHI 2000, The Hague, The Netherlands.
David Garlan, Bradley Schmerl, Vishal Dwivedi, Aparup Banerjee, Laura Glendenning, Mai
Nakayama, and Nina Patel. Swift: A tool for constructing workflows for dynamic network
analysis. http://acme.able.cs.cmu.edu/pubs/show.php?id=333, 2011.
F.H. Post and W. Barth, Construction Techniques of Graphic, Direct-Manipulation User
Interfaces, EUROGRAPHICS ’91 / Elsevier Science Publishers B.V. Eurographics
Association, 1991.
28
References…








Carnegie Mellon University, School of Computer Science
Halbert, Dan (November 1984). Programming by Example. U.C. Berkeley (PhD
dissertation)
J.F. Pane, "A Programming System for Children that is Designed for Usability," Ph.D.
Thesis, Carnegie Mellon
Joseph Lawrance, Steven Clarke, Margaret M. Burnett, Gregg Rothermel: How Well Do
Professional Developers Test with Code Coverage Visualizations? An Empirical Study.
VL/HCC 2005: 53-60
K. T. Stolee and S. Elbaum, "Refactoring Pipe-like Mashups for End User Programmers,"
International Conference on Software Engineering (ICSE), Honolulu, Hawaii, May 2011. to
appear.
Margaret M. Burnett, “Visual Programming” In the Encyclopedia of Electrical and
Electronics Engineering (John G. Webster, ed.), 1999
Morch, A., Stevens, G., et al. (2004) Component-based technologies for end-user
development. Communications of the ACM, 47(9), pp. 59-62.
Pausch, R., Conway, M., & DeLine, R. (1992). Lesson Learned from SUIT, the Simple User
Interface Toolkit. ACM Transactions on Information Systems, 10(4), 320-344. ACM DL Ref
Scaffidi C., Myers B.A., AND Shaw M. 2008. Topes: Reusable abstractions for validating
data. International
29