History and Future of Interaction
Techniques (HFIT)
Brad Myers
Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
bam@cs.cmu.edu
www.cs.cmu.edu/~bam
1
© 2014 - Brad Myers
You Can Get the Slides

These slides are available on my website:

www.cs.cmu.edu/~bam/uicourse/2014inter/HFIT3hrworkshop.pptx

16.8 MB
Or go to www.bradamyers.com and click on the
link


Useful
links
embedded
in slides
2
© 2014 - Brad Myers
Full semester course

Excerpted from:
05-899A/05-499A: Interaction Techniques,
Spring, 2014

http://www.cs.cmu.edu/~bam/uicourse/2014inter/

All lectures were videotaped and are available
off of the schedule:

http://www.cs.cmu.edu/~bam/uicourse/2014inter/schedule.html

Guest lectures by:




David Canfield Smith - inventor of icons
Larry Tesler - inventor of cut-copy-paste
Ted Selker – inventor of the pointing stick
Chris Harrison – on the future of interaction techniques
3
© 2014 - Brad Myers
Instructor

Brad Myers






Human Computer Interaction Institute
Office: Newell-Simon Hall (NSH) 3517
Phone: x8-5150
E-mail: bam@cs.cmu.edu
http://www.cs.cmu.edu/~bam
Office hours: By appointment.
4
© 2014 - Brad Myers
Why am I teaching this course?



I was at MIT Media Lab (then “Architecture Machine Group),
1976-1979
At Xerox PARC, 1976-1980
Designed one of the first commercial window managers,
1980-1984







First to put progress bars into icons, and collect
icons in a window, etc.
Studies of two handed UIs and progress bars
with Bill Buxton, 1984 – 1988
Significant influence on progress bars
“All the Widgets” history video, 1990, vimeo
"A Brief History of Human Computer Interaction
Technology.” ACM interactions, 1998, pdf
With student, invented new text input technique
Significant consulting on patents on interaction
techniques, 1988-present
© 2014 - Brad Myers
5
“Interaction Techniques”

Scroll bars, buttons, text fields

But also:




Drawing a new object in an editor
Copy-and-paste
Selecting a cell in a spreadsheet
How high level? Text editor widget,
but not Word

Scroll bar is composed
of buttons, etc.
6
© 2014 - Brad Myers
Some examples


Visual Basic
Physical controls
7
© 2013 - Brad Myers
Other names

“Widgets” (Wikipedia: “GUI Widget”)





GUI “elements”
“Gadgets”


(Windows)
“Components”


But not the same
as Scott Hudson’s
“Controls”


Note that there are no cross references in Wikipedia
between “Interaction Technique” and “Widget”
See my video “All the Widgets”
But not the same as Apple dashboard widgets
Too generic
“Behaviors”

= the interaction
part
8
© 2014 - Brad Myers
Definitions

My definition:
An “interaction technique” starts when the
user does something that causes a computer
to respond, and includes the direct feedback
from the computer to the user. Interaction
techniques are generally reusable across
various applications.
9
© 2014 - Brad Myers
Definitions

Wikipedia’s definition:
“An interaction technique, user interface
technique or input technique is a combination
of hardware and software elements that
provides a way for computer users to
accomplish a single task.”
10
© 2014 - Brad Myers
What is not an interaction
technique?



Whole applications (Microsoft Word)
Dashboard “widgets” – small apps for the desktop
Output only (no interactions)

Visualizations





But many come with specialized interactions, then they might
count?
Animations
Movies
Progress bars
…
11
© 2014 - Brad Myers
Why Study Interaction
Techniques?

Used extensively



Interesting historically




Everyone who uses a computer
uses copy-paste, etc.
So can have an enormous impact
Why do we do things the way we do?
Is there a good reason?
Example: which way does the arrow point in a scroll bar?
And new interaction techniques are created all the time:

Patent on “Bounce at end of scrolling” for iPhone submitted by Bas Ording in
2007 (right before 1st iPhone was released in 2007) US 7,469,381


“Pull down to refresh” – patent submitted in 2010 by Twitter, became popular
in 2013!


Try it! iPhone vs. Samsung
US 8,448,084
Many new CHI & UIST conference papers every year with new ones
12
© 2014 - Brad Myers
Why Study Interaction
Techniques, cont.

Interaction Techniques have a high economic value

Often the subject of patents and lawsuits




Can’t patent overall look and feel
“Apple Wins Over Jury in Samsung Patent Dispute,
Awarded $1.05 Billion in Damages (Live Blog)” link
“Jury orders Samsung to pay $290M to Apple in patent
case” link
Need new ones

“Desktop metaphor” is getting tired



Macs & PCs look and work pretty similar to each other and to
the designs of the 1980’s (30 years ago)
Text entry on smartphones is a big barrier
Selecting individual elements, characters on smartphones
13
© 2014 - Brad Myers
Problem

April 29, 1991
14
© 2014 - Brad Myers
Problem

Appliances are too complex
15
© 2014 - Brad Myers
Problem

Too many remotes
16
© 2014 - Brad Myers
Why are Interaction
Techniques Hard to Design?



Surprisingly large number of design decisions
Individual differences and preferences
Lots of details that impact human performance

How far does the cursor move when you move the
mouse 1 inch?



Trick question – depends on mouse speed
Complex formula developed through experimentation
How far does the content move on an iPhone when you
flick your finger?


Needs to work for long distance, and highly accurate local
movements
Nokia phones released just after the iPhone got this all wrong
17
© 2014 - Brad Myers
Example: check box

How many “states” can it be in?







Checked, not-checked,
Disabled, not-disabled
Hover, not-hover (can’t be hover+disabled)
Pressed-inside, pressed-outside, notpressed (can’t be pressed + disabled, can’t
be pressed-inside + not-hover)
Keyboard focus, not-focus
2^4 * 3 = 48, but many are not possible
Often forget about the release-outside case &
interface gets confused (Flash implementations)
© 2014 - Brad Myers
18
Example2: Drawing a new object

What happens when move upwards past
start point?
19
© 2014 - Brad Myers
Affordances

“Perceived and actual properties of the
thing, primarily those fundamental
properties that determine how the thing
could possibly be used.” (Norman
DOET book, p. 9)


“When affordances are taken advantage of,
the user knows what to do just by looking”
Helps people understand what to do
with the control
20
© 2014 - Brad Myers
History of Personal Computers
and Their Interaction Techniques
21
© 2014 - Brad Myers
“Character Terminals”


Still around as “DOS Cmd prompts” and console
windows
But we are more interested
in graphics….
22
© 2014 - Brad Myers
Ivan Sutherland’s
Sketchpad, 1963
23
© 2014 - Brad Myers
SketchPad, 1963



Lincoln Labs TX-2 computer
“Light pen” pointing device
Invented many important interaction techniques

Direct manipulation




“Rubber Band Lines”
Snapping to a grid or other objects
Constraint-based drawing






Master with multiple copies,
Can edit the master to affect all copies
Almost arbitrary scaling of the whole drawing
Lots of individual switches and knobs to control the drawings
3D drawings added by others to Sutherland’s original SketchPad
program


Maintains connectivity of lines
Vertical, horizontal lines
Prototype-instance drawing


Uses a “light pen”
Including hidden line elimination
First flow chart – graphical programming

Ivan’s brother: William Sutherland!
24
© 2014 - Brad Myers
SRI and the
Mouse


Stanford Research Institute (SRI)
Bill English and Doug Engelbart
credited with the invention of the
mouse
[W.K. English, D.C. Engelbart and M.L. Berman. “Display Selection
Techniques for Text Manipulation,” IEEE Transactions on Human
Factors in Electronics. Mar, 1967. HFE-8(1).]

NLS, or the “oN-Line System”


"The Mother of All Demos” on December
9, 1968 at the Fall Joint Computer
Conference in San Francisco
Never really had a decent user
interface
25
© 2014 - Brad Myers
Xerox PARC


Palo Alto Research Center (PARC)
Founded by Xerox in 1970


Incredible collection of talent


Still exists today, as a semi-autonomous research lab
Hired many people from SRI, and many researchers and engineers
Incredible collection of inventions, 1970-1982

Hardware

Invented workstations, laser printing, the Ethernet







Bitmapped displays
Software


Only part that Xerox made money on
Invented many of the standard OS and systems principles
Object oriented programming (Smalltalk)
Model-View-Controller architecture
Interpress, a resolution-independent graphical page-description
language and the precursor to PostScript
User Interfaces
(and Ubiquitous Computing in the 1990s)
26
© 2014 - Brad Myers
Xerox Alto Machine

Everyone else at the time was using
mainframes or “mini computers” that were
shared


Alto was one of the first “personal
workstations”



Starting about 1973
No operating system – each program had
its own libraries and low-level access
mechanisms
Three button mouse with two opposing
roller wheels



“Time Sharing”
Red, Yellow, Green vertically
Later replaced with left, middle, right, with
single metal roller
Was secret for a long time
but later distributed to many
universities
© 2014 - Brad Myers
27
Brad Myers with an Alto, 1979

From my Dad’s scrapbook for that year, with my annotations!
28
© 2014 - Brad Myers
“Bravo”

Butler Lampson, Charles Simonyi and colleagues in 1974





Simonyi went to Microsoft and created Microsoft Word
First WYSIWYG text editing
Multiple fonts, bold, italics, etc.
Justification
Interaction techniques are quite
different



Left mouse button – select character,
middle – select word, right – extend
selection
Left – scroll up, right – scroll down,
middle - thumb
Highly moded commands:


“r” for replace, “d” delete, “I” insert,
“ESC” for stop inserting, …
“EDIT”
© 2014 - Brad Myers
29
Smalltalk







Started about 1972 as the first purely
object-oriented language by Alan Kay
Alan Kay proposed the idea of overlapping
windows in his 1969 doctoral thesis
Overlapping windows first appeared in
1974 in the Smalltalk’74 system
Also used popup windows, scroll bars, etc.
Larry Tesler invented the “browser” for code
for Smalltalk
Smalltalk’80 is best known – Byte article,
generally released and described
I worked with Smalltalk in 1977
30
© 2014 - Brad Myers
Various Drawing Programs on Alto


Draw – cubic splines for curves
Markup – in-place pop-up context menus
Source: http://toastytech.com/guis/saltodraw.png
© 2014 - Brad Myers
31
Larry Tesler

Xerox PARC 1973



Rejected highly moded interactions of Bravo
With Tim Mott, et. al, invented non-moded
interactions for Gypsy
editor including Copy
and Paste about 1974
Reaction to awkward
editing in NLS
32
© 2014 - Brad Myers
Larry Tesler, cont.


Copy and Paste added to Smalltalk
in 1976
He went to Apple in 1980

In charge of the Lisa design team
33
© 2014 - Brad Myers
“Workstations”


Alto
Lisp Machines (LMI & Symbolics)


Sun, Apollo, PERQ, Silicon Graphics




About 1979-1995
About 1982 - 2000
About $10,000 each
For scientists, engineers,
programmers
Had mouse, window
managers
34
© 2014 - Brad Myers
Xerox Star


Released 1982
Designed for executives


Large team of designers who
were not from PARC







Their building was next door to PARC
Extensive user interface studies guided designs
Key innovations


Too expensive for secretaries
Desktop metaphor
Many modern widgets
WYSIWYG editing and drawing
No PowerPoint or Spreadsheet
programs
Mostly closed – only Xerox made
applications
Too expensive and seemed slow
Images: http://toastytech.com/guis/star2.html
© 2014 - Brad Myers
35
Apple



Xerox wanted to invest in Apple
In exchange, Steve Jobs got the right to use
all of Xerox’s ideas
Steve & his team (Bill Atkinson) were given a
demo of various Alto programs in 1979




Mouse
Smalltalk – overlapping windows – thought they
updated
Bravo WYSIWYG editing
Apple hired Larry Tesler & others, 1980
36
© 2014 - Brad Myers
Apple “Lisa”


1983
Original design
for desktop







Bill Atkinson &
others
Novel pull-down menus (at top of screen)
Dialog boxes
Many other UI innovations
Doesn’t look or work like the Star
One button mouse
Amazing programming expertise to get it to work
on a tiny, inexpensive machine Image: http://toastytech.com/guis/lisa.html
37
© 2014 - Brad Myers
Original Macintosh




1984
Much cheaper
than Lisa
No harddisk – just one floppy
128 k-bytes of memory


Much of code re-implemented in assembly
Famous 1984 Super Bowl ad by Ridley Scott
38
© 2014 - Brad Myers
HyperCard


Bill Atkinson, 1987
Intention – “programming
for the rest of us”



Many UI innovations




One of the first “prototyping” systems
But not used for many “real” applications
Tear off menus
Pages that overlay each other
Animated transitions
Programmed in “HyperTalk”

English-like language
39
© 2014 - Brad Myers
PCs & Windows




IBM PC – 1981
(IBM had missed the
“minicomputer” phase dominated by DEC)
Used Microsoft’s DOS 1.0 and shipped with
VisiCalc
Windows 1.0 released in Nov, 1985 as DOS
extension



Windows 1
from Wikipedia
Tiled window manager
Windows 2.0 was overlapping 1987
Windows 3.0 in 1990, 3.1 in 1992


Was a real operating system
Added virtual memory, protected multiple processing, etc.
40
© 2014 - Brad Myers
History of Handhelds
(PDAs, Smartphones & Tablets)
and their Interaction Techniques
© 2014 - Brad Myers
41
“Computers”
(cite,
slide 24, 25)
42
© 2013 - Brad Myers
Early Handwriting Input

Handwriting recognition has
been an active research
topic since 1960’s:

Rand Tablet: 1964:
http://www.rand.org/content/dam/rand/pubs/research_memoranda/2005/RM4122.pdf



Used term: “pen-computing”
Early: hand printing
Lots of work on handwriting and
gestures

E.g., W. Buxton, E. Fiume, R. Hill, A. Lee, C. Woo,
“Continuous hand-gesture driven input,” Graphics
Interface '83 (1983), pp. 191–195
43
© 2014 - Brad Myers
“Ubiquitous Computing”

Term coined by Mark Weiser
at Xerox PARC, 1988




Mark Weiser. “The Computer for the 21st
http://en.wikipedia.org/wiki/File:Mark_weiser.jpg
Century”, Scientific American, 94-104, Sep 1991.
Mark Weiser. “Some Computer Science Issues in Ubiquitous Computing,” CACM. July,
1993. 36(7). pp. 74-83.
(Died at 46 in 1999 of cancer)
“I called these three sizes of computers boards, pads, and tabs, and adopted the
slogan that, for each person in an office, there should be hundreds of tabs, tens of
pads, and one or two boards.” [p. 76]
44
© 2014 - Brad Myers
PARC Tab




~1989
Low speed wireless network using IR
Touch-sensitive screen
Quick writing – unistrokes, write on top of each other

David Goldberg and Cate Richardson. “Touch Typing with a Stylus,” Human Factors in
Computing Systems, Proceedings INTERCHI'93. Amsterdam, Netherlands, Apr, 1993.
pp. 80-87.
45
© 2014 - Brad Myers
Go Corp’s “PenPoint” OS

Founded 1987, released in 1991





Hardware by NCR, IBM and EO
Styled to look like a tabbed notebook
Conventional tapping on menus
Lots of gestures for editing,
page turning, etc.





One of the founders was Robert Carr from Xerox PARC;
Alto designer
Flick to scroll and turn pages, circle, insert
space, cross-out, insert word, get help, …
Press and hold to start moving or selecting
Hand printing for text entry
Hyperlinks
Instant on-off
© 2014 - Brad Myers
46
Apple Newton





Started 1987, released 1993
Newton “MessagePad”
Coined term “Personal Digital
Assistant (PDA)
Was on sale for 6 years
Fairly large & heavy
John Sculley III
47
© 2014 - Brad Myers
Apple Newton

Key issue: handwriting recognition was main
input technique
http://www.computerhistory.org/revolution/mobile-computing/18/319/1714

Often not successful

Famously panned for an entire week by Doonesbury
(August 1993)
48
© 2014 - Brad Myers
Early phone + PDAs

IBM Simon


Nokia 9110 Communicator



1996
Added full physical keyboard
Typical PDA features:


Shipped in 1994 by BellSouth
Address book, calendar
Slow
49
© 2014 - Brad Myers
Palm


US Robotics (1995), 3Com (1997),
Handspring (1998), Palm (2000), HP
(2010)
First released version: 1996 = “Pilot”



Name changed due to lawsuit
They did lots of user testing with
prototypes created using HyperCard
Graffiti for data entry
50
© 2014 - Brad Myers
Palm Graffiti

Had seen Xerox QuickWriting


Designed to be easier to
learn



Lawsuit
Still required practice
Unistroke except for “X”
Two sides – numbers look the
same as some letters
51
© 2014 - Brad Myers
Palm’s design Principles
“Designing the Palm Pilot: A conversation with Rob Haitani”, by Eric Bergman and Rob Haitani, chapter 4 in
Information Appliances and Beyond, Eric Bergman, ed. (2000)

Fast access to key features on small screens ->


Only a few commands used a lot
Leave commands off main screen, even if not symmetric






new vs. delete
(think stapler and stapler remover)
Note that violates consistency
Tap and then type in schedule and to-do
Only four buttons – which ones?
Vs. Windows CE -> if know PC, this is familiar

But usage models are different


PC: infrequent long usage
Palm: frequent short bursts of usage
52
© 2013 - Brad Myers
Palm Watch

Fossil,




Announced 2002, shipped 2003-5
160 x 160 illuminated screen
with a stylus integrated into the band,
8MB internal memory, rechargeable
battery and
standard Palm platform features
$250
Heavy, short battery life, tiny stylus
53
© 2014 - Brad Myers
Palm Phones

Kyocera QCP-6035 about 2001



Physical phone buttons, or regular
Palm
Low-speed internet
Handspring (then Palm) Treo


Blackberry-like keyboard
replaces Graffiti
Starting 2002
54
© 2014 - Brad Myers
RIM Blackberry



Starting 1999
Research in Motion (RIM)
Blackberry 850



Email & pager
Originally, proprietary network
Key features:


Two-thumb keyboard
Roller dial (“scroll wheel”) for navigation




Eventually, became 2D navigation
Later, regular phone networks
Awkward attempts at full-screen
touchscreen


Moved to side of device
Attempted to be backwards
compatible with old applications
Insufficient 3rd party applications

Late to have good APIs
55
© 2014 - Brad Myers
Early wireless phone UIs



1993 – first Nokia soft keys & scrolling
Standardized on 2 or 4 directions,
2 action keys
Motorola Razr – 2004


Text entry by multi-tap or T9


Thinner is better
Note: not touch screens
WAP – starting 1997



Wireless Application Protocol
Bring web-like access to these
devices
Terrible usability

Nielsen study
56
© 2014 - Brad Myers
Windows TabletPC




2001 spec (Windows XP), first
devices in 2002
Bill Gates said it would be big (2002)
Handwriting recognition was much
better, but still not sufficiently
accurate
Windows UI not
changed for pen


Lower accuracy than mouse
Quite poor UIs for
correction
57
© 2014 - Brad Myers
iPod



2001
Apple iPod lauded for design and
user interface
Unique dial interaction technique




Enabled easy access to thousands
of songs
Highly tuned speed ratio
iTunes  entire service design
5 GB hard drive that put “1,000
songs in your pocket.”
58
© 2014 - Brad Myers
iPhone


Starting 2007
Went against the conventional wisdom in
many aspects





No blackberry-style keyboard
Capacitive screen (multi-touch)
No stylus
Only one button – focus on
easy to use
Some unique interaction
techniques

Scroll bounce, swipe login, …
59
© 2014 - Brad Myers
Phone MarketShare
http://en.wikipedia.org/wiki/File:World_Wide_Smartphone_Sales_Share.png
60
© 2013 - Brad Myers
iPad


2010
Very different from TabletPC




Media machine
Little text entry facilities
Interactions same as a Phone,
instead of mimicking a PC
Focuses on ease of use
61
© 2014 - Brad Myers
Future of Pointing

Chris Harrison, Desney Tan, and Dan Morris. 2010. Skinput:
appropriating the body as an input surface. In Proceedings of the
SIGCHI’10, 453-462. http://doi.acm.org/10.1145/1753326.1753394
62
© 2014 - Brad Myers
Future of Pointing

Chris Harrison, Julia Schwarz, and Scott E. Hudson. 2011.
TapSense: enhancing finger interaction on touch surfaces. In ACM
UIST '11. 627-636. http://doi.acm.org/10.1145/2047196.2047279

Can identify which part of finger using
microphone
63
© 2014 - Brad Myers
Exercise – How fast can you tap?

Compare mouse, pointing stick, touchpad, direct
finger tapping on phone or tablet, (anything else?)

http://www.cs.cmu.edu/~bam/uicourse/2014inter/fittslaw/

Go to this URL
Click on the blue bar as
fast and accurately as
you can
What is your total time
and error percent?


64
© 2014 - Brad Myers
Deep Dive: Scrolling Techniques
© 2014 - Brad Myers
65
Early Systems

SketchPad (1963): Pan and zoom with knobs

“The size and position of the part of the total picture seen on the
display are controlled by the four black knobs just above the
table.”
66
© 2014 - Brad Myers
Early Systems, 2

Bravo, 1974





Move to left margin, cursor changes shape
Press to see cursor, release to do operation
Left = Scroll up – line next to cursor goes to top
Right = Scroll down – line at top of window goes
to cursor
Middle = “thumb” – jump to that percent of the
document, with indicator of where you are
67
© 2014 - Brad Myers
Smalltalk




Smalltalk, 1977
Scroll bar pops up to the left of the
document
Focus window has a scroll bar
Three regions:




Right region – text moves up
Left region – text moves down
Center – drag thumb smoothly
Thumb shows percent visible
68
© 2014 - Brad Myers
Keyboard keys



WordStar, June 1979, etc.
Scroll using keyboard keys
Scrolls to keep cursor on the screen
69
© 2014 - Brad Myers
InterLisp, 1980



Popup on left of window, as move out to
the left (same as Smalltalk)
Thumb showing percent visible
Left button – scroll up




Same as Bravo – line next to cursor to top
Right button – scroll down
Middle button – thumb
Same cursors as Bravo
70
© 2014 - Brad Myers
Star (1981-1982)

Scroll bar on right to get it out of
the way


Scroll arrows point in the direction the contents
will move





Reduce the visual clutter
Based on user studies
+, - buttons to scroll by pages
Thumb fixed size diamond, independent
of how much of document is visible
Clicking in thumb “elevator” region jumps
to that part of the document
Viewpoint (1985)

When press and hold, can move outside the scroll bar


“Reduce the hand-eye coordination problems users were
experiencing
Right button – move by percent, or by window rather
than page
71
© 2014 - Brad Myers
Lisa




1983
Arrows pointing up and
down
Page buttons
Fixed size thumb
72
© 2014 - Brad Myers
Macintosh


1984
Arrows point up and down



Removed the page buttons




“Click in a grey region”
Hold down for auto-repeat – issue?
Press in thumb and drag to get to a particular
point in the document


“Point towards data that will be exposed when the
arrow is pressed”
Arrows auto-repeat
Abort by dragging (far) out of scroll bar before release
No scroll bar shown if can see whole document
Empty scroll bar when not the focus window
73
© 2014 - Brad Myers
Alternate Reality Kit (ARK)



1985
Hand at edge causes scrolling to start from
that side
“Teleporters”
74
© 2014 - Brad Myers
OpenLook







1988
(One of the Unix “X/11” look and feels – see
Lecture 8)
Novel elevator – put arrow keys on it
Clicking on cable moves by pages
Auto-repeat – pushes the pointer along
“Cable anchors” – beginning or end of the
document
Drag from center of elevator
75
© 2014 - Brad Myers
NeXT




1989
Scroll bars moved back to the left side
of windows
Proportionally sized thumb
Arrows are together at bottom



Auto repeat
Alt-key then moves by window fulls
Drag thumb

Alt-key while dragging moves more slowly
76
© 2014 - Brad Myers
Mouse Scroll Wheel


Popularized by the Microsoft IntelliMouse in 1996
along with support for the mouse wheel in Microsoft
Office 97. – Wikipedia
Turn to scroll by increments




Can set how much that is
Can press wheel
for “middle” mouse
button
New: some move
smoothly – no
notches
Now used for
zoom in Google
Maps, etc
77
© 2014 - Brad Myers
Rate-controlled scrolling

Can enable press-and-hold of mouse wheel


Faster if move further from press-point
Also, press “middle” button on IBM Thinkpads
and pull with pointing stick
78
© 2014 - Brad Myers
Palm Pilot


1997
Conventional scroll bar


Drag thumb or tap on
arrows with stylus
Arrow buttons
79
© 2014 - Brad Myers
RIM Blackberry dial




1999
Move with right thumb
Can press in to activate
selected item
Not a touch screen
80
© 2014 - Brad Myers
Macintosh recent scrollbars

2001 – thumb now proportional


Moved arrow buttons to bottom
2011 – no more buttons on end
so looks more like iOS version
81
© 2014 - Brad Myers
iPhone


2007
Flick to scroll




Flick to scroll
existed well before
iPhone
iPhone has a highly tuned momentum
function
Stops when touch the screen
Innovation: bounce at end

Patent on “Bounce at end of scrolling”
for iPhone submitted by Bas Ording in
2007 (right before 1st iPhone was
released in 2007) US 7,469,381
82
© 2014 - Brad Myers
iPhone scroll bars


Displays scrollbar with letters
for jumping around in
contacts, etc.
“regular” scrollbar in
web browser, other
applications

Just output – not
touchable
83
© 2014 - Brad Myers
iPhone, cont.

Two-finger drag





Takes advantage of multi-touch screen
Can also flick with momentum
Also two-finger rotate,
zoom
Imported into Mac
touchpad (when?)
Now available on
most touchpads
84
© 2014 - Brad Myers
Other Scrolling
Mechanisms

Hand to grab the contents and scroll




Dial on original iPod (2001)
Tap at edge or flick to go page-bypage on eReaders
“Infinite scroll” like on Twitter &
Facebook



MacPaint (1984), Adobe Acrobat, etc.
Usability problems
Scroll bars with marks for search
results, errors, etc.
… what else?
85
© 2014 - Brad Myers
“Sliders”



Part of most widget sets to select numbers in a
range
Usually look different than scrollbars, but behave
similarly
Two-handled
“range sliders”
86
© 2014 - Brad Myers
Research System

1978

Spatial Data Management
System (SDMS)



MIT “Architecture Machine Group”
now MIT Media Lab
Use a small, zoomed out version
for 2-D scrolling


One monitor for “world view”
Main (large) screen for area of
current interest
87
© 2014 - Brad Myers
Research Paper

William Buxton and Brad Myers. "A Study in Two-Handed
Input," Proceedings SIGCHI '86: Human Factors in Computing
Systems. Boston, MA. April 13-17, 1986. pp. 321326. pdf or html and video. (required)

Explored two handed interactions



Clicking, resizing, scrolling
Clicking and resizing done in parallel
Scrolling was not, but still faster to
use 2 hands
88
© 2014 - Brad Myers
Research Paper

George W. Fitzmaurice. 1993. Situated information spaces and spatially aware palmtop
computers. Commun. ACM 36, 7 (July 1993), pp. 39-49.
http://dl.acm.org/citation.cfm?doid=159544.159566

3D sensor on hand-held device
Moving device causes contents to move as if
contents were fixed in place




“Move to scroll”
Also zooming if move towards and away from the
user
Later implemented
using Smartphone
camera and
“optical flow”
image analysis
89
© 2014 - Brad Myers
Research Paper

Christopher Ahlberg and Ben Shneiderman. 1994. The
alphaslider: a compact and rapid selector. In Proceedings of
the SIGCHI Conference on Human Factors in Computing
Systems (CHI '94). ACM, pp. 365-371.
http://dl.acm.org/citation.cfm?doid=191666.191790

New designs for a more accurate slider
(scroll bar)






User test with 10,000 items
Position and scrollbar
fastest
Rate controlled
failed


One divides thumb into 3 regions, of different
scroll speeds
Another: depends on speed of mouse
movements
Another: speed depends on vertical distance
from slider
Too much
overshoot
Now this is used
for video playback
90
© 2014 - Brad Myers
Research Paper

Jun Rekimoto. 1996. Tilting operations for small screen interfaces. In Proceedings of
the 9th annual ACM symposium on User interface software and technology (UIST
'96). ACM, pp. 167-168. http://dl.acm.org/citation.cfm?doid=237091.237115

Tilt to scroll
Also, tilt to select menu items
Usability issues [Hinckley 2000]


91
© 2014 - Brad Myers
Research Paper

Brad Myers, Kin Pou ("Leo") Lie and Bo-Chieh ("Jerry") Yang, "Two-Handed Input Using a
PDA and a Mouse", Proceedings CHI'2000: Human Factors in Computing Systems. April 1-6,
2000. The Hague, The Netherlands. pp. 41-48. local pdf.

See if could use PDA in left hand
Clicking, scrolling, etc.
Created a variety of scrollers on PDA screen
Used design from IBM study of scrolling techniques





10 pages from IBM manual with embedded links
Mouse fastest, buttons close, rate-controlled worst
92
© 2014 - Brad Myers
Research Paper

Selina Sharmin, Oleg Špakov, and Kari-Jouko Räihä. 2013. Reading
on-screen text with gaze-based auto-scrolling. In Proceedings of the
2013 Conference on Eye Tracking South Africa (ETSA '13). ACM, pp.
24-31. http://dl.acm.org/citation.cfm?doid=2509315.2509319

Participants preferred to read only at
the top of the screen
So auto-scrolled when looked below
that
No statistically significant difference
in reading speed
or preference


93
© 2014 - Brad Myers
Exercise #2 –
How fast can you scroll?

Try with different kinds of scrollers

http://www.cs.cmu.edu/~bam/uicourse/2014inter/ScrollExperiment/

What is your total time?
94
© 2014 - Brad Myers
Past to Future: Interactions in 3D,
Very Large Displays, and Virtual
Reality
© 2014 - Brad Myers
95
Why is 3D Harder?

Objects have six degrees of
freedom (DoF)



Also camera position



Occlusion and resolution issues
Difficulty of orienting oneself
People are not very good at 3D
manipulation or reasoning


X, Y, Z
Roll, pitch, yaw
Mouse is basically 2D
Generally, dealing with
complex, hierarchical objects
96
© 2014 - Brad Myers
Why Hard, cont.


Rick Carey, Tony Fields, Andries van Dam, Dan Venolia. 1994. Why is 3-D
interaction so hard and what can we really do about it? (panel). In Proceedings
SIGGRAPH '94. ACM, pp. 492-493. http://doi.acm.org/10.1145/192161.192299
3D picking is hard – which object is selected?


Designing widgets for 3D manipulation is hard



Occlusion, hierarchy, accuracy of pointing device
Interfere with graphics
Should they have shadows?
Harder to get interactive speeds for direct
manipulation
97
© 2014 - Brad Myers
Where 3D displayed?


Desktops – just on a screen in the usual way
3D “Cave” or other large displays (ACM ref)


Display on 1 or up to all walls and ceiling
Virtual Reality (VR) or Augmented Reality (AR)
headsets

AR – can see through the display, so
pictures are superimposed on the view
98
© 2014 - Brad Myers
3D Control

Regular Mouse or touch – 2D


“Mouse in the air” tracked in 3D = “bat”; 6 DoF





Possibly with extra knobs or buttons
“bat” translates to fledermaus in German
(mouse that flies through the air)
Fixed camera tracking object in 3D space
Moving the end of an articulated motorized arm
3D physical objects incorporating the above
99
© 2014 - Brad Myers
Types of 3D sensors


Earliest: Boxes with sets of knobs
for each dimension
Polhemus trackers (“bat”)







DataGlove




Starting in 1969
Magnetic cube on part to be tracked
and nearby receiver
6 DOF
Limited sensing area
Company still selling similar products
Often attached to gloves, head-trackers, etc.
Starting about 1982
Measured finger bending =
pose of hand
Incorporated Polhemus tracker
on the wrist
Nintendo “PowerGlove” – 1989

Unsuccessful – only 2 games
© 2014 - Brad Myers
100
Virtual reality on five
dollars a day

Randy Pausch. 1991. Virtual reality
on five dollars a day.
In Proceedings of the SIGCHI
Conference on Human Factors in
Computing Systems (CHI '91),
ACM, pp. 265-270.
http://dl.acm.org/citation.cfm?doid=
108844.108913

Combined with
inexpensive virtual
reality headset
101
© 2014 - Brad Myers
Minority Report, 2002


Using data gloves to interact with large 2-D
displays in the air (or on a surface)
MIT Media Lab advised on science (John
Underkoffler)
102
© 2014 - Brad Myers
History of 3D sensors,
cont.

Lots of motion capture research
and systems




Kinect




Motion capture rooms with cameras
Used for movies, etc.
(But no interaction techniques)
Introduced 2010
Two cameras
Next lecture (w/ game controllers)
Leap Motion


2013
Camera based – designed to look
upwards
103
© 2014 - Brad Myers
3D “arm” Controllers


Motors to measure 3D movements
and provide force feedback
Phantom OMNI from Sensable
Technologies



Medical Applications, etc.
3D editing and drawing
Falcon from Novint

Marketed for gaming
104
© 2014 - Brad Myers
Special-purpose input devices
for Manipulation

Head and cutting plane


Ken Hinckley, Randy Pausch, Dennis Proffitt, and Neal F. Kassell. 1998. Two-handed
virtual manipulation. ACM Trans. Comput.-Hum. Interact. 5, 3 (September 1998), 260302. http://dl.acm.org/citation.cfm?doid=292834.292849
Articulated “marionette”

Wataru Yoshizaki, et. al. 2011. An actuated physical puppet as an input device for
controlling a digital manikin. In Proceedings CHI '11. ACM, pp. 637-646.
http://dl.acm.org/citation.cfm?doid=1978942.1979034
105
© 2014 - Brad Myers
Mouse-Based 3D manipulation

Common 4-panel display


Mouse works in conventional way in each panel
Still tricky to manipulate
AutoDesk Maya
106
© 2014 - Brad Myers
One screen mouse approaches

Michael Chen, S. Joy Mountford, and Abigail Sellen. 1988. A study in interactive
3-D rotation using 2-D control devices. In Proceedings SIGGRAPH '88. ACM pp.
121-129. http://doi.acm.org/10.1145/54852.378497

6D control with 2D mouse





Sliders for each dimension
2D areas (2D “sliders”) for each pair of
dimensions
Menus – pick axes to
manipulate first
Button modifiers – so can
quickly change
“Virtual Sphere Controller”
for 3D rotation

Pretend object is in a sphere so
can rotate around outside
© 2014 - Brad Myers
107
3D Handles


Extend idea of handles on 2D objects to 3D
Need handles for move, stretch, rotate, etc. in
each dimension

Many approaches for doing this. E.g.,

Scott S. Snibbe, Kenneth P. Herndon, Daniel C. Robbins, D. Brookshire
Conner, and Andries van Dam. 1992. Using deformations to explore 3D
widget design. In Proceedings SIGGRAPH '92, ACM, pp. 351-352.
http://doi.acm.org/10.1145/133994.134091
108
© 2014 - Brad Myers
Research: 3D miniatures

Randy Pausch, Tommy Burnette, Dan Brockway, and Michael E. Weiblen. 1995.
Navigation and locomotion in virtual worlds via flight into hand-held miniatures.
In Proceedings of the 22nd annual conference on Computer graphics and interactive
techniques (SIGGRAPH '95), ACM, pp. 399-400.
http://dl.acm.org/citation.cfm?doid=218380.218495

Miniature version of the world held in your
hand
Can move objects in
miniature version
to edit
Move avatar to
change camera view


109
© 2014 - Brad Myers
Research: Manipulating
Objects

Jeffrey S. Pierce, Andrew S. Forsberg, Matthew J. Conway, Seung Hong,
Robert C. Zeleznik, and Mark R. Mine. 1997. Image plane interaction techniques
in 3D immersive environments. In Proceedings of the 1997 symposium on
Interactive 3D graphics (I3D '97). ACM, 39-ff.
http://dl.acm.org/citation.cfm?doid=253284.253303

Head-crusher, lifting palm
Renders hand in the scene, detects pose of
hand, detects objects in relation to the pose

110
© 2014 - Brad Myers
Research: 3D on Touch
Screens

Aurélie Cohé, Fabrice Dècle, and Martin Hachet. 2011. tBox: a 3d
transformation widget designed for touch-screens. In Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems (CHI '11).
ACM, pp. 3005-3008. http://dl.acm.org/citation.cfm?doid=1978942.1979387

Surround 3D object with a cube
1 or 2 fingers moving on cube have special
meanings
Translate, stretch,
move camera, etc.



Video on ACM site
111
© 2014 - Brad Myers
Research: 3D drawing by
“Sketching”

Robert C. Zeleznik, Kenneth P. Herndon, and John F. Hughes. 1996.
SKETCH: an interface for sketching 3D scenes. In Proceedings of
SIGGRAPH '96. ACM, pp.163-170.
http://graphics.cs.brown.edu/research/pub/papers/sig96-sketch/sig.html

Only use 1 button to draw (another to move
camera)
Constructive Solid Geometry (CSG) = adding
and removing pieces of geometric elements


Cubes, spheres, etc.

Uses heuristics to guess what various drawings
mean

Video on
ACM site
112
© 2014 - Brad Myers
Research: Sketching Soft
Objects

Takeo Igarashi, Satoshi Matsuoka, and Hidehiko Tanaka. 1999. Teddy: a
sketching interface for 3D freeform design. In Proceedings of SIGGRAPH '99.
ACM, pp. 409-416. http://dx.doi.org/10.1145/311535.311602

Smooth sketches turn into
smooth objects
Frequent rotations so can draw
on the image plane
Extrudes or cuts the shape
(YouTube) 5:01min



113
© 2014 - Brad Myers
New Research

Still an active research area. E.g.

Interactions with Google Glass (web page)



Gestures on the touch sensor on side
Head gestures
From UIST'13:



Mime: Compact, Low Power 3D Gesture Sensing for
Interaction with Head Mounted Displays (ACM DL)
uTrack: 3D Input Using Two Magnetic Sensors (ACM DL)
BodyAvatar: Creating Freeform 3D Avatars using FirstPerson Body Gestures (ACM DL)
114
© 2014 - Brad Myers
Thanks!
115
© 2014 - Brad Myers
For more information

05-899A/05-499A: Interaction Techniques,
Spring, 2014

http://www.cs.cmu.edu/~bam/uicourse/2014inter/

“All the Widgets” history video, 1990,
https://vimeo.com/61556918
Brad A. Myers. "A Brief History of Human Computer
Interaction Technology." ACM interactions. Vol. 5, no. 2,
March, 1998. pp. 44-54. pdf

116
© 2014 - Brad Myers