Lecture 26:
Past to Future: Artificial
Intelligence (AI) in Interaction
Techniques
Brad Myers
05-440/05-640:
Interaction Techniques
Spring, 2016
1
© 2016 - Brad Myers
Announcements

Schedule for final presentations posted in
googledoc:
https://docs.google.com/spreadsheets/d/1aHnqR0KpbJTG8UAGrX67AgIDBuRohJF-fhEgIBR3h8g/edit#gid=0

Optional meetings with me today – still slots open
random
number
0.025497
0.071698
0.103625
0.154695
0.363748
0.584315
0.6203
0.687627
0.731451
0.782284
0.928622
Group
Watch band text entry
Better tests
Knobs
Watch Tilt Text entry
Norm Cox Wiki
Pull-to-Refresh wiki
Virtual Reality
Leap Motion
Apple Watch
Scrolling wiki
Scroll study
talk length tentative presentation date
15
Wed, 4/20
30
Wed, 4/20
15
Mon, 4/25
15
Mon, 4/25
15
Mon, 4/25
15
Mon, 4/25
15
Mon, 4/25
15
Wed, 4/27
30
Wed, 4/27
15
Wed, 4/27
2
15
Wed, 4/27
© 2016 - Brad Myers
Intelligent User Interfaces

Lots of work in Intelligent User Interfaces in
general



http://iui.acm.org/2016/ - yearly since 1997
But most are not “interaction techniques”
Also, lots of work on AI to build UIs

E.g., automatic selection of interaction techniques


Not covered here. See slides from 05-830
I selected a few interaction techniques to cover:




Speech and natural language user interfaces
“Data detectors”
Squiggly underlining
Intelligent agents (“Clippy”)
3
© 2016 - Brad Myers
What makes a UI “Intelligent”?
4
© 2016 - Brad Myers
What makes a UI “Intelligent”?





“a user interface (UI) that involves some aspect
of Artificial Intelligence (AI or Computational
Intelligence). There are many modern examples
of IUIs, the most famous (or infamous) being the
Microsoft Office Assistant, whose most
recognizable agentive representation was called
"Clippy".” – Wikipedia
Using heuristics that may be wrong
Using elaborate pattern matching algorithms
Recognition-based interfaces
Knowledge based interfaces
5
© 2016 - Brad Myers
Key Evaluation Issues

Trust



Will users trust that the system is doing the right things?
High stakes vs. low stakes interactions
Components:

Accuracy






Visibility of System Status (Nielsen heuristics)


“False negatives” – misses something it should do
“False positives” – does something it should not
Often can reduce one by increasing the other
“Smarter” interface lowers all errors
“90% accuracy” – still gets 1 out of 10 wrong!
Can users tell what the system is doing?
User Control (Nielsen heuristics)

Can users fix it when it does the wrong thing?
6
© 2016 - Brad Myers
Speech and natural
language user interfaces




Speech recognition and natural language understanding has been a
CS research topic since at least the 1960’s
Very slow & steady progress with machine speeds and new
algorithms
Now “reasonably” accurate for conventional requests for people with
conventional speech
Speech: two phases

Recognition into a transcript



Natural language processing


Problems with common sense, references (pronouns), sentence structure, etc.
It turns out that dictating is difficult while thinking


Problems with words sounding alike, accents, background noise, pauses, etc.
Natural ways to correct are to hyper-articulate & talk slower, which often makes
recognition do worse
Especially given the need to be error free
Special “sub-languages” difficult to learn


Not clear what you are allowed to say
Interface needs to guide the user into saying things that will work.

Used to be much more strict about what can say – e.g., collect calls “yes”/”no”7
© 2016 - Brad Myers
Speech & NL

Key advantages:






Average human’s fastest output mechanism
Able to “jump around” and combine tasks
Can handle ambiguity and partial descriptions
Versus direct manipulation
Example: “Schedule a meeting the day before
CHI with everyone in my group.”
Key disadvantages


Inaccuracies, misrecognitions, unclear scope
Difficulties of corrections when wrong
8
© 2016 - Brad Myers
Today’s speech systems

Apple Siri, Google Now


Microsoft’s “Cortana” -- ref
Amazon Echo & equivalent

Always listening




Recognition by the “cloud”
Single statement phrases and questions


As opposed to “push to talk” like cars
Video ad
Doesn’t remember a dialog
“Direct answers” also available in Google
9
© 2016 - Brad Myers
Can we get some of those
advantages without speech?



PhD research of Toby Li
Ineffective use of smartphones
Data is all “siloed” in apps


Multi-app tasks
Inefficient or repetitive data or command
entry
10
© 2016 - Brad Myers
“Data Detectors”

Pattern matcher that looks for specific kinds
of data in plain text



Enables various operations on that text
E.g., recognizing phone numbers, people
names, URLs, email and physical addresses,
etc.
Nardi, B.A., Miller, J.R., and Wright, D.J.
“Collaborative, programmable intelligent agents.”
Comm. ACM 41, 3 (1998), pp. 96–104.
“Apple Data Detectors”
US 5,946,647 – “System and method for performing an
action on a structure in computer-generated data” by
Thomas Bonura, James R. Miller, Bonnie Nardi, David
Wright, Filed: Feb 1, 1996,


https://www.google.com/patents/US5946647

In the Apple v. Samsung case
11
© 2016 - Brad Myers
Research related to Data
Detectors

Grammex

Lieberman, H., Nardi, B.A., and Wright, D. Grammex: defining grammars by
example. Demo at CHI'98, ACM (1998), pp. 11–12.
http://web.media.mit.edu/~lieber/Lieberary/Grammex/Grammex-Intro.html


Define the pattern by giving a bunch of examples
Listpad – use data detectors to recognize structure in plain text lists


Kerry S. Chang, Brad A. Myers, Gene M. Cahill, Soumya Simanta, Edwin Morris
and Grace Lewis. "Improving Structured Data Entry on Mobile Devices", ACM
Symposium on User Interface Software and Technology, UIST'13, October 8-11,
2013, St. Andrews, UK. pp. 75-84. acm dl or local pdf and video (5:00)
Combine with web services to make data entry easier
12
© 2016 - Brad Myers
Squiggly (Wavy) underlining

In Word: for misspellings (red), grammar
problems (green), and formatting problems
(Blue) – reference


Originally, grammar
checker was quite
bad, but significantly
improved over time



Introduced in Word 95 for Windows – cite
Too many “false positives”
AI researchers at Microsoft Research helped with
better language models
Now used for errors in code as well as regular
documents
13
© 2016 - Brad Myers
Intelligent Agents

A common aspiration of
AI is a personified agent


Microsoft’s “Office Assistant”, known as
“Clippy” – video (0:38)






E.g., “Knowledge Navigator”
video from Apple, 1987
Office 1997 to 2003
“Smithsonian Magazine called Clippy “’one of the
worst software design blunders in the annals of
computing’". – cite
Too often useless and wrong (false positives)
Animates even when you are not supposed to use it.
Whole thesis on “Why People Hate the Paperclip:
Labels, Appearance, Behavior and Social
Responses to User Interface Agents” – pdf
Clifford Nass’s work on personification
© 2016 - Brad Myers
14
Automatic Design of UIs

Specify some property of design and system
picks widgets


Simple example: html <input type="radio">
Long history of more elaborate rules
15
© 2016 - Brad Myers
Early Example: Mickey

D. R. Olsen, Jr.. 1989. A programming language
basis for user interface. In Proceedings of the
SIGCHI Conference on Human Factors in
Computing Systems (CHI '89), K. Bice and C. Lewis
(Eds.). ACM, New York, NY, USA, 171-176.
http://dx.doi.org/10.1145/67449.67485

Programmer specifies UI by putting
special comments in a Pascal file.
Uses the Apple Macintosh
guidelines
Pre-processor to parse the Pascal
code and generate the Macintosh
resources.
Maps Procedures into Menu items.




If parameter is one of a standard set,
pops up appropriate dialog box or
waits for input


File to be read, file to be written
New point, line or rectangle
16
© 2016 - Brad Myers
Personal Universal Controller


Jeff Nichols’ PhD work (2006)
Problem:


Appliance interfaces are too complex and too
idiosyncratic.
Solution:

Separate the interface from the appliance and use
a device with a richer interface to control the
appliance:


PDA, mobile phone, etc.
Goal: Generate high-quality UIs
17
© 2016 - Brad Myers
Approach
Use mobile devices to control all appliances in the environment
Specifications
Control
Feedback
Mobile Devices
Appliances
Key Features
Two-way communication, Abstract Descriptions, Multiple Platforms,
Automatic Interface Generation
18
© 2016 - Brad Myers
Properties of PUC Language

State variables & commands

Each can have multiple labels


Typed variables



Useful when not enough room
Base types: Boolean, string,
enumerated, integers,
fixed-point, floating-point, etc.
Optional labels for values
Hierarchical Structure

Groups
19
© 2016 - Brad Myers
Generating Consistent UIs



Personally consistent
Reduce learning time
Add unique functions
20
© 2016 - Brad Myers
Generating Combined UIs



For multiple appliances, such as home
theaters
Specify content flow
Combined controls
21
© 2016 - Brad Myers
Summative Study



Compared PUC to manufacturer’s
interfaces for HP and Canon
printer/fax/copiers
PUC twice as fast, 1/3 the errors
Consistent: another factor of 2 faster
18
16
1200
Number of Failures
Average Time (sec)
1400
1000
800
600
400
14
12
10
8
6
4
200
2
0
0
HP
HP-PUC
HPConsistent
Canon
Canon-PUC
CanonConsistent
© 2016
- Brad Myers
HP
HP-PUC
HPConsistent
Canon
Canon-PUC
22
CanonConsistent
Recommender Systems

Collaborative filtering – based on other people’s
choices, as well as yours

Dates back to early research at Xerox PARC (1992)
David Goldberg , David Nichols , Brian M. Oki , Douglas Terry, Using collaborative
filtering to weave an information tapestry, Communications of the ACM, v.35 n.12,
p.61-70, Dec. 1992. http://dl.acm.org/citation.cfm?id=245121

Also content-based or
hybrid
23
© 2016 - Brad Myers
Other examples






Personalized autocomplete like Google
search bar
Natural language like google search bar, etc.
Programming-by-example in general
User models, like for intelligent tutors
Robotics – Roomba (not really an interaction
technique)
…
24
© 2016 - Brad Myers