What Users Want
Daniel Weld
University of Washington
Two Interface Trends
Usage
4x
24-Jul-16
Variation
625x
Daniel S. Weld / Univ. Washington
2
“Steelcase-Inspired Software”
-- David Gelernter

One Size
Need
Increased
Fits All Customization
• Beyond changing buttons on the toolbar




Overriding inappropriate adaptation
High-level functionality
Programming by demonstration
Need Increased Adaptivity
• Beyond inconsistent defaulting




Adapt
Adapt
Adapt
Adapt
to
to
to
to
available devices, connectivity, …
user location
user tasks & goals
user calendar & current time
(Some overlap with Contextual Computing)
24-Jul-16
Daniel S. Weld / Univ. Washington
3
Adaptivity & Customization

Deep Deployment ~ OS layer

Consistent Across Applications


Adaptation in every ‘dialog’
Bridging Applications
Data gathering
 Transformations

24-Jul-16
Daniel S. Weld / Univ. Washington
4
Outline
High-level
Customization
 Motivation
 Deliberative
Software Agents
 Programming by Demonstration
 Adaptive Websites
 Adaptive User Interfaces
Pure
Adaptation
24-Jul-16
Daniel S. Weld / Univ. Washington
5
Genesis: Internet Softbots

Interface Principles:
Goal-Oriented
 Integrated
 Balanced
 Safe

24-Jul-16
[Etzioni & Weld CACM 94]
User says what she wants
Single,
Agentexpressive
determines &how &
uniform
when interface
to achieve it
DT-lite: “Softbot must
E.g.,
printing
files
vs.
balance
the cost
of finding

planning-based
Asimov’s first law… softbots
FTPingon
them
information
its own with
Safety, tidiness &
the nuisance of asking the
thriftiness
user.”
constraints in planner
Daniel S. Weld / Univ. Washington
6
Softbot
UW Softbot
Project
Family
Observations
Tree

Rodney
Specifying goals is bottleneck


Logical spec. language ?!$@#!
Simon
Forms interface
• Limited to anticipated use
• Not scalable
Info Manifold

BargainFinder
MetaCrawler
Grouper Ahoy
Often easier to just do task!
ShopBot
• Except: data integration tasks
Occam

Natural language interfaces


Popescu et al.
Yates et al.
} Wed 10:15
ILA
Wrapper Induction
Razor
…
Tukwila
LSD
Mulder
GLUE
Piazza

Programming by demonstration
24-Jul-16
Daniel S. Weld / Univ. Washington
Gplex
7
Outline
High-level
Customization
 Motivation
 Deliberative
Software Agents
 Programming by Demonstration
 Adaptive Websites
 Adaptive User Interfaces
Pure
Adaptation
24-Jul-16
Daniel S. Weld / Univ. Washington
8
Programming by Demonstration
[Lau & Weld IUI-99]
[Wolfman et al. IUI-01]
[Lau et al. ICML-01]
If it’s too hard for users to specify goals
 Let’s watch them…

And try to help
 Like plan recognition


Initial domains:
Email
 Text editing
 Cross domain

24-Jul-16
Daniel S. Weld / Univ. Washington
9
Gentle-Slope Systems
C
Click’nCreate
VBasic
Hypercard
Hypertalk
Basic
Difficulty
of use
C
C Plugins
kCmds
MFC
(Multimedia Fusion)
Ideal
Sophistication of what can be created
24-Jul-16
Daniel S. Weld / Univ. Washington Adapted from Myers et al.10
Objectives
Macro Record On / Off
Straight Ln Brittle
VBasic Programing Turing Cplt
Robust
None
Varying
Lots
Previous PBD
Little
Varying
SmartEdit
SmartPython
On / Off
Loops,Cond Robust
24-Jul-16
None
Daniel S. Weld / Univ. Washington
Minimal
11
SmartEdit 1
[Lau et al. ICML-01]
24-Jul-16
Daniel S. Weld / Univ. Washington
12
2
24-Jul-16
Daniel S. Weld / Univ. Washington
13
3
24-Jul-16
Daniel S. Weld / Univ. Washington
14
4
24-Jul-16
Daniel S. Weld / Univ. Washington
15
5
24-Jul-16
Daniel S. Weld / Univ. Washington
16
6
24-Jul-16
Daniel S. Weld / Univ. Washington
17
PBD as a learning problem

Action is function : input state  output state


Editor state: text buffer, cursor position, etc.
Actions: move, select, delete, insert, cut, paste,…
Move
to next
<!-
Given a state sequence, infer actions


Delete
to next
-->
Many actions may be consistent with one example
Challenge: Weak bias + low sample complexity
24-Jul-16
Daniel S. Weld / Univ. Washington
18
Version Space Algebra


Version space = set of complex functions
Define version space hierarchically

Combine simpler version spaces with
algebraic operators
• Union  : analogous to set union
• Join
: cross product with consistency predicate
• Transform: convert functions to different types

Can factor a version space
24-Jul-16
Daniel S. Weld / Univ. Washington
19
SMARTedit's version space
…
24-Jul-16
…
…
…
Daniel S. Weld / Univ. Washington
20
Fast Consistency


Test consistency of example against
entire version space
Quickly prune subtrees
Action
Move
Paste
Insert
Select

Copy
Delete
Cut
Innovations:

24-Jul-16
Independent join allows BSR representation
Daniel S. Weld / Univ. Washington
21
Preliminary User Study




6 undergrad CS majors
7 repetitive tasks with & later w/out SMARTedit
Tasks: 4 to 27 iterations, 1-5 min to complete
Evaluation metrics:



Time saved completing task with SMARTedit's help
% user actions (keyboard + mouse) saved
User feedback
24-Jul-16
Daniel S. Weld / Univ. Washington
22
Time saved using SMARTedit
300
Time (sec)
(sec)
Cost Time savingsSaved
250
200
150
100
A
B
C
D
50
0
X
X
E
F
- 50
Six
Users
- 100
- 150
- 200
1
2
3
4
5
6
Task
Task Number
24-Jul-16
Daniel S. Weld / Univ. Washington
23
Action Savings with SMARTedit
100
Cost
Saved
Percent savings
Percent of Actions
80
60
40
20
A
B
C
0
D
E
X
- 20
X
XXXX
Six
Users
F
- 40
- 60
1
2
3
4
5
6
Task Number
Task
24-Jul-16
Daniel S. Weld / Univ. Washington
24
Observations from PBD

Overhead of macro recorder UI is high
Most repetitive tasks short
 How many shell scripts do you write / day?


Focus on pure adaptivity

E.g., automatic segmentation
24-Jul-16
Daniel S. Weld / Univ. Washington
25
Outline
High-level
Customization
 Motivation
 Deliberative
Software Agents
 Programming by Demonstration
 Adaptive Websites
 Adaptive User Interfaces
Pure
Adaptation
24-Jul-16
Daniel S. Weld / Univ. Washington
26
Early Adaptation: Mitchell,Maes

Predict:
Email message priorities
Meeting locations, durations

Principle 1:
Principle 2:
Defaults minimize cost of errors
Allow users to adjust thresholds

24-Jul-16
Daniel S. Weld / Univ. Washington
27
Adaptation in Lookout: Horvitz
24-Jul-16
Daniel S. Weld / Univ. Washington Adapted from Horvitz
28
Resulting Principles
[Horvitz CHI-99]

Decision-Theoretic Framework
Graceful degradation of service precision
 Use dialogs to disambiguate

(Considering cost of user time, attention)
24-Jul-16
Daniel S. Weld / Univ. Washington Adapted from Horvitz
29
Principles About Invocation
Allow efficient invocation & dismissal
 Timeouts minimize
cost of prediction
errors

24-Jul-16
Daniel S. Weld / Univ. Washington
30
Adapting to Small Screens
24-Jul-16
Daniel S. Weld / Univ. Washington
31
Web Site Adaptation in Proteus
[Anderson et al. WWW-01]
Architecture

Visitor
Proteus
Web server
Personalizing in two steps:

1. Learn model of visitor from access logs
2. Transform content per learned model
Hill-climbing thru space of websites



24-Jul-16
Transforms: shortcuts & elision
Decision-theoretic guidance
Daniel S. Weld / Univ. Washington
32
Guiding the Search
Expected utility based on model of visitor
Model learned by mining server access logs
Sum value of each screen
of each page
Discount by difficulty of
reaching screen from p
=p
Depends on how many
links followed and how
much scrolling required
24-Jul-16
Daniel S. Weld / Univ. Washington
33
Proteus Empirical Study
Observe real users on the desktop

Info-seeking goals drawn from random
distribution
Personalize based on observations
Measure performance on mobile device
Number of links and scrolls, amount of time
 Compare unmodified and personalized sites

•
24-Jul-16
Half users did unmodified first, others vice versa
Daniel S. Weld / Univ. Washington
34
Average number links followed
5
Average number of links followed
Unmodified
Personalized
4
3
2
1
0
cs.washington.edu
24-Jul-16
finance.yahoo.com
ebay.com
Daniel S. Weld / Univ. Washington
cnn.com cnet.com
35
Analysis of Proteus
Why Proteus worked well
Suggested useful shortcuts
Elided mostly unnecessary content
Why Proteus worked poorly
Sometimes elided useful content
Users didn’t find shortcut, tho it existed
Flaws with implementation more than
concept
24-Jul-16
Daniel S. Weld / Univ. Washington
36
Principles
Saliency of new UI operations is crucial
How name shortcuts?
Eliminating features is dangerous
Must partition dynamicity
Maintain separate dynamic & static “areas”
Always allow previous navigational methods
Duplicate functionality if necessary
Accurate prediction also crucial
24-Jul-16
Daniel S. Weld / Univ. Washington
37
Partitioned Dynamism
24-Jul-16
Daniel S. Weld / Univ. Washington
38
Partitioned Dynamism
24-Jul-16
Daniel S. Weld / Univ. Washington
39
Partitioning Failure
24-Jul-16
Daniel S. Weld / Univ. Washington
40
Principles
Must partition dynamicity
Accurate prediction also crucial
24-Jul-16
Daniel S. Weld / Univ. Washington
41
Predicting User Behavior
[Anderson et al. IJCAI-01]

Model as Sequential Process

Markov Models
P(sd) =
# times sd was followed
Total # visits to s
Mixtures of Markov Models
 Second-Order…
 Conditioning on Position in Trace
 Etc.

24-Jul-16
Daniel S. Weld / Univ. Washington
42
Weakness of Markov Models

Each state is trained independently



Abundant training data at one state cannot
improve prediction at another state
Large state models require vast training data
Problematic since Web trace data is sparse


24-Jul-16
A single visitor views ~0% of any site
New & dynamic content not in training data
Daniel S. Weld / Univ. Washington
43
Reasoning about Uncertainty
DPRM
PRM
DBN
Structure
Bayes Net
RMM
Sequence
24-Jul-16
MM
Daniel S. Weld / Univ. Washington
44
Relational Markov Models
[Anderson et al. KDD02]

Domains often contain relational
structure

Each state is a tuple in relational DB sense
ProductPage
ProductName
Apple_iMac
Palm_m505
StockLevel
in_stock
backorder
Structure enables state generalization
 Which allows learning from sparse data

24-Jul-16
Daniel S. Weld / Univ. Washington
45
Defn: Relational Markov Model
 Q: set of states
Pages in a web site
 Each state ~ a relation

ProductPage(Apple_iMac, in_stock)
 p: init prob distribution
 A: transition probability matrix
 D: a set of hierarchical domains
 R: a set of relations
24-Jul-16
Daniel S. Weld / Univ. Washington
46
Domain Hierarchies
Instance of relation
with leaf values is a
state, e.g.
ProductName
AllProducts
…
AllComputers
AllPDAs
…
ProductPage(iMac, in_stock)
AllDesktops
…
AppleDesktops …
…
24-Jul-16
iMac
…
Daniel S. Weld / Univ. Washington
47
Domain Hierarchies
Instance of relation with
non-leaf values is a set of
ProductName
AllProducts
…
AllComputers
AllPDAs
states: an abstraction, e.g.
…
ProductPage(AllComputers, in_stock)
AllDesktops
…
AppleDesktops …
…
24-Jul-16
iMac
…
Daniel S. Weld / Univ. Washington
48
E-commerce Site: Markov
RMMVer.
MainEntryPage()
ProductPage(AllProducts,
AllStockLevels)
CheckoutPage()
…
ProductPage(AllProducts,
backorder)
main.html
m505_backorder.html
checkout.html
…
ProductPage(AllProducts,
instock)
iMac_instock.html
dell4100_instock.html
24-Jul-16
Daniel S. Weld / Univ. Washington
49
RMM generalization

Want to estimate P(s



d) … but no data!
Use shrinkage
Can do this with abstractions of d and s

Let  be an abstraction of s and  of d
P( s  d )  
 P( | s) P(   ) P(d |  )


 Abs( s ) Abs( d )
s
s
24-Jul-16
dd


Daniel S. Weld / Univ. Washington
50
Calculating Shrinkage Weights
Intuitively, the  should be large when
Abstractions are more specific
Training data is abundant
Three methods for assigning weights
Uniform
Heuristic
EM
24-Jul-16
(Based on lattice depth and number of examples)
(Data intensive)
Daniel S. Weld / Univ. Washington
51
Gazelle
Avgerage negative log likelihood
8.5
RMM-uniform
RMM-rank
RMM-PET
PMM
8.0
7.5
7.0
6.5
6.0
5.5
5.0
4.5
4.0
3.5
10
24-Jul-16
100
1000
10000
Number training examples
Daniel S. Weld / Univ. Washington
100000
1000000
52
Outline
High-level
Customization
 Motivation
 Deliberative
Software Agents
 Programming by Demonstration
 Adaptive Websites
 Adaptive User Interfaces
Pure
Adaptation
24-Jul-16
Daniel S. Weld / Univ. Washington
53
The Google Generation

Most WWW traces very short


24-Jul-16
Can’t beat |trace| = 2
Not true in desktop apps
Daniel S. Weld / Univ. Washington
54
Striving for Duplex
24-Jul-16
Daniel S. Weld / Univ. Washington
55
Still Striving for Duplex
24-Jul-16
Daniel S. Weld / Univ. Washington
56
Finally!
24-Jul-16
Daniel S. Weld / Univ. Washington
57
Confirm (Twice!)
24-Jul-16
Daniel S. Weld / Univ. Washington
58
State machine (partial)
main
ok,
cancel
^P
printer,
range,
copies,
frames,
links
ok,
cancel
features
properties
print
setup
color
ok,
cancel
ok,
cancel
services
ok,
cancel
Six clicks required!
24-Jul-16
Daniel S. Weld / Univ. Washington
59
Remember: Partitioned Dynamicity
 Why Proteus worked poorly
 Users didn’t find shortcut, tho it existed
 Saliency of new UI operations is crucial
 Must partition dynamicity
 Maintain separate dynamic & static “areas”
 Duplicate functionality
24-Jul-16
Daniel S. Weld / Univ. Washington
60
With Controlled Adaptation
Maintain
Stable
Navigation
Optimize
For
User
Behavior
24-Jul-16
Daniel S. Weld / Univ. Washington
61
Or Rather…
Curry to Boolean
24-Jul-16
Daniel S. Weld / Univ. Washington
62
Future Work

Conceptual user study


What do users want?
Interface description language

Enhance Pebbles representation?
Transformation algorithms
 Implementation & experiments

24-Jul-16
Daniel S. Weld / Univ. Washington
63
Conclusion




Goal-oriented softbots
Programming by demonstration
Adaptive interfaces / websites

Pure
Adaptation
Principles

High-level
Customization
Partitioned Dynamicity
Techniques


24-Jul-16
Version-Space Algebra
Relational Markov Models
Daniel S. Weld / Univ. Washington
64
Acknowledgements

Corin Anderson
Oren Etzioni
Pedro Domingos
Keith Golden
Cody Kwok
Tessa Lau
UW AI Group

NSF, ONR, NASA, DARPA






24-Jul-16
Daniel S. Weld / Univ. Washington
65