Automatically Personalizing
User Interfaces
Daniel Weld
University of Washington

10-Apr-20
Acknowledgements








Corin Anderson
Oren Etzioni
Pedro Domingos
Krzysztof Gajos
Keith Golden
Cody Kwok
Tessa Lau
UW AI Group

NSF, ONR, NASA, DARPA, WRF
Daniel S. Weld / Univ. Washington 2

Two Interface Trends
Usage
10-Apr-20
4x
Variation
625x
Daniel S. Weld / Univ. Washington 3

“Steelcase-Inspired Software”
-- David Gelernter




• Beyond inconsistent defaulting
Adapt to available devices, interaction modes
Adapt to user location, tasks & goals
Adapt to user calendar & current time
Adapt to network connectivity, …

Need Increased Customization

• Beyond changing buttons on the toolbar
Overriding inappropriate adaptation


High-level functionality
Programming by demonstration
10-Apr-20 Daniel S. Weld / Univ. Washington 4


Consistent Across Applications

Adaptation in every ‘dialog’

Bridging Applications

Data gathering

Transformations

Deep Deployment ~ OS layer
10-Apr-20 Daniel S. Weld / Univ. Washington 5

Outline
High-level
Customization

Motivation

Deliberative Software Agents

Programming by Demonstration

Adapting User Interfaces to

Screen Size
Pure

Adaptation
User Behavior
10-Apr-20 Daniel S. Weld / Univ. Washington 6

Genesis: Internet Softbots
[Etzioni & Weld CACM 94]

Interface Principles:




Goal-Oriented
Integrated
Balanced
Safe
User says what she wants
Agent determines when
& to achieve it how &
DT-lite: “Softbot must
 planning-based softbots information on its own with the nuisance of asking the
10-Apr-20 Daniel S. Weld / Univ. Washington 7





Rodney Specifying goals is bottleneck
Logical spec. language ?!$@#!
Forms interface
Simon
• Limited to anticipated use
BargainFinder
MetaCrawler
Grouper
• Not scalable
Info Manifold
Often easier to just do task!
• Except: data integration tasks
Ahoy
ShopBot
Occam Wrapper Induction
ILA
LSD
Razor

Natural language interfaces

[Popescu et al; Yates et al 03]
Tukwila
Piazza

Programming by demonstration
10-Apr-20 Daniel S. Weld / Univ. Washington
…
KnowItAll
Mulder
GLUE
8

Outline
High-level
Customization

Motivation

Deliberative Software Agents

Programming by Demonstration

Adapting User Interfaces to

Screen Size
Pure

Adaptation
User Behavior
10-Apr-20 Daniel S. Weld / Univ. Washington 9

Programming by Demonstration
[Lau & Weld IUI-99]
[Wolfman et al. IUI-01]
[Lau et al. ICML-01]



If it’s too hard for users to specify goals
Let’s watch them…


And try to help
Like plan recognition
Initial domains:



Email
Text editing
Cross domain
10-Apr-20 Daniel S. Weld / Univ. Washington 10

Objectives
Macro Record
On / Off Straight Ln Brittle
VBasic Programing Turing Cplt Robust
Previous PBD Little Varying Varying
SmartEdit
SmartPython
On / Off Loops,Cond Robust
None
None
Lots
Minimal
10-Apr-20 Daniel S. Weld / Univ. Washington 11

SmartEdit 1
[Lau et al. ICML-01]
10-Apr-20 Daniel S. Weld / Univ. Washington 12

2
10-Apr-20 Daniel S. Weld / Univ. Washington 13

3
10-Apr-20 Daniel S. Weld / Univ. Washington 14

4
10-Apr-20 Daniel S. Weld / Univ. Washington 15

5
10-Apr-20 Daniel S. Weld / Univ. Washington 16

6
10-Apr-20 Daniel S. Weld / Univ. Washington 17

PBD as a learning problem



Action is function : input state  output state
Editor state: text buffer, cursor position, etc.
Actions: move, select, delete, insert, cut, paste,…
Move to next
<!--
Delete to next
-->



Given a state sequence, infer actions
Many actions may be consistent with one example
Challenge: Weak bias + low sample complexity
10-Apr-20 Daniel S. Weld / Univ. Washington 18

Version Space Algebra




Version space = set of complex functions
Define version space hierarchically
Algebraic operators for combing VSs
• Union  : analogous to set union
• Join : cross-product with consistency predicate
• Transform : convert functions to different types
Can factor a version space
10-Apr-20 Daniel S. Weld / Univ. Washington 19

SMARTedit's version space
…
…
…
…
10-Apr-20 Daniel S. Weld / Univ. Washington 20

Fast Consistency


Test consistency of example against entire version space
Quickly prune subtrees
Action
Move Paste
Insert
Copy

Select Cut
Delete

Innovations:
Independent join allows BSR representation
10-Apr-20 Daniel S. Weld / Univ. Washington 21

Preliminary User Study




6 undergrad CS majors
7 repetitive tasks with & later w/out SMARTedit
Tasks: 4 to 27 iterations, 1-5 min to complete



Evaluation metrics:
Time saved completing task with SMARTedit's help
% user actions (keyboard + mouse) saved
User feedback
10-Apr-20 Daniel S. Weld / Univ. Washington 22

Time saved using SMARTedit
300
250
200
150
100
50
0
- 50
- 100
- 150
- 200
1 2 3 4
X X
5 6
A
B
C
D
E
F
Six
Users
10-Apr-20 Daniel S. Weld / Univ. Washington 23

Action Savings with SMARTedit
100
80
60
40
20
0
- 20
- 40
- 60
1 2 3 4
X X
XXXX
5 6
D
E
A
B
C
F
Six
Users
10-Apr-20 Daniel S. Weld / Univ. Washington 24

Observations from PBD

Overhead of macro recorder UI is high


Most repetitive tasks short
How many shell scripts do you write / day?

Should focus on pure adaptivity

E.g., automatic segmentation
10-Apr-20 Daniel S. Weld / Univ. Washington 25

Outline
High-level
Customization

Motivation

Deliberative Software Agents

Programming by Demonstration

Adapting User Interfaces to

Screen Size
Pure

Adaptation
User Behavior
10-Apr-20 Daniel S. Weld / Univ. Washington 26

Early Adaptation: Mitchell,Maes

Predict: Email message priorities
Meeting locations, durations


Principle 1: Defaults minimize cost of errors
Principle 2: Allow users to adjust thresholds
10-Apr-20 Daniel S. Weld / Univ. Washington 27

Adaptation in Lookout: Horvitz
10-Apr-20 Daniel S. Weld / Univ. Washington Adapted from Horvitz 28


Resulting Principles
[Horvitz CHI-99]
Decision-Theoretic Framework


Graceful degradation of service precision
Use dialogs to disambiguate
(Considering cost of user time, attention)
10-Apr-20 Daniel S. Weld / Univ. Washington Adapted from Horvitz 29

Principles About Invocation

Allow efficient invocation & dismissal

Timeouts minimize cost of prediction errors
10-Apr-20 Daniel S. Weld / Univ. Washington 30

Adapting to Device Characteristics
Func
Interface
Spec
+ Device
Model
Custom
Interface
Rendering
Hierarchy of
State vars +
Methods
Screen size
Approaches:
•Templates
•Expert System
•Optimization
10-Apr-20 Daniel S. Weld / Univ. Washington 31

Optimize Widgets & Layout


Available widgets = F(type(state-var))
E.g. for an integer

For hierarchy

Tabs, adjacent panes, …
10-Apr-20 Daniel S. Weld / Univ. Washington 32



SUPPLE
[Gajos & Weld, draft 2003]
Optimization methods


Branch & bound vs. local search
CSP: MRV, FC
Cost function = minimize user effort


Navigation + manipulation
Component importance (  trace data)
10-Apr-20 Daniel S. Weld / Univ. Washington 33

SUPPLE Output
10-Apr-20 Daniel S. Weld / Univ. Washington 34

SUPPLE Output
10-Apr-20 Daniel S. Weld / Univ. Washington 35

SUPPLE Output



260,000,000 possibilities
Simulated annealing fastest
Can improve with bin-packing methods
10-Apr-20 Daniel S. Weld / Univ. Washington 36

Web Pages on Small Screens
10-Apr-20 Daniel S. Weld / Univ. Washington 37

Web Site Adaptation in Proteus
[Anderson et al. WWW-01]
Visitor Proteus Web server


Personalizing in two steps:
1. Learn model of visitor from access logs
2.
Transform content per learned model


Hill-climbing thru space of websites
Transforms: shortcuts & elision
Expected utility of candidate site =
• Sum of utility of each screen of each page
• Discounted by access difficulty (#links, scrolling)
10-Apr-20 Daniel S. Weld / Univ. Washington 38

Analysis of Proteus
 Why Proteus worked well
 Suggested useful shortcuts
 Elided mostly unnecessary content
 Why Proteus worked poorly
 Sometimes elided useful content
 Users didn’t find shortcut, tho it existed
 Flaws with implementation, not concept
10-Apr-20 Daniel S. Weld / Univ. Washington 39

Principles
 Eliminating features is dangerous
 Accurate prediction is crucial
 Saliency of new UI operations is crucial
 How name shortcuts?
 Must partition dynamicity
 Maintain separate dynamic & static “areas”
 Always allow previous navigational methods
 Duplicate functionality if necessary
10-Apr-20 Daniel S. Weld / Univ. Washington 40

Partitioned Dynamism
10-Apr-20 Daniel S. Weld / Univ. Washington 41

Partitioned Dynamism
10-Apr-20 Daniel S. Weld / Univ. Washington 42

Partitioning Failure
10-Apr-20 Daniel S. Weld / Univ. Washington 43

Principles
 Must partition dynamicity
 Accurate prediction also crucial
10-Apr-20 Daniel S. Weld / Univ. Washington 44

Predicting User Behavior
[Anderson et al. IJCAI-01]

Model as Sequential Process

Markov Models
# times s
 d was followed
P ( s
 d ) =
Total # visits to s



10-Apr-20

Mixtures of Markov Models
Second-Order…
Conditioning on Position in Trace
Etc.
Daniel S. Weld / Univ. Washington 45

Weakness of Markov Models

Each state is trained independently

Abundant training data at one state cannot improve prediction at another state

Large state models  vast training data

Problem: Web trace data is sparse


A single visitor views ~0% of any site
New & dynamic content not in training data
10-Apr-20 Daniel S. Weld / Univ. Washington 46

Reasoning about Uncertainty
PRM
Bayes Net DBN
DPRM
Sanghai et al.
Thurs 12:10
Cholula 3
RMM
10-Apr-20
Sequence
Markov Model
Daniel S. Weld / Univ. Washington 47

Relational Markov Models
[Anderson et al. KDD02]

Domains often contain relational structure

Each state is a tuple in relational DB sense
ProductPage
ProductName
Apple_iMac
Palm_m505
StockLevel in_stock backorder


Structure enables state generalization
Which allows learning from sparse data
10-Apr-20 Daniel S. Weld / Univ. Washington 48


Q : set of states


Pages in a web site
Each state ~ a relation
ProductPage(Apple_iMac, in_stock)
 p : init prob distribution

A : transition probability matrix

D : a set of hierarchical domains

R : a set of relations
10-Apr-20 Daniel S. Weld / Univ. Washington 49

Domain Hierarchies
ProductName
AllProducts
Instance of relation with leaf values is a state , e.g.
… AllComputers …
ProductPage(iMac, in_stock)
AllPDAs
AllDesktops
…
10-Apr-20
…
AppleDesktops … iMac …
Daniel S. Weld / Univ. Washington 50

Domain Hierarchies
…
ProductName
AllPDAs
AllProducts
AllComputers …
Instance of relation with non-leaf values is a set of states: an abstraction , eg
ProductPage(AllComputers, in_stock)
AllDesktops
…
10-Apr-20
…
AppleDesktops … iMac …
Daniel S. Weld / Univ. Washington 51

MainEntryPage() main.html
ProductPage(AllProducts,
AllStockLevels)
…
ProductPage(AllProducts, backorder) m505_backorder.html
CheckoutPage() checkout.html
10-Apr-20 instock) iMac_instock.html
dell4100_instock.html
Daniel S. Weld / Univ. Washington 52

RMM generalization

Want to estimate P(s

d) … but no data!

Use shrinkage
10-Apr-20 s d
Daniel S. Weld / Univ. Washington 53

RMM generalization


Want to estimate P(s

d) … but no data!

Use shrinkage
Can do this with abstractions of d and s

Let  be an abstraction of s and  of d
10-Apr-20
 s d

Daniel S. Weld / Univ. Washington 54

RMM generalization


Want to estimate P(s

d) … but no data!

Use shrinkage
Can do this with abstractions of d and s

Let  be an abstraction of s and  of d
P (

| s ) P (
  
) P ( d |

) s s d d 

10-Apr-20 Daniel S. Weld / Univ. Washington 55

RMM generalization


Want to estimate P(s

d) … but no data!

Use shrinkage
Can do this with abstractions of d and s

Let  be an abstraction of s and  of d
P ( s
 d )

 
 

Abs ( s )

Abs (

 d )
P (

| s ) P (
  
) P ( d |

) s d
10-Apr-20 Daniel S. Weld / Univ. Washington 56

Calculating Shrinkage Weights
 Intuitively, the 
 should be large when
 Abstractions are more specific
 Training data is abundant
 Three methods for assigning weights
 Uniform
 Heuristic (Based on lattice depth, # of examples)
 EM (Data intensive)
10-Apr-20 Daniel S. Weld / Univ. Washington 57

8.5
8.0
7.5
7.0
6.5
6.0
5.5
5.0
4.5
4.0
3.5
10
Gazelle
10-Apr-20
100 1000 10000
Number training examples
Daniel S. Weld / Univ. Washington
RMM-uniform
RMM-rank
RMM-PET
PMM
100000 1000000
58

The Google Generation

Most WWW traces very short

Can’t beat |trace| = 2

Not true in desktop / mobile apps


Adapt to user behavior
Adapt to device characteristics
10-Apr-20 Daniel S. Weld / Univ. Washington 59

Striving for Duplex
10-Apr-20 Daniel S. Weld / Univ. Washington 60

Still Striving for Duplex
10-Apr-20 Daniel S. Weld / Univ. Washington 61

Finally!
10-Apr-20 Daniel S. Weld / Univ. Washington 62

Confirm (Twice!)
10-Apr-20 Daniel S. Weld / Univ. Washington 63

State machine (partial) main ok, cancel
^P printer, range, copies, frames, links properties ok, cancel print setup ok, cancel ok, cancel features services
Six clicks required!
ok, cancel
10-Apr-20 Daniel S. Weld / Univ. Washington color
64

Remember: Partitioned Dynamicity
 Proteus: Users didn’t find shortcuts!
 Saliency of new UI operations is crucial
 Must partition dynamicity
 Maintain separate dynamic & static “areas”
 Duplicate functionality
10-Apr-20 Daniel S. Weld / Univ. Washington 65

With Controlled Adaptation
Maintain
Stable
Navigation
Optimize
For
User
Behavior
10-Apr-20 Daniel S. Weld / Univ. Washington 66

Or Rather…
10-Apr-20
Curry to Boolean
Daniel S. Weld / Univ. Washington 67

Challenges for AI




Formalizing interface descriptions

Pebbles, iCrafter, XIML, …
Customization languages

Rec start/stop; shortcut; make default
Datamining the clickstream

Improved prediction;  dim +  data
Interface transformation algorithms

Complex constraint satisfaction
10-Apr-20 Daniel S. Weld / Univ. Washington 68

Conclusion



Goal-oriented softbots
Programming by demonstration
Adaptive interfaces / websites
High-level
Customization
Pure
Adaptation

Principles

Partitioned Dynamicity

Techniques


Version-Space Algebra
Relational Markov Models
10-Apr-20 Daniel S. Weld / Univ. Washington 69

The End
10-Apr-20 Daniel S. Weld / Univ. Washington 70

10-Apr-20
Abstract

Today’s computer interfaces are one size fits all. Users with little programming experience have very limited opportunities to customize an interface to their task and work habits.
Furthermore, the overhead induced by generic interfaces will be proportionately greater on small form-factor PDAs, embedded applications and wearable devices. Searching for a solution, researchers argue that productivity can be greatly enhanced if interfaces anticipated their users, adapted to their preferences, and reacted to high-level customization requests. But realizing these benefits is tricky, because there is an inherent tension between the dynamism implied by automatic interface adaptation and the stability required in order for the user to predict the computer’s behavior and maintain control. This talk will list challenges for the field, describe principles governing effective adaptation, and present new algorithms for data mining user action traces and dynamically transforming interfaces.
Daniel S. Weld / Univ. Washington 71

Gentle-Slope Systems
C
Click’nCreate
VBasic
Hypercard
Difficulty of use
10-Apr-20
Ideal
Sophistication of what can be created
Daniel S. Weld / Univ. Washington Adapted from Myers et al. 72

Learning programs from traces

Illustrated VS algebra with SMARTedit



Grand vision: cross-domain PBD
Learn an abstract programming language
SMARTpython: learn Python programs


Lau PhD thesis
Loops, conditionals, …
10-Apr-20 Daniel S. Weld / Univ. Washington 73

Req’s for Interface Representation

Same as Pebbles Project?




State variables
Typing
Commands
Hierarchical organization
10-Apr-20 Daniel S. Weld / Univ. Washington 74

Adaptation in Pebbles
See next talk…
10-Apr-20 Daniel S. Weld / Univ. Washington 75

State abstractions form lattice
ProductPage(AllProducts, AllStockLevels)
ProductPage(AllProducts, in_stock) ProductPage(AllComputers, AllStockLevels)
ProductPage(AllComputers, in_stock) ProductPage(AllDesktops, AllStockLevels)
ProductPage(AllDesktops, in_stock) ProductPage(AppleDesktops, AllStockLevels)
ProductPage(AppleDesktops, in_stock) ProductPage(iMac, AllStockLevels)
10-Apr-20
ProductPage(iMac, in_stock)
Daniel S. Weld / Univ. Washington 76

Guiding the Search
 Expected utility based on model of visitor
 Model learned by mining server access logs
 Sum value of each screen of each page
= p
 Discount by difficulty of reaching screen from p
 Depends on how many links followed and how much scrolling required
10-Apr-20 Daniel S. Weld / Univ. Washington 77

Specifics of Screen Evaluation
Expected utility of s ij
Similarity to past content
Frequency of past visits
E [ U ( s ij
)]

 sim sim ( s ij
)
  freq freq ( s ij
)

Intrinsic utility
P ( scroll
 links
[ P (
L k
L k
)
) ( E [ U ( s i , j

1
)]
( E [ U ( L k
.
dest )]
Probability
 
 
Expected utility
( scroll ))
( L k
))]

Cost of action taken of destination taking action
Extrinsic utility
10-Apr-20 Daniel S. Weld / Univ. Washington 78

Proteus Empirical Study
 Observe real users on the desktop

Info-seeking goals drawn from random distribution
 Personalize based on observations
 Measure performance on mobile device


Number of links and scrolls, amount of time
Compare unmodified and personalized sites
•
Half users did unmodified first, others vice versa
10-Apr-20 Daniel S. Weld / Univ. Washington 79

Average number links followed
5
Unmodified
Personalized
4
3
2
1
0
10-Apr-20 cs.washington.edu
finance.yahoo.com
ebay.com
Daniel S. Weld / Univ. Washington cnn.com cnet.com
80

RMM generalization


Want to estimate P(s

d)

Use shrinkage
Can do this with abstractions of d and s

Let  be an abstraction of s and  of d
P ( s
 d )

  

Abs ( s )
 
Abs ( s d
P (
  
)



)
P ( ( | | s s ) ) P ( (

 i
 



P ( i d
|



)
 j
 
P ( i

) ) P ( ( j )



| | ) )
10-Apr-20 Daniel S. Weld / Univ. Washington 81

Backup


Want to estimate P(s

d) … but no data!

Use shrinkage
Can do this with abstractions of d and s

Let  be an abstraction of s and  of d
P ( s
 d )

 
 

Abs ( s )

Abs (

 d )
( (

| | s s ) ) P ( ( ) ) P ( ( d | | ) ) s s d d 

10-Apr-20 Daniel S. Weld / Univ. Washington 82