Stanford HCI Group
Adobe Advanced Technology Labs
Two Studies of
Opportunistic
Programming: Interleaving Web
Foraging, Learning,
and Writing Code
Joel Brandt
with Philip Guo (S), Joel Leweinstein (S), Mira Dontcheva (A), Scott Klemmer (S),
http://hci.stanford.edu/opportunistic
“good grief, I don’t
even remember
the syntax for
forms!”
2
3
4
5
6
7
8
9
10
11
12
13
14
The Web has fundamentally changed
the cost structure of information
access.
Enables an opportunistic
approach to programming
15
People program opportunistically to
prototype, ideate, and discover
Emphasize speed and ease of
development over code robustness and
maintainability
16
Designers…
http://flickr.com/photos/royalsapien/2387707860/
17
Scientists…
http://flickr.com/photos/niosh/2492023651/
18
Makers…
http://www.flickr.com/photos/samthor/3198975900/
19
By 2012:
13 million who program as
part of their job…
…but only 3 million who
are
professional
programmers
[Scaffidi 2005]
20
How do programmers use
online resources?
21
Related Work
Web Search for Programmers
[Stylos 06, Hoffmann 07]
Code Cloning in Software Engineering
[Kim 04, Ducasse 99]
Learning Barriers of Programming
[Ko 04]
Learning on the Web
[Torrey 09]
22
Study 1: Prototyping
a
Web-based chat room
Study 2: Log
Analysis of
Programmers’ Web
Queries
23
METHOD
Study 1: Web Chat Room
 20 participants (9 Graduate, 11
Undergrad)
 “basic knowledge” of PHP, JS, and
HTML
 2.5 hours each
 use any resources
 3 researchers observed,
one asked questions to prompt think
aloud
24
METHOD
Chat Room Specifications
1. Set username
2. Post messages
3. Update without page reload
4. Messages have timestamp
5. Limited chat history
25
26
RESULTS
subjects
Most Subjects Met All
Specs























































































Specification Met
Specification Attempted







27
RESULTS
subjects
Frequent Web Use
time (minutes)
28
RESULTS
subjects
Frequent Web Use
time (minutes)
29
RESULTS
subjects
Frequent Web Use
time (minutes)
30
RESULTS
subjects
Frequent Web Use
time (minutes)
31
RESULTS
length (seconds)
The Web plays multiple
roles
session (sorted by length)
32
RESULTS
length (seconds)
The Web plays multiple
roles
session (sorted by length)
33
RESULTS
length (seconds)
The Web plays multiple
roles
session (sorted by length)
34
RESULTS
Three Intentions Behind Web
Use
Learning
just-in-time acquisition of new skills
Clarification
connecting high-level knowledge to implementation
details
Reminder
offloading memory to external resources
35
RESULTS
Just-in-Time Learning-byDoing
 Copy and modify code before
reading prose
36
“There’s some stuff in
this code that I don’t
really know what it’s
doing, but I’ll just try it
and see what
happens.”
37
RESULTS
Just-in-Time Learning-byDoing
 Copy and modify code before
reading prose
 Not concerned with deep
learning
38
“I think it’s less expensive for
me to just take the first code I
find and see how helpful it is
at … a very high level … as
opposed to just reading all
these descriptions and
text”
39
RESULTS
Just-in-Time Learning-byDoing
 Copy and modify code before
reading prose
 Not concerned with deep
learning
 Learn new terminology from
result snippets
40
Example
Query
Terms
Time
Number of
Refinements
Number of
Result Clicks
Types of
pages visited
Lines of
code copied
41
Learning
Example
Query
Terms
Time
Number of
Refinements
Number of
Result Clicks
Types of
pages visited
Lines of
code copied
42
Learning
Example
Query
ajax tutorial
Terms
Time
Number of
Refinements
Number of
Result Clicks
Types of
pages visited
Lines of
code copied
43
Learning
Example
Query
Terms
ajax tutorial
English
Time
Number of
Refinements
Number of
Result Clicks
Types of
pages visited
Lines of
code copied
44
Learning
Example
Query
ajax tutorial
Terms
English
Time
10s of minutes
Number of
Refinements
Number of
Result Clicks
Types of
pages visited
Lines of
code copied
45
Learning
Example
Query
ajax tutorial
Terms
English
Time
10s of minutes
Number of
Refinements
2-3
Number of
Result Clicks
Types of
pages visited
Lines of
code copied
46
Learning
Example
Query
ajax tutorial
Terms
English
Time
10s of minutes
Number of
Refinements
2-3
Number of
Result Clicks
5+
Types of
pages visited
Lines of
code copied
47
Learning
Example
Query
ajax tutorial
Terms
English
Time
10s of minutes
Number of
Refinements
2-3
Number of
Result Clicks
5+
Types of
pages visited
tutorials,
how-to articles
Lines of
code copied
48
Learning
Example
Query
ajax tutorial
Terms
English
Time
10s of minutes
Number of
Refinements
2-3
Number of
Result Clicks
5+
Types of
pages visited
tutorials,
how-to articles
Lines of
code copied
dozens
49
RESULTS
Clarification of Existing
Knowledge
 Web search enables translations
and “language analogies”
 Know exactly what code does
when they see it
 Copied code often not
immediately tested
50
Learning
Example
Query
ajax tutorial
Terms
English
Time
10s of minutes
Number of
Refinements
2-3
Number of
Result Clicks
5+
Types of
pages visited
tutorials,
how-to articles
Lines of
code copied
dozens
51
Learning
Example
Query
Clarification
ajax tutorial
Terms
English
Time
10s of minutes
Number of
Refinements
2-3
Number of
Result Clicks
5+
Types of
pages visited
tutorials,
how-to articles
Lines of
code copied
dozens
52
Example
Query
Learning
Clarification
ajax tutorial
javascript
thread
Terms
English
Time
10s of minutes
Number of
Refinements
2-3
Number of
Result Clicks
5+
Types of
pages visited
tutorials,
how-to articles
Lines of
code copied
dozens
53
Learning
Clarification
ajax tutorial
javascript
thread
Terms
English
mix of code and
English
Time
10s of minutes
Example
Query
Number of
Refinements
2-3
Number of
Result Clicks
5+
Types of
pages visited
tutorials,
how-to articles
Lines of
code copied
dozens
54
Learning
Clarification
ajax tutorial
javascript
thread
Terms
English
mix of code and
English
Time
10s of minutes
about 1 minute
Example
Query
Number of
Refinements
2-3
Number of
Result Clicks
5+
Types of
pages visited
tutorials,
how-to articles
Lines of
code copied
dozens
55
Learning
Clarification
ajax tutorial
javascript
thread
Terms
English
mix of code and
English
Time
10s of minutes
about 1 minute
Number of
Refinements
2-3
0-1
Number of
Result Clicks
5+
Types of
pages visited
tutorials,
how-to articles
Lines of
code copied
dozens
Example
Query
56
Learning
Clarification
ajax tutorial
javascript
thread
Terms
English
mix of code and
English
Time
10s of minutes
about 1 minute
Number of
Refinements
2-3
0-1
Number of
Result Clicks
5+
1-2
Types of
pages visited
tutorials,
how-to articles
Lines of
code copied
dozens
Example
Query
57
Learning
Clarification
ajax tutorial
javascript
thread
Terms
English
mix of code and
English
Time
10s of minutes
about 1 minute
Number of
Refinements
2-3
0-1
Number of
Result Clicks
5+
1-2
Types of
pages visited
tutorials,
how-to articles
API docs,
blogs, forums
Lines of
code copied
dozens
Example
Query
58
Learning
Clarification
ajax tutorial
javascript
thread
Terms
English
mix of code and
English
Time
10s of minutes
about 1 minute
Number of
Refinements
2-3
0-1
Number of
Result Clicks
5+
1-2
Types of
pages visited
tutorials,
how-to articles
API docs,
blogs, forums
Lines of
code copied
dozens
several
Example
Query
59
RESULTS
Reminding about forgotten
details
 Explicitly choose not to learn
things
 Search serves as “Web-enabled
auto-complete”
60
61
62
RESULTS
Reminding about forgotten
details
 Explicitly choose not to learn
things
 Search serves as “Web-enabled
auto-complete”
 Despite utility, can break flow
63
Learning
Clarification
ajax tutorial
javascript
thread
Terms
English
mix of code and
English
Time
10s of minutes
about 1 minute
Number of
Refinements
2-3
0-1
Number of
Result Clicks
5+
1-2
Types of
pages visited
tutorials,
how-to articles
API docs,
blogs, forums
Lines of
code copied
dozens
several
Example
Query
64
Learning
Clarification
ajax tutorial
javascript
thread
Terms
English
mix of code and
English
Time
10s of minutes
about 1 minute
Number of
Refinements
2-3
0-1
Number of
Result Clicks
5+
1-2
Types of
pages visited
tutorials,
how-to articles
API docs,
blogs, forums
Lines of
code copied
dozens
several
Example
Query
Reminder
65
Learning
Clarification
Reminder
ajax tutorial
javascript
thread
fetch_array php
Terms
English
mix of code and
English
Time
10s of minutes
about 1 minute
Number of
Refinements
2-3
0-1
Number of
Result Clicks
5+
1-2
Types of
pages visited
tutorials,
how-to articles
API docs,
blogs, forums
Lines of
code copied
dozens
several
Example
Query
66
Learning
Clarification
Reminder
ajax tutorial
javascript
thread
fetch_array php
Terms
English
mix of code and
English
mostly code
Time
10s of minutes
about 1 minute
Number of
Refinements
2-3
0-1
Number of
Result Clicks
5+
1-2
Types of
pages visited
tutorials,
how-to articles
API docs,
blogs, forums
Lines of
code copied
dozens
several
Example
Query
67
Learning
Clarification
Reminder
ajax tutorial
javascript
thread
fetch_array php
Terms
English
mix of code and
English
mostly code
Time
10s of minutes
about 1 minute
less than 1
minute
Number of
Refinements
2-3
0-1
Number of
Result Clicks
5+
1-2
Types of
pages visited
tutorials,
how-to articles
API docs,
blogs, forums
Lines of
code copied
dozens
several
Example
Query
68
Learning
Clarification
Reminder
ajax tutorial
javascript
thread
fetch_array php
Terms
English
mix of code and
English
mostly code
Time
10s of minutes
about 1 minute
less than 1
minute
Number of
Refinements
2-3
0-1
0
Number of
Result Clicks
5+
1-2
Types of
pages visited
tutorials,
how-to articles
API docs,
blogs, forums
Lines of
code copied
dozens
several
Example
Query
69
Learning
Clarification
Reminder
ajax tutorial
javascript
thread
fetch_array php
Terms
English
mix of code and
English
mostly code
Time
10s of minutes
about 1 minute
less than 1
minute
Number of
Refinements
2-3
0-1
0
Number of
Result Clicks
5+
1-2
0-1
Types of
pages visited
tutorials,
how-to articles
API docs,
blogs, forums
Lines of
code copied
dozens
several
Example
Query
70
Learning
Clarification
Reminder
ajax tutorial
javascript
thread
fetch_array php
Terms
English
mix of code and
English
mostly code
Time
10s of minutes
about 1 minute
less than 1
minute
Number of
Refinements
2-3
0-1
0
Number of
Result Clicks
5+
1-2
0-1
Types of
pages visited
tutorials,
how-to articles
API docs,
blogs, forums
API docs,
result snippets
Lines of
code copied
dozens
several
Example
Query
71
Learning
Clarification
Reminder
ajax tutorial
javascript
thread
fetch_array php
Terms
English
mix of code and
English
mostly code
Time
10s of minutes
about 1 minute
less than 1
minute
Number of
Refinements
2-3
0-1
0
Number of
Result Clicks
5+
1-2
0-1
Types of
pages visited
tutorials,
how-to articles
API docs,
blogs, forums
API docs,
result snippets
Lines of
code copied
dozens
several
none to several
Example
Query
72
How do these results generalize?
73
METHOD
Study 2: Query Log
Analysis
 Adobe Community Help logs from July 2008
 ~25,000 programmers performing 101,289
queries about Adobe Flex Framework
 Grouped queries into 65,955 search sessions
 Analyzed three properties of sessions:
 Format of query strings
 Process of query refinement
 Types of Web pages visited
 Hand-coded 300 sessions for intent
74
Query format predicts
types of pages visited
75
RESULTS
Query format predicts types of pages
visited
Page type
code only
English + code
English only
Adobe APIs
38%
16%
10%
Adobe tutorials
31%
33%
39%
blogs
15%
22%
19%
3%
7%
6%
13%
22%
27%
forums
unclassified
All inter-row differences significant at p < 0.001
76
RESULTS
Query format predicts types of pages
visited
Page type
code only
English + code
English only
Adobe APIs
38%
16%
10%
Adobe tutorials
31%
33%
39%
blogs
15%
22%
19%
3%
7%
6%
13%
22%
27%
forums
unclassified
All inter-row differences significant at p < 0.001
77
RESULTS
Query format predicts types of pages
visited
Page type
code only
English + code
English only
Adobe APIs
38%
16%
10%
Adobe tutorials
31%
33%
39%
blogs
15%
22%
19%
3%
7%
6%
13%
22%
27%
forums
unclassified
All inter-row differences significant at p < 0.001
78
RESULTS
Query format predicts types of pages
visited
Page type
code only
English + code
English only
Adobe APIs
38%
16%
10%
Adobe tutorials
31%
33%
39%
blogs
15%
22%
19%
3%
7%
6%
13%
22%
27%
forums
unclassified
All inter-row differences significant at p < 0.001
79
Reminder sessions are typically
code-only queries
80
RESULTS
Query format indicative of
intention
query format
reminding
learning
code only
56%
21%
English + code
10%
29%
English only
34%
50%
All inter-row differences significant at p < 0.05
81
Programmers rarely need to
refine queries
82
RESULTS
number of queries
Programmers refine rarely
query number in session
83
RESULTS
percent of queries
…and they start out with either words or
code
query number in session
84
Five Key Insights
for Tool Design
85
Just-in-time learning is common
Create tutorials that teach
“how the code works”
86
Copied code often not immediately
tested
Demarcate copied code,
highlight what is executed
87
Web search often used as a “translator”
Improve search ranking with synonyms,
perform searches automatically
88
Programmers choose not to learn
syntax
Make Web-enabled auto-complete
89
Query refinement is often predictable
Automatically augment queries
90
Preview of Ongoing Work
Blueprint
Enabling Example-Centric
Programming
91
92
Stanford HCI Group
Adobe Advanced Technology Labs
http://hci.stanford.edu/opportunistic
95
Learning
Clarification
Reminder
ajax tutorial
javascript
thread
fetch_array php
Terms
English
mix of code and
English
mostly code
Time
10s of minutes
about 1 minute
less than 1
minute
Number of
Refinements
2-3
0-1
0
Number of
Result Clicks
5+
1-2
0-1
Types of
pages visited
tutorials,
how-to articles
API docs,
blogs, forums
API docs,
result snippets
Lines of
code copied
dozens
several
none to several
Example
Query
96