>>:
I'd like to introduce Joel Brandt, who's a Ph.D.
student at Stanford.
hint.
He is graduating in a year, hint,
And here to talk with us about something I think
that's is really fascinating that he calls opportunistic
programming.
And take it away.
>> Joel Brandt:
Thank you, Gina.
So, good morning.
Like
Gina said, my name is Joel Brandt, a Ph.D. candidate at
Stanford.
I'm in the Human Computer Interaction Group
there I'm also an intern researcher in the Adobe Creative
Technologies lab.
So the sentence I'm about to say is
probably the most important in my talk.
None of this would
have been possible without the help of my wonderful
collaborators.
At Stanford I worked with Philip Guo who
actually was an intern here this summer, Joel Lewenstein,
William Choi, and my adviser Scott Klemmer.
And Adobe I've
had the pleasure of working with, very closely with Mira
Dontacheva and also with Marcos Weskamp as part of David
Salesin's research group.
And I really want to thank Gina again for
setting up everything for my visit today.
It's so far been
awesome and I -- and I don't expect that to change in the
next few hours.
So today's topic will largely be about how
programmers use the web, both understanding how they are
using it now and understanding how we can build tools that
support that.
And so I'd like to start out by presenting a
vignette from one of our exploratory studies.
So, Jenny was working on building a user
interface for a web-base the chat room.
So she was staring
at a blank E-MAX window and she said, "Good grief, I don't
even remember the syntax for forms."
Now, Jenny is a great programer, in fact
she's a Ph.D. candidate, and if she see saw the syntax in
HTML form, she'd know exactly how to use them.
She doesn't
need to learn anything, she just needs an external memory
aide.
Web search makes finding this very straightforward.
She searched Google for HTML forms.
She clicks on the
first result, and she quickly scrolls to code on the page.
A quick copy and paste later, and a few typing changes, and
now she's already got a mock up of the user interface.
All this took about 60 seconds, something
that would have taken orders of magnitude longer in the
days before the web.
In short, we believe that the web has
fundamentally changed programming practices.
It is now
feasible for programmers to build entire applications by
interactively searching for, modifying, combining the short
box and code.
And this enables what we call an
opportunistic approach to programming, where individuals
inner weave this web foraging learning and writing code.
So we've defined, and actually folks at
Microsoft have also designed opportunistic programming as
the kind of programming that's done to prototype, ideate
and discover.
It emphasizes speed and ease of development
over code robustness and maintainability.
It's by no means
the only type of programming that gets done, but we believe
a large portion of code today is written in this manner.
It's certainly done by professional software developers,
but it's also done by amateurs by designers mocking up user
interfaces, by scientists writing code to run experiments,
and by weekend hackers building the next generation of
physical computing devices.
So Skafitti Shawn Meyers recently estimated
that by 2012, which is coming up, there will be thirteen
million people that do programming as part of their job,
but only 3 million of these people will be called full-time
programmers.
So we believe that for the most part, these
other ten million people will be primarily engaged in this
sort of opportunistic programming.
And we believe there's
a significant value in understanding and designing for this
large population of amateur programmers.
Sure.
>>:
We've got some questions.
>> Joel Brandt:
>>:
You can ask them now, sure.
Your definition, can you just deliver it one more
time, what is the definition of professional programer?
>> Joel Brandt:
So, I mean, I don't think that there's a
strict dichotomy here.
I mean, I think, you know, it's
certainly a sliding scale.
I think there's people that
would self-describe as professional programmers, and those
are people that work on development teams and, you know,
sit down and roughly 40 hours a week write code.
And then
there's lots and lots of other people that just fall into
the practice of coding.
Either, you know, they start out
using Excel or, you know, like I said, they are scientists
and they said, "I've got to hook this thing up to this
thing, and so I'm going to write some code."
You know,
these people that have technical acumen and are willing to
jump into development, but approach it from a different
mind-set.
They don't need code that works all the time,
they just need code that runs once.
And, you know, that's
a perfectly valid reason to write code.
>>:
I'm just a little concerned because, you know, if you
define in a professional program somebody who doesn't know
writing code for money, you know.
>> Joel Brandt:
Uh-huh.
>>:
Or fun, but basically 40 hours a week, it's pretty
much their full-time job.
>> Joel Brandt:
>>:
Uh-huh.
And then you define nonprofessional as just, you know,
people who do it in the meantime.
Is that a fair
definition?
>> Joel Brandt:
it.
I don't think that's how I would define
I wouldn't say it's something that's done in their
free time.
I mean, I think for lots of people, this sort
of amateur programming is done as -- as part of in-service
of their job, right?
It's sort of -- I mean, another way
you could draw this distinction in another domain is, you
know, how things were with mathematicians, right?
There's
professional mathematicians, and they do math all the time,
right, in whatever their area is.
There's also lots of
people that do math as part of their job.
statistics.
whatever.
job.
They do
Maybe they do statistics terribly, I mean, but
They do those sorts of things as part of their
And we don't think it's weird that math is just
something that everybody takes in college.
But, you know,
we are just getting to this point where computer science or
programming is becoming that thing.
It's a tool that
people use even if that isn't their, you know, sole
profession.
Does that sort of clear things up?
So anyway, so this observation leads to the
two main researchers that I'll address in my talk today.
First, how do programmers use online resources?
And
second, how can we build new programming interfaces that
support and more importantly amplify the role the web plays
in software development?
So the remainder of my talk will
be divided into two parts addressing each of these
questions in turn.
To answer the first question, so we
conducted a -- two exploratory studies, one in the lab and
one in the wild.
Our first study observed 20 individuals
in the lab while they prototyped a web-based chat room.
In
this study, we found that the web played a principle role
in all participants' development practices and participants
used the web with a varying range of intentions.
To understand if these results generalized,
we conducted a second study where we analyzed a month of
queries to an online programming search engine.
And we also saw traits in this study that
suggested what was being used for this spectrum of
intentions and found that a query's lexical structure, the
way queries were refined and how people visited page were
very indicative of their different types of intentions.
So in our laboratory study, like I said we
observed 20 participants prototyping a web chat room.
When
recruiting, we asked that all participants have a basic
knowledge of PHP, javascript, and HTML; however, all but
two participants rated themselves at novices in at least
one of these technologies.
Each session lasted two and a half hours
and participants were compensated $99.
Participants were allowed to use any
resource while programming.
They were told ahead of time
that they could bring books or codes to the session.
participant brought code on his laptop.
One
No participants
brought any printed materials, and all participants used
the web.
>>:
[inaudible.]
>> Joel Brandt:
information.
So we gave them a little bit of
We said they would be building a web
application in PHP, javascript, or HTML.
We didn't say,
"It's going to be an AJAX web chat room."
>>:
[inaudible] brought code.
>> Joel Brandt:
Yes.
So he -- I mean, he -- he -- he --
you know, he -- I wouldn't say he selected code to bring
along.
He brought along his laptop which had code on it.
He brought it along because he knew that he could go and
look at it.
>>:
Yeah.
Does that clarify things?
>> Joel Brandt:
So three researchers, one of them actually
was Philip, and three researchers observed all the
participants.
And one researcher, myself, asked open-ended
questions as they programmed to get them to think allowed
about particular aspects of what they were doing
specifically relating to how they were using the web when
they were engaged in programming.
The chat room that we asked them to build,
we asked them to try to meet five specifications.
First,
the user of the chat room needed to be able to set their
user name and obviously post messages.
Message updates needed to happen without a
page reload, so obviously they should use AJAX, but we
didn't use that term when telling them.
Messages should also have a time stamp.
And the chat history, the length of the
chat window should be limited to ten lines.
This last requirement was somewhat unique.
You know, that's not how most chat clients work, but we
added it so that all participants would have to write some
code.
They couldn't just go and get a PHP application
that -- that does this chat and be done.
So here is one participant's chat room that
met all the specifications.
And indeed, most -- most
participants met all of the specifications.
So this table lists one subject per row,
and shows what specifications they met.
And an unfilled
circle means they started on a specification, but didn't
complete it.
As you can see, 13 of the 20 participants
met them all, and most met almost all of them.
And, well, we don't have hard data on this
because there's really no way to test it.
We believe that
only one participant could have met all of the
specifications without being allowed to use the web as a
resource.
So, as I mentioned, all the participants
used the web, but how frequently did they use it?
So the
Gann Turner I'm about to show spans the length of the
entire study and has one line per subject.
And
participants' web usage will be shown as white bars.
So that's a ton of data.
And for the
purposes of this talk, there's really only a few things
that I'd like you to get out of this chart.
First,
participants used the web a lot and used it throughout the
entire course of this study.
On average, they use the web
19 percent of the time they were writing code.
So about
25 minutes and they averaged 18 distinct web use sessions.
Second, the length of the sessions varied
greatly.
Some lasted tens of minutes like this one that's
highlight, and some lasted just a few seconds, like the one
that's highlighted in the upper right.
So what is the distribution of session
links look like?
Suppose we sort all of the web sessions
by length and then plot the length of each session in
order.
Now, if all the web uses were for basically the
same purpose, you might expect a roughly uniform
distribution.
Instead, we see a parallel-like
distribution.
This suggests that the web may be playing
very different roles in these different sessions.
Indeed,
when we were looking at the data ->>:
[inaudible] what exactly they were doing during those
sessions?
>> Joel Brandt:
I mean, we watched them work.
And so we
tracked them in a qualitative way.
>>:
So do you know, for example, what that person did for
10 minutes versus the person that did like in 5 seconds.
>> Joel Brandt:
Yes.
Stay tuned for the rest of the talk.
That's what the next 10 minutes or so we'll be talking
about.
So when analyzing the data, you know, you
need to figure out some way of breaking down ->>:
I didn't know you were going to go into this, but do
you have [inaudible] that three or four of the participants
didn't finish.
>> Joel Brandt:
>>:
Uh-huh.
Is there a patent in their web use that correlates
with ->> Joel Brandt:
Well, actually, I'd love to talk more
about this, we should maybe take it offline.
I think the
reasons they didn't finish had less to do with their
particular web-use strategies and more to do with their
overall programming strategies.
So interestingly, the
most -- I would say the -- it's hard to generalize from
three data points of people that didn't finish, right?
But
the thing that was the strongest predictor of not finishing
is if they chose to start with a large body of code, like,
you know, use something like a content management system
like Drupal and build on top of that.
counterintuitive, right?
So that's sort of
You'd think if they started with
something that was almost what they wanted, that would be a
huge benefit.
In fact, we saw them make very, very rapid
initial progress.
So one of the participants that didn't
finish had met, I think, four of the five specifications in
10 minutes.
But the last specification, the one we added
where we asked them to limit the chat history, he spent two
and a half hours not meeting that specification.
So, you
know, whereas these other participants that built things up
from small chunks where they understood each chunk and had
an overall goal for how to complete the task were much more
successful.
So, yeah, I'd love to talk more about that.
Anyway, so when analyzing the data, we
found that understanding program intention for each of
these web accesses was key to explaining how they were
using the web.
So this intention is clearly a spectrum,
but we broke it into three separate categories for the
purpose of our analysis.
First, programmers engaged in just-in-time
learning on new skills and approaches.
Second, they clarified and extended their
existing knowledge to sort of implementation level details.
And third, they reminded themselves of
details deemed not worth remembering or not worth typing.
So now I'll go through each of these
intentions and talk about some key characteristics of each
so we can talk about how they differ.
When participants engaged in just-in-time
learning, they often copied and modified code before
reading a tutorials prose.
So this is really interesting.
One participant said, there's some stuff in this code that
I really don't know what it's doing, but I'll just try it
and see what happens.
So we believe that this is because
participants really require a how-the-code-works
instructions, which is not how -- which is very different
than how-to-do-a-task instructions.
tutorials typically provide.
And that's what
They say, "F you want to do
this thing, follow these ten steps."
So instead, experimenting with and breaking
the code that they were provided, you know, structured this
more guided reading of the tutorial where they could say,
"I don't understand this part, I need to go read that
portion of the tutorial."
Next, participants seemed to be not very
concerned with deep learning.
One participant said, "I
think it's just less expensive for me to just take the
first code I find and see how helpful it is at a very high
level, as opposed to just reading all these description and
text."
And finally, participants often learn knew
terminology from search snippet results -- or search result
snippets.
For example, one participant began with the all
English query, "PHP update page without reloading," and
then saw the term AJAX in the result snippet.
Before
clicking on any links, he revised his query to "PHP AJAX
tutorial."
So people are learning things from the snippets
and taking little bits of knowledge they have and bringing
them into making their search experience better.
Yeah?
>>:
It seems like part of this learning is because -- the
quickness of the learning is because they've sort of
perceived that speed was important, like, "Let me get out
of this user study as fast as I can."
>> Joel Brandt:
hours.
>>:
Yep.
We made them stay for two and a half
But no, no, no, you have a very good point.
So the question could be, what do you think -- how do
you think you could modify a design of the experiment to
make them take the two and a half hours and actually
effectively use the two and a half hours rather than try to
speed through the study?
>> Joel Brandt:
So that's a really good question.
Experimental design is hard.
I think -- I think there's a
couple of responses to your question.
First off, one of
the interesting things we saw in this study was that -- was
in effect of this time pressure.
interesting way.
So -- but sort of an
So for a lot of participants, we'd see
them spend, let's say, the first hour doing whatever part
of the programming process they enjoyed.
through that.
And going slowly
So maybe they were, you know, really a
designer and liked visual things, they would spend, you
know, an hour getting the HTML right for their interface
when really that didn't matter for the results of the
study.
And then they would, you know, start to
feel this crunch and try and get things done.
But I think
that actually feeling that time crunch is a pretty
realistic situation.
I think for a lot of people,
especially people that are just programming in service
getting some other part of their job done they want to get
it done as quick as possible.
I think it's really hard to
do a design study that doesn't have any sort of time
pressure.
And I would be happy to talk more about that if
you have some ideas.
So anyway, returning to these just-in-time
queries.
I'll summarize the characteristics we saw in each
of these, and we'll build the table up for all three of the
intentions as we go.
So first, for learning queries, a good
example learning query is something like AJAX tutorial.
The terms in the query would almost always be English.
They would spend tens of minutes on the web, and do two to
three refinements, and click on lots and lots of results,
five or more.
They would typically visit tutorials or
how-to articles and copy dozens of lines of code.
We'll
return to this table for each of the rest of the
intentions.
So let's move onto clarification.
we often saw participants use the web as -- as a
First,
translator.
One participant received an error that read,
"XML filtering predicate operator called on incompatible
functions."
So he mumbled what does that mean?
And
instead of trying to figure it out, because it was
inscrutable, he copied the error message into the web
search window wholesale and immediately found a forum post
that said, "This is what you have," followed exactly by the
line of code that he had, and then, "This is what you
should have," followed by a corrected line of code.
So he
copied and pasted the code and had no actual idea what the
actual bug was.
And things like this, I mean, this is an
extreme example, but things like this were very common.
Similarly -- similarly participants would
make language analogies to sort of adapt their knowledge
from one domain to another domain.
So one participant
said, "Perl has a function that formats dates and strings,
so PHP must as well."
In these uses, participants often
could not write the required code from scratch, but in all
cases, they knew exactly what the code did when they saw
it, which is very different than these learning queries.
Finally, we observed that participants
trusted the code they found in these uses, and often didn't
immediately test it.
This caused debugging problems when
they made mistakes in adapting the code.
So interestingly,
the code was most of the time correct, but they introduced
bugs by copping and pasting it and, for example, not always
changing variable lanes consistently or things like that.
And then they would assume that errors they experience were
in code they wrote rather than in the code they pasted.
Again, let's review the characteristics of
these clarification uses.
So example queries here tend to
contain a mix of code terms and English terms.
And
sometimes they'll contain code terms that aren't
particularly relevant for the language being used.
javascript doesn't have threads.
So
But you'll see a query
javascript thread that will lead people to a forum post
that tells them to use set time out instead, which is a
threadlike functionality.
So these term -- the query terms usually
would be 9a mix of code and English.
These axes will take
about a minute, so usually do zero to one query refinements
and one to two result clicks.
These types of uses will
take them to API documentation, blogs, forums, and they'll
copy several lines of code.
Finally, participants use the web as an
external memory to remind themselves about forgotten
details.
things.
In some cases, they explicitly chose not to learn
After one participant searched for and immediately
copied code, we asked him if he ever copied it before?
said, yes, hundreds of times.
He
And then he went on to say
he never bothered to learn it because he knew it would
always be there.
In some cases, web search served as web
enabled auto complete.
If they would forget, for example,
the beginning portion of a method name, auto complete
wouldn't help them, but web search would quickly recover
it -Here the participant forgot the exact name
of the mice equal fetch array function in PHP.
He searched
for PHP fetch array, and the first result contained exactly
the information needed.
He didn't even have to click on a
result.
We did observe that despite the utilities
of these uses, they often broke programer flow because the
participant had to shift out of his current programming
environment and into web browser.
So finally, let's detail
the characteristics of these reminder uses.
>>:
I have a question about --
>> Joel Brandt:
>>:
Yeah.
[inaudible] development environment did they use.
>> Joel Brandt:
point.
So we -- that's a little complicated of a
We provided for everyone Aptana which is an open
source ID for web development.
complete, especially for PHP.
It provides decent auto
It's obviously not as good
as you can for a statically typed language.
But, I mean,
in general, that's a problem with web programming.
We also
let participants install their own ID's if they -- or
development environments if they wanted to.
>>:
[inaudible] language to say let's choose, let's say,
java, okay, or C Sharp.
And they install AP user or tar
pit or which has a pretty good support for [inaudible].
Do
you think that would change?
>> Joel Brandt:
I think -- well, this actually is
foreshadowing a later part of our talk where I'll talk
about building tools for development environments that
bring web search into the environment.
I think it would
change the results, but not dramatically because to use
auto complete, you need to have knowledge about exactly
what you want to do and how to term it still, right?
So
how to -- how to phrase what you want to do, which, you
know, the web plays a big role in helping you do that sort
of thing.
And second, even it you auto complete to a
method name, or, you know, a class name or something like
that, that still doesn't tell you how to use that thing for
these slightly higher level tasks.
a database.
You know, connecting to
Just finding in, you know, in your auto
complete the method that connects to a database doesn't
tell you, you know, the three steps you need to do to
connect to a database.
For that you need to either jump
down and do some documentation or find some example code or
something like that.
So I certainly think it would change
the results, but I think not dramatically so.
So let's go through these things, reminder
so we can contrast things.
So these -- an example query
here would be something like fetch array PHP.
are almost always code.
than one minute.
The terms
And these axis generally take less
They usually require no query refinement
because people know exactly what they are looking for and
require zero to one result clicks.
They typically take people to API docks or
just the result snippets and, you know, sometimes people
copy no code from these axises, sometimes they copy
several, it really varies.
So we learned a great deal from this lab
study, but we want to better understand if these results
generalized.
analysis.
So we conducted a large scale query log
We obtained one month of search and result
click -- result click logs from Adobe's Community Help
portal which is a Google custom search engine which indexes
all kinds of Adobe flex specific content throughout the
web.
As you can see from this screen shot, the search
results are presented in a manner very similar to popular
general purpose search engines.
The Community Help data contained over
100,000 queries from approximately 25,000 programmers.
All
of the queries in the status that were about to hit Adobe
flex framework.
For analysis, we grouped these queries and
result clicks in 65,000 sessions where a session was
identified as a string of queries from a single user with
gaps no longer than six minutes.
This methodology is
pretty consistent with previous literature.
After this, we programatically analyzed
three properties of each session.
First, the format of the
query strings; that is whether they contained plain
English, code specific terms, or both.
Second, the process of query refinement.
And third, the types of web pages visited.
For example, whether they went to tutorials, blogs, or API
documentation.
Finally, we also hand coded 300 sessions
with respect to this intention -- notion of intention I
introduced earlier.
Sessions were coded as either being
closer to the learning or reminding end of the attention
spectrum.
We didn't code for the middle clarification
intentions sort of due to insufficient context.
Again, I only have time to present a few
high level results in this talk.
There's a paper on this,
and so I refer you to do if you would like to see more
details on the quantitative analysis.
First, we found that query format predicted
the types of pages visited.
What do I mean by that?
This
table presents the types of web page visited such as
tutorials or forms broken down by formatted queries.
Either code only, English and code, or English.
column here sums to a hundred percent.
Each
And so numbers
present a percentage of, for example, code-only queries
that go only to a particular page type.
So looking at each row, we see that Adobe
API pages were primarily visited as a result of code-only
queries.
This suggests that API pages are only helpful
when participants know exactly what they are looking for.
Next, tutorials were most commonly visited
by English only queries.
This is not surprising.
Participants need tutorials when they have a high level
description of the problem, but don't know how to do it.
Finally blogs and forums were most often
arrived at through queries that contained both English and
code.
We speculate that this is because such content is
most helpful in clarifying situations where participants
know only some of the specific pieces involved in solving a
problem.
Based on our hand-coded data, we observed
that sessions on the reminder side of the attention
spectrum were typically code-only queries.
So this table
presents a breakdown of query format by session intention.
Again, all the columns go a hundred percent.
We see that
the code-only queries commonly drive reminding uses.
In
English and code or English-only queries drives learning
uses.
Third, we found that programmers rarely
need to refine queries.
So this stacked histogram presents
a summary of the amount of query refinement that happens
during search sessions.
In this graph, the nth bar shows
the total number of sessions that have a an nth query .
the first bar contains all sessions.
So
The second contains
sessions that have at least two queries and so on.
Each
bar is broken up by the format of that query.
As you can see, the histogram tails off
rapidly.
There are an average of 1.7 queries per session,
and this is significantly smaller than for the general
population.
The lowest published we could find, not a lot
of people publish data on this, was about 2.2 queries per
session.
So now if we take this histogram and
normalize the height of each bar to a hundred percent, we
can get a view of how query content changes when queries
are performed.
Interestingly, the first queries are either
code only or English only.
And as code-only queries are
refined, programmers tend to add English.
The portion of
English-only queries, however, tends to remain relatively
constant.
So this suggests that when the programmers
start off with English-only queries, they may not know how
to make them more precise with code terms.
So now I would like to present five key
insights that came from these two studies that offer some
implications for tool design.
First, this just-in-time
learning is common and programmers do this learning by
experimenting with the code.
This suggests that we should
maybe create tutorials differently.
Tutorials should teach
how a code works and perhaps provide a sandbox for users to
users experiment with that code.
Yeah?
>>:
How does the code work?
Because all -- we figure out
is that people are much more interested in, I don't know,
learning how to shop data, tech their weight ->> Joel Brandt:
>>:
Uh-huh.
Or to data finding [inaudible] because they have no
idea that the reason quickly controlled the data bonding
they have no motive before.
>> Joel Brandt:
Uh-huh.
>>:
So if we specify a tutorial in terms of how they would
phrase it, we are much more successful.
And if we are only
shown how it gets done, but we wouldn't explain how the
code works, it's just, "Okay, this is how we are going to
do a task."
>> Joel Brandt:
>>:
Uh-huh.
And so I don't know what you mean by how the code
works.
>> Joel Brandt:
So it would great to talk about the work
that you've done, because we haven't explicitly looked at
different types of -- you know, different ways to create
tutorials and how that plays out.
I would love to talk
about that.
Our intuition has -- if I'm understanding
you correctly, our intuition is not so much about how to,
you know, structure the intuition or purpose that that
tutorial is provided for.
I mean, I think you're probably
exactly right about trying get people to the right
tutorials.
But, you know, but -- but, about the necessity
of providing information about how a particular block of
code in that tutorial works so that if people experience
problems with it, they can fix those problems, right?
if, you know, any code that they find in a tutorial, if
they just want to follow that tutorial through step by
step, great, a tutorial would be perfect for that.
But
So
most likely, they want to do something slightly different
than that tutorial.
And as soon as they have to change
code, you know, tutorials don't teach them how that code
might be changed.
You know, so it shouldn't be that every
single line in a tutorial is explained.
lots of boilerplate in there.
You know, there's
But the lines that people
are going to need to adopt to do what their job is, they
need to understand how those lines of code are working.
So hopefully that sort of addressed your
question, and I would love to chat more about what you
found.
Second, we saw that copy code is not
immediately tested.
So to aide debugging, it would be
really helpful to demarcate which code is copy code, and
then after the execution of the program, highlight code
that was and wasn't executed.
In fact, we saw people waste
a tremendous amount of time thinking that code ran that
didn't even run or vice versa.
Third, we saw that search was often used as
a translator, so I'm certainly not an Information retrieval
expert, but perhaps we can automatically extract these
synonyms for code-specific searches and use this to improve
ranking.
Additionally, perhaps we can automatically
search the web for the content of error messages so that we
can help people find bugs sooner.
Fourth, we saw that programmers often
choose not learn syntax.
So this suggests that we should
put search inside a development environment and make, you
know, current auto complete tutorials sort of web enabled.
And finally, we saw that when participant
refined queries, the query refinement was often
predictable.
For example, they would often add version of
library or framework to improve the result set.
So this additional context is available in
the development environment, and perhaps we could use this
context that's present in the development environment to
automatically augment queries so that programmers can find
good results faster.
So I would like to take these design
guidelines as a dryer for the second part of my talk,
investigating how we can build tools that support and
amplify the use of the web during programming.
Given the great diversity of web use that
I've just described, we believe there's significant value
in building very task specific search interphases for each
of these intentions.
So returning to this graph from earlier,
our recent work that I'm about to describe focuses on this
long tail of tasks and this clarifying reminding end of the
spectrum.
Improving the just-in-time learning aspects of
web use remains exciting future work for us, and I would be
happy to talk more about the things that you guys are doing
in that space as well.
The main insight behind the tool I'm about
to describe is that milliseconds matter.
Specifically, we
believe that lowering the threshold to complete what are
already brief, but very common web search tasks, we do more
than just get a programer home for work 90 seconds earlier.
Instead, we believe that -- we hoped and believed it would
cause a fundamental change in the way that web searched was
used during programming.
So we began with the observation that tools
programmers use to searching the web are wholly separate
from those that they use to write code.
The web browser
and search engine has no access to the incredible amount of
context in the users code despite the fact that the code is
intended to be read by computers.
Similarly, the idea has
no notion of code providence and instead it seems that
every character that entered the editor has been typed by
hand.
What benefits might be realized by removing this
banner -- barrier?
To explore the question, we built
Blueprint.
Blueprint is a plug-in for the Adobe flexible
and development environment that authorizes code search and
code authorizing experiences.
So I would like to show you
-- begin by showing you a scenario of blueprinting use.
And this scenario we'll follow a programer as she uses
Blueprint to build a web application that allows people to
graph and compare their power consumption.
So this programer begins by creating a
project and adding some boilerplate code.
The first thing
that the programer wants to do is load some data from the
web.
So I start by creating a load data function.
And
then they say, "I've done this before, I have some
knowledge about how to do it, and I know that there's a
class involved that begins with URL."
auto complete -- uh-oh.
PowerPoint is not ideal.
problem.
So they bring up
Video control.
Video control in
Perhaps that's only a Mac only
I meant to pause play back.
So create some boilerplate code and then
starts out by using auto complete.
So it says, you know, I
know I've done this before, I know one of the classes
involved has a name that begins with URL.
So we'll open up
their auto complete and say, "Oh, yeah, URL loader, that's
it.
That's the class I'm interested in using."
But this
is as far as auto complete will get you some time.
So you
say, "Great, I want to you URL editor, but I have no idea
how to use it."
This is when programmers drop down into,
you know, Firefox and a search engine and try to find an
example code.
Blueprint allows you to just hit another
hot key to go and perform that search on the web without
leaving your development environment.
Now, the result view that we present in our
search engine is very example code centric.
thing that's foregrounded.
So that's the
So as you scroll threw it here,
you see code that uses the URL loader object.
And if
you've done this before, parsing the code is fairly
straightforward.
You look through it and you see several
different examples that are doing different things.
This
one has lots of air handling which might be nice, another
one further down here automatically loads the resulting URL
into an XML object and parses it.
So in this example, the
programer finds that and says, "This is exactly the code I
want."
So they go and just select that code and they hit
enter to bring it into their development environment.
We provide information about where the code
came from so if they ever need to get back to the web page,
they can do that, also how they access the code.
So the programer runs that code after
replacing the URL, and confirms in the debugger that she's
actually getting the X in all she expected.
So now the programmer -- so this is another
common way that people enter the Blueprint user interface.
They say, "There's something I want to do that I know is
one line of code," like setting a busy cursor.
But if you
don't know the class involved in doing that, you simply
can't use auto complete to start that process.
So in our
search interface you can search with plain English.
Just
type something like "busy cursor" and perform that search.
And immediately you see, oh, there's this line of code
right here, cursor manager, if you don't know it was called
cursor manager dot set busy cursor.
So it's a very simple
call, one line of code, and this allows you to get to that
very rapidly.
So the user here copies these two lines of
code for setting a busy cursor and removing a busy cursor.
And so we'll see that occur here.
So now when she compiles a code, she sees
that she's missing an import statement from busy cursor.
So sure, she could write this, but she can also very
quickly return to the web and get that import statement.
You can also imagine doing some semantic code analysis type
stuff to try and help people get the right information.
This third page shows another way to use
Blueprint.
So you can also dock Blueprint within your
development environment.
Here the user has searched for
charting and is looking through different charts to -to -- to get an idea of how she might want to visualize the
data that -- that came back.
And the next thing we can do
is in some situations we can pull out running examples as
well as source code and present those running examples
alongside the source code so people can see what the code
does.
So here, the user selects some code for a
line chart, adapts it real quickly to bind it to the data.
And in a matter of minutes, there's a mocked-up user
interface and you'll see that in just a second.
So let's real quickly take a look at the
interface.
ways.
First, users can initiate queries a number of
They can hot key on a line of code that has a short
amount of text, and that entire line of text is used as a
query.
Or they can select some text in the editor and hit
a hot key to use that as the query term.
Or they can
simply type into the query box and search in a traditional
manner.
In all cases we provide additional queries
suggestive to the user.
Each query result has a title
which is taken from a source web page and an English
description that we automatically distract, and also a link
to the source web page if they want to go and see the
example in context.
The most prominent feature is the example
code.
>>:
Query terms are highlighted within -This course content, everything that you have --
>> Joel Brandt:
>>:
[inaudible].
>> Joel Brandt:
>>:
Uh-huh.
Yep.
This is coming from the -- the documentation for the
flex or is that coming from [inaudible] forms?
>> Joel Brandt:
whole.
queries.
It's coming from essentially the web as a
I'll talk about that next, about how we service
But to give you a really brief answer, this is
sitting on top of that exact same Google custom search
engine that I was talking being earlier in the talk, that
indexes Adobe flex content from all over the web.
So we'll
talk about that in just a second, how our particular
extraction and query servicing works.
So in the code result view, we highlight
all the query terms which as I'll mention later proved to
be vitally important for people using the interface.
Finally, when we are table to, we sometimes
are able to extract a running example of the code so that
users can interact with it and see what the code does.
So now, getting to your question, I'll give
you a brief overview of how Blueprint works.
consists of three main parts:
Blueprint
The client plug-in which
provides the interface you saw, our Blueprint server, and
then an underlying search engine here this Google custom
search engine that I talked about.
When a user creates a
query like chart which you just saw, the Blueprint client
takes that and augments the query with contextual
information, things like the language you're using,
particular versions of framers you're using and sends all
the contextual information off to the Blueprint server.
The Blueprint server then uses some rules to take the
contextual information and turn it into what we consider a
better Google query.
We send the query off to our Google
custom search engine and it returns a list of custom URLS
and also a list of auto complete questions.
The Blueprint
server then parses this list of URLS, goes and looks at all
of the pages, and tries to extract example code and English
descriptions from those pages.
This is actually -- this
actually occurs off line and is cached so we can do this in
a realtime manner.
The Blueprint server then sends all of this
data, along with some additional data, things like
documentation pages that we know are relevant and the query
suggestions we got from Google back to the interface which
renders the examples.
So I would be happy to talk more about how
we extract example code from these web pages and things
like that.
That's a little bit nuance, so probably not
appropriate for the scope of this talk, but I'd be very
happy to talk about that stuff.
>>:
The example that [inaudible] that does what it says in
English then.
>> Joel Brandt:
So, actually, it turns out, the answer to
that question is mostly legal issues.
So we only -- it
turns out, we only extract running examples, so running
swift files in this case because it's Flash from Adobe web
pages.
We don't extract them from third-party web pages
and there's lots of good reasons for that.
security reasons and things like that.
Lots of good
So for those Adobe
web pages, we have reasonable confidence that, you know,
the people working at Adobe are trying to create the right,
you know, right running examples.
question?
>>:
I mean --
I'm just curious.
>> Joel Brandt:
yeah.
Does that answer your
Yes, so it's really good point.
I mean,
There's all these really interesting security issues
around serving up other people's content and also lots of
legal issues about serving it up.
So we have an opt-in
program that allows people to specify whether or not they
want all of their code included in the search engine or
just some of it.
And I don't want to go into that in this
talk because it's just legal stuff.
It has nothing to do
with the research.
But I'd be happy to talk about it
later.
>>:
[inaudible] getting back to the Blueprint server
saying this code was useful, this code is not useful.
>> Joel Brandt:
So we do -- I don't know if you do -- let
me see if I can go back to that really easily.
I didn't
mention this in the interface, we in fact -- we do have
opportunities for people to rate examples explicitly.
one does this.
>>:
No
Literally no one.
Right.
>> Joel Brandt:
Like we have thousands and thousands of
queries and thousands and thousands of copy and paste
events and six people have rated queries.
>>:
[inaudible].
>> Joel Brandt:
So there's also, I think, a lot of
opportunities for sort of implicit feedback.
There are
unfortunately ->>:
[inaudible.]
>> Joel Brandt:
So like did you leave this code around in
your -- did you leave the pasted code around in your code?
I mean, I think it would be huge to be able to use that
information.
Unfortunately, there are also lots of legal
issues with that.
So that -- that's been a really
interesting trade off in doing this research.
Obviously,
I'm Stanford grad student, but we've been doing it in
conjunction with Adobe because there's so many benefits we
get with working with a real corporation, right?
I mean,
we can get it out to literally thousands of users, we can
build, you know, on top of existing data sets.
We can get
access to logs that we would never get in academia, which
is awesome, but it also means that we have substantially
more legal constraints than we would if we were just a
bunch of university students causing trouble.
So, again, I'd be happy to talk about that.
I wish I could give better answers, but, you know.
So, going on.
So the next thing I want to
talk about is how we've evaluated Blueprint.
And I
personally think this is the most exciting part of this
work.
So to better understand how Blueprint changes the
search experience, we conducted, again, two studies.
The
first was a comparative lab study with 20 participants, and
the second was a three-month deployment.
We had over 8,000
people install Blueprint and a little over 2,000 of those
actually used it for real.
pretty normal drop off.
Which I think is a pretty --
Those 2,000 users constructed over
17,000 queries in a three-month span.
So as with the studies I discussed earlier,
I don't have time to detail all the results, so instead
I'll present the most salient findings from each of them.
Our first study was conducted with 20
participants.
All of those participants where Adobe
employees who used Flex as part of their job.
Participants
were randomly assigned one or two conditions.
They would
either use Blueprint or Adobe's Community Help search
engine which you saw before in a regular word process to
find word processors by completing programming tests and
that was the only resource they could use.
All
participants followed a tutorial task that was basically
identical that taught them how to use the search interface
they signed.
tasks.
All participants were asked to complete two
The first was very direct.
They were asked to,
just like you saw in the URL and asked to write this code
as quickly as possible.
The second was exploratory.
They were
asked to visualize the data set that was already in their
code and introduced what they believed was the best
visualization of that data.
We limited each task to
10 minutes, and for both tasks we measured the time to the
first code paste to the time of task completion.
And we
had independent experts outside of our study rate the
quality of code produced and the visualization produced.
So in our directed test we found that participants selected
and copied codes significantly faster with Blueprint than
with Community Help.
So this graph shows the rank order of
participants' time first paste.
users of Blueprint.
And the solid squares were
Blueprint users average 50 seconds
of -- 57 seconds from starting the task to their first code
paste whereas Community Help users average just over
2 minutes.
Task completion time was very strongly
correlated with time to first paste.
So this sort of
suggestions that Blueprint helps people write code faster.
Additionally, and perhaps most
interestingly, participants who used Blueprint wrote
significantly better code as rated by an outside expert.
So code that had appropriate error handling and things like
this.
And we think this is because Blueprint users could
very quickly scan lots and lots of example codes, so they
were able to select a better starting example.
Go ahead.
>>:
[inaudible].
>> Joel Brandt:
Well, I mean, you know, when people are
asked to hurry, not everybody chooses the best example,
right?
I mean, code was -- code was pretty similar across
users, and so -- I mean, there's probably 10 good examples
of how to do with URL loading task on the web.
And so if
you get two people that choose the same one, their code is
likely to turn out about the same in the end.
So I
wouldn't -- I think the high-level picture is a lot more
important here than particular data points, but that's a
great point.
So the results from the exploratory test
were a -- task were a little less dramatic.
As expected,
all participants used all 10 minutes, so there's no time
data there as they were trying produce the best design they
could.
When those designs were rated by an outside
expert, we found know statistically significant difference
in quality.
So it's certainly good that Blueprint didn't
make participants perform worse, but we would have liked to
see them perform better.
And I think the biggest reasons
why they didn't perform better were user interface bugs.
So it's not particularly easy in Blueprint right now to
compare multiple examples at the same time.
You can't see
all of the different type of charts that you could create,
for example.
And you can't do that very well on the web
either, but we certainly don't make that task easier and I
think you can sort of imagine interphases that would make
this comparison a lot easier.
So again, we wanted to understand, you
know, we had an idea that we were on the right track with
Blueprint, but to understand if we were really changing how
people use the web through this task specific interface, we
needed to put Blueprint out there and see what people
really did after they had it for a while.
a three-month deployment.
So we conducted
In fact, it's still out there,
you can download and use it, and examined user behavior
through log analysis and interviews.
Blueprint was employed on the Adobe
website.
2,224 users performed 17,012 queries over these
84 days.
We logged queries and user interface action so
things like copy and paste.
At the end of the study, we conducted
structured interviews with four users recruited through a
special prompt in the interface.
So one problem with just
doing log analysis, with enough data, you can sort of tell
any story you want to, right?
So the principle purpose of
these interviews wasn't to gather real quantitative hard
data, but instead to drive the creation to meaningful
hypotheses that were really worth testing in this data
rather than finding something because there's lots of data.
We compared Blueprint logs to the logs
again from this Adobe Community Help search engine.
And I
want to remind you that the Blueprint user -- Blueprint
uses this exact same underlying search engine, and the user
populations using these two search interphases are the
same.
So really the only thing that differed in these two
log sets were the search interface use.
>>:
How many programmers are using this technology you
think?
Like what percentage of overall Adobe technology
users are using search engines?
>> Joel Brandt:
large number.
So, it's actually a very surprisingly
And one of the reasons for that is if you
search from within the application, so just through the
regular help interface, it takes you to this -- this
Community Help search engine.
people are arriving there.
And so that's how most
I think the interface to get to
that search engine just by browsing the web is incredibly
cumbersome.
So I think it's very unlikely people are
arriving there by chance or by just general web browsing.
I think everybody is either arriving at this through, you
know, the help system in their development environment or
something they've bookmarked or something like that after
once arriving at it.
Or, obviously, people using Blueprint
arrived at Blueprint through labs and through the promotion
we did.
>>:
Does that --
[inaudible] it means most or all are professional
flash developers.
>> Joel Brandt:
Yeah, yeah, that's true, so we should --
we could talk more about this offline.
Professional flash
developers is kind of a funny term because interestingly,
most people who write flash professionally have no computer
science training.
So it's real interesting population, but
anyway, we could -- I would love to chat more about that.
So anyway, now I'll present -- there was
also more log in analysis and for each of these I'm offer
our high level insight that came from the interviews and
state the hypothesis that came from that and then I'll
describe what we found when testing the hypothesis.
First, all participants indicated that the
benefits of having this consistent result view
substantially outweigh the drawbacks of removing examples
from their original context.
So specifically, we asked
one person -- well, we asked all the interviewees, but when
we asked one of them about whether additional contacts was
ever necessary.
key.
He said, "Highly knew the search code is
I scroll through the results quickly, and I can find
code that has it and I can understand the code much faster
than I could English."
So this indicates that some help is
required, but -- but really, when you're engaged in the act
of coding, there's this very small gulf of evaluation to
understanding the code is something you want, and a lot of
the context gets in the way in these cases.
So, broadly speaking, additional context is
rarely necessary.
We would expect Blueprint researchers
would have significantly lower click-through than Community
Help searches.
So in the Blueprint interface you can click
through and get that context if you want to.
It appears in
the same window and it's very east toy get that context.
Indeed, Blueprint users clicked substantially less with an
average of .38 clicks per session versus 1.32 clicks in
Community Help, so that's a huge difference in the amount
of click through.
Second, many interviewees mentioned that
they use Blueprint in conjunction with other IDE features.
For example as you saw in the video, they would use auto
complete define classes or properties of interest for some
task that they wanted to complete, and then they would use
the code generated by auto completed to initiate a search
in Blueprint.
So in short, Blueprint seemed to word
symbiotically with existing IDE features, each kind of
making the other more useful.
itself in the logs?
So how would this manifest
Well, one way to look at this behavior
is to look for occurrences of what you might call correctly
formatted code.
So code that is produced by IDE features
like auto complete would be correctly formatted, having all
the necessary, you know, capitalization and things like
that that people might not bother to carry over to a search
engine when they are typing by hand.
So if such code is
indeed more frequently occurring in Blueprint than it does
in Community Help, this would suggest that Blueprint is
really being used in this manner.
Now, we have this nice property that
because the flex framework uses this coding convention
where everything is in camelCase, it's very easy to look
for cases where camelCase is used.
So indeed camelCase
words appear substantially more frequently in Blueprint
queries than in Community Help queries.
Almost half of the
queries contain in Blueprint whereas only 16 percent of
them do in Community Help searches.
Finally, our participants indicated that
they use Blueprint very heavily for these clarifying and
reminding tasks.
One participant said, "I can find what I
need 68 percent of the time without having to go to API
docs."
Another participant mentioned that he would often
use Blueprint to make sure he wasn't forgetting anything in
a task he did routinely.
So it wasn't about copying code.
Instead, he would, for example, routinely pull up a block
of example code to make sure he hadn't left out a step
when, his example was, when setting up transformation
matrixes for the flash stage.
So there's three or four
steps you have to do in that process and just bringing up
example code he could compare it to his code and say, "Oh,
yeah, I forgot that step," or "Yeah, this is the right
thing."
He went onto say that Blueprint works well
for that mid space between when you don't know what you're
doing at all and when you need help -- and when you don't
need any help at all bus you know exactly what you're
doing.
So following these intuitions, we believe
that refining content would be substantially more frequent
with Blueprint than with Community Help because it's being
used for these reminding, clarifying and repeat tasks.
Indeed, this hypothesis -- we found this
hypothesis to be true as well.
12.2 percent of the
sessions from high frequency users from Blueprint, again we
only looked at people that used it a substantial amount of
time as with Community Help contained queries that were
identical to a prior section compared to only 7.8 percent
of the queries from Community Help.
Finally, we took a look at the top 50
queries from both interphases.
So as you might expect, the
most common queries -- the most common searches are quite
similar.
Ten of the top 20 firms for each list appeared in
the other list.
or canvass.
These are searches for things like button
So this isn't particularly interesting, but
what it is interesting is that the relative frequency of
terms were sometimes quite different in these -- in these
two search interphases.
And perhaps the most dramatic case
is that of the query for alert.
So queries for alert,
which is the call necessary to display a simple pop up
dialogue very similar to javascript were substantially more
common in Blueprint.
This was ranked 8th in Blueprint and
34th in Community Help and the frequency of things between
8th and 34th, the frequency change is huge.
this be?
So why might
All right, well, first alert statements in flex
are slightly more complicated than you might be familiar
with in javascript.
They have three parameters all of
which have to be sent.
things.
So you have to do a number of
It's a little more complicated.
Second, alerts are most commonly used in
debugging, right?
familiar with that.
And I'm sure many of you programmers are
They are convenient for conveying
information to the programer and allowing the programer to
stay within the interface.
So in some situation it's much
more preferable to print lines and trace statements things
like that.
But there are, of course, simpler alternatives.
It's much easier to use a trace statement or a print line.
So let's look at the result interfaces for
this query.
As you see, Blueprint immediately produces the
one line of code necessary to display dialogue right here.
Someone use Community Help would click on the first result
and wait for a while for the page to load then scroll
through several screens of content to find the first line
of code because the first result in an API page that
doesn't have code until you get to the bottom.
So we believe this is a specific example of
how task interface services can change the search is a
viable tool.
Without Blueprint, users may instead be
choosing, you know, to take suboptimal approaches like
using trace statements because the cost of searching isn't
worth the benefit.
So in conclusion, we believe that these
results show that task specific search interphases can
cause a fundamental change in how and when individuals
search the web.
Moreover, we believe that programmers
increasing reliance on the web is changing the nature of
what it means to be a programer.
In fact, in the future,
we suggest that programming may become less about knowing
how to do a task and more about knowing how to find the
right answer.
So thank you so much for your time today.
I'd be happy to take any more questions.
>>:
[Applause.]
>> Joel Brandt:
>>:
Thank you.
You mentioned fairly early that you saw that these
opportunistic nonprofessional programmers, it was very
important that they get to the solution quickly, less
expensively.
Do you actually think that's different to
whatever the opposite is of opportunistic professional
programmers?
>> Joel Brandt:
that.
Well, I think there's two things to say to
First off, I -- first off I think there are
differences.
But -- but I think more importantly,
professional programmers certainly engage in this
opportunistic activity when they are prototyping new things
and stuff like that.
I think that there's -- that part of
being a professional programer means you're often in
situations where you need make sure that things are done
right that some thing is done systematically that something
is done with a minimum number of bugs with away that is
viable.
So I think in those situations, I mean, you always
want to get your job done and go home and sit in front of
the TV, but I think the trade offs between doing something
quickly and doing something right change for professional
programmers a lot more than they do for these amateur
opportunistic programmers.
>>:
Is there research specifically about the difference?
>> Joel Brandt:
Not that I am aware of.
So Steven Clark
and Monte Hammitry who are both on the visual studio team
here at Microsoft were as far as I know the first people
that sort of coined this idea of opportunistic in
programming.
And there's -- there's -- I think this is one
of these things where there's lots and lots of anecdotal
evidence for it, but there's no -- you know you can't -- I
don't know of numbers you can point to that say this
percent of the time they are thinking about this.
And this
percent of the time this other population is thing being
this.
I think it would be fascinating to get that data it
would be interesting to talk about how to get it.
>>:
[inaudible] title approach and say.
>> Joel Brandt:
>>:
Uh-huh.
And if you are working -- if you are designing a class
library, you know, doing the framework, you have to make
sure that ->> Joel Brandt:
>>:
It's gain is --
>> Joel Brandt:
>>:
Yes.
Yep.
Is onto it [inaudible].
But if you approach -- you
develop for a customer and this customer is 10, 15 people
working with a software solution, you don't care about the
architecture.
There is no need for an interior
architecture.
There is no need that your database
application ->> Joel Brandt:
>>:
Yeah.
That your database can't handle millions millions of
transaction per seconds.
>>:
I think --
>>:
Just slap it together and steer it around and --
>>:
I think it's nice to think that, but rather than -- I
think it's more to do with people and I think the
probability that is more opportunistic programming going on
than we like to think is pretty high.
I think most people
care about getting to a solution, and yes, they know they
should be carrying about beautiful elegant code, but they
don't.
>> Joel Brandt:
Well, and I mean, to carry your ideas to
the extreme, I think you're both absolutely right.
You
know, I mean, I'm a fairly proficient programer, and I
write tons of code in this very, very opportunistic manner.
Code that I would never want the world to see, because it's
code that's going to run exactly one time.
You know,
it's -- you know, I don't care if my algorithm to sort this
stuff that I'm extracting from this log file runs
particularly quickly because I'm going to process this log
file once, right.
And that's a very different type of
software development and a very valid form of software
development compared to, you know, our traditional models
and yet we are using exactly the same tools to do those
sort of things.
>>:
If we look in this the space of this opportunistic
development, I mean, they usually -- the customers, they
serve, they have a budget of $500, a thousand bucks, 1,500,
so they can't afford to spend 10 days, 15 days to get
everything right.
The more projects they can do, they need
more money than they have in their pocket.
There's no time
for them ->>:
I mean, if given the choice between taking a long time
to do a task and a little bit of time to do a task, I'm
always going to choose the little bit of time to do a task.
>>:
I would say so.
If you're customer -- okay.
Like if
I would go to my planning team where they have an
application where they scan my badge and all the doors lock
when I come in I don't have it, I mean, how much would the
client be willing to pay for this application?
>>:
$500?
For instance a professional [inaudible] like if
somebody is paying you for this kind of job, like I think
we are talking about different parameters now.
And I would
suggest that like there's this whole end user software
engineering consortium of people who study the end user end
of basically like people who are mainly like PHP websites
for their church or people who are like doing a spreadsheet
for a budget for their boss and they have lots of
calculations in the spreadsheet.
They don't necessarily
even know what code elegance means.
They don't have any
idea what testing is or that they should be doing it.
what maintenance is or coding for maintainability.
Or
They
have very different -- their goals are basically, I need to
get this job done because somebody told me to do it.
There's this Permian thing I have to do in order get it
done I'll do the Permian thing to get it done and then I'm
done and I don't have to think about it again.
>>:
My point is we like to think that people working on
Windows or Studio or Office don't do the quick
opportunistic programming and I [inaudible].
>>:
Even if they do it just as a past zero, then they kind
of pick and choose from it and say, "Okay, now I've got to
add the security hardening or whatever it is."
>> Joel Brandt:
>>:
Uh-huh.
I think they will obviously still be using the
reminder code, the API service is just too large to have
everything.
>> Joel Brandt:
>>:
Yep, yep, I mean --
I mean how often have we seen applications that
prototype and the percentage of customer and the developer
intend that, okay, this is the prototype.
This should
never ever be productive.
>> Joel Brandt:
>>:
Uh-huh.
Now let's make it real, and the customer says that's
good enough it worked.
>> Joel Brandt:
Yeah.
And Fred Brooks has a great
commentary about this sort of thing.
The problem is that
it's actually too easy to reuse code, right?
That there's
lots of code that was written, that if you built -- if you
were doing -- if you were a mechanical engineer and you
built a physical model, you would have thrown it away, like
that's what would have happened, and it doesn't happen in
software development.
>>:
Like two orders of magnitude user [inaudible] for this
plug-in or something like this.
What do you think would
change about the kinds of research you could do?
>> Joel Brandt:
That's a great question.
I think -- well,
I mean, I think -- I think it would be great to be able to
sort of segment the population that we are looking at into
different smaller segments and understand how different
people are doing things differently.
So if we could look
at all of the people -- you know, if you could, for
example, compare, you know, all of the people that had a
computer science background and all of the people that had
a design background and see how they differed, I think that
would be fascinating.
I think you'd need a slightly larger
population to be able to do that, but you also need more
contextual information that we don't have.
So i think that
would be an interesting thing to do I also think our most
interesting next research steps are sort of parallel to the
things we've done rather than in depth in what we are doing
now.
I think there's a lot of really interesting
questions about how you change the editing and code
execution and debugging environment, you know, with the
intuition that lots of code that comes in your editor, you
didn't write, right?
Nobody is really thinking being how
the editing process should change for this new type of
development.
>>:
May I ask because there's clearly no [inaudible] here
so the [inaudible] in taking off especially if you made it
available in more languages.
>>Joel Brandt:
>>:
And you are right you're going to get all this data.
>> Joel Brandt:
>>:
Uh-huh.
Yep.
And like you said, there's legal and morale issues.
>> Joel Brandt:
Yep.
Well, and I think I mean, I think
also we talked about this a little bit.
I think that if we
could get more data on how people are using the code, I
think there's tons of really interesting things that you
could do with that.
Especially if you had a great
background in information retrieval.
So I think there's
lots you could learn about what code examples were good,
you improve ranking.
Also there's really neat things you
could do where if you could watch how let's say ten people
used this example what things they change.
Then for the
11th person you could say, okay, well you you're probably
going to want to change these three literals.
Out of this
boilerplate, you need this and line, but you don't need all
this stuff.
And don't forget that line, everybody copied
that, right?
I mean, I think, so as we start to get lots
of date a you could do lots of things.
I think all of that
work would be absolutely fascinating.
>>:
If you also control the content piece of that and then
put the content piece with it.
>> Joel Brandt:
>>:
Yep.
That's great input to tune it.
>> Joel Brandt:
Yeah, and it's also a great -- I don't
know if you were saying great input to tune the interface
or great input to tune the actual content.
>>:
By tune I meant content.
>> Joel Brandt:
same thing.
Okay, yeah, then we are saying exactly the
Yeah, I totally agree.
Great any other questions?
I'd also, of
course, be happy to chat about anything else offline, so
thank you so much for inviting me.
[applause.]