>> Chris Bird: It's my pleasure to introduce Emerson Murphy-Hill. He is, he got his
PhD from Portland State, and then spent two years at UBC under Gail Murphy as a post
doc, and just recently started as assistant professor at North Carolina State. So he's
visiting us today and he'll also be around for the rest of the week, so if anybody would
like to send him any questions by e-mail, people watching over video or meet with him
they can e-mail me, my email is cbird or Emerson directly. So with that we’ll turn it over
to him.
>> Emerson Murphy-Hill: Okay. Thanks Chris, and thanks for inviting me, happy to be
here. Thank you for all, to all of you for taking your morning to come and see me talk.
So I wanted to talk about why software developers don't use their tools. And I'm going to
start by telling you a little story. So this is the PDP-1 computer and this came out almost
exactly 50 years ago. And this is a replica; I think this is from the Computer History
Museum in San Jose. And so the computer you see in the back with the monitor is some
later, some later computer. It came out about 50 years ago. It cost about $100,000. It
came out with, kind of one of, what you might consider the first editor. They didn't really
have an official name for it, but they called it the colossal typewriter or the very
expensive typewriter.
And what's interesting to me about this system is that although it's got an editor and
although you could do some programming on it, there wasn't actually a monitor to see
what you were programming, okay. And if you wanted to type something in and then see
what you had, what you had written, you had to use the printer, and there's a little printer
in the back, all right. So, which of course makes debugging your programs a bit of a
challenge. So if anybody tells you that real programmers don't use Visual Studio they
use Emacs, you can tell them that real programmers don't use either of those. They don't
even use a screen.
So I bring up this example just because I want to say that 50 years ago this is maybe,
gives you kind of an idea of what software development looked like. And it was fairly,
you know, you could do your job, but it was fairly primitive, okay. There was a limited
amount of things you could do, a limited amount of tools that were available to you. So
contrast that with today. So this is what may be an average software developer's
workstation looks like. We've gone from you now having all computer and no screen to
basically all screen and very little computer. And we have a million, a thousand different
tools at her fingertips. We have bug trackers; we have integrated development
environments where we can, you know, format our source code with a single key press.
We can get an outline of our code, we can navigate our code, we can do find call
hierarchies we can find references, we can change the structure of our code, we can
generate new code. There's a million, not a million, there's thousands of different tools
within Eclipse or Visual Studio or any modern integrated development environment.
And so, the point of this example is that I’d say that life has really changed.
Technologically, we have a lot more at our fingertips as software developers. There's a
lot more things that we can do. However, if you look at all the different software features
that are available say in Eclipse, and you look at how many people are actually using, the
number’s quite small. So most developers use less than one percent of the available
commands in Eclipse. And so how we know this is that Eclipse makes public its usage
data so if you use Eclipse and you agree to submit your usage data, each time you use a
command, Eclipse records it and then puts it on a server in the Eclipse foundation. And
so we can go through that data and see which commands people are using, and also kind
of infer which commands they're not using. Question?
>>: Do you know how that compares to the percentage of other software tools like Word
or, you know?
>> Emerson Murphy-Hill: So I don't know anything about other development
environments. I know my guess is Visual Studio is keeping track, and I know InteliJ is
keeping track too. But I think that's private. I believe there's public Word data from
maybe ten years ago.
>>: What about non-programming tools?
>> Emerson Murphy-Hill: Non-programming tools, it’s actually quite similar as well. I
don't know if it's maybe quite as extreme as this, but take you know, take another
example outside of programming like, the, the Apple App Store. How many pieces of
software in the App Store do you use, of the, you know, hundred thousand that are
available? Maybe if you use a lot, you use a hundred, right? So it's pretty small. Yeah?
>>: Is it the same one percent?
>> Emerson Murphy-Hill: Is it the same one percent? No. There's, for any tool you'll
probably find a good one or two users. Now some of them are very esoteric and very few
people know about them so, for example, in Eclipse there's a tool called hippie complete.
Okay. It's like auto complete. It has a history and I don't remember what it is exactly,
but it comes from a previous editor. And the way hippie complete works is it's a braindead auto complete. It doesn't know anything about the semantics of your program. But
if you’re typing on a token, it looks back at the previous few letters that you've typed, all
right, and then finds that token in the previous, basically looks at the previous code or the
previous text or whatever that you've already written, and then auto completes from that
token above, okay. So it's kind of a, it's a weak version of auto complete. Now it's
actually pretty useful if you're using a programming language or a text file that there's no
auto complete for, and it's a feature that Eclipse offers but very few people actually use.
Other features that are really useful, I'll give you an example soon, are used by a broader,
broader population than just one percent. But it's still, you know, if you have such usage
data available to you, what I encourage you to do is take a tool that you think everybody
should know or that you assume everybody does know, and look at how many people are
actually using it. My guess is you're going to see numbers less than twenty percent. Uhhuh?
>>: [inaudible] leading essentially to the [inaudible] because they found that so many
features [inaudible] and that in fact the features people wanted to use were too hard to use
and the other ones were undiscoverable. I don't know if they've done a follow-up study
to understand whether users after [inaudible] aware of more commands, but certainly
there was a lot of work that went in because fundamentally the file [inaudible] the
traditional file menu system [inaudible] number of features [inaudible]. And so whether
the ribbon scales, I don't know but it certainly helps you discover things, I think.
>>: [inaudible] anecdote there and I don’t know if it's a case of the purist [inaudible] is
people kept asking them for those very features that had already be in the product for ten
years. So they were reaching constantly at the top of the request list for things that they
wanted ten years ago so what are they talking about? And that was one of the big drivers
of the change. But I don't know if that same thing going on here with Eclipse [inaudible]
but people still want the 99%.
>>: But if you look at like Visual Studio and Eclipse, they both subscribe to sort of the
old-style user interface.
>> Emerson Murphy-Hill: I would actually guess that--I think there's actually a fairly
fundamental difference between Word and tools like Visual Studio or Eclipse or
whatever. And that's with, with Word most of the features that you want to use, what
you'll find is that people have a problem and they need, they go out and they search for
and sometimes they can't find it. So they know that say text needs to be, they want their
text darker, so they're going to look for something like bold. Maybe they don't know the
name of it, but they're looking for something for it.
All right, with Eclipse, what are the core features you need to do programming? You
need a compiler and you need an editor. The rest of it's just kind of fluff, right? The rest
of it's just sort of productivity enhancements. Whereas, with Word, you know,
everything, I think well, not everything, but a lot of the tools are things that, you know
you should have, but it's just a matter of finding them. So, you know, contrasting you
know, fifty years ago against today, we have more tools, but we're not really using them
that much. All right, so, let me put it to you this way. So this is what programming
looked like 50 years ago, and this is kind of what I think it looks like today. All right, it's
basically the same picture right but we have a pretty UI on top of the development
environments that we use.
So my research is about helping software developers make use of the tools that we've got.
We spent a lot of time and we spent a lot of money making really great software
development tools. Things can really improve people’s productivity; things that can save
a lot of time, things can help connect people together, things that can make software
better in general. But if people don't use them, right, those benefits aren't reaped. So I
want to talk about sort of a slice of the research that I've done and that I'm planning to do.
And we’re going to talk about sort of three pretty different areas. And the first one is
going to be re-factoring, and I'll talk about a little more in a minute. Re-factoring is
restructuring code. Will talk about some re-factoring tools and some problems with those
and why people use them and don't use them. And then I'm going to talk about discovery
in general. How do people find out about features in the tools that they use? If they are
using Eclipse, how do they know that there's a re-factoring tool in it? If they're using
Word how do they know that there's a compare documents feature in it? And the last one
I want to talk about is aging. Which is probably not the kind of aging that we usually talk
about, software aging; this is actually human aging. How does the tools that we use, how
does that change over time as we get older?
Okay. So let's start with re-factoring. So this is some work that I did when I was in grad
school that my PhD is based on. So let's start at the beginning here little bit. So refactoring has been a hot topic in the last ten years or so. And re-factoring is just changing
the structure of code without changing the way that it behaves. So although it's been
popular for the last ten years, of course, it's been done forever. Even, it's been done since
we've had structured software at least. And so there's nothing, nothing particularly new
about it except that we have a name for it and we've kind of done some categorization of
different types of re-factoring. So why would we want to change the structure of code
without changing the way it behaves, right? We're paid to make software that does
something more, that has fewer bugs. Well, a re-factoring helps enable those different
types of changes. So if you need to make an enhancement, changing the structure of code
may make it easier to make that enhancement. Or if you need to fix a bug, changing the
structure may actually expose the bug in a way that's not obvious in the code that you
have now.
So and factoring is something, like I said, something that's been around for a long time
and in the past twenty years ago, if you wanted to restructure your code, you would you
basically do it manually. So if you to say rename a class Okay. renaming is typical refactoring that's, that a lot of people do. You want to take a class; you picked a name
before that doesn't capture what the class does anymore. The name wasn't good for some
reason. If you wanted to do it in the past what you would have to do is you would have
to change the name and then find all the references to that name and update all of the
references. And in the last few years what you've gotten is tools to help you do that.
What we are calling re-factoring tools. And so re-factoring tools are in most big
development environments now Clips, NetBeans, Visual Studio and as plug-ins to those
environments too. And re-factoring tools don't do anything amazing other then do what
you would normally do by hand. They just automate what you would have to do without
the tool. So again take this rename example, if I have a class and I need to rename it, I
can use a tool and I can, you know, say, this is the thing I went to rename. This is the
new name and the tool knows the semantics in the program and so it will find all the
references and update all of the references.
And so basically with re-factoring tools you get two things. One it's basically
guaranteeably correct. Okay. the change that it's, that will make will not affect the
behavior of your program. And they also do it very quickly as well. Even if you're pretty
sure that the change you are making isn't going to change semantics, you can do it much
faster with the tool. The tool can do it basically at the speed of the compiler. So that's
why you might want to use a re-factoring tool over factoring by hand. An important
point here is you always have the choice. Well, if you have a re-factoring tool available
to you in your development environment, you can do it with the tool you can do it
without the tool.
Okay. and I have a little demo here. So this is some re-factoring tools in Eclipse. I'm
going to change name of this results variable, Okay. I'm going to change it to results set
and it's going to change all of the references, which is nice. I'm going to do one other
little re-factoring here which is I'm going to take this hunk of code, and I'm going to put it
in a new method. It's what we call extract a method. All right, say extract method. All
right, it made it into a method here, put a reference to that method, and I can change the
name as I did before. So a little demo about what kind of re-factoring is about. And
there are smaller factoring and big ones. Bigger ones include things like restructuring
inheritance hierarchy in moving things up and down, excuse me, extracting classes, in
lining classes, lots of different types of re-factoring. So that's re-factoring and refactoring tools.
So what's the problem here? Well, we do know that re-factor--developers re-factor a lot.
Whether or not they recognize it, whether or not they sit down and say Okay. I'm going
to re-factor right now. It's something that people just do. And so we did--there's been
previous work that has, say, taken automated tools and look through the version of, look
through the version history of large pieces of software. And seeing where people have
changed class names, taken big classes and broken them down into smaller classes. They
have observed that re-factoring's happened. We also did a study where we manually
looked at code changes and found that re-factoring a lot. Happens daily, hourly at a high
level in changing the class structure around, at low levels just changing statements around
a little bit. So developers re-factor a lot. But they don't use their tools very much, okay.
They don't use re-factoring tools. And, in fact, we did, when we did a study what we
found was that developers only use them about ten percent of the time. And so the
methodology here is a little, little complicated but I think I can summarize it.
What we did is we looked through version history, Okay. and we did a random sampling
of before someone did a commit and after they didn't commit, and we compared the two
together, and we inferred whether some re-factorings happened. We saw a class that's
name is changed but the content hadn't changed all. We said, oh it looks like they do a
rename, re-factoring, okay. Okay. we did that for a bunch of different changes and we
also had re-factoring tool logs. The people that actually made these changes, they gave
us the history of which re-factoring tools they invoked when. So we were able to do with
that, we took when they used tools and tried to figure out, they must've done that between
two different commits, all right. So what we could do is we could see the re-factoring in
code, and we can see when they did it with the tool. Or sometimes we saw a re-factoring
in code and there was no history of doing it with the tool. And I case we said they didn't
do it with tool. Question?
>>: Were you looking only for re-factorings that matched the list of what could be done
automatically by the tools?
>> Emerson Murphy-Hill: That's a very good question. So we, when we looked
originally we didn't constrain ourselves by things that are just available by tools or even
re-factoring that have a name. We were just looking at changes and we thought really
hard about it, and we thought was this something that would've changed behavior or not.
I think we found a couple that didn't have names, but for the most part they were named.
And actually for the most part there was tool support. So when I talk about ten percent,
ten percent of the time they didn't use tools. That was only for the re-factorings for
which there was tool support. Okay.
>>: [inaudible] the driven by instrumentation of clips were what's the data collection
[inaudible].
>> Emerson Murphy-Hill: So, so in terms of looking through the repository, there wasn't
anything special there that was outside of Eclipse. That was us looking to the version
histories. In terms of instrumentation for which tools they were using, that is inside of
Eclipse. Eclipse keeps track of which re-factorings you do. And it does that for the
purpose of library migration. So if you do some re-factoring, you can ship those refactorings along with your code and so any clients of your code can kind of rerun those
re-factorings on their code and they can update it. But we can use that for our study too.
>>: [inaudible] control system to sort of match the changes to what happened
[inaudible] the re-factoring piece of the tool actually corresponded to [inaudible] and that
was manually done?
>> Emerson Murphy-Hill: That was, yes, manually done. And there are some threats
there. So for instance it's possible that we didn't get the full histories. But even if you
look at, even if you do a more conservative analysis, the number is still low. The number
only goes up to about 30%. Thirty percent of re-factorings are done with tools.
>>: [inaudible] adequate. Somebody actually would have to turn on the re-factoring for
that special [inaudible] to take place? That's not a default [inaudible]? So when they turn
that on and you know that the re-factoring tool is active?
>> Emerson Murphy-Hill: That's right.
>>: And then for certain other types of re-factorings, you actually have to explicitly say
I'm doing this re-factoring, lifting out the method.
>> Emerson Murphy-Hill: Yes, if you use the tools, you always have to say which type
of re-factoring you're doing.
>>: I see so this is the recording is recording the type of re-factoring [inaudible].
>> Emerson Murphy-Hill: Yes, that's right. Okay. So, developers are only using their
re-factoring tools about ten percent of the time. Then we did a couple of different case
studies and actually that number was quite stable. We were surprised. We were
surprised by that. And I think there's a bunch of possible reasons why people don't use
their re-factoring tools. But I'm going to give you one and one of them is that the refactoring tools have poor usability, okay. The re-factoring tools are, have some problems
with the user interfaces. So I'm going to give you an example here, this example says-occasionally when you do a re-factoring, you try to use a tool. The tool will refuse to do
it; it will pop up an error message. For instance this one says “ambiguous return value.
Selected block contains more than one assignment to local variable.” So maybe you are
smarter than I am, but when I first saw that it was, it didn't really make a lot of sense.
After, you know, careful study it's actually, it is correct. It's actually quite accurate. But
at its face is kind of hard to figure out what's going on.
And so we did a, a formative study where we had people do some re-factoring and, and
occasionally they got these, these error message and we asked them to either explain
what was going on, tell me what was, tell me what this error message means. And all the
time what they did, they got most of it; they got say 75% of understanding what the error
message meant. But the other 25% there was something slightly off, or more commonly
people just don't pay any attention to the error messages at all. They say yeah, Okay.
whatever, I get that. Okay.
So I'm going to talk about a solution to the error message problem and then I'll talk about
some other usability solutions too. So what we did here was we took the error messages
and we replaced them with a graphical representation. Okay. So what you see on top is a
graphical representation of the same error message you saw before. So if you select this
piece of code with these pink dots, Okay. And you tried to extract it to a new method, put
it into a new method below, you can't do it, because there are these two variables that are
being assigned to, and if you try to assign to them you can only return one or the other,
Okay. not both. And there are technical solutions to this, but basically this is what the
previous error message was trying to tell you, just in a graphical form, all right.
So we did this, we built an implementation for Eclipse, and then we also built some
prototypes. So it turns out if you look at the error messages that re-factoring tools throw
in Eclipse, there's a lot of them. There's actually more than I expected. There's about 600
different error messages that you can get from the Eclipse re-factoring tools. And that
was just one example. So I didn't have the time to reimplement all 600 in a graphical
form, but what I did instead is I took all 600 broken them down into taxonomies, similar
ones are grouped together. Some of them talk about data flows, some of them talk about
control flow, some of them talk about clashes between names. I took those broken them
down, and then made some sort of prototype UIs for what, what a graphical
representation of that error message might look like, sort of a more expressive
representation.
So, and based on those we kind of came up with some principles for what we think a
better display of these re-factoring error messages look like. And so, for instance, the
error representation, no matter what it is, should help me estimate the amount of work
that's required to fix the problem. So for example, we saw that last error message that
said more than one assignment to local variable. Okay. how many? It matters how many
because the more there are the harder it is for me to fix the problem. Another guideline is
the error should be distinguishable. And what we saw with the formative study is that
people saw one error message, Okay. and they read it, and they were pretty diligent about
it, and then I saw another, and they didn't even look at it because they assumed it was the
same as the first one. Okay. the fact that they were both kind of indiscreet text boxes,
they both had you know a sentence worth of text, they didn't even bother to look at the
second one. So our guiding line here was that you should be able to distinguish, sort of,
at a glance, the differences between different error messages. And these guidelines aren't
particularly new. These guidelines you get from sort of any standard usability text. But
they are sort of specialized just for this context. These are the ones that we think are
particularly important for these types of error messages. We also think actually that
they're probably generalizable to other error messages that you might get, in your
development environment.
So we did a study on this. Okay. where we had, we pitted the graphical version versus
the textual version. We had people do kind of some simulated re-factoring. We showed
them the error messages or we showed them the graphical version. We asked them to
explain what the problem was, point out what the problem points are in the code, and tell
us whether which kind of one they liked more. So a standard usability test. So using the
graphical version people were faster. They were about three times faster understanding
what the problem was. They were more accurate, depending on how you measure
accuracy. They were sometime, somewhere between two and 10 times more accurate
using the graphical version. And they also enjoyed using the graphical version more.
We actually did two different versions of the study. One where we gave them training
with both of the--we essentially told them what this meant, and we told them what this
meant and we did the experiment. And we did it another way when we didn't tell them
what either of them meant, all right. And those results were consistent as well.
Okay. So that's kind of an improvement to the error messages that you get with refactoring tools. We did some work in some other areas with re-factoring too. So on the
left, this is a tool, that's what they call a smell detector. So a smell detector is a tool that
finds and identified smells. And a code smell is something that's in your code that
indicates that a re-factoring might be necessary. Okay. So it's kind of a, you might call it
a design defect, and they use the word smell because it's something that might possibly
be wrong with your code. But you have to take a look at our first, like something that
smells in your refrigerator. You know, there's probably something wrong, but maybe it's
just some delicious cheese or something. And so what we did is we looked at the existing
smell detectors that were out there, which either take the form of this visualization that
you run through your whole codes set and you get this visualization and you pick through
it, or you get the standard underlining that you get in code with compiler warnings or
compiler errors. And we kind of combined these two together into this ambient
notification thing, which always stays in the same place as you're moving around in the
code. It's always behind the code. And it's not intended to be in-your-face but it's sort of
intended to be kind of a heads up display to give you some information about the code
smells and if you want more information you can kind of click on these individual pedals
and it will tell you, it will give you a deeper explanation of what's going on there. So is
kind of a different way to display static analysis results. We did some experiments there
too which were, good.
And then another problem that we tackled with re-factoring tools is--so it's been brought
up one problem with re-factoring, or one thing about re-factoring tools is you always
have to know which re-factoring you want to do before you do it. Okay. so things like
rename, people are pretty good at that. I'm about to rename a class, they recognize that.
But there's a lot of other re-factorings that maybe aren't so obvious. So for instance, what
is introduce indirection mean? Okay. If you were going to introduce indirection by hand
would you know to go to a menu item that said it introduce indirection? And our feeling
was no. The names aren't actually particularly good indicators of which, of what the
restructuring does. So we did instead is rather than having people rely on the names, we
had, we used gestures instead. We had kind this model of where re-factoring should
appear in a gesture.
So what's kind of nice about re-factoring is there's somewhat structural, so things like
pushing methods up and down, or pushing fields up and down, naturally kind of belongs
in an up-and-down position, okay. So if you want to invoke the re-factoring, we
implemented this with pie menus, so rather than going to the menu and saying, or going
to your contacts menu and saying pull up, you can just kind of gesture in the up direction,
or if you don't know which direction it is, you can invoke the pie menu. You can use
your mouse or you can do a hotkey, a hotkey scheme with that to. Another nice thing
about why re-factorings lend themselves well to sort of this directional scheme, is that a
lot of times re-factorings are opposite. Some things like extract method and in-line
method. Those kind of naturally lend themselves to being on, in opposite directions, and
so we leverage that to put re-factorings on these pie menus.
And we did, we didn't do, we haven’t done a field study with that yet, but what we did do
is we showed that people can intuitively--if we show, if we show them where these items
appear on the menu, they can kind of form a mental model of where they should appear
and can re-create that. What's even cooler is if you show some examples and you ask
them to do a re-factoring that you've never showed them before, Okay. but naturally fits
with the model, they can actually guess pretty well which direction they should gesture to
invoke that re-factoring. Which suggests that they don't actually have to know the name
of the re-factoring to use it.
And one other piece of work--so this is with tools but I'm also interested in not only how
developers use the tools that they use, but also I'm interested in the language, the
language features that they use to. So, for instance, this is a study that I did with Chris
Parnin and Chris Bird where we were looking at Java generics. Okay. and we wanted to
sort of evaluate some of the claims about Java generics, because nobody--there hasn't
really been many empirical studies on it, and certainly no large-scale empirical studies.
So one of the claims about generics, so generics are things like if you have a collection,
you want to type that collection, you want to say what the items in that collection. So if
you have a list, maybe you want to say it's a list of strings.
One claim with that is that if you use generics then you can reduce the number of casts
you have in your program. Casts allow you to force the type system to recognize that
when you say, get something out of the collection, it is of the specific type, and you have
to use a cast to do that. And so there's always been this claim that generics should help
reduce casts. So we evaluated that and it turns out it works, it's true sometimes, but a lot
of the times it's not.
So in this graph what we have is the number of casts in a program normalized for
program length, Okay. so you see it’s relatively, there’s some initial turmoil and then it
kind of it’s kind of more or less flat. And then the blue line is the number of. the number
of generics essentially. Okay. and we see that generics were actually introduced in 2004
and so we see a little bit of generic use from then on and some increasing use. And as
people are using generics more and more, we don't actually see a big change in the
number of casts in a program. And there didn't seem to be a strong correlation between
the two. So, this is interesting too because it tells us something about how people are
actually are adopting the language features that they’re adopting. When they're doing it
and whether the, the ideas that we have about how people should be using, using
language features, are really panning out.
Okay. so I want to return back to our factoring a little bit. So here's our development
environment, here’s Eclipse. So suppose that I’ve made, we’ve got these great refactoring tools. My research has this huge impact. Eclipse totally changes the way the
re-factoring tools are made, or Microsoft now goes out and changes all of its re-factoring
tools. They're all totally usable; their great. So now are people going to use the refactoring tools that we’ve given them? Are we going to see this big uptick? Well, I think
we'll see some, but are they going to find the re-factoring item on the menu? And are
they going to click on it? My guess is no, because the way that, if you don't use refactoring tools the way that you see your development environment is like this, you don't
even really recognize that there is re-factoring menu. You kind of just skip over it. And
this isn't a bad thing, right; this is how we deal with complex environments like this. Is
we don't, you know, we don't have to understand everything about it. We skip over the
things that we don't understand and sometimes we come back and try to figure out what
they are. But a lot of the times we just blank them out. And so for me there's a lot of
things in Eclipse that I don't know what it does, and I don't, don't register for me and all
those kind of appear as I just skip over, when I'm using my development environment.
And so all of these are kind of unused and so no matter how good the usability is of these
particular features in my development environment, if I don't really recognize them, I
don't even try them, I'm not, I won't use them. So I'll give you one other example of this.
So this is the open resource tool in Eclipse. You may not have used it, but you've almost
certainly use a tool just like it. So the way open resource works is if you use the open
resource command, it'll bring up a dialog box that looks like this. You can type, you can
start typing something and the characters that you type will match the different files that
Eclipse knows about. So, and it also works with camel case which is kind of neat. And
so it's, it's actually, it's really a good tool. It's quite, it's pretty usable. It could be a little
better, but it's quite good. And if you look online what people are saying about it they
really like it. So it's number one on the list of the ten navigation shortcuts every
programmer should know. It's one of the most useful tools in Eclipse. It's a command
that I use religiously. And it's the biggest timesaver I've stumbled upon. So this is a tool
that a lot of people really like in Eclipse.
But if you, again if you look at who's using it and who's not, if you say take one month, I
took May, 2009. Only about 12% of people actually use, who use Eclipse actually use
open resource. So even though the tool is really fantastic, it's really great, it's actually
independent of whatever programming environment, you’re, whatever programming
language you use, it'll work for anything you're using resources. People, a lot of people
still don't use it. Okay. and a maybe because they don't know about it.
So, what I wanted you to know about this is, what you really want to know is why aren't
people using those tools? Why didn't they find out about a tool? And that's a hard
question to answer. And so, what I did instead is, well, if we can't figure out what's not
working, let's figure out what is working. Okay. let's find out about how people really do
discover tools. So if you're working in this environment, how might you find out about
re-factoring tools? So we came up with a list of different possible ways that you might
discover a new tool. And we just sort of brainstormed these. This is with Gail Murphy.
And so for instance we said one way you might discover a new tool is through something
and we'll call peer observation, where you observe someone else using a tool while their
programming that she did know about. So say you're doing pair programming and you
see some else use a tool. Or you just walk into their office and you see them use it. And
you say hey that was cool. What did you just do? You learn it that way. Or kind of the
opposite thing happens, we'll call it peer recommendation, where someone observes you
programming and suggests a new tool and says you know you're doing something slightly
inefficiently. There's a tool that actually does that better for you. It'll save you a bunch
of steps; it'll save you a bunch of time,
There's tool encounter where you just happen to find a tool by exploring the user
interface of your development environment, or any other environment like Microsoft
Word. A tutorial, you're reading or watching a tutorial, mentions a new tool and you
discover it that way. There's sort of written descriptions, like standard textbook, or we'll
lump web pages in there too. So you're reading through one of these publications and it
mentions a tool. You could also discover a tool through Twitter or RSS feeds.
Occasionally, you'll see tweets where someone says, if you're in Eclipse and you press alt
shift t, it does this cool thing, of course on 140 characters or less. But it is one way that
you could discover it. We have seen people at least broadcasting that they discovered
new tools.
And the last one we thought of is discussion thread, where you learn about a new tool
after reading it on a list of comments or a forum or an e-mail discussion. And what we're
really interested in here is examples of where someone discovers a new tool where they
didn't know they had a problem, okay. Where it's not that they're going out to find a
solution; it's not that they're going to search Google; or not that there going to go ask a
friend, it's that they find something that they didn't know about in the past. Okay. So
we've got these seven different categories, and what we want to know about them is how
often do they happen, okay. And how well do they work. Are they effective if you learn
a tool, if you are discover a tool via one of these methods, how likely are you to use it
three months later or six months later?
So what we did is for frequency, all right, we asked people just to tell us some stories,
Okay. We did interviews. This is the first study that we've done. We did interviews
with developers and we said, we give them a list of different tools. We had a bunch of
tools from Emacs, and a bunch of different tools from Vim, a bunch of different tools
from Visual Studio and a bunch of tools from Eclipse. And we said now here's a
development environment you might use. Here's a tool, tell me how you discovered that
tool. How did you find out about it in the first place? And people told us stories. And
what we assumed was the more frequent they told the story, all right, the more frequent
the type, the more likely that type happened. So for instance, if everybody told us they
discovered that tool from Twitter, we would say that Twitter is a frequent mechanism for
discovering new tools. Now that relies on people's memories, which is questionable and
I can answer any questions about that.
Why we think that was an okay method to use. So that's how we figured out frequency
and I'll give you the results in a second. In effectiveness, we asked people to self rate in
ctiveness. So, we said here's seven different categories of ways that you could learn
about a new tool. Tell me what you think the most effective way is. What's the best way
that works for you in terms of using the tool later again in the future? And so here's what
the results looked like. Each of these boxes represents a single person's response. And
so what you find is that tool encounter, the one at the top, that's what people mentioned as
the most common way they discovered a new tool. Okay. they were poking through the
user interface, and they said oh there's a re-factoring menu. I wonder what this does and
they try it. And previous results actually predict that. That's not a big surprise. It turns
out that that is the most frequent way that people discover new tools. What previous
results don't tell us is whether it's really effective or not. And the answer to that
according to the people we interviewed was actually no, Okay. essentially not particularly
effective, especially compared to learning it from someone else.
So we did the study partially because we were especially interested in these first two,
learning socially from someone else. And what we found is learning from someone else
actually doesn't happen particularly frequently, but people like doing it a lot. It's quite
effective. And so the challenge here is it's, it's--we'll call these two together peer
interaction, interacting with a peer. So why is it effective, and why doesn't it happen
often? Okay. so that some answers to that based on the interviews. So why does it work
so well to learn from someone else? Well it works well because typically the person that
you learn from is someone that you work with as part of your normal work. If someone's
likely to walk over to you and look over your shoulder, that's probably someone you
work with rather than someone random who comes in off the street. So there's a high
probability of relevance there. What's also nice about this is the tool is demonstrated on a
real problem. It's not like you have to come up with a contrived example of why this tool
is great. You get to see it in the context of use. So it's on a real problem.
The learner feels like she discovered it herself, which is, which is nice so when you see
someone else do it and it's not necessarily like someone is forcing it upon you. It's more
like, it's kind of self discovery. Which people reported was important to them. The
learner associates the tool with its context of use which makes it memorable. Okay. So
they see the tool used in its context and what they report is next time they're in that
context they think of using the tool again. All right, so why doesn't it is happen to
frequently? Does anyone want to take a guess as to why learning from someone else
wouldn't happen frequently?
>>: Because programmers don't have a lot of friends.
>> Emerson Murphy-Hill: Because programmers don't have a lot of friends, great.
Nobody mentioned that. Nobody wanted to own up to that. What you think, what do you
think they reported as the biggest barrier to learning from someone else?
>>: They don't want to show you [inaudible]
>> Emerson Murphy-Hill: Don't want to show your toys, your tools?
>>: [inaudible].
>> Emerson Murphy-Hill: Yep, that will come in. It wasn't number one. The number
one thing was, of course, physical isolation. Okay. people not being in the situation
where they would naturally look over someone's shoulder. That's kind of obvious right.
Lots of people work from home at least part of the time. Or in some cases people work
from home a lot. So physical isolation makes this hard, but not impossible. Some people
talked to us about doing some remote pair programming with someone else. They
weren't using anything fancy like videoconferencing. But what they were doing is they
had a remote vim session open and they had a chat session open and they switch back and
forth. And one of them reported it as, I saw some text move around on the screen and
then I had asked my pair, my buddy what was going on with that. And he explained oh if
you press this particular combination of keys, you get that affect.
Okay. So physical isolation makes it hard but not impossible. Different programming
environments definitely make this hard. You know of some people on your team are
using Emacs and some of them are using Visual Studio, then different programming
environments, the two can't learn from each other say quite as much as they could if they
were using the same programming environment. But even if they are using the same
programming environment, at some point your team is going to kind of, you might call it
annealing, right, they all get to the kind of, the same level of tool proficiency. So once,
once I learned a lot from you, at some point there's not going to be much more that I can
learn assuming you're not learning new tools outside of that relationship. So acclimation
to each other's toolsets is one reason this may not happen very often.
Sometimes company policy dictates which tool you should use. This one came up a few
times. And so if the company says you must use Visual Studio, and you don't have a
choice even though you see your friend using something else, and you don't have a
choice to switch then obviously, basically that's a barrier to learning from someone else.
However, even within, even if you're, typically when someone dictates that you must use
this tool, typically it's a high-level tool. Is something big like Visual Studio or something
big like Eclipse, and even within that there's lots of new features or new tools that you
can learn. Time pressure makes this hard for sure. If you're working under a deadline
people reported well, I don't have the time to, you know, share my knowledge with
someone else. So when there's time pressure sometimes people don't share. And
sometimes there's an unwillingness to. This was kind of rare but some people talked
about sometimes people in the organization were not willing to share a tool with someone
else partially because, that is kind of what makes developers proficient sometimes. What
gives them their skills is the toolsets that they use, right. That's kind of a, kind of, it's
kind of like if I told about my tools I'd be giving away my secret sauce, so another reason
why it doesn't happen very often.
So here's the research challenge for me. So what we know Okay. Is that peer interaction
doesn't happen very often but it works really well. So if we could make it happen more
often, Okay. and people actually like doing it. They're not forced into a situation where
they don't want to do it. If people had the opportunity to learn from others more, they'll
probably learn more tools. So how can we make, how can we sort of tackle the number
one problem, people working in different places. How are people who work remotely
and say asynchronously, they're working on different parts of the globe at different times,
how can they learn from each other? Okay.
Okay. so I'll give you a proposal, all right. So one way we can have developers learn
from each other if they're working asynchronously, is that for each developer we'll just
keep this say into one team. We won't talk about the global ecosystem of developers.
For one developer we'll keep track of what tool she's using, okay. And so each time you
use a tool, make a note. And we have certainly got infrastructure to do that. And also
keep a full movie or a full screen cast of the developer’s history from beginning to end,
okay. Just a full, a full screen cast, all right, so that at any time you want to go back in
time and you can see what you were doing last week, or you can see what you are doing
last month. So a lot of you are thinking, okay, there's privacy issues here. I'm not going
to let anybody else see this sort of thing.
We can talk about that. But, what we can do with this data is we can look at the tool
histories and we can compare them between, let's say, all developers on your team, all
right. And by doing the comparison, we can use something like say collaborative
filtering to make recommendations about tools. Okay. we can say that, you know, there's
one tool that I don't use that everybody on my team knows about. Okay. And maybe I
should know about that too. We actually built, or Gail Murphy has built a tool with a
master’s student, where they actually did bank recommendations like this. And one
difficulty of these sorts of, just making recommendations, in saying, you know,
everybody else uses this, but you don't. Is people don't, people don't take very well to
those sort of recommendations. We get to the clippy the problem right. People don't like
the sorts of recommendations where they are kind of, where they're sort of out of context.
But what we can do, if we take recommendations from individual people, and we say
who it is, who's making the recommendation, and maybe if we just connect people
together, they'll learn from each other. So what we could do is we could say, you know,
we'll call one guy Rob and the other Tim. And we could say Rob your coworker Tim
uses open resource. Would you like to see an example of him using it yesterday? And
again, privacy issues, there will have to be some mediation going on here. But what you
can do is you can go through Tim's history, you go back in time to the last time he used it
or one time before that. You can take out a little screen cast, ship it off to the network
and then Rob can learn from what he did. Okay. It's not synchronous, because this
person did in the past. And it doesn't require any extra effort because the example’s
already there.
>>: Yeah this could be great [inaudible] relevance to what your current task is.
[inaudible] ah ha, with peer interaction, working on something [inaudible] I wrote a
really long e-mail saying [inaudible] and he said do you have time for a chat? And so
like he called me up and then he said, will you share your screen so I can see like visually
what's happening that's confusing you? And so like a communicator I just shared my
screen and he said, Okay. I understand now what you meant in that long e-mail and then
he says, Okay. can I see the source code now? Then he turned his browser on he said
well, you know this other thing that I did that is similar to what you want to do. And like
in five minutes like we were done.
So like, I think that I like the making peer interaction more frequent. Here was a guy
across campus and you know just the ability, I mean e-mail is like stupid. I should've
called him and said look at what's happening, I don't want this to happen and then we
could have this like really [inaudible] discussion. Instead I sat for like twenty minutes
typing this e-mail and trying to make it a perfect description, formal text [inaudible] so I
think somehow, somehow there's, if you can't get face-to-face you know you can talk to
the person, you can talk to the person and do these sort of things. [inaudible] have a
priority right? First I should really talk to the person who is the expert in this [inaudible].
If I can't do that then I need these, then I need these proxies from the person right of what
they did. [inaudible] you know even before I [inaudible] would just make it easier in an
organization you know to just contact people and get their time. This guy actually
happen to be very willing to share his time and you know suggested, wow, share your
computer screen with me, and I'm like, wow, that's an novel idea. And five minutes later
we were done.
So I really would just encourage you to think wider than just, the direct communication
like, wow, I really solved my problem quickly. I was such a happy camper because I
could have spent hours drilling down into the code, and browsing and thinking and
learning the code and, yeah, that might have been worthwhile but in the end getting the
solution which is good enough for what I to do now. So I'm thinking there's also that
balance, you know, I had a task to get done verses, browsing state. It may be in relevant
on may get some learning from it on the way getting [inaudible].
>> Emerson Murphy-Hill: The timing is definitely important. And this doesn't address it
at all. When you interrupt someone, or if you don't interrupt, if you wait for them to go
out and get the information they need, that's an open question.
>>: But I mean the title was absolutely right on. I think that enabling that within the
whole business of the privacy in history and so on, you know, I can't imagine that if this
guy had reported what he done like the little tidbit that he gave me would have been
captured, how I would've found it. [inaudible] an interesting search problem. I got
something that I want to do right like some thing somebody else did in this case.
>>: It seems like there are different levels [inaudible] like the suggestion here which is
completely unaltered to recapture. [inaudible] there's another level of investment which
is authoring something, some people will put together blog posts, you know, I love this
tool here’s why I use it, here's how to use it. There's another level which is this real-time
on my shoulder, over the shoulder [inaudible].
>>: What this peer interaction really means, like if it's mediated it would be a blog. So I
was taken the interaction more literally as he [inaudible].
>>: No I don't think given one, you're going to nail one…
>>: No, no I'm not suggesting that at all. I'm just saying that this sort of things here are
not, our proxy for peer interaction; they aren't peer interaction.
>> Emerson Murphy-Hill: Okay. thanks. That's good points. So, so in this first of
situation I'm talking about, we're saying, you know, there's something you don't know,
maybe you should find out about it. Okay. and that may or may not work. People may
not want to go out and sort of admit their, admit their ignorance of something. But you
could do it; you could flip it around to. You consider say to the other guy, you could say,
Rob, your coworker, or Tim, your coworker Rob doesn't seem to know about open
resource. Would you like to share an example with him using with it? So the example
that you just brought up of say putting it on your blog, for example, that might be kind of
an indirect way of to do it. We could just kind of take that snippet of your, of your
history, you know, and putting it on Facebook, for instance, that might help too. Yeah?
>>: [inaudible] a couple weeks ago and I was wondering if you thought about this which
is I guess I have to call it peer competition. It was the idea of trying to help people
discover natural abilities of the tool. Which was just to me was, here’s some
achievements that could be personally interesting for you to do it as an individual, but I
suppose we could explore the same space as a team [inaudible] were every group is my,
is the ones that we learn from. Competition [inaudible].
>> Emerson Murphy-Hill: Yeah, the gaming, yeah, the gaming aspect could definitely
be leveraged. This worked well for things like stack over flow where there is a certain
level, the more points you get, the higher status you are. But yeah, as you said even
within your own team. I think one question, one open question there though is how much
do people actually value that? I don't know. So you could take another example like
Olo, which rates open source developers against one another. That one I'm not sure that
people actually take seriously or not. So what's the difference between something like
stack over flow that people would, I would guess people take more seriously than they
take overflow. So I think, you know, using, maybe using what we've learned about
games in the past in the development context could, is definitely promising for sure.
Okay. Great. So maybe one solution to making peer interaction more frequent.
I want to share one other thing that we noticed by doing these interviews and that
discovery didn't happen uniformly across different groups of people. And one thing we
actually notice was there was an age difference between older developers and younger
software developers. So someone from an older teammate in a group said the junior
members tend to be more voracious in their desire to learn new APIs and tools and stay
plugged in to what's going on with languages and stuff. My time is spent digging into
more bugs and more things that I'm responsible for delivering, and I have less time to do
independent research. No one's upset when a junior member says they have a better way
to do things.
And I kind of saw the same affect from the younger side too. So someone, the younger
teammate said there's a fair amount of bias towards me teaching other engineers
something. I'm a student and intern in the process of learning as many tools as I can
from, learning as much as I can from as many tools as I can. Several developers I know
especially those that have ten, twenty, thirty, forty years of experience tend to say that
they have, say that they know the tools that they use and they don't necessarily have the
time, or more commonly the patience, to sit down and fiddle with a new tool. And I do
know about you but that kind of resonates with my experience too.
So what was interesting was that this sort of this dynamic where the tool knowledge
maybe comes in through the younger developers where they have time to mess around
and poke through the user interface and waste their time not actually developing
software, but playing with the tools they have and kind of filtering up socially to the older
software developers. And I think that's kind of, and I want to kind of go off into talk a
little bit more abstractly about this. I think there something there. You know we know a
little bit about gender differences between software developers, but we don't actually
know much about an aging perspective.
And we have reason, we have reason to suggest that it's important. So, for example, the
study that I just talked about suggests that technology introduction often starts with
younger workers and eventually may be reaching older adults, assuming that the peer
interaction actually happens. But one other interesting happened, is that older adults
reported they were actually both less likely to learn and teach others via peer interaction.
That was somewhat unexpected. Learning less, I can believe that. They’re, you know
they’re experts; they've been at it for a while. But teaching less was actually a surprise.
Okay. They said that it actually happen less often than their younger peers. And if you
look at sort of the other literature, there's not much about software development, but
there’s certainly a lot of research into aging and certainly aging and technology. If we
look at that, we can also get sort of a perspective on aging. So other studies have
suggested that there's a belief in practice that technology careers are very intense and
focused. So stereotypically we think that, you know, it's a young man's game and, you
know, between the ages of ten and thirty, you know, you are going to be working 80
hours a week or something. And then after that what happens, right? You burn out.
And so what if we thought about how could that be, what's going on there? Why are
people saying not, why isn't, why are so few interested in continuing, continuing to do
software development after these intense careers? Maybe they need to be less intense.
And finally, you know if we kind of look at other sort of physical and cognitive factors
that tells us interesting things too. So, for instance, as we get older our ability to filter out
irrelevant visual information goes down. So, for instance, what you might notice is that
kind of by analogy it happens, it happens in audio too. If you're in a party and there's a
bunch of noises going on, bunch of other conversations and you're having a conversation
with one person, typically your ability, you can filter out the other information, you can
filter out the other conversations and you can focus on your one, on the one person. And
the same happens visually, okay. Typically you have your one task. And you can filter
out other parts of the, the whatever task you're working on. If you're adding numbers and
there's dollar signs, you can filter out those dollar signs because they are not relevant to
your task. But as it turns out as you get older your ability to filter out the visual
information not relevant to your task goes down. Okay. I think it's interesting because
development environments are really, really complicated and we talk about you know
filtering, how it's important to be able to filter out features in your development
environment so that you can kind of deal with it effectively. Maybe, you know, because
of this factor our ability to deal with the complexity of our development environments
goes down as well. Just kind of some speculation.
>>: Some of my favorite Eclipse features is double click the tab, full-screen. You
missed that.
>> Emerson Murphy-Hill: Yeah, yeah, that's a good one for sure.
>>: Thanks.
>> Emerson Murphy-Hill: Yeah. Peer interaction. So kind of this is a little more farreaching. I don't have much research to show on this, but I think we don't have a very
good understanding of the opportunities and challenges faced by older adults in software
development. And I think we have a lot to learn there. Co I wanted to leave you with
just this, just as one last slide kind of summarizing what we talked about. So with our
factoring, there are still some open questions here. So how does increased usability of
tools translate to increased use? And, kind of in general, how can we communicate
effectively with our development environment? Our environment has a lot of things to
tell us. It has lots of information that we, that it would like us to be aware of, static
analysis information, compiler information. How can we effectively communicate with
it? In terms of discovery, what personal practices—ignoring, you know, the technology
solutions, what personal practices can developers follow to stay aware of new tools? Is it
about spending a lot of time on Life Hacker looking for interesting pieces of software? Is
it about spending more time pair's programming, spending an hour week doing it and
rotating between programmers? Don't know. And how can physically, as we talked
about, how can physically distributed developers discover, discover tools from one
another? And in terms of aging, how can we effectively, how can we lengthen the
effective working career of a software developer? And is that even something that we
want to do? And what challenges and opportunities do older adults face in software
development? Okay. so this is, I'll be around for the rest of the week, like I said. And so
if you'd like to meet with me, I would be happy to talk with you. If you got some great
examples, I like to hear them. I like to hear about your experience so thank you.
>>: [applause].
>> Emerson Murphy-Hill: Questions?
>>: Has this research been written up in a white paper of some sort or?
>> Emerson Murphy-Hill: The aging part?
>>: Just, the whole presentation in general. Because I see you had several references of
different studies.
>> Emerson Murphy-Hill: Yeah, there isn't a single, single unifying paper. This is, this
presentation is probably it. But you can look for my NSF career award, which will
probably be quite similar. Other questions? Points of contention? I don't agree with this.
Yeah?
>>: So I know you didn't do too much research into aging, did you get any sense for
whether if at all there was transition between younger and older developers? Like are
technologies adopted because your developers are sharing, or because just a generational
shift?
>> Emerson Murphy-Hill: That's a good question. I think, I don't have--so your point is
that there are two, two different factors that might be going on. One in terms of learning
new features, one of them is that maybe people are using the same tools just because
that's what they started with, right. And there continuing to use them. And the other one
is that maybe they transition to new tools because they find out about them through some
mechanism. And I don't know. Actually can't say much about whether one happens
more than the other. Which factor is more important? But there's definitely, there's
definitely, you know, in addition to, you know, your ability to be in the situation where
you're going to learn something from someone else, there's also kind of your willingness,
your willingness to learn something from someone else. And sort of the, one of our
interview respondents kind of called it developer inertia. Where, you know you got what
you have and it works pretty well, so why should you, why should you transition to some
tool that may or may not save you a few seconds? But it's a good question. I don't know.
Any questions or comments? Okay. Thanks, thank you all for coming.
>>: [applause]