>> Merrie Morris: It's my pleasure to introduce Michael Twidale from the
University of Illinois at Urbana Champaign. Today Michael's going to talk about
extreme research. And Michael is a professor in the School of Library and
Information Sciences at UIUC where he does a lot of research on computer
supported cooperative work, computer supported cooperative learning, human
computer interaction and information visualization.
Currently he's been studying informal social learning of technology, technological
appropriation, collaborative approaches to managing data quality, and the use of
mash-ups to create lightweight applications. And of course the usability of open
source software.
His approach involves the uses of interdisciplinary techniques to develop high
speed and low cost methods to better understand the needs of people and their
difficulty using existing computer applications as part of the process to design
future, more effective systems. And I think we'll be learning a lot about that
today, when Michael challenges us to think about the future of HCI research. So,
thank you, Michael.
>> Michael Twidale: Thanks very much. So my title is extreme research
question mark, and so I don't have an answer. I have a lot of rants. And I'm
hoping to discuss with you guys some possibilities of addressing this larger
problem. So this is also kind of like a verbal grant proposal. I'm trying to put
together my argument of how I might get the NSF to let me work on inventing
some methods because I think our examining methods are very powerful but
never for what I want to do. They always seem to work for other people.
So what is it I do? Well, as Mary sort of summarized, I sort of feel I'm
somewhere in the intersection between computer supported cooperative work
and computer supported collaborative learning, human computer interaction, and
library information science. But really what I obsess over is learning. How do
people learn how to use an application initially? How do they learn how to get
better with it? Why do they not get better with it?
And increasingly I'm releasing that you can't just study a person on their own
because lots of learning is social, and you can't just study one person or two
people around an application because outside the lab people have five
applications open at the same time. And they're copying and pasting between
them.
And when computer scientists do that, we call it a mashup. But other people do
it, too, just using basic copy paste. So I want to -- I've been thinking about
mashups as a technology that allows rapid prototyping but also kind of like a
metaphor a way of looking at research problems that you combine together.
So I think it's always useful to try to understand somebody by looking at their
book shelves. So I'm just going to take you through my bookshelf very, very
quickly. These are books that I've read that I found inspirational. And if you
recognize these books and run them through in your head, you get a sense of
where I'm coming from. So all of those, very, very quickly. Glance on the
bookshelf. And what do you get as the distillation of all those books?
I think my design approach, although I've got a background in computer science,
I don't really want to automate, I want to augment. And a people are smart.
They can do things themselves if we design to help them. And what the things
that they do?
Well, they do a lot of learning and helping each other. But if I just say I'm going
to study learning, I go in, what would I think is going to be a learning interaction
but then it can turn into a problem solving. Because the learner asks the help
giver, and it turns out the help giver didn't know either. So now they've got to try
to figure it out together.
And a lot of it then leads to what you might innovation and what's the difference
between learning and innovation? Well, it's kind of a weird one. But I would say
it's learning if somebody knows it and I don't know and it's innovation the nobody
knew it either such a bit like research. And a lot of this is sort of fiddling around
with stuff and tweaking, the kinds of things that computer scientists find very
comfortable to do with technologies. A lot of non computer scientists don't find
that very comfortable, but some people are innovating even though they swear
blind they're not a computer scientist. So I'm interested in how can we help
people do this with other people even if they're not computer scientists? What
are the kinds of tools and resources and interfaces that would help them do that?
So a few more rants. Why is it that I feel that the methods that we have in
research don't help me do what it is that I want to do? And I think this -- part of it
is this distinction between science and engineering. So, you know, I would say
science is about finding fundamental truths about the world and engineering is
about making things better. And I want to make things better. I want to help
people learn more efficiently, more effectively, do more interesting things with the
software to make the world a better place.
This is clearly not just a technical problem. It's called a sociotechnical problem
it's about people who are using these systems. And so if you want to study the
sociotechnical effects of designed artifacts, it's kind of difficult to do it in the lab
setting. Because often in the lab setting in order to do proper science you control
away all the messy complicating factors of the real world and it's precisely those
messy complicating factors of the real world that I'm interested in.
So you know, the simplest sociotechnical issue I can think of password design.
So passwords are the only thing I thought of as something where there's a
technical fix. Oh, that's a bad password, I as a systems administrator am going
to force you to have a longer password because longer passwords are safer and
longer passwords that also include numbers and various, you know, top-level of
your keyboard things of asterisks, explanation points, they are also safer. So
longer is safer.
And in a technical perspective that's true. But in a sociotechnical perspective it's
not true because the longer you make the password the greater the probability
people will write it down. And that lowers the safety, the reliability of the
sociotechnical system even though I've improved the safety of the technical
system. That's why you've got to see it outside the lab.
It's also hard because when we're talking about programs, these technical -technological artifacts could be otherwise. You know, there's always a design
space. I could have designed my program a million different ways. And when I
evaluate it, am I evaluating this particular manifestation of the idea or do I want to
claim this applies to all possible manifestations?
Also, there's all this new stuff coming along, all new programs, new opportunities.
So I haven't got time to rigorously test everything. So how do I decide what I'm
going to bother testing? So my frustration is I feel that a lot of the current
research methods are kind of like a scalpel, you know. If you know what you
want, you can get to it very precise and accurate. And I feel I'm trying to, you
know, hack my way through the jungle with a scalpel and it's not appropriate.
What I want is to invent a machete.
A lot of traditional science is about figuring out a question and then answering
that question. And the figuring out of a question, nobody every talks about the
method of how you go about doing that. And in these sort of wide open areas of
whole new kinds of technologies and new people using it, the whole point is the
starting point. How do you even figure out where is the question that you want to
ask in the first place, and what might be some methods to help you invent the
right question to ask?
And then, you know, you can use your methods to try to answer the question.
But even figuring out the right question to ask is really important. Because the
danger is that if you have got a research method in your hand, maybe that
influences the kind -- the kinds of questions that you even think are possible to
ask because they're answerable by the method you're holding. So you just may
have a whole blindness to whole category of other questions just because of the
method you've got.
So here is the famous waterfall method from software engineering often claimed
to be invented just in order to be knocked down. But I think we all recognize it
from software engineering classes as this is how some people say some
software development projects should be done and then generally they've shown
that in order to say but it doesn't work for another category of software
development methods and so I've invented another software development project
and so I've invented a new method that's better than it.
So when might the waterfall method work in well, maybe if you've done this time
and time again and when is it very likely not to work when you've never done
anything like this before? And so I put it to you that this is where you see the
waterfall method. Well, my application to NFS says I'll do that, then I'll do that,
then I'll do that, then I'll do that, then I [inaudible]. And it's very strange when
we're talking about research that you can turn research into such a production
line.
And then we write it up and well I did that and then obviously logically did I that
and then obviously logically I did that and then obviously that popped out
because I'm a genius. And this -- this is very disconcerting to a lot of graduate
students who read papers that imply everybody else does that and [inaudible] all
over the place and I'm doing that and I tried that and it's like I'm obviously not a
good researcher because what I do bears no relation to what I read everybody
else says they're doing. To which you have to say that's because they lied.
They tidied it up after the event.
So we have this method, the waterfall method that sort of raised -- may well work
when you know what you're doing, seems to be clumsy when you're doing
innovative software development. When you're trying to do research, you
wouldn't think that the waterfall method would work as an analog for how you go
about doing research.
So makes you wonder well, if the waterfall method of doing research -- and I
don't like it -- what might be some other ways of doing research? Well, we've got
things like the spiral method and the problem is I think it's a great way of doing it.
You try it out, you analyze it, you figure it out, you do it again and then you do it
again and then something good pops out at the end. And the problem is I'm not
so sure that many research proposals say oh, I'm going to do this, I'm going to try
it and then I'll discover I've done it wrong, so I'll do something different, please
give me the money. No.
But I do think that that's what people actually do very often is just they don't say
that's what they're going to do. So [inaudible] what would happen if we actually
told the truth about how we really do research in our grant proposals and in our
write-ups? And, you know, Bruno Latour studied scientists but I wonder if some
of the things that he found about how a lot of the worker of lab scientists is about,
you know, taking an idea and basically building up a stronger and stronger
rhetorical case for why this idea works. And then you assemble more and more
evidence until more and more people believe it. And your job is to sort of be
convincing everybody else of it.
Maybe you could be thinking about things like that. Why is this a prop? Well, if
we're messing around with sociotechnical systems as I am, they're very grubby,
and they're very messy, and so I would like some methods that embrace this
mess. And I don't think I'm going to find many of these methods sitting on the
shelf, research methods and say well, grab that one. So I think I'm going to have
to be thinking about how to actually design that methods.
And it's strange we don't often talk about method design. We talk about method
selection. Okay. Put these methods on a shelf. You have good reasons. This
method is good for that, this method's good to that. So you figure out what you
want to do and you pull it off the shelf. But just as, you know, it's a group of
computer scientists that forever building things, you know, the act of design of
building a computer program is something that people do. And you can study the
processes of design and say okay, yeah, this guy not only builds things but these
got some very interesting design processes and I can learn from that and teach
somebody else.
How might you actually talk about the process of designing your research
methods? As opposed to saying oh, well, where did that come from? Oh, I had
a flash of genius because I'm a genius and that just popped into my head. You
know, how did you design your program? Oh, I had a flash of genius, it popped
into my head. Well, no, surely you have some methods of how you went about
designing your program. So surely you might have some methods about how
you go about designing your research.
Now, some existing research methods I claim cope with mess by tidying it away.
And a lot of controlled experiments do that, you know, rigorous controlled
experimental design is about getting rid are of all the mess so you just measure
the thing that you want. And I'm saying well that's perfectly legitimate if you know
what it is you want. But often I worry that I may tidy away something in the mess
that's really what I care about. So I want methods that embrace mess rather than
tidy it away.
So here's an inspiration which is extreme programming. And I need to say I have
never done extreme programming. Read about it, sounds like a nice idea. I'm
intrigued by it because it seems to me to have been designed, developed and
refined precisely to address some of the problems that programmers have with
traditional software development processes. They seem to me strongly
analogous to the problems I have with the doing of research. And just a sort of,
you know, as the waterfall method is extreme programming so extreme research
is to these more traditional rigorous scientific methods.
So might we look at some of the elements of extreme programming, and might
we look at some of the design processes that led to the design of extreme
programming to help us think about how we might design some research
methods that will help us embrace messiness.
So why did these people come up with XP? Well, a whole load of reasons. But
some that intrigue me is it's really hard to find out what the requirements are.
And you're dealing with clients that are not computer scientists so they don't
necessarily know what to ask for. They know what it is they do, but there's an
awful lot of things they do that they don't actually remember that they do. So you
build what they ask for, and then think say no, that's not what I want, I want
something else. So then you build it again.
And so you have this problem of understanding what the real requirements are
and how that's different from the articulated requirements. And even if you can
find the underlying implicit requirement, the world keeps on changing. They have
this billing system and first of all, they describe and say well this is what we do.
And then you say oh, do you mind if I sort of follow you around? And you say oh,
well what about this, oh, yeah, that's an exception. That's what we do on this
one. What about this one? Oh, that's an exception. What about this one? Oh,
that's an exception.
And a certain stage you start to think like is there anything around here that's not
an exception? And sometimes it was they give the ideal state which is how they
used to do it years ago, but they don't anymore, the world's moved on [inaudible]
exception, or sometimes it's the world is so complicated that they've created this
idealized billing that's like a centroid and everything else can now be very nicely
explained as just little exception toss that. But they never actually do the one
they just told you about. But it helps you make sense of all the others.
So it's a very good human problem solving way of explaining to another human
being what we do as an idealization and exceptions. But it's an absolute disaster
if you try to program if you think that's the central one.
So everything is kind of changing. So how does XP address it? Well, in many
ways similar to what happens in research. You start off with one thing, you build
it rigorously, you test it, you check it out, and then you gradually add things to it.
So sounds nice. It intrigues me that extreme programming and software
engineering look very different. But they seem to be kind of compatible. So
again, if we've got software engineering and its manifestation of extreme
programming, maybe we have research and this new inventive manifestation
unthought of yet which is extreme research.
So what are some other things? And again, I'm just going to bundle together a
whole describe of work on this extreme programming, agile program, all these
others. What are some of the things that they do, particular things they do and
then some of the approaches to design that we might be thinking of?
So one is pair programming. Now, why do they do that? I can think of at least
two reasons. One is you can spot bugs and the person will say why did you do
that? And you externalize the design rationale. Why do you that? Now, we
know it's very hard to make any computer scientist document their code. Why?
Well, you know, coding is what I'm here for. And writing stories about what I'm
coding isn't -- my boss makes me do it but I don't like it.
And why is this being documented? Oh, so that in 20 years time, when you've
gone somewhere else and somebody is maintaining your code and they'll find
that useful. So you're writing for a hypothetical person you're going to never
meet and you can't really image them and you're not very good at writing which is
why you got into programming in the first place. And is it surprising people don't
do it?
But if you see the nexus operator, why did you do that? Firstly you may say I
didn't mean to do that, that was a mistake, so that's the debugging. The second
is you are actual going to verbally give a design rationale. So we've now
externalized the design rationale.
And the third bonus is if I say to you why did you do that, you're going to explain
to me, and I may learn something about your programming techniques, so I may
be learning about the process and you are -- you are necessarily reflecting on the
process in order to explain it to me. So you may actually get better at doing it as
well. So lots of benefits in pair programming.
There's a very strong emphasis on testing. Now, this is where it looks really
promising but we're going to have to do some work to turn it into extreme
research. Because as I understand extreme programming, they try to automate
the testing. And I have great difficulties imagining how I can automate the kind of
sociotechnical testing that I want to do.
But they do testing early and often, okay? And I want to do testing early and
often. Probably not automated but it's interesting to think how do the testing
methods we call evaluation and research they're normally very rigorous and very
slow and very careful, and how could we do them earlier, quicker and dirtier and
then go round the loop many more times? They focus on lots of little changes.
And always having something that works at any one time when you can actually
expand the functionality. And I think that's kind of interesting and different.
Too often what we try and do in research is find some really interesting research
question and do one big experiment to definitively answer that question, this big
block of stuff as opposed to these little incremental changes and little incremental
tests. And what would little incremental bits of research be that's useful as
opposed to sort of you know pejoratively publishable unit.
They don't try and get it all right in one go. And certainly XP really focuses on
the functionality. And I'm more interested in sort of the interface and usability.
But this might give me some ways of thinking about it. And I'm sort of
challenging those of you who have done any sort of extreme or agile
programming is sort of run through in your mind some of the techniques you've
used and think about how they might be either directly mapped into solving a
research problem or indirectly mapped.
So, you know, we may not be using the same technique, but we might use the
same heuristic of how you got from regular program to extreme to get from
regular research to extreme research. So be thinking of those, and I shall invite
you to share those a bit later on.
So how are we going to do this? Well, we know that regular research methods
are slow, expensive, but very careful and very rigorous. And why do I have this
problem? Well, they don't seem to work in the kinds of things that I'm interested
here, which is a sort of, you know, horribly embedded situated people being
creative, I want to learning episodes in the workplace and the most successful
ones are the ones who take 10 seconds. So you're hanging around doing
[inaudible] nothing's happening work, work, work, all learning episode, work,
work, work, learning episode. So they're rare events, you know, if they're
working well. And when they're slow is when they go wrong.
So studying them even with using ethnographic methods, which I'm a big fan of,
it's still a rather inefficient method. There's an awful lot hanging around. So what
might we do to speed up this process? I'm interested in learning. And the more I
study learning the more I find it sort of smooshes out in a million different
directions. So as I said, it starts off you think you're looking at learning but then it
gets into leaved with work and then it gets into leaved with problem solving and
then I get stuck and ask you for help, and before you can even tell me what the
right thing to do you've got to help me get out of the mess I'm in because I tried
to learn on my own and I've now messed up my system.
And then people adopt these technologies and mix them around. And the
interface can totally swamp the functionality, those of you who have done ACI will
know. Genuinely in a negative way, bad interface can get in the way of really
good functionality. But sometimes it's the other way around.
And some of my frustrations with a lot of research papers is they do all this very
careful analysis and then they come up with these very feasible design
implications. It would be better if the interface was easier to use. Well, yes,
that's true. But you really have to spend, you know, all those hours of work,
something that's so blindingly obvious.
Sometimes they have the really rigorous definitive proof of something that I
always thought was true anyway. So you've not told me anything other than
finally somebody's actually proved that the sun rises every morning. And yeah, it
is strange nobody actually bothered to prove it. But I didn't really -- you know, I
want to be taught something that's surprising, something that's counterintuitive.
And so many people go and do an ethnographic study and then tell me things
and I think well isn't that obvious? Couldn't you have like written down and put in
a sealed envelope all the things you thought you were going to see in the study,
then you do the ethnographic study, then you open the envelope and say oh,
yeah, got that, got that, got that, got that, so confirm those, but I really can't
prevent I discovered those in the ethnography because already wrote them and
they were in the envelope. But here were some things that I predicted I would
see and didn't see, oh, that's surprising. And here are some things I found that I
didn't predict. That's also interesting.
But pretending that everything you saw in the ethnography is a justification for
doing the ethnography seems to me a bit of a cheat because I'm sure you could
have sat down and thought have some of those in advance. And so the question
is now what we're trying to do here? Are we trying to verify something or validate
it or are we trying to just discover it? You know, what are we trying to prove?
So I'm interested in this sort of learning approach, and, you know, I sort of one of
my heuristics is follow the learning and focus in on that where ever it happens. A
lot of it is collaborative. And I've studied lots of different places, including a
castle. But the reason for trying to study lots of different applications by lots of
different people and lots of different places is to look for, you know, what is
surprisingly similar, you know, in what way do mashup programmer computer
scientists have something in common with secretaries in common with post-docs
in astrophysics. Or all may be combining existing applications together to do
something.
So you get these recurrent themes, situated social informal learning, and when
you look at individuals the same person can do some things completely
differently. So lots of things going on. We now also have crowds and clouds.
So there's resources out there that people can use. And these may be other
people and they may be other software applications, little Web resources that
you can use.
So we've got a certain degree of software abundance. It's not just that people
are building stuff from scratch. And so this is very nice for the people, building by
assembly, it's easier than building by scratch. But studying it can be can really
hard.
So I've got various examples. I'm just going to go through a couple of these. But
if you want to ask me about any of the others later on, I want to emphasize that
this talk is inspired by various different projects. I keep on seeing the same kinds
of problems, same kinds of thematic issues, and I suspect the same kinds of
design research methods might have helped get to the truth a bit faster.
So the first one is going to be talking about mashups. And again, I just want to
show my age and say when I was a graduate computer scientists you started
with a black screen, you know, and then you spent many, many months building
a program and then because I was interested in AI, intelligent tutoring systems, I
then stick somebody in front of it and evaluate it. And then it would all go horribly
wrong, and then I would go and rebuild it and test it again. And that's what you
did.
And now, the young people of today, what happens in they do something very
different. And because of this available software that's out there, some of my
doctoral students I sort of give them the challenge to do, and they'll wonder off
and I -- you know, a couple of days later they'll come up with a mashup. And
what they've done is taken some existing Web services and fitted them together
and the gluing of the code in mashup programming is still a bit Archean. But
even now what you can do is program by Google. You just type in and say oh,
how do I convert the import from some Web service into a format that Google
Maps wants?
And if you're lucky, somebody has already written it and there's a bit of code you
can copy and paste in. If you're really lucky, it will work as is. More than likely it
needs a bit of tweaking to make it happen. But again, you're not building from
scratch. You're using some preexisting resources to assemble. And that is
really, really fast. So I'm claiming this software abundance out there. And we'll
still got constraints of time and money and certainly expertise. Mashup
programming is the most ugly code I have ever seen in my life. It's, you know,
really hard to do these conversions between one input process output to the next
input process output. But again, you've got programming by Google. You can
leverage off what some other people have done.
And so you can -- you can apply this sort of remix culture that we normally talk of
in terms of like, you know, music videos, but I think it equally applies to
innovative software development of just throwing things together that may worker
at least enough as a proof of concept and if it's a matter of -- if it's a sequence of
Web services it's now on the Web so I can test it outside the lab. And that's very
different from when I was a grad student and my flaky little prototype worked on
one Sun work station and I never managed to support it anywhere else, it was a
whole load of idiosyncrasies, so I had to bring people into the lab to test it.
Now, if these things are Web services, they're online so they are testable, and
you can also invite people either to take part in your experiment or say we've got
some swear if you'd like to use it feel free and, you know, some people will try it
out out of interest. But then if they can keep on using it, it must be meeting some
of their needs. So we now actually have a situated experiment. People are
choosing to use my application, not because I'm paying them to come into my lab
but because somehow they're getting some use out of it.
So we've now got authentic situated use which is normally very hard to design an
evaluation for. You kind of get it for free if you hack together some Web based
applications.
So here's an example of one. And this was a doctoral student. I was teaching a
class in rapid prototyping and evaluation. And he had this vision because he was
a computer scientist now in the library score and he'd noticed that he and many
other graduate students had this sort of problem of writing papers. And the more
diligent ones were quite happy reading, reading, reading lots more papers but
never quite got around to writing their stuff. And so here we're saying well,
wouldn't it be interesting if you could more tightly integrate the writing, searching,
and reading process as opposed to having them on separate applications.
And he built some paper prototypes that we were doing paper prototyping and it
sounded quite of nice. But none of us in the class really quite knew what he was
going on about. And then the next week was I going to challenge -- try and build
it and he sort of hacked together a little mashup. And again, I'm not going to try
and risk running a demo but on -- up here is word processor, so I think this
[inaudible] now Google docs.
So it's a Web based word processor. And then hopped it up to a number of
different Web services that try and do keyword and phrase extraction which
would be these keywords up at the top right, and then he passed those on to
various digital libraries as queries to produce some results that you see at the
bottom right.
So what happens is you go in, you brainstorm your ideas into the word
processor. The system is just extracting keywords and making suggestions of
things you might want to read on the right-hand side. And the first remarkable
thing which as soon as he hacked this together, everybody in my class, including
a group of computer scientists who really have no clue what he was on about
when he had done a paper prototyping suddenly got it. We suddenly realized
what it was that he was going on about. Because we had to see it working.
And it is a moment of revelation. Because I had said oh, yes, non-computer
scientists have real, really great difficulties in envisioning what a piece of
software, piece of innovative software would be because they can't kind of run it
in their heads because they're not computer scientists. And half the people in
that room are computer scientists. We couldn't envision it until we actually saw it
working. So that's sort of stage one.
And stage two was as we saw that, we realized that there's something weird
about these hack together things that sometimes it doesn't have to be good, it
just has to be good enough. So as you're typing in it's pulling out some
keywords. Often it pulse out completely useless keywords, so it's not that great
at doing it. And then when he does these searches in ACM digital library it
comes up with some suggested readings. And these belong in various
categories. One is the completely wrong and irrelevant category. The other is
highly relevant but I've already read it, so it's not doing me any good. And then
the very rare categories, oh, that's really interesting, I think I actually want to read
that. And that's a very small proportion.
So if it analyzed its suggestions from the perspective of traditional information
retrieval, you'd say this is rubbish, you know. I mean, we were better than that in
1960. You know, this is really not very good. But when we tried it out on people,
they really quite liked it. And as a tour to help them write their essays. And the
question is why. Partly it's ambient nature. You're not typing inquiries, you're
just writing your essay, and suggestions come up. So people seem to be very
tolerant about not very good suggestions most of the time if they don't have to do
much work.
And the fact that it made suggestions of things that they already knew about,
they didn't mind because that increased their trust in the system. It's like oh, he's
doing some things right it's just not very useful because I already know that. And
occasionally, just because it only occasionally pulls a rabbit out of a hat, they
think that's really exciting. So they're much more tolerant of it than if they had to
go to Google and type it in and get, you know, three relevant results out of 50.
The other interesting thing was because he'd done it as a mashup and now all of
us in the class could see what he was doing, many of you were saying oh, why
don't you use this other key extraction unit? Why don't you use this other word
process? Why don't you link it up to this other digital library? Why don't you, you
know, have things where you can save things and throw away -- so we had a
whole lot of suggestions. And it was quite remarkable to me that every other
time that I've been advising a CS student who has gone off and built a really
quite interesting piece of code and then I come back and say why don't you
change it, they're very, very resistant like I have invested 50 weeks of my life in
this algorithm and now you're saying people can't use it? People are stupid. You
know, they must be reconfigured. You know, my algorithm is right because I've
worked so hard on it. And I'll tweak my algorithm. But there's no way I'm going
to throw way my algorithm and start again just because you've told me, you
know, you've evaluated it, it didn't work.
And we're saying oh, yeah, yeah, that algorithm isn't very nice. I'll just rip it out
and put in another one because it was just a Web service and they didn't care.
And so we would rip out one algorithm, keyword extraction, plop in another that
we suggested, find out that was actually worse than the one we had before, take
it out, put it back in again, and we didn't care because it was not emotional
invested in this code. You know, he was just assembling the Lego bricks, he
wasn't hand crafting the Lego bricks.
So that was really kind of intriguing to me is how this prototyping culture can lead
to a letting go of your code as opposed to sort of the enormous emotional
investment you get with it. So that was one example of very rapid prototyping.
Here's another example of the other end of it is if we can now get to the stage
where we can build in the morning can we test in the afternoon? Now the
moment we -- I think, within computer science we're getting better and better and
better at developing rapid prototyping techniques. So we can assemble these
things, we can build really fast, but if we're talking about sociotechnical systems
we need to build and then evaluate. And if it's build them in the morning, test in
over three months, build in the morning, test over three months, that's not really
the extreme programming cycle, the build, run the automated test, rebuild, run
the automated test.
So how might we speed up the evaluation process? Now, we really clearly we're
going to have to give things up. We're not going to be able to do the kind of
rigorous evaluation that you do when you test over three months. But I think it's
worth asking provocative questions. And a big argument that's been running for
many, many years in human computer interaction is how big an N do you need
for doing a user test? [inaudible] those have been advocating for various sort of
pragmatic reasons that actually test one more person you find more and more
usability bugs. But the code starts to flatten surprisingly at five. And there's a
huge fuss and bother and people say oh, no, five's far too low, argue, argue,
argue, back and forth there's lots of tests, lots of careful analysis of it.
And I'd like to ask a different question, what can you get out of the user test with
a sample size of one? And don't be definitive, but is it enough to get you to the
next stage? And now we're down to that. How even more extreme can we do it?
Can we do a user test where we just do 10 minutes preparation? Can we do a
user test that only lasts 10 minutes? Can we do a user test that takes 10
minutes of analysis and then go back and do more iterations, more redesigns?
What are the things you can measure this way, and what are the things you will
totally fail to measure that just wouldn't crop up in those circumstances?
And I think it's interesting to explore how you might design different kinds of
extreme methods like this. So I'm not saying oh, I've invented this method and
everybody should adopt it. I'm saying I found a point in the design space of
evaluation methods and everybody else's methods are way over here and I seem
to be here on my own and it's not like I've got the right one. But there's a whole
space here. We don't even talk about it. How might we design evaluations to get
what we need out of it so it's rigorous enough.
And this partly came out of some outreach worker I was doing at the conference
museums in the Web. So this is with a load of culture heritage professional for
her [inaudible] built testing is totally alien and so we were running things like
workshops and tutorials on how to do user testing. And we did some public user
tests which we sort of ran as a game show, somebody come up on to the stage
and we do a user test in front of them and try and encourage the audience not to
shout out no, no, click on that button there.
But the point of this was to illustrate people who had never seen user testing that
it was not a mystical process and they should go home and do regular rigorous
user testing.
However, what we found was to the best of our knowledge in validating with
people whose site it was, the results we were finding from a 10 minute user test
they were taking furious notes, oh, we're going to change that when we get back
home. So we had these lots, we had an hour -- half hour slot, we'd have a -- take
a volunteer outside the room who hadn't used the site before, do 10 minutes of
analysis, 10 minute user test and 10 minutes of what we see in the user test and
design recommendations and on to the next slide. And that's to say 10 minutes
of user tests every 30 minute whole slot we would get results that people have
said were useful.
So that made me realize that, you know, the technique of use testing can stand
an awful lot of abuse, you know. This is not science. I don't want to pretend it's
science. But there's things you can do if you do this carefully that you can still
get some useful results out of. So I've decided now that, you know, the trick is to
find what are the big changes and so I'm not interested in these sort of tiny
percentage changes because then you have to use the rigorous methods. And I
think these crude methods worker -- this is the low hanging fruit, the really huge
changes in how to go about doing that.
And so here are just some other techniques that I have been playing around with
to help do this kind of analysis when you don't have a lot of people, you don't
have a lot of time but you can look closely at things, and I think there's a lot to be
done by trying to do something like affordance analysis where you look at an
interface and say well what's it likely to be good at, what's it likely to be bad at,
what kinds of features is it likely to enable? Let's stick somebody in front of it,
see what we see. Let's be very careful at notifying what we see that is surprising
to us. What are the things we expected, and can we triangulate this with
something that somebody else has mentioned in passing in the literature? And if
we've got a very tiny N let's concentrate on within subject variations. How come
he found that there really kind of easy and this bit really kind of hard? It's the
same person. And particularly when that is surprisingly easy and that bit is
surprisingly hard can we see what's in there that will give us some clue about
changing the design? And one of the reasons that I've got into open source
usability is just because it's open that you get an awful lot of these externalized
discussions of the design processes and -- but also in the tech support forums
where people just sort of talking about what they're finding confusing. So in a
sense they've done part of the user testing for me. So I've got a large data set
out there to look at.
So one last example to illustrate this kind of approach we're coining this word
patchwork prototyping to say that, you know, you can do rapid prototyping and
build from scratch or you can do it like building a patchwork quilt where you get
existing bits of stuff and you stick it together. And those existing bits of stuff
might be commercial off the shelf or it might be Web services or it might be open
source or it might be any combination of those and can you put them together
sufficiently quickly that when you stick them in front of somebody and say that's
not what I want, I want something else, you can change it very, very quickly?
So here's one setting where we were trying to do it and part of a larger project
which is looking at a whole lot of different projects that are digitizing cultural
heritage artifacts in museums and archives and there's 800 of these collections,
and our job is to use the OAI metadata harvesting to suck all of this stuff into our
central repository. This was funded by the IMLS who funded all these separate
800 projects and they basically wanted a one-stop shopping for all their projects.
And now you've got that. You have to say well, why would we -- why were they
doing all this digitizing in the first place. It's often to help humanity scholars do
the things that humanity scholars do. So they won't access the primary materials
digitized stuff works very well. But how do you help the humanity scholars to get
a sense of what you've got. If you're a scholar and you walk into a library or you
walk into a museum or indeed you walk into an archive they're very good at
getting a sense of what's there just by wandering around. The physicality of the
place and the physicality of the layout gives them a sense of what's there, what
might be interesting, or what's surprising. And they're very good at looking for
things that to them are anomalous or interesting, they want to delve into more
depth.
And I think we have designed the equivalent of the closed stack library. So the
open stack library you walk in, you run the shelves and then you serendipitously
find things you're interested in. A closed stack library a little cubbyhole and
there's a librarian on the other side and you say I would like this rare book and
the librarian scuttles off, brings it back to you and says here it is and you say
thank you very much and you read it.
So that's fine if you know exactly what it is you want. But you can't go to that little
cubbyhole and say what have you got in and they're going to say oh, 20 million
artifacts. You say can I see them and they say no, just tell me what you want
and I'll go get it for you.
And I would claim that what we had done in this project is done that, but instead
of the little cubbyhole with the librarian in this side we have a little search box.
We have an algorithm on the other side which is fine if you know exactly what
you want. But you can't type into that little box what have you got?
So we are trying to say how might you draw some pictures that? So I think what
you need is a data visualization. It needs to be low cost because we don't got
any money to do anything better, but much worse many of these humanity
scholars are very good at texts but they really don't -- they're not used to
diagrams. You can't say to them oh, what picture would you like of the data,
what data visualization would you like the [inaudible] before, pie chart?
Witchcraft, you know. So they're not used to this.
And it's an interesting design challenge. So what we try to do was go to a
museums in the Web conference where there were humanity scholars who had
some sort of techie background and try them out with a dashboard, here is a
paper prototyping one, of various kinds of visualizations that might give you a
sense of what is there in a collection and then we sort of produce this using I
hope you can recognize many little Web services using Google Maps, using
word, all of those are publically available resources and we're just plopping in the
data.
And then the Indianapolis Museum of Art had assembled these resources which
was more for management stuff into this dashboard. So we're taking the
metaphor of a dashboard which is being used in sort of commerce, and we're
trying to do it to help say to answer the question what is it that you've got? And
again, each of these visualizations is very cheap. They don't take much effort to
do. None of them is particularly good, but if you've got enough of them all
together, maybe they can -- maybe they're good enough. And again, this
intrigues me. It's a different approach to doing information visualization than our
colleagues up the road, the National Center for Supercomputing Applications.
When they do data visualization it's a tornado costs 10 million dollars and it's
gorgeous. When I do visualization it's like I slap something into Google docs,
takes 15 minutes, I stick it in front of a humanity scholar and say that's kind of
nice but couldn't you make it do this instead?
So it's that sort of low road of little tweaks and changes along the way. Here's
another one. This is probably the most ugly visualization I've ever been involved
with. This was an undergraduate project at the -- involving the green stem digital
library developed in [inaudible] New Zealand. These are fills in the metadata for
doubling core. And basically you scroll down a set and you say there's a blue
blob if there's a value in that field for that record and others there's not. And you
scroll down and you can look for anonymous shapes in there. So if this is your
data set, you will notice weirdnesses or not weirdnesses in here and that can be
a very quickly way of getting a sense of maybe while harvesting was wrong.
Maybe there's some errors when we harvest it and maybe there's some errors in
there. Do you see huge loads of blank spaces or do you see these weird anti
phases? Is there always the case that that one or that column but never the 2?
Scroll down, get a sense of it. Then you can ask the expert is that normal? Is
that what you want? Or is there something wrong with it?
So a very crude visualization, very quick and easy to develop but is useful
enough for the experts to start doing things with it. So I shall end there and
entertain questions. And I've got many more slides to hopefully illustrate any of
the questions you may have and I'd be really interested if you can think of other
metaphors of extremeness or agility that may work in the context of doing
research.
So thanks very much.
[applause].
>> Michael Twidale: All right.
>>: As part of your rants you talked about the waterfall method of paper writing
and how ->> Michael Twidale: Yes.
>>: That's sort of what's accepted in academic circles. And so I was wondering
how you found to be trying to publish research using these kind of approaches.
Are people receptive to that? Do you actually write it up differently? Do you
write it up in a way that reflects the style more and [inaudible]?
>> Michael Twidale: I have tried to. But I have to say I'm on sabbatical this year.
One of my big jobs in the sabbatical is to write the papers to justify these
methods.
Because of course what happens is I write a paper and again if it's a CHI paper,
you've only got 10 pages. So you've got to spend so much time explaining what
the problem is in the first place. And then maybe how you've build it and then
something about how you evaluate it. And I've got like two paragraphs to explain
why I'm doing it this way as opposed to the way that all the reviewers thing I
should be doing a controlled experiment. And there's never enough room to
squeeze it in to a regular paper.
So I think part of my job is to right another paper to justify these methods. So I
tried to, but I've had problems in squishing it in.
>>: Well, then I guess a [inaudible] is if approaches to do extreme research is
there an equivalent extreme publishing.
>> Michael Twidale: Yes.
>>: Publishing everything via tweets or ->> Michael Twidale: Yes. Yes. No, but again I want to try blogging as one way
of doing this. I'm not sure what -- I think I'm too old for tweeting. Still find
blogging a bit kind of noble. But I think one way is to try these alternative
publications because, you know, one of the trends of eScience is to share more
and more of the stuff. So the idea is you share your dataset and maybe you'll
share your dataset analysis programs that you did or the macro you did to XL so
somebody can replicate what it is you've done. And I think it's incumbent on on
me to say I've invented this method, here's what it is, here's why I think it's good
and here's why you can try it at home too.
>>: Because you can imagine [inaudible] just to build on that, part of the benefit
of extreme programming is that one person sort of points out errors to the other.
You can imagine that you might get more benefit even out of your 10 minute user
study sessions if instantaneously after the session was over you do the
one-minute write-up.
>> Michael Twidale: Yes. Yes.
>>: What you learned and then you can get feedback from other people about
do they agree with your interpretation of the data, that might help you then refine
your ->> Michael Twidale: That is really interesting because of course what we've
done on this is we write up for our benefit immediately after. And I found it very
useful to do [inaudible] so if there's two of us observing the user study you get a
lot out of it. But you're saying you could broaden it out even more. Yeah. I
hadn't thought about that. That would be really good.
>>: What if you [inaudible] these of the extreme approaches on depth of
[inaudible].
>> Michael Twidale: Yeah.
>>: [inaudible] part of the thing you're getting with grad students is trying to teach
them how to ask a good research question?
>> Michael Twidale: Yes.
>>: And presumably how to design a good visualization? And if the
methodology is just throw a billion things out there and eventually you'll figure out
one of them that's successful and you'll learn what the attribute was, it seems like
it's almost a waste of their time with lots of wasted end user's time to look at all
that stuff if it could have been done with one more thoughtful experiment for this
graduate student to have tried.
>> Michael Twidale: Yeah. I think this is a very fair question and one I struggle
with. So the way I would address it is I think a lot can be done by a little bit more
thought about what you're going to do going in in advance. And I see this
problem both with when I'm teaching design to computer scientists the risk is,
you know, [inaudible] problem, they come up with an idea and then they want to
go and implement it. And I say okay, slow down. Let's thing of six other ideas
before we decide which one we're going to pick because, you know, ideas are
not so precious that as soon as the lightbulb goes off you've got to run off an
code it. So in that sense I want to slow them down but I want to say look, I'm not
saying we're going to spend months on that, I'm just saying let's just spend an
hour or 15 minutes charting out the design space.
I think there's something similar you could do with just a few minutes charting out
sort of the analysis space. And I think that's part of the sort of the sealed
envelope heuristic that I'm advocating is something I don't see as done. It
shouldn't take too long to do, but I think it can force some reflection. So, yeah, I
really do want to have my cake and eat it too. I want to be able to have these
fast methods encouraging reflection at the same time. And I do recognize that
students could misinterpret this as being oh, you could just do any old crap and
it's like Darwinian evolution. Just like breed lots of them and one amazing thing
will pop out.
>>: [inaudible] computer surveying, you know, on a million different [inaudible]
you could try and [inaudible] variations on the search page to see which one
might have an effect.
>> Michael Twidale: Yeah. Yeah.
>>: Wait for the [inaudible] to come in and tell you how you should have been
thinking about it.
>> Michael Twidale: Yes.
>>: The same analogy exists in extreme programming, agile programming.
People argue and they're like oh, you don't do any design. It's not true at all. It's
just ->> Michael Twidale: Yeah.
>>: So we've been actually doing -- using methodology [inaudible] and so what
we've done is we've [inaudible] and you know, you'll meet with our program
manager if you evaluate the producer [inaudible] appropriate for this kind of
methodology. And at that point make the call as to whether actually to use the
agile session [inaudible].
>> Michael Twidale: So how do you decide between the two? What's -- what
kinds of problems do agile extreme methods seem to be particularly good for?
>>: So I work in enterprise software, you know, so the problem space
[inaudible].
>> Michael Twidale: Yeah.
>>: So generally the problems that tend to fall better into the more agile space
are the smaller well constrained problems.
>> Michael Twidale: Yeah.
>>: And if they're big and ugly questions, then we need to do, you know, more
involved technology. You know. And so kind of, you know, bring them together
a little bit, you know. It may be that, you know, a tweaking UI and agile session
I'll also add in, you know, some qualitative questions and so forth to, you know,
provide, you know, rough data gathering that will beat us into [inaudible].
>> Michael Twidale: Yeah. That is interesting because I often thing you know
sociotechnical systems are a bit messy, horrible spaces but I think my job as a
researcher is to try to break those down into more manageable bite size research
questions. However.
>>: [inaudible].
>> Michael Twidale: Yeah.
>>: Agile method.
>> Michael Twidale: Yes.
>>: But at the same time you're feeding that data into the larger dataset.
>> Michael Twidale: Yes. And I think by -- the question is when you can have
the iterations enable you to be rigorous. But also I think that there may be
methods of how you break down things into bite sized chunks. And the memo I
see those -- you know, it's weird -- when we talk about research methods in
university, all we ever talk about is evaluation. Now, research methods means
evaluation methods. It doesn't mean how do you scope a research question,
how do you figure out that there is a research space that is strongly analogous to
a design space? And, you know, gradually we're starting to realize we continue
to design by talking about concepts in design space and how you just -- how you
cope with that. But we don't do it in research space.
And I see part of what I want to work with is look at the methods of how people
carve these problems into manageable units. So I suspect that is articulable and
teachable. But at the moment you know, it implies you're just good at it, just
have a flash of iteration, that's what makes me good or why should I tell you how
to do that? That will be giving away the secret sauce.
And which is really ironic seeing the whole point of science is to share
techniques. But we don't want to seem to want to share that one. And, you
know, it's hard enough to do, as I say, teaching design to computer scientists
because we're not really used to it and my experience learning computer science
I was never taught design.
And if I was lucky, they created opportunities where I could figure it out for
myself, you know, they constrain it so I would have a good learning [inaudible] if I
actually taught you how to do design. I don't know that we actually teach how to
scope out research problems, we just teach. Now you've got, magically got the
research question which method to choose. We don't actually teach how to
design methods much. Yeah?
>>: [inaudible] talk about [inaudible] research question. You haven't talked at all,
at least in this talk, about how do you figure out a research question when you
have no idea what you could be looking for.
>> Michael Twidale: No, so I haven't talked about it in this talk, but I do -- yeah.
Yeah. So I think there's a pretalk or talk temporally how do you scope out these
kinds of research questions. And I think that can be done using relatively quick
methods as opposed to just sitting staring at it. But, you know, I'm not sure I can
make those up on the fly here. I mean, I think it's ->>: What you're implying is that you start with one and you enter it on that and
then you may discover that you're asking the wrong question.
>> Michael Twidale: Yeah. So that is certainly one part of it. But I think there is
something about trying to scope out a research space which is where you try and
list a set of the areas in it and then you may use reasoning by analogy from other
domains to say okay there might be some areas in here. And so I think you can
keep on pointing to this as if it was a giant white board, so you could write down
some of the issues in the area and then say okay, we're now going to hook in on
that.
But I -- I agree -- firstly I agree, I haven't talked about it here, and secondly I don't
think that's talked about much anywhere as a technique to figure out, you know,
where you are in the larger space of research.
>>: [inaudible] and from also where you talked about there's a difference
between communicating an idea and telling somebody else exact how you did it.
>> Michael Twidale: Yes.
>>: In order to teach them how to do it?
>> Michael Twidale: Yes.
>>: And then all the papers and things that people are writing about [inaudible]
all about just communicating an idea so that somebody can understand it as
opposed to here's how [inaudible] you also see that when you look at [inaudible]
and they're doing their dissertation talk or their job talk, they usually distill down
their entire research area to a graph with two axes, and they say all the other
research fits on these points, these most important ones and mines up here at
the upper right because that's exactly where you want to be of two difficult acts.
And it seems that people don't really get to the point where they can
communicate their ideas clearly until they have a really good understanding of
the research questions and the research space. And then at that point, then they
could start asking what are the real research questions I should have been doing
instead of wasting my time with four years on ->> Michael Twidale: Yeah. Yeah. Yes. So I suppose, you know, one sense I
want to speed that up a bit, you know, so you can make various mistakes earlier.
And I think certainly there are ways of doing microstudies just to try things out.
I'm sure it's the case that you only -- it's only at the end when you really
understand it and you can say okay, this is where I fit. But it's almost like you
need two documents. There's the -- here's the Reader's Digest condensed
version. I've hacked my way through the jungle and now it's really easy for you
and I can tell you how to get from A to B in the most direct way possible so it will
save you time if all you care about is getting to B.
But there's another story about how I actually really got from A to B, which would
be very useful if you want to get from C to D. Because I can then show you the
technique of how I coped in the hacking process through the jungle and how I
dealt with all the twists and turns, and that's very different -- it's different but
you're talking to people who want to get from C to D or the people who want to
get from A to B because you've put it through, and that's where the nice tidied up
research paper throws away all the dead ends. But it doesn't help anybody who
is going to be doing new research who needs to know how to cope with dead
ends and maybe recognize the dead end faster than I did when I was going
through.
>>: I think on that topic there is a distinction between the research paper and the
research notebook.
>> Michael Twidale: Yes.
>>: And we just don't have a tradition of sharing out our research notebooks
except for Da Vinci's, which are the only one that really gets shared out and
people adore that one.
>> Michael Twidale: Yes.
>>: So maybe we just need a better tradition of sharing out our research
notebooks and of learning how to keep them better.
>> Michael Twidale: Yeah. I mean, I think that -- yeah.
>>: That is analogous to [inaudible] to the end user.
>>: [inaudible].
>>: Well, I think even so -- I mean, even that analogy if you run an open source
program and your whole revision history is mercurial and everybody is checking it
out every time they check out your source, it's still way too much information to
wade through, so really I mean you're sharing it in the sense that it's there and it
exists and if you wanted to you could bisect to find bugs or what not, but it's not
there in a way that we would like it to be there. It's not there as you know the tale
of how I got from A to B, it's there from, you know, here's a ridiculous amount of
data and if you were to spent your lifetime digging through it, you could maybe
figure out how I got from A to B.
>>: [inaudible] some interesting [inaudible] to be done.
>>: I'm not [inaudible] how to understand how this could apply to research in
mathematics or [inaudible] it feels like the launch part of this is good enough, is
good enough and I expect earlier to end users but there aren't really end users in
[inaudible] good enough as in good enough. I mean, what we've been talking
now about sharing the process of research for extreme publication might help,
but even that's hard to see.
>> Michael Twidale: Yeah. I mean I think a lot of this is inspired by the
messiness of sociotechnical systems where you don't even know what the
variables are you should be worrying about in the first place, so there's this fear
of asking the wrong questions. And we know from computer supported
cooperative work that a lot has been learned from ethnographic studies going in,
pointing out whole categories of things you didn't even know you should be
worrying about in the first place. So it's inspired by that, but also acknowledging
that I think [inaudible] is incredibly slow, so is there a way of speeding that up?
So it may be that within certain aspects of math an theoretical psychics it's
constrained enough that you can black box around, say, okay, this is a really
hard problem but I know everything is inside this box is everything I need to care
about and everything outside this box I can safely ignore. And if that is indeed
the case, then I don't suppose any of these methods -- because my methods are
to do without leaking in and out of the box of what's going on. But you might
want to look at the processes used by say a mathematician so it's more like the
[inaudible] kind of argument, how do mathematicians get to an understanding
and how do they develop proofs?
So if you're interested in the process of proof development and you want to study
that -- I mean nobody wants to [inaudible] in order to teach other people how to
do it then these methods analogous to this might be helpful to say well, you
know, what I'd really like to do is open up your brain and look inside but you
know, I can't do that. Or I could just keep on interrupting saying why are you
doing that, why are you doing that, why are you doing that? But then you taught
me how to do it.
So for those kinds of things it would. But now you see what I've done is sneakily
brought the human being back inside the box. And these -- I think these
methods always work around human beings. So when I'm studying the
mathematician, I think they worker. When you're studying the math they may not
work.
>>: I think that's the style of research, ethnology, so [inaudible].
>> Michael Twidale: As it currently stands one big fear I have is it will only look
at short-term things. So if you're just doing a user study in 10 minutes, you can
only study the things that people do in 10 minutes. So that's a risk. I think that
one can be mitigated if you are doing a whole series of 10 minute studies over a
longer period of time so you can aggregate out of it.
>>: [inaudible].
>>: [inaudible].
>> Michael Twidale: This is the interesting thing. There's many ways in which I
would want to study the same person over a very longer period of time. I think
there's a lot that we can get out of that. And it's just I want -- you know, other
people are already doing large end studies. So I think I want to find -- to ask the
question what can you get out of a small end?
And it's clearly you lose -- what you lose by small end, but what you can gain is
some degree of depth of, you know, really getting to know that person and if it's
over time how is their understanding evolving? And then you want to somehow
triangulate and say well, was that person just mad, are they unrepresented, are
they unlikable human and are there -- you know, are they like everybody that's
also unlikable or are there certain categories?
Now, how many more people do I have to study before I'm [inaudible] the same
dumb thing time and again? So I think my bias is always to depth rather than
breadth, and I had think you'll just sort of kicking against the problems that you
have with large end. You have to ask very, very focused questions and you may
lose something.
But I think there may be other things I'm losing, too. And I think that's just
something I want to go away and think about more.
>>: [inaudible] as something we see here because the more people you have
and potentially the more times you see the same person the bigger the
researcher scheduling problem is. Because now you actually have to put them
on your calendar somewhere.
>>: Yeah.
>>: And [inaudible] the larger amount of time just makes space for this
experiment. If a particular target population [inaudible].
>> Michael Twidale: So larger scale collaborative interactions are just harder to
study?
>>: Yes.
>> Michael Twidale: Because either you have to try to invite them all into the
same place at the same time or you go to where they are. But even there you've
got to be there at the right place at the right time to [inaudible].
>>: [inaudible] visualizations and potentially really fast trials to much longer
potentially issues about the mechanics of scheduling and the mechanics of
getting people in the same room, mechanics of running the study which are, you
know, from my experience, those are the parts that suck as opposed to actually
once they have started watching them use it and figuring out what's going on.
>> Michael Twidale: So I have one trick that we addressed that problem, one
particular case where we wanted to talk to lots of people with a background in
cultural heritage and [inaudible] technology so we took the photos of everybody
doing the design was done at a conference. So rather than, you know, going to a
conference to present the results, we went to the conference to do data
collection. And so that's like well we went there because that's where all the
historians are. So this was like the demo session.
So everybody else, everybody else's table was here's this neat computer
application we've built, isn't it good, do you want to use it? And ours is like we
haven't built it yet, tell us what you want. But because it was in the demo setting
everybody knew the game was you wound it up, you stood around on the
periphery. If you were interested you gradually come closer and closer in. You
can stay as long as you like and then you wander off to the next demo session.
So we're playing off the demo genre and which allows people to understand what
they're letting themselves in for, which in many ways sidesteps a lot of the
problems of recruiting subjects for user studies like what am I getting into, it's like
lab coats and electrodes and how long is it going to take?
And this is like ooh, you know, so there were several people who hovered on the
outside and maybe shout out a few questions [inaudible] involved but then they
still stay there and shout out a few more and then gradual will come in. So that
was one way of sidestepping this whole recruitment problem. Because having to
go to these people's home offices all the way across the understand would have
been infeasible.
But again, this is not going to be universal solution. I think we have to invent
ways of getting you know, access to it.
>>: Does your IRB have to be consulted for every single instance have every
piece, every design that you make?
>> Michael Twidale: That's coming. That's the IRB paper coming soon. Yes.
This is the big problem that IRB hasn't caught up with the invention of new
methods. We did have to get IRB permission for this. And of course people say
you had to get IRB to go to a conference? Yes. Because we're doing data
collection. But, you know, I said to the IRB well, we can't make people sign
forms because the whole point is they're going to give -- that, you know, that
woman standing in the back may be shouting out a suggestion. She hasn't even
got close enough to pick up a form yet. But they were quite comfortable with us
posting an informed consent from, you know, just behind where I'm standing. So
that was okay. But it -- yes, it does lead to these sort of bizarre things like what
even counts as data collection in these settings. Yes?
>>: Have you thought about how the sample could change over time depending
on whether you're in the [inaudible] there might be different samples [inaudible]
different parts of the lifecycle of the research?
>> Michael Twidale: Yeah. Yeah. Yes, I have. And I think -- I'm trying to figure
out how to argue this one. Early on you can -- if you discount carefully you can
make do with unrepresentative uses. So for example, one way we're doing user
testing of museum websites we were doing this at museums in the Web
conference. So we had volunteers who had never used the website before. But
they were not like the average museum visitor. These were people who paid
money to go to a conference on museums in the Web.
So they were museum professionals and they were sufficiently techie to go to the
museum in the Web conference. But if you can make the a fortiori argument
that I'm testing this museum website, I've picked some representative path, I
asked this person who is a museum informatics professional who has never seen
this website before to say you know, imagine you are coming to this museum for
the first time and you're going to be on holiday and you want to know where is it,
what its opening times and how much is it going to cost for a family of four and
they can't do that. Then how much more so is it going to be difficult for people
who have less skills?
So that argument doesn't work if you -- if you're going to say this person is finding
it more hard, more difficult than the average person will be. But I'm saying that
for those these were unrepresentative because they had a whole lot of skills and
everything that they found hard is interesting data that probably ought to be fixed
because your actual intended users will find it at least as hard. But everything
they find easy you can't say whether it's good or bad.
So you can only focus on the negative evidence. But that's okay because that's
only what you do in formative evaluation. And so if I had a limited set of
volunteers and if I was able to outrank it so that I'd test out on the more skilled
experienced people first, to catch, you know, some of the more blatant errors and
then work my way down to the really intended users, the more novice ones. But
there are times when you may want to test some of the more advanced features
and those that say well the absolutely beginners, they wouldn't be using those.
But I think it is a way of when you've got a very precious set of users who are you
going to pick first and where can you get away with? But it requires a certain art
of discounting to know what are you seeing, what is the background of the
person you're seeing it with, and also is there any evidence that this is being
seen elsewhere? And so, you know, in our very tiny samples here we'd find
things like people would just get impatient and go off and do something else. Or
they say well I've got a -- I would have gone and done this if I wasn't sitting here
doing this user study.
And there's quite a lot of studies who have also found this degree of impatience.
So you can say even though I've seen this in that in this one person or two
people, I think it's fairly wide spread because there's evidence of that kind of
impatience popping up. And notice that a lot of the people in that kind of
audience will say oh, yes, young people are impatient. The Nintendo generation,
they're impatient. And now we've just seen it on a 40 year old museum
professional, she got board with your site. So it's not like the 16 year old that's
going to be any less bored. So again that's that kind of discounting. And it would
be very difficult to argue that in a paper. And again it would be a whole load of
stuff to argue in the paper but I think in terms of making pragmatic decisions of
what needs fixing and I find five things and which things you want to prioritize it
might be enough to make the decision of where to allocate the resources. Well,
thank you very much, everybody. If you have any ideas of extreme methods that
might work in research and this has triggered any thoughts, please send me an
e-mail. Thank you.
[applause]