16829.
>> Andy Wilson: Thanks everybody for coming. It's my pleasure to be hosting Bjoern Hartmann
from Stanford. Bjoern has done a lot of really interesting things in ETI, as many of you know.
And he is finishing up his Ph.D. work right now and he'll be presenting some of that work.
We've had the privilege of working with him last spring. He was an intern here, and we worked
with Mary Morris, myself and Banco on a large format surface, which is actually still running
upstairs. We're still doing some work on that, and I know some of you here have seen that. If
you haven't, I'll happily give you a demo of that system.
I think one of the interesting things about Bjoern is his approach to ATI and it's his looking at tools
for designers and various strategies for prototyping and making tools for prototyping. And I think
it's some things that resonate with some of the activities that are going around on campus.
So Bjoern himself has done a lot of, as I mentioned, a lot of interesting work in ATI. He's
garnered several best paper awards at West and ACI. And he's editor of Ambidextrous Journal
at Stanford University. And reading through his CV is really interesting because he's worked all
over the world. I think he's had jobs in France and Netherlands and Italy and been involved with
a record company and all sorts of interesting things.
He has a lot of different experiences for a computer scientist. Thank you. He's going to be
talking about enlightened trial and error. I'll give it over to Bjoern.
>> Bjoern Hartmann: Thank you for the introduction. I have to warn you ahead of time, if I'm not
quite my bubbly self today, it is because I caught a cold about two days ago on the East Coast.
But, anyway, I'm very glad to be here, excited to be here. Great time in the three months that I
spent here last year. And today I want to talk to you about my dissertation research, which is
really concerned with new prototyping tools for the design of physically embodied user interfaces.
Now, in particular I will introduce systems in my talk today that address three important concerns
for prototyping new user interfaces. First, how to enable a larger group of designers to rapidly
create functional ubiquitous computing prototypes.
Second, demonstrate how to support iteration in the design process by integrating test feedback
from testing and prototype right back into the design tool.
And, third, I'll introduce ways to -- oops -- enable simultaneous exploration of multiple design
alternatives in parallel. But maybe let's first take a step back and ask the question: Why focus on
prototypes?
Well, if you believe practicing designers and people who study design, then they will tell you that
prototypes are central to good design in any domain, and that prototyping is the pivotal activity
that structures innovation and collaboration, whether you're designing furniture or medical
devices or user interfaces.
And the goal in prototyping is not the artifact, per se. It's feedback and iteration. Prototypes are
cognitive artifacts that enable exploration and iteration.
Let's look at exploration and iteration in a little bit more detail. This diagram may be familiar to
you since it came from Bill Buxton's book. So any sufficiently complex problem has a space of
possible solutions.
To map that space, designers frequently generate multiple possible solutions to a problem and
they do that simultaneously. And afterwards they then select either desirable candidates or
desirable features from different candidates to move forward.
Now, iterative design just acknowledges that finding a fit between a problem space and a solution
space can only be achieved through repeated testing and improvement if what you're concerned
with is the user experience.
Let me give you two examples. Up here you see Paul Bradley, industrial designer at IDEO, and
when they were tasked with designing an early version of the Microsoft mouse, he rapidly built
about 80 different foam models to just explore the space of possible shapes.
This is Liza Richert in the UK works for Buchard Design. Recently worked on the open source
content management system. They have about 300,000 users, and the way they went about that
was to post even early wire frames to the community of existing users, gather factor one week,
and then incorporate that feedback into the next iteration, and then wire frames turned into more
functional prototypes and into the next iteration and the next and the next.
Now, given that exploration and iteration are core activities in design, it's somewhat exciting the
tools neither of these two activities are well supported. The focus of today's tools are really on
getting a single complete polished robust artifact built.
And, of course, that's important, but it makes the wrong set of trade-offs in earlier stages of the
design process.
So let's see what I propose we can do about that. So, first, I'm going to talk to you about how to
enable more rapid offering of prototypes in the realm of off-the-desktop user interfaces. So
information appliances, smart products, ubiquitous computing interfaces, and think of rapid
authoring as really the baseline of support, without which none of the other work can happen.
If you can't build a single prototype quickly enough, you're unlikely to explore multiple different
alternatives in parallel. You're unlikely to go through the iterative cycle a number of times
because it just takes too many sources to get started in the first place.
Now, as a good user-centered designer, of course you have to go out into the world and learn
about your user. So, for example, I served as a technical consultant at a Bay Area product
design company for the design of the physical user interface for a medical surgery device and
that just allowed me to go into the company and get kind of firsthand data of how those design
teams work.
I just want to highlight one of the findings from this initial inquiry. And that is that professional
design companies actually do have access to resources and the expertise required to build, well,
almost anything.
However, that expertise is nonuniformly distributed. So at many design companies, for example,
there will be one electrical engineer on staff. And that one person now is a bottleneck for all
projects that will have some kind of sensing and actuation component in them.
What that means is that shiny functional prototypes are getting built but oftentimes, because of
those resource constraints, only as kind of models that you show off at trade shows to wow
potential clients, not as tools for design thinking itself, for reasoning through the problem space.
Now, to address that opportunity, I, together with collaborators, built D Tools, which is a hardware
and software toolkit that leverages the storyboard paradigm familiar to many designers. It's the
first toolkit that designs the behavior of what your prototypes should do together with the design
of what kind of IO components are there, how many buttons are there, what kind of sensing can
your device do.
And let me give you an example to make clear what level of fidelity details targets. This is an
example taken from a local university. The mobile text entry problem.
In the absence of a touch screen, if you have less buttons than letters in the alphabet you'll need
some kind of multi-step disambiguation process to select maybe some subset and select a letter
from that subset.
So people at UW proposed you take that small number of buttons and combine it with tilled
information that you may extract from an accelerometer. That's becoming ubiquitous in our
mobile devices. The larger question is how well does such an input technique work?
And this is a question that I submit that instead of going through quite involved step of spinning
out a custom circuit board and fabricating the hardware, really you want to be able to answer in
about an afternoon.
So here is a prototype of tilt based technique we built using our own toolkit. It's a little larger, but
there's some trade-offs you make in prototyping.
Let me just show you. This is an action. We were able to prototype the published technique of
first using buttons to select the letter subset followed by tilt to select a letter within that subset.
Now, more importantly details was rapid enough to ask what happened if you reverse the
structure so you first use tilt and then buttons to disambiguate. Turns out that because tilting is
more error prone, you have less feedback. The second technique actually outperforms the first
technique. However, this second technique was not what was published. So my larger argument
here is having a rapid enough tool supported the serendipitous discovery of better spots in the
design space that just thinking hard about the problem itself sometimes does not get you to.
All right. Well, how do we build such a prototype in D Tools? Designers begin by plugging in
physical components such as sensors and actuators into the D Tools interface. As they do that
details counterparts appear. The designer is laying out an input what input and output exists.
With that blueprint in place, you can then offer interactions through visual states which
encapsulate output and link them with transitions which get triggered by input events.
And any such author interaction can be tried out in real time or it can be simulated if, for example,
you're sitting in an airport and you don't have your hardware with you. Now, of course, the
complexity of what you can author in any visual scheme is limited.
And so to break through that complexity barrier our toolkit offers methods to augment any visual
state with procedural code. Just to look at the user interface here of D Tools, again in the lower
left-hand corner you have a graphical user interface, GUI editor. We know how to build those, yet
they still need to be part of kind of a larger set.
What's novel is really on the upper left-hand corner that you have this virtual representation of
what the physical thing is you are building. And I'd argue that having this representation serves
as a bridge to reduce the cognitive friction of focusing on one type of work that happens on your
screen where you author the behavior and another kind of work when you switch over to your
desk and you have a bunch of hardware in your hands. To make this plug-and-play architecture
interaction feasible that I showed you, we built our own hardware interface that's built based on
networking, smart components on a shared bus. So here every button, every LED and sensor
has a tiny, cheap micro controller underneath it so when you plug it in it can announce its identity
and its capabilities to our toolkit.
Now, of course, the main editor. This is where the prototype's behavior is defined. And as I
mentioned, states encapsulate output think control flow, herald state charts and then control is
transferred from one state to another through events that come in through the infidel moments.
That raises another question, that is where do these events come from. It's fairly trivial when
you're thinking about discrete inputs but those are actually the less interesting case. The more
interesting case is working with continuous sensors such as accelerometers for resistors, photo
transistor, rangers, et cetera.
Now the challenges that these kind of inputs pose is any time you're sensing the real world, the
real world turns out to be noisy and nonlinear. If you don't have a background in signal
processing, this presents quite a challenge to the interaction designer.
Observe where the challenge is. It's not in kind of scoping what should be detected. We're
focusing on the domain of sensor-based interaction. So take the Wii tennis game. As a designer
I know I want to detect the swinging of the tennis racket. I can perform that action itself.
The problem comes in symbolically specifying to the system how to detect such as an action. So
the research question is can we use the performance of the action itself to do a significant
amount of the work of specifying the recognition algorithm.
Well, it kind of sounds like a machine learning problem. And, yes, of course, what I'm talking
about here is using recognition algorithms kind of under the hood to help you with more complex
event definitions for your prototypes.
However, from the perspective of an HCI researcher, you're actually not done at that point if you
have the right algorithms in hand. Because there's another problem, that is, what are the
appropriate techniques in the user interface that make real time demonstration and testing
feasible for designers? How do we elicit an appropriate example set and how do we deal with
breakdown cases such as false positives or missed recognition?
And the guiding insight here is you really want to combine the smarts of recognition algorithm on
one side with the smarts of the designer on the other side of the screen.
And I believe the way to do that is by combining direct multiplication with a rich information
visualization interface about what's going on.
So here's a quick demonstration of this extension to D Tools which is called Exemplar. So once
again you begin by connecting sensors. In this case, the two accelerometer, and then you
demonstrate what you want to recognize. Here it's a simple shake. Then mark up in the signal
graph, the part of the signal that corresponds to the demonstration.
You can then perform that action again, and see in real time a visualization of what was
recognized and why it was recognized.
Now, the crux, of course, is that generalization step. How do we go from one or a small number
of demonstrations to a more generally applicable rule? Well, I'll make two suggestions here. The
first one is just to use thresholds. These are computationally straightforward. However, they're
very powerful from the point of view of the designer, because they have a straightforward visual
interpretation. I can just track my signal, see when it dips into the right region. And I don't know if
you can see on this projector, we then highlight throughout the history of the signal regions that
match.
Now simple thresholds, of course, are not that useful when you're dealing with noisy signal data.
So that's why experts employ techniques such as historesis or double threshold and turns out
these techniques also have good visual representations.
So in exemplar, you can adjust parameters, split thresholds, define minimum event durations, and
as you do that interactively with each step we revisualize how your new event would have been
recognized throughout the history of the signal.
Now, that gives you two ways of checking whether you're done. The first one is inspection of the
signal. And the second one is just another demonstration in real time.
Now, thresholds do have limited power. For example, they don't let you capture anything where
the performance depends on variation over time. It's not good for gesture detection.
So for gestures, exemplar introduces pattern matching techniques where, from the user's point
the interaction is the same. Give an example, mark it up and then threshold. However, what
you're thresholding against now is a second graph that we plot into that same window and that
graph is an error metric of the pattern recognition algorithm.
The particular algorithm we're employing here is the dynamic time warping which comes out of
the speech recognition community. It's been around for a while, since the late '70s. The idea
behind that, given one demonstration and new input, compute how much you have to distort the
new input in both time and space to match the demonstration. That gives you an error metric and
then we plot that error metric back into the UI.
Speech recognition has moved on. They're mostly not using dynamic time warping anymore.
However, when it makes this algorithm especially promising in our setting is it works with a single
demonstration, where most of the techniques like hidden Markov models and others that are used
today require larger sample sets.
You don't want to turn the designer into this example provision machine. Another technical detail,
this algorithm is quadratic in time and space. But we don't really care about that because it's
entirely reasonable to sample human motion around 100 Hertz. So N ends up being fairly small
and it is not a problem to run a large bank of these recognizers in parallel.
So I'm showing you kind of two classes of techniques. Visual authoring and demonstration based
definition of events and the next question is how well does this work. And one of the challenges
in design tool research is oftentimes you lack a clear comparison for doing an A/B test.
In the absence of a clear comparison, what you can do, though, is a triangulation of methods to
examine the software artifact you've built from multiple different perspectives.
So that can start with inspection by experts using, for example, the cognitive dimensions of
notation questionnaire which we've employed, which comes from cognitive science.
Of course, first use lab studies which have two important limitations, though, with respect to the
kind of tools I'm building. The first one is you're always conflating learning curve with usability
and utility of your tool. And you are actually only interested in the latter for any tool that's going to
be used for any significant amount of time. The second one is if what you're testing is a tool to
build for something else you have to give people tasks to build something else.
It's quite tricky, oftentimes, to come up with tasks that you feel confident generalized to the task
you would encounter in a real world setting.
To work around that, we also tried to do class and industry deployment of our tools. And kind of
longer term settings.
Let me give you a few impressions of our user studies. So these were lab user studies of details
and exemplar. One task we gave people was to, from scratch, build an interactive prototype,
working prototype of an MP3 player in less than 45 minutes.
Another task was that given a set of mouse and keyboard control games and a box of sensors,
we'll give you 30 minutes to come up with new, more compelling interaction schemes to control
those games. I'll show you three quick clips from this. Here a participant with no prior exposure
to sensing techniques builds a booth where hitting the walls and stomping on the floor would
navigate a spaceship.
Another participant who uses accelerometer to detect rocket fire ranges to detect left and right
stepping.
And here's someone used the game where you aim and shoot rubberbands and you control that
game with a hand-held mini joystick and a bend sensor on the table where flicking would trigger
shooting the rubber band.
What do we get out of these studies instead of a few funny engaging data clips? Because we
have that fine grained data we can look at traditional measures like task completion time. This is
now data for building the MP3 player for that part of the study. It turns out, however, that task
completion time is not that meaningful when you have an open ended design task, because
there's no clear notion of what it means to be done with it.
So to get meaningful data, then, you can delve one layer deeper into the data and split the video
analysis, split and tally what did people spend their time doing.
And here I'm showing you a breakdown between time spent in software and the black bar and
time spent just assembling and modifying the physical prototype, in the shaded area.
What's exciting here is we're starting to approach parity and time spent between prototyping the
behavior and the form. Here in prototyping the foam core was sandwiched between layers of
cardboard. That's a successful material in prototyping for industrial design because it allows you
to do massing models really quickly.
Now, I also mentioned class deployment. And I just wanted to bring up here one example of a
successful pass collaboration with industry. So Leapfrog, the electronic toy manufacturer,
approached us about using D Tools internally for their advanced concepts group to prototype new
toys.
And in return for me training them, they then actually had one of their electrical engineers improve
our hardware design and manufactured a whole set of hardware tool kits which we were then
able to give out to students. So we taught an interaction design course with 50 students that all
used our tool kits.
Bubbling up from this section of the talk. So what are the important notions for this first section?
First hopefully showed that D tool successfully enabled nonprogrammers to program novel
functional ubiquitous sensor user interfaces. And the techniques that enabled that were first in
visual offering having that virtual model of a physical device that you're building and combine with
plug and play hardware.
And the second technique was raising the complexity of the event definitions that interaction
designers can work with by employing programming by demonstration.
All right. Moving on now to the second topic of this talk, how can we aid iteration. So the
repeated refinement and modification of prototypes in our design tools. Well, a second result of
our inquiry was that user tests, even of incomplete rough prototypes at design companies today,
are frequently video recorded.
And that is because video is essentially free. Doesn't cost much in terms of equipment or
storage. However, what's expensive is revisiting that video later on. So a typical use case would
be I just collected an hour or two or three of user study data and really all I care about for now are
three five-second snippets that distil the key insights so I can communicate those to my team.
Now, the insight in research was that it's possible to integrate the design, the test and the
analysis of user interfaces into a single tool so that we can then afterwards leverage the
correlation of the design model with the test data of testing that model.
So I'm going to show you how such a test would look like. So we have a prototype built with our
toolkit. We now just point a camera over the shoulder of the tester. Instead of recording to RAM,
we just record to flash, and we record straight to the computer and annotate in real time the video
with the states that the software model was in, all possible events that occurred by user
interaction, and then in addition we can provide the experimenter itself with a big red button that
says review this later.
Now, after test session is completed, you can then leverage tight synchronization between the
graphical interaction model, the video and the test data, kind of the meta data that ties the two
together.
This is a prime example for leveraging a multi-monitor setup that many of us already work on. So
now we have the design environment running on the left-hand screen and the video review
application on the right-hand screen. And the interactions that are now possible are that you can
get context of what happened in the video in the software model.
So as the play head scrubs through the video, we can highlight in our software model where the
user was at that point and also where they were coming from and because the test is over where
they're going to next.
So that gives you additional context. More importantly, though, is probably the other direction,
where you can now use your state model as a query interface into the video. So I can click on
any particular state to execute the query, show me only those video sequences where the user
was in that particular state.
And that can either be done in software or actually through demonstration on the physical
prototype itself. So, for example, the query show me all video segments where the user moved
the volume slider on the device can be executed by picking up the device and moving the volume
slider.
All right. So built it , prototyped it tested it and looked through and analyzed the results of that
test. That most likely suggests some changes that need to be done for the next revision.
If there are no changes, we actually didn't learn anything from the test and the test kind of was a
waste. Now, designed as a team sport, undertaken by people from different backgrounds, so
most likely the people, the person reviewing is actually not the person who built the prototype in
the first place.
So that raises the question of what would suitable interfaces for advising interaction designs look
like? And this is really early work that's been kind of in progress, so I'm going to give you a quick
snippet of this.
Really, our approach is to say that, well, interaction designers really concerned with two separate
activities that are interrelated. It's about the definition of appearance and the definition of
behavior. So maybe we can take inspiration from the different realms that are concerned with
behavior in the one sense and pure appearance on the other hand. So, of course, for definition of
behavior we have a long history of successful tools in revision control and versioning, which
operate on the level of the source representation itself and show you what changes were made.
However, if you look in the domain of appearance, you actually get very different tools. So on the
left-hand side is a shot from Pixar where directors, after reviewing the daily 3-D rendering of a
scene annotate on a Tablet PC how things should change for the next rendering.
Now, observe none of the changes here are done. They're just suggested in a more abstract
medium. So to me that says we actually want to have both of these functions available in a
revision tool. So here's the first stab at what a revision tool should look like. To enable sketching
this is pen driven. You want to be able to have both high level feedback through sketching as
well as semantic actions think track changes in word that visualize insertions, modifications,
deletions to the structure, to the kind of information architecture of your prototype.
Bubbling back up a level. So what's important in this section about supporting integration.
Integration of analysis data into a design tool can radically shorten the time required to work with
that test data.
And the second point that I would like to make is that integration is really enabled by having this
high level visual representation of the software model. Now in D Tools you kind of get that high
level representation for free because that also happens to be the representation that you author
in, that control flow from state to state.
Now, these techniques don't directly apply if, for example, your entire software was written in
C++, because there's a huge granularity mismatch between tying a second of video to lines of
code that executed. So we may actually, for those cases, maybe automatically create some kind
of higher level representation of that software.
And then, finally, design is a team sport. So new revision tools are needed to enable and capture
and express insight between iterations.
And the last part of my talk, I'd like to really ask the following research questions: Research
question: How can tools support the creation of multiple user interface alternatives in parallel?
And that question didn't just arise out of a vacuum. It was an interesting study published by Brad
Meyers at Real HCC last year where he did a large survey of 259 interaction designers. And one
of the key findings of that paper was that designers frequently wanted to have multiple
alternatives of a behavior side by side but that there was no way to do that in current
implementation tools.
Now, when we started this project, I asked the question: Maybe we can gain inspiration, once
again, from neighboring domains. Maybe they already have some good solutions prepared that
we can leverage and work off of. And kind of the most relevant things I found were design
galleries, research done at Meryl, which looked at the problem of finding your way to good points
in the design space of large 3-D rendered scenes.
So take, for example, particle systems. We have a large number of dimensions. And it's not
clear a priori where to set those parameters to get good visual results. Their solution was we'll
just off line overnight generate a hopefully space-spanning set and then let you select the right
alternatives.
Mark Lavoy proposed using spreadsheets where the spreadsheet paradigm is inherently
supportive of having multiple possible what-if scenarios living side by side.
Unfortunately, these kind of solutions don't translate straightforwardly to the task of interaction
design, and that is because visual tools like D Tools, notwithstanding, actually a lot of the nuts
and bolts of interactions happens to be done in code. So whether it's action script in Flash or
HTML and CSS in DreamWeaver, or if you look on the open source side, the most popular tools
right now are Duono and Processing which give you thin wrappers around C and Java.
Now once you're working with programmed interaction, of course, now we have two
representations you have to worry about. There's the source code where you make changes,
where you express the alternatives and the running program where you then observe them later
on.
So in the absence of good existing tools for supporting exploration, you can ask the question:
What do people already do to kind of get around those limitations?
So we conducted a small study where we asked interaction designers to come into our office and
bring the source code of their last project. And this was a little dicy to convince people to open up
their code. But we got a number of respondents. And really from that exercise derived three
separate requirements.
The first is creating a satisfying user experience often involves tweaking the dials of your
application. So the code stays the same and you're just experimenting with different parameter
settings.
Now, a second requirement is that to compare different behaviors, changes to logic may be
required as well. Don't worry about what this function does. This is a code someone brought in
during the inquiry.
The important part here is that there are actually two different alternatives co-existing side by side
within the same function body and then to switch between testing one and the other between
executions, this participant would move the comment lines up and down.
So I think tool support, creation management of different code alternatives in a better way than
the moving of comment lines.
And then the final observation was that management of code and data alternatives really has to
span both the editing and the execution environments because you express them in one but
observe them in the other.
So you really want to play around with them not when the code is up but actually when the
program is up and running. Now, experts already do that today through providing separate slider
panels. So at some point in a complex project, someone will just say I've had enough of this
messing around and reexecuting, I'm just going to write a slider panel that puts all the relevant
dimensions right there on my screen while the program is executing.
Now, this practice right now is, first it's really reserved to expert programmers; and, second, this
is a whole bunch of effort you expend that is not part of the final deliverable at all. It's code that's
written solely to get some hooks, get some instrumentation into the behavior of your program at
runtime. So wouldn't it be nice if you could just generate those for free automatically?
All right. To see how we could address these three requirements in practice, we will juxtapose,
which is an experimental both code editor and runtime environment. Excuse me, which manages
source alternatives in an augmented editor. Those alternatives are then executed side by side in
runtime, and augmented by automatic generation of control interfaces to let you modify
parameters of the running application.
All right. Working with multiple code alternatives. Kind of sounds like a big headache for the
developer. How do you keep track of what the alternatives are. How do you express them.
So I think there are really two requirements that your authoring environment has to deliver on.
The first is offer techniques to manipulate content across different alternatives in parallel.
And the second one is visualize. Make in some way visually apparent of what's shared and
what's not shared between your different alternatives.
So in our editor which we wrote for Action Script, compiles and runs Flash files, alternatives are
expressed in tabs. So tabs are no longer different files, they're alternatives of the same file.
And then there's a linked edit mode which basically determines where your edits are applied. So
with linked edit mode on, anything you type gets propagated to all alternatives that exist. And to
know where the right spot is to apply those alternatives, you basically use the DIF algorithm
underneath the hood, so we can keep cursor correspondents in one location what corresponds to
the right in another alternative in the text documents.
When linked edit is turned off, your changes are going to be made locally to only to the document
that you currently see, and then to make that apparent what's unique we just highlight it in a
different background color.
So I was thinking about what's kind of a small but salient example that demonstrates how a
designer would use parallel editing for exploration. And luckily someone else provided the
answer for me. This is Chris Harrison at WIS 2007 who had this nice insight and published a
paper on it that varying the function that maps actual progress in the progress bar to displayed
progress actually has implications for how long people think -- how long people think it took
regardless of how long it actually took.
So he came up with a bank of different functions for mapping actual progress to displayed
progress, and then measured the difference that people felt. So here the application harness is
the same each and every time, right?
I have some window that has some functions and then a progress bar. And the only thing I'm
varying is that one line of code that expresses this function. So here's this project we
implemented in Flash. I created four alternatives. They really only change in two lines of code.
One in the label and the other one in the mapping function.
When I hit run, I now compile the set of applications and they get tiled on my screen so I can
rapidly look at them side by side. Now it may also be advantageous sometimes to provide
parallel input instead of sequential input. Here we add support for intercepting elements and
inject ghost events in all the other alternatives.
Now to explore parameter spaces really the goal of juxtapose is basically just from the definition
of suitable parameters variables in your source code to automatically generate control panels.
And here's another example of when that might be useful. Many of you here may be familiar with
this particular project of Patrick that in a collaborative setting, if you have multiple people looking
at the same screen, it's hard for the person who is not controlling the mouse to make sense of
what just happened last in the interface.
So Patrick had this great idea of using basically afterglows of widget manipulations. Now those
afterglows -- there's a trade-off space of how visually salient do you make them; you want to
make sure they don't get missed, but you also don't want them to clutter the user interface.
There's lots of dynamic and interface parameters for these parameters that you want to
experiment with.
Here's a reimplementation of [inaudible] that I wrote in Flash, and you see on the right-hand side
here we get a completely automatically generated control panel that allows me to at runtime
modify the behavior of these afterglows.
Just to peek underneath the hood. Can we generate these control interfaces? Well, first I would
argue that you actually want them to run in a different process. So you don't more than
necessary interfere with the timing of your original application. So the UI is going to run, the
control UI is going to run in a different process. The user process here, before we compile the
program, we add a little library stub that on start-up then basically wraps the main entry point to
the application and uses language reflection to inspect the user's application, package up a
message containing names, types, initial values of variable, send that across using remote
procedure call into our runtime process, which, from that information, can then generate the
control UI.
When that control UI is modified we just do a remote procedure call in the other direction into our
library, which can then update the running application.
So let me just spend a couple minutes talking about evaluation here. One of the questions we
asked was well, when is this tuning really most useful?
And the way we try to get at the answer was through a closed-ended task which allowed for
precise measurement of completion times and other metrics, where we gave people procedural
code that recursively drew this tree, and to hopefully make this a little more externally valid we
actually took this task from a book on authoring dynamic flash applications.
So we get people code that drew this tree and we said now your task is to produce these four
trees by varying the parameters of the code. Here's what one of these sessions would have
looked like.
And so here now we actually have a good A/B comparison. We can in one case just let people
use our tool and the other one we just get rid of this panel but have them use the same editor and
the same runtime environment otherwise.
Here's the result from the obvious department. Of course, if you give people sliders to change
parameters, they'll test many more variations than they ever could if they had to recompile.
Direct manipulation is good. But we already believed in that. I think more interesting thing is the
question where does it provide benefits.
You delve a little deeper in the data and you actually see there were only significant differences
for these two trees. Why is that? And it turned out that this code was not written by computer
scientists it was written by designers.
There were some strange interaction effects between different parameters that made it hard for
these two trees after initially reasoning about what the parameters should be to then, by
reasoning alone, close the remaining gap.
So it was precisely when reasoning could not get you closer, when trial and error was the way to
go that tuning made most of a difference.
Now, that actually adds a little wrinkle to the story I told you before because if really the value is in
exploring multi-dimensional interactions between parameters, then a mouse-controlled interface
is definitely suboptimal because you always have only a single point of control.
So the design response to that is to make it possible to use a multi-dimensional input device.
Here you can map different parameters to an actuated media slider board that musicians use.
That that actually turns out to have three benefits. The first one is more than one dimension of
control in parallel. The second one is with the mouse it's a hand-eye coordination task to adjust
your control sliders.
Here, your eyes are free to focus on the application that you want to tune. And the last one is
because the mouse is still free, you can, in parallel, tune interactions that depend on mouse input.
So I think there's a good case for kind of moving the control interface off on to dedicated
hardware.
Now, the first two projects I showed you were really in the realm of ubiquitous interaction. And
here I show you unique alternatives for exploring off the desktop. The natural next question is
can we extend these benefits to pervasive computing?
And I looked at two particular points in that space. The first one is interaction design for mobile
phones. So here mobile phones are, of course, the new commodity hardware. So mobile
developers already frequently test with a whole bank of devices that they own.
So this now gives us the opportunity to leverage these device collections by rendering out
different alternatives to different phones.
So you may test your alternatives by simply picking a second phone up that runs a second
alternative and put the first phone down. Now, the interesting part here is that you still have a
control interface that runs on the PC. So you still have hooks into the application that runs on
your phone, but that phone now actually can be taken outside the lab and you could potentially
monitor as well as influence what's happening on the phone application from a remote location.
The second domain I briefly want to talk about is the realm of working with micro controllers. So
those are the tool of choice if you want to experiment with haptic interactions or other sensing
interaction. I picked the Argueno [phonetic] a typical stand in for the hardware that's used for this
region of work today.
So you typically find something like an eight bit risk chip, 16 K of memory with fairly low level -well, at this point it's already having, sees already a high level tool, statically compiled languages
and really small footprints.
So just one final engineering detail that we ran into is the question of how would you actually
implement this variable tuning in a static language that does not have reflection and no other
good runtime information?
Well, instead of leveraging reflection, you can actually, before you compile the cold, you can
parse the code and you can't get all variable information, but you can definitely know what the set
of global variables is. You can then build a symbol table that contains names, types, and runtime
pointers to those variables. And then emit that symbol table as C code back into the original
program.
You can then compile the modified program, and that's what you ship off to the micro controller.
Along with some communication routines, so you can talk back to the PC.
And that then allows us at runtime when a tuning widget is manipulated or to send a message
and use our symbol table to do pointer referencing and change the value of the variable in the
running program.
From pointer to referencing back to the high level. So what's important here? Well, the high level
point is that a structured approach to supporting alternatives can eliminate a lot of the rough, the
nasty practices that people retrofit on their current tools, and plain enable more interaction. The
tool support for doing that requires connecting both the authoring and the runtime environments
of the programs.
And juxtaposed in particular, introduce three techniques, linked editing to manage code
alternatives. Executing programs side by side with either serial or parallel interaction and
automatic generation of tuning interfaces.
As a quick summary and a future agenda, I introduced today systems that address three
important concerns for the prototyping of new user interfaces. First enabling a larger group of
interaction design practitioners to build functional prototypes of user interfaces that live off the
desktop and have some real embodied physical presence.
Second, introduced techniques that allow you to much more rapidly make sense of feedback you
gathered in the iterative development of user interface prototypes.
And, third, juxtapose introduced techniques for exploring multiple user interfaces in parallel.
Briefly wanted to mention some kind of reception of this work. Unbeknownst to me and to our
group there was a research group in Germany which took D Tools because it was open source
and just extended it to also record timing characteristics of user interaction with prototypes and
they published a paper at CAI on this and there are two recent O'Riley books which discuss these
tools. The first is Tomago Making Things Talk teaches in the interactive televisions program at
NYU and this is a book for students and enthusiasts to get a foothold in computing. And the
second book came out maybe a month ago from John Safford. He was a consultant at Adaptive
Path. This book is how to help them reason through making the transition between desktop
interfaces and [inaudible] controlled interfaces.
In terms of a future research agenda, there are, of course, many concrete topics that fall directly
out of the research from the prior system. So one concrete example juxtaposes, I haven't talked
at all about enabling exploration of alternatives of hybrid authoring environments like blend or
Flash that have both code and visual direct manipulation editing. That's concrete.
In terms of larger other ideas, the first is how can -- well, languages or maybe execution
environments and IDs be co-designed to facilitate prototyping. So a good example was that we
found in juxtapose, it's actually very valuable to have some kind of additional keywords or
modifiers in the source language that can inform the automatic generation of these control
interfaces at runtime, and that's just one instance of a much larger space of kind of development
environment support.
I'm interested in the question of design and programming by example modification. So very little
code these days gets written by firing up Visual Studio and just typing from scratch. Much more
code gets found by initial search through either company databases or search on the web for
public resources, and there's an interesting, if somewhat complex, analogous practice in product
design where during the initial phases you oftentimes your team will construct kind of a project
room or a war room and put up as many and as diverse a set of examples of other existing
products in that space as you can find.
So you find practices built around working with examples both in programming and design. It
would be nice to delve into that deeper.
And then as a larger project, there's the question of what will the interaction design studio of the
future look like. The work I presented today had the commonality that I used, graphical user
interfaces to develop nongraphical user interfaces. All the tools were desktop-based. The target
domain is off the desktop. So what happens if we flip that first bit?
Can we actually make use of surface computing and other ubiquitous computing technologies to
help how, especially team design happens. Can we transform the designers' work spaces?
And really the first impetus to that work was done right here with Mary and Banco and Andy last
year at MSR, where we built a large surface to support design team meetings. And the
motivation was that in observing what happens currently at design team meetings, that there's
kind of an uneasy co-existence between physical and digital artifacts, whether it's sketches in
them [inaudible] printed out schedules for the team, and together with PowerPoints and other
videos of prototypes. If we let our design tools move into the physical world we have a much
better chance of integrating these practices.
So there are some initial interaction techniques that we worked on that allow you, if you can track
where objects are on your surface, they'll allow you to rapidly capture high resolution, digital
images of any artifact, or go into the other direction if you want to walk out of the meeting with
something take a digital artifact and basically use your table as an impromptu light table to quickly
sketch on paper.
Of course, these are kind of the first inklings of techniques in a larger plan to holistically support
design activity and we'd be happy to chat with you about that later.
And to conclude, I want to point out that none of the work I presented today was done by me
alone. And it was a fantastic set of collaborators at MSR and undergrads master students and
Ph.D. students who worked with me over the years. And also just want to point out kind of the
variety of companies who have both used design tools to give us feedback or served as sites
where we could do contextual inquiry.
More information is in recent papers and all of those can just be found on the website. Thank
you.
[Applause].
>> Andy Wilson: We have time for some questions.
>>: I have a question. In Juxtapose, there's kind of two benches that I've seen for the system.
One is that you could see the actual behavior simultaneously from multiple instances and the
other one is this automatic control, generation controls. Did you find that one was more useful to
the other? In particular, was it useful to have multiples running at the same time?
>> Bjoern Hartmann: Yeah, so the multiples. The tuning of parameters only gets you a win if the
code stays constant, right? Any part where you want to change the code itself, that is much
better expressed by creating multiple alternatives and then running them -- whether you run them
side by side or focus in one versus focusing in on the other is one question, but certainly having
authoring support for keeping track of those changes was definitely used. So I didn't in this talk
report on the task where we had people design -- we basically gave them a code framework for
rendering and interacting with a map and gave them tasks to think about, if this were for a GPS
device, what changes when you're designing for pedestrians versus bicyclist versus drivers.
And there we saw a lot of use of these, of code alternatives and also get executed side by side to
both think about changes and also have it be a lightweight, smaller scale versioning mechanism,
right?
In a way, some of this, it shares most of the back end architecture with version control, but
because we're targeting a different time scale of interaction and a different kind of scale of
changes, the interaction techniques end up being very different, and the feel for what people do
with those tools end up being very different.
So, yeah, both we definitely saw both in use.
Other questions?
>>: I'll throw one in here. So one thing that strikes me about all of your work was the active
programming now is more of performance, like it's an ongoing activity and it's more like you start
with a lump of clay and you build it and one of the key components there is you always have
something that's there working doing something. I know in my own work life I'm very nervous
when I don't have something running, and I was wondering if you felt that people who were from
traditional computer science background were comfortable with that way of doing things or if you
found that some other, maybe we're better off having somebody who is more into, I don't know,
some other area of expertise would be better at doing this kind of stuff. Maybe a mechanic might
be better off at doing this.
>> Bjoern Hartmann: Well, a couple of things I want to touch upon in my answer. The first is that
we see actually a lot of experimentation in this space of applications, really moved out of
computer science and moved into the realm of it would be the professional design shops or
hobbyists, because the promise is there. It's very appealing for people to get started and it's not
that hard to make the first steps.
But then subsequent steps, once you delve into it, if you used the current tool, you often find out
at some point you've bitten off more than you can swallow.
And there's certainly I've aimed these tools more at interaction designers and I have less data to
report on how computer scientists use them.
I want to touch upon, though, that issue of how -- how close do you want that integration to be
between something that's always working or when do you want to say: Stop, now I really need to
go back to the drawing board. Give me two hours to make a big change and come back. And I
think there definitely is a space -- you need to provide support for moving along that continuum.
Far on the other end of the spectrum is some other work in on the fly programming that people
like Guong who came from Princeton who is now at Stanford, do for live programming of audio
performances where everything you type gets interpreted on the fly and you definitely get -somehow that puts limits on the amount of exploration you're going to do in real time. Because
sometimes you need to do a larger refactoring, whether that's code refactoring or mental
refactoring of how things work.
So I realize that as something too play close attention to, but I don't have any good principles to
report about it.
>> Andy Wilson: Any more questions? Let's thank Bjoern.