>> Ken Hinckley: All right, thanks, everyone. Thanks for coming. I'm very excited to have an
awesome slate of speakers today, kind of covering all ranges of HCI. First up, I have Roel
Vertegaal from Queen's University, and he's going to talk to us about real displays and real
realities, a topic that we're all intimately familiar with, but perhaps we could do a lot more with
in HCI.
>> Roel Vertegaal: Thanks, Ken. Hi, everyone. I want to talk a little bit about some of the
developments at the Media Lab in terms of using shaped displays. About 10 years ago, it
occurred to us that all software lives on flat displays and therefore lives in flat land. And we've
been working ever since to try and change that and trying to include shape of the display as a
design factor, so I'm going to talk a little bit about that work, but mostly about some of the
challenges that we see going ahead and towards making this a real reality. So today's computer
design is stuck not just in the 20th century. It's stuck in the 1920s of the 20th century. Above
here, you see this is actually 1950s design. This is Dieter Rams' design of a radio for Braun, and
what you see right next to it is Jony Ive's design for the iPod. It's very clear that Jony takes his
cues from Braun design in the 1950s. Very beautiful design, but very much aimed towards
modernism, this thought that you have a clean interface, you have white pixels, you have
Helvetica. You have straight lines, squares, flat surfaces. And that was a trend in the early
1900s, but since then, architecture has certainly learned a lot. What you see here is one of the
main protagonists of this movement, of the Modernist movement, Ludwig Mies Van Der Rohe,
who was a director of Bauhaus in Germany but ended up in Chicago, which is one of the reasons
why Chicago has got such awesome architecture. This is the Barcelona Pavilion, and you can
see how much current-day user interfaces take from the literal windows that we see in this
particular pavilion. This was in the 1920s in Barcelona, during the exhibit. So we take our
inspiration from a different architect, who's also from Chicago, Frank Lloyd Wright. And in
1939, Frank Lloyd Wright came up with this idea for an organic architecture. This idea that a
building can be placed in situ not in contrast to nature but as a part of it, sort of harmonizing
human design elements, such as what you see here, with the cantilevered balconies, as well as
natural elements, which you see here with the natural stone, which actually cements the house
into a waterfall. And it's a really peculiar spot. When he was commissioned to do this, the
owner of the property said, well, we really love the waterfall, and so they thought that he would
build a house overlooking the waterfall, like everybody else does. Instead, he built the house
right on top of the waterfall. But what's cool about that is there's a real flow going through this
house. There's a real motion to it, and this is consequently one of the most spectacular pieces of
architecture from the 20th century. This spawned a whole movement of organic architects,
which got tied to sustainability movement, promoting recycling, reuse, using natural materials in
building, but also using shapes that are based on natural shapes rather than the kind of rectilinear
shapes that were part of the Modernist movement. And so we looked at some of the design
principles that these architects came up with and derived a number of principles for what we call
organic user interfaces, so user interfaces with a particular shaped display. Firstly, and this was - now, it may be obvious, but back in 2007, when we first published this, it wasn't, input equals
output. So this whole idea of control, controlling a device, is very much an engineer-y way of
thinking about things, and with the subsequent popularization of tablet PCs, we see that users
like to use their fingers right on the graphics, and that is a principle that we hold dear. That also
means that, for example, if you have a shaped display and you can change the shape of the
display, the change of shape can be input into whatever computational operation you have. And
we like to stop thinking about these things as devices, but rather as things, very much in the trend
of Internet of things. Why are computers not just things? Apparently, they're not sufficiently
advanced for us to think of them as things. Fire was once a technology. A spoon was once a
technology, but we don't think of them as technologies anymore, because they're just things.
They're there. Form equals function. It doesn't follow it. Following function was something
that was very much a Modernist thing. Frank Lloyd Wright proposed that a shape actually
affords a particular functionality directly, and so change the shape, changes the functionality.
They have to be in sync. They equal one another. So we used that as a design principle, and
form follows flow, well, you see it right here in this building design. We live in a world of
multitasking of activities rather than tasks, and so to really think of a single device that supports
a particular task is not really appropriate anymore. You need to be able to change the shape of a
device, for example, in order to support different tasks. A great example of that is a flip phone,
where if you get a phone call, you literally open the phone call, and it also fits better to the shape
of your body when you do that. So we need to get back to that kind of thing. So as I said,
organic user interfaces are user interfaces with non-flat displays. Part of this is that they may
actively or passively change physical shape. They certainly will have a physical shape, but if
they're flexible, they can actually change that shape, and this was in Communication of the ACM
in 2008. Now, I argue that display technology has been a driving force behind HCI. Most of the
major revolutions in our field have been the result of a new display technology being introduced,
and every new display technology renders obsolete the previous display technology. How many
of you still own a CRT or even know what that stands for. And oscilloscopes don't count. So
the LCD displaced the CRT, and we are fully confident that the new OLED and FOLED
technology for some really simple principles of market dynamics will replace all LCDs and
potentially also projectors. These displays are so thin that you can apply them as wallpaper. As
soon as they're big enough, that will happen, and you won't need a projector anymore. So let's
have a look at a couple of these. So the CRT was invented by Karl Ferdinand Braun in 1897. It
resulted in the invention of the trackball Canadian inventors Tom Cranston and Fred Longstaff.
Not too many people know this, but the graphical user interface was not invented in the United
States. It was invented in Canada in 1951 by Ferranti. This evidence of this is the 10-pin -- or
sorry, the 5-pin bowling ball that was used in order to create this trackball. And it was
subsequently followed by work by Engelbart and Sutherland using light pins and mice 10 years
later. It resulted in the graphical user interface that we're all using today, Windows. This is
Bravo, which is the Xerox Star implementation, and really what we see is that very little have
changed. I think this didn't have movable icons, but it had pretty much everything that a current
graphical user interface still has. One important thing to note here is that the reason we have
Windows is because we only have one display, and Windows are designed to carve up the
display into multiple sub-displays. If you have very cheap and many and plentiful and
lightweight displays, you don't necessarily need Windows anymore, because you can just have
one window per display. The LCD was invented in 1968 by George Heilmeier at RCA, and this
allowed Alan Kay to immediately dream up the form factor called the tablet PC. It was known
as Dynabook, and I don't know if you've ever read this paper. It was targeted towards a device
for children of all ages to learn. It is uncanny how much it describes the iPad. It even talks
about if end users would program on this device, the first thing that they would program would
be an ad blocker. It talks about getting books from the library digitally, sort of alluding to the
Kindle and Amazon.com. This in turn led to ubiquitous computing, because we had these flat
form factors and small form factors and different display sizes, and of course, multitouch,
pioneered amongst others by a fellow Canadian and Microsoftie, Bill Buxton. So in the early
2000s, flexible displays came along, and I saw this video, and I knew it was going to change
everything, not just because this is so cool. It's just fundamentally different from anything we'd
seen before. At the time, there was lots of pixel breakage and all sorts of issues. These issues
have largely been solved in the following 10 years, but the idea that you could wear this as a
wristband or apply it to, I don't know, any kind of day object like a cup or something, was very,
very compelling, leading to this notion that we could have, for example, things that are
interactive computers and not necessarily things first and computers second. So why flexible
displays, and why are they going to take over the market? Well, there are some real simple
physics and economical advantages. The first is that they are very lightweight. I didn't mention
in this list that they are nearly indestructible, so if you hit them with a hammer, they don't break.
So if you drop your phone, it would basically fall like a leaf, and it wouldn't damage. The only
way you can damage them right now is by creasing them. So they're more mobile, and it's much
easier to carry more displays, so you can have many. They're thin, and that means that they fit
thin form factors like watches or wearables or clothing. They're much cheaper to manufacture.
That's not true right now, because they are being produced in volume now. They are a part of the
LG Flex, for example, and I'm pretty sure that the Galaxy S5 also has a FOLED, because it kind
of curves its screen over the edge, and that's hard to do with an OLED. These factories churn out
about maybe a million units or more a year, but they're all old LCD factories, so there's no rollto-roll printing yet. Potentially, these things can be printed like newspaper, and that would yield
about a third of the cost, and the cost of a display in touchscreen in a cell phone is the biggest
cost factor. It's about $75 in iPhone for the multitouch screen, and that could go down to $25 or
potentially even less. Not true right now, but will be 10 years from now. Much higher color
contrast than the LCD. The graphics just look better. They flexibly conform to your body, and
Bentgate has taught that this is certainly now of user concern as phones get thinner. It starts to
become hard to maintain their structural integrity, and maybe we shouldn't even try and just not
have this brick in your pocket but actually have something that follows the human body. But for
us, what was really important was the last two things. They allow for a 3D display form factor
that can be flexible, cylindrical, maybe arbitrary, and in the case of a flexible application, they
can be bendable. And what's cool about this is that you get haptics for free. Because we're
dealing with shape now, we get passive haptics of shape. With bendable screens, we get haptics
from bending them and so that means that the amount of bend is represented in your kinesthesia
feedback. So early on, this was 2004, we started thinking about what would happen if you would
have these displays, and of course, we didn't have them, and we certainly didn't have them at the
size of a US letter, so we designed this projected system. It was one of the first, I believe,
projection mapping systems, certainly one of the first that was sensing shape actively and
changing the graphics accordingly so that we could have pixels that were the correct size, no
matter what the shape of in this case the paper display was. And it made us think about many
displays and being able to move data between displays and other things like that. And then once
this was about 10 years later, the real thing came along. We did PaperTab.
>>: PaperTabs represent apps as physical windows, working together to create a seamless
experience. The location of each PaperTab is tracked on a desk through an electromagnetic
tracker. Tabs held by the user act like full screen physical windows. Tabs at arm's reach in the
warm zone show thumbnails of documents, while could tabs that are out of reach show app
icons. These apps can be opened by touching them with another PaperTab. When a tab is held
by the user, its flexible touchscreen allows full editing of its contents. Here, we see a user
picking up a PDF document by touching one PaperTab showing PDF icons with another. The
document is opened and navigated by bending the display, allowing the user to flip through its
contents without requiring buttons or swipe gestures. Here, we see how a user uses a bend
gesture to fast-forward through a video. The user reverses the action by dog-earing the display
in the opposite direction. Here, a user is creating a larger view by placing two paper tabs
together. Because the displays are thin, it's very easy for users to draw or drag graphics across
multiple tabs, as you would with paper. By working together, PaperTabs make it easier to work
with multiple documents than traditional tablets. Emails are simply opened by touching the
inbox tab with an empty tab. A quick dog-ear creates a reply. Attaching a photo is as
straightforward as tapping the reply with the photo. The email is sent by placing it in the outbox.
In PaperTab, each app is a tablet.
>> Roel Vertegaal: And so we did a study to validate whether the following parameters of
flexible displays were really driving change in the future. There have been quite a bit of
resistance in the HI community towards the thought that you needed actually to adopt true paperlike form factors, and I think the stance was that you could use a regular tablet PC and have
many of these and simulate the kinds of interactions with paper. And we wanted to show that
this is not true, that you actually need lightweight and very thin and flexible displays in order to
truly create a desk-based system where you have physical windows. So here we see how
PaperTab works. We have these hot zones, warm zones, cold zones. It's based on work by
Abigail Sellen on how people manipulate documents. So in the cold zones, there's archives, and
in the hot zone is active documents. They are typically full screen open. In the cold zone,
they're typically thumbnailed, and you can pick them up by tapping them. So this whole action
of tapping became an interesting subject of the study. So what we did in the first task was we
used the displays on area cursor, so instead it's a Fitts Law task, but instead of having a certain
width and then having a pointer move between these two objects of a certain width, the display
itself is the target or provides a target width. We also made sure that within the display, we
could do smaller targets by painting them, dynamically updating them. And then you'd move the
display between two laser-pointed targets on a desk. In the second task, we were using a display
as a pointing device into another display, and so you could actually tap within pixels on the
display, and this was a classic Fitts Law study. And so the two parameters we studied were
flexibility and mass, and we controlled very carefully for this. We also added paper, just as sort
of a baseline to see what that would do, and somewhat to our surprise, we actually got a
significant and substantial gain from the displays' being flexible. The effect of mass has been
well known since Fitts's first study, where he actually studied the effect of mass on pointing
performance, so what we see here is the index of performance in this task for the various
conditions, so we had a two by two, where we have flexibility and we have mass, and we see that
the flexible low-mass condition wins. And so flexibility is an advantage most likely because
when you're tapping on a table, the display gives way so that your hand doesn't get caught
between the tablet PC and the table, and this facilitates the pointing task in both cases. Paper did
not do as well, and we hypothesize that this was because it was too flexible and would start
actually being -- becoming turbulent when moved quickly around, because of air flow. So here
we see the regression analysis of this, and flexible low mass has got the lowest line there. This
was the results for within-display pointing, and we see a similar effect. We left out paper for this
one, because we knew it wasn't going to perform, but again, we're seeing an effect of flexibility,
and we're seeing an effect of mass, meaning that you cannot simply assume that if you want a
many-display system, you can just grab a tablet PC like a Surface and take a couple of those,
simply because users will not be able to move these bulky systems around like they can with
paper. And if you ask the questions, like, well, why is this important? Well, why do you have a
printer? It means there's something wrong with your screen, that you still have to print out
paper. And one of the chief features of paper is the multitasking capability, the fact that you can
compare data on one page very easily with data on another page, which is very hard to do when
you need to scroll in a document on a single display. And here are the regression analysis for
that. All right, so that was form factor number one. Form factor number two is applying these
displays to rigid objects. So now they follow a certain shape, and what you see here is one of our
first commercial products coming out of this line of research, which is called Vessel. It's a
startup located in San Francisco, and this is a beverage container that has a computer inside it,
and it has analysis equipment that can determine what kind of beverage you pour into it for the
purpose of tracking the other side of the calorie equation, of calorie intake. About 50% of your
calorie intake is done through beverages, in particular pop. And this thing can tell the difference
between Pepsi, Coke. It can tell the difference between you drinking a coffee with one sugar or
two sugars, and it can calculate your calorie count, offset it with your calorie expenditures, but it
can also maintain hydration levels, warn you if you haven't been drinking enough, can maintain
caffeine intake and all sorts of other parameters. But the research that inspired that was this, so
we were using the same technique we used for PaperWindows, where we were projecting on -this is very grainy footage, but it's all I have. We were projecting on an actual Coke can,
thinking about a form factor that was so different from a real computer, and of course, Coke cans
weren't the thing to do. We were able to make them interactive. It's certainly possible to put a
flexible screen on them, but they only cost $0.05 to manufacturer, and it's not something that will
ever live in the real world. Plus, there's real reusability issues with that, disposable device.
Here's another. Okay, let me just -- here's another cylindrical form factor. We published this at
TEI, but we did the work about a year and a half ago. Here, we're using a display to create a
smartwatch that has got -- I believe this is a seven-inch display. I forget. But it's a 240-degree
field of view around your arm, and this is the kind of form factors we're thinking about. Of
course, you'd want to do this with a FOLED, but we did it with an electrophoretic screen. Lots
of cables, still issues like that, but those will disappear very, very shortly. So the question then
becomes, what else can we do? Well, here's case study number three, three-dimensional. So
here we see PaperFold. This was also presented at TEI this year, and this is a display that is
made out of panels that are clicked together with magnets, and as you fold the display, the
change in shape of the device is sensed, and it produces a view port change. Here, you just saw
someone browsing through some photos and adding a panel for being able to edit the photos.
Here, you see someone in a phone call using both sides of the display when it's folded up, so we
have a cell phone here, essentially, that has displays on both sides, ideally completely surrounded
in displays. And then the user shares a link, and you can open that up by opening up the phone.
Here, we see the phablet in a configuration that's much like an ultra-notebook. So it shows a
keyboard when you do that. When you then flatten it, it shows Google Maps. When you put it
into a curved location, you get a Google Earth view, and in this case, we're actually picking up
the sketch of model of a triangular building by putting the device in a triangular shape. And this
is then printed out on a 3D printer, just for kicks. But the challenge is what else can we do, so
what other shapes can we do? This is actually something we do on a regular basis. We can cut
these screens and then keep working, but if you cut them the wrong way, you cut lines away, and
parts of it stop working, and so there's only certain shapes you can do. You can do a triangle.
We can do a circle and a couple of others, but with the cable management and all that, it
becomes really, really hard to do polyhedrons, for example. We'd love to be able to do some of
these polyhedrons, but the display technology right now simply isn't there. There are, however,
some developments out in research. Stretchable circuits/displays, this was recently announced,
so the ability to print onto stretchable materials means that you can potentially drape or stretch
these displays over 3D-printed objects. This is just a circuit, but you can assume that this would
be possible with displays. Sharp recently announced and showed a FOLED that actually is
cuttable in different shapes without cutting the circuits, because they are wired in a semirandom
pattern. I don't exactly understand how they did it, but here's an example of how they can do
very complicated forms. These are not three dimensional, but they're certainly interesting
shapes. And this summer, we did some work on 3D printing, in this case, circuits. So this is
called PrintPut, and we're using a carbon filament in order to print input devices right into the
3D-printed object. And so you see here a couple of things like a bent sensor. That's 3D printed,
buttons that are 3D printed, even touch inputs that are 3D printed, with the point of not having to
do these non-deformable operations on flexible input devices but printing that capability right
into the object. So here you see an example of that. The reason it looks slightly dirty is because
the printing process isn't perfect, and so carbon tends to be transferred to other parts of the
object. But it's relatively easy to print three-dimensional objects with, in this case, resistive
touch input. We're moving forward towards being able to properly print circuits and being able
to drop chips into 3D printed objects. Here, we see a recent development called a Voxel8
printer. It allows you to do just that, so it can print with a silver particle and do proper circuits.
It can also print with plastics, and it can integrate chips into that form factor. And so we're very
excited by this development, and they're available for preorders. I think they ship in the fall.
This is some work by a university in Saarland in Germany, where they are printing
electroluminescent displays on various objects. And here, you see it printed on a leaf of a plant.
And this is very, very promising, and we want to move towards displays that are printed on any
kind of 3D object. So what is lacking right now is displays of arbitrary 2D form. Sharp may
have addressed that, but I haven't seen them in my hands yet. These can be cut without severing
circuitry. They allow folding of displays around geometric objects of any type. We need
stretchable displays, displays that can be stretched over complex shapes, 3D printed displays for
arbitrary 3D forms, where the display is printed right onto the object. And, of course, we need
corresponding touch input technologies. Why do we need it? Being able to do this utilizes our
haptic system to a much greater extent than flat screens will ever do. Just the fact that you can
sense the shape of an object immediately and its orientation is something that allows for eyesfree operation, which is particularly critical in things like cars, that have been designed for such.
We get a flexible fit around the human body, as well as human activities, so you can change the
shape of your device according to what you're doing. We get real pixels on real objects in the
real world. Rather than simulating them in augmented or virtual reality, users can live in the real
world, and they can synchronize their real-world spatial abilities with objects in the real world
that have pixels on them. And it would allow for a plurality of shapes that fit a particular
function, so the idea is just like you have a staple for stapling, you would have a different
computational device for various activities that fit the task rather than this one brick fits all
approach that we have with the current iPhones, etc. And finally, we would have real
synesthesia between touch kinesthetic and sound, vision, producing the kind of interface that
currently you only see in dimensions of music. If you think about the violin as an input device
and how it feeds back to the body in terms of vibrations, it is just awesome, and we haven't, and
nobody at Microsoft, nobody anywhere, has even come close to developing an input device that
has this kind of synesthesia and is able to produce in this particular case an emotional transfer
between humans through remote touch through sound. And that's really an objective of ours.
All right. That's it. Thank you very much. I don't know if there's time for questions.
>> Ken Hinckley: Maybe like one or two quick ones. Awesome talk. Thank you.
>>: So the Voxel8, that's out of Jennifer Lewis's lab. We're a beta lab, Flight Sciences Group
here, and for that. I'm pretty psyched by it. Have you thought about -- first of all, do you have
one yet?
>> Roel Vertegaal: We preordered one. They're not out yet.
>>: So are we, but I've gone back and checked it out in the lab. But have you thought about
putting a pick in place with that and kind of really getting deep on putting components embedded
fully? Have you gone that way?
>> Roel Vertegaal: Yes, that is something that you can do with that printer. If you leave cavities
for components, you can put Arduinos or whatever the processor is you're using inside. We
haven't -- well, we've dabbled with it with the kind of PrintPut stuff that we've been doing. But,
ideally, you would also be able to print the processors themselves. But this is not work that we
can really realistically engage in ourselves, because we're an HCI lab.
>>: So the other question is you ship the glove. I think that's out of Purdue, right?
>> Roel Vertegaal: Yes.
>>: So yes, are you going to go down that track and kind of try to replicate or push that
envelope?
>> Roel Vertegaal: Yes, it's increasingly easy for us to print flexible circuits. We used to etch
them, but now we can print them, and printing them on flexible objects would be something that
->>: [Indiscernible] as I recall, a colored PDMS.
>> Roel Vertegaal: Yes, that's correct. Once it gets to lithography and those kind of processes,
it gets increasingly hard for us to do that in house. But certainly the DIY community has been of
great use for us, making it much easier to do hardware.
>>: Thanks.
>> Ken Hinckley: All right, so let's go head and thank Roel.
>> Michael McGuffin: All right, so I'm a professor at an engineering school in Montreal called
ETS, and two of the things I'm most interested in are interaction and visualization. So on the
interaction side, one thing that we've been doing is incorporating multitouch into our courses at
the undergraduate level, so we have a multimedia lab with several multitouch screens. We also
tablets, and so we get students at the undergraduate level to do projects on this. For example,
some students made the games that we see here. On the left, there's a slingshot game where you
stretch out an elastic and then let the stone go. On the right, you see a laser and mirrors game,
where you can use your fingers to reposition mirrors. We put together a simple real-time
strategy game that students can modify. We've incorporated music into the projects, so here
there is a student who made this entirely from scratch. It's a piece of software for remixing a
WAV file. And also recently, we've gotten more interested in pen input, at least I've gotten
interested in pen input. We started to acquire Cintiq displays, so we've added that to the lab.
And then the other topic, which we've pursued more on the research side, is visualization. And
as an example of that, one of my PhD students, [Maxim de Meux] is doing visualization of
financial data, and one of the projects he completed recently was a survey of over 80 papers that
proposed different ways of visualizing financial data, and there's a website where you can
interactively browse all of these papers and filter and search by keyword and so on. So today, I'll
show you two projects to go into more detail of recent stuff we've done. The first one is on the
visualization side, and here, the goal in this project was to find a way to visualize a database, or
more specifically, a table within a relational database. So if you have one table, you've got
multiple columns that correspond to different variables. How do you show a visual overview of
this table? So I'll show you some simple examples first. Here, I've got a very simple table with
two columns. I've got sales as a function of region, and I'll distinguish between two types of
variables. Some of the variables I'll call dimensions, so dimension is a variable that's
independent and it's categorical. The other types of variables will have our measures, so a
measure is a dependent variable that is also quantitative. So if you have one measure that's a
function of one dimension, often, you can show this very simply with a bar chart. That usually
works pretty well, or sometimes use a line chart, depending on the type of dimension you have.
Another simple case is if you have two dimensions, so here I have a single measure, sales, which
is a function of two dimensions. And one way to show an overview of that is with a heat map, so
the shades of red show the sales. And another simple case is if you have two measures -- here,
I've got sales and expenses, which are both functions of a single dimension. To see the two
measures visually, you can use a scatterplot. So these are simple cases. If you get a bigger table
with more columns, it gets more challenging. Here, I've got a fictitious example. Imagine a
company that manufactures nuts and bolts, and we've got data about the three locations where
they have factories, so there's three regions, numbered 0, 1 and 2. We've got data over 12
months. We've got two products, nuts and bolts 0 and 1. So I've got three regions times 12
months times two products. That's a total of 72 combinations, so I've got 72 rows in this table,
and for each row, I've got values for the sales and the costs of equipment and labor. So those
three columns on the right are my measures. They are the functions of the three columns on the
left, which are the three dimensions. So how do we show a visual overview of this data? There
are previous approaches we could try out, like the scatterplot matrix, which dates back to the
'70s. The idea here is that we take all our variables as columns, and we cross them with the same
variables as rows, we get a matrix, and then in each cell of that matrix, we show a scatterplot of
the two variables. So that works well if you've got measures, like in the lower right, you can see
here scatterplots of things like cost versus sales, and you can look for things like clusters,
outliers, correlations. It doesn't work so well at the top, in the top left, because here I've got
dimensions versus dimensions. For example, I've got month versus region, and what I end up
seeing is simply a grid of dots that shows me that all combinations occurred, and that's not really
very useful. So the scatterplot matrix works well with measures, but not so well with
dimensions. We see a similar problem with parallel coordinates, which is another way of
showing many variables at once. So the idea in parallel coordinates is every variable is shown as
a vertical axis. Every row in the table becomes a polygonal line that interests the axes at the
appropriate locations, and on the right, the three axes on the right are the three measures, and
there we can see some maybe trends in the data, or we get an idea of the distribution and the
outliers. But on the left, the left three axes, that's where we have the dimensions, and again, what
we're seeing is that all combinations of region, month and product occur, and that's not
particularly useful. So another approach we can see in the software product Tableau, which is a
popular piece of software for loading up a database and then allowing the user to explore the
data, Tableau, as far as I know, doesn't show any visual overview of a database when you first
open up a database. Instead, what it shows you is a list of all the variables it found. So those are
listed here on the left as dimensions and measures, and then the user can drag and drop those up
to the top to construct a sort of query, and that causes this tiled display to be generated with
slices of the data shown as charts. So we could call this approach dimensional stacking. That's
one of the terms that's used in the previous literature. So I took my nuts and bolts database and
applied dimensional stacking to it, and it gave these two examples here, where I'm showing four
variables at a time in each example. So on the left, for example, I've got the variables of product,
sales, month and region, so I'm not showing all six variables. I'm only showing four. On the
right is another possible combination of four variables. You can see on the right, I'm actually not
showing month. If I were to add month as a fifth variable on the right, the charts would get
much more -- the entire tabular view would get much larger. We'd increase by a factor of 12 the
amount of space we'd need. Here's another possible subset of four variables, and again, this is
already pretty big. So if we tried to show all six variables using this dimensional stacking
approach, it would get very big, so it doesn't scale so well when you've got lots of categorical
variables. So here's a summary of what I just showed you. On the left, we have simple cases.
On the right are the more challenging ones, and in the lower right, we see if you've got many
measures, you can use the scatterplot matrix or parallel coordinates, which I already showed you,
or you can use glyphs, which I didn't explain but probably many of you know glyphs already.
The more challenging case is if you have lots of dimensions in your data. If you have a few
dimensions, you can use dimensional stacking, like in Tableau, but if you have a lot, that's
something that has not really been addressed much in the previous literature, almost not at all.
So here's a new approach for dealing with many dimensions. You take your variables and you
cross them with themselves. You get a matrix, just like in the scatterplot matrix case. But now,
inside each cell of this matrix, we're not going to show a scatterplot. We're going to show a chart
that's chosen based on the type of variables that we have. So on the lower right, if we've got
measures crossed with measures, we use scatterplots. In the middle, we've got measures that are
a function of a dimension, so we can use bar charts for that, and in the top, we've got dimensions
versus dimensions. We can use heat maps to show those. So this idea is not something that we
weren't the first ones to come up with this idea. There is a little bit of previous literature on this,
but we were the first ones to develop an interactive prototype, and we were the first ones to
evaluate it experimentally with users. So here's the prototype that my former student JeanFrancois came up with, and this is showing us more than 100 million flights and the reasons that
they were delayed. So depending on the day of month or the weekday and so on, we can see
different types of delay. Notice that there's one category -- let's see, it's a distance group that's
been selected, so it's shown in blue here. So the user selected this, and one interesting thing is
that the distance group down here also corresponds to a distance group over here, so we can
show that with the sort of curved highlight through the heat maps and also through the bar charts.
Another thing that's done here is that when the user selected one category, in the other bar charts,
we highlight the fraction of the bars that corresponds to that selection. So the bar charts I have
over here, these are showing a sum for each weekday over all the distance groups, and if a
particular distance group has been selected, then the fraction that's in blue corresponds to that
distance group.
>>: So what does [indiscernible]. What does that mean, dissent group into the ->> Michael McGuffin: I don't remember. I'm sorry, I don't. It's some way of dividing up the
flights according to some criterion.
>>: Like long flight versus short flight or something?
>> Michael McGuffin: That sounds very plausible. Yes, so this idea of having the user select
something in one bar chart and then showing the fractional highlighting in another bar chart, that
itself is not new, but the bar charts we generated here, they use the sum aggregation. We can
also aggregate by averages, and that's what's done here, and here, we do have a new way of
showing the correspondence between different charts. So in the second column here in my bar
charts, these white bars are showing the average for each weekday over all distance groups, and
the user selected one particular distance group in the third column, and so the average
corresponding to that is shown with the blue dots here, so that's another -- in the paper, we call
this associative highlighting. All right, so we also did a little experimental validation, and we
found that our approach can be faster than Tableau for certain tasks, can be significantly faster.
So getting back to the topic of multitouch, another recent project is this multitouch radial menu,
which I'll show you. So imagine that you have a two-dimensional space within which you've got
some photos or images or documents, and you want to move them around with direct
manipulation. So you put your purple hand down here with one finger, you touch an image, and
then simply by dragging your hand down, you can reposition the image, right? With two fingers,
you can move your hands together to scale down this object and reposition it, so this is direct
manipulation. You can also -- in systems like this, you can often put a finger down on the
background, and then that gives you access to camera operations, like pan and zoom, so by
moving your finger, you can slide the entire scene sideways, or with two fingers on the
background, you can then move your hands together or further away to zoom, right? So in this
case, if I wanted to examine the details of this document, maybe I'd move my hands apart and
gradually zoom in, and I might get to this point where now the document's filling my screen, and
I can easily read the details. But now, if I want to get back to where I was before, I'm kind of
trapped. I no longer have access to the background. I can't access the camera operations, so I'm
forced to directly manipulate the document and scale it down. Then, there's some background
that's visible. I can then zoom out, but now my document's really small, and that's not what I
wanted to do. So how do we avoid this? Well, a really easy approach, a really simple thing to
do, is just add a toolbar, and we see that a lot, in a lot of software. Toolbars seem to work well
on smaller screens, like phone-sized screens, so very simply the user touches an icon to go into
the mode for that for camera operations or for direct manipulation, and then they can touch the
object they want to manipulate. But imagine if you had a really large screen, like maybe a wallsize display. Maybe you don't want to have toolbars in that case, because you don't want the user
reaching up, or you don't want lots of toolbars for different users cluttering up your display.
Maybe you also want to avoid having the user move back and forth between the toolbar and their
data, right? So there are actually several ways that have been proposed in the literature to avoid
having a toolbar, to allow the user to get to a command without having to move far away. And
often, this is done using some type of gesture, but most of the previous techniques that have been
proposed are designed for pen input or a mouse. There hasn't been as much done with
multitouch, and I'm going to show you a new way to do it with multitouch.
>>: In this interface, if I put down a single finger, there's a menu that pops up.
>> Michael McGuffin: I'll turn up my volume here.
>>: With commands I can select from. To manipulate an object here, a shape, there's a threestep process. First step, I put my finger down on the object I want to manipulate. Second step, I
select the command, which in this case is manipulate. Third step, my two fingers provide the
arguments to control that command. I can do all that with a single hand just by positioning my
fingers appropriately to grab the command in the menu. In the same menu, to the north, there's a
camera command which allows me to pan and zoom the entire view port. So it's just like in my
earlier scenario. I could zoom in on a single shape to examine some details, and now I'm not
trapped, like I was earlier. I can still zoom out, simply by getting to the appropriate command in
the menu. There's a few other commands in our menu. For example, there's an eastern item to
make a copy of an object. After copying, my two fingers can manipulate the copy to scale, rotate
and translate it. I can also change the color of a shape, so I can give this shape more green, give
this one more blue, and in the menu to the east -- sorry, to the west, there's a lasso command,
which allows us to select multiple shapes. A few things to note about the menu -- in the center,
there's actually a default command. You can see the name of it there is ink, and the way the user
invokes that command is to simply press and drag. This allows the user to create shapes quickly.
A second thing to note is that let's say I wanted to manipulate this shape over here. A natural
thing would be to position my fingers this way, but the way our menu is designed, you actually
have to position your fingers this way first, so first, you finger down on the shape, second, a
finger down here. And this might seem like an odd positioning of the fingers, but once you're in
this mode, you're actually free to reposition your fingers. As long as you keep one finger in
contact with the screen, you stay in this manipulate mode, and you can reposition your fingers.
The same thing with the camera command. Once you're in camera mode, you're free to swap
fingers and reposition them, and you remain in that mode. Finally, because we can distinguish
between finger and stylus, you can double the number of commands accessible, so for example,
finger could launch one menu, and then stylus could be used to complete the command, and the
stylus could launch a different menu, which could double the number of commands available to
you. Here's a second version of our solution. Now, in the menu, the items have been duplicated,
so that the menu is now a symmetrical. You can see, in the north and the south, we have the
same camera command. In the east and west, we have the same lasso command, and in the
northeast and southwest, we have the same manipulate command. This allows the user to select
commands faster, because they don't need to be careful about which finger touches first. For
example, to get to the camera command, it doesn't matter if my thumb touches first or if my
index finger touches first. The result is the same. I just need to be careful about the orientation
of the line connecting my fingers -- in this case, vertical to get to the camera command. Let's say
we want to draw a house, so we add a little roof here. We can add a side wall. Now, if I open up
the menu, of course, to get to a command, I press on the corresponding pie slice. I can also press
outside the menu, and that allows me to invoke the ink command on each of my fingers. So I
can easily draw some grass or add a little forest here, and then with the lasso command, one way
I can lasso select is to encircle the strokes that I'm interested in, like that. Now that they're
selected, I can go grab the manipulate command in the menu, and so with two fingers, I can scale
or rotate my selection. Another way I can use the lasso command is to simply drag out a
rectangle, so for example, I can drag out this rectangle, and then the system selects whatever is in
that rectangle, like that. So the system is smart enough to infer whether I'm encircling things I
want to select or I'm dragging out the diagonal of a rectangle. It figures that out by looking at the
shape of my stroke. Another thing to note is that the north and south items here, the camera pie
slices, are much more extensive radially than the other ones. That's because of the way a user
uses the camera command. If they want to zoom in, they'll naturally start with their fingertips
close together and then spread them apart. But if they want to zoom out, it's natural to start with
fingertips far apart, so we made the pie slices for camera log enough radially to capture distant
fingers in a command like that. And then finally, you'll see in the menu, there's a more
command. That's to open up a sub-menu, and in there, there's options to perform other
operations, like I can change the color of my selection, so the grass I selected earlier, I can make
that grass green. I could select my house and then change it to be let's say purple. I could also
make a copy of the house and with another finger, I could scale down that copy. I could flip it
from left to right, make another copy, change the color of that, and so by holding open the submenu here, I can invoke multiple commands on the same selection.
>> Michael McGuffin: All right, so let's come back here. Okay, so let me show you a couple
examples of previous work, which also allows the user to access commands, often with sort of
gestural interaction. Scriboli was developed here at MSR, and it's designed for pen input. And it
combines several things into a single gesture. So the user starts off by drawing a lasso selection,
to select some ink that they want to change. Then they perform a pigtail gesture over here. That
transitions them into a radial menu, a type of radial menu called a control menu, and then they
can select a command, and then they can keep dragging past that command to provide the
argument that goes with that command. So this combines lasso selection, command and
argument all into a single drag, and it works great for pen input, but it's not designed to support
multitouch gestures like the sort of pinch gesture, where you'd have four degrees of freedom
when you're using it. Another example from previous work is the Toolglass, a very elegant
technique. Usually, it uses two mice. So with your non-dominant hand, you move around a tool
palette. With your dominant hand, you control a cursor, and you can click through tool palette to
select a command and apply it to an object underneath or apply it to a canvas underneath, so this
combines the selection of objects and commands into a single click. Works great if you have
two mice. We tried this on a multitouch display, and actually, it didn't work well, somewhat to
our surprise. Part of the reason is that most multitouch displays don't support hover, and this
technique works a lot better if you do have hover. Another problem is the fat finger problem,
which hides what's under the fingers.
>>: [Indiscernible].
>> Michael McGuffin: So if you take your finger and you push on a command in the tool
palette, your finger hides what's going on underneath, right? So one way to get around that fatfinger problem is to say, yes, I'm going to have an offset between the cursor and my finger. And
if you don't have detection of hover, you have to press before you see the cursor, and then you
end up having to drag your finger across the screen. And we found that this felt very unnatural
and slow. So this is two of many techniques from the previous literature. Here, we've got a table
comparing a whole bunch of them, all the ones that we thought were relevant. Every technique
is a column, so Scriboli is the second column here. Toolglass is this column, and then our new
technique is the rightmost column. And then in the rows, we have different criteria, and these
have been phrased in such a way that if you can answer yes to a criterion, it's a positive thing.
It's desirable, right? So the more yeses you have in a column, the more green you've got, the
better it is, or at least that's our theoretical argument. So design is all about tradeoffs, right? I
think I read that in Jenny Preece's HCI textbook. There's no perfect solution here, but look at our
column on the right. It is mostly green. And if I were to sum up with distinguishes our
technique from the previous ones, it's that we support multitouch gestures with four degrees of
freedom, and Toolglass can also support four degrees of freedom, but it doesn't work well on a
multitouch display. So thanks to the people who made this research possible and to our
collaborators. I think there might be time for questions.
>>: [Indiscernible] anyone want to ask any?
>> Michael McGuffin: Yes.
>>: Each time, you present your menu thing, you seem so -- it seems -- the idea seems really
hard, but then when you actually play with it, so when you demo it, I'm always blown away by
how you can make all of these things happen.
>> Michael McGuffin: Actually, I was very discouraged, because every time I showed the menu
to anyone, it took them at least 10 minutes to understand how it works. But when ->>: Here's my question. Did you have other people achieve the same fluidity that you're
manipulating it? Did you actually ask people to use that menu and then see how they ->> Michael McGuffin: No, and we have an experiment planned coming up, where we'll get a
chance to find that out.
>>: Because it reminds me of this hot box you had presented, which is crazy menu, but then
when you -- it's like magical. I go, I've got to [indiscernible] those things, but could I do it? So
that's -- yes. It would be interesting to see.
>> Michael McGuffin: Yes, that's an important issue, especially for commercialization.
>>: What do you think is the cognitive thing that people have to get over to grok it?
>> Michael McGuffin: Well, one problem is that when people put their finger down and they
see a radial menu, they immediately think, oh, let's drag in the direction of the command, but
actually, you don't want to do this in this menu. And then people end up doing strange things
with their hands, like they move their hands in very awkward poses, because they think they
have to maintain touches when they don't have to. But because I'm the one who programmed it,
I know exactly what to do, and it's -- yes. So that's an issue.
>>: Okay. Excellent presentation.
>> Michael McGuffin: Thank you.
>> Pourang Irani: Okay, great. Hi, everyone. Thanks for the invitation to come here. So unlike
the other two talks here, my talk is going to be much more focused. I think Ken advertised it as
being something secret. It's far from anything secret. It was something that was actually
published and presented at CHI last year. I think the only reason why it's secret, is because it
was actually secret to myself until last night, or two days ago. So sort of last minute planning on
this end, but at the same time, not to say that this is not an important topic. In fact, it's probably I
think one of the more important topics that we are looking at currently, because it gets us to
rethink a lot about how we are designing and how we are thinking about natural user interfaces at
large. So in here, I want to present work done by students in the lab, Juan David, postdoc, Xiang
and Paymahn, who were undergrad interns at the time, working with Juan. And I'm at the
University of Manitoba, as some of you know. So I want to talk about what we refer to as
consumed endurance, a metric to measure arm fatigue. Here, I'm going to talk about what that
metric is, look at consumed endurance as a measurement tool, how we can use it to actually
measure and assess it, and finally, present an example where we talk about or showcase the use
of consumed endurance as a design tool, as a way to look at new designs. So needless to say, I
think for most of us in the crowd here, we know about natural interactions, and is this turning
on? Why is my video not working?
>>: Interfaces facilitate natural ->> Pourang Irani: Sorry.
>>: Air interfaces facilitate natural user interactions as user ->> Pourang Irani: Sorry, guys.
>>: Users freely move their arms in space to interact with the computing system. However, a
frequent complaint among users of such systems is the feeling of heaviness and tiredness of the
arm, a condition colloquially termed as the gorilla arm effect.
>> Pourang Irani: So this is probably something that we are very familiar with, not only
interacting with systems, but also interacting on a daily basis by picking objects, maintaining
your arms in the air and so on and so forth. So we are familiar with this. In fact, this was
something that appeared, crept up in the '80s, when touch systems came out. And in fact, it was
one of the reasons that led to the demise of these systems, that as these touch systems evolved,
particularly as the sensing techniques on these systems, sensing methods, required users to keep
interacting with them for longer periods to get an action completed, it required people to do it
longer, inducing fatigue, and ultimately, the experience for interacting with these systems was
diminished. So the question is, we know that this is the case. If I were to ask you, well, let's try
to hold our hands up for a minute or less than a minute, how long do you think you could do
things, actions, such as maybe panning an interface, maybe spiraling to zoom in or zoom out.
What would you guess? Roughly, how long do you think? Two minutes, at this distance over
here? So you are almost close, and this is the kind of answer we like to get with the kind of
system of where we are going at with this work, is try to find out how long can we actually
sustain certain natural interactions, such as mid-air input, particularly, and how can we actually
reduce the impact on fatigue? So, of course, this has been done before throughout the systems,
so you can look at them as being more or less quantitative assessment tools, and there's a lot of
them in the literature looking at electromyograms, looking at blood pressure and heartbeat. Also
looking at blood samples and even more obtrusive methods, where you actually can stick needles
into different muscles and be able to assess the level of exertion that the muscle is applying to a
certain action. However, if you think about natural user interfaces, you may say that it's a bit of
an overkill, particularly since most of us don't actually have this kind of equipment in our labs.
But not only that, it's actually very obtrusive. I can't imagine myself -- I mean, to some extent,
maybe something like this, but anything more obtrusive than this to get precise or somewhat
adequate measurements can be difficult, such as blood samples and whatnot. We can use
qualitative measurement tools, and this has been used a lot in the literature in HCI, particularly.
So Likert scale surveys, system usability questionnaires. The NASA TLX is something that's
actually used a lot in HCI papers, and one that's used in HCI but also coming from other
communities is the Borg CR10, which is considered to be one of the more reliable metrics for
fatigue, and they use it in medical scenarios and so on and so forth. However, these are a set of
qualitative assessment tools, so they can be subjective according to the mood of the participant.
They happen after the interactions take place, so as a means of designing a system and being able
to capture it throughout the interactions, they are somewhat limited. So the question we had had
is how to measure arm fatigue in a sort of unobtrusive and hopefully somewhat of a more
quantitative, objective manner? To do this, we looked at something that was known, that came
out in the '60s by Rohmert and has actually been enriched ever since. In one of the sort of more
interactive books in the area of biomechanics that came out about 10 years ago, it's been, again,
re-featured. And Rohmert talks about something referred to as the endurance time, so the
maximum amount of time a muscle can maintain an exertion level before needing rest. So what
you have on the X-axis here is the maximum amount of force that's being applied at any given
point in time for a particular action, and that maximum percentage of the force you can apply for
that action, the amount of the maximum force being applied, triggers what's referred to as the
endurance time, meaning this is how long you can maintain that action before you actually need
to rest or you need your muscle to rest. So if you look at something to this effect, where you're
exerting a force of less than, say, 15%, and if you notice, the graph is actually asymptoted at
15%, and this has been verified from multiple studies. So anything below 15%, so if your arm is
at rest, which is typically the range where you're really not exerting much muscles, and I should
specify, you're looking at -- when I say arm muscles, I'm looking potentially at the shoulder, the
shoulder muscles happening or the shoulder, because we kind of consider the entire hand as
being one unit. So you can actually maintain this position for a long period of time, and that's
because you're exerting a maximum percentage of force under that 15% ratio. Something at 45
degrees, you can sort of maintain this at about 300 seconds, and I think [indiscernible] mentioned
something that if you kept your hands up straight -- for instance, this is sort of at the point of
about 75 seconds. So just slightly over one minute to be able to leave your hand in this position,
interacting with objects in space. So from this result, we basically derive what we called an -and there are more details for these equations in the paper. We have about three or four
appendices in the paper itself that you can refer to to get the details on this, but this is sort of the
brief idea for the consumed endurance metric, is we look at the interaction time -- that is, the
length of time it takes you to do an action, over the endurance time. The endurance time was the
amount of time that you can use before you have to actually rest your arm. And if you look at it
as a ratio, so a percentage of that interaction time. So you're kind of bound to some extent by
endurance time in this. Yes.
>>: Is the last chart implying that if you hold at 100 degrees, that you will not be able to hold the
arm at all?
>> Pourang Irani: No. All it means is that -- so it all depends, and I'll talk about this a little bit
more, but it depends on the forces that are being applied to the arm at the moment that you're
holding it here. You can of course hold it higher than that, but then what that means is it could
be anywhere in this chart. In fact, holding it at a certain level, even higher than 90 degrees,
doesn't necessarily mean that you're actually down here. This is the percent of the actual force.
It's not actually -- so this is not the degree at which you're holding your arm. It's the percentage
of the force being applied at that.
>>: But can you assume this graph is pre-given, because it could be changing due to small
variation in the motion?
>> Pourang Irani: Yes. Actually, I don't have the equation here, but this formula is actually a
given, and this is what Rohmert actually is attributed to having derived from his studies. There
are constants in this function that change, and they change according primarily due to the
physiology of the users or the people involved. So much of my data here that I'll talk about, it's
at the 50th percentile bodyweight and length, body arm length and mass weight. But, of course,
they vary slightly according to the masses.
>>: Let's say you hold a hand straight or you hold a hand straight and with small fluctuations.
It's a different.
>> Pourang Irani: Of course, yes. No doubt. What we are doing is we are actually accumulating
the gestures, and I'll show you in a minute here. We are accruing the force applied on the limb
throughout the interaction process, and as you do the entire accumulation of it, then we derive
the consumed endurance. So it's not that we are looking at the consumed endurance in a static
point. We are looking at it over the length of a gesture, and as you're moving your hand -- for
instance, you could be at a certain consumed endurance level here, but if you start moving it
here, it goes lower, and we accrue that to specify what the -- yes. So, basically, this is our
system here.
>>: We implemented CE using a Microsoft Kinect as the tracking system. To calculate
consumed endurance in real time, our implementation follows the arm as it moves in three
dimensions and estimates the forces involved at any given point of time using standard bodily
measures of arm length and weight. For instance, a user can hold his arm for 70 seconds at 90
degrees, and after 10 seconds, the consumed endurance of this position is 12%. This means that
the user can sustain their arm in this position for a little over one minute before needing rest.
Similarly, a user can hold his arm at 45 degrees for 130 seconds, and after 10 seconds, the
consumed endurance for this position is 8%.
>> Pourang Irani: Notice that this person is keeping their hand here, but these endurance values
are constantly changing because of the tracking information that we are getting there. And
according to the tracking information, of course, you can get 100%, even though the person is
actually holding or they might be moving it slightly. But it's mostly due to the slight noise in
tracking.
>>: So doesn't that change over time? So at the beginning, you may be able to do this, but after
an hour of contracting, then you've caught ->> Pourang Irani: Exactly, so this is what I was saying. The endurance that we compute is
accumulated over the entire period of the interaction. In this first phase of the study -- we had
three parts. The first one, we were just looking at just simply validating this metric. So we
asked people, hold your hand there as well as you can, and we will try to compute consumed
endurance for that value, and we'll do it for the 90 degree, as well as the 0 degree, and we
actually validated against the Borg CR10 metrics to see whether we actually have fallen in line
with what people subjectively reported at these gestures. So for this first validation, they weren't
doing any interactions here nor here. They were just simply holding their hand, and we were just
simply computing both the endurance value, and of course, we have the interaction time, which
we consider interaction time, and the ratio of these gives us the consumed endurance metric over
there.
>>: Means that the user can sustain their arm in this position for a little over two minutes before
needing rest. Finally, a user can hold his arm for an undefined amount of time when the arm is at
rest.
>> Pourang Irani: So going into some of the details of how we compute this, we kind of
estimate the entire limit as being unit, so we have these different joints. We estimate the center
of mass for each of these values over here. The center of mass for the forearm is at B, and for
the arm, the upper arm, is at A. And we compute the center of masses this way, and basically,
this is basically computed by the -- of course, we have gravity acceleration helping us, so it's
fairly easy to compute the forces applied, and we look at the general forces then through all these
points ultimately to compute what are the forces applied at this joint throughout an entire motion,
because you have the entire set of forces being applied there. We also use, as I mentioned, the
50th percentile for all our studies. There are ways to actually compute it for per individual basis,
so there have been studies looking at it, for, for instance, the length of a person, the height of the
person, the height of the limbs, can give you an estimation of what these values would be, and
you can easily do this with some of the tracking systems. For our initial validation, we didn't
want to introduce that noise in the metric, so we actually stayed away from that. However, they
are different for male and female, as you notice that, and we used that accordingly for the
different participants, either male or female, that we had in the validation. So is CE a valid
metric for arm fatigue? And indeed, in the regression, we actually see it's a very strong
correlation to the Borg CR10, which was basically the person maintained their arm there, and
then at the end of it, and they did this over multiple trials, they were asked to rate on the Borg
CR10 scale, which is not a linear scale, by the way. It's a very odd scale, if I may say, but one
that has been used reliably for the last three or four decades. And, basically, we see a fairly good
fit and regression to the Borg CR10. These were students, so university students, were our
participants. They were half male and half female, and this is an important distinction to make
here, because as I said, there is a lot that comes into effect when you're looking at the physiology
of the participants. So we stuck to the 50th percentile metrics, which is what I showed you
earlier. So we moved past the first step to show that it was potentially -- it's a potentially valid
metric in comparison to the Borg CR10, which I said was a qualitative metric. And next we
looked at can it be used as a way to measure fatigue in interactions, so we looked at it, and we
said, well, when you're doing natural interactions, there are many things that can happen. We
looked at multiple factors. We had two experiments. In the first one, we looked at the idea of
plane location, so are you interacting in this plane over here versus this plane over here? And if
you think about lots of our interactions in natural user interfaces, particularly with systems that
are maybe head-mounted displays that have a tracking system on your body or on your chest,
you might be working in this space over here, so you have sort of the shoulder height. We also
looked at your arm being extended or bent, so you could have your arm bent and interacting still
around this plane or lower over here, which was the arm bent and the center plane. In
experiment two, we looked at other factors, such as the size of the plane, so small or big, and
different selection methods that could induce fatigue assessed you interacted with your natural
input. So the experiment looked something -- I don't think there is a video here. So basically, it
was a number of targets at fixed size in the first experiment, and at different lengths, so
something happened to appear in red. We had only a mouse click, so we had the participant hold
the mouse in the left hand, so when they got to that point, so basically, the clicking did not
induce any extra fatigue in this, because we were simply interested in movement, and it kept on
going like this for a large number of trials. So 12 participants, mean age of 26 years old, and
what we see indeed is significant differences in the different conditions, where the center bent
condition induces the least amount of fatigue, the shoulder bent is the second, and of course, the
straight over here and then this one here, so this condition here being the worst, as we would
expect, however, with some quantitative attributes assigned to that judgment. So this is the kind
of position. So when you're thinking about interaction, this is an interesting application for
natural user interfaces, is do we really think about natural user interfaces as something where we
interact in this space. It can be very tricky. It really defines how we have to really think about
where we put our trackers, how we put them, how we orient them, so that if you're interacting,
particularly with things like head-worn displays, how do we go about doing things not, for
instance, in this space over here, but particularly if it's a modeling, 3D modeling task where
you're putting something together, how do we rethink that and how do we go about -- so it really
gets us to think about how do we shape this whole space, if you want to really extend this
interaction, user experience in this interaction. So continuing on with experiment two, we
looked at different selection methods, so I think I have a video for those over here.
>>: We evaluated the following selection methods, click.
>> Pourang Irani: Sort of a baseline.
>>: Second arm.
>> Pourang Irani: I think someone mentioned that, but I think that's a bit awkward.
>>: Dwell. Swipe.
>> Pourang Irani: So very similar setup, not very complicated. Again, a four-by-two
experimental design with four different selection techniques, a big and a small plane, so small
plane of 25 by 25 or 35 by 35 I'd like to say centimeters, but I'll have to confirm that. Fairly
younger. Mean age on this set of participants, there were again 12, and here, what we see is -so, of course, the second-handed techniques induced less fatigue, because we are measuring
fatigue primarily on the dominant shoulder, where the actions are taking place. Of course, we
see here the larger the plane, so the larger the plane of interaction, the more fatigue is being
induced, because you're potentially moving -- you're having bigger motions in there, both this
way and this way. And when you're doing it with a single technique versus like a swipe or a
dwell, the dwell actually winds up doing not that badly, which was a bit surprising to us, because
you're actually holding your arm there for maybe 100, 200 milliseconds longer, but yet, if you
think about a swipe, to be able to register that. So there may be ways of doing this a little bit
better, like for instance maybe push gestures and so on and so forth and other ways of doing it
that could be considered. But this is what it gets you to think about. So again, ideal position is
center bent with a smaller plane and something like a dwelling selection. So here, we see
through these two experiments, CE can effectively be used as a way to measure how certain
interactive techniques that have been used in the past perform. So the next thing is, can we use
CE to help us design new techniques, and this was what the last goal of the paper was, and we
looked at it for the case of looking at redesigning mid-air text entry. So if you think about this
grid as your grid for entering text with your QWERTY characters, what we looked at was where
in the cells on this grid, with the center-bent position, and we used the larger grid because it had
fewer errors in the performance of the gestures, do we did use the larger one here. But you look
at the distribution of the forces applied, and you see that of course the higher you move your
hand, the amount of force that you need to apply to maintain those gestures -- for a period of
about 10 seconds -- we assume that you need 10 seconds to go to that place and do a dwell or
move to that location, give us this kind of distribution, which makes you rethink the complete
layout of your keyboard. And so this led to -- for instance, this is what we call the SEATO -- or
SEATO, I don't know how you want to say it. But it's not your QWERTY keyboard, and these
are your most frequent character counts, with the other ones here. Your enter and the backspace
are given more space, both for accuracy, but also for positional importance, and then things that
are less important are given in areas of higher demand. So we did this with the QWERTY and
the mean age of about 24 years old, and what we actually see is that the SEATO layout actually
induces less fatigue. It's interesting that we actually see this difference over multiple blocks,
which you kind of see with the QWERTY. The QWERTY levels out. After four blocks,
possibly, people are starting to get tired, which you may see as a spike over here. I wish we had
done maybe five or six blocks, but I think that was extending the time for the experiment, and it
would have been interesting to actually see where it goes back up there. I'm guessing -- my
guess, my hunch, would have been that it actually goes up a little bit more. So this is an example
of showing how you can think about redesigning an interaction technique -- in this case,
inputting text using these systems with CE.
>>: [Indiscernible] time, though?
>> Pourang Irani: Yes, I have a bar for that.
>>: Is that similar?
>> Pourang Irani: Yes, no statistical differences, but there is a bit of twist to that story. There
were more errors in the SEATO layout, I am guessing because of the unfamiliarity with the
layout, because you see it in the block for kind of leveling out. But despite that, the consumed
endurance was still less, so it made me think of things like improving performance by training
and so on and so forth. I'll show you a graph at the end that discusses. So what are some
implications? If you're designing menus for natural user interfaces, think about them in the
lower region, directly from these results. But you could think about actually evaluating them.
We talk about natural user interfaces using direct input, but indirect input can actually be quite
interesting, because indirect input, your hand can be here. Your hand can be in your pocket, can
be actually lower over here, and of course, that breaks away the model of a natural kind of
natural input interface, natural user interface, in the sense that you what to move as in the real
space. But if you're really working with a real system, do you really want to be having your
hands in the air for extended periods of time, or do you want to use those for very specific
actions like dropping an object here and doing all your modeling in the lower space here, instead
of having it up here. So it makes you really think about these things. We have a workbench.
You can actually download this and use it to test your own designs. You can actually integrate
this with your code. This shows an example of how you manually capture that, so you have a
start and stop button. The red ball there is showing you the amount of consumed endurance at
the shoulder length at any given point in time, so it grows and shrinks according to that motion,
and you can start and stop for multiple trials, so depending on what you're doing, and you can
integrate that into your code. The libraries are made available to you, so you could use this as
well, and you can download it from here if you're interested. And, of course, you can ask us if
you have any questions. We are still maintaining the code and improving it, so if you have
questions, you can ask us in terms of providing you that. So some takeaway points. We
introduced CE to estimate arm fatigue, and we have validated against sort of a well-known
metric in the literature, the Borg CR10, and show fairly good regression to a -- response to that
metric. We found that the center-bent position for 2D selection in the plane seemed to be ideal.
Dwell is a reasonably good selection technique, and it can be useful in making new designs, such
as the SEATO layout for things like text entry, but potentially you could think of other
techniques involving natural user interfaces that need to be used for this. The larger question is
how do we rethink NUIs in consideration of fatigue, and the user experience not only involves
the visuals, the interaction techniques themselves, but the element of do I really want to do this,
and is it going to be something that I'm comfortable doing? We looked at it primarily with
participants under the age of 30 or so, but as you start getting up there, of course, you can
imagine how this is even more of a concern and more of a problem. So thank you for your
attention, thanks for the invitation here, and hopefully, it's not as much of a secret anymore as it
was planned to be. Thanks. Questions?
>>: I said something about input equals output, and this kind of negates that in the sense that
there are many reasons why we'd want an indirect input device. Do you have any comments on
that, like to what extent? Because this also applies to desktop situations, where people
experience fatigue, for example, through tilting their heads, not just through arm movement, but
all sorts of other things. And I can sort of see this become a quantified self thing, where you
wouldn't use a Kinect, but you'd use eye views or something like that on the body in order to
continuously monitor position. Do you have any thoughts? To what extent -- so the question is
twofold.
>> Pourang Irani: Yes, can you repeat those?
>>: Sorry?
>> Pourang Irani: Can you repeat the two questions?
>>: Yes, so firstly, to what extend will we continue to use indirect input to avoid fatigue in cases
where the display needs to be elevated, for example? And the second is to what extend have you
considered using on-the-body measurement instead of Kinect?
>> Pourang Irani: I wish I could predict the future to say how long we'll continue using indirect
input. I think we still will. I think indirect input has a lot going for it. If you're thinking about
interacting with a large surface like this over here for an extended period of time, really, direct
input really starts getting troublesome. And people have looked at this in the last 10 years or so
about things like the hybrid cursor, how to move from one point to the other or how to transform
an input device that's direct and make it an indirect input device. So there have been
considerations and some thought put into using indirect input devices. I think that's important. I
think the mouse, even Engelbart said that he didn't expect the mouse to live so long, and I think
one of the reasons why it's lived so long is that because it's actually resting. You're resting your
hand on the desk. You're not actually having to move your hand in air and do this thing, because
if you were, I bet you actually -- someone would come up with a better tool, a better design to
actually use this, interact with the system. So I think this kind of notion is ingrained in designers'
minds going way back, and the idea of coming up with suitable designs. And the use of indirect
devices actually worked very nicely with this concept of diminishing the fatigue that's induced
by natural actions, which is something that we don't do for sustained periods of time, even in the
physical world. We may pick an object and put it there, but hands generally go down, or we may
sit down. In terms of using IMUs and other bodily sensors for interacting, sure. For certain
applications, you can. Gaming, like the Kinect for gaming, is great. You can move around. You
can interact with the system. Maybe fatigue is actually part of the whole gameplay, where the
person starts getting tired, and therefore your opponent can actually take them down or whatever
it may be. So entire body sensors, sure. But in terms of diminishing fatigue and looking at that,
I don't really have a clear answer per se. Maybe you have some thoughts we can share later, but
I don't know.
>>: I thought it would be kind of cool to have this quantified self-monitoring system that would
do this all the time.
>> Pourang Irani: Yes, to tell us how ->>: The Kinect is not very appropriate for that, because it needs an instrumented room,
essentially.
>> Pourang Irani: Right, to do this kind of stuff. Yes. We got away with a lot doing with just
the Kinect.
>>: But you can't walk it outside or [indiscernible] or lying in bed and whatever it might be that
you're doing.
>> Pourang Irani: Yes, but then it becomes more complex, because it's not just simply one limb.
There's a whole body. There's things like climate, temperature, your own conditions and
everything else that comes in, so that's another whole space, I guess.
>>: It's interesting, because all of those videos people with media interaction, they're always
standing, but they don't quite do anything with their feet or anything. So I wonder if there is a
difference if actually you do the same thing sitting? Because then you can hold your hand and
do lots of things. So why do people do it with folks standing instead of just trying to see where
you're at your office and you're sitting?
>> Pourang Irani: I think standing has this idea that you're working in a larger collaborative
space, or you're dealing with -- you're walking, maybe a presentation, where you have three or
four people, or a gaming system in a media room at home. I definitely think if you're doing
swipe on your computer and you're typing here, you may swipe a little bit here, but after a
certain period of time, if you can, if it's convenient, you're actually going to start resting your
arm to do stuff, the same as we did some studies with Ken last year, looking at how people
interact with Surfaces and so on. And the idea of resting their arms came out quite a bit, and
basically the postures, the grips that they would use for holding the Surface, or the grips or how
they would hold, leave their hand on the tablet, so that it's not -- it's something that's comfortable
for them while they're reading something or while they're flicking or scrolling or something to
that effect. So I think it just depends on the task. It depends on the environment. If you're
collaborating with other people, you may need to stand in these kinds of systems. Of course, if
you're alone, sitting is probably the thing you would want to do, and you would probably do it
after a period of time anyway. But whether you're sitting -- and I guess when you're sitting, this
part of your body doesn't move much, and you're kind of in already a comfortable position, so
that's probably why we also can use a mobile phone for extended periods of time, because you're
not kind of doing this kind of stuff. And on top of that, you actually have -- so we didn't have
any support systems here, but you could think of how to extend this model with what if you had
a Surface or a tablet, and your hand had some moments of rest. What would that do to the
forces, because you'd have ->>: The ground, I think [indiscernible] instead of something different.
>> Pourang Irani: The floor, yes. I think for some things, definitely, the floor is interesting.
>> Ken Hinckley: Yes, thanks, Pourang. Let's move to our last talk just to stay on schedule.
>> Pourang Irani: Yes, thank you.
>> Ken Hinckley: All right, thanks, Shen.
>> Shendong Zhao: Okay, thank you. It's great to be back, and thanks for having me here. My
name is Shen, and I'm an assistant professor in the Computer Science Department of National
University of Singapore, so this talk, I want to share with you some initial thoughts I have on
innovation patterns in HCI. So we are in the age of innovation, with globalization and digital
revolution, now a lot of people can actually perform a lot of innovations to do a lot of interesting
things today, and countries usually are very emphasized on innovation, because they think
innovation is the driver of the economy. So the question is, how can we actually more
effectively perform innovation? So this is a very interesting question, and I'm looking to some
other disciplines in computer science. In software engineering, there are books about design
patterns, which are reusable elements which occurs over and over again, which you can kind of
put in the hands of engineers. And this is the 23 patterns coming up with by the Gang of Four
book, and many of these patterns are being commonly used today, and a lot of software
engineers actually learn about that. So I'm looking into innovation HCI as researchers, it doesn't
seem like we have such tools, so this makes me thinking, it might be interesting to actually
explore this area and come up with something hopefully similar. Another thing, motivation I
have, is that we are at the age of information overloading, as so many things. If you're just
looking at the conference that we often submit to, CHI, and you can see the number of growth in
terms of number of submissions and number of papers we accepted, and if you look at the
numbers, there's already 18,000 papers being submitted and about 4,000 papers being accepted.
I used to do this kind of reading lab to kind of go over every year's CHI papers, but now it's
almost impossible, because it's 400 papers. Even if you have a group of people reading it, it's
going to take a few months, which is too time consuming. And I'm teaching a class, a graduatelevel class, which is kind of a seminar class about different topics in HCI, and I have a problem
of finding out what are the proper topics, just a subset of very nice topics to teach students so that
they can learn about HCI research. Instead of more domain, this is about crossover thing, this is
about ubiquitous computing, which can change over time. I want something that's more
universal, so this is the second motivation I wanted to think about this topic, although this should
be probably a topic that somebody much more senior to think about, but I was really curious
about it, so I had my initial shot about this. So what are innovation patterns? Let me just give
you some analogies of the definition that I learned from the design patterns from the software
engineering community. Well, a design pattern is actually a generally reusable solution to a
commonly occurring problem with a given text of software design. So based on that, I come up
with my definition of innovation pattern, which is a generally reusable approach -- so you can
see the difference here, I'm not really a solution. It's an approach to find novel and better
solutions to accomplish an HCI task, so a human computing task, so this is all driven by coming
up with better solutions and better things and what kind of approach they can use. So to just give
you a quick preview of some of the things that I've been thinking about and seems to be
recurring in the past in the 10 years of my own research, short career. You can see some of the
first ones I see is finding alternatives. Then we have reuse patterns, and you can invent based on
a theory, balance humans and computers, enhance directness, explore concurrency complement,
facilitate transition and define appropriate chunks and timing. So these are the things I've been
thinking which is always recurring, and if you're looking at this list, of course, this initial list, this
list, the top three is actually kind of universal. We all do that. I think even in other disciplines,
people are looking for alternatives, looking for patterns, toolkits and using theories to drive
innovations. The next four is -- I think is more unique to the area of human-computer
interactions, and I've been using them unconsciously in my own research quite a lot. So this talk,
actually, I will be very much focusing on the next four, the four patterns here that I think is more
interesting. And the last one I listed there, which I think it's also a pattern, but I don't have very
good examples yet, so I will just throw it out for discussion. So what this innovation is for? I
think innovation patterns is for HCI researchers and for innovators, somebody who wanted to go
and -- it's not really for people who are designers, not really for people who use the interface -for example, testers. It's for people who wanted to innovate and doing better things and better
solutions. And the question is that when do you want to use them? I think it's at the strategy
planning stage, where you ask the question, how can I achieve a better performance for a
particular task? And then you can look into this laundry list of strategies and maybe pick things
that can apply and hopefully help you to better come up with some interesting solutions. And
also, it can be useful when you actually try explaining your solution. Sometimes, you come up
with something and it works really well, but how do I explain it so that people think this is not
just a random invention. It can also maybe use the innovation patterns for that. All right, so I'm
just going to go through the four things, four patterns, in a little bit more detail in this talk, and I
think that's all the time we have, as well. The first pattern, I'll call it balanced humans and
computers. So what it is, is what we're actually looking at solving problems and accomplishing
goals for people. We come up with solutions that involves some computing tasks and some
human tasks. This is basically all the solutions we will have to involving those components and
the interaction in between. A lot of times, we're looking at things a little bit separately. We're
looking at how do you come up with a better algorithm, maybe a more faster database or better
networking, so these are just for computing tasks, optimizing computing tasks. Or maybe we're
just looking at interfaces itself, how do you come up with a faster menu and things like that? But
I think it's important to look at a little bit of the high level, so if I want to complete a task, it has
sub-components, how do I assign the different components or different sub-tasks to the
appropriate agent to complete, so that the entire system is optimized? So this comes into how do
you attribute the right task to the right agent, whether it's a human task or whether the computer
task. So let me give an example of part of my own research. This is the paper in CHI 2012, with
my PhD Student, Kazi, together with Takeo and Richard Davis. So this is actually the problem
we're actually looking at, just pen and ink illustrations in art. We can see that these are three
examples of pen illustrations, and what you see in common is that there is actually a lot of
texture patterns that's actually really repetitive. If you look at the task analysis of this particular
task, what you find out is that the typical pipeline involves you first define the pipeline, and then
you put more details to make the outline more fine-tuned, and then you would start with some
sample textures, and then you repeatedly apply these textures to cover the entire illustration, and
finally, you do some touch-ups. If you look at these tasks, step one, if you think about which
task is better for which agent, human or computer, you can see that step one and two is probably
more humanlike. Because humans, we have ideas. We define what we want. Actually, this also
doesn't take too much effort. Just a few strokes, you get -- first step you already know is kind of
a bird. Second step, you know it's a duck or goose, and then step three, also, it's probably a
human task, because the type of texture you want to define, you need to define by human. And
step four, on the other hand, is a very repetitive task. If you think about what computers are
good at, computers are very good at repetitive tasks. So perhaps humans don't like that. We
don't like repetitive things, so maybe this is a candidate for computers to do automation on the
top of it. Step five, again, I think is a human task, because you know what's the final shape you
want to look like. So based on this principle and based on this analysis, we redesigned the
interface and called it Vignette to do drawing and sketching for pen and ink illustrations. Let me
just show you a video of that. So by redesigning the interface, we allow a more fluid and
somewhat natural interface for people to come up with fairly complex drawings and illustrations.
Follow-up work is Draco, and also done by Rubaiat, together with Fanny, Tovi and George. It's
extending this idea to animations. We will not have time to talk about this here, but you can take
a look at the video. So for this pattern, what is the innovation path, or what is the storage that
you can apply when you think about using it? Well, firstly, you want to identify what task you
want to actually work on, and then you want to do a task and workflow analysis and look at what
are the components of this task. And then you want to discover if there's any imbalance. If you
find things that's very repetitive, probably it's better for computer to handle that. If you find
things that requires a lot of decisionmaking, then maybe it's for humans. And then you want to
redesign, assign the sub-tasks to the appropriate agent, and finally come up with an interface to
support that, or algorithms to support that. So if you look at the example of Vignette, the task is
pen and ink illustration, and we analyze it using the task analysis, and then we assign the
different stages to human or to computers, and finally, we compute. We come up with interface
Vignette to improve the process. The next pattern is enhanced directness, so that's also a very
familiar topic in HCI, because we talk about the direct manipulation all the time. So just to kind
of sharpen the definition a little bit, what do we mean by directness? Which direct here means
the shortest path or the straightforwardness between two places. And what is that? The place is
actually more kind of imaginary. It's the user's goals with the physical system, so we call this
gulf of execution and gulf of evaluation by Norman. And the directness here means that it
consumes the fewest amount of resources from the human, cognitive resources from the human.
So the more directness means that the less resources are taken from the human, easier cognitively
or physically. So the goal is to minimize that. What are the kind of manifestation of that already
in HCI? Well, we already use direct manipulation to reduce the cognitive distance between
interactions between tasks and features, and also direct feedback is also useful, and direct access,
as more forms of making this more direct. Just to show you an example of research we've done
in our lab in CHI 2013, my student, TJ, together with Kevin and Anshul. So the motivation here
is to help people to navigate through online videos better, because video is not necessarily a very
easy to access media, because the structure of it is linear and you have to scroll through to look
for information. And if you want to look for a particular -- for example, as a student, you're
looking through a video. You want to see, when did Professor A talk about this equation, and
what did he say about it? The current way, approach, is people actually have to scroll through
the video to find out where it is, which is not very direct. So how can we improve that? Here's a
video of what we did to improve that indirect operation.
>>: Blackboard-style lecture videos, like Khan Academy videos, are getting popular as effective
learning tools for users. But if users wish to quick navigate the video to a particular concept,
what are the existing solutions other than fast-forwarding and guessing? To solve this, we
propose NoteVideo, a novel tool for navigating blackboard-style lecture videos easily and
intuitively. NoteVideos allows users to navigate blackboard-style lecture videos by interacting
on visual elements in the board interface. These elements are extracted from the video by the
NoteVideo processor and laid out in such a way that it retains the position of the element in
relation to the video. When the user clicks on a visual element, it starts the video on the time that
the element appears in the video. By allowing them to already see the overview of the video,
users can use these information sense to lead them to their target information.
>> Shendong Zhao: So the idea here is that -- well, maybe I'll just explain what is this
innovation path for this pattern. Well, first, you want to identify the user intention, and then you
want to discover any indirectness in the existing potential. Then you propose a more direct
approach, and then finally, you have to design the interface and algorithm to support that. So in
the case of NoteVideo, the user actually has the goal of I wanted to find out the videos, talk
about a particular equation, when the lecture was talking about that. The traditional approach
was scrubbing, which is not very direct. It requires a lot of cognitive resources to actually find it
and physical resources to find it. So what we did is that we designed a more direct way, which is
kind of making that visible in the beginning, and you can use that as hyperlink to jump that to
directly, and we have the built algorithms and interface to support that, which is the NoteVideo.
So this is the second pattern. The third pattern is also very common in HCI, is explore
concurrency and complement. So I will just show that using two examples. The first example is
Tilt Menu, which is not our work, but I think it's a very good example of this. Show the video.
So the basic idea of Tilt Menu is basically you can leverage the tilt of the pen to make selections
of menu items, so this actually can act concurrently with your pen gestures, so you can make a
stroke. At the same time, you can tilt your pens to make selections. So you make those two
things happen concurrently. It allows you to perform certain tasks better and faster. So this is
exploring concurrency. The second example I want to show is actually complement and hybrid
solutions. This is the work we did together with Michael and Mark in InfoVis '05, which is a
very old paper. Well, we look at hierarchical visualization, and we know that node-link diagram
is good at showing the structure, but it doesn't scale very well, because it gets really
[indiscernible] in the end. But the tree map is very good at showing -- it's more scalable, and you
can show more leaf nodes at the same time, but it's not very good at showing the structure. So if
I wanted to compare two sub-branches from a tree, from a hierarchy, and while I want to see the
kind of context more clearly and see the leaf nodes and examine them in more details, you can
use like a hierarchy, which is combining the two approaches to make this happen. So if you see
two things that are complementary to each other, then a hybrid solution is potential ways to
innovate, as well. So the innovation path for this particular pattern is, again, you identify the
task, and then you want to do a design space exploration, and you want to look at what are the
dimensions, as well as the different ways they can do it. And then you seek opportunities for
concurrency, as well as hybrid, and then you design new interface algorithms to support that.
And this is how we did for our elastic hierarchies, which we were looking at the task of hierarchy
visualization, and we analyzed the possible forms where you can come up with different types of
hierarchies. And then we find that the first two here are complementary to each other, and then
we build a hybrid form, which was elastic hierarchy to enhance this particular task. And the last
pattern I want to talk in detail is facilitate transition, so when we talk about transition here, what I
mean is the different state of a person when they're actually performing a task. So one thing we
know is that the only thing that's constant in this world is change, But a lot of times, when we
design for something, we often design for a particular state and neglect that there is actually
changes happening all over the place, which are from novice to expert, sometimes for elderlies,
from capable person to a disabled person, and all that transition can happen, and how do you
facilitate that is also a research opportunity. So let me give you an example of early work also
by myself and together with Pierre, Mark and Ravin and Patrick in CHI 2007 called earPod. The
motivation is to come up with a kind of eyes-free interaction in the mobile environment, where
your visual attention is taken away. But try to come up with something that's nonvisual is
actually not so easy, and especially the performance can be a little bit problematic, so you're
looking at this example. If you want to select a menu, a visual menu, if I want to choose the
items, say print, it takes about two seconds to just go through and select, even for novice users.
But if you want to do it for an auditory menu, such as the IVR system, interactive voice response
system, you will take a bit longer time.
>>: Welcome to TD Visa. For customer service, account balance and statement information,
please press one. For credit limit increase or status of an application, please press two. For a
report a lost or stolen card, please press three. To repeat these options, press six.
>> Shendong Zhao: It's only four menu options, but it takes about 20 seconds to just finish
listening to that, so that's the shortcoming of auditory-based menus. So we come up with
something called earPod, and I'll show you a video of how it works.
>>: Color, instrument, job, animal, country, fruit, clothing, fish, color -- fish, clothing. Color,
instrument, job, and country, fruit, clothing, fish, color, fish, clothing [indiscernible] clothing.
Clothing.
>> Shendong Zhao: So one thing you see from this video is that we designed appropriate -different kind of interactions for different stages of the user in terms of their expertise. So we
will compare this with the traditional interactive voice response system, which is the one you just
heard, the TD Visa one. You will find out that they actually do support the novice mode and
expert mode. Novice mode, you just listen to the voices and you make selections. Expert, you
can directly type in the code to do that. However, they didn't design anything that facilitates
transition, because user will not be able to transition from listening to that to expert mode very
easily, but the earPod one, we try to facilitate transition by first to help people to remember
different items and the different locations by using a spatial audio type of arrangement. And
when you actually find a novice to expert, as a novice user, you will probably go through a
slightly larger circle where you actually wind up looking for the final item, but as you become a
little bit better, you probably will remember that the item is on that side, so I will start from here
and it will go a little bit closer. And as you use it more and more often, naturally, you will reach
the expert state, where you can just tap on the expert location directly. And this is happening
spontaneously, just through the usage, actually by itself. So as a result, we try to make this
transition more smooth, and in the end, we think we designed a better interface. So for this one,
the innovation path is, again, identified task and list the possible user states in performing such
tasks and then design interface to facilitate and support smooth transition among these different
tasks. And that's how we did in our eyes-free menu selection, where we're looking at the
different states and we designed earPod to support these states mostly. So let's go back to these
patterns. The top three, so we talked about these four -- the top three is actually quite common,
finding alternatives. If you're looking at the CHI committees, there's one committee called
Capacity and [indiscernible]. It's basically just talking about alternative ways of doing things.
Reused patterns, there's a lot of toolkits being invented, Profuse, D3, and there are toolkits being
done by Yang Li on gesture design. There's a lot of toolkits being also come up with, which is
fitting to this innovation pattern. And invent based on theory, we have a few theories in HCI -not too many, but we have a few, and I think Bubble Cursor is a very good example of that,
actually, designing based on the theory. And the last one here, define appropriate chunk and
timing, I think there's example here done by Ken earlier, is where you try to enter a series of
commands, but they are chunked together. And to prevent more errors, you can use kinetic static
feedback to avoid those mode errors, so I think there is this notion of how do you group tasks
and group the tasks so that it will perform in the optimal state. I think that's also an important
dimension, and that you can look into, but I don't have very good examples, but I didn't really
talk about it very much. To recap, innovation patterns is for -- hopefully for HCI researchers and
innovators, and I think it could be useful when you're actually planning to working on new
innovation, as well as once you finish innovation, it can help you to explain your innovation
better. And there's a lot of future work. This is just a starting of the sort. I did this because I
wanted to teach a class, and I wanted to organize the themes in the massive papers a little bit
better, but I think there is a lot of future works that can be done to define other patterns and
define them more clearly, so that people can use them more effectively. And I wanted to
acknowledge Richard Davis and Michael McGuffin. I had some discussion with them before I
did this presentation, and finally, open for questions, if you have any. And I think people's
consumed endurance is high right now.
>>: Just one thought I have is that it seems like some of these are -- at least it's there
particularly, and now they seem they're probably more appropriate to students or please just
entering the field, as opposed to people who have been working in the area for a long time. Do
you think there's some more subtle ones that you could suss out that might maybe lead to some
more surprising techniques? I guess I'm just jaded, like I've seen everything this morning.
>> Shendong Zhao: Yes, I think that would be interesting, maybe dig deeper into analyze all the
papers, and then maybe I can identify some other patterns that's more surprising. So be looking
forward to thinking about that a little more, and if you have any other patterns, please let me
know.
>>: I think a common experience is that you can get stuck into one mode of thinking, right, and
then it's only when you bounce ideas off a colleague that they make you think in some new way,
and in hindsight it might be obvious, right? So like there are decks of cards you can buy to
stimulate new ideas, where you pick a card at random, and it might make you think of something
new. And it occurs to me, if this list of patterns gets too long, it might get less useful. So I guess
it's a tradeoff in how finely you want to distinguish different patterns.
>> Shendong Zhao: Yes, I agree. I think it would be nice if it's only less than 20 patterns. And
then I think that would be more useful. I personally find it useful for myself, when I was
thinking about projects, so yes.
>>: Oh, I didn't have a question. I'm happy to make a comment. I'm kind of intrigued by these
patterns, because it's kind of this idea that you would have a recipe for innovation. I must say, I
do question -- I see the value of the patterns, but certainly the way we work, or I work, is very
different. We only, in my mind, do obvious things, so very often we find a new topology,
something that didn't exist before, and then we think of an obvious application and we do it.
And we try not to iterate too much. I see that -- to me, that's a mistake that a lot of junior
designers make, is that they iterate and iterate and iterate, and they start really focusing on
details, for example, of 3D-printed stuff, early on and they never really get to the result. So I'm
not sure if you could comment on the shortcut pattern. You start with the end.
>> Shendong Zhao: Yes, I certainly don't think these are the only ways to do innovation. I think
these are -- when you're stuck, you can start looking at these kind of things and see if they can
help you to perhaps spark a new idea that you haven't thought about before, and if you're just
looking at my own projects in the past, a lot of these projects are investigating one direction and
then failed and data come in and point a new direction, and then all of a sudden comes out, and
then some of these, from the hand side, I can think about, oh, that's what I'm actually doing. But
recently, do I see that after thinking about this, there's occasions where I think about these first
and then I say, oh, this problem, let's just explore concurrency on that. And then it seems like it
allows me to do things a little quicker, but I don't think you should be just locked into these
patterns, because innovation should be endless and more surprising the better, right?
>>: My favorite design pattern is KISS.
>> Shendong Zhao: Yes, keep it simple. That's great. Yes.
>> Ken Hinckley: I think we better get to lunch, but thanks very much.