18194
>>: Andy Wilson: It's my pleasure today to introduce Dr. Roel Vetegaal from Queen's
University. He's, I guess, the Director of the Human Media Lab there. And he's been doing a lot of interesting things over the years with display technology. Some of you here know a little bit about that. And, in particular, he's going to be talking today about organic user interfaces. I think the thing that impresses me the most about Roel is his very multi-disciplinary nature and some of the conversations I've had with him over the years about many far-ranging topics going from display technology into, in particular, lately, how to evaluate whether or not a particular interactive technology is pleasurable or fun to use. I think these are very interesting topics. I'm hoping we'll touch on those a little bit.
So without any further delay, I'll introduce Roel.
>>: Roel Vetegaal: Thanks, Andy. Welcome everyone. So my story begins with the
Egyptian Monk Cosmos, who, in 547 A.D. in his, topographic Christiani argued that the earth was flat, a parallelogram enclothed by four oceans.
It should fit the arc of the covenant which was supposedly a parallelogram as well. It comes as no surprise, then, that the tools we use to see the world greatly affect our world view. Unfortunately, in about 1500 years or so, a little has changed.
You may think that this is arguably the hottest computing technology, I know I'm speaking at Microsoft, is the iPhone but I have news for you. Its user interface is really not that different than, say, the Mac 2 color PC that came out in 1987.
Except for maybe a couple of multi-touch gestures, but really there were only a couple.
Sure, it's a lot flatter. Sure, it's a lot smaller. But flat is the operative word. If you were looking at Google Maps, for example, one of the issues here is that it's a very, very small screen. So wouldn't it be cool if we had something like this instead.
So that's the kind of thing we're shooting for. Foldable displays that can open up real estate when you need it. And, of course, the metaphor there is the metaphor that's been used so much in human computer interaction as the metaphor of the book.
So we all know that the earth didn't turn out to be flat, and just because this monk's tool was flat didn't mean the earth was, but it required a new tool. One of the earliest information technologies, a tool in itself was not flat to change this world view over the course of about a millennium. The movable type book, which was popularized in the western world by John Johannes Gutenberg. Unlike the scroll, which required linear navigation, book had random access, which was really cool at the time.
And those pages carried bits of information that could be easily syndicated like an RSS feed today through printing. It was also a key factor in the European renaissance, which is one of the reasons why we're here today as researchers. It essentially opened up the market for research.

However, imagine what it would have been like for a monk to be faced with this burst of information technologies, the book. Would it be truly usable? Would there be a help desk for a book? [video]
You can watch the full video on YouTube. One of the messages here is that learnability is overrated. If you introduce a technology to an 80-year-old, they're not going to be able to learn it very easy. One thing with the graphical user interface, it got introduced to essentially people that were beyond my daughter's age, who is here in the audience.
But my daughter grew up with these technologies. And for her the GUI is basically writing with a pen.
Now, imagine that you would teach 75-year-old to learn how to read or write. You would have a similar challenge. So I think learnability is overrated. However, another message to take home here is physicality translates into learnability. The idea of picking stuff up, being able to manipulate it in your hands, living in the real world not being a technology affects the learnability of that technology.
So I think we need to put a little less emphasis in HEI on learnability and focus on some other things instead. But we'll get to that in a second.
So the book has always been the metaphor for HEI. The first big project in HEI was done by Vanover Bush in 1945 called Mimics, the first mechanical book. It had actually thin film displays, which is really cool. They were the kind that you find on microfilm. It was a microfilm reader. Completely mechanized. It was never built because it was impossible. It had limited form of storage that sort of resembles the way we use Web today and resembles Doug Englebert's design of the graphical interface and the mouse used today.
Another critical prototype in computer interaction was Dynabook done by Alan Kay in
1969.
The thing about this Dynabook which essentially was the first tablet PC, is that it was never real. It was actually done in cardboard. So what you see here heavily influenced the computer used today. You can even see the keyboard layout and some of the fonts are almost identical. The only difference between the MacBook Pro here and Dynabook is that it hinges. But Alan Kay worked for Apple during the time that the power books were introduced, so there's a direct lineage back to the Dynabook there.
Now, Alan Kay on Dynabook. Another thing we saw in 1968 was a tiny 1-inch square first flat panel display down at the University of Illinois. We realized it was going to be only a matter of years until you took all electronics on a flex machine on the flat panel display, later came to call Dynabook. Back in 1968 I made this cardboard model, thought of it as the machine of future and thought what it would be like to have millions of people have this machine the metaphor became the pencil. What would it be like to have something like this in the world, like pencil and paper?
Could people actually use it? And the answer in 1968 and early '70s was no. Now, Alan went on at Xerox to create the Alto.
Which was the first real graphical interface that led to the Macintosh.

So taking paper and taking books as a metaphor is one thing. But we have to recognize that these elements still live in very, very different realms. We have the technology realm of the computer, and it's a virtual realm that's very distinct from the realm of paper such as here on my desk.
Or as P.L. Relner at Xerox put it, we interact with documents in two separate worlds, the electric and physical world. Interaction styles between these two worlds do not resemble each other.
That's one of the chief reasons we still use paper. Abigail Salen did a book on this topic and concluded one of the reasons why paper has remained really important is because
GUIs are not good at navigation. Input is not on the object. There's at least two people in the audience here who are trying to fix that.
One-handed input of single XY coordinate. Again, we're fixing that with multi-touch. But this notion that you can only move in a single plane in two dimensions is extraordinarily limiting when you think about the fact we have two hands and ten fingers and we can move in 3D. A lack of tactile feedback in motor tasks makes it hard to learn certain gestures.
And, of course, limited screen real estate when you're having overlapping windows. This is particularly an issue with small devices. By contrast, paper supports multi-touch, annotation law reading, and multi-tasking, because you can easily mix different documents and put them out in your peripheral vision.
So Pierre Rela [phonetic] realized that, did probably one of the most seminal pieces of work in the recent history of HEI. And that's Digital Desk. Essentially, the technologies we've been using are very similar to what he's been using. Computer vision for tracking finger movements. Projection for projecting on a flat surface on a desk, and then being able to manipulate a digital calculator projected over regular documents on the desktop.
You see -- I'm not going to run the full video. You see a piece of paper here and he can circle a number and enter it into his calculator that way. And this was done in 1990.
Since then, as I said, multi-touch has evaluated. We now have these cool screens. This is Jeff Hand's work, that allows to express a lot more dimensions in input, which I think is really important if you want to do creative stuff with computers.
We've also seen the evolution of shape in input. This is work by Robin Belakrishna and
Toby Grothman on shape tape. The idea being you can take bending and use it as a direct way of representing a shape. In this case, three-dimensional model, which is purportedly a lot more efficient than having to draw this with a mouse.
And an evolution of tangible user interfaces. Tangible user interfaces being essentially user interfaces that live in the real world that you can hold in your hand. This is some early work by Hiroshi Ishi at MIT, trying to devise a clay interface, three-dimensional clay gets sent by a laser scanner and projections get updated accordingly and projected back on to the clay model.

And this is what I consider to be probably the world's first organic user interface. This was done in 2002, presented at KAI.
So we're seeing a landscape analysis model here now, if you depress or change the landscape, the water runs differently over the landscape.
So my little moment was when I saw this display at Polymer Vision about six years ago now. That's a Phillips spin-off that recently moved to Taiwan. And this is flexible E-Ink.
So E-Ink is the number one technology that's really, really interesting. And you've all seen the e-book revolution unfolding itself. But this is actual flexible E-Ink. This is getting really, really close to the kind of paper metaphor that we were originally inspired by.
And that led to the formulation of a new kind of user interface, an organic user interface.
User interfaces with nonplanar displays that may actively or passively change physical shape.
So these are user interfaces that really live in the three-dimensional world. You can wrap them around objects, but you could also treat them as flexible and treat them as paper, in which case you can still bend them and they would live in the three-dimensional world in that respect.
So I think the quick take-away message from this talk is that organic user interfaces is going to mean that products will be computers. So here we see a piece of radical design we did using projector ink technology. I'll show a video later of the thing that I thought was most unlike a computer, most like a product that I could think of. That was a Coke can. It's a brand. It's not even a drink, it's a brand first. So we made this interactive.
We did this through painting the can with reflective paint and then tracking the can with
Vicon computer vision tracker. And put buttons on it so you could interact with it in real time, projecting back a 3D model of the can texture mapped with the interactive graphics. That allowed us to toy around with the idea what it would be like if we had these kinds of what we call display objects.
But what it also means is that industrial design is going to be really, really crucial in the next phase of human computer interaction. Like graphing designers took over Web design, industrial designers will take over the design of most computer interfaces.
Once computer interfaces become integrated intimately with products such as these or, say, cereal boxes or other kinds of mundane products, we are going to not just to need the material expertise and the expertise in form and function that industrial designers have to offer.
As technology commoditizes, there will be less of a need for technologists in order to make this work and technology will become one of those things you can use as a material in your design.
And the other notion here is that once technology commoditizes, which it's already doing, the only way in which you can differentiate products is through industrial design.

And that's nothing new. I stole a slide from Bill Buxton. This is the Kodak Series 3 pocket vanity by Walter Doren Teague, designed in the 1930s as a way of marketing
Kodak cameras to a female population. And the way in which this was done was through the industrial design trick of giving it candy colors.
And what we see here is that that trick has been replicated at Apple with the iPod Nano.
Simply coloring things, making it seem more like a fashion article, differentiates your product. And that is definitely something that is in the realm of industrial design.
But what I'm proposing is not just this kind of styling, which is why the superficial, but actually integrating the function of a device completely with user interface such that it only works in limited contexts.
So, for example, if you had a product that had a screen integrated, it would only know about the functions of that product. You wouldn't necessarily use it to check e-mail.
Instead, you would have a larger array of various products that have various functionalities and they would talk to each other and allow you to have this sort of microcosm of information that's hypercontextualized.
All right. Before we start talking about organic design principles, first, what it is not. How does this differentiate from the work by Jonathan Eisen at Apple, for example? Here's a pretty cool design of something called the Barcelona Pavilion, which was done by
Ludwig [indiscernible] very famous German architect and director of Bowhaus, but it was done in 1930. So what we're seeing here is very straight lines, very rigid forms, rectangles, wearing the inner on the outer; but in particular we see rigid and planar devices. This is the kind of aesthetic that we're seeing in properly industrially designed computer products such as Apples today. And it's an aesthetic very much of the 20th century. It's almost an aesthetic of the 19th century. About 100 years old aesthetic.
We've moved beyond that in architecture. I want to talk to you about that a little bit.
Think of devices as devices in and of itself is wrong. When we think of a book, a book is not really a device. It's not really a technology. It's just a thing. We're designing it as a thing. It's not just because it's so old, but it's also because it's been engineered in such a way you don't have to think about interacting with it anymore. It lives totally in the world. Design aesthetics should not really be equated with engineering efficiency.
One of the issues with engineered devices is that they don't age gracefully. And so here we see, does anyone know what this picture is? What building this is? It's
[indiscernible] designed by Rogers and [indiscernible] Peano. And it gets really dirty after a couple of years, and they have to clean the whole thing to make it look new again so that it looks cool again.
>>: What is it?
>> Roel Vetegaal: It's a museum. But it wears -- this literally doesn't just wear its construction inner on its outer, but it wears its plumbing on its outer. It's really cool but it gets dirty, then loses its coolness. Does not age gracefully. So that's a problem with modernist designs, is that the materials really have to be plasticy and shiny all the time.
Another thing we need to get rid of is the idea of control or control devices. So the mouse is very much a control device. I use this box and it talks to my computer and it controls an XY coordinate on my screen. Wrong. That's not how humans think. That's

not how humans walk. That's not how humans talk. If I have to think in those terms in order to be able to give this presentation, I probably would be taking three hours.
So we need to get rid of this notion of control. In a world of 24/7 productivity, we can question this emphasis on efficiency that is sort of comes out of the engineering aesthetic. We don't need to be more efficient. We need to be more creative and we need to be able to have more down time and be productive at the same time.
Another notion that I think it's not is this idea of cleanliness and youth that's permeating today's society. The idea of not being able to ever grow old and having children at a very late age of 60 or 65, or performing plastic surgery at a very old age in order to continue to look young is a complete fallacy that we somewhat see reflected in devices such as this iPod, previous generation iPod, which has a really clean design. It's completely white.
And its aesthetic is very, very simple in that sense. So those are all aesthetics of the
20th century. They're not aesthetics of the 21st century. They're not post-modern aesthetics. The same is true for the principles of usability. For those who aren't familiar with these, Brian Sheckel, amongst others, in 1990 formulated four principles of usability. The first two being learnability. Well, I've already dismissed that one. And the second being efficiency and effectiveness. And I've already dismissed the efficiency part.
The effectiveness will remain very important. I personally believe that the goal that you have with using a computer is the most important thing. And if you're a musician, you realize that it might take ten years for you to learn an instrument. Why would you expend so much time learning an instrument? Well, there's really only one reason, that's because it allows you to be effective with that instrument, otherwise you won't be effective. We haven't really been designing computer systems for that kind of expert use, other than in the military. That's an interesting notion. But efficiency, well, I think we're already extremely efficient. And I think we have other things to think about, such as the other two remaining principles, which are very ill-defined.
Flexibility. Nobody really knows what Sheckel meant by that. One interpretation is that users can change around and customize their user interface. We don't really know how to evaluate that. You can have people customize or not customize and measure how efficient they are, but that doesn't really seem to capture the spirit. Could we start digging a little bit deeper and truly think about flexibility in terms of, for example, the flexibility of clothing, and the functionality shape shifting that occurs there.
If I, for example, wear my clothing, the clothing is flexible enough to fit my body.
However, if I want to store it, it folds up neatly in a small volume. That is really cool.
Satisfaction, finally -- and this we talked about a little bit this morning, Andy and I -- is something that's extremely hard to measure, knowing that some interaction is satisfactory. We all know the feeling, that's maybe why we buy Apple products, but we really don't know what it means. We really don't know what it means, and we really don't know how to measure satisfaction very, very well.

But there seems to be something engaging about certain designs that other designs don't have. And it's almost this intangible property that industrial designers know how to capture very well.
All right. So let me ask this question: Now, obviously this is a ruin. But is it more or is it less? Let's not think about this too long. How many hands for more? We've got two hands for more, three hands for more.
How hands for less. Two hands for less. So we kind of have a split. A couple of nonvoters. Well, there's a couple of arguments for saying it's less. Well, it used to be a completely functional building and people would use it, so it's not longer that, so that means it's less, so why do tourists flock to buildings like this? It's because in terms of aesthetics, it's more. This building is actually, in many ways, prettier than the original building because now it has moss over it and high information entropy. It has a cool aesthetic that's quite rough. Yet, this aesthetic is not captured at all by any computer devices we have today. In fact, in today's society it is not valued at all.
Now, there are other societies where that's not true. So in the states they would probably throw this out as a misbaked piece of pottery. In Japan, they would pay
$10,000 for it. The art of subtle imperfection is a concept known as Wabi-Sabi [phonetic] in Japan. It means sad beauty. And it finds profundity in nature, accepting cycles of growth, decay and death. It's simple, just like the modernist design, but in a very different way. It's simple in its earthiness. In its simple, slow, uncluttered authenticity.
So it's H wood, not Pergo. Rice paper, not glass. It celebrates cracks and crevices and other marks of time, weather and loving use leaves behind. It reminds us we're transient beings and this planet as well as our bodies exists in the material world for a limited amount of time.
Through Wabi-Sabi we learn to embrace rust, frayed edges and the march of time that they represent.
Now, it's no coincidence that the young architect Frank Lloyd Wright, one of the greatest architects America has ever known, once studied and lived and worked in Japan. He built the Imperial Hotel in Tokyo in the 1920s which is now destroyed. But came back as a transformed being. He never really advertised what he learned in Japan, but instead coined the term organic architecture. To articulate many of the aspects of Wabi-Sabi and the balance between humans and the environment they create as well as nature.
Frank Lloyd Wright in 1939 coined the term organic architecture. This sounds really crazy, but this is a direct paraphrase. So here I stand before preaching organic architecture, not cherishing any preconceived from fixing upon us either past, present or future, but instead exalting the simple laws of common sense or super sense if you prefer determining form by way of the nature of materials.
A well-known example of this kind of architecture is Falling Water, perhaps the most famous piece of post-modern architecture in the world. Wright had many choices to place the house on land. But he interviewed the family. They said we really like the waterfall. What they meant was we like the view of the waterfall. He thought it would be a nice engineering challenge to place the house right on top of the waterfall, which is what he did. But in doing so he created a masterpiece. Why? Because he managed to

perfectly balance the cantilevered aesthetics of mankind, these concrete striations here with the flows of nature that literally go into the rock. He used natural rock for this pillar.
Balancing the aesthetics of mankind with the flows and aesthetics of nature in an almost perfect sense.
This got picked up by what's called the Gaia charter of architects. I won't go into Gaia, has to do with earth balance. But it was an interesting charter devised by David Pearson in 1989. He's a sustainable architect. And he said architecture should be inspired by nature, be sustainable, healthy, conserving and diverse. It should unfold like an organism from the seed within. It should exist in the continuous presence and again and again. It should follow the flows and be flexible and adaptable. Satisfy social, physical, and perhaps most of all spiritual needs. Grow out of the site and be unique. Celebrate the spirit of youth, play and surprise. It doesn't negate what I said earlier. I don't think there's anything wrong with being useful and playing at an old age. I just think there's something wrong with faking it.
And to express the rhythm of music and power of dance. So being a scientist, I thought that was, you know, fluffy. So I took that and translated it into a couple of design principles for organic user interfaces. First of all, it means that input devices should be output devices, and I apologize for even using the term devices here. Input equals output. If we want to live in the real world, then whatever effect we have on the real world should be directly on the artifact.
That means that if we have an input device it really cannot be disassociated from the output device. And in simple that means touch screens. But it goes a little bit beyond that.
Display shape and deformation, for example, could be used as a source of input.
Secondly, form follows flow. And this stems directly from designs like Falling Water.
Form may follow any physical shape found in the natural environment. That means we can create computers in any shape or form now. They should also mimic the flexibility of those materials. Including being able to unfold and to move according to, for example, the physiology of the human, which is a really important ergonomic principle. They should also flow around activities. Allowing better use of multiple devices during multi-tasking.
And create continuous fluid experiences that are imbued with physics rather than creating binary experiences that are sometimes hard to follow.
And finally, and this again comes from Frank Lloyd Wright, form does not follow function.
That was actually coined by his mentor Sullivan, as a modernist idea. Form follows function, it means you can reduce all embellishments and just create what's functional.
This is what America is built on. What Frank Lloyd Wright proposed the two are interrelated, cannot go without each other. When we're talking about user interfaces, software user interfaces, the form is somewhat disassociated from the function because it's flat. But if we have these user interfaces running on a three-dimensional shape, then of course the shape of that object is going to affect the software user interface as well as utility very, very intimately. That means physical affordances are going to make a comeback. Being able to feel what the function of an object is is going to be really, really important. It also means that we can begin designing for a different kind of activity. And

I'm talking about things like creative activities where very often you need to manipulate objects. But also social and spiritual activities and to have a certain physicality in your interactions. I'll get back to what I mean by the spiritual in a sec, because that still sounds pretty efemoral.
So input equals output, I've shown you an example of that. This is one form of input equals output. Form equals flow. And this is something that I hate my iPhone not having, this ability to shape shift. Does anyone have a clam shell phone here?
Everybody has an iPhone, I take it. Just kidding.
Well, what's really cool about clam shell phones is that you can open them up and that actually communicates, it's a great affordances for opening up communications, but it also unfolds so you can hold it against your head and it follows the shape of your head as you do so. The microphone ends up over there and speaker over there. But what's cool, when you close it, not only do you close the communication channel, but protect the keyboard and display from any disruption in the environments, scratches, etc. This is exactly the thing what I mean by physical affordance, but also form following the flow of your activity. When you're on the phone, open it up form changes, you close it, change, boom.
Now, here's a design by Karin Rasheed that's really interesting in this regard. This is just a soap bottle. It's not even a computer. But what we're seeing here is that one of the issues of soap bottles is that when they're almost empty, you have to put them upside down, right, to squeeze that little bit out. And so Karin redesigned the soap bottle and talked about, well, why don't we put the bottle upside down to begin with. Now all I have to do is pick it up like so and it can fold around the hand and I can squeeze it.
Perfect example of form equals function. Unfortunately, I do think it was productized, ended up coming off the shore shelves now they have this, I'm happy to say this on video, terrible design that doesn't stand up on its -- upside down, because it will just keel over. It also doesn't squeeze very well. The reason was I believe that this design was too expensive to produce.
But this design is absolutely brilliant, I believe.
>>: [indiscernible].
>> Roel Vetegaal: Well, I actually got this from a Web page and already said that. So I don't necessarily subscribe to that. But it's interesting to see the dynamics of the shape here going from somewhat more rectangular to more of a body shape. And I do think that that communicates this idea of fitting to the body. Picking that bottle up requires very little energy, whereas picking that bottle up, first of all, you need to fit your hand around it, which is hard to do. And you have to tilt it like so.
So here's my summer office. This is my backyard in Kingston, just north of Kingston,
Ontario. And I'm not showing you this just to show off. This was actually the environment that another very famous architect grew up in and probably got inspired by.
Frank Gehry, Canadian architect, who is very famous for this particular building, which transformed the city of Bilbao from somewhat rusty industrial wasteland to a tourist attraction. That's what I mean by designing for spirituality.

Here, he designed a building that probably doesn't even hold paintings very well.
Because nothing in the building is rectangular. What he designed was a landmark that transformed an entire city using one building. That's spiritual design.
That's thinking bigger than the actual function of the architect or of the architecture.
What's also interesting here this is a high tech building because it mimics some of the shapes you find in nature. All of these curves are created as nerve surfaces in his special software that really is the key to his success, being able to do finite element modeling of the strains and stresses that are placed on these surfaces has been a key to make these buildings happen. So we're looking at a $300 million building right there.
But to mimic nature is a high tech endeavor. It's not a low tech endeavor.
Now, much older example of how computers can integrate into shapes that are nonflat and into buildings and create skins for buildings is of course the NASDAQ display, which is literally a skin around a building. And probably one of the first examples not just of a display integrated into an architecture but also a display that is nonflat and very, very large.
All right. Brief word about the technology. We've already seen this. Polymer vision roleable display. Uses ink on a flexible substrate. The challenge here is to actually print the circuitry right on a flexible substrate which is easy to break.
The E-Ink itself is pretty robust. This is how it works for those who don't know. You polarize a plate just like you do with an LCD display. You have polarized ink capsules that either repel or attract based on the polarization. If you want gray, you do a neutral and you'll get a gray mix of ink capsules.
The big downside of this technology is that it's very slow. The big pro, it consumes very little electricity when not changed because it's just ink and it will just stay in its current form.
So this is the biggest piece of vaporware in the history of flexible computing at least.
This is the polymer vision radius, which I actually can -- you have to believe me, I held in my hand in a fully operative form in 2005. And it hasn't made it to market for various reasons. But it is a really interesting example of how you could have a display that rolls and unrolls, and thus creating screen real estate when you need it, yet a small form factor when you don't need it.
Here's another technology that's going to make this happen. And that's flexible O-leds.
This is a Sony prototype. You could easily imagine this wearing a wristband, something like this around your, on your clothes or as an arm band. This is completely flexible technology. This is currently at the state where the LCD was when Alan Kay saw it.
Okay. So I'm not predicting that this is going to be everywhere immediately. However, many of these technologies have come out a couple of years ago now, so I'm confident to say within the next five to ten years we'll see products. In fact, we're already seeing some products come out and I have an example later.
Right now we're still focusing on rigid surface screens. This is however very, very much like paper. We see here cardboard piece of machinery that actually is a computer. This is the plastic logic ink pad that was introduced last year. And it is in fact a computer. So

it does sense touch. And this is their entry into the e-book reader market, recently got introduced at CES in a somewhat different form. What I like about the form factor it really looks like paper. The only problem is it does not flex yet. And I believe that that's not necessarily because they can't do it. There are other reasons for that that I won't get into.
Here's one piece of technology we have been able to lay our hands on. It's Phillips
Lumalive, a pillow you can stick under your shirt, or a display on your shirt. It's fairly high res. It's about, I don't know, 128 pixels or so. You can actually show messages on there. And I'll show an example of a little experiment we did with Phillips Lumalive last summer.
So there are a number of technologies coming out that will make this happen. However, many of them have pros and cons in terms of their sustainability. And I just wanted to mention that. When we speak of Gaia we need to speak of sustainability. One really cool notion is that if we can actually wrap flexible displays around products is that we'll have displays and computers everywhere. However, if we don't do it the right way, we won't be able to recycle them. Imagine the horror of having a display on every Coke can. And Esquire Magazine already made that mistake in 2008, I believe, they came out with a flexible E-ink screen. It was just a banner that basically yelled, it had three pixels that were printed as words.
And it yelled "buy me," essentially. And there was no way to recycle that. Instead what they should have done is they should have cancelled the print subscription and go on to the Kindle instead. That would have been a radical move that would have actually been beneficial to the environment.
And the reason, of course, is that these E-Ink screens, potentially use less electricity when reused. So Kindle is great because you keep using the same device for different content, and of course it also means you don't need to cut trees for paper. It also means you don't have to use gas in order to ship bits over atoms. So there are a bunch of reasons why this is potentially a green technology. However, if we don't get it right it will fail as a green technology. And the key there is reusability, not recycleability.
This technology will be very hard to recycle. It will be very easy to reuse. So if we focus on manufacturing products that can be reused over and over and over again, then that means we can actually make a contribution to the environment by delivering bits where normally atoms would be shipped.
So an example of that is, for example, having a flexible screen that you would use in the morning to read your E-newspaper, and in the evening you might read a magazine on it instead. And of course that's very good for the environment.
All right. Now, a couple of examples of some of the things we've done and some of the things other people have done. This is fairly limited and mostly focusing on what we've done.
There are three categories of organic user interfaces. The first is the flexible form factor.
The second is the rigid nonplanar form factor, and the final is a programmable form factor. I see that very much as a thing of the future.

So, first, flexible form factors. This is a concept mockup of a credit card display that could be used as a map. It got prototyped as a rigid screen with a bendable acrylic called gummy by Sony in 2004, by Carson Trasec and Ivan Pupyrev. And the idea was you could bend to zoom on a map, bend out to zoom out of that map and you could use back-of-the-device touch screens in order to move around.
That same year we did a system called Paper Windows to sort of push back and see like how far can we push the envelope in terms of interaction techniques if we actually had really paper displays, really thin film paper displays. We did that by capturing using a very expensive Vicon motion capture system, nine markers on pieces of paper, and then fitting a nerve surface through that, texture mapping it, projecting that back on to the display or on to the paper. And that means that pixels which would be enlarged because of the shape of the paper and the way the projection projected light interacts with that fold get preshrunk in the 3D model and enlarged again when you actually project them on the paper. That basically takes care of all skewing problems. You can take any shape, pretend it's a real piece of paper. Gives a real interesting feel. The experience I had when I was using the system was that of using a leaflet. What you're seeing here instead browsing through Web pages back and forth by rotating the piece of paper.
>>: So do you have a Web page?
>> Roel Vetegaal: Johnny Lee who works at Microsoft replicated this work at CMU a couple of years ago where he used a $20 device to do the same thing. This is a
Wiimote that he basically using active markers used to create this foldable newspaper.
>>: This format is a large sheet folded in twice. It allows the user to customize the desired viewing area from half a page to viewing two full pages.
>> Roel Vetegaal: Here we're seeing an example of increasing real estate with real ease, yet keeping it in a foldable pocketable format.
>>: The user decides to reduce the viewing area, simply fold the display in half again.
>> Roel Vetegaal: And he also had fan-shaped devices. Here's a piece of work we did about two years ago, recently got published at TEI. The notion that board games are really a very fun and social family activity. But we're seeing computer games take over this world. And, you know, mom might be playing behind her computer, and the kid might be playing with her iPod Touch or whatever it might be.
Wouldn't it be fun if we could integrate the two worlds and have displays that actually are active on cardboard. What we see here is cardboard Settler of Katan gamepiece.
What's cool about that game, it's actually a reconfigurable board made out of hexagonal tiles. And we're tracking these tiles with little tiny IR markers and projecting a real time strategy game on it so we could have the best of both worlds. We can sit around the table, construct these huge tesselated screens that have full gaming content. But what it also means is that we can actually use physical interaction techniques in order to interact with the characters, such as, for example, pouring information from one tile to the next. Here we see sort of a liquid motion. Just to demonstrate the idea.

And the way that could be used in a game, for example, is pouring troops on to a village or, in this case, we connect the ship moves one tile and then we take the tile and pour the troops that are on the ship out on the beach.
Now, you can of course do all sorts of regular things with this that we're used to in GUIs such as, for example, being able to pop up a context or marking map or something like that.
Here's actually the Esquire Magazine flexible display. First flexible E-Ink display we laid our hands on in the lab. Here we're seeing a bending gesture with bend sensor and
Ardwino [phonetic] scroll through the information, which is three lines, three pixels that are pixillated awards, essentially.
But we're now moving towards making this a reality. Going from a projected interface to real flexible screens that, for example, have these kinds of interaction techniques in order to be able to page through them. So what you're seeing here is bookmarking. I'm going to run that again. What we're seeing here is paging down on one side. Paging up on the other. Using bend sensors. And the user creates a bookmark, which gets place then navigates bookmarks using an earmarking gesture.
One of the interesting things about this is we're going to see very different way of interacting with flexible screens than we do with rigid screens that's because there's no normal for us with flexible screens.
So if you press a button on a flexible screen, the button doesn't push back, that means it can be very hard to actually depress with any force unless you hold your hands behind the screen. And that means you always have to hold the screen in a certain way in order to be able to depress.
Now, of course, capacity of touch screens are problematic because you get spurious interactions all over the place when you roll these things up and stick them in your pockets. That's one of the reasons why we think bending is actually a very good solution for navigating other than the fact that we're all used to this. When we read books, we're used to sticking our fingers in between pages, putting physical bookmarks down. So there's a lot of transfer that you get from that.
And our initial evaluation show that this actually decreases the mental load with regards to paging, as compared to, for example, push button navigation page up-and-down interfaces.
Here's another flexible display technique that was projected. This is work by Julian
[indiscernible] works now with Robin Belakrishnon where he was interested in surgical operating theaters and trying to support imaging right on the body. Here we have a flexible display that's projected on a piece of textile, could be a surgical piece of textile.
Project the images right on the patient and you use gestures over the flexible display.
So this is not actually touching the display. This is hovering over the display in order to peel image data so you can look inside the human being.
Here's another example of the same technology where he takes a reusable textile form factor, places it on a cereal box. This means the cereal box itself doesn't need to

necessarily have a display. Not every cereal box needs to have a display, still can create an interactive experience with that cereal box because the textile form factor basically takes on the form factor of the underlying object and now it becomes an interactive cereal box. You can read up your ingredients.
And this is a piece of Lumalive that I was talking about earlier. We are very interested in multi-touch on these kinds of form factors, and of course the problem is that you can't necessarily do multi-touch on somebody else's body and so we're really interested in finding where can you touch and where you can't touch. This is a social touch experiment that we're conducting right now. It turns out that experiment actually hasn't been done in social science. All these questions have been done with questionnaires instead because people are touch towing around being able to touch for real. I think it's important to do it for real and of course the original results suggest that the shoulder pads are the best place to put this kind of sensor.
Yes.
>>: What's the context of being able to touch someone else to control a display? The scenario.
>> Roel Vetegaal: You can imagine using, here we're seeing the tapping action. I'll replay that. Replay that one more time.
You could of course imagine using a remote control or cell phone in order to interact with somebody else's shirt, but there are obvious issues there in terms of privacy and how to manage connections, how do you set up a bluetooth connection with somebody else's shirt and do I want you to change my content. What's cool about being able to touch is the person standing right in front of you, if you don't want them to touch, there's several ways we've evolved over years to make it happen.
>>: Alluding to, is there an application where that makes sense?
>>: Is there an application, a lot of stuff you showed me, I rock it. I understand why you might want the cereal box to tell me about all this stuff. I haven't seen these displays before. So I don't know the context, but it seems to me I could see myself wearing that kind of signal for the world to know something about myself. But why would I want somebody to affect what I'm showing. I'm missing the scenario.
>> Roel Vetegaal: One interesting idea is that you want to customize or explain a relationship, for example. So let's say you and a potential girlfriend are in a disco, and your girlfriend wants to find an icon or a picture or something like that, that sort of signifies the relationship, then being able to use touch on each other is actually very appropriate mechanism in order to change the content of the screen.
I can of course imagine, there are many other scenarios, I can of course imagine all the other ways of interacting as well, but this is a particularly intimate way of interacting that's interesting because we don't actually do this. So it's a great example of things we don't do with computers that maybe we should consider doing with computers, and we'll figure out what happens next.
>>: Can you just change the language, translational, French [indiscernible].
>> Roel Vetegaal: So the second category is rigid nonplanar form factors. Here we see the first iPod. Sorry to keep talking about Apple. I really didn't mean to do that. But I

kind of have to. This was a piece of styrofoam with no display. It's really difficult to evaluate this early on with users because you have to put them behind a computer to show them the software interface. So there are lots of reasons why you would want to prototype in styrofoam. There's no computer support for that. We did a project called display objects, where we took the same type of stuff we did for Windows and instead of having to draw faces on these styrofoam objects, you simply project the user interface on there. What's interesting about the notion you can start thinking that hardware really isn't important anymore.
If any object can be your hardware, then really the functionality becomes a software downloadable item.
So instead of -- oops, there goes my power. Okay. So here we see an iPod being projected on to this piece of styrofoam. Now I can actually pick it up. It's just a piece of styrofoam but it has all the functionality of an iPod. So I can customize flexibly change the layout of anything and then when I hit the green button it turns into sort of play mode.
And now it's a fully functional iPod. And, of course, you can do the same thing with the same device and change it into an iPod touch or into a Blackberry or whatever your fancy is. You can imagine a future of display objects that has displays on all sides that have sufficient power to do the functionality that you want from the device and you just onload the functionality from a store. [Ringing].
And this is already happening, this is a Samsung camera that came out recently that actually has displays all over the place. We've been dabbling in spherical displays, vie con $20 globe. Bang co-did a better job of that with Microsoft sphere which was done in
2008. And here again a interesting question is what do we need this for. We built a bigger sphere trying to figure that out. And one of the interesting things about spheres is that not only are they social in the sense you can stand opposite each other and work together on stuff. They're also really cool for certain navigational tasks such as browsing information that's inherently physical such as Google earth here.
It also squeezes a lot of pixels into a very small space. So you actually have about twice the space requirements for a regular display. This is a similar kind of display, but here's the Coke can. This is actually an interactive cylindrical display. This is again projected from the exterior, not the interior. So we're seeing what that might look like some day.
Coke cans will be full of commercials. I dread that day.
Finally, programmable matter. This notion of displays that can actually actuate in the environment and change shape. One early example of that was lumen by Ivan
Poupyrev at Sony. But really still this is pretty much a pipe dream of the future and what we're seeing here is the Claytronics project at CMU which promises a world in which we have these displays that are real reality three dimensional displays where tiny robot pixels climb on top of each other and they have LEDs or whatever the technology might be to give you a hands-on three-dimensional experience.
And right now they're working on sort of the smart dust level, a lot of the problems are actually theoretical computer science, how can you build a structure like that and move a hole through it without, while maintaining structural integrity. How do you communicate with that many nodes of pixel, et cetera? These are interesting challenges that guys like
Jeff Goldstein are trying to address.

So with that I wanted to wrap up, concluding that here ends the tyranny of the task of the device. And begins design for the human organism. Natural high tech materials that use form for input as well as difficulties plays and designs that flow with our activities that serve more than just productivities.
Organic user interfaces, user interfaces are industrial designs. Input equals output.
Form equals flow and function equals form. If you want to read up on that please check out the communications of ACM Special Issue on this topic edited by Ivan Poupyrev and myself. I believe it's June 2008, and you can learn more about what people are doing in this realm.
And of course I would like to invite you to visit our Web pages. Humanmedialab.org for a live blog of all the stuff what we're up to. I've survived my battery crisis, and would like to open up the floor for some questions.
So if you thought this talk went pretty fast, that's because I was aware of the battery issue.
>>: Sorry, what did you say the website was?
>> Roel Vetegaal: Humanmedialab.org. Organicui.org gives you content of all of the communications of the ACM articles.
>>: So I have a question. So one side you argued for I think simplicity of having forms that are completely, the form itself has all the sensors built in and all the display capabilities built in. We can assume any function. But on the other side you're arguing how broad the human evolution, there's forms that have specific purpose, and the form itself is so integral to the purpose. I've not fully seen how those two ideas merge.
>> Roel Vetegaal: Does anybody have a Blackberry here? Or another phone, say?
>>: Sure.
>> Roel Vetegaal: So how are these two different. They're just made by different manufacturers. So if I want to change my sum sung into this kind of iPhone really all I should be doing is going to the Apple store downloading the iPhone app onto the hardware. I'm saying yes we need different form factors -- I won't steal that -- we need different form factors for different functions, but we don't need different form factors for similar functions, but we may need different software skins or different applications, in fact the app store is an example of that already where you can change the functionality of your device.
>>: I'm more thinking about the input part of it. The problem -- so if you have -- one of the nice things about specific forms is that they're very highly optimized for a particular input. And when you go to these nondescript blog forms, nothing wrong with them. But they tend to be -- they tend to resolve inputs through kind of suboptimal ways, I would argue.
>> Roel Vetegaal: I'm not arguing against that. I'm just saying that there is a space within this design space which is huge for having software-only functionality that when you have a display wrapped around the entire device, anything that has, that does well in a form factor that's a brick could be downloaded onto that thing and that thing would have accelerometers. It would have multi-touch interaction. It would have all these

GPS, whatever you can think of, already prebuilt in there, so that when you need to be able to use a different kind of interaction modality for that particular app you can. You know, with Google maps I might be using my GPS. I don't use the GPS for the mail tool.
So I think that represents really the flexibility in changing hardware and software and addresses a really important concern of mine we keep buying new cell phones. Now they all look the same and all have basically the same functionality. We keep buying new cell phones, and one reason is what I'm proposing it's industrial design. It's fashionable to have the latest greatest gear. Not necessarily because it's better, because a phone is a phone is a phone. If I can talk to someone I can talk to someone.
It's because it has the product DA die manned or this or that let's make it a software option allows us to mitigate some of the risk for increasing consumerism through design, answer your question? I'm not necessarily -- I'm actually proposing that we have different form factors for different kinds of functionalities, but that's not to say that the brick won't be one of those form factors, that you can do stuff within the brick.
>>: Large brick or small brick?
>> Roel Vetegaal: Exactly. Let's call it the iPad.
Sorry, you had a question.
>>: I thought it was interesting when you were prototyping with projected, prototypes basically on surfaces, projecting on to surfaces. I was wondering whether there was a certain -- whether you test the extremes surfaces people would be willing to interact with.
Did you ever check on water and see people say liquid display is interesting.
>> Roel Vetegaal: I will, if my computer will allow me to --
>>: In some ways I'm wondering if that's con straining our what our options because of what users --
>> Roel Vetegaal: I disagree with that notion. I think that users are willing to put up with a lot more if we were to offer them more. I don't think there is a very diverse ecosystem in computing devices at all. No, that was it. Yeah, there isn't really too much choice at all. I mean basically everything we have is either that form factor the form factor, it's either small or big. Some diversity, please. Then we can pick and choose.
So I'm not sure if we're still on camera, but apparently purportedly one of the reasons why the plastic logic display is not flexible is they thought that users wouldn't like it, because flexible displays are hard to interact with.
But I believe that they're not necessarily hard to interact with. They're hard to interact with if you use the wrong kind of interaction modalities. That's really the problem. How do we match the kind of sensors you need with that kind of technology with the kind of form factors you need. If you get that wrong -- you have to get everything right in these designs. I think that's a little bit different than the kind of design we're used to today, where we can make a lot of mistakes because we're fairly limited to what we can do and it's all planar. And people are fairly for giving because they think of it as a technology.
Once you start thinking of something as a soup bottle you can't squeeze soap out of it they toss it in the bin. You have to get the mappings right. To me that actually sounds like a challenge for human computer interaction. It also means better usability. Because

there will be an system of products that work and not work that get bought or not bought.
So, yes, I'm arguing for diversity.
>>: I had a question. With respect to displays that are shapes that are kind of nonflat or nontraditional, I mean, even consider flexible displays particularly being planar, except they're kind of bent in many ways. But with one of the experiences that I had with spherical display is that it almost calls for a completely different kind of data.
>> Roel Vetegaal: Yeah.
>>: And unfortunately most of the data that we have available right now is not well suited for nonplanar displays. So have you thought about the suitability of what items are really suitable for some of these displays?
>> Roel Vetegaal: We have. We had some problems justifying the hemispherical display, other than the functions I just mentioned. There are some really cool things you could do with, for example, art pieces where you have particle systems floating around the displays. There's cool stuff, cutting sort app doesn't cut it. Better for a screen. We took the opposite stands and trying to proving an experiment that spherical displays are no different from flat displays, just that the surface is larger and you can't see the other side.
In the case of a large screen, we have about this size of display, you can't even see the other person. There are, however, instances that were not looking at in this particular experiment that would be very interesting to observe. And that is when people move over or around the display and maybe you've seen that happening, where you can sort of have your own private space and I could have my own private space, but there's sort of this in between area where we could share information. And I think that's a really interesting area for study, where, with flat screens, essentially everything is public all the time.
And so we're doing a task in both the collaborative and competitive environment and see how that impacts performance. When you can see what somebody else is doing and when you cannot see what somebody else is doing. Essentially it's a flat screen with a curtain in between.
>>: On the other hand, if you have a dome projection, there's a heck of a lot more than there was just a couple years ago.
>> Roel Vetegaal: That's right. A lot of it will probably involve browsing or browsing-like datasets. That's a good example of what you could use this for.
>>: A lot more directional demand than there ever was.
>> Roel Vetegaal: Yes, great example for the complexity of dealing with sphere interactions is zooming in to an earth when it becomes flat. And so then the sphere should actually be flat because the data becomes flat when you zoom in. When you hit the street level you can get the street views, and they're great for spherical displays because they're omnidirectional like that. It's like a fractile. It's cool, because it relates to what I was saying about organic architecture. Sort of begin and again. You see the features come back where the form factor starts interacting with the data in intricate ways. I don't necessarily have solutions for that. I'm happy to take either side of the fence, say like spherical displays are the greatest things on earth and here's what we

move them for. The other side I could say there's really nothing you can do with a spherical screen that you can't do with a flat screen.
>>: I do have one question. CES this year, the dominant presentations were around 3D closure. Couldn't walk ten feet without seeing one.
>> Roel Vetegaal: Yes.
>>: That's obviously a three-dimensional image but within a two-dimensional kind of projection system, when there's interaction. Do you think that potentially is going to help this type of thing?
>> Roel Vetegaal: I think it's a bit of a fad. They really tried to push beyond HDTV, like, cool, we just sold a bunch of new TVs to everybody, but now the market is saturated.
What else can we do, they came up with 3D, I think mostly because lack of other ideas.
I think the Wii and interactivity is stuff that's much more interesting.
That said, I do think that there's a space for 3D, we are working on some of those things.
I do think that screens will eventually become 3D, but there are many problems such as what do you do in multi-user situations you want a viewpoint that's different from somebody else's. Lenticular screens addressed that to some extent but not really.
Once you're talking about an ecosystem of users, multiple users, 3D is kind of sucky.
It's a fad. It's going to go away. It's not transient. Next CES they'll be talking about something else.
>>: 4K -- resolution.
>> Roel Vetegaal: No, 4D. The slides that tickle you. Now they've got these 4D experiences now in IMax and stuff where you get sprayed on and rumble seats and so forth. All right.
>>: Andy Wilson: Okay. Let's thank Roel.
>> Roel Vetegaal: My pleasure.