>> Meredith Ringel Morris: Welcome, and welcome people who are watching over the Internet also. So a lot of invisible audience is also watching you online right now. So I'm pleased to welcome Brygg Ullmer here. Brygg is an assistant professor at Louisiana State University and many are probably familiar with his research in tangible interaction. Brygg has been a pioneer in that area. His paper on the metaDESK that many of you are familiar with was given a UIST
Lasting Impact Award, and he's one of the founders and sort of key people behind the TEI conference on tangible and embedded interaction which has become increasingly prominent in the past few years.
And so we're really excited to have Brygg here today to tell us about what he's been up to at Louisiana State.
>> Brygg Ullmer: Thank you so much, Meredith, and to all of you. It's a real thrill to be here. So the title I picked for today is entangling surfaces, tangibles, webs, clouds and architectural spaces.
This relates somewhat to a talk that, or a paper I published last year in
Interactions on this subject. There are a set of words that everyone is very intimately familiar with. The argument is that they're not purely independent but perhaps they're entangled and perhaps actionably entangled and perhaps it's even more actionably than actionably, perhaps there's a common fate in some of those words is one of the ideas I've been exploring recently.
To explore that I wanted to begin with some motivating entanglements through a colleague Freeman Dyson. Go back to some tangibles in pre-history and history and use that to talk about some of the work I've been doing in the last couple of years and some of the future directions I've been heading.
So in terms of personal entanglements, these are a couple of the different places that I've been at over the last few years leading to the present. One actually I could say more about some research dimensions, the work I did with
Clark [indiscernible] on behalf of Microsoft's behalf for a while. But I'll be saying more about these as I go along. Currently in addition to my role as assistant professor at Louisiana State University I have two major hats I wear.
One of them I do a lot of computational genomics work. So roughly half of my life ties into that. I'm leading a group, not my main choice, but the two previous successors were both British and I think they both had this --

wouldn't it be cool to be a director of the BBC. So I'm the interim director of the bioinfomatics biostatics biology core for a statewide biomedical research initiative.
And the other side I have ten faculty in art, computer science, electrical computer engineering, business infomatics, communications, music, and growing that are part of a group of cultural leading five divisions of the center I'm present at.
You'll see bits of that as I go along as well.
So for my outlook, when you think about points of momentum in human computer interaction, today, yesterday tomorrow as well, there seems to be a lot of activity in the physical and the online world in terms of social media, Bing spaces, you name it, physical world, ranging from the Kinects to the smartphones and so on and so forth.
Those are often viewed independently. But I think clearly there's an intersection there. Sometimes looser, sometimes stronger. And we can begin parsing that into several other subspaces. You can think about legacy forms.
The physical world is nothing new. The online world is nothing new. There have been modes of interaction in those spaces that existed many, many decades.
But there are also some very new specific forms. Uniquely new to physicality reaches computation to online reaches where we are, and especially this sort of intersection between those. And the prospect for bridge forms. What are the ways that bridge kinds of computation that we've known for a long while with the new kinds supported by different social media, physical entanglements. And so this particular space with the bridge forms and those intersections drives a lot of the research that has kept me occupied for quite some time.
Systems from my graduate period and the period at Sony. And then a number of systems that I'll be saying more about in parts for my post-doctoral work, work to the present.
Some of that has some slightly different forms, some of the computational genomics work that I mentioned and I'll refer to that briefly mostly at the end of the talk.

So starting at a point 1997 that is one of the more known papers that Meredith was referring to, my first project at the media lab involved the metaDESK and a couple of other projects. I'll show you in the next slide.
And was a point of departure. Had some interesting properties. I wish I remembered that what Andy saw more clearly than I that we actually with our vision system if we took off these objects and put our hands down we could actually have multi-touch interaction, but it was really, really focusing on the tangible dimensions about what these kinds of surfaces could be.
Something that we talked about with the tangible bits paper was several different points in architectural space. And a drawing saying that these at least co-existed. What I don't think we said and what I don't think has been the future of that work is that maybe again there was a co-dependence rather than just doing ambiance or just doing surfaces, what were the prospects if you more actively interlinked those together, more intensely entangled those as it were.
And I'll say more about that as I go along. After that work, I realized that I knew a decent amount about graphics but I really didn't know that much about physical objects. Hadn't shaped them as much with my own hands. And so I did a series of projects here a subset of those and the rest of my years in Boston as well as some work at Sony that really influenced.
And as I was going through that work, I was trying to again wrestle with this notion that humans are not new to the physical interaction with the world seemed to me on first principle if we learned something more about our last hundred, thousand, 10,000 years of interaction with physical objects that might give us a view towards the future.
And in particular, one individual I encountered at that time, Freeman Dyson, some of you may know, probably the majority not, has had some really interesting tools for thinking about the future. One of them, a book called
Imagined Worlds, a chapter called Evolution, he explores the idea: What can we say with relative confidence about ten years from now, 100 years from now? A thousand years from now? 10,000 years from now, 100,000 years from now? A million years from now and about the unbounded future?
And I think actually as I've been reflecting on this the last couple of days,
Dyson was entangled in the Manhattan project. He was leading something called

Project Orien had a reasonable prospect of having men on Saturn by mid '70s.
They had the technology right, actually, dropping off strings of nuclear weapons behind you to propel you on their force impacts.
But again I think part of his impact with that the future that led out led him to think a lot about the future. One of the simplest way he introduces that chapter let's think about the two shortest time spans ten years from now, 100 years from now. Ten years from now, his argument goes: We hope that all of us will be here. Ten years ago we were all here. 100 years from now, I don't see anyone in the room that's more than 100. I don't know, this is not wishing maybe Patrick will be in a different state, but in 100 years most of us will probably be gone.
As we think about what can be known and unknown at these different periods, but our families, families existed 100 years ago depending on which continent or others bits of our culture existed a thousand years ago, we can think about a thousand years into the future.
So a set of tools by looking to the past to see the future, and another sort of way, just looking a little bit at my immediate past, the activities that led me into that work with intangible interfaces into the present, certainly I had been really active with robotics and sensors, speech, super computing, graphics, super computing will become more relevant later, but I was also working very much in the context of the Internet.
I had built a first C-Span site on the Internet in the late '92 early '93, as I went to interval for the first time had sort of these two dual offers, one Mark
Andreessen invited me to be a person on the team with the Mosaic browser and opportunity to go to interval to work with Terry Winograd and some other people on what led me to the intangible space.
In particular with the interval time, I arrived as a 19-year-old, one of the first two interns there. And there was this interesting proposition that we were presented with. The idea that Paul Alan as Microsoft's co-founder and
David Lindell's interest was not really so much in coming up with a few new products, not even so much coming up with a few new companies.
The real interest was by putting a couple hundred million in if we could come up with one or two new industries, wouldn't that be interesting.

Turns out to be hard. And there was a really interesting tool also for that.
One of the probably three most influential courses in my life, two of them I took at the same time. One of them was my remedial admission to the MIT media lab. They said I wasn't ready to come until I did some remedial night school education in graphic design, which is probably the top of my list of the most influential pieces of instruction I've had in my life.
The other was of course that David was teaching at the same time accounting for electrical engineers. Of course he taught as a short course at Stanford but he was doing it at this time as an internal tool for the company employees.
And so there was a matrix, probably Boston Consulting Group, probably roughly
90. And the idea was if we're thinking about existing industries and new industries, existing products and new products. As best I understand this, and
I was confirming this with him a couple days ago, I haven't found the original source article, but especially as a new company and perhaps as any company, this is sort of a safer space, an industry exists, a line of products exists, let's do a next iteration on that.
Taking a new product or an existing product into a new industry or a new product into an existing industry. Those are more reaches, but also reasonable things.
Apparently the intuition, though, is trying to both create a new industry and create new products at the same time. At least from this early '90s perspective of the Boston Consulting Group, perhaps there lie dragons. Perhaps that's hard. So to me towards the TEI context that Mary was describing, I'm super, super excited about the tangible space. I was seeing amazing transformative stuff at interval when I was there. Turns out there's a much longer history as I'll share. But I don't think you can really say at present there's a tangibles industry.
And also to me, as I look around the world, it would be shocking to me if there's any entity in the world who has put even 1/10 the investment, human investment, product investment, capital investment, as Microsoft into this space, whether you look at the four generations of Surface, whether you look at
Gadgeteer and hundreds of other different things.
For me, having been trained my whole career, excited my whole career about new industries, enormously enthusiastic to see Microsoft be inordinately successful

in establishing, populating this new industry. Not only would it be good for
Microsoft but it would be good for the whole ecosystem to have success there.
So I'm really excited about that and excited to be here to learn as much as I can about activities that have gone on and perhaps in time hope to contribute to prospects in those directions.
So I mentioned tangible history, prehistory. Let me go backwards in time. So when Hiroshi and I were meeting up in late 1995 and doing our first projects together in '96, we hadn't known that much about longer histories in Boston, for instance, relating to tangibles. That wasn't quite true because I was sitting right next to David Small doing amazing Lego work and other things, but turned out over the next couple of years we kept discovering things. So there's amazing lists that Steve and others are familiar and he called reuse.
And one day the reuse list had this series of tangibles that were actually meant to be sort of internal small computing nodes doing super computing simulations of fluid dynamics.
In 1988. Really interesting tangible interfaces dealing with the computational sciences, an area that I spent a lot of activity since. Further back, how many of you are familiar with the name Danny Hillis? Anyone know about -- this is sort of giving it away -- but Danny Hillis's undergraduate work in Boston? So it turns out that by '76, Roddy Perlman, interspace in the Internet space,
Roddy said she wasn't good at cutting acrylic. She didn't have a laser cutter at that point.
So Danny did all the heavy work in terms of the cutting of these elements to make what he called the slot machine. Actually, the predecessor to this was called the button box. And the park folks talk about that with a pigmalian
[phonetic] thesis DC Smith as one of the inspirations thinking about icons in the GUI space but the next project was something for 5-year-olds that let them do recursion, complex conditionals and really program that with both their robotic turtle as the screen base sort of turtle. And so this again was really educational, partly because I don't know of many projects in the last 30 or 40 years that have gotten to the level of expressiveness, of expressing recursion conditionals, et cetera, with tangibles. But they did it with an interesting mixture of visual forms, physical forms, connecting those in different sorts of fashion.

And Hiroshi and I, for instance, were really steering away from text, steering away from diagrammatics, those were two traditional, focused on the physical form. So to this to me was really educational. Maybe there was something we missed, opportunities lost.
1933, I don't know if many of you are familiar with at least the history of the
Vinylite house. But I really enjoy this quote, this idea, special vision of the future where every surface, every artifact, aside from the humans themselves, was a plastic. And on the one hand that's sort of familiar to us.
We see things a heck of a lot more like that today than it was in 1933. What does this have to do with the future?
So certainly there's the prospect that a decade from now all of these, all of those, all of those are Foleds [phonetic] other display elements, to an extent,
I think even in Bill Gates own home that's been around for a long time. King
Abdullah, KAUST, a lot of experimentations, and a lot of trends that suggest we're heading in that direction, how would that be to live in.
And there was an assertion that maybe that actually wouldn't be the optimal space to live in. It turns out how many of you are familiar with the design heuristic crap? Any of you? Super. You from my classroom. Super. So contrast, repetition, alignment, proximity. Contrast. There usually used in terms of graphical contrast. But certainly we see a lot of materials contrast, that this room is different having this aluminum next to this fabric, next to this other. And then again in terms of mediums and message, mixers of dynamic surfaces with passive surfaces, how do these different elements mix. There's some beautiful aluminum bezels in the active columns in your art space there.
And that fits the crap heuristic very well, but again asks us: What are these interesting entanglements? So something we've gone through and courses there.
People when they look at tile displays, when you were doing your hand-helds talking with the tile display projects, looking at the bezels as a bug.
But maybe the bezels are interesting features, because actually smudging your pizza stained fingers all over the glass have limits in high traffic spaces.
So maybe really focusing a lot of emphasis on the bezels and the kind of interactivity that might want to live there towards the crap heuristic. That's even better than having the bezel-free sorts of versions of the room of the future. At the same time I do look at these walls and they do look so naked to me in some ways and thinking about all of the entanglements this space could

have. Preferably from my eyes I'm really glad they're not high resolution screens right now because if there was a remote audience and I'd be seeing them my eyes would be dragged over there and I would find it very difficult to concentrate.
But if this was a massively online course and there were 100,000 remote people,
10,000 different places, are there ways with the lower resolution servos or
LEDs and other things to be aware in different senses? Like with Skype.
I taught once together with Miriam a course on tangible interfaces at Hong Kong
Polytech, and a couple of years later we taught it remotely from midnight to
7:00 a.m. in the morning, Saturdays and Sundays in Louisiana to the folks there. One of the things with dealing with those audiences in that timeframe I had to be far more interactive, asking far more questions to keep them awake and me awake in that context.
Again, if you're thinking how does Skype let you engage with hundreds of thousands millions perhaps an Olympic context, billions of people, what's the right medium if it's not going to be a thousand or a million faces dealing with the technologies and deal with [inaudible] message that becomes important and interesting.
This object. Something I also really enjoyed discovering from a migratory tribe in Africa. So my understanding is from a culture that was oral in nature, each of these objects -- in a sense this is a kind of subway map, a subway map of the last few hundred years of this particular tribe. I haven't found really good sources, and I certainly haven't interviewed any primary users of this. I've bought interesting things related to this if people have interest.
But if you can imagine this representing a kind of mountain or water resource or different elements there, and you can say what if I put a Microsoft
Gadgeteer in this, just a Gadgeteer wires and nothing else? What could that do?
Well, if it wasn't for the heart attack of a grandmother of a student, I was going to send one of my team members to interview, one of the last five speakers of the [indiscernible] language on LSU's campus, we have a
500,000-year-old Indian mound, and this idea about there are five people left who speak a language of a native tribe in our space, and ten years from now

none of those five people will be left. Most likely. They're all upper 80s, early 90s.
What if we just connect wires from each of these prospectively metal pins to those gadgeteers backs a touch sensor. For speakers of that language or nonspeakers of that language to let that for the blind to be a haptic map for recording stories, accessing stories, this sort of tradition, and even towards something I've spent a fair amount of time looking at, maybe not so much. But the scrapbooking movement. Maybe seven years ago passed golf in terms of number of active practitioners. So all of these sort of Robo knives coming out to help people with scrapbooking, maybe these things with the Gadgeteer and
Surfaces let people craft objects like this are most alive when they sit on the fourth or third generation of the surface, or core with Gadgeteer or both with each addition because the Gadgeteer could be functioning 50 years from now, most of the fourth generations won't be functioning 50 years from now. This idea how do we got we very familiar with paperback, hardback additions of physical books but with tangibles, who is producing them and what are the different editions and are they mediated and how could those be entangled. I won't spend too much time going back 10,000 years ago which I think is a favorite place, has a lot to say about the Euclidian spaces, the tangibles, primary way of doing business for roughly 5,000 years of human history.
But this one seems very, very relevant. My favorite is your faster core image.
And so as you look at this, with school children, mothers, merchants, street people, senators, all interacting on this counting table, which there's some evidence in another set of pictures that the tables were used interchangeably for gaming, early forms of backgammon and more accounting, how business was done.
To me, I see a great deal of the future of surface computing in this image from the past. For example, there was a popular thing 15, 20 years ago taking different notions, Apple interface guidelines and inverting them. One thing if we look at that table and look at some prospects, here is a user experience guideline for Microsoft Surface, must in terms of orienting always to the person involved.
And if we look back at this, we might say, for instance, that this failed that particular property. Now, probably in this original incarnation they weren't looking at large interactive display elements. So maybe that was the only way you could do things.

But I think there's also a real possibility some of the work that Miriam, myself, Ally Mazalek at Georgia Tech are trying to do are trying to take invert that and what we say actually what we're looking at when we see this this is
Excel. Tabula, the word for about a thousand years, they didn't use abacus.
That was the Roman columns tabula were the way this way computation was done before paper was common. After paper was common.
So maybe again if we think about the future of Excel, maybe it's woven into these sort of conference tables and other sorts of elements and being able to have many things that do have a specific orientation. So here with the concept sketch something we've been proposing for computational biology, computational genomics. We're letting people quickly flip the orientations to orient the text in different ways and really taking very highly structured maybe you could say one challenge of a number of tabletop interactions is that they've been not only insufficiently entangled but insufficiently structured that maybe that structure, if we're dealing with Steve's sort of large scale information visualizations, all at Bing. If we've got billions of different elements that are always reorienting, reorienting for some things exactly the right way but for other things maybe that's been a challenge that's been hard in some ways to get over.
Or the last element. I know one person in the room who can read these hieroglyphics, but anyone else for any of these hieroglyphics elements, things that are legible?
So to me the first time I introduced programming, I start with this slide, and it relates to my German and my biology and a lot of other sorts of things. It turns out that anywhere you see something like this, that's a hyperlink. It's naming something which is both virtual and physical, a woman or man who is both a human and a God. So anywhere you see those, whether here on the rosetta stones decoded by a bunch of undergraduates here on an obelisk moved over to
Rome here on ceilings on jewelries, that's a hyperlink. And to me it's sort of interesting to compare that with a QR bar code and say are we further ahead or were the Egyptians.
Why? Because I really think if it's not already the case that most mothers and fathers will be telling their kids two, three years ago, don't you dare ever, ever load a naked QR bar code, because how do you know that that's not the virus that's going to take down your device.

You would expect a disproportionate number of those things would take you places you don't want to be. So again where I haven't studied that much the legibility here, even those who don't know how to read the rest of the elements we can find one or two subelements and compare similar to different to other elements that were on this here, this mark here looks similar to that mark.
Maybe that's actionable. Maybe not. But this idea about Cartouche's in several different forms that's perhaps the most common. But here if you ask a cartographer what's a Cartouche, they'll say that's a Cartouche, that, those are Cartouches, they're the meta information that makes the rest legible. And they turn out to be really important. Because, for instance, if you don't have this element or this element, what's the nearest century. That's not necessarily especially if I were to continue north, you wouldn't find the fidelity reputation of Seattle that you might like today. In fact, I'm pretty confident on that map Seattle doesn't exist.
Yes? So those are important Cartouches, these maker marks in different sorts of ways whether it's on weaponry or other sorts of things those are notions of
Cartouches as well. I spent a lot of time thinking, if we want to create structures that don't have an expected lifetime of two years but perhaps 20 or
200 or 2,000, what are some of the different tools or crossing many different widely divergent cultures where Microsoft like almost no other company does that in terms of their computational things, what are some conceptual tools, physical tools, technical tools that could take us there?
So I'll move more into the present. The way I've structured my research group over the past, well, eight years at LSU and two years as a post-doc has been taking this sort of interplay of tangible architectures and tangible applications. My engineering side is sort of on the one side looking at the physical dimensions.
What are the pages, the books, the walls, the buildings, what are ways we can weave these technologies into the spaces in which we live, professionally, domestically, in other ways. We've spoken about the Cartouches, what are some concepts that could generalize that aren't specific to tablets or embedded technologies with gadgeteers, if we want to view all these areas as actively entangled, what are some ideas that cross cut. There was a great idea that hasn't developed into a paper yet presented the future of research funding for technology at NSF. The right answer is never one is the theme. The idea in

that talk, I'm blanking on the guys's name, he co-led, structured on one mouse and one keyboard and one screen and one element. What if the one thing you do you put an exclamation point in front of it and never say just one, never say just this surface, but pluralities of different surfaces, most of the things we're doing with hard-backed books, have a Microsoft Surface, Android, not a
Microsoft Surface inside, and this idea that heterogeneity with different application space, is actually more interesting, harder but more interesting than having this same kind of device in each of those two different covers.
So technical spaces, if we're trying to, as I showed pictures earlier, transform large walls, other sorts of things, whether with Gadgeteers or other sorts of things, what are some scaling mechanisms that let us think about rearchitecting this room with hundreds of thousands of different little points.
It's not that difficult to already, if you take the largest, well, Microsoft, few 100,000 employees, right now we've got a million pixels there. So certainly taking each pixel as a person is possible in this space.
But, again, if we want to do that in your atrium or do that here and we want to do the architecture that can live adapt, grow through time, there's some different kinds of technical tools, different types of data models. So in the tangible space I don't know if I've seen a single paper, mine included because the one I did it got rejected the first time around in the TAI conference, that really talks about equivalencies to HTML and how do we represent data.
One of the whole goals in the TAI conference is that we have thousands and thousands of different point systems. But what was done with Microsoft Windows
1.0, Apple Macintosh, the star is saying maybe individual applications don't get you there. Maybe it's only when you have this ecology, this conceptual functional critical mass where you actually have a usable space, whether that space is this big or a larger building.
And I'll say a little bit later about how interactive super computing could relate to what every one of us does. Certainly as you touch a key on Bing you're interacting with an interactive super computer, right? Thousands of different compute nodes, millisecond by millisecond come up to project how is that next interaction going to play out. My interest is really in the tangible space, physical space, surface space, what are the equivalencies of that sort of computer-based interaction, all of us know from Bing and Google and other sorts of things.

And to ground that, I've been looking at the two particular subspaces. Really mostly this first one. If you look at where the Web came from, [indiscernible]
Bush was writing about computational science. Science in a sense. Tim
Berners-Lee working at CERN was big science. Mark Andreessen for super computing applications. All of these people were talking about things that they hoped and it did turn out generalize but at least on the academic side where you're trying to fund the work, get the work, heavily used in places that could afford it, the ability to sort of dive into some particular computational space where the science, the technology and engineering and arts or mathematics can only happen through the thousands or millions of compute cores that are bringing that to life.
Part of that idea also is that every one of your tablets or laptops, they all today have more than one compute core but most of them are fewer than ten. And so the amazing flip of Moore's law that happened about four or five years ago is from the increasing clock to increasing core count. And so if Moore's law predicts roughly a power of 100 per decade, I think it's very reasonable to expect that every computational device to the extent it still exists a decade from now has 100 times the number of compute cores.
And if we're still thinking so often in terms of single threaded modes of execution for our interfaces what if we sort of flip that and have to think about in something that scales to at least something in at least 100 compute cores. I'll talk a little bit about that as well as attempting to generalize this work with the work I mentioned with native Americans or thinking about blind, other people, other special communities where we can generalize out that in different sorts of ways.
So my first exercise post-Boston with that, I worked on a large grid and cloud computing project involving ten countries in the European Union. And they were two sorts of tasks we were looking at. One of them was numeric relativity, colliding black holes. It turns out that the National Science Foundation has invested more towards understanding colliding black holes than any single other research project, roughly a billion dollars they've spent. There's this interesting facility in Louisiana coupled with an interesting facility in
Washington, one form of very active entanglement between our two states where your facility and our facility are several miles long by several miles wide.
Acting like something that's at least the size of the continent of North
America. And they've been trying to put one in India or China so they have something that's the size of a globe. Looking for colliding black holes,

because if we don't find them it turns out Einstein will be wrong with one of his major predictions and we'll have to sort of get back to the drawing board of understanding gravity as we know it.
It involves a heck of a lot of computation, or in that case biomedical application. So one of the sort of two or three side projects I worked on in my two years in Germany was with a medical doctor who was in the business of breaking people's faces.
Now, most of us I don't see any one in the room who looks like they're aspiring to have their face broken but if your jaw runs at roughly 90 degree angle where it runs currently there's a set of people in the world that have maxiofacial cranial surgery that need to have their face broken into five or ten pieces rearranged and some of them actually have an interest in what they'll look like when their face is reassembled.
So existing project was advanced revelations of how the tissue folds over -- but the interesting details were the doctor wasn't in this lifetime going to be mastering those computational techniques. He was already working his 100 hour weeks and there weren't a lot of times to master that. The patient is a vested party. The patient wasn't going to be mastering these techniques, how could you be thinking what are some of the kinds of furniture in the waiting room of the doctor or other sorts of rooms where there are a couple different cards that have things about the patients and kinds of different protocols they're working through to make it where just in a couple minute interaction professionally or couple minutes with the patient where they can do what ifs and see brought to life on the sort of the R theaters what their evolving faces would look like. This was the project, taking Cartouche's idea how do we mix the visual and physical representations and how do we take what you have seen earlier with the parameter tiles and externalizing it in some sort of fashion.
In Boston -- sorry, in Louisiana, when I moved there, I saw a lot of work in large scale video conferencing, especially with these big science projects.
The first big science project where I actually had authorship status, two cover articles in Science, the main article had 250 papers or 250 co-authors. I was at the low end of the time I put in. I put in only 700 hours personal coding into that. My wife put in several thousand hours. So that was again big projects.

So it's a business about how we communicate the kinds of science, the kinds of weekly conference calls, that's actually a pretty small project. My wife right now is on a thousand genomes project where by now 5,200 different human genomes, verging on the million and billion genome projects and figuring out how to have these discussions and make these things work. You're seeing in
2007 where we're putting these different sets of tangibles in this space, in the remote spaces coupled with the visualizations, all towards the goal of how do we have a good argument. And the main access grid things, all people from the things, didn't control their own PowerPoint in the early days of access grid remotely saying next slide, next slide to the tech in the back.
If we can put enough things especially why the physical things, if the eyes are supposed to be on the people on the data and not the device that's here, so this even connects so one of the patents I had at Interval, back in '95, we had some amazing tablet interactions with large screen televisions to have much more active entanglement of Internet data of any project I'm aware of today because we're not geckos, it's really difficult to have our eyes equally on two radically diverse surfaces at the same time. So maybe we're whether driving or whether we're driving our cars or driving these devices we ourselves are functionally blind. Yes because we're trying to let our fingers do the walking while our visual attention is allocated elsewhere. And so this is exactly that case. Now we're actually putting the Microsoft Surface tablets other sorts of things as mediators but still trying to give some physical controls to help us keep our visual attention to the places where the action really is and then let our fingers do the walking. There with sort of one page and now with some that
I brought with me and I'll show later having five pages, 10 pages, 20 different sort of haptic constraint pages whether it's for a blind person in the visual sense or a blind person like all of us when our attention is allocated elsewhere have done a lot of different mentions and I think in the interests of time I'll skip over these quickly evolving with these Cartoucian negotiations, how if we want to do these technologies that function on the tablets and tables and embedded electronics and especially designed to function with technologies that won't exist for another five or ten years, what are some of the core concepts for how we represent content, how we represent people, how we represent activity.
And again I can come back to any of those if people have interest. Just one example of that, which was a couple month summer project right after Deep Water
Horizon, turns I moved to Louisiana six months before Katrina, had a first whole round of tangibles that were about kids interact with kids in North

Carolina to interactively change the models of Katrina to understand that. It was a nice case for computational science, because there was a super computer that was used to do the storm surge models. A lot of scientists were very angry because they had computations running for months that were killed off for this wild and crazy idea that when a big hurricane came, maybe the priority should be simulating the hurricane, even if their code couldn't checkpoint and they were going to lose three months worth of week. So Deep Water Horizon, too, having ideas first gen table, some of the devices made directly by the surface, some mediated by the iPad and other ways. But the idea of tearing off different entangled elements because as you know maybe four, six, eight, ten people around one of your commercial surfaces but certainly if you're scaling to 15, 50, 150, a thousand fifty, 10 million, those sorts of coincident models of interaction start breaking down.
And so some interest that again I could refer to here again on some custom electronics versus mediated directly by the Surface versus one that's mediated by maybe a $50 digitizer surface that's on the way to being $5.50.
Computational steam. My largest grant right now, around $1.2 million, 40 different investigators from 12 different departments, is saying if you look at
Ivan Sutherland's work, transformative sort of first dissertations on what could computer graphics be.
I believe TX-2 if I'm remembering correctly was the name of the computer that was involved. Does anybody recall what TX-2 was used just before Ivan
Sutherland got his hands on it and just after. It's a batch super computer that was briefly co-opted to sort of do this wild and crazy thing of interactivity. More than $300,000 in current mode. I wasn't adventurous enough to write it in the proposal, but the idea what if we get a large several hundred core processor and pretend that's our PC?
I have a really good way to use that currently with my genomics folks in lots of different areas of computational science. That's what they do every day of the week and classroom right now using two cores but if you want to simulate at scale and let it be interactive than you need 100 or a thousand. That's a great motivation.
So again here, and I'll show you as my last slide a couple other elements of --
I've actually shown you some of where we're taking that. But last three slides of sort of the future. I had this interesting opportunity at the KAI PC

meeting the time before last where Hiroshi's first question to me over dinner was Brygg, great conversation starter. How do you want to be remembered when you die? I said Hiroshi, at that point I'm 38 years old, I'm hoping I may have a little bit of time to sort of figure that out.
But the next morning, we're on Lockery, I ran into him. He said Brygg we'd love you to write an article for Interactions. The request is it not deal with anything remotely tangible or anything you've talked about in the last 20 years. This is a couple-month project, the figure you'll see a current
Microsoft employee, holding some different things.
But with large scale super computer, an idea for 50 years having a head node, the place you textually sort of interact. We were really interested what if we can have face nodes so as we're walking down the hallways, we're aware of remote people, remote computation, taking those invisible things inner workings of Bing and those things making those visible. Intangible things tangible.
And here at the top it's really rough so you're seeing me, as of two weeks from now I'll be required to spend 80 percent of my time in each of four of my buildings and of course the numbers don't add up. The students always chasing me, I'm always chasing the students and how to be aware as I'm walking through the halls in different sorts of places. So here you're seeing some very emerging lighting legend. This is tough although it's the most important thing does the color represent the Louisiana digital media Center Patric Hall or
Johnson Hall that give me nothing but anyone but you, but tangle that with my calendar. So it's not really truth but at least for that which I'm already making publicly aware, can I begin letting people see for all the different faculty, all the different students, both physically and virtually, what's my
Skype status, what's my I-chat status, do how do we do it with the people and the data and for the cloud-based representations and here for the computer representations you're saying STEAM, technology mathematics and here days of the week and times of the day and seeing which percentages of the super computer we have are allocated towards computational arts which is actually our huge economic driver and the thing that's sort of helped our center. We are building moving into the week after next, the third floor is EA Sports is the first floor is my center and the first floor is both and by tax credits IBM said they're moving 800 employees to Baton Rouge, funny tax credits related to movie industry, game industry other sorts of things that have been again attracting all these things but how to mediate that and to show how that super computing, whether it's touching the film rendering or other sorts of things.
So again I could say more about that later. And also the pull-down surface

that we're having on the gas springs to sort of float back and forth on the wall.
Books. So here, and this is another sort of unpublished element. Had this interesting meeting with some French and German folks. So there are three countries on the planet. None of them being the U.S. That have sort of larger consortiums of researchers doing tangible intangible things. The
Surface entangled activities, but 15 different institutions and 10 institutions. In the German and French cases strangely perhaps, maybe not in other ways, the publications are published in French and German in the local conferences. So I was brought last summer to try to help them team up to do some big summer schools, big interactive workshops, but there's this challenge if I don't speak the other language, it's almost like this blind case again over that Meredith mentioned to me, how do I sort of navigate more fascially than we can do today with Microsoft academic or Google Scholar but to dance across the different sorts of literatures. So to me perhaps the pictures are universal in some sort of sense. And so here to let someone touch 2010, 2013, see all of the institutions, see all the venues, where tangible entangled things sort of bulk exported from the ACM digital library to [inaudible] to relatively legible form compared to alternate representations, and then start drilling down. And again here is a two surface interaction, one on the right that's showing us the details. One on the left that's mediating that, but it's the same if we're doing this with tele interaction, too, having bunch of people in different rooms and having some interaction or lapse. Some on the shared surfaces. So this one has exactly the same layouts. This is how I spent my so-called last Japanese vacation. My wife was invited to speak for a week so I hung out in the Kyoto hotel and the Olympics were playing. There wasn't sumo but a lot of things the Japanese were interested in. Maybe I touch gold and I see all the countries light up that have a gold medal or all the different diving medals, some sort of intangible interactions. This is the idea of the book, several different pages.
I have some of it here and I can grab afterwards. But tearing those pages out of the book and putting them back on the wall. And so with these magnetically coupled things we're trying to see how these occupy larger, more persistent pieces of real estate. And let that persistence -- if tangibles are fundamentally persistent let's treat that as a feature rather than a bug. It isn't always. But here perhaps maybe this doesn't. Maybe we care about the groups in Microsoft Research, about time, what was happening, or was something happening before. Will something be happening after. If something is

happening after the room has a different dynamics than if it isn't happening after. Why not have at least that presence. Right now there may be zero people remotely watching or larger numbers of people remotely watching. When I defended my dissertation it turned out that largest number of people attending my mom had 15 people from her bridge group watching remotely I probably would have explained things differently if I knew there was a large contingent of
60-year-old bridge players that was important I didn't need to see their faces but I needed something to see about their presence and accommodate to what I was doing to that context.
And sort of the last element then again I've already told you about the tabletop and so I'll just show you what we showed at super computing last time which was starting to tear the pages out of the book put them on the surface and see interactively with the different videos and other sorts of elements how those tangled elements starting to do more things around the bezel which you would have seen earlier and I could say more about.
>>: Design galleries here?
>>: Yes, that was actually the very first concept drawing I had with metaDESK was taking sort of a Scrabble board metaphor like this wrapping around the outer grid in case with the design algorithms, CT scan, is it the reputation, the bones, the tissue, the blood, the answer is, of course, what's the answer?
Which is of course it depends it's the answer to most things. So trying to make that actively entangled trying to make that accessible and wrapping the people or the projects, the more stable things around the bezels, but, yes, that is.
So last slide before acknowledgments and questions. Again, trying to begin making this argument about engaging, leveraging what I do see is these entanglements between these very different kinds of systems but systems that are perhaps much stronger taken together than they are individually. Telling you a little bit about the story of how we've been diving, even though probably there are not too many computational genomics people in the room but this was
Vanderbar Bush's case, Tim Berners-Lee case and a set of cases that people understand about that a tool may generalize very broadly that's why you have seen the dual pronged approach with the last decade half my life in the computational science and half thinking about the generalization and concluding with the four surface Gadgeteer nexus and natural recognition and beyond I'm thrilled and deeply appreciative for Microsoft for the investments you've made

over the last decade. I do see if there's an industry to be had and there's no question if there is an industry, the question is when, but I do see Microsoft as having a push those bounds further than any other entity on the planet.
Super excited over the next day and last night to learn more about that and if
I could perhaps contribute to that. Lots of different hands, minds behind all of this but for the moment I appreciate your presence and attention and would love to take any questions.
[applause].
>>: I was going to talk about the STEAM effort which is I guess -- we always heard STEM but you're adding art.
>> Brygg Ullmer: Not my doing. First NSF funded adding the arts are actionably entangled. Would not disagree.
>>: How is that kind of playing out? Is that sort of federally funded across different labs or is that -- is that your kind of center that's focusing on ways to teach that? I'm not sure how that fits into --
>> Brygg Ullmer: Good. So I think it's a complicated tapestry I could take a number of ways in. At NSF it was creative IT. I thought it was pronounced creativity or something like this, that was trying to fund week at the intersection at arts and cognitive science and actionable work with the sciences that was funding my work-related to K through 12 entanglement of this work bringing it into the science center and so forth so that's quite independent for me. The word STEAM has got some momentum, workshops at RISD, other places saying how is the art side not about making something pretty but as you were arguing last night that that's existential perhaps to articulating making that work, legible, actionable. And perhaps aspirational and inspirational as well. In my particular case, again, because my wife and I, with our own genomics work, a lot of the computational folks, but more and more the education of the art folks, and trying to use Maya and 3-D studio max, whatever at scale taking that into the classroom. And letting the instructor again have their full attention on the students, it's really, really difficult to have half your mind debugging the networking and the other sort of console interaction while you're trying to let alone this classroom, especially if you're trying to have multiple remote classrooms that are actively entangled that's where we're trying to say can we put the Surface tablets on the desk in other sorts of things, give a few more persistent elements that channel my fingers to activate a few functions to change a few parameters, some tiny

percentage of the functionality but if that's infinitely more than none of the functionality in the classroom setting both for the arts and science and the other sorts of domains, with the conference calls, the conference calls for the thousand whatever genome projects they go purely audio because they have at least 50 perhaps more than a hundred co-participants and I guess you could say they've just not caught up to the video conferencing or perhaps having the video conferencing with a few hundred participants is challenging. And so what are the alternate mediations then? And the alternate tools if we're really trying to -- they're barely at the stage they send around their PowerPoints and we all load it up and we say verbally where are we going to go and we watch that dead data. My interests again both for that as well as for the arts cases is to make that live data. And let people figure out the clutching mechanisms the hand off mechanisms the raising the hands sorts of mechanisms link that to the high performance computation, people are doing the interactive super computing every time they touch Bing and Google why not try to think about the next couple of steps as that permeates other types of applications other than search alone. So that's a half answer and I could try to dive more into sub elements, but it's at least half hopefully. There were a couple of others, yes?
>>: So is -- do you think tangibles is an industry or going to be an industry or what's your proposed trajectory of that?
>>: So like big hurricanes in Louisiana, like large scale information warfare,
I think partly just because of the last 10,000 years of art, human history with objects and the fact that I'm really excited, all of us having bodies, I think that's a feature, of course, rather than a bug. It's not an if, another a when. And which when I think again are you, the questions I was asking a lot of times this morning are you approaching something mostly from the commercial case, from the domestic case, from the government space, from the public, I think that there's not to me what I see in my mind as an industry that recognizes itself as an industry. But to me again what -- I hadn't really realized until I was talking with Nick Vilar in Barcelona that you guys really have productized the Gadgeteer and I think it's really interesting to, with one eye looking at Gadgeteer and the other eye looking at tables and tablets, say what if I fuse those two together and are those two independent things? One of them just about people who are gidgeting around in their sort of home lab and the other about people who are running Excel on the airplane as I was doing I think I did all of this on my Microsoft RT on the airplane so I really appreciated that aspect. But I'm interested in how those fuse together and how

I think they can be much bigger taken together than either taken individually in themselves. I think especially the big lesson I learned as you guys were funding me to help defend one of your patents, not your patents you were wanting me to break your patent in order to not have a big issue relating to the bottom dock on Windows XP. And so that was what I was being funded on your side. And this whole idea about window managers, you know I think that window managers, we punt with the tablets and say let there be one window. But it's harder to punt and different to punt when you get to the table. And just as sort of the overlapping versus tiled windows was the fundamental existential emergence with GUI and perhaps the idea punting with windows as we came to tablets, getting the window manager right in that particular case, and that's again something I think isn't quite right but how that could entangle with the things on the bezels, for instance, that maybe it's the gadgetee are that runs the bezels and tablets with the gadget and if you take it together it gets you to critical mass applications that without the bezel maybe it's harder to.
That was a pretty long and indirect. I think the answer is saying I don't think that there's a tangible industry in motion today but I think you individually as a company and a lot of other companies in that ecosystem have all the pieces that with that right constellation of applications, it's sort of like 1993 with the Web browser again, all of the right pieces from the modem hardware to other sorts of things are in place it takes some knitting together and showing some really aspirational applications to show where that wants to go.
>>: Briefly a book project we had different kinds of tablets, meeting up the dual screen. Could you go to a little bit more talk about that.
>>: Yes. So to me being an interesting moment of our physical books on the verge of death. To me it starts to say if each page of that book becomes
10,000 websites, 1 billion genomes is live and forever live whenever I take it out having the book be more legible a decade from now as the hardware catches up to the book here's an example and I have only -- I left most of the pages I was trying to bring. But here is a page and this page is just something I sketched in turkey for an abstract parameter study that works on a big surface.
It works on a small surface. It works in between to sort of haptically guide your fingers. First time I brought this to TAI in Cambridge England I handed it to a blind woman and it was the first interaction Sheila, I can't pronounce her last name but she avoided tablets like the plague but suddenly back from DG days in the knob she could be fluent in manipulating things, here's the case where I was dealing with LED strips to illuminate whether it's the one line or

whether it's the four lines that you saw here with the different pages, having a series of pages, using the NSFC. So at least with the Surface Pro you've got a USB port I can put RFID readers, one to identify the book of pages and magnets to figure out the page and fully existing hardware can deal with these pages full of actionable entanglements that work there with that but work there with other tablets and work on the other walls in other sorts of places so we're trying to set up these hallways please take these objects away please make them into the, this one, you know, giving away the sorts of clothing most of our tablets and phones have only one set of clothing if any wanting to just likely give away T-shirts can we give away these sorts of housings, maybe giving them without the electronics, but then letting you plug into the
Gadgeteers to make them more and more functional on their own. So that's a little bit. And here actually assembled at the 711 so I had Google Amazon mailing me all sorts of fabrication technologies in my Kyoto hotel room and printing them out at the 711. You're seeing these pages align with this page and so if you've got ten pages there, you can touch through to the tablet underneath, touch through to the capacitive sensors around the LED strip underneath by having the tablet on both sides, then again you can flip through a series of things having the mediated on the right. Mediated on the left. So and emerging sort of thing that we're trying to use -- in terms of the eating your own dog food, I've sort of shifted a little bit from this to selling lots of things in the tangibles conference, lots of things in the ITS conference, and if charity begins at home trying to do something that's really useful at least to the researchers in a community and if they like that for selling their own work, maybe again we can generalize to other sorts of content like the
Olympics content or LSU sports content or other things beyond. So that was the sort of again a mangled answer. But hopefully at least that's a little more clear with some tangibles in the hands.
>> Meredith Ringel Morris: Probably need to wrap up. Thank you very much.
>> Brygg Ullmer: Thank you so very much all again.
[applause]