>> Hrvoje Benko: All right. Well, it's a great pleasure today to host Daniel Wigdor. But before
that, my name is Hrvoje Benko. I am a researcher here up at Natural Interaction Research
Group, and I've had the pleasure to actually work with Daniel on several occasions in the past.
Daniel is a fantastic speaker, so I think we are all here in for a treat. Welcome Ken, I know it's
early. He is currently a professor at University of Toronto. He was also a student at University
of Toronto, but he did this in a roundabout way of going out. He has an illustrious research
career working for Mitsubishi Research Labs, working for Microsoft, he worked in a Surface
Group as well as has a stint here at Research we did a bunch of projects together; and in the
meantime he also worked at Harvard, I believe with Cha Shen[phonetic], on the different
projects there. So he's been around. Many people here know him, and he always gives some
fun talks. He's been really active in the touch and gesture community, but he'll tell you right
now that's so passé and he doesn't work on that anymore. So now he’s turned his attention,
laser focus on a new topic for him which has been developing tools and applications for kind of
ubiquitous computing scenarios. He calls them, what, Symphony of Things?
>> Daniel Wigdor: Symphony of Devices.
>> Hrvoje Benko: Symphony of Devices. Sorry. I messed it up. One thing that I tried to do
correctly I messed up. Anyways, so without further ado I'll give you Daniel.
>> Daniel Wigdor: Thanks, Benko. It's always so fun to come back to 99. Thank you everyone
for coming out. As Benko mentioned I’m now a professor at University of Toronto and codirector of the Dynamic Graphics Project Lab. There are two major projects that my group
focuses on. The first is high-performance interactive systems and purging all latency from the
world; but our early collaborations were all with MSR, and I didn't want to repeat things that
people had seen before, so I'm not going to be talking about that today. Instead I'm going to be
focused on this Symphony of Devices Project that Benko told you about a little bit.
Now I should say as a professor my job isn't really to run my own research projects directly
anymore, right? My job is to go out and find money and to help other people to do exciting and
fun research. Really more of a middle manager than it is a research dig. But what it does mean
is that I get to work with a lot of really fun students who have done wonderful projects, and I'll
be sharing four of them with you here today. Stephanie, who is a Master’s student, and did she
did a field study that I'm going to be telling you about shortly; Peter, who has just moved onto
his PhD and is now interning at Yahoo, did some of this work; Anthony, who is right here in the
room, was interning at Autodesk and I had the privilege of collaborating with that team and I'm
going to tell you about that project as part of the larger Symphony Project; and of course Jishuo
as well who is also my Master’s student.
So the talk is largely comprised of these four papers, so if you want to go back and get the
source material, that's what it is. So Anthony, in his talk, had what I think is a really startling
contrast that I think is a great way of proving the point that the error of ubiquitous computing is
here. So this is the people conclave for Benedict the 16th in 2005, the crowd eagerly awaiting
the announcement; and this is Pope Francis in 2013. Notice the contrast. Can anyone really
argue that the error of ubiquitous computing if not having arrived, if we haven't yet achieved
the true vision, at least it has begun.
So what does that mean? Well, what it means is that we have this huge number of devices, and
this too comes from Anthony's talk, this is what my lab looks like. I don't know if everyone's
living room looks quite like this, but we are soon to be flooded with devices. Now, many people
think of this as an error in which you might have a smart phone, maybe a desktop computer,
although that's becoming increasingly unpopular, and maybe also a tablet. But the way I think
about this and I think the way a lot of people in ubiquitous computing have been thinking about
this for quite a while is not to think about these as individual devices and certainly not to think
of them as devices that are owned by an individual, but that eventually we will think of them
the same way we think of this, a ream of paper sitting on the shelf.
Now what I love about this photo of this ream of paper is that it's such an unimportant product
that they didn't even bother to spell it properly. If you look at the description of the price tag
down at the bottom it's receyled or something, not recycled. This is how unimportant paper is.
You don't distinguish yourself with paper. Maybe you think oh, I want to be good for the
environment so I'll buy the recycled stuff, right? But this is how you think about it. It's cheap, if
I were to walk into someone's office when I was meeting with them today and there was a
stack of paper, I wouldn't even think about taking one and writing with it, right? But right now I
would think about twice about grabbing their iPad and taking notes with their iPad. But soon
enough this will be the way we start to think about these ubiquitous computing devices. These
technologies will become as cheap and as plentiful as paper.
Now, for anyone who studies ubiquitous computing this is not in and of itself an original
thought, but the key that we see to this at the University of Toronto is moving from the giant
clutter vision of the world and the disparate large number of devices of the world to what we
call a Symphony of Devices. And the core idea behind the Symphony of Devices is rather than
just having individual screens that you have content for developed for, instead, think of the
large collection of devices that the person has and target applications for the person and not
for their technologies.
So this has started to happen in a few places for specific form factors. So this is Microsoft's
attack which is called SmartGlass, am I getting that name right? Where you can write an
application that runs on the phone and it runs on the Xbox. So if you're sitting in the living
room and you’ve got your Xbox running you can interact with it with your phone. If the person
has created an experience the developer, the people who made the game, created an
experience that allows it to span that way. It's not just Microsoft. Samsung also sees this as
their future, so they have applications that run on their phones, and as long as you have a
Samsung TV you can control your television from your phone. And there are lots of these sort
of targeted tools. What we are trying to achieve with the Symphony though is a little bit
different. What we're trying to achieve is a way for developers to create applications without
having to focus on the form factor and without having to focus on the context of use because
the problem is right now you can develop a SmartGlass application, you say okay, what's the
scenario? Well, there's a television and there's a phone. And maybe there are two phones.
And that's sort of a scale that is reasonable to ask them to design their applications for. But if
you think about what's going to happen and what has already begun to happen is this explosion
of different device form factors.
Now this has created a couple of problems, one of which has been talked about for decades,
which is writing an application once and having it run everywhere. [indiscernible]. What I'm
interested in is similar but subtly different from that. I'm interested in enabling development
teams to build applications that yes, can run everywhere, and I think the closest to winning the
write once run everywhere is probably HTML, but you might have a different opinion about
different tools. What I'm interested in is similar to that except I want our experiences to grow
and to shrink across all of the user’s available devices. So if right now when I'm giving a talk I
have my computer, and I have available to me a large screen, the application should spread out
and take advantage of those screens. I'll give you a more specific example in just a second.
Now the role of the cloud in all of this of course is to be the connective tissue between all of
these devices. But it's important to understand that the approach that's been taken so far
doesn't yet allow us to achieve it in a meaningful way. We have experiences like iCloud, and I
forget what Apple calls their ability to play video from the phone and it shows up on the
television. Does anyone remember what that's called?
>>: AirPlay.
>> Daniel Wigdor: AirPlay. Yes, AirPlay. Thank you. So this starts to get there, right? And with
iTunes if I buy a song once and my wife is on the same iTunes account, so she buys a song, it
shows up on the phone, right? And that certainly starting to get to what all of us as computer
scientists have wanted to envision, but what this is about is about replicating experiences
across devices. So if I pick up my phone or I pick up my tablet I can have roughly the same
experience. The video is there, the song is there, the applications are there.
SkyDrive, I have to get a new screenshot, the Ministry of Truth has decided this is called
something else now, right? One drive. Okay. I apologize for that. I should’ve fixed that slide
before coming over. I figured it might change names again this morning though, but anyway.
So SkyDrive is a wonderful tool. I use this a lot. But it too is about replicating experiences,
right? I was really impressed that when I got my new PC and it backed up, restored from
Windows 8 it had all my passwords still. I mean that's great, but again it's about replicating
experiences across devices. Netflix, same thing. I can watch a movie on my tablet on the bus,
you get home, you walk in the living room and you resume and it plays from the same place.
Amazon, same thing. I'm reading a book on one device, I put it down, I pick up another device,
it knows what page I was on. Replication, replication, replication.
So what is it that we want to achieve? Well, this is a version of the cloud that we're starting to
see as a more popular thing where the applications start to live in the cloud and actually get
delivered to the devices. Adobe recently went to a subscription model for their software,
Microsoft has gone to a subscription model for their software, we have this vision of the
application living here and being sent down. Now we've built up before. A project that I did at
Harvard University, as Benko mentioned, looked at how it can we enable people to take pixels
and control signals and share them across devices. So I'm going to show you that project very
quickly. Here's an old video.
[video demo]
[inaudible] environment for collaborative research. Users bring their laptops to an interactive
table and connect to a server with a lightweight client and share their screen images on a high
resolution data wall. Each laptop screen on the table is rendered with a control bar with
buttons that allow users to switch the priority of the image. These images are automatically
laid out based on the priority of each display. Users are also free to manually manage the
layout above the table and wall surfaces. Gestural input on the table controls the magnification
and placement of the laptop images on both table and the wall. Double tapping on a laptop
image enters a full-screen mode with mouse simulation. In this mode the synchronized view
between the table and wall is severed. User actions on the table are interpreted as mouse
input and sent to the client laptop. After finishing the mouse commands this user taps on the
toolbar to switch back to the synchronized view.
>> Daniel Wigdor: So that was a tool that allowed astrophysicists, and of course Paul you
recognize this, right? This is the main room at Mural where we get a lot of this research. This
allowed astrophysicists to bring their research projects into a large room and do screen sharing
so that their application would show up on the table, and it's actually their laptop, and they
could go in and edit their content. It would also get mirrored up onto the wall. So this is very
popular. But this too is different from what we are trying to accomplish. What we want is
applications that automatically grow and shrink across all of the available devices.
So finally, a concrete example. Panelrama is the tool that I'm going to tell you about at the end
of the talk that allowed us to create these applications. So let me show you this app. So
imagine if instead of using PowerPoint to give this talk I was using this slide presentation tool.
So imagine this is up on the screen I'm giving the talk this way. You see of course the main
slide, you see the speaker notes, you see the list of slides that are available to come up next,
and if you look carefully you can see a timer in the bottom right corner that shows how long the
talk has been going.
Now, in PowerPoint I plug in a projector and I hit F5, it will make this full-screen. And there are
people who have written special applications where if I connect my phone into this I can
advance the slide. But what if I want something a little bit different? What I want is if I plug
into the screen I get this experience but I want to be able to give a more dynamic talk. I don't
want to have to be glued to the podium. So what I want to have happen is I connect my phone
into the Symphony and what it will do is it recognizes that the phone is a small private screen
with a good touchscreen, so it's connecting in, and this application doesn't know anything
about the phone except for those with physical properties. And you see the notes, the slides,
and the timer have all transitioned down to the phone and the current slide stayed on the big
screen.
Now this was not custom written for this phone. The developer of this application didn't know
that there would even be a phone involved. What they knew was that they wanted to be able
to have their application always keep the current slide on the biggest screen that was public
and move the list of slides, the notes, and the timer to something that was more private. Okay?
So let's say you want to give a more slightly more dynamic presentation. I've shown you that.
And let's say you want now to be able to walk around. Well, you can connect your Pebble into
the system and it gets recognized as a device that has time, so in a moment it will pull the
timers down onto the Pebble so you can see how long your talk has been, so instead of seeing a
clock you'll see contextual information. It also recognizes the presence of buttons and so the
user’s going to be able to push buttons on the side of the device and it will transition the slide
from one to the next. And here again the developer didn't know about Pebble, the developer
didn't know that there would even be a watch connected. What they knew was they wanted
the timer to be on something that could be very close to the user and they needed to be able to
push buttons to advance the slides and it recognized that that too was close.
There's a problem with this though. If I'm walking around with a Pebble on the stage and I'm
giving my talk, I've lost my speaker notes. I can't see them anymore. So I end up going back
over here. You don't really want speaker notes on this little screen, right? You don't want to be
going like this whole time you’re talking. So what do we do for that? Well, we connect in a
Google Glass and the system recognizes that this screen on the side of Google Glass is small and
private and so the speaker notes automatically move over to the Google Glass, and so now a
speaker gets a personal teleprompter right there in front of them. And yes, I see Paul sort of
making this sort of head gesture while you’re giving your talk, so you too can look like a zombie
while you’re giving your talk, right? The next version of Google Glass they put a little bit lower
and it’s a little less apparent.
Anyway, so this is the tool that we are trying to achieve, this automatic reallocation, and it
doesn't stop at one user. So if someone is connected with a different account into this
presentation what they see on their tablet, which is different from the speakers, is the list of
the slide that's currently being displayed. So you can follow along with the slides with it in your
lap, maybe take some notes, and in addition they can hit a button that shows them additional
information about a slide. So if someone just shows you a bar graph but you want the
underlying data you can see the publicly accessible content of the presentation. Let's think of
this is like a public version of the notes for each slide. This is a feature that doesn't exist in
PowerPoint yet, but if you added it to the application and you said this is public but I want it to
be on a private screen, on a small display, it would put it here automatically.
So this is what we’re trying to get to. And this application has been built. We built the tool kit.
Let me show you what we're going to build into it and why. So to achieve our Symphony of
Devices we underwent three steps to the work. The first was to understand how do people
currently use many devices? What are people doing in the world? Are there people who use
tablets and other devices in parallel? We then look at interaction designs. So what are the
things people want to do with these devices? How can we achieve the great functionality? And
then finally we built developer tools.
So, I know this is the show me building, so I'll go quickly through the ethnographic stuff. Mary
is not here, right? No. Okay. So what do we do? We went out into the world and we found
people who had distributed workspaces. We did this through advertising on Facebook and
Craigslist and we said, we want professionals who have what we refer to as distributed
workspaces. And the way we defined this in the call was you had to own a few devices, you had
to use them in different places, so you did work from home, you did work from the office, etc.
What did we get back from our call? Well, we got a whole bunch of people. These are just nine
of them, but a large variety of professions. These aren't just undergraduate students in
computer science between the ages of 17 and 19; we have design consultants, architects,
managers, physicians, etc. In fact, the population of the study is distributed across creative
endeavors, some engineering, healthcare, business and marketing, etc.
So what did we see when we went out to these people? We met with them in their workplace;
we interviewed them to understand what it was they were doing. We saw workplaces like this.
So this is one person's desk. And this is far from atypical. Jonathan will certainly appreciate the
use of multi-monitors here on the table. But in addition, of course, to the multiple monitors we
have the laptop that's over here on the side and when this person is at their office they plug the
laptop in and it’s driving all three of those monitors. They also have front and center an iPad
right where you would expect the keyboard to be. So what is the role that the iPad plays
because a lot of people have this vision that the iPad is the thing you use when you leave your
office, right? I know Ken and I have had many conversations about people actually use the iPad
at the same time. So how is that happening? And then of course just as interesting to all of us I
think is the presence of all this paper for hand notes and real artifacts, right? We haven't
reached the paperless office yet is no surprise to anyone.
Additional workplaces, this is a developer who does engineering work; we see a couple of
keywords and input devices, we see a bunch of screens etc. And here too, again, we see a
tablet sitting off to the side that they use simultaneously with their work. Same thing again,
this is an architect’s workplace. We see an iPad and a phone sitting and facing her while she's
doing her architectural work. And then this one I threw in just for Ken. We see a laptop and of
course a pen tablet off to the side here so that they can give different forms of input.
So what does all this, what do we do with all this information? We did a couple of things. We
built what we call the Artifact Analysis Matrix, and I promise I won't take you through all 10,000
or so cells of this thing, but essentially what it was was that for every piece of technology, and
by technology I include things like paper notebooks and cameras, we analyze what role does
that play in their work, how does it get used, and how does the information flow from that
device to another device? So this is more or less a classic technique of artifact analysis. Now,
come on Presenter View. Where have you put me? I love Presenter View 99 percent of the
time. There we go. Okay. And we understood what is their multi-device workflow, how do
they use all of these technologies, and then especially focused what are the things they want to
do that they can't?
So what did we find? Well, Stephanie developed these landscape diagrams which are actually
an incredibly useful tool. This is one person's technology and their workflow for one of their
tasks. So we see the things like the collaborator and the client sending information via
corporate e-mail into their Windows laptop. From the Windows laptop they used techniques
like getting information into Evernote so that it can then go to their Android phone because
there isn't a good way to get directly from the Windows computer into the Android phone. I'll
take you through a couple of these in a little bit more detail, but understand the arrows are
flows of information and the little pictures are people or artifacts and then we have services,
cloud services, and applications. We've got a couple dozen of these for each of the
participants.
Now, there are a couple sort of obvious things that emerge from this tool. Why does it keep
jumping around on me? I'll try the touchscreen. More touch friendly. Here's one. Does
anyone notice anything interesting about this graph?
>>: [inaudible]?
>> Daniel Wigdor: Split in half, right? Yes. Completely split in half. We have professional and
personal. What do you see, what are the technologies you when it's split in half? There's
always one of these. If you’ve got a Blackberry, and other people have noticed this in the
industry, if work makes you use a Blackberry, generally you erect a firewall and then you have
all your Mac stuff at home, right? And Windows laptop of course too.
So we're able to use this to understand things like what are people's workflows, but then also
what are the patterns that people follow when they're trying to use all of these technologies to
do their work? So I'll take you through a few. We have different information flow patterns, we
have the findings about the specializations of the devices, we have findings on data
fragmentation; and again, I'll get to do cool stuff, to the toys quickly for this crowd, but let me
give you a little bit of intuition why we care about this.
So first thing we talk about serial and parallel patterns. What we find when we say this is how
are people using these technologies one after the other or how are they using them
simultaneously? So an example of a serial pattern is what we call the Producer-Consumer. And
this is sort of the classic vision for how tablets get used, right? You use your laptop to write a
document in the first place and then you send it over to your tablet. And almost invariably this
is done by e-mailing it to yourself or by sending it through a cloud tool like SkyDrive or Dropbox.
So the Producer-Consumer pattern we found over and over and over again that people would
use these devices with better input technologies and then they would put it away so they could
fit more comfortably and use their mobile device, or they’d do that that way so they could use
it on the bus or they’d do it that way so they could take it to show someone else.
Also interesting to us was the frequency of the Performer-Informer pattern. So what that
means is we have the primary device like the laptop but simultaneously they would use a
second device, usually a tablet but sometimes a phone, to look up information. And I imagine
now that I'm showing this to you, you realize you do this too, right? You’ll be maybe writing
your program on one thing and looking up APIs on your tablet. Now this, although we all sort
of realized that we do this if you have a tablet, you stop and you realize this is actually a pretty
painful way to work in some ways, right? One, the input device sucks for this other device but
fine, they like having the portability, they like having this dedicated screen that they can use in
parallel; but the other thing is, if you're looking at the API documentation here and then
wanting to follow it in the program that you're running over here what’s the problem?
>>: Copy and paste.
>> Daniel Wigdor: Copy and paste, right? There’s no clipboard between these devices. These
are the sorts of really obvious findings that come out of this once you see the technique. So we
see things like, we asked people why is it you use all of these devices? Why don't you just have
the documentation open up another window on your laptop, or why don't you just have a
second monitor and put the information over there? But what we got over and over again are
people who say things like this, there's something about the geometry of one's desk, you can
have some notes sitting literally on the meatspace as opposed to the computer desktop even
with three screens, it's handy. They like the physicality. They like being able to have that API,
and only some of them were engineers, but I'm going to keep putting it in engineering terms;
they like having that API documentation right where they can reach and grab it. Alt tab isn't
fast enough for these people.
So, what else? Things like Performer-Informer-Helper is another design pattern. So this is
similar to Performer-Informer except we add this Helper device, I’ve got to stop using the
mouse to click on Presenter View; it jumps to different slide. Are there any office people in the
room? I’m going to put the mouse over here. Someone yell at me if I reach for it, okay?
Anyway, Performer-Informer-Helper, the only difference with this is they add yet another
device to use things like they’ll have a calculator up on their phone and they'll have this
designated tool you can always go to and find a calculator running on their phone. You think
why don't they just run the calculator app in Windows? But people like the physicality of this
other device despite the fact that it has the copy and paste problems.
All right. I'll let you read the paper for a much more detailed analysis of all these design
patterns. I think it's really interesting, but I know we want to get to the toys. Certainly we
found a lot of specialization of the devices where, just as we would expect, if you have a
desktop computer that gets used to do things like composing information, composing content,
if you have a smartphone that's used for things like collecting data out in the field, it might be
taking pictures of a site and bringing them into the computer etc., tablets used for reading and
having an extra display, exactly the sorts of specialization that we expect from all of this but the
surprising finding that we found was people much more quickly go to their specialized devices.
Even when they're sitting at their desk, often people will pull out their phone and run the
calculator app on their phone instead of running the one in Windows, even though there's no
copy and paste. So they end up retyping things and having transcription errors because they
like the physicality.
Last, oh good, so it's happening on the touch screen now too so at least it's consistent. Data
fragmentation. So this comes as no surprise to anyone, I think. Getting information from one
device to another sucks. What is the fastest ad hawking network technique that people use for
information sharing? Who thinks it's Dropbox?
>>: Email.
>> Daniel Wigdor: Everyone said it. Who thinks it's e-mail? Yes. We all do that, right? You'll email things to yourself just to get it onto your Windows tablet. So we have this is sort of a
workflow. You save your content, you open the e-mail up, navigate to the file, attach it, pick
your device, all the way back through. And of course what this does is it moves it off of a
shared file system, so now you have this extra version of the file hanging out in your e-mail.
And you save it there, it doesn't save back properly; and I should say I am completely in love
with the new Office for iPad because of its one drive integration is awesome because it solves
many of these problems.
All right. What else do people want? Over and over and over again we see people trying to
perform tasks together on devices. I mentioned the calculator model, someone's using a
calculator on their phone, I mentioned reading documentation from one device, people want
this. I want it, you want it, we want to be able to have all of our devices share information well.
But it isn't just about running an application on one device and moving it over to another app,
it's also about being able to take the information and divide it up, or the application itself and
divide it up and have specialization. So if you have an application that has all of this
functionality, we don't just want to have it running on one screen and then have this other
thing hanging on another screen, we actually want to be able to divide it up and use one
application broken up across whatever devices I have available to me at a given time and
critically, to be able to understand if that device is good for something, if it's good for data
entry, if it’s good for having a big screen, if it’s good for having a good touch input, that the
content that gets put there, the portion of the application is appropriate for that device.
All right. Let’s talk about technology. I'll take questions on the way through. Does anybody
have a question about the ethnography? No. We want the toys, right? Yes.
>>: Do you also explore the idea of public spaces and user devices in public spaces? I saw some
examples of Office space [inaudible] on settings-
>> Daniel Wigdor: Yes.
>>: But did you answer [inaudible] big spaces in museums or [inaudible]?
>> Daniel Wigdor: Yes. So the question was about public spaces. Did we look at how people
use devices in public? Some of our users work exclusively in public spaces. So their notion,
their distributed workspace is that they go to a coffee shop every morning and camp out. So
there are lots of examples of people using private devices in public spaces. In terms of public
devices, like being able to use a large screen that you can walk up to at the airport or
something, there wasn't, I don't think there are any examples of that. Maybe it's just that
Toronto isn’t high-tech enough to have large public displays that we can share information on
yet, although I keep wanting to be able to send info to the CN Tower because it's got this light
display that I want be able to send things to. Yes.
>>: [inaudible] ethnographic is that written up in Stephanie’s theses? Where is [inaudible]>> Daniel Wigdor: It's a UB Comp paper 2013.
>>: That's the most complete?
>> Daniel Wigdor: Her thesis is more complete. I'm happy to share it with you if you want to
see it.
>>: And then the other thing which maybe we can take offline so you can get onto the showing
stuff, but one key question here is you say that people prefer to have the stuff on their devices,
and this a possibility. Now the other possibility is that the way it’s so hard to arrange things and
access them conveniently on larger devices and I think that's a critical thing to sort out. So it
may actually be that they really prefer physically a way or may just be hard to, that may just be
the easiest way to access it and [inaudible] in that. Do you have a feeling on that?
>> Daniel Wigdor: Yeah. Certainly I agree with you that if, eventually it won’t just be like you
have one desktop monitor and then you’ve got this space down here and in order to get, so the
question is do they really want to have it on a device or is that they want to have it on the
desk? And if it were one operating system that stretched across all of that would they be happy
just to sort of send the window down to the desk or is it that they like the physicality of this
other device? Am I getting that right?
>>: Well, or I mean the third possibility actually it would be faster if it was all up in one place
where they can move around, but that may actually prefer to just have it in a separate place.
So I think there's three possibilities. I mean, I agree with you that the second is preferable to
what we have now, that's where you’re headed, but there's still-
>> Daniel Wigdor: To be clear, it's definitely faster. So if you want to use a calculator, it’s way
faster to hit the start button, type C-A-L-C, Enter, and then type versus pulling out your phone,
turning it on, entering your unlock code, scrolling to the application, it's definitely faster to have
it on the desktop.
>>: That’s also [inaudible] just your primary task, right? If I bring up something document on
top of the thing I'm working on maybe it’s faster to do it there but then I have to>>: That's what I'm getting at.
>> Daniel Wigdor: So we have a study that starts to scratch the surface of that that I'm going to
sneak in in the middle of the toy presentation. So let me show you that, and then I think that
will set context for some of this conversation. Yeah.
>>: So maybe I’m just one of the old guys in the room.
>> Daniel Wigdor: Well, you are, to be clear.
>>: Thanks for pointing that out. But I find that I will often pull up the calculator for my phone
because I forget that you can just hit Windows and type CALC.
>> Daniel Wigdor: Right.
>>: It’s not a question I prefer it on my phone, I actually don't, but it's just easier to remember
oh yeah, I have this on my phone. So it's not because I prefer it>> Daniel Wigdor: Yes.
>>: It’s just because in my mental model that was just easier.
>> Daniel Wigdor: It’s definitely it’s in the mental model that it's easier. And a lot of what
people said to us, and this will really come out in the study I'm going to show you sort of
halfway through what's coming next, is that having a physical token that is that object is very
helpful to people and when people have that option they like it. So, for example, what we
found with our participants was, sticking with the calculator example, it’s not that they want it
on their phone, they want a calculator, like a real physical calculator that is on their desk and
they can use it. And it's not about it's faster to use, it’s about they remember that it's there and
they have the physicality of that to sort of have this token.
>>: But what if you have a whole wall display, you had your calculator up in on corner and it
was always there, nothing went over it and it didn't go over anything. Would that be good
enough, or would you really still rather have a physical device up closer? I don't know.
>> Daniel Wigdor: It’s an excellent question, it’s one that has to be answered. I don't know the
answer yet either. If Window Managers were as painless as physical arrangement, eventually
degenerate case is really what it comes down to is do you prefer pixels or physicality? And I
think on that particular question there's a divide. A lot of people really prefer physicality, and it
does help you perform the task really well, but I don't know yet.
>>: Part of its [inaudible] in your hand because no matter where I hold my hand I know where
to look at to see that, right? If it's in some big wall display, even it stays fixed, I need to
remember where to look.
>> Daniel Wigdor: Yeah. I think let's all talk at lunch and we’ll figure out the answer to this
question. All right. So let's get into the toys that got built. So Peter has this idea, Peter is my
Master’s, he's now a PhD student, his idea is to target, I talked about watches and heads up
displays and wearables; his vision is this, his vision is when you're working with technology
you'll have a whole bunch of tablets that you can be able to use simultaneously. Are they
tablets? Maybe, maybe not, but they are physical devices that you can dedicate Windows to.
And the way to think about this work starting off is think about these as individual windows of
information. Instead of having a Window Manager you have these tablets. These are all
running Android, and he’s developed tools to start to solve this problem of how do you flow
information from one device to another. And to be very clear, neither he nor I is ideological
about whether or not there's a very large monitor front of all of this. But given that people are
excited to use all of these devices and to have the physical token, I think, then we started to
solve this problem with this system that he calls Conductor.
So to build Conductor he focused on creating six applications, and by using these applications
on individual devices and trying to solve the problem of how to flow information between
devices he was able to build his Conductor. So what does it look like? This is the application
that I'm going to be using throughout the example. He built this for CHI which was just in
Toronto, so the example we have is a tourist’s guide to Toronto. So you can imagine the single
device version of this application where you have a list of different things that are very
important to see, of course the building where my office is the most important, where our lab is
is important, and if you see something you like you can tap it and then you'll see it, right? And
you can imagine what the smartphone application looks like. Now he built it for a multi-device
environment. So sure, he can bring up information about the CN Tower, but when he adds a
second device like this now he wants to enable people to have in their hand, as Ken put it very
well, to have it dedicated to their hand, here are the list of things that I want to be able to see,
the tourist applications, but I want to be able to send content from the phone over onto the
screen so that I can see information about the tourist site on the screen. And by solving this for
a tourist application and for a few other applications he has a generalizable solution.
The question is how does he do it? So again, as we've gone through now together a couple of
times, right now what you might do is you try to e-mail yourself a link into the app and you
bring up the app on the tablet and you load it up. He developed this idea of targeted
transmissions. So what does that look like? Well, from the phone, from the first device, he can
perform a gesture on any item and from this pie menu to see a list of all the other devices in
the space. And it has the name of the device, just Nexus 7 because all the tablets he had were
Nexus 7, but they're also color-coded in a couple of ways. There’s a bar at the top, you can see
this is the orange device, and then he also colored them orange so it had a physical bumper.
So what does that look like? Well, so here is an arrangement of many devices, he's got his
application on the left side, and he’s going to send information from one to a different device.
And you can see as he goes through the list we get this little shutter on the other device that
says this is one you're pointing at right now. So if you let go of your thumb right now that's
where you're going to send the information from one device to the other. So think of this as
being sort of like copy and paste, but it’s not quite a copy and paste because it’s sort of a forced
copy and paste, right? As soon as you copy and then you select Paste it changes the application
that's running on the other device. So it's a little over the top. We have additional fidelity.
So the question I put to him was this doesn't seem physical enough. What we found was
people really like to be able to lay out their devices on the desk in particular places and they
wanted to be able to have that physicality mean something. There’s great work out of Calgary
and other places that look proxemics so can do things like if you're holding a phone you can
reach it towards another device and share information or affect that device. But we wanted
this to work outside of an icon instrumented lab.
So how do we achieve having the physicality matter? If you've got these two tablets, you want
to be able to send information from here to here without having to sort of go for a list and
target it, right? So what he does is an approach called broadcasting. Now what broadcasting
does is when he selects to broadcast it sends it to all of the devices in the Symphony
simultaneously. Now the key realization here was this is a single user application, or a single
user thing that he was trying to solve for multiple devices, if you're trying to send information
from device A to device B, if you look at device B it doesn't matter that the information is
arriving on your 14 other devices on the table, it only matters that it shows up at the one you're
looking at.
>>: [inaudible]?
>> Daniel Wigdor: Let me show you how it works. So he picks an item and he chooses to have
it broadcast, fine. What does that look like? When he broadcasts it, it shows up is what he calls
a queue over here on the side of the screen. Now, instead of forcing the device to go to a
different mode or whatever, it just shows up as this little icon on the left side and you just
ignore it. It will go away after a few seconds. The key features here are though we get the
color that shows where it came from, we get an icon that represents the application that it
came from, and we get a label that says what it is that this thing does. Now when you get the
queue you can just tap it and say, I want the default action. So if I send the CN Tower
information I tap it, it will load up a full-screen about the CN Tower on the tablet.
You can also choose to do what are called contextual actions. Let me take you through these.
So he taps it and it gets the default action, so here it's going to load the CN Tower information,
fine. Now what you’ll see here you can bring out old cues from the side of the screen, so if
they’ve timed out you can just get them back again, they live in a little tray on the side, and
more cues can show up, you can just keep ignoring them, and again you can tap it to get the
default action, but just as interesting is what we call contextual actions. This is sort of the paste
version of this thing.
So you've got your cue and instead of tapping it you drag and drop it, and when you do that it
pastes it into the current application. So a minute ago if you just tapped that it would've loaded
the tourist information application, brought up the page, here he dragged and dropped it into a
web browser. So what does it do? Well, it finds the URL of the thing that he dropped and it
goes to that website. If he had targeted it towards a text field it would've pasted it like text.
Now what Peter is going to show here is how to quickly get information from one device to
another. So on the very left we have the contact list. This is a list of people in his contact book.
Next he’s going to move from that into an e-mail application and he'll move from that into a
map which shows right now is just in a map app. But if he brings the cue into the map app and
he's asked for the contextual action it will show you where the person lives. So let's watch him
do that. So he chooses the individual, he broadcasts it, that goes out to all devices, he taps it so
it loads the default action, he then broadcasts out the contact information again, and when he
drags and drops it onto the map it performs the contextual action of loading it into the map
application. So very quickly moving information from one device to another and always having
the right context. You can either preserve the context of what was running on that device
before or you can choose and say I want to switch applications.
Now this doesn't quite get all of the multi device scenarios we want. This requires that you
continuously send these requests to affect another device and they have to get it sucked it over
and over and over again. What we've created here is the ability to temporarily form a link
between two devices, and we call this link a duet. And a duet is these two devices are going to
temporarily play together. Now the way that we procreate this connection is, so for example
let's say I’ve got a list of the videos here and I want it to be that every time I tap one of the
videos it loads it onto this tablet. Or if it's not a tablet imagine it's your living room television.
You want to be able to watch a movie when you tap it. So that's done by one of these
functionally bonded duets. So here's how it works.
We have on the left our tourist application, we have on the right the information with the CN
Tower, recall if I wanted to see a different one I’d have to broadcast it, come over here and
accept it over and over and over again, instead we are going to form our duet. So here's how.
You broadcast a cue and then instead of putting it in the usual place he’s going to put it over
into this link. And that says I want to create a connection between these two devices. So now
anytime he taps one of the list items on the phone it automatically brings it up on the tablet.
No need to over and over and over again set the same thing.
Now, duets are great, but how do you manage them all? And by manage I mean there's a
couple of things. One is there’s going to be an action that this will cause to reform. If I tap CN
Tower on one list it makes sense it would bring up the information with the CN Tower on
another devices, but what if instead I wanted to do something else? Can't you form other types
of connections?
So in the developer framework the developer of the application can choose what type of duet
they want to create. So, for example, here we've got two map views and he’s going to form
two different types of duets. First he’s going to create one that just synchronizes the view.
Then he'll change it and say I don't want to synchronize this view anymore, I want to see what
the other one is looking like a world in miniature, and I want to have information about the
viewport of the other screen. So let's watch that happen. So he brings it over, forms the duet,
and now any time he scrolls one map the other catches up complete with latency to make it
authentic, and then he's going to grab the duet and put it into the other socket which says I
want to create the other type. The user gets a little information that describes the duet that
will be formed. So now when he zooms in it doesn't zoom the map, it just shows what is the
viewport of the other map.
So pick your favorite application and think about what are the ways you might want screens to
get synced. If there's only one way that would just be the default duet. If there's more than
one then you can create them.
>>: I have a quick question.
>> Daniel Wigdor: Yes.
>>: So you made a duet look small like you have a master device and you have a slave device
and you can control your little slave device from your master? Is there a way to>> Daniel Wigdor: It goes both ways. So if you have the screen synced, for example, and you
grab it and move it here it will send it out to the other device as well. So it's bidirectional. And
the developer can also choose to make it unidirectional if you want to. But that's a very good
point. We really want to get away from this idea of master and slave because I think it's holding
back a lot of multi device computing. A lot of companies are thinking about it as the phone is
master, the TV is the slave, or, the phone is the master the watch is the slave, or the phone is
the master and the heads-up display is the slave. But it's always about having a master device.
We're trying to break away from that completely. This is a democracy in our Symphony.
We have here cross-application duets. So this is a map application running in parallel with the
tourist application. So now whenever he taps one of the tourist sites he sees the current
location on the map. And that happens because the duet defines this as a geographical point
and the map knows how to receive it and so it does it automatically.
There's a nice property that this brought out which is the ability to share peripheral devices
easily. We found this to be really painful. We had 20 tablets and we wanted to be able to pair
a keyboard with any of them. A Bluetooth keyboard. You have no idea how long it takes to pair
a Bluetooth keyboard until you try to do it 20 times in a row. Here's how you can handle a
Bluetooth connection in Symphony. This, in terms of the Bluetooth pairing, this one is paired
with of the devices in the Symphony. We don't even know which one anymore, okay? When
you type you get a cue that shows up and if you bond the cue with the device it sends the
keyboard over there, if you form a duet with the keyboard. And now what you see is you’ll
break that duet by throwing it in the trash and now it goes back to be on the previous device.
So much better than the, I watched him doing it.
All right. Last bit of functionality, being able to manage all of these devices. So there is a
delicate balance I argue, and we're going to talk a lot more about it, that the physicality of the
device matters. And what we found in a study I’m going to tell you about in just a second is
that over and over and over again people would dedicate particular physical devices to
particular tasks. This is my e-mail. This is the map. And they would put it in a particular
physical place on their desk. Now the corollary to that is, or the counterpoint to that is this is
all just digital information. So shouldn't I be able to, on the device that is currently my list of emails, shouldn't I instead be able to grab the screen from something else? Now, we provided
that functionality, but interestingly it would get used a little bit but not very often because they
like this direct physicality. So what you see here is if you hit the Home button your home view
is all of the devices that are running, not just your home view on your particular phone, and you
can clone a view by just tapping it and zooming in, so then he got the copy of what was on the
green tablet, and you're also able to grab a view and move it. So if you want to have the
session from the green device on the orange device you just zoom out to your Task Manager
and drag and drop it and then the orange device becomes the green device. So it completely
gets rid of this idea of an application running on a device. It just is running in your cloud of
applications. You can just zoom in. Yeah.
>>: I always imagined that hoping that I can, I have a duet tablet but to pull up the virtual
keyboard is so difficult. You only see a slide of maybe your website. Can I do like typing in my
phone and just transfer that?
>> Daniel Wigdor: Yeah. If you form a text entry duet between your phone and your tablet
your phone can be a keyboard for the tablet the same way. So I share your opinion that that's
painful. Now the study is not just about proving out the functionality. The study is about
looking at how do people really use this thing. So what we did was this replicates a technique
that was used for very large screens at CHI previously.
We used the VAST 2006 contest, this is from North and his research group that did this
originally for very large screens, and we said let's have the people try to solve this task. So what
it was was the following, you are a CIA analyst and you have 230 odd newspaper entries, it’s
mostly stuff that's in the press, you have a map so you know where this fictitious town is in
California, at least I think Alderwood is fictitious but it's set in California, and your job is to find
the 8 articles, excuse me, 10 articles that really do describe this funny business that's going on
in the town. You're trying to solve a crime. This apparently is a task; this is something that CIA
analysts have to do a lot. What we gave people was access to all 230 documents in a list. If
they wanted to they could just pick up one device and just go through the documents one at a
time, read them, they had the ability to take notes, etc. I don't know why Presenter View just
does not like me.
>>: CIA is not supposed to be investigating crimes.
>> Daniel Wigdor: I apologize. You're right. Domestic crime is FBI, yes. In Canada we don't
have any of this stuff.
>>: [inaudible].
>> Daniel Wigdor: Oh, that makes me nervous. They don't like me. So eight participants came
in and did this task. Most of them were computer science graduate students just to set myself
up the way I did others earlier. We gave them a whiteboard, we gave them a paper and pen so
if they just wanted to take handwritten notes they could, we give them five Nexus 10’s which is
a ten inch tablet, five Nexus 7’s, one Bluetooth keyboard and it was running our system. And
what we wanted to know was well; first, would they use all of these devices? Would they
actually make use of having multiple devices?
Now every user started off the same way. Every participant I mean to say started off the same
way. They had this stack. All of them were running the application that let them see the
contents but they could also switch to a note-taking application, the basic apps that are in
Android; and some of them would start off just reading through the list of documents. But
very, very quickly all eight participants started having a desk that looked like this. And what
they would do is they would scroll through and find a document that they thought was
interesting and then they would physically put that document aside. They’d say okay, I want to
save this one and put it over here. Now they could've used a flagging feature in the application,
put like a little check mark on the document so they could go back to it just like in your e-mail,
didn't use that. Instead they would use the physical piling to say this is one I want to
remember. So they used them. How did they use it? Well, I mentioned that physical piling
behavior before, and we see over and over and over again the participants using spatial
memory to keep track of these documents like they were physical documents.
So what we found in post-interviews, and I'm going to let you go to the paper if you want to see
all the details of the user study, but the point is most of the participants used all 10 tablets and
said they wished they'd have more. Almost no one took handwritten notes although that may
be a function of the fact that they were computer scientists. And people did over and over
again do this physical piling. So if we asked them questions about a document they would
know where it was on the desk. They didn’t even have to open their eyes, right? They wouldn't
even have to look at it. They’d say oh, that's over here.
>>: Did you have [inaudible] they were using [inaudible] desktop [inaudible]. Because I've seen
the same task in one the information [inaudible] and it was the same [inaudible], exactly the
same [inaudible] with so many documents in the other tool which analyzed things for us and
allowed us to bookmark it; and it seemed reasonable for us to do it on a desktop that was like
21 inches. So they didn't need as many tablets. If I feel that it is, this study is sort of like looks
biased towards having multiple devices in front of you and that's why you're using them. You
want more obviously [inaudible].
>> Daniel Wigdor: Because it's a small.
>>: Yeah.
>> Daniel Wigdor: Yeah. So we didn't, but you're right in saying that we replicated the
experiment. So North and his students, who did this experiment originally for very large
screens, gave people a desktop computer and a wall sized display and looked at the differences
in behavior. And we sort of latched onto that as doing another condition. So you want to see
what the baseline was. What North and his, I wish I remembered the name of the student who
was the first author, but Chris North is at Virginia Tech, his, their findings were you can do the
task on a single device, right? If forced to you can do single task on an individual tablet. But
what they found was however much space you gave people they would expand out to use it.
So what was interesting to us about the tablets was we give them more pixels, they want to use
more pixels. That's sort of interesting but I think to be expected. What we were really pleased
by was how they physically arranged things. The fact that they didn't just used more pixels.
They could’ve just put everything in a giant grid and created a larger screen and used it like a
big monitor. But instead they did this physical piling behavior and physical arranging behavior
where the physical position of the device was interesting.
>>: I wonder if the information study, the field study [inaudible]>> Daniel Wigdor: Yeah.
>>: You've found a task which was common enough that you could have replicated that task
with these multiple screens and your application helping them arrange the screens such that
the task would become easier than what they were actually doing. And we started with the
premise that the mental model. [inaudible] was using a device for calculator, [inaudible] device
for calculator and a device for something else. All of the functionality existed in the main
desktop and I could have [inaudible] access.
>> Daniel Wigdor: Sure.
>>: Now I wonder if taking one of those tasks you could have replicated this multiple screen
scenario and could've made that easier somehow and then could've commented saying that
our application made it easier for them to communicate between the device and then solve
that problem.
>> Daniel Wigdor: Sure. We could've done that, but it's trivially obvious that this makes it
easier to communicate information between devices, right? Like just by giving people a
clipboard you make it easier. I don’t need to run a lab site to do that. But there are all sorts of
lab studies that can be run next, and I'm happy to talk to you about it after the talk.
All right. Anthony, I'm sorry, I'm going to have to skip your talk. What this means is you guys
are going to have to ask Anthony to present his best paper and best talk winning presentation
from CHI at some point this summer.
So the last thing I want to talk to you about is how do you actually build these applications? So
if you're a developer I've talked about this vision of being able to write once and run anywhere
on all the available devices. And this guy promised that a long time ago. Java [inaudible] to a
certain degree but we talk about this problem. If you want to develop an application for
Android versus the IOS what is the first problem that you think confronts you and you see
complained about over and over and over again in creating Android applications? It’s one word
and starts with F, but it's not four letters.
>>: Fragmentation.
>> Daniel Wigdor: Fragmentation. You hear this all the time. And what does fragmentation
mean? You sit down with a developer and you say well what is it, Android is fragmented, what
does that mean? Well, it means sometimes the phone is the phone has a 4.3 inch screen and
sometimes it has a 4.6 inch screen. You think that's fragmentation? You want to see
fragmentation try making it run on a watch, glasses, and a phone or a tablet and writing it once.
That's the problem we are trying to solve, this fragmentation of the number of devices.
So how do we do it? Well, we built something that we call Panelrama. And what Panelrama
allows you to do is to write an application once and run it on all of the available devices
simultaneously. There’s a great video that goes through all of the technical details, I'm going to
give you the two-minute version now so we don't go much over time, but understand that what
it’s intended to do is you write the application and then every time a new device is connected
or a device disconnects it scans the devices that are connected to this Symphony and it
allocates the UI appropriately.
So what does the developer to have to do in order to accomplish this? Three things and then
we do the fourth. They define that the building blocks of the UI, which we call the panels, then
they define the attributes of the panels and the attributes of the devices they want them to
target towards and then it gets optimized. So let me quickly show you that. Here's an example
application. Everyone’s seen this one before. So there's work that's called distributed user
interfaces that has looked at this problem before of how you write an application that will span
devices. What the focus of the Panelrama is is in minimizing the pain of the developer
experience. So if you could write a normal HTML 5 application then you can write a Panelrama
application with very minimal modification.
So here's an existing one. And you say I want to divide this up into panels. So what are the
panels? Well, you look at it and you say well, the playback controls are a bunch of individual UI
elements but clearly they belong on the same device, right? The play and the pause button and
the full-screen button all that is going to be together. So you define that as a panel. Next we
would define the video as a panel. Say it's a separate piece. What are the other panels? Well,
this list of related videos. And the nice thing about HTML 5, and I imagine many of you know
how to write stuff in HTML 5, is that this is usually divided up in the DOM already. So we are
not talking about rejiggering your entire application but finding the right branch of the tree.
What are the other components? We've got a comments box and the text that goes in it.
What do we then do? We modeled the attributes. So what does that mean? Well, it means
you say this panel cares about physical size. So I want the video to play on whichever is the
biggest screen in the Symphony at that moment. Next, these controls I care about proximity to
the user, how ready to hands the device is. For this I care about having a good keyboard so that
I can enter text in the comments, and here I care about having a good touch input fine, mouse
input, whatever, but the point is you want to be able to point. And that developer gives a score
for all of these attributes. We did a study where we designer or redesigned existing Android
applications and determined that there were seven physical attributes that would matter the
most in our designs but the developer can define more, for any given panel you give a score to
the attribute that that panel cares about.
Next, the individual devices need to have those attributes modeled. Now fortunately there are
information services that you can subscribe to. They're pretty expensive as a developer, but if
you're writing an application it costs you about 5000 dollars a year to subscribe to this
database. And it can, from the user agent information and other attributes of the device, tell
you not just this is IOS but it can say this is an iPad mini and that it has all the physical
characteristics. So you can subscribe to that database. So using that database we get a score
for each of these physical attributes. So in this particular Symphony at this moment there's a
50 inch TV, a Dell laptop, an iPad, and a Nexus phone. You can see decreasing physical size,
how good is the keyboard, how good is the touch, etc., etc. And then we stick it into a linear
optimization and the linear optimization allocates the different panels to the different device so
you always get the right panel on the right thing.
The panel tag will be the last thing I show you, this is the code, some of the code for a drawing
application. We've got the drawing canvas here. This does not have the panel stuff in it yet,
there's just normal HTML 5. Drawing canvas up here is inside a div, we’ve got the buttons that
change the color palette, red, green, and blue. Pretty basic application. So the developer
would say well, I want to canvas to be in one panel and the controls would be in another panel.
How do they do that? Well, they just add some panel tags and they say this is the palette
panel, that’s the canvas panel, and a little bit of class information that ties it back to the panel
definition. And the panel definition which is here, this is new code that they would have to
write, all they have to provide in addition to sort of the obvious things like giving it a type and a
name are these infinity scores that I showed you before. This is where they would say I care
about big screens, I care about quality of touch input. And we get applications.
[video demo]
Global panel state need not be completely synchronized in this map application. We use the
global map panel to synchronize the local viewports of three tablets, but not the type of map
being displayed, allowing each map panel to provide different information on the same
location. Similarly, a street view application could synchronize the GPS coordinates but not the
viewports making it possible to achieve a panoramic view of the road. Panelrama also allows
this PDF reader to display consecutive pages on additional devices. Synchronization ensures
that all pages flip simultaneously while an unsynchronized device can be used to display a
helpful diagram or a list of citations. Unsynchronized local panels may access the global panel
on demand. In this video player a user uses the push function to broadcast the local video to
the global panel for group viewing. Other users may use the pull function to copy the global
panel contents to their local channels for private interactions.
>> Daniel Wigdor: Okay. The paper has a lot more details on the technical implementation. It
has a lot more details on a developer study that we ran; it was first time I ever had to pay
participants 150 dollars an hour to come be part of a study. That was fun getting that by the
grant people. Anyway, so the nice thing about Panelrama is that a team has spun it out and
they're actually building this as part of a startup operation led by someone named Robert
Levey[phonetic] who is a toolkit architect from Microsoft previously and soon will be available
at a store near you for further use. I don't know why the store would carry it. It will be
available online for you to use for free.
So that's the Symphony of Devices Project. As I’ve said, all of these are much more fleshed out
in the appropriate papers, and here are all the references, and again I apologize Anthony that I
didn't have time to show your work. We had good discussions instead. So thank you for that
contribution. Anyway, so thank you very much.
>>: We have time for a few more questions.
>>: So I didn’t ask before because I think it might be off topic, but very interesting that
experiment because most of us are raised from like, we are train to use paper and pencil,
calculator so that we kind of have the memories that using that way but the new kids may be
growing in the new century they might get used to all these new technologies. What do you
think, like if you replicated the same experiment in about 20 years. What do you think the
result?
>> Daniel Wigdor: Yeah. So there's sort of a nature versus nurture conversation to be had
here, right, of does spatial memory come about from learning and is the physicality that is
associated with it inherently joined with spatial memory, or can you have all of the same sense
but entirely virtually? So if a baby was born and you put a VR helmet on them and the way that
they point of things is with their eyes and they never use their arms would they be just as good
at being able to find things quickly? And to Ken's point not interrupt the task that they're
doing. My intuition, despite the fact that it’s the exact opposite thesis of the book that I wrote
while I worked here, is that there really are inherent properties in psychology that we have to
tap into. This is not an original idea in HCI. We borrow from psych all of the time and there's a
lot of really good work on spatial memory that we draw from.
I think the perennial project that I would think of when I think about what is the virtual version
of spatial memory is probably Data Mountain which was something that was done here a long
time ago. Not that long ago. 20 years ago, give or take? And it’s sort of the virtual version of
all of this. So would people be just as happy with the Data Mountain as they would be with all
of these physical devices? I don't know the answer to that question, but I think that's worth
answering.
>>: So following on that,I don't know the answer either>> Daniel Wigdor: Yes.
>>: But if, let's say that they developed a new device that would enable you to detect how far it
is to other planes that are flying near you. Would a pilot rather that on their phone or would
they rather have it as another panel on this physical control panel that’s one panel that doesn't
move? And I think we know that for that one they would rather have it up there. Now the pilot
may not control the entertainment system, but if they did then they might prefer to have the
entertainment system controls in their pocket.
>> Daniel Wigdor: Right.
>>: So I think that the point here is that I think it's pretty complex space and it's probably not all
or none and it’s going to be based on the different tasks involves in the context. So I think that
these are really interesting questions. And I think the Windows Management System clearly
pushes us in certain directions, and so if we did have a better one it could change sort of the
balance.
>> Daniel Wigdor: I think that's true. And I think one of the works that any student who comes
on the Symphony Project, they have a stack of papers they have to read, and one of is your
multi-monitor work. And one of the conclusions I'll tell you, in case you don't know, is that two
monitors is not the same as one monitor of the same size. And yes, it's about the Window
Manager and the way Windows treats that and when you hit the maximize button what does it
do to the different screens?
>>: 15 years ago we didn't have as many devices, but I did see cases people would have the first
smart phone or other devices on their desk just showing their calendar, for example, so>> Daniel Wigdor: I agree with you. I think it depends on the task. And I think physical,
physicality matters in a couple of ways. One is you want to always know where it is, and you go
into someone's office and you say where is this paper? I actually had to tell my assistant to go
cash a check that I had forgotten to deposit in my office, and she was poring through papers.
Wouldn’t it have been great if she could have just like typed search and hit where’s the check
or known it would be in the right place.
My favorite example of this now is a feature that apparently Land Rover is actually going to
come out with which is people drive these giant cars and the hood is in the way so you can’t see
if you're going to run over someone's dog. So they have the backup cameras because people
don't want to turn the necks; and you see the thing down here, but they've added, Land Rovers
added cameras to the front of the car so you think well, where is that display? They have
registered it and they do it as a heads-up display on the windshield where the hood is so it
looks like you're looking through the hood to the video display of the thing. And what I argue
there is it’s not just about the fact you always know where it is but the fact that it's in the right
place is helpful too and less disruptive to the task. So I think yes, tons of issues that come to
bear on all of this and ripe for plucking at.
>>: I have a technical part of the question.
>> Daniel Wigdor: Okay.
>>: So you some optimization program to determine which screen, which device you use for
certain tasks?
>> Daniel Wigdor: Yes.
>>: Is it running locally on some of the device or it’s on the cloud?
>> Daniel Wigdor: It's running in the cloud. So it's a cloud, so the question was about the linear
optimization function and we base our work on that, by the way, from a fellow who is now at
Harvard named Christophe[phonetic] whose last name I can't pronounce, who did linear
optimization for allocating individual elements and figuring out what type of button should be
shown or should be something else. Is it really that simple? So Christophe Gaios[phonetic].
Anyway so his, anyway backing up, is in the cloud. Yes, it's a cloud service. And then that begs
the question well, what if you're traveling to the US and you live in Canada. You don't have SIM
cards in all your devices. Does this just stop working? That's why I gave the talk as videos
instead of having tablets and actually showing you Panelrama dividing up my actual slide
presentation.
So taking a dependency on the cloud creates user problems that some of us sort of forget which
has become very apparent to me with my SkyDrive set up when I don't have Internet access on
in my tablet when I'm in the States, or on my laptop actually. Anyway, so it's in the cloud, we
need to fix that too.
>>: So I think [inaudible] example kind of along these lines is what's happened to electronic test
equipment, and it's probably not an area that you follow; but in my youth we had all these
analogue scopes [inaudible] generators and network analyzers and stuff like that. And each
separate pieces of equipment with an insane number of knobs and dials for setting everything.
And sometime around maybe the 90s or so all this stuff got converted to visual form. So when
the first digital instruments came out they basically looked like PCs. They had screens with
keywords and people hated them. It was a total failure in the market. And so people went
back to basically embedding the PC in these things and then connecting up knobs and dials and
switches and stuff so it would look like the old instrument.
Now at the same time you started see this market for virtual instruments where you could buy
modules that you would stick in your PC network, the entire scope minus the screen and the
keyboard and the interface, right? And that's caught on to some minor degree. If you walk
through almost any lab that does hardware Microsoft you're going to see physical equipment
and you still see scopes marketed as wow, this scope has a seven inch screen even though it
still has a VGA connector off the backs and it’s just running Android typically inside. So it’s just
a PC; and you can make the argument that wow, this is really dumb. We really should've gone
to this virtual instrument model a long time ago, but people don't like it. They find it much
easier to deal with these physical knobs and dials with separate pieces of equipment replicating
the screens over and over again even though a nice PC would do much better and be far
cheaper.
>> Daniel Wigdor: Right. Yeah, it's apparent to me. And in fact, just because I picked on you
earlier, I had an experience that made me feel very old when a new student came and joined
and wanted to do this, he wanted to extend the Conductor; and the Connector app that Peter
wrote just ran on tablets and other Android devices. He wanted to put it on the watch to start
to do some of the stuff Anthony had done while he was at Autodesk. And so we bought him a
watch, and I don't remember which one it was, and he starts wearing it around everywhere and
trying it out and just also having a smart watch, and then we asked him what do you like most
about the smart watch? And I asked him, do you play music on it? Yeah, kind of. Do you use
this, do you use that? You know what my favorite thing is is that it has the time on it, and I
know I can always just look at my wrist and know what time it is. I was like okay.
>>: Charge it or something.
>> Daniel Wigdor: That's right. I said that to him. I only have to replace the battery in this
every three years. Oh my God.
>>: How much did you have to pay for it?
>>: I feel like you pushed to heterogeneous spaces of the different devices. But it would be
more interesting to push it towards more heterogeneous interactions for [inaudible] sets
because I feel like current examples are mostly based on the pointing>> Daniel Wigdor: Yeah.
>>: So, for example, [inaudible] device [inaudible] then the way the countenance that has been
transmitted from there to here should be [inaudible] then. Maybe it's just for all the users,
developers because developers should care about how to make the countenance interaction
diagnostic [inaudible], and the users should [inaudible] adapt their mindset to different
[inaudible], and the researchers, some researchers are developing new interaction vocabulary.
They should care about the compatibility between the new one and the old one. Should that
be another line of [inaudible] of these devices?
>> Daniel Wigdor: Yes. So I absolutely agree. And so the point, where we started Panelrama
was by saying we're going to divide up the application and then intelligently send the panels to
the right devices. And then a clear natural next step is once the panel gets there what should it
do? Should it just be the same and or should be customized to the input capability? And I’ll
give you two pointers. One is Christophe Gaios[phonetic], whose last name I know how to
pronounce properly, who looked at some of this for mobile devices a long time ago; and then
the other is, in fairness, that was one of the key things from Anthony's project that I didn't have
time to show you, so you should see the paper, where he looked at if you have a watch and a
phone working together what are the right interactions to put on the watch, what are the right
things to do on the phone and how does the sensing capability of each complement each
other?
And so what I think of as the next step for Panelrama, which is the toolkit although the
company is called Conductor just to be really confusing, the next step is to say given this huge
number of interaction vocabularies and all of the great work people do all over the world of
what is the right way to interact with a touch screen when it's a phone, what is the right way to
do accelerometers on a device that has no screen, and having sort of a Rosetta Stone that helps
to translate all of those things sort of like the Google translate I guess of all of this to make the
panels do the right thing when they get there. And I think the choice that we made with
Panelrama which is to say we're going to use HTML 5. And so if you're using a tool like
Bootstrap which is the Twitter UI toolkit that’s become incredibly popular, whatever that can
do to adapt your UI to the different screen we get that for free because you can still use
Bootstrap, I’m using the right word, right? Bootstrap?
>>: Yeah, Bootstrap.
>> Daniel Wigdor: If you're using Bootstrap it will do the right thing when the panel gets there
but it's not enough. It doesn't say you need to point at this list but there isn't a touchscreen on
your watch, but there is an accelerometer so how do you do the right thing? So I one hundred
percent agree it’s the right next step for this project. I wish he would stay and do a PhD, but he
wants to go get a job instead.
>>: All right folks. Let's give another round of applause for Daniel.
>> Daniel Wigdor: Thank you.