>> Lee Dirks: We have a joint presentation this time. And we have colleagues from
inside the company, Deb Robinson from the Microsoft library. Yay!
[applause]
>> Deborah Robinson: I haven't even said anything.
>> Lee Dirks: So yes, another honest to god librarian here at Microsoft. There's, I'm
going to guess there's probably 75 or so people with library degrees or information
science degrees around the company. And there's a lot obviously from the I School here
in Washington, but from all over the place as well. Where was your degree from?
>> Deborah Robinson: Columbia.
>> Lee Dirks: Columbia, that's right. Just like, when there was still a school there. Yes.
And also Jacquelyn Krones, who's going to be from the Bing team. She's going to be
giving us some information about the website that I understand now has 27% market
share as of this morning, so.
>> Deborah Robinson: Powered by.
>> Lee Dirks: Powered by. Very good. [laughter] All right. So first I will hand it over
to Deb to talk to us about what's going on within the firewall with the Microsoft
corporate library.
>> Deborah Robinson: Thank you Li. Thank you and it's great to see all the Microsoft
library bags that you are using so I know Meryl gave you a quick tour of the physical
library the other day. So I hope you enjoyed that; it's a beautiful space. You may have
noticed that you didn't see a lot of the staff there. That's because the full-time employees
are actually not in the physical library. We’re actually sitting in a different building. The
people that manage the physical library in building 92 is actually a vendor company and
they get to sit in the library but we don't, sadly.
So I first started with the Microsoft library in 1996; I think the same week Li did. I was
here for five years and then I decided to change careers, and then I decided to change
back, and came back to Microsoft in the library in ‘05, the very end of ’04, the beginning
of ‘05. So it's been about 10 years off and on with the Microsoft library, and I love my
job. What can I say?
So here are some fast facts, but I'm not going to read this; I'm going to tell you other
things about the library. It was founded in 1983, 20 software packages and books. But
primarily the library was founded for a software collection. The primary audience at that
time was the software support engineers and they needed software, mostly competitive
software, or software that works with Microsoft software to troubleshoot when a
customer was having a problem. So the library was really built to house a software
collection. Actually it has books; it's library. But the primary focus was software. And
as we all know as software has developed and digital right management has developed
we don't really have a software collection anymore. It's very small; you really can't
circulate software, so it is largely books, and online resources. As I mentioned the fulltime employees there are about 12 of us that manage and run and strategize about library
services for Microsoft. And there's a really large contingent staff that manage the day-today operations of the physical library. Unlike a lot of special libraries, we still have a
physical collection. Scientists, as you probably know, love their books, so we will
probably for a very long time to come still have a physical book collection. We circulate
books globally so anybody that has access to the corporate network can look at the library
catalog, search for books and check them out, and we will send them to you wherever
you are located. If you are remote and you work in Rye New Hampshire or if you're in
China or in India, we will send you physical books.
So we have a site here in Redmond and we also have a library in Silicon Valley. We
have built relationships with a library in Hyderabad India and there's also a Microsoft
research library in China. And we partner with those libraries to help them organize their
materials. We do some producing for them and we’re just kind of sister libraries.
The library supports the entire Microsoft organization, all employees. The largest user
group is the sales organization, not surprisingly, they need a lot of competitive
intelligence. They need a lot of company information; they need a lot of product
information. It's also probably the largest employee group as well; but they are our
largest audience. In addition to the sort of the corporate library we recently also took
over the legal library. Law and Corporate Affairs or LCA has a legal library. They have
some books; is largely electronic. There's a lot of electronic research databases not
surprisingly for the lawyers. And that used to be managed by another group and a couple
of years ago we started talking and we have the contract and the knowledge expertise
within the corporate library so we decided to take on that responsibility as well. So we
support everybody from a corporate business perspective and we also now support the
law group as well. And to that point the collection consists of business and technical
sources. We will purchase materials that are asked of us if they fall within the realm of
what would make sense for Microsoft employees, so business and technical.
Foreign language is another really popular source. We used to get requests continually
for new additions of different languages and CDs and books and we finally a couple of
years ago signed an enterprise license for Rosetta Stone, and it's one of the most popular
resources that we have. So we do support foreign language learning as well.
One of the more interesting things that we have done lately is started a chat service so if
you go to the library portal, right; I have a slide for the portal. And the idea is if you're
on the portal and you can't find what you need you can just chat with a librarian and
somebody that mans the front desk at the library will respond to you via chat, and a lot of
the time people are just asking about how to check out a new book or renew a book
where they lost a book, so if it's a lot of administrative stuff around the collection but
they can just do an online chat with a librarian during the business hours. We hope to
expand the service, but right now it's Redmond-based 9-to-5 hours.
So here are some collection stats. As I mentioned business and technical are probably the
most prevalent resources that we have. We are going to try to build the legal collection a
little bit. Right now we have half as many electronic books as hard copy books but the
purchasing is going such that we’re buying just as many electronic books as hard copy.
Like I said the scientists and engineers love their hard copy books, but when you're
remote, it's a lot easier to access a book via Safari or download it from Overdrive. So
where we can we’re trying to go digital as much as we can.
We also have been focusing lately on the use of the e-readers. We get a lot of requests or
people will come to us and say, hey, I have a Kindle, you know, how can I use Overdrive
with my Kindle? And we made a pitch to buy some of the hardware and circulate them
the way that some public libraries do and some academic libraries do, but we got shot
down for that. But we do have copies. We have a Nook and we have a Sony Reader and
we have a Kindle, and we were allowed to get those so we could experience what it's like
for our users who want to use our resources on those e-books. So we have a lot of
podcasts on here's how you use Overdrive to downloadable to your e-reader. So it's kind
of fun. Actually I got a Nook Color for Christmas, love it. It's the greatest thing. So
that's little bit about our collection.
One of the more popular areas and it really always surprises me when we see what people
are checking out it’s not surprisingly, it's all about technical languages support languages,
but there's a lot of career development that people are pursuing using the library for how
to communicate better with your team, or how to communicate better when you have a
remote team. So there's a lot of--it's very interesting to see that people at Microsoft really
care a lot about their careers and they went to learn to be better in their jobs, and be better
employees and be better managers. So we see a lot of that which is, it's really interesting
to see that.
And it's interesting, I manage a lot of the electronic on-line resources and I do a fair
amount of research for people, and I have a little bit of my hands in the book collection
but when we have the, we had a big snowstorm the week of Thanksgiving and people
couldn't, the people who run the library couldn't get there, so I was working at the front
desk that day. There were people coming in and out and the calls that I would get were
from people who live in the area and had a book that was due back that day, and they
couldn't come to the library to return it. And is there anything I can do for them because
they could make it to the library so it was just, I'm always amazed at how much people so
much care about their books, and they take these things really seriously. And one of the
more popular features of the library portal is the ability to manager checked out items.
You can see which items you have checked out, you can see which items are overdue,
you can see where you're on a waitlist. It's fascinating to see how much people still love
their books.
We also have a service and this goes back to at least 1996, when I started, the daily news
alert service. It's a topical or should I say 13 different topical e-mails that we send out to
people who subscribe. It's a way to get a quick view of the news. So we have one that
looks at the news of the day regarding Microsoft, so it's a little bit about everything about
Microsoft. And we have these other topical alerts. So we’ll one on cloud computing.
And these are managed by library people. It's not an automated service. We have
librarians that cull through the online news every day and pick out the most relevant
stories on those topics, create an e-mail and send them out to employees. And it's one of
the most well-regarded services that we have from the library. When I talk with other
librarians that kind of current awareness program often comes up and there’s so much
that's automated now and, you know, why not just send a Google news alert or a Bing
alert or Yahoo, but people love this, and because it's hand selected news. It's not just a
list of stuff that comes from the web. It’s people who are actually looking at the news
and pulling out the most relevant stories. So it's, you know, it's an expensive service to
run, but we continue to run it because people just love it so much and it gives us a lot of
credibility and it’s kind of a win-win for everybody.
The most popular resources, Wall Street Journal.com; we have an enterprise license for
the Wall Street Journal. Directions on Microsoft, which is a monthly or every two month
publication that focuses just on what's going on at Microsoft. It’s supposed to be for
people outside of Microsoft but it's one of the most popular resources for Microsoft
employees. It gives you a lot of insight into where the company is going with different
initiatives and it’s really helpful for everybody in the company. Get Abstract is another
one of the most popular resources. You know five-page summaries of business books.
And this is what the homepage for the library portal looks like. This is a SharePoint site.
When we launched the site in 2007, it was on the ‘07 version. It was a lot of custom
work. We work really close with a SharePoint development team. And a lot of it we had
customized to SharePoint code. We recently migrated to 2010, and we now have a fast
search. But what you can see here is when you come to the library portal, you get sort of
a quick view of everything that you would find if you were to dive deeper into the library
portal. So you see the tabs across the top for the homepage, the books, the news, market
research, so those of the different content areas of the library that we manage.
And one question that we get a lot is well what about internally produced content, either
from Microsoft research in their papers or the sales organization produces a lot of content
for selling and strategies around selling. And we made a decision a few years ago to not
manage and not house internally produced content. Mostly because we didn't have a
good way to manage it well enough so a person had a good experience when they were
trying to find it. We didn't apply the metadata; the content was given to us and it was
searchable on our site and you could see search results, because we didn’t manage the
data, you wouldn't always get--you couldn’t find what you were looking for, necessarily
or it wasn’t a really great experience. So we kind of said, you know what, the library
licenses third-party content, and that’s what we’re going to focus on the library portal;
And that's what you'll find here. We license a very large number of research databases.
We partner with MSR and Co Fund IEEE. We have Nature and Science and Nexus
Direct and Dow Jones and Hoover's, vast number. In some ways it's like a small
academic library because we do support the entire company. We have content that
supports a great number of job functions. So we have a wide variety of resources. And I
think that's all I'm going to say. Let me see if there's anything else.
No, we do partner as I said so, like we partner with MSR on purchasing some content.
We partner with LCA to manage their collection. We do some funding with other groups
for a limited license content. And one of the partnerships that we also have is with Bing
and one of the great things that we’ve been able to do with Bing in the past year is
actually go to a couple of conferences, like ALA, and work the Bing booth and talk to
librarians about the main experience. So it's been a really nice way to be part of what
Microsoft is doing from a product perspective, and it's really great that Bing values what
we can do and bring to their audience as well. So with that I'm going to turn it over to
Jacquelyn.
>> Jacquelyn Krones: Hi I'm Jacqueline Krones and I'm in the Bing Product
Management Group. Instead of doing a demo today I wanted to have a discussion about
how consumers think about creating knowledge. So as information scientists, you guys
are so far ahead of what I'm going to talk about today. So when I talk about this don't
think of it in the context that she thinks she's telling us something new, but think of it in
the context of would the average person, your neighbor, how would they think about how
they search for information and they build knowledge. What's interesting is that the role
of the search engine is actually changed in consumer's mind from 2007 to 2010. And I'd
like to talk about that. It has--it presents some interesting challenges for us as a new
search engine. And then the second topic that I'd like to talk about is how consumers
think about social information. Which I was talking to a couple of librarian friends and
they said this is a pretty controversial topic, but again, it's from understanding how
consumers are thinking about it, or users, or just people, who are not experts. So these
are some of the things that are influencing the way the search engines are changing today.
The things that Bing is specifically focused on, or planning to focus on in the future, are
social information, mobile usage, and visual which includes mapping of spatial
information because those are the areas where we see there's the most unmet need.
When we think about knowledge creation, essentially what consumers tell us, and a lot of
this is based on some research that I did this summer where we met with 48 people for
four hours each in their homes in eight different countries. We call it ethnography, but I
really call it ethnography light, because a lot of it is a structural interview and then some
structured activities rather than just being part of the environment. But essentially what
we find is that the way things are changing is that in 2007, which was the last time we did
this work, people would tell us that knowledge lived in Google. And when I say Google
of not saying anything good or bad about their product, but as you know they have the
most market share and so when you talk about search, that's what people are going to talk
about. So please don't think I'm disparaging them when I challenge them, when I give
you the bad news about what consumers are saying. So essentially in 2007, people
believed that knowledge lived in Google. And they didn't really feel--we expected them
to have lots of habits around finding ways to save what they found, and synthesize it, and
store it and make use of it. And there was very little of that going on. Because people's
belief was I can just go back and find it again.
Well, of course, you’ve had this experience, and consumers have now had that
experience, where they find that it's much more difficult to re-find information. But the
more important change is that consumers feel more responsible for looking at several
sources and integrating that into their own personal knowledge, or in the academic world
you'd call a sense making, right? Most people didn't really feel that they were going
through this process when we last talked to them in 2007. And now people actually did
and it does change their behaviors and also their expectations. So I have a quote from
user who said that what he's searching for doesn't have a personality until he's given the
get go. He's just bland and boring. Once I’m able to make use of him for what I need he
becomes chipper, cheerful and brained up.
So this was from a technique called Z Matt where it's a more psychological approach of
an interview with an individual user. And so you get these kind of quotes from users that
are little more esoteric, but you get the idea. And the good thing is that using search
engines and using the internet is allowing people to feel like they have more control over
their lives. The challenge is that it's making them aware that there's a lot of holes in the
process.
So the big insights for us are that, you know, from a consumer’s perspective, I'm the
consumer; I'm the control, and my knowledge is personal. Until I've had the experience
of bringing information together and using it, it's not really knowledge to me. We have a
quote from Tim from the UK who said, and he was awesome. I loved his interview. He
said, “Search engine results are not knowledge. That's just sources. I make knowledge
from that data and from the information on those sites. But until I actually use it, it's not
really knowledge.” Another impact that the change is having on users is that they are
actually starting to save a lot more information that they did, and to do interesting things
with it. So I have an example of one of our respondents and I could show you video of
him, but since we’re running late, I’m not going to spend a lot of time on that. I'll just
say what he said, which is essentially, he's holding this book. It's a 300 page book full of
clips of outfits from the Internet. And what he says is, he's actually in Korea, and he says
I'm really interested in American fashion. And it's actually really hard for me if I'm
getting ready for the day to go try to find that on the fly. So what I do is I essentially go
find it when I have some free time, and then I pull it out and put in my book. Well, that
specific artifact is unique to him, but we found similar things for different kinds of
people. Another person has, was going on a trip, and instead of just pulling out her hotel
and her flight and all of that information, she had this 20 page booklet of information that
she took with her that included tables she’d created of things that she wanted to buy and
where they were and how much they cost online, and where the nearest Metro stops were
and three ideas of where she might eat afterwards. She had done all of this research to
synthesize this information before she left.
They believe that as a result of going through this process it's, they’re, the knowledge that
they’re building is really unique to them. And I think that anybody who’s participated in
any sort of discussion online anywhere will see that this is actually true. That really the
concept of authority has left the building and the best that we can get to with what we
used to call authoritative sources is really official sources. And the part of the challenge
for the user is that because they believe there is no external authority, they actually have
to spend a lot of time figuring out where all the information is and how to bring it
together in a meaningful way.
So another sentiment, another insight for us was that people are sure curating sites that
they use as a solution for particular topic. So again in 2007, people would pretty much
just use Google as the center of their Internet experience. Now people are saying well
obviously Google is not the best solution for a traveler. It's not the best solution for
shopping, or it’s not the best solution for my weird hobby, whatever it is. Mine is
machine embroidery. And I've actually taken the time to go out and figure out where the
best resources are, so I just bypass the search engine and I go to those places. Well, if
you think about how the Internet works today, that's not actually an easy thing to do, so if
there are four sites that I always use when I'm going to go look for an embroidering
pattern, I have to go from site to site to site and go through the same process at each site
looking for the information, and then somehow, you know, deciding across those places
what's the best thing for me right now.
The challenge for us as a search engine is that we end up in a situation where we’re not
essential to the person's life anymore, because they're just not there. And even though we
may have believed that a search engine was always more like white or yellow pages than
anything else, the consumer used to believe that it was a fundamental part of their online
experience. So one of the users expressed this as, I use Google more than I did three
years ago, but it's less important to me. So that's a challenge for us. The next big thing is
that users are looking both to official sources and to social data in order to make their
knowledge. So the interesting thing for us was that subjective information is becoming
perceived as being more objective, largely by accumulating a large amount of it, right?
So the best example, and this is actually a kind of an anomaly right now, but it's an
interesting anomaly. Is that we found that, I found users in San Francisco have now
learned that they can take the shortcut which is the following. If I'm walking down the
street with you and we’re going to go have dinner, I look at what's near me on Yelp, I
find, I want Italian, there's 300 reviews; it has four and a half stars; I'm done, right?
What we used to hear and what we would still hear about a special occasion is that I have
to go to the Yelp; I have to look at the most popular, the highest reviews and the lowest
reviews. I've got to look for clues in the reviews that say things that will help me figure
out whether this matches my preference or not. Did they say garlic mayonnaise or do
they say aIOLi? This is an actual example from a user. If they say aIOLi, they're
probably more of a foodie and they probably will have more ability to influence my
decision than if they say garlic mayonnaise, right? So what was surprising to me after
hearing that for several, for several years was for people to say actually I can make that
leap now. The social information has hardened up based on the number of, the number of
opinions and the high score, which is obviously infallible in lots of cases, and in many
cases there is just not that much information available.
The other interesting thing was you've probably seen some, in the news, there were, about
a year ago, it started talking about how, you know, Facebook was the largest search
engine after AOL or something right? And this idea that what was going to be next was
people searching on their social networks. That actually doesn't happen at all. And I'll
talk about why on the next slide. But what people are doing is using social information
that's available in lots of different areas especially forums and then of course review sites.
The challenge is that there are still times when people’s only or best recourse is a search
engine. But I came away from spending on that time with customers realizing that that's
where we’re failing them the most, right? So I was talking to Dr. Marti Hearst last week
at a conference and we were talking about this shift in users’ behavior because she'd seen
the Google as a source of knowledge, but she hadn't seen the shift to actually, no, I make
the knowledge from all of the resources.
And one of the things that she brought up on the pedal issues on that I thought was very
interesting was kind of this point. If you think about searchers tasks, they generally fall
in three buckets. There's missions, where essentially know exactly what you want and
you probably know where you're going to get it, and it doesn't take much time, and you
want to optimize your experience for efficiency, right? You want to get to the weather,
get the score, get the stock quote, buy a turkey for Thanksgiving. Then there's
excavations where you actually are going through this process where you’re going to
spend a lot of time with it, because what you're optimizing for is thoroughness, right?
You feel like you have to get the right answer. So healthcare often falls into this bucket,
but a lot of shopping scenarios do as well. And travel scenarios do as well, right? And
then the third one is exploration where you're really optimizing for novelty. So things
like Pandora, the toolbar one that I'm totally spacing on right now, but essentially where
you can say these are some of my interests and I need you to show me a bunch of things.
That's, you know, a very different mode than the first two.
Excavation is the--mission is the mode that I really see most people in tech jumping on
right now, especially in mobile, right? There are relatively concrete problems to solve.
There's lots of information in your environment actually about what your mission likely
is right now, where I am what time of day it is, what I searched for before. And so what
everyone wants to do, is kind of intuit what the user wants and [snaps her fingers] give
them that answer. And so there's lots of--I see lots of technology being developed there.
What Dr. Hearst said and what I agree with is that actually excavation is a much bigger
problem and harder problem solve. And there's less innovation happening in that domain
right now. And I’d actually say that probably the most important innovation is happening
with you, right? Because much of the domain that you're covering falls into excavation
when you're helping people with really pretty complex problems. So what are the main
problems?
Well, there's problems, you know, problems due to information overload, compounded
by Spanish content managing to find its way to the top of search engine results. There is
a privacy of text context/content. Although there are some solutions to this problem,
what we hear from users they don't know about it, right? So they say what's this green
plant in my yard? How am I going to put that into text? Plants, Nebraska green? You
know, how to I get there from the search engine box. Obviously there’s language issues.
So I gave you the example of the person who created the book with the 300 pages of
outfits. His technique was to go into image search, put in one of the style icons. He likes
Ken Begley and Justin Timberlake. And then look at the images, and use that as a way to
figure out where the US sites that will be most interesting to him likely are, right? So
he'll use the images then to navigate back to a site to find more examples of what he
might like. We found this to be pretty common and especially in Asia there was a real
thirst for having the rest of the world opened up to them. We found that to be less true in
the UK because their little insular and more true in the rest of Europe. So…Pardon me?
>>: The best of Europe is insular.
[laughter]
>> Jacquelyn Krones: Oh, oh, so this is what we heard. So the people in the UK were
less interested in seeing results outside the UK, right? The query formulation continues
to be a problem for people. And interestingly we have all these smart people at Microsoft
working on Bing with the president of our division, and he's an extremely smart and
bright person, scientist. And I was going through this data with him and he said, really?
When would people not know what their query was? And I think that you probably are
very aware of the pain that people have in trying to formulate a query. It's not a simple
thing to do. So I wanted to talk about social for a minute. I need to get all my transitions
up; I hate transitions. So this is a mind map of how people think about search. It's
actually based on a pretty small sample. There were 16 people in this study. And this
was based on this Zemont process which was the more psychological approach that I
talked about. So, while I don't totally buy into the whole mind map, I’d really like to get
another 40 or 60 people before I felt really good about this. One of the things that was
fascinating and I absolutely believe is true based on ethnography is that there's no
connection between people's social network and finding wisdom from other people's
opinions. So when I first looked at this I was kind of surprised because there's all this
hype around how people are searching on their social network, but the real purpose of
social network, of course, is to be informed about people and to belong. Where, where
they're getting wisdom, well they’re getting other people's wisdom from specific sites and
from Google is their perception, right?
But when we dig deeper with them what they tell us is that primarily they get those from
forums, from blogs and from review sites. And forums are a really interesting, a really
interesting space, because my model of what I heard users tell me was that the perfect
source of social information would overlap people where I trust their expertise. Our
preferences and needs are aligned and there's at least several people, right? So I don't
really find this on Facebook. On Facebook there are several people, right? And my
primary source of data for whether I have an expertise in a preference match is a personal
relationship, which is why people say Facebook's great because that's where people know
the people and so they have this information right? But as you know, you don't really
know everything that your friends and your Facebook contacts know. The secondary
cues for preference and expertise matches are really varied and they can be very subtle,
right? So people will say things like, again you know, an example of that would be the
subtle cue of, is it garlic mayonnaise aioli? And the second secondary cues for expertise
I find are generally social. So the example of this is people really like forums because
they get the concentration of expertise as opposed to like a Yahoo Q&A, where you got
this big general population. And they can look at how other people respond to the person
with the opinion as a way to gauge their expertise, right? So to the extent that other
people agree with me when I say that I think natural medicine is the way to go for
indigestion, then I'm, I accrue some expertise; I accrue some reputation, right? So the
other thing that we found which is the thing that I've referred to before, is that if you get a
mass of people agreeing on something, that essentially becomes as objective as you can
get, right? As close to authoritative as you can get.
What the then diagram represents to me is that there is no good solution right now,
because while there's a lot of good information in forums, it takes an awful lot of effort to
get to a place we could actually trust what people are saying in a forum and find any sort
of consensus that you could trust about anything or any other way to verify it. So I’m
going to give you an example of a respondent we had in Germany. A young woman who
has a rare disorder and she's been prescribed steroids for the rest of her life. And she
really feels like that's not a good thing for her health. And so she was looking for other
alternatives. Her doctors were kind of giving her no option and she found on this forum,
she found people discussing another option in newer medicine. And people would talk
about how that new drug made them feel, the change in their experience, and that's just
information that she could never get from an official source, right?
She then took the information about the drug. She went to her doctor and said, hey, what
about this and he said, no. So she was an academic, so she then went back and went to
the academic literature and found some support and brought that back, and then he said
yes, right? Most people don't have access to that kind of information or even the ability
to understand it, so there’s clearly a lot of opportunity here to find better ways to make
social information more useful and I would say safer, right?.
So just quickly here is an example of what a real user session looks like for a health
related query. The point that I wanted to make is if you look at the time that they spent
on this, they spent far over an hour. And if you look at the kinds of sites they clicked on.
This is my classification, they clicked on unprofessional information, which was my nice
way of saying spam sites, advertisements and then finally they got to some user driven
Q&A, went back to an advertisement. Then they got to an official source, another
official source, got back to spam, got to an official source, got to some more social source
and finally back to a professional source of information. So as you can see, there were
lots of times in here when the person really wasn't getting the best information possible.
Not just from the perspective of whether they, the person should actually be talking about
medicine anyway, but from the perspective of what the person was trying to do. Were
they trying to sell them something and were they just trying to get them to click some
ads?
So there's a lot more work for us to be doing here, both from helping people figure out
how to vet the official information and helping them make better use of social
information. Any questions?
[applause]
>> Jacquelyn Krones: Thank you very much.
>>: Is this a [inaudible]?
>> Jacquelyn Krones: Yes, it’s just an example.
>> Lee Dirks: Any other questions? Anyone? All right. Then I'll invite Jen up to get
started. Thank you very much Jacqueline and to Deb.
[applause]
>> Lee Dirks: Please let me introduce Jennifer Lynn from the Silverlight Team. She's
going to be talking to us about an extremely cool tool called Pivot. Thank you.
>> Jennifer Lin: Will that come up to me? Okay. Hi. It's on. It just sounds a little
crackly. I'd like to introduce you all to the Silverlight…
>> Lee Dirks: You might have to move it up just a little bit.
>> Jennifer Lin: Okay. Hopefully that will work. I want to introduce you to the
Silverlight Pivot Viewer control. This control is a multifaceted multimedia experience of
a data set that provides you an opportunity to look at trends within the data, and kind of
observe larger forces at work and drill down into specifics, and then formulate hypothesis
so that you can move from the data to something closer to knowledge. The first example
I wanted to show you is one in environmental info graphics, and I think it just popped up
behind my, there we go. Okay. So I'll just give you a quick tour of Pivot Viewer the
control. So each of these data points for this environmental data is a capture from an
array of video cameras in Tanzania. Professor Ilius Sloslasky of UCSD is doing some
research on the impact of climate change on various species in that area. So what we've
got here is what we call the filter pane. It kind of gives you different kinds of data about;
this is one, camera location here, all the snap shots that were taken there. You can look at
the latitude and longitude of the locations. The species identified and their genus and
family, and then the moon phase and temperature at the time that the snapshot was taken.
So when I was working with the researchers who provided this data we came up with
some really interesting trends that really were not coming up in any other way that they
were interacting with this data. Probably the coolest is this. You can kind of get a sense
from the data the distribution of temperatures in this region, but if you look per species at
the temperatures at which they were making appearances in these video cameras, they
found that they were very specific distributions that were not just distributed evenly
across different temperatures. So basically for each species there some prime
temperature that gets them out and socializing in the wider environment. And therefore
when you deviate from these temperatures with climate change that could, you know,
show that the species will have a harder time interacting with one another reproducing
etc.
Also very interestingly, is when you look per species at the moon phase at which they
were appearing, in this instance there was a very short period within the month that they
were studied where the species was appearing. Here you kind of get more of a
distribution, but definitely a clear preference to one specific end of the moon phase cycle.
And then for each species is kind of a different pattern. Really each of these discoveries
is leading to papers that are going to show up in Nature. There are things that were not
transparent when you look at this in a spreadsheet. And just the richness of being able to
kind of get some really granular information about the time and appearance and the
species, but then also look back and say, like okay, well what are the larger trends in the
data and kind of take data points and granulate, and take the granulated data points and
kind of facilitate the ability to formulate a thesis which you can then apply to other data
sets and see how they apply.
So I just saw this as a great introduction as to how scientific discovery can be aided by
really useful and intuitive tools. Ah, oop, ah, here we go. So here's a little more of an
introduction in to the tool. The viewer is a control based on Silverlight, which is a web
platform, which is then hosted inside a webpage or a web script. So basically when
you're developing against a Pivot Viewer application, you're going to have dependencies
on the Silverlight platform and you can kind of interact with it in a broader web
experience. So maybe you have a Pivot Viewer and you also have some table that shows
related data.
The next kind of tourist look at a scientific area that I wanted to show you is the human
genome. Just wanted to give people some background in genetics to kind of couch the
conversation. To give you a sense of kind of the scale of what it means to go zooming
from a chromosome level to the actual nucleotides, the actual genetic data, you can see
for each of these buckets, there's actually a very small section that has to be zoomed quite
large until you actually get to the one all the way on the left which is the actual, you
know, genetic material, the actual nucleotides themselves. And the other problem that we
have with this data is that there's a lot of information, but the noise to signal ratio is just
horrendous. There so much genetic garbage that's accumulated through the process of
evolution. So really the genes that we have identified are the nuggets of actual
information that we can take action upon, that live within just enormous data sets.
So the approach that I'm going to show you is another. Pivot View application. And I
wanted to kind of explain the visual language that we’re using for this collection. So
you'll see that there's a trade card that we have for each of the genes, and it gives you the
name, the band, the location within the chromosome and some more information about
the gene. For example, you know, the high affinity imu, immuno globulins. And then
when you get below is a colorful representation.
So each of the colors represents a different amino acid. An amino acid is a trio of
nucleotides so to encode the same data more succinctly you can translate from
nucleotides, the AGCTs, into amino acids, gives the same information but makes it a
little more compact. And then we just use different color representations to make it easy
to kind of scan across and look for trends. So I'm going to bring up the pivot viewer
application for this one. Here we go. So this first collection that we’re looking out as a
representation of the human chromosomes. Every human has 23. So this is the number
of chromosomes and this guy is X and this guy is Y. The length, or the height on the
screen is representing the length of the chromosome and the blocks, the bands that you
see, represent the density of gene data that's been identified to have some specific
purpose. So I happen to know that chromosome 2 is pretty interesting. So we'll load that
one up.
And what I like to do when I'm looking at these chromosomes, is I like to look at where
do they land within--these are each genes, so where does the gene land within the
chromosome? Interestingly enough there's kind of a nice distribution, a little bit shorter
on the beginning, which makes sense because often the ends of chromosomes have less
dense material or less dense content. And what I'm going to do, I'm going to look over
here. I'm kind of looking for what are some interesting patterns that we see and right
away I've come across some of my favorite genes which are these guys right here. It's
kind of funny to have favorite genes. And what's really fascinating about these is as you
can see, that there's this repeating pattern. And there's these genes that just kind of go
across in a diagonal throughout this whole gene here. And then interestingly enough the
one next to it also has this repeating pattern. But if you look around at other genes, that
kind of structure just isn't showing up.
So one thing I happened to have noticed is that these are collagen genes. So to just kind
of test this theory that maybe collagen has something to do with straight patterns, I'm
going to come do a search for collagen throughout the gene, or throughout the
chromosome. And so we get some results. I'll kind of condense them so you can see
them together. And you see that definitely there's some kind of theme here around
straight patterns. This one is kind of hidden in like the middle. There's something going
on there, right? It kind of gives you this hint. They got this little hunch that might be
able to develop into a thesis. And my coworker who worked on this data with me, she
went to a geneticist and kind of asked, like how come when I visualize my collagen genes
they have this interesting pattern, this kind of structure to them? And he said, well of
course, collagen would have a structure. It's a structural protein. Like, that's what it
does. So it's just kind of interesting to be able to come up with a thesis and validated
even if as kind of a biological truth.
You know this isn't my area of specialty but you can kind of get these interesting patterns
just from having rich data that's visualized in a way that's accessible to the human mind.
I'm going to just take one second and show you--let me open this guy--if you use kind of
a more traditional visualization of this same data, what that looks like in your usual web
browser, so this UCSC project is the most commonly used genome browsing experience.
And one thing you'll notice is that zooming requires clicking between HTML pages.
Every time I click I totally lose the sense of context because this whole area here updates
to show the new data. And then eventually, I think at this level I can just click on here
and actually get the detailed metadata. So basically the same metadata that I got just by
clicking and zooming in, in the pivot viewer experience, requires many clicks and, you
lose the sense of where you are in the data. You know, you're drilling in but you don't
know, well where was I in the gene? I don't remember. Was I at the beginning, in the
middle, I don't know. So this kind of gives you a sense of kind of like the standard has
been for this kind of experience.
I know it's getting to be a little late so I'm just going to go through the slides pretty
quickly. If you want to use the Pivot Viewer control, we have an API that passes data
between the control and the hosting webpage. I really don't want to restart. [laughter]
No problem, no problem. And kind of the collections are the visualizations I was
showing you. This is kind of the first example is a simple question. There's no linkage
between this and any other data. It's just a discrete data set. And then the second one I
showed it, you know, you had the chromosomes that you could zoom into and click on
and get to the more rich data set which had the actual gene data. And I wanted to put up
a few links if you wanted to play around with this on your own, it's publicly available and
we would love to hear feedback from you. This is definitely a fun little bit of Silverlight
technology that we've been working on, and we will be working on for the next release.
So, thank you.
[applause]
>> Jennifer Lin: Any questions.
>>: Yes, one question.
>>: [inaudible] in the long term on Silverlight. Because I know that there's a movement
over all, the technology is to move away. I know that the plug-in architecture is what
Microsoft pushes, but I know that overall across the technology [inaudible] there's a
movement away in the future for the next five years to move away from proprietary plugin architecture and move more towards
>> Jennifer Lin: More to HTML 5 kind of open…
>>: Yeah more towards HTML 5 technology.
>> Jennifer Lin: Yeah. Right now we don't see it as a competition yet, because basically
this kind of rich visualization isn't available in HTML 5. Definitely it's something that
we’re keeping in mind. You know, we started this project on WPF which is an even
more constraining platform, so we see this, these kind of ideas as being platform
independent and certainly I hope that we see it developed across different products. And
right now this is easily accessible in Silverlight, if you wanted to use it for something, for
a project you're working on.
>>: Okay. Cool.
>> Jennifer Lin: All right. Thank you.
>> Lee Dirks: Thank you very much. We appreciate it. And now I'd like to introduce
Brett Brewer from Office Labs. Brett's a general manager in Office Labs. I think was
recently with Live Labs. So you've transferred from one applied research team, kind of
as it were, in Microsoft to another.
>> Brett Brewer: Just can't get enough.
>> Lee Dirks: Exactly, exactly. So a brief overview about some of the cutting-edge
technology and projects that are going on in productivity apps. I'll it over to Brett.
>> Brett Brewer: Thanks. Yeah, I actually was on the team that originally created the
concepts behind Pivot Viewer so, that was a lot of fun. And there's a lot of promise for
that project going forward, certainly. Let's see, we have to figure out this high-tech stuff.
So yes, my name's Brett Brewer. I'm a development manager in Office Labs. Office
Labs is a lab directly within the Office development team. So it's a little different than,
you know, there are variations on this theme with innovation teams being within
corporate environments. Some are outside the product teams trying to look over them.
Some are directly within the projects. Some are directly within the same sort of
management structure as a product group and that happens to be the case with Office
Labs. It affords us some really nice features of sort of being right there in with the
product team and to be able to have somewhat of a greater influence on the product
teams, their actions going forward. In fact, with Office Labs--actually in the same
development, or same organization as the planning team for Office. So it's nice to be
directly in with the teams who are envisioning what are those next steps for the next
release, but also as you'll see, what are those, what is that vision for those future
productivity that we see 7, 10 years out in the future?
So Office Labs in brief, it’s about four years old now. There's 51 team members. This
spans across disciplines from development to program management design user research,
so it's an interesting mix of disciplines. We have a hedge portfolio of projects. Think of
this as just as you would manage your monetary portfolio. You want to have a different,
a wide range of projects across the different varying, mounts and dimensions like how
long they'll take. You know, here's a two-month project. Another one might take more
than a year or two. You might have some that are really technology focused innovations
versus something that's more about maybe a design or business focus. And you also
might have things that are more validated of an idea, like as soon as you hear the idea you
go yeah, I get that. Or it might be a little bit more on crazy scale and I'll show you how
we sort of determine that a little bit later.
We also in the team have grassroots innovators, community support. It's just a long way
of saying how do we help people within Microsoft create any idea that they think is a
good idea, without having to go through getting, you know, management approval and
have it be part of their job. How can someone who's in the developer division in Visual
Studio create something that's great for Bing? And so we have this sort of crosspollination of ideas and just an empowering to any person within Microsoft to build
whatever they want and give them a forum to be able to expose that idea.
And then the envisioning efforts are really about, as part of Office, trying to paint the
picture of what 7 to 10 years out my look like for an enterprise worker. So in some ways
that actually starts to overlap, feeling more like an entertainment thing where the ideas
about tell a story. Tell me a story of why the future will look a certain way. And that is
influenced by technology trends as well as trends in just how information workers go
about doing their business. So the mission is pretty simple. It's to accelerate productivity
and innovation at Microsoft. Accelerate, you know we want to be a multiplier to the kind
of innovation that already happens and hopefully fill in the gaps where they need to be
filled. It's about productivity, so, kind of like all forms of getting things done. It
certainly is aimed more directly at the Microsoft Office division, but we don't let that stop
us from, you know, finding good ideas if they span across different divisions. And of
course, we don't limit ourselves in terms of the technology we use. We'll build web
services. We'll build mobile applications. We'll build standalone client applications.
So here's what we effectually called the crazy chart. This is probably like the worst info
graphic you’ve seen all day. So, I'm a developer; forgive me. So this is what we call our
crazy chart and what we’re looking at this is, how do we decide what projects we work
on? Well, if it's already in progress within the Office division, it's probably not all that
smart for us to be working on it to. If it's obvious like oh, this is high priority for us and
everyone knows it's going to be in the next release or it’s currently in development, well
it's probably not all that smart for us to work on either. That would be duplicative work.
But the things that are non-obvious or, you know, a phrase that's been thrown around a
lot recently is adjacent possible, right? Ideas that are, well if you combine this idea with
that idea it's something completely new. That starts to get into where a lab, into its sweet
spot. How do you take something that's not quite validated and bring it to the area where
the product groups believe that yes, that's something that we should be investing in.
And then you kind of have the longtail of crazy. Which is some ideas just sound really
stupid when you first hear them, you know, if someone told you about Twitter five years
ago, you would probably go gee, I’m not really sure if that's all that interesting. But over
time what happens is things start to move from the crazy to the in-progress or obvious.
You can imagine this is sort of like tectonic plates moving in where these bars get higher,
there's more ideas that have been validated, that you're moving from the crazier nonobvious space. So the job of the lab is to take the ideas in here, in this adjacent possible
area, extend them in interesting ways such that they can become part of the plan of Office
or do inventions that are somewhat more crazy or harder to believe upfront, but with a
little bit of validation, can actually be shown to be something that's really quite
worthwhile. One interesting thing, aspect of the crazy chart is kind of a revisionist
history that happens. Things that were crazy three years ago, you can even take like
Kinect, you might've thought that was crazy two years ago, but now it's in progress and
already being done, right?
And so a little bit of revisionist history is things start out crazy, but once they become
obvious or in progress, it's kind of hard to think of them as crazy anymore. And so it's an
interesting look at how labs move things through this process. And of course the box is
just a way of showing here's our sweet spot for the lab’s to work on.
I wanted to show in honor of the crazy chart [laughter] a little bit of a crazy idea that if
you saw this, you might think, wow, this guy really is nuts. But the underlying vision
that's behind it is there is this theme about gamification in the enterprise. And if I'm
trying to teach somebody how to use Office how might I go about doing that in the way
that is most fun and most engaging for that user? Well if I do it with a game where there
are things like a very deterministic set of goals and achievements, and that helps them
understand how to do and use their tools better, that could be an excellent way of
integrating games within the enterprise. And so this is one of the ones we came up with
more than a year ago where clearly even today with the amount of laughter I heard, you
might expect that that's in the crazy side of the chart, right? But we’re on the second
version of this coming out in, in a month or so, and it really isn't angry birds, it's just a-probably a copyright violation there probably. Uh oh, I didn't say that.
But this is one of--so also as a lab you want to have centers of excellence, right? You
don't want to be a place where, well I can just send contract engineering to them to do
work. You want to have centers of excellence so people can come to you and say, gee,
we’re thinking about some kind of gaming thing. Well, who has the expertise in that?
Well these labs guys have tried these things in multiple ways, have validated or not
validated certain things, can we get to them for their expertise in how to go about doing
that? And so Ribbon Hero was about seeing if maybe one of those centers of excellence
is about gaming in the enterprise. And so it really is just a game played in Office
applications. You know, you use different features of the ribbon to expose how to
essentially just use Office better. And it comes with a really nice interactive immersive
environment for--one of the stages is an ancient Greece, you know, one is in the medieval
times, one’s in Egypt. And it's just a more engaging way of figuring out how to best
utilize the Office Suite better than you currently are. We're also seeing in a less geeky
way how this affects the sort of customer satisfaction with the applications over time.
But it gives you an idea of the kind of span of things that we'll work on within the lab.
There's two other parts of the lab that I want to quickly touch. I know I don't have much
time. One is the garage that I talked about and one is envisioning. I'm going to fly
through this because I know we don't have much time. The garage is really for
employees who like to build stuff. The garage is not a team. There is no organization
around it. In fact it's just one PM, one program manager on our team who basically is a
caretaker for the whole thing. But when you talk about multipliers for an organization
there's one PM who has been the caretaker for the garage now for a couple of years. And
has created this groundswell of grassroots innovation within Microsoft where everyone
feels empowered to just say, I got a great idea. It's not part of my regular job; how do we
go about building it? Well, maybe it requires some expertise I don't have. How do I get
connected with people who have like-minded who have that expertise? I might need
some server resources to actually it externally, how do I get that? These are the kinds of
resources that the garage would supply. And it really is about, you know, having that
mechanism for employees to experiment on their own. And if you look at the next slide
which is sort of a mishmash of different things that came out of the garage. There are
probably no developers here, but if you were a developer you would know Fiddler right
away, a very well-known debugging environment for the web. So these are the kinds of
things that come out of this grassroots innovation that would not otherwise have come
through the regular sort of planning and development process in a product group.
You know, Office Talk is something that's being integrated directly with our IT
environments internal websites. Open Off was, you know, was the first.net
implementation of O Off. Some really, really interesting work that's being done, some
very small, admittedly, right? But some could grow up to be really important things and
it also is a big differentiator for the general morale of any developer to say I've got a good
idea; I can make it happen. Envisioning is a part, sort of a sister team to the lab. It is
about the seeing what is that vision or painting a picture of what productivity looks like 7
to 10 years in the future. So if you can think of the garage as between now and six
months from now, right, that that kind of level of project. The labs are working at the six
months to two years type timeframe. Envisioning looks at the, you know, 5, 7, 10 years
out. And it's really understanding the trends, both technology and business trends for
painting a picture and how that picture gets painted is mostly through some really
fantastic videos they've created that really bring home how we think people work in the
enterprise, you know, 7 to 10 years in the future. And part of that is also having rooms
set up in our executive briefing center that brings some of this stuff to light where you
can see the video, but you can also see demos that start to say, these aren't videos that just
show some crazy future that we don't think will be realized. We can actually show
demos that can be building blocks for how some of these things will come to fruition over
time. And so if you want to learn more, Office Labs.com is where our external projects
are. You can find Ribbon Hero there if you want to beef up on your Office skills. And
then the envisioning video there as well, and you can see that for the Office division as a
whole, where do we see enterprise productivity going? So with that, do you have any
questions?
>> Lee Dirks: Thank you Brett.
[applause]
>> Lee Dirks: Any questions for Brett as our next speaker steps up?
>>: One more stupid question. So can you implement Ribbon Hero in the Mac version
of Office? Because we feel left out. [laughter]
>> Brett Brewer: We do not have plans to do that, no, sorry. We’re not that crazy.
[laughter]
>> Lee Dirks: All right. Very good. Well, we’re going to get set up here. One thing I
would like to say is, and this is Keith Steury from the Xbox, specifically the Kinect team.
He's going to--he is a research guy and he's going to be showing us some of the Kinect
research that was done. And I configure you're going to find that quite cool. But he was
also very, kind enough to bring a few Xbox games that we can give away. So for
everyone that registered, we kept your name, we've produced little slips for everyone that
was registered in the workshop today, and at the conclusion of this talk we’re going to do
a drawing and five games will be yours. So that's a nice lead-in, no?
[laughter]
>> Keith Steury: It looks like I'm going to be here for the rest of the day and I figured if I
didn't bring gifts, no one would stick around. And we will, I know, we’re running off a
little bit but were going to be done at 3:15, and we'll get you on the bus as quickly as
possible.
So a little bit about my team, I'm part of the game's user research team here at Microsoft.
We're a team of about 22 full-time staff and about 17 vendors and contingent staff. And
our goal or our mission as a team is to connect our target customers as we build games
using controlled research psychological methods. The reason we do this is to help teams
realize their design vision and to ensure that our games are engaging approachful and fun.
And we do this, when we do this, we want to make sure that our data is accurate, that we
it in real-time so that it's useful for the team to make use of it and that it’s specific enough
that they can take action on it. So it's not just theoretical data, but it's actually data that
they can act on.
We work on every game that Microsoft does. But today I'm just going to talk about the
work that we did working on Kinect games. Obviously a lot of work went into Kinect.
There was a lot of really cool smart people working on technology and art and audio and
all that. I'm just going to focus on the user research part. And specifically, I'm going to
focus on how that kind of overlapped with the Kinect technology as well as game design,
and kind of how we bring all three of these parts together, and bring it together to create a
fun and compelling experience.
So user research is really about understanding user behavior, right? So that's kind of
what it comes down to. And Kinect is a natural user interface. So this should be easy,
right? It’s user behavior. We finally broken through the barrier of the terrible controller
that what, if you're a gamer you think it's great. But if you're a non-gamer and we've
tested you, you sit in the lab and you go [acts out with controller] and try to figure out
how to use it. Well we've gotten rid of that. We no longer have this awful controller.
We can just tell people to jump. Everyone knows how to jump. They'll jump, problem
solved, we can all go home. Not a lot of work to do. It's easy. Except that what happens
when I tell a person to throw, to throw a ball? It seems like everyone's going to throw,
right? Ah, unless they throw side arm. And maybe they throw underhanded. Maybe
they do some sort of over the head kind of throw. What about people who use two hands
to throw? What about this guy? The problem is, is that while the controller, even though
it's an unnatural user interface, the problem is that there's user behavior, but then there's
the technology that translates that behavior. And that's where we run into the problem
and the difficulties. So while this controller is unnatural, it's great for development
because there's 14 buttons on it. And as technology all we need to do is look for an input
from one of those 14 sources and we know exactly what to do. Our lovely Kinect
product, the number of inputs it has is infinite. You can throw a ball here, here, here,
here, here, here, here. And then do that in a variety of different ways. So how do we
solve this problem? The ideal solution would be let's adapt the technology to the
behavior. Let's just bring it together. The problem is that doesn't exactly work. And the
reason it doesn't work is due to classical conditioning. So this is where I get to put on my
psychologist hat. So as most of you know classical conditioning is this idea that you can
get an animal to repeat a behavior by reinforcing that behavior. Well it turns out that that
works in humans as well. One of the classic studies in this field was that not only did
they reward, say, a pigeon for doing a specific behavior they just randomly rewarded
pigeons. So every once in a while the door would open and the pigeon would get food.
What happened is that the pigeon develop superstitious behavior. The pigeon started to
believe that it was doing things to make that door open. So it would jump on one foot
waiting for it to open. It would spin in circles and the door would open.
So what we did when we tried to adapt this technology to fit the person, is we did the
exact same thing. And the reason that we did it was because we started opening up a
variety of things that would trigger the outcome that's wanted. So a great example is an
early prototype that we worked on was just catching a ball. So you know you would line
up, the ball’s coming towards you. You would catch it. We’re not really good at
tracking fingers yet, so we needed a way to drop that ball after you caught it. So the idea
was well let's just have the user put their hand down. That's seems easy enough. You
dropped the ball. But then there's questions, okay how rigid do we want to be about that?
Does it have to be straight like this? What it the hand is over here? What if it's here?
We decided, well let's just kind of open it up. Let's just say a sphere right around here, as
long as their hand kind of crosses that plane, that'll work. So that's what we did. And I
didn't bring the video, because we have a policy of not showing embarrassing videos of
our participants. But we brought a participant in and she caught the ball and then she
went oh, I need to drop it. Started moving her hand around, and went like that, and the
ball dropped.
So the rest of the session this participant would catch the ball, slap her stomach, catch the
ball, slap her stomach, catch the ball and slap her stomach. And we saw this in a variety
of forms and a variety of ways. So it became really difficult to figure out how do we kind
of deal with this. So the way that we dealt with it is we figured out we need to get on top
of it. We need to start understanding users before we get to the technology part. We just
need to bring users in and observe what they do. So have them drop a ball by dropping
their hands. Understand what that looks like. Understand what behaviors are going to
happen before we start development, so then we can develop the technology as best as we
can around that user.
So we have pretty much a standard usability lab. It's a little bit different than the rest of
Microsoft in that it's really big, so that it's like a living room. It has a big TV and people
can come in and they can play these active games. And so what we did is we just had
people, like I said, come in and we'd save bowl a ball. And we'd watch them bowl a ball.
And we'd see them bowl this way. We'd see them bowl that way. And we recorded it not
only for say design purposes, but we also recorded it through the Kinect technology. So
that when a developer then decided that they wanted to make a bowling game, they
already knew how users were going to bowl and could design the game around it. Again
this is part of us trying to get timely information to developers. So this information is
available now before they start coding, rather than building a bowling game, bringing
people in, seeing where all the problems are and then building around that.
So we talked about Kinect and technology and that overlap. I'm going to talk about user
behavior in the game design overlap. I'm going to give you an example from Kinect
Adventures. This is a title that shipped with the Kinect sensor. And this was a mechanic;
it was jumping. So this one's easy. Users know how to do it. There's not a lot of
variation or least variation that we can easily code for. So this really wasn't much of a
problem. The problem arises is that now the user has to coordinate this in conjunction to
what's happening on the game screen. So one of those things was just an obstacle course.
Where you need to just need to jump over this bar as it approaches. And what you can
see happening here, and this happened across nearly all participants, is people are
jumping too early. So we're doing something wrong here. Obviously in real life you can
do this quite well. There are spatial cues that you pick up on that tell you when to jump.
But we're just not doing it for some reason in this game. So we tried alternative ways of
doing it. We tried swinging arms. We tried putting indicators in the game to tell you
when to jump. And it just didn't seem to work at all.
People are still too early. I'm going to jump ahead. Oh yes. So the solution for this was
what we weren't able to do with the Kinect technology. That was to change the game
design to match the user behavior. So what we did, and this is pretty subtle, is that we
changed the game so that the person there is ducking and then she jumps, actually her
timing there was fine. But I think it's this next one, that was fine and the next one she
jumps too early. But she still had success. Why is that? Well, the reason is, is that when
a person jumps, we speed up the cart that they're on. So when they jump too early what
ends up happening is where they would've come down on the obstacle, we've now passed
that obstacle under them. So we didn't worry about fixing the problem of people jumping
too early we just adapted the system to it. And the really cool thing is that this actually
turned into a game play mechanic. So in this game you are rewarded for finishing it
earlier. You get more rewards, but obviously if you're jumping through the whole thing
you're going to perform worse so you can have it, it's going to be more difficult to
perform the tasks that you need to do while you're going through the system. So it added
this game play trade-off that actually ended up working out quite well.
So we've got all three of those pieces. And now the job is to bring them all together. So
where we’re at now at this point in development is we know the sensor’s working, for the
most part. During launch it was quite a trek with the technology constantly changing and
sometimes getting better and sometimes getting worse. But we'll say that technology is
working. We know how people are interacting with the game. Now we just need to
make, bring all of these together and create a fun and compelling experience. So to do
that, we don't need to watch them anymore. We know how they're behaving with the
game. That's a known quantity. All we care about now is getting as many people as we
can to try out the game and give us their feedback on it. So to do this we have other
facilities called play pods. It's cut off there at the bottom. These are just rooms that are
kind of, I don't know, maybe 8 x 10. People can come in, there's a TV Kinect sensor and
there is a computer where they answer questions. And we just have them play the game
and then we asked them a bunch of questions about their experience and get their
feedback.
So we might ask them things like, about the Kinect technology. How accurate was it?
How responsive was it? We might ask about their behavior? How difficult was it to do
that move? Was it too easy? Was it too hard? And ask them about game design. What
did you like about the game? What did you dislike about the game? What can we
change? How is the pace? Was it too fast? Was it too slow? How was the length of the
game, too long, too short? All of those types of questions.
And we repeat this process over and over and over again until we've created a fun and
compelling experience. So that is my ten minute talk on how we worked on Kinect. And
a little data, we did four launch titles within Microsoft Game Studios for the Kinect.
They were Kinectimals, Adventures, Joy Ride and Sports. We ran 113 studies total on
those four titles. And that resulted in using about 3250 participants. We did all of this
within about a 15-month period of time. So thank you to everyone who actually worked
on this. And also just a quick plug. If you're in the Seattle area and you want to sign up
to be a participant and come in, you can go to Microsoft.com/playtest, and sign up there.
If you have kids that like to play games, we’re looking for lots of kids and a broader
audience. Obviously we’re not making just Halo anymore, so were looking to really
expand the kind of people that we test. So thanks.
>> Lee Dirks: Awesome. Thank you, Keith.
[applause]
>> Lee Dirks: I have a question for the audience. How many of you want Keith's job?
[laughter]
>> Lee Dirks: I have a question over here I believe.
>>: I have a question about how do you [inaudible].
>> Keith Steury: Yeah, we actually have… So the question was how much we thought
about, really about accessibility issues and people who have different physical abilities.
We have a group within Microsoft that is focused on that. We did the best that we could.
We tried to identify ways that, you know, hey we don't need to use the lower body for
this. We did some things on a couple of games where it's kind of no fail game play. So if
you couldn't say jump, you wouldn't fail; you just maybe wouldn't score as well. But it
was one of those things that in the rush to launch, it was really hard to prioritize that. But
that is an initiative that we are looking at within Microsoft Kinect Studios.
>> Lee Dirks: One more question, or who wants to do the drawing? Can we do the
drawing?
>>: I have one question. So how does games ability research, how do you classify that
as being different from doing [inaudible]?
>> Keith Steury: Yeah. So the short answer is that in traditional usability research,
faster, quicker, shorter is better. So the most usable game in the world is a big button that
you press and it says you win. That's not a compelling experience. So that is where the
attitudinal stuff becomes really important. Traditional usability research is used to
remove blockers to enjoying the game, but then it becomes an issue of how do we test for
attitudes? How we test for game play strategies? And really thinking about what is the
optimal path for the player? What is going to be the most fun experience and testing to
that as opposed to testing to, easy is better.
>> Lee Dirks: All right. Let's do some drawing. I'm going to hold the box and I'm going
to ask you to shuffle around and then pull out a name. This will be for… Who is it?
>> Keith Steury: Oh you're going to make me say names. John Petko.
>>: Whoa, that's me.
[applause]
>> Lee Dirks: All right next. Table 3.
>> Keith Steury: Ernie Hood. Is Ernie here?
>>: I'm Ernie.
[laughter]
>> Keith Steury: Must be present to win. Ronald Larson.
>> Lee Dirks: I saw you. Yeah, there you are. It's a flight simulator 10.
>> Keith Steury: I knew this was going to happen. Lee.
>> Lee Dirks: Jackson Lee? Ah.
>> Keith Steury: And Alan Wake. Andre Brock.
[laughter] [applause]
>> Lee Dirks: We have one more. Halo Reach.
>> Keith Steury: Brian Landry.
>> Lee Dirks: Is Brian here? Brian? No.
>> Keith Steury: Mohammed Jahari?
>> Lee Dirks: Yes. All right.
[applause]
>> Lee Dirks: All right everyone this here concludes the meeting. Thank you so much
for coming out and for your patience and flexibility and have a fantastic conference.
[applause]