>> Evelyne Viegas: Okay. Welcome to the final session for of the last few days.
This session is all talks are all by Microsoft employees. And the first one is going
to be a little bit different from all of the technical talks that we've had the last
couple of days. It will be by Willie Callison who is European Privacy Director and
Michael Brick who is the attorney for Virtual Earth, and they're going to talk about
legal implications of geographic data.
>> Willie Callison: Thank you very much. Can you hear me? Is the mike on well
enough? This is a really bad idea actually to try to get two attorneys into 20
minutes and late in the afternoon and especially Microsoft attorneys because we
tend to drone on and on and on.
I'm going to talk about one of the bigger legal issues that affects location based
services, which is privacy. And then Michael will talk about a slice of the
technology in the different legal issues that affect it and that's more aerial
imagery and the different aspects of aerial imagery.
So I'll try to keep the slides fairly light. We tried to trim the text a little bit. First
what I want to do is kind of lay the ground work for what we're talking about when
we talk about privacy and what the considerations are, then a list of a few things
you should avoid or areas that cause concerns and then a few things that you
should look at trying to do as you're developing products.
The first bullet here, personal data could be the topic of an entire seminar on its
own. Personal data depending on where you are, what you're looking at is a very
broad concept. And I should talk about first the geographical jurisdictions of
privacy. I mean, you obviously have been United States and if you're services
are limited to the United States that's well and good.
If you have a global service you also have to think about Europe, you have to
think about Canada, you have to think about individual states who have privacy
laws, like South Africa and Russia and various Middle East turn countries now
have their own privacy regimes. Those different regimes will define personal
data in different ways and the way Microsoft approaches it we try to set the
highest bar possible so that we have a global program.
So by and large what that means is we look at European rules. The European
rules are the most stringent privacy rules around the world and they are being
exported to other countries. So the Russian rules adopted two years ago are
very similar to the European framework.
The concept at a thumbnail level, the concept of personal data is any data, piece
of data that could be used directly or incorrectly to trace back to an individual, a
specific individual. So it's not data related to an entity, it's not data related to a
place, it's to a person.
It's also data that indirectly can relate back to them through the use of other data,
so associated data also becomes personal data and also has to be used and
controlled within the data protection rules. So the obvious things are names,
email addresses, phone numbers, physical addresses, pictures, serial numbers
or social security number and so forth.
The less obvious ones and the ones that affect technology services most fall into
areas like cookies and IP addresses and similar types of machine identifiers or
client application identifiers and so forth.
Under the European rules, the rules by and large say that you can't process data
unless you get user concept, give user notice, give user control of what you're
doing. The European rules are a little -- have a few tighter aspects in that even
with clear user notice, clear user consent they will add certain requirements like
concept of necessity for purpose.
So if you're providing a service to the user and you're collecting things like
income data and school and education data that doesn't relate to the service
you're fulfilling for the user, European rules will say that's overboard you can't
necessarily collect that unless you do have a legitimate purpose benefitting the
user itself.
In Europe there's a split between the concept of opt in and opt out and this is
really a default setting so opt out is you essentially say you give the user a notice
form that's prechecked and it says can I use this data? And the user unless they
uncheck it they've opt -- they have allowed you.
Opt in is them making an affirmative selection that you know, checking the box
that allows you to use the data. Europe is split and some countries require opt in
for everything, France, Germany are the most predominant.
Europe differs from the U.S. and there's a big argument going on right now
around IP address and by and large what we're recommending product groups
right now is that IP addresses in and of itself should be considered personal data
in constructing our services and products.
In most cases you have associated other data with IP address enough that it is
personal. I won't get into some of the other topics. Anonymization is a concept
of scrambling or encrypting an identifier so that it becomes unidentifiable.
We do that with the live ID quite a bit, we don't necessarily do it with IP address.
And the concept of irreversible there is a concept that means essentially we
throw away the keys, so we make it unrecognizable and the company cannot
possibly go back and trace back to the original user.
These are again at a thumbnail level and conclusionary level some of the areas
where you should start worrying about your service or your product. Improper
notice, the concept here is that you've given the user notice of what you're going
to collect, what data you're going to use, what you're going to use it for, but if it's
not sufficient you can run into data protection problems.
Unauthorized sources, what I mean here is really you collect user data about an
individual but they're not your user. You collect data from a user about other
people. If you don't have consent from the owner of that data, you can also run
into problems. The conversely, if your user gives you data and you make that
data available to another person, another user, you don't do it properly, again you
run into problems.
Unnecessary usages, usually this is for marketing and data mining. So you
collect data, you provide a service, you provide a customized service you
optimize it for a user but then you're also taking that data and you're using it for
market surveys and trying to commercialize it.
Empowering big brother, this is really a U.S., European situation right now. By
and large Europe is changing its privacy rules with a focus on protecting
European user data from the CIA and U.S. law enforcement. So there is a very
active movement right now where the European privacy rules are looking at
minimizing the data that law enforcement could come after.
The last one is very fuzzy concept and it's not really something that they've really
concretely started enforcing but we started hearing noise from the enforcers.
And this is the idea that especially around social networks and those types of
services you are encouraging users to overexpose themselves online or through
social networks. And that if you don't go overboard or you don't thoroughly give
the users control and mechanisms to change -- to protect their own data, you're
in essence misleading them into disclosure stated intent.
So very quickly, what to do. Most of these come at the product planning and the
product engineering level. The first part is to try to minimize the amount of data
you collect in the first place. You're not subject to the rules if you don't collect
any perm data at all, so that's the ideal. You rarely can get there with online
services.
But the idea is to minimize the amount of data you need and try to scramble or
anonymize whatever you can, so that if you can create an unrecognizable
account number and track that account number without knowing who that
account belongs to, that's preferable.
Obviously you have to disclose correctly. Engineering your services so that user
control is as thorough as possible. So the user has the right to tell you, I don't
want to be tracked this time, I don't want to be tracked on your service today or I
don't want to be tracked anymore or I want to erase everything that's been stored
up till now and start over, those mechanisms go a long way with privacy
regulators.
Strong takedowns, procedures and mechanisms. Mechanisms we often
engineer in, but the procedures also have to be there, so if you get notice of a
violation you have to be in a position where you're going to take action on that
notice and actually take down the offending page.
And finally, we haven't had time to go into it deeply, but there's -- there are a lot
of complicated rules about sharing the data with other third party commercial
services so managing third party access and cross user access to data is also
very, very important. So I'd be happy to take questions at the end, but I'll turn to
Michael now.
>> Michael Brick: Thanks, Willie. For those of you who I haven't met, my name
is Michael Brick, and I am the product group attorney for the Virtual Earth product
team and business development teams here at Microsoft.
And so I'm going to, there we go, I'm going to focus my comments today sort of
on legal issues related to the intersection of location data and imagery. And
Virtual Earth and the Virtual Earth platform is a great background for this topic
because it is incredibly rich and has so much location data and the ability for end
users to sort of correlate that data with imagery is growing, you know, with each
release. So it's a great exciting product.
I'll focus first on really the regulatory regime related to the collection and
acquisition of satellite and aerial imagery and also touch a bit about a sort of
popular topic in the press, which is the rights of people caught on film through
street level imagery. My comments or my short comments here today really
focus on U.S. law and I'll make a few distinctions, and also I won't be talking
about security regulations. Those vary by country and it's hard to make
sweeping generalizations, particularly in such a short amount of time.
But most of you know this, but Virtual Earth has made up predominantly or in
large part with third party content and third party imagery, imagery that we
license in generally satellite and aerial imagery. And the way that the legal
regimes work at least in the United States is that regulation on the collection of
satellite and aerial imagery really focus on the generators, the creators of that
imagery.
So in terms of satellite imagery, it's the satellite companies, the private
companies which own those satellites which are regulated.
And in particular the U.S. Congress passed what's called the Land Remote
Sensing Policy Act of 1992 and that act gave the U.S. Department of Commerce
the power to create sort of a licensing and administrative system for the
regulation of private satellites and private remote sensing space systems.
The Department Of Commerce immediately delegated that licensing power to
Noah, and that agency is the agency today which issues licenses for satellite
companies. And gives them the permissions, the permission to take satellite
imagery. They do inspections, audits, et cetera.
So again, the in terms of satellite imagery the regulatory regime is really around
controlling the satellites licensing process.
In terms of aerial imagery and I have a picture up here of one of our ultra cam
cameras and a plane from the Netherlands which flies an ultra cam camera. In
the United States the Federal Aviation Administration regulates flight plans. So
they do not tell you what you can and cannot take pictures of, but they tell you
where your plane can fly.
So it's an important but subtle distinction. If you're camera in the airplane that's
taking the aerial imagery can take a picture of something in a restricted flight
zone, like in Washington, D.C. that's not taking of that building, take a picture of
that building in the restrict flight zone is not prohibited. Again, the only
restrictions are where the relevant aircraft can fly.
So what's the legal question. You know. The legal issue is really can or is it
illegal to fly over a private property with a satellite or airplane and take a picture?
And the answer is no. In the United States it's not illegal. Courts have
consistently held that capturing and viewing satellite and aerial images of a
private person's property does not rise to the level of trespass. Trespass is in
many ways the primary claim that would be brought, you know, intrusion or
interference with a landowner's ownership of their property.
So you can even fly over Disneyland and take a picture of Disneyland even
though or the Microsoft campus, even though it is private property.
And there's a relatively famous case out of California where Barbara Streisand
tried to sue a nonprofit organization which was collecting imagery of the
California coastline saying, she didn't try to argue trespass, she tried to argue,
you know, interference or invasion of her privacy. And she lost and the court
held that the environmental group's pictures of the California coastline, which
happened to include Barbara Streisand's mansion in Malibu, the capturing that
mansion was just incidental to the use of the imagery, and so the group was able
to keep the imagery up.
But it is important to remember, though, there are special issues around the use
of celebrity, images of celebrities and images of celebrity properties.
Google made a big splash last year with their launch of Street View. It is a
product that got privacy advocates around the world very worked up and
promoted a lot of discussion in the press and on blogs. You can see here an
example of some Microsoft street level view imagery where it's not particularly
good resolution, this particular image, but there are some individuals caught on
film there, caught in the crossfire. And so the legal issue or question is whether
taking an individual's photograph without their consent is considered an invasion
of privacy, and in the United States it's not. You know, the holdings that have
continually come down from courts, the legal precedent is that the mere taking of
someone's paragraph in public without their consent is not ordinarily considered
an invasion of privacy. In the United States we have a very low expectation the
courts tell us a low expectation of privacy in the public space.
So the issue really is or the question really should be was the photo taken from a
public or private space. And there are certainly issues if the photo is taken of an
individual in a private space. And we're really here I'm talking about, you know,
images where, you know, we catch a person's face and can make out who they
are and their likeness or, you know, perhaps we capture their license plate
number. And again, there's -- there aren't a lot of restrictions on the use of
images which capture people in public spaces.
This is much different in Canada so we're told and much different in Europe as
well. In particular, the privacy commissioner of Canada has been very outspoken
since Google launched the Street View product, and she said -- has said very
emphatically on a number of occasions that individual consent would be needed
unless the pictures were sufficiently and irreversibly blurred to the extent
necessary so as not to be able to identify individuals.
And I understand from the same article where I took this quote that the privacy
commissioner also said that she has worked out a deal with Google and their
Street View product that they would always blur faces and license plates. And
the UK Commissioner has weighed in as well and indicated that, you know, if the
resolution is sufficient to identify the individual there will likely be privacy issues in
the UK with similar type products.
So I'll just close as I'm out of time, but I just wanted to emphasize what Willie
stated and encourage all of you to really focus on privacy by design. That's a
term that's come out of meetings from the European Privacy Commissioners.
And what they've said and, you know, what is true, it's much easier to develop
and create a product that has privacy designs during the development phase
rather than to be told that you have to go back in and try and implement some
sort of privacy or takedown measures after the fact within the product. So just,
you know, be aware that there's certainly a lot more wiggle room in the United
States with regard to privacy at the moment. But to the extent that, you know,
our location based products continue to aggregate personal data and images and
pictures of people's homes and their backyards and how much they paid for the
house, you know, everything that's going on about their lives, regulators will get
more involved here in the United States and certainly been the movement in
Canada and in Europe.
So just food for thought. Thank you guys.
(Applause)
>> Evelyne Viegas: Thank you, Willie and Michael. I believe they might still be
here after the session for questions. We actually don't have time for questions.
I'm sorry.
The next speaker is John Krumm who most of you know he's a researcher at
Microsoft research in Redmond, and he's going to be talking about what kinds of
things you can do with lots and lots of data about where you've been.
>> John Krumm: Working on. Pardon me? I am, yeah. That works. Okay. The
first project is data mining collections for finding points of interest, and the second
project is using markup models to predict where a driver is going to turn when he
gets to an intersection.
So this first project on data mining for points of interest. So a lot of organizations
have maps and they have a database of points of interest, but it's hard to keep
those up to date. For instance, new things come along, this is a new park that's
in downtown Seattle. There are colloquial names for places that you don't get
any other way. And some people at the company might just not be brave enough
to put in certain points of interest like this one up here.
So what we were doing is looking at Virtual Earth collections in order to pull out
points of interest and so we got a lot of collection data thanks to Steve Lombardi.
Anyone can go make a collection in Virtual Earth, which is just a series of push
pins, and maybe you could indicate your favorite golf courses, your favorites
Mexican restaurants or something like that. Each push pin has a latitude and
longitude because you click where it's supposed to go, and then you can add
some text, a title and some notes describing the push pin.
So our idea was could we look at this text over here to try to pull out possible
names of points of interest.
And we looked at 40,000 push pins just around the Seattle area. These
collections are growing pretty fast. There are 25,000 new collections that are
made public that are coming up every month.
The basic approach is what we do is we break apart those collections into
individual push pins and then we do geometric clustering on the push pin. So
let's say we found this cluster of push pins down here, this is in a park called
Merrimar Park, we look at all the text that people typed in these push pins and
we looked for short phrases one to three words long and we try to find phrases
that occur frequently inside this cluster but infrequently outside the cluster. So
it's a TF IFD kind of thing, if you're familiar with document analysis.
And so if you can find a phrase in here which occurred frequently but didn't occur
frequently outside the cluster, that means that that phrase is probably specific to
that area and that might be something that's good to pull out and add to the map.
So we did a little user study on that. What we did is we pulled out a bunch of
potential points of interest names. Some of them we are able to find in a
database of points of interest, so they were already known by our mapping group
and some of them were not. So we made maps, paper maps that we showed to
people that had six points of interest on them. Three of them were found in the
database and three of them were not. The second three were the ones that we
found.
And we asked people look the a those maps and say, you know, if you recognize
the points of interest and it's about the right location, circle it. If it's, you
recognize it and it's wrong, cross it out, and if you don't recognize it put a
question mark by it. And so the question was are the points of interest that we
found that weren't in the database about as good as the ones that were already
in the database?
So here are the existing points of interest that were in the database, here are the
new ones. About 16 percent of the existing points of interest people said were
okay. About 15 percent of ours were okay. 1.4 percent of the existing ones
people said were wrong. About five percent of ours were wrong. And then by far
the vast majority of people just didn't recognize.
But this shows that the new points of interest that we were able to find were
about as good as the ones that were already in the database. So this is
potentially a good way to keep maps fresh and up to date.
These are some of the points of interest that we found that weren't yet in the
database.
The other thing you can do with this is quite a bit simpler and potentially also kind
of useful. I apologize for the fonts. When I imported this Power Point or this
Excel spreadsheet.
So you can rank points of interest by how popular they are. So let's say you
have a small screen, you can only draw a small map, or maybe you want to
decide, well, if you can draw only a small map you can only show certain points
of interest on because you just have -- don't have enough room. Or let's say that
you're trying to decide which buildings you want to render in 3D. Well, maybe
you could use a technique like this to figure out which points of interest or which
buildings are most popular and you should pay attention to.
So these are ranked by popularity. Of that region around Seattle, those 40,000
push pins, these are ranked number 31 through 60 popular points of interest, and
that's how many push pins we found. And this one over here, I'll just let you
maybe try to guess before I show the spreadsheet what the most popular point of
interest in the Seattle area was and what number two was and what number
three was. I think if you're familiar with this area at all, it's probably pretty easy to
guess what number one would be. When you see number two I think you'll say I
might have not guessed that but it makes sense, and then number three was a
little bit of a surprise to me.
So I've got this dramatic building here. (Laughter)
So if you said Space Needle you were right. You know, that's by far the most
popular point of interest. Pike Place Market is a popular tourist place, so that it
seems to make sense that that ranks up here.
And then number three oddly is Redmond Town Center which is a shopping mall
that's really not all that far from here. So there maybe were a lot of Microsoft
people putting in push pins.
(Laughter)
But so okay, it's number three, it's a little bit unusual but it's kind of in this big
equivalence class of things that are about as popular as all the rest here. So
these first two are way, way more popular than all the rest of these. Okay. So
that's that project.
The other one I want to talk about is a markup model for driver turn prediction.
Here is say someone gets in their car and starts driving, can we predict the turns
they're going to make as they make a trip. So you might be following this white
Bronco that's driving around and you want to know which bridge to go stand on
next to see this thing drive by. So what we do is we say, well, we have observed
the person on the current drive on this segment, this segment, this segment and
this segment, can we derive some probability for what they're going to do when
they get to this intersection over here? Applications of this might be anticipatory
warnings, something is wrong on the road that you might want to know about
before you get there, points of interest or even advertising. It would be nice to
give an ad before someone is actually driving by the store that you're advertising.
And even automatic vehicle behaviors.
And you can imagine, you know, for some people maybe want to turn off their
blinkers automatically. (Laughter)
We base this research on a bunch of GPS data that we've been gathering over
the past three years. This is a picture that's made -- that was made by one of my
summer interns. We've got about 16,000 trips in this database so far, about two
weeks from each person. And so what we do is we build up a markup model of
people's behavior as they're driving. So we have to represent their trips in just
some kind of finite alphabet to describe the roads they drove on, and so what we
use is just road segments, where road segments end at intersections or where
the road name changes or it dead ends.
So what we would with a markup model is say well, we've seen you drive here,
here, and here, maybe we've seen you do that ten times, and nine times you
went that way and one time you went that way, so next time we see you do that,
those are the probabilities that we will assign to what we think you're going to do
with this intersection. We had to do some map matching to take our GPS data to
get to the actual roads.
Here's what our accuracy looks like. This is one-ahead prediction accuracy. If
we look at the past one road segment you've driven on, we can predict with
about 60 percent accuracy what you're going to do with the next intersection.
And as we look further into the past, we can do a better job of predicting what
you're going to do at the next intersection. So we're up here about 90 percent
accuracy.
There's quite a big jump from looking one road segment into the past to looking
two road segments into the past and I think that's probably because looking two
road segments into the past we have some sense of your are direction on the
road. If all we know is that you're here then you could go to any one of these six
other roads. But if we've seen you drive here and here, that kind of cuts down
the possibility. So that's why I think there's a big jump there.
We can also predict more than just one road segment ahead. We can predict
however far ahead we want to. And so this is our accuracy if we're looking at ten
road segments into the past, how well are we doing predicting the future. So one
road, if we were predicting one road segment into the future, again we're 90
percent accurate and as you're predicting further and further ahead the accuracy
drops down to about 50 percent here. Road segments are about .15 miles long,
so this is predicting out one and a half miles into the future. So I still think that's
pretty good. And with these two lines here, they are what you get if you were just
rolling the dice and just doing random prediction into the future.
So when you're out ten road segments in the future, basically, you know, your
prediction actually is going to be zero.
Okay. So that's the end of my talk. I talked about data mining using these push
pins and pulling out points of interest and then using driver's past history in a
markup model to do probabilistic predictions on what they're going to do at the
next intersection.
So thanks for listening.
(Applause)
>> Evelyne Viegas: All right. Let's take one question. All right. Or zero
questions. Thank you, John.
>> John Krumm: All right. You bet.
>> Evelyne Viegas: Okay. The next speaker is going to be Steve Lombardi, who
is a senior program manager at Virtual Earth, and he's going to talk about
collections.
>>: Just talked about collections.
>> Steve Lombardi: Better. Low. Okay. Sorry about that. I'm going to show
you some new stuff that we just released in Live Map/Virtual Earth last week,
show you a few more statistics that update some of the stuff that John showed
you a moment ago and, most importantly, APIs. I'm going to leave you with
some secret hidden rest APIs that let you get at all this data if you want to use it
in projects that you're working on.
So collections, these are things that Live Search maps that people create push
pins, 3D models now cruncher layers, polygons, poly lines, they group them
together, it helps them tell a story.
The GeoWeb is sort of a superset of the collections on our site and all of the
other stuff that's going on on the Web where people are contributing little bits of
geoinformation. It could be in the form of Google's equivalent of our collections
features, Google My Maps, there's historically been KML files coming from
Google Earth, Websites like Palatial have popped up and on an on and on.
There's tons of user contributed geocontent out there.
And I hate buzz words as well as anybody, but GeoWeb seems to have caught
on as the catch-all for all of this information that's out there.
We have -- we're now crawling all of that content and shoving it into our big
spacial database so all of the features that I'm about to show you don't just work
on collections, although it still says collections in our UI, but we're really operating
on the superset of all that data.
So here's my stats. Right now in this one I was hustling around trying to get an
update on this one this morning. We just did some reprocessing of the whole
database and we have a lot of dupes in it and we just fixed a bug that's going to
knock out a lot of the dupes.
I heard that there's up to 24 million entities in the database right now but there's a
lot of dupes so the real number is probably at about 18. So i put 20. It sounded
a little bit more impressive. But more important than how many we have right
now is the growth trajectory that we're on. And that's really killer. And that's
500,000 new entities per week, and that's from across general Web crawl that's
finding KML files for us as well as our own collections.
And this third bullet's important, too. This isn't a U.S. only feature, although our
site only supports four countries, and you can only really -- well, that's our claim
is that we support four markets as sad as that sounds at this point in the game.
But people aren't waiting for us to support their country, they're contributing
everywhere. I'd like to get all of this data into a database that I could do ad hoc
queries on so I've been curious, maybe one of you will help solve this problem for
me.
What is the biggest area, the biggest square area on earth right now that's not
covered by any content from our GeoWeb database? And I bet it's a really small
area because it's amazing to go into explore mode and see just how much
content is out there.
So what's new. These are the features that we just added. I have a screen shot
of each one, or almost all of them. I think the only one I didn't grab a screen shot
of was import, export. So we've got neighborhood subscribe which lets you
zoom into an area and pluck an RSS feed for that area, throw it into your RSS
reader, it's actually a Geo-RSS feed and stay up to date as we crawl the Web
and throw stuff into this big user contributed content database your RSS reader
will be continually updated with new information.
We've had import for a while. We've just added export. And now that we have
both import and export you can do simple conversion. So if you have a GPX file
and you wish it was a KML file you can actually do that on our site right now.
MapCruncher support, everybody here loves MapCruncher, right? It's one of my
favorite features -- it's one of my favorite things that came out of Microsoft
Research. We now support it in Live Maps directly so that you can create your
crunched layer and display it in Live Maps directly and share it with people so
they don't have to find your mashup or come across your layer in another way.
If you don't know what Cruncher is, you'll see a screen shot in a minute. This
one was very important. As we start to get 20 million, 30 million entities, we have
to give people really good tools for sorting and filtering on all that information.
And lastly we've got the API, which I'll show you at the end.
So here's our subscribe feature. I've zoomed into a really tight view of my
neighborhood. I live a little bit that way, so I like to know what's going on in that
area. So I zoomed in, turned on Explorer, I clicked subscribe to collections in this
map. I grabbed my RSS feed you can see on my life.com page, now I can see
what's going on. That's actually a different feed, but for illustration purposes
that's my West Seattle feed actually, but you get the idea.
Here's Cruncher. So I started with a bitmap, just a -- I think it was a JPEG image
of the seating chart at Quest Field. I ran it through MapCruncher, which is a
really fun tool to use. Actually I always have a good time placing my tie points in
there.
It kicks out a bunch of tiles. You have to upload your tiles to a publically
accessible Web server, and as your tiles were created it kicked out a little
manifest file, a little XML file, and now you can go to Live Maps and to a
collection add that XML file, and that tells us everything we need to know to draw
that layer. So people are using this to bring customarial imagery in. We had a
partner, image patch that just flew over the site of the Beijing Olympics and sent
us a nice chunk of aerial imagery, and it got crunched up and it looks beautiful
laid into Live Maps.
And I talked about the importance of filtering on information when there's so
much information in the system. You want to see the good stuff, so our static
rank has improved quite a bit, so we're always by default bubbling the better stuff
up towards the top. But sometimes you don't want to see the better stuff,
sometimes you have more specific needs. So we've got this filtering panel now,
and you can choose a tag to filter on. So I've zoomed in in Amsterdam, and I
said I only want to see entries that are tagged Wikipedia. And as of yesterday
we loaded all the Wikipedia entries that are georeferences, about 160,000 of
those. And that's a lot of fun. Turn on the Wikipedia layer or filter by the
Wikipedia tag and just cruise around the Earth with that on. It's really awesome.
You could say well I just want to see 3D buildings or I just want to see
MapCruncher layers, or I just want to see items with photos.
And here's the secret API is my last slide, or I think I have one that goes with this.
You don't have to write this down because the slides are available. And it's
basically a simple rest API, and there's two ways that you can query this
information.
We've got text queries up here and almost identical syntax, but basically what
differentiates these two queries is this one goes against the Monarch search
index, and this goes against our Vessy (phonetic) index. But they basically
return GRRS feeds of points that come from this database. And here you can
see a simple mashup that I built that uses that, and there's the code up there,
and you can see it in practice. I'm just panning around the map, and I'm filtering.
I think in this case my term was Wikipedia. Yeah. Because that's a Wikipedia
entry over there. And with just that little bit of code you can now visualize
information from the database on a map.
And that's it. Any questions? I think I got two minutes. Two minutes. Two
minutes worth of questions.
>>: Did you see (inaudible)?
>> Steve Lombardi: Oh, yeah, no, it's secret in that it's undocumented. And
thanks for pointing that out. We haven't officially supported API coming out in the
fall as part of our new platform services, API collections. You'll be able to read
and write collections actually, it's going to be super cool, so if you're building an
application as long as you implement passport or live ID on your site you're going
to be able to read and write collections into our data store which is really cool.
So this API that we have right now back here, it's unsupported. We actually use
it. These are the URLs that we're using when you do a GRRS subscribe, for
instance. So it's not like these are untested, they're just subject to change and
yadaya. They're fun to play with right now, they won't below up anything.
Any others? All right. That's about it.
>> Evelyne Viegas: Thanks, Steve.
(Applause)
>> Evelyne Viegas: Okay. Before we go on to Drew, I have some bad news and
good news. Bad news is Drew is not the last speaker in the session, as it says.
But the good news is that our next speaker promises a very exciting fun demo.
So Drew Steedly is a scientist in Live Lab. He's going to talk to us about
modeling 3D buildings.
>> Drew Steedly: Okay. I'm Drew Steedly, I'm a scientist in LiveWebs. I've
been working on Photosynth for the fast two years, the synther side of things,
and one of the side projects that came out of this was Photosynth generates a
point cloud and allows you to jump from image to image but it doesn't actually
generate a full 3D model, like what you see in VE. It would be fantastic if we
actually had full-on models inside of VE. So we brought in Sidipta Sinha
(phonetic), who is an intern from UNC, as an intern this past summer, and Rick
and I worked with him on this modeling project to see what we could do with a
little bit of user interaction and a Photosynth.
So the goal, if we have a full 3D model and we put that inside a Photosynth and
we project the images on to that, the transitions from image to image are going to
look a lot more beautiful, they're going to be -- every part of the scene is going to
be stabilized instead of just whatever plane or proxy is being used now.
And also if we can generate 3D models that are textured from Photosynth, well,
we could geolocate them and upload them to VE and add user-generated
models, high resolution textured models to VE, which would also be very cool.
So this is my house, and so this is part of the emphasis for why we would want to
upload these models to VE because it's on this road but it's not captured.
Here's a birds-eye image overlaid on top of it, and you can see my backyard
here, and this is the playhouse that I built. But it's still fairly low resolution. You
can't see a whole lot of detail.
But these are the photos that I can take in my backyard and add these. VE is not
going to be able to drive around and take street-side images and see into my
backyard and model this. I don't know that I want them to, but if I want to add
this in here, I can take this photos and build a Photosynth out of this fairly easily.
But the output of the Photosynth is just a point cloud and the camera position,
you know, aside from the images, I mean this is what you get, right, this is not
going to look pretty in VE, and I don't think they want us uploading these. But
these are the original images that I took, and so the first thing that we did is we
added a little bit of processing to the synther. So in addition to just extracting
features, point features and matching them between images, we also start
extracting lines and vanishing points.
So in each image we extract a set of three vanishing directions that are
orthogonal to each other, and this gives us three directions and 3D space. We
clustered these over all the images, and this gives us the three major axes of the
model. So you can see them here as blue, green, and red. And we use that to
actually further refine the bundling of the camera positions.
And in addition, there are some of these vanishing directions which are not on
the major coordinate axis, and so these can be added in by the user by just
drawing two lines that specifies the vanishing direction. It goes through all the
other images, clusters them together and adds them into the data set. So here
we added in two more vanishing directions.
So from this, you can start tracing on top of the image. So when you're drawing,
it's snapping to these vanishing directions, so this is our rectangle drawing tool.
And you can see that it's snapping on these vanishing directions.
And when I do this, I'm lassoing some points this were reconstructed in the
scene. So when I let go of this, I've told the systems that there are two vanishing
directions that I care about, which tells me the normal of the plane that I'm
drawing. So the system knows the normal the plane, the only thing it doesn't
know is the depth of the plane. But because we have a point cloud underlying
this, we can -- we only have to estimate this one parameter, which is the depth of
this plane, and we can just do a robust fit to the point cloud to get this plane in
there. And so you can see in black here are the points that were considered
inliers to this plane fit. So this is a 1D estimation problem. It's actually something
that we can do which is fairly reliable.
So I'm just going to spend a little bit of time just showing off, you know, how quick
you can model something here and, you know, how little texture you actually
need to be able to get some models in here, right. So there's not a whole lot of
texture on this facade, but there's enough points and we've -- the user has told
the system what the plane normal should be, and so it's not actually that hard of
a fitting problem.
So again fairly quickly you get a pretty tight model. You can also see that all of
the images are being projected on to this
model at the same time, and so you can see the blending artifacts right here
where actually multiple images are being projected on to this.
The reason that you have trouble seeing this blending here is it's a fairly planer
object and you don't see many artifacts. But if we go over here, there are some
pillars that are in front of this where you'll see the pillars are projected from
different images on to this plane in the background.
The actual portions of the front of the playhouse that are in focus are at the
correct depth.
So this is the part of it that requires a little bit of computer vision. There are a lot
of other things that you can do that don't require any computer vision. Such as
just selecting two edges and creating a plane out of that. But still having a
Photosynth and being able to project the images on to it gives you feedback
about whether or not you have the geometry right. So we can see that the edge
of this is wrong, so I can just go and correct it and have it line up with the images.
And so similarly, there's a gap between these two. So I can just generate a weld
between these two to tie them together, to make it a little more watertight. And
then let me go ahead and just put in the floor here.
And so pretty quickly you can get a rough model of what it's supposed to look
like. Another trick that we do is we take advantage of occlusion. So here there
are all these points on the front of the playhouse here that are occluding it and -but we've already modelled this, so we know that these points won't be part of
this plane that I'm clicking on, and so it snaps to the one in the back here. And
so that's -- it doesn't cause a problem.
We also have features to allow you to automatically weld, so if you're drawing it
will snap, it will snap the existing geometry which really helps to avoid having to
do the inference problem, so here now I'm snapping, snapping to the existing
wall and it just snaps into place.
So really quickly, five minutes you can generate a model of this. But now, as you
can see, there's all these texturing artifacts, so what we really want is a textured
3D model out. But you can see all of the multiple images projected on to this
don't look particularly pretty.
So we also did some work on graphic cut texturing. So I'm just going to play a
video to show the texturing because it's a little cumbersome to do it real-time.
This is just putting each one of the images on to the model.
Here this -- so this problem is a lot like the mosaicing problem. If you take a
panorama they can run through images, you want to stitch them into a mosaic,
you choose seams between the images and blend across the seams to create
one big seamless image. Well, that's exactly the same thing that we have here.
We have a bunch of images that see this one plane, so we project them all on to
it, run a graph cut to two seams between the images, blend across the seams,
and that gives us a texture map for everything that we saw on this wall here. And
here you can see the label image. These are the labels for the images that the
texture was taken from.
And here again is the final model with the texture map. So these are the texture
maps that we get out, and you can see that some of the -- all of this texture is
projected back on to the wall, and you get multiple copies of these chairs and
that kind of stuff but in general it looks a lot better than just blending it.
We also added a feature to allow a little bit of user interaction for when the graph
cut is not quite perfect. And so here you can see there's three images here, you
can see there are trees in front of the face here and all of the images. And so the
graph cut does a good job of getting some of it out but leaves these two trees in
here. A little user interaction to say, no, I don't want this tree, I don't want these
pixels or pixels like this from any other image to be in to the texture map and you
can correct the model with just a few strokes.
So here is another model that we generated. This took about 25 minutes of user
interaction to model. And there's a ton of detail in here. There are a lot of things
such as the front here with these pillars that a stereo algorithm's just not going to
get right and requires manual interaction.
All of the trees have been removed from the texture maps and the bushes and
that was just done with a little bit of user interaction. I guess I'm out of time. But
one other trick that we use in the graph cut is that if we have points in the scene
like trees that would project on to the texture we use those as automatic negative
brush strokes so if we have some reconstructive points in the point cloud that
correspond to trees, we use those to say don't take these pixels in the texture
map from this area. So. That's it.
(Applause)
>> Evelyne Viegas: Questions?
>>: (Inaudible) I suppose you only put a rectangle on some of the points and
planes. Can it extend out to planes to where it's not seen in the image maybe?
>> Drew Steedly: Yes. So one of the goals of this is to make it seamlessly
integrated with a regular 3D modeler. So if you only see a few of the points,
you've specified the normal. The only thing that you have to do is you have to hit
one point, and it's going to snap to that point. If you hit a certain number of
points, it's going to try to robustly fit a consensus of them. And then just like any
other CAD modeling tool, you could just extent the plane to cover any area that
you want. And that's actually what we do a lot of the times and allows us to get
areas of -- model areas of the scene that are only seen by one image. So you
lasso -- you know, you can draw a rectangle and maybe only points are in a
certain quadrant of the image or of the plane and maybe you can't see the entire
plane in a single image, but it will snap to the right depth, and then you can go
into 3D editing tool and extend the boundary of the plane.
>> Evelyne Viegas: Thank you, Drew.
(Applause)
>> Evelyne Viegas: The last speaker for the day is Jason Christie from MSN UK.
He's a lead producer, and hopefully he'll tell us what that means. He's going to
talk about life space vision.
>> Jason Christie: Really short. I'm from MSN UK. I'm a producer. So I don't
really deal with academic stuff, so it's an absolute privilege to be here, and some
of the stuff I've seen has been mind-blowing. I feel quite humble to get to show
you a little mashup that I put together, having found a feed on a blog belonging to
Rob Dolan in Windows Live Spaces.
There's nothing up there, so I might need your help. (Laughter) So what's
happened is it's gone over there.
Okay. I'll crack on. I think I've solved this. Thanks. So I start this up here, and
this RES, that was a reply from my colleague in London, Rick Stalk, who is an
evangelist and he was the one person who went with this feed and sort mumbled
and fumbled and said what I want to do is I want to rip off total vision, that's what
I want to do, and I'm a (inaudible) to feed, and so he kindly for the purpose of
they said you guys who will understand this, this is what he did. A little bit of
Java script and a proxy in order to get us to a Twitter Vision type of experience.
What I wanted to ask, can anybody raise their hands, has anybody heard of
Twitter Vision? So that's quite remarkable. Anybody heard of Flickr Vision?
Okay. Spin TV? And Wikipedia Vision? Okay. So it's interesting. Anyone
heard of Rickrolling? Anybody been Rickrolled? (Laughter) Okay. I have gone
one here. I don't want to Rickroll the whole audience. But what those are,
they're content genres, and I deal with content. Our game is page views. MSN
UK is the biggest portal. Windows Live Spaces is the world's greatest social
network by terms of active users. There's, you know, every Hot Mail user has a
Windows Live Space, so it's the biggest.
The reason I ask those about content types is because this is a content genre,
and it's maybe going to fit on the screen there. You see down there it's parsing
the feed and if any luck it will start rolling. So this is a content genre along lines
of Rickrolling, blogging, Twitter Vision and so on. This is the pop of content
genres.
And what you're seeing as was on the bullet points, these are near lifetime
updates to the Windows Live Space network from around the world, and it uses a
publically available feed and a couple of little search parameters. Really simple
stuff where everybody in this room who has any sort of computer science
background. So it's a mashup and that is it. I'll leave it with you.
(Applause)
>> Evelyne Viegas: That was very short and sweet. All right. Any questions?
>>: People who are twittering right now are just showing (inaudible) is that
right?
>> Jason Christie: That's right. These people have recently updated their
Windows Live Space. So if you visit that space, you'll see the person in
Venezuela just updated.
>> Evelyne Viegas: Any other questions? Okay. Thank you, Jason.
>> Jason Christie: Thanks.
(Applause)