>> Arjmand Samuel: Welcome everyone to App [inaudible] Fest. Tonight’s talk is about OData,
and we are very lucky to have Mark here. He's a senior program manager here at Microsoft.
He's going to talk to us about OData. He’s also going to give some great demos. And then this
talk will be recorded, and you’ll be able to view it again later if you missed something. Don’t
worry about, hey, I didn't get that one step that he did because it will all be available on Resnet.
So, take it away.
>> Mark Stafford: All right. Am I on? Can people hear me? Test, test. Please let me know if
you can’t hear me. And I will warn you in advance that I’ve had like four cups of coffee today
and I'm now drinking caffeinated water. So if I move too fast, please slow me down. We are
here to talk about OData. OData is a protocol for exposing data over the Internet. So it's not
directly related to any apps but you need data in your app most likely. And OData is a really
great way to get data into your applications. So we're going to talk about both how to expose
OData services as well as how to consume them with inside of Win-8 apps and other platforms
as well. I did steal a deck that I used to present on OData at user groups, so I may skip over
some the content depending on how well we’re engaging.
So when I go visit these user groups, I posit a problem and that problem is that data APIs
frequently create more problems than solutions. The reason that they create more problems
than solutions is because there's so much idiosyncrasy around data APIs. Every data API that’s
out there has kind of its own look and feel. And the really big data APIs like Facebook and
Twitter, they're called fiat APIs, and other APIs, they may follow restful principles, but by and
large there is very little consistency across the board. So we're going to talk about one possible
solution to this problem.
So I do have some assumptions going in. Number one, you know that data APIs exist. So
hopefully you know that you can access Twitter’s data through an API. Hopefully you know that
you can access Facebook's data through an API. Most major websites that are out there today
are actually exposing some sort of an API and then building their website actually on top of that
API to dog food the API and make sure that it's consumable by the general populace. What you
guys probably want to do is be able to bring that data into your Win-8 application. Number
two, you should have basic familiarity with HTTP, XML, and JSON. For the purposes of this
room I think just knowing that they exist is probably enough. And you should know that REST
exists. You may not need to know what it means right now because we’re going to go through
some of that.
So, like I said, this is a really dense deck. Just so I know how fast to move through some of this
material, this is just kind of a quick poll. How many people know what the Web is? Hopefully
everybody. Good. Thank you. How many people are comfortable with HTTP and XML and
JSON? Good. At least half the room. How many people have a basic understanding of the term
REST? Wow. Awesome. And how many people have actually created a data API? Okay.
Fewer. Fine. And the last one, like there are some people who are REST wonks and I say that
lovingly. People who are REST wonks, we can have a conversation, it's just that this forum is
probably not the place for us to have the conversation. There are people who really know REST
inside in and out who love OData, and there are people who really know REST inside and out
that hate OData. And they tend to be polarized one way or the other. I'd love to have that
conversation with you. If you fall into that category, please catch me afterwards.
So is there really a problem? Let's start with the most basic question is, if so, what is it? So I
want to run through a really quick quiz here. With Facebook, who thinks they know what fields
they'll get when you read a post? When you ask for a post back? Take a guess if you don't
know.
>>: The username and the post.
>> Mark Stafford: Okay. Yeah. And probably a time and maybe some other stuff. But we don't
really know, right, until we actually go look at the documentation and this is part of what we
get back. There’s actually a lot more fields attached to a post. So on Facebook, how do you
unlike a post? Who thinks they know how to unlike a post?
>>: You go into interface and you click the unlike button.
>> Mark Stafford: I saw that one coming. Okay. All right. So you have to send an HTTP delete,
which that part makes sense, but you actually have to send it to post ID slash likes because a
like doesn’t have its own ID. So you’ve got to kind of know that the likes URL exists. And one or
two more, so with Twitter, how do I get a list of the users following me? I'll just kind of skip
through and say, you send a get to friendships, which that part makes sense, right? Like the
actual request process makes sense. I don't know whether this is a bug in their docs or what
but like this person that came back, Taylor Singletary, there's no connection. So I don't know
whether this is a bug in their docs, a bug in their API, or if that's expected behavior. And last
but not least, how do I unfollow somebody? You actually send a post to friendship slash
destroy, which I thought was kind of funny, and then you get back this big response that has all
this data in it that I don't know why you need that back. I guess I need to delete that.
So if you don't know how to use these APIs, how do you learn how to use these APIs? What the
first thing you do?
>>: Go get [inaudible].
>> Mark Stafford: Go get, okay. You're jumping ahead.
>>: Search the web for examples of uses.
>> Mark Stafford: Okay. Yeah. Documentation, right? So theoretically, they use docs and
there’s Google up docs, and there’s Twitter docs, and there’s Facebook docs, and Get
[inaudible] docs, and Amazon docs, and Yelp[phonetic] docs, and so on and so forth, right?
There's many, many documentation sites out there. So is there a problem? Yes. Developers
don't like and don't use documentation. They immediately try to go somewhere else and my
friend over here was jumping the gun. So does OData magically cure the problem? No, but it
helps. And we're going to see how.
So if developers don't go look at the documentation, what do they actually do? Well, they go
pick up an SDK or they pick up one of the console apps that helps them to understand what the
payloads look like. So in theory, they may use an SDK if it exists, so here's a number of different
SDKs that are out there, the developer consoles are actually really cool. The developer consoles
help you to understand exactly what the request should look like so that you can formulate it
properly if you're using WebClient or something like that. And this is actually really helpful.
But, it’s still a problem, right? Like, it would take you forever to go through this and then to
actually replicate that in code and these APIs change all the time.
So SDKs are great, but here's the problem: if you’re Twitter, if you’re Facebook, if you're
somebody else you can probably get away with it. If you're a little service, you probably can't.
So what can you do? And let's look at the counter example to what we just looked at which is
the confusion of existing APIs. So if data APIs were like LINQ instead, I think you guys will do
much better on this quiz. So with linked objects, what extension method or what type of clause
do you use to get statuses from the last 24 hours?
>>: [inaudible].
>> Mark Stafford: Where? Exactly. Context dot status dot where. This is what the thing might
look like. So how do I do it with LINQ to SQL? Same way. How do I get the 50 most recent
statuses with linked objects? What two clauses do I need to use? The three most recent
statuses. Take and order by, right? And the same way with LINQ to SQL, right? So the point
here is that you guys know that stuff because there's consistency in the APIs. So uniformity
decreases the time necessary to learn an API. And it also decreases the amount of time
necessary to design an API. So for those few people who have actually been through the
process of creating a Web API, it takes a long time to figure out what your URL structure should
look like, and what HTTP methods you want to support, and what you're going to do if
somebody's behind a firewall that blocks get and put requests, right? There's this thing called
post tunneling that you can use. But if you're designing your API from scratch, you need to go
through all that stuff. Are you going to support JSON? Are you going to support XML? What do
the payloads look like? Let's skip the quote. It’s a good quote though. Yeah.
>>: [inaudible], what's JSON?
>> Mark Stafford: We will see it in just a minute. But JSON is just the JavaScript object
notation. It’s JavaScript serialization format that is much more pleasant to look at for humans
than XML. It's also much more compact on the wire, typically. Okay. So we've established
hopefully that data APIs should have some level of uniformity, but you might be asking now,
well, isn't that what REST is for? And so we kind of need to run quickly through what REST is.
So when Roy Fielding, the guy who came up with the term REST, talks about REST, this is what
he means: resources should be uniquely addressable, application state on client, server is kept
separate, resources should be connected to each other via links, right, hypermedia, and this is a
principle he really, really pushes on hard. In fact, he says there should be one entry point to the
service and everything else inside the service should be discoverable from that one entry point.
Services should have a uniform interface. This is the one that nobody really understands, but
typically, it's interpreted to mean the HTTP methods. So you consistently use get, you
consistently use delete, you consistently use put, that type of stuff. And messages should be
self-descriptive. We can largely kind of discard this slide because most of this stuff was
asserted in the thesis and what it looks like in the real world is a little bit different.
So what most people mean when they say REST is that simplicity is important and that
resources are identifiable. And that's kind of in a nutshell what most people mean when they
say REST. A lot of people want to try to attach the concept of JSON, but those things have yet
to gain enough momentum to be widely accepted. So kind of slowing down for a second in
recapping, we've established there's lots of different Web APIs out there, most of them follow
their own little format and that own little format isn't all that useful to your application because
you either have to use an SDK to consume it and the SDK has a proprietary thing attached to it,
are you going to use that SDK in a particular way, or you've got to go bare bones and use
WebClient and to attach the service that way.
So why isn’t REST enough? So if we think that REST is out there for building services and REST is
this thing that people point to is the epitome of a good service, why isn't it enough? Well, most
importantly, the conventions are insufficiently prescriptive. REST does not say a lot of things. It
says a few things and what it says it says well, but there are a lot of things that it doesn't say. It
doesn't talk about URL structure. So how do you want your URLs to look? It doesn't talk about
things like search; it doesn't talk about things like how to use methods consistently, right? So
how do you create a new resource? REST doesn't actually define that. And services that are
just restful services require significant documentation. So most of those services that we
pointed out earlier, the Facebook API and Google APIs, most of those claim to be restful
services, some people will argue about whether or not they’re restful, but they still vary pretty
dramatically. And, also importantly we talked about this briefly, but we don't account for
things like search, ordering, paging, things like that.
All right. Skipping forward, let's get to some interesting stuff. So what's good about
conventions? Conventions enable you to move faster, right? So we talked about LINQ, we
talked about how you guys immediately know how to use some LINQ providers because you're
familiar with LINQ. So you know how to use LINQ to SQL, you know how to use LINQ to EF, you
know how to use LINQ to objects, you know how to use NHibernate’s LINQ provider, you know
how to use Lucene’s LINQ provider just because you're familiar with LINQ overall. You can also
enable generic tooling to produce and consume services. So our vector, the data movement
vector, actually dynamically generates OData services. And similarly, on the client side, we can
generate code that makes it easier to consume OData services. We'll see some of why in just a
little bit here.
You can build frameworks that allow you to codify interactions. What this means is that we
know how to delete something, right? So we can build a framework that enables you to delete
certain objects. Oh. This last point is actually really interesting. It's just that I'm going to make
you wait to get to it. So conventions enable a more compact payload. We can actually see a 90
percent savings over our XML format just because of some of the things that we can guess on
the wire. And I'll come back to this in a little bit.
So how is OData better than REST? Let’s skip forward a little bit. We have some conventions,
right? An overview of the conventions. So there are three main categories of conventions in an
OData service. There are some well-known endpoints, so one place where you enter the
service, and theoretically for that hypothetical hypermedia client that can navigate the entire
service from the entry point, this will work. The data model is exposed in most or all OData
services at an endpoint called dollar metadata. This end point helps you to understand what
the model looks like, right? And that XML that's exposed at that endpoint allows you to
generate code the client can more easily use. And we provide a consistent model for batching
requests and creating change sets. So all OData services do batching in the same way. We
have a really powerful set of URL conventions for filtering and paging and we have payload
conventions that you'll see.
So a service document. This is what the service document looks like. This is the entry point,
right? So we talked about the three well-known endpoints. This is one of the three. This is the
entry point. This tells you what you have available. So in this particular case, this is a service
called Soupon and we have two sets of things available to us, we’ve got Soupons and users and
the URLs unsurprisingly are Soupons and users for those they could be different, that's why
there are separate fields there. We'll see some of this in just a second here, too in the browser.
Dollar metadata goes out and it actually describes the entire service. So this is a snippet of
what it looks like. The actual service kind of goes on for some extent, but this is describing an
entity called Soupon. Soupon is identified by its ID, which is an int[phonetic] 32 and it’s also got
these other fields attached to it.
So let's take a quick break there and actually jump over to Visual Studio, maybe, and see what
this kind of means in the real world. So if I open up, start in IE actually. Okay. So this is service
that you can't see very well. This is a service that I've just recently started building. It's called
Trip In, it's going to be a combination of the Trip It and Pinterest so that you can flag trips that
you like and so on and so forth. But right now it's mainly got some airport information in it that
we’re going to use in a different demo down the road. But we can see that this is the main
entry point to the service. And this is the atom representation for it. If we want to see the
JSON representation for it, it’s actually a little bit easier to see in Chrome, don't tell anybody I'm
using Chrome. The main reason that it's easier to see in Chrome is because IE doesn't natively
handle the JSON data type so it asks you what you want to do with it.
And we’ll blow this one up too. So just by appending dollar format equals JSON to the URL, we
are able to get back the JSON format instead of the XML format. It's actually the AtomPub
format, but we can also go out and see the dollar metadata for it which describes the entire
service and all the different entities in it. So far, so good? Everybody with me? We have any
questions so far?
>>: So is the data only hierarchal?
>> Mark Stafford: What do you mean?
>>: If I want to have a directed graph representation of something that's [inaudible] can I use
this? Will that>> Mark Stafford: So the question was: is the data only hierarchal? Can I have a directed
graph? So let me maybe just explain a little bit about some of the modeling concepts. We’ll get
to some of this later too, but there are two primary types. There's an entity type and a complex
type. And they're used to create structure for the service overall. Right now, in OW-3, entity
types can inherit from other entity types, so you’ve got type inheritance and so on and so forth.
Complex types will be able to do that in OW-4, which is coming up at the end of this year. And
they can have relationships to each other, so entity types especially have a much stronger way
to be related to other entities in a means that allows you to bring back only some of the entities
unless you specifically ask for the related entities. So if you’ve got a graph of people who have
friends, right, if you just ask for people, all you're going to get back are people. You have to
specifically ask for people and their friends or friends of their friends in order to get back a
more complex query. Make sense?
>>: [inaudible]?
>> Mark Stafford: No. Well, so entity types and navigation properties are available right now
and have been for all the versions of OData. It’s, the new thing in V-4 is inheritance and
complex types. And hopefully some of that will make a little bit more sense as we make more
progress through the slide deck. Okay. Let me get back to the slide deck hopefully.
So looking at some of the URL conventions, there are three main parts to an OData URL, and
this will be useful as we start to look at some of the other parts of OData services. So there's a
service route, and with WCF data services, which is one of Microsoft's main stacks for building
OData services, that will typically end with something called MyService.SBC. That doesn't have
to be that way but out-of-the-box it’s kind of the way that it is. Then there will be some
component that is a path to some place in the service. So we talked about the fact that when
you go to the service route you see a set of sets, right? You see all the different things that you
can access. So in this particular service we can access a set called Employees, and then, in this
case, we are casting it to Managers and we're doing some additional stuff with it.
And then finally, there's the query string portion of the URL. So three main parts to the URL
overall. The URL path, so that center part, right? We're not going to talk about the service
route. All that is is kind of where we start. The URL path has a few different parts to it. It can
have an entity set, which is always the entry point or frequently the entry point; it can have key
segments that are attached to entity sets, that helps us to identify a particular user, and so we
can actually replicate this in the browser in just a second, let me get to this slide; and then we
can just have like navigation segments. In this particular case, we're navigating, we are starting
at the entity set for users where finding the particular user with ID Jun Hao[phonetic], we are
going to the person that referred him, we’re going to the person that referred that person,
we're going to their billing address and we're finally going to the city. So let's actually take a
look at that in the browser. If we can find it. Let me, where is my mouse? There it is.
So here we are at the route of the service and if we go to users and we get a list of all the
different users, here, let's do this in JSON format just for kicks. Dollar format equals JSON.
Okay. So we’ve got all the different users that are available. There's only three in this
particular case. We’re going to find a particular user with ID Jun Hao[phonetic], and then we
can append additional path segments to navigate. So we can see who referred at Jun
Hao[phonetic], and it happens to be Hao Chen. We can see who referred Hao Chen, that was
Polara Akerman[phonetic]. We can already kind of see the address here. There's a particular
reason for that that we’ll get to later. Billing address, and I'm going to just going to jump
straight to city. So we can jump all the way down to a particular place in the service. So we've
got the service route, which gets you into the OData service overall, we've got the path that
gets you through the service to a particular place, and then we have the query string, and
there's one other thing that we should call out here. So you can actually do typecasts. We
talked about entity type inheritance in the service and this is a typecast to a particular type.
So the query string allows you to do other special things. You can do filtering, you can do
paging, you can do ordering, you can do all sorts of different stuff with the query string, and
we're going to spend a little bit of time on some of the URL conventions that are out there. So
with dollar filter, you can do a few different things. You can do a very simple property match,
so you can say, hey, I want all the users where the display name is Hao Chen and there's a
series of logical operators, you can do equals, not equals, greater than, greater than or equal,
less than, less than or equal, etc., etc., etc. And more importantly, you can do and, or, and not,
which allow you to combine different types of filter statements together.
There's arithmetic operators, string operators, and other functions as well. You can do any all,
this is new in OData V-3, but we can do things like hey, show me all the Soupons where one of
the ingredients contains the word garlic. And we can create compound conditions, and they
can be arbitrarily complex literally up the limits of what your service is willing to accept or the
limits of a URL length overall, which is pretty big. Questions on filter before I move on?
Order by is very similar to filter. We can do simple ordering where we’re ordering by the
display name. We can do more advanced ordering, in this particular case, we are selecting
users but we are ordering by the user ID of the person that referred them and since the user ID
of the person that referred them isn’t actually coming back, like the payload as you get it would
look nonsensical. It wouldn't appear that there was any ordering to do it, but it actually is
ordered. And then multiple field ordering of course, and more complex types of things we can
do to sending, ordering, etc., etc. No surprises, right? Things that you would expect to be able
to do with an API.
You can do client-side paging with top and skip. These work exactly the way you expect them
to, the way that they do in SQL Server. There's also another type of paging. So client-side
paging is all well and good, but as a service, if you're returning multiple billion rows worth of
data, you can’t trust the client to use top and skip appropriately, and so you need to implement
server driven paging. Server driven paging is the same as client-side paging, it's just that it uses
a nextlink as kind of this opaque token for how to get the next page worth of results. So you
can enforce paging and good OData clients are expected to understand what a next page link
looks like.
Select and expand. Select is column filtering. It is not productions. So you can't do math in
there. You can’t assign names to things. That may eventually come in, but right now it's
definitely just column filtering. You can also include related entities. So we talked about kind
of this graph concept. You don't want to get back your entire database by default in a query.
And so by default, a query is scoped to a particular entity type or something like that. Expand
allows you to bring back related entities as well. And then eventually, near the end of this year
we'll actually be able to do much more advanced queries where we can actually ask for users
and their orders where the total is greater than 1000 and the top 10 most recently shipped
orders where the total is greater than 1000. We can't do that quite yet.
>>: [inaudible]?
>> Mark Stafford: Yeah.
>>: Can you take one step back when you say that we will be, you mean the standard will>> Mark Stafford: The protocol.
>>: The protocol will define it.
>> Mark Stafford: Yeah. In fact, yes. So that's a great distinction to make. So there's the
protocol, which is the OData protocol, and then there's our implementations and stacks that
expose OData services. We have WCF data services and Web API. We'll talk more about those
in just a little bit. So both of those are built on a core set of libraries called OData Lib. And
OData Lib actually already internally has support for a lot of the new expands syntax, it's just
that it's not a valid part of V-3. So we actually, we implemented the core libraries late enough
that it was easier to implement the V-4 syntax and translate the V-3 syntax to the V-4 syntax for
expand. If that makes sense.
And most importantly, you can combine these URL conventions. So you can come up with a
complex query like this where you want the third and fourth Soupon where any of the
ingredients is garlic and etc., etc., etc. Now if we take a big step back here and kind of go back
to the beginning of the presentation where we postulated that data APIs create headaches and
you don't know how to use them, what we solved here is the ability for people to walk up to
any OData service and intuitively know how to use it just because they know what OData is like,
right? So as soon as you learn OData on service A, you can approach service B and easily start
using it. You know how to do filtering, you know how to do ordering, you don't have to learn all
sorts of new conventions for creating new entities, or deleting new entities, or deleting existing
entities.
We’ll move through the payload conventions really quickly and get to the demos, hopefully. So
success in error codes are standardized. The overall formats are standardized. We'll look at
them in just a second here. Nuances, right? Like what does PATCH mean? Should we use
PATCH? Should we use Merge? What does a null payload look like? So the JSON format is the
one that we recommend. The JSON format is super friendly to JavaScript, especially the new
JSON format. There is a legacy JSON format that, if anybody knows about, they should just
keep their mouths shut. The new format is 90 percent more compact than AtomPub. If you G
zip it, it's still 30 percent more compact and it looks like a custom API. So if you were to go out
and design a custom API from scratch you'd wind up with something looking like this. We look
at a few of those payloads, right? So if we kind of come back to Chrome maybe, so it looks and
feels very natural. Maybe not that one, but we go back a few pages, right? It looks very natural
in how you actually would use the service. There's very little OData gunk to it. Looks like this.
There is also an Atom format. The Atom format was one of the original formats in OData. It's
still around because there are Atom parsers in almost every single language out there. So even
if we don't have a rich OData client in a particular language, if there's an Atom parser then you
can do a lot of stuff natively with that response. Looks ugly, but there it is.
And kind of what else can we do with OData services? So kind of getting out there on the edge.
So, great. We can define basic entities, we can do CRUD, what else can we do? So there's
these things called actions and functions that allow you to do nonCRUD operations. Again, this
is something that's not standardized in REST. There’s no conventions for it, but OData has
conventions for how to do actions and functions. And the only difference between actions and
functions is one is side-effecting and one is not. We have defined conventions for PATCH and
Deep Insert. Deep Insert means if I want to create a customer and an order and put them in a
database together, how do I do that? Something called containment. If you don't know what
containment is, maybe you know what the aggregate route pattern is. If you don't know what
either of those are then probably this feature doesn't matter to you.
And vocabularies are extensibility mechanism. So vocabularies allow you to put custom
information into any kind of a payload, whether that's dollar metadata or whether that's the
actual payload. And this is something that [inaudible] uses to extend error messages, right?
They provide more detailed error information with annotations and other people are using
annotations to apply permissions to feeds. So if you don't get a property back like, is it because
the property didn't exist? Is it because the property was empty? Or is it because you didn't
have read permissions to the property? So you can use annotations to deal with those types of
scenarios.
All right. Let's see if there's anything else important. Let’s fly through this and we’ll get into
demos. So back to this slide. Let's say that you buy the basic argument, right? You need some
sort of a standard for your data API. What else should you know before you jump on the OData
bandwagon? I'm a very realistic developer. I've only been at Microsoft for two years. And I
remember what it was like to be not at Microsoft. So I want to be realistic with OData.
So OData isn't always the best fit, right? If you've got some really unusual service that's really,
really heavily RPC oriented, right? So you are rebooting VMs or something like that, then
maybe OData wouldn't be the best fit. But OData is a great fit for flexible data oriented APIs in
general. It's not good for bulk data transfer, at least it didn't used to be, with the new JSON
format, it's a lot better but still, probably if the only thing that you're doing is transferring a
table of 1 billion rows from SQL Server to Excel, TDS is a much better protocol for doing that
sort of thing. And APIs that are dramatically different in shape from the back end, it's
sometimes hard to do in OData. This has gotten much easier with the advent of Web API, WCF
data services was a little bit more tied to the back end.
OData doesn't support all queries. So we don't currently support aggregation. There is an
extension to OData V-4 that supports aggregation, but you can't ask for customers that have at
least two orders right now. You can't support filtering expansions right now. You get back all
the orders if you ask for the orders. Again, this is something that's fixed in OData V-4. And we
have limited support for graph style queries. You can ask for a friend of a friend, but there’s
limitations around it. The limitation is mainly kind of what we saw briefly here. So if I go to
users June How[phonetic] I can say format equals JSON and [inaudible] expand equals referred
by and that will bring back the person that referred June How[phonetic]. Here's referred by
right here. But then I got to ask for, comma referred by slash referred by in order to get the
person that referred them, right? There’s not a simple way to say hey, I want all the people
that referred this particular person out to four levels. Again, this is something that's fixed inside
of OData V-4. We made a lot of progress in OData V-4.
So it is not a panacea. And then there's some people out there, this is a post by a guy called
Myths Demos Bellow[phonetic]. If you're familiar with him he wrote Service Stack, smart,
smart guy, hates OData. Absolutely hates OData. And I think his criticism is unwarranted, but I
mean again, being realistic, if you go out there and research OData, you're going to find both
criticisms and praises. On the plus side, OData is becoming a standard, and it's not just
Microsoft participating in the standard. We are going through OASIS ultimately ISO. The ETA
for OASIS is November of this year. And we are in a standardization body with IBM, with CA,
with SAP and many other big companies as well. Skip, skip, skip. Microsoft is investing heavily>>: Hold on. Go back to>> Mark Stafford: Yeah. This?
>>: When you talked about [inaudible].
>> Mark Stafford: Oh, yes. Here. So Microsoft clearly cares about OData, right? OData started
here and we have huge inroads already inside of Microsoft. So if you're trying to consume
Microsoft data, SharePoint already expresses exposes OData, many parts of Asher expose
OData. Dynamics exposes OData. Their OData service is interesting because they've got such a
flexible back end and you can define your own types and everything like that. So their OData
service is pretty interesting. And this is just the tip of the iceberg. It says that because there
are parts of Microsoft that I can’t talk about that are doing OData services and it's not available
publicly.
>>: Can you speak to SQLs [inaudible] anything like that?
>> Mark Stafford: So SQL doesn't expose OData natively and I don't think they're all that
interested in exposing OData natively mainly because it's so easy to bootstrap an OData service
on top of SQL server. And there’s two really, really easy ways to do that. Entity framework
works really, really well with WCF data services currently, and it's super easy to put together an
entity framework model that just kind of reflects on your database, generate all your types, and
then expose that through an OData service. It literally takes all of about 15, 16 lines worth of
code. Now if you actually want to go through and secure things properly and stuff then you
need more code, but it's so easy to do it on top of SQL server already that I don't think they're
interested in exposing a native OData endpoint. Does that make sense?
>>: Yeah. I'll buy that.
>> Mark Stafford: Okay. All right. SAP is super heavily invested in OData. If I had to name one
other company that was most involved in the OASIS TC, the technical committee, it’s by far SAP.
They’ve contributed as much or more to the OData spec as we have.
Okay. So the rest of what I have to show are demos. We can take a break. We've been going
for 40 minutes and it's been really, really dense content. I get that. Are you guys ready for a
break? Do you want a break? You want to just jump in and do demos?
>>: I have a question. You mentioned the entity framework. Where's the authentication come
in or being able to put together>> Mark Stafford: That's a great question.
>>: There’s public data that everyone can see everything and looks it looks fine so far, but
where is, I mean the first thing you want to do on your sprocks[phonetic] is [inaudible] imager
ID, the token>> Mark Stafford: Yeah. I totally get it. So OData, the protocol doesn't specify what you're
supposed to use to authenticate the service. And it does that for a very specific reason,
primarily because authentication changes so fast, right, and there's so many other standards
out there for authentication. So we expect that the OData protocol will work nicely with
authentication, but most authentication is driven by headers on HTTP request realistically,
right? So if you’ve got an O off flow then you go out and you get your token and then you stick
that token onto one of your headers, right? If you’ve got forums authentications, very similar
except for it's in a cookie, etc. etc. etc. So what we expect you to do is we expect you to
authenticate the service however you want to and then use appropriate authorization within
the service and hopefully our frameworks make it easy for you to do that. So inside of either
Web API or WCF data services you can just ask for the HTTP context dot identity or whatever
request dot identity, whatever the property that actually has the identity on it, and if you use
Windows Off or something like that then it’s just sitting there and you have access to the
identity.
We have examples of people using this in production for both Web API services as well as WCF
data services, so we know that it works. We've even got examples of people who are doing like
row level authentication, right? They're restricting the rows that come back based upon a
user's identity. And there are specific ways to do that with the WCF data services and Web API
as well. Anything else? Other questions before we look at demos?
All right. Well, let's start with some client-side demos because we've been doing a lot of really
serious stuff. Maybe. If I can get out of PowerPoint. Sorry. I'm working with a really low
resolution monitor here so I’m doing the best but I cannot my computer. It doesn't like when I
change my display mode. All right. So let's start here.
So this is a particular solution that I put together that’s just called the Soupon. You probably
can’t see this solution Explorer over here very well. I wish I'd put, I'm going to cause something
to crash if I do too much. Here, let's try to do magnifier. And I've no idea how to use magnifier
actually. Oh, look at that.
So we have a window store application in here and we have a service as well. What we're going
to look at first is kind of the window store application. So you can see that there's a service
reference here that we just created by doing right-click, add service reference. So I can actually
go take that service reference and just choose to update it, but let's actually look at the
application running and then we’ll go through the process of adding the service reference and
kind of explain what’s happening throughout the service.
So let's see if we can get out of magnifier and I'm just going to debug this application and it's
amazing, it's wonderful, we can navigate all these Soupons. Soupon is my brilliant idea how to
retire from Microsoft. It's going to be a clone of Groupon except we sell hot soup and bread
apparently. So we can navigate back and forth full kind of Windows 8 interactivity. You can
even do search. I want to do chowder. I want to look in the Soupon. And I've got the ability to
go out and search various types of Soupons as well. No big surprise, especially if you've already
written Windows 8application. To do that we just use the service reference, which we’ll look at
in just a second, and then literally all I did was kind of modify the sample data context, if I can
find that in here, somewhere, somewhere. It is impossible to see. I am not able to read what's
on my solution Explorer.
Thank you. No, it’s not under [inaudible]. It's under a separate folder called sample data
source but, well here. Maybe I can just control T to it. So this is a class that comes stock with
your Windows 8application out-of-the-box; it’s part of the product template, right? So literally
all I did was to do some modifications in line here. I used the types that were generated with
the add service reference and then I just made some very minor modifications to this. I made
as few modifications as possible because I was posting a blog post about this and I wanted to
make the minimum set of modifications. So one thing that I did need to do at the time was I
had to add some extension methods at the bottom here for doing things asynchronously. We
have since released a set of portable libraries that will work inside of Windows store apps,
Windows phone apps, etc., etc., etc. And those already have the a-synch methods built into
them.
So let's just add a service reference to a different solution. So if I just go add service reference,
it’s literally just kind of, the same way they got any existing soap service reference etc., but I’m
going to add a service reference OData dot, I'm sorry services dot OData dot org slash OData
slash OData dot service, I click go, get the service here, I can actually see the entity types
underneath it, advertisements, categories, products, suppliers, and then I can assign a
namespace and it will actually create the context for me and Code Gen everything. Similarly,
there's an application called LINQPad, so if I go, it’s in my downloads, LINQPad actually uses our
client internally and it will also do Code Gen. So if I choose to add a connection here, I choose
WCF data services, next, services dot OData dot org slash OData slash OData dot SVC, and we’ll
tell it to use the JSON format because that one's better. We’ll say okay. It goes out; it
generates proxies for all the things, and then I can do things like advertisements dot take two
and it will go out and get the top two advertisements.
Now our client actually ships internally with a full-fledged LINQ provider. So it actually will take
your LINQ query tree and translated to an OData URL. And if you want to go see what that
OData URL looks like, if you haven't gone out and read the URI conventions, then you can
actually go look at that request log here and it will tell you exactly what query was necessary in
order to get those things. So this is the same client that's used in all of our dot net stacks. You
can use it inside of the Windows store apps, you can use it inside of Windows phone apps, etc.
So far so good, any questions?
This makes it really easy to consume data from SharePoint. Talking about the service side
specifically, let's see how easy it is to actually create a couple services. So this is the first
solution that I'm going to create. I'm actually going to create the two solutions together
because it used to be that it took my machine a long time to create the Web API solution, I'm
not sure why, but we're just going to start with empty Web application for WCF data services
and so we just called it the WCF DS dot demo. I will expand the code, or make the code bigger
when we actually see the code. So we'll say okay. I'm going to start the other one creating as
well just because. So this will be our Web API demo. This one I'm using the ASP.net MVC four
template I've just called it Web API dot demo. I'm just going to let that go and we’ll kind of go
back to the other one. We'll just pick up the Web API template here.
Okay. So WCF data services, we have the ability to actually go out and choose add other new
item. And we want a WCF data service which is down at the bottom. We'll even call it WCF
data service 1 dot SVC. Make this a little bit bigger. All right. And literally, to get going with
WCF data services, all you have to do is give it a context. So we are going to say, we'll call it
actually we’ll call it user’s context. That did not work right. We’ll cheat to create the context.
And all this context needs to have is I query able stuff. So public I query able user, users get set,
and we'll create the user class, public ent[phonetic] ID, get set, and we’ll get public string
name, get set. And we’ll just create a couple of these to actually pick up.
Again, no magic happening here, we're just creating sample data and we’ll say name, all right.
Oh boy. I missed something. Okay. So all we've done is create a little bit of sample data to give
us something to look at and now we need to actually look at but this data service configuration
is allowing us to do. So we need to be able to expose our entity set, so we can do that with
config dot set dot entity set access rule and in this particular case, since it’s a demo, I'm just
going to say all entity sets, and I can change this from all read to a variety of different things,
right? I’ve got all, all read, all write, read multiple, read single, etc. But for right now we’ll just
leave it all read. This is not what you would want to use for normal authentication. It's just
kind of a basic entity set level access. And you can set your max protocol version. It's probably
not important for most people. If we [inaudible] this and wait, and wait, and wait. Wow.
>>: [inaudible] user?
>> Mark Stafford: Yeah, it does. Oh, it's loading symbols, I'm sorry. I was working with symbols
earlier today so we're going to just have to wait a minute here. But it is still working. It will
load the symbols eventually. So there are three main types of providers with WCF data
services. There is the reflection provider, which is what we just used. We can go users, and we
can say we want to ask for that in the JSON format, format equals JSON, we can ask for just the
top one, etc., etc. So WCF data services kind of comes with three providers out-of-the-box. It
comes with a reflection provider, which is what we just used. The reflection provider is good
for exposing arbitrary data sources. So if your data source is coming from Lucene or something
like that, as long as you have a LINQ provider to hand the query off to, then you're good to go
and you can use the reflection provider.
The entity framework provider makes it really, really easy to use entity framework. To use that
instead of the reflection provider, you literally just kind of pass a DB context here instead of this
custom context that I created. And then since DB set is, inherits from I query able it all works
properly.
And then there's a custom provider, if you have something more advanced that you need to do
that you can use to expose any one of a number of things and some of our more advanced
partners use the custom provider in order to expose things that don't necessarily have type
definitions, right? So I had to create the user type in order to use their reflection provider
because it's reflecting over the types. If you don't have types or you're dealing with unschematized data or something like that, then the custom provider is where you want to go.
Any questions on WCF data services?
>>: [inaudible]?
>> Mark Stafford: It releases approximately every six weeks although we're in a more
moratorium right now. But over the past couple of years we've released every six weeks or so.
Web API is the other way to create OData services. This one is a little bit more involved to get
started but it is important to see. So let's start with this. We'll just call this users controller. So
were going to do three things with Web API. We need to create a controller that can handle
the request, we need to set up the routing convention, and we need to build a model that Web
API can use for dollar metadata.
So the first thing we're going to do is create the controller because we happen to be sitting
there on the right page. So we’re going to say entity set controller, I'm going to try and create
this effect if I can see it. That's not what I wanted. Let's just create it manually. Okay. So we’ll
just kind create the same shape we did before. And now our entity set controller is happy. We
need to get rid of some of the stuff that came in the prefab class that was sitting here. And the
way that the entity set controller works is it uses the generic types that you pass to it in order
to define method signatures. So we can go override, and we can override get, if I can find it, so
we want to override get and in this particular case we need to return something, actually I'm
just, yeah. And we'll just [inaudible] up. So we're not actually going to give this one any data
and we need to do this as query able.
Okay. So literally all we've done in our entity set controller is say, hey look you can get back all
the entities. If you want to be able to get back entities by key and stuff like that then there are
other methods that you need to override. So we say override, get entity by key, return
something here. In this particular case we’re returning all. So again, three steps. We need to
implement an entity set controller, we've done that, we need to go set up our route, so we do
that inside of Web API config, so we don't need this one anymore, whoops, I guess let's make
that bigger because I can't see it either, so we don't need the default HTTP route anymore
because we’re not using just regular Web API, we’re just using the OData part of the Web API.
So we're going to say map OData route and we need to give it a route name. Let’s say OData.
We need to give it a route prefix. We’ll just say API and we need to give it a model.
So we said that we had to build that model somehow. Web API has a class built-in that makes
it much easier to build your model overall. So we can do OData convention model builder and
this guy just has some nice fluent API on top of it to be able to do things like model builder dot
entity set user, we’ll call it users, oh boy, now I have to get the right one. Okay. So all we've
done here is we've built a model, we've said that there’s one entity set of type user called users
and here we’ll call users two just for kicks, and now we need to give that model to the OData
route, so we’re going to say model builder dot get EDM model. All right. And, if we cross our
fingers, we should be able to load this guy up and I’m actually just going to do this one in
Chrome immediately. I'm just loading symbols anyway. And we need to go to API because that
was our route prefix. It's thinking.
While that’s loading, are there any questions about Web API? You guys are familiar with
ASP.net? How many people have used ASP.net in production systems? Okay. A lot of people.
Have you used Web API? Yeah. A couple. So here's our service. And we can ask for users,
although we won't get anything back because we didn't have any data in there. So 404. So all
of that said, we've got two stacks, WCF data services and Web API, we’ve got clients for .net
desktop, we got clients for Silverlight, we have clients for Windows store Windows phone, and
we’ve got clients build into Excel, so I'm going to try one more demo here and this is the one
that may fail spectacularly, we'll see.
So one of the things that you can do with OData services is spatial queries. And so, if I can find
my spatial query here, maybe it's in, so I'm going to local host. So I will have to use the
magnifier here to help you understand what's going on. But what we've actually done is we've
asked for the five most remote airports from where we are right now. So one of the types that
OData supports is a spatial type, and hopefully you can see that we've chosen to order by the
distance from here, I'm sorry from here, which is negative 122 comma 47 to the particular
location where the airport is located. And because we’ve sorted by that descending, it picks up
the farthest airports from where we are physically. If we were to go modify that and say, so
right now we've got airports that are showing up in Mauritius or however you pronounce that,
whereas if I remove the descending, then of course I get the close airports, right? Boeing Field,
and SeaTac and everything else. So I can go find the five closest airports. It doesn't like
JavaScript somewhere.
So if we want to consume that inside of Excel we can actually use that data to print out a pretty
map. How many people have used Power Query? How many people know that Power Query
exists? We’ve got two people that know that Power Query exists. All right. Good. This is a
cool application. So you can just go out to Microsoft's download site and download Power
Query. It is super cool. I worked [inaudible] Power Query tab in Excel here and I'm going to say
from other sources and I’ve got a big prominent OData feed. Now I've also got, on the data tab,
I can go from other sources and I can go to from OData feed. These are two different providers.
So the one that's sitting here has actually been in Excel since Power Pivot 2010, it came in with
Power Pivot 2010, and it supports a lot of stuff. It doesn't support the same shaping that we
are going to see inside of Power Query. So I would recommend if you have the ability, to use
Power Query rather than this provider for OData. If we go over to Power Query then we can
say from the sources and we’re going this guy and we’ll go pick up that URL that we were just
looking at for airports. And let's get our magnifier running again. Okay.
So this is what we have available to us. So we can name the query if we want to and you can
actually share the queries out and so on and so forth. Over on this side, if we get all the way
over to this side, you'll see that there's little steps that shows up and we’ll come back and look
at that in just a minute here. But what we want to do is we want to shape this data a little bit.
It's pretty good, but it's not exactly what we want. We can see that we’ve got this, the three
letter code and four letter code, whatever those are, we've got the name of the airport, we’ve
got this thing called the city that says record, and we've got these locations here that are kind
of in a format that aren't very easy to use right now. So let’s see if we can fix this up some. So
we're going to ask for some specific information back from city. And as expand that out, now
we've literally gone and done an expand on the OData query and it's bringing back that data in
addition. Oh boy. Come over.
If we go to location, we want to fix this up because the data doesn't look like we want it to, so
we're going to right-click and we're going to replace values and we’re going to replace SRAD
equals, so we're just going to trim off the start date and the end date and then we’re going to
split that column up. So we’re going to say four, three, two, six, semicolon, point, [inaudible].
So we’re going to replace that leading text with nothing. We’re going to replace the trailing
text with nothing. Then we’re going to split the column on space and then we’re going to
rename those longitude, latitude, and we’ll bring in this one just for kicks. Country.
All right. And so now if I go back over to my steps we can see that it's actually put together the
steps for me. Wow. And I can see that I can go in and modify any of those steps. The really
cool part about this is it allows the entire query pipeline to be refreshable. So we started from
an OData feed that was already fairly smart and if we've chosen to filter these rows by
something like the country equals United States or something like that then that would all
actually be built into the OData query itself, and then we can kind of build on top of that with
things that OData doesn't necessarily supports like renaming columns and flattening things out.
Okay. So we are going to try and click okay on this. We're done, rather. And it's going to dump
that data into Excel and we can just use it there if we want to. If we wanted to go into Power
Pivot we can go in and use it there as well. But we are going to do is we’re actually going to
insert a map, and I think here's where we’ll kill the magnifier and that gives us a globe that we
can use. So we need to identify some information for actually mapping this stuff. So I'm going
to pick up longitude and latitude. And you might not be able to see it, but it's actually mapped
itself because it's got the right names. So I’m going to pick country as well and just so you can
see when it doesn't map right, I'll pick city dot region, and it doesn't know what to map it to so
it maps it to city because it started with city.
So I'm actually going to pick state province for that. I'm going to click map it and the first thing
that we’ll do, so out-of-the-box we kind get these placements for all the different airports. The
first thing that we're going to do is kind of take a look at the name and we’ll put that
underneath height. Right now it doesn't do anything interesting other than draw a lot of red
bars. But if we actually change our mapping type to country it will start to draw bars that are
varying in height based upon the number of airports in a particular country. And so if we go
over to Europe, we can figure out which country has the most airports based on the size of the
bar. And it happens to be France, right? And if there was any question about that then we can
actually go click on France and see 112 and Germany is 73 it looks like. We can change the
view to a heat map, etc., etc.
We can also, this is one of my favorite things to do even though it doesn't look all that good, we
can also see the actual locations of all the different airports around the world and kind of what
their named and stuff like that. And you can actually search this control for certain locations.
So you can punch in city names or other things like that. So if you want to find the closest
airport to a particular city then you can go out and find that stuff.
So the moral of the story is that OData has really rich support across the board both on the
server side and on the client side. Things that we didn't talk about as far as ecosystem are
concerned are JavaScript. So we produce a JavaScript library as well. It's called data JS and
there's another company called, I'm going to draw a blank on their name, but they produce a
stack called J data that's really very cool for working with OData services as well as a number of
other data types. We are working on a C plus plus client. There are existing clients for Java in
IOS, although we're working on improving the client for Java and the one for IOS needs
attention. But yeah, it's a very robust ecosystem. It's a great way to bring data into your
existing apps. So, with that, I’ve been talking for an hour and 15 minutes almost nonstop. I'm
sure it’s really overwhelming. What questions do we have at this point? Or have I completely
overwhelmed you? Yeah.
>>: What is Google's stance, Twitter’s stance, and Facebook stance for providing OData feeds
of their [inaudible]?
>> Mark Stafford: That's a great question. So Mike Edmondson[phonetic], the guy who wrote
Restful Web Services awhile back, or at least was involved with it and is writing the upcoming
Restful Web APIs, calls those fiat Web APIs. I kind of mentioned that briefly at the start. And
the reason that he calls them that is that because they have the ability to decree that their API
is going to look a particular way, right? So they do what's best for them, and that's
understandable. I’m not trying to criticize that, I'm just pointing out that they're doing what's
best for them. More importantly, their Web APIs change all the time. Facebook actually has a
90 day breaking change policy. So they'll guarantee that your application will work for the next
90 days, but not necessarily beyond that. So they’ll give you a 90 day warning from when
they're going to change their API and they may not keep the legacy API around.
>>: Do the types of changes [inaudible] APIs also [inaudible] client?
>> Mark Stafford: That's a good question. Sorry. I should probably be repeating the questions
for the recording. The original question was what about Google and Twitter and what’s their
stance towards OData? Would similar changes break an OData client? Well, it depends on the
OData client and whether you did Code Gen and things like that. So if you have a strongly
typed OData service and you've done Code Gen, our client will tend to handle that in certain
cases as long as you've done the right things in our client. There are probably clients who
wouldn't handle it as gracefully. But OData, like one of the features of it that we didn't
necessarily cover are these things called open types which means that you don't even have to
declare what the properties are on your type. And so especially as we move into the V-4
timeframe, it’s expected that clients will be receptive of things that aren't necessarily described
inside of dollar metadata. But that said, the OData protocol has really strong guidelines around
how to version your services and what to do and things like that. So typically what we would
expect is if you're really doing breaking changes inside your service you should version your
service and bring up a different time point. There's no magic solution. Yeah.
>>: So you mentioned [inaudible] search for OData bunch of crappy links. So where are the
good links like that blog you mentioned or good starting points?
>> Mark Stafford: Okay. So we should probably, do we have, I assume that there's some sort
of a DL for this group. And I can follow up with links there. There is a DL?
>>: Everything that you need should be available [inaudible].
>> Mark Stafford: I will try to communicate this via e-mail as well, but we have a blog on MSDN
called Astoria Team. Currently it's still kind of a legacy name for what WCF data services was,
and then OData.org is the first place that we always point people. We're trying to put more
attention into that site right now, but it has all sorts of information about the protocol itself.
It's not focused on Microsoft’s implementations because it’s designed to be the protocol’s
website as opposed to Microsoft's implementations. So for WCF data services stuff you go to
our team blog which is blogs.MSDN.com [inaudible] Astoria Team. ASP.net maintains their own
blog. So that's where you would go for the OData stuff for them. And then OData.org for
everything else. Yeah.
>>: If you have multiple [inaudible] sources can [inaudible]?
>> Mark Stafford: So there is not a way to formulate a single web query that will work across
multiple services just because the queries are rooted at a particular place. But certainly you can
you do one of two things. Number one, you can mash up the data in a client like we just saw,
and Data Explorer, I'm sorry. It used to be called Data Explorer. Power Query makes it really
easy to mash up those data sources or you can do it actually at a service side. So you can have
a service that’s set up like a proxy that matches up those two services together for your clients.
Yeah.
>>: Remember when I was talking to you about it makes sense for a single server not to spark
16 lines of code?
>> Mark Stafford: Uh huh.
>>: That's the type of scenario right there where you want to match up two different data
sources or otherwise you are trying to do data analytics that like want to support.
>> Mark Stafford: So SQL Server has other things to work about too, right, like they would need
to think about plumbing their permissions through that OData service and how do they do that
stuff. So as far as like SQL Server is concerned it's probably not a 16 line solution.
>>: And I’m talking about data consumption, not necessarily [inaudible].
>> Mark Stafford: Sure. Like I'm talking strictly permissions, right? Like SQL service position is
not going to be, we'll make it really easy for you to bring up an OData head that bypasses all the
other restrictions on your data, right? That’s just not something that they're going to do. If
they were going to expose an OData service and make it a turnkey, you right click here and
expose OData head over your database, then they would need to do more work in order to
plumb their permissions that are set in the database all the way through to that OData service.
>>: Sorry. I'm talking about, and I know I’m coming from [inaudible] different space, but when
you're writing an ETL that is taking in data from a whole bunch of different sources and writing
it to database tables, one of those sources you’d like to be able to use [inaudible] the same way
you just used it in Excel, that’s the types of, we wouldn’t be worried about all that security’s
plumbing. I just want to pull the data in.
>> Mark Stafford: So if you're using SSIS, I know at one point in time they were working on an
OData source. I'm not sure what the status of that is. But I can check on it and follow up. And
certainly it would not be all that difficult to kind of write your own. I mean I've written my
share of SSIS components. So I can imagine that it probably wouldn't be more than a day or
two's worth of effort even to make it generic, right?
>>: Thank you very much Mark.
>> Mark Stafford: All right. Well, my alias is here. My alias is [inaudible]. I'm sure it's attached
to something, but I'll try to get it back up on the slides. If you do have other questions or your
team is working on services, please reach out and I'm happy to help you diagnose whether or
not OData is the right tool for you. That was the demo failure that I was trying to prevent by
the way. I literally have, I'll show you. Well, I can’t see it. I have a file on my desktop called
demo fail. Maybe you see it right here. Just in case my demo crashed. I had it backed up in a
place where I could show [inaudible] demos. My alias is [inaudible]. I’ll see if I can bring it up
here while people are leaving. If you have any other questions please feel free to reach out.
>>: Thank you very much. If anyone would like to go to the break yard room it’s outside and to
the left. Feel free to grab an extra sandwich; I think they’ve set some chips and stuff up.