>> Ratul Mahajan: Thank you. Good morning, and thanks everybody for coming.
Today we have another rising star from the U Dub’s rising star factory this year
[laughter], Colin Dixon. Some of you may actually know him. He spent a summer with
us and then some change. He is going to tell us about how we should look at networks as
programmable platforms rather than as computers, individual computers as
programmable platforms.
>> Colin Dixon: That's close enough, [laughter]. So thanks for the wonderful
introduction, Ratul. So as he said, today I'm going to talk about the network as a
platform and it is basically how we should think about programming our networks as a
single object rather than a bunch of single proprietary little objects that do their own thing
on their own. And so I am going to start off by PowerPoint disagreeing with me. Let's
see, there we go. Let's start off talking about by way of an analogy about cell phones.
And so if you think back sort of way back to five years ago before the iPhone came out,
before the android came out, before finally the Windows phone came out, remember your
phone was basically a, you know, basically something you bought. It made phone calls
may have a calendar that you never used because you had to press the stupid little button
seven times in order to get each letter. But it was sort of the phone that you got and you
kept for the entire time that you had that phone. It wasn't going to change. It wasn't
going to add new features.
And then all of a sudden things started to really shift. It's hard to even remember what it
was like before this but I'm sure all of you have one of the devices at the bottom. Either
you know an android or Windows phone, probably at least half Windows phones, I'm
thinking, [laughter] in your pocket, which is just a radically different kind of object. I
actually do have my Windows phone just so I'm off the hook for making jokes. And you
know it sort of radically shifted. These phones are not things that have only the
functionality that you buy them with. They have applications that you can install and
run, and they have sort of brought about a radical way of innovation. And so what
changed? These old phones were actually pretty decent. They had not terrible processors
in them; they had data connections that almost worked, and the functionality wasn't really
missing. What was missing was the extensibility. It was very hard for anybody except
for a very small subset of people to add functionality to them. Nobody could write apps
for them. Nobody could do that.
And so what happened was phones became a platform. A couple of companies got
together and decided they were going to try to make it easy to build applications, easy to
get those applications out to lots of different people. And this sponsored a huge wave of
innovation and in the process sort of made cell phones what are likely the dominant
computing platform for the next 10, 15 years. And so the question that I am going to
hope to answer in this talk is can we do something similar for networks? And in
particular I am going to talk about two different kinds of networks. First I am going to
talk about networks of electronic devices in homes. So things like, you know, light
switches doorbells cameras televisions, that kind of thing. And second about networks of
computers and enterprises.
So can we turn these into sort of an extensible programmable platform we can enable the
same degree of innovation. And this raises a bunch of challenges because networks aren't
phones. They have a lot of very different properties that we are going to have to do deal
with. First of all you have to coordinate many different devices to carry out some kind of
centralized task. And to do this we are going to treat the network as a distributed system
and we’re going to try to use standard distributed system techniques in order to cause
them to basically carry out a given task together without having to worry about doing
different things in different places.
Second we need to overcome the limited extensibility of a bunch of these different
devices. If you are talking about trying to write a program that runs on a light switch,
you are not going to run a program on a light switch. So you are going to need some way
of thinking like you could program them, even though you can't. So we are going to
proxy these devices have sort of general purpose hardware, in order to give us a uniform
platform that looks kind of the same everywhere. And lastly we need simple extractions
and APIs. We actually want this to be a platform; we need to make it easy to write
applications and get them to run on top of it. And so we are going to build two different
sets of APIs, one in the home and one in enterprise networks and just sort of prove that
you can build these APIs.
So the key idea of this talk is that we are going to provide an extensible software platform
on top of, you know, heterogeneous network hardware. We're going to be able to build
new and interesting applications easily. This includes all of the things that you may
expect from a platform. Here people probably have a better idea of what a platform
means than they do in most of the rest of the places where I give this talk, but basically
you need an abstraction layer, so that we can think about particular devices, some kind of
security model, a programming model of what you can think of, standard things of what
you think in your programming against when you write an application.
And I am going to talk about this in the context of two systems that I have built. The first
one is work done here with some of the people that are in this room called HomeOS.
And its goal is to make it easy to build cross device applications in the home, so to make
it easy to write things to coordinate different devices all across your home and also make
it easier for home users to manage that technology, to easily bring in innovative
applications and new functionality into their home. And the second thing I am going to
talk about is ETTM, which is focused on how do you make it easier to run and provide
the interesting services that you would like from your enterprise network and doing so by
running these network control on and hosts on your network rather than running a
proprietary middle boxes in the middle.
And so the basic outline of my talk, I finished up basically the introduction and
motivation. I am going to talk about HomeOS first and then into the middle, and then I
will move on to related work and I will conclude and I'll take your questions. Also if you
have a really burning question you want to ask me in the middle, you can interrupt me
and I will decide whether I like you or not.
So I'm going to start off with HomeOS just by bringing up how much technology has
made its way into modern homes. So there is a huge amount of technology. You
probably recognize all of the devices up here, or almost all of them. You probably have
something like half of them in your house, from, you know, locks to smart phones to
laptops to cameras to televisions to access points to gaming systems, tablets. There's just
this huge amount of technology that has migrated its way into homes and increasingly it's
connected or programmable in some way. At least you should theoretically be able to
reach out and tell it to do things or ask what it's currently doing. But we find that despite
that home users are struggling with technology in their homes, and for two key reasons.
First reason is there is a management nightmare around each device with its own
interface, its own semantics, its own particular details that you need to configure. I need
to make sure you understand that and do it for each device. And since, you know, that
maybe the number of devices in your home is doubling every two years, you are not
going to have twice as much time to manage your home every two years.
Imagine you are putting a Wi-Fi chip in every thing John, so every, so you have a number
of Wi-Fi chips and devices in your home doubling every two years. That seems like it's
been happening for the past five or six years. And the second problem is integration
hurdles, which is that, well even if you have these devices and they could theoretically
talk to each other, in practice getting different systems to talk to each other is very
difficult. So you might get lucky in your Xbox and your Windows computer and your
Zune to all play nice together, but if you want that to talk to your ATC security systems
that way you can use your Windows phone to view your cameras, that's unlikely to work
very well.
So even when you do get these islands of connectivity, getting them to talk to each other
is very difficult. And so it makes building things that you might kind of like hard. And
so in this example let's say you have a digitally controlled door lock which you can buy
from Schlage today, a doorbell that's connected and will send a signal when it's been
rung, and a security system with a camera. It would be really nice if I was going to be
out and about during the day and I knew somebody was going to come over and visit that
I could have it set up where when they came over and rang the doorbell it would take a
picture of who was at the door, send it to my phone and I would say oh it's Ratul; I was
expecting Ratul to come over today. And it would actually ask me if I wanted to unlock
the door for him. I would say yes. It would reach back and unlock the door and Ratul
would be able to come in. And this is something I just described in 60 seconds. But I'm
sure that you can imagine why it is that it would be difficult to build. I am going to step
through it…
>>: [inaudible] [laughter]
>> Colin Dixon: So imagine it's somebody that you liked better than Ratul. [laughter]
>>: It was your mom; she just showed up to visit
>> Colin Dixon: Yeah. But the idea is, you should be able to, imagine it's the babysitter
that is coming over and you just want to let them in and you don't want to give them a
key, so you might imagine, this is actually…
>>: [inaudible] don't have a babysitter. [laughter] [inaudible]
>> Colin Dixon: Anyway. So you might imagine that with all this technology coming
into homes and the fact that people seem to have trouble exposing it, that the developers
would help and build the applications that I just went through and described. But
developers aren't, you don't know a whole lot of people that are writing apps for the
home. It's not like you hear about this huge home system development thing; you hear
about iPhone apps; you hear about Windows apps; you hear about that. And so there are
some good reasons for that. Basically today you have to write an application on top of
the hardware. There is almost no support between you and the hardware. And this
involves dealing with a whole set of problems in terms heterogernating the home. So
first you have to worry about topology. You have to worry about how am I actually
connecting to each individual device? Is it on Wi-Fi? Is it on 3G? Is it on ZigBee or ZWave, and even different subnets within the home? And even if you deal with that, you
then deal with the actual diversity of devices. So there are 100 different kinds of cameras
in the world, and each one with a slightly different interface, or a radically different
interface. So you need to actually be able to speak the same language as the device. And
then if you've done that, you still, you may not be able to assume that you are the only
application running in the home; you probably can't.
So what do you do if the security application is set to say all of the doors and windows
should stay closed and locked when there aren't people in the home? But the climate
application says, it's a nice day outside we can save energy by opening the windows?
You need some way to manage these conflicts in these coordination issues. And
implementing that co-application seems pretty difficult. And then even if you've done all
that right, different users want to do different things with their homes. So you have to
somehow express all of the preferences of any user using your application would like in
some reasonable way so that they can do it. And then when you're done with all of that
you can write the 12 lines of code that you actually wanted to write in the beginning that
constitutes the actual application logic of when the doorbell rings, take a picture, send it
to the phone, ask them to unlock the door and if they say yes, actually unlock it. And
that's, you know, and then you can finally write those 12 lines of code.
And then to add insult to injury all of these stuff is application agnostic. There is nothing
in particular about the unlocking the door application that required that I have all of these
pieces. It is just a property of finding the right application in the home. And so you can
sort of start to get an idea of why developers aren't helping. Developers have this huge
mountain of work that they have to do for themselves in order to get anything to work.
So you might say well what about device spenders? Those are the people actually
making the devices. A bunch of the problems I just talked about or having to do with the
devices, you know the particular devices. They already know what it is. They probably
already have drivers for them. And the vendors don't seem to be helping either. They
seem to be building only islands of sort of usefulness. So Schlage makes a door lock;
they make an iPhone app, that's it. They don't try to communicate with other devices, try
to use other people's things, and when they do try to make use of other people's devices
and to integrate they run into a sort of the same issues that I just talked about. How do
you actually coordinate with the other devices in the home? How you expose the
functionality in a way that you can make sure that different users can express their
preferences? And lastly you might think that maybe one vendor was going to take over,
but no single vendor is likely to control all the devices in your home in the near future.
And you can imagine this is going to be true ever. You probably don't want Apple or
Microsoft making your stove for instance; it's just not their area of expertise.
And so we are not the first people to note that we have this problem with this massive
number of devices and no really good way to try and get them to work together. There
have been two approaches to try to solve this problem. The first is interoperability and
interoperability even if it did happen entirety isn't sufficient. So you can imagine,
imagine that all devices could pair wise communicate with each other. This would make
it possible to go and build applications that might coordinate a couple of different devices
in order to do something useful. But it still is not going to solve the higher issues of
preferences and also, you know coordination. So imagine if you had a camera-based
entry system and a remote lock application, how are they going to both share the lock?
How are they actually going to figure out who should have access to it? Who's going to
get the final say? You still need to build these separate uses. So we make lots of use of
interoperability protocols and what we end up building, really don't hope you build a
complete system for how it is you would write applications that would run in the home.
On the other end we have monolithic systems. These are things like security systems,
entertainment systems, and they are sort of commercial home automation systems. And
these systems aren't, they're just inextensible. Unless you buy the particular hardware
they want you to buy, you end up not been able to extend the system and you offload the
set of applications that you get to run our limited by what you have. And if you want to
try to combine two of the systems together like your security system with your stereo
system, say you want to look at your security camera on your smart phone, getting these
two things tie together is usually very difficult, and it's often above the pain threshold of
what you're willing to do even if it is technically possible.
And so we are proposing an alternative approach of a home wide operating system,
where the operating system will run on top of all of the devices, logically talk to all of
these devices and bring them together into one place where they can mediate access to
those devices on behalf of applications. And furthermore if you have this central place
you can give users one place to control the technology which is in their homes; they can
actually express their preferences as to how things should work. And you can even
couple this with an application store or market place that would let you say this
application is compatible with your home versus this one isn't or other people like
applications like this. If you buy this device it will enable a bunch of new applications.
And so we think that this is a much more compelling way to try and think about solving
the problem and it's going to leverage the interoperability and the other ideas, but it's
going to end up with something that looks quite different.
So in building HomeOS, we had three main goals and these correspond to the three
stakeholders in the home ecosystem. The first is this thesis of HomeOS is that the way to
expose functionality and new features to homeowners is through applications. And so we
wanted to make it really easy for developers to build new applications for the home and
get them out to people. So the first goal was really to simplify the applications involved
to make it easy for people to write co-applications and get them into the hands of
homeowners.
Second, in order to make things easy to program we have to standardize things a little bit
and you run the risk whenever you standardize things are shooting yourself in the foot by
preventing new devices from working with what you have, meaning that the device
vendors cannot differentiate their devices from other people's, so the second goal is for
the device builders want to make it really easy to build new features into your device or
totally new devices and rapidly expose that functionality out to application developers.
And lastly none of this is going to matter if we can simplify the tasks of users managing
technology in their homes. So if you can't simplify user management of technology,
none of this is going to work. So the third thing is we really focus on trying to cater on
what users want out of their homes and make sure that the tasks that they want to carry
out are easy to do. So I am going to step through each of these goals and explain how we
satisfy them with HomeOS. And I'm going to start off with simplifying application
development. And I am going to walk through each of the difficulties that I mentioned
that the developers have in building applications in the home one at a time and build up
the pieces that have been developed to make this easy. So first to deal with topology
differences we have, we use drivers. This is nothing new but we use drivers to logically
bring the devices across the network to one location where we can program them as
though they were directly connected to a single PC.
And we are going to do with the differences between devices; we have drivers export the
functionality at a functional layer via services. And I will come back to services and little
bit on the next slide and explain what I mean there. To deal with coordination issues of
which applications get to control which devices at what times. We build an access
control area so we can make principal decisions about which accesses are allowed in
which accesses aren't and why. And lastly to deal with user preferences we give users a
management user interface in order to control the access rules in their house. So what
accesses are legal and what are not and under what circumstances. And this is how we
give users control over what applications are allowed to do in the house and to what
devices and at what times. This is our model for how it is to give users control what
happens in their homes.
So I said I would come back to services and I will, right now. Services are what users,
what programmers develop against. So services export functional descriptions of these
drivers, functional descriptions called roles. Roles are things like light switches,
televisions, displays, speakers etcetera, and you have functions on them, we call them
operations, like light switch.invoke or light switch.turn on or light switch.turn off, that
kind of thing. And application developers program against these but anybody who wants
to can make a new role. And this enables vendors to innovate and differentiate their
products. So somebody comes out with a brand-new kind of strange device they can
invent a new role for it and wrap that functionality into it for developers to program
against. Or if Sony wants to expose the cool new features of their Sony Bravia TV as
opposed to Vizio's base line TV, they can actually make that kind of decision and they
can say, here are the new extensions that enable you to make use of these cool features.
And this enables us to allow our vendors to rapidly bring these features to market and
expose them to developers at the same time as sort of letting commodity vendors actually
produce inexpensive devices and get them out there so you can have the basic
functionality.
And lastly in order to simplify user management tasks, you actually need to know what
users want to do in the homes. And so a bunch of my colleagues here at Microsoft did
field studies of modern homes in order to try to find out what people with technology in
their homes do with it and what is the extent of home automation technology. And there
was a whole bunch of takeaways from this. There was a whole CAI paper about it which
I suggest you go read if you're interested. But the three takeaways from the systems
willing perspective, in particular for access control rules which is how you give users
preference; the first one is that users want applications as security principles. And this is
coming to be true in sort of modern smart phone OSs, but it is less true in commodity
OSs today. And it is a different approach.
Second users want to talk about time and access control decisions and this is for two
reasons. First reason is they always want to talk about guests. They want to invite
people into their home and give them some permissions but not all of them, whether it's a
house guest or somebody coming to do work on the house, they want to be able to say
we're going to give you some permissions for this limited window of time and then it's
going to go away and I won't have to worry about it anymore. The second thing is for
parental controls, which is that you don't want kids playing Xbox after 11 o'clock at
night, that kind of thing. And lastly, this is the most interesting to me; users are actually
much better configuring things than they think that they are. But they are very bad at
convincing themselves that they've done it correctly. And so giving users a way to be
able to say figure out did I configure it the way I wanted to configure it, is critical for
them to actually be, have confidence in their configuration plus trust the system to do
what they wanted to do.
And so when we built our management parameters we kept these sort of three things in
mind. So the first thing which we sort of built from the ground up is reliable reverse
perspectives to build confidence. And this is the ability to ask questions like which
applications can open the front door? And actually be able to give a concrete answer to
that question, and in modern sort of operating system access control systems with
dynamic delegation, this is a very hard question to ask. Give me a list of all the people
that can open you know system .conf is going to be a difficult question to answer. And
so we have restricted our set of access control rules to being positive things under the
form user can access device using application within a certain window of time. And this
enables us to do these diverse creations quickly and easily and be able to let users build
confidence.
In addition to specifying time and access control rules, users actually have validity time
windows to enable guests, which is something we found many people wanted. And you
might imagine that requiring people to specify access control rules for every user for
every application for every device would be difficult, but we made, because of this sort of
unqued problem, but we make this pretty easy by using application manifests.
Applications specify which devices they require in addition to the devices that they can
use in any way shape or form so you can radically constrain the number of possible rules
that make sense in the system. You know your music player application isn't going to get
access to your front door locks because it doesn't make sense. So we've actually built
this. It's about 15,000 lines of C sharp that run on top of the.net CLR. Only about 2500
lines of this are in sort of a kernel which has to do with sort of mapping roles and services
back and forth. The rest of the code is in applications and drivers. We have built 11
applications, although there might be more by now. We are continuing to work on this
stuff. And sort of two example applications of the types of things that we've been
building our Music Follows the Lights, which is basically where the lights are, you play
music when the lights are off you don't play music and you can transfer it back and forth.
This is just using light as a proxy for where people are in the house. You can imagine
using motion detectors or something else as they became available. And the second thing
we built was a two factor authentication program. Which uses a spoken password and
face recognition in order to grant access to you know, open your front door or disable
parental controls or that kind of thing. And you can see our test bed here on the right. It
consists of a couple of laptops, some programmable lamps that have dimmer switches
attached to them, smart phones, a camera, a television. We now have a door opening and
closing, a sensor on a couple of other things.
And so now that we've actually built it this is at least technically a systems talk, so I need
to give you some kind of CDF of performance. So the top graph on the right here you
can see is basically a function of latency for service invocation operation, our operation
invocation which is something like, light switch turn on. And what you can see that it is
a latency of the function of the rate of operation locations per second so. And it's both for
using application domains as an isolation mechanism to isolate applications from the
kernel, and with no isolation whatsoever.
And what you can see is that with no isolation for like 10,000 operations per second on
the commodity PC with isolation is about 8000. It stays sort of somewhere under a
millisecond when you do either one of these operations sort of regardless of how fast you
do them. And so this seems pretty reasonable but this is just for one application querying
one device. And so the second graph is showing what happens to latency as you increase
the number of applications and devices. So for each number of applications we have
four, for each application we have for devices and the application queries each of those
four devices 10 times per second. So this is about 400-- this is about 40 operations per
second provocation. So with 100 applications are talking about having, you know, a
hundred applications querying 400 devices each 10 times for second about 4000
operations per second overall. And you still see it's about 5 1/2 ms when we are doing an
operation and on average and the vast majority of them fall less than 10. And so with
interactive guidelines of about 100 ms you can do, you know some working 10 or 20 of
these things serially and many, many more than that in parallel. And so we think this
works pretty well for you, small businesses and homes. If you wanted to make this work
for the Sears Tower you may have to think a little bit about how we could change things.
Do you have a question?
>>: You said that you need to write some drivers, you wrote some drivers in C sharp for
instance you, how…
>> Colin Dixon: It depends on the driver. For some of the drivers, yes, others of the
drivers we wrapped the, we wrapped existing Windows drivers in C sharp so we can
make calls to. So for instance, we used Zwave lamps, so there's a Zwave driver that is
written in native code, which we have a C sharp wrapper too. But in the case of like
Webcams we actually have a pure C sharp wrapper that goes that makes queries against
the web interface of the camera. So it depends.
>>: [inaudible] wired or wireless or ran in combination. I'm just wondering if you put
this in context it seems like querying anything thousands of times per second over
wireless, isn't going to happen anyways. I'm wondering if this is ever the model
[inaudible] were the real [inaudible]?
>> Colin Dixon: My goal with this graph was to demonstrate that we are not the
bottleneck. And I think that that's probably what this does. I think you are exactly right
which is to say that the bottleneck is going to be, you know, getting out across a wireless
link to get to a camera, or getting across a Zwave to get somewhere. So we are
suspecting a random smattering of connectivity options, kind of increasingly wireless and
increasingly less wired. And so this actually is a dummy driver. There is no actually
device involved in this top graph; it is just going out to ask it. If you actually need to go
get an image from the camera, like, you know, right now the web interface is kind of
slow so it takes almost a second in order for it to get back with an actual picture from the
camera. And that's just sort of an artifact of the particular hardware that you are running
on top of. We also don't feel like the environment in the home, if it takes a second to get
a picture from the front door, that's probably okay. You probably don't need nano second
kind of high-frequency trading style performance in the home.
>>: My other question about this is would you ever actually want to query a device 500
times per second as opposed to have an sensor [inaudible] interface?
>> Colin Dixon: We have location interface as well. This was sort of just a test, the
application boundary crossing and what the performance implications of that were. So
the more interesting evaluations of HomeOS actually are not about, not about particular
performance. Because I think performance is good enough for building most applications
that I can think of in the home, but we did both a programmability study and a usability
study. I am going to omit the usability study results from here, but if you want to talk to
me later about them I would be happy to talk to them. But the programmability study we
actually had 10 students and developers come in and try their hand at writing an
application for HomeOS.
And we asked that they had no prior experience with HomeOS; we asked them to write
one of two applications, either a custom lights for user application which asks users to
look for rooms that had cameras and lights and then see if it recognized any faces in the
camera and if it did set the lights for the appropriate level in that room, and The Music
Follows Light application that I discussed on the slide previously. And eight out of the
ten people finished this in less than two hours. And the two people that didn't finish, one
person spent their time building a GUI for about an hour before actually trying to
accomplish the task that we asked them to complete. And the second person had a
misunderstanding about the driver model for an extended period of time without asking
for any help and also spent about an hour doing nothing. And so the claim is not that,
you know, that 80% of all applications for HomeOS can be built in to hours, but rather
that, you know, reasonable developers can build interesting stuff in a very short
timeframe with no prior experience with the system. And so we think this is pretty
compelling evidence that this approach has legs in terms of making it easy for developers
to build interesting applications in the home, quickly.
In summary I really do feel like homes are worth phones were like five years ago. There
is a huge amount of technology, you know floating around in them and we haven't really
figured out how to harness it yet. I think with the right infrastructure we could really
create a similar wave of innovation where suddenly we can rapidly expose all kinds of
new functionality and have it shift the way we think about our daily lives. And we are in
the process of working with researchers at a bunch of universities in order to try to get
this around and actually having people using it, and as of about a month ago it's running
in a real person's house that we didn't ask them to run it; they actually came to us and said
they wanted to have a security system that was made cheaply out of these parts, and so
we did that. And so it's real in whatever sense that means, it is real. But I think this is the
right approach I really do think that we could create a huge difference by getting this out
there.
>>: For an analogy like assume that Windows mobile is there and that it is
programmable before [inaudible] so I am going to ask,
>> Colin Dixon: Did you use it?
>>: I used it, so what was [inaudible]
>> Colin Dixon: I can tell you what people inside Microsoft have told me a list of what
was missing in Windows mobile. I'm not; I think that what was missing was sort of a
good experience end-to-end across the system. And what I've been told about why that is
it's in part because the hardware was very varied which was radically informed, the
design of Windows 7 with a sort of specified chassis and things like that to make sure
that the performance was sufficient rather than like a nice interface and experience across
the whole system. And that that difference made, getting one set of hardware to give you
consistent and sort of expected behavior was a big part…
>>: [inaudible] where you're headed which is homes don't have that property. Just make
the argument that it doesn't have a consistent set of hardware because if they did security
companies [inaudible]
>> Colin Dixon: So I think that's true but there is a caveat which makes this problem
little easier, which is that HomeOS itself is probably going to run on a dedicated
appliance, which is either a PC or, you know, your Xbox, set up box. And so that wheel
could have the property which is that you have one consistent set of hardware that
actually provides reasonable expected behavior.
>>: The apps are going to expect some suite of devices… Somebody writes a phone
expects that there will be some sort of device…
>> Colin Dixon: And we have mechanisms to specify that expected support and actually
check for it in the home, and so it's true that, you know, if you want to make this work in
practice in the real world, you would want to come up with applications that had
relatively small numbers of requirements at first that most homes had, and I think there is
kind of a course of things like that. You can imagine the multimedia experience in your
home being something that, you know, probably the hooks for are in many homes, and
then from there moving out towards other things. You know you can buy a wireless
controlled light switch for 20 or $30 right now. That's compared two dollars for a regular
one. If you didn't have HomeOS already in your home, you probably wouldn't be willing
to pay the overhead, but if you already had something there for another reason it would
probably be worth it to you; it's only 20 bucks. I think you are absolutely right.
>>: I feel at least right now the future very few homes have a PC running 24-7. What is
your position on that? I mean even people here; any people have a PC running 24-7?
[laughter]
>> Colin Dixon: This is not a good sample. So I think that there is some validity to that.
But I feel like integrating this with some other sort of always on or always standby
appliance like an Xbox or something else, getting something which can go to sleep and
wake up on demand and is, doesn't seem like a really hard problem here. There is lots of
good work around that. I will take your questions. I should move on but…
>>: Sorry.
>> Colin Dixon: No, no, no. There's a whole second half of the talk, which will be fun.
>>: I will make it quick. You talked about a problem of access control and your access;
your principles seem to be a combination of user and application. How are you going to
do it with [inaudible] because that seems like something that how you identify the user
when the main reasons for identifying the user varies [inaudible]
>> Colin Dixon: Our imagination is that most rules will actually specify star for user and
that access will actually be granted on behalf of an application. And that for the ones
which don't we can use, we haven't really spent a bunch of time thinking about how you
authenticate users. We imagine that username and password in some instances, using
your cell phone from which you're accessing it from is useful in other instances. We are
sort of getting around that. We can imagine a bunch of things if you can get access to the
music player app inside the house, we don't really care who you are. We are just going,
so physical access is something which buys you a lot. Let's just say if the only displays
you are going to render our inside the house, just saying that music player live have
access to the speakers is probably sufficient.
>>: But that would restrict [inaudible] figure out who the user is [inaudible] different
domains in the way that they different languages in the way you describe the user is
different than the audio like song versus voice or face or something. It's hard to imagine
all of the contraptions going into a single [inaudible] I guess is what I'm saying.
>> Colin Dixon: I think that the combination of applications getting questions from
devices followed by if for whatever reason you need to do something which just the
application is asking that person to do asking you to authenticate yourself either by
pressing accept on your phone or unlocking or using your password, I think would handle
a bunch of the stuff in the home. And our, I mean yeah, and having talked and gone
through what people say about how they use their technology in their homes it leads me
to think that that would get you a long way towards there.
>>: So I found the HomeOS very similar to the traditional OS where you have to
services you have to program each access each device and application [inaudible] so what
is unique about the HomeOS from traditional OS?
>> Colin Dixon: So I think the difference is in access control as I mentioned earlier, sort
of one piece which is that, you know, may be for reasons that just have to do with
historical reasons, you don't have applications as priorities. Time tends not to be a role in
access control decisions. And there's a lot less effort in convincing people that you
configured it correctly and actually letting people interact with the decisions that they
made about the access that they've done. I think that's one part of it. And then the other
part of it is simply the standardization model for how it is that you standardize the
interface which you write applications against. So the model in a traditional OS today is
that Microsoft says this is the interface to a hard drive and it looks like this. And your
files are, or this is the interface to the camera and it looks like this. And there isn't sort of
a first-order way to shift that easily on a device tender and actually send that functionality
up to developers. And so this focusing on a more dynamic interface between devices and
applications is also I think pretty different from how commodity OSs are built today.
Now that I've talked about HomeOS, for a while and I would be happy to take questions
more. I'm going to meet with a bunch of you later on during the day. I want to talk about
ETTM, which is about sort of how do you manage enterprise network in a different and
more scalable way. So before I went to grad school for all the power and money that a
PhD gets you, I worked IT and I ran networks for people. And so I have some idea of
what it is that I want from my enterprise network in terms of what I want it to do. And
this is not a complete list but it's a list of sort of a bunch of things that I think you would
absolutely want from your network. You probably want a NAT whether it's just a share
of your IP addresses off of a lot of computers or these days to deal with IPv4 exhaustion,
you are going to want some kind of firewall to keep the bad guys out, traffic prioritization
to make sure that, you know, bit torrent traffic and bulk downloads don't interfere with
your video conferences, a transparent web cache for all the nice things that we love web
caches for. Remote access so that we people can work from home, and you want some
type of pervasive intrusion detection, so that you can tell when something has gone
wonky inside of your network.
In order to have this today, you have to buy a bunch of proprietary boxes and put them in
your network. You have to buy a firewall. You have to buy some kind of IDS. If you're
lucky they might be bulked together. The NAT might be there too but you'll probably
want a different one. Different people want specializations. This is sort of expensive but
not only is it expensive, it's as you scale it out and you buy multiple of these in order to
scale, it’s potentially inconsistent. It's an extensible. You have to sort of hope that
someone has built the box that you want already. And it doesn't scale up automatically,
which is if you add twice as many hosts, you have to hope that the boxes are either under
provisioned or you have to buy new ones. And so just to give you an example, so these
limitations I am going to talk about intrusion detection. And you need to think about how
this would apply to other applications I talked about. But intrusion detection is typically
employed at the edge of your network today. And it, can you imagine the intrusion
detection box being this red firewall in the middle of your network and sort of as traffic
comes in, it is going to classify some traffic as good and nothing to worry about, and
other traffic is going to be classified as attacks. It's actually going to raise an alarm, let
some IT person know that this might be an attack and he might want to go check on that
post. It might even actually block the traffic.
But because it's deployed only, and it can even do things like detect outgoing traffic and
say, it looks like somebody inside is launching an attack outside. That host might be evil.
We should go check if it's infected. But because it's deployed only at the edge of the
network, it's difficult to catch internal threats. So if you have sort of a host inside of your
network attack another host inside of your network, it's often difficult to try to catch these
things. And you might say well this is just a problem of increased deployment. I can buy
more boxes and drop them in. But not only is that kind of expensive just because now
you have a bunch of these different boxes, it's that most of these services are state full.
They actually require, and so this requires coordination amongst these devices, which is
difficult or impossible depending on whether the devices you happen to be buying
support this.
And so this sort of ends up being sort of hard to provide the pervasive intrusion
protection that you would really like to have a network today. And so we are going to
raise sort of a radically different approach as to how you might try to and get this
pervasive network services that we want in networks. And we are calling this approach
an end to the middle. And this is sort of invoking the end argument of pushing
functionality back to the edge and leaving the core as a simple straightforward thing
which basically just forwards packets. And it's based on the realization that we don't
need physically centralized hardware to make logically centralized decisions. And we
can actually implement the network functionality we want as a virtual middle boxes
running as distributed software across the PCs in the network. And so to give you an idea
of how this might work, let's return to the intrusion detection example. So let's assume
that we have a trusted shim at each host that's monitoring and giving us control of the
traffic which arrives there and leaves. This gives us pervasive monitoring so when
somebody actually starts to launch an attack inside your network, you can detect that they
are doing something strange. And it also gives you the ability to identify that they are
probably infected and actually quarantine that host, either at that particular host or the
other host blocking traffic that comes from them.
And so this is sort of the approach that we take in ETTM. And there are a bunch of
challenges which I will obviously go through and explain how we will overcome them.
But first I just want to say a little bit about what our assuming in order to build this. So
we are assuming that we are advising an enterprise network not a transit network, not a
data center, not a home network. So we have some kind of IT staff that wants to
accomplish goals. It's really about trying to service the actual hosts that are on the
network, not trying to trend a lot of traffic. And it's not in a data center. These are actual
users using these hosts. And it's a single administrative domain. It's about giving one
network administrator or a network staff more control at lower cost of their network.
>>: Is Microsoft a single administrative domain?
>> Colin Dixon: Probably not, right? Probably Microsoft Research is one domain, and
you might even do it for building. I think there is, most organizations have a pretty good
idea of what the network administrator domain is and isn't at this point, which is who has
control over the network, who actually administers the switchers and the hardware. Does
that make sense?
>>: Yes. I was trying to see what your underlying assumptions about single
administrative domain is?
>> Colin Dixon: With regard to single administrative domain, I am not trying to merge
conflicting policies or anything like that. Now there's going to be one policy that's
actually agreed upon by somebody and that is what is being implemented as opposed to
trying to figure out how to sort of negotiate policy differences and find the lowest
common denominator kind of thing. That's what I mean. So I'm going to make a set of
assumptions about posts in particular that they have trusted computing hardware in them,
that they are multi-cores and that they are virtualized. And I am going to assume that
switches provide some sort of access control. Now this might seem like a lot of
assumptions to make about hardware, but unfortunately the assumption I make about
hosts is increasingly true about commodity PCs today. And the assumption that they
should provide axis control can be provided by 802.1X which is commonly supported on
commodity networking gear today, or by OpenFlow which is an emerging standard for
programmable networking hardware.
>>: [inaudible]
>> Colin Dixon: For the purposes of what we built it's a TPM. I can imagine using
other, in fact I can imagine other trusted hardware being much more useful, but for the
purposes of this we built on top of TPM's.
>>: Did you consider ETTM as a replacement to the current process or complementary
to the [inaudible]
>> Colin Dixon: Are you asking me in my academic researcher hat, or how I would
actually get it to work in practice on a real network in the short term hat?
>>: That answered the question right there.
>> Colin Dixon: I think that in the long run this would be interesting and you might do
hardware acceleration for some tasks that require it, over time. In practice you want to
get this working today you would probably start off with providing services that are hard
to provide in a reasonable way or a reasonable cost with middle boxes and solutions like
that; nothing’s going to happen overnight. The other thing which I can imagine
happening is for small and medium businesses that affording all of the sets of hardware
that provide the properties you actually want for your network is difficult especially in
providing people that can actually run them is also difficult. And so the idea that you
might provide a software solution that they could go get for a reasonable cost and provide
them the properties that they wanted, seems like they might get almost all of their
functionality from this kind of system.
>>: Yes, but there are some benefits to doing in process rather than do everything
[inaudible]
>>: [inaudible] you always want to stop our category as early as possible.
>> Colin Dixon: But what's earlier than the host which is sending the traffic? That's
actually a place where…
>>: [inaudible] your host, right? So you still need all of the new boxes to stop the traffic
[inaudible]
>>: Not if you believe the argument that Colin’s virtualized, you know virtualized midboxes are robust [inaudible] it doesn't matter that there lost.
>> Colin Dixon: With his claim is that how are you going to protect that?
>>: [inaudible] switches that can do that?
>> Colin Dixon: So yes, you can imagine having some, and I can't spend as much time
on this as I would like. If you end up with OpenFlow boxes and they implement the new
extension to this spec, then you can actually do things like rate limit them. And you
might imagine that rate limiting is something that would want the network to natively
expose and allow you to actually use. There is a question about what functionality you
do want to eventually end up with the network, and so the purpose of this talk I am going
to assume basically it's just packet forwarding, and turning ports on and off. You can
imagine the slightly extending that if they got you significant benefits in terms of
manageability or performance.
>>: [inaudible] I don't quite see why you actually need it and it definitely comes to
performance even if it's [inaudible]
>> Colin Dixon: Why don't you ask that question again after I get to that point and you
can explain to me why it is that I don't need it? So we have three key goals in building
ETTM. The first is we are going to make it easy to extend network functionality, make it
easy for people to build or build applications that give them the functionality that they
want on their networks or deploy something that somebody else has built and actually get
that. The second goal, and in order to do this, we are going to have one platform with
simple APIs that enables people to build the applications as well as be able to play them
on the network. Second we want to automatically scale with network demand, and in
order to do this we are going to run management on already present hosts in the network.
So we are actually run this stuff on top of the hosts which come into your network which
means they bring their own management resources with them. And lastly if you are
going to run management tasks on people’s machines, you don't want Bob in
accounting’s machine going down to take down your network. So you need to function
pervasively, consistently and fault tolerantly in the face of this distributed
implementation. And to do this were going to use a standard distributed systems
techniques in and we’re going to use a consensus algorithm called Paxos. And I can
describe how we use it a little bit later.
>>: For your first [inaudible] where do you expect to the new functionality to run
virtualized pure [inaudible]
>> Colin Dixon: Ask that question in three slides if you don't understand. So there's a
bunch of challenges, two slides if you don't understand. There are a bunch of challenges
that come up when you propose this approach. The first and foremost is can we even
trust commodity PCs to faithfully carry out network management tasks? Existing
commodity OSs are famous for getting infected and famous for having users that are
selfish and will configure them whatever way gives them the best service.
>>: That's why we have them [inaudible] I mean if we could trust them, then we
wouldn't need to put them behind a firewall to protect them [inaudible]
>> Colin Dixon: Yes. And there are reasons why you may be able to trust at least a
subset of what they do, and I'll explain that. So can users’ commodities actually quickly
make decisions; can we make them fast enough. And lastly how can we enable
developers and administrators to build new applications and get them out on networks
easily? And so I am going to address each of these challenges in the process build up the
architecture of a single machine and what it is going to look like in an ETTM network.
So the first question is can we trust PCs. And the answer is, as John said, I think
obviously not in a canonical way that you run them. Commodity OSs aren't up to date.
They get infected. Users are selfish and configure them however they want to configure
them in order to get the best performance. And so you might throw a buzzword at the
problem and say virtualization; virtualization is what we use to solve problems.
Hypervisor in this case actually does help you. It gives you a point of control so
something is less likely to be infected and it does actually give you a place where you can
talk about stopping traffic which commodity OS sends.
But there is no way to actually tell if a hypervisor is running on a host. It will lie to you
to get a better service. So we actually extend the hypervisor with support for TPM which
lets us actually get an attestation from the machine which says in fact it did boot the
given hypervisor. We can actually tell that it booted that. But that's not quite enough
because just knowing that it's running what you want or not doesn't help you if you can't
keep it off of your network if it isn't running what you want. And so we actually extend
the 802.1X network authorization protocol in order to basically deny access to hosts until
they've sent us an attestation that they are running the particular software stack that we
trust.
And even that's not quite enough. We don't really want to stick all of the functionality
inside the hypervisor, or we add to boot and add what we have to trust in order to keep
things isolated. Instead we are going to run our management tasks inside of a test
execution environment. I think this answers your question. Which is a specialized
virtual machine on top of the hypervisor which is used to both filter traffic as it comes in
and out of the host and also to run any other network management tasks that you might
want to run in a trustworthy manner.
>>: Did you just claim that putting these allocations inside the [inaudible] inside the MM
takes them out as a trusted computing base.
>> Colin Dixon: No it doesn't take them out of the trusted computing base. It does give
you another layer which you sort of, which you don't have to…
>>: Does it or not? Come on buddy.
>> Colin Dixon: So it's trusted from the point of view of the network. So the network,
but it's not trusted from the point of view of the commodity OS, which is to say now
you've given the user some assertion, so it depends trusted from whose point of view.
Also this model makes it easier to be portable across different hypervisors, which is
critical if you want this to actually work.
>>: [inaudible] we go back to [inaudible] question if your goal is to, it's not safe unless
everything is going to filter which is why people usually put a firewall at the chokepoint.
What you're doing is saying well we’re going to distribute the chokepoint to make sure
that the attack services no different a little piece of the firewall’s on every host and have
you plug in printers or embedded devices or anything else it doesn't do that [inaudible]
firewall.
>> Colin Dixon: The short answer to that which I'll explain in a bit is that you can create
secure tunnel to, but you can run into you get [inaudible] environment on some other
commodity host that's embedded. And you can create a secure rate limited tunnel to it
across your network. And that that's a pretty reasonable thing to expect of the switches in
your network to be able to forward traffic that way.
>>: So you mentioned that there was a bug in the hypervisor that allowed the [inaudible]
to be compromised. [inaudible] inconceivable but let's say it happened. What do you do
at that point? Is it over? How do you…
>> Colin Dixon: Let me answer your question with a question. What if there was a bug
that gave you control of things in Cisco's IOS that gave you access to the network? Is a
game over? And so, yes, the answer is yes. But I don't think it's any different, in fact the
last time I checked I think Cisco; in 2009 Cisco IOS had something like 20% more
vulnerability days than Windows.
>>: That's not surprising.
[multiple speakers] [inaudible]
>> Colin Dixon: We are pretty good at updating; I mean updating PCs is something we
are pretty good at, especially inside a corporate environment.
>>: [multiple speakers] [inaudible]
>>: If your PC you are updating is the same everywhere, getting them all updated isn't
particularly harder than getting all of your system boxes updated.
>>: If you do an update here…
>>: [inaudible] and so in order to fix that you've got to go out and touch the TPM on all
of the machines in your network to tell them that that old signature isn't good anymore.
>> Colin Dixon: So this is actually something that you could fix with, it's the same
property as upgrading your IOS boxes. You need to deploy some, so we actually store
the information about what's a valid signature and what isn't actually in one place on the
network. So you just need to update that file, push out a new validation so that to the
people, so that just kick everybody off the network, if that makes sense.
>>: Yeah but [inaudible] I now own the network interface in their I'm not going to look
at your revocation list anymore,
>> Colin Dixon: But the switch will. That is the assumption, I guess. If you break that
and you break the switch, so that way the switch doesn't listen. It's true. Then you have
taken over everything that…
>>: How can the switch tell?
>> Colin Dixon: So the switch is using 802.1X, right, so we can actually push out
assigned, close the internet port for all devices and require everybody to re-authorize
themselves.
>>: But I thought the model that you are trusting [inaudible] the hypervisor because the
ETTM said it was.
>> Colin Dixon: So yes, and then you kicked me out of my model that would do that,
right? You said the hypervisor was broken, and I am giving you what you would do…
>>: Because it will be. You know it is.
>> Colin Dixon: But does that need to print from the switches inside your network being
compromised. And I guess I feel like it is a slightly different trust bundle. I'm not
convinced that it's actually concretely different, if that makes sense. So to say the
switches can be compromised as well, and we don't seem to worry about that. I don't
know why.
>>: It's easy to get the host. Know the reason why is because switches have a very
isolated control path, which is data path which is why usually packets pass through
switches…
>> Colin Dixon: And usually packets just pass through hypervisor's too.
>>: What I'm saying is that the host contains the hypervisors code. In other words if you
own the host then you can start writing bits into the hypervisor presumably. If there is
this hypothetical [inaudible] you can't even talk to them. They don't even have their
external IP addresses [inaudible]
[multiple speakers] [inaudible]
>> Colin Dixon: My claim is in practice, you know, switches have just as many
vulnerabilities, just as long of a windows opening as commodity OSs, actually slightly
worse. We just don't worry about it. And I don't know why we don't worry about it.
Maybe I should put a slide of my talk about you should be worrying about your switches
in your routers. I think the actual reason why they don't get compromised as much is
because they have less value, which is to say that end hosts have value in the resources
which you can steal from them. And compromising a switch is much harder to get value
out of. But in that sense the commodity OS would be the target of the thing and this
hypervisor and that AEE would still be of relatively little value, except as a way to get to
the [inaudible] OS.
>>: Hypervisor has just as much value as [inaudible] replace it.
>> Colin Dixon: With that long digression, these are all useful things and certainly
conversations that I have had, and it would be interesting to talk about it. I actually have
a separate site product that I would like to do about TPM's for network hardware which is
that we should not just trust network hardware because it says it's trustworthy. We
should actually try to do similar stuff to this and get ourselves an extra layer. So
secondly can commodity PCs consistently make quick decisions? I'm going to Paxos to
do this and I will talk about the particular interface as opposed to Paxos; Bill will talk you
about how to make Paxos fast if you want to know. I think he knows more than I do.
And lastly how can we enable developers and administrators to build applications on top
of this model actually get them out on your network? And so we have an interface for
filters were how we give applications control over the packets that flow in and out, and
we a micro virtual router that invokes filters on the packet flows. And the consensus API
combined with this filter and micro virtual router API is the API that you actually write
network services against. And so you end up with binary that you can go actually and
distribute to people and it works. And so I am going to talk a little bit about each of these
challenges and the pieces of technology that we built in a bit including about how we do
trust authorization and how we do legacy hosts. Go ahead
>>: [inaudible] environment is Lennox [inaudible]
>> Colin Dixon: It's a BUN-2 server at the moment. It's just, some VM that has a
reasonable way to do network processing across hypervisor Cs. I mean it happens to be
Lennox because that what was easy for me to work in when I was building; there is
nothing special about Lennox that I rely on. And you can imagine building a really
stripped-down something to do this that had basically a
>>: Is actually a [inaudible] on how to make sure it's secure and still easier to [inaudible]
right
>>: [inaudible] developer it's still hard.
>> Colin Dixon: So right now it's about 400 MB. And so I'd like it to be about 10 and
actually I think it may be interesting to talk to, I guess it's not quite related to the
drawbridge stuff but it's similar.
>>: [inaudible] your challenge to use software trying to solve this problem obviously
you're solving it from a different angle.
>> Colin Dixon: Okay so, the way trust authorization works in this, we have a switch.
The 802.1X protocol causes the switch when it sees new hosts on it to actually go and say
you are a new host. I am not giving you access until you tell me who you are and why
you should have access. That takes about 50 ms to figure that out and you can fix that by
sending a request explicitly, but you think that your touch that but once you've done that,
the host then responds and in this case we've overridden it so that the identity which it
uses is of the public key that it's actually going to sign the authentications with, rather
than any normal identity that you would use to that 802.1X in EEP.
That's been forwarded to the verification server. And the verification server you can
imagine being something like a dedicated box or you can do [inaudible] what we do run it
inside of in test execution environment on another host, which so you don't actually need
any dedicated hardware to do this if you have other hosts that are up. And at which point
[inaudible] challenge nonce to make sure that whatever [inaudible] it gets back is fresh
and this takes about 20 ms for the verification server. We use the open radius server and
it’s a little bit slow. But that doesn't really matter because signing the registers that
specify the state is even slower. It takes about a second to do it for the hardware we
have. I understand there is a newer hardware that reduces that by a factor 2ish. This is
just because TPM's, nobody cares about their performance, and nobody's optimized it.
It's likely to be in the future that this will either migrate into the CPU or simply have a
slightly faster processor and then this issue should go away.
Then the verification server and the host will negotiate new policies or new software that
needs to be deployed and finally you'll open up the port. And all in all this takes about a
second, a little bit more. This can be overlaid with commodity OS boot time making it
basically trivial. And in the future as I said, if TPMs are actually used them get much
faster and this will be something on the order of how long it would take you to get an IP
address anyway on a to get onto a network..
I said I would talk about legacy hosts, so let's talk about hosts that either don't want to or
can't run their own execution environment. So they are going to show up on your
network, try to connect and be denied because they are not running the software you
asked them to do. The nice thing is that since the AEEs are actually virtual objects, they
need to run on the device whose traffic they are filtering. So we can substantiate
[inaudible] somewhere else on the network and as long as we create a secure rate limited,
that only goes to that AEE, we can then actually give that host access to the network.
And so this enables you to get printers that are protected and actually have their traffic
filtered, iPads whatever, X boxes, and so this does sort of mean that you can have
everything on your network. You don't need to have all odd devices to be able to run this
kind of stuff. And actually when I started off I thought this was going to be a real
problem because everything is going to be a phone and it's not going to have
virtualization or trust and computing hardware, but in the two years that I've been
working on this project, phones have started to get virtualization and trust computing
hardware in them, and so you might actually be able to do this on phones natively, and I
think that would be kind of interesting to go and play around with.
So in order to have consistent state that applications run against we use a shared,
consistent shared table abstraction so the table consists of one row per application. And
each cell is the result of a Paxos round, which is to say that each cell is consistently
agreed-upon. Also the ordering of the cells within a row is agreed-upon. And so this
way you can have sort of you know everybody sees a prefix [inaudible] row. In between
rows we don't do any inconsistency so we have an application that wants to have
crossroads to implement that. But this gives us scalability because each row is thus
actually independent and can make progress. For those of you that are really familiar
with Paxos, you'll realize that Paxos only makes progress as long as the majority of the
nodes are up. And if you're talking about running on commodity hardware, it's possible
that the majority of your nodes could fail simultaneously. Imagine you know a building
goes out, your power goes out, or floor goes out, and so in order to deal with this we
allow for unsafe progress explicitly. So for instance if the NAT application sees that one
of the nodes goes down, it can actually say that it would like to fork like Fiat declare a
new Paxos group which is the set of up nodes right now and continue to make a separate
set of safe progress here, and then later on we might have to merge any conflicts. And
while in theory merging these conflicts is very difficult, in practice from anything it's like
a NAT, relatively straightforward. You can go through and look and say oh these two
external port and IP address allocations are different, so it needs to be reset in each
direction and start blocking traffic.
>>: I think I missed [inaudible] and all risk and embarrassing question [inaudible] what
is it you are trying to keep consistently using Plaxo’s [inaudible]
>> Colin Dixon: So one of the features that I give to applications is a Paxos facts hash
table, things like that. So you can agree upon anything that you want. In particular, I use
it to build data structures that are sort of snapshots of some point, some recent point in
time.
>>: [inaudible] hash
>> Colin Dixon: So for instance imagine that you want to keep your network address
translation table consistent so that you know that IP address 1.2.3.4 port 50 isn't assigned
to different connections. So the only way to do that is to know what the other allocations
are. And so you can use this to build up a table of externally visible addresses to
internally visible addresses and know that you have a double assigned something. And
know that it won't go away if a host drops.
>>: [inaudible] corner at all [inaudible] in other words DHP manages this essentially
without…
>> Colin Dixon: In a common case we use leader Paxos, and so that's effectively what
we do. We just use this as a way to be resilient against that leader failing as well and also
have consistent state in case you know…
>>: [inaudible] at what point were you sitting writing code and you reached a problem
when you said oh we can't solve this without distributed agreement and so we to put
Paxos in. I am not quite clear on that.
>> Colin Dixon: So imagine that you are writing a NAT in a distributed fashion, and you
want to assign, and you have a new connection, and you want to assign it to an externally
invisible IP address and port. So now if we assume that there is some designated leader
we can just, on the IP server, we can just go out and ask for it. And they will come back
and do it. Now if you want to be resilient against that person going down because they
are also a commodity host, you need some replication strategy. And you could use twophase commit, you could use, you know, and that has advantages and disadvantages with
respect to Paxos, you could use something else. We chose to use Paxos because it doesn't
impose much overhead over other replication strategies and it buys you a lot.
>>: [inaudible] actually the wrong analogy here. What's happening is I'm [inaudible] for
my connection and you do [inaudible]
>>: There's an algorithm [inaudible]
[multiple speakers] [inaudible]
>>: So imagine that you have a some application [inaudible]
>> Colin Dixon: So if you want to do a femoral assignment of ports in this case, you
have to have some way of assigning resources to people in your network. And you can
do that with a central person. If that person fails, you need some replication strategy to
make sure that you don't lose the configuration data and Paxos is basically as fast as any
stupid replication system you can do, and it has a lot of better features than you get from
other replication things.
>>: [inaudible]
>> Colin Dixon: Applications can choose to be always safe and not always live, or they
can choose to be explicitly live with one particular boundary at which something can go
wrong, which is to say that you know that internally in this timeline it's consistent, and
internally in this timeline it's consistent and that's much better than having no guarantees
about anything afterwards. So it has some…
>>: [inaudible] can to do any better than that? See really can't ask them to do better than
that.
>>: Well know, if you're [inaudible] hey everything is safe, except when it's not then
why bother providing safety? That's what I don't understand.
>> Colin Dixon: So you can choose to say everything is safe, and just not be live if the
majority of the hosts go down, if that's what you want. Or you can choose to say that I
will be safe for this period of time, and I will be safe for this period of time and I will
have to explicitly know that you could at one point end up with two timelines that are
internally consistent but need to be merged. And that is pretty clear. And so I think that
that is, the idea that you could actually have two timelines that are internally consistent
but need to be merged is a lot different than you could have arbitrary bad differences
between your timelines. So it is not quite as nice as any one timeline, but it's a lot better
than the alternative.
>>: So therefore [inaudible] has questioned [inaudible] I ask you when I see you in the
afternoon [inaudible]
>> Colin Dixon: And so eventually once you've merged all the changes back in you can
actually garbage collect this row and it goes away. So in order to actually give people
access to the flows that go into and out of hosts we have a micro virtual router which
interposes on individual traffic. We have a high rate implementation. One part of it uses
open vSwitch which gives us line speed up to a couple gigabits but only has OpenFlow
commands and so those are only for header operations. So NAT works really well in this
environment. Deep packet inspection does not. Then we have a separate implementation
built-in ip-tables which actually gives you the full packet, let's you write your filters in
C++ and this actually lets you build things like deep packet inspection and do HDP
caching on content and all of this good stuff.
The filters themselves have a relatively straightforward API and this is actually what we
write stuff against. These first five functions are really just metadata about when the
micro virtual router should run the filter on what packets and in what order. And these
last three are really the ones where you actually do things where you can filter, capture,
delay, re-write, drop and inject packets. And we actually have separate versions of these
when we use headers versus when you don't use headers so we can tell which particular
substantiation of the micro virtual router and the filter off to. And we also [inaudible]
function on these filters in order to go out there and make changes update state and be
able to collect other things.
So before I go into evaluation stuff, I want to go through one example where we put all of
these things together and show how they work in practice to implement the network
address translator. So on the left here we have a particular host. On the right we have the
table for the Nat. And for those of you just a quick introduction of how a NAT works for
those of you who don't know what it is. It shares one IP address among were a few IP
addresses among a large number of our hosts and traditionally it does this by being in a
centralized location like this router, where it can actually see new flows, when it makes
the mappings when it needs to and apply those mappings to traffic as they come back and
forth. So instead in the ETTM implementation, when a new flow gets created in a
commodity OS and application, it gets sent out and the hypervisor will forward it over to
the text execution environment, where the micro virtual router will say it needs to be
NATed so we are going to send it up to the Nat filter. The Nat filter will note it is a new
flow and so we actually need a new mapping, so it will request a new mapping from the
distributed consensus algorithm. This will actually go out and talk to all of the other
hosts on the network or whoever is participating in the consensus algorithm, and this will
run around eventually agreeing on some mapping. In this case C maps to D. this result
will be communicated back to the host, which is actually resending the packet, and the
mapping can be returned up. The packet can be rewritten according to the mapping and
sent out to the general intranet. And then later on when a packet according to that flow
comes back in, it can actually go through, it gets routed out to the Nat filter and can be
rewritten without any consensus operation needed because it has mapping.
>>: How does it get routed to the right…
>> Colin Dixon: I have three different importations of how that works right now. One of
them assumes OpenFlow and we actually install the routes inside the network. Another
one uses…
>>: [inaudible] you are installing…
>> Colin Dixon: You install, you install [inaudible] inside the network as to how to route
packets, and if they don't match then the OpenFlow controller which is us, gets notified
and we can go back and re-…
>>: [inaudible] the NAT then…
>> Colin Dixon: Well no, not necessarily, because the Nat has hard state, and we don't.
So if a Nat box got re-set, then the connection would break. Whereas if a switch gets
reset, the connection is still live.
>>: This is the control half of it.
>> Colin Dixon: The second invocation we actually spoof Arps for the global IP address
to be the broadcast Mac address, which is a horrible abuse of the way networks work, but
actually works pretty well in practice.
>>: [inaudible]
>> Colin Dixon: Externally there is an IP address. We actually arped the external IP
address with the Mac address, for the broadcast Mac address, so everybody receives all
traffic destined to the global IP address, and just only, the person who wants, deserves it
will actually accept. And that's a horribly simple, it does work, it works pretty well
actually. And the last thing we do is we just rotate the IP address across hosts. We
actually just make one of them [inaudible]. So there are a bunch of different ways of
getting around this, but yes, you are right. It's kind of tricky. And you actually caught
this where I think only one person in the CI, caught the fact that that was sneaky.
>>: What else are you sneaky about?
>> Colin Dixon: That's actually detailed in the paper pretty extensively. It was just
omitted because it's a painful thing to have to go through when people are explaining
distributed sentences is probably a bad move, never mind getting into the details of lowlevel NATS. So we implemented a whole bunch of stuff on top of this even though there
is a Nat deep packet inspection, web cache, traffic prioritization, a worm scan detector, a
generic firewall. Most of them are only 100 lines of code on top of this. And so we think
it's pretty easy to build a bunch of interesting stuff on top of it. And in fact, I'm going to
talk a little bit about that traffic prioritization stuff later on which it seems like, which is
one of the harder things to write build on top of this.
Isn't this going to suck in terms of performance? We’re running this on commodity
hardware on the edge of the network rather than running it on [inaudible]. So we really I
am going to focus my evaluation on sort of the performance and the underlying pieces.
>>: [inaudible] only applications that require distributive [inaudible]
>> Colin Dixon: No. They'll do. Deep packet inspection doesn't. The worm scan
detection actually needs to share views of traffic sketches across it in order to be able to
say this host is talking to a lot of other hosts so therefore it's probably doing something
bad. The firewall that we built is not state full but you can imagine one that is. The
traffic prioritization actually does per round trip time per flow bandwidth reservations in
order to make sure that we are allocating bandwidth according to some policy. So most
of these involve some form of distributed consensus.
>>: [inaudible]
>> Colin Dixon: It's okay. [laughter]
>>: It's a [inaudible] share information that how to reach a consensus.
>> Colin Dixon: So consensus doesn't mean everybody gets equal power. Consensus
just means everybody eventually decides the same thing. Sorry this is the difference
between what we say consensus means in normal world and what consensus means in a
distributive systems world. In the real world consensus means that we all get together
and we agree and it's happy and it's an egalitarian process. Consensus in the distributed
systems world means that everybody will make the same decision. It might have been
decided by a dictator that says everything is exactly what I say, but it will just be the
same decision everywhere.
>>: Really need expansion [inaudible] consensus
>> Colin Dixon: Is a consensus fast? The answer is yes, as long as you don't write to
disk and you are making decisions in less than a millisecond and you are doing this on a
LAN, you can make fault time decisions in about a millisecond, less if you're running on
a smaller number of nodes and if you sort of pick a subset of the nodes that is highly
available and will connected this works really well and really fast. But it's just not the
bottleneck that you might imagine it would be. So the second question you asked is, does
it scale? How many decisions can you make per second? How many new things can you
do? And so depending on the group size…
>>: [inaudible]
>> Colin Dixon: That was a number of people participating in consensus. We actually
ran that on a subset of the nodes. So you can have 500 computers and you would have
some core of 10 of them that are well-connected and fast that you decided run the
consensus and 10 is plenty, for most reliability point of view. The trick is to have enough
so that way they don't go down often.
>>: You want to run it on [inaudible]
>> Colin Dixon: Yea, you want to run on some numbers between five and 20. I think
Chubby is running on 13. So in terms of scalability, we can add somewhere between
1700 and 8000 cells per row for each second. And they are really good techniques for
scaling this up if you want to scale this up. You can partition things. You can partition
across applications. This just isn't going to be your Achilles' heel in this kind of system.
>>: [inaudible] so your equivalent 8000 [inaudible]
>> Colin Dixon: Yes, and so what we have written is about is a few hundred lines of C.
We haven't spent a ton of time optimizing this. So the other question you might ask do
filters hurt performance. Actually we are doing some packet processing on each packet
as it leaves. The first thing I want to note is this is flow size as in the X axis and this is
the throughput on the Y axis, so for large flows both using the NAT filter and not using
the NAT filter, we get the full gigabyte per second. There is basically no difference.
And in fact for all flows bigger than 10 kB there is no difference between the two flows.
And the only reason there's any difference for very short flows is because we are using a
Paxos time to start off the ground trip time which slows down slow start for the first
round. And this is actually two hosts in the same network so the round-trip time is
actually about 180 µs and we are imposing about a millisecond delay before we make the
connection. If you were running on a real connection where the runtime is about 30 ms
this would be totally invisible, and so filters don't really seem to her performance much.
And I know what you're thinking; well all you're talking about is simple filters, things
that run on one place. What about something more complicated like trying to do
bandwidth allocation, trying to actually do fair share bandwidth? So on this topic graph
you can see what happens if you don't do anything and you have a big queue. So if you
run bit torrent while you are trying to make request to Google.com, what you can see is
when you're not running BitTorrent, queues are fine; everything is okay. Suddenly you
start running BitTorrent, and it's going to fill the queues to capacity until it sees loss.
And this is going to result in a taking basically as long as your queues are occasionally in
order to make requests. In this case like 10 seconds and we didn't [inaudible] that badly.
And so in practice what you'd like to do is something different. You would like to
actually allocate bandwidth in a fair way so that the BitTorrent connection cannot blow
your queues quite that badly. And so we actually implement a system where we make
reservations for bandwidth. Every round-trip time, each flow actually would trigger a
reservation that says value was this much bandwidth this time I would like to get that
much bandwidth again this time and there are details in the paper, but in the end we
basically don't allocate more bandwidth than what we have. We know how much
bandwidth we have in the access link, and the result is much more stable latency because
you are not blowing these queues way out of proportion.
And the cool part about this is that it actually works. We can make per round-trip time,
per flow decisions and actually go out and do bandwidth allocation in a distributed
fashion on a network and that's kind of surprising. I don't think many people would've
thought you could do bandwidth allocation on end hosts in a consistent way, and have it
actually work.
>>: [inaudible] latency is couple hours of magnitude better than your round-trip latency
on the internet.
>> Colin Dixon: Yes, because it is in practice. We haven't looked at how to extend this
to wireless quite as much and that might change some things because round-trip times
would get to be kind of strange. But I have some ideas there. So in summary enterprise
network management today is complex expensive and ultimately unsatisfying providing
the kind of properties that you actually want out of a network. And I would urge people
to not consider Microsoft. Microsoft is the best managed enterprise network I have ever
worked at. It is really well run. There are lots of places out there where it's just terrible
and people really struggle to get these things to work.
>>: Did you say Microsoft was unsatisfying [inaudible]
[laughter]
>> Colin Dixon: No. So I used to work IT. And it used to drive me and nuts because I
knew what I needed to do in order to get the problem done and then I couldn't. And I just
wanted to write code to make it happen. And so that's where this actually came from.
Here I am seven years later; I come back and I want to write the code that makes the
thing that I want to happen on the network happen. And so that's sort of what I mean by
ultimately unsatisfying. So we propose a radical new approach implementing network
management tasks as software on end hosts. We make it easy to write and distribute it
applications for networks. We can manage resource management, scale with demand
because they are the hosts that are coming onto your network. Tasks run pervasively
consistently and tolerate faults, sort of by design. And we implemented a bunch of
applications on top of this. Obviously I am not the first person to realize that you need to
manage your networks. There is a lot of related work in network management. In
particular industry HP IBM and CA have written tools, but these are mainly around
visibility. They are about what network links are up or down and how much traffic they
are carrying.
A bunch of other work has worked on how it is that you actually pick paths for packets
across your network, but not really anything other than the particular paths that the
packets are going to take. Other work including from people that are currently in the
room, focuses more on isolation albeit not performance isolation or security isolation.
But again not the pervasive nature of all the tasks you want carry out on your network.
And there is also been other work that involves end hosts but usually as a side piece to
other network management. Network exception handlers try to use end hosts under
certain exception conditions in order to be better and system monitors tries to use end
hosts in order to get visibility into the network but then again not actually control.
On the other end there's been a lot of work on programming network devices kind of like
things in the home. But it tends to be either focused on specific domain or just be a
couple of ideas that were tossed out there without implementation or very restricted
programming environment. And other stuff as in a monolithic system b-day commercial
academic tend to be closed above, close below or both. And we really want to be open
below to all possible devices that we might be able to get people to write the right drivers
for and open above in this really distinguishes what it is that we've done here from other
stuff.
I have a bunch of other things I would like to work on. The first is I would really like to
build a fully integrated programmable network, something like HomeOS on the top for
programming applications across the devices. And something like ETTM below so that
you could actually talk about something that's happening on your network, and actually
have security policies that across back-and-forth like debugging and diagnoseability seem
like much easier when you have some knowledge of what the network is doing below.
So I think marrying these two ideas actually has a lot of interesting things that will drop
out of it. I am really interested in home security and not your front door lock, although I
also like to teach lock picking so I can do that. But homes are increasingly networked,
increasingly wireless and this poses a huge number of challenges as to how we're going
to make them secure. Both from the pure security model of how to make these devices
not easy to hack, but also how do you get users to express what they want to happen? We
don't really have enterprise access control working in the enterprise today. And it's even
worse in the home; you don't have anybody that's around. But it has some really cool
advantages, because I'm not talking about some file on some server somewhere; I am
talking about that TV, that one over there in the corner. That's what we are talking about.
This gives people a mental model to route things in where you might actually be able to
solve some of these seemingly intractable problems in the home. I think that would be
really cool to work on. And lastly I'm interested in trying to build a peer to peer cloud
and you can imagine ETTM as the first step to get there in terms of suddenly having a
place where you can run code that you don't have, that you can trust it will run correctly.
But I think it provides for some interesting alternate economic models for how it is that
you run a cloud, rather than the traditional sort of pay rent or watch ads. It allows for
something more like the current model which is you go buy a computer you drop it
somewhere; that's how you get your computing resources. And this gives you that same
model, potentially gives you similar things. There are huge challenges to reliability
performance. We don't really have the premium models correct yet for the cloud. We are
just beginning to get that figured out, never mind for a cloud that that's on end hosts.
But I think it's cool and I don't quite know if it would fit as a side piece to existing cloud
infrastructure that does certain things better, or if its own particular thing for certain
applications. So I would like to thank my collaborators, my advisors, Tom and Arvind at
UW, Dane, Hardeep and Vjeksolav who are really great undergrads who built real stuff.
Undergrads are great; you should work with them if you get a chance. And my
collaborators here at Microsoft who worked with me on HomeOS. And so in conclusion
I think I hope I convinced you that networks are crucially important but they are really
hard to store a extend manage and secure. We have proposed a new approach of turning
the networks into programmable software objects, where we can solve these problems in
software rather than in hardware. And we present two case studies as to why it is you
might be able to do this and want to do this, and those were ETTM for enterprise
networks and HomeOS for homes. And with that I am done four minutes early and I can
take a few questions. It was 1030 to noon on my calendar.
>> Ratul Mahajan: No, actually we're twenty minutes late.
[applause]
>> Colin Dixon: It was 10:30 to noon on my calendar.