22431
>> Sudipta Sengupta: I am Sudipta Sengupta, and it is my pleasure to introduce
Shreeshankar Bodas, who goes by the name of Shree. Shree is currently a
post-doc at Laboratory for Information and Decision Systems at MIT.
The LIDS lab is the same lab which is home to folks like Bob Gallagher,
Demetrius Butikus [phonetic] and John [inaudible].
Shree did his Ph.D. in ECD from University of Austin in September of 2010 and
his research interests include resource allocation in data centers and wireless
scheduling algorithms.
Today he will talk about his work on fast averaging, which appeared in this year's
ISIT, International Symposium on Information Theory, and he will also point out
applications to Map-Reduce. I'll hand it over to Shree.
>> Shreeshankar Bodas: All right. Thanks for the introduction Sudipta.
And good morning, everyone, and thanks for attending this talk on fast averaging
with applications to Map-Reduce and consensus on graphs.
So before we begin, let me tell you that most of this talk is going to be about
applications to Map-Reduce. And after the main body of the talk we are going to
also consider the consensus on graphs as an application of fast averaging.
This is joint work with Devavrat Shah. So the particular problem I'd like you to
keep at the back of your minds throughout this talk is this problem of auto
complete suggestions for a search engine; that is, so there is one user that is,
this is a snapshot from Bing, and I started typing a query with F, and this is the
suggestions that Bing gave to me.
Facebook, Firefox, Foxwoods, and so on. And so the question is how does a
search engine give me such queries? These are based on the query logs in the
recent past by different users. And based on the value of a given keyword, these
are going to be presented to me in some order.
So we want to determine in an efficient and fast and online way a way of coming
up with these suggestions. So that this is an application that has motivated us.
So this application had certain features as we're going to see that, first of all, it
is -- it's possible to approximate the computation that goes on behind this without
hampering the results. So this is the particular application that I would like you to
keep at the back of your minds.
So let's we'll what happens when someone starts typing with F and waits for the
auto complete to suggest something.
So this is the picture of the back end of the system, where there is, let's say, a
master server. There are these bunch of slave servers, and these are the
different query logs in terms, in the form of files.
So when a query comes in, this master server wants to know, let's say for the
time being, that there are only three possibilities here. There is a Facebook,
Firefox and FedEx. I want to see the related ranking or related importance of
these keywords and based on the previous search queries.
So a simple number that I want is the following: That indeed search queries, how
many of them contain the word "Facebook" and how many contain the word
"Firefox" and "FedEx." And based on those results I'll rank them and present
them to the user. We want to speed up this operation, that's the way I'm going to
present to you the fast averaging algorithms, and we are going to see how these
are applicable in this setup.
Because this is essentially a counting problem. You want to count the frequency
of something and that can be approximate.
So at the heart of the problem is this following mathematical operation; that is, I'm
given a long vector of N numbers, think of N as large, because I have these huge
query logs, I want to search over them fast. I have these long vector of N
numbers and what I'm interested in is this continuity XI. So XI represents the
number of times a given keyword, let's say Firefox, came in the search query.
It's 01, 2, 3, number of times ->>: Do you know ahead of time? If you do, then average sum are the same
problems.
>> Shreeshankar Bodas: Are the same problems. Yes, if you know them, then
some of are the same problems. But for mathematical reasons we're going to
have this one in front because we want a multiplicative approximation error as
opposed to an aggregator error. So as you said, they're the same problem.
We're interested in this summation, and equivalently we're interested in one by N
times this summation. So mathematically this problem is not hard. This problem
just wants you to add up all these N numbers and just divide it by N.
As can you do it in a distributed fashion as I've shown it here that there are these
N numbers, suppose I got these 100 servers that can do the task. Then I'm
going to give, let's say, one-hundredth of the fraction of my numbers to first
server and the second server and the hundred server and so on.
These guys are going to compute the integer sums and then they'll report the
sums to the central guy. The central guy will add up all the hundred numbers
and then divide it by N. And that's your mu.
So this is the basic simplest algorithm you can do. And right now I have shown
here two levels of hierarchy, but it could have more levels of hierarchy.
So this mathematically the problem is not hard. What makes the problem hard is
the data size and your delay constraints that you want to respond quickly and
you want to process large volumes of data.
Nevertheless, so this deterministic algorithm has certain advantages and certain
issues that we need to address. So the advantages are first it returns an exact
answer, which is always a good feature of any algorithm.
It's clearly a distributed algorithm, as I've shown here, it can be done in parallel
by hundred servers or multiple servers, multiple level hierarchy, because the
addition operation is inherently distributed operation. So this algorithm can be
implemented in a distributed form.
What are some of the issues with this algorithm? First issue is complexity or
latency. And so we know that if you want to sum up N numbers, then there is a
theta of N complexity bound. You cannot do less than, or rather N minus 1
additions if you want the exact answer in the worst case.
But the point here is you may not want, you may not care for the exact answer,
because this addition operation of the averaging operation can be a means to an
end. Even if my actual answer to this averaging problem is not exact, maybe the
final thing I'm interested in, which is the ranking of my results, may still be good.
So if I'm willing to trade off some accuracy for speed, then I want to beat this
latency bound because this T'd off end may be prohibitively large. That's one
issue we need to address. Another issue is robustness.
If one of these servers is slow or one of these servers fails its task, then in my
implementation that I've shown here, first of all, the bottleneck is the slowest
server. I cannot, on the fly, decide okay that this server should not report.
Because I'm interested in the exact answer in this algorithm.
Or if this server fails, then somebody else has to complete his task. And it's
going to add to your other -- it's going to affect your answer or it's going to add to
your network bottlenecks. So these are a couple of issues that complexity and
robustness that we need to address for this algorithm.
So let's look at the issue of latency in a little more detail. So suppose I've got
these file servers and I have a bunch of numbers I want to compute the average
of. This is the central server who is going to do the final averaging operation.
So what happens here is suppose there is this server that is the slowest server.
It's always the case that the slowest server will determine when your answer is
ready.
But if this guy is overloaded because there's some background loading in the
datacenter that it is doing some other operations or there are some network
bottlenecks for which this guy is too slow to respond; then although all the other
numbers are in single digit milliseconds, this guy is in 50 milliseconds. You have
time to finish, time to finish your operation is here in this case 54 milliseconds.
But, hey, what happens is if I'm not interested in the exact answer, maybe I can
say that, okay, forget this guy. I'm going to compute the average of the rest of
the people and suitably normalize it, and maybe my answer is going to be
approximate, but my time to finish is 13 milliseconds for the numbers I have
chosen here.
This is a recurring theme that as we know from the simple MM1Q Markov chain
analysis, if you are very close to capacity and if you slightly reduce the load, then
your delay comes down by a large amount.
So that's the same thing that is happening here; that if you reduce the number of
servers that you need to respond by, let's say, one percent or five percent, then
you get dramatic speedup because these are the slowest servers that are going
to determine how fast your answer is really.
It's true that the slowest server is always the bottleneck, but you're making the
bottleneck wider in this case. And for all we know it may not be a bad thing for
averaging operation. Because our numbers are not just some arbitrary numbers.
They could have some correlation between them. So depends upon the
structure of the underlying data and how much error you are willing to tolerate
that determines how much area you're going to get in your final ranking.
So suppose in the extreme case I know that the X size, the underlying numbers I
want to average, are all equal. Suppose all of them are equal to some number. I
don't know what they're equal to, but in the extreme case they're all equal I can
just sample one of them and that's it that's my final answer.
So this complexity bound of theta of N it's actually in the worst case. That is,
these given numbers can come from what has to be no distribution underlying it.
But if the numbers exhibit some regularity, then I can get away with smaller
amount of sampling. That's the main point here.
One thing I would like to mention here, so this business of averaging with or
without replacement has been around forever, like people have beaten this to
death, and you can find classic large deviations results on how many samples
you need if you want to estimate the mean within a plus server and an epsilon
guarantees.
But most of those results, in fact I should say all the results that are known at
least in classic textbooks, they are in terms of the entries of the given vector. So
as opposed to that we are representing results in terms of the moment of the
given underlying sequence.
So it's a different take on the problem. And for Map-Reduce kind of problems
that we are interested in, this is a new dimension in which you can do the
trade-off, for accuracy, for speed, which if this is a means to an end averaging
operation, then it can be, it can be used without affecting the final result. So here
is the summary of our contribution to the problem.
We propose a simple randomized algorithm for averaging, and we analyze its
performance. We analyze the trade-off between accuracy and latency. Basically
we're counting latency in terms of how many samples do you need for getting an
approximate answer.
The improved job completion types of Map-Reduce applications using these
ideas from fast averaging, and the main intuition I'd like you to take away from
this talk, so forget about the actual system or forget about the exact result, the
one thing is if the numbers are regular, numbers are concentrated at least in
approximate sense around its mean, then the mean computation is an easy
thing. So I don't have to sample each and everything if I can sample, let's say,
half of them or root N of them. Even that can give me a good answer.
So before or without keeping you in any more suspense, here's a centralized
version of our algorithm. It's a very simple algorithm. It's the first thing that
would come to your mind.
Given this sequence of N numbers, these are the numbers, and I'm going to pick
any R out of them. R is a parameter that we want to choose later. But pick a
fraction of these numbers and average them and report their average. So this is
a very simple algorithm.
One slight modification that I would like to make to the final version of the
algorithm is that instead of picking R numbers out of N, let's pick every number
with some probability. So that this becomes a distributable algorithm. Because if
I had to pick R numbers out of N deterministically, then it's difficult to distribute
the choices among different servers.
So the algorithm is this: Pick every number uniformly at random with some
probability according to some random variable and report their average.
The advantage of this approach is that each number is going to be picked with
some probability already, but the probability that you pick a number is close to 0.
Because this number N is large I will be averaging enough of them, but I can still
substantially reduce the amount of sampling I need.
So this is the algorithm. Take a subset, average it, and report its average as a
sample. Mathematically, this is what is going on here.
These are the given N numbers, X1, X2, XN. This is a fixed given underlying
sequence. This is the number of occurrences of your keywords in the file. This
is not random. Randomness comes from the algorithm.
This Y1, Y2, YN, are our random variables, these are Bernoulli random variables
or binomial random variables. And you're going to sample these numbers X1,
X2, XN. These YI number of times. You can think of the YIs as the weight.
Compute the weighted sum of the XIs and divide by total weight you put in the
system and that is your estimate of mu. That is your mu hat.
So this is a nice simple algorithm. And as presented here it's a centralized
algorithm. But it's clear it could be implemented in a distributed way because the
binomial random variables and because these choices are independent you can
distribute this algorithm over multiple servers. So here are some other features
of this algorithm.
Although it's a centralized algorithm as -- you had a question?
>>: Need to distribute the summation of the YIs.
>> Shreeshankar Bodas: Exactly.
>>: So ->> Shreeshankar Bodas: These YIs -- so think of it as follows: So XI is one of
the entries in the given vector. I've got these bunch of slave servers that are
going to do the computation that are going to compute the XIs or agree with the
XIs. Say I flip a Bernoulli coin and decide for every server every file I flip a
Bernoulli coin and decide to sample that file, then the total number of people that
are sampling a give file are binomial. In that way the algorithm is distributable.
>>: Is there a hidden assumption that the variables are 0 and 1 because as
they're widely skewed, like if you try to estimate mean amount of money in
Medina, gates, one missing sample cannot be totally out of whack.
>> Shreeshankar Bodas: That's true. That's true. So the assumption is this: So
let's talk about what we are going to do with these samples. So we are
interested in giving auto complete kind of suggestions. So even in the
undersampling, heavy hitters like Firefoxes and Facebooks are not going to be
underrepresented. Let's say a million entries thousands of Facebooks. If I
sample it by 50 percent, then the related ranking of Facebook and Firefox with
high probability will be what I want. So if you had some obscure entry like FQ Q
of X that's not going to be presented but that's okay.
Okay. So this algorithm can be distributed. So that's the good news. This
algorithm is an online algorithm. So I don't need to know the system or statistics
or anything. If it is available, it can be used. But it can nevertheless be an online
algorithm. I don't need to know the statistics of anything else. It's just a sample
algorithm. This algorithm can clearly trade off accuracy for speed because if my
number of samples is very small I cannot give good guarantees for my
performance.
But my speedup will be large, for two reasons. First I'm doing less computation
and second traffic is reduced there. I'm reporting less amount of data over the
network. If network bottleneck is the issue even that bottleneck is widened and
this algorithm, because of its randomized nature, it's robust in all failures.
So here is the main result of the algorithm. Now, I typically don't like to put
equations on my slides. But given this is MSR and given this expression actually
tells us something, let's go over this expression.
So it says that under our algorithm, if R, the number of samples are big are
greater than this one by epsilon square log of 2 by delta times some constant,
then we get a probability approximately correct answer.
That is the estimate of your mean, that is mu hat, minus mu, divided by mu, the
probability that this exceeds a given constant epsilon is no more than delta.
And this constant depends upon how regular your underlying sequence is, the
number of finite moments of the given sequence. So we're relating the
probability of successful probability approximately correct answer to the number
of moments of this underlying sequence.
That is, if this sequence is extremely regular, suppose it has all finite moments,
like it's an exponential distribution, then this constant will be small.
If the number of finite moments is small, like three or four, then this constant will
be larger. Nevertheless, it's the right asymptotic scaling in the following way.
Suppose I want to estimate the mean of an unknown distribution, there's an
unknown distribution from which I can draw samples and I want such probability
approximately correct estimate of the mean.
Then how many samples do I need from this? We know from Chernof bounds
that the number of samples you need scales approximately like one by epsilon
square log of one by delta and this expression matches almost exactly with this
expression.
So what this says is the algorithm that you are presenting, although it's nice and
simple and runny algorithm, beyond constant factors it's essentially
unimprovable. That if you can improve this algorithm, you can improve a classic
bound random variable like the Chernoff bound, which itself would be a big
result.
So this is essentially the best that you can do. So this is the algorithm has the
correct asymptotic scaling. So, so far so good. We have this randomized
averaging problem and we have come up with an algorithm that looks like it's
giving the right asymptotic scaling.
So how can we use this in a datacenter network? So this is what a large
datacenter network looks. Conceptually it's a big network of thousands of
interconnected servers and disks and the queries come in, the queries or jobs
come in from outside. They get processed by one or typically more of these
servers and they're reported, the results are reported out.
These are just some canonical situations for anybody who needs to process
large volumes of data like terabites or petabytes of data, like Microsoft, Google,
anybody.
So one key feature of these asymptotes is that your job needs to be processed in
a certain way. Because you're sensitive to delays and you want to make sure
when the system is underloaded the response times are good in a networking
setup.
So some of the questions in this datacenter are one natural question is what's the
topology that's right for these servers, because as you might be aware the
servers come in form of racks. Interrack communication is much slower than
intrarack communication. And so what is the right topology from a networking
point of view. And so that's definitely one problem that I'd like you to understand
more.
But for the purpose of this talk we are going to focus on what all functions can
compute in this framework when I have a datacenter. And the particular
computational framework that has emerged over the years is Map-Reduce, which
is a distributed computation framework.
It's a very simple abstraction of the kind of computations you can do here. So
there are two functions, function called map and function called reduce,
user-defined functions. Once these functions are specified, the black box takes
care of the rest. It works well with thousands of servers, extremely scaleable.
And according to some of these rather old numbers Google was processing
something like 20 petabytes of data per day. This is January 2008 -- in
May 2011 this must be closer to, what, 50, I would guess.
So Map-Reduce is here to stay and we want to understand what all Map-Reduce
can do. If we can speed up at least some applications that use Map-Reduce,
then that would be a good thing to do.
This is Map-Reduce. There's two parts of computation. There's map part and
reduce part. And these are user-defined functions. The map part is basically the
number crunching part. You divide the input, give it to your slave servers, and
these slave servers will do computations based upon the map function.
The reduce function is a simple combination of your outputs of the slave servers,
of the measured outputs, and it's typically a counting operation like summation or
simple operations that just combine the data.
Map-Reduce can do other operations that are not distributed like you can do
medium computation or person by computation, but the majority of the
applications are summation-based and even if there are these other applications
as we're going to see later, although the algorithm has not been analyzed for
that, the algorithm can be extended and you could get results.
This is Map-Reduce. Let's go over a simple example of Map-Reduce. Suppose
the search query is, let's say, it's Federal. And as a first, the zero level
approximation as how important the keyword is. Want to count the number of
occurrences of this given keyword in this large dataset.
There are these files and contents that are given to me. The map function is
going to just read each one of these files and count the number of occurrences
and the reduce function is going to add, let's introduce some technology, there's
a key called federal and it's going to add all the values, the inter measured
outputs corresponding to this particular key federal.
Simple operation. This is the natural thing you would do if you want to distribute
your operation. And this reduce function typically outputs just 0 or 1 values, and
it does simple operations like summation.
So let's get back to our problem of autocomplete suggestions here. So I have
started typing with F. And these are the 11 results according to Bing, this is a
snapshot.
So how do we come up with these suggestions fast? So this is what will happen
in this datacenter framework. And then this query starting with the F comes to
the datacenter. I want to find relative ranks of Facebook, Firefox and FedEx,
over the large query log. One way of doing this would be to compute the brute
force way, that is count the number of times Facebook was accessed and the
number of times this was accessed and so on.
But if all that I'm interested in is the related ranking then I don't need to do this
brute force computation. And coming back to the question that was asked a
couple of minutes ago. What happens if my data is extremely skewed. That's
not going to happen if the auto complete suggestions you're going to give are at
least for famous or important queries. Even if our undersample is data by 50
percent. It's believable that the related rankings are not going to change. You
can make it precise mathematically. But that is the idea. The heavier represent
even in undersampling. That's what you care for.
So this basically is a summary of Map-Reduce. What is typically used for.
Typically used for word count, or URL access count or for searching for text or
generating reverse web link graph, who all is responding to a given web page to
count its importance or the max of an array, or basically all the operations you
can do if you can count a histogram of your given input that is corresponding to
the key one, I want to know how many times this key one occurred in my data,
key two, key three, so on. And if I can compute the histogram of this then
basically any function that can use a histogram can be computed using
Map-Reduce.
And getting the histogram is basically a summation operation. I want to count the
frequency of each one of these keywords. So most of the operations under
Map-Reduce, most of the interesting operations fall under this category where
reduce function is just a simple combination, which is just a summation. That's
what brings us to the motivation that fast averaging can clearly benefit this kind of
Map-Reduce setup.
Here's what's going to happen in the datacenter. When a query comes in from
outside, there is a central server. This will do the reduce operation. The central
server has a bunch of slave servers connected to it. There's these files F1, F2,
F3, FN and these numbers are query-specific numbers.
So suppose my query was federal. This file F1, number X1 is the number of
times that the query came in in this particular file. And file 2 and file 3 and so on.
Typically what happens is the number of servers is much fewer than the number
of files. Think of asymptotic scaling of M to be like square root of N. So we have
a thousand servers and million files or even more. You're interested in is the
number of sums or the summation of XIs or the average of XIs equivalently. So
let's see how this algorithm, the proposed algorithm for approximate computation
in a Map-Reduce setup will work.
Query comes into a slave server. This slave server is going to decide randomly
out of these N files which files am I going to sample. It's going to sample in this
case file 2 and file N. Then there's no approximation beyond that point.
It's going to exactly compute the numbers X2 and XN and it's going to report to
the central server these two numbers, X2 plus XN, that's the total weight of my
contribution, and the number of files that I sampled. That is two.
And the central server is going to sum up the -- as mentioned before, is going to
sum up XIYI and divide that by summation YI, the number of files you sample,
the total weight that you assigned. And that is going to be your approximate
estimate of the mean.
So how does this work? Why do we expect this to work? So as mentioned
before, if the underlying sequence is regular then the earlier result applies and
you can get away with far fewer samples than shat is given to you. This makes
intuitive sense. Suppose I'm given a uniform random variable between let's say
0 to 1. If I'm given million samples of this, I can say, okay, I don't need a million
samples, maybe I can get away with a thousand samples and estimate a mean.
And that mean is not going to be too far away from what I would get if I just did
the brute force computation for the mean for a million numbers.
So because these numbers are coming from, for famous queries like Facebook,
so the heavy hitters well represented, even under the undersampling. So if all
we're interested in is the relative ranking of the important results, then it can be
quickly computed.
So far we had looked at -- by the way, so were there any questions so far?
>>: I had one question. X1 to XN, that N number is usually not no when you
process those files. How do you distribute those files, the files and server?
>> Shreeshankar Bodas: Sorry, so these ->>: Is that what you have this number, right?
>> Shreeshankar Bodas: Yeah.
>>: When you are processing your data, do you know what N is?
>> Shreeshankar Bodas: Yeah.
>>: That way how do you distribute those files to different servers say, okay, you
have N and you have N weight different words, for example, and then you use M
machines to process and you choose N files, because any is unknown. How do
you do that?
>> Shreeshankar Bodas: Is the concern that this N is not known?
>>: Yes. Because the problem is when you have a lot of files you don't know
how many is taken the values that you have.
>>: You mean the number of keywords I have is not known?
>>: Yeah.
>> Shreeshankar Bodas: So this is, I would say this is a module for computing
how many keywords I have. So it's like this, so this is an example that I've
shown as for one particular keyword.
But in practice I understand your concern that the keywords themselves are not
known and related frequencies are not known. So I want the following.
I want to -- so it's like this. There are two keywords, let's say keyword one and
keyword two. There could be many more. I want to count the related
frequencies of these keywords, right? So this is for a particular query.
So when -- so the idea is that you just randomly sample a subset of the files and
compute the histogram. It's an approximation to the histogram. So you sample a
bunch of files, you'll get an approximate word count or approximate count for the
number of hits for the given query, and you do this in a distributed fashion on
each of the slave servers and merge those histograms.
So because each one of those histograms is approximately right, at least for the
heavy hitters, when you merge the result, it remains approximately right. So
although you don't know the keywords to begin with, you don't know what you're
looking for, you compute the approximate histogram.
>>: Could you approximate the stage, first stage was and is [phonetic] and under
second stage it could do that, an operation.
>>: Those files, N is the number of logs.
>>: So N is logs. So XN is keyword.
>> Shreeshankar Bodas: XN is the number of occurrences of my query in the
given file log. These are not files on the Internet or anything.
>>: Number of keywords.
>> Shreeshankar Bodas: Number of keywords in the file. That's right.
>>: Number of keywords in the file.
>> Shreeshankar Bodas: Number of keywords corresponding to my interest.
>>: I just don't understand how that -- and N is known.
>> Shreeshankar Bodas: N is known. Sorry for the confusion, was that a
question?
>>: Should that be a vector?
>>: I have a question. This sampling technology, clearly statistics, some of the
statistics is relatively stable. Some of the statistics it is not, basically let's just say
average salary, I think.
>> Shreeshankar Bodas: Yeah.
>>: Average figure is actually very sensitive how you sample it, very few people,
the accountant in the file average is going to be significant swing, but if you
come ->> Shreeshankar Bodas: I understand. So even going back to the motivation.
So does your application allow for an approximate computation is one question?
>>: Many of the complex queries at least when you just do the query itself, it may
not be easy to distinguish how this query is sensitive to that. I mean, let's say
give you quite complex two queries, you submit, how do I know whether this
query, I mean, basically when I use the sampling technology, how accurate my
curve ->>Shreeshankar Bodas: I see what you're saying. So if you want to compute
the mean of a distribution, which is like heavy tail, where there are a few people
that have huge values and there are a lot of people that have small values, then
you want to make sure that these people with huge values or huge salaries are
sampled. Otherwise, you're going to be extremely totally off your target. So
that's true.
That's a property of your underlying even numbers, and it's a mathematical
problem. You can't deal with it, because so what is going to happen in that case
is that your underlying sequence is not regular enough. That is, it does not have
finite second or third moments if you were to extend the distribution.
So that's true. Your underlying sequence has to have certain regularity for these
approximate computations to work.
>>: Here you can kind of expect X to be all identical, very close to the numbers.
The lead could be --
>> Shreeshankar Bodas: The range could be small. And so what happens is if
the typical values of XIs is let's say 100 and the range is let's say plus or minus
10 or 20 or even more, but true if the XIs are close to their mean then only you
can estimate their mean by some sparse sampling.
>>: By one the log of yesterday, the previous day.
>> Shreeshankar Bodas: And XIs are not ->>: The XIs, a fraction of fairly [inaudible] not using.
>> Shreeshankar Bodas: Yes, but if this log is from 2000, then the distributive ->>: XIs is not going to be heavy tailed.
>> Shreeshankar Bodas: That's true. Distribution of XIs is not going to be heavy
tailed for the applications we have in mind. But it could be heavy tailed, in which
case you are to be careful while applying the result. That's true. That is the
assumption here that not everything can be computed in this way, but there are
certain things that you can speed up. Yes?
>>: Do you sample the servers, the site machines?
>> Shreeshankar Bodas: Yes. So every machine samples one of the files with a
Bernoulli coin toss.
>>: Every machine?
>> Shreeshankar Bodas: Yes, three machine samples every file with a Bernoulli
coin toss. Suppose there's ten servers and 100 files, I'll have ten times 100 coin
tosses, and that's going to determine who samples which file.
>>: [inaudible] this is just a picture of one slave.
>> Shreeshankar Bodas: This is a picture of one slave server.
>>: You have this many files rely on this machine?
>> Shreeshankar Bodas: Yes. Yes?
>>: Since the query that you talk about, the crosslink nature they have an arc
max at the end like a [inaudible] case you want the averages but you want R
max, if it's helping you a lot because it's orienting you toward the head of the
distribution, do you think you could somehow abstract out, sort of talk about the
second layer of what are the types of queries for which you'll have these nice
physical properties versus others, if you're doing min obviously you're screwed
because [inaudible] is there any way to organize the space of queries to think of
classes that would be well served so you could analyze each class?
>> Shreeshankar Bodas: So that's a good question. So I would say that -clearly this heavy hitter you had to have heavy hitters other it's not going to work,
approximation for auto complete for these kind of things that are not sensitive to
some one or two people skewing the distribution.
These kind of queries were off the top of my head, let's see, so if I want to
compute the page rank, then I want to count the number of incoming links as a
basic building block. Then suppose I take a Web page like New York Times,
then if I approximately compute the number of incoming links, even that is going
to be good enough because it's going to be like millions of links pointing to that.
If somebody does not have enough links pointing, then even if I do an exact
computation that person is not important, that person or side is not important. So
maybe for these applications it can be used.
Was there another question? Okay. So so far we had seen this approximate
computation in a static setup, that is, we're going to briefly go over what happens
if there are system dynamics, that is, so far what we have talked about is all the
servers are empty.
I just want to see what happens if I submit a given query and what happens to its
completion time and how many samples do I need. It could happen that these
servers already are loaded by some of these queries.
That is, I have this yellow query coming in from outside and it brings in a certain
amount of work for each one of these servers because corresponding to this
server this server is going to sample a subset of files and server two is going to
sample a subset of the files and so on. It's like there's a Q in front of these
servers. And every incoming query is going to bring in some amount of work and
these servers could have earlier query that are not yet complete. Suppose this is
the amount of residue that is there from the previous work.
So this query comes in. It brings in two units of or two packets into the first
queue and so. These files, as I mentioned, are the search query logs. So what
happens in this dynamic setup is our system, first of all, stable? Does it give
good qeueing performance, good delivery performance, and how can we extend
the algorithm or how can we even analyze it in a dynamic setup. So there is one
simple way of bounding a response time that looks nevertheless useful in this
case.
So let's assume that I'm going to slightly modify the system and I'm going to
batch the incoming queries into units of time T. So these are some of the green
queries and the blue queries and so on.
And I'm going to give precisely T units of time for each batch of this T queries.
So remember that because there is underlying randomness in the algorithm,
although the number of queries per unit time, even if that is constant, I'll have a
random amount of workload that comes into the system.
I'm going to say that, okay, you are given some T units of time for your overall
process to complete. If there is some amount of work that is left in the system,
then I'm going to just terminate all that work and start processing the new
queries. So this brings in two points. First of all, it gives deterministic delay
guarantees for response time, but this might additionally introduce some errors in
your estimate because of your sampling, because of this termination event.
So it turns out that even this system can be analyzed in more or less closed form.
So you can get expressions like probability of error. In the earlier case this was a
fixed number. Now this is a function of your epsilon, the conference you want.
E, the amount of time you give. And the sampling parameter, P. And this
expression can be, this function F can be computed in closed form but I'm not
putting it up here. But the main message here is that you can trade off accuracy
response time and confidence. If you put T equal to infinity, then you get the
earlier result back. But even if you're constrained by how much time you can
afford to give for a particular job, you can analyze the trade-off between your
confidence and your response time.
>>: Would the [inaudible] norm for example come into ->>: Right. So that's a good question. So what happened here is because we are
doing this binomial kind of sampling, because binomial random variable sharply
concentrates around invariants same order so there is not much variation here.
But that's true if your sampling is a bursty [phonetic] scheme, if it's not just
Bernoulli coin toss, if it's more complicated sampling the burstiness of your arrival
process will determine how your function F changes.
>>: I understand this logic of T. If one is the server did sample from ability based
on its role what link ->>: That's a [inaudible] sampling you're saying?
>>: Each or some, that's right, with your distribution. So the server one just picks
two files, server, because it's [inaudible] and two is lightly loaded five files.
>> Shreeshankar Bodas: That's a good point. So that will even out the lure in a
more better way. That will give a sharper -- that will give us more of a function of
our agree. But we have not analyzed that. I should think about the sampling
strategy. Okay.
So this is what we have seen so far. So Map-Reduce is the sample distributed
computing framework where for most of the cases of interest this reduce function
is at least in the exact form the reduce function at summation. We believe this
summation can be approximated at least in some ways that gives us, first of all,
network condition benefits and also completion time benefits, and there are
simple randomized algorithms for mean computations that are robust and fast
and that can trade off your accuracy and response time and confidence.
Again, not everything can be spread out this way; like if you are extremely
sensitive to your estimation error, then this kind of approach is not the right one.
But there are certain applications for which this kind of randomized calculation of
the mean is good enough and it saves you computation time.
So let's see if this area of mean computation can be extended beyond these
Map-Reduce type of math computations. There's a whole community that
worries about consensus on graph. The graph you can think of it as a
communication graph. These could be let's say sensors in a field and these XIs
are the numbers. Each XI's a real number every node has a real number. You
want to compute summation XI at each of these nodes. So why is this problem
interesting, why do we care for this problem? It could be, for example, these are
some unmanned vehicles or robots that are moving in a certain direction and
they just want to compute a consensus; that is, I'm going in this particular
direction.
Or this consensus can also be used in CSMI type applications where this is a
black box for doing more complicated tooling operations, or these could be, as I
mentioned this could be sensors in a field and you want to count, measure the
temperature.
The underlying feature is that these XIs in at least the sensor network setup
these could be correlated to each other that the temperatures in a given room are
not going to be that different from each other, at least we can expect that.
Nevertheless, if that assumption is valid, then you can use these ideas of
approximate mean computation to quickly count them or quickly measure the
average of XIs.
So the idea is simple. We know that I can take random walk on this graph
according to this graph structure, that is, an edge represents presence that if I'm
in this, on this node of times 0 I can go to one of its neighbor at time 1. So I'm
going to take a random walk on this graph, and I'm going to have a token with
me. This token is going to sample the values of XIs as and when it goes to those
nodes and not all the time but let's say every T mix times where T is the mixing
time of the underlying Markov chain.
I'm going to take a random walk. And if the underlying Markov chain is nice and
carefully constructed, in particular if it has a uniform stationary distribution, then
I'm sampling from that distribution of XIs. And there is a main idea here that you
have this token, it's just going to take a random walk here. It's as it is passing on
the graph, it's going to collect samples and it's going to average them.
We have some theoretical results even for this part. You can show for certain
structures of the graph like ring graph this approximate computation of mean can
improve on the earlier known bounds but, of course, you have to assume that the
underlying XIs have certain distribution, it's not completely arbitrary numbers. So
this is just one small application that showed that these results are, these ideas
of approximate mean computation can also be used in other fields.
So this is what I'd like to work on in the future as regards to the Map-Reduce
computations. So one thing and I'm sure already all of you have asked this
question, what happens on the real data; that is, it's all okay if it works in theory.
Does it work in practice, and that's one thing that I'll be focusing on.
Another question is the smart randomized algorithm. This is the basic vanilla
version of the algorithm where it's all uniform sampling, maybe you want to do
smarter things because there are network bottlenecks or because it's
heterogeneous environment where server speeds are inherently unequal or there
could be other smarter things you can do to save on sampling. So these are a
couple of directions that I want to take in the future.
So here is the summary of what we have talked about so far: The main message
here is that if the underlying sequence was regular, then mean computation is an
easy problem.
Your numbers are nicely concentrated around the mean. Just sample a few of
them and you'll get a good estimate of the mean. This idea can be used in a
datacenter framework, in a Map-Reduce setup, where this Map-Reduce uses
simple mean computation, simple mathematical operations like averaging, that
have this feature that reduce function is a simple summation and then that can
be spread up.
And this randomized algorithm for fast averaging can give you a trade-off
between accuracy and completion time, which is one direction in which, well, at
least the trade-off as far as IO has not been studied, it's a one more dimension to
your toolbox.
So that's it. Thank you. If you have any questions I'll be glad to answer.
>>: If your canonical versions are essentially 01 variables it seems to be
standard random sampling, that's kind of made famous.
>> Shreeshankar Bodas: It is. It is exactly that, but your XIs -- exactly. If the XIs
are just coming from 0, 1 or any finite support, then given lots of samples it's not
going to benefit my -- there's just a diminishing return to how much accurate you
can get the estimate, that's true, exactly.
We're not doing something that is mathematically heavyweight or anything, it's
just a nice observation that can be used in the setups to reduce the amount of
completion time. Any other questions? All right, then, thank you.
[applause]