>> Dengyong Zhou: I am glad to have Ran Gilad-Bachrach speaking today. Ran is a researcher in the machine learning group at Microsoft Research. Before joining the department here he was an applied researcher at Bing. Ran has a PhD from Hebrew University in computer science. >> Ran Gilad-Bachrach: Thank you Denny. I'm going to be talking today about the Median Hypothesis. This is joint work with Chris Burges and to lay the groundwork I'm going to discuss a Pac Bayesian point of view about learning. We will need that to define the median and actually we will find out that will need also to define a depth function by which we will define the median. Once we have that, we will provide generalization bounds using the median hypothesis which is actually what we will call deep hypothesis. So this will be the first part of the talk, and in the second part we will start discussing algorithms. So the first one would be how do we measure the depth of a hypothesis and then we will discuss how do we find the deepest hypothesis which is actually the median. So this is kind of the outline of the talk So let's start by looking at the Pac Bayesian point of view on learning. Many learning algorithms look kind of like that. We have some sort of scoring function. I will call it the energy function. We give a score for every hypothesis based on some regularization term; pick your favorite, L2, L1, whatever; and a loss function on some observation that he had. From Pac Bayesian point of view we can look at it in the following way. We can say that actually the regularization term actually represents some prior belief that we had. So think about the regularization term as the log likelihood of the prior belief that we had. And then given the observation we can generate a posterior belief, and this is what we use then to select the hypothesis and make a prediction. It's very important to note that these are beliefs. So it doesn't mean that we assume that something in the real world really behaves according to these probability measures or distributions or whatever you call them. This is just a representation of our internal beliefs on what is going on out there. So from kind of the way that I would like you to view it is that the process of learning is made of three stages. We begin with some prior about the world and then we see some evidence. For example, it could be label, the nonlabeled examples from which we generate the posterior. And then once we have this posterior belief, we select the hypothesis with which we will go ahead and make our forecast and our predictions. My talk today is going to focus on this part. Once we have our posterior belief, what hypothesis should we choose to use to make our predictions. Let's look at some common methods of selecting the hypothesis once we have our posterior, so one option is to use the maximal posterior. This is actually what SVM and LASSO do; actually minimizing the energy function is equivalent in my language to selecting the maximal posterior. Another option is to use the Bayes function, which means that every time I want to make a prediction, I will just hold the vote among all hypotheses. I will do weighted voting and make my prediction. I can use Gibbs sampling which means just sample at random a hypothesis according to your belief. And I can use in ensembles which are kind of an extension of the Gibbs sampling, which is instead of just selecting one random hypothesis, I will select several and just hold the voting between them. So I would like to compare these methods, but this is very qualitative comparison and to make sure that you don't think that there is anything accurate about it, I drew it by hand [laughter] to emphasize that. And I went to look at to axes, one is the runtime. So when I want to make a prediction, how complicated is it to make this prediction. And the second axis is accuracy but think about accuracy more as how well does it capture my beliefs. So I had the posterior belief. Does the hypothesis, how well does it capture all of the structure of my beliefs. So let's look at some examples. If we look at the Gibbs method which is just I got my belief, I selected a random hypothesis and then I am just going to use it to make predictions. So the runtime is really easy. I just need to evaluate a single hypothesis and see how well it works. But does it capture all of the complexity of my belief? To a very small extent I would say. If we look at the maximal posterior, it seems to capture more, but still it just looks at the peak. My belief can change dramatically and still the peak, as long as it remains in the same place, I would pick the same one. So it doesn't seem to capture all that I have learned in the learning process. On the other hand, if we look at the Bayes classifier it uses a lot of information about my belief, but in terms of runtime, I really need to hold a vote every time I want to make a prediction. Ensemble method provides kind of a trade-off. If I use an ensemble of just one hypothesis, then this is just a Gibbs sampling and so it is going to be fast but not very accurate. Whereas as I get my ensemble to grow larger, then I capture more structure for my beliefs, but it becomes lower. >>: You are making some assumptions. >> Ran Gilad-Bachrach: Of course. >>: For example, for linear [inaudible] classes if you're just [inaudible] linear classifiers, then you can show that the Bayes optimum is also an accepted linear classifier. So somehow the fact that the Bayes… >> Ran Gilad-Bachrach: No, the Bayes is not--if you are making a binary classification, so it is true that if you--it is not true that the Bayes, if you take the Bayes classifier over linear classifier is a linear classifier. It is simply not true. >>: That depends on what your distributions are. When they do the general--when they use the--when you have Bayesian generalization bounds for SVM, then in that state it turns out that the Bayes classifier is in fact very fast. >> Ran Gilad-Bachrach: Again, this is very--this is why I didn't, why I drew it by hand. It is, think about the concept, not about the very specific cases in which things might be different. Generally speaking, computing the Bayes classifier is hard. Although there are examples in which it is easy, generally speaking for general class with general distribution, this tends to be hard. Okay? >>: So another question. If I think about this from a Bayesian respect here, there are two types of uncertainty. There is model uncertainty and there is parameter uncertainty. So it sounds like you are talking about just model uncertainty in this schematic or diagram, is that right? >> Ran Gilad-Bachrach: Again, I am isolating all the problem of estimating the parameters and everything. I had my prior belief and I saw some evidence, and this is currently my belief. This is my belief and this is it. But now I am asking myself okay given that this is my belief, how should I make predictions? >>: So you are thinking about distribution to the model classes and the parameters for the [inaudible]. [inaudible] both the model and parameters. >> Ran Gilad-Bachrach: Exactly. >>: Okay thanks. >> Ran Gilad-Bachrach: What I'm about to propose is a new way to select a hypothesis from my belief which is to use the median. Now the median is going to be just a single hypothesis so in terms of runtime it is going to be equivalent to the Gibbs and the maximal posterior, but I will select this hypothesis that captures as much as possible information about my belief. So this is where we are heading to. Okay, is it…? So now I want to start defining what it is I mean by median, but it turns out in order to define a median we will need to define something which we will call a depth, a depth function. So let's start with just the case that we all know about the one-dimensional univariate median. So assume while I have this sample here of points and I am going to find a median of these points. One way to look at it is to say the following; every point kind of splits the sample into two parts, all the points that are to the left and all of the points that are to the right of this point. And we associate depth with each point being the size of the smaller part. For example, for this point it had the depth three because here on this side there are only three points, where this point has depth one because if I split the space only this point is on this side. Now given this definition then this point is the deepest point and it is the median. Once we define the median via a depth function, we can now move to multivariate cases. >>: [inaudible] so your points [inaudible]. >> Ran Gilad-Bachrach: Sorry? >>: [inaudible] so your points [inaudible] distance [inaudible]. >> Ran Gilad-Bachrach: No. I do not consider distance; just count the number. Let's try to see if we can extend this definition to the multivariate case. When we moved to the multivariate case this is called the Tukey depth function and the Tukey median. So how do we find it in the multivariate case? So I assume again that I have this sample but now it is two-dimensional, and I want to measure the depth of every point. So what I do is I look at hyper planes. So if I want to compute the depth of this point, I look at all the hyper planes that go through this point. Every hyperplane split the sample into two, and I take the smaller part of it. I take the minimum over all hyper planes, and this would be the depth of this point. So for example, you can see that for this point, the depth would be two because every hyperplane will have this point and at least one additional point with it. Whereas, for this point the depth would be three because every hyperplane will have at least this point and two other points with it. So this point is deeper than that point. And again, once we have a depth function, automatically we get a median associated with it which is just the deepest point. A word of advice is you have to know that in the multivariate case actually there are many definitions of multivariate medians. So we can define a different definition of a depth function; each one of them would lead to different median. >>: [inaudible] example, if your points are in a convex shape, then all of them are medians. >> Ran Gilad-Bachrach: No. >>: So then you're saying--no. Because all of them would be [inaudible] the point one. >> Ran Gilad-Bachrach: Remember that the median does not have to be a sample point. So if I have points on a circle, so let me draw that. I have points on a circle, right? This is my sample. So you are right to say, you are correct that every one of these points has a depth of one. But actually, that point is much deeper. >>: Oh, so it's any… >> Ran Gilad-Bachrach: I can compute to any, for any point in space. And in this case you can actually see that the median is actually the center. So it is what you expect it to be. >>: Well I think that you [inaudible] to compute [inaudible]. What if you can't compute function in the [inaudible]? >> Ran Gilad-Bachrach: There are depth functions that are defined only on your sample and there are depth functions that are defined everywhere. There are different definitions. Each one of them has its pros and cons. So I don't--it is an interesting topic. It is a very interesting topic but I will have to skip on all of that. So I just wanted to point out the fact that this is not the only way to define a multivariate median. But this is the one that I will be inspired by, so this is why I brought it up and not other possible definitions. >>: [inaudible]. >> Ran Gilad-Bachrach: For example, like the univariate case, there could be more than one median. You know it from the univariate case that the median is not a single point. It could be--but here you could show for example that the level sets of the depth functions are convex. Therefore if you have more than one median, it will always be a convex set of points which are the median. So you can show all sorts of things. So there are a lot of studies on multivariate median and just as an anecdote I will tell you that about 10 years ago was the first time I encountered that, and I thought that I invented the multivariate median, because I searched for multidimensional median and when you search for multidimensional median I couldn't find anything. And I thought, you know, this is my big thing, right? And then I don't know--I know why. A friend told me, you know that in statistics they don't call it multidimensional; they call it multivariate. [laughter]. Yeah, so this is why I brought this just to mention the fact that there is a lot of literature, a lot of different definitions, and unlike the univariate case which when you say the median, we all know what you're talking about, when you go to the multivariate case, this is no longer true. But now I am motivated by this very specific definition of the median that we just saw. I am going to define a median for function classes. One thing that is important to note is unlike what we discussed before in which we thought of the sample of points and we wanted to find the median of these points, now we are talking about functions. We are not going to find the median of our sample, but the median of our hypothesis. Just a little bit of notation that we are going to use, our sample space is going to be, I am going to use the letter X for that and the function class is F and the most important notation is Q which is our belief, our posterior belief. This is the distribution over our sample class. And here is how we define the depth in this case. First I have a function that I want to measure its depth. So first of all I take an instance and define the depth of this function with respect to this instance X to be the probability of a random function agreeing with a label of F that F assigns to X. Now I define the depths of F to be just the infimum over all instances. It is similar to what we did in the Tukey depth thing in which we said every hyperplane defines some sort of a depth, because it splits a space into two and I take the smaller part and then I take the infimum over all hyper planes. This is what we do over here. >>: In the function family here like any, does it have to be [inaudible] space? >> Ran Gilad-Bachrach: I can compute the depth of the function which is not in my function class, for example. It is defined for every function. Once we have a definition of a depth function, we have a median, which is just the deepest point. >>: So in the previous slide you said that the range of F is just minus one, plus one, right? So there are just two depths. >> Ran Gilad-Bachrach: No. For every, again for every X there could be… >>: Oh, X is either one or minus one. >> Ran Gilad-Bachrach: Yes. But then I take that infimum of all Xs. >>: Wait a minute. For a given X, there are just two types of X, one that are deep and one that are shallow. >> Ran Gilad-Bachrach: Exactly. >>: Okay. >>: Wait, wait, there is a probability that when you draw F [inaudible] at random Q. >> Ran Gilad-Bachrach: Yes, what [inaudible] says is that if I have for a certain point, and we will see it in a second, for a second point, 80% of the functions say that the label is +1, and 20% won't say that. So either you are in this class or in that class. >>: There are two sums; there are two terms in this [inaudible]. >> Ran Gilad-Bachrach: Yes, but once you take the infimum, and then this function becomes continuous. You are absolutely correct. Now… >>: Wait [laughter] so the infimum is over all of X regardless of [inaudible] if there is an X there is no probability of this ever occurring then you still… >> Ran Gilad-Bachrach: With your question, wait five minutes. >>: So the [inaudible] is the amount support that [inaudible] were the least popular [inaudible]. >> Ran Gilad-Bachrach: Exactly. So for example, you can see that if I compute the depth of the Bayes classifier, it will always be at least half. Do you see that? Because the Bayes classifier always takes the majority vote. So for every X at least 50% of the population will agree with that and therefore the infimum has to be greater than half, right? This is an example to keep in mind. A depth of half is the best we can hope for in any realistic setting. And the question would be how close to half can we get. I will not go over all the details here, but we can actually see that the Tukey depth that we discussed before is actually a special case of this definition, and so when the hypothesis class which is really some sort of linear classifiers, then actually the Tukey depth actually becomes you know a special case of this definition. But there are some interesting things about the Tukey depth. For example, we know that the Tukey depth if we work on D dimensional space, there is always a point with depth greater than 1 over D +1, regardless of the distribution. There is always a point with depths 1 over D +1. We know further, that if that distribution Q is log concave, then there is always a point with depths 1 over E, regardless of the dimension. Now1over E is very good. Remember that the optimum we hope for is one half, so 0 over E is very close to that. >>: So the Bayes point, isn't always the Bayes function isn't always in this space F, is that what you're saying? >> Ran Gilad-Bachrach: No. It doesn't have to be in space F. >>: You're saying [inaudible]. [multiple speakers]. [inaudible]. >>: [inaudible] realizable so you might not be able to realize in the space of the measure Q that you can get a half [inaudible]. >> Ran Gilad-Bachrach: If we go back to the beginning of our discussion and we said many learning algorithms look like that, and the energy function looks like that, and we actually like the loss function to be like convex and regularization turn to be convex, which means that the energy function is convex, which means that Q is log concave. So actually in many of the cases which we really work with, we are exactly in the setting where our posterior belief is log concave, and therefore we are guaranteed to have a deep function, a function with a depth of at least 1 over E. So I hope you get some intuition with respect to the median and the depth and what does it mean, and now I would like to convince you that the median hypothesis is actually good. But before I do that, a warning, don't always use the median. [laughter]. But in our case, it is actually good to use the median and so here are the first results. So let's say we have some target. So the word is distributed with some distribution U that I don't know about it. And say we will just use our R nu of F is just a generalization error of F. So we can prove that for every function F its generalization error is bounded by one over its depth time the expected generalization error of if I would have just sampled hypothesis from Q. And the proof is trivial. The proof, we have to consider just two cases. Pick a point. There could be two things. If the majority vote is done with large majority, then my function since it is deep must agree with the majority. And therefore my risk, my loss on this would be very similar to the risk of just a random hypothesis. The other case is in which Q is not decisive. In which case if Q is not decisive my competition, which is just a random hypothesis, would err a lot regardless of what it proved, and therefore, I can't do much worse than that. So again, this is kind of vaguely speaking, but when you write it down it is just two lines and you get the proof. The nice thing about it is that we can connect that with more traditional Pac Bayesian analysis. So we had just about a month ago we had Mario Marchand over here and what he showed is something of this type of a theorem and trying to clear things around. Basically what it says is that with large probability, that generalization error of just selecting a hypothesis at random, the expected error of a random hypothesis is bounded by a term which is made of two things. One is it is training error and the other one is kind of the KL divergence between the prior and the posterior. And now we can just take this result, take the theorem that I just showed you and plug them together and we get a generalization bound that says that for every hypothesis, its generalization bound is bounded by 1 over its depth times the same term that we had before. But now there is something really annoying about our definition and [inaudible] asked about it. We looked at the infimum, when we defined the depth, we looked at the infimum and this seems to be kind of very harsh. It could be that we have a function that agreeing with the majority on almost all of the points, but say one point. So it seems not the right way to look at it. And therefore we can define a relaxed version of the depth in which we say instead of taking the infimum, we say okay, you are allowed to put aside a proportion Delta of the points and compute the depths only on the rest of them. So when we look at this relaxed depth over here, actually what it means is that for the majority of the points your depth is greater than that, but there might be a proportion Delta in which actually you do worse than that, but that is fine. And again we can repeat the same kind of proof that we have seen before, the same kind of theorems using this relaxed version of the depth function, so we can bound the generalization error in terms of this relaxed depth function and have a Delta term over here and again we can plug it into that PAC Bayesian bounds and get a generalization bound in terms of this relaxed depth function. >>: [inaudible]? >> Ran Gilad-Bachrach: Uhh. >>: Or we can choose [inaudible] whatever I want? >> Ran Gilad-Bachrach: I think you can, but I need to verify that. I think it does. I hope that by now I have motivated you that finding a deep hypothesis is something worthwhile and now with looking for, to find a deep hypothesis and the first thing is actually to be able to measure the depth of a hypothesis, so if someone comes with a hypothesis can I know what its depth is. And we want more than just estimating its depth; we want a uniform estimation, meaning that if I have a class of function, I want to be able to estimate the depths of all of them simultaneously and make sure that my estimate holds for all of them. So here's what we're going to do. Assume I have a bunch of points. As we discussed earlier, for each point I can look at what is the size of the population that labels it plus 1 and what is the size of the population that labels it minus 1? So say this is my, these are my sample points. And now I want to evaluate a special function. So this function for the first point it provides the label plus 1. I know that it depths on this point is 20%, because only 20% of my functions I agree that the label here is plus 1. And then I can take the label on the second point and again see what is the size of the population that agrees with it on the second point. And so on, and so forth I can go over all of the points in each one of them mark what is the depths with respect to this first specific point. Eventually the depth is just a minimum over that. So in this case on this point, I had the smallest agreement and therefore the depth of this function is .2. And this is what we are actually going to do. What we are going to do is instead of computing the infimum over all points, we are just going to take a sample and evaluate the depth only on this sample, and when we want to evaluate what is the agreement here, again, we will take a sample functions and use them to evaluate what is the proportion of the population that is labeled +1 and what is the proportion of the population that is labeled -1. And this actually is what the algorithm does, so you take two samples. One is unlabeled samples of instances and one is samples of functions from your belief and you actually compute for every point X you compute what is a proportion of the population that agree with the function that you are trying to evaluate and you take the minimum and this is your estimate. So this is what we call the empirical depth of the function F. We can prove the following property, and again I will try to clear up all of the details here, but the main thing is that with high probability we will have the following, the depth, the empirical depth we compute is going to be greater than the true depth minus epsilon, and it is going to be not larger than the relaxed depth plus epsilon. So it is going to be somewhere in this range, but this holds with high probability holds simultaneously for all functions, so it is uniform. This is a uniform bound that holds for all functions. If we look at the probability in terms of what we mean by high probability, let's look first over here. We have here the fife function, the gross function; we see the dimension that we used to see, and we see that we have some polynomial term, here but exponentially small term over here and the same thing over here. So we can choose the sizes of the sample that we need such that the probability will be as high as we want, is close to 1 as we want. >>: So epsilon [inaudible] back to the fraction and you relaxed that and [inaudible] parameter, right? In other words you go back to the definition of relaxed depth, there's a [inaudible] you choose Delta? >> Ran Gilad-Bachrach: Delta. Here it is. >>: Okay, so that Delta is… >> Ran Gilad-Bachrach: That Delta is here, and here and… >>: Oh, I see it. Okay. >> Ran Gilad-Bachrach: And again, I'm going to skip that. And the proof is very simple. The main ingredient is to note the following, that if I have in my sample--so we have this likeness between the true depth of the function F and it's kind of relaxed depth. We have sort of a selectness here. If my sample of points consists of a point for which DF of given X is somewhere in this range, then my estimate would be smaller than the relaxed depth. And my estimate will always be greater than this depth because this is infinite. So all I need to guarantee is that my sample is actually a hitting set, which means that it hits all of these regions so I have instances in all of these regions for all of the functions. So this is called a hitting set and in the machine learning literature this is called an epsilon net, so we know that if this dimension is finite, a random sample will actually hit all of these regions with high probability. So this is actually, you know, the main ingredient in this proof. So now we know how to measure depth and we know how to measure it uniformly for all of the functions in the class, but now we would like to find the median, or approximate the median. We want to find a deep point. And this is what we are going to do now. Again, I want to start with motivation. I am going to show it by pictures and then go over the algorithms. I have a sample of points and actually I will tell you in advance what we are going to do. Instead of trying to find really the deepest point, the deepest function, we are going to find the function that maximizes the empirical depths and not the true depths. And it turns out that it is easy to find this function. Assume that I have this sample of points and on each one of them I know what is the size of the population that labels a +1 and what part of the population labels it -1. The deepest possible points will actually agree with the majority vote on all of these cases. So the Bayes classifier will give the label -1 here and +1 over here and +1 over here because the majority says +1. So if I can find a function like that, this is the best that I can hope for. But now assume that I cannot. So I can't find any function in my class that actually gives all of the correct labels in all of these cases. So I am going to eliminate one of the points from the sample, saying that I allow the function that I am looking for to kind of miss-label this point. But which point do I want to eliminate? If for example, I eliminate this point, I might get a classifier that labels here +1 and its depth is going to be .2, so this is not very favorable. >>: I have a question. In drawing these graphs, how [inaudible]? >> Ran Gilad-Bachrach: Yes. I want to eliminate the point on which there is the smallest margin in the voting, because I know that for this point, for example, for this point although 55% of the classifiers label it -1, but still 45 say it’s +1. So if I got the label +1, this is not too bad. So I can delete this point and I can say can I find a classifier which agrees with all of the remaining labels? And I can keep going, and every time I will delete the point on which there is the smallest margin. I will keep going until I find a consistent classifier and not only that, once I have found one, I know that it is actually maximizing the empirical depth, I can also know what is the empirical depth and the empirical depth is actually the size of the minority set on the last point that I deleted. Now this is basically it. So again the algorithm itself will just have, will receive two samples, one is unlabeled sample of points. The other one is sample functions for my belief distribution, and the output would be a function F which actually maximizes the empirical depths and its empirical depth. >>: [inaudible] sample point starting with the same thing [inaudible] the sample the same sample? >> Ran Gilad-Bachrach: Label sample is something that I, the only thing that I get is a posterior. You might have used--I don't know what you use to reach with this posterior. Maybe you didn't use label sample or maybe you have some Oracle that gives you some hints. I don't know. But once you have this posterior, this is what I need in order to select the median hypothesis. Does it make any sense? >>: Yes. But I am saying what bounds can you use…? >> Ran Gilad-Bachrach: I can use label sample, but it doesn't add anything. I don't use the labels. >>: What I am trying to say is having a labeled sample doesn't [inaudible]. I mean when you are training an algorithm you have several different points, so using additional unlabeled points is not adding to anything either. >> Ran Gilad-Bachrach: You're asking whether--if I trained using a labeled example, can I reuse the sample again for this part? >>: Yes. And I think you should be. >> Ran Gilad-Bachrach: You should be. You can kind of reuse the Delta term in the confidence divided by two and you should be fine. >>: You have T. You can build it on sample over T. Is it true that this ensemble will always be better than your median? >> Ran Gilad-Bachrach: I don't know. It needs to be validated. But again, this ensemble in terms of what I want to really use it, it is much heavier than having just a single hypothesis, right? >>: Is N times… >> Ran Gilad-Bachrach: N times, but on runtime. Here I use it only in training. >>: I understand. But is it true that the best you can do, the best that you can hope to do is as good as the ensemble on the [inaudible] because you are using, you are hypothesizing… >> Ran Gilad-Bachrach: It makes sense, but can I prove it? I don't know if I can… >>: The ensemble you can just take it all over zero but the median. >> Ran Gilad-Bachrach: Sir? >>: You can have the ensemble all the way to zero but 1. >>: But here is the proof. So the proof is very easy. You have Q primes which is not the posterior but it is the empirical posterior, just as the posterior is defined by this. Then the ensemble on T is the base is the base optimum on that comparable posterior. >> Ran Gilad-Bachrach: That is true, but it is not my, but then you have to prove that this empirical posterior is better than the… >>: [inaudible] average, right? [inaudible]. >>: [inaudible]. >> Ran Gilad-Bachrach: On that Q prime, but we are asking on Q, right? >>: Right. So I think the answer is that the S upper bound for this, for this a sample is better than the best upper [inaudible] that you can get, but we don't know about lower bounds. >>: So I was just going to say that in the case of the Bayes point machine is the classic one where, and you were referring to this earlier, where in fact, the actual [inaudible] is an approximation because the actual classifier that you use may not live with in the function [inaudible]. It could be at the best-known ensemble out of T does pretty well, but if you happen to have a Q prime that you are using and [inaudible] against it that had that [inaudible] function, you could do better than what was actually in the ensemble. >> Ran Gilad-Bachrach: If you can use in ensemble, yep, and if you can use the Bayes classifier, probably would be the best thing that you can use, but if you can't… >>: You know what this reminds me of? Is Rich Coran and his whatchamacallit [inaudible] the thing where he made this gigantic loaded mega-classifier, the mega ensemble to triggered; is Rich here? >>: There's Rich. >>: He called it [inaudible]. The giant enormous classifier out of everything and the kitchen sink. >>: The ensemble collection. >>: The ensemble collection, so that is sort of your F and if that is sort of going back to Chris [inaudible]'s question, what is this? Is this like a parameter space? No. This really very weird space over--you are saying that your belief is, I don't know. I am going to try [inaudible] trees and all sorts of things. And that is sort of which are giant belief domain is. And instead what you are doing is you are sort of, I predict you are sort of boosting your averaging together. He is saying you could train the little classifier using his method maybe, and that is sort of a guidance on how he would train a little classifier… >>: The problem is you have to find one guy who agrees with one of the other guys. >> Ran Gilad-Bachrach: But this is usually easier. So if you think of the task, usually when we say we want to find a classifier that minimizes the number of errors and stuff like that. This is hard, right? But finding, what I need is an algorithm. This is what I call the consistent learning algorithm and it is an algorithm that says I found a hypothesis that agrees with everything or I failed. >>: [inaudible] see something because you have zero training… >>: When you are building ensembles, you are usually building ensembles that are [inaudible] simply Bayes hypothesis. >>: Everything, every imaginable algorithm was in the ensemble. >>: The ensemble is much more expressive than any one of the individual [inaudible]. >>: Right. >>: And here Rani is saying I want to find one guy who tends to agree, let's say uses all these things… [multiple speakers]. [inaudible]. >> Ran Gilad-Bachrach: To simplify the notation, I assume that the functions here are from the same class from where I am trying to select the final hypothesis, but actually you don't have to make that assumption. It just, otherwise I have to hold two function classes and all of the discussion becomes cumbersome. But actually it could be that the ensemble is made of some stumps and then you try to build a tree. >>: [inaudible] I think you are doing is picking and… >>: So we all have to… What? >>: You're missing, if you take 1 million things and you build an ensemble over them, and you are trying to find one thing which tends to agree with that ensemble, I agree that Rani is saying that it could be like a big huge tree, but it is going to be a big, huge tree. So you are going to pay again with computational complexity because if you want a very expressive function, you're going to have to make it complicated. >>: That was the beautiful thing that Rich had done, because he [inaudible] trained all of this insanely complicated ensemble. I mean they were just enormous, I don't know 100 times lower than you would ever want to use, but you would label in fact a very large training set. That is sort of different from here where you would label a very large unlabeled set and then you could get a much larger data set. Here he is saying well, maybe what you do is you just define the largest training set that you can get zero training error on that matches the labels of the giant mega ensemble, so just a different way of creating Riche’s giant data ensemble. >>: But you got to understand that I think you're missing a very good [inaudible] but let's take it off-line. There is a problem with what you're saying. You can't eliminate the Bayes variance [inaudible]. You can't get something for nothing. >> Ran Gilad-Bachrach: Think I agree with you. If you don't have any limitations in runtime, then yeah, use your more sophisticated ensemble. Use the Bayes optimal, right? I agree with you, but if you have limitations in runtime, and you want to moderate the size of your hypothesis because you have this limitation, so this is how I propose to do that. >>: That is exactly the motivation for ensemble selection, so that… >> Ran Gilad-Bachrach: You can build during the training process; you can build this huge ensemble. And this is actually here. This is the ensemble. But now still on the training process you have to prune it. So you can look at it, I think this is what John is suggesting, you can view all of this as a pruning process. >>: I guess the motivation is little bit different, because I think… Like my perspective on, well maybe it's just rattling back, Rich’s work was like, I agree with you philosophically. It feels very similar. But he was kind of like, his small set and then this small set I can't train a really simple thing to generalize well, but a really complex algorithm that can do a good job, so I pull a ton of them and then I train, then I [inaudible] deal with that to effectively expand the size of my [inaudible] data and then I train the super classifier on it. But here it doesn't seem to be so much, that at least from the software perspective; it doesn't seem to be so much that I have too little data to work from. It is more, well, maybe not so different. But it feels like you are trying to do something about generalization, well maybe it comes down to the same thing. >> Ran Gilad-Bachrach: Basically I separated the task, the training data is just out of the picture over here. It is just out of the picture. And the same thing--in what Rich did, once you train these low models and now use them to label--you didn't add new information. You didn't add new information to this project. This is just a method by which you actually select your frontal hypothesis. So here I just instead of just putting, gluing them together, I said okay, do whatever you want to do in the first part, here is what to do on the second part. And I am just analyzing the second part. Do whatever you want to do on the first part. >>: So I am just trying to understand your algorithm a little better. How does it prevent situations like that on the board that she just drew in that [inaudible] set up sample functions none of which are particularly deep? >> Ran Gilad-Bachrach: The function that I'm going to find eventually does not have to be one with these end functions. It is not going to be a 1 of F1 to FN. It could be any function in my class. So I have this algorithm MA for which I am going to feed it with samples. And it just either tells me no I can’t; I am going to feed with label samples. >>: Labels by the… >> Ran Gilad-Bachrach: They are going to be labeled by this ensemble. And I am going to feed it samples, labeled samples and this algorithm is either going to tell me I can't find any hypothesis which is consistent with the sample or it will say here is a hypothesis which is consistent with this sample. And this hypothesis doesn't have to be from this F1 to FN. It could be anywhere. And actually, and this is, I didn't want to talk about it, but it doesn't have to be in the same class. The algorithm, what I do in, first I compute for every point, what is the proportion of classifier in my ensemble that actually that label it +1 and this is a Pi plus. And then I compute the size of the minority group on each point, and sort the points, such that the first point will have the largest minority group, and the sizes of the minority groups are actually decreasing. I compute the label off the majority vote, what is the label that I really want to have and start iterating. So the first thing is I will just put all of the points in with the labels of the majority vote and see if I can find a classifier that just classifies correctly all of them. So I will close the algorithm and see if it can find a consistent hypothesis. If it does not, I will remove the first point. The first point is the one with the largest minority, so I will remove it. And again, send it to the algorithm and I will keep going, keep going until eventually the algorithm will return a function. >>: Why not do a binary search? Why do a linear scan? It seems like you're asking for a lot. >> Ran Gilad-Bachrach: You can do that. >>: Okay, it just seems like that is log n versus n number of trainers that you have to… >> Ran Gilad-Bachrach: Actually, yeah. But it doesn't matter. >>: But you are saying knockoff half, knock off half the [inaudible]. >>: Yeah, because otherwise it is [inaudible] neural net work or whatever [inaudible]. >> Ran Gilad-Bachrach: You're right. >>: Does it guarantee that it's true? >>: Is it… >>: I mean couldn't it be the case that it's not [inaudible]? >> Ran Gilad-Bachrach: No. And eventually once the algorithm returns a function which is consistent, then I can just compute its depth using the formula here and we can prove the properties of this algorithm so first of all it will always terminate. So it could be that you will keep the leading points and end up with an empty sample. And the reason is very trivial and the last point you give it a single point with the majority that, the label that the majority gave this point. So there must be [inaudible] hypothesis that agrees with that. And we can show that the function that this algorithm returns as actually the maximizer of this empirical depth, and the depth that it computes is actually the correct depth for this function. And again, trying to clear the clouds around this formula, the important thing is the following, that the relaxed depth of the return function is at least as big as the depth, actually the depths of the median. So this is a super [inaudible] of all of the depths, so this is actually the depths of the median point -2 epsilon. So actually it does approximate the median. So before I conclude, just a few small notes. So one is we discussed in the beginning actually the motivation for the definition of the depth were the Tukey depth in the Tukey median. So now we can go back to that and say that actually the algorithm that we described is actually, if we use them on this very specific case, they approximate the Tukey depth and the Tukey median, and they are polynomial algorithm and actually this is the new result. So all of the previous results in approximating the Tukey depth and Tukey median are exponential in terms of the dimension and this is a polynomial in the dimension, but I must say that usually the approximation that they consider is different than the type of approximation that I am considering here. But nevertheless, this is something that is a new result. I am not going to go over the details here, but we can also discuss the geometrical properties of these depth functions and if it is convex in some sense, but I'm going to skip that. But this is the big but. As you know I didn't have any empirical study over here which is kind of why don't you have any, if it's good, show us that it's good. And the problem is that I made the assumption that I can sample functions from my belief. So all algorithm used, actually if we look at the algorithm that approximates the median, we needed three things as inputs. We needed a sample of unlabeled instances, which is usually easy to get. We needed a consistent algorithm, which in many cases is easy to come up with. But what is hard to come up with is just a sample of functions from my belief. This is something that for interesting belief distributions is not so easy to achieve. So we can do heuristics, you can think about begging and random force and everything like that, this is actually what they do. So we can do heuristics. We can also show that if I can sample from the true distribution, but I can sample with something that is close to the distribution, it is fine. We can correct for those mistakes. But the reason why I didn't yet do the empirical studies is once you start using the heuristics and if it works, great. If it doesn't work, then is it because all of this method is broken or is it because the heuristics did not deliver what you expected it to deliver. You won't be able to say. So this is why I preferred first to, you know, complete the theory and know exactly what I wanted to achieve, and then separately try different heuristics and see how they perform with respect to this method. >>: Is there any way to measure how [inaudible] any way to measure how good heuristic is with respect to a fair sample [inaudible]? >>: Sure its Bayes error is low. I mean not the lowest one, but you can compare heuristics, but compare… >>: Compare heuristics but the can you tell if your… >> Ran Gilad-Bachrach: But you can't compare it. You want to compare it, again, you want to compare this Bayes error with--if its Bayes error is low, then you are good. But if it is not low, is it because your heuristic is broken or because your posterior is not very decisive or not very good? It is hard to isolate these two things in this is why I want to isolate the theory from the empirical because there is additional unknown over here. >>: I go by the hard Q comment. So it seems like in a lot of cases in [inaudible] Monte Carlo is a sample from the Q? >> Ran Gilad-Bachrach: Approximate. So even if… >>: So the question is whether or not it converged or not, but I think… >> Ran Gilad-Bachrach: For example, I think that even if you have in the simple case where you have uniform distribution over convex [inaudible]. So yes, there are polynomial algorithm for something from that and their polynomial, they have improved so that the complexity is something, so the first algorithm is something like, with the complexity like D to the power of 23, and they manage to reduce it all the way now to three or four, but the constant here is something like two to the power of 100. >>: But even [inaudible] finding mean instead of the posterior and a lot of approximation algorithms approximations like actually give you a very good assumption of Q. [inaudible] find the family… >> Ran Gilad-Bachrach: Again, there are a lot of heuristics on how to do that and there is evidence that in many cases it worked fine, but the only result that I know that is actually samples from the distribution at least in some cases, they have this complexity. So again once you start… >>: There are certain other cases where you know that the Markov chain is missing and you're going to guess [inaudible]. >>: You are making a very strong assumption that you have a sample from [inaudible] only sampling continuous values or [inaudible] but that is [inaudible] specific… >> Ran Gilad-Bachrach: That's true, but then if you say okay, let's sample only from Qs from which I want to sample, so maybe these Qs, so imagine now that I am going to apply to a certain problem. First of all let me just before we dive into this discussion, the first thing that I want to do is really go ahead and evaluate this. So this is obvious. But there is a reason why I wanted to isolate it from this theoretical work. Because once I want to do that, I have to either restrict the type of Qs, you know the posterior that I am allowed to use, to only Qs I know how to sample from, or say I am not going to restrict Qs, but I am going to use some heuristics to sample from this distribution. Anyway, if it works fine, that is great. But if it doesn't work, what does it mean? Does it mean that this full method is broken, or does it mean that if I restricted my family of distribution, restricted it in a way that this distribution do not work well for this problem. >>: By definition, since your method has to sample from Q, it is broken if it doesn't work. >> Ran Gilad-Bachrach: Yes, but then you can say okay, I can say there is another item now is how to sample from Q. So I went that far and obviously, you know, there is still a way to go. And I am very clear about it. But if I just evaluate now and see if it is working, we are happy. But if it doesn't work, what does it mean? That this whole line is broken, or that we need to improve our methods from sampling from Q? >>: Probably the latter. >>: Do you have any ideas on what types of Qs you are interested in sampling from? >> Ran Gilad-Bachrach: So for example, [inaudible]. >>: Exactly, [inaudible]. The kind of form, the function that you have. They haven't done work in trying to approximate the posterior [inaudible] like in the Bayesian [inaudible] community, but even SVM lost function [inaudible] laws and… >> Ran Gilad-Bachrach: As far as I know and I will be happy to know that I am wrong, is that none of them has provable guarantees on the difference between the true distribution from which you sample and the sample, the distribution that you are trying to sample from. All of them are well justified, seems to be working, but for none of them apart from this result from [inaudible] and one of those and I don't remember who else, none of them is really provable. >>: No, no. Here is the problem. So you talk about sample selection that comes closer to solution. And you were saying that your method [inaudible] immediate is a good one. And then I have [inaudible] evaluate in the mod, I have the method to evaluate the mean and in these cases people have shown that the mean might be better than mod, right? And I cannot pinpoint where does this [inaudible] lie in that space, unless there is some in particular evaluation. And hence my questions, what would it require for you to show as comparative as those [inaudible]? >> Ran Gilad-Bachrach: So definitely I am going to evaluate that and definitely I am going to try these heuristic methods either of sampling with the inaccuracy of the sampling process and see if it works, or otherwise restrict the family of Qs such that the families from which I know how to sample. But I wanted to separate that from this work because once I do that I add another unknown. The evaluation would be problematic. >>: Here's another way of looking at it. If maybe the sampling or the sample of posterior I said is really my posterior because I can adjust my prior. It is whatever I say. So I can adjust my prior maybe if there are certain [inaudible]'s that I don't know, because that is my prior. It's a crazy prior, but how can you argue with me? It's my prior and the prior happens to match where the posterior has to sample. >> Ran Gilad-Bachrach: But again, you run out to design an experiment, right? And you think okay, I am going to design an experiment. Let's see what are the possible outcomes? If the outcome is that it doesn't generalize well, is it because of your sampling technique, or is it because of the algorithm that I described here? >>: But [inaudible] I don't think this will [inaudible] maybe they are just inseparable. I think it doesn't make any sense [inaudible]. >> Ran Gilad-Bachrach: If it would have been inseparable, it would have been inseparable if I simply knew how to sample Q from a general Q, so for example energy function of SVM, if I could sample from this energy function. And then I could compare the result of the result of SVM, then I know exactly that this is… >>: Yeah, that is one choice of prior. >> Ran Gilad-Bachrach: Yes. This is one choice of prior, but an important one. >>: Yeah. [inaudible]. >> Ran Gilad-Bachrach: That's true. But then again, what is the broken component here? Is it that the prior is broken or is it that the algorithm is broken? So there is a problem in running this type of experiment. It is an inherent problem in this kind of an experiment; that you cannot control for all of the free variables. >>: That is true. >> Ran Gilad-Bachrach: So this is why I prefer to go all the way to establish a theory and know exactly what I want to achieve, analyze it and then once I do that, yeah, I am going to run the experiment, but I know that there are, that my ability to evaluate the results of these experiments is going to be limited. >>: [inaudible] you are saying that the death of the Bayes classifier is half and then your bounds have one over the depth, so you're going to pay factor of two in your error compared to the Gibbs sampling no matter what. So… >> Ran Gilad-Bachrach: Remember when [inaudible] over here said we get this annoying two factor for analyzing the Bayes classifier. We tried to get rid of it and didn't manage to do that. >>: No [inaudible] there's a paper where the two goes to almost epsilon in some settings but… >> Ran Gilad-Bachrach: I don't know. At least here when the studies were going on they said that we have this annoying two factor and we tried to get rid of it and we couldn't and… >>: [inaudible] years ago [inaudible]. >> Ran Gilad-Bachrach: You can see cases in which you can get better then two, for example, the relaxed depth, so we can say why is it two? It's two if you have many points on which the majority vote is very marginal. But if you actually, for most of the points the majority is very decisive. Actually this bound will show you that you can get better than factor two, kind of two is for the worst case in the classifier. It could be better. But even in their analysis actually what they show, the best bound that they get is for the expected Gibbs classifier, and then they say okay, the Bayes classifier cannot be much worse than that. And this is the same thing that we showed here. >>: I will show you the result where this two to [inaudible]. The other thing I wanted to say is that the trendy thing in learning theory now is to talk about the importance of strong convex [inaudible] and you didn't mention this at all. I am really [inaudible] required was the [inaudible] property of the posterior. Now what happens, I don't know what it would be called, when you have a posterior which is defined as normalized X of a strong convex function rather than just a convex function, which would be achieved by our strong convex regular [inaudible]. And I think what you get is that you have this guarantee that says that there exists a function which is at least at depth [inaudible] for E. >> Ran Gilad-Bachrach: [inaudible] I don't need the strong convex… >>: Right, but I'm saying I think that if you look that 1 over E in the general case. If you have a ball, you actually have half. And I think strong convexity [inaudible] interpolated between the two, right? This would be now just so you talking this result just for general convex body the center of mass you can cut it such that 1 over E is on one side. So I am assuming that that is like this 1 over E. if it was a ball, you could actually cut it in half and then if you had the strong convex [inaudible] like L2 regularizer think about, think of the last function is just an L2 regularizer. No empirical loss. So the weight of the empirical loss is zero. You are just quadratic. Then you have the ball. Then you would have half. >> Ran Gilad-Bachrach: Interesting direction. It is not something that I can come up with from the top of my head. Actually you can also show that if a function is not log concave actually there is a larger family of these distributions which is called ra concave function and for each one of them you can get a bound which is a function of ra. You can refine the results I'm sure. >>: Question about the, so the energy function that you were showing, regularizing the function, it doesn't always lead to [inaudible] solutions [inaudible] so if you just exponentiate the energy [inaudible]? >> Ran Gilad-Bachrach: Oh, no you always have… >>: [inaudible] for SVM, for [inaudible], it does not. >> Ran Gilad-Bachrach: So you always have the partition function. >>: Yeah, but I think that some people [inaudible]. So consider the case where you have, the SVM case, then you regularize it is a [inaudible] relative loss function. >> Ran Gilad-Bachrach: So you say that the integral might go to infinity and you cannot--could be, but then if you bound it, if you bound everything… >>: So if you can approximate it somehow? >> Ran Gilad-Bachrach: Thank you very much. [applause].