Generalization Central challenge in ML is that our algorithm must perform well on new, previously unseen inputs, not just those on which our model was trained The ability to perform well on previously unobserved inputs is called generalization Typically, when training a ML model, we have access to a training set We can compute some error measure on the training set, called the training error and we reduce this training error This is just an optimization problem What separates ML for optimization is that we want the generalization error, also called the test error, to be low as well Generalization error is defined as the expected value of error in a new input We estimate the generalization error of a ML model by measuring its performance on a test set on a test set of examples that were collected separately from the training set For example, in linear regression, we trained the model by minimizing the training error, But we really care about the test error Data-generating distribution The training and test data are generated by a probability distribution over datasets called the data-generating process We make some assumptions about the process called the iid assumption o 1. Examples in each dataset are independent from each other o 2. Training and test said are identically distributed, meaning they are draw from the same probability distribution Same distribution is used to generate every train example and every test example We call the shared underlying distribution as the data-generating distribution Ideal model know the true probability distribution that generates the data, but even if we this model would have error on many problems because there may still be some noise in the distribution Overfitting and Underfitting The factors that determine how well a ML algorithm will perform are its ability to: 1. Make the training error small 2. Make the gap between training and test error small These two factors correspond to two challenges in machine learning: underfitting and overfitting Underfitting: model is not able to obtain a low error in training set Overfitting: gap between training error and test error is too large Capacity We can control whether a model is more likely to overfit or underfit by playing around with its capacity A model’s capacity is its ability to fit a wide variety of functions Models with low capacity may struggle to fit a training set Models with high capacity can overfit by memorizing properties of the training set that do to generalize well on the test set One way to control the capacity is by choosing a model’s hypothesis space – the set of functions that the algorithm is allowed to select as being the solution Linear Regression Example Linear regression algorithm has the set of all linear functions of its input as its hypothesis space We can generalize linear regression to include polynomials in its hypothesis space This increases the model’s capacity Introudce x^2 as another feature provided to the model, we can learn a model that is quadratic as a function of x We can continue to add more powers of x as additional features to increase the polynomial degree Even though the model implements a quadratic function of its input, the output is still a linear function of the parameters, so we can still train the model in closed form Linear Regression Example We fit three models to this example training set where the underlying distribution is quadradic LEFT – a linear function fit to the data suffers from underfitting – it cannot capture the curvature that is present in the data CENTER – quadratic function fit to the data generalizes well to unseen points, does not suffer from significant amount of overfitting or underfitting RIGHT – polynomial of degree 9 for to the data, suffers from overfitting, passes through all training points, deep valley between two training points, sharp increase on left side In general, ML will perform best when their capacity is appropriate for the true complexity of the task Models with insufficient capacity are unable to solve complex tasks Models with high capacity can solve complex tasks, but when their capacity is higher than needed, they may overfit Relationship between capacity and error Simpler functions are more likely to generalize, we still need to choose a sufficiently complex hypothesis to achieve low error Typically, as you increase model capacity, the training error decreases until it reaches a minimum possible error value Generalization error has a U-shape curve At the left end of the graph, training error and generalization error are both high – this is the underfitting zone As we increase capacity, training error decreases, but the gap between training and generalization error increases Eventually, the size of the gap outweighs the decrease in training error and this is the overfitting zone where the capacity is too large No Free Lunch Theorem Averaged over all possible data-generating distributions, every classification algorithm has the same error rate when classifying previously unobserved points In other words, no ML algorithm is universally better than any other The most sophistical algorithm has the same average performance (over all possible tasks) as merely predicting that every point belongs to the same class Luckily, these results hold only when we average over all possible data-generating distributions If we make assumptions about the kinds of probability distributions we encounter, we can design learning algorithms perform well on these distributions Goal of ML is not to seek a universal learning algorithm Our goal is to understand what kinds of distributions are relevant to the “real world”, and what kind of ML algorithms perform well on data draw from data-generating distributions Regularization No free lunch theorem implies that we must design our ML algorithms to perform well on a specific task We do so by building a set of preferences into the learning algorithm With the linear regression example, we discussed increasing the model’s capacity by adding functions to the hypothesis space of the solutions the learning algorithm is able to choose from We can also give a learning algorithm preference for one solution over another in its hypothesis space, another way of controlling model’s capacity We can regularize a model that learns a function by adding a penalty to the cost function Regularization is any modification we make to a learning algorithm that is intended to reduce its generalization error but not its training error Point estimation Statistics offers user concepts such as parameter estimation, bias and variance to formally characterize generalization, underfitting and overfitting Point estimation is the attempt to find a single “best” prediction of some statistic Bias This can be a single parameter or a vector of parameters in some parametric model, such as the weight in out linear regression example If the true value of our parameter is theta, we denote a point estimate as theta hat Given a set of data points, a point estimate is any function of the data This definition is very general and does not even require that g return a value close to the true theta A good estimator is a function whose output is close to the true underlying theta that generated the data Frequentist perspective – we assume the true parameter theta is fixed but unknown, while the point estimetta is a function of the data the bias of an estimator is the difference between this estimator's expected value and the true value of the parameter being estimated. this estimator's expected value is equal to the true value of the parameter Whiteboard example: gaussian distribution estimator of the mean - a common estimator of the Gaussian mean parameter is the sample mean - To determine the bias of the sample mean, we are interesting in calculating its expectation Variance - expected value of the squared sampling deviations - indicate how far, on average, the collection of estimates are from the expected value of the estimates - Variance of an estimator is a measure of how we would expect the estimate we vary as we independently resample the dataset from the underlying data-generating process Mean squared error of estimator - Bias and variance measure two sources of error in an estimator - MSE measures the overall expected deviation between the estimator and true value of parameter theta - Evaluating MSE incorporated both bias and variance - A Good estimate has a small MSE and these are esimators tha manage to keep both bias and variance in check Bias variance tradeoff - As capacity increases, bias tends to decrease and variance tends to increase - U-shaped curve of the generalization error as a function of capacity - You want to balance out the bias and variance of your model to the point where your test error (validation error) and training error have reached their combined minimum.
