CLASSIFIER ACCURACY Intuition During the training phase, measuring accuracy plays a relevant role in model selection: parameters are selected in order to maximize prediction accuracy on training samples. At the end of the learning step, accuracy is measured to assess the model predictive ability on new data. Being learning algorithms trained on finite samples, the risk is to overfit training data: the model might memorize the training samples instead of learning a general rule, i.e. the data generating model. For this reason, a high accuracy on unseen data is an index of the model generalization ability. ROC curve 1. Receiver operating characteristic (ROC) analysis is one of the most popular technique used to measure the accuracy of the prediction model. 2. ROC analysis is well suited for binary outcome variable i.e. when the outcome variable has two possible outcome values. 3. The ROC curve is defined as a plot between sensitivity on the y-coordinate and 1-specificity on the x-coordinate models at different possible threshold values (cut-off points) Concept of Confusion Matrix Let us assume for simplicity to have a two-class problem: as an example, consider the case of a diagnostic test to discriminate between subjects affected by a disease (patients) and healthy subjects (controls). • TP or true positives, the number of correctly classified patients. • TN or true negatives, the number of correctly classified controls. • FP or false positives, the number of controls classified as patients. • FN or false negatives, the number of patients classified as controls. The sum of TP, TN, FP and FN equals N , the number of instances to classify. These values can be arranged in a 2 × 2 matrix called contingency matrix. Performance Measures in Classification, their pros and cons All model performance measures presented face some limitations. For that reason, many measures are available, as the limitations of a particular measure were addressed by developing an alternative. For instance, RMSE is frequently used and reported for linear regression models. However, as it is sensitive to outliers, MAE was proposed. In case of predictive models for a binary dependent variable, the measures like accuracy, F1 score, sensitivity, and specificity, are often considered depending on the consequences of correct/incorrect predictions in a particular application. However, the value of those measures depends on the cut-off value used for creating the predictions. For this reason, ROC curve and AUC have been developed and have become very popular. Given the advantages and disadvantages of various measures, and the fact that each may reflect a different aspect of the predictive performance of a model, it is customary to report and compare several of them when evaluating a model’s performance. Factors affecting accuracy of Models 1. Overfitting: In learning, overfitting occurs when the model learns noise rather than depicting the relationship. ● When the training error is much lower than the generalization or testing error, the model predicted is said to be over fitted. 2. Underfitting: In supervised learning, underfitting happens when a model unable to capture the underlying pattern of the data. These models usually have high bias and low variance. 3. Bias and Variance Errors Variance : Variance is the amount that the estimate of the target function will change if different training data was used. ● Low Variance: Suggests small changes to the estimate of the target function with changes to the training dataset. ● High Variance: Suggests large changes to the estimate of the target function with changes to the training dataset, means that the learning algorithm varies a lot depending on the data that is given to it. Examples of low-variance machine learning algorithms include: Linear Regression, Linear Discriminant Analysis and Logistic Regression. Examples of high-variance machine learning algorithms include: Decision Trees, k-Nearest Neighbors and Support Vector Machines. Bias :Bias are simplifying assumptions made by a model to make a target function easier to learn. Low Bias: Suggests less assumptions about the form of the target function. High-Bias: Suggests more assumptions about the form of the target function.
