# Classification ```Classification
Definition
Classification is a task that requires the use of
machine learning algorithms that learn how to
assign a class label to examples from the
problem domain.
An easy to understand example is classifying
emails as “spam” or “not spam.”
Classification Predictive Modeling
In machine learning, classification refers to a
predictive modeling problem where a class label
is predicted for a given example of input data.
Examples of classification problems include:
-Given an example, classify if it is spam or not.
-Given a handwritten character, classify it as one of the
known characters.
-Given recent user behavior, classify as churn or not.
Class labels are often string values, e.g. spam, not spam,
and must be mapped to numeric values before being
provided to an algorithm for modeling. This is often
referred to as label encoding, where a unique integer is
assigned to each class label, e.g. “spam” = 0, “no spam” =
1.
Types of Classification
•
•
•
•
Binary Classification
Multi-Class Classification
Multi-Label Classification
Imbalanced Classification
Binary Classification
Binary classification refers to those classification
tasks that have two class labels.
Binary Classification
Examples include:
-Email spam detection (spam or not).
-Churn prediction (churn or not).
Binary classification tasks involve one class that
is the normal state (class label 0) and another
class that is the abnormal state (class label 1).
Popular algorithms
Algorithms that can be used for binary classification include:
• Logistic Regression
• k-Nearest Neighbors
• Decision Trees
• Support Vector Machine
• Naive Bayes
Some algorithms are specifically designed for binary
classification and do not natively support more than two
classes.
-examples include Logistic Regression and Support Vector
Machines.
Multi-Class Classification
Multi-class classification refers to those
classification tasks that have more than two
class labels.
Multi-Class Classification
Examples include:
-Face classification.
-Plant species classification.
-Optical character recognition
In multi class classification, examples are
classified as belonging to one among a range of
known classes.
Popular algorithms
Algorithms that can be used for multi-class
classification include:
– k-Nearest Neighbors.
– Decision Trees.
– Naive Bayes.
– Random Forest.
Other Classifications
Multi-Label Classification
-Multi-label classification refers to those
classification tasks that have two or more class
labels, where one or more class labels may be
predicted for each example.
Imbalanced Classification
-Imbalanced classification refers to classification
tasks where the number of examples in each
class is unequally distributed.
Assessing Classification Performance
Why?
-Multiple methods are available to classify or predict
-For each method, multiple choices are available for
settings
-To choose best model, need to assess each model’s
performance
Accuracy Measures (Classification)
Misclassification error
• Error = classifying a record as belonging to one
class when it belongs to another class.
• Error rate = percent of misclassified records
out of the total records in the validation data
Confusion Matrix
A confusion matrix is a table that is often used
to describe the performance of a classification
model (or &quot;classifier&quot;) on a set of test data for
which the true values are known.
Confusion Matrix
A confusion matrix (Kohavi and Provost, 1998)
contains information about actual and predicted
classifications done by a classification system.
Performance of such systems is commonly
evaluated using the data in the matrix.
Example confusion matrix for a binary classifier
-There are two possible predicted classes: &quot;yes&quot; and &quot;no&quot;. If we
were predicting the presence of a disease, for example, &quot;yes&quot;
would mean they have the disease, and &quot;no&quot; would mean they
don't have the disease.
-The classifier made a total of 165 predictions (e.g., 165 patients
were being tested for the presence of that disease).
-Out of those 165 cases, the classifier predicted &quot;yes&quot; 110 times,
and &quot;no&quot; 55 times.
-In reality, 105 patients in the sample have the disease, and 60
patients do not.
true positives (TP): These are cases in which we
predicted yes (they have the disease), and they do
have the disease.
true negatives (TN): We predicted no, and they
don't have the disease.
false positives (FP): We predicted yes, but they
don't actually have the disease. (Also known as a
&quot;Type I error.&quot;)
false negatives (FN): We predicted no, but they
actually do have the disease. (Also known as a
&quot;Type II error.&quot;)
• Accuracy: Overall, how often is the classifier
correct?
– (TP+TN)/total = (100+50)/165 = 0.91
• Misclassification Rate: Overall, how often is it
wrong?
– (FP+FN)/total = (10+5)/165 = 0.09
– equivalent to 1 minus Accuracy
– also known as &quot;Error Rate&quot;
• True Positive Rate: When it's actually yes,
how often does it predict yes?
– TP/actual yes = 100/105 = 0.95
– also known as &quot;Sensitivity&quot; or &quot;Recall&quot;
• False Positive Rate: When it's actually no, how
often does it predict yes?
– FP/actual no = 10/60 = 0.17
• True Negative Rate: When it's actually no, how often
does it predict no?
– TN/actual no = 50/60 = 0.83
– equivalent to 1 minus False Positive Rate
– also known as &quot;Specificity&quot;
• Precision: When it predicts yes, how often is it correct?
– TP/predicted yes = 100/110 = 0.91
• Prevalence: How often does the yes condition actually
occur in our sample?
– actual yes/total = 105/165 = 0.64
ROC Curve
• Started in electronic signal detection theory
(1940s - 1950s)
• Has become very popular in biomedical
• Also used in machine learning applications to
assess classifiers
• Can be used to compare tests/procedures
ROC curves: simplest case
• Consider diagnostic test for a disease
• Test has 2 possible outcomes:
– ‘positive’ = suggesting presence of disease
– ‘negative’
• An individual can test either positive or
negative for the disease
ROC Analysis
• True Positives = Test states you have the
disease when you do have the disease
• True Negatives = Test states you do not have
the disease when you do not have the disease
• False Positives = Test states you have the
disease when you do not have the disease
• False Negatives = Test states you do not have
the disease when you do
Specific Example
Threshold
Some Definitions…
Some Definitions…
Some Definitions…
Some Definitions…
Moving the Threshold: right
Moving the Threshold: left
Threshold Value
• The outcome of a logistic regression model is a
probability
• Often, we want to make a binary prediction
• We can do this using a threshold value t
• If P(y = 1) ≥ t, predict positive
– If P(y = 1) &lt; t, predict negative
– What value should we pick for t?
Threshold Value
• Often selected based on which errors are “better”
• If t is large, predict positive rarely (when P(y=1) is large)
– More errors where we say negative , but it is actually
positive
– Detects patients who are negative
• If t is small, predict negative rarely (when P(y=1) is small)
– More errors where we say positive, but it is actually negative
– Detects all patients who are positive
• With no preference between the errors, select t = 0.5
– Predicts the more likely outcome
THANK YOU!
```