UGBA104: Introduction to Business Analytics
Week 06 Predictive Analytics
k-NN Classifier and Undersampling
Professor Thomas Y. Lee
Operations and Information Technology Management
Overview
Week 06
• Monday
– Why:
– What:
D
M
D
Predict
Evaluating Binary Classification (11.2)
• Overfitting, Imbalanced Data
• Tradeoffs and Targeting
kNN (k-nearest neighbor) (Camm 11.4)
•
Wednesday
– How:
kNN (k-nearest neighbor) (Camm 11.4)
Next Week
• What:
– Classify with Trees
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 2
Confusion Matrix
Actual\Predicted
0
1
0
a
1
thomasyl@berkeley.edu
b
At the arrow on the ROC curve above, the
FPR is 0.124 and the TPR is 0.543
Consider the confusion matrix at the arrow.
What is the value in [a] expressed as a
count of observations rounded to the
nearest integer
What is the value in [b] expressed as a
count of observations rounded to the
nearest integer
UC Berkeley, Haas School
Slide 3
[a] is TN
TN/(TN + FP) = TNR
specificity = 1-FPR
From confusion matrix, TN + FP is 1157
So w/ algebra, TN/1157 = 1 – FPR
So TN = 1157*(1-FPR) = 1157*(1 – 0.124) = 1014
[b] is TP
TP/(TP + FN) = TPR = sensitivity
From confusion matrix, TP + FN = 348
So w/ algebra, TP/348 = TPR
So TP = TPR * 348 = .543 * 348 = 189
thomasyl@berkeley.edu
Confusion Matrix
Actual\Predicted
0
1
0
a
1
b
FPR is 0.124 and the TPR is 0.543
UC Berkeley, Haas School
Slide 4
Confusion Matrix
Actual\Predicted
0
1
thomasyl@berkeley.edu
0
[a]
1
[b]
At the arrow on the ROC curve above, the
FPR is 0.124 and the TPR is 0.543
Consider the confusion matrix at the arrow.
What is the value in [a] expressed as a
count of observations rounded to the
nearest integer
What is the value in [b] expressed as a
count of observations rounded to the
nearest integer
UC Berkeley, Haas School
Slide 5
[a] is FN
FN/(TP + FN) = FNR = 1-TPR
From confusion matrix, TP + FN = 348
So w/ algebra, FN/348 = 1-TPR
So FN = 348 * (1 -.543) = 159
[b] is FP
FP/(TN + FP) = FPR
From confusion matrix, TN + FP is 1157
Confusion Matrix
Actual\Predicted
0
1
0
1
[b]
[a]
So w/ algebra, FP/1157 = FPR
FPR is 0.124 and the TPR is 0.543
FP = 1157*(FPR) = 1157*(0.124) = 143
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 6
It was fall 2021 when [Kate] Trumbull, chief brand officer for Domino’s, and her team
began strategizing [The Emergency Pizza] promotion … to add new members to its
loyalty program, as well as encourage current members to increase their order
frequency….
As Domino’s tested the deal with focus groups for roughly a year, the company’s
finance executives had basic questions: Could the deal … bring back infrequent
customers? With potentially hundreds of thousands of free pizzas being offered, they
recommended reserving it for carryout customers, a part of the business the chain
had been working to expand … [or should] it also benefit delivery customers.
What are examples of binary outcomes that Domino's wanted to predict.
For one observation (customer), what is the variable that Domino’s is trying to predict?
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 7
Card Magic – Memorial Stadium
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 8
Does This Student Have a Blue Card?
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 9
Does This Student Have a Blue Card?
Intuition
Ask their “friends”
Guess “is the card Blue?”
Who is a “friend”?
How many friends to ask?
What if friends disagree?
*Intuition: “Birds of a Feather” … friends sit together and friends “have the same color card.”
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 10
Does This Student Have a Blue Card?
Intuition
Algorithm
Applied
Ask their “friends”
Pick the data sample
Student section
Pick the variables (that
A student
describe an observation)
Guess “is the card Blue?”
What is the outcome
Is card Blue?
Who is a “friend”?
What are the inputs
Seat and Row Number
Distance measure
Euclidean
How many friends to ask?
Set the value of (k)
5 closest seats
What if friends disagree?
What is the cut-off?
Majority rules (50%)
*Intuition: “Birds of a Feather” … friends sit together and friends “have the same color card.”
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 11
Recruiting New Music Fans (W06-DataMusicFan.xlsx)
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 12
Recruiting New Music Fans (W06-DataMusicFan.xlsx)
Download the file (W06-DataMusicFan.xlsx) from bCourses.
To do:
Partition the data:
• Use the OSPartition Worksheet in the Excel workbook.
• This data is not really “imbalanced” but it gives us an example.
Understand how to use validation data to tune a hyper-parameter and select a model:
Build at least two different k-NN classifiers. One with k=8 and one with k=5.
Describe an observation (a student):
• Your outcome variable is “Fan”
• Your input variables are only “Lat” and “Long” (yes, you could include Size but two
dimensions are easier to visualize for this exercise)
Set your cutoff at 0.5
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 14
Questions
For your “best” classifier on “oversampled” data, estimate w/ the Test partition:
• What % of the potential fans that you predict will become fans really will?
• You only have enough swag for 20 potential fans (music listeners). If you
distribute swag to the top 20 candidates as predicted by your classifier, how
many actual “fans” do you expect to reward with swag? (i.e. not all people
that you predict are “fans” are genuine fans).
• If you picked 20 listeners at random, how many “fans” would you expect
among the group?
• If you want to get “10 new fans”, how many potential fans (music listeners)
do you need to target?
What difference does it make for (im)balanced data?
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 16
kNN (Binary) Classification and ASP
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 19
kNN (Binary) Classification and ASP
Specify the outcome variable
(the variable to “predict”)
Select those variables from
which to build the model.
If the outcome variable is categorical,
ASP will automatically ask for the
number of classes (as opposed to
estimating a continuous value).
thomasyl@berkeley.edu
kNN voting rule: 50% = “majority rule”
UC Berkeley, Haas School
Slide 20
kNN (Binary) Classification and ASP
Partition the data if
not already done
If you did not use
Data Science | Partition
to create a new worksheet w/
data partitions
Lat and Long already
on same scale
Sensitivity analysis to empirically
determine the optimal number of
neighbors, k.
thomasyl@berkeley.edu
Estimating the population
distribution of 1’s and 0’s
(“global mean”) by looking
at the empirical distribution
of 1’s and 0’s in the data.
UC Berkeley, Haas School
Slide 21
kNN (Binary) Classification and ASP
Excel creates a number of new worksheets with Data Mining results
Generate
predictions
for “new”
observations
with no class
labels
Results for
Validation data
Results for
Training data
New worksheet with overall classifier results
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 22
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 25
Recruiting New Music Fans (W06-DataMusicFan.xlsx)
Download the file (W06-DataMusicFan.xlsx) from bCourses.
To do:
Partition the data:
• Use the OSPartition Worksheet in the Excel workbook.
• This data is not really “imbalanced” but it gives us an example.
Understand how to use validation data to tune a hyper-parameter and select a model:
Build at least two different k-NN classifiers. One with k=8 and one with k=5.
Describe an observation (a student):
• Your outcome variable is “Fan”
• Your input variables are only “Lat” and “Long” (yes, you could include Size but two
dimensions are easier to visualize for this exercise)
Set your cutoff at 0.5
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 27
Questions
For your “best” classifier on “oversampled” data, estimate w/ the Test partition:
• What % of the potential fans that you predict will become fans really will?
• You only have enough swag for 20 potential fans (music listeners). If you
distribute swag to the top 20 candidates as predicted by your classifier, how
many actual “fans” do you expect to reward with swag? (i.e. not all people
that you predict are “fans” are genuine fans).
• If you picked 20 listeners at random, how many “fans” would you expect
among the group?
• If you want to get “10 new fans”, how many potential fans (music listeners)
do you need to target?
What difference does it make for (im)balanced data?
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 29
Picking Models: Validation Partition
and tuning hyperparameters – the value of k in k-NN
Pick the hyperparameter, the
value of “k” that minimizes
%misclassification
Same cut-off value
Changing “k” (number of neighbors)
changes the value of the
performance metric. Excel/ASP
uses overall misclassification… You
may pick your own metric
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 32
Picking Models: Validation Partition
and tuning hyperparameters – the value of k in k-NN
kNN for k=8
kNN for k=5
Same cutoff
value
kNN for k=8
kNN for k=5
Confusion
Matrix changes
based upon the
value of “k” …
you can use
your own criteria
for picking the
“best k” on
Validation data
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 33
Picking Models: Validation Partition
and tuning hyperparameters – the value of k in k-NN
Higher AUC.
Comparing
classifiers at
all cutoff
values
(i.e. at all
different
confusion
matrices)
kNN for k=5
kNN for k=8
See:
Validation Lift
Chart
worksheet
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 34
Lessons Learned
•
•
Tuning Hyperparameters (like the value of “k” in k-NN)
– Use Validation Data (Validation Partition)
– Pick appropriate performance metric (e.g. % misclassification or AUC)
Comparing (Evaluating) Classifiers. Not all errors are created equal
– ROC Curve (Receiver Operating Characteristic Curve is TPR v. FPR)
– AUC (Area Under the ROC Curve) one metric for all cutoff values
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 35
Applying Your Prediction Machine (your classifier)
“Unbiased estimate” of how well the classifier
predicts for new, previously unseen observations
… (using the “test” partition
• What % of the
potential fans that
you predict will
become fans, really
will?
• This is just the
“Precision” which is
computed from the
column of “Positive
predictions”
• 13/(6+13) or 13/19
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 38
Applying Your Prediction Machine (your classifier)
Using the Test Partition (Test Lift Chart) to estimate performance on “unseen” data
•
You only have enough swag for
20 potential fans (music
listeners). If you distribute swag
to the top 20 candidates as
predicted by your classifier, how
many “fans” do you expect to
discover?
thomasyl@berkeley.edu
•
If you picked 20 listeners at
random, how many “fans” would
you expect to discover?
•
Rate of “fans” in the population is
30/42…. So 20 *(30/42) Yikes! Not
a good classifier. We do just as
well w/ random guessing ☺
UC Berkeley, Haas School
Slide 39
Applying Your Prediction Machine (your classifier)
• If you want to get
“10 new fans”, how
many potential fans
(music listeners) do
you need to target?
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 40
Lessons Learned
•
•
•
•
Tuning Hyperparameters (like the value of “k” in k-NN)
– Use Validation Data (Validation Partition)
– Pick appropriate performance metric (e.g. % misclassification or AUC)
Comparing (Evaluating) Classifiers. Not all errors are created equal
– ROC Curve (Receiver Operating Characteristic Curve is TPR v. FPR)
– AUC (Area Under the ROC Curve) one metric for all cutoff values
Making decisions
– Use Test Data (Test partition)
– Unbiased estimate of how well model performs on new, “unseen” data
– What might you expect among the new observations
Over (under) sampling
– What is the impact of Over (under) sampling on learning performance
– How does ASP/Excel output indicate over (under) sampling in training, validation,
testing
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 41
Key Ideas
k-NN
Output (Class) Variable
Logistic Reg
What are you trying to predict (1, 0)
Input Variable(s)
Measuring similarity
Explaining the log odds
Tuning
Scale or weight variable
values, influencing who is a
neighbor
Estimate (fit) coefficients,
marginal impact of each
variable on log odds
k (number of neighbors)
Interpreting cutoff
Voting rule %(k) neighbors
Above cutoff probability
Data Sample
Training data *is* the model; Estimate coefficients to fit
the neighborhood
the training partition
Interpretability
Birds of a feather; flocking
Influence of input variables
Evaluating probabilistic classifiers
• Varying the cutoff exchanges Class-1 errors for Class-0 errors; not all
errors are created equal.
• Gain and Decile-Wise Lift Chart visualize improvement over ”guessing”
(where guessing matches the population averages); bang for buck.
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 42
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 43
Bang for Buck: Lift Curves
Lift = Ratio of Cum TP to Cum TP if you guess at random
= Ratio of “Classifier TPR” to “TPR if you guess at random”
thomasyl@berkeley.edu
UC Berkeley, Haas School
“Cumulative TP”
from Random
Guessing
Slide 44
Bang for Buck: Lift Curves
How much better is this classifier
than “random guessing”
Note: this classifier is not much
better, but a “good” classifier would
be, as illustrated by the gap between
the blue and red lines.
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 45
Question: How is the Gain curve different from the Lift curve? Hint: How is the
information in the curves different?
Hint: Think about the units on each axis. How does “Cumulative TP” relate to TPR
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 46
Bang for Buck: Decile Wise Lift
Decile Wise Lift:
Ratio of Decile Mean to “Global Mean”
Global Mean = Expectation from random guessing = 40/56 in this example.
Decile Mean = % of Success (TP) in each decile
Q: If you target the first decile (the first 10th of the sample ranked by prediction probability) how many
fans would you expect?
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 47
Lessons Learned
•
•
•
•
Tuning Hyperparameters (like the value of “k” in k-NN)
– Use Validation Data (Validation Partition)
– Pick appropriate performance metric (e.g. % misclassification or AUC)
Comparing (Evaluating) Classifiers. Not all errors are created equal
– ROC Curve (Receiver Operating Characteristic Curve is TPR v. FPR)
– AUC (Area Under the ROC Curve) one metric for all cutoff values
– Lift: Cumulative TP when you focus on the highest probability first
– Decile-wise Lift: Ratio of Cumulative TP to random guessing when you focus on
highest probability first
Making decisions
– Use Test Data (Test partition)
– Unbiased estimate of how well model performs on new, “unseen” data
– What might you expect among the new observations
Over (under) sampling
– What is the impact of Over (under) sampling on learning performance
– How does ASP/Excel output indicate over (under) sampling in training, validation,
testing
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 49
Monday Lecture
For each prediction probability (imagine the “cutoff”
is set to this prediction probability.
Every observation with a prediction probability
greater than or equal to this cutoff is a “positive
prediction”
Count the total, cumulative number of True
Positives at this cutoff
Compute the TPR = Cumulative TP divided by Total
Positive Cases (40 in this example),
thomasyl@berkeley.edu
DecreasingProbability
Bang for Buck: Gain Curves
Imagine a perfect classifier.
The 40 positive observations have high probabilities.
The remaining observations have low probabilities
The Cumulative TP and TPR have no errors
UC Berkeley, Haas School
Slide 50
Monday Lecture
Bang for Buck: Gain Curves
Gain Curve
1
0.9
0.8
0.7
TPR
0.6
0.5
TPR
0.4
Perfect TPR
0.3
0.2
0.1
0
0
10
20
30
40
50
60
Num Observations (by decreasing prediction probability)
Horizontal Axis: Number of observations (in this
example, 1 to 56
thomasyl@berkeley.edu
Predictive Positive Rate (support) is
Number of observations as a percentage
e.g. 1/56, 2/56, 3/56 … 56/56
UC Berkeley, Haas School
Slide 51
Monday Lecture
Bang for Buck: Gain Curves
Gain Curve
1
0.9
0.8
0.7
TPR
0.6
0.5
TPR
0.4
Perfect TPR
0.3
0.2
0.1
0
0
10
20
30
40
50
60
Num Observations (by decreasing prediction probability)
If you target the “top 20” observations (music listeners) as predicted by your
classifier, how many “fans” do you expect to discover?
If you want to get “10 new fans”, how many potential observations (music
listeners) do you need to target?
If you picked 20 observations at random, how many “fans” would you expect
to discover?
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 52
Lessons Learned
•
•
•
•
Tuning Hyperparameters (like the value of “k” in k-NN)
– Use Validation Data (Validation Partition)
– Pick appropriate performance metric (e.g. % misclassification or AUC)
Comparing (Evaluating) Classifiers. Not all errors are created equal
– ROC Curve (Receiver Operating Characteristic Curve is TPR v. FPR)
– AUC (Area Under the ROC Curve) one metric for all cutoff values
– Lift: Cumulative TP when you focus on the highest probability first
– Decile-wise Lift: Ratio of Cumulative TP to random guessing when you focus on
highest probability first
– Gain: Cumulative TP (TPR) when you focus on the highest probability
predictions first.
Making decisions
– Use Test Data (Test partition)
– Unbiased estimate of how well model performs on new, “unseen” data
– What might you expect among the new observations
Over (under) sampling
– What is the impact of Over (under) sampling on learning performance
– How does ASP/Excel output indicate over (under) sampling in training, validation,
testing
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 54
Key Ideas
k-NN
Output (Class) Variable
Logistic Reg
What are you trying to predict (1, 0)
Input Variable(s)
Measuring similarity
Explaining the log odds
Tuning
Scale or weight variable
values, influencing who is a
neighbor
Estimate (fit) coefficients,
marginal impact of each
variable on log odds
k (number of neighbors)
Interpreting cutoff
Voting rule %(k) neighbors
Above cutoff probability
Data Sample
Training data *is* the model; Estimate coefficients to fit
the neighborhood
the training partition
Interpretability
Birds of a feather; flocking
Influence of input variables
Evaluating probabilistic classifiers
• Varying the cutoff exchanges Class-1 errors for Class-0 errors; not all
errors are created equal.
• Gain and Decile-Wise Lift Chart visualize improvement over ”guessing”
(where guessing matches the population averages); bang for buck.
thomasyl@berkeley.edu
UC Berkeley, Haas School
Slide 55