MORE CLASSIFIERS
AGENDA
Key concepts for all classifiers
Precision vs recall
Biased sample sets
Linear classifiers
Intro to neural networks
RECAP: DECISION BOUNDARIES
With continuous attributes, a decision boundary
is the surface in example space that splits
positive from negative examples
x2
x1>=20
T
x2>=10
F
x2>=15
F
T
F
T
T
F
x1
BEYOND ERROR RATES
4
BEYOND ERROR RATE
Predicting security risk
Predicting “low risk” for a terrorist, is
far worse than predicting “high risk”
for an innocent bystander (but maybe
not 5 million of them)
Searching for images
Returning irrelevant images is worse
than omitting relevant ones
5
BIASED SAMPLE SETS
Often there are orders of magnitude more
negative examples than positive
E.g., all images of Kris on Facebook
If I classify all images as “not Kris” I’ll have
>99.99% accuracy
Examples of Kris should count much more than
non-Kris!
FALSE POSITIVES
True decision boundary
Learned decision boundary
x2
x1
7
An example
incorrectly predicted
to be positive
FALSE POSITIVES
True decision boundary
Learned decision boundary
x2
New query
x1
8
An example
incorrectly predicted
to be negative
FALSE NEGATIVES
True decision boundary
Learned decision boundary
x2
New query
x1
9
PRECISION VS. RECALL
Precision
Recall
# of relevant documents retrieved / # of total
documents retrieved
# of relevant documents retrieved / # of total relevant
documents
Numbers between 0 and 1
10
PRECISION VS. RECALL
Precision
# of true positives / (# true positives + # false
positives)
Recall
# of true positives / (# true positives + # false
negatives)
A precise classifier is selective
A classifier with high recall is inclusive
11
REDUCING FALSE POSITIVE RATE
True decision boundary
Learned decision boundary
x2
x1
12
REDUCING FALSE NEGATIVE RATE
True decision boundary
Learned decision boundary
x2
x1
13
PRECISION-RECALL CURVES
Measure Precision vs Recall as the decision
boundary is tuned
Recall
Perfect classifier
Actual performance
14
Precision
PRECISION-RECALL CURVES
Measure Precision vs Recall as the decision
boundary is tuned
Recall
Penalize false negatives
Equal weight
Penalize false positives
15
Precision
PRECISION-RECALL CURVES
Measure Precision vs Recall as the decision
boundary is tuned
Recall
16
Precision
PRECISION-RECALL CURVES
Measure Precision vs Recall as the decision
boundary is tuned
Recall
Better learning
performance
17
Precision
OPTION 1: CLASSIFICATION THRESHOLDS
Many learning algorithms (e.g., probabilistic
models, linear models) give real-valued output
v(x) that needs thresholding for classification
v(x) > t => positive label given to x
v(x) < t => negative label given to x
May want to tune threshold to get fewer false
positives or false negatives
18
OPTION 2: WEIGHTED DATASETS
Weighted datasets: attach a weight w to each
example to indicate how important it is
Instead of counting “# of errors”, count “sum of
weights of errors”
Or construct a resampled dataset D’ where each
example is duplicated proportionally to its w
As the relative weights of positive vs negative
examples is tuned from 0 to 1, the precisionrecall curve is traced out
LINEAR CLASSIFIERS : MOTIVATION
Decision tree produces axis-aligned decision
boundaries
Can we accurately classify data like this?
x2
x1
PLANE GEOMETRY
Any line in 2D can be expressed as the set of
solutions (x,y) to the equation ax+by+c=0 (an
implicit surface)
ax+by+c > 0 is one side of the line
ax+by+c < 0 is the other
ax+by+c = 0 is the line itself
y
b
a
x
PLANE GEOMETRY
In 3D, a plane can be expressed as the set of
solutions (x,y,z) to the equation ax+by+cz+d=0
ax+by+cz+d > 0 is one side of the plane
ax+by+cz+d < 0 is the other side
ax+by+cz+d = 0 is the plane itself
z
c
a
b
x
y
LINEAR CLASSIFIER
In d dimensions,
c0+c1*x1+…+cd*xd =0
is a hyperplane.
Idea:
Use c0+c1*x1+…+cd*xd > 0 to denote positive
classifications
Use c0+c1*x1+…+cd*xd < 0 to denote negative
classifications
PERCEPTRON
x2
+
+
x1
+
-
-
+
xi wi
+
S
g
xn
y = f(x,w) = g(Si=1,…,n wi xi)
y
x1
-
w1 x1 + w2 x2 = 0
-
g(u)
24
u
A SINGLE PERCEPTRON CAN LEARN
x1
xi wi
S
g
y
xn
A disjunction of boolean literals x1 x2 x3
Majority function
25
A SINGLE PERCEPTRON CAN LEARN
x1
xi wi
S
g
y
xn
A disjunction of boolean literals x1 x2 x3
Majority function
XOR?
26
PERCEPTRON LEARNING RULE
θ θ + x(i)(y(i)-g(θT x(i)))
(g outputs either 0 or 1, y is either 0 or 1)
If output is correct, weights are unchanged
If g is 0 but y is 1, then the value of g on attribute
i is increased
If g is 1 but y is 0, then the value of g on attribute
i is decreased
Converges if data is linearly separable, but
oscillates otherwise
27
PERCEPTRON
+
+
x1
xi wi
?
-
S
+
g
xn
y = f(x,w) = g(Si=1,…,n wi xi)
y
-
+
+
-
g(u)
28
u
UNIT (NEURON)
x1
xi wi
S
g
y
xn
y = g(Si=1,…,n wi xi)
g(u) = 1/[1 + exp(-u)]
29
NEURAL NETWORK
Network of interconnected neurons
x1
xi w
x1
i
xi w
i
xn
S
g
y
S
g
y
xn
30
Acyclic (feed-forward) vs. recurrent networks
TWO-LAYER FEED-FORWARD
NEURAL NETWORK
w1j
Inputs
w2k
Hidden
layer
Output
layer
31
NETWORKS WITH HIDDEN LAYERS
Can represent XORs, other nonlinear functions
Common neuron types:
Soft perceptron (sigmoid), radial basis functions,
linear, …
As the number of hidden units increase, so does
the network’s capacity to learn functions with
more nonlinear features
How to train hidden layers?
32
BACKPROPAGATION (PRINCIPLE)
Treat the problem as one of minimizing errors
between the example label and the network
output, given the example and network weights
as input
Sum this error term over all examples
Error(xi,yi,w) = (yi – f(xi,w))2
E(w) = Si Error(xi,yi,w) = Si (yi – f(xi,w))2
Minimize errors using an optimization algorithm
Stochastic gradient descent is typically used
33
Gradient direction 𝛻𝐸 is orthogonal to the level sets (contours) of E,
points in direction of steepest increase
Gradient direction 𝛻𝐸 is orthogonal to the level sets (contours) of E,
points in direction of steepest increase
Gradient descent: iteratively move in direction −𝛻E
Gradient descent: iteratively move in direction −𝛻E
Gradient descent: iteratively move in direction −𝛻E
Gradient descent: iteratively move in direction −𝛻E
Gradient descent: iteratively move in direction −𝛻E
Gradient descent: iteratively move in direction −𝛻𝐸
Gradient descent: iteratively move in direction −𝛻𝐸
STOCHASTIC GRADIENT DESCENT
For each example (xi,yi), take a gradient descent
step to reduce the error for (xi,yi) only.
43
STOCHASTIC GRADIENT DESCENT
Objective function values (measured over all
examples) over time settle into local minimum
Step size must be reduced over time, e.g., O(1/t)
44
NEURAL NETWORKS: PROS AND CONS
Pros
Bioinspiration is nifty
Can represent a wide variety of decision boundaries
Complexity is easily tunable (number of hidden
nodes, topology)
Easily extendable to regression tasks
Cons
Haven’t gotten close to unlocking the power of the
human (or cat) brain
Complex boundaries need lots of data
Slow training
Mostly lukewarm feelings in mainstream ML
(although the “deep learning” variant is en vogue
now)
NEXT CLASS
Another guest lecture
0
