Support Vector Machine
Classifiers
Based on lecture slides by
Glenn Fung, O. L. Mangasarian,
Rob Lothian, and Kristin Bennett
1
Outline
• Support Vector Machines for Classification
– Linear Discrimination
– Nonlinear Discrimination
• SVM Mathematically
• Extensions
• Application in Drug Design
• Data Classification
• Kernel Functions
2
Definition
• ‘Support Vector Machine is a system for
efficiently training linear learning machines in
kernel-induced feature spaces, while respecting
the insights of generalisation theory and
exploiting optimisation theory.’
– AN INTRODUCTION TO SUPPORT VECTOR
MACHINES (and other kernel-based
learning methods)
N. Cristianini and J. Shawe-Taylor
Cambridge University Press
2000 ISBN: 0 521 78019 5
– Kernel Methods for Pattern Analysis
John Shawe-Taylor & Nello Cristianini
Cambridge University Press, 2004
3
The Scalar Product
a
q
b
a  b  a b cosq
The scalar or dot product is, in some sense, a
measure of Similarity
4
Decision Function
for binary classification
f (x )  R
f ( xi )  0  yi  1
f xi   0  yi  1
5
Support Vector Machines
• SVMs pick best separating hyperplane according to
some criterion
– e.g. maximum margin
• Training process is an optimisation
• Training set is effectively reduced to a relatively
small number of support vectors
6
Feature Spaces
• We may separate data by mapping to a higherdimensional feature space
– The feature space may even have an infinite
number of dimensions!
• We need not explicitly construct the new feature
space
7
Kernels
• We may use Kernel functions to implicitly map to a
new feature space
• Kernel fn:
K x1 , x2  R
• Kernel must be equivalent to an inner product in
some feature space
8
Example Kernels
xz
Linear:
P x  z
Polynomial:
Gaussian:


exp  x  z /  2
2

9
Perceptron Revisited: Linear Separators
• Binary classification can be viewed as the task of
separating classes in feature space:
wTx + b = 0
wTx + b > 0
wTx + b < 0
f(x) = sign(wTx + b)
10
Which of the linear separators is optimal?
11
Best Linear Separator?
12
Best Linear Separator?
13
Best Linear Separator?
14
Best Linear Separator?
15
Find Closest Points in Convex Hulls
d
c
16
Plane Bisect Closest Points
wT x + b =0
w=d-c
d
c
17
Classification Margin
wT x  b
r
w
• Distance from example data to the separator is
• Data closest to the hyperplane are support vectors.
• Margin ρ of the separator is the width of separation between
classes.
ρ
r
18
Maximum Margin Classification
• Maximizing the margin is good according to intuition and
theory.
• Implies that only support vectors are important; other training
examples are ignorable.
19
Statistical Learning Theory
• Misclassification error and the function complexity
bound generalization error.
• Maximizing margins minimizes complexity.
• “Eliminates” overfitting.
• Solution depends only on Support Vectors not number
of attributes.
20
Margins and Complexity
Skinny margin
is more flexible
thus more complex.
21
Margins and Complexity
Fat margin
is less complex.
22
Linear SVM Mathematically
• Assuming all data is at distance larger than 1 from the
hyperplane, the following two constraints follow for a
training set {(xi ,yi)}
wTxi + b ≥ 1 if yi = 1
wTxi + b ≤ -1 if yi = -1
• For support vectors, the inequality becomes an
equality; then, since each example’s distance from the
• hyperplane is
wT x  b
r
w
the margin is:
2

w
23
Linear SVMs Mathematically (cont.)
• Then we can formulate the quadratic optimization problem:
Find w and b such that

2
w
is maximized and for all {(xi ,yi)}
wTxi + b ≥ 1 if yi=1; wTxi + b ≤ -1 if yi = -1
A better formulation:
Find w and b such that
Φ(w) =½ wTw is minimized and for all {(xi ,yi)}
yi (wTxi + b) ≥ 1
24
Solving the Optimization Problem
Find w and b such that
Φ(w) =½ wTw is minimized and for all {(xi ,yi)}
yi (wTxi + b) ≥ 1
• Need to optimize a quadratic function subject to linear constraints.
• Quadratic optimization problems are a well-known class of mathematical
programming problems, and many (rather intricate) algorithms exist for
solving them.
• The solution involves constructing a dual problem where a Lagrange
multiplier αi is associated with every constraint in the primary problem:
Find α1…αN such that
Q(α) =Σαi - ½ΣΣαiαjyiyjxiTxj is maximized and
(1) Σαiyi = 0
(2) αi ≥ 0 for all αi
25
The Optimization Problem Solution
• The solution has the form:
w =Σαiyixi
b= yk- wTxk for any xk such that αk 0
• Each non-zero αi indicates that corresponding xi is a support vector.
• Then the classifying function will have the form:
f(x) = ΣαiyixiTx + b
• Notice that it relies on an inner product between the test point x and the
support vectors xi – we will return to this later!
• Also keep in mind that solving the optimization problem involved
computing the inner products xiTxj between all training points!
26
Soft Margin Classification
• What if the training set is not linearly separable?
• Slack variables ξi can be added to allow misclassification of difficult or
noisy examples.
ξi
ξi
27
Soft Margin Classification Mathematically
• The old formulation:
Find w and b such that
Φ(w) =½ wTw is minimized and for all {(xi ,yi)}
yi (wTxi + b) ≥ 1
• The new formulation incorporating slack variables:
Find w and b such that
Φ(w) =½ wTw + CΣξi is minimized and for all {(xi ,yi)}
yi (wTxi + b) ≥ 1- ξi and ξi ≥ 0 for all i
• Parameter C can be viewed as a way to control overfitting.
28
Soft Margin Classification – Solution
• The dual problem for soft margin classification:
Find α1…αN such that
Q(α) =Σαi - ½ΣΣαiαjyiyjxiTxj is maximized and
(1) Σαiyi = 0
(2) 0 ≤ αi ≤ C for all αi
• Neither slack variables ξi nor their Lagrange multipliers appear in the dual
problem!
• Again, xi with non-zero αi will be support vectors.
• Solution to the dual problem is:
But neither w nor b are
needed explicitly for
w =Σαiyixi
classification!
b= yk(1- ξk) - wTxk where k = argmax αk
k
f(x) = ΣαiyixiTx + b
29
Theoretical Justification for Maximum Margins
• Vapnik has proved the following:
The class of optimal linear separators has VC dimension h bounded from
above as
 D 2 

h  min  2 , m0   1
  

where ρ is the margin, D is the diameter of the smallest sphere that can
enclose all of the training examples, and m0 is the dimensionality.
• Intuitively, this implies that regardless of dimensionality m0 we can
minimize the VC dimension by maximizing the margin ρ.
• Thus, complexity of the classifier is kept small regardless of
dimensionality.
30
Linear SVMs: Overview
• The classifier is a separating hyperplane.
• Most “important” training points are support vectors; they define the
hyperplane.
• Quadratic optimization algorithms can identify which training points xi are
support vectors with non-zero Lagrangian multipliers αi.
• Both in the dual formulation of the problem and in the solution training
points appear only inside inner products:
Find α1…αN such that
Q(α) =Σαi - ½ΣΣαiαjyiyjxiTxj is maximized and
(1) Σαiyi = 0
(2) 0 ≤ αi ≤ C for all αi
f(x) = ΣαiyixiTx + b
31
Non-linear SVMs
• Datasets that are linearly separable with some noise work out great:
x
0
• But what are we going to do if the dataset is just too hard?
x
0
• How about… mapping data to a higher-dimensional space:
x2
0
x
32
Nonlinear Classification
x   a, b 
x w  w1a  w2b

q ( x)   a, b, ab, a , b 
2
2
q ( x) w  w1a  w2b  w3ab  w4 a  w5b
2
2
33
Non-linear SVMs: Feature spaces
• General idea: the original feature space can always be mapped to some
higher-dimensional feature space where the training set is separable:
Φ: x → φ(x)
34
The “Kernel Trick”
• The linear classifier relies on inner product between vectors K(xi,xj)=xiTxj
• If every datapoint is mapped into high-dimensional space via some
transformation Φ: x → φ(x), the inner product becomes:
K(xi,xj)= φ(xi) Tφ(xj)
• A kernel function is some function that corresponds to an inner product into
some feature space.
• Example:
2-dimensional vectors x=[x1 x2]; let K(xi,xj)=(1 + xiTxj)2,
Need to show that K(xi,xj)= φ(xi) Tφ(xj):
K(xi,xj)=(1 + xiTxj)2,= 1+ xi12xj12 + 2 xi1xj1 xi2xj2+ xi22xj22 + 2xi1xj1 + 2xi2xj2=
= [1 xi12 √2 xi1xi2 xi22 √2xi1 √2xi2]T [1 xj12 √2 xj1xj2 xj22 √2xj1 √2xj2] =
= φ(xi) Tφ(xj), where φ(x) = [1 x12 √2 x1x2 x22 √2x1 √2x2]
35
36
Positive Definite Matrices
A square matrix A is positive definite if
xTAx>0 for all nonzero column vectors
x.
It is negative definite if xTAx < 0 for all
nonzero x.
It is positive semi-definite if xTAx  0.
And negative semi-definite if xTAx  0
for all x.
37
What Functions are Kernels?
• For some functions K(xi,xj) checking that K(xi,xj)= φ(xi) Tφ(xj) can be
cumbersome.
• Mercer’s theorem:
Every semi-positive definite symmetric function is a kernel
• Semi-positive definite symmetric functions correspond to a semi-positive
definite symmetric Gram matrix:
K(x1,x1)
K(x1,x2)
K(x1,x3)
K(x2,x1)
K(x2,x2)
K(x2,x3)
…
K(x1,xN)
K(x2,xN)
K=
…
…
…
…
…
K(xN,x1)
K(xN,x2)
K(xN,x3)
…
K(xN,xN)
38
Examples of Kernel Functions
• Linear: K(xi,xj)= xi Txj
• Polynomial of power p: K(xi,xj)= (1+ xi Txj)p
• Gaussian (radial-basis function network): K(xi,xj)= e

xi x j
2
2 2
• Two-layer perceptron: K(xi,xj)= tanh(β0xi Txj + β1)
39
Non-linear SVMs Mathematically
• Dual problem formulation:
Find α1…αN such that
Q(α) =Σαi - ½ΣΣαiαjyiyjK(xi, xj) is maximized and
(1) Σαiyi = 0
(2) αi ≥ 0 for all αi
• The solution is:
f(x) = ΣαiyiK(xi, xj)+ b
• Optimization techniques for finding αi’s remain the same!
40
SVM applications
• SVMs were originally proposed by Boser, Guyon and Vapnik in 1992 and
gained increasing popularity in late 1990s.
• SVMs are currently among the best performers for a number of
classification tasks ranging from text to genomic data.
• SVM techniques have been extended to a number of tasks such as
regression [Vapnik et al. ’97], principal component analysis [Schölkopf et
al. ’99], etc.
• Most popular optimization algorithms for SVMs are SMO [Platt ’99] and
SVMlight [Joachims’ 99], both use decomposition to hill-climb over a subset
of αi’s at a time.
• Tuning SVMs remains a black art: selecting a specific kernel and
parameters is usually done in a try-and-see manner.
41
SVM Extensions
•
•
•
•
•
Regression
Variable Selection
Boosting
Density Estimation
Unsupervised Learning
– Novelty/Outlier Detection
– Feature Detection
– Clustering
42
Example in Drug Design
• Goal to predict bio-reactivity of molecules to
decrease drug development time.
• Target is to predict the logarithm of inhibition
concentration for site "A" on the Cholecystokinin
(CCK) molecule.
• Constructs quantitative structure activity relationship
(QSAR) model.
43
LCCKA Problem
• Training data – 66 molecules
• 323 original attributes are wavelet coefficients of TAE
Descriptors.
• 39 subset of attributes selected by linear 1-norm SVM (with no
kernels).
• For details see DDASSL project link off of
http://www.rpi.edu/~bennek
• Testing set results reported.
44
LCCK Prediction
LCCKA Test Set Estimates
1
Predicted Value
0.8
0.6
0.4
0.2
0
0
0.1
0.2 0.3
0.4 0.5 0.6
True Value
0.7 0.8
0.9
1
45
Many Other Applications
•
•
•
•
•
•
•
•
Speech Recognition
Data Base Marketing
Quark Flavors in High Energy Physics
Dynamic Object Recognition
Knock Detection in Engines
Protein Sequence Problem
Text Categorization
Breast Cancer Diagnosis
• Cancer Tissue classification
• Translation initiation site recognition in DNA
• Protein fold recognition
46
Hallelujah!
•
•
•
•
•
•
•
Generalization theory and practice meet
General methodology for many types of problems
Same Program + New Kernel = New method
No problems with local minima
Few model parameters. Selects capacity
Robust optimization methods
Successful Applications
BUT…
47
HYPE?
•
•
•
•
•
•
•
Will SVMs beat my best hand-tuned method Z for X?
Do SVM scale to massive datasets?
How to chose C and Kernel?
What is the effect of attribute scaling?
How to handle categorical variables?
How to incorporate domain knowledge?
How to interpret results?
48
Support Vector Machine Resources
• SVM Application List
http://www.clopinet.com/isabelle/Projects/SVM/applist.html
• Kernel machines
http://www.kernel-machines.org/
• Pattern Classification and Machine Learning
http://clopinet.com/isabelle/#projects
• R a GUI language for statistical computing and graphics
http://www.r-project.org/
•
Kernel Methods for Pattern Analysis – 2004
http://www.kernel-methods.net/
• An Introduction to Support Vector Machines
(and other kernel-based learning methods)
http://www.support-vector.net/
• Kristin P. Bennett web page
http://www.rpi.edu/~bennek
• Isabelle Guyon's home page
http://clopinet.com/isabelle
49
Support Vector Machine References
•
•
•
•
•
•
•
•
•
•
•
•
•
Duda R.O. and Hart P.E.; Patter Classification and Scene Analysis. Wiley, 1973.
T.M. Cover. Geometrical and statistical properties of systems of linear inequalities with applications in
pattern recognition. IEEE Transactions on Electronic Computers}, 14:326--334, 1965.
V.Vapnik and A.Lerner. Pattern recognition using generalized portrait method. Automation and Remote
Control}, 24, 1963.
V.Vapnik and A.Chervonenkis. A note on one class of perceptrons. Automation and Remote Control}, 25,
1964.
J.K. Anlauf and M.Biehl. The adatron: an adaptive perceptron algorithm. Europhysics Letters, 10:687-692, 1989.
N.Aronszajn. Theory of reproducing kernels. Transactions of the American Mathematical Society, 68:337-404, 1950.
M.Aizerman, E.Braverman, and L.Rozonoer.Theoretical foundations of the potential function method in
pattern recognition learning. Automation and Remote Control 25:821--837, 1964.
O. L. Mangasarian. Linear and nonlinear separation of patterns by linear programming. Operations
Research , 13:444--452, 1965.
F. W. Smith. Pattern classifier design by linear programming. IEEE Transactions on Computers , C17:367--372, 1968.
C.Cortes and V.Vapnik. Support vector networks. Machine Learning, 20:273--297, 1995.V.Vapnik. The
Nature of Statistical Learning Theory}. Springer Verlag, 1995.
V.Vapnik. Statistical Learning Theory}. Wiley, 1998.A.N. Tikhonov and V.Y. Arsenin. Solutions of Illposed Problems. W. H. Winston, 1977.
B.Schoelkopf, C.J.C. Burges, and A.J. Smola, Advances in kernel methods - support vector learning, MIT
Press, Cambridge, MA, 1999.
A.J. Smola, P.Bartlett, B.Schoelkopf, and C.Schuurmans, Advances in large margin classifiers, MIT Press,
Cambridge, MA, 1999.
50
Unlabeled Data Classification
• Given a large unlabeled dataset
• Use a k-Median clustering algorithm to select a
small (5% to 10%) representative sample.
• Representative sample is labeled by expert.
• Combined labeled-unlabeled dataset is classified
by a Semi-supervised Support Vector Machine.
• Test set correctness within 5.2% of a linear
support vector machine trained on the entire
dataset labeled by an expert.
51
Data Selection for Support Vector
Machines Classifiers
• Extract a minimal set of data points from a given
dataset.
• Minimal set used to generate a Minimal Support
Vector Machine (MSVM) classifier.
• MSVM classifier as good or better than that
obtained by training on entire dataset.
• Feature selection is incorporated into procedure to
obtain a minimal set of input features.
• Data reduction as high as 81% and averaged 66%
over seven public datasets.
52
Unlabeled Data Classification
• Given a completely unlabeled large data set.
– Costly to label points by an expert or an oracle.
• Two Question arise:
– How to choose a small subset for labeling?
– How to combine labeled and unlabeled data?
• Answers:
– Use k-median clustering for selecting
“representative” points to be labeled.
– Use semi-supervised SVM to obtain a classifier
based on labeled and unlabeled data.
53
Unlabeled Data Classification
Unlabeled Data Set
k-Median
clustering
Chosen
Data
Remaining
Data
Expert
Labeled
Data
Semi-supervised
SVM
Separating Plane
54
K-Median Clustering Algorithm
• Given m data points.
• Find k clusters of these points such that the sum of
the 1-norm distances from each point to the closest
cluster center is minimized.
55
K-Median Clustering Algorithm
*
*
*
*
56
K-Median Clustering Algorithm
57
Shortcomings of
the Existing Clustering Methods
• Parametric approaches
– pre-assuming number of clusters
• Non-Parametric approaches
– sensitive to the control parameters used in
the algorithm
– dependent on the distribution of data
– not able to accurately approximate the
overlapped distributions
58
Nonparametric Clustering Problem Description
the sample data
Gaussian approximation of the whole class
multi-Gaussian approximation
3
2
1
y
0
-1
-2
-3
-2
-1
0
x
1
2
3
59
Data Density Clustering (DDC)
Algorithm
Extraction of the seed clusters
 Coverage of the training data signals by the
seed clusters
 Size of the seed cluster
Merging of the clusters with a similar
distribution
60
Cluster Growth Program
61
Distribution of Minimum Distance (MD)
in 2D space
the distribution of MD with n=400
0.09
0.08
0.07
the PDF of MD
0.06
0.05
0.04
0.03
0.02
0.01
0
0
0.02
0.04
0.06
Y
0.08
0.1
the MD value
0.12
0.14
0.16
62
Distribution of Average Minimum Distance
(AMD) in 2D space
the distribution of AMD with n=400
0.35
0.3
the PDF of AMD
0.25
0.2
0.15
0.1
0.05
0
0
0.005
0.01
0.015
the AMD value
0.02
Z
0.025
0.03
X
63
Definition of Growth Threshold
MD - Minimum Distance
AMD - Average Minimum Distance
X - the mean value of AMD
Z - the lower bound of AMD
Y - the upper bound of MD
Growth Threshold 

MminD
Y
Z
Y
 MminD  GR  MminD
Z
Where MminD is the mean of the minimum
distances in the current cluster.
and
Growth Ratio (Ratio)
GR=Y/Z
64
Growth Ratio in m-Dimensional Space
m=1,…,4
65
The Cluster Merging Algorithm
• Focus - how to identify the clusters whose distributions are
consistent?
The Non-overlapped area (NOA)
1.4
Non_overlapped area
1.2
1
pdf
pdfM
0.8
pdfW
pdfA
0.6
0.4
pdfB
0.2
0
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
66
The Touch Point of Two Clusters
The parameters
affecting the NOA
• Distance - the
distance between
the means of two
clusters
• R_size - the
ratio of two
clusters’ sizes
67
NOA with different Distance and R_size
68
Merging Threshold vs. R_size
0.7
the ANO vs R_size when
two clusters touch each other
the linear estimate
0.6
0.5
0.4
0.3
0.2
0.1
0
0
2
4
6
8
10
12
14
When the two groups of data just
touch each other, the corresponding NOA
value is used as the threshold for merging
16
18
20
69
Merging of Clusters
• Using the threshold to
decide whether to
merge the clusters in
the same class
• Using the NOA value
to the nearest cluster
from a different class
to terminate the
selection for the
current cluster
70
Sample Clustering Results
71
Definition of Cluster Based Classifier
72
The Misclassification Ratio With and Without
Clustering
73
Unlabeled Data Classification
Unlabeled Data Set
k-Median
clustering
Chosen
Data
Remaining
Data
Expert
Labeled
Data
Semi-supervised
SVM
Separating Plane
74
Semi-supervised SVM (S3VM)
Separate Triangles & Circles
Hollow shapes represent
labeled data
Solid shapes represent
unlabeled data
SVM
S3VM
75
76
77
Numerical Tests
78
Part 2: Data Selection for Support Vector
Machines Classifiers
Labeled dataset
1-norm SVM
feature selection
Support vector
suppression
MSVM
Smaller dimension
dataset
Separating surface
79
Support Vectors
80
Feature Selection using
1-norm Linear SVM ( ö small.)
81
Motivation for the Minimal
Support Vector Machine (MSVM)
82
Motivation for the Minimal
Support Vector Machine (MSVM)
• Suppression of error term y:
– Minimizes the number of misclassified points.
– Works remarkably well computationally.
– Reduces positive components of multiplier u and
hence number of support vectors.
83
MSVM Formulation
84
MSVM Formulation
85
86
Numerical Tests
87
Conclusions
• Unlabeled data classification:
– A fast finite linear programming based
approach for Semi-supervised Support Vector
Machines was proposed for classifying large
datasets that are mostly unlabeled.
– Totally unlabeled datasets were classified by:
• Labeling a small percentage of clusters by an expert
• Classification by a semi-supervised SVM
– Test set correctness within 5.2% of a linear
SVM trained on the entire dataset labeled by an
expert.
88
Conclusions
• Data selection for SVM classifiers:
– Minimal SVM (MSVM) extracts a minimal subset
used to classify the entire dataset.
– MSVM maintains or improves generalization over
other classifiers that use the entire dataset.
– Data reduction as high as 81%, and averaged 66% over
seven public datasets.
• Future work
– MSVM: Promising tool for incremental algorithms.
– Improve chunking algorithms with MSVM.
– Nonlinear MSVM: strong potential for time & storage
reduction.
89