Support Vector Machines
CS 6243 Machine Learning
Modified from the slides by Dr. Raymond J. Mooney
http://www.cs.utexas.edu/~mooney/cs391L/
And by Dr. Andrew W. Moore
http://www.cs.cmu.edu/~awm/tutorials
1
Perceptron Revisited: Linear Separators
• Binary classification can be viewed as the task of
separating classes in feature space:
wTx + b > 0
wTx + b = 0
Classification function:
f(x) = sign(wTx + b)
wTx + b < 0
2
Linear Separators
• Which of the linear separators is optimal?
• Need to find a decision boundary:
ax + by − c = 0
• Lots of possible solutions for a, b, c.
• Some methods find a separating
hyperplane, but not the optimal one
• E.g., perceptron
• Support Vector Machine (SVM) finds
an optimal solution.
3
Maximum Margin Classification
• SVMs maximize the margin around the separating hyperplane.
• A.k.a. large margin classifiers
• Efficient algorithms to find optimal solution
• Elegantly extensions to handle non-linearly-separable data
• Currently widely seen as one of the best classification methods.
wTx + b = 0
wTx + b > ρ/2
ρ/2 ρ/2
Classification function:
f(x) = sign(wTx + b)
wTx + b < -ρ/2
4
Why Maximum Margin?
1. Intuitively this feels safest.
2. If we’ve made a small error in the
location of the boundary (it’s been
jolted in its perpendicular direction)
this gives us least chance of causing a
misclassification.
3. LOOCV is easy since the model is
immune to removal of any nonsupport-vector datapoints.
4. There’s some theory (using VC
dimension) that is related to (but not
the same as) the proposition that this
is a good thing.
5. Empirically it works very very well.
5
Classification Margin
wT xi  b
r
w
• Distance from example xi to the separator is
• Examples closest to the hyperplane are support vectors.
• Margin ρ of the separator is the distance between support vectors.
ρ
x
r
x′
Derivation of finding r:
Dotted line x’−x is perpendicular to
decision boundary so parallel to w.
Unit vector is w/|w|, so line is rw/|w|.
x’ = x – yrw/|w|.
x’ satisfies wTx’+b = 0.
So wT(x –yrw/|w|) + b = 0
Recall that |w| = sqrt(wTw).
So, solving for r gives:
r = y(wTx + b)/|w|
6
Linear SVM Mathematically
• Let training set {(xi, yi)}i=1..n, xiRd, yi  {-1, 1} be separated by a
hyperplane with margin ρ. Then for each training example (xi, yi):
wTxi + b ≤ - ρ/2 if yi = -1

wTxi + b ≥ ρ/2 if yi = 1
yi(wTxi + b) ≥ ρ/2
• For every support vector xs the above inequality is an equality.
After rescaling w and b by ρ/2 in the equality, we obtain
that
T
distance between each xs and the hyperplane is r  ys (w xs  b) 
w
1
w
• Then the margin can be expressed through (rescaled) w and b as:
  2r 
2
w
7
Linear SVMs Mathematically (cont.)
• Then we can formulate the quadratic optimization problem:
Find w and b such that

2
w
is maximized
and for all (xi, yi), i=1..n :
yi(wTxi + b) ≥ 1
Which can be reformulated as:
Find w and b such that
Φ(w) = ||w||2=wTw is minimized
and for all (xi, yi), i=1..n :
yi (wTxi + b) ≥ 1
8
Solving the Optimization Problem
Find w and b such that
Φ(w) =wTw is minimized
and for all (xi, yi), i=1..n :
yi (wTxi + b) ≥ 1
• Need to optimize a quadratic function subject to linear constraints.
• Quadratic optimization problems are a well-known class of mathematical
programming problems for which several (non-trivial) algorithms exist.
• The solution involves constructing a dual problem where a Lagrange
multiplier αi is associated with every inequality constraint in the primal
(original) problem:
Find α1…αn such that
Q(α) =Σαi - ½ΣΣαiαjyiyjxiTxj is maximized and
(1) Σαiyi = 0
(2) αi ≥ 0 for all αi
9
The Optimization Problem Solution
• Given a solution α1…αn to the dual problem, solution to the primal is:
w =Σαiyixi
b = yk - Σαiyixi Txk for any αk > 0
• Each non-zero αi indicates that corresponding xi is a support vector.
• Then the classifying function is (note that we don’t need w explicitly):
f(x) = ΣαiyixiTx + b
• Notice that it relies on an inner product between the test point x and the
support vectors xi – we will return to this later.
• Also keep in mind that solving the optimization problem involved
computing the inner products xiTxj between all training points.
10
Soft Margin Classification
• What if the training set is not linearly separable?
• Slack variables ξi can be added to allow misclassification of difficult or
noisy examples, resulting margin called soft.
ξj
ξi
11
Soft Margin Classification Mathematically
• The old formulation:
Find w and b such that
Φ(w) =wTw is minimized
and for all (xi ,yi), i=1..n :
yi (wTxi + b) ≥ 1
• Modified formulation incorporates slack variables:
Find w and b such that
Φ(w) =wTw + CΣξi is minimized
and for all (xi ,yi), i=1..n :
yi (wTxi + b) ≥ 1 – ξi, ,
ξi ≥ 0
• Parameter C can be viewed as a way to control overfitting: it “trades off”
the relative importance of maximizing the margin and fitting the training
data.
12
Soft Margin Classification – Solution
• Dual problem is identical to separable case (would not be identical if the 2norm penalty for slack variables CΣξi2 was used in primal objective, we
would need additional Lagrange multipliers for slack variables):
Find α1…αN such that
Q(α) =Σαi - ½ΣΣαiαjyiyjxiTxj is maximized and
(1) Σαiyi = 0
(2) 0 ≤ αi ≤ C for all αi
• Again, xi with non-zero αi will be support vectors.
• Solution to the dual problem is:
Again, we don’t need to
w =Σαiyixi
b= yk(1- ξk) - ΣαiyixiTxk for any k s.t. αk>0
compute w explicitly for
classification:
f(x) = ΣαiyixiTx + b
13
*Theoretical Justification for Maximum Margins
• Vapnik has proved the following:
The class of optimal linear separators has VC dimension h bounded from
above as
 D 2 

h  min 2 , m0   1
  

where ρ is the margin, D is the diameter of the smallest sphere that can
enclose all of the training examples, and m0 is the dimensionality.
• Intuitively, this implies that regardless of dimensionality m0 we can
minimize the VC dimension by maximizing the margin ρ.
• Thus, complexity of the classifier is kept small regardless of
dimensionality.
14
Linear SVMs: Overview
• The classifier is a separating hyperplane.
• Most “important” training points are support vectors; they define the
hyperplane.
• Quadratic optimization algorithms can identify which training points xi are
support vectors with non-zero Lagrangian multipliers αi.
• Both in the dual formulation of the problem and in the solution training
points appear only inside inner products:
Find α1…αN such that
Q(α) =Σαi - ½ΣΣαiαjyiyjxiTxj is maximized and
(1) Σαiyi = 0
(2) 0 ≤ αi ≤ C for all αi
f(x) = ΣαiyixiTx + b
15
Non-linear SVMs
• Datasets that are linearly separable with some noise work out great:
x
0
• But what are we going to do if the dataset is just too hard?
x
0
• How about… mapping data to a higher-dimensional space:
x2
z  ( x, x )
2
0
x
16
Non-linear SVMs: Feature spaces
• General idea: the original feature space can be mapped to some higherdimensional feature space where the training set is separable:
x2
x22
Φ: x → φ(x)
x1
x1
2 x1 x2
x  ( x1, x2 )
z  ( x , x , 2 x1x2 )
2
1
2
2
17
1




2 x1 


2 x2 


:



2 xd 


2
x1




2
x
2




:


2
x
d

Φ(x )  
 2 x1 x2 


 2 x1 x3 


:


 2 x1 xd 
 2x x 
2 3


:


 2x x 
1 d


:


 2x x 
d 1 d 

Constant Term
Linear Terms
Quadratic Basis Functions
Number of terms (assuming d input
dimensions) = (d+2)-choose-2
Pure
Quadratic
Terms
Quadratic
Cross-Terms
= (d+2)(d+1)/2
= (as near as makes no difference) d2/2
Potential problems with highdimensional feature space:
• Overfitting
• Separation in high
dimensional space is trivial
• High computational cost
18
Quadratic Dot Products
1
1

 


 

2a1  
2b1 


2a2   2b2 

 

:
:

 


2ad   2bd 

 

2
2
 a1
  b1

 a 2   b2 
2
2

 


 

:
:

 

2
2
a
b
d
d


Φ(a)  Φ(b)  
 2a1a2   2b1b2 

 

 2a1a3   2b1b3 

 

:
:

 

 2a1ad   2b1bd 
 2 a a   2b b 
2 3
2 3

 

:
:

 

 2 a a   2b b 
1 d
1 d

 

:
:

 

 2 a a   2b b 
d 1 d  
d 1 d 

1
+
d
 2a b
i i
i 1
+
d
2 2
a
 i bi
i 1
+
d
d
  2a a b b
i 1 j i 1
i
j i
j
O(d2) to compute
19
Quadratic Dot Products
(a.b  1) 2
 (a.b) 2  2a.b  1
m


   ai bi   2 ai bi  1
i 1
 i 1

Φ(a)  Φ(b) 
m
m
m
m
m
1  2 ai bi   a b    2ai a j bi b j
i 1
2
m
i 1
2 2
i i
m
m
  ai bi a j b j  2 ai bi  1
i 1 j 1
i 1
i 1 j i 1
m
m
m
m
  (ai bi )  2  ai bi a j b j  2 ai bi  1
2
i 1
i 1 j i 1
i 1
20
Quadratic Dot Products
(a.b  1) 2
 (a.b) 2  2a.b  1
m


   ai bi   2 ai bi  1
i 1
 i 1

Φ(a)  Φ(b) 
m
m
m
m
m
1  2 ai bi   a b    2ai a j bi b j
i 1
2
m
i 1
2 2
i i
m
m
  ai bi a j b j  2 ai bi  1
i 1 j 1
i 1
i 1 j i 1
m
m
m
m
  (ai bi )  2  ai bi a j b j  2 ai bi  1
2
i 1
i 1 j i 1
i 1
They’re the same!
And this is only O(d) to
compute!
21
Higher Order Polynomials
R: number of instances
m: number of attributes
Poly-nomial
f(x)
Cost to
build Qkl
matrix
traditionally
Cost if 100 inputs
f(a).f(b)
Cost to
build Qkl
matrix
sneakily
Cost if
100
inputs
Quadratic
All d2/2 terms
up to degree 2
m2 R2 /4
2,500 R2
(a.b+1)2
m R2 / 2
50 R2
Cubic
All d3/6 terms
up to degree 3
m3 R2 /12
83,000 R2
(a.b+1)3
m R2 / 2
50 R2
Quartic
All d4/24 terms
up to degree 4
m4 R2 /48
1,960,000 R2
(a.b+1)4
m R2 / 2
50 R2
22
The “Kernel Trick”
• The linear classifier relies on inner product between vectors K(xi,xj)=xiTxj
• If every datapoint is mapped into high-dimensional space via some
transformation Φ: x → φ(x), the inner product becomes:
K(xi,xj)= φ(xi) Tφ(xj)
• A kernel function is a function that is eqiuvalent to an inner product in
some feature space.
• Example:
2-dimensional vectors x=[x1 x2]; let K(xi,xj)=(1 + xiTxj)2,
Need to show that K(xi,xj)= φ(xi) Tφ(xj):
K(xi,xj)=(1 + xiTxj)2,= 1+ xi12xj12 + 2 xi1xj1 xi2xj2+ xi22xj22 + 2xi1xj1 + 2xi2xj2=
= [1 xi12 √2 xi1xi2 xi22 √2xi1 √2xi2]T [1 xj12 √2 xj1xj2 xj22 √2xj1 √2xj2] =
= φ(xi) Tφ(xj), where φ(x) = [1 x12 √2 x1x2 x22 √2x1 √2x2]
• Thus, a kernel function implicitly maps data to a high-dimensional space
(without the need to compute each φ(x) explicitly).
23
Examples of Kernel Functions
• Linear: K(xi,xj)= xiTxj
– Mapping Φ:
x → φ(x), where φ(x) is x itself
• Polynomial of power p: K(xi,xj)= (1+ xiTxj)p
– Mapping Φ:
x → φ(x), where φ(x) has
 d  p  dimensions


 p 
• Gaussian (radial-basis function): K(xi,xj) = e

xi x j
2
2 2
– Mapping Φ: x → φ(x), where φ(x) is infinite-dimensional: every point is
mapped to a function (a Gaussian); combination of functions for support
vectors is the separator.
• Higher-dimensional space still has intrinsic dimensionality d (the mapping
is not onto), but linear separators in it correspond to non-linear separators
in original space.
24
Non-linear SVMs Mathematically
• Dual problem formulation:
Find α1…αn such that
Q(α) =Σαi - ½ΣΣαiαjyiyjK(xi, xj) is maximized and
(1) Σαiyi = 0
(2) αi ≥ 0 for all αi
• The solution is:
f(x) = ΣαiyiK(xi, xj)+ b
• Optimization techniques for finding αi’s remain the same!
25
SVM applications
• SVMs were originally proposed by Boser, Guyon and Vapnik in 1992 and
gained increasing popularity in late 1990s.
• SVMs are currently among the best performers for a number of classification
tasks ranging from text to genomic data.
• SVMs can be applied to complex data types beyond feature vectors (e.g. graphs,
sequences, relational data) by designing kernel functions for such data.
• SVM techniques have been extended to a number of tasks such as regression
[Vapnik et al. ’97], principal component analysis [Schölkopf et al. ’99], etc.
• Most popular optimization algorithms for SVMs use decomposition to hillclimb over a subset of αi’s at a time, e.g. SMO [Platt ’99] and [Joachims ’99]
• Tuning SVMs remains a black art: selecting a specific kernel and parameters is
usually done in a try-and-see manner.
26
SVM software and resources
• SVM software
– LibSVM (C++)
– SVMLight (C)
• As well as complete machine learning toolboxes that include SVMs:
– Torch (C++)
– Spider (Matlab)
– Weka (Java)
• Useful websites
– www.kernel-machines.org
– svms.org
27