SVMs in a Nutshell
What is an SVM?
• Support Vector Machine
– More accurately called support vector
classifier
– Separates training data into two classes so
that they are maximally apart
Simpler version
• Suppose the data is linearly separable
• Then we could draw a line between the
two classes
Simpler version
• But what is the best line? In SVM, we’ll
use the maximum margin hyperplane
Maximum Margin Hyperplane
What if it’s non-linear?
Higher dimensions
• SVM uses a kernel function to map the
data into a different space where it can
be separated
What if it’s not separable?
• Use linear separation, but allow training
errors
• This is called using a “soft margin”
• Higher cost for errors = creation of more
accurate model, but may not generalize
• Choice of parameters (kernel and cost)
determines accuracy of SVM model
• To avoid over- or under-fitting, use cross
validation to choose parameters
Some math
•
•
•
•
•
Data: {( x1, c1), (x2, c2), …, (xn, cn)}
xi is vector of attributes/features, scaled
ci is class of vector (-1 or +1)
Dividing hyperplane: wx - b = 0
Linearly separable means there exists a
hyperplane such that wxi - b > 0 if positive
example and wxi - b < 0 if negative example
• w points perpendicular to hyperplane
More math
• wx - b = 0
Support vectors
• wx - b = 1
• wx - b = -1
Distance between
hyperplanes is
2/|w|, so minimize
|w|
More math
•
•
•
•
For all i, either w xi - b  1 or wx - b  -1
Can be rewritten: ci(w xi - b)  1
Minimize (1/2)|w| subject to ci(w xi - b)  1
This is a quadratic programming problem
and can be solved in polynomial time
A few more details
• So far, assumed linearly separable
– To get to higher dimensions, use kernel function
instead of dot product; may be nonlinear transform
– Radial Basis Function is commonly used kernel:
k(x, x’) = exp(||x - x’||2) [need to choose ]
• So far, no errors; soft margin:
– Minimize (1/2)|w| + C i
– Subject to ci(w xi - b)  1 - i
– C is error penalty