CS364: Machine Learning
Semester 2- year 1444
Level 6 BSc CS
Support Vector Machines
Instructor:
Prof. Hachemi BinNaceur
Dr. Waad Alhoshan
Outlines
• Motivation -- Linear Support Vector Machines
• Basic Terminologies
• Decision Boundary: Maximum Margin
• Soft Margin and Misclassification
• Support Vectors
• Finding Optimal Margin: Optimization
• Non-Linear Classification: Kernel Trick
• Advantages & Disadvantages
5/16/2025
2
Motivation
• Simple classification problems could be solved with a separator line
that separates two classes from each other.
• Example: predicating the person BMI with a specific wights (Kg), and
select a threshold point to determine if the person is overweight or
not.
Overweight
Not overweight
Weights (Kg)
threshold
• What about if we have more data (i.e., more features) such as wight,
age, height, color, has kids or not, etc.?
• Mm! we need more powerful way to separate the classes.
• What about project the training points in a space and find a hyperplane …
okay.. That’s SVM
5/16/2025
3
Support Vector Machines
• Support Vector Machines (SVM): very popular approach for “off-theshelf” supervised learning.
• Proposed by Cortes Corinna and Vladimir Vapnick in 1995*.
• SVM= attractive model
• Main idea-class separation-based learning: find the best separation between
classes that is the decision boundary with the largest possible distance to the
data instances.
*Cortes, Corinna; Vapnik, Vladimir N. (1995). "Support-vector networks" (PDF).
Machine Learning. 20 (3): 273–297.
5/16/2025
4
Basic Terminologies with SVM
5/16/2025
5
Support Vectors
Key idea: some points are more important than others and taking them into
account can lead to better generalization.
Find the maximum margin
separator using support vectors.
Support vectors:
5/16/2025
6
Cont.
• Support vectors: data points that are closest to the separating plan.
• Support vectors: they hold up the separating plan.
• Support vectors: they are fewer than examples. SVM gain some
benefits of parametric models.
• Support vectors: using them can lead to better generalization.
• SVM address the issue: instead of minimizing expected empirical loss
on the training data, SVM try to minimize expected generalization
loss.
5/16/2025
7
How does SVM work?
• Main idea: How to find the best separation (i.e., hyperplane) between the
different classes in the feature space.
dénote +1
Equation of the hyperplan
w x + b>0
w x + b=0
dénote -1
W,b
X
w x + b<0
f
Y
f(x,w,b) = sign(w x + b)
Case of linear separation
Parameters to be optimized
5/16/2025
8
5/16/2025
9
Cont.
• With SVM, several separators (or hyperplanes) can be created and
one of them could represent the best or let’s say the maximum
margin.
• What is the best candidate decision boundary?
5/16/2025
10
Cont.
Supports vector
x+
M=Margin
X-
SVM create a linear separating
hyperplan.
Classifying data:
• w . x+ + b = +1
• w . x- + b = -1
• w . (x+-x-) = 2
5/16/2025
11
SVM Optimization
• Objective: maximize the margin M which means minimize|𝑤|
• Process: Classify the data points as follows:
• if wxi+b >=+1 Then class yi=+1
• if wxi+b <=-1 Then class yi=-1
• in both cases yi (wxi+b)>=1
5/16/2025
12
5/16/2025
13
Cont.
• Formulation: Optimization problem
𝐌𝐢𝐧𝐢𝐦𝐢𝐳𝐞 𝝋 𝑾 =
𝟏
𝟐
𝑾𝒕𝐖
𝑺𝒖𝒃𝒋𝒆𝒄𝒕 𝒕𝒐 ∶ 𝒚𝒊(𝑾𝒙𝒊 + 𝒃) ≥ 𝟏
That means it may have two variables multiplied by each
other, or the square of some variable
• Non-linear optimization problem (quadratic programming) for
which there exists a unique solution because the problem is
convex.
FYI .. A differential equation is an equation which
contains one or more terms and the derivatives of one
variable (i.e., dependent variable) with respect to the
other variable
Optimization in Machine Learning
• Optimization is a crucial step in practical machine learning
algorithms.
• In machine learning, an objective function and constraints are defined
over a set of data to which the algorithms have to find an optimum
solution
• Convex Function
• A convex function refers to a function whose graph is shaped like a
cup U
• A strictly convex function has exactly one local minimum point, which is
also the global minimum point
5/16/2025
15
Soft Margins
This idea is based on a simple premise: allow SVM to make a certain number of
mistakes and keep margin as wide as possible so that other points can still be
classified correctly. This can be done simply by modifying the objective of SVM, and
cross-validation!
Misclassified Point
5/16/2025
16
Support Vector Machines
• SVM considers to be a parametric model with linearly classified data,
what that means?
• Parametric model is a learning model that summarizes data with a
set of parameters of fixed size (independent of the number of
training examples).
• No matter how much data you push at a parametric model, it won’t
change its observance about how many parameters it needs.
• With parametric models to predict new data, we only need to know
the parameters of the model.
• So, basically it pays attention to the underlying data distribution
unlike the non- parametric model.
5/16/2025
17
Cont.
• SVM could benefit from non-parametric properties with non-linear
data as SVM
Kernel
Trick
• Can handle nonlinear separable classes efficiently.
Linearly separable data
5/16/2025
Non-Linearly separable data
18
Kernel Trick
• Often, data are not linearly separable in the input space and are easily separable
in the higher dimensional space.
• Case of non linear separation: Mapping data into a higher dimensional space.
Φ: x → φ(x)
2D space
3D space
Kernel Trick
5/16/2025
20
Support Vector Machines
• Advantages:
1. More efficient classifier than the other models.
2. Able to deal with high dimensional data.
3. Combine the advantages of non-parametric and parametric models.
They have the flexibility to represent complex functions and they are
resistant to overfitting.
4. They generalize well because of the use of a margin seperator.
• Disadvantages:
1. Difficult to understand and to interpret.
Support Vector Machines
• Extensions:




Multiclass SVM
Support-vector clustering (SVC)
Regression (SVR)
Transductive support-vector machines (semi-supervised learning)
References
• Kelleher, John D., Brian Mac Namee , and Aoife
Fundamentals of machine learning for predictive
data analytics: algorithms, worked examples, and
case studie s. MIT Press, 2015
23