Vapnik-Chevonenkis Dimension and its use in machine learning
We have some data with real world observations. Given an input of the data, the resulting observations depend on a function 𝑐 (𝑥) ∈ 𝐶. By using the data and training a model, we try to
define a function ℎ(𝑥) ∈ 𝐻, an approximation of the concept function. The objective is to
minimize the generalization error for a chosen ℎ(𝑥), so that it is as near as possible to the
function 𝑐 (𝑥) that generated the data. We want to define a threshold error value 𝜖 that a hypothesis in the class 𝐻 can have and define the probability of having a ℎ(𝑥) greater than the
threshold value in the hypothesis class 𝐻. For a finite hypothesis class, this can be defined by
the sample complexity bound:
1
1
𝜖
𝛿
𝑚 ≥ (𝑙𝑛|𝐻| + 𝑙𝑛( ),
where m indicates the number of training data points needed to have a generalization error to
be at most 𝜖 with probability at most 𝛿 . This bound is valid for finite hypothesis classes, but
it is not valid for infinite hypothesis classes since 𝑙𝑛|𝐻| cannot be calculated. This is where
the Vapnik-Chervonenkis dimension is used.
VC dimension is defined as the size of the largest finite subset 𝑋 that can be shattered by a
hypothesis space H. It indicates the number of datapoints for a given dichotomy of 𝑋 that a
given hypothesis ℎ(𝑥) ∈ 𝐻 can distinguish. A dichotomy of a set is defined as a partition of
X into two disjoint subsets (for example, labeling the dataset into two distinct classes). 𝑋
then, is shattered by the hypothesis space 𝐻 if and only if for every dichotomy of 𝑋 there exists some hypothesis ℎ(𝑥) ∈ 𝐻 that is consistent with the dichotomy (for example a hypothesis that can separate all the positive labelled data points from the negative ones).
If for a given number n of datapoints there is a dichotomy that cannot be shattered by a hypothesis, then the VC dimension is defined as n -1. The VC dimension of any d-dimensional
hyperplane is defined as d+1, thus in a 2D hyperplane, the VC dimension is at most 3 for any
hypothesis class in the same dimension.
To define then a bound for the generalization error, the sample complexity in the case of infinite hypothesis classes can be defined as:
𝑚 ≥
1
2
13
(4 log 2 ( ) + 8𝑉𝐶(𝐻) log 2 ( )
𝜖
𝛿
𝜖
Which answers the question: How many training datapoints suffice to guarantee a generalization error of at most ϵ with probability at least (1 − δ)?