Training, Validation, Testing
Workshop Fundamentals of Data Science
Prof. Johannes Binswanger
University of St. Gallen
Fall 2025
1 / 21
Bias-Variance Trade-Off
2 / 21
Limits to empirical risk minimization
■
■
■
■
Previously, we have introduced Empirical Risk Minimization as a
rule to fit or pick our model.
We have already hinted towards the fact that this might not always
be the best approach.
For instance, we saw that a larger model, which nests a smaller one,
will always perform at least as well on our dataset.
Let us now push this idea even further to see how the ERM paradigm
might fail. . .
3 / 21
A special predictor
■
■
■
Consider a dataset {(x (i) , y (i) )}ni=1 .
Consider now the following predictor:
( (i)
y , if x = x (i) ,
f (x) =
0,
otherwise.
What does such a predictor do?
4 / 21
What does the special predictor do?
5 / 21
Overfitting
■
This phenomenon is called overfitting.
■
Overfitting occurs when a predictor fits the data it has seen too well.
In turn, such a predictor will not generalize well to new, unseen data.
ˆ) instead of the expected
■ This comes from minimizing the empirical risk L̂(f
ˆ
risk L(f ) which is unobservable (see below for the definitions of the two
functions).
■
Recall that the two functions are defined as follows:
L(f ) := E ℓ(y , f (x))
n
L̂(f ) :=
1X
ℓ(y (i) , f (x(i) )).
n
i=1
■
What can we do in such a situation?
Relate the two loss measures to the present, past, and future. Do they have
a different relationship to each of them, respectively?
6 / 21
Underfitting
■
■
Conversely, when our model does not fit the data it has seen well, we
speak of underfitting.
Underfitting occurs when our model is too simplistic for the
underlying relationship.
■
Such a model does not exploit the information we have in our data.
■
We want to find the sweet spot between underfitting and overfitting!
7 / 21
Towards the bias-variance trade-off: Start
■
■
■
Recall that we are interested in minimizing the expected risk
h
i
L(fˆ) = E ℓ y , fˆ(x) .
Because this quantity is unobservable, we focus on its empirical
counterpart
n
1 X (i) ˆ (i) L̂(fˆ) =
ℓ y , f (x ) .
n i=1
Let us study the expected risk for a squared loss a bit further. . .
8 / 21
Bias-variance trade-off: Line of attack
We can decompose the expected mean squared error as
L(fˆ) = E (y − fˆ(x))2
= E y 2 − 2y fˆ(x) + fˆ(x)2
(i)
= E[y 2 ] − 2 E[y fˆ(x)] + E[fˆ(x)2 ],
where (i) follows from the linearity of expectations.
Let us now consider the components individually. . .
9 / 21
Towards the bias-variance trade-off: Step 1
First, recall that by the definition of the variance, for a random variable X ,
we have
2
Var(X ) = E[X 2 ] − E[X ] .
Thus, it follows that
h
i
2
E fˆ(x)2 = Var fˆ(x) + E[fˆ(x)] .
10 / 21
Towards the bias-variance trade-off: Step 2
Second, by the true relationship y = f ∗ (x) + ϵ, we have
E[y 2 ] = E (f ∗ (x) + ϵ)2
= E f ∗ (x)2 + 2 f ∗ (x) ϵ + ϵ2
(i) ∗
= E f (x)2 + 2 E f ∗ (x) ϵ + E[ϵ2 ]
(ii)
= f ∗ (x)2 + 2 f ∗ (x) E[ϵ] + E[ϵ2 ]
(iii)
= f ∗ (x)2 + Var(ϵ) ,
where (i) follows from the linearity of expectation, (ii) from the fact that
f ∗ (x) does not depend on the (random) data, and (iii) from E[ϵ] = 0 (so
E[ϵ2 ] = Var(ϵ)).
11 / 21
Towards the bias-variance trade-off: Step 3
Third, also by the true relationship, we have
E y fˆ(x) = E (f ∗ (x) + ϵ) fˆ(x)
= E f ∗ (x) fˆ(x) + ϵ fˆ(x)
(i) ∗
= E f (x) fˆ(x) + E ϵ fˆ(x)
(ii) ∗
= f (x) E fˆ(x) + E[ϵ] E fˆ(x)
(iii) ∗
= f (x) E fˆ(x) ,
where (i) follows from linearity of expectation, (ii) from f ∗ (x) not
depending on the random data and from the independence of ϵ and fˆ(x),
and (iii) from E[ϵ] = 0.
12 / 21
Bias-variance trade-off: the result
Putting it all together, we obtain
L(fˆ) = E (y − fˆ(x))2
= E[y 2 ] − 2 E y fˆ(x) + E fˆ(x)2
2
= f ∗ (x)2 + Var(ϵ) − 2 f ∗ (x) E fˆ(x) + Var fˆ(x) + E[fˆ(x)]
2
= f ∗ (x)2 − 2 f ∗ (x) E[fˆ(x)] + E[fˆ(x)] + Var fˆ(x) + Var(ϵ)
2
= f ∗ (x) − E[fˆ(x)] + Var fˆ(x) +
Var(ϵ) .
| {z }
|
{z
}
| {z }
Bias2
Variance
Irreducible error
Thus, our expected risk depends on the bias of our predictor, the variance
of our predictor, and an irreducible error, which we cannot reduce.
13 / 21
Bias-variance trade-off graphically
14 / 21
Training, Validation, Testing
15 / 21
Training, validation, testing
■
■
■
Fitting a model on our dataset, and using this same dataset to
evaluate the model’s quality is not a good approach.
It is common to split the data into a train(ing), a validation, and a
test(ing) sample.
There is no clear-cut rule on how large each sample should be; a
50%/25%/25% or 70%/15%/15% split is often used in practice.
16 / 21
Training, validation, and testing samples
■
Training sample
•
■
Used to fit/train our model. We proceed by ERM as before, but we
don’t use all the available data, just the training data.
Validation sample
Used to assess the quality of our model.
It is used to tune the hyperparameters of our model (e.g., the λ
parameter in Lasso and Ridge models, or the number of polynomials
in a linear regression).
• It is also used to select the best model among a set of models.
•
•
■
Testing sample
•
•
A hold-out data sample which we don’t use to fit our model.
Useful to assess the performance of our model on new, unseen data.
It gives a better picture of how well our model generalizes.
17 / 21
Cross-Validation (CV)
■
■
■
■
Resampling procedure used to evaluate machine learning models on a
limited data sample.
Aims to reduce overfitting, providing insights into how the model will
generalize to an independent dataset.
Leverages different subsets of data for training and validation,
ensuring the model trains on multiple splits of our dataset.
Gives a more robust performance assessment and insight into the
model’s variance and bias.
18 / 21
Cross-Validation algorithm
1. Initialize
•
•
Choose K (number of folds).
Partition the dataset into K equal-sized subsets (folds).
2. For each fold k = 1, 2, . . . , K :
•
•
Train the model on all folds except fold k.
Evaluate the model on fold k and store the result (validation score).
3. Compute the average validation score (e.g., mean squared error) and
choose the model that minimizes it.
19 / 21
Cross-Validation graphically
20 / 21
The logic behind K -fold Cross-Validation
■
■
■
■
The dataset is partitioned into K equal-sized subsets (folds). One fold
becomes the validation set while the rest are used for training. This
rotation happens K times.
We fit the model on the K − 1 subsets.
Every data point is used for both training and validation, making it
suitable for limited datasets. Common choices for K include 5 or 10,
balancing computational cost and performance estimation reliability.
A special case where K = n (number of data splits equals the number
of observations). The model trains on all data points except one,
which becomes the test set. This is called Leave-One-Out
Cross-Validation (LOOCV).
21 / 21