Cameron McLurcan – 1094458
MSc Stream 1
Machine Learning – Assessment 4
The paper chosen for task 2 of assessment 4 is D. Margineantu & T. Dietterich, (1997), Pruning
Adaptive Boosting. The problem addressed is that ensemble learning approaches such as
AdaBoosting require large memory storage capabilities – capabilities that may not be feasible for
many situations. The authors therefore propose a technique called ‘pruning’ which only selects a
subset of the hypotheses. This allows a similar level of performance while using less memory.
The principle of pruning is based off of the “accuracy/diversity tradeoff,” where two classifiers of
high accuracy would reliably provide similar data. Outliers on the other hand would very likely be
inaccurate in comparison. The authors devised five pruning algorithms for testing: Early Stopping,
KL-divergence Pruning, Kappa Pruning, Kappa-Error Convex Hull Pruning, and Reduce-Error Pruning
with Backfitting.
Early Stopping is simply halting production of classifiers after M constructions of classifiers from the
AdaBoost algorithm.
KL-divergence Pruning is an algorithm that chooses a subgroup of diverse classifiers to discard with
the Kullback-Leubler Divergence algorithm to prune. Once measured, the selection algorithm used
is:
Kappa Pruning measures differences between classifiers with a variation of the Kappa algorithm,
which measures the level of ‘agreement’ between them. The variation provides a much lower
probability of the classifiers agreeing by chance due to skewed class numbers. Once calculated, the
predetermined classifiers are selected for use with the most ‘agreeable’ pairs of classifiers given
priority.
Kappa-Error Convex Hull Pruning is similar to Kappa pruning, but also takes the error rate of the
classifiers into consideration. A scatter plot is drawn of each pair of classifiers, with x being Kappa
values, and Y being the error rate. The subgroup of classifiers to be used are then taken from the tip
of the cluster with the highest Kappa (which generally also has the lowest error rate).
The final pruning procedure is Reduce-Error Pruning with Backfitting, which requires the splitting of
the training set into a pruning set and a sub-training set. The sub-training set is used to train the
AdaBoost classifier, whereas the pruning set is used to select which of the classifiers to keep and
which to discard. A method called backfitting is used in conjunction with the pruning set in order to
select the classifiers with the lowest pruning set error rates.
It was found that the best pruning methods were Kappa Pruning and Reduce-Error Pruning, where
both retained a high level of performance (above 0.8 in relation to the unpruned control group) until
pruning percentages of 60% and above. Even radical pruning procedures between 60%-80% proved
fairly reliable. It was also interesting to note that with these two methods, pruning the 20% most
different classifiers resulted in a performance rate above that of the unpruned control group, having
eliminated the outliers in the classifier sets.
Asides from the fact that pruning 20% of the classifiers for those methods provided a more reliable
result than AdaBoosting on its own, the fundamental weakness of pruning is the reduction of
accuracy when the subgroup of classifiers to be pruned is large enough to make a significant
difference to memory costs. The paper was written in a time when memory was much scarcer than
it is today, and so the value of memory in comparison with accuracy of results has shifted
accordingly. Pruning is therefore less necessary in most cases than when the paper was released.