On Comparing Classifiers:
Pitfalls to Avoid and
Recommended Approach
Published by
Steven L. Salzberg
Presented by
Prakash Tilwani
MACS 598
April 25 th 2001
Introduction
Classification basics
Definitions
Statistical validity
•
Bonferroni Adjustment
•
•
Statistical Accidents
Repeated Tuning
A Recommended Approach
Conclusion
Agenda
Introduction
Comparative studies – proper methodology?
Public databases – relied too heavily on them?
Comparison results – are they really correct or just statistical accidents?
T-test
F-test
P-value
Null hypothesis
Definitions
T-test
The t-test assesses whether the means of two groups are from each other.
statistically different
Ratio of difference in means to variability of groups.
F-test
It determines whether the variances of two samples are significantly different.
Ratio of variance of two datasets
Basis for “Analysis of Variance” (ANOVA)
p-value
It represents probability of concluding
(incorrectly) that there is a difference in samples when no true difference exists.
Dependent upon the statistical test being performed.
P = 0.05 means that there is 5% chance that you would be wrong if concluding the populations are different.
NULL Hypothesis
Assumption that there is no difference in two or more populations.
Any observed difference in samples is due to chance or sampling error.
Statistical Validity Tests
Statistics offers many tests that are designed to measure the significance of any difference.
Adaptation to classifier comparison should be done carefully.
Bonferroni Adjustment – an example
Comparison of classifier algorithms
154 datasets
NULL hypothesis is true if p-value is <
0.05 (not very stringent)
Differences were reported significant if a t-test produced p-value < 0.05.
Example (cont.)
This is not correct usage of p-value significance test.
There were 154 experiments. Therefore,
154 chances to be significant.
Actual p-value used is 154*0.05 (= 7.7).
Example (cont.)
Let the significance for each level be
Chance for making right conclusion for one experiment is 1
Assuming experiments are independent of one another, chance for getting experiments correct is (1 ) n n
Chances of not making correct conclusion is 1-(1 ) n
Example (cont.)
Substituting =0.05
Chances for making incorrect conclusion is 0.9996
To obtain results significant at 0.05 level with 154 tests
1-(1 ) 154 < 0.05 or < 0.003
Example - conclusion
Rough calculations but provides insight to problem
The use of wrong p-value results in incorrect conclusions
T-test overall is wrong test as training and test sets are not independent
Simple Recommended
Statistical Test
Comparison must consider 4 numbers when a common test set to compare two algorithms (A and B)
A > B
A < B
A = B
~A = ~B
Simple Recommended
Statistical Test (cont.)
If only two algorithms compared
Throw out ties.
Compare A>B Vs A<B
If more than two algorithms compared
Use “Analysis of Variance” (ANOVA)
Bonferroni adjustment for multiple test should be applied
Statistical Accidents
Suppose 100 people are studying the effect of algorithms A and B.
At least 5 will get results statistically significant at p <= 0.05 (assuming independent experiments).
These results are nothing but due to chance.
Repeated Tuning
Algorithms are “tuned” repeatedly on same datasets.
Every “tuning” attempt should be considered as a separate experiment.
For example if 10 tuning experiments were attempted, then p-value should be
0.005 instead of 0.05.
Repeated Tuning (cont.)
Datasets are not independent, therefore even Bonferroni adjustment is not very accurate.
A greater problem occurs while using an algorithm that has been used before: you may not know how it was tuned
(one disadvantage of using public databases).
Repeated Tuning –
Recommended approach
Break dataset into k disjoint subsets of approximately equal size.
K experiments are performed.
After every experiment one subset is removed.
Trained system is tested on held-out subsystem.
Repeated Tuning –
Recommended approach (cont.)
At the end of k-fold experiment, every sample has been used in test set exactly once.
Advantage: test sets are independent.
Disadvantage: training sets are clearly not independent.
A Recommended Approach
Choose other algorithms to include in the comparison. Try including most similar to new algorithm.
Choose datasets.
Divide the data set into k subsets for cross validation.
Typically k=10.
For a small data set, choose larger k, since this leaves more examples in the training set.
A Recommended Approach
(cont.)
Run a cross-validation
For each of the k subsets of the data set
D, create a training set T = D – k
Divide T into T1 (training) and T2
(tuning) subsets
Once tuning is done, rerun training on T
Finally measure accuracy on k
Overall accuracy is averaged across all k partitions.
A Recommended Approach
(cont.)
Finally, compare algorithms
In case of multiple data sets, Bonferroni adjustment should be applied
Conclusion
We don’t mean to discourage empirical comparisons but to provide suggestions to avoid pitfalls.
Statistical tools should be used carefully.
Every details of the experiment should be reported.