10-Fold Classifier Accuracies Classifier Naïve Bayes (multinomial) SVM (radial basis kernel) Language model SVM (linear kernel) Feature Selection Method (number of features) None Chi-2 (5613*) KLD (5000) KLD (10000) 0.93500140 N/A 0.98562993 0.88252699 0.95426271 N/A 0.90905059 0.88021420 0.96144794 N/A * this is the size of the set containing the top 300 features from each class We are looking for the following accuracies CNB: 95.75% WCNB: 95.75% TWCNB: 94.25% 0.90783690 N/A Support Vector Machine Classification We used LIBSVM to build an SVM classifier for the newsgroup data. Since this is a multiclass problem we used the one-against-all approach described in . To eliminate noise in the dataset we preprocessed the messages using the Chi-squared feature selection method that is described in the 2nd deliverable. To train the SVM, we first converted each message to a normalized tf-idf vector. Then, for each newsgroup i , we constructed a training file train.i and a testing file test.i from the tf-idf vectors. The train.i file contains 9/10 of the messages from newsgroup i and test.i contains the remaining 1/10. All messages from newsgroup i are labeled with +1 (as positive training examples). A matching number of negative examples (i.e. messages from newsgroups other than i ) were included in train.i and test.i with an equal number of examples selected from each newsgroup other than i . We experimented with three kernel types: linear, polynomial and the radial basis (i.e. "Gaussian" kernel). We found that the polynomial kernel (with default parameters) performed poorly. However, this was probably due to our inability to systematically explore the polynomial kernel parameter space (i.e. there is no analogous grid.py tool for the polynomial kernel). The parameter-less linear kernel performed quite well during testing but was outperformed by a radial basis kernel with tuned parameters. To determine the radial basis parameters we used the grid.py tool provided with LIBSVM. Treating the radial basis parameters as variables and 5-fold cross validation as the objective function, it tries to solve the optimization problem of maximizing classifier accuracy. Applying this process to training data from the 20 newsgroups data set revealed that gamma=0.5 and C=8 are the best radial basis parameters for this particular classification problem. With knowledge of which kernel parameters to use, we implemented an SVM classifier for the 20 newsgroups data set. During training, it converts each message to a normalized tf-idf vector and then builds 20 one-against-all models in memory. It is possible to save the training, testing and model data to the hard drive by changing the SAVE_FOR_OFFLINE_USE member variable to true. See the "10-fold classifier accuracies" table for a summary of the SVM classifier's performance on the 20 newsgroups data set. Feature Selection with KL Divergence The results shown in the "10-fold classifier accuracies" table are telling. Despite the author's claim to the contrary, the KL divergence-based feature selection method (KLD) described in  does not perform as well as the approach based on mutual information (Chi-2). One reason for this is that the Chi-2 guarantees that the set of retained features K contains N top features from every class (i.e. each class will be equally represented in the set K). This can be done because the feature score (i.e. mutual information) is a function of feature and class, allowing the best features within a class to be identified. On the other hand, the score in KLD a function only of feature. This means that it not possible to select the top K features within a particular class. Therefore one risk in using KLD is that a particular class may be overrepresented/underrepresented in the set of top features. It could be the case that one of the classes consists of a larger proportion of high-scoring KLD words, in which case the set of top features will contain a disproportionate number of words from this class. Should this happen then the weaker signals present in the other classes will be inadvertently excluded from the training data. References   C.-W. Hsu and C.-J. Lin. A comparison of methods for multi-class support vector machines. IEEE Transactions on Neural Networks, 13(2):415–425, 2002. K. Schneider. 2004. A new feature selection score for multinomial naive Bayes text classification based on KL-divergence. In The Companion Volume to the Proceedings of 42nd Annual Meeting of the Association for Computational Linguistics, Barcelona, Spain, July.