See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/221534216 Multiclass classiﬁers vs multiple binary classiﬁers using ﬁlters for feature selection Conference Paper · July 2010 DOI: 10.1109/IJCNN.2010.5596567 · Source: DBLP CITATIONS READS 13 2,481 4 authors, including: Noelia Sánchez-Maroño Amparo Alonso-Betanzos University of A Coruña University of A Coruña 86 PUBLICATIONS 2,796 CITATIONS 263 PUBLICATIONS 5,450 CITATIONS SEE PROFILE SEE PROFILE Some of the authors of this publication are also working on these related projects: SMARTEES: Social Innovation Modelling Approaches to Realizing Transition to Energy Efficiency and Sustainability View project LOCAW (Low-Carbon at Work: Modelling agents and organisations to achieve transition to a low-carbon Europe View project All content following this page was uploaded by Verónica Bolón-Canedo on 06 July 2016. The user has requested enhancement of the downloaded file. Program at a glance Organizing Committee Welcome Message Plenary Sessions Announcements Detailed Program Detailed Program Detailed Program Invited Sessions Invited Sessions Invited Sessions Author Index Paper Index Program Committee Author Index Paper Index Program Committee © 2010 IEEE Author Index Paper Index Program Committee Hybrid Program Preface Sebastian Stober, Christian Hentschel and Andreas Nuernberger . . . . . . . 2780 Multi-Label Classification by ART-based Neural Networks and Hierarchy Extraction (N-0133) Fernando Benites, Florian Brucker and Elena Sapozhnikova . . . . . . . . . . . . . 2788 Multi-model Ensemble Forecasting in High Dimensional Chaotic System (N0794) Michael Siek and Dimitri Solomatine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2797 Multi-Resolution State-Space Discretization for Q-Learning with Pseudo-Randomized Discretization (N-0307) Amanda Lampton, John Valasek and Mrinal Kumar . . . . . . . . . . . . . . . . . . . . . 2805 Multi-scale Support Vector Regression (N-0815) Stefano Ferrari, Francesco Bellocchio, Vincenzo Piuri and N. Alberto Borghese 2813 Multi-task Learning for One-class Classification (N-0240) Haiqin Yang, Irwin King and Michael R. Lyu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2820 Multiagent Reinforcement Learning in the Iterated Prisoner’s Dilemma: Fast Cooperation through Evolved Payoffs (N-0644) Vassilis Vassiliades and Chris Christodoulou . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2828 Multiclass classifiers vs multiple binary classifiers using filters for feature selection (N-0355) Noelia Sanchez-Marono, Amparo Alonso-Betanzos, Pablo Garcia-Gonzalez and Veronica Bolon-Canedo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2836 Multiobjective Multiclass Support Vector Machine Based on the One-against-all Method (N-0597) Keiji Tatsumi, Masato Tai and Tetsuzo Tanino . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2844 Multiple Kernel Learning with ICA: Local Discriminative Image Descriptors for Recognition (N-0358) Siyao Fu, Shengyang Guo and Guanghou Zeng . . . . . . . . . . . . . . . . . . . . . . . . . 2851 MuSeRA: Multiple Selectively Recursive Approach towards Imbalanced Stream Data Mining (N-0581) Sheng Chen, Haibo He, Kang Li and Sachi Desai . . . . . . . . . . . . . . . . . . . . . . . 2857 N2Cloud: Cloud Based Neural Network Simulation Application (N-0552) Altaf Ahmad Huqqani, Xin Li, Peter Beran and Erich Schikuta. . . . . . . . . . . .2865 Naive Bayes texture classification applied to whisker data from a moving robot (N-0408) Nathan Lepora, Mat Evans, Charles Fox, Mathew Diamond, Kevin Gurney and Tony Prescott . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2870 32 WCCI 2010 IEEE World Congress on Computational Intelligence July, 18-23, 2010 - CCIB, Barcelona, Spain IJCNN Multiclass classifiers vs multiple binary classifiers using filters for feature selection N. Sánchez-Maroño, A. Alonso-Betanzos, Member, IEEE, P. Garcı́a-González and V. Bolón-Canedo Abstract— There are two classical approaches for dealing with multiple class data sets: a classifier that can deal directly with them, or alternatively, dividing the problem into multiple binary sub-problems. While studies on feature selection using the first approach are relatively frequent in scientific literature, very few studies employ the latter one. Out of the four classical methods that can be employed for generating binary problems from a multiple class data set (random, exhaustive, one-vsone and one-vs-rest), the two last were employed in this work. Besides, four different methods were used for joining the results of these binary classifiers (sum, sum with threshold, Hamming distance and loss-based function). In this paper, both approaches (multiclass and multiple binary classifiers), are carried out using a combination method composed by a discretizer (two different were employed), a filter for feature selection (two methods were chosen), and a classifier (two classifiers were tested). The different combinations of the previous methods, with and without feature selection, were tested over 21 different multiple data sets. An exhaustive study of the results and a comparison between the described methods and some others on the literature is carried out. I. I NTRODUCTION Classification problems with multiple classes are common in real life applications. However, while binary classification problems have been studied intensively, only very few works have been carried out for studying multiclass classification [1], [2], [3], [4]. There are two basic approaches to deal with classifying multiple classes: one is to use classification algorithms that can deal directly with multiple classes; and the alternative is to divide the original problem into several binary classification problems. Some authors conclude that there is not a multiclass method that outperforms every other one, and that the method to be used in order to obtain the best results will depend on the problem, and also on other userdefined constraints, such as the desired level of accuracy, the time available for obtaining a solution, etc.[5] Besides the multiclass characteristic, nowadays the majority of the applications deal with data sets with high dimensionality. Problems with multiple classes and high dimensionality have been even less studied. Feature selection is one of the methods that can be used to reduce dimensionality. These methods aim to reduce the number of input attributes of a given problem, eliminating those that are unnecessary or redundant, and obtaining a reduction in the computational Department of Computer Science, University of A Coruña, Spain (email: nsanchez@udc.es, ciamparo@udc.es, pablo.garcia.gonzalez86@gmail.com, vbolon@udc.es). This work was supported by Spanish Ministerio de Ciencia e Innovación (under projects TIN 2006-02402 and TIN 2009-10748) and Xunta de Galicia (under project 2007/000134-0), all of them partially supported by the European Union ERDF. 978-1-4244-8126-2/10/$26.00 c 2010 IEEE resources needed, and most of the times, improving the performance of the classification algorithms employed. Among the different approaches than can be employed, filters are the common option when the number of input features is very high, presenting also two other interesting advantages: they are independent of the evaluation function used and they consume less computational resources than the alternative wrapper methods [6], [7]. In this paper, a combination method that uses a discretizer, a filter and a classification algorithm has been tested over 21 different multiclass problems. Four different discretizers, two filters and two classifiers have been selected to carry out an exhaustive comparative study. In order to check the adequacy of the feature selection step, the results of the classification method without previous filtering have been obtained for the first approach (multiclass problem) and the second approach (multiple binary classifiers). Regarding the later, there are several strategies that can be used for dividing the multiclass problem in several binary problems, such as the one-versusrest, one-versus-one, exhaustive, etc. In this work, the results of the two first methods are included in the comparative study. Also, there are different algorithms that can be used to integrate the information of the various binary classifiers that select the same class. In this work, sum, sum with threshold, Hamming decodification, and Loss-based decodification have been used [8]. At the end, 12 different combinations were tested over the 21 different data sets, and the results obtained were compared, so as to be able to reach conclusions about their performance. II. M ULTIPLE CLASSES APPROACHES There are two main approaches for classification problems involving more than two classes. One transforms the multiclass problem into several binary problems, while the other deals directly with the multiclass problem. This last strategy has as disadvantage: a possible overtraining of the algorithm, favoring those classes that are most represented in the sample, or are easier to separate [9]. Besides, another possible problem is that as the number of classes increases, the more difficult can be for a feature set to provide an adequate separation among them [7]. However, the first strategy has also its drawbacks, such as how to integrate the information that comes from each of the binary classifiers, or the fact that there exists enough representation of a specific class in the training set generated from the original one to train the binary classifier. There are different schemes to transform the multiple classes problem into several binary problems, but all of them 2836 can be generalized by the Error-Correcting Output-Codes (ECOC) method [10]. In this method, instead of providing each classifier with a set of inputs and its corresponding output, the output is transformed by using a matrix Ml×c which columns are the number of classes (c) and its rows the number of classifiers (l). There are four different schemes: • Random: it forms l different classifiers. Each classifier randomly selects a subset of classes as positive and another subset, mutually exclusive, as negative class. • One-versus-rest: it transforms a problem with c classes in c binary problems, i.e., it uses one classifier for each class (l = c). The elected class represents the positive samples and the remaining classes the negative ones. • One-versus-one: it generates a classifier for each pair of classes, i.e., each classifier compares two different classifiers. classes. Therefore, there are l = c(c−1) 2 • Exhaustive: it generates one classifier for each possible combination between classes. Any meaningless combination is not taken into account, for example, considering all classes as positive. Notice that this approach includes the schemes one-versus-rest and oneversus-one. Transforming the multiple class problem into several binary problems is a coding technique that requires a counterpart, i.e., a decoding technique. This decoding technique should integrate the information provided by all the classifiers into an unique solution, that is, finally the sample is classified in only one class. The most common decoding techniques are Hamming distance and loss-based functions [8], subsequent paragraphs briefly explain both. An illustrative example that shows how different decoding techniques can lead to different results can be consulted in [8]. Consider a problem with c classes that has been codified using a matrix Ml×c with l classifiers. Each class i (i = 1, . . . , c) has one of three possible values for each classifier j (j = 1, . . . , l): positive (+), negative (-) or ignored (*). Then, the pattern of class i is a vector of length l where each position j denotes the value of class i for classifier j. For example, considering the one-vs-rest scheme, the pattern for class 1 would be {+, −, . . . , −, −}, being the number of negative classes equal to l − 1 (there are no ignored classes in this scheme). Once the classifiers are trained and a new pattern is provided to them, each classifier returns a solution forming an ”objective” pattern. Then, this ”objective” pattern is compared to each class pattern. The class which ”distance” is minor compared to the ”objective” pattern would be elected as output class. There are different ways to measure this ”distance”: • The Hamming distance between two vectors of equal length is the number of positions at which the corresponding symbols are different. Put another way, it measures the minimum number of substitutions required to change one vector into the other, or the number of errors that transformed one vector into the other. In this case, the differences are weighted to assign a higher ”distance” value to those positions than differ in two • values {+, −} than those that only differ in one ({+, ∗} or {−, ∗}). The previous distance measure does not consider the values returned by classifiers causing a loss of information. The loss-based function takes under consideration this information and it is adapted for each classifier. In this work, due to the classifiers employed, the logistic regression function was chosen, defined as: L(z) = log(1 + e−2z ), where z is equal to the product of each position of the ”objective” pattern and the same position at the class pattern. The value L(z) is computed for each position of a class pattern, all these values are added and so, the ”distance” for the class is obtained. Notice that the ”objective” patterns are like {0.90, 0.30, . . . , −0.40}, i.e., they reflect the value that the classifiers provided. Another problem of the multiple binary classifiers approach is the need for enough number of samples in the training set. This aspect is critical in those cases in which there is a great difference between the number of samples of one class and the number of samples of others. In those cases, the capacity of generalization of the learning algorithm can be seriously affected, as it could end up ignoring the samples of the minority class, better than trying to learn it. In order to mitigate this problem, there exist different alternatives that consist on downsizing the majority class (undersampling), upsizing the minority class (oversampling) or altering the relative costs of misclassifying the small and the large classes [11]. In this work the last strategy was adopted, i.e. oversampling, because the unbalanced data sets employed have not many samples available. Thus, some samples of the minority class were randomly replicated so as to obtain the same number of samples of the majority class. III. T HE METHODOLOGY In this paper a combination method for multiclass classification problems has been tested. The method is divided in three steps: • First, a discretizer method is applied over the input data, with the aim of solving problems of unbalanced values, and prepare the attributes of the sample to be processed by the feature selection algorithm of the next step. Several discretizers have been chosen to test their influence on the classification problem, specifically EWD (Equal Width Discretization), EFD (Equal Frequency Discretization), PKID (Proportional K-Interval Discretization) [12] and EM (Entropy Minimization) [13]. • After discretization, feature selection is carried out using filters. Two different methods based in different metrics were tested, Consistency-based filter (CBF) [14] and Correlation-based feature selection (CFS) [15]. • Finally, a classifier is applied. In our case, C 4.5 [16] and naı̈ve Bayes [17] were selected, because both can be used for the direct multiclass approach. Besides, their 2837 Fig. 1. An scheme of the several approaches tested in this work use of computer resources is affordable, an important factor in our study due to the high dimensionality of some of the data sets employed. The combination method above has been used over the two different general approaches than can be used for multiclass classification problems: • A multiclass approach. This is feasible for both the classification methods selected, C4.5 and naı̈ve Bayes. • A multiple binary classification approach. In this work the approaches one-versus-rest and one-versus-one have been chosen. The random scheme is not expected to return good performance results, whereas the exhaustive scheme is very computer-resource demanding due to its requirement of training a high number of classifiers. So, one-versus-rest and one-versus-one are the coding techniques adopted. The rest of the section regards to the decoding techniques used. Initially, it is necessary to explain that the results returned by the classifiers are probability values. For example, a possible output value of a classifier could be 0.9, which means that a pattern should be assigned to the positive class with a high probability. An opposite value could be 0.4 which means that this pattern should be consider negative with reservations. The decoding technique used for the one-versus-rest scheme consists on assigning as result class the class with the highest probability value. Notice that those classifiers that return as ”winning” class the negative class (”rest” class) are ignored. For the oneversus-one scheme, Hamming and loss-based (see section II) were used as decoding techniques. A threshold was used for computing both distances. The threshold is used to denote if a probability value should be considered positive, negative or ignored for the Hamming distance or for weighting this probability value for the Loss-based function. Apart from these distances, we developed and applied two more ”adhoc” distance measurements: • Sum is an union method based in the probability assigned by the binary classifier to the “winning” class for each sample. Therefore, instead of calculating distances to determine the class from the l different results, the • accumulative probability sum of each class is computed. Then, the desired output is the one with the highest value. Notice that this measure is similar to the one used for the one-versus-rest scheme. Sum with threshold is a method that modifies the previous one and takes under consideration the fact that test patterns include “ignored” classes, i.e., classes not used for the learning of the classifier. Then, this technique only computes those probabilities that are over an established threshold to guarantee that only clearly winning classes are computed. As we have seen, there are two multiple class classification approaches. In the first one, called ”multiclass”, a data set with c classes is discretized, filtered and classified, and no union method is required since the prediction results are directly obtained. On the other hand, when the multiple binary classification approach is chosen, the problem turns into several binary problems, depending on the scheme adopted, one-vs-one or one-vs-rest. Each classifier requires the previous steps of discretization and filtering and, besides, after obtaining the outputs of the l binary classifiers, a union method is required in order to join data and achieve an unique prediction. At this point, as explained above, four union methods are available for the one-vs-one scheme: Sum, Sum threshold, Hamming and Loss-based and they will provide us the final result ( see figure 1). Besides, for all those approaches, in order to compare the performance with and without filtering and see the benefits related with feature selection, we can eliminate the filter step (dashed in the center of Figure 1). Therefore, there are 16 different combinations (4 discretizers × 2 filters × 2 classifiers) when using feature selection plus 8 more without using it (4 discretizers × 2 classifiers). Each combination is applied for each approach, i.e., multiclass, one-vs-rest and one-vs-one with 4 different decoding techniques. The number of results achieved is enormous even for only one data set and cannot be included in a paper, so next section will try to summarize them. 2838 IV. T HE EXPERIMENTAL RESULTS The direct multiple approach and the two multiple binary approaches, with and without feature selection, and with the different methods of discretizers, filters and classifiers were tested using Weka[18] and MatLab[19]. The 21 different data sets shown in Table I were selected [20], attempting to include several aspects such as different number of classes, different number of samples, different ratios features/samples, unbalanced data sets, etc. Specifically, there are 4 data sets with clearly unbalanced classes: Glass, Connect-4, Dermatology and Thyroid. The oversampling technique was applied to the glass data set which has 3 of 6 classes with very reduced number of samples (lower than 20), making extremely difficult the adequate learning of the classifiers. A 10-fold crossvalidation was used to obtain results in percentage of correct classification and in features selected by the methods that use filters. For each data set, in order to check if the several methods used exhibit statistically significant differences in performance, a multiple comparison statistical test was employed using ANOVA (ANalysis Of VAriance) [21], if the normality hypothesis is assumed, or using an alternative non-parametric procedure, the KruskalWallis test [22]. The prefix MFeat employed in some data sets of Table I means Multi-feature, whereas the prefix MLL in Leukemia data set denotes the type of leukemia being tackled. Both prefixes will be ignored in the rest of this paper. TABLE I T HE DATA SETS EMPLOYED IN THE EXPERIMENTAL STUDY Data set Iris Vehicle Wine Waveform Segment Glass Connect 4 Dermatology Vowel KDD SC Splice Thyroid Optdigits Pendigits Landsat MFeat-Fourier MFeat-Factor MFeat-Karhounen MFeat-Pixel MFeat-Zernike MLL-Leukemia Classes 3 4 3 3 7 6 3 6 11 6 3 3 10 10 6 10 10 10 10 10 3 Samples 150 846 178 5000 2310 214 67557 366 990 600 3190 3772 3823 7494 4435 2000 2000 2000 2000 2000 57 Features 4 18 13 21 19 10 42 34 13 60 61 21 64 16 36 76 216 64 240 47 12582 For each data set, 16 different combinations with feature selection and 8 without filtering were used for each approach considered. Then, there are 24 performance results for both multiclass and one-vs-rest approach and, moreover, 96 (24 × 4 union techniques) one-vs-one results. Trying to show all results becomes untractable, so the best results for each scheme with and without feature selection are shown. It turns to 12 different results for each data set. As an example, in Figure 2 the results obtained for the Thyroid data set are shown. As it can be seen, the best performance with the smallest set of features is obtained by the one-vs-one approach using the sum method to make the union of the multiple binary classifiers and the combination EM+Consistency-Based Filter+C4.5. The precision obtained was 99.52 ± 0.34 using 5 features (the number of attributes used by this approach is 23.81% of the total). Fig. 2. Best results obtained for the Thyroid data set. Percentage of features selected is represented at the left y axis, while on the right side, the accuracy obtained by the approach is drawn. On the x axis, the best results for each approach without and with feature selection are displayed. In any case, Figure 2 shows that all approaches in this data set obtained good results when using feature selection. Both multiclass and one-vs-one approaches (using sum and sum with threshold as union techniques) achieved better accuracy values using feature selection than not using it. On the other hand, two of the one-vs-one versions (using Hamming and loss-based) and the one-vs-rest approach obtained lower values for the accuracy by using feature selection. This latter case is the one with the lowest accuracy value when feature selection is applied (99.23 ± 0.36), a slightly lower value than using the same approach without feature selection (99.44 ± 0.34). However, the difference is not statistically significant and the reduction in the number of features used is important (21 vs 8). Figure 3 shows the result of applying a multiple comparison function (0.05 significance level) between the number of features selected by the approaches multiclass, one-vs-rest and one-vs-one Sum (for the sake of completeness also the set containing all features was included) for the Thyroid data set. The graph displays each group mean represented by a symbol and an interval around the symbol. Two means are significantly different if their intervals are disjoint, and are not significantly different if their intervals overlap. Therefore, it can be seen that there is a significant statistical difference among the approaches with and without feature selection. Notice that these differences are not so notorious as bars in figure 2 may suggest because X-axis denotes average group ranks. Ranks are found by ordering the data from smallest to largest across all groups, and taking the numeric index of this ordering. The rank for a tied observation is equal to the average rank of all observations tied with it. So, in figure 3, 40 different values were ranked (10 values, one per fold, for each one of the 4 approaches considered). Clearly, there 2839 TABLE II B EST RESULTS FOR EACH DATA SET. P ERFORMANCE WITH AND WITHOUT F EATURE S ELECTION (FS AND W ITHOUT FS) APPROACHES . F OR EACH ONE , MEAN AND STANDARD DEVIATION ACCURACY OBTAINED FROM THE 10- FOLD CV (ACC ) AND RANKING OCCUPIED BY THE APPROACH (R K ). W HEN USING FS, PERCENTAGE OF CHARACTERISTICS SELECTED (% F EAT.). O F THE one-vs-one APPROACHES , ONLY THE RESULTS USING THE SUM ARE DISPLAYED , BECAUSE THEY TEND TO SHOW THE BEST PERFORMANCES . AT THE END OF THE TABLE AVERAGE ACCURACY AND AVERAGE RANKING IS SHOWED FOR THE SIX APPROACHES . Multiclass Iris Vehicle Wine Waveform Segment Glass Connect 4 Dermat. Vowel KDDSC Splice Thyroid Optdigits Pendigits Landsat Karhounen Factor Fourier Pixel Zernike Leukemia Accuracy Ranking Without FS Acc Rk 95.99±3.44 6 69.73±6.02 8 98.88±2.34 2 80.98±1.72 10 91.38±1.99 12 73.83±8.56 9 80.94±0.74 2 98.35±2.31 4 74.04±5.15 2 96.99±2.19 9 95.74±0.94 9 99.28±0.60 10 92.67±1.05 8 89.61±0.93 11 85.27±1.40 8 92.50±1.90 4 93.30±1.70 6 77.40±2.75 9 93.50±1.24 4 72.05±1.97 11 91.33±12.09 12 87.80 7.43 Acc 97.99±3.22 71.62±4.33 98.33±2.68 80.86±1.82 92.42±2.07 70.15±9.06 81.16±0.51 98.91±1.89 74.64±3.67 96.16±2.83 96.08±1.57 99.44±0.56 92.62±0.69 89.32±0.93 84.75±1.45 93.10±1.64 95.90±1.32 78.60±1.76 93.30±1.18 72.35±3.68 96.67±7.03 88.30 1VsRest FS Rk 1 2 3 11 9 11 1 1 1 10 6 3 10 12 12 1 1 3 5 9 1 5.48 % Feat 50.00 38.89 84.62 71.42 63.15 77.77 83.33 67.64 84.61 91.66 55.74 28.57 57.81 62.50 88.88 92.18 49.53 69.73 57.50 68.08 4.79 Without FS Acc Rk 96.00±6.44 3 72.11±4.83 1 98.3±2.74 6 79.78±1.78 12 92.12±1.96 11 72.01±8.20 10 80.61±0.50 3 97.79±2.87 11 54.04±5.37 4 95.00±1.76 11 96.14±1.00 3 99.23±0.34 11 92.62±1.91 10 93.50±1.21 8 84.82±2.14 11 92.50±1.56 4 88.70±2.2 8 77.10±2.54 10 91.70±1.33 11 71.65±2.40 12 96.67±7.02 1 86.78 7.67 Acc 96.65±3.51 67.60±6.12 97.71±4.83 82.36±1.76 92.42±2.25 68.83±8.51 80.61±0.51 97.00±2.36 64.44±4.82 82.33±19.11 95.89±1.63 99.44±0.36 91.68±1.55 93.40±1.10 84.98±1.57 90.95±1.78 92.90±2.52 76.25±2.40 91.15±1.39 72.15±3.44 96.67±7.03 86.45 1vs1 Sum FS Rk 2 12 10 6 9 12 3 12 3 12 8 3 12 9 9 8 7 12 12 10 1 8.19 % Feat 75.00 100 100 95.23 94.73 100 97.62 76.47 84.61 96.66 55.73 38.09 85.93 100 100 100 84.72 98.68 90.00 95.74 0.86 Without FS Acc Rk 94.00±7.34 9 71.27±4.70 3 97.77±3.88 7 81.02±1.72 8 94.31±1.61 5 85.81±4.58 3 80.38±0.56 5 98.07±2.28 8 -±0 97.00±2.19 6 96.21±0.82 1 99.52±0.44 1 93.83±1.94 5 95.78±0.65 1 86.89±1.21 3 89.75±2.58 10 88.00±2.51 10 78.40±2.36 4 92.95±1.42 7 75.20±2.62 1 93.33±8.61 5 89.47 5.10 Acc 96.00±4.66 71.05±6.23 97.19±2.96 82.64±1.50 94.37±1.54 85.34±8.51 80.26±0.55 98.34±1.93 -±98.33±1.11 96.14±1.12 99.44±0.34 94.19±1.04 94.15±1.40 87.10±1.36 92.50±2.27 95.65±1.85 78.90±3.71 94.15±1.05 74.90±4.21 93.33±11.65 90.20 FS Rk 3 4 11 4 4 4 8 7 0 1 3 3 3 6 1 4 2 1 2 2 5 % Feat 50.00 100 53.85 90.47 84.21 100 97.62 94.11 95.00 63.93 23.81 84.37 100 100 100 99.07 88.15 97.91 100 0.70 3.90 example are the results of the Leukemia data set, which has a much higher number of features (12582) than samples (57). Besides, it is representative of a type of data sets that are receiving considerable attention in research, the microarray data sets. In these type of data sets, feature selection can have a high impact, because eliminating unnecessary features can help the biologists for a better explanation of the behavior of genes involved in cancer research. The results obtained for the Leukemia data set can be seen in Figure 4. Again, better results are obtained by the feature selection versions of the methods, and the best is that of the one-vs-rest approach. Although accuracy is the same for multiclass with feature selection, and one-vs-rest with and without feature selection, the drastic reduction in the number of features needed (108 out of 12582) makes it worthwile to use EWD+CFS+NB. Fig. 3. Multiple comparison results using all features (ALL) and the features selected by the multiple classes (MC), the one-vs-rest(1R) and onevs-one (11) approaches for the Thyroid data set are 10 tied values at the last positions of this rank, one per fold of the approach with all features. On the other hand, the features subset returned by the one-vs-one approach is most of the times at the top. The important reduction in the number of necessary attributes achieved by the feature selection methods make their use worthwhile, specially in those cases in which the elimination of features can contribute to better explanation of clinical situations, such as it is the case in some data sets used in this work (i.e., Leukemia, Thyroid,...). So, another interesting Fig. 4. Best results obtained for the Leukemia data set. Percentage of features selected is represented at the left y axis, while on the right side, the accuracy obtained by the approach is drawn. On the x axis, the best results for each approach without and with feature selection are displayed. 2840 TABLE III R ESULTS IN ACCURACY OBTAINED FOR EACH OF THE APPROACHES A. Analysis of multiclass versus multiple binary approaches In order to be able to establish a general comparative picture between all methods and both alternatives, in Table II the results obtained for all 21 data sets by the Multiclass, one-vs-rest and one-vs-one approaches with (FS) and without (without FS) feature selection are shown. For the latter approach, only the results achieved by the sum as union method are shown, because it is the one obtaining best results in average. A column labeled Rk has been added with the idea of detecting which approximation is the best in average. To be able to do this, the 12 different approaches are listed in order of percentage obtained for each data set. Subsequently, the average ranking position is computed for each of the 12 approaches, so as to compare the different methods. This value is shown in the last row of the table. Also, the average in accuracy is displayed for each method in the previous row. It is necessary to note that the ranks are computed over all the 12 approaches, although only the best 6 of them are shown in the table. From results in Table II it can be concluded that, although the multiclass approach appears to behave better in number of features selected and with similar accuracy than the alternative multiple binary classifiers, the approach with the best average results, in both ranking and accuracy, is onevs-one using sum and feature selection. The ranking value of the latter, 3.9 is clearly separated from the rest, that obtain values higher than 5. The same behavior can be observed for the accuracy, in which the best value is obtained again by one-vs-one using sum and feature selection, although in this case the difference with the other methods is smaller. Notice that the scheme one-vs-one was not carried out for the Vowel data set, because its 10 different classes will imply the training of 45 classifiers in this approximation. A deeper analysis of Table I suggests that the one-vs-one approach clearly surpasses the others when there exists a high ratio number of samples/number of features and enough samples per class. In general, the one-vs-rest approximation exhibits a poor behavior, even when using feature selection, but surprisingly it achieves the best results when leading with the Leukemia data set that has a very low ratio number of samples/number of features. Besides, the results obtained by this approach worsen as the number of classes increases. This is due to the consequent unbalanced division into positive and negative classes for each of its classifiers. The good results achieved by the multiclass approach were unexpected and a further analysis was done in order to get more insight. It is important to remember that different combinations of discretizer, filter and classifier were done for each approach and the best combination was selected for each one. An example showing all the combinations for the Factor data set is depicted in Table III. The first block of rows of this table shows the results achieved when using the C4.5 classifier, while the last block is devoted to the naı̈ve Bayes classifier. The best accuracy obtained for each combination is emphasized in bold font. The last row indicates the average accuracy achieved by each approach. It can be seen that the best accuracy (marked with grey Factor DATA SET. AVERAGE ACCURACY OBTAINED FOR M ULTICLASS AND BOTH MULTIPLE BINARY CLASSES APPROACHES ARE SHOWN IN LAST ROW OF THE TABLE . TESTED FOR THE Combin. EWD+CFS EFD+CFS PKID+CFS EM+CFS EWD+CBF EFD+CBF PKID+CBF EM+CBF Combin. EWD+CFS EFD+CFS PKID+CFS EM+CFS EWD+CBF EFD+CBF PKID+CBF EM+CBF Average C4.5 Classifier Multiclass 1vsRest 95.00 ± 1.81 92.150 ± 2.30 94.65 ± 1.47 91.50± 1.83 92.50 ± 2.12 90.85 ± 1.93 95.90 ± 1.33 92.90 ± 2.53 84.90 ± 3.00 90.45± 2.73 82.10 ± 1.90 88.65 ± 1.68 69.75 ± 3.84 84.35 ± 1.89 84.90 ± 3.34 92.55 ± 1.74 NB Classifier Multiclass OnevsRest 79.20 ± 1.89 84.60 ± 2.85 74.10 ± 2.53 85.45 ± 1.76 52.75 ± 3.07 71.00 ± 4.60 80.65 ± 3.07 87.75 ± 3.16 78.60 ± 3.70 85.60 ± 2.39 76.05 ± 3.13 85.00 ± 2.20 53.95 ± 2.99 71.50 ± 2.59 80.80 ± 3.15 87.30 ± 2.39 79.74±2.65 86.35±2.41 1vs1 Sum 94.65 ± 2.07 94.65 ± 1.55 94.25 ± 1.21 95.65 ± 1.86 92.35 ± 1.29 90.90 ± 1.88 90.95 ± 2.78 92.15 ± 1.97 1vs1 Sum 89.75 ± 2.53 88.95 ± 2.06 88.05 ± 2.31 88.95 ± 2.60 89.20 ± 1.89 88.75 ± 2.23 89.40 ± 2.13 89.55± 2.24 91.13±2.04 TABLE IV B EST ACCURACY AND AVERAGE ACCURACY OBTAINED FOR M ULTICLASS AND BOTH MULTIPLE BINARY CLASSES APPROACHES FOR THE Dermatology AND Karhounen DATA SET. Data set Dermat. Karhounen Multiclass 98.91 ± 1.90 94.34 ± 4.06 93.10 ± 1.64 71.72 ± 3.16 1vsRest 96.70 ± 3.61 94.66 ± 4.16 89.60 ± 2.20 73.73 ± 2.85 1vs1 Sum 98.08 ± 3.43 96.62 ± 3.52 91.60 ± 1.43 87.12 ± 2.33 background) is obtained by the combination of minimum entropy discretizer + Correlated filter selection + C4.5 classifier using the multiclass approach. However, the last row denotes that the one-vs-one scheme achieves the best result in average, and moreover, this scheme obtains the best accuracy in 12 of 16 combinations, whereas the multiclass approach only gets the best values in 3 of them. Table IV shows similar results for Dermatology and Karhounen data sets, but summarized. For each data set, the first row shows the values for each approach when the best accuracy is achieved (EFD+CFS+NB combination for Dermatoloy data set and EWD+CFS+NB combination for Karhounen data set). The second row indicates the mean average. Again, the multiclass approach gets the best performance result in a combination, however it is clearly surpassed by the one-vs-one scheme when focusing on averages. B. Best discretizer, filter and classifier combination In this work, 16 different combinations of discretizer, filter and classifier were tried over 21 data sets. Moreover, for reasons of completeness, the filtering step was considered optional. Table V attempts to determine which combination gets the best accuracy values. If two combinations obtain 2841 TABLE VI DATA SETS WHICH IMPROVES THE RESULTS ACHIEVED BY METHODS identical accuracy, both are computed in this table, and so all values in table V add up to 24, not to 21. Several conclusions can be extracted from this table attending to different issues: • Feature selection (with or without): the number of combinations using feature selection and reaching the best performance values are 17 from 24, which denotes the adequacy of its use. • Discretizer: Entropy minimization discretizer forms part of the ”best” combination in 13 occasions. On the other hand, PKID is only included in a ”best” combination using C4.5, although it is suited for naı̈ve Bayes classifier, but it is suboptimal when learning from training data of small size [12]. • Filter: CFS seems to be a good filter combined with NB classifier either using EWD or EM discretizers. However, this filter does not achieve the best results when it is applied together with C4.5 classifier; in this case, the consistency based filter is preferred. • Classifier: Naı̈ve Bayes obtains better results in more data sets than C4.5. Specifically, naı̈ve Bayes gets the best values for 12 different data sets, while C4.5 only for 8 of them (both classifiers achieve the same accuracy for the Dermatology data set). EXISTING IN THE BIBLIOGRAPHY. ACC STANDS FOR ACCURACY AND %F EAT THE PERCENTAGE OF FEATURES EMPLOYED Data Wine Glass Connect4 Dermatol. Splice Thyroid ACCURACY AND Data Iris Vehicle Waveform Segment Vowel KDD SC Optdigits Pendigits Landsat Karhounen Factor Fourier Pixel Zernike Leukemia RESPECTIVELY. W 4 M 4 W 0 F 0 CFS K 1 M 0 Naı̈ve Bayes Classifier CBF W F K M 1 0 0 2 C4.5 Classifier CBF W F K M 2 0 0 2 W 1 Without FS F K M 0 0 2 W 1 Without FS F K M 0 0 3 C. Comparative study with other methods In this section we will try to compare our results with those existing in the literature. Notice that it is not a ”fair” study in the sense that the validation methodologies may differ from one study to another. As can be seen in Table VI, we obtained better results than those achieved by other methods for 6 data sets. This is a remarkable fact because we are comparing our method with a group of methods, in some cases specially designed for an specific problem, such as the Parzen method for image data sets [28]. It is important to notice the difference in accuracy for the Glass data set (up to a 17% of improvement). It is also worth to mention the reduced set of features used for the Thyroid data set while there is a slight increment in the accuracy. Analogously, in Connect 4 results, accuracy is improved while the number of features is reduced in 16,7%. Table VII reflects those data sets where our performance results did not surpass the existing ones. It is important to remark than some methods are highly computer-resource demanding, for example the EECOC method, that is, the Best value achieved Acc % Feat. 98.9 100 87.7 100 81.2 83.33 98.9 100 96.2 100 99.5 23.81 TABLE VII DATA SETS WHICH DOES NOT IMPROVE THE RESULTS ACHIEVED BY METHODS EXISTING IN THE BIBLIOGRAPHY. ACC STANDS FOR TABLE V N UMBER OF TIMES A COMBINATION GETS THE BEST RESULTS . W, F, K AND M STANDS FOR EWD, EFD, PKID AND EM DISCRETIZERS , CFS F K 1 0 Best value in bibliography Acc Method 97.8 NB-Back [26] 70.9 C4.5+EECOC’s [23] 79.2 C4.5 [24] 97.5 NB [25] 95.4 NB [25] 99.4 C4.5+ DMIFS [25] %F EAT THE PERCENTAGE OF FEATURES EMPLOYED . LR MEANS LOGISTIC REGRESSION . Best value in bibliography Acc Method 100 RNA [25] 75.8 AFN-FS [26] 86.6 LR+EECOCs [23] 97.5 C4.5+ EECOCs [23] 93.2 C4.5+ECCOCs [23] 98.4 Naı̈ve Bayes [25] 98.1 C4.5+ECCOC’s [23] 99.1 C4.5+ECCOC’ [23] 91.0 MLP+ SCG [27] 96.3 Parzen [28] 96.6 Bayes lineal [28] 82.9 Parzen [28] 96.3 Bayes lineal [28] 82.0 Parzen [28] 98.2 SVM + 3NN [29] Best value achieved Acc % Feat 98.0 50 72.1 100 84.1 90.47 94.8 100 74.6 84.61 98.3 95.00 94.3 84.67 95.8 100 87.1 100 93.1 92.18 95.9 49.53 78.9 88.15 94.3 97.91 75.2 100 96.7 0.86 exhaustive ECOC commented in section II. For some data sets, the performance accuracy obtained by the combination method proposed in this paper is lower than other methods’ accuracy; however the reduction in the number of features is very significant, see for example the Factor or Leukemia data sets in table VII. Notice that this latter data set has a very high ratio number of features/number of samples, making extremely difficult the adequate learning of any classifier. Nevertheless the combination only employs 0.86% of the features and obtains a good level of accuracy in the test data. V. C ONCLUSIONS The main goal of this paper was to study the combination of discretizers and feature selection methods in multiple class problems. Four discretizers, two filters and two classifiers were taken into account, obtaining 16 different combinations that were tested over 21 data sets broadly used in the bibliography. Moreover, two approaches were considered to deal with multiple class problems: the first one consists on applying a suitable classifier for those problems; the second one divides the problem into several binary problems, the one-vs-rest and one-vs-one schemes were used to generate this division. In the latter scheme, the results provided for each classifier were gathered using 4 decoding techniques, 2842 although only the results obtained by the best of them are shown in this paper. Therefore, 16 combinations were applied using three different approaches, one of them using 4 ways to join its results. Besides, for the sake of completeness, the same approaches were also tested without using the feature selection step, i.e., without filtering. The experimental results support the hypothesis that using feature selection leads to better performance results than not using it. Moreover, it eliminates the associate cost of acquisition and storage of those discarded features. On the other hand, comparing the different approaches to deal with multiple class problems, the one-vs-one scheme obtains better accuracy results in average than the others, although using a higher number of features. This approach is also more computational demanding than the others, because each sample must be tested for different classifiers, unifying their results later to obtain the desired output. From the experimental results achieved, the one-vs-one scheme should be used when there are enough samples per class and features, while, on the contrary, the one-vs-rest can be used with data sets with large number of features and reduced number of samples. Nevertheless, a deeper theoretical analysis needs to be done to support these hypothesis, trying to relate some properties of the data set (separability, number of classes, ratios samples/features, etc.) with the adequacy of a determined approach. Regarding the numerous combinations checked, several of them exhibited a good behavior, so it becomes difficult to select one. The entropy minimization discretizer seems to be more adequate than the rest of discretizers, and CFS filter is preferred when using naı̈ve Bayes classifier while consistency based filter is the best when applying C4.5. A comparative study was done to test the effectiveness of the combinations proposed compared to other existing methods. Six data sets obtained better performance results than those provided by other authors. As was shown in this study, the exhaustive methods for generating binary problems achieved good performance results and suggest a future line of research. Another interesting line can be the application of more advanced classifiers, such as Support Vector Machines that may obtain better performance results, also using them as part of the feature selection process. Finally, the combinations return very good results for the Leukemia data set that has a very high number of features with a reduced set of samples, therefore, in a more exhaustive study, the combinations would be applied to this appealing type of problems. R EFERENCES [1] T. Li, C. Zhang and M. Ogihara. A comparative study of feature selection and multiclass classification methods for tissue classification based on gene expression, Bioinformatics, vol. 20, no. 15, pp. , 24292437, 2004 [2] C. H. Yeang, S. Ramaswamy, P. Tamayo and S. Mukherjee and R. M. Rifkin and M. Angelo and M. Reicha and E. Lander and J. Mesirov and T. Golub. Molecular classification of multiple tumor types, Bioinformatics, vol. 17, pp. 316–322, 2001 [3] G. Madzarov, D. Gjorgjevikj and J. Chorbev. A multiclass SVM classifier using binary decision tree, Informatica,vol. 33, pp. 233-241, 2009 [4] Y. Ivar Chang and S. Lin. Sinergy of logistic regression and Support Vector Machies in Multiple-class classification, In Proc. IDEAL, LNCS, vol. 3177, pp. 132-141, 2004 [5] A. Golestani, K. Almadian, A. Amiri and M. JahedMotlagh. A novel adaptive-boost-based strategy for combining classifiers using diversity concept, 6th IEEE/ACIS Int. Conf. on Computer and Information Science, (ICIS). 0-7695-2841-4/07, 2007 [6] R. Kohavi and G.H. John. Wrapper for feature subset selection, Artificial Intelligence Journal, Special issue on relevance, vol. 97 no. 1–2, pp. 273–324, 1997 [7] I. Guyon, S. Gunn, M. Nikravesh and L. Zadeh, Feature extraction. Foundations and applications, Springer, 2006 [8] E.L. Allwein, R.E. Shapire and Y. Singer, Reducing Multiclass to Binary: A Unifying Approach for Margin Classifiers, Journal of Machine Learning Research, vol. 1, pp. 113-141, 2001 [9] G. Forman, An extensive empirical study of feature selection metrics for text classification, Journal of Machine Learning Research, pp. 1289– 1305, 2003 [10] T.G. Dietterich and G. Bakiri, Solving multiclass learning problems via error-correcting output code, Journal of Artificial Intelligence Resarch,vol. 2,pp. 263–285, 1995 [11] N. Jaopkowicz and S. Stephen, The class imbalance problem: A system study, Intelligent Data Analysis vol. 6(5), 2002 [12] Y. Yang and G.I. Webb, Proportional k-Interval Discretization for Naive-Bayes Classifiers, In Proceedings of the 12th European Conference on Machine Learning, pp. . 564-575, 2001 [13] U.M. Fayyad and K.B. Irani, Multi-Interval Discretization of Continuous-Valued Attributes for Classification Learning, Proceedings of the 13th International Joint Conference on Artificial Intelligence, pp. . 1022-1029, Morgan Kaufmann, 1993 [14] M. Dash and H. Liu, Consistency-based Search in Feature Selection, Journal of Artificial Intelligence, vol. 151, no. 1-2, pp. 155-176, 2003 [15] M.A. Hall, Correlation-based Feature Selection for Machine Learning, PhD thesis, University of Waikato, Hamilton, New Zealand, 1999 [16] J.R. Quinlan, C4.5: Programs for Machine Learning, Morgan Kaufmann, 1993 [17] I. Rish, An Empirical Study of the naı̈ve Bayes Classifier. Proceedings of IJCAI-01 Workshop on Empirical Methods in Artificial Intelligence, vol. 335, 2001 [18] I.H. Witten and E. Frank. Data mining: Practical machine learning tools and techniques, 2nd Edition, Morgan Kaufmann, San Francisco, 2005. http://www.cs.waikato.ac.nz/ml/weka/. Last access: February 2010 [19] The Mathworks. Matlab tutorial, 1984. http://www.mathworks. com/academia/student\_center/tutorials/. Last access: February 2010 [20] A. Asuncion and D.J. Newman. UCI machine learning repository, University of California, Irvine, School of Information and Computer Sciences, http://mlearn.ics.uci.edu/MLRepository.html. Last access: February 2010 [21] R. Fisher, Statistical methods for research workers, Oliver and Boyd, 1925 [22] W. H. Kruskal and W. A. Wallis, Use of ranks in one-criterion variance analysis, Journal of the American Statistical Association, vol. 47, no. 260, pp. 583621, 1952 [23] E. Frank and S. Kramer. Ensembled for Nested Dichotomies for MultiClass Problems, Proceedings of International Conference on Machine Learning, ACM Press, pp. 305-312, 2004 [24] N. Kerdprasop and K. Kerdprasop. Data partitioning for incremental data minig, The 1st International Forum on Information and Computer Science (IFICT) , Shizuoka University, Japan, pp. 114-118, 2003 [25] H. Liu and H. Zhang. Feature selection with dynamic mutual information, Pattern Recognition, vol. 42, no. 7, pp. 1330-1339, 2009 [26] N. Sánchez-Maroño, A. Alonso-Betanzos, R. M. Calvo. A wrapper method for feature selection in multiple classes datasets, J. Cabestany et al. (Eds.), IWANN, Part I, LNCS 5517, pp. 456-463, 2009 [27] http://www.mathworks.com/academia/student\ _center/tutorials/, Last access: February 2010. [28] P. W. Duin and A. K. Jain and J. Mao. Statistical Pattern Recognition: A review, 2000 [29] C. J. Alonso-Gonzlez, Q. I. Moro, O. J. Prieto, M. Aránzazu Simón. Selecting few genes for micro-array gene expresion classification, Actas Conferencia Española para la Inteligencia Artificial, pp. 21–31, 2009 2843 View publication stats