Assignment 5
Overview
Investigate performance evaluation in Weka.
Questions
1. Explore the differences between evaluating an algorithm by (1) testing on the
training set, (2) using a held-out test set, and (3) using cross-validation. In this
exercise, we will evaluate both the (1) J48 and the (2) DecisionStump tree
learning algorithms on several datasets. Use the following five datasets that come
with Weka: (1) contact-lenses, (2) iris, (3) labor, (4) soybean and (5) weather.
Construct a table of classifier vs. datasets vs. testing procedure, and in each cell
enter the accuracy of that classifier for that dataset and test procedure. In your
discussion of your results consider the following: (1) the size of the dataset, (2)
the number of attributes, and (3) the complexity of the model.
2. Next, we will use Weka to generate ROC curves for NaiveBayes and J48 on the
labor dataset. First, we need to generate and save ROC curve data.
Instructions






Using the Weka Explorer open the labor dataset under the Preprocess tab.
Under the Classify tab, choose the NaiveBayes classifier and click Start to
perform a 10-fold cross-validation test.
In the Result list window, right-click on the NaiveBayes entry and choose
Visualize Threshold Curve and class "good". The visualization window
will appear.
If necessary, change the X axis to be False Positive Rate, and change the
Y axis to be True Positive Rate. You should now see the ROC curve.
Click Save and store the results to a file in ARFF format.
Exit the visualization window and repeat the above for the J48 classifier
with default settings.
Now, we need to load the data into Excel (or some other charting software) to
visualize the ROC curves for both classifiers at once. Here's an outline of the
process for Excel.

Edit the two ARFF files containing the threshold curve results saved
above and remove everything above and including the "@data" line. Note
that the False Positive Rate and True Positive Rate values are the
sixth and seventh entries, respectively, in each line.




Open Excel and choose Data -> Import External Data -> Import Data.
Browse to the first ARFF file and load it as a comma-delimited file. Do
the same for the second ARFF file.
Insert a chart of type scatter line plot and put two lines on the plot: one is
TP vs. FP for NaiveBayes, and one is TP vs. FP for J48.
This chart will now show the two ROC curves for NaiveBayes and J48 on
the labor dataset.
Format your chart with a title, correct axis titles, and correct legend titles.
Discuss your conclusions about the performance of NaiveBayes and J48 on
the labor dataset based on the appearance of the ROC curves.