Machine Learning for Language Technology (2015) – DRAFT July 2015
Lecture 06: LAB Assignment
Weka: Perceptron (2): Parameter tuning
ACKNOWLEDGEMENTS : THIS LAB ASSIGNMENT IS BASED ON THE CONTENT OF THE WEKA BOOK .
SOME TASKS HAVE BEEN BORROWED FROM MARTIN D. SYKORA’S TUTORIALS. S OME TASKS HAVE
BEEN ADAPTED FROM WEKA TUTORIAL, PP. 577-578
Required Reading for this Lab Assignment
 Daume III (2014): 46-50; 60-62
 Witten et al. (2011): 147-156; 577-578
ATT: datasets can be downloaded from here:
<http://stp.lingfil.uu.se/~santinim/ml/2015/datasets/>
Free material:
Free Weka book (2005), 2nd edition:
<http://home.etf.rs/~vm/os/dmsw/Morgan.Kaufman.Publishers.Weka.2nd.Edition.200
5.Elsevier.pdf>
Learning objectives
In this lab assignment you are going to:
 use the VotedPerceptron classifier
 gain more insights into validation and parameter behaviour
Preliminaries
In Lab Assignment 03, we touched upon classification and looked at outputs provided
under Summary, Detailed accuracy by class and Confusion Matrix sections of the Classfier
output window. In this assignment we look at classifiers again and their validation..
Learning algorithms have parameters that can affect the outcome of the learning. We are
going to explore how parameter settings affect the performance.
Tasks
G tasks: pls provide comprehensive answers to all the questions below:
(1) Start Weka, launch the Explorer window and upload the spambase dataset. Select
the Classify. Under the Classifier section, select Voted Perceptron, leave the parameters as the
defaults. Run the model using each of the differerent Test options in turn.
(a) For Cross-validation, try different Folds values e.g. 2, 3, 5, 10,15, 30, 45, 90. What do
these values mean? What is interesting about making Folds equal to 90?
(b) For Percentage split, try different % values e.g. 20, 40, 60, 80. What do these values
mean?
Machine Learning for Language Technology (2015) – DRAFT July 2015
(c) For Supplied test set, use discretized_past_tense_testing.arff; what is the performance?
(d) Holdout (Percentage split), Cross-validation etc. are common techniques for
assessing accuracy based on randomly sampled partitions of the given data. The use of
such techniques to estimate accuracy increases the overall computation time, yet is
useful for model selection. How do each of the test options affect the performance of the
model (look at the visualisations as well as the text output)? Which test option gives the best
performance?
(2) Under the Classifier section, if you click on the textbox next to the Choose button, the
Object Editor will appear, clicking the More button brings up information about the
classifier itself. VotedPerceptron’s parameters are:
▪
▪
▪
▪
I: The number of iterations to be performed. (default 1) Range: real; -?-+?
E: The exponent for the polynomial kernel. (default 1) Range: real; -?-+?
S: The seed for the random number generation. (default 1) Range: real; -?-+?
M: The maximum number of alterations allowed. (default 10000) Range: real; -?-+?
Change the default parameters and motivate your choices. Run the classifier with your
hand-selected new parameters (select 10-fold Crossvalidation). Compare these results
with the performance you got with default parameters. How does changing the
classifier’s parameters with the values you selected affect the performance? Briefly
explain and discuss the results.
(3) Theoretical questions: which are the main differences between a perceptron, a
weighted perceptron and an average perceptron.
VG tasks: pls provide comprehensive answers to all the questions below:
(4) Weka’s metalearner CVParameterSelection searches for the best parameter settings
by optimising cross-validated accuracy on training data. By default, each setting is
evaluated using tenfold cross-validation. The parameters to optimize are specified using
the CVParameters filed in the Generic Object Editor window. Click on the More button in
the Generic Object Editor window for more information and examples. Run
CVParameterSelection with the VotedPerceptron and the spambase dataset to find the
best parameter settings. Compare the output you get here to the output you got with the
default parameters above. Has accuracy changed? What parameter values were selected
by CVParameterSelection? Briefly explain and discuss the results.
(5) Theoretical question: What are perceptron’s limitations?
To be submitted
A one-page written report containing the reasoned answers to the questions above and
a short section where you summarize your reflections and experience. Submit the report
to santinim@stp.lingfil.uu.se no later than 29 Nov 2015.
*-*-*