From: FLAIRS-01 Proceedings. Copyright © 2001, AAAI (www.aaai.org). All rights reserved.
Skill
Refinement Through Competence Feedback
David Patterson
wd.patterson@ulst.ac.ukSchoolof Informationand SoftwareEngineering, University of Ulster, NorthernIreland BT370QB
Sarabjot S. Anand
ss.anand@ulst.ac.ukSchoolof Informationand SoftwareEngineering, University of Ulster, NorthernIreland BT370QB
John Hughes
jg.hughes@ulst.ac.ukSchoolof Informationand SoftwareEngineering, University of Ulster, NorthernIreland BT370QB
Abstract
Learningis generally performedin two stages, knowledge acquisition and skill refinement. Developments
within machinelearning have tended to concentrate
on knowledgeacquisition as opposedto skill refinement. In this paper we develop mechanismsfor skill
refinementwithin the context of lazy learning by incorporating competencefeedback into the exemplar
base. The extent of competencefeedback dependson
the richness of the data collected during task execution. Wepresent two techniques for competencefeedback, exception spaces and knowledgeintensive exception spaces (KINS).The techniques differ in the
extent of competencefeedback and the resulting degree of skill refinement. Previousdefinitions of exception spaces and KINSare extendedand the resulting improvementis evaluated using six data sets. A
genetic algorithmis utilised to optimise the definitions of the exception spaces and KINSfor each exemplarin the exemplarbase. Wealso provide a visualisation of KINSand exceptionspaces with an aimto
exemplify howthese mechanismsfor skill refinement
affect the structure of the exemplarbase.
1
Introduction
A general model of learning consists for four main components: the environment, knowledgebase, learning element and performance element [1]. The environment provides the inputs (raw data) into the leaming task through
various sensors. The environment may also take on the
role of teacher and provide the expected outcome for a
set of inputs, during learning. Based on the inputs from
the environment, the performance element is expected to
perform a task of some description with some degree of
competence using knowledge it has access to within the
knowledgebase. The learning element is central to this
system and aims to bridge the gap between the level of
information available from the environment and that required by the performance element. Thus, the learning
element converts inputs from the environment into
knowledge stored within a predefined representation, a
process known as knowledge acquisition. Based on the
394
FLAIRS-2001
amount and complexity of the computation that the
learning task must undertake, it may be considered as a
lazy or eager learner. The latter characteristically generalize/abstract their input values into more general representations such as rules or decision trees which are subsequently stored in the knowledge base and used by the
performance element. On the other hand lazy learners,
such as the nearest neighbor (k-NN), perform no processing of inputs initially and wait until a request is made
for task execution. Whencalled upon to perform a task,
the performance element utilises the previously stored
inputs to perform the task. Apart from performing the
task itself, the performance element must also perform
the function of learning. A characteristic of the lazy
learners is that any learning carried out during task execution is discarded after the task has been performed.
Other characteristics
include a large memoryrequirement, low training costs, efficient solution reuse, high
perspicuity and inefficiency in knowledgedeployment.
In addition to performing the task, whose performance
enhancement is the goal of learning, the performance
element may also provide feedback on its own competence. This feedback maybe utilised to refine the learnt
knowledge, leading to improved competence and goaloriented learning, knownas skill refinement. For example, the backpropagation algorithm used in learning neural network weights uses competence (measured by the
observed error in prediction/classification) as a driver of
search within the hypothesis space. In comparison, decision tree induction algorithms do not utilise such feedback while performing a greedy search of the search
space.
The goal of this paper is to present two techniques for
incorporating competencefeedback into a nearest neighbor model with the aim of improving its competence on
future tasks.
The rest of the paper is in the following format. The next
section (Section 2) presents a summaryof previous related work that forms the motivation for the research preCopyright
©2001,
AAAI.
Allrights
reserved.
sented in this paper. In Section 3 we describe the implementation of competence feedback within the context of
the k-NN algorithm using exception spaces and KINS.
This is followed in Section 4, by a visual interpretation of
KINSthat demonstrates how they generalize the definition of exception spaces and alter the structure of the exemplar base. The empirical evaluation of this implementation of competence feedback using six data sets from
disparate domains is presented in Sections 5 and 6. Finally, Section 7 draws conclusions from the evaluation
and suggests future work in the development of KINS
and exception spaces.
2
Related
work
Whenpresented with a target example, the 1-Nearest
Neighbour (I-NN) [3] retrieves the most similar exemplar from the exemplar base (based on the Euclidean distance metric) and allocates the class of the retrieved exemplar to the target example. The 1-NNhas been criticized on a number of accounts: expensive due to their
large storage requirements, sensitive to the choice of
similarity function, cannot easily work with missing attribute values, cannot easily work with nominal attributes
and does not yield concise summaries of the concept [4].
The IB series [5] of enhancements to the basic nearest
neighbour algorithm addressed a number of these criticisms, as did work by Cost and Salzberg [6].
Anand et al. [2] classify the extensions to the nearest
neighbour along the following four dimensions: kdimension, attribute weight dimension, exemplar weight
dimension, distance metric dimension. The development
of exception spaces and KINS is along the exemplar
weight dimension and thus we shall concentrate our discussion primarily on extensions proposed along this dimension.
Cost and Salzberg [6] weighted an exemplar according to
howreliable a classifier it proved itself to be. Thus, implementing the notion of competencefeedback for classification goals, resulting in the restructuring of the exemplar base. Goodclassifiers were assigned a small weight
(close to 1) which had the effect of making them appear
closer to a target exemplar and therefore more likely to
be retrieved from the exemplar base. Poorer classifiers
were assigned larger weights (>1) which has the opposite
effect on them. The rationale behind this is that poor
classifiers are either noise or exceptions within the exemplar base describing regions of the instance space where
normal rules are not applicable. Geometrically exemplar
weights define circular regions around exemplars, called
exception spaces The size of the exception space is inversely proportional to the size of the weight. For an exemplar to be retrieved from the exemplar base when presented with a target exemplar, the target must fall within
the region bounded by the exemplar’s exception space
(see Section 4).
In IB3, Aha et al. [5] first introduced the notion of competence feedback. The principle focus of this feedback
was to make the algorithm more tolerant to noisy exemplars. IB3 applies a selective filter to prevent noisy exemplars from being used to misclassify new exemplars.
Exemplarsare excluded based on their classification record. A poor classification record indicates that an exemplar’s most similar neighbours in the exemplar space have
a different classification to itself and that it is therefore a
noisy exemplar. Exemplarsare acceptable if their classification accuracy is significantly greater (statistically)
than their class’s observed frequency and excluded if it is
significantly
less. System competence is improved
through time by applying this filter through continual
feedback. In this way an exemplar already included
within the exemplar base can be excluded as the system
changes dynamically with the addition of new exemplars.
Note that as opposed to Cost and Salzberg’s exception
spaces, exemplars within IB3 are either in the exemplar
base or out, there is no grading based on competence.
IB4, in addressing the issue of attribute relevancy under
differing contexts, also uses competence feedback. The
weights reflect the relevance of the attributes used to describe each exemplar.
3
Competence
Feedback
using
Exception Spaces and Knowledge Intensive
Exception
Spaces
Anandet al. [2] extended the exception space definition
to apply not only to classification problems (Cost & Salzberg model) but also to regression problems through the
use of a goodness membership function (GMF) which
redefines the measure of usefulness (goodness) of an exemplar. The GMFprovides a methodology whereby exemplars can be weighted according to their competency
despite having a continuous output field. The global
weight of an exemplar is defined ultimately by a
weighted quartile measure formula (see later), which
uses the GMF,and is defined as the ratio of the uses of
the exemplar to the correct uses of the exemplar. For
classification problems an exemplar is said to have been
retrieved correctly if it is used to classify a target problem into its correct class. For regression problemsit is a
bit moredifficult to define.
Cross validation using the training data and employing
the nearest neighbor algorithm results in a distribution of
predictive errors of individual exemplar retrievals which
can be described using the statistical measure of spread
namely quartiles. Those exemplars resulting in a low predictive error will fall into quartile I(Q1), those with the
worst errors will fall into Q 4, with exemplars exhibiting
errors between these extremes falling into Q 2 & Q3 respectively. A GMFis defined on the quartiles according
to the degree to which a retrieval of an exemplar, producing an error in that quartile, is considered a good use
MACHINELEARNING 39S
of the exemplar. Therefore a GMFof {1.0, 0.7, 0.3, 0}
implies that the retrieval of an exemplarwhenit produces
an error falling into Q 1 is good, with a gradual deterioration in goodness downto Q 4 where the retrieval of an
exemplar producing an error in this quartile is poor (a
goodness membershipvalue of 0). As an exemplar can be
retrieved a number of times throughout the cross validation process it may produce errors in more than one
quartile. Therefore the weighted quartile measure formula is used to calculate its overall goodness and hence
its global weight as shownbelow.
4
iE nx
i=1
Wx
4
i=|
where, P.i is the membershipvalue for quartile i as defined by th¢ GMF
and
n i is the no. of retrievals of x in quartile i
x
As defined by Cost and Salzberg, a small global weight has
the effect of makingan exemplarappear closer to a target.
A large global weight has the opposite effect and makesit
appear to be further awayand therefore less likely to be
retrieved. Theseweights can be visualized (section 4)
circular boundaries around exemplars, knownas exception
spaces. The radius of the exceptionspace is inversely proportional to the size of the global weight. Smallweights
therefore produce large exception spaces. For an exemplar
to be retrieved for a target the target mustfall into the exception space of the exemplar. Exemplarswith small exception spaces (defined by large global weights) are therefore less likely to be usedin retrieval.
Exemplar #
I
I
2
3
3
Error Quartile
I
3
I
3
4
Frequency.
3
2
4
4
I
Table I Competence
Feedbackdata used to define Exception
Spaces
The competencefeedbackdata utilized within the definition
of exception spaces takes the form shownin Table 1. Here,
examplefeedback on the competenceof the first three exemplars from one of the housing data sets (see Section 5 for
a brief description) is shown.This data is interpreted as
follows: Exemplar1 was retrieved five times in total. Of
these, three times the error resulting from the retrieval was
in the first quartile of the error distribution resulting from
the use of the exemplarbase as a whole, while twice the
retrieval resulted in errors in the third quartile of the error
distribution.
396
FLAIRS-2001
Usingthe GMF
defined previously of { 1.0, 0.7, 0.3, 0} and
the weighted quartile measureformula this will producea
global weight for exemplarl as follows.
w] = 5/(3"1 + 2*0.3)= 1.38
If exemplar1 was retrieved again, this time producingan
error in Q4its newglobal weight wouldbe
wl -6/(3"1 + 2*0.3 + 1"0)= 1.67
In this wayfeedbackfromretrievals is used to constantly
refine the size of the exceptionspaces and therefore the
structure of the exemplarbase.
While some exemplars always produce poor predictions
(error in Q4)for a range of targets and others alwaysproduce good predictions (error in Q1), moreoften exemplars
display a range of goodand bad usage for different target
exemplars. That is, exemplarsare generally not universally
goodor bad predictors. The reason for this mayeither be
noise or that these differing results occur underdifferent
retrieval circumstances. Onewayof describing the retrieval
circumstancesis in terms of the individual attribute distanees betweenthe feature vectors describing the exemplar
and target example.Oneshortfall in the reasoning behind
exception spaces is that they do not take the circumstances
of retrieval into consideration. That is, evenif an exemplar
producesgood predictions sometimesit will be discriminated against by the times it producespoor predictions.
WhatKINSattempts to accomplish is to removethis bias by
attaching a weight to an exemplarwhich is determined by
the circumstancesof its retrieval. Thereforeundercertain
circumstances KINSwill attach a small weight to an exemplar and under other circumstances a larger weight. As more
knowledgeis being used to define these new exception
spaces, they are knownas KnowledgeINtensive exception
Spaces (KINS). To enable this dynamicallocation
weights, whichare dependenton the retrieval circumstances, additional competencefeedbackdata is generated
for each exemplarduring cross validation including:
1) the individual attribute distances betweenthe exemplar
and the target
2) the quartile into whichits error in prediction fell.
Examplecompetencefeedback data for a single exemplaris
shownin Table 2 which was generated using a data set from
the colorectal cancer domain(see Section 5 for a brief description of the data set). Notethat only the last rowin Table 2 is used whendefining exception spaces.
As can be seen from Table 2, KINSutilizes moredetailed
competencefeedbackthan exception spaces. A classifier is
built using this data for each exemplar.The individual attribute distances are the independentattributes that are input
to the classifier and the error quartiles are the classification
labels that the classifier must learn to discriminate between.
For example let us assume that the classifier for a particular
exemplar is defined as KINS1. This classifier maybe interpreted as follows: if the distance between the values of attribute Age of the target exemplar and the retrieved exemplar is less then 0.2 then assign a weighting of I(Q1). If the
distance between them for attribute Age is >= 0.2 and the
distance between them for attribute Dukes Stage < 0.6 then
assign a weight of 2 (Q2). If the distances for Age is >= 0.2
and for Dukes Stage is >= 0.6 then assign a weight of 3
(Q3).
the structure of the exemplar base. Note that using a rule
based classifier is not necessary for defining KINS. Any
type of classifier may be used. Weuse rules here as it makes
the visualization of KINSmore intuitive.
KINSI: if d(Age) <0.2
then weight = 1
else if d(Age) >=0.2 & d(Dukes Stage) <
then weight = 2
else if d(Age) >=0.2 & d(Dukes Stage) >= 0.6
then weight =3
4
Attribute
Sex
Pathological
Type
Polarity
Tubule Confi~uration
Tumour Pattern
Lymphocytic
Infiltration
Fibrosis
Venous Invasion
Mitotic Count
Penetration
Differentiation
DukesStage
A/~e
Obstruction
Site
Error Quartile
Use#1
0
0
Use#2
0
0
Use#3
0
0
Use#4
0
0
Use#S
0
0
0
0
0
0
0
0.111
0
0
0
0
0
0
0
0
0
0
0
0
0.25
0
0.25
0
0
0
0.25
0
0
0.56
0
0
0.027
0
0
0.062
0.0008
1
0
3
0.25
0.062
0
0.25
0.0008
0
0
4
0.25
0
0.25
0.062
0.02
0
0
3
0.11
0
0
0.062
0.016
0
0
3
0
0
0
0.062
0.058
0
0
3
Now, when a new target example is encountered the KINS
rules for an exemplar are used to classify the current retrieval circumstances (individual attribute distances) into
one of four error categories (quartiles). The exemplar
weight associated with the relevant quartile is then used to
compute the "true" distance of the target from the exemplar.
Visualizing
Exception Boundaries
Exception spaces and KINS effectively
incorporate competence feedback into the exemplar base by altering its
structure. While exception spaces define circular boundaries (assuming a two dimensional space) around an exemplar, KINS define more complex spaces. As described
earlier,
in exception spaces the weight assigned to the
exemplar defines the radius of the boundary using an
inverse relationship
whereby the larger the weight the
smaller the boundary. Figure I shows the exception
spaces defined for exemplar X using exemplar weights of
1,2 and 3. In this example, exemplar X will only be retrieved for a target, T, if the exemplar weight associated
with X, at the time of retrieval, is less than or equal to 2.
o
Table 2 CompetenceFeedbackdata utilised by KINSshowing
the difference in attribute values betweentarget exemplarsand
a retrieved exemplar.
KINS, as defined above, removes the bias introduced by
exception spaces which associate only one global weight
with an exemplar. The definition of K1NSis a dynamic process where the competency of the system improves over
time through introspective feedback from exemplar retrievals. Each time an exemplar is retrieved, competence feedback data in the form shown in Table 2 is presented to the
learning element and the classifier is used to refine the
learned knowledge that defines the KINS. This dynamically
modifies the shape of the exception boundary around an
exemplar through introspective feedback, thus modifying
Age
Figure 1: Visualising Exception Spaces defined on exemplarX
KINS enable a less rigid and more realistic
boundary to
be defined around exemplars as seen in Figures 2 and 3.
To help visualize how KINS changes the structure
of the
exemplar base let us consider the previous KINS rule
(KINS1) and the one defined as KINS2 both based on the
coloreetal cancer data. For simplifying the discussion, we
assume here that the only attributes
found to be significant in defining the KINS are Age and Dukes Stage.
KINS2: if d(Age) <0.2
then weight = 1
else if d(Age) > =0.2 & d(Dukes Stage) >
then weight = 2
else if d(Age) >=0.2 & d(Dukes Stage) <= 0.6
then weight =3
MACHINE
LEARNING397
KINS2can be interpreted in a similar way as KINS1.
Note here that the weights used in defining KINSincrease as the estimated error increases. On the other
hand, in the case of the GMF,as the defining weights are
measures of goodness, they reduce as the estimated error
¯ increases.
Nowlet us see how these two KINScan be visualized.
Figures 2 and 3 show two exemplars, X and T. Here, X is
an exemplar within the exemplar base and T is the target
exemplar, for which a prediction is to be made. Weassume that KINSI and KINS2 are alternative
KINS defined around exemplar X. The three concentric circles
around exemplar X correspond to the exception boundaries around the exemplar, defined by the three weights l,
2 and 3 within the definitions of KINS1and KINS2.The
smallest weighting of 1 corresponds to the largest circle.
Notice that up until this point we have not considered
antecedents of the rules in the KINSdefinition. The two
rectangular areas bounding the exemplar correspond to
the region on the Age axis where the distance between
the target and retrieved examplesis less then 0.2 and on
the Dukes Stage axis where the distance is less then 0.6.
These areas correspond to the antecedent expressions in
the KINSdefinitions.
Age
Figure2: Visualizing KINSIdefined on Exemplar
X
Figure 2 shows the exemplar with the regions defined by
KINSI shaded while Figure 3 illustrates
KINS2. If exemplar X had KINS1 defined on it, exemplar X would
not be retrieved when attempting to make a prediction for
the target exemplar as it lies outside the KINSboundary
defined. On the other hand, as T falls within the KINS
boundary region defined by KINS2, X would be retrieved
in this case. Thus, KINSIand KINS2result in different
restructuring of the original exemplarbase.
Age
Figure 3: Visualizing KINS2defined on Exemplar
X
5
Competence
Methodology
Feedback
Evaluation
To evaluate the effectiveness of the proposed competence
feedback mechanismswe empirically evaluate them using
the followingsix data sets.
1. Housing1 consists of 565 records and I0 attributes
taken from a housing database supplied by the Valuation and Lands Agencyof Northern Ireland. The goal is
to build a modelfor predicting house price.
2. Housing2 consists of 584 records and 10 attributes
taken from a different housingdatabase. It also is a regression problemaimedat predicting house price
3. Loansconsists of 430 records each with 23 attributes
taken from a lending institution database. This is a binary classification learning problemof predicting
whetherto give a customera loan or not. For the purposes of this paper we transformed the probleminto
one of predicting the propensity of the applicant being a
good investment. Thus the outcomeattribute was a
continuousattribute with domain[0,1 ].
4. Car Insuranceconsists of 600 records and 19 attributes. The aimis to predict the propensity of a customer
to renewa ear insurance.
5. Breast Cancerconsists of 683 records and 9 attributes
taken from at the UCIMachineLearning Repository
[13]. The data was originally part of the Wisconsin
Breast CancerDatabases.It is a binary classification
learning problem,the aim being to predict whether a
tumor is benign or malignant. Onceagain the problem
was transformedto that of predicting the likelihood of
the tumor being malignant.
6. Colorectal Cancerconsists of 188 records each with 15
attributes taken from a database of patients from the
RoyalVictoria Hospital and the Belfast City Hospital,
Belfast, NorthernIreland. This is a regression problem,
the aim being to predict the numberof monthsthe patient is expected to survive after being diagnosedwith
colorectal cancer.
In the original definition of exception spaces for regression problems, Anand et al. suggested defining the GMF
398
FLAIRS-2001
using error quartiles [2]. The exemplar weight that defines the exception space around an exemplar was defined by the weighted quartile measure formula. A drawback of the initial GMF
definition, (an integral part of the
formula) was that the GMF(each of the ~i’s) is user
fined. To rectify this we use a genetic algorithm to search
the space of all possible GMF
definitions.
Additionally the original definition of exception spaces
was based on quartiles of the error distribution. This was
chosen arbitrarily as an initial basis of investigating the
validity of the technique in extending the definition of
exception spaces for regression models. In this paper we
investigate the effects of extending this work through the
implementation of a continuous definition of the GMF
(which therefore no longer requires the weighted quartile
measure formula to define exemplar weights). The continuous GMFis defined as:
/a(e) emax - e
ernax
- emi
n
where, e is the actual error value
and emax and emi n are the maximumand
minimumerror produced by all retrievals
This definition of the GMFis a simple linear function of
the error due to exemplar retrieval. Wealso investigate
the effects of dividing the error distribution into eight and
twelve intervals to determine if this improves the model
further. Clearly, the numberof error intervals on which
the GMFis defined will define the complexity of the
search space to be searched by the genetic algorithm. For
example, if we assume the membership values, P-i, all
have a precision of two decimal points, the size of the
search space would be 100", where n is the number of
intervals that the error distribution has been split into
within the GMFdefinition. Thus, the more complex the
GMFdefinition, the more computationally expensive will
be the search undertaken by the genetic algorithm.
Initial results reported by Anandet al. using KINSwere
encouraging [2]. In this paper we also address some of
the deficiencies within the preliminary definition of
KINS. The deficiencies within KINSdefinition are the
same as those with the GMFdefinition in relation to exception spaces. In the original definition of KINSthe
quartile interval weights were assigned arbitrarily as was
the choice of the numberof intervals into which the error
distribution was split. In this paper we use a GAto optimise the interval weights and we investigate the effects
on competency of splitting the error distribution into
eight and twelve intervals instead of using quartiles. The
numberof intervals into which the error predictions must
be split clearly has a direct effect on the sparsity of the
competence feedback data (Table 2) used to create the
classification function that defines the KINSfor an exemplar. Increasing the numberof intervals would clearly
have a negative influence on the accuracy of the discovered classifier as less data wouldbe available to achieve a
useful generalisation. The improvement in competence of
the nearest neighbor model when using KINS would be
affected by the accuracy of the classifier defining the
KINS. So any reduction in competence of the classifier
will have a direct, derogatory impact on the value of using the KINS.Additionally, the complexity of the search
for the optimal weights will also dependon this choice of
numberof intervals, as in the ease of the GMF.
6
Results
of Evaluation
The k-NN algorithm [10] employed in the experiments
used the Euclidean distance metric, retrieved five neighbors for each target and arrived at a prediction using a
weighted sum of the output values for the retrieved
neighbors. The weights used in the prediction were based
on the distance of the neighbors from the target.
To evaluate the competence improvement achieved by
using exception spaces and KINS, we used three fold
cross validation. Thus, three models were generated for
each data set and the results shown in Table 3 are the
average mean absolute errors for the three training and
test data sets for each of the data sets. Three-fold cross
validation was used to minimize any sampling errors in
the reported results.
To obtain optimal KINSand exception space definitions,
a genetic algorithm was employed to search the search
space of all GMFsand KINS. The genetic algorithm used
a population size of 50, a crossover probability of 0.8, a
mutation probability of 0.1, linear sealing of fitness, 2nd
level of incest prohibition [11] (i.e. crossover between
two chromosomesthat shared either parents or grandparents was prohibited) and uniform crossover. Selection
was based on survival of the fittest using the stochastic
remainder sampling with replacement method [ 12]. Tenfold cross-validation
employing the kNNalgorithm on
the training data, with the parameter settings discussed
above, was used as the fitness evaluation function. The
results presented here were the optimal solution found by
the genetic algorithm in 50 generations.
For the GMF,it clearly makes sense for the goodness
value associated with intervals pertaining to large errors
to be smaller than that associated with an interval pertaining to smaller errors. Thus, the genetic algorithm was
provided with the domain knowledge that the optimal
solution will be a monotonically decreasing function. For
KINS,the exemplar weights increase as the error quartile
increases. Thus, in this ease, the genetic algorithm was
provided with the domain knowledge that the optimal
solution will be a monotonicallyincreasing function.
As can be seen from Table 3, competence feedback improves the competence of the k-NNmodel for five out of
the six data sets. Also, in general, in cases where competence feedback produces worse results on the test data, it
also does so on the training data set - clearly a desirable
MACHINELEARNING 399
feature for any model. Twoexceptions to the rule are
K.INSin the Housing1 data and exception spaces in the
colorectal cancerdata set.
Competence MAEon
Data Set Feedback Training
data
Housing1 None
4751.76
Exception 4777.22
~paces
Continuous 4962.67
KINS
4550.34
MAEon ImproveTest
ment on
data
Test data
5209.31
5393.93 -3.54%
Exception Spaces KINS
HousingI. ¯ 1,0,96~0.92,0~92.~(.’,.,.
i:I~,:~’8~8~~~..
Housing
2
0.93,0.93,0.9,0.87
1.85,1.98,1.99,1.99
Loans
¯ 0.99, 0.99,0.:i,0 ) li~l~O~l!i’9~~i
CarInsurance
0.95,0.95,0.89,0.15 1,1.14,1.2,1.61
Breast Cancer 0.88,0.75;0.75,0.g51:2~ i~-52~1~3,~(~f8
Colorectal
0.7,0.64,0.64,0.15 1.3,1.4,1.67,1.98
Cancer
¯ " .’: .~ ~... " ~ ’ :’ r,.: .... ,. "~ ",’;’~
Loans
Breast
Cancer
C~tiil-tioUs
7.5’04"~3!-:~:’:
KINS
7146;55
None
0.665
Exception 0.446
spaces
Continuous 0.441
0.465
KINS
0.465
0.497
None
0.068
Exception 0.039
spaces
0.12
0.097
expected to suffer if the numberof intervals was increased due to the competencefeedback data becoming
too sparse for buildingcompetentclassifiers for the individual exemplars.To test these hypothesesweused eight
and twelve intervals on the colorectal cancer data. The
results of these experimentsare summarizedin Table 5.
As can be seen fromTable5, using eight or twelve intervals reduces the gain in competenceexperienced when
usingonly four intervals.
Table 4: OptimalGMF
and KINSdefinitions for 4 Intervals
29.4%
24.5%
4 Intervals 8 Intervals 12 Intervals
Test Train Test Train Test Train
Exception 26.36 29.96 26.5430.03 28.51 30.79
Space
KINS
21.35 27.96 21.63 28.08 21.36 28.84
Table5: Mean
AbsoluteErrorwhenusingmorethan4 intervals for definingKINS
andException
Spaces
One surprising outcomefrom the experiments was that
the KINSmodel of competence feedback didn’t show
consistent improvement
over the exception spaces model.
This is not an unreasonable assumptionto makeas KINS
is utilising moredetailed knowledgeto calculate exemplar weights. Out of the five data sets wherecompetence
feedback improved the competencyof the k-NNmodel
KINSproduced modelsof similar competencyto exception spacesin three data sets, it faired worsein one data
set and improvedthe competencyin one data set. Obviously there is an increase in the computationaloverheads
Table3: Effect of Competence
Feedback
on modelcompetence associated with the KINSmodelwhichdecreases the efficiency of the system. This overhead is acceptable if
Table 4 shows the optimal definitions for the GMF
and
there is an associated benefit in terms of competency.If
KINSweights as discovered by the genetic algorithm.
Interestingly, there is little variationin the weightsacross there is no benefit then the exercise is a futile one. There
are a numberof possibilities, whichmayexplain this. The
the four quartiles for the two housing data sets. This
six data sets used to evaluate the KINSand exception
wouldsuggest that the competencefeedbackdata in these
space modelsof competencefeedback were quite small.
sets could not improve much upon the competence
This in turn wouldmeanthat during the initial ten-fold
achieved by the unweightedk-NNmodel. This is clearly
cross validation used to produce the competencefeedreflected within the results in Table3 for these two data
back data, each exemplarwouldonly be retrieved a limsets.
ited numberof times. Thefeedbackdata for an individual
exemplaris used to define its KINSthroughthe building
In Section 5 we discussed howincreasing the numberof
of a classification function for the exemplar.This lack of
intervals into whichthe initial error distribution is split
affects the complexityof the search space for the genetic data mayhavelimited the effectiveness of the classifier
to define accurate KINSfrom the introspective feedback
algorithm. Wesuggested that the increased size of the
search space mayresult in the genetic algorithmreturning process. Additionallythe choiceof classifier, particularly
its ability to avoid over-fitting, mayalso havehadan efsub optimal solutions. Wealso noted that KINSwouldbe
4OO FLAIRS-2001
16.7%
feet on the effectiveness of KINS.The results in Table 3
suggest that over-fining is occurring when using KINS,
especially in the case of the Colorectal Cancer and Breast
Cancerdata sets.
Conclusions
& future
work
7
In this paper, we introduced two mechanisms for skill
refinement through competence feedback within the
context of lazy learning. These techniques, exception
spaces and KINS,differ in the extent to which they use
the available feedback data. The techniques are independent of the distance metric employed and can potentially be extended to handle structured representations of
exemplars. The techniques have been validated using six
data sets. There is significant improvement in competence of the exemplar base when using competence feedback. In this paper, we assumedall independent attributes
to have equal relevance to the task of predicting their
relevant output fields. However,Anandet al. have shown
previously that KINSand exception spaces are effective
techniques even whenusing attribute weightings [2].
A further extension to this evaluation of the two competence feedback models based on synthetic data sets will
be carried out to attempt to characterize circumstances
where KINSimproves over exception spaces and when it
is not beneficial to use competencefeedback at all.
Oneextension to this work is to investigate the effects of
larger data sets on defining KINSto see if the provision
of more examples to the classifier during training improves the KINSdefinition leading in turn to more competent results. Related to this is the need to investigate
the use of different classifiers.
A direct extension of the work presented in this paper is
an investigation into whether the use of characteristics of
the error distribution in defining the continuous GMFcan
add value. Also, the effectiveness of the genetic algorithm’s parameters in improving its convergence characteristics needs to be undertaken. Techniques for reducing
over-fitting
by the genetic algorithm of the GMFand
KINSdefinitions, to the training data also need to be investigated. The authors have previously suggested the use
of a fitness function that takes into account not only the
mean absolute error but also the variance of the model
across the cross-validation folds [10] with a view to controlling over-fitting. This approach needs to be evaluated
in this context. The competence feedback mechanisms
defined in this paper assumethat the exemplars are represented as feature vectors and that the features are independent of each other. An extension to these mechanisms
would be to incorporate structured exemplars and exemplars with interacting features [8].
matching case that cannot be adapted, adaptation knowledge should be considered during retrieval to choose the
most closely matching case which can be adapted [9].
Therefore, to be used effectively in CBR, these competence feedback mechanisms must be extended to take
adaptation knowledgeinto consideration within its definition.
8
References
[1] Barr and Feigenbaum. The Handbook of Artificial
Intelligence, vol. 3. pp 326-333, 1982.
[2] S. S. Anand, D. Patterson, J. G. Hughes. Knowledge
Intensive Exception Spaces, Proceedings of AAAI-98,pp
574-579
[3] Cover, T.; and Hart, P. Nearest Neighbour Pattern
Classification, IEEETransactions on Information Theory,
13(1): 21-27, 1967.
[4] L. Breiman, J.H. Friedman, R.A. Olshen, C.J. Stone.
Classification and Regression Trees, Chapmanand Hall,
1990
[5] Aha, D.; and Kibler, D. Noise Tolerant instance-based
learning algorithms. In Proceedings of the 1 l th International Joint Conference on Artificial Intelligence, 794799. MelnoPark, Calif., 1989
[6] Cost, S.; and Salzberg, S. A Weighted Nearest
Neighbour Algorithm for Learning with Symbolic Features. MachineLearning 10: 57-78, 1993.
[7] Clementine User Guide, Integral Solution Limited,
1998.
[8] D. Aha, D. Wettschereck (ed.). Proceedings of the
ECML-97 Workshop on Case-Based Learning: Beyond
Classification of Feature Vectors, 1997
[9] Smyth, B. & Keane, M. T. (1998) Adaptation-Guided
Retrieval: Questioning the similarity Assumptionin Reasoning. Artificial Intelligence, (102)2, pp. 249-293
[I0] S. S. Anand, J. G. Hughes. Hybrid Data Mining
Systems: The Nest Generation, Proceedings of 2nd Pacific-Asia Conference in KnowledgeDiscovery and Data
Mining, 1998.
[11] R. Craighurst, W. Martin. Enhancing GAPerformance through Crossover Prohibitions Based on Ancestry,
Proc. of the 6th International Conference on Genetic Algorithms and Applications, pp. 130 - 135, 1995.
[12] M. Srinivas, L. M. Patnaik. Genetic Algorithms: A Survey, IEEEComputer,Vol 27, No. 6, June, 1994.
[13] C. L. Blake, C. J. Merz. UCIRepository of machine
learning databases, Irvine, CA:University of California,
Department of Information and ComputerScience, 1998.
The technologies developed within this paper have important implications to retrieval in case-base reasoning
(CBR).As there is little advantage on retrieving a closely
MACHINELEARNING 401