From: AAAI Technical Report WS-94-01. Compilation copyright © 1994, AAAI (www.aaai.org). All rights reserved.
A ComparativeUtility Analysis of
Case-Based Reasoning and Control-Rule Learning Systems
Anthony G. Francis, Jr. and Ashwin Ram
Collegeof Computing,GeorgiaInstitute of Technology
Atlanta, Georgia30332-0280
{centaur, ashwin}@ce.gatech.edu
learner,2a type of systemthat attempts to improveits
problem-solvingperformanceby learning search-control
knowledge,called control rules, that reduce the amount
of search it needs to perform by eliminating dead-end
paths and selecting profitable ones. WhatMintonand
others noticed about systems like PRODIGY/EBL
was
that the system could actually get slower after having
learnedcontrol rules, rather than faster. At eachstep in
the search space, a control-rule problemsolver has to
matchall of its controlrules against the currentstate to
determineif they shouldfire. As that library of control
rules growsin size, the cost of matchingthe control rules
increases to the point that it outweighsor "swamps"
the
savings
in
search
they
provide.
1. Introduction
This side effect of learning wascalled the "utility
All intelligent systemsthat learn can suffer from the
problem": learning designed to improve the system’s
utility problem, whichoccurs whenknowledgelearned
performance ended up degrading performance instead
in an attempt to improve a system’s performancede(HOLDERET AL.1990). Since Minton’sdiscoveryof this
grades performanceinstead (1VIINTON1990). In this pa"swamping"utility problem in PRODIGY,
researchers
per, weanalyze the utility problemand examineits efhaveidentified manydifferent types of utility problems,
fects in case-based reasoners and control-rule problem each manifestingitself in slightly different ways. Besolvers. Our methodologyfor this analysis couples a
cause sometypes of utility problemsare affected by the
functional analysis of an AI systemwith a performance hardware architecture of the system and others are
analysis of the system’s algorithmic and implementa- largely independentof hardwareconcerns, wecan group
tional components.Sucha computationalmodelallows
the different types of utility problemsinto two rough
the identification of the root causesof the utility prob- classes: architecturalutility problems,whicharise from
lem, whichare combinationsof algorithmic characterisinteractions betweena system’s learning and its hardtics of an AI system(e.g., serial search of memory)
with ware architecture, and search-spaceutility problems,
particular "parameters"that affect its operation (e.g.,
which arise from interactions between learning and
knowledge
base size). Identifyingthe precise algorithmic problemsolvingalgorithms.Afull discussionof the difnature of a root cause allowsus to predict the threshold ferent types of utility problemsis containedin (FRANCIS
conditionsunder whichit will affect a systemseverely & RAM
1994). In this paper, wewill focus on swamping,
enoughto cause a utility problem.
whichis the utility problemmostcommonly
encountered
Our analysis provides a general and theoretical
in learning systems.Wewill reservethe term"the utility
frameworkfor addressing this problemin systemsthat
problem"for the general utility problem,and will refer
havebeenstudied empirically(e.g., control-rule learning to specific versions of the utility problem, such as
(CRL)systems) and in systemsfor whichlittle utility
swamping,by their names.
analysis has been performed(e.g., case-basedreasoning
3. A Methodologyfor Utility Analysis
(CBR)systems). Weuse this frameworkto compare
and CRLsystems, and find that CBRsystems are more Weproposethe use of algorithmic complexitytheory as
resistant to the utility problemthan CRLsystems.
a tool for the analysisof the generalutility problem.Our
methodologyinvolves analyzing different types of AI
2. Analyzing the Utility Problem
systemsand decomposing
their cognitive architectures
Theutility problemwas first detected in PRODIGY/EBL
into lower-level functional units, including problem(MINTON1988). PRODIGY/EBL
is control-rule
solving engines and memory
systems, that can be repreAbstract
Theutility problem
in learningsystemsoccurswhen
knowledge
learned in an attemptto improvea system’sperformance
degradesperformance
instead. We
presenta methodology
for the analysisof the utility
problemwhichuses computational
modelsof problem
solvingsystems
to isolatethe rootcausesof a utility
problem,to detect the thresholdconditionsunder
whichthe problem
will arise, andto designstrategies
to eliminateit. Wepresentmodels
of case-based
reasoningandcontrol-rulelearningsystemsandcompare
themwith respect to the swamping
utility problem.
Ouranalysissuggeststhat CBR
systemsare moreresistantto the utility problem
thanCRL
systems,
l
i Thisresearch
wassupported
bytheUnited
StatesAirForceLaboratoryGraduate
Fellowship
Program
andthe Georgia
Instituteof Technology.Alongerversionof this paper(FRANCIS
&RAM
1994)is ttp-able
from
24.ps.Z
ftp. cc. gatech, edu :/pub/ai/ram/git-cc-
94-
36
2 Etzioni (1992) uses the term meta-level problemsolvers for controlrule learning systems. Wehave avoided this term because of the possible
confusion with metacognition, which includes systems that "knowwhat
they know" (metaknowledge)and systems that reason about their own
reasoningprocesses (metareasoning,or introspection).
sented by formal algorithmic models. Our algorithmic
approachincorporates both functional-level aspects of
the computation,such as the system’scognitive architecture and its knowledgebase, and implementation-level
aspects, such as the performancecharacteristics of the
system’shardwarearchitecture. This multi-level analysis
is crucial for the study of the utility problembecause
manyutility problemsarise due to interactions between
the functional level of the systemand the waythat functional computationis actually implemented.Our methodologycan be usedto identify potential utility problems
as well as to designcopingstrategies to eliminatetheir
effects. In this paper, wefocus on a comparative
analysis
of case-basedreasoningandcontrol-rule systems.
matchtime and knowledgebase size; in a CBRsystem, a
similar interaction exists betweencase retrieval time and
caselibrary size.
Wecan formally define an interaction to be a combination of a set of parameters,a module,and a set of effects. Themodulerepresents the part of the CAthat is
responsible for the relationship betweenthe parameters
andtheir effects. Parameters
represent characteristics of
the system’sknowledge
base, while effects represent the
performancemeasuresthat affected by the interaction.
Thus,an interaction defines a function betweenlearning
(changes in the knowledge base) and performance
(changes in the evaluation metric), mediated by the
characteristics of the algorithmiccomponent
of the interaction
(the
module).
3.1. AI Systems and Learning
Utility problemsarise whena learning modulein the
Formally,wecan describe an AI systemas a triple (CA, system causes parameter changes whichinteract with
KB, HA). The cognitive architecture CAspecifies a
someCAcomponentto produce"side" effects that imsystemin terms of separate functional modulesthat carry pact the performancemeasuresa learning moduleis deout fixed subtasks in the system, while the knowledge signed to improve. This kind of coupling between a
base KBrepresents the internal "data" that the CAuses learning moduleand an interaction is a potential root
to performits computations.Thehardwarearchitecture
causeof a utility problem.For a particular root cause,
HAdefines the operations that a systemcan performat
the calculated improvement
Fc is the savings that the
the implementation
level, as well as their relative costs.
learning moduleis designedto perform,while the actual
Thecost (and hencethe utility) of an operation may
cost Fa is the actual changein performance
taking into
different on different HA’s:for example,retrieval might account
the side effects of the interaction.
take longeron a serial machinethan it wouldon a paralBycomparingthe algorithmicbehaviorof the learning
lel machine.
module,the root cause interaction it is paired with, and
the cost and savings functions that they contribute, we
3.2. Utility and the Utility Problem
Utility can only be defined in terms of "performance" can computethreshoMconditions -- limiting values for
the parameterchangesthat the systemcan tolerate before
measuresthat judge the efficiency of a reasoner, such as
the actual costs exceed the calculated improvement
and
executiontime, numberof states searched, storage space the systemencountersa utility problem.Eliminatingthe
used, or evenquality of solution. Theseevaluationmet- general utility probleminvolves identifying the root
rics measurethe costs that a system incurs during its
causesof particular utility problemsthat can arise in a
reasoning.Givena particular evaluationmetric, the utilsystemanddesigningcopingstrategies that preventtheir
ity of a learned item can be definedas the changein exthresholdconditionsfrombeingsatisfied.
pectation values of a problemsolver’s performanceon
the metric across a problem set (MARKOVITCH~ SCOTT 4. Modeling CRL and CBR Systems
1993). Tocomputethe utility of a changeto the system’s To compare CBRand CRLsystems, we must develop
knowledgebase with respect to somemetric, wewant to
computational models of these systems and compare
computethe costs that the systemwill incur for different learning andproblemsolving in each.
problemsweightedby the probability that the systemwill
The baseline for comparisonof the computational
actually encounter those problems. Thus, utility is a
model approach is the unguided problem solver. Unfunction not only of the learned item but also of the
guided problemsolvers (UgPS)are a class of problem
learning system, the problemset, problemdistribution,
solvers that operate without search control knowledge.
and the evaluation metric. The utility problemoccurs The algorithm of a UgPSis a knowledge-free weak
whena learning system makesa change to its KBwith
the goal of improvingproblemsolving utility on some
Problem
Space
metric by a calculated improvementFc, but whichhas
Initial
State~
~
the side effect of degradingproblemsolving utility for
I
another (possibly identical) metric by someactual
Difficulty
d=
’:~ii
Solution
Path
amountFa that outweighsthe savings(i.e., Fc<Fa).
path
length
3.3. Dissecting the Utility Problem
,:,:,;,
In general, utility problemsare not global, emergent
Goal
State
properties of computationbut can instead be tied to
Branching.
Fact.or
b=
specific interactions betweenthe CA,the KB,and the
average
numcer
orcnolcea
pernode
performancecharacteristics of the HA.In a CRLsystem,
Figure 1. UnguidedSearch
the interaction of interest is the relationship between
37
methodguaranteedto find a solution if one exists, such
as breadth-first search; the only knowledge
it uses is its
operatorlibrary. For a given problemp with a solution
of path length d operating in a search space with
branchingfactor b, a UgPSwill examineba nodes during
search (see Figure1). Thenumberof nodesthat the system expandsis termedthe complexity of a problemand
is denotedCp. Becausethe UgPSsolves problemsin exponentialtime, it can serve as a "baseline"against which
moreefficient learning systemscan be compared.Much
of "intelligence" can be viewedas attemptsto reducethis
combinatorialexplosionthroughthe use of heuristics or
other techniques (NEWELL ~ SIMON1975; RAM&
HUNTER
1992; SCHANK
& ABELSON
1977).
4.1. Control-Rule Problem Solvers
Learning systems improve over the UgPSby finding
waysto reduce or eliminate search. A CRLsystem reducessearch by retrieving and applyingcontrol rules at
eachstate it visits duringproblemsolving, givingit the
ability to select or reject states. Thiscontrolknowledge
is
a completelydifferent kind of knowledgethan operator
knowledge
and must be stored in a separate control rule
library. If a system’scontrol rule library is emptyand
ot .C-.
PROBLEM
Elaborated
Problem~~-n~::~
Problem
Old Case
& /
ProUam
& "-,,.I
~ed Case
(so’ruuon)
~
problem
Bolutl~
~
& Outcome@
~. Costof CaseRetrieval= Rc
/
~
~ I Old rSolution Pstfl
lk,,
Cost of
~1(,~t,.,c..) ~ ~ .=~..on
I
I
/~N
/~,:~ ~::::."2 ::: ~’~
~ I New Solution
Pa~
Cpffi TOtel
of Nodes
~AdaptedCase F.Jmrnlned
Inlq~h ,S~ce
~(SOfUtlon)
Figure3. Searchin the Spaceof ProblemPaths
solution path is achieved(see Figure 3). Oncethe new
solutionis found,it is storedin the case library indexed
for futureretrieval.
5. Analyzing Retrieval Costs
Ouranalysis of these modelsfocuseson retrieval costs in
CRLand CBRsystems---howmanyretrievals are made,
and howmuchdoes each retrieval cost? Retrieval is ofProblem Space
ten cited as the core source of powerfor CBRsystems,
Initial State------~-¢~ ....,._ A PRlication of Control .Rules
yet the retrieval cost is a critical factor in the swamping
Keauces ~pace ~earonea
utility problem.Ananalysis of retrieval costs in CRLand
CBRsystemsbefore and after learning reveals interestSolution
Pa~~,,
.
/~~--~Potential
Search Space:
Cp
ing differencesin howeachdeals with retrieval.
paceActually Searched:Cp’
For our analysis, wewill assumethat both the CRL
Total
Sav
ngs
of
and
CBRsystemsoperate on the sameproblemset, and
_
. ~ ..............
~oal State--’--GuioedSearch: Cp- Cp’
that their problemspaces are defined by the sameoperator library. Wefurther assumethat the costs of adaptaFigure 2. SearchGuidedby Control Rules
tion operatorsare roughlyequivalentto those of regular
operators; this assumptionmayor maynot be true for
control rules are not available, the problemsolverresorts
particular systemsbut is reasonablefor this analysis.
to blind search. Oncea solution path has been found,
Wedefine a basic operation of retrieval, R, which
correct decisionscan be cachedin the library as control
rules that will guidethe systemin the future (see Figure chooses an item in a knowledgelibrary based on some
2). This model, while simplified, approximates many matchingfunction. In general, for a given hardwarearchitecture HA,the cost of retrieval for a knowledge
liexisting systems,including Soarand Prodigy.
braryi, denotedRi, is a functionof boththe library i and
4.2. Case-Based Reasoners
the item to be retrieved, r: Ri =fir, i). For a serial hardware architecture, HAs,the most importantvariable in
Case-basedreasoning is primarily experience-based;
whena CBRsystem encounters a newproblem, it checks this cost function is the numberof items in the knowledge library, Ki. Wewill approximatethis serial cost
its case library of past problemsolving episodes, or
cases, lookingfor a similar casethat it can adaptto meet function with Ri = cKi, wherec is a constant multiplier
that approximatesthe (nearly) linear cost function for
the needs of the new problem. Our model of CBRhas
matchingon serial systemslike 3Has.
twoprimaryknowledge
libraries: the case library itself,
Theparticular interaction wewill examineis the reindexed so that the most appropriate case can be retrieved in newproblem-solvingsituations, and an adap- trieval time interaction: the relationship betweenthe
retrieval operation Ri, knowledgelibrary size Ki, and
tation library that stores adaptationoperatorsthat are
runningtime t. Becausethe learning operations in CBR
used to transform the cases once they are retrieved.
and
CRLsystemshavethe effect of increasing the size of
Whena CBRsystemis presented with a problem, it retrieves an appropriate past case based on the problem’s knowledgelibraries in the system, their learning modules coupledwith the retrieval time interaction formpofeatures, its goals, andthe indicesit has in its case library. Then, the case is adaptedby performingsearch in
3 Anindexed or parallel memorysystemmight improveon this retrieval
the space of problempaths: the adaptation operators
transformentire paths into newpaths until a satisfactory function (or, then again, might not, depending on the domain; see
~
DOORElqI3OS
1993 and TAMBE
ET AL, 1991),
38
tential root causes of the utility problem. Now,let’s examine the dynamics of learning and retrieval in CBR
and CRLsystems and attempt to establish the threshoM
conditions for the utility problemin each.
5.1. Retrieval in Control-Rule Problem Solvers
In its initial state, without control rules, a CRLsystemis
equivalent to the UgPS.It searches Cp states, retrieving a
set of operators at each step with a cost of Ro. Nocontrol
rules exist, so the cost of control-rule retrieval Rc = O.
Thus, the total cost in retrievals of the initial systemis
CpRo.After the systemhas learned a set of control rules,
it has the capacity to guide its search. The number of
states searched is reduced to Cp’, where Cp’ < Cp. However, in addition to retrieving a set of operators, it also
needs to retrieve control rules at each step; thus, the cost
for solving a problemrises to Cp’(Ro+Rc).
The expected savings that guided problem solving
brings are the costs of the states that the problemsolver
avoids: (Cp - Cp)Ro. The added costs are the costs of
matching the control rules at each step: Cp’(Rc’ - Rc)
cp’ARc. (Note: because Rc = O, ARc = Re). The utility
problem will arise whenthe added costs exceed the expected savings. Thus, the threshold condition is (Cp
Cp)Ro < Cp’ARc; in other words, the swampingutility
problem arises when the added cost of retrieval outweighs the benefits of individual rules. But will this
threshold condition ever be met?
In the limit, the maximumsearch reduction is to a
single path (Cp’ = d), and operator retrieval costs axe
constant (Ro’ = Ro) since the library of operators the
system uses does not change in size. The maximumexpected savings possible for any problem are thus (Cp
d)Ro. In contrast, the cost of retrieving control rules (Re)
increases without boundas the control base Kc increases
in size; in the limit, the added costs associated with a
nflebase are dRc’ = dc(Kc) and thus can outweigh the
maximumpossible savings. Therefore, the threshold
conditions can be met and the CRLsystem will encounter the utility problem.
These results indicate that swampingis a function of
the potential speedupof learned items, the cost function
of retrieval (which is itself dependent on retrieval
strategies and machine architecture), and the numberof
items a system needs to learn. If the system converges on
a bounded set of learned items and the hardware slowdownnever approaches the utility of those items, the
4 If the learned items are
system will never be swamped.
of low utility, or if the learner never converges on a
bounded set, as might be the case for an open-world or
4 For example, on a parallel machinewith a logarithmic cost ~nction
Ri = c(log Ki), the threshold condition (Cp-Cp~Ro< Cp’c(log Kc) may
never be met in a closed-world domainin which a small set of knowledge
items learned by rote are adequatefor performance.If the learning system
successfully converges on a small enoughset, the logarithmic slowdown
will be negligible comparedto the potential savings. This condition can
arise on serial architectures as well, but becausethe cost function is linear
in the size of the knowledge
base the constraints on the size of the learned
set are muchmoresevere.
39
multidomain system, then the swamping problem can
eliminate the benefits of the learned rules.
5.2. Retrieval in Case-Based Reasoners
To analyze utility effects in CBRsystems, we need to
measure the performance of a CBRsystem as it learns.
To provide a basis for this measurement,we assume that
a CBRsystem that does not have an appropriate case in
memorycan resort to somemethod(e.g., adaptation of
"null case" or "from-scratch" reasoning) that is costequivalent to the UgPS. Many existing CBRsystems
have such a last-resort
method (e.g., KOTON
1980,
KOLODNER
& SIMPSON1988).
A CBRsystem that resorts to null-case adaptation beginning with no experiences must still incur the cost of
retrieving the null case (Re) and then search the space of
problempaths until the case has been adapted into a satisfactory solution. Under our earlier assumptions, the
total numberof paths the system examinesis Cp, and one
adaptation retrieval (Ra) occurs per step. Thus, the total
cost of case adaptation before learning is Rc + CpRa.
After the system has learned a library of cases, it will
still need to retrieve a case but each case will require
muchless adaptation, reducing the numberof paths examined to Cp’ where Cp’ << Cp. The cost of retrieving
cases will increase to Re’ where Re’ >Re. Thus, the total
costs are Re’ + ¯CpPRa
TOevaluate these results we must again examine the
benefits and costs of case retrieval. The expected savings
are the costs of the states that the problemsolver avoids:
(Cp - Cp) Ra, while the added costs are the increased
costs of retrieval of cases Re’ - Re = ARc. In the limit,
the cost of retrieval increases without boundas the case
library increases in size: Re’ = c(Kc). However,as we
approach the limit the case library contains manyappropriate cases and little adaptation needs to be dono-perhaps only one or two steps. In general, wheneverthe
threshold condition (Cp - Cp) Ra < AReis met, the cost
of retrieval outweighs the benefits of case adaptation;
under these conditions, CBRsystems will be swamped.
5.3. Advantages of Case-Based Reasoning
Whilethis analysis reveals that both control-rule learners
and CBRsystems can suffer from the swampingutility
problem, it also reveals that CBRsystems have important advantages over CRLsystems.
The primary advantage CBRsystems have over control-rule problemsolvers is that cases are retrieved only
once during the lifetime of problem solving. For a control rule problem solver to avoid swamping,the increase
in cost of retrieval of a control rule mustbe less than the
fraction of total states that the system avoids in guided
problem solving times the cost of an operator:ARc <
Ro(Cp- CpS/Cp’. For a CBRsystem, on the other hand,
the increase in cost of a case retrieval must be less than
the cost of the numberof adaptation steps avoided: ARc
< Ra(CP- C )P ¯The missing C ’p term in the denominator
of the CBRequation arises because the increased cost of
retrieval of control rules are incurred at each step in the
search space, whereasthe increased cost of case retrieval
is incurred only once during problem solving for a CBR
system. In other words, CBRsystems amortize the cost
of case retrieval across all adaptations, making them
muchmoreresistant to increases in’ retrieval costs than
CRLsystems.
This amortization also makes CBRmore amenable to
solutions to the swampingproblem, such as deletion
policies or indexing schemes. In order to be effective,
any coping strategy needs to reduce retrieval time to the
point that the threshold conditions are never satisfied.
For CRLsystems, this upper limit is Re’ < Ro(Cp
Cp)/Cp’; for CBRsystems, this upper limit is Rc’ <
Ra(Cp - Cp%a muchhigher (and hence muchless stringent) limit on the maximum
time retrieval can take for
system to be guaranteed to avoid swamping.
5.4. Future Comparative Analysis
The above analysis does not close the bookon the utility
problem in CBRand CRLsystems. There are a number
of other differences that can contribute to the utility
problem, including differences in operator costs, degree
of search reduction, and knowledgebase size.
Normalproblem solving operators and adaptation operators do not necessarily have comparable costs. A
traditional problem solving operator makesa change to a
single state, while an adaptation operator can potentially
make several changes to a case. Thus, the same reduction in Cp’ in CBRand CRLsystems could provide different degrees of savings because the individual steps
being saved maynot cost the same.
Another difference between CBRand CRLsystems is
that they are not guaranteed to produce the same reduction in C/after being exposedto the sameset of training
examples. While both types of systems are senstitive to
the structure of the domain, search reduction in CRL
systems depends on the learning and matching policy for
control rules, but in CBRsystems depends on the indexing schemeand similarity metric used for case retrieval.
Therefore, the same training set might produce different
search reductions for CRLand CBRsystems.
Furthermore, depending on the learning biases chosen,
different systems could extract varying numbersof cases
or rules from the same set of examples. Therefore, even
if the search reduction is the same, there is no guarantee
that the size of the systems’ knowledgelibraries will be
comparable, and hence no guarantee that their performance on the utility problemwill be identical.
Providing a principled account of the effect contributing factors like these have on the utility problemis the
primary goal of our research. While we believe we have
identified one factor -- amortization -- which contributes to the performance differences of CBRand CRL
systems, it is only one contributing factor amongmany
that determine the systems’ respective performances.
Accountingfor other contributing factors, such as operator costs, degree of search reduction, and knowledgebase
size, will allow a principled comparison of CBRand
CRLsystems performance on the utility problem.
For example, there have been few attempts to systematically evaluate the cost-utility tradeoffs in CBR
4O
systems with very large case libraries. However,reports
of the speedup provided by cases (e.g., KOTON
1989) and
control rules (e.g., TAMBE
ETAL1990) suggest that cases
can provide large improvements, up to an order of
magnitude greater than the speedups provided by control
rules. It is possible that the utility of eases maybe high
enough to allow CBRto avoid swamping, but it is not
clear whetherthis will alwaysbe the case.
6. The Bottom Line
CBR’spatterns of retrieval makeit resistant to the utility
problem. Becausethe cost of case retrieval is amortized
over many adaptation steps, ideal CBRsystems suffer
less severely from the same overhead and are more amenable to coping strategies
than CRL systems. This
analysis suggests several future lines of comparativeresearch on CBRand CRL,such as operator costs, degree
of search reduction and size of knowledgebases.
References
Doorenbos,R.B. (1993) Matching100,000 LearnedRules.
Proceedingsof the EleventhNationalConferenceon Artificial Intelligence, p290-296.AAAI
Press/MITPress.
Etzioni, O. (1992) AnAsymptoticAnalysis of SpeedupLearning. In MachineLearning:Proceedingsof the 9th International Workshop,1992.
Francis, A. and Ram,A. (1994). ComputationalModelsof the
Utility Problemand their Applicationto a Utility Analysisof
Case-BasedReasoning. Technical Report GIT-CC-94-24,
Collegeof Computing,
GeorgiaInstitute of Technology.
Holder, L.B.; Porter, B.W.;Mooney,R.J. (1990). Thegeneral
utility problemin machinelearning. In MachineLearning:
Proceedingsof the SeventhInternationalConference,1990.
Kolodner,J.L. &Simpson,R.L. (1988). TheMediator: A case
study of a case-basedreasoner. GeorgiaInstitute of Technology, Schoolof Informationand ComputerScience, Technical
Reportno. GIT-ICS-88/11.
Atlanta, Georgia.
Kolodner, J.L. (1993). Case-basedReasoning. MorganKaufmann,1993.
Koton,P. A. (1989). A Methodfor Improvingthe Efficiency
Model-BasedReasoningSystems. Laboratory for Computer
Science, MIT,Cambridge,MA.HemispherePublishing.
Markovitch,S. and Scott, P.D. (1993) InformationFiltering:
Selection Methodsin Learning Systems.MachineLearning,
10: 113-151.
Minton,S. (1988). Quantitativeresults concerningthe utility
of explanation-basedlearning. In Proceedingsof the Seventh
NationalConference
on Artificial lntelligence, 1988.
Minton,S. (1990). Quantitativeresults concerningthe utility
of explanation-based
learning. Artificial Intelligence, 42(23), March1990.
Newell, A. &Simon, H.A. (1975). ComputerScience as Empirical Inquiry: Symbols and Search. Reprinted in
Haugeland,J., (ed.). MindDesign:Philosophy,Psychology,
Arnficial Intelligence, chapter1, pp 35-66.MITPress, 1981.
Ram,A. &Hunter, L. (1992) The Use of Explicit Goals for
Knowledge
to GuideInferenceand Learning.AppliedIntelligence,2(1 ):47-73.
Schank,R. &Abelson, R. (1977). Scripts, Plans, Goalsand
Understanding.LEA,Hillsdale, NJ.
Tambe,M.; NeweU,A.; Rosenbloom,P. S. (1990). The Problemof ExpensiveChunksand its Solutionby Restricting Expressiveness. MachineLearning,5:299-348,1990.