From: AAAI Technical Report SS-94-01. Compilation copyright © 1994, AAAI (www.aaai.org). All rights reserved.
Representing ProblemSolving for Tutoring and Critiquing Medical Domains
Kathy A. Johnson, Todd R. Johnson, and Jack W. Smith, Jr.
Laboratory for Knowledge-Based Medical Systems
Division of Medical Informatics
The Ohio State University
571 Health Sciences Library
376 W. 10th Ave.
Columbus, Ohio 43210
Email: kj@med.ohio-state.edu,tj@med.ohio-state.edu,smith.30@magnus.ohio-state.edu
1 Introduction
Medicalproblemsolving often involves large amountsof
d~t~, This posesproblemsfor expertssolvingthe problemas
well as for studentslearningto do so. Thereare often many
different waysto solve a single problem. Wehave been
studyingthe constructionof tutoring systemsfor problems
involvingthe interpretation of data. Asystemwasbuilt to
detect errors in solving the problemof red blood cell
antibody identification. The system has at its core a
monitoringshell and a representation of problemsolving.
Theshell providesa generalmethodfor monitoringthat can
be used in any domain for which an appropriate
representation can be built. Theshell and representation
could also be used as part of a critiquing systemto aid
medicalproblemsolving.
2 Representing Tutoring Knowledge
Manyrepresentationsfor tutoring havebeenexploredas
well as techniquesfor tutoring variousdomains.Giventhat
a student has had somebasic training in a problemsolving
area, tutoring can be viewedas a process during whicha
systemmust1) monitora student for mistakes,2) determine
the causeof the mistake,and3) give the studentappropriate
informationto correct his mistake.Monitoringfor mistakes
maybe as simple as checking the final answer of an
arithmetic problem or as complicated as detecting an
incorrect sequenceof steps during medicaldiagnosis. An
exampleof a medicaltutoring systemtaking this viewof
tutoring is Clancey’s GUIDON
system which uses the
knowledgebase from MYCIN
to tutor students in the
methodfor doingmedicaldiagnosis (Clancey,1987). Other
medicalsystemshave approachedtutoring in a different
mannerbest characterized by the term critiquing. Rather
than correcting a student’s problem-solvingmethod,these
systemspresentrelevantinformationandalternatives to the
student. Miller’s VQ-Attending
is an exampleof this type of
system(Miller, 1986).
Foreither tutoring strategy, it is importantto understand
the problemsolving taking place. This is obviousfor a
systemthat is trying to teach problem-solvingstrategy.
* Thisresearchis supportedby NationalHeartLungandBlood
Institute grant HL-38776,
andNationalLibraryof Medicine
grant
LM-04298.
74
However,a critiquing system must also knowsomething
about the problemsolving in order to present relevant
information.
In orderto filter all the information
availableto
aid problemsolving, the systemneeds knowwhatgoals the
informationwill be serving.
Wehave been studying the domainof red blood cell
antibody identification whichis a multi-step problemsolving process similar to medicaldiagnosis. Data from
manysources mustbe incorporated to determinethe best
answeras to the patient’s antibodies.Thereis a great deal of
variability in correct problemsolvingbehavior.Thestudent
should be allowedto pursue any correct problem-solving
path. In order to build such a system, we needed a
representation of problem solving that is capable of
encodingall possible correct problem-solving
paths.
Oneof the mostdifficult aspects of describingflexible
problem solving is finding a wayto easily represent
alternative methodsto solve a problem. The standard
representation for problem solving is a procedural
descriptionin whicheachoperator is listed in order along
with conditionals and loops. Such descriptions are
notoriously inflexible and brittle (Newell, 1990). The
ordering of the operatorsmustbe completelyspecified and
the justification for choosingoneoperatorbeforeanotheris
lost in the encoding. Analternative representation for
problem-solvingmethodsis embodiedin the problemspace
computation model (PSCM)of NeweU
(Newell, 1990).
the PSCM,
all problemsolvingis viewedas searchfor a goal
state in a problemspace. Theproblemspace contains a set
of operatorswhichwhenapplied in the appropriatesequence
will achieve the goal. Knowledge
about whenparticular
operators are applicable to a state can be specified
independentof knowledgeabout whichoperator to prefer.
For example,conditions maybe appropriate for moving
BlockXor BlockY(e.g. both are clear). Other kinds
knowledge
mayindicate that it makesno differencewhichis
moved
first, or perhapsthat it is better to moveBlockXfirst.
Operatorsmaybe encodeddirectly or they maythemselves
be set up as a goal in a sub-problem
spacewith their ownset
of operators andcontrol knowledge.For example,movinga
block maybe a primitive operator tied to a motorsystem
whileclearing a block mightrequire a subgoalto determine
the sequenceof operatorsneededto removethe block on top
will be achieved, and the behavior by whichthe function is
achieved. The behavior is a sequence of state transitions
connectedby subfunctions. In our case, a function is a goal
to be achieved,the result is the state of problemsolving after
achievingthe goal, there are still preconditions, and wehave
a method(or methods) for achieving the goal instead of
specific device behavior. The difference between a device
behavior and a problem-solvingmethodis ~n the variability
of the steps. A device usually behaves in a deterministic
mannersince a device is a physical system that works in a
linear fashion (seldom do devices randomly execute
functions unless they are designed to do so). The methodfor
problemsolving should allow for variability in sequenceand
is represented as a set of subgoals with whatever control
knowledgeis necessary (or desired).
Goal: Identify-Allo-Antibodies
It: not auto-antibodies,
desiredallo-antibodiesidentified
To-Make:Complete,Consistent, Parsimonious
explanationfor data
By: Methodl
Method:Methodl
Goals: Make-abstract-hypotheses,
Match-hypothesis-toantigram,Rule-out, Explaindatum
Control:Rule-outis better than Explaindatum
Otheroperatorsare indifferent
Goal: Make-abslract-hypotheses
If: datumneedsto be explained
To-Make:antibody a explains datum
By: Group-by-rows,
Group-by-columns,
Group-by-likereactions
Control:Group-by-rows,Group-by-columns,
and Groupby-like-reactions
are indifferent.
3 Monitoring Students Using a Functional
Representation of Problem Solving
Goal: Match-hypothesis-to-antigram
If: antibody<a>doesnot havea specificity
To-Make:
antibody<a>has a specificity
By: Method3
The functional representation of problemsolving for red
blood cell antibody identification was extracted from a task
analysis that was used to develop an expert system(Johnson,
et. al. 1991). Thetask analysis wasfacilitated by the use of
problemspaces describing generic tasks (Johnson, 1991).
portion of the functional representation is shownin Figure 1.
A monitoring system (Johnson, 1993) using this functional
representation has been built in Soar. It is capable of
mappinga simulated student’s intermediate results into the
goal that he waspursuingand determiningif there is an error
in the student’s reasoning1. Figure 2 showsthe interactions
of the componentsof the monitoring shell.
Goal: Rule-Out
If: desiredlimit-possible-explanations
To-Make:
antibody<b>ruled-out (i.e. not consideredas
part of the explanation)
By: Method4
Goal: Explain datum
If: datumneedsto be explained
To-Make:antibody a explains datum
By: Method5
3.1 Monitoring Goals
The following are the goals that comprise the monitoring
task:
¯ Get-student-result: Get the student’s intermediate conclusion.
¯
Generate-explanations-for-result: Determinethe goals
that the student could have been pursuing that would
generate such a result
¯ Determine-possible-goals: Determinethe goals that are
possible from the previous goal that generate visible
results.
¯ Evaluate-validity-of-goal: For a particular goal generated by generate-explanations-for-result,
determine
whether the goal was valid by checking the preconditions and search-control.
Figure 1: Partial modelof goals and methodsfor red blood
cell antibodyidentification
of it. A description of problemsolving in these terms can
provide a more dynamicor flexible description than does an
algorithm because operator proposal knowledgeand searchcontrol knowledgecan be sensitive to the current state of
problem solving. In addition, alternative methods for
achieving goals can be represented as different sets of
subgoals and knowledge to choose between alternative
methodsmay or maynot be provided.
The PSCM
allows us to describe flexible problem-solving
methods that we need for monitoring a student’s problem
solving. To makeuse of this knowledge, however, we need
to explicitly represent what each problemspace and operator
does. Wehave chosen to represent that knowledge using
terminology borrowed from the functional representation
developed at Ohio State (Sembugamoorthy and
Chandrasekaran, 1986). Previously, the functional
representation had been used to describe devices, plans, and
computer programs (Allemang, 1990; Chandrasekaran, et
al., 1986; Keuneke,1989; Sticklen, 1987). The traditional
functional representation consists of a function, whatresult
is expected, what conditions mustbe true before the function
1. Thecurrentmonitoris in a versionof Soarwrittenin Lispandis
too slowto be usedwithactual students.Additionally,a user interface needsto be developed
to capturethe student’sresults. Thedata
usedto test the monitorwasrepresentativeof student’sactual results.
75
state and the modelof the student’s state is available for
further processing.
The general methodfor composingthese goals is:
1. The system scans the problem-solvingsituation that is
given to the student to get a preliminary modelof the
situation.
Tutor
Monitor
;Student’sConclusion
error indicadon
student’sstate
Causeof
Shell
2.
Solvina
Model"
3.
Student’s
Goal: O1
Result: RI
.
Correction
.
Correction
Student’s
Imermediam
Conclusions
ExternalWorldi.1.
}1I I I I I I I II II II II II II
II IIIII
I II I
Intermediate
Conclusions
Resultsof
pro.blem.splving
and outside
information
Figure 2: The high level problem-solving breakdownof the
tutor and the monitor
¯
.
Determine-explanation: Of the valid explanations
(goals) generated, determine which fits the student
best.
¯
Evaluate-validity-of-method: Determine whether the
conclusion was correct for the goal being pursued. This
is generally accomplishedby running the appropriate
portion of the expert system and comparingthe result
to the student’s intermediate conclusion.
These are general goals that rely on having an explicit
functional model of problem solving and some way of
checking the correctness of a conclusion. That is, they are
useful in any system that has those properties. They are
incorporated into the expert systems and are used whenthe
system has a goal of monitor-student-for-error rather than
the expert’s usual domain goal such as identify-alloantibodies in the case of red blood cell antibody
identification. Get-student-resuh has domain-specific parts
that dependon the particular information available such as
menuselections on a screen, mouseclicks in active areas,
text input, etc. Likewise, evaluate-validity-of-methodneeds
to have access to a knowledge-basedsystem in the domain.
Whenthe monitoringshell detects an error, it is noted in its
Get-student-resultis set as a goal and it returns the student’s intermediate conclusion.
Generate-explanations-for-resuh is used to determine
the goals that the student could have been pursuing.
Determine-applicable-goals generates a set of goals
that are appropriate in the current situation by finding
goals whosepreconditions match. Search control is not
checkedfor these goals, it is deferred to evaluate-valid.
ity-of-goal.
Evaluate-validity-of-goal is used for each goal in the
set of goals returned from generate-explanations-forresult. If the goal beingtested does not appearin the set
of applicable goals generated by determine-applicablegoals, then the goal is markedinvalid and a reason of
not-applicable is listed. If there is a match,then search
control is checked. If the search control indicates that
the goal is not the best one at this time, then the goal is
markedinvalid and a reason of wrong-timeis listed. If
the search control checks out, then the goal is marked
as valid.
Determine-explanationis used after all the goals have
been checkedfor validity. If no goal is valid, then an
error is determinedand the set of goals and reasons for
them being invalid is madeavailable for further error
diagnosis. If one goal is valid, then it is assumedthat
the student is pursuingthis goal even if there wereother invalid goals. If morethan one goal is valid, then
one is picked at randomand the others are kept in the
student modelas alternative goals.
Evaluate-validity-of-method is then used to check the
actual value of the student’s conclusion to makesure
that it matchesthe correct value. At this point, the expert systemis run to lind the correct value. If it does
not match, then an error is determined. If it matches
then the student modelis updated with the conclusion.
If no errors were indicated, then the process begins again
from step 2.
.
3.2 Classes of error
Thereare several classes of error that the monitoringshell
detects.
1. WrongtimingmThe student may be pursuing a goal
that is applicable, but is not the best one at this time.
That is, the preconditions for the goal were met, but
76
detecting errors in problemsolving. This shell maybe used
at the core of a tutoring systemor a critiquing system.It
providesthe capability to followa problem-solving
path in
a flexible manner.Theproblem-solving
representationthat
is a keypart of the shell providesa goal-orientedindexof
methods and could be expanded to also index data
relevancy. Future workwill explore both the tutoring and
critiquingaspectsof the monitoring
shell.
search-controlindicated that there wassomethingelse
that shouldbe donefirst.
2. Goal not applicable---The student was attempting a
goal whosepreconditionswerenot met.
3. Goalnot part of current method--Thestudent wasattemptinga goal not in the current method.Whether
the
newgoal’spreconditionsweremetor not, this is an error becausethe currentgoal hasn’t beenachievedyet.
4. Bad implementation of goal--The student was attemptinga correct goal, but the result of executingthe
goal wasnot correct.
It is necessaryto detect these types of errors in any
domain. Each error type will have domain-specific
explanations. There are also process-specific errors and
explanations for reasoning strategies such as abductive
assembly.Theseerrors andexplanationswouldbe useful in
any domainemployingthe methodof abductiveassembly.
Acknowledgments
Wethank the AIMgroup for their assistance and
comments
on this paper and the workit reports. Wealso
thank the membersof the Soar communityfor their
intellectual andtechnicalsupport.Thisresearchis supported
by National Heart Lungand BloodInstitute grant HL38776,and National Library of Medicinegrant LM4)4298.
References
Allemang, D. T. (1990) UnderstandingProgramsas
Devices. Ph.D., TheOhioState University.
Chandrasekaran,B. (1986). Generic Tasks in Knowledge-BasedReasoning:High-LevelBuildingBlocksfor Expert SystemDesign.IEEEExpert, 1(3), 23-30.
Chandrasekaran,B., Josephson, J., and Keuneke,A.
(1986). FunctionalRepresentationsas a Basis for Generating Explanations.In Proceedingsof the 1986IEEEInternational Conferenceon Systems, Manand Cybernetics (pp.
726-731).Atlanta, Georgia:IEEEComputerSociety Press.
Clancey,W.J. (1987). Knowledge-Based
Tutoring; The
GuidonProgram.MITPress.
Johnson,K. (1993) Exploiting a FunctionalRepresentation of ProblemSolvingfor ErrorDetectionin Tutoring.
Ph.D., OSU.
Johnson,K., Johnson,T., Smith, J. W., DeJongh,M.,
Fischer, O., Amra,N., and Bayazitoglu, A. (1991). RedSoar--A Systemfor RedBloodCell AntibodyIdentification. In Symposiumon ComputerApplications in Medical
Care. Washington,D.C.
Johnson,T. R. (1991) GenericTasks in the ProblemSpace Paradigm:Building Flexible KnowledgeSystems
While UsingTask-LevelConstraints. Ph.D., OSU.
Keuneke,A. (1989) MachineUnderstandingof Devices: CausalExplanationsof DiagnosticConclusions.Ph.D.,
TheOhioState University.
Laird, J. E., Newell,A. andRosenbloom,
P.S. (1987).
SOAR:
AnArchitecturefor GeneralIntelligence. Artificial
Intelligence, Vol.33(No.1).
Newell,A., Yost., G., Laird, J.E., Rosenbloom,
P.S. and
Altmann, E. (1990 September24-26). Formulating The
ProblemSpaceComputationalModel.In Proceedingsof the
25th AnniversarySymposium,School of ComputerScience,
CarnegieMellonUniversity. Pittsburgh, PA.
Miller, P. L. (1986)ExpertCritiquingSystems:Practice-Based MedicalConsultation by Computer.NewYork:
Springer-Verlag.
Sembugamoorthy,
V., and Chandrasekaran,B. (1986).
Functional Representationof Devicesand Compilationof
Diagnostic ProblemSolving Systems.In J. Kolodner&C.
Reisbeck(Eds.), Experience, Memory
and Reasoning(pp.
47-73). LawrenceErlbaumand Associates.
S ticklen, J. (1987)MDX2:
An IntegratedMedicalDiagnostic System. Ph.D. OSU.
4 CritiquingUsinga Functional
Representationof ProblemSolving
Wehave shownhowa functional representation of
problemsolving can be used to monitora student for errors
(an essential subtaskof tutoring). Thesamemonitormay
used in the context of a critiquing system. A critiquing
systemprovidesa personwith supportfor problemsolving.
It may indicate possible omissions of steps and
inconsistencies in conclusions. It can provide access to
descriptionsof alternative methodsto achievea particular
step. It can also help to organizethe informationpresented
to a person.Thesefunctionsare possibleonlyif the system
is able to determinethe goal of the person solving the
problem.
Themonitoringshell andfunctional representationcould
be used in a critiquing systemas follows. Themonitoring
shell infers the problem
solver’s goal. If the problem
solver
has omitteda step, the monitordetects this andputs up a
warning.Asthe personsolves the problem,the monitoralso
compares
the person’sanswerto its expert systemanswer.If
these are foundto be significantly different, the system
warnsthe person. Since the critiquing systemcontains a
description of the problemsolving organizedaroundgoals
andmethods,it has easy access to a list of methods.These
couldbe madeavailableuponrequest. Finally, the problemsolving methodsin the system makereference to certain
types of data. Enhancement
of the functional representation
wouldprovide a goal-oriented index to the relevant data.
Thiscouldthen be usedto filter the data for displayto the
humanproblemsolver.
5 Conclusion
Medical problem solving is a complexprocess often
involving large amountsof data. Solving such problems
createsdifficulties for expertssolvingthe problem
as wellas
for students learning to do so. Wehavebuilt a shell for
77