From: AAAI Technical Report FS-97-03. Compilation copyright © 1997, AAAI (www.aaai.org). All rights reserved.
DiagrammaticRepresentationand Reasoning:
SomeDistinctions
B. Chandrasekaran
Departmentof Computer& InformationScience
TheOhioState University
Columbus,OH43210
Email:chandra@cis.ohio-state.edu
1. Introduction
When
I co-edited the book,
"Diagrammatic
Reasoning,"[5] I often pnzzled
overthe fact that rather differentsorts of things
were dubbeddiagrammaticreasoning by
different authors.I then spentsometime
trying to
categorizethe differenttypes of use of diagram.~
andthe different tasks that they served.This
shortarticle is the result of that study.The
distinctionsI makehere applynot simplyto socalled diagrammatic
reasoning,but the larger set
of activities that havebeencalled spatial and
visual reasoning.
2. Representation
Let us fLrStconsiderthe notionof
representationitself. Oneof the striking things
about the current wonkin cognitiveneuruscience
is the degreeto whichit is impessiblefor the
researchersto avoidtalking aboutinformation
beingrepresented,projected,fed backetc.
Information
aboutobjectsis said to be
representedin this area of the brain, aboutshape
in that area, andso on. Thus,antirepresentationalism
is not a practical optionany
longer, informationand representationare an
inevitable part of the neededvocabulary.Onthe
other hand, buyinginto informationbeing
representeddoes not necessarilyimplyhavingto
buy into "informationprocessing"or
"representationprocessing."Amechanicalcoinsorter that operatesby passingthe coinsthrough
levers andholesof differentsizes canbe
analyzedin termsof its usinginformationabout
weightsanddiametersof the coins, but saying
that it performsinformationprocessingdoesnot
agreewith everyone’sintuitions. Moredirectly
relevant to our topic, Paul Churchland
has
describedneural structuresthat are involvedin
the visuo-motor
coordinationof a frog’s catching
of its prey, a fly. Heshowsthat, in a suitably
transformedspace, the solution to the conlml
problemcan be expressedas a linear relationship
betweenthe (trart~formedvaluesOf) the location
of the prey anda motorcontrol parameter.In the
frog’s neural structure, visual informationabout
the prey is representedin onemurallayer, and
the motorcontrol informationin anotherlayer
directly belowit. Thetwolayers are so
connectedthat the prey location information
directly sets the motorcontrol parameter.Again~
information
of varioustypes is clearly
represented,but whetherthe best wayto describe
whathappensto it as "processing"is unclear.
At least amongAI people,talk of
informationrepresentationsoonleads to talk of
representation
processingwhichsoon
leads
to
talk ofreasoning.Conversely,
this also sets upa
counter-rhetoricof"no representation"[1]. With
referenceto ourtopicof interest, not all spatial
representationin the brain
istobe takenas grist
for somespatial reasoningmill if reasoningto
be takenas being performed
explicitly. Onthe
other hand knowledgeand inference are
appropriateterms if only a Knowledge
Level
account[2] is intended.
3. Visual versus Spatial versus
Diagrammatic
Whilemanypeople makedistinctions between
"visual," "spatial," and"diagrammatic"
representations,there doesnot seemto universal
agreementon the meaningof these terms.
Glasgow[3] distinguishes between"visual" and
"spatial"byusingthe formerto refer to
uninterpretedpixel-like representationsandthe
latter to refer to representations
in whichobjects
andtheir mutualspatial relationsare indicated.
Butmanyother peopleseemto use "visual" to
denoterepresentations,interpretedor not, arising
fromthe visual modality,while"spatial" is used
to denoteknowledge
of space, its occupant~and
their mutualspatial relations, leavingopenwhich
modalityor modalities- visual, auditor,
kinestheticor haptic- that suppliedthe
information.Diagram~and visual
representationsin generalcan also havecolor
andtextureas part of the representational
repertoire, whilespatial representationsdonot
seemto involvethese properties. Motionis
potentiallypart of all three: diagramscanuse
animation,andvisual andspatial representations
can involve sequencesof images. Thereseems
to be muchmoreof a generalagreement,at least
onan intuitive level, onthe notionof
diagrammatic
representation.It is takento be a
formof spatial representation,explicitly
constructedandintendedto be visually
processed,containingelementsthat havea
conventionalsemantics,displayingthe spatial
relations amongthe elements. Diagrammatic
representationsmayalso haveelements,such as
labels or other annotations,intendedto be
interpreted as non-spatial, andelementswith
visual properties,suchas color andtexture, that
are usedas part of the representational
vocabnlary.Iwasaki[16] proposesthat one of
their importantpropertiesis that they represent
abstractions,i.e., theyabstract out someof the
informationandrepresent other information.Of
course,diagramsalso havedetails that are not to
be taken seriously. Theconventionalityin
dementsemanticsprovidesclues to the
consumer
of the representationabout which
aspectsof the diagramare to be takenseriously
andwhichare to be takenas incidentalto the
representation.
Generalizing
this to use of spatial information
in
humanor machinecognition, the three types of
spatial informationprocessingare predicate
extractionandprojection, reasoning,and
simulation.(Thesethree types are in additionto
algorithm~ that use spatial information
to
computesomethingof interest andthat haveno
special standingas modelof anypart of spatial
cognition.)
4.1. PredicateExtractionand
Projection
Givena Sl~_tial representation,a rmmber
of facts corresponding to the representationcan
be asserted;e.g., objectAis in the picture,object
Ais in frontof objectB, Ataller than B, andso
on. Theset of spatial predicatesis open-ended,
and domain-and task-dependent. For example,
in this sessionat the workshop,
the paperby
Epsteinand Gelfanddiscusses a game-playing
programthat constructs various predicatesgiven
a boardrepresentation.Thesepredicates are
intendedto capturepotentiallyrelevantfacts
aboutthe configuration-However,
by definition,
all the spatial predicatesmustbe comp~_a_oble
fromthe spatial informationin the
representation.Manyso-calledspatial or
diagrammatic
reasoningprogram.~
in fact focus
on predicateextraction,i.e., computing
a given
set of predicatesfroma spatial representation.
Thereis a complementary
process, predicate
projection, whichhas not received much
attentionin the literature. Givena set of spatial
predicates pertaining to a situation, this
projectionprocesscreates a spatial or
diagrammatic
represera_otion,or modifiesan
existing one, so as to matchthe assertions. What
spatial predicatesare usefulfor whatta~qk~q
and
howto computethemefficiently are reviewedin
somedetail in Chapter6 of [6].
4. Reasoning,PredicateExtraction
andProjection, andPrediction by
Simulation
Theword"reasoning"is often usedas a
generic equivalentof compm_ation
- "geometric
reasoning"or "spatial reasoning"program.q
often
turn out to be programsthat perform
computations
involvinggeometricor spatial
information.Thereis, however,a narroweruse
of the term"reasoning,"In this senseof
reasoning,the agentstarts with somegiven
assertions, andmakesadditionalassertions as
inferences,usingrules of inference.For
example, whenBarwiseand Etchemendy
[4]
speakof diagrammatic
reasoning,they intend for
the diagramsto be assertions of somefacts or
hypotheses,andthe goalis to arrive at inferences
that satisfy the standardnotionsof valid
inference.
Goingbackto the motivationfor this
paper,a perusalof the literature led meto the
conclusion
that that at least three distinct typesof
use of diagrammatic
informationare generally
includedin the term"diagrammatic
reasoning."
4.2. Reasoning
Asmentioned,
sp,9_riO1reasoningis
makingtrustworthyinferenceswith spatial or
diagrammaticrepresentations. Asimple model
of spatial reasoningis onein whichthe agent
starts witha spatial representation,
extracts some
predicatesfromit, and, usingother axiomsand
rules of inference,proceedsto makeinferences.
Statedthis way,there is nothingto distinguish
the actual processof spatial reasoningfromany
other formof reasoning,exceptfor the facts that
the problem
gets its start fromthe spa!i_al
representationandgenerationof initial predicates
requiresuse of a predicateextractionmodule.
2
This is what has often p.~led manypeople who
havenot beenenthusedabout special claimsfor
diagrammatic
reasoning. However,in fact, there
are a munber
of propertiesof spatial reasoning
that justify special treatment.Spatial
representationsare not of generalquantified
propositions,but of concreteinstances.
Nevertheless,weseem,in somecases, to be able
to makecorrect inferencesabout general
situations. Studyingthe conditionsunderwhich
suchgeneralinferencesare supportedis a major
concernin the logic of spatial reasoning.
Second,spatial reasoningoften involvesspecial
spatially sensitive axiomsandrules of inference
(suchas the transitivityof right-of, or inside-of
predicates).Studyof spatial reasoningis an
opportunityto focus on suchmodality-specific
rules of inference.Thirdly,often, spa__tial
reasoningdoesnot simplyinvolvejust extracting
predicates and then proceedingwith reasoningas
usual, but insteadinvolvesprojectiononto
spatial representationsandextractionof new
predicates.Thatis, the spatial representationis
inextricably involvedin the wholeprocessof
reasoning. A useful modelis one wherethe
agent’scognitivestate is representedas having
two components,
a spatial one and a
corresponding
predicaterepresentation,with the
two componentsmatchingone another. There
are nowtwomodulesthat operateon the side, a
predicateextractionmodule,as before, anda
predicateprojectionmodulethat constructsor
modifiesthe spatial representationto matchthe
predicate component.The famous"Behold"
proof of the PythagoreanTheorem
(see Chapter
1 of [7] for a versionof this proof)is just a series
of diagrams,with no linguistically represented
assertions alongthe way.Butevenin these
instancesthe reader is assumed
to be making
implicit inferencesin her headas she examines
the diagrams.
A major concernin diagrammatic
reasoningis the conditionsunderwhichsuch
inferencesusingdiagramsare reliable.
Conversely,the creation and use of such
representationto solve reasoningproblemsthat
are not intrinsically spatial (suchas reasoning
aboutsets) are also mattersof intenseinterest.
BarwiseandEtchemendy
[4] note that it has
generallybeenassumed
by logicians that
diagramsare a mereheuristic aid, and"real
proofs"haveto be purelysymbolicin character.
Amajorfocusof their workis showingthat valid
diagrammaticproof systemsmayindeed be
constructed.
Becausespatial representationsare
specific instancesrather thanquantified
propositions,they often providea convenient
representationfor modelsof propositions.The
researchof Johason-Laird
(presentedat this
workshop)
showsthat this propertyof sp-atial
representationis useful in l~lmannon-specialist
syllogistic reasoning.Givena syllogistic
reasoningproblem,humanreasoners modelthe
variouspropositions,representthemspatially,
andtry to see if counterexamples
can be
constructedto the propositionto be proved.
Glasgow’s
work,presentedin this session,
discussesthe use of spatial representationsas
models.
4.3. Simulation
Manysp-ati_al "reasoning"program~
or
cognitivemodelsdo not reasonin the senseof
moving
fromcognitivestate to cognitivestate by
processingtruth-valuesof assertions. Instead,
they makeuse of simulationto generatethe next
state. Theproblemsfor whichsuchsimulationis
appropriateare generallypredictionproblems:
givena spatial si_~J-afionandsomeproposed
actions on, or interaction between,the elements
in it, whatwill be the spatial represe~ation
correspondingto the newsituation? The
simulationis supposedto mimicthe spatial
tran~ormations
involvedin the transition from
initial state to final state. Thesimulationmight
additionally mimicanyphysical laws involvedin
effecting the changes.In A.I, the workof Funt
[8], DeCuypers,
at al [9], andGardinandMeltzer
[10] are examples
of this kindof sp-afial
simulation. DeCuypers’programcomputesthe
results of diffusionof gas in a room,andGardin
and Metzler’sprogramsimilarly performs analog
compta-ations
to predict the behaviorsof strings
andother objects in three-spaceundervarious
forces. Typically,the computation
is
implementedby somekind of array model,in
whicheach processor performssome
compraation
andpasseson the results to its
neighborsin the array. Glasgow
[3] also uses a
spatial array to performspatial reasoning.The
simulationapproachesare often characterizedas
analogprocessesfor obviousreasons.
In contrastto AI, wherethere is a
cert-ain freedomin choosingcomputational
approachesto get the job done,there are more
constraints in explaininghuman
spatial
informationprocessingabilities. Thescopeof
suchanalog computationalprocessesin hnman.q
is among
the mosthotly contestedof issues. A
gooddeal of the debate on the nature of imagery
- propositional versus imagistic representations
- can be traced to this issue. The mentalrotation
experiments of Shepard, Cooperand colleagues
[11] have often been interpreted as implyingthe
existence of internal "analog" processes in
hmnansfor performingrotation and translationKosslyn[12] proposes the existence of some
analog processes in humanspatial reasoning.
The alternative to the hypothesis of such analog
spatial information-processing
is that rotation,
translation and other such spatial reasoning
processes are actually solved by reasoning
involving spatial predicates and predicate
projections.
yet haveacconnt.qof all formsof valid spatial
reasoning. Howand whendiagrnm.q are effective
in aiding reasoning in general remains relatively
poorly understood. Goel [15] argues that,
contrary to intuitions, diagram.~can not only be
vague, but designers and other problemsolvers
actually exploit this vaguenessfor effective
problem solving. The degree to which human
spatial cognition employsanalog processes for
performingprediction remain.~ controversial, but
in AI special purpose architectures that perform
analog simulation for prediction remain
promising. It seemsto methat the emer~i’ng
trend towards reformulating manyAI problems
in the context of robotic behavior, wherethe
robots have sensory, action and reasoning
abilities, is very importantfor progressin sp_Rtial
reasoning. Broadeningthe notion of internal
imagesfrom the visual modality to an integrated
multi-modalrepresentation that includes
conceptual information as well and using this
internal multi-modal"image"as the basic
encoding of experience promises to open up new
ways of thinking about learning.
5. Control of Reasoning
Earlier whenI discussed reasoning I
focused just on spatial predicates. However,a
morerealistic modelis one wherethe state of the
agent is multi-modal,i.e., informationin several
perceptual modesand also in the conceptual
modeis present at each state. For example,if the
agent were thinking of an apple, her cognitive
state wouldhave a visual representation of the
shape, texture and color of the apple. In
addition, she might have knowledgeabout apples
relevant to the context, say, their prices and
where to buy thent She might also have an
anticipation of its taste. Representational
changesin one modetypically give rise to
corresponding changes in other modalities,
giving the multi-modal representation some
degree of inter-modal coherence. Problemsolving is driven by the most relevant
information in any domain. The reasoning might
be driven by spatial predicates at one moment
and by conceptual predicates at the next. Each
moveforward results in a new state and in
updatingof representations in all modalities.
Koedinger and Anderson’s geometry program
[13] uses a representation in whichboth
conceptual and diagrammaticinformation is
present at each state. Narayananand
Chandrasekaran[14] investigate the control
issues involved in the use of such multi-medal
representations. Epstein and Gelfand [17]
discuss a game-playing programwhich involves
the use of such multi-modal representationtheir program’sgamestate has both spatial and
propositional content.
7. Acknowledgments
Parts of this paper were included in a
session report at the NSFWorkshopon Visual
Cognition and Decision-Makingin the Spatial
Domain,May15-17, 1997. I thank the
organizers for the invitation. I also thank
Michael Anderson and Had Narayanan for their
comments
on an earlier draft of this paper,
thoughI haven’t fully incorporated their
suggestions in this version. I thank John and
SusanJosephsonfor discussions on this topic.
Researchthat forms the basis for this
paper was supported by DARPA,Order mlmber:
D594, monitored by Office of Naval Research,
Grant #: N00014-96-1-0701.
References
1.
Brooks, R.A., Intelligence without
representation,. ArtLOcialIntelligence,
1986.47(1-3): p. 139-159.
Newell, A., The KnowledgeLevel, in
.
AIMagazine. 1981. p. 1-19.
3.
Glasgow,J.I. and D. Papadias,
Computational Imagery. Cognitive
Science,1992.3:p. 355-394.
Barwise,J. andJ. Etchemendy,
Visual
.
Informationand Valid Reasoning,in
Visualization in Mathematics,
W.
ZimmermanEditor. 1990,
6. Summary
The task of formalizing the logic of
spatial reasoning has just begun, since we do not
4
.
6.
.
.
.
10.
11.
12.
Mathematical Association of America:
Washington D. C.
Glasgow,J., N.H. Narayanan,and B.
Chandrasekaran, eds. Diagrammatic
Reasoning: Cognitive and
ComputationalPerspectives, 1995, The
MITPress: Cambridge, MA.780 +
xxvii.
Davis, E., Representations of
CommonsenseKnowledge. 1990, San
Mateo, CA: Morgan Kaufmann
Publishers, Inc.
Allwein, G. and J. Barwise, eds. Log/cal
Reasoning wtth Diagrams.. 1996,
Oxford University Press: NewYork,
NY.
Funt, B.V., Problemsolving with
diagrammaticrepresentations. Artificial
Intelligence, 1980. 13: p. 201-230.
De~r, J., D. Keymeulen, and L.
Steels, A hybrid architecture for
modelingliquid behavior, in [5], p. 731752.
Gardin, F. and B. Metzer, Analogical
Representations of Naive Physics.
Artificial Intelligence, 1989.38:p. 139159.
Shepard, R.N. and L.A. Cooper, Mental
Images and Their Transformations.
1982, Cambridge, MA:MITPress.
13.
14.
15.
16.
17.
5
Kosslyn, S.M., Images andMind. 1980,
Cambridge, MA:Harvard University
Press.
Koedinger, K.R. and J.R. Anderson,
Abstract Planning and Perceptual
Cbunk~:Elements of Expertise in
Geometry,Cognitive Science, 1990. 14:
p. 511-550.
Narayanan~N.H. and B.
Chandmsokaran~
Reasoning vis!lally
about spatial interactions. Proceedings
of Twelfth International Joint
Conferenceon Artificial Intelligence.
1991. Sydney, Australia. San
Francisco: Morgan Kaufmznrt
Goel, V., Sketches of Thought. 1995,
Cambridge, MA:The MIT Press/A
Bradford Book.
Iwasaki, Y. 1997. Session Report, NSF
Workshopon Visual Cognition and
Decision-M~king
in the Sl:m_l_ial
Dornain~ May15-17, 1997. This
material will be part of the Workshop
Reportto appear, edited by S. Epstein,
J. Gelfand, and M. Marefat.
Epsteni, S. L, J. Gelfand,and J.
Lesniak, Pattern-based learning ad
spatially-oriented concept formation in
a multi-agent, decision-m~kingexpert.
CognitiveScience, to appear.