Tutorial
Inferential and Expressive Capacities of Graphical
Representations
Survey and Some Generalizations
Diagrams 2004
University of Cambridge
March 23, 2004
Atsushi Shimojima
School of Knowledge Science
Japan Advanced Institute of
Science & Technology
ATR Media Information Science
Labs
Purpose
• To understand three
• In my personal
concepts useful to
terminology:
capture the inferential– Free ride
expressive capacities
– Over-specificity
of many graphical
– Derived meaning
systems.
Why Important?
These concepts are very often
alluded to in the literature
but
Their exact contents seldom defined
Their ranges of application never explicated in full
thus
Different people used different terms to refer
to them, sometimes missing important
connections of their ideas and findings
Plan for the hour
Free ride
Examples
Analysis & definition
Connections
Over-specificity
Derived meaning
Outstanding questions
1. Free Ride
A toy example:
Suppose:
• Jon, Ken, Gil, Bob, and Ron run races
• of the kind with no “ties” in arrival
Compare:
• Different ways of expressing the
information that Jon defeated Bob :
Defeated(Jon,Bob)
Jon Bob
Jon defeated Bob.
Atomic sentence of a first-order
language (FOL) with:
• two-place predicate Defeated
• its arguments Jon and Bob
Defeated(Jon,Bob)
Jon Bob
Jon defeated Bob.
Representation of PD system
(position diagrams) where:
Horizontal relation of names
indicate arrival order of people.
Defeated(Jon,Bob)
Jon Bob
Jon defeated Bob.
Sentence of English describing
the arrival order of two people.
Defeated(Jon,Bob)
Jon Bob
Jon defeated Bob.
PD system (a bit more precisely)
Syntactic rules:
– Two or more of the
names “Jon”, “Ken”,
“Gil”, “Bob”, and “Ron”
appear in a horizontal
row.
– The same name
appears at most once.
Semantic rules:
– If the name X appears
to the left of the name
Y, the bearer of X
defeated the bearer of
Y.
Look Similar
FOL
Defeated(Jon,Bob)
PD
Jon Bob
English
Jon defeated Bob.
But behave quite differently when more
information expressed
Difference 1
Express information:
Jon defeated Bob.
Ken lost to Bob.
FOL
PD
English
Defeated(Jon,Bob) & Lost_to(Ken,Bob)
Jon Bob Ken
Jon defeated Bob and Ken lost to Bob.
Express information:
Jon defeated Bob.
Ken lost to Bob.
PD
Jon Bob Ken
The PD system expresses an
additional piece of information…
Express information:
Jon defeated Bob.
Ken lost to Bob.
FOL
Defeated(Jon,Bob) & Lost_to(Ken,Bob)
English
Jon defeated Bob and Ken lost to Bob.
…while FOL and English don’t.
• In PD, expressing certain sets of
information results in the expression of
additional, consequential information.
= Free rides
Another example: Venn diagrams
Express information:
All As are Bs.
No Bs are Cs.
As
Bs
As
Cs
Bs
As
Cs
Bs
Cs
Expressing certain sets of information results in the
expression of additional, consequential information
Another example: Euler diagrams
Express information:
A⊂B
C∩B=φ
A
C
A
B
B
Expressing certain sets of information results in the
expression of additional, consequential information
Another example: Maps
Express information:
B’s house is in front of F’s house across the river.
Expressing certain sets of information results in the
expression of additional, consequential information
Sloman (1971)
Of course we cannot always do the manipulations in our
heads: we may have to draw a diagram on paper, or rearrange parts of a scale model, in order to see the
effects.…The main point is that the ability to apply such
subroutines to parts of analogical configurations enables
us to generate, and systematically inspect, ranges of
related possibilities, and then…to make valid inferences,
for instance about the consequences of such possibilities.
(p. 220.)
Barwise & Etchemendy (1990)
Diagrams are physical situations….As such, they obey
their own set of constraints…By choosing a
representational scheme appropriately, so that the
constraints on the diagrams have a good match with the
constraints on the described situation, the diagram can
generate a lot of information that the user never need
infer. Rather, the user can simply read off facts from the
diagram as needed.
Larkin & Simon (1987)
We have seen that formally producing perceptual
elements does most of the work of solving the geometry
problem. But we have a mechanism---the eye and the
diagram---that produces exactly these “perceptual”
results with little effort. We believe the right assumption is
that diagrams and the human visual system provide, at
essentially zero cost, all of the inferences we have called
“perceptual.” As shown above, this is a huge benefit. (p.
99.)
Other conceptions
• Non-deductive
representation systems
where the operation of the
construction process
entails the “making” of the
inferences (Lindsay 1988, p.
112)
• Inference by recognition
(Novak 1995)
• Inference by inspection
and transformation (Olivier
2002, p. 72--74)
• Emergence effect (Kulpa
2003, p. 90)
• Emergent properties
(Koedinger 1992, as cited by
Olivier 2001)
• Emergent relations
(Chandrasekaran, Kurup, and
Banerjee 2004)
But
What, more exactly, is the “free-ride” capacity?
What is the general condition---semantic
mechanism---for a system to have that
property?
Basic Assumption
A representation X expresses information about the
represented object Y by having a property that indicates
the corresponding property of Y.
a property
Represented object Y
indicates
Representation X
a property
Example: PDs
a particular
running race
[Jon defeated Bob]
indicates
a particular
position diagram
Jon Bob
[the name “Jon” appears
to the left of the name “Bob”]
Example: Euler diagrams
a particular group
of objects
[A ⊂ B]
indicates
a particular
Euler diagram
A
B
[the circle “A” appears
inside the circle “B”]
Condition for Free Ride: PD system
Jon defeated Bob.
Ken lost to Bob.
indicates
indicates
The name “Jon”
appears to the
left of the name
“Bob”
The name “Ken”
appears to the
right of the name
“Bob”
constraint
Jon defeated Ken.
indicates
constraint
The name “Jon”
appears to the left
of the name
“Ken”.
Condition for Free Ride: Euler Diagram
A⊂B
C∩B=φ
indicates
indicates
A circle “A”
appears inside
a circle “B”.
A circle “C” and
the circle “B” has
no overlap.
constraint
C∩A=φ
indicates
The circle “C” and
the circle “A” has
constraint no overlap.
Condition for Free Ride: General
(Shimojima 1996a, 1996b)
………
indicates
constraint
indicates
………
indicates
constraint
Constraints on representations themselves track
constraints in the represented domain.
Thus:
A system with a free-ride property supports deductive
inference through physical manipulation of representations
on an external display, not in the head.
A (paradigm) case of distributed cognition
Connection: AI systems
Some AI systems utilize the free-ride capacities of
graphical systems by installing some manipulationinspection abilities on diagrams.
• WHISPER for the prediction of the collapsing of
objects (Funt 1980)
• REDRAW I & II for the deflection shape problem
(Tessler, Iwasaki, and Law 1995a, 1995b)
• KAP for the prediction of the movements of camfollower pairs and meshing gears (Olivier, Ormsby,
and Nakata 1996)
• DRS component (Chandrasekaran et al. 2004)
Connection: graphical simulations
Some graphical simulations can be considered free rides
in an extended sense, combining computer-controlled
dynamic constraints with geometrical-topological
constraints on graphics.
• Dynamic behaviors of strings, flexible rods, and rings,
falling in free space, etc. (Gardin and Meltzer 1995)
• Liquid behaviors in and out of containers with complex
shapes (Decuyper, Keymeullen, and Steels 1995)
Connection: studies of sketching in design
Free-ride capacities may be an essential factor of the
utility of pictorial sketches in design process.
Schoen (1982)
Each move is a local experiment which contributes to
the global experiment of reframing the problem. Some
moves are resisted (the shape cannot be made to fit the
contours), while others generate new phenomena. As
Quist reflects on the unexpected consequences and
implications of his moves, he listens to the situation’s
back talk, forming new appreciations which guide his
further moves. (p. 94.)
Lawson (1997)
Thus the drawing represents a sort of hypothesis or
“what if” tool. With a plan, for example, the architect can
say, what if the kitchen were here, the dining-room next
to it and the living-room there? How could I then organize
the entrance and the stairs? (p. 242 , colored emphasis
by me.)
Also:
• Goldschmidt (1994)
– “One reads off the sketch more information than was invested in
its making” (p. 164)
– Such reading-off of unexpected reading-off as an essential step
in “interactive imagery” in design
• Suwa, Gero, and Purcell (2000)
– Relationship between unexpected discoveries in sketches and
invention of new design requirements
Warning: Recognition problem
Free rides only guarantee the expression of consequential
information in the representation, not its recognition by the
user.
• “Cheap rides” (Gurr, Lee, and Stenning 1998, Gurr 1999)
• Expertise in diagram construction to facilitate the
recognition of useful consequences (Novak 1995)
2. Over-Specificity
Difference 2
Express information:
Jon defeated Bob.
Ken defeated Bob.
FOL
PD
English
Defeated(Jon,Bob) & Defeated(Ken,Bob)
Jon Ken Bob
?
Ken Jon Bob
Jon defeated Bob and Ken defeated Bob.
?
Express information:
Jon defeated Bob.
Ken defeated Bob.
LD
Jon Ken Bob
?
The PD system can’t express the
info without additional info…
Ken Jon Bob
?
Express information:
Jon defeated Bob.
Ken defeated Bob.
FOL
Defeated(Jon,Bob) & Defeated(Ken,Bob)
English
Jon defeated Bob and Ken defeated Bob.
…while FOL and English can.
• In PD, certain sets of information cannot
be expressed without expressing
additional, non-warranted information.
=
Over-specificity
Another example: Euler diagrams
Express information:
A⊂ B
C∩B≠φ
How do you place
This circle?
C
A
C
A
B
C
B
Certain sets of information cannot be expressed without
expressing additional, non-warranted information.
C
Another example: Maps
Express information:
K’s house is between A’s house and B’s house.
Where do you place
this icon?
K
Certain sets of information cannot be expressed without
expressing additional, non-warranted information.
Stenning and Oberlander (1995)
Specificity
=
``the demand by a system of representation
that information in some class be specified in
any interpretable representation'' (p. 98)
Lawson (1997)
However, there are some ways in which a picture can
often carry too much information or indicate a degree of
precision which may be inappropriate….It would be
difficult to construct a drawing which did not suggest
other features of the form of the finished product which
might restrict a future designer. (p. 242.)
Aristotle (350 B. C. E.)
[In geometrical proofs,] though we do not for the purpose
of the proof make any use of the fact that the quantity in
the triangle (for example, which we have drawn) is
determinate, we nevertheless draw it determinate in
quantity (As cited by Kulpa 2003, p. 101.)
Other conceptions
• Analog property of
representation systems
as opposed to digital
property (Dretske 1981)
• Smaller degree of
discretion (Norman 2000, p.
110)
• Particularity feature (Kulpa
2003, p. 96, p. 101)
Not in the sense of
Goodman (1982)!
Analysis of over-specificity: Euler Diagram
constraint
A⊂B
C∩B≠φ
C ∩ A =φ
indicates
indicates
indicates
A circle “A” A circle “C” and
appears inside the circle “B”
a circle “B”. has some
overlap.
The circle “C”
and the circle
constraint “A” has some
overlap.
C∩A≠φ
indicates
The circle “C”
and the circle
“A” has no
overlap.
Analysis of over-specificity: Position
Diagram
(3) Jon defeated Ken
(4) Ken defeated Jon
(1) Jon defeated Bob
(2) Ken defeated Bob
constraint
(1)
(2)
(3)
(4)
indicates
indicates
indicates
indicates
(1’)
(2’)
(3’)
(4’)
constraint
(1’) The name “Jon” appears to
the left of the name “Bob”.
(3’) The name “Jon” appears to
the left of the name “Ken”.
(2’) The name “Ken” appears
to the left of the name “Bob”.
(4’) The name “Ken” appears
to the left of the name “Jon”.
Analysis of over-specificity: General
(Shimojima 1996b)
constraint
………
indicates
indicates
………
………
indicates
constraint
indicates
………
Thus:
A system with an over-specificity property prohibits the
exclusive expression of certain (small) sets of information
even when they are in the system’s expressive coverage.
The expressive efficacy of a system depends
on “what it allows you to leave unsaid” as well
as “what it allows you to say” (Levesque 1988,
p.370).
In contrast:
“Knowledge expressed in propositional format can
determine part of the state of the world while
conveniently leaving other parts undetermined.” (Ioerger
1992, as cited by Kulpa 2003, p. 101.)
“[Descriptions] need only express their intended
information; they can leave unsaid and indeterminate
some aspect of what is described; they need not convey
more than is required.” (Norman 2000, p. 110.)
Also:
Expressing such a set of information in the system
produces a representation with semantic contents beyond
that set.
=
Accidental features
Berkeley (1710)
“Having demonstrated that the three angles of an
isosceles rectangular triangle are equal to two right ones,
I cannot therefore conclude this affection agrees to all
other triangles which have neither a right angle nor two
equal sides” (Introduction, paragraph 16).
More recent discussion:
Problem of “unintended exclusion”
(Giaquinto 1993)
Problem of “over-looked divergence”
(Kulpa 2003)
Thus:
An over-specificity property of a representation system
poses a challenge to some (all?) attempts to build a formal
deductive system based on that system.
• Hyperproof (Barwise and Etchemendy 1994)
• System for geometry proof (Luengo 1995, Winterstein
et al. 2000)
3. Derived Meaning
Difference 3
Express information:
Jon defeated Bob.
Gil defeated Jon.
FOL
PD
English
Bob defeated Ken.
Ken defeated Ron.
Defeated(Jon,Bob) & Defeated(Bob,Ken) &
Defeated(Gil,Jon) & Defeated(Ken,Ron)
Gil Jon Bob Ken Ron
Jon defeated Bob and Bob defeated Ken and Gil
defeated Jon and Ken defeated Ron.
Express information:
Jon defeated Bob.
Gil defeated Jon.
PD
Bob defeated Ken.
Ken defeated Ron.
Gil Jon Bob Ken Ron
You can count the number of names to find out the
count of people satisfying a certain condition…
Express information:
Jon defeated Bob.
Gil defeated Jon.
FOL
English
Bob defeated Ken.
Ken defeated Ron.
Defeated(Jon,Bob) & Defeated(Bob,Ken) &
Defeated(Gil,Jon) & Defeated(Ken,Ron)
Jon defeated Bob and Bob defeated Ken and Gil
defeated Jon and Ken defeated Ron.
…while you can’t, in FOL and English.
Express information:
Jon defeated Bob.
Gil defeated Jon.
FOL
English
Bob defeated Ken.
Ken defeated Ron.
Defeated(Jon,Bob) & Defeated(Bob,Ken) &
Defeated(Gil,Jon) & Defeated(Ken,Ron)
Jon defeated Bob and Bob defeated Ken and Gil
defeated Jon and Ken defeated Ron.
The count of names don’t mean the count of people.
• In PD, some additional meaning relation holds
that does not hold in FOL and English.
• Moreover, that relation is derivative in that it is not
written in basic semantic rules.
= Derivative Meaning
Another example: Tables
Gil Jon Bob Ken Ron
Jon
Jon
Bob
Ken
○
○
○
○
○
Bob
Ron
○
Ken
Gil
Gil
○
○
○
○
Ron
Count of circles in a row means
the count of people to the right
Count of circles in a
column means the
count of people to the
left
Additional meaning
relation
Another example: Scatter plots
From Tufte (1983)
I
15
II
15
10
10
5
5
0
0
0
10
20
The shape formed by dots means a
general fact about the distribution:
• Existence of correlation,
• Its strength,
• Existence of an exceptional
instance, etc.
0
10
20
Additional meaning
relation
Kosslyn (1994)
Scatter plots...employ point symbols (such as dots,
small triangles, or squares) as content elements. The
height of each point symbol indicates an amount. These
displays typically include so many points that they form a
cloud; information is conveyed by the shape and the
density of the cloud. (p. 46.)
Another example: Data maps
From Tufte (1983)
The concentration of dots along the
Broad Street band means a
concentration of deaths along Broad
Street.
Additional meaning
relation
Another example: Node-edge graphs
Concentration of lines on one node
means that the corresponding station is
a “hub” of the subway system.
Additional meaning
relation
Another example: Node-edge graphs
K
G
H
Olivier (2001)
C
A
D
E
F
H
J
“H” node’s being lower in the same
group as “K” node means that K is an
ancestor of H.
Additional meaning
relation
Analysis of derived meaning: PDs
A particular way
(1) holds
(1) At least two
people defeated
Jon
…
constraint
indicates
(1*) At least two
names appear to
the left of the
name “Jon”
indicates
indicates
…
constraint
A particular way
(1*) holds
indicates
Enlarged view
A particular way
(1) holds
(1) At least two
people defeated
Jon
constraint
(1*) At least two
names appear to
constraint
the left of the
name “Jon”
Gil defeated Jon
Bob defeated Jon
indicates
indicates
The name “Gil”
is to the left of
the name “Jon”
The name “Bob”
is to the left of
the name “Jon”
A particular way
(1*) holds
Analysis of Derived Meaning: General
(Shimojima 1999, 2002)
A particular way
holds
…
constraint
indicates
…
constraint
indicates
…
A particular way
holds
…
indicates
…
indicates
…
Thus:
A representation system with a meaning derivation property
allows the simultaneous presentation of local information
and global information implied by the local information.
Kulpa (2003)
Concerning the original numbers, they can be easier and
more accurately read off from a list of numbers, without
the expense of producing that graph. What such graphs
are really for is something different---namely, a possibility
to see at a glance some general conclusion, i.e., a result
of some reasoning that follows from the interaction of
these numbers. (p. 111)
mln $
30
20
10
0
'96
-10
'97
'98
'99
Lowe (1989)
Indeed, the central purpose of many scientific diagrams
is to depict relationships and interactions….If students
are to understand such diagrams, they need to be able to
do more than just decode the symbols used. They must
also be able to uncover and assimilate salient
relationships between the symbols that constitute a
diagram and appreciate how these relationships map
onto the real-world situation being represented. (p. 28.)
Also:
Petre (1995): Generally critical about the insensitive use
of graphics in programming environment, but mentions
“gestalt response” as an “informative impression of the
whole that provides insight into the structure” (p. 42) and
admits it as a potential benefit of graphics.
Ratwani, Trafton, and Boehm-Davis (2003): Assumes the
difference of global/trend reading and local reading, and
goes on to demonstrate that different mental operations
are involved in them.
Connection: expert reading of graphics
Correctly assessing and recognizing derived meaning may
be a major component of expertise in reading graphical
representations.
For example:
• Lowe (1989): Compared the professional meteorologists’
way of inspecting (incomplete) meteorological charts with
non-meteorologists’ and find their greater appreciation of
large-scale patterns of organization.
• Winn (1991): Discusses expertise of reading topographical
maps as perceptual chunking of contours to form larger
features, such as valleys.
Summary
Free ride
Over-specificity
Distributed
cognition
Formal deduction
systems
Graphical
simulation
Justificatory status
of diagrammatic
proofs
Design studies
AI systems with
diagrams
Design studies
Derived meaning
Graphics
semantics
Expertise in
graphics reading
Information
graphics/
visualization
Useful and central set of concepts
Concepts not covered today
Mental transformation
of graphics
Narayanan, Suwa &
Motoda (1995), Schwartz
(1995), Trafton & Trickett
(2001), Shimojima (2003)
Low-encoding diagrams
Cheng (2003) etc.
Free ride
Auto-consistency
Over-specificity
Derived meaning
Perceptual chunking
of graphical elements
Lowe (1989), Anderson
& Koedinger (1992), etc.
Barwise & Etchemendy
(1995), Stenning & Inder
(1995), Lemon & Pratt
(1997)
Spatial indexing of
information
Larkin & Simon (1989), etc.