Lecture 4. Representing Space II:
Shortcomings of the inner space
proposals and a possible alternative

The problem of nonconceptual representation of space
and scenario content
 The internalized space options
 Internalizing constraints: Marr & Shepard
 Functional Space
 Literal (Neural) space

The empirical evidence for a special form of
reconstructed spatial representation: Some imagery
demonstrations and experimental findings
A quick summary of some empirical
findings involving mental imagery: or
What people mean when they say
images are ‘spatial’
Two general kinds of studies
1. Studies showing that imagery ‘involves’ the
visual system

I will briefly mention these because most of them
lead to the ‘images are spatial’ view
2. Studies showing that image representations
have spatial properties
Summary of some empirical findings on
mental imagery and vision

Imagery and Vision
 Early studies of interference between vision and
imagery (Brooks)
 Activation of visual cortex during imagery
Why is this relevant?
 Because V1 is retinotopic!
 Image-vision superposition studies

Most of these studies relating imagery and
vision are of interest to us only because they
were thought to support a picture-theory of
images in whch images are laid out in space
Is mental imagery is carried out by
the early (modular) visual system?
Suppose that mental imagery involved the visual system, what
would that tell us about how space is represented? Before
drawing conclusions, consider:
 Virtually all mental imagery phenomena (scanning, mental rotation,
etc) are exhibited by congenitally blind people.
 Evidence for the involvement of primary visual cortex is not
univocal. The majority of studies report that many cortical areas are
involved in mental imagery
 There is clear evidence of spared visual imagery with severely
damaged visual perception and vice versa (this applies to
achromatopsia, agnosia, visual neglect and cortical blindness)
 There is good evidence that information in imagery is not
reperceived through vision but only inferred
Summary of some empirical
findings concerning the spatial
character of mental images.
What makes images “spatial”?
Why do we think images are spatial?

The strong intuition that things in an image have a location, in
some relevant sense – e.g., you can point to them!
 How could a “symbolic” format have scenario content? Directions?

Spatial interaction between location in an image and location in a
stimulus
 Spatial Interference (Brooks)
 Attentional enhancements (e.g., Podgorny & Shepard)
 Illusions in combined image-perception displays

Effects of image distances (e.g., scanning)
 The connection with the motor system



Eye movements during mental imagery (Brandt &Stark, 1997)
Visual-motor adaptation (Finke)
The Simon effect (S-R compatibility with images)
Brooks’ spatial interference study
Respond by pointing to symbols in a table or by saying the words left or right
Shepard & Podgorny experiment
Both when the displays are seen and when the F is imagined,
RT to say whether the dot was on the F was fastest when the
dot was at the vertex of the F, then when on an arm of the F,
then when far away from the F – and slowest when one square
off the F.
Visual illusions with projected images
Bernbaum & Chung. (1981)
Studies of ‘mental scanning’
Do they show that images have metrical properties?
1.8
1.6
1.4
Latency (secs)
1.2
1
0.8
0.6
0.4
0.2
0
Relative distance on image
Other reasons for thinking that images are
“spatial”: The connections among vision,
imagery and motor-control

Eye movements during imagery are similar to those during
visual perception of the same figure

Visual-motor adaptation occurs with imagined hand position
that is similar to adaptation with prism glasses (Finke, 1979)

Stimulus-response compatibility effects are observed between
location in an image and the side of the response

Patients with visual neglect often show imaginal neglect
 Famous 1978 Milan Duomo experiment of Eduardo Bisiach

A patient with post-operative tunnel vision also showed
“tunnel imagery”(But by the time she was tested, the patient know how things looked to her)

It takes longer to report details from a “small” image!
If images are spatial, where is the space?

Is it in the head?
 Is it an abstract representation?
 Functional space?
 Analogue space?
 Tacit (conceptual) knowledge of space?
 Is it in the world that is concurrently perceived
and in which objects are indexed?
Are images displayed in a functional space?

Can we explain the spatial character of images by
appealing to a “functional space”?
 What properties does a functional space have?
 In virtue of what (natural laws, principles,
architectural properties of mind) does the functional
space have the properties it does?
The picture theory and “Functional Space”
“The space in which the points appear need not be physical…,
but can be like an array in a computer, which specifies spatial
relations purely functionally. That is, the physical locations in
the computer of each point in an array are not themselves
arranged in an array; it is only by virtue of how this
information is “read” and processed that it comes to function
as if it were arranged into an array (with some points being
close, some far, some falling along a diagonal, and so on).”
(Kossyln, 1994)
Is there any coherent sense of “space” that is neither
literal (real, physical) space nor merely knowledge of
spatial properties, with no commitment about the
format in which this knowledge is represented?
In what way is a matrix data structure spatial?
Must the cells be accessed in any particular order? Which cells are “between” two
given cells? Which cells are collinear? What makes that so?
In what way is a matrix data structure spatial?
Must the cells be accessed in any particular order? Which cells are “between” two
given cells? Which cells are collinear? What makes that so?
In what way is a matrix data structure spatial?
Must the cells be accessed in any particular order? Which cells are “between” two
given cells? Which cells are collinear? What makes that so?
Questions about the Functional Space option
Where does the “spatial” property of “functional space” come
from? Why does a matrix seem to have the spatial properties
assumed in picture-theories of mental imagery?
 Must Pythagoras’s theorem hold of the matrix? Why?
 Why does attention have to pass through “intermediate” cells in
getting from A to B.
 What makes these cells “intermediate”? In what sense are they
“between” A and B?
 Why does it take longer to move attention through more cells?

Why does adding point C not change the relation between points A
and B?
 What makes it the case that when C is added, then the location of B
is between A and C?
Spatial properties of functional space
There are two possibile reasons why spatial
properties apparently hold of functional (matrix)
space:
 The principled alternative. These are principled
(lawlike, inherent) properties because the matrix is
(tacitly) assumed to be just a simulation of real space, or
 The ad-hoc alternative. The properties are not inherent
properties of the functional space. Rather, they are just
a way of depicting the properties that were observed in
the data (this is just like listing the findings without
explaining them).
Ways of explaining the scanning effect:
1. If images have spatial extent, then the
explanation is straightforward:
Time = Distance
Speed
2. If images represent distance, then
(distance) ?
Time = Representation-of
Representation-of (speed)
Only (1) constitutes an explanation!
Aside: Is image space represented
in an analogue form?

An analog representation is not subject to the arguments
presented against “functional space”
 This is a much more difficult proposal than generally
recognized – which is why nobody has seriously proposed an
analogue representation of space other than space itself
 It would require a representation that not only results in
Time = Rep(distance)  Rep (speed), but the properties that
represent distance and speed would also have to meet many
other requirements – e.g., they would have to model Euclid’s
Axioms, and much of basic contact physics (since these are
obeyed in imagery).
 This would still not do because most imagery phenomena are
Cognitively Penetrable so they do not arise from properties of
the architecture
A puzzle …

Why do we find it natural to assume that brain processes
underlying mental imagery share certain properties with
what is represented, but not others?
 Do the brain processes involved in representing a certain event
take the same amount of time as the duration of the imagined
event? A monotonic function of real time?
 Does the size of the brain state corresponding to a large image
reflect the relative size of what is imagined?
 Are distances in the neural representation of a certain scene a
monotonic function of distances in the scene?
 Does the color (volume, mass, temperature) of the brain state
vary with the equivalent represented property?
The Intentional Fallacy strikes again!
The literal space option
This was not taken serously until the recent
findings that:
 The early visual cortex (V1) is spatially
mapped, and
 Visual cortex is active (as determined by PET
and fMRI) when people examine a visual image
Activity on visual cortex of primates is retinotopic
Tootell, R. B., Silverman, M. S.,
Switkes, E., & de Valois, R. L.
(1982). Deoxyglucose analysis of
retinotopic organization in
primate striate cortex. Science,
218(4575), 902-904.
Does the retinotopic mapping of visual
cortex support the view that images are
displayed in real space in V1?


The “display” in visual cortex is retinotopic, which
means it is narrow in scope, moves with eye
movements, and is inherently 2-dimensional!
But scenario content is panoramic and 3-dimensional!
 Mental images are experienced as being in allocentric
coordinates and are panoramic or even cycloramic
 Every imagery finding applies equally in 3D as in 2D
 Image size does not map onto size in V1, so the “hard to
see” explanation of small images does not work
The topographical structure of the visual cortex
could not support mental images
The 2D mapping of retinal activity in V1 cannot be
identified with the mental image which is panoramic,
3-dimensional, dynamic and has many other
properties that could not be mapped onto V1, so we
would need a different theory for them.
…and there are many other differences …

Image content is not visually reinterpreted: it does not
have the signature properties of visual interpretation (e.g.,
amodal completion, spontaneous 3D interpretation, visual
ambiguity, Emmert’s law, etc etc)

Image content cannot be accessed in free order

Image content is whatever we want it to be! i.e., it is
cognitively penetrable because it is your image!
e.g., our mental scanning findings
Exactly the same applies to the effect of image size, visual
angle of the mind’s eye, the oblique effect, etc
Imagine two parallelograms
Imagine two parallelograms
Imagine two parallelograms
Amodal completion in imagery?
Amodal completion in imagery?
Slezak figures
Pick one of these animals and memorize what
they look like. Now rotate it in your mind by
90 degrees clockwise and see what it looks like.
Rotated Slezak figures

No subject was able to recognize the
mentally rotated figure

Subjects remembered the figures well
enough so if they drew it they could
recognize the rotated figure
PS. There have been differing claims about such results!
…and there are many other differences …

Image content is not visually reinterpreted: it does not
have the signature properties of visual interpretation (e.g.,
amodal completion, spontaneous 3D interpretation, visual
ambiguity, Emmert’s law, etc etc)

Image content cannot be accessed in free order

Image content is whatever we want it to be! i.e., it is
cognitively penetrable because it is your image!
e.g., image size?
e.g., our mental scanning findings
Exactly the same applies to the visual angle of the mind’s
eye, the oblique effect, and dozens of other findings
Studies of ‘mental scanning’
Do they show that images have metrical properties?
2
1.8
1.6
Latency (secs)
1.4
scan image
imagine lights
show dire ction
1.2
1
0.8
0.6
0.4
0.2
0
Relative distance on image
(Pylyshyn & Bannon. See Pylyshyn, 1981)
An alternative view of the source of
spatial properties of images
There is something special about
images that are ‘projected’ onto a
perceived scene
 Can phenomena involving projected
images be explained in terms of
indexing of objects and binding them
to imagined things?
Shepard & Podgorny experiment
Both when the displays are seen and when the F is imagined,
RT to say whether the dot was on the F was fastest when the
dot was at the vertex of the F, then when on an arm of the F,
then when far away from the F – and slowest when one square
off the F.
Visual illusions with projected images
Bernbaum & Chung. (1981)
Visual adaptation and aftereffects
through images (Ron Finke)

Instead of seeing their hand in the wrong location (due to wedge
prisms) subjects were told to imagine that there hand (which
was actually under a screen) was at some shifted location

After moving their hands a few times and each time imagining
them to be at the false location, subjects were allowed to see
their hand reaching and it did show a negative aftereffect

But the conditions for adaptation are known to be (1) a
mismatch between felt and seen position and (2) a conviction
that the hand is actually at the given location. The same result
can be obtained by a light held by the hand.
 No actual visual properties of the hand are required
S-R Compatibility effect with a visual display
S-R Compatibility effect with a mental image
What happens when you use
imagery with your eyes closed?

The existence of a very accurate spatial sense suggests
that there are mechanisms similar to visual indexes in
other modalities

We don’t need to have a representation of empty places
so long as we have coordinate transformation
operations linking gestures and representations of
filled locations (tactile perceptions are of tactile
objects)
Can indexing convert an apparently
mental-space phenomenon to a
perceived-space phenomenon?

When the imagined stimulus is projected onto a
visually perceived background (as in the S-R
compatibility example) it is easy to see how it can
become a visual phenomenon – subjects must respond
to a place selected in the actual display (or near it)
 Can we apply this approach when there is no visual
input to which to bind imagined objects?
 This is where the idea of spatial sense plays a role. We
can convert between sensed modalities with coordinate
transformation processes.
A spatial sense is very different
from a mental picture
 Exercise of the spatial sense need not be accompanied by a
perceptual experience. It is amodal and typically free of any
conscious content
 It does not require that any properties of imagined objects be
represented – just their identity (indicated by binding them to
sensory objects)
 The spatial sense exploits visual, auditory, kinesthetic and
proprioceptive information
 The sense of space does not need an internal spatial medium.
It arrises purely from the capacity to bind object files
(possibly empty) to sensed objects.
We don’t need a spatial display in our head if we
have a way to pick out and keep track of a few
sensory objects in real space
1. None
of the experiments that suggest a spatial display in
visual cortex need to appeal to anything more than a
small number of imagined locations (e.g., Podgorny
Shepherd , Finke, Tlauka, …).
 All spatial imagery phenomena have been found in the blind
2. If
we can pick out a small number of occupied locations
in real space we can use these to allocate attention or to
target eye movements. This is likely what goes on in the
Shepard & Podgorny, Bernbaum & Chung and other
cases involving combined vision and imagery.
We don’t need a spatial display in our head...
3. If
these selected perceived objects are also bound to
objects of thought (e.g., “Object Files”) it can result in the
objects of our thought having persisting spatial locations
and spatial relations (e.g., obeying metrical Euclidean
axioms).
4. This would also explain why some patients with visual
neglect sometimes show neglect to the same side of their
image (Bisiach & Luzzatti, 1978).
 Patients may have projected imagined objects onto real space in
the course of their imagining, and their visual neglect resulted in
failure to orient to the indexed objects in the neglected field.
Neglect appears to involve a failure to orient attention (Bartolomeo
& Chokron, 2002)
 The patient whose surgery resulted in tunnel vision and tunnel
imagery may simply have been showing she knew how things
looked to her post-surgery
Should you try to rescue the picture theory?
Summary



I have presented a view of how conceptual representations
might be grounded in causal connections to the world,
through a perceptual mechanism that enables individuation,
selection, and reference to individual sensory objects
I have proposed that this mechanism picks out a limited
number of sensory individuals in the world – individuals
that usually turn out to be physical objects, although they
are picked out nonconceptually and therefore not as objects
While identifying and tracking individuals in general
requires a conceptual apparatus for individuation and
identity, perceptual systems are equipped to provide this
function in a way that almost always works in our kind of
world. Without such a mechanism, grounding of concepts
in experience would not be possible
Summary

The theory of Indexing (FINST theory), developed for quite
different reasons, postulates that there is a primitive capacity
in the modular perceptual architecture for identifying and
keeping track of individuals in a nonconceptual manner
 This nonconceptual “picking out” is closely related to focal
attention and is able to independently select several (~4-5)
causally efficacious entities. These are usually physical
objects, as opposed to empty places, since empty places do
not have index-capturing causal powers
 This view has consequences for a theory of sentience and for
any theory of symbol-grounding (or concept-grounding). It
provides a different foundation for sentience than that of
feature-placing, adopted by Strawson, Clark and others.
Summary
 This
account assumes a spatial sense, viewed as a
perceptual-motor skill, whose exercise requires referring to
causally efficacious indexed things in the perceived world
and computing coordinate transformations only among the
selected individuals and only as needed.
 The theory shuns the assumption of a nonconceptual mental
representation with rich scenario content, such as that which
we consciously experience, and denies the existence of
picture-like representations in either vision or imagery.
Spatial properties are assumed to be conceptually encoded
from the world itself, rather than from a nonconceptual
picture-like mediating representation. Since such property
encoding occurs only for indexed objects, this also provides a
solution to the binding problem
Summary & End

The FINST index theory claims that certain spatial
properties of mental images are inherited from the
world by a process of associating objects of thought
with indexed objects in several modalities. Since
properties of indexed objects can be visually evaluated,
this makes it possible to carry out certain spatial
inferences without the need for a rich nonconceptual
spatial representation beyond that provided by the
selection of individuals through FINST indexes.
More information can be found at:
http://ruccs.rutgers.edu/faculty/pylyshyn.html