Lecture 3: Table of Contents
Cognizing Space 1: Nonconceptual content
and the impression of space
1
Introduction: Where we stand ........................................................1
2
The problem of the sense of space ...............................................2
3
The role of conscious experience in the study of perception .....3
3.1 The experience of space ........................................................................ 5
4
Nonconceptual content and the experience of space ..................5
4.1 What is the problem of spatial representation? ................................. 6
5
The genesis of our “Sense of Space”: ..........................................7
5.1 The role of visuomotor experience: Poincaré’s insights .................... 7
5.2 Coordinate Transformation as the basic operation in sensing space . 9
5.3 Do we pick out spatial locations in a unitary frame of reference? . 10
5.4 The coordinate transformation function and as-needed translation
11
Pylyshyn
Lecture 3
Lecture 3. Representing Space 1:
Nonconceptual content and the sense of space
1
Introduction: Where we stand
I have so far argued that early vision contains a small number of reference pointers or
Indexes, called FINSTs, for referring to Visual Objects. I have occasionally allowed myself to
call them simply Objects, because that’s what FINSTs are for – they are for picking out and
maintaining the identity of real objects. [By the way, when I say that FINSTs are for picking
out and keeping track of objects I don’t mean this as a teleological justification, but merely as a
generalization of what role a mechanism like that would play in a human’s interaction with the
world. Generalizations of this kind are essential in cognitive science simply because we find
ourselves regularly needing to distinguish between a mechanism’s general or regular behavior
and its aberrant behavior. All hypothesis about the functioning of a mechanisms are postulated
against a background of its assumed normal function. Were it not for such background
assumptions we could not distinguish between illusions and veridical perception since in an
important sense all vision goes beyond the information given – ie beyond what it is logically
entitled to reconstruct.]
Of course FINSTs cannot pick out all and only physical objects, even in principle, because
Objecthood is a conceptual category and a FINST is a mechanism operating on the
nonconceptual level. What they do they do because that is in their nature: evolution wired
them so that they are able to select and track physical objects under the usual conditions that
obtain in our kind of world. (They also can’t pick out all objects because infinitely many of
them are too small, too big, or simply invisible).
I have also argued that we perceive properties in the world as properties of indexed objects,
not simply as properties that exist in the visual scene. This suggests that properties of only
about 4 or 5 objects (the ones that are Indexed) are encode or conceptualized. The concepts or
predicates that describe those objects may be stored in what are called Object Files, which are
linked by FINST pointers to the objects themselves. The rest of the visual landscape is not
conceptualized and therefore is not available for thought. This view accounts both for the
apparent paucity of encoded information in a visual scene and for how the binding problem is
solved (it is solved because the conjoined properties are stored in the same object file and thus
associated with the same visual object). According to this story, locations in space are not
indexed and therefore are not conceptualized. But the account says nothing about
nonconceptual representations of space itself. This may strike you as a very odd omission for a
topic that makes much of the notion of nonconceptual representation and in some cases (e.g., in
1
2/12/2016 6:32 AM
Pylyshyn
Lecture 3
Asten Clark’s book) even speaks of sentience. The representation of space and spatial relations
would seem, at least prima facie, to be something that demands a different story from one that
claims not only a sparse set of conceptual representations, but also a very limited
nonconceptual mechanism for connecting with the world (viz, the visual indexes). The sensory
impression of space appears to be central to all sentience and appears to be far from being
addressed by a theory that claims that only a few objects are nonconceptually picked out. It
appears to us that space is something you sense everywhere at once.1 So at least on the face of
it, the omission of space from this account leaves it disconnected from the clearest case of
nonconceptual representation. Today I want to address this omission by discussing spatial
representation, both nonconceptual and conceptual as well as the general skill I will refer to as
our sense of space.
2
The problem of the sense of space
For those interested in the nature of sentience – in how we experience the world through
our senses – it has been an article of faith that this question is closely related to understanding
how we sense and experience space and spatial relations among objects in space (some
philosophers have even argued that our sense of self is dependent on our awareness of the
location of our bodies in relation to other objects, e.g., Grush, 2000). From the perspective of
these lectures I am interested in the question of the nonconceptual representation of space for
two reasons. One is to see whether this question presents any special problem for the view I
have been advocating, in which it is not places but things (typically physical objects) that
constitute our first causal contact with the world. The second is to see whether the FINST
index theory can help us to understand how representations of spatial layouts attain some of
their spatial properties, where by “attain spatial properties” I mean more that that just
representing qualitative relations of the sort that we can capture in language.
Our grasp of space is subtle, complex and extremely fine-grained. Our experience of space
is all-pervasive; we experience ourselves as being totally immersed in the space around us
which remains fixed as we move through it or as objects other than ourselves move through it.
Our spatial abilities are remarkable. We can orient ourselves in space rapidly and effortlessly
and can perceive spatial layouts based on extremely partial and ambiguous cues. We can recall
spatial relations and recreate spatial properties in our imagination. Animals who may not have
concepts exhibit amazing powers of navigation which establish beyond doubt that they have an
accurate quantitative representation of the space through which they travel. Although vision
science is arguably the most developed of the cognitive sciences there are many areas of vision
science where it is dubious that we have posed the problems correctly, and the problem of
2
2/12/2016 6:32 AM
Pylyshyn
Lecture 3
spatial cognition strikes me as an extremely likely candidate for one of those problems. Before
I look at some of the scientific problems we face, I will take a detour to examine the role that
conscious experience plays in the study of perception. The reason for this detour is that
nowhere does the content of conscious experience play a more central role than in our
apprehension of space, including both in the perception of space and in reasoning in terms of
spatial layouts, particularly in spatial mental imagery.
3
The role of conscious experience in the study of perception
Vision science has had a deeply ambivalent relation with conscious experience. On one
hand, the way things look to us or how they appear in our conscious experience has always
constituted the primary data of theories of vision. Traditionally, from Koffka to Kanizsa,
experiments in visual perception begin with the question: What do you see? Even when they
do not begin with such a sweeping question, it is still true that when one thing looks bigger in
one condition than in another or when something looks to be moving faster under one
condition than another, or when colors appear different under one context than another, these
are considered primary data to which theories of vision are expected to respond. As Kenneth
Pike pointed out many years ago, and as Julian Hochberg reiterated in the context of the
psychophysics of vision, it is how we experience things, as opposed to how they are, that is the
basis for reliable empirical generalizations about cognition (e.g., mixing red light with yellow
light results in our seeing orange light, regardless of which of an infinite variety of mixtures of
wavelengths were responsible for the light we experienced as red and as yellow).
Yet despite the fact that vision science begins with phenomenal appearance, the content of
such experience has also proven to be an extremely misleading base from which to build
theories of visual perception. Furthermore, it is in the analysis of our representation of space
and of spatial patterns where we are most readily led astray. Perhaps this is because the
temptation to assume what (Pessoa, Thompson, & Noë, 1998; Thompson, Noe, & Pessoa,
1999) refer to as analytical isomorphism is particularly strong in that case (that’s the doctrine
that there must be an isomorphism between neural activity and how things seem). For
example it led to the widespread belief that vision provides a dense manifold of panoramic
information about spatial structure and locations – a view that has been shown repeatedly to be
untenable.
Whatever one thinks of the role of phenomenology in vision science (and France is one
place where people do have strong views about this question) it is clear that (1) Taken at face
value it can be extremely misleading and (2) Theories that take conscious experience as their
starting point have no place for the growing body of evidence of vision-without-awareness,
3
2/12/2016 6:32 AM
Pylyshyn
Lecture 3
including change blindness, blindsight, intact visual-motor control in the absence of
recognition and other sources of neuropsychological data.
So we appear to be stuck between needing the data of visual experience and yet not being
able to trust it as the basis for building our theories. How are we to reconcile these
differences? There is no general solution to this problem. The question of how to interpret a
particular type of observation can only be resolved as we unearth broader generalizations or as
we build more successful theories. The situation we are in is very similar to that which
linguistics has been in during the last 60 years. Intuitions of grammatical structure led early
linguistics astray by focusing on surface phenomena. But as generative linguistics became
better able to capture a wide range of generalizations, it found itself relying more, rather than
less, on linguistic intuitions. What changed is that the use of the intuitions was now under the
control the evolving theory. Even such general questions as whether a particular intuitive
judgment is relevant to linguistics became conditioned by the theory itself. Take Chomsky’s
famous sentence “Colorless green ideas sleep furiously” which was introduced to show the
distinction between grammaticality and acceptability. This example engendered considerable
debate because what constitutes grammaticality as opposed to acceptability is not given by
intuition but comes from the nascent theory itself.
So as broader general principles are formulated, they will direct us in the interpretation of
evidence from conscious experience. For example, they will show us how to interpret such
findings as those of (Wittreich, 1959). Wittreich confirmed the well-known observation that
when people walked across the floor of the Ames distorted room they appeared to change in
size, as shown in the figure. But he also found that this did not happen when the walkers were
well-known to the observer, e.g., if it was the observer’s spouse, even if the person was
accompanied in the walk by a stranger (whose size did change!). Even now I think we are in a
fairly good position to be incredulous of the theoretical significance of this finding, given that
we know that reports of conscious experiences (even private reports to yourself) can be
cognitively penetrable, hypnotism being an extreme example of this. Sometimes we can show
this fairly directly by comparing measures from which the response factors have been
statistically factored out, as we do when we use the signal detection measure d′ rather than
percent correct. But sometimes we make the decision on the grounds that a theory that takes
certain observations at face value will simply miss the deeper underlying principles.
4
2/12/2016 6:32 AM
Pylyshyn
4
Lecture 3
Nonconceptual content and the experience of space
4.1 The experience of space
For many philosophers interested in the problem of nonconceptual representation the task
begins with providing a characterization of the experience of space. Consider, for example, the
important philosophical work of Christopher Peacocke. Peacocke (Peacocke, 1992) presents
an insightful analysis of such experiences, in which he introduces the notion of scenario
content – a highly detailed nonconceptual representation of the spatial layout around a person.
Peacocke describes it as the possible ways of filling the space that could be discriminated in
experience. But what can we make of the experience of the space around the body?
The problem is that much of what is in scenario content does not appear to enter the
information processing stream and much of what does enter our information processing in a
substantial way does not appear to be part of our conscious experience. The experience of
spatial layout is particularly problematic because our experience reveals a stable panoramic
layout of allocentric spatial locations, some of which are filled with objects, but most of which
are empty. This led many people to postulate an inner replica of the perceived world which
constitutes the experiential content of our perceived space – our scenario content. If we
assume that the content of our spatial experience must arise from a representation of space, and
that the representation is based on the information we receive through vision, then there is the
immediate problem of how such a representation could possibly be constructed, given the
poverty of the incoming information.2 The incoming information consists of a small peephole
view from the fovea that jumps in rapid saccades several times a second, during which we are
essentially blind, and so on (the information available to the brain has been described in detail
and is a familiar story, see e.g., O'Regan, 1992). So the gap between our visual experience and
the available visual information requires some explanation. While there are many ways to try
5
2/12/2016 6:32 AM
Pylyshyn
Lecture 3
to fill the gap (some of which appeal to visual indexes) the natural way, given the form of the
experience, is to try to postulate an internal facsimile that carries the contents of the experience
– i.e. that is stable in an allocentric frame of reference, that is complete, informationally dense
and panoramic, along the lines of the figure below:
But as we now know, this theory is patently false as an account of the informationprocessing that occurs in the brain – there is no inner picture of any kind in our head, and
indeed no information corresponding to such a picture, regardless of its format. What has gone
wrong? What’s gone wrong is that we are using a particularly natural description of
phenomenological experience as the explanandum: we are trying to explain the content of the
experience in terms of properties of a representation. But we are not entitled to assume that the
content of experience reflects either the structure of a representation or its available
information content – the first is the intentional fallacy, which equates properties of the
perceived world with properties of its representation; the second is merely empirically untrue.
Yet so long as we take the content of the perceptual experience as our primary data this is
where it will lead us.
4.2 What is the problem of spatial representation?
Let me turn now to the scientific question of spatial representation. It seems to our modern
sensibilities that space consists of a dense array of points which can be connected by straight
lines. But these notions, which have been enshrined in our view of space since Euclid, may not
be the right terms in which to describe the space that we perceive – and especially the terms in
which we represent space in our mind when we think about it or imagine events taking place in
it. But what does it mean to say that these notions may not be the right ones for understanding
the experience of space? What options are there?
One of the few people who asked that fundamental question was Jean Nicod. For Nicod
the problem was that the basic building blocks of the Euclidean view were points and lines,
together with the relation of congruity, none of which seemed to him to be the sorts of things
that perceptual systems are equipped to detect. Nicod saw them as complex types that
6
2/12/2016 6:32 AM
Pylyshyn
Lecture 3
collapsed collections of sensory experiences into categories whose virtue was solely that they
made the statement of geometrical principles simple, but they did so at the cost of making their
connection with sensory data opaque. Nicod suggested that since there are infinitely many
models of the Euclidean axioms (the Cartesian model of space as triples of real numbers being
the best known) we should seek instead a way to capture Euclidean spatial properties in terms
of primitives more suited for creatures with sensory systems like ours. After considering a
variety of such possible primitives, he developed several “sensible geometries” based on the
geometry of volumes and the relation of volume-inclusion (which he called “spatio-temporal
interiority”). He argued that this basis is closer to our sensory capacities than one based on
points and lines (for example, volume inclusion is detectable and is invariant with viewpoint so
it can be sensed as we move through space). With the addition of other novel ideas (such as
the idea of succession and global resemblance) Nicod set out a new direction for understanding
what space might consist in for a sentient organism. While in the end he did not succeed in
developing a complete formalization of geometry based on these sensory primitives he did
point the way to the possibility of understanding sense-based space in a way that is radically
different from the Euclidean, Kantian, and Cartesian approaches that now seem so natural to
those of us brought up in the Euclidean-Cartesian tradition. If Nicod had been able to carry out
his program it might have provided a set of tools for thinking about space that would have been
more useful to us than the view that is so thoroughly embedded in our way of thinking. But he
did show us that thinking in terms of points and lines may not be the only way and indeed it
may not be the most perspicuous way for cognitive science to proceed in studying the
experience of space. I concur with this suggestion and will examine an alternative, not for all
of geometry, but for personal space as it may be explored by perceptual-motor systems.
5
The genesis of our “Sense of Space”:
5.1 The role of visuomotor experience: Poincaré’s insights
In what follows I will examine what I call our sense of space by focusing on the relation
between our experience of space and our visual-motor abilities. This will give us a way to
think about the problem that is very different from the most common approaches which rely on
some notion of internalization – on the idea that we apprehend space because space or spatial
constraints have been internalized in the brain. This will be the topic of the next lecture where
I will develop the proposal that rather than internalizing space, the converse actually holds; the
mind actually externalizes space by projecting spatial concepts onto the perceived world using
a nonconceptual motor- and proprioception-based capacity.
7
2/12/2016 6:32 AM
Pylyshyn
Lecture 3
The basic idea for this direction comes from Henri Poincaré. In one of his last essays,
written almost a century ago, Poincaré describes how a three-dimensional impression of space
might arise in a sentient organism confronted with information in many forms and modalities
and potentially in many dimensions. In what follows I will use Poincaré’s terminology, which
is not how we might talk about these problems nowadays. A starting assumption for Poincaré
is that an organism must distinguish between experiences that correspond to changes in
position and those that do not. According to Poincaré the key to being able to recognize this
difference depends, in turn, on being able to distinguish between changes brought about by our
own actions and changes in position that are externally caused. Here Poincaré makes use of
the notion of the reversibility of certain sensations. An externally caused change can be
reversed by a voluntary action that brings back the earlier visual or tactile sensation. It is also
important that this same “renewal” of the tactile sensation can be accomplished by any of an
equivalence class of sequences {S1, S2, S3, …}. According to Poincaré what you, or your
genetic ancestors have learned is that if you are touching an object and the object moves, you
can once again renew the sensation of touching the object by carrying out a motor sequence
that is in the equivalence class. Thus the basis for your sense of space is a certain skill, it is the
skill of moving in such a way as to bring back a tactile or visual sensation after an object
moves away from external causes.
The equivalence-classes of movements define distinct “spaces” and the spaces marked out
by each finger or limb are then merged by virtue of the fact that when two fingers or limbs
touch each other they define a common ‘place’ and so lead to the convergence of what were
initially distinct spaces. [Poincaré then goes on to argue that the reason that our representation
of space has 3 rather than 2 or 4 dimensions is tied to the way that the equivalence classes are
established, together with the boundary condition that we should not count as equivalent two
sequences of sensations that fail to renew a tactile sensation (i.e., that take us to the same final
position, nor should we count as distinct two sequences of sensations that take us to the same
final positions (where the tactile sensation is renewed). It is these boundary conditions that
force the tri-dimensionality of perceptual space.] The reason I have spent as much time as I
have on this point is that apart from providing an elegant account of the basis for the threedimensionality of space, Poincaré’s analysis touches on several issues that are relevant to our
present discussion, not the least of which is his appeal to fingers! 3
The details of this analysis don’t carry much conviction these days, and indeed the
reversibility-of-sensation condition was criticized by Jean Nicod, but the main ideas remain
sound. For example, Poincaré’s distinction between two kinds of changes in sensory states;
those that signal a difference in location and those that signal a difference in some sensory
8
2/12/2016 6:32 AM
Pylyshyn
Lecture 3
quality, such as color or texture, is central in the work of Austen Clark (who devotes an entire
chapter to it, without acknowledging Poincaré’s earlier recognition of the same point). Another
important idea concerns the emphasis placed on sequences of motor actions and to equivalence
classes of such sequences. This is a remarkably modern idea, although it is not expressed in
this way in current writings. What Poincaré’s description shares with contemporary analysis is
the idea that a sense of space may be a construction based on mechanisms that compute the
equivalences among otherwise very different sequences of muscular actions. Computing the
mappings between representations of the position of limbs, sensors, and other movable parts of
the body is arguably one of most general and perhaps best understood functions of the brain –
functions carried out primarily in the posterior parietal cortex, but also in the superior
colliculus, in the motor and premotor cortical areas and elsewhere.
5.2 Coordinate Transformation as the basic operation in sensing space
Computing a representation of one position given a representation of a different position is
commonly referred to as a coordinate transformation. A coordinate transformations is thus a
function from a representation of the orientation of an articulated body part (e.g., a limb or the
eye in its orbit) to the representation of the body-part in a different orientation or relative to a
different frame of reference. It also applies to computing a representation of the orientation of
a different body part or to a representation within the reference frame of one modality to a
corresponding representation in the reference frame of another modality. The relevant
representations of limbs in these cases is typically expressed within a framework that is local to
the parts in question – such as the states of the muscles that control the movements, or the joint
angles that characterize their relative positions, or to endpoint locations relative to the body.
The relevant representations of sensory inputs may similarly be in proximal coordinates – e.g.,
positions on the retina or on the basilar membrane.
The importance of these ideas in the present context relates to the theme of nonconceptual
contact between mind and world. In particular, since I have been arguing that this contact does
not begin with the selection of spatiotemporal regions I need to say how places in space are
specified for purposes of motor actions – and indeed whether places are ever represented as
such. Last week I discussed problems with the traditional view that our first, nonconceptual
(or causal) contact with the world occurs through the detection of features-at-locations (the
idea embodied in Strawson’s notion of a feature-placing language). What I want to do now is
suggest how the apparent function of spatial selection might be achieved using the coordinate
transformation function, without any actual selection of places specified in a unitary global
frame of reference.
9
2/12/2016 6:32 AM
Pylyshyn
Lecture 3
5.3 Do we pick out spatial locations in a unitary frame of reference?
There are a number of reasons to resist the idea that we have a nonconceptual
representation of location-in-space.4 For a start, there are a very large number of distinct
frames of reference. Some are required because of the way in which the sensory information is
initially presented. The visual system, for example, receives information in an eye-centered
frame of reference, but the information may be required, say, for controlling a hand in a frame
of reference that includes joint angles or a 6 degree of freedom frame of reference of location
and orientation. How is the relevant conversion performed? One possibility might be that
each frame of reference is mapped onto a single global frame of reference – say a bodycentered or allocentric frame of reference. But another alternative, for which there is
supporting evidence, is that the representation of objects in pairs of reference frames is only
mapped for the objects that are relevant and only as the information is needed. This idea
receives support from a number of studies. For example, (Henriques, Klier, Smith, Lowry, &
Crawford, 1998) studied an open loop pointing task under conditions in which subjects either
kept their eyes fixated or performed a saccade to a peripheral location. They found that
pointing errors were highly correlated with ocular fixations and not with head orientation,
leading them to propose what they refer to as “…a ‘conversion-on-demand’ model of
visuomotor control in which multiple visual targets are stored and rotated … within the
oculocentric frame, whereas only selected targets are transformed further into head- or bodycentric frames of motor execution.”
The idea that there are a very large number of different frames of reference for spatial
phenomena will not come as a surprise to this audience since a great deal of the work on
multisensory motor coordination has been done in France – particularly in Lyon, Marseille and
Paris by researchers like Marc Jeannerod, Denis Pélisson, Alain Berthoz and Jacques Paillard
(whose work on deafferented patients is particularly relevant to this discussion). The most
famous distinction between types of frames of reference is the distinction between ventral and
dorsal visual systems introduced over 30 years ago by (Ingle, 1973) and developed further by
(Ungerleider & Mishkin, 1982). This idea was illustrated most famously by patients such as
DF (studied by Milner & Goodale, 1995) . DF is a seriously impaired visual agnosic who
despite her inability to recognize the simplest visual patterns, managed visuomotor
coordination at near normal performance levels. Such findings show that even within one
modality different functions (in this case motor control vs object recognition) may involve
distinct frames of reference. These include the ventral system which uses a relatively local
frame of reference and represents primarily qualitative rather then metric spatial relations, and
10
2/12/2016 6:32 AM
Pylyshyn
Lecture 3
the dorsal system which uses a body-centered frame of reference and represents relatively
precise spatial magnitudes (see also Bridgeman, Lewis, Heit, & Nagle, 1979).
The use of multiple frames of reference is also illustrated by cases of visual neglect – a
deficit in attention due to damage in parietal cortex – in which patients fail to notice or respond
to objects in half of their visual field. Even so clearly a spatial deficit appears to show the
many different frames of reference that may be involved, including retinocentric, bodycentered, and environment-centered frames of reference that can be also specific for stimuli
presented at particular distances (Colby & Goldberg, 1999, p320-321).
Properties of many of these frames of reference have been investigated, often with
surprising results. For example, there appear to be integrated visual-tactile representations in
peripersonal space surrounding the hand and face. Visual stimuli presented near the body tend
to be processed together with tactile stimuli so that when one modality shows deficits, such as
extinction5, the other tends to show similar deficits. The visual experience of the region near
the hand or the face appears to be tied to the somatosensory experience of the body part itself,
so that it moves with the body part, it appears with “phantom limb” experiences of amputees,
and has even been shown to be extended with tool use (Làdavas, 2002).
Visual and motor frames of reference are very closely linked in other ways as well. For
example, people can accurately point to a seen object after the eyes are closed, but are poor at
pointing from a different imagined location – unless the person actually moves to the new
location even without vision during the move (Farrell & Thomson, 1998; Gallistel, 1990).
It seems therefore that we have a very large number of distinct frames of reference within
which we represent where things are and that many of these frames are automatically updated
when we move or when we otherwise need to coordinate between them. What does this tell us
about how places are represented nonconceptually?
5.4 The coordinate transformation function and as-needed translation
The ability to coordinate across modalities has frequently been cited as evidence for the
existence of spatial representation as the lingua franca tying modalities together (and therefore
serving to solve the cross-modal binding problem) (e.g., Clark, 2004). But as we have just
seen, the evidence suggests an alternative account that does not require a single global frame of
reference, and in fact does not require that empty locations be represented at all. That is the
proposal that specific sensory objects are translated as needed from one frame of reference to
another. A large number of brain centers have been identified that appear to specialize in
computing coordinate transformation (thoroughly documented by Gallistel, 1999) and have
11
2/12/2016 6:32 AM
Pylyshyn
Lecture 3
even modeled as simple neural networks. Coordinate transformation appear to be among the
basic operation in the central nervous system.
It also makes sense to avoid postulating a single allocentric framework since the origin and
coordinates of that framework are not given by sensory data.6 Also we do not need a unitary
representation of space in order to deal with objects that are located at certain places, so long as
we can coordinate among the many frames of reference from which we do receive direct
information through the senses. This idea is very much in the spirit of Poincaré’s proposal
since implicit in his story was the assumption that people or, as he put it, our “genetic
ancestors” learned the equivalence sets of sequences of motor-commands or proprioceptive
sensations.
Thus the study of visuomotor coordination further supports the claim I made in the last
lecture, viz., that there is no nonconceptual selection or representation of empty places and that
cross-modal coordination is most readily accounted for in terms of Coordinate Transformations
carried out between local sensory representations. While such representations do carry
information about various magnitudes (e.g., joint angles) they carry it in a form that is specific
to the frame of reference in which it originates. According to the terminology I suggested in
the last lecture, this constitutes a special form of representation (or quasi-representation) that I
referred to as analogue. What makes it analogue is not that it is continuous (though it generally
is) but that it represents physical magnitudes in terms of other physical magnitudes and
therefore that the information it carries is specific to the mechanism in which it occurs. It
generally cannot be conceptualized or used to solve the binding problem. Such forms of
information are more closely related to how information is carried by thermostats, planetary
systems, and simple one-cell organisms, than to how it is carried in clear cases of
representations.
So where does that leave us on the matter of spatial experience? I have said very little
about the conscious experience of space – the phenomenology of seeing a spatial layout. I
believe that descriptions such as Peacocke’s scenario content are probably correct – our
experience of space is very fine-grained, richly textured and wholly fills our conscious vista.
But this is clearly not true of our conceptualization of it. This, as many experiments have
shown, is surprisingly sparse, partial and abstract. Studies have repeatedly shown that very
little information is retained between visual fixations. Our studies of the FINST mecahnism
seem to show that among the information that is retained is information that allows us to move
attention back to certain salient objects in the scene for further encoding; that’s what FINST
indexes allow one to do. This ability is also consistent with the results of a number of
12
2/12/2016 6:32 AM
Pylyshyn
Lecture 3
experimental results in other laboratories (Ballard, Hayhoe, Pook, & Rao, 1997; Burkell &
Pylyshyn, 1997; Henderson & Hollingworth, 1999). For example, the findings of Ballard,
Hayhoe and colleagues, which shows that what people appear to retain from a visual fixation is
little more than one property plus the means by which to return for more information – just
exactly what the FINST mechanism provides.
Ballard slides
What happens to the rest of the scenario content? What function does it play? What
underlies and supports its presence? Why does it fail to have a discernable effect on observed
behavior? The thesis of supervienience, which is universally accepted in cognitive science
(and which is weaker than the notion of analytical isomorphism), insists that every experiential
distinction must be mirrored by some formal and physical properties in the brain. 7 This
means that conscious states must be real informational states in Dretske’s sense, so it is a
puzzle why they very often do not have observable behavioral consequences. For example, if
our visual experience is of a panoramic view of the world, then why do we appear to only have
information about the small part of it that we conceptualize? Either there is some principled
reason why we are prevented from conceptualizing certain conscious information-carrying
states, or else we are deceived in thinking that we have in fact represented all that information,
that it is in some sense available information.
Notwithstanding the deep philosophical puzzles that surround the problem of conscious
contents, the evidence is pretty clear that our conscious experience not only fails to tell us what
is going on but may in fact have led us astray on the heartland of cognition – in the study of
thought, problem-solving, and imagination. It just reinforces what I tell my undergraduate
cognitive science students: That cognitive science is a delicate balance between the prosaic –
the everyday, commonplace – and the incredible, between what your grandmother knew which
turns out to be true and what the scientific community thinks it knows which turns out to be
false. In the next lecture we will see the clearest case of this last moral. We will also see that
the notion “sense of space” that I introduced in today’s lecture plays a special role in
understand mental space – or the space of mental images.
13
2/12/2016 6:32 AM
Pylyshyn
Lecture 3
References
Ballard, D. H., Hayhoe, M. M., Pook, P. K., & Rao, R. P. N. (1997). Deictic codes for the embodiment
of cognition. Behavioral and Brain Sciences, 20(4), 723-767.
Bridgeman, B., Lewis, S., Heit, G., & Nagle, M. (1979). Relation between cognitive and motor-oriented
systems of visual position perception. Journal of Eperimental Pyschology: Human Perception and
Performance, 5, 692-700.
Burkell, J., & Pylyshyn, Z. W. (1997). Searching through subsets: A test of the visual indexing
hypothesis. Spatial Vision, 11(2), 225-258.
Clark, A. (2004). Feature placing and proto-objects. Philosophical Psychology(forthcoming).
Colby, C. L., & Goldberg, M. E. (1999). Space and attention in parietal cortex. Annual Review of
Neuroscience, 22, 319-349.
Farrell, M. J., & Thomson, J. A. (1998). Automatic spatial updating during locomotion without vision.
Quarterly J Experimental psychology A, 51(3), 637-654.
Fodor, J. A. (2004). Revenge of the Given.Unpublished manuscript.
Gallistel, C. R. (1990). The Organization of Learning. Cambridge, MA: MIT Press (A Bradford Book).
Gallistel, C. R. (1999). Coordinate transformations in the genesis of directed action, Bly, Benjamin
Martin (Ed); Rumelhart, David E. (Ed). (1999). Cognitive science. Handbook of perception and
cognition (2nd ed.). (pp. 1-42). xvii, 391pp.
Goodman, N. (1968). Languages of Art. Indianapolis: Bobbs-Merrill.
Grush, R. (2000). Self, world and space: The meaning and mechanisms of ego- and allocentric spatial
representation. Brain & Mind, 1(1), 59-92.
Henderson, J. M., & Hollingworth, A. (1999). The role of fixation position in detecting scene changes
across saccades. Psychological Science, 10(5), 438-443.
Henriques, D. Y., Klier, E. M., Smith, M. A., Lowry, D., & Crawford, J. D. (1998). Gaze-centered
remapping of remembered visual space in an open-loop pointing task. Journal of Neuroscience,
18(4), 1583-1594.
Ingle, D. J. (1973). Two visual systems in the frog. Science, 181, 1053-1055.
Làdavas, E. (2002). Functional and dynamic properties of visual peripersonal space. Trends in
Cognitive Sciences, 6(1), 17-22.
Milner, A. D., & Goodale, M. A. (1995). The Visual Brain in Action. New York: Oxford University
Press.
O'Keefe, J., & Nadel, L. (1978). The Hippocampus as a cognitive map. Oxford: Oxford University
Press.
O'Regan, J. K. (1992). Solving the "real" mysteries of visual perception: The world as an outside
memory. Canadian Journal of Psychology, 46, 461-488.
Peacocke, C. (1992). A Study of Concepts. Cambridge, MA: MIT Press/Bradford Books.
Pessoa, L., Thompson, E., & Noë, A. (1998). Finding out about filling in: A guide to perceptual
completion for visual science and the philosophy of perception. Behavioral and Brain Sciences,
21(6), 723-802.
Thompson, E., Noe, A., & Pessoa, L. (1999). Perceptual completition: A case study in phenomenology
and cognitive science. In J. Petitot & F. J. Varela & B. Pachoud & J.-M. Roy (Eds.), Naturalizing
Phenomenology: Issues in contemporary phenomenology and cognitive science. Stanford,
California: Stanford University Press.
Ungerleider, L. G., & Mishkin, M. (1982). Two cortical visual systems. In J. Ingle & M. A. Goodale &
R. J. W. Mansfield (Eds.), Analysis of visual behavior (pp. 549-586). Cambridge, MA: MIT Press.
Wittreich, W. J. (1959). Visual perception and personality. Scientific American, 200 (April), 56-75.
1
The general view is that spatial impressions are nonconceptual, at least initially. Most writers have claimed
that they are represented by an analogue form of representation. For present purposes, analogue representations
of space are nonconceptual because they fail three tests: (1) They do not represent spatial locations under a
description, or they do not represent them in terms of categories, and (2) they do not constitute constituents of
thought – at least not until they are conceptually interpreted. (3) An even stronger test that they fail is one
proposed by (Fodor, 2004) – they do not represent space in terms of a canonical constituent structure. Analogue
representations of space do not have standard parts that function compositionally to determine the content of the
14
2/12/2016 6:32 AM
Pylyshyn
Lecture 3
complex. If they have parts at all these are arbitrary and are not interpreted the way constituents are; rather every
part of an analog representation (of space) represents some part of the represented (space). This is not to say that
every analogue representation is nonconceptual; it is possible for individual atomic concepts representing
magnitudes to be part of what (Goodman, 1968) called a “dense symbol system” and thus qualify as components
of a conceptual representation.
2
I think part of what is wrong is that a certain assumption that has been central to our thinking about
representation may not apply in this case. That assumption, which goes back to at least Ferdinand De Saussure’s
semiotics, is that referring or signifying is mediated by a vehicle, a sign or a symbol, which carries the meaning.
Modern computer science (at least since Turing), and before that, the formalist movement in mathematics, makes
this quite concrete by explicitly dealing with the signifier or the symbol as the physical means by which reference
is achieved. This assumption is certainly at the heart of what is called the computational theory of mind, which is
the foundation of cognitive science. The importance of this assumption cannot be exaggerated: it is the hope of a
materialistic response to Brentano’s problem of how a material object (i.e., a brain) can have states that are about
something – that have intentional content.
So if the experience we have is an experience of the visible world, then it must be mediated by a medium, a
physical representation which carries the content. The natural implication therefore, is that there is a
corresponding representational state that carries that content of our conscious experience. This means that all
experiences are encoded in some form – if not in thought then in some nonconceptual form. As I suggested in the
last lecture, there may be forms, which I called analogue, that lack many of the properties of conceptual
representations and yet provide inputs for visual-motor control and other low-level encapsulated visual functions.
But whatever the form of this sort of representation might be, the thesis of supervienience insists that every
experiential distinction must be mirrored by some formal distinction within the representation – within the formal
properties of the code. (Davidson (1970, p98): "[M]ental characteristics are in some sense dependent, or
supervenient, on physical characteristics. Such supervenience might be taken to mean that there cannot be two
events alike in all physical respects but differing in some mental respect, or that an object cannot alter in some
mental respect without altering in some physical respect. ") This thesis claims that any differences in conscious
experience correspond to differences in some information-carrying code in the mind/brain. This means that
conscious states are real informational states so it becomes a puzzle why they very often do not have observable
behavioral consequences. If our visual experience is as of taking in an entire panoramic view of the world, why
do we appear to only have information about the small part of it that we conceptualize? Either there is some
principled reason why we are prevented from conceptualizing everything that we have represented in
nonconceptual representation, or else we are deceived in thinking that we have represented all that information. I
can imagine developing either version of this bifurcation, but I will not pursue this question here.
Poincaré’s examples use fingers and the capacity to sense the locations of fingers. I can now confess that
his essay was very much on my mind at the time I was formulating the FINST Index theory and is the reason for
the appearance of “finger” in FINST.
3
4
I am taking the position that such spatial representations, if they exist at all, are nonconceptual on the
grounds that they are claimed to be analogue. For present purposes, analogue representations are nonconceptual
because they fail two tests: (1) They do not represent spatial locations under a description, or they do not represent
them in terms of categories or as something, and (2) they do not form constituents of thought – at least not until
they ARE conceptually interpreted. Another even stronger test is one proposed by Fodor (forthcoming) – they are
not represented as having a canonical constituent structure.
5
A deficit in processing two stimuli presented together bilaterally, when neither is impaired when presented
individually.
6
Note that the impressive work by (O'Keefe & Nadel, 1978), which argues for an allocentric representation
of space on the grounds of the existence of “place cells” in the rat that respond selectively to unique places in a
room, need not be in conflict with the present view. Place cells are not cells that fire when the animal thinks
about a certain place, but only when it gets there. Getting there may be a matter of coordinate transformations
from the various sensed inputs and motor actions. It is not known whether the animal can consider or plan in
terms of the relative direction of A and B in a room when it is situated at some place C different from A or B.
What the work on the hippocampus has shown is the remarkable capacity of the navigation module to compute
the equivalence of different movements in reaching a particular allocentric place.
7
Davidson (1970, p98) puts it this way: "[M]ental characteristics are in some sense dependent, or
supervenient, on physical characteristics. Such supervenience might be taken to mean that there cannot be two
events alike in all physical respects but differing in some mental respect, or that an object cannot alter in some
mental respect without altering in some physical respect."
15
2/12/2016 6:32 AM