Ann. N.Y. Acad. Sci. ISSN 0077-8923
A N N A L S O F T H E N E W Y O R K A C A D E M Y O F SC I E N C E S
Issue: The Year in Cognitive Neuroscience
Measuring neural representations with fMRI: practices
and pitfalls
Tyler Davis1 and Russell A. Poldrack1,2,3,4
1
Imaging Research Center, 2 Department of Psychology, 3 Center for Learning and Memory, and 4 Section of Neurobiology,
The University of Texas at Austin, Austin, Texas
Address for correspondence: Tyler Davis, Imaging Research Center, University of Texas at Austin, 1 University Station, R9975,
Austin, TX 78712. Thdavis@mail.utexas.edu
Recently, there has been a dramatic increase in the number of functional magnetic resonance imaging studies seeking
to answer questions about how the brain represents information. Representational questions are of particular
importance in connecting neuroscientific and cognitive levels of analysis because it is at the representational level
that many formal models of cognition make distinct predictions. This review discusses techniques for univariate,
adaptation, and multivoxel analysis, and how they have been used to answer questions about content specificity in
different regions of the brain, how this content is organized, and how representations are shaped by and contribute to
cognitive processes. Each of the analysis techniques makes different assumptions about the underlying neural code
and thus differ in how they can be applied to specific questions. We also discuss the many pitfalls of representational
analysis, from the flexibility in data analysis pipelines to emergent nonrepresentational relationships that can arise
between stimuli in a task.
Keywords: representation; fMRI; MVPA; adaptation
Introduction
A critical distinction that underlies research in neurobiology and psychology is the difference between
processes and representations. Representations are
the codes with which we store information about
the world around us. Processes are the mechanisms
that create and operate upon these representations
so that we can use stored information to guide our
behavior. For example, in the domain of long-term
memory, our first encounter with a bat may result
in the process of encoding a representation of the
features of the animal and other contextual details
of the event. When the next bat is encountered,
retrieval processes may allow us to fondly recall
the representation of the first. A complete cognitive theory details the form of representations and
processes, and how they relate within a domain.1
Given that one of the primary goals of cognitive neuroscience is to delineate how cognition is
supported by the brain, it is highly desirable that
neuroscience methods like neuroimaging be able to
answer questions about both processes and representations. Neuroimaging methods like functional
magnetic resonance imaging (fMRI) have long been
thought to be sensitive to processes in many domains. Univariate voxel-wise measures of activation/deactivation have been used to isolate the neural locus of cognitive processes involved in almost all
domains, from long-term memory2,3 to attention4,5
and categorization.6–8
Recently, there have been a number of efforts to
use fMRI and other neuroimaging techniques to
uncover neural representations and test hypotheses
about how they are used by cognitive processes.9
Many of these studies have pushed the boundaries
of what was originally thought possible with neuroimaging and show promise in further integrating neuroimaging research with cognitive theory.
Indeed, it is at the representational level (as opposed to the process level) that many opposing
theories of cognition differ. For example, theories
of categorization, whether they are rule-based,10
prototype,11 exemplar,12 or cluster-based models,13
doi: 10.1111/nyas.12156
108
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 Davis & Poldrack
all posit that categorization involves a density estimation process but differ in the structure of the underlying representations.14,15 Likewise in long-term
memory, spirited debates arise about the content of
mnemonic representations, such as whether regions
represent contextual or item-level information,16
vivid recollection or general familiarity,17,18 or overlapping or distinct representations,19 even though
nearly all models posit similar encoding and retrieval processes. Being able to decode the form and
content of neural representations using neuroimaging is therefore of central importance for linking
cognitive theories with neural function.
This review discusses the different fMRI analysis
approaches that have been used to measure neural representations, how they relate to each other,
and the assumptions that they make about the underlying neural code (Table 1). Next we discuss the
specific types of questions about the content, organization, and topography of neural representations
that have been addressed by these methods in different research domains across cognitive neuroscience,
and how particular analysis methods may be better
suited to specific questions. Finally, we discuss the
interpretational pitfalls of representational analysis,
with the goal of establishing some general guidance
for concluding that any specific analysis is sensitive
to representational content.
Although we draw on broad examples from across
the literature, we are primarily concerned with reviewing the techniques used to answer representational questions with fMRI, their characteristics,
and potential interpretational pitfalls. We do not
attempt to answer theoretical questions about representational content or organization within any of
the content domains from which these examples are
drawn. For example, there are a variety of important and timely questions about neural representations that are beyond the scope of this review, such
as whether word and concept meaning are coded
in amodal symbols or embodied (e.g., whether the
verb “kick” depends upon simulations of kicking
in motor cortex);20 whether representations are isomorphic to the physical world and accurately reflect environmental statistics, or whether they are
shaped by our goals and interactions with them;21
and whether small groups of neurons explicitly
encode highly specific information (e.g., grandmother, cardinal, Jennifer Aniston), or whether
such specific information arises from combina-
Representational analysis using fMRI
tions of general features coded across networks of
neurons.22
Imaging techniques used to study
representations
The main assumption underlying the use of fMRI to
measure representations in the brain is that the true
underlying neuronal representation of psychological or physical stimulus properties at the cellular
or network level can impact the observed blood
oxygenation level–dependent (BOLD) activation
patterns elicited for stimuli at the level of tens of
thousands of neurons. These properties or features
encoded by the brain may include, but are not limited to, intrinsic aspects of the stimuli, such as the
color, shape, size, or weight; contextual information such as the geographical or temporal context in which a stimulus was encountered; information about the category a stimulus comes from;
or relational and encyclopedic information, such
as how the stimulus relates to and interacts with
other stimuli. Most fMRI research on representation thus far has focused on perceptual representations in vision and the various sensory modalities; however, representation has also become a key
topic of interest in work on emotions, decision making, reward and language processing, and long-term
memory.
The dominant theoretical underpinning of representational analyses in most content areas of fMRI
research is that stimulus representations can be
thought of as points in an n-dimensional space.23–26
This characterization of neural representations in
terms of n-dimensional spaces follows from influential work in cognitive psychology on how psychological representations can often be characterized as points in a representational space,27,28 and
how a variety of cognitive processes, such as stimulus generalization, categorization, and memory,
can be modeled as geometric operations on these
representations.12,29,30 It is important to keep in
mind that not all representations and processes discussed by cognitive psychology are amenable to a
spatial approach,31 a point we expand upon below
in the section on the future of representational analysis. However, this review focuses primarily on examples in vision, categorization, and memory that
lend themselves well to the n-dimensional spatial
framework. This focus on spatial theories, both in
the present review and likely in the broader fMRI
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 109
Representational analysis using fMRI
Davis & Poldrack
Table 1. Pros and cons of reviewed analysis techniques
Analysis technique
Measures
Pros
Cons
Limited to representations
that differ along a single
continuum or
measuring
representations via their
effect on cognitive
processing
Misses multidimensional
representations coded
across voxels
Misses representational
relationships coded by
neurons encompassed
by a single voxel
May not correspond in a
one-to-one manner
with underlying neural
tuning
Magnitude and direction
of adaptation can be
susceptible to top-down
effects from task goals
Less localizable
Less efficient estimation in
time domain
Likely sensitive to a variety
of signals that covary
with stimulus features
and conditions
Differences between
results found for MVPA
and other analysis
techniques may reflect
sensitivity differences
and are not conclusive
evidence of a
combinatorial or
representational effect
Univariate activation
Overall engagement of a
voxel or brain region
Easily implemented
Efficient testing of
representational change
in time domain
Good anatomical
localization
Adaptation
Change in BOLD
activation within a
voxel or brain region
between two temporally
adjacent stimuli
Only method that can
measure
representational
relationships coded
across neurons
encompassed within a
single voxel
Multivoxel pattern analysis
Relationships between
across-voxel patterns of
activation
Allows for combinatorial
effects across voxels
Potentially a more direct
measure of
multidimensional
stimulus
representations
Often has greater
sensitivity than other
techniques
literature, follows from the ease with which spatial
theories can be directly related to statistical measures, such as activation, adaptation, and similarity,
that we discuss later.
Applied to neural representations, spatial approaches assume that when the brain perceives a
stimulus, it will lead to the firing of neurons that represent this point in a neural representational space,
110
and that activation at the voxel-level can be taken as a
proxy measure of the underlying neural tuning. The
goal of representational analyses is to examine which
aspects of physical or psychological representations
are coded in these neural representational spaces. In
practice, testing hypotheses about the content and
structure of neural representational spaces inherently involves measuring the relationships between
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 Davis & Poldrack
Representational analysis using fMRI
Predacity
guinea pig
cow
elephant
bison
rat
goat
hampster
rabbit
mouse
horse
squirrel
rhinocerous
chipmunk
deer
Size
bear
dog
wolf
bat
fox
sea lion
tiger
dhole
honey badger
weasel
cat
lion
mongoose
leopard
Figure 1. A depiction of the example mammal space with size
and predacity dimensions used to illustrate the assumptions
underlying the different representational analysis techniques.
neural signals elicited for different stimuli, under
the assumption that stimuli that are nearby in the
neural representational space will share a relationship that is absent (or barely present) between two
distant stimuli.
Because there are no existing studies that employ all of the possible representational analysis
techniques, we will use examples from an idealized mammal space with the dimensions of size and
predacity (Fig. 1) to illustrate the assumptions that
different types of analyses make about how representational spaces are coded in the brain. This space
conforms roughly to seminal findings in the semantic categorization literature on how people organize
their mammal concepts.32 After introducing each of
the primary analysis types in relation to this idealized space, we will review specific applications of
these methods in real-world neuroimaging analysis
problems.
Univariate activation
Univariate activation33,34 is the most extensively
used method for delineating the neural systems associated with cognitive processes, but has also enjoyed some success as a measure of neural representation in a wide variety of perceptual, semantic, and
reinforcement learning domains.35–39 Univariate activation analysis can be used to measure representa-
tions by showing that a region of interest (ROI) or
voxel’s mean activation level differs between stimuli
that differ along a representational dimension. In
terms of the idealized mammal space, univariate activation could be used to measure whether a voxel or
ROI represents the dimensions of size or predacity
by showing that the region’s mean activation differs between stimuli or categories that are separated
along this dimension. For example, if a region codes
for size, activation elicited for large mammals should
differ from that elicited for small mammals, or will
vary continuously with size, whereas differences in
predacity should not affect activation (Fig. 2).
Model-based fMRI has recently extended the
types of representational questions that can be addressed using univariate activation by providing a
concrete psychological measure of how representations of a property or category change over the
course of a task.40–43 In model-based fMRI, a formal theory of how representations combine to influence cognitive processing is incorporated into data
analysis and used to assess voxel-wise activation in
the brain. These model-based predictions can be
straightforwardly related to a representation such as
in reinforcement learning, where models are used to
make predictions about how the representation of
expected rewards for different choices change over
the course of a task.44 However, model-based analysis can also allow inferences about the structure of
neural representations based on a model’s predictions for how representations should impact processes like memory retrieval. For example, if two
models make different predictions for how their representations combine to influence memory, the one
that better predicts retrieval-related activation will
be favored.41 It is important to note that in this latter case, univariate activation is not interpreted as
a direct measure of neural stimulus representation,
but rather a reflection of how underlying stimulus
representations contribute to processes engaged by
a region.
Because univariate analysis does not take into account relationships between voxels, it is an inherently localist measure and will succeed as a measure
of representation to the extent that individual voxels or regions as a whole distinguish between different types of representational content, as opposed to
content being represented in a combinatorial code
across regions. Finally, univariate activation for a
stimulus or condition is always measured relative to
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 111
Representational analysis using fMRI
Davis & Poldrack
Univariate Activation
Stimulus Presentation
Neural Response
Activation
elephant
+
goat
e
Tim
+
hamster
A
Figure 2. A depiction of how univariate activation is hypothesized to relate to neural representation. On each trial of the task,
neurons within a voxel that are sensitive to the size dimension of the example mammal space fire homogenously to differences in
size, leading to changes in overall voxel activation that correlate with size.
another stimulus or condition and depends critically
on what content is being compared. Thus, univariate
activation is often not thought of as giving a stimulus’ absolute location in a representational space,
only its distance in the space relative to a contrast
set.
Adaptation measures
Aside from univariate activation, adaptation measures have been one of the longest used measures of neural representation.45–48 Neuroimagingbased adaptation measures (e.g., fMR-adaptation)
are based on observations at the single-cell level
that rapid sequential presentation of repeated or
similar stimuli will result in lower neural activity
for the second stimulus in the case that a cell is
sensitive to some psychological or physical prop-
112
erty shared across repetitions.49 This reduced activity or adaptation for the second stimulus is thought
to reflect the tuning of the neuron for the property. A neuron that is finely tuned to small differences along a dimension will release from adaptation for small differences between stimuli, whereas
a neuron that is broadly tuned toward a property
may not release from adaptation for even large
differences.
As applied in neuroimaging, where the neural
measurements combine across hundreds of thousands of neurons, adaptation techniques answer the
question of whether there are neurons within the
population encompassed by a voxel or ROI that code
a particular stimulus feature or dimension. For example, if an ROI or voxel contains neurons that
are sensitive to differences along the size dimension
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 Davis & Poldrack
Representational analysis using fMRI
Univariate Activation
Adaptation
e
Tim
e
Tim
Figure 3. An example context in which univariate activation would fail to successfully measure neural coding of size, but where
adaptation measures would be successful. Assuming a single voxel contains an equal number of neurons that code for large (e.g.,
elephants), medium (e.g., goats), and small mammals (e.g., hamsters), the overall activation of the voxel would be invariant to
changes along the size dimension. However, sequential presentation of stimuli may still lead to adaptation effects to the extent that
overlap in the representational tuning of neurons within the voxel relates to differences in size. Sequential presentations of stimuli
that are closer in size (e.g., elephant, goat) will adapt size-sensitive neurons more than sequential presentation of stimuli that are
far apart in size (e.g., elephant, hamster).
in the example mammal space, there will be more
adaptation between objects that are closer in size,
such as an elephant and a goat, than objects that are
of vastly different sizes, such as an elephant and a
hamster (Fig. 3).
Although the observed level of adaptation on a
given trial will clearly be sensitive to overall univariate activation levels for a stimulus (e.g., if a region’s
overall activation correlates with mammal size), the
sequential and relational properties of adaptation
techniques enable them to answer questions about
neural representations in contexts for which standard univariate contrasts would fail. One context
in which adaptation allows for measurement of a
neural representational space, but in which univariate activation fails, is when a single voxel contains
neurons that code an entire representational space
(e.g., the entire range of mammal sizes; Fig. 3).46,47
If all points in the space are equally represented by
neurons within a voxel, then the overall activation of
the voxel will be invariant to differences within the
space. For example, if a voxel contains neurons sensitive to large (e.g., elephant), medium (e.g., goat),
and small (e.g., hamster) mammals, then none of
these will elicit greater overall activation. However,
if the selectivity of neurons within the voxel mirrors the properties of the representational space,
then similar objects should still elicit greater adaptation in sequential presentations than do dissimilar
stimuli (Fig. 3).
It is important to note that, like univariate activation, adaptation for any given two-stimulus sequence only gives a measure of the second stimulus’
distance from the first stimulus in a representational
space. However, by using continuous adaptation designs in which stimulus order is counterbalanced
across first-order pairings and each stimulus serves
as both an adaptor and a probe,50 it is possible to
build adaptation matrices that measure the adaptation between all pairs of stimuli. Like behavioral
similarity matrices, these pairwise adaptation matrices can be used to triangulate the specific location of a stimulus in multidimensional space using
scaling techniques or be compared to a predicted
physical or psychological similarity matrix. For example, neural adaptation between pairs of simple
perceptual stimuli has been shown to correlate significantly with their psychological distance, suggesting that adaptation can recover the basic structure
of subjects’ perceptual spaces.24
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 113
Representational analysis using fMRI
Davis & Poldrack
Although the theory of fMR adaptation is based
on sound physiological principles, using adaptation
to study neural representation is not without its
pitfalls. One of these is that adaptation does not
always correspond in a one-to-one manner to the
representational specificity of the underlying neural
code and may be susceptible to experimental demands, top-down processing, and subjects’ goals,
particularly for exact stimulus repetitions.51–54 For
example, neuronal adaptation has been shown to
be less than expected for two stimuli that a neuron was equally sensitive to relative to exact stimulus repetitions,52 suggesting that, in some cases,
comparing adaptation effects between two different stimuli and exact repetitions may overestimate
representational specificity. In other instances, exact repetitions of stimuli, particularly when they
are rare, can lead to a release from adaptation or
larger activation to exact repetitions.53 The specific
mechanisms and contexts that lead to adaptation or
enhancement are currently a matter of debate.54,55
However, care should be taken in interpreting adaptation findings as a measure of neural representation, particularly for exact stimulus repetitions.
Multivoxel pattern analysis
Multivoxel pattern analysis (MVPA) has recently
gained widespread acceptance as the premier technique for measuring neural representations in fMRI
data.56–59 In this approach, stimulus representations
are taken to be reflected in the multivariate patterns of activation elicited for stimuli across voxels
within a ROI or across the whole brain. To this
end, unlike univariate or adaptation measures, the
precise activation/deactivation patterns across voxels in multivoxel analysis are assumed to code an
item’s position within a representational space. This
makes many multivoxel techniques easily relatable
to vector-space models that have been used to understand representational spaces in several different
domains of cognitive science.13,19,60 To the extent
that the assumption is true that multivoxel patterns
reflect an item’s location in neural representational
space, MVPA techniques are also more direct measures of representation than are univariate activation or adaptation.
Representational similarity analysis. There are a
variety of ways that pattern information analysis
techniques can be used to measure representations
from neuroimaging data. At a general level, MVPA
114
measures whether there is an isomorphism between
stimulus structure (such as categorical or dimensional structure) and activation patterns. The most
basic methods for testing these questions are encompassed by representational similarity analysis
(RSA),61 in which a similarity/distance metric, often a correlation, is computed between multivoxel
activation profiles elicited for different stimuli to
test whether the region represents a category or dimension. For example, if a region represents the
size dimension of mammals, mammals with more
similar sizes (e.g., elephant and goat) should elicit
activation patterns that are more similar than mammals with different sizes (e.g., elephant and hamster;
Fig. 4). In RSA, the relationships between the activation patterns elicited for different items, given as
a pairwise dissimilarity matrix, serve as the basic
unit of measurement instead of the overall activation level.61
Dissimilarity matrices contain a vast amount of
information that can be probed using a variety of
specialized analysis techniques. Exploratory multivariate tests were among the first MVPA techniques
used to study neural representational spaces and
continue to be one of the primary methods for analyzing dissimilarity matrices.23,61 Exploratory tests
include clustering and multidimensional scaling
analyses that project the pairwise neural pattern
similarities onto a lower dimensional space. These
methods are highly useful for visualizing the representational code of a particular region and discovering which aspects of a representational space
the region codes. For example, multidimensional
scaling has been used to show that objects in the
lateral occipital cortex clustered on the basis of
shape and roughly mirrored clustering based on psychological similarity.24 Likewise, similarity spaces
of real-world objects constructed from activation
patterns in human and monkey inferior temporal
cortex revealed not only a broad category-level clustering that was consistent between species but also
fine-grained within-category relationships between
objects.25
There are also a number of confirmatory methods that can be used to test specific hypotheses about
the representational spaces that dissimilarity matrices code. One of these is to test whether there is a
significant relationship between the observed neural
dissimilarity matrices and those predicted by some
model of the underlying representational space.50,61
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 Davis & Poldrack
Representational analysis using fMRI
Stimulus Presentation
Neural Response
Activation Pattern
Dissimilarity Matrix
elephant
elephant
goat
hamster
0
low
high
goat
low
0
low
hamster
high
low
0
+
elephant
goat
e
Tim
+
hamster
A
Figure 4. An example of how changes in size could be coded in patterns of activation, such as those used by MVPA measures.
The pattern of activation across the voxels is dissimilar for elephants and hamsters but more similar for elephants and goats, or
hamsters and goats. The mean univariate activation across the ROI (all three voxels) would fail to recover the neural coding of size.
This method has been used to establish relationships between psychological and neural representational spaces by showing, for example, that neural
similarity matrices for mammals are significantly
correlated with subjects’ perceptions of similarity.62
Other confirmatory techniques can be used to
test questions about representational spaces that go
beyond the pairwise relationships between items,
such as questions about the topography of the neural representational space or how the distribution of
different properties changes across a space. One important topographical question is which items are
in regions of higher density or are more likely to be
in terms of the distribution of items across the representational space. Many theories of how psychological processes shape or arise from the structure
of a representational space emphasize how processing is influenced by the density of regions within a
representational space. For example, in the domain
of categorization, formal models posit that objects
that are more similar to other category members,
or are in regions of higher density with respect to a
psychological category space, are more typical.11,30
A nonparametric measure of the density of a representational space with respect to a specific object,
akin to kernel density estimators,14,15 can be obtained by summing the pairwise similarities between
that object and others in the task (Fig. 5). Summed
similarity is used by exemplar models to explain
typicality and familiarity effects.30 One recent study
found that a summed similarity measure of the density of neural activation space with respect to categories containing simple geometric bird stimuli significantly correlated with psychological measures of
typicality, suggesting that neural and psychological
category spaces have similar topographies.63
Classification and machine learning methods.
Classification and machine learning methods are
among the most widespread of MVPA techniques
used to study representation.56,57,64,65 The goal of
these techniques is to predict the value of a dimension or stimulus class for an object based on
the pattern of activation elicited for the object and
activation elicited for a number of training examples. For example, in the mammal space, a classifier
may be used to predict whether or not an object
is predatory based on the relationships between its
activation pattern and that elicited for other predatory and nonpredatory mammals. If a brain region
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 115
Representational analysis using fMRI
Davis & Poldrack
contains activation patterns that can be used to successfully classify these differences at a rate greater
than chance, it is often inferred that the brain region contains information that represents these categories or stimulus dimensions.
At a computational level, the basic comparisons
that classification and machine learning methods
make to predict the features or category of an object are often highly related to the methods used to
compute similarities in RSA. However, unlike RSA,
which relies on a raw similarity metric, classification and machine learning methods often employ
sophisticated algorithms for learning and emphasizing specific features of the neural activation space
(e.g., regularization techniques)66–68 or training examples that contain diagnostic information.65,69
Support vector machines (SVMs) are one type
of machine learning algorithm that can be used to
learn training examples that contain information
that is diagnostic of category membership.57,65,70
SVMs learn a classification problem by finding the
boundary between categories that maximizes the
margin (or separation) between categories. To represent this boundary and use it to classify new untrained examples, SVMs rely on a subset of training examples called support vectors, which are the
hardest-to-classify items that lie along the margin.
In the example mammal space, a trained linear-␯
SVM (␯ = 0.1) relied upon five training examples
to represent the boundary between predatory and
nonpredatory mammals (Fig. 6). In this example,
the data are linearly separable, and perfect accuracy
can be obtained over a wide range of parameter
A)
B)
C)
D)
Figure 5. Depictions of how familiarity (A) and attention
(B–D) relate to the topography of a space. (A) A nonparametric
estimate of the density of particular points (/animals) in the
116
←−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−
space can be computed from summing the pairwise similarities
between a point and the other exemplars in the space. The density of points in a multidimensional space relates to familiarity
in models of long-term memory, or typicality when summed
only across members of a class. (B) An example of how dimensional selective attention can affect the similarity between representations. By emphasizing (i.e., attending to) the dimension
size and deemphasizing the dimension predacity, the similarity
between goat and mammals that differ in size are enhanced,
whereas differences along the predacity dimension are ignored.
(C) An example of how attention to a point in space can sharpen
the similarity gradient enhancing differences between goats and
nearby stimuli (D) or lower the specificity, thereby deemphasizing differences.
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 Davis & Poldrack
Representational analysis using fMRI
guinea pig
cow
elephant
bison
rat
goat
hampster
rabbit
mouse
horse
squirrel
rhinocerous
chipmunk
deer
bear
dog
wolf
bat
fox
sea lion
tiger
dhole
honey badger
weasel
cat
lion
mongoose
leopard
Figure 6. An example of how specific instances are represented by SVMs with the goal of optimally defining a boundary that
maximizes the margin between classes. Bold objects are the support vectors. The SVM was trained to distinguish predatory from
nonpredatory mammals.
settings, but in many real-world data problems,
there will be a trade-off between the number of
training examples that are support vectors and the
number of training examples that are misclassified
or on the wrong side of the margin. High accuracy
within the training set can be obtained by increasing
the number of training examples that are support
vectors, but this can lead to overfitting of the training data and can reduce generalization to test data.
In many contexts, interpretations of basic classifier output, such as accuracy, will differ from
those of RSA because classifier accuracy emphasizes diagnostic information, whereas topographical measures, such as density or similarity to other
category members, emphasize characteristic information. For example, mammals that are accurately
classified with respect to other members of their category will tend to be those that are far from the margin or boundary between categories. In contrast, in
RSA measures of similarity to a category, mammals
that elicit patterns of activation that are like other
members of their category will be those that have
the most characteristic activation patterns. This difference occurs because, in classification, an item’s
similarity to its own category is only one piece of
the information that goes into selecting a category;
how similar an item is to other categories also affects accuracy. If an item is somewhat similar to
other categories, even though it is highly similar
to other members of its own category, it will tend
to be less accurately classified than items that are
maximally dissimilar to all categories but their own
or, equivalently, are the farthest from the margin.
Although classification accuracy is the most common measure employed in machine-learning analyses and differs from standard RSA measures in interpretation, most machine-learning methods are
highly flexible and can often be used to answer similar questions when additional measures or different
training modes are employed. For example, SVMs
can be used not only for classification and regression, but also to estimate the density of regions of
a space.71 When trained to predict stimulus identity instead of class, classifier accuracy can also be
used as a similarity metric and has been shown
to perform equivalently to raw similarity metrics
in some studies.62 Moreover, although classification techniques are supervised methods, comparisons of misclassifications (i.e., confusion matrices)
can be submitted to clustering analyses that can
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 117
Representational analysis using fMRI
Davis & Poldrack
potentially find emergent relationships between
stimuli. The strong relationship between RSA and
classification techniques suggests that RSA would
benefit from regularization techniques used in many
machine learning methods to reduce noise from irrelevant voxels.66–68 One method that has shown
some promise in reducing the impact of noise from
irrelevant voxels in RSA is Diedrichsen et al.’s pattern component model.72 However, this model becomes unwieldy in many frequent condition-rich
RSA designs due to the number of separate components that require estimation.
Encoding models. Encoding models are another
set of MVPA techniques that have been employed
to test hypotheses about how information is represented in the brain.73–76 Mathematically, encoding
models bear some resemblance to the classification
and machine learning methods more commonly
employed in fMRI analysis.76 The key difference between classification and encoding approaches is in
the direction of the mapping between the physical or
psychological representational space and the neural
space. Instead of trying to map neural similarity relationships between activation patterns onto physical or psychological stimulus features, as occurs in
classification or RSA, encoding models map a generative model of a hypothesized underlying physical
or psychological feature space onto activation patterns via a set of feature weights. Here, generative
means that the stimulus space is characterized by
some set of features that are common across stimuli
(e.g., size and predacity for mammals), which are
mixed together to generate a representation of each
observed stimulus. The obtained neural weighting
of specific features can then be used to predict patterns of activation for novel stimuli that were not
included in the test set. The generative model of
the stimulus space is taken to be a good description
of the representational content of a particular ROI
insofar as the encoding model is able to accurately
predict activation patterns elicited for new stimuli.
Applied in the example mammal space, an encoding approach would first estimate the weighting
of the generative size and predacity dimensions with
respect to each voxel (e.g., obtain a slope describing
how much each voxel responds to increases in size
or predacity) and would then use these weights to
predict activation patterns for a new mammal of a
known size and predacity. In one real-world appli118
cation, an encoding model was used to test theories
of how the early visual cortex represents images.73
By modeling the correspondence between basic image properties characterized by Gabor wavelets and
neural activation, images that subjects were viewing were reconstructed with impressive accuracy.
Another study examining neural representation of
higher-level semantic categories used an encoding
model based on semantic relationships in WordNet
to test how semantic spaces are distributed across the
cortex.26 The results supported a model whereby
real-world visual object categories are represented
in terms of a continuous semantic space that is distributed across much of the visual cortex.
Relationships between MVPA and other measures. Although the information-rich activation
vectors used in MVPA appear vastly different from
the relative activation measures used in univariate
analysis, MVPA techniques are sensitive to the same
signals that make up univariate activation measures
at the ROI level. Many distance metrics used in RSA
(e.g., Euclidean or city-block distance) will be sensitive to whether the aggregate across-voxel activation
differs between stimuli, and as we discuss later, even
measures that are not sensitive to aggregate activation (e.g., correlations) may still be sensitive to
patterns induced by activation in many real-world
contexts. All else being equal, stimuli with larger differences in aggregate activation will be more distant
than stimuli that elicit more equivalent aggregate
activation. Likewise, most classifiers, multivariate
regression methods, and encoding models allow for
the possibility that differences between categories
or along a stimulus dimension are coded solely in
the aggregate (mean univariate activation) across
an ROI. Thus, although the inputs and mathematics of the two approaches often seem wildly different, there are contexts in which the basic conclusions
reached from univariate activation and MVPA could
be the same.
Although MVPA techniques can be sensitive to
the same signals measured by univariate activation, multivoxel techniques differ from standard
univariate activation techniques in a variety of important ways and will be successful at measuring
representations in a wider number of analysis situations. Foremost, as alluded to above, because multivoxel analyses make use of patterns of activation
across voxels instead of the aggregate direction of
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 Davis & Poldrack
activation, they are able to investigate finer-grained
relationships between stimuli. This follows simply
from the basic combinatorial properties of the voxel
space. Aggregate univariate activation over a single
voxel or summed across an ROI can, at best, code
differences along a single dimension. Adding additional voxels (or not distilling activation to a single
summary for an ROI) thus increases the number of
dimensions that can potentially be encoded within
the ROI.
The combinatorial properties of MVPA provide
one potential explanation for findings suggesting
that MVPA is more sensitive to representational content than are univariate77 and adaptation methods78
that ignore the relationships between voxels. Adaptation and univariate methods will both fail to
uncover representations in any case where the relationships between stimuli are coded in a combinatorial code that depends upon multiple voxels.
However, there are also contexts in which MVPA
may be less sensitive than adaptation measures. Because standard MVPA does not explicitly take into
account adaptation responses, like univariate measures, it will fail to measure representations in contexts where an entire stimulus space is represented
within a single voxel.24,48
An additional important question is which methods are best suited toward measuring representational changes that occur over time. We know that
in many domains, like category learning, how subjects represent a stimulus space will change as they
learn.21,41 So far, studies of representational change
over time have relied primarily on model-based
studies employing univariate activation. Much of
the work so far in MVPA has either tacitly or explicitly assumed that the activation patterns elicited for
items stay constant over time, and have therefore relied upon a single time-averaged activation pattern
for each trial or stimulus. However, there have been
a number of recent efforts to extend MVPA measurements into the temporal domain to assess how
representations evolve over time,79–82 which should
increase the use of model-based methods in MVPA
research.
A final issue concerns localization or testing questions regarding which brain regions are responsible for representing a specific property. Unlike
univariate analysis, which is often conducted individually for voxels across the entire brain, early
work employing MVPA was primarily restricted to
Representational analysis using fMRI
testing hypotheses using a set of broad and predefined anatomical regions, which is ill-suited toward
making strong claims about localization. However,
in more recent MVPA research, searchlight methods
have been introduced that allow iterative testing of
multivoxel hypotheses in all possible n-voxel groupings of adjacent voxels across the brain.83 Searchlight
methods have the benefit of facilitating novel discoveries and allowing for conclusions about localization, like whole brain univariate analysis. However, the geometric properties of searchlights can
potentially distort the underlying spatial extent of a
pattern and in less predictable ways than smoothing that is carried out in univariate analysis.84
Thus, univariate methods remain the method
of choice for making strong conclusions about
the anatomical specificity of a representation or
effect.
It is important to note that there is no single
best method for investigating all of the different
questions that may be studied with representational
fMRI (Table 1). Because of the differences in the
properties of representations that the different analysis techniques are sensitive to, it may be useful to
employ designs in which a number of methods can
be used simultaneously.50
A taxonomy of representational questions
Representational analyses have been used to tackle
a host of different questions in specific content domains across psychology and neuroscience. Much of
this work was originally intended to answer pressing theoretical questions within specific content domains, and their impact is better understood by
considering more focused reviews within these areas. Here, we attempt to integrate across content
domains to develop a general framework for the
types of questions representational analyses have
been used to address and which methods may be
best equipped to answer them.
The neural localization of representational
content
Much of cognitive neuroscience so far has centered on the goal of localizing cognitive processes within particular brain regions. In this regard, representational analysis has progressed much
in the same manner as work on cognitive processes and has a general goal of localizing specific
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 119
Representational analysis using fMRI
Davis & Poldrack
representational content in different regions of the
brain.
Much of the seminal research on localizing representational content has relied on univariate activation analysis, which has been used to map the
regions that show specificity for a wide variety of
sensory domains and categories.35–39 Establishing
that an area is sensitive to a particular type of content is straightforward using univariate activation
and involves testing whether the region preferentially activates for specific content when compared
to a suitable set of controls. For example, in research establishing the fusiform face area (FFA) as
a region of the ventral visual pathway that is selective for face processing, greater activation in the FFA
was shown for faces relative to a number of control
stimuli, including objects, scrambled faces, houses,
and hands.35
Adaptation methods have also played a large role
in mapping the representational content of different regions of the brain.45–48 In adaptation studies
of representational content, the goal is to show that
repeated sequential presentations of stimuli lead to
adaptation to the extent that they share a stimulus
dimension or category. If a region or voxel is sensitive to particular content, than it should exhibit
adaptation with respect to repetitions of stimuli that
share that content, and no adaptation, or a release
from adaptation, for stimuli that do not share the
content. One study used adaptation to test which
brain regions were involved in representing the distinction between /ba/ and /da/ phonemes.84 These
phonemes show very subtle differences in the underlying physical waveform, but are treated as categorically different by English speakers, suggesting
that somewhere in the brain, the difference between
them is accentuated. They found that a number of
regions, including the supramarginal gyrus, exhibited a release from adaptation for pairs of exemplars
crossing the /ba/–/da/ boundary, but this did not
occur in early auditory cortex, suggesting that categorical processing occurs later in the processing
stream.
The advent of MVPA methods brought about a
sea change in the way that representational analysis
of content coding in the brain was conducted and
interpreted.85 Whereas univariate analysis focuses
on differences in mean signal across regions of the
cortex, MVPA focuses on the informational content
120
of activation patterns coded in different regions.
By focusing on informational content, MVPA studies have shown that regions of the brain previously
thought to encode representational spaces specific
to a certain type of representational content may
code information about a variety of object classes.
For example, Haxby et al. were among the first to
use MVPA classification techniques to study the informational content of the ventral visual stream,
finding that activation patterns in stimulus-selective
regions contained information that could also discriminate between a number of nonpreferred stimulus categories.64 Likewise, a recent multivariate
application to content coding employed RSA and
classification of activation patterns elicited for
words, sounds, faces, and scenes to investigate representational specificity in the medial temporal lobe
(MTL) cortex.86 Contrary to theories suggesting
that subregions of the MTL cortex code a face or
scene space, the analysis of the neural activation pattern space revealed that all stimulus types (words,
sounds, faces, and scenes) were discriminable and
formed distinct clusters in the neural pattern similarity space throughout the MTL cortex, even when
the stimulus types that activated particular regions
of cortex the most (faces or scenes) were removed
from the analysis.86
Encoding models may ultimately lead to additional refinements in the way that we think about
the localization of content. Classification methods
have been highly successful in establishing that different regions of the brain code information about
specific types of content, but they do not directly test
theories about the structure of the information in
particular regions. Encoding models can do this by
building a generative model of the underlying representational space into the data analysis and examining how the features of this model are distributed
across the cortex. By mapping a feature space onto
the brain, encoding models can potentially lead to
a more general description of the neural code and
explain why a particular brain region discriminates
between the categories that it does. One recent study
that employed encoding models to map the distribution of representational content across the cortex used WordNet-based87 content coding of movie
scenes to predict subjects’ activation patterns during
passive viewing.26 The results revealed that the semantic factors underlying the content coding of the
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 Davis & Poldrack
movie scenes were encoded broadly across the cortex and were preserved across individual subjects.
Explaining the structure of representational
spaces
Recent representational research using fMRI has expanded from simply mapping how different representational content is distributed throughout the
brain to questions about the manner in which information is organized.25,50,61,62 This amounts to testing specific theories of how representational spaces
coded by particular regions are structured. For reasons alluded to above, testing theories of how representations are organized is more difficult using simple classification and univariate activation methods
because these methods are largely tied to the particular content explicitly contrasted or included in
the training set. To test theories about organization,
models are needed of how the particular comparisons or contrasts between stimuli should relate to
one another.
Recent studies have used adaptation to establish
which regions code fine-grained differences between
stimuli in terms of a psychological or physical stimulus space. Instead of simply showing aggregate adaptation, these studies examine whether the degree
of adaptation between pairs of stimuli differs as a
function of the psychological or physical distance
between stimuli.50 In one recent example of the use
of adaptation as a measure of the organization of a
representational space, pairwise adaptations in the
hippocampus between landmark stimuli were significantly associated with the landmarks’ physical
distances.88
MVPA techniques used to establish whether a region codes the fine-grained relationships between
stimuli have largely relied upon RSA. In these
analyses, the pairwise similarities between activation patterns elicited for different stimuli are compared to some psychological or physical predictions
for the similarity relationships between stimuli. A
significant correlation between predicted and observed similarity relationships within a region suggests a possible isomorphism between psychological (/physical) and neural representational spaces.
In one of the first studies to employ this method,
Weber et al. found that similarity between activation patterns elicited for different mammals in
object-sensitive visual cortex predicted psychological perceptions of similarity.62 Recent follow-up
Representational analysis using fMRI
studies have found isomorphisms between the neural activation patterns elicited by viewing images
of birds and insects and perceptions of biological
similarity.89 Likewise, a variety of computational
models of vision and object perception were compared to explain the similarities between natural and
artificial objects in the inferior temporal cortex.61
More recent studies have used similar methods
to explain how psychological dimensions of simple geometric objects are coded in lateral occipital
regions,24,90 and to test predictions about coding in
the visual word form area.91 It is important to note,
however, that the potential isomorphisms revealed
by these methods are purely statistical and the relationships are only evidence of a greater than chance
relationship between similarity spaces, or a relative
advantage for one particular model of a similarity
space compared to others. Establishing a perfect isomorphism between neural and psychological spaces
would require much more stringent criteria and is
not likely, given current neuroimaging methods.
Like RSA, one of the main uses of encoding models is to test specific theories of how representations
are organized within a region. If a generative feature space that underlies an encoding model fits
the underlying neural data and generalizes to novel
training examples, it is supported as a model of the
organizing principles of the underlying neural representational space. For example, although the goal
of the study by Huth et al.26 was to map semantic
content onto the cortex, to the extent that the model
effectively accounts for the neural data, it also serves
to support the WordNet factors used as a model
of representational organization in the brain. Likewise, encoding models that have employed word
cooccurrences as a generative model in studies of
word meaning lend support for the role of word
cooccurrence in the organization of semantic representations in the brain.74
Beyond testing models of the dimensions and
relationships underlying neural representational
spaces, it is also possible to test questions related
to the topography of representational spaces and
how they align. Here, we refer to the topography of
a representational space as the manner in which different properties, like the density of observations,
vary across the space. The density of a representational space can be approximated nonparametrically by taking sums over the pairwise similarities for
each of the stimuli within the space.14,15 In a recent
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 121
Representational analysis using fMRI
Davis & Poldrack
study,63 sums of similarities between activation patterns elicited for items and other members of their
category were computed and compared to subjects’
typicality ratings, a measure thought to be related
to psychological density30 and to objective estimates
of the item’s density with respect to the physical category space. Summed similarity in regions of early
visual cortex and the ventral lateral temporal and
MTL cortex correlated significantly with subjects’
typicality ratings, suggesting that the neural and
psychological spaces have similar topographies.63
Adaptation measures have recently been hypothesized to be sensitive to longer-term information
about the organization of representations in addition to information about stimuli that are temporally adjacent.92,93 Insofar as adaptation observed
on a given trial reflects an integration of the stimuli
experienced in a task over time, longer-term components can be interpreted, like summed pattern similarities, as measures of a representational space’s topography. This is the assumption underlying recent
tests of norm-based coding in perception, whereby
objects that are central for their category are found
to elicit greater overall adaptation.92
Although testing of hypotheses about the relationships between stimuli cannot be easily accomplished using standard univariate activation measures and classification techniques, these measures
can be used to derive pairwise similarity matrices in the same way as described for adaptation
and RSA. For example, similarity matrices based
on mean signal across an ROI have been compared
to those obtained from more conventional pattern
similarity metrics (i.e., correlation and Euclidean
distance).61 Likewise, confusion matrices from classification analyses can be used as a similarity metric
and may yield comparable results to standard RSA
metrics.62
How the structure of representational space
contributes to and is shaped by cognitive
processes
Testing theories regarding the localization and organization of representational content in the brain
is one step in bridging the gap between cognitive
theory and neuroscience; however, to fully connect
the different levels of analysis, it is necessary to
explain how these neural representations are used
and influenced by cognitive processes. Many existing computational models make precise predictions
122
for how representational spaces are shaped by and
contribute to cognitive processes, and thus provide a
useful starting point for studies exploring how neural representations and cognitive processes relate.
There are a variety of cognitive mechanisms that
have been proposed to shape or emphasize dimensions in representational space. The most straightforward of these is selective attention. Selective attention is instantiated in cognitive models as weights
on particular dimensions12 or points in a representational space.94,95 By highly weighting a dimension or
location in representational space, stimuli along this
dimension or near this point in space are emphasized, whereas stimuli in other regions of space or
differences along unattended dimensions are deemphasized. For example, in the mammal space, high
selective attention to the size dimension may decrease the similarity between mammals that differ in
size (Fig. 5B). Likewise, depending upon task goals,
attentional mechanisms that emphasize points in
space may sharpen the peaks around attended points
in the representation space, thereby decreasing the
similarity between items that mismatch any features
(Fig. 5C and D).
In univariate studies, spatial selective attention
can explain how activation is enhanced for objects falling in attended regions of retinotopic space
and decreased for objects falling outside of these
regions.5,96,97 Likewise, learned dimensional selective attention may explain recent adaptation effects in the artificial category learning literature.
For example, after learning novel categories of multidimensional stimuli, the amount of adaptation
between two stimuli was found to be stronger for
stimuli that differed along a previously attended dimension (e.g., a dimension that was diagnostic during category learning) than to stimuli that differed
along an irrelevant dimension.98
Selective attention has also been a central topic in
recent MVPA studies. In terms of dimensional selective attention, several studies have found enhanced
decoding of the precise stimulus dimensions that
subjects are attending to in a visual stimulus.99–102
A study that examined stimulus-specific attention
found that after category learning, stimuli that fell
in regions of a category space that contain highly
typical and diagnostic stimuli elicited patterns of
activation that were more similar to other category
members, or equivalently, were in higher density regions of the neural category space.63 One possible
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 Davis & Poldrack
explanation that the authors suggested for this finding was that selective attention mechanisms boosted
the weighting of highly diagnostic stimuli in subjects’ category representations stored in memory.
Other cognitive processes can be modeled as
arising from a representational space as opposed
to shaping it. Working memory is thought to involve sustained activation of points in representational space. MVPA studies have found that it is
possible to decode the contents of working memory after a stimulus has been removed from a visual display,81,103–105 and when a memory has been
retrieved from long-term memory and is being
pondered.106
Long-term memory is another process that is
sensitive to the contents of the underlying representational space. Successful long-term memory is
thought to involve a reactivation of points in representational space. MVPA studies have measured reactivation by showing that it is possible to decode the
broad category that subjects are retrieving on trials
associated with successful memory107,108 and that,
compared to forgotten stimuli, activation patterns
for successfully remembered stimuli are more similar or replicable across repeated presentations.109
Beyond simple reactivation of item-specific
memory traces, formal long-term memory and categorization models predict that an object’s similarity
to other items plays a strong role in memory.30,110–113
The degree to which an objects’ representation globally matches those of all other objects stored in memory is thought to determine familiarity. In terms of a
representational space, this global similarity can be
thought of as a density measure,14,15 and depending upon the structure of the representational space
can be measured psychologically using exemplarbased summed similarity measures,30,110 or other
types of parametric and semiparametric density estimators that have been instantiated as psychological models.15 Objects in regions of higher density
with respect to subjects’ psychological representations are often found to be more familiar.30,110,114
For example, in our mammal space, mammals that
are in regions of higher density in the representational space (e.g., cat, tiger, bison, mouse) would
be predicted to be more familiar than mammals in
regions of lower density (e.g., mongoose, sea lion,
elephant, guinea pig; Fig. 5A). There have been several recent studies where density-like measures of
neural similarity spaces have been used to predict
Representational analysis using fMRI
long-term memory. For example, for broad face and
scene categories, the summed similarity or density
of an object’s activation patterns with respect to
members of its category correlated significantly with
memory success.115 Likewise, in the MTL cortex, an
item’s summed similarity to other items was positively correlated with memory judgments, whereas
in the hippocampus, similarity between items correlated negatively with memory judgments.116
Interpreting representational analysis
Many important theoretical and conceptual distinctions have yet to be fleshed out with regard to representational fMRI analysis, some of which are specific
to particular content domains while some pervade
representational analysis as a whole. One recent review elegantly covers many of the pressing philosophical questions for representational analysis,59
such as whether the voxel-wise adaptations and
distributed activation patterns measured by fMRI
have a causal role in guiding behavior or whether
they are coarser reflections of processing that is occurring at fine-grained levels (e.g., at the level of
the synapse) that are not directly measurable with
fMRI. The answer to this question may differ depending on the particular region and image resolution, and ultimately may not discount from the
utility of representational analysis for localizing and
understanding the structure or content of neural
representations.
Here, we skirt many of the deeper questions
surrounding how neuroimaging measurements are
connected to behavior at a causal level, and focus
primarily on interpretational issues, such as how
different types of signals can affect representational
analysis, and how this affects our ability to interpret
their results. To this end, one of the main challenges of representational analysis is the presence of
similarity relationships between stimuli that are not
related to the underlying stimulus representations.
Analytic confounds
One class of similarity relationships between stimuli arises based on the properties of neuroimaging
measurement or the statistical properties of techniques used to establish representations. These are
important to be aware of when designing experiments but are often possible to overcome using
well-established methods.
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 123
Representational analysis using fMRI
Davis & Poldrack
Temporal autocorrelation is a ubiquitous source
of measurement confounds in representational
analysis.117–119 Temporal autocorrelations arise in
fMRI from a number of sources, including temporally autocorrelated noise due to physiological process (respiration, heart rate) and motion; temporal
correlations in the hemodynamic response due to its
sluggish nature; and collinearity caused by preprocessing (temporal filtering) and estimation of the
hemodynamic response. For example, beta series
models120,121 that are used to estimate the multivoxel pattern elicited for individual stimuli in a task
will often have collinearity problems in estimating
temporally adjacent trials, leading to trade-offs and
correlated parameter estimates.122
Other temporal effects can occur from interactions between the structure of the representational space and temporal properties of adaptation effects, particularly in designs that seek to
tease apart long-term components of representation
(e.g., norm-based coding) from short-term adaptation effects.92,93 In some cases, it is possible to
predict greater adaptation effects for prototypical
stimuli just based on the assumption that adaptation reflects the distance between two adjacent
stimuli and without the need for long-term representations of a stimulus or category’s location in
a multidimensional space. For example, in a representational space able to be characterized by a
single prototype (e.g., only the large, low predacity mammals), the average distance between pairs
of stimuli will be shorter for prototypical stimuli
(e.g., bison) than for extreme stimuli on the peripheries (e.g., elephants). When summing over repeated presentations, because of these shorter distances, prototypical stimuli will tend to be associated with more overall adaptation even in cases
where no long-term information about the category structure is stored. All that would be required
to explain the greater adaptation for prototypical
stimuli (bison) is the basic assumption that a voxel
is sensitive to the similarity between temporally adjacent stimuli in the space and not that it stores any
long-term information about their actual locations
in the space. For example, bison will be associated
with greater overall adaptation than elephants if a
voxel codes relationships in the example mammal
space and even if all that is represented on a pair
of adjacent trials is a short-term characterization of
the pairwise similarities (Fig. 7). Whether and how
124
these temporal/sampling confounds affect activation patterns measured with MVPA remains an open
question.63
Designs that counterbalance stimulus orders
(continuous carryover, Refs. 50 and 123) or randomize presentations of stimuli and conditions
greatly reduce the possibility of temporal effects. In
the case of adjacency effects in adaptation designs,
statistical methods can be used to simultaneously
estimate long-term (norm-based) and short-term
adaptation effects.93 However, it is important to
note that because of the ubiquity of temporal autocorrelations in neuroimaging data, stimuli within
a run will almost always have some baseline similarity that stimuli from different runs do not share.
Thus, the null distribution of correlations or similarity values between activation patterns for two
stimuli within the same run will almost always be
centered above zero and will often vary systematically with the temporal distance between these stimuli within a run. For these reasons, in many cases,
the interpretability of MVPA studies, regardless of
whether it is RSA or classification, will benefit from
methods used to ensure independence between activation patterns, such as employing a leave-onerun-out cross-validation to estimate classifiers or
similarity values.124
In most cases, this recommendation to crossvalidate between runs entails using a larger number
of shorter runs124 (for additional reasons for employing a number of short runs, see Ref. 125). However, sometimes this recommendation is impractical due to the added variance that a large number of
runs can add to the estimation of the neural response
for individual stimuli. Moreover, if presentation order is sufficiently randomized or counterbalanced
across subjects, parameter-free methods like RSA
may remain unbiased for particular comparisons
and interactions. For example, if RSA were used to
compare the mean within-condition similarity for
two conditions A and B, neither would be associated
with a zero within-condition similarity if similarities were measured within run, but the difference
between conditions would still be expected to be
zero insofar as there was sufficient randomization
in the trial order between subjects. This is a critical
point for fMRI experimental design because current optimization practices often encourage using
a small number of highly optimized presentation
sequences that are repeated across subjects. In the
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 Davis & Poldrack
Representational analysis using fMRI
103
cow
elephant
89
rhinocerous
72
bison
48
goat
horse
34
Predacity
Predacity
53
cow
elephant
deer
81
goat
bison
136
89
horse
115
rhinocerous
65
deer
156
Size
Size
Figure 7. An example of how pairwise adaptation can be confounded with measures of prototype or norm-based coding in
adaptation tasks. The sum of pairwise distances between bison and all other large nonpredacious mammals (377) is substantially
shorter than for elephants and all other large nonpredacious mammals (664). This suggests that on average the amount of adaptation
should be higher for bison even if no long-term information about the category structure is represented.
case where full randomization or counterbalancing is not accomplished, there may be predictable
between-condition differences in classification
accuracy or similarity based on the presentation sequence orders alone.
Measuring representations versus processes
There are also a number of more theoretical or
conceptual issues that can arise in representational
analysis. One conceptual issue that pervades representational studies across many domains concerns
the difference between representations and processes, and whether representational analysis techniques are truly measuring representations—the
codes with which information about stimulus features, associations, and relationships are encoded in
our neural spaces—or some sort of process that operates upon or otherwise covaries with the organization or structure of these codes. Because all fMRI
methods (activation, MVPA, or adaptation) are sensitive to process-level differences between stimuli,
significant relationships between two stimuli may
arise because of representational similarity or because the stimuli engage a common process.
In many cases, the distinction between representations and processes can just be a philosopher’s
problem and break down into questions of semantics (e.g., “What do we mean by a representation?”).
Moreover, in the case of testing theories on content
coding, the distinction between processes and representations can often be inconsequential to interpretation of a study, or inseparable in practice, because
neuroimaging methods likely can only measure rep-
resentations that are in use or are being operated on
by some process. For example, in domain-specific
or vertical processing (for review see Ref. 126), such
as occurs in color vision, interpreting an activation
pattern as indicating that subjects are processing a
particular color may be functionally equivalent to
interpreting the activation pattern as a representation of color insofar as the necessary criteria for
either inference is met (see Ref. 127).
In the case of studies examining how representations are organized and operated upon by more horizontal or domain-general processes, such as longterm memory, categorization, decision making, and
executive control, it can be more critical and difficult
to rule out whether it is a process or representation
driving activation, adaptation, or pattern similarity
relationships between stimuli. Like domain-specific
processes, domain-general processes are sensitive to
the content and organization of representations and
will therefore often correlate systematically with representational relationships within a task. However,
unlike for domain-specific processes that activate
if and only if a particular type of content is being activated, the engagement of a domain-general
process does not, in and of itself, indicate that any
particular type of representational content is being activated. For example, activation in the MTL
during retrieval of a successfully encoded number
cannot be automatically interpreted as retrieval of
number information, nor can adaptation or pattern
similarity between encoding and retrieval because
the MTL does not activate selectively for the representation of number. The fact that no analysis
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 125
Representational analysis using fMRI
Davis & Poldrack
strategies automatically allow for a representational
interpretation is a critical point because the success
of MVPA and adaptation measures at uncovering
representations for domain-specific processes (e.g.,
angle and orientation tuning in early visual cortex or
shape similarity in lateral occipital complex) often
lead authors to automatically interpret results from
these analyses as inherently more representational
than results from univariate activation. There are
no analysis pipelines that automatically allow inferences about representations by virtue of the signals
that they measure (Table 1).
Even in cases of domain-specific processes, it is
important to be cognizant of how different domaingeneral processes may be engaged and impact the
similarity relationships between stimuli in a given
task. Depending on task goals, subjects may rely
on domain-general processes that are engaged to
different degrees between stimulus classes or between stimuli within the same class, leading to
changes in activation patterns due to processing differences alone. For example, color processing, while
fairly domain specific, may involve top-down affective components that differ systematically between
subjects,128 leading to activation differences or clustering of activation patterns between colors based on
affect instead of coding of actual color information
per se.
Likewise, conclusions about the tuning sensitivity
or organization of representations within a particular region can be affected by the stimulus space and
how it is represented. For example, in a representational space that can be characterized by a single
prototype structure (e.g., a space encompassed by
only large, nonpredacious mammals), stimuli that
are more prototypical will often be processed more
fluently and remembered better, whereas stimuli on
the peripheries will be processed less fluently.11,129
Representational analysis in this context may suggest that a region is tuned toward a color prototype because the central items are most activated/deactivated or more similar to other items,
but this conclusion may be tenuous because prototypicality will likely be confounded with fluency
and memory (Fig. 8A and B).
Cognizant of the interpretational problems that
engagement of common process can create in representational analyses, experimenters have devised
a variety of statistical and experimental methods
for reducing their impact. In terms of experimental
126
controls, one common technique for reducing the
impact of processes is to measure neural representations during tasks that are orthogonal to the underlying stimuli space. For example, subjects could
be engaging in a secondary task while being presented with, or asked questions about, stimuli that
are not related to the representational aspects being
studied.
How effective experimental controls are at removing psychological processes is a matter of debate. Theoretically, unsupervised learning processes
that are geared toward learning the statistical properties of the stimulus space will be engaged at all
times,130,131 even when the statistical regularities are
orthogonal to the task that subjects are asked to do.
This could lead to increased fluency or memory for
stimuli in high density regions of the space (e.g., the
prototypical stimuli in the mammal space), making it difficult to conclude that similarity analysis
is not picking up on some process-level similarities
between stimuli.
Perhaps the best experimental solution is to use
experimental designs in which the coding of content
can be easily established (stimulus sets that have a
well-defined representational space) and where any
likely underlying processes are not fully confounded
with the dimensions of the space. By showing that
an ROI distinguishes between stimuli with different
content but equivalent memory or fluency, it is more
likely that activation, adaptation, or the similarity
relationships between stimuli within the ROI are not
being solely driven by engagement of common processes. For example, to rule out processing fluency
as the sole cause of the higher summed similarity
for typical category members, a recent study compared the similarity between highly typical members of the same category to the similarity between
highly typical members of different categories.63
Highly typical items of different categories should
be processed as fluently but represented distinctly
if a region is sensitive to featural or category-level
information. Within-category similarity was greater
than between-category similarity for the highly typical items, suggesting that fluency alone was not
the cause of the heightened summed similarity for
typical category members and that the neural activation patterns contained some information about
the stimulus features or category.
A second set of proposed solutions for reducing the interpretational problems associated with
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 Davis & Poldrack
Representational analysis using fMRI
A)
B)
C)
D)
Figure 8. An illustration of how processes can be confounded with representational similarities between stimuli. Pairwise distances
between objects can be used to compute a measure of density, which is hypothesized by formal cognitive models to give rise to
familiarity. In the case of a single-category prototype task (e.g., only large nonpredator mammals; Fig. 7), the pairwise distances
between objects (A) in the representational space are correlated (r = 0.37) with pairwise differences in density (B), suggesting that
any correlation between the pairwise distance matrix and a neural dissimilarity matrix could be due to familiarity processes (i.e.,
process-level similarity) as opposed to representational similarity per se. However, in the larger multiple-category space, pairwise
distances in the representational space (C) are well balanced with respect to density/familiarity, leading to almost zero correlation
(r = −0.006).
engagement of common processes is to try to remove
the effect of processes statistically. These statistical
solutions are often based on ad hoc assumptions
about the sensitivities of various analysis pipelines
to representational or process-level information and
are likely tenuous at best. For example, in MVPA
studies of representation, it has become common
to remove the mean from an ROI in an attempt
to make the analysis orthogonal to univariate activation analysis.61 As a control for processes, the
assumption is that univariate activation is more sen-
sitive to processes whereas multivoxel patterns give
information about representations.
On a theoretical level, removing univariate activation seems to go against many of the findings
reviewed in this paper suggesting that univariate
activation can be interpreted as a measure of representation in select contexts. Further, given current knowledge of the brain, there is no reason to
suspect that processes may not produce distributed
multivariate effects within an ROI. On a statistical level, this solution also only works to remove
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 127
Representational analysis using fMRI
Davis & Poldrack
univariate activation under very select circumstances where the impact of activation within a region can be sufficiently described with a single mean.
This is almost never the case in neuroimaging analysis because univariate activation often manifests as
a spatial mixture within any given ROI; there will be
voxels that are unresponsive, and those that are responsive, to different degrees. Subtracting the mean
will leave the general shape of the spatial activation
pattern within ROIs that contain any type of mixture or variability between voxels, and these patterns
will continue to affect multivoxel analysis. For these
reasons, we recommend against using centering or
controlling for mean activation level in an ROI as a
rhetorical tool for making multivariate and univariate analyses orthogonal or ruling out the impact of
processes. However, there may be select cases where
removal of these signals is theoretically or empirically justified for other reasons.
In other cases, it might be possible to make a
principled model of how a psychological process
will affect activation within a region so as to remove activation patterns or factors associated with
these processes prior to representational analysis or
to control for the predicted effects of a process by
using multiple regression. For example, in studies
of how the density or topography of a representational space relates to long-term memory, it may
be possible to remove factors or activation patterns
associated with long-term memory prior to modeling the representational topography. Although this
would be a more principled way of removing the
effect of processes than controlling for or removing
univariate activation, in practice it may be difficult, and process-level information may remain in
the residuals if the processes are mismodeled. Further, omitting a number of processes from data may
render the resulting activation patterns no longer
interpretable and potentially may remove real representational information.
Although an inability to rule out the influence
of simultaneous processes in any particular design
precludes making strong representational conclusions, there are many contexts in which useful results can still be obtained and a representational
interpretation will be useful for guiding new research. One case where representational analysis can
yield insightful information, even when the influence of processes cannot be fully ruled out, occurs
in model-based approaches, particularly when al128
ternative models are directly being compared. In
the case of model-based analysis, showing that the
neural data and predictions from a model share a
relationship may yield support for the model even
if process-level differences are partially responsible for the relationships. For example, the finding
that taking into account information about the semantic relationships between words improves the
performance of neural decoding74 lends support to
the notion that these relationships are represented
somehow in the brain, even if the relationships
are measured via the engagement of common processes between words (e.g., similar affective engagement). Similarly, in recent studies on the structure
of category representation,41 it was possible to dissociate cluster-based representational models from
standard exemplar representations using univariate activation because, in the design employed, the
models predict different patterns of engagement for
mnemonic processes across stimuli within the task.
Because the patterns of engagement predicted by
the models were unique, the greater correlation between a cluster-based model’s predictions and univariate activation in the MTL lent support to the
model’s representations even though the activation
itself may only be measuring processes. Generally
speaking, the more models compared in a study, the
stronger the conclusions that can be drawn about
the processes and representations underlying neural activation patterns. A simple greater-than-zero
correlation between the predictions of a model and
neural activation patterns may not always admit
strong conclusions in and of itself.
Dissociating process-level and representational
accounts is a difficult conceptual problem. Researchers should be careful to design studies so
that psychological and physical relationships between stimuli are well defined, and where it is
likely that processes and representations are not fully
confounded. For example, in well-balanced, multicategory tasks (e.g., the full mammal space), processes like familiarity are often uncorrelated with
the dimensions of the representational space because there will be objects in each of the categories
that are highly familiar (Fig. 8C and D). By showing
that activation patterns within a given region behave
like representations and discriminate between stimuli that are representationally distinct but processed
similarly, it is more straightforward to conclude
that the analysis is providing information about
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 Davis & Poldrack
neural representations. However, assuming that any
analysis technique is measuring representations by
virtue of the methods used (univariate, multivariate,
adaptation) is not tenable.
Wiggle room
There is a growing awareness in neuroimaging that
the flexibility in analysis pipelines and hypotheses
that can be addressed with a data set can lead to
large numbers of false positives.132–134 Thus, when
evaluating the results of representational analyses,
it is worthwhile to consider whether the observed
relationships between stimuli are a function of the
wiggle room available in representational analysis.
Here, we covered three basic methods of establishing
neural representation with neuroimaging, within
each of which are a plethora of different decisions
researchers can make. Univariate analyses can often afford a number of independent data processing
pipelines in terms of preprocessing steps taken, the
specific statistical models used, and choice of statistical thresholding. MVPA compounds the number
of choices that can be made by inheriting all of
the options from univariate analysis and introducing a large number of potential classification algorithms for machine learning methods, in addition to
a variety of similarity metrics, clustering algorithms,
and projections methods that can be used in RSA.
For example, in only a few years, researchers have
employed an impressive number of similarity metrics for RSA, including Euclidean distance,23,61 Mahalanobis distance,25 correlation distance,24,50,61,63
classification confusions,62 and summed squared
distance.62
Given the cost of fMRI data and the large number of ways in which representations could be coded
within an ROI, there is a temptation to try to informally learn from the data what analyses techniques
are best for modeling representations in the task.
In the context of MVPA, such informal learning, if
done on even a moderate scale, will undoubtedly
lead to large numbers of false positives. To avoid
these, researchers should make principled decisions
about how to approach representational questions
in their data set prior to conducting analyses and
present these decisions in a written analysis plan.
Reviewers and journal editors can help in this regard by encouraging authors to make the rationale
for their decisions explicit and by limiting the num-
Representational analysis using fMRI
ber of parallel analyses that ostensibly target the
same question.
Frustratingly, however, even for univariate analysis, there is little agreement on which analysis decisions are the right ones, and there may not even
be a single answer to which classifiers, similarity
metrics, or processing pipelines are the best overall.
What constitutes a principled analysis plan can often
be a matter of opinion, and sticking with arbitrary
decisions can lead to false negatives. As an alternative to strong planning, replication and nested
cross-validation methods can be used to build the
best model possible for the observed data in a given
experiment and then test this model on independent data sets (for an introduction to model selection and validation strategies, see Ref. 135). Coupled
with the within-run autocorrelations discussed earlier, this suggests that researchers should use a large
number of short runs for representational analysis,
when possible, instead of small numbers of long
runs. This recommendation is purely pragmatic; it
is easier to hold data out and ensure that model selection and validation are independent with larger
numbers of runs.
The future of representational analysis
There are a number of questions yet to be addressed in representational analyses. Many of these
involve straightforward extensions of the methods
discussed here to build ever stronger ties between
the way in which fMRI data is analyzed and formal
cognitive theory. However, as more content domains
become interested in representational analysis, new
questions will undoubtedly arise.
One pertinent question is how the spatial model
of neural representation that has driven much of the
work that we cite and has provided a backbone for
the present review will be able to interface with questions in higher-level cognition where spatial forms
of knowledge representation are often insufficient
or the axioms underlying the theory of representational spaces have been violated.31,136 For example, in explaining higher-level reasoning, structured
representations, such as graphs, decision trees, and
predicate calculus, often provide a better account
of behavior than spatial forms of knowledge representation. An important question is whether the
spatial measures used in neuroimaging will provide
a useful understanding of how the brain represents
information in these contexts.
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 129
Representational analysis using fMRI
Davis & Poldrack
One possibility is that, at the level of neuroimaging analysis, these finer-grained logical relationships
become more amenable to measurement with spatial metrics. In this respect, neuroimaging similarity
measures may bear resemblance to corpus similarity measures, such as latent semantic analysis or
Google similarity.60,137 Word cooccurrences across
Web pages may reflect broad similarity spaces that
are easily amenable to spatial analysis, but the use
of two words on any specific Web page may be better represented by a more structured representation.
For example, across many Web pages, the word bison
will appear in a variety of contexts, and its similarity to different words can be captured with statistical distributions. However, a single Web page may
only discuss bison in relation to a specific topic (e.g.,
the U.S. government’s sanctioning of the widespread
slaughter of bison herds to force native peoples onto
reservations). This specific similarity relationship
between the U.S. government and bison cannot simply be expressed as a distance between two points in
a representational space. Likewise, at the voxel level,
activation may measure a more continuous aggregate of the different relationships that words bring
to mind, whereas specific structured relationships
between words may be coded at the neuronal level.
Another question that deserves consideration as
representational fMRI progresses is the extent to
which all of the content domains in cognitive neuroscience that are currently exploring representational questions each need to postulate their own
separate representations and processes, or whether
theoretical consolidation is required to prevent unnecessary proliferation of representational systems
in cognitive neuroscience theory. Although it is now
generally agreed that the structure of the environment is too impoverished to explain behavior and
thought without some sort of representational capacity (for review, see Ref. 138), it will be important
to ensure that cognitive neuroscience does not inadvertently fulfill the prophesies of behaviorists and
Gibsonians who warned against postulating excessive representations and ignoring interactions with
the environment altogether. For example, one important question is whether the representations that
we form to encode long-term memories and learn
new categories are specific to these cognitive processes, or whether they may be combinations of
representations coded in lower-level perceptual areas that are joined together using top-down pro130
cesses based on the affordances of the environment.
In terms of categorization, early neurobiological research suggested a category learning system in early
visual cortex operated via priming and was fundamentally separate from a rule-based categorization
system in the prefrontal cortex,7 but recent research
suggests that these systems may work together such
that rules instantiated in the prefrontal cortex work
in a top-down fashion to emphasize diagnostic stimulus representations in early visual cortex.63 In the
future, it will be critical to continue to integrate
theories across cognitive domains to avoid creating
more representational systems than are necessary to
explain cognitive and neural results.
In conclusion, the major goal of cognitive
neuroscience—to connect cognitive theory and
neural function—depends not only on our ability to understand how processes are instantiated in
the brain, but also on our ability to answer questions about the nature of the representations upon
which these processes depend. There are a number
of analysis techniques available to neuroimaging researchers that have shown potential in their ability to
answer fundamental questions about not only where
and how representations are coded in the brain, but
also how they are harnessed by cognitive processes
to guide behavior. Each of these techniques makes
somewhat different assumptions about the underlying neural code, and thus each may be a useful
measure of representation in different contexts. By
developing an understanding of the assumptions
that different analysis techniques make, in addition
to identifying their strengths and weaknesses for
specific representational questions, it will be possible for the field to continue to make headway in
discovering how the brain makes cognition possible.
Conflicts of interest
The authors declare no conflicts of interest.
References
1. Marr, D. 1982. Vision: A Computational Investigation into
the Human Representation and Processing of Visual Information. New York, NY: Henry Holt and Co.
2. Simons, J.S. & H.J. Spiers. 2003. Prefrontal and medial
temporal lobe interactions in long-term memory. Nat. Rev.
Neurosci. 4: 637–648.
3. Eichenbaum, H., A.R. Yonelinas & C. Ranganath. 2007. The
medial temporal lobe and recognition memory. Annu. Rev.
Neuro. 30: 123.
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 Davis & Poldrack
4. Corbetta, M. & G.L. Shulman. 2002. Control of goaldirected and stimulus-driven attention in the brain. Nat.
Rev. Neurosci. 3: 215–229.
5. Kastner, S. & L.G. Ungerleider. 2000. Mechanisms of visual
attention in the human cortex. Annu. Rev. Neuro. 23: 315–
341.
6. Poldrack, R.A. & K. Foerde. 2008. Category learning and
the memory systems debate. Neurosci. Biobehav. Rev. 32:
197–205.
7. Smith, E.E. & M. Grossman. 2008. Multiple systems of
category learning. Neurosci. Biobehav. Rev. 32: 249–264.
8. Seger, C.A. & E.K. Miller. 2010. Category learning in the
brain. Annu. Rev. Neuro. 33: 203–219.
9. Rissman, J. & A.D. Wagner. 2012. Distributed representations in memory: insights from functional brain imaging.
Annu. Rev. Psychol. 63: 101–128.
10. Ashby, F.G. & R.E. Gott. 1988. Decision rules in the perception and categorization of multidimensional stimuli. J.
Exp. Psychol. Learn Mem. Cogn. 14: 33–53.
11. Rosch, E. & C.B. Mervis. 1975. Family resemblances: studies
in the internal structure of categories. J. Cogn. Psychol. 7:
573–605.
12. Nosofsky, R.M. 1986. Attention, similarity, and the
identification–categorization relationship. J. Exp. Psychol.
Gen. 115: 39.
13. Love, B.C., D.L. Medin & T.M. Gureckis. 2004. SUSTAIN:
a network model of category learning. Psychol. Rev. 111:
309–322.
14. Ashby, F.G. & L.A. Alfonso-Reese. 1995. Categorization as
probability density estimation. J. Math. Psychol. 39: 216–
233.
15. Rosseel, Y. 2002. Mixture models of categorization. J. Math.
Psychol. 46: 178–210.
16. Davachi, L. 2006. Item, context and relational episodic encoding in humans. Curr. Opin. Neurobiol. 16: 693–700.
17. Diana, R.A., A.P. Yonelinas & C. Ranganath. 2007. Imaging recollection and familiarity in the medial temporal
lobe: a three-component model. Trends Cogn. Sci. 11:
379–386.
18. Squire, L.R., J.T. Wixted & R.E. Clark. 2007. Recognition
memory and the medial temporal lobe: a new perspective.
Nat. Rev. Neurosci. 8: 872–883.
19. Norman, K.A. & R.C. O’Reilly. 2003. Modeling hippocampal and neocortical contributions to recognition memory:
a complementary-learning-systems approach. Psychol. Rev.
110: 611–645.
20. Barsalou, L.W. 1999. Perceptual symbol systems. Behav.
Brain Sci. 22: 577–660.
21. Love, B.C. 2005. Environment and goals jointly direct category acquisition. Curr. Dir. Psychol. Sci. 14: 195–199.
22. Quiroga, R.Q., L. Reddy, G. Kreiman, et al. 2005. Invariant
visual representation by single neurons in the human brain.
Nature 435: 1102–1107.
23. Edelman, S., K. Grill-Spector, T. Kushnir & R. Malach.
1998. Toward direct visualization of the internal shape representation space by fMRI. Psychobiology 26: 309–321.
24. Drucker, D.M. & G.K. Aguirre. 2009. Different spatial scales
of shape similarity representation in lateral and ventral
LOC. Cereb. Cortex 19: 2269–2280.
Representational analysis using fMRI
25. Kriegeskorte, N., M. Mur, D.A. Ruff, et al. 2008. Matching categorical object representations in inferior temporal
cortex of man and monkey. Neuron 60: 1126–1141.
26. Huth, A.G., S. Nishimoto, A.T. Vu & J.L. Gallant. 2012. A
continuous semantic space describes the representation of
thousands of object and action categories across the human
brain. Neuron 76: 1210–1224.
27. Edelman, S. 1998. Representation is representation of similarities. Behav. Brain Sci. 21: 449–467.
28. Gärdenfors, P. 2004. Conceptual Spaces: The Geometry of
Thought. MIT Press. Cambridge, MA.
29. Shepard, R.N. 1987. Toward a universal law of generalization for psychological science. Science 237: 1317–1323.
30. Nosofsky, R.M. 1988. Exemplar-based accounts of relations
between classification, recognition, and typicality. J. Exp.
Psychol. Learn Mem. Cogn. 14: 700–708.
31. Tversky, A. 1977. Features of similarity. Psychol. Rev. 84:
327–352.
32. Smith, E.E., E.J. Shoben & L.J. Rips. 1974. Structure and
process in semantic memory: a featural model for semantic
decisions. Psychol. Rev. 81: 214–241.
33. Friston, K.J., P. Jezzard & R. Turner. 1994. Analysis of functional MRI time-series. Hum. Brain Mapp. 1: 153–171.
34. Worsley, K.J., C.H. Liao, J. Aston, et al. 2002. A general
statistical analysis for fMRI data. Neuroimage 15: 1–15.
35. Kanwisher, N., J. McDermott & M.M. Chun. 1997. The
fusiform face area: a module in human extrastriate cortex
specialized for face perception. J. Neuro. 17: 4302–4311.
36. Epstein, R., A. Harris, D. Stanley & N. Kanwisher. 1999.
The parahippocampal place area: recognition, navigation,
or encoding? Neuron 23: 115–125.
37. Lotze, M., M. Erb, H. Flor, et al. 2000. fMRI evaluation
of somatotopic representation in human primary motor
cortex. Neuroimage 11: 473–481.
38. Shmuelof, L. & E. Zohary. 2005. Dissociation between ventral and dorsal fMRI activation during object and action
recognition. Neuron 47: 457–470.
39. O’Doherty, J.P. 2004. Reward representations and rewardrelated learning in the human brain: insights from neuroimaging. Curr. Opin. Neurobiol. 14: 769–776.
40. Forstmann, B.U., E.J. Wagenmakers, T. Eichele, et al. 2011.
Reciprocal relations between cognitive neuroscience and
formal cognitive models: opposites attract? Trends Cogn.
Sci. 15: 272–279.
41. Davis, T., B.C. Love & A.R. Preston. 2012. Learning the
exception to the rule: model-based fMRI reveals specialized
representations for surprising category members. Cereb.
Cortex. 22: 260–273.
42. Davis, T., B.C. Love & A.R. Preston. 2012. Striatal and
hippocampal entropy and recognition signals in category
learning: simultaneous processes revealed by model-based
fMRI. J. Exp. Psychol. Learn Mem. Cogn. 38: 821–839.
43. White, C.N. & R.A. Poldrack. 2013. Using fMRI to constrain
theories of cognition. Perspect. Psychol. Sci. 8: 79–83.
44. Daw, N.D. & K. Doya. 2006. The computational neurobiology of learning and reward. Curr. Opin. Neurobiol. 16:
199–204.
45. Kourtzi, Z. & N. Kanwisher. 2000. Cortical regions involved
in perceiving object shape. J. Neuro. 20: 3310–3318.
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 131
Representational analysis using fMRI
Davis & Poldrack
46. Grill-Spector, K. & R. Malach. 2001. fMR-adaptation: a tool
for studying the functional properties of human cortical
neurons. Acta Psychol. 107: 293–321.
47. Kourtzi, Z. & N. Kanwisher. 2001. Representation of perceived object shape by the human lateral occipital complex.
Science 293: 1506–1509.
48. Malach, R. 2012. Targeting the functional properties of
cortical neurons using fMR-adaptation. Neuroimage 62:
1163–1169.
49. Desimone, R. 1996. Neural mechanisms for visual memory
and their role in attention. Proc. Natl. Acad. Sci. U. S. A. 93:
13494–13499.
50. Aguirre, G.K. 2007. Continuous carry-over designs for
fMRI. Neuroimage 35: 1480–1494.
51. Tolias, A.S., Z. Kourtzi & N.K. Logothetis. 2003. Functional magnetic resonance imaging adaptation: a technique for studying the properties of neuronal networks. In
Exploratory analysis and data modeling in functional neuroimaging, F.T. Somner and A. Wichert (eds). pp. 109–125.
MIT Press Cambridge MA.
52. Sawamura, H., G.A. Orban & R. Vogels. 2006. Selectivity of
neuronal adaptation does not match response selectivity: a
single-cell study of the FMRI adaptation paradigm. Neuron
49: 307–318.
53. Summerfield, C., E.H. Trittschuh, J.M. Monti, et al. 2008.
Neural repetition suppression reflects fulfilled perceptual
expectations. Nat. Neurosci. 11: 1004–1006.
54. Mur, M., D.A. Ruff, J. Bodurka, et al. 2010. Face-identity
change activation outside the face system: “release from
adaptation” may not always indicate neuronal selectivity.
Cereb. Cortex 20: 2027–2042.
55. Larsson, J. & A.T. Smith. 2012. FMRI repetition suppression: neuronal adaptation or stimulus expectation? Cereb.
Cortex 22: 567–576.
56. Norman, K.A., S.M. Polyn, G.J. Detre & J.V. Haxby. 2006.
Beyond mind-reading: multi-voxel pattern analysis of fMRI
data. Trends Cogn. Sci. 10: 424–430.
57. Cox, D.D. & R.L. Savoy. 2003. Functional magnetic resonance imaging (fMRI)“brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual
cortex. Neuroimage 19: 261–270.
58. Mur, M., P.A. Bandettini & N. Kriegeskorte. 2009. Revealing
representational content with pattern-information fMRI—
an introductory guide. Soc. Cogn. Affect Neurosci. 4: 101–
109.
59. Raizada, R.D. & N. Kriegeskorte. 2010. Pattern information
fMRI: new questions which it opens up and challenges
which face it. Int. J. Imaging Syst. Technol. 20: 31–41.
60. Landauer, T.K. & S.T. Dumais. 1997. A solution to Plato’s
problem: the latent semantic analysis theory of acquisition,
induction, and representation of knowledge. Psychol. Rev.
104: 211–240.
61. Kriegeskorte, N., M. Mur & P. Bandettini. 2008. Representational similarity analysis—connecting the branches
of systems neuroscience. Front. Syst. Neurosci. 2: 4.
62. Weber, M., S.L. Thompson-Schill, D. Osherson, et al. 2009.
Predicting judged similarity of natural categories from their
neural representations. Neuropsychologia 47: 859–868.
132
63. Davis, T. & R.A. Poldrack. 2013. Quantifying the internal
structure of categories using a neural typicality measure.
Cereb. Cortex. doi: 10.1093/cercor/bht014.
64. Haxby, J.V., M.I. Gobbini, M.L. Furey, et al. 2001. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293: 2425–2430.
65. Formisano, E., F. De Martino & G. Valente. 2008. Multivariate analysis of fMRI time series: classification and regression of brain responses using machine learning. Magn.
Reson. Imaging 26: 921–934.
66. Yamashita, O., M.A. Sato, T. Yoshioka, et al. 2008. Sparse
estimation automatically selects voxels relevant for the decoding of fMRI activity patterns. Neuroimage 42: 1414–
1429.
67. Ng, B., A. Vahdat, G. Hamarneh & R. Abugharbieh. 2010.
Generalized sparse classifiers for decoding cognitive states
in fMRI. Mach. Learn. Med. Imag. 6357: 108–115.
68. Ryali, S., K. Supekar, D.A. Abrams & V. Menon. 2010. Sparse
logistic regression for whole brain classification of fMRI
data. Neuroimage 51: 752.
69. Pereira, F., T. Mitchell & M. Botvinick. 2009. Machine learning classifiers and fMRI: a tutorial overview. Neuroimage
45: S199–S209.
70. Hanson, S.J. & Y.O. Halchenko. 2008. Brain reading using
full brain support vector machines for object recognition:
there is no “face” identification area. Neural Comput. 20:
486–503.
71. Mukherjee, S. & V. Vapnik. 1999. Support vector method
for multivariate density estimation. Center for Biological
and Computational Learning. Department of Brain and
Cognitive Sciences, MIT. CBCL, 170. http://cbcl.csail.
mit.edu.ezproxy.lib.utexas.edu/projects/cbcl/publications/
ps/dense_memo.ps.
72. Diedrichsen, J., G.R. Ridgway, K.J. Friston & T. Wiestler.
2011. Comparing the similarity and spatial structure of
neural representations: a pattern-component model. Neuroimage 55: 1665–1678.
73. Kay, K.N., T. Naselaris, R.J. Prenger & J.L. Gallant. 2008.
Identifying natural images from human brain activity. Nature 452: 352–355.
74. Mitchell, T.M., S.V. Shinkareva, A. Carlson, et al. 2008. Predicting human brain activity associated with the meanings
of nouns. Science 320: 1191–1195.
75. Naselaris, T., K.N. Kay, S. Nishimoto & J.L. Gallant. 2011.
Encoding and decoding in fMRI. Neuroimage 56: 400–410.
76. Naselaris, T., R.J. Prenger, K.N. Kay, et al. 2009. Bayesian reconstruction of natural images from human brain activity.
Neuron 63: 902–915.
77. Jimura, K. & R.A. Poldrack. 2011. Analyses of regionalaverage activation and multivoxel pattern information tell
complementary stories. Neuropsychologia 50: 544–552.
78. Sapountzis, P., D. Schluppeck, R. Bowtell & J.W. Peirce.
2010. A comparison of fMRI adaptation and multivariate
pattern classification analysis in visual cortex. Neuroimage
49: 1632–1640.
79. Mourao-Miranda, J., K.J. Friston & M. Brammer. 2007.
Dynamic discrimination analysis: a spatial-temporal SVM.
Neuroimage 36: 88–99.
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 Davis & Poldrack
80. Soon, C.S., M. Brass, H.J. Heinze & J.D. Haynes. 2008.
Unconscious determinants of free decisions in the human
brain. Nat. Neurosci. 11: 543–545.
81. Harrison, S.A. & F. Tong. 2009. Decoding reveals the contents of visual working memory in early visual areas. Nature
458: 632–635.
82. Turner, B.O., J.A. Mumford, R.A. Poldrack & F.G. Ashby.
2012. Spatiotemporal activity estimation for multivoxel
pattern analysis with rapid event-related designs. Neuroimage 62: 1429–1438.
83. Kriegeskorte, N., R. Goebel & P. Bandettini. 2006.
Information-based functional brain mapping. Proc. Natl.
Acad. Sci. U. S. A. 103: 3863–3868.
84. Raizada, R.D. & R.A. Poldrack. 2007. Selective amplification of stimulus differences during categorical processing
of speech. Neuron 56: 726–740.
85. Haxby, J.V. 2012. Multivariate pattern analysis of fMRI: the
early beginnings. Neuroimage 62: 852–855.
86. Liang, J.C., A.D. Wagner & A.R. Preston. 2012. Content
representation in the human medial temporal lobe. Cereb.
Cortex 23: 80–96.
87. Miller, G.A., R. Beckwith, C. Fellbaum, D. Gross, & K.J.
Miller. 1990. Introduction to wordnet: An on-line lexical
database. Int. J. Lexicography 3: 235–244.
88. Morgan, L.K., S.P. MacEvoy, G.K. Aguirre & R.A. Epstein.
2011. Distances between real-world locations are represented in the human hippocampus. J. Neurosci. 31: 1238–
1245.
89. Connolly, A.C., J.S. Guntupalli, J. Gors, et al. 2012. The
representation of biological classes in the human brain. J.
Neurosci. 32: 2608–2618.
90. Op de Beeck, H.P., K. Torfs & J. Wagemans. 2008. Perceived
shape similarity among unfamiliar objects and the organization of the human object vision pathway. J. Neurosci.
2840: 10111–10123.
91. Braet, W., J. Wagemans & H.P. Op de Beeck. 2011. The visual
word form area is organized according to orthography.
Neuroimage 59: 2751–2759.
92. Panis, S., J. Wagemans & H.P. Op de Beeck. 2011.
Dynamic norm-based encoding for unfamiliar shapes
in human visual cortex. J. Cogn. Neurosci. 23:
1829–1843.
93. Kahn, D.A. & G.K. Aguirre. 2012. Confounding of normbased and adaptation effects in brain responses. Neuroimage 60: 2294–2299.
94. Sakamoto, Y., T. Matsuka & B.C. Love. 2004. Dimensionwide vs. exemplar-specific attention in category learning
and recognition. In Proceedings of the 6th International Conference of Cognitive Modeling. 261–266. Mahwah, NJ, US:
Lawrence Erlbaum Associates Publisher.
95. Rodrigues, P.M. & J.M. Murre. 2007. Rules-plus-exception
tasks: a problem for exemplar models? Psychon. Bull. Rev.
14: 640–646.
96. Kastner, S., P. De Weerd, R. Desimone & L.G. Ungerleider.
1998. Mechanisms of directed attention in the human extrastriate cortex as revealed by functional MRI. Science 282:
108–111.
Representational analysis using fMRI
97. Brefczynski, J.A. & E.A. DeYoe. 1999. A physiological correlate of the ‘spotlight’ of visual attention. Nat. Neurosci. 2:
370–374.
98. Folstein, J.R., T.J. Palmeri & I. Gauthier. 2012. Category learning increases discriminability of relevant object dimensions in visual cortex. Cereb. Cortex 23:
814–823.
99. Kamitani, Y. & F. Tong. 2005. Decoding the visual and
subjective contents of the human brain. Nat. Neurosci. 8:
679–685.
100. Haynes, J.D. & G. Rees. 2005. Predicting the orientation
of invisible stimuli from activity in human primary visual
cortex. Nat. Neurosci. 8: 686–691.
101. MacEvoy, S.P. & R.A. Epstein. 2009. Decoding the representation of multiple simultaneous objects in human occipitotemporal cortex. Curr. Bio. 19: 943–947.
102. Serences, J.T., S. Saproo, M. Scolari, et al. 2009. Estimating
the influence of attention on population codes in human
visual cortex using voxel-based tuning functions. Neuroimage 44: 223–231.
103. Ester, E.F., J.T. Serences & E. Awh. 2009. Spatially
global representations in human primary visual cortex
during working memory maintenance. J. Neurosci. 29:
15258–15265.
104. Reddy, L., N. Tsuchiya & T. Serre. 2010. Reading the mind’s
eye: decoding category information during mental imagery.
Neuroimage 50: 818–825.
105. Serences, J.T., E.F. Ester, E.K. Vogel & E. Awh. 2009.
Stimulus-specific delay activity in human primary visual
cortex. Psychol. Sci. 20: 207–214.
106. Lewis-Peacock, J.A. & B.R. Postle. 2008. Temporary activation of long-term memory supports working memory. J.
Neurosci. 28: 8765–8771.
107. Kuhl, B.A., J. Rissman, M.M. Chun & A.D. Wagner. 2011.
Fidelity of neural reactivation reveals competition between
memories. Proc. Natl. Acad. Sci. U. S. A. 108: 5903–5908.
108. Zeithamova, D., A.L. Dominick & A.R. Preston. 2012. Hippocampal and ventral medial prefrontal activation during
retrieval-mediated learning supports novel inference. Neuron 75: 168–179.
109. Xue, G., Q. Dong, C. Chen, et al. 2010. Greater neural
pattern similarity across repetitions is associated with better
memory. Science 330: 97–101.
110. Nosofsky, R.M. 1991. Tests of an exemplar model for relating perceptual classification and recognition memory. J.
Exp. Psychol. Hum. Percept. Perform. 17: 3–27.
111. Gillund, G. & R.M. Shiffrin. 1984. A retrieval model for
both recognition and recall. Psychol. Rev. 91: 1.
112. Hintzman, D.L. 1988. Judgments of frequency and recognition memory in a multiple-trace memory model. Psychol.
Rev. 95: 528–551.
113. Shiffrin, R.M. & M. Steyvers. 1997. A model for recognition memory: REM—retrieving effectively from memory.
Psychon. Bull. Rev. 4: 145–166.
114. Sakamoto, Y. & B.C. Love. 2004. Schematic influences on
category learning and recognition memory. J. Exp. Psychol.
Gen. 133: 534–553.
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134 133
Representational analysis using fMRI
Davis & Poldrack
115. Kuhl, B.A., J. Rissman & A.D. Wagner. 2012. Multi-voxel
patterns of visual category representation during episodic
encoding are predictive of subsequent memory. Neuropsychologia 50: 458–469.
116. LaRocque, K.F., M.E. Smith, V.A. Carr, et al. 2013. Global
similarity and pattern separation in the human medial
temporal lobe predict subsequent memory. J. Neurosci. 33:
5466–5474.
117. Weisskoff, R.M., J. Baker, J. Belliveau, et al. 1993. Power
spectrum analysis of functionally-weighted MR data:
what’s in the noise. Proc. Soc. Magn. Reson. Med. 1: 7.
118. Purdon, P.L. & R.M. Weisskoff. 1998. Effect of temporal
autocorrelation due to physiological noise and stimulus
paradigm on voxel-level false-positive rates in fMRI. Hum.
Brain Mapp. 6: 239–249.
119. Woolrich, M.W., B.D. Ripley, M. Brady & S.M. Smith. 2001.
Temporal autocorrelation in univariate linear modeling of
FMRI data. Neuroimage 14: 1370–1386.
120. Rissman, J., A. Gazzaley & M. D’Esposito. 2004. Measuring
functional connectivity during distinct stages of a cognitive
task. Neuroimage 23: 752–763.
121. Mumford, J.A., B.O. Turner, F.G. Ashby & R.A. Poldrack.
2012. Deconvolving BOLD activation in event-related designs for multivoxel pattern classification analyses. Neuroimage 59: 2636–2643.
122. Mumford, J.A., T. Davis & R.A. Poldrack. In prep. Bias in
representational similarity analyses using single trial parameter estimates.
123. Aguirre, G.K., M.G. Mattar & L. Magis-Weinberg. 2011. de
Bruijn cycles for neural decoding. Neuroimage 56: 1293–
1300.
124. Hanke, M., Y.O. Halchenko, P.B. Sederberg, et al. 2009.
PyMVPA: a Python toolbox for multivariate pattern analysis of fMRI data. Neuroinformatics 7: 37–53.
134
125. Coutanche, M.N. & S.L. Thompson-Schill. 2012. The advantage of brief fMRI acquisition runs for multi-voxel
pattern detection across runs. Neuroimage 61: 1113–
1119.
126. Fodor, J.A. 1983. The Modularity of Mind: An Essay on
Faculty Psychology. The MIT Press. Cambridge MA.
127. Poldrack, R.A. 2006. Can cognitive processes be inferred
from neuroimaging data? Trends Cogn. Sci. 10: 59–63.
128. Palmer, S.E. & K.B. Schloss. 2010. An ecological valence
theory of human color preference. Proc. Natl. Acad. Sci.
U. S. A. 107: 8877–8882.
129. Posner, M.I. & S.W. Keele. 1968. On the genesis of abstract
ideas. J. Exp. Psychol. 77: 353–363.
130. Love, B.C. 2003. The multifaceted nature of unsupervised
category learning. Psychon. Bull. Rev. 10: 190–197.
131. Pothos, E.M. & N. Chater. 2002. A simplicity principle in
unsupervised human categorization. Cogn. Sci. 26: 303–
343.
132. Carp, J. 2012. On the plurality of (methodological) worlds:
estimating the analytic flexibility of fMRI experiments.
Front. Neurosci. 6: 149.
133. Carp, J. 2012. The secret lives of experiments: methods
reporting in the fMRI literature. Neuroimage 63: 289–300.
134. Poldrack, R.A. 2012. The future of fMRI in cognitive neuroscience. Neuroimage 62: 1216–1220.
135. Harrell, F.E. 2010. Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival
Analysis. New York: Springer-Verlag.
136. Markman, A.B. 1998. Knowledge Representation. Lawrence
Erlbaum.
137. Cilibrasi, R.L. & P.M. Vitanyi. 2007. The google similarity
distance. IEEE Trans. Knowl. Data Eng. 19: 370–383.
138. Laurence, S. & E. Margolis. 2001. The poverty of the stimulus argument. Br. J. Philos. Sci. 52: 217–276.
C 2013 New York Academy of Sciences.
Ann. N.Y. Acad. Sci. 1296 (2013) 108–134