1
Identifying Human Memory Encoding Mechanisms
from Physiological fMRI data via Machine Learning
Techniques
Asaf Gilboa1, Hananel Hazan2, Ester Koilis2, Larry M. Manevitz2, Tali Sharon3*
1
2
The Rotman Research Institute, Canada
Computer Science Department, University of Haifa, Israel
3
Psychology Department, University of Haifa, Israel
Abstract - Neuropsychological theories postulate that there are multiple memory systems in the brain
but there is controversy as to whether declarative memory is a unitary memory system. In this study,
we succeeded in classifying two distinct declarative memory acquisition mechanisms directly from
physiological data by the use of machine learning techniques on functional MRI (fMRI) scans of
subjects, thereby adding explicit physiological justification to the existence of multiple declarative
memory systems.
The data were gathered in previous experiments which were designed so that subjects acquired
identical declarative information, but used different processes in doing so. The analysis was based on
the multi-voxel pattern analysis of neural information obtained from fMRI signals. Support Vector
Machines (SVM) type classifiers identified the memory patterns from complex, high dimensional and
noisy fMRI activations evoked by participants while they acquired novel information in one of two
methods: fast mapping encoding and explicit encoding enabling prediction of whether the subject
succeeded in the recollection attempt for data acquired with each of two encoding methods. A further
classifier succeeded in distinguishing the type of encoding used for novel knowledge acquisition - fast
mapping or explicit encoding. Finally, applying a multivariate “searchlight” method assisted in
construction of qualitative brain maps for both paradigms enabling identification of activation patterns
associated with each method and highlighting the physiological differences between them.
Keywords - Machine Learning, fMRI, SVM, Memory Encodings, Human, Word Learning
I. Introduction
The present work uses machine learning
techniques to demonstrate the uniqueness of FastMapping (FM). FM is a neurocognitive
mechanism enabling rapid acquisition of
declarative
novel
information
(arbitrary
associations) independently of the hippocampus
(Sharon, Moscovitch, & Gilboa, 2011). This
mechanism is known to support vocabulary
acquisition in children as fast as after only a single
exposure to the word-object association (Carey &
Bartlett, 1978). The FM mechanism allows for a
rapid mapping to be created between a word and
its referent by the child based on logical
hypothesis formation that probably relies on
disjunctive syllogism. Despite the literature on the
various aspects of FM in children as a word
learning mechanism, little is known about the
characteristics of this mechanism in adults, or
2
about its neural substrate. It could be that FM
serves as a general learning mechanism, not solely
dedicated for word learning and as such should be
accessible to adults. Sharon et al. (Sharon,
Moscovitch, & Gilboa, 2011) have recently
demonstrated that adults with extensive damage to
the Medial Temporal Lobe and the hippocampus
were able to acquire novel declarative associations
through FM despite a profound impairment in
declarative learning through explicit encoding.
The goal of the present study is to investigate the
neural basis of FM learning in adults, and the
possible role of FM as a neurocognitive mediator
for the acquisition of novel declarative semantic
memories. The hypothesis is that FM declarative
learning depends on cortical structures that are
distinct from those essential for learning
declarative associations through a matched explicit
episodic encoding control paradigm (EE).
To address this question, the neuroanatomical
correlates of FM and EE learning were explored
using an event related functional Magnetic
Imaging method (ER-fMRI) (Sharon, 2010). fMRI
is a noninvasive technique for investigating the
neural correlates of cognitive processes in which
the hemodynamic response (i.e. the change in
blood oxygenation level) related to neural activity
in the brain is measured. The fMRI combines high
spatial resolution anatomic imaging capabilities of
conventional MRI with the hemodynamic
specificity of nuclear tracer techniques (positron
emission tomography), allowing spatially accurate
mapping of human brain function to underlying
anatomy. ER-fMRI is a more recently developed
fMRI paradigm designed to measure regional
responses to single sensory or cognitive events, in
contrast to "blocked" designs in which activity was
measured over blocks consisting of several trials.
The fMRI data were gathered during the
information recollection task performed by
participants. The task was designed so that
successful acquisition of novel associations was
based either on incidental fast mapping - FM or on
explicit encoding – EE. In (Sharon, 2010) the data
were analyzed using the tools of SPM5 (SPM,
2011) to identify the regions of interest appropriate
to the task.
In this work the data collected during the previous
experiments were used (Sharon, 2010). The
following questions were asked in regard to the
abilities of machine learning techniques used for
analysis: 1) Is it possible to distinguish between
“recollection success” and “recollection failure”
conditions in EE-based tasks? 2) Is it possible to
distinguish between “recollection success” and
“recollection failure” conditions in FM-based
tasks? 3) Can we predict which of the original
mapping paradigms, FM or EE, were used by
participant in “recollection success” condition? 4)
Can we identify the brain activity areas associated
with FM and EE mechanisms?
In this paper, we show that the answer to these
questions is affirmative.
Interpreting brain image experiments requires
analysis of complex, multivariate data. Methods
used for the analysis depend on the specific
research question. It may be retrieval or decoding
stimuli, mental states, behaviors and other
variables of interest from the raw data and thereby
showing the data contain information about them –
brain decoding (answering the question of “is
there information about a variable of interest”).
Questions 1-3 belong to this category of tasks.
However, it is usually not enough, and the research
question requires finding out how the information
is mapped to the activity patterns in the particular
brain regions – brain mapping (answering the
question of “‘where the information resides inside
the brain”). Question 4 falls to this category.
In addition, fMRI analysis methods can be
categorized according to the number of variables
included into the analysis. Univariate methods
perform voxel-wise analysis, multivariate methods
provide inference about larger parts or the entire
brain simultaneously. Univariate methods are
widely applied in the neuroscience domain. The
standard method is Statistic Parameter Mapping
(SPM), which is based upon the hypothesis of
linear correlation between neuro-activities and
tasks, and utilizes general linear model (GLM) to
3
do regression analysis (Friston, Holmes, Worsley,
Poline, Frith, & Frackowiak, 1994). The
motivation for using multivariate learning
techniques in this work stems from the known
limitations of GLM.
One of the limitations is related to the univariability of this method. Possible between-voxel
interactions are not taken into consideration during
the analysis thus weakening a general inferring
strength of this method. Another significant
disadvantage is the assumptions of a particular
fMRI response model driving the regression - the
voxels in GLM are rated by univariate analysis of
the correlation between the real signal and the
estimated Hemodynamic Response Function
(HRF). There are recent and sophisticated HRF
models trying to better capture the complex
structure of the fMRI response (Zheng,
Martindale, Johnston, Jones, Berwick, & J., 2002).
Nonetheless, these parametric models still encode
the ideal expected fMRI signal not considering
confounds in the design protocol and not including
the dependencies due both to the brain structure
(e.g., proximity of a big vessel, location) and to the
cognitive/ perceptual tasks under investigation.
Actually, most cognitive fMRI research to date
appears to be exclusively focused on estimating
the magnitude of evoked activations and does not
pay much attention to co-action of different areas
or HRF variability. As revealed by a recent survey
of 170 fMRI studies, 96% of experiments used a
canonical HRF model, thus ignoring the difference
in shape between individuals or areas of the brain
(Grinbald, Wager, Lindquist, & Hirsch, 2008).
In general, is it feasible to use machine learning
techniques for the prediction of complex cognitive
states? There is no convincing answer to this
question. However a growing number of studies
(Cox & Savoy, 2003; Haxby, Gobbini, Furey,
Ishai, Schouten, & Pietrini, 2001; Haynes & Rees,
2005; Kamitani & Tong, 2005; Mitchell, et al.,
2004; Mitchell, et al., 2008; Kriegeskorte, Goebel,
& Bandettini, 2006) show that machine learning
techniques can be used to extract new information
from the neuroimaging data. Both brain decoding
and brain mapping techniques are explored in
these works. The approach is usually multivariate,
with different strategies used for variables subset
selection. Thus, it was demonstrated (Shinkareva,
Mason, Malave, Wang, Mitchell, & Just, 2008)
that one may observe differences in neural activity
using fMRI, as people think about different items,
and train a machine learning classifier to discover
the patterns of activity associated with these items.
Moreover, it was shown (Mitchell T. , et al., 2008)
that a machine learning classifier trained on fMRI
data collected from a group of people could
successfully distinguish which item a new person
was thinking about, despite the fact that the
classifier had never seen data from this person
(although accuracies vary by person).
The recollection task discussed in this paper is
much more complex as it involves additional
cognitive dimensions, for example, decision
making or response production.
Various machine learning classifiers can be used
for decoding the different variables of interest.
Classification is the analogue of regression when
the variable being predicted is discrete, rather than
continuous. Also, classifiers are used in the reverse
direction, predicting parts of the design matrix
from many input variables. A classifier is a
function that takes the values of various features
(independent variables or predictors, in regression)
in an example (the set of independent variable
values) and predicts the class that that example
belongs to (the dependent variable). In
neuroimaging, the features are voxels and the class
is usually the type of stimulus the individual was
looking at when the voxel fMRI signals were
recorded. The trained classifier is essentially a
model of the relationship between the features and
the class label. Once trained, the classifier can be
used to determine whether the features used
contain information about the class of the example.
Different types of classifiers exist, but in this work
we will concentrate on the most basic form – a
linear classifier, whether the classification function
is defined as a linear combination of the features.
As there are usually much more voxels than data
points in the fMRI data sets, often it is advisable to
perform feature selection - a process reducing the
4
number of features by selecting the significant
ones only. Reducing the ratio of features to data
points decreases the chance of overfitting, as well
as gets rid of the non-informative features to
enable the classifier to focus on the informative
ones. Moreover, this process also able to reduce
feature redundancy which decreases noise on the
input.
Both univariate and multivariate approaches for
feature selection exist. In univariate selection, the
features are ranked by a given criterion where each
feature is scored individually, and features with
best ranking are selected. In multivariate selection,
new features are picked according to by how much
impact they have on the classifier, given the
features already selected on the previous step.
Alternatively, in reverse, the initial set may
include all the features to begin with, and the
features are removed until the performance does
not decrease.
pictures and a reminder for the manner of response
appeared for an additional 2 seconds. Next, the
participants were given 1.5 seconds in order to
respond while the pictures and the reminder were
still presented on screen. Next, the subjects
received relevant feedback for their response; the
feedback was presented on the screen for 0.5
seconds. Finally, a red fixation cross was
presented for either 4 seconds in half of the events
or 6 seconds in the other half. Thus, each event
lasted either 9 or 13 s, a mean of 11 s per event.
In the FM trials, the stimuli were two pictures of a
novel and a familiar animal/fruit/vegetable/flower
and the question presented was a perceptual
question regarding one of these pictures, the target
picture (for example, “Does the chayote has
leaves?”). The participants were instructed to press
either the right button on the response box in order
to answer 'yes' and the left button if their answer
was 'no'. No mention was made about a later
memory test.
II. Materials and Data Gathering
The full details of human experiments briefly
described below are found in (Sharon, 2010).
Participants
Twenty five healthy volunteers participated in this
study: thirteen participants who performed the FM
paradigm and twelve who performed the EE
paradigm. Of these participants, 15 were males
and 10 females, their mean age 26.64 (SD=3.41).
No significant difference was found neither in the
age of the participants in the FM (M=25.38,
SD=2.36) and EE paradigms (M=28.09, SD=3.93)
[t(23) =2.03, ns], nor in the gender distribution [ 
(1)
2
=0.03, ns].
Experimental Paradigm
During the experiment, participants were given
either FM or EE series of tasks. In each paradigm,
novel and familiar target trials (either FM or EE
trials) were intermixed with base line trials. Each
trial, whether an FM, EE or base line trial was
composed of the following steps: at first a
question/statement was presented both visually
and auditory for 3 seconds. Next, the relevant
Figure 1 FM stimuli example.
In the EE trials, one picture, either novel or
familiar, was presented alongside a scrambled
picture and participants were explicitly instructed
to remember the item for a later test (for example,
“Try to remember lornec”). The participants were
also requested to look for an x under either the
picture of the scrambled picture and, as in the FM
paradigm, to press the right or left response
buttons on the response box in order to answer the
question.
Finally, in the base line trials, the participants were
presented with two scrambled pictures (the
original pictures from the FM paradigms were
scrambled) and were asked "Is the picture on the
right brighter?" Again, participants were to answer
5
using the response box similarly to the FM and EE
trials.
Figure 2 EE stimuli example.
Each experiment, either FM or EE, was
transmitted in 3 runs. The first two runs included
40 events and lasted 8 minutes and 2 seconds each.
The last run included 44 events and lasted 8
minutes and 52 seconds. The events were
organized in 5 sequences (E-Prime "lists") of 8
events in the first two runs and an additional
sequence of 4 events in the third run. The events
were pseudo randomly assigned such that each
sequence of 8 events contained 4 novel FM/EE
target trials, 2 familiar FM/EE target trials and 2
base line trials. Every run began with 12 seconds
of a presentation of either a reminder of the
instructions (on the first run) or a blank screen (the
second and third runs). The images acquired
during these 12 seconds were intended to allow
global image intensity to reach equilibrium, and
they were later excluded from data analysis.
Between every 8 events a blank screen appeared
for duration of 6 seconds. Memory was tested
outside the magnet using a 4-alternative forced
choice recognition in which the label appeared in
the center and four pictures around them.
Participants had to select the correct picture to go
with the label.
fMRI Procedure
Imaging was performed on a GE 3T Signa HDx
MR system with an 8-channel head coil located at
the Whol Institute for Advanced Imaging in Tel
Aviv Sourasky Medical center. The scanning
session included T1-weighted anatomical 3D
sequence spoiled gradient (SPGR) echo sequences
(TR=9.14 ms, TE=3.6 ms, flip angle =13º)
obtained with high-resolution 1-mm slice
thickness and no interslice skip and a 256x256
matrix. .In addition T2*-weighted functional axial
images (TR=2000 ms, TE=40 ms, flip angle =90º)
were acquired from the bottom of the cerebrum to
the top in 32 contiguous slices aligned parallel to
the AC–PC plane, of 5 mm thickness with no
interslice skip, a field of view of 20 cm and a
64x64 acquisition matrix. The functional images
covered the whole cerebrum and yielded 3x3x5
mm voxels. The images were acquired in 3 runs.
In the first 2 runs, 241 images were acquired
during each run (7712 slices per run). In the third
run, 266 images were acquired (8512 slices). At
the beginning of each run, six images were
acquired to allow global image intensity to reach
equilibrium; these were later excluded from data
analysis.
fMRI Data Processing
Data were preprocessed using SPM5 (SPM, 2011).
The functional images were corrected for
differences in slice acquisition timing by
resampling all slices in time to match the middle
slice. This was followed by a realignment of the
time series of images to the first image of the run
performed after acquisition of the anatomical
image (for most subjects this was the third run).
The data were then spatially normalized to MNI
space and smoothed with a 5-mm FWHM of the
Gaussian smoothing kernel.
Each data point used for analysis was constructed
using scan data obtained for TR=1 (related to the
stimuli exposition and the reminder) composing a
vector of 517845 features. The selection of TR=1
was motivated by the pre-test classification results
obtained for TR=0..4 and revealed the best
classification accuracy. Each feature vector was
detrended (session number SN = 3) normalized
independently of others before analysis procedure.
Two different labels were assigned to each data
point, specifying the status of recollection action –
“recollection success” in the case of a post-scan
correct answer, or “recollection failure” in the case
of a wrong answer, and in addition the paradigm
this data point belongs to – “FM” or “EE”.
6
III. Methods
Basic contrasts defined in Questions 1-3 belong to
the Brain Decoding domain, asking whether the
classification information exists in a given dataset.
Traditional univariate analysis of fMRI data does
not provide the direct answer to these questions.
The indirect classification information is usually
fetched from the statistical analysis of data
contained in the time course of the individual
voxels (Friston, Holmes, Worsley, Poline, Frith, &
Frackowiak, 1994). A multivariate analysis used in
this study takes the advantage of knowledge
contained in the activity patterns across the entire
brain volume, from the multiple voxels
(Formisano, De Martino, & Valente, 2008). A
trained classifier takes the values of various voxels
(features) in a data sample and predicts the class
that this sample belongs to. A class of the sample
is selected from the set of different stimuli defined
for a given contrast. In our case we are interested
in two types of contrasts with the following classes
defined for each one of them: 1) “recollection
success” and ”recollection failure” for Explicit
Encoding (EE) tasks; 2) “recollection success”
and ”recollection failure” for Fast Mapping (FM)
tasks; and 3) “FM” and ”EE” for inter-paradigm
classification. Each set contains 2 class labels
only; therefore the classification of this kind is
called two-class classification. A linear Support
Vector Machine (Vapnik, 1999) was used as an
underlying classifier for all study experiments. The
classes in the data acquisition were selected to
have equivalent frequency.
Classification results were evaluated using 3-fold
cross-validation for within-subject experiments
(according to the number of the collected sessions)
and leave-one-out cross-validation for crosssubject experiments. In all analyses, the accuracy
of prediction was based only on test data that was
completely disjoint from the training data.
Considering the high data dimensionality used in
the current study, feature selection procedure was
performed in order to decide which voxels should
be included into the multivariate classification
analysis. Feature selection process was performed
three separate times based on different scoring
methods ranking features by the individual voxel
performance under each of the corresponding
scoring methods. In each case, all voxels were
sorted according to the assigned score in the
descending order. The 1000 voxels having the
highest ranking scores were included in the
analysis.
The following feature selection methods were
explored: (i) Activity - selects the voxels that are
active in at least one condition relative to a control
baseline, (ii) Accuracy - scores a voxel by how
accurately an SVM classifier can predict the
condition of each example in the training set,
based only on this voxel, and (iii) SVM-RFE – a
multivariate eliminating approach to the feature
selection process (Guyon, Weston, Barnhill, &
Vapnik, 2002) starting from a complete feature set
and then eliminating 15% of the tail-ranked
features (the rank is based on a feature weight
obtained in the multivariate SVM classification)
during each execution round, until the number of
features is reduced to 1000. In all cases, the crossvalidation success averaged rate was used as a
voxel score.
Figure 3. A classification scheme used in the
experiments. A feature selection process is followed by
the classifier training.
To evaluate the statistical significance of the
observed classification accuracy, classification
results were compared to those obtained by using
random selection of data classes.
The prediction accuracy was evaluated for both
within-subject (Questions 1 and 2 only) and crosssubject cases, using different spatial and temporal
aspects of the input data.
7
For the within-subject case, the accuracy value
was produced for each participant individually,
and then the average accuracy was calculated. For
cross-subject case, the accuracy was produced on a
dataset combined of the individual participant’s
datasets, using the leave-one-out cross validation
method. The accuracy was calculated as an
average over all cross-validation folds.
the relevant patterns obtained during the crosssubject analysis.
Unlike the contrasts classification (Questions 1-3),
discovering the brain areas associated with each
paradigm - FM or EE (Question 4)) - requires
constructing brain maps. In machine learning
terms, brain mapping is a process of highlighting
voxels contributing most strongly and reliably to
the classifier’s success. It may be achieved by
determining which voxels are being selected by a
classifier and also how their classification weight
affects the classifier prediction. The major issue
with this straightforward approach is that a group
of voxels appearing in a conjunction of all crossvalidation fold sets is relatively small (as a result
of the initial information redundancy) and cannot
be used as a completely reliable source for brain
mapping. Information-based functional brain
mapping method (Kriegeskorte, Goebel, &
Bandettini, 2006) overcomes this limitation. The
main idea of this method is to train classifiers on
many small voxel sets which, put together, cover
the entire brain. For example, we may train a
distinct classifier for each voxel, using only the
voxels spatially adjacent to it. Then the search area
may be enlarged to include every voxel
neighborhood in succession (This technique is
often referred to in the fMRI machine learning
community as training ‘searchlight classifiers’).
The empirical analysis aims to assess the ability of
chosen classification model to predict the
classification targets in a non-random manner. The
encouraging results show that the model was able
to predict the required targets in all 3 contrasts
(Questions 1-3) although with different levels of
the accuracy.
IV. Results
Experiment 1. Recollection Status and Memory
Paradigm Prediction.
Contrast 1. EE task – recollection status.
Ranking
Metric
Prediction
Accuracy
SD
Within-Subject
Analysis
Type
The classification results for EE paradigm are
presented in Table 1. The results are significant
statistically. Random choice will give a level of
0.5, and all results are significant statistically
found more than 2 SD above this value. The best
classification results are obtained by using the
multivariate SVM-RFE feature selection method a mean value of 78% for correct class predictions
in within-subject analysis, and a mean value of
73% for correct class predictions in CV crosssubject analysis.
Cross-Subject
SVM (Support Vector Machines) was used as the
underlying classifier in this study. The resulting
brain map reconstructed the accuracies of a
classifier trained on the spherical neighborhoods
of a radius r=4mm. Voxels with a statistically
significant accuracy were inserted into the brain
map (in these brain maps the highlighting color
strength reflects the accuracy rate). Both withinsubject and cross-subject maps were produced.
The disjunction of within-subject maps was
constructed enabling even stronger highlighting of
The software used for these experiments was
developed on Python programming language and
based on pyMVPA library (Hanke, Sederberg,
Hanson, Haxby, & Pollmann, 2009).
Accuracy
0.66354255
0.04433817
Activity
0.67992270
0.04095269
SVM-RFE
0.7778667
0.0237164
Accuracy
0.60715518
0.04960175
Activity
0.60125534
0.04527102
SVM-RFE
0.7322059
0.0619211
Table 1 Experiment 1, Contrast 1. Classification
accuracy for EE paradigm.
8
Contrast 2. FM task – recollection status.
SD
Accuracy
0.73157783
0.05037422
Activity
0.71005632
0.03937601
SVM-RFE
0.80722163
0.03902201
Accuracy
0.66204481
0.06090074
Activity
0.65448720
0.03683927
SVM-RFE
0.76072566
0.03072572
Ranking
Metric
Prediction
Accuracy
Cross-Subject
Within-Subject
Analysis
Type
The experiment environment was identical to that
of Contrast 1 except for the data set source. The
data set for this experiment was collected from the
fMRI of participants performing the FM task. In
similar to Contrast 1, the trained model was able to
predict the recollection status under the Fast
Mapping (FM) paradigm.
Ranking
Metric
Prediction
Accuracy
SD
Accuracy
0.80210345
0.03642382
Activity
0.60213476
0.03240215
SVM-RFE
0.88794379
0.05643578
Table 3. Experiment 1, Contrast 3. Classification
accuracy for FM vs. EE paradigms.
Using the above results we were able to move to
the construction of brain maps.
Experiment 2. Brain Activation Mapping
Table 2. Experiment 1, Contrast 2. Classification
accuracy for FM paradigm.
Classification results for FM are presented in
Table 2. They are slightly higher than those
obtained for EE experimental data.
Contrast 3. FM vs. EE – paradigm prediction.
The FM vs. EE classification experiment is the
most intriguing because of its ability to point to the
essential difference between these two memory
mapping paradigms.
The analysis was based on two-class classification,
with target class labels “FM” and “EE”. Only
successful trials for both mapping types (the
“recollection success” data points) were taken into
account. The classification results showed that the
difference between FM and EE indeed exists and
can be detected at the 88% level (Table 3).
The results presented below obtained with a
“searchlight” algorithm show that for both the
individual maps and the cross-subject maps, the
FM is characterized by the activity in a temporal
pole area and the backline parts of the cerebellum,
while the EE is more associated with the medial
temporal regions. One may expect that there is no
perfect match between different participants’
activation areas and the exact point of the
information storage and retrieval has a sufficient
interpersonal variation. This is why the disjunction
we present of the within-subject maps was
produced. In this map, the areas associated with
two different declarative memory paradigms are
clearly visible (see Figure 4 and Figure 5). Again,
the large spots of the activity in the temporal pole
area and the backline parts of the cerebellum are
associated with Fast Mapping. We see from these
maps that the extensive medial temporal regions
are associated with EE while more moderate maps
are produced for FM. From the other side, FM is
characterized by activations of the unique anterior
temporal lobe and polar areas not produced for EE.
Note that the hippocampus shows up less in this
paradigm as compared to the Explicit Encoding
paradigm.
9
Figure 4 Experiment 2. A disjunction of the participants’ brain maps for Contrast 1 (Explicit Encoding). Axial view with
4 mm interslice spacing. Active areas are shown in yellow.
Figure 5 Experiment 2. A disjunction of the participants’ brain maps for Contrast 2 (Fast Mapping). Axial view with 4
mm interslice spacing. Active areas are shown in yellow.
10
Experiment 3. Spatial Analysis – Hippocampus
versus Temporal Pole
Given the brain maps, we were able to evaluate a
contribution of the individual areas highlighted in
the maps to the classification accuracy. We were
especially interested in the hippocampus and the
temporal pole areas found in the maps and known
from the previous studies (Sharon, 2010) as
differentiating between the paradigms. For this
purpose, the classification procedure for withinsubject and cross-subject data was repeated for
various brain cuts including: (i) the entire brain
(All), (ii) the hippocampus only (Hippocampus
Only), (iii) the temporal pole only (Temporal Pole
Only), (iv) the entire brain with a hippocampus
excluded from the analysis (All w/o
Hippocampus), (v) the entire brain with a temporal
pole excluded from the analysis (All w/o Temporal
Pole), and (vi) the putamen (Putamen Only) – a
control area with a size comparable to the size of
the hippocampus and mostly not associated with
any of two paradigms. This area was used for
evaluation of random prediction accuracy.
μ (SD)
Brain Cut
Within-Subj.
Cross-Subj.
All
0.77786665
(0.02371642)
0.73220590
(0.06192115)
Hippocampus
Only (BA36)
0.73320062
(0.04249911)
0.696540652
(0.04492871)
Temporal Pole
Only (BA38,21)
0.70080948
(0.02278488)
0.66303595
(0.06915410)
All w/o
Hippocampus
0.77695866
(0.02371641)
0.73531368
(0.04931522)
All w/o
Temporal Pole
0.77746900
(0.02393828)
0.73424218
(0.05413447)
Putamen Only
0.57933221
(0.04887691)
0.59243121
(0.06249911)
Table 4 Experiment 3. Classification accuracy for
Contrast 1 (Explicit Encoding paradigm).
μ (SD)
The classification results are shown in the tables
below. They depict the brain cut classification
accuracy for within-subject and cross-subject
analysis methods. For within-subject method, a
mean value of subjects’ classification accuracy is
reported, with a standard deviation shown in the
braces. For cross-subject method, a mean value of
one-leave-out cross-validation is reported, with a
standard deviation between different folds shown
in braces.
Contrast 1. Explicit Encoding task (EE) –
recollection status for different brain cuts.
For Contrast 1, the classification accuracy is
significantly higher than a baseline random level
(0.5) for all tested brain cuts, except for the
randomly selected control area (the putamen).
Contrast 2. Fast Mapping task (FM) – recollection
status for different brain cuts.
For Contrast 2, the classification accuracy is
significantly higher than a baseline random level
(0.5) for all tested brain cuts, except for the
randomly selected putamen area.
Brain Cut
Within-Subj.
Cross-Subj.
All
0.80722163
(0.03902227)
0.76072566
(0.03072455)
Hippocampus
Only (BA36)
0.72348420
(0.04597861)
0.68615242
(0.06431254)
Temporal Pole
Only (BA38,21)
0.75566112
(0.04737763)
0.71372193
(0.05971212)
All w/o
Hippocampus
0.80723226
(0.03902227)
0.76525352
(0.04215641)
All w/o
Temporal Pole
0.80705919
(0.03902227)
0.76017063
(0.05447689)
Putamen Only
0.56666919
(0.05971527)
0.55744710
(0.05214834)
Table 5 Experiment 3. Classification accuracy for
Contrast 2 (Fast Mapping paradigm).
Again, the classification accuracy obtained for FM
paradigm is slightly higher than for EE paradigm.
11
Classification results for Contrast 1 and Contrast 2
look a little bit controversial. Removing regions
seems to contribute very little to classification
success. This behavior may be explained by
experiment participant composition assembled
from healthy people only. Unlike with the real
patients, healthy participants have all the available
temporal structures in place during the brain
encoding, and so the information in the rest of the
brain reflects that fact and enables robust
classification. On the other hand, looking at the
classification success of each structure alone
compared with the whole brain reveals the real
relations between different brain cuts.
Because of the sufficient differences in the wholebrain classification accuracy between FM and EE
(the classification is more accurate for FM than for
EE), we compare the percentages. The two
conditions, FM and EE, and the two structures, the
hippocampus (H) and the temporal pole (TP) show
a reverse pattern (Figure 6) which is the same as
observed in patients (Sharon, 2010).
Figure 6. Experiment 3. Reduction in SVM-RFE
prediction accuracy caused by Hippocampus area
removal compared to the reduction in SVM-RFE
prediction accuracy caused by Temporal Pole area
removal. A reverse pattern of prediction accuracy
reduction is observed for FM and EE tasks.
For EE, using H cut only reduces the classification
success by 5.7% and using TP cut only there is a
9.9% reduction in the classification success. The
reverse pattern is seen in FM (10.3% and 6.3%
respectively). If one looks at the residual
classification over random level (50%) then the
figures of the reduction in classification success
are even more pronounced (for EE, 16% for H and
27.7% for TP; for FM, 27.2% for H and 16.7% for
TP). All results are statistically significant. They
lead to the conclusion that the hippocampus cut
produces better classification results for EE than
for FM; from the other side, the temporal pole cut
produces better classification results for FM than
for EE.
V. Discussion
A basic question being addressed in the current
study is whether the registered fMRI signal carries
information about the particular patterns of
knowledge acquisition and retrieval. In other
words, it was concerned with pattern
discrimination. It appears that although both FM
and EE lead to the acquisition of declarative
memory as reflected in the post-scan recognition
performance, they do so by recruiting very distinct
neuronal networks that can be efficiently
distinguished using SVM.
In the first phase of the study, three different
feature selection methods were used at the
preprocessing stage of the classification process.
The univariate methods selecting the individual
voxels according to some predefined rank enabled
to classify data points according to the given
contrast, however, with relatively low prediction
accuracy (up to 70%). Using an SVM-RFE, a
multivariate feature selection method, based on
pruning the features associated with the low
absolute weight values produced by Support
Vector Machine during the classification process,
enabled to increase the accuracy of prediction by
10% in average. Thus, the study shows that using
the multivariate methods for feature selection and
classification purposes brings dramatic increase to
the classification performance. Unfortunately no
production
SPM-level
software
exists
implementing these methods leading to the almost
complete ignorance of them by the wide
neuroscientific audience.
For the next stage, we leveraged these results to try
to address the question as to where the
12
discriminative patterns reside in the brain - pattern
mapping. It was important to clarify which
memory structures are involved in information
retrieval for both fast mapping and explicit
encoding designs. In the second part of the study,
we were interested in finding the brain regions
correlated with the formation of memory through
EE and especially through FM paradigms. Our
hypothesis was that underlying the FM learning
would be a network of brain regions distinct from
the network known to mediate the EE (the episodic
memory). Indeed, according to the brain maps
constructed using multivariate “searchlight”
method, this network included amongst others
regions positioned more lateral in the temporal
neocortex, and specifically in the anterior temporal
lobe and polar area, as opposed to medial temporal
regions critical for episodic memory.
informative than basic univariate methods. Using
these advanced techniques, we showed that Fast
Mapping engages distinctly different regions than
those activated by an explicit encoding tasks
presumably relying on episodic memory encoding.
In both cases, medial and lateral pre frontal
activations along with superior and medial
posterior parietal regions were found, however
distinct areas within these regions were active for
FM and EE tasks.
Another point concerns investigating the role of a
hippocampus and the surrounding medial-temporal
cortices for relational memory functioning. This is
a point of discussion in the neuropsychological
community. We hope and expect that current study
based on empirical fMRI data and advanced
machine learning techniques contributed to the
discussion.
In conclusion, the results above were obtained
using mainly multivariate machine learning
techniques proven to be more accurate and
References
1. Carey, S., & Bartlett, E. (1978). Acquiring a single new word. Papers and Reports on Child Language
Development , 15, 17-29.
2. Cox, D. D., & Savoy, R. L. (2003). Functional magnetic resonance imaging (fMRI) ‘brain reading’:
detecting and classifying distributed patterns of fMRI activity in human visual cortex. Neuroimage ,
19, 261–270.
3. Formisano, E., De Martino, F., & Valente, G. (2008). Multivariate analysis of fMRI time series:
classification and regression of brain responses using machine learning. Magnetic Resonance Imaging
, 28, 921–934.
4. Friston K. J., H. A., Worsley, K. J., Poline J. P., F. C., & S., F. R. (1994). Statistical parametric maps
in functional imaging: A general linear approach. Human Brain Mapping , 2 (4), 189-210.
5. Grinbald, J., Wager, T., Lindquist, M. F., & Hirsch, J. (2008). Detection of time-varying signals in
event-related fMRI designs. Neuroimage , 43, 509-520.
6. Guyon, I., Weston, J., Barnhill, S., & Vapnik, V. (2002). Gene Selection for Cancer Classification
using Support Vector Machines. Machine Learning , 46 (1-3), 389 - 422.
7. Hanke, M. H., Sederberg, P. B., Hanson, S. J., Haxby, J. V., & Pollmann, S. (2009). PyMVPA: A
Python toolbox for multivariate pattern analysis of fMRI data. Neuroinformatics , 7, 37-53.
13
8. Haxby, J., Gobbini, M., Furey, M., Ishai, A., Schouten, J., & Pietrini, P. (2001). Distributed and
overlapping representations of faces and objects in ventral temporal cortex. Science , 293, 2425–2430.
9. Haynes, J. D., & Rees, G. (2005). Predicting the orientation of invisible stimuli from activity in
primary visua cortex. Nature Neuroscience , 8, 686–691.
10. Haynes, J., & Rees, G. (2005). Predicting the stream of consciousness from activity in human visual
cortex. Current Biology , 16, 1301–1307.
11. Kamitani, Y., & Tong, F. (2005). Decoding the visual and subjective contents of the human brain.
Nature Neuroscience , 8, 679–685.
12. Kriegeskorte, N., Goebel, R., & Bandettini, P. (2006). Information-based functional brain mapping. In
Proceedings of National Academy of Science USA (Vol. 103, pp. 3863–3868).
13. Mitchell, T., Hutchinson, R., Niculescu, R., Pereira, F., Wang, X., Just, M., et al. (2004). Learning to
decode cognitive states from brain images. Machine Learning , 57, 145–175.
14. Mitchell, T., Shinkareva, S., Carlson, A., Chang, K., Malave, V., Mason, R., et al. (2008). Predicting
human brain activity associated with the meanings of nouns. Science , 320, 1191–1195.
15. Mitchell, T., Shinkareva, S., Carlson, A., Chang, K., Malave, V., Mason, R., et al. (2008). Predicting
Human Brain Activity Associated with the Meanings of Nouns. Science , 320, 1191.
16. Sharon, T. (2010). Bypassing the hippocampus. Rapid neocortical acquisition of long term arbitrary
associations via Fast Mapping. A PhD Thesis, Department of Psychology, University of Haifa.
17. Sharon, T., Moscovitch, M., & Gilboa, A. (2011). Rapid neocortical acquisition of long-term arbitrary
associations independent of the hippocampus. PNAS .
18. Shinkareva, S., Mason, R., Malave, V., Wang, W., Mitchell, T., & Just, M. (2008). Using fMRI Brain
Activation to Identify Cognitive States Associated with Perception of Tools and Dwellings. PLoS
ONE , 3 (1).
19. SPM. (2011). Wellcome Department of Cognitive Neurology, London, UK. Retrieved from SPM site:
www.fil.ion.ucl.ac.uk/spm
20. Zheng, Y., Martindale, J., Johnston, D., Jones, M., Berwick, J., & J., M. (2002). A Model of the
Hemodynamic Response and Oxygen Delivery to Brain. NeuroImage , 16 (3), 617–637.