Long Term Memory – Reinforcement Learning
Task Name
Description
Cognitive Construct Validity
Probabilistic
Reversal
Learning
Subjects select one stimulus from an
array of concurrently presented stimuli
(preferably 3); typically, more than one
set of discriminanda are presented, in a
randomized order across trials. Each
choice is followed by positive or
negative feedback (correct choice or
incorrect choice). The relationship
between the stimulus presented and the
feedback given can be deterministic or
probabilistic. Subjects must, based
upon feedback alone, learn which cue in
each discriminanda set are associated
with positive feedback and then
maintain optimized performance of the
rule (rule maintenance). Once the
subject meets pre-set performance
criteria, feedback is adjusted without
warning. In other words, the rules are
"reversed" in one or more of the
discriminanda sets. Subjects must
update their behavior based upon the
change in rules. Optimal reversal
performance is often considered to
involve cognitive control over pre-potent
responding.
Acquisition of a multiple
choice visual discrimination is
believed to reflect implicit
learning processes. Optimal
performance of a visual
discrimination is thought to
reflect rule maintenance.
Reversal of a learned visual
discrimination is thought to
measure cognitive control over
pre-potent responding
(response inhibition).
(Waltz & Gold, 2007)
(Lee, Groman, London, & Jentsch,
2007)
(Frank & Claus, 2006)
MANUSCRIPTS ON THE WEBSITE:
Cools, R., Altamirano, L., & D'Esposito,
M. (2006). Reversal learning in
parkinson's disease depends on
medication status and outcome valence.
Neuropsychologia, 44(10), 1663-1673.
Waltz, J. A., & Gold, J. M. (2007).
Probabilistic reversal learning
Neural Construct
Validity
Interactions between
declarative systems
(hippocampal and
prefrontal) and implicit
systems (striatum) are
predicted across initial
learning of a
discrimination, such
that increasing
activation in the
striatum, as a function
of trials performed,
occurs along with
learning. The
ventrolateral and
ventromedial frontal
cortex are particularly
involved in the
effective inhibition of
responses at reversal.
(Dias, Robbins, &
Roberts, 1996, 1997)
(Fellows & Farah,
2005)
(Fellows & Farah,
2003)
Reliability
As this is a
"learning" task,
test-retest
reliability is
unavailable.
Psychometric
Characteristics
Practice effects are
unknown, but are the
subject of active
investigation. Depending
upon how the test is
conducted, there can be
ceiling effects for
discrimination learning.
The use of multiple
discriminada sets, at least
3 stimuli in each set and
probabilistic feedback can
avoid this.
Animal Model
Stage of Research
Analogous tests
for mice, rats and
monkeys exist.
There is evidence
that this specific task
elicits deficits in
schizophrenia.
(Lee et al., 2007)
(Boulougouris,
Dalley, & Robbins,
2007)
(Chudasama &
Robbins, 2006)
We need to assess
psychometric
characteristics such
as test-retest
reliability, practice
effects, and
ceiling/floor effects
for this task.
We need to study
whether or not
performance on this
task changes in
response to
psychological or
pharmacological
intervention
BUT: (Cools,
Altamirano, &
D'Esposito, 2006;
Cools, Lewis, Clark,
Barker, & Robbins,
2007)
impairments in schizophrenia: further
evidence of orbitofrontal dysfunction.
Schizophrenia Research, 93(1-3), 296303.
Pizzagalli
Reward Task
Recently, we described a probabilistic
reward task based on a differential
reinforcement schedule that allowed us
to objectively assess participants’
propensity to modulate behavior as a
function of reward history. In this 15-20min computerized task, participants are
confronted with a choice between two
responses that are linked to different
probabilities of reward. Due to this
probabilistic nature, participants cannot
infer which stimulus is more
advantageous based on the outcome of
a single trial and so need to integrate
reinforcement history over time in order
to optimize their choices. Behavioral
performance is analyzed using signaldetection theory by calculating both
response bias toward the more
frequently rewarded stimulus and
overall discriminability (in addition to
reaction time and hit rates). Unix scripts
are available for automatic data quality
check, outlier detection, and
computation of behavioral variables.
In prior studies in non-clinical
samples, subjects reporting
elevated depressive
symptoms showed reduced
responsiveness to the more
frequently rewarded stimulus
(Pizzagalli et al., 2005).
Moreover, reward
responsiveness negatively
correlated with self-reported
anhedonic symptoms (Bogdan
and Pizzagalli, 2006;
Pizzagalli et al., 2005), and
predicted these symptoms one
month later (Pizzagalli et al.,
2005). These findings have
recently been extended to two
clinical groups: euthymic
patients with bipolar disorder
(Pizzagalli et al., 2008) and
unmedicated subjects with
unipolar depression (Pizzagalli
et al., under review) were
characterized by reduced
reward learning and reward
responsiveness, respectively.
(Pizzagalli, Jahn, & O'Shea, 2005)
MANUSCRIPTS ON THE WEBSITE:
Pizzagalli, D. A., Goetz, E., Ostacher,
M., Iosifescu, D. V., & Perlis, R. H.
(2008). Euthymic Patients with Bipolar
Disorder Show Decreased Reward
Learning in a Probabilistic Reward Task.
Biol Psychiatry.
Pizzagalli, D. A., Jahn, A. L., & O'Shea,
J. P. (2005). Toward an objective
characterization of an anhedonic
(Pizzagalli, Jahn, & O'Shea,
2005)
(Pizzagalli, Bogdan, Ratner, &
Jahn, 2007)
(Pizzagalli, Goetz, Ostacher,
Iosifescu, & Perlis, 2008)
(Bogdan & Pizzagalli, 2006)
In healthy controls, we
have also shown that
response bias was
reduced under an
acute stress condition
(Bogdan & Pizzagalli,
2006) and by a
pharmacological
manipulation affecting
the dopaminergic
system (Pizzagalli et
al., 2008) Conversely,
response bias was
increased when a
nicotine patch was
applied to healthy nonsmokers (Barr et al.,
2007). Finally, ongoing
ERP and fMRI studies
have provided
preliminary evidence
that response bias is
associated with
feedback-related
negativity (FRN) and
activation in the dorsal
anterior cingulate and
basal ganglia.
(Pizzagalli, Bogdan,
Ratner, & Jahn, 2007)
(Frank, Seeberger, &
O'Reilly R, 2004)
(Bogdan & Pizzagalli,
2006)
(Barr, Pizzagalli,
Culhane, Goff, &
Evins, 2007)
Information about
test-retest
reliability can be
found in Pizzagalli
et al. (2005). After
an average of 38
days, the testretest correlation
between bias
scores was 0.57.
Practice effects might
emerged with repeated
administration. This can be
circumvented by asking
participants to respond to
different stimuli in different
sessions. For an example,
see: Bogdan &
Pizzagalli, 2006
A federal grant is
currently under
review to develop
a rodent model of
our probabilistic
reward task.
This specific task
needs to be studied
in individuals with
schizophrenia.
Data already exists
on psychometric
characteristics of this
task, such as testretest reliability,
practice effects,
ceiling/floor effects.
There is evidence
that performance on
this task can
improve in response
to psychological or
pharmacological
interventions.
phenotype: a signal-detection approach.
Biological Psychiatry, 57(4), 319-327.
Probabilisiti
c Selection
Task
The probabilistic selection (PS) task
measures participants' ability to learn
from positive and negative feedback,
both on a trial-to-trial basis, and in
integrating reinforcement probabilities
over many trials. It also probes
performance in a novel 'test' phase
which permits evaluation of whether one
implicitly learned more from the positive
or negative outcomes of their decisions.
This "Go" or "NoGo" learning bias is
very sensitive to dopaminergic
manipulation and dopamine-related
genetics.
(Frank, Seeberger, & O'Reilly R, 2004)
(Waltz et al., 2007)
(Frank, Moustafa, Haughey, Curran, &
Hutchison, 2007)
MANUSCRIPTS ON THE WEBSITE:
Frank, M. J., Seeberger, L. C., &
O'Reilly R, C. (2004). By carrot or by
stick: cognitive reinforcement learning in
parkinsonism. Science, 306(5703),
1940-1943.
Waltz, J. A., Frank, M. J., Robinson, B.
M., & Gold, J. M. (2007). Selective
reinforcement learning deficits in
schizophrenia support predictions from
computational models of striatal-cortical
dysfunction. Biological Psychiatry.
Performance in this task is
defined by the ability to
choose the probabilistically
most optimal stimulus.
Relative positive and negative
feedback learning is similarly
modulated by dopamine
manipulation in this task and
others that are meant to
measure similar constructs
using different stimuli, motor
responses, and task rules.
Probabilistic positive
and negative feedback
learning are sensitive
to dopaminergic
manipulation.
Increases in
dopaminergic
stimulation, likely in
the striatum, lead to
better positive learning
but cause impairments
in negative feedback
learning. Dopamine
depletions, as in
Parkinson's disease
and older seniors
(greater than 70 years
of age) are associated
with relatively better
negative feedback
learning. Genes that
control dopamine
function in the striatum
are predictive of
probabilistic positive
and negative learning,
whereas genes that
control dopamine
function in prefrontal
cortex are predictive of
rapid trial-to-trial
learning from negative
feedback. Negative
feedback learning is
also associated with
enhanced errorrelated negativity
(brain potentials
originating from
anterior-cingulate
cortex) and activation
of this same region in
fmri.
In healthy
populations, there
is not a very large
test-retest
reliability (some
participants can
show a positive
learning bias in
one session and a
negative bias in
another). On the
other hand, there
are robust genetic
effects in this task,
with large effect
sizes, so on
average the task
must measure a
reliable trait
characteristic.
Thus individual
variability is likely
due to either (i)
state-dependent
changes; or (ii)
intrinsic noise in
the measure (for
example a
participant might
be labeled a
'positive' learner
after choosing
stimulus A, but
any one subject
might happen to
prefer stimulus A
from the outset
(before receiving
any reinforcement)
simply due to its
surface features).
These problems
can be mitigated
Practice effects have been
assessed in Frank &
O'Reilly (2006). On
average participants are
faster to learn the task
after multiple sessions, but
this practice does affect
relative positive versus
negative feedback
learning.
(Frank & O'Reilly R, 2006)
Currently being
used for rat testing
in collaboration
with Claudio
DaCunha's lab in
Brazil.
There is evidence
that this specific task
elicits deficits in
schizophrenia.
Data already exists
on psychometric
characteristics of this
task, such as testretest reliability,
practice effects,
ceiling/floor effects.
There is evidence
that performance on
this task can
improve in response
to psychological or
pharmacological
interventions.
(Frank et al., 2007)
(Frank, Woroch, &
Curran, 2005)
(Klein et al., 2007)
(Frank & O'Reilly R,
2006)
Weather
Prediction
Task
This is a 200 trial learning task. The
stimuli for this task consist of four “tarot”
cards that each have a unique
appearance that consists of an
arrangement of circles, squares,
triangles or diamonds. On any given
trial, between 0 and 3 of these cards are
presented, and there are 14 possible
card combinations. Each tarot card is
associated with one of two outcomes
(rain or sun) with a fixed probability.
The probability of each outcome (rain or
sun) on any given trial is computed as
the conditional probability of each
outcome and card occurring together.
After the set of cards is presented for
that trial, participants have up to 5
seconds to respond. Subjects are then
presented with their prediction accuracy
and the actual weather for that trial (rain
or sun). A running score on the side of
the screen increases or decreases as a
function of the accuracy of the subject’s
predictions.
(Knowlton, Mangels, & Squire, 1996;
Knowlton, Squire, & Gluck, 1994)
MANUSCRIPTS ON THE WEBSITE:
Gluck, M. A., Shohamy, D., & Myers, C.
(2002). How do people solve the
"weather prediction" task?: individual
variability in strategies for probabilistic
category learning. Learn Mem, 9(6),
408-418.
Keri, S., Juhasz, A., Rimanoczy, A.,
Szekeres, G., Kelemen, O., Cimmer, C.,
Research on how subjects
solve this task suggests three
different possible learning
strategies (Gluck, Shohamy, &
Myers, 2002): 1) multi-cue –
respond to each pattern on the
basis of associated of all four
cues with each outcome; 2)
one-cue – respond on the
basis of the presence or
absence of a single cue,
ignoring the other cues; or 3)
singleton – learn only about
patterns (4 total) that have
only one cue present an all
other absent. Participants
who adopt a multi-cue strategy
do better than participants
who adult either of the other
two strategy, for which
performance does not differ
(Gluck et al., 2002).
It is also thought that early
learning on the WPT is
dependent on implicit,
striatally governed processes,
while later learning can
engage explicit classification
learning processes (Price,
2005).
Parkinson’s patients do poorly
on versions of the task
involving feedback based
learning, but are not impaired
on versions requiring
observational learning
Evidence that this task
measures striatally
based learning comes
from studies showing
impaired WPT
performance in
patients with
Parkinson’s disease
(Knowlton, Mangels et
al., 1996),
Huntington’s Chorea
(Knowlton, Squire et
al., 1996), and
Tourette’s Syndrome
(Keri, Szlobodnyik,
Benedek, Janka, &
Gadoros, 2002).
Neuroimaging studies
in healthy individuals
reveal activation of the
striatum (Gluck,
Poldrack, & Keri,
2008; Poldrack &
Foerde, 2008;
Poldrack & Gabrieli,
2001; Poldrack,
Prabhakaran, Seger,
& Gabrieli, 1999).
There is evidence that
individuals with
schizophrenia are not
impaired in
performance on this
task (Keri et al., 2005;
Keri et al., 2000;
Weickert et al., 2002),
unless they are taking
by testing
participants in
multiple blocks
with different
stimuli and
averaging.
Unknown
Unknown
Unknown
This task does not
seem to elicit deficit
sin schizophrenia.
We need to assess
psychometric
characteristics such
as test-retest
reliability, practice
effects, and
ceiling/floor effects
for this task.
There is evidence
that performance on
this task can
improve in response
to psychological or
pharmacological
interventions.
et al. (2005). Habit learning and the
genetics of the dopamine D3 receptor:
evidence from patients with
schizophrenia and healthy controls.
Behav Neurosci, 119(3), 687-693.
(Shohamy et al., 2004).
Parkinson’s patients are less
likely than control to use a
multi-cue strategy and more
likely to use a singleton
strategy (Shohamy et al.,
2004).
antipsychotics that
block dopamine
receptors in the
striatum (Beninger et
al., 2003).
REFERENCES:
Barr, R. S., Pizzagalli, D. A., Culhane, M. A., Goff, D. C., & Evins, A. E. (2007). A Single Dose of Nicotine Enhances Reward Responsiveness in Nonsmokers: Implications for Development of Dependence. Biol
Psychiatry.
Beninger, R. J., Wasserman, J., Zanibbi, K., Charbonneau, D., Mangels, J., & Beninger, B. V. (2003). Typical and atypical antipsychotic medications differentially affect two nondeclarative memory tasks in
schizophrenic patients: a double dissociation. Schizophr Res, 61(2-3), 281-292.
Bogdan, R., & Pizzagalli, D. A. (2006). Acute stress reduces reward responsiveness: implications for depression. Biol Psychiatry, 60(10), 1147-1154.
Boulougouris, V., Dalley, J. W., & Robbins, T. W. (2007). Effects of orbitofrontal, infralimbic and prelimbic cortical lesions on serial spatial reversal learning in the rat. Behav Brain Res, 179(2), 219-228.
Cools, R., Altamirano, L., & D'Esposito, M. (2006). Reversal learning in parkinson's disease depends on medication status and outcome valence. Neuropsychologia, 44(10), 1663-1673.
Cools, R., Lewis, S. J., Clark, L., Barker, R. A., & Robbins, T. W. (2007). L-dopa disrupts activity in the nucleus accumbens during reversal learning in parkinson's disease. Neuropsychopharmacology, 32(1), 180189.
Chudasama, Y., & Robbins, T. W. (2006). Functions of frontostriatal systems in cognition: comparative neuropsychopharmacological studies in rats, monkeys and humans. Biological Psychiatry, 73(1), 19-38.
Dias, R., Robbins, T. W., & Roberts, A. C. (1996). Dissociation in prefrontal cortex of affective and attentional shifts. Nature, 380, 69-72.
Dias, R., Robbins, T. W., & Roberts, A. C. (1997). Dissociable forms of inhibitory control within prefrontal cortex with an analog of the Wisconsin Card Sort Test: restrictions to novel situations and independence
from "on-line" processing. The Journal of Neuroscience, 17(23), 9285-9297.
Fellows, L. K., & Farah, M. J. (2003). Ventromedial frontal cortex mediates affective shifting in humans: evidence from a reversal learning paradigm. Brain, 126(Pt 8), 1830-1837.
Fellows, L. K., & Farah, M. J. (2005). Different underlying impairments in decision-making following ventromedial and dorsolateral frontal lobe damage in humans. Cereb Cortex, 15(1), 58-63.
Frank, M. J., & Claus, E. D. (2006). Anatomy of a decision: striato-orbitofrontal interactions in reinforcement learning, decision making, and reversal. Psychol Rev, 113(2), 300-326.
Frank, M. J., Moustafa, A. A., Haughey, H. M., Curran, T., & Hutchison, K. E. (2007). Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning. Proc Natl Acad Sci U S A, 104(41),
16311-16316.
Frank, M. J., & O'Reilly R, C. (2006). A mechanistic account of striatal dopamine function in human cognition: psychopharmacological studies with cabergoline and haloperidol. Behav Neurosci, 120(3), 497-517.
Frank, M. J., Seeberger, L. C., & O'Reilly R, C. (2004). By carrot or by stick: cognitive reinforcement learning in parkinsonism. Science, 306(5703), 1940-1943.
Frank, M. J., Woroch, B. S., & Curran, T. (2005). Error-related negativity predicts reinforcement learning and conflict biases. Neuron, 47(4), 495-501.
Gluck, M. A., Poldrack, R. A., & Keri, S. (2008). The cognitive neuroscience of category learning. Neurosci Biobehav Rev, 32(2), 193-196.
Gluck, M. A., Shohamy, D., & Myers, C. (2002). How do people solve the "weather prediction" task?: individual variability in strategies for probabilistic category learning. Learn Mem, 9(6), 408-418.
Keri, S., Juhasz, A., Rimanoczy, A., Szekeres, G., Kelemen, O., Cimmer, C., et al. (2005). Habit learning and the genetics of the dopamine D3 receptor: evidence from patients with schizophrenia and healthy
controls. Behav Neurosci, 119(3), 687-693.
Keri, S., Kelemen, O., Szekeres, G., Bagoczky, N., Erdelyi, R., Antal, A., et al. (2000). Schizophrenics know more than they can tell: probabilistic classification learning in schizophrenia. Psychol Med, 30(1), 149155.
Keri, S., Szlobodnyik, C., Benedek, G., Janka, Z., & Gadoros, J. (2002). Probabilistic classification learning in Tourette syndrome. Neuropsychologia, 40(8), 1356-1362.
Klein, T. A., Neumann, J., Reuter, M., Hennig, J., von Cramon, D. Y., & Ullsperger, M. (2007). Genetically determined differences in learning from errors. Science, 318(5856), 1642-1645.
Knowlton, B. J., Mangels, J. A., & Squire, L. R. (1996). A neostriatal habit learning system in humans. Science, 273(5280), 1399-1402.
Knowlton, B. J., Squire, L. R., & Gluck, M. A. (1994). Probabilistic classification learning in amnesia. Learn Mem, 1(2), 106-120.
Knowlton, B. J., Squire, L. R., Paulsen, J. S., Swerdlow, N. R., Swenson, M., & Butters, N. (1996). Dissociations within nondeclarative memory in Huntington's Disease. Neuropsychology, 10, 538-548.
Lee, B., Groman, S., London, E. D., & Jentsch, J. D. (2007). Dopamine D2/D3 receptors play a specific role in the reversal of a learned visual discrimination in monkeys. Neuropsychopharmacology, 32(10), 21252134.
Pizzagalli, D. A., Bogdan, R., Ratner, K. G., & Jahn, A. L. (2007). Increased perceived stress is associated with blunted hedonic capacity: potential implications for depression research. Behav Res Ther, 45(11),
2742-2753.
Pizzagalli, D. A., Goetz, E., Ostacher, M., Iosifescu, D. V., & Perlis, R. H. (2008). Euthymic Patients with Bipolar Disorder Show Decreased Reward Learning in a Probabilistic Reward Task. Biol Psychiatry.
Pizzagalli, D. A., Jahn, A. L., & O'Shea, J. P. (2005). Toward an objective characterization of an anhedonic phenotype: a signal-detection approach. Biological Psychiatry, 57(4), 319-327.
Poldrack, R. A., & Foerde, K. (2008). Category learning and the memory systems debate. Neurosci Biobehav Rev, 32(2), 197-205.
Poldrack, R. A., & Gabrieli, J. D. (2001). Characterizing the neural mechanisms of skill learning and repetition priming: evidence from mirror reading. Brain, 124(Pt 1), 67-82.
Poldrack, R. A., Prabhakaran, V., Seger, C. A., & Gabrieli, J. D. (1999). Striatal activation during acquisition of a cognitive skill. Neuropsychology, 13(4), 564-574.
Price, A. L. (2005). Cortico-striatal contributions to category learning: dissociating the verbal and implicit systems. Behav Neurosci, 119(6), 1438-1447.
Shohamy, D., Myers, C. E., Grossman, S., Sage, J., Gluck, M. A., & Poldrack, R. A. (2004). Cortico-striatal contributions to feedback-based learning: converging data from neuroimaging and neuropsychology.
Brain, 127(Pt 4), 851-859.
Waltz, J. A., Frank, M. J., Robinson, B. M., & Gold, J. M. (2007). Selective reinforcement learning deficits in schizophrenia support predictions from computational models of striatal-cortical dysfunction. Biological
Psychiatry.
Waltz, J. A., & Gold, J. M. (2007). Probabilistic reversal learning impairments in schizophrenia: further evidence of orbitofrontal dysfunction. Schizophrenia Research, 93(1-3), 296-303.
Weickert, T. W., Terrazas, A., Bigelow, L. B., Malley, J. D., Hyde, T., Egan, M. F., et al. (2002). Habit and skill learning in schizophrenia: evidence of normal striatal processing with abnormal cortical input. Learn
Mem, 9(6), 430-442.