Differentiating Models of Associative Learning: Reorientation, Superconditioning,
and the Role of Inhibition
Brian Dupuis (bdupuis@ualberta.ca)
Michael R.W. Dawson (mdawson@ualberta.ca)
Department of Psychology, University of Alberta
Edmonton, Alberta, Canada
Keywords: Rescorla–Wagner model; artificial neural
networks; operant choice; reorientation; superconditioning.
locations with the correct geometric configuration in
absence of the correct feature.
Although developed with reorientation in mind, the
Miller-Shettleworth model shows signs of a serious
mathematical mistake when tested on this task. At high
learning rates, the model predicts dramatic fluctuations in
associative strength (Figure 1) and in choice probabilities,
eventually culminating in a global divide-by-zero error.
The Miller-Shettleworth Model
One broad class of models of associative learning, based
on Rescorla and Wagner’s (1972) original model, views
stimuli as collections of cues that compete with each other
for associative strength. Miller and Shettleworth (2007,
2008) attempted to apply a variation of this model to the
field of spatial learning. Their model attempts to capture the
role of choice behavior in spatial navigation tasks –
specifically, how agents estimate the likelihood of
reinforcement based on their experience, and how their
choices are influenced by these estimates. They observe,
correctly, that this makes such tasks exercises in operant
conditioning, while the well-established Rescorla-Wagner
model captures classical conditioning.
In their model, Miller and Shettleworth multiply the
Rescorla-Wagner equation by a “probability” term – a ratio
of the associative strengths a single choice’s cues to the total
associative strength of all cues at all possible choices.
However, this produces a model that is empirically and
mathematically flawed (Dupuis & Dawson, in press). Here,
we describe these empirical and formal flaws, and supply an
alternative associative model for investigating these tasks.
Figure 1: The Miller-Shettleworth model’s associative
strength on a reorientation task at a high (0.7) learning rate.
Superconditioning
Superconditioning arises when an excitatory cue paired
with an inhibitor produces greater excitation during further
training than it produces when paired with a neutral cue.
This is a prediction of the Rescorla-Wagner (1972) model
and is well-established in animal experimental literature.
A recent experiment (Horne & Pearce, 2010) established
that this effect also occurs in spatial learning – that is, rats
trained with an inhibitory feature responded to geometryonly probe trials with greater probability than rats trained
with a neutral feature. However, when the MillerShettleworth (2008) model attempts to model this effect, it
predicts the opposite result (experimental 0.91, control
0.94). Horne and Pearce observed that the model was not
assigning sufficient inhibitory associative strength to cues
that are present at both reinforced and non-reinforced
locations.
Empirical Shortcomings
The Miller-Shettleworth model’s flaws are evident when
one investigates how it behaves in a pair of spatial tasks.
Reorientation
A ‘reorientation task’ is a common experimental paradigm,
used to explore spatial and geometric learning, where agents
are placed inside a controlled arena – typically rectangular,
with an assorted set of feature information or landmarks
such as colored panels over the corners. The subjects must
learn which locations are or are not reinforced. Systematic
change to this arena after training produces regular effects,
most famously “rotational error”. If the colored panel is
moved from the reinforced corner to a different corner,
agents will follow it, but they will also return to its original
corner… as well as the corner rotationally opposite (which
has walls in the same configuration as the original corner).
In an associative context, each location is defined as the
collection of cues present at that location – the geometric
configuration of the walls, the type of features present, and
so on. Rotational error is explained as a response to
Why Does It Fail?
This weakness in handling inhibition is indicative of why
the Miller-Shettleworth model produces such unusual
results. The root cause lies with their decision to implement
operant choice by scaling the Rescorla-Wagner equation.
The Rescorla-Wagner model includes an implicit measure
of the time that passes with each iteration of the equation
within its learning rate parameter – a term that is held
constant (and thus suppressed), because to do otherwise
322
would “beg justification” (Rescorla & Wagner, 1972).
When one calculates the change in associative strength as
this change in time approaches zero, the Rescorla-Wagner
equation produces the instantaneous time derivative of
associative strength.
Miller and Shettleworth (2007, 2008) multiply the
Rescorla-Wagner equation by some “probability” ratio. The
problem is that both of these equations are functions of
current associative strength – meaning both equations
contain a (suppressed) time term. In order to correctly
control for this additional time dependency, the chain rule
must be applied to the equation or the form of the derivative
will change. Miller and Shettleworth did not do this.
In effect, this uncontrolled time dependency is equivalent
to allowing the learning rate to vary independently for each
type of cue, with none of the justification that Rescorla and
Wagner “beg”.
is, it is not normalized), which has important theoretical
implications.
Empirical Robustness
When tested on the same reorientation task illustrated
above, the operant perceptron converges upon the expected
solution at low (0.15), high (0.7), and extremely high (1.0)
learning rates, illustrating that it does not succumb to the
scaling problems seen in the Miller-Shettleworth model. (A
discussion of perceptrons and reorientation is found in
Dawson et al. (2010).)
Presenting Horne and Pearce’s (2010) superconditioning
experiment to the operant perceptron leads to a prediction
consistent with their animal experiments (experimental
0.998, control 0.970). Examining the operant perceptron’s
behavior over time shows that it assigns substantially more
inhibitory associative strength to partially-reinforced cues
than the Miller-Shettleworth model – a result of applying
the full learning rule at some frequency rather than a scaled
rule.
Solution: The Operant Perceptron
In light of these empirical and mathematical results, we
must recommend that the Miller-Shettleworth model be
abandoned. However, this need not spell the end for
associative theory in exploring these areas. An alternative
model – a simple artificial neural network known as a
perceptron – has been shown to successfully model many of
the standard reorientation task results (Dawson, Kelly,
Spetch, & Dupuis, 2010), even though it employs a
classical-conditioning training algorithm. This perceptron is
presented a pattern of cues representing a location in a
reorientation arena, which is then sent through weighted
connections to an output unit. This output unit sums the
weighted signals together and responds with the logistic
function of this value; mistakes in this response are then
used to modify the connection weights using a gradientdescent learning rule based on the Rescorla-Wagner
equation.
A simple modification to this algorithm to capture the
probabilistic choice behavior in operant conditioning
produces an “operant perceptron”. After a location’s cues
are presented, the operant perceptron generates its logistic
output – a number that must fall between 0 and 1. This has
been shown to literally be the estimate of the conditional
probability of reinforcement given the pattern of cues
(Dawson & Dupuis, 2012). Therefore, this output response
is used as the probability of the operant perceptron
investigating a location. If it does not investigate a location,
its weights are not changed – as in operant conditioning, the
operant perceptron only learns from its experience.
A critical difference between the operant perceptron and
the Miller-Shettleworth model is that the latter applies a
scaled Rescorla-Wagner equation with every iteration, while
the former applies a normal, unscaled Rescorla-Wagnerstyle equation on some subset of iterations. This allows the
operant perceptron to bypass the calculus error described
above. Furthermore, the operant perceptron generates
conditional probabilities for each option independently (that
References
Dawson, M. R. W., & Dupuis, B. (2012). The equilibria of
perceptrons for simple contingency problems. IEEE
Transactions in Neural Networks and Learning
Systems, 23(8), 1340–1344.
doi:10.1109/TNNLS.2012.2199766
Dawson, M. R. W., Kelly, D. M., Spetch, M. L., & Dupuis,
B. (2010). Using perceptrons to explore the
reorientation task. Cognition, 114(2), 207–26.
doi:10.1016/j.cognition.2009.09.006
Dupuis, B., & Dawson, M. R. W. (in press). Differentiating
Models of Associative Learning: Reorientation,
Superconditioning, and the Role of Inhibition. Journal
of Experimental Psychology: Animal Behavior
Processes (36 pages, accepted February 4, 2013).
Horne, M. R., & Pearce, J. M. (2010). Conditioned
inhibition and superconditioning in an environment
with a distinctive shape. Journal of Experimental
Psychology: Animal Behavior Processes, 36(3), 381–
94. doi:10.1037/a0017837
Miller, N. Y., & Shettleworth, S. J. (2007). Learning about
environmental geometry: An associative model.
Journal of Experimental Psychology: Animal
Behavior Processes, 33(3), 191–212.
doi:10.1037/0097-7403.33.3.191
Miller, N. Y., & Shettleworth, S. J. (2008). An associative
model of geometry learning: a modified choice rule.
Journal of Experimental Psychology: Animal
Behavior Processes, 34(3), 419–22.
doi:10.1037/0097-7403.34.3.419
Rescorla, R. A., & Wagner, A. R. (1972). A theory of
Pavlovian conditioning: Variations in the
effectiveness of reinforcement and nonreinforcement.
In A. H. Black & W. F. Prokasy (Eds.), Classical
conditioning II: current research and theory (pp. 64–
99). New York, NY: Appleton-Century-Crofts.
323