Classical conditioning mechanisms can differentiate between seeing and doing in rats Munir G. Kutlu and Nestor A. Schmajuk Supplementary Online Material The attentional-associative model Schmajuk, Lam, & Gray model (SLG, 1996) presented an attentional-associative model of classical conditioning. The network incorporates (a) a mechanism that modulates attention to the conditioned stimuli (CSs) in proportion to the total novelty detected in the environment, and (b) a network that forms CS-CS and CS-unconditioned stimulus (US) excitatory and inhibitory associations, according to a real-time competitive rule. The model assumes that total novelty increases when (a) a predicted CS or predicted the US is absent, or (b) an unpredicted CS or unpredicted US is present. Figure 1 shows a simplified diagram of the model that illustrates the different mechanisms involved in the generation of a conditioned response (CR) when a given CS is presented. Node 1 receives input from a short-term memory trace of the CS, τCS, and the prediction of that CS, BCS, by other CSs or the context (CX). In order to modulate attention to the CSs in proportion to the novelty detected in the environment, the output of Node 1, (τCS + BCS), becomes associated through zCS with the normalized value of the total novelty detected in the environment, Novelty’. Node 3 receives input from Node 2, XCS, as well as from the error term (US - BUS). The synaptic weight connecting Node 2 to Node 3, VCS-US, reflects the (excitatory or inhibitory) association of XCS with the US. Changes in VCS-US are proportional to a common error term (US - BUS), which reflects the 1 difference between the predicted, BUS, and the real value of the US. The model describes forward (and backward sensory preconditioning in the version used in this paper), because a CS can be predicted by another CS through BCS in Node 1. Activation of Node 1 by BCS results in the activation of XCS (through zCS) in Node 2, and of BUS (through VCS-US) in Node 3. Finally, the CR is a non-linear function of BUS. Figure 1 about here 1. Short-term memory and feedback. Node 1 in Figure 1 is activated by a shortterm memory trace, τCS, of the CS. Changes in τCS are given by d τCS/dt = K1 (λCS - τCS), [ A1 ] where K1 is the rate of increase and decay of τCS and λCS is the salience of the CS. By Equation A1, τCS increases over time from zero to a maximum when the CS is present and then gradually decays back to its initial value when the CS is absent. In order to make the trace of the CSs and the Bar Press long enough to have an effect after 40 t.u. of their offset, we assumed that K1 is larger when λCS > τCS than when λCS < τCS, that is, τCS decays at a slower rate than it growths. The output of Node 1 is given by (τCS + K3 BCS), where BCS is the prediction of the CS by (a) itself, through CSi-CSi associations, (b) other CSs, through CSi-CSj associations, and (c) the context (CX), through CX-CSi associations. In the parenthesis, K3 is a feedback constant, K3 < 1 to avoid instabilities. 2.Attention. Attentional memory, zCS, reflects the association between (τCS + K3 BCS) and Novelty’. Changes in zCS are given by d zCS /dt = (τCS + K3 BCS) ( K5 Novelty’ (1- zCS) - K6 ( 1 + zCS) ), [ A2 ] 2 where K5 is the rate of increase of zCS, K6 is the rate of decay of zCS, and Novelty’ is given by Equation A8. The synaptic weight (represented by the triangle in Figure 1) connecting Node 1 to Node 2 reflects the positive value of attention z CS, which can vary between 1 and –1. The initial value of zCS is zero. When Novelty’ is larger than a certain value, zCS gradually increases; when Novelty’ is smaller than another value, zCS decreases. Notice that Novelty’ need not change for zCS to increase or decrease. Because attention zCS is controlled by Novelty’, changes in zCS always lag behind changes in Novelty’. Therefore, zCS reflects the history of exposure of the CS to previous values of Novelty’. The output of Node 2 is the attention-modulated representation of the CS, XCS. Importantly, the more negative zCS becomes the longer it will take to become positive again and have an effect on XCS. Therefore, whereas positive values of zCS can be interpreted as a measure of the attention directed to CS, negative values of zCS can be interpreted as a measure of the inattention to CS (see Lubow, 1989, page 192). Representation XCS, is given by XCS = K2 (τCS + K3 BCS) (K4 + zCS). [ A3 ] When zCS becomes negative, input (τCS + K3 BCS) activates XCS only through an unmodifiable connection K4 (not shown in Figure 1). This connection ensures that Node 1 and Node 2 are always minimally connected. Latent inhibition and compound preexposure. Equation A3 allows the model to describe latent inhibition as well as its many properties (Lubow, 1989). In addition, the model describes the retardation in conditioning to an element of a compound following 3 compound preexposure (Honey & Hall, 1989), which is similar to the case when LightTone presentations precede Light-Food presentations in the CC case and when ToneLight presentations precede Light-Food presentations in the CH case. 3. Novelty. The novelty of event k (a CS, CX or the US) is computed as the absolute value of the difference between the average observed value of that event, and the average of the sum of all predictions of that event by all active CSs and CXs: Noveltyk = | k - B k|, [ A4 ] where k is the average observed value of event k, and B k is the average aggregate prediction of event k. The average observed value of event k is given by k = ( 1 - k ) λk - K8 k, [ A5 ] where K8 is the rate of decay of k. The average aggregate prediction of event k is given by B k = ( 1 - B k ) Bk - K8 B k, [ A6 ] where K8 is the rate decay of B k. Total novelty, Novelty, at a given time is given by the sum of the novelty of all stimuli present or predicted at a given time. Novelty is given by Novelty = Σk | k - B k|, [ A7 ] 4 where k includes all CSs and the US. The normalized value of Novelty, Novelty’, is given by Novelty’ = Novelty2 / ( K92 + Novelty2 ). [ A8 ] Novelty’ is used by the attentional system to define the value of zCS (Equation A2). 4. Changes in CS-US associations. Figure 1 shows that Node 3 receives input from Node 2, XCS, as well as from the US. The synaptic weight connecting Node 2 to Node 3, VCS-US, reflects the (excitatory or inhibitory) association of XCS with the US. Changes in this association, VCS-US, are given by dVCS-US /dt = K7 XCS (λUS - BUS) ( 1 - |VCS-US|), [ A9 ] where XCS is the internal representation of the CS, BUS is the aggregate prediction of the US by all X's active at a given time (See Equation A3), and λUS is the strength of the US. As in the Rescorla-Wagner (R-W) (1972) model, changes in VCS-US are proportional to a common error term given by the difference between predicted and real values of US, (λUS - BUS). As in the Rescorla-Wagner model, conditioned inhibition is the result of presenting an excitatory CS together with a neutral CS. As suggested by Zimmer-Hart and Rescorla (1974), in contrast to the Rescorla-Wagner model and in order to prevent the extinction of conditioned inhibition (Zimmer-Hart and Rescorla, 1974) or the generation of an excitatory CS by presenting a neutral CS with an inhibitory CS (Baker, 1974), our model assumes that BUS (and BCS) takes on only positive values (BUS = 0 when BUS < 0). 5 According to Equation A9, changes in associations VCS-US are also proportional to (1 - |VCS-US|). This term (a) limits VCS-US, -1< VCS-US < +1, and (b) makes changes in VCS-US relatively slow when VCS-US approaches 1 or –1 (and relatively fast when VCS-US is close to 0). Mediated acquisition and extinction. Because XCS in Equation A9 is proportional to τCS + K3 BCS (see Equation A3), changes in VCS-US might take place even in the absence of the CS, if the CS is predicted by other CSs or the CX. That is the model is able to described mediated excitatory and inhibitory acquisition (Holland & Sherwood, 2008) and mediated extinction (Shevill & Hall, 2004). Protection from extinction. Notice that by Equation A9, dVCS-US /dt = 0 whenever λUS = BUS, that is when the US is perfectly predicted. In that case, if an excitatory CS is present together with an inhibitory CS and BUS = 0 (see Equation A10), the excitatory association will not decrease even when the US is absent, λUS = 0. We use this mechanism, referred to as “protection from extinction” (Chorazyna, 1962; Soltysik, 1985; Rescorla, 2003), to explain why responding during Observation is relatively strong even after testing in the Intervention condition. 5. Changes in CS-CS associations. Although not shown in Figure 1, the model also stores (1) associations of each XCS with its corresponding CS, e.g., VCS1-CS1; (2) associations between XCS with other CSs, e.g., VCS1-CS2 (see Rescorla and Durlach, 1981); (3) associations of XCS with the context (CX), VCS-CX (see Rescorla, 1984); (4) associations of XCX with the CS, VCX-CS (see Marlin, 1982); and (5) associations of XCX with the US, VCX-US (see Baker et al., 1981). Changes in the CS1-CS2 associations, VCS1CS2, are given by 6 dVCS1-CS2 /dt = K7 XCS1 (λCS2 - BCS2) ( 1 - |V CS1-CS2|), [ A9’] where XCS1 is the internal representation of CS1, BCS2 is the aggregate prediction of CS2 by all XCS 's active at a given time, and λCS2 is the value of CS2. Forward and backward sensory preconditioning. Equation A9’ allows the original model to describe forward sensory preconditioning, by establishing CS1-CS2 associations which can be combined with CS2-US associations. This is accomplished because the output of Node 1 is given by (τCS2 + K3 BCS2), and BCS2 = XCS1 VCS1-CS2. Therefore, when CS1 is presented in the absence of CS2, the output of Node 1 is XCS1 VCS1-CS2, which activates zCS2 and VCS2-US, to generate a CR to CS1. Even if the model is able to combine CS1-CS2 with CS2-US associations and describe forward preconditioning when CS1 preceded CS2, Equation A9’ does not yield the CS2-CS1 associations needed to describe backward sensory preconditioning. In the present study, we initially thought that backward sensory preconditioning was needed to explain why Light-Tone and LightFood presentations (CC training) result in the Tone being able to predict the Food and the conditioned nose pokes controlled by that prediction. Schmajuk and Larrauri (2006, page 10) proposed to modify Equation A9’ by replacing the actual value of the predicted CS2 (Tone), λCS2, by its trace, CS2. dVCS1-CS2 /dt = K7 XCS1 (CS2 - BCS2) ( 1 - |V CS1-CS2|), [ A9’’] This change allows the model to build VCS2-CS1 (Tone-Light) associations even when CS1 (Light) precedes CS2 (Tone) because their traces overlap. Schmajuk and Larrauri (2006) demonstrated that computer simulations run with this modification yield backward sensory preconditioning (Ward-Robinson and Hall, 1996). Therefore, in the present study 7 we use the Schmajuk-Larrauri (2006) modification of the SLG model to describe associations formed during CC training. Notice that the same CS1-CS2 associations that describe sensory preconditioning can describe second-order conditioning, in which CS1US presentations precede CS1-CS2 presentations. For consistency, λUS in Equation A9 was also replaced by λUS which allows the model to describe backward conditioning. 6. Aggregate predictions of the US and CS. The output of Node 3 in Figure 1 is the aggregate prediction of the US by all CSs with representations active at a given time, BUS, given by BUS = CS BCS-US = CS XCS VCS-US, [ A10] where BCS-US is the prediction of the US by CS and VCS-US is the association of XCS with the US. BUS is used to compute VCS-US in Equation A9, and determines the magnitude of the CR. As mentioned, although VCS-US can be either positive (excitatory) or negative (inhibitory), BUS is always positive. Not shown in Figure 1, the aggregate prediction of CS by all CSs with representations active at a given time, BCS, given by BCS = i BCSi-CS = i XCSi VCSi-CS, [ A10' ] where BCSi,CS is the prediction of CS by CSi and VCSi, CS is the association of XCSi with CS. 7.CR strength. The output of Node 3 is BUS, which directly controls the CR: according to a non-linear function CR = (1- K10 OR) BUS2 / (K112 + BUS2). [ A11 ] 8 Although the original model assumed that the orienting response (OR = Novelty’) inhibits the CR, Schmajuk and Larrauri (2006) explained that this is not the case when (a) a suppression paradigm is used because both appetitive and aversive behaviors are affected and the effects possibly cancel out, or (b) when the conditioned response is freezing, because both the CR and the OR work in the same direction. Therefore, they assumed that K10= 0. For simplicity, in the present paper we assume that CR = BUS. [ A11’ ] Similar results are obtained with Equation A11. Parameter values used in the simulations. Parameters values used in the simulations are K1= .2, K2= 2, K3= .4, K4= .1, K5= .02, K6= .005, K7= .005, K8= .005, K9= .75, K10= 0, and K11= .15. These values are identical to those used in Schmajuk et al. (1996), Schmajuk and Larrauri (2006), Schmajuk et al. (2007) and Larrauri and Schmajuk (2008), with exception of K1. As in previous publications, the rate of growth of all short-term memory traces, K1 = .2 when λCS > τCS or λBAR > τBAR. In order to extend the duration of their short-term memory traces, the rate of decay of all CS traces, including that of the Bar Press, is K1 = .02. 9 Interventions but not exogenous cues affect responding following common-cause training Leising et al.’s (2008) Experiment 1 showed that, following CC training responding was decreased following Intervention but not following Observation or the presentation of a new, “Exogenous” Cue. Causal-model theory vs. attentional-associative theory. According to the causal reasoning approach, whereas Interventions result in the removal of the Tone-Light link, that connection remains intact in the Observation and Exogenous-Cue condition (Figure 2, Upper Panel). Causal models expect that observing the barometer reading (Tone) leads to the assumption that atmospheric pressure changes (Light) were also present but missed—either in the absence or the presence of the Exogenous Cue—, and therefore expect weather changes (Food). Our associative model predicts stronger responding to Exogenous Cues than to Interventions because XEXOGENOUS CUE at the time of the Tone presentation (Figure 2, Lower Panel) is much weaker than XBAR PRESS (Figure 4, Upper Panel) thereby providing less activation to similarly weak inhibitory connections with Food. Figure 2 about here Experimental data. In Leising et al.’s (2008) Experiment 1, rats received 24 Light→Tone pairings followed by 24 Light→Food pairings (CC design). After training, a lever was placed into the testing chamber for the Intervention and Observation groups, with no consequence for the latter. Subjects in the Intervention Group received a Tone presentation with every Bar Press, while subjects in the Observation Group were yoked pairs of the Intervention Group. Subjects in an Exogenous-Cue Group were presented with a 10-s Clicker followed immediately after its offset by a 10-s Tone during the test 10 trials. Testing consisted of 60 min sessions over 1 day. For all groups, the number of nose pokes into the food magazine was recorded. Figure 3 (Upper Panel) shows that the number of nose pokes was significantly lower for the Intervention Group than those of the other two groups. Responses of the Exogenous-Cue and Observation groups did not differ. Figure 3 about here Simulated Results. Simulations with the SLG model consisted of 48 Light→Tone trials followed by 48Light→Food trials and (CC design), and 4 test trials either with (a) the Exogenous Cue followed by Tone, (b) Bar Press followed by Tone, or (c) Tone alone. During Light→Tone trials, the 40-t.u. Light preceded the 40-t.u. Tone. During Light→Food trials, the 40-t.u. Light preceded the 40-t.u. Food US. During testing of the Intervention Group, the 40-t.u. Bar Press CS started simultaneously and fully overlapped with the 40-t.u. Tone. During testing of the Exogenous-Cue group, a 40-t.u. Clicker was presented preceding the Tone. During testing of the Observation Group, the 40-t.u. Tone was presented alone. Testing consisted of 3 test trials over “1 day.” Because Bar Pressing had no consequence in the Observation groups, during the test trials a Bar lever CS was simulated in the Intervention, but not for the Observation groups. All the CS (including the Bar Press and Exogenous Cue), Magazine saliences, US strength, and ITI were as in the simulations for Leising et al.’s (2009) Experiment 3. Figure 3 (Lower Panel) shows that the CR is weaker in the Intervention case than in the Exogenous-Cue and Observation cases, which did not differ. Exogenous Cue Group. As explained so far for the Intervention groups, VBAR PRESS-FOOD becomes inhibitory during the test trials. As shown in Figure 4, VEXOGENOUS 11 CUE-FOOD FOOD, also becomes inhibitory because the absent Food is predicted by adding VMAG- VTONE-FOOD and the combination of VTONE-MAG VMAG-FOOD associations. Although the magnitudes of the inhibitory VBAR PRESS-FOOD and VEXOGENOUS CUE-FOOD associations are very similar (Figure 4, Lower Panel), the internal representation of the Bar Press (XBAR PRESS) is much stronger than the internal representation of the Exogenous Cue (XEXOGENOUS CUE) during the Tone presentation (Figure 4, Upper Panel), because the Bar Press CS is more salient than those of the Exogenous Cue. Therefore, the inhibition exerted by XBAR PRESS on responding to the Tone (Equation 2) is much stronger than that exerted by XEXOGENOUS CUE in Equation 7. CRCC-EXOGENOUS CUE (Tone) = XMAG VMAG-FOOD + XTONE VTONE-MAG VMAG-FOOD – XBEXOGENOUS CUE VEXOGENOUS CUE-FOOD [7] Predictions of the model. A prediction of our associative model, but not of causalmodel theory, is decreased responding by (a) increasing the number of test trials using the Exogenous Cue, (b) presenting the Exogenous Cue starting simultaneously and perfectly overlapping with the Tone and (c) increasing the intensity of the Exogenous Cue. The last change would make the Exogenous Cue more similar to the Bar Press CS. However, because salience is a function not only of the intensity of a CS, but also of attentional, emotional, motivational and cognitive (e.g., sensory-motor feedback) factors, it is possible that even an intense Exogenous Cue will not decrease nose poking to the low level attained by a Bar Press. Conversely, but also expected by causal-model theory, the model predicts that increasing the Bar Press-Tone interval (making the Intervention more like the Exogenous Cue used in Leising et al.’s Experiment 1) will decrease the inhibition exerted by the Intervention on responding. 12 References Baker, A. G., Mercier, P., Gabel, J. & Baker, P. A. (1981). Contextual conditioning and the US preexposure effect in conditioned fear. Journal of Experimental Psychology: Animal Behavior Processes, 7, 109–128. Chorazyna, H. (1962). Some properties of conditioned inhibition. Acta Biologiae Experimentalis, 22, 5–13. Holland, P.C., & Sherwood, A. (2008). Formation of excitatory and inhibitory associations between absent events. Journal of Experimental Psychology: Animal Behavior Processes, 34, 324-335. Honey, R. C., & Hall, G. (1989). Attenuation of Latent Inhibition after Compound Preexposure: Associative and Perceptual Explanations. The Quarterly Journal of Experimental Psychology, 418, 355-368. Larrauri, J.A., & Schmajuk, N.A. (2008). Attentional, associative, and configural mechanisms in extinction, Psychological Review, 115, 640–676. Lubow, R. E. (1989). Latent Inhibition and Conditioned Attention Theory. Cambridge: Cambridge University Press. Marlin, N. A. (1982). Within-compound associations between the context and the conditioned stimulus. Learning and Motivation, 13, 526–541. Rescorla, R. A. (1984). Associations between Pavlovian CSs and context. Journal of Experimental Psychology: Animal Behavior Processes, 10, 195–204. Rescorla, R. A. (2003). Protection from extinction. Learning and Behavior, 31, 124–132. 13 Rescorla, R. A., & Durlach, P. J. (1981). Within-event learning in Pavlovian conditioning. In N. E. Spear & R. R. Miller (Eds.), Information processing in animals: Memory mechanisms (pp. 81–112). Hillsdale, NJ: Erlbaum. Rescorla, R. A. & Wagner, A. (1972). A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and non-reinforcement. In Classical conditioning II: Current Research and Theory, eds. A. H. Black & W. F. Prokasy. New York: Appleton–Century–Crofts. Schmajuk, N., Lam, Y., & Gray, J.A. (1996). Latent inhibition: a neural network approach. Journal of Experimental Psychology: Animal Behavior Processes, 22, 321–49. Schmajuk, N.A., & Larrauri, J.A. (2006). Experimental challenges to theories of classical conditioning: Application of an attentional model of storage and retrieval. Journal of Experimental Psychology: Animal Behavior Processes, 32, 1–20. Schmajuk, N.A., Larrauri, J.A., & LaBar, K.S. (2007). Reinstatement of conditioned fear and the hippocampus: an attentional-associative model. Behavioural. Brain Research, 177, 242–253. Shevill, I., & Hall, G. (2004). Retrospective revaluation effects in the conditioned suppression procedure. Quarterly Journal of Experimental Psychology: Comparative and Physiological Psychology, 57(B), 331–347. Soltysik, S. (1985). Protection from extinction: New data and a hypothesis of several varieties of conditioned inhibition. In R.R. Miller & N.E. Spear (Eds.), Information Processing in Animals: Conditioned Inhibition. Hillsdale, NJ: Lawrence Erlbaum. 14 Ward-Robinson, J., & Hall, G. (1996). Backward sensory preconditioning. Journal of Experimental Psychology: Animal Behavior Processes, 22, 395–404. Zimmer-Hart, C. L. & Rescorla, R. A. (1974). Extinction of Pavlovian conditioned inhibition. Journal of Comparative and Physiological Psychology, 86, 837–845. 15 Predicted CS B CS CS-US Association Attention to CS CS US - B US Novelty’ 3 3 2 2 1 1 z CS X CS V CS,US CR B US Figure 1. A simplified diagram of the SLG model. CS: conditioned stimulus; US: unconditioned stimulus; CS: trace of the CS; zCS: attention to the CS; XCS: internal representation of the CS; VCS-US: CS-US association; BUS; prediction of the US; CR: conditioned response. Triangles: variable connections (associations) between nodes that modulate the activation of the node. Arrows: inputs that control the output of the node. Solid circles: inputs that modify connections z CS and VCS-US without affecting outputs X CS and CR. 16 Causal-Model Theory Light Exogenous Cue Tone Food SLG model MAGAZINE Light Exogenous Cue Tone - Food Figure 2. Explanation of the effect of exogenous cues. Causal-model theory. Upper Panel: Exogenous Cue Intervention preserves the Tone-Light link (Solid Arrow). SLG Model. Lower Panels: Responding in the Common Cause-Intervention Group is determined by (a) the strong prediction of Food based on VMAG-FOOD associations (MAG: Magazine), (b) the weak direct prediction of Food based on the VTONE-FOOD association, (c) the weak indirect prediction of Food based on the combination of VTONE-MAG (MAG: Magazine) and VMAG-FOOD associations, and (d) the VEXOGENOUS CUE-FOOD inhibitory association activated by the weak XEXOGENOUS CUE (Thin Box). The minus signs indicate inhibition. Dotted arrows and lines represent, respectively, weak excitatory and inhibitory associations. 17 Leising et al. (2008) - Experiment 1 Mean Nose Pokes 25 20 15 10 5 0 Intervention Observation Exogenous-Cue Groups Simulations Conditioned Response 0.2 0.15 0.1 0.05 0 Intervention Observation Exogenous-Cue Groups Figure 3. Responding during presentation of the Tone in the Intervention, Observation, and Exogenous Cue groups. Upper Panel: Data from Leising et al.’s (2008) Experiment 1. Lower Panel: Simulated results with the SLG model showing the average strength of the CR during the presentations of the Tone during 3 test trials. 18 Internal Representations (Xs) 2 Arbitrary Units 1.6 1.2 0.8 0.4 0 XTONE XMAGAZINE XBAR PRESS XEXO CUE VBAR PRESSFOOD VEXO CUEFOOD Associations (Vs) Arbitrary Units 0.6 0.4 0.2 0 -0.2 VTONEFOOD VTONEMAGAZINE Intervention VMAGAZINEFOOD Observation Exogenous-Cue Figure 4. Average associations and representations for Intervention, Observation, and Exogenous Cue groups during testing. Upper Panels: Representations of the Magazine (XMAG), Tone (XTONE), Exogenous Cue (XEXOGENOUS CUE), and Bar Press (XBAR PRESS). Lower Panels: Corresponding associations VMAG-FOOD, VTONE-FOOD, VTONE-MAG, VEXOGENOUS CUE-FOOD, and VBAR PRESS-FOOD, for Intervention (Open Bars), Observation (Solid Bars) and Exogenous Cue (Diagonally Striped Bars) groups. 19