Psychology 209: Models of Cognitive Processes
Winter, 2012-2013
Syllabus
Key:
†
*
~
_
+
!
Foundational material you should know – read if you don’t already know this
Essential readings for homework or in-class discussion
Readings very closely related to lecture, adding more detail, for reference
Other readings, sometimes historical
Models we won’t cover, possible starting places for projects
Quantitatively challenging materials, sometimes very long, for those interested
Jan 7: Introduction to PDP
~ McClelland, J. L., Rumelhart, D. E. & Hinton, G. E. (1986). The Appeal of Parallel
Distributed Processing. In D. E. Rumelhart, J. L. McClelland, and the PDP Research
Group (Eds.), Parallel Distributed Processing: Explorations in the Microstructure of
Cognition. Vol I. Cambridge, MA MIT Press. Chapter 1.
~ McClelland, J. L. (2009). The place of modeling in cognitive science. Topics in
Cognitive Science, 1(1), 11-38.
† Marr, D. (2001). The Philosophy and the Approach. Chapter 1 of Vision.
San Francisco: Freeman.
Jan 9: Information integration in neurons and behavior
* McClelland, J. L. (2013). Bayesian inference, generative models, and probabilistic
computations in interactive neural networks. Draft, Jan. 6. 2013, Department of
Psychology, Stanford University. Pages 1-28.
~ Saltzman, C. Daniel and Newsome, William T. (1984). Neural Mechanisms for
Forming a Perceptual Decision. Science, 264, 231-7.
† Kolb, B. and Whishaw, I. Q. (1980). Physiological organization of the nervous system.
Chapter 2 of Fundamentals of Human Neuropsychology (pp. 31-42). San Francisco:
Freeman. (General background on cellular physiology for those with no prior exposure.
Stop at EEG Recording).
Jan 11: Introduction to the PDPtool software: the IAC model of representation of
general and specific information
* McClelland, J. L. (in prep). Explorations in Parallel-Distributed Processing: A
Handbook of models, programs, and exercises. Second Edition. Chapter 1:
Introduction.
* McClelland, J. L (in prep). Explorations in Parallel-Distributed Processing: A
Handbook of models, programs, and exercises. Second Edition. Chapter 2: Interactive
Activation and Competition. Sections 2.1 & 2.2. Exercises will be specified in
Homework.
Jan 14: Discussion of IAC Model, Relation to Exemplar Models
HOMEWORK #1 DUE
* McClelland, J. L. (1981). Retrieving general and specific information from stored
knowledge of specifics. Proceedings of the Third Annual Conference of the Cognitive
Science Society 170-172.
~ Nosofsky, R. M. (1984). Choice, similarity, and the context theory of
classification. Journal of Experimental Psychology: Learning, Memory,
and Cognition, 10, 104–114. doi:10.1037/0278-7393.10.1.104
~ Kumaran, D. & McClelland, J. L. (2012). Generalization through the recurrent
interaction of episodic memories: A model of the hippocampal system. Psychological
Review, 119, 573-616.
Jan 16: Collective computation and optimal perceptual inference
* McClelland, J. L (in progress). Explorations in Parallel-Distributed Processing: A
Handbook of models, programs, and exercises. Second Edition. Chapter 3: Constraint
Satisfaction Models. Sections 3.1, 3.2, and 3.6. Exercises will be specified in
Homework.
* J. J. Hopfield, "Neural networks and physical systems with emergent collective
computational abilities", Proceedings of the National Academy of Sciences of the USA,
vol. 79 no. 8 pp. 2554-2558, April 1982.
Jan 18: Interactive Activation: Behavioral and Brain Evidence and the IA Model
* McClelland, J. L. and Rumelhart, D. E. (1981). An interactive activation model of
context effects in letter perception: Part 1. An account of basic findings. Psychological
Review, 88, 375-407.
_ Marr, D. (1982). From images to surfaces. In Vision (pp. 99-111). San Francisco:
Freeman. An alternative perspective on interactivity
~ Lee, T. S., & Nguyen, M. (2001). Dynamics of subjective contour formation in the
early visual cortex. Proceedings of the National Academy of Sciences, 98(4), 1907-11.
~ McClelland, J. L. Mirman, D., and Holt, L. L. (2006). Are there interactive processes in
speech perception? Trends in Cognitive Sciences, 10(8), pp. 363-369. Reply by
McQueen, Norris and Cutler. Response to reply by Mirman, McClelland and Holt.
Current state of debate on whether there are interactive processes in speech perception.
Note: There are three separate files, all fairly short, with TiCS in their titles for this item.
Jan 21: Martin Luther King Day, Jr., Day (holiday, no classes)
Jan 23: Interactive Activation and Bayesian Inference
Homework #2 Due
* McClelland, J. L. (2013). Bayesian inference, generative models, and probabilistic
computations in interactive neural networks. Draft, Jan. 6. 2013, Department of
Psychology, Stanford University. Pages 28-end. Note: Find pdf in 01_09 directory.
! Movellan, J., and McClelland, J. L. (2001). The Morton-Massaro Law of Information
Integration: Implications for Models of Perception. Psychological Review, 108, 113-148.
Computational analysis extending the response to Massaro to address broader
foundational issues.
_ Dean, T. (2005). A computational Model of Cerebral Cortex. Preprint of a paper in the
Proceedings of the AAAI. Contemporary take on computational analysis of brain and
perceptual inference from a Bayesian perspective.
Jan 25: Dynamics of Decision Making
* Bogacz,R., Usher, M., Zhang, J. & McClelland, J. L. (2012). Extending a biologically
inspired model of choice: multi-alternatives, nonlinearity and value-based
multidimensional choice. In A. K. Seth, T. J. Prescott and J. J. Bryson (Eds.), Modelling
Natural Action Selection. pp 91-119. Cambridge, UK: Cambridge University Press.
~ Gold, J. I. and Shadlen, M. N. The Neural Basis of Decision Making. Annual Review
of Neuroscience, 30, 535-574
! Wong, K.F. and Wang, X.J. A recurrent network mechanism of time integration in
perceptual decisions. Journal of Neuroscience, 26, 1314-1328. Very interesting because
it links from detailed neuronal models to behavior. The on-line supplement is also
available in the class readings directory for those who are interested.
! Bogacz, R., Brown, E., Moehlis, J., Holmes, P., & Cohen, J.D. (2006). The physics of
optimal decision making: A formal analysis of models of performance in two-alternative
forced choice tasks. Psychological Review, 113, 700-65. This is a very technical paper
and is provided only for those interested.
Jan 28: Hebbian and Competitive Learning
* McClelland, J. L. (2006). How far can you go with Hebbian learning, and when does it
lead you astray? In Munakata, Y. and Johnson, M. H. Processes of Change in Brain and
Cognitive Development: Attention and Performance XXI. pp. 33-69. Oxford: Oxford
University Press.
† Gazzaniga, M. S, Ivry, R. B., & Mangan, G. R. (1998). Cellular Basis of Memory. In
Cognitive Neuroscience: The Biology of Mind (pp 283, 285-288). New Your: Norton.
(This article is background for those with no prior exposure to synapses and the biology
of synaptic modification.)
~ Wikipedia article on Kohonen’s Self-Organizing Map:
http://en.wikipedia.org/wiki/Self-organizing_map
! Miller, K.D. (1990a). Correlation-based models of neural development, in
Neuroscience and Connectionist Theory, M.A. Gluck and D.E. Rumelhart, Eds.
(Lawrence Erlbaum Associates, Hillsdale NJ), pp. 267-353. (This is a harder-thanaverage article that explains in depth how ocular dominance columns may arise from a
simple learning process. Another Miller paper (Miller, 1995) covering a wider range of
models is available in the archive. Its first two figures are missing, but are the same as
Fig 1a and 1b of the above paper.)
+ McClelland, J. L (in prep). Explorations in Parallel-Distributed Processing: A
Handbook of models, programs, and exercises. Second Edition. Chapter 6: Competitive
Learning. Read from on-line handbook if interested.
+ Vallabha, G. K., McClelland, J. L., Pons, F., Werker, J. and Amano, S. (2007).
Unsupervised learning of vowel categories from infant-directed speech. Proceedings of
the National Academy of Science, 104, 13273-13278.
Jan 30: Pattern Association: One-layer networks and learning rules
* McClelland, J. L. (in prep). Explorations in Parallel-Distributed Processing: A
Handbook of models, programs, and exercises. Second Edition. Chapter 4: Learning in
PDP Models: The Pattern Associator.
_ Rosenblatt, F. (1958). The Perceptron: A probabilistic model for information storage
and organization in the brain. Psychological Review, 65,386-408.
_ Willshaw, D. J. (1981) Holography, associative memory and inductive generalisation.
In Parallel models of associative memory. G E Hinton & J A Anderson (eds), New
Jersey, Erlbaum, 83-104.
_ Kohonen, T. Oja, E. and Lehtio, P. (1981). Storage and processing of information in
distributed associative memory systems. In Parallel models of associative memory. G E
Hinton & J A Anderson (eds), New Jersey, Erlbaum, 105-143.
Feb 1: Distributed Representations in Memory and Knowledge Representation
* McClelland, J. L. and Rumelhart, D. E. (1985). Distributed memory and the
representation of general and specific information. Journal of Experimental Psychology:
General, 114, 159-197.
_ Hinton, G. E., McClelland, J. L., & Rumelhart, D. E. Distributed Representations. In
Rumelhart, D. E., McClelland, J. L., and the PDP research group. (1986). Parallel
distributed processing: Explorations in the microstructure of cognition. Volume I.
Cambridge, MA: MIT Press.
_ Bowers, J. S. (2009). On the biological plausibility of grandmother cells: Implications
for neural network theories in psychology and neuroscience. Psychological Review, 116,
220-251.
_ Plaut, D. C., & McClelland, J. L. (2010). Locating object knowledge in the brain: A
critique of Bowers' (2009) attempt to revive the grandmother cell hypothesis.
Psychological Review, 117, 284-288.
Feb 4: The Past Tense Debate
HWK #3 Due
* McClelland, J. L., Patterson, K., Pinker, S. and Ullman, M. (2002). The Past Tense
Debate: Papers and replies by S. Pinker and M. Ullman and by J. McClelland and K.
Patterson. Trends in Cognitive Sciences, 6, 456-474.
~ Rumelhart, D. E., & McClelland, J. L. On learning the past tenses of English verbs. In
McClelland, J. L., Rumelhart, D. E., and the PDP research group. (1986). Parallel
distributed processing: Explorations in the microstructure of cognition. Volume II.
Cambridge, MA: MIT Press.
Feb 6: Back Propagation Learning Algorithm
* McClelland, J.L. (in preparation). Explorations in Parallel-Distributed Processing: A
Handbook of models, programs, and exercises, Second Edition. Chapter 5: Training
Hidden Units.
~ Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning internal
representations by error propagation. In Rumelhart, D. E., McClelland, J. L., and the
PDP research group. (1986). Parallel distributed processing: Explorations in the
microstructure of cognition. Volume I. Chapter 8, pp. 318-362. Cambridge, MA: MIT
Press.
Feb 8: Workshop: Applying Backprop to Semantic Cognition
Jay out: Cynthia to lead class
* Rumelhart, D.E. and Todd, P.M. (1993). Learning and Connectionist Representations.
Attention and Performance XIV, Synergies in Experimental Psychology, Artificial
Intelligence & Cognitive Neuroscience, D.E. Meyer and Sylvan Kornblum, Eds.
Cambridge, MA MIT Press, pp. 3-30.
_ Rogers, T.T. and McClelland, J.L. (2005). A PDP approach to semantic cognition:
Applications to conceptual development. In L. Gershkoff-Stowe and D. Rakison (Eds),
Building Object Categories in Developmental Time. Mahwah, NJ: LEA.
Feb 11: Complementary Learning Systems
HWK #4 Due
* Read one of these papers (the second is a condensed version of the key points in the
first).
McClelland, J. L., McNaughton, B. L., and O'Reilly, R. C. (1995). Why there are
complementary learning systems in the hippocampus and neocortex: Insights from the
successes and failures of connectionist models of learning and memory. Psychological
Review, 102, 419-457.
McClelland, J. L. (1996). Role of the hippocampus in learning and memory: A
computational analysis. In T. Ono, B. L. McNaughton, S. Molitchnikoff, E. T. Rolls and
H. Nichijo (Eds.), Perception Memory, and Emotion: Frontier in Neuroscience. Oxford:
Elsevier Science, Ltd. 601-613.
NOTE: Rest of Syllabus is Subject to Change
Feb 13: Simple Recurrent Networks: Finding Structure in Time
* Elman, J.L. (1990). Finding Structure in Time. Cognitive Science 14, 179-211. A
seminal article introducing an alternative approach to time and syntax.
~ Servan-Schreiber, D., Cleeremans, A., and McClelland, J.L. (1991). Graded State
Machines: The Representation of Temporal Contingencies in Simple Recurrent
Networks. Machine Learning 7, 161-193. Builds on Elman (1990) and provides the basis
for the exercises for the SRN
+ McClelland, J.L. (in preparation). Explorations in Parallel-Distributed Processing: A
Handbook of models, programs, and exercises, Second Edition. Chapter 7: Simple
Recurrent Networks.
_ Elman, J.L. (1993). Learning and development in neural networks: the importance of
starting small. Cognition, 48, 71-99. This and the next article address an interesting issue
but come to very different conclusions. A side topic for those interested.
_ Rohde, D.L.T. and Plaut, D.C. (1999). Language acquisition in the absence of explicit
negative evidence: how important is starting small? Cognition, 72, 67-109.
Feb 15: Connectionist Approaches to Language Processing
Meet with Jay or Cynthia by this Date to Plan Class Project
* Elman, J. L. (2009). On the meaning of words and dinosaur bones: Lexical knowledge
without a lexicon. Cognitive Science. 33(4), 547–582.
~ McClelland, J.L., St. John, M., and Taraban, R. (1989). Sentence Comprehension: A
Parallel Distributed Processing Approach. Language and Cognitive Processes, 4 , 287335.
_ Rohde, D.L.T. (1999). A Connectionist Model of Sentence Comprehension and
Production. Unpublished PhD thesis proposal, School of Computer Science, Carnegie
Mellon University, Pittsburgh, PA.
_ Tabor, W., Juliano, C., & Tanenhaus, M. K. (1997). Parsing in a dynamical system: An
attractor-based account of the interaction of lexical and structural constraints in sentence
processing. Language and Cognitive Processes, 12, 211-271.
Feb 18: President’s Day: No Class
Feb 20: Learning in Recurrent Networks
* Williams, R. J. and Zipser, D. (1995). Gradient-based learning algorithms for recurrent
networks and their computational complexity. In: Y. Chauvin and D. E. Rumelhart (Eds.)
Back-propagation: Theory, Architectures and Applications, Hillsdale, NJ: Erlbaum.
~ Movellan, J. R., & McClelland, J. L. (1993). Learning continuous probability
distributions with symmetric diffusion networks. Cognitive Science, 17, 463-496.
~ O'Reilly, R.C. (1996). Biologically Plausible Error-driven Learning using Local
Activation Differences: The Generalized Recirculation Algorithm. Neural Computation,
8, 895-938.
+ McClelland, J.L. (in preparation). Explorations in Parallel-Distributed Processing: A
Handbook of models, programs, and exercises, Second Edition. Chapter 8: Recurrent
Backpropagation.
Feb 22: Disorders of Lexical and Semantic Processing
One-Page Project Proposals Due
* Dilkina, K., McClelland, J. L. & Plaut, D. C. (2008). A single-system account of
semantic and lexical deficits in five semantic dementia patients. Cognitive
Neuropsychology, 25(2), 136-164.
_ Rogers, T. T., Lambon Ralph, M. A., Garrard, P., Bozeat, S., McClelland, J. L.,
Hodges, J. R., and Patterson, K. (2004). The structure and deterioration of semantic
memory: A neuropsychological and computational investigation. Psychological Review,
111, 205-235.
_ Plaut, D. C. and Shallice, T. (1993). Deep dyslexia: A case study of connectionist
neuropsychology. Cognitive Neuropsychology, 10, 377-500.
_ Plaut, D. C., McClelland, J. L., Seidenberg, M. S., and Patterson, K. (1996).
Understanding normal and impaired word reading: Computational principles in quasiregular domains. Psychological Review, 103, 56-115.
Feb 25: Executive functions and attention
* Cohen, J. D., Dunbar, K. and McClelland, J. L. (1990). On the Control of Automatic
Processes: A Parallel-Distributed Processing Account of the Stroop Effect. Psychological
Review, 97, 332-361.
~ Botvinick, M. and Plaut, D. C. (2004). Doing without schema hierarchies: A recurrent
connectionist approach to normal and impaired routine sequential action. Psychological
Review, 111, 395-429.
~ Moody, S. L., Wise, S. P., di Pellegrino, G., & Zipser, D. (1998). A model that
accounts for activity in primate frontal cortex during a delayed matching-to-sample task.
The Journal of Neuroscience, 18 (1), 399-410.
_ Cohen, J. D., Servan-Schreiber, D., & McClelland, J. L. (1992). A parallel distributed
processing approach to automaticity. American Journal of Psychology, 105, 239-269.
_ Miller, E. K. and Cohen, J. D. (2001). An Integrative Theory of Prefrontal Cortex
Function. Annual Review of Neuroscience, 24, 167-202.
Feb 27: Deep Belief Networks
* Hinton, G. E. and Salakhutdinov, R. R.(2006). Reducing the dimensionality of data
with neural networks. Science, Vol. 313. no. 5786, pp. 504 - 507.
_ Ranzato, M. Boureau, Y-L, and LeCun, Y. (2007). Sparse feature learning for deep
belief networks, Advances in Neural Information Processing Systems (NIPS 2007).
_ Bengio, Y., Lamblin, P., Popovici, D. & Larochelle, H. (2006) Greedy layer-wise
training of deep networks. Technical Report, 1282, Univ of Montreal.
Mar 1: Reinforcement Learning
*/+ McClelland, J.L. (in preparation). Explorations in Parallel-Distributed Processing: A
Handbook of models, programs, and exercises, Second Edition. Chapter 9: Temporal
Difference Learning.
Mar 4: Efficient Sensory Coding
* Olshausen, B. A. and Field, D. (2004). Sparse coding of sensory inputs. Current
Opinion in Neurobiology, 14:481-487.
~ Smith, E. C. & Lewicki, M. J. (2006). Efficient auditory coding. Nature, 439, 978-982.
~ Karklin, Y., and Lewicki, M. S. Emergence of complex cell properties by learning to
generalize in natural scenes. Nature, 457, 83-86.
March 6: The Binding Problem and Complementary Object processing systems in
Dorsal and Ventral Cortex
Lecture by Cynthia Henderson
Readings to be determined
March 8: Bayesian and Connectionist Approaches to Learning in Neural Networks
Readings to be determined
Mar 11, 13: Project Presentations
March 15: Wrap-up in class discussion
Mar 20, 5:00 pm. Final Project Due