PSYCHOLOGY 394V - Methods, Design, & Analysis
Bob Josephs
Department of Psychology
University of Texas-Austin
Spring, '07
This course serves as an introduction to some of the issues and meta-issues surrounding scientific
inquiry, with a strong emphasis on the social and behavioral sciences, and psychology in
particular. The essence of science lies in its methodology, and the fate of any particular scientific
endeavor is, in large part, a function of the quality of the associated methodology. Methodology
is as much a way of thinking as it is a set of specific rules, guidelines, and techniques. This is not
a “how to” course. Rather, it is my hope that these readings and the discussions they provoke
will help guide your future inquiries.
At least 12 hours prior to each week’s meeting, each of you will be required to submit at least
two substantive questions based on that week’s readings. Questions will serve as a catalyst for
that week’s discussion. Send the questions via electronic mail to josephs@mail.psy.utexas.edu
Section 1- Hypothesis Generation
Nisbett, R.E. The Anticreativity Letters: Advice From a Senior Tempter to a Junior Tempter.
September, 1990. American Psychologist.
Brush, S.G. Should the history of science be rated “X”? Science, 183, 1974, 1164-1172.
Wegner, D.T. (1992). The premature demise of the solo experiment. PSPB, 504-508.
Kuhn, T.S. The Structure of Scientific Revolutions. Chaps 2-4.
*Selection 9--Karl Popper "The Myth of Inductive Hypothesis Generation."
Gholson, B., & Barker. P. Kuhn, Lakatos, and Laudan: Applications in the History of Physics
and Psychology. July 1985 American Psychologist Vol. 40, No. 7, 755-769.
Masterman, M. The nature of a paradigm. In I. Lakatos and A. Musgrave (Eds.), Criticism and
the Growth of Knowledge. Cambridge: Cambridge University Press, 1970.
McGuire, W.J. The Ying and Yang of progress in social psychology: Seven koan. Journal of
Personality and Social Psychology, 1973, 26, 446-456.
Section 2 - Hypothesis Testing I
*Selection 12--Karl Popper "Science, Pseudo-Science, and Falsifiability."
Chamberlin, T. C. The method of multiple working hypotheses. Journal of Geology, 1897, 5,
837-48.
1
Platt, J. R. Strong Inference. Science, 1964,146(3642), pp. 347-353.
*Selection 15--Robert Sekuler "In Praise of Strong Inference."
Section 3 - Hypothesis Testing II
*Introduction (Hypothesis Testing)
Mahoney, M.J. Publication prejudices: An experimental study of confirmatory bias in the peer
review system. Cognitive Therapy and Research, 1977, 1, 27-31.
Sterling, T.D., Rosenbaum, W.L., & Weinkam, J.J. Publication decisions revisited: The effect of
outcome of statistical tests on the decision to publish and vice versa. The American
Statistician, 1995, 49, 108-112.
Greenwald, A. G., Pratkanis, A. R., Leippe, M. R., & Baumgardner, M. H. Under what
conditions does theory obstruct research progress? Psychological Review, 1986,93(2) 216229.
Greenberg et al. Reaction to Greenwald et al. Psychological Review.
Petty et al. (1996). Understanding data when interactions are present or hypothesized.
Psychological Science, 7, 247-257.
Section 4 - Statistical Inference
Schmidt, F.L., & Hunter, J.E. Eight common but false objections to the discontinuation of
significance testing in the analysis of research data. In Harlow, L.L., Mulaik, S.A., &
Steiger, J.H. (Eds.). What if there were no significance tests? London: Erlbaum, 1997
(pps. 37-64).
Abelson, R.P. A retrospective on the significance test ban of 1999 (If there were no significance
tests, they would be invented). In Harlow, L.L., Mulaik, S.A., & Steiger, J.H. (Eds.). What
if there were no significance tests? London: Erlbaum, 1997 (pps. 117-141).
The following four readings were taking from Morrison, D.E., & Henkel, R.E. (1970). THE
SIGNIFICANCE TEST CONTROVERSY: A READER. Selections # 5, 17, 19, 22, 25, & 27.
Bakan, D. The test of significance in psychological research. From On Method, by David Bakan,
1967.
Lykken, D.T. Statistical significance in psychological research. From Psychological Bulletin, 70,
(1968), 151-159.
Zeisel, H. The significance of insignificant differences. From Public Opinion Quarterly, 17,
2
(1955), 319-321.
Skipper, J.K,, Guenther, A.L., & Nass, G. The sacredness of .05: A note concerning the uses of
statistical levels of significance in social science. From The American Sociologist, 2, (1967), 1618.
Folger, R. Significance Tests and the Duplicity of Binary Decisions. Psychological Bulletin
1989, Vol. 106, No. I. 155-160
Trafimow, D. Hypothesis Testing and Theory Evaluation at the Boundaries: Surprising Insights
From Bayes’s Theorem. Psychological Review, 2003, Vol. 110, No. 3, 526–535
Cumming, G. & Finch, S. Inference by Eye: Confidence Intervals and How to Read Pictures of
Data. February–March 2005 ● American Psychologist Vol. 60, No. 2, 170–180
Baron, R.M., & Kenny, D.A. The moderator-mediator distinction in social psychological
research: Conceptual, strategic, and statistical considerations. JPSP, 51, 1173-1182.
Zuckerman, M., et. al. To predict some of the people some of the time: In search of moderators.
JPSP , 1989, 57, 279-293.
Meehl, P. E. Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress
of soft psychology (pp. 826-834 optional). Journal of Consulting and Clinical Psychology,
1978, 46, 806-834.
Berscheid, E. Mea Culpas and lamentations: Sir Francis, Sir Isaac, and “The slow progress of
soft Psychology”. In The Emerging Field of Personal Relationships, 1986, 267-287.
Serlin, R.C., & Lapsley, D.K. Rationality in Psychological Research: The Good-Enough
Principle. January 1985 • American Psychologist Vol. 40, No. 1, 73-83
Schmidt, F.L. What do data really mean: Research findings, meta-analysis, and cumulative
knowledge in psychology. American Psychologist, 1992, 47, 1173-1181.
Prentice, D., & Miller, D. When small effects are impressive. Psychological Bulletin, 1992 112,
160-164.
Cohen, J. Things I have learned (so far). American Psychologist, 1990, 45, 1304-1312.
Cohen, J. The earth is round (p < .05). American Psychologist, 1994, 49, 997-1003.
Petty, R.E., Fabrigar, L.R., Wegener,D.T., & Priester, J.R. Understanding data when interactions
are present or hypothesized. Psychological Science, 1996, 7, 247-252.
Carver, R.P. The case against statistical significance testing. Harvard Educational Review, 1978,
48, 378-399.
3
Zuckerman, M., . . ., Rosenthal, R. Contemporary issues in the analysis of data: A survey of 551
psychologists. Psychological Science, 1993, 4, 49-53.
Hagen, R.L. (1997). In praise of the null hypothesis statistical test. American Psychologist, 52,
15-24.
*Selection 31--Amos Tversky and Daniel Kahneman "The Law of Small
Numbers."
Maxwell, S.E. (2004). The persistence of underpowered studies in psychological research:
Causes, consequences, and remedies. Psychological Methods, 9, 147-163.
Section 5 - Statistical and Probabilistic reasoning
Fox, R. Prejudice and the unfinished mind: A new look at an old failing. Psychological Inquiriy,
1992, 3, 137-152.
Section 6 - The ideology of experimental psychology
Sampson, E. E. Cognitive psychology as ideology. American Psychologist, 1981, 36, 730-743.
Section 7 – The limits of Psychological Science
Price, G. R. Science and the supernatural. Science, 122, 1955, 359-67.
(responses by Scott, Haskell, & Mundle in Nature, 1973, 245, 52-54).
Diaconis, P. Statistical Problems in ESP Research. Science, 201, 1978 131-136.
Meehl, P.E., & Scriven, M. Compatibility of science and ESP. Science, 123, 1956 14-15.
Price, G.R. Where is the definitive experiment? Science, 123, 1956 17-18.
Rhine, J.B., Comments on “Science and the Supernatural.” Science, 123, 1956 11-14.
Soal, S.G. On “Science and the Supernatural.” Science, 123, 1956 9-11.
Bem, D.J., & Honorton, C. Does Psi exist? Replicable evidence for an anomalous process of
information transfer. Psychological Bulletin, 1994, 115, 4-18.
Hyman, R. Anomoly or artifact? Comments on Bem and Honorton. Psychological Bulletin,
1994, 115, 19-24.
Bem, D.J. Response to Hyman. Psychological Bulletin, 1994, 115, 25-27.
Storm, L, & Ertel, S. Does Psi Exist? Comments on Milton and Wiseman's (1999) Meta-Analysis
of Ganzfeld Research. Psycholocical Bulletin 2001. Vol. 127. No. .1. 424-43.
4
Milton, J., & Wiseman, R. Does Psi Exist? Reply to Storm and Ertel (2001). Psychological
Bulletin 2001, Vol. 127, No. 3, 434-438.
Milton, J., & Wiseman, R. Does Psi Exist? Lack of Replication of an Anomalous Process of
Information Transfer. Psychological Bulletin 1999, Vol. 125, No. 4, 387-391.
Section 8 - Ecological validity versus laboratory generalizability
Zimbardo, P.G. Does Psychology Make a Significant Difference in Our Lives? July–August 2004
● American Psychologist 339 Vol. 59, No. 5, 339–351
Section 9 – Judging importance and significance of research
Davis, M. S. That's Interesting! Towards a phenomenology of sociology and a sociology of
phenomenology. Phil. Soc. Sci. I, 1971, 309-344.
Sternberg, R.J., & Gordeeva, T. The anatomy of impact: What makes an article influential.
Psychological Science, 1996, 7, 69-75.
*Note - Those selections designated with an "*" are from: Tweeney, R.D., Doherty, M.E., &
Mynatt, C.R. (Eds.). On Scientific Thinking. New York: Columbia University Press,
1981.
5