Research Problems – Spring 2015: Willcutt
1
RESEARCH PROBLEMS IN CLINICAL PSYCHOLOGY
PSYC 5423
Spring Semester, 2015
Instructor: Erik Willcutt, Ph.D.
Office: Muenzinger D-313C
Work phone: 303-492-3304
Home phone: 303-926-8844
Cell phone: 303-916-9148 (it's fine to call or send text messages to my cell)
Email: erik.willcutt@colorado.edu
Schedule: Tuesday 11:30 - 2:00; Muenzinger D216
Course wiki:
All course materials will be provided in PDF format on the course wiki. You should be able to open the wiki using your
department ID and password (ask Ernie Mross if you aren't sure what this is).
Wiki address: http://psych.colorado.edu/wiki/doku.php?id=courses:willcutt:5423:home.
COURSE DESCRIPTION
Course Goals:
Psyc 5423 is an intensive graduate course that provides a survey of research design, research criticism, and proposal
writing. The first primary aim of the course is to enable students to become proficient with the fundamentals of research
design. Primary topics will include methods for systematic literature reviews, hypothesis formulation and operationalization,
inference testing, advantages and disadvantages of design alternatives, measurement and assessment strategies for clinical
research, and selection and implementation of appropriate statistical procedures. The second overarching goal of the course
is to facilitate the development of writing skills that will enable students to write empirical papers and research proposals that
will be selected for publication or external funding.
Course Format:
To accomplish these objectives, students will be exposed to information through class lectures, assigned readings,
and class discussion. These sources of information are designed to complement one another. The readings will broadly
cover topics related to research design. The content of the lectures will overlap with the readings to a certain extent, but
will also provide specific context and applied examples which will facilitate the learning process. The course will focus
heavily on the application of research design and will emphasize class discussion.
Textbook:
The following textbook is under consideration for adoption in future years. You don’t need to buy it this year, but I will be
assigning several chapters for you to read in addition to the readings listed on the syllabus. I always appreciate comments
(whether positive or negative) about all of our readings, but am especially interested in your impressions of this book.
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized
causal inference. Boston: Houghton-Mifflin.
Other readings:
All readings are provided in PDF format on the course wiki. For most class sessions there will be 3 - 5 required readings.
In addition, the reading list that starts on page 8 also includes a number of supplemental readings for each major content area.
The supplemental readings are not required for the course. One of my primary goals for this course is to provide you with
resources regarding different aspects of research design that may be useful to you later in graduate school and in your future
career as an independent researcher. Therefore, for each topic the reading list includes a range of "classic" papers that
provide useful overviews or different perspectives, along with papers that discuss more specific topics that may only be
relevant for some research questions or study designs. On a related note, I am always looking for useful new articles to add to
the list, so if you find any articles that are especially helpful during the course or later in your training, please forward them to
me.
Research Problems – Spring 2015: Willcutt
2
COURSE REQUIREMENTS
I. Class attendance and participation:
I.A. Overall attendance and participation (20% of final grade): Although the content of the course requires much of our
time to be devoted to presentation of information by lecture (especially early in the semester), I have structured the course to
empasize discussion as much as possible. Students are expected to read the assigned materials prior to class and to be
prepared to discuss those materials during class. In addition, you will complete several brief homework assignments to
help you to consolidate the information presented in class. These will be announced during class.
II. Research reviews:
II.A. Manuscript review #1 (10% of final grade; due in class on February 3rd). Read Reynolds & Nicolson (2007)
and prepare a "bullet point" critique summarizing your reaction to the paper (positive and negative) for discussion in class.
Think about both the specific content / logic of the paper and your more general "gut-level" reactions to the style of
presentation. I realize that you don't have detailed knowledge about this area of research - the goal of this assignment is
just to get us all thinking about these issues, so please don't let this one stress you out.
II.B. Manuscript review #2 (10% of final grade; due March 10th). You will review an empirical paper that I will
distribute in the format that is used for blind reviews for a clinical psychology journal. I will provide several sample reviews as
examples before you are required to write your own review.
II.C. Manuscript review #3 (10% of final grade; due April 21st). Read the meta-analysis by Rind et al. (1998) and
prepare a bullet point critique for discussion in class.
II.D. Review of popular press article (10% of final grade; can be completed anytime, due by email by May 9th).
As you read articles in the popular press, watch for articles that make an "error in thinking" that is relevant to the issues
covered in this course ("relevant" can be interpreted liberally). Newspapers, magazines, and the internet are all fine - just
make a copy of the article or send along the link so I know what you read. In no more than 2 - 3 pages, describe the error
that you spotted, and explain why it is an error. Then, in your role as empathic skeptic, discuss why you think the error
was made. Things to think about could include:
1. Why does the error matter in the big picture?
2. Why was the author of the article susceptible to the error? Did the original source of the information play a role in
the error? (i.e., did the author of the article just misinterpret the source, miss a subtle point, or frame the
information from the source in a way that contradicts its content, or was the original source material misleading?)
3. Why might members of the public be susceptible to believe the error?
4. How would you change the story? Is the information in the article simply wrong, or is it a more subtle mistake that
could be presented more appropriately by providing adequate context, discussion of caveats, etc.?
III. Foundation of NIH F31 individual fellowship proposal (40% of final grade; due May 9th)
The final assignment is to write the framework for a proposal for an NIH Ruth Kirschtein National Research Service
Award Individual Predoctoral Fellowship. In a perfect world each of you will use the final product from this course as the
foundation for a submitted proposal for the NRSA deadline in December of your second year in the program. We will
discuss the specific details of this assignment extensively throughout the semester. You will complete the following
sections by the end of the course:
Specific Aims: Full written draft
Significance: Full written draft
Approach: Full written draft
Data analysis: detailed outline
Training Plan: detailed outline
Research Problems – Spring 2015: Willcutt
3
POLICIES
Accommodations for a documented disability:
I encourage students with documented disabilities, including nonvisible disabilities such as chronic diseases,
learning disabilities, head injury, attention deficit / hyperactivity disorder, or psychiatric disabilities, to discuss with me
possible accommodations so that your learning needs may be appropriately met.
If you qualify for accomodations because of a disability, please see me during the first two weeks of class.
University policy requires that you provide documentation from Disability Services (disabilityservices.colorado.edu; 303492-8671, dsinfo@colorado.edu). Disability Services determines accommodations based on documented disabilities.
Please be assured that I will keep any information that you provide confidential.
Academic Honesty:
All students of the University of Colorado at Boulder are responsible for knowing and adhering to the academic
integrity policy of this institution. Violations of this policy may include: cheating, plagiarism, aid of academic dishonesty,
fabrication, lying, bribery, and threatening behavior. All incidents of academic misconduct shall be reported to the Honor
Code Council (honor@colorado.edu; 303-735-2273). Students who are found to be in violation of the academic integrity
policy will be subject to both academic sanctions from the faculty member and non-academic sanctions (including but not
limited to university probation, suspension, or expulsion). Other information on the Honor Code can be found at
http://www.colorado.edu/policies/honor.html and at http://honorcode.colorado.edu.
Classroom behavior policy:
Students and faculty each have responsibility for maintaining an appropriate learning environment. Those who fail to
adhere to such behavioral standards may be subject to discipline. Professional courtesy and sensitivity are especially
important with respect to individuals and topics dealing with differences of race, color, culture, religion, creed, politics,
veteran’s status, sexual orientation, gender, gender identity and gender expression, age, disability, and nationalities.
Class rosters are provided to the instructor with the student's legal name. I will gladly honor your request to address you
by an alternate name or gender pronoun. Please advise me of this preference early in the semester so that I may make
appropriate changes to my records.
Professional courtesy and sensitivity are especially important with respect to individuals and topics dealing with
differences of race, color, culture, religion, creed, politics, veteran’s status, sexual orientation, gender, gender identity and
gender expression, age, disability, and nationalities. The University of Colorado Boulder and the instructors for this course will
not tolerate acts of discrimination or harassment on the basis of race, color, national origin, sex, age, disability, creed, religion,
sexual orientation, or veteran status. Individuals who believe they have been discriminated against should contact the Office of
Discrimination and Harassment (ODH) at 303-492-2127 or the Office of Student Conduct (OSC) at 303-492-5550. Information
about campus resources available to assist individuals regarding discrimination or harassment can be obtained at
http://hr.colorado.edu/dh/.
Observation of religious holidays and obligations:
University of Colorado at Boulder policy regarding religious observances requires that faculty make every effort to
deal reasonably and fairly with all students who, because of religious obligations, have conflicts with scheduled exams,
assignments or required attendance. In this class we have attempted to schedule the four examinations on dates that do
not conflict with religious holidays. However, we recognize that we may not be aware of all potential conflicts. Please
review the dates of the four exams listed in the course schedule, and let me know during the first two weeks of class if you
will be observing a religious holiday on one of those dates so that we can arrange appropriate accommodations.
Research Problems – Spring 2015: Willcutt
4
TENTATIVE COURSE SCHEDULE AND ASSIGNMENTS
NOTE: ALL READINGS ARE AVAILABLE ON THE COURSE WIKI
1/13: WHAT IS THIS CAREER THAT WE HAVE CHOSEN? (AND WHY IN THE WORLD DID WE MAKE THAT CHOICE?)
Professional issue: The pros and cons of a tenure-track or other academic life.
1/20: INTRODUCTION TO GRANT WRITING AND THE F31
**Due: Very brief (3 - 5 minute at most) summary of your research ideas**
F31: Please read the SF424 Individual Fellowship Application Guide (PDF under "F31 resources" on the wiki)
Professional Issues: Writing a successful grant proposal: solutions to common problems
READINGS:
Bordage, G., & Dawson, B. (2003). Experimental study design and grant writing in eight steps and 28 questions.
Medical Education, 37, 376 - 385. [A nice resource - you can focus on big picture points in this one.]
Calof, A. (1999). Grant-writing amnesia. Current Biology, R869.
Oetting, E. R. (1986). Ten fatal mistakes in grant writing. Professional Psychology: Research and Practice, 17,
570 - 573.
Rasey, J. S. (1999). The art of grant writing. Current Biology, R387.
1/27: PHILOSOPHY OF SCIENCE AND CAUSAL INFERENCE
F31: work on your Specific Aims
Professional Issues: Writing peer reviews. Please download the sample reviews that will be posted on the wiki.
READINGS:
Kuhn, T. (1991) Scientific Revolutions. In R. Boyd, P. Gasper, and J. D. Trout (Eds.), The philosophy of science,
pp. 139 - 158. Cambridge, MA: MIT Press. [A dense but relatively concise summay of the philsophical
perspective of one of the most important voices in science.]
Lovejoy, T.I., Revenson, T.A., & France, C.R. (2011). Reviewing manuscripts for peer-review journals: a primer
for novice and seasoned reviewers. Annals of Behavioral Medicine, 42, 1-13.
Shadish, Cook, & Campbell, Chapter 1: Experiments and generalized causal inference (pp. 1 - 32).
Shermer, M. (1997). How thinking goes wrong: Twenty-five fallacies that lead us to believe strange things. In Why
People Believe Strange Things (pp. 44 - 61). New York: Freeman.
West, S. G., & Thoemmes, F. (2010). Campbell's and Rubin's perspectives on causal inference. Psychological
Methods, 15, 18-37. [A concise summary of two of the most prominent theoretical models of causal inference
in the field. Don't worry if a few of the more dense statistical sections are difficult to follow, and just read it for
"big picture" content]
2/3: THEORETICAL MODELS AND PUBLIC SKEPTICISM
**Due: Assignment II.A.: Bullet-point critique of Reynolds and Nicolson (2007)**
Due Friday 2/6 by 5 PM: Detailed outline / rough draft of aims.
Professional issue: Why is there public skepticism about psychological research?
READINGS:
Lilienfeld, S. O. (2012). Public skepticism of psychology: why many people perceive the study of human behavior
as unscientific. American Psychologist, 67, 111-129.
Richard, F. D., Bond, C. F., & Stokes-Zoota, J. J. (2001). “That’s completely obvious . . . and important”: Lay
judgments of social psychological findings. Personality and Social Psychology Bulletin, 27, 497-505.
Wicker, A. W. (1985). Getting out of our conceptual ruts: Strategies for expanding conceptual frameworks.
American Psychologist, 40, 1094-1103. [this paper begins to address the importance of theoretical models to
guide both the literature review and new empirical studies]
Research Problems – Spring 2015: Willcutt
5
2/10: INTERNAL AND STATISTICAL CONCLUSION VALIDITY
F31: Continue to work on your aims and begin to work on your Significance section. (Note: this is often a good time
to meet with me one on one if you are feeling stuck)
READINGS:
Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155-159. [overview of power by one of the most
influential figures in this specific area. Includes his take on estimates of small, medium, and large effect sizes]
Shadish, Cook, & Campbell, Chapter 2: Statistical conclusion validity and internal validity.
2/17: CONSTRUCT VALIDITY, RELIABILITY, AND SCALE DEVELOPMENT
**DUE BY FRIDAY 2/20: FULL DRAFT OF AIMS**
F31: Complete full draft of Specific Aims and continue to work on the Significance section
READINGS:
Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues in objective scale development.
Psychological Assessment, 7, 309 - 319.
Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281302.
Podsakoff, P. M., MacKenzie, S. B., Lee, J., & Podsakoff, N. P. (2003). Common method biases in behavioral
research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88,
879-903. [A nice summary of strengths and weaknesses of different designs. The tables are an especially
nice resource]
Shadish, Cook, & Campbell, Chapter 3: Construct validity and external validity (pp. 64 - 82) [Note that only the
first half of the chapter is due this week].
2/24: EXTERNAL VALIDITY AND GENERALIZATION
F31: Continue to work on the Significance section
Professional Issue: Designing studies that take culture and diversity into consideration.
READINGS:
Sears, D. O. (1986). College sophomores in the laboratory: Influences of a narrow data base on social
psychology’s view of human nature. Journal of Personality and Social Psychology, 51, 515 - 530.
Shadish, Cook, & Campbell, Chapter 3: Construct validity and external validity (pp. 83 - 92).
Smedley, A., & Smedley, B. D. (2005). Race as biology is fiction, racism as a social problem is real:
Anthropological and historical perspectives on the social construction of race. The American Psychologist,
60, 16 - 26. [Key points about the biological meaning of our racial categorizations (especially "census level"
categories) with important implications for prominent (and inflammatory) hypotheses about racial differences.]
3/3: STUDY DESIGN I: RANDOM SAMPLING AND RANDOMIZED EXPERIMENTS
**DUE BY FRIDAY 3/6: DETAILED OUTLINE OF SIGNIFICANCE SECTION**
Professional Issue: Finding adequate samples with limited resources.
READINGS:
Shadish, Cook, & Campbell, Chapter 8: Randomized experiments: Rationale, designs, and conditions conducive
to doing them (pp. 246 - 277).
Shadish, Cook, & Campbell, Chapter 9: Practical Problems 1: Ethics, participant recruitment, and random
assignment. (pp. 279 - 313).
Wainer, H. (1999). The most dangerous profession: A note on nonrandom sampling error. Psychological
Methods, 3, 250 - 256. [describes potentially important implications of the cases that do not get included by
your sampling design]
Research Problems – Spring 2015: Willcutt
6
3/10: STUDY DESIGN II: QUASI-EXPERIMENTAL AND CASE-CONTROL DESIGNS
**DUE: ASSIGNMENT II.B. REVIEW #2.**
READINGS:
Shadish, Cook, & Campbell, Chapter 4: Quasi-experimental designs that either lack a control group or lack
pretest observations on the outcome. (pp. 103 - 134).
Shadish, Cook, & Campbell, Chapter 5: Quasi-experimental designs that use both control groups and pretests.
(pp. 135 - 169).
West, S.G., & Thoemmes, F. (2008). Equating Groups. In P. Alasuutari, L. Bickman, & J. Brannen. The SAGE
handbook of social research methods. London: SAGE.
3/17: STUDY DESIGN III: TREATMENT STUDIES AND OTHER LONGITUDINAL DESIGNS
**DUE BY FRIDAY 3/20: FULL DRAFT OF AIMS AND SIGNIFICANCE**
READINGS:
Shadish, Cook, & Campbell, Chapter 10: Practical problems 2: Treatment implementation and attrition. (pp. 314 340).
Clarke, G. N. (1995). Improving the transition from basic efficacy research to effectiveness studies:
Methodological issues and procedures. Journal of Consulting & Clinical Psychology, 63, 718 - 725.
Hsu, L. M. (1989). Random sampling, randomization, and equivalence of contrasted groups in psychotherapy
outcome research. Journal of Consulting & Clinical Psychology, 57, 131 - 137. [A classic paper summarizing
the difference between random sampling and randomization for treatment studies]
Lilienfeld, S. O., Ritschel, L. A., Lynn, S. J., Cautin, R. L., & Latzman, R. D. (2014). Why ineffective
psychotherapies appear to work: A taxonomy of causes of spurious therapeutic effectiveness. Perspectives
on Psychological Science, 9, 355-387.
3/24: NO CLASS, SPRING BREAK
3/31: STATISTICAL INFERENCE AND INTERPRETATION I: OVERVIEW AND THREATS TO STATISTICAL CONCLUSION VALIDITY
**DUE BY FRIDAY 4/3: DETAILED OUTLINE OF APPROACH**
Professional issue: Real-life data management
READINGS:
Cohen, J. (1990). Things I've learned (so far). American Psychologist, 45, 1304-1312. [Classic summary of "Big
Picture" issues as we dive into statistical analysis.]
DeCoster, J., Iselin, A. M., & Gallucci, M. (2009). A conceptual and empirical examination of justifications for
dichotomization. Psychological Methods, 14, 349-366. [examines researchers' justifications for
dichotomization, then tests whether their rationale is supported empirically. Read this one for big picture
points, and don't worry about the details of the simulation models.]
Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7,
147-177. [A classic paper that describes the strengths and weaknesses of a range of solutions to deal with
missing data]
4/7: STATISTICAL INFERENCE AND INTERPRETATION II: SIGNIFICANCE TESTS, CONFIDENCE INTERVALS, AND EFFECT SIZES
READINGS:
Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997-1003. [Position paper by one of
the most influential methodologists in our field. Argues against null hypothesis significance testing.]
Krueger, J. (2001). Null hypothesis significance testing: On the survival of a flawed method. American
Psychologist, 56, 16-26. [summarizes key arguments that have been advanced for and against null
hypothesis significance testing]
Schmidt, F. L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for
training of researchers. Psychological Methods, 1, 115-129. [argues against null-hypothesis significance
testing and in favor of point estimates of effect size with confidence intervals]
Schmidt, F., & Hunter, J. (2002). Are there benefits from NHST? American Psychologist, 57, 65-71. [a brief
response to Krueger]
Research Problems – Spring 2015: Willcutt
7
4/14: STATISTICAL INFERENCE AND INTERPRETATION III: STATISTICAL POWER AND REPLICATION
**DUE BY FRIDAY 4/18: FULL DRAFT OF APPROACH**
Professional issue: How do we handle the fact that our studies are nearly always underpowered?
G*Power3 computer program: http://www.psycho.uni-duesseldorf.de/abteilungen/aap/gpower3/
READINGS:
Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155-159. [overview of power by one of the most
influential figures in this specific area. Includes his take on estimates of small, medium, and large effect sizes]
Hallahan, M., & Rosenthal, R. (1996). Statistical power: Concepts, procedures, and applications. Behaviour
Research and Therapy, 34, 489-499. [Nice overview of the issues with suggestions regarding ways to
increase power]
Kraemer, H. C., Mintz, J., Noda, A., Tinklenberg, J., & Yesavage, J. A. (2006). Caution regarding the use of pilot
studies to guide power calculations for study proposals. Archives of General Psychiatry, 63, 484 - 489.
Makel, M. C., Plucker, J. A., & Hegarty, B. (2012). Replications in psychology research: How often do they really
occur? Perspectives on Psychological Science, 7, 537-542.
Stanley, D. J., & Spence, J. R. (2014). Expectations for replications: Are yours realistic? Perspectives on
Psychological Science, 9, 305-318.
4/21: META-ANALYSIS, DATA SYNTHESIS, AND ETHICAL ISSUES IN RESEARCH
**DUE: ASSIGNMENT II.C. RIND CRITIQUE**
READINGS:
Brown, S. D., Furrow, D., Hill, D. F., Gable, J. C., Porter, L. P., & Jacobs, W. J. (2014). A duty to describe: Better
the devil you know than the devil you don't. Perspectives on Psychological Science, 9, 626-640.
Ferguson, C. J., & Brannick, M. T. (2012). Publication bias in psychological science: Prevalence, methods for
identifying andf controlling, and implications for the use of meta-analysis. Psychological Methods.
4/28: OTHER ETHICAL ISSUES
**DUE BY FRIDAY 5/1: FULL DRAFT OF TRAINING PLAN**
Professional Issue: Human Subjects and writing an IRB proposal
READINGS:
APA Publications and Communications Board Working Group on Journal Article Reporting Standards. (2008).
Reporting standards for research in psychology: Why do we need them? What might they be? American
Psychologist, 63, 839-851. [APA publication describing the rationale and requirements for reporting of
research]
Antonuccio, D. O., Danton, W. G., & McClanahan, T. M. (2003). Psychology in the prescription era: building a
firewall between marketing and science. American Psychologist, 58, 1028-1043.
Ioannidis, J. P. A. (2012). Why science is not necessarily self-correcting. Perspectives on Psychological Science,
7, 645-654.
Pachter, W. S., Fox, R. E., Zimbardo, P., & Antonuccio, D. O. (2007). Corporate funding and conflicts of interest:
a primer for psychologists. American Psychologist, 62, 1005-1015. [report and recommendations of an APA
Task Force on conflicts of interest in research]
Stroebe, W., Postmes, T., & Spears, R. (2012). Scientific misconduct and the myth of self-correction in science.
Perspectives on Psychological Science, 7, 670-688.
WEEK OF 5/4, TIME TO BE DETERMINED: FINAL PRESENTATION OF RESEARCH IDEAS (I'LL BUY LUNCH FOR THIS ONE)
**DUE BY 5/9: ASSIGNMENT II.D.: REVIEW OF POPULAR PRESS ARTICLE**
**DUE BY 5/9: ASSIGNMENT III: FINAL DRAFT OF ALL PARTS OF F31**
Research Problems – Spring 2015: Willcutt
8
SUPPLEMENTAL READINGS
Just to be sure this is clear to everyone: The supplemental readings are not required for the course. One of my
primary goals for this course is to provide you with a list of resources regarding different aspects of research design that may
be useful to you later in graduate school and in your future career as an independent researcher. Therefore, for each topic the
reading list includes a range of "classic" papers that provide useful overviews or different perspectives, along with papers that
discuss more specific topics that may only be relevant for some research questions or study designs. On a related note, I am
always looking for useful new articles to add to the list, so if you find any articles that are especially helpful during the course
or later in your training, please forward them to me.
CAREER DEVELOPMENT AND GRANT WRITING
JANUARY 20, 2015
General training and career development:
Aiken, L. S., West, S. G., & Millsap, R. E. (2008). Doctoral training in statistics, measurement, and methodology in
psychology: replication and extension of Aiken, West, Sechrest, and Reno's (1990) survey of PhD programs in
North America. American Psychologist, 63, 32-50.
Aiken, L. S., West, S. G., & Millsap, R. E. (2009). Improving training in methodology enriches the science of
psychology. American Psychologist, 64, discussion 51-52.
Bray, J. H. (2010). The future of psychology practice and science. American Psychologist, 65, 355-369.
Eissenberg, T. (2003). Teaching successful grant writing to psychology graduate students. Teaching of Psychology,
30, 328 - 330.
Hitt, E. (2008). Seeking the skills for a successful career in academia. Science Careers, 499 - 502.
Illes, J. (1999). The strategic grant-seeker: A guide to conceptualizing fundable research in the brain and behavioral
sciences (chapter 6 and chapter 9). Lawrence Erlbaum: London.
Roberts, M. C. (2006). Essential tension: specialization with broad and general training in psychology. American
Psychologist, 61, 862-870.
Zerhouni, E. (2003). The NIH Roadmap. Science, 302, 63 - 72.
Zimiles, H. (2009). Ramifications of increased training in quantitative methodology. American Psychologist, 64, 51.
Grant writing:
Eissenberg, T. (2003). Teaching successful grant writing to psychology graduate students. Teaching of Psychology,
30, 328 - 330.
Illes, J. (1999). The strategic grant-seeker: A guide to conceptualizing fundable research in the brain and behavioral
sciences (chapter 6 and chapter 9). Lawrence Erlbaum: London.
Marsh, H. W., Jayasinghe, U. W., & Bond, N. W. (2008). Improving the peer-review process for grant applications:
reliability, validity, bias, and generalizability. American Psychologist, 63, 160-168.
Zerhouni, E. (2003). The NIH Roadmap. Science, 302, 63 - 72.
--------------------------------------------------------------------------------------------------------------------------------------------------------- -------PHILSOPHY OF SCIENCE AND CAUSAL INFERENCE
JANUARY 27, 2015
Philosophy of science
Dar, R. (1987). Another look at Meehl, Lakatos, and the scientific practices of psychologists. American Psychologist,
42, 145-151.
Gough, B., & Madill, A. (2012). Subjectivity in psychological science: from problem to prospect. Psychological
Methods, 17, 374-384.
Haig, B. D. (2008). How to enrich scientific method. American Psychologist, 63, 565-566.
Lau, H. C. (2007). Should scientists think? Comment on Machado and Silva (2007). American Psychologist, 62, 686688.
Machado, A., & Silva, F. (2007). Toward a richer view of the scientific method: The role of conceptual analysis.
American Psychologist, 62, 671 - 681.
Meehl, P. E. (1993). Philosophy of science: Help or hindrance? Psychological Reports, 72, 707-733.
Sternberg, R. J. & Grigorenko E. L. (2001). Unified Psychology. American Psychologist, 56, 1069-1079.
Causal inference: empirical and theoretical papers
Erceg-Hurn, D. M., & Mirosevich, V. M. (2008). Modern robust statistical methods: an easy way to maximize the
accuracy and power of your research. American Psychologist, 63, 591-601.
Research Problems – Spring 2015: Willcutt
9
Jo, B. (2008). Causal inference in randomized experiments with mediational processes. Psychological Methods, 13,
314-336.
Platt, J. R. (1964). Strong Inference. Science, 146, 347-353.
Rubin, D. B. (2010). Reflections stimulated by the comments of Shadish (2010) and West and Thoemmes (2010).
Psychological Methods, 15, 38-46.
Scarr, S. (1997). Rules of evidence: A larger context for the statistical debate. Psychological Science, 8, 16-17.
Shadish, W. R. (2010). Campbell and Rubin: A primer and comparison of their approaches to causal inference in field
settings. Psychological Methods, 15, 3-17.
Wilson, T.D. (2005). The message is the method: Celebrating and exporting the experimental approach.
Psychological Inquiry, 16, 185-193.
Causal inference: popular press
Gigerenzer, G. (2002). Calculated Risks. How To Know When Numbers Deceive You. New York: Simon & Shuster.
Paolos, J. A. (2001). Innumeracy. New York: Hill and Yang.
Paulos, J. A. (1995). A Mathematician Reads the Newspaper. New York: Basic Books.
Sagan, C. (1996). The demon-haunted world: Science as a candle in the dark. New York: Random House.
Salsburg, D. (2001). The lady tasting tea: How statistics revolutionized science in the twentieth century. New York: W.
H. Freeman.
Schick, T., & Vaughn, L. (1999). How to think about weird things. Mountain View, CA: Mayfield Publishing Company.
Silver, N. (2012). The Signal and the Noise. Penguin Press: New York.
Stanovich, K. E. (2001). How to think straight about psychology (6th Edition). Boston, MA: Allyn and Bacon.
Tal, J. (2001). Reading Between the Numbers. Statistical Thinking in Everyday Life. New York: MacGraw Hill.
Taleb, N. N. (2004). Fooled by randomness: the hidden role of chance in life and in the markets. New York: Random
House.
Taleb, N. N. (2010): The black swan: the impact of the highly improbable (2nd Edition). New York: Penguin.
---------------------------------------------------------------------------------------------------------------------------------------------------------------THEORETICAL MODELS AND PUBLIC SKEPTICISM
FEBRUARY 3, 2015
The importance of theory
Cacioppo, J. T., Semin, G. R., & Berntson, G. G. (2004). Realism, instrumentalism, and scientific symbiosis:
Psychological theory as a search for truth and the discovery of solutions. American Psychologist, 214-223.
Meehl, P. E. (1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34,
103-115.
Meehl, P. E. (1996). Appraising and amending theories: The strategy of Lakatosian defense and two principles that
warrant it. Psychological Inquiry, 1, 108-141.
Meehl, P. E. (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological
Reports, 66, 195-244.
Meehl, P. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft
psychology. Journal of Consulting and Clinical Psychology, 46, 806-834. N/A
Roberts, S. & Pashler, H. (2000). How persuasive is a good fit? A comment on theory testing. Psychological Review,
107, 358-367.
Trafimow, D. (2003). Hypothesis testing and theory evaluation at the boundaries: Surprising insights from Bayes's
theorem. Psychological Review, 110, 526-535.
----------------------------------------------------------------------------------------------------------------------------- -----------------------------------CONSTRUCT VALIDITY, RELIABILITY, AND SCALE DEVELOPMENT
FEBRUARY 17, 2015
General scale development:
Achenbach, T. M., Dumenci, L., & Rescorla, L. A. (2003). DSM-oriented and empirically based approaches to
constructing scales from the same item pools. Journal of Clinical Child and Adolescent Psychology, 32, 328-340.
Bauer, D. J., & Hussong, A. M. (2009). Psychometric approaches for developing commensurate measures across
independent studies: traditional and new models. Psychological Methods, 14, 101-125.
Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437455.
Willcutt, E. G., Boada, R., Riddle, M. W., Chhabildas, N. A., & Pennington, B. F. (2011). A parent-report screening
questionnaire for learning difficulties in children. Psychological Assessment, 778 - 791.
Research Problems – Spring 2015: Willcutt
10
Reliability:
Bartels, M., Boomsma, D. I., Hudziak, J. J., van Beijsterveldt, T. C., & van den Oord, E. J. (2007). Twins and the study
of rater (dis)agreement. Psychological Methods, 12, 451-466.
Bonett, D. G. (2010). Varying coefficient meta-analytic methods for alpha reliability. Psychological Methods, 15, 368385.
Botella, J., Suero, M., & Gambara, H. (2010). Psychometric inferences from a meta-analysis of reliability and internal
consistency coefficients. Psychological Methods, 15, 386-397.
Feldt, L. S., & Charter, R. A. (2003). Estimating the reliability of a test split into two parts of equal or unequal length.
Psychological Methods, 8, 102-109.
Green, S. B. (2003). A coefficient alpha for test-retest data. Psychological Methods, 8, 88-101.
Hoyt, W. T. (2000). Rater bias in psychological research: when is it a problem and what can we do about it?
Psychological Methods, 5, 64-86.
Kraemer, H. C., Measelle, J. R., Ablow, J. C., Essex, M. J., Boyce, W. T., & Kupfer, D. J. (2003). A new approach to
integrating data from multiple informants in psychiatric assessment and research: mixing and matching contexts
and perspectives. American Journal of Psychiatry, 160, 1566-1577.
Osburn, H. G. (2000). Coefficient alpha and related internal consistency reliability coefficients. Psychological Methods,
5, 343-355.
Overall, J. E., & Woodward, J. A. (1975). Unreliability of difference scores: A paradox for measurement of change.
Psychological Bulletin, 82, 85-86.
Rodriguez, M. C., & Maeda, Y. (2006). Meta-analysis of coefficient alpha. Psychological Methods, 11, 306-322.
Schmidt, F. L., Le, H., & Ilies, R. (2003). Beyond alpha: An empirical examination of the effects of different sources of
measurement error on reliability estimates for measures of individual-differences constructs. Psychological
Methods, 8, 206-224.
Schmitt, N. (1996). Uses and abuses of coefficient alpha. Psychological assessment, 4, 350-353.
Schuster, C., & Smith, D. A. (2002). Indexing systematic rater agreement with a latent-class model. Psychological
Methods, 7, 384-395.
Vanbelle, S., Mutsvari, T., Declerck, D., & Lesaffre, E. (2012). Hierarchical modeling of agreement. Statistics in
Medicine, 31, 3667-3680.
Weijters, B., Geuens, M., & Schillewaert, N. (2010). The stability of individual response styles. Psychological Methods,
15, 96-110.
Item-response theory:
Brown, A., & Maydeu-Olivares, A. (2012). How IRT can solve problems of ipsative data in forced-choice
questionnaires. Psychological Methods.
Meijer, R. R. (2003). Diagnosing item score patterns on a test using item response theory-based person-fit statistics.
Psychological Methods, 8, 72-87.
Reise, S. P., & Waller, N. G. (2003). How many IRT parameters does it take to model psychopathology items?
Psychological Methods, 8, 164-184.
Reise, S. P., & Waller, N. G. (2009). Item response theory and clinical measurement. Annual Review of Clinical
Psychology, 5, 27-48.
Waller, N. G., Thompson, J. S., & Wenk, E. (2000). Using IRT to separate measurement bias from true group
differences on homogeneous and heterogeneous scales: an illustration with the MMPI. Psychological Methods, 5,
125-146.
-------------------------------------------------------------------------------------------------------------------- --------------------------------------------EXTERNAL VALIDITY AND GENERALIZATION
FEBRUARY 24, 2015
Albright, L., & Malloy, T. E. (2000). Experimental validity: Brunswik, Campbell, Cronbach, and enduring issues.
Review of General Psychology, 4, 337-353.
Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix.
Psychological Bulletin, 56, 81-105.
Hunsley, J., & Meyer, G. J. (2003). The incremental validity of psychological testing and assessment: Conceptual,
methodological, and statistical issues. Psychological Assessment, 15, 445-455.
Kraemer, H. C., Measelle, J. R., Ablow, J. C., Essex, M. J., Boyce, W. T., & Kupfer, D. J. (2003). A new approach to
integrating data from multiple informants in psychiatric assessment and research: mixing and matching contexts
and perspectives. American Journal of Psychiatry, 160, 1566-1577.
Meyer, G. J., Finn, S. E., Eyde, L. D., Kay, G. G., Moreland, K. L., Dies, R. R., Eisman, E. J., Kubiszyn, T. W., &
Reed, G. M. (2001). Psychological testing and psychological assessment: A review of evidence and issues.
American Psychologist, 56, 128-165.
Research Problems – Spring 2015: Willcutt
11
Smith, G. T. (2005). On construct validity: Issues of method and measurement. Psychological Assessment, 17, 395408.
Westen, D., & Rosenthal, R. (2005). Improving construct validity: Cronbach, Meehl, and Neurath's ship. Psychological
Assessment, 17, 409 - 412.
Willcutt, E. G., Boada, R., Riddle, M. W., Chhabildas, N. A., & Pennington, B. F. (2011). A parent-report screening
questionnaire for learning difficulties in children. Psychological Assessment, 778 - 791. [Don't worry about the
substantive detasils of our results - this paper just illustrates a number of the methods we will cover in class.]
Willcutt, E. G., Nigg, J. T., Pennington, B. F., Carlson, C. L., McBurnett, K., Rohde, L. A., Solanto, M. V., Tannock, R.,
& Lahey, B. B. (2012). Validity of DSM-IV attention-deficit/hyperactivity disorder symptom dimensions and
subtypes. Journal of Abnormal Psychology, 121, 991 - 1010. [same as previous]
Issues around culture, race, gender, and other key aspects of diversity
Okazaki, S. & Sue, S. (1995). Methodological issues in assessment research with ethnic minorities. Psychological
Assessment, 7, 367 - 375.
Ossorio, P., & Duster, T. (2005). Race and genetics: controversies in biomedical, behavioral, and forensic sciences.
American Psychologist, 60, 115-128.
Rowe, D. C. (2005). Under the skin: On the impartial treatment of genetic and environmental hypotheses of racial
differences. American Psychologist, 60, 60-70.
Shields, A. E., Fortun, M., Hammonds, E. M., King, P. A., Lerman, C., Rapp, R., & Sullivan, P. F. (2005). The use of
race variables in genetic studies of complex traits and the goal of reducing health disparities: a transdisciplinary
perspective. American Psychologist, 60, 77-103.
Sue, S. (1999). Science, ethnicity, and bias: Where have we gone wrong? American Psychologist, 54, 1070-1077.
Zahn-Waxler, C., Crick, N. R., Shirtcliff, E. A., & Woods, K. A. (2006). The origins and development of psychopathology in
females and males. In D. Cicchetti & D. Cohen (Eds.), Psychopathology, Vol. 1: Theory and Method (2nd Edition).
Hoboken, NJ: Wiley.
----------------------------------------------------------------------------------------------------------------------------- -----------------------------------STUDY DESIGN I: RANDOM SAMPLING AND RANDOMIZED EXPERIMENTS
M ARCH 3, 2015
General research design issues:
Keppel, G. & Wickens, T. (2004). Design and Analysis: A Researcher’s Handbook, 4th Ed. Prentice Hall: Upper
Saddle River, NJ.
McClelland, G. H. (1997). Optimal design in psychological research. Psychological Methods, 2, 3-19.
Rodgers, J. L. (2010). The epistemology of mathematical and statistical modeling: a quiet methodological revolution.
American Psychologist, 65, 1-12.
Rutter, M. (2007). Proceeding from observed correlation to causal inference: the use of natural experiments.
Perspectives on Psychological Science, 2, 377 - 394.
Spencer, S. J., Zanna, M. P., & Fong, G. T. (2005). Establishing a causal chain: Why experiments are often more
effective in examining psychological process than mediational analyses. Journal of Personality and Social
Psychology, 89, 845-851.
Randomized Experiments:
Aiken, L. S., West, S. G., Schwalm, D. E., Carroll, J. L., & Hsuing, S. (1998). Comparison of a randomized and two
quasi-experimental designs in a single outcome evaluation. Evaluation Review, 22, 207-255.
West, S. G., & Sagarin, B. J. (2000). Participation selection and loss in randomized experiments. In L. Bickman
(Ed.), Research design: Donald Campbell’s Legacy (pp. 117-154). Thousand Oaks, CA: Sage.
West, S.G., Naihua, D., Pequegnat, W., Gaist, P., Des Jarlais, D.C., Holtgrave, D., Szapacznik, J., Fishbein, M.,
Rapkin, B., Clatts, M., & Mullen, P. (in press). Alternatives to the randomized controlled trial. American Journal of
Public Health.
----------------------------------------------------------------------------------------------------------------------------- -----------------------------------STUDY DESIGN II: QUASI-EXPERIMENTAL, CASE-CONTROL, AND OTHER DESIGNS
M ARCH 10, 2015
Kraemer, H. C. (2010). Epidemiological methods: about time. Int J Environ Res Public Health, 7, 29-45.
Preacher, K. J., Rucker, D. D., MacCallum, R. C., & Nicewander, W. A. (2005). Use of the extreme groups approach:
a critical reexamination and new recommendations. Psychological Methods, 10, 178-192.
Wampold, B. E., Davis, B., & Good, R. H. (1990). Hypothesis validity of clinical research. Journal of Consulting and
Clinical Psychology, 58, 360-367.
Research Problems – Spring 2015: Willcutt
12
West, S. G., & Thoemmes, F. (in press). Equating groups. In J. Brannon, P. Alasuutari, & L. Bickman (Eds.),
Handbook of social research methods. London: Sage.
Qualitative research designs
Kidd, S. A. (2002). The role of qualitative research in psychological journals. Psychological Methods, 7, 126-138.
Madill, A., & Gough, B. (2008). Qualitative research and its place in psychological science. Psychological Methods,
13, 254-271.
Rennie, D. L. (2012). Qualitative research as methodical hermeneutics. Psychological Methods, 17, 385-398.
----------------------------------------------------------------------------------------------------------------------------- -----------------------------------STUDY DESIGN III: TREATMENT STUDIES AND OTHER LONGITUDINAL DESIGNS
M ARCH 18, 2015
Design issues:
Atkins, D. C., Bedics, J. D., McGlinchey, J. B., & Beauchaine, T. P. (2005). Assessing clinical significance: does it
matter which method we use? Journal of Consulting and Clinical Psychology, 73, 982-989.
Clarke, G. N. (1995). Improving the transition from basic efficacy research to effectiveness studies: Methodological
issues and procedures. Journal of Consulting & Clinical Psychology, 63, 718-725.
Kazak, A. E., Hoagwood, K., Weisz, J. R., Hood, K., Kratochwill, T. R., Vargas, L. A., & Banez, G. A. (2010). A metasystems approach to evidence-based practice for children and adolescents. American Psychologist, 65, 85-97.
Kazdin, A. E. (2011). Evidence-based treatment research: Advances, limitations, and next steps. American
Psychologist, 66, 685-698.
Kelley, K., & Rausch, J. R. (2011). Sample size planning for longitudinal models: accuracy in parameter estimation for
polynomial change parameters. Psychological Methods, 16, 391-405.
Khoo, S.-T., West, S. G., Wu, W., & Kwok, O.-M. (2006). Longitudinal methods. In M. Eid & E. Diener (Eds.),
Handbook of psychological measurement: A multimethod perspective. Washington, DC: American Psychological
Association books.
Kraemer, H. C. & Kupfer, D. J. (2006). Size of treatment effects and their importance to clinical research and practice.
Biological Psychiatry, 59, 990-996.
Kraemer, H. C., & Frank, E. (2010). Evaluation of comparative treatment trials: assessing clinical benefits and risks for
patients, rather than statistical effects on measures. Journal of the American Medical Association, 304, 683-684.
Kraemer, H. C., Frank, E., & Kupfer, D. J. (2011). How to assess the clinical impact of treatments on patients, rather
than the statistical impact of treatments on measures. Int J Methods Psychiatr Res, 20, 63-72.
Kraemer, H. C., Morgan, G. A., Leech, N. L., Gliner, J. A., Vaske, J. J., & Harmon, R. J. (2003). Measures of clinical
significance. Journal of the American Academy of Child and Adolescent Psychiatry, 42, 1524-1529.
Lambert, M. J., Okiishi, J. C., Finch, A. E., & Johnson, L. D. (1998). Outcome assessment: From conceptualization to
implementation. Professional Psychology: Research & Practice, 29, 63-70.
Masten, A. S., & Cicchetti, D. (2010). Developmental cascades. Developmental Psychopathology, 22, 491-495.
Nathan, P. E., Stuart, S. P., & Dolan, S. L. (2000). Research on psychotherapy efficacy and effectiveness: Between
Scylla and Charybdis? Psychological Bulletin, 126, 964-981.
Persons, J. B., & Silberschatz, G. (1998). Are results of randomized controlled trials useful to psychotherapists?
Journal of Consulting & Clinical Psychology, 66, 126-135.
Reichardt, C. S. (2006). The principle of parallelism in the design of studies to estimate treatment effects.
Psychological Methods, 11, 1-18.
Rutter, M., Kim-Cohen, J., & Maughan, B. (2006). Continuities and discontinuities in psychopathology between
childhood and adult life. Journal of Child Psychology and Psychiatry, 47, 276-295.
Velicer, W. F., & Fava, J. L. (2003). Time series analysis. In J. A. Schinka & W. F. Velicer (Eds.), Handbook of
psychology (Vol. 2). Research methods in psychology (pp. 581-606). New York: Wiley.
West, S. G., Biesanz, J. C., & Kwok, O-M. (2003). Within-subject and longitudinal experiments: Design and analysis
issues. In C. Sansone, C. C. Morf, & A. T. Panter (Eds.), Handbook of Methods in Social Psychology. Thousand
Oaks, CA: Sage.
Statistical issues:
Blozis, S. A. (2004). Structured latent curve models for the study of change in multivariate repeated measures.
Psychological Methods, 9, 334-353.
Cole, D. A., & Maxwell, S. E. (2009). Statistical methods for risk-outcome research: being sensitive to longitudinal
structure. Annual Review of Clinical Psychology, 5, 71-96.
Crits-Christoph, P., Tu, X., & Gallop, R. (2003). Therapists as fixed versus random effects-some statistical and
conceptual issues: a comment on Siemer and Joormann (2003). Psychological Methods, 8, 518-523.
DerSimonian, R., & Laird, N. (1986). Metaanalysis in Clinical-Trials. Controlled Clinical Trials, 7, 177-188.
Research Problems – Spring 2015: Willcutt
13
Enders, C. K. (2011). Missing not at random models for latent growth curve analyses. Psychological Methods, 16, 116.
Gibbons, R. D., Hedeker, D., & DuToit, S. (2010). Advances in analysis of longitudinal data. Annual Review of Clinical
Psychology, 6, 79-107.
Hogan, J. W., Roy, J., & Korkontzelou, C. (2004). Handling drop-out in longitudinal studies. Statistics in Medicine, 23,
1455-1497.
Holmbeck, G. N. (1997). Toward terminological, conceptual, and statistical clarity in the study of mediators and
moderators: Examples from the child-clinical and pediatric psychology literatures. Journal of Consulting & Clinical
Psychology, 65, 599-610.
Khoo, S.-T., West, S. G., Wu, W., & Kwok, O.-M. (2006). Longitudinal methods. In M. Eid & E. Diener (Eds.),
Handbook of psychological measurement: A multimethod perspective. Washington, DC: American Psychological
Association books.
Kraemer, H. C., Frank, E., & Kupfer, D. J. (2006). Moderators of treatment outcomes: clinical, research, and policy
importance. Journal of the American Medical Association, 296, 1286-1289.
Kraemer, H. C., Wilson, G. T., Fairburn, C. G., & Agras, W. S. (2002). Mediators and moderators of treatment effects
in randomized clinical trials. Archives of General Psychiatry, 59, 877-883.
Kuljanin, G., Braun, M. T., & Deshon, R. P. (2011). A cautionary note on modeling growth trends in longitudinal data.
Psychological Methods, 16, 249-264.
Lix, L. M., & Sajobi, T. (2010). Testing multiple outcomes in repeated measures designs. Psychological Methods, 15,
268-280.
McArdle, J. J., Grimm, K. J., Hamagami, F., Bowles, R. P., & Meredith, W. (2009). Modeling life-span growth curves of
cognition using longitudinal data with multiple samples and changing scales of measurement. Psychological
Methods, 14, 126-149.
Muthen, B., Asparouhov, T., Hunter, A. M., & Leuchter, A. F. (2011). Growth modeling with nonignorable dropout:
alternative analyses of the STAR*D antidepressant trial. Psychological Methods, 16, 17-33.
Muthen, B., Asparouhov, T., Hunter, A. M., & Leuchter, A. F. (2011). Growth modeling with nonignorable dropout:
alternative analyses of the STAR*D antidepressant trial. Psychological Methods, 16, 17-33.
Nagin, D. S., & Odgers, C. L. (2010). Group-based trajectory modeling in clinical research. Annual Review of Clinical
Psychology, 6, 109-138.
Nagin, D. S., & Odgers, C. L. (2010). Group-based trajectory modeling in clinical research. Annual Review of Clinical
Psychology, 6, 109-138.
Nagin, D. S., & Tremblay, R. E. (2001). Analyzing developmental trajectories of distinct but related behaviors: a
group-based method. Psychological Methods, 6, 18-34.
Overall, J. E., & Woodward, J. A. (1975). Unreliability of difference scores: A paradox for measurement of change.
Psychological Bulletin, 82, 85-86.
Raudenbush, S. W., & Liu, X. (2000). Statistical power and optimal design for multisite randomized trials.
Psychological Methods, 5, 199-213.
Velicer, W. F., & Fava, J. L. (2003). Time series analysis. In J. A. Schinka & W. F. Velicer (Eds.), Handbook of
psychology (Vol. 2). Research methods in psychology (pp. 581-606). New York: Wiley. Chp
Venter, A., Maxwell, S. E., & Bolig, E. (2002). Power in randomized group comparisons: The value of adding a single
intermediate time point to a traditional pretest-posttest design. Psychological Methods, 7, 194-209.
Venter, A., Maxwell, S. E., & Bolig, E. (2002). Power in randomized group comparisons: The value of adding a single
intermediate time point to a traditional pretest-posttest design. Psychological Methods, 7, 194-209.
Wilson, D. B., & Lipsey, M. W. (2001). The role of method in treatment effectiveness research: Evidence from metaanalysis. Psychological Methods, 6, 413-429.
Wu, W., West, S. G., & Taylor, A. B. (2009). Evaluating model fit for growth curve models: Integration of fit indices
from SEM and MLM frameworks. Psychological Methods, 14, 183-201.
Research Problems – Spring 2015: Willcutt
14
------------------------------------------------------------------------------------------------------------------------------------------------ ----------------STATISTICAL INFERENCE AND INTERPRETATION
M ARCH 31 - APRIL 21, 2015
General Statistical Inference:
Abelson, R. P. (1985). A variance explanation paradox: When a little is a lot. Psychological Bulletin, 97, 129-133.
Abelson, R.P. (1995). Statistics as Principled Argument. NJ: Lawrence Erlbaum Associates.
Berger, J. O. & Berry, D. A. (1988). Statistical analysis and illusion of objectivity. American Scientist, 76, 159-165.
Best, J. (2001). Damned Lies and Statistics. Untangling Numbers from the Media, Politicians, and Activists. CA:
University of California Press.
Best, J. (2004). More Damned Lies and Statistics. How Numbers Confuse Public Issues. CA: University of California
Press.
Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997-1003.
Erceg-Hurn, D. M., & Mirosevich, V. M. (2008). Modern robust statistical methods: an easy way to maximize the
accuracy and power of your research. American Psychologist, 63, 591-601.
Paulos, J.A. (1995). A Mathematician Reads the Newspaper. NY: Basic Books.
Platt, J. R. (1964). Strong Inference. Science, 146, 347-353.
Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological
Methods.
Significance Tests and Confidence Intervals:
Abelson, R. P. (1997). On the surprising longevity of flogged horses: Why there is a case for the significance test.
Psychological Science, 6, 12-15.
Belia, S., Fidler, F., Williams, J., & Cumming, G. (2005). Researchers misunderstand confidence intervals and
standard error bars. Psychological Methods, 10, 389-396.
Chow, S. L. (1988). Significance test or effect size? Psychological Bulletin, 103, 105-110.
Cumming, G., & Finch, S. (2001). A primer on understanding, use, and calculation of confidence intervals that are
based on central and noncentral distributions. Educational and Psychological Measurement, 61, 532-574.
Cumming, G., & Finch, S. (2005). Inference by eye: confidence intervals and how to read pictures of data. The
American psychologist, 60, 170-180.
Cumming, G., & Maillardet, R. (2006). Confidence intervals and replication: where will the next mean fall?
Psychological Methods, 11, 217-227.
Frick, R. W. (1996). The appropriate use of null hypothesis testing. Psychological Methods, 1, 379-390.
Gardner, M. J., & Altman, D. G. (1986). Confidence intervals rather than P values: Estimation rather than hypothesis
testing. British Medical Journal, 292, 746-750.
Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1-20.
Hagen, R. L. (1997). In praise of the null hypothesis statistical test. American Psychologist, 52, 15-24.
Harris, R. J. (1997). Significance tests have their place. Psychological Science, 8, 8-11.
Howard, G. S., Maxwell, S. E., & Fleming, K. J. (2000). The proof of the pudding: An illustration of the relative
strengths of null hypothesis, meta-analysis, and Bayesian analysis. Psychological Methods, 5, 315-332.
Hsu, L. M. (2000). Effects of directionality of significance tests on the bias of accessible effect sizes. Psychological
Methods, 5, 333-342.
Jones, L. V., & Tukey, J. W. (2000). A sensible formulation of the significance test. Psychological Methods, 5, 411414.
Keselman, H. J., Algina, J., Lix, L. M., Wilcox, R. R., & Deering, K. N. (2008). A generally robust approach for testing
hypotheses and setting confidence intervals for effect sizes. Psychological Methods, 13, 110-129.
Keselman, H. J., Miller, C. W., & Holland, B. (2011). Many tests of significance: new methods for controlling type I
errors. Psychological Methods, 16, 420-431.
Mulaik, S. A., Raju, N. S., & Harshman, R. A. (1997). There is a time and place for significance testing. In Lisa A.
Harlow, Stanley A. Mulaik, and James H. Steiger , Eds. What if there were no significance tests? (pp. 65-116).
Mahwah, NJ: Lawrence Erlbaum Associates.
Nickerson, R. S. (2000). Null hypothesis significance testing: A review of an old and continuing controversy.
Psychological Methods, 5, 241-301.
Rindskopf, D. M. (1997). Testing "small," not null, hypothesis: Classical and Bayesian Approaches. In Lisa A. Harlow,
Stanley A. Mulaik, and James H. Steiger (Eds). What if there were no significance tests? (pp. 319-332). Mahwah,
NJ: Lawrence Erlbaum Associates.
Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological
Methods.
Research Problems – Spring 2015: Willcutt
15
Schmidt, F. L. & Hunter, J. E. (1997). Eight common but false objections to the discontinuation of significance testing
in the analysis of research data. In Lisa A. Harlow, Stanley A. Mulaik, and James H. Steiger (Eds.) What if there
were no significance tests? (pp. 37-64). Mahwah, NJ: Lawrence Erlbaum Associates.
Tryon, W.W. (2001). Evaluating statistical difference, equivalence, and indeterminancy using inferential confidence
intervals: An integrated alternative method of conducting null hypothesis statistical tests. Psychological Methods,
6, 371-386.
Wainer, H. (1999). One cheer for null hypothesis significance testing. Psychological Methods, 6, 212-213.
Effect sizes:
Abelson, R. P. (1985). A variance explanation paradox: When a little is a lot. Psychological Bulletin, 97, 129-133.
Borenstein, M. (2009). Effect sizes for continuous data. In H. Cooper, L. V. Hedges & J. C. Valentine (Eds.), The
handbook of research synthesis and meta-analysis (pp. 279-293). New York: Russel Sage Foundation.
Chow, S. L. (1988). Significance test or effect size? Psychological Bulletin, 103, 105-110.
Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155-159.
Durlak, J. A. (2009). How to select, calculate, and interpret effect sizes. Journal of Pediatric Psychology, 34, 917-928.
Greenwald, A. G., Gonzalez, R., Harris, R. J., & Guthrie, D. (1996). Effect sizes and p values: What should be
reported and what should be replicated? Psychophysiology, 33, 175-183.
Grissom, R. J., & Kim, J. J. (2001). Review of assumptions and problems in the appropriate conceptualization of effect
size. Psychological Methods, 6, 135-146.
Hedges, L. (1981). Distributional theory for Glass' estimator of effect size and related estimators. Journal of
Educational Statistics, 6, 107-128.
Olejnik, S., & Algina, J. (2003). Generalized eta and omega squared statistics: measures of effect size for some
common research designs. Psychological Methods, 8, 434-447.
Rosenthal, R. (1994). Parametric measures of effect size. In H. Cooper & L. V. Hedges (Eds.), The Handbook of
Research Synthesis. New York: Russell Sage Foundation.
Approaches to deal with missing data:
Allison, P. D. (2003). Missing data techniques for structural equation modeling. Journal of Abnormal Psychology, 112,
545-557.
Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147177.
Dichotomization:
Coghill, D., & Sonuga-Barke, E. J. (2012). Annual research review: categories versus dimensions in the classification
and conceptualisation of child and adolescent mental disorders--implications of recent empirical study. Journal of
Child Psychology and Psychiatry, 53, 469-489.
Helzer, J. E., Kraemer, H. C., & Krueger, R. F. (2006). The feasibility and need for dimensional psychiatric diagnoses.
Psychological Medicine, 36, 1671-1680.
Kraemer, H. C., Noda, A., & O'Hara, R. (2004). Categorical versus dimensional approaches to diagnosis:
methodological challenges. Journal of Psychiatry Research, 38, 17-25.
MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D. (2002). On the practice of dichotomization of
quantitative variables. Psychological Methods, 7, 19-40. [Nice summary - focus on the big picture points.]
McGrath, R. E., & Walters, G. D. (2012). Taxometric analysis as a general strategy for distinguishing categorical from
dimensional latent structure. Psychological Methods, 17, 284-293.
Rhemtulla, M., Brosseau-Liard, P. E., & Savalei, V. (2012). When can categorical variables be treated as continuous?
A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions.
Psychological Methods, 17, 354-373.
Sanchez-Meca, J., Marin-Martinez, F., & Chacon-Moscoso, S. (2003). Effect-size indices for dichotomized outcomes
in meta-analysis. Psychological Methods, 8, 448-467.
Widiger, T. A., & Trull, T. J. (2007). Plate tectonics in the classification of personality disorder: Shifting to a
dimensional model. American Psychologist, 62, 71-83.
Statistical power:
Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives
on Psychological Science, 7, 543-554.
Cohen, J. (1988). Statistical power analyses for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum Associates.
Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and
Social Psychology, 65, 145-153.
Maxwell, S. E. (2004). The persistence of underpowered studies in psychological research: causes, consequences,
and remedies. Psychological Methods, 9, 147-163.
Research Problems – Spring 2015: Willcutt
16
Rossi, J. S. (1990). Statistical power of psychological research: What have we gained in 20 years? Journal of
Consulting and Clinical Psychology, 58, 646-656.
Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies?
Psychological Bulletin, 105, 309-316.
Mediators and moderators:
Baron, R. M., & Kenny, D. A. (1986). The moderator-mediator distinction in social psychological research: Conceptual,
strategic, and statistical considerations. Journal of Personality and Social Psychology, 51, 1173-1182.
Holmbeck, G. N. (1997). Toward terminological, conceptual, and statistical clarity in the study of mediators and
moderators: Examples from the child-clinical and pediatric psychology literatures. Journal of Consulting & Clinical
Psychology, 65, 599-610.
Judd, C. M., Kenny, D. A., & McClelland, G. H. (2001). Estimating and testing mediation and moderation in withinsubject designs. Psychological Methods, 6, 115 - 134.
Kraemer, H. C. & Blasey, C. M. (2004). Centering in regression analyses: a strategy to prevent errors in statistical
inference. International Journal of Methods for Psychiatry Research, 13, 141-151.
Muller, D., Judd, C. M., & Yzerbyt, V. Y. (2005). When moderation is mediated and mediation is moderated. Journal of
Personality and Social Psychology, 89, 852-863.
Factor analyses and other latent variable approaches:
Fabrigar L. R. , Wegener D. T., MacCallum R. C., Strahan E. J. (1999). Evaluating the use of exploratory factor
analysis in psychological research, Psychological Methods, 4, 272-299.
Grice, J. W. (2001). A comparison of factor scores under conditions of factor obliquity. Psychological Methods, 6,
67-83.
Grice, J. W. (2001). Computing and evaluating factor scores. Psychological Methods, 6, 430-450.
Heene, M., Hilbert, S., Draxler, C., Ziegler, M., & Buhner, M. (2011). Masking misfit in confirmatory factor analysis
by increasing unique variances: a cautionary note on the usefulness of cutoff values of fit indices. Psychological
Methods, 16, 319-336.
Jackson, D. L., Gillaspy, J. A., & Purc-Stephenson, R. (2009). Reporting practices in confirmatory factor analysis:
an overview and some recommendations. Psychological Methods, 14, 6-23.
Linting, M., Meulman, J. J., Groenen, P. J., & van der Koojj, A. J. (2007). Nonlinear principal components
analysis: introduction and application. Psychological Methods, 12, 336-358.
Loehlin, J. C. (1992). Latent variable models: An introduction to factor, path, and structural analysis. Hillsdale, NJ:
Lawrence Erlbaum Associates.
Muthen, L. K., & Muthen, B. O. (2009). Mplus User's Guide: Fourth Edition. Los Angeles, CA: Muthen and
Muthen.
Reise, S. P., Waller, N. G., & Comrey, A. L. (2000). Factor analysis and scale revision. Psychological
Assessment, 12, 287-297.
Replication
Bones, A. K. (2012). We knew the future all along: scientific hypothesizing is much more accurate than other forms of
precognition: A satire in one part. Perspectives on Psychological Science, 7, 307-309.
Francis, G. (2012). The psychology of replication and replication in psychology. Perspectives on Psychological
Science, 7, 585-594.
Frank, M. C., & Saxe, R. (2012). Teaching replication. Perspectives on Psychological Science, 7, 600-604.
Klein, O., Doyen, S., Leys, C., Magalhaes de Saldanha da Gama, P. A., Miller, S., Questienne, L., & Cleeremans, A.
(2012). Low hopes, high expectations: Expectancy effects and the replicability of behavioral experiments.
Perspectives on Psychological Science, 7, 572-584.
Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific Utopia: II. Restructuring incentives and practices to promote
truth over publishability. Perspectives on Psychological Science, 7, 615-631.
Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on
Psychological Science, 7, 531-536.
Wagenmakers, E. J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., & Kievit, R. A. (2012). An agenda for purely
confirmatory research. Perspectives on Psychological Science, 7, 632-638.
----------------------------------------------------------------------------------------------------------------------------- ----------------------------------Overview of literature review methodologies:
Anzures-Cabrera, J., & Higgins, J. P. T. (2010). Graphical displays for meta-analysis: An overview with suggestions
for practice. Research Synthesis Methods, 1, 66-80.
Cooper, H., & Patall, E. A. (2009). The relative benefits of meta-analysis conducted with individual participant data
versus aggregated data. Psychological Methods, 14, 165-176.
Research Problems – Spring 2015: Willcutt
17
Curran, P. J. (2009). The seemingly quixotic pursuit of a cumulative psychological science: introduction to the special
issue. Psychological Methods, 14, 77-80.
Curran, P. J., & Hussong, A. M. (2009). Integrative data analysis: the simultaneous analysis of multiple data sets.
Psychological Methods, 14, 81-100.
Fehrmann, P., & Thomas, J. (2011). Comprehensive computer searches and reporting in systematic reviews.
Research Synthesis Methods, 2, 15-32.
Hopewell, S., Clarke, M., Lusher, A., Lefebvre, C., & Westby, M. (2002). A comparison of handsearching versus
MEDLINE searching to identify reports of randomized controlled trials. Statistics in Medicine, 21, 1625-1634.
Ioannidis, J. P. A. (2010). Meta-research: The art of getting it wrong. Research Synthesis Methods, 1, 169-184.
Jackson, D., Riley, R., & White, I. R. (2011). Multivariate meta-analysis: Potential and promise. Statistics in Medicine.
Meehl, P. E. (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological
Reports, 66, 195-244.
Shadish, Cook, & Campbell, Chapter 13: Generalized causal inference: Methods for multiple studies (pp. 417 - 455).
[A systematic summary of issues relevant for meta-analyses. You can skim pages 429 - 445].
Shrout, P. E. (2009). Short and long views of integrative data analysis: comments on contributions to the special
issue. Psychological Methods, 14, 177-181.
Thomson, D., Russell, K., Becker, L., Klassen, T., & Hartling, L. (2010). The evolution of a new publication type: Steps
and challenges of producing overviews of reviews. Research Synthesis Methods, 1, 198-211.
Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006). The poor availability of psychological research data
for reanalysis. American Psychologist, 61, 726-728.
Meta-analysis: overviews and critiques:
Cohn, L. D., & Becker, B. J. (2003). How meta-analysis increases statistical power. Psychological Methods, 8, 243253.
Cooper, H., & Koenka, A. C. (2012). The overview of reviews: unique challenges and opportunities when research
syntheses are the principal elements of new integrative scholarship. American Psychologist, 67, 446-462.
Ferguson, C. J., & Brannick, M. T. (2012). Publication bias in psychological science: prevalence, methods for
identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17, 120-128.
Hofer, S. M., & Piccinin, A. M. (2009). Integrative data analysis through coordination of measurement and analysis
protocol across independent longitudinal studies. Psychological Methods, 14, 150-164.
Ioannidis, J. P. A. (2010). Meta-research: The art of getting it wrong. Research Synthesis Methods, 1, 169-184.
Meehl, P. E. (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological
Reports, 66, 195-244.
Normand, S. T. (1999). Tutorial in biostatistics: Meta-analysis: formulating, evaluating, combining, and reporting.
Statistics in Medicine, 18, 321-359.
Rosenthal, R. (1984). Meta-analytic procedures for social research. Beverly Hills, CA: Sage. Bk
Schimmack, U. (2012). The ironic effect of significant results on the credibility of multiple-study articles. Psychological
Methods.
Sutton, A. J., & Higgins, J. P. (2008). Recent developments in meta-analysis. Statistics in Medicine, 27, 625-650.
Meta-analysis: Estimating effect sizes:
Borenstein, M. (2009). Effect sizes for continuous data. In H. Cooper, L. V. Hedges & J. C. Valentine (Eds.), The
handbook of research synthesis and meta-analysis (pp. 279-293). New York: Russel Sage Foundation.
Deeks, J. J. (2002). Issues in the selection of a summary statistic for meta-analysis of clinical trials with binary
outcomes. Statistics in Medicine, 21, 1575-1600.
Gillett, R. (2003). The metric comparability of meta-analytic effect-size estimators from factorial designs.
Psychological Methods, 8, 419-433.
Hedges, L. (1981). Distributional theory for Glass' estimator of effect size and related estimators. Journal of
Educational Statistics, 6, 107-128.
Kraemer, H. C. (1983). Theory of estimation and testing of effect sizes: Use in meta-analysis. Journal of Educational
and Behavioral Statistics, 8, 93-101.
Kuhnert, R., & Bohning, D. (2007). A comparison of three different models for estimating relative risk in meta-analysis
of clinical trials under unobserved heterogeneity. Statistics in Medicine, 26, 2277-2296.
Morris, S. B., & DeShon, R. P. (2002). Combining effect size estimates in meta-analysis with repeated measures and
independent-groups designs. Psychological Methods, 7, 105-125.
Rosenthal, R. (1994). Parametric measures of effect size. In H. Cooper & L. V. Hedges (Eds.), The Handbook of
Research Synthesis. New York: Russell Sage Foundation. Chp
Rosenthal, R., & Rubin, D. B. (2003). r equivalent: A simple effect size indicator. Psychological Methods, 8, 492-496.
Research Problems – Spring 2015: Willcutt
18
Sanchez-Meca, J., Marin-Martinez, F., & Chacon-Moscoso, S. (2003). Effect-size indices for dichotomized outcomes
in meta-analysis. Psychological Methods, 8, 448-467.
Meta-analysis: Publication and other biases in the literature search
Cooper, H., DeNeve, K., & Charlton, K. (1997). Finding the missing science: The fate of studies submitted for review
by a human subjects committee. Psychological Methods, 2, 447-452.
Edwards, P., Clarke, M., DiGuiseppi, C., Pratap, S., Roberts, I., & Wentz, R. (2002). Identification of randomized
controlled trials in systematic reviews: accuracy and reliability of screening records. Statistics in Medicine, 21,
1635-1640.
Fehrmann, P., & Thomas, J. (2011). Comprehensive computer searches and reporting in systematic reviews.
Research Synthesis Methods, 2, 15-32.
Ferguson, C. J., & Brannick, M. T. (2012). Publication bias in psychological science: prevalence, methods for
identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17, 120-128.
Hopewell, S., Clarke, M., Lusher, A., Lefebvre, C., & Westby, M. (2002). A comparison of handsearching versus
MEDLINE searching to identify reports of randomized controlled trials. Statistics in Medicine, 21, 1625-1634.
Orwin, R. G. (1983). A fail-safe N for effect size in meta-analysis. Journal of Educational Statistics, 8, 157-159.
Rosenthal, R. (1979). The "file drawer problem" and tolerance for null results. Psychological Bulletin, 85, 638-641.
Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2005). Publication bias in meta-analysis: Prevention, assessment,
and adjustments. Chichester, England: Wiley and Sons. Bk
Shrier, I. (2011). Structural approach to bias in Meta-analyses. Research Synthesis Methods, 2, 223-237.
Thomas, J., McNaught, J., & Ananiadou, S. (2011). Applications of text mining within systematic reviews. Research
Synthesis Methods, 2, 1-14.
Vevea, J. L., & Woods, C. M. (2005). Publication bias in research synthesis: sensitivity analysis using a priori weight
functions. Psychological Methods, 10, 428-443.
Viechtbauer, W., & Cheung, M. W. L. (2010). Outlier and influence diagnostics for meta-analysis. Research Synthesis
Methods, 1, 112-125.
Williamson, P. R., Gamble, C., Altman, D. G., & Hutton, J. L. (2005). Outcome selection bias in meta-analysis.
Statistical Methods in Medical Research, 14, 515-524.
Meta-analysis: detecting and accounting for heterogeneity among studies
Hedges, L. V., & Pigott, T. D. (2004). The power of statistical tests for moderators in meta-analysis. Psychological
Methods, 9, 426-445.
Higgins, J. P. T., Jackson, D., Barrett, J. K., Lu, G., Ades, A. E., & White, I. R. (2012). Consistency and inconsistency
in network meta-analysis: concepts and models for multi-arm studies. Research Synthesis Methods, 3, 98-110.
Higgins, J. P., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21,
1539-1558.
Higgins, J. P., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003). Measuring inconsistency in meta-analysis.
British Medical Journal, 327, 557-560.
Howard, G. S., Maxwell, S. E., & Fleming, K. J. (2000). The proof of the pudding: An illustration of the relative
strengths of null hypothesis, meta-analysis, and Bayesian analysis. Psychological Methods, 5, 315-332.
Huedo-Medina, T. B., Sanchez-Meca, J., Marin-Martinez, F., & Botella, J. (2006). Assessing heterogeneity in metaanalysis: Q statistic or I2 index? Psychological Methods, 11, 193-206.
Jackson, D., White, I. R., & Riley, R. D. (2012). Quantifying the impact of between-study heterogeneity in multivariate
meta-analyses. Statistics in Medicine, 31, 3805-3820.
Kulinskaya, E., & Koricheva, J. (2010). Use of quality control charts for detection of outliers and temporal trends in
cumulative meta-analysis. Research Synthesis Methods, 1, 297-307.
Lijmer, J. G., Bossuyt, P. M., & Heisterkamp, S. H. (2002). Exploring sources of heterogeneity in systematic reviews
of diagnostic tests. Statistics in Medicine, 21, 1525-1537.
Pereira, T. V., Patsopoulos, N. A., Salanti, G., & Ioannidis, J. P. A. (2010). Critical interpretation of Cochran's Q test
depends on power and prior assumptions about heterogeneity. Research Synthesis Methods, 1, 149-161.
Sterne, J. A., Juni, P., Schulz, K. F., Altman, D. G., Bartlett, C., & Egger, M. (2002). Statistical methods for assessing
the influence of study characteristics on treatment effects in 'meta-epidemiological' research. Statistics in
Medicine, 21, 1513-1524.
Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003). Adjusting for publication bias in the presence of heterogeneity.
Statistics in Medicine, 22, 2113-2126.
Thompson, S. G., & Higgins, J. P. (2002). How should meta-regression analyses be undertaken and interpreted?
Statistics in Medicine, 21, 1559-1573.
Research Problems – Spring 2015: Willcutt
19
Supplemental readings: statistical procedures for meta-analyses
Bax, L., Yu, L. M., Ikeda, N., & Moons, K. G. (2007). A systematic comparison of software dedicated to meta-analysis
of causal studies. BMC Medical Research Methodology, 7, 40.
Bond, C. F., Jr., Wiitala, W. L., & Richard, F. D. (2003). Meta-analysis of raw mean differences. Psychological
Methods, 8, 406-418.
Bonett, D. G. (2007). Transforming odds ratios into correlations for meta-analytic research. American Psychologist,
62, 254-255.
Bonett, D. G. (2008). Meta-analytic interval estimation for bivariate correlations. Psychological Methods, 13, 173-181.
Bonett, D. G. (2009). Meta-analytic interval estimation for standardized and unstandardized mean differences.
Psychological Methods, 14, 225-238.
Bonett, D. G. (2010). Varying coefficient meta-analytic methods for alpha reliability. Psychological Methods, 15, 368385.
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2010). A basic introduction to fixed-effect and
random-effects models for meta-analysis. Research Synthesis Methods, 1, 97-111.
Botella, J., Suero, M., & Gambara, H. (2010). Psychometric inferences from a meta-analysis of reliability and internal
consistency coefficients. Psychological Methods, 15, 386-397.
Cai, T., Parast, L., & Ryan, L. (2010). Meta-analysis for rare events. Statistics in Medicine, 29, 2078-2089.
Cheung, M. W., & Chan, W. (2005). Meta-analytic structural equation modeling: a two-stage approach. Psychological
Methods, 10, 40-64.
Darlington, R. B., & Hayes, A. F. (2000). Combining independent p values: extensions of the Stouffer and binomial
methods. Psychological Methods, 5, 496-515.
DerSimonian, R., & Laird, N. (1986). Metaanalysis in Clinical-Trials. Controlled Clinical Trials, 7, 177-188.
Field, A. P. (2001). Meta-analysis of correlation coefficients: a Monte Carlo comparison of fixed- and random-effects
methods. Psychological Methods, 6, 161-180.
Field, A. P. (2005). Is the meta-analysis of correlation coefficients accurate when population correlations vary?
Psychological Methods, 10, 444-467.
Furlow, C. F., & Beretvas, S. N. (2005). Meta-analytic methods of pooling correlation matrices for structural equation
modeling under different patterns of missing data. Psychological Methods, 10, 227-254.
Hafdahl, A. R., & Williams, M. A. (2009). Meta-analysis of correlations revisited: attempted replication and extension
of Field's (2001) simulation studies. Psychological Methods, 14, 24-42.
Hedges, L. V., & Pigott, T. D. (2001). The power of statistical tests in meta-analysis. Psychological Methods, 6, 203217.
Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent
effect size estimates. Research Synthesis Methods, 1, 39-65.
Henmi, M., & Copas, J. B. (2010). Confidence intervals for random effects meta-analysis and robustness to
publication bias. Statistics in Medicine, 29, 2969-2983.
Rodriguez, M. C., & Maeda, Y. (2006). Meta-analysis of coefficient alpha. Psychological Methods, 11, 306-322.
Sanchez-Meca, J., & Marin-Martinez, F. (2008). Confidence intervals for the overall effect size in random-effects
meta-analysis. Psychological Methods, 13, 31-48.
Thorlund, K., Wetterslev, J., Awad, T., Thabane, L., & Gluud, C. (2011). Comparison of statistical inferences from the
DerSimonian-Laird and alternative random-effects model meta-analyses: an empirical assessment of 920
Cochrane primary outcome meta-analyses. Research Synthesis Methods, 2, 238-253.
Valentine, J. C., Pigott, T. D., & Rothstein, H. R. (2010). How many studies do you need? A primer on statistical
power in meta-analysis. Journal of Educational and Behavioral Statistics, 35, 215-247.
Research Problems – Spring 2015: Willcutt
20
ETHICAL ISSUES IN RESEARCH
APRIL 28, 2015
Fly, B. J., Van Bark, W. P., Weinman, L., Kitchener, K. S., & Lang, P. R. (1997). Ethical transgressions of psychology
graduate students: Critical incidents with implications for training. Professional Psychology: Research & Practice,
28, 492-495.
Pipes, R. B., Blevins, T., & Kluck, A. (2008). Confidentiality, ethics, and informed consent. American Psychologist, 63,
623-624; discussion 624-625.
Rosnow, R. L. (1997). Hedgehogs, foxes, and the evolving social contract in psychological science: Ethical challenges
and methodological opportunities. Psychological Methods, 2, 345-356.
Steinberg, L., Cauffman, E., Woolard, J., Graham, S., & Banich, M. (2009). Are adolescents less mature than adults?:
minors' access to abortion, the juvenile death penalty, and the alleged APA "flip-flop". American Psychologist, 64,
583-594.