PSYCHOLOGY 865
APPLIED RESEARCH METHODS
FALL 2010
Instructor: Dr. Ann Marie Ryan
Office: 333 Psychology Building
Phone: 517-353-8855
Email: ryanan@msu.edu
Office hours: by appointment
Course website is on ANGEL
Class meeting time: 9:10-12, Wednesdays
Class meeting location: 325 Psychology Building
Objectives:
 To develop skills to conduct high quality, relevant applied research.
 To familiarize one with common designs and concerns related to conducting
research in applied settings.
The course focuses on planning and executing applied research, with a particular focus on
differences from highly controlled research settings. For example, in planning we focus
on the contrast between problem identification in applied settings and traditional
hypothesis generation, and on issues such as determining resource needs. In discussing
the execution of applied research, we focus on ruling out validity threats in quasiexperimental designs, project management, and reporting of results to applied audiences.
We also focus on writing for publication. The course will mix basic readings on design
with exercises. Specifically, you will be asked to do assignments to help you selfevaluate your skills and as preludes to class discussions– these are not just “busy work”.
To enhance the relevance of the class to your specific research interests, discussions are
predicated on your preparation for class with specific examples from your research
domain. For example, you may be asked to read an article of your choosing in a domain
of interest to you to discuss in class as illustrating the principles we are learning. You
may be asked to come prepared to describe how you would design a study in your
research area along the lines of the design elements we are discussing. Thus, class
preparation is critical.
Texts:
Required:
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2001). Experimental and quasiexperimental designs for generalized causal inference. Boston, Houghton
Mifflin Company.
Huff, A.S. (2009). Designing research for publication. Sage Publications.
Readings are available as PDFs on the class website (in Angel).
1
Suggested additional reading:
Lance, C.E. & Vandenberg, R.J. (2009). (Eds.). Statistical and methodological
myths and urban legends. New York: Routledge. (we read several chapters from
this volume and they are available as PDFs on Angel but you might find some of
the other content really useful and worth purchasing).
Grading Criteria
Participation (class preparation, discussion, attendance)
Method Presentation
Theory assignment
Inferences assignment
Reviewer assignment
Grant Proposal
Weight
25%
25%
5%
5%
5%
35%
Due
weekly
TBD
10/6
11/17
12/1
12/14
Information on specific assignments can be found under the Lessons tab on Angel.
Attendance Policy: For graduate courses, there is a lot of in-class exchange of ideas and
discussion of readings. Missing class is problematic and will be considered in awarding
of participation points. Absences will be excused only in accordance with ombudsmen’s
website on Attendance Policy (see www.msu.edu/unit/ombud)
Academic Integrity: Article 2.3.3 of the Academic Freedom Report states that “The
student shares with the faculty the responsibility for maintaining the integrity of
scholarship, grades, and professional standards.” In addition, the Psychology Department
adheres to the policies on academic honesty as specified in General Student Regulations
1.0, Protection of scholarship and grades, the all-University Policy on Integrity of
scholarship and Grades, and Ordinance 17.00, Examinations (see MSU website).
Therefore, unless specifically directed otherwise, you are expected to complete all course
assignments, including homework, papers and exams, without assistance from any
source. You are expected to develop original work for this course; therefore, you may
not submit course work you completed for another course to satisfy the requirements for
this course. Students who violate MSU rules will receive a failing grade in this course.
Accommodations: If you require special accommodations with regard to a disability,
please discuss that with me. If you will be missing class or assignment due to a religious
observance, please let me know in advance so alternative arrangements can be made.
Other: Commercialization of lecture notes and university-provided course materials is not
permitted in this course.
2
Week 1: What to Study (September 8)
Objectives:
 Cover course logistics and expectations
 Discuss how research proceeds in the “ideal” and reality compromises.
 Examine factors that make research interesting and significant
Huff, Ch 1 and Ch 2, p 2-37; Ch 7 p127-146
Martin, J. (1982). A garbage can model of the research process. In J. E. McGrath, J.
Martin, & R. A. Kulka (Eds.), Judgment calls in research (pp. 17-40). Beverly Hills,
CA: Sage.
Ryan, A. M. & Pulakos, E.D. (2007). Conducting meaningful research in a fast-paced
and volatile work world: challenges and opportunities. In J.P. Hodgkinson & J.K. Ford
(Eds.). International review of industrial and organizational psychology, London, Wiley.
Preparation: Research Self-efficacy Assessment (on the website)
Questions to think about (on the website)
Huff exercise 4 (prepare to discuss in class)
Week 2: Literature review/theory development (Sept 15)
Objectives:
 Understand what makes research “theory building”
 Formulating hypotheses
 Introduce basic concepts related to inferring causality and generalizing causality
Huff, Ch 8, p 147-166 ONLY; Ch 11, p217-247 (note that the exercises in the chapter
will form basis for an assignment due 10/6)
Shadish, Cook, & Campbell, Chapter 1 (pp. 1-32)
Feldman, D.C. (2004). What are we talking about when we talk about theory? Journal of
Management, 30 565-567.
Sutton, R.I. & Staw, B. M. (1995). What theory is not. Administrative Science
Quarterly, 40, 371-384.
Weick, Karl E (1995). What theory is not, theorizing is. Administrative Science
Quarterly; 40,385-391.
Leavitt, K., Mitchell, T. R. & Peterson, J. (in press). Theory pruning: strategies to
reduce our dense theoretical landscape. Organizational Research Methods,
Preparation: Questions to think about (on the website)
3
Week 3: Research Design Fundamentals (September 22)
Objectives:
 Understand what contributes to construct validity, statistical conclusion validity,
internal validity, and external validity
 Review basic elements of design
Shadish, Cook, & Campbell, Chapter 2 and 3 (pp. 33-102)
Huff, Chapter 5, p85-106
Chapter 9, 179-200
Combs, J.G. (2010). Big samples and small effects: let’s not trade relevance and rigor
for power. Academy of Management Journal, 53, 9-14.
Brutus, S., Gill, H. & Duniewicz, K. (in press). State-of-science in industrial and
organizational psychology: a review of self-reported limitations. Personnel Psychology,
Preparation: online tutorials on internal validity
Questions to think about (on website)
Huff exercise 12 (you do not have to write this out – just give it
some thought)
Week 4: Research ethics (September 29)
Objective:
Discuss key ethical concerns specific to research in applied settings
Define your stance on debated ethical issues
Lefkovitz, J. (2003). Ethics and values in industrial-organizational psychology.
Lawrence Erlbaum, Mahwah, NJ.
Chapter 13: Research Ethics: I. Informed consent and confidentiality p331-358
Chapter 14: Research Ethics: II. The use of deception p359-386
Rosenthal, R. (1994). Science and ethics in conducting, analyzing, and reporting
psychological research. Psychological Science, 5, 127-134.
Schminke, M. (2009). The better angels of our nature – ethics and integrity in the
publishing process. Academy of Management Review, 34, 586-591.
Preparation: do the CITI modules
Questions to think about (on website)
DUE: PRESENTATION TOPIC – email your idea to me
4
Week 5: Resources for research (October 6)
Objectives:
 Familiarize yourself with funding sources
 Discuss challenges in obtaining funding
 Understand what contributes to successful proposals
Grant workshop led by Jon Harrison, meet at Main Library Basement Instruction
Room.
Jelinek, M. & Griffith, T.L. (2005). Organizational science and the NSF: funding for
mutual benefit. Organizational Science, 16, 550-559.
Munsey, C. (2009). 8 Tips for funding your dissertation. APA Monitor, 64-67.
Preparation: Questions to think about (on website)
DUE: Theory development exercise – be prepared to discuss
Week 6: Experimental Designs (October 13)
Objectives:
Understand what makes something an experiment and why they are
valuable
Consider challenges in conducting experiments in applied settings
Shadish, Cook, & Campbell, Chapters 8 & 9 (pp. 246-313).
Highhouse, S. (2009). Designing experiments that generalize. Organizational Research
Methods. 12, 554-566
Collins, L. M., Dziak, J.J. & Li, R. (2009). Design of experiments with multiple
independent variables: a resource management perspective on complete and reduced
factorial designs. Psychological Methods, 14, 202-234.
Preparation: Questions to think about (on website)
5
Week 7: Quasi-experimental Designs (October 20)
Objectives:
Familiarize yourself with various quasi-experimental designs and their
limitations
Uncover ways of ruling out alternative explanations
Shadish, Cook, & Campbell, Chapters 4 & 5 (pp. 103-170).
Grant, A. M. & Wall, T.D. (2009) The neglected science and art of quasiexperimentation: why-to, when-to, and how-to advice for organizational researchers.
Organizational Research Methods, 12, 653-686.
Preparation: Questions to think about (on website)
DUE: PROPOSAL TOPIC – email me your general idea
Week 8: Survey research: sampling, response rates, and CMV (October 27)
Objectives:
Discuss key issues in surveying, esp. sampling strategies and sample
representativeness issues, lessening non response, and CMV
Henry, G. T. (1998). Practical sampling. In L. Bickman & D. J. Rog (Eds.), Handbook of
applied social research methods (Chapter 4, pp. 101-126). Thousand Oaks, CA: Sage.
Rogelberg, S.G. & Stanton, J.M. (2007). Understanding and dealing with organizational
survey nonresponse. Organizational Research Methods, 10, 195-209.
Podsakoff, P.M., MacKenzie, S.B., Lee, J. & Podsakoff, N.P. (2003). Common method
biases in behavioral research: A critical review of the literature and recommended
remedies. Journal of Applied Psychology, 88, 879-903.
Pace, V. (2010) Method variance from the perspectives of reviewers: Poorly understood
problem or overemphasized complaint? Organizational Research Methods, 13, 421434.
Lance, C.E., Dawson, B., Birkelbach, D. & Hoffman, B.J. (2010). Method effects,
measurement error, and substantive conclusions. Organizational Research Methods,
13, 435-455.
Preparation:
Questions to think about (on website)
Bring in a copy of a survey (your own or someone else’s) to discuss
Over next several classes, part of time will be given to Methods Presentations by class
members.
Objectives:
 To provide focus on methodological issues of interest to individual students
 To gain skill in synthesizing and summarizing information for others
6
Week 9: Qualitative Research (Nov 3)
2 people will present
Objectives: To gain a broad overview of some of the techniques and aims of
qualitative research
To understand some of the challenges in conducting qualitative research
Huff, Chapter 9, p179-200
Locke, K. (2002). The grounded theory approach to qualitative research. In. F. Drasgow
& N. Schmitt (Eds.), Measuring and analyzing behavior in organizations: Advances in
measurement and data analysis (pp. 17-43). San Francisco: Jossey-Bass.
Eby, L.T., Hurst, C.S. & Butts, M (2009). Qualitative research: the redheaded stepchild
in organizational and social science research? In Lance, C.E. & Vandenberg, R.J. (2009).
(Eds.). Statistical and methodological myths and urban legends. New York: Routledge
P219-246.
Pratt, M. G. (2009). For the lack of a boilerplate: tips on writing up (and reviewing)
qualitative research. Academy of Management Journal, 52, 856-862.
Preparation: Bring in example qualitative article in your area to discuss
Questions to think about (on website)
Week 10: Mitigation: Attrition, missing data, statistical control (November 10)
1 person will present
Objectives: To understand how to mitigate some common problems encountered
Shadish, Cook, & Campbell, Chapter 10
Meade, A.W., Behrend, T.S. & Lance, C.E. (2009). Dr. StrangeLOVE or how I learned
to stop worrying and love omitted variables, p 89-106. In Lance, C.E. & Vandenberg,
R.J. (2009). (Eds.). Statistical and methodological myths and urban legends. New York:
Routledge
Newman, D.A. (2009). Missing data techniques and low response rates: the role of
systematic nonresponse parameters. Lance, C.E. & Vandenberg, R.J. (2009). (Eds.).
Statistical and methodological myths and urban legends. New York: Routledge p7-36.
Becker, T. E. (2005). Potential problems in the statistical control of variables in
organizational research: a qualitative analysis with recommendations, Organizational
Research Methods, 8, 274-290.
Spector, P. & Brannick, M. (in press). Methodological urban legends: the misuse of
statistical control variables. Organizational Research Methods.
Preparation: Questions to think about (on website)
7
Week 11: Generalized Causal Inference (Nov 17)
2 people will present
Objectives:
To gain knowledge of what limits and enhances generalizability of
inferences
Shadish, Cook, & Campbell, Chapters 11 & 12 (pp. 341-455).
West, S. & Thommes, F. (2010). Campbell’s and Rubin’s perspectives on causal
inferences. Psychological Methods, 15, 18-37.
Shadish, W.R. (2010). Campbell and Rubin: A Primer and Comparison of Their
Approaches to Causal Inference in Field Settings, Psychological Methods, 15,3-17.
Johns, G. (2006). The essential impact of context on organizational behavior. Academy
of Management Review, 31, 386-408.
Preparation: Questions to think about (on the website)
DUE CAUSAL INFERENCE EXERCISE
Week 12: Methods: synthesizing research (November 24)
2 people will present
Objective: To gain an overview of methods of synthesizing research (i.e., literature
synthesis, meta-analytic synthesis)
Shadish, Cook, & Campbell, Chapter 13
Huff p 166-177 only
Carlson, K.D. & Ji, F.X. (2009). Citing and Building on meta-analytic findings: a
review and recommendations. Presented at SIOP, New Orleans.
Rousseau, D. M., Manning, J., Denyer, D. (2008). Evidence in management and
organizational science : assembling the field’s full weight of scientific knowledge
through syntheses, Academy of Management Annals, 2, 475-515.
Preparation: Bring example meta-analysis in your area of interest
Questions to think about (on the website)
8
Week 13: Reporting and presenting research in academic settings (December 1)
Objective:
To discuss some key issues in the preparing research for publication
To enhance your skill as a reviewer
To gain self-awareness regarding writing skills
Huff, Ch 12, p251-268
Bem, D. (2003). Writing the empirical journal article. In J.M. Darlye, M.P. Zanna, &
H.L. Roediger (Eds.). The Compleat Academic: a practical guide for the beginning
social scientists. Washington DC: American Psychological Association.
Roediger, H.L. (2007). Twelve tips for authors. APS Observer, 20, 39-41
Lane, D. M. & Sandor, A. (2009). Designing better graphs by including distributional
information and integrating words, numbers, and images. Psychological Methods, 14,
239-257.
Zahra, S.A. & Neubaum, D.O. (2006). Revising to be published: building trust to win
the acceptance of journal editors and reviewers. In Y. Baruch, S.E. Sullivan & H.N.
Schepmyer (Eds). Winning reviews: a guide for evaluating scholarly writing.205-223.
Feldman, D.C. (2006). Communicating more effectively with editors: strategies for
authors and reviewers. In Y. Baruch, S.E. Sullivan & H.N. Schepmyer (Eds). Winning
reviews: a guide for evaluating scholarly writing. 236-250.
Lepak, D. (2009). What is good reviewing? Academy of Management Review, 34, 375381.
Preparation: Questions to think about (on the website)
DUE: Manuscript review
9
Week 14: Translation, implementation, and impact (Dec 8)
Objective:
To understand the challenges in translating research into application
To discuss how to design research to facilitate relevance and
implementation
Huff, Chapter 10, p201-216
Rynes, S.L. (in press). The research-practice gap in I/O Psychology and related fields:
challenges and potential solutions. In s. Kozlowski (Ed). Oxford Handbook of
Industrial and Organizational Psychology.
Bartunek, J.M. & Rynes, S.L. (2010). The construction and contributions of
“implications for practice”: What’s in them and what might they offer? Academy of
Management Learning & Education, 9, 100-118.
Briner, R. & Rousseau, D. (in press). Evidence based I-O Psychology: Not there yet.
Industrial and Organizational Psychology: Perspectives on Science and Practice
PROPOSALS DUE at class time on DEC 14
Final: December 14, 7:45-9:45. Breakfast
Please note that we are REQUIRED to meet during finals week, either for an exam or
regular class meeting. Do not consider this meeting optional, although we’ll try to
make it a fun ending.
10