PSYCHOLOGY 865
APPLIED RESEARCH METHODS
FALL 2011
Instructor: Dr. Ann Marie Ryan
Office: 333 Psychology Building
Phone: 517-353-8855
Email: ryanan@msu.edu
Office hours: by appointment
Course website is on ANGEL
Class meeting time: 11:30-2:20, Wednesdays
Class meeting location: 325 Psychology Building
Objectives:
 To develop skills to conduct high quality, relevant applied research.
 To familiarize one with common designs and concerns related to conducting
research in applied settings.
The course focuses on planning and executing applied research, with a particular focus on
differences from highly controlled research settings. For example, in planning we focus
on the contrast between problem identification in applied settings and traditional
hypothesis generation, and on issues such as determining resource needs. In discussing
the execution of applied research, we focus on ruling out validity threats in quasiexperimental designs, project management, and reporting of results to applied audiences.
We also focus on writing for publication. The course will mix basic readings on design
with exercises. Specifically, you will be asked to do assignments to help you selfevaluate your skills and as preludes to class discussions– these are not just “busy work”.
To enhance the relevance of the class to your specific research interests, discussions are
predicated on your preparation for class with specific examples from your research
domain. For example, you may be asked to read an article of your choosing in a domain
of interest to you to discuss in class as illustrating the principles we are learning. You
may be asked to come prepared to describe how you would design a study in your
research area along the lines of the design elements we are discussing. Thus, class
preparation is critical.
Texts:
Required:
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2001). Experimental and quasiexperimental designs for generalized causal inference. Boston, Houghton
Mifflin Company.
Huff, A.S. (2009). Designing research for publication. Sage Publications.
Required readings are available as PDFs on the class website (in Angel).
1
Grading Criteria
Participation (discussion, attendance)
Theory assignment #1
Theory assignment #2
Experiment critique
Quasi experiment critique
Nonexperimental critique
Inference critique
Method Presentation
Meta analysis critique
Reviewer assignment
Grant Proposal
Weight
10%
5%
5%
5%
5%
5%
5%
20%
5%
5%
30%
Due
weekly
Sept 14
Sept 21
Oct 12
Oct 19
Oct 26
Nov 9
Nov 16
Nov 23
Nov 30
Dec 16
Detailed directions for specific assignments can be found under the Lessons tab on
Angel.
Attendance Policy: For graduate courses, there is a lot of in-class exchange of ideas and
discussion of readings. Missing class is problematic and will be considered in awarding
of participation points. Absences will be excused only in accordance with ombudsmen’s
website on Attendance Policy (see www.msu.edu/unit/ombud)
Academic Integrity: Article 2.3.3 of the Academic Freedom Report states that “The
student shares with the faculty the responsibility for maintaining the integrity of
scholarship, grades, and professional standards.” In addition, the Psychology Department
adheres to the policies on academic honesty as specified in General Student Regulations
1.0, Protection of scholarship and grades, the all-University Policy on Integrity of
scholarship and Grades, and Ordinance 17.00, Examinations (see MSU website).
Therefore, unless specifically directed otherwise, you are expected to complete all course
assignments, including homework, papers and exams, without assistance from any
source. You are expected to develop original work for this course; therefore, you may
not submit course work you completed for another course to satisfy the requirements for
this course. Students who violate MSU rules will receive a failing grade in this course.
Accommodations: If you require special accommodations with regard to a disability,
please discuss that with me. If you will be missing class or assignment due to a religious
observance, please let me know in advance so alternative arrangements can be made.
Other: Commercialization of lecture notes and university-provided course materials is not
permitted in this course.
2
Week 1: What to Study (September 7)
Objectives:
 Cover course logistics and expectations
 Discuss how research proceeds in the “ideal” and how reality leads to
compromises.
 Examine factors that make research interesting and significant
Huff, Ch 1 and Ch 2, p 2-37; Ch 7 p127-146
Martin, J. (1982). A garbage can model of the research process. In J. E. McGrath, J.
Martin, & R. A. Kulka (Eds.), Judgment calls in research (pp. 17-40). Beverly Hills,
CA: Sage.
Ryan, A. M. & Pulakos, E.D. (2007). Conducting meaningful research in a fast-paced
and volatile work world: challenges and opportunities. In J.P. Hodgkinson & J.K. Ford
(Eds.). International review of industrial and organizational psychology, London, Wiley.
Preparation: Research Self-efficacy Assessment (on the website)
Questions to think about (on the website)
Huff exercise 4 (don’t write anything – just prepare to discuss)
3
Week 2: Literature review/theory development (Sept 14)
Objectives:
 Understand what makes research “theory building”
 Formulating hypotheses
 Introduce basic concepts related to inferring causality and generalizing causality
Huff, Ch 8, p 147-166 ONLY; Ch 11, p217-247 (note that the exercises 38-40 in the
chapter will form basis for an assignment due next week so read through them)
Feldman, D.C. (2004). What are we talking about when we talk about theory? Journal of
Management, 30 565-567.
Sutton, R.I. & Staw, B. M. (1995). What theory is not. Administrative Science
Quarterly, 40, 371-384.
Weick, Karl E (1995). What theory is not, theorizing is. Administrative Science
Quarterly; 40,385-391.
Leavitt, K., Mitchell, T. R. & Peterson, J. (2010). Theory pruning: strategies to reduce
our dense theoretical landscape. Organizational Research Methods, 13, 644-667
Alvesson, M. & Sandberg, J. (2011). Generating research questions through
problematization. The Academy of Management Review 36. 2 : 247-271
Corley, K. G., & Gioia, D. A. (2011). Building theory about theory building: what
constitutes a theoretical contribution? Academy of Management Review, 36(1), 12-32.
doi:10.5465/AMR.2011.55662499
Preparation: Questions to think about (on the website)
Due: Theory assignment #1(dropbox)
Optional: Blog on lit review/search tips
4
Week 3: Resources for research (September 21)
Objectives:
 Familiarize yourself with funding sources
 Discuss challenges in obtaining funding
 Understand what contributes to successful proposals
Grant workshop led by Jon Harrison, meet at Main Library Basement Instruction
Room.
Huff, Chapter 5, p 85-106.
Jelinek, M. & Griffith, T.L. (2005). Organizational science and the NSF: funding for
mutual benefit. Organizational Science, 16, 550-559.
Munsey, C. (2009). 8 Tips for funding your dissertation. APA Monitor, 64-67.
Preparation: Questions to think about (on website)
Due: Theory development #2 (dropbox)
Optional: Blog on gaining access
Week 4: Research Design Fundamentals (September 28)
Objectives:
 Understand what contributes to construct validity, statistical conclusion validity,
internal validity, and external validity
 Review basic elements of design
Shadish, Cook, & Campbell, Chapters 1, 2 and 3 (pp. 1-102)
Huff, Chapter 9, 179-200
Combs, J.G. (2010). Big samples and small effects: let’s not trade relevance and rigor
for power. Academy of Management Journal, 53, 9-14.
Brutus, S., Gill, H. & Duniewicz, K. (2010). State-of-science in industrial and
organizational psychology: a review of self-reported limitations. Personnel
Psychology,63, 907-936.
Preparation: online tutorials on internal validity (for review)
Questions to think about (on website)
Optional: blog: learning from disappointing results
5
Week 5: Research ethics (October 5)
Objective:
Discuss key ethical concerns specific to research in applied settings
Define your stance on debated ethical issues
Lefkovitz, J. (2003). Ethics and values in industrial-organizational psychology.
Lawrence Erlbaum, Mahwah, NJ.
Chapter 13: Research Ethics: I. Informed consent and confidentiality p331-358
Chapter 14: Research Ethics: II. The use of deception p359-386
Rosenthal, R. (1994). Science and ethics in conducting, analyzing, and reporting
psychological research. Psychological Science, 5, 127-134.
Schminke, M. (2009). The better angels of our nature – ethics and integrity in the
publishing process. Academy of Management Review, 34, 586-591.
Preparation: CITI modules (your choice; do workers as vulnerable population)
Questions to think about (on website)
Due: Presentation topic (email for approval)
Optional: Blog on ethical concerns
Week 6: Experimental Designs (October 12)
Objectives:
Understand what makes something an experiment and why they are
valuable
Consider challenges in conducting experiments in applied settings
Shadish, Cook, & Campbell, Chapters 8 & 9 (pp. 246-313).
Highhouse, S. (2009). Designing experiments that generalize. Organizational Research
Methods. 12, 554-566
Example article: Fan, J. & Wanous, J.P. (2008). Organizational and cultural entry: a
new type of orientation program for multiple boundary crossings, Journal of Applied
Psychology, 93, 1390-1400.
Preparation: Questions to think about (on website)
Due: Experiment critique (dropbox)
6
Week 7: Quasi-experimental Designs (October 19)
Objectives:
Familiarize yourself with various quasi-experimental designs and their
limitations
Uncover ways of ruling out alternative explanations
Shadish, Cook, & Campbell, Chapters 4 & 5 (pp. 103-170).
Grant, A. M. & Wall, T.D. (2009) The neglected science and art of quasiexperimentation: why-to, when-to, and how-to advice for organizational researchers.
Organizational Research Methods, 12, 653-686.
Example article: Davidson et al. (2010). Sabbatical leave: who gains and how much?
Journal of Applied Psychology, 95, 953-964.
Preparation: Questions to think about (on website)
DUE: PROPOSAL TOPIC – email me your general idea
Due: Quasi critique (dropbox)
Week 8: Survey research: sampling, response rates, and CMV (October 26)
Objectives:
Discuss key issues in surveying, esp. sampling strategies and sample
representativeness issues, lessening non response, and CMV
Henry, G. T. (1998). Practical sampling. In L. Bickman & D. J. Rog (Eds.), Handbook of
applied social research methods (Chapter 4, pp. 101-126). Thousand Oaks, CA: Sage.
Rogelberg, S.G. & Stanton, J.M. (2007). Understanding and dealing with organizational
survey nonresponse. Organizational Research Methods, 10, 195-209.
Podsakoff, P.M., MacKenzie, S.B., Lee, J. & Podsakoff, N.P. (2003). Common method
biases in behavioral research: A critical review of the literature and recommended
remedies. Journal of Applied Psychology, 88, 879-903.
Lance, C.E., Dawson, B., Birkelbach, D. & Hoffman, B.J. (2010). Method effects,
measurement error, and substantive conclusions. Organizational Research Methods,
13, 435-455.
Preparation:
Questions to think about (on website)
Due: Nonexperimental critique (Drop box)
7
Week 9: Mitigation: Attrition, missing data, statistical control (November 2)
Objective:
To understand how to mitigate some common problems encountered
Shadish, Cook, & Campbell, Chapter 10
Meade, A.W., Behrend, T.S. & Lance, C.E. (2009). Dr. StrangeLOVE or how I learned
to stop worrying and love omitted variables, p 89-106. In Lance, C.E. & Vandenberg,
R.J. (2009). (Eds.). Statistical and methodological myths and urban legends. New York:
Routledge
Newman, D.A. (2009). Missing data techniques and low response rates: the role of
systematic nonresponse parameters. Lance, C.E. & Vandenberg, R.J. (2009). (Eds.).
Statistical and methodological myths and urban legends. New York: Routledge p7-36.
Spector, P. & Brannick, M. (2011). Methodological urban legends: the misuse of
statistical control variables. Organizational Research Methods, 14, 287-305.
Atinc, G., Simmering, M.J. & Kroll, M.J. (in press). Control variable use and reporting
in macro and micro management research. Organizational Research Methods.
Preparation: Questions to think about (on website)
Optional: Blog: Mitigation
8
Week 10: Qualitative Research and Generalized Causal Inferences (Nov 9)
Objectives:
To gain a broad overview of some of the techniques and aims of
qualitative research
To understand some of the challenges in conducting qualitative research
To gain knowledge of what limits and enhances generalizability of
inferences
Shadish, Cook, & Campbell, Chapters 11 & 12 (pp. 341-455).
Locke, K. (2002). The grounded theory approach to qualitative research. In. F. Drasgow
& N. Schmitt (Eds.), Measuring and analyzing behavior in organizations: Advances in
measurement and data analysis (pp. 17-43). San Francisco: Jossey-Bass.
Eby, L.T., Hurst, C.S. & Butts, M (2009). Qualitative research: the redheaded stepchild
in organizational and social science research? In Lance, C.E. & Vandenberg, R.J. (2009).
(Eds.). Statistical and methodological myths and urban legends. New York: Routledge
P219-246.
Pratt, M. G. (2009). For the lack of a boilerplate: tips on writing up (and reviewing)
qualitative research. Academy of Management Journal, 52, 856-862.
Preparation: Bring in example qualitative article in your area to discuss
Questions to think about (on website)
Inference critique (dropbox)
Week 11: Method Presentations (Nov 16)
9
Week 12: Synthesizing research (November 23)
Objective: To gain an overview of methods of synthesizing research (i.e., literature
synthesis, meta-analytic synthesis)
Shadish, Cook, & Campbell, Chapter 13
Huff p 166-177 only
Carlson, K.D. & Ji, F.X. (in press). Citing and Building on meta-analytic findings: a
review and recommendations. Organizational Research Methods.
Aytug, Z.G., Rothstein, H.R., Zhou, W. & Kern, M.C. (in press). Revealed or concealed?
Transparency of procedures, decisions, and judgment calls in meta-analyses.
Organizational Research Methods.
Rousseau, D. M., Manning, J., Denyer, D. (2008). Evidence in management and
organizational science: assembling the field’s full weight of scientific knowledge
through syntheses, Academy of Management Annals, 2, 475-515.
Briner, R.B. & Rousseau, D.M. (2011). Evidence-based I-O psychology: not there yet.
Industrial and organizational psychology: perspectives on science and practice, 4, 3-22.
Burke, M.J. (2011). Is there a fly in the “systematic review” ointment? Industrial and
organizational psychology: perspectives on science and practice, 4, 36-39.
Banks, G.C. & McDaniel, M.A. (2011). The kryptonite of evidence-based I-O
Psychology. Industrial and organizational psychology: perspectives on science and
practice, 4, 40-44.
Preparation: meta-analysis critique (in your area of interest) (dropbox)
Questions to think about (on the website)
10
Week 13: Reporting research in academic settings (Nov 30)
Objective:
To discuss some key issues in the preparing research for publication
To enhance your skill as a reviewer
To gain self-awareness regarding writing skills
Huff, Ch 12, p251-268
Bem, D. (2003). Writing the empirical journal article. In J.M. Darlye, M.P. Zanna, &
H.L. Roediger (Eds.). The Compleat Academic: a practical guide for the beginning
social scientists. Washington DC: American Psychological Association.
Roediger, H.L. (2007). Twelve tips for authors. APS Observer, 20, 39-41
Zahra, S.A. & Neubaum, D.O. (2006). Revising to be published: building trust to win
the acceptance of journal editors and reviewers. In Y. Baruch, S.E. Sullivan & H.N.
Schepmyer (Eds). Winning reviews: a guide for evaluating scholarly writing.205-223.
Feldman, D.C. (2006). Communicating more effectively with editors: strategies for
authors and reviewers. In Y. Baruch, S.E. Sullivan & H.N. Schepmyer (Eds). Winning
reviews: a guide for evaluating scholarly writing. 236-250.
Lepak, D. (2009). What is good reviewing? Academy of Management Review, 34, 375381.
Preparation: Questions to think about (on the website)
DUE: Manuscript review (dropbox)
Week 14: Translation, implementation, and impact (Dec 7)
Objective:
To understand the challenges in translating research into application
To discuss how to design research to facilitate relevance and
implementation
Huff, Chapter 10, p201-216
Johns, G. (2006). The essential impact of context on organizational behavior. Academy
of Management Review, 31, 386-408.
Rynes, S.L. (in press). The research-practice gap in I/O Psychology and related fields:
challenges and potential solutions. In S. Kozlowski (Ed). Oxford Handbook of
Industrial and Organizational Psychology.
Bartunek, J.M. & Rynes, S.L. (2010). The construction and contributions of
“implications for practice”: What’s in them and what might they offer? Academy of
Management Learning & Education, 9, 100-118.
11
Week 15:
PROPOSALS DUE at finals time on DEC 16, 10-12
Final: Please note that we are REQUIRED to meet during finals week, either for an
exam or regular class meeting. Do not consider this meeting optional, although we’ll
try to make it a fun ending.
12