PSYCHOLOGY 865
APPLIED RESEARCH METHODS
FALL 2012
Instructor: Dr. Ann Marie Ryan
Office: 333 Psychology Building
Phone: 517-353-8855
Email: ryanan@msu.edu
Office hours: by appointment
Course website is on ANGEL
Class meeting time: 9:10-12, Wednesdays
Class meeting location: 325 Psychology Building
Objectives:
 To develop skills to conduct high quality, relevant applied research.
 To familiarize one with common designs and concerns related to conducting
research in applied settings.
The course focuses on planning and executing applied research, with a particular focus on
differences from highly controlled research settings. For example, in planning we focus
on the contrast between problem identification in applied settings and traditional
hypothesis generation, and on issues such as determining resource needs. In discussing
the execution of applied research, we focus on ruling out validity threats in quasiexperimental designs, project management, and reporting of results to applied audiences.
We also focus on writing for publication. The course will mix basic readings on design
with exercises. Specifically, you will be asked to do assignments to help you selfevaluate your skills and as preludes to class discussions– these are not just “busy work”.
To enhance the relevance of the class to your specific research interests, discussions are
predicated on your preparation for class with specific examples from your research
domain. For example, you may be asked to read an article of your choosing in a domain
of interest to you to discuss in class as illustrating the principles we are learning. You
may be asked to come prepared to describe how you would design a study in your
research area along the lines of the design elements we are discussing. Thus, class
preparation is critical.
Texts:
Required:
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2001). Experimental and quasiexperimental designs for generalized causal inference. Boston, Houghton
Mifflin Company.
Huff, A.S. (2009). Designing research for publication. Sage Publications.
Required readings are available as PDFs on the class website (in Angel).
1
Grading Criteria
Participation (discussion, attendance)
Theory assignment #1
Theory assignment #2
Experiment critique
Quasi experiment critique
Nonexperimental critique
Method Presentation
Meta analysis critique
Reviewer assignment
Grant Proposal
Weight Due
15%
weekly
5% Sept 12
5% Sept 26
5% Oct 10
5% Oct 17
5% Oct 24
20% Oct31,Nov7&14
5% Nov 21
5%
Nov 28
30%
Dec 10
Detailed directions for specific assignments can be found under the Lessons tab on
Angel.
Attendance Policy: For graduate courses, there is a lot of in-class exchange of ideas and
discussion of readings. Missing class is problematic and will be considered in awarding
of participation points. Absences will be excused only in accordance with ombudsmen’s
website on Attendance Policy (see www.msu.edu/unit/ombud)
Academic Integrity: Article 2.3.3 of the Academic Freedom Report states that “The
student shares with the faculty the responsibility for maintaining the integrity of
scholarship, grades, and professional standards.” In addition, the Psychology Department
adheres to the policies on academic honesty as specified in General Student Regulations
1.0, Protection of scholarship and grades, the all-University Policy on Integrity of
scholarship and Grades, and Ordinance 17.00, Examinations (see MSU website).
Therefore, unless specifically directed otherwise, you are expected to complete all course
assignments, including homework, papers and exams, without assistance from any
source. You are expected to develop original work for this course; therefore, you may
not submit course work you completed for another course to satisfy the requirements for
this course. Students who violate MSU rules will receive a failing grade in this course.
Accommodations: If you require special accommodations with regard to a disability,
please discuss that with me. If you will be missing class or assignment due to a religious
observance, please let me know in advance so alternative arrangements can be made.
Other: Commercialization of lecture notes and university-provided course materials is not
permitted in this course.
2
Week 1: What to Study (September 5)
Objectives:
 Cover course logistics and expectations
 Examine factors that make research interesting and significant
 How to write the initial section of papers
Huff, Ch 1 and Ch 2, p 2-37; Ch 7 p127-146
Colquitt, J. A., & George, G. (2011, June). From The Editors. Academy of Management
Journal. pp. 432-435. doi:10.5465/AMJ.2011.61965960
Leong, F. T., Schmitt, N. & Lyons, B. (in press). Developing Testable and Important
Research Questions. In APA Handbook of Research Methods by Harris Cooper, Debra
Long, Paul Camic, Richard Gonzalez, Abigail Panter, and Kenneth Sher.
Grant, A. M., & Pollock, T. G. (2011, October). Publishing In Amj-Part 3: Setting The
Hook. Academy of Management Journal. pp. 873-879. doi:10.5465/amj.2011.4000.
Preparation: Research Self-efficacy Assessment (on the website)
Questions to think about (on the website)
Huff exercise 4 (don’t write anything – just prepare to discuss)
Week 2: Literature review/theory development (Sept 12)
Objectives:
 Understand what makes research “theory building”
 Formulating hypotheses
Huff, Ch 8, p 147-166 ONLY; Ch 11, p217-247 (note that the exercises 38-40 in the
chapter will form basis for an assignment due next week so read through them)
Feldman, D.C. (2004). What are we talking about when we talk about theory? Journal of
Management, 30 565-567.
Sutton, R.I. & Staw, B. M. (1995). What theory is not. Administrative Science
Quarterly, 40, 371-384.
Weick, Karl E (1995). What theory is not, theorizing is. Administrative Science
Quarterly; 40,385-391.
Leavitt, K., Mitchell, T. R. & Peterson, J. (2010). Theory pruning: strategies to reduce
our dense theoretical landscape. Organizational Research Methods, 13, 644-667
3
Corley, K. G., & Gioia, D. A. (2011). Building theory about theory building: what
constitutes a theoretical contribution? Academy of Management Review, 36(1), 12-32.
doi:10.5465/AMR.2011.55662499
Smithey Fulmer, I. (2012). Editor's Comments: The Craft of Writing Theory ArticlesVariety and Similarity in AMR. Academy Of Management Review, 37(3), 327-331.
Greenwald, A.G. (2012). There Is Nothing So Theoretical as a Good Method
Perspectives on Psychological Science, vol. 7, 2: pp. 99-108.
Sparrowe, R. T., & Mayer, K. J. (2011). Publishing In Amj--Part 4: Grounding
Hypotheses. Academy Of Management Journal, 54(6), 1098-1102.
doi:10.5465/amj.2011.4001
Preparation: Questions to think about (on the website)
Due: Theory assignment #1(dropbox)
Optional: Blog on lit review/search tips
Week 3: Research Design Fundamentals (September 19)
Objectives:
 Understand what contributes to construct validity, statistical conclusion validity,
internal validity, and external validity
 Review basic elements of design
Shadish, Cook, & Campbell, Chapters 1, 2 and 3 (pp. 1-102)
Huff, Chapter 9, 179-200
Combs, J.G. (2010). Big samples and small effects: let’s not trade relevance and rigor
for power. Academy of Management Journal, 53, 9-14.
Brutus, S., Gill, H. & Duniewicz, K. (2010). State-of-science in industrial and
organizational psychology: a review of self-reported limitations. Personnel
Psychology,63, 907-936.
Bono, J. E., & McNamara, G. (2011, August). From The Editors: Publishing In Amj-Part 2: Research Design. Academy of Management Journal. pp. 657-660.
doi:10.5465/AMJ.2011.64869103.
Zhang, Y. & Shaw, J.D. (2012). Crafting the methods and results. Academy of
Management Journal, 55, 8-12.
4
Carlson, K.D. & Herdman, A.O. (2012). Understanding the impact of convergent
validity on research results. Organizational Research Methods, 15, 17-32.
Preparation: online tutorials on internal validity (for review as you need them)
Questions to think about (on website)
Optional: blog: learning from disappointing results
Week 4: Resources for research (September 26)
MEET IN BEAUMONT INSTRUCTIONAL ROOM IN MAIN LIBRARY
Objectives:
 Familiarize yourself with funding sources
 Discuss challenges in obtaining funding
 Understand what contributes to successful proposals
 Gaining access to and conducting research in applied contexts
Grant workshop led by Jon Harrison, meet at Main Library
Huff, Chapter 5, p 85-106.
Jelinek, M. & Griffith, T.L. (2005). Organizational science and the NSF: funding for
mutual benefit. Organizational Science, 16, 550-559.
Munsey, C. (2009). 8 Tips for funding your dissertation. APA Monitor, 64-67.
Ryan, A. M. & Pulakos, E.D. (2007). Conducting meaningful research in a fast-paced
and volatile work world: challenges and opportunities. In J.P. Hodgkinson & J.K. Ford
(Eds.). International review of industrial and organizational psychology, London, Wiley.
P265-281 ONLY
Berger, L.K., Begun, A.L. & Otto-Salaj, L.L. 2009. Participant recruitment in
intervention research: scientific integrity and cost-effective strategies. International
Journal of Social Research Methodology, 12, 79-92.
Preparation: Questions to think about (on website)
Due: Theory development #2 (dropbox)
Optional: Blog on gaining access
Week 5: Research ethics (October 3)
Objective:
Discuss key ethical concerns specific to research in applied settings
Define your stance on debated ethical issues
Lefkovitz, J. (2003). Ethics and values in industrial-organizational psychology.
Lawrence Erlbaum, Mahwah, NJ.
5
Chapter 13: Research Ethics: I. Informed consent and confidentiality p331-358
Chapter 14: Research Ethics: II. The use of deception p359-386
Rosenthal, R. (1994). Science and ethics in conducting, analyzing, and reporting
psychological research. Psychological Science, 5, 127-134.
Schminke, M. (2009). The better angels of our nature – ethics and integrity in the
publishing process. Academy of Management Review, 34, 586-591.
Tyldum, G. (2012). Ethics or access? Balancing informed consent against the application
of institutional, economic or emotional pressures in recruiting respondents for research.
International Journal of Social Research Methodology, 15, 199-210.
Honig, B. & Bedi, A. (2012). The Fox in the Hen House: A Critical Examination of
Plagiarism Among Members of the Academy of Management. Academy of Management
Learning & Education. 11, 101-123.
John, L.K., Loewenstein,G. & Prelec, D. (2012). Measuring the Prevalence of
Questionable Research Practices With Incentives for Truth Telling Psychological
Science, 23, 524-532.
Preparation: CITI modules (your choice; do workers as vulnerable population)
Questions to think about (on website)
Due: Presentation topic (email for approval)
Optional: Blog on ethical concerns
Week 6: Experimental Designs (October 10)
Objectives:
 Understand what makes something an experiment and why they are valuable
 Consider challenges in conducting experiments in applied settings
 Particular focus on generalizability from lab to field
Shadish, Cook, & Campbell, Chapters 8 & 9 (pp. 246-313).
Highhouse, S. (2009). Designing experiments that generalize. Organizational Research
Methods. 12, 554-566
Mitchell, G. (2012). Revisiting Truth or Triviality: The External Validity of Research in
the Psychological Laboratory. Perspectives on Psychological Science, 7, 109-117.
Johns, G. (2006). The essential impact of context on organizational behavior. Academy
of Management Review, 31, 386-408.
Example article: Vuori, J.; Toppinen-Tanner, S.; Mutanen, P. (2012). Effects of
resource-building group intervention on career management and mental health in work
6
organizations: Randomized controlled field trial. Journal of Applied Psychology 97: 273286.
Preparation: Questions to think about (on website)
Due: Experiment critique (dropbox)
Week 7: Quasi-experimental Designs (October 17)
Objectives:
Familiarize yourself with various quasi-experimental designs and their
limitations
Uncover ways of ruling out alternative explanations
Consider issues in establishing mediation
Shadish, Cook, & Campbell, Chapters 4 & 5 (pp. 103-170).
Grant, A. M. & Wall, T.D. (2009) The neglected science and art of quasiexperimentation: why-to, when-to, and how-to advice for organizational researchers.
Organizational Research Methods, 12, 653-686.
Aussens, M.E, Boomsma, A. & Snijders, T.A.B. (2011). The use of quasi-experiments in
the social sciences: a content analysis. Quality & Quantity, 45, 21-42.
MacKinnon, D.P. & Baraldi, A. N. (2012). Guidelines for the investigation of mediating
variables in business research. Journal of Business and Psychology, 27, 1-14.
Example article: Leiter, M. P.; Laschinger, H. K.; Day, A.; Oore, D. G. (2011). The
impact of civility interventions on employee social behavior, distress, and attitudes.
Journal of Applied Psychology 96: 1258-1274.
Preparation: Questions to think about (on website)
DUE: PROPOSAL TOPIC – email me your general idea
Due: Quasi critique (dropbox)
Week 8: Survey research: sampling, response rates, and CMV (October 24)
Objectives:
Discuss key issues in surveying, esp. sampling strategies and sample
representativeness issues, lessening non response, and CMV
Henry, G. T. (1998). Practical sampling. In L. Bickman & D. J. Rog (Eds.), Handbook of
applied social research methods (Chapter 4, pp. 101-126). Thousand Oaks, CA: Sage.
Shen, W., Kiger, T. B., Davies, S. E., Rasch, R. L., Simon, K. M., & Ones, D. S. (2011).
Samples in applied psychology: Over a decade of research in review. Journal of Applied
Psychology, 96(5), 1055-1064. doi: 10.1037/a0023322
7
Rogelberg, S.G. & Stanton, J.M. (2007). Understanding and dealing with organizational
survey nonresponse. Organizational Research Methods, 10, 195-209.
Fauth, T., Hattrup, K., Mueller, K. & Roberts, B. (published online April 2012).
Nonresponse in employee attitude surveys: a group-level analysis. Journal of Business
and Psychology.
Meade, A.W. & Craig, S.B. (published online April 2012). Identifying careless
responses in survey data. Psychological Methods.
Podsakoff, P. M., MacKenzie, S. B., & Podsakoff, N. P. (2012). Sources of method bias
in social science research and recommendations on how to control it. Annual Review
of Psychology, 63, 539.
Preparation:
Questions to think about (on website)
Due: Nonexperimental critique (Drop box)
Week 9: Mitigation: Attrition, missing data, statistical control; Methods
presentations (October 31)
Objective:
To understand how to mitigate some common problems encountered in
field research
Shadish, Cook, & Campbell, Chapter 10
Meade, A.W., Behrend, T.S. & Lance, C.E. (2009). Dr. StrangeLOVE or how I learned
to stop worrying and love omitted variables, p 89-106. In Lance, C.E. & Vandenberg,
R.J. (2009). (Eds.). Statistical and methodological myths and urban legends. New York:
Routledge
Newman, D.A. (2009). Missing data techniques and low response rates: the role of
systematic nonresponse parameters. Lance, C.E. & Vandenberg, R.J. (2009). (Eds.).
Statistical and methodological myths and urban legends. New York: Routledge p7-36.
Atinc, G., Simmering, M.J. & Kroll, M.J. (2012). Control variable use and reporting in
macro and micro management research. Organizational Research Methods, 15, 57-74.
Carlson, K.D. & Wu, J. (2012). The illusion of statistical control: control variable
practice in management research. Organizational Research Methods, 15, 413-435.
Preparation: Questions to think about (on website)
Optional: Blog: Mitigation
3 people will present
8
Week 10: Qualitative Research; Methods Presentations (Nov 7)
Objectives:
To gain a broad overview of some of the techniques and aims of
qualitative research
To understand some of the challenges in conducting qualitative research
O’Reilly, K., Paper, D. & Marx, S. (2012). Demystifying grounded theory for business
research. Organizational Research Methods, 15, 247-262.
Eby, L.T., Hurst, C.S. & Butts, M (2009). Qualitative research: the redheaded stepchild
in organizational and social science research? In Lance, C.E. & Vandenberg, R.J. (2009).
(Eds.). Statistical and methodological myths and urban legends. New York: Routledge
P219-246.
Pratt, M. G. (2009). For the lack of a boilerplate: tips on writing up (and reviewing)
qualitative research. Academy of Management Journal, 52, 856-862.
Bansal, P. & Corley, K. (2012) What's Different about Qualitative Research? Academy of
Management Journal. 55, 509-513.
Acocella, I. (2012). The focus groups in social research: advantages and disadvantages.
Quality & Quantity, 46, 1125-1136
Bergman, M.M. (2011). The good, the bad, and the ugly in mixed methods research and
design. Journal of Mixed Methods Research, 5, 271-275.
Preparation: Bring in example qualitative article in your area to discuss
Questions to think about (on website)
2 members of the class will do presentations
Week 11: Emerging Methods and Issues; Method Presentations (Nov 14)
To discuss new and emerging methods and methodological issues
Chan, D. (2011). Advances in analytical strategies In S. Zedeck (Ed.) APA handbook of
industrial and organizational psychology, Vol 1: Building and developing the
organization, 85-113. Washington, DC, US: American Psychological Association.
Uhlmann, E.L., Leavitt, K., Menges, J.I., Koopman, J., Howe, M. & Johnson, R.E. (in
press). Getting explicit about the implicit: a taxonomy of implicit measures and guide
for their use in organizational research. Organizational Research Methods,
Wilson, R.E., Gosling, S.D. & Graham, L.T. (2012). A review of Facebook research in
the social sciences. Perspectives on Psychological Science, 7, 203-220.
9
Miller, G. (2012). The Smartphone Psychology Manifesto Perspectives on Psychological
Science, 7, 221-237.
Ployhart, R.E. & Ward, A-K (2011). The “quick start guide” for conducting and
publishing longitudinal research. Journal of Business and Psychology, 26, 413-422.
3 members of the class will do method presentations
Week 12: Synthesizing research (November 21)
Objective: To gain an overview of methods of synthesizing research (i.e., literature
synthesis, meta-analytic synthesis)
Shadish, Cook, & Campbell, Chapter 13
Huff p 166-177 only
Chan, M.E. & Arvey, R.D. (2012). Meta-Analysis and the Development of Knowledge
Perspectives on Psychological Science, 7, 79-92.
Carlson, K.D. & Ji, F.X. (in press). Citing and Building on meta-analytic findings: a
review and recommendations. Organizational Research Methods.
Aytug, Z.G., Rothstein, H.R., Zhou, W. & Kern, M.C. (in press). Revealed or concealed?
Transparency of procedures, decisions, and judgment calls in meta-analyses.
Organizational Research Methods.
Ferguson, C. J., & Brannick, M. T. (2012). Publication bias in psychological science:
Prevalence, methods for identifying and controlling, and implications for the use of metaanalyses. Psychological Methods, 17(1), 120-128. doi: 10.1037/a0024445
Rothstein, H. R., & Bushman, B. J. (2012). Publication bias in psychological science:
Comment on Ferguson and Brannick (2012). Psychological Methods, 17(1), 129-136.
doi: 10.1037/a0027128
Dalton, D.R. & Aguinis, H., Dalton, C. M., Bosco, F.A. & Pierce, C.A. (2012).
Revisiting The File Drawer Problem In Meta-Analysis: An Assessment Of Published
And Nonpublished Correlation Matrices. Personnel Psychology, 65, 221-249.
Rousseau, D. M., Manning, J., Denyer, D. (2008). Evidence in management and
organizational science: assembling the field’s full weight of scientific knowledge
through syntheses, Academy of Management Annals, 2, 475-515.
Preparation: meta-analysis critique (in your area of interest) (dropbox)
Questions to think about (on the website)
10
Week 13: Reporting research in academic settings (Nov 28)
Objective:
To discuss some key issues in preparing research for publication
To enhance your skill as a reviewer
To gain self-awareness regarding writing skills
Huff, Ch 12, p251-268
Bem, D. (2003). Writing the empirical journal article. In J.M. Darlye, M.P. Zanna, &
H.L. Roediger (Eds.). The Compleat Academic: a practical guide for the beginning
social scientists. Washington DC: American Psychological Association.
Roediger, H.L. (2007). Twelve tips for authors. APS Observer, 20, 39-41
Zahra, S.A. & Neubaum, D.O. (2006). Revising to be published: building trust to win
the acceptance of journal editors and reviewers. In Y. Baruch, S.E. Sullivan & H.N.
Schepmyer (Eds). Winning reviews: a guide for evaluating scholarly writing.205-223.
Feldman, D.C. (2006). Communicating more effectively with editors: strategies for
authors and reviewers. In Y. Baruch, S.E. Sullivan & H.N. Schepmyer (Eds). Winning
reviews: a guide for evaluating scholarly writing. 236-250.
Lepak, D. (2009). What is good reviewing? Academy of Management Review, 34, 375381.
Day, N. (2011). The Silent Majority: Manuscript Rejection and Its Impact on Scholars.
Academy Of Management Learning & Education, 10(4), 704-718.
doi:10.5465/amle.2010.0027
Preparation: Questions to think about (on the website)
DUE: Reviewer assignment (dropbox)
Week 14: Translation, implementation, and impact (Dec 5)
Objective:
To understand the challenges in translating research into application
To discuss how to design research to facilitate relevance and
implementation
To consider the challenges of conducting research across cultures
Huff, Chapter 10, p201-216
Rynes, S.L. (2012). The research-practice gap in I/O Psychology and related fields:
challenges and potential solutions. In S. Kozlowski (Ed). Oxford Handbook of
Industrial and Organizational Psychology.
11
Bartunek, J.M. & Rynes, S.L. (2010). The construction and contributions of
“implications for practice”: What’s in them and what might they offer? Academy of
Management Learning & Education, 9, 100-118.
Mohrman, S.A. & Lawler, E.E. (2012). Generating Knowledge That Drives Change.
Academy of Management Perspectives. 26, 41-51.
Rousseau, D. M. & Boudreau, J.W. (2011). Sticky findings: research evidence
practitioners find useful. In S.A. Mohrman & E.E. Lawler (Eds.). Useful research:
advancing theory and practice. San Francisco: Berrett-Koehler Publishers. P269288.
Geletkanycz, M. & Tepper, B.J. 2012. Discussing the Implications. Academy of
Management Journal, 55, 256-260.
Ryan, A. M. & Pulakos, E.D. (2007). Conducting meaningful research in a fast-paced
and volatile work world: challenges and opportunities. In J.P. Hodgkinson & J.K. Ford
(Eds.). International review of industrial and organizational psychology, London, Wiley.
P281-286 ONLY
Gelfand, M. J. (2012). The trials and tribulations of cross-cultural research. In A. M.
Ryan, F. T. L. Leong & F. L. Oswald (Eds.), Conducting multinational research:
Applying organizational psychology in the workplace. (pp. 179-199). Washington,
DC, US: American Psychological Association, Washington, DC. doi: 10.1037/13743007
Week 15:
PROPOSALS DUE at finals time on DEC 10, 7:45-9:45 AM
Final: Please note that we are REQUIRED to meet during finals week, either for an
exam or regular class meeting. Do not consider this meeting optional, although we’ll
try to make it a fun ending.
12