PSYCHOLOGY 865
APPLIED RESEARCH METHODS
FALL 2013
Instructor: Dr. Ann Marie Ryan
Office: 333 Psychology Building
Phone: 517-353-8855
Email: ryanan@msu.edu
Office hours: by appointment
Course website is on D2L
Class meeting time: 9:10-12, Wednesdays
Class meeting location: 325 Psychology Building
Objectives:
 To develop skills to conduct high quality, relevant applied research.
 To familiarize one with common designs and concerns related to conducting
research in applied settings.
 To develop strategies for mitigating concerns
 To enhance understanding of the publication process
The course focuses on planning and executing applied research, with a particular focus on
differences from highly controlled research settings. For example, in planning we focus
on the contrast between problem identification in applied settings and traditional
hypothesis generation, and on issues such as determining resource needs. In discussing
the execution of applied research, we focus on ruling out validity threats in quasiexperimental designs, project management, and reporting of results to applied audiences.
We also focus on writing for publication. The course will mix basic readings on design
with exercises. Specifically, you will be asked to do assignments to help you selfevaluate your skills and as preludes to class discussions– these are not just “busy work”.
To enhance the relevance of the class to your specific research interests, discussions are
predicated on your preparation for class with specific examples from your research
domain. For example, you may be asked to read an article of your choosing in a domain
of interest to you to discuss in class as illustrating the principles we are learning. You
may be asked to come prepared to describe how you would design a study in your
research area along the lines of the design elements we are discussing. Thus, class
preparation is critical.
Texts:
Required:
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2001). Experimental and quasiexperimental designs for generalized causal inference. Boston, Houghton
Mifflin Company.
Huff, A.S. (2009). Designing research for publication. Sage Publications.
1
Silvia, P.J. (2007). How to Write a Lot: A Practical Guide to Productive
Academic Writing. Washington, DC: American Psychological Association.
Required readings are available as links on the class website (in D2L).
Grading Criteria
Participation (discussion, attendance)
Theory assignment #1
Theory assignment #2
Experiment critique
Quasi experiment critique
Nonexperimental critique
Method Presentation
Meta analysis critique
Reviewer assignment
Grant Proposal
Weight Due
15%
weekly
5% Sept 11
5% Sept 18
5% Oct 9
5% Oct 16
5% Oct 23
20%Oct30,Nov6 &13
5% Nov 20
5%
Nov 27
30%
Dec 11
Detailed directions for specific assignments can be found on D2L.
Extra credit
There are no extra credit opportunities –there are multiple opportunities for you to
improve your grade.
Late or missed assignments. Handing in assignments later than the start of class on the
date due is not acceptable. Late assignments ARE NOT ACCEPTED. Note that the D2L
dropbox for assignments automatically stops taking submissions within minutes of the
start of class on dates when assignments are due – do NOT wait until the last minute to
submit the assignment.
Grade posting
Grades will be posted in a timely fashion. It is your responsibility to check the posted
grades throughout the term for accuracy.
Attendance Policy: For graduate courses, there is a lot of in-class exchange of ideas and
discussion of readings. Missing class is problematic and will be considered in awarding
of participation points. Absences will be excused only in accordance with ombudsmen’s
website on Attendance Policy (see www.msu.edu/unit/ombud)
Academic Integrity: Article 2.3.3 of the Academic Freedom Report states that “The
student shares with the faculty the responsibility for maintaining the integrity of
scholarship, grades, and professional standards.” In addition, the Psychology Department
adheres to the policies on academic honesty as specified in General Student Regulations
1.0, Protection of scholarship and grades, the all-University Policy on Integrity of
scholarship and Grades, and Ordinance 17.00, Examinations (see MSU website).
Therefore, unless specifically directed otherwise, you are expected to complete all course
assignments, including homework, papers and exams, without assistance from any
2
source. You are expected to develop original work for this course; therefore, you may
not submit course work you completed for another course to satisfy the requirements for
this course. Students who violate MSU rules will receive a failing grade in this course.
Accommodations: If you require special accommodations with regard to a disability,
please discuss that with me. If you will be missing class or assignment due to a religious
observance, please let me know in advance so alternative arrangements can be made.
Other: Commercialization of lecture notes and university-provided course materials is not
permitted in this course.
Instructor Availability:
While I do have office hours on M and W for my undergraduates, I am available to meet
with graduate students at other times by appointment. However, my schedule is a full
one and so dropping by may not be the best strategy – please email to schedule
appointments. I do check email frequently. Also, a little common courtesy in your email
is more likely to generate an equally courteous and timely response.
Classroom Rules of Conduct
Cell phones must be turned off before class starts. Individual conversations outside of set
discussion are disruptive to other learners – if you are disruptive, you will be asked to
leave the class. The course is structured to encourage discussion and interaction – please
treat other members of the class with courtesy and respect. Texting, emailing, game
playing, and web surfing during class are counterproductive to learning, lessen your
capacity to engage with those around you, and can be rude. Leave the outside world
outside of class and be fully present. If for some reason you must arrive late or leave
early (e.g., illness, car trouble), please take a seat near the door and minimize disruption
of others with your arrival/departure. While I do not mind beverages in class, please
refrain from eating unless you have enough to share with everyone.
Online expectations
D2L is new to me and to many of you as well so please be patient with any snags or
missteps. Some assignments will require you to post to a discussion board your thoughts
and ideas. While I encourage you to be yourself in your expressions, I also expect
courtesy and respect for others. Please refrain from language that others might find
offensive. There are topics we will discuss where individuals may have very different
opinions and views – please express your dissenting viewpoint diplomatically.
Emergency Procedures
If an emergency should occur that would require the cancellation of class, I will send an
email via D2L.
Tips for Success
This is not a class of lecturing – class time will be spent on integration with the
assumption that you are coming to class prepared. Some hints for being successful:
 Read any assignment BEFORE class.
3




Finish the assignments early
Ask questions.
Share your own experiences.
Be engaged– you get more if you give more.
Tentative course schedule
(this schedule is tentative and subject to change)
4
Week 1: What to Study (Sept 4)
Objectives:
 Cover course logistics and expectations
 Examine factors that make research interesting and significant
 Review how to write the initial section of papers
Huff, Ch 1 and Ch 2, p 2-37; Ch 7 p127-146
Colquitt, J. A., & George, G. (2011, June). From The Editors: topic choice. Academy of
Management Journal. pp. 432-435.
Leong, F. T., Schmitt, N. & Lyons, B. (in press). Developing Testable and Important
Research Questions. In APA Handbook of Research Methods by Harris Cooper, Debra
Long, Paul Camic, Richard Gonzalez, Abigail Panter, and Kenneth Sher.
Pilluta, MM & Thau, S (2013). Organizational sciences’ obsession with “that’s
interesting!”: consequences and an alternative. Organizational Psychology Review, 3,
187-194.
Alvesson, M & Gabriel, Y (2013). Beyond formulaic research: in praise of greater
diversity in organizational research and publications. Academy of Management Learning
& Education, 12, 245-263.
Grant, A. M., & Pollock, T. G. (2011). Publishing In Amj-Part 3: Setting The Hook.
Academy of Management Journal. pp. 873-879.
Preparation: Questions to think about (on the website)
Huff exercise 4
For both of these: don’t write anything – just prepare to discuss
Week 2: Literature reviews and theory development (Sept 11)
Objectives:
 Understand what makes research “theory building”
 Review formulating hypotheses
Huff, Ch 8, p 147-166 ONLY; Ch 11, p217-247 (note that the exercises 38-40 in the
chapter will form basis for an assignment due next week so read through them)
Feldman, D.C. (2004). What are we talking about when we talk about theory? Journal of
Management, 30 565-567.
Sutton, R.I. & Staw, B. M. (1995). What theory is not. Administrative Science
Quarterly, 40, 371-384.
5
Weick, Karl E (1995). What theory is not, theorizing is. Administrative Science
Quarterly; 40,385-391.
Leavitt, K., Mitchell, T. R. & Peterson, J. (2010). Theory pruning: strategies to reduce
our dense theoretical landscape. Organizational Research Methods, 13, 644-667
Corley, K. G., & Gioia, D. A. (2011). Building theory about theory building: what
constitutes a theoretical contribution? Academy of Management Review, 36(1), 12-32.
doi:10.5465/AMR.2011.55662499
Smithey Fulmer, I. (2012). Editor's Comments: The Craft of Writing Theory ArticlesVariety and Similarity in AMR. Academy Of Management Review, 37(3), 327-331.
Sparrowe, R. T., & Mayer, K. J. (2011). Publishing In Amj--Part 4: Grounding
Hypotheses. Academy Of Management Journal, 54(6), 1098-1102.
doi:10.5465/amj.2011.4001
Schaubroeck, J (2012). Pitfalls of appropriating prestigious theories to frame conceptual
arguments Organizational Psychology Review, 1-12.
Preparation: Questions to think about (on the website)
Due: Theory assignment #1(dropbox)
Week 3: Resources for research (September 18)
MEET IN BEAUMONT INSTRUCTIONAL ROOM IN MAIN LIBRARY
Objectives:
 Familiarize yourself with funding sources
 Discuss challenges in obtaining funding
 Uncover helpful tips for use of literature databases and bibliographic tools
 Understand what contributes to successful proposals
 Gaining access to and conducting research in applied contexts
Grant workshop led by Jon Harrison and Database Information by Jill Morningstar, meet
at Main Library
Huff, Chapter 5, p 85-106.
Munsey, C. (2009). 8 Tips for funding your dissertation. APA Monitor, 64-67.
Salas E How to get #3 million, TIP
Roch, SG An investigation of research grants in published articles, TIP.
6
Berger, L.K., Begun, A.L. & Otto-Salaj, L.L. 2009. Participant recruitment in
intervention research: scientific integrity and cost-effective strategies. International
Journal of Social Research Methodology, 12, 79-92.
Bamberger & Pratt (2010) Moving forward by looking back: reclaiming unconventional
research contexts and samples in organizational scholarship. Academy of Management
Journal, 53, 665-671.
Preparation:
Questions to think about (on website)
Week 4: Research Design Fundamentals (September 25)
NOTE THIS IS A HEAVY READING WEEK – READ AHEAD!
Objectives:
 Understand what contributes to construct validity, statistical conclusion validity,
internal validity, and external validity
 Review basic elements of design
Shadish, Cook, & Campbell, Chapters 1, 2 and 3 (pp. 1-102)
Huff, Chapter 9, 179-200
Brutus, S., Gill, H. & Duniewicz, K. (2010). State-of-science in industrial and
organizational psychology: a review of self-reported limitations. Personnel
Psychology,63, 907-936.
Bono, J. E., & McNamara, G. (2011, August). From The Editors: Publishing In Amj-Part 2: Research Design. Academy of Management Journal. pp. 657-660.
Zhang, Y. & Shaw, J.D. (2012). Crafting the methods and results. Academy of
Management Journal, 55, 8-12.
Preparation: online tutorials on internal validity (for review as you need them)
Questions to think about (on website)
Optional: blog: learning from disappointing results
Week 5: Research ethics (Oct 2)
Objective:
Discuss key ethical concerns specific to research in applied settings
Define your stance on debated ethical issues
Lefkovitz, J. (2003). Ethics and values in industrial-organizational psychology.
Lawrence Erlbaum, Mahwah, NJ.
7
Chapter 13: Research Ethics: I. Informed consent and confidentiality p331-358
Chapter 14: Research Ethics: II. The use of deception p359-386
Colquitt JA (2013) Data overlap policies at AMJ. Academy of Management Journal, 56,
331-333.
Colquitt JA (2012). Plagiarism policies and screening at AMJ. Academy of Management
Journal, 55, 749-751.
Stroebe, W., Postmes, T., & Spears, R. (2012). Scientific misconduct and the myth of
self-correction in science. Perspectives on Psychological Science, 7(6), 670-688
Preparation: CITI modules (your choice; do workers as vulnerable population)
Questions to think about (on website)
Due: Presentation topic (email for approval)
Optional: Blog on ethical concerns
Week 6: Experimental Designs (October 9)
Objectives:
 Understand what makes something an experiment and why they are valuable
 Consider challenges in conducting experiments in applied settings
 Particular focus on generalizability from lab to field
Shadish, Cook, & Campbell, Chapters 8 & 9 (pp. 246-313).
Highhouse, S. (2009). Designing experiments that generalize. Organizational Research
Methods. 12, 554-566
Mitchell, G. (2012). Revisiting Truth or Triviality: The External Validity of Research in
the Psychological Laboratory. Perspectives on Psychological Science, 7, 109-117.
Colquitt, JA 2008 Publishing laboratory research in AMJ: A question of when, not if.
Academy of Management Journal, 51, 616-620.
King, EB, Hebl MR Morgan WB & Ahmad AS (2012). Field experiments on sensitive
organizational topics. Organizational Research Methods,
Example article: Vuori, J.; Toppinen-Tanner, S.; Mutanen, P. (2012). Effects of
resource-building group intervention on career management and mental health in
work organizations: Randomized controlled field trial. Journal of Applied
Psychology 97: 273-286.
Example article: Lester, PB Hannah ST, Harms PD Vogelgesang GR & Avolio
BJ 2011. Mentoring impact on leader efficacy development: a field experiment.
Academy of Management Learning and Education, 10, 409-429.
8
Preparation: Questions to think about (on website)
Due: Experiment critique (dropbox)
Week 7: Quasi-experimental Designs (October 16 9)
Objectives:
Familiarize yourself with various quasi-experimental designs and their
limitations
Uncover ways of ruling out alternative explanations
Shadish, Cook, & Campbell, Chapters 4 & 5 (pp. 103-170).
Grant, A. M. & Wall, T.D. (2009) The neglected science and art of quasiexperimentation: why-to, when-to, and how-to advice for organizational researchers.
Organizational Research Methods, 12, 653-686.
Connelly BS Sackett PR & Waters SD 2013. Balancing treatment and control groups in
quasi-experiments: an introduction to propensity scoring. Personnel Psychology, 66,
407-442.
Example article: Leiter, M. P.; Laschinger, H. K.; Day, A.; Oore, D. G. (2011).
The impact of civility interventions on employee social behavior, distress, and
attitudes. Journal of Applied Psychology 96: 1258-1274.
Example article (LOOK AT STUDY 1 ONLY): Wagner, D. T., Barnes, C. M.,
Lim, V. K. G., & Ferris, D. L. (2012). Lost sleep and cyberloafing: Evidence from
the laboratory and a daylight saving time quasi-experiment. Journal of Applied
Psychology, 97(5), 1068-1076
Example: Parker, S. K., Johnson, A., Collins, C., & Nguyen, H. (2013). Making
the most of structural support: Moderating influence of employees' clarity and
negative affect. Academy of Management Journal, 56(3), 867-892.
Preparation:
Questions to think about (on website)
DUE: PROPOSAL TOPIC – email me your general idea
Due: Quasi critique (dropbox)
Week 8: Survey research: sampling, response rates, and CMV (October 23)
Objectives:
Discuss key issues in surveying, esp. sampling strategies and sample
representativeness issues, lessening non response, and CMV
Henry, G. T. (1998). Practical sampling. In L. Bickman & D. J. Rog (Eds.), Handbook of
applied social research methods (Chapter 4, pp. 101-126). Thousand Oaks, CA: Sage.
9
Rogelberg, S.G. & Stanton, J.M. (2007). Understanding and dealing with organizational
survey nonresponse. Organizational Research Methods, 10, 195-209.
Fauth, T., Hattrup, K., Mueller, K. & Roberts, B. (2013). Nonresponse in employee
attitude surveys: a group-level analysis. Journal of Business and Psychology. 28, 1-16
Meade, A.W. & Craig, S.B. (2012). Identifying careless responses in survey data.
Psychological Methods. 17, 437-455
Podsakoff, P. M., MacKenzie, S. B., & Podsakoff, N. P. (2012). Sources of method bias
in social science research and recommendations on how to control it. Annual Review
of Psychology, 63, 539.
Preparation:
Questions to think about (on website)
Due: Nonexperimental critique (Drop box)
Week 9: Mitigation: Attrition, missing data, statistical control; Methods
presentations (October 30)
Objective:
Understand how to mitigate some common problems encountered in field
research
Shadish, Cook, & Campbell, Chapter 10
Newman, D.A. (2009). Missing data techniques and low response rates: the role of
systematic nonresponse parameters. Lance, C.E. & Vandenberg, R.J. (2009). (Eds.).
Statistical and methodological myths and urban legends. New York: Routledge p7-36.
Aguinis, H., Gottfredson, RK & Joo, H (2013). Best-Practice Recommendations for
Defining, Identifying, and Handling Outliers. Organizational Research
Methods 16: 270-301
Carlson, K.D. & Wu, J. (2012). The illusion of statistical control: control variable
practice in management research. Organizational Research Methods, 15, 413-435.
Preparation: Questions to think about (on website)
Methods presentations
Week 10: Qualitative Research; Methods Presentations (Nov 6)
Objectives:
To gain a broad overview of some of the techniques and aims of
qualitative research
To understand some of the challenges in conducting qualitative research
10
Gephardt. RP 2013. Doing research with words: qualitative methodologies and
industrial/organizational psychology. In JM Cortina & RS Landis (Eds). Modern
research methods for the study of behavior in organizations. Routledge: New York
265-318.
O’Reilly, K., Paper, D. & Marx, S. (2012). Demystifying grounded theory for business
research. Organizational Research Methods, 15, 247-262.
Pratt, M. G. (2009). For the lack of a boilerplate: tips on writing up (and reviewing)
qualitative research. Academy of Management Journal, 52, 856-862.
Bansal, P. & Corley, K. (2012) What's Different about Qualitative Research? Academy of
Management Journal. 55, 509-513.
Acocella, I. (2012). The focus groups in social research: advantages and disadvantages.
Quality & Quantity, 46, 1125-1136
Gioia, DA, Corley, KG & Hamilton AL (2013). Seeking Qualitative Rigor in Inductive
Research: Notes on the Gioia Methodology Organizational Research Methods, . 16, 1531.
Example article: Wanberg, C, Basbug, G, van Hooft, EAJ & Samtani A 2012
Navigating the black hole: explicating layers of job search context and
adaptational responses. Personnel Psychology, 65, 887-926
Preparation: Bring in example qualitative article in your area to discuss
Questions to think about (on website)
presentations
Week 11: Emerging Methods (Nov 13)
To discuss emerging methods and methodological issues
Dimotakis, N, Ilies R & Judge TA 2013. Experience sampling methodology. In JM
Cortina & RS Landis (Eds.). Modern research methods for the study of behavior in
organizations. Routledge: New York p319-348
Vancouver, J. B., & Weinhardt, J. M. (2012). Modeling the mind and the milieu:
Computational modeling for micro-level organizational researchers. Organizational
Research Methods, 15(4), 602-623.
Stanton JM 2013 Data mining: a practical introduction for organizational researchers.
In JM Cortina & RS Landis (Eds.). Modern research methods for the study of behavior
in organizations. Routledge: New York p199-232.
11
Uhlmann, E.L., Leavitt, K., Menges, J.I., Koopman, J., Howe, M. & Johnson, R.E. (in
press). Getting explicit about the implicit: a taxonomy of implicit measures and guide
for their use in organizational research. Organizational Research Methods,
Ployhart, R.E. & Ward, A-K (2011). The “quick start guide” for conducting and
publishing longitudinal research. Journal of Business and Psychology, 26, 413-422.
Krasikova, DV & LeBreton JM 2012. Just the two of us: misalignment of theory and
methods in examining dyadic phenomena. Journal of Applied Psychology, 97, 739-757.
presentations
Week 12: Synthesizing research (November 20)
Objective: To gain an overview of methods of synthesizing research (i.e., literature
synthesis, meta-analytic synthesis)
Shadish, Cook, & Campbell, Chapter 13
Huff p 166-177 only
Carlson, K.D. & Ji, F.X. (2011). Citing and Building on meta-analytic findings: a
review and recommendations. Organizational Research Methods, 14, 696-717
Aytug, Z.G., Rothstein, H.R., Zhou, W. & Kern, M.C. (2012). Revealed or concealed?
Transparency of procedures, decisions, and judgment calls in meta-analyses.
Organizational Research Methods. 15, 103-133
Humphrey, SE 2011 What does a great meta-analysis look like? Organizational
Psychology Review, 1, 99-103.
Aguinis, H, Pierce CA, Bosco FA, Dalton DA, Dalton CM 2011 Debunking myths and
urban legends about meta-analysis, Organizational Research Methods, 14, 306-311.
Kepes, S., Banks, GC, McDaniel, M, Whetzel DL (2012). Publication Bias in the
Organizational Sciences. Organizational Research Methods. 15, 624-662.
Rousseau, D. M., Manning, J., Denyer, D. (2008). Evidence in management and
organizational science: assembling the field’s full weight of scientific knowledge
through syntheses, Academy of Management Annals, 2, 475-515.
Preparation: meta-analysis critique (in your area of interest) (dropbox)
Questions to think about (on the website)
12
Week 13: Reporting research in academic settings (Nov 27)
Objective:
To discuss some key issues in preparing research for publication
To enhance your skill as a reviewer
To gain self-awareness regarding writing skills
Silva, p 3-132.
Huff, Ch 12, p251-268
Bem, D. (2003). Writing the empirical journal article. In J.M. Darlye, M.P. Zanna, &
H.L. Roediger (Eds.). The Compleat Academic: a practical guide for the beginning
social scientists. Washington DC: American Psychological Association.
Roediger, H.L. (2007). Twelve tips for authors. APS Observer, 20, 39-41
Zahra, S.A. & Neubaum, D.O. (2006). Revising to be published: building trust to win
the acceptance of journal editors and reviewers. In Y. Baruch, S.E. Sullivan & H.N.
Schepmyer (Eds). Winning reviews: a guide for evaluating scholarly writing.205-223.
Feldman, D.C. (2006). Communicating more effectively with editors: strategies for
authors and reviewers. In Y. Baruch, S.E. Sullivan & H.N. Schepmyer (Eds). Winning
reviews: a guide for evaluating scholarly writing. 236-250.
Lepak, D. (2009). What is good reviewing? Academy of Management Review, 34, 375381.
Ragins, BR (2012) Reflections on the craft of clear writing. Academy of Management
Review, 37: 493-501.
Shaw JD (2012). Responding to reviewers. Academy of Management Journal, 55, 12611263.
Pollock TG & Bono JE 2013 Being Scheherazade: the importance of storytelling in
academic writing. Academy of Management Journal, 56 629-634.
Preparation: Questions to think about (on the website)
DUE: Reviewer assignment (dropbox)
Week 14: Translation, implementation, and impact (Dec 4)
Objective:
To understand the challenges in translating research into application
To discuss how to design research to facilitate relevance and
implementation
To consider the challenges of conducting research across cultures
Huff, Chapter 10, p201-216
13
Rynes, S.L. (2012). The research-practice gap in I/O Psychology and related fields:
challenges and potential solutions. In S. Kozlowski (Ed). Oxford Handbook of
Industrial and Organizational Psychology.
Bartunek, J.M. & Rynes, S.L. (2010). The construction and contributions of
“implications for practice”: What’s in them and what might they offer? Academy of
Management Learning & Education, 9, 100-118.
Mohrman, S.A. & Lawler, E.E. (2012). Generating Knowledge That Drives Change.
Academy of Management Perspectives. 26, 41-51.
Geletkanycz, M. & Tepper, B.J. 2012. Discussing the Implications. Academy of
Management Journal, 55, 256-260.
Week 15:
PROPOSALS DUE at finals time on DEC 11, 7:45-9:45 AM
Final: Please note that we are REQUIRED to meet during finals week, either for an
exam or regular class meeting. Do not consider this meeting optional, although we’ll
try to make it a fun ending.
14