EPS 780: RESEARCH PRACTICUM ON MIXED METHODS ANALYSIS
Associate Professor Sara Goldrick-Rab
Educational Policy Studies and Sociology
MEETING DETAILS:
Fall and Spring 2011-2012
Class meetings: Wednesdays, 9:15 am-12:00 pm, L150 Education Building
STAFF:
Sara Goldrick-Rab—Email preferred: srab@education.wisc.edu. I can be reached by
phone at 608-265-2141. Office is 211 Education Building. Office hours: Wednesdays,
1:00pm-2:30pm, 211 Education Building
Peter Kinsley—Peter will be the class’s teaching assistant. I have asked him to provide
you with methodological help (e.g., advice on statistics or Dedoose) and insights
regarding WSLS. Peter can be reached at pkinsley@wisc.edu. He will hold weekly labs
on Wednesdays from 12:00pm-1:00pm in Education L177.
Alison Bowman—Alison is the WSLS project manager. You should contact her
regarding any logistics questions (IRB, Dedoose password, etc.). She can be reached at
ambowman@wisc.edu. Her office is in 575 Education Sciences.
DESCRIPTION:
This two-semester course covers the theory and practice of mixed methods research in the
social sciences, with a special emphasis on educational research. It is a practicum in
which each student undertakes a project with linked qualitative and quantitative
components, with the intention of producing a publishable research paper. The course
covers the nuts and bolts of constructing research, at the same time maintaining an
overall emphasis on the specific approach known as “mixed methods” research.
A central advantage for enrolled students is the opportunity to use survey, administrative,
and interview data from the first three years of the Wisconsin Scholars Longitudinal
Study (WSLS), a randomized trial of need-based financial aid (www.finaidstudy.org).
That study, directed by Sara Goldrick-Rab and Douglas N. Harris, is housed at the
University of Wisconsin-Madison. It is the first statewide, longitudinal, mixed-methods
study to examine how and why financial aid affects the undergraduate experience. The
rich data included in the study facilitates the study of numerous topics not related to
financial aid, including: the roles of identity, motivation, and self-efficacy in the
educational attainment process; the process of college transfer; racial/ethnic differences
in the college experience; the formation of social networks during college; the role of
1
institutional practices and actors in student outcomes; and the influence of health and
healthy behaviors on educational attainment. Students are encouraged to think broadly
about the potential uses of the WSLS data and develop research questions related to their
own areas of training and study. They are also permitted to write papers that compare the
WSLS data to other data sources (for example, national datasets).
PREREQUISITES:
All graduate students interested in mixed-methods approaches to data analysis are
welcome. Prior coursework in higher education is not assumed. I recommend that
students have completed at least one course in quantitative methods and in qualitative
methods—but this is not required. The course is appropriate for both new and continuing
graduate students, as there is space provided for students to work at their own pace, and
time built in for instruction in “building block” research skills. The course meets the
“education in context” and “measurement” requirements of the Interdisciplinary Training
Program. It also counts toward the “policy” and “sociology” concentrations in
Educational Policy Studies (and possibly others, according to the preferences of students’
advisers). In the past, students taking the course have made significant progress on a
master’s thesis or dissertation proposal during the year.
REQUIREMENTS:
1. I expect every student to commit to attend and actively participate in every class
session.
This course has limited enrollment and is intended to be a seminar in which the students
and the professor actively engage and interact with one another in a mutually beneficial
way. This only works if all students regularly attend. I ask that you make every effort
not to miss any class—I have already devoted several weeks to your independent work on
papers, allowed time off for the key education conference, and recognized holidays. I also
expect you to do the readings assigned in advance of each class. I reserve the option of
introducing pop quizzes (to constitute 10% of the final grade) if attendance is uneven or
if reading is not done.
I also highly recommend regular attendance at my office hours. Office hours are the
most appropriate time during which to seek advising on your paper, which should be an
ongoing effort throughout the year.
2. I require compliance with the Wisconsin Scholars Longitudinal Study human subjects
and authorship policies.
Through this course you are granted access to a restricted-use dataset that is still being
constructed. This is a rare opportunity that could pay dividends if you are successful at
writing publishable papers.
Access to this data requires that you:
2
(a) Complete human subjects training and sign on to the Institutional Review Board
paperwork for the project (to be done during the initial weeks of class) and
(b) Sign the class authorship agreement (to be introduced during the first week of class).
No one is required to complete either; however access to the data is not allowed until
both are done.
3. I require completion of the course assignments over the two-semester academic year.
All of these are intermediate steps en route to the final assignment.
ASSIGNMENTS
The course assignments follow. Please submit all memos and papers via Learn@UW in
the form of a PDF. I grade using an iPad and return comments to you that way.
MEMO #1: Statement of Interest (DUE 9/28)
This memo is a 1-2 page (double-spaced) description of a student’s chosen WSLS
research topic. Please provide some theoretical and/or policy basis for selecting your
topic, an indication of the literature that will form the backdrop for the research (a short
bibliography is acceptable).
MEMO #2: Study Motivation and Literature Review (DUE 10/26)
This memo (approximately 6-8 pages) should build on Memo #1, clearly stating the
theoretical or policy concern you plan to address (make the case, why is this important?),
describe what we need to know (and currently do not) about the issue, and discuss the
relevant theory and literature. I will return feedback on Memo #2 by 10/30; you should
plan to discuss it in office hours with me during the follow week or two as needed.
MEMO #3: Study Motivation, Literature Review, and Research Questions (DUE 11/16)
This memo (approximately 7-9 pages) should be a revision of Memo #2 and include 2-3
specific research questions that should explicitly motivate a mixed methods analysis.
MEMO #4: Proposed Sample and Methodology (DUE 12/21)
This memo (approximately 5 pages) should outline the sample (from the WSLS and any
other dataset you choose) you want to utilize and the details of your proposed
methodology. Include estimated sample sizes, planned measures for key concepts (e.g.,
draft codes), and an overview of your analytic plan.
3
MEMO #5: Revised Sample and Methodology (DUE 2/8)
Following the feedback I provide as well as the results of the group workshop, revise
Memo #4 and resubmit.
MEMO #6: Preliminary Analysis (DUE 3/7)
Provide a set of tables and/or figures that demonstrates where you think the “gold” is in
your analysis. Write a brief description of what key concerns or questions you are now
facing.
MEMO #7: Draft of Full Paper (DUE 4/18)
Include all sections from prior memos along with a draft set of results and analysis.
PRESENTATION (5/2 and 9)
On a designated day, you will present for 10 minutes a powerpoint describing your study.
FINAL PAPER (DUE 5/16)
The paper should be a full, complete draft that would be acceptable for submission to an
academic conference. It should be no more than 30 pages, excluding tables, references,
and appendices. It should include references that conform to either APA or Chicago
style. Select papers will receive the opportunity to present at the 2012 WSLS conference
and/or be included in an edited volume of WSLS papers.
GRADING:
This is a two-semester course requiring activities and deliverables across both semesters.
Therefore, at the conclusion of semester 1, students will be given a grade of “pass.” At
the end of semester 2, a grade will be assigned that will serve for both semesters (taking
the place of the pass, and assigned to semester 2). If this grading approach presents a
significant challenge, please notify me at the start of the first semester.
The grading rubric is as follows:
Memo 1:
Memos 2-6:
Memo 7:
Presentation:
Final Paper:
5% of total grade, graded on scale of 1-5
10% of total grade each, graded on scale of 1-20 (total=50%)
15% of total grade, graded on scale of 1-100
5% of total grade, graded on scale of 1-10
25% of the total grade, graded on scale of 1-100
4
I will assign letter grades corresponding to the following structure:
A
B+
B
C+
94-100
90-93
84-89
80-83
COURSE MATERIALS:
I strive to keep the costs of this course to a minimum. Except for books (see below) all
readings and homework materials are accessible via the course website at Learn@UW.
I’ve placed them into a single zipped folder for ease of downloading. Feel free to
download them to your laptops and bring laptops to class for note-taking, rather than
printing paper. However, I ask that you TURN OFF your internet connection during
class unless explicitly instructed otherwise. Email or Facebook during class will not be
tolerated.
BOOKS:
The following three books are required. They are available on Amazon, in paperback, for
under $25 each. Please obtain copies quickly.
Duncan, Huston, and Weisner. 2009. Higher Ground: New Hope for the Working Poor
and Their Children. Russell Sage Foundation.
Lieberson, S. 1987. Making it Count: The Improvement of Social Research and Theory.
University of California Press. (Earlier versions are fine)
Yoshikawa, Weisner, and Lowe. 2009. Making It Work: Low-Wage Employment, Family
Life, and Child Development. Russell Sage Foundation.
In addition, I urge those of you who have not taken graduate statistics courses to obtain
the following book and read it early in the course.
Vickers, Andrew. 2010. What is a p-value anyway? 34 Stories to Help You Actually
Understand Statistics. Addison-Wesley.
SOFTWARE:
Dedoose software is used in the class; this is accessible via Web and I am paying all fees
associated with your usage and will also work to train you. You will keep that license
until the end the calendar year, following the term in which the class concludes, to allow
you to revise and publish your paper. Alison Bowman is in charge of initiating your
Dedoose account, and any general questions or problems with Dedoose should be
directed to her. Questions related to analysis should be directed to Peter Kinsley or
myself.
5
All other software packages you need can be accessed via the Social Sciences Computing
Cooperative (SSCC).
DATA:
Depending on whether you have prior experience working with the WSLS, you will
either utilize a redacted version of the WSLS quantitative dataset provided via
MyWebSpace (password and access information to be provided in class on the designated
date) or you will utilize the full dataset through the WSLS website (information provided
separately).
In addition, I will make interview transcripts available to you via MyWebspace
(password and access information to be provided in class on the designated date) so that
you can read interviews in their entirety. However, in addition, interview data is
preloaded into Dedoose for your use when coding.
SPEAKERS
We will have several guest speakers during the year who will meet with us in person or
via the Bascom Hill Telepresence Classroom to share their perspectives as practitioners
of mixed-methods research. It is especially important that you plan to attend class on the
days they will join us. The speakers include:
Regina Deil-Amen, Associate Professor of Higher Education, University of Arizona.
Stefanie DeLuca, Associate Professor of Sociology, Johns Hopkins University.
Eli Lieber, Associate Research Psychologist in the UCLA Department of Psychiatry and
Biobehavioral Sciences and co-director of the Fieldwork and Qualitative Data Research
Laboratory, Center for Culture and Health.
Katherine Magnuson, Associate Professor of Social Work, UW-Madison
Fran Schrag, Professor Emeritus of Educational Policy Studies, UW-Madison
Thomas Weisner, Professor of Anthropology and Psychiatry at UCLA.
6
SCHEDULE OF READINGS AND ASSIGNMENTS
DATE
TOPICS
9/7/11
Course logistics; Overview of
higher education research and
areas for further research
Intro to the Wisconsin Scholars
Longitudinal Study; Discussion
of potential topics for class
papers
9/14/11
9/21/11
Methodology: What it means,
why it matters
9/28/11
Mixed methods research:
Defining terms, considering
examples (part 1)
10/5/11
Mixed methods research:
Defining terms, considering
examples (part 2)
Research questions:
Formulations & hypothesis
generation
10/12/11
10/19/11
10/26/11
Mixed methods research:
Defining terms, considering
examples (part 3)
Designing mixed methods
research
DUE ON THIS
DATE
NOTES
Complete IRB
training; GoldrickRab et al., 2009;
Goldrick-Rab et al.,
2011; listen to
assigned audio files
Leahey, 2008;
Lieberson Ch 1
We will review toplevel reports in
today’s class—
continue reviewing
them afterward
You will have
access to interview
transcripts by
today—begin
reading
MEMO 1 DUE
Johnson,
Onwuegbuzie, &
Turner, 2007;
Kinsley &
Goldrick-Rab 2011;
Figueiredo-Brown
2011; Carter, 2010
Higher Ground
(whole book)
Becker, 1996;
Lieberson Ch 5;
Schrag, 1989;
Small, 2009
Making it Work
(whole book)
11/2/11
Coding and creating scales
MEMO 2 DUE
Creswell & Plano
Clark, 2007 Ch 4;
Gibson & Duncan,
2005; Yoshikawa et
al., 2008
Weisner, 2011
11/9/11
Art and science of interviewing
Hochschild, 2009
Guest Speaker: Fran
Schrag
Guest Speaker: Tom
Weisner
Peter leads class
Access to quant
7
11/16/11
(we will discuss audiofiles in
class)
Statistical inference, selection,
comparisons
11/23/11
11/30/11
HAPPY THANKSGIVING
Dedoose training
12/7/11
Causality and experimentation
12/14/11
Workshop on Memo 3
12/22/11- WINTER BREAK
1/24/12
1/25/12
Workshop on Memo 4 (part 1)
2/1/12
Analyzing mixed methods data
2/8/12
Workshop on Memo 4 (part 2)
2/15/12
Strategies for collecting data for
mixed methods research,
including how to structure
interview protocol, survey
design, how to work with a
team, timeline (community
engagement)
Ethical dilemmas in mixed
methods research
Writing and displaying mixed
methods research
2/22/12
2/29/12
data begins...
Use Vickers book
MEMO 3 DUE
Lieberson (Ch 2,-4, as needed to
6, 7); Murnane
supplement
&Willett, 2011 Ch 3
Read interview
transcripts and visit
Dedoose website to
look around
Howe, 2004;
Lieberson Ch 8-11;
Maxwell, 2004
Read Memo 3 for
members in your
assigned group
MEMO 4 Due
12/21
Guest speaker Eli
Lieber (10-12).
Coding of qual data
begins!
Work on analyzing
data to get ahead!
Read Memo 4 for
members in your
assigned group
Caracelli & Greene,
1993; Teddlie &
Tashakkori, 2009
MEMO 5 DUE
Harris & GoldrickRab, 2012; Teddlie
& Yu, 2007
Leahey, 2007; Fine,
2011
Bryman, 2007;
Mertens, 2011;
Alise & Teddlie,
2010; Dougherty
Workshop by Eli
8
3/7/12
3/14/12
Planning, budgeting, and
fundraising for mixed methods
research
(how to put proposals and how
they are evaluated)
Workshop on Memo 6 (group
1)
MEMO 6 DUE
Creswell et al.,
2011
Read Memo 6 for
members in your
assigned group
Deil-Amen &
DeLuca, 2010;
Deil-Amen &
Goldrick-Rab, 2010
Read Memo 6 for
members in your
assigned group
3/21/12
Conducting mixed methods
research
3/28/12
Workshop on Memo 6 (group
1)
4/4/12
4/11/12
SPRING BREAK
Conducting mixed methods
Weiss, 2008
research; Research meets Policy
4/18/12
4/25/12
NO CLASS-- AERA
Conducting mixed methods
research
PRESENTATIONS
PRESENTATIONS
5/2/12
5/9/12
MEMO 7 DUE
DeLuca &
Rosenblatt, 2010
Guest speaker
Regina Deil-Amen
Guest speaker
Katherine
Magnuson
Guest speaker
Stefanie DeLuca
FULL PAPER
DUE 5/16
READINGS
Alise, M.A., & Teddlie, C. (2010). A continuation of the paradigm wars? Prevalence
rates of methodological approaches across the social/behavioral sciences. Journal of
Mixed Methods Research, 4(2), 103-126.
Becker, H. S. (1996). The epistemology of qualitative research. In R. Jessor, A. Colby, &
R. Schweder (Eds.), Essays on ethnography and human development (pp. 53-71).
Chicago: University of Chicago Press.
Bryman, A. (2007). Barriers to integrating quantitative and qualitative research. Journal
of Mixed Methods Research, 1(1), 8-22.
Caracelli, V. J., & Greene, J. C. (1993). Data analysis strategies for mixed-method
evaluation designs. Educational Evaluation and Policy Analysis, 15(2), 195-207.
Carter, P. L. (2010). A venture into mixed methods: race and cultural flexibility among
students in different multiracial schools. Teachers College Record.
9
Creswell, J. W. et al. (2011). Best practices for mixed methods research in the health
sciences. Bethesda, MA: National Institutes of Health Office of Behavioral and Social
Sciences Research. Retrieved from http://obssr.od.nih.gov/mixed_methods_research
Creswell, J. W., & Plano Clark, V. (2011). Choosing a mixed methods design. Designing
and conducting mixed methods research (2nd ed.). Thousand Oaks, CA: Sage.
Deil-Amen, R., & DeLuca S. (2010). The underserved third: How our educational
structures populate an educational underclass. Journal of Education for Students Placed
at Risk, 15, 27–50.
Deil-Amen, R., & Goldrick-Rab, S. (2010). Institutional transfer and the management of
risk in higher education (WISCAPE Working Paper). Madison, WI: Wisconsin Center
for the Advancement of Postsecondary Education. Retrieved from
http://wiscape.wisc.edu/uploads/media/a4e789cc-7150-4081-854b-6dee33c830ea.pdf
DeLuca, S., & Rosenblatt, P. (2010). Does moving to better neighborhoods lead to better
schooling opportunities? Parental school choice in an experimental housing voucher
program. Teachers College Record, 112(5), 1443-1491.
Figueiredo-Brown, M. R. (2011). Family support of Latino/a college students: How and
why does gender matter? (Unpublished working paper). Madison, WI: University of
Wisconsin.
Fine, C. (2011, July 30). Biased but brilliant. New York Times. Retrieved from
http://www.nytimes.com/2011/07/31/opinion/sunday/biased-but-brilliant-scienceembraces-pigheadedness.html
Gibson, C. M., & Duncan, G. J. (2005). Qualitative/quantitative synergies in a randomassignment program evaluation. In T. Weisner (Ed.), Discovering successful pathways in
children’s development: New methods in the study of childhood and family life (pp.283315). Chicago: University of Chicago Press.
Goldrick-Rab, S., Harris, D. N., & Trostel, P. A. (2009). Why financial aid matters (or
doesn’t) for college success: Toward a new interdisciplinary perspective. Higher
Education: Handbook of Theory and Research, 24, 1-45.
Goldrick-Rab, S. Harris, D. N., Benson, J., & Kelchen, R. (2011). Conditional cash
transfers and college persistence: Evidence from a randomized need-based grant
program (IRP Discussion Paper 1393-11). Madison, WI: Institute for Research on
Poverty. Retrieved from http://www.irp.wisc.edu/publications/dps/pdfs/dp139311.pdf
Harris, D. N., & Goldrick-Rab, S. (2012). Improving the Productivity of Education
Experiments: Lessons from a Randomized Study of Need-Based Financial Aid.
(Unpublished working paper). Madison, WI: University of Wisconsin.
10
Hochschild, J. (2009). Conducting intensive interviews and elite interviews. In M.
Lamont & P. White (Ed.), Workshop on Interdisciplinary Standards for Systematic
Qualitative Research. Washington, DC: National Science Foundation.
Howe, K. R. (2004). A critique of experimentalism. Qualitative inquiry, 10(1), 42-61.
Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a definition of
mixed methods research. Journal of Mixed Methods Research, 1(2), 112-133.
Kinsley, P., & Goldrick-Rab, S. (2011). Weighing the alternatives: Financial constraints,
opportunity costs and early college enrollment trajectories among low-income university
students. (Unpublished working paper). Madison, WI: University of Wisconsin.
Leahey, E. (2007). Convergence and confidentiality? Limits to the implementation of
mixed methodology. Social Science Research, 36(1), 149-158.
Leahey, E. (2008). Methodological memes and mores: Toward a sociology of social
research. Annual Review of Sociology, 34, 33-53.
Maxwell, J. A. (2004). Causal explanation, qualitative research, and scientific inquiry in
education. Educational Researcher, 33(2), 3-11.
Mertens, D. M. (2011). Publishing mixed methods research. Journal of Mixed Methods
Research, 5(1), 3-6.
Murnane, R. J., & Willett, J. B. (2011). Methods matter: Improving causal inference in
educational and social science research. New York: Oxford University Press.
Schrag, F. (1989). Values in educational inquiry. American Journal of Education, 97(2),
171-183.
Small, M. L. (2009). Lost in translation: How not to make qualitative research more
scientific. In M. Lamont & P. White (Ed.), Workshop on Interdisciplinary Standards for
Systematic Qualitative Research. Washington, DC: National Science Foundation.
Teddlie, C., & Tashakkori, A. (2009). The analysis of mixed methods data. Foundations
of mixed methods research: Integrating quantitative and qualitative approaches in the
social and behavioral sciences. Thousand Oaks, CA: Sage.
Teddlie, C., & Yu, F. (2007). Mixed methods sampling: A typology with examples.
Journal of Mixed Methods Research, 1(1), 77-100.
Weisner, T. (2011). New Hope ecocultural family interview coding manual examples. Los
Angeles: University of California, Los Angeles.
11
Weiss, C. et al. (2008). The fairy godmother--and her warts: Making the dream of
evidence-based policy come true. American Journal of Evaluation, 29(1), 29-47.
Yoshikawa, H. et al. (2008). Mixing qualitative and quantitative research in
developmental science: Uses and methodological choices. Developmental Psychology,
44(2), 344-354.
12