+
Conducting a
Mixed Method
Research
Synthesis
(MMRS)
Dr. Elizabeth G. Creamer, Professor, School of
Education, VA Tech
Dr. Denise R. Simmons, Ass’t Professor, Myers
Lawson School of Construction and
Engineering Education, VA Tech
Rongrong Yu, Doctoral Student, EDRE, School
of Education, VA Tech
CIDER 2015 Conference
February 5, 2015
Blacksburg, VA
Creamer, Simmons & Yu, CIDER2015
+
Overview of Topics Covered
1. Describe types and purposes of systematic reviews (SR) of
the literature.
2. Review stages in the process a SR.
3. Distinguish QUAL, QUANT, and MM reviews
4. Provide an example of a MMRS
5. Summarize the value-added by a mixed method approach.
6. Identify challenges to systematic reviews
7. Suggest avenues for future research.
Creamer, Simmons & Yu, CIDER2015
+
Defining Mixed Methods Research:
Mixing at Any of 4 Stages
QUAL
Research
Question/Desig
n
Data
Sampling or
Analysis
Synthesis or
Visualization
Creamer, Simmons & Yu, CIDER2015
QUANT
MM
+
How Much of the Research Literature in
Education is Mixed Methods?
Discipline
Research Studies
(n=2381)
MM Studies
(n=332 (14%))
Literacy
424
66 (16%)
Mathematics
617
86 (14%)
Science
1182
167 (14%)
Social Studies
158
13 (8%)
Reference: Truscott et al., 2010
Creamer, Simmons & Yu, CIDER2015
+ Systematic Literature Reviews – 4
Design Choices
Type of datanumbers or words
Articles
to
ReviewQUAL,
QUANT,
and/or
Mixed
Creamer, Simmons & Yu, CIDER2015
Type of analytical
procedures-QUAL,
QUANT, or both
Ways to
visualize
– words,
numbers
or both
+
Definition – Research Synthesis

A systematic review or research synthesis is a type of
literature review that uses systematic procedures to synthesizes
the literature in order to pool or aggregate the results or to
evaluate quality features.

Mixed methods research synthesis(MMRS)-review includes
empirical articles using qualitative, quantitative, and mixed
methods

The goal in this type of research is to generalize results
across studies and to produce a synthesis that combines
words and numbers.
Creamer, Simmons & Yu, CIDER2015
+
Different Terms Distinguish QUAL, QUANT,
and MIXED Research Synthesis
Analytical Logic
Descriptors
QUALITATIVE
Research Synthesis, Meta
Narrative, Meta Ethnography,
Case Narrative
QUANTITATIVE
Meta-synthesis
MIXED METHODS
Mixed method research synthesis
Mixed method mixed research
synthesis*
Integrative mixed method review
Creamer, Simmons & Yu, CIDER2015
+
GENERIC STAGES
Sandelowski et al., (2012, p. 320) identified the following generic
steps in conducting a synthetic review.
1.
Define the research purpose and questions.
2.
Set parameters for a systematic literature search, including
criteria for inclusion and exclusion, and conducting the
search.
3.
Identify categories, themes, and/or relationships about results
or outcomes using using either an inductive or deductive
approach.
4.
Summarize result by counting, plotting (as through cluster
analysis), using figures, or a narrative summary.
Creamer, Simmons & Yu, CIDER2015
+
Sampling in Systematic Reviews

Both purposeful (QUAL) and exhaustive (QUANT)

Goal: 20-40 articles for a mixed method review
Creamer, Simmons & Yu, CIDER2015
+
Value Added of a MM Approach to an
Outcome Oriented Review

VERY persuasive foundation for a grant proposal or
publication.

Offers the potential to compare outcomes for different
groups of students and by type of activity.

Can be used to answer how and why questions and to
summarize the literature as a causal model.
Creamer, Simmons & Yu, CIDER2015
STEP 1 of ANALYSIS: Data
Reduction
Kearney, M. H. (2001). Enduring love: A grounded formal theory of women’s
experience with domestic violence. Research in Nursing & Health, 24, 270282.
Creamer, Simmons & Yu, CIDER2015
+
Data Matrix - Another Type of Data
Consolidation
Theme 1
Article 1
XXX
Article 2
Article 3
Theme 2
XX
XXXX
X
Article 4
Creamer, Simmons & Yu, CIDER2015
Theme 3
XX
XX
X
LAST STEP: VISUAL/GRAPHIC DISPLAY – A GT LOGIC DIAGRAM
Whittemore, R. (2005). Analysis of integration in nursing science and
practice. Journal of Nursing Scholarship, 37(3), 261-267.
Creamer, Simmons & Yu, CIDER2015
ANOTHER EXAMPLE OF A VISUAL DISPLAY – A CONCEPT MAP
From: Attride-Stirling, 2001, p. 388
Creamer, Simmons & Yu, CIDER2015
+
CAREER: Investigating Co-Curricular
Participation of Students
Underrepresented in
Engineering, Dr. Denise Simmons, PI
The authors would like to gratefully acknowledge the National Science
Foundation for their support of this work under the CAREER grant #EEC1351156. Any opinions, findings, conclusions, or recommendations
expressed in this presentation are those of the authors and do not
necessarily reflect the views of the National Science Foundation.
Creamer, Simmons & Yu, CIDER2015
+
Background of Project

Specific outcomes result from involvement in “out of class”
experiences

Engineering students are less likely to participate

Few studies investigate


participation decisions of engineering related to these
experiences

various types activities when investigating student outcomes
Develop survey to collect data and investigate gaps

Mixed Method Research Synthesis
Creamer, Simmons & Yu, CIDER2015
+ Research Questions for Co-Curricular
Involvement Project

Who benefits from participation in co-curricular activities?

What factors influence students’ decision to participate or not
participate in the co-curricular activities?

What are the outcomes of participating in the co-curricular
activities?

Does the impact of co-curricular activities vary by types of
activities?

How does co-curricular activities influence students’ learning
outcomes, persistence in their majors, and career intentions?
Creamer, Simmons & Yu, CIDER2015
+
Conceptual Framework for Mixed
Method Research Synthesis
Four steps:

information extraction,

inductive analysis,

cluster analysis, and

analysis of outcomes
Creamer, Simmons & Yu, CIDER2015
+
Steps

Data collection: identified 51 articles with 36 quantitative
studies, 7 qualitative studies, and 8 mixed methods studies.

Information extraction: (1) identified the outcomes related to
co-curricular involvement; and (2) extracted the information
using NVIVO 10.
Creamer, Simmons & Yu, CIDER2015
+
Steps

Data analysis—inductive (qualitative) phase
(1)
generated descriptive codes through a process of open
coding
(2)
organized the initial codes into groups and developed
categories with a higher level of abstraction
Creamer, Simmons & Yu, CIDER2015
+
Data Consolidation & Comparison
(1) Cluster analysis: grouping categories into clusters
by the words similarity
Creamer, Simmons & Yu, CIDER2015
+
Steps
Data
analysis—deductive (quantitative)
phase
(2) Quantizing the results: sort and count the clusters by types of
co-curricular activities
Outcomes
Frequency (Percentage)
Rank
Intellectual development
Personal and social development
Civic development
Interpersonal development
Career and professional development
Satisfaction with college experiences
Academic and social engagement
Belonging and persistence in the major
23 (51.1%)
11(24.4%)
2(4.4%)
6(13.3%)
13(28.9%)
3(6.7%)
10(22.2%)
9(20.0%)
1
3
8
6
2
7
4
5
Creamer, Simmons & Yu, CIDER2015
+
Systematic Reviews: Challenges
1.
There’s no way you can use this method if you are
squeamish about associating numbers with qualitative data
or if you see qualitative and quantitative methods as polar
opposites.
2.
Getting an adequate sample size (20-40) of articles on the
same construct or intervention to conduct statistical
procedures.
3.
Developing creative figures/models/visualizations.
4.
Examples of a mixed methods mixed synthesis (MMMR)
are few and far between.
Creamer, Simmons & Yu, CIDER2015
+
Discussion Questions: Innovative
Approaches to Research Synthesis

What applicability do you see from what we have talked
about to your own research?

How could you do a systematic review AFTER you’ve
completed a traditional literature review chapter for your
chapter?
Creamer, Simmons & Yu, CIDER2015
+
Making Your Mark in Systematic
Reviews
A Systematic Review Squared (MRS-squared)
1.
Develop a grounded theory from a review of the literature
about how and why an outcome like engagement or interest
in STEM is promoted.
2.
Use a QUAL approach to identify the methodological and
graphic characteristics of top-cited articles (ie. More than 50
cites in 5 years) in your field

Or reviews published in the Review of Educational Research

Or the visualizations (e.g. figures and tables) presented in these
Creamer, Simmons & Yu, CIDER2015
+
Relevant References

Attride-Stirling, J. (2001). Thematic networks: An analytical tool for qualitative
research. Qualitative Research, 1(3), 385-405).

Kearney, M. H. (2001). Enduring love: A grounded formal theory of women’s
experience with domestic violence. Research in Nursing & Health, 24, 270-282.

Onwuegbuzie, A. J., & Teddlie, C. (2003). A framework for analyzing data in mixed
methods research. In A. Tashakkori and C. Teddlie (Eds.), Handbook of mixed
methods in social and behavioral research (pp. 351-383). Thousand Oaks,
CA.SAGE.

Sandelowski, Voils, Leeman, & Crandall. (2012). Mapping the mixed methodsmixed research synthesis terrain. Journal of Mixed Methods Research, 6 (4), 3-17221.

Whittemore, R. (2005). Analysis of integration in nursing science and practice.
Journal of Nursing Scholarship, 37(3), 261-267.
Creamer, Simmons & Yu, CIDER2015