Opportunities, Challenges and
Recent Trends in Meta-Analysis:
A Journal Editor’s Perspective
Andreas Schwab, Ph.D.
Associate Professor
Iowa State University
Visiting Senior Fulbright Scholar
Institut Teknologi Bandung
June 24, 2014
.Copyright
© 2014 Andreas Schwab -- All rights reserved.
Personal Background
Institutional Affiliations
 US Senior Fulbright Scholar


Visiting faculty member Institute of Technology Bandung
since September 2013
Associate Professor of Management

College of Business at Iowa State University
Personal Background
Involvement in Methodological Change Initiatives
 Alternatives to Null-Hypothesis Significance Tests




PDWs at Annual Academy of Management Conference
Schwab et al. (2011) Organization Science
Numerous invited presentations
Bayesian Statistics: Potential Advantages and
Applications

PDWs at Annual Academy of Management Conference
Personal Background
Editorial Expertise
 Contributing Editor for Research Methodology and
Statistics at Entrepreneurship, Theory and Practice


(5Y-
ImpF: 2.2)
Member Editorial Boards



Organization Science (5.5)
Strategic Entrepreneurship Journal (3.1)
Journal of Small Business Management (2.1 )
Special Issue Editor


Organization Studies "Temporary Organizations“ (3.2)
IJESB “Sustainable Entrepreneurial Eco-Systems”
Whom else you would like to be
here?
Ray Ragby
Baylor University
Danny Miller
HEC Montreal
Angelo S. DeNisis
Tulane University
Andreas Schwab
Iowa State University
Senior Editorial Board of
Entrepreneurship Theory and Practice
Today's Agenda
Introduction into Meta Analysis (brief)
 History
 Effect Size vs Statistical Significance
 Meta-Analytic Thinking and Scientific Progress
Implications for Academic Journals (speculative)
 Submission Process
 Review Process
 Publication Process
If I get to excited or confusing,
slow me down with a question!
What to expect from today's session
 Motivation to consider meta analysis as a
useful methodology for your own research
 Appreciation of opportunities of meta
analysis for accumulation of knowledge
 Anticipate and contribute to related
appropriate changes in publication process
Brief History of Meta Analysis
Behavioral Sciences
Gene Glass, University of Colorado Boulder
 Presentation to American Educational
Research Association (1976)
 Quantitative methodology to aggregate
findings from multiple prior empirical studies
Brief History of Meta Analysis
Medical Sciences
Thomas Chalmers
, US physician and medical
administrator


Quantitative approaches to
systematically combine findings from
available studies (mid-1970)
Institutionalization of systematic
collection, archiving and analysis of all
controlled clinical trials (Cochrane
Collaboration)
Accumulative Nature of Scientific
Progress
Any single study is not enough …
 Statistical power issues
 Measurement issues
 Unknown boundary conditions
 Dynamic nature of research context
… scientific progress depends on the
accumulation of knowledge across studies.
Traditional “Qualitative” Aggregation of
Prior Empirical Findings
Key Limitations of “Traditional” Review Articles
 Conflicting findings
 Complex differences between studies
 Large number of relevant studies
Current Solution: Focus on a few “high quality”
studies
 No general agreement on “quality” criteria
 Only move from “one study” to “few studies”
Meta Analysis
Quantitative integration of findings in prior
related empirical studies
 Estimates standardized effect size
 Estimates ES credibility intervals, prediction
intervals and confidence intervals
 Options to account for:
 Sampling error
 Measurement errors
 Other artifacts and causal factors
Meta Analysis
Potential Advantages
 Draws on all relevant studies
 Increases statistical power
 Can capture design and empirical context
related moderating effects
 Focus on effect sizes (not stat. significance)
 Focus on credibility, prediction, and
confidence intervals (not p-values)
Type of Effect Size Measure for Meta
Analysis
Effect sizes measures estimate
 A measure of associational strength
 Comparable across studies
 Independent of sample size
 With a computable standard error
Alternative Measures include:
 Bivariate correlation coefficient
 Partial correlation coefficient
 Others
Effect Sizes for Meta Analysis:
Unique vs Standardized
Dialectic tension between alternative
approaches to effect size evaluations
 Adjust to specific empirical research question,
research design and empirical context
versus
 Use similar measures across studies for the
aggregation of findings across studies
Solution: Estimate, interpret, and report both!
Weighting of Effect Sizes
Effect sizes in some studies more precise than
in others
 Alternative indices of precision
 Sample size weight
 Inverse variance weight
 They provide a statistical basis for estimating
 Standard error of the mean effect size
 Confidence intervals
 Homogeneity tests
Comprehensive Meta-Analytic Study:
Alternative Key Models
Several relevant estimates of linear model
 Overall size of an effect (HOMA )
 Mean simple (or partial) correlation coefficient
 Homogeneity of ES estimates to determine need
for moderator investigation
 Probing for publication bias
 Effects of moderators (MARA)
 measurement method, research context,
estimation procedure, model specification
 Entire theoretical models (MASEM analysis)
Probing for Publication Bias
Funnel Graphs
 Scatter diagram of precision (e.g., sample size,
1/SE) vs. estimated effect (e.g., r)
 Estimates vary randomly and symmetrically
around true effect in the absence of
publication bias
 Inverted funnel shape is dictated by
predictable normal distribution of chancebased heteroskedasticity
Funnel Plots: An Example
Publication Bias vs Questionable
Research Practices
Likely upward bias in mean effect sizes and
correlations (Bakker et al., 2012; Levine et al. 2009)
 Publication bias more likely for small sample
studies (lack of power)
 Questionable research practices prevalent and
serious threat in management research
(Simmons et al. 2011)
 Statistical significance of <5% offers only
weak false-positives protection
Starbuck, 1988).
(Webster &
Recommended literature
Cumming, Geoff (2011):
Understanding the new
statistics: Effect sizes,
confidence intervals, and
meta-analysis.
Routledge, New York.
(Good fundamental
introduction)
Meta Analysis Literature:
Books
 Cooper & Hedges (2009), The handbook of
research synthesis and meta-analysis. (Great
book; combination of very relevant papers)
 Lipsey & Wilson (2001), Practical metaanalysis. (Good introductory text)
 Stanley & Doucouliagos (2012), Meta-
regression in economics and business. (Most
advanced)
Meta Analysis Literature:
Journal Articles





Aguinis, Pierce, Bosco, Dalton and Dalton (2010) Debunking
myths and urban legends about meta-analysis. Organizational
Research Methods, 14, 306-331.
Cortina (2003), Apples and oranges (and pears, Oh my!): The
search for moderators in meta-analysis, Organizational Research
Methods, 6, 415-439.
Geyskens, Krishnan, Steenkamp and Cunha (2009) A review and
evaluation of meta-analysis practices in management research.
Journal of Management, 35, 393-419.
Stanley (2001), Wheat from chaff: Meta-analysis as quantitative
literature review, Journal of Economic Perspectives, 15, 131150.
Stanley and Jarrell (2005), Meta-regression analysis: A
quantitative method of literature surveys, Journal of Economic
Surveys, 19, 299-308.
Meta Analysis: Key Challenges
Current limitations of Meta Analysis applications

Key information not reported in prior empirical studies

Key design and context differences not
reported/recognized in prior empirical studies
Focus on single highly standardized effect size
measures across studies (e.g., bivariate correlations)
Need to further improve methods and methods
understanding


(requiring often problematic conversions)
Increasing Number of Meta Analyses in
Entrepreneurship Research
Figure 1
Number of Meta-Analytic Studies in
Leading Entrepreneurship and Management Journals (1990-2013)
18
Management
Journals
16
14
12
10
8
6
Entrepreneurship
Journals
4
2
0
1990
1992
1994
1996
1998
2000
2002
Years
2004
2006
2008
2010
2012
Meta-Analytic Thinking
and Scientific Progress
Promising approach to theory development

Move away from “single study” theory test
approaches

Embrace iterative development and adjustment of
models and theory
… and goes far beyond the concept of “replication
studies”

Meta-Analysis and Replication Studies
Replication studies are not enough! (Schmidt 2014)

Exact future replications only probes for sampling
error effects

Future studies that differ in a few key features can
probe for other moderating effects (e.g.,
measurement error, boundary conditions)

Accumulation of knowledge in pool of studies
Reasonably heterogeneous > homogeneous
Opportunities, Challenges and
Recent Trends in Meta-Analysis
How can academic journals
help?
One
Journal Editor’s Perspective
My Personal Convictions (Biases)
Combination of single study empirical theory tests using
statistical significance is detrimental to scientific
progress
 Inherent power problems
 Statistical significance flawed indicator for importance
of effects for theory or practice
Meta Analysis has potential to support meaningful
iterative model and theory development across studies
 Improves upon straight replications
 Improves upon conceptual review articles
Meta Analysis Opportunities and
Publication Process
Potential Implication for Academic Publication
Process
 Stronger focus on meta-analysis compared to
traditional review articles
 Provide necessary information for future meta
analyses
Proposal to Support Meta Analyses in
Entrepreneurship Journals
Changes in all stages of academic publication
process




Pre-Submission
Review
Copy Editing and Distribution
After-Print
Review-Oriented Journals
Review Journals should (primarily) publish
meta analyses
 Encourage collaborations among
phenomenon experts and meta-analysis
experts
 Develop editorial board with metaanalytical expertise
Journals Publishing
Original Empirical Studies
Journals should collect the necessary information for
future meta analyses
 Effect size
 Credibility, Prediction and Confidence intervals
 Relevant study design features


Measurement, sampling process, data coding
Empirical context features
Solution
 Adjust submission guidelines (= new requirements)
 Reinforce requirements during review and copyediting stage
Journals Publishing
Original Empirical Studies
Archive and distribute MA-relevant information for future
meta analyses using
 Printed article



Text
Appendix
Online storage and retrieval systems



Systematic
Detailed
Open-access
Solution: Online journal archives!
Publication Bias:
Journal Archives are not enough
Need for institutionalized collection, archiving
and dissemination of prior empirical studies
similar to Cochrane Collaboration
 Government (NSF)
 Professional associations (AOM)
 Research foundations (Kauffmann)
Publishing in Top Entrepreneurship Journals:
Opportunities, Challenges and Recent Trends
Questions and Thoughts
.
Opportunities, Challenges and
Recent Trends in Meta-Analysis:
A Journal Editor’s Perspective
Further
Thoughts and Ideas?
.
Opportunities, Challenges and
Recent Trends in Meta-Analysis:
A Journal Editor’s Perspective
Thank You!
.
Opportunities, Challenges and
Recent Trends in Meta-Analysis:
A Journal Editor’s Perspective
Andreas Schwab, Ph.D.
Associate Professor
Iowa State University
Visiting Senior Fulbright Scholar
Institut Teknologi Bandung
June 24, 2014
.
Entrepreneurship, Theory and
Practice
Aims and Scope of ETP
 Leading ENT scholarly journal
(others JBV, ETP, SEJ)
 Original conceptual and empirical
papers that advance the field of
entrepreneurship
 Official journal of "United States
Association for Small Business and
Entrepreneurship" (USASBE)
Entrepreneurship, Theory and
Practice
Review Process Structure
 Single action editor
 'Double-blind' review process
 Three independent reviewers
 Three-month review cycle
 Rarely (= never) straight acceptance
 1 - 3 rounds of respectfully worded and constructive
criticism
 Acceptance rate < 10%