Examination of Biased Outcome
Reporting in Educational Research
Terri Pigott & Ryan Williams
Jeff Valentine & Dericka Canada
Loyola University Chicago
University of Louisville
Publication bias in research
• Well-known tendency for studies to be
published in peer-review journals when the
results are statistically significant
• Publication bias can also work within
published studies
– Important details about the study methods are
missing
– Outcomes gathered in the original research are not
reported in the published work
Biased outcome reporting
• Identified problem in clinical trials in
medicine,e.g., Vedula, Bero, Scherer &
Dickersin (2009), NEJM.
– Studies of off-label use of gabapentin
– Compared outcomes described in published reports
with those described in internal research
documents from the industry sponsor
– Many documents for study obtained as a result of
litigation against drug companies
Figure 1. Vedula et al. (2009)
Vedula et al. (2009)
• In the example here, the published reports lead
to a more favorable assessment of gabapentin’s
efficacy for unapproved indications
Implications for Inferences
• If published studies selectively report outcomes of an
intervention, any inferences drawn about the effects
of an intervention will also be biased
• Extent of this problem depends on the relation
between the observed and the unobserved outcomes
– If strong (e.g., two highly correlated measures of a
construct), then problem is likely minimal
– If weak (e.g., two measures of two different constructs)
then problem will lead to biased inferences
• What evidence do we have in education research of
selective outcome reporting?
Our study
• No research registries in educational research
• Instead, we focused on dissertations completed
between 2001 and 2005 at the 96 Carnegie
designated Research Universities/Very High
Research Activity (RU/VH)
• For this presentation, we randomly sampled 26
of the 96 RU/VH institutions
Methods
• Using DAI, we searched for dissertations
completed between 2001 and 2005 at each
sampled RU/VH institution with keyword
“Education”
• Using the abstracts, we identified those
reporting on an intervention for student
outcomes in PreK – 12
• We focused on interventions for ease of
identification of primary outcomes
Methods (continued)
• After identifying the set of dissertations that
studied an intervention with PreK-12 students,
we searched for a published version of the
dissertation primarily using Google Scholar
• In the matched dissertations, we coded each
outcome and its associated p-value.
• In the matched publication, we coded whether
each outcome was reported
Results
• This sample contains 4,102 dissertations
• Of those, 199 (5%) were intervention studies
with PreK-12 student outcomes
• Of these 16 (8%) had an identifiable and
comparable published outcome that met our
inclusion criteria
Results Continued
• Our search thus far has located 16 studies with
209 different treatment outcomes
• Statistically significant outcomes were more
likely to appear in the published version of the
study than were non-statistically significant
outcomes (p = .003)
– OR = 2.34 with CI95% (1.35, 4.14)
– RR = 1.48 with CI95% (1.15, 1.92)
Next Steps
• Next steps
• Complete coding of all 96 university
dissertation sets
• Multilevel analysis to model bias across
studies
• Some evidence of outcome bias in education
research
• What are the practical implications?
Observations about Ed Research
• Few dissertations focus on interventions for
PreK-12 students
– IES reports majority of their applications and
funded studies are Goal 1 and 2 (exploration and
development) and not Goal 3 or 4 (testing an
intervention strategy and scaling up)
– Why are we seeing few intervention studies in
education?