Education Research 101
A Beginner’s Guide for S STEM Principal Investigators
KATEWINTEREVALUATION.com
Welcome and Overview
• Structure of today’s webinar
• About 30 minutes of content and time for Q&A
• Quick overview of educational research
• Brief synopsis of the Common Guidelines
• Pulse checks (interaction points) to gauge engagement
• About me
KATEWINTEREVALUATION.com
Objectives
• Increase awareness of the “Common Guidelines for
Education Research and Development,” including:
• The 3 broad categories of research
• The 6 general types of research, with their:
•
•
•
•
Purpose
(Very) Abbreviated theoretical and empirical justifications
Abbreviated guidelines for evidence produced
Guidelines for external review and feedback
KATEWINTEREVALUATION.com
What is Education Research?
• Research that uses various methodologies to address
educational processes and outcomes
• Research that provides the evidence behind “evidencebased” practices/strategies/interventions in education
• Increasingly commonly by following established guidelines for rigor
and significance set forth by IES and NSF, including the What
Works Clearinghouse (WWC) standards
KATEWINTEREVALUATION.com
Interaction #1
What was your background with the Common Guidelines
for Education Research and Development
PRIOR to this webinar?
KATEWINTEREVALUATION.com
Disclosure
• In *many* places, this presentation draws text verbatim
from the Common Guidelines. When you have time, I highly
encourage you to find, bookmark, and review the full
document from IES and NSF.
http://ies.ed.gov/pdf/CommonGuidelines.pdf
KATEWINTEREVALUATION.com
Assumptions behind the Common Guidelines
• While the 6 research types generally follow a sequence of
development:
• Knowledge development is not linear and can be informed from
various places along the continuum
• Investigations can skip “phases” (MOOCs make large-scale testing
possible without prior small-scale testing)
• Single studies may include aspects of more than one research type
KATEWINTEREVALUATION.com
Foundational Research & Early-Stage or Exploratory Research
• Contribute to Core Knowledge in education, including basic
understanding of teaching and learning (i.e., cognition);
components and processes involved in learning and
instruction; the operation of education systems; and, models
of systems and processes
• Include:
• Research Type #1: Foundational Research
• Research Type #2: Early-Stage or Exploratory Research
KATEWINTEREVALUATION.com
Design and Development Research
• Develops solutions to achieve a goal related to education or
learning, such as improving student engagement or mastery
of a set of skills
• Represents Research Type #3
KATEWINTEREVALUATION.com
Impact Studies
• Contribute to evidence of impact, generating reliable
estimates of the ability of a fully-developed intervention or
strategy to achieve its intended outcomes
• Includes:
• Research Type #4: Efficacy Research
• Research Type #5: Effectiveness Research
• Research Type #6: Scale-Up Research
KATEWINTEREVALUATION.com
Interaction #2
Which statements are TRUE?
KATEWINTEREVALUATION.com
Research Type #1: Foundational Research
• Provides the fundamental knowledge that may contribute to
improved learning and other relevant education outcomes
• Seeks to test, develop, or refine theories of teaching or
learning and may develop innovations in methodologies
and/or technologies that will influence and inform research
and development in different contexts
KATEWINTEREVALUATION.com
Foundational Research (cont.)
• Purpose: To advance the frontiers of education and learning;
develop and refine theory and methodology; and provide
fundamental knowledge about teaching and/or learning
• Justifications: Should address important research problems
or questions related to education and learning; should
outline the theoretical and empirical bases for the project
and why it is necessary for core knowledge generation
KATEWINTEREVALUATION.com
Foundational Research (cont.)
• Evidence to be produced:
• Advances in theory, methodology, and/or understanding of
important constructs in education and learning
• Findings that could serve as the basis for future studies

May examine phenomenon without establishing explicit links to
education outcomes
KATEWINTEREVALUATION.com
Research Type #2: Early-Stage or Exploratory Research
• Examines relationships among important constructs in
education and learning to establish logical connections that
may form the basis for future interventions or strategies to
improve education outcomes
• Connections are usually correlational rather than causal
KATEWINTEREVALUATION.com
Early-Stage or Exploratory Research (cont.)
• Purpose: To investigate approaches to education problems to
establish the basis for design and development of new
interventions or strategies, and/or to provide evidence for
whether an established intervention or strategy is ready to
be tested in an efficacy study
• Justifications: Should address a practical education problem
and provide a compelling case for how the study will inform
the development, improvement, or evaluation of education
programs, policies, or practices
KATEWINTEREVALUATION.com
Early-Stage or Exploratory Research (cont.)
• Evidence to be produced:
• Empirical evidence regarding associations between malleable
factors and student learning outcomes
• A well-specified framework behind these associations
• Justification for proceeding to a Design and Development project,
an Efficacy study, or if additional foundational or early-stage or
exploratory research is needed first
KATEWINTEREVALUATION.com
Interaction #3
Who has engaged in a study or project that would be
considered foundational research or early-stage or
exploratory research?
KATEWINTEREVALUATION.com
Research Type #3: Design and Development Research
• Draws on existing theory and evidence to design and
iteratively develop interventions or strategies (solutions)
• May include pilot tests of fully developed interventions to
determine if they achieve their intended outcomes under
various conditions
• Results could lead to additional work to better understand
the foundational theory behind the results or could indicate
that the intervention or strategy is sufficiently promising to
warrant more advanced testing
KATEWINTEREVALUATION.com
Design and Development Research (cont.)
• Purpose: To develop new or improved interventions or
strategies to achieve well-specified learning goals or
objectives, including making refinements on the basis of
small-scale testing. Typically entails four components
• Justifications: Explain why the proposed project has the
potential to improve learning or education outcomes or
increase efficiencies in the education system or institutional
setting beyond what current practice provides
KATEWINTEREVALUATION.com
Design and Development Research (cont.)
• Evidence to Be Produced:
• Fully developed version of the proposed design/research
• A well-specified theory of action
• Descriptions of the major iterations and the resulting evidence to
support or question key assumptions about the theory of action
• Description and empirical evidence of the adjustments that
resulted from design testing
• Measures with evidence of technical quality for assessing the
intervention and data demonstrating the project’s success
• Pilot data on the intervention’s promise for generating the
intended beneficial learner outcomes
KATEWINTEREVALUATION.com
Interaction #3
Who has engaged in a study or project that would be
considered design and development research?
KATEWINTEREVALUATION.com
Research Type #4: Efficacy Research
• Allows for testing of a strategy or intervention under “ideal”
circumstances
• May limit the investigation to a single population
KATEWINTEREVALUATION.com
Efficacy Research (cont.)
• Purpose: To determine whether an intervention or strategy
can improve outcomes under “ideal” conditions
• Justifications: Must compelling explain why testing the
impact under “ideal” circumstances is warranted, with
evidence from prior studies/implementation that supports
the value of the intervention and the need for additional
testing
KATEWINTEREVALUATION.com
Efficacy Research (cont.)
• Evidence to Be Produced (same for all Impact Studies):
• Final report as outlined by the What Works Clearinghouse (WWC)
• Reliable estimates of the intervention’s average impact, with
estimates by sample subgroups as appropriate
• Implementation details for the intervention and counterfactual
condition, with connections to outcomes as appropriate
• Implications of the findings for theory of action:
• Replication details for projects demonstrating favorable impacts
• Otherwise, possible reasons why no favorable impact was demonstrated
KATEWINTEREVALUATION.com
Research Type #5: Effectiveness Research
• Examines the effectiveness of a strategy or intervention
under circumstances that would typically occur
KATEWINTEREVALUATION.com
Effectiveness Research (cont.)
• Purpose: To estimate the impacts of an intervention or
strategy when implemented under conditions of routine
practice
• Justifications: Must compelling explain why testing the
impact under routine circumstances is warranted, with
evidence from prior studies/implementation that supports
the value of the intervention and the need for additional
testing
KATEWINTEREVALUATION.com
Research Type #6: Scale-Up Research
• Examines effectiveness in a wide range of populations,
contexts, and circumstances, without substantial developer
involvement in implementation or evaluation (i.e., under
“normal” conditions)
KATEWINTEREVALUATION.com
Scale-Up Research (cont.)
• Purpose: To estimate the impacts of an intervention or
strategy when implemented under conditions of routine
practice AND across a broad spectrum of populations and
settings
• Justifications: Must compelling explain why testing the
impact under routine circumstances and with a broad
sample is warranted, with evidence from prior
studies/implementation that supports the value of the
intervention and the need for additional testing
KATEWINTEREVALUATION.com
Guidelines for External Review and Feedback
• All research types should be subject to a series of external,
critical reviews of design, implementation, analysis and
reporting
• External reviews should be independent and rigorous
• Appropriate review activities include:
• Peer review of proposed projects
• Ongoing monitoring and review by grant-making personnel
• External review panels or advisory boards
• Third-party evaluator
• Peer review of publications and/or conference presentations
KATEWINTEREVALUATION.com
Interaction #4
Who has engaged in a study or project that would be
considered an impact study?
KATEWINTEREVALUATION.com
Recap
• Three categories:
1. Foundational research and early-stage or exploratory research
2. Design and development research
3. Impact studies:
• Efficacy research
• Effectiveness research
• Scale-up research
• OK to mix different approaches in a single project, if it makes
sense to do so
KATEWINTEREVALUATION.com
Questions or Comments?
Raise your hand to be un-muted, or
Type your question in the question box
Please let us know if you would be interested in additional webinars on the specific
research types, including aspects related to design plans and WWC standards
KATEWINTEREVALUATION.com