Randomized Control Trials: What, Why,
How, When, and Where
Mark L. Davison
MESI Conference
March 11, 2016
Department of Educational Psychology
What Is A Randomized Control
Trial (RCT)?
• A randomized control trial is an experiment in
which the people being studied are randomly
assigned to one of several treatment
conditions.
– People can be students, teachers,
administrators, etc.
– Can also randomly assign schools, districts
– Treatment conditions, e.g.
• No treatment vs. treatment
• New treatment vs. business-as- usual
Department of Educational Psychology
Why Do an RCT?
• RCT designs support causal inferences; that is if
there is a difference in outcomes for treatment and
control, RCT supports conclusion that treatment
caused the difference
• As an educator, you need to know what actions
you can take that will CAUSE an improvement in
outcomes
– We need to understand the causal processes affecting
outcomes if we are to do education well
• Not the only design that can support causal
inferences
How to Do An RCT?
• Logic Model (or Theory of Change)
• Sampling Plan
• Measurements
– Independent Variables
– Dependent Variables
– Covariates
– Mediators
– Moderaters
• Treatments, fidelity measures, and fidelity observations
Logic Model
Theory of how intervention has its effects
Include the various variables that you will
include in study as measured variables.
Text to accompany figure that explains how the
variables are related to the change process
How to Do an RCT: Planning
Process?
• Logic Model which includes
– Intermediates steps between the intervention
and the outcome (mediators)
– Factors that may weaken or strengthen the effect
of the intervention (moderators)
– Alternative variables that may explain why the
treatment group may have better outcomes than
the control group (covariates)
RCT Variables
• Independent variable = treatment condition
• Dependent variable = outcome variable
• Covariates = variables to be statistically controlled in the
analysis
– Must be measured in data collection
– E.g. prior achievement in studies of achievement
• Mediator variables = processes through which independent
variables has its effects
• Moderator variables = variables that influence the size of the
treatment effect
Department of Educational Psychology
Logic Model
Summer
School
Attendance
Reading
Mediator
Achievement
IV
Free
Prior
Reduced
Lunch
Reading
Achievement
Moderator
Covariate
Department of Educational Psychology
Random Sampling and the
Independent Variable
•
Random Assignment options (influence analysis strategy)
– Randomly assign student
• When people receive the treatment independently
• Students in same classroom can receive different treatments
Randomly assign by teacher or classroom
When treatment is administered by classroom (all students in same classroom
receive same treatment)
When treatment is by teacher (all students with same teacher receive same
treatment)
Studies of classroom or teacher, policy, or interventions
Randomly assign by school if treatment administered at school level (all students in same
school receive same treatment)
Studies of school policy
Avoids cross-over effects
Generally not feasible for district research (not enough schools)
Stratified Random Sampling
• Simple random sampling controls
confounding variables only within limits of
random sampling
• Stratified random sampling identifies
variables to be controlled, called strata, and
samples in way that controls
– Sex as stratifying variable in reading
• Half males are randomly assigned to treatment and
control; half females are randomly assigned to
treatment and control
Stratified Random Sampling
• Sex and ESL as stratifying variables
– Four groups: boys/ESL, boys/non-ESL,
girls/ESL, girls/non-ESL
– Half of each group is randomly assigned to
treatment and controls
• Stratified random sampling actually controls
stratifying variables, not just within limits of
sampling
Sample Size
• Select sample size so power of detecting
effect is large (e.g. > .8)
• Power is the probability of detecting an
improvement resulting from the treatment if
there is one
• Consult applied statistician and websites
– http://sitemaker.umich.edu/groupbased/optimal_design_software
Dependent Variable
• If students are not the target of the
intervention, do you want a student outcome
variable? For example,
– Teacher intervention designed to improve
reading achievement
• Teacher measures e.g. pedagogical knowledge,
fidelity of implementation in classroom
– Student outcome
• Reading achievement measure, number of books read
independently
Covariates
• Alternative explanations for an effect (if it occurs) and that
were not controlled by stratification
• Used to determine if randomization well-controlled the
variable
• Used to statistically control for alternative explanation
– Test hypothesis of whether there is still a treatment effect
controlling for differences in gender, prior achievement,
etc.
• In studies of achievement, use prior achievement as
covariate; increases power and controls for alternative
explanation
Mediators
• Variables that represent intermediate steps
in the processes leading from the
intervention to the outcome
• School processes, teacher processes,
student processes
• Mediator variables are used to assess
whether the intervention led to the
intermediate processes and whether
intermediate process are related to outcome
Moderators
• Used to test hypotheses about whether the
intervention is equally effective for all
E.g. wonder whether intervention will be equally
effective for boys and girls? Include sex as a
moderator?
Used to test hypotheses about whether
intervention will help close gaps?
E. g. Wonder if intervention will close gaps
between blacks and whites? Include race as a
moderator
Treatments
• Describe each treatment
• Treatment fidelity
– Develop or select a measure of how well treatment is
administered in treatment and control conditions
• Quantitative treatment fidelity index can be used as a
mediator and to see if outcome increases as function
of how well it is implemented
• Observe classrooms (treatment and control) to see what
goes on in each treatment and how different they really are
• Qualitative observations of treatment and control
– RCTs tend to be mixed quantitative and qualitative studies
When RCT?
• RCT canNOT be used if
– Independent variable is not malleable (it is not
under the control of educators)
• E.g. sex of students, race of students
– Manipulation of independent variable by
researchers would be ethically unacceptable or
practically not feasible
• E.g. randomly assigning students to a suspension and
no suspension condition
• E.g. randomly assigning people to high income and
low income conditions
When RCT? Alternatives to RCT
• Regression discontinuity designs
Assignment to treatment is based on a strict cut-off on a
continuous variable
• Single-subject designs, such as ABA designs in which intervention is
turned off and on at designated points to see if outcome changes as the
intervention is turned off and on
– Recognized by special education more than regular education (NCSER, but not
NCER)
– Must distinguish between single subject designs that show effects in one-on-one
settings vs those that show effectiveness in class settings
– Used when treatments have strong, immediate effects
that are reversible
When RCT? As Research
Capstone
• My opinion. RCT’s are best used
– After “bugs” have been worked out of the intervention
– http://www.whale.to/vaccine/bayly.html#THE%20SALK%
20VACCINE%20DISASTER
– After research methods have been piloted in less
rigorous quasi-experimental studies
– After pilot studies (quasi-experimental studies or small
RCTs) have provided some evidence for the intervention
– In short, after a lengthy program of intervention
development and pilot studies
Summary
• What?
– RCT is an experiment in which people are
randomly assigned to treatment groups
• Why?
– RCT permits causal inference, and educators
need to know what actions they can take to
cause an improvement in student outcomes?
Summary
• How
– Based on plan including logic model, sampling plan,
variables (independent, dependent, mediators,
moderators, covariates), treatment description, and
fidelity measures and observations
• When and where
– If feasible and ethically acceptable
– If independent variable is malleable educational factor
– If regression discontinuity or single subject design is less
appropriate
– If research design and treatment implementation have
been well worked out
References
• Booth, J. L., Oyer, M. H., Pare-Blagoev, E. J.,
Elliott, A. J., Barbieri, C., Augustine, A., &
Koedinger, K. R. (2015). Learning algebra by
example in real-world classrooms. Journal of
Research on Educational Effectiveness, 8(4), 530 –
551.
• Borman, G. D. & Dowling, N. M. (2006).
Longitudinal achievement: Effects of multi-year
summer school: Evidence from the Baltimore
randomized field trial. Educational Evaluation and
Policy Analysis, 28, 25 – 48.
References
• Fehr, C. N., Davison, M. L., Graves, M. G., Sales,
G. C. & Seipel, B. E. (2012). The effect of
individualized, online vocabulary instruction on
picture vocabulary scores: An efficacy study.
Computer Assisted Language Learning, 25(1), 87102.
• Food and Nutrition Service, U. S. Department of
Agriculture (February, 2004). Evaluation of the
school breakfast program pilot project.
Washington, D. C.: Author.
What Works Clearinghouse
• ies.ed.gov/ncee/wwc
• Contains evaluations of research evidence
on various interventions
– Evidence for a causal effect of the intervention
Group Work: Research Plan
• Statement of research question
• Logic Model
• Research Plan
– Sampling Plan
– Independent Variable
– Dependent Variable(s)
– Covariates
– Mediator variables
– Moderator variables
– Treatment conditions
– Fidelity measure