Regression
Discontinuity Design
Thanks to Sandi Cleveland and Marc Shure (class of 2011) for some of these slides
RD Designs
 A pretest-posttest, program-comparison group
strategy
 Review:
 Advantages of Pre-tests?
 Detect differences between groups
 Detect potential vulnerability to internal validity threats
 Helps with statistical analysis
 Advantages of Comparison groups?
 Helps control sources of error
 Helps support the counterfactual inference
Underuse of RD? Why?
 It’s new.
 Key criteria must be met for use.
 Perhaps it’s just misunderstood.
Overview of RD
OA
C
OA
C
X
O2
O2
 “pre” is ANY continuous variable that correlates with the
outcome of interest
 Assignment based on cutoff score
 Regression line should have vertical displacement at the
cutoff score if there is an effect
No Treatment Effect
Positive Effect
Examples
1. Campell & Stanley’s Ivy League Education Example
2. Trochim’s Hospital Administration Example
Hospital Quality of Care
More about assignment
 Assignment variables:
 Must be continuous (or ordinal)
 Can be a pretest on a dependent
variable
 Can be by order of entry into study
 Cannot be caused by treatment
 May or may not be related to the
outcome
If implementing an RD design in your
area of research, what variables would
you choose for assignment?
Choosing the Cutoff Score
 Now referring to the assignment variable(s) you
identified, how would you arrive at a cutoff score?
 Substantive grounds: professional judgment
 Need or Merit
 Clinical diagnosis
 Practical grounds:
 Available data sets
 Available resources
Choosing the Cutoff Score
 Mean of the distribution of assignment scores
 Politically defined thresh-holds
 Composite scores of assignment variables
Important: Assignment must be controlled! (It is as
important as proper random assignment.)
Additional Considerations
 Functional form relating the assignment and outcome
variables
 A defined population in which it is possible for all units
in the study to receive Tx regardless of the choice of a
certain cutoff point
 Intent-to-Treat? : Tx diffusion and cross-over
participants
Variations
1. Compare 2 treatment groups
2. Compare 3 conditions
3. Different dose treatment groups
4. Multiple cutoff points
5. …and many more creative ways to think of
Theory of RD – How does this
work?
 RDs as Treatment Effects in REs
 RE pretest means of Tx and Control groups nearly identical at
would be the cutoff score in an RD design through random
assignment
 Cutoff-based assignments creates groups with different pretest
means and non-overlapping pretest distributions
 RD compares regression lines, not means
 Both RDs and REs control for selection bias
 Unknown variables do not determine assignment
 Pretests have no error IF used as the selection variable
 Regression lines are not affected by posttest errors
Adherence to the Cutoff
 Overrides of the cutoff
 Crossovers
 Attrition
 “Fuzzy” regression discontinuity
Threats to Validity
 Statistical Conclusion Validity
 Nonlinearity
 Interactions
 Internal Validity – must occur at the cutoff point




History
Maturation
Mortality
Selection-instrumentation
Interaction
Group Exercise: RD Design
Analytical Assumptions
 No exceptions to the cutoff
 Adhere to true function of the pre-post relationship
 Uniform delivery of pretest and program
Combining RD with
Randomized Experiments
7 combo examples
3 advantages:
 Increased power
 Allows estimation of both
groups at the overlap interval
 Adds clarity to the cutoff
point
RD – Quasi-experiment?
 shortfalls are not yet clear
 Requires more “demanding statistical analysis”
 Less statistical power
 see table 7.2 in SCC (pg. 243)
Analysis Problems
 The Curvilinear Problem
Steps to Analysis
 Transform the pretest
 Examine the relationship visually
 Specify high order terms and interactions
 Estimate the initial model
 Refine
Comparing RD Designs with
Experimental Designs
 IN theory, both designs should produce similar results
when all exemplary conditions of each method type exist
 Question remains do they produce similar results and
standard errors in practice (real world settings)?
 Under exemplary conditions, experiments are 2.75 times
more efficient than RDDs
 If otherwise, the degree of this efficiency will vary
 Central Question: How to compare these two design
options in field settings?
Cook, Shadish & Wong 2008
Statistical Power for GRT and
RDD
 the RDD has approximately 36 per cent the efficiency
of the GRT.
 This implies that the RDD will require approximately
2.75 times more groups than a GRT with the same
power.
 The same result was found by Schochet [16] for
hierarchical models in education
 and by Cappelleri and Trochim [31] for trials targeting
individuals rather than groups.
Pennell et al., 2010
Within Study Comparisons:
 Proposed methodology from LaLonde
 Causal estimates derived from an experiment
compared with estimates from a non-experiment
 Same Tx Group
 Different Control Group
 Modifications needed to use for RDDs
7 Criteria to Improve Interpretation of
Within-Study Comparisons
1. Must demonstrate variation in types of
methods being contrasted
2. Both assignment mechanisms cannot be
correlated with other factors related to
outcome variables
3. The RE must “deserve” its status of the
causal “Gold Standard”
4. The non-experiment design must also be
good
7 Criteria to Improve Interpretation of
Within-Study Comparisons
5. Both study types should estimate the
same causal quantity
6. Explicit criteria must be raised on how the
two design estimates relate to each other
7. Blind that data analyst!
Further Discussion?
Nagging Questions?
…or
Inspirations?