EMR 6550:
Experimental and QuasiExperimental Designs
Dr. Chris L. S. Coryn
Kristin A. Hobson
Fall 2013
Agenda
• Quasi-experimental designs that use
both control groups and pretests
• Interrupted time-series designs
• Design and power problems
Designs that Use Both Control
Groups and Pretests
Untreated Control Group
Design with Dependent Pretest
and Posttest Samples
NR
O1
NR
O1
X
O2
O2
• A selection bias is always present,
but the pretest observation allows for
determining the magnitude and
direction of bias
Outcome Pattern 1
Treatment
Control
•
Both groups
grow apart at
different
average rates in
the same
direction
• This pattern is consistent with treatment
effects and can sometimes be causally
interpreted, but it is subject to numerous
threats, especially selection-maturation
Outcome Pattern 2
Treatment
•
Spontaneous
growth only
occurs in the
treatment group
Control
• Not a lot of reliance can be placed on this
pattern as the reasons why spontaneous
growth only occurred in the treatment group
must be explained (e.g., selection-maturation)
Outcome Pattern 3
Treatment
Control
•
Initial pretest
differences
favoring the
treatment group
diminish over
time
• Same internal validity threats as outcome
patterns #1 and #2 except that selectionmaturation threats are less plausible
Outcome Pattern 4
Control
Treatment
•
Initial pretest
differences
favoring the
control group
diminish over
time
• Subject to numerous validity threats (e.g.,
selection-instrumentation, selection-history),
but generally can be causally interpreted
Outcome Pattern 5
Treatment
Control
•
Outcomes that
crossover in the
direction of
relationships
• Most amenable to causal interpretation and
most threats cannot plausibly explain this
pattern
Modeling Selection Bias
• Simple matching and stratifying
– Overt biases with respect to measured
variables/characteristics
• Instrumental variable analysis
– Statistical modeling of covariates believed to
explain selection biases
• Hidden bias analysis
– Difference with respect to unmeasured
variables/characteristics
– Sensitivity analysis (how much hidden bias would
need to be present to explain observed differences)
• Propensity score analysis
– Predicted probabilities of group membership
– Propensities then used for matching or as covariate
Effect-Decay Functions
Delayed Effect
Immediate Effect,
No Decay
Large
Response
Small
Program
Onset
Program
Termination
Response
Small
Program
Onset
Program
Termination
Response
Small
Time
Early Effect, Slow
Decay
Immediate Effect,
Rapid Decay
Large
Large
Time
Program
Onset
Program
Termination
Time
Program
Onset
Program
Termination
Time
Large
Response
Small
Untreated Control Group
Design with Dependent Pretest
and Posttest Samples Using a
Double Pretest
NR
O1
O2
NR
O1
O2
X
O3
O3
• Permits assessment of selection-maturation on
the assumption that the rates between O1 and O2
will continue between O2 and O3
• Testable only on the control group
Untreated Control Group
Design with Dependent Pretest
and Posttest Samples Using
Switching Replications
NR
O1
NR
O1
X
O2
O2
O3
X
O3
• A strong design and only a pattern of historical
changes that mimics the time sequence of the
treatment introductions can serve as an
alternate explanation
• The addition of treatment removal (X) can
strengthen cause-effect claims
Untreated Control Group
Design with Dependent Pretest
and Posttest Samples Using
Reversed Treatment Control
Group
NR
O1
X+
O2
NR
O1
X-
O2
• Interpretation of this design depends on
producing two effects with opposite signs
• Adding a control is useful
• Ethically, often difficult to use a reversed
treatment
Interrupted Time-Series
Designs
Interuppted Time-Series
• A large series of observations made on the
same variable consecutively over time
– Observations can be made on the same units
(e.g., people) or on constantly changing units
(e.g., populations)
• Must know the exact point at which a
treatment or intervention occurred (i.e., the
interruption)
• Interrupted time-series designs are
powerful cause-probing designs when
experimental designs cannot be used and
when a time series is feasible
Types of Effects
• Form of the effect (slope or
intercept)
• Permanence of the effect (continuous
or discontinuous)
• Immediacy of the effect (immediate
or delayed)
Analytic Considerations
• Independence of observations
– (Most) statistical analyses assume observations are
independent (one observation is independent of
another)
– In interrupted time-series, observations are
autocorrelated (related to prior observations or
lags)
– Requires a large number of observations to
estimate autocorrelation
• Seasonality
– Observations that coincide with seasonal patterns
– Seasonality effects must be modeled and removed
from a time-series before assessing treatment
impact
Simple Interrupted TimeSeries Design
O1
O2
O3
O4
O5
X
O6
O7
O8
O9 O10
• The basic interrupted time-series
design requires one treatment group
with many observations before and
after a treatment
Change in Intercept
20
18
Intervention
16
14
12
10
8
6
Change in intercept
4
2
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Change in Slope
35
Intervention
30
25
20
15
10
Change in slope
5
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Weak and Delayed Effects
35
Intervention
30
25
20
15
10
5
Impact begins
0
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41
Validity Threats
• With most interrupted time-series
designs, the major validity threat is
history
– Events that occur at the same time as the
treatment was introduced
• Instrumentation is also often a threat
– Over long time periods, methods of data
collection may change, how variables are
defined and/or measured may change
• Selection is sometimes a threat
– If group membership changes abruptly
Additional Designs
O1
O1
O2
O2
O3
O3
O5
O5
X
O6
O6
OA1 OA2 OA3 OA4 OA5
OB1 OB2 OB3 OB4 OB5
X
X
OA6 OA7 OA8 OA9 OA10
OB6 OB7 OB8 OB9 OB10
O1 O2 O3 O4
O4
O4
X
O7
O7
O8
O8
O9 O10
O9 O10
O5 O6 O7 O8 X O9 O10 O11 O12
• (1) nonequivalent control group, (2)
nonequivalent dependent variable, and
(3) removed treatment
Nonequivalent Control Group
80
Intervention
70
Treatment group
60
50
40
Control group
30
20
10
0
1
2
3
4
5
6
7
8
9
10
11
12
13
Nonequivalent Dependent
Variable
Intervention
35
30
25
Nonequivalent dependent variable
20
Dependent variable
15
10
5
0
1
2
3
4
5
6
7
8
9
10
11
12
13
Removed Treatment
Introduction
30
Removal
25
Treatment period
20
15
10
5
0
1
2
3
4
5
6
7
8
9
10
11
12
13
Design and Power Problems
Problem #1
• A school administrator wants to know
whether students in his district are scoring
better or worse than the national norm of
500 on the SAT
• He decides that a difference of 20-25 points
or more from this normative value would be
important to detect
• He anticipates that the standard deviation
of scores in his district is about 80 points
– Determine the number of students necessary
for power at 95% to detect a difference of 20
and 25 points
– Graph both
– Diagram the design of the study
Problem #2
• Patients suffering from allergies are
nonrandomly assigned to a treatment and
placebo condition and asked to rate their
comfort level on a scale of 0 to 100
• The expected standard deviation is 20 and a
difference of 10-20 is expected (treatment
= 50-60 and placebo = 40)
– Determine the number of patients necessary
for power at 95% to detect a difference of 10
and 20 points
– Graph both
– Diagram the design of the study
Problem #3
• The cure rate for two current
treatments are 10% and 60%,
respectively
• The alternative treatments are
expected to increase the cure rate by
10%
– Determine the number of patients
necessary for power at 95% to detect a
difference of 10% for both scenarios
– Graph both
– Diagram the design of the studies