Quasi-Experimental Designs
Manipulated Treatment Variable
but
Groups Not Equated
Pretest-Posttest Nonequivalent
Groups Design
N O X O
N O
O
• Cannot assume that the populations are
equivalent prior to treatment.
• Selection and Selection Interactions are
threats to internal validity.
• Can try to select subjects or intact groups
in ways that make it likely that the groups
are similar, but what about unknown
variables on which the groups may differ.
Double-Pretest Nonequivalent
Groups Design
N
N
O
O
O
O
X
O
O
• Some control for Selection x Maturation.
• If groups are maturing at different rates,
that may be shown in the two pretests.
Regression-Discontinuity Design
C
C
O
O
X
O
O
• ‘C’ indicates subjects are assigned to
groups based on score on covariate.
• Groups are deliberately nonequivalent.
• I shall illustrate with a hypothetical
example
Evaluating an Online Tutorial
• IV = Student completes online tutorial or
not.
• DV = Student’s score on statistics course
exam.
• Pretest/Covariate = Student’s score on a
test of statistics aptitude.
• How do I assign students to groups?
Evaluating an Online Tutorial
•
•
•
•
Let the students self-select into groups.
This gives me a pretest-posttest
nonequivalent groups design.
I use pretest scores as covariate in
ANCOV.
This does not, however, remove all
possible confounds.
How might the groups have differed other
than on statistics aptitude???
Evaluating an Online Tutorial
Randomly assign students to groups.
• This would be an experimentally sound,
randomized pretest-posttest control group
design.
• And you would live to regret trying it.
– Students complain.
– Their parents complain.
– The Chair of the Department intervenes.
– The IRB revokes your authority to do
research.
Evaluating an Online Tutorial
Try a switching-replications design.
• Those who have to wait until the second
half of the class would be disadvantaged
– if you don’t learn the beginning material well,
the later material will very hard to learn.
• Those who have it taken away at midsemester will complain.
Evaluating an Online Tutorial
Apply the treatment only to those most
in need of it, those lowest in statistics
aptitude.
• Those not selected may complain that
they could benefit from it too.
– Tough, there is an American tradition of
favoring the underdog.
Evaluating an Online Tutorial
• Those selected may complain about
having to do extra work.
– You can’t please everybody every time.
Convince them they need to do it.
• May be cases when you want to give the
tutorial only to those with highest aptitude
– purpose of tutorial is to allow brightest
students to finish course early, allowing prof
more time to spend in class with others.
Evaluating an Online Tutorial
• Suppose I pick a cutoff on the covariate
(stats aptitude) so that lowers ½ get the
treatment, upper ½ don’t.
• Simulated data are in file RegD0.txt .
• C = control group, T = Treatment group.
• 2nd score is posttest score.
• 3rd score is pretest score.
• I defined the treatment effect to be zero in
the population.
Evaluating an Online Tutorial:
Population Treatment = 0
• In the population,
2

Post = 7 + 1.35Pre + error,  = .9, error  1.71
• In the sample, ignoring group,
Post = 7.58 + 1.27Pre + error, r = .85, and
MSE = 2.13
• Look at this plot of the data:
Ignoring Group
Evaluating an Online Tutorial
• Now I compute two separate regressions,
one for each group.
• T: Post = 8.09 + 1.17Pre + error, r = .62,
and MSE = 2.13.
• C: Post = 6.33 + 1.43Pre + error, r = .72,
and MSE = 2.29.
• The plot shows how little the two lines differ:
Separate Regressions, by Group, No Treatment Effect in Population
Evaluating an Online Tutorial
• I re-simulated the data, with a 3 point
treatment effect built in.
• The data are at RegD1.txt.
• T: Post = 11.27 + 1.07Pre + error, r =
.82, and MSE = 1.35.
• C: Post = 7.90 + 1.18Pre + error, r = .82,
and MSE = 1.25
• The plot shows a clear regression
discontinuity:
Separate Regressions, by Group, 3 Point Treatment Effect
Evaluating an Online Tutorial
• The dotted line shows the expected
regression for the treatment group if there
were no treatment effect.
• Hard to imagine how any threat to internal
validity would create the observed
regression discontinuity.
• Caution: This analysis assumes the
regression is linear, not curvilinear.
Proxy-Pretest Design
•
•
•
•
N O1 X O2
N O1
O2
You have a nonequivalent groups posttest
only control group design.
The treatment has already been
administered.
Now you decide you want a pretest too.
Can’t warp time, can find an archival proxy
pretest.
PSYC 2210 and Understanding Stats
• Mid-semester, I ask myself “does taking
2210 improve students understanding of
stats?”
• I’ll compare students in current 2210 class
with those in another class (excluding any
who have already taken 2210).
• I want a pretest too, but the treatment is
already in progress.
PSYC 2210 and Understanding Stats
• I use, as a proxy pretest, students’ final
averages from PSYC 2101.
• Conduct an ANCOV
– IV = took 2210 or not
– DV = end of course stats achievement test
– COV = the proxy pretest
Separate Pre-Post Samples
Design
• Pretest subjects different than
posttest subjects.
• I want to evaluate online
tutorial in stats.
• Both I and my friend Alex
taught stats this semester and
last semester.
• Both of us gave our students a
end of course standardized
exam.
N
N
N
N
O
X
O
O
O
Separate Pre-Post Samples
Design
• Row 1: My students last
semester, no tutorial.
• Row 2: My students this
semester, with tutorial.
• Row 3: Alex’s students last
semester, no tutorial
• Row 4: Alex’s students this
semester, no tutorial.
• Selection problems likely.
N
N
N
N
O
X
O
O
O
Nonequivalent Groups Switching
Replications Design
N
N
O
O
X
O
O
X
O
O
• I am teaching two sections of stats.
• I make the experimental tutorial available
the first half semester to one class
• and the second half semester to the other.
• Might reduce complaints, until students
from the two classes meet each other.
Nonequivalent Dependent
Variables Design
N
•
•
•
•
•
O1
O2
X
O1
O2
Only one group of subjects, but two DVs.
One DV you expect to be affected by X.
The other you expect not to be affected by X.
The second DV serves as a control variable.
Should be similar enough to 1st DV that it will be
affected in same way by history, maturation, etc.
Nonequivalent Dependent
Variables Design
• I want to evaluate effect of stats remedial
tutorial given to all PSYC 2210 students.
• DV1 = stats knowledge measured at start
and end of semester.
• DV2 = Vocabulary test given at start and
end of semester.
• More impressive if have multiple control
variables and an a priori prediction of
extent to which each will be affected.
Nonequivalent Dependent
Variables Design
•
•
•
•
•
•
Stats Knowledge (DV1) – most affected
Logical Thinking – next most
Verbal Reasoning – same as LT
Arithmetic Skills – next most
Vocabulary – next most
Artistic Expression – least affected by
treatment
Regression Point Displacement
Design
N(n
N
•
•
•
•
= 1)
O
O
X
O
O
Only one subject in the treatment group
Several or many in the control group.
X = novel economic development plan.
Treatment unit = your hometown, in which
the plan was just initiated.
Regression Point Displacement
Design
• You consult state economic database.
• Pick 20 cities comparable to your city,
these serve as the control group.
• Pretest = value of criterion variable (such
as unemployment rate) last year.
• Posttest = value of same variable two
years later.
Regression Point Displacement
Design
• Plot Post x Pre for the Control Group.
• Draw in regression for predicting Post from
Pre.
• Plot experimental unit data point.
• If it is displaced well away from regression
line, you have evidence of a treatment
effect.