Quasi-Experimental Designs
• Whenever it is not possible to establish
cause-and-effect relations because there
is not complete control over the
variables in the study
Common Quasi-Experiments
• The two most common quasi-experimental
designs include
– Nonequivalent control group design
– Interrupted time series designs
• Both of these are very common in program
evaluation research
Nonequivalent Control Group Design
• Design
– Experimental Group
• Pre-test --> Treatment --> Post-test
– Control Group
• Pre-test --> Nothing --> Post-test
• The distinguishing aspect of this design is that the
experimental and control groups are not equivalent
at the beginning of the experiment
Example
• Testing the usefulness of a new technique for
teaching research methods
• Design
– Experimental Group (Texas Tech students)
• Pre-test (Final from Fall) --> Treatment (using the new technique
Spring) --> Post-test (Final from Spring)
– Control Group (Minnesota students)
• Pre-test (Final from Fall) --> Nothing (i.e., they keep teaching via
their same technique) --> Post-test (Final from Spring)
Example
• Because of the non-equivalence of the groups
at the beginning of the study, the researcher
needs to be especially concerned about the
various threats to internal validity.
Example
• History
– If something significant happens during Spring ‘05
at Tech (e.g., abnormal amounts of bad weather,
unrest on campus, etc.), then this could affect
test scores and effect the outcome of the study
• All you can do is record these kinds of events and try to
defend that nothing like this happened during your
study
Example
• Selection
– The experimental and control groups each contain
people from different schools. If students at one
school were more capable than students at the
other, then this could create post-test differences
• To deal with this, you look not just at post-test scores,
but the improvement from pre-test to post-test for both
groups
Cook & Campbell (1979)
• When doing a non-equivalent control group
design, pay special attention to the following
threats to internal validity
– Maturation
– Instrumentation
– Regression to the Mean
– Selection/History
Interrupted Time Series Design
• Design
– Experimental Group
• Pre-test --> Pre-test --> etc. --> Treatment --> Post-test --> Posttest --> etc.
• The distinguishing aspects of this design are
– there is no control group
– measurements are taken for an extended period of time
before and after the treatment
Example
• Testing the usefulness of a new technique for
teaching research methods
• Design
– Experimental Group (Texas Tech students)
• Pre-test (Final from Fall) --> Pre-test (Final from Spring) --> Pretest (Final from Fall)
• Treatment (using the new technique Spring)
• Post-test (Final from Spring) --> Post-test (Final from Fall) -->
Post-test (Final from Spring)
Example
• Because of the lack of a control group, the
researcher needs to be especially concerned
about the various threats to internal validity
Example
• History/Selection
– If Tech changes its entrance requirements
around the time of the implementation of
the new technique, then this might account
for any improvement in test scores
• To deal with this, it would be best to examine
grades in other unrelated courses (which
essentially would serve like a control group)
Example
• Instrumentation
– If the new teaching technique is associated
with new tests, then the pre-test/post-test
measures for the experimental group would
be different.
• Ideally, you would use the same test before
and after treatment, but that may not always
be possible.
Example
• Attrition
– If the post-treatment drop-out rate is higher than
the pre-treatment drop-out rate, then (assuming
that folks not doing well drop the courses) this
could inflate the post-treatment grades
• To deal with this, you would need to record the dropout rate and demonstrate that a change in those rates
did not account for your results
Cook & Campbell (1979)
• When doing a interrupted time series design,
pay special attention to the following threats
to internal validity
– History
– Maturation
– Instrumentation
Summary
• Quasi-experiments
do
not
have
the
components to allow for causal statements
• In addition, the missing components (e.g.,
control groups, random assignment, etc.)
increase the likelihood of encountering
problems with the threats to internal validity