Quasi-Experiments – Outline
1. True Experiments
a. Characteristics
b. Threats to validity controlled by experiments
c. Threats not controlled by experiments
d. Obstacles to true experiments in the field
2. Quasi-experiments
a.The logic of quasi-experiments
b.Non-equivalent control group design
•
Example – Langer & Rudin (1976)
c. Interrupted time-series design
•
Example – Campbell (1969)
Quasi
True Experiments - Characteristics
• True experiments are
characterized by:
• A manipulation
• A high degree of control
• An appropriate comparison
(the major goal of exerting
control)
• Manipulation in the
presence of control gives
you an appropriate
comparison.
Quasi
Threats to validity controlled by true
experiments
• History
• occurrence of an event
other than the treatment
Quasi
Threats to validity controlled by true
experiments
• Maturation
• participants always change
as a function of time. Is
change in behavior due to
something else?
Quasi
Threats to validity controlled by true
experiments
• Testing
• improvement due to
practice on a test
(familiarity with
procedure, or with testers
expectations)
Quasi
Threats to validity controlled by true
experiments
• Instrumentation
• especially if humans are
used to assess behavior
(fatigue, practice)
Quasi
Threats to validity controlled by true
experiments
• Regression
• when first observation is
extreme, next one is likely
to be closer to the mean.
Quasi
Threats to validity controlled by true
experiments
• Selection
• if differences between
groups exist from the
outset of a study
Quasi
Threats to validity controlled by true
experiments
• Mortality
• if exit from a study is not
random, groups may end
up very different
Quasi
Threats to validity controlled by true
experiments
• Interactions of selection…
• with History
• with Maturation
• with Instrumentation (ceiling
effects)
Quasi
Note difference between these threats:
• Maturation
• One group; performance
better on post-test than on
pre-test
• Interaction of Maturation
& Selection
• Two or more groups
• Performance difference
larger on post-test than on
pre-test
Quasi
Threats to validity not controlled by experiments
• Contamination
• Cook & Campbell (1979):
• communication of
information about the
experiment between groups
of subjects
Quasi
• resentment
• ‘compensatory rivalry’
• diffusion of treatment:
control subjects use
information given to others
to change their own
behavior.
Contamination – an example
• Craven, Marsh, Debus, &
Jayasinghe (2001)
• Journal of Educational
Psychology
• Teachers trained to
improve students’
academic self-concept
through praise
• Internal control
• External control
Quasi
Contamination – an example
• Craven, Marsh, Debus, &
Jayasinghe (2001)
• Next slide shows T2 (posttest) academic selfconcept scores as a
function of T1 scores for
control children only.
Quasi
T2 acad self concept
1.0
0.5
No diffusion
Resentful demoralization?
Overzealous cooperation?
0.0
Low focus group
consistently higher than
external control
-0.5
Diffusion
-1.0
Low
Medium
External control
Internal control
High T1 acad self concept
Internal high focus
Internal low focus
Threats to validity not controlled by experiments
• Threats to external validity
• best way to deal with this is
replication
Quasi
Threats to validity not controlled by experiments
• Hawthorne effects
• changes in a person’s
behavior due to being
studied rather than the
manipulation.
• a special kind of reactivity.
Quasi
Hawthorne effects
• Demand characteristics
• cues communicated by
researcher
• subject’s under-standing of
their role
Quasi
Hawthorne effects
• Role of “research subject”
• Is subject behaving the
way he thinks a person in
that role should behave?
• (E.g., hypnotized person)
Quasi
Hawthorne effects
• Orne (1962)
• ‘good subjects’ think they
are contributing to science
by complying with
researcher’s demands
Quasi
Hawthorne effects
• What to do about
Hawthorne effects?
• Orne (1962): Use quasicontrol subjects as “coinvestigators”
• They do your task, reflect
on demand characteristics
of the experiment.
Quasi
Obstacles to true experiments in the field
• Sometimes, we cannot
bring the phenomenon we
want to study into the lab,
so we have to work in the
field.
• Can we do experiments in
the field?
Quasi
Obstacles to true experiments in the field
• Can’t get permission from
individuals in authority?
• Your study may involve
some time and effort on
their part. But what’s in it
for them?
• In schools, parents also
have to agree.
Quasi
Obstacles to true experiments in the field
• Can’t assign subjects to
groups randomly?
• have to work with intact
groups (e.g., classes in a
school)
Quasi
Quasi-Experiments
• Quasi-experiments
resemble true
experiments…
• …but they are not true
experiments.
• lack high degree of control
that is characteristic of true
experiments.
• usually include a
manipulation, and provide a
comparison.
Quasi
Quasi-Experiments
• Quasi-Experiments are
compromises
• They allow the researcher
some control when full
control is not possible.
Quasi
Quasi-Experiments
• Because full control is not
possible, there may be
several “rival hypotheses”
competing as accounts of
any change in behavior
observed.
• How do we convince others
that our hypothesis is the
right one?
Quasi
The Logic of Quasi-Experiments
• Eliminate any threats you
can
• Show how each threat to
validity on list given
above is dealt with in
your study.
• Argue that others don’t
apply.
• using evidence or logic
Quasi
Two kinds of quasi-experiments
•
Non-equivalent control
group
•
Quasi
“non-equivalent” because
not randomly assigned
Two kinds of quasi-experiments
•
Interrupted time-series
design
•
Quasi
a series of observations
over time, interrupted by
some treatment
Non-equivalent Control Group design
• Control group is “like” the
treatment group.
• Chosen from same
population
• Pre- and post-test measures
obtained for both groups,
so similarity can be
assessed.
Quasi
Non-equivalent Control Group design
• Control group is not
equivalent
• subjects are not randomlyassigned to control &
treatment groups
• so best you can do is argue
that comparison is
appropriate.
Quasi
Non-equivalent Control Group design
• If the groups are
comparable to begin with,
this design potentially
eliminates threats to
internal validity due to:
•
•
•
•
•
Quasi
History
Maturation
Testing
Instrumentation
Regression
Problems with the NECG design
• Threats to validity due to
interactions with selection
may not be eliminated using
the NECG design.
• Selection and maturation
• Most likely when treatment
group is self-selected (as in
psychotherapy cases –
people who sought help).
Quasi
Problems with the NECG design
• Selection and history
• Does one group experience
some event that has a
positive or negative effect
(e.g., teacher of one class
leaves)?
Quasi
Problems with the NECG design
• Selection and
instrumentation
• Does one group show
ceiling or floor effects?
Quasi
Problems with the NECG design
• Regression to the mean
• Are one group’s pretest
scores more extreme than
the other group’s?
Quasi
Possible NECG study outcomes
• both experimental and
control groups show
improve-ment from
pretest to posttest
• appears not to be any
effect of the treatment
Pretest
Posttest
Control group
Quasi
Possible NECG study outcomes
• Looks like a treatment
effect, but there may be a
threat due to
• selection and maturation,
• selection and history
Pretest
Posttest
Control group
Quasi
Possible NECG study outcomes
• Selection and maturation
could be a threat
• Or interaction of selection
and
•
•
•
•
history
testing
instrumentation
or mortality.
Pretest
Posttest
Control group
Quasi
Possible NECG study outcomes
• Interaction of selection and
regression looks like a
serious threat here
• Selection and maturation
probably not a threat here.
Pretest
Quasi
Posttest
Possible NECG study outcomes
• Crossover effect
• Clearest evidence for an
effect of the program of any
of these graphs.
• Selection and
instrumentation not a
problem – no ceiling or
floor effects
Pretest
Quasi
Posttest
Quasi-experiment example
• Langer & Rudin (1976)
• Research conducted in
retirement home.
• Residents on one floor
given more control over
their daily lives
• Residents of another floor
given same interaction
with staff, but no increased
control.
Quasi
Langer & Rudin (1976) – Measures
• Ratings
• Objective measures
• Self-report of feeling of
control from residents
• Staff assessments of mental &
physical well-being, by ‘blind’
assessors
Quasi
• record of movie attendance
• participation in “Guess how
many jelly-beans” contest on
each floor
L & R (1976) – limits on control
• L & R had no control over
• who entered the home
• who was assigned to either
floor.
• no control over staff hiring
or firing / resigning.
Quasi
L & R (1976) – Possible Problems
• Interaction of Selection and
Maturation
• even if groups have similar
pretest scores, they may
differ on things pretest
didn’t measure
• probably not a problem
here – people on both
floors had similar SES
• assigned to floors randomly,
not by health status.
Quasi
L & R (1976) – Possible Problems
• Selection and history
• suppose a popular (or
unpopular) nurse left one of
the floors during the study.
That might influence wellbeing.
• L & R did not address this
issue.
Quasi
L & R (1976) – Possible Problems
• Selection and
instrumentation
• did one group show ceiling
or floor effects?
• L & R say, no.
Quasi
L & R (1976) – Possible Problems
• Regression
• were one group’s pretest
scores more extreme than
the others?
• L & R say, no.
Quasi
L & R (1976) – Possible Problems
• Observer bias and
Contamination
• observers in the L & R study
were not aware of the
hypothesis.
• L & R reported there was
little communication
between floors.
Quasi
L & R (1976) – Possible Problems
• Hawthorne Effect
• cannot be ruled out, but L
& R took care to give both
floors same attention.
• Message varied between
floors, but “face time” was
the same.
Quasi
L & R (1976) – Possible Problems
• External Validity
• might be an issue.
• home involved was rated
“one of the finest” in the
state
• subjects may have been
atypical in their desire for
control
Quasi
Two kinds of quasi-experiments
Non-equivalent control
group
• Interrupted time-series
design
•
•
Quasi
a series of observations
over time, interrupted by
some treatment
Time-Series Designs
• In T-S designs, performance
is measured both before
and after a treatment.
• If there is an abrupt change
in performance at time of
treatment, we conclude
that treatment worked.
Quasi
Time-series designs example
• Campbell (1969)
• Effect of speed limit
reduction on traffic fatalities
in Connecticut
• incidence of traffic fatalities
in years before and after
the speed limit reduction,
• conclusion: speed limit
change had a modest effect.
Quasi
Campbell (1969)
• Any threat to internal
validity?
• other explanations for any
change in traffic fatality
incidence:
• Changes in car safety
• Weather
• Record keeping
Quasi
Campbell (1969)
• Any threat to internal
validity?
• Such effects should be
similar in neighboring states
• Campbell found no change
in fatality incidence in those
states.
Quasi
Campbell (1969)
• Any threat to external
validity?
• E.g., would treatment have
same effect in other states,
or are people in
Connecticut more lawabiding?
Quasi
Campbell (1969)
• Time-series design
eliminates most other
threats to validity – e.g.,
maturation, testing,
regression.
• For example, maturation
would probably not
produce a sudden change in
performance of the kind
found in Time-Series
Designs.
Quasi