Outline
1. True Experiments
a. Characteristics
b. Threats to validity controlled by experiments
c. Threats not controlled by experiments
d. Obstacles to true experiments in the field
2. Quasi-experiments
a.The logic of quasi-experiments
b.Non-equivalent control group design
• Example – Langer & Rudin (1976)
c. Interrupted time-series design
• Example – Campbell (1969)
True Experiments Characteristics
• True experiments are
characterized by:
– A manipulation
– A high degree of
control
– An appropriate
comparison (the major
goal of exerting
control)
• Manipulation in the
presence of control
gives you an
appropriate
comparison.
Threats to validity controlled by true
experiments
• History
• occurrence of an
event other than the
treatment
Threats to validity controlled by true
experiments
• History
• Maturation
• participants always
change as a function
of time. Is change in
behavior due to
something else?
Threats to validity controlled by true
experiments
• History
• Maturation
• Testing
• improvement due to
practice on a test
(familiarity with
procedure, or with
testers expectations)
Threats to validity controlled by true
experiments
•
•
•
•
History
Maturation
Testing
Instrumentation
• especially if humans
are used to assess
behavior (fatigue,
practice)
Threats to validity controlled by true
experiments
•
•
•
•
•
History
Maturation
Testing
Instrumentation
Regression
• when first observation
is extreme, next one
is likely to be closer to
the mean.
Threats to validity controlled by true
experiments
•
•
•
•
•
•
History
Maturation
Testing
Instrumentation
Regression
Selection
• if differences between
groups exist from the
outset of a study
Threats to validity controlled by true
experiments
•
•
•
•
•
•
•
History
Maturation
Testing
Instrumentation
Regression
Selection
Mortality
• if exit from a study is
not random, groups
may end up very
different
Threats to validity controlled by true
experiments
•
•
•
•
•
•
•
•
History
Maturation
Testing
Instrumentation
Regression
Selection
Mortality
Interactions of
selection…
• with History
• with Maturation
• with Instrumentation
(ceiling effects)
Note difference between these
threats:
• Maturation
– One group;
performance better on
post-test than on pretest
• Interaction of
Maturation &
Selection
– Two or more groups
– Performance
difference larger on
post-test than on pretest
Threats to validity not controlled by
experiments
• Contamination
– communication of
information about the
experiment between
groups of subjects
• Cook & Campbell
(1979):
– resentment
– ‘compensatory rivalry’
– diffusion of treatment:
control subjects use
information given to
others to change their
own behavior.
Contamination – an example
• Craven, Marsh,
Debus, & Jayasinghe
(2001)
• Journal of
Educational
Psychology
• Teachers trained to
improve students’
academic selfconcept through
praise
• Internal control
• External control
Contamination – an example
• Craven, Marsh,
Debus, & Jayasinghe
(2001)
• Next slide shows T2
(post-test) academic
self-concept scores
as a function of T1
scores for control
children only.
T2 acad self concept
1.0
0.5
No diffusion
Resentful demoralization?
Overzealous cooperation?
0.0
Low focus group
consistently higher than
external control
-0.5
Diffusion
-1.0
Low
Medium
External control
Internal control
High T1 acad self concept
Internal high focus
Internal low focus
Threats to validity not controlled by
experiments
• Contamination
• Threats to external
validity
• best way to deal with
this is replication
Threats to validity not controlled by
experiments
• Contamination
• Threats to external
validity
• Hawthorne effects
• changes in a person’s
behavior due to being
studied rather than the
manipulation.
• a special kind of
reactivity.
Hawthorne effects
• Demand
characteristics
• cues communicated
by researcher
• subject’s understanding of their role
Hawthorne effects
• Role of “research
subject”
• Is subject behaving
the way he thinks a
person in that role
should behave?
• (E.g., hypnotized
person)
Hawthorne effects
• Orne (1962)
• ‘good subjects’ think
they are contributing
to science by
complying with
researcher’s
demands
Hawthorne effects
• What to do about
Hawthorne effects?
• Orne (1962): Use
quasi-control subjects
as “co-investigators”
• They do your task,
reflect on demand
characteristics of the
experiment.
Obstacles to true experiments in
the field
• Sometimes, we cannot
bring the phenomenon
we want to study into
the lab, so we have to
work in the field.
• Can we do
experiments in the
field?
Obstacles to true experiments in
the field
• Can’t get permission
from individuals in
authority?
• Your study may
involve some time and
effort on their part. But
what’s in it for them?
• In schools, parents
also have to agree.
Obstacles to true experiments in
the field
• Can’t get permission
from individuals in
authority?
• Can’t assign subjects
to groups randomly?
• have to work with
intact groups (e.g.,
classes in a school)
Quasi-Experiments
• Quasi-experiments
resemble true
experiments…
– usually include a
manipulation, and
provide a comparison.
• …but they are not true
experiments.
– lack high degree of
control that is
characteristic of true
experiments.
Quasi-Experiments
• Quasi-Experiments
are compromises
• They allow the
researcher some
control when full
control is not possible.
Quasi-Experiments
• Because full control is
not possible, there
may be several “rival
hypotheses”
competing as
accounts of any
change in behavior
observed.
• How do we convince
others that our
hypothesis is the right
one?
The Logic of Quasi-Experiments
• Eliminate any threats
you can
– Show how each
threat to validity on
list given above is
dealt with in your
study.
• Argue that others don’t
apply.
– using evidence or
logic
Two kinds of quasi-experiments
•
Non-equivalent control
group
•
“non-equivalent”
because not randomly
assigned
Two kinds of quasi-experiments
•
Non-equivalent control
group
• Interrupted time-series
design
•
a series of
observations over
time, interrupted by
some treatment
Non-equivalent Control Group
design
• Control group is “like”
the treatment group.
• Chosen from same
population
• Pre- and post-test
measures obtained for
both groups, so
similarity can be
assessed.
Non-equivalent Control Group
design
• Control group is not
equivalent
• subjects are not
randomly-assigned to
control & treatment
groups
• so best you can do is
argue that comparison
is appropriate.
Non-equivalent Control Group
design
• If the groups are
comparable to begin
with, this design
potentially eliminates
threats to internal
validity due to:
•
•
•
•
•
History
Maturation
Testing
Instrumentation
Regression
Problems with the NECG design
• Threats to validity due
to interactions with
selection may not be
eliminated using the
NECG design.
• Selection and
maturation
• Most likely when
treatment group is
self-selected (as in
psychotherapy cases
– people who sought
help).
Problems with the NECG design
• Selection and
maturation
• Selection and history
• Does one group
experience some
event that has a
positive or negative
effect (e.g., teacher of
one class leaves)?
Problems with the NECG design
• Selection and
maturation
• Selection and history
• Selection and
instrumentation
• Does one group show
ceiling or floor effects?
Problems with the NECG design
• Selection and
maturation
• Selection and history
• Selection and
instrumentation
• Regression to the
mean
• Are one group’s
pretest scores more
extreme than the
other group’s?
Possible NECG study outcomes
• both experimental
and control groups
show improve-ment
from pretest to
posttest
• appears not to be any
effect of the treatment
Pretest
Posttest
Control group
Possible NECG study outcomes
• Looks like a treatment
effect, but there may
be a threat due to
– selection and
maturation,
– selection and history
Pretest
Posttest
Control group
Possible NECG study outcomes
• Selection and
maturation could be a
threat
• Or interaction of
selection and
–
–
–
–
history
testing
instrumentation
or mortality.
Pretest
Posttest
Control group
Possible NECG study outcomes
• Interaction of selection
and regression looks
like a serious threat
here
• Selection and
maturation probably
not a threat here.
Pretest
Posttest
Possible NECG study outcomes
• Crossover effect
• Clearest evidence for
an effect of the
program of any of
these graphs.
• Selection and
instrumentation not a
problem – no ceiling or
floor effects
Pretest
Posttest
Quasi-experiment example
• Langer & Rudin
(1976)
• Research conducted
in retirement home.
• Residents on one
floor given more
control over their daily
lives
• Residents of another
floor given same
interaction with staff,
but no increased
control.
Langer & Rudin (1976) – Measures
• Ratings
– Self-report of feeling of
control from residents
– Staff assessments of
mental & physical wellbeing, by ‘blind’
assessors
• Objective measures
– record of movie
attendance
– participation in “Guess
how many jelly-beans”
contest on each floor
L & R (1976) – limits on control
• L & R had no control
over
• who entered the home
• who was assigned to
either floor.
• no control over staff
hiring or firing /
resigning.
L & R (1976) – Possible
Problems
• Interaction of Selection
and Maturation
– even if groups have
similar pretest scores,
they may differ on
things pretest didn’t
measure
• probably not a
problem here – people
on both floors had
similar SES
• assigned to floors
randomly, not by
health status.
L & R (1976) – Possible
Problems
• Selection and history
• suppose a popular (or
unpopular) nurse left
one of the floors
during the study. That
might influence wellbeing.
• L & R did not address
this issue.
L & R (1976) – Possible
Problems
• Selection and history
• Selection and
instrumentation
• did one group show
ceiling or floor
effects?
• L & R say, no.
L & R (1976) – Possible
Problems
• Selection and history
• Selection and
instrumentation
• Regression
• were one group’s
pretest scores more
extreme than the
others?
• L & R say, no.
L & R (1976) – Possible
Problems
• Selection and history
• Selection and
instrumentation
• Regression
• Observer bias and
Contamination
• observers in the L & R
study were not aware
of the hypothesis.
• L & R reported there
was little
communication
between floors.
L & R (1976) – Possible
Problems
• Selection and history
• Selection and
instrumentation
• Regression
• Observer bias and
Contamination
• Hawthorne Effect
• cannot be ruled out,
but L & R took care to
give both floors same
attention.
• Message varied
between floors, but
“face time” was the
same.
L & R (1976) – Possible
Problems
• Selection and history
• Selection and
instrumentation
• Regression
• Observer bias and
Contamination
• Hawthorne Effect
• External Validity
• might be an issue.
• home involved was
rated “one of the
finest” in the state
• subjects may have
been atypical in their
desire for control
Two kinds of quasi-experiments
•
Non-equivalent
control group
• Interrupted timeseries design
•
a series of
observations over
time, interrupted by
some treatment
Time-Series Designs
• In T-S designs,
performance is
measured both before
and after a treatment.
• If there is an abrupt
change in
performance at time of
treatment, we
conclude that
treatment worked.
Time-series designs example
• Campbell (1969)
• Effect of speed limit
reduction on traffic
fatalities in
Connecticut
• incidence of traffic
fatalities in years
before and after the
speed limit reduction,
• conclusion: speed limit
change had a modest
effect.
Campbell (1969)
• Any threat to internal
validity?
• other explanations for
any change in traffic
fatality incidence:
– Changes in car safety
– Weather
– Record keeping
Campbell (1969)
• Any threat to internal
validity?
• Such effects should be
similar in neighboring
states
• Campbell found no
change in fatality
incidence in those
states.
Campbell (1969)
• Any threat to external
validity?
• E.g., would treatment
have same effect in
other states, or are
people in Connecticut
more law-abiding?
Campbell (1969)
• Time-series design
eliminates most other
threats to validity –
e.g., maturation,
testing, regression.
• For example,
maturation would
probably not produce
a sudden change in
performance of the
kind found in TimeSeries Designs.