EXPERIMENTS
© Pine Forge Press, an imprint of Sage Publications, 2004
Different Types of
Experimental Design
true experiments
quasi-experiments
evaluation research
nonexperimental designs
© Pine Forge Press, an imprint of Sage Publications, 2004
Example of non-experimental design
[before-after/pretest-posttest]
The Imaginary Seattle
Bike Patrol Study
Pretest
O
Posttest
X
O
Look at change in crime rates
Subtract pretest score from posttest
O
X
O
Could have been;
Change in Economy
Change in weather
Change in media coverage
etc.
Instead of X, or combined with X
“Confounding”
Combining a second variable with the
independent variable is Confounding
External events
Such additional variables in the research
situation which provide an alternative
explanation to the one that X is changing Y,
in this case are classified as “external
events” because they are occurring outside
the study at the same time as the
independent variable is occurring.
HISTORY
Eternal events
HISTORY - ENVIRONMENTAL CHANGES …all
events that occurred during the time of the study that might
affect the individuals studied and provide a rival
explanation for the change in the dependent variable…all
events between the pretest [if given] and the posttest…
Changes in the economy
Changes in the weather
Changes in information provided by the media
War
Natural disaster
Etc.
Exogenous events
Note: “history” variables are also known as
“exogenous” events. This means that the
events are “outside” the study.
How can we ensure that the change was due
to X and not one or more of the external
events?... “History”
Classical experimental design:
Pretest
Bike patrol
Posttest
No bike patrol
Posttest
RA
• Pretest
We have randomly selected a bunch of
cities for the E group and the C
group…then…
We compare changes in each group.
Simultaneous variation
Pretest
Bike patrol
Change in economy
Change in weather
Media coverage
Etc.
Posttest
Pretest
no bike patrol
change in Economy
Change in weather
Media coverage
Etc
Posttest
RA
Simultaneous variation
Simultaneous variation allows for equal changes
in all other variables in the experimental group
and control group.
Since randomization would equalize [out] the
effects in both groups, the only reason for different
results in the experimental and control group
should be due to X.
S.V. allows for changes to occur in each group yet
it “CONTROLS” for effects of other variables.
N.B.
Be able to explain how simultaneous
variation and random assignment do their
work to “control.”
Review
History is controlled by random assignment
and allowing the History Variables to make
changes in Y equally in both the
experimental group and the control group.
Hence the only difference in changes in the
two groups can be attributed to X.
If this doesn’t make immediate sense ask
now, work on it, and get help!
True Experiments
True experiments must have at least one experimental
group (subjects who receive some treatment) and at least
one comparison group (subjects to whom the
experimental group can be compared).
01
01
© Pine Forge Press, an imprint of Sage Publications, 2004
True Experiments
True experiments must have at least three things:
•Two comparison groups (in the simplest case, an
experimental and a control group)
• Variation in the independent variable before assessment of
change in the dependent variable
•Random assignment to the two (or more) comparison
groups
© Pine Forge Press, an imprint of Sage Publications, 2004
True Experiments
True experiments must have at least one experimental
group (subjects who receive some treatment) and at least
one comparison group (subjects to whom the
experimental group can be compared).
01
01
© Pine Forge Press, an imprint of Sage Publications, 2004
True Experiments
All true experiments have a posttest—that is,
measurement of the outcome in both groups after the
experimental group has received the treatment. Many true
experiments also have pretests that measure the
dependent variable prior to the experimental intervention.
A pretest is exactly the same as a posttest, just
administered at a different time.
01
01
© Pine Forge Press, an imprint of Sage Publications, 2004
INTERNAL VALIDITY
The issue of X being responsible for the
changes to Y is an issue of INTERNAL
VALIDITY.
Since we ruled out all the outside factors,
history factors, external events which could
alternatively explain why the bike patrols
worked, we may tentatively assume that the
study indicates internal validity.
Why tentatively? RA & INTERACTION
Tentativeness
RA…random assignment doesn’t always
truly equalize the E group and the C group.
Interaction allows for the possibility that
bike patrols work sometimes and not others
and we may not “uncover” a hidden
relationship that work in different
directions.
RA doesn’t always equalize E & C group in values
of Y.
Or, Interaction may mask actual relationship:
Z1 O1
O2
Z2 O1
X1
O2
X2
Internal Validity
The 5 subtypes of internal validity examined here
are:
1. External [exogenous ] events / History
2. Endogenous events / Maturation, Instrumentation, Testing,
outside
inside
Regression
3. Selection bias / Selection Mortality
4. Treatment misidentification / Reactivity, Experimenter
Bias, Demoralization, Compensation, Placebo effect
5. Contamination / Experimental Diffusion, Contamination
Causal (Internal) Validity
There are four basic sources of noncomparability
(other than the treatment) between a comparison group and an
experimental group. They produce four of the five sources of
internal invalidity:
When characteristics of the experimental and
comparison group subjects differ SELECTION BIAS
When the subjects develop or change during the
experiment as part of an ongoing process independent of
the experimental treatment ENDOGENOUS EVENTS.
When something occurs during the experiment, other
than the treatment, which influences outcome scores
HISTORY.
When either the experimental group or the comparison
group is aware of the other group and is influenced in the
posttest as a result (Mohr, 1992)CONTAMINATION.
Causal (Internal) Validity
The fifth source of internal invalidity can be termed
TREATMENT MISIDENTIFICATION: Variation in the
independent variable (the treatment) is associated with
variation in the observed outcome, but the change occurs
through a process that the researcher has not identified.
© Pine Forge Press, an imprint of Sage Publications, 2004
Review
We have done history and found that it is
controlled by
E
RA
C
SIMULTANEOUS VARIATION
Endogenous Change
The next set of variables which may
interfere with internal validity are
exogenous variables. “When the subjects
develop or change during the experiment as
part of an ongoing process independent of
the experimental treatment.” These include:
Maturation, Instrumentation, Testing, &
Regression artifact.
Maturation
The bike patrol period was the beginning of
a baby bustlet a small baby bust for folks
growing into their teen age years. Fewer
teenagers…probably fewer crimes. Folks
grow out of lots of crimes when they get
older, marry, get decent jobs etc.
Maturation
MATURATION – GROWING OLDER,
WISER, TIREDER, DEVELOPMENTAL
CHANGES…biological, psychological, or
social processes that produce changes in the
individuals or units studied with the passage of
time that are not produced by the independent
variable(s) under study in the experiment.
Changes in the age distribution of the population
Changes in the interest in participating in the
dependent variable [getting tired of it]
How is it controlled?
E
O
RA
X
O
Bike patrol
Age changes
C
O
O
Age changes
SIMULTANEOUS VARIATION
INSTRUMENTATION
Police could have learned to be more
vigilant…thus may have caught more
crooks.
Alternatively, police may have grown tired
and thus been less vigilant.
INSTRUMENTATION
INSTRUMENTATION – INSTRUMENT
“DECAY” OR IMPROVEMENT…changes in
the measuring instrument between the pretest
and posttest.
Note changing springs on a scale…
Note changing experience of interviewers…
Note changing ability of coders…
How to control?
O
X
O
Bike patrol
Police Reporting changes
RA
O
O
Police Reporting changes
SIMULTANEOUS VARIATION
TESTING
The study of crime may have made the
potential victims more careful or may have
made the criminals more careful and thus
less likely to be caught.
TESTING
TESTING – PRACTICE OR
AWARENESS CHANGES THE
SUBJECT…the possible reactivity of
measurement …testing itself may change
the phenomenon being
measured…through awareness &
reflection, through experience &
practice…
How to control?
E
X
& changes in testing
RA
C
changes in testing
SIMULTANEOUS VARIATION
REGRESSION ATRIFACT
Crime scores were unreliably measured at
the top & the only way they could go was
down as they varied on second test.
Example using unreliable scale
“Extreme” scores can only go down.
300 [297]
300 [400]
300 [295]
300 [298]
300 [296]
____
1500
1500/5=300
292
300
293
294
300
____
1479
1479/5=295.8
300-295.8=4.2
REGRESSION ATRIFACT
REGRESSION ATRIFACT –
UNRELIABLE MEASURES OF GROPUS
WITH EXTREME SCORES; THERE’S
ONLY ONE WAY TO GO…if only extreme
cases are selected, and if the measurement
instrument is unreliable, when remeasuring
the only way unreliably low scores can go is
up and …high scores…down
How to control for “regression
artifact”
RA and simultaneous variation
Sam Ting w.r.t. changes due to unreliable
measures with only one way to go will
happen in the E group and the C group
Review
All endogenous changes involve changes within the
experiment other than X. They are controlled in the same
way. SAM TING!
They are controlled by SIMULTANEOUS VARIATION –
changing equally in the E group and the C group. RA
should place the same kinds of cases in both groups so that
change [aside from X] is “equalized/”
Hence SIMULTANEOUS VARIATION.
O
X
Other stuff
O
RA
O Other stuff O
Selection Bias
e.g.1 Seattle may have already had a “thing” about crime
such that the community was going to reduce it and the
police on bikes was just a temporal coincidence.
Seattle was already hell bent on changing crime and so the
selection of bikes coincided with an already established
tendency.
So Seattle has “selected” to make the change apart from
the bike patrol.
e.g.2 the film may not be the cause of
change when a survey is taken on film
effects. Filmgoers may have been more
liberal before they saw the film.
Confounding by selection
No X = not going to film
Non liberal folks avoid
film
X = film
Liberal folks go to such a
film
It isn’t film exposure .
It’s the initial liberalism of
the audience
Film exposure doesn’t
change people.
How to control for selection
See above
Sam Ting
RA will place the same kinds of cases in the
E group and the C group. Folks won’t
select themselves.
INTERACTIONS WITH SELECTION
– SUBJECT SELECTION - FOLKS
CHOOSE WHICH GROUP THEY’RE IN
SO GROUPS AREN’T REALLY EQUAL
subject may be selected with a
predilection to change or may appear to
change because of how they have selected
themselves…
EXPERIMENTAL MORTALITY
Perhaps all the crooks were jailed …thus
there was a smaller population of criminals
committing crime at time 2.
EXPERIMENTAL MORTALITY “DYING” / DIFERENTIAL ATTRITION
/ DROPPING OUT OF THE
EXPERIMENTAL OR CONTROL
GROUP…dropout problems that prevent
the researcher from obtaining information
on all cases.
How to control for mortality?
Same as above…assuming that the
experimental treatment or the control
treatment doesn't “kill off” subjects at a
different rate.
Treatment misidentification
some process of which the researcher may
not be aware is responsible for the apparent
effect of treatment
Treatment misidentification
N.B. Controlling for these sources of
internal invalidity involves procedures
different from employing a “true”
experimental design.
REACTIVITY
Realizing that the bike “experiment” was in
place, criminals may have been on guard.
Potential victims may have protected each
other.
Reactivity
REACTIVITY – KNOWLEDGE ONE IS IN AN
EXPERIMENT CHANGES HOW ONE ACTS
[“REACTS”]…sometimes inappropriately known
as the “Hawthorne effect” …subjects sometimes
become aware that they are in an experiment and,
because of this awareness, “react”
differently…Subjects may “feel special” and work
harder. Alternatively they may suffer from an
“audience effect.”
How to control?
Have both the E group and the C group
think they’re the “experimental” group.
E.g. they’re “blind” to what group they’re
in.
Experimenter bias
Maybe there was not less crime but the cops
liked the bikes so they cheated.
Maybe the cops misperceived.
EXPERIMENTER BIAS –
INADVERTANTLY GUIDING
SUBJECTS TOWARDS RESULTS TO
“CONFIRM” THE HYPOTHESIS ;
OUTRIGHT CHEATING
How to control?
The experimenter must be “blind” to what
condition the subject is in…during initial
instructions…conducting the experiment
and coding / analyzing the data.
Demoralization
DEMORALIZATION – SUBJECTS IN
THE CONTROL GROUP GIVING UP
Note: not relevant in the Seattle study,
because no control group
How to control?
Have subjects “blind” to which group
they’re in
EXPECTANCIES OF THE
EXPERIMENTAL STAFF/
COMPENSATION
EXPECTANCIES OF THE
EXPERIMENTAL STAFF/
COMPENSATION – STAFF
PROVIDING “EXTRAS” TO THE
CONTROL GROUP
How to control?
Have staff “blind to which group subjects
are in.
When not possible train and monitor the
staff.
PLACEBO EFFECT
Anticipating cop’s bike patrols’
effectiveness, criminals did less crime.
PLACEBO EFFECT
Placebo effect: Subjects change because
of expectations of change, not because of
treatment itself
How to control
o bike patrol
o
o
o
RA
faux bike patrol?
Sugar-pill analogy
S’s [and experimenters] must be BLIND w.r.t.
nature of treatment. Both must see the subject as
in the experimental group.
Review: Treatment Misidentification
Random Assignment is insufficient here.
For all these issues, either the subjects or
the experimenters must be “blind.”
You may have heard of medical
experiments wherein the procedures are
“double blind.” This is to ensure that
neither knowledge of the experimenter or
the subject will alter the results.
Contamination
Think of stuff from the experimental group
analogous to leeching through the soil to the
control group or vice versa.
EXPERIMENTAL DIFFUSION
EXPERIMENTAL DIFFUSION –
SUBJECTS ON ONE EXPERIMENTAL
CONDITION INFLUENCING
ANOTHER…by sharing information.
How to control?
Isolate…sepArate groups.
CONTAMINATION
CONTAMINATION – COMPARISON
[CONTROL] GROUP IS AFFECTED BY OR
AFFECTS THE EXPERIMENTAL GROUP
Compensatory rivalry or “John Henry effect.”
…or demoralization…is different from # 13 in
that in #13 information is passed on whereas in
14 only the knowledge of what group one is in is
passed on such that group members behave
differently. Wienir sees it as analogous to # 8
and doesn’t feel it is the same sort of
contamination as in # 13.
How to control?
Separate groups and keep members form
communication about which group they’re
in.
Comparison of two contamination
situations:
Diffusion: sharing content form experiment.
Contamination: One set of subjects letting
the other know who the “experimental
subjects” are.
Review
Type of invalidity:
History
Endogenous variables
Selection bias
Treatment
misidentification
Contamination
How to control:
RA & simultaneous variation
RA & simultaneous variation
RA & simultaneous variation
Blind
Separate
N.B.
Be able to explain fully how controlling
works in the situations in the slide above!
True Experiments
An Example:
Richard Price, Michelle Van Ryn, and Amiram Vinokur
(1992) hypothesized that a job-search program to help
newly unemployed persons could reduce the risk of
depression among this group.
The researchers tested this hypothesis with a sample of
unem-ployed persons who volunteered for job-search help
at Michigan Employment Security Commission offices.
© Pine Forge Press, an imprint of Sage Publications, 2004
True Experiments
The unemployed volunteers were randomly assigned either
to participate in eight 3-hour group seminars over a 2-week
period (the treatment) or to receive self-help information in
the mail on how to conduct a job search (the comparison
condition).
Those in the seminars were more likely to obtain jobs,
which would naturally decrease their risk of depression.
© Pine Forge Press, an imprint of Sage Publications, 2004
Quasi-experiments
A quasi-experimental design is one in which the
comparison group is predetermined to be comparable to
the treatment group in critical ways, such as being eligible
for the same services or being in the same school cohort
(Rossi & Freeman, 1989:313).
•Nonequivalent control group designs have
experimental and comparison groups that are
designated before the treatment occurs and are not
created by random assignment.
•Before-and-after designs have a pretest and posttest
but no comparison group. In other words, the subjects
exposed to the treatment serve, at an earlier time, as
their own controls.
© Pine Forge Press, an imprint of Sage Publications, 2004
Quasi-experiments
Ruth Wageman (1995) used a quasi-experimental design to
investigate how the way tasks were designed and rewards
allotted affected work team functioning.
Her research question was whether it was preferable to
organize work tasks and work rewards in a way that stressed
team interdependence or individual autonomy.
© Pine Forge Press, an imprint of Sage Publications, 2004
Quasi-experiments
David P. Phillips’s (1982) study of the effect of TV soapopera suicides on the number of actual suicides in the
United States illustrates a more powerful multiple group
before-and-after design.
© Pine Forge Press, an imprint of Sage Publications, 2004
Nonexperimental Designs
The ex post facto
control group
design has
experimental and
comparison
groups that are
not created by
random
assignment.
Unlike the groups
in nonequivalent
control group
designs, the
groups in ex post
facto (after the
fact) designs are
designated after
the treatment has
© Pine Forge Press, an imprint of occurred.
Sage Publications, 2004
Other Nonexperimental Designs
Cross-sectional designs, termed “one-shot case
studies” in the experimental design literature, are
easily able to establish whether an association exists
between two variables, but we cannot be anywhere
near as confident in their conclusions about
appropriate time order or nonspuriousness as with
true experiments or even quasi-experiments.
Longitudinal designs improve greatly our ability to test
the time order of effects, but they are unable to rule out all
extraneous influences.
© Pine Forge Press, an imprint of Sage Publications, 2004
External Validity
Researchers are often interested in determining
whether treatment effects identified in an experiment
hold true for subgroups of subjects and across different
populations, times, or settings.
There is always an implicit tradeoff in experimental
design between maximizing causal validity and
generalizability. The more that assignment to
treatments is randomized and all experimental
conditions are controlled, the less likely it is that the
research subjects and set-ting will be representative
of the larger population.
© Pine Forge Press, an imprint of Sage Publications, 2004
Although it may be possible to test a
hypothesis with an experiment, it may not
always be
desirable to do so.
The intersecting
complexity of societies, social relationships, and
social beings—of people and the groups to
which they belong—is so great that it often
defies reduction to the simplicity of a laboratory
or
restriction to the requirements of experimental
design.
© Pine Forge Press, an imprint of Sage Publications, 2004
True Experiments
Randomization, or random assignment, is what makes the comparison
group in a true experiment such a powerful tool for identifying the effects
of the treatment. A randomized comparison group can provide a good
estimate of the counterfactual—the outcome that would have occurred if
the subjects who were exposed to the treatment actually had not been
exposed but otherwise had had the same experiences
© Pine Forge Press, an imprint of Sage Publications, 2004