Research
Problems
Hawthorne Effect
 Western Electric’s Hawthorne Plant 1939 study of
light intensity
 productivity went…up
 Potential Solutions:
 run experiment for a longer period
 use a control group
John Henry Effect
 legend of black railway worker
 control group overcompensates
 Potential Solutions:
 don’t do threatening experiments
 don’t set up obviously competitive situations
 don’t tell control group that they are control group
• conduct in another school somewhere else
 unfortunately, produces new variable of different school,
neighbourhood, etc.!
Placebo Effect
 introduce placebo in attempt to make conditions in
treatment and control groups identical
 placebo effects are reactions that (after taking
the placebo) cannot be explained by the chemical
or medical effects of the placebo.
— psychological factors
Placebo Effect
Potential Solutions
 double-blind experiment
 secrecy
 but then violate principle of informed consent
 screen out or balance number of placebo reactors
in treatment & control groups
the Pygmalion effect
 Rosenthal and Jacobson, 1968
 self-fulfilling prophecy
 Potential solution:
 do not tell subjects what experiment is about
 but what happens to principle of informed consent?
Demand Characteristics
 rumour
 setting
 instructions
 status and personality of researcher
 unintentional cues from experimenter
 experimental procedure itself
Demand Characteristics
Potential Solutions

reduce clarity of demand characteristics

generate alternative demand characteristics

reduce subjects’ motivation to respond to demand
characteristics
Validity

Internal Validity
– control experiment to eliminate
extraneous variables

External Validity
– outcome can be generalized to other
populations in other settings
Threats to Internal Validity
 History
 change producing events in addition to the
experimental treatment
 Maturation
 subjects grow older, learn more, can do more
Threats to Internal Validity
 Testing
 in pre-test/post-test designs — subjects may
remember questions from pre-test
 Instrumentation
 changes in measurement
 changes in observer
 changes in subjects (testing effects)
Threats to Internal Validity
 Differential Selection
 Differences between treatment and control
group
 E.g., volunteers vs non-volunteers
 Experimental Mortality (or Attrition)
 subjects that drop out of experiment may
differ from others in important ways
Threats to Internal Validity
 Testing & Experimental Treatment
Interaction
 In pre-test/post-test design, pretest may
interact with experimental treatment to
exaggerate result
 Statistical Regression
Threats to Internal Validity
 Not Paying Attention to What’s Really
Going On
External Validity

1. Population Validity
– can results be generalized from specific sample
to the population from which sample was drawn

2. Ecological validity
– can results be generalized from contrived
conditions created by experimenter to another
set of environmental conditions (i.e., real world)
Experimental Design
Experimental Design

Purpose is to design an experiment that
controls for as many extraneous variables
as possible
– Campbell and Cook, classic 1968 paper
categorized experimental designs by how many
variables they controlled
Weak Designs
Single Group Designs
The One-Shot Case Design
X
0
X = treatment
o = observation or measurement
The One-Shot Case Design

hardly experiment at all
– can’t be sure result was result of treatment, not history,
maturation, etc.
 no control over group selection
 could have problem with subject mortality (e.g., transient
student population)
 since only tested once, can’t even measure gain — maybe
students knew it before we started

consequently, results are largely meaningless
One Group
Pre-test/Post-test Design
O
1
X
0
2
One Group
Pre-test/Post-test Design

Uncontrolled Extraneous Factors (all of them!)
 history e.g., Hawthorne effect
 maturation
 testing
 Pre-test may have contributed to higher scores on post-test due
to greater familiarity with types of questions or focus on certain
topics
 instrumentation
 pre-test and post-test the same? Observer the same?
 selection: could be an atypical group
 mortality
 interaction of testing & experimental treatment
 statistical regression
One Group
Pre-test/Post-test Design


useful for studying stable dependent variable
justified when
– extraneous factors can be estimated with a high degree
of certainty or
– can be safely assumed to be nonexistent
• E.g., not maturation because change is too dramatic for
maturation to explain it
Control Group Designs
Posttest-Only
Nonequivalent Groups Design
X
O
1
-----------------------------------------
O
2
(dotted line me ans no t random selection)
Posttest-Only
Nonequivalent Groups Design




also called “static group comparisons”
still a relatively weak design
advantage over single one shot design that one can compare,
so cancel out maturation, etc.
Problems:
 Differential selection
 experimental mortality
Non-Equivalent
Control Group Design
Expt
O
1
X
O
2
-----------------------------------------
Cntl
O
3
O
4
can compa re average (mean ) gain O2-O1
with average gain O4-O3
Non-Equivalent
Control Group Design



stronger design (weakest of the strong designs)
common design in education because usually can’t
randomize assignment of students to classes
pre-test measures whether initial groups are
similar on tested variable
– could also match subjects based on pre-test
 but may miss other factors which could impact results
(e.g., better teacher in one group)
Non-Equivalent
Control Group Design

Factors controlled by inclusion of a control group






history
maturation
testing
instrumentation (assuming same for both groups)
statistical regression
Factors still NOT controlled
 differential selection
 experimental mortality (if any)
 testing and experimental treatment interaction
Pre-test/Post-test
Control Group Design
R
R
O
O
1
3
X
O
O
2
4
where R = randomly selected groups
Pre-test/Post-test
Control Group Design
 still two uncontrolled factors

Intersession History
– events that are specific to one group and not the other which
occur during the group sessions
– To control for interssion history, need to balance (control) such
factors as
• experimenters
• time of day
• day of week
by randomly assigning to groups of subjects
Pre-test/Post-test
Control Group Design

Interaction of testing and treatment
– Cannot be avoided when there is a pretest
– eliminating the pre-test would therefore
improve design
The Post-test Only
Control Group Design
R
R
X
O
O
1
2
The Post-test Only
Control Group Design


do not need pre-test with random selection
disadvantages
– random assignment may not be fully successful in eliminating
initial differences between control and experimental groups
– cannot form subgroups (i.e., high, medium, & low) to determine
whether the experimental treatment has a different effect on
subjects at different levels of the variable as measured by the
pretest (because no pretest)
Solomon
Four-Group Design
R
R
R
R
O
O
X
X
O
O
O
O
Solomon
Four-Group Design


basically, a pretest and non-pretest
experiment at the same time
controls for everything, except
differential mortality
Time Series Design
OOOXOOO
Time Series Design



Similar to one-group pre-test /post-test design
(weak #2) but additional pre- and postmeasurements add power
additional measurements enable researcher to rule
out maturation
testing effects as sources of influence shift from
pretest to posttest