Introduction to Research
Design
External Validity
Simple Research Designs
Extraneous Variable Control
External Validity
The extent to which the results of the
study can generalize beyond the
specifics of the experimental situation.
• Would you get the same results if you
– used different subjects
– manipulated the IV in a different way
– measured the DV in a different way
– The setting was different
– etc.
Threats to External Validity:
Testing x Treatment
Pretest Postest
Control Group Design
O
O
X
O
O
• Is the effect of the treatment the same
for pretested subjects as for subjects
not pretested?
• Can use a Solomon Four Group Design
• Or a Posttest Only Control Group Design
Threats to External Validity
Selection x Treatment :
Does the treatment interact with
characteristics of the subjects?
• Can you generalize your results to
different sorts of subjects?
– from laboratory rats to feral rats?
– from laboratory rats to humans?
– from US college students to those in China?
– from college students to farmers?
Threats to External Validity:
Reactive Effects of Experimental Arrangements
Can you generalize the results found
with observed subjects to subjects who
are not being observed?
• Difficult or impossible to observe without
affecting.
• Heisenberg uncertainty principle.
• Demand characteristics.
Threats to External Validity:
Multiple Treatment Interference
Do results found with subjects used in
many experiments generalize to
subjects without multiple treatment
experience?
• Expensive monkeys used in medical
research.
• Trained human subjects used in
perceptual research.
Simple Research Designs:
One-Shot Case Study
(X)
•
•
•
•
•
O
No variable is manipulated.
Just find group that has experienced X.
Example: Post tornado research here.
No pretest data, no comparison group.
Of little if any scientific value.
Randomized Pretest-Posttest
Control Group Design
R
R
O
O
X
O
O
• Random assignment to groups – can
assume groups equivalent prior to
treatment.
• Thus, no selection threat and
• No Selection x Maturation threat.
Randomized Pretest-Posttest
Control Group Design
R
R
O
O
X
O
O
• Effects of history, maturation, testing,
instrumentation can be assessed by prepost comparison in control group.
• They are controlled if they exist to the
same extent in both treatment and control
groups.
Randomized Pretest-Posttest
Control Group Design
R
R
O
O
X
O
O
• Testing x Treatment remains a threat to
external validity.
• Problems with random assignment
– Compare the two pretest means
– The Elmira College Study
Randomized Pretest-Posttest
Control Group Design
R O X O
R O
O
• Factorial ANOVA: Time x Group
– Mixed between subjects and within subjects
– Primary interest is in the Time x Group
interaction.
• Independent t comparing the two groups’
change scores (equivalent to interaction
analysis)
Randomized Pretest-Posttest
Control Group Design
R
R
O
O
X
O
O
• Analysis of Covariance
– pretest scores as covariate
– posttest scores as dependent variable
• All three of these have more power than
independent t on posttest scores.
Switching Replications Design
R
R
O
O
X
O
O
O
X
O
O
• This is a modification of the basic design.
• The control group gets the treatment too,
but after the other group gets it.
• This is useful when the treatment is
something that should be of value to all
subjects.
• Also known as the waitlist control group
design.
Randomized Switching
Replications Design
Intervention
Group
Pre (no X)
Post X
Followup
Waitlist Control Time 1 (no X) Time 2 (no X) Time 3 (no X)
Followup 2 Followup 3
Post X
Followup
Randomized Posttest Only
Control Group Design
R
R
X
O
O
• For controlling threats previously
discussed, this is the strongest design.
• But it usually has less power than designs
that include within subjects comparisons.
• This is a threat to statistical conclusion
validity.
Extraneous Variable Control
• Important to control extraneous variables
to
– Reduce noise
– eliminate confounds
• I’ll discuss five methods of controlling
extraneous variables.
Constancy
• Hold values of EVs constant across
subjects and groups.
– for example, use only female participants
– test all subjects at same time of day, etc.
• Eliminates confounds.
• Reduces noises, raising power.
• May reduce external validity
– do results generalize to men, etc?
Balancing
• Distribution of EV constant across groups
– if 60% female in one group, same in others
– if 70% tested during day in one group, same
in others
• Prevents confounding.
• Does not reduce noise.
Randomization
Assumed to produce balancing, thus
• Confounds are eliminated.
• Noise is not reduced.
Matching
• Subjects matched up on EV(s) into blocks
of k, where k = # of treatments.
• Within each block, randomly assign one
subject to each treatment.
• Matched pairs, randomized blocks, splitplot.
• Balances groups, eliminating confounds.
• Allows power-enhancing statistical
analysis.
Statistical Control
• Statistical techniques, such as ANCOV
and ANOVA, can be used to remove the
effect of extraneous variables and thus
increase power.
• I’ll discuss these in more detail in the next
unit of this class.