External Validity
 Generalization—Applying experimental results to a different situations or populations
than those found in the study.
 The extent to which we can generalize the results of the study to other
populations, settings, times, measures, etc.
 The goal of good research is to make a connection between results of “this study”
and the situations/ people in the real world .
 Population Generalization
 The degree to which a study’s results can be assumed true of people not included
in the study. (from sample to pop)
 Would the same results be obtained with a different sample from the pop?
 The Replication Fix
 Replication—conducting a study in exactly the same way it was conducted
before.
 Goal:
 More confidence in results consistently found upon multiple replications
 Replication with extension—replicating a previous study but with different participants
or under different conditions
 Often used in cross-cultural and cross-group research.
 Results apply to Native Americans? Older persons? People of the WWII
generation? Etc.
 Environmental Generalization
 The degree to which similar results would be obtained in environments/situations
different than that of the experiment itself.
 If results occur only in the lab, then their practical application is limited (i.e., the
study is pointless)
 Temporal Generalization
 Applying results to a time different from the time during which the study was
conducted.
 As times change, old knowledge/findings become less relevant
 Occasionally discover "timeless” findings: findings replicated across time, eras,
and seasons.
 Example: fight or flight mechanisms, tendency of men and women to process
information differently
 Tend to be more biological findings, but even those are often subject to changes
over time because the brain is plastic and fluid (not a stone, more like clay)
 Threats to External Validity
 Any characteristic of the study that limits the generalizability of the results
 May come from
 Experimental Methods
 Participants
 Method’s-Based Threats to
External Validity
 Interaction of testing and treatment—taking a pretest may influence
participants’ responses to a post-test.
 Responses to the treatment may be different due to the pretest (serves as a primer
that may confound results)
R
O1
R
O3
O2 (control group)
X
O4 (experimental group)
 Particularly troublesome for studies investigating attitude change
 Example: Measure attitudes toward teachers, watch Mr. Holland’s Opus, measure
attitudes again
 A way to control this threat is to have non-pretesting group(s)
 Method’s-Based Threats to
External Validity
 Interaction of selection and treatment—when results (or the treatment effect) are
found only in a specific sample of participants.
 A particular threat for studies that require hard-to-find participants. It is more likely
these participants will be uniquely different from the population (not representative)
 Example:
 Intentional sampling: A sample of the uber-rich, a sample of hospice nurses
 Unintentional sampling: summer school students, English teachers
 Method’s-Based Threats to
External Validity
 Multiple-Treatment Interference—occurs when a set of findings is due to
participants receiving multiple treatments in the same experiment
 Example: results on a memory task (learning a list of words) for participants who had
already learned several word lists. Their previous treatment is impacting (likely
heightening) their ability to learn a new task. Thus, results are NOT generalizable to
persons without these previous treatments.
 Example: Test-taking skills of seniors vs. freshman. Results from seniors are not
applicable to freshman.
 Method’s-Based Threats to
External Validity
 Reactive arrangements—when the experimental situation alters participants’
behavior.
 Due to need for control, experimental situations are often highly contrived
/artificial
 Since experimental settings do not exist in the real world, we cannot say that
similar results would be found in the real world.
“The play-acting, outguessing, up-for-inspection, I’m-a-guinea-pig, or whatever attitudes so
generated are unrepresentative of the [real world], and seem to be qualifiers of the effect of X
[the treatment], seriously hampering generalization.”
(Campbell & Stanley, 1966, p. 20).
 Participant-Based Threats to
External Validity
 Good Subject
 Attempt to corroborate experimenter’s hypothesis
 Negativistic Subject
 Attempt to refute experimenter’s hypothesis
 Faithful Subject
 Follow instructions and behaves in socially appropriate ways more readily
than they may in real-world situations
 Apprehensive Subject
 Overly concerned with evaluations of their performance
 Participant-Based Threats to
External Validity
 The infamous white rat
 The Norway white rat represents .001% of all living creatures that could
be studied.
 Between 1993 and 1995, 75% of studies published in the Journal of
Experimental Psychology: Animal Behavior Processes reported results
obtained from rats and pigeons.
 Participant-Based Threats to
External Validity
 College students are the human researcher’s lab rat and extra credit is the cheese.
 Sample of Convenience—including certain participants simply because
they are readily available, often does not involve random sampling.
 Can college students represent an entire population. Absolutely NOT!
 Example: attitude change—traditionally aged college students have less
crystallized attitudes than persons older than they.
 Example: learning/memory differences due to academic demands
 Example: some psychological disorders disproportionately impact younger
(schizophrenic break early 20s) or older persons (Dementia, personality
disorders calm down)
 Participant-Based Threats to
External Validity
 The Gender bias
 Three quarters of all psychology majors are women (problem for academic
researchers)
 The Caucasian bias
 Cannot automatically generalize results based on Caucasian persons to
non-Caucasian persons.
 The American bias
 Cannot automatically generalize results based on American participants—
even American participants of minority status (Beware acculturation).