Pelham, B. W., & Blanton, H. (2003). Conducting research in psychology: Measuring the
weight of smoke (2nd Ed.). Toronto: Thompson Wadsworth
Chapter 4: How do we misinterpret? Common threats to validity
The chapter focuses on four general categories of threat to the validity of a set of
research findings (each divided into three more specific sub-categories).
 People are different
 People change
 The process of studying people changes people
 Variables that systematically accompany a treatment (confounds) change people
People are different (p. 86)
 Individual differences
o Pseudo-experiments are prone to this threat (absence of some kind of
comparison group or control group) … individual differences are a very
good alternate explanation for the findings observed in pseudoexperiments
o Individual differences are a threat to internal validity
o Experimental method is relatively immune to threat to internal validity,
including those based on individual differences
 Selection bias
o Refers to sampling from an unrepresentative bias
o Threat to external validity
 Nonresponse bias
o Respondents themselves are the source of bias because people who
choose to answer surveys are systematically different from people who
choose not to do so
o Surveys that have low response rates may yield information that is
misleading
 Relying on convenience sample of college students may represent a serious
shortcoming of much psychological research
 It is important to realize that the efficacy or utility of a sample technique depends
as much upon the group of people about whom we would like to generalize as
much as it does on the group of people we have sampled
People change (p. 89)
 People change over time and across situation
 If people change for reasons that have nothing to do with the treatment, then
these changes can lead to some inappropriate conclusions about the treatment
 History
o Refers to changes that are occurring more or less across the board in a
very large group of people such as a nation or culture
o Common threat to internal validity
 Maturation
o Refers to the specific developmental or experiential changes that are
occurring in a particular person, or particular age cohort, over time
o Common threat to internal validity
 Regression toward the mean
o
o
o
o
o
o
Refers to the tendency for people who receive high or low scores on a
particular measure to score closer to the mean on a subsequent testing
Changes that are mistaken for experimental treatments occur when
research participants do not experience any true changes (such as
maturation) but score differently than they did originally when they take a
retest of a particular measure of personality or performance
People fail to appreciate the potency and ubiquity of regression toward
the mean
True scores are more stable than observed scores (influenced by the true
score and error) … regression toward the mean exists because
performance is always the joint product of skill and luck
Threat to internal validity
Pre-post studies that do not include a control group are most susceptible
The process of studying people changes people (p. 94)
 Hawthorne effect
o Studying people can dramatically change the way that people behave
 Testing effects
o Refer to the tendency for most participants to perform better on a test or
personality measure the second time they take it
o Probably reflect learning on the part of the test taker
o In the case of many psychological tests, test effects may represent a form
of attitude polarization (allowing people to give a little thought to their
attitudes often leads them to become more extreme in these attitudes)
o There are several ways to correct the problem of testing effects
 Conduct a true experiment in which you estimate the pretest
(requires some faith in random assignment)
 Wait as long as possible between pre and post test
 Use an alternate version of the test
 Experimental mortality (attrition)
o Refers to the failure of some participants in an experiment to complete
the study
o Can represent a threat to both internal and external validity
o Homogeneous attrition: equal levels of attrition across all experimental
conditions (threat to external but not internal validity)
o Heterogeneous attrition: the attrition rates in two or more conditions of an
experiment are noticeably different (threat to internal and external validity)
… basically erases the normal benefits of random assignment
o Ways to reduce attrition include: Provide realistic expectations,
incentives, emphasize the scientific importance of attrition
 Participant reaction bias
o Occurs when people realize they are being studies and behave in ways
that they normally wouldn’t
o Three basic types of reaction
 Participant expectancies: People may try to do what they think the
researcher expects
 Participant reactance: People may try to do the opposite of what
they think the researcher expects

o
o
o
Evaluation apprehension: Concerns about being judged favorably
or unfavorably by another person. In an experiment, people may
try to do whatever will make them look good
These represent threats to internal validity
Demand characteristics: characteristics of an experiment that subtly
suggest how people are expected to behave
Ways to reduce participants reaction bias include:
 guarantee anonymity
 use a cover story (a false and often elaborate story about the
purpose and nature of the study to create expectancies unrelated
to the real predictions of the experiment) … deception (ethics?)
 unobtrusive observations (participants do not realize they are
being observed, or if they do, they don’t realize what is being
observed)
 use a bogus pipeline (convince participants that researchers can
read people’s minds)
 use indirect measures of people’s attitudes and opinions (e.g., ask
about reactions to another’s behaviour) … pencil-and-paper
version of an unobtrusive observation
 measure attitudes people don’t know they posses (e.g., IAT)
Variables that accompany a treatment change people (p. 105)
 researchers sometimes unintentionally allow things to vary along with their
independent variables
 can be hard to detect, but represent threats to internal validity
 Experimenter bias
o Refers the influence of the experimenter’s expectations about their
studies on their experimental observations
o Sometimes see what we expect to see
o Experimenters may treat their participants differently according to their
expectations about how their participants should perform (double-blind
procedure can overcome this problem)
 Confounds
o A broad term for any design problem in which some additional variable
varies systematically along with the independent variable …sometimes
referred to as a third variable problem
o Confounds can influence correlational and experimental studies
o Relative to other designs, true experiments are less likely to contain
confounds
o Represent a serious threat to internal validity
o Ways to avoid confounds include:
 Be careful,
 read the existing literature
 measure any variables that are likely to be confounded with the
variable(s) in which you are interested