Unit 1c: Essential
Elements of Research
PSYC 4310
COGS 6310
MGMT 6969
Michael J. Kalsher
Department of
Cognitive Science
© 2015, Michael Kalsher
1
Internal vs. External Validity
Internal Validity
Extent to which causal/independent variable(s) and no
other extraneous factors caused the change being
measured.
External Validity (generalizability)
Degree to which the results and conclusions of your
study would hold for other persons, in other places,
and at other times.
© 2015, Michael Kalsher
2
Threats to Internal Validity:
Factors that reduce our ability to draw valid conclusions
Selection
History
Maturation
Repeated Testing
Instrumentation
Regression to the mean
Subject mortality
Selection-interactions
Experimenter bias
© 2015, Michael Kalsher
3
Reducing Threats to Internal Validity
The role of Control
Behavior is influenced by many factors termed—confounding
variables—that tend to distort the results of a study, thereby
making it impossible for the researcher to draw meaningful
conclusions. Some of these may be unknown to the
researcher.
Control refers to the systematic methods (e.g., research
designs) employed to reduce threats to the validity of the study
posed by extraneous influences on both the participants and
the observer (researcher).
© 2015, Michael Kalsher
4
Group/Selection threat
Occurs when nonrandom procedures are used to assign
subjects to conditions or when random assignment fails
to balance out differences among subjects across the
different conditions of the experiment.
Example:
A researcher is interested in determining the factors most likely to
elicit aggressive behavior in male college students. He exposes
subjects in the experimental group to stimuli thought to provoke
aggression and subjects in the control group to stimuli thought to
reduce aggression and then measures aggressive behaviors of the
students. How would the selection threat operate in this instance?
© 2015, Michael Kalsher
5
History threat
Events that happen to participants during the
research which affect results but are not linked to
the independent variable.
Example:
The reported effects of a program designed to improve
medical residents’ prescription writing practices by the
medical school may have been confounded by a self-directed
continuing education series on medication errors provided to
the residents by a pharmaceutical firm's medical education
liaison.
© 2015, Michael Kalsher
6
Maturation threat
Can operate when naturally occurring biological or
psychological changes occur within subjects and
these changes may account in part or in total for
effects discerned in the study.
Example:
A reported decrease in emergency room visits in a long-term
study of pediatric patients with asthma may be due to subjects
outgrowing childhood asthma rather than to any treatment
regimen introduced to treat the asthma.
© 2015, Michael Kalsher
7
Repeated testing threat
May occur when changes in test scores occur not
because of the intervention but rather because of
repeated testing. This is of particular concern when
researchers administer identical pretests and
posttests.
Example:
A reported improvement in medical resident prescribing
behaviors and order-writing practices in the study previously
described may have been due to repeated administration of the
same short quiz. That is, the residents simply learned to provide
the right answers rather than truly achieving improved
prescribing habits.
© 2015, Michael Kalsher
8
Instrumentation threat
When study results are due to changes in instrument
calibration or observer changes rather than to a true
treatment effect, the instrumentation threat is in
operation.
Example:
In Kalsher’s Experimental Methods and Statistics course, he
evaluates students progress in understanding principles of research
design at week 3 of the semester. A graduate T.A. evaluates the
students at the conclusion of the course. If the evaluators are
dissimilar enough in their approach, perhaps because of lack of
training, this difference may contribute to measurement error in
trying to determine how much learning occurred over the semester.
© 2015, Michael Kalsher
9
Statistical Regression threat
The regression threat can occur when subjects
have been selected on the basis of extreme
scores, because extreme (low and high) scores in
a distribution tend to move closer to the mean (i.e.,
regress) in repeated testing.
Example:
if a group of subjects is recruited on the basis of extremely high
stress scores and an educational intervention is then implemented,
any improvement seen could be due partly, if not entirely, to
regression to the mean rather than to the coping techniques
presented in the educational program.
© 2015, Michael Kalsher
10
Experimental Mortality threat
Experimental mortality—also known as attrition,
withdrawals, or dropouts—is problematic when there
is a differential loss of subjects from comparison
groups subsequent to randomization, resulting in
unequal groups at the end of a study.
Example:
Suppose a researcher conducts a study to compare the effects of a
corticosteroid nasal spray with a saline nasal spray in alleviating
symptoms of allergic rhinitis (irritation and inflammation of the nasal
passages). If subjects with the most severe symptoms preferentially
drop out of the active treatment group, the treatment may appear
more effective than it really is.
© 2015, Michael Kalsher
11
Selection Interaction threats
A family of threats to internal validity produced
when a selection threat combines with one or
more of the other threats to internal validity.
When a selection threat is already present, other
threats can affect some experimental groups,
but not others.
Example:
If one group is dominated by members of one fraternity
(selection threat), and that fraternity has a party the night
before the experiment (history threat), the results may be
altered for that group.
© 2015, Michael Kalsher
12
Threats to External Validity:
Ways you might be wrong in making generalizations
People, Places, and Times
Demand Characteristics
Hawthorne Effects
Order Effects (or carryover effects)
© 2015, Michael Kalsher
13
People threat:
Are the results due to the unusual
type of people in the study?
Example:
You learn that the grant you submitted to assess average
drinking rates among college students in the U.S. has been
funded. In late November, you post an announcement
about the study on campus to get subjects for the study.
100 students sign up for the study. Of these, 78 are
members of campus fraternities; the other 22 are members
of the school’s football team.
© 2015, Michael Kalsher
14
Places threat:
Did the study work because of the
unusual place you did the study in?
Example:
Suppose that you conduct an “educational” study in a
college town with lots of high-achieving educationallyoriented kids.
© 2015, Michael Kalsher
15
Time threat:
Was the study conducted at a peculiar time?
Example:
Suppose that you conducted a smoking cessation study
the week after the U.S. Surgeon General issued the well
publicized results of the latest smoking and cancer studies.
In this instance, you might get different results than if you
had conducted the study the week before.
© 2015, Michael Kalsher
16
Demand Characteristics
Participants are often provided with cues to the
anticipated results of a study.
Example:
When asked a series of questions about depression, participants
may become wise to the hypothesis that certain treatments may
work better in treating mental illness than others. When participants
become wise to anticipated results (termed a placebo effect), they
may begin to exhibit performance that they believe is expected of
them.
Making sure that subjects are not aware of anticipated outcomes
(termed a blind study) reduces the possibility of this threat.
© 2015, Michael Kalsher
17
Hawthorne Effects
Similar to a placebo, research has found that the mere
presence of others watching a person’s performance causes
a change in their performance. If this change is significant,
can we be reasonably sure that it will also occur when no one
is watching?
Addressing this issue can be tricky but employing a control
group to measure the Hawthorne effect of those not receiving
any treatment can be very helpful. In this sense, the control
group is also being observed and will exhibit similar changes
in their behavior as the experimental group therefore negating
the Hawthorne effect.
© 2015, Michael Kalsher
18
Order Effects (carryover effects)
Order effects refer to the order in which treatment
is administered in a within-subjects/repeatedmeasures design and can be a major threat to
validity if multiple treatments are used, and no
attempt to counterbalance the order is made.
Example:
If subjects are given medication for two months, therapy for another
two months, and no treatment for another two months, it would be
possible, and even likely, that the level of depression would be least
after the final no treatment phase. Does this mean that no treatment
is better than the other two treatments? It could mean that the
benefits of the first two treatments have carried over to the last phase,
artificially elevating the no treatment success rates.
© 2015, Michael Kalsher
19