Variance in Research Design
Sources, threats to internal validity,
and “Noise”
Sources of Variance
• There are three “sources” of variance:
1) Primary Variance: the variability in the DV
that occurs as a result of (or is caused by) the
influence of the IV
2) Error Variance: unexplained variance.
Variability due to true chance happenings such
as moment-to-moment fluctuations in your
subject’s attention or fluctuations in your ability
to accurately measure your DV due to chance
variations in accuracy of equipment.
3) Secondary Variance: Variance in the DV
that occurs as a result of the influence of
secondary variables.
• As the experimenter, you want to maximize
primary variance, minimize error, and
control secondary variance.
• You cannot completely prevent error variance.
Some will occur.
• Secondary variance is where the researcher
has the best opportunity to improve the
chances of obtaining good internal and
external validity.
Problems caused by Secondary
Variance
• Secondary Variance can cause problems in two
ways:
1) If the secondary variable (2nd variable)
co-varies along with the Independent variable
(IV) then the secondary variable will create a
“threat to internal validity” or confound.
2) If secondary variables cause a lot of overall
variability in your DV then this may “mask” or
hide any effects of the IV. The 2nd variables
create “noise” in the data that make it harder to
detect an effect of your IV.
Problem #1:Threats to Internal Validity
(Confounds) page 314-317 9th ed
• In chapter 10 for ALL versions of text
• Five different threats to internal validity
(Campbell and Stanley, 1966)
Two threats resulting in
non-equivalent groups
• Occur in independent group (or between
subject) type designs.
• Example of threat #1 : The Effects of viewing
a violent stimulus on future aggressive
tendencies
• Selection: occurs when participants/subjects
in one level of the IV differ initially from
participants/subjects in another level of the IV,
due to systematic selection differences. This
is usually the result of the use of “intact
groups” or lack of random assignment of
subjects to groups. It is a “between-subject”
or independent group type issue.
Example of threat #2: Is punishment more
effective than reward?
• Mortality/attrition: occurs when you lose
subjects from one level of the IV more than
from some another level, systematically.
Differential loss of subjects from levels of
the IV.
• It is a “between-subject” or independent
group type issue.
Three threats that involve time-related
issues
• Occur in repeated-measures type designs
• Example of threat #3: Effect of a remedial
math course for entering freshmen on math
grades in college-level Math courses.
• Regression toward the mean (aka statistical
regression): extreme scores will become less
extreme with repeated measurement.
• Very low scores will regress upward toward
the mean and very high scores will fall down
toward the mean.
• Not every subject’s score will do this but on
average, in the long run, extreme scores will
become less extreme and thus they will move
toward the mean when subjects are remeasured.
• This is a potential problem in a repeatedmeasures type design.
• Example of threat #4: The effects of a
program on the benefits of alternative energy
sources on attitudes toward the use of
alternative energy sources.
• History: the occurrence of an EVENT other
than the treatment that produces changes in
the participants’ behavior. The event is not
under the researcher’s control.
• This is a repeated measures problem. The
longer the time between measurements, the
more likely it is that such an event will occur
and internal validity will be jeopardized.
• Example of threat #5: The effects of a preschool story-time program on adjustment to a
school environment.
• Maturation: Changes occur to your
participants (older, wiser, weaker, stronger
etc.) between measurements.
• These changes are not related to your IV.
Practice, boredom, and fatigue are considered
threats to internal validity of
maturation…even though these are short-term
changes.
• this is a potential issue in a repeatedmeasures type manipulation.
Controlling “Noise”
• Your chances of detecting a real effect of your
IV will be better if you reduce variability
caused by 2nd variables.
• Two ways in which you can reduce “noise” by
controlling secondary variables
1) Isolation: Insolate the research in a
“controlled” environment (laboratory).
Allows control over many environmental
variables (lighting, temperature, noise)
1) Holding constant: Hold some 2nd
“individual differences” variables constant to
eliminate variability due to 2nd variables
• Isolation and holding constant will eliminate
“noise” that might “mask” the effect of your
IV
• It will increase “internal validity” by
eliminating potential confounds
• But it will also reduce “external validity”
(your ability to take the results of your
research and make statements about other
populations, settings, and conditions)
• Researcher must find the balance between
controlling 2nd variables (to increase internal
validity but lower external validity) and
allowing 2nd variables to occur randomly
( to increase external validity but potentially
lower internal validity).