Experimental Research Design
Causality & Validity
• Threats to Validity
– Construct (particular to experiments)
– Internal
– External – already discussed
Threats to Construct Validity
• Mono Method bias
– Remove common method variance
– Establish convergent & discriminant
validity
• Hypothesis guessing
• Inappropriate level of construct
– Usually lower than what occurs in natural
setting
Threats to Internal Validity
• Time
– History, Maturation
• Measurement
– Testing, Instrumentation
• Selection
– Randomization, Statistical regression, Mortality
• Contamination
– Diffusion, Compensation
• Interaction among threats
Internal Validity
• Establish that cause led to the effect
– Identify non-experimental (uncontrolled)
factors that affect the dependent variable
or the relation b/w the independent and
dependent variables
10 Threats to Internal Validity
Time
• History
– Factors occurring along with the
independent variable
• An event occurring at the time of the
intervention that overshadows (or reduces the
effect of iv) and causes changes in dv
– E.g., change in manufacturing material along with
training influences performance quality
Copyright © 2003 John Wiley & Sons, Inc. Sekaran/RESEARCH 4E
FIGURE 7.1
10 Threats to Internal Validity
Time
• Maturation
– Biological or psychological processes or
factors that occur w/the passage of time &
affect the dependent variable
• E.g., job experience, socialization etc. affect
performance
Copyright © 2003 John Wiley & Sons, Inc. Sekaran/RESEARCH 4E
FIGURE 7.2
10 Threats to Internal Validity
Measurement
• Testing
– Studies w/pre-post measures or multiple
measures
• Increased or decreased sensitization to posttest measures
• Changes in participants due to measurement
itself
• Increased fatigue or resentment
• Self-consistency effects
10 Threats to Internal Validity
• Instrumentation
– Change in the measures from pre to post
test
• Change in observers’ frames of reference
• Changes in participants’ frames of reference
(e.g., beta & gamma changes p. 43 Saks)
• Change in instrumentation quality (e.g.,
performance quality measurement)
10 Threats to Internal Validity
• Instrumentation
– Use of different measurement techniques
• Establish expected convergent & discriminant
validity
• Importance of using different measures to
overcome percept-percept problem
Threats to Internal Validity
Selection
• Randomization
– Randomly assign to experimental
conditions
• Only in experimental designs
• Matching on key variable if random assignment
not possible
Threats to Internal Validity
Selection
• Randomization
– Randomly select from population
• Calculate response rate
• Establish comparability of respondents to
population on key features
• Stratified (v. cluster?) sampling
10 Threats to Internal Validity
Selection
• Statistical Regression
– Upon repeat measurement, scores usually
“regress” or “move toward” the mean
• Usually when extreme scorers are selected into
the study
– Can be classified as ‘error’ variation
10 Threats to Internal Validity
Selection
• Mortality
– Features of participants who do not
complete the study affect the results
– Reduced power in testing effects
10 Threats to Internal Validity
Contamination
• Diffusion/imitation of treatments
– When people in control groups
inadvertently know about or are affected
by intervention and that changes their
behavior
10 Threats to Internal Validity
Contamination
• Compensatory Effects
– Knowledge about intervention and
changes in control group members’
behavior is more purposeful
10 Threats to Internal Validity
• Interaction among threats
– When multiple threats operate at the same
time obscuring the effect of the IV on the
DV
Threats to Internal Validity
• Time-related
– History, maturation
• Measurement
– Testing, Instrumentation
• Selection
– Randomization, Statistical regression, mortality
• Contamination
– Diffusion, Compensation
• Interaction among threats