Research Design and Validity
Threats
Evaluation Designs
• Quantitative versus Qualitative
• Combination
Categories of Research Designs
• Nonexperimental or Pre-experimental designs
• One group, little validity control
• Quasi-experimental
• Experimental and comparison group, but no random
assignment or selection
• Experimental
• Random assignment of experimental and control groups
Terminology
• Internal validity
• Extent to which an observed outcome can be
attributed to a planned intervention
• External validity
• Extent to which an observed outcome can be
attributed to a replicable intervention and
generalized to other settings and populations
Internal validity threats
• History
• An event that occurs during the intervention that
could have an impact on the results
• Maturation
• Bias from biological, natural, or social events that
can bias results
Internal validity threats
• Testing
• Testing might cue a person in to change behavior,
regardless of the program
• Instrumentation
• Bias in data collection instruments
Internal validity threats
• Statistical Regression
• Bias from selecting a group with unusually high
or low scores on something
• Selection
• Comparison groups are unequal
Internal validity threats
• Attrition/subject mortality
• Dropouts of subjects; if there is more than one
group, then unequal dropouts between groups
• Interactive effects
• Combinations of the above
Other internal validity issues
• Diffusion
• Contamination of comparison condition or
intervention condition
• Demoralization
• Subjects upset they are not receiving the other
condition
External Validity
• Threats
•
•
•
•
•
Social desirability
Expectancy effect
Hawthorne effect
Placebo effect
Novelty effect
Research Designs
• Key to abbreviations:
O = data collection
X = treatment/intervention
R = random assignment
Solid line separating groups – equal groups
Dashed line separating groups – unequal groups
Pre-experimental designs
• One group, pretest, post-test
O X O
• Good for pilot testing
• Does not control for IV threats
• Be certain to use valid and reliable instruments
Pre-experimental designs
• One shot case study
X O
• No control for validity threats, no pretest
measures
Quasi-experimental designs
• Nonequivalent comparison group
O X O
----O O
• A comparison group is added, but they are not
equal
Quasi-experimental designs
• Time series
O O O O X O O O O
• Several measures to assess if there is a trend
• No comparison group
Quasi-experimental designs
• Multiple Time series
O O O O X O O O O
-----------------O O O O O O O O
• Added a comparison group
Experimental Designs
• Pre-test, post-test, control group design
R O X O
R O O
• Randomly assigned to groups
Experimental Designs
• post-test only, control group design
R X O
R
O
• Randomly assigned to groups
Experimental Designs
• Solomon Four Group Design
R O X O
R O O
R X O
R
O
Why do you care?
• Having this knowledge will help you
determine the quality of research studies,
which will impact your conclusions regarding
the results.