Research Design and Validity Threats

advertisement
Research Design and Validity
Threats
Evaluation Designs
• Quantitative versus Qualitative
• Combination
Categories of Research Designs
• Nonexperimental or Pre-experimental designs
• One group, little validity control
• Quasi-experimental
• Experimental and comparison group, but no random
assignment or selection
• Experimental
• Random assignment of experimental and control groups
Terminology
• Internal validity
• Extent to which an observed outcome can be
attributed to a planned intervention
• External validity
• Extent to which an observed outcome can be
attributed to a replicable intervention and
generalized to other settings and populations
Internal validity threats
• History
• An event that occurs during the intervention that
could have an impact on the results
• Maturation
• Bias from biological, natural, or social events that
can bias results
Internal validity threats
• Testing
• Testing might cue a person in to change behavior,
regardless of the program
• Instrumentation
• Bias in data collection instruments
Internal validity threats
• Statistical Regression
• Bias from selecting a group with unusually high
or low scores on something
• Selection
• Comparison groups are unequal
Internal validity threats
• Attrition/subject mortality
• Dropouts of subjects; if there is more than one
group, then unequal dropouts between groups
• Interactive effects
• Combinations of the above
Other internal validity issues
• Diffusion
• Contamination of comparison condition or
intervention condition
• Demoralization
• Subjects upset they are not receiving the other
condition
External Validity
• Threats
•
•
•
•
•
Social desirability
Expectancy effect
Hawthorne effect
Placebo effect
Novelty effect
Research Designs
• Key to abbreviations:
O = data collection
X = treatment/intervention
R = random assignment
Solid line separating groups – equal groups
Dashed line separating groups – unequal groups
Pre-experimental designs
• One group, pretest, post-test
O X O
• Good for pilot testing
• Does not control for IV threats
• Be certain to use valid and reliable instruments
Pre-experimental designs
• One shot case study
X O
• No control for validity threats, no pretest
measures
Quasi-experimental designs
• Nonequivalent comparison group
O X O
----O O
• A comparison group is added, but they are not
equal
Quasi-experimental designs
• Time series
O O O O X O O O O
• Several measures to assess if there is a trend
• No comparison group
Quasi-experimental designs
• Multiple Time series
O O O O X O O O O
-----------------O O O O O O O O
• Added a comparison group
Experimental Designs
• Pre-test, post-test, control group design
R O X O
R O O
• Randomly assigned to groups
Experimental Designs
• post-test only, control group design
R X O
R
O
• Randomly assigned to groups
Experimental Designs
• Solomon Four Group Design
R O X O
R O O
R X O
R
O
Why do you care?
• Having this knowledge will help you
determine the quality of research studies,
which will impact your conclusions regarding
the results.
Download
Related flashcards
Estimation theory

11 Cards

Econometricians

55 Cards

Create flashcards