Chapter 6
Research Validity
Research Validity:
Truthfulness of inferences made from a
research study.
Statistical Conclusion Validity:
Valid statements of the
co-variation (relationship)
between the Independent
and Dependant Variable.
Video
e.g., Heterogeneous results – present
atypical cases.
Statistical Conclusion Validity:
Using Statistics as evidence for the
null hypothesis. Failing to reject the
null does not mean there is no
difference between conditions anymore
than failing to convict a defendant
means that they are innocent. The
study may not have had enough
POWER to find differences.
Construct Validity
Is the construct adequately represented by
the measures used in the research study?
Participant Reactivity
Participant Motivations and tendencies.
Demand Characteristics
Any cues in the experiment (e.g., instructions,
rumors, others behaviors that influence the
responses of the participants.
e.g., Yawn Study
Subliminal Perception study
- Positive self presentation
- “screw you” attitude!
A. Experimenter Effects
Actions and characteristics of researcher
that influence the responses of participants.
B. Experimenter Expectancies
• Recording Errors
• Interpretation Biases
• Unintentionally treating one group
differently than another.
C. Experimenter Attributes
• Biosocial (age, sex, race etc.)
Controlling for
Demand Characteristics
Can’t get rid of demand characteristics.
• Withhold the hypotheses until after the study.
• Distract participants from real variables.
e.g., asked participants to rate the
commercials for emotions etc.
• Do not tell participant which condition they are
in.
BE SURE DEMAND CHARACTERISTS DO
NOT VARY ACROSS CONDITIONS!!
Internal Validity - The extent to which all
explanations for changes in the DV between
conditions have been eliminated -- other than
the IV.
ie(7a)
12
Extraneous variable - any variable other than
IV that effects DV.
Control
Extraneous
Variable
Experimental
ie(7a)
13
Control
Confound
Experimental
Confounding Extraneous Variable - any
extraneous variable that affects one condition
differently than it affects other conditions.
ie(7a)
14
Between Subjects Designs
-Two (or more) Groups of Participants
compared to each other. Each group has a
different level of the IV.
Experimental (Treatment) Group
Control Group
ie(7a)
15
Major Potential Confound: Individual
Differences (I.e., perhaps the participants in
one group are not comparable in many ways
to participants in the other group).
ie(7a)
16
Within Subjects Design
One Group of participants measured under more
than one condition of the IV.
Experimental Condition
Control Condition
Potential Confounds
Since subjects cannot be in more than one
condition at one time, anything that is not the
same at each of the times of measurement (other
than the IV) is a potential confound.
ie(7a)
17
Pre-Post Test Designs (type of WS design)
Pre-test - DV measured before Treatment
- serves as baseline control condition
Post-test - DV measured after Treatment
ie(7a)
18
Other than the treatment (IV) what could cause
difference in DV between conditions (I.e., what
possible confounds could there be?)
Confounds with Pre-Post Designs
History - any changes that occurred between
Pre and Post Test other than IV.
ie(7a)
19
Maturation - biological/psychological changes
between pre-post test.
ie(7a)
20
Instrumentation - changes in measurement
device or operational definition between pre
and post test.
ie(7a)
21
Are rates of Diabetes really
increasing, or are we simply
diagnosing more cases???
Attention Deficit Disorder?
DUIs?
ie(7a)
22
Testing : Changes is a person’s score for
the post conditions that results from
having been tested in the pre-test.
- Sensitization
- Boredom
- Practice
ie(7a)
23
Attrition - subjects dropping out of the study.
ANYTHING that is a difference between the
before and after condition other than the IV is
a confound.
ie(7a)
24
Hokey Pokey
ie(7a)
25
Artifact - Effect caused by the procedure
rather than by the IV.
Statistical Regression Artifact.
- problem when subjects are assigned to groups
based on the pre-test scores.
- group scores will be pulled towards the mean
of the DV.
- Pre-test high scorers will appear to do poorer.
Pre-test low scorers will appear to do better.
- problem with unreliable measures
ie(7a)
26
Not a problem if the DV is very reliable.
ie(7a)
27
Internal Validity?
Confounds (in pre and post designs)
•History
•Maturation
•Instrumentation
•Testing
•Attrition (Mortality)
•Statistical Regression Artifact*
ie(7a)
28
Controlling for these confounds.
Use a pre-post control group.
Treated the same as treatment Group except
IV is not manipulated between tests.
Control group affected by same, history,
Maturation and Regression effects. Any
difference between the treatment and control
groups are not due to confounds.
ie(7a)
29
Experimental Group
Pre-test Treatment
plus confounds
Control Group
Pre-test
confounds
ie(7a)
Post-test
Post-test
30
Selection – confound due to assignment of
subjects to the Control and Treatment Groups.
Compare Pre-test scores. Are they the same to
begin with?
But could be a difference that interacts with the
treatment.
- use random assignment!!!!
ie(7a)
31
Selection Confound
-subjects assigned to treatment and control group
biased on a criteria (bias).
Selection X (interactions)
Any of the five confounds effect the control group
differently than the treatment group.
i.e., Alcohol treatment study
- compare volunteers to non-volunteers
e.g., Differential History
ie(7a)
32
Selection X (interactions)
Any of the five confounds effect the control group
differently than the treatment group.
Differential History
Differential Attrition
i.e., Alcohol treatment study
- compare volunteers to non-volunteers
ie(7a)
33
Differential Attrition
Weight loss Study
(Diet and Exercise condition)
No-Treatment Control Group
Type of people that drop put of the study might
depend on which study they are in.
ie(7a)
34
How can we ensure the subjects in each condition
are comparable?
Random Assignment
Extraneous variables still effect DV, but it
should not be a confound.
ie(7a)
35
Pre-test
Post test
Treatment
Control
Treatment group = size of treatment effect
plus confounds
Control group = estimate of the size of prepost confounds.
ie(7a)
36
Pre-test
Post test
Treatment
Control
Is the pre-post test change significantly greater
in the Treatment Condition than in the control
condition?
ie(7a)
37
External Validity
Goal of Psychology: to determine the
Underlying “laws” of behavior.
External Validity – the extent to which the
results of an experiment can be applied to and
across different persons, settings, and times.
(ie14)
38
Population Validity – ability to generalize
results from the sample to a larger population.
Random Selection – from a population should
insure generalization to the experimentally
accessible population.
(ie14)
39
Target Population - larger population to which
results are generalized.
(ie14)
40
ALL PERSONS WITH DEPRESSION
Persons Diagnosed
Willing
Control
Studied
(ie14)
41
Ecological Validity: extent to which results can
be generalized across settings or environmental
conditions.
(ie14)
42
Temporal Validity – generalizability across
time.
Seasonal Variation
- fixed-time variation
- variable time variation
Cyclical Variation
Personological Variation
e.g., are you planning a mid-life
crisis?
(ie14)
43
Treatment Variation Validity
- Degree the results of a study can be generalized
across variations in the treatment
Outcome validity
- Degree to which the results of a study
generalize across related dependant variables.
(ie14)
44
Relationship between Internal & External
Validity
To obtain high Internal Validity we often create
very artificial study settings.
To establish external validity we often need to
determine if the finding applies across setting,
locations, cultures etc.
(ie14)
45