Types of Research and Designs
• This week and next week…
• Covering
– Research classifications
– Variables
– Steps in Experimental Research
– Validity
– Research Designs
– Common Sources of Error
VALIDITY
VALIDITY
• In Step 4, the validity of measurement for
the DVs comes into question
• There are two (2) important classification
of validity in experimental research:
– Internal validity
– External validity
Internal Validity
• How valid the findings are within, or
internal to, the study.
• Did the treatments cause the outcome to
occur, or did other extraneous factors
cause the outcome?
Internal Validity
•
Eight (8) factors that threat the internal validity
of a study:
1.
2.
3.
4.
5.
6.
7.
8.
History
Maturation
Testing
Instrumentation
Statistical regression
Selection
Experimental mortality
Interaction of Selection and Maturation or History
History
• Specific things that happen while
conducting the research study that affect
the final scores of the participants in
addition to the effect of the experimental
treatment.
– The effects of linear vs. nonlinear
periodization models for in-season volleyball
players
– Resistance training vs. aerobic training for
physical fitness in middle-aged men.
Maturation
• Subjects grow older during the course of
an experiment – this may affect the
outcome of the study
– The time-course for training-induced
increases in strength for adolescents during a
25-week resistance training program
– The effects of 12 weeks of aerobic training on
VO2max (from September to December)
Testing
• The “learning” effect
– Subjects get better at scoring well on a test
the more times they take it.
• The effects of 2 days of resistance training on
muscle strength and power output.
Instrumentation
• Changes in the adjustment or calibration
of the measuring equipment or use of
different standards among scorers may
cause a difference in outcome scores.
Statistical Regression
• The tendency for groups with extremely
high or low scores on one measure to
score closer to the mean score of the
population on a second measure.
– Can be mistaken for a treatment effect
– Example: Low-IQ and high-IQ groups are
formed. A treatment is applied to both groups
for twelve weeks. A physical ability test is
administered to both groups. It is found that
both groups are equal in physical ability.
Selection
• The selection of subjects and their
assignments to groups can bias the
outcome of a study.
– Subjects that are not representative of the
population
• Kinesiology majors vs. all college-aged males and
females
– Groups that are not equal in their abilities
prior to the experiment
• VO2max and muscle strength
Experimental Mortality
• Subject attrition
– Excessive loss of subjects throughout the
course of a study
– Sample of subjects is no longer
representative of the population
Interaction of Selection and
Maturation or History
• The maturation effect or history effect is
not the same for all groups selected for the
research study – may affect the outcome.
– Groups that differ in age or background may
respond differently to the same treatment
– If a group starts out unequal in ability due to
maturation or history, they may not all have
the same potential for improvement or
change.
– Disproportional experimental mortality
External Validity
• The degree to which findings in a research study
can be inferred or generalized to other
populations, settings, or experimental
treatments.
• Can the findings of a particular study be inferred
to the entire population?
• Are the findings unique to the participants of the
study, or do they apply to other groups?
– Examples: A study using prison inmates. Good
internal validity.
External Validity
•
Four threats to external validity
1. Interaction effect of testing
2. Interaction effects of selection bias and
experimental treatment
3. Reactive effects of experimental setting
4. Multiple-treatment interference
Interaction Effect of Testing
• This effect occurs when the pre-test itself
changes the group’s response to the
treatment.
– Example: A pre-test may identify deficiencies
that the subject may try to correct during the
course of the treatement
Interaction Effects of Selection Bias
and Experimental Treatment
• The participant or groups selected in a
biased manner respond to the
experimental treatment in a unique way so
they are not representative of any
population.
– Novice vs. intermediate vs. elite weight lifters
– This threat generally occurs as a result of a
convenient group is tested and an attempt is
made at generalizing the results
Reactive Effects of Experimental
Settings
• The experimental setting is not the same
as a subject’s ambient environment – this
may affect the study outcome.
– Subjects may respond differently around
investigators
– Laboratory vs. natural settings
– Subjects may think that a particular treatment
is supposed to do “something” and respond
like they are “supposed to.”
Multiple-Treatment Interference
• Multiple treatments administered at the
same time on the same group of subjects.
– Subject involved in two studies at the same
time
– Competing influences of treatments
Control of Variables
• Control of all variables operating in an
experimental study is highly desirable, but
seldom (if ever) accomplished.
– Sleep, food, exercise, etc.
• Too much control can harm external validity
• Not enough control can harm internal validity
Validity in Summary
• In many cases, internal and external
validity cannot be obtained.
– Must decide which is most important
• Good internal validity is necessary for
external validity
– Internal validity is the basic minimum (LCD)
for an experimental design
• Investigators must consider the threats to
validity when designing research studies.