Chapter 18
Experimental and Quasi-Experimental Research
Research Methods in Physical Activity
Experimental research
Experimental research attempts to establish cause-and-effect
relationships. Three criteria must be present to establish cause and
effect:
1. The cause must precede the effect in time. For example, the starting
gun in a race precedes the runners’ beginning the race; the runners’ beginning does
not cause the starting gun to go off.
2. The cause and effect must be correlated with each other. Just
because two variables are correlated does not mean one causes the
other; cause and effect, however, cannot exist unless two variables
are correlated.
3. The correlation between cause and effect cannot be explained by
another variable. Recall that the relationship between the academic
performance of elementary children and shoe size was explained by a third
variable, age.
Research Methods in Physical Activity
Experimental research
Cause and effect in must be defined terms of necessary and
sufficient conditions. For example, if the condition is necessary and sufficient
to produce the effect, then it is the cause.
Alternative situations exist as well:
♦ Necessary but not sufficient: Some related condition likely
produces the effect.
♦ Sufficient but not necessary: Some alternative condition is likely
the cause.
♦ Neither necessary nor sufficient: Some contributing condition is
likely the cause.
Research Methods in Physical Activity
Experimental Research
Also, remember that statistics do not establish cause and effect.
Statistical techniques can only reject the null hypothesis (establish that
groups are significantly different) and identify the percentage of
variance in the dependent variable accounted for by the independent
variable or the effect size.
Neither of these procedures establishes cause and effect (they are
necessary but not sufficient).
Cause and effect can be established only by the application of logical
thinking to well-designed experiments.
This logical process establishes that no other reasonable
explanation exists (reject Null for alternate hypothesis) for the
changes in the dependent variable except the manipulation of the
independent variable.
Research Methods in Physical Activity
Experimental Research
The logical process of establishing changes in the dependent variable
require:
♦ The selection of a good theoretical framework
♦ The use of appropriate participants
♦ The application of an appropriate experimental design
♦ The proper selection and control of the independent variable
(treatment)
♦ The appropriate selection and measurement of the dependent
variable
♦ The use of the correct statistical model and analysis
♦ The correct interpretation of the results
Thus, the above are the methods required to establish valid findings
allowing the researcher to establish cause and effect.
Research Methods in Physical Activity
Sources of Invalidity
Both internal validity and external validity are important in experiments, but
they are frequently at odds in research planning and design.
Gaining internal validity involves controlling all variables so that the
researcher can eliminate all rival hypotheses as explanations for the
outcomes observed.
When controlling and constraining the research setting to gain internal
validity, the researcher places the generalization (external validity) of
the findings in jeopardy.
Thus, the researcher must decide whether it is more important to be certain
that the manipulation of the independent variable caused the observed
changes in the dependent variable or more important to be able to
generalize the results to other populations, settings, etc.
Research Methods in Physical Activity
Threats to Internal Validity in Research Design (see
examples of each pp. 332-334)
♦ History: events occurring during the experiment that are not
part of the treatment
♦ Maturation: processes within the participants that operate as
a result of time passing (e.g., aging, fatigue, hunger)
♦ Testing: the effects of one test on subsequent administrations
of the same test
♦ Instrumentation: changes in instrument calibration, including
lack of agreement within and between observers
♦ Statistical regression: the fact that groups selected based on
extreme scores are not as extreme on subsequent testing
♦ Selection bias: choosing comparison groups in a nonrandom
manner
♦ Experimental mortality: loss of participants from
comparison groups for nonrandom reasons.
Research Methods in Physical Activity
Threats to Internal Validity in Research Design (see
examples of each pp. 332-334)
♦ Selection–maturation interaction: the passage of time
affecting one group but not the other in nonequivalent group
designs
♦ Expectancy: experimenters’ or testers’ anticipating that
certain participants will perform better
Research Methods in Physical Activity
Controlling Threats to Internal Validity in Research
Design
Many threats to internal validity are controlled by making the
participants in the experimental and control groups as alike as
possible. This objective is most often accomplished by randomly
assigning participants to groups.
Random Assignment
1) Randomization allows the assumption that the groups do not
differ at the beginning of the experiment.
1) The randomization process controls for history up to the
point of the experiment; that is, the researcher can assume
that past events are equally distributed among groups
Research Methods in Physical Activity
Controlling Threats to Internal Validity in Research
Design
Random Assignment
3) Randomization also controls for maturation because the
passage of time is equivalent in all groups.
4) Statistical regression, selection biases, and selection–maturation
interaction are controlled because they occur only when
groups are not randomly formed.
Research Methods in Physical Activity
Controlling Threats to Internal Validity in Research
Design
Random Assignment
Ways other than random assignment of participants to groups are
sometimes used to control threats to internal validity. These
Include the matched-pair technique, and within-subjects designs.
Matched-Pair Technique - The matched-pair technique matches
pairs of participants who are equal on some characteristic and
then randomly assigns each to a different group.
Within-subjects designs - The participants are used as their own
controls. This means that each participant receives both the
experimental and the control treatment. In this type of design, the
order of treatments should be counterbalanced; that is, half the
participants should receive the experimental treatment first and
then the control, and the other half should receive the control
treatment first and then the experimental treatment.
Research Methods in Physical Activity
Controlling Threats to Internal Validity in Research
Design
Placebos, Blind Setups, and Double-Blind Setups
Other ways of controlling threats to internal validity include
placebos and blind and double-blind setups.
Placebo - Method of controlling a threat to internal validity in
which a control group receives a false treatment while the
experimental group receives the real treatment.
Blind setup - Method of controlling a threat to internal validity in
which the participant does not know whether he or she is
receiving the experimental or control treatment.
Double-blind setup - Method of controlling a threat to internal
validity in which neither the participant nor the experimenter
knows which treatment the participant is receiving.
Research Methods in Physical Activity
Uncontrolled Threats to Internal Validity in Research
Design
Three Threats to internal validity remained uncontrolled by the
randomization process : 1) Reactive or Interactive Effects; 2)
Instrumentation; and, 3) Experimental Mortality.
1) Reactive or interactive effects of testing can be controlled
only by eliminating the pretest.
2) Instrumentation problems cannot be controlled or evaluated
by any design. Only the experimenter can control this threat
to internal validity.
3) Experimental mortality is not controlled by any type of
experimental design. The experimenter can control this
threat only by ensuring that participants are not lost (at all, if
possible) from groups.
Research Methods in Physical Activity
Threats to External Validity in Research Design (see
examples of each pp. 334-336)
Threats to external validity, or the ability to generalize results to
other participants, settings, measures, etc:
♦ Reactive or interactive effects of testing : The pretest may
make the participant more aware of or sensitive to the upcoming
treatment. As a result, the treatment is not as effective without the
pretest.
♦ Interaction of selection bias and the experimental
treatment: When a group is selected on some characteristic, the
treatment may work only on groups possessing that characteristic.
♦ Reactive effects of experimental arrangements:
Treatments that are effective in constrained situations (e.g.,
laboratories) may not be effective in less constrained settings
(more like the real world).
♦ Multiple-treatment interference: When participants receive
more than one treatment, the effects of previous treatments may
influence subsequent ones
Research Methods in Physical Activity
Controlling Threats to External Validity in Research
Design
External validity is generally controlled by selecting the
participants, treatments, experimental situation, and tests to
represent some larger population.
The results of a particular treatment can be generalized to a
larger group if the sample is representative, this generalization may
apply only to the specific situation in the experiment. If the
experiment is conducted under controlled laboratory conditions,
then the findings may apply only under controlled laboratory
conditions.
Does the study have enough characteristics of real-world
settings that participants respond as if they were in the real
world ? (ie. Is ecological validity present ? )
Research Methods in Physical Activity
Types of Research Designs
Types of research designs are divided into three categories:
Pre - experimental designs,
True experimental designs, and,
Quasi-experimental designs. (not reviewed in this lecture)
With the following notation:
♦ Each line indicates a group of participants.
♦ R signifies random assignment of participants to groups.
♦ O signifies an observation or a test.
♦ T signifies that a treatment is applied; a blank space in a line where
a T appears on another line means that the group is a
control.
♦ A dotted line between groups means that the groups are used
intact rather than being randomly formed..
Research Methods in Physical Activity
Types of Research Designs
Notations (cont) :
♦ Subscripts (ex. O1, T2 ) indicate either the order of observations
and treatments (when they appear on the same line) or observations
of different groups or different treatments (when they appear on
different lines). For example, when the terms T1 and T2 appear on
different lines, they refer to different treatments; when they appear
on the same line, they mean that the treatment is administered more
than once to the same group.
Research Methods in Physical Activity
Types of Research Designs
Pre - experimental Designs
Pre-experimental design - One of three types of research design that
control few of the sources of invalidity and that do not have random
assignments of participants to groups: one-shot study, one-group
pretest - posttest design, and static group comparison.
One-shot study
In a one-shot study design, a group of participants receives a
treatment followed by a test to evaluate the treatment:
T
O
This design fails all the tests of good research. All that can be said is
that at a certain point in time this group of participants performed at
a certain level. In no way can the level of performance (O) be
attributed to the treatment (T).
Research Methods in Physical Activity
Types of Research Designs
Pre - experimental Designs
Pretest - posttest design
The one-group pretest–posttest design, although very weak, is better
than the one-shot design. At least we can observe whether any
change in performance has occurred:
O1 T O2
If O2 is better than O1, we can say that the participants improved.
Complications with this design :
1) The improvement could be due to the treatment, but it could also
be due to history. Some event other than the treatment (T) may
have occurred between the pretest (O1) and the posttest (O2);
Research Methods in Physical Activity
Types of Research Designs
Pretest - posttest design
Complications with this design (cont) :
2) Some event other than the treatment (T) may have occurred
between the pretest (O1) and the posttest (O2); (ex. the participants
may have exercised at home on the other days. )
3) Maturation is a rival hypothesis. The participants may have gotten
better (or worse) just as a result of the passage of time.
4) Testing is a rival hypothesis; the increase at O2 may be the result
only of experience with the test at O1.
5) If the group being tested is selected for some specific reason, then
any of the threats involving selection bias can occur.
Research Methods in Physical Activity
Types of Research Designs
Pre - experimental Designs
Static Group Comparison
The static group-comparison design compares two groups, one of
which receives the treatment and one of which does not:
T
O2
O1
Complications with this design :
1) We do not know whether the groups were not equivalent when
the study began, as indicated by the dotted line between the groups.
This means that the groups were selected intact rather than being
randomly formed. Therefore, we are unable to determine whether
any differences between O1 and O2 are because of T or only because
the groups differed initially.
Research Methods in Physical Activity
Types of Research Designs
Static Group Comparison
Complications with this design (cont) :
2) This design is subject to invalidity because of selection biases and
the selection–maturation interaction.
Research Methods in Physical Activity
Types of Research Designs
True Experimental Designs
True experimental design - Any design used in experimental research
in which groups are randomly formed and that controls most
sources of invalidity.
This controls for past (but not present) history, maturation (which
should occur equally in the groups), testing, and all sources of
invalidity that are based on nonequivalence of groups (statistical
regression, selection biases, and selection–maturation interaction).
Only the experimenter can make sure that nothing happens to one
group (besides the treatment) and not the other (present history),
that scores on the dependent measure do not vary as a result of
instrumentation problems, and that the loss of participants is not
different between the groups (experimental mortality).
Research Methods in Physical Activity
Types of Research Designs
True Experimental Designs
True experimental designs include:
1) Randomized–Groups Design
2) Pretest–Posttest Randomized–Groups Design, and
3) Solomon Four-Group Design
Research Methods in Physical Activity
Types of Research Designs
True Experimental Designs
Randomized–Groups Design
The randomized-groups design resembles static group comparison
except that groups are randomly formed:
R
R
T
O1
O2
If the researcher controls the threats to internal validity that are not
controlled by randomization, and has a sound theoretical basis
for the study, and meets the necessary - and-sufficient rule, then
this design allows the conclusion that significant differences
between O1 and O2 are due to T.
An independent t test is used to analyze the difference between O1 and
O 2.
Research Methods in Physical Activity
Types of Research Designs
True Experimental Designs
Randomized–Groups Design (cont)
This design can be extended to any number of levels of an
independent variable:
R
R
R
T1
T2
O1
O2
O3
Here, three levels of the independent variable exist, where one is the
control and T1 and T2 represent two levels of treatment.
This design can be analyzed by simple ANOVA, which contrasts the
dependent variable as measured in the three groups (O1, O2, O3).
Research Methods in Physical Activity
Types of Research Designs
True Experimental Designs
Pretest–Posttest Randomized–Groups Design
In the pretest–posttest randomized-groups design, the groups are
randomly formed, but both groups are given a pretest as well as
a posttest:
R
R
O1
O3
T
O2
O4
The major purpose of this type of design is to determine the amount
of change produced by the treatment; that is, does the
experimental group change more than the control group?
Research Methods in Physical Activity
Types of Research Designs
True Experimental Designs
Pretest–Posttest Randomized–Groups Design (cont)
This design threatens the internal validity through testing, but the
threat is controlled because the comparison of O3 to O4 in the
control group and the comparison of O1 to O2 in the
experimental group includes the testing effect. Thus, although
the testing effect cannot be evaluated in this design, it is
controlled.
This design can (best) be analyzed by repeated measures ANOVA.
Research Methods in Physical Activity
Types of Research Designs
True Experimental Designs
Solomon Four-Group Design
The Solomon four-group design is the only true experimental design to
specifically evaluate one of the threats to external validity: reactive or
interactive effects of testing. The design is depicted as follows:
R
O1
T
O2
R
O3
O4
R
T
O5
R
O6
This approach combines the randomized-groups and the pretest–posttest
randomized-groups designs. The purpose is explicitly to determine whether
the pretest results in increased sensitivity of the participants to the
treatment.
Research Methods in Physical Activity
Types of Research Designs
Solomon Four-Group Design (cont)
R
R
R
R
O1
O3
T
T
O2
O4
O5
O6
This design allows:
1) a replication of the treatment effect (is O2 > O4? and is O5 >
O6?),
2) an assessment of the amount of change due to the treatment (is
O2 – O1 > O4 – O3?),
3) an evaluation of the testing effect (is O4 > O6?), and,
4) an assessment of whether the pretest interacts with the treatment
(is O2 > O5?).
Research Methods in Physical Activity
Types of Research Designs
Solomon Four-Group Design (cont)
In addition, no good way exists to analyze this design statistically. The best
alternative (one that does not use all the data) is a 2 X 2 ANOVA set up as
follows:
Pretested
Unpretested
No T
T
O4
O6
O2
O5
Thus, independent variable 1 has two levels (pretested and not pretested), and
independent variable 2 has two levels (treatment and no treatment).
In the ANOVA, the F ratio for independent variable 1 establishes the effects of
pretesting, the F for independent variable 2 establishes the effects of the
treatment, and the F for interaction evaluates the external validity threat of
interaction of the pretest with the treatment.
End of Lecture
Research Methods in Physical Activity