Educational Research
250:205
Writing Chapter 3 – Part II
Chapters 9 and 13: Procedures and Experimental Design
I.
A Systematic Process for Writing the Methods Chapter:
Subjects.
Instrumentation.
Procedures.
Experimental Design.
Statistical Analysis.
II.
Procedures:
A. Describes how the data was collected:
1. This section of the procedures is the most detailed section.
a. In general, the following areas should be included:
A.
B.
C.
D.
E.
b. The importance of pilot testing:
B. Procedures section should be used in descriptive, associational and
intervention research.
III.
Experimental Design:
A. What is it?
B. Validity:
1. As you may already know, the term validity is used in a variety of
contexts in research. You have already learned about instrument
validity:
a. Instrument validity: Does your instrument measure what it was
intended to measure.
2. There are two other types that are based on the experiment itself as
opposed to just the instrument:
a. Internal validity:
b. External validity:
3. There is a tension between internal and external validity:
C. Internal Validity: Minimizing threats via experimental design.
1. There are many threats to the internal validity of your experiment and
your weapons against these threats are “random sampling, sound
procedures and sound experimental design.”
a. The threats to internal validity include:
i.
Subject characteristics:

Examples:

How can you control for subject characteristics:
ii.
Mortality:

Example:

How can you control for mortality?
iii.
Location:

iv.
How can you control for location:
Instrumentation Decay:

Examples:

How can you control for instrumentation?
v.
Data Collector Characteristics:

Example:

How can you control for data collector characteristics?
vi.
Researcher bias:

vii.
How to prevent researcher bias?
Multiple testing:

Example:

How can you control for multiple testing?
viii.
History:

Example:

How can you control for history?
ix.
Maturation:

Example:

How can you control for maturation?
x.
Sampling methodology:

Example:

How can you control for sampling methodology?
xi.
Experimental treatment diffusion:

Example:

How do you control for experimental treatment diffusion?
xii.
Compensatory rivalry:

Example:

xiii.
How do you control for compensatory rivalry?
Resentful demoralization:

Example:

How do you control for resentful demoralization?

Hawthorne effect:
How to prevent this threat to internal validity?
xiv.
D. External Validity: Minimizing threats.
1. External validity is threatened as researcher attempts to increase the
control of the experiment. This is not necessarily bad.
E. Experimental Designs:
1. Experimental design describes how the researcher manipulates
treatments and observes the effects.
2. Experimental design is the most powerful tool for minimizing threats to
internal validity.
3. Terminology:
a. Independent variable:
b. Dependent variable:
4. In other words, the researcher is trying to determine the effect of the
“independent variable/treatment on the dependent
variable/measurement.”
5. There are many different types of experimental designs:
a. Single-Group Designs:
i.
One-Shot Case Study:
ii.
One-Group PreTest Posttest Design:
b. Quasi-Experimental Designs: Almost like true experiments (both
contain control groups) however the subjects in the study have not
been chosen at random. Groups are typically intact (convenience
samples). These designs are typical in the classroom.
i.
Quasi version of the Single-Shot Case Study: Static-Group
Comparison Design:
ii.
Quasi version of the One-Group Pretest Posttest Design:
Static Group Pretest-Posttest Design:
c. Experimental Designs: The next step up from the QuasiExperimental Design. Involves randomization of subjects which is
harder than you think.
i.
Randomized Posttest-Only Control Group Design:
ii.
Randomized Pretest Posttest Control Group Design:
d. Common Extensions to Research Designs:
6. Other experimental design solutions for educational research:
a. Education researchers must often perform research on intact
groups (classes) and are not able to use control groups (everyone
wants the treatment). Here are some other designs that are used
in educational research:
i.
Matching-Only Design: Used when randomization is
impossible.
ii.
Counterbalanced Design: Used when everyone wants the
treatment.
iii.
Time Series Design: Used when a control group is
impossible.