Educational Research:
Experimental Studies
EDU 8603
Educational Research
Richard M. Jacobs, OSA, Ph.D.
Research...

The systematic application of a
family of methods employed to
provide trustworthy information
about problems
…an ongoing process based on many
accumulated understandings and
explanations that, when taken together
lead to generalizations about problems
and the development of theories
The basic steps of research...
Scientific and disciplined inquiry is an
orderly process, involving:
 recognition and identification of a
topic to be studied (“problem”)
 description and execution of
procedures to collection information
(“method”)
 objective data analysis
 statement of findings (“results”)
Research methods...
Quantitative…
…collects and analyzes numerical data
obtained from formal instruments
Quantitative methods...
descriptive research (“survey research”)
 correlational research
 causal-comparative research
(“ex post facto research”)
 experimental research


experimental research
…the researcher selects participants
and divides them into two or more
groups having similar characteristics
and, then, applies the treatment(s) to
the groups and measures the effects
upon the groups
Conducting an experimental study…
1. select the problem
2. select participants and instrument
3. selection and execution of a
research plan
4. data analysis and formulation of
conclusions
Types of experimental comparison…
1. comparison of two different approaches
(A versus B)
2. comparison of an existing approach
to a new approach (A and ~ A)
3. comparison of differing
amounts of a single approach
(A and a or a and A)
where:
A – experimental (“treatment”) group
B – control (“no treatment,”
“nonmanipulated”) group
Variable...
…a concept (e.g., intelligence, height,
aptitude) that can assume any one of
a range of values
Research variables...
Independent…
…an activity of characteristic believed
to make a difference with respect to
some behavior
…(syn.) experimental variable, active
variable, cause, treatment
Dependent…
…the change or difference occurring a
a result of the independent variable
…(syn.) criterion variable, assigned
variable, effect, outcome, posttest
Confounding…
…the fact that the effects of the
independent variable may intertwine
with extraneous variables, such that
it is difficult to determine the unique
effects of each variable
…(syn.) criterion variable, assigned
variable, effect, outcome, posttest
Group experimental designs…
1. single-variable
2. factorial

single-variable designs
…involve one manipulated independent
variable
pre-experimental
quasi-experimental
true experimental

types of pre-experimental designs
one-shot case study
X O
…a single group exposed to a
treatment ( X ) and then
posttested ( O )
one-group pretest-posttest design
O X O
…a single group is pretested ( O ),
exposed to a treatment ( X ) and,
then, is posttested ( O )
static group comparison
X1 O
X2 O
…involves at least two groups ( X ), one
receiving a new, or experimental
treatment ( X1 ) and another receiving a
traditional, or control treatment ( X2 )
and, then, are posttested ( O )

types of quasi-experimental designs
nonequivalent control group
design
O X O
…random assignment of intact groups
that are pretested ( O ), exposed to a
treatment ( X ) and then posttested
(O)
time-series design
O O O O X O O O O
…a single group is pretested ( O )
repeatedly until pretest scores are
stable, exposed to a treatment ( X )
and, then, is repeatedly posttested
(O)
counterbalanced design
X1 O X2 O X3 O
X3 O X1 O X2 O
X2 O X3 O X1 O
…all of the groups receive all
treatments but in a different order;
the number of groups and
treatments must be equal

types of true experimental designs
pretest-posttest control group design
R O X1 O
R O X2 O
…at least two groups are formed by
random assignment ( R ),
administered a pretest ( O ), receive
different treatments ( X1, X2 ), are
administered a posttest, and posttest
scores are compared to determine
effectiveness of treatments
posttest-only control group design
R
X1 O
R
X2 O
…at least two groups are formed by
random assignment ( R ), receive
different treatments ( X1, X2 ), are
administered a posttest, and posttest
scores are compared to determine
effectiveness of treatments
Solomon four-group design
R O X1 O
R O X2 O
R
X1 O
R
X2 O
…four groups are formed by random
assignment ( R ) of participants, two
groups are pretested ( O ) and two are not,
one pretested and one unpretested group
receive the experimental treatments ( X1,
X2 ), each group is are administered a
posttest on the dependent variable, and
posttest scores are compared to
determine effectiveness of treatments

factorial designs
…involve two or more independent
variables with at least one
independent variable being
manipulated by the researcher

examples of factorial designs
two-by-two factorial design (four cells)
2 X 2
…two types of factors (e.g., method of
instruction) each of which has two
levels (e.g., traditional vs. innovative)
A 2 X 2 factorial design…
Dependent Variable
manipulated
Independent Variable
A
B
O
Group #1
Group #2
O
Group #3
Group #4
not manipulated
Cells
A 2 X 2 factorial design…
A
No interaction
between factors
B
A 2 X 2 factorial design…
No interaction
between factors
A
B
A 2 X 2 factorial design…
A
Interacting
factors
B
A 2 X 2 factorial design…
B
Interacting
factors
A
two-by-three factorial design (six cells)
2 X 3
…two types of factors (e.g., motivation;
interest) each of which has three levels
(e.g., high, medium, low)
Single-subject experimental designs…
1. A – B – A withdrawal
2. multiple baseline designs
3. alternating treatments designs

simple A – B design
…baseline measurements ( O ) are
repeatedly made until stability is
established, then the treatment ( X ) is
introduced and an appropriate number
of measurements ( O ) are made during
treatment implementation

simple A – B design
O O O
X O X O X
baseline
phase
treatment
phase
A
|
B
O

A – B – A withdrawal designs
…baseline measurements ( O ) are repeatedly
made until stability is established, then the
treatment ( X ) is introduced and an
appropriate number of measurements ( O )
are made during treatment implementation,
followed by an appropriate number of
baseline measurements ( O ) to determine
stability of treatment ( X )

A – B – A withdrawal designs
O O O
X O X O X
O O O
baseline
phase
treatment
phase
baseline
phase
A
|
B
|
A

multiple-baseline designs
…used when a return to baseline
conditions is difficult or impossible
since treatment effects oftentimes do
not disappear when a treatment is
removed
…“multiple” refers to the study of more
than one behavior, participant, or
setting
…instead of collecting baseline data on
one specific behavior, data are
collected on: (1) several behaviors for
one participant, (2) one behavior for
several participants, or (3) one
behavior and one participant in several
settings
…then, over a period of time, the
treatment is systematically applied to
each behavior (or participant, or
setting) one at a time until all
behaviors (or participants or settings)
have been exposed to the treatment

multiple baseline design
Example: one treatment for three behaviors in three settings
A
behavior 1
behavior 2
behavior 3
B
O O OXOXOXOXOXOXOXOXOXOXOXO
O O O O O
OXOXOXOXOXOXOXO
O O O O O
O O O
OXOXOXO
the baseline remains
the same…
setting #1
setting #2
setting #3
…while the treatment
is applied at other
settings

alternating treatments design
…the relatively rapid and random
alternation of treatments ( T1 T2 …Tn )
for a single subject but not
necessarily within fixed intervals of
time
T1 - T2 - T1 - T1 - T2 - T1 - T1 - T2
The concept of validity…
…the experiment tests the variable(s)
that it purports to test
Threats to validity…
…internal: factors other than the
independent variable that affect the
dependent variable
…external: factors that affect the
generalizability of the study to groups
and settings beyond those of the
experiment
Threats to internal validity…
1. history
2. maturation
3. testing
4. instrumentation
5. statistical regression
6. differential selection of participants
7. mortality
8. selection-maturation interaction

history
…the occurrence of events that are not
part of the experimental treatment
but that occur during the study and
affect the dependent variable

maturation
…the physical, intellectual, and
emotional changes that occur
naturally in a study’s participants
over a period of time

testing
…refers to improved scores on a
posttest as a result of having taken a
pretest

instrumentation
…the unreliability or lack of
consistency in measuring
instruments that can result in an
invalid assessment of performance

statistical regression
…the tendency of participants who
score highest on a test to score
lower on a second, similar test and
vice versa

differential selection of participants
…the outcome when already formed
groups are compared raising the
possibility that the groups were
different before a study even begins

mortality
…the case in which participants drop
out of a study which changes the
characteristics of the groups and
may significantly affect the study’s
results

selection-maturation interaction
…if already-formed groups are used in
a study, one group may profit more
(or less) from a treatment or have an
initial advantage because of
maturation, history, or testing factors
Threats to external validity…
1. pretest-treatment interaction
2. selection-treatment interaction
3. multiple treatment interference
4. specificity of variables
5. treatment diffusion
6. experimenter effects
7. reactive effects

pretest-treatment interaction
…the situation when participants
respond or react differently to a
treatment because they have been
pretested

multiple-treatment interference
…the situation when the same
participants receive more than one
treatment in succession

selection-treatment interference
…the situation when participants are
not randomly selected for treatments

specificity of variables
…the situation when a study is
conducted with (1) a specific kind of
participant; (2) is based on a
particular operational definition of
the independent variable; (3) uses
specific dependent variables; (4)
transpires at a specific time; and, (5)
under a specific set of circumstances

treatment diffusion
…the situation when different treatment
groups communicate with and learn
from each other

experimenter effects
…the situation when the researchers
present potential threats to the
external validity of their own studies

reactive arrangements
…the situation when a number of
factors associated with the way in
which a study is conducted interacts
with or shapes the feelings and
attitudes of the participants involved
Types of reactive arrangements…
…Hawthorne effect: any situation in
which participants’ behavior is
affected not by the treatment per se
but by their knowledge of participating
in a study
…compensatory rivalry: the control
group is informed that they will be the
control group for a new, experimental
study (“John Henry effect”)
…placebo effect: the situation in which
half of the participants receive no
treatment but believe they are
…novelty effect: the situation in which
participant interest, motivation, or
engagement increases simply because
they are doing something different
Controlling for extraneous
(confounding) variables…
1. randomization
2. matching
3. comparing homogeneous
groups or subgroups
4. using participants as their own
controls
5. analysis of covariance (ANCOVA)

randomization
…the process of selecting and
assigning participants in such a way
that all individuals in the defined
population have an equal and
independent chance of being
selected for the sample

matching
…a technique for equating groups on
one or more variables, usually the
ones highly related to performance
on the dependent variable (e.g.,
pairwise matching)

comparing homogeneous groups or
subgroups
…a technique to control an extraneous
variable by comparing groups that
are similar with respect to that
variable (e.g., stratified sampling)

using participants as their own controls
…exposing a single group to different
treatments one treatment at a time

analysis of covariance (ANCOVA)
…a statistical method for equating
randomly formed groups on one or
more variables by adjusting scores
on a dependent variable for initial
differences on some other variable
Data analysis and interpretation…

for single-subject research
…a visual inspection and analysis of
graphical presentations of results
…focuses upon: adequacy of the
design; an assessment of treatment
effectiveness (clinical vs. statistical
significance)
Mini-Quiz…

True and false…
…experimental researcher is the only
type of research that can test
hypotheses to establish cause-andeffect relationships
True

True and false…
…the manipulation of the independent
variable is the one characteristic that
differentiates correlational research
from other types of research
False

True and false…
…an experimental study is guided by
at least one hypothesis that states
an expected causal relationship
between two variables
True

True and false…
…the “experiment” in an experimental
study is conducted to conform or
disconfirm the research problem
False

True and false…
…experimental research has both
random selection and assignment
whereas causal-comparative
research has random selection not
assignment
True

True and false…
…it is correct to state that a control
group always receives no treatment
in experimental studies
False

True and false…
…in an experimental study, the
researcher makes every effort to
ensure that the two groups start as
equivalently as possible on all
variables except the independent
variable
True

True and false…
…in an experimental study, the
researcher collects data on the
independent variable from the
groups to determine whether there is
a significant difference between their
performances
False

True and false…
…in an experimental study, the
researcher endeavors to make the
groups as similar as possible so that
the only major difference between
them is the treatment variable(s)
manipulated by the researcher
True

True and false…
…even though experimental research
is the only type of research that can
truly establish cause-effect
relationships, it is not universally
appropriate for all research
problems or studies
True

True and false…
…an experiment is valid if results
obtained are due only to the control
variable and if they are generalizable
to individuals or contexts beyond the
experimental setting
False

True and false…
…internal validity focuses upon
threats or rival explanations that
influence the outcomes of an
experimental study but are not part
of the independent variable
True

True and false…
…if a study’s results cannot be
replicated in other settings by other
researchers, then the study has low
external reliability
False

True and false…
…to conduct a valid experiment, all the
researcher has to do is to maximize
internal and external validity
False

True and false…
…true experimental designs control for
all sources of internal and external
validity
False

True and false…
…the more narrowed and controlled
a research situation is, the less
realistic and generalizable it
becomes
True

True and false…
…a study that is not internally valid
is worthless
True

True and false…
…the longer a study lasts, the more
likely it is that history will be a
threat to the study’s internal
validity
True

True and false…
…the way a researcher controls for
selection-maturation interaction is
to select a design that controls for
potential problems or to make
every effort to determine if
selection-maturation interaction is
operating in the study
True

True and false…
…pretest-treatment interaction
makes a study’s results
generalizable only to other
pretested groups
True

True and false…
…the seriousness of pretest-treatment
interaction threat is dependent upon
the research participants, the nature
of the independent and dependent
variables, and the duration of the
study
True

True and false…
…even if intact groups are randomly
selected, the possibility exists that
the experimental group is in some
important way different from the
control group and/or from the larger
population
True

True and false…
…selection-treatment interaction is an
uncontrolled variable in research
designs involving randomization
True

True and false…
…it is possible to know what a
researcher means by the terms
assigned to variables without clear
operationalized descriptions
False

True and false…
…good designs control for many
sources of invalidity, poor designs
control few
True

True and false…
…ANCOVA is used to “correct” or
adjust posttest scores for initial
pretest differences
True

True and false…
…ANCOVA is assumes that the
relationship between the
independent and covariate variables
is curvilinear
False

True and false…
…a researcher should not totally
accept or reject a research design
because of how it controls (or fails to
control) for sources of invalidity
True

True and false…
…a researcher should decide which
research design is most appropriate
not only by the controls provided but
also by the nature of the study and
the setting in which it is to be
conducted
True

True and false…
…in a static-group comparison design,
each group serves as a control or
comparison group for the other
group(s)
True

True and false…
…in a counterbalanced design, the
average performance of the groups
on each treatment are calculated and
compared
True

True and false…
…in a true experimental designs,
participants should be randomly
selected or randomly assigned
False

True and false…
…in the pretest-posttest control group
design, if groups are essentially the
same on the dependent variable at
the start of a study, a t-test can be
used for analysis; if not, ANCOVA is
used
True

True and false…
…the more dissimilar the intact groups
in a study are, the stronger the study
is
False

True and false…
…when describing and symbolizing
factorial designs, the control variable
is placed first
False

True and false…
…the purpose of a factorial design is to
determine whether the effects of an
independent variable are
generalizable across all levels or are
specific to particular variables
True

True and false…
…factorial designs are very effective
for testing research hypotheses
that cannot be tested with a singlevariable design
True

True and false…
…for all types of research, the more
a study’s results are replicated, the
less confidence there is in the
procedures that produced those
results
False

Fill in the blank…
…the two main ways that researchers
equate groups participating in
experimental research
simple random sampling
stratified random sampling

Fill in the blank…
…the researcher decides what
treatment(s) will constitute the
independent variable and which
group will get which treatment
manipulation

Fill in the blank…
…the researcher’s efforts to remove
the influence of any extraneous
variable that might affect scores on
the dependent variable
control

Fill in the blank…
…the researcher’s concern with
threats or factors other than the
independent variable that affect the
dependent variable
internal validity

Fill in the blank…
…the researcher’s concern with the
extent to which the study results
can be generalized to groups and
settings beyond those of the
experiment
external validity

Fill in the blank…
…a threat to validity involving the
tendency for scores to more
toward a average or expected
score
statistical regression

Fill in the blank…
…a threat to internal validity that may
be overcome by testing for initial
equivalence between groups
differential selection of participants

Fill in the blank…
…the population from which a
researcher endeavors to extract a
sample
target population

Fill in the blank…
…the actual population from which a
researcher extracts a sample
accessible population

Fill in the blank…
…the influences on the part of the
experimenter over which the
experimenter has no direct control
but may affect study procedures
passive experimenter effects
experimenter personal-attributes effects

Fill in the blank…
…the situation in which a
researcher's expectation of the
study affect the researcher’s
behavior and the researcher
outcomes
active experimenter effect
experimenter bias effect

Fill in the blank…
…an external variable that effects
the independent variable and
intertwines with other extraneous
variables such that it is difficult
to determine the unique effects
of each
confounding variable

Fill in the blank…
…the term referring to a design that
has more than one independent
variable
factor(ial)

Fill in the blank…
…a study to compare the achievement
of two groups: one group taught in
an anxiety-producing environment
and one group taught in an anxietyreducing environment
experimental

Fill in the blank…
…the nonmanipulated variable in a
factorial design
control variable

Fill in the blank…
…a second study conducted by a
different investigator, with different
participants, behaviors, and settings
systematic replication

Fill in the blank…
…additional studies that add
confidence to the procedures that
produced those results
replication

Fill in the blank…
…a second study involving the
development of treatment packages,
composed of two or more
interventions that have been found
to be effective individually, designed
for persons with complex behavior
disorders
clinical replication

Fill in the blank…
…a second study conducted on a
number of participants with the
same problem, location, or time
simultaneous replication
This module has focused on...
experimental studies
...which test hypotheses to establish
cause-and-effect relationships
The next module will focus on...
descriptive statistics
…the set of mathematical
procedures for describing,
synthesizing, analyzing, and
interpreting quantitative data