EXPERIMENTS
© LOUIS COHEN, LAWRENCE
MANION & KEITH MORRISON
STRUCTURE OF THE CHAPTER
• Designs in educational experimentation
• True experimental designs
• A quasi-experimental design: the non-equivalent
control group design
• Single-case research: ABAB design
• Procedures in conducting experimental research
• Threats to internal and external validity in
experiments
• The timing of the pretest and the post-test
• Examples from educational research
• The design experiment
• Internet-based experiments
CAUSALITY
• Experiments are held up to be able to identify
causality through control and manipulation of
variables.
• Examine the effect of an independent
variable on a dependent variable.
• Identifying the effects of causes by
implementing interventions in a controlled
environment.
• Held up to be able to offer explanations for
outcomes.
INDEPENDENT AND
DEPENDENT VARIABLES
Development
planning
Parents
and
community
Teaching and
learning
School
Effectiveness
Professional
development
Management
Leadership
Culture and
climate
RANDOMIZATION
• Random sampling and random allocation to either a
control or experimental group.
• Randomization allows for the many additional
uncontrolled and, hence, unmeasured, variables that
may be part of the make-up of the groups in question.
• Randomization operates the ceteris paribus condition
(all other things being equal), assuming that the
distribution of extraneous variables is more or less
even and perhaps of little significance.
• Randomization strives to address Holland’s (1986)
‘fundamental problem of causal inference’, which is
that a person may not be in both a control group and
an experimental group simultaneously.
CONCERNS IN EXPERIMENTS
• It may not be possible or desirable to isolate
and control variables under laboratory
conditions.
• The ‘real world’ is not the antiseptic, artificial
world of the laboratory.
• Cannot assume that a single cause produces
a single effect.
• The setting affects the outcomes.
BLIND AND DOUBLE-BLIND
EXPERIMENTS
• Blind experiment: participants do not know to
which group they are assigned.
• Double blind experiment: neither the
researcher nor the participants know to which
group the participants are assigned.
KINDS OF EXPERIMENT
• Laboratory experiments (controlled, artificial conditions):
–
–
–
–
–
–
–
–
–
Pretest-post-test control and experimental group
Two control groups and one experimental group pretest-post-test
Post-test control and experimental group
Post-test two experimental groups
Pretest-post-test two treatment
Matched pairs;
Factorial design;
Parametric design;
Repeated measures design;
• Field experiments (controlled conditions in the ‘real world’):
– one-group pretest-post-test;
– non-equivalent control group design;
– time series
• Natural experiments (no control over real world conditions)
FEATURES OF A TRUE EXPERIMENT
•
•
•
•
•
•
•
•
Random allocation of the sample to control or
experimental groups;
Identification and isolation of key variables;
Control of the key variables;
Exclusion of any other variables;
Special treatment (the intervention) given to the
experimental group (i.e. manipulating the
independent variable) whilst holding every other
variable constant for the two groups;
Ensuring that the two groups are entirely separate
throughout the experiment (non-contamination);
Final measurement of outcomes to compare the
control and experimental groups and to look at
differences from the pre-test results (the post-test);
Comparison of one group with another.
Randomly assign subjects
to two matched groups:
control and experimental group
Stages in an
experiment
Conduct pre-test
Isolate and control variables,
exclude other variables
Administer intervention to
experimental group
Conduct post-test and compare
control and experimental groups
‘TRUE’ EXPERIMENTAL DESIGN
EXPERIMENT
Matched on
Pre-test
Random group
assignation
CONTROL
Intervention
EXPERIMENT
PLUS
Isolate,
control and
manipulate
variables
Post-test
CONTROL
MEASURING EFFECTS
Average causal effect (A):
(A) = (E1E2)  (C1C2)
where:
–
–
–
–
E1 = post-test for experimental group;
E2 = pretest for experimental group;
C1 = post-test for control group;
C2 = pretest for control group.
CAMPBELL’S AND STANLEY’S NOTATION
• X represents the exposure of a group to an
experimental variable or event, the effects of which are
to be measured.
• O refers to the process of observation or measurement.
• Xs and Os in a given row are applied to the same
persons.
• Left to right order indicates temporal sequence.
• Xs and Os vertical to one another are simultaneous.
• R indicates random assignment to separate treatment
groups.
• Parallel rows unseparated by dashes represent
comparison groups equated by randomization, while
those separated by a dashed line represent groups not
equated by random assignment.
CAMPBELL’S AND STANLEY’S
SYMBOLIC REPRESENTATION OF
‘TRUE’ EXPERIMENTS
RO1
RO3
X
O2
O4
Campbell, D. T. and Stanley, J (1963)
Experimental and Quasi-experimental
Designs for Research on Teaching. Boston:
Houghton Mifflin Co.
TWO CONTROL GROUPS AND ONE
EXPERIMENTAL GROUP PRETESTPOST-TEST DESIGN
Experimental
Control1
Control2
RO1
RO3
X
X
RO2
RO4
RO5
THE POST-TEST CONTROL AND
EXPERIMENTAL GROUP DESIGN
Experimental
Control
R1
R2
X
O1
O2
THE POST-TEST TWO
EXPERIMENTAL GROUPS DESIGN
Experimental1
Experimental2
R1
R2
X1
X2
O1
O2
THE PRETEST―POST-TEST TWO
TREATMENT DESIGN
Experimental1
Experimental2
RO1
RO3
X1
X2
O1
O4
THE TRUE EXPERIMENT ONE
CONTROL AND TWO EXPERIMENTAL
GROUPS
Experimental1
Experimental2
Control
RO1
RO3
RO5
X1
X2
O1
O4
O6
THE PRE-TEST TWO TREATMENT
DESIGN
Experimental1
Experimental2
RO1
RO3
X1
X2
O1
O4
MATCHED PAIRS DESIGN
Step One: Measure the dependent variable.
Step Two: Assign participants to matched pairs,
based on the scores and measures established
from Step One.
Step Three: Randomly assign one person from
each pair to the control group and the other to
the experimental group.
Step Four: Administer the intervention to the
experimental group and, if appropriate, a placebo
to the control group. Ensure that the control
group is not subject to the intervention.
Step Five: Carry out a measure of the
dependent variable with both groups and
compare/measure them in order to determine the
effect and its size on the dependent variable.
FACTORIAL DESIGN
Performance in an examination may depend on availability of
resources and motivation for the subject studied
INDEPEND
ENT
VARIABLE
Availability
of
resources
motivation
for the
subject
studied
LEVEL
ONE
LEVEL
TWO
LEVEL
THREE
limited
moderate
high
availability (1) availability (2) availability (3)
little
motivation (4)
moderate
motivation (5)
high
motivation (6)
9 combinations: 1+4; 1+5; 1+6; 2+4; 2+5; 2+6; 3+4; 3+5; 3+6
Motivation for mathematics
100
80
60
Males
40
Females
20
0
15
16
17
Age
18
Factorial designs
must address
the interaction of
the independent
variables.
Difference for motivation in mathematics is not constant
between males and females, but varies according to age
of participants: an interaction effect (age and sex)
PARAMETRIC DESIGN
• Participants are randomly assigned to groups
whose parameters are fixed in terms of the
levels of the independent variable that each
receives.
• Parametric designs are useful if an
independent variable has different levels or a
range of values which may have a bearing on
the outcome (confirmatory research) or if the
researcher wishes to discover whether
different levels of an independent variable
have an effect on the outcome (exploratory
research).
REPEATED MEASURES
• Participants in the experimental groups are
tested under two or more experimental
conditions.
• The order in which the interventions are
sequenced may have an effect on the
outcome (e.g. the first intervention may have
an influence – a carry-over effect – on the
second, and the second intervention may
have an influence on the third).
• Early interventions may have a greater effect
than later interventions.
• Repeated measures designs are useful if it is
considered that order effects are either
unimportant or unlikely.
REPEATED MEASURES
(two groups receiving both conditions)
Group 1
With no
intervention
Group 1
With
intervention
Matched on pre-test
Post-test
Random allocation to
groups
Group 2
With
intervention
Group 2
With no
intervention
Independent
groups
Noise condition
No noise
condition






Sara Rob Peter
Jane Jack Jim






Joan Susan John Lyn Sally Alan
Repeated
measures
Noise condition
No noise
condition






Sara Rob Peter
Jane Jack Jim






Joan Susan John Lyn Sally Alan






Jane Jack Jim
Sara Rob Peter




Lyn Sally Alan


Joan Susan John
QUASI-EXPERIMENTS: NONEQUIVALENT CONTROL GROUP
DESIGN
• Pre-experimental design: the one-group
pretest―post-test
Experimental
O1
X
O2
• Pre-experimental design: the one-group posttest only design
Experimental
O1
• The Post-Tests only non-equivalent groups
design
Experimental
Control
O1
---------O2
QUASI-EXPERIMENTS: NONEQUIVALENT CONTROL GROUP
DESIGN
• The pre-test―post-test non-equivalent
group design
Experimental O1 X
O2
---------Control
O3
O4
PROCEDURES IN CONDUCTING
EXPERIMENTS
1. Identify research problems
2. Formulate hypotheses
3. Select appropriate levels at which to test the
independent variables
4. Decide which kind of experiment to adopt
5. Decide population and sampling
6. Select instruments for measurement
7. Decide how the data will be analyzed
8. Pilot experimental procedures
9. Carry out the refined procedures
10. Analyze results
11. Report the results
A TEN-STEP MODEL FOR
CONDUCTING EXPERIMENTS
Step One: Identify the purpose of the experiment.
Step Two: Select the relevant variables.
Step Three: Specify the level(s) of the intervention
(e.g. low, medium high intervention).
Step Four: Control the experimental conditions and
environment.
Step Five: Select appropriate experimental design.
Step Six:
Administer the pretest.
Step Seven: Assign the participants to the group(s).
Step Eight: Conduct the intervention.
Step Nine: Conduct the post-test.
Step Ten: Analyze the results.
PROCEDURES IN CONDUCTING
EXPERIMENTS: HYPOTHESES
• Null hypothesis (H1)
• Alternative hypothesis (H0)
• Direction of hypothesis: states the kind of
difference or relationship between two
conditions or two groups of participants
• One-tailed (directional): e.g. ‘people who study
in silent surroundings achieve better than those
who study in noisy surroundings’
• Two-tailed (no direction): e.g. ‘there is a
difference between people who study in silent
surroundings and those who study in noisy
surroundings’
OPERATIONALIZING HYPOTHESES
• Hypothesis: ‘people who study in quiet
surroundings achieve better than those who
study in noisy surroundings’
• What do ‘work better’, ‘quiet’ and ‘noisy’ mean?
Define the operations:
– ‘work better’ = obtain a higher score on the
Wechsler Adult Intelligence Scale
– ‘quiet’ = silence
– ‘noisy’ = CD music playing
• Operationalized hypothesis: ‘people who study
in silence achieve a higher score on the
Wechsler Adult Intelligence Scale than those
who study with CD music playing’
DIRECTIONAL AND NONDIRECTIONAL HYPOTHESES
Directional (one-tailed):
People who do homework without the TV
on produce better results than those who
do homework with the TV on.
Non-directional (two-tailed):
There is a difference between work
produced in noisy or silent conditions.
HISTORY
MATURATION
TESTING
DIRECTION
OF CAUSALITY
TYPE 1 AND
TYPE 2
ERRORS
INSTRUMENTATION
THREATS TO
VALIDITY AND
RELIABILITY
EXPERIMENTAL
MORTALITY
OPERATIONALIZATION
CONTAMINATION
REACTIVITY
TIMING OF PRE-TEST AND POST-TEST
• Pre-test: as close to the start of the experiment as
possible (to avoid contamination of other variables).
• Post-test: as close to the end of the intervention as
possible.
• Too soon a post-test: misses long-term/delayed
effect and only measures short-term gain (which may
be lost later).
• Too long a time lapse before a post-test: becomes
impossible to determine whether it was a particular
independent variable that caused a particular effect,
or whether other factors have intervened since the
intervention, to produce the effect.
INTERNET-BASED EXPERIMENTS
Four types:
1. Those that present static printed materials
(e.g. printed text or graphics)
2. Those that make use of non-printed
materials (e.g. video or sound)
3. Reaction-time experiments
4. Experiments that involve some form of
interpersonal interaction
INTERNET-BASED EXPERIMENTS
• Check download speeds and time, anticipate
problems of different browsers and platforms.
• Can experience greater problems of dropout
than conventional experiments.