The Two-Group Randomized
Experiment
The Basic Design
R
R


X
O
O
Note that a pretest is not necessary in this design.
Why?
Because random assignment assures that we have
probabilistic equivalence between groups
The Basic Design
R
R



X
O
O
Differences between groups on posttest indicate a
treatment effect.
Usually test this with a t-test or one-way ANOVA.
Why no pretest?
Internal Validity
History 
Maturation
Testing
Instrumentation
Mortality
Regression to the mean
Selection
Selection-history
Selection- maturation
Selection- testing
Selection- instrumentation
Selection- mortality
Selection- regression
Diffusion or imitation
Compensatory equalization
Compensatory rivalry
Resentful demoralization
R X O
R
O
Examples
Experimental Design Variations


The posttest-only two group design is
the simplest; there are many variations.
To better understand what the
variations try to achieve, we can use the
signal-to-noise metaphor.
Signal to Noise
What we observe can be divided into
what we see
Signal to Noise
What we observe can be divided into
Signal
what we see
Signal to Noise
What we observe can be divided into
Signal
Noise
what we see
Signal to Noise
Experimental designs can take two
approaches:
Signal to Noise
Experimental designs can take two
approaches:
Signal
Focus on (enhance)
the signal
Signal to Noise
Experimental designs can take two
approaches:
Signal
Focus on (enhance)
the signal (what is
this?)
Signal to Noise
Experimental designs can take two
approaches:
Signal to Noise
Experimental designs can take two
approaches:
or reduce the noise
Noise
(what is this?)
Signal to Noise
Experimental designs can take two
approaches:
or reduce the noise
Noise
Signal to Noise
Signal
enhancers
Noise
reducers



Factorial designs
Covariance designs
Blocking designs
Noise and Signal in Significance Tests

For interval and ratio dependent
variables, you can conduct a
difference in means test:
( X 2  X 1 )  2  ( std .error1 )  ( std .error 2 )
2
Signal
std .error1 
ˆ1
N1
Noise
2
Factorial Designs
A Simple Example
R
R
R
R
X11
X12
X21
X22
O
O
O
O
Factor 1: Time in Instruction
Level 1:
1 hour per week
Level 2:
4 hours per week
Factor 2: Setting
Level 1:
In-class
Level 2:
Pull-out
A Simple Example
Setting
Time in Instruction
Factors:
Major independent variables
A Simple Example
Time in Instruction
In-class
Pull-out
Setting
1 hour/week
4 hours/week
Levels:
subdivisions
of factors
A Simple Example
Time in Instruction
factors
1 hour/week
4 hours/week
In-class
Group 1
average
Group 3
average
Pull-out
Setting
levels
Group 2
average
Group 4
average
Usually, averages are in the cells.
Multiplicative Notation
A 3 x 4 factorial design
How many factors?
How many levels?
How many cells with
averages?
Multiplicative Notation
A 3 x 4 factorial design
The number of numbers tells
you how many factors
there are.
There are 2 factors
because there are 2
numbers.
Multiplicative Notation
A 3 x 4 factorial design
The number values tell you
how many levels are in
each factor.
Multiplicative Notation
A 3 x 4 factorial design
The number values tell you
how many levels are in
each factor.
Factor 1 has 3 levels.
Factor 2 has 4 levels.
The Null Case
4 hrs
Out
5
5
5
In
5
5
5
5
5
8
7
6
5
4
3
2
in
out
1hr
4hrs
The lines in the graphs
below overlap each
other.
8
7
6
5
4
3
2
out
in
4hrs
1 hr
1hr
Setting
Time
A Main Effect


A consistent difference between levels
of a factor
For instance, we would say there’s a
main effect for time if we find a
statistical difference between the
averages for the hours of instruction
between groups
Main Effects
Out
1 hr
4 hrs
5
7
6
Main Effect of Time
5
7
5
7
8
7
6
5
4
3
2
6
in
out
1hr
4hrs
8
7
6
5
4
3
2
out
in
4hrs
In
1hr
Setting
Time
Main Effects
Out
1 hr
4 hrs
5
5
5
Main Effect of Setting
7
7
6
6
8
7
6
5
4
3
2
7
in
out
1hr
4hrs
8
7
6
5
4
3
2
out
in
4hrs
In
1hr
Setting
Time
Main Effects
1 hr
4 hrs
Out
5
7
6
In
7
9
8
6
8
10
Main Effects of
Time and Setting
10
8
1hr
4hrs
6
4
8
out
in
6
4
in
out
4hrs
2
2
1hr
Setting
Time
An Interaction Effect



When differences on one factor depend
on the level you are on on another
factor
An interaction is between factors (not
levels)
You know there’s an interaction when
can’t talk about effect on one factor
without mentioning the other factor
Interaction Effects
4 hrs
Out
5
5
5
In
5
7
6
5
6
8
7
6
5
4
3
2
in
out
1hr
4hrs
The in-class, 4-hour per
week group differs
from all the others.
8
7
6
5
4
3
2
out
in
4hrs
1 hr
1hr
Setting
Time
Interaction Effects
4 hrs
Out
7
5
6
In
5
7
6
6
6
8
7
6
5
4
3
2
in
out
1hr
4hrs
The 1-hour amount works
well with pull-outs while
the 4 hour works as well
with in class.
8
7
6
5
4
3
2
out
in
4hrs
1 hr
1hr
Setting
Time
Advantages of Factorial Designs



Offers great flexibility for exploring or
enhancing the “signal” (treatment)
Makes it possible to study interactions
Combines multiple studies into one
Randomized Block Designs
The Basic Design
R
R
X
O
O
The Basic Design
R
R
X
O
O
R
R
X
O
O
R
R
X
O
O
R
R
X
O
O
Homogeneous
groups
The Basic Design
R
R
X
O
O
R
R
X
O
O
R
R
X
O
O
R
R
X
O
O
Homogeneity
on the
dependent
variable
(observations
in group one
tend to have
higher levels of
the dependent
variable than
observations in
group two, etc.)
Homogeneous
groups
Randomized Blocks Design
R
R
X
O
O
R
R
X
O
O
R
R
X
O
O
R
R
X
O
O
Each replicate
is a block.
Randomized Blocks Design



Can block before or after the study
Can block on a measured variable or on
unmeasured perceptions
Is a noise-reducing strategy
How Does Blocking Reduce Noise?
Posttest
100
90
80
70
60
50
40
30
20
20
30
40
50
Pretest
60
70
80
How Does Blocking Reduce Noise?
Range of x
Posttest
100
90
80
70
60
50
40
30
20
20
30
40
50
Pretest
60
70
80
How Does Blocking Reduce Noise?
Range of x
Posttest
100
90
Variability of y
80
70
60
50
40
30
20
20
30
40
50
Pretest
60
70
80
How Does Blocking Reduce Noise?
Range of x
Posttest
100
80
Variability of y
Mean difference
90
70
60
50
40
30
20
20
30
40
50
Pretest
60
70
80
How Does Blocking Reduce Noise?
For block 3
Posttest
100
90
80
70
60
50
40
30
20
20
30
40
50
Pretest
60
70
80
How Does Blocking Reduce Noise?
For block 3
Range of x in block
Posttest
100
90
80
70
60
50
40
30
20
20
30
40
50
Pretest
60
70
80
How Does Blocking Reduce Noise?
For block 3
Range of x in block
Posttest
100
Variability of y in block
90
80
70
60
50
40
30
20
20
30
40
50
Pretest
60
70
80
How Does Blocking Reduce Noise?
For block 3
Range of x in block
Posttest
Variability of y in block
Mean difference
100
90
80
70
60
50
40
30
20
20
30
40
50
Pretest
60
70
80
How Does Blocking Reduce Noise?
Posttest
For block 3
Range of x in block
Variability of y in block
Mean difference
100
90
80
70
60
50
40
30
20
Same mean difference,
20
30
40
50
but lower variability
on both x and y
Pretest
60
70
80
Conclusion


Instead of having one treatment effect
based on the full variability of y, you
have three treatment effects based on
reduced variability (but with the same
mean difference).
The average of the three estimates
gives a less noisy estimate than the
nonblock one.
Analysis of Covariance Experimental
Designs
Design Notation
R
R



O
O
X
O
O
Uses a pre-measure
Can be a pretest, but doesn’t have to be
Can have multiple covariates
The Covariance Design
Posttest
100
90
80
70
60
50
40
30
20
20
30
40
50
Pretest
60
70
80
The Covariance Design
Posttest
The range of y is
about 70 points.
100
90
80
70
60
50
40
30
20
20
30
40
50
Pretest
60
70
80
How Does a Covariate Reduce Noise?
Posttest
100
We fit regression lines to
describe the pretest-posttest
relationship.
90
80
70
60
50
40
30
20
20
30
40
50
Pretest
60
70
80
How Does a Covariate Reduce Noise?
Posttest
100
90
80
70
60
50
40
30
20
20
30
40
50
We want to “adjust” the
posttest scores for
60pretest
70 variability.
80
Pretest
How Does a Covariate Reduce Noise?
Posttest
100
We do this by “subtracting
90
out” the
pretest -- by
80
“subtracting
out” the line.
70
60
50
40
30
20
20
30
40
50
We want to “adjust” the
posttest scores for
60pretest
70 variability.
80
Pretest
How Does a Covariate Reduce Noise?
Posttest
100
We do this by “subtracting
90
out” the
pretest -- by
80
“subtracting
out” the line.
70
Get the
difference
between the
line and each
point.
60
50
40
30
20
20
30
40
50
Pretest
60
70
80
How Does a Covariate Reduce Noise?
Here is the plot with the effect of the pretest
removed. Notice the range on y is now only about
50 points instead of 70 (although the difference
between the means remains the same).
40
30
20
10
0
-10
-20
-30
-40
20
30
40
50
Pretest
60
70
80
Summary


The Analysis of Covariance adjusts
posttest scores for variability on the
covariate (pretest).
This is what we mean by adjusting for
the effects of one variable on another.
Summary


You can use any continuous variable as
the covariate, but the pretest is usually
best.
You can use multiple covariates, but if
they are highly intercorrelated, you don’t
improve the adjustment (and you pay a
price for each covariate).
Hybrid Experimental Designs
Hybrid Designs



Variations on randomized designs
Help to address specific threats
Incorporates different design features
Two will be illustrated here.
The Solomon Four-Group Design
R
R
R
R


O
O
X
X
O
O
O
O
Explicitly addresses a testing threat
Assesses the effect of taking a pretest
Possible Outcomes
Treatment Effect -- No Testing Effect
18
16
14
Tpre-post
Cpre-post
Tpost
Cpost
12
10
8
6
Pre
Post
Possible Outcomes
Treatment Effect and Testing Effect
18
16
14
Tpre-post
Cpre-post
Tpost
Cpost
12
10
8
6
Pre
Post
Switching Replications Design
R
R


O
O
X
O
O
X
O
O
Switching: the groups switch roles over
the course of the study.
Replications: the treatment is repeated
or replicated.
Possible Outcomes
Short-term Persistent Treatment Effect
18
16
14
Group 1
Group 2
12
10
8
6
Pre
Post1
Post2
Possible Outcomes
Long-term Continuing Treatment Effect
18
16
14
Group 1
Group 2
12
10
8
6
Pre
Post1
Post2
Switching Replications




Very strong in internal validity
Looks at short and longer term effects
Strong ethically because all participants
get the treatment
Works well with some institutional
structures (for example, semester
system in schools)