Randomized Experimental Design
When defining the essence of an experiment, Campbell and Stanley stressed the
random assignment of subjects to experimental treatments. I stress the manipulation of
the treatment variable. In this document I shall discuss designs where the treatment
variable is experimentally manipulated and the subjects are randomly assigned to
experimental conditions. In a later document I shall discuss the so-called quasiexperimental designs, where a treatment is manipulated but subjects are not randomly
assigned to experimental conditions.
Random Selection and Random Assignment
Let me remind you of the difference between random selection and random
assignment. Random selection is accomplished when you randomly select subjects
into your research sample from a specified population. Random assignment refers to
the process of deciding which subjects get which treatments. If those decisions are
made randomly, then random assignment has been accomplished
Random Selection
Sampling N data points from a population is random if every possible different
sample of size N was equally likely to be selected.


We want our samples to be representative of the population to which we are
making inferences. With moderately large random samples, we can be
moderately confident that our sample is representative.

The inferential statistics we shall use assume that random sampling is employed.

In fact, we rarely if ever achieve truly random sampling, but we try to get as close
to it as is reasonable.

You may have seen a definition of random sampling that looked something like
this: "Each and every element of the population has an equal chance of being
selected."). A sampling procedure may this definition but not mine. For example,
sampling from a population of 4 objects (A,B,C,& D) without replacement, N = 2,
contrast sampling procedure X with Y:
Copyright 2013, Karl L. Wuensch - All rights reserved.
Research-7-RanExpDesign.docx
2
Sampling Procedure
Sample
X
Y
AB
1/2
1/6
AC
0
1/6
AD
0
1/6
BC
0
1/6
BD
0
1/6
CD
1/2
1/6
Note. The tabled values are probabilities.

If each time a single score is sampled, all scores in the population are equally
likely to be selected, then a random sample will be obtained.
Random Assignment
We have already discussed the Randomized Pretest-Posttest Control Group
Design and the Randomized Posttest Only Control Group Design.
R
O
R
O
X
O
R
O
R
X
O
O
In both of these designs we wish to evaluate the effectiveness of the treatment
by comparing the two groups, one which received the treatment and one which did not.
If we can consider our two groups to have been equivalent prior to the administration of
the treatment, then their differing after the treatment is strong evidence that the
treatment caused that difference. Random assignment to groups allows to consider the
two groups as equivalent prior to the treatment. Technically it is the two populations
from which our samples were drawn that we can consider to be equivalent -- we can
expect that our two samples will differ at least a bit on this and that due to sampling
error.
Noise Reducing Designs
Some designs, and their associated statistical analysis, are known to enhance
power. Power, you should recall, is the conditional probability of detecting an effect of
the experimental treatment, given that it really does have an effect.
Randomized Blocks Design
Earlier I mentioned the power-enhancing quality of the randomized blocks
design. Let me give you more detail now. Suppose you were going to evaluate the
effectiveness of three different methods of teaching young children the alphabet. To
enhance power, you wish to use a randomized blocks design. You administer to every
3
potential subject a test of readiness to learn the alphabet, and then you match (block)
subjects on that variable. Next you randomly assign them (within each block) to groups.
In your statistical analysis, the effect of the matching/blocking variable is taken out of
what would otherwise be “error variance” in your statistical model. Such error variance
is generally the denominator of the ratio that you use as the test statistic for a test of
statistical significance, and the numerator of that ratio is generally a measure of the
apparent magnitude of the treatment effect. Lets look at that ratio.
treatment effect
difference between means
, for example, t 
, or
noise
standard error of difference
among groups variance
.
F
error variance
Test statistic 
Look at this pie chart, in which I have partitioned the total variance in the DV into
variance due to the treatment, due to the blocking variable, and due to everything else
(error). If we had just ignored the blocking variable, rather than controlling it by using
the randomized blocks design, the variance identified as due to blocks would be
included in the error variance. Look back at the test statistic ratio. Since error variance
is in the denominator, removing some of it makes the absolute value of the test statistic
greater, giving you more power (a greater probability of obtaining a significant result).
I used the term “split-plot” design earlier as a synonym for a randomized blocks
design. The term “split plot” comes from agricultural research, in which a field is divided
into numerous plots and each plot is divided into k parts, where k is the number of
treatments. Within each plot, treatments are randomly assigned to each part -- for
example, seed type A to one part, seed type B to a second part, etc. Statistically, the
plots here are just like the blocks we use in a randomized blocks design.
Some of you have already studied “repeated measures” or “within subjects”
designs, where each subject is tested under each treatment condition. This is really just
a special case of the randomized blocks design, where subjects are blocked up on all
subject variables. If you have not studied such designs or need a refresher on
important topics such as order effects and counterbalancing, please read the document
An Introduction to Within-Subjects Analysis of Variance.
Blocks
Error
Error
Treatment
Blocks
Treatment
Analysis of Covariance. Another way to reduce noise and increase power is to
have available for every subject data on one or more covariate. Each covariate should
be an extraneous variable which is well correlated with the dependent variable. We can
4
then use an ANCOV (analysis of covariance) to remove from the error term that
variance due to the covariate (just like the randomized blocks analysis does), but we
don’t need to do the blocking and random assignment within blocks. This analysis is
most straightforward when we are using it along with random assignment of subjects to
groups, rather than trying to use ANCOV to “unconfound” a static-group design (more
on this later in the semester).
Factorial ANOVA
If the extraneous variable you wish to control is a categorical variable, one
method to remove its effect from the error variance is just to designate the extraneous
variable as being a classification (“independent”) variable in a factorial ANOVA. Please
read my document An Introduction to Factorial Analysis of Variance. A 2 x 2
factorial design would be represented like this, using the design notation we have
adopted:
R
X11
O
R
X12
O
R
X21
O
R
X22
O
Other Randomized Experimental Designs
There are many, I’ll illustrate a couple here.
Solomon Four Group Design
R
O
This design is a combination of the pretest-posttest
control group design and the posttest only control group design.
It is as good at controlling threats to internal and external validity
as the posttest only control group design, but superior to that
design with respect to statistical conclusion validity. However, it
gains that advantage at the cost of considerably greater expense
and effort in the collection and analysis of the data.
R
O
R
R
X
O
O
X
O
O
To test the hypothesis of no testing by treatment interaction, one could
arrange all four groups’ posttest scores into a 2 x 2 factorial ANOVA design, (pretested
or not) X (given the experimental treatment or not). A significant interaction would
indicate that the simple main effect of the treatment among those that were pretested
was not the same as the simple main effect of the treatment among those that were not
pretested. The main effect of pretesting and the main effect of the treatment could
also be assessed with such an ANOVA. If there existed a significant testing by
treatment interaction, one could test the simple main effect of the treatment for the
pretested subjects and for the not pretested subjects. One might also want to analyze
the data from the pretest-posttest control group part of the design with those techniques
5
which are appropriate for such a design (such as ANCOV on the posttest scores using
pretest scores as a covariate).
Here are some examples of patterns of results from the Solomon four group
design. Each cell contains a posttest mean. Where a pretest was given, I have
indicated the pretest mean in parentheses.
Treatment effect but no testing or Testing x Treatment interaction
Treatment
Pre-tested
Control
X
marginal
no
10
30
20
yes
10 (10)
30 (10)
20
10
30
20
marginal
Treatment and testing effects but no Testing x Treatment interaction
Treatment
Pre-tested
none
X
marginal
no
10
30
20
yes
20 (10)
40 (10)
30
15
35
25
marginal
Treatment and testing effects and a Testing x Treatment interaction
Treatment
Pre-tested
none
X
marginal
no
10
30
20
yes
20 (10)
60 (10)
40
15
45
30
marginal
Randomized Switching-Replications Design
This design represents an attempt to control
R O X O
O
some of the social threats to internal validity -- that is,
threats that arise when members of one experimental
R O
O X O
group know that they did not get the special treatment
but that the other group did. What we do is give both groups the special treatment, but
one group gets the special treatment first and the other group gets it later. Of course,
there still may be social effects with respect to who gets the special treatment first.
6
Here are three patterns of results for this design. Group 1 is that which gets the
special treatment first.
Temporary Treatment Effect
Group
Pretest
Posttest 1
Posttest 2
1
150
130
150
2
150
150
130
Imagine that the dependent variable is a measure of the amount of anxiety a
patient feels and the treatment is administration of a drug designed to lower anxiety.
The results above show that the treatment has a temporary 20 point effect.
Persistent Treatment Effect
Group
Pretest
Posttest 1
Posttest 2
1
150
130
130
2
150
150
130
Imagine that the treatment is a particular type of psychotherapy. The results
above indicate that the treatment has a persistent effect of lowering anxiety by 20
points. Of course, if we took a fourth measurement later in time we might find that the
effect does not persist indefinitely.
Continuing Treatment Effect
Group
Pretest
Posttest 1
Posttest 2
1
150
130
120
2
150
150
130
Here we see that anxiety continues to decline beyond the first post-treatment
observation. For example, our patients may have been given some cognitive
psychotherapy that teaches them ways to avoid irrational, anxiety-provoking thoughts.
Although improvement was shown already at the time of the first post-treatment
observation, the patients continue to improve beyond that, as they become better and
better at employing the cognitive techniques.
Copyright 2013, Karl L. Wuensch - All rights reserved.

Fair Use of this Document