Chapter 5: Experimental Design
Introduction
The design of any experiment is of utmost importance because it has the power to be the most
rigid type of research. The design, however, is always dependent on feasibility. The best
approach is to control for as many confounding variables as possible in order to eliminate or
reduce errors in the assumptions that will be made. It is also extremely desirable that any
threats to internal or external validity be neutralized. In the perfect world, all research would
do this and the results of research would be accurate and powerful. In the real world, however,
this is rarely the case. We are often dealing with human subjects, which in itself confound any
study. We are also dealing with the restraints of time and situation, often resulting in less than
perfect conditions in which to gather information.
There are three basic experimental designs, each containing subsets with specific strengths and
weaknesses. These three basic designs include: (1) pre-experimental design; (2) quasiexperimental design; and (3) true experimental design. They will be discussed below and as you
will discover, are addressed in order of effectiveness.
Pre-Experimental Design
Pre-experimental designs are so named because they follow basic experimental steps but fail to
include a control group. In other words, a single group is often studied but no comparison
between an equivalent non-treatment group is made. Examples include the following:
The One-Shot Case Study. In this arrangement, subjects are presented with some type of
treatment, such as a semester of college work experience, and then the outcome measure is
applied, such as college grades. Like all experimental designs, the goal is to determine if the
treatment had any effect on the outcome. Without a comparison group, it is impossible to
determine if the outcome scores are any higher than they would have been without the
treatment. And, without any pre-test scores, it is impossible to determine if any change
within the group itself has taken place.
One Group Pretest Posttest Study. A benefit of this design over the previously discussed
design is the inclusion of a pretest to determine baseline scores. To use this design in our
study of college performance, we could compare college grades prior to gaining the work
experience to the grades after completing a semester of work experience. We can now at least
state whether a change in the outcome or dependent variable has taken place. What we
cannot say is if this change would have occurred even without the application of the treatment or
independent variable. It is possible that mere maturation caused the change in grades and not
the work experience itself.
The Static Group Comparison Study. This design attempts to make up for the lack of a control
group but falls short in relation to showing if a change has occurred. In the static group
comparison study, two groups are chosen, one of which receives the treatment and the other
does not. A posttest score is then determined to measure the difference, after treatment,
between the two groups. As you can see, this study does not include any pre-testing and
therefore any difference between the two groups prior to the study are unknown.
Table 5.1: Diagrams of Pre-Experimental Designs
Quasi-Experimental Design
Quasi designs fair better than pre-experimental studies in that they employ a means to compare
groups. They fall short, however on one very important aspect of the experiment:
randomization.
Pretest Posttest Nonequivalent Group. With this design, both a control group and an
experimental group is compared, however, the groups are chosen and assigned out of
convenience rather than through randomization. This might be the method of choice for our
study on work experience as it would be difficult to choose students in a college setting at random
and place them in specific groups and classes. We might ask students to participate in a onesemester work experience program. We would then measure all of the students’ grades prior to
the start of the program and then again after the program. Those students who participated
would be our treatment group; those who did not would be our control group.
Time Series Designs. Time series designs refer to the pretesting and posttesting of one
group of subjects at different intervals. The purpose might be to determine long term effect of
treatment and therefore the number of pre- and posttests can vary from one each to many.
Sometimes there is an interruption between tests in order to assess the strength of treatment
over an extended time period. When such a design is employed, the posttest is referred to as
follow-up.
Nonequivalent Before-After Design. This design is used when we want to compare two groups
that are likely to be different even before the study begins. In other words, if we want to see how
a new treatment affects people with different psychological disorders, the disorders themselves
would create two or more nonequivalent groups. Once again, the number of pretests and
posttests can vary from one each to many.
The obvious concern with all of the quasi-experimental designs results from the method of
choosing subjects to participate in the experiment. While we could compare grades and
determine if there was a difference between the two groups before and after the study, we could
not state that this difference is related to the work experience itself or some other confounding
variable. It is certainly possible that those who volunteered for the study were inherently different
in terms of motivation from those who did not participate. Whenever subjects are chosen for
groups based on convenience rather than randomization, the reason for inclusion in the study
itself confounds our results.
Table 5.2: Diagrams of Quasi Experimental Designs
True Experimental Design
True experimental design makes up for the shortcomings of the two designs previously
discussed. They employ both a control group and a means to measure the change that
occurs in both groups. In this sense, we attempt to control for all confounding variables, or at
least consider their impact, while attempting to determine if the treatment is what truly caused the
change. The true experiment is often thought of as the only research method that can adequately
measure the cause and effect relationship. Below are some examples:
Posttest Equivalent Groups Study. Randomization and the comparison of both a control and
an experimental group are utilized in this type of study. Each group, chosen and assigned at
random is presented with either the treatment or some type of control. Posttests are then given
to each subject to determine if a difference between the two groups exists. While this is
approaching the best method, it falls short in its lack of a pretest measure. It is difficult to
determine if the difference apparent at the end of the study is an actual change from the possible
difference at the beginning of the study. In other words, randomization does well to mix subjects
but it does not completely assure us that this mix is truly creating an equivalency between the two
groups.
Pretest Posttest Equivalent Groups Study. Of those discussed, this method is the most
effective in terms of demonstrating cause and effect but it is also the most difficult to perform.
The pretest posttest equivalent groups design provides for both a control group and a measure of
change but also adds a pretest to assess any differences between the groups prior to the study
taking place. To apply this design to our work experience study, we would select students from
the college at random and then place the chosen students into one of two groups using random
assignment. We would then measure the previous semester’s grades for each group to get a
mean grade point average. The treatment, or work experience would be applied to one group
and a control would be applied to the other.
It is important that the two groups be treated in a similar manner to control for variables such as
socialization, so we may allow our control group to participate in some activity such as a softball
league while the other group is participating in the work experience program. At the end of the
semester, the experiment would end and the next semester’s grades would be gathered and
compared. If we found that the change in grades for the experimental group was significantly
different than the change in the grades of our control group, we could reasonably argue that one
semester of work experience compared to one semester of non-work related activity results in a
significant difference in grades.
Table 5.3: Diagrams of True Experimental Designs
Chapter Conclusion
The experiment, especially the true experimental design is often the measure of choice when
attempting to determine a cause and effect relationship. Utilizing randomization and the pretesting and post-testing of both an experimental group and a control group allows us to control for
more confounding variables than any other research method. These confounding variables,
when not addressed, can often result in inaccurate findings.
Controlling for confounding variables is important in research and especially important in the
experimental designs. This process helps us assure valid results both internally and externally.
The threats to internal validity, those that apply to the experimental situation itself, and external
validity, those relating to the generalizability of our results to the real world are also issues of
great concern to researchers. As the saying goes: garbage in, garbage out. If we start with a
flawed design we will end up with flawed results.
As the degree of control for each of the designs discussed increases, the difficulty in performing
the research also increases. Feasibility is always an issue and even when the most stringent
control is used, the mere fact that the subjects have agreed to participate in the experiment may
have a negative effect on the study’s generalizability. Are volunteer subjects truly representative
of the population at large? As you can see, there are varying degrees of experimental research,
but there is no perfect experiment that controls for all possible variables and assures us of 100%
generalizability.