Designing Experiments
AP Statistics – Section 5.2
Definition:
An observational study observes individuals and measures variables of interest but does not attempt to influence the
response.
Example 1 (Statistics in Action):
Suppose you want to study accident rates at a dangerous intersection to see if the rate on rainy days is higher than the rate on
dry days.
a.
Explain why this is an observational study.
b.
If it turns out that most of the rainy days you have available for study are weekends and almost all of the dry days
are during the week, can you see a problem with your results? Is there a confounding variable involved in this study?
An experiment can help eliminate (or at least try to minimize the effects of) lurking variables.
Definition (BPS):
An experiment deliberately imposes some treatment on individuals in order to observe their responses.
Example 2 (Statistics in Action)
A psychologist wants to compare children from first, third and fifth grades to determine the relationship between grade level
and how quickly a child can solve word puzzles. Would this be an observational study or an experiment?
Example 3 (Statistics in Action)
Some buttercups grow in bright, sunny fields; others grow in woods, where it is darker and damper. A plant ecologist wanted
to know whether buttercups in one habitat had adapted in a way that made them less successful in the other habitat. For his
study, he dug up 10 plants from each habitat. Five out of each 10, chosen at random, were planted in the habitat they came
from; the remaining five were planted in the other habitat. At the end of the study, he compared the sizes of the plants.
1.
Why did he bother to dig up the plants that were just going to be replanted in the same habitat?
2.
Is this study observational or experimental?
Page 1 of 6
Terms Used in an Experiment
1.
Experimental units are individuals on which the experiment is being done. If the units are people they are called
subjects.
2.
The treatment is the specific experimental condition applied to the units. The level of treatment is measured by the
explanatory variable and the level of the variable we’re interested in is measured by the response variable.
3.
Factors are the explanatory variables in an experiment (i.e., the changes that are being imposed on the subjects in
the experiment). Experiments may have several factors.
Example 4 (Statistics in Action):
Does the type of lighting or the music in a dentist’s waiting room have any effect on the anxiety level of the patient?
Consider an experiment to study this question, represented by the nine categories shown in the table below.
Type of Music
Pop
Brightness
Classical
Jazz
Low
Medium
High
a.
What are the factors? What are the levels of each factor?
b.
Describe a possible response variable.
Example 5 (Statistics in Action):
You want to compare two different textbooks for the AP Statistics course. Ten classes with a total of 150 students will take
part in the study. To judge the effectiveness of the books, you plan to use the scores of the students when they take the AP
exam at the end of the year. The classes are randomly divided into two groups of five classes each. The first group of five
classes, with 80 students, uses the first of the two books. The remaining five classes, with 70 students total, use the other
book. Identify the treatments and the experimental units in this scenario, and then tell the sample size.
Now that we know what an experiment is and some of the terms used, we need to focus on what makes a well-designed
experiment. Part of that discussion also involves what constitutes a poor experimental design.
Page 2 of 6
Explain why each of the next two designs create poorly designed experiments.
1.
Take a group of experimental units ---> Impose a treatment ---> Observe
2.
Observe a group of experimental units ---> Impose a treatment --> Observe
A well-designed experiment almost always satisfies the following three principles.
3 Basic Principles of Experimental Design
I.
Control the effect of lurking variables on the response variable. It is important to know that the changes in the
response variable are due to the explanatory variable and not some other variable not involved in the study.
One commonly used method designed to control the effects of lurking variables is to have a second group (called a
control group) involved in the experiment. Control groups are often given a placebo (a “sugar pill”) instead of the
treatment being studied. The results of the treatment group and the control group are then compared. Another type
of control involves using a comparison group (for example, comparing new and improved cereal to what is
currently on the market.) We’ll often refer to both types of groups as the control group.
II. Randomization ensures that individuals are assigned to treatment groups by chance.
III. Replication of the experiment many times or on many subjects ensures that the results are not just due to chance
variation.
Three more important definitions are listed below.
Definition (Statistics in Action):
An experiment that has random assignment and a control or comparison group is called a randomized comparative
experiment.
Definition (BPS):
An observed effect too large to attribute plausibly to chance is called statistically significant.
Definition (paraphrased from Statistics in Action):
The placebo effect occurs when a subject receiving a placebo reacts favorably to it (even though there is no medicinal benefit
in the pill).
Page 3 of 6
Example 6 (BPS):
A food company assesses the nutritional quality of a new “instant breakfast” product by feeding it to newly weaned male
white rats. The response variable is a rat’s weight gain over a 28-day period. A control group of rats eats a standard diet but
otherwise receives exactly the same treatment as the experimental group. If 30 rats are to be used in the experiment, explain
(diagram) how the experiment should be designed.
Example 7 (BPS):
A manufacturer of food products uses package liners that are sealed at the top by applying heated jaws after the package is
filled. The customer peels the sealed pieces apart to open the package. What effect does the temperature of the jaws have on
the force needed to peel the liner? To answer this question, engineers obtain 20 pairs of pieces of package liner. They seal
five pairs of each at 250F, 275F, 300F and 325F. Then they measure the force needed to peel each seal.
a.
What are the experimental units?
b.
There is one factor (explanatory variable). What is it, and what are its levels?
c.
What is the response variable?
d.
Use a diagram to design a completely randomized experimental design for this experiment.
Definitions:
An experiment is blind if the subjects involved in the study do not know which treatment they are receiving. An experiment
is double blind if neither the experimenter or the subjects know who is receiving which treatment.
Question: Can you think of a type of experiment that might require the subjects to be blind?
The experiments we’ve examined so far have been examples of completely randomized comparative designs. We will now
examine two other types of designs.
Definition (text):
A block is a group of experimental units or subjects that are known before the experiment to be similar in some way that is
expected to affect the response to the treatments. In a block design, the random assignment of units to treatments is carried
out separately within each block.
Note of importance (text):
In a block design, the assignment of subjects to blocks is not random but the assignment of subjects within each block to the
various treatments is done randomly.
Page 4 of 6
Commonly used variables by which subjects are blocked:
Example 8 (text):
What are the effects of repeated exposure to an advertising message? The answer may depend on the length of the ad and on
how often it is repeated. An experiment investigated this question using undergraduate students as subjects. All subjects
viewed a 40-minute television program that included ads for a 35 mm camera. Some subjects saw a 30-second commercial;
others, a 90-second version. The same commercial was repeated 1, 3 or 5 times during the program. After viewing, all of the
subjects answered questions about their recall of the ad, their attitude toward the camera, and their intention to purchase it.
These are the response variables.
a.
How many factors does this experiment have? How many levels are in each factor?
b.
How many different treatments does this experiment have?
c.
Suppose you decide to use a completely randomized design in this experiment. If you have 36 students who will
serve as subjects, outline the design.
d.
Suppose you have 36 subjects: 24 women and 12 men. Men and women often react differently to advertising. You
therefore decide to block design with the two genders as blocks. Outline the design with a diagram.
A special type of block design is called a matched pairs design and an experiment that contains such a design is called a
matched pairs experiment. In a matched pairs experiment, one of two set ups occur:
1.
(the most common example) Subjects are matched with themselves and given two different treatments (in random
order).
Examples: “Before and after experiments,” preference for Coke or Pepsi,
2.
Similar subjects are matched and each assigned a different treatment.
Examples:
Page 5 of 6
Example 9:
Design a matched pairs experiment to determine whether people who drink cola prefer Coke or Pepsi.
Example 10 (Problem #2 from the free response section of the 2002 AP Statistics exam):
A manufacturer of boots plans to conduct an experiment to compare a new method of waterproofing to the current method.
The appearance of the boots is not changed by either method. The company recruits 100 volunteers in Seattle, where it rains
frequently, to wear the boots as they normally would for 6 months. At the end of the 6 months, the boots will be returned to
the company to be evaluated for water damage.
a.
Describe a design for this experiment that uses the 100 volunteers. Include a few sentences on how it would be
implemented.
b.
Could your design be double blind? Explain.
Homework: #5.31-5.39, 5.41-5.45, 5.47-5.53, 5.56, 5.57
Page 6 of 6