LISA: BASIC PRINCIPLES OF
EXPERIMENTAL DESIGN
By Chris Franck
1
Laboratory for Interdisciplinary Statistical
Analysis
LISA helps VT researchers benefit from the use of
Statistics
Collaboration:
Visit our website to request personalized statistical advice and assistance with:
Experimental Design • Data Analysis • Interpreting Results
Grant Proposals • Software (R, SAS, JMP, SPSS...)
LISA statistical collaborators aim to explain concepts in ways useful for your research.
Great advice right now: Meet with LISA before collecting your data.
LISA also offers:
Educational Short Courses: Designed to help graduate students apply statistics in their research
Walk-In Consulting: M-F 12-2PM in 401 Hutcheson Hall for questions requiring <30 mins
All services are FREE for VT researchers. We assist with research—not class projects or homework.
www.lisa.stat.vt.edu
2
Laboratory for Interdisciplinary Statistical
Analysis
To request a collaboration meeting go to
www.lisa.stat.vt.edu
www.lisa.stat.vt.edu
3
Laboratory for Interdisciplinary Statistical
Analysis
To request a collaboration meeting go to www.lisa.stat.vt.edu
1. Sign in to the website using your VT PID and password.
2. Enter your information (email address, college, etc.)
3. Describe your project (project title, research goals,
specific research questions, if you have already collected
data, special requests, etc.)
4. Wait 0-3 days, then contact the LISA collaborators
assigned to your project to schedule an initial meeting.
www.lisa.stat.vt.edu
4
About LISA
 Laboratory for Interdisciplinary Statistical
Analysis
 www.lisa.stat.vt.edu
 FREE services: Collaboration, walk-in consulting,
short courses
 We are here to help you! Goal is to contribute to
good research across the Virginia Tech
community.
5
Eric Vance
Director
Tonya Pruitt
Administrative
Superstar
Chris Franck
Assistant Director
6
5 Lead Collaborators (20 hours/week)
7
Structure of talk
 PLEASE SIGN THE SIGN IN SHEET!
 Cover fundamental aspects of experimental
design.
 Scenarios come from consulting experience.
 Highlight LISA services available.
 Key words in red
 ?? Interesting questions.
8
Central messages
If you face statistical uncertainty at any stage of
your research, please come to LISA.
Best time to involve the statisticians: Before the
data has even been collected.
Speaking of the pre-data collection phase…
9
Why are we here?
 Choices made at the design stage have the
potential to drastically impact the results of
any study.
 Good experimental design gives the
researcher an improved chance of a
successful experiment.
 A poorly considered or implemented design
can have a ruinous effect on the
investigation.
10
The plan
 We will discuss basic elements of
experimental design: randomization,
replication, and blocking.
 Real world experiments.
 Interpretation of experimental results will be
compared and contrasted with interpretation
of results from observational studies.
11
A note on examples
presented.
 Examples chosen present unique challenges
and complications – chosen from hundreds of
collaborations.
 Magnitude of challenges in these examples is
greater than what is typical for a LISA
collaboration.
 Worst case scenarios!
12
13
Study design: food science
 Research question: Among three genetic
varieties of sweet potatoes, which type will
brown the least when fried? Also take
storage time into account.
 Measurement of browning done with a
machine – beyond the scope of this talk.
 The following graphic shows the design
layout.
14
Sweet potato design
Cook
Order
1
2
3
4
5
6
7
8
9
1
■
■
■
■
■
■
■
■
■
2
■
■
■
■
■
■
■
■
■
3
■
■
■
■
■
■
■
■
■
4
■
■
■
■
■
■
■
■
■
Legend
■
■
■
Week
5
■
■
■
■
■
■
■
■
■
6
■
■
■
■
■
■
■
■
■
7
■
■
■
■
■
■
■
■
■
8
■
■
■
■
■
■
■
■
■
9
■
■
■
■
■
■
■
■
■
Cultivar 1
Cultivar 2
Cultivar 3
Oil Change
15
Features of the design
 Suppose we conduct this experiment and
conclude the third variety browns the most,
and the first variety browns the least.
 Is this necessarily due to the genetic
differences in the potato types?
 Is there another plausible explanation?
16
What about cooking order?
 Notice that for a given week, all of the
potatoes are cooked in the same oil.
 Also, the varieties are always cooked in the
same order, making the effect of variety and
the effect of cook order inseparable. The
effect of variety on browning is confounded
with the effect cooking order has on
browning.
17
A randomized design
Cook
Order
1
2
3
4
5
6
7
8
9
1
■
■
■
■
■
■
■
■
■
2
■
■
■
■
■
■
■
■
■
3
■
■
■
■
■
■
■
■
■
Legend
■
■
■
4
■
■
■
■
■
■
■
■
■
Week
5
■
■
■
■
■
■
■
■
■
6
■
■
■
■
■
■
■
■
■
7
■
■
■
■
■
■
■
■
■
8
■
■
■
■
■
■
■
■
■
9
■
■
■
■
■
■
■
■
■
Cultivar 1
Cultivar 2
Cultivar 3
Oil Change
18
Why randomize?
 Randomization is a fundamental feature of
good experimental design.
 In this case, randomization will eliminate the
known confound between cooking order and
potato variety.
 Randomization makes groups similar on
average, and hence eliminates unknown
confounding effects as well!
19
Sweet potato remarks
 Since the effect of interest (genetic variety) is
confounded with cooking order in the current
experiment, recommendation is to repeat the
experiment with a randomized design.
 Many randomized designs exist! Perhaps
changing oil more frequently can also
improve the project.
20
Costly
 In general randomization is not difficult to
perform.
 In this case the cost of repeating experiment
in randomized fashion was moderate (about
12 weeks time + materials).
 Repeating an experiment can be VERY costly.
(3 years + materials - PhD research)
21
Another randomization example
 A professor observes that students who sit in
the front of a large lecture class tend to get
better grades. Can she conclude that sitting
in front causes students to get better grades?
22
HW problem
 A grape researcher is interested in testing the
effect of 4 pesticides on the disease rate on
his grapes. For his experiment he has 16 total
vines arranged in four plots. Each vine has a
trunk at the center and two cordons
extending from the trunk. Many grape
clusters grow on each cordon.
23
24
Basic grape vine anatomy
25
How to assign pesticides?
 To administer the pesticides, the researcher
randomly assigns one pesticide (labeled A, B,
C, and D) to each of the plots. He then sprays
the assigned pesticide on all four vines in
each plot, walking from north to south in
each case.
 Call pesticide treatment.
26
How many replicates for each
treatment?
27
How many reps for each
treatment?
 A) 4 reps/treatment since there are four vines
that receive each pesticide.
 B)8 reps/pesticide since there are eight
cordons that receive a given treatment.
 C) Many replicates: depends on the number
of grapes which grow, since each grape might
or might not have the disease.
 D) Something else.
28
Answer:
 Number of experimental replicates:
 Why!? What went wrong?
 The experimental unit is the smallest unit in
the experiment to which separate treatment
assignments are made.
 What was the experimental unit in this
experiment?
29
Definition of replicate
 The number of replicates for a given
treatment is equal to the number of times the
treatment was assigned to the experimental
unit.
30
Consequences of this design
 We cannot perform usual statistical inference
in this experiment. That is, we cannot
perform hypothesis tests, construct
confidence intervals, etc.
 The resulting data might suggest a difference
in the treatments, but we can’t quantify the
uncertainty of the results with confidence
levels, p-values, etc.
31
Improvements
 Instead of using 4 total plots, we might use 8,
12, 16, etc. This would give 2,3,4 replicates
per treatment.
 Instead of randomizing the treatments to the
plots, perhaps we can randomize the
treatments to the vines themselves?
32
33
Randomizing treatments to
vines
 Now the vine is the experimental unit.
 4 replicates for each treatment instead of 1.
 ?? What if our treatment is sprayed on the
vines in such a way that adjacent vines get a
little bit of the wrong treatment? Windy day?
 This is an example of a carryover effect – we
can address these advanced issues at LISA
collaboration meeting.
34
Another replication example
 A researcher is developing a compound
needed to create some advanced textiles.
Four temperature levels are crossed with 4
molecular compounds. Each combination of
temperature and compound is randomized to
a flask, and five samples are taken from each
flask for measurements.
 Interaction between temperature and compound
is of primary interest.
35
Umm, Chris, did you really
randomize?
 Plot 1 has three instances of D!
 Not one single plot has each of the
treatments!
 What if plot 1 has ideal characteristics?
(irrigation, soil quality, sunlight)
 ?? Won’t treatment D seem better than it
otherwise would since it appears in the best
plot three times?
36
37
Yes, and Yes
 I did randomize, using a completely
randomized design.
 Yes, if plot 1 has different characteristics than
the other plots, and D appears frequently by
chance in plot 1, then the three observations
on treatment D are a function of both the
treatment and the enhanced plot
characteristics.
38
In general
 If the plots 1,2,3 and 4 have are not identical
in terms of the response (disease rate), then
plot to plot or inter-plot variability is present.
 Maybe some of the plots are on inclines, soil
characteristics may be different, etc.
 Call the impact of the different plots on the
response the plot effect.
39
How do we handle the plot
effect?
 We are interested in the disease rates of
grapes. We believe the various treatments
will affect these disease rates. We also
believe that the plots will also have an impact
on the disease rates.
 We don’t really care about the plot effect – our
primary goal is to determine how the
treatments affect the response. Plots are
simply an extra source of variability
40
Treat the plots as blocks
 Blocking is a strategy that may be
implemented in order to account for known
sources of variability in the experimental
material.
 In our case, the plots may show variability we
wish to account for.
41
To implement blocking
 Blocking is implemented during the design
phase of the experiment.
 We want to assign the treatments into the
blocks so that each treatment appears in
each block exactly one time.
 Assigning treatments to blocks in this fashion
is a form of restricted randomization.
42
Randomized complete block
design
43
Notes about RCBD
 Notice each treatment appears in each block
exactly once.
 Statistical model for this design:
yij = μ + αi + βj + εij
i=1,…,a is the number of treatments (4)
j=1,…,b is the number of blocks (4)
yij is the response for the ith treatment, jth
block.
44
More terms
 μ is the overall mean.
 αi is the treatment effect for the ith
treatment.
 βj is the effect of the jth block.
 εij is an error term associated with response at
ith treatment and jth block.
45
Take home message
 By implementing the randomized complete
block design, we can:
 Compare the performance of the four treatments.
 Account for variability in the plots that might
otherwise obscure the treatment effects.
 But suppose you did not randomize the
treatments into blocks before collecting data.
Can I still use above technique?
46
Can I?
 Maybe, maybe not.
 If you used a completely randomized design (as I
did initially), you may not be able to fit the RCBD
model!
 This is because some of the parameters in the
model may be non-estimable depending on how
your randomization works out e.g. in completely
randomized design, no information about how
treatment A behaves in plot 1.
 Come see LISA when designing experiments!
47
Row effects?
 ?? RCBD seems good, but what if I also have a
row effect in addition to a plot effect.
 E.g. perhaps there is a fertility gradient within
each plot.
 Treatment C appears three times in the
second row in RCBD plot. Don’t we have the
same problem even with RCBD.
 Answer:
48
Latin Square layout
49
Another blocking example–
hemophilia project
 Hemophilia refers to a set of genetic
disorders which impairs an individual’s blood
from clotting.
 Hemophiliacs do not have the ability to
produce certain proteins which are needed
for blood clotting.
 In our project we study Factor 9 (F.IX).
50
Chapel Hill hemophilia dogs
Hemophilia A
Hemophilia B
Hemophilia overview
 This work is done in collaboration with The
Francis Owen Blood Research Laboratory
(FOBRL) in Chapel Hill, NC.
 FOBRL maintains a colony of hemophilic
dogs which are used for animal models.
 Gene therapy – Use a modified virus to insert
genes into an individual’s cells and tissues to
treat hemophilia.
52
Gene therapy outline
 Development of gene therapies is an involved and
expensive process.
 Hemophilic dogs are given doses of gene therapy
vector with the hope that this therapy will give the
dog the ability to produce the missing factor.
 If the dog is producing trace amounts of the factor
then the dose of the vector can be increased. This is
less expensive than developing a new gene therapy.
This is also why it is vital to have a test that can
detect the factor with high sensitivity.
53
Measuring clotting time
 Activated Partial Thromboplastin Time
(APTT) – expensive, highly technical assay
which relies on processed blood plasma.
 Whole Blood Clotting time (WBCT) – Less
expensive simpler assay which is performed
on unprocessed blood.
54
Whole blood clotting time
procedure
 Draw 1 mL of blood from subject. divide
between two test tubes in 28◦ C water bath.
 Incubate tubes for 1 minute, then begin
tipping the first tube every 30 seconds.
 As soon as a clot forms in the first tube, begin
tipping the second tube every 30 seconds.
 When the second tube forms a clot, the total
time is the WBCT.
55
Research question
 What percentage of F.IX (relative to a non-
hemophilic individual) can be detected by the
WBCT procedure?
 More specifically, is the clotting time for
WBCT assay significantly shorter than 60
minutes (baseline for untreated dog) at 0.01%
F.IX?
56
Randomized complete block
design
 4 dogs each with 10 concentrations of F.IX.
 4*10 = 40 data points.
 Each clotting time modeled as a function of
overall mean, F.IX concentration, dog, and
random error.
 What is the treatment? The blocks?
57
Statistical formulation of
research hypotheses
 The null hypothesis (denoted H0) is the
hypothesis of no treatment effect.
 The alternative hypothesis (denoted HA) is
the hypothesis that the treatment has an
effect.
 H0: F.IX concentration does not have an effect on
WBCT.
 HA: F.iX concentration does have an effect on
WBCT.
58
59
Analysis of F.IX project
 SAS was used to conduct this statistical
analysis.
 We found that concentration had a significant
effect on WBCT (p-value < 0.001).
 We found that even at the lowest concentration
present (0.01% F.IX) the average clotting time was
significantly less than 60 minutes. (p-value
<0.001).
 These p-values represent the probability data
as “extreme” as ours would arise if H0 were
true.
60
Hemophilia wrap-up
 The WBCT assay yields clotting times lower
than 60 minutes on average even for F.IX
dilutions as low as 0.01%.
 Hence this procedure is very sensitive to the
presence of F.IX.
 This is useful information since it helps guide
researchers when deciding whether to
increase the dose of the current gene therapy
vector instead of developing a new gene
therapy.
61
Strawberry Cover Example
 Data was collected on strawberry yields: Row
covers and spring covers.
 Covers help regulate the temperature for
strawberries, allow sunlight in, and protect
from frost. Microclimate.
 Extend the growing season.
 Client appeared Thursday, had a conference
Monday.
62
Strawberry rows
63
Strawberry row covers
64
Experimental layout
 16 plots in four blocks.
 Plots randomly assigned to one of following
treatments: No row cover (control), or one of
three application dates of row covers (D1, D2,
D3).
 Each plot was also split, and a Spring row
cover regimen was applied to half of the plot
chosen randomly.
65
Factorial arrangement of
cover
66
Split each plot so we can
assign Spring cover treatment
67
Each plot split into two,
another experiment inside
68
What is this experiment
called?
69
More advanced example –
Split plot with Blocks
 Split plot designs are somewhat advanced –
unlikely in an introductory course!
 Main feature: One experiment inside another.
 Can involve blocking terms.
 Model for split plot with blocks:
 Yijk = μ + αi + Rk + (αR)ik + βj + (αβ)ij + Eijk
 Greek letters fixed effects, Capital Roman letters
random effects
70
Yijk = μ + αi + Rk + (αR)ik + βj + (αβ)ij + Eijk
 Y is response, i= 1,…,a, j=1,…,b, k=1,…, r.
 μ is overall mean, αi, are whole plot treatment
effects, βj are split plot treatment effects, (αβ)ij
are whole plot by split plot interaction effects.
Assume sum to zero constraints.
 Rk represents the block effects, (αR)ik represents
whole plot by block interaction, Eijk represents
residual error. These random effects have
normal distributions with appropriate variance
components.
71
Analyzing Split plot with
blocks model
 Not trivially easy, but not really hard either.
 However, the above experimental description
differs from the experiment which was
actually conducted!
72
Forgot to add split plot
treatment to control plots!
73
Now what!?
74
Analyzing full data set not
so simple…
 Expert opinion suggested the spring cover
may not be particularly useful for improving
yield.
 What if we consider only the non-control
plots, and then test for a spring cover effect?
75
What type of experiment is
this?
76
Strawberry Wrap-up
 Concluded no spring cover effect. Considered
the full data set ignoring spring cover term,
analyzed as two way factorial experiment
with subsamples. Sufficient.
 Was this the optimal experimental design?
 Could the be a more optimal statistical
approach?
 Deadline was met – conference presentation
went well.
77
One benefit of experiments
vs. observational studies
 In general, experiments provide more
evidence of causal relationships between
variables.
 Observational studies can show associations
between variables, but are NOT sufficient to
demonstrate causality.
 E.g. Survey grape farms, ask which treatment
they use to control disease, find one
treatment better than others. Causal?
78
Conclusions
 The design phase of an experiment is a crucial
time to plan carefully. Careful design sets the
stage for success. Overlooking this stage can
lead to disastrous results.
 LISA can help you design experiments.
 Nobody (including LISA) can fully rescue an
experiment that has design flaws.
79
Other LISA short courses
Date
Course Title
Instructor
02/01/2011
Basic Principles of Experimental Design
Chris Franck
02/07/2011
Using JMP for Statistical Analysis Part I
Wandi Huang
02/08/2011
Using JMP for Statistical Analysis Part II
Wandi Huang
02/15/2011
Regression
Jennifer Kensler
02/21/2011
Intro to SAS
Mark Seiss
02/22/2011
Intro to SAS
Mark Seiss
02/28/2011
Introduction to R
Sai Wang
03/01/2011
Introduction to R
Sai Wang
03/14/2011
Bayesian Methods for Regression in R
Nels Johnson
03/15/2011
Bayesian Methods for Regression in R
Nels Johnson
80
Thanks!
81