Experimental Designs
stat/engl 332
Outline
• Aspects of experimental design:
- treatment, response, randomization, covariates
• Analysis of results
Designing
Which design is ‘best’?
Experiments
Male
80
Thu
OR
0
10
M
20
40
20
0
Thu
Fri
Sat
Day
Sun
Gender
F
Female
Male
30
40
60
50
Female
Male
Thu
Fri
Sat
Day
Sun
Fri
Day
Sat
Sun
What is an experiment?
• Actively control the variation. By controlling all sources of variability, we are able
to assess cause-and-effect in a designed
experiment. • Manipulate an explanatory variable and observe a
response. Then the change in the response can be
attributed to the explanatory variable. • In an observational study we just observe - in an
experiment we control the variables.
Terminology
• Experimental units - who or what the experiment is
performed on, eg People - Subjects, “participants”. • Factors – Explanatory variable. Need at least one factor for every experiment. • Levels (different values of a Factor)
Need at least two levels for each factor
Terminology
• Treatment – The combination of factors and levels given to
experimental units. Each experimental unit receives
one treatment. • Response Variable – experimental units’ response to
the treatment. Could measure more than one
response variable.
Three principles of
Experimental Design
• Control: Only aspect that affects response is treatment. Make
everything as equal as possible for experimental
units. Only difference in treatment of units is the
TREATMENT.
• Randomization: Treatments are applied to units randomly. Controls
for unseen factors. Units do not pick the treatment
they receive.
Three principles of
Experimental Design
• Replication: Within the experiment, each treatment is given to
more than one unit. Entire experiment should be
repeated on a different group of units.
Example: 2d or 3d pie?
• Question: Is it easier for readers to
interpret a 2D pie chart, or a 3D pie chart? • Hypothesis: Readers will better interpret
the values from a 2D pie chart. • Factor: Type of pie chart (levels: 2D, 3D) • Response: Percentages reported from each
category. Example: 2d or 3d pie?
• Experimental units: Subjects available to
complete the study. (how do we get them?)
• Randomization: The subjects will be
randomly divided into two groups and
assigned to report on one of the two
charts. • Control: Each subject will have 10s to
respond. Each will have a view of the chart
at the same distance and same angle. Let’s do that!
Everyone gets a
Everyone gets a
number …
number
1
2
3
5
4
6
7
7
Check
the
of
Check
themake-up
make-up
the two
groups
of the
groups
Number
Gender
Major
Yrs ISU
#Pets
Number
Gender
Major
Yrs ISU
#Pets
1
M
ACC
3
0
7
M
GR.DES
2
1
2
M
ARCH
5
0
8
F
STU.ART
3
1
3
M
T.COM
3
2
9
M
LIB
5
0
10
M
ACC
5
0
4
M
ACC
5
0
11
F
T.COM
3
0
5
F
ECON
2
0
12
M
ACC
5
0
6
F
ACC
4
0
13
M
HCI
6
1
8
Random assignment
• If your number is 6 or less close your eyes
• If your number is 7 or higher keep your
eyes open
Write down the proportions for each color
ALS$Ice$Bucket$Challenge$
Movember$
Komen$Race$for$the$Cure$
Ride$to$End$AIDS$
Jump$Rope$for$Heart$
Out$of$Darkness$Overnight$Walk$
Step$out:$Walk$to$Stop$Diabetes$
Firght$for$Air$Climb$
• If your number is 6 or less open your eyes
(and keep them open)
• If your number is 7 or higher close your
eyes
Write down the proportions for each color
ALS$Ice$Bucket$Challenge$
ALS$Ice$Bucket$Challenge$
Movember$
Movember$
Komen$Race$for$the$Cure$
Komen$Race$for$the$Cure$
Ride$to$End$AIDS$
Ride$to$End$AIDS$
Jump$Rope$for$Heart$
Jump$Rope$for$Heart$
Out$of$Darkness$Overnight$Walk$
Out$of$Darkness$Overnight$Walk$
Step$out:$Walk$to$Stop$Diabetes$
Step$out:$Walk$to$Stop$Diabetes$
Firght$for$Air$Climb$
Firght$for$Air$Climb$
Record the data
ID
ALS
1
Movem Komen Ride to
ber
Race
end
Jump
Rope
Out of
Darkne
Step
out
Fight
for Air
Design!
2d
2
2d
3
2d
4
2d
5
2d
6
2d
7
3d
8
3d
9
3d
10
3d
11
3d
12
3d
Truth
48.9
23.2
20.9
6.1
0.3
0.3
0.2
0.2
—
Blinding
• Subjects should not know which treatment
they receive. Could inadvertently influence response. • Researchers in contact with subjects should
not know which treatment subject receives.
Could influence way subjects are treated. • Experiment should have both blindings. Called
double-blind.
Confounding Variables
• Instructor wants to determine effect of teaching
style on student’s attitude about class. • Example:
• Factor – Teaching style (1) Animated (spring
semester) (2) Laid-back (fall semester)
• Response – student attitudes about course.
• Teaching style is confounded with semester. No
way to tease apart whether difference in student
response is due to semester or to teaching style.
Process
• Determine a question of interest • Define the treatments, and response
variable • How will you measure the response? • Identify the subjects, and a scheme for
assigning subjects randomly to treatment
group
•
Example
Example
•
Compute difference between reported percentage and
BRIEF ARTICLE
percentage
for
each
subject.
Compute difference between reported percentage
truea percentage
fordifferences
each subject
Make
dot plot of the
for each of the treatm
•and
groups.
THE AUTHOR
(dottplot or boxplot by treatment)
••Visualize!
Calculate mean and standard deviation for each treatm
• Calculate
group. mean and standard deviation by treatment
x̄
x̄
1
2
Compute
the
test
statistic:
•
Compute
the
test
statistic
•
s
s
2
1
n1
!
+
2
2
n2
the statistical significance (table or
••Determine
Determine the statistical significance, by (1) “scramblin
permutation test)
treatment labels, many times (2) computing the test stat
each, (3) counting the proportion of times the real test s
Example
Example
8
8
6
6
4
Difference
Difference
4
2
2
0
-2
0
-4
-2
-6
2D
-4
3D
Treatment
-6
2D
3D
Treatment
•
Visually not much
difference between
2D and 3D times.
Example
Mean
SD
2D
3D
1.10
0.49
3.10
2.25
20 times and compare the
test statistics for the random
samples with the real data.
• The test statistic for the real
data is within the range of
those observed for random
data.
0.4
0.2
• There is no statistically
0.0
tstat
• Test statistic=0.40
• We’ll scramble the labels
-0.2
-0.4
-0.6
scrambled
type
true
significant difference
between the accuracy in
reading % from 2D or 3D
pie charts.
19
Examining
the
response
Examining the response
•
Compute summary statistics for the response
separately for each treatment, and make plots.
Treatment
2
Mean 183.1 138.2 201.0
SD
12.1
9.3
200
3
14.9
Response
1
220
180
160
140
Means plotted (as dots, point
Treatment
estimates), and intervals
extend out 2 SD.
Treatment 2 is significantly different
from treatments 1,3 because the
intervals do NOT overlap.
120
1
2
3
What if there are
differences?
• Average response is different between
treatment groups. • The difference could be due to treatment, or it
could be due to random variation. • Differences are statistically significant if the
differences are large enough not to be due to
random variation.