Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecturer:
Dr. Michael Stuart,
Department of Statistics
email: mstuart@tcd.ie
Lectures:
Tuesday, Thursday,
Laboratory: Thursday, March 12th,
Tuesday, March 31st,
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
6.00 - 8.00pm
6.00 - 8.00pm
6.00 - 8.00pm
Lecture 1.1
1
© 2015 Michael Stuart
Design and Analysis of Experiments
Course Outline
• The need for experiments
– experimental and observational studies
– cause and effect
– control
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
2
© 2015 Michael Stuart
Design and Analysis of Experiments
Course Outline
• Basic design principles for experiments
– Control
– Blocking (pairing)
– Randomization
– Replication
– Factorial structure
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
3
© 2015 Michael Stuart
Design and Analysis of Experiments
Course Outline
• Standard designs
– Completely randomized designs
– Randomized blocks
– Two-level factors
– Split units
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
4
© 2015 Michael Stuart
Design and Analysis of Experiments
Course Outline
• Analysis of experimental data
– Exploratory data analysis
– Effect estimation and significance testing
– Analysis of variance
– Statistical models, fixed and random effects
– Model validation, diagnostics
– Software laboratories
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
5
© 2015 Michael Stuart
Design and Analysis of Experiments
References
Mullins, E., Statistics for the Quality Control
Chemistry Laboratory, Royal Society of Chemistry,
2003, particularly Chapters 4-5, 7-8.
(EM)
Available as an electronic resource
Montgomery, D.C., Design and analysis of
experiments, 8th ed., Wiley, 2013.
(DCM)
Dean, Angela and Voss, Daniel, Design and analysis
of experiments, Springer, 1999.
(DV)
Available as an electronic resource
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
6
© 2015 Michael Stuart
Design and Analysis of Experiments
Further reading
Box, G.E.P, Hunter, J.S. and Hunter, W.G., Statistics
for Experimenters, 2nd. ed., Wiley, 2005.
(BHH)
Daniel, C., Applications of Statistics to Industrial
Experimentation, Wiley, 1976.
(CD)
Mead, R., Gilmour, SG and Mead, A, Statistical
Principles for the Design of Experiments:
Applications to Real Experiments, Cambridge
University Press, 2012.
(MGM)
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
7
© 2015 Michael Stuart
Design and Analysis of Experiments
Lecture notes and supplements
Module Web Page
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
8
© 2015 Michael Stuart
Assessment
• 3-hour written examination
– 3 questions. Questions 1 and 2 carry 30 marks
each, Question 3 carries 40 marks.
– Appendix gives tables of critical values of the
t-distribution and selected critical values of the
F distribution.
– Non-programmable calculators are permitted
for this examination
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
9
© 2015 Michael Stuart
Assessment
Examination dates:
Monday 27 April to Friday 22 May 2014 inclusive
Examination Timetables will be available in March
"The onus lies on each student to establish the dates,
times and venues of examinations by consulting the
relevant timetable on the College website. No timetable
or reminder will be sent to individual students by any
office."
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
10
© 2015 Michael Stuart
Course assessments
• Module assessment, as for Base Module
• End of Lecture, Minute Tests
– How much did you get out of today's class?
– How did you find the pace of today's class?
– What single point caused you the most
difficulty?
– What single change by the lecturer would have
most improved this class?
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
11
© 2015 Michael Stuart
Lecture 1.1
1. Introduction to Course
2. What is an experiment?
3. Case study: Industrial process improvement
− three design principles
4. Case study: Clinical trial
− how randomization works
5. Multifactor Designs
6. Other application areas
7. Experimental vs Observational Studies
8. Strategies for Experimentation
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
12
© 2015 Michael Stuart
Part 2
What is an experiment?
Try something, to see what happens
Try something different, to see the difference in
what happens
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
13
© 2015 Michael Stuart
Experiment as demonstration
Pendulum
– length L
– period T
T  2
L
g
L
g  4 2
T
2
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
14
© 2015 Michael Stuart
Newton's colour demonstration
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
15
© 2015 Michael Stuart
Newton's colour demonstration
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
16
© 2015 Michael Stuart
Newton's colour demonstration
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
17
© 2015 Michael Stuart
Thought experiments
• Aristotle (4th century BC):
– speed of falling objects is proportional to
weight
• Galileo (17th century AD):
– not true!
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
18
© 2015 Michael Stuart
Comparative experiments
• Galileo's pendulum experiments
• A comparative experiment is a programme of
actions undertaken to study the effects of making
changes to a process or system.
• “To find out what happens when you change
something, it is necessary to change it”.
(BHH, p. 404)
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
19
© 2015 Michael Stuart
Control, a key feature of
comparative experiments
• Complete control
– the counterfactual argument
• Practical control of study environment
– chance variation if no change introduced
– comparing results of change to no change involves
a test of statistical significance
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
20
© 2015 Michael Stuart
Lecture 1.1
1. Introduction to Course
2. What is an experiment?
3. Case study: Industrial process improvement
− three design principles
4. Case study: Clinical trial
− how randomization works
5. Multifactor Designs
6. Other application areas
7. Experimental vs Observational Studies
8. Strategies for Experimentation
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
21
© 2015 Michael Stuart
Part 3
Case study on process improvement
•
Comparison of standard (old) process and new
process for manufacture of electronic
components
•
Key criterion:
– number of defective components
Ref:
EM Notes, Ch 4, Example 1, pp. 3-6
Hahn.xls
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
22
© 2015 Michael Stuart
Experimental design
• 50 components sampled per day,
• 6 days per week,
• 8 weeks,
• Systematic layout, as follows
Week Number
1
2
3
4
5
6
7
8
Monday
Old
New
Old
New
Old
New
Old
New
Tuesday
New
Old
New
Old
New
Old
New
Old
Wednesday
Old
New
Old
New
Old
New
Old
New
Thursday
New
Old
New
Old
New
Old
New
Old
Friday
Old
New
Old
New
Old
New
Old
New
Saturday
New
Old
New
Old
New
Old
New
Old
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
23
© 2015 Michael Stuart
Sampling plan
50 components sampled per day
Measurement:
X = number of defectives in sample of 50
Why 50?
Why not 1?
100?
the whole lot?
For fair comparison, let p = X/n
SE(p) =
(1   )
n
Ref: EM Notes Ch 3 p 2
Measurement precision
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
24
© 2015 Michael Stuart
Results
Numbers of defectives per daily sample of 50
for 48 days (8 weeks)
Day Defectives
1
0
2
0
3
6
4
3
5
3
6
3
7
4
8
1
9
0
10
2
11
0
12
0
Day Defectives
13
1
14
0
15
3
16
1
17
0
18
2
19
0
20
1
21
2
22
0
23
1
24
3
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Day Defectives
25
0
26
0
27
0
28
2
29
0
30
0
31
1
32
1
33
0
34
0
35
0
36
2
Day Defectives
37
2
38
0
39
0
40
0
41
0
42
0
43
1
44
0
45
2
46
0
47
0
48
0
Lecture 1.1
25
© 2015 Michael Stuart
Comparison of two processes
over eight weeks
Numbers of Defectives
in Samples of 50 Units
Day
Old
New
Difference
pair Process Process (New – Old)
1
0
0
0
2
6
3
–3
3
3
3
0
4
1
4
+3
5
2
0
–2
6
0
0
0
7
1
0
–1
8
3
1
–2
9
0
2
+2
10
1
0
–1
11
0
2
+2
12
3
1
–2
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Numbers of Defectives
in Samples of 50 Units
Day
Old
New
Difference
pair Process Process (New – Old)
13
0
0
0
14
0
2
+2
15
0
0
0
16
1
1
0
17
0
0
0
18
2
0
–2
19
2
0
–2
20
0
0
0
21
0
0
0
22
0
1
+1
23
0
2
+2
24
0
0
0
Lecture 1.1
26
© 2015 Michael Stuart
Comparison of two processes
over eight weeks
Numbers of Defectives
Summary
Old
Process
New
Process
Difference
(New – Old)
25
22
–3
2.08
1.83
–0.25
Total
8 week averages
per cent
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
27
© 2015 Michael Stuart
Differences in numbers defective,
with control limits
8
6
4
2
Difference
0
-2
-4
-6
-8
4
8
12
16
20
24
Day Pair
No statistical significance!
Ref: EM Notes Ch 1 § 1.7
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
28
© 2015 Michael Stuart
Calculating the control limits
Numbers of Defectives
in Samples of 50 Units
Day
Old
New
Difference
pair Process Process (New – Old)
1
0
0
0
2
6
3
–3
3
3
3
0
4
1
4
+3
5
2
0
–2
6
0
0
0
7
1
0
–1
8
3
1
–2
9
0
2
+2
10
1
0
–1
11
0
2
+2
12
3
1
–2
SD(Differences) = 1.57
Numbers of Defectives
in Samples of 50 Units
Day
Old
New
Difference
pair Process Process (New – Old)
13
0
0
0
14
0
2
+2
15
0
0
0
16
1
1
0
17
0
0
0
18
2
0
–2
19
2
0
–2
20
0
0
0
21
0
0
0
22
0
1
+1
23
0
2
+2
24
0
0
0
Control limits: 0  3xSD =  4.7
Ref: hahn.xls
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
29
© 2015 Michael Stuart
Formal significance test
D0
Z
SE(D )

D
SD / n
From Summary table, sum of differences = – 3
From control limit calculation, SD = 1.57
Z
 3 / 24
1.57 / 24
 0.39
-4
-3
-2
-1
0
1
2
3
4
not statistically significant
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
30
© 2015 Michael Stuart
Alternative design
(proposed by engineers)
Week Number
1
2
3
4
5
6
7
8
Monday
Old
Old
Old
Old
New
New
New
New
Tuesday
Old
Old
Old
Old
New
New
New
New
Wednesday
Old
Old
Old
Old
New
New
New
New
Thursday
Old
Old
Old
Old
New
New
New
New
Friday
Old
Old
Old
Old
New
New
New
New
Saturday
Old
Old
Old
Old
New
New
New
New
Assume this design was used;
check for no effect
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
31
© 2015 Michael Stuart
Defect rates, per cent, with differences,
for the first and second four week periods
First
Period
Second
Period
Difference
Both Processes
3.0
0.9
2.1
Old Process
3.3
0.8
2.5
New Process
2.7
1.0
1.7
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
32
© 2015 Michael Stuart
Testing statistical significance
P̂  P̂
P̂P̂11 1P̂
P̂P̂22 2
Z

P̂
Z
 P̂  (100  1P̂ ) 2 P̂  (100  P̂ )
Z
1 (100 
1 P̂
2 (100
2
P̂
)


P̂
)
P̂

P̂
P̂

(
100

P̂
)

1
1
2
2
1
1
2
2
P̂1  (100  P̂1 ) P̂2  (100  P̂2 )

n1
n2
n
n
n
n1
n 22
1
Ref: EM Notes Ch 3 p 11
2
.000.0.99.9
333.3.00
.0  0 .9
 3397
97
9
99
99
.1
99
.1..11
3 97
9700.0
0.99..9
99
 1200
1200
1200
1200
1200 1200
1200
2
2...111
2


00..56
56
 33..75
75
-4
-3
-2
-1
0
1
2
3
4
highly statistically significant!
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
33
© 2015 Michael Stuart
Classwork 1.1.1
Assess the statistical significance of the difference
in defect rates, %, between the first period and
second period for the old process.
Homework 1.1.1
Assess the statistical significance of the difference
in defect rates, %, between the first period and
second period for the new process.
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
34
© 2015 Michael Stuart
How can this be?
Numbers defective in time order
6
5
4
Defectives
3
2
1
0
6
12
18
24
30
36
42
48
Day
Long term downward trend,
systematic bias
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
35
© 2015 Michael Stuart
How to avoid systematic bias
• Make comparisons under
homogeneous experimental conditions
• 1
Systematic arrangement, as implemented:
avoids known biases
• 2
Random allocation:
within each day pair, allocate old and new
processes at random
avoids known and unknown biases
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
36
© 2015 Michael Stuart
Random vs Systematic allocation
Week Number
1
2
3
4
5
6
7
8
Monday
Old
New
Old
New
Old
New
Old
New
Tuesday
New
Old
New
Old
New
Old
New
Old
Wednesday
Old
New
Old
New
Old
New
Old
New
Thursday
New
Old
New
Old
New
Old
New
Old
Friday
Old
New
Old
New
Old
New
Old
New
Saturday
New
Old
New
Old
New
Old
New
Old
Suppose there is an additional "other factor",
unknown to the experimenter
with settings Up, Down,
settings alternate every day, including Sunday
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
37
© 2015 Michael Stuart
Random vs Systematic allocation
Week 1
Week 2
Experimental
Factor
Other
Factor
Experimental
Factor
Other
Factor
Monday
Old
Up
New
Down
Tuesday
New
Down
Old
Up
Wednesday
Old
Up
New
Down
Thursday
New
Down
Old
Up
Friday
Old
Up
New
Down
Saturday
New
Down
Old
Up
Sunday
Up
Old and Up always coincide,
New and Down always coincide.
Factors are "confounded"
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
38
© 2015 Michael Stuart
Random vs Systematic allocation
Random allocation minimises chances that
experimental factor settings pattern
coincides with
other factor settings pattern.
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
39
© 2015 Michael Stuart
Two design principles
• Blocking (or local control)
– identify homogeneous blocks of experimental
units
– assess effects of experimental change within
homogeneous blocks
– average effects across blocks
• Randomization
– allocate experimental settings to units at
random
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
40
© 2015 Michael Stuart
Another design principle
• Replication
– 24 comparisons
• Why 24
• Why not 1? 50? 100?
SD( D )   / n
– power calculation
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
41
© 2015 Michael Stuart
Lecture 1.1
1. Introduction to Course
2. What is an experiment?
3. Case study: Industrial process improvement
− three design principles
4. Case study: Clinical trial
− how randomization works
5. Multifactor Designs
6. Other application areas
7. Experimental vs Observational Studies
8. Strategies for Experimentation
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
42
© 2015 Michael Stuart
Part 4
Clinical trial of heart disease treatments
• 596 patients suffering from heart disease
• to be treated by drugs or by surgery
• each patient assigned at random to one treatment
– 310 (52%) assigned to Drugs
– 286 (48%) assigned to Surgery
• Was the randomization successful?
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
43
© 2015 Michael Stuart
Was the randomization fair?
Z

P̂  50
P̂  (100  P̂) / n
52  50
52  48 / 596
2

2.05
 0.98
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
44
© 2015 Michael Stuart
Balance with respect to Covariates
Drugs
per cent
Surgery
per cent
Limitation in ordinary activity
94
95
History of heart attack
59
64
Heart attack indicated by electrocardiogram
36
41
Duration of chest pain >25 months
50
52
History of high blood pressure
30
28
History of congestive heart failure
8.4
5.2
History of stroke
3.2
2.1
History of diabetes
13
12
Enlarged heart
10
12
High serum cholesterol
32
21
Covariate
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
45
© 2015 Michael Stuart
Balance with respect to Covariates
Drugs
Surgery
per cent
per cent
Limitation in ordinary activity
94
95
-0.5
Histroy of heart attack
59
64
-1.3
Heart attack indicated by electrocardiogram
36
41
-1.3
Duration of chest pain >25 months
50
52
-0.5
History of high blood pressure
30
28
0.5
History of congestive heart failure
8.4
5.2
1.6
History of stroke
3.2
2.1
0.8
History of diabetes
13
12
0.4
Enlarged heart
10
12
-0.8
High serum cholesterol
32
21
3.1
Covariate
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Z(Diff-0)
Lecture 1.1
46
© 2015 Michael Stuart
How randomization works
• Balance with respect to
– known covariates
AND
− unknown covariates
(not achieved by systematic assignment)
• Minimize experimenter bias
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
47
© 2015 Michael Stuart
1. Class count
2. Random number
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
48
© 2015 Michael Stuart
Lecture 1.1
1. Introduction to Course
2. What is an experiment?
3. Case study: Industrial process improvement
− three design principles
4. Case study: Clinical trial
− how randomization works
5. Multifactor Designs
6. Other application areas
7. Experimental vs Observational Studies
8. Strategies for Experimentation
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
49
© 2015 Michael Stuart
Part 5
Multi-factor Designs
• Traditional versus statistical design
– efficiency
– interaction
Ref: EM §5.2
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
50
© 2015 Michael Stuart
Multi-factor designs are efficient
Illustration:
• Yield of a chemical manufacturing process
affected by
– operating pressure,
– operating temperature
• Choose between
– Low and High pressure
– Low and High temperature
• Resources available for 12 experimental runs
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
51
© 2015 Michael Stuart
Traditional “one-at-a-time” design,
Y5 Y6 Y7 Y8
High
Pressure
Low
(best)
Y1 Y2 Y3 Y4
Low
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Temperature
High
Y9 Y10 Y11 Y12
Lecture 1.1
52
© 2015 Michael Stuart
Fisher’s two-factor design
Y10 Y11 Y12
Y7 Y8 Y9
High
Pressure
Low
Y1 Y2 Y3
Low
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Temperature
High
Y4 Y5 Y6
Lecture 1.1
53
© 2015 Michael Stuart
Calculation of effect estimates
Pressure main effect, traditional design:
(Y5+Y6+Y7+Y8)/4 – (Y1+Y2+Y3+Y4)/4
SE:
2
4
Pressure main effect, Fisher design
(Y7+Y8+Y9+Y10+Y11+Y12)/6 – (Y1+Y2+Y3+Y4+Y5+Y6)/6
SE:
2
6
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
54
© 2015 Michael Stuart
Multi-factor designs
find best operating conditions
75 best
60
High
Pressure
Low
best
65
Low
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Temperature
High
70 best
Lecture 1.1
55
© 2015 Michael Stuart
Multi-factor designs
reveal interaction
Classwork 1.1.2:
Calculate Pressure effect at Low Temperature
and at High Temperature;
calculate the difference
Calculate Temperature effect at Low Pressure
and at High Pressure;
calculate the difference
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
56
© 2015 Michael Stuart
Multi-factor designs
reveal interaction
75
60
Pressure effect
Low T: 60 – 65 = –5
High T: 75 – 70 = +5
Diff:
5 – (–5) = 10
High
Pressure
Temperature effect
Low
65
Low
Temperature
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
High
70
Low P: 70 – 65 = 5
High P: 75 – 60 = 15
Diff:
15 – 5 = 10
Lecture 1.1
57
© 2015 Michael Stuart
Interaction defined
Factors interact when the effect of changing one
factor depends on the level of the other.
Interaction displayed
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
58
© 2015 Michael Stuart
Lecture 1.1
1. Introduction to Course
2. What is an experiment?
3. Case study: Industrial process improvement
− three design principles
4. Case study: Clinical trial
− how randomization works
5. Multifactor Designs
6. Other application areas
7. Experimental vs Observational Studies
8. Strategies for Experimentation
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
59
© 2015 Michael Stuart
Part 6
Other application areas
• Agriculture
• Genetics
• Biological Sciences
• Physical Sciences
• Engineering
• Psychology
• Social Sciences?
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
60
© 2015 Michael Stuart
Lecture 1.1
1. Introduction to Course
2. What is an experiment?
3. Case study: Industrial process improvement
− three design principles
4. Case study: Clinical trial
− how randomization works
5. Multifactor Designs
6. Other application areas
7. Experimental vs Observational Studies
8. Strategies for Experimentation
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
61
© 2015 Michael Stuart
Part 7
Experimental vs Observational Studies
Example:
Process improvement study,
old or new process
Observational study:
new process is run,
old process inventory is sampled,
product from old and new processes compared
Experiment:
process is changed from day to day,
under controlled conditions
• Current control vs historical control
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
62
© 2015 Michael Stuart
Example:
Clinical trial,
drugs or surgery
Observational study:
check patient records,
compare drug and surgery
Experiment:
assign patients at random,
compare drug and placebo
 Retrospective vs Prospective
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
63
© 2015 Michael Stuart
Number of Deaths from
Cancer
Lurking Variables
Number of Churches
• Lurking variable = Population size
• Covariance Analysis ?
or try number of deaths per thousand
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
64
© 2015 Michael Stuart
Lurking Variables
80
Population ('000)
75
70
65
60
55
50
100
150
200
Number of Storks
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
250
300
Ref: BHH Ch 1 p 8
Lecture 1.1
65
© 2015 Michael Stuart
Experiment
vs
Observation
• control of input factors;
• no control of input
factors (happenstance);
• control of environment;
• environment may vary;
• blocking to control known
non-experimental factors;
• matching to control nonexperimental factors;
• randomization to minimse
the effects of unknown
non-experimental factors
• randomization
impossible; "lurking"
variables possible
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
66
© 2015 Michael Stuart
Cause and effect
• Fisher's randomized controlled experiment,
– the "gold standard"
• Rubin's matching via propensity scoring
• Pearl's Structural Causal Model
• etc.
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
67
© 2015 Michael Stuart
Caustic comments
... large segments of the statistical research
community find it hard to appreciate and benefit
from the many results that causal analysis has
produced in the past two decades.
Pearl (2009) Statistics Surveys Vol. 3 96–146
I appreciate the opportunity to expand on the
essential point of Shrier’s and Pearl’s letters,
because I think that it has fostered, and continues
to foster, bad practical advice, which is based on
an unprincipled and confused theoretical
perspective.
Rubin (2009) Statist. Med., 28:1415–1424
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
68
© 2015 Michael Stuart
Fisher on smoking and lung cancer
"The evidence linking cigarette smoking with lung
cancer, standing by itself, is inconclusive, as it is
apparently impossible to carry out properly controlled
experiments with human material.
Observations not fulfilling the requirements of decisive
experimentation might be suggestive, not conclusive,
and may be afforded a confidence which is more than
their due.
Association is not causation."
RA Fisher, quoted in "Cigarette-cancer links disputed",
New York Times, Dec. 29, 1957
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
69
© 2015 Michael Stuart
Regression analysis and
cause and effect
"The justification sometimes advanced that a
multiple regression analysis on observational data
can be relied upon
if there is an adequate theoretical background
is utterly specious and disregards the unlimited
capability of the human intellect for producing
plausible explanations by the carload lot".
K.A. Brownlee, 1965
Big Data
Analytics
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
70
© 2015 Michael Stuart
Lecture 1.1
1. Introduction to Course
2. What is an experiment?
3. Case study: Industrial process improvement
− three design principles
4. Case study: Clinical trial
− how randomization works
5. Multifactor Designs
6. Other application areas
7. Experimental vs Observational Studies
8. Strategies for Experimentation
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
71
© 2015 Michael Stuart
Part 8
Strategies for Experimentation
Box on strategy:
When you see the credits roll at the end of a
successful movie you realize there are many more
things that must be attended to in addition to
choosing a good script.
Similarly in running a successful experiment there
are many more things that must be attended to in
addition to choosing a good experimental design.
(BHH, End notes)
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
72
© 2015 Michael Stuart
Robinson's outline
Ref: GKR p.6, see also p.7
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
73
© 2015 Michael Stuart
Lecture 1.1
1. Introduction to Course
2. What is an experiment?
3. Case study: Industrial process improvement
− three design principles
4. Case study: Clinical trial
− how randomization works
5. Multifactor Designs
6. Other application areas
7. Experimental vs Observational Studies
8. Strategies for Experimentation
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
74
© 2015 Michael Stuart
Minute test
– How much did you get out of today's class?
– How did you find the pace of today's class?
– What single point caused you the most
difficulty?
– What single change by the lecturer would have
most improved this class?
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
75
© 2015 Michael Stuart
Reading
Lecture 1.1
EM
Sections 4.3, 4.5.1, 4.5.3, 4.6, 5.2
Supplementary reading:
DCM Chapter 1, Section 2.5
DV
Chapter 1, Sections 2.2, 2.3
Next lecture:
EM Notes, Chapter 4
Postgraduate Certificate in Statistics
Design and Analysis of Experiments
Lecture 1.1
76
© 2015 Michael Stuart