Design and Analysis of
Multi-Factored Experiments
Engineering 9516
Dr. Leonard M. Lye, P.Eng, FCSCE, FEC
Professor and Associate Dean (Graduate Studies)
Faculty of Engineering and Applied Science, Memorial University of
Newfoundland
St. John’s, NL, A1B 3X5
L. M. Lye
DOE Course
1
DOE - I
Introduction
L. M. Lye
DOE Course
2
Design of Engineering Experiments
Introduction
•
•
•
•
•
Goals of the course and assumptions
An abbreviated history of DOE
The strategy of experimentation
Some basic principles and terminology
Guidelines for planning, conducting and
analyzing experiments
L. M. Lye
DOE Course
3
Assumptions
• You have
–
–
–
–
–
a first course in statistics
heard of the normal distribution
know about the mean and variance
have done some regression analysis or heard of it
know something about ANOVA or heard of it
• Have used Windows or Mac based computers
• Have done or will be conducting experiments
• Have not heard of factorial designs, fractional
factorial designs, RSM, and DACE.
L. M. Lye
DOE Course
4
Some major players in DOE
• Sir Ronald A. Fisher - pioneer
– invented ANOVA and used of statistics in experimental
design while working at Rothamsted Agricultural
Experiment Station, London, England.
• George E. P. Box - married Fisher’s daughter
– still active (86 years old)
– developed response surface methodology (1951)
– plus many other contributions to statistics
• Others
– Raymond Myers, J. S. Hunter, W. G. Hunter, Yates,
Montgomery, Finney, etc..
L. M. Lye
DOE Course
5
Four eras of DOE
• The agricultural origins, 1918 – 1940s
– R. A. Fisher & his co-workers
– Profound impact on agricultural science
– Factorial designs, ANOVA
• The first industrial era, 1951 – late 1970s
– Box & Wilson, response surfaces
– Applications in the chemical & process industries
• The second industrial era, late 1970s – 1990
– Quality improvement initiatives in many companies
– Taguchi and robust parameter design, process robustness
• The modern era, beginning circa 1990
– Wide use of computer technology in DOE
– Expanded use of DOE in Six-Sigma and in business
– Use of DOE in computer experiments
L. M. Lye
DOE Course
6
References
• D. G. Montgomery (2008): Design and Analysis
of Experiments, 7th Edition, John Wiley and Sons
– one of the best book in the market. Uses Design-Expert
software for illustrations. Uses letters for Factors.
• G. E. P. Box, W. G. Hunter, and J. S. Hunter
(2005): Statistics for Experimenters: An
Introduction to Design, Data Analysis, and Model
Building, John Wiley and Sons. 2nd Edition
– Classic text with lots of examples. No computer aided
solutions. Uses numbers for Factors.
• Journal of Quality Technology, Technometrics,
American Statistician, discipline specific journals
L. M. Lye
DOE Course
7
Introduction: What is meant by DOE?
• Experiment – a test or a series of tests in which purposeful changes
are made to the input variables or factors of a system
so that we may observe and identify the reasons for
changes in the output response(s).
• Question: 5 factors, and 2 response variables
– Want to know the effect of each factor on the response
and how the factors may interact with each other
– Want to predict the responses for given levels of the
factors
– Want to find the levels of the factors that optimizes the
responses - e.g. maximize Y1 but minimize Y2
– Time and budget allocated for 30 test runs only.
L. M. Lye
DOE Course
8
Strategy of Experimentation
• Strategy of experimentation
– Best guess approach (trial and error)
• can continue indefinitely
• cannot guarantee best solution has been found
– One-factor-at-a-time (OFAT) approach
• inefficient (requires many test runs)
• fails to consider any possible interaction between factors
– Factorial approach (invented in the 1920’s)
•
•
•
•
L. M. Lye
Factors varied together
Correct, modern, and most efficient approach
Can determine how factors interact
Used extensively in industrial R and D, and for process
improvement.
DOE Course
9
• This course will focus on three very useful and
important classes of factorial designs:
– 2-level full factorial (2k)
– fractional factorial (2k-p), and
– response surface methodology (RSM)
• I will also cover split plot designs, and design and analysis of computer
experiments if time permits.
• Dimensional analysis and how it can be combined with DOE will also be
briefly covered.
• All DOE are based on the same statistical principles
and method of analysis - ANOVA and regression
analysis.
• Answer to question: use a 25-1 fractional factorial in a central
composite design = 27 runs (min)
L. M. Lye
DOE Course
10
Statistical Design of Experiments
• All experiments should be designed experiments
• Unfortunately, some experiments are poorly
designed - valuable resources are used
ineffectively and results inconclusive
• Statistically designed experiments permit
efficiency and economy, and the use of statistical
methods in examining the data result in scientific
objectivity when drawing conclusions.
L. M. Lye
DOE Course
11
• DOE is a methodology for systematically applying
statistics to experimentation.
• DOE lets experimenters develop a mathematical
model that predicts how input variables interact to
create output variables or responses in a process or
system.
• DOE can be used for a wide range of experiments
for various purposes including nearly all fields of
engineering and even in business marketing.
• Use of statistics is very important in DOE and the
basics are covered in a first course in an
engineering program.
L. M. Lye
DOE Course
12
• In general, by using DOE, we can:
–
–
–
–
–
Learn about the process we are investigating
Screen important variables
Build a mathematical model
Obtain prediction equations
Optimize the response (if required)
• Statistical significance is tested using ANOVA,
and the prediction model is obtained using
regression analysis.
L. M. Lye
DOE Course
13
Applications of DOE in Engineering Design
• Experiments are conducted in the field of
engineering to:
– evaluate and compare basic design configurations
– evaluate different materials
– select design parameters so that the design will work
well under a wide variety of field conditions (robust
design)
– determine key design parameters that impact
performance
L. M. Lye
DOE Course
14
INPUTS
(Factors)
X variables
OUTPUTS
(Responses)
Y variables
People
Materials
PROCESS:
Equipment
responses related
to performing a
service
Policies
responses related
to producing a
produce
Procedures
A Ble nding of
Inputs which
Ge ne rates
Corresponding
Outputs
responses related
to completing a task
Methods
Env ironment
L. M. Lye
Illustration of a Proce ss
DOE Course
15
INPUTS
(Factors)
X variables
OUTPUTS
(Responses)
Y variables
Type of
cement
compressive
strength
Percent water
PROCESS:
Type of
Additiv es
Percent
Additiv es
Mixing Time
modulus of elasticity
Discov e ring
Optimal
Concre te
M ixture
modulus of rupture
Poisson's ratio
Curing
Conditions
% Plasticizer
L. M. Lye
Optimum Concre te M ixture
DOE Course
16
INPUTS
(Factors)
X variables
OUTPUTS
(Responses)
Y variables
Type of Raw
Material
Mold
Temperature
Holding
Pressure
PROCESS:
% shrinkage f rom
mold size
Holding Time
Gate Size
thickness of molded
part
M anufacturing
Inje ction
M olde d Parts
number of defective
parts
Screw Speed
Moisture
Content
L. M. Lye
M anufacturing Inje ction M olde d
Parts
DOE Course
17
INPUTS
(Factors)
X variables
OUTPUTS
(Responses)
Y var iables
Impermeable lay er
(mm)
Initial storage
(mm)
PROCESS:
Coef f icient of
Inf iltration
Coef f icient of
Recession
Rainfall-Runoff
M ode l
Calibration
R-square:
Predicted vs
Observed Fits
Soil Moisture
Capacity
(mm)
Initial Soil Moisture
(mm)
L. M. Lye
M ode l Calibration
DOE Course
18
INPUTS
(Factors)
X v ariables
OUTPUTS
(Responses)
Y v ariables
Brand:
Cheap vs Costly
PROCESS:
Taste:
Scale of 1 to 10
T im e:
4 min vs 6 min
Power:
75% or 100%
Making the
Best
Microwave
popcorn
Bullets:
Grams of unpopped
corns
Height:
On bottom or raised
Making microwave popcorn
L. M. Lye
DOE Course
19
Examples of experiments from daily life
• Photography
– Factors: speed of film, lighting, shutter speed
– Response: quality of slides made close up with flash attachment
• Boiling water
– Factors: Pan type, burner size, cover
– Response: Time to boil water
• D-day
– Factors: Type of drink, number of drinks, rate of drinking, time
after last meal
– Response: Time to get a steel ball through a maze
• Mailing
– Factors: stamp, area code, time of day when letter mailed
– Response: Number of days required for letter to be delivered
L. M. Lye
DOE Course
20
More examples
• Cooking
– Factors: amount of cooking wine, oyster sauce, sesame oil
– Response: Taste of stewed chicken
• Sexual Pleasure
– Factors: marijuana, screech, sauna
– Response: Pleasure experienced in subsequent you know what
• Basketball
– Factors: Distance from basket, type of shot, location on floor
– Response: Number of shots made (out of 10) with basketball
• Skiing
– Factors: Ski type, temperature, type of wax
– Response: Time to go down ski slope
L. M. Lye
DOE Course
21
Basic Principles
• Statistical design of experiments (DOE)
– the process of planning experiments so that
appropriate data can be analyzed by statistical
methods that results in valid, objective, and
meaningful conclusions from the data
– involves two aspects: design and statistical
analysis
L. M. Lye
DOE Course
22
• Every experiment involves a sequence of
activities:
– Conjecture - hypothesis that motivates the
experiment
– Experiment - the test performed to investigate
the conjecture
– Analysis - the statistical analysis of the data
from the experiment
– Conclusion - what has been learned about the
original conjecture from the experiment.
L. M. Lye
DOE Course
23
Three basic principles of Statistical DOE
• Replication
– allows an estimate of experimental error
– allows for a more precise estimate of the sample mean
value
• Randomization
– cornerstone of all statistical methods
– “average out” effects of extraneous factors
– reduce bias and systematic errors
• Blocking
– increases precision of experiment
– “factor out” variable not studied
L. M. Lye
DOE Course
24
Guidelines for Designing Experiments
• Recognition of and statement of the problem
– need to develop all ideas about the objectives of the
experiment - get input from everybody - use team
approach.
• Choice of factors, levels, ranges, and response
variables.
– Need to use engineering judgment or prior test results.
• Choice of experimental design
– sample size, replicates, run order, randomization,
software to use, design of data collection forms.
L. M. Lye
DOE Course
25
• Performing the experiment
– vital to monitor the process carefully. Easy to
underestimate logistical and planning aspects in a
complex R and D environment.
• Statistical analysis of data
– provides objective conclusions - use simple graphics
whenever possible.
• Conclusion and recommendations
– follow-up test runs and confirmation testing to validate
the conclusions from the experiment.
• Do we need to add or drop factors, change ranges,
levels, new responses, etc.. ???
L. M. Lye
DOE Course
26
Using Statistical Techniques in
Experimentation - things to keep in mind
• Use non-statistical knowledge of the problem
– physical laws, background knowledge
• Keep the design and analysis as simple as possible
– Don’t use complex, sophisticated statistical techniques
– If design is good, analysis is relatively straightforward
– If design is bad - even the most complex and elegant
statistics cannot save the situation
• Recognize the difference between practical and
statistical significance
– statistical significance  practically significance
L. M. Lye
DOE Course
27
• Experiments are usually iterative
– unwise to design a comprehensive experiment at the
start of the study
– may need modification of factor levels, factors,
responses, etc.. - too early to know whether experiment
would work
– use a sequential or iterative approach
– should not invest more than 25% of resources in the
initial design.
– Use initial design as learning experiences to accomplish
the final objectives of the experiment.
L. M. Lye
DOE Course
28
DOE (II)
Factorial vs OFAT
L. M. Lye
DOE Course
29
Factorial v.s. OFAT
• Factorial design - experimental trials or runs are
performed at all possible combinations of factor
levels in contrast to OFAT experiments.
• Factorial and fractional factorial experiments are
among the most useful multi-factor experiments
for engineering and scientific investigations.
L. M. Lye
DOE Course
30
• The ability to gain competitive advantage requires
extreme care in the design and conduct of
experiments. Special attention must be paid to joint
effects and estimates of variability that are provided
by factorial experiments.
• Full and fractional experiments can be conducted
using a variety of statistical designs. The design
selected can be chosen according to specific
requirements and restrictions of the investigation.
L. M. Lye
DOE Course
31
Factorial Designs
• In a factorial experiment, all
possible combinations of
factor levels are tested
• The golf experiment:
–
–
–
–
–
–
–
–
Type of driver (over or regular)
Type of ball (balata or 3-piece)
Walking vs. riding a cart
Type of beverage (Beer vs water)
Time of round (am or pm)
Weather
Type of golf spike
Etc, etc, etc…
L. M. Lye
DOE Course
32
Factorial Design
L. M. Lye
DOE Course
33
Factorial Designs with Several Factors
L. M. Lye
DOE Course
34
Erroneous Impressions About Factorial
Experiments
• Wasteful and do not compensate the extra effort with
additional useful information - this folklore presumes that
one knows (not assumes) that factors independently
influence the responses (i.e. there are no factor
interactions) and that each factor has a linear effect on the
response - almost any reasonable type of experimentation
will identify optimum levels of the factors
• Information on the factor effects becomes available only
after the entire experiment is completed. Takes too long.
Actually, factorial experiments can be blocked and
conducted sequentially so that data from each block can be
analyzed as they are obtained.
L. M. Lye
DOE Course
35
One-factor-at-a-time experiments (OFAT)
• OFAT is a prevalent, but potentially disastrous type of
experimentation commonly used by many engineers and
scientists in both industry and academia.
• Tests are conducted by systematically changing the levels
of one factor while holding the levels of all other factors
fixed. The “optimal” level of the first factor is then
selected.
• Subsequently, each factor in turn is varied and its
“optimal” level selected while the other factors are held
fixed.
L. M. Lye
DOE Course
36
One-factor-at-a-time experiments (OFAT)
• OFAT experiments are regarded as easier to implement,
more easily understood, and more economical than
factorial experiments. Better than trial and error.
• OFAT experiments are believed to provide the optimum
combinations of the factor levels.
• Unfortunately, each of these presumptions can generally be
shown to be false except under very special circumstances.
• The key reasons why OFAT should not be conducted
except under very special circumstances are:
– Do not provide adequate information on interactions
– Do not provide efficient estimates of the effects
L. M. Lye
DOE Course
37
Factorial vs OFAT ( 2-levels only)
Factorial
OFAT
• 2 factors: 4 runs
• 2 factors: 6 runs
– 3 effects
– 2 effects
• 3 factors: 8 runs
• 3 factors: 16 runs
– 7 effects
– 3 effects
• 5 factors: 32 or 16 runs
• 5 factors: 96 runs
– 31 or 15 effects
– 5 effects
• 7 factors: 128 or 64 runs
• 7 factors: 512 runs
– 127 or 63 effects
L. M. Lye
– 7 effects
DOE Course
38
Example: Factorial vs OFAT
Factorial
OFAT
high
high
Factor B
B
low
low
low
high
low
A
Factor A
E.g. Factor A: Reynold’s number,
L. M. Lye
high
DOE Course
Factor B: k/D
39
Example: Effect of Re and k/D on friction factor f
•
•
•
•
•
Consider a 2-level factorial design (22)
Reynold’s number = Factor A; k/D = Factor B
Levels for A: 104 (low) 106 (high)
Levels for B: 0.0001 (low)
0.001 (high)
Responses: (1) = 0.0311, a = 0.0135, b = 0.0327,
ab = 0.0200
• Effect (A) = -0.66, Effect (B) = 0.22, Effect (AB) = 0.17
• % contribution: A = 84.85%, B = 9.48%, AB = 5.67%
• The presence of interactions implies that one cannot
satisfactorily describe the effects of each factor using main
effects.
L. M. Lye
DOE Course
40
DESIGN-EASE Pl ot
Ln(f)
Interaction Graph
-3.42038
k/D
2
2
X = A: Reynol d's #
Y = B: k/D
-3.64155
Desi gn Poi nts
Ln(f)
B- 0.000
B+ 0.001
-3.86272
2
-4.08389
2
-4.30507
4.000
4.500
5.000
5.500
6.000
Reynold's #
L. M. Lye
DOE Course
41
DESIGN-EASE Pl ot
Ln(f)
0.0010
Ln(f)
X = A: Reynol d's #
Y = B: k/D
Desi gn Poi nts
0.0008
k/D
-3.56783
-3.86272
-3.71528
0.0006
-4.01017
0.0003
-4.15762
0.0001
4.000
4.500
5.000
5.500
6.000
Reynold's #
L. M. Lye
DOE Course
42
DESIGN-EASE Pl ot
Ln(f)
X = A: Reynol d's #
Y = B: k/D
-3.42038
-3.64155
-3.86272
Ln(f)
-4.08389
-4.30507
0.0010
0.0008
0.0006
6.000
k/D
5.500
0.0003
5.000
4.500
0.0001
4.000
Reynol d's #
L. M. Lye
DOE Course
43
With the addition of a few more points
• Augmenting the basic 22 design with a center
point and 5 axial points we get a central composite
design (CCD) and a 2nd order model can be fit.
• The nonlinear nature of the relationship between
Re, k/D and the friction factor f can be seen.
• If Nikuradse (1933) had used a factorial design in
his pipe friction experiments, he would need far
less experimental runs!!
• If the number of factors can be reduced by
dimensional analysis, the problem can be made
simpler for experimentation.
L. M. Lye
DOE Course
44
DESIGN-EXPERT Pl ot
Log10(f)
Interaction Graph
B: k/D
-1.495
X = A: RE
Y = B: k/D
Desi gn Poi nts
-1.567
Log10(f)
B- 0.000
B+ 0.001
-1.639
-1.712
-1.784
4.293
4.646
5.000
5.354
5.707
A: RE
L. M. Lye
DOE Course
45
DESIGN-EXPERT Pl ot
Log10(f)
X = A: RE
Y = B: k/D
-1.554
-1.611
-1.668
Log10(f)
-1.725
-1.783
0.0008828
0.0007414
B:0.0006000
k/D
0.0004586
0.0003172
4.293
4.646
5.000
5.354
5.707
A: RE
L. M. Lye
DOE Course
46
DESIGN-EXPERT Pl ot
Log10(f)
0.0008828
Log10(f)
Desi gn Poi nts
X = A: RE
Y = B: k/D
0.0007414
B: k/D
-1.668
0.0006000
-1.706
-1.592-1.630
-1.744
0.0004586
0.0003172
4.293
4.646
5.000
5.354
5.707
A: RE
L. M. Lye
DOE Course
47
DESIGN-EXPERT Pl ot
Log10(f)
Predicted vs. Actual
-1.494
Predicted
-1.566
-1.639
-1.711
-1.783
-1.783
-1.711
-1.639
-1.566
-1.494
Actual
L. M. Lye
DOE Course
48
DOE (III)
Basic Concepts
L. M. Lye
DOE Course
49
Design of Engineering Experiments
Basic Statistical Concepts
• Simple comparative experiments
– The hypothesis testing framework
– The two-sample t-test
– Checking assumptions, validity
• Comparing more than two factor levels…the
analysis of variance
–
–
–
–
L. M. Lye
ANOVA decomposition of total variability
Statistical testing & analysis
Checking assumptions, model validity
Post-ANOVA testing of means
DOE Course
50
Portland Cement Formulation
Observation
(sample), j
Modified Mortar
(Formulation 1) y1 j
Unmodified Mortar
(Formulation 2)
y2 j
1
16.85
17.50
2
16.40
17.63
3
17.21
18.25
4
16.35
18.00
5
16.52
17.86
6
17.04
17.75
7
16.96
18.22
8
17.15
17.90
9
16.59
17.96
10
16.57
18.15
L. M. Lye
DOE Course
51
Graphical View of the Data
Dot Diagram
Dotplots of Form 1 and Form 2
(means are indicated by lines)
18.3
17.3
16.3
Form 1
L. M. Lye
Form 2
DOE Course
52
Box Plots
Boxplots of Form 1 and Form 2
(means are indicated by solid circles)
18.5
17.5
16.5
Form 1
L. M. Lye
Form 2
DOE Course
53
The Hypothesis Testing Framework
• Statistical hypothesis testing is a useful
framework for many experimental
situations
• Origins of the methodology date from the
early 1900s
• We will use a procedure known as the twosample t-test
L. M. Lye
DOE Course
54
The Hypothesis Testing Framework
• Sampling from a normal distribution
• Statistical hypotheses:
H 0 : 1   2
H1 : 1   2
L. M. Lye
DOE Course
55
Estimation of Parameters
1 n
y   yi estimates the population mean 
n i 1
n
1
2
2
2
S 
( yi  y ) estimates the variance 

n  1 i 1
L. M. Lye
DOE Course
56
Summary Statistics
Formulation 1
Formulation 2
“New recipe”
“Original recipe”
y1  16.76
y2  17.92
S  0.100
S  0.061
S1  0.316
S2  0.247
2
2
2
1
n2  10
n1  10
L. M. Lye
DOE Course
57
How the Two-Sample t-Test Works:
Use the sample means to draw inferences about the population means
y1  y2  16.76  17.92  1.16
Difference in sample means
Standard deviation of the difference in sample means
 
2
y
2
n
This suggests a statistic:
Z0 
y1  y2
 12
n1
L. M. Lye

 22
n2
DOE Course
58
How the Two-Sample t-Test Works:
Use S and S to estimate  and 
2
1
2
2
2
1
The previous ratio becomes
2
2
y1  y2
2
1
2
2
S
S

n1 n2
However, we have the case where     
2
1
2
2
2
Pool the individual sample variances:
(n1  1) S  (n2  1) S
S 
n1  n2  2
2
p
L. M. Lye
2
1
2
2
DOE Course
59
How the Two-Sample t-Test Works:
The test statistic is
y1  y2
t0 
1 1
Sp

n1 n2
• Values of t0 that are near zero are consistent with the null
hypothesis
• Values of t0 that are very different from zero are consistent
with the alternative hypothesis
• t0 is a “distance” measure-how far apart the averages are
expressed in standard deviation units
• Notice the interpretation of t0 as a signal-to-noise ratio
L. M. Lye
DOE Course
60
The Two-Sample (Pooled) t-Test
(n1  1) S12  (n2  1) S 22 9(0.100)  9(0.061)
S 

 0.081
n1  n2  2
10  10  2
2
p
S p  0.284
t0 
y1  y2
16.76  17.92

 9.13
1 1
1 1
Sp

0.284

n1 n2
10 10
The two sample means are about 9 standard deviations apart
Is this a "large" difference?
L. M. Lye
DOE Course
61
The Two-Sample (Pooled) t-Test
• So far, we haven’t really done any “statistics”
• We need an objective basis for deciding how large the test
statistic t0 really is
• In 1908, W. S. Gosset derived the reference distribution
for t0 … called the t distribution
• Tables of the t distribution - any stats text.
• The t-distribution looks almost exactly like the normal
distribution except that it is shorter and fatter when the
degrees of freedom is less than about 100.
• Beyond 100, the t is practically the same as the normal.
L. M. Lye
DOE Course
62
The Two-Sample (Pooled) t-Test
• A value of t0 between
–2.101 and 2.101 is
consistent with
equality of means
• It is possible for the
means to be equal and
t0 to exceed either
2.101 or –2.101, but it
would be a “rare
event” … leads to the
conclusion that the
means are different
• Could also use the
P-value approach
L. M. Lye
DOE Course
63
The Two-Sample (Pooled) t-Test
• The P-value is the risk of wrongly rejecting the null
hypothesis of equal means (it measures rareness of the event)
• The P-value in our problem is P = 0.000000038
L. M. Lye
DOE Course
64
Minitab Two-Sample t-Test Results
Two-Sample T-Test and CI: Form 1, Form 2
Two-sample T for Form 1 vs Form 2
N
Mean
StDev
SE Mean
Form 1
10
16.764
0.316
0.10
Form 2
10
17.922
0.248
0.078
Difference = mu Form 1 - mu Form 2
Estimate for difference:
-1.158
95% CI for difference: (-1.425, -0.891)
T-Test of difference = 0 (vs not =): T-Value = -9.11
P-Value = 0.000 DF = 18
Both use Pooled StDev = 0.284
L. M. Lye
DOE Course
65
Checking Assumptions –
The Normal Probability Plot
Tension Bond Strength Data
ML Estimates
Form 1
99
Form 2
Goodness of Fit
95
AD*
90
1.209
1.387
Percent
80
70
60
50
40
30
20
10
5
1
16.5
17.5
18.5
Data
L. M. Lye
DOE Course
66
Importance of the t-Test
• Provides an objective framework for simple
comparative experiments
• Could be used to test all relevant hypotheses
in a two-level factorial design, because all
of these hypotheses involve the mean
response at one “side” of the cube versus
the mean response at the opposite “side” of
the cube
L. M. Lye
DOE Course
67
What If There Are More Than
Two Factor Levels?
• The t-test does not directly apply
• There are lots of practical situations where there are either
more than two levels of interest, or there are several factors of
simultaneous interest
• The analysis of variance (ANOVA) is the appropriate
analysis “engine” for these types of experiments
• The ANOVA was developed by Fisher in the early 1920s, and
initially applied to agricultural experiments
• Used extensively today for industrial experiments
L. M. Lye
DOE Course
68
An Example
• Consider an investigation into the formulation of a
new “synthetic” fiber that will be used to make ropes
• The response variable is tensile strength
• The experimenter wants to determine the “best” level
of cotton (in wt %) to combine with the synthetics
• Cotton content can vary between 10 – 40 wt %; some
non-linearity in the response is anticipated
• The experimenter chooses 5 levels of cotton
“content”; 15, 20, 25, 30, and 35 wt %
• The experiment is replicated 5 times – runs made in
random order
L. M. Lye
DOE Course
69
An Example
• Does changing the
cotton weight percent
change the mean
tensile strength?
• Is there an optimum
level for cotton
content?
L. M. Lye
DOE Course
70
The Analysis of Variance
• In general, there will be a levels of the factor, or a treatments, and n
replicates of the experiment, run in random order…a completely
randomized design (CRD)
• N = an total runs
• We consider the fixed effects case only
• Objective is to test hypotheses about the equality of the a treatment
means
L. M. Lye
DOE Course
71
The Analysis of Variance
• The name “analysis of variance” stems from a
partitioning of the total variability in the response
variable into components that are consistent with a
model for the experiment
• The basic single-factor ANOVA model is
 i  1, 2,..., a
yij     i   ij , 
 j  1, 2,..., n
  an overall mean,  i  ith treatment effect,
 ij  experimental error, NID(0,  2 )
L. M. Lye
DOE Course
72
Models for the Data
There are several ways to write a model for
the data:
yij     i   ij is called the effects model
Let i     i , then
yij  i   ij is called the means model
Regression models can also be employed
L. M. Lye
DOE Course
73
The Analysis of Variance
• Total variability is measured by the total sum of
squares:
a
n
SST   ( yij  y.. )2
i 1 j 1
• The basic ANOVA partitioning is:
a
n
a
n
2
(
y

y
)

[(
y

y
)

(
y

y
)]
 ij ..  i. .. ij i.
2
i 1 j 1
i 1 j 1
a
a
n
 n ( yi.  y.. ) 2   ( yij  yi. ) 2
i 1
i 1 j 1
SST  SSTreatments  SS E
L. M. Lye
DOE Course
74
The Analysis of Variance
SST  SSTreatments  SSE
• A large value of SSTreatments reflects large differences in
treatment means
• A small value of SSTreatments likely indicates no differences in
treatment means
• Formal statistical hypotheses are:
H 0 : 1  2 
 a
H1 : At least one mean is different
L. M. Lye
DOE Course
75
The Analysis of Variance
• While sums of squares cannot be directly compared to test
the hypothesis of equal means, mean squares can be
compared.
• A mean square is a sum of squares divided by its degrees
of freedom:
dfTotal  dfTreatments  df Error
an  1  a  1  a (n  1)
SSTreatments
SS E
MSTreatments 
, MS E 
a 1
a (n  1)
• If the treatment means are equal, the treatment and error
mean squares will be (theoretically) equal.
• If treatment means differ, the treatment mean square will
be larger than the error mean square.
L. M. Lye
DOE Course
76
The Analysis of Variance is
Summarized in a Table
• The reference distribution for F0 is the Fa-1, a(n-1) distribution
• Reject the null hypothesis (equal treatment means) if
F0  F ,a 1,a ( n 1)
L. M. Lye
DOE Course
77
ANOVA Computer Output
(Design-Expert)
Response:Strength
ANOVA for Selected Factorial Model
Analysis of variance table [Partial sum of squares]
Sum of
Mean
F
Source Squares
DF
Square
Value Prob > F
Model 475.76
4
118.94
14.76 < 0.0001
A
475.76
4
118.94
14.76 < 0.0001
Pure Error161.20
20
8.06
Cor Total 636.96
24
Std. Dev. 2.84
Mean
15.04
C.V.
18.88
PRESS 251.88
L. M. Lye
R-Squared
Adj R-Squared
Pred R-Squared
Adeq Precision
DOE Course
0.7469
0.6963
0.6046
9.294
78
The Reference Distribution:
L. M. Lye
DOE Course
79
Graphical View of the Results
DESIGN-EXPERT Pl ot
Strength
One Factor Plot
25
X = A: Cotton Wei ght %
Desi gn Poi nts
20.5
2
Strength
2
2
2
16
2
11.5
7
2
2
15
20
25
30
35
A: Cotton Weight %
L. M. Lye
DOE Course
80
Model Adequacy Checking in the ANOVA
•
•
•
•
•
•
Checking assumptions is important
Normality
Constant variance
Independence
Have we fit the right model?
Later we will talk about what to do if some
of these assumptions are violated
L. M. Lye
DOE Course
81
Model Adequacy Checking in the ANOVA
DESIGN-EXPERT Pl ot
• Examination of residuals
Strength
99
eij  yij  yˆij
• Design-Expert generates
the residuals
• Residual plots are very
useful
• Normal probability plot
of residuals
95
90
Normal % probability
 yij  yi.
Normal plot of residuals
80
70
50
30
20
10
5
1
-3.8
-1.55
0.7
2.95
5.2
Res idual
L. M. Lye
DOE Course
82
Other Important Residual Plots
DESIGN-EXPERT Plot
Residuals vs. Predicted
Strength
PERT Plot
Residuals vs. Run
5.2
5.2
2.95
2.95
2
Res iduals
Res iduals
2
0.7
2
2
0.7
-1.55
-1.55
2
2
2
-3.8
-3.8
9.80
12.75
15.70
18.65
1
21.60
7
10
13
16
19
22
25
Run Num ber
Predicted
L. M. Lye
4
DOE Course
83
Post-ANOVA Comparison of Means
• The analysis of variance tests the hypothesis of equal
treatment means
• Assume that residual analysis is satisfactory
• If that hypothesis is rejected, we don’t know which specific
means are different
• Determining which specific means differ following an
ANOVA is called the multiple comparisons problem
• There are lots of ways to do this
• We will use pairwise t-tests on means…sometimes called
Fisher’s Least Significant Difference (or Fisher’s LSD)
Method
L. M. Lye
DOE Course
84
Design-Expert Output
Treatment Means (Adjusted, If Necessary)
Estimated
Standard
Mean
Error
1-15
9.80
1.27
2-20
15.40
1.27
3-25
17.60
1.27
4-30
21.60
1.27
5-35
10.80
1.27
Mean
Treatment Difference
1 vs 2
-5.60
1 vs 3
-7.80
1 vs 4
-11.80
1 vs 5
-1.00
2 vs 3
-2.20
2 vs 4
-6.20
2 vs 5
4.60
3 vs 4
-4.00
3 vs 5
6.80
4 vs 5
10.80
L. M. Lye
DF
1
1
1
1
1
1
1
1
1
1
Standard
Error
1.80
1.80
1.80
1.80
1.80
1.80
1.80
1.80
1.80
1.80
DOE Course
t for H0
Coeff=0
-3.12
-4.34
-6.57
-0.56
-1.23
-3.45
2.56
-2.23
3.79
6.01
Prob > |t|
0.0054
0.0003
< 0.0001
0.5838
0.2347
0.0025
0.0186
0.0375
0.0012
< 0.0001
85
For the Case of Quantitative Factors, a
Regression Model is often Useful
Response:Strength
ANOVA for Response Surface Cubic Model
Analysis of variance table [Partial sum of squares]
Sum of
Mean
F
Source Squares
DF Square Value Prob > F
Model
441.81
3 147.27
15.85 < 0.0001
A
90.84
1
90.84
9.78
0.0051
2
A
343.21
1 343.21
36.93 < 0.0001
A3
64.98
1
64.98
6.99
0.0152
Residual 195.15
21
9.29
Lack of Fit 33.95
1
33.95
4.21 0.0535
Pure Error 161.20
20
8.06
Cor Total 636.96
24
Coefficient
Factor Estimate
Intercept 19.47
A-Cotton % 8.10
A2
-8.86
A3
-7.60
L. M. Lye
Standard 95% CI 95% CI
DF Error
Low
High
1
0.95
17.49 21.44
1
2.59
2.71 13.49
1
1.46 -11.89
-5.83
1
2.87 -13.58
-1.62
DOE Course
VIF
9.03
1.00
9.03
86
The Regression One
Model
Factor Plot
DESIGN-EXPERT Plot
Strength
25
Final Equation in Terms of
Actual Factors:
X = A: Cotton Weight %
Design Points
This is an empirical model of
the experimental results
2
2
Strength
Strength = 62.611 9.011* Wt % +
0.481* Wt %^2 7.600E-003 * Wt %^3
20.5
2
16
2
11.5
7
2
2
15.00
L. M. Lye
2
DOE Course
20.00
25.00
30.00
A: Cotton Weight %
35.00
87
DESIGN-EXPERT Pl ot
Desi rabi l i ty
One Factor Plot
1.000
Predict 0.7725
X
28.23
X = A: A
Desi gn Poi nts
0.7500
5
Desirability
5
0.5000
5
0.2500
6
6
0.0000
15.00
20.00
25.00
30.00
35.00
A: A
L. M. Lye
DOE Course
88
L. M. Lye
DOE Course
89
Sample Size Determination
• FAQ in designed experiments
• Answer depends on lots of things; including what
type of experiment is being contemplated, how it
will be conducted, resources, and desired
sensitivity
• Sensitivity refers to the difference in means that
the experimenter wishes to detect
• Generally, increasing the number of replications
increases the sensitivity or it makes it easier to
detect small differences in means
L. M. Lye
DOE Course
90
DOE (IV)
General Factorials
L. M. Lye
DOE Course
91
Design of Engineering Experiments
Introduction to General Factorials
•
•
•
•
•
General principles of factorial experiments
The two-factor factorial with fixed effects
The ANOVA for factorials
Extensions to more than two factors
Quantitative and qualitative factors –
response curves and surfaces
L. M. Lye
DOE Course
92
Some Basic Definitions
Definition of a factor effect: The change in the mean response when
the factor is changed from low to high
40  52 20  30

 21
2
2
30  52 20  40
B  yB  yB 

 11
2
2
52  20 30  40
AB 
DOE Course  1
2
2
A  y A  y A 
L. M. Lye
93
The Case of Interaction:
50  12 20  40
A  y A  y A 

1
2
2
40  12 20  50
B  yB   yB  

 9
2
2
12  20 40  50
AB 

 29
2
2
L. M. Lye
DOE Course
94
Regression Model & The
Associated Response
Surface
y   0  1 x1   2 x2
 12 x1 x2  
The least squares fit is
yˆ  35.5  10.5 x1  5.5 x2
0.5 x1 x2
 35.5  10.5 x1  5.5 x2
L. M. Lye
DOE Course
95
The Effect of Interaction
on the Response Surface
Suppose that we add an
interaction term to the
model:
yˆ  35.5  10.5 x1  5.5 x2
8 x1 x2
Interaction is actually
a form of curvature
L. M. Lye
DOE Course
96
Example: Battery Life Experiment
A = Material type; B = Temperature (A quantitative variable)
1.
What effects do material type & temperature have on life?
2.
Is there a choice of material that would give long life regardless of
temperature (a robust product)?
L. M. Lye
DOE Course
97
The General Two-Factor
Factorial Experiment
a levels of factor A; b levels of factor B; n replicates
This is a completely randomized design
L. M. Lye
DOE Course
98
Statistical (effects) model:
 i  1, 2,..., a

yijk     i   j  ( )ij   ijk  j  1, 2,..., b
k  1, 2,..., n

Other models (means model, regression models) can be useful
Regression model allows for prediction of responses when we
have quantitative factors. ANOVA model does not allow for
prediction of responses - treats all factors as qualitative.
L. M. Lye
DOE Course
99
Extension of the ANOVA to Factorials
(Fixed Effects Case)
a
b
n
a
b
i 1
j 1
2
2
2
(
y

y
)

bn
(
y

y
)

an
(
y

y
)
 ijk ...
 i.. ...
 . j. ...
i 1 j 1 k 1
a
b
a
b
n
 n ( yij .  yi..  y. j .  y... )  ( yijk  yij . ) 2
2
i 1 j 1
i 1 j 1 k 1
SST  SS A  SS B  SS AB  SS E
df breakdown:
abn  1  a  1  b  1  (a  1)(b  1)  ab(n  1)
L. M. Lye
DOE Course
100
ANOVA Table – Fixed Effects Case
Design-Expert will perform the computations
Most text gives details of manual computing
(ugh!)
L. M. Lye
DOE Course
101
Design-Expert Output
Response:
Life
ANOVA for Selected Factorial Model
Analysis of variance table [Partial sum of squares]
Source
Model
A
B
AB
Pure E
C Total
L. M. Lye
Sum of
Squares
59416.22
10683.72
39118.72
9613.78
18230.75
77646.97
DF
8
2
2
4
27
35
Mean
F
Square Value
7427.03 11.00
5341.86
7.91
19559.36 28.97
2403.44
3.56
675.21
Std. Dev. 25.98
Mean
105.53
C.V.
24.62
R-Squared
Adj R-Squared
Pred R-Squared
0.7652
0.6956
0.5826
PRESS
Adeq Precision
8.178
32410.22
DOE Course
Prob > F
< 0.0001
0.0020
< 0.0001
0.0186
102
Residual Analysis
DESIGN-EXPERT Plot
Life
Normal plot of residuals
DESIGN-EXPERT Plot
Life
Residuals vs. Predicted
45.25
99
95
18.75
80
70
Res iduals
Norm al % probability
90
50
30
20
10
-7.75
-34.25
5
1
-60.75
49.50
-60.75
-34.25
-7.75
18.75
76.06
102.62
129.19
155.75
45.25
Predicted
Res idual
L. M. Lye
DOE Course
103
Residual Analysis
DESIGN-EXPERT Plot
Life
Residuals vs. Run
45.25
Res iduals
18.75
-7.75
-34.25
-60.75
1
6
11
16
21
26
31
36
Run Num ber
L. M. Lye
DOE Course
104
Residual Analysis
DESIGN-EXPERT Plot
Life
Residuals vs. Material
45.25
18.75
18.75
-7.75
-7.75
-34.25
-34.25
-60.75
-60.75
1
2
3
1
2
3
Tem perature
Material
L. M. Lye
Residuals vs. Temperature
45.25
Res iduals
Res iduals
DESIGN-EXPERT Plot
Life
DOE Course
105
Interaction Plot
DESIGN-EXPERT Plot
Life
Interaction Graph
A: Material
188
X = B: Temperature
Y = A: Material
A1 A1
A2 A2
A3 A3
Life
146
104
2
2
62
2
20
15
70
125
B: Tem perature
L. M. Lye
DOE Course
106
Quantitative and Qualitative Factors
• The basic ANOVA procedure treats every factor as if it
were qualitative
• Sometimes an experiment will involve both quantitative
and qualitative factors, such as in the example
• This can be accounted for in the analysis to produce
regression models for the quantitative factors at each level
(or combination of levels) of the qualitative factors
• These response curves and/or response surfaces are often
a considerable aid in practical interpretation of the results
L. M. Lye
DOE Course
107
Quantitative and Qualitative Factors
Response:Life
*** WARNING: The Cubic Model is Aliased! ***
Sequential Model Sum of Squares
Sum of
Mean
Source
Squares DF
Square
Mean
4.009E+005
1
Linear
49726.39
3
2FI
2315.08
2
Quadratic 76.06
1
Cubic
7298.69
2
Residual 18230.75 27
Total
4.785E+005 36
F
Value
Prob > F
4009E+005
16575.46
1157.54
76.06
3649.35
675.21
13292.97
19.00
1.36
0.086
5.40
< 0.0001
0.2730
0.7709
0.0106
Suggested
Aliased
"Sequential Model Sum of Squares": Select the highest order polynomial where the
additional terms are significant.
L. M. Lye
DOE Course
108
Quantitative and Qualitative Factors
A = Material type
B = Linear effect of Temperature
B2 = Quadratic effect of
Temperature
AB = Material type – TempLinear
AB2 = Material type - TempQuad
B3 = Cubic effect of
Temperature (Aliased)
L. M. Lye
DOE Course
Candidate model
terms from DesignExpert:
Intercept
A
B
B2
AB
B3
AB2
109
Quantitative and Qualitative Factors
Lack of Fit Tests
Source
Linear
Sum of
Squares
9689.83
DF
Mean
Square
F
Value
Prob > F
5
1937.97 2.87
0.0333
2FI
7374.75 3
Quadratic 7298.69 2
Cubic
0.00 0
Pure Error 18230.75 27
2458.25 3.64
3649.35 5.40
0.0252
0.0106
Suggested
Aliased
675.21
"Lack of Fit Tests": Want the selected model to have insignificant lack-of-fit.
L. M. Lye
DOE Course
110
Quantitative and Qualitative Factors
Model Summary Statistics
Source
Std.
Dev.
Adjusted
Predicted
R-Squared R-Squared R-Squared PRESS
Linear
29.54
0.6404
0.6067
0.5432
35470.60 Suggested
2FI
29.22
Quadratic 29.67
Cubic
25.98
0.6702
0.6712
0.7652
0.6153
0.6032
0.6956
0.5187
0.4900
0.5826
37371.08
39600.97
32410.22 Aliased
"Model Summary Statistics": Focus on the model maximizing the "Adjusted R-Squared"
and the "Predicted R-Squared".
L. M. Lye
DOE Course
111
Quantitative and Qualitative Factors
Response:
Life
ANOVA for Response Surface Reduced Cubic Model
Analysis of variance table [Partial sum of squares]
Sum of
Source Squares DF
Model
59416.22 8
A
10683.72 2
B
39042.67 1
B2
76.06
1
AB
2315.08
2
2
AB
7298.69
2
Pure E 18230.75 27
C Total 77646.97 35
L. M. Lye
Mean
F
Square Value
7427.03 11.00
5341.86 7.91
39042.67 57.82
76.06
0.11
1157.54
1.71
3649.35
5.40
675.21
Prob > F
< 0.0001
0.0020
< 0.0001
0.7398
0.1991
0.0106
Std. Dev. 25.98
Mean
105.53
C.V.
24.62
R-Squared
Adj R-Squared
Pred R-Squared
0.7652
0.6956
0.5826
PRESS
Adeq Precision
8.178
32410.22
DOE Course
112
Regression Model Summary of Results
Final Equation in Terms of Actual Factors:
Material A1
Life
=
+169.38017
-2.50145
* Temperature
+0.012851 * Temperature2
Material A2
Life
=
+159.62397
-0.17335
* Temperature
-5.66116E-003 * Temperature2
Material A3
Life
=
+132.76240
+0.90289 * Temperature
-0.010248 * Temperature2
L. M. Lye
DOE Course
113
Regression Model Summary of Results
DESIGN-EXPERT Plot
Life
Interaction Graph
A: Material
188
X = B: Temperature
Y = A: Material
A1 A1
A2 A2
A3 A3
Life
146
104
2
2
62
2
20
15.00
L. M. Lye
42.50
70.00
97.50
B: Tem perature
DOE Course
125.00
114
Factorials with More Than
Two Factors
• Basic procedure is similar to the two-factor case;
all abc…kn treatment combinations are run in
random order
• ANOVA identity is also similar:
SST  SS A  SS B 
 SS ABC 
L. M. Lye
 SS AB  SS AC 
 SS AB
K
DOE Course
 SS E
115
More than 2 factors
• With more than 2 factors, the most useful
type of experiment is the 2-level factorial
experiment.
• Most efficient design (least runs)
• Can add additional levels only if required
• Can be done sequentially
• That will be the next topic of discussion
L. M. Lye
DOE Course
116