Experimental Design
μ-3σ μ-2σ μ-σ
μ μ+σ μ+2σ μ+3σ
Lecture 12
0909.400.01 / 0909.400.02
Dr. P.’s
Clinic Consultant Module in
Probability & Statistics
in Engineering
Today in P&S
μ-3σ μ-2σ μ-σ
μ μ+σ μ+2σ μ+3σ
 Experimental Design
ª Things to remember when designing statistical experiments and tests
© 2006All Rights Reserved, Robi Polikar, Rowan University, Dept. of Electrical and Computer Engineering
Normality?
μ-3σ μ-2σ μ-σ
μ μ+σ μ+2σ μ+3σ
 We often make the assumption that the data is normal. Can we check whether this is
true?
ª Yes, in several ways.
 The easiest one is the normal probability plot.
Normal Probability Plot
x = normrnd(10,1,25,1);
normplot(x)
0.99
0.98
0.95
0.90
Probability
0.75
Notice that the vertical axis is not
linear – the differences indicate the
distances of percentiles in the normal
distribution.
0.50
0.25
0.10
0.05
0.02
0.01
8.5
9
9.5
10
10.5
Data
11
11.5
12
© 2006All Rights Reserved, Robi Polikar, Rowan University, Dept. of Electrical and Computer Engineering
Normality
μ-3σ μ-2σ μ-σ
μ μ+σ μ+2σ μ+3σ
x = exprnd(10,100,1);
normplot(x)
Normal probability plot of exponential distribution
0.997
0.99
0.98
0.95
0.90
Probability
0.75
0.50
0.25
0.10
0.05
0.02
0.01
0.003
0
5
10
15
20
25
Data
30
35
40
45
© 2006All Rights Reserved, Robi Polikar, Rowan University, Dept. of Electrical and Computer Engineering
Normality
μ-3σ μ-2σ μ-σ
μ μ+σ μ+2σ μ+3σ
 Two additional tests that can be used to test normality are Kolgomorov-Smirnov test
and the Lilliefors test.
 H = kstest(X, alpha)
ª H = kstest(X) performs a Kolmogorov-Smirnov test to compare the values in the data
vector X with a standard normal distribution (that is, a normal distribution having mean 0
and variance 1). The null hypothesis for the Kolmogorov-Smirnov test is that X has a
standard normal distribution. The alternative hypothesis that X does not have that
distribution. The result H is 1 if you can reject the hypothesis that X has a standard normal
distribution, or 0 if you cannot reject that hypothesis. You reject the hypothesis if the test is
significant at the 100(1-alpha)% level.
 H=lillietest(X, alpha)
ª The Lilliefors test evaluates the hypothesis that X has a normal distribution with unspecified
mean and variance, against the alternative that X does not have a normal distribution. This
test compares the empirical distribution of X with a normal distribution having the same
mean and variance as X. It is similar to the Kolmogorov-Smirnov test, but it adjusts for the
fact that the parameters of the normal distribution are estimated from X rather than
specified in advance.
© 2006All Rights Reserved, Robi Polikar, Rowan University, Dept. of Electrical and Computer Engineering
μ-3σ μ-2σ μ-σ
μ μ+σ μ+2σ μ+3σ
Experimental Units &
Treatments
 In any statistical experiment, we have
ª Experimental units – the sample on which we observe the data of interest
ª Treatments – the procedure to be assigned to the experimental units
• For example, in medical trials, the patients are the units (sample) and the drug(s) are the
treatments.
© 2006All Rights Reserved, Robi Polikar, Rowan University, Dept. of Electrical and Computer Engineering
μ-3σ μ-2σ μ-σ
Experimental Units &
Treatments
μ μ+σ μ+2σ μ+3σ
 Recall the mileage example, where we compared two types of gas to find if
one of them provided better gas mileage on the taxis then the other:
ª The taxicabs are the experimental units
ª The treatments to be compared are
Gas A and Gas B
 For comparing chip speeds for
two (or more) designs
ª The microchips are the experimental
units
ª Different designs are the treatments
applied to these microchips (units)
© 2006All Rights Reserved, Robi Polikar, Rowan University, Dept. of Electrical and Computer Engineering
Experimental Design
μ-3σ μ-2σ μ-σ
μ μ+σ μ+2σ μ+3σ
 Statistical experimental design is used extensively not only in medicine, but
also in industrial process optimization, all social sciences (in particular,
sociology and psychology), financial sciences, and of course, engineering.
 All experimental designs use three basic
principles
ª Replication
ª Local Control
ª Randomization
© 2006All Rights Reserved, Robi Polikar, Rowan University, Dept. of Electrical and Computer Engineering
replication
μ-3σ μ-2σ μ-σ
μ μ+σ μ+2σ μ+3σ
 The same treatement(s) are assigned to many different experimental units
ª Both aspirin and placebo are given to many patients
ª Each type of gas is used on many taxicabs
ª The speed of each chip design is tested on many many chips
 Without replication, it would be impossible to
assess the natural variability and measurement error
ª Say, we test the aspirin and placebo on one
patient each, and indeed the person taking
placebo gets a heart attack, while the other
does not. Is this sufficient evidence that
aspirin does really prevent heart attacks…?
© 2006All Rights Reserved, Robi Polikar, Rowan University, Dept. of Electrical and Computer Engineering
Local Control
μ-3σ μ-2σ μ-σ
μ μ+σ μ+2σ μ+3σ
 Local control is the method that accounts for and reduces natural
variability in the data that may be due other factors than the treatment itself
 One approach is to group similar experimental units into blocks.
ª In the taxi example, each gas type is used on all cabs (with mean drivers and gentle
drivers- where the driving habits may account for the variability). Each type of
driver would then be a block.
ª For heart attack example, we may want to separate all subjects to whether they
smoke or not.
• A multicomponent analysis can then tell us whether the difference is due to drug or due
to smoking habits. Each group of people would then constitute a block.
© 2006All Rights Reserved, Robi Polikar, Rowan University, Dept. of Electrical and Computer Engineering
Randomization
μ-3σ μ-2σ μ-σ
μ μ+σ μ+2σ μ+3σ
 The must-have step in all statistical experiments!
 Treatments must be assigned randomly to experimental units.
ª For the taxi example, we assign each gas randomly. Say, for example, we assign gas
A to cab 1 on Tuesday and Gas B on Wednesday, or vice versa by flipping a coin
ª This would also prevent different weather conditions to have an effect on the gas
mileage.
 Bottom line: All treatments are
assigned to all subjects by random!!!
© 2006All Rights Reserved, Robi Polikar, Rowan University, Dept. of Electrical and Computer Engineering
μ-3σ μ-2σ μ-σ
Complete
Randomized Block
μ μ+σ μ+2σ μ+3σ
 Suppose we would like to investigate the effect of two brands of tires as well
as two types of gasolines on the gas mileage
 We have four possible treatments for each cab. The two factors are gas and
tire make. Note that this would lead to a two-factor anova test.
Shelf Gas Amigo Gas
Good-week
tires
a
b
Michell’s-in
tires
c
d
 We can assign four treatments at random to four different days for each cab.
All four treatments (a, b, c, d) are repeated within each block (cab). This is
complete randomized block
© 2006All Rights Reserved, Robi Polikar, Rowan University, Dept. of Electrical and Computer Engineering
Latin Square
μ-3σ μ-2σ μ-σ
μ μ+σ μ+2σ μ+3σ
 An n2–by–n2 table with each of the n-treatments appearing only once in each row
and column is called a Latin Square.
 What if we had more then 4 cabs or 4 days? Then we can
increase the number of experimental units by repeating
the experimental designs:
ª For example, if we had 8 cabs, we can divide them into two
groups of four and then repeat the design with 4each group
ª The randomization step then picks a single Latin square from a list of all possible four-way
latin squares.
© 2006All Rights Reserved, Robi Polikar, Rowan University, Dept. of Electrical and Computer Engineering
Multiway Anova
μ-3σ μ-2σ μ-σ
μ μ+σ μ+2σ μ+3σ
 So, how do we handle such a complex design, particularly when there are many
other factors and many different treatment plans…?
 Multiway ANOVA testing. We allocate the total variability in the data among
different sources (cars, days, tires, gas and random error). Anova partitions the total
variability into these sources and we look at how much of the variability can be
explained with which of the sources. – Done using statistical software, e.g, SPSS
© 2006All Rights Reserved, Robi Polikar, Rowan University, Dept. of Electrical and Computer Engineering
μ-3σ μ-2σ μ-σ
What have we learned
this semester?
μ μ+σ μ+2σ μ+3σ
 Probability
ª Axioms of probability, fundamental principle of counting, permutations,
combinations, conditional probability, Bayes rule
 Random variables
ª Continuous and discrete probability distributions, probability mass and density
functions, cumulative probability function, random variables associated with
different distributions,
 Specific Distributions
ª Binomial, hypergeometric, negative binomial, geometric, poisson, normal, chisquared, t-, f-distributions, their parameters, joint distributions, marginal
distributions, mean and variance, co-variance, correlation, correlation coefficient,
covariance
© 2006All Rights Reserved, Robi Polikar, Rowan University, Dept. of Electrical and Computer Engineering
μ-3σ μ-2σ μ-σ
What Have we
Learned This Semester
μ μ+σ μ+2σ μ+3σ
 Statistics – Population vs. Sample
 Random sampling
ª independent and identically distributed samples, sampling distribution (Gaussian),
effect of sample size, central limit theorem
© 2006All Rights Reserved, Robi Polikar, Rowan University, Dept. of Electrical and Computer Engineering
μ-3σ μ-2σ μ-σ
What Have we
Learned This Semester
μ μ+σ μ+2σ μ+3σ
 Confidence intervals
ª Confidence interval
and confidence levels
for means and
populations, large vs. small
sample size
α/2
-zα/2
1-α
0
⎛
α/2
μ = ⎜ x ± zα / 2
⎝
+zα/2
s ⎞ ⎛⎜
⎛
p = ( pˆ ± zα / 2σ pˆ ) ≈ ⎜ pˆ ± zα / 2
⎟ = pˆ ± zα / 2
n ⎠ ⎜⎝
⎝
p(1 − p ) ⎞
⎟
⎟
n
⎠
s ⎞
⎟
n⎠
μ = x ± tα 2, n −1 ⋅
s
© 2006All Rights Reserved, Robi Polikar, Rowan University, Dept. of Electrical and Computer Engineering
n
μ-3σ μ-2σ μ-σ
μ μ+σ μ+2σ μ+3σ
What Have we
Learned This Semester
 Hypothesis testing: Compare different scenarios to understand whether a
given statistical result be due to chance or due to some factor
ª Null hypothesis, alternative hypothesis, level of test, p-value, z-test, t-test, type I
error, type II error, two sample z- and t- tests, paired tests, pooled procedures, Ftest for variances
© 2006All Rights Reserved, Robi Polikar, Rowan University, Dept. of Electrical and Computer Engineering
μ-3σ μ-2σ μ-σ
What Have we
Learned This Semester
μ μ+σ μ+2σ μ+3σ
 ANOVA – Analysis of variance
ª Testing for comparing three or more populations, do the means of all are the same
or is any one of them different then the others?
ª Mean square for treatment, mean square error, total sum of squares, treatment sum
of squares, error sum of squares, F-statistic, ANOVA table
 Multiple component analysis
ª What if the null hypothesis is rejected, that is not all sample means are identical.
Which ones are different?
ª Simplified Tukey’s test
 MATLAB commands for commonly used statistical tests
© 2006All Rights Reserved, Robi Polikar, Rowan University, Dept. of Electrical and Computer Engineering
What Next?
μ-3σ μ-2σ μ-σ
μ μ+σ μ+2σ μ+3σ
 We have covered most of classical statistics in one credit, one
semester course
ªYou now have a working knowledge of statistical concepts
ªEven if you do not remember the exact equations (you can always
look them up in books, or use statistical software), you now know
what tools are available.
ªApply these tools to a variety of engineering or non-engineering
problems.
© 2006All Rights Reserved, Robi Polikar, Rowan University, Dept. of Electrical and Computer Engineering
The End
μ-3σ μ-2σ μ-σ
μ μ+σ μ+2σ μ+3σ
 Understand when and how statistics can be
used to make intelligent inferences
ª Remember, it is very easy to lie with statistics
– do not believe every statistical result you
read / hear unless you know using what
statistical tests and assumptions were used in
computing those results
ª As a Rowan educated engineer, it is your
responsibility to use statistics judiciously and
honestly and interpret your results accordingly.
ª Do not make claims unless you can justify
your them with compelling statistical evidence
obtained through well designed statistical
experiments.
© 2006All Rights Reserved, Robi Polikar, Rowan University, Dept. of Electrical and Computer Engineering