Effect Size &
Power Analysis
0246-511 Statistics for Graduate Study I
1
Learning Objectives
 Understand common effect size measures
 Which to use
 When to use different types
 Learn to employ effect sizes to plan study
 A priori power analysis
 Two groups
 ANOVA
 Observed power
2
Effect Size
 Two Broad Types (in ANOVA)
 Standardized mean differences
 Proportion of variance accounted for
 Compared to statistical significance
 APA Guidelines
 Use in Meta-Analysis
3
Effect Size: Comparing
Two Means
Standardized Mean Differences
Population Value
Estimated from sample
1  2
12 
 error
X1  X 2
d
MSW
Proportion of Variance Accounted for
2
t
r  2
t  df
2
4
Concepts (Two-Groups)








Power
Type I & Type II Error
Effect Size
γ (gamma)
δ (delta)
Estimating Power
Estimating Needed Sample Size
Observed Power
5
Review




Power
Type I Error
Type II Error
Factors that affect power




6
Independent Samples t-test:
Things we need to estimate power
 Estimate Effect Size
 Estimate Gamma
 Estimate Delta
 Estimate Power
 Working backwards to get N
7
An example
Fire Safety Instruction
 Two methods of
teaching
 Pilot Study
 Need to estimate
sample size needs to
get funding
 Power of .80 at alpha
.05 and .01
Score
1
X1
77
X2
77
2
3
4
67
60
79
80
81
82
5
6
7
8
65
71
68
77
60
70
74
74
9
10
69
80
77
83
8
We will need to
 Estimate of Effect Size
 Gamma
 Estimate Delta
1  2
x1  x2

,d 

sp
 Estimate (observed)
Power
 Work backwards to get n
n
 
2
n
2
2
2
9
Results from Pilot Study
Score
1
2
3
4
5
6
7
8
9
10
Mean
Sum
SS
Std Dev
Variance
pooled var
std err
t
X1
77
67
60
79
65
71
68
77
69
80
71.30
713.00
402.10
6.68
44.68
46.09
3.04
-1.48
X2
77
80
81
82
60
70
74
74
77
83
75.80
758.00
427.60
6.89
47.51

71.3  75.8
 0.66
46.09
10
  0.66
 1.48
2
Observed Power (.05) ≈ 0.32
Observed Power (.01) ≈ 0.14
Note: strictly speaking, all we need to
estimate n for power of .80 is gamma, we
compute observed power for illustration
and because it is sometimes helpful.
10
Obtaining estimate of n
2(2.802 )
n.05 
 36.0
2
0.66
2(3.402 )
n.01 
 53.08
2
0.66
Note1: these are not
obtained delta estimates,
but based on what is
needed to obtain desired
power level
Note 2: these are “n”
values not “N” values,
thus need this many per
condition.
11
Unequal Sample Sizes
For unequal sample sizes we often use something called a harmonic
mean. It is calculated as follows:
k
Xh 
1

Xi
If our group sizes had been 14 and 9:
Xh 
2
1 1

14 9
 11.11
12
Effect Sizes: More than
two means
Population Based Measure
2
2



 2  total 2 error
 total
Estimated from Sample
SSM  (k  1) MSW
 
SSTOT  MSW
2
Another sample measure
SS M
 
SStot
2
Also called R2
13
A brief digression
ω2 can be derived from the F statistic as well:
(k  1)( F  1)
 
(k  1)( F  1)  kn
2
You will care shortly. 
14
Cohen’s Effect Size
Estimate
 Cohen defines a population ANOVA ES
estimate as:
M
f 

 Stevens provides an estimate of this as:
(k  1) F
f 
N
^
 Keppel & Wickens provide another estimate:
2
f 
1 2
^
15
Extend previous example
Score
1
2
3
4
5
6
7
8
9
10
Mean
SS
Source
Method
Within
Total
X1
77
67
60
79
65
71
68
77
69
80
71.3
402.1
X2
77
80
81
82
60
70
74
74
77
83
75.8
427.6
X3
79
74
69
70
71
63
69
69
72
75
71.1
166.9
SS
141.267
996.600
1137.867
df
2
27
29
MS
70.633
36.911
2 
141.267  (3  1)36.911
 .06
1137.867  36.911
F
1.914
0.167
16
Use power chart to obtain a
noncentrality paramter: Φ
17
Using power charts to get
sample size
What is a noncentrality parameter?
From examining a power chart, Φ≈1.8
Now we substitute our obtained Φ value into the following equation:
n 
2
1


2
2
1  .06
n  1.8
 50.76
.06
2
Rounding up to 51, this implies 3(51-1)=150 for dfdenom.
We need to iterate again. Using the infinity line, we get…
1  .06
n  1.83
 52.44
.06
2
18
Strategy for assessing
power
 Determine form of experiment/research
 Decide on hypothesis test
 Carefully consider which effect(s) is (are)
important to detect
 Select desired power (& alpha)
 Determine likely effect size (or range of effect
sizes)
 Conduct power analysis
 Consider whether study is feasible as currently
designed
19
In-Class Example
 Students’ (MA Thesis) involves trying to
find a hypothesized effect.
 Wish to detect effect it if it is at least
ω2=.03
 df numerator will be 1
 How many subjects will she have to
recruit?
20
Software Options
 SPSS (observed power)
 G*Power (consistent with Dr. Jackson’s
2nd principle of economics)
 Web-based power calculators
 Others
21