Sample Size Estimations Demystifying Sample Size Calculations Graphics contributed by Dr. Gillian Bartlett © Nancy E. Mayo 2004 Choosing the Study Population Question Background Population Reasonable Question © Nancy E. Mayo 2004 Study Population Exposure ? © Nancy E. Mayo 2004 1. 2. 3. COMMON QUESTIONS How many subjects (specimens) do I need? How do I analyze my data? What do I put in the data analysis section? 1. 2. 3. 4. 5. COMMON ANSWERS What is your question? What is your outcome? How is it measured? How big an effect do you want to see? Is the effect meaningful? © Nancy E. Mayo 2004 Clinically Meaningful Change Meaningful to whom? • Clinician - usually impairments • Patient – function (disability), quality of life • Society - health services utilization, cost • Payer – disability, prescription medication © Nancy E. Mayo 2004 Clinically Meaningful Change • Norm referenced – refers to changes that would put someone within normal values or within a % of normal • Criterion referenced – change anchored in future benefit – change is associated with increased probability of distant outcomes – relevant when impact is on pathology but benefit not reaped for years © Nancy E. Mayo 2004 Clinically Meaningful Change • Content referenced – for outcomes measured by scales – translates change into what would have had to have changed on the scale – e.g. 5 points on Barthel Index - changed 1 level on 1 item. • Minimally detectable change – Subjects can detect improvement © Nancy E. Mayo 2004 How BIG is BIG? Effect size: ratio of change to variability 0.2 - 0.3 – small 0.5 – moderate 0.8 - large © Nancy E. Mayo 2004 Change greater than “noise” signal is difficult to detect against excessive background noise © Nancy E. Mayo 2004 Raw vs. Cooked Data (order rare) Raw Data Cooked Data (<50, >=50) 39 0 43 0 68 1 56 1 78 1 22 0 34 0 49 0 50 1 51 1 35 0 55 1 48 0 29 0 33 0 78 1 56 1 69 1 53 1 66 1 Mean 50.6 SD 15.8 Ratio 3.2 Mean 0.55 SD 0.51 Ratio 1.1 © Nancy E. Mayo 2004 Examples of the Pitfalls of Cooking Data Raw Data Cooked Data (<50, >=50) 22 0 29 0 33 0 34 0 35 0 39 0 43 0 48 0 49 0 50 1 51 1 53 1 55 1 56 1 56 1 66 1 68 1 69 1 78 1 78 1 Mean 50.6 SD 15.8 Ratio 3.2 Mean 0.55 SD 0.51 Ratio 1.1 © Nancy E. Mayo 2004 DEMYSTIFIED Sample Size Formula = SD / delta Effect size = delta / SD Delta = difference © Nancy E. Mayo 2004 Relationship between Effect Size and Sample Size Sample Size per Group 100 80 60 40 20 0 0 0.5 1 1.5 Effect Size (Two group design) © Nancy E. Mayo 2004 2 2.5 Calculation of Sample Size for Comparing Two Independent Means n= 2 ( za – zb ) SD ___________ xexp - xcon 2 Where: Za = z value for the risk of a Type I error (significance level) 1.96 for 0.05 Zb = z value for the risk of a Type II error (power) 1.96 for 0.95 (two-tailed) -1.65 for 0.95 (one-tailed) SD = standard deviation of outcome in the general population xcon = mean of control group xexp = mean of experimental group n = number of subjects per group © Nancy E. Mayo 2004 Calculation of Sample Size for Comparing Two Independent Proportions n= za √ 2 pcon (1 - pcon ) – zb √ pexp (1 – pexp ) + pcon (1 - pcon ) ______________________________________________ pexp - pcon 2 Where: za = z value for the risk of a Type I error (significance level) 1.96 for 0.05 zb = z value for the risk of a Type II error (power) 1.96 for 0.95 (two-tailed) -1.65 for 0.96 (one-tailed) pcon = prevalence of outcome in control group pcon = prevalence of outcome in experimental group n = number of subjects per group © Nancy E. Mayo 2004 Colton (pg 168-169) Sample Size Required Per Group for Comparing Two Independent Means Ratio of SD to difference ∆ between means (∆/SD) POWER .80 .90 .95 0.50 (2.0) 5 7 8 1.0 (1.0) 17 23 27 1.25 (0.8) 26 34 42 1.50 (0.67) 37 49 60 2.0 (0.5) 60 86 105 © Nancy E. Mayo 2004 Sample Size Required Per Group for Comparing Two Independent Proportions: 80% Power PREVALENCE OF OUTCOME IN CONTROL GROUP Prevalence of outcome in experimental group .05 .10 .20 .30 .40 .10 475 .15 160 726 .20 88 219 .25 59 113 1134 .30 43 72 313 .35 51 151 1417 .40 38 91 376 .45 30 62 176 1574 .50 25 45 103 408 68 186 .55 .50 .60 107 408 .65 70 183 .70 49 103 .75 36 66 .80 28 45 .90 17 25 © Nancy E. Mayo 2004 Sample Size Required Per Group for Comparing Two Independent Proportions: 95% Power PREVALENCE OF OUTCOME IN CONTROL GROUP Prevalence of outcome in experimental group .05 .10 .20 .30 .40 .50 .10 758 .15 251 1174 .20 137 349 .25 90 177 `850 .30 65 111 505 .35 78 241 2318 .40 58 144 609 .45 45 97 281 2578 .50 36 70 163 661 .55 53 107 299 2630 .60 41 75 170 661 .65 33 56 109 293 .70 43 75 163 .75 34 55 103 41 70 25 36 .80 .90 © Nancy E. Mayo 2004 More complex data situations • Convert each component to the simple 2group comparison or correlation • Estimate (calculate) sample size for the contrast that has the smallest effect and build up • Remember if using correlation as the base, you are not testing it against 0 you are testing it against a correlation that you do not think is important © Nancy E. Mayo 2004 …More • Consider the impact on power to maintain a given effect size if other variables are in the model © Nancy E. Mayo 2004 Regression • Green indicates that adequate power (80%) can be achieved for moderate effect sizes with a sample size N > 50 + 8m, where m is the number of covariates to be modeled. © Nancy E. Mayo 2004 Adjustment only • no parameters are estimated • no hypotheses tested • to maintain the same degree of power, only 1 additional subject is required per level (l) or per degree of freedom (df) inherent in the covariate © Nancy E. Mayo 2004 Summary • • • • Variable under study N > 50 + 8m (moderate effect size, 80% power) Adjustment only Continuous = + 1 per covariate df Dichotomous = 5-9 events per covariate • Sub-group analysis • Sample size for main effect * 4 for interaction with group © Nancy E. Mayo 2004 Marking Scheme for Protocol • • • • • • • • • Background 10 Question 5 Population 5 Design 5 Procedures 5 Measures 10 Analysis 5 Sample Size 5 Bonus points – above and beyond the call of duty © Nancy E. Mayo 2004