advertisement

Linear Contrasts and Multiple Comparisons (§ 8.6) • • One-way classified design AOV example. Develop the concept of multiple comparisons and linear contrasts. • Multiple comparisons methods needed due to potentially large number of comparisons that may be made if Ho rejected in the one-way AOV test. Fisher’s Protected LSD Terms: Tukey’s W MCP One-way classification Studentized range distribution Linear Contrasts Experimentwise error rate Multiple comparisons Comparisonwise error rate Data dredging Student-Newman-Keuls procedure Mutually orthogonal contrasts Duncan’s new MRT Waller-Duncan k-ratio MCP STA 6166 - MCP 1 One-Way Layout Example A study was performed to examine the effect of a new sleep inducing drug on a population of insomniacs. Three (3) treatments were used: Standard Drug New Drug Placebo (as a control) What is the role of the placebo in this study? What is a control in an experimental study? 18 individuals were drawn (at random) from a list of known insomniacs maintained by local physicians. Each individual was randomly assigned to one of three groups. Each group was assigned a treatment. Neither the patient nor the physician knew, until the end of the study, which treatment they were on (double-blinded). Why double-blind? A proper experiment should be: randomized, controlled, and double-blinded. STA 6166 - MCP 2 Response: Average number of hours of sleep per night. Placebo: Standard Drug: New Drug: 6.5, 5.7, 5.1, 3.8, 4.6, 5.1 8.4, 8.2, 8.8, 7.1, 7.2, 8.0 10.6, 6.6, 8.0, 8.0, 6.8, 6.6 yij = response for the j-th individual on the i-th treatment. Standard Placebo Drug New Drug 5.60 8.40 10.60 5.70 8.20 6.60 5.10 8.80 8.00 3.80 7.10 8.00 4.60 7.20 6.80 5.10 8.00 6.60 sum 29.900 47.700 46.600 mean 4.983 7.950 7.767 variance 0.494 0.455 2.359 pooled variance 1.102 SSW 16.537 variance of the means 2.764 Between mean SSQ (SSB) 16.582 Degrees Sums of of Source Squares Freedom Between Groups 33.16 2 Within Groups 16.54 15 Total 49.70 17 TSS (y ij y ) 2 i, j Hartley’s test for equal variances: Fmax = 4.77 < Fmax_critical = 10.8 Mean Square F statistic 16.582 15.04 1.102 P-value 0.00026 SSW SSB ( yij yi ) 2 ni ( yi y ) 2 i, j i SSB t 1 SSW 2 MSW sw nT t MSB F ~ Fdfb ,dfw MSW 2 MSB sb STA 6166 - MCP 3 Excell Analysis Tool Output Anova: Single Factor SUMMARY Groups Placebo Standard Drug New Drug ANOVA Source of Variation Between Groups Within Groups Total Count 6 6 6 SS 33.163333 16.536667 49.7 Sum Average Variance 29.9 4.983333 0.493667 47.7 7.95 0.455 46.6 7.766667 2.358667 df MS F 2 16.58167 15.04082 15 1.102444 P-value F crit 0.00026 3.682317 17 What do we conclude here? STA 6166 - MCP 4 Linear Contrasts and Multiple Comparisons If we reject H0 of no differences in treatment means in favor of HA, we conclude that at least one of the t population means differs from the other t-1. Which means differ from each other? Multiple comparison procedures have been developed to help determine which means are significantly different from each other. Many different approaches - not all produce the same result. Data dredging and data snooping - analyzing only those comparisons which look interesting after looking at the data – affects the error rate! Problems with the confidence assumed for the comparisons: 1-a for a particular pre-specified comparison? 1-a for all unplanned comparisons as a group? STA 6166 - MCP 5 Linear Comparisons Any linear comparison among t population means, m1, m2, ...., mt can be written as: l a1m1 a2m2 at mt t ai 0 Where the ai are constants satisfying the constraint: i 1 Example: To compare m1 to m2 we use the equation: l m1 m2 with coefficients a1 1 a2 1 a3 a4 at 0 Note constraint is met! m2 m3 l m1 (1)m1 ( 21 )m 2 ( 21 )m 3 2 STA 6166 - MCP 6 Linear Contrast l m 1 m 2 y1 y 2 A linear comparison estimated by using group means is called a linear contrast. l m m 2 m 3 (1)y ( 1 )y ( 1 )y 1 1 2 2 2 3 2 Variance of a linear contrast: V (l) 2 a1 sw n 1 2 2 a2 n2 at n t t ai yi Test of i 1 MSl significance t ai2 i 1 ni H : l = 0 vs. H : l 0 o 2 2 2 sw t i 1 2 ai ni sw2 MSE ( MSW ) MSl F ~ F1,nT t MSE a STA 6166 - MCP 7 Orthogonal Contrasts lˆ1 a1 y1 a2 y2 at yt lˆ b y b y b y 2 1 1 2 2 t t These two contrasts are said to be orthogonal if: t l1 l2 a1b1 a2b2 at bt ai bi 0 i 1 in which case l1 conveys no information about l2 and viceversa. A set of three or more contrasts are said to be mutually orthogonal if all pairs of linear contrasts are orthogonal. STA 6166 - MCP 8 ˆl y y2 y3 1 1 2 lˆ y y 2 2 a1 1 1 a2 2 1 a3 2 Compare average of drugs (2,3) to placebo (1). Contrast drugs (2,3). 3 b1 0 Orthogonal b2 1 b3 1 Non-orthogonal Contrast Standard drug (2) to placebo (1). Contrast New drug (3) to placebo (1). lˆ3 y1 y2 lˆ4 y1 y3 a1 1 b1 1 a2 1 b2 0 a3 0 b3 1 STA 6166 - MCP 9 Drug Comparisons Standard Placebo Drug New Drug 5.60 8.40 10.60 5.70 8.20 6.60 5.10 8.80 8.00 3.80 7.10 8.00 4.60 7.20 6.80 5.10 8.00 6.60 sum 29.900 47.700 46.600 mean 4.983 7.950 7.767 variance 0.494 0.455 2.359 pooled variance 1.102 SSW 16.537 variance of the means 2.764 Between mean SSQ (SSB) 16.582 F MSl1 ~ F1, nT t MSE MSlˆ1 33.10 F1 30.04 MSE 1.102 MSlˆ2 0.10 F2 0.09 MSE 1.102 F1,15,.05 4.54 Degrees Sums of of Source Squares Freedom Between Groups 33.16 2 Within Groups 16.54 15 Total 49.70 17 Mean Square F statistic 16.582 15.04 1.102 P-value 0.00026 y y 7.95 7.77 lˆ1 y1 2 3 4.98 2.88 2 2 lˆ y y 7.95 7.77 0.18 2 2 3 2 (1) 2 (.5) 2 (.5) 2 a22 a32 2 a1 ˆ 1.102(0.25) V l1 sW 1.102 n n n 6 6 6 2 3 1 2 (0) 2 (1) 2 (1) 2 b22 b32 2 b1 ˆ 1.102(0.33) V l2 sW 1.102 n n n 6 6 6 2 3 1 2 2 lˆ1 2.88 33.10 ai2 0.25 ni i 2 2 lˆ2 0.18 ˆ ˆ SSl2 MSl2 0.10 bi2 0.33 STA 6166 - MCP i n i SSlˆ1 MSlˆ1 10 Importance of Mutual Orthogonality Assume t treatment groups, each group having n individuals (units). • • t-1 mutually orthogonal contrasts can be formed from the t means. (Remember t-1 degrees of freedom.) Treatment sums of squares (SSB) can be computed as the sum of the sums of squares associated with the t-1 orthogonal contrasts. (i.e. the treatment sums of squares can be partitioned into t-1 parts associated with t-1 mutually orthogonal contrasts). contrasts orthogonal SSl1 SSlt 1 SSB t-1 independent pieces of information about the variability in the treatment means. STA 6166 - MCP 11 Example of Linear Contrasts Objective: Treatments: Treatment A B C D Test the wear quality of a new paint. Weather and wood combinations. Code m1 m2 m3 m4 Combination hardwood, dry climate hardwood, wet climate softwood, dry climate softwood, wet climate (Obvious) Questions: Q1: Is the average life on hardwood the same as average life on softwood? Q2: Is the average life in dry climate the same as average life in wet climate? Q3: Does the difference in paint life between wet and dry climates depend upon whether the wood is hard soft? STAor 6166 - MCP 12 Treatment Mean (in years) ni Population parameter A B C D 13 3 14 3 20 3 21 3 m1 m2 m3 m4 Q1 MSE= t= nt -t= 5 4 8 Q1: Is the average life on hardwood the same as average life on softwood? 1 0 H : m1 m 2 m3 m 4 2 2 Comparison: OR m1 m 2 m3 m 4 2 2 0 l1 ( 12 )m1 ( 12 )m2 ( 12 )m3 ( 12 )m 4 l̂1 ( 12 )y1 ( 12 )y 2 ( 12 )y3 ( 12 )y 4 Estimated Contrast Test H0: l1 = 0 versus HA: l1 0 Test Statistic: F What is MSl1 ? MSl1 MSE Rejection Region: Reject H0 if F F1,nT t,a STA 6166 - MCP 13 2 t 2 a y 1 1 1 1 i i y y y y 1 2 3 4 2 2 2 2 MSl1 i1t 2 ai 1 2 1 2 1 2 1 2 2 2 2 2 n i 1 i n2 n3 n4 n1 2 t 2 1 1 1 1 ai yi 13 14 20 21 2 2 2 2 49 MSl1 i1t 2 147 2 2 2 2 1 ai 1 1 12 12 2 2 3 n i1 i 3 3 3 3 MSl1 147 = = 29.4 MSE 5 F1,8,0.05 = 5.32 F= Conclusion: Since F=29.4 > 5.32 we reject H0 and conclude that there is a significant difference in average life on hard versus soft woods. STA 6166 - MCP 14 Treatment Mean (in years) ni Population parameter A B C D 13 3 14 3 20 3 21 3 m1 m2 m3 m4 Q2 MSE= t= nt -t= 5 4 8 Q2: Is the average life in dry climate the same as average life in wet climate? H 02 : m1 m 3 2 m2 m4 2 OR m1 m 3 m 2 m 4 0 2 2 Comparison: l2 ( 12 )m1 ( 12 )m 2 ( 12 )m3 ( 12 )m 4 l̂2 ( 12 )y1 ( 12 )y 2 ( 12 )y3 ( 12 )y 4 Estimated Contrast Test H0: l2 = 0 versus HA: l2 0 Test Statistic: F = MSl2 MSE Rejection Region: Reject H0 if F > F1,nT - t,a STA 6166 - MCP 15 2 t 2 a y 1 1 1 1 i i y y y y 1 2 3 4 2 2 2 2 MSl2 i1t 2 ai 1 2 1 2 1 2 1 2 2 2 2 2 n i 1 i n2 n3 n4 n1 2 t 2 1 1 1 1 ai yi 13 14 20 21 2 2 2 2 12 i1 MSl2 3 2 2 2 2 2 t 1 ai 1 1 12 12 2 2 3 n i1 i 3 3 3 3 MSl2 3 = = 0.6 MSE 5 F1,8,0.05 = 5.32 F= Conclusion: Since F=0.6 < 5.32 we do not reject H0 and conclude that there is not a significant difference in average life in wet versus dry climates. STA 6166 - MCP 16 Treatment Mean (in years) ni Population parameter A B C D 13 3 14 3 20 3 21 3 m1 m2 m3 m4 Q3 MSE= t= nt -t= 5 4 8 Q3: Does the difference in paint life between wet and dry climates depend upon whether the wood is hard or soft? H30 : m1 m 2 m3 m 4 OR (m1 m 2 ) (m3 m 4 ) 0 Comparison: l3 (1)m1 ( 1)m2 ( 1)m3 (1)m 4 l̂3 (1)y1 (1)y 2 ( 1)y3 (1)y 4 Estimated Contrast Test H0: l3 = 0 versus HA: l3 0 Test Statistic: F MSl3 MSE Rejection Region: Reject H0 if F F1,nT t,a STA 6166 - MCP 17 2 t 2 a y i i 1 y 1 y 1 y 1 y 1 2 3 4 MSl3 i1t 2 ai 12 12 12 12 n i 1 i n2 n3 n4 n1 2 t 2 ai yi 1 13 1 14 1 20 1 21 02 i1 MSl3 0 2 2 2 2 2 t ai 1 4 1 1 1 3 n i1 i 3 3 3 3 MSl3 0 = = 0 MSE 5 F1,8,0.05 = 5.32 F= Conclusion: Since F=0 < 5.32 we do not reject H0 and conclude that the difference between average paint life between wet and dry climates does not depend on wood type. Likewise, the difference between average paint life for the wood types does STA 6166 - MCP not depend on climate type (i.e. there is no interaction). 18 Mutual Orthogonality Contrast a1 a2 a3 a4 l1 1 2 1 2 12 12 l2 1 2 1 2 1 2 12 l3 1 1 1 1 l1 l2 14 14 14 14 0 l1 l3 12 12 12 12 0 l2 l3 12 12 12 12 0 The three are mutually orthogonal. SSl1 = MSl1 SSl2 = MSl2 SSl3 = MSl3 Treatment SS = = = = 147 3 0 150 The three mutually orthogonal contrasts add up to the Treatment Sums of Squares. Total Error SS = dferror x MSE = 8 x 5 = 40 STA 6166 - MCP 19 Types of Error Rates Compairsonwise Error Rate - the probability of making a Type I error in the comparison of two means. (what we have been discussing for all tests up to this point). Experimentwise Error Rate - the probability of observing an experiment in which one or more of the pairwise comparisons are incorrectly declared significantly different. (Type I error.) Number of Type I Experimentwise comparsons Error Rate Error Rate 1 0.05 0.050 2 0.05 0.098 3 0.05 0.143 4 0.05 0.185 5 0.05 0.226 6 0.05 0.265 7 0.05 0.302 8 0.05 0.337 9 0.05 0.370 10 0.05 0.401 11 0.05 0.431 12 0.05 0.460 13 0.05 0.487 14 0.05 0.512 15 0.05 0.537 16 0.05 0.560 17 0.05 0.582 18 0.05 0.603 19 0.05 0.623 20 0.05 0.642 STA 6166 - MCP 20 Error Rates: Problems Suppose we make c mutually orthogonal (independent) comparisons, each with Type I comparisonwise error rate of a. The experimentwise error rate, e, is then: c e 1 (1 a ) (If the comparisons are not orthogonal, then the experimentwise error rate is smaller.) Number of Type I Experimentwise comparsons Error Rate Error Rate 1 0.05 0.050 2 0.05 0.098 3 0.05 0.143 4 0.05 0.185 5 0.05 0.226 6 0.05 0.265 7 0.05 0.302 8 0.05 0.337 9 0.05 0.370 10 0.05 0.401 11 0.05 0.431 12 0.05 0.460 13 0.05 0.487 14 0.05 0.512 15 0.05 0.537 16 0.05 0.560 17 0.05 0.582 18 0.05 0.603 19 0.05 0.623 20 0.05 0.642 Solution (Bonferroni): set e=0.05 and solve for a. But there’s a problem… E.g. if c=8, we get a=0.0064! STA 6166 - MCP 21 Multiple Comparison Procedures Terms: • If the multiple comparison procedure (MCP) requires a significant overall F test, then the procedure is labeled a “Protected” method. • Not all procedures produce the same results. • The major differences among all of the different MCPs is in the calculation of the “yardstick” used to determine if two means are significantly different. The yardstick can generically be referred to as the least significant difference. Any two means greater than this difference are declared significantly different. y i y j " yardstick" " TabledValue"" SEof difference" • Yardsticks are composed of a standard error term and a critical value from some tabulated statistic. • Some procedures have “fixed” yardsticks, some have “variable” yardsticks. The variable yardsticks will depend on how far apart two observed means are in a rank ordered list of the mean values. • Some procedures control Comparisonwise Error, other Experimentwise Error, and some attempt to control both. STA 6166 - MCP 22 Fisher’s Least Significant Difference - Protected Mean of group i (mi) is significantly different from the mean of group j (mj) if if all groups have same size n. y i y j LSD LSD t a 2 ,dferror 2 1 1 sw ni n j t a 2 ,dferror MSE 2 n {tabled value}{standard error of difference} Type I (comparisonwise) error rate = a This procedure controls Comparisonwise Error. Experimentwise error control comes from requiring a significant overall F test prior to performing any means comparisons. How well does it work? STA 6166 - MCP 23 Tukey’s W (Honestly Significant Difference) Procedure Means are different if: yi y j W W = qa (t , nT - t ) sw2 (1n ) = qa (t , nT - t ) MSE n {Table 11 - critical values of the studentized range.} Experimentwise error rate = a This MCP controls experimentwise error rate! Comparisonwise error rates is thus very low. How well does it work? STA 6166 - MCP 24 Student Newman Keul Procedure Rank the t sample means from smallest to largest. For two means that are r “steps” apart in the ranked list, we declare the population means different if (modified Tukey’s MCP): y i y j Wr Wr qa (r , nT t ) qa (r, nT t ) 2 1 sw n MSE n {Table 11 - critical values of the studentized range. Depends on which mean pair is being considered!} y [1] min r=2 y [ 3] y [2] r=3 y [5] y [4] r=4 r=5 y [ 6] max r=6 varying yardstick STA 6166 - MCP 25 Duncan’s New Multiple Range Test Neither an experimentwise or comparisonwise error rate control alone. Based on a ranking of the observed means. Introduces the concept of a “protection level” (1-a)r-1 Number of steps Apart, r 2 3 4 5 6 7 y i y j Wr Protection Level (0.95)r-1 .950 .903 .857 .815 .774 .735 Probability of Falsely Rejecting H0 .050 .097 .143 .185 .226 .265 Wr q a (r , nT t ) MSE n {Table A -11 in these notes} STA 6166 - MCP 26 Waller-Duncan k-ratio MCP (Protected) A MCP that uses the sample data to help in determining whether we need to use a conservative rule (e.g. Tukey’s MCP) or a nonconservative rule (e.g. Fisher’s MCP). y i y j LSD LSD t c MSE n2 tc is obtained from Table A-12 or A-13 (in these notes) and is based on: • k - the error weight ratio which designates the seriousness of a Type I error to a Type II error. (typical values 50, 100, 500. • df1 - the model degrees of freedom. (i.e. t-1). • df2 - the error degrees of freedom (i.e. t(n-1)). • F - the F statistics from the overall model effects test. • the assumption of equal group sample sizes. STA 6166 - MCP 27 Scheffé’s S Method For any linear contrast: Estimated by: With estimated variance: l a1m1 a2m2 at mt l̂ a1y1 a2 y 2 at y t V̂( l̂ ) s a12 2 w n1 a 22 n2 To test H0: l = 0 versus Ha: l 0 For a specified value of a, reject H0 if: where: Confidence interval: a 2t nt s t 2 w i1 ai2 ni lˆ S S V̂( l̂ ) ( t 1)F( t 1),(nT t ),a l̂ S l l̂ S STA 6166 - MCP 28 Geometric Mean If the sample sizes are not equal in all groups, the value of n in the previous equations is replaced with the geometric mean of the sample sizes: 1 t nG exp ln( ni ) t i 1 E.g. Tukey’s procedure becomes: W = qa (t , nT - t ) s 2 w ( 1 nG ) = qa (t , nT - t ) STA 6166 - MCP MSE nG 29 Comparisonwise error rates for different MCP STA 6166 - MCP 30 Experimentwise error rates for different MCP STA 6166 - MCP 31