Limitations of Hypothesis Tests 1. Do not confuse Statistical significance with Practical significance. When the sample size gets really big, even very small differences in the mean or proportion will be statistically significant. Ex. Even though the population mean SAT math score may be greater than 400, if it is 400.7 does that really matter? 2. A small p-value means that if Ho is true then the observed sample is highly unlikely to have occurred. It does not necessarily mean that the actual mean or proportion is far from μo or po. 3. When our p-value is not small and we do not reject Ho, that does not mean we have evidence that Ho is true. This is why you always need to state the claim in the words of the problem and when we do not reject Ho, we say “there is not sufficient evidence to support the claim,” and not “there is evidence to support Ho.” 4. When we started hypothesis testing I told you that you although technically it is not correct, you can think of the p-value as the “probability that Ho is true.” It would have been more accurate to state: When the p-value is small then there is evidence that Ho is not true. When the p-value is large then there is no evidence that Ho is not true. 5. Be wary of articles that just report statistically significant results. If they are using a 5% significance level, then 1 out of 20 statistical results will be significant by chance. 6. Confidence Intervals are becoming more popular than Hypothesis Tests, because they convey more information. You can see when a mean or proportion is statistically different from a value (μo or po) but you can also see how far or close the interval is from the hypothesized value. Ex. Two researchers Al and Bob conduct separate studies to test: Ho: p = .5 Ha: p ≠ .5 (Claim) Each sample has n = 400. Al gets 220 successes and Bob gets 219 successes. Run the hypothesis tests for each researcher at 5% significance level, be sure to report the test statistics p-values and whether or not you accept or reject Ho. Al: p-hat = .550 TS = 2.00 Reject Ho., support claim p-value = .046 Bob: p-hat = .5475 TS = 1.90 p-value = .057 Accept Ho., do not support claim Even though the difference in the p-values is .011, Al would support the claim and Bob would not. If the test statistics and p-values are not reported, this can be very misleading. Find 95% confidence intervals for the population proportion for Al and Bob. Al 95% CI for p: (.501, .599) Bob 95% CI for p: (.499, .596) First note that .5 is not included in Al’s interval, so we are 95% sure that p is not equal to .5. Also notice that the intervals are incredibly close to one another. Practically there may be no difference for these samples, but that will depend on the problem. Ex. A survey is taken to determine one’s love of Math. The scale goes from 0 to 10 with 0 meaning no love of Math and 10 meaning adoration of Math. A researcher claims that the average score for the population is more than 3. A sample of 10000 people is taken and the average is 3.1 with a standard deviation of 2.3. Test this claim at 5% significance level. Find a 95% Confidence Interval for the population mean math love score. Ho: μ ≤ 3 Ha: μ > 3 TS = 4 6.9 * 10 -6 = .0000069 Reject Ho. There is sufficient evidence to support the claim that the population mean is greater than 3. 95% CI for μ: (3.055, 3.145) Is there a practical difference between 3 and 3.05?