Limitations of Hypothesis Tests
1.
Do not confuse Statistical significance with Practical significance. When the sample
size gets really big, even very small differences in the mean or proportion will be
statistically significant.
Ex. Even though the population mean SAT math score may be greater than 400, if
it is 400.7 does that really matter?
2.
A small p-value means that if Ho is true then the observed sample is highly unlikely
to have occurred. It does not necessarily mean that the actual mean or proportion
is far from μo or po.
3.
When our p-value is not small and we do not reject Ho, that does not mean we
have evidence that Ho is true. This is why you always need to state the claim in the
words of the problem and when we do not reject Ho, we say “there is not sufficient
evidence to support the claim,” and not “there is evidence to support Ho.”
4.
When we started hypothesis testing I told you that you although technically it is not
correct, you can think of the p-value as the “probability that Ho is true.” It would
have been more accurate to state:
When the p-value is small then there is evidence that Ho is not true.
When the p-value is large then there is no evidence that Ho is not true.
5.
Be wary of articles that just report statistically significant results. If they are using a
5% significance level, then 1 out of 20 statistical results will be significant by
chance.
6.
Confidence Intervals are becoming more popular than Hypothesis Tests, because
they convey more information. You can see when a mean or proportion is
statistically different from a value (μo or po) but you can also see how far or close
the interval is from the hypothesized value.
Ex. Two researchers Al and Bob conduct separate studies to test:
Ho: p = .5
Ha: p ≠ .5 (Claim)
Each sample has n = 400. Al gets 220 successes and Bob gets 219 successes. Run the
hypothesis tests for each researcher at 5% significance level, be sure to report the test
statistics p-values and whether or not you accept or reject Ho.
Al:
p-hat = .550
TS = 2.00
Reject Ho., support claim
p-value = .046
Bob:
p-hat = .5475
TS = 1.90
p-value = .057
Accept Ho., do not support claim
Even though the difference in the p-values is .011, Al would support the claim and Bob
would not. If the test statistics and p-values are not reported, this can be very
misleading.
Find 95% confidence intervals for the population proportion for Al and Bob.
Al 95% CI for p: (.501, .599)
Bob 95% CI for p: (.499, .596)
First note that .5 is not included in Al’s interval, so we are 95% sure that p is not equal
to .5.
Also notice that the intervals are incredibly close to one another.
Practically there may be no difference for these samples, but that will depend on the
problem.
Ex. A survey is taken to determine one’s love of Math. The scale goes from 0 to 10 with
0 meaning no love of Math and 10 meaning adoration of Math. A researcher claims that
the average score for the population is more than 3. A sample of 10000 people is taken
and the average is 3.1 with a standard deviation of 2.3. Test this claim at 5%
significance level. Find a 95% Confidence Interval for the population mean math love
score.
Ho: μ ≤ 3
Ha: μ > 3
TS = 4
6.9 * 10 -6 = .0000069
Reject Ho. There is sufficient evidence to support the claim that the population mean is
greater than 3.
95% CI for μ: (3.055, 3.145)
Is there a practical difference between 3 and 3.05?