AP Statistics
Notes
Name: ____________
Date: _____________
Lesson 11.3C: Statistical Significance vs.
Practical Significance
Big Idea
How can we assess the truth of a claim about a population parameter using
information gained in a simple random sample from that population?
Objectives: U
I. Statistical Significance vs. Practical Significance



The purpose of a test of hypothesis is to give a statement of the amount of
evidence provided by the sample against the null hypothesis. The P-value
provides this degree of evidence.
If you must make a decision based on your inference, choose a level of  in
advance as your standard. Then determine if your evidence reaches this
standard (i.e. P- value <  ).
If you do use a fixed  level, choose it by asking how much evidence is required
to reject the null hypothesis. Things to consider:
– How plausible is H 0 ?
If the value of H 0 has been accepted as true for generations, you will
need strong evidence (small  ) in order to reject it.
– What are the consequences of rejecting H 0 ?
If rejecting H 0 will require major dollar and/or time commitments, you
will need strong evidence (small  ) in order to reject it.
Note #1:
There is no sharp border between “significant” and “insignificant,”
only increasingly strong evidence as the P-value decreases.
Example: Suppose that in the absence of special preparation, Scholastic
Assessment Test mathematics (SATM) scores vary normally with mean
 = 475 and  = 100. One hundred students go through a rigorous
training program designed to raise their SATM scores by improving their
mathematical skills. Carry out a test of hypothesis using:
H0:
Ha :
a. The students’ average score is x = 491.4.
Is this result significant at the 5% level?
b. The students’ average score is x = 491.5.
Is this result significant at the 5% level?
c. Are the differences between these two averages important?
Note#2:
When large samples are drawn, even small deviations from the null
hypothesis will be significant.
Example: Suppose that SATM scores in the absence of coaching vary normally
with mean  = 475 and  = 100. Suppose also that coaching may
change  but does not change  . An increase in the SATM score from
475 to 478 is of no importance in seeking admission to college, but this
unimportant change can be statistically very significant. To see this,
calculate the P-value for the test of
H0:
Ha :
a. A coaching service coaches 100 students. Their SATM scores
average x = 478.
b. By the next year, the service has coached 1000 students. Their SATM
scores average x = 478.
c. An advertising campaign brings the number of students coached to
10,000 students. Their average SATM score is still x = 478.
Note #3:
Tests of hypotheses are overused.
A test of hypothesis only tells if an effect is too large to occur by
chance alone.
A confidence interval estimates the size of the effect.
Confidence intervals are not used enough.
Example: Give a 99% confidence interval for the mean SATM score  after
coaching in each part of the previous example. For large samples, the
confidence interval tells us, ”Yes, the mean score is higher than 475 after
coaching, but only by a small amount.”
II. Cautions “Fancy formulas can not rescue badly produced data!”
– Data must be a SRS, nothing more complex like a multistage or stratified sample.
– If the data is not randomly drawn, then no method of inference is correct.
– Examine your data for extreme outliers. They strongly influence x , and hence any
test of hypothesis based on x . If possible, remove extreme observations.
– Examine your data for skewness and other signs of nonnormality, particularly if your
sample is small. If possible, draw a large sample.
– Testing a hypothesis on the same data that suggested the hypothesis can invalidate
a test. Decide what effect you are seeking, design a study to search for it, and then
use a test of hypothesis to weigh the evidence that you get.
– Do not test many hypotheses at the same time by doing multiple tests of
hypotheses. Some significant results will occur by chance alone, even if all the null
hypotheses are true. (See cell phone example on page 720.)