What is the difference between an alpha level and a p-value?
Science sets a conservative standard to meet for a researcher to claim
that s/he has made a discovery of a real phenomenon. The standard is the
alpha level, usually set of .05.
Assuming that the null hypothesis is true, this means we may reject the
null only if the observed data are so unusual that they would have
occurred by chance at most 5 % of the time. The smaller the alpha, the
more stringent the test (the more unlikely it is to find a statistically
significant result).
Once the alpha level has been set, a statistic (like r) is computed. Each
statistic has an associated probability value called a p-value, or the
likelihood of an observed statistic occurring due to chance, given the
sampling distribution.
Alpha sets the standard for how extreme the data must be before we can
reject the null hypothesis. The p-value indicates how extreme the data
are. We compare the p-value with the alpha to determine whether the
observed data are statistically significantly different from the null
hypothesis:
If the p-value is less than or equal to the alpha (p< .05), then we reject
the null hypothesis, and we say the result is statistically significant.
If the p-value is greater than alpha (p > .05), then we fail to reject the
null hypothesis, and we say that the result is statistically nonsignificant
(n.s.).