Statistical Tests
• How to tell if something (or somethings)
is different from something else
Populations vs. Samples
• Remember that a population is all the
possible members of a category that we
could measure
– Examples:
• the heights of every male or every female
• the temperature on every day since the
beginning of time
• Ever person who ever has, and ever will, take a
particular drug
Populations vs. Samples
• So a population is kind of abstract typically you couldn’t ever hope to
measure the entire population
– Notable exceptions include:
• Standardized tests (mean IQ is 100 with std.
dev. of 15)
• Special populations such as rare diseases or
isolated groups of people
Populations vs. Samples
• A sample is some subset of a
population
– Examples:
• The heights of 10 students picked at random
• The participants in a drug trial
Populations vs. Samples
• The notation
– Sample statistics are usually regular letters
like s and X
– Population statistics are usually greek
letters like:
 - the population mean
 - the population standard deviation
Populations vs. Samples
• Test your intuition:
– Under what circumstances does the mean
of a sample equal the mean of the
population from which it was drawn?
– What about the standard deviation?
– What if your sample was very small relative
to the population?
Populations vs. Samples
• Test your intuition:
– Most importantly: What if you took more
than one sample
Central Limit Theorem
•
There is a distribution of sample
means
Central Limit Theorem
•
There is a distribution of sample
means
The population of IQ scores:
100
Central Limit Theorem
•
There is a distribution of sample
means
The population of IQ scores:

100
Your
Sample
x = 95
Central Limit Theorem
•
There is a distribution of sample
means
The population of IQ scores:

100
Your
Sample
x = 103
Central Limit Theorem
•
There is a distribution of sample
means
The population of IQ scores:

100
Your
Sample
x = 99
Central Limit Theorem
•
•
There is a distribution of sample
means
This is the sampling distribution of the
mean
Central Limit Theorem
•
What is the mean of the sampling
distribution of the mean?
– mean of the sampling distribution
approaches the mean of the population
with many resamplings
x  u
Central Limit Theorem
•
What is the standard deviation of the
sampling distribution of the mean?
– The standard error of the mean
X 

n
Notice it will always be less than the
standard deviation of the population!
Central Limit Theorem
•
What is the shape of the sampling
distribution of the mean?
– Central Limit Theorem: the sampling
distribution of the mean is normal
regardless of the shape of the underlying
distribution !
– This means you can use the Z transform
and use the Z table
The Logic of Statistical Tests
Statistical Tests
• Consider a simple example:
– you are testing the hypothesis that eating
walnuts makes people smarter by feeding
walnuts to a group of 30 subjects and then
testing their IQ
Statistical Tests
• Consider a simple example:
– you are testing the hypothesis that eating
walnuts makes people smarter by feeding
walnuts to a group of 30 subjects and then
testing their IQ
– If you are right, then eating walnuts will
make the average IQ of your subjects be
higher than the average IQ of all people
(the population) since, mostly, those other
people don’t eat walnuts much
Statistical Tests
• Consider a simple example:
– Put another way:
• Is this sample (entirely) of walnut eaters
different from the population of mostly nonwalnut-eaters
Types of Errors
• There are two “mistakes” you could
make:
Types of Errors
• There are two “mistakes” you could
make:
– Type I error or False-Positive - you decide
the walnut treatment works when it doesn’t
really
– Type II error or False-Negative - you
decide the walnuts don’t work when really
they do
Types of Successes
• There are two ways to succeed:
– Hit or True-Positive: You decide the
walnuts do make people smarter and, in
fact, they really do
– Correct-Rejection or True-Negative: You
decide the walnuts don’t work and, in fact
they really don’t
Outcome Matrix
Actual Situation
Works
“Works”
Doesn’t
Work
True Positive
Type I
Your
Conclusion
“Doesn’t
Work”
Type II
TrueNegative
Statistical Tests
• Consider a simple example:
– Your subjects turn out to have a mean IQ
of 107.5 (1/2 S.D. from the mean of the
population) after eating walnuts
Statistical Tests
•
What are two reasons why the mean
IQ of your subjects might be greater
than the mean of the population?
1. you happened to pick 30 very smart
people (i.e. university students)
– WARNING: Type I error is possible!
Statistical Tests
•
What are two reasons why the mean
IQ of your subjects might be greater
than the mean of the population?
1. you happened to pick 30 very smart
people (i.e. university students)
– WARNING: Type I error is possible!
2. the walnuts worked
Statistical Tests
•
Usually we are worried about making
a type I error so we need to know:
– What fraction of all possible groups of 30
subjects would have a mean IQ of 105 or
less?
Statistical Tests
•
Usually we are worried about making
a type I error so we need to know:
– What fraction of all possible groups of 30
subjects would have a mean IQ of 105 or
less?
•
In other words, we are interested not in the
distribution of IQ scores themselves, but
rather in the distribution of mean IQ scores
for groups of 30 subjects
The Z Test
…as it is more formally known
Example Z Test
• Using our example in which we are
testing the hypothesis that walnuts make
people smarter
• null hypothesis is that they don’t
X = 107.5
 = 100
 = 15
Example Z Test
• Using our example in which we are
testing the hypothesis that walnuts make
people smarter (null hypothesis was that
they don’t)
X = 107.5
  100
 = 15
• We wantto know how many standard
errors from the mean (of the sampling
distribution of means) is 107.5
Example Z Test
Here’s what we’ve got:
X = 107.5   ux 100  = 15
n = 30
Here’s what we can compute:

Zx 
x  x
x
X 

n
That’s what we’re after so that we can use the Z
table
Example Z Test
Here’s what we’ve got:
X = 107.5   ux 100  = 15
n = 30
Here’s what we can compute:


15
X 

 2.739
n
30
Which is much less than 15!
Example Z Test
Here’s what we’ve got:
X = 107.5   ux 100  = 15
n = 30
 X  2.739
Here’s what we can compute:

Zx 

x  x
x
107.5 100

 2.738
2.739
Example Z Test
Here’s what we’ve got:
X = 107.5   ux 100  = 15
n = 30
 X  2.739
Zx  2.738
ThusX = 107.5 isn’t half a standard deviation

from the sampling distribution mean!
It’s actually more than two and a half standard deviations
from the sampling distribution mean !
Example Z Test
Here’s what we’ve got:
X = 107.5   ux 100  = 15
n = 30
 X  2.739
Zx  2.738


Looking up 2.739 in the Z table reveals that only .0031 or .31%
of the means in the sampling distribution of mean IQs (for
groups of 30 people each) would have a mean equal to or
greater than 107.5!
Example Z Test
• What this means is that you have only a
0.31% chance of making a type I error if
you conclude that walnuts made your
subjects smarter !
Example Z Test
• What this means is that you have only a
0.31% chance of making a type I error if
you conclude that walnuts made your
subjects smarter !
• Put another way, there is only a 0.31%
chance that this sample of IQs is taken
from the regular population…walnut
eaters are different
Alpha
• Is .31% small enough? What risk of
making a Type I error is too great?
Alpha
• Is .31% small enough? What risk of
making a Type I error is too great?
• There is no absolute answer - it
depends entirely on the circumstances
Alpha
• Is .31% small enough? What risk of
making a Type I error is too great?
• There is no absolute answer - it
depends entirely on the circumstances
• 5% or probability (p) = .05 is generally
accepted
Alpha
• Is .31% small enough? What risk of
making a Type I error is too great?
• There is no absolute answer - it
depends entirely on the circumstances
• 5% or probability (p) = .05 is generally
accepted
• This rate of making Type I errors (ie.
number of Type I errors per 100
experiments) is called the Alpha Level
Statistical Significance
• So we conclude that walnuts have a
statistically significant effect on IQ with
a probability of a Type I error of less
than 5%
– In a research article we might say “the
effect of walnuts on IQ was significant
(one-tailed Z test, p = .0031)”