Statistical inference: CLT,
confidence intervals, p-values
Statistical Inference
The process of making
guesses about the truth from
Sample statistics
a sample.

n
x
̂  X n 
i 1
n
n
Truth (not
observable)
(x  X
i
ˆ 2  s 2 
n)
2
i 1
n 1
Sample
(observation)
*hat notation ^ is often used to indicate
“estitmate”
Population
parameters
N

x
i 1
N
N
(x  )
2
i
2 
i 1
N
Make guesses
about the whole
population
Statistics vs. Parameters

Sample Statistic – any summary measure calculated from
data; e.g., could be a mean, a difference in means or
proportions, an odds ratio, or a correlation coefficient



E.g., the mean vitamin D level in a sample of 100 men is 63 nmol/L
E.g., the correlation coefficient between vitamin D and cognitive
function in the sample of 100 men is 0.15
Population parameter – the true value/true effect in the
entire population of interest


E.g., the true mean vitamin D in all middle-aged and older European
men is 62 nmol/L
E.g., the true correlation between vitamin D and cognitive function in
all middle-aged and older European men is 0.15
Examples of Sample Statistics:
Single population mean
Single population proportion
Difference in means (ttest)
Difference in proportions (Z-test)
Odds ratio/risk ratio
Correlation coefficient
Regression coefficient
…
Example 1: cognitive function
and vitamin D


Hypothetical data loosely based on [1]; crosssectional study of 100 middle-aged and older
European men.
Estimation: What is the average serum vitamin D in
middle-aged and older European men?


Sample statistic: mean vitamin D levels
Hypothesis testing: Are vitamin D levels and cognitive
function correlated?

Sample statistic: correlation coefficient between vitamin D
and cognitive function, measured by the Digit Symbol
Substitution Test (DSST).
1. Lee DM, Tajar A, Ulubaev A, et al. Association between 25-hydroxyvitamin D levels and cognitive performance in middle-aged
and older European men. J Neurol Neurosurg Psychiatry. 2009 Jul;80(7):722-9.
Distribution of a trait: vitamin D
Right-skewed!
Mean= 63 nmol/L
Standard deviation = 33 nmol/L
Distribution of a trait: DSST
Normally distributed
Mean = 28 points
Standard deviation = 10 points
Distribution of a statistic…




Statistics follow distributions too…
But the distribution of a statistic is a theoretical
construct.
Statisticians ask a thought experiment: how much
would the value of the statistic fluctuate if one could
repeat a particular study over and over again with
different samples of the same size?
By answering this question, statisticians are able to
pinpoint exactly how much uncertainty is associated
with a given statistic.
Distribution of a statistic

Two approaches to determine the distribution
of a statistic:

1. Computer simulation



Repeat the experiment over and over again virtually!
More intuitive; can directly observe the behavior of
statistics.
2. Mathematical theory


Proofs and formulas!
More practical; use formulas to solve problems.
Example of computer
simulation…


How many heads come up in 100 coin
tosses?
Flip coins virtually



Flip a coin 100 times; count the number of
heads.
Repeat this over and over again a large
number of times (we’ll try 30,000 repeats!)
Plot the 30,000 results.
Coin tosses…
Conclusions:
We usually get
between 40 and 60
heads when we flip a
coin 100 times.
It’s extremely unlikely
that we will get 30
heads or 70 heads
(didn’t happen in
30,000 experiments!).
Distribution of the sample mean,
computer simulation…
1. Specify the underlying distribution of vitamin D
in all European men aged 40 to 79.








Right-skewed
Standard deviation = 33 nmol/L
True mean = 62 nmol/L (this is arbitrary; does not affect
the distribution)
2. Select a random sample of 100 virtual men from
the population.
3. Calculate the mean vitamin D for the sample.
4. Repeat steps (2) and (3) a large number of
times (say 1000 times).
5. Explore the distribution of the 1000 means.
Distribution of mean vitamin D
(a sample statistic)
Normally distributed! Surprise!
Mean= 62 nmol/L (the true
mean)
Standard deviation = 3.3 nmol/L
Distribution of mean vitamin D
(a sample statistic)



Normally distributed (even though the
trait is right-skewed!)
Mean = true mean
Standard deviation = 3.3 nmol/L


The standard deviation of a statistic is
called a standard error
s
The standard error of a mean =
n
If I increase the sample size to
n=400…
Standard error = 1.7 nmol/L
s
33

 1.7
n
400
If I increase the variability of
vitamin D (the trait) to SD=40…
Standard error = 4.0 nmol/L
s
40

 4.0
n
100
Mathematical Theory…
The Central Limit Theorem!
If all possible random samples, each of size n, are
taken from any population with a mean  and a
standard deviation , the sampling distribution of
the sample means (averages) will:
1. have mean:
x  
2. have standard deviation:

x 
n
3. be approximately normally distributed regardless of the shape
of the parent population (normality improves with larger n). It all
comes back to Z!
Symbol Check
x
x
The mean of the sample means.
The standard deviation of the sample means. Also
called “the standard error of the mean.”
Computer simulation of
the CLT:
1. Pick any probability distribution and specify a mean and
standard deviation.
2. Tell the computer to randomly generate 1000 observations from
that probability distributions
E.g., the computer is more likely to spit out values with high
probabilities
3. Plot the “observed” values in a histogram.
4. Next, tell the computer to randomly generate 1000 averagesof-2 (randomly pick 2 and take their average) from that
probability distribution. Plot “observed” averages in histograms.
5. Repeat for averages-of-10, and averages-of-100.
Uniform on [0,1]: average of 1
(original distribution)
Uniform: 1000 averages of 2
Uniform: 1000 averages of 5
Uniform: 1000 averages of 100
~Exp(1): average of 1
(original distribution)
~Exp(1): 1000 averages of 2
~Exp(1): 1000 averages of 5
~Exp(1): 1000 averages of 100
~Bin(40, .05): average of 1
(original distribution)
~Bin(40, .05): 1000 averages
of 2
~Bin(40, .05): 1000 averages
of 5
~Bin(40, .05): 1000 averages of
100
The Central Limit Theorem:
If all possible random samples, each of size n, are
taken from any population with a mean  and a
standard deviation , the sampling distribution of
the sample means (averages) will:
1. have mean:
x  
2. have standard deviation:

x 
n
3. be approximately normally distributed regardless of the shape
of the parent population (normality improves with larger n)
Central Limit Theorem caveats
for small samples:

For small samples:

The sample standard deviation is an imprecise estimate of
the true standard deviation (σ); this imprecision changes
the distribution to a T-distribution.


A t-distribution approaches a normal distribution for large n
(100), but has fatter tails for small n (<100)
If the underlying distribution is non-normal, the
distribution of the means may be non-normal.
Review Question 1
Standard error is:
a.
b.
c.
d.
e.
For a given variable, its standard deviation
divided by the square root of n.
A measure of the variability of a sample
statistic.
The inverse of sample size.
A measure of the variability of a
characteristic.
All of the above.
Examples of Sample Statistics:
Single population mean
Single population proportion
Difference in means (ttest)
Difference in proportions (Z-test)
Odds ratio/risk ratio
Correlation coefficient
Regression coefficient
…
Distribution of a correlation
coefficient?? Computer simulation…
1. Specify the true correlation coefficient






Correlation coefficient = 0.15
2. Select a random sample of 100 virtual
men from the population.
3. Calculate the correlation coefficient for
the sample.
4. Repeat steps (2) and (3) 15,000 times
5. Explore the distribution of the 15,000
correlation coefficients.
Distribution of a correlation
coefficient…
Normally distributed!
Mean = 0.15 (true correlation)
Standard error = 0.10
Distribution of a correlation
coefficient in general…

1. Shape of the distribution




Normally distributed for large samples
T-distribution for small samples (n<100)
2. Mean = true correlation coefficient
(r)
2
1 r
3. Standard error 
n
Many statistics follow normal
(or t-distributions)…

Means/difference in means



Proportions/difference in proportions
Regression coefficients


T-distribution for small samples
T-distribution for small samples
Natural log of the odds ratio
Estimation (confidence
intervals)…

What is a good estimate for the true
mean vitamin D in the population (the
population parameter)?

63 nmol/L +/- margin of error
95% confidence interval



Goal: capture the true effect (e.g., the
true mean) most of the time.
A 95% confidence interval should
include the true effect about 95% of
the time.
A 99% confidence interval should
include the true effect about 99% of
the time.
Recall: 68-95-99.7 rule for normal distributions! These is a 95%
chance that the sample mean will fall within two standard errors of
the true mean= 62 +/- 2*3.3 = 55.4 nmol/L to 68.6 nmol/L
Mean - 2 Std error=55.4
Mean
Mean + 2 Std error =68.6
To be precise, 95%
of observations fall
between Z=-1.96
and Z= +1.96 (so
the “2” is a rounded
number)…
95% confidence interval


There is a 95% chance that the sample mean
is between 55.4 nmol/L and 68.6 nmol/L
For every sample mean in this range, sample
mean +/- 2 standard errors will include the
true mean:

For example, if the sample mean is 68.6 nmol/L:


95% CI = 68.6 +/- 6.6 = 62.0 to 75.2
This interval just hits the true mean, 62.0.
95% confidence interval



Thus, for normally distributed statistics, the
formula for the 95% confidence interval is:
sample statistic  2 x (standard error)
Examples:

95% CI for mean vitamin D:


63 nmol/L  2 x (3.3) = 56.4 – 69.6 nmol/L
95% CI for the correlation coefficient:

0.15  2 x (0.1) = -.05 – .35
Simulation of 20 studies of
100 men…
Vertical line indicates the true mean (62)
95% confidence
intervals for the mean
vitamin D for each of the
simulated studies.
Only 1 confidence
interval missed the true
mean.
Confidence Intervals give:
*A plausible range of values for a population
parameter.
*The precision of an estimate.(When
sampling variability is high, the confidence
interval will be wide to reflect the uncertainty
of the observation.)
*Statistical significance (if the 95% CI does
not cross the null value, it is significant at
.05)
Confidence Intervals
The value of the statistic in my sample
(eg., mean, odds ratio, etc.)
point estimate  (measure of how confident
we want to be)  (standard error)
From a Z table or a T table, depending
on the sampling distribution of the
statistic.
Standard error of the statistic.
Common “Z” levels of confidence

Commonly used confidence levels are
90%, 95%, and 99%
Confidence
Level
80%
90%
95%
98%
99%
99.8%
99.9%
Z value
1.28
1.645
1.96
2.33
2.58
3.08
3.27
99% confidence intervals…

99% CI for mean vitamin D:


63 nmol/L  2.6 x (3.3) = 54.4 – 71.6 nmol/L
99% CI for the correlation coefficient:

0.15  2.6 x (0.1) = -.11 – .41
Testing Hypotheses


1. Is the mean vitamin D in middleaged and older European men lower
than 100 nmol/L (the “desirable” level)?
2. Is cognitive function correlated with
vitamin D?
Is the mean vitamin D
different than 100?




Start by assuming that the mean = 100
This is the “null hypothesis”
This is usually the “straw man” that we
want to shoot down
Determine the distribution of statistics
assuming that the null is true…
Computer simulation (10,000
repeats)…
This is called the null
distribution!
Normally distributed
Std error = 3.3
Mean = 100
Compare the null distribution
to the observed value…
What’s the
probability of
seeing a sample
mean of 63 nmol/L
if the true mean is
100 nmol/L?
It didn’t happen in
10,000 simulated
studies. So the
probability is less
than 1/10,000
Compare the null distribution
to the observed value…
This is the p-value!
P-value < 1/10,000
Calculating the p-value with a
formula…
Because we know how normal curves work, we can exactly calculate
the probability of seeing an average of 63 nmol/L if the true average
weight is 100 (i.e., if our null hypothesis is true):
63  100
Z
 11.2
3.3
Z= 11.2, P-value << .0001
The P-value
P-value is the probability that we would have seen our
data (or something more unexpected) just by chance if
the null hypothesis (null value) is true.
Small p-values mean the null value is unlikely given
our data.
Our data are so unlikely given the null hypothesis
(<<1/10,000) that I’m going to reject the null
hypothesis! (Don’t want to reject our data!)
P-value<.0001 means:
The probability of seeing what you saw or something
more extreme if the null hypothesis is true (due to
chance)<.0001
P(empirical data/null hypothesis) <.0001
The P-value


By convention, p-values of <.05 are often
accepted as “statistically significant” in the
medical literature; but this is an arbitrary cutoff.
A cut-off of p<.05 means that in about 5 of
100 experiments, a result would appear
significant just by chance (“Type I error”).
Summary: Hypothesis
Testing
The Steps:
1. Define your hypotheses (null, alternative)
2. Specify your null distribution
3. Do an experiment
4. Calculate the p-value of what you observed
5. Reject or fail to reject (~accept) the null
hypothesis
Hypothesis Testing
The Steps:
1. Define your hypotheses (null, alternative)

The null hypothesis is the “straw man” that we are trying to shoot down.

Null here: “mean vitamin D level = 100 nmol/L”

Alternative here: “mean vit D < 100 nmol/L” (one-sided)
2. Specify your sampling distribution (under the null)

If we repeated this experiment many, many times, the mean vitamin D
would be normally distributed around 100 nmol/L with a standard error
33
 3.3
of 3.3
100
3. Do a single experiment (observed sample mean = 63 nmol/L)
4. Calculate the p-value of what you observed (p<.0001)
5. Reject or fail to reject the null hypothesis (reject)

Confidence intervals give the same
information (and more) than hypothesis
tests…
Duality with hypothesis tests.
Null value
95% confidence interval
50
60
70
80
90
Null hypothesis: Average vitamin D is 100 nmol/L
Alternative hypothesis: Average vitamin D is not 100
nmol/L (two-sided)
P-value < .05
100
Duality with hypothesis tests.
Null value
99% confidence interval
50
60
70
80
90
Null hypothesis: Average vitamin D is 100 nmol/L
Alternative hypothesis: Average vitamin D is not 100
nmol/L (two-sided)
P-value < .01
100
Summary: Single population
mean (large n)

Hypothesis test:
observed mean  null mean
Z
s
n

Confidence Interval
s
confidence interval  observed mean  Z/2 * ( )
n
Single population mean (small
n, normally distributed trait)

Hypothesis test:
observed mean  null mean
Tn 1 
s
n

Confidence Interval
s
confidence interval  observed mean  Tn 1,/2 * ( )
n
2. Is cognitive function correlated
with vitamin D?


Null hypothesis: r = 0
Alternative hypothesis: r  0


Two-sided hypothesis
Doesn’t assume that the correlation will be
positive or negative.
Computer simulation (15,000
repeats)…
Null distribution:
Normally distributed
Std error = 0.1
Mean = 0
What’s the probability of our
data?
Even when the true
correlation is 0, we get
correlations as big as 0.15
or bigger 7% of the time.
What’s the probability of our
data?
This is a two-sided hypothesis
test, so “more extreme”
includes as big or bigger
negative correlations (<-0.15).
P-value = 7% + 7% = 14%
What’s the probability of our
data?
Our results could have
happened purely due to a
fluke of chance!
Formal hypothesis test
1. Null hypothesis: r=0


Alternative: r  0 (two-sided)
2. Determine the null distribution



3. Collect Data, r=0.15
4. Calculate the p-value for the data:




Normally distributed
Standard error = 0.1
Z=
0.15  0
 1.5
.1
Z of 1.5 corresponds to a
two-sided p-value of 14%
5. Reject or fail to reject the null (fail to reject)
Or use confidence interval to
gauge statistical significance…



95% CI = -0.05 to 0.35
Thus, 0 (the null value) is a plausible
value!
P>.05
Summary: correlation
coefficient (large n)

Hypothesis test:
Z

r -0
1 r2
n
Confidence Interval
1 r 2
confidence interval  observed r  Z/2 * (
)
n
Correlation coefficient (small
n)

Hypothesis test:
Tn  2 

r 0
1 r 2
n2
Confidence Interval
1 r 2
confidence interval  observed r  Tn 2,/2 * (
)
n2
Examples of Sample Statistics:
Single population mean
Single population proportion
Difference in means (ttest)
Difference in proportions (Z-test)
Odds ratio/risk ratio
Correlation coefficient
Regression coefficient
…
Example 2: HIV vaccine trial


Thai HIV vaccine trial (2009)
 8197 randomized to vaccine
 8198 randomized to placebo
Generated a lot of public discussion about pvalues!
51/8197 vs. 75/8198
=23 excess infections in the
placebo group.
=2.8 fewer infections per 1000
people vaccinated
Source: BBC news, http://news.bbc.co.uk/go/pr/fr/-/2/hi/health/8272113.stm
Null hypothesis


Null hypothesis: infection rate is the
same in the two groups
Alternative hypothesis: infection rates
differ
Computer simulation assuming
the null (15,000 repeats)…
Normally distributed,
standard error = 11.1
Computer simulation assuming
the null (15,000 repeats)…
If the vaccine is
completely
ineffective, we
could still get 23
excess infections
just by chance.
Probability of 23
or more excess
infections = 0.04
How to interpret p=.04…



P(data/null) = .04
P(null/data) .04
P(null/data)  22%
*estimated using Bayes’ Rule (and
prior data on the vaccine)
*Gilbert PB, Berger JO, Stablein D, Becker S, Essex M, Hammer SM, Kim JH, DeGruttola VG. Statistical
interpretation of the RV144 HIV vaccine efficacy trial in Thailand: a case study for statistical issues in efficacy
trials. J Infect Dis 2011; 203: 969-975.
Alternative analysis of the
data (“intention to treat”)…

56/8202 (6.8 per 1000) infections in the
vaccine group versus 76/8200 (9.3 per
1000)
Computer simulation assuming
the null (15,000 repeats)…
Probability of 20
or more excess
infections = 0.08
P=.08 is only slightly
different than p=.04!
Confidence intervals…

95% CI (analysis 1): .0014 to .0055

95% CI (analysis 2): -.0003 to .0051

The plausible ranges are nearly
identical!
Review Question 2
A randomized trial of two treatments for depression
failed to show a statistically significant difference in
improvement from depressive symptoms (p-value
=.50). It follows that:
a.
b.
c.
d.
e.
The treatments are equally effective.
Neither treatment is effective.
The study lacked sufficient power to detect a
difference.
The null hypothesis should be rejected.
There is not enough evidence to reject the null
hypothesis.
Review Question 3
Following the introduction of a new treatment regime in a
rehab facility, alcoholism “cure” rates increased. The
proportion of successful outcomes in the two years
following the change was significantly higher than in the
preceding two years (p-value: <.005). It follows that:
a.
b.
c.
d.
e.
The improvement in treatment outcome is clinically important.
The new regime cannot be worse than the old treatment.
Assuming that there are no biases in the study method, the new
treatment should be recommended in preference to the old.
All of the above.
None of the above.
Homework




Problem Set 4
Coursework articles: p-values and
confidence intervals
Reading: Vickers 16-17
Journal article/article review sheet