Section IV
Sampling distributions
Confidence intervals
Hypothesis testing and p values
1
Population and sample
We wish to make inferences
(generalizations) about an entire target
population (ie, generalize to “everyone”)
even though we only study one sample
(have only one study).
Population parameters=summary values for
the entire population (ex: μ,σ,ρ,β )
Sample statistics=summary values for a
sample (ex: Y, S, r, b)
2
Samples drawn from a population
Population
Sample is drawn “at random”.
Everyone in the target population
is eligible for sampling.
sample
3
True population distribution of Y
(individuals)- not Gaussian
Original distribution of Y-individuals
30%
25%
20%
15%
10%
5%
0%
1
2
3
4
Y
Mean Y=μ= 2.5,
SD=σ=1.12
4
Possible samples & statistics from
the population (true mean=2.5)
sample (n=4)
(statistic)
1,1,1,1
…
2,2,4,3
…
4,4,4,4
mean
1.00
2.75
4.00
5
Distribution of the sample means (Ys)
- Sampling distributioneach observation is a SAMPLE statistic
sampling distribution
50
frequency
40
30
20
10
0
1.00 1.25 1.50 1.75 2.00 2.25 2.50 2.75 3.00 3.25 3.50 3.75 4.00
__
Y
Mean Y = 2.5, SEM = 0.56, n=4
SEM = SD/n
the square root n law
6
Central Limit Theorem
For a large enough n, the distribution of any
sample statistic (mean, mean difference,
OR, RR, hazard, correlation coeff,regr coeff,
proportion…) from sample to sample has a
Gaussian (“Normal”) distribution centered at
the true population value.
The standard error is proportional to 1/√n.
(Rule of thumb: n> 30 is usually enough. May
need non parametric methods for small n)
7
8
Funnel plot - true difference is δ= 5
Each point is one study (meta analysis)
400
350
sample size (n)
300
250
200
150
100
50
0
-15.0
-10.0
-5.0
0.0
5.0
10.0
15.0
20.0
25.0
sample mean difference
9
Resampling estimation
(“bootstrap”)
One does not repeatedly sample from the
same population, (one only carries out the
study once). But a “simulation” of repeated
sampling from the population can be obtained
by repeatedly sampling from the sample with
replacement & computing the statistic from
each resample, creating an “estimated”
sampling distribution. The SD of the statistics
across all “resamples” is an estimate the
standard error (SE) for the statistic.
10
Samples drawn from a population
sample
Population
sample
Original sample
sample
Sample is drawn “at random” with
replacement. Everyone in the original
sample is eligible for sampling.
sample
11
Confidence interval (for μ)
We do not know μ from a sample.
For a sample mean Y and standard error
SE, a confidence interval for the
population mean μ is formed by
Y - Z SE, Y + Z SE
(sample statistic is in the middle)
For a 95% confidence interval, we use
Z=1.96 (Why?) and compute
Y – 1.96 SE, Y + 1.96 SE
lower
mean
upper
12
Confidence Intervals (CI)
and sampling dist of Y
Sampling
Distribution
2.5%
2.5%
-1.96(/n)

1.96(/n)
95% CI: Y  1.96 (/n)
13
95% Confidence
intervals
95% of the
intervals will
contain the true
population value
But which ones?
14
Z vs t (technical note)
Confidence intervals made with Z assume
that the population σ is known. Since σ is
usually not known and is estimated with
the sample SD, the Gaussian table areas
need to be adjusted. The adjusted tables
are called “t” tables instead of Gaussian
tables (t distribution). For n > 30, they are
about the same.
15
Z distribution vs t distribution, about the same for n > 30
16
t vs Gaussian Z percentiles
%ile
85th
90th
95th
97.5th
99.5th
Confidence
70%
80%
90%
95%
99%
t, n=5
1.156
1.476
2.015
2.571
4.032
t, n=10
1.093
1.372
1.812
2.228
3.169
t, n=20
1.064
1.325
1.725
2.086
2.845
t, n=30
1.055
1.310
1.697
2.042
2.750
Gaussian 1.036
1.282
1.645
1.960
2.576
What did the z distribution say to the t distribution?
You may look like me but you're not normal.
17
Confidence Intervals
Sample Statistic ± Ztabled SE
(using known variance)
Sample Statistic ± ttabled SE
(using estimate of variance)
Example: CI for the difference between two means:
__ __
(Y1 – Y2) ± ttabled (SEd)
Tabled t uses degrees of freedom, df=n1+n2-2
18
CI for a proportion
“law” of small numbers
n=10, Proportion = 3/10 = 30%
What do you think are the 95% confidence
bounds?
Is is likely that the “real” proportion is more
than 50%?
19
CI for a proportion
“law” of small numbers
n=10, Proportion = 3/10 = 30%
What do you think are the 95% confidence
bounds?
Is is likely that the “real” proportion is more
than 50%?
Answer: 95% CI: 6.7% to 65.3%
20
Standard error for the difference
between two means
__
Y1 has mean μ1 and SE = √σ12/n1 = SE1
__
Y2 has mean μ2 and SE = √σ22/n2 = SE2
SEd
SE2
For the difference between two means
(δ=1 - 2)
SEδ =
√(σ12/n1
+ σ2
2/n
SEd = (SE12 + SE22)
SE1
2)
SEd is computed from SE1 and SE2
using “Pythagoras’ rule”.
SEd2 = SE12 + SE22
21
Statistics for HBA1c change
from base to 26 weeks (Pratley et al, Lancet 2010)
Tx
n
Mean
SD
SE
Liraglutide
225
-1.24
0.99
0.066
Sitaglipin
219
-0.90
0.98
0.066
__
Mean difference = d = 0.34 %
Std error of mean difference= SEd=[0.0662 + 0.0662] = 0.093%
Using t{df=442}=1.97 for the 95% confidence interval:
CI: 0.34% ± 1.97 (0.093%) or (0.16%, 0.52%)
22
Null hypothesis & p values
Null Hypothesis- Assume that, in the population,
the two treatments give the same average
improvement in HbA1c. So the average
difference is δ=0.
Under this assumption, how likely is it to observe
a sample mean difference of d= 0.34% (or more
extreme) in any study? This probability is called
the (one sided) p value.
The p value is only defined for a given null
hypothesis.
23
Hypothesis testing
for a mean difference, d
d =sample mean HBA1c chg difference,
_
d = 0.34%, SEd = 0.093%
95% CI for true mean difference = (0.16%, 0.52%)
But, under the null hypothesis, the true mean difference (δ) should be zero.
How “far” is the observed 0.34% mean difference from zero (in SE
units)?
tobs = (mean difference – hypothesized difference) / SEdiff
tobs = (0.34 – 0) / 0.093 = 3.82 SEs
p value: probability of observing t=3.82 or larger if null hypothesis is true.
p value = 0.00008 (one sided t with df=442)
p value = 0.00016 (two sided)
24
Hypothesis test statistics
Zobs = (Sample Statistic – null value) / Standard error
Z (or t)=3.82
p value
-3.0
-2.0
-1.0
0.0
1.0
2.0
3.0
z
25
26
Difference & Non inferiority
(equivalence) hypothesis testing
Difference Testing:
Null Hyp: A=B (or A-B=0), Alternative: A≠B
Zobs = (observed stat – 0) / SE
Non inferiority (within δ) Testing:
Null Hyp: A > B + δ, Alternative: A <= B + δ
Zeq = (observed stat – δ )/ SE
Must specify δ for non inferiority testing
27
Non inferiority testing-HBA1c data
For HBA1c data, assume we declare non inferiority if
the true mean difference is δ=0.40% or less. The
observed mean difference is d=0.34%, which is
smaller than 0.40%. However, the null hypothesis
is that the true difference is 0.40% or more versus
the alternative of 0.40% or less. So
Zeq=(0.34 –0.40)/0.093=-0.643, p=0.260 (one sided)
We cannot reject the “null hyp” that the true δ is larger
than 0.40%. Our 95% confidence interval of
(0.16%, 0.52%) also does NOT exclude 0.40%,
even though it excludes zero.
28
Confidence intervals
versus hypothesis testing
Study
(1-8)
equivalence demonstrated only from –D tp +D
(brackets show 95% confidence intervals)
Stat
Sig
1. Yes ----------------------------------------------------------------------------------------------- < not equivalent >
2. Yes -----------------------------------------------------------------------------< uncertain >-------------------3. Yes ------------------------------------------------------------------< equivalent >----------------------------------4. No ---------------------------------------------------< equivalent >--------------------------------------------------5. Yes ----------------------------------< equivalent > ---------------------------------------------------------------6. Yes ---------------------< uncertain>---------------------------------------------------------------------------------7. Yes -< not equivalent >----------------------------------------------------------------------------------------------8. No ---------<___________________________uncertain________________________________>-----|
-D
O
true difference
|
+D
Ref: Statistics Applied to Clinical Trials- Cleophas, Zwinderman, Cleopahas 2000
Kluwer Academic Pub Page 35
29
Non inferiority
JAMA 2006 - Piaggio et al, p 1152-1160
30
Paired Mean Comparisons
Serum cholesterol in mmol/L
Difference between baseline and end of 4 weeks
Subject
1
2
3
4
5
6
mean
SD
SE
chol(baseline)
9.0
7.1
6.9
6.9
5.9
5.4
6.87
1.24
0.51
chol(4 wks)
6.5
6.3
5.9
4.9
4.0
4.9
5.42
0.97
0.40
difference(di)
2.5
0.8
1.0
2.0
1.9
0.5
1.45
0.79
0.32
_
Difference (baseline – 4 weeks) = amount lowered: d = 1.45 mmol/L
SD = 0.79 mmol/L SEd = 0.79/6 = 0.323 mmol/L, df = 6-1=5, t0.975 = 2.571
95% CI: 1.45 ± 2.571 (0.323) = 1.45 ± 0.830 or (0.62 mmol/L, 2.28 mmol/L)
t obs = 1.45 / 0.32 = 4.49, p value < 0.001
31
Confidence Intervals
Hypothesis Tests
Confidence intervals are of the form
Sample Statistic +/- (Zpercentile*) (Standard error)
Lower bound = Sample Statistic- (Zpercentile)(Standard error)
Upper bound = Sample Statistic + (Zpercentile)(Standard error)
Hypothesis test statistics (Zobs*) are of the form
Zobs=(Sample Statistic – null value) / Standard error
* t percentile or tobs for continuous data when n is small
32
Sample statistics and their SEs
Sample Statistic
Mean
Mean difference
Proportion
Proportion difference
Log Odds ratio*
Log Risk ratio*
Slope (rate)
Hazard rate (survival)
Transform (z) of the
Correlation coefficient r*
*Form
Symbol
__
Y
__ __ _
Y1 – Y2 =d
P
P1 – P2
logeOR
logeRR
b
h
z=½loge[(1+r)/(1-r)]
r = (e2z -1)/(e2z + 1)
Standard error (SE)
S/√n = √[S 2/n] = SEM
√[S12/n1 + S22/n2]= SEd
√[P(1-P)/n]
√[P1(1-P1)/n1 + P2(1-P2)/n2]
√[ 1/a + 1/b + 1/c + 1/d]
√[1/a -1/(a+c) + 1/b - 1/(b+d)]
S error / Sx√(n-1)
h/√[number dead]
SE(z)=1/√([n-3])
CI bounds on transformed scale, then take anti-transform
33
Handy Guide to Testing
Sample Statistic &
Comparison
Population null hypothesis
Comparing two means
True population mean difference is zero
Comparing two proportions
True population difference is zero
Comparing two medians
True population median difference is zero
Odds ratio (comparing odds)
True population odds ratio is one
Risk ratio=relative risk (comparing risks)
True population risk ratio is one
Correlation coefficient (compare to zero)
True population correlation coefficient is zero
Slope=rate of change=regression coefficient
True population slope is zero
Comparing two survival curves
True difference in survival is zero at all times
34
Nomenclature for Testing
Delta (δ) = True difference or size of effect
Alpha (α) = Type I error = false positive
= Probability of rejecting the null hypothesis when it is true.
(Usually α is set to 0.05)
Beta (β)
= Type II error = false negative
=Probability of not rejecting the null hypothesis when delta is not zero
( there is a real difference in the population)
Power
=1–β
= Probability of getting a p value less than α
(ie declaring statistical significance)
when, in fact, there really is a non-zero delta.
We want small alpha levels and high power.
35
Statistical Hypothesis Testing
Statistic/type of comparison
Mean comparison-unpaired
Mean comparison-paired
Median comparison-unpaired
Median comparison-paired
Proportion comparison-unpaired
Proportion comparison-paired
Odds ratio
Risk ratio
Correlation, slope
Survival curves, hazard rates
Test/analysis procedure
t test (2 groups), ANOVA (3+ groups)
paired t test, repeated measures ANOVA
Wilcoxon rank sum test, KruskalWallis test*
Wilcoxon signed rank test on differences*
chi-square test (or Fishers test)
McNemar’s chi-square test
chi-square test, Fisher test
chi-square test, Fisher test
regression, t statistic
log rank test*
ANOVA = analysis of variance
* non parametric – Gaussian distribution theory is
not used to get the p value
36
Parametric vs non parametric
Compute p values using ranks of the data.
Does not assume stats follow Gaussian distribution – particularly
in distribution “tails”.
Parametric
2 indep meanst test
3+ indep meanANOVA F test
Paired meanspaired t test
Pearson correlation
Nonparametric
2 indep mediansWilcoxon rank sum test=MW
3+ indep mediansKruskal Wallis test
Paired mediansWilcoxon signed rank test
Spearman correlation
37