Review
Recall: Representative samples matter.
1
Stepping Back: Anatomy of a Sample
Ideally, we want to study a target population composed of
units of observation. In practice, we can study a sample
frame, from which we draw sampling units which compose a
sample.
2
Defining Selection bias.
Selection bias occurs when some part of the target
population is not in the sampled population, or, more
generally, when some population units are sampled at a
different rate than intended by the investigator.
3
Selection can arise in various ways.
▶ Target population ̸= sampling frame
▶ Unit of observation ̸= sampling unit
4
Sampling (Part 2)
Karthik Srinivasan
University of Michigan
October 8, 2025
Simple Random Sample
▶ A simple random sample (SRS) of size n is taken when
every possible subset of n units in the population has
the same chance of being the sample.
5
What are some other
ways to sample?
6
Answers
▶ Stratified sample
▶ Cluster sample
7
Error Framework
▶ Sampling error: Error in estimation due to taking a
sample instead of measuring every unit in the
population.
▶ Nonsampling error: An error from any source other than
sampling error.
• measurement error
• selection bias
8
The most common statistic
The mean, average, or expected value is given by
x̄N =
N
1 X
xi
N i=1
9
An example of sampling error.
▶ Suppose the class heights are given by: 5’2,5’4,5’6,5’8,5’10.
▶ The true population mean height is 5’6.
▶ Suppose we sample 2 members from the class: 5’6, 5’10.
▶ Our sample mean height is 5’8, and our sampling error is
2 inches.
10
How can we estimate
the size of the sampling
error?
11
Recall: Stats 101
p
− µ
Law of Large Numbers (LLN): X̄N →
As sample size N increases, the sample mean converges in
probability to the true mean.
d
2
µ, σN
Central Limit Theorem (CLT): X̄N →
− N
The normalized sample mean approaches a standard normal
distribution as N grows large.
12
Recall: Normal distribution N µ, σ
2
13
What is σ?
d
2
− N µ, σN
Central Limit Theorem (CLT): X̄N →
σ is the population standard deviation. It measures how
spread out the values xi are around the true mean µ.
r
1 PN
2
i=1 (xi − µ)
N
r
1 Pn
2
σ̂ = n−1
i=1 (xi − x̄ )
σ=
14
Estimating a 95% confidence interval
We construct a 95% confidence interval for the unknown
population mean µ using sample data:
CI95% = x̄N ± 2 × σ̂
r
=2∗
1 Pn
2
n−1 i=1 (xi − x̄ )
√
n
15
Apply LLN to a binary
question: Do you prefer
cake to pie?
16
Apply CLT to a a binary
question: Do you prefer
cake to pie?
17
Applying Stats 101 to Cake
The Law of Large numbers: As we sample from more
members of the class, the mean cake preference converges
to the true cake preference.
The Central Limit Theorem : The mean cake preference is
distributed standard normal.
18
What does the margin of error in a
poll account for?
A. Measurement Error
B. Selection Bias
C. Sampling Error
19
Correcting Selection Bias
How can we know if our
sample is selected?
20
Quantifying bias using external benchmarks
▶ Compare sample characteristics to trusted population
data (e.g., Census, ACS, administrative records).
▶ Quantify bias as the difference between sample and
population means: b = x̄sample − x̄pop
21
What if we care about a
population without
government
benchmarks? (E.g. App
users)
22
Other Selection Detection Mechanisms
▶ Use validation data where outcomes are known for both
respondents and non-respondents: metadata, user logs
▶ Use attrition responses as a proxy: differential attrition?
23
How can we correct for
selection into the
sample?
24
Reweighting using post-stratification
▶ Divide the population into known strata (e.g., gender,
age, education).
▶ Compute weights so that sample proportions match
population proportions.
P
▶ Weighted mean: x̄w = Pi wwi xi , where wi = PPpop (stratum)
(stratum) .
i
i
sample
25
Example: Post-stratification with two groups
▶ Suppose in the population: 50% rural, 50% urban.
▶ In our sample: 30% rural, 70% urban.
▶ Share preferring reps: rural = 0.6, urban = 0.4.
▶ Unweighted mean preference: 0.3(0.6) + 0.7(0.4) = 0.46
▶ Weighted mean (using population proportions):
0.5(0.6) + 0.5(0.4) = 0.50
26
The curse of dimensionality
▶ Post-stratification works when there are few adjustment
variables.
▶ As the number of variables grows, the number of strata
increases exponentially.
▶ Many cells become empty or contain very few
observations. (e.g. white Men > 50 who play candy crush)
▶ Propensity score methods address this by summarizing
many variables into a single score.
27
Reweighting using propensity scores
▶ Model the probability that unit i is included in the
sample: p̂i = P(i in sample | Xi ). (Use of Machine
Learning!)
▶ Assign each observation a weight inverse to its
estimated inclusion probability: wi = p̂1i .
▶ Generalizes post-stratification to many covariates.
28
Design-based fixes
▶ Stratified sampling: oversample underrepresented
groups.
▶ Cluster sampling: select natural groupings (schools,
precincts, etc.).
▶ Multi-stage sampling: combine stages to improve
coverage and reduce cost.
▶ Use sampling frames that align more closely with the
target population.
29
Unobserved biases
▶ Weighting and regression can only adjust for observed
differences.
▶ If unobserved variables affect both selection and
outcomes, bias remains.
▶ Controlling for an imperfect proxy can worsen bias!
▶ Even large N cannot eliminate bias from unobserved
differences
30
0
