POPULATION RESEARCH SEMINAR SERIES
Sponsored by the Statistics and Survey Methods Core of the U54 Partnership
Post Survey Adjustments
Lee Hargraves, Ph.D. & Lois Biener, Ph.D.
Center for Survey Research
University of Massachusetts Boston
Statistical Areas of Concern
• Nonresponse is a problem if there are
systematic differences in who responds to a
survey, especially if it is related to key
measures
Weight Adjustments:
Post-stratification
• Need to know or estimate characteristics of
the population you want to generalize to.
• For example, gender, race, age, Hispanic
origin, education level, income level.
• Such estimates may come from Census data or
ACS (American Community Survey) data.
• For health care surveys, know characteristics
may come from administrative data, claims, or
electronic health record data
Weight Adjustments: Post
Stratification
• You need to measure these same
characteristics in your survey.
• You can compare the distributions from your
survey to the distributions from the “best”
source (or “controls”).
Example 1: Statewide Medicaid Survey
Gender
% among Respondents
% among Sample
Female
34.5
35.1
Male
65.5
64.9
Race
% among Respondents
% among Sample
Black
15.0
17.3
White
80.0
78.9
Other
5.0
3.8
Spanish Language
% among Respondents
% among Sample
Yes
13.7
11.2
No
86.3
88.8
Source: A 2012 survey among members of Medicaid health plans, conducted by mail (3 contacts) and telephone.
Example 1: Statewide Medicaid Survey
Age
% among Respondents
% among Sample
18 – 24
9.5
13.0
25 – 34
15.4
20.5
35 – 44
19.8
20.6
45 – 54
25.6
22.9
55 – 64
19.2
14.1
65 – 74
6.5
5.4
75 or older
4.0
3.4
Source: A 2012 survey among members of Medicaid health plans, conducted by mail (3 contacts) and telephone.
Example 1: Statewide Medicaid Survey
Age
% among Respondents
% among Sample
18 – 24
9.5
25 – 34
15.4
35 – 44
19.8
20.6
45 – 54
25.6
22.9
55 – 64
19.2
14.1
65 – 74
6.5
5.4
75 or older
4.0
3.4
24.9
13.0
20.5
33.5
Source: A 2012 survey among members of Medicaid health plans, conducted by mail (3 contacts) and telephone.
Weight Adjustments:
Post Stratification
• If they differ, can consider adjusting the
weights to “make them match.”
• This is what is called post-stratification weight
adjustment.
What is a Survey Weight?
• A value assigned to each case in the data file.
• Typically weights are used to make statistics computed
from the data more representative of the population.
• Each case has a non-zero value that indicates how
much it will count in statistical procedures.
• Examples:
– A weight of 1 means that the case only counts as one case
in the dataset.
– A weight of 10 means that the case counts in the dataset
as ten identical cases.
– Weights can be a fraction, but are always positive and nonzero.
Example 2: Men may be less likely
to respond to surveys
Gender
Men
Women
100%
90%
More accurate
representation of
the state’s men and
women.
80%
70%
60.5%
60%
52.1%
47.8%
50%
40%
39.5%
30%
20%
10%
0%
unweighted
weighted
Source: A statewide survey of 4608 adults (age 18 +) using random digit dialing (RDD) of landlines with a
supplement of mobile phone surveys. To ensure that the sample represented the population in terms of age,
gender, and ethnicity, data were weighted to match Census data. (2012, a state in New England)
Example 2: Effect of differential
non-response on key measures
Has a Usual Source of Medical Care
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
No
Yes
89.0%
85.6%
More accurate
representation of
people with access
problems.
11.0%
unweighted
14.4%
weighted
Source: A statewide survey of adults (age 18 +) using random digit dialing (RDD) of landlines with a supplement
of mobile phone surveys. To ensure that the sample represented the population in terms of age, gender, and
ethnicity, data were weighted to match Census data. (2012, a state in New England)
Example 2: Effect of differential
non-response on key measures
Body Mass Index
< 18.5
18.5 - 24.9
25.0 - 29.9
30.0 and above
100%
90%
80%
70%
60%
50%
40%
33.5%
35.7%
34.6%
30%
34.2%
23.3%
More accurate
representation of
the percent of adults
who are obese.
21.7%
20%
10%
1.9%
1.5%
0%
unweighted
weighted
Source: A statewide survey of adults (age 18 +) using random digit dialing (RDD) of landlines with a supplement
of mobile phone surveys. To ensure that the sample represented the population in terms of age, gender, and
ethnicity, data were weighted to match Census data. (2012, a state in New England)
How are weights calculated?
• A topic for a longer session
– Post-stratification are one potential adjustment.
– Non-response adjustments are another.
– Both depend on the sample design.
– Both are affected by the response rate.
• Best advice for non-statisticians:
– Take a course on survey sampling.
– Work with a statistician!
Working with survey organization
• You should ask those who collected your data:
– Did they post-stratify weight adjust?
– If so, what factors did they use in this adjustment?
– Can they provide information about the size of the
weight adjustment factors?
– Better yet, can they place on the final data file,
values of weights before and after the
adjustment?
POPULATION RESEARCH SEMINAR SERIES
Sponsored by the Statistics and Survey Methods Core of the U54 Partnership
Questions? Comments?
Send us an email!
u54.ssmc@gmail.com