Statistical Sampling
Part IV – When the MOE Doesn’t Apply
This video is designed to accompany
pages 41-76
in
Making Sense of Uncertainty
Activities for Teaching Statistical Reasoning
Van-Griner Publishing Company
Non-Sampling Errors
The Margin of Error is a nice mathematical way of
addressing sampling variability, also called
“random sampling error.”
There are lots of other “errors” that can affect data
collection and the MOE simply doesn’t apply.
Working Definition
Non-Sampling Error – An error or discrepancy
caused by something other than the fact that a
sample was selected instead of the entire
population. These errors include, but are not
limited to, data entry errors, nonresponse, biased
questions in a questionnaire, question order, and
false information provided by respondents.
Muddled Questions
“Do you plan to leave your job and look for another one
during the coming year?”
“Have you often, sometimes, hardly ever, or never felt bad
because you were unfaithful to your wife?”
(Harris Poll: One percent said often, 14 percent said sometimes or hardly ever,
and 85 percent said they never felt bad because of this.)
“Should we increase taxes in order to get more housing and
better schools, or should we keep them about the same?”
http://exporters-sources.com/designing-the-questionnaire-question-wording-a-problem-of-communications/
Sensitive Questions
Roger Tourangeau and Ting Yan, Psychological Bulletin, 2007, Vol. 133, No. 5, 859–883
“Since 1971, the federal government has sponsored a series of recurring
studies to estimate the prevalence of illicit drug use. Other surveys ask
national samples of women whether they have ever had an abortion or ask
samples of adults whether they voted in the most recent election. “
“An important question about such
surveys is whether respondents
answer the questions truthfully.
Methodological research on the
accuracy of reports in surveys about
illicit drug use and other sensitive
topics … is a major source of error,
more specifically of bias, in the
estimates derived from these surveys.”
Percentage
Nonresponse
Total household income
8.15
No. of lifetime male sexual partners
3.05
Received public assistance
2.22
No. of times had sex in past 4 weeks
1.37
Age of first sexual intercourse
0.87
Blood tested for HIV
0.65
Age of first menstrual period
0.39
Highest grade completed
0.04
Item
How bad is it?
No hard data to compare sampling errors to non-sampling errors, but it is
generally thought that non-sampling errors might be the larger of the two.
Harris Poll Disclaimer:
All sample surveys and polls, whether or not they use probability sampling,
are subject to multiple sources of error which are most often not possible to
quantify or estimate, including sampling error, coverage error, error
associated with nonresponse, error associated with question wording and
response options, and post-survey weighting and adjustments. Therefore,
Harris Interactive avoids the words “margin of error” as they are misleading.
What to do?
Large body of literature on things to try and things that
seem to work
1. Use of inducements for non-responders
2. Use of technology-assisted confidential
interview techniques
3. Awareness of psychology of question order
4. Awareness of experiences with question
wording
5. Use of internal consistency checks
One-Sentence Reflection
Non-sampling errors can substantially
detract from the accuracy of survey
data, and are not addressed by the
margin of error.