Sources of Systematic Error in Surveys
Slide 1
This lecture is on the sources of systematic error in surveys.
Slide 2
This cartoon illustrates why we care about systematic error in surveys. In the example, a survey
of Lennon University graduates indicated that their average annual income was $80,000. Yet, in
the background we see a less successful alum. What happened was that less successful alums
didn’t complete and return questionnaires, and thus didn’t report on their current salaries. As a
result, they weren’t included in an analysis of the survey data, which created upwardly biased
results because it’s clear the average graduate doesn’t earn $80,000 annually.
Slide 3
Here we see the two sources of error in surveys; random sampling error and systematic error.
Random sampling error is, by definition, beyond our control. It’s the luck of the draw, in terms of
who in the population was chosen or was excluded from responding to our survey. Systematic
error is something under the researcher’s control, at least to some extent.
Slide 4
Here’s a formal definition of random sampling error. It should be obvious why such error is
beyond the researcher’s control.
Slide 5
In contrast, systematic error is caused by one or more mistakes introduced by the researcher.
These mistakes could be in the way the research was designed or in the way the research was
executed.
Slide 6
This graphic summarizes the various sources of survey error. I’ll chat about each of these in
subsequent slides.
Slide 7
Clearly, respondents can introduce error into our survey results. As this slide indicates, they can
introduce bias in two ways: through their responses or by their decision not to respond. If, in
fact, the people who choose not to respond are systematically different from those who choose
to respond, as in the original cartoon regarding Lennon University, then our results will be
biased.
Slide 8
Non-response error can be caused by our inability to successfully contact some respondents;
for example, people who are excessively busy (like physicians and CEOs), people who are
rarely home, or people who refuse to cooperate. People who are unenthused about the purpose
of our research may pass on it. They may be preoccupied at the moment you contact them and
you never reach them again on a call back (if we’re talking telephone interviews). There are
many reasons why potential respondents become non-respondents.
Page | 1
There’s also the issue of self-selection bias. People may choose, because of some personal or
study-related characteristic, to participate or not to participate in our surveys. People who
choose to cooperate in a study are more interested in it and its influence on the decisions of the
sponsoring organization. As a result, there’s often over-representation of extreme opinions and
under-representation of indifferent opinions.
Finally, non-response error tends to vary by type of interview. For example, it’s a far lesser issue
for personal interviews than for mail or internet surveys; response rates for the latter types of
surveys may be as low as a few percent.
Slide 9
There are four reasons why people cooperate with survey researchers: to be supportive, to
have a social interaction, curiosity about the research, and to be compensated for participating
in a study.
They may want to be supportive or helpful, especially if the interviewer is professional and
polite, and the research problem seems interesting. Perhaps a university professor is
conducting the study and they want to be supportive of basic university research.
It’s a chance for a social interaction. Think about retirees and the elderly, who may crave the
opportunity to talk to someone for an hour, even if it’s about laundry detergent or hand soap.
They may be curious about the research. I once spent 45 minutes on the telephone for a survey
about cottage cheese. I couldn’t believe anyone would field such an extensive survey on such a
minor product. I wanted to discover the kinds of questions the researchers had the field service
ask respondents.
Finally, there’s compensation for their time and effort. When people do cooperate in surveys,
typically they receive a token remuneration in advance, such as a dollar or a free pencil or pen.
Alternatively, in return for participation they may be given the opportunity to receive the results
of the survey or to access those results online.
Slide 10
Why do people opt out of our research and refuse to participate? There are several reasons.
They may be afraid. They may not know to what purposes the research will be put. If somebody
asks personal information, respondents may be concerned that the information may fall into the
wrong hands. They may be concerned about invasion of privacy; some people don’t want others
to know about their lives and their activities. They may be hostile to the survey sponsor; for
example, if I was a disgruntled new graduate of NMSU and I was contacted by an NMSU faculty
member to participate in a survey, I might refuse on that basis alone. They may be hostile to the
study topic. Some studies may be about sensitive topics that people may choose to avoid. For a
personal interview, it could be that a respondent is hostile to the interviewer because of the
interviewer’s appearance or manner.
Slide 11
Here’s an excellent definition of response bias: the bias that occurs when respondents answer
questions with a certain slant that consciously or unconsciously misrepresents the truth. It’s not
a matter of respondents choosing to lie to us; rather, they may respond inaccurately without
awareness that those responses are inaccurate. Although unintentional, the data for analysis is
inaccurate.
Page | 2
Slide 12 (No Audio)
Slide 13
Unwillingness to respond accurately could be caused by concerns about invasion of privacy,
time pressure and fatigue, constraints imposed by the physical or social environment, the nature
of the questionnaire being fielded, or respondents just opting to be mischievous.
People may be concerned about invasion of privacy. After all, it’s impossible to be certain how
one’s responses will be used. Perhaps the IRS will compare your self-reported income to your
tax return!
It’s possible that people are time pressured or fatigued. This problem may arise when we
contact someone at the end of the day or in the evening, or when we contact a harried shopper
who’s been at the mall for several hours and is anxious to leave. It’s also possible that a
respondent who’s been kept on the telephone for half an hour and is now fatigued or has other
pressing matters is no longer willing to answer later questions as fully as earlier questions.
The social or physical environment may induce response errors. If a housewife with three
screaming kids in the background is asked excessively many questions during a telephone
interview, then her responses may become increasingly curt or careless. Respondents in mall
interviews may worry that their answers will be overheard by other mall patrons.
The nature of the questionnaire—the way the questions are worded in a self-administered or
internet questionnaire and how the questions are displayed—may induce an unwillingness to
respond accurately. For example, questions with a seemingly liberal slant may anger politically
conservative respondents.
Finally, what I call the mischievous respondent—and I know some of you listening to this lecture
are people who, for whatever reason, decide to answer survey questions erroneously merely to
foul up the research effort—may choose to answer inaccurately. I know from querying students
in my in-class marketing research course—who should be predisposed to respond otherwise—
that roughly 10% have admitted to responding mischievously at least once.
Slide 14
Here’s an example of what I refer to as a mischievous respondent. In this case, the respondent
is using the research as an excuse to be funny rather than as an opportunity to foul up
someone’s research. Regardless, the respondent is providing answers that are inaccurate.
Slide 15
It’s possible that people respond inaccurately for reasons beyond their control. To an extent,
articulateness relates to the ability to respond accurately, and some folks are more articulate
than others. It’s also possible that people respond inaccurately because they don’t know the
correct answer. One example in which a respondent legitimately could be ignorant about the
answer to a survey question: we could ask about other people’s attitudes and behaviors. We
often conduct surveys of households, yet we limit our questioning to one household head. If we
ask that household head about the attitudes or behaviors of the other household head, as we
might in the case of a traditionally married couple, then we may find that our respondent knows
little or nothing about the other household head’s attitudes or behaviors.
Page | 3
Respondents may be ignorant in the sense that they can’t predict their own behaviors. As you’ll
see from the next slide, people generally are poor predictors of their own behaviors. It’s likely
you’re a poor predictor of what you’ll eat for lunch tomorrow, let alone whether or not you’ll buy
a consumer durable, like an automobile or a home, six months or a year from now. Asking
people to predict their own behaviors is a very ‘iffy’ proposition and I usually recommend that
rather than asking people to predict their own behaviors, we ask them about their most recent
behaviors. People are far better at remembering their most recent behaviors, and surprisingly,
their most recent behaviors are a far better predictor for their future behaviors than any
prediction they might make about their future behaviors.
Finally, we might mistakenly ask someone a question that was meant for another person; in
other words, we might ask a question that the respondent is not qualified to answer. For
example, I teach marketing research but I don’t teach sales management. Occasionally, I’ll
receive questionnaires from other faculty asking me to respond to a series of questions about a
course that I have not ever taught, like sales management. Clearly, I can’t answer those
questions intelligently because those questions are meant for someone else.
Slide 16
This slide is based on a previous research study that shows people are poor predictors of their
subsequent behavior. In this case, 100 people predicted that they would buy Brand A in the next
12 months, when in fact only 44 of those 100 people even bought a product like Brand A, and of
those 44, only 30 people bought Brand A. These people may have been thoughtful in trying to
predict their purchase behavior in the next 12 months, but only 30% of them were correct.
Slide 17
This cartoon illustrates a situation in which someone has been asked a question that is clearly
ill-suited to them. It’s quite obvious that this ‘gentleman’ isn’t in the market for a twin engine,
private aircraft.
Slide 18
Here’s an example of respondent ignorance that should have been obvious to the researcher.
This cartoon is humorous because the researcher is not asking the woman how many people
live in the U.S., but she interprets the question that way.
Slide 19
Another reason people are unable to respond accurately is faulty memory. Human memories
are far from perfect, especially when we ask about behaviors over time or in the distant past and
those behaviors concern relatively trivial matters, like the purchase of shampoo or dining at fast
food restaurants. When we ask about the frequency of behaviors over an extended period,
people create mental models of their behaviors, and those models tend to be inaccurate. What
people do when we ask them how often they’ve eaten at McDonalds in the last six months,
they’ll work though something like the following mental calculus: “I tend to eat there about once
a week, there are 26 weeks in six months, so I’ve eaten at McDonalds 26 times in the last year.”
That answer is unlikely to be accurate. To avoid problems with asking about the frequencies of
behaviors over time, we only should ask about recent behaviors. Instead of asking someone
how often they’ve eaten at McDonalds in the last six months, we should ask them how often
they’ve eaten at McDonalds in the last two weeks.
Page | 4
Also, peoples’ memories are subject to telescoping. People tend to remember things that are
memorable and they remember them as having occurred more recently than was the case. If
you ask me the last time I dined at a high-end restaurant, I might recall having done so in the
last few months, when in fact I may not have done so in the last half year.
Slide 20
Now I’ll talk briefly about other reasons for unconscious misrepresentations by respondents.
Slide 21
Acquiescence bias, or yeah saying, is a type of response bias due to some people’s tendency to
agree with all questions or to concur with a particular position. The problem with acquiescence
bias is that people who tend to agree tend to limit their responses to the ‘strongly agree’ end of
our multi-point scales. When those extreme answers are averaged with other people’s answers,
those averages will be exaggerated.
Slide 22
Other people just tend to use the extreme points of scales; they either ‘strongly disagree’ or
‘strongly agree’ with whatever we ask them. That’s an example of extremity bias. Other people
have the opposite tendency; they tend to see everything in shades of gray rather than as either
black or white. As a result, they tend to avoid the endpoints of response scales. Either way,
that’s a form of response-style bias.
Slide 23
Interviewer bias is just what the words say: a type of response bias caused by the presence of
an interviewer. The mere presence of an interviewer could influence the type of answers that
people provide. For example, males asked personal lifestyle questions by a female interviewer
may be reluctant to answer accurately because they’re embarrassed. Alternatively, younger
people may respond differently and more enthusiastically to attractive younger interviewers—to
whom they relate better—than to unattractive older interviewers. Regardless, such bias affects
the quality of responses.
Slide 24
This cartoon illustrates interviewer bias. In this case, it’s not merely that the interviewer is an
attractive female, but also that she is providing a free steak dinner along with the interview.
Clearly, the answers this respondent would give are very different under these circumstances
then they would be to a telephone interview.
Slide 25
Auspices or sponsor bias is caused by respondents being influenced by the organization
conducting the study. For example, I might be far more conscientious in responding to an
academic survey then I would be to a commercial survey. In one case, I know I’m helping a
fellow researcher; in the other case, I’m merely helping a company earn additional profit.
Slide 26
Social desirability bias is caused by respondents’ desire, either conscious or unconscious, to
gain prestige or appear in a different social role.
Page | 5
Slide 27
Here are two example of what I mean by social desirability bias. In this first cartoon, we see a
mechanic trying to justify his purchase of an expensive automobile. This is a very old cartoon,
so $14,000 is the equivalent of $50,000 today. The assumption in this cartoon is that the
mechanic doesn’t have the salary or lifestyle to support that kind of expenditure. Nonetheless,
he’s trying to justify his purchase of an expensive vehicle to an interviewer, and rather than
explaining that it was on a whim or that it was purchased in response to a mid-life crisis, he
offers bogus economic reasons for his purchase.
Slide 28
In this second example of social desirability bias, a shopper is being asked about her use of unit
pricing information when buying groceries. Clearly, everyone recognizes that using pricing
information, especially unit pricing information, should result in wiser shopping decisions. To
appear as an informed shopper, one might falsely admit using this type of information.
Slide 29 (No Audio)
Slide 30
Another type of systematic error is administrative error. Some examples of this error type
include improper research administration, a failure to record returned surveys, and a failure to
double check that surveys contained every page.
Interviewer cheating is another example, although it’s only relevant to personal and telephone
interviews. Interviewers typically earn base pay plus a commission per completed interview. As
a result, interviewers have an interest in fabricating data; hence, their faking of answers. Most
commercial research companies certify at least 10 to 15% of interviews to ensure that they were
conducted in the manner specified by the client. Certification entails contacting respondents,
verifying the survey was administered at the appropriate time and place, and double checking
several answers to the questions.
Data processing error is another form of administrative error. In transferring data from
questionnaires to a computer database, it’s possible that numbers could be entered incorrectly.
Typically, a commercial data entry company will enter all data twice and then compare the two
sets. Whenever there’s a discrepancy between the two sets, the original questionnaire is reexamined to determine which entries, if either, are correct. The odds of twice mis-entering the
same data identically are minimal, so we wouldn’t worry about missing those errors. Errors in
data analysis also may occur. It’s possible that a statistical program could be set so that it
recognizes missing values as zeros, as used to be the case in SPSS. Data analysis often
requires us to recode data, and in recoding it, errors may be introduced. When computer
software is used to clean data by searching for discrepancies in answers, the programmer’s
logic may be faulty.
Errors could be introduced at the sample selection stage; either the sample design was
inappropriate or efforts to execute the sampling procedures were mishandled. Finally, there’s
interviewer error, which I’ll talk about more on the next several slides.
Slide 31
Interviewer error often is so subtle that the interviewer is unaware of it. Interviewers must train
themselves to avoid non-verbal responses to people’s answers, either affirmative or negative. If
Page | 6
an interviewer shakes his or her head approvingly after a respondent answers each question,
and suddenly fails to shake his or her head after an answer, then he or she has inadvertently
given a subtle cue about ‘right’ answers. Only a highly skilled interviewer can avoid giving such
cues.
Appearing incompetent is another type of interviewer error. As a respondent, I would be
discouraged from continuing to participate in a study if I sense the interviewer was clueless
about proper procedures. I’d be reluctant to continue because I’d assume the interviewer is the
public face of the study, and if the interviewer is incompetent, then it’s likely everyone
associated with the study is incompetent, and therefore my efforts to respond completely and
carefully are likely wasted. For me, it’s important the interviewer appear competent as a
surrogate indicator of the research team’s overall competence.
Slide 32
In this cartoon, I assume it’s the interviewer who’s incompetent because no one would ask “Who
are you voting against for President?” Such a question is clearly negative and would discourage
anyone from taking this survey seriously.
Slide 33
One truly important advantage of a personal or telephone interview is the interviewer’s ability to
probe respondents further. If a respondent gives an interesting yet incomplete answer, the
interviewer can ask the respondent to say more about that answer. If the interviewer’s probes
are ill conceived, then much of the value of a personal or telephone interview is eliminated.
Slide 34
Here are two humorous examples of what I mean by poor probing. In this first cartoon, the
interviewer responds to the gentleman’s answer by telling him that his answer is loony. That
type of a comment is unlikely to encourage a respondent to elaborate further or perhaps even to
continue to answer questions posed by this interviewer.
Slide 35
In this second cartoon, the interviewer’s failure is not in being hostile or negative towards the
respondent. Instead it’s merely a matter of the interviewer being insufficiently precise, in his
language, so that the wizard answers the wrong question.
Slide 36
Sample selection error entails the improper selection of a sample design or improper execution
of sampling procedures. For example, mall-based interviewers might be instructed to approach
every tenth person who passes them, regardless of race, gender, age, apparent busyness, or
anything else. That way, they—in theory—will be sampling a representative group of people
who visit the mall. However, it may be very tempting for these clipboard holders to approach
younger people or people who seem less busy because those people might provide more
entertaining answers or there’s a greater likelihood they would be willing to participate in the
survey. Such a practice would create sample bias because people who are younger or who are
more willing to participate are likely to differ from mall visitors in general.
Another important issue, in terms of interviewing and sampling, is selecting unqualified
respondents. As I mentioned earlier, there’s an incentive for interviewers to complete interviews
Page | 7
because they receive incentive pay per completed interview. As a result, interviewers might be
encouraged to interview people who are willing to participate but aren’t qualified. Suppose a
study about parents with pre-schoolers and a potential respondent someone with only
teenagers. As an interviewer, I might want to survey that person to earn additional money; as
the researcher in charge of the study, I’d want you to disqualify such people, regardless of their
interest, because they aren’t members of the target sample population.
Slide 37
Finally, here’s a quick checklist of the things that make a questionnaire unacceptable, especially
a mail questionnaire. Questionnaires returned and major portions or key questions unanswered
are useless for data analysis purposes. If there’s evidence the respondent didn’t understand the
task or didn’t take the questionnaire seriously, then we’d want to ignore that questionnaire as
well. If there are missing pages, so the respondent couldn’t answer a large fraction of questions
and responses are anonymous—it’s impossible to re-contact the respondent and get him or her
to complete those missing pages—then that questionnaire is unusable. If the respondent is
unqualified because he or she isn’t a member of the target population, his or her questionnaire
should be ignored. Finally, questionnaires returned after the cutoff date for study completion
should also be excluded because of possible history effects. Events after the cutoff date may
have influenced answers; as a result, those answers won’t be typical of answers submitted
before the cutoff date.
Slide 38
To recap, the goal of this lecture was to alert you to the various sources of systematic error in
survey research. Some of these sources are related to respondents and their ability or
willingness to answer questions accurately. Other sources are related to the administrator, who
may have planned a poorly designed study. Still other sources are related to the interviewer,
who may interact inappropriately with respondents.
Page | 8