MEASUREMENT
Measurement Error &
Survey Construction
1
Measurement: Measurement Error in
Survey Research
•One Requirement of Effective Research Design:
Minimizing measurement error (Remember MAXMINCON
Principle!)
• What is measurement Error?
Measurement Error = Observed Score - True Score
• What are the sources of measurement error in survey
research and how do we minimize it?
2
Measurement Error in Survey Research
Types of Measurement Error/Bias:
• Instrumentation Error/Bias
• Error that results from lack of fidelity of the
measurement instrument (e.g., poor instructions,
wording, scales, response options)
• Response Error/Bias
• Error that stems from the mentality and/or
predisposition of study participants
3
Measurement Error in Survey Research-Instrumentation Bias
Sources of Instrumentation Bias -- How to Minimize It?
A. Poor Wording of Items/Instructions:
– Guideline #1: Are some words in the item ambiguous? Spell
out acronyms, define difficult to understand terms, and
reword the items using clearer and more commonly
recognized vocabulary.
– Guideline #2: Does the question state the necessary/
appropriate frame of reference/criterion for answering? If not,
the criterion must be clearly indicated.
– Guideline #3: Does the question overemphasizes/
exaggerates/dramatizes some condition. If so, it must be toned
down and stated in less dramatic terms.
4
Measurement Error in Survey Research-Instrumentation Bias
– Guideline #4: Is the question as specific as it can reasonably be?
If the item is too general, state it more specifically.
– Guideline #5: When possible, avoid using negatives and doublenegatives in statements--they are confusing.
–
Guideline #6: Avoid putting a blank in the middle
of an item.
5
Measurement Error in Survey Research-Instrumentation Bias
– Guideline #7: Use your pronouns carefully when writing both, the items
as well as the instructions.
– Guideline #8: Choose proper and clear wording and sentence structure
for the items and instructions.
– Guideline #9: Is the question likely to lead respondents toward a
particular answer (i.e., is it a leading question)? If so, the leading phrase
must be removed.
6
Measurement Error in Survey Research-Instrumentation Bias
B. Poor Content of Items/Questions:
– Guideline#10: Is the question applicable to all respondents? If not,
reword it and/or use a detour around it for those to whom it does not
apply.
– Guideline#11: Avoid branching if at all possible. The two items above
can easily be rewritten as one question that applies to everyone.
– Guideline #12: Does the item contain an example that is also a possible
answer? If so, change or discard the example.
7
Measurement Error in Survey Research-Instrumentation Bias
– Guideline #13: Does the question require respondents to remember too much
detail or to recall distant events? If so, it must be modified or generalized to
make recall easier.
– Guideline #14: Is the question as free from threat to respondents as possible?
If not, change it to reduce the threat; consider depersonalizing it and/or asking
about hypothetical situations.
– Guideline #15: Does the question include more than one issue (i.e., a doublebarreled item)? If so, it must be split or modified to include only a single
question .
– Guideline #16: Is the question “lladed” with a reason for responding in a
particular way? If so, the reason must be deleted.
8
Measurement Error in Survey Research-Instrumentation Bias
– Guideline #17: Underline, italicize, or use bold print to draw
respondent’s attention to important terms in the instructions.
C. Poor Response Format/Options:
– Guideline #18: Make sure the list of response options is exhaustive.
– Guideline#19: Are “Yea-sayers” or “nay-sayers” likely to always
choose one answer? Reword the item to avoid using “Yes” and “No”
as choices.
– Guideline #20: Use “Always” and “Never” with caution; “Almost
Always” and “Almost Never” may be much better substitutes.
9
Measurement Error in Survey Research-Instrumentation Bias
– Guideline #21: Note that not all rating scales are bipolar or have to be
balanced/symmetrical with a neutral mid-point (e.g., importance, likelihood, etc.).
– Guideline #22: When the respondent is to select only one choice from a list of
several options, make sure the choices are mutually exclusive.
– Guideline #23: Make sure that the formats of your items and your response
options match (are consistent).
10
Measurement Error in Survey Research-Instrumentation Bias
D. Organization and Order:
– Guideline #24: Be cognizant of position/order bias.
– Guideline #25: When appropriate, use the funnel approach to avoid the
potential assimilation effect. That is:
 The more general first, followed by the more specific
– Guideline #26: Use duller demographic questions in a section at the end of
the questionnaire (the opposite in interviews)
– Guideline #27: Use more interesting and attention-grabbing set of questions
earlier in the survey.
11
Measurement Error in Survey Research-Instrumentation Bias
– Guideline #28: Use more sensitive, personal and/or threatening items later in the
survey.
– Guideline #29: Organize questionnaire by arranging items into subsections (based
on content or/and response format), each with a clear and short instruction.
– Guideline #30: At the end of the survey, indicate what your respondents are
expected to do next—e.g., mail the survey to whom, at what address, by when.
12
Survey Research
MEASURMENT ERROR:
SOURCES OF RESPONSE ERROR/BIAS
13
Measurement Error in Survey Research-Response Bias
Response Error/Bias:
Error that stems from the mentality
and/or predisposition of study participants.
Sources of Response Bias -- How to minimize it?
– SOCIAL DESIRABILITY:
Response based on what is perceived as being socially
acceptable or respectable.
14
Measurement Error in Survey Research-Response Bias
–ACQUIESCENCE:
Response based on respondent’s perception of what
would be desirable to the sponsor.
– YEA- AND NAY-SAYING:
Response influenced by the global tendency toward
positive or negative answers.
15
Measurement Error in Survey Research-Response Bias
–PRESTIGE:
Response intended to enhance the image of the
respondent in the eyes of others.
– THREAT:
Response influenced by anxiety or fear instilled by
the nature of the question.
16
Measurement Error in Survey Research-Response Bias
–HOSTILITY:
Response arising from feelings of anger or
resentment engendered by the response task.
– AUSPICES:
Response dictated by the image or opinion of the
sponsor rather than the actual question.
17
18
–MENTAL SET:
Cognitions or perceptions based on previous items
influence response to later ones (the “priming”
problem).
– ORDER:
The sequence in which a series is listed affects the
responses to the items.
– EXTREMITY:
Clarity of extremes and ambiguity of midrange
options may encourage extreme responses.
18