Developing Establishment Survey
Questionnaire Design Guidelines
at the U.S. Census Bureau
Rebecca L. Morrison
Outline
Introduction & Background
Development of the Guidelines
Preliminary Guidelines
Application of Guidelines
Conclusion & Next Steps
2
Introduction & Background
Introduction & Background
U.S. Census Bureau:
– “…leading source of quality data…nation’s
people and economy”
– Tries to minimize burden of data collection
Burden = time, level of effort
Reduce cognitive burden through visual
design, structure of data request
4
Introduction & Background
(cont’d)
Paper does not focus on specific question
wording and order issues.
Paper presents preliminary guidelines in
questionnaire design – formatting,
layout, navigation, instructions, and
data request phrasing and style.
5
Development of the Guidelines
6
Development of the Guidelines:
Census Bureau
Agency standards for various survey
quality issues
No standard for design of questionnaires
Economic surveys moving toward
consistent “look and feel”
7
Development of the Guidelines:
ESMS
Establishment Survey Methods Staff
– Group of survey methodologists
– In-house consultants to economic
programs
– Noticed differences across questionnaires
– Survey programs work with ESMS on
questionnaire design and pretesting
– Catalyst towards consistency
8
Development of the Guidelines:
Methods
Questionnaire design principles from
standard texts
Special attention to literature on visual
design
Based on pretesting a variety of
questionnaires with respondents
9
Development of the Guidelines:
Considerations
List of design elements to consider
Used to assess tradeoffs within specific
survey conditions
Not a cookbook
10
Preliminary Guidelines
Guideline A: Text styles
Be aware of text styles, and how they are
used for emphasis within a survey
instrument.
12
Guideline A: Example
2004 Annual Survey of Local Government
Finances
13
Guideline A: Example
14
Guideline A: Text styles
These are favored by respondents:
– Print item numbers in reverse-print
bubbles, e.g., , , 
– Print questions in bold black text.
– Print instructions in plain text or italics.
– Use an 8-point font or larger.
15
Guideline A: Text styles
Answer spaces:
– Open, non-delineated vs. delineated
– Respondents do not seem to have a
strong preference
– Be consistent
16
Guideline A: Text styles
Key codes / Punch codes:
– De-emphasize processing codes for
respondents
– Print in a darker shade of background
color
– Example:
17
Guideline B: Reduce visual
clutter. Use natural navigational
paths.
Reduce clutter on the page. Use
navigational paths and layouts that are
natural and readable for respondents.
18
Guideline B: Reduce visual
clutter. Use natural navigational
paths.
19
Guideline B: Reduce visual
clutter. Use natural navigational
paths.
Format in 1997 Economic Census
confusing:
– Lines were “speedbumps” to navigation
– Not clear how respondent was to navigate
two adjoined columns
Format changed for 2002
– Single column of questions
– Generally, a single column of response
options
20
Guideline B: Reduce visual
clutter. Use natural navigational
paths.
Two columns of questions
– Occasionally done when questions:
• Are shorter
• Do not involve extensive instructions
• Do not ask for numerical information
– Survey of Business Owners (SBO)
• Information collected is categorical or ordinal,
not interval or ratio
21
Guideline B: Reduce visual
clutter. Use natural navigational
paths.
Spread out the questions across more
pages
Use bulleted lists
More open space = more “user-friendly”
22
Guideline C: Instructions
Place instructions close to questions, or
incorporate them into questions.
Where possible, convert instructions
into questions.
23
Guideline C: Instructions
Instructions convey specifications, intent
of question
Respondents tend not to pay attention to
instructions, or only look when they
think they need them
24
Guideline C: Instructions
2 Goals for instructions:
– Eliminate, or reduce, amount of
instructions located separate from
question
– Place instructions/information where it is
most needed
25
Guideline C: Instructions
Convert instructions into questions so
respondents attend to them
– When content critical to correct
interpretation of later questions
– When it helps clarify/correct reported data
Example (2004 Medical Expenditure
Panel Survey):
26
Guideline D: Avoid sentence
fragments and key words
Phrase data requests as questions or
imperative statements, not as sentence
fragments or key words.
27
Guideline D: Avoid sentence
fragments and key words
Types of data requests
– Question (question word, question mark)
– Imperative statement (report, enter, add)
– Sentence fragment (key words, no verb)
28
Guideline D: Avoid sentence
fragments and key words
Example, 2002 Economic Census:
– Is this establishment physically located
inside the legal boundaries of the city,
town, village, etc.?
Versus:
– Type of municipality where this
establishment is physically located
29
Guideline D: Avoid sentence
fragments and key words
Cognitive interview study: respondents
prefer questions over sentence
fragments
Survey methodology grad students:
questions more effective, imply
respondent has to do something
30
Guideline E: Simple vs.
complicated questions
Ask additional, simple questions, rather
than fewer, more complicated ones.
31
Guideline E: Simple vs.
complicated questions
Length and complexity of question affects
how long it takes for respondent to
understand it
More complicated questions might also
be more likely to be double-barreled
May be easier for respondents to answer
series of shorter, simpler questions
32
Guideline E: Simple vs.
complicated questions
Example: 2002 Industrial Research &
Development Survey
33
Guideline E: Simple vs.
complicated questions
2006 R&D Survey
34
Guideline F: Use matrices
judiciously
Use matrices judiciously, and consider
the likely respondents’ background
when deciding whether or not to use
them.
35
Guideline G: Be consistent!
Items should be numbered in the same
way
Bold and italicized print should always
mean the same thing
Navigation path should remain constant
Data requests should be in the same form
Variation can be confusing for respondents
36
Application of Guidelines
37
Guideline A: Text Styles
Work in progress among estab surveys at
Census Bureau
Economic Census automation required
consistent design features
– Plain text: questions
– Italics: instructions, definitions,
include/exclude lists
– Bold: Separating items into categories, sums
of added lines, emphasis within questions
38
Guideline A: Text Styles
Survey of Business
Owners:
– Questions: plain
– Instructions: bold
and/or italics
– Emphasis within
questions: bold
and/or italics
39
Guideline B: Reduce visual
clutter. Use natural navigational
paths.
Survey of Residential Alterations and
Repairs (SORAR)
– Two columns to one column
– Removed or lightened lines
– Use of bulleted lists
– More open space
40
Guideline B: Reduce visual
clutter. Use natural navigational
paths.
Old
SORAR
41
Guideline B: Reduce visual
clutter. Use natural navigational
paths.
New SORAR
42
Guideline B: Reduce visual
clutter. Use natural navigational
paths.
Two columns:
Survey of
Business
Owners
43
Guideline B: Reduce
visual clutter. Use
natural navigational
paths.
BEA quarterly foreign
direct investment (FDI)
Complex navigational
path
Crowded text
44
Guideline B:
Reduce
visual clutter.
Use
natural
navigational
paths.
BEA form
revised
45
Guideline C: Instructions
Economic census continues to use
separate instruction sheets and
booklets
Two questionnaires moved many
instructions to questionnaire:
– 2007 Commodity Flow Survey
– BEA quarterly FDI form
46
Guideline C: Instructions
47
Guideline D: Avoid sentence
fragments and key words
Mixed success in application across
surveys
48
Guideline D: Avoid sentence
fragments and key words
Commodity Flow Survey uses mix of
questions and imperative statements
49
Guideline E: Simple vs.
complicated questions
Example:
2002 Survey
of Business
Owners
50
Guideline E:
Simple vs.
complicated
questions
Example: 2007
Survey of Business
Owners
(still in cognitive
pretesting)
51
Guideline F: Use matrices
judiciously
Example:
BEA
quarterly
FDI
52
Guideline F: Use matrices
judiciously
53
Conclusion & Next Steps
54
Conclusion & Next Steps
Guidelines are preliminary, based on
principles of questionnaire design
Further refinement is necessary before
they can be applied to establishment
surveys at Census Bureau
55
Conclusion & Next Steps
Other agencies/organizations may
develop guidelines that work better for
their environment
There is no cookbook for questionnaire
design
56
Thank you!
Rebecca L. Morrison
301-763-7595
Rebecca.L.Morrison@census.gov
57