Questionnaire Design: Part 2
Slide 1
In addition to the issues surrounding the creation of good questions, there also are many issues
related to the creation of good questionnaires overall. The goal of this lecture is to discuss many
of the issues related to overall questionnaire design.
Slide 2 (No Audio)
Slide 3
For many of the reasons I’ve mentioned in Part 1 of this lecture, here’s an example of a badly
formatted questionnaire.
Slide 4
Here’s what the questionnaire, from the previous slide, would look like had it been formatted
properly.
Slide 5 (No Audio)
Slide 6
Here’s an example of a double-column formatted questionnaire meant for traditional 8½” x 11”
paper.
Slide 7
If, instead of 8½” x 11” sheets, corner stapled, one chose to use a booklet format, here’s how
that booklet format might appear on legal-sized paper.
Slide 8
Cover letters are important for encouraging people to respond to questionnaires. In essence,
the cover letter is a sales pitch. The researcher is trying to encourage people to spend time
responding to a questionnaire, so they are giving of themselves in return for some cause. That
cause could be helping out or receiving compensation; regardless, the cover letter is a sales
pitch and if people aren’t sold on responding to a questionnaire, then they’ll just trash it.
Slide 9
Different cover letter appeals will be relatively more or less effective in encouraging people to
respond to surveys, depending on the sponsor for the research. This slide summarizes the
results of a study done several years ago of Madison Wisconsin retailers. Different people
received cover letters with different appeals and the researchers wanted to know which appeal
would receive the highest percentage of returns. They used four different appeals in the
different cover letters.

A social utility appeal, which encouraged people to respond because it will help to make
the world a better place. This translated into “Your assistance is needed. Your attitudes
and opinions can provide information that contributes to understanding how consumers
can be better served by local retail shopping facilities. Your cooperation is truly
appreciated.”
Page | 1

A ‘help the sponsor’ appeal, which encouraged people to respond because it would help
the researcher, who would really appreciate the assistance. Hence, this second appeal:
“We need you assistance!! Your attitudes and opinion are very important to our
successful completion of this study. We truly appreciate your cooperation!”

A third appeal suggested that participating in the research benefitted the respondent in
some way. Hence the third appeal: “Your opinions are important. It is important for you
to express your opinion so Madison’s retailers will know the types of products and
shopping facilities you would like to have available. Thanks for expressing your
opinions.”

Finally, a combined appeal in which all three of these features appeared: “Your opinions
are important and useful. Your attitudes and opinions are important for three reasons;
they can provide information that leads to an understanding of how consumers can be
better served by local retail shopping facilities, second, they will enable Madison’s
retailers to know the types of products and shopping facilities you would like to have
available, and third, they will help us successfully complete this study. Thank you for
your cooperation.”
Slide 10
Which appeal was more successful? The researchers mailed some questionnaires indicating
the sponsor of the study was a local university, and they mailed other questionnaires indicating
the sponsor was a commercial marketing research firm. This table shows that if the sponsor
was identified as a university, then a social appeal—helping to make the world a better place or
please help us out—was more effective. In contrast, if it was a commercial survey, then a ‘make
the world a better place, we appreciate your assistance’ appeal was not as effective. Roughly
10% fewer people responded to those appeals for a commercial survey. However, if people
received an appeal that suggested it was in their best interest to respond to a survey, they were
10% more likely to respond if it was a commercial survey instead of a university-sponsored
survey. When a combined appeal was used, it didn’t matter if it was a university-sponsored or a
commercial survey because roughly 40% of people chose to respond in either case.
Slide 11
Here are several general guidelines for the topics that should be addressed in a cover letter.
You should indicate to respondents that they are participating in a bona fide survey. You should
identify the sponsor of that survey as early in the letter as possible. To encourage people to
participate, you should describe the survey’s purpose clearly and simply, and indicate how that
person was chosen to participate in the study. Finally, you should use an appeal like the ones I
just discussed.
Slide 12
What follows on the next nine slides are examples of cover letters for university-sponsored
surveys. Notice that some of these letters address second and third mailings.
Slide 13 to Slide 21 (No Audio)
Slide 22
Here’s a basic checklist for things you should include in cover letters. Does the letter give the
reader enough information to provoke the desired response? Does it answer all potential
Page | 2
questions? Are only essential facts included? (Remember that people won’t want to read more
than about one page, so you have to convince them to respond within that single page). Have
all unnecessary words been eliminated? Is the language clear and easy to understand on the
first reading? (Remember, if your cover letter is difficult to read, potential respondents will
assume that is characteristic of your entire questionnaire and as a result they’re likely to refuse
to participate in your study). Has all factual information been double-checked for accuracy? Has
the letter been proofread by a qualified person for grammatical spelling and punctuation errors?
(Remember, you want your cover letter to appear professional because that would indicate the
professionalism of the entire research project, and respondents who believe the research is not
being conducted professionally are far less likely to respond). Is the letter clean? Is it formatted
for a pleasing appearance? Have all the ten-dollar words been replaced by shorter words that
have the same meaning? If you go through your cover letters and appraise them, relative to this
checklist, you’ll find your final cover letter will be better than your first draft.
Slide 23 (No Audio)
Slide 24
Fielding a survey is an expensive proposition. I’ve found in my consulting experiences that the
data collection costs often exceeded 50% of the total research budget. Therefore, it’s best to
ensure that your questionnaire is properly constructed before you go to the expense of creating
and fielding them. Here are the things you might like to do or consider in pretesting.

What items should be pretested? Typically, we want to pretest all items that have not
been used in previous studies. If we’re using existing scales that already have been
tested, then pretesting is of lesser importance.

How should the pretest be conducted? Typically, we want the pretest to be under
circumstances as similar as possible to the circumstances under which respondents will
answer our questions. If respondents will be in a noisy and distracting environment, then
the pretest should be conducted in such an environment as well.

Who should conduct the pretest? The researcher should conduct the pretest because
the researcher is in the best position to know how to make modifications based on the
results of that pretest.

Who should the respondents be for a pretest? Pretest respondents should be as similar
to the ultimate respondents as possible. If I’m conducting a study in which many
respondents will be elderly, then it makes little sense to pretest the questionnaire on
college students, who may have totally different predispositions, vocabularies, and
reading abilities.

How large should the sample be for a pretest? There’s no need for an expensive pretest
with many respondents. Also, there is no sense in wasting real respondents on a pretest.
Basically, convenience samples—people who are readily available at low cost and who
are relatively similar to the ultimate respondents—are preferred.
Slide 25
As internet surveys are becoming increasingly popular for many reasons--especially cost—I
should spend a few minutes discussing that type of survey.
Page | 3
Slide 26
Having participated in several internet-based surveys, I can vouch for a basic layout for such
questionnaires. Typically they’re written in html, and as a result they’re graphical. Often, I’ve
needed to page from one screen to another and just as often I’ve needed to scroll down a page
before going to the next page. In responding to questions, I’ve needed to click on a radio button.
Because software often precludes paging to the end of web-based questionnaires—to get a
sense for how long they’ll take to complete—I’m often provided with a status bar indicating the
percent of the questionnaire I’ve completed to that point. This will give me an idea of how much
longer I’ll need to finish the questionnaire.
Slide 27
Internet-based questionnaires tend to require responses using one of the following formats:



Radio buttons, in which all the choices are provided on screen and respondents need to
click on the button adjacent to a choice that most corresponds to their attitude or
behavior;
Drop- down boxes, which are like the drop-down menus in most current software; and
Check boxes and open ended boxes, in which respondents type an open-ended
response to a question.
Slide 28
Internet-based questionnaires have welcome screens in the same way that self-administered
mail questionnaires have cover letters. The welcome screen provides the relevant information
about the sponsor and the purpose of the research, and is meant to encourage people who’ve
reached that screen to continue on and complete the questionnaire.
Slide 29
To ensure that the proper respondent is answering the questionnaire and to avoid multiple
responses—either intentional or unintentional—by a single person, it’s best to send via email a
password and instructions to potential respondents so that they’ll enter that information into the
questionnaire and ensure that only one response has been received from a qualified person.
Slide 30
What follows on this and the next four slides are some sample screen shots for an internet
survey.
Slide 31 to Slide 34 (No Audio)
Slide 35
Here are four advantages for internet surveys relative to self-administered mail surveys.

Internet surveys can be programmed so that the appropriate skips and logic is followed.
As a result, people only answer the questions that are intended for them, either based
on who they are or their previous responses.

Variable piping means that text can be inserted within a question based on one or more
previous responses and/or the respondent’s profile. (Think about the junk mail you’ve
Page | 4
received with your name inserted into the cover letter; that’s an example of variable
piping.)

It’s possible to evaluate answers as they’re being entered. Because internet-based
survey software can identify errors during the interview process, it can inform
respondents that they’ve answered inappropriately and should try again.

Internet-based questionnaires can include online help, which is impossible to provide for
self-administered mail questionnaires. If respondents have problems with a selfadministered mail questionnaire, their only recourse is to e-mail or telephone the
researcher, both which are likely to take more than a few minutes and provide additional
barriers to questionnaire completion. As a result, these respondents are more likely to
terminate before completion. Interactive help is available immediately, so respondents
are more likely to continue once they’ve received it.
Slide 36
Here’s a recap of this lecture on questionnaire design. After summarizing basic considerations
before designing questionnaires, I covered screener questions, filter questions, and skip
patterns. Next, I talked about general question organization; which questions tend to come
initially in questionnaires and which ones come toward the end of questionnaires. I talked about
the aesthetics of questionnaires in terms of typeface and blank spacing. Next, I talked about
different answer formats for questions and the possibility of pre-coding questionnaires to ease
data entry. Then, I provided several sample questionnaire layouts with some do’s and don’ts. I
talked about cover letters and different appeals, guidelines for designing cover letters, and
showed many examples of cover letters. I closed with discussions about pretesting and internet
surveys.
Page | 5