School of Nursing’s Research
Workshop and Seminar Series
Survey Methods, Questionnaire
Design & Implementation
Paul Krueger, Ph.D.
Associate Professor
Dept. of Clinical Epidemiology & Biostatisitcs
kruegerp@mcmaster.ca
January 19, 2009
Surveys

Many study designs use surveys to collect
information (exposures, confounders, outcomes,
opinions, preferences, KABP)

Systematic method of collecting data

Structured and standardized questionnaire

Sample of the population
Advantages of Surveys



Most methods allow collection a lot of
information, from many people, over a wide
geographic area, in a relatively short time period
Everyone is asked the same questions in the
same way (minimize interviewer bias)
If sampled appropriately, results can be
representative of a larger population (within a
certain degree of error)
Types of Surveys

Mail





focus of this presentation
Telephone


(e.g. survey of all ER and Family MDs about CAP)
(e.g. air pollution - random sample)
Becoming more difficult (e.g. cell phones, answering machines,
call display, call blocking, unlisted, do not call lists)
Fax (e.g. physician survey)
Face-to-face (e.g. LTCF resident QOL/satisfaction)
Internet (e-mail and web based)
Combined methods (e.g. CAPIS, OARS…)
Dillman Approach to
Mail Surveys

Total Design Method (1978)





What aspects of the survey process affect
quality or quantity of response
A detailed and systematic way to design and
implement surveys
Social exchange theory (see below)
High quality mail surveys
Acceptable response rates
Dillman Approach Cont’d

The Tailored Design Method (2000)

Many changes since 1978
Technology – computers, scanners…
 More complete understanding of social exchange
principles in survey research
 Research on survey methods
 Need to tailor the method vs same procedure


Note: Internet, Mail, and Mixed-Mode Surveys:
The Tailored Design Method 3rd edition 2009
Social Exchange Theory

Theory of human behaviour


“The theory asserts that actions of individuals
are motivated by the return these actions are
expected to bring, and in fact usually do bring
from others” (Dillman, 2000)
People will do things that are rewarding
Social Exchange Theory

Three elements for predicting action

Rewards


Costs


What one expects to gain from an activity
What one gives up/spends to obtain rewards
Trust

Expectation that in the long run, rewards of doing
something will outweigh the costs of doing it
Application of Social Exchange Theory

In terms of questionnaire design &
implementation:



Increase rewards for responding
Reduce perceived costs
Establish trust so the rewards outweigh costs
of responding
Ways of Providing Rewards

Show positive regard


Say thank you


e.g. “we appreciate your help”, “thanks in advance”
Ask for help, advice, assistance


Show respect (e.g. give reason for survey, provide
number to call, personally address letters)
Provides a sense of reward
Support group values

appeal to widely shared values – the study’s social
usefulness
Ways of Providing Rewards

Give tangible rewards


Make the questionnaire interesting


Improve layout and design
Give social validation


Include a financial incentive (token) vs promise
Let them know that others have responded
Opportunities to respond are scarce

Respond quickly so information can be used
Ways of Reducing Social Costs

Avoid subordinating language


Avoid embarrassment


Ask for a favour e.g. to solve a problem vs it is
necessary for you to complete this survey because….
E.g. avoid complex questions and instructions at the
start of the questionniare
Avoid inconvenience

E.g. include return postage paid envelope – real
stamp
Ways of Reducing Social Costs

Make questionnaire appear short and easy


Lesson the perceived cost of responding
Minimize requests to obtain personal information

Offer explanations why info is important & ensure
confidentiality
Ways of Establishing Trust

Provide a token of appreciation in advance


Small amount – gesture of trust – vs payment for
your time
Including actual stamps on return envelope

Sponsorship by legitimate authority (e.g. university

Make the task appear important
or gov’t sponsored vs marketing research)



Personalized cover letters on letterhead paper
Professional looking questionnaire
Something useful will be done with results
Dillman’s Five Needed Elements
for High Response Rates
1.
2.
3.
4.
5.
Respondent friendly questionnaire
Up to 5 (varied) contacts
Inclusion of stamped return envelopes
Personalized correspondence
Token financial incentive
Respondent Friendly Questionnaire




Easy to understand questions
Well ordered
Easy to follow layout
Overall appearance
Writing Questions

Develop questions that every potential
respondent will:



Interpret the same way (and the same way as
the investigators)
Be able to respond accurately
Be willing to answer
Dillman’s examples of things to do
when writing questions








Use simple words
Do not be vague
Keep it short
Avoid being too specific
Do not talk down to respondents
Avoid bias
Avoid objectionable questions
Etc.

However, many of the above can be contradictory!
Things to do….






Use equal numbers of pos and neg categories
Distinguish between undecided and neutral
Create mutually exclusive response categories
Avoid double-barrelled questions
Provide appropriate time referents
Etc. ……

See Dillman, Choi (and others) for many other things
that should and should not be done when writing
questions
Questionnaire Construction

Respondent friendly questionnaire design



Will improve response rates (and response rates
among those least likely to respond)
Reduces measurement error (e.g. prevents items or
answer categories from being missed)
Issues to consider:




Questionnaire format
Question order
Appearance of individual questions
Front and back covers
Questionnaire Format

Booklet format is recommended



Handled more or less automatically without
error
People are familiar with them
See Dillman for design issues and rationale
Ordering Questions


A questionnaire is like a conversation
Choose the first question carefully







apply to everyone
be easy
be interesting
be connected to the study purpose
Needs to be a logical flow
Group questions with similar content


Should
Should
Should
Should
Sections with “transition statements”
More salient to least salient
Objectionable questions near end…….
Appearance of Questions








Respondents should see and understand every word
of every question
Font, font size, bold, italics can ease the task of
understanding and responding
Place instructions where needed
If combining questions with same answer categories
– do it carefully
Number questions consecutively
More space between questions than subcomponents
List answer categories vertically (one column)
Etc……..
Front Cover





Opportunity to motivate and increase
response rates
Title (short and simple)
Neutral graphic
Name and address of study sponsor
Extension of cover letter (well known and
legitimate source)
St. Joseph’s Lifecare Centre
2007 FAMILY MEMBER SATISFACTION WITH CARE SURVEY
The purpose of this survey is to collect information that will be used to help improve care and services provided for
your loved one at the St. Joseph’s Lifecare Centre. Your participation is voluntary and all information will remain
confidential. Answers will be pooled together so that no individual response can be identified. With this in mind, we
are interested in your honest opinion, whether it be negative or positive. If you wish to comment on any question,
please feel free to use the space in the margins. Your comments will be read and taken into account.
Thank you in advance for your participation.
Working together to enhance the quality of life by providing love, care, and dignity
St. Joseph’s Lifecare Centre
99 Wayne Gretzky Parkway, Suite 105
Brantford, Ontario
N3S 6T6
Attention: St. Joseph’s Health System Research Network
Back Cover





Keep it simple
Invitation to make additional comments
(chance for people to vent)
Thank you
Lots of “white space”
Avoid questions on back cover

Last few questions tend to be the most
objectionable (e.g. income)
Pretesting

Review by knowledgeable colleagues



Content validity (included all necessary questions;
exclude unneeded questions; appropriate response
categories….)
Diverse expertise (content + methodology)
Potential survey respondents




All words understood
All questions interpreted the same
All inclusive list of response categories
Mailout package create a positive attitude etc….
Pilot Testing




Revise questionnaire and then pilot test all
proposed procedures
Examine response rates, item nonresponse,
variable distributions
Make revisions based on findings
Then, have someone “fresh” that can review the
questionnaire one last time
Questionnaire design is much more complex and
takes much longer than most people expect
Mail Survey
Implementation



Questionnaire design is only 1 part of a
well designed survey
It is not the main determinant of high
response rates
Survey implementation procedures
collectively have a greater influence on
response rates
Dillman Recommends
5 Contacts
Multiple, varied contacts are essential
1. Prenotice (advance notification) letter
2. Questionnaire mailout (package)
3. Thank you postcard (thank you / reminder
letter)
4. Replacement questionnaire (package)
5. Final (special) contact
Prenotice Letter
(First Contact)






Notice to expect a request to participate in an
important study (all letters - low education level)
Brief, personalized, positively worded
Should build anticipation
Courtesy (people like to be told ahead of time)
Mention incentive
Mailed (first class) a few days to week before
questionnaire
Prenotice Letter








Letterhead paper (helps to legitimize survey)
Actual Date / Respondent name and address /
Dear…
What will happen (you should receive a
questionnaire)
What it is about (brief description + sponsor)
Usefulness of survey (how it will help)
Thank you
Real signature + title and affiliations
P.S. Incentive notice (if one is to be offered)
Advance Notification Letter (on Letterhead Paper)
Insert Mailing Date
Insert Caregiver’s Name
Insert Caregiver’s Address
Dear Insert Caregiver’s Name:
I am writing to request your participation in a study to measure family member
satisfaction with care at the [facility name]. The study will collect information
that will be used to help improve care and services provided at the [facility name].
Your participation is voluntary and involves completing a questionnaire that will
be mailed to you in the next week or so. The questionnaire will take about 5 to 10
minutes to complete. Although your help is essential to the study’s success, you
are under no obligation to complete this survey and your nonparticipation will in
no way affect the care of your family member at the [facility name].
I am writing in advance because many people appreciate being advised that a
research study is in process. If you have any questions or concerns regarding this
study, please let me know. Thank you in advance for your help and participation.
Sincerely,
Administrator’s Name
Questionnaire Mailout
(Second Contact)


Sent a few days to 1 week after prenotice
Includes:



Cover letter
Questionnaire (with an ID#)
Token of appreciation ($, TH coupons, pen etc.)
One of the best ways to increase response
 Should accompany questionnaire vs afterwards


Return postage paid envelope (with stamps)
Cover Letter


1 page on letterhead paper
Exact mailing date / Name & address / Dear…







Specifically written to the person vs a form letter
What this letter is about (your request)
Why were you selected (and how)
Usefulness of survey (how this will help)
Confidentiality
ID# on questionnaire (don’t hide it)
Voluntary (if refuse ask for reason)
Cover Letter Cont’d





Enclosures (stamped return envelope + token)
Who to contact with questions (accessibility)
Thank you for helping with this important study
Real signature (contrasting ink – pressed)
Postscript (thanks again - token)
Cover Letter (on Letterhead Paper)
Insert Mailing Date
Insert Caregiver’s Name
Insert Caregiver’s Address
Dear Insert Caregiver’s Name:
About a week ago I mailed you a letter saying that you would be receiving a
questionnaire about family member satisfaction with care at the [facility name].
Enclosed you will find the questionnaire and a postage-paid return envelope. The
information collected will be used to help improve care and services provided at
the [facility name].
Your participation is voluntary. Your questionnaire will be treated confidentially.
Your name will never be placed on the questionnaire or appear in the study results.
You will notice an identification number on the front cover of the questionnaire.
This number is used for mailing purposes so that we may check your name off the
mailing list when your questionnaire is returned. Although your help is essential to
the study’s success, you are under no obligation to complete this survey and your
nonparticipation will in no way affect the care of your family member at the
[facility name].
Please answer all of the questions and return the questionnaire as soon as possible.
We are interested in your honest opinion, whether it be negative or positive. Thank
you for your participation and help. If you would like a copy of the study results,
please let me know. I would be happy to answer any questions you have about the
survey.
Sincerely,
Adminstrator’s name
Assembling and Mailing







Attention to detail
Questionnaire ID# matches the name on letter
and mailing label
Everything is included
Dillman “stuffing procedure” to ensure that none
of the contents are left in the envelope
Larger outgoing envelope (avoid folding)
1st class mail
Select mailout time (avoid major holidays) and
day
Thank you / Reminder
(Third Contact)









Mail 1 week after questionnaire
Thank you / reminder post card (or letter)
Objective is to jog memories
Very short
Thanks those who completed and returned
Reminds others to “please do so today”
Statement of how important each recipient is to
the success of the study
If you didn’t receive it, call and one will be sent
Signature
Thank you / reminder letter (on Letterhead Paper)
Insert Mailing Date
Insert Caregiver’s Name
Insert Caregiver’s Address
Dear Insert Caregiver’s Name:
About a week ago you were sent a questionnaire about family member
satisfaction with care at the [facility name]. If you have already
completed and returned the survey, please accept my sincere thanks. If
not, would you please do so today. In order that the results will truly
represent the opinions of all family members, it is important that each
questionnaire be completed and returned. Only through your help can
the success of this study be ensured.
If by some chance you did not receive the questionnaire, or it has been
misplaced, please call me at [phone number] and you will be mailed
another one today.
Thank you for your help.
Sincerely,
Administrator’s name
First Replacement Questionnaire
(Fourth Contact)



Send 2 weeks after thank you / reminder
Only sent to non responders (need to track)
Cover letter - more persuasive/insisting tone








Questionnaire not yet received
Others have responded/importance to study (representative)
Not eligible? (return questionnaire)
Confidential / voluntary (still return it unanswered)
Note of appreciation
Personally signed / P.S. (any questions)
Replacement questionnaire (with ID#)
Return postage paid envelope - but no token
Follow-up Letter (on Letterhead Paper)
Insert Mailing Date
Insert Caregiver’s Name
Insert Caregiver’s Address
Dear Insert Caregiver’s Name:
About 3 weeks ago, you were sent a letter and questionnaire about
family member satisfaction with care at the [facility name]. As of today,
we have not yet received your completed questionnaire.
I am writing to you again because of the importance each questionnaire
has to the usefulness of this study. In order for the results of this survey
to be truly valuable, it is important that each questionnaire be completed
and returned.
In the event that your questionnaire has been misplaced, I have enclosed
a replacement for you.
Your cooperation is greatly appreciated.
Sincerely,
Administrator’s name
Special Contact
(Fifth Contact)


4 weeks later to non respondents [PK 2 wks]
Impact is due to a different “special” method
which shows respondent & study are important






Telephone call (encourage/send new one/complete)
Registered mail (requires signature)
Special delivery
Courier
More expensive follow-up
“New” cover letter, questionnaire (ID#), postage
paid envelope
Survey Error (Dillman)

Sampling error


Coverage error


Occurs when list does not include everyone
Measurement error


Precision is limited by the number surveyed
Results from poor wording and construction
Nonresponse error

When significant numbers don’t respond
Example Readings for Other Ideas

Increasing Response Rates to Postal
Questionnaires (Edwards et al. 2002)




Systematic review of RCTs
292 RCTs reviewed
75 strategies for influencing response
A Catalog of Biases in Questionnaires (Choi and
Pak, 2005)


Identified 48 common types of bias
Categorized into 3 main sources of bias
 Question design
 Questionnaire design
 Questionnaire administration
Problems/Issues

Undelivered questionnaires


People move/incomplete address or errors
Attempt to correct problems and resend


Handling respondent questions




New mail out dates – additional work
Should attempt to answer all questions
Should anticipate questions (e.g. ID#, no proxies…)
Thank them for asking
Evaluating early returns

Identify problems (e.g. printing, item response,
certain questions causing problems….)
Final Note About
Mail Survey Response Rates


Mail surveys often criticized for low response
Example Response Rates (PK using “Dillman approach”)

Surveys of physicians




Surveys of other health professionals



Physician Survey (elder abuse) 65%
Physician Survey (pneumonia) 77%
LTC Medical Director Survey 61%
LTC Director of Care Survey 76%
Canadian Optometrist Survey 90%
Other “general population” health surveys


Contact Lens Use Survey 93%
LTC Family Member Satisfaction Surveys (80-86%)