Survey Research
Pierre-Auguste Renoir: Luncheon of the Boating Party, 1881 [D]
Topics Appropriate for Survey Research
1. Data Collection from Large Numbers
• Survey research is a relatively inexpensive
procedure for collecting information from a
large number of elements.
• They are efficient means of learning about
attitudes, intentions to act.
• Unfortunately, they are overused by marketing
and political organizations, to the point where
many persons refuse to respond to any type of
survey.
Issues Related to Survey Research
1. Survey Response Rate
• When survey response rates are low then
questions arise about how well the results can
be generalized to the population.
• What were the opinions of those elements
who did not respond?
• Various types of extrapolation procedures can
be used to estimate the responses of those
who did not complete the questionnaire, but no
procedure is widely accepted by the community
of scholars.
Issues Related to Survey Research
1. Survey Response Rate (Continued)
• A large volume of research addresses the issue
of how to improve survey response rates.
• Key features that affect response rates:
• The topic of the survey, sponsorship,
incentives, survey mode, time/day of
contact, question wording, question ordering,
formatting of the questionnaire, follow-ups,
interview style, interviewer style, envelope
style, cover letter, time to complete.
Issues Related to Survey Research
1. Survey Response Rate (Continued)
• The survey response rate is calculated as:
Number of completed questionnaires
(Number of attempted contacts
- Unavailable for contact)
• For university-sponsored research, survey
response rates of 50% are acceptable, 60% is
preferred, 70% is very good.
Issues Related to Survey Research
2. Item Response Rate
• The item response rate is the percentage of
responses to an item among all returned and
completed questionnaires.
• Questionnaires might be returned and
otherwise complete except for missing
responses to certain questions.
• Example: Responses to questions about
income can receive low response rates.
• Key features that affect item response rates:
• Topic, question wording, question ordering,
questionnaire formatting, interviewer style.
Guidelines for Asking Questions
1. Choose Appropriate Question Forms
• Open Ended Questions: Respondents are
asked to give information in their own words.
• Closed-Ended Questions: Respondents are
asked to state which response most closely
represents their answers to questions.
2. Respondents Must be Competent to Answer
• Typically, surveys are administered to adults.
• Sometimes, screening questions are needed to
insure that the respondent is qualified to
answer (i.e., “Do you live in the city limits of
Sappville?”).
Guidelines for Asking Questions
3. Respondents Must Be Willing to Answer
• Respondents must consent to complete the
survey and must be competent and legally able
to give their consent.
4. Questions Should be Relevant
• Questions should apply to most respondents.
• Questions should address the main topic of the
survey.
5. Short Items are Best
• Persons are more likely to understand and
respond to simple, short questions.
Guidelines for Asking Questions
6. The First Question
• Should directly address the main topic of the
survey, even if the survey covers sensitive
topics (e.g., sex, religiosity, employment).
• Attempts to “warm up” the subject with
background questions or ones not directly
related to the topic will give the impression of
being evasive, coy, misleading in purpose.
• Should apply to all potential respondents.
• Should be short and easy to answer.
Guidelines for Asking Questions
7. Question Wording
• See the: Question Wording website for detailed
directions and examples of how to avoid
common mistakes in writing survey questions.
General Questionnaire Format
1. Formats for Respondents
• The questionnaire should present an easy to
understand format for respondents.
• It should be clear about how to respond.
• The questionnaire should have a professional
appearance. Money spent on professional
formatting and graphics is money well spent.
• The response options should tend to flow down
the page to give the sense of completing the
questionnaire quickly.
General Questionnaire Format
2. Contingency Questions
• These questions screen respondents for
characteristics and then direct them to the next
appropriate question on the survey.
3. Matrix Questions
• A lead in question is followed by a series of
statements that complete the question.
• Example: “How much do you trust...
1. Federal legislators.
2. State legislators.
3. Local legislators.
General Questionnaire Format
4. Ordering Items in a Questionnaire
• The ordering should flow logically from one
topic to the next.
• Question ordering affects responses. It is
difficult to avoid this bias altogether.
5. Questionnaire Instructions
• A self-administered questionnaire will need to
be especially easy to understand.
• Short introductions help the respondent grasp
the intent of the questions that follow.
• Providing opportunities to give additional
information helps gain validity and responses.
General Questionnaire Format
6. Pretesting Questionnaire
• Pretesting helps uncover many flaws types of
flaws that can occur when writing a
questionnaire.
• Pretesting will reveal the approximate time it
takes to complete the questionnaire.
The Mailed Survey
1. Introduction
• Questionnaires are mailed to potential
respondents and returned by mail.
• Business reply envelopes can be used for the
return of the questionnaire.
• The questionnaire is accompanied by a cover
letter.
• Sometimes, incentives are used to increase the
response rate.
The Mailed Survey
2. Comparison with Other Modes
 Least expensive of traditional surveys.
 Best mode for collecting sensitive, extensive, or
complex information.
 Must have strong content validity of items.
 Most time to gather data.
 Requires entry of data after collection.
 Does not allow for clarification or probing.
 Does not allow for editing questions.
 Must have very clear directions.
 No control over who completes the
questionnaire.
The Mailed Survey
3. The Total Design Method
• The Total Design Method, developed by Don
Dillman, has been very effective in improving
survey and item response rates.
• The TDM includes a wide range of guidelines
for questionnaire development and survey
administration.
• One key feature of the TDM is the suggested
guideline for administering follow-up
questionnaires in a mailed survey.
The Mailed Survey
3. The Total Design Method (Continued)
• The TDM approach to follow-ups:
Day 1: Mail questionnaire and cover letter.
Day 7: Mail a reminder postcard to all elements.
Day 21: Mail a replacement questionnaire and cover
letter to those elements who have not yet
replied.
Day 49: Mail a replacement questionnaire by
certified mail and cover letter to those
elements who have not yet replied.
The Mailed Survey
3. The Total Design Method (Continued)
• The TDM approach to follow-ups increases the
cost of conducting a survey.
• Be prepared to hear complaints from elements,
especially after the fourth mailing (See: Angry
Letter).
• Instructor’s Note: I feel justified in using the
TDM approach to follow-ups when I am
conducting research with taxpayer’s money
because I think that elements have a
responsibility to reply on behalf of other
members of the society.
The Telephone Survey
1. Introduction
• The preferred method for marketing research
and political polling.
• The direct contact and rapid data acquisition of
a telephone survey are well suited to collection
of information on a few, easy to understand
items.
• Telephone surveying is not restricted by the “Do
Not Call” registry.
The Telephone Survey
2. Comparison with Other Modes
 Best mode for collecting a few, easy to answer
questions in a short time frame.
 Does not require entry of data after collection
(Computer Assisted Telephone Interviewing).
 Allows for clarification or probing.
 Allows for editing questions.
 More expensive than a mailed survey.
 Questions cannot be overly complicated and
must be answered with knowledge at hand.
 Requires trained interviewers.
The Personal Interview
1. Introduction
• There is no substitute for a personal interview
when the research requires much in-depth
information from elements.
• Personal interviewing is an excellent mode for
exploratory research.
• Personal interviewing allows the researcher to
collect additional information not included in the
interview itself: body language, setting,
condition of the home, nuances of meaning,
personal contact with hard-to-reach elements.
The Personal Interview
2. Comparison with Other Modes
 Best mode for collecting in-depth responses.
 Can collect information other than what is
asked on the questionnaire.
 Allows for clarification or probing.
 Allows for editing questions.
 Allows for unstandardized interviewing.
 The only reasonable procedure for collecting
data from some elements.
 Most expensive type of survey.
 Requires trained interviewers.
 Data entry after collection might be required.
The Internet Survey
1. Introduction
• This method is gaining in popularity as more
persons have home computers and access to
the internet.
• Internet surveys can combine some of the
advantages of mailed and telephone surveys.
• The quality and affordability of software
packages is improving rapidly.
The Internet Survey
2. Comparison with Other Modes
 Even less expensive than mailed surveys.
 Can ask many questions.
 Can ask detailed information.
 Respondents can complete the survey at their
own pace and at a time convenient to them.
 Allows for randomization of questions by
respondent.
 Immediate data entry.
 Still unfamiliar to many respondents.
 Not a good way to contact older or less affluent
adults.
New Technologies
1. Introduction
• CAPI: Computer assisted personal interviews.
• Computer is used to collect data in person.
• CASI: Computer assisted self-interviews.
• Researcher brings the computer to the
person’s location. The person completes the
questionnaire on the computer.
• CASQ: Computer self-administered
questionnaire. The person completes the
questionnaire online.
New Technologies
1. Introduction (Continued)
• TDE: Touchtone data entry. The person calls
telephone number and responds to a
computerized interviewer with entries on the
telephone key pad.
• VR: Voice recognition. The person calls a
telephone number and responds by voice.
• WWW Surveys: Person enters or links to a URL
and responds to questions on a web-based
questionnaire.
New Technologies
2. Comparison with Other Modes
 Can be very inexpensive.
 Subjects respond when convenient.
 Data seem as valid as traditional modes.
 Do not need to train interviewers.
 Becoming more accepted by the public.
 Ideal for some audiences.
 No need for data entry after collection.
 Concerns about representativeness.
 Requires specialized software.
Secondary Analysis
1. Introduction
• Analysis of data from existing surveys.
• Typically, this term refers to analysis of largescale, government sponsored surveys.
• Various censuses of the population.
• National Longitudinal Survey
• National Survey of Families and Households
• General Social Survey
• National Health and Nutrition Examination
Survey
• Many others....
Secondary Analysis
2. Comparison with Other Modes
 Large number of subjects.
 Longitudinal studies.
 Data available to many researchers.
 Expert data collection procedures.
 Take questions as they come.
 Questions over time do not always match.
 Can be slow to adopt new theories and
concepts.