Survey Design
So Many Surveys

Political ‘push polls’
–

Marketing polls
–

Trying to influence, not collect data
Some are just advertisements
News Polls
How to write a good survey

Think in terms of ‘questionnaire’, not
questions
–

How are questions related to each other?
Have a clear topic
–
–
A research question
Know what concepts you are measuring

–
Think in terms of hypotheses, IVs and DVs
Every question should have a clear reason for
inclusion
The Instrument – the whole package

Make it look/sound appealing
–
–

Clear directions that make it sound
interesting/important
shorter is always better (i.e. 15 minutes)
Consistency is good
–
Use similar response formats for questions
Introductions and Instructions


What is the study about?
Informed and Voluntary Consent
–


Institutional Review Boards
Contact information for researcher
Let them know they can have the results
Example of Instructions
This survey is part of a research project about your parish directed by Dr. Matthew Loveland of Le
Moyne College. Completing the survey indicates your voluntary, informed consent to participate
in the project. Your responses are anonymous and confidential. If you are under 18, please do
not complete the survey. Please circle the number to indicate your answer.
1. How often do you attend mass at this parish?
1. Daily
2. Weekly
3. Few times a month
4. Few times a year
5. I’m a visitor today
6. I don’t know
Thanks for your participation. More information about the research project can be found at:
www.lemoyne.edu/parishproject. With questions, call Dr. Loveland at 315 445 5462, or by Email
at lovelam@lemoyne.edu
Question Order

Early questions make identities salient
–
Example: Political ID and Political Issues


Early questions should not be threatening
–
–

How often do you watch pornography?
Have you ever been convicted of a crime?
Early questions should not be boring
–

So, Put space between the questions
How old are you?
Of course, this all depends
–
–
Different survey settings call for different strategies
It’s a judgment call
Bad question order example
(and probing)
1. Are you of the Catholic faith?
1. Yes
2. No (end)
3. Not sure (Do Not Read) (end)
2. Aside from weddings and funerals, how often do you attend Mass? (Do not Read Responses, Use Categories as
Probes if Necessary)
1. Never (skip to question 9)
2. once a year or less
3. only on religious holidays
4. several times a year
5. once a month
6. 2 to 3 times a month
7. Every week
8. More than once a week
9. Not sure
Do previous questions
influence answers?
The following set of items asks about your attitudes about church leadership. For each of these statements, please
tell me whether you strongly agree, somewhat agree, somewhat disagree, strongly disagree, or have no opinion.
[Read No Opinion/ Don’t Know]
27. Pope Benedict is doing a good job leading the Church
1. Strongly agree
2. Somewhat agree
3. Somewhat disagree 4. Strongly disagree 5. No Opinion/DK
28. The American bishops are doing a good job leading the U.S. Church.
1. Strongly agree
2. Somewhat agree
3. Somewhat disagree 4. Strongly disagree 5. No Opinion/DK
29. [Skip if q24=1]The pastor of the parish I attend is doing a good job leading our congregation.
1. Strongly agree
2. Somewhat agree
3. Somewhat disagree 4. Strongly disagree 5. No Opinion/DK
30. I think it would be a good idea if parishes were to choose their own priest from among available ordained priests.
1. Strongly agree
2. Somewhat agree
3. Somewhat disagree 4. Strongly disagree 5. No Opinion/DK
Social Desirability

Sensitive Topics
–


sex, drugs, crime, embarrassment, church
attendance, etc….
People will sometimes tell you who they wish
they were, not who they really are
Advice: write questions so that undesirable
answers are made to sound ok.
Social Desirability Example
1. How often do you attend mass at this parish?
1
2
3
4
5
6
Daily
Weekly
Few times a month
Few times a year
I’m a visitor today
I don’t know
Open and Closed Ended



Closed ended: give responses
Open ended: let respondent give comments
If closed:
–
Mutually exclusive and exhaustive


Allow don’t know / not sure ?
If open:
–
–
Will have to code after data collection
Might effect length of interview

Could be quicker or slower
Open vs. Closed Example
•
How many people do you consider close
friends? ________
•
How many people do you consider close
friends?
1.
2.
3.
4.
0
1 to 5
6 to 10
More than 10
Avoid Double Barreled Questions

A single question that really asks two
separate things.
–
–

Watch out for this if you use the word “and”
Try not to preface any question with something
people might disagree with
Do you think the United States government
should raise taxes and spend more money
on education?
Double Barreled
The government should guarantee jobs to insure a basic standard of living
1. Strongly Agree 2. Agree 3. Neutral 4. Disagree 5. Strongly Disagree
Skip Patterns

Contingency Questions
–
Answer to a question sends you down a different
path



If yes, go to question number X
Use it to shorten survey for respondent
Too many can get complicated
–
Use with caution on a self administered survey
Skip Pattern Example
These questions are about your previous place of worship. If you do not have a previous place of
worship, you may skip to question # 14.
11. How long were you involved at your previous place of worship?
1.
2.
3.
4.
5.
6.
Less than 5 years
5 to 10 years
11 to 20 years
21 to 40 years
More than 40 years
No prior place
Negative or Biased Terms


Negative terms may be confusing
The death penalty should be illegal
–
–

SDA DA N A SA
What does it mean if you disagree?
Avoid Biased Terms
–
–
Biased questions may encourage a certain
answer
Depends on connotation of words
“Forbid” versus “Not Allow”



Do you think the U.S. should forbid public
speeches against democracy?
Do you think the U.S. should allow public
speeches against democracy?
“Americans are much more willing to not
allow speeches than they are to forbid them.”
Some general tips

If asking about the past, use a time period
people can remember
–

Only ask about personal experiences
–

“In the last week…”
Don’t ask about other people’s impressions
Beware of hidden contingencies
–
In the past month, have you crossed the street to
avoid someone you found frightening?
Response Rate



% of those sampled who participated
Larger RR usually means more
representative data
General Guidelines, no clear rule
–
–
–
Shoot for 70%
Happy with 50%
Don’t be surprised with 30% or lower
Methods of Administration
– Mail Survey

Self Administered
–
Relies on Participant Enthusiasm




Low Response Rates
Appearance is Most important in this case
Give an Incentive?
Slow




Send a prior letter, maybe call
Send the survey
Send a reminder letter to those who haven’t replied
Contact again
Methods of Administration
–Phone Survey

Random Digit Dial (RDD)
–
–


Very Fast
Very Expensive
Participation gets worse every year
Cooperation Rates
–
–
20% to 30%, or lower
Statistically, in 30% range is not that different
from in 60% range as long as sample is
representative
Methods of Administration
-In person interview

Best response rate & Most Expensive
–
–

Well trained interviewers
–
–

Contact ahead of time to arrange meeting
Sit with respondent while collecting data
Dress, Race, Gender, etc.
Threatening Situations
Mixed Methods
–
–
Computer Assisted
Self administered portions of an in person interview
Methods of Administration
-Web Survey



Still a new method, very few clear guidelines
Visually, I’d say follow same guidelines as a
a mail survey
How do you get your sample?
–

Do you trust online identities?
Can be very affordable
–
Survey Monkey
General comments on survey error


Sample surveys are way to generate
estimates
There are multiple sources of error
–
–
–
Sampling error
Low response rates
Poorly written questions