This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike License. Your use of this
material constitutes acceptance of that license and the conditions of use of materials on this site.
Copyright 2012, The Johns Hopkins University and Freya Sonenstein. All rights reserved. Use of these materials
permitted only in accordance with license rights granted. Materials provided “AS IS”; no representations or
warranties provided. User assumes all responsibility for use, and all liability related thereto, and must independently
review all materials for accuracy and efficacy. May contain materials owned by others. User is responsible for
obtaining permissions for use from third parties as needed.
ISSUES IN SURVEY
RESEARCH
Measurement
Measures Depends on What is Focus
Status Characteristics
Accounts of Behavior or Events
Attitudes, Values or Opinions
Accounts of Emotional States
Status Characteristics
•  Usually Nominal Measures (mutually exclusive
categories)
•  Race/Ethnicity Example
–  Measured categorically—nominal scale
–  How many dimensions?
•  White, Black, Other
•  Hispanic Origin
–  Mixed race or ethnicity
•  Varied Effects of Offering Different Responses
Accounts of Behavior/Events
•  Contraception example
–  At last intercourse
–  Frequency?
•  Sometimes, always, never
•  Number of Condoms/Number of Acts of Intercourse
•  Proportion of times used
•  Partner by Partner
•  Services received by clients
Avoid Vague Questions
•  How many times have you seen or heard
gunshots?
–  Never
–  Once
–  A few times
–  A lot of times
More Precise Response
Categories
 How many times have you ever seen or
heard gunshots?
 ____________ (leave continuous/open-ended- 0
to…)
Response Categories
•  How often did you hear guns shots during
the past 12 months? ”
–  Not at all
No Times
–  1-2 times
1-5
–  About once a month
6-10
–  About two to three times per month 11-20
–  About once a week
21-50
–  More than once a week
51 or
more
Sensitive Questions
•  Information requested is personal (e.g. income)
•  Words convey implication about which
respondents have negative feelings
•  Questions that are incriminating (e.g stealing)
•  Can use self administered data collection mode
but there are also other ways to address
Use of Series of Questions
Have you ever shoplifted anything?
1. Yes
2. No
Revision:
As you may know, there is a great deal of discussion
about shoplifting in this community, and questions as to
how it should be handled. Some people feel it is a
serious problem about which something should be done.
Others feel that it is not a serious problem. How about
yourself? Do you consider shoplifting to be a serious,
moderate, slight or not a problem in our community?
1. Serious 2. Moderate 3.Slight 4.Not at all serious
During the past few years, do you think that the
frequency of shoplifting has increased, stayed about
the same or decreased in this community?
1. Increased
2. Stayed about the same
3. Decreased
Do you recall personally knowing anyone who took
something from a store without paying for it?
1. Yes
2. No
How about yourself? Did you ever seriously
consider taking something from a store without
paying for it?
1. Yes
2. No
If you said yes, did you actually take it?
1. Yes
2. No
Other Methods for Objectionable
Questions
•  Everybody does it- Even the calmest parent
get angry with their children some time. Did
your child do anything in the past seven
days that made you angry?
•  Assume the behavior- How many cigarettes
do you smoke each day?
•  Use authority to justify behavior- Many
doctors now think that drinking wine
reduces heart attack, have you drunk any
wine in the past year?
•  Casual approach- Did you happen to…
•  Reasons why not-Many drivers report that
wearing seat belts is uncomfortable and
makes it difficult to reach switches such as
lights and windshield wipers. Thinking
about the last time you got into a car, did
you use a seat belt?
Other Method for Sensitive
Question
•  Randomized response technique
–  Respondents randomly assigned to sensitive or
a non sensitive question
–  Interviewer does not know which question the
respondent is answering
–  Example in book : ever arrested for drunk
driving vs is your birthday in June
–  Calculation of point estimated based on known
probability of getting the question (50%) and
known probability of being born in June
So How do You Decide
•  What level of precision does your research
objectives require?
•  What level of precision is your respondent
able to provide reliably?
–  How complicated is the task?
•  Shorter questions do not necessarily take
less time to answer– how many takes
calculation on the part of the respondent
Recall Difficulty
•  Ways to improve reporting of dates &
estimation?
–  Bounding
–  Length of reference period
–  Use of salient events to aid recall
–  Use of event history calendar
Event History Calendar
Opinions/Values/Attitudes:
•  Positive to Negative dispositions towards
objects eg.
–  Policies, programs
–  Certain types of people
–  Behaviors
–  Ideals
–  Etc.
Why Start from Scratch?
Another Resource
•  Health & Psychosocial Instruments (HaPI)
http://www.welch.jhu.edu/
1.  Need the appropriate key words –eg
desirability
2.  Check literature for psychometric properties
3.  Check literature for populations asked
4.  Ask the creator for the instrument
Title Marlowe-Crowne Social Desirability Scale--Short Form--1(10).
Acronym M-C 1(10)
Instrument Author Strahan, Robert F.; Gerbasi, Kathleen Carrese.
Source Code Secondary Source
Source Cella, D. F., Tulsky, D. S., Gray, G., et al . (1993). The Functional
Assessment of Cancer Therapy Scale: Development and validation of the general
measure. Journal of Clinical Oncology, 11(3), 570-579.
Language English
Measure Descriptors Need for Approval*..Social Desirability*
Sample Descriptors Adult Adulthood ..Aged Breast Neoplasms
..Colorectal Neoplasms .Female .Head and Neck Neoplasms
..Hospitalized Patients ..Human Females Human Males .Inpatients
..Lung Neoplasms..Outpatients
..United States
References Strahan, R., & Gerbasi, K. C. (1972). Short, homogeneous versions of
the Marlowe-Crowne Social Desirability Scale. Journal of Clinical Psychology,
28(2), 191-193.
Scaling
•  Scaling evolved to measure "unmeasurable"
constructs
•  Aims to assign a single number representing a
person's overall attitude or belief toward some
“object .”
•  Uses multiple items to triangulate on a general
disposition towards some “object”
Source: Research Methods Knowledge Base
http://www.socialresearchmethods.net/kb/scallik.php
Steps to creating a Likert Scale
1.  Define the Focus. it is assumed that the concept you want to measure
is one-dimensional in nature
2.  Create the set of poten/al scale items 80-­‐100 • To be rated on a 1-­‐to-­‐5 or 1-­‐to-­‐7 Disagree-­‐Agree • Brainstorming: a good source? 3. Rating of Items by a group of judges. Usually a 1-to-5 rating scale
where:
1= strongly unfavorable , 2= somewhat unfavorable, 3= undecided
4= somewhat favorable 5 = strongly favorable to the concept
4. Compute the intercorrelations between all pairs of items, based on the
ratings of the judges.
5. Discard items that have a low correlation with the total (summed)
score across all items
6. For each item, get the average rating for the top quarter of judges and
the bottom quarter. Then, do a t-test of the differences between the
mean value for the item for the top and bottom quarter judges. Higher
t score = better discriminator items
Issues
•
•
•
•
•
Bipolar versus unipolar : midpoint?
Number of categories
Category labels
Don’t know filters
Acquiescence
Bipolar vs. Unipolar
Assumptions about the continuum is needed
–  Literature
–  Knowledge
–  Question of interest
Assumptions must be made about midpoint (Bipolar)
–  Omit- assume that respondents have a position but
lack intensity
–  Omit- assume that it attracts individuals with no
opinion or “satisfier”
–  Include- assume that respondents do place themselves
in the middle, legitimate response
Bipolar – Midpoint Resolved
•  If direction in which people are leaning
is what’s need
– Don’t offer a middle position
•  If the desire is to sort out those
individuals with more definite
convictions on a issue
– Offer middle position or use different
approach
Number of Response
Categories
•
Opinion: Marijuana should be made legal in
the US.
•
Balance between increasing discrimination
potentially available with more categories VS. the
capacity of respondents to make distinctions reliably
VS. the content you want
Acquiescence
  Statements to be agreed or disagreed most common format
  Advantages: simple to construct & easy to answer
  Disadvantage: the tendency to agree irrespective of item
content.
  Consider your population
  Occurs disproportionately in less educated
  Some solutions
  Balancing - posing the same number of statements on
each side of an issue
  Force choice
  Include social desirability scale in a survey so that you
can later use it as a control variable or for other quality
control purposes
Multiple Question Sequence:
Addresses Acquiescence
•  1. Do you have an opinion about whether
marijuana should be made legal in the US?
2. Do you favor legalizing the use of
marijuana?
3. How strongly do you feel about legalizing
(not legalizing) marijuana?
Number of Categories
  Use five to nine categories.
  Always remember to label your categories but it is
important how you label them!
1.  Words & Numbers
2.  Smiley Faces
1
2
3
4
Not at all
1
5
Very
2
3
Very
  Mode is important in making a decision.
  Always keep your population in mind.
4
5
Not at all
Evaluating Your Question
•  Questions are vague
•  Avoid jargon
•  Data is too difficult to remember, calculate or
estimate
•  Wording that is misunderstood by the respondent
•  Objectionable Questions
•  Double barrel questions
•  Allowing response categories to overlap at the
boundaries
•  Responses do not match your question
•  Sequencing
Double Barrel Questions

Do you favor legalization of marijuana for use in
private homes but not public places?
1.
2.
Yes
No
Unfamiliar/Technical Words &
Question/Response Mismatch
•  When should you put on a condom?
–  Before ejaculation
–  After ejaculation
•  During the past 90 days, did you have
receptive anal sex with your partner?
–  Yes
–  No
Too Much Precision?
Readability
•  Flesch–Kincaid Grade Level
•  These readability tests are used extensively in the field of
education. The "Flesch–Kincaid Grade Level Formula"
translates the 0–100 score to a U.S. grade level, making it
easier for teachers, parents, librarians, and others to judge
the readability level of various books and texts. It can also
mean the number of years of education generally required
to understand this text, relevant when the formula results in
a number greater than 10.
Sequencing
•  Where should questions go, especially
sensitive ones?
–  End?
–  Beginning?
–  Middle?
•  What order should questions appear? –
questions at beginning can influence later
answers (KAB)