February 17, 2016
Catherine R. Messina PhD

― Claude Lévi-Strauss (1908-2009), French anthropologist

 No matter how good your study design, your sampling strategy, your response/participation rate, etc …………
The quality of your measurement instruments will directly impact your ability to draw meaningful conclusions about relationships studied

What is a survey or questionnaire?
…….. a systematic method for collecting information from a sample of individuals for the purpose of quantitatively describing the attributes of the larger population of which the sample are members.

Groves, Fowler, Couper, Lepkowski, Singer and Tourangeau, 2004.
What is a good survey or questionnaire?
…….. One that yields responses that are reliable (i.e., consistent) and valid (i.e., measures what it’s supposed to measure) measurements of what you are trying to describe


Benefits of surveys
Can reach a large number of people relatively easily
 Provide quantifiable answers
 Relatively easy to analyse
Drawbacks of surveys
 Provides only limited insight into problem
 Information obtained is limited by questions asked

 Always possible that wrong questions are asked
Varying response – respondent may misunderstand or misinterpret questions
 Need to get it right first time
 Can’t chase after missing data – or go back and try again

Making inferences from survey data
Characteristic(s) (attribute(s)) of Population
You infer….
Characteristic(s) (attribute(s)) of Respondent
Characteristic(s) (attribute(s)) of Sample
You infer….
Respondent answers question(s)

 Measurement is a systematic and repeatable process that quantifies or classifies events or characteristics
 Measurement tools (i.e., surveys/questionnaires) need to yield measurements that are reproducible
(reliable) and accurate (valid)!
 Reliability - the property of reproducability
 does the measure consistently yield a similar finding?
 Validitydoes the measure really assess (i.e. measure) the concept/object that it is intended to?
How accurate is the measure?

 Measurement is represented by – 3 related concepts:
 observed value (aka the “measured value”)

 true value and measurement error
Observed value = true value + measurement error
 Well designed measurement tools (i.e., surveys) minimize measurement error so that the observed value is very close to the true value

 First, determine clearly what it is you want to measure in order to answer your research question(s) or hypotheses
 Be very clear about what you want to know
 Decisions about relevant variables are based on the research question (or hypothesis)
 Research question / hypothesis should be stated in measureable terms
 Define your survey sample

 Know your respondents!
What are their needs and capabilities regarding survey administration (e.g., literacy levels; access to internet, etc.)


 List all relevant variables!
 Dependent variables
 Independent variables
Consider Covariates ! - A secondary variable
( and of interest to the researcher) that is observed and which can affect the relationship between the IVs and the DV


Will gender or age affect outcomes?
Parent education? Family size? Family income? Insurance? Co-morbid conditions?
Gestational age? ETC………

Identifying Existing Measures
 Next, determine if questions that you want to include have already been developed ( and tested) by other researchers
 Review the literature!!!!… you do not have to reinvent the wheel!
 Saves development time and can facilitate comparisons with other studies

Identifying Existing Measures





Use standardized measures as available
Review the literature for standardized measures
Look for validity and reliability information, appropriate to your sample
Usually not a good idea to “adapt” or change – but if you do, describe this is your Methods section and note this as a potential limitation in Discussion section
 Be cautious if using only a subset of questions from an existing survey as this may change the meaning of the summary scores. Check whether a shorter version of the instrument exists that has also been validated
Re: primary research articles – if questions not included look for on-line supplemental info

What if there are no existing measures on my topic, and I need to develop my own survey or questionnaire?

Developing Surveys – General Approach
 Determine the mode of survey administration: self-administered or interviewer administered
 Each has its own advantages and disadvantages ……
 and will determine your survey format

Selfadministered (e.g. “paper and pencil”)
• Hand out
• Mailed
• Email / web-based
Interviewer administered
• Face to face
• telephone

 Advantages:
 Relatively inexpensive and easy to administer
 Preserves confidentiality (can be anonymous)


Can be completed at respondent's convenience
No influence by interviewer
 Disadvantages:
 Low response rate – especially email surveys which sift to bottom of inbox very quickly
 Questions can be misunderstood
 No control by interviewer

 Advantages:
 Facilitates participation by people with low literacy
 Interviewer can clarify ambiguity
 Disadvantages:



May introduce interviewer bias
Needs more resources – more expensive
Difficult for sensitive issues

Developing Surveys – General Approach
 Decided on operational definitions
 How the researcher chooses to measure a particular variable
 E.g.: your child’s age – can be operationally defined as”
 DOB
 How old your child was on her/his last birthday
(in years)
 What age group best describes your child
 Use measure that offers most flexibility

Developing Surveys – General Approach
 Consider using standard demographic questions
 Facilitates comparing your sample to others in the literature
 Examples from national health surveys
 Look at the way these are asked in similar studies
 Only ask for necessary demo information
 Race / ethnicity – always should be comprised of two questions (NIH format):
1.
2.
Are you Hispanic or Latino? (yes vs. no)
What is your race? (allow for multiple answers)

Developing Surveys – General Approach
 Create a pool of new questions or revise existing questions – write more questions than will be included in the final draft
 Put questions in sequence
 Group questions that are similar


Put them in a logical order
Place demographic questions at the beginning


Put any sensitive or difficult questions at the end
Put any open-ended questions at the end

Developing Surveys – General Approach
 Format survey draft




Give your questionnaire a topic-related title
Include a brief explanation of the survey’s purpose
(short paragraph)
This includes all instructions for filling out survey and possible skip patterns!
Include appropriate white space to get a good estimate of the length

Developing Surveys
 If possible, ask experts to review for content
 Content validity: do the items in a questionnaire adequately represent the universe of items relating to a specific construct? How accurately does a measurement instrument tap into the various aspects of a specific construct?

Developing Surveys
 Get feedback (e.g., advisor; peers) on your 1 st draft
 Face validity: is the measurement logical on the
“face of it”? The degree to which the purpose of a measurement instrument is obvious to those using it or responding to it, i.e., “looks like” or appears to respondents as a measure of depression.


Advantage of good face validity: respondent really understands the point of the questions and interprets them accurately
Disadvantage of good face validity: respondent really understands the point of the questions and
“tailors” their responses to “fake good” or “fake bad”


Developing Surveys
Get feedback (e.g., advisor; peers) on your 1 st draft – con’t





Which questions measure IV(s)?
Which questions measure DV(s)?
Which questions measure covariate(s)?
Are questions clear and “answerable”?
Are any questions potentially offensive?


Does format make sense? Is it logical?
Length of survey?
 REVISE!!
 Get more feedback (e.g., advisor; peers) on your revised draft

Developing Surveys







Pilot-test on a small sample of respondents who are similar to your study sample
Are questions clear and “answerable”?
Are any questions potentially offensive? Questions should be culturally sensistive!!
Does format make sense? Is it logical?
 Length of survey and how long to complete? (consider participant burden)
Cognitive interviewing and “think aloud”
 Ask a few people in pilot sample to read the questions out loud to you and then explain to you, what each question means to him/her and how he/she thought about their answer
Test-drive your coding system – prelim analyses
REVISE!

Developing Questions
 If possible, pilot-test the near final draft of the survey with new pilot sample
 Pilot-test all procedures and logistics for administering survey
 Self or interviewer administered?
 Who will administer?
 Who is responsible for collecting completed surveys?
 Where will these be stored before pick-up?

Developing Surveys – Specifics

Characteristics of effective survey questions
Your goal is to write survey questions that :
• Consistently understood – i.e., all respondents will interpret the same way (and interpret in the way that you interpret them)
• Respondents are able to respond accurately – all respondents have the information to answer the question (except in cases of knowledge questions)
• Respondents are willing to answer

Writing good survey questions



Only include questions you plan to use
 Too many questions (especially if they seem unrelated to the purpose of your survey) can irritate respondents (an IRB issue too)
Avoid lengthy or complex questions – keep it simple
Write at a low reading level – understand the literacy level of your respondents
 Use conventional language!
 Use words relevant to respondents - terms and concepts should be familiar to respondents and easy for them to understand
 Avoid jargon and acronyms

Writing good survey questions
To ensure that respondents consistently understand your questions:
 Provide clear and concrete definitions of key terms
 Sugary drinks include soda, sweetened ice teas, all fruit juices and fruit drinks
 Provide all needed information so that respondents can answer the question
 E.g., Not counting any sugary drinks your child may have been served outside your home, in the past week how many times did you serve surgery drinks at home.
 Instructions need to be placed exactly where needed - not at the beginning of the entire survey.
 Use italics or bolding or underline to emphasize instructions and directions

Writing good survey questions
Write questions that respondents are able to respond to accurately:
 E.g., Survey examining video game exposure among children:
 In a typical weekday, how many hours does your child spend playing video games at home?
 0
 1- 3
 4-6
 more than 6
WHAT DOES THE “0” RESPONSE MEAN?

Writing good survey questions
Write questions that respondents are able to respond to accurately:
 DON’T MAKE ASSUMPTIONS!
 Do you have a video game player at home?
 No (skip next question and go to question X)
 Yes
 If yes then, in a typical weekday, how many hours does your child spend playing video games at home?
 0
 1- 3
 4-6
 more than 6

Writing good survey questions

Avoid “and” “or” (double-barreled) questions
 Trying to respond to a question measuring two or more ideas is confusing to respondent and their response is difficult for the researcher to interpret
 E.g., Do you think that increasing physical activity and
(or) losing weight is beneficial to you child’s health?
 E.g., Do you think that eye contact and (or) using words you understand improve communication between you and your child’s doctor
 E.g., Were you satisfied with the timeliness and quality of the health care service your child received?

Writing good survey questions
 Avoid leading questions
 Do you think that the food in the hospital made your child sick?
 Do you agree that the hospital staff were overworked?

Writing good survey questions
 Avoid bias by using both positive and negative sides in the question stem
“To what extent do you agree….” vs.
“ To what extent do you agree or disagree with this statement…. “

Writing good survey questions
 Avoid vague quantifiers
How often did you use aspirin for pain relief in the last month?
 Never
 Rarely
 Occasionally
 Regularly
 Vs……..

Writing good survey questions
How often did you use aspirin for pain relief in the last month?
 Not at all
 About once in the last month
 Two or three times in the last month
 About once a week
 More than once a week

Writing good survey questions
 Ask questions that people are willing to answer!


Do not be too personal or risky unless absolutely needed
Be culturally sensitive
 People are more likely to answer questions about behaviors and health compared to income or education
 Using income categories rather than asking for exact $ amounts is a better strategy

Writing good survey questions
 Ordering questions
 Two key considerations:
 Questions should be ordered such that the response to one question does not influence the response to the question that follows it
 Questions should be ordered such that respondents are more likely to complete the survey
 Fact-based questions before opinion based questions
 Start with non-threatening or least sensitive questions
 Most important questions should be placed first
 Vary question format to reduce likelihood that respondents will “auto-pilot” response sets

Writing good survey questions
 Good questions follow comfortably from the previous question.
 Requires good writing skills
 Smooth transitions between questions or groups of questions
 Group similar questions together
 Avoid non-response (i.e., missing data) by including response options for “Don’t Know” and “Refused” or “Prefer not to answer”

Writing good survey questions
 Recall of past events – people have limited recall of past events (this will vary with type of event)
 Improve recall by providing reasonable prompts
 Past year
 Past 6 months
 Past 3 months
 Past 2 weeks
 Use addition prompts, if needed:


“What season was it when your child was last vaccinated?”
“Was it near a holiday or someone’s birthday?”

Two main question formats
 Closed format
 forced choice
Yes
No
Don’t know 


Always

Sometimes
Never

 Open format
 free text

Please describe your child’s most distressing symptom?
________________________________________________
________________________________________________

Close-Ended Response Formats
 the Likert-type response format
“Indicate the strength of your agreement for each of the following statements, on a scale from 1 to 5, where a response of 1 indicates that you strongly disagree, a response of 5 indicates that you strongly agree, and 3 indicates that you are uncertain.”


 Odd number of categories is preferred – captures respondents who are indifferent or undecided
Reduces non-response
No more than 5 response options

 Semantic Differential Scale - used to assess the meaning of a variable to the respondent
 uses a pair of discrete descriptor words, which name opposing positions, as anchors for the response scale
 Everything in between represents the continuum of choices between the 2 anchors
Very satisfied __I__I__I__I__I__I__I__I__ Very dissatisfied
Very confident __I__I__I__I__I__I__I__I__ Not at all confident
Strongly agree __I__I__I__I__I__I__I__I__ Strongly disagree
NOTE: odd number of spaces

Rating Scales - used to assess the meaning of a variable to the respondent
 Consider a scale from 1 to 10, where 10 represents the best your child’s health has been in the past month and 0 represents the worst your child’s health has been in the past month – how would you rate your child’s health today.
 DESIGN SURVEY ITEMS WITH AN UNDERSTANDING OF
HOW YOU PLAN TO ANALYZE YOUR DATA!!!!!

 Advantages:
 Simple and quick
 Easy to code, record, analyse
 Easy to compare
 Easy to report results
 Disadvantages:


Restricted number of possible answers
Loss of information

 Respondents answer with their own ideas, opinions, language


What is you biggest concern about your child’s condition?
Please list your three most important barriers to breast feeding your baby:
 What difficulties have you encountered while trying to feed your child more fruits and vegetables?
 Do you have any additional comments about breastfeeding that you would like to share?

 Advantages
 Rich and detailed response – respondents are not constrained by researchers choices
 Unexpected responses possible
 Good for exploring knowledge and attitudes
 Disadvantages
 Cognitively demanding for respondent and researcher (coding is challenging and time intensive)
 Interviewer bias
 Difficult to analyse
 Difficult to compare groups

Question content
 Asking questions about:
 Knowledge
 Attitudes and beliefs
 Behavior

Measuring Knowledge
 True/False and multiple choice “fact” questions
 Open-ended response formats
 E.g., list all of the problems related to X that come to mind
 Close-ended response formats (response options are provided by the survey)
 Quicker, less burden compared to open-ended
 Less detail but easier to code and analyze
 If you plan to intervene on knowledge, include questions at pre and post-test that measure knowledge specifically addressed by intervention

Measuring Attitudes and Beliefs (subjective states)


Close-ended response formats are very common when measuring attitudes / beliefs
Attitude questions can be standalone items but …
 Responses often combined (i.e., summed) as a
“Scale” which means that many items are used to assess a single attitude or belief
 often using the same response format… e.g. strongly agree …… strongly disagree



Self report : “I did not smoke any cigarettes today.”
 By phone, mail, computer, interview
Observation Report of respondent’s behavior by others (family member, health professional)

“I saw him/her smoke cigarettes today.”





Consider using surveys that you think are attractive as a template
Professional looking format good use of “white space”


Space response scales widely enough so that it is easy to circle or check the correct answer without the mark accidentally including the answer above or below.
Open-ended questions: the space for the response should be big enough to allow respondents with large handwriting to write comfortably in the space.
Closed-ended questions: line up answers vertically and precede them with boxes or brackets to check, or by numbers to circle, rather than open blanks.
Use larger font size (e.g., 14) and high contrast (black on white).


