Chapter 7
Designing Surveys and Data Collection
Instruments
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Survey research
• Typically used in descriptive/conclusive research
• Answers questions such as Who? Where? When?
How? Why?
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Three Data Collection Methods in
Survey Research
• Questionnaires: by mail, email, internet
•
Advantages
•
i.
Inexpensive
•
ii.
Can be confidential or anonymous
•
iii.
Easy to score most items
•
iv.
Standardized items and procedures
•
•
b.
•
v.
Response rate may be small
•
vi.
Cannot probe or explain items
•
vii.
Only used by people who can read
Disadvantages
• Interviews
• Telephones
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Imagine this scenario!
• If a survey is done for Albertson’s stores and the data
show that 36% of the respondents rate Albertson’s
checkout as the best in town, this 36% figure should
be an accurate reflection of feelings among the
population at large.

True or False or Maybe
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Types of
Survey Error
Total
survey
error
Random
sampling
error
Measurement
error
Processing
error
Systematic
error
Surrogate
information
error
Interviewer
error
Response
error
Nonresponse
bias
Instrument
bias
Sample
design
error
Selection/
error
Frame/
coverage
error
Population
specification
error
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Sources of Survey Error
•
1.
Sampling error: The result of surveying only some, and not all,
elements of the survey population
•
2.
Coverage error: The result of not allowing all members of the
survey population have on equal or known nonzero chance of being
sampled for participation in the survey
•
3.
Measurement Error: The result of poor question wording or
questions being presented in such a way that inaccurate and
interpretable answers are obtained
•
4.
Nonresponse error/bias: The result of people who respond to a
survey being different from sampled individuals who do not respond, in
a way relevant to the study
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Types of Errors (contd.,):
Measurement Errors

Surrogate Information Error
- A discrepancy between the information
actually required and the information being
sought by the researcher.

Interviewer Error
- Interactions between the interviewer and
respondent may influence answers.

Instrument Bias
- Problems with the measurement instrument
or questionnaire.

Processing Error

Mistakes when entering information from the survey
documents.
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Instrument Bias
• Q: What is your income?
Response bias
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Mail Survey
• In December 2004, a mail survey was sent
out to 1000 managers of industrial equipment
to gauge their interest in a new product. 40
questionnaires were returned indicating a
response rate of 4%. 90% of these responses
were favorable toward the new equipment.
So, the marketing managers have decided to
launch the new product.
• What are your thoughts?
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Question
• In the past three months, how many times have you
eaten at fast food places?
• In the past three years, how many times have you
been involved in automobile accidents?
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Types of Errors: Measurement
Errors
• Non-response bias
Are those who did respond different from
those who did not respond?
• Response bias
- A tendency for people to answer a particular
question in a certain way
deliberate falsification
unconscious misrepresentation
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Function and Importance of
Questionnaires
• A questionnaire is a formalized method for
collecting data from respondents.
• questionnaires measure:



past purchase and usage behavior
attitudes and opinions, intentions, awareness and
knowledge, ownership
variety of respondent characteristics
• To avoid measurement error, questions must
pose realistic alternatives or trade-offs.
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Questionnaire Components
A. identification data, such as:



respondent's name
address
phone number
B. request for cooperation
C. instructions
D. information sought
(longest portion of the questionnaire)
E. classification data
(characteristics of the respondent, primarily
"geodemographic" data)
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
D: The blueprint
• Blueprint Table:.
• This table serves as an organizer that frames the
major content categories, attitudes, beliefs to be
assessed. The proportion of the tasks or items that
will be included on the instrument or overall
performance should correspond roughly with how
important the domain is relative to other domains.
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Example: Life Satisfaction survey
(100 items)
Number of items
Content Base Category
Emotions such as: Guilt, Anger,
Depression and Hopelessness,
and Anxiety
30
Cognitive Functioning
25
Decision Making
15
The Capacity for Pleasure
10
Movement Toward Goals
10
Self Care and Daily Functioning
5
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Questionnaire Design
Steps/Best practices in questionnaire design:
review
preliminar
y
considera
tions
decide
question
content
decide
response
format
decide
question
wording
decide
question
sequence
decide
physical
characteri
stics
pretest
and
revise
make
final
draft
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Questionnaire Design (cont.)
Preliminary Considerations
• review issues already decided by research design:






detailed listing of information needs
sources of data
definition of the target population
detailed sampling plan
specification of scales and communication media
visualization of the research findings
• link between data-collection and information needs

Each question should relate to a specific
information need.
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Questionnaire Design (cont.)
Question content
Consider ways question will not be answered
accurately or at all:
1. respondent unable to provide the data
2. respondent uninformed
3. respondent forgetful
4. respondent misremembers
5. respondent unwilling to respond accurately
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Response Format
1. open-ended questions – free-response, usually in
respondent's own words
2. multiple-response questions – requires choice from
an explicit list of options
3. dichotomous questions – offers choice of two
responses and often a third neutral alternative
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Question Types
What do you feel about buying Tide Detergent?
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
Do you own a(n) __________________ at home?
Personal Computer
VCR
Entertainment Center
Yes
Yes
Yes



No
No
No



What is your age? (Please check one)
Under 18
r
18 - 24
r
25 - 34
r
35- 44
r
45 - 54
r
55 - 64
r
65 and over
r
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Decide on Question Wording
• use simple language
• use unambiguous words





does the word truly convey what the researchers
intended?
can respondents extrapolate any alternative
meaning?
if so, does context help make the intended
meaning clear?
is there any word with similar pronunciation or
spelling that could be confused with it?
could we use a simpler word or phrase instead?
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Decide on Question Wording
(cont.)
• avoid leading questions
• avoid biasing questions
• avoid implicit alternatives
• avoid implicit assumptions
• avoid estimates
• avoid double-barreled questions
• consider frame of reference
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Decide on Question Sequence
• sequence questions to retain respondent
interest without introducing bias
• use intriguing, readily understood opening
question
• ask general questions first
• place more difficult questions in the middle of
the sequence
• position sensitive, threatening, and
demographic questions at the end
• arrange questions in logical order
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Questionnaire Design
Steps/Best practices in questionnaire design:
review
preliminar
y
considera
tions
decide
question
content
decide
response
format
decide
question
wording
decide
question
sequence
decide
physical
characteri
stics/
layout
pretest
and
revise
make
final
draft
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Issues to Address When Evaluating
the Questionnaire Layout
•Is each Question Necessary/ Is the Questionnaire Too Long?
• Will the Questions Provide the Desired Information to Accomplish the
Research Objectives?
• Look at the Appearance of Mail and Self-Administered Questionnaires:
-
Avoid a Cluttered Look
-
Allow Plenty of Space for Open-Ended Responses
-Instructions Printed Within The Questionnaire Should Be In Capital
Letters (or set apart in some manner)
PleaseTurn
Page
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Survey implementation
• First Contact: Prenotice Letter/Incentive
• Second Contact: The Questionnaire Mailout
• Third Contact: The Postcard Thank You/Reminder
• Fourth Contact: The First Replacement
questionnaire
• Fifth Contact: Invoking Special Procedures
•
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Observational Forms
(alternative to surveys)
• make explicit the types of observations to be made
• make explicit how observations are to be
measured
• design should flow logically from the listing of
information needs:




who is to be observed?
what is to be observed?
when is observation to be made?
where should observations be made?
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.
Acceptability
Totally unacceptable, Unacceptable, Slightly unacceptable, Neutral, Slightly acceptable, Acceptable,
Perfectly acceptable
Agreement
Strongly disagree, Disagree, Somewhat disagree, Neither agree or disagree, Somewhat agree, Agree, Strongly
agree
Amount of Use
Never use, Almost never, Occasionally/Sometimes, Almost every time, Frequently use
Appropriateness
Absolutely inappropriate, Inappropriate, Slightly inappropriate, Neutral, Slightly appropriate, Appropriate,
Absolutely appropriate
Awareness
Not at all aware, Slightly aware, Somewhat aware, Moderately aware, Extremely aware
Beliefs
Very untrue of what I believe, Untrue of what I believe, Somewhat untrue of what I believe, Neutral, Somewhat
true of what I believe, True of what I believe, Very true of what I believe
Concern
Not at all concerned, Slightly concerned, Somewhat concerned, Moderately concerned, Extremely concerned
Familiarity
Not at all familiar, Slightly familiar, Somewhat familiar, Moderately familiar, Extremely familiar
Frequency
Never, Rarely, Sometimes, Often, Always
Importance
Not at all importance, Slightly important, Somewhat important, Very important, Extremely important
Likelihood
Not at all likely, Slightly likely, Somewhat likely, Moderately likely, Very likely
Priority
Not a priority, Low priority, Medium priority, High priority, Essential
Probability
Not probable, Somewhat improbable, Neutral, Somewhat probable, Very probable
Quality
Very poor, Poor, Fair, Good, Excellent
Reflective
Very untrue of me, Untrue of me, Somewhat untrue of me, Neutral, Somewhat true of me, True of me, Very true
of me
Satisfaction
Completely dissatisfied, Mostly dissatisfied, Somewhat dissatisfied, Neither satisfied or dissatisfied, Somewhat
satisfied, Mostly satisfied, Completely satisfied
Satisfaction
Not at all satisfied, Slightly satisfied, Moderately satisfied, Very satisfied, Extremely satisfied
© 2008 Thomson, a part of the Thomson Corporation. Thomson, the Star logo, and Atomic Dog are trademarks used herein under license. All rights reserved.