Designing Questionnaires
Dr. Roger Harris
Visiting Professor
Research and Innovation Management Centre (RIMC)
and
Institute of Social Informatics and Technological Innovations (ISITI)
roger.harris@rogharris.org
http://www.rogharris.org/
@HarrisrwhRoger
September 2013
1. Survey Research
Dr. Roger Harris
Visiting Professor
Institute of Social Informatics and Technological Innovation
roger.harris@rogharris.org
http://www.rogharris.org/
@HarrisrwhRoger
September 2013
Survey questionnaires
• If the researcher is looking to test and quantify
hypotheses and the data is to be analysed
statistically, a formal standardised questionnaire is
required.
• If the data to be collected is qualitative or is not to be
statistically evaluated, it may be that no formal
questionnaire is needed
• The heart of a survey is its questionnaire; the main
means of collective quantitative primary data
• Survey results depend crucially on the questionnaire
• Best practices stem from experience and
methodological research
Questionnaires
• A questionnaire is a formatted set of questions
for obtaining information from respondents.
• The objective of questionnaire design is to
translate the researcher’s information needs
into a set of specific questions that respondents
are able and willing to answer.
• Questionnaires collect quantitative data in a
standardised way so that the data is internally
consistent and coherent for analysis.
• There is no scientific principle to guarantee an
optimal or ideal questionnaire.
Survey implementation choices
• Direct interview
– costly; slow; high quality data
• Telephone interview
– Costly; needs expertise.
• World Wide Web
– Efficient; cheap; limited range of question types; hard to apply
full range of implementation procedures; respondents need
access.
• Self-administered by Mail
– Well studied; moderately expensive; can do well
• Self-administered by eMail
– Cheaper; efficient; respondents need access
• Multi-method
– Complex.
Quiz 1: True or False?
A questionnaire can be used to test and quantify a hypothesis
True
Survey results depend on questionnaires
True
There is a scientific method to design questionnaires
False, best practices stem from experience and methodological research
The objective of questionnaire design is to translate the
researcher’s information needs into a set of specific questions
True
Direct interviews are cheap to conduct
False
Telephone interviews are costly
True
The 5 most important elements for
achieving high response rates
• Respondent-friendly questionnaire
• The 4 contacts;
–
–
–
–
–
Pre-notice invitation
The questionnaire
Thank you
(Replacement questionnaire reminder)
Final outcome notification
• For mail – return stamped envelopes
• Personalised correspondence
• Prepaid token incentive
Three types of survey information
• Dependent variables
– Information we are primarily interested in
• Independent variables
– Information which might explain the dependent
variables
• Confounding variables
– Other factors related to both dependent and
independent factors which may distort the results
and have to be adjusted for
Characteristics of survey questionnaires
• Prescribed wording and order of questions,
– to ensure that each respondent receives the same
stimuli
• Prescribed definitions or explanations for each
question,
– to ensure interviewers handle questions consistently
and can answer respondents' requests for clarification
if they occur
• Prescribed response format,
– to enable rapid completion of the questionnaire
during the interviewing process.
Quiz 2: True or False?
1. Questionnaires should be respondent-friendly
True
2. There are 5 contacts with respondents during a questionnaire
survey
False; there are 4
3. The three types of variables in a survey are; dependent,
independent and confusing
False, they are dependent, independent and confounding
4. Dependent variables explain the independent variables
False, independent variables contain information that might explain the dependent
variables
5. Prescribed wording and order of questions enables rapid
completion of the questionnaire during the interviewing process.
False. Prescribed wording and order of questions, to ensure that each respondent
receives the same stimuli
However
• Given the same task and the same
hypotheses, six different people will come up
with six different questionnaires that differ
widely in their choice of questions, line of
questioning, use of open-ended questions
and length.
• There are no hard-and-fast rules about how
to design a questionnaire, but there are a
number of points that can be borne in mind
A well-designed questionnaire should :
• Meet the research objectives
• Obtain the most complete and accurate
information possible
• Ensure that respondents fully understand the
questions
• Be organised and worded to encourage
respondents to provide accurate, unbiased and
complete information
• Be easy for the interviewer to record the answer
• Keep the interview brief and to the point
How people answer questions; ideally
1. First, they interpret the question and
deduce its intent, next ,
2. They must search their memories for
relevant information, and,
3. Then integrate whatever information comes
to mind into a single judgment.
4. Finally, they must translate the judgment
into a response, by selecting one of the
alternatives offered by the question.
This process is known as optimizing
How people may answer questions;
1. By automatic compliance; with no intrinsic
motivation to make the answers of high quality
2. They may become fatigued, disinterested, or
distracted and may be less thorough in
comprehension, retrieval, judgment, and response
selection
3. Instead of attempting the most accurate answers,
respondents settle for merely satisfactory answers
4. Respondents may interpret each question
superficially and select what they believe will appear
to be a reasonable answer
This process is known as satisficing
Minimising satisificing
• The likelihood of satisficing is thought to be determined by three
major factors:
– task difficulty,
• question-specific attributes, e.g., the difficulty of interpreting a question and
of retrieving and manipulating the requested information
• attributes of the questionnaire’s administration, e.g., the pace at which an
interviewer reads the questions and the presence of distracting events
– respondent ability,
• the extent to which respondents are adept at performing complex mental
operations, practiced at thinking about the topic of a particular question, and
equipped with pre-formulated judgments on the issue in question
– respondent motivation
• Motivation is influenced by need for cognition, the degree to which the topic
of a question is personally important, beliefs about whether the survey will
have useful consequences.
• Efforts to minimize task difficulty and maximize respondent
motivation are likely to pay off by minimizing satisficing and
maximizing optimising.
Remedies for satisficing
• Minimise task difficulty
– Minimise the number of words in questions
– Avoid double-barreled questions
– Decompose questions when needed
• Use ratings not rankings
• Label response options
• Minimise ‘response effects’
– Avoid blocks of ratings on the same scale (prevents ‘straight lining’)
– Avoid ‘no response’ options
– Avoid ‘agree/disagree’, ‘yes/no’, ‘true/false’ questions
• Maximise motivation
–
–
–
–
–
Describe purpose of the survey
Provide instructions to think carefully
Include random probes (“give your reason”)
Keep surveys short
Put important questions early.
Quiz 3: True or False?
1. Given the same task and the same hypotheses, six different
people will come up with the same questionnaire
False
2. There are no hard-and-fast rules about how to design a
questionnaire
True
3. The 4 steps in respondent optimising are; interpret, search,
integrate and translate
True
4. Satisficing includes when respondents interpret each question
superficially
True
5. The likelihood of satisficing is thought to be determined by 3
factors; task difficulty, respondent ability and the weather.
False. The likelihood of satisficing is thought to be determined by 3 factors; task
difficulty, respondent ability and respondent motivation
Example of a survey
1. Review the literature to define the
research question
2. Define the population to be
surveyed
3. Isolate the variables of interest
4. Organise the variables into
hypotheses that reflect the
research question
5. Operationalise the variables into
tangible indicators
6. Develop questions that will enable
measurement of the indicators
7. Formulate a questionnaire
8. Carry out the survey
9. Analyse the data
1. Adoption rates of personal
computers varies among people
2. Knowledge workers
3. Computer anxiety
4. “Respondents with higher levels of
computer anxiety will be less likely
to use personal computers”
5. The Computer Anxiety Rating Scale
with 19 items; able to learn,
confidence, apprehension, etc.
6. “I feel insecure about my ability to
interpret a computer printout “ “I
am confident that I can learn
computer skills “
2. Purpose and Types of
Questionnaires
Dr. Roger Harris
Visiting Professor
Institute of Social Informatics and Technological Innovation
roger.harris@rogharris.org
http://www.rogharris.org/
@HarrisrwhRoger
September 2013
Approaches
• Structured
– All respondents provide answers to a set of questions
on a standardised questionnaire. Results can be
compared to each other and easily summarised
• Unstructured
– Allows the respondent to talk about whatever aspects
of the survey topic they wish. Best suited for
qualitative data
• Semi-structured
– Mixes both. Uses qualitative data to supplement
quantitative.
Administering questionnaires
• Self administered questionnaires may be sent by post, email, or
electronically online.
• Advantages
–
–
–
–
Cheap and easy to administer.
Preserve confidentiality.
Can be completed at respondent’s convenience.
Can be administered in a standard manner.
• Read out by interviewers. Questionnaires may be by telephone or face to
face.
• Advantages
– Allow participation by illiterate people.
– Allow clarification of ambiguity.
– More reliable response rate
• The method depends on who the respondents are. For example, university
lecturers may be more appropriately surveyed by email; older people by
telephone interviews; train passengers by face to face interviews.
Comparisons of data collection modes
Parameter
Interview
Telephone
Mail/Self
Most expensive
Medium expensive
Cheapest
Depend on
interviewer
Depend on
interviewer
Good & reduces bias
Coverage
Depend on personal
contact
Depend on personal
interviewer
Wide and scattered
reach
Observation
Listen, watch, assess
Listen, assess
None
Probing
Allow
Allow
Not possible
Literacy
Not required
Not required
Required
Language
Language
Not needed
Response rate
Highest
Medium
Lowest
Privacy
Difficult
Some anonymity
Good
Consent
Easy to gain
Possible
Difficult
Cost
Standardisation
Respondent skills
3. Asking Questions
Dr. Roger Harris
Visiting Professor
Institute of Social Informatics and Technological Innovation
roger.harris@rogharris.org
http://www.rogharris.org/
@HarrisrwhRoger
September 2013
Common Wisdom
1.
2.
3.
4.
5.
6.
Use simple, familiar words
Avoid technical terms, jargon, and slang
Use simple syntax
Avoid words with ambiguous meanings
Use wording that all respondents will interpret in the same way
Strive for wording that is specific and concrete as opposed to
general and abstract
7. Make response options exhaustive and mutually exclusive
8. Avoid leading or loaded questions that push respondents toward
an answer
9. Ask about one thing at a time
10. Avoid double-barreled questions (if there’s an ‘and’ in the
question, there are probably two questions
11. Avoid questions with single or double negations.
Optimize question order
1.
2.
3.
4.
5.
6.
Early questions should be easy and pleasant to answer, and
should build rapport between the respondent and the
researcher.
Questions at the very beginning of a questionnaire should
explicitly address the topic of the survey, as it was described to
the respondent prior to the interview.
Questions on the same topic should be grouped together.
Questions on the same topic should proceed from general to
specific.
Questions on sensitive topics that might make respondents
uncomfortable should be placed at the end of the questionnaire.
Filter (branching) questions should be included, to avoid asking
respondents questions that do not apply to them.
Open versus closed questions
• Open questions permit respondents to
answer in their own words
• Closed questions require respondents to
select an answer from a set of choices
• The vast majority of survey questions are
closed
• Closed questions must be grouped into a
relatively small number of categories
• This requires the development of a coding
scheme
Open versus closed questions
• Open questions are usually preferable
for measuring quantities, e.g. age,
number of doctor visits, hours devoted
to housework,
• Closed quantity categories for this type
of open question (e.g., less than 1 h, 1–3
h, more than 3 h) can produce errors
Comparison of open and closed
Open questions
•
•
•
•
•
•
•
•
Advantages
Allows exploration of the range of possible
themes arising from an issue
Respondent is asked to give a reply to a
question in his/her own words
Can be used even if a comprehensive range
of alternative choices cannot be compiled
Reveal the issues which are most important
to the respondent
Respondents can 'qualify' their answers or
emphasise the strength of their opinions
Disadvantages
Respondents may find it difficult to
'articulate' their responses
Respondents may not give a full answer if
they may forget to mention important points
Verbatim comments have to be interpreted,
coded and reduced to manageable
categories
Closed questions
•
•
•
•
•
•
•
Advantages
Easy and quick to fill in, less reliance on
memory
Can specify answer categories most suitable for
the purposes
Minimize discrimination against the less literate
(self administered questionnaire) or the less
articulate (interview questionnaire)
Easy to code, record, and analyze results
quantitatively
Easy to report results
Disadvantages
Do not allow the respondent to give a different
response to those suggested.
Suggest answers that respondents may not
have considered before.
Categorical judgments
• Such as the ‘‘what is most important
problem’’
• Closed categorical question can be used only
if its answer choices are comprehensive.
• Usual to include other category.
• Respondents tend to restrict their answers to
the substantive choices that are explicitly
offered.
Open response-option questions
• Both open-ended and also include specific
response-options as well. For example,
• What features of this implement do you like?
–
–
–
–
–
Performance
Quality
Price
Weight
Others mentioned:
• Eliminates the disadvantages
Types of closed questions format
• Choice of categories, for example, What is your marital status?
–
–
–
–
[
[
[
[
] Single
] Married
] Divorced
] Widowed
• Likert style scale, for example, Statistics is an interesting subject
–
–
–
–
–
[
[
[
[
[
] Strongly disagree
] Disagree
] Cannot decide
] Agree Strongly
] Agree
• Differential scales, for example, How would you rate the presentation?
Extremely interesting
[1] [2] [3] [4] [5] [6] [7] [8] [9] [10]
Extremely dull
• It may also be diagrammatic;
• Extremely interesting
Extremely dull
Types of closed questions format
•
Checklists, for example, Circle the clinical specialties you are particularly
interested in:
General medicine
General surgery
Ophthalmology
Paediatrics
Obstetrics and gynecology
Orthopaedics
Accident and emergency
General practice
• Ranking, for example, Please rank your interests in the following
specialties
(1= most interesting, 8=least interesting)
–
–
–
–
–
–
–
–
[
[
[
[
[
[
[
[
] General medicine
] Obstetrics and gynecology
] General surgery
] Orthopaedics
] Ophthalmology
] Accident and emergency
] Paediatrics
] General practice
Question checklist
1. Is this question sufficient to generate the required information?
2. Can the respondent answer the question correctly?
3. Are there any external events that might bias response to the
question?
4. Do the words have the same meaning to all respondents?
5. Are any of the words or phrases loaded or leading in any way?
6. Are there any implied alternatives within the question?
7. Will the question be understood by the type of individual to be
interviewed?
8. Is there any ambiguity in my questions?
9. Are any words or phrases vague?
10. Are any questions too personal or of a potentially embarrassing
nature?
11. Do questions rely on feats of memory?
Put questions into a meaningful order
and format
• Opening questions
– Opening question easy and pleasant to answer
– Not threatening
– Questions of special importance should be earlier
• Question flow
– Flow in some kind of logical order
– One leads easily and naturally to the next.
– Questions on one subject should be grouped together
• Question variety
– Vary the respondent's task from time to time
– An open-ended question here and there (even if it is not
analysed)
– Questions involving showing cards/pictures to respondents can
help vary the pace and increase interest.
The effect of one word
Balanced questions
• Do you support India taking military action against
Pakistan?
• Do you support or oppose India taking military action
against Pakistan?
• If you heard that candidate X supported a higher
minimum wage, would that make you more likely to vote
for her?
• If you heard that candidate X supported a higher
minimum wage, would that make you more likely to vote
for her, less likely to vote for her, or wouldn’t it make much
difference?
• Did you happen vote in the presidential election in 2010?
• Did you happen vote in the presidential election in 2010,
or not ?
“Don’t know” – offered or volunteered
Using scales to get to “the most”
How questions can affect one another
• Funnelling. Ask general questions before
specific priority questions in order to obtain
unbiased responses.
• Pay attention to how one question might
influence answers to the next:
– In considering whether or not to vote for a political
candidate, how important is the candidate’s
position on HIV/AIDS in making your choice?
– What do you think are the two most important
health care issues for the government to address?
4. How to use Scales
Dr. Roger Harris
Visiting Professor
Institute of Social Informatics and Technological Innovation
roger.harris@rogharris.org
http://www.rogharris.org/
@HarrisrwhRoger
September 2013
Types of scales for closed questions
• The four levels of measurement:
• Nominal
• Ordinal
• Interval, and
• Ratio
Nominal scales
•
•
•
•
•
•
•
•
Simplest form of measurement scale
Classifies individuals, companies, products, brands or other entities into categories
where no order is implied.
Often referred to as a categorical scale.
It is a system of classification and does not place the entity along a continuum.
Involves a simply count of the frequency of the cases assigned to the various categories
Example of a nominal scale
Which of the following food items do you tend to buy at least once per month? (Please
tick)
1.
Okra
2.
Palm Oil
3.
Milled Rice
4.
Peppers
5.
Prawns
6.
Pasteurized milk
The numbers have no arithmetic properties and act only as labels.
Ordinal scales
• Involve the ranking of individuals, attitudes or items along the
continuum of the characteristic being scaled.
• For example, if a researcher asked farmers to rank 5 brands of
pesticide in order of preference.
• Example of an ordinal scale used to determine farmers'
preferences among 5 brands of pesticide.
• Brand Order of preference
1.
2.
3.
4.
5.
Rambo
R.I.P.
Killalot
D.O.A.
Bugdeath
• Provides the order of preference but nothing about how much
more one brand is preferred to another - no information about the
interval between any two brands.
Interval scales
• The interval scale has equal units of measurement, thus making it possible
to interpret not only the order of scale scores but also the distance
between them.
• Allows the use of the arithmetic mean as the measure of average.
• Two respondents with scale positions 1 and 2 are as far apart as two
respondents with scale positions 4 and 5
• Cannot be said that a respondent with score 10 feels twice as strongly as
one with score 5.
• Temperature is interval scaled, being measured either in Centigrade or
Fahrenheit. We cannot speak of 50°F being twice as hot as 25°F. No fixed
origin.
• Interval scales may be either numeric or semantic.
• Most of the common statistical methods of analysis require only interval
scales in order that they might be used.
Ratio scales
• The highest level of measurement
• Has the properties of an interval scale together with a
fixed origin or zero point.
• Examples; weights, lengths and times.
• Permit the researcher to compare both differences in
scores and the relative magnitude of scores.
• For instance the difference between 5 and 10 minutes
is the same as that between 10 and 15 minutes, and 10
minutes is twice as long as 5 minutes.
• Virtually all statistical operations can be performed on
ratio scales.
Likert-type scales
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Most often uses 5 points;
“I would use this tool in my work”
[Strongly Disagree ] [Disagree] [Neither Agree Nor Disagree] [Agree] [Strongly Agree]
Semantic scale
“Was the product easy or difficult to use?”
Very Difficult [ ] [ ] [ ] [ ] [ ] Very Easy
Scales might have 7or 11 points.
The points offered should cover the entire measurement continuum, leaving out no regions
Points must appear to be ordinal, progressing from one end of a continuum to the other
The meanings of adjacent points should not overlap.
Each respondent must have a relatively precise and stable understanding of the meaning of each
point on the scale.
Most or all respondents must agree in their interpretations of the meanings of each scale point
Offering a neutral midpoint may encourage satisficing, or
Eliminating the midpoint will force people with no opinion to pick a point measurement.
Cross-sectional validity and test–retest reliability increases from 2- to 3- to 5-point scales but
were equivalent thereafter for 7-, 9-, and 14-point scales.
Some labels that work
Bipolar intervals
•
•
•
•
•
•
•
Extremely bad
Moderately bad
Slightly bad
Neither good nor bad
Slightly good
Moderately good
Extremely good
•
•
•
•
•
•
•
Dislike a great deal
Moderately dislike
Dislike a little
Neither like nor dislike
Like a little
Moderately like
Like a great deal
Unipolar intervals
•
•
•
•
•
Not at all
Slightly
Moderately
Very
Extremely
•
•
•
•
•
Definitely won’t
Probably won’t
Might or might 7not
Probably will
Definitely will
•
•
•
•
•
Never
Sometimes
About half the time
Most of the time
Always
Quiz 5: True or False?
1. There are three levels of measurement; Nominal, Ordinal, and
Interval
False. There are four levels of measurement; Nominal, Ordinal, Interval, and Ratio
2. Ordinal scales involve the ranking items along the continuum of
the characteristic being scaled
True.
3. In an interval scale, two respondents with scale positions 1 and 2
are as far apart as two respondents with scale positions 4 and 5
True.
4. Interval scales can only be numeric
False. Interval scales may be either numeric or semantic
5. A ratio scale has the properties of an interval scale together with
a fixed origin or zero point
True
5. Questionnaire Appearance
Dr. Roger Harris
Visiting Professor
Institute of Social Informatics and Technological Innovation
roger.harris@rogharris.org
http://www.rogharris.org/
@HarrisrwhRoger
September 2013
Physical appearance
• The physical appearance of a questionnaire can have
a significant effect upon both the quantity and quality
of data obtained
• Ill-designed questionnaires can give an impression of
complexity, and too big a time commitment
• Unnecessarily confusing layouts making it more
difficult for interviewers, or respondents in the case
of self-completion questionnaires, to complete
• Keep questionnaires as short as possible
• In a rural situation an interview should not last longer
then 30-45 minutes.
Format
• Booklets
– In the place of loose or stapled sheets of paper,
make it easier for interviewer or respondent to
progress through the document. Fewer pages
tend to get lost
• Simple, clear formats
– The clarity of questionnaire presentation can also
help to improve the ease with which interviewers
or respondents are able to complete a
questionnaire.
Design tips
• Creative use of space and typeface
– Tendency to put too much information on a page. This is counterproductive since it gives the questionnaire the appearance of being
complicated.
– Questionnaires that make use of blank space appear easier to use,
enjoy higher response rates and contain fewer errors when
completed.
• Colour coding
– Can help in the administration of questionnaires. It is often the case
that several types of respondents are included within a single survey
(e.g. wholesalers and retailers). Printing the questionnaires on two
different colours of paper can make the handling easier.
• Interviewer instructions
– Place alongside the questions to which they pertain. Instructions on
where the interviewers should probe for more information or how
replies should be recorded are placed after the question
Distributing questionnaires
• If you’re sending a cover letter, be sure to use
letterhead stationary and have the principal
investigator or someone of renown from the
funding agency sign it to add prestige.
• Your cover and advance letters should include a
brief explanation of the study, an explanation of
how you obtained the respondent’s name, why
it is important that each "sampled" respondent
cooperate, and a short assurance of
confidentiality.
• Include a stamped self-addressed envelope for
the respondent to return the completed form.
Design tips
• Clear instructions how to mark answers:
– “Please tick the box that best describes your
answer”
• Instructions for each question:
– “Tick one box only”
– “Write the number inside the box”
– “Circle or underline the choice that corresponds
most closely to your preference”
Design tips
• Questions in a meaningful order; natural
flow, progression
• Place definitions in the question
• Respondents rarely read instruction booklets
• Indicate prominently which questions are
mandatory
• Include a “thank you” at the end
• Ensure the respondent or interviewer knows
what to do with the completed
questionnaire.
Design elements for a mail
questionnaire
•
•
•
•
•
Booklet
Descriptive title
No questions on front or back
Size appropriate for mail envelope
Easy to read font; sans serif, 10 pt+, Ariel.
(this is sans serif
•
•
•
•
Colour
Attractive, inviting design
Number questions
Embedded instructions
this is serif)
6. Testing and Evaluating
Questionnaires
Dr. Roger Harris
Visiting Professor
Institute of Social Informatics and Technological Innovation
roger.harris@rogharris.org
http://www.rogharris.org/
@HarrisrwhRoger
September 2013
Piloting
• Until it has actually been used in interviews and with respondents,
it is impossible to say whether it is going to achieve the desired
results.
• Pre-test the questionnaire before it is used in a full-scale survey,
to identify any mistakes that need correcting.
– whether the questions as they are worded will achieve the desired
results
– whether the questions have been placed in the best order
– whether the questions are understood by all classes of respondent
– whether additional or specifying questions are needed or whether
some questions should be eliminated
– whether the instructions to interviewers are adequate.
• Usually a small number of respondents are selected for the pretest. The respondents selected for the pilot survey should be
broadly representative of the type of respondent to be
interviewed in the main survey.
Testing methods
• Expert review
– one or more experts critiques the questionnaire. The
experts are typically survey methodologists, but they
can be supplemented with specialists in the subject
matter(s) of the questionnaire. Reviews are done
individually or as part of a group discussion.
• Pretesting
– involves administering a questionnaire to a small
sample of the relevant population under conditions
close to, or identical to, those of the main survey. Ask
them how the questions worked, which are typically
reported at a group debriefing discussion (cognitive
interviewing).
Cognitive interviewing
• After testing participants have completed the
questionnaire, ask them:
– In your own words, what do you think the
questionnaire is trying to discover?
– What was the question asking?
– What did you consider in determining your
response?
– Was there anything difficult about this question?
– Do you feel that something important might be
missing?
What am I measuring? Reliability
• Questionnaire reliability has to do with the
quality of measurement
• Reliability is a measure of the consistency or
repeatability of your measures
• Internal consistency is one type of reliability
measure. Cronbach’s alpha measures it by
how well a set of indicators measures a single
uni-dimensional variable.
What am I measuring? Validity
• Construct validity
– The degree to which inferences can legitimately
be made from the operationalisations in the
questionnaire to the theoretical constructs on
which they are based
• External validity
– The degree to which the conclusions from the
questionnaire data would hold for other persons
in other places and at other times
Quiz 6: True or False?
1. The physical appearance of a questionnaire has no effect on either
the quantity and quality of data obtained
False.
2. In a rural situation an interview should not last longer then 30-45
minutes.
True.
3. Expert review and pre-testing are two methods of piloting a
questionnaire
True.
4. Cognitive interviewing involves asking test participants about the
questionnaire
True.
5. Reliability is a measure of how truthful respondents are
False. Reliability is a measure of the consistency or repeatability of your measures
7. Survey Conduct
Dr. Roger Harris
Visiting Professor
Institute of Social Informatics and Technological Innovation
roger.harris@rogharris.org
http://www.rogharris.org/
@HarrisrwhRoger
September 2013
Getting respondents to respond
•
•
•
•
Introduce your questionnaire
Motivate people to complete it
Write a few lines to introduce your survey.
Introduction should be short and includes:
–
–
–
–
–
–
–
Purpose of the survey
Why it is important to hear from the respondent
What may be done with the results
Promise of confidentiality
What possible impacts may occur with the results.
Person to contact for questions about the survey.
Due date for response
Steps in designing a survey
1.
2.
3.
4.
5.
6.
7.
8.
9.
Review the literature to define the research question
Define the population to be surveyed
Isolate the variables of interest
Organise the variables into hypotheses that reflect
the research question
Operationalise the variables into tangible indicators
Develop questions that will enable measurement of
the indicators
Formulate a questionnaire
Carry out the survey
Analyse the data
Steps in designing a questionnaire
1. Decide the information required.
2. Define the target respondents.
3. Choose the method(s) of reaching your target
respondents.
4. Decide on question content.
5. Develop the question wording.
6. Put questions into a meaningful order and
format.
7. Check the length of the questionnaire.
8. Pre-test the questionnaire.
9. Develop the final survey form.
Decide the information required.
• Do not start by writing questions
• Decide 'what are the things I need to know
from the respondent in order to meet my
survey objectives?‘
• From the research brief and the research
proposal a small number of preliminary
informal interviews with target respondents
will give a glimpse of reality
Define the target respondents
• Must define the population about which the
researcher wishes to generalise from the
sample data to be collected
• E.g., existing users of a product type and/or
non-users
• Take into account factors such as the age,
education, etc. of the target respondents.
Choose the method(s) of reaching
your target respondents
• The method of contact will influence the
questions and the phrasing of those
questions
• The main methods available in survey
research are:
· personal interviews
· group or focus interviews
· mailed questionnaires
· telephone interviews.
Decide on question content
• Always ask, "Is this question really needed?“
• No question should be included unless the data it
gives rise to is directly of use in testing one or more of
the hypotheses
• Occasions when seemingly "redundant" questions
might be included:
– Opening questions that are easy to answer and which are
not perceived as being "threatening", and/or are perceived
as being interesting, can greatly assist in gaining the
respondent's involvement in the survey and help to
establish a rapport.
– "Dummy" questions can disguise the purpose of the survey
and/or the sponsorship of a study
– Validating questions to test the validity of other answers
. Develop the question wording
• Three forms of question,
–closed,
–open-ended and
–open response-option
questions.
Put questions into a meaningful order
and format
• Opening questions:
– Easy to answer
– Not threatening
– The first question is crucial as the respondent's first exposure to the
interview and sets the tone for the nature of the task to be performed.
• Question flow:
– Questions should flow in some kind of psychological order, so that one
leads easily and naturally to the next. Questions on one subject, or
one particular aspect of a subject, should be grouped together.
• Question variety:
– Respondents become bored quickly and restless when asked similar
questions. Vary the respondent's task from time to time. An openended question here and there (even if it is not analysed) may provide
much-needed relief from a long series of questions in which
respondents have been forced to limit their replies to pre-coded
categories. Questions involving showing cards/pictures to respondents
can help vary the pace and increase interest.
Quiz 4: True or False?
1. Surveys can be structured, unstructured or stratified
False. Surveys can be structured, unstructured or semi-structured
2. Self administered questionnaires are relatively expensive
False.
3. Double-barreled questions ask about guns
False. Double- barreled questions contain two questions
4. Questions on sensitive topics that might make respondents
uncomfortable should be placed at the end of the questionnaire
True
5. The vast majority of survey questions are closed.
True
6. Closed questions allow respondents to give a different response
to those suggested
False
8. Automated Techniques
Dr. Roger Harris
Visiting Professor
Institute of Social Informatics and Technological Innovation
roger.harris@rogharris.org
http://www.rogharris.org/
@HarrisrwhRoger
September 2013
Internet questionnaires
• Facilitates
–
–
–
–
•
•
•
•
•
Design; templates available
Distribution
Completion
Analysis
Lowers costs
Accelerates survey
Only for computer literate online respondents
No face-to-face
Concerns with confidentiality and privacy
Aspects of online questionnaires
• Graphical user interface
– Control appearance, colours, fonts for attractive
design
• Layout
– Paging from screen to screen
– Scrolling simplifies navigation
• Push buttons
– Buttons for specified actions and options; ‘Submit’
‘Next’
Aspects of online questionnaires
• Status bar
– Visual indicator of respondent’s progress
• Radio button
– Circular icon to activate one response choice and
de-activates the others
• Drop-down box
– Space-saving device to present choices when they
are needed but hides them when they aren’t
Aspects of online questionnaires
• Check boxes
– Small boxes next to question that the respondent
clicks on to select their answer; when an x
appears in it.
• Open-ended boxes
– Boxes in which respondents can type in their own
answers to open-ended questions
• Pop-up boxes
– Boxes that appear at selected points containing
information or instructions to respondents.
Software aspects of online
questionnaires
• Variable piping
– Allows variables to be inserted into a questionnaire as
a respondent is completing it.
• Error trapping
– Controls the flow
• Forced answering
– Prevents respondents from continuing with a
questionnaire if they fail to answer a mandatory
question
• Interactive help desk
– Live, real-time support for respondents with a
problem.
Example: Survey Monkey
• Founded in 1999
• In 2013,
SurveyMonkey had
15 million users
• Over 200 employees
• Used a lot for market
surveys, and by SMEs
Survey Monkey pricing options
Quiz 7: True or False?
1. Construct validity degree to which inferences can legitimately be
made from the operationalisations in the questionnaire to the
theoretical constructs on which they are based
True.
2. External validity is a measure of the appearance of the front
page.
False. The degree to which the conclusions from the questionnaire data would hold for
other persons in other places and at other times
3. Automated techniques facilitate design, distribution, completion
and printing
False. Automated techniques facilitate design, distribution, completion and analysis
4. The status bar is a measure of how important the respondent is
False. It is a visual indicator of respondent’s progress
5. Survey Guerrilla is an example of an online survey tool.
False. Survey Monkey is an example of an online survey tool.
9. Examples of Question Design
Dr. Roger Harris
Visiting Professor
Institute of Social Informatics and Technological Innovation
roger.harris@rogharris.org
http://www.rogharris.org/
@HarrisrwhRoger
September 2013
A poor question
Do you support gun control?
The problem
Improved version
Unclear question because of
vague words
Do you believe that guns do
not belong in schools?
Yes
No
Don’t know
Do you believe that guns and
knives do not belong in
schools?
Yes
No
Don’t know
Two or more questions
Do you believe that knives do
not belong in schools?
Whenever violence occurs in
schools, weapons are typically
found in school lockers. Do
you believe that students
should keep guns in their
lockers?
Yes
No
Don’t know
Wordy or lengthy question
Yes
No
Don’t know
Yes
No
Don’t know
Should students keep guns in
their lockers?
Yes
No
Don’t know
A poor question
The problem
Improved version
Should students pack a 45 at
school?
Yes
No
Don’t know
Question contains jargon
Should students carry a
handgun to school?
Yes
No
Don’t know
Students should not carry
weapons and should not have
them in their lockers. Do you
agree?
Strongly agree
Agree
Undecided
Disagree
Strongly disagree
Question contains negatives
Should students keep guns in
their lockers?
How many times have you
seen a student carry a
handgun?
0 times
1-2 times
2-3 times
More than 3 times
Response categories overlap
Yes
No
Don’t know
How many times have you
seen a student carry a
handgun?
0 times
1-2 times
3-4 times
More than 4 times
A poor question
The problem
Improved version
How often have you seen
Respondent does not
students carry semi-automatic understanding to answer the
weapons at school?
question
None
1 time
2 times
3 or more times
How often have you seen
students carry a rifle at
school?
None
1 time
2 times
3 or more times
To what extent do you feel
that handguns are a problem
at your school?
A great extent
Some
Not very important
Not a problem
To what extent do you feel
that handguns are a problem
at your school?
A great extent
Some extent
Little extent
Unbalanced response option
No labels
What does the *
mean?
19 indicators
Bit confusing here; age/sex
What’s this?
Instructions
Definition
Overlapping
< >?
Two questions
Two questions
Uneven frequency flow
Two questions
Two questions
Two questions
Two questions
Dense appearance
Confusing layout
What this?
No instructions
Yes or no?
What does it mean?
Not much space
Comments about what?
No instructions what to put here
Wordy
No alternatives or graduated
response
Some observations
• Almost every questionnaire you come across
has something wrong with it
• Most are the result of inappropriate
approaches;
– Starting with the questions
– No clear articulation of the purpose
– Not following simple guidelines
– Insufficient thought put into the process
– Underestimation of what’s required
10. Recording and Coding
Questionnaire Responses
Dr. Roger Harris
Visiting Professor
Institute of Social Informatics and Technological Innovation
roger.harris@rogharris.org
http://www.rogharris.org/
@HarrisrwhRoger
September 2013
• Closed question answers should be coded in
order to facilitate recording.
• Number all questions
• Record coded answers in a spreadsheet
• For ease of recording, show the code next to the
response:
1.
2.
3.
4.
5.
Strongly disagree
Disagree
Cannot decide
Agree Strongly
Agree
Recording responses
Q1
1
2
3
4
5
6
7
8
9
Respondent
Respondent
Respondent
Respondent
Respondent
Respondent
Respondent
Respondent
Respondent
Q2
Q3
Q4
Q5
Q6
Q7
Sample spreadsheet recording
eBorneo
Question
1
2
Gender
Ethnicity
3
4
5
6
Age
Education
Employment
Income
10-20=1
Resp. No.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Village
Bakelelan
Bakelelan
Bakelelan
Bakelelan
Bakelelan
Bakelelan
Bakelelan
Bakelelan
Bakelelan
Bakelelan
Bakelelan
Bakelelan
Bakelelan
Bakelelan
Bakelelan
Bakelelan
M
Lun Bawang=L
Kelabit=K
Penan=P
21-30=2
31-40=3
41-50=4
51-60=5
61-70=6
71-80=7
None=N
Primary=PI
Junior High=J
Senior High=S
College=C
University=U
Unskilled=U
Skilled=S
Farmer=F
Business=B
Government=G
Housewife=H
Up to RM250 =1
RM251-RM500=2
RM501-RM750=3
RM751-RM1,000=4
F
Other=O
81+=8
Post-Graduate=PG
None/other=O
RM1,001+=5
F
F
F
M
M
F
F
F
F
F
M
M
F
F
M
M
L
L
L
L
L
L
L
L
L
L
L
L
L
L
L
L
J
S
PI
PI
J
PI
J
U
J
J
PI
J
S
PI
C
PI
F
B
F
F
F
F
H
B
F
F
F
B
B
B
B
F
4
2
5
5
6
3
4
3
3
4
4
5
2
3
3
4
2
2
2
1
1
2
2
5
2
1
1
5
2
3
2
2
Spreadsheet analyses
100
Gender
90
Q
u
a
n
t
i
t
y
Q
u
a
n
t
i
t
y
80
70
62
60
50
40
33
30
20
100
90
80
70
60
50
40
30
20
10
0
Education
28
19
24
9
2
12
1
10
0
Male
Female
Gender
100
Age
90
Q
u
a
n
t
i
t
y
Education Level
Q
u
a
n
t
i
t
y
80
70
60
50
40
25
30
25
24
20
10
12
100
90
80
70
60
50
40
30
20
10
0
Employment
35
23
6
8
9
7
1
1
0
71-80
81+
0
10-20
21-30
31-40
41-50
51-60
Age Range
61-70
Employment Category
5
9
SPSS for statistical analyses
11. Group Exercises
Dr. Roger Harris
Visiting Professor
Institute of Social Informatics and Technological Innovation
roger.harris@rogharris.org
http://www.rogharris.org/
@HarrisrwhRoger
September 2013
Exercise
• Nomophobia is the fear of being out of mobile
phone contact.
• A research organisation looked at anxieties suffered by
mobile phone users.
• The study found that nearly 53% of mobile phone users in
Britain tend to be anxious when they "lose their mobile
phone, run out of battery or credit, or have no network
coverage".
• The psychographic variables were:
– Attitude towards usage of cellular phones,
– Mobile phones dependence, and
– Associated anxiety
• Devise a set of questions to measure nomophobia.
The questionnaire focusing on
nomophobia had 7 components
1. Duration of having mobile phone with self;
2. Anxiety and stress experienced because of faulty
connections;
3. Loss of mobile and battery discharge;
4. Amount spent per month on recharge
5. Reaction shown to phone ringing at
inappropriate times;
6. Frequency of change of phone/sim cards
7. Reactions to inability of using the phone for a
period of one week.
Answers
Physical sensations
Never
1. Physical sensations
Accelerated heart beat, heart pounding
Face reddening or hot flashes
Tightness of the chest, chest pain or pressure
Numbness or tingling sensations of arms or legs
Lightheadedness, dizziness or feelings of
unsteadiness
Shortness of breath or difficulty breathing
Sweating
Shaking or trembling
Intense fear, panicky feelings, anxiety
Seldom
Some
times
Often
Always
Thoughts
2. Thoughts
I worry that something bad might happen to myself
I worry that something bad might happen to someone
else
I worry that something bad might happen to the device
(getting lost, stolen, or broken)
I worry about others being unable to contact me
I worry about missing important news
I worry that I might get lost
I am preoccupied with thoughts about how to complete
tasks without the device
I am preoccupied with thoughts about how I could have
acted differently to be able to access the device
Never
Seldom
Some
times
Often
Always
Emotions
3. Emotions
I feel lonely
I feel anxious and/or embarrassed because I feel unprepared
I feel anxious because I cannot access certain information
I feel stressed when I am not able to check on certain things
on my device
I feel uncomfortable communicating with friends, family,
colleagues, etc. without my device (I prefer to avoid
communicating face-to-face)
I feel guilty for not contacting someone
I feel guilty because I am unable to do work or other
important tasks
I feel angry that I allowed myself to lose access to the device
I feel angry that someone else led me to lose access to the
device
I feel sad that I will not be able to use my device
Never
Seldom
Some
times
Often
Always
Other
4. Other
I feel bored
I just don’t feel “right”
Never
Seldom
Some
times
Often
Always
Review Questions
1.
2.
3.
4.
5.
6.
7.
8.
Summarise the qualities of a good questionnaire.
Where should interviewer instructions pertaining to responses to a
particular question be placed on the questionnaire?
The textbook says that one does not start by writing questions. How should
the researcher begin?
What are two occasions when apparently "redundant" questions should be
found in a questionnaire?
Name the three advantages of open-ended questions.
What are three reasons why a respondent might be unable to answer a
question?
What is the recommended duration of interviews carried out in rural
situations?
What are the key characteristics of opening questions in a questionnaire?