General Issues in Survey
Design
Dr Desmond Thomas, MA TESOL
University of Essex
Useful References
• Denscombe, M., 1998, The Good Research
Guide for Small-scale Social Research
Projects, Open U. Press
• Foddy, W. 1993, Constructing Questions for
Interviews and Questionnaires - Theory and
Practice in Social Research, Cambridge:
Cambridge University Press
• Oppenheim, A.N., 1992, Questionnaire
Design, Interviewing and Attitude
Measurement, London: Pinter
The quantitative-qualitative divide
• Q-Q Divide: Numbers vs words
• Traditionally some methodologies (inc
surveys) labelled ‘quantitative’ while
others (eg ethnography) are considered
qualitative
• Methods are divided similarly eg
structured questionnaires (quantitative)
• This view of research is now challenged
Descriptive vs analytic
• A more useful distinction for surveys
• Descriptive surveys count – entire
populations (as in a census) or representative
samples
• Analytical surveys explore relationships
between different variables. These can be
causal but are more likely to explore
associations. Often less emphasis on
representativeness. Linking ideas is the aim.
Questionnaire or interview?
What’s the difference?
• Not a great deal, when questionnaire items
are read out and responses filled in by the
researcher
• But a world of difference between an
anonymous structured survey questionnaire
and an unstructured 1-to-1 interview
• QUESTIONNAIRE OR INTERVIEW: WHICH
SUITS YOUR RESEARCH?
• And what kind of interview?
A research questionnaire should …
• Be designed to collect information which can
be used as data for analysis: this can be
quantitative, qualitative or a mix
• Gather information by asking people about
issues directly related to a research project
(demonstrating validity)
• Consist of a written list of questions with
each respondent reading an identical set and
following identical procedures (reliability)
Questionnaires are useful when …
• Used with large numbers of
respondents in multiple locations
• Information required is straightforward
• Standardized data from identical
questions is required
Questionnaire issues: Sampling
• Is a sample representative of a total
population really necessary?
• Or can the sample in a qualitative survey
represent itself?
• If a representative sample is needed are
there ready-made sampling frames?
• Will sampling be random or in clusters?
• How important is sampling size?
Questionnaire issues: Piloting
• What should be piloted? Everything!
• Questionnaire layout, length, question
types, question wording, order of
questions, rubrics all need to be tested
• It is impossible to get things right at a
first attempt; second attempts, in turn,
will need to be piloted
Advantages of questionnaires
• Supply a large quantity of data for a
relatively low cost: not labour-intensive
• Standardized pre-coded answers can
enable speedy data collection,
management and even analysis
• Eliminate ‘interviewer bias’ – at least to
a certain extent
• Face validity
Questionnaire problem areas
• Low response rates
• Frustration for the respondent: boxticking can deter respondents when no
answer seems appropriate
• Frustration for the researcher: no scope
for clarification of answers
• One chance only to ‘get it right’ –
especially in terms of item wording
Creating web-based questionnaires
• Go to: http://www.surveymonkey.com/
• Design and edit your survey
• Send the link by email to potential
respondents
• Collect and analyse the data
• Note: limitations of the free vs the paid
version of this software
Types of 1-to-1 interviews
• Structured: with tight control over
question format and possible answers.
Like a face-to-face questionnaire
• Semi-structured: with a clearly-defined
question schedule but some flexibility
and more open-ended answers
• Unstructured: a narrative prompted by
one general question, perhaps
1-to-1 Interviews are useful when …
• Detailed information is needed from
respondents
• A smaller number of respondents is
acceptable
• Attitudes or feelings are investigated
• Sensitive issues are explored (?)
• ‘Key players’ are targeted
• Ideas for a questionnaire need to be explored
or ‘fine-tuned’
Advantages of 1-to-1 interviews
•
•
•
•
Depth of information
Insights eg gained from key informants
Flexibility of formats
Validity: direct contact means that data can
be checked for accuracy and for relevance
• High response rate
• Opportunity for targeted individuals to make
their voice heard – element of advocacy
Interview problem areas
• The ‘interviewer effect’ and power
relationships: face-to-face contact can directly
influence answers
• The ‘instant position’ effect: respondents feel
obliged to supply answers of some sort
• Investment of time and resources
• Complexity of data analysis
• Reliability issues for multiple interviews
Reliability issues
• How can we know if respondents
consistently understand the question?
Or if they have misinterpreted it?
• How can we know if respondents are
consistently telling the truth? Or if they
are misleading the interviewer
deliberately or at a subconscious level?
Features of interview design
•
•
•
•
•
•
•
How many questions?
In what order?
How long/short?
Format? (open qqs, statements etc.)
Follow-up probes and/or prompts?
Wording of questions?
Mode of recording/analysing answers?
Conducting an interview
• Establishing a relationship/trust
• Explaining aims & procedures
• Low key presentation of self and
involvement
• Active listening
• Clarifications, probes and prompts
• Recording, transcribing and analysing
Focus groups
• Consist of a small group of people who
are brought together by a trained
‘moderator’ (usually the researcher) to
explore attitudes, feelings and ideas
about a particular topic or set of issues
Main characteristics of FGs
• A question schedule is prepared by the
moderator to focus the discussion
• Place value on interaction within the
group as a means of eliciting
information – a social experience
• No requirement to reach consensus
• Less pressure to hold a fixed point of
view – in fact opinions can develop
FGs are useful when …
• Topics or issues need to be explored in
depth, or attitudes revealed
• Typical groups of key informants are to
be targeted eg language teachers
Advantages of FGs
• A less directive interview format
• ‘Interviewer bias’ can be greatly
reduced
• Provide a more natural social forum for
the exchange of ideas. Interviewees are
able to take the initiative – not just
respond
• The right to silence
FG problem areas
• Less directive means less predictable
• Huge amounts of data can be collected,
much of which may be unusable
• Perception that FGs are a market
research tool (Tony Blair & New Labour)