Handbook of Online Research Methods Colloquium 28-29 March 2007
Overview: Online Surveys
Vasja Vehovar
University of Ljubljana, Slovenia
http://WebSM.org
Introduction
Structure
• Computer-assisted survey information collection
• Key methodological issues
• Related issues
• Applications
Computer-assisted survey information collection
Technology and surveys
1930s: applications of probability sampling
1960s: expansion of telephone surveys
1970s: computer technology appears in surveys
1980s: computer-assisted surveying
1990s: Internet mediated surveys
Computer-assisted survey information collection
Early CASIC interviewer assisted modes
• CATI – Computer-assisted telephone interviewing
• CAPI – Computer-assisted personal interviewing
• CASI – Computer-assisted self-interviewing
Computer-assisted survey information collection
CASIC benefits
• Reduced time and costs for data input
• Elimination of errors during data transcription
• Implementation of advanced features:

automatic skips and branching

randomization of questions and response options

control of answers

inclusion of multimedia elements…
Computer-assisted survey information collection
Computerized self-administered questionnaires (CSAQ)
Different modes of CSAQ:

disk-by-mail

touch-tone data entry (TDE)

interactive voice response (IVR)

e-mail surveys

web surveys
Computer-assisted survey information collection
Online, Internet, Web, CASIC, CSAQ, CADAC
CASIC
(e.g. CATI, CAPI,
CASI, etc.)
Online surveys
Internet surveys
Internet surveys
Web surveys
Computer-assisted survey information collection
Online, Internet, Web, CASIC, CSAQ, CADAC
CSAQ
CASIC
(e.g. CATI, CAPI,
CASI, etc.)
Online surveys
Internet surveys
Internet surveys
Web surveys
Computer-assisted survey information collection
Online, Internet, Web, CASIC, CSAQ, CADAC
CSAQ
CASIC
(e.g. CATI, CAPI,
CASI, etc.)
Online surveys
Internet surveys
Internet surveys
Web surveys
CADAC
Computer-assisted survey information collection
Interviewer-less and paper-less surveys
Interviewer
involvement
Survey mode
Paper and pencil
CASIC
Interviewer
presence
Paper and pencil (face-toface) interviewing (PAPI)
CAPI, CASI,
Audio/Video CASI
Remote
interviewer
Paper assisted telephone
interviewing (PATI)
CATI, CAVI (computer
assisted video
interviewing)
No interviewer
Self-administered paper
questionnaires (mail
questionnaires)
CSAQ telesurveys
(web CSAQ, TDE, IVR,
Virtual interviewer,...)
Computer-assisted survey information collection
Interviewer-less and paper-less data collection
Interviewer
involvement
Survey mode
Paper and pencil
CASIC
Interviewer
presence
Paper and pencil (face-toface) interviewing (PAPI)
CAPI, CASI,
Audio/Video CASI
Remote
interviewer
Paper assisted telephone
interviewing (PATI)
CATI, CAVI (computer
assisted video
interviewing)
No interviewer
Self-administered paper
questionnaires (mail
questionnaires)
CSAQ telesurveys
(web CSAQ, TDE, IVR,
Virtual interviewer,...)
Computer-assisted survey information collection
CSAQ input-output technology
Asking question
(INPUT)
Recording of responses (OUTPUT)
Manual recording
Automatic voice recognition
Written
questions
Standard CSAQ
(PC, TV, PDA,
Mobile,...)
Visual CSAQ with AVR
Audio questions
Video CSAQ, Audio
CSAQ,
TDE
IVR, Video IVR,
TTS with AVR, Virtual
interview
Computer-assisted survey information collection
CSAQ input-output technology
Asking question
(INPUT)
Recording of responses (OUTPUT)
Manual recording
Automatic voice recognition
Written
questions
Standard CSAQ
(PC, TV, PDA,
Mobile,...)
Visual CSAQ with AVR
Audio questions
Video CSAQ, Audio
CSAQ,
TDE
IVR, Video IVR,
TTS with AVR, Virtual
interview
Computer-assisted survey information collection
Technological aspects
• Developments of the Internet and related technologies
• Importance of broadband Internet access
• Possibilities of distribution across various platforms and devices
• Fast and easy implementation using specialized software tools
Key methodological issues
1. Probability vs. non-probability sampling
• Two major problems of probability samples in Internet surveys:

non-coverage

sampling frame problems
• Image of Internet surveys as inherently non-probability ones
Key methodological issues
The spurious link
Issue of a statistical inference:
probability vs. non-probability samples
Information-communication
technologies
Internet
survey
Management of the survey
process
Mode of survey data
collection
Key methodological issues
2. Sampling and invitation
Sampling
type
Solicitation
Type of invitation
Personal invitation
General invitation
Probability samples
Samples from closed
population with email,
Pre-recruited with mail,
telephone, f2f
Web intercept surveys
Non-probability
samples
Self-recruited Internet
panels,
Opt-in lists of e-mail
adresses
General banner-ad
invitation
Key methodological issues
3. Non-response
• Generally low and variable response rates to web surveys
• Complexity of the response process
• Measures to increase response rates:

incentives (e.g. using PayPal)

multiple follow-up contacts

other motivations
Key methodological issues
Involvement process
Target population
Complete respondents
Operational population
Partial respondents
Frame population
Starting units
Sampled units
Solicitation
Clickers
Absorbed units
Informed units
Aware units
Key methodological issues
Participation in web surveys
Respondent’s
characteristics
No researcher’s
control
Social
environment
Researcherrespondent
interaction
Technological
environment
Partially under
researcher’s
control
Survey design
Key methodological issues
4. Questionnaire design
• Question types and visual elements
• Advanced features of computerized questionnaires
• Inclusion of multimedia
• Problems and considerations:

standardization

impact on respondent’s answers

technical problems
measurement error
Key methodological issues
5. Post-survey adjustments
• Commonly utilized for correcting:

deviations from probability selections

non-coverage and non-response problems
• Importance for non-probability Internet surveys
• Promising approaches:

calibration methods

propensity score weighting

modelling and causal analysis
Related issues
1. Costs, errors and management
• The managerial process of survey implementation:

costs optimization

data quality

relations between data quality and costs
• Cost effectiveness of web surveys:

costs for increasing response rates

evaluations of costs in relation to errors
Related issues
2. Survey mode
• Web surveys vs. other modes:

lower response rates (Lozar Manfreda et. al., 2007)

comparable or higher level of validity and reliability
(e.g. Fricker et al., 2005; Chang & Krosnick, 2002)
Related issues
Mixing survey modes
Solicitation
Contact with the
respondent
Interview
PAPI, CAPI, CATI,
CASI,…
personal, telephone,
e-mail, mail,…
Centralized
data
management
Related issues
Selecting the right mixture
Structure of
the data
(MAR)
Willingness
to participate
Cost models
and impact of
environment
Modelling mixed
modes
Response
rates and
behaviour
Data quality
requirements
Optimal
combination
Related issues
Programing of the questionaire
User-friendly
interface
Paper
questionnaire
Standardized
questionnaire
description in QML
Internet survey
CAPI
CATI
CASI
Related issues
3. Mixing research methods
• Quantitative and qualitative research methods
• Flexible combinations of approaches (Morgan, 1998):

qualitative preliminary

quantitative preliminary

qualitative follow-up

quantitative follow-up
• New possibilities with the Internet
Related issues
Flexible on-line mixed methods (FOMM)
Analysis
Qualitative
method
Conclusion
Beginning
Quantitative
method
Related issues
4. Ethics, guides and standards
• New ethical dilemmas in survey research:

solicitation process, spam and privacy

data protection

children and minors

combining data…
• Development of new standards (e.g. ESOMAR, AoIR, MRA…)
Current and future applications
Applications
• Different applications according to:

type (internet access panels, on-line voting, intercept…)

organizations (official, academic, commercial)

topics (marketing research, polling opinions, health…)

complexity (simple one-shot surveys vs. complex integrated
data collection)
Current and future applications
Related areas

psychological research
 online experiments

paradata and online measurement

specific research fields (usability research, HCI...)

emerging areas (e-learning, e-government…)
Current and future applications
Trends
• Expansion of CASIC modes, options and mixtures
• Integration of devices (TV, mobile,..)
• Speech recognition and touch-screens
• Multimedia and interactivity
• Integration with other CADAC (GPS, paradata, observations,...)
• Specialisation and professionalisation