From face-to-face to web
in a cross-national study
Ineke Stoop
SCP/CST
ESS
Academic survey in 30+ European countries
Attitudes, values and beliefs
Biennial, start in 2002
1+ hour, face-to-face, effective sample size ~ 1500
Funders
• National Science organisations
• European Commission, Framework Programmes
• European Science Foundation
Management model
• Central coordination
• National implementation
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Aims ESS
To monitor and interpret
public attitudes and values
within Europe and to
investigate how they
interact with Europe’s
changing institutions
• Provide data on beliefs,
attitudes and values for
scientific and policy
making purposes
• Measure attitude change
in a changing Europe
To advance and consolidate
improved methods of crossnational survey
measurement in Europe
and beyond
• Collect data according to
highest standards
• Generate methodological
research
• Develop and disseminate
new best practices
• Develop and improve
social survey research
infrastructure in Europe
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Content questionnaires ESS
Every round
Socio-demo-graphic items
•
•
•
•
•
household composition
education
housing
occupation
social structure
Attitudes and behaviour patterns
•
•
•
•
•
•
•
religious affiliation and identity
ethnic and national identity
political and social trust
party affiliation and voting
behaviour
media consumption
value orientations
social exclusion
www.europeansocialsurvey.org
Round specific
R1
R2
R3
R4
R5
R6
Immigration
Citizenship, involvement and
democracy
Family, work and well-being
Opinions on health and careseeking
Economic Morality
Personal & Social Well-being
The Timing of Life
Ageism
Welfare attitudes
Family, work and well-being in
times of economic recession
Trust in police and courts
Personal & Social Well-being
Understanding, evaluation
democracy
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Access ESS
Round 1-5 available
Fully documented
Free access
Tables and graphics online, or download data
Training package for young researchers
Everything available
All deviations documented
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
ESS top data user countries (>50)
(June 2012) (N=48,973)
Germany
Belgium
United Kingdom
Slovenia
United States
Spain
Norway
Poland
Netherlands
France
Italy
Switzerland
Denmark
Austria
Portugal
Finland
Ireland
Sweden
Hungary
Estonia
Greece
Israel
Russian Federation
Ukraine
Czech Republic
Romania
Canada
Turkey
Slovakia
Latvia
Lithuania
Bulgaria
Australia
Philippines
China
Cyprus
Croatia
Iceland
Japan
Iran, Islamic…
Luxembourg
Korea, Republic of
Mexico
Brazil
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Participation
Austria
Belgium
Bulgaria
Croatia
Cyprus
Czech
Denmark
R1
R2
R3
R4
R5
22
26
25
31
28
Estonia
Finland
France
Germany
Greece
Hungary
Iceland
Ireland
Israel
Italy
Latvia
Lithuania
Luxembourg
Netherlands
Norway
Poland
Portugal
Romania
Russia
Slovakia
Slovenia
Spain
Sweden
Switzerland
Turkey
UK
Ukraine
Total
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Methods: high quality
Strictly random sampling
• No substitution, no proxy
Population: all inhabitants aged 15+
High target response rates
• 70% target response rate, max. 3% noncontact
Face-to-face fieldwork
• Gold standard, high response rates, universally applicable
• Increasingly expensive
• Rare in some countries
Pilots, pretesting and fieldwork monitoring
Analysis of cross-national differences
• Nonresponse bias, measurement error, sampling design effects
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Methods: central coordination
Central specifications for fieldwork
Close monitoring national fieldwork (in advance
and during fieldwork)
Sampling design
Translation procedures
Guidelines response enhancement
Guidelines interviewer training
Contact forms, response calculation
Context, events
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Can we compare countries?
Attitudes,
values, beliefs
Survey results
Survey design,
methodology,
implementation
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Harmonisation strategies
(Körner and Meyer, 2005)
Harmonised
concept
Harmonised
concept
Harmonised
concept
Measurement
procedure
Measurement
procedure
Measurement
procedure
National survey/
Specific concept
National survey/
Specific concept
National survey/
Specific concept
Input harmonisation
Ex-ante output
harmonisation
Ex-post output
harmonisation
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Output harmonisation
Ex post output harmonisation
• Use existing sources
Ex ante output harmonisation
•
Set up new survey (or develop new instrument)
Output harmonisation
• Best national quality
• Or, national survey tradition
• But, what about optimal comparability
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Output harmonisation: problems
•
•
•
•
•
•
•
•
•
Interview mode
Context question
Lay-out questionnaire
Question text
Introductory text
Answer scales
Handling don’t know
Size of boxes
Use of illustrations
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Input harmonisation
Input harmonisation
• Design new survey
• Optimal comparability
Or, are identical methods and instruments really
equivalent in different countries?
• But, what about optimal quality
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
From face-to-face to web in the ESS
Mixed mode research and experiments
Different survey modes over time
Different survey modes across countries
Mixed mode within countries: different composition
over time
Which modes:
• Face-to-face
• Telephone
• Web
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
ESS studies
Martin, Peter (2011) A Good Mix? Mixed Mode Data Collection and
Cross-national Surveys. ASK Research & Methods, Vol. 20 (1,
2011) 5–26.
Martin, Peter (2011) What makes a good mix? Chances and
Challenges of Mixed Mode Data Collection in the European Social
Survey. CCSS Working Paper No. 02.
Martin, Peter, and Lynn, Peter (2011) The Effects of Mixed Mode
Survey Designs on Simple and Complex Analyses. CCSS Working
Paper No. 02.
Roberts, Caroline (2007) Mixing Modes of Data Collection in
Surveys: A Methodological Review. ESRC National Centre for
Research Methods. NCRM Methods Review Papers, NCRM/008.
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
From face-to-face to web in the ESS
Coverage
Sampling
Nonresponse
Questionnaire design
Role interviewers
Measurement
Analysis
Cross-national comparison
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Coverage
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Internet access and broadband connections in
households, EU27 (%)
80
70
60
50
40
30
20
10
0
2006
2007
2008
Internet access
2009
2010
2011
Broadband connections
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Individuals who used the internet at least once
a week, 2011 (%)
100
90
80
70
60
50
40
30
20
10
0
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Individuals who used the internet on average at least once a
week, by sex, age and education, EU27, 2011 (%)
100
80
60
40
20
0
All
individuals
Men
Women
16-24 years 25-54 years 55-74 years
Low
education
Medium
education
High
education
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Sampling principles
Identical population definition
Complete coverage
• No upper age limits
• (Institutional population)
• (Non-native language speakers)
Probability sampling
Effective sample size
Optimal sampling design in each country
• Address, household, individual
• Address and household: selection of target respondent in the field
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
ESS R2 sampling frames (Häder and Lynn)
Country
Austria
Belgium
Czech Republic
Denmark
Estonia
Finland
France
Germany
Greece
Hungary
Iceland
Ireland
Luxembourg
Netherlands
Norway
Poland
Portugal
Slovakia
Slovenia
Spain
Sweden
Switzerland
Turkey
UK
Ukraine
Frame
Telephone book
National register
Address register UIR-ADR
Danish Central Person Register
Population register
Population register
None
Registers from local residents’ registration office
None
Central registry
National register
National Electoral Register
Social Security Register
Postal address list
Population register
National register of Citizens
None
Central Register of Citizens
Central Register of Population
Continuous Census
Register of population
Telephone register
Cluster of addresses
Postal address list
None
Remarks
Additional non-telephone households sampled in the field
Select streets, followed by field enumeration
Area-based sampling
Area-based sampling
Area-based sampling
Area-based sampling
Simple random sampling, or simple stratified sampling (no design effect)
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Comparison sampling frames
Achim Koch (GESIS)
In household and address samples: selection of
respondent by interviewer
Leads to overrepresentation of women in
household/address samples
Related to higher response rates
Close control on who answers the questionnaire in a
household even more problematic in web survey
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
ESS 1 – 4:
Sample of individuals: Bias X response rate
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
25
ESS 1 – 4:
Sample of hhs/addresses: Bias X response rate
r = .44
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
26
Nonresponse ESS
Response targets
• 70% response rate
• 3% noncontact rate
Standards in
• Response rate calculation
• Contact form
• Nonresponse analysis
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Response efforts and monitoring
Guidelines response enhancement
• Incentives/Brochure
• Refusal conversion
Call schedule requirements
• Minimum number of calls
• Timing of calls (evening, weekend)
Contact form
• Interviewer observations
• Call records
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Response rates ESS (selection all rounds)
80
70
60
50
40
30
20
10
0
CH
FR
ES
UK
DE
BE
2002
NO
2004
DK
NL
2006
PT
2008
SE
HU
SI
FI
PL
2010
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Response rates over time
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Nonresponse
Response rate highest for face-to-face
Functional illiteracy (differs across countries)
Nonresponse analysis
•
•
•
•
•
What happens in web
Noncontact, refusal, not able
Reasons for refusal
Interviewer judgment
Refusal conversion
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Questionnaire design
Questionnaire length
• Split questionnaire
Matrix design
Over time
Question design
• Unimode
• Developed for face-to-face
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Role interviewers
Select respondents (+)
Obtain cooperation (+)
Conduct interview
•
•
•
•
•
Explain questions (-)
Explain process (+)
Sensitive questions (+/-)
Validity answers (+)
Social desirability (-)
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Respondent understands questions
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Never
Almost never
Now and then
Often
Very often
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
7
Denmark
y = 2.8879x - 7.9073
R² = 0.4241
6.5
Norway
Sweden
6
Netherlands
Switzerland
Ireland
Estonia
Israel
UK
Belgium
Spain
Germany
Czech Republic
5.5
respondent: social trust
Finland
5
4.5
Slovenia
Ukraine
Russia
4
Romania
Cyprus
France
Croatia Poland
Slovakia
Latvia
Hungary
Greece
Portugal
Bulgaria
3.5
3
2.5
Turkey
2
3.5
4
4.5
interviewer: respondent understands questionnaire
5
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Measurement
Mode effects
Part of analysis
No simple adjustment
Mixed modes:
• Selection effects and mode effects
• Always comparison face-to-face
• Mode in analyses
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Cross-national comparison
Comparison over time
• Mode change or time change
Comparison across countries
• Mode difference or cultural difference
Mixed modes within countries
• Selection effect or mode effect
Mode distribution over time
Feasibility of web for ESS (length, type of questions)
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Thank you for your attention
www.europeansocialsurvey.org
4th International Workshop on Internet Survey Methods,
Statistics Korea,2012