A poor prognosis for survey methods? Assessing the
health of survey methods in social research
McCaughey Centre
Survey Methods Seminar
May 2008
The Topic
Survey methods have been a stalwart tool for social science research, however,
traditional approaches to survey methods are becoming increasingly difficult to
undertake. Letterboxes are overflowing with ‘junk’ mail, households are fatigued by
unsolicited telephone calls and electronic home protection systems are turning
homes into fortresses. With low response rates - what does this mean for ideals of
‘representativeness’ and what does ‘representativeness’ mean in contexts of
increasing diversity within populations?
For survey methods to adapt to these changing contexts for social science
research and to continue to provide reliable and valid insights into people’s lives
there is a need for innovative approaches.
Sample frames
Response Rates
Methodological Innovations
2
Credentials
About the Social Research Centre
•
Established in 2000 as a specialist provider of survey research
services to government and academia
•
Operates as a social research supplier under the umbrella of the
market and social research industry
• 25 permanent staff
• 180 telephone interviewers
• 110 seat Call Centre
•
Our aim is to enhance knowledge and contribute to informed
decision-making by undertaking high quality social and health
research
3
About the Social Research Centre
Services provided
•
large-scale survey design, conduct and management
•
longitudinal surveys
•
population health research
•
community attitudes research
•
campaign evaluation research
•
research amongst persons of culturally and linguistically
diverse backgrounds
•
Programme evaluation and policy development research
4
About the Social Research Centre
Research Areas
social capital and community
strength
environmental attitudes and
behaviours
community well-being and
connectedness
welfare reform and income support
population health monitoring
poverty and social exclusion
health promotion and social
marketing
parenting and families
cancer research
tobacco, drugs and alcohol
employment and unemployment
mental health
education and training
Racism
skills shortages
immigration and multiculturalism
carers and persons with
disabilities
violence
Indigenous research
crime, justice and victimisation
road safety
5
Random Digit Dialling (RDD)
6
RDD
Background
The rise of RDD and the demise of the Electronic White Pages (EWP)
•
Up until quite recently the Electronic White Pages was
considered complete and accurate for most survey research
purposes
•
The final version of the Desktop Marketing Systems (DtMS)
Electronic White Pages (EWP) product was released in July
2004 and is still widely used by market research companies
but is no longer used for high quality social research
•
Random Digit Dialing (RDD) methodologies are now the norm
rather than the exception
•
One of the main barriers to the efficient use of RDD has been
overcome by the development of automatic dialing
technologies
7
EWP versus RDD
•
Methods of RDD generation
Generation process involves
Launched in 2003
•
Undertaking a random selection of “seed” numbers from
Government’s
year plan for
Queensland
the 20045Electronic
White
Pages women
Initiatives to address areas where women experience inequity,
•
Retaining the eight
digit stem and deleting the last two
under-representation
and need
digits
•
Randomly generating two numbers to append to the
eight digit stem
•
Washing the resultant randomly generated ten digit
numbers against business listings
•
Send those records for which we get a confirmed
address a primary approach letter
8
EWP versus RDD Selected characteristics
Selected demographic characteristics of persons with listed and unlisted phone numbers
Characteristics
Listed
Unlisted
VPHS 2005 (n=7,560)
5,449
2,111
Employed
50.7
60.3
Retired
33.2
15.4
University educated
22.9
28.3
Household income <$40k pa
49.7
46.6
Language spoken at home - English only
89.8
84.5
Dependent children in household
32.0
42.3
Divorced / separated
11.0
16.4
Resident in area less than one year
1.7
18.9
Renting
10.6
30.9
9
EWP versus RDD Selected characteristics
Selected survey measures for persons with listed and unlisted phone numbers
Characteristics
Listed
Unlisted
VPHS 2005 (n=7,560)
5,449
2,111
Holds private health insurance
52.5
43.6
Daily smoker
14.6
22.0
Has an alcoholic drink every day
18.9
13.6
Sought professional help for mental health related illness
8.6
12.2
Could raise $2,000 within 2 days in an emergency
85.0
79.4
ICVS 2004 (n=6,000)
4,730
1,270
Victim of at least one crime in last 5 years
51.3
58.0
Victim of violent crime in last 5 years
17.5
24.4
Feels safe waiting for public transport after dark
60.7
55.9
All of the results are statistically significant at the 95% confidence interval.
10
RDD
•
Improved representation
RDD samples can access different persons in terms of:
Launched in 2003
•
Age distribution
Government’s 5 year plan for Queensland women
•
Demographic characteristics
Initiatives to address areas where women experience inequity,
under-representation
and needor behavioural variables
•
Key usage, attitudinal
•
Inclusion of persons with unlisted numbers in the sample
universe by using RDD techniques can achieve a more
representative sample of the general population
•
BUT IN THE SPACE OF A DECADE THE
REPRESENTATIVENESS OF SAMPLES
OBTAINED FROM RDD FRAMES IS STARTING
TO BE ERODED…
11
RDD
Limitations of RDD frames
•
Little or no coverage of areas poorly represented in
source
listing
e.g. emerging growth areas
Launched
in 2003
•
The
emergence
of plan
‘mobile
only’ households.
Non
Government’s
5 year
for Queensland
women
coverage
ofaddress
households
without
a landline
to be
Initiatives to
areas where
women
experience(used
inequity,
under-representation
and need
1-2%
but now probably
around 13%)1,2
•
Decreasing ability to obtain address matches and send
out primary approach letters
1
12.8% of households in the US in 2006. National Health Interview, Public Opinion Quarterly,
Vol. 71
2
One in four adults aged 18 to 24 years, Ibid.
12
Response Rates
Are response rates declining?
Generally, probably yes but not inevitably so
Launched in 2003
In market research it has been predicted that …
Government’s 5 year plan for Queensland women
“Reach is increasing but co-operation is declining and nonInitiatives
address areas
wheretowomen
experience
inequity, rate
contact to
is increasing,
leading
a 3% decline
in response
under-representation and need
per annum” (Bednall, 2000).
Is the same true in social research …Possible yes but not
inevitably so and certainly not to the same extent …. It
depends on how hard you work…
13
Response Rates
Selected findings
Victorian Population Health
2005Survey
2006
Response Rate (%) 67.9
61.5
2007
69.3
Health system
Feb-05 May-06 Aug-06 Nov-06 Feb-07 May-07 Aug-07 Nov-07
Government’s
5 year plan
59.3
59.2women
60.8
54.7
65.9
61.6
Response
Rate (%)
67.9
61.5for Queensland
Smoking and Health Survey
2004
Response Rate (%)
55.2
2005
54.9
Community Attitudes to Road Safety
Response Rate (%)
NTS
Response Rate (%)
National Drug Survey
Response Rate (%)
2002
43.4
2003
43.4
2004
38
2006
43.3
2007
51
2003
68
2004
64
2004
52.3
2005
44.2
2005
73
2006
30.8
2006
66
2007
40.2
2007
42
14
Response Rates
The current state of play
Overall only marginally lower than they were several years ago
or at least not substantially lower. Certainly no precipitous
Launched in 2003
decline.
This has been achieved by:
•
Government’s
5 year plan for Queensland women
Extended
call regimes
•
Longer fieldwork periods
•
Increased use of Primary Approach Letters
•
Increased reliance on bilingual interviewing
•
More sophisticated interviewer training
•
Leaving messages on answering machines
•
Increased reliance on survey sponsors to promote their surveys
•
Creating an internet presence for surveys
•
More calls to mobiles to obtain interviews with selected respondents
15
Population Research
The future?
• Access to the Integrated Public Number Data Base is needed to
overcome these deficiencies
• Mixed
/ multi modes
data
Government’s
5 yearof
plan
for collection
Queensland women
• On line access panels1
• Access to mobile phone directories
•
Advocacy is needed
•
Further research into advances in survey methodology is
needed
•
Survey methods are certain to adapt to the changed
environment as the alternative is sub standard population
research
1.
2006 Census, 58% of households had an internet connection, 37% had broadband.
16
Population Research
The future?
A poor prognosis for survey methods?
No, just continued innovation …
17
CATI
Computer Assisted Telephone Interviewing
CATI
Quick

Economical

Good response rates
??
Direct data entry
Computerised survey management


system (SMS)
Random Digit Dialling (RDD)
Reaches silent numbers and recent
connections
?
More disconnected, non-household, fax
__
Not completely random
??
18
RDD
Random Digit Dialing
The proceedings from the first CATI Population Health
Forum in 1998 record the following advantages of CATI and
RDD:
CATI
 quick
 economical
 good response rates
 direct data entry
 computerised survey management system (SMS)
Random Digit Dialling (RDD)
 reaches silent numbers and recent connections
 more disconnected, non-household, fax
 not completely random
19