Telephone Surveys and the Mobile Phone Only Population
Workshop Program - 17 July, 2012
The Institute for Social Science Research and the Social Research Centre
Telephone Surveys and the Mobile Phone Only population
Workshop Program
Page i
Contents
OVERVIEW OF PROGRAM ............................................................................................................................. 1
DETAILED PROGRAM ..................................................................................................................................... 2
PRESENTERS .................................................................................................................................................. 7
The Institute for Social Science Research and the Social Research Centre
Telephone Surveys and the Mobile Phone Only population
Workshop Program
Page 1
Overview of program
Start time
Session
Speaker
9:00
Coffee and registration
9:30
OPENING SESSION
9:30
Introductory comments and recap of last workshop
Darren Pennay
10:00
NSW Population Health Survey - Dual frame results
Margo Barr
10:30
Mental health literacy and stigma - Dual frame results
To be advised
ACMA Mobile Phone Only study 10:45
Surveying
Australia’s communications users: Integrating the
Jennifer Newton
mobile only population into a single survey vehicle
11:00
Morning Tea
11:15
MORNING SESSION
Social Research Centre Dual-frame Omnibus
11:15
Methodology
Darren Pennay
11:35
Transport and Road Safety (Selected findings)
Michael Nieuwesteeg
11:55
Gambling (Selected findings)
Alun Jackson
12:15
12:35
13:00
Sexual activity and sexual experiences (Selected
findings)
Fieldwork and financial considerations in undertaking
dual-frame surveys
Anthony Smith
Graham Challice
Lunch
AFTERNOON SESSION
Methodological Issues
13:30
Optimal approaches to weighting
Michele Haynes
13:50
Measures of data quality across the frames
Paul Lavrakas
14:10
It’s not just the mobile phone frame you need to
worry about
14:30
Latest developments and emerging issues
15:00
Afternoon Tea
Graham Challice
Paul Lavrakas
FINAL SESSION
15:15
Advocacy initiatives since last workshop
Darren Pennay
15:30
Summing up and next steps.
Mark Western
15:45
Close
The Institute for Social Science Research and the Social Research Centre
Telephone Surveys and the Mobile Phone Only population
Workshop Program
Page 2
Detailed program
9:00am
Registration and coffee
OPENING SESSION
9:30am
Opening Remarks and Recap of last workshop
Darren Pennay
This workshop follows on from the successful workshop held in March 2011, the purpose of which was to
raise awareness of the gap in the coverage of traditional landline surveys due to the increasing proportion of
the population living in ‘mobile phone only’ households. A main focus of the 2011 workshop was to learn
how the U.S. survey research community responded to this issue and to explore the findings of what was
believed to be the first ever Australian dual-frame survey (the Social Research Centre / ISSR Dual-frame
Demonstration Survey). The workshop concluded that the exclusion of the ‘mobile phone only’ population
from traditional landline surveys was a source of non-ignorable bias. It was agreed that there was need for
advocacy and capacity building to enable the Australian survey research community to better respond to
some of these emerging methodological issues.
The 2012 workshop will pick-up on these themes and also have a very practical focus with regard to the
developments in survey methodology which affect this area. Topics for discussion include:
•
Updating participants on the latest Australian developments in this area
•
Presenting the findings from the Social Research Centre’s dual-frame omnibus survey and other
recently commission dual-frame surveys
•
Presenting a summary of the key developments in this area based on the relevant papers /
presentations from the preceding International Social Science Methodology Conference. This will
include some very practical sessions on data quality, sample design and weighting.
•
A discussion of emerging issues and latest developments from the US, and
•
The continuing need for advocacy in this area.
10:00am
NSW Population Health Survey
Margo Barr
This presentation will cover details about how mobile only phone users were incorporated into the NSW
Population Health Survey (an existing landline RDD health survey on health behaviours, health status, and
social determinants) using an overlapping dual-frame design in the first quarter of 2012.
Data collection was kept as consistent as possible with the previous years’ landline only RDD surveys and
between frames. Over 3000 interviews were completed with approximately a third from the mobile frame and
two thirds from the landline frames.
Presentation will include information on:
•
Data collection considerations, sample frame performance, costing, acceptance by staff,
respondents profile and response and participation rates
The Institute for Social Science Research and the Social Research Centre
Telephone Surveys and the Mobile Phone Only population
Workshop Program
•
Page 3
Weighting options, assumptions required and final strategy used as well as some preliminary results,
and
•
What is currently being explored to assess the impact of including mobile only phone users (through
a dual-frame) on the integrity of the time series.
10:30am
Mental Health Literacy and Stigma
To be advised
The Orygen Youth Health Research Centre at the University of Melbourne commissioned the Social
Research Centre to undertake a dual-frame telephone survey to measure aspects of mental health literacy
and mental health stigma. The 2011 survey was the third in a series with previous surveys undertaken in
1995 and 2003. The overall survey design featured two main components. The main survey comprised a
national sample of approximately 6,000 persons aged 15 years and over with 4,500 of these interviews
obtained from a fixed-line RDD telephone sample and 1,500 from a mobile phone RDD sample. The overall
design also featured a separate survey targeting young people aged 15 to 25 years. This ‘Youth Survey’
comprised 1,800 interviews from a fixed-line RDD sample and 1,200 from a mobile phone RDD sample.
This paper will describe and evaluate the survey methodology and profile the respective samples. The
paper will also cover differences in levels of psychological distress and mental health literacy of the landline
and mobile phone samples. Mental health literacy being defined as ‘knowledge and beliefs about mental
disorders that aid in their recognition, management and prevention’.
10:45am
Australian
Communications
and
Media
Authority
–
Surveying
Australia’s
communications users: Integrating the mobile only population into a single survey
vehicle
Jennifer Newton
Each year, the Australian Communications and Media Authority undertakes a survey of Australian adults to
gauge levels of take-up and use of, and community attitudes toward, communications technology. Data from
this survey feeds into a range of ACMA functions including regulatory reviews, research reports and the
annual flagship publication, the Communications report.
This session will outline the techniques used to reach mobile-only consumers for the past two ACMA
consumer surveys. It will reveal the findings of these surveys to show that far from being a permanent
communications choice, for many Australian consumers, going mobile-only is a temporary phase based on
lifestyle and financial pressures.
11:00am
MORNING TEA
The Institute for Social Science Research and the Social Research Centre
Telephone Surveys and the Mobile Phone Only population
Workshop Program
Page 4
MORNING SESSION
11:15am
Social Research Centre Dual-frame Omnibus (Methodology)
Darren Pennay
One of the action points arising from the 2011 workshop was to assess the level of interest among the
research community in supporting a Dual-frame Omnibus Survey. The objectives of such as survey were to
further developing our knowledge of dual-frame surveys, and to provide subscribers with a better
understanding of any biases associated with the ongoing exclusion of the mobile phone only population from
their established survey estimates.
It eventuated that there was sufficient interest in this topic to enable the Social Research Centre to undertake
a Dual-frame Omnibus Survey in December, 2011. This session will present a methodological overview of
this survey and act as a segue for the following presentations which focus on specific topic areas – Transport
and Road Safety (Michael Nieuwesteeg), Gambling (Alun Jackson) and (Sexual activity and Sexual
experiences (Anthony Smith).
11:35am
Transport and Road Safety
Michael Nieuwesteeg
The Victorian Transport Accident Commission (TAC) conducts a number of surveys among Victorian road
users that investigate a range of road safety issues. In recognition of the decreasing population coverage of
landline telephones, declining response rates in landline-based surveys, the rise of on-line research panels
and the proliferation of mobile phones, the TAC has embarked on a program of renewal of its range of social
surveys.
The TAC purchased several questions on the Social Research Centre’s Dual-frame Omnibus Survey
covering topics such as transport usage and driving patterns, drivers licence and crash history, attitudes to
selected road safety measures and self-reported driving behaviours in relation to issues such as speeding
and drink driving. This presentation will explore the differences in the survey results between the landline
and mobile phone only respondents and consider the broader issues for general community road safety
surveys.
11:55am
Gambling
Alun Jackson
This presentation will argue that in terms of gambling behaviour, the dual frame methodology provides a
more nuanced picture than in landline sample frames which have been the norm in problem gambling
population studies. The ‘mobile phone only’ subset of the total mobile sampling frame, including dual landline
and mobile users, were significantly less likely as a group to have gambled in the last 12 months, but
nevertheless significantly more likely to endorse items in a lifetime measure of problematic gambling. In
addition, levels of anxiety, depression and hazardous drinking were also found to be higher in the mobile
phone sample.
The Institute for Social Science Research and the Social Research Centre
Telephone Surveys and the Mobile Phone Only population
Workshop Program
12:15pm
Page 5
Sexual activity and Sexual experiences
Anthony Smith
The first Australian Study of Health and Relationships (ASHR1) was conducted in 2001/2, involved
telephone interviews with 19,307 people aged 16-59 and provides much of the contemporary evidence base
concerning sexuality and sexual health in Australia. The second such study (ASHR2) will be competed in
2012/13 and will similarly involve telephone interviews with 20,000 people aged 16-69.
Whereas sole reliance on a landline sample in 2001 could be justified, sampling for ASHR2 provides
different challenges.
12:35pm
Fieldwork and financial considerations in undertaking dual-frame surveys
Graham Challice
There has been an increasing number of dual-frame surveys commissioned since the Dual-frame
Demonstration Survey in September, 2010.
This session presents the acquired learnings from these
surveys with a focus on research ethics, call procedures, response dynamics, response maximisation, and
the cost of undertaking dual-frame surveys.
1:00pm
LUNCH
AFTERNOON SESSION
1:30pm
Optimal approaches to weighting
Michele Haynes
Surveying mobile phone users introduces new complexities in coverage and sampling, nonresponse,
measurement and weighting that survey researchers must address. Through a comparison of the fixed line
and mobile phone samples, this presentation illustrates some of the consequences of failing to include
mobile phone users in telephone surveys, and examines and evaluates some of the different approaches to
combining results from these samples in dual-frame designs.
1:50pm
Measures of data quality across the sample frames
Paul J. Lavrakas
There are myriad reasons to hypothesize that data quality from those interviewed via a mobile device will be
inferior to that gathered from those interviewed on a landline. This session will address what we know about
these issues and point out the circumstances under which differential data quality is most likely to result.
2:10pm
It’s not just the mobile phone frame that you need to worry about?
Graham Challice
This presentation switches the focus from the mobile phone component of dual-frame surveys back to the
landline frame and documents the findings from a major review of the entire Victorian RDD landline sample
frame. The Social Research Centre commissioned this review to test all non-working residential numbers in
Victoria which had not been tested for six months or more (some 2.8 millions numbers). The process
resulted in a 19% increase in the pool of working residential landlines in Victoria (from 1,387,332 numbers to
1,651,222). This updated sample frame was then used for the Victorian Population Health Survey (VPHS).
The Institute for Social Science Research and the Social Research Centre
Telephone Surveys and the Mobile Phone Only population
Workshop Program
Page 6
Very preliminary interim results from the 2011-12 VPHS are used to look at the profile of the newly included
telephone numbers.
2:40pm
Latest developments and emerging issues in dual-frame surveys
Paul J. Lavrakas
Since the 2010 Cell Phone Surveying Task Force report was released by AAPOR, survey researchers in the
USA have continued to learn a great deal about how to design the most cost effective dual frame RDD
surveys for the purposes to which those must be put. This session will provide an update of what is now
known (and suspected) about these “best practices” using newly released findings from the just completed
2012 AAPOR annual conference. This will include new information about several of the issues I discussed
in the 2011 Melbourne workshop, including the optimal allocation of interviewing across the two frames, and
how to gather reliable data to screen and weight dual frame RDD surveys.
3:00pm
AFTERNOON TEA
3:15pm
Advocacy undertaken since the last workshop
Darren Pennay
3:30pm
Summing up and next steps
Mark Western
3:45pm
CLOSE
The Institute for Social Science Research and the Social Research Centre
Telephone Surveys and the Mobile Phone Only population
Workshop Program
Page 7
Presenters
Darren Pennay - Managing Director and Head of Research at the Social Research Centre, Adjunct
Professor, University of Queensland, Institute for Social Science Research
Darren Pennay is the founder and Managing Director of the Social Research Centre and an Adjunct
Professor with the University of Queensland’s Institute for Social Science Research (ISSR). Darren has
worked in social research and survey design since 1984. He has previously worked with the Australian
Institute of Multicultural Affairs, Australian Bureau of Statistics, Reark Research and the Wallis Consulting
Group.
Darren formed the Social Research Centre late 2000 to develop a ‘centre for excellence’ in
government and social research and to offer government and academic researchers the choice of working
with an agency specialising in this area of research. In the twelve years since, the Social Research Centre
has grown from a single person start up entity (working out of a spare bedroom) to Australia’s leading
supplier of survey research services to government and academic institutions.
Darren’s specific interests include survey methodology, large-scale survey design and management,
longitudinal surveys, population health surveys and community attitudes research. He is a full member of
the Australian Market and Social Research Society and has QPMR (Qualified Practicing Market Researcher)
accreditation. Darren is also a member of the American Association of Public Opinion Research, the World
Association of Public Opinion Research, the European Survey Research Association and the Australasian
Evaluation Society.
Margo Barr - Principle Epidemiologist and Manager, Health Behaviour Surveillance, Centre for
Epidemiology and Evidence, NSW Health
Margo Barr is the Principal Epidemiologist and Manager, Health Behaviour Surveillance, Centre for
Epidemiology and Evidence, NSW Ministry of Health. She has worked with the NSW Ministry of Health since
2001 running their population health survey program. Margo is an epidemiologist with over 20 years of
experience in public health and surveillance methodology having also worked at PNG Department of Health,
Queensland University of Technology, Queensland Health and the Australian Government, Department of
Health and Ageing. She has a BSc, a MPH and, although mainly working in the public sector, has published
over 30 research articles on public health, service provision, surveillance and survey methodology. She is
currently completing her PhD in survey methodology at the Centre for Statistical and Survey Methodology,
University of Wollongong. Margo is also a member of the Australasian Epidemiological Association, the
International Epidemiological Association and the European Survey Research Association.
Nicki Reavley?
The Institute for Social Science Research and the Social Research Centre
Telephone Surveys and the Mobile Phone Only population
Workshop Program
Page 8
Jennifer Newton - Policy Analyst, Communications Analysis, Australian Communications and Media
Authority
Jennifer Newton has been a researcher and policy analyst at the ACMA for over five years, specialising in
quantitative design, questionnaire development and sampling techniques in the area of communications
technology adoption and behaviour. Her research work has explored internet usage, unsolicited
communications, e-Security issues and emergency telecommunications.
As statistical liaison officer for the ACMA, Jennifer also acts as a primary contact between the ACMA and the
Australian Bureau of Statistics with respect to business survey development, assisting staff to design fit-forpurpose questionnaires that minimize the burden on Australian businesses while achieving specific research
objectives.
Jennifer holds a PhD in Geography and Archaeology from the University of Melbourne.
Michael Nieuwesteeg - Research Manager Transport Accident Commission
Michael Nieuwesteeg (B.Science (Honours - Statistics) is Research Manager for Road Safety and Marketing
at the Transport Accident Commission. He is responsible for the evaluation of the TAC's marketing and road
safety program, which is based largely on social research and analysis of administrative datasets. He has
worked for over ten years as an analyst in a range of government agencies predominantly in crime and
accident prevention.
Alun Jackson - Director, Problem Gambling Research and Treatment Centre, University of Melbourne
Alun is Professor and Director of the Problem Gambling Treatment Research Centre at the University of
Melbourne, a Professorial Fellow of the Melbourne Graduate School of Education, an Honorary Research
Fellow of Peking University and an Honorary Research Fellow of the Centre on Behavioural Health at the
University of Hong Kong. Professor Jackson’s previous positions include: Chair of Social Work at the
University of Melbourne from 1997 to 2007; Chairman of the Centre for Migrant and Intercultural Studies at
Monash University; and Director of the HIV/AIDS Socio-behavioural Research Unit at the University of
Melbourne. He is the President of Drummond Street Services, a not-for profit family mental health service.
He has been involved in the design and direction of many large scale gambling research projects including a
study on family violence and problem gambling; a study of risk factors for the children of problem gamblers;
an evaluation of the Victorian Gambler’s Help service; a study of the social and economic impact of gambling
in Tasmania; and the development of the NH&MRC-endorsed Clinical Guidelines for Screening, Assessment
and Treatment of Problem Gambling.
Anthony Smith - Professor, Faculty of Health Science, Australian Research Centre in Sex, Health and
Society
Anthony Smith is a leading authority on the sexual behaviour and sexual health of the Australian population.
He is a Professor at the Australian Research Centre in Sex, Health & Society at La Trobe University.
The Institute for Social Science Research and the Social Research Centre
Telephone Surveys and the Mobile Phone Only population
Workshop Program
Page 9
Graham Challice - Executive Director, Head of Survey Operations, the Social Research Centre
Graham has over 21 years experience in market and social research operations with a reputation for
effectively managing large-scale, resource intensive, high stakes government research projects to industrybest standards. Graham is a full member of the Australian Market and Social Research Society and has
previously held positions with Wells Research, Reark Research, AC Nielsen and NCS Pearson. His
expertise is in large-scale survey management, complex multi-mode projects and the development and
execution of quality operational processes, with specific skills in response maximisation techniques,
interviewer performance and enhanced data quality.
Michele Haynes - Associate Professor, Social Statistics, Program Leader, Research Methods &
Social Statistics
Michele Haynes is Leader of the Research Methodology and Social Statistics Program in the Institute for
Social Science Research (ISSR), The University of Queensland. She is also Executive Director of the
Queensland node of the Australian Social Science Data Archive. Michele has been the senior statistician in
ISSR (and previously the UQ Social Research Centre) since 2003 where she has participated in and
managed academic and numerous contract research projects for government agencies. Michele has had
extensive experience in the application of statistical methodology to both cross-sectional and longitudinal
datasets arising in the social sciences. Her current research interests include developing methodologies for
analysing multilevel and longitudinal social survey data with applications to understanding transitions
throughout the life course including partnership, employment and housing transitions and interrelationships
with important outcomes.
Paul J. Lavrakas - Independent Research Consultant
Paul is a research psychologist, and currently is serving as a methodological research consultant for several
organizations, including Google and The Associated Press. From 2000-2007 he was Vice President and
chief methodologist for Nielsen Media Research. He was a tenured Professor at Northwestern University
(1978-1996) and Ohio State University (1996-2000), and was the founding faculty director of the
Northwestern University Survey Lab (1982-1996) and the OSU Center for Survey Research (1996-2000).
Among his publications, he has written two editions of a widely read book on telephone surveying
(Telephone Survey Methods: Sampling, Selection and Supervision) which were published by Sage in 1987
and 1993. Since then he has authored several handbook chapters on telephone survey methods, the most
recent appearing in the Handbook of Survey Research (Elsevier, 2010). In 2007, he was the editor of a
special issue of Public Opinion Quarterly on mobile phone surveying in the USA. He also was co-editor for
Advances in Telephone Survey Methodology (Wiley, 2008). He also organized and chaired the 2003 and
2005 U.S. Cell Phone Sampling Summits and the 2007-2008 and 2009-2010 AAPOR Task Forces on Cell
Phone Surveying. Among his other publications, Dr. Lavrakas is the editor of the Encyclopedia of Survey
Research Methods published by Sage in 2008. He is the current President of the American Association for
Public Opinion Research (AAPOR), and looks forward to reconstituting AAPOR’s Cell Phone Surveying Task
Force in 2013, with anticipation of issuing a report from that initiative by early 2014.
The Institute for Social Science Research and the Social Research Centre
Telephone Surveys and the Mobile Phone Only population
Workshop Program
Page 10
Mark Western - Director of the Institute for Social Science Research
Mark Western is Director of the Institute for Social Science Research at The University of Queensland and a
Fellow of the Academy of Social Sciences in Australia. He is a sociologist who has previously worked at the
University of Tasmania and the Australian National University and held visiting appointments at the
University of Wisconsin-Madison, the University of Manchester and the Institute of Education. He has
published on topics relating to social and economic inequality, political behaviour, and family demography
using Australian and international academic and government survey data. He has also led and participated
in a number of research projects for Federal government departments and agencies. In addition to his
substantive research he also has research interests in survey methodology and statistical methods for
categorical data analysis, and longitudinal and multilevel data structures.
The Institute for Social Science Research and the Social Research Centre