Sport England
Methodology of measuring sports participation
Methodology of measuring sports participation
Consultation document
August 2015
1
Sport England
Methodology of measuring sports participation
2
Contents
Introduction ......................................................................................................................................... 3
Background ........................................................................................................................................ 4
Reasons for Change ........................................................................................................................... 5
Pilot studies ........................................................................................................................................ 6
Proposal ............................................................................................................................................. 7
What will this mean for me? ................................................................................................................ 9
Timing............................................................................................................................................... 10
Quality Commitment ......................................................................................................................... 11
How Can I Respond? ........................................................................................................................ 12
Sport England
Methodology of measuring sports participation
Introduction
Sport England wishes to consult with users and other interested parties on future
arrangements for the measurement of sports participation.
This document and an ONLINE SURVEY forms this consultation. Please click on this link to
respond to the consultation.
If you have any questions, please contact:
research@sportengland.org
OR
Research, Evaluation and Analysis
Sport England
1st Floor
21 Bloomsbury Street
London
WC1B 3HF
The consultation will be open for 8 weeks, closing at midnight on 14th October 2015.
3
Sport England
Methodology of measuring sports participation
Background
Sport England is an insight-led organisation tasked with getting more people active. To
create the right conditions for this we need a broad and deep knowledge of how much sport
people play, what sports they play and which sports they’re interest in playing more of. This
will inform our decision around who we invest in, and our understanding of how sport can
deliver wider objectives.
Through our commitment to measuring sports participation over the past decade, we have
unparalleled data both internationally and amongst other sporting and cultural agencies in
the UK. Central to this commitment has been Sport England’s Active People Survey (APS).
APS has existed since 2005/06 and over that time, its design has remained largely
unchanged. The cross sectional, random digit dial (RDD), landline telephone design has
provided an effective mechanism to measure a wide range of sports and provide local
authority level data.
We are committed to continue to lead in this area over the next five years and beyond, as
Sport England begins to consider its strategic priorities for the period until 2021. We believe
there is a natural opportunity to review both what we measure (to ensure it aligns with the
Government’s, and our own, new strategy) and how we measure it (reflecting the best
available methods). This consultation is specifically about how we undertake the collection of
data. In parallel, both Sport England and DCMS are undertaking a wider consultation to
establish the strategic priorities for community sport over the next five years.
Sport England previously consulted users of APS data about modernising the survey design
in 2013/14. There was overwhelming support for the introduction of a new survey design that
reflected more modern methods of data collection (e.g. mobile phone and online). However,
respondents also emphasised the importance of retaining consistency with previous results.
In response to the consultation, Sport England carried out further testing of different modes
of data collection. With this work now completed, we believe it is the right time to set out
plans (approach and timing) for the future measurement of sports participation.
4
Sport England
Methodology of measuring sports participation
Reasons for Change
APS is a large landline telephone survey (annual sample size around 165,000) which
focuses on sports participation and physical activity amongst people aged 14 years and over
in England. The sample size is large enough to be able to measure robustly lower
participation sports at the national level as well as providing headline sports measures at
local authority level.
However, given changing patterns of telephone use, surveys based purely on residential
landlines do not have complete or even coverage of the target population. Whilst all of the
validation, testing and development we have undertaken gives us confidence the current
design remains fit for purpose, changes to our business requirements and the changing use
of technology mean new and alternative approaches should be considered as we look to
ensure the right arrangements are in place for the next 5-15 years.
Survey designs reflecting newer methods, such as online and mobile phone, have been
explored and proven to be viable alternatives to meet our strategic and performance
measurement requirements in the future.
5
Sport England
Methodology of measuring sports participation
Pilot studies
Having tested alternative approaches alongside the current APS model, Sport England
believe there are other survey designs (briefly described below) that are viable and can meet
Sport England’s requirements.
Dual-frame (mobile and landline) design
The trials undertaken used random digit dial telephone interviews with both a landline
and mobile sample. Through testing this dual frame approach, it was concluded that:
 It is feasible to carry out a suitable sport and physical activity survey using a
mobile RDD sample
 The dual frame design provides coverage of 99% of the target population
 The mobile phone sample was effective in reaching demographic groups that
were typically under-represented in the landline sample achieving a more
balanced sample overall
 However, the response rate achieved on the mobile phone interviews was lower
than the landline interviews
 The cost per interview for a dual-frame RDD survey is higher than the cost per
interview for a RDD survey based on landline numbers only.
CAWI (Postal-to-web surveys)
The trials undertaken used an online data collection method based on an approach
trialled for the Cabinet Office’s Community Life Survey. With this method, a sample of
addresses was drawn (using random methods) from the Postcode Address File and
the target population at the sampled addresses were invited, by letter, to complete
the survey online. Respondents were also provided with (by request or with the
second survey reminder) a shortened paper version of the questionnaire, enabling
the offline subset of the population to also participate in the survey. A monetary
incentive, conditional on completion of the survey- was offered. The trials provided
evidence that:
 It is feasible to carry out a suitable sport and physical activity survey using online
data collection
 The online design provides coverage of 99% of the target population
 The online sample is more effective in reaching demographic groups that are
typically under-represented in the landline sample achieving a more balanced
sample overall
 However, the response rate achieved by the online design was lower than the
landline telephone interviews
 The cost per respondent of the online survey was broadly equivalent to the cost
of landline telephone interviews
 Online data collection will have an impact on the level of sports participation
reported.
Whilst not exhaustive, the pilot work gives us confidence that it would be possible to develop
and implement a high quality alternative to the current APS design that meets the needs of
Sport England and its key partners.
6
Sport England
Methodology of measuring sports participation
Proposal
Sport England is planning to appoint an appropriate social research agency, and to work
with them on the development and implementation of new arrangements to measure sports
participation. To manage the risk of any change and to ensure the availability of robust
sports participation data, Sport England is planning a period of parallel running whereby the
current APS survey design continues alongside the development and implementation of the
new survey arrangements.
Through the development work undertaken, it is clear there are several alternative designs
that could be developed and implemented. Rather than prescribe a particular technical
solution, Sport England believes it is more appropriate to state its measurement
requirements and priorities and allow leading social research agencies to use their
experience and expertise to develop proposals that meet these.
Input gathered through this consultation process will help to form those requirements and
priorities. We believe this approach will encourage innovation and result in the best solution
being identified and implemented.
On this basis, Sport England has developed the following requirements:
The new survey will be required to:
• Accurately track regular participation at a population level, by:
– Demographic group (age, disability, gender, ethnicity, socioeconomic status,
working / educational status, faith, sexuality, other demographics)
– Local area (The survey will need to provide credible estimates at a national,
regional, county sports partnership, and local authority level. APS has been
structured to provide the same sample size for every local authority area in
England. Whilst credible estimates will be retained for all local authority areas,
we will be considering the ability to have different sample sizes depending on
size of authority.
– Sport / activity (In addition to overall participation, provide accurate and
precise estimates of regular participation for a range of specific sports and
activities. For larger sports and activities, this will include additional detail
about specific forms / disciplines of the sport.)
Taken together, this is likely to mean the new survey will be of a similar size and scale
to APS.
•
Measure the overall activity of individuals, rather than their involvement in each
sporting activity separately. This will be kept under review and develop in response
to the requirements of the new strategy, but key considerations are likely to include:
– Understanding the full range of sporting behaviour from regular, wellestablished patterns to infrequent and sporadic activity.
7
Sport England
Methodology of measuring sports participation
–
–
–
–
•
Better recognition that the nature of participation in different sports can vary
significantly based on a range of factors (duration, frequency, formality,
setting, etc.).
How people combine different sports and activities at a given moment in time
as part of an active lifestyle.
How individuals combine different sports and activities over the course of a
year as part of an active lifestyle.
How and why participation in sport fluctuates over time (e.g. motivation,
illness/injury, constraints of time, access to opportunities, etc.) and that
sometimes actual behaviour in a given period may not represent typical
patterns of participation.
Measure the contribution of sport and physical activity to public health. We
anticipate the new survey will continue to include questions about walking and all forms
of cycling (not just sport / recreational) and are keen to continue to work positively with
Public Health England on areas of common interest.
The design of the new survey will:
• Be high quality, providing statistically robust estimates of participation that have the
confidence of key stakeholders by using sound statistical methods and principles
quality assured by professional statisticians1.
• Remain fit for purpose over the medium to long term. As previously noted, Sport
England wants to establish new survey arrangements that will remain fit for purpose for
the next 5-15 years.
• Minimise the survey burden on respondent. We are very conscious of the burden
surveys place on respondents. As part of new survey arrangements, we want to
minimise this.
• Support a transparent and easily understood measurement framework. As part of
implementing any new survey arrangements, Sport England will review and seek to
develop and improve the reporting and dissemination of results (e.g. reports, tables,
charts, interactive tools). We believe it is appropriate to retain the current six-monthly
cycle of reporting.
1 Considerations
of quality will include: population coverage; sampling approach; sample size and
profile (geographic and demographic); response rates; questionnaire design and testing; fieldwork
procedures; data processing and weighting; reporting and dissemination
8
Sport England
Methodology of measuring sports participation
What will this mean for me?
New survey arrangements will be developed and implemented with extreme care and closely
monitored following implementation. Should any issues arise, we would take the necessary
action to manage them.
A new survey, designed to support a new strategy, will mean new data and information
being presented in new ways. Through effective consultation and dialogue, we will ensure
we deliver outputs that are of value and relevance to stakeholders.
Sport England is keen to emphasise that they regard this as a new survey, not the
continuation or evolution of the current Active People Survey. Whilst we would like to get
continued value from the time series data we have, our primary focus is on making the right
choice for the next 5-15 years.
9
Sport England
Methodology of measuring sports participation
Timing
Sport England is currently in the process of selecting an appropriate social research agency
to work with on the development of the new survey. Once appointed, we will work with this
agency to implement new survey arrangements in the autumn. We intend to appoint the
supplier to oversee an initial development and implementation period which, if successful,
will then lead to ongoing delivery over the five year period of the new strategy for sport (until
2020/21). Any responses received from users of sports participation statistics will be
important as we ensure the arrangements are put in place.
We will continue APS until 31 March 2016. Through this period of parallel running, we will be
able to continue to provide sports participation data, manage the risk of any changes, and
understand the impact of any changes.
10
Sport England
Methodology of measuring sports participation
Quality Commitment
APS is subject to the Code of Practice for Official Statistics.
Sport England believes reviewing current arrangements for the measurement of sports
participation will increase public trust and confidence in sports participation statistics.
Sport England’s Research, Evaluation and Analysis Team are working closely with experts
in social and market research to evaluate alternatives and will continue to do so if any new
arrangements are implemented.
11
Sport England
Methodology of measuring sports participation
How Can I Respond?
We value your views on this consultation. You can respond by completing the ONLINE
SURVEY which will remain open until midnight on 14th October 2015. This survey provides
structured questions and space for free text response. If you wish your response to be
confidential, you will have the option to specify this at the end of the survey.
If you would prefer to respond by post, a word version of the survey is available to download
on the consultation page of the Sport England website or please contact the Research,
Evaluation and Analysis team (details in the introduction to this document).
12