Understanding and Measuring
Uncertainty Associated with the
Mid-Year Population Estimates
Joanne Clements
Ruth Fulton
Alison Whitworth
1
Context
• Improving Migration and Population Statistics
(IMPS) Project
• Quality Strand
• “Establish quality measures for population
statistics”
• No international precedent for this work
2
Issues
• Estimates compiled from a wide range of
administrative sources plus some survey and
Census data
• Source data subject to sampling and nonsampling errors
• Lack of independent data with which to
corroborate
• How to estimate each potential error and
combine these in one measure?
3
Aim and Objectives
• Aim
Improve understanding, measurement and reporting
of the quality of population estimates
• Objectives
– Describing the sources of uncertainty
– Developing methods for measuring uncertainty for
each issue and combining them into one measure
– Eventually feeding findings into ONS quality
reports
4
Presentation Outline
• Summarise population estimates
methodology
• Summarise previous research on quality
• Detail proposed error measurement
methodology
– Illustrate by applying to local authority (LA) mid2006 population estimates
• Outline emerging proposals for further work
to achieve robust quality measures
5
Calculating LA Population Estimates
e.g. Southampton UA
6
Calculating LA Population Estimates
(cont)
Adjustment
7
Calculating LA Population Estimates
8
Previous Research
Quality of Population Estimates
• Past experience of inter-censal errors
• Sampling error and expert opinion of nonsampling error in components of estimates
Quality of Population Projections
• Accuracy of past projections
• Use of variant projections
• Simulation methods using error distributions
for the components of projections (stochastic
forecasting)
9
Proposed Methodology:
Initial Assessment of Quality Issues
• Map out the procedures and data sources
used to derive population estimates
• Identify associated quality issues
• Identify the importance of these issues
10
LA Population Estimates:
Initial Assessment of Quality Issues
• Brief assessment of the evidence for each
component
• For example: Internal Migration
– Relies on GP registration data
– Assumes patients reregister within a month of
moving (known issue for students leaving
university)
11
Proposed Methodology (cont):
Detailed Investigation of Quality Issues
• Quantify uncertainty using statistical theory,
empirical evidence and / or expert opinion
• Both sampling and non-sampling errors
12
LA Population Estimates:
Detailed Investigation of Quality Issues
• Attributing a potential uncertainty range and
distribution to each component
• Not each quality issue
• Made relatively simplistic distribution
assumptions (Normal or Uniform)
• Assumed same level of uncertainty across
LAs
13
LA Population Estimates:
Detailed Investigation of Quality Issues
• Estimating uncertainty relative to size of local
authority component
• For example: Could assume potential error in
annual local authority births estimate
N(0, X% of estimated births)
– Assume similar error distributions by year
14
Proposed Methodology (cont):
Overall Quality Measure
• Mathematically complicated to combine a
large number of potential error measures into
one quality measure
– Errors may be correlated
– Distributions not all normally distributed
• Developed a Simulation methodology
15
LA Population Estimates: Simulation
• For each local authority randomly generate
errors for each component
– Using previously developed error distributions
Mid-2001 error estimate
+ Births 01/02 error estimate
- Deaths 01/02 error estimate
+ Internal In-Migrants 01/02 error estimate
- Internal Out-Migrants 01/02 error estimate
+ …..
+ Births 02/03 error estimate
- Deaths 02/03 error estimate
+…
16
LA Population Estimates: Simulation
•
Calculate error in mid-2006 estimate by
combining the errors generated for each
component in each year up to 2006
•
Repeat process 1000 times
•
Obtain distribution of potential error in mid2006 local authority estimate
17
Findings: Potential Error Distribution
18
Findings:
Measuring Uncertainty in Population Estimates
• Simulation methodology allows measures of
uncertainty to be calculated for population
estimates.
• But, in reality, there is uncertainty in these
measures of uncertainty, as…
– Only as good as the error assumptions made for
each issue / component of change
– Very difficult to exactly measure non-sampling
error
19
Findings:
Key Components of Uncertainty
• Uncertainty in population estimates related to:
– Size of each component
– Error distribution assumptions
• Key components driving uncertainty in LA
estimates:
–
–
–
–
Mid-2001 base population estimate
Internal Migration
International Migration (IPS)
Specific components important in specific LAs
e.g. Foreign Armed Forces
20
Extending the Methodology
• Current assumptions in the estimation of
uncertainties are inadequate
– Need to examine issues within each component
– Consider LA variation in uncertainties within each
component
• Currently focussing on refining error
distributions for key drivers of uncertainty
within LA estimates
– Internal Migration
– International Migration
21
Estimating Uncertainty - Internal Migration:
Emerging Proposals for Further Work
Building upon previous work, investigate
uncertainty in estimates related to:
• Time lags between moving and reregistering
• Moves not captured by GP registers because
patients were not registered when data were
extracted
• The scale of constraining GP register data to
NHSCR
22
Estimating Uncertainty - International Migration:
Emerging Proposals for Further Work
• Calculating sampling error of IPS estimates
• Investigating uncertainty around migrant and
visitor switcher estimates
• Investigate uncertainty within methods used
to calculate LA migration estimates from the
IPS
– For example, in LA emigration model used
23
Future Outcomes of this Work
• Increased understanding of sources of error in
the population estimates and their relative
importance
• Ability to focus resources for research on key
sources of uncertainties
• Additional information which could feed into
Quality Reports
This work is intended to improve our
understanding of the uncertainty in population
estimates, rather than provide exact estimates
of uncertainty
24
Summary
Measuring Uncertainty of Population Estimates
• Estimating their error margin is complex
• Detailed quality assessment of each
component required to obtain a robust
measurement
• Simulation methods are a plausible approach
to approximately measure the overall quality
of an estimate
• Ongoing work on estimating uncertainty in
migration components
25