Overview of Census Evaluation
Methods
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Why evaluate ?
 The census is a huge operation including many stages
 It is not perfect and errors can and do occur at all stages
of the census operation
 Many countries have recognized the need to evaluate the
overall quality of their census results and have employed
various methods for evaluating census coverage as well as
certain types of content error
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Purpose of census evaluation
 To provide users with some measures of quality of
census data to help them interpret the results
 To identify types and sources of error in order to assist
the planning of future censuses
 To serve as a basis for constructing the best estimate of
census aggregates, such as total population, or to
provide adjustment of census results
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Planning Census Evaluation Programme



Census evaluation programme should be developed as part of
the overall census programme and integrated with other
census activities
Census errors can happen at all phases of census operation
such as questionnaire design, mapping, enumeration, data
capture, coding, editing, etc.
Evaluation of data accuracy may have two parts:



Preliminary evaluation will enable the identification of any problem
areas that have not been previously detected in its process
More extensive evaluation should be undertaken on data items
where problems have been identified
The results of the evaluation should be made available to
census data users
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Scope and methods of evaluation
 Scope of evaluation of census data should be decided
during the planning phase
 Scope might be determined for example as:



Estimate coverage error at national level, regional level or
provincial level
Analyze evidence of age misreporting
Compare census data with independent data sources
(surveys, registers)
 Methods of census evaluation should be determined
according to resources and objectives
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Institutional organization
 Census evaluation team should be established




Team should be trained for the evaluation methods
including demographic methods, other sources of data, etc.
Team should consist of staff having experience on different
census topics such as demography, education, housing,
labor force, etc.
Team should have knowledge on historical events and
transition in population structure in the country
Team should collaborate with related institutions
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Institutional organization
 Equipment needed for census data evaluation including
hardware and software should be assessed in the initial
stage of planning
 Cost of evaluation should be included in overall census
budget
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
What are census errors?
 Coverage errors:
 Errors in the count of persons or housing units resulting from
cases having been “missed” or counted erroneously
 Content errors:
 Errors in the characteristics of persons or housing units resulting
from the interview operation (enumerators/respondents),
coding, editing, etc.
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Coverage errors
Omissions
 Missing housing units, households and/or persons during
census enumeration
 If the whole housing unit is missed, all households and
persons living in the housing unit will also be missed
 Major causes of omissions are:
 Failure to cover whole land area of a country in creating
EAs
 Mistakes made by enumerators in canvassing assigned
areas
 Ambiguous definitions of EAs, unclear boundaries of
EAs, faulty maps or coverage error during the pre-census
listing exercise
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Coverage errors
Omissions contd.
 In addition, omissions within EAs can result because all
or some of the members of the household were not
present at the time of enumeration
 Proxy respondents can inadvertently or deliberately omit
some members of a household
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Coverage errors
Duplications
 Occur when persons, households or housing units are
counted more than once
 Reasons for duplications include:
 Overlapping of enumerator’s assignments owing to errors
done during pre-census listing and delineation
 Failure by enumerators to clearly identify boundaries
 In practice, the number of omissions usually exceeds the
number of duplicates (net under-counts)
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Coverage errors
Erroneous Inclusions
 This includes:
 Housing units, households and persons enumerated while
they should have not been enumerated (e.g. babies born after the
census reference date)
 Housing units, households and persons enumerated in a
wrong place
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Coverage errors
Gross error
 This is the sum of duplications, erroneous inclusions and
omissions
Net error
 This is the difference between over-counts and undercounts
 Net census under-count exists when number of omissions
exceeds the number of duplicates and erroneous
enumerations
 Net census over-count is the opposite
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Content errors
 Every phase of census data collection and processing has
the potential for introducing errors into the census results
 The interviewing operation in which enumerators and
respondents can make errors
 Many other operations such as coding, editing where the
personnel or the procedures cause errors which affect the
census content
 Content errors add bias and nonsampling variance
components to the total mean square error of a census
statistic
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Methods for evaluation of census errors
Single Source of Data
 Demographic analysis of the census
 Interpenetration studies
Multiple Sources of Data
 Non-matching studies
 Demographic analysis using previous censuses
 Comparison with administrative sources or existing surveys
 Matching studies
 Post Enumeration Surveys
 Record checks
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Single Source of Data
Demographic Analysis of the Census
 Average number of persons per household
 Sex- and age- ratios
 Tabulations...
 For an overall assessment of quality:
 an age pyramid is a standard method
 stable population analysis can be undertaken as long as
assumptions pertaining to constant fertility and mortality
and no migration are met, for countries with declining
mortality a quasi-stable model may be appropriate
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Single Source of Data
Strengths and weaknesses
 Methods that depend on a single data source provide less
insight into the magnitude and types of errors in the
census data
 The advantage is that the methods using such sources do
not require additional data to be collected
 No need for sophisticated matching although this is also a
limitation
 It provides a general impression of quality of the census
data
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Single Source of Data
Interpenetration studies
 Method involves drawing subsamples, selected in an
identical manner, from the census frame
 Each subsample should be capable of providing valid
estimates of population parameters
 Assignment of personnel (i.e. enumerators, coders, data
entry staff, etc.) is done randomly
 The method helps to provide an appraisal of the quality of
census information and procedures
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Single Source - Interpenetration studies
Strengths and weaknesses
 Gives good idea of different contribution of component
errors to total census error
 Helps to identify operational stages that contribute to
census error, thus identifying procedural limitations in a
census
 Disadvantages include:
 That it is an expensive operation demanding many field
staff, intensive training and close supervision
 Relatively complex in designing and implementation
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Multiple Sources of Data – Non matching studies
Demographic Analysis
 Comparison with data from administrative registers such
as vital registration system
 Demographic analysis based on successive censuses such
as the cohort component method
 More discussion of demographic methods is the focus
of this workshop
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Multiple Sources of Data – Non matching studies
Comparison with existing household surveys
 In theory any probability sample of households or persons
can be used to evaluate coverage and content error in a
census if:




They have identical items with same concepts and definitions
They are independent from the census
They have been conducted close to the census date
There is a sufficient identification information to facilitate
accurate matching
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Multiple sources - Non-matching studies
Strengths and weaknesses
 Review census results at aggregate rather than unit level
i.e. provides only estimates of net census error
 Evaluates very limited characteristics such as sex and age
distributions
Advantage
 They are relatively cheap compared to matching studies
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Non matching methods - Demographic Analysis
Strengths and weaknesses
 No additional data is needed to be collected to perform
the analysis
 Less costly
 In statistical offices with sufficient numbers of
demographers there is no need for additional staff to do
the technical analysis
 On the negative side these methods provide less insight
into the different contributions of component errors to
total error in the census
 Quality of sources (Vital Statistics…)
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Multiple Sources of Data – Matching studies
Record checks
 Census records are matched with a sample of records
from registration systems such as the vital registration
system
 The relevant respondents to the census questionnaire are
traced to the time synchronized with the census
 Sources include:





Previous census
Birth registrations
School enrolment
Citizen registration card
Immigration registers etc.
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Multiple Sources of Data – Matching studies
Record checks contd.
 Both coverage and content errors could be measured
through the above comparisons
To evaluate coverage efficiently the following
preconditions are essential:
 A large proportion of census population should be covered
in registration system
 The census and registration system should be independent
from each other
 There should be sufficient information in records
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Multiple Sources of Data – Matching studies
To evaluate content efficiently the following
preconditions are essential:
 The registers system should contain some relevant items
covered in the census such as age, sex, education,
relationship, marital status etc.
 Definitions of items should be identical between the
census and the registers
Countries that have used record checks include:
Denmark, Finland, Norway, Sweden and Canada
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Multiple Sources of Data – Matching studies
Strengths and weaknesses
 It provide separate estimates of coverage and content error
 Prospects of evaluating more characteristics compared to
what can be done with non-matching studies
Challenges
 Calls for high level technical skills including managerial
 Matching is expensive
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Multiple Sources of Data – Matching studies
Post Enumeration Survey (PES)
 Consists of two separate coverage studies :
 A survey conducted using a sample frame independent of
the census. Persons from this survey are matched to the
census to estimate the number of persons missed in the
census
 A survey conducted using a sample drawn from persons
enumerated in the census. This sample is re-enumerated to
determine if the sample person or unit was erroneously
enumerated (inc. erroneously located)
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Multiple Sources of Data – Matching studies
Post Enumeration Survey (PES) contd.
 Results can be used to evaluate the reliability of some
characteristics such as sex, age, marital status, relationship to
reference person or head of the household.
 For some countries the results of PES can be used to
adjust some census results
 Facilitates better interpretation of census results
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Post enumeration survey
Advantages:
 Its results can be used to independently evaluate census
coverage and content error, including reliability of
selected characteristics collected in a census
 Incorporates matching of individuals or units between the
census and PES
 Its results are generally more reliable than those of the
census i.e. it is justification for evaluation
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Post enumeration survey
Challenges:
 Requires highly skilled field and professional staff
 Matching is complex
 As it is supposed to be carried out immediately after the
census at times there is lack of adequate funds and staff
to implement the PES exercise
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011
Thank You!
Sub-regional Workshop on Census Data Evaluation,
Phnom Penh, Cambodia, 14-17 November 2011