Supporting Researchers and Institutions in Exploiting
Administrative Databases for Statistical Purposes:
Istat’s Strategy
Giovanna D’Angiolini, Pierina De Salvo, Andrea Passacantilli
Istituto Nazionale di Statistica - ITALY
Abstract The paper presents the Istat’s strategy for supporting both the traditional and the new
users of administrative data sources for statistical purposes. Such a strategy encompasses several
activities which are aimed at enumerating the available administrative data sources, documenting
their information content and quality, making it easier to modify their content in accordance with
statistical standards. The collected information about the available administrative data sources is
disseminated to the potential statistical users through a dedicated web-based metadata
management system, called DARCAP. Moreover in order to support in-depth quality analyses of
the most important administrative data sources we are studying a new Quality Assessment
Framework for Administrative Data Sources.
Keywords Administrative data source, Administrative data documentation, Administrative data
quality, Data source ontology, Statistical data production
1
1. Istat’ strategy for supporting the statistical usage of administrative data sources:
foundation and activities
Today a large number of NSIs exploits administrative data for statistical purposes, in order to
improve the quality of statistical outputs, to reduce the statistical burden on respondents and to
minimize costs [2] [8]. Therefore documenting the content of the available administrative data
sources and assessing the quality of the acquired administrative data collections is a traditional
concern for NSIs. However in this scenario the administrative source’s documentation is produced
when needed, and the administrative data quality is generally evaluated from the viewpoint of the
particular NSI’s data production processes in which the administrative data are involved, as input
data or auxiliary data [3] [9].
This traditional scenario is rapidly evolving. Today the usage in the NSI’s data production
processes is not the only possible statistical usage for administrative data anymore. Due to the
spreading of the Data Warehouse approach in the latter years more and more non statistical
organizations have been implementing their own decision support systems, which exploit
administrative data for monitoring the context and the effects of the organization’s activities. Such
systems in fact employ statistical techniques even if their purpose is not the statistical data
production.
This decision support usage of administrative data requires that the exploited data exhibit a good
quality when regarded as measures of real world phenomena, that is to say it requires a good data
quality from a statistical viewpoint. As a consequence the interest in administrative data quality
issues is spreading across several research communities, such as the database research
community [1]. More recently the open data vision is strengthening this trend.
In such a new scenario the NSIs are required to play a new important role. They must devise and
release guidelines, methods and tools for supporting any kind of user which need to exploit
administrative data for acquiring knowledge about real world phenomena.
2
In Italy particularly many organizations belonging to the Italian national statistical systems, such as
government institutions which need to monitor the effects of their adopted policies, are building
large data warehouses which may encompass their own administrative databases together with
survey data and external administrative databases.
Nevertheless often the potential of the administrative data sources as sources of statistical
information is limited, because of the lack of proper information about their content and quality or
because they adopt administration-oriented definitions, classifications and data management
conventions.
In order to deal with such limitations Istat has undertaken a general strategy aimed at making the
available administrative data sources more understandable and usable [5].
Generally speaking the Istat’s strategy is aimed at:
•
collecting information about the available administrative data sources and producing
standard documentation about their information content and quality
•
modifying, when possible, the content of the available administrative data sources through
adopting standard statistical definitions, classifications and data management conventions.
Providing the users with proper knowledge about the content and the quality of the administrative
data sources is the first step for boosting their statistical exploitation. In order to accomplish such a
task Istat is launching several systematic documentation activities, which concern different kinds of
administrative data sources.
The central government institutions manage large information systems made up of several
administrative data sources which are fed and exploited by administrative procedures. In such
context, the Istat’s experts together with the data source’s experts jointly perform a dedicated
investigation on each administrative data source and its related administrative forms, in a
systematic way. An administrative data source investigation is an analysis and documentation
activity which follows a standard template in order to collect comparable information about the
3
content and the quality of the data source, as explained in the following section 2. The collected
information is managed by means of a dedicated web-based metadata management system,
called DARCAP (Documenting Public Administration Archives) in order to disseminate it to any
potential statistical user of the documented administrative data sources. The DARCAP system is
briefly illustrated in section 3.
Such dedicated investigations enable us to thoroughly document the information content of the
available administrative data sources, but they collect only a limited amount of qualitative remarks
about the quality of such data sources. For the most important and complex data sources, the
statistical users may need additional information about the source’s data quality. In order to
support an in-depth quality analysis, we are studying a new Quality Assessment Framework for
Administrative Data Sources, which is briefly illustrated in section 4.
Unlike central institutions, local government institutions often manage a large number of
independent administrative databases aimed at supporting a large spectrum of heterogeneous
administrative tasks which concern several subject matters, ranging from environment
management to employment monitoring. In order to gain knowledge about such administrative data
sources, Istat organizes dedicated administrative data sources’ surveys together with those
agencies which represent local government institutions. Such surveys enumerate the existing
administrative data sources and classify them by subject matter. Moreover they collect some
pieces of information for each administrative data source, such as the main observed collectives
and variables. The collected information is stored into the DARCAP system too.
All the above described activities are oriented to provide the potential statistical users of
administrative data sources with proper information about the content and the quality of such
sources.
Istat is also launching another activity, which is aimed at making it easier to modify the content of
the administrative data sources. This activity is the supervision on changes and innovation projects
concerning the administrative data sources and their related forms. It is worth noting that according
4
to the Italian statistical law the administrative data sources’ owner institutions should enforce any
Istat’s recommendation concerning their managed administrative data sources and forms, but it is
often difficult to attain such an enforcement in practice. This activity is aimed at overcoming such a
problem.
For the most important administrative data sources, the owner institution is required to notify Istat
each time it plans a change in the source’s information content. Such a notification concerns all
kinds of changes, periodic changes in the forms for income declaration as well as major innovation
projects such as a new data warehouse.
On the basis of the received notifications, Istat may give feedback and release proper
recommendations. Examples of such recommendations are: using official instead of non-official
classifications, improving the identification code system, improving the quality control procedures.
The DARCAP system provides the administrative data sources’ owner institutions with a dedicated
subsystem for supporting the change notification activities. All the received notifications together
with their related recommendations are stored into the DARCAP system. Moreover Istat’s experts
may analyze the information content of the new designed administrative data sources and forms,
as they do for already existing data sources and forms.
All the above described activities are coordinated by a Committee for Harmonizing Administrative
Forms whose members are nominated by Istat and the most important administrative data sources’
owner institutions, which is supported by a Network of experts.
2. The administrative data sources’ investigation activity: documenting the content and the
quality of the available administrative data sources
5
The investigation on an administrative data source is performed by means of analyzing the
available documentation and interviewing the source’s experts belonging to the owner institution as
well as the source’s users. The collected documentation is then structured according to the
DARCAP’s database structure, in order to be stored into such a database.
The investigation activity consist of three actions: 1) specifying a source’s general description, 2)
analyzing and documenting the source’s information content, 3) collecting information about the
source’s data quality.
1)
Specifying a source’s general description: we specify the denomination and the purposes of
the administrative data source, the owner and the other managing institutions, the laws that
instituted it and other regulating laws, the related administrative procedures, the set of
administrative forms or other tools which are currently used for feeding the administrative data
source with data.
2)
Analyzing and documenting the source’s information content: the documentation activity
aims at producing a standard and therefore comparable specification of the content of each
available administrative data source in terms of observed real-world objects. According to a
widespread usage, we call such an information content’s specification the ontology of the
documented administrative data source.
More precisely, an ontology of an administrative data source is a structured description of its
information content, based on a standard conceptual model. In order to define such a conceptual
model, we have analyzed the life-cycle of the administrative data and singled out the different kinds
of real-world objects to which they are referred, and we have put such objects into correspondence
with those objects to which any statistic is currently referred, namely collectives and variables. Our
conceptual model is oriented towards supporting the statistical exploitation of the administrative
6
data sources, but it can be easily translated into other general-purpose conceptual models and
languages for ontology specification [4].
Administrative data sources collect information about several kinds of real world objects in order to
support administrative activities. First, any administrative activity entails collecting data about those
entities which the activity addresses. Such entities are subsets of the two general populations of
persons, on one side, and entities which perform economic activities, on the other side, or they are
subsets of related populations such as households, enterprises’ territorial units. Moreover,
information is collected about those particular sets of events which may involve these entities and
are of interest for the purposes of the administrative activity. The observed populations and sets of
events are linked by relationships. For both observed populations and sets of events proper
information is collected about their characteristics, which may change in time.
As an example, the Ministry for Public Education continuously collects information about the
students, the schools and the universities with their characteristics as well as about sets of events
such as the degree course enrolments, the examinations, the degree earnings with their
characteristics. Each element of these collectives has qualitative or quantitative characteristics,
such as date of birth, residence, date of the enrolment, examination score, as well as relationships
with elements in other collectives.
Therefore we document the observed populations, which correspond to those collectives which are
the target of the administrative procedure, and their related sets of events, each one with its
definition. We also document the main characteristics which are possessed by the single elements
belonging to the specified collectives with their definitions, and the associated classifications (list of
modalities) for qualitative characteristics. From a statistical viewpoint, the quantitative
characteristics and the qualitative characteristics together with their associated classifications are
regarded as variables.
This work of conceptual description of the administrative data source’s content produces a source
ontology’s specification which encompasses: the main collectives, which may be populations or
7
sets of events, the main characteristics of these populations or sets of events, and also the
relationships which link populations and sets of events.
The result of this work is a network of main populations or sets of events, linked by 1-1 or 1-N
relationships, in which every collective has its own definition and associated characteristics.
A further analysis of the administrative data source may lead to single out more populations or sets
of events which have associated their distinguished characteristics and relationships and are linked
with the main collectives by means of subset or partition relationships. A subset relationship simply
links two collectives when one gathers a part of the elements of the other. A partition relationship
links a collective with many collectives which jointly partition it, that is: each element of the
partitioned collective belongs to one and only one of the partitioning collectives.
3)
Collecting information about the source’s data quality: by means of a dedicate
questionnaire, we interview the source’s experts in order to collect information which is used for a
first evaluation of the source’s quality.
For this purpose we ask the source’s experts for information concerning each population or set of
events. For each population we document the enter and exit events and the way by which their
registration influence the population’s coverage. For each set of events we document the way by
which the single events are recorded into the source and the time distribution of events as well as
the coverage problems due to: registration scope, namely the capability of effectively registering all
the single expected events, registration systematic distortion related to the purposes of the
administrative registration procedure, registration timeliness, namely the time lag between the
occurrence of the event and its registration.
The main problems and the possible interventions concerning the collectives’ definitions, the
suitability of the used classifications and their correspondence with standard classifications, the
identification codes which may be used for the exact linking with other data sources are also
8
evaluated. For the administrative data source as a whole, the main problems and the possible
interventions concerning its statistical usability and its diffusion timeliness are also evaluated
together with the related innovation strategies.
In such a way we obtain a first qualitative evaluation of the source’s quality. For the purpose of a
deeper analysis of the quality of administrative data sources it is useful and necessary to calculate
standard numerical indicators too. As described in section 4, the Quality Assessment Framework
for Administrative Data Sources defines concepts, methods and specific indicators for such an indepth quality evaluation.
3. Managing
and
disseminating
the
collected
information
about
the
available
administrative data sources: the DARCAP system
As we have seen, DARCAP (Documenting ARChives of Public Administrations) is the web-based
information management system for supporting the administrative data sources’ investigations and
other documentation initiatives in order to provide the administrative data sources’ potential users
with structured documentation of their content and features [6].
This tools also supports the administrative data sources’ owner institutions in sending Istat their
notifications about any change which may affect their managed administrative data sources or the
related administrative forms, and documents the released Istat’s recommendations.
More precisely, DARCAP encompasses three subsystems:
•
DARCAP-Documenta: it provides Istat experts with the functionalities for documenting the
information content and the quality of the most important administrative data sources managed by
central administration institutions, by means of storing the results of the dedicated investigation
activities which are described in the foregoing Section 2. Moreover it provides functionalities for
9
storing the results of surveys on those administrative data sources which are managed by local
administration institutions;
•
DARCAP-Innova: it provides the administration institutions with the functionalities for
notifying Istat each time they plan a change in their managed administrative data sources or forms.
It allows the Istat’ experts to give feedback on the designed innovation projects and release proper
recommendations. Moreover, it allows the Istat’ experts to document the information content of the
new designed administrative data sources or forms when needed, by calling the dedicated
functionalities of the DARCAP-Documenta subsystem
•
DARCAP Consultazione: this is the inquiry subsystem, which is aimed at disseminating the
collected information about the available administrative data sources to the potential statistical
users.
In particular, DARCAP Consultazione provides the end user with two distinguished environments
for accessing the documentation of the innovation projects or navigating through the
documentation of the existing administrative data sources or forms, respectively.
Accessing the documentation of the innovation projects: it is possible to search for an innovation
project by project’s name and institution’s name and display all the general or specific features of
any innovation project, including the documentation of the new designed administrative data
sources or forms when produced, as well as the Istat’s recommendations.
Navigating through the documentation of the existing administrative data sources or forms: this
environment provides the end user with two different search functions.
The first one is the search for an administrative data source or an administrative form by name and
other criteria, which depend on the kind of the owner institution.
The search by name requires a string specification. For those administrative data sources or
administrative forms which are owned by central institutions the other search criteria are: validity
period, data source’s type, managing institution’s name. For those administrative data sources or
10
administrative forms which are owned by local institutions the other search criteria are: validity
period, managing institution’s type and name, region, related administrative procedure’s type,
general thematic area and specific thematic area. The latter criteria correspond to an official
classification of the administrative data source’s subject matter. Proper choice lists are displayed
for each criterion. The system shows the list of those administrative data sources or administrative
forms that satisfy the specified criteria, among which the end user can choose.
The second one is the search for a an administrative data source or an administrative form by
information content: given a string specification, the system shows all the collectives,
characteristics and classifications whose name contains the specified string, and for each of them
its containing administrative data sources or administrative forms, among which the end user can
choose.
Once the end users choose a particular data source or administrative form they can browse
through its related documentation. More precisely they access:
•
Name, description and validity period, and a simple list of the observed collectives,
characteristics and classifications;
•
A graphical presentation of the data source’s ontology, namely the network of collectives
and their relationships, with the possibility, for each collective, of viewing the list of its
characteristics with their associated classifications and the network of those collectives which are
its subsets.
•
Other general features such as: the owner institutions and the other managing institutions,
the related administrative procedures and regulating laws, for the administrative data sources their
feeding administrative forms, datasets, or other administrative data sources, and other information
including enclosed documents and web sites’ addresses.
For the administrative data sources, it is possible to download a pdf document which contains the
filled Istat’s questionnaire on the administrative data source quality, which gathers information
11
about several aspects such as: the actual or potential uses of the administrative data source, the
information collecting procedures and the estimated coverage of the observed collectives.
In the second version of DARCAP, for the administrative forms it will be possible to view their
information content referred to the different sections which build their structure. It will be possible to
highlight a section in the layout and open a window with the specification of its particular
information content.
4. In-depth evaluating the quality of administrative data sources: the Framework of quality
indicators for administrative data sources
The Quality Assessment Framework for Administrative Data Sources is the Istat’s methodological
tool for evaluating the quality of the available administrative data sources [7].
As we have seen trends such as the widespread development of data warehouses and the
increasing usage of administrative data sources for non-administrative purposes oblige the NSIs to
take responsibility for a new methodological coordination task, namely to define a rich and flexible
enough sets of standard and repeatable quality assessment procedures for administrative data
sources, as they currently do for surveys [5].
Therefore the Quality Assessment Framework for Administrative Data Sources aims to define a
well-reasoned quality indicators’ frame in order to drive anyone outside or inside a NSI, particularly
the administrative data source’s owners themselves, to assess the quality of any given
administrative data source.
In order to meet such a requirement we have based the Framework on a careful analysis of the
particular goals and features of the administrative data collection process and their effects on the
quality of the collected data.
12
Such an analysis has been performed for each one of the different kinds of observed objects which
form any data source’s ontology [6]. Our approach is innovative because the ontology-based
description of the content of a data source is not a familiar practice among statisticians, despite the
fact that today the ontology-based data documentation is a common practice. By means of
anchoring the proposed indicators to the data source’s ontology we ensure a systematic
specification of the indicators and we provide the quality evaluators with directions for choosing
among calculable indicators as well as for interpreting the calculated indicators.
The Framework is organized according to the structure that has been proposed by Statistics
Netherlands [9], which distinguishes three different views on quality, namely the Source view, the
Metadata view, and the Data view. Each of these views called “hyperdimension” encompasses a
number of dimensions, quality indicators and methods.
In the Source hyperdimension, the quality aspects relate to the administrative data source as a
whole, the data source’s owner, and the delivery conditions. The Metadata hyperdimension
specifically focuses on the metadata related aspects of the administrative data source. It is
concerned with the existence and the adequacy of the documentation and with the kind and the
structure of the identification codes. The Data hyperdimension gathers all the quantitative
indicators which are calculated from data and are aimed at measuring the traditional quality
dimensions for sets of collected data: as an example, the coverage of the observed collectives and
the accuracy of the collected values for the observed characteristics.
For the Source and Metadata hyperdimensions the Framework proposes a set of qualitative
indicators which are similar to those ones which have been proposed in the BLUE-ETS project.
Note that in addition to requiring the administrative data owners to certify the availability of the
administrative source’s documentation we also provide them with the proper standard tool for
managing such documentation, namely the DARCAP system.
13
As to the Data hyperdimension, we are now specifying a new richer and more structured set of
indicators, strictly based on the administrative data source’s ontology. It includes both qualitative
and quantitative indicators.
The qualitative indicators in the Data hyperdimension are specified by exploiting the investigation
activity, which collect a first quality assessment distinctly for each collective (populations and set of
events) in the administrative data source.
As to the quantitative indicators, namely those indicators which are calculated from data and
therefore require the availability of the data set, they must be calculable by the administrative data
owner as well as by the NSI when it acquires the data set. The best scenario is when a
collaborative calculation procedure is applied.
In order to define such quantitative indicators, first we have discriminated between possible errors,
on one side, and ways of checking them, on the other side. The possible errors are defined in
terms of those objects that may be present in an administrative data source’s ontology, in the
following way.
For each object in an ontology, namely a collective, a characteristic, or a relationship, we can build
belonging statements concerning observed elements. The administrative data sources
continuously collect and store data which are in fact proper combinations of such belonging
statements.
As an example, let us suppose that a new student is registered in a Students’ register, that, is, a
new element enter in the population Students, an a new element enter in the set of events
Enrolments. If the new student is given an identifier code n and its enrolment is given an identifier
code i, the Students’ register accepts two new records: 1) a new record which combines the
statement Student (n) with the statement Residence (n, Milan) and others similar statements
concerning the registered characteristic for the new student, 2) another new record which
combines the statement Enrolment (i) with the statements Enrolment_Student (i, n),
14
Enrolment_Course (i, Statistics) and possibly other statements concerning the registered
characteristics for the enrolment itself.
It may happen that some of these statements are false, and that some true statements are not in
the data set. Therefore at any given time we may have in the administrative data source:
• Inclusion errors: false statements (definitely or temporarily) accepted in the data source
• Exclusion errors: true statements (definitely or temporarily) excluded from the data source
Other errors may concern wrong identification of the involved elements, because of problems in
the identification code system, such as: syntactical errors in identifiers, identifiers for not existing
elements, lack of identifiers for existing elements, more than one identifier for each element,
elements which share identifiers.
For each collective (population or set of events) the inclusion or exclusion errors correspond to the
well-known over-coverage and under-coverage errors respectively, and by means of combining
them with identification errors we obtain a specification of all possible errors which concern
belonging to the collective.
For each mandatory characteristic we may have an exclusion error, which corresponds to a
nonresponse error, as well as a combined exclusion and inclusion error if the element is linked with
a wrong item in the classification or a wrong numerical value, which corresponds to a measure
error; for not mandatory characteristics we may have inclusion errors too. Identification errors may
affect the observed characteristics too, when a change in a characteristic is registered for an
element which is already in the data set, such as a new residence town for a student. Errors which
may concern relationships are specified in a similar way.
The available quality check methods are mainly: searching evident errors, such as duplicate
identification codes, linking with other data sources, using logical constraints (mandatory or
incompatible combinations between various belonging statements), calculating time lags between
the moment of the events’ occurrence and the moment of their registration .
15
Until now we have defined a quality indicators’ frame concerning the collectives’ coverage and the
elements’ identification by means of properly combining possible errors and quality checks
methods. We are now analyzing the possible errors on characteristics and relationships in order to
define two other quality indicators’ frames concerning all kinds of nonresponses, measure errors,
relationship errors.
Note that our proposed indicators are distinctly calculable for each collective, characteristic and
relationship in the administrative data source’s ontology, in order to effectively support any
statistical usage of the collected information by any interested user.
5. Present and future work
At present we are carrying on data source investigations on a first set of important administrative
data sources owned by central government institutions and their related administrative forms. We
have also stored inside the DARCAP system the results of a first survey on the administrative data
sources owned by local government agencies. We plan to enlarge the investigation activity by
addressing more and more administrative data sources, and launch the supervision activity on the
administrative data sources’ changes and innovation projects.
Moreover we are carrying on our work of specifying indicators in the Data hyperdimension on the
basis of a careful analysis of possible errors based on the objects which may be present in an
administrative data source’s ontology. Finally the Quality Assessment Framework for
Administrative Data Sources will contain qualitative indicators for a preliminary quality assessment
in the Source and Metadata hyperdimensions together with a rich set of both qualitative and
quantitative indicators for an in-depth and customizable quality assessment in the Data
hyperdimension. This work is suggesting new interesting directions for the research on data quality
too.
16
REFERENCES
[1] M. Benedikt, P. Bohannon, G. Bruns Data Cleaning for Decision Support. First Int'l VLDB
Workshop on Clean Databases (2006)
[2] G.J. Brackstone, Issues in the use of administrative records for statistical purposes, Survey
methodology (1987)
[3] P. Daas, S. Ossen, M. Tennekes, L.. Zhang, C. Hendriks, K. Foldal Haugen, F. Cerroni, G. Di
Bella, T. Laitila, A. Wallgren, BLUE – ETS Deliverable 4.2 - Report on methods preferred for the
quality indicators of administrative data sources (2011)
[4] G. D’Angiolini, Manuale per la documentazione di archivi, moduli e dataset nel sistema
DARCAP, Istat document (2013)
[5] G. D’Angiolini, P. , De Salvo, A. Passacantilli, Istat’s new strategy and tools for enhancing
statistical utilization of the available administrative databases, European conference on quality in
official statistics, Vienna (2014)
[6] G. D’Angiolini, P. De Salvo, A. Passacantilli, E. Patruno, T. Saccoccio, C. De Rosa, E. Valente,
DARCAP: a tool for documenting the information content and the quality of the available
administrative databases, European conference on quality in official statistics, Vienna (2014)
[7] G. D’Angiolini, P. , De Salvo, A. Passacantilli, F. Pogelli, Framework per la qualità degli archivi
amministrativi, Istat document (2013)
[8] United Nations Economic Commission for Europe (UNECE), Using Administrative and
Secondary Sources for Official Statistics: A Handbook of Principles and Practices, United Nations
Publication (2011)
[9] R. Vis-Visschers, J. Arends-Tóth, Checklist for the Quality evaluation of Administrative Data
Sources, Discussion paper by Statistics Netherlands (2009)
17