Assessing Quality through Auditing
and Self-Assessment
Signore M. , Carbini R., D’Orazio M., Brancato G.,
Simeoni G.
Marina Signore
Head of Service “Audit for Quality
Istat
Auditing and Self-assessment
Presentation
• Describe the new procedure for assessing
quality launched by Istat and how it fits into
Istat Quality Policy
• The new
procedure was launched in
October 2009 and is now in the testing
phase
DatQAM map
External Environment
Users Requirements
Labelling
Self-Assessment
Tools
Process
Variables
Quality
indicators
Measurements
Preconditions
Standards
Certification (ISO 20252)
Audit
Quality Reports
User Surveys
Documentation
Auditing and Self-assessment
Istat Quality Policy
Harmonized with international quality
frameworks
European Statistics Code of Practice (since 2005)
Eurostat Quality dimensions
Centralized Structure for Quality Assurance
(management and assessment)
Auditing and Self-assessment
Istat Framework for Quality
Actions
Improvement
Actions
Communication on Quality to Users
Internal Audit
Assessment
Self-Assessment
Assessment of Standard Quality Indicators
Measurements
Survey Assurance- Quality Control Systems
Quality Guidelines
Preconditions
Documentation system
Standard Quality Indicators
Auditing and Self-assessment
Quality Assessment
Current surveys
• Analyses of SQI managed in SIDI
• In-depth analyses of survey process
 self-assessment
 internal auditing
New Surveys
• Procedure for quality labeling
 developed and tested on 2 surveys
 adherence to few principles (relevance, questionnaire design
and test, response burden reduction, quality assurance system)
 not yet launched
Auditing and Self-assessment
Quality Assessment for current processes
Analyses of SQI managed in SIDI
• analyses of trends (response rates, timeliness,…)
• evaluation of Istat policies
• reports prepared by central service for Quality
Regular evaluation for most statistical processes but
not a specific tool for promoting quality improvements
Auditing and Self-assessment
Quality Assessment for current processes
Development (October 2009) of a strategy that combines
audit and self-assessment
• way for highlighting strengths and weaknesses
• tool for improving weaknesses
• tool for collecting and disseminating good practices
Concrete way of improving quality
Auditing and Self-assessment
Quality Assessment: main features
 The audits are carried out by internal auditors (face-to-face
interview)
 The output of the assessment process consists in the
identification of improvement actions, resources and timetable
 The improvement actions resulting from self-assessment
are reviewed by a team of reviewers
 Additional resources or methodological support in order to
implement the improvement actions
 User consultation on the improvement actions and for
setting priorities
 Establishment of a Quality Committee in order to
coordinate and sponsor the assessment activity
Auditing and Self-assessment
Quality Committee
Established in February 2010
Composition
• Chaired by the Head of Department of Statistical
Production
• Co-chaired by the Head of Service “Audit for Quality”
• Members: Head of services
Auditing and Self-assessment
Quality Committee
Mandate
 To coordinate audit and self-assessment activity:
• Evaluation of final reports
• Approval of improvement actions
• Monitoring of improvement actions
 To develop procedures for monitoring the quality of
statistical processes
 To coordinate and sponsor quality assessment in the
ISS
 To promote the adoption of the CoP in the ISS
Auditing and Self-assessment
The tools
• Quality guidelines
• Audit questionnaire
• Self-assessment questionnaire (electronic version)
• Operative manual
Auditing and Self-assessment
Quality Guidelines
Reference guide to design, carry out
and control a statistical process
Part I: Process quality
Survey purposes
Survey design
Registers
Data collection
Data processing
Data dissemination and documentation
Part II: Product quality
Components of Eurostat Quality definition
Auditing and Self-assessment
Quality Guidelines
For each section
 Definition of the Principle to be followed
 Text describing what should be done in order to fulfill the
principle
 Quite general descriptions and references to technical
documents
 Light document less than 50 pages
Auditing and Self-assessment
Audit and Self-assessment questionnaires
Structure
Part I: Process quality
Part II: Product quality
Part III: Process management and quality management
For each section
 The Principle to be followed is stated
 Some questions in order to evaluate the degree of
compliance to the Principle
 Slight differences in the two questionnaires
 Input: DESAP checklist and OECD questionnaire
Auditing and Self-assessment
The testing phase
 To test the tools and the procedure
 To evaluate the impact and sustainability
Features
• on a voluntary basis
• Audit: 2 processes (2 teams of 3 experts each)
• Self-Assessment: 5 processes (3 teams of 3 reviewers
each)
Auditing and Self-assessment
Timetable of the new procedure
• Development of the procedure and tools (October December 2009)
• Internal discussion on the tools (January 2010)
• Testing phase (February- June 2010)
Auditing and Self-assessment
Final remarks
 Great opportunity for improving the quality of
Istat statistical processes
 Analyses of the testing phase but first results
seem to be very positive
 Further developments will concern processes
based on administrative sources and secondary
studies (e.g. National Accounts)