ESCM Chapter 8:
Data Quality and Meta Data
United Nations Oslo City Group on Energy Statistics
OG7, Helsinki, Finland
October 2012
1
Introduction
 IRES Chapter 9: deals with Data Quality
Assurance and Meta Data
 Under IRES, countries are encouraged to:
• Develop national quality assurance programs
• Document these programs
• Develop measures of data quality
• Make these available to users
2
Prerequisites of Data Quality
 Institutional and organizational conditions,
including:
 Legal basis for compilation of data
 Adequate data-sharing and coordination between
partners
 Assurance of confidentiality and security of data
 Adequacy of resources – human, financial, technical
 Efficient management of resources
 Quality awareness
3
Promoting Data Quality






4
Make quality a stated goal of the organization
Establish standards for data quality
Track quality indicators
Conduct regular quality assurance reviews
Develop centres of expertise to promote quality
Deliver quality assurance training
What is a Quality Assurance
Framework?
 All planned activities to ensure data produced are
adequate for their intended use
 Includes: standards, practices, measures
 Allows for:
•
•
•
•
Comparisons with other countries
Self-assessment
Technical assistance
Reviews by international and other users
 See Figure 8.1 for examples of quality frameworks
5
Quality Assurance Framework
 Six Dimensions of Data Quality, based on
ensuring “fitness for use”
1.
2.
3.
4.
5.
6.
6
Relevance
Accuracy
Timeliness
Accessibility
Interpretability
Coherence
Quality Measures and Indicators
 Should cover all elements of the Quality
Assurance Framework
 Methodology should be well-established,
credible
 Must be easy to interpret and use
 Should be practical – reasonable, not an overburden
 For Key Indicators, see Chapter 8, Table 8.2
7
Sample Quality Indicators
 From IRES Table 9.2, linked to QA Framework
 Relevance: user feedback on satisfaction, utility of
products and data
 Accuracy: response rate, weighted response rate, number
and size of revisions
 Timeliness: time lag between reference period and release
of data
 Accessibility: number of hits, number of requests
 Interpretability: amount of background info available
 Coherence: validation of data from other sources
8
Quality assurance must be built into
all stages of the survey process
Survey Stages:
Quality Assurance
Framework
9
1.
2.
3.
4.
5.
6.
7.
8.
9.
Specify needs
Design
Build
Collect
Process
Analyze
Disseminate
Archive
Evaluate
1. Specify Needs
Activities:
 Determine needs: define
objectives, uses, users
 Identify concepts,
variables
 Identify data sources and
availability
 Prepare business case
10
Quality Assurance
 Consult with users and
key stakeholders
 Clearly state objectives,
concepts
 Establish quality targets
 Check sources for quality,
comparability, timeliness
 Gather input and support
from respondents
2. Design
Activities:
 Determine outputs
 Define concepts,
variables
 Design data collection
methodology
 Determine frame &
sampling strategy
 Design production
processes
11
Quality Assurance
 Consult users on outputs
 Select, test & maintain frame
 Design & test questionnaire
and instructions
 Use established standards
 Develop processes for error
detection
 Develop & test imputation
3. Build
Activities:
 Build collection
instrument
 Build processing system
 Design workflows
 Finalize production
systems
12
Quality Assurance
 Focus test questionnaire
with respondents
 Test systems for
functionality
 Test workflows; train staff
 Document
 Develop quality measures
4. Collect
Activities:
 Select sample
 Set up collection
 Run collection
 Finalize collection
13
Quality Assurance
 Maintain frame
 Train collection staff
 Use technology with built
in edits
 Implement verification
procedures
 Monitor response rates,
error rates, follow-up
rates, reasons for nonresponse
5. Process
Activities:
 Integrate data from all
sources
 Classify and code data
 Review, validate and edit
 Impute for missing or
problematic data
 Create and apply weights
 Derive variables
14
Quality Assurance
 Monitor edits
 Implement follow-ups
 Focus of most important
respondents
 Analyze and correct
outliers
6. Analyze
Activities:
 Transform data to outputs
 Validate data
 Scrutinize and explain
data
 Apply disclosure controls
 Finalize outputs
15
Quality Assurance
 Track all indicators
 Calculate quality indicators
 Compare data with
previous cycles
 Do coherence analysis
 Validate against
expectations and subject
matter intelligence
 Document all findings
7. Disseminate
Activities:
 Load data into output
systems
 Release products
 Link to meta data
 Provide quality indicators
 Provide user support
16
Quality Assurance
 Format & review outputs
 Verify that tools do not
introduce errors
 Verify disclosure control
 Ensure all meta data and
quality idicators are
available
 Provide contact names
for user support
8. Archive
Activities:
 Create rules and
procedures for archiving
and disposal
 Maintain catalogues,
formats, systems
17
Quality Assurance
 Periodic testing of
processes and systems
 Ensure meta data are
attached
9. Evaluate
Activities:
 Conduct post mortem
reviews to assess
performance, identify
issues
 Take corrective actions or
make new investments,
as required
18
Quality Assurance
 Consult with clients about
needs, concerns
 Monitor key quality
indicators
 Periodic data quality
reviews
 Perform ongoing coherence
analysis
 Compare with best
practices elsewhere
Meta Data
 Important for assessing “fitness for use” and
ensuring interpretability
 Required at every step of the survey process
 Critical for enabling comparisons with other data
 Should include results of data quality reviews
 Figure 8.4: generic set of meta data
requirements
19
Future of Meta Data
 Should become a driver of survey design
 Can be used proactively to prescribe definitions,
concepts, variables and standards
 Can support the harmonization of international
surveys and data
 Efforts are underway to create an integrated
approach for producing and recording meta data
20
Thank you!
Andy Kohut, Director
Manufacturing and Energy Division
Statistics Canada
Section B-8, 11th Floor, Jean Talon Building
Ottawa, Ontario
Canada K1A 0T6
Telephone: 613-951-5858
E-mail: andy.kohut@statcan.gc.ca
www.statcan.gc.ca
21