National Strategy and Toolkit for NHSN Data
Validation
Kathryn E. Arnold MD
Medical Officer, Division of Healthcare Quality Promotion
2012 CSTE Annual Conference
June 3-7, 2012
National Center for Emerging and Zoonotic Infectious Diseases
HAI Data Validation is Important

Credible data are vital for prevention, public reporting,
and incentivizing improvements in clinical
performance
 Concerns about uneven data quality
• Always important, now more than ever
 Validation can improve fairness

Need for training on all levels
 Validation findings help guide training
What Do We Mean by Validation?
What Do We Mean by Validation?

Assure production of high quality surveillance data





Ability to generate correct denominator data
Ability to identify all candidate events in real time
Routine assessment and tracking of candidate events
Ability to correctly apply case-definitions
Minimized data-entry error
How Do We Develop a Standardized, Scalable
Approach to Validation That Can Work in Any
State?
States as Validation Laboratories, 2010-2011

States created innovative approaches under ARRA
 Central Line-Associated Bloodstream Infection (CLABSI):
•
•
•
•
•
Structure of sampling frame
Numerator sampling approaches
Checklists for case-classification
Denominator methods surveys
Risk-factor (location mapping) investigations
 Surgical Site Infection (SSI):
• Data linkage to enrich targeted samples (procedures) for SSI
• In house and post-discharge case-finding surveys
• Risk-factor audits in access database
* Citations, references, and credits – Myriad Pro, 11pt
CLABSI Externally Validated by State, as of 2012
Dots: CLABSI Mandate by 2012
NH
VT
ID
MA
PA
PA
OH
RI
CT
MD
NJ
WV
DE
MD
DC
HI
PR
SSI Externally Validated by State, as of 2012
Dots: SSI Mandate by 2012
NH
VT
ME
ID
MA
PA
PA
RI
CT
OH
NJ
WV
DE
MD
DC
HI
PR
State and CMS Validation are Complementary,
but Different
Approach
State
CMS
Differs state-by-state
Nationwide probability sample
Constrained Statute (access to data), and
by
resources
Statute (scope), resources, and
existing infrastructure
Validates
Numerator;
denominator methods;
risk adjustment variables;
Numerator
Sampling
Varies; often targeted
Small sample from all IPPS
hospitals, at least every 4 years
Primary
goals
Improve surveillance practices;
understand weaknesses for
teaching;
optimize data quality at all
levels
Assure compliance;
validate accuracy of metric;
motivate internal improvement
National Strategy for NHSN Data Validation



Document and characterize need for NHSN validation
Recognize CMS role in motivating facility engagement
Demonstrate unique value of states in conducting
NHSN validation
 Because ALL data cannot be validated, states use data to assure
competence, identify weaknesses in surveillance, and enable
improvement by teaching




Develop guidance, determine costs
Identify funding
Sustain and enhance capacity
Harmonize work among stakeholders
* Citations, references, and credits – Myriad Pro, 11pt
2012 Validation Guidance and Toolkit:
CLABSI and SSI

Chapter 1:
 Overview and Framework
• Intrinsic (built-in) validation
• Internal (to NHSN and reporters) validation
• External (to NHSN or reporters) validation
 Types of External Validation
 Examples of SHD Validation Approaches
• Targeted External Validation
• Probability Samples for External Validation
• Hybrid approaches
* Citations, references, and credits – Myriad Pro, 11pt
Approaches to External Validation

Targeted External Validation, TN (others)
 Perfect for efficiently improving data quality and teaching to
reporting errors

Probability Samples OR (CT, CMS, WA)
 Needed for extrapolation of performance estimates, and preferred
for longitudinal assessment.
* Citations, references, and credits – Myriad Pro, 11pt
Chapters 2-4: CLABSI

Internal validation (Quality Assurance)
 For reporting facilities
 For group users



Targeted External Validation
External Validation using Probability Samples
CLABSI Validation Tools
* Citations, references, and credits – Myriad Pro, 11pt
CLABSI Validation Tools









Access Database (New York)
Facility Self-Validation Tool
Denominator Collection Methods Survey
Algorithmic Use of NHSN Analysis to Target Facilities
Example Letter Requesting External Validation Site
Visit
Checklists for Validation (Tennessee)
Template for Audit Discrepancies Report
Example Validation Follow-up Letters, With and
Without Problems
Scalable Self-weighting Sample Using Probability
Proportional to Size
* Citations, references, and credits – Myriad Pro, 11pt
Chapters 5-7: SSI

Internal validation (Quality Assurance)
 For reporters
 For group users



Targeted External Validation
External Validation using Probability Samples
SSI Validation Tools
* Citations, references, and credits – Myriad Pro, 11pt
SSI Tools






Expected and Unusual Values for Surgery Variables
Admission Surveillance Practices Survey
Post-Discharge Surveillance Practices Survey
Developing an Enriched Sampling Frame for Targeted
SSI Validation
ICD-9 Procedure Codes, and ICD-9 Diagnostic Codes
Suggestive of SSIs
Expected Length of Stay for NHSN Procedures
* Citations, references, and credits – Myriad Pro, 11pt
Quality Improvement for the Toolkit


Post-Validation Analysis to Help with Future Iterations
of the Toolkit
Rate the Toolkit
* Citations, references, and credits – Myriad Pro, 11pt
Pre-clearance Input


We are not seeking to distribute the document widely
yet; we are seeking feedback
We invite reviewers who are willing to read and provide
meaningful input for this first (pre-clearance) iteration
of the Guidance and Toolkit
 If you are interested, please let us know (KEA3@CDC.GOV)

Please come to Rachel Stricof’s Roundtable for more
discussion of targeted vs. probability sampling
 Roundtable Tuesday 5:45 Herndon
* Citations, references, and credits – Myriad Pro, 11pt
Thank You !
(CSTE) Rachel Stricof
(State Partners) Lynn Janssen (CA), Richard Melchreit (CT), Carole Van
Antwerpen and Valerie Haley (NY) , Paul Cieslak and Zintars Beldavs (OR),
Marion Kainer and Brynn Berger (TN), David Birnbaum (WA), Many others
(CDC) James Baggs, Maggie Dudeck, Jonathan Edwards, Ryan Fagan, Scott
Fridkin, Teresa Horan, Paul Malpiedi, Daniel Pollock, Cathy Rebmann, Philip
Ricks, Dawn Sievert, Arjun Srinivasan, Nicola Thompson, Elizabeth Zell
The findings and conclusions in this report are those of the author and do not necessarily represent the official position of the
Centers for Disease Control and Prevention.
National Center for Emerging and Zoonotic Infectious Diseases