VALIDATING AHRQ PATIENT SAFETY
INDICATORS with the VA NATIONAL
SURGICAL QUALITY IMPROVEMENT
PROGRAM POSTOPERATIVE
ADVERSE EVENTS
Dennis Tsilimingras, M.D., M.P.H.
Center for Health Quality, Outcomes, & Economic
Research (Bedford VAMC)
Boston University School of Public Health
Co-Investigators
• Patrick Romano, M.D., M.P.H.3, William Henderson, Ph.D.4,
Peter Rivard, M.H.S.A.1,2, Susan Loveland, M.A.T.1,2, Shibei
Zhao, M.P.H.1,2, Cindy Christiansen, Ph.D.1,2, Anne Elixhauser,
Ph.D.5, Shukri Khuri, M.D.6, Amy K. Rosen, Ph.D.1,2
•
•
•
•
•
•
1Center
for Health Quality, Outcomes, & Economic Research
(Bedford VAMC)
2Boston University School of Public Health
3University of California at Davis
4NSQIP Denver Data Analysis Center
5Agency for Healthcare Research and Quality
6Surgical Service, VA Boston Healthcare System
Funding Organization: Department of Veterans Affairs
HSR&D Project IIR 02-144-1
Introduction
• Patient Safety Indicators (PSIs):
- developed by the UC-Stanford and AHRQ
collaboration
- purpose to screen for potential safety problems using
ICD-9-CM codes
- limitation: PSIs have not been validated in the past
Research Objectives
(1) To validate the AHRQ PSIs with postoperative
adverse events in the VA National Surgical Quality
Improvement Program (NSQIP).
(2) To test alternative PSI definitions based on clinical
information from NSQIP for the purpose of future
refinement of the PSIs.
Methods
AHRQ
NSQIP
Accepted PSIs:
Postoperative Adverse
Events:
(1) Postoperative Sepsis
(2) Postoperative Respiratory
Failure
(1) Systemic Sepsis
(2) Unplanned Intubation for
Respiratory Failure
or Failure to Wean from
Ventilator > 48hrs
(3) Deep Vein Thrombosis
& Pulmonary Embolism
(3) Postoperative Pulmonary
Embolism or Deep Vein
Thrombosis
(4) Postoperative Wound
(4) Dehiscence
Dehiscence
(5) Postoperative Physiologic
(5) Acute Renal Failure
and Metabolic Derangements
Methods
Study Sample
• VA administrative inpatient data from the Patient
Treatment File (PTF):
- from 130 VA hospitals
- matched with NSQIP chart-abstracted records
- FY 2001 (10/00 – 09/01) from both data sources
Methods
• NSQIP:
- purpose is to provide reliable, valid, and
comparative information regarding morbidity &
mortality to all VA medical centers
- employs trained nurse data collectors
- collects preoperative patient characteristics and
30-day postoperative outcomes
Methods
• A hospitalization had a PSI if it met AHRQ’s
administrative data definition that includes ICD-9-CM
diagnostic and procedure codes.
• A hospitalization had a NSQIP event if it met NSQIP’s
clinical definition which was translated into ICD-9-CM
diagnostic and procedure codes.
• Our alternative definition was based on a translation
of NSQIP clinical definitions into ICD-9-CM diagnostic
& procedure codes.
Methods
• Using NSQIP as the “gold standard”:
- measured the sensitivity & positive predictive value
of “all” 5 AHRQ PSIs
- using the current AHRQ PSI definition
- compared with the best alternative PSI definition
• These definitions were developed using PTF data
and validated with NSQIP medical chart-abstracted
data.
Postoperative Respiratory Failure
Definitions
ICD-9-CM
Diagnostic codes
Procedure Codes
AHRQ
518.81 & 518.84
NSQIP
518.5
96.04 & 96.70-96.72
518.5, 518.81, &
518.84
96.04 & 96.70-96.72
Alternative
Results
Sample
• 561,436 inpatient hospitalizations (PTF)
• 101,548 with surgical Diagnostic Related Groups
• 55,752 were matched with NSQIP data, which were
used to calculate sensitivities and positive predictive
values
Results
Accepted PSIs:
(1) Postoperative Sepsis (12,011 hospitalizations)
Current AHRQ PSI Definition
Alternative PSI Definition (NSQIP)
Sensitivity(%) PPV(%)
32
44
37
45
(2) Postoperative Respiratory failure* (24,273 hospitalizations)
Current AHRQ PSI Definition
Alternative PSI Definition (NSQIP)
Sensitivity(%) PPV(%)
19
74
67
66
*PSI includes diagnostic and procedure codes.
Results
(3) Postoperative Pulmonary Embolism or Deep Vein
Thrombosis* (55,682 hospitalizations)
Current AHRQ PSI Definition
Alternative PSI Definition (NSQIP)
Sensitivity(%) PPV(%)
56
22
58
22
(4) Postoperative Wound Dehiscence* (16,904 hospitalizations)
Current AHRQ PSI Definition
Alternative PSI Definition (NSQIP)
Sensitivity(%) PPV(%)
29
72
61
57
*PSI includes diagnostic and procedure codes.
Results
(5) Postoperative Physiologic and Metabolic Derangements*
(27,722 hospitalizations)
Current AHRQ PSI Definition
Alternative PSI Definition (NSQIP)
Sensitivity(%) PPV(%)
44
54
48
63
*PSI includes diagnostic and procedure codes.
Results
Comparing our results with Best and colleagues.
AHRQ PSI
Sepsis
Our Results
Best et al 2002 Results†
Sens(%)PPV(%) Sens(%)PPV(%)
37
45
40
29
Respiratory Failure
67
66
9
52
PE or DVT
58
22
Dehiscence
61
57
PE 49
DVT 12
25
48
18
23
Physiologic & Metabolic
Derangements
48
63
16
4
†Best
et al. Identifying Patient Preoperative Risk Factors & Postoperative
Adverse Events in Administrative Databases: Results from NSQIP. J Am Coll
Surg. 2002;194:257-266.
Conclusions
• NSQIP clinical data are an excellent source for
examining the validity of administrative-data-based
indicators, such as the PSIs, as identifiers of potential
inpatient surgical patient safety events.
• All 5 PSIs were improved by a simple modification to
perform better in the VA.
Significance
• PSIs may be a useful screening tool for identifying
inpatient safety events that warrant further
investigation.
• Such a tool can be constructed with the incorporation
of clinical data definitions to administrative data
definitions.
• The latest release of the AHRQ PSI software has
incorporated certain findings from this study.