Validating the Patient Safety Indicators ( PSIs ) in
advertisement
Validating the Patient Safety Indicators (PSIs) in the Veterans Health Administration (VA) Ann Borzecki, MD, MPH Center for Health Quality, Outcomes, and Economic Research, Bedford VAMC, and Boston University Schools of Public Health and Medicine Acknowledgements Funding: VA Health Services Research and Development (HSR&D) Service SDR 07-002 Amy Rosen, PhD, (PI) Haytham Kaafarani, MD, MPH Amresh Hanchate, PhD Susan Loveland, MAT Hillary Mull, MPP Marlena Shin, JD, MPH Joanne Cuny, RN, BSN, MBA Dijon Fasoli, RN, PhD, MBA Kathleen Hickson, RN Kamal Itani, MD Julie Lynch, RN Sally McDonald, RN Michael Shwartz, PhD, MBA Patrick Romano, MD, MPH Kristine Ruggiero, RN Shibei Zhao, MPH Borzecki, Academy Health 2009 2 Background - PSIs Set of indicators designed to screen for potentially preventable complications of care in inpatient setting – Complications following surgical/non-surgical procedures and medical care Use administrative data - hospital discharge data Cost-efficient Standardized definitions of adverse events Borzecki, Academy Health 2009 3 PSI Development Initial set of 13 PSIs developed by AHRQ in early 2000s through review of literature and ICD-9-CM codes (Miller, Health Serv Res 2001) Existing PSIs evaluated and revised by UCStanford EPC through a four-step process: – Literature review (Complications Screening Program; Iezzoni, QRB Qual Rev Bull 1992) – Evaluation of candidate PSIs by clinical panels – Expert review of ICD-9-CM codes in candidate PSIs – Empirical analyses of candidate PSIs Borzecki, Academy Health 2009 4 Accepted PSIs (N=20) Selected postoperative complications Technical difficulty with procedures –Iatrogenic pneumothorax * –Accidental puncture or laceration * –Foreign body left in during procedure*‡ –Postoperative pulmonary embolism or deep vein thrombosis * –Postoperative respiratory failure –Postoperative sepsis Other –Postoperative physiologic and –Complications of anesthesia metabolic derangements –Death in low-mortality DRGs * –Postoperative wound dehiscence * –Death among surgical patients with –Postoperative hip fracture treatable serious complications *† –Postoperative hemorrhage or –Transfusion reaction *‡ hematoma Obstetric trauma and birth trauma (Not Relevant to VA) Selected technical adverse events –Birth trauma – injury to neonate * –Decubitus ulcer –Obstetric trauma – vaginal delivery –Selected infections due to medical with instrument care † * NQF Endorsed † Recently revised ‡ Counts, not rates –Obstetric trauma – vaginal delivery without instrument –Obstetric trauma – cesarean section delivery Borzecki, Academy Health 2009 5 Uses of PSIs Intended Uses Screening tools for patient safety problems Case-finding tools for targeted QI efforts Monitoring care over time Benchmarking organizations’ safety progress and performance Hospital profiling, public reporting, pay-forperformance Unintended Uses Borzecki, Academy Health 2009 6 How Valid are the PSIs? Face validity – Established through expert panel consensus process (AHRQ Pub. No. 02-0038) Construct validity – Correlations and factor analysis (AHRQ Pub. No. 02-0038) Predictive validity – Increased LOS and in-hospital death, costs (Zhan, JAMA 2003; Rivard, Med Care Res Rev 2008) Criterion validity – Relatively little known both in and outside VA (Romano, Health Serv Res 2009) Borzecki, Academy Health 2009 7 Project Objectives 1. Develop collaborations with key stakeholders to guide in PSI selection and validation 2. Investigate criterion validity of the PSIs – Positive predictive value = True Positives Flagged Cases 3. Identify processes and structures of care associated with individual PSIs 4. Revise and improve PSIs using multiple data sources and settings of care 5. Assess utility validity of PSIs for QI and performance measurement Borzecki, Academy Health 2009 8 Methods Retrospective observational study using VA administrative and electronic medical record (EMR) data from FY04-07 Data Sources: – VA National Patient Care Database Patient Treatment File • Inpatient discharge information: diagnoses, procedures, DRGs, admission and discharge dates – VistAWeb - EMR Applied AHRQ PSI software (v. 3.1a) to inpatient dataset - individual PSI and composite rates Borzecki, Academy Health 2009 9 Hospital Sampling Strategy N=158 N=158 Hospitals Hospitals Stratify Hospitals by # of Observed and Expected PSIs Group Group 11 N=28 N=28 >4 Observed & & Expected Expected >4 Observed Group Group 22 N=33 N=33 >2 >2 Observed Observed & & Expected Expected Group Group 33 N=18 N=18 >1 Observed & & Expected Expected >1 Observed Rank Hospitals by PSI Composite Rate Within Each Group Select top 3 & bottom 3 hospitals from each group Randomly select from remaining hospitals in each group: gp 1=4, gp 2=4, gp 3=2* Group Group 11 N=10 N=10 Group Group 22 N=10 N=10 Group Group 33 N=8 N=8 * Geographic distribution & ICU severity also taken into account Borzecki, Academy Health 2009 10 Methods – Chart Abstraction Identified 112 cases for each indicator (N=28 x 4) Trained nurse abstractors (2 RNs/PSI) Modified existing AHRQ chart abstraction tools, developed new tools as required – Clinician review, pre-pilot testing, piloting, IRR Completed chart abstraction: – Post-operative PE/DVT, Accidental Puncture or Laceration, Iatrogenic Pneumothorax – Post-operative Sepsis, Wound dehiscence and Hemorrhage or Hematoma Borzecki, Academy Health 2009 11 Results: Post-op PE/DVT Total # of cases: 112 False Positives True Positives 49 cases, 44% (CI 34%-53%) Present on admission (POA) 16 cases (14%) Pre-procedure diagnosis 13 cases (12%) Coding-related inaccurate diagnosis 14 cases (13%) Arterial thrombosis 4 cases Negative PE/DVT workup 4 cases “Rule out PE” 3 cases Superficial thrombosis or thrombophlebitis 3 cases Remote history of PE/DVT 10 cases (9%) Miscellaneous 10 cases (9%) 63 cases (56%) Borzecki, Academy Health 2009 12 Accidental Puncture or Laceration (APL) Total # of cases: 112 False Positives True Positives True Positives 96 cases, 86% True Positives 96 cases, 86% 96 cases, 86% (CI 78%-92%) No accidental puncture during admission 6 cases (5%) POA 6 cases (5%) Miscellaneous 4 cases (4%) 16 cases (14%) Borzecki, Academy Health 2009 13 Iatrogenic Pneumothorax (PTX) Total # of cases: 112 False Positives True Positives 90 cases, 80% True Positives 90 cases, 80% (CI 72%-87%) No PTX during admission 6 cases (5%) Spontaneous PTX 6 cases (5%) POA 5 cases (4%) Previous exclusionary procedure 4 cases (4%) Remote history of PTX 1 case (1%) 22 cases (20%) Borzecki, Academy Health 2009 14 Summary PSI algorithms for APL and PTX demonstrate high positive predictive values for detecting true PSI events PE/DVT algorithm does not For all 3 PSIs, most common cause of FPs was due to problem being present on admission Borzecki, Academy Health 2009 15 Study Limitations Not all potentially useful data elements were available for abstraction, e.g., anesthesia reports; surgery reports (missing on occasion, although rare) Inter- and intra-hospital variation in medical record documentation Borzecki, Academy Health 2009 16 Study Strengths Methods – Administrative data source – reliable, accurate – Comprehensive, integrated, central EMR – Hospital sample selection – representative – Abstraction process • Tool development, iterative revision • Inter-rater reliability testing • Physician review of ambiguous cases Borzecki, Academy Health 2009 17 Implications/Significance Accuracy of all PSI algorithms could be improved by relatively simple coding enhancements: – Use of POA codes – Explore use of “997” complication code as part of PSI algorithm to capture post-operative PSIs Training: – Targeted education to help coders understand how to code adverse events and specific complications of care Borzecki, Academy Health 2009 18 Contact Information Ann Borzecki, MD, MPH Center for Health Quality, Outcomes & Economic Research (VA Center of Excellence) Boston University Schools of Public Health and Medicine, Departments of Health Policy and Management and Internal Medicine E-mail: ann.borzecki@va.gov Borzecki, Academy Health 2009 19 Positive Predictive Value (PPV) Comparison Across Studies PPV % (95% CI) N VA AHRQ UHC PE/DVT 44 (34-53) 112 60 121 62 452 APL 86 (78-92) 112 91 (86-94) 249 ----- PTX 80 (72-87) 112 78 (70-82) 205 Borzecki, Academy Health 2009 ----- 20
