Validating the Patient
Safety Indicators (PSIs) in
the Veterans Health
Administration (VA)
Ann Borzecki, MD, MPH
Center for Health Quality, Outcomes, and Economic
Research, Bedford VAMC, and Boston University
Schools of Public Health and Medicine
Borzecki, AcademyHealth 2010
Background - PSIs
 Set of indicators use administrative data to screen for
potentially preventable complications of inpatient care
 Intended to highlight areas requiring further study
 Increasing use for hospital profiling / public reporting
– Several NQF endorsed
– CMS added 4 PSIs + composite PSI to measures
tracked by hospital reporting initiative
 Relatively little data on certain aspects of validity
– Do they identify true events?
– Are these events preventable?
Borzecki, AcademyHealth 2010
2
Objectives
 Investigate criterion validity of Postoperative
Hemorrhage or Hematoma (PHH) & Postoperative
Respiratory Failure (PRF)
– Positive Predictive Value = True Positives
Flagged Cases
 Examine false positive cases in further detail to
determine reasons for incorrect flagging & improve
PSIs
Borzecki, AcademyHealth 2010
3
Methods
 Retrospective observational study
 Data Sources (2003-2007):
– VA National Patient Care Database Patient Treatment
File (inpatient discharge information)
– VistAWeb - EMR
 Applied AHRQ PSI software (v. 3.1a) to inpatient
dataset
individual PSI and composite rates
 Selected representative sample of VA hospitals based
on PSI rates and geographic distribution (N=28)
 2 RNs abstracted charts of 112 flagged cases per PSI
Borzecki, AcademyHealth 2010
4
Postop Hemorrhage or
Hematoma (PHH) – Def’n
Numerator:
 Discharges with secondary diagnosis code for
hemorrhage or hematoma and procedure code for
hemorrhage control or hematoma drainage
Denominator:
 All surgical discharges > age 18
Exclusion criteria:
 PHH present on admission (POA)
 Hemorrhage control or hematoma drainage procedure
–
–
Occurred prior to first OR procedure
Only OR procedure or part of initial OR procedure
Borzecki, AcademyHealth 2010
5
Results: PHH
Total # of cases: 112
False Positives
True Positives
81 cases,
72% (CI 63%-80%)
Present on admission (POA)
7 cases (6%)
Intra-op bleeding controlled
during original procedure
7 cases (6%)
No PHH identified
7 cases (6%)
I&D
3 cases
Pseudoaneurysm / groin
abscess
2 cases
No procedure required
4 cases (4%)
Non-eligible procedure
3 cases (3%)
Miscellaneous
3 cases (3%)
31 cases (28%)
Borzecki, AcademyHealth 2010
6
Postop Respiratory Failure (PRF)
– Def’n
Numerator:
 Discharges with secondary diagnosis code for acute
respiratory failure or procedure code for intubation or
mechanical ventilation (timing specific)
Denominator:
 All elective surgical discharges > age 18
Exclusion criteria (include):
 Respiratory failure POA
 Tracheostomy:
– Only OR procedure or before 1st OR procedure
 MDC 4 or 5 (respiratory or circulatory disorders)
 Neuromuscular disorder (ICD-9-CM diagnosis code)
Borzecki, AcademyHealth 2010
7
Results: PRF
Total # of cases: 112
False Positives
Failed coding criteria
True Positives
90 cases, 80%
True Positives
75 cases,
67% (CI 57%-76%)
7 cases (6%)
Ventilated post-op for
< specified time
2 cases
Not documented in EMR
4 cases
No OR procedure
1 case
Failed clinical criteria
30 cases (27%)
Non-elective admission
28 cases
Trach during initial OR
1 case
Post-op vent for airway
protection
1 case
37 cases (33%)
Borzecki, AcademyHealth 2010
8
Implications/Significance
 Improve accuracy (PPV) of PHH/PRF algorithms by:
– Coding enhancements
• PHH - POA codes
- Codes indicating timing of control of bleeding
• PRF - VA adoption of codes indicating admission
status
– Coder Training
• Help coders understand how to code adverse
events and specific complications of care
Whether these PSIs identify preventable events
unclear
Use with caution for public reporting or P4P
Borzecki, AcademyHealth 2010
9
Acknowledgements
Funding: VA Health Services Research and Development
(HSR&D) Service SDR 07-002







Amy Rosen, PhD, (PI)

(COLMR, VA Boston)

Haytham Kaafarani, MD, 
MPH

Shibei Zhao, MPH


Qi Chen, MD, MPH

Susan Loveland, MAT

Marlena Shin, JD, MPH 

Arlene Ash, PhD
Joanne Cuny, RN, BSN, MBA
Dijon Fasoli, RN, PhD, MBA
Amresh Hanchate, PhD
Kathleen Hickson, RN, MN
Kamal Itani, MD
Sally MacDonald, RN
Hillary Mull, MPP
Patrick Romano, MD, MPH
Michael Shwartz, PhD, MBA
Garth Utter, MD, MSc
Borzecki, AcademyHealth 2010
10
Contact Information
Ann Borzecki, MD, MPH
Center for Health Quality, Outcomes & Economic
Research (VA Center of Excellence)
Boston University Schools of Public Health and
Medicine, Departments of Health Policy and
Management and Internal Medicine
E-mail: ann.borzecki@va.gov
Borzecki, AcademyHealth 2010
11