Inpatient Pediatric Quality Indicators:
Limitations of Small Numbers
Naomi Bardach, MD1,2
Alyna Chien, MD3
R. Adams Dudley, MD MBA2
June 27, 2010
AcademyHealth Conference
1 Division
of General Pediatrics, UCSF
2 Institute for Health Policy Research, UCSF
3 Boston Children’s Hospital, Harvard University
• We have no financial relationships or
conflicts of interest to disclose
• Funding: T32 HD044331 NICHD
Background
• Variation
V i ti iin iinpatient
ti t quality
lit off care,
leading to poor patient outcomes
• Dearth of pediatric inpatient quality
measures
• Federal legislation, 2009 (CHIPRA):
– Funded development of pediatric quality
measures. Intended uses:
• Public reporting
• Payment programs
Background
• The Agency for Healthcare Research and
Quality
y ((AHRQ)) developed
p PeDiatric
quality Indicators (PDIs)
• Measure important adverse events in
admitted medical and surgical pediatric
patients
• Designed for use with administrative data
• Risk
Risk--adjusted
Background
• Some
S
PDIs
PDI have
h
b
been nationally
ti
ll endorsed
d
d
and are already used for public reporting
of hospital comparison
• National rates of the PDIs noted to be low
• Some hospitals have low volumes of
pediatric admissions
• The combination of low rates and low
volumes
l
may lilimitit th
the use off PDI
PDIs ffor
hospital comparison
McDonald, Pediatrics July 2008
Objective
• Determine the percentage of hospitals with
adequate
q
volume to meaningfully
g y compare
p
performance using the PDIs
– Hypothesis: There may not be adequate
volumes of pediatric patients in some
hospitals
Study design and Data source
• Cross sectional design, 20052005-2007
• California
California’s
s Office of Statewide Health
Planning and Development (OSHPD)
d t b
database
• Administrative discharge data from all
non--federal hospitals in CA
non
• Included ages <18 years
AHRQ indicators used
•
•
•
•
•
•
•
•
•
Iatrogenic Pneumothorax Non
Non--neonates
Accidental Puncture or Laceration
Postoperative Wound Dehiscence
Decubitus Ulcer
Postoperative Hemorrhage or Hematoma
Selected Infections Due to Medical Care
Postoperative Respiratory Failure
Postoperative Sepsis
Pediatric Heart Surgery Mortality
Methods: Analyses
• State level analyses
– Event rates for each indicator
– Number of eligible patients needed to detect
performance two times worse the event rate
• One
One--sided sample size calculation with 2x effect
size and power of 80%
• Hospital level analyses
– Number of eligible patients at each hospital
• Used AHRQ
Q publicly
p
y available software
Methods: Outcomes
• Proportion of hospitals with adequate
volume of eligible
g
p
patients to detect
differences in performance for each
measure
• Proportion of patients eligible for each
indicator seen at a hospital with adequate
volume
Results
• In 20052005-2007, 381 hospitals in CA
admitted any
yp
patients <18 yyrs old
• N(2005
N(2005--2007)=2,333,556
AHRQ indicator rates
Indicator
Mean Rate/1000
Iatrogenic Pneumothorax
Accidental Puncture or Laceration
Postop
p Wound Dehiscence
Decubitus Ulcer
Postop Hemorrhage
Infections Due to Medical Care
0.2
0.6
Postop Respiratory Failure
8.1
81
Postop Sepsis
Heart Surgery Mortality
19.0
19
38.0
38
0
0.9
1.4
2.0
2.6
AHRQ indicator rates
Indicator
Mean Rate/1000
Iatrogenic Pneumothorax
Accidental Puncture or Laceration
Postop
p Wound Dehiscence
Decubitus Ulcer
Postop Hemorrhage
Infections Due to Medical Care
0.2
0.6
Postop Respiratory Failure
8.1
81
Postop Sepsis
Heart Surgery Mortality
19.0
19
38.0
38
0
0.9
1.4
2.0
2.6
Percent of hospitals with adequate volume
to detect 22-fold worse performance
Measure
Iatrogenic Pneumothorax
Accidental Puncture or Laceration
P t Wound
Postop
W
d Dehiscence
D hi
Decubitus Ulcer
Postop Hemorrhage
Infections Due to Medical Care
Postop Respiratory Failure
Postop Sepsis
Heart Surgery Mortality
Minimum
sample size
% Hospitals
p
exceeding
g
or meeting number of
eligible patients
49,869
49
869
x9,165
9,165
x3,922
3,922
3 922
x5,956
5,956
12,516
0.0
0
0
0.0
0
0.7
07
1.1
3.0
xx976
976
xx419
419
3,125
xx201
201
3.9
6.6
19.0
19
25
25.0
Percent of patients seen at hospitals with adequate
volume to detect 22-fold worse performance
%
Iatrogenic Pneumothorax
Accidental Puncture or Laceration
Postop Wound Dehiscence
D
Decubitus
bit Ul
Ulcer
Postop Hemorrhage
Infections Due to Medical Care
Postop Respiratory Failure
Postop Sepsis
Heart Surgery Mortality
00
0
27
00
0
24
22
66
64
80
92
Percent of patients seen at hospitals with adequate
volume to detect 22-fold worse performance
%
Iatrogenic Pneumothorax
Accidental Puncture or Laceration
Postop Wound Dehiscence
D
Decubitus
bit Ul
Ulcer
Postop Hemorrhage
Infections Due to Medical Care
Postop Respiratory Failure
Postop Sepsis
Heart Surgery Mortality
00
0
27
00
0
24
22
66
64
80
92
Summary
• 25% or less
l
h
hospitals
it l iin CA h
have enough
h
eligible patients to detect meaningful
differences in PDI performance even using
3 years of data
• For some of the measures, the majority of
eligible
e
g b e pat
patients
e ts a
are
e see
seen at hospitals
osp ta s with
t
adequate volume
• For others
others, most eligible patients are seen
at hospitals without adequate volume
Limitations
• California data
– Sensitivityy analyses
y
• Clinically significant differences in
performance may occur at lower than 2
2-fold differences
Implications and Conclusions
• Using
U i th
the PDI
PDIs ffor quality
lit measurementt
may underunder-recognize poor performance
for many hospitals caring for children
• Solutions:
– Measure care only at high volume hospitals
– Delineate minimum number of patients
needed to report on a given measure
– Develop measures that are applicable to a
broader group of children, seen at all
hospitals,
p
, not jjust children’s hospitals
p
Acknowledgements
UCSF Phillip R
R. Lee
Lee-Institute for Health
Policyy Studies
• Adams Dudley
• James Anderson
Boston Children
Children’s
s
• Alyna T. Chien
UCSF Division of
General Pediatrics
• Michael Cabana
• Bonnie Halpern
Halpern-Felsher
Measure rates with and without
POA conditions
Measure
Rate
Rate
Iatrogenic Pneumothorax
Accidental Laceration
Postop
p Wound Dehiscence
Decubitus Ulcer
Postop Hemorrhage
Infections Due to Medical Care
0.2
0.8
0.2
0.6
0.9
3.1
2.4
3.6
0.9
1.4
2.0
2.6
Postop Respiratory Failure
9.1
91
8.1
81
Postop Sepsis
Heart Surgery Mortality
31.0
31.0
38.0
38
.0
0
19.0
19.0
38.0
38
.0
0
California and National distributions of
eligible pediatric discharges per hospital
Measure
Iatrogenic Pneumothorax
Accidental Laceration
Postop
p Wound Dehiscence
Decubitus Ulcer
Postop Hemorrhage
Infections Due to Medical Care
Postop Respiratory Failure
Postop Sepsis
Heart Surgery Mortality
California
Nation
372 (67
(67--1097)
373 (67
(67--1109)
198 (30
(30--618)
215 (32
(32--666)
9 ((3(3-27))
21 (5
(5--88)
5 (2(2-16)
313 (48(48-876)
6 ((2(2-15))
10 (3
(3--39)
6 (2(2-15)
154 (21
(21--526)
4 (1(1-16)
4 (1(1-17)
6 (2(2-15)
3 (1
(1--10)
3 (1
(1--85)
4 (1
(1--145)
Inpatient Pediatric Quality Indicators:
Limitations of Small Numbers
Naomi Bardach, MD
Alyna Chien, MD
R. Adams Dudley, MD MBA
June 27, 2010
AcademyHealth Conference