STANDARDIZED MORTALITY RATIOS FOR THE MOST
FREQUENT CARDIAC SURGICAL PROCEDURES
UTILIZING PEDIATRIC INDEX OF CARDIAC SURGICAL
INTENSIVE CARE MORTALITY
Gall C, Katch A, Rice T, Jeffries HE, Kukuyeva I, and Wetzel RC
OBJECTIVES
• Verify the performance (including calibration
and discrimination) of PICSIM on the most
frequently performed cardiac surgical cases in
the VPS database.
• Use PICSIM to evaluate post-operative
mortality outcomes by calculating
Standardized Mortality Ratios.
BACKGROUND
• The Pediatric Index of Surgical Intensive Care
Mortality (PICSIM) score was developed to
support robust risk-adjustment for the pediatric
cardiac surgical population cared for in pediatric
Intensive Care Units (ICUs) [1].
• While PICSIM performs well in a general pediatric
cardiac ICU surgical population, and outperforms
PIM 2, PRISM 3, and STAT severity of illness
scores overall [2], its performance in predicting
post-operative outcomes for individual diagnostic
categories is unclear.
BACKGROUND cont.
• Development of a risk-adjustment tool that enables a
consistent comparison of mortality outcomes both for
the overall pediatric cardiac surgical sample as well as
for cohorts of children receiving the same cardiac
surgical interventions will enable surgical techniques,
pre-, peri-, and post-operative care strategies to be
effectively evaluated.
• To explore the use of PICSIM to evaluate the ICU
outcomes of children following cardiac surgery and
post-operative ICU care, data from the Virtual PICU
Systems, LLC (VPS) database were selected to inform
this question.
BACKGROUND cont.
• VPS, LLC is the largest network in pediatric critical care.
• Formed by the National Association of Children’s
Hospitals and Related Institutions (NACRI, now CHA),
Children’s Hospital Los Angeles, and Children’s Hospital
of Wisconsin to develop a tool for prospective data
collection.
• VPS database has nearly 1,000,000 ICU encounters.
• Trained clinicians collect and enter VPS data.
• VPS performs quarterly inter-rater reliability (IRR)
testing & quality validation
• IRR of VPS data is consistently above 90%
METHODS
• This study was waived for approval by Seattle Children’s IRB.
• Cardiac surgical patients were defined as patients with a cardiac
diagnosis that had an STS (Society of Thoracic Surgeons) registered
surgical procedure.
• Inclusion criteria for PICSIM:
– ICUs that managed postoperative pediatric cardiac surgical
patients.
– Patients that have estimates for both the Pediatric Index of
Mortality 2 (PIM 2) and the pediatric risk of mortality (PRISM 3)
scores.
– Cardiac surgical patients that had surgery either before or after
admission to the ICU.
– Patients with a STAT score that could be derived from the index
procedure.
METHODS
• Step 1
Find the most frequently performed cardiac surgical
procedures. We focused on the 25 most frequently
performed cardiac surgical procedures by volume.
These procedures represented 66% of all cardiac
surgical procedures in the VPS database.
METHODS cont.
• Step 2
Evaluate PICSIM model accuracy via AUC. One way to
evaluate model accuracy is by finding the area under the
curve (AUC) of the Receiver Operating Characteristic (ROC)
curve. The Receiver Operating Characteristic Curve shows
how well the model correctly and incorrectly classifies patient
deaths. Accuracy is measured by the Area Under the ROC
Curve: an area of 1 represents perfect classification; an area
of 0.5 shows a classification made via random classification of
patients into the two categories.
METHODS cont.
• Step 3
Evaluate PICSIM model calibration via HosmerLemeshow. One way to assess model calibration is
via Hosmer-Lemeshow (HL). HL evaluates how close
the observed and predicted values are, for a given
group of patients. A specified number of (equallysized) groups is chosen a priori and patients are
assigned to the groups according to their predicted
mortality rates.
METHODS cont.
• Step 4
Evaluate PICSIM model calibration via Standardized Mortality ratios
(SMRs). Another way to determine how well a model is calibrated
is to analyze the SMRs and confidence intervals of the 25 surgical
procedures. An SMR is a ratio of the total number of observed
deaths for a given procedure to the total number of expected
deaths for the same procedure. Across procedures, we expect the
SMRs to fluctuate around 1. If a majority of the SMRs deviate from
1, this suggests that the model has demonstrated 'drift' and is not
precise. SMRs were calculated to compare observed deaths with
expected deaths, using PICSIM for prediction. SMRs were
computed for each of the top 25 cardiac surgical index procedures
by volume, as well as over all top 25 procedures. Confidence
Intervals (CI) at the 95% significance level were calculated for each
procedure and results were plotted (in the figure)
RESULTS
• 66 VPS ICUs participating (July 2009 - March 2014) were
included in the analysis (n=20,076). PICSIM showed satisfactory
discrimination (AUC=0.89) & calibration via HL (HL: P=0.181) and
SMR criteria.
• The combined SMR for all top 25 procedures was 0.94 (95% CI:
0.86-1.02).
• The SMRs and confidence intervals for each of the top 25 most
frequently performed cardiac surgical procedures by volume is
shown in the figure; procedures were ranked by most to least
frequent, by volume.
• Two procedures had no associated deaths; SMRs could not be
generated.
• 18 of the remaining 23 procedures and the aggregate CIs
included 1.0.
CONCLUSIONS
• The results demonstrate that PICSIM’s capacity to
predict mortality for the most common surgical
categories is robust.
• PICSIM SMRs represent a reasonable approach
for comparison of mortality among ICUs for select
cardiac surgeries.
• Further development of PICSIM performance will
enable us to assess the impact of site, unit and
care delivery team characteristics on pediatric
ICU mortality for children and adults requiring
cardiac surgical intervention..
REFERENCES
•
Jeffries HE, Soto-Campos G, Katch A, Gall C, Rice TB, Wetzel RC: Pediatric Index of
Cardiac SurgicalIntensive Care Mortality (PICSIM) risk score for pediatric cardiac
critical care. Pediatric Critical Care Medicine. 2014. In review.
•
Wetzel RC, Jeffries HE, Katch A, Gall C, Kukuyeva I, Rice TB. Validation of the
Pediatric Index of Cardiac Surgical Intensive Care Mortality (PICSIM) score.
Accepted for poster presentation at the Pediatric Cardiac Intensive Care Society
Annual Meeting in Miami, FL, December 11-12, 2014.
•
LaRovere JM, Jeffries HE, Sachdeva RC, Rice TB, Wetzel RC, Cooper DS, Bird GL,
Ghanayem NS, Checchia PA, Chang AC, Wessel DL. Databases for assessing the
outcomes of the treatment of patients with congenital and pediatric cardiac
disease – the perspective of critical care. Cardiol Young 2008;18(Suppl 2): 130-36
PICSIM SMRs for VPS Cardiac
Procedures