WHICH RISK
ADJUSTMENT MODEL
SHOULD WE USE? A
FINNISH POINT OF VIEW
16.3.2011
Matti Reinikainen
North Karelia Central Hospital, Joensuu
Pohjois-Karjalan sairaanhoito- ja sosiaalipalvelujen kuntayhtymä
www.pkssk.fi
THE FINNISH INTENSIVE
CARE CONSORTIUM
1994
2007
• So far, benchmarking in the Finnish Intensive
Care Consortium has been mainly based on
SAPS II
– Based on “The Severity Study”
– 13 152 patients (720 from 7 Finnish hospitals)
– Le Gall JR, Lemeshow S, Saulnier F. A new Simplified
Acute Physiology Score (SAPS II) based on a
European/North American multicenter study. JAMA
1993; 270: 2957-63.
• APACHE II data is also collected
APACHE II vs. SAPS II
• same basic principle, values of physiologic
parameters from the first 24 hrs in the ICU
• APACHE II (Acute Physiology And Chronic
Health Evaluation II): the diagnostic category
weight is added to the logit
• SAPS II (Simplified Acute Physiology Score II):
the diagnosis is not needed; instead the type of
admission (scheduled surgical, unscheduled
surgical, medical) affects the score
ARE THE OLD MODELS GOOD
ENOUGH?
• APACHE II
- from 1985
- not always easy to choose the right diagnostic
category
• SAPS II
- from 1993
- advantage: no diagnosis needed
- disadvantage: does not take into account the
diagnosis
DOES THE RISK PREDICTED BY
SAPS II REFLECT REALITY?
• A patient example:
- HR 110/min
- age 65 years
- SAPs 84 mmHg
- no difficult chronic diseases
-Tc 38 ºC
- a medical admission
- consciousness, renal
function, blood cell counts,
electrolytes quite OK
- respiratory insufficiency,
need for mechanical ventilation, PaO2/FIO2 250 mmHg
(33.3 kPa)
- HCO3- 18 mmol/l
• PROBABILITY OF IN-HOSPITAL DEATH ?
DOES THE RISK PREDICTED BY
SAPS II REFLECT REALITY?
• A patient example:
- HR 110/min
- age 65 years
- SAPs 84 mmHg
- no difficult chronic diseases
-Tc 38 ºC
- a medical admission
- consciousness, renal
function, blood cell counts,
electrolytes quite OK
- respiratory insufficiency,
need for mechanical ventilation, PaO2/FIO2 250 mmHg
(33,3 kPa)
- HCO3- 18 mmol/l
• SAPS II score 32 points → probability 0.128
• SAPS II –score 32 → probability 0.128
- the database of the Finnish Consortium, 19982007, readmissions excluded: 2319 patients, with a
SAPS II score of 32 points
- hospital mortality 8.4%
• SAPS II –score 32 → probability 0,128
- the database of the Finnish Consortium, 19982007, readmissions excluded: 2319 patients, with a
SAPS II score of 32 points
- hospital mortality 8.4%
- diabetic ketoacidosis (n = 26): mort 0%
- drug intoxication (n = 108): mort 0.9%
- congestive heart failure (n = 49): mort 22.4%
CAN SAPS II STILL BE USED?
• It overestimates the risk of death – leads to ”grade
inflation”
• If most intensive care units are graduating with
honors, is it genuine quality or grade inflation?
Popovich MJ, Crit Care Med 2002
• Recalibrations are needed
SMR 1998 – 2007, FINNISH INTENSIVE CARE CONSORTIUM
1.80
25000
1.60
20000
1.40
SMR based on new
calibration
1.20
15000
1.00
N
SMR
N
Konsortio SMR Original
Konsortio SMR 2008
0.80
10000
SMR based on original
SAPS II model
0.60
0.40
5000
0.20
0.00
0
1998
1999
2000
2001
Tehohoidon laatupäivät Helsingissä 1.4.2008
Päivitetty 09.04.2008
2002
2003
2004
2005
2006
2007
CAN SAPS II STILL BE USED?
• It can be used for monitoring changes in a unit’s
own results
• Can be used for benchmarking purposes if the units
to be compared have similar case-mix
• Should not be used to compare results of units with
major differences in case-mix
SAPS 3 WAS CONSIDERED IN
FINLAND TOO - IS IT A GOOD
ALTERNATIVE?
• Values of physiological parameters ± 1 h of ICU
admission
• Reason for ICU admission documented more
precisely than in SAPS II
• Takes into account pre-ICU care
• Prognostic performance?
• Quality of data collected??
The SAPS 3 Study
Metnitz et al ICM 2005: 31:1336-1344. (Part 1)
Moreno et al. ICM 2005: 31:1345-1355. (Part 2)
• At first 22,791 admissions
• Exclusions: readmissions (1455), < 16 yrs (628),
those without ICU admission or discharge data
(1074) and those that lacked an entry in the field
”ICU outcome” (57)
- SAPS 3 basic cohort: 19,577 patients
The SAPS 3 Study
Metnitz et al ICM 2005: 31:1336-1344. (Part 1)
Moreno et al. ICM 2005: 31:1345-1355. (Part 2)
• SAPS 3 basic cohort: 19,577 patients
• More exclusions: patients with a missing entry in
the field of ”vital status at hospital discharge”
(2540) and those still in hospital (253)
– SAPS 3 Hospital outcome cohort: 16,784 patients
• Quality of data? – at first, 5.5% of patients excluded
because of missing data; then 13% of the remaining
population excluded because of missing data on
vital status
The SAPS 3 Study
Metnitz et al ICM 2005: 31:1336-1344. (Part 1)
Moreno et al. ICM 2005: 31:1345-1355. (Part 2)
• How about data completeness?
– ”Data completeness was found to be satisfactory with 1
[0-3] SAPS II parameter missing per patient”
• How many SAPS 3 parameters were missing?
– ???
– Were the physiological values obtained within ± 1 h?
SAPS 3 – even if data quality in the study was
less than perfect, does it work?
• Ledoux D et al. SAPS 3 admission score: an external
validation in a general intensive care population. Intensive
Care Med 2008; 34: 1873-7.
– single-centre (Belgium), 802 patients
– “the SAPS 3 … model customised for Central and Western Europe
… was not significantly better than the SAPS II.”
• Poole D et al. External validation of the Simplified Acute
Physiology Score (SAPS) 3 in a cohort of 28,357 patients
from 147 Italian intensive care units. Intensive Care Med
2009; 35: 1916-24.
– “…the SAPS 3 score calibrates inadequately in a large sample of
Italian ICU patients and thus should not be used for benchmarking,
at least in Italian settings”
• Sakr Y et al. Comparison of the performance of SAPS II,
SAPS 3, APACHE II, and their customized prognostic
models in a surgical intensive care unit. Br J Anaesth 2008;
101: 798-803.
– single-centre (Germany), 1851 patients
– “… the performance of SAPS 3 was similar to that of APACHE II
and SAPS II. Customization improved the calibration of all
prognostic models.”
• Metnitz B, Schaden E, Moreno R, Le Gall JR, Bauer P,
Metnitz PG; ASDI Study Group. Austrian validation and
customization of the SAPS 3 Admission Score. Intensive
Care Med 2009; 35: 616-22.
– 22 ICUs in Austria, 2060 patients
– “The SAPS 3 … general equation can be seen as a framework … For
benchmarking purposes, region-specific or country-specific equations
seem to be necessary...”
• 2 ICUs in Norway, 1862 patients
• “The performance of SAPS 3 was satisfactory, but not
markedly better than SAPS II.”
• SAPS II showed better discrimination
• SAPS 3 equations showed better calibration
• “…in our experience the scoring process is more timecomsuming and complex than that for SAPS II.”
SAPS 3, CONCLUSION:
• Does it work? – Yes!
• However, prognostic performance is NOT better
than that of SAPS II
• the scoring process is more time-comsuming and
complex than that for SAPS II (experience from
Norway)
• on the other hand: according to many studies, the
calibration of SAPS II is poor and customisation is
needed
QUESTION DISCUSSED IN
FINLAND:
• Should we implement a new risk-adjustment
model (SAPS 3) that
– is not better than the old ones
– is more time-consuming
– would require customisation
• Or should we go on with one of the old models
(that also require customisation)?
FINNISH (at least temporary)
SOLUTION: OWN CUSTOMISED
PREDICTION MODEL
• One objective: no need to exclude patient groups
for benchmarking
– neuro- and cardiac surgical patients are not excluded
• We did not want to increase the burden of data
collection – no new parameters added
• SAPS II –based data collection preserved
– possible to compare the results with those of previous
years
– possible to describe the population using a well-known
scoring system
OWN CUSTOMISED MODEL
- M Reinikainen, P Mussalo, V Kiviniemi, V Pettilä, E Ruokonen
•
•
•
•
•
Based on patients treated in 2007-2008
Readmissions excluded
Age ≥ 18 yrs
Those discharged to another ICU excluded
n = 25 801
OWN CUSTOMISED MODEL
• Outcome variable (to be predicted) ”DEATH IN
HOSPITAL”
• Explaining covariates:
–
–
–
–
–
Emergency admission or planned beforehand
Surgical postoperative or medical
SAPS II score without admission type points
ln ((SAPS II score without admission type points) + 1)
Diagnostic groups having an independent impact on the
probability of death
• First a binary variable (0,1) was made of every APACHE
III –dg group; everyone of these was tested separately
• 31 dg groups with an independent effect were included in
the model
LOGISTIC REGRESSION ANALYSIS
logit = β0 + β1X1 + β2X2 + … + βiXi
- the regression analysis produces the constant β0 and the
coefficients βi
-the logit can be calculated when the parameter values Xi are
known
- the logit (log odds) can also be expressed as
R
logit  ln
1 R
and thus
elogit
R
1  elogit
CALCULATING THE RISK R
R
logit  ln
1 R
e
logit
R

1 R
R  elogit - R(elogit )
(1  elogit )R  elogit
elogit
R
1  elogit
LOGIT = -7,796 + 0,049 x (SCORE_SAPS_WITHOUT_ADM_TYPE_POINTS)
+ 1,013 x (ln(SAPS_WITHOUT_ADM_TYPE_POINTS + 1))
+ 0,767 (if emergency admission) - 0,219 (if post-operative admission)
+ 1,229 (if DG_NONOP_CARDIOGENIC_SHOCK) + 0,364 (if DG_NONOP_CARDIAC_ARREST)
– 0,796 (if DG_NONOP_RHYTHM_DISTURBANCE) + 0,348 (if DG_NONOP_ACUTE_MYOCARDIAL
INFARCTION) + 0,422 (if DG_NONOP_BACTERIAL_OR_VIRAL_PNEUMONIA)
– 1,619 (if DG_NONOP_MECHANICAL_AIRWAY_OBSTRUCTION)
+ 0,306 (if DG_NONOP_OTHER_RESP_DISEASES) + 0,795 (if DG_NONOP_HEPATIC_FAILURE)
+ 0,703 (if DG_NONOP_GI_PERFORATION_OR_OBSTRUCTION) + 0,643 (if DG_NONOP_GI
BLEEDING_DUE_TO_VARICES) + 0,431 (if DG_NONOP_OTHER_GI_DISEASES)
+ 0,790 (if DG_NONOP_INTRACEREBRAL_HAEMORRHAGE)
+ 0,654 (if DG_NONOP_SUBARACHNOID_HAEMORRHAGE) + 0,400 (if DG_NONOP_STROKE)
– 1,427 (if DG_NONOP_NEUROLOGIC_INFECTION) - 1,266 (if DG_NONOP_SEIZURE)
– 0,486 (if DG_NONOP_OTHER_NEUROLOGIC_DISEASES) - 0,679 (if DG_NONOP_MULTIPLE
TRAUMA_WITHOUT_HEAD_TRAUMA) – 0,658 (if DG_NONOP_METABOLIC_COMA)
– 2,126 (IF DG_NONOP_DIABETIC_KETOACIDOSIS) – 2,245 (if DG_NONOP_DRUG_OVERDOSE)
– 1,150 (if DG_NONOP_OTHER_METABOLIC_DISEASES) – 0,752 (if DG_NONOP_OTHER
MEDICAL_DISEASES) + 0,340 (if DG_POSTOP_DISSECTING_OR_RUPTURED_AORTA)
– 0,701 (if DG_POSTOP_CABG) + 0,701 (if DG_POSTOP_PERIPH_ARTERY_BYPASS_GRAFT)
+ 0,470 (if DG_POSTOP_GI_PERFORATION_OR_RUPTURE) + 0,411 (if DG_POSTOP
GI_OBSTRUCTION) – 0,522 (if DG_POSTOP_SUBDURAL_OR_EPIDURAL_HAEMATOMA)
– 0,885 (if DG_POSTOP_CRANIOTOMY_FOR_NEOPLASM)
– 1,620 (if DG_POSTOP_OTHER_RENAL_DISEASES)
PROB = EXP(LOGIT) / (1 + EXP(LOGIT))
PATIENT EXAMPLE
• A patient example:
- HR 110/min
- age 65 years
- SAPs 84 mmHg
- no difficult chronic diseases
-Tc 38 ºC
- a medical admission
- consciousness, renal
function, blood cell counts,
electrolytes quite OK
- respiratory insufficiency,
need for mechanical ventilation, PaO2/FIO2 250 mmHg
(33.3 kPa)
- HCO3- 18 mmol/l
• SAPS II score 32 points → probability 0.128
PATIENT EXAMPLE
• SAPS II score 32 → probability 0.128
• New customised model:
– If none of the diagnoses included in the model:
probability 0.082
– dg bacterial pneumonia: probability 0.12
– dg drug intoxication: probability 0.0094
AUROC
- APACHE II: 0.84
- SAPS II: 0.84
- new customised
model: 0.87
- H-L test for new
model: p = 0.127
CONCLUSIONS
• SAPS 3 works, but its prognostic performance is
not better than that of SAPS II
• If you want to use SAPS 3, you should probably
customise it
• If you want to use SAPS II, you should probably
customise it
• Idea for future research: to create a Nordic risk
adjustment model, predicting 6-month or 1-year
mortality