Patient Safety Indicators’
Experiences in France and in
Switzerland
Jean-Marie Januel, PhD, MPH, RN
Senior Researcher
University hospital of Lausanne, Institute of Social and Preventive Medicine,
Lausanne, Switzerland
Health Services Research Seminar Series
UC Davis, Center for Healthcare Policy and Research
Sacramento, CA, US – May 9, 2013
IUMSP
Institut universitaire de médecine sociale et préventive, Lausanne
1
Disclosure Statement
Speaker’s Verbal Disclosure Statement:
Have you (or your spouse/partner) had a personal financial relationship in the last
12 months with the manufacturer of the products or services that will be discussed
in this CME activity?
___ Yes
_X_ No
(If yes, please state disclosures and resolutions)
Educational objectives
 To describe the French experience of PSI (focusing on
feasibility, validity and measurement variations between
providers)
 To describe the Swiss experience of PSI (focusing on
the use of PSI for assessing the impact of DRG
implementation in all Swiss hospitals)
 To discuss about an appropriate reference standard
for comparing Patient Safety Indicators (example
provided by a pilot study to compare the PSI 12 –
postoperative pulmonary embolism and deep vein
thrombosis – in hospitalized patients undergoing hip
arthroplasty between France and Switzerland)
BACKGROUND
Hypothesis to Develop New Indicators

To use Administrative Databases for assessing
adverse events related to healthcare that could
be preventable
McDonald K, Romano P, Geppert J, Davies SM, Duncan BW, Shojania KG, et al.
Measures of Patient Safety Based on Hospital Administrative Data - The Patient
Safety Indicators. Technical Review 5. AHRQ Publication No. 02-0038 . Rockville,
MD: Agency for Healthcare Research and Quality

Approach derived from the Complication
Screening Program (CSP) by Iezzoni LI et al.
Iezzoni LI, Foley SM, Heeren T, Daley J, Duncan CC, Fisher ES, Hughes J. A
method for screening the quality of hospital care using administrative data:
preliminary validation results. QRB Qual Rev Bull. 1992;18(11):361-371.
PSI Algorithm Model
ICD Codes for Secondary Diagnoses
related to Adverse Event Identification
PSI =
Population at risk, defined using
DRG codes, Diagnostic codes,
Procedure codes
Projects

20 Patient Safety Indicators (PSI) initially developed by
the Agency for Healthcare Research and Quality (AHRQ)
using ICD9-CM
McDonald K, et al. Measures of Patient Safety Based on Hospital Administrative
Data - The Patient Safety Indicators. Technical Review 5. AHRQ Publication No.
02-0038 . Rockville, MD: Agency for Healthcare Research and Quality

ICD10 adaptation of 15 PSI by the International
Methodology Consortium for Coded Health Information
in 2007 (www.imecchi.org)
Quan H., et al. AHRQ Advances in Patient safety 2: News Directions and
Alternatives Approaches. Vol 1. Assessment. Taxonomies and Measurement.
Agency for Health Care Research and Quality Publication 2008.
Januel JM, et al. Rev Epidemiol Sante Publique 2011;59 :341-350.
Achievements for PSI

Benchmark (external comparisons)
o Between healthcare providers (hospitals)
o Between healthcare systems (countries)

Surveillance (internal comparisons)
o Longitudinal studies (to assess epidemiological
peak)
o To assess changes in robustness relating of
changes in ICD coding rules / version used
The Patient Safety Indicators in the Literature
40
36**
Total citations for PSI
35
33
Review / Commentary / Editorial
30
Fréquency, n
Original study
26*
25
20
17
14
15
10*
10
5
6
5
1
1
2001
2002
0
2003
2004
2005
2006
2007
Year of publication
*Including one citation without abstract; **including two citations without abstract.
2008
2009
2010
The French experience of PSI
French Pilot Study – Objectives

To explore French data using PSI
algorithms (ICD10 version - IMECCHI)

To assess feasibility for PSI by
calculating a selection of PSI from
nationwide database in 2005 and 2006,
respectively

To validate selected PSI using chart
review
PSI estimates calculation for feasibility
Frequency, N
PSI #
Denominator
Prevalence
Numerator
/1000
stays
(σ)
Incidence density
/1000
days hosp.
(σ)
Mean Prevalence
at hospital level
/1000
stays
(σ)
1
3’124’476
707
0.23 (0.02)
0.05
(0.00)
0.14 (0.78)
3
2’923’535
20’734
7.09 (0.10)
0.66
(0.00)
10.11 (17.06)
5
8’973’343
426
0.05 (0.00)
0.01
(0.00)
0.03 (0.11)
7
6’078’340
2’177
0.36 (0.02)
0.06
(0.00)
0.33 (2.01)
10
3’101’929
21’605
6.97 (0.09)
1.29
(0.01)
4.79 (15.36)
12
3’123’112
16’719
5.35 (0.08)
0.91
(0.01)
3.62 (10.44)
13
1’169’288
6’074
5.19 (0.13)
0.61
(0.01)
2.70 (5.10)
15
7’989’654
5’819
0.73 (0.02)
0.12
(0.00)
0.56 (1.15)
16
8’973’561
53
0.01 (0.00)
0.00
(0.00)
0.00 (0.06)
17
756’768
2’392
3.16 (0.13)
0.68
(0.01)
1.43 (7.26)
18
57’499
803
13.97 (0.96)
2.89
(0.10)
5.00 (18.10)
19
570’404
1’788
3.13 (0.15)
0.71
(0.02)
1.22 (3.08)
20
150’808
10
0.07 (0.04)
0.01
(0.00)
0.02 (0.36)
Stratified analysis for PSI #12
(Postoperative PE /DVT)
120
30
2005 HMen
100
2006 HMen
2005
2005 FWomen
20
2006 FWomen
15
10
PSI /1000 discharges
PSI /1000 discharges
25
60
40
5
20
0
0
[18-40[
[40-65[
[65-75[
Age Categories
[75-85[ >= 85 ans
2006
80
1
[2-4[
[4-8[
[8-15[ [15-22[ [22-29[ >= 29
Lenght of Stay in Hospital
(days)
Estimating PPV based on three PSI

To compare PSI positive cases to a
retrospective chart review of medical records

Between 80 and 150 inpatients stays with
positive PSI at the university hospital of Lyon,
France

To calculate the Positive Predictive Value (PPV)

To identify potential reasons for false positive
cases
PPV of PSI #12 (postoperative PE/DVT)
University Hospital, Lyon, France (2008)
True Positive Cases
Positive PSI 12
Sample
N
Total cases
154
VPP
(%)
N
(100.0)
%
123
79.87
[CI 95%]
[72.66-85.89]
Sex
0,607
Men
61
(39.6)
45
73.77
[60.93-84.20]
Women
93
(60.4)
78
83.87
[74.80-90.68]
Type of surgery
<0,001
Hip Arthroplasty*
43
(27.9)
38
88.37
[74.92-96,11]
Knee Arthroplasty**
58
(37.7)
56
96.55
[88.09-99.58]
Femur Fracture
(Osteosynthesis)
19
(12.3)
15
78.95
[54.43-93.95]
Other orthopedic procedures
13
(8.4)
8
61.54
[31.58-86,14]
Digestive procedure
13
(8.4)
6
46.15
[19.22-74.07]
Thorax & Pulmonary
procedure
8
(5.2)
0
0.00
[0.00-36.94]
Chart review
Paper record only
Both paper and electronic
P
0,232
114
(74.0)
83
72.81
[63.67-80.72]
40
(26.0)
40
100
[91.19-100]
PPV of PSI #7 (Bacteremia-related to Catheter)
University Hospital, Lyon, France (2008)
True Positive Cases
Positive PSI 7
sample
Total cases
PPV
N
(%)
N
55
100
16
%
29.09
[CI 95%]
P
[17.63-42.90]
Sex
0.857
Men
40
72.73
12
30.00
[16.56-46.53]
Women
15
27.27
4
26.67
[7.79-55.10]
Hospital unit
0.445
Medicine
19
34.55
7
36.84
[16.29-61.64]
Infectious diseases
14
25.45
5
35.71
[12.76-64.86]
Surgery
16
29.10
2
12.50
[1.55-38.35]
Clinical Nutrition
3
5.45
1
33.33
[0.84-90.57]
ICU
3
5.45
1
33.33
[0.84-90.57]
Chart review
0.255
Paper record only
18
32.73
4
22.22
[6.41-47.64]
Electronic record only
32
58.18
9
28.13
[13.75-46.75]
Both paper and electronic
5
9.09
3
60.00
[14.66-94.73]
PPV of PSI #13 (postoperative sepsis)
University Hospital, Lyon, France (2008)
True Positive Cases
Positive PSI 13
sample
Total cases
PPV
N
(%)
N
%
[IC 95%]
81
(100)
21
25.93
[16.82-36.86]
Sex
P
0.351
Men
48
(59.26)
14
29.17
[16.95-44.06]
Women
33
(40.74)
7
21.21
[8.98-38.91]
Lenght of Stay
0.006
<20 days
40
(49.38)
5
12.50
[4.19-26.80]
≥20 days
41
(50.62)
16
39.02
[24.20-55.50]
Chart review
<0.001
Paper record only
27
(33.33)
14
51.85
[31.95-71.33]
Electronic record only
54
(66.67)
7
12.96
[5.37-24.90]
43
(53.09)
2
4.65
[5.68-15.81]
Coding rule
Code « R578 »
<0.001
The Swiss experience of PSI
IDoC Project (Switzerland)
 To assess the Impact of Diagnosis related groups
(DRG) implementation on patient Care and
professional practice in Swiss hospitals
o Overall Hospitals DRG implementation in Switzerland for
inpatient stays payment on 2012, January 1st
 Design
o 5 sub-projects based on several outcomes (Ethic; Law;
Nursing sensitive; AMI; and PSI)
 Sub-project “PSI”
o To monitor the possible effects of the generalization of DRGbased hospital reimbursement using Patient Safety Indicators
(PSI)
To take into account the number of SDx in
Models (hierarchical) better
 At inpatient Level
o To control differences in case mix (using
categorical variable)
 At Hospital Level
o To control differences in quality of coding
o By assessing adjusted average number of SDx
for each hospitals using negative binomial
regression models
SDx in a Previous Analysis of PSI #12
2008
2009
2010
Mean # of SDX
(observed)
2.11
(2.10 – 2.12)
2.19
(2.18 – 2.19)
2.34
(2.33 – 2.35)
Mean # of
adjusted SDX
(random effects of
hospital)
2.48
(2.47 – 2.49)
2.52
(2.51 – 2.53)
2.66
(2.65 – 2.67)
Between hospitals
variance
0.89
(0.88 – 0.90)
0.73
(0.72 – 0.73)
0.61
(0.61 – 0.62)
Mean # of
adjusted SDX
(random effects of
hospitals + fixed
effect for sex, age,
los and type of
hospitals)
2.66
(2.64 – 2.67)
2.66
(2.65 – 2.68)
2.78
(2.77 – 2.79)
Between hospitals
variance
0.74
(0.73 – 0.74)
0.62
(0.62 – 0.63)
0.55
(0.55 – 0.55)
% of variance
explained by fixed
effects variables
16.85%
15.07%
9.84%
Impact of Changes in ICD10 version
on PSI 12 estimates
 Changes in ICD10 version occurred between
2008 and 2010 in Switzerland
o 2008: ICD10-WHO
o 2009: ICD10-GM2008
o 2010: ICD10-GM2010
 Trend comparisons across years (2008, 2009
and 2010) using ICD-10-WHO
 To compare ICD-10-WHO to ICD-10-GM using:
o
Data from 2009
o
Data from 2010
Hierarchical Logistic Regression Model for PSI #12
2008
ICD-10-WHO
2009
ICD-10-WHO
2010
ICD-10-GM2008
ICD-10-WHO
ICD-10-GM2010
% postoperative
VTE using the PSI
#12 (observed)
0.34
(0.30 – 0.36)
0.38
(0.36 – 0.37)
0.38
(0.36 – 0.37)
0.38
(0.37 – 0.40)
0.38
(0.37 – 0.40)
% postoperative
VTE using the PSI
#12 (random effect
of hospitals)
0.34 (0.34 – 0.34 )
0.37
(0.37 – 0.37)
0.37
(0.37 – 0.37)
0.38
(0.38 – 0.38)
0.38
(0.38 – 0.38)
Between hospitals
variance
1.23
(0.84 – 1.80)
1.06
(0.72 – 1.56)
1.06
(0.72 – 1.56)
1.20
(0.82 – 1.75)
1.20
(0.82 – 1.75)
ICC
0.27
(0.20 – 0.35)
0.24
(0.18 – 0.32)
0.24
(0.18 – 0.32)
0.27
(0.20 – 0.35)
0.27
(0.20 – 0.35)
MOR
2.88
C-statistic
0.720
(0.709 – 0.730)
2.67
0.719
(0.709 – 0.729)
2.67
0.719
(0.709 – 0.729)
2.84
0.721
(0.711 – 0.731)
2.84
0.721
(0.711 – 0.731)
To Develop a New “PSI 13”
 To refine definition from Postoperative Sepsis to
Postoperative Hospital-Acquired Infections
 To develop a New Algorithm
 To compare New Algorithm to PSI 13 (AHRQ)
 To repeat models using data from 2008 to 2010,
with respect to changes in ICD-10 version used
each year in Switzerland
Approach for New Algorithm (ICD10)
Infections codes at SDx Fields
(3 scales indicator)
Criteria for Severity
using codes at SDx
Fields
Bacteremia / Blood Stream
Infection (A49, B95, B96)
Pneumonia (J13 to J18)
Nosocomial pneumonia (U6900)
Sepsis (A40, A41, A419 according
ICD10 version, and B377)
Septic shock (A419 or R572,
according the ICD10 version)
SIRS (R65)
Acute Organ
Dysfunction (AOD) –
Inspired from Angus et al.
(2001), Martin et al.
(2003), and Sundararajan
et al. (2005)
Healthcare-Related /
Nosocomial criteria at SDx
Fields
T-Codes for Healthcarerelated infections (T802,
T814, T826, T827, T835,
T836, T845, T846, T847,
T857, T874)
Y-Codes for “Nosocomial”
(Y95, Y69, Y62 according the
ICD10 version)
New Algorithm vs. PSI #13 (AHRQ)
2008 (ICD10-WHO)
N
PSI #13 (AHRQ
algorithm)
(%)
Nosocomial
criteria (%)
2009 (ICD10-GM2008)
2010 (ICD10-GM2010)
N
N
(%)
Nosocomial
criteria (%)
(%)
Nosocomial
criteria (%))
1 727 (0.50)
(1.91)
1 835 (0.52)
(13.51)
2 582 (0.54)
(14.56)
18 235 (5.23)
(0.67)
22 077 (5.90)
(4.96)
23 646 (4.95)
(6.71)
Sepsis /
Severe Sepsis
809 (0.23)
(0.26)
1 327 (0.35)
(13.79)
2 617 (0.74)
(18.27)
Septic Shock
637 (0.18)
(2.35)
800 (0.21)
(11.88)
913 (0.19)
(15.66)
New Algorithm
(including 3
levels of
severity)
Bacteremia /
Pneumonia
PSI rate “high” or “low ”?
A pilot study to compare Swiss and
French data for PSI #12
(Postoperative PE/DVT)
To which Benchmark / Reference Standard
Compare PSI 12 (postoperative PE/DVT)?
 The Hypothesis
o Need a BENCHMARK for assessing comparisons
across clusters (e.g., hospitals, countries…) in
studies for healthcare quality improvement
 Possible benchmarks / reference standards
o Risk-adjusted observed vs. expected rate (Funnel
plots)
o Benchmark vs. « best in class » (Forest plots)
o Zero event
o Evidence-based data / information
Development of an evidence-based benchmark
for PSI 12 (postoperative PE/DVT)
 Proposal: to compare actual values to an
evidence-based reference standard
 Three step methodology
o To develop a “reference standard” using the baseline risk of
Postoperative PE/DVT occurring in hospitalized patients
undergoing hip arthroplasty, under appropriate prophylaxis,
using a systematic review with meta-analysis
o To estimate adjusted occurrence rates of PSI 12 in patients
undergoing hip arthroplasty (pilot study using data from
Switzerland and France)
o To compare these rates against the “reference standard” as a
benchmark (or a target) developed using the meta-analysis
Systematic Review
(Januel JM, et al. JAMA 2012;307(3): 294-303)
 Original RCT and observational studies published
from 1996 to 2011 (PubMed-Medline, EMBASE,
Cochrane)
 Adult patients undergoing hip arthroplasty with
appropriate VTE prophylaxis (updated guidelines
during the corresponding period)
 Proportions of symptomatic VTE occurring between
arthoplasty and hospital discharge using pooled
occurrence rate with random effects
 GRADE1 method for assessing quality of included
studies and for evaluating the evidence-basis of
our systematic review results
(1)
GRADE working group. www.gradeworkinggroup.org
Systematic Review – RESULTS
 27 studies
 21’369 adult patients undergoing hip arthroplasty
 38 prophylactic treatment subgroups
 58 to 70 years old on average
 Follow-up after surgery ranged from 8 to 17 days
 Pooled VTE
0.53% (95% CI, 0.35% to 0.70%)
Heterogeneity: I2=49.4% P<0.001
Pooled estimates by type of prophylaxis
%
(95% CI)
I²
P
LWMH (OS)
0.83
(0.19 – 1.48)
67.3%
0.230
LWMH (RCT)
0.51
(0.26 – 0.76)
45.4%
0.010
Direct Inhibitor, Factors
IIa/Xa (RCT)
0.31
(0.03 – 0.59)
32.8%
0.070
Indirect Inhibitor, Factors
IIa/Xa (RCT)
0.68
(0.26 – 0.97)
0.0%
0.380
0.53
(0.35 – 0.70)
49.4%
<0.001
TOTAL
PSI 12 estimates using Swiss / French Data
 Routine Data (ICD10) from overall Swiss /
French hospitals (3 consecutives years)
 Hierarchical Two-Level Logistic Regression
Model for estimating PSI #12 outcomes
o Level-1 = Inpatients
o Level-2 = Hospitals
Risk-adjustment Model

Fixed effect variables at each level of the
model:
o Inpatients
 Case mix (Sex, Age, Death, # of Secondary Diagnoses coded,
Selection of comorbidities from Charlson and Elixhauser
indices)
 Usual practice for detecting DVT (e.g., lower extremities
ultrasound before hospital discharge)
o Hospitals
 Average number of secondary diagnoses coded (adjusted on
sex, age and length of stay using negative binomial regression
models)
 Number of hip arthroplasty procedures
Adjusted PSI 12 using Two-Level Logistic
Regression Models
France
Switzerland
2008
2009
2010
2006
2007
2008
0.35%
0.34%
0.35%
1,36%
1,31%
1,38%
(0.31 – 0.40)
(0.30 – 0.38)
(0.32 – 0.39)
(1.17 – 1.55)
(1.15 – 1.47)
(1.22 – 1.55)
0.266 (0.215)
0.256 (0.217)
0.090 (0.172)
1.064 (0.104)
0.812
0.782
0.842
0.849
(0.766 – 0.857)
(0.729 – 0.835)
(0.801 – 0.883)
VTE (PSI #12)
Expected
Variance (SE)
C-statistic
1.052 (0.102) 1.026 (0.099)
0.850
0.857
(0.841 – 0.857) (0.842 – 0.858) (0.849 – 0.865 )
Factors associated to VTE
 Switzerland
o Some comorbidities are different across
years (lower frequency of some
comorbidities, thus not included in models)
 France
o The systematic use of lower extremity
ultrasound for screening DVT before
discharge in almost 20% of inpatients
Comparisons of Swiss and France (PSI 12)
against reference standard (meta-analysis)
 Stratified on Length of Stay
 Displayed on a same figure including:
o Adjusted proportions of VTE using the
PSI #12 (data from Switzerland and
France)
o Pooled proportion of VTE occurring
before hospital discharge, using metaanalysis (LMWH prophylaxis according to
ENDORSE Study)
3
Proportion of VTE (%)
Comparisons stratified
on length of stay
2
Switzerland
76.4%
France
74.1%
1
LMWH = 0.58% (0.35% - 0.81%)
0
< 8 days
[8 - 18[
≥ 18 days
Expected values (from Models) using Swiss and French Nationwide
Data, stratified by Length of Stay and by Year
International Perspectives
 Enlargement of the Swiss-French Pilot Study to
several other country (Canada, Germany,
Australia, New-Zealand, South-Korea, USA …)
 Expected issues
o Overall nationwide data or 10% (or more)
representative sample?
o Which countries do not have sufficient data quality
for assessing practice using procedure codes?
o Which other potential reasons for explaining
differences between countries
Conclusions
Facts …
 We are able to compare all outcomes that
could be measured for HSR…
 But we are only able to interpret measures
using the same definition / the same
metrology / the same data quality…
 Or, for which we are able to control potential
biases…
A Questionnaire for Conducting Studies to
Assess Quality of Healthcare
 Which variations across and between
providers (hospitals, health systems as
countries)?
 What are the sources of these variations
(Clinical definition, Algorithm codes and Coding
rules, Data quality, Case-mix, Quality of care,
Random effects)?
 How to control these potential biases (models,
risk-adjustment variables, interpretation)?
Back to Basic…
 Conditions for assessing healthcare quality
using patient safety indicators based on the
relation between a Process, an Outcome, and a
Structure
o Before hospital discharge (AEs related to
healthcare in hospitals / potentially no sufficient
consistency and accuracy for data after discharge)
o Outcome compared to “state of the art”
practices (guidelines, recommendations,
systematic review with meta-analysis)
o Results should be interpretable for decision
making
Approach for Modeling Comparisons
between Providers / Health Systems
 Take into account (adjustment, stratification)
o Difference in practice (use of prophylaxis to
prevent VTE, see ENDORSE Study)
o Difference in measurement (use of different
methods for assessing a diagnosis)
o Difference in coding diagnoses (Coding rule
for Major Diagnosis, # of Secondary Diagnoses
coded)
o Difference between Health Systems and
Hospitalization habits (average length of stay
between countries)
Giving Sense to our Findings
"We have to assess the quality of our
measures and make sure that they
make sense to clinicians, make sure
providers can act upon them and
that we account for variation in how
sick the patients are."
Prof. Patrick S. Romano
From CHPR Website
Jean-Marie.Januel@chuv.ch
IUMSP
Institut universitaire de médecine sociale et préventive, Lausanne
46
Inpatients Characteristics
VARIABLES
Switzerland *
France **
P
P
YEARS (Strate in analyses), N (%)
1= 2008* / 2008**
20 685
(32.48)
137 136
(32.81)
2= 2009* / 2007**
21 228
(33.33)
139 243
(33.32)
3= 2010* / 2006**
21 774
(34.19)
141 559
(33.87)
HOSPITALS (Level-2),
N / Average vol. of hip arthroplasty
procedures [min-max]
Year #1
137
217
[1 – 887]
728
219
[1 – 3 643]
Year #2
134
228
[1 – 811]
722
221
[1 – 3 651]
Year #3
128
238
[1 – 825]
699
239
[1 – 3 917]
INPATIENTS (Level-1)
Sex (Women), N (%)
35 305
(55.44)
0,334
252 687
(60.46)
<0.001
Age, Mean (SD)
70,06
(12.36)
0,558
72,00
(12.56)
<0.001
Lenght of Stay, Mean (SD)
11,85
(7.04)
<0,001
11,78
(7.25)
<0.001
Secondary Diag., Mean nb. (SD)
2,29
(2.71)
<0,001
2,33
(2.59)
<0.001
Death, N (%)
669
(1.05)
0,836
5 167
(1.24)
0.205
236
(0.37)
0,946
5 898
(1.41)
0.264
PSI #12,
VTE (both PE+DVT), N (%)
2008
2009
0+
50
0[
-5
0
0[
[4
50
-4
5
0[
[4
00
-4
0
0[
[3
50
-3
5
0[
[3
00
-3
0
0[
50
-2
5
0[
010
<5
[5
50
-5
0
50
-4
5
[4
[4
00
-4
0
50
-3
5
[3
00
-3
0
[3
[2
50
-2
5
00
-2
0
[2
50
-1
5
[1
0[
00
<5
010
[5
[1
Samplesize from hospitals
[2
0.0
0[
0.0
00
0.5
-2
0
0.5
[2
1.0
0[
1.0
50
1.5
[1
1.5
-1
5
2.0
00
2.0
[1
2.5
0
2.5
0+
3.0
0[
3.0
0[
3.5
0[
3.5
0[
4.0
0[
4.0
0[
4.5
0[
4.5
0[
5.0
0
5.0
Adjusted # of SDx (hospital level)
Adjusted Average # of SDx at hospital Level
using Negative Binomial Regression
Samplesize from hospitals
2010
2006
2007
2008