Key Performance Indicators,
Centre Reports, and more
Stephen McDonald

Barbecue talk
120
100
80
60
40
20
1970 1980 1990
Year
Rate
Incident rate
95% CI
2000
Incident RRT, Australia only
2010

More “good” news
20
15
10
5
0-24
Age-specific incident RRT rates
Australia
25-34
70
60
50
40
30
200
150
100
50
65-74 75-84
600 300
45-54
85+
500
0
1980 1990 2000 2010
400
200
200
100
0
1980 1990 2000 2010
Year
Rate 95% CI
0
1980 1990 2000 2010
Graphs by age group

Indigenous incidence rates
Aboriginal & TSI, Australia
500
400
300
200
100
0
1985 1990 1995
Year
2000
Rate 95% CI
2005 2010

Background
• A number of ongoing work themes exist within ANZDATA for generating output
– Stock and flow figures
– Annual Report
– Contributor requests
• Responses to information needed for various projects
– Research projects (internal and external analyses)
– Outcomes reporting

Outcomes reporting
• Recent years have seen a growth of interest in outcomes reporting
• Centre reports have been part of
ANZDATA for many years, with increasing emphasis in recent years
– At “parent hospital level”
– Limited distribution historically

4
Why measure outcomes?
2
1
.5
.2
0 20 40 units
60
O/E 98% CI
All Australia & NZ Dialysis Units, 98% confidence intervals
80

Dialysis outcome
4
2
1
.5
.2
0 20 40
Units, ranked by RR
60
RR 95% CI
Mortality rate during dialysis treatment in Australia 2006-10, adjusted for demographics and comorbidities
80

50
.5
.25
2
1
20
10
5
Variation in transplant outcomes
0 5 10
Units
15
RR 95% CI
Fully adjusted 1 year graft survival, by unit
All transplant units, Australia and New Zealand, patients transplanted
2005-2019, followup to 2010
20

What is happening to centre reports?
• Greater reporting of demographics and comorbidities
• Adjusted analyses in transplanting centre and dialysis reports
– Details of models supplied
• Graphs
– Funnel plots
– CUSUM plots (transplant)

Centre reports – graph 1
Survival from 90th Day of Treatment
1.00
0.75
0.50
0.25
0.00
0
CNAR
Australia
New Zealand
1 2
Years
3 4 5

Centre reports – graph 2
Technique Survival - PD at 90 days
1.00
0.75
0.50
0.25
0.00
0
CNAR
Australia
New Zealand
1 2
Years
3 4 5

100
80
60
40
20
0
Everywhere else
But....
CNARTS

1.3
1.2
1.1
1
.9
.8
.7
CNAR
Adjusted graphs
Adjusted SMR (95% CI)
Australia New Zealand

Adjusted graphs
2.5
2
1.5
1
.5
0
0
CNAR
50 100 150
Expected Number of Deaths
200

How are reports derived?

You need a model
• Logistic regression model (transplant),
Poisson model (dialysis)
• Adjusted for demographics, comorbidities
(donor and XM variables)
• With this model, derive a probability of
“expected” failure for each person / graft based on covariate matrix
• Compare this with actual outcomes www.anzdata.org.au

Which predictors are important?
0,7
0,6
0,5
0,4
0,3
0,2
0,1
Harrell's C
Somer's D
0
Recipient age gender & graft number
+comorbidities + HLA matching
+ ischaemic time
+ donor age + cause donor death
Predictive power of multivariate Cox model predicting graft survival, all DD transplants 2001-2009, with sequential addition of covariate groups

Don’t adjust for…
• Factors within the control of centre
– These may be why a particular centre gets good or bad results
• Factors that occur as a result of treatment decisions
• For example, don’t adjust for
– Choice of dialysis modality, HD access
– Use of immunosuppressives, rejection, 1 month graft function… www.anzdata.org.au

Other graphical demonstrations of output
• Funnel plots are a static measure and summarise performance (relative to a comparator) over a fixed period of time.
– Lack a dynamic element
– Weight recent and distant results equally

Adding time – CUSUM
2
0
-2
-4
4
Twoway CUSUM for a transplant centre
400
300
200
100
0
Tx date

4
3
2
1
0
5
Removing credit for good deeds
Oneway CUSUM for for a hospital
Tx number

Do we need to do more?

Why KPIs?
• Mortality is an insensitive and late indicators of problems
– Hopefully rare
– Outcome of complex series of events
• Incompletely ascertained
– Important to monitor as best we can
• Key Process indicators
– Simpler to understand, easier to address
– Need to be valid and correctable (and related to meaningful outcomes)

KPI Project
• Dialysis KPI project commenced 2011
– At instigation of DNT committee
• 2 markers chosen – Peritonitis and HD access at first treatment
– Deliberately limited to existing data collection
• NO additional data collected
– Based on real time ANZDATA data collection

Variation in HD access
1
.8
.6
8
5 5
31
40
65
18 28
10
5
7
17
25
22
113
36
68
8
10
15
7
47
19
12
29
52
75
434452
29
11
34344144
12
44
64
20
10
17
314433
1923
13
9
28
15
54
6 8
66
.4
.2
0
0 20
Centres
Proportion
ANZDATA, access at first HD where first dialysis
40
95% CI
60

27
1.5
1
.5
2.5
2
3
Variation in peritonitis rate
Peritonitis rates by treating unit
2009 only
4
6
12
24
Confidence intervals not shown where upper limit >3
Units with <5 person-years PD over 2009 not shown

KPI reporting -- access
• Quarterly identified feedback to units

Peritonitis reporting

Where to from here?
• COMMUNICATE
• Improve data collection
• Improve access to results
• Enhance reporting
– Add peritonitis rates
– Access subdivided by late referral
– Graphs etc etc
• Or is it all just too hard?

How do we view quality?

Centre reports -- SMR