A ‘PROM’ising Future
The Relationship Between Patient
Reported and Other Process
Outcomes at Trust Level
Esther Kwong
Academic F2
Dept Primary Care and Public Health
Project Supervisor: Dr Paul Aylin
Educational Supervisor: Dr Graham Easton
Dept Primary Care and Public Health
Contents
The Context –Setting the Scene for PROMs NHS
What are Outcomes
Measuring Patient Reported Outcomes
National PROMs Program Overview
Research Question
Methods
Results
Discussion & Conclusions
What have I learnt…
The Past
“Despite a century of developments in
medical technology, and vast
improvements in the ability of medical
science to prevent, diagnose and treat
disease and ill health, attempts to measure
the outputs of health care in terms of their
impact on patients’ health have not
progressed beyond Florence Nightingale’s
time.”
Getting the most out of PROMs Kings Fund 2010
The Context
Reasons for Healthcare:
Live Longer
Better Quality of Life
= Better Health Outcome
But health services
traditionally focused on
one outcome
Mortality
Setting the Scene for PROMS
Darzi Review “High Quality for All”
NHS White Paper
Equity and Excellence: Liberating the NHS
Appraisal of new technologies –
PRO data incorporated in the evaluation of new technologies
Routine measurement of pre/post elective surgery PROMs
Since April 2009
DH Long Term Conditions PROMs Pilot
Since 2010
Patient Centred
If quality is to be at the heart of
everything we do, it must be
understood from the
perspective of patients.
Patients pay regard both to
clinical outcomes and their
experience of the service...
Lord Darzi
The ultimate measure by
which to judge the quality
of medical effort is
whether it helps patients,
as they see it.
Donald Berwick
What Are Outcomes
Traditional Ways of Planning = Measuring in terms of OUTPUT
• Quantifying what is produced, implemented, provided, and developed
in the health service
Increasing Focus = Measuring in terms of OUTCOME
• Quantifying extent of any health impact on patients
• Change in various dimensions ~physiological (e.g. functional status)
or psychological (e.g. attitudes)
• Can be harnessed from different sources
Sources of Outcomes
Measuring Patient Reported Outcomes
Patient Reported Outcomes (PRO)
• Health status as perceived by the patient
Patient Reported Outcomes Measures (PROMs)
• Measurement tools to harness this information
• Can be used in two points in time to record change in health status
• Can be assessed against patient progress or health interventions
received
• Various types available
• Much dedicated research and analysis on validating questionnaire
types
Types of PROMS
EQ5D
Oxford Hip
Score
For any
condition
Generic
Different
Disease
states
Aggregation
and
comparison
Economic
evaluations
Condition
Specific
Outperform on
sensitivity
Centred on a
particular
aspect/ clinical
detail
Focused –
useful for
informing
National PROMs Program Overview
Since 1 April 2009
Providers required to collect and report PROMs
Four key NHS funded elective interventions
• Unilateral hip replacements
• Unilateral knee replacements
• Groin hernia surgery
• Varicose vein surgery
Expected to invite patients to complete a pre-operative PROMs
questionnaire (Q1)
Post-operative questionnaires (Q2) are then sent to patients following their
operation after a specified time period.
theNHS
NHSwill
willbe
bethe
thefirst
firsthealth
healthcare
caresystem
……the
in
system
world to
measure
what itin terms
the
worldintothe
measure
what
it produces
in terms
ofinhealth,
in
ofproduces
health, rather
than
termsrather
of the than
production
of care.
the production of health care.
ofterms
health
Getting the most out of PROMs Kings Fund
Getting the most out of PROMs Kings Fund
PROMs Used for the National Program
EQ5D index
score
• Multi-dimensional – five areas
• Responses record three levels of severity
• Scores are weighted and combined to give a single
index
EQ5D Visual
Analogue
Scale
• Self rating health related quality of life
• Places self reported health state on a point in a line
• Line ranges from 0 to 100
Oxford Hip
Score
• Validated tool specific for Total Hip Replacements
• 12 questions to assess function and pain, 0-4 points
• Given as a single summed score from 0 to 48
Oxford Knee
Score
• Validated tool specific for Total Knee Replacements
• 12 questions to assess function and pain, 0-4 points
• Given as a single summed score from 0 to 48
Research Topic
Aim:
To explore the relationship between routinely collected patient
reported and other process outcomes at trust level
Null Hypothesis:
There is no relationship between patient reported and other process
outcomes at trust level
Methods:
Aggregate analysis conducted using STATA 11 on trust level data
Participation and Coverage 2010
Participation rate of 69.7%.
•245,488 eligible hospital episodes
•171,080 pre-operative questionnaires returned
Return rate of 75.8%
•147, 974 post-operative questionnaires sent out
• 112,163 returned
National PROMs Key Final Results 09-10 Overview
EQ-5D Index score
Oxford Hip and Knee Score
87.2% of hip replacement respondents
77.6% of knee replacement respondents
95.7% of hip replacement respondents
91.4% of knee replacement respondents
Recorded an increase in general
health following operation
Recorded an improvement following
operation
Data Sources
Dr Foster Data
Hospital Episode Statistics (HES)
PROMs data 2010 August 2011
Publication Used
Orthopaedic Revision Rates
Orthopaedic Readmission
Caseload
Staff to bed ratio
Aggregate Trust
HSMR
Level Data
2010
National Joint Registry Data
Orthopaedic Procedures
Caseload
HES Inpatient Data
Elective Surgery Waiting Times
Emergency Admission Caseload
Data Comparison
Other Process Outcomes/ Hospital
Indicators Compared
PROMs Outcomes 2010 Data- Hip
and Knee Data
Case Adjusted Health Gain
(Q2-Q1)
•
•
•
EQ5D Index
EQ5D Visual Analogue Scale
Oxford Hip/ Knee Score
1. Hospital Standardised Mortality
Ratios
2. Dr Foster Orthopaedic Revision
Relative Rate
3. Dr Foster Orthopaedic Readmission
Caseload
4. National Joint Registry Data
Orthopaedic Procedure Caseload
5. Hospital Episode Statistics Elective
Surgery Waiting times
6. Hospital Episode Statistics
Emergency Admissions Caseload
7. Hospital Staff to Bed Ratios
Descriptive Results Hip
EQ5D Index
22,270
127
21
0.395
0.0364
EQ5D VAS
21,653
128
20
7.53
2.23
3.362
Oxford Hip
Score
24,682
131
17
19.3
1.28
20.1
Missing
Mean
Trusts
Interquartile
Range
0.4215
14
16
18
20
Oxford Hip Score Case Adjusted health gain
22
.25
.3
.35
.4
Case Adjusted EQ5d HG
.45
.1
0
0
0
.05
.1
5
Density
Density
10
.15
.3
.2
Standard
Deviation
.2
15
.4
Outcome
Case
Numbers
Number of
Trusts
(Observations)
0
5
10
EQ5D VAS Case Adjusted health gain
15
HSMR
.45
.4
.35
.3
.25
0
500
1000
NJR no. Hip operation procedures
0.0847
-0.0275
0.0777
-0.0225
-0.0143
-0.1431
10
-0.0919
-0.1263
5
0.0308
0.0606
-0.137
0
-0.05
Fitted values
15
Case Adjusted EQ5d HG
1500
0
500
1000
NJR no. Hip operation procedures
0.1939
0.1567
0.1924
0.1881
-0.0295
-0.0645
-0.0402
-0.0801
-0.0437
0.0573
0.0451
0.0334
0.1258
-0.066
18
20
0.2042
Fitted values
22
EQ5D VAS Case Adjusted health gain
1500
16
Nurse to Bed
Ratio
Staff to bed
Ratio
OHS
14
Orthopaedic
Procedures
Caseload
EQ5D
VAS
OHS Case Adjusted Health Gain
Revision
Relative Risk
Readmission
Caseload
Surgery
Wait Times
(days)
Emergencies
Caseload
Hip Procedures
Caseload
EQ5D
Index
EQ5D VAS Case Adjusted Health Gain
Indicator
EQ5D Index Case Adjusted Health Gain
Correlations Hip
0
500
1000
NJR no. Hip operation procedures
Oxford Hip Score Case Adjusted health gain
1500
Fitted values
b Coefficient
R2
Value
Confidence
Intervals
0.0398
0.0000276
0.0334
1.31 x 10-5
539 x 10-5
0.0612
0.0015296
0.0275
-7.27 x 10-5
313 x 10-5
Regression Model
Hip
F Probability
EQ5D index
Hip Operation
Caseload
EQ5D VAS
Hip Operation
Caseload
Regression Hip
OHS
Hip Operation
Caseload
0.0460
0.0009363
EQ5D index
Orthopaedic
Operation Caseload
0.0453
0 .0000138
0.0317
0.0291 x 10-5
2.72 x 10-5
EQ5D VAS
Orthopaedic
Operation Caseload
0.0640
0.000775
0.027
4.58 x 10-5
159 x 10-5
OHS Orthopaedic
Operation Caseload
0.0742
0.000425
0.0245
4.33 x 10-5
90.2 x 10-5
EQ5D VAS
Waiting Time
0.0836
0.0378
0.0267
-510 x 10-5
8030 x 10-5
OHS
Waiting Time
0.0734
0.0223
0.0281
-210 x 10-5
4670 x 10-5
0.0305
1.69 X 10-5
186 x 10-5
Description Results Knee
Outcome
Case
Numbers
Number of
Trusts
(Observations)
23,318
180
219
0.299
0.0369
0.45
EQ5D VAS
22,591
177
222
1.836
2.105
3.035
Oxford
Knee Score
25,413
189
210
14.81
1.43
1.653
Interquartile
Range
.2
Density
.1
0
0
0
.05
.1
5
Density
.15
10
.2
.3
EQ5D Index
Missing Mean Standard
Trusts
Deviation
.2
.25
.3
.35
EQ5D case adjusted HG
.4
.45
-5
0
5
EQ5D VAS Case Adjusted HG
10
10
12
14
16
18
Oxford Knee Score case adjusted HG
20
EQ5D
VAS
16
14
12
EQ5D
Index
18
20
Indicator
OKS
10
OKS Case Adjusted Health Gain
Correlations Knee
50
Revision
Relative Risk
Readmission
Caseload
Surgery
Wait Times
(days)
Emergencies
Caseload
Hip Procedures
Caseload
0.2176
0.1014
Orthopaedic
Procedures
Caseload
0.2215
0.0389
Nurse to Bed
Ratio
-0.2407
-0.1899
Staff to bed Ratio
-0.3242
-0.4102
-0.0795
0.1524
0.0124
-0.0408
0.0388
-0.0447
-0.1586
-0.1459
Oxford Knee Score case adjusted HG
200
250
Fitted values
5
-0.0283
0
0.1257
-5
EQ5D VAS Case Adjusted Health Gain
0.2188
9
0.181
9.5
10
10.5
11
HES Emergency Admissions Caseload
11.5
Fitted values
.4
.25
-0.3408
.3
.35
0.1693
-0.185
.2
EQ5D Index Case Adjusted Health Gain
.45
EQ5D VAS Case Adjusted HG
9
HSMR
150
Foster RR Knee Revision
10
-0.09
100
0.102
0.104
0.0625
9.5
10
10.5
11
HES Emergency Admissions Caseload
EQ5D case adjusted HG
Fitted values
11.5
Regression Knee
Regression model F Probability
b Coefficient
R2
Value
Confidence
Intervals
EQ5D index
Emergency Admissions
Caseload
0.0003
-0.0158
0.135
-0.024
-0.007
EQ5D VAS
Emergency Admissions
Caseload
0.0341
-0.488
0.0372
-0.94
-0.037
OKS
Emergency Admissions
Caseload
0.0331
-0.350
0.0382
-0.77
-0.033
Discussion – Important Findings 1
Weak positive correlations between Hip and Orthopaedic
Procedures Caseload and all Hip PROMs health gain
• Suggests the more procedures a trust does the better its quality
of hip replacement procedure perceived by patient
• This is an expected correlation direction
Weak negative correlations between Emergency Admission
Caseload and all Knee PROMs health gain
• Suggests the more emergency admissions a trust has the worse
the patient perceived outcome for a knee replacement procedure
• unexpected correlation direction, warrants further exploration into
relational factors – such as trust specialisation and quality
relationship
Discussion -Important Findings 2
Weak Positive Correlation Between EQ5D Visual Analogue
Scale and Oxford Hip Score health gain for hip patients and
Waiting Times for Elective Surgery
• Suggests the longer a patient waits for elective surgery in a trust
the more health gain perceived from hip operation
• Unexpected correlation direction
• Disease progression factors are adjusted for
• May be explained by expectation management
‘Patient Satisfaction = Patient Experience - Patient Expectation’
• Longer waiting times may decrease expectation affecting
perceived outcome
• Lead time difference
Discussions Limitations
Recruitment Bias
• LSHTM Report to Dept of Health on PROMs recorded correlation
coefficient of -0.38 between EQ5D score and every 20% increased
recruitment, suggesting low recruitment rates can introduce bias
• The report recommended a target recruitment rate of 80%
Response Bias
• Studies suggest non responders were younger in all PROMs, This is
particularly evidenced in orthopaedic PROMs
Patient Reported Outcome Measures (PROMs) in Elective Surgery Report to the Department of Health, London School of Hygiene and
Tropical Medicine
Conclusions
Weak/ Lack of Correlations suggests Patient Reported Outcomes are
capturing an added dimension of quality that traditional process outcome
and clinical indicators were not measuring
Weak correlations findings at trust level maybe due to aggregation, this
could eliminate clinical variation within and between hospital services as
well as patient characteristics
• Evidence from clinical governance concluded acute hospitals services were
‘A mix of good and bad’
• Analysis of PROMs at clinical level and unadjusted data may provide further
explanations and strengthen correlations
Lack of evidence/ data available for statistical relationship significance for
correlations
• Further work building larger aggregate data set on PROMs
• Analysis on new PROMs data, or analysis spanning two years of PROMs data
What I learnt from this Academic Rotation
Nature of Rotation
2 days a week in GP surgery clinical duties
3 days a week dept based research and
teaching activities
Research
Literature search on PROMs
Insight in health services research
Handling aggregate data
Data management
Statistical analysis on STATA
Seminars and Journal Club
Experience of life as an academic!
Teaching
Formal Teaching courses/
training
Clinical Methods teaching 3rd
year Imperial Students
Problem Base Learning
facilitator
GP
Wealth of clinical experiences
Primary care setting exposure
Consultation simulation training
References
Bevan G , Skeller M, 2011 Competition between hospital and clinical quality BMJ 2011; 342:d3589
Berwick D, Hiatt H, Janeway P, Smith R. 1997 An ethical code for everybody in health care BMJ 1997;315:1633
Black N, Browne J, Cairns J. 2006. Health care productivity. British Medical Journal 333: 312–313.
Brooks R, Rabin R, de Charro F. 2003. The Measurement and Valuation of Health Status using EQ-5D: A European Perspective. Kluwer: Dordrecht.
Browne J, Jamieson L, Lawsey, J, van der Meulen J, Black N, Cairns J, Lamping D, Smith, S, Copley L,Horrockes, J. 2007. Patient Reported Outcome
Measures (PROMs) in Elective Surgery. Report to theDepartment of Health. Available from: www.lshtm.ac.uk/hsru/research/PROMs-Report-12-Dec-07.pdf.
Burge P, Devlin N, Appleby J, Gallo F, Nason E, Ling T. 2006. Understanding patients’ choices at the point of referral. Technical report TR359-DOH,
Cambridge: RAND Europe. Available from: www.rand.org/pubs/technical_reports/TR359/.
Darzi L. 2008. High Quality Care for All. NHS Next Stage Review: Final Report, Department of Health, London.
Available from: www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/
DH_085825.
Dawson J, Rogers K, Doll H, Using Patient Reported Outcomes Routinely: An example in context of Shoulder surgery Open Epidemiology Journal 2010,
3,42-52
Department of Health. 2008. Guidance of the Routine Collection of Patient Reported Outcome Measures (PROMs).Department of Health document
DH_081179[1].pdf.
Devlin N, Appleby J. 2010. Getting the Most Out of PROMs: Putting Health Outcomes at the Heart of NHS Decision Making. Kings Fund/Office of Health
Economics: London.
Dolan P. 1997. Modelling valuations for EuroQol health states. Medical Care 35(11): 1095–1108.2010).
Greenhalgh J, Long A, Flynn Rob, 2004 The use of patient reported outcome measures in routine clinical practice: Lack of theory or lack or impact. Social
Science and Medicine 60 (2005) 833-843
EuroQol Foundation. Springer: Rotterdam.
Hospital Episode Statistics: Finalised Patient Reported Outcome Measures (PROMs) in England: April 2009 – March 2010
Isis Outcomes Patient Reported Outcome Measures from the University of Oxford, Orthopaedic Pros http://www.isis-innovation.com/outcomes/orthopaedic/
London School of Hygiene and Tropical Medicine Patient Reported Outcomes on Elective Surgery, Report To Department of Health Dec 2007,
http://www.lshtm.ac.uk/php/hsrp/research/proms_report_12_dec_07.pdf
NHS North West. 2010. Advancing quality. Available from: www.advancingqualitynw.nhs.uk
National Council on Ageing and Older People, 1998 health promotion strategy for older people in Ireland. Adding years to life and life to years
National Network of Libraries of Medicine Guide 3: Define Measurable Goals, Outputs and Outcomes http://nnlm.gov/outreach/community/goals.html
Office of Health Economics. 2008. NHS Outcomes, Performance and Productivity. Report of the Office of Health Economics Commission. OHE: London.
Szende A, Oppe M, Devlin N. 2007. EQ-5D Valuation Sets: An Inventory, Comparative Review and Users’ Guide.
Acknowledgements
I want to thank all those in the department who have contributed their
expertise and advice towards this project and towards my
educational development
•
•
•
•
Dr Paul Aylin, Dr Graham Easton, Dr Jenny Lebus,
Dr Michael Soljak, Dr Sonia Saxena
Dr Fiona Hamilton, Dr Matthew Harris, Dr Eszter Vamos
Elizabeth Cecil, Farzan Rahman, Dr Ghasem Yadegarfar