Approaches to Evaluating and Measuring
Outcomes in Integrated Care
Key Issues for Consideration
Dr Nick Goodwin
CEO, International Foundation for Integrated Care
Senior Associate, The King’s Fund
Paper to: Health Quality and Safety Commission New Zealand
Workshop: Towards Integrated Care in New Zealand
Wellington, New Zealand, November 14th 2013
What is a ‘programme evaluation’?
• A systematic method of collecting, analysing and using
information to answer questions about projects, policies or
programmes
• In health care systems, they are particularly concerned with
quality of care, patient safety, system efficiency and/or cost
effectiveness
• They also seek to examine whether programme goals are, of
have been, appropriate and/or useful so can be used to
change and adapt strategic directions
• They tend to utilise both qualitative as well as quantitative
methods
• They can be ‘formative’ or ‘summative’
Typical components of a ‘programme evaluation’
• Assessment of the need for the programme
– Needs assessment/gap analysis/population health planning
• Assessment of the design and/or theory and logic of the
programme in supporting its desired influence
– Are the assumptions of the programme justified?
• Assessment of how the programme is being implemented
– Process evaluation - is it going to plan?
• Assessment of the programmes outcome or impact; cost and
efficiency
– Outcome level, outcome change, programme’s effect
Rossi, Lipsey and Freeman (2004) Evaluation: a systems approach, Sage
Understanding what to evaluate
in an integrated care programme
What are you evaluating – some key questions
• Who and what is the programme seeking to influence?
Need to clarify aim and design of the integrated care
intervention by looking at the needs of patients/users
• What is the timeframe over which outcomes are expected
to be achieved?
Given this timeframe, which categories of outcomes have
the potential to be improved?
• Is there sufficient opportunity in a given population to
achieve this targeted improvement in outcomes?
• How can you measure the impact? How can you ensure
attribution?
What are you evaluating – some key questions
Before developing questions and/or survey instruments to
examine the experience and impact of integrated care from a
person’s perspective, there is a need to understand four
things:
 the programme theory of change – what are the
assumptions that lie behind the programme (why?)
 the (set of) problems to be addressed (where and who?)
 the (set of) interventions best suited to address the
problem (what and who?)
 the strategy best suited to develop, implement, and
evaluate the (set of) interventions (how, when and who?)
What are you evaluating – some key questions
For integrated care to be successful, it needs to execute the
following three functions:
 accurate identification of individuals within target
population (e.g. reliable predictive modelling, health risk
assessment, medication list and/or laboratory values from
EMRs);
 individuals must be enrolled and actively participate in the
program for a meaningful period of time (e.g. readiness to
change, motivational interviewing, incentives);
 the program must include a set of interventions that modify
or close deficits in participant and provider behaviour (e.g.
tailoring to needs).
Key Points to Consider
 Baseline data
 Define a comparison group
 Define nature and structure of integrated care being
implemented
 Include measures of the professionals’ perspective where care
is delivered through multidisciplinary teams
 Identify what good looks like from a patients’ perspective and
evaluate this through user feedback
 Include analysis of utilisation and costs of care
Experiences, care outcomes, utilisation & costs
Understanding what to measure
in an integrated care programme
Measurement Types - 1
Care Outcome Measures
 Patient outcomes
– e.g. mortality, morbidity, functional status, quality of life
 Cost and utilisation outcomes
– e.g. hospital admissions, bed days, LOS, nursing home placements
Care Process Measures
 Occurrence of recommended care activities
– e.g. presence of a care plan; patient follow-up – often processes that
are set out in best practice guidance
The measurement tools we have on outcomes and processes are mostly
disease-specific - for people with multiple needs, the process of care is
less well understood
Examples of outcome measures
 Hospital utilisation
– Emergency admissions; hospital readmissions; lengths of stay; number of bed days etc …
– Disease-specific hospital admissions etc …
 ‘Social care’ utilisation
– Levels of home care support packages
– Rates of long-term nursing home/residential home stays
 Mortality and disease-specific mortality
 Short-term clinical outcomes
– e.g., glycated haemoglobin levels for diabetic patients
 Functional status
– e.g., for CHF patients
 Quality of life
– e.g., functional dependence
 Other patient outcomes
– e.g., missed school days for children due to illness; experiences with system
 Treatment and service adherence
– e.g., remaining in contact with services for mentally ill patients
Measurement Types - 2
Measures of care co-ordination
 Information exchanges and transfers
 Relational co-ordination between organisations and professionals
– levels of awareness/interaction among participants
– is there a common understanding of care activities and goals
– shared culture and mission?
Co-ordination Mechanism Measures Approaches




Direct observation
Interviews and staff surveys (self-report)
Medical record audits on information transfer
Measures of inter-professional collaboration within teams and
organisations
Examples:
ITMA - Integrated Team Monitoring and Assessment Tool – see
http://www.readiness-tools.com/tool-full.aspx?toolguid=0d6382ad-f0174623-8d10-93f2f314e346
POET – Partnership Outcomes Evaluation Toolkit – see
www.dhcarenetworks.org.uk/asset.cfm?aid=1479
Care Co-ordination Measures Atlas
McDonald KM, Schultz E, Albin L, Pineda N, Lonhart J, Sundaram V, Smith-Spangler C, Brustrom J, and Malcolm E. Care Coordination Atlas Version
3 AHRQ Publication No.11-0023-EF. Rockville, MD: Agency for Healthcare Research and Quality. November 2010.
http://www.ahrq.gov/qual/careatlas/careatlas.pdf and http://www.ahrq.gov/qual/careatlas/careap4.pdf - 64 different survey tools
Domains for measuring care co-ordination
Co-ordination activity:
Service delivery approaches:














Establish accountability/negotiate
responsibility
Communication – informational and
inter-personal
Facilitate transitions – e.g. across
settings or as coordination needs
change
Assess multiple needs and goals
Pro-active care planning
Monitor, follow-up, review
Support self-management
Link or refer to community resources
Align resources to meet individual or
community needs
Care management
Medicines management
Healthcare at Home
Multi-disciplinary teams
ICT-enabled integrated care (e.g.
telehealth)
Perspectives:
Family/patient
Professional
System/organisation
Measurement Types - 3
Patient, carer, family-reported perceptions
 PROMs
•
•
do you feel better?
have your symptoms improved?
 PREMs
•
•
recommend to a friend?
how satisfied are you?
 * PACIC patient assessment of chronic illness care
http://www.improvingchroniccare.org/downloads/2004pacic.doc.pdf
 * Care Transition Measures (CTM) – patient assessment of discharge from
hospital http://www.caretransitions.org/documents/CTM-15.pdf
The more ‘robust’ measures are those that ask about specifics of care co-ordination
rather than generalities, and which link data sets together to understand
association between responses. Two key things: measures that are specific to a
particular group of people; measures that translate to actionable service
improvements
Patient Reported Outcome Measures in England
Since 2009, PROMs have been collected for
four elective procedures: hip surgery,
knee surgery, hernia repair and varicose
veins – these costs the English NHS
£800m/year
Cataract surgery was to be included, but
concerns on methods of data collection
250,000 patients a year invited to complete
survey – all NHS providers (100%) collect
data
PROMS being extended to cover: anxiety and
depression, cancer care, asthma, COPD,
diabetes, epilepsy, heart failure, stroke
It’s a careful process – identify the right
PROMs instrument; pilot before roll-out;
implement data collection; evaluate
programme
Uses EQ-5D – part 1 about mobility, selfcare, usual activities, pain/discomfort,
anxiety/depression; part 2 gives overall
assessment of health on 0-100 scale
NOTE: Not sensitive to looking at people
with multiple conditions/needs
Some Conclusions on Measuring Integration
 Many different tools available:




Need to define the client group
Need to understand the goal in terms of outcomes to patients and service users
Need to create ‘measurable’ outcomes and experiences
Measures need to mean something – i.e. that actions can follow
 Patients and users tend to understand the term ‘care co-ordination’ or
‘continuity of care’ – e.g. to what extent they feel that care is co-ordinated
around their needs
 Baseline on measures required on which to base progress over time
 Link measures to other data – e.g. on clinical outcomes, utilisation, costs
 Where possible, benchmark performance with others or investigate with a
matched ‘control’
 Use data in ‘real time’ to monitor progress and drive performance
Case Example
Integrated Care Metrics
NHS London (2012)
Activity metrics currently used in IC systems in London, across all settings of care
Metric
Definition
Emergency admissions
(No.)
Number of unplanned admissions for ACS conditions that should not usually
require hospital admission
Number of emergency admissions for the IC cohort (over 65/75/ top ICD 10
codes/ community ward patients)
Case file audit results for reviews of avoidable ED admissions
Emergency bed days (No.)
Number / total number of emergency hospital bed days for IC cohort (over
65/75/ top ICD 10 codes/ community ward patients)
Emergency bed days associated with multiple acute hospital admissions
SUS/ HA data
ONEL, NWL,
ELC
A and E attendances (No.)
Number / Total number of A and E attendances for the IC cohort (over 65/75/
top ICD 10 codes/ community ward patients)
SUS data
NWL, ONEL,
NCL, KHP, ELC
Acute re-admissions (No/
%)
All emergency re-admissions that occur within 30 days of discharge for IC
cohort (over 65/75/ top ICD 10 codes/ community ward patients)
SUS / HA data
NWL, ONEL,
KHP
Length of stay (No.)
Data source
SUS / HA data
Difference between the discharge time and the admittance time for both
Dr Foster/ SUS / HA
elective and non elective episodes divided by the total number of spells for IC
data
cohort
Average Length of Stay for patients in the IC cohort (over 65/75/ top ICD 10 codes/
community ward patients)
ICS
NWL, ONEL,
NCL,
ONEL,ELC,
KHP
NWL, ONEL,
NCL, ELC
Permanent admissions to
residential and nursing care
homes (No./ %)
Number of patients in long term care homes as a proportion of the total
number of patients in the pilot (%)
Number of new permanent admissions to residential and nursing care homes,
65+/ -65
Case file audit results for reviews of recent care home admissions
Referral Assessment
Provision (RAP) stats
KHP, NWL,
Greenwich,
NCL, ONEL
Number of people
completing re-ablement
(No.)
Proportion of service users independent following re-ablement
Readmission within 1 year for patients who have completed re-ablement
Referral Assessment
Provision (RAP) stats
Greenwich
RiO
NWL
Community nursing hours
per person (No.)
Total number of community nursing visits or units delivered divided by
number of patients in pilot
People supported at home
with low/med/high care
packages (No.)
Number of people 65+supported at home with low/med/high care packages
Referral Assessment
Provision (RAP) stats
KHP,
Greenwich
Urgent GP response
DH survey (Q14-17 seen on same day within primary care 'how convenient'
score)
QOF
KHP
Process metrics currently used by IC systems in London
Metric
Definitions
Source
ICS
Number of health assessments
completed (No.)
Number of assessments complete versus target
Outcomes of health assessments
Emergency admissions of patients who had health assessments
Local ICS collection
KHP
Number of people case managed
(No.)
Numbers of case managed patients
Caseload, analysed by length of stay, source of referral and
state of care plan
% currently case managed patients with no ED contact
‘Referral bounce’ for case managed patients
Local ICS collection
KHP
Number of care plans completed
(No.)
Numbers of patients with personalised care plans
Number of people in cohort on a care plan as a proportion of
total number of people that should be on a care plan
Local ICS collection
ONEL , NWL
Adherence to care plan (No.)
Number of patients with one or more delay or incomplete
actions as a proportion of the total number of patients in the
pilot (%)
Local ICS collection
NWL
MDT operation (No.)
Numbers of case managed patients discussed at MDT for
whom follow up actions took place
Local ICS collection
KHP
Attendance at MDT meetings
(No.)
Number of meetings scheduled / held / fully attended
Number of clinicians attending monthly MDT case conferences
and quarterly review meetings as proportion of total number
expected at these meetings (%)
Local audit
KHP
Frequency of MDT meetings
(No.)
Virtual Ward: number of monthly MDT meetings held
Local ICS collection
ELC
Degree of joint working
Attendance rates at joint meetings
Exception reporting –
local ICS collection
ONEL
Outcome metrics currently used by IC systems in London
Metric
Control measures
Hard outcomes
Definition
Source
Patients with T1/T2 DM with HbA1c less than or equal to 10
QOF
% of patients with blood pressure targets achieved as per NICE
guidance
Percentage of patients with cholesterol less than or equal to 5
BMI less than or equal to 30
proportion of patients receiving medication review who should
Speed of referral for recognised foot complications,
QOF
Amputation rate below the ankle,
Falls rate among the frail elderly,
Number of fractures
ICS
NWL
NWL
Patient and staff experience measures used by IC systems in London
Metric
Definition
Source
ICS
PROMS
Measure of quality from the patient perspective covering four
clinical procedures: hip replacements, knee replacements,
hernia, varicose veins
NHS Information Centre
NWL, ONEL
PREMS
Measure of patients experience based on range of different
possible metrics: patient recommendation, overall
satisfaction, complaints, patient confidence
Patient surveys and
questionnaires, focus groups,
complaints data, one on one
patient interviews
NWL , ONEL
Patient
satisfaction
surveys
Results of ICS service user survey
Percentage of people with LTCs who feel supported to
manage their condition
The proportion of people who use services who feel safe
DH GP survey 032
KHP, ONEL
LTC6 survey
1. Did you discuss what was most important for you in managing
your own health?
2. Were you involved as much as you wanted to be in decisions
about your care or treatment?
3. How would you describe the amount of information you
received to help you to manage your health?
4. Have you had enough support from your health and social care
team to help you to manage your health?
5. Do you think the support and care you receive is joined up and
working for you?
Local health observatory
6. How confident are you that you can manage your own health?
Staff satisfaction
Staff survey to determine satisfaction of IC pilot, to
determine number of staff responding very satisfied and
satisfied to survey as proportion of total number of staff
surveyed
Local ICS collection
NWL
Contact
Dr Nick Goodwin
CEO, International Foundation for Integrated Care
nickgoodwin@integratedcarefoundation.org
www.integratedcarefoundation.org
@goodwin_nick
@IFICinfo
Appendix 1: From a measure to an indicator that can be used
for quality improvement
In developing ‘indicators’ to judge comparative performance, the following
criteria are important:
 Statistical validity




Accurate – measures what is says it measures!
Reliable – can be tracked over time
Consistent – data collection robust and reproducable
Avoids bias
 Data considerations






Data source – sample or full population; existing or new data source?
Unit of assessment – country, region, locality, practice etc
Client groups covered – what is the targeted population? account for bias/case mix?
Significance – smaller groups, reduced statistical power
Patient reported data is indirect – how cross-compare to other data (multi-methods)
Coverage – health, health & social care, health, social care and housing ….
 Face validity



Meaningful to public; clinically credible;
Potential to support quality improvement;
Cost and value for money
Appendix 2: Key Resources on Care Co-ordination Measures
McDonald KM, Schultz E, Albin L, Pineda N, Lonhart J, Sundaram V, Smith-Spangler C, Brustrom J,
and Malcolm E. Care Coordination Atlas Version 3 AHRQ Publication No.11-0023-EF. Rockville,
MD: Agency for Healthcare Research and Quality. November 2010.
http://www.ahrq.gov/qual/careatlas/careatlas.pdf
ttp://www.ahrq.gov/qual/careatlas/careap4.pdf - 64 different survey tools
King et al - 1995/2004 - Measures in the process of care – MPOC
http://www.canchild.ca/en/measures/mpoc56_mpoc20.asp
Flocke SA. Measuring attributes of primary care: development of a new instrument. J Fam Pract.
1997 Jul;45(1):64-74.
Safran DG, Kosinski M, Tarlov AR, et al. The Primary Care Assessment Survey: tests of data quality
and measurement performance. Med Care. 1998 May;36(5):728-39.
Cassady CE, Starfield B, Hurtado MP, Berk RA, Nanda JP, Friedenberg LA. Measuring consumer
experiences with primary care. Pediatrics. 2000 Apr;105(4 Pt 2):998-1003.
Starfield B, Cassady C, Nanda J, Forrest CB, Berk R. Consumer experiences and provider
perceptions of the quality of primary care: implications for managed care. J Fam Pract.
1998;46(3):216-26.
Grimmer K, Moss J. The development, validity and application of a new instrument to assess the
quality of discharge planning activities from the community perspective. Int J Qual Health
Care. 2001 Apr;13(2):109-16
McGuiness C, Sibthorpe B. Development and initial validation of a measure of coordination of
health care. Int J Qual Health Care. 2003 Aug;15(4):309-18.
Radwin L, Alster K, Rubin KM. Development and testing of the Oncology Patients' Perceptions of
the Quality of Nursing Care Scale. Oncol Nurs Forum. 2003 Mar- Apr;30(2):283-90.
Billings J, Coxon K, Alaszewski A. Empirical research methodology for ‘Procare’ version 3.
University of Kent at Canterbury: Centre for Health Services Studies; 2003. Available from:
http://www.kent.ac.uk/chss/docs/procare_version3.pdf.
Morita T, Hirai K, Sakaguchi Y, Maeyama E, Tsuneto S, Shima Y. Measuring the quality of structure
and process in end-of-life care from the bereaved family perspective. J Pain Symptom
Manage. 2004 Jun;27(6):492-501.
Coleman EA, Mahoney E, Parry C. Assessing the quality of preparation for posthospital care from
the patient's perspective: the care transitions measure. Med Care. 2005 Mar;43(3):246-55.
(also CTM website)
Glasgow RE, Wagner EH, Schaefer J, Mahoney LD, Reid RJ, Greene SM. Development and
validation of the Patient Assessment of Chronic Illness Care (PACIC).Med Care. 2005
May;43(5):436-44
Drewes et al (2012) PCIC+ at http://www.ijic.org/index.php/ijic/article/view/862/1896
Haggerty J, Roberge D, Freeman G, Beaulieu C, Breton M. Review. When patients encounter
several providers: validation of a generic measure of continuity of care. Annals of Family
Medicine (In press) and Breton et al at
http://www.ijic.org/index.php/ijic/article/view/682/1534
Hibbard JH, Stockard J, Mahoney ER, Tusler M. Development of the Patient Activation Measure
(PAM): Conceptualizing and Measuring Activation in patients and consumers. Health Services
Research 2004: 39:1005-1026.
Powell, R. Powell, H. Baker, L. & Greco, M. (2009) Patient Partnership in Care: A new instrument
for measuring patient–professional partnership in the treatment of long-term conditions.
Journal of Management & Marketing in Healthcare. Vol. 2 No. 4. PP 325–342.
Long Term Conditions 6 (LTC 6) (part of the QIPP programme) Department of Health and also
Living with your Long Term Condition (LWYLTC)
Safran DG, Karp M, Coltin K, Chang H, Li A, Ogren J, et al. Measuring Patients' Experiences with
Individual Primary Care Physicians Results of a Statewide Demonstration Project. Journal of
General Internal Medicine. 2006;21(1):13-21.
Adair CE, McDougall GM, Mitton CR, Joyce AS, Wild TC, Gordon A, et al. Continuity of Care and
Health Outcomes Among Persons With Severe Mental Illness. Psychiatric Services.
2005;56(9):1061-9.
King M, Jones L, Nazareth I. Concern and continuity in the care of cancer patients and their
carers: A multi-method approach to enlightened management. London: National
Coordinating Centre for NHS Service Delivery and Organisation 2006.
Dolovich LR, Nair KM, Ciliska DK, Lee HN, Birch S, Gafni A, et al. The Diabetes Continuity of Care
Scale: the development and initial evaluation of a questionnaire that measures continuity of
care from the patient perspective. Health Soc.Care Community. 2004;12(6):475-87
Kowalyk KM, Hadjistavropoulos HD, Biem HJ. Measuring Continuity of Care for Cardiac Patients:
Development of a Patient Self-report Questionnaire. Canadian Journal of Cardiology.
2004;20(2):205-12.
Hadjistavropoulos H, Biem H, Sharpe D, Bourgault-Fagnou M, Janzen J. Patient Perceptions of
Hospital Discharge: Reliability and Validity of a Patient Continuity of Care Questionnaire.
International Journal for Quality in Health Care : Journal of the International Society for
Quality in Health Care / ISQua. 2008;20(5):314-23.
Borowsky SJ, Nelson DB, Fortney JC, Hedeen AN, Bradley JL, Chapko MK. VA community-based
outpatient clinics: performance measures based on patient perceptions of care. Med Care.
2002;40(7):578-86.