The Measurement and
Monitoring of Safety
Charles Vincent
Health Foundation Professor of Psychology
University of Oxford
Charles
Vincent
Jane
Carthey
Susan
Burnett
10% patients
harmed, half
judged
preventable
UK National Reporting & Learning System
Hospital Episode Statistics: 11.8M
hospital admissions in England
2004/5
But incident
reporting only
detects 5% of
harmful events
We do not know whether we
are making progress or not
Just tell me - are we safe?
Commissioning. How do we know care is safe?
• Tools and approaches
to measuring safety
• Provide a future
direction
• Jane Jones, Jonathan
Bamber
Methods (1)

Reviews of research literature and reports from
organisations:
–
–
–
–

Safety relevant industries
Conceptual approaches and models of systems safety
Measurement and monitoring in healthcare
The role of patients and families
Interviews with senior staff in national
organisations
Methods (2)

Case studies in healthcare organisations in the UK
and USA
– Acute & specialist trusts
– Mental Health
– Primary care
– Combined organisations
– Clinical services: maternity care, care of the
elderly, anaesthesia
The fundamental questions





Has patient care been safe in the past?
Are our clinical systems and processes reliable?
Is care safe today?
Will care be safe in the future?
Are we responding and improving?
Patient Safety
Safety
in NHS
High Risk
Industries
Models
of Safety
Case Studies
Conceptual Structure of Report
Safety in
NHS
Reliability
High Risk
Industries
Sensitivity
to
operations
Models of
Safety
Anticipation
&
Preparedness
Case Studies
Integration &
Learning
Principles of Safety Measurement and Monitoring
Patient Safety Concepts
Past Harm
What do we mean by harm?







Treatment specific harm
Harm due to over treatment
General harm from healthcare
Harm due to failure to provide appropriate
treatment
Harm due to failed or inadequate diagnosis
Psychological harm and feeling unsafe
Harm due to neglect and dehumanisation
Adverse events in older people
•
•
•
•
Errors, omissions
Operative/procedural complications
Hospital acquired infections
Adverse drug events
Adverse
events
affecting all
age groups
•
•
•
•
Falls
Pressure sores
Incontinence
Functional ± mobility
decline
Delirium
Depression
Nutritional decline
Dehydration
The
geriatric
syndromes
•
•
•
•
+
Should be thought of as adverse events
• Preventable?
• Lead to prolonged hospital stay
• Increased morbidity and mortality
NHS Safety Thermometer: Falls with Harm
Proportion of NHS Safety Thermometer Falls with Harm National
p chart
Temporary: UCL = 1.53, CTL = 1.24, LCL = 0.94
Inspected Mean = 12,477.77, Counts Mean = 154.31
2.0
UCL
1.6
1.4
CTL
1.2
1.0
20
12
01
/0
5/
20
12
01
/0
4/
20
12
01
/0
3/
20
12
01
/0
2/
20
12
01
/0
1/
20
11
01
/1
2/
20
11
01
/1
1/
20
11
01
/1
0/
20
11
01
/0
9/
20
11
01
/0
8/
20
11
01
/0
7/
20
11
01
/0
6/
20
11
01
/0
5/
01
/0
3/
20
11
LCL
20
11
0.8
01
/0
4/
Proportion of patients with harm from a fall (%)
1.8
Month
Data sourced from the NHS Safety Thermometer national data set April 2012. This data highlights the national mean proportion of patients
with harm from a fall (based on a monthly point of care survey) is 1.2%. Caution must be taken regarding the sample – the control limits of
this p-chart highlight the changes in sample size which includes pilot periods when submitting organisations were providing data on only 50%
of pilot wards (4) through to 100% of patients in hospital care on one day. In March 2011 the sample was 5700. In March 2012 this was
49,917.
Are our clinical systems and processes
reliable?
• Measuring and testing reliability: the WISER study –
– Clinical information availability at the point of decision
making
– Prescribing for hospital inpatients
– Equipment in theatres
– Equipment for inserting IV lines
– Handover between wards
I’m looking
for...
•
•
•
•
•
•
•
•
•
Past medical history
Referral letter/other specialty letter
Discharge summary
Current medication
Radiology/imaging results
Diagnostic test results
Procedure notes/anaesthetic record
Electrocardiogram (ECG) report
Blood results
Missing information overall
Site
Total number of
patients in the
sample
Number of patients with
missing information
(% of all patients in
sample)
A
411
18 (4%)
E
423
113 (27%)
G
327
44 (13%)
TOTAL
1161
175 (15%)
Differences between organisations
Reliability of
equipment availability
in operating theatres
Missing & faulty equipment
Site
Total
operations
studied
Number of
operations with
equipment
problems
A
258
50
56
19%
D
67
25
28
37%
F
165
19
19
12%
Total
490
94
103
19%
Number of Percentage operations
equipment
with one or more
problems equipment problems
‘We always need a colposcope with that
list and time and time again it isn’t there
or it’s broken or it isn’t back or nobody
knows where it is’
Surgeon 3 Organisation A
Sensitivity to operations



At the coal face, minute by minute, safety may either be
eroded by the actions and omissions of individuals or,
conversely created by skilful, safety conscious
professionals
Clinicians monitor their patients, watching for subtle signs
of deterioration or improvement, but also have to monitor
their teams for signs of discord, fatigue or lapses in
standards.
Managers have to be alert to the impact of staff shortages,
equipment breakdowns, sudden increases in patient flow
and a host of other potential problems.
Soft intelligence






Safety walk-rounds
Using designated patient safety officers
Operational meetings, handovers and ward rounds
Briefings and debriefings
Day to day conversations
And above all …. the patient voice
Anticipation and Preparedness:
Will care be safe in the future?






WHO Surgery Checklist
Risk assessments
– (falls, pressure ulcers, self harm)
Risk registers
Safety culture assessments
Safety cases
Bringing available information in the organisation
to anticipate safety in the future
Predicting mortality by day of the week
There were 27 582 deaths within 30 days after 4,133,346
inpatient admissions for elective operating room procedures


Crude mortality rate 6.7 per 1000)
The adjusted risk of death was higher if the procedures were
carried out on Friday (+ 44%) or a weekend (+ 82%)
compared with Monday.

Nurse Staffing and Quality of Care

Hospitals with low nurse staffing levels tend to
have higher rates of poor patient outcomes such as
pneumonia, shock, cardiac arrest, and urinary tract
infections,

http://www.ahrq.gov/research/findings/factsheets/services/
nursestaffing/index.html. AHRQ Research in Action, Issue
14 (2004)
Integration & learning. Are we
responding and improving?
Berwick Report
“Most Health care organisations at present have very
little capacity to analyse, monitor, or learn from
safety and quality information. This gap is costly
and should be closed and that early warning signals
can be valued and should be maintained and heeded”
(Berwick, 2013, p26)
Safety Information System



A safety information system should really be seen
as an ‘information, analysis, learning, feedback
and action’ system.
Many organisations currently expend most of their
efforts on data collection, to the detriment of other
aspects.
Too narrow a focus on reporting means less
invested in the more critical areas of feedback and
learning.
Integration of information

Data sources could include:
–
–
–
–
–
–
–
incidents reported,
Indicators administrative data,
complaints, health and safety
incidents, inquests, claims,
Clinical audits, routine data,
Observations, both quantitative and qualitative
informal conversations with patients, families and staff.
Great Ormond St: team level







Number of days since the last serious incident (SI)
– narrative, lessons learnt and recommendations
Central venous line, MRSA (MSSA) infection rates
Hand hygiene compliance rate
WHO Surgical Safety Checklist compliance rate per
clinical unit
Common themes identified in executive walk-rounds
Medication errors
Top three risks from the clinical unit’s risk register.
Intermountain Healthcare



Online reports portal with 80 quality and patient safety
metrics patient safety metrics
Use of electronic records and data provided by care
provider as part of clinical workflow
Web-enabled reporting and SPC charts on demand
including:
– Centres for Medicare and Medicaid Services (CMS)
– The Joint Commission core measures,
– Quality Forum (NQF) etc. Intermountain captures
patient harm from existing
10 Guiding Principles (1)
1. A single measure of safety is a fantasy
2. Safety monitoring is critical and does not receive
sufficient attention
3. ‘Leading indicators’ are needed to anticipate and
be proactive
4. Safety information is fragmented in healthcare
organisations – integration and learning needs
investment in technology and data analysis
5. Safety information needs to be customised to the
different levels in the organisation
10 Guiding Principles (2)
6. There is a need for a blend of externally and
internally agreed metrics
7. Clarity of purpose is when developing safety metrics
8. Empowering and devolving responsibility for the
development of safety metrics is essential
9. Collaboration between the regulator and regulated is
essential
10. Beware of perverse incentives!
What information do you have in your
organisation which will tell you:






Has patient care been safe in the past?
Are your clinical systems and processes reliable?
Is care safe today?
Will care be safe in the future?
Is your organisation integrating, learning
responding and improving?
Where do you need to focus attention in future to
measure and monitor safety?
http://www.health.org.uk/publications/the-measurement-and-monitoring-of-safety/