Overview of the
Hospital Safety Score
March 2013
Missy Danforth, Senior Director of Hospital Ratings, The Leapfrog Group
Presentation Overview
•
•
•
•
Why is Leapfrog issuing a Hospital Safety Score
Who is getting a Hospital Safety Score?
Important Dates
Scoring Overview –
• Measure Selection
• Measure Weights
• Changes to the Scoring Methodology Since November 2012
• Bound on negative z-scores
• Dealing with Missing data
• Details of the data review process
• Questions
2
What is the Hospital Safety Score?
Why is Leapfrog issuing one?
• The Hospital Safety Score is an A, B, C, D, or F letter grade
reflecting how safe hospitals are for patients. For the first time
ever, this score empowers healthcare consumers to make
informed decisions based on the safety of their hospital.
• Leapfrog is strategically expanding its focus to get consumers
to use information on safety and quality that can save their
lives.
3
What Hospitals Will Receive a
Hospital Safety Score?
• General, acute-care hospitals for which there is adequate
public data
• Excluded Hospitals include:
• Specialty Hospitals (i.e. surgical centers, cancer hospitals,
women’s hospitals, etc.)
• Critical Access Hospitals
• Free –Standing Pediatric Facilities
• Non-IPPS participating hospitals (hospitals from the state of MD)
• Hospitals Missing Too Much Data:
• More than 9 process measures
• More than 3 outcome measures
4
Important Dates
• January 2013 – Expert panel re-convened to review feedback from November 2012 safety
score release and develop recommendations for the scoring methodology.
• March 2013 – Leapfrog’s board approved recommendations of expert panel.
• March 2013 - Letter sent to CEOs of hospitals receiving a Hospital Safety Score. Letter
included:
• Information about the Hospital Safety Score
• A username/password to a secure website where hospitals can review the source data that
Leapfrog used to calculate their numerical safety score
• Links to the Hospital Safety Score help desk and helpful documents
• March 20th to April 8th - Data Preview Period
• Mid April – data preview website refreshed with any changes received during data review
period.
• Late April - Hospitals will be able to preview letter grades 48 hours prior to the pubic
release (www.HospitalSafetyScore.org).
• For more information about important dates, visit: http://www.hospitalsafetyscore.org/forhospitals/updates-and-timelines-for-hospitals
5
SCORING OVERVIEW
6
Measure Selection Criteria
• Publicly-reported from national data sources, reflecting
individual hospital results
• Leapfrog Hospital Survey
• CMS’s Hospital Compare
• Endorsed or in use by a national measurement entity
• Linked to patient safety (“freedom from harm”)
• Directly quantifying patient safety events
• Assessing processes that lead to better outcomes
• Identified by experts as important to patient safety
• Forty-five candidate measures identified; 26 recommended
for inclusion in score (note PSI 11 not available for the scoring
period)
7
Measures included in the Hospital Safety Score
Measure Name
Computerized Physician Order Entry (CPOE)
Primary Data Source
Process and Structural Measures (15)
2012 Leapfrog Hospital Survey
ICU Physician Staffing (IPS)
Safe Practice 1: Leadership Structures and Systems
Safe Practice 2: Culture Measurement, Feedback and Intervention
Safe Practice 3: Teamwork Training and Skill Building
Safe Practice 4: Identification and Mitigation of Risks and Hazards
Safe Practice 9: Nursing Workforce
Safe Practice 17: Medication Reconciliation
Safe Practice 19: Hand Hygiene
Safe Practice 23: Care of the Ventilated Patient
SCIP INF 1: Antibiotic within 1 Hour
SCIP INF 2: Antibiotic Selection
SCIP INF 3: Antibiotic Discontinued After 24 Hours
SCIP INF 9: Catheter Removal
SCIP VTE 2: VTE Prophylaxis
Foreign Object Retained
Air Embolism
Pressure Ulcer – Stages 3 and 4
Falls and Trauma
CLABSI
PSI 4: Death Among Surgical Inpatients
PSI 6: Iatrogenic Pneumothorax
PSI 11: Postoperative Respiratory Failure
PSI 12: Postoperative PE/DVT
PSI 14: Postoperative Wound Dehiscence
PSI 15: Accidental Puncture or Laceration
[i] AHA Annual Survey © 2011 Health Forum, LLC
2012 Leapfrog Hospital Survey
2012 Leapfrog Hospital Survey
2012 Leapfrog Hospital Survey
2012 Leapfrog Hospital Survey
2012 Leapfrog Hospital Survey
2012 Leapfrog Hospital Survey
2012 Leapfrog Hospital Survey
2012 Leapfrog Hospital Survey
2012 Leapfrog Hospital Survey
CMS Hospital Compare
CMS Hospital Compare
CMS Hospital Compare
CMS Hospital Compare
CMS Hospital Compare
Outcome Measures (11)
CMS Hospital Compare
CMS Hospital Compare
CMS Hospital Compare
CMS Hospital Compare
Leapfrog Hospital Survey
CMS Hospital Compare
CMS Hospital Compare
CMS Hospital Compare
CMS Hospital Compare
CMS Hospital Compare
CMS Hospital Compare
Secondary Data Source
AHA Annual Survey (2009 Technology
Supplement to the AHA Annual
Survey) i
2010 AHA Annual Survey i
CMS Hospital Compare
8
Weighting Process
• Two (2) measure domains, each weighted 50%:
1.
2.
Process/structural measures
Outcome measures
• Three (3) criteria for weighting individual measures
• Strength of Evidence (rating of 1 or 2)
• Opportunity (rating of 1, 2, 3), based on coefficient of variation
• Impact (rating of 1, 2, or 3), based on no. of patients possibly
affected by the event and severity of harm to individual patients
• Weight score: [Evidence + (Opportunity x Impact)]
9
Z-Score Methodology
• Standardizes data from individual measures with different
scales
• Counts how many standard deviations a hospital’s score on
the measure is away from the mean
• Mean always equals 0; worse than mean = negative z-score ;
better than mean = positive z-score
• Translate raw score on measure to z-score:
• Process/Structural Measures = [(Hospital Score – Mean)/Standard
Deviation]
• Outcome Measures = [(Mean – Hospital Score)/Standard
Deviation]
10
Overall Score
• Summation of z-score for each measure × weight for each
measure
3.0 + CPOE z-score × CPOE weight + IPS z-score x IPS weight +
CLABSI z-score × CLABSI weight . . . . etc.
• If measure has missing data, then weight for that measure is
re-apportioned to other measures within the same domain
• 3.0 was added to each hospital’s final score to avoid possible
confusion with interpreting negative patient safety scores
11
CHANGES TO THE SCORING METHODOLOGY SINCE
NOVEMBER 2012
12
Bounds on Z-scores
• Issue: For those measures in which hospitals are clustered at
the ‘best performing’ end of the scale (e.g., rare outcomes;
some process measures), hospitals that do not perform well
on the measure can have a substantial negative z-score (i.e. -7
to -20). Even though each individual measure carries a
relatively low weight, this substantial negative z-score on one
measure can have more influence on the hospital’s overall
safety score than was intended.
• Change to scoring methodology: In an effort to reflect the
actual variation in hospital performance, but to minimize the
possibility of a single measure overwhelming a hospital’s
overall score, we are implementing a lower bound on z-scores.
Based on analysis of the November 2012 safety score data, we
are setting the lower bound for z-scores at -5.0.
13
Small Denominators
• Issue: Minimum reporting requirements vary among data sources and
measures. For measures that have a publicly reported numerator and
denominator, Leapfrog will establish reporting requirements that meet
Leapfrog’s standard for scientific acceptability.
• Change to scoring methodology: When a measure’s denominator is
publicly available, Leapfrog applies the following minimum reporting
requirement for using the measure in the safety score: the number of
cases in the denominator must be >= 30.
• This minimum reporting requirement was identified from the literature,
which suggests that thirty cases is generally the point when a nonnormal distribution begins to approximate a normal distribution, which
is important given the Safety Score’s use of z-scores for standardizing
data across disparate data sets. The minimum sample size of 30 has also
been used by other organizations that are engaged in evidence-based
public reporting of health care performance data.
14
DETAILS OF THE
DATA REVIEW PROCESS
15
Secure Website
• http://www.leapfroggroup.org/data-validation/validationlogin
Details on what public
reports were used to
obtain source data. Links
to source data with
instructions.
Hospitals must confirm
their Medicare Provider
Number before moving on
to the next page.
16
Source Data
Instructions for hospital to
review their source data and a
link to the help desk
For each measure, hospitals
are provided with the
measure name, type of
measure, data source
(primary or secondary),
reporting period, measure
score (i.e. points, rate, SIR,
etc)
Hospitals are asked to confirm
that each Measure Score
matches their score (i.e. rate,
SIR, etc) from the Data Source
(i.e. Leapfrog Hospital Survey
Results, CMS Hospital
Compare, etc)
17
Where to Locate Source Data –
Hospitals That Submitted a Leapfrog Hospital Survey by February 28, 2013
• CPOE: Visit www.leapfroggroup.org/cp, find the column
labeled “prevent medication errors.” Leapfrog score used to
determine points towards safety score (i.e. 4 bars equals 100
points, 3 bars equals 50 points, etc).
• ICU Physician Staffing: Visit www.leapfroggroup.org/cp, find
the column labeled “Appropriate ICU Staffing.” Leapfrog score
used to determine points towards safety score.
• CLABSI Standardized Infection Ratio (SIR): Visit
www.leapfroggroup.org/cp, find the column labeled “prevent
ICU infections.” The SIR was used to calculate in the safety
score calculation, and is located in the first sentence in the
first paragraph above the table.
18
Where to find your hospital's CLABSI SIR
on Leapfrog’s results website
19
Where to Locate Source Data –
Hospitals That Did Not Submit a Leapfrog Hospital Survey by February 28, 2013
• CPOE: Leapfrog used information you provided to the
American Hospital Association through the 2009 AHA
Technology Supplement.
• ICU Physician Staffing: Leapfrog used information you
provided to the American Hospital Association through the
2011 AHA Annual Survey.
• CLABSI Standardized Infection Ratio (SIR): Leapfrog used
information published by CMS. Visit www.data.medicare.gov.
Select the “Healthcare Associated Infections” report and
search for hospital by name or MPN (do not type in the
hyphen sometimes included with the MPN).
20
Where to find your hospital’s CLABSI SIR
on CMS’ results website
21
22
Where to Locate Source Data –
CMS Data for All Hospitals
• SCIP Measures: Hospital rates are published by CMS at
https://data.medicare.gov/dataset/Hospital-Process-of-CareMeasures-Surgical-Care-Im/iz5u-4mk2
• HACs: Hospital rates are published by CMS at
https://data.medicare.gov/dataset/Hospital-AcquiredCondition-Measures/qd2y-qcgs
• PSIs: Hospital rates are published by CMS at
https://data.medicare.gov/dataset/Agency-For-HealthcareResearch-And-Quality-Measure/vs3q-rxc5
• You can download your hospital’s results into a CSV or XLS
formatted file.
23
What if the Measure Score doesn’t
match the public report?
• Hospitals are asked to contact
the help desk immediately
once they have confirmed the
measure and reporting
period.
• Hospitals must provide a copy
of the public report that
shows a different score
• If we find a recording error,
we will update the score and
re-issue a numerical safety
score
24
Preview Numerical Safety Score
25
More Information
• Hospital Safety Score Help Desk scorehelp@leapfroggroup.org
• Hospital Safety Score Website – www.HospitalSafetyScore.org
• Data Review Website - http://www.leapfroggroup.org/datavalidation/validation-login
•
26