“The public is concerned
about hospital quality but
has less information about
selecting a hospital than
they do when purchasing a
toaster or a car” (Malone, 1999)
“Puffery”



“…representation, statement, or
conduct that clearly overexaggerates…
…is not intended to be an offer to be
relied upon.”
Consumer protection
(2000 Legal on-line Dictionary)
Methods
1.
2.
3.



Solucient’s “Top 100 Hospitals”
US News & World Report’s “Best Hospitals”
American Nurses Credentialing Center’s
“Magnet Recognition Program”
Descriptive statistics
Spatial mapping
Review of empirical literature
US News & World Report’s
“Best” Hospitals
“If you are looking for
the ‘best’ in medical
care…”
—web-based guide.
“designed to assist you
in your search for the
highest level of
medical care.”
(US News & World Report)
US News: Selecting the “Best”







National Opinion Research Center
Each July since 1990
Ranked by specialties (17)
“Only rigorous assessment of all US hospitals”
Eligible -- “tertiary care hospitals”
 COTH membership or
 Affiliation with medical school or
 Availability of 9 out of 17 prescribed
technologies
Hospital Quality Index (HQI)
Honor Roll – score well in at least 6 specialties
US News continued…

Measures: produce HQI
 Structure = AHA Annual Survey data
FTE “on-staff” Nurse to bed staffing ratios
• Services available (technology index)
– # of specialty-specific technology services
Process = (“proxy measure”)
• Physician nominations
• Newest specialties reputation only
Outcomes =
• Medicare risk-adjusted mortality by specialty
• Pool most recent 3 years
• Mortality ratio = # inpatient deaths/expected
•


Solucient’s “Top 100”
Hospitals

“Identifies
industry
benchmarks by
recognizing
hospitals that
demonstrate
superior clinical,
operational, and
financial
performance.”
Selecting the “Top 100”








Since 1993 -- December
“Management results that separate these
hospitals from the pack.”
“Which hospitals offer superior care at
lower costs.”
“Data obsession--common trait.”
Enterprise = benchmark reports for
purchase
Eligibility restricted: 5 categories (n=20)
Overall + select specialties (ICU, CV,
Ortho, Stroke benchmarks)
Data source – Medicare Cost Report
Solucient “TOP” – 5 strata
Major Teaching (400 or more beds &
resident to bed ratio of at least .25)
 Teaching (at least 5 residents or
resident to bed ratio of .01-.24)
 Large Community (250 or more beds)
 Medium Community (100-249 beds)
 Small Community (25-99 beds)

“Top 100”selection…

Ranked by strata on following measures—20 best rankings
per strata selected
 Financial management =
• Expense/adjusted discharge: case mix and wage
adjusted
• Profitability
• Proportion of outpatient revenues
• Index of total facility occupancy
 Clinical performance =
• Medicare risk-adjusted mortality index
• Risk-adjusted complication index
 Operational performance =
• Severity-adjusted average LOS
• Case-mix and wage-adjusted expense per
discharge
ANCC Magnet Recognition
Program for Excellence in Nursing
(1994)
ANCC Recognizes “excellence”
in 4 target areas
Management philosophy & practices
2.
Adherence to standards for improving
the quality of care
3.
Leadership of the CNO in supporting
professional practice and continued
nursing competence
4. Attention to the cultural and ethnic
diversity of patients, their significant
others, and providers
1.
Magnet Recognition
Process
Self evaluation: ANA’s Scope & Standards
 ANA quality indicator data (NDNQI)
 Site visit
 Community input
 Could range from $30,000 - $75,000
 Reevaluated every 4 years or more often with
leadership change

Continuous Process: submit data annually
Diffusion of “Magnetism”
by year and total number recognized
120
100
80
60
40
20
0
1994
1996
1998
2000
2002
2004
“Creep” & Overlap - 2001 & 2003
Designation
Best
Top
Magnet
Best & Magnet
Best & Top
Top & Magnet
Best, Top &
Magnet
2001
173
101
47
14
15
2
2003
203
101
91
25
13
12
1
6
Best, Top, & Magnet
1.
2.
3.
4.
5.
6.
University of CA Irvine
Poudre Valley Hospital, Fort Collins
CO
Mayo Clinic, Rochester MN
NC Baptist, Wake Forest Un., NC
Methodist Hospital, Houston, TX
Inova Fairfax Hospital, Falls Church
VA
Summary – Best, Top & Magnet
Best (14yrs)
Top (11yrs)
Magnet (10yrs)
Measurement
Framework
norm referenced
norm
referenced
criterion
referenced
Focus
structure,
process,
outcomes
outcomes
structure, process
Self-nomination no
no
yes
Self-appraisal
no
no
yes
Site visit
no
no
yes
Nursing
yes
no
yes
Purpose
“inform
consumer
choice”
“excellence in
management”
“excellence in
nursing care”
Empirical
evaluations
3
3
4
U.S. News “Best Hospitals” - 2003
Solucient’s “Top 100 Hospitals” - 2003
ANCC’s Magnet Hospitals - 2003
Best, Top, & Magnets - 2003
Regional dominance =
“Pockets of excellence” ??




High preponderance – northeast, south
6 states – “empty”
Mountain, west north central, west south
central regions
Some states in highly designated regions
had fewer hospitals than those in the
“empty regions”
CA -- 450 hospitals (3 magnets) = .7%
 NJ – 74 hospitals (14 magnets) = 19%

The US NEWS “Best” -- 3 Empirically
based publications




Different rankings when different decision models
used (Teasley, 1996)
CABG outcomes in PA and NY (Hartz et al., 1997) “Best did not perform better than rest”
Significantly lower AMI mortality rates (Chen et
al. (1999)
 Clinical practice processes
 Asprin & beta blockers
Numerous reports of methodological “flaws” &
conceptual issues


What is examined
Who does the nomination
“best colleges”
 “best party schools”

Solucient “Top” -- 3 empirically based
publications
1.
CHEN et al., 1999
•
•
2.
Griffith & Alexander 2002
•
3.
AMI patients over 65 Significantly lower LOS
and costs than “rest”
No significant difference in mortality rates
Found that 8 of the Solucient measures of
hospital performance were valid and reliable
Griffith, Knutzen & Alexander, 2002
•
“a disjuncture” between “Top” outcome
measures and JCAHO evaluations in nonfederal hospitals
ANCC Magnet Hospitals – 4 Empirically
based publications

Aiken, Havens, & Sloane, 2000







support for nursing practice
nurse reported quality
nurse satisfaction
better nurse staffing ratios
more highly educated nurses
nurse burnout
Havens, 2001





strong departments of nursing
quality ratings
organizational support for nursing practice
difficulty recruiting nurses
patient and family complaints

Upenieks, 2002, 2003
nurse satisfaction in ANCC magnet hospitals
-- linked to stronger nurse manager leadership

levels of RN reported empowerment


Havens, (Forthcoming)


use of “outside” nurses
vacancy rates
Comparison of JCAHO Hospital
Evaluation Score Means
(2002 JCAHO file)
Hospital Type
mean
score
SD
All Community Hospitals (Federal,
psych and other specialties
eliminated)
90.84
4.43
U.S. News “Best”
91.66
3.68
Solucient “Top 100”
93.13
3.05
ANCC Magnet Hospitals
94.43
1.27
Hospitals with Multiple
Designations
96.34
2.04
JCAHO “Commended”
96.94
1.61
Overall Evaluation Scores
Comparative Data - 2002
Do JCAHO scores contribute to
designation?

No statistically significant differences between
designated hospitals and the rest on JCAHO
scores (t-test)

Logistic regression models of designation (does
JCAHO score predict designation or not?)
 Unrelated to receiving best, top, or magnet
 Higher JCAHO scores only slightly (and not
significantly) predict being designated.
 Model fit is poor.
Havens, Vasey, & Kellogg 2003,
Unpublished
What is the take home?






Concern about the quality of hospital care
Adverse events and staffing deficiencies
cause alarm
Scant information to assess performance
No universal, “consumer friendly”
measure of quality
Growing # of ratings / rankings
Competition and marketing campaigns
AHRQ advice re: selecting
quality care…

“Look for a hospital that:

Is accredited by JCAHO.

Is rated highly by the state, consumer groups,
or other organizations.”
http://ahrq.gov/
“PR Bonanzas”










“Americans have a love affair with rankings.”
“Public relations goldmine”
Marketing campaigns
News media pick up
Web-sites
“Get a 2nd opinion from the ‘best’ hospitals”
“Helps our development efforts”
US News – “hottest marketing tool in the healthcare
industry”
Financial boon for awarding organizations…
And ….more
Research Implications








Paucity of research
Mixed findings
Are the “Best really better than the rest”?
Are some of the best better than other bests?
Best on what?
Is this meaningful to consumers - The big picture
paradigm?
Reliance on structure and process only
What are the key questions?
 Safety
 Patient perspectives
Policy Implications










A case for moving research to inform policy
Consumer info, mis-info, or dis-info?
As competition grows—release of comparative
information will increase
“Desire to hold healthcare organizations
accountable…need accurate measures.” (Devers)
How to uniformly gauge quality?
Some argue that public does not understand or will not
use…the public is requesting
“top 10” is part of American culture
How to depoliticize – big $ maker for magazines
Use of e-health sites is growing
Being touted as a decision guide for consumers
 Web sites
 Ladies Home Journal
 Reader’s Digest
We are on the job!
Stay tuned