Measuring & Understanding Quality
Improvement in Healthcare
Steve Meurer, MBA / MHS, PhD
Vice President - Operations
St Mary Medical Center
Langhorne, PA
Focus
Continuing the Journey
1) Research Questions
• Experience in Healthcare Operations
Oral Exam
2) Theory / Models
•Search and Study
•Develop Questions and Hypotheses
3) Develop / Test
Defense
•Develop 2 into something that can
help answer 1
4) Examine Results
• How does 3 answer 1
Initial Research Questions
Developed from 8 years of frustration in healthcare
management knowing that I wasn’t equipped to
provide appropriate support to clinicians
What I Could Provide
What I Needed to Provide
Leadership
Understanding of Healthcare
Financial Direction
Strategic Direction
Management Capabilities
A More Balanced Approach to
Managing
Assessment
Data Management
Study Design
Initial Research Questions

How do I know something works?


Do patients get better?


Continued requests for equipment, supplies and
instruments
The End of Medicine
Is there one way to do a procedure that is
better than another?

Significant variation in preference cards
How do I measure quality?
Defining Quality


IOM – The degree to which health services for
individuals and populations increase the
likelihood of desired health outcomes and are
consistent with current professional knowledge.
Donabedian - The systematic measurement and
evaluation of the predetermined outcomes of a
process, and the subsequent use of information
to improve the process based on expectations
of the customer.
Theories / Models – Oral Exam

Ernest Amory Codman


Florence Nightingale (late 1800s) and Walter Shewhart
(1920s)



structure – process – outcome
Implicit vs. explicit criteria
Everett Rogers


Continuous Quality Improvement (CQI)
Avedis Donabedian


Statistical Process Control
Edwards Deming, Joseph Juran and Kauru Ishikawa


end results idea – 1920s
Dissemination of Innovation
Don Berwick, Paul Batalden, Brent James and Steve
Shortell

Recent literature
Structure – Process - Outcome
Quality of healthcare can be assessed on the basis of
structure, process (how care is delivered), and outcome
(mortality, functional status, quality of life, and patient
satisfaction)
good measures of the first two are those that have a clear
relationship to the third
structure must proceed process which must proceed outcome

Structure Definition
Something arranged in a definite pattern of organization
 Organization of parts as dominated by the general
character of the whole

Implementing CQI is Largely Structure
Medical
Outcomes
Scott
Organization
Division of labor
Specialty Mix
Coordination
Resources
Training/Experience
Work Load
Power
Access
Resources
Buildings
Information
Leadership
Policies/Procedures
Tasks
CQI
HR / Training
Customer Focus
Planning
Process
Innovation
Supplier Partner
Information
Leadership
Structural Dimensions of CQI
Implementation
Strategic
Cultural
Technical
Organiz.
Result
No
Yes
Yes
Yes
Yes
No
Yes
Yes
Yes
Yes
No
Yes
No significant results
on anything important
Small, temporary
effects
Frustration & False
starts
Yes
Yes
Yes
No
Yes
Yes
Yes
Yes
Inability to capture the
learning & diffusion
Lasting process
change
Adapted from Shortell et al. 1996
Translating Theory To Research –
Putting Shape to my Frustrations
1. The healthcare system is broken
The IOM reports
2. Appropriate structure – “the forgotten, but important
component of the quality triad” – is wrongly assumed
as present
 JCAHO
3. CQI, if implemented properly, can be the appropriate
structure
Managerial Philosophy
4. Healthcare providers are are finding it difficult to
implement CQI
Problem Solving Methodology
Research Question

Why haven’t healthcare organizations
been able to use CQI to differentiate
themselves in terms of quality?
Answer: Accountability & Assessment
Accountability
Individual motivation has not been successful
Midnight at the Waldorf-Astoria
Rhetoric, not Reality (The Halothane Study)
Large scale environmental change is needed
Environment, Organization, Micro-System, Pt
Current Motivators that may force change
Patient Safety – Medical Errors
“report cards” - PA, NY and CA
Increase in Costs and Premiums
Increase in the use of Alternative Medicine
Variation in processes
Increase in litigation
Assessment Issues & Research Questions
1. Low CQI knowledge level of senior leadership
Do step by step instructions exist that assists researchers in studying CQI,
and healthcare leaders in implementing CQI?
2. Implementing only a subset of the CQI domains
Does a comprehensive survey and scale of CQI implementation exist?
3. Poor measurement strategies
Are psychometrics examined appropriately in measuring CQI implementation?
4. Rhetoric does not equal the reality
Does a measure exist that can be used to develop a quick snapshot of CQI
implementation efforts in a hospital and is there a corresponding descriptive
scale?
Step 1: Search the Literature
Does a comprehensive, valid and easy to administer
measurement tool exist that provides organization leaders
with a descriptive scale and instructions for next steps?

Methods Used Ovid databases with a focus on business,
psychology, sociology and healthcare
 Keywords: measurement quality, continuous quality
improvement, total quality management,
implementation
 Scrolled through quickly at least 10,000 references



most focused on implementing only a few domains of CQI
Articles not deleted included: Measuring Quality;
Domains and Implementation of Quality
Quality Overload
Using Medline through PubMed

from 1995 to the present / English only
‘quality improvement’ = 8,848
 ‘continuous quality improvement’ = 1,100
 ‘quality’ in title = 17,466
 ‘quality improvement measure’ in title = 3


from 2000 to present / English only

‘quality improvement’ in title = 350
 50
usable, 30 ‘easily findable’, 15 good, 1
measurement
Worldwide Measurement for QA/QI Structure
AWARDS - too time intensive, gold standard domains

The Malcolm Baldrige Award
Leadership, HR/Training, Process, Business Results, Customer
Focus, Information Systems, Planning, Partnership
EFQM
 US State-Based Awards

ACCREDITATION & CERTIFICATION - questions on
validity and too time intensive
JCAHO
IS0
9002 - 2000
Inspection, Contract, Public Responsibility, Innovation, Product
Control, Servicing
SURVEY INSTRUMENTS
Survey Instruments
Eleven worldwide surveys examining CQI as a
managerial philosophy were analyzed (Tables - pges
1 & 2)
8 from the US, 1 from Canada, the Netherlands, and
Australia
5 were specific to healthcare, including the ‘gold standard’
from Shortell et al.
Most examined psychometrics while very few provided a
scale
The shortest (22 questions) was also the least
comprehensive
The Baldrige Domains dominated
Survey Instruments
Weaknesses of the current surveys included:
Relatively few domains other than the Baldrige were
even mentioned
The most comprehensive surveys are much too long
(depth vs. breadth)
Strong potential for respondent bias if survey is only
given to one level employee in an organization
Very few surveys provided a scale
Published in the International Journal for Quality in
Health Care 2001: Volume 13, Number 3: pp 197-207
Step 2: Develop and test a comprehensive and
concise measure of CQI implementation

Initial Survey
14 domains (Baldrige, EFQM, ISO) and 70 items, with
each domain containing at least 4 items
 All but a very few items were from the 11 surveys
analyzed in Step 1
 Items were chosen by the researchers using a
subjective analysis and whether or not the question
could be answered using a 5 point Likert scale

Content Validity

The benefits of a content validity study for this study
True experts in the field of CQI
Past measures have gone through psychometric testing
Excellent method of data reduction
Methods

Statistical
method described in Grant & Davis (1997) and
Lynn (1986)
1. Panel of Experts
All either attend an invite only CQI symposium sponsored by
Dartmouth, have recently taught CQI at a Masters level or are
positional leaders of QI efforts in a healthcare organization
 Best to have between 7 and 10 - this study had 7 from the
US and 1 from England
Content Validity

Methods (cont.)
2. Scoring Grid (See Sample Grid - pge 3)
 Each expert was emailed the scoring grid with
definitions and instructions.
Is the item clear and understandable?
 4 point scale
Does the item represent CQI?

4 point scale
Match the item with a domain.
 1 through 14 representing each domain & 15 representing
unable to classify
Content Validity

Methods (cont.)
3. Indices
Inter-rater agreement (IR) = # of raters who scored an
item as high / total # of raters
 high
defined as a 1 or 2 on both 4 point scales
 acceptable IR > / = .70
Content Validity Index (CVI) = # of items where all
experts rated high / # of items
 acceptable
CVI > / = .80
Domain congruence = % of time where experts chose
the same domain as the investigators
Content Validity Results

After 4 analysis iterations where poorly rated items
were deleted, the questionnaire included:
22 items
 8 domains
 Clarity IR of .91 (range of .85 - 1)
 Representativeness IR of .93 (range of .87 - 1)
 Clarity CVI of .73 using Lynn’s (1986) method
 Representativeness CVI of .91 using Lynn’s (1986)
method
 Overall, the experts chose the same domain as the
investigators in the original measure 76% of the time

Content Validity Results
 Investigators added 6 items to ensure that every
domain except for Supplier Partnership contained 3
items
 Investigators changed some wording to increase
clarity
 Final survey was sent back to experts for comments
Other Results
The scoring grid took a lot longer to complete than originally
thought
The leadership domain had the highest rate of agreement
QI must be differentiated from QA
Baldrige criteria dominate
Step 3: Develop a corresponding scale of
CQI implementation

A 5 level corresponding scale (pge 4) was developed by
the investigators from:

Roger’s Diffusion of Innovations


Deming’s Continuous Quality Improvement


match domains with appropriate level
Samsa & Matchar


agenda setting, matching, redefining / restructuring, confirmation,
clarifying, routinizing
CQI as a problem solving methodology vs. a managerial
philosophy
Characteristics:



CQI is a developmental process
Time is important
Scale provides focus for future quality initiatives
Steps 2 & 3 - Content Validity and Scale
Published and Voted Best Student-Led Paper in
the 2002 Business and Health Administration
Proceedings,pges 198-204
Will be further published in Either Quality in
Health Care or Hospital Topics
Step 4: Is the survey and scale easy to use
and are the results practical?
Pilot Study

Worked with the Missouri Hospital Association for contacts.



83 Missouri hospitals eligible (above 40 beds), 40 participated
5 responses from each hospital: CEO/COO, Director of
Quality, a non-salaried MD, and 2 managers
Survey and results disseminated via email
Hypotheses based on Paper 1 Weaknesses
1. There will be measurable differences between and
within hospitals.
2. The survey will have high known-groups validity.
3. The items and domains will differentiate between levels
as hypothesized by the conceptual scale.
Pilot Study Methods

Known Groups Validity

Examined the relationship between the survey and:
state quality team winners >/= Level 2
 state quality organization winners >/= Level 3
 national quality award finalists >/= Level 3
 subjective quality assessment at 10 of the 40 hospitals
 question 1 (pge 5) asking the participants to categorize
their quality structure


Reliability

Cronbach’s Alpha for each domain, each title, and the
overall measure
Pilot Study Methods

Between Hospital Variation

One-way ANOVA & Bonferroni


Within Hospital Variation

Repeated Measures ANOVA & Bonferroni


by size, region and ownership model
by title
Item and Domain Analysis

ANOVA & Bonferroni

determine which items and which domains discriminate
well between different levels of the scale
Pilot Study Hospital Total Level
N = 40 (min of 90, max of 130)
16
14
12
10
8
# of hospitals
6
4
2
0
Level 1
(</= 100)
Level 2
(101-110)
Level 3
(111-120)
Level 4
(121-130)
Level 5
(131-140)
Pilot Study Results

Sample Characteristics (pge 6)



40 hospitals are significantly larger and more likely to be
for profit and part of a system
Of the 200 returned surveys, there was less than 5%
missing values and ‘I don’t know / NA’
Known Groups Validity

2 of the 12 did not score as hypothesized



hospitals were not significantly different than others
4 of 9 (44%) similar for the subjective assessment
40% agreement for question 1 assessment
Pilot Study Results

Reliability

Cronbach’s Alpha ranged from .54 (HR/Training) to
.84 (Innovation) for the domains
 Information
.69; Process, Planning .77; Customer .78;
Leadership .83

Cronbach’s Alpha ranged from .88 (Director
/Manager and QI Director) to .92 (MDs) for different
groups

Cronbach’s Alpha was .94 for the overall measure
Pilot Study Results

Between Hospital Variation


Region was only attribute that was significant
Within Hospital Variation
Senior Executives significantly lower than QI Directors
 QI Directors significantly higher than Managers /
Directors
 MDs significantly higher than Managers/ Directors

 supports
surveying more than one level employee
Pilot Study Results

Item and Domain Analysis (pge 7)

Of the 28 items, 4 did not show good differentiation
between any of the levels
 these
should be either reworded or changed
Leadership showed significant differentiation between
all levels studied
 Planning showed significant differentiation between 2 of
the 5 levels

Pilot Study Results
Q uality I mprovement Scale
Level
1. Q uality Assurance
Components after Domain Analysis
2. Q I Low
High Focus – Leadership ( visibility) ; Customer Focus
M edium Focus – I nnovation
3. Q I M edium
High Focus – Process; HR / Training; Planning
M edium Focus – Leadership ( support)
4. Q I High
M edium Focus – I nformation; Supplier Partnership
Low Focus – Leadership ( consistency) ; Planning
5.Q I - Absorbed
Findings





The survey is easy to administer
The survey provides a reliable and valid snapshot of
CQI implementation in a healthcare organization
No known group exists
The scale is a practical method of providing hospital
leaders with a roadmap for CQI implementation
Leadership is the most important component of
implementing CQI
Submitted to Health Services Research
A Likely Future Scenario
1.
2.
3.
4.
Patient Safety provides accountability to analyze
quality and outcomes
Healthcare leaders see CQI as a methodology to
improve patient outcomes
Hospitals use the survey and scale to help
assess & implement CQI appropriately, which in
turn eliminates structure issues discussed
Because of this, hospitals can effectively assess
their processes and improve their outcomes
Future Research Questions
Can a clearer snapshot of CQI implementation emerge using
line worker responses, and senior leadership interviews?

Administered the survey to a 40 random line workers at 10
hospitals

Initial results include:




50% return rate
high number of ‘I don’t know / NA’ responses
all hospitals overall employee score < 100 (Quality Assurance)
Senior hospital leadership meetings to discuss quality
structure

Initial results include:


low knowledge of ‘quality’ among the senior leaders
structures developed with little statistical or facilitation resources
Future Questions
Does a higher level of CQI implementation lead to better
financial, quality and safety outcomes? If not, why?
 Develop and Find Financial, Operational and HR
Effectiveness and Efficiency Measures
Counte & Glandon, 1995
 Build one, clean database with CQI implementation
scores and measures
 Analyze to assess relationships
Future Questions
Is CQI, as its described in the literature, an effective
method for improving quality outcomes?

Change the four items that did not differentiate well
1. How many multi-disciplinary teams currently work to improve the
processes of care in your organization? (Process)

more statistics, less teams
• Human Factors Research
• Toyota
• Six Sigma
2. Do people in your organization know who their customers are?
(Customer Focus)

not clear
3. Are employees in this organization encouraged to try new and
better ways of doing things? (Innovation)
4. Is creativity actively encouraged in this organization? (Innovation)

healthcare has typically not been innovative and so these questions
may need to be more specific
Acknowledgements and Questions
Mentor – Dr. Counte
Committee – Drs. Arrington, Rubio &
Burroughs
Dr. Dunagan
Gretchen