How to Publish Your Quality
Improvement Project—
Designing and Executing Your QI Project as
Scholarly Activity
DAVID COOPERBERG, MD
DEPARTMENT OF PEDIATRICS
SECTION OF HOSPITAL MEDICINE
Acknowledgements


SCHC Resident QI Curriculum Team

Celeste Chamberlain, PhD, Director of Quality

Cheryl Gebeline-Myer, MS, former Director of Process Improvement

Mackenzie Frost, MD

Paul Shore, MD, MS
Improving Adolescent HIV Screening Team

Lorena Pereira, MD

Clint Steib, HIV Coordinator

Peter Osgood, MD

Roberta Laguerre, MD

Dan Conway, MD

Doug Thompson, MD, MMM

Evan Weiner, MD

Mario Cruz, MD (co-designed handout)

Katie McPeak, MD and the Improving Time to Third Next Available Newborn Clinic Team

Aarti Patel, MD and the Improving Nurse-Resident Communication Team

Tess Woehrlen, MPH and the Improving Hospitalist-Primary Care Provider Handoff Team
Objectives
1.
Design a high-impact QI project
2.
Apply the SQUIRE guideline to draft a QI manuscript
Disclosure

I have no relevant disclosures
Disclaimer

This talk is not about Quality Assurance

Office of Research provides additional support and random reviews
through the QA/QI division

For more information:

Contact: Office of Research

Karen Skinner, MSN, RN, NHA, CCRP, Director QA/QI

Kirtanaa Voralu BSc, MStat, Analyst

Drexel University1601 Cherry Street, Suite 10-444, Philadelphia, PA 19102

Tel: 215.255.7883 | Fax: 215.255.7874

http://www.drexel.edu/research/compliance/qa/
Resources

www.squire-statement.org

Handout: Quality Improvement Scholarship: Taking Your QI Project
to the Next Level

Handout: Examples

QI Posters

QI Abstract accepted as platform presentation

Published QI Reports
Brief Review of Quality Improvement
in Healthcare

Systematic approach to problem identification
and improvement

Multidisciplinary team process

Objective, data driven process

Minimizing variation in
processes to improve outcomes
7
Outline

Steps in Designing and Executing QI Project

Use Standardized Quality Improvement Reporting Excellence
(SQUIRE) Guideline to Draft QI Report Manuscript

Use example of Improving Adolescent HIV Screening Rates at SCHC
Adequate Preparation
Before You Get Started…

Check with Drexel IRB

Letter of determination

IRB exempt

Expedited IRB review
Adequate Preparation

CITI Training

Perform Literature Review
 Existing

programs/projects
Discuss with local/regional experts in field
Building Your Improvement Team

Identify Key Stakeholders
Stakeholder

Definition: Anyone who can help affect change or may be
effected by the potential interventions
Consider Influence and Interest
Influence
Interest
Option 1
High
High
Option 2
High
Low
Option 3
Low
High
Option 4
Low
Low
Jonathan Boutelle: Understanding Organizational Stakeholders for Design Success, 2004.
http://www.boxesandarrows.com/view/understanding_organizational_stakeholders_for_design_success
Prioritize Stakeholders
Influence
Interest
Option 1
High
High
Option 2
High
Low
Option 3
Low
High
Option 4
Low
Low
Jonathan Boutelle: Understanding Organizational Stakeholders for Design Success, 2004.
http://www.boxesandarrows.com/view/understanding_organizational_stakeholders_for_design_success
Prioritize Stakeholders
Influence
Interest
Option 1
High
High
Option 2
High
Low
Option 3
Low
High
Option 4
Low
Low
Jonathan Boutelle: Understanding Organizational Stakeholders for Design Success, 2004.
http://www.boxesandarrows.com/view/understanding_organizational_stakeholders_for_design_success
Prioritize Stakeholders
Influence
Interest
Option 1
High
High
Option 2
High
Low
Option 3
Low
High
Option 4
Low
Low
Jonathan Boutelle: Understanding Organizational Stakeholders for Design Success, 2004.
http://www.boxesandarrows.com/view/understanding_organizational_stakeholders_for_design_success
Self- and Team-Reflection
 Does
 Do
your improvement team have the requisite skills?
team members represent relevant perspectives?
 Who
else should join your improvement team?
Team Roster
Project Year 2 (2012-2013)

Team leader: Lorena Pereira

Residents:

HIV Coordinators:

Barbara Bungy

Clint Steib

Yasmin Bahora

Daria Ferro

Zoabe Hafeez

Diamond Harris

Daniel Conway

Elisabeth Heal

David Cooperberg

Ji Kong


Meyeon Shin
Jill Foster

Peter Osgood

Katie McPeak

Yesha Patel

Roberta Frederick-Laguerre

Lauren Weaver

Doug Thompson

Mentors:
Getting Started


Selecting a High-Impact Project

High Risk

High Volume

Problem-Prone

Evidence-based (Recognized Standard)
Align with Organizational Priorities
Institute of Medicine Aims for Improvement
The Triple Aim

Improve patient experience

Improve the health of populations

Reduce per capita costs of health care
Berwick DM, Nolan TW, Whittington J. The Triple Aim: Care, Health, and Cost. Health Aff 2008;27:759-769
The Improvement Model
What are we trying to accomplish?
How will we know that a change is
an improvement?
What changes can we make that will result in
improvement?
Act
Plan
Study
Do
Langley et al The Improvement Guide 2009
Introduction
Introduction

Brief Literature Review

Nature and Severity of Local Problem

Specific Aim

Measures

Primary and Secondary Study Questions
Brief Literature Review

Incidence of AIDS has increased by 21% in youth aged 13-24 years1

Half of all new HIV infections occur in 13-24 year olds2

Risk-based testing associated with delayed diagnosis3

The 2006 CDC guideline4



Recommend HIV screening all patients ages 13-24 years if prevalence > 1/1000
Two sites describe improved screening rates in patients ages 13-64 years5

8% to 53% (New York)

3% to 17% (Louisiana)
No published studies describe improvement in adolescent screening rates
2Spiegel
1AAP,
Pediatrics 2011
H, Current HIV/AIDS Reports 2009
3CDC, MMWR 2011
4MMRW 2006
5Lin X, et.al, MMRW 2014
Nature and Severity of Local
Problem

In Philadelphia, the incidence of HIV is FIVE TIMES the national
average exceeding 1/1000

Prior to 2011, HIV screening at St. Christopher’s Hospital for Children

Risk based

Inconsistent

Not always documented
The Improvement Model
Aim Statement
What are we trying to accomplish?
How will we know that a change is
an improvement?
What changes can we make that will result in
improvement?
Act
Plan
Study
Do
Langley et al The Improvement Guide 2009
Establish Clear Goals

Global Aim

Specific (SMART) Aim
Global Aim

Improve HIV Screening Rates in Adolescents
“SMART” Aim Statement
Specific
Measurable
Action-oriented
Realistic
Time-bound
Specific SMART Aim

By June 30, 2013, 70% of patients aged 13 years and
above presenting to the SCHC CCAH ambulatory clinic
will have a documented annual HIV screen
The Improvement Model
What are we trying to accomplish?
Measure
How will we know that a change is
an improvement?
What changes can we make that will result in
improvement?
Act
Plan
Study
Do
Langley et al The Improvement Guide 2009
Measurement for QI

You can’t improve what you can’t (or don’t) measure

Measures tell a team if the changes they make are making a
difference

Should speed improvement, not slow it down

Measurement is not the goal
Definitions

Process Measure: whether an activity has been accomplished (i.e. was
PDSA cycle carried out as planned)

Outcome Measure: relate directly to aim; offer evidence that changes
are having an impact at the system level

Balancing Measure: make sure that other important measure does not fall
off
Langley, et.al, The Improvement Guide, 2nd edition
Process Measure Definition:
HIV Screening in Adolescents in Ambulatory Clinic

Numerator: # patients age >/= 13 years for whom HIV screening test performed
within the past 12 months in the Ambulatory Clinic
Denominator: # patients age >/= 13 years who present to the Ambulatory Clinic

Reported monthly
Other measures (not specified in example)

Outcome Measure
Examples would be:


% of newly-diagnosed patients presenting with AIDS

% of newly-diagnosed patients presenting with primary HIV infection
Balancing Measure (example would be rate of pregnancy testing in
female adolescents presenting to Ambulatory Clinic)
Primary and Secondary Study
Question
Primary Study Question

Can we design and implement interventions that reliably improve
HIV screening in adolescent patients in the ambulatory clinic?
Secondary Study Question (not stated)
METHODS
Methods

Ethical considerations

Setting

Planning the Intervention

Planning the Study of the Intervention

Methods of Evaluation

Analysis
Setting

Describes local context and local processes
Planning the Intervention

Process Map of local process

Key Driver Diagram
Key Driver Diagram
Design Changes/
Interventions
Key Drivers
Aim
An effective tool for breaking down complex questions or improvement goals and
structuring them into smaller, more-focused “drivers”
Each Key Driver

Focused

Answers the question, “What has to go right in order to accomplish
our aim?”

Stated in the affirmative
Key Driver Diagram
Key Drivers
Aim
By June 30, 2013, 70%
of patients aged 13
and above presenting
to the pediatric
ambulatory and acute
care clinics at SCHC
will have
documentation of a
HIV screen
Immunology provides
the tests to each clinic
and is available for
each positive test
Nurses screen patients
and administer test
Patients accept testing
Residents screen
patients and orders
test/reviews form
Administration supports
program
Design Changes/
Interventions
Immunology monthly check-in with
each clinic
Provide resident/nurse incentives
Nurse managers buy-in to testing
Screening barriers elicited Education
re: screening barriers
HIV screening process defined/shared
Screening incentives provided
Staff signs HIV consent form/test
ordered
Patient fills data form in clinic
Acute Retroviral curriculum established
Screening barriers elicited Education
re: barriers
HIV screening process defined/shared
Frequent poster/email reminders
Screening incentives provided
Administration buy-in to policy
change.
Routinely screen patients >/= 13 years
Plan-Do-Study-Act cycle ramp
The Improvement Model
What are we trying to accomplish?
How will we know that a change is
an improvement?
Interventions
What changes can we make that will result in
improvement?
Act
Plan
Study
Do
Langley et al The Improvement Guide 2009
Standardized Process for HIV Screening
NO
NO
Screening
opportunity
missed
Resident
screens
patient,
fills out
form and
tasks
nurse
HIV test
offered
by nurse
YES
Test
performed
Results
reviewed
with patient
YES
Patient
fills out
data
form
NO
Screening
opportunity
missed
Intervention: Reminder Cards
Table of Interventions (examples)
Category of
Intervention
Specific Intervention
Setting
Date Initiated
Identifying Risks and Collaboration with front-line staff
to identify barriers to testing
Barriers
Primary Pediatrics Clinic
Emergency Department
April 2013
April 2013
Implementing
Resources
HIV Coordinator hired
60-second HIV test implemented
Ambulatory Clinic
Emergency Department
March 2012
Education
Peer-to-peer review of screening
process
Nursing/MA Education on HIV
testing kit
Retroviral Curriculum for Residents
Ambulatory Clinic
August 2012
Ambulatory Clinic
September 2012
Hospital-wide
January 2013
Incentives
Nursing Incentives
Resident/Nursing Incentives
Ambulatory Clinic
Hospital-wide
May 2012
March 2013
Provide Reminders
Resident reminders
Ambulatory Clinic
December 2012
Survey/Evaluation
Residents, Nurses, Medical
Assistants survey of knowledge,
attitudes, practices
Ambulatory Clinic
March 2013
Policy
Medical Executive Committee
Approval
Hospital-wide
March 2, 2013
Table of Interventions
Planning the Study of the Intervention

Study Design
Observational time series
with multiple planned sequential interventions
Planning the Study of the Intervention

Methods to Ensure Internal Validity of Data
 Improve
documentation of screening tests performed
 (Compare
to laboratory administrative data)
Planning the Study of the Intervention

Methods to Promote External Validity/Generalizability
RELIABILITY
Level 1:
Straw
Level 2:
Wood
Level 3:
Brick
LEVEL 1 RELIABILITY
Education
Training
Feedback
Try Harder
LEVEL 2 RELIABILITY
Standardize
process
Redundancy
Decision aids
LEVEL 3 RELIABILITY
Alter habits
Real-time
Review
Force-function
Planning the Study of the Intervention

Methods to Promote External Validity/Generalizability
 Level
1 reliability interventions
 Reminders
 Education
 Incentives
 Level
2 reliability interventions
 Standardize
 Policy
 Level
screening process
change
3 reliability interventions
 Alter
everyday habits
Methods of Evaluation

HIV coordinator tracks testing via outpatient clinic form

# tests performed

patient age

location of testing
Analysis

Statistical Process Control Charts created using QI Charts©1

Rules for Detecting Special Cause Variation2 were used to
differentiate special cause from common cause variation
2Shewhart
1QI
Charts©, Scoville Associates 2009
WA. The Economic Control of Quality of Manufactured Product 1931
Results
Results

Outcomes
Results

Outcomes

During the study period (January 2012 – April 2013)

Rate of HIV screening in Ambulatory Clinic improved from 34% to 84%

3 new patients were diagnosed with HIV

1 previously known HIV+ patient who had been lost-to-follow-up was reintroduced to specialty care
Statistical Process Control
Statistical Process Control

Powerful tool for quality improvement projects

Foundation in theory of variation

Shows when changes are occurring due to Special Cause
Variation v. Common Cause Variation
Involves Control Limits and tests of change

Plot Data in Run Order

Calculate the Center Line

Calculate Control Limits
• Average rate +/- 3*standard deviation
Langley G, The Improvement Guide 2009
100%
Incentives provided to residents
Ambulatory Screening
Rates
90%
Change in policy: age of screening
lowered from 15 to 13 years
80%
70%
Incentives provided
to nursing staff
60%
50%
40%
Nursing staff
education
30%
Resident education and approval
to screen in Acute Care as well
20%
10%
4/1/13
3/1/13
2/1/13
1/1/13
12/1/12
11/1/12
10/1/12
9/1/12
8/1/12
7/1/12
6/1/12
5/1/12
4/1/12
3/1/12
2/1/12
1/1/12
0%
Understanding Variation

Common Cause Variation: variation expected within a given
system


Example: The high temperature in Philadelphia in March (variation from
-3 degrees to 60 degrees Fahrenheit may be expected)
Special Cause Variation: variation beyond what is expected in a
given system
Rules for Detecting Special Cause

A single point outside the control limits

8 consecutive points on one side of the mean

6 consecutive points increasing or decreasing

2 of 3 consecutive points in the outer 1/3 approaching the
control limit
Shewhart WA. The Economic Control of Quality of Manufactured Product 1931
100%
Incentives provided to residents
Ambulatory Screening
Rates
90%
Change in policy: age of screening
lowered from 15 to 13 years
80%
70%
Incentives provided
to nursing staff
60%
50%
40%
Nursing staff
education
30%
Resident education and approval
to screen in Acute Care as well
20%
10%
4/1/13
3/1/13
2/1/13
1/1/13
12/1/12
11/1/12
10/1/12
9/1/12
8/1/12
7/1/12
6/1/12
5/1/12
4/1/12
3/1/12
2/1/12
1/1/12
0%
HIV Screening Rates: Patients >/= 13 years by Month (Jan 2012 – April 2013)
Overall HIV screening rates
30%
25%
20%
15%
10%
4/…
3/…
2/…
1/…
2/…
3/…
4/…
1/…
2/…
3/…
4/…
1…
1…
1…
9/…
8/…
7/…
6/…
12…
11…
10…
9/…
8/…
7/…
6/…
5/…
4/…
3/…
2/…
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
1/…
4/…
3/…
2/…
1/…
Primary Peds Screening Rates
12…
11…
10…
9/…
8/…
7/…
6/…
5/…
4/…
3/…
2/…
1/…
Ambulatory Screening Rates
5/…
3/…
2/…
1/…
4/…
3/…
2/…
0%
1/…
0%
1…
5%
1…
5%
1…
10%
9/…
10%
8/…
15%
7/…
15%
6/…
20%
5/…
20%
4/…
25%
3/…
25%
2/…
30%
1/…
30%
4/…
Inpatient Screening Rates
ED screening rate
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
1/…
1…
1…
1…
9/…
8/…
7/…
6/…
5/…
4/…
3/…
2/…
0%
1/…
5%
Discussion
Discussion

Summary

Relation to other evidence

Limitations

Interpretation

Conclusion
Summary

Keys to Success
 Inter-professional
 Front-line
collaboration
staff engagement
 Resident
Leadership
 Broadcasting
 Peer-to-Peer
‘Great Catches’
Model of Education
Relation to other evidence

Recently published study (June 2014 MMWR)
 Improved
screening rates in NY and LA in patients ages 13-64 years
 Still
well-short of universal screening
 Did
not specifically address adolescent HIV screening
Limitations

Internal Validity
 Data
not cross-referenced with laboratory administrative data
 Possible
that testing in ED is not documented
Limitations

External Validity/Generalizability
 Local
Context
Limitations

External Validity/Generalizability
 Dedicated
 The
HIV coordinator (funded via Gilead)
Dorothy Mann Center (funded via Ryan White Fund)

6 physicians (1 immunologist, 3 infectious disease specialists, 1 general pediatrician, 1 psychiatrist)

1 physician’s assistant

1 nurse practitioner

1 licensed practicing nurse

2 social workers

2 case managers
 Local
QI Culture
Interpretation
Next Steps

Utilize technology (in EHR) to facilitate screening
 Decision
aids
 Pre-checked

orders for screening
Spread this intervention through a multisite collaborative
Conclusion
Implications for future study of improvement interventions

Links between routine screening
 Earlier

detection/treatment
 Decreased
rate of presenting with advanced HIV infection
 Decreased
high-risk behaviors
 Decreased
transmission of HIV
 Decreased
incidence of HIV in adolescents in one community
 Decreased
healthcare costs
A multisite collaborative may accelerate improvement
Other

Funding
Other

Funding

Grant from Gilead


Rapid HIV tests

HIV tester/data collector
Dorothy Mann Center is funded by Part A and Part B

Ryan White HIV/AIDS Treatment Modernization Act

AIDS Activities Coordinating Office in Philadelphia
Examples
Examples
Examples
Example: Abstract Submission accepted as Platform Presentation

Title: Effect of a Newborn Access Program on Third Next Available Appointment and No-Show Rates in
an urban, underserved academic healthcare center
Katie E McPeak, M.D.1, Deborah A Sandrock, M.D.1, David Cooperberg, M.D.1, Selima N ShulerJenkins1, Bruce A Bernstein, PhD.1 and Lee M Pachter, D.O.1. 1Section of General Pediatrics, St.
Christopher's Hospital for Children, Philadelphia, PA, United States.

Background: Timely access (72 hrs per AAP) to post-discharge newborn (NB) care can be
challenging. Prior to our intervention, NBs in our center were scheduled into resident continuity clinics.
NB scheduling problems resulted in overbooking, high no-shows, and delayed access to care.

Objective: To decrease time to third next available appointments (TNAA) by 50% within 6 months.

Design/Methods: A multidisciplinary team was formed, consisting of practice leadership, clinical and
non-clinical staff. Utilizing planned sequential interventions, this team developed and piloted a NB
clinic with a new scheduling model, aiming to have NBs seen in a single point of entry to care. TNAA is
calculated weekly utilizing methods published on www.ihi.org (Institute for Healthcare Improvement).
No-show rates were extracted via Next Gen EPM. We used QI Charts© to create statistical process
control charts. The API rules for detecting special cause variation were applied.

Results: TNAA decreased by 78% within 2 months. No-show rates decreased after intervention (9.6%
vs. 12.2%, p=0.05). An additional 63 NBs were seen in 6 months, at an average charge of $145/visit.
This amounts to a return on investment of roughly $18,000/year.

Conclusions: Timely access to NB care was achieved, aligning our center's aim with AAP
recommendations. Keys to success were: a dedicated NB team, warm-call reminders, and a supplydemand scheduling model. Significant reduction in TNAA was accomplished. While temporally
related, this intervention does not demonstrate causality. Next steps are to measure sustainability and
replicate this model at other centers. Future outcome measures include breastfeeding sustainability
and re-admission rates for jaundice and breastfeeding issues.
Results
#scheduled
newborn
appointments
# kept newborn
appointments
# no-show
newborn
appointments
% No show rate
(#no
show/#scheduled)
Pre-Intervention
1128
860
137
12.15%
Post-Intervention
1132
923
109
9.63%
Delta (change) Post
vs. Pre
Implementation
+4
+63
-28
-2.52%
Time to Third Next Available Appointments
Examples of Published QI Reports

Brady PW, Muething S, Kotagal U, et.al, Improving situation awareness
to reduce unrecognized clinical deterioration and serious safety events.
Pediatrics. 2013;131(1):e298-308.

Volpe D, Harrison S, Damian F, et.al, Improving timeliness of antibiotic
delivery for patients with fever and suspected neutropenia in a
pediatric emergency department. Pediatrics. 2012;130(1):e201-10.

Fischer D, Cochran KM, Provost LP, et.al, Reducing central lineassociated bloodstream infections in North Carolina NICUs. Pediatrics.
2013;132(6):e1664-71.
