GETTING YOUR MONEY’S WORTH:
RETURN ON INVESTMENT
Craig Thomas, CDC-OSTLTS
Greg Randolph, North Carolina Center for Public Health Quality
Brynn Riley, Maine Department of Health
Measuring the Impact of Improvement:
Lessons from the National Public Health Improvement
Initiative (NPHII)
Craig Thomas, PhD
Division of Public Health Performance Improvement
Office for State, Tribal, Local and Territorial Support
Centers for Disease Control and Prevention
Open Forum for Quality Improvement
June 20, 2012
Office for State, Tribal, Local, and Territorial Support
Division of Public Health Performance Improvement
National Public Health Improvement Initiative
Five year initiative for health departments
 Adopt and institutionalize cross-cutting performance
management and quality improvement methods
 Improve accountability, efficiency, and effectiveness of public
health programs and services
 Advance health departments’ readiness for public health
accreditation
NPHII Return On Investment Workgroup

Why is ROI for improvement important?
 Greater accountability in times of scare resources
 Stronger evidence base for what works to improve business
practices and service delivery for greater health impact
 Promotes a business culture of improvement in public health

Purposes:
 Identify practical approaches to measuring the costs and
benefits of improvement with select NPHII grantees
 Provide technical assistance as requested
 Develop and share ROI methods, tools, and approaches
Analytic Approaches to ROI

Cost Benefit Analysis
 QI methods on health outcomes is important from societal
perspective
 Not always feasible

Budget Impact Analysis
 Offers practical approach to assessment of the costs and
benefits of QI methods
• Calculates the cost and benefits of improving business processes
and service delivery
• Focused on efficiency and effectiveness outcomes
• Documents both tangible and intangible benefits of QI
Key QI Outcome Measures

Efficiency: Doing something well with the least
amount of waste
 Time saved
 Money saved
 Reduced steps in a process

Effectiveness: Achieving a purpose or desired result




Quality enhancement (e.g., completeness and accuracy of data)
Increased customer service satisfaction
Increased reach of services
Implementation of evidence-based practices
Other QI Outcome Measures

Improvements that relate to both efficiency and
effectiveness:




Consolidation and sharing of services
Leverage of additional funds / revenue streams
Integration of networks
Pilot testing and evaluation of innovative ideas
Overview of ROI Methodology for QI

Planning:
 Develop objectives; ‘AIM’ Statement, measurement and data
collection plans

Data Collection:
 Establish baseline, collect cost data and outcomes before,
during, and after implementation of improvement effort

Data Analysis:
 Control for alternative hypotheses; convert hard date (e.g., time
and staff) to monetary value; calculate the ROI (net project
benefits/project costs) times 100

Identify Intangible Benefits:
 Increased customer satisfaction; improved buy-in and
organization support; greater credibility within the community
NPHII: Strategies to Increase Efficiencies
Out of 74 grantees, the following have prioritized strategies to
target efficiencies
Percent
Focus
70
Decreasing cost to deliver services
69
Decreasing time to deliver services
57.5
Decreasing time to award contracts
54
Decreasing staff allocation to deliver services
NPHII: Increasing Efficiencies through QI

Early examples in saving time and money
 Lab data reporting to CDC: Reduced lag time from 14-21 days to
2–3 days
 IT costs: Identified strategies to save over $1.2 million annually
NPHII: Strategies to Improve Effectiveness
Out of 74 grantees, the following are prioritizing strategies to
strengthen effectiveness
Percent
Focus
94.4
Using data to prioritize activities for maximum impact
94.5
Identifying and implementing evidence-based
practices
92
Engaging in cross-jurisdictional partnerships
93
Using an organization-wide performance
management system
NPHII: Improving Effectiveness thru QI

Early successes achieved by some grantees
 Increased enrollment in Medicaid family planning program by
109 percent in one year
 Streamlined clinic flow of family planning visits to improve
patients’ experience
 Increased referrals of pregnant women in WIC to smoking
cessation programs
Resources
Phillips, J., & Phillips, P. (2007). Show me the Money: The Use of
ROI in Performance Improvement, Part 1. Performance
Improvement; V. 46, No 9.
OSTLTS Performance Management / Quality Improvement
Resources:
http://www.cdc.gov/stltpublichealth/Performance/index.html
For more information please contact Centers for Disease Control and
Prevention
1600 Clifton Road NE, Atlanta, GA 30333
Telephone, 1-800-CDC-INFO (232-4636)/TTY: 1-888-232-6348
E-mail: cdcinfo@cdc.gov Web: www.cdc.gov
The findings and conclusions in this report are those of the authors and do not necessarily represent the official
position of the Centers for Disease Control and Prevention.
Office for State, Tribal, Local and Territorial Support
Division of Public Health Performance Improvement
Application of ROI to Quality
Improvement Projects
Greg Randolph MD, MPH
NC Center for Public Health Quality
Why is ROI Important?
• Need to improve performance given substantial
burden of population health issues and health
disparities in US
• PH agencies are increasingly applying QI methods
to their work – involves a substantial upfront
investment
• Sound business practices can help in a time of
fiscal scrutiny and dwindling budgets/resources
How We Measure and Use ROI…
• ROI Model
• Template linked to aim statement’s
measurable goals (see handout)
• Communications
ROI Process Model
Planning
Data
Collection
Develop AIM
and
Measurement
Plan
-----------------Think about
Potential
Business
Impact
Before
Improvements
(Baseline)
-----------------After
Improvements
------------------Capture costs
throughout
Data Analysis
More Clues
Waste
identified via
VSM and
observation
-----------------QI Tools used
for
improvements
Convert
Measurement
and Benefits
Data to
Monetary
Values
Calculate ROI
Using Cost and
Benefit Values
Identify
Intangible
Benefits
Reporting
Generate Report with
Conclusions and
Lessons Learned
Lou Anne Crawley-Stout, MBA, CLSSBB, PMP
Adapted from “ROI at Work” by Jack J. Phillips and Patricia P. Phillips
Communicate Results
to Target Stakeholders
17
Terminology and Formula
ROI (return on investment): A performance measure used to evaluate the
efficiency of an investment.
ROI = Benefits-Costs/Costs
EI (economic impact): Refers to costs and benefits of an activity.
EI = Benefits-Costs
Example: NC Institute for Public Health (NCIPH)
Project Benefits (annualized savings)
Archiving (paper/printing)
$ 360
Rent elimination
1490
Labor/time savings
16330
Overtime reduction
2300
$20480
Project Costs
Staff time/travel
Archiving supplies
$ 9850
200
$10050
ROI: $20480-10050/10050 = $1.04
For every $1 invested in QI, NCIPH received in return $1.04 after costs.
EI: $20480-10050 = $10430
18
Communication Tips
• Do stakeholder analysis early and plan for
when/how/what will communicate to stakeholders
– customize to specific stakeholder audience
– Since ROI relatively new for public health, best to report faceto-face
• Stress that ROI data should not be used to reduce
budgets further
– by cutting budget of organization generating substantial ROI,
would starving most innovative and effective organizations and
halt spread of use of ROI
• Communicate how ROI illustrates you are better able to
achieve your mission and other intangibles
Lessons Learned
• “Begin with the end in mind”
– Begin with your aim statement
– Capture tangibles and intangibles throughout
• When converting tangible benefits into
monetary values, be conservative
– Vet with your team and selected stakeholders
• Intangible Benefits (non-monetary benefits)
are extremely valuable
Resources
Additional ROI Results in Jan / Feb
2012 issue of Journal of Public
Health Management & Practice
“Applying Lean Principles and Kaizen
Rapid Improvement Events in Public
Health Practice”
http://journals.lww.com/jphmp/toc/2012/01000
21
Resources
• ROI at Work: Best-Practice Case Studies from
the Real World; Jack J. Phillips and Patricia
Pulliam Phillips
• www.ncpublichealthquality.org
• www.ies.ncsu.edu/
Greg Randolph: greg.randolph@ncphf.org
Brynn Riley
Performance Improvement Manager, Maine
June 20, 2012


Quality Improvement (QI) teams have an
opportunity to show projects’ return on
investment (ROI) to stakeholders
QI teams should consider the:
upfront investment before applying QI methods
costs and benefits throughout the project’s
duration.

Accounting for costs and benefits will
 Help improve the organization’s case for undertaking a QI
project
 Help prioritize competing interests
 Help account for community or societal benefits where data
doesn’t exist

Include intangible benefits, which are a significant
and growing part of an organization’s and
community’s worth.
Developed by B. Riley - Brynn.Riley@maine.gov and J. Moran – jmoran@phf.org
6/10/12

Consider how it translates into public health and
community value.
 Part of public health is preventing adverse health
outcomes for a population, essentially creating “nonevents”.
 Including intangible benefits in cost benefit calculations
account for the organizations added value to the
community that may not otherwise be included.

Concerns:
 Intangible benefits cannot be easily measured in dollar
terms
 The benefits may not be viewed as valid by some people
Developed by B. Riley - Brynn.Riley@maine.gov and J. Moran – jmoran@phf.org
6/10/12

Use the AIM statement as a springboard to
hone the a QI Project’s focus and evaluate
return on investment.
 The Aim is sometimes called the Charter or Opportunity
Statement. It is a goal.
 A written discrete, measurable, and time-bound
description.
 Includes the scope of an improvement effort, as viewed
by the organization and the team/individual striving for
improvement.
 It should be developed with input from Senior
Leadership to ensure support for the Team and
alignment with the strategic goals of the organization.
Developed by B. Riley - Brynn.Riley@maine.gov and J. Moran – jmoran@phf.org
6/10/12
 What
will improve?
 When
 How
 For
will it improve?
much will it improve?
whom will it improve?
Developed by B. Riley - Brynn.Riley@maine.gov and J. Moran – jmoran@phf.org
6/10/12



When developing the AIM statement stay at a
strategic level - try not to get too operational.
Determine baseline measurements while
framing the AIM statement
Outline the expected tangible and intangible
benefits
Developed by B. Riley - Brynn.Riley@maine.gov and J. Moran – jmoran@phf.org
6/10/12

AIM statements are not written once and set in cement.

A regular team task is to review the aim statement
 "Remember, we’re here to increase immunizations by 50% within
12 months,"
 How are we doing?
 Regular review of progress quantitatively

It is acceptable to tweak the aim, as the team learns more
about their work or new information changes the team’s
direction.
Developed by B. Riley - Brynn.Riley@maine.gov and J. Moran – jmoran@phf.org
6/10/12

Section 1: Problem Description, Boundary, and Team Composition

Describe the problem or opportunity to be addressed:
◦ Example: Physician/Immunizing Stakeholders are not utilizing the Immunization
Information System (IIS) tools provided, resulting in lower than acceptable immunization
rates.

This process is important to work on now because of: (describe the impact this
problem or opportunity is having on the agency, program, customers,
employees, or the community
Example: Not utilizing the provided public health improvement tools in the IIS results in
improperly immunized people, wastage of State supplied vaccine, client susceptibility to
disease, and community risk of disease.

Team Sponsor: _______________________

Team Leader: _________________________

Team Members: ______________________
Area of Expertise: ________________
Developed by B. Riley - Brynn.Riley@maine.gov and J. Moran – jmoran@phf.org
6/10/12


Section 1: Problem Description, Boundary, and Team Composition
The problem or opportunity starts with… and ends with…
For Example:
 The problem or opportunity starts with the lack of utilization of IIS tools* and
with the assignment of staff to conceptualize and recommend an improvement
plan.

…and ends with the publishing of a results document that demonstrates an
increase in immunization rates via the developed methodologies and that also
quantifies the cost of the improvement effort in order to identify the cost
associated with sustainability

IIS* = Immunization Information System (ImmPact), Maine’s tracking system
Developed by B. Riley - Brynn.Riley@maine.gov and J. Moran – jmoran@phf.org
6/10/12
Section 2: Internal and External Benefit and Cost Description:
This opportunity has the following estimated potential benefits internally and to
the external community:
Internal
and
External
Tangible Benefits (cost benefit)
Intangible Benefits
Are benefits produced by an
investment that is
immediately obvious and
measurable. (i.e. IT software)
Are indirect benefits
which cannot be felt or
touched - such as
improved health access,
staff morale, or the
image of the health
department by
stakeholders.
A tangible benefit can usually
be expressed in terms of a
monetary savings.
Developed by B. Riley - Brynn.Riley@maine.gov and J. Moran – jmoran@phf.org
6/10/12
Section 2: Internal and External Benefit and Cost Description:
What is unique to many public health organizations is the need for a way to
account for intangible benefits to the community being served.
For example, tangible benefits may be a decrease in the smoking rate, an
increase in the immunization rate, reduced infant mortality, etc.
 An intangible benefit may be a program that contributes to increased selfesteem resulting in greater use of preventive services and adherence to an
asthma management plan.
Developed by B. Riley - Brynn.Riley@maine.gov and J. Moran – jmoran@phf.org
6/10/12
Section 2: Internal and External Benefit and Cost Description:
This opportunity has the following estimated potential benefits internally and to
the external community:
Internal:
Tangible Benefits (cost benefit)
Intangible Benefits




External:
Reduced vaccine wastage
Increased number of doses
recorded within the IIS*
Increased Immunization Rates


Increased Stakeholder
awareness of the IIS*
Increased Stakeholder
satisfaction with the IIS*
Increased use of IIS tools
Tangible Benefits (cost benefit)
Intangible Benefits





Increased Client Visits
Increased Services Rendered
Increased Billing Activities
Increased Immunization Rates


Additional opportunity to
serve the client
Increased Community
Marketing Opportunity
Improved Assessment Rates
IIS = Immunization Information System (ImmPact), Maine’s tracking system
Developed by B. Riley - Brynn.Riley@maine.gov and J. Moran – jmoran@phf.org
6/10/12
Some questions to consider are:
 If we implement a solution to this problem what are the potential cost
categories that will be impacted and by how much?
 How much will it cost to develop a solution and implement it?
 What is the cost of doing nothing?
 What is the cost of sustaining the solution?
 Are there possible unintended consequences, and if so, what are the
costs?
 If we plan to use existing resources, what will we have to stop doing to
divert resources to this problem/solution?
Developed by B. Riley - Brynn.Riley@maine.gov and J. Moran – jmoran@phf.org
6/10/12




Section 3: Current State Performance and Desired Future State
Describe the current state of the problem or opportunity and its current
performance (baseline data):
◦ External Stakeholders are not using the integrated Public Health
Improvement tools resident in the IIS for their use. These tools include,
but are not limited to;
 Immunization Coverage Reports (ICR),
 Reminder/Recall Letters (R/R),
 Client Vaccine Forecasting (ACIP Forecasting)
Describe the ideal future state:
What are the key driving and restraining forces impacting the current state
and the ideal future state?
Developed by B. Riley - Brynn.Riley@maine.gov and J. Moran – jmoran@phf.org
6/10/12
Section 4: Improvement Description:

This effort should improve the current state by:

Describe the improvement goals to be achieved:

The timing of these improvements:

We will measure improvements by:
Developed by B. Riley - Brynn.Riley@maine.gov and J. Moran – jmoran@phf.org
6/10/12
Section 5: Internal and External Customer Identification:

For the following customers/clients (customers, staff or those

For Example:
affected by the process under improvement)
 Maine Immunization Program Staff
 Consumer Services Staff
 IIS Support Staff
 Administrative Stuff
 Physician’s Office
 Nurse Staff
 Administrative Support
 Billing/Fiscal Support
Developed by B. Riley - Brynn.Riley@maine.gov and J. Moran – jmoran@phf.org
6/10/12
This presentation is based on the article:
Calculating the Real Value of Process
Improvement: Factoring in Intangible
Benefits. J. Moran and B. Riley, Process
Excellent Network, April 2012
Web Address when ready:
Process Excellence Network April 2012
http://www.processexcellencenetwork.com/people-performance-and-change-in-processimproveme/articles/calculating-the-real-value-of-processimprovement/&mac=SSIQ_OI_Featured_2011&utm_source=processexcellencenetwork.com&utm_m
edium=email&utm_campaign=HrOptIn&utm_content=4/12/12
Developed by B. Riley - Brynn.Riley@maine.gov and J. Moran – jmoran@phf.org
6/10/12