Mary Davis, DrPH, MSPH; NC Institute for Public Health
Kristin Adams, Ph.D., CHES; Indiana State Department
of Health



Performance measurement is the “regular
collection and reporting of data to track work
produced and results achieved”
Performance measure is “the specific
quantitative representation of capacity,
process, or outcome deemed relevant to the
assessment of performance”
Performance measurement is NOT
punishment
Infrastructure
Process
Coordination
Results
 Leadership &
Decision making
 Boards of Health
 Policies &
Procedures
 Enforcement/
Investigation
Protocols
 Financial
management
 Information &
reporting
 Set of Core
Indicators
 Program
performance
goals/ objectives
 Assessment
processes
 CD investigation
case write-ups
 EH enforcement
action and case
files
 Education
sessions
 Work of
community
groups and
coalitions
 Communication
mechanisms
 Public and
private work on
access to services
 Hand-offs
between local
health and state
programs
 Program
evaluation
results
 Key
indicator
outcomes
(CD/EH/PP)
 Financial
performance
Monitoring the results of performance review
against PHAB Standards and Program Evaluation
Monitoring
Indicators
and
Outcomes
Dr. Mary Davis is a Senior Investigator at
the NC Institute for Public Health and
Adjunct Assistant Professor in the
Department of Health Behavior, both part
of the UNC, Gillings School of Global Public
Health. Since Dr. Davis conducts public
health systems and services research and
program evaluations using participatory
approaches; specialty areas are
accreditation, quality improvement, and
public health partnerships.
Her experiences in these areas have been shaped by participating in the
Multi-State Learning Collaborative, evaluation of the NC Local Health
Department Accreditation program, and working with the NNPHI and CDC
to evaluation the National Public Health Improvement Initiative. Dr. Davis
earned her doctorate from the Johns Hopkins University School of Public
Health and her masters in science in public health from the University of
Alabama at Birmingham.
Dr. Kristin Adams is the Director of the Office of Public
Health Performance Management at the Indiana State
Department of Health (ISDH). She oversees agency
performance management, quality improvement, and future
public health accreditation efforts. She was the Principal
Investigator in Indiana for the Multi-State Learning
Collaborative grant. She is currently the Principal
Investigator for the National Public Health Improvement Initiative
cooperative agreement from CDC.
Prior to joining ISDH, Dr. Adams worked for a national non-profit agency
focusing on program evaluation and training for health programs targeted
toward girls and young women. Dr. Adams has taught numerous health
education courses and has presented at national and international
conferences. Dr. Adams holds a BS and MA in School Health Education from
Indiana State University, a Ph.D. in Health Education from Southern Illinois
University Carbondale, and is a Certified Health Education Specialist.
Performance
Measurement:
Nuts and Bolts
Basics
Mary V. Davis, DrPH, MSPH
Open Forum for Quality Improvement
November 20, 2013
Outline
• What is a performance measure?
• Where are performance measures in a
performance management system?
• What should be in place before establishing
measures?
• What’s a system of work?
• What are the characteristics of a useful process
measure?
• How do you go about establishing useful
process measures?
What is a Performance Measure?
• Performance Measurement is regular collection
and reporting of data to track work produced and
results achieved.
• Performance Measure is quantitative
representation of
– Capacity
– Process
– Outcome
Where are Performance Measures in a
Performance Management System?
YOU ARE HERE!
Standards and Measures: PHAB Example
• Standard 9.1: Use a performance
management system to monitor
achievement of organizational objectives.
– Measure 9.1.2 A Implement a performance
management system.
– Use a process to determine and report on
achievement of goals, objectives, and
measures set by the performance
management system.
Results help to inform the planning process
Strategic Planning
and
Process Improvement
Opportunities
Improvement
Opportunities
Mission and Values
Goals and Objectives
Strategies
Vision
Improvement
Opportunities
Performance Measures
and Operational Indicators
Operational/
Business Plans
Quality Improvement Projects
Performance Monitoring and Analysis
Washington State Department of Health Example:
Results and
Customer, Stakeholder, Partner Feedback
Performance Monitoring and Analysis
Line of Sight Framework
Restaurant
inspections
conducted
and reports
produced
Work Process
Measure
restaurants
achieving
acceptable
sanitation
grades can
post grades
Process
Outcome
Public uses
restaurants
with
acceptable
sanitation
grades
Short term
Outcome
Reduction in
food-borne
illnesses and
disease
outbreaks
Long term
outcome
What is a System of Work?
Factory: How?
Widget: What?
Customers: Who?
Outcomes: Why?
Focus on the Widgets
• What’s a Widget?
Widgets
•
•
•
•
Are Things
Are Deliverables
Can be Counted
Are Specific
Common Public Health Agency Widgets
•
•
•
•
•
Environmental Permits, Notices of Violations
Rules and regulations
Vaccinations
Epidemiology Reports
Infrastructure
– Filled Vacancies
– Contracts
What Widgets Do You Produce?
• Take a minute: write down 5 widgets you
produce
• What examples do you have to share?
Measures: General Considerations
•
•
•
•
Measures reveal our values
Measures drive behavior
Measures can inspire us
Measures can help us learn
Useful Measures: Specifics
• Meaningful
• Focused on customer
needs
• Accurate
• Valid
• Reliable
• Responsive
• Functional
• Credible
• Simple enough to be
understood
• Available
• Abuse-Proof
• Cost effective to
collect and report
• Comparable data
over time
SMART Measures
•
•
•
•
•
Specific
Measurable
Aggressive
Results-Oriented
Time-bound
Developing Measures
• Talk with customers and those involved in the
process
• Identify which process measures
–
–
–
–
Are of value to customers and producers
Will drive behavior
Will inspire
Will promote learning
• Determine how process measures fit into other
measurement processes in the agency
(performance standards, strategic plan)
What to Include in a Measure
•
•
•
•
•
Statement of the Measure
Target (population, process)
Numerator and denominator
Target goal
Who will collect the information
What to Include in a Measure
•
•
•
•
Form or tool to collect information
How often data will be collected
Who will conduct data analysis
How often will the data be reported
Data Description Form
Performance measure:
Target population:
Numerator:
Denominator:
Which are you using—a target or
benchmark?
What is the target/benchmark?
SMART objective:
Source of data:
Who will collect the information?
How often will the data be analyzed?
How often , by what mechanism and
who reports these data and analysis?
Baseline measurement data and
date(s):
Definitions, such as accuracy and
validity, and other comments:
MarMason Consulting
Resources
• “We Don’t Make Widgets” Ken Miller
• Turning Point Resources: Guidebook for
Performance Measurement
• Marni Mason tools and presentations
• Washington State Department of Health
QI Plan
Thank You!
Mary V. Davis, DrPH, MSPH
mary_davis@unc.edu
Kristin A. Adams, Ph.D., CHES

Ask a series of questions:
◦ Who is your customer
◦ What do they care about
◦ What is the purpose of your funding/mission/vision
of the program area
◦ Does this goal and objective tie to the strategic plan
 Does your program have a role in the key priorities of
the agency
◦ Can you measure it
 Is it meaningful???

Several areas of resources to determine
measures:
◦
◦
◦
◦
◦
State law/code
Program requirements
Healthy People 2020
Historical data of the program
Benchmarks by national groups (i.e., March of
Dimes)

Develop the key outcome measure

Identify key activities to get you to making a
change in the measure
◦
◦
◦
◦
Decrease Smoking Rates
Decrease Obesity Rates
Increase Immunization Rates
Decrease the late payment fees
◦ Increase number of people receiving coaching from
quitline
◦ Increase the complete streets communities throughout
the state
◦ Reminder recalls--% of those who become compliant due
to the reminder notice
◦ Decrease invoice wait time; increase invoice processing
time
Sometimes it is all about
Process





Ensure your epidemiologists are involved
Ensure the experts to the program are
involved
If you are doing a “data pull” ensure the
IT/database manager is involved
Involve Executive Leadership
Do a “mass” education campaign