Evaluating Partnerships:
Why, When and How
February 17, 2015
B. J. Tatro, PhD
ASTDD, bj@bjtatro.com
Presented in Collaboration with the Centers
for Disease Control and Prevention
1
General Reminders
• This webinar will be recorded and archived on the
ASTDD website (including speaker notes)
• Questions will be addressed after the speakers are
finished. Please type your question into the “chatbox”
that will appear at the end of the webinar and then
click on the bubble to the right of where you type your
question to send it to the moderator
• Please respond to the polling questions at the
conclusion of the webinar
2
• This presentation was supported by
Cooperative Agreement 5U58DP004919-02
from CDC, Division of Oral Health. Its contents
are solely the responsibility of the authors and
do not necessarily represent the official views
of CDC
3
Desired Results
• State Oral Health Programs (SOHPs) have
increased awareness of the importance of
evaluating their partnerships and of engaging
their partners in the evaluation process
• SOHPs have increased awareness of the
importance of matching their partnership
evaluation questions and tools to their
partnership’s stage of development
• SOHPs are familiar with resources available to
them to inform their planning, implementation,
and use of partnership evaluation
4
Some Preliminary Thoughts
on Partnerships
• What are they?
– Nature of the partnership
– Purpose of the partnership
5
Some Preliminary Thoughts
on Partnerships
• Why do we need them?
• What does it take to develop, maintain, and
grow a successful partnership?
6
Some Preliminary Thoughts on
Evaluating Partnerships
• Why evaluate partnerships?
• How do we know if we are ready to evaluate a
partnership?
• How do we approach the design and
implementation of a partnership evaluation?
7
Step 1: Engage Stakeholders
• Identify who should be involved in the
evaluation
• Specify the roles they will play
• Think about how to engage these stakeholders
throughout the evaluation process
8
Step 2: Describe the Partnership
• Develop your logic model
9
State of Hawaii
Purpose = Develop & coordinate partnerships to improve oral health
Inputs 
Hawaii Oral Health Team (staff)
Internal Partners
Family Health Services Division Programs:

Office of Primary Care & Rural
Health

CSHCN

MCH (including MIECHV, Parenting
Support, Adolescent & Youth, etc.)

WIC
Other DOH:

Chronic Disease Management &
Control Branch (Asthma, Cancer,
Diabetes, Heart/Stroke, Obesity,
Tobacco)

Office of Planning, Policy & Program
Development (Preventive Health &
Health Services Block Grant)

Developmental Disabilities
External Partners
State Medicaid Office
Hawaii Federally Qualified Health Centers
(FQHC) & Rural Health Clinics (RHC)
Hawaii Chapter of the American Academy of
Pediatrics
Insurance providers
Legislators
Community representatives
CDC Funding & TA
ASTDD TA
Activities 
Outputs 
1. Identify priorities & establish, enhance, &
maintain a collaborative relationship with
internal DOH partners, with priority focus on
CDC-funded chronic disease programs,
CSHCN, & MCH
a. Contacts identified, documented
2. Identify priorities & establish, enhance, &
maintain a collaborative relationship with
external partners, with priority focus on
FQHCs & RHCs
d. Shared interests & common goals
identified, documented
For both internal & external partners:
a. Identify contacts
b. Conduct 1:1 outreach
c. Become familiar with each other’s
mission & activities
d. Identify shared interests & common
goals
e. Identify potential benefits of working
together
f. Identify options for collaborative
efforts
g. Make decision about whether & how
to collaborate
h. Establish partnership objectives &
develop work plans
i. Implement work plans
j. Evaluate collaborative strategies
f. Options identified, documented
b. Communication established, documented
c. Information shared, documented
e. Benefits identified, documented
g. Decision made, documented
h. Objectives established & work plans
developed
i. Work plans implemented, documented
j. Evaluation conducted, documented
Outcomes
Short Term
Ongoing communication & collaborative
action
Increased awareness of the relevance of oral
health to partners’ target populations
Increased awareness of the benefits of
prevention, early identification, & treatment
to overall health

Intermediate Term
Oral health addressed in partners’ burden
documents if applicable, activities,
messaging, plans, policies, &/or practices
Resources devoted to oral health expanded/
leveraged

Longer Term
Increased number & commitment of oral
health champions
Increased attention to prevention, early
identification, & treatment
Increased number of children have dental
sealants

Impact
Reduced dental caries, oral cancer,
periodontal disease
Improved health equity
Improved overall health
10
Step 2: Describe the Partnership
(continued)
• Identify what you need and what you have
11
Partnership Membership Assessment Tool
from Fundamentals of Evaluation Partnerships/Evaluation Guide (National Health Disease & Stroke Prevention Program)
Want
Have
Roles
Partnership roles
Leader
Committee leader
Want
Have
Skills/Expertise
Want
Have
Representation
Data analysis, worksites
State Emergency Services
Data analysis, healthcare
State Obesity Program
Reviewer, medical content
State Diabetes Program
Task leader
Writer
State Tobacco Program
Meeting planner
Advocate for stroke
State Epidemiology
Meeting facilitator
Advocate for heart disease
State Office of Minority Health
Strategic planner
Legislative advocate
Hospital Association
Medical expert
Primary Care Association
Training
Cardiologist
State Legislature/Policy Makers
Financial support
Neurologist
Schools (as worksites)
Content reviewer
Healthcare quality improvement
Community health clinics
Budget management
Nursing
Private insurers
Spokesperson
Pharmacy
Medicaid/Medicare
Funder
Media communications
Prevention Research Center
Champion, healthcare
Workplace wellness
Chambers of Commerce
Champion, public health
State policy change
Unions
Champion, worksites
Community policy change
Business coalition on health
Strategic implementer
Training for healthcare
State American Heart Association
Resource linker – connection to
groups with influence or resources
Evaluation
Disparate groups (race/ethnicity,
geographic, gender. SES. Etc.)
Communications
Marketing
12
Step 2: Describe the Partnership
(continued)
• Identify the stage of development of your
partnership
– Formation
– Building
– Maintenance
• Engage stakeholders
13
Step 3: Focus the Evaluation
Design
• Develop and prioritize evaluation questions
that are matched with your partnership’s
stage of development
• Engage stakeholders
14
Step 4: Gather Credible Evidence
• Complete your evaluation plan:
– Indicators
– Data Sources (tools)
– Methods of data collection
– Frequency of data collection
– Study design
– Implementation roles, responsibilities, and
timelines
• Engage stakeholders
15
State of Minnesota
Evaluation Question
Indicator/
Performance Measure
Method
Data Sources
Frequency
Responsibility
How were the opportunities
identified and leveraged to integrate
oral health efforts with other
programs that address related
chronic disease/risk factors?
# of meetings, campaigns,
summits organized
Meeting notes,
summit reports
review
Meeting notes and
summit reports
Annually
Program
coordinator/
evaluator
How many partnerships were
developed with a focus on prevention
interventions in a year?
# of partnerships that focus on
prevention intervention and that
provided a letter of support or MOA
Program reports
review
Program reports
Annually
Letters of support
Partnership database
Program
coordinator/
evaluator
How was the integration of oral health
into overall health
encouraged/institutionalized?
# of meetings with other
chronic disease partners
Meeting notes
and reports
review
Meeting notes and
reports
Biannually
Program
coordinator/
evaluator
How was the documentation
provided for collaboration with
partners outside state health
department?
# of meetings, meeting notes,
emails, letters indicating
agreement to collaborate
Meeting notes
and reports
review
Meeting notes and
reports
Biannually
Program
coordinator/
evaluator
What are types of activities are
partners involved in?
# of specific tasks
accomplished
Meeting notes
and reports
review
Meeting notes and
reports
Biannually
Program
coordinator/
evaluator
What are the reasons for long term
sustained partnerships?
List of reasons for long term
sustained partnerships
Partners’ survey
Survey questionnaire
Once in a grant
period
Program
evaluator
What are the challenges
partners feel that hinder
intended progress?
List of reasons for long term
sustained partnerships
Partners’ survey
Survey questionnaire
Once in a grant
period
Program
evaluator
16
Indicator/
Performance Measure
Evaluation Question
Method
Data Sources
Frequency
Responsibility
Meeting notes
and reports
review
Meeting notes and
reports
Biannually
Program
coordinator/
evaluator
# of members from each sector
of CDC coalition framework
Coalition member
roster
Coalition member
roster
On-going
Program
Coordinator
# of members from each sector
attending coalition meetings
Member group is
identified on the
membership
application
Stakeholder database
What were the steps taken to
facilitate institutionalization and
sustainability of coalition?
501c3 status, written
vision/mission statement,
priorities /plans/strategies
Consistency of membership
Review bylaws and
coalition work plan
By-laws and work plan
Annually
Coalition
members/
program
coordinator
Which programs and policy changes
have resulted from the coalition
efforts?
# of policy changes as a result of
coalition efforts
Review policy
documents
Policy document
Annually
List of program initiated
Review reports
Coalition reports
Annually
Coalition
members/
program
coordinator
How does the coalition focus on the
following?
•
Oral health infrastructure
development and sustainability
•
State oral health plan
revision and
implementation
•
School-based/-linked
sealant programs
•
Community water
fluoridation efforts
What sectors of CDC coalition diversity
framework do coalition members
represent?
Coalition initiatives in the areas of
Oral health
infrastructure
development and
sustainability
•
State oral health plan
revision
•
School-based/-linked
sealant programs
•
Community water
fluoridation
•
Data
Coordinator
17
Evaluation Question
Indicator/
Performance Measure
Method
Data Sources
What types of communication are being
used by partners?
Types of
communications by
partners
Review reports
Progress reports
Member survey and
review of agenda,
minutes and reports
Member survey and
Coalition agendas,
minutes and reports
On going
Use of website for
communications
Coalition website
On going
Review program
progress reports
Program progress
reports
Quarterly
Program
coordinator/
evaluator
Annually
Program
coordinator
Once in the grant
period
Program
evaluator
# of member that feel the
communication is adequate
How do collaborative partners focus on
oral health issues and prevention
strategies as outlined in the state oral
health plan?
Coalition website
updated quarterly
# of partner involved in
initiating new activities
outlined in state oral health
plan
Frequency
On going
Responsibility
Coalition
secretary/
program
coordinator
# of partner participating in
activities outlined in state oral
health plan
What are the forms of documentation of
commitment or support available from
stakeholders e.g. delineating
accountability and clear lines of
responsibility for activities?
Letters of support from
stakeholders
Review coalition
reports
Description of roles and
responsibilities of partners for
activities
Review of coalition
manual
How did partnerships influence oral
health policies, practices and systems?
List of policies, practices and
systems influence
Review report and
survey results
Coalition reports
Coalition manual
Annual reports and
partners’ survey
18
Step 5: Justify Conclusions
•
•
•
•
•
Ensure accuracy of data and data entry
Analyze the data
Make sense of what you have learned
Develop recommendations
Engage stakeholders
19
Step 6: Ensure Use and Share
Lessons Learned
•
•
•
•
Develop a communication plan
Communicate results
Incorporate into ongoing planning
Engage stakeholders
20
Resources
•
•
•
•
ASTDD website (slides/speaker notes from this webinar will also be posted on the
ASTDD website) and B. J. Tatro (ASTDD evaluation consultant) bj@bjtatro.com
There are also a variety of resources related to oral health collaborations on the
site.
nd
Collaboration: What Makes It Work (2 ed), P. Mattessich, M. Murray-Close, & B.
Monsey (Fieldstone Alliance, 2001) Includes The Wilder Collaboration Factors
Inventory. This publication includes a review of the literature on factors influencing
successful collaboration and The Wilder Collaboration Factors Inventory TOOL.
Collaborative Effectiveness Assessment Activity (Prevention Institute, 2004) This
TOOL is designed to stimulate thinking about some elements of effective
collaboration.
Evaluating Partnerships to Prevent and Manage Chronic Disease, F. Butterfoss,
Preventing Chronic Disease, April 2009, Vol. 6, No. 2 This article describes how to
develop a comprehensive evaluation strategy based on partnership theory, select
outcome indicators, choose methods and tools, and use results for accountability
and improving effectiveness.
21
Resources
•
•
•
Evaluation Guide: Fundamentals of Evaluating Partnerships, National Heart
Disease & Stroke Prevention Program (Centers for Disease Control and Prevention,
2008) This is one of a series of evaluation technical assistance tools. The guides
supplement the CDC program evaluation framework. This particular guide provides
information on how to implement the framework when evaluating partnerships at
various levels (annual assessment, basic evaluation, and enhanced evaluation).
Appendices include examples of evaluation questions, a variety of TOOLS, and
other helpful information.
Framework for Evaluation (CDC, 2012) CDC provides a six-step framework for
conducting program evaluation. CDC also offers guides for developing logic models
and developing evaluation plans; see Developing and Using a Logic Model and
Developing an Effective Evaluation Plan
Guide to Evaluating Collective Impact, H. Preskill, M. Parkhurst, & J. Splansky
Juster (FSG, Collective Impact Forum, 2014) In addition to three guides, this site
links the user to a knowledge exchange related to collective impact.
22
Resources
•
•
•
Handbook on Planning, Evaluating and Improving Interagency Collaboration in
State Oral Health Programs (ASTDD, 2012) The handbook and companion
workbook provide a framework for evaluating collaboration between state oral
health programs and other entities. A variety of TOOLS, as well as oral health
examples, are provided.
Measuring Collaboration Among Grant Partners, B. Frey, J. Lohmeier, S. Lee, & N.
Tollefson, American Journal of Evaluation, 2006, 27, 383 This TOOL can be used to
assess levels of collaboration—ranging from networking to collaboration for each
partner.
PARTNER This is an online social network analysis tool designed to collect, analyze,
and interpret data to improve collaboration within community networks.
23
Resources
•
•
•
Partnership Evaluation: Guidebook and Resources, National Center for Chronic
Disease Prevention & Health Promotion, Division of Nutrition, Physical Activity &
Obesity (Centers for Disease Control and Prevention, 2011) This is one of a series
of evaluation technical assistance tools. The guides supplement the CDC program
evaluation framework. This particular guide provides information on how to
implement the framework when evaluating partnerships. Appendices include a
variety of TOOLS and other helpful information.
Partnership Self-Assessment Tool (Center for the Advancement of Collaborative
Strategies in Health, 2002) This TOOL can be used to assess how well a
collaborative process is working and to identify specific areas for improvement. It
is designed for partnerships of five or more that have been in existence for at least
six months.
Tools for Partnership Evaluation (CDC, 2012) This “coffee break” webinar was
presented by S. Ladd, CDC Division for Heart Disease and Stroke Prevention
Applied Research and Evaluation Branch. It covers three types of TOOLS—
evaluability assessment, membership and roles, and process and functioning.
24
Thank You!
Thank you to all who provided information for and
input into this webinar, including:
– Harder+Company, for sharing a matrix of
partnership/collaboration evaluation tools that they
researched and prepared
– John Shoemaker, for sharing a variety of
partnership/collaboration tools and other resources
– The many CDC-funded states that provided copies of
their partnership/collaboration evaluation logic
models and evaluation plans, especially those who
allowed us to draw examples from their work
And thanks to all of you who participated today!
25
Polling Questions
• Are you planning to do a partnership
evaluation within the next few years?
– Yes, No, Not Sure
• Have you developed your partnership
evaluation plan yet?
– Yes, No, Not Applicable (no study planned)
• What could ASTDD do to help you
plan/conduct your partnership evaluation?
26