Evaluating Our Coalition:
Are We Making an Impact?
Presentation at
The Conference for Family Literacy
Louisville, Kentucky
By Apter & O’Connor Associates
April 2013
Why a Coalition?
Critical
stakeholder
mass
Greater
efficiencies
Broader goals
accomplished
Community
benefits
Why Evaluate?
Evaluation can guide the change process
Evaluation can monitor the health of your
coalition
Evaluation can “tell your story”
Evaluation can “dig deeper”
Evaluation can measure your success and
answer the “so what” questions
Why Evaluate?
Provide Accountability to
Community, Funders & Stakeholders
Quality
Efficiency
Effectiveness
Effectiveness
Effectiveness
Step 1: Defining YOUR Coalition
Network
• Dialogue
• Informal linkages
• Information
exchange
Cooperative Group
• Short term goals/tasks
• Linkages may be
formalized but still
advisory
Partnership
Collaborative
• Share/merge
resources
• Joint planning but
individualized
authority
• Frequent
communication
• Common issue
• Long-term commitment
• Formal agreements
• Shared leadership and
decision making
• Shared community-wide
indicators of success
SO WHAT!!
The Impact of Your Coalition
Step 2:
Meeting Evaluation Challenges
1. Lack of clarity about the expected
outcomes from the coalition: What to
evaluate
2. Coalitions take time to build
3. Coalitions are not static
4. Measurement: Attribution vs. Contribution
(who gets “credit”)
Challenge #1: Lack of Clarity
about Expected Outcomes
Some Solutions
• Reflect back to your logic model:
reasonable, feasible, focused outcomes
• Clarity about what the coalition will bring to
the community that individual programs
cannot.
• The evaluation must match the level of
anticipated outcome
Level of Impact
Communities
Systems
Agencies/
Organizations
Individuals/
clients
Individuals
Change in behaviors,
attitudes, skills
Agencies/Organizations
Change in behaviors, attitudes,
knowledge and skills
Change number and type services
provided
Change in policies, procedures
Leadership development
Systems
Change in behaviors, attitudes,
knowledge and skills
Increase in
coordination/collaboration/cohesion
within or across systems
Change in service delivery
Joint decision making
Community
More resources/services for the
community
Change in legislation, regulation, policy
Heightened community awareness/civic
action
Demonstrated change in social, economic,
environmental conditions
Challenge #2: Coalitions Take
Time to Build
Some Solutions
• Clarify the “development phase” of your
coalition and the key tasks that need to be
accomplished
• That will lead you to appropriate questions
and important data to collect
Evaluating
the right Evaluation”:
thing at the right
“Developmental
Evaluating the Righttime!
Thing at the Right Time…
•
•
•
•
Planning and forming
Why do we need this collaboration?
Who should be involved?
Do we have a common vision?
Formative Evaluation:
--Readiness and capacity
Evaluating the Right Thing at the
Right Time
Developing
• Are we on track?
• Is our coalition meeting the needs of
members?
• Are we achieving early outcomes?
Process Evaluation:
-- Beginning to implement
Evaluating the Right Thing at the
Right Time
Maturing and Sustaining
• What have we accomplished?
• What are the benefits for our
coalition and for whom?
• What is the value of our effort?
Outcome/summative evaluation
-- Accountability
Challenge #3: Service Systems
and Communities Are Not Static
Some Solutions
• Revisit your logic model:
– Are the new strategies are still in line with
your vision
– Does the coalition membership still include
the “right” stakeholders
Challenge #4: Attribution vs.
Contribution
• How do we insure “self interest” of member
agencies/organizations are met?
• How do we distinguish outcomes of the
coalition vs. outcomes by programs funded
by the coalition vs. other contextual factors?
Some Solutions
• Do a good job of collecting the right data
– Reflects the coalition expected level of impact
– Reflects the coalition developmental stage
• Critical importance of process to outcome:
continue to measure perception of members
• Shared measurement systems
Not everything that counts can be counted!
Phases of Shared Measurement
System
STEP I
Design
•Agree on system and
relation to logic model
STEP 2
Develop
STEP 3
Deploy
•Develop platform and
tools
•Ongoing support
•Review current state of
knowledge/data
• Test and refine
platform/tools
•Agree on governance
structure
•Staff for data
management and
synthesis
•Identify approach,
metrics, confidentiality,
etc.
•Learning forums
•Review, refine,
improve, ongoing
evaluation of usability
and impact
Source: FSG Analysis, www.fsg.org 2011
The COMET® System
Cautions….
• Clarity among organizations about
WHY you are collecting this data and how it will be
used
• Getting organizations to agree on
a set of shared indicators that reflect their work
• Silo nature and reporting requirements of funders
• And………….
Evaluation Types are All Related….
Formative
Measure
infrastructure,
functioning and
procedures of the
coalition
Process
Measure indicators
of extent of
implementation
Outcome
Measure change and
realization of vision
Tools of the Trade
SURVEYS: member benefits, perceptions
of stakeholders, informants, community
INTERVIEWS/FOCUS GROUPS
stakeholders, beneficiaries, community
members
OBSERVATIONS: site visits, meetings
RECORDS, DOCUMENT REVIEW
CASE STUDIES: policy analysis,
beneficiaries, build a story of the
coalition’s work and impact
INDICATORS: statistical analysis for
significance, trend data, change in
targets, cost-benefits
Tools of the Trade
Collect information from SEVERAL
SOURCES (community residents at
large, targeted clientele, elected
officials)
Collect information using SEVERAL
METHODS (survey, focus group and
site observations for perceptions)
Consider OUTSIDE EVALUATORS (to
add credibility, expertise, objectivity)
Collect both QUANTITATIVE (the
numbers) and QUALITATIVE(the story)
data
USE your findings
Successful Coalitions
Pre-Conditions
• Urgency for change
• Adequate financial
resources
• Influential champions
Kania and Kramer, Collective Impact,
2011 www.ssireview.org
Conditions for having impact
• A common agenda (vision)
• Continuous communication
• A shared measurement
system with a “short list” of
indicators
• Mutually reinforcing
activities (relating to vision
• A centralized backbone
structure
Contributions of a Literacy Coalition
1. Selection of a core set of community outcome
indicators by the literacy coalition
2. Provide training and technical assistance in
performance measurement
3. Assist community literacy programs to track
and use their own outcomes
4. Provide support for community-wide outcome
data collection
5. Provide support for analysis of the outcome
information.
Adapted by Apter & O’Connor from National Institute for Literacy, Guide to Performance Management
for Community Literacy Coalitions, Washington, DC 20008
Contributions of Evaluation
6. Use the findings to help attract funding
7. Use the findings to identify and report
community literacy program needs and literacy
condition
8. Use the findings to help identify and
disseminate successful (“best”) practices in
your community and elsewhere
9. USE THE FINDINGS TO CELEBRATE GOOD
LITERACY OUTCOMES
Adapted by Apter & O’Connor from National Institute for Literacy, Guide to Performance Management
for Community Literacy Coalitions, Washington, DC 20008
For more information…
Literacy Powerline
www.literacypowerline.com
Apter & O’Connor Associates
www.apteroconnor.com
dianne@apteroconnor.com
cynthia@apteroconnor.com
For more information…
The Literacy Powerline Site
www.literacypowerline.com
or
The Apter & O’Connor Associates Site
or