Outcome Measurement
and Evaluation
For Nonprofit Learning Point
S
Overview
S Introduction to Evaluation and Measurement
S Why Should You Measure Outcomes?
S Creating a Framework for Identifying Appropriate
Outcomes for Your Program – the Logic Model
S How to Measure Outcomes
S Brief Review of Results-Based Accountability
Gauging Your Needs
S What is the mission of your organization?
S What services do you provide?
S Why are you interested in this class?
S What do you hope to take away today?
S Who do you plan to share it with?
3
Seize the Opportunity
Continual improvement is an unending journey.
- Lloyd Dobens
4
Balanced Perspectives
I don't have a particular recommendation other than that we base
decisions on as much hard data as possible. We need to carefully look
at all the options and all their ramifications in making our decisions.
- Dorothy Denning
5
What is Evaluation?
S Program evaluation is carefully collecting and analyzing
information about a program or some aspect of a program
in order to make necessary decisions.
S The type of evaluation you undertake to improve your
program depends on what you want to learn.
6
Evaluation and Measurement
S Measurement techniques help us collect useful feedback. The
purpose of measurement is to raise our awareness,
presenting opportunities for improvement.
S Data results from our efforts to measure our program
services, participants, and results.
S Information results from processing data.
“The height of insanity is doing things the same way and hoping for
a different result.”
7
Myths About Evaluation
S Evaluation is a useless activity that generates lots of
boring data with useless conclusions.
S Evaluation is about proving the success or failure of a
program.
S Evaluation is a highly unique and complex process.
S Evaluation is only for developing organizations.
S Nobody cares about program evaluations.
8
What Does Evaluation Help Us Do?
9
What Does Evaluation Help Us Do?
S Determine where we actually are
10
What Does Evaluation Help Us Do?
S Determine where we actually are
S Set goals for where we want to be
11
What Does Evaluation Help Us Do?
S Determine where we actually are
S Set goals for where we want to be
S Develop a plan for how to get there
12
What Does Evaluation Help Us Do?
S Determine where we actually are
S Set goals for where we want to be
S Develop a plan for how to get there
S Track progress on implementing the plan
13
What Does Evaluation Help Us Do?
S Determine where we actually are
S Set goals for where we want to be
S Develop a plan for how to get there
S Track progress on implementing the plan
S Change the plan, if necessary
14
Why Measure Outcomes?
S Gather information that will be useful for decision
making/program development
S Identify strong and weak components of programs
S Assist in demonstrating a program’s value and
effectiveness
S Help justify existence
S Guide budget development and resource allocations
Roadblock to Meaningful Evaluation
S Key Problem:
Priority measurement issues not identified
S What areas are most in need of improvement?
S What benefits do we need to demonstrate?
S What data do we have? Is it too much? Not enough? Does
the data really tell us what we want to know?
16
Know Your Audience…
Exercise
Possible
Audience
What Do They
Already Know?
What Do They
Need to Know?
17
How Will They
Use the Data?
Develop Balanced Measures
S Include both process and outcomes measures
S Process –activity/output/efficiency
S Number and type of volunteer hours provided, volunteers
trained, workshops held, applications processed, etc.
S Outcomes – effectiveness/results/impact
S Can include client and volunteer satisfaction, change in clients’
level of awareness or behaviors, program referrals from
participants, number of new processes developed, etc.
S Examples:
S Avg. contact time with participants (process)
S Avg. number of participants who demonstrate increased
knowledge of effective parenting skills (outcomes)
18
Develop Balanced Measures
S Address successes as well as challenges
S Successes –accomplishments
S
Percentage of clients who recommend the service to others,
percentage of students who attained a GED, etc.
S Challenges – areas for improvement
S
Error rates, complaints, terminations, etc.
19
Make Your Goals Measurable
Examples
S Process:
S Increase the number of families served to 50 by September 30,
2007
S Outcomes:
S Achieve a 90% exceptional rating on client satisfaction surveys
within 6 months
S Increase reported use of substance abuse refusal skills by 20%
after 10 weeks
S 75% of all participants will achieve independent living status by
January 31, 2007
20
Make Your Goals Measurable
Increase reported use of substance abuse refusal skills by
20% after 10 weeks
Audience: Who do you plan to impact?
Indicator:
Target:
Timeline:
Baseline:
Use of SA refusal skills
20% increase
10 weeks
Pre-intervention use of refusal skills
21
Be Realistic
S Collect data that is actionable
S Don’t measure A and hope for B
S Measuring the hours a counselor spends with a client is not a
measure of how that contact has impacted the client.
22
Creating a Framework for Identifying
Appropriate Outcomes - the Logic Model
What is a logic model? A logic model is a simple
description of how your program works that shows the
linkages between:
SProblem you are attempting to address
SProgram components
SProgram activities
SOutcomes (both short- and long-term)
Why Should You Develop a Logic
Model for Your Program?
S Useful for designing/developing your program
S Provides a useful framework for evaluating your program
S Provides a focal point for stakeholders, requiring them to
work together to identify the components of the model and
to think about expected outcomes.
Existing Problem
S Identification of risk factors or needs in the community
S Availability of program resources in the community
S Availability of fiscal resources
S Type of community partnerships that already exist in the
community
Program Components
Components are closely related groups of:
S Activities in your program that are logically linked to the
conditions identified as part of the program planning
process.
S The number of components depends on the size of your
program and how you administer it. For a large program,
there could be several components in the logic model.
Smaller programs might consist of just one component.
Examples of Program
Components
S Parent component – a group of program activities
conducted with parents
S Family component – a group of program activities
conducted with the entire family
S Youth component – a group of activities conducted with
children/youth in the program
Program Activities
S Activities are the services a program provides for its
participants and the means through which the desired
outcomes will be achieved.
S Activities are logically connected to the identified program
components.
Examples of Program Activities
S Within a parent component, activities might include a 12-
week session of parenting classes and a parent support group
held twice a week
S Within a family component, activities might include family
counseling sessions and family recreational activities held
twice per week
S Within a youth component, activities might include youth
recreational activities and youth counseling sessions twice
per week
Program Outcomes
S Outcomes are benefits or changes for program participants
during or after their involvement with a program that flow
logically from the precipitating program activity.
S Outcomes can be short-term or long-term
S Outcomes are not process measures
Short-Term Outcomes
Short-term outcomes should:
S Flow logically from the precipitating activity;
S Be relevant to the program’s overall goal;
S Be reasonable to achieve; and
S Represent achievements or improvements on the
part of program participants that occur either
during or immediately after program participation ends.
Examples of Short-Term
Outcomes
S Parents in parenting class increase knowledge of
parenting skills
S Youth attending a drop-out prevention program
improve school attendance
Long-Term Outcomes
Long-term outcomes should:
S Have a logical connection to the short-term outcomes;
S Represent benefits or changes for program participants
over an extended period of time;
S Be collected during a follow-up period with program
participants.
Examples of Long-Term
Outcomes
S Parents completing a parenting class improve parenting skills,
including ability to supervise and discipline their children
S Youth participating in a drop-out prevention program stay in
school and graduate
S Youth completing a diversion program are not re-adjudicated
one year after program participation ends
What Information Should be
Included in a Program Outcome?
Outcomes are simple goal statements that identify
the following components:
S The target population
S What benefit or change is occurring
S How the benefit or change will be measured
What is the Target Population?
The target population is the individuals, groups, or
communities to whom the program’s services are
directed. This is your program’s priority population or
the intended “audience” for your program.
Examples might be:
S Males between the ages of 14 and 16
S Parents with children between the ages of 3 and 5
S Families in which abuse is occurring
What Benefit or Change is
Occurring?
S Knowledge
S Attitude
S Skills
S Values
S Behavior
How are Outcomes Measured?
S Setting Outcome Targets
S Defining Amount of Change in Target Population
S Describe Participant Eligibility to Meet Outcome
S Selection of Instruments or Data Collection Tools
Setting Outcome Targets
S Outcome targets are the numerical objectives for the number
and percent of participants who will achieve the desired
outcome in the next reporting period.
S Examples include:
S 75% of participating students will improve grades
S 50% of families will improve family functioning
Defining the Amount of
Change in Target Population
S The amount of change describes specifically how much
impact you expect the program to achieve.
S The amount of change should ideally be measurable.
S For example:
S 75% of participating students will improve their grade point
average by one letter grade
Participant Eligibility
S What level of program participation is required
before you would expect to achieve the desired
outcomes?
S Allows you to assess differences in outcomes for
clients who follow the base program requirements
vs.those who do not.
S For example:
S 75% of students who attend 30 of 40 program sessions will
improve their grade point average by one letter grade
Selection of Instruments or
Data Collection Tools
Things to Consider When Selecting Instruments
or Data Collection Tools:
S Purpose and Feasibility
S Administration of Instrument
S Utility and suitability
S Psychometric qualities
Purpose and Feasibility
S Is the tool’s purpose consistent with your needs?
S Does the instrument measure issues that relate directly to
your desired outcomes?
S Are there clear directions for administration?
S Is instrument easy to administer and score?
S Is training needed prior to administration?
S Does program have adequate resources to cover cost of
instrument, scoring, and training, if applicable?
Administration of Instrument
S Number of items
S Length of time to complete
S Method of administration (self-report, rating by trained
clinician, other)
S Recommended administration schedule
Utility and Suitability of
Instrument
S How will data be used?
S What reading and writing levels are needed by respondents?
S Is tool culturally sensitive?
S Is tool available in other languages?
S Does tool have good clinical fit with other instruments
program is using?
Psychometric Qualities of
Instrument
S Tested groups
S Reliability
S Validity
Specific Ways to Measure
S Surveys
S Personal interviews
S Focus groups
S Other data collections tools
47
Who Can Be Helpful?
S Depends on the purpose – who can answer your questions?
S If not apparent, ask for guidance from key informants.
S If potential population is large, generate a list of candidates
and key characteristics then randomly select.
48
What is a Survey?
A quantitative research project in which a relatively large
number of people are queried, each being asked a standard
set of questions, posed in the same way each time.
49
Why Use Surveys?
S To collect specific, standardized data across
respondents.
S To collect data across many people as efficiently as
possible.
S To increase understanding.
S To collect data quickly, even across multiple groups.
S Open-ended quotes may add impact and credibility.
50
Types of Surveys
Method
S Snail Mail
S Web-Based
S Phone
S Blended Approach
Question Types
S Open-ended
S Closed-ended
S Combination
51
The Survey Tool
DO:
S
Keep the survey as short as possible.
S
Balance white space and length.
S
Keep questions neutral.
S
Be specific in your wording.
S
Pretest your questions.
S
Address only one construct per question.
S
Use a blend of closed and open-ended questions, if reasonable.
S
Mix types of questions throughout.
S
Provide an opportunity for additional comments.
52
The Survey Tool
DON’T:
S Use leading questions.
S Make your questions too complicated.
S Ask controversial questions first.
S Mix reference points.
53
How to Enhance Participation
DO:
S Provide a reason for the survey.
S Use clear instructions.
S Explain confidentiality protections.
S Give a realistic sense of the time involved.
S Track respondents.
S Use reminders.
54
How to Enhance Participation
DO:
S Provide your contact information.
S Request contact information from the respondent, if
appropriate.
S Establish an end date.
55
Using Survey Data
S Prepare your analysis plan.
56
What is an Interview?
An interview is the collection of data by asking people
questions and following up or probing their answers, usually
conducted face-to-face or by phone.
57
Why Use Interviews?
S As an exploratory first step to creating quantitative
tools.
S Enhance understanding of interesting findings which
emerged from other processes.
S Real-world quotes may add impact and credibility.
S To collect specific, standardized data across
respondents.
58
Types of Interviews
Method
S Face-to-face
S Telephone
Style
S Conversational: go with the flow
S Interview guide
59
Conversational Approach
S Questions emerge through conversation
S Highly individualized
S May provide greater insights
S Requires interviewer with strong content and interpersonal skills
S Not systematic
S Can be more difficult to analyze
60
Interview Guide Approach
S Outline of topics exists, but wording and order can vary
S Topics covered is more systematic
S Interview setting remains fairly informal
S Also required considerably skilled interviewer
S Inhibits spontaneous discussion of new topics
61
Open-Ended Structured Approach
S Strict script for question, but free-response format for
answers
S Most efficient of qualitative techniques
S Reduces bias and guides less experienced interviewers
62
The Interview Tool
DO:
S Keep the interview as short as possible.
S Keep questions neutral.
S Be specific in your wording.
S Pretest your questions.
S Address only one construct per question.
S Use a blend of closed and open-ended questions, if reasonable.
S Provide an opportunity for additional comments.
63
The Interview Tool
DON’T:
S Use leading questions.
S Make your questions too complicated.
S Ask controversial questions first.
S Mix reference points.
64
Recruitment Techniques
DO:
S Provide a reason for the interview.
S Give a realistic sense of the time involved.
S Explain how your interview candidate fits into the bigger
goal.
65
Recruitment Techniques
DON’T:
S Be vague about why you need the interview.
S Candy coat the commitment.
66
Preparing for the Interview
DO:
S Plan an appropriate interview location.
S Exchange contact information with the interviewee.
S Prepare to capture information.
S Create response cards, if needed.
67
Preparing for the Interview
DON’T:
S Choose a location with lots of distractions.
S Forget to take the interviewees phone number with you!
68
Building Rapport
DO:
S Be on time.
S Review the purpose of the interview.
S Discuss how the results will be used.
S Reiterate confidentiality protections.
S Acknowledge the time frame you have set aside.
S Provide your contact information on paper.
S Ask if they have questions before you begin.
69
Building Rapport
DON’T:
S Be a robot!
70
General Tips
S Don’t be judgmental!
S Use appropriate non-verbal cues.
S Attend to non-verbal cues.
S Repeat and clarify when needed.
S Practice active listening.
S Let the interviewee know when you are switching gears.
S Keep the interviewee focused.
S Don’t make assumptions.
71
What is a Focus Group?
A focus group is a group discussion. Participants are
brought together in a neutral location for the specific
purpose of discussing an issue or responding to ideas or
materials of interest.
72
Why Use Focus Groups?
S To increase understanding.
S As an exploratory first step to creating quantitative
tools.
S Enhance understanding of interesting findings
which emerged from other processes.
S Real-world quotes and rich feedback may add
impact and credibility.
73
Types of Focus Groups
Method
S Exploratory
S Issue-Focused
74
Focus Group Model
DO:
S
Plan on a 2 hour session
S
Secure 8-12 participants and a few alternates
S
Identify 5-7 key questions beforehand
S
Provide background information to group.
S
Keep questions neutral.
S
Record all input.
S
Have SMEs available to answer questions.
S
Use an objective facilitator.
75
The Focus Group Session
DON’T:
S Use leading questions.
S Accept too many participants.
S Try to squeeze in too many questions.
S Forget to document results.
76
Recruitment Techniques
DO:
S Provide a reason for the focus group.
S Give a realistic sense of the time involved.
S Explain how your focus group candidate fits into the bigger goal.
S Establish a potential participant list.
S Invite participants by letter, followed by phone calls.
S Consider offering incentives.
77
Preparing for the Session
DO:
S Secure a roomy, centrally-located venue
S Select a facilitator
S Exchange contact information with the participants
S Prepare background information
S Prepare to capture information
78
Building Rapport
DO:
S Be on time.
S Review the purpose of the focus group.
S Discuss how the results will be used.
S Reiterate confidentiality protections.
S Acknowledge the time frame you have set aside.
S Provide your contact information on paper.
S Ask if they have questions before you begin.
79
Other Tools
S Observations
S Checklists (process documentation)
S Attendance logs
S Case records
S Existing databases
80
Data Management
S Develop a plan to maintain program and outcome
information
S Excel can be a useful tool
What is Results-Based
Accountability (RBA)?
S Draws a clear distinction between community
accountability for population results vs. manager
accountability for program client results.
S Addresses accountability from the highest level view across
systems and across communities to accountability for the
smallest program in the bureaucracy, and everything in
between.
Results-Based Accountability Implementation Guide (www.raguide.org)
82
How Does RBA Fit Into the
Levels of Outcomes?
Outcomes Level
Community
Systems
Collaborative
Organizational
Program
Client
Description
Collected across organizations/programs
to address changes in community-level
needs or conditions
Collected across and/or outside
organizations to assess changes in
capacity or systems
Collected, across programs or
organizations, to assess results for
clients of collaborative service efforts
Collected, across multiple programs, to
assess results for clients of an
organization
Collected, across multiple clients, to
assess results in a group of individuals
who have received a service or set of
services
Collected to assess changes in an
individual client or a group of clients as
a result of a provided service
Often Driven By:
Community Collaboratives,
Funding Collaboratives
(Results-Based Accountability)
Systems Change Agents,
Organizations, Funders
Collaborative Partners,
Organizational Leadership,
Funders
Organizational Leadership,
Funders
Program Staff, Organizational
Leadership, Funders
Program staff, Organizational
Leadership, Funders
83
Other Supporting Data:
•Administrative
•Implementation (progress,
barriers, integrity,
compliance)
•Workload (hours
invested)
•Output/Process (#s
served)
•Financial (costs)
•Program Development
(satisfaction, attendance)
RBA –
Population Accountability
S Population accountability is accountability:
S BY THE COMMUNITY (city, county or state)…
S TO THE COMMUNITY (city, county or state)…
S FOR THE WELL-BEING (RESULTS) OF A POPULATION
(children, adults, families, all citizens...).
Results-Based Accountability Implementation Guide (www.raguide.org)
84
RBA –
Performance Accountability
S Performance accountability is accountability:
S BY THE MANAGERS (of the program, agency or service
system)…
S TO THE CUSTOMERS (and other stakeholders)…
S FOR THE PERFORMANCE OF THE PROGRAM (agency
or service system).
Results-Based Accountability Implementation Guide (www.raguide.org)
85
Other Key Concepts of RBA
S Results Decision-Making
S Results-Based Budgeting
S Language of Accountability
Results-Based Accountability Implementation Guide (www.raguide.org)
86
The Community Foundation’s
Implementation of RBA
S Long-Term Goals
S Performance Measures:
S How Much Do You Do?
S How Well Do You Do?
S Will Anyone Be Better Off ?
S Was Anyone Better Off Last Year?
S Data Collection Methods
87
Parting Tips
S Plan ahead, plan ahead, plan ahead
S Focus on continuous improvement
S Include outcomes/results measures
S Keep it simple
88
Questions?
Please contact:
Trina Willard
Principal
Knowledge Advisory Group
804-564-6969
Trina@KnowledgeAdvisoryGroup.com
89