PROGRAM
IMPACT & OUTCOME
EVALUATION:
Is The Pain Worth The Gain?
Gary A. Hachfeld
Extension Educator, Ag Business Management
University of Minnesota Extension
RME Conference
March 2009
Reno, NV.
Copyright 2009. University of Minnesota. All Rights Reserved.
Today’s Outline:
Definition of Evaluation
Perspectives on Evaluation
Evaluative Philosophy & Tips
Evaluative Approaches
Evaluative Framework
Evaluation & Program Example
Uses of Evaluative Data & Summary
Copyright 2009. University of Minnesota. All Rights Reserved.
Program Evaluation:
• Definition of Evaluation*: systematic process
of determining the worth of a program.
• Continuous Process*: is an essential part of
all stages of educational programming:
planning, design, implementation and
evaluation.
*Seevers, Barbara et.al. (1997) Education Through Cooperative Extension. Delmar Pub.
Copyright 2009. University of Minnesota. All Rights Reserved.
Perspectives on Evaluation:
• Evaluation is a burden. Evaluation is boring
and a huge waste of time. Evaluation requires
too many resources to accomplish.
OR
• Evaluation is an integral part of program
development and implementation. It is NOT an
inconvenience or “add on”. Evaluative data can
be a valuable tool when reported to funders,
organizational administration, the public, etc.
Copyright 2009. University of Minnesota. All Rights Reserved.
Evaluative Philosophy:
• The Principle of Leadership and Mission.*
• Begin With the End in Mind!
*Covey, Steven R. (2004) The 7 Habits of Highly Effective People. Free Press
Copyright 2009. University of Minnesota. All Rights Reserved.
Evaluation Tips:
• Involve your program team*.
• Ask yourself the following questions*:
– Do I have a program (a logical, sequential set
of events or activities that accomplish an
agreed upon set of goals or objectives)?
– Is it worthy of evaluation?
*Krueger, Richard A. (2000) Think Like an Evaluator
Copyright 2009. University of Minnesota. All Rights Reserved.
Evaluation Tips:
• Think about WHAT your program goals and
purposes are (“proposed results”).
– WHAT are you trying to accomplish with your
program.
– Design your evaluation process to determine if
you have met WHAT it is you are trying to
accomplish with the program.
Copyright 2009. University of Minnesota. All Rights Reserved.
Evaluation Tips:
• Think about WHO you are doing the
evaluation for*.
– Grant and other financial funders
– Program participants
– Organizational administration
– Your program team
– General public
*Krueger, Richard A. (2000) Think Like an Evaluator
Copyright 2009. University of Minnesota. All Rights Reserved.
Evaluation Tips:
• Think about WHY you are evaluating the
program*.
– Demonstrate accountability to funders
– Identify consequences of the program
– Leverage funding for other programs
– Build organizational support
– Feedback to people: public, clients, sponsors
– Improve future programs
*Krueger, Richard A. (2000) Think Like an Evaluator
Copyright 2009. University of Minnesota. All Rights Reserved.
Approaches to Evaluation:
• Formative Evaluation:
– Determine what works with a particular
situation or audience.
– Determine how we are doing as educators.
– Increase the impact and effectiveness of the
program.
– Insight into the evolution of the program.
Copyright 2009. University of Minnesota. All Rights Reserved.
Approaches to Evaluation:
• Summative Evaluation:
– Assess program impact and outcome.
– Supply of evidence for reporting program
impact and outcome.
– Funder accountability, public awareness, etc.
– Organizational support.
– Personal performance measures.
Copyright 2009. University of Minnesota. All Rights Reserved.
Evaluative Data:
• Quantitative Evaluative Data:
– Numbers, statistics, measurements, etc.
• Qualitative Evaluative Data:
– Comments, stories, quotes, interviews,
descriptions, perceptions, case studies,
journals, etc.
Copyright 2009. University of Minnesota. All Rights Reserved.
Evaluation & Program:
• Evaluative clarity:
– Start by thinking about what you want to
accomplish programmatically.
– Think about the who & why of the evaluation
process.
• Evaluative clarity will result in a quality
program curriculum and will enable you to
report substantive impacts and outcomes.
Copyright 2009. University of Minnesota. All Rights Reserved.
Evaluation & Program:
• Many evaluation/ program development models:
– Linear Logic Models:
• Newtonian Causality Logic Model
• Nascent Feedback-Based Systems Logic Model
– Interdependent System Relationship Models:
• Web of Interconnections
– Complex Linear Dynamics.
– Theory of Action Model.
Copyright 2009. University of Minnesota. All Rights Reserved.
Evaluative Framework:
7. End Results
6. Practice Change
5. KASA Change
4. Reactions
3. People Involvement
2. Activities
1. Inputs
*Bennett, Claude F. (1977) Hierarchy of Evidence for Program Evaluation
Copyright 2009. University of Minnesota. All Rights Reserved.
Evaluative Framework:
4. Reactions
3. People Involvement
2. Activities
1. Inputs
Degree of interest, like or
dislike for program.
Participants: number, gender,
age, etc.
Program promotion, presentation, etc.
Staff/volunteer time, costs, resources.
*Bennett, Claude F. (1977) Hierarchy of Evidence for Program Evaluation
Copyright 2009. University of Minnesota. All Rights Reserved.
Evaluative Framework:
Social, economic.
7. End Results
environmental, &
(impact & outcome) individual consequences.
6. Practice Change
(impact)
5. KASA Change
Adoption & application of
knowledge, attitudes, skills, and
aspirations to work, lifestyle, etc.
Knowledge, Attitudes, Skills, Aspirations.
(impact)
*Bennett, Claude F. (1977) Hierarchy of Evidence for Program Evaluation
Copyright 2009. University of Minnesota. All Rights Reserved.
Evaluative Framework:
7. End Results
6. Practice Change
5. KASA Change
4. Reactions
3. People Involvement
2. Activities
1. Inputs
*Bennett, Claude F. (1977) Hierarchy of Evidence for Program Evaluation
Copyright 2009. University of Minnesota. All Rights Reserved.
Evaluation & Program
Example:
Farm Transition
&
Estate Planning
Copyright 2009. University of Minnesota. All Rights Reserved.
Evaluation Process:
• End-of-meeting evaluation
– Increased knowledge – 5 point Likert scale (B5)
– Status of transfer & personal estate plans (B3)
– Comments about the program (B4)
• Post-meeting evaluation (6 months later)
–
–
–
–
Progress on plans & barriers encountered (B6)
Usefulness of workbook (B4)
Quality of program (B4)
Topics for a second meeting (B5)
Copyright 2009. University of Minnesota. All Rights Reserved.
End-of-Meeting Evaluation:
• 19 workshops (B1)
– October 2007 – April 2008
– 23 local sponsors
• 587 farm family members (B3)
–
–
–
–
247 farm business units
Participants from 151 different communities
Ages: 22 to 89 (68.5% over age 55)
45.5% female; 54.5% male
• 301 surveys completed
(51.3%)
Copyright 2009. University of Minnesota. All Rights Reserved.
End-of-Meeting Evaluation: (B3)
Yes
No
Up-to-date estate plan?
18%
82%
Up-to-date farm transfer plan?
11%
89%
If no, plan to start farm transfer
process due to workshop
98%
2%
Copyright 2009. University of Minnesota. All Rights Reserved.
End-of-Meeting Evaluation: (B5)
“Strongly agree or agree”
Need for clear goals and
communication
Transfer strategies
Importance of assessing
financial strength
Tax issues
Estate planning strategies
and issues
How to write and utilize a
transition plan
100%
95%
99%
91%
98%
90%
Copyright 2009. University of Minnesota. All Rights Reserved.
Post-Meeting Evaluation:
• Process:
– Mail-out survey with return envelope
– Followed final program by six months
– 247 farm units surveyed
– 107 farm units returned surveys (43.3%)
Copyright 2009. University of Minnesota. All Rights Reserved.
Post-Meeting Evaluation:
• Farm transfer plan progress: (B6)
– 72.9% reported starting the process
– 15.7% had completed the process
• Estate plan progress: (B6)
– 79.2% reported starting the process
– 17.1% had completed the process
Copyright 2009. University of Minnesota. All Rights Reserved.
Outcome of Program:
• Average balance sheet asset value Minnesota
farm family (FINBIN 2007):
– Farm business assets - $1,373,612 per farm family
– Personal assets - $197,068 per farm family
• Follow-up evaluative results:
– 15.7% of 107 farms reported implementing transfer plan
– 17.1% of 107 farms reported implementing estate plan
Copyright 2009. University of Minnesota. All Rights Reserved.
Outcome of Program:
• Actual Program Outcome: (B7)
– 15.7% of 107 farms transfer plan - $22.0 million
– 17.1% of 107 farms estate plan - $3.5 million
• Total Actual Program Outcome: $25.5 million
Copyright 2009. University of Minnesota. All Rights Reserved.
Outcome Evaluation:
• Think of it as the “so what” of your program efforts.
Change in
Action or
Participation*  Knowledge &  Practice  Results
Understanding
Change
(outcomes)
(impacts)
*Bennett, Claude F. (1977) Hierarchy of Evidence for Program Evaluation
Copyright 2009. University of Minnesota. All Rights Reserved.
Outcome Evaluation:
Participation
Change in
 Knowledge & 
Understanding
Attend

Increased

Workshop
Understanding
Action or
Practice  Results
Change
(outcomes)
(impacts)
Develop  $25.5 million
Implement
Plans
Copyright 2009. University of Minnesota. All Rights Reserved.
Program Impact & Outcome
Evaluation:
Lets Practice !
Copyright 2009. University of Minnesota. All Rights Reserved.
Program Evaluation:
• Share evaluative findings with:
Agency administration !
Copyright 2009. University of Minnesota. All Rights Reserved.
RESULTS:
• Farm Transfer Estate Planning:
Funding:
– Risk Management Grants: $52,000
– Minnesota Corn Growers Assn.: $3,600
– Sponsorship fees: $26,400
Recognition:
– Extension Dean & Director Team Award
– NACAA Program Team Award
Copyright 2009. University of Minnesota. All Rights Reserved.
RESULTS:
• Winning the Game:
Funders:
– MN Soybean Research Promotion: $195,000
– Sponsorship fees: $78,000
Recognition:
– Extension Dean & Director Team Award
– AAEA Outstanding Program Award
– Featured in “Source” magazine
Copyright 2009. University of Minnesota. All Rights Reserved.
Summary:
• Think about evaluation and develop your own
philosophy.
• Start with the end in mind. What do you want
to accomplish?
• Think about the who & why of the
evaluative process as it relates to your
program.
• Involve the program team.
• Determine what evaluative approach and data
type you are going to utilize.
• Ask yourself “so what”?
Copyright 2009. University of Minnesota. All Rights Reserved.
Hints:
• Collect evaluative data/information using multiple
methods & levels for same program effort:
– Pre-test & post-test, post-test perceptions, surveys and
questionnaires, face-to-face interviews, phone interviews,
Turning Point Technology, Survey Monkey, etc.
– Can effectively report findings at multiple levels.
• Be Creative - use results in multiple ways:
– Formal written reports (short & long), verbal reports,
press releases, bullet items in marketing brochures,
target audience description in grants, grant results, etc.
Copyright 2009. University of Minnesota. All Rights Reserved.
Hints:
• Avoid “Myths of Reporting”:*
– One written report is enough:
• We do not all learn the same way.
– People read written reports:
• Know your audience.
• “Leak” information to influentials.
– Complexity is impressive to the audience:
• Know your audience.
– Build to the most important point-save it for last:
• Not for evaluative data – time constraints, interruptions, etc.
*Krueger, Richard A. (2000) Think Like an Evaluator
Copyright 2009. University of Minnesota. All Rights Reserved.
Hints:
• Avoid “Myths of Reporting”:*
– A 15 minute report means a 15
minute presentation:
• Audiences want time for questions,
clarification, etc.
– The audience knows why they
are getting the report:
• Indicate why the report was given and any action that may be
recommended.
– Everything should be reported:
• Be brief and report most critical information.
*Krueger, Richard A. (2000) Think Like an Evaluator
Copyright 2009. University of Minnesota. All Rights Reserved.
PROGRAM
IMPACT & OUTCOME
EVALUATION:
Is The Pain Worth The Gain?
You have to decide !
Copyright 2009. University of Minnesota. All Rights Reserved.
Contact information:
Gary A. Hachfeld
Extension Educator
Ag Business Management
507-389-6722
hachf002@umn.edu
Copyright 2009. University of Minnesota. All Rights Reserved.
Questions?
Comments?
Copyright 2009. University of Minnesota. All Rights Reserved.