Thinking Evaluatively
Evaluation Framework for Measuring Program Outcomes
Nav Ghimire
Agriculture Agent, Green Lake County
Team Member, ELST
What is Evaluation ?
Judgment about how well your
program objectives are being met….
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Evaluation is…….
Developing Criteria
Gathering Evidence
Forming Judgments
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Criteria
The measure against which judgments are made about.
“Outcome/Program Objective / Evaluative Question”
Evidence
Data that is relevant to evaluative question to make decision.
May be obtained from a variety of measurement tools, survey,
interview, observations, reports, records, etc.
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Judgment
Judgments should be made based upon the degree to which
the evidence indicates that the criteria are being met.
Judgment About What ?
* Appropriateness –
Looks at process than products.
- Were participant needs met?
- Was what learned important to them?
- Were teaching methods appropriate?
* Effectiveness –
Measures program accomplishments
in terms of changes in participant behavior.
- Knowledge, skills , attitude, and application of practice?
* Efficiency –
Compares time, resources, and effort expended in
terms of accomplishments.
- Was accomplishment sufficient for resources used?
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Steps in Evaluative Thinking
Process Evaluation
How is program/ training
implemented?
Are activities delivered as intended?
Are participants being reached as
intended?
Outcome Evaluation
Ability to rigorously collect data
on the program outcomes
Impact Evaluation
Collect data on program
impacts
Changes in participant’s
Changes in
Knowledge
Economic
Skills
Social
Attitude
Environmental
What are participant reactions?
conditions of the individuals
and community
Was teaching method appropriate?
Educator’s self–assessment?
Aspiration
Behavior – application /change
of practice
Source: Journal of Agricultural Education Volume 53, Number 4, pp 85–97; UW-Extension, PD& E
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Process Evaluation
How is program/ training implemented?
e.g., venue, training hall, noise, seating arrangements, food, timing etc.
Are activities delivered as intended?
e.g., curriculum, content etc.
Are participants being reached as intended?
e.g., targeted audience, number of participants etc.
What are participant reactions?
e.g., educator’s knowledge, skills, presentation style, learning material,
delivery style.
Educator’s self–assessment?
e.g. what went wrong, how can I improve next time etc.
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Outcome Evaluation
Usually Conducted
Ability to rigorously collect data on the
program outcomes
Changes in participant’s
Attitude
Immediately after
the program
Immediately or
after some time
After some time
Aspiration
After some time
Behavior – application /change of
practice
After some time
Knowledge
Skills
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Usually Conducted
Impact evaluation
Collect data on program impacts
Changes in - Economic
After few years
Social
After few years
Environmental
conditions of the individuals and
communities
After few years
Nav Ghimire, Green Lake County Agriculture Agent, 2014
How to Measure Program Outcomes

List intended outcomes from the program.

Write measurable program objective(s) to reach each outcome.

List program activities to achieve each objective.

Write evaluative question(s) aligned to each objective.

Identify key variable(s) that evaluation question(s) contain.

Operationalize the variable(s) and identify indicator(s) to assess

Identify source(s) of data.

Determine data collection method(s).
Nav Ghimire, Green Lake County Agriculture Agent, 2014
How to Write a Good Objective

Specifies a result, not an activity.

Each objective describes just one result.

Emphasize what will be done but does not explain how to do it.

It is specific, measurable, achievable, possible, verifiable.

Should have a subject, verb, and object (SVO)

It is designed with and understood by those responsible for its
implementation - Extension educators based on needs assessment.
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Example of Nutrient Management Program
Outcome: Number of farmers developing a nutrient management
plan for their farm will be increased.
Objective 1:
Farmers will gain knowledge and skills about
phosphorous management.
Correct:
Farmers will gain knowledge about the importance of
phosphorous management in their farm /soil.
Objective 2:
Farmers will learn skills to manage phosphorous in their farm
Objective 3:
Farmers will learn skills to manage potassium in their soil.
Objective 4:
Farmers will gain knowledge about the importance of a
nutrient management plan for their farm.
Objective 5:
Farmers will learn skills for writing a nutrient management
plan for their farm.
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Example of Family Living / Youth Program
Outcome:
Numbers of parents/caregivers developing nurturing abilities to
effectively raise their children will be increased
Objective 1:
Parents/ caregivers will gain knowledge about the importance
of nurturing to support their children.
Objective 2:
Parents/caregivers will learn nurturing skills to support their
children / effectively raise their children.
Outcome:
Number of youth developing abilities to resist negative peer
pressure will be increased.
Objective 1:
Youth will gain knowledge about the consequences of
negative peer pressure in the school.
Objective 2:
Youth will learn skills to resist negative peer pressure to avoid
bad habits in the school / life.
Impact:
Number of better citizen in the community will be increased.
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Program objectives Vs Programs Outcomes
Program Objective:
They are learning objectives that you want your participants to
learn as a result of your program. It guides program activities
you want to implement as an educator to facilitate learning.
Program Outcomes:
What participants will achieve / are going to do as a result of
your program / educational activities.
Evaluation:
Measures to what extents program achieved its objectives to
produce a desired outcomes.
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Evaluative Questions
What is it that you want to know
about your program?
operationalize it (make it measurable)
Do not move forward if
you cannot answer this
question.
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Evaluative Questions
 Provides guidelines to direct your evaluation

Demonstrates what area of programming outcomes you
are interested to know about

Primarily not designed as a question to put in survey

Can be a part of the survey question, if needed
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Evaluative Questions
Nutrient Management Training Program
Question 1:
Did farmers increase knowledge about the importance of
phosphorous management in their soil?.
Correct Q.1:
How much did program / training increased farmers’
knowledge about the importance of phosphorus management?
Question 2:
Did farmers develop skills for writing a nutrient management
plan for their farm?
Correct Q.2:
To what extent the training / program developed farmers’
capacity to write a nutrient management plan for their farm?
Question 3:
How well did farmers correctly developed nutrient
management plans for their farm?
Question 4:
How well did farmers applied nutrient management plan in
their farm.
Question 5:
How much did farmers saved cost from improved nutrient
management practices?
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Evaluative Question
Family Living / Youth Program
Question 1:
How much did program/training increased parents /
caregivers knowledge about the importance of nurturing?
Question 2:
How well did parents / caregivers applied nurturing skills to raise
their children?
Question 3:
To what extent youth developed their capacities / abilities to
resist negative peer pressure ?
Question 3:
How well did youth applied negative peer pressure skills to
avoid bad habits?
Question 4:
How much could youth able to resist negative peer pressure to
avoid bad habits ?
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Examples of Good Evaluative Question(s)
?

How well the program / training was implemented?

To what extent has the program / training contributed to
meet the needs of participants.

How much did program / training increased participants’
knowledge and skills?

To what extent the program / training was enough to solve
participant’s farming problems?
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Evaluative Question Contd…

How significant was the program’s contribution to
strengthening the partner organization(s)?

To what extent did the program work well to all
participants?

To what extent the possible program’s positive
results are likely to last after the project ends?
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Evaluation Framework
Nutrient Management Training Program
Variable
Q1.
Knowledge
gain on P
Indicators/ Measure
Number of farmers gained
knowledge in phosphorous
management
management
Q2.
Nutrient
Number of farmers
Management
developing nutrient
plan developed management plan
Data Source
Surveying participants
Surveying participants
Checking with Land and
Water conservation to see
how many farmers
submitted the plan
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Evaluation Framework
Family Living /Youth Program
Variable
Q1.
Knowledge
gain on
nurturing
Q2.
Ability to
resist negative
peer pressure
Indicators/ Measure
Data Source
Number of parent/caregivers Surveying participants
gained knowledge in nurturing
skills
Number of youth increased
ability to resist peer pressure
1. Surveying participants
2. Surveying parents /
school teachers /mentors.
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Questionnaire for Data Collection
 Use self-constructed questionnaire

Each question should focus on one idea only

Easy to answer (Check rather than write), if possible

Keep the same scale throughout the questionnaire

Maintain floor and celling in the scale; very low – very high

Ask only what you need to meet your objective(s)

Use open ended questions when you want details

Give your friends / farmers to read and advise you
Nav Ghimire, Green Lake County Agriculture Agent, 2014
An Example of Questionnaire
Circle the appropriate number to indicate change in your knowledge on the listed
topics after you participated in Nutrient Management Training
1 = Very low
2 = Low
3 = Moderate
4 = High
5 = Very high
Topic Areas
Before Training
After Training
1
2
3
4
5
1
2
3
5
5
Apply nutrient recommendation
from soil test report
1
2
3
4
5
1
2
3
5
5
How pH affects nutrient
availability
1
2
3
4
5
1
2
3
5
5
Crediting N from manure and
legumes
1
2
3
4
5
1
2
3
5
5
P availability from fertilizers and
manure
1
2
3
4
5
1
2
3
5
5
How to sample soil
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Types of Question for A Program Evaluation Survey
Reactions to the Training / Program
Please rate the training on the following items:
Items
Poor
Fair
Average
Good
Content
1
2
3
4
Very
Good
5
Organization
1
2
3
4
5
Teaching
1
2
3
4
5
Involvement of
participants
1
2
3
4
5
Workbook Material
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Types of Question Contd…..
Perception / Attitude
Please indicate to what extent do you agree with the todays’ training
Items
Agree
Disagree
N/A
3
Strongly
Disagree
4
Stimulated me to think
1
Strongly
agree
2
Motivated me to learn more
1
2
3
4
5
Motivated me to do
something different
1
2
3
4
5
Motivated me to talk about
this with my friend
1
2
3
4
5
5
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Types of Question Contd…..
Confidence
Please indicate your level of confidence to perform the skills that you learned
form the training:
Items
Managing phosphorous in soil
Not
A little
confident confident
1
2
Somewhat
Highly
confident Confident confident
3
4
5
Writing a nutrient management
plan for my farm
1
2
3
4
5
Applying the nutrient
management plan in my farm
1
2
3
4
5
Helping my friend to develop
his nutrient management plan
1
2
3
4
5
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Types of Question Contd…..
Intentions
As a result of this training, please indicate your intentions for applying learned
skills / information in your farm / family / work:
Items
No Plans
to apply
May
apply
Will
definitely
apply this
Need
Already
More
applying this
Help To
apply this
4
5
Managing phosphorous in soil
1
2
3
Writing a nutrient management
plan for my farm
1
2
3
4
5
Applying the nutrient
management plan in my farm
1
2
3
4
5
Helping my friend to develop
his nutrient management plan
1
2
3
4
5
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Types of Question Contd…..
Behavior Change
To what extent did you change the following practice that you learned from the
training /program:
Items
None
10% of
my farm
25% of my 50% of my
farm
farm
100% of
my farm
Managing phosphorous in soil
1
2
3
4
5
Writing a nutrient management
plan for my farm
1
2
3
4
5
Applying the nutrient
management plan in my farm
1
2
3
4
5
Nav Ghimire, Green Lake County Agriculture Agent, 2014
Workshop
1.
Identify a program.
2.
Write 1 intended outcome.
3.
Write 2 program objectives.
3.
Write 1 evaluative question.
4.
Display your work on the wall.
Nav Ghimire, Green Lake County Agriculture Agent, 2014