If we build a partnership will
they come?
Integrating Needs Assessment,
Process Evaluation,
and Impact Evaluation
Ronald Jester and Robert Wilson,
University of Delaware
rcjester@udel.edu, rwilson@udel.edu
A Program Evaluation may include
•
•
•
Process Indicators
or
Impact Indicators
or
A combination of Process and Impact
Indicators
A Process Evaluation Analyzes
Program Inputs, such as
•Outreach Activity
•Services Delivered
•Adherence to Grant
Proposal
•Day-by-day Staff
Activity
•Costs
•Materials
An Impact Evaluation Analyzes
Program Outputs, such as
Productivity
•Farm output
•Individual farmer or
waterman output
•Efficiency
•Benefits/Costs
Client Satisfaction with
services
Improvements in Quality of
Life
AgrAbility evaluations may include
both process and impact indicators
The Delaware Maryland Program
(DMAP) employs a systems model
for program evaluation




This model starts with the target population -2400 farmers with disabilities
The model tracks the target population
through a service delivery system
The system includes a number of filters that
regulate the flow of AgrAbility clients
The model includes a number of efficiency
measures that identify impediments to
efficient client flow
The AgrAbility Process includes 5 steps:
1.
2.
3.
4.
5.
Identify farmers with disabilities.
Diagnose their problems.
Identify the services that will alleviate the
problems.
Find resources to support for the
services.
Deliver the services.
The Delaware-Maryland project relies
on “partnerships” to deliver services.
The efficiency of the project
depends on
1.
2.
3.
4.
5.
6.
Locating the target population
Engaging the target population
Linking the target population with the
partners
Routing the clients to the partners in the
network
Delivering appropriate services
Producing specific outcomes (impacts)
as a result of the services
Overall efficiency is gauged by the
proportion of the target population
•
•
•
•
•
Located
Diagnosed
Referred to services
Provided services
Helped by services
Areas of Evaluation
Process Evaluation




Locating Clients
Diagnosing Clients
Referring Clients to
services
Providing Services
Impact Evaluation

Measuring the Impact
of Services
The Model (Figure 1) portrays the flow of
a target population through a series of filters.
As the target population of 2400 farmers flows
through the service delivery system an increasing n
drop out at each stage.
How to interpret Figure 1







The target group is the estimated 2400 disabled farmers who reside in
Delaware and Maryland. This estimate was generated from a random
survey.
A farmer may seek help the AgrAbility program (columns B and C) or a
program can reach out to a farmer (columns D,E, and F).
Line 1 shows that 280 of the estimated 2400 disabled farmers were
contacted either by AgrAbility or another program.
Line 2 shows that 168 farmers out of the 280 farmers who were
contacted were subsequently evaluated.
Line 3 shows that 120 of the farmers who were evaluated were
subsequently referred for service.
Line 4 shows that 105 of those who were referred for service
subsequently received a service.
Line 5 shows that 90 out of the 105 farmers who received a service
subsequently experienced an improvement in their quality of life.
Figure 1
Systems Model of Delaware-Maryland
AgrAbility Program (DMAP),
Target Population = 2400 Disabled Farmers
Sought Help From
C
B
1
2
3
Did Not Seek Help Contacted by
D
F
E
DMAP
Other
DMAP
Other
None
110
110
60
?
?
Assessed
Assessed
Assessed
Assessed
110
?
58
?
Not
Assessed
Referred
Referred
Referred
80
?
40
Referre
d
Not Referred
?
4
5
Service
Provided
Service
Provided
Service
Provided
60
?
45
Quality of
Life
Improved
Quality of
Life
Improved
?
Quality of
Life
Improved
50
40
Service
Provided
?
No Service
Provided
Quality of
Life
Improved
?
Quality of
Life
Improved
?
Six Process Measures
are employed
Each measure indicates the
efficiency of a different part
of the system
1. Market Penetration
The percentage of the target population that
contacts social service agencies for help.
Include all agencies- AgrAbility and other
agencies that provide services to farmers with
disabilities
Market Penetration Measurement
is based on Target Population:
the estimated 2400 disabled
farmers residing in MD and DE
The farmers who were contacted
are indicated in line 1 of Figure 1
Figure 1
Systems Model of Delaware-Maryland
AgrAbility Program (DMAP),
Target Population = 2400 Disabled Farmers
Sought Help From
C
B
1
2
3
Did Not Seek Help Contacted by
D
F
E
DMAP
Other
DMAP
Other
None
110
110
60
?
?
Assessed
Assessed
Assessed
Assessed
110
?
58
?
Not
Assessed
Referred
Referred
Referred
80
?
40
Referre
d
Not Referred
?
4
5
Service
Provided
Service
Provided
Service
Provided
60
?
45
Quality of
Life
Improved
Quality of
Life
Improved
?
Quality of
Life
Improved
50
40
Service
Provided
?
No Service
Provided
Quality of
Life
Improved
?
Quality of
Life
Improved
?
1. Market Penetration
(220)/(2400)
The percentage of the target
population that contacts social
service agencies for help
1. Market Penetration
(B1+ C1)/(2400)=.09
The percentage of the target
population that contacts social
service agencies for help
Market Penetration is a crude
measure of public service
advertising reach.
2. Outreach
(B1 + C1+DI)/TP
% of the target population (TP)
identified and contacted by
a specific program
The target population that was
reached is shown in cells B1, C1,
and D1 of Figure 1.
Figure 1
Systems Model of Delaware-Maryland
AgrAbility Program (DMAP),
Target Population = 2400 Disabled Farmers
Sought Help From
C
B
1
2
3
Did Not Seek Help Contacted by
D
F
E
DMAP
Other
DMAP
Other
None
110
110
60
?
?
Assessed
Assessed
Assessed
Assessed
110
?
58
?
Not
Assessed
Referred
Referred
Referred
80
?
40
Referre
d
Not Referred
?
4
5
Service
Provided
Service
Provided
Service
Provided
60
?
45
Quality of
Life
Improved
Quality of
Life
Improved
?
Quality of
Life
Improved
50
40
Service
Provided
?
No Service
Provided
Quality of
Life
Improved
?
Quality of
Life
Improved
?
3. Assessment Efficiency.
(B2+D2)/(B1+D1)
Assessment Efficiency is the percentage of potential clients
eligible for assessment that was assessed
The clients who were assessed are show in cells B2 and D2 of
Figure 1.
Clients eligible for assessment are shown in cells B1 and D1
of Figure 1
3. Assessment Efficiency.
(100 +58)/
(110 +60)=.93
The percentage of potential
clients eligible for for
assessment that was assessed
4. Referral Efficiency.
(B3+D3)/(B2 with positive
screen + D2 with positive
screen)
Referral efficiency is the percentage of clients
with a positive screen that was referred for
service.
Clients who were referred for services are shown
in cells B3 and D3.
Those with a positive screen are in cells B2 and
D2.
Figure 1
Systems Model of Delaware-Maryland
AgrAbility Program (DMAP),
Target Population = 2400 Disabled Farmers
Sought Help From
C
B
1
2
3
Did Not Seek Help Contacted by
D
F
E
DMAP
Other
DMAP
Other
None
110
110
60
?
?
Assessed
Assessed
Assessed
Assessed
110
?
58
?
Not
Assessed
Referred
Referred
Referred
80
?
40
Referre
d
Not Referred
?
4
5
Service
Provided
Service
Provided
Service
Provided
60
?
45
Quality of
Life
Improved
Quality of
Life
Improved
?
Quality of
Life
Improved
50
40
Service
Provided
?
No Service
Provided
Quality of
Life
Improved
?
Quality of
Life
Improved
?
5.
Service Delivery Efficiency.
(B4 + D4)/B3 +D3)=
(60+ 45)/(80 + 40)=
.88
Service delivery efficiency is the
percentage of clients referred for service
that received service.
Clients who received a service are show
in cells B4 and D4.
Clients who were referred for service are
shown in cells B3 and D3.
The Delaware-Maryland Project
employs the SF-36 Quality of Life
Instrument as an Impact Measure
Additional impact measures that are
specific to farmers with disabilities
and assistive technology will be
incorporated in the evaluation
These measures are being
developed by a committee drawn
from AgrAbility programs and
national program staff
The SF-36 is the most widely used
quality of life instrument in health
and social services
The SF-36 is calculated for 8 health
scales (0-100)
1.
2.
3.
4.
5.
6.
7.
8.
Physical limitations
Limitations in social activities
Limitations in work and home roles
Bodily pain
Psychological distress and wellbeing
Limitations because of emotional problems
Energy and fatigue
General health perceptions
The SF-36 has been
validated through a
numerous studies
Several thousand scientific
articles employ the SF-36 as a
quality of life measure for
measuring program impact
All active Delaware and Maryland clients
are currently completing SF-36 interviews
The SF-36 serves two
purposes


The SF-36 is used in the initial assessment to
develop a service plan.
The SF-36 is also used in a 1-year follow-up
evaluation to assess changes in the quality of
life subsequent to the initial assessment.
6.
Improved quality of Life:
(# of Clients with improved quality of life) /
(# of Clients that Received AgrAbility Service)
Clients with improved quality of life are shown
in cells B4 and D4. Those who received a
service from AgrAbility are shown in cells B3
and D3.
(B4 + D4)/(B3 +D3)=
(50 + 40)/(60 + 45)=
90/105=
.86
Finally, 2 questions for the
audience:


Which of the measures focus on the
operation of partnerships in AgrAbility
programs?
What other techniques should be used to
evaluate partnerships?