Exploring Impact Assessment Methodology for FSRD1
Mallick, R. N., Subash Dasgupta and Musleh Uddin Ahmed2
ABSTRACT
The history of Farming Systems Research and Development (FSRD) is over two decades
and impact assessment is a common request by most donors in many countries.
Currently, there are several FSRD projects being implemented. However, funding
agencies frequently inquire about the effectiveness of these projects. Though, most
projects have been evaluated and the output of the project is known, the final outcome is
not well documented. This occurs mainly due to the complexity of the impact study.
There is a lack of concrete and appropriate methodology for an accurate FSRD impact
study because it is a relatively new innovation in agriculture. An effort to develop a
methodology for better documentation was tested by the Thana Cereal Technology
Transfer and Identification (TCTTI) Project in 2000/01. The main purpose was to
correctly describe the process, understand critical differences in terminology, identify
evaluation techniques and identify time horizon for impact assessment. Participatory
Rural Appraisal (PRA) was helpful for selecting Ex ante impact assessment indicators at
various levels with explicit illustrations of qualitative and quantitative standards by
interdisciplinary approach. Details of the methodology and assorted results of the study
are illustrated in this poster, so that it may serve as a guideline for comparable projects.
Stakeholders should select indicators for assessment at the time of project design,
although a few impact assessment indicators are common across FSRD projects. Some of
the results of a case study of the Ex post impact assessments that TCTTI conducted at
various levels are also illustrated.
Some of the Indicators:
Varying impacts at personal, family, village,
union, Upazila, and national levels
Promoting integrated water management
Promoting women' roles
Promoting integrated pest management
Safeguarding the environment
Strengthening country level food selfsufficiency impacts and country success in
global context
Improving performance of agricultural research
Increased production of seeds and food
Building information resources
Promoting sustainable agriculture
INTRODUCTION:
The criteria of success of a project are impact, sustainability and contribution on capacity
development. Impact refers to the results of a project that are assessed with reference to
the long-term development objectives (Figure 1). In this sense, impact represents changes
in a situation, whether planned or unplanned, positive or negative that a project brings
about. The purpose is to assess how the target beneficiaries, their households,
communities and country are going to benefit in the long term as the result of the project.
1
Prepared for the 17th Symposium of the International Farming Systems Association to be held from
November 17-20, 2002, Lake Buena Vista, Orlando, Florida USA.
2
Farming Systems Professional, National Professional Officer, and Director General Department of
Agricultural Extension, Khamarbari, Farm gate, Dhaka 1215, Bangladesh
1
Impact assessments broadly evaluate the effects of the program on people in economic,
social and environmental aspects. For example, the first level of impact is directly on the
participating farmers of the FRSD project (Figure 2). They may acquire new knowledge
about the production technologies and take a lead role on field days at Farming Systems
School. The second level of impact is on their households. This is measured in terms of
increased production and income. The third level of impact is on their communities. This
may be an increase in employment, social dignity and empowerment. The fourth level of
impact is at the national level. Skilled farmers of the nation with increased productivity
will ultimately increase the Gross Domestic Product of the nation.
METHODOLOGY:
Impact assessments may be direct, indirect and intermediate.
Conceptual Framework: Framework for impact assessment includes identifying the
tools and techniques, selecting appropriate tools and techniques and developing an
evaluation framework for impact assessment. Brain storming sessions, group discussions,
lectures, data collection on various formats and classroom exercises are considered as
alternate methods. Stakeholders are beneficiaries such as farmers, project staff, extension
officers, research scientists, NGOs, and donor professionals (Figure 3).
Tools and techniques: A decision system that uses several techniques to look at changes
from different angles will provide results that are more robust and reliable.
Quantitative and qualitative tools measure magnitude of changes and provide conclusions
about the types of changes that have occurred. Case studies, focus group discussions,
participatory rapid rural appraisals using timelines, Venn diagrams, seasonal calendars,
flow chart and matrix tests may be used to assess social changes and ownership of assets
as economic criteria (Figure 4). It is important to keep in mind the dynamic nature of how
a project impact and therefore the impact assessment. Impacts can be accurately assessed
only after a significant period of time has elapsed after the completion of a project.
Sustainability is the durability of positive project results after the termination of the
technical cooperation during the project. As a criterion of success the contribution made
to capacity development relates to the extent to which a project enables target groups to
be self-reliant and makes it possible for government institutions and the private sector to
use positive results of the project in addressing broader development issues such as
TCTTI model project.
Capacity development is both a mean and an end for sustainable human development. It
empowers people to realize their potentials and to use their capabilities better and assures
ownership and sustainability of the process and results of the project.
Relevance is the degree to which the objectives of a project remain valid and pertinent
either as originally planned or as subsequently modified due to changing circumstances.
An assessment of the performance looks at the progress that is being made by the project
relative to its objectives.
INDICATORS:
Indicators constitute a critical component of a project that shows changes in certain
conditions from specific interventions. They provide evidences of progress of the project
in attaining the development objectives. Project indicators are pre-established signs that
are relevant to good performance and to achieve project objectives.
2
Criteria and Indicators: The proper assessment of interventions made during the project
against the project objective is based on criteria, which are considered representative for
the decision to be taken. Criteria usually refer to certain parameters to be taken into
account in evaluation, such as production, input use and living conditions of the project
beneficiaries. The criteria are ultimately translated into indicators, which more concretely
represent yardsticks to measure the impacts
Selection of Indicators: Indicators should be selected during formulation of the project
when the objectives are also being established. The following questions should be
answered as part of the process of establishing the indicators. Such as, what are the
objectives of the project? Who are the target groups and what are their needs and
expectations? What changes are anticipated as a result of the project? And to what extent
and how efficiently is the project achieving its objectives (Figure 5)?
Quantitative, Qualitative and Proxy Indicators: Both quantitative and qualitative
indicators should be selected based on the nature or the particular aspect of the project
that is being assessed. Efficiency, for instance, leads itself easily to quantitative
indicators. On the other hand, measuring dynamic sustainability, which involves people's
adaptability to a changing environment, necessitates some qualitative assessment of
attitudes and behaviors. However, there are some methodologies such as beneficiary
assessment, RRA, PRA and structured interviews that can be used to convert qualitative
indicators into quantitative indicators. In case where complexity, cost and timeliness of
data collection prevent a result from being measured directly proxy indicators can be
used to reveal performance trends and make managers, administrators, planners and
donors aware of potential problems or areas of success.
Limiting the Number of Indictors: A good balance should be achieved between theory
and practice. That is what should be and what can be measured. An ideal set of
indicators includes indicators of relevance, performance and success. An indicator
should adequately account for the immediate and long-term objectives of the project,
ownership by beneficiaries and other stakeholders and cost-effectiveness of data
collection.
Characteristics of a Good Indicator: Popular code to remember the characteristics of
good indicators is SMART: S-Specific, M-Measurable, A-Attainable, R-Relevant, and TTraceable.
Collection and Analysis of Data for Indicators: Based on selected indicators, crosssection and/or time-series data are collected since the inception of the project and
analyzed (Figure 6). The project should establish types and sources of data required,
methods and frequency of data collection, methods of data analysis, personnel
responsible for data collection and analysis and the audience of the analytical results.
Uses of Indicators: The indicators are used during the project to measure progress,
including the identification of potential problems or successes. Finally, they will be a part
of the overall evaluation. Donors, NGOs and GO officers, all benefit from the
sophisticated data collection and analysis techniques. Criteria for designing a simple
evaluation system are cost, time, easy to use and analyze, easy to present results, results
useful for stakeholders. A few examples of indicators for impact assessment are yield of
R&D plot relative to farmers’ non demonstration plots yields, increase in production for
each crop in project areas, value-added of crops, coverage of new crops or new varieties
in project areas, total increase in income and nutritional values, training of farmers,
3
NGOs, entrepreneurs, extension and research officers, and other stakeholders depending
upon locations.
Output versus Outcomes: Output indicates what will be the result of number of
activities. For example, assume a project takes certain initiatives on training. Then
conducting training need assessment, developing training curriculum and conducting
training for demonstration cooperators would be necessary features of the initiative. The
first output will be a complete training curriculum, and the second will be a group of
trained personnel with better
techniques and skills.
Outcomes are immediate benefits
from the outputs. For example,
assume a Boro rice demonstration is
done with farmers and the yield is
6t/ha. This 6t/ha is output of the
project but outcome will be what
benefit farmers get by achieving this
level of yield. Assume the increase
in production is 1 t/ha, and then the
outcome depends on what they did with surplus production (Figure 6). Did they sale at a
greater price and used the surplus income for children education, cloths, health care,
nutritious food, fertilizer for the next crop or produced more rice. There may be several
such examples of difference between output and outcome. Relevance, performance and
success should be assessed in an integrated manner in order to have a sound basis for
making recommendations and drawing lessons learned from experience to improve
project quality.
Case Study: Impact of TCTTI
The process of impact assessment at Upazila level started after 4 years of project
implementation (Figure 1,2,3,4,5 and 6). The format for collection of data was developed,
reviewed and supplied to 64 UAOs (Contact R N Mallick drrnmallick@hotmail.com for
further information). Among many indicators let us examine one example of TCTTI.
Impact of BRRIDHAN-29 rice variety can be estimated by the adoption. In 1995 this
variety was not cultivated at farmers’ level. In 1996 the acreage for this variety was 3620
ha through TCTTI and in 2002 increased to 92,057 ha in the project area (Figure 6). At
the margin the increase of one ton per hectare is 92,057 tons of extra produce available.
Over 50% of the total produce was used as seed so value addition was in 276,171 tons of
rice seed which can be planted in over 11 million hectare so 11 million tons of extra rice
production by this one initiative in four years. Similarly, various other parameters can be
calculated to assess impact.
Approximately 4.6 million farmers have been reached indirectly through the direct
exposure to 134,423 farmers that were directly trained by the Project organized in 2,952
block demonstration on T. Aman rice, 1,571 block demonstrations of Boro rice, 187
block demonstrations of hybrid maize, 162 block demonstrations on wheat, 831 block
demonstrations on green manure and 109 demonstrations on grain legumes throughout 64
targeted Upazilas by June 2002. At the national level the impact can be assessed in terms
of promotion of sustainable agriculture, integrated water management, empowerment of
4
women, protection of the environment, skill development and research, information and
resource management and improvement of the performance of agriculture research of
these results.
Figure 1: Objectives in relation to Impact
5
Figure 3: Conceptual Framework of Impact Assessment-Criteria and Indicators
Figure 2: Levels of Impact at Individual, Household and Upazila levels
6
Figure 5 Level of Impact on Organizations and Hierarchy
Area covered by Brridhan-29 in 64 TCTTI Upazilas
100000
92057
90000
80000
70000
Hectare
64387
60000
50000
40000
34784
30000
20000
18237
10000
3620
0
1996
1997
1998
1999
2000
Year
Figure 6 Impact of Brridhan 29 rice variety with time series
7
Figure 4: Schematic diagram of PRA Methods and Approach, Mallick et. al. 2000.
Reference:
BRAC. 1998. Poverty alleviation and empowerment-The second impact assessment study of BRAC's Rural
Development Program.
CARE. 1996. Impact evaluation of income generation projects-A training course for NGO staff on
assessment of the economic and social benefits of Income generation projects.
IFAD 1986. Monitoring and evaluation guiding principles.
Mallick, R. N., K. G. Pillai, Q. T. Hossain, M. A. Awal, M. Nuruzzaman, Subash Dasgupta, and Wazed Ali
Shah. 2000. Participatory rural Appraisal Practical Handbook, GOB/UNDP/FAO, Dhaka, Bangladesh.
PCARRD 1990. Technology assessment for agriculture in the Philippines Book series No 92/1990.
Social Impact Analysis 1982 - A model and strategy for implementation in Development Assistance.
TCTTI 2002a. Highlights and Achievement. GOB/UNDP/FAO, Dhaka, Bangladesh.
TCTTI 2002b. Project Findings and Recommendation. GOB/UNDP/FAO, Dhaka, Bangladesh.
8