Scope of Work for MidTerm Evaluation of Food
for Peace, Title II
Development Food Aid
Projects
Guidance Notes
FSN/TOPS Mike
References:
Bonnard, Patricia. 2002. Title II Evaluation Scopes of Work. Food and Nutrition Technical Assistance
Project, Academy for Educational Development, Washington, D.C. 2002, USA
Catholic Relief Services (CRS). 2008. Consortium Alignments Framework for Excellence (CAFE),
Partnership, Program Quality and Support Department, Catholic Relief Services, Baltimore, MD, 21201,
USA
Management Systems International (MSI). 2011. Evaluation Stetements of Work. Good Practice
Examples, 600 Water Street, SW, Washington DC., 20024, USA
Office of Food for Peace (FFP). 2009. Food for Peace Information Bulletin 09-06, Monitoring and
Evaluation Responsibilities of Food for Peace Multi-Year Assistance Programs Awardees. US Agency for
International Development, DCHA/FFP/PTD, 1300 Pennsylvania Avenue, NW., Washington DC 20523,
USA
USAIDa. 2010. Performance Monitoring and Evaluation. TIPS, Preparing and Evaluation Statement of
Work.
USAIDb. 2012. ADS Chapter 203, Assessing and Learning.
Contents
1.
Introduction: ..................................................................................................................................... 1
2.
Description of the project2................................................................................................................ 2
3.
Evaluation Purpose and Objectives .................................................................................................. 2
4.
Composition of Evaluation Team ...................................................................................................... 2
5.
Methodology of the Evaluation ........................................................................................................ 3
5.1 Quantitative Survey ...................................................................................................................... 4
5.2
Validation of the Initial Observations ....................................................................................... 5
6.
Key Evaluation Questions ................................................................................................................. 5
7.
Project Responsibilities ..................................................................................................................... 6
8.
Deliverables....................................................................................................................................... 7
9.
Methodological Strengths and Limitations3...................................................................................... 8
10.
Presentations ................................................................................................................................ 8
11.
Timeframe ..................................................................................................................................... 9
Annex 1: Example of Key Evaluation Questions ............................................................................................ i
Guidance Notes on Scope of Work (SOW)
For Mid-term Evaluation of USAID/FFP Supported Title-II Development Food Aid Project
1. Introduction:
Mid-Term Evaluations (MTE) must be carried out approximately halfway through the life of a Title II
Development Food Aid Project award. The MTE of a Title II Development Food Aid Project should be a
formative evaluation instead of a summative. As a formative evaluation, the MTE represents an
opportunity to make mid-course corrections in program strategy as well as to address implementation
issues that are constraining effective implementation. The primary focus of the MTE is to review
implementation processes for producing planned outputs and the program logic reflected in the results
framework to determine whether the program is likely to achieve the intended results, objectives and
impact. The MTE seeks to identify successful strategies, bottlenecks, ways for improvements and
lessons learned rather than focusing on measuring progress compared to IPTT targets.
The following set of objectives1 can be used in designing an MTE for a Title II Development Food Aid
project.
a) Learn from successes and also acknowledge and learn from challenges and issues
b) Assess implementation progress and bottlenecks rather than focus on outputs, outcomes
and/or impact
c) Solicit systematic feedback from program participants and other stakeholders to get their views
on what is working and what adjustments need to be made.
This guidance note is produced to help Title II Development Food Aid Project awardees to develop a
standard Scope of Work for the Mid-Term Evaluation.
Following is an outline of a Scope of Work.
a) Introduction
b) Description of the project
c) Evaluation purpose and objectives
d) Composition of the evaluation team
i.
Scope of work for the Team Leader
ii.
Scope of Work for each members
iii.
Core competencies of the team members
e) Evaluation Methodology
1
Office of Food for Peace/USAID. 2002
f)
Key Evaluation Questions
i.
Technical components
ii.
General program quality and crosscutting areas
iii.
Implementation processes
g) Project Responsibilities
h) Deliverables
i) Presentations
j) Timeframe
2. Description of the projectError! Bookmark not defined.
a)
b)
c)
d)
e)
Goal of the project, including strategic objectives and intermediate results
A brief description of key interventions and implementation strategies
Targeted Impact Groups and geographic coverage of the project
Description of key partners and how activities are coordinated
Implementation history and issues to date, which may include:
i.
Operating environment (i.e., political, institutional, climatic, economic)
ii.
Factors affecting implementation
iii.
Changes in targeted populations
iv.
Changes in the design of key interventions
3. Evaluation Purpose and Objectives
The purpose statement in an evaluation SOW explains why an evaluation is being conducted. The
purpose of a Mid-Term evaluation is to improve program or project effectiveness or to inform
specific future decisions. MTE is a formative evaluation to make mid-course corrections in
program strategy and to make recommendations for improving the effectiveness or efficiency of
implementation. Evaluation objectives include those listed above in the introduction as well as
any unique topics that need to be investigated for a particulalar program, such as a major issue
affecting implementation, a major constraint or problem that the program is facing or an
important cross-cutting issue.
4. Composition of Evaluation Team
The composition of the MTE team depends on the technical components of the project, general
program quality issues and implementation processes to be evaluated. The number of team
members will depend on the need for specific technical expertise and resource availability.
2|Page
It is a better practice to assign one person as the Team Leader (TL). This person can also cover one
or more of the technical areas. The role of the TL is to work with the project management on
evaluation planning and logistics, to work with the evaluation team to allocate evaluation topical
responsibilities, especially those related to implementation processes, based on their skills and
capacity, to provide support to evaluation team members to insure that all team members fulfill
their obligations, to organize and facilitating team interaction, to plan and facilitate a “Validation”
workshop to validate the initial observations and interpretations, and to plan in-country
presentations to program stakeholders and the donor(s). The individual who would fill this position
should possess strong evaluation management skills and prior experience in leading evaluation
teams. TL needs to be given sufficient time to carry out the responsibilities.
The Scope of Work of the team leader and the team members need to be defined and included in
this section. In addition it is suggested to include the core competencies of the team leader and the
team members in this section.
5. Methodology of the Evaluation
In this section, you need to describe the methodology of the evaluation. Food for Peace (FFP) does
not require an MTE to be quantitative or population based. FFP encourages participatory qualitative
methods for Mid-Term evaluation. However, evaluators should certainly review available
quantitative information being generated by the M&E systems and should also be thinking about
how the Final Evaluation will be able to assess impact (see the section below on Knowledge
Management).
The Mid-Term Evaluation needs to use a qualitative and participatory approach, which should
include all key stakeholders, including representatives from the government, field staff,
implementing partners, consortium members, community volunteers, farmers and mothers’ groups.
There will be significant involvement from staff (include partner/ consortium member staff) in the
project and country office in all aspects of evaluation including developing lists of project sites,
sharing project documents, answering questions, sharing successes and challenges, recommending
changes, validating data collected by the MTE team, help interpreting the results, and reviewing
draft reports. The qualitative methods tend to use purposive sampling hence if the MTE team’s
observations do not represent the overall project it is important for the project staff to point this
out to the MTE team. The focus will be on improving the process of project implementation (i.e;
understand the quality of services delivered by the project, and how the results are being achieved,
challenges and solutions for improving outcomes).
The MTE team should review the project proposal, baseline report, annual reports, routine
monitoring reports, outcome monitoring reports, and all other special study reports produced by
the project as well as other relevant qualitative and quantitative information to determine the
targeting, project strategies, service delivery mechanisms and the quality of services, quality of
3|Page
project results, evaluate the theory of change, quality of program management, cross cutting issues,
and the likelihood of achieving the targets.
In addition, the evaluation team should use qualitative assessment tools and techniques for primary
data collection (e.g., focus group discussions and key informant interviews with beneficiaries,
implementing staff, partners, USAID/Food for Peace, other donor agencies, organizations operating
in the area, relevant government staff; observing the sensitization sessions and other service
delivery mechanisms used by the project) to complement the available data. These could include:






Key informant interviews with key project personnel and stakeholders –USAID, local
government organizations, relevant government ministries, local community committees
and selected community participants.
Review of project records – to establish outputs and financial accountability.
Direct observations of service delivery sessions through site visits- assess technical practices,
quality of activities, confirm recorded outputs, and assess likelihood of achieving outcomes.
Focus group and group interviews with the project participants (household and institutional)
– to determine benefits of the project, who receives the benefits, quality of service delivery,
perception of the communities, unanticipated consequences and possible areas of
modification and design.
Interview project staff to determine the quality of service delivery and level of knowledge
on the subject matter
Review of program and organizational documents – to assess institutional strengthening
activities of various partners and organizations.
The MTE team should also conduct data quality assessments to determine the quality (validity,
reliability, timeliness, precision, and integrity) of data collected and reported; M&E structures and
responsibilities, data collection and reporting forms/tools, data management processes; data quality
assurance mechanisms, and M&E capacity and system feedback mechanisms.
5.1 Quantitative Survey
Should the project decide to conduct a quantitative survey as a part of an MTE, the survey needs to
be carefully designed to achieve the objective of the evaluation.
Typically the objective of a quantitative survey in an MTE is to measure progress on key outcome
indicators and to evaluate the theory of change/ development hypothesis of the project.
The Mid-Term survey should be designed to collect information from beneficiaries (not a population
based survey). It is a better practice to collect and analyze quantitative survey data before the
qualitative review takes place. This will allow the MTE team investigating the reasons for poor
4|Page
adoption of strategies promoted by the project and also strategies that are well adopted. The
quantitative data will also provide an insight to identify bottlenecks and recommend changes.
5.2 Validation of the Initial Observations
The MTE team should plan to present its initial observations to the project staff (including partners) to
validate their findings and interpretations. In the event of extreme disagreements, the MTE team may
need to revisit the data or communities to make sure that their findings are based on valid and reliable
information.
6. Key Evaluation Questions
SOW for a Mid-Term Evaluation is likely to focus on processes and often include questions about a
project’s approach, implementation management, or relationships among project partners, as well as
questions about a project’s early results—many of which may have been anticipated and budgeted for
during project design2.
It is important to outline a set of key evaluation questions by topical area in the Scope of Work. This
would clearly communicate your expectation from the MTE and will guide the MTE team to develop
topical outlines and survey instruments. The questions to be outlined in the Scope of Work will help the
MTE team to focus on the key evaluation questions. The following guiding principles can be used to
develop a list of key evaluation questions:
a) Consider the time, budget and resources available to the evaluation in developing the questions.
While a participatory process ensures wide ranging input into the initial list of questions, it is
equally important to reduce this list to a manageable number of key questions. Keeping in mind
the relationship between budget, time, and expertise needed, every potential question should
be thoughtfully examined by asking a number of questions3.
b) Review the Theory of Change (ToC) or Development Hypothesis (DH) of the project to identify
key evaluation questions; The ToC/DH should be the basis of most of the key evaluation
questions.
c) Identify all evaluation questions for which gender-disaggregated data are expected; also identify
questions for which an examination of gender specific or gender differential effects are
expected4;
d) Each evaluation question should be answerable with the highest-quality and most credible
evidence possible, given time and budget constraints. To ensure that a team will be able to
gather adequate evidence to sufficiently address each of the evaluation questions a limited
number of evaluation questions be included in a SOW.
e) The questions should be clearly defined, precise and “researchable”. It is important to ask how
important and essential the question is to the purpose and the users of the evaluation.
List of key evaluation questions can be organized by technical sectors, program quality and crosscutting
areas and implementation processes.
2
MSI, 2011
USAIDa, 2010
4
USAIDb, 2012
3
5|Page
a) Technical sectors
i.
MCHN
ii.
Agriculture and Natural Resource management
iii.
Water, Sanitation and Hygiene
iv.
Disaster Risk Reduction
v.
Education
vi.
[insert technical areas specific to your project]
b) Program Quality and Crosscutting Areas
i.
Targeting
ii.
Integration
iii.
Sustainability/ Exit Strategies
iv.
Gender
v.
Social and Behavior Change
c) Implementation Processes
i.
Monitoring and Evaluation
ii.
Knowledge Management
vi.
Partnership/ Consortium Quality
vii.
Commodity Management
iii.
General Management
 Human Resource Management
 Financial Management
This guide includes example of questions by topical area in Annex 1. The long list of questions is only to
help you to develop specific questions for your project. However the actual list of questions for your
evaluation will be much shorter and will depend on your project components, purpose and scope of the
evaluation; and resource availability.
7. Project Responsibilities
The project management will be responsible for the following:




Provide administrative and logistics support to the evaluation team
Ensure effective coordination of the Mid-Term evaluation logistics, scheduling appointments
with the stakeholders, coordinate with the consortium members or partners to facilitate the
consultants in undertaking their assignment including travel, stay, and per-diem
Provide consultants with background documents, reports, data, materials
Develop a list of all operational communities, classify them based on the progress (i.e., high
performing, moderate, poor performing) and make it available to the MTE team
6|Page

Share the draft report with all key stakeholders including key project staff, partners, donor
representative and government; provide them with a timeline to forward their feedback to a
point person in the project; review comments, compile them and forward the consolidated
comment to the evaluation team.
8. Deliverables
The consultants will be responsible to produce a report [combining qualitative and quantitative data5]
using the following outline:
i.
ii.
iii.
iv.
v.
vi.
vii.
5
Executive summary
Background
a. Overview of Project Strategies
b. Project History and Operating Context
c. Evaluation Methodology
Evaluation purpose and objectives
Intermediate Results 1.1 under Strategic Objective 1
a. Brief description of interventions
b. Service delivery strategies and approaches – quality, successes and challenges
c. Implementation progress and achievement of results
d. Meeting targets
e. Other achievements
f. Discussion of general evaluation questions
Intermediate Results 1.2 under Strategic Objective 1
a. Brief description of interventions
b. Service delivery strategies and approaches – quality, successes and challenges
c. Implementation progress and achievement of results
d. Meeting targets
e. Other achievements
f. Discussion of general evaluation questions
Intermediate Results 2.1 under Strategic Objective 2
a. ….
[Similar sections for all intermediate results]
Program Quality and Crosscutting Areas
a. Partnership/ Consortium Quality
b. Targeting
c. Gender
d. Social and Behavior Change
e. Integration
If a quantitative survey is carried out as a part of the Mid-Term evaluation
7|Page
viii.
ix.
x.
f. Sustainability/ Exit Strategies
Implementation Processes
a. Commodity management
b. Monitoring and Evaluation
c. Knowledge Management
d. General Management
 Human Resource Management
 Financial Management
Recommendations
a. Critical Priority Recommendations
b. Other Recommendations
Annexes
a. List of acronyms
b. Evaluation SOW
c. Evaluation Plan and Schedule
d. Composition of the team
e. Evaluation Methods and tools (topical outlines and survey questionnaire)
f. List of sites visited
g. List of key informants and communities visited
h. Updated Indicator performance tracking tables (IPTT)
i. Summary Tables on Finance, Commodities and Human Resources
9. Methodological Strengths and Limitations2
USAID’s evaluation policy states that any methodological strengths and limitations are to be
communicated explicitly in evaluation SOWs. When likely study limitations are described in an
evaluation SOW, the users of the reports and USAID are in a better position to decide whether to go
ahead with the evaluation in spite of these or to adjust the evaluation questions, timeline, or budget to
eliminate or minimize their impact on the quality of the evidence an evaluation yields.
10.
Presentations
The Scope of Work should outline the presentation requirements and the potential audiences. It is a
good practice to plan for two presentations. For the internal audience (i.e., project managers,
technicians, and implementing partner organizations) a detailed presentation of the findings including
preliminary recommendations is preferable. However a formal presentation of the key findings needs to
be made to Food for Peace representatives, Government counterparts and other stakeholders.
8|Page
11.
Timeframe
The Scope of Work should offer a realistic timeframe for the key evaluation activities. Following is an
example6 of key activities which can be adapted to your evaluation context. It is important to develop a
realistic timeframe.
Key evaluation activities
Pre-Planning
 Meeting with the partners to plan for the Mid-Term evaluation, finalize Scope of
Work and finalize core competencies and level of experience of the evaluation team
members and the number of member.
 Contact possible Team Leader / member candidates/ firms
 Finalize the evaluation team (members and the team leader).
 Organize all documents and make them available to the evaluation team
 Develop a list of all operational villages/ communities (by partner) and classify them
by level of project performance (i.e. excellent, moderate, and needs lots of support)
Planning
 Review of existing reports, documents and data (quantitative survey, project records)
 Develop qualitative survey tools
 Develop an evaluation plan
 Develop and share final implementation and logistic plan with stakeholders
 Identify project staff who will participate in the review process
 The evaluation team select sample communities/ villages
 Arrange all logistics
 Discussion with partner and staff to ensure that appropriate steps are being taken
before implementation
Implementation
 Introductory meeting between the evaluation team and the stakeholders (partners,
key project staff, donor)
 Briefing of Mid-term review team (objectives, organizations, methods, approaches,
tools) and finalization of qualitative study tools
 Field work (interviews, FGDs, observations, analyses, triangulations)
 Make presentation on the preliminary observation to validate the findings and
interpretations;
 Re-visit sites (if there are extreme disagreements on evaluation team findings
between the evaluation team and the project staff)
 Make a detailed presentation of the results including recommendations to the
consortium members,
6
Adapted from PROSHAR (ACDI/VOCA) Scope of Work for Mid-Term Evaluation, 2012.
9|Page
Timeframe

Make a summary presentation of the key findings to Food for Peace and Government
counterparts.
Reporting
 Prepare a draft report following the guideline and share with the prime awardee;
 Awardee shares the draft report with the key stakeholders, collect and compile
comments and forward to the evaluation team
 Address the comments and incorporate inputs from the stakeholders
 Finalize the mid-term report
10 | P a g e
Annex 1: Example of Key Evaluation Questions
Technical Components
You may want to organize the technical areas by Strategic Objectives and Intermediate Results. The
following examples of key questions may help you to develop your project specific questions.
Evaluation Team is expected to answer the following questions (if a program is implementing
interventions in a specific area referenced by the question).
a) Example of MCHN Questions














Is the level of effort among different interventions and activities relevant to the MCHN problems
facing the community?
Are there clear linkages and coordination with public and private health, nutrition and social
services in the community?
Does the community have a clear understanding about the services offered by the project and
who are eligible to receive them?
How successful has the activity been in increasing access to government preventative and
curative health and related social services?
How successful has the activity been in getting the community linked with potential government
financial resources (i.e., construction of WatSan related infrastructures)?
Do field staff and community volunteers have appropriate BCC materials that focus on a limited
number of harmonized messages for optimal infant and young child feeding?
Are staff qualified and knowledgeable of the purpose and methods used in the program?
What is being done to improve the capabilities of the staff and local partners to respond to
community needs and meet the objectives of the program?
Are structured supervision plans and tools in place to monitor those responsible for providing
health and nutrition assessment/care/treatment?
Is there a problem with the attrition of community health or nutrition volunteers? If so, why do
they drop out and what can be done to reduce the rate of attrition?
What is the quality of growth promotion sessions? Has there been follow up discussions with
the mothers?
Are staff aware of common micronutrient deficiencies? Do they know how to link people to
available services? If the project has a micronutrient activity, are staff and community
volunteers clear on recommended behaviors and the rationale for these?
Are the program eligibility and graduation criteria appropriate given the objectives and
assumptions of the program? If not, how should it be modified?
Which MCHN strategies have been more effective so far and why? What new knowledge have
clients acquired and what knowledge is lacking? Has new knowledge translated into new



practices and recommended behaviors? If yes, what are the contributing factors to this
achievement? If no, does the program understand the obstacles to optimal behavior and have a
plan to address?
Do those responsible for promoting health and nutrition behavior change have SBC
materials that are appropriate – tailored to the user, actionable, accurate and linked to growth
promotion messages? Which materials need strengthening, if any, and how? Are users of these
materials able to select the appropriate messages and provide effective counseling? If not, what
skills need strengthening?
Through what process were the health and nutrition behavior change and communication (BCC)
materials developed, tested and applied?
Is there an exit or graduation strategy developed?
b) Example of Agriculture, Natural Resource Management and Income Generation Activities
Questions
General






What are the main opportunities addressed by the project e.g. increasing volume of production,
increasing volume of sales, improving the quality of produce, building business expertise, etc.?
Does the program take adequate advantage of these opportunities? What has worked really
well and what needs to be addressed and adapted going forward?
Are there ways to enhance the effect of income-generating activities on household food security
beyond what has already been done or planned?
Have any new potential practices or tools been used during the project that could be replicable
elsewhere?
What are the main agricultural and natural resource management elements hindering the
progress of the project, and what new opportunities have been identified that could be
addressed in the remaining part of the project?
What is the likelihood of sustaining the increased agricultural production (vegetable, fish etc)
and/or value addition methods introduced during the project? Are microenterprises targeted by
the project likely to sustain their operations without project support? If not, why and what can
be done to enhance their sustainability?
What systems or activities have been put in place to ensure sustainability, and how likely is that
system to be sustainable beyond the life of the project?
Technologies and practices

To what extent have project promoted agricultural and natural resource management
technologies and practices been adopted by project participants? What gaps are there that still
need to be addressed?
ii | P a g e







Are the technologies and practices being promoted established and suitable to the local agroecological environments?
If demonstration plots have been used during the project, do they accurately reflect the real
conditions facing farmers? What challenges have been identified in using these plots, and what
needs to be changed to improve them?
What are farmers’ perceptions about the project promoted technologies and practices and their
perceived benefits?
Will farmers be able to continue project promoted technologies and practices beyond the life of
the project?
Are farmers and other stakeholders able to access technical support related to agriculture and
natural resource management practices without program assistance (free or subsidized)? If not,
what would be required in order that they could do so?
Are the project promoted activities linked to the Department of Agriculture, fisheries, livestock
or the national or international agricultural research centers and has a communication protocol
been established? Are the linkages sustainable? If not, why and suggest solutions.
If the project has introduced productive infrastructure, such as small irrigation systems, are
farmers and other stakeholders able to maintain those new infrastructures on their own? If not,
why and what could be done to address this limitation?
Market access








Has the project conducted a comprehensive value chain analysis? What activities were initiated
in response to the value chain analysis and how successful have they been? What were the main
challenges related to this?
Is there an accessible market for the products or services produced by the microenterprises
promoted by the project?
Did the project introduce market linkages? If so how successful have they been, and how likely
are they to remain after the end of the project?
If farmer groups have been established, what capacity do they have to sustain the established
relationships with the private sector (producer, buyers, suppliers, retailers etc.)?
If collection points have been established during the project, are they functioning regularly and
are they likely to be sustained? If so how?
Did the project conduct an assessment of gender roles in production, resource management,
control over resources and workload in developing project strategies for this sector? How were
the results used in developing or revising the strategies to improve an equitable participation
and benefits?
Does the project have systems in place to address gender equity in promoting the techniques
related to agriculture and natural resource management? What could be done to improve
equitable participation, workload distribution, and the benefits from interventions?
Are the income generating activities promoted by the project appropriate for the most
vulnerable and geographical context? If not what could be the better option?
iii | P a g e
Access to finance

Are micro-finance programs (VS&L – Village Savings and Loans, Micro-finance institutions)
designed and implemented according to standard best practices? If not, why and how can credit
practices be improved?

Are there certain groups/individuals within the target population better able to access loans,
and why? If there are groups/individuals who are unable to access loans, should the program be
broadened to include these groups and how can the program be modified to incorporate them?
Have microfinance activities contributed directly to household food security, and how?

c) Example of Disaster Risk Reduction Questions








Which activities have been most effective as perceived by the community? Which activities
were not successful and why? How might these activities be improved?
How well did the DRR activities integrated with interventions to promote food access,
availability and utilization?
How well the activities integrated with the government promoted emergency preparedness or
disaster risk reduction activities?
What roles did the communities and individuals have in the development of the DRR activities?
Are government and community stakeholders aware of the project activities and outputs? How
successful has the program been in assuring government buy-in/support?
What evidence, if any, exists to indicate improved community preparedness for response to and
recovery from disasters?
Are there additional direct or indirect benefits derived from infrastructure construction or
rehabilitation that are not currently being captured?
Are there any unintended negative environmental impacts stemming from infrastructure
activities? If so, are there sustainable mitigation measures being implemented? What additional
measures can be implemented?
d) Example of Water, Sanitation and Hygiene (WASH) Questions


What strategies are employed to influence water, sanitation and hygiene (WASH) behaviors of
men, women and children? What is the level of successes of these strategies in influencing
WASH behaviors of target population? What is the quality (i.e. interactive sessions, length of the
session, topic selection, quality and use of visual aids, quality and use of flipcharts, participation,
and demonstrated knowledge of the facilitator on the topic) of behavior change sessions? How
it can be improved?
Which WASH activities have been most effective in addressing assessed needs? Which activities
were not successful and why? How might these activities be improved?
iv | P a g e








Did the project build or distribute or rehabilitate potable water system (s) (borehole pump or
tube-well) in the community? Are all of the pumps working? How they are maintained? Who are
benefiting from the water systems? Who were excluded?
How has the community been involved in deciding the type and location of water sources? How
the sources have been maintained? Are there examples of community maintained water
sources? What are the key factors behind the success of this model? What are the challenges in
community led management of water points? How it can be addressed?
Is there a functioning water committee? Does the water committee have a savings account?
Are there households in the community that do not receive this water supply? If yes how do
these households get water? Why don’t these household receive water?
Does the water committee collect monthly fees from the households to maintain the water
system? Are the monthly fees collected from the households enough to pay for the operation
and maintenance of the water system? Where do the necessary funds come from for
maintenance or repair?
What are men and women’s preferences for location of water points? What preferences are
there for design and technology?
What communication methods are preferred by women and men to provide them with the
information they need about the WASH activities? Are these formal or informal? Are there
differences between women and men in willingness and capacity to pay for water services?
What is the level of training of staff to promote health and hygiene interventions? Is there
knowledge about behavioral change and its links to project hierarchy adequate? What
additional training, if any, would be recommended for project staff?
e) Example of Infrastructure Questions78



Does the design and implementation of the infrastructure adhere to local technical codes and
guidelines? If not, why not, and how can the Development Food Aid Project activities be brought
up to code? What is the quality of the infrastructure? What is the perception of the community
about the quality and effectiveness of the infrastructures?
Did the project follow USAID environmental regulations in constructing these infrastructures?
How the indicators with negative determination with conditions are being monitored? Are there
any unintended negative environmental impacts stemming from infrastructure activities? If so,
are there sustainable mitigation measures being implemented? What additional measures can
be implemented?
Has the rehabilitation of rural roads had an effect on seasonal road use and transportation time
and costs for the targeted beneficiaries? Has it had an effect on seasonal availability of food and
other important commodities?
7
Bonnard, Patricia. 2009
Infrastructure is often not stand alone in a project strategy, but rather supports other components including the
ANR component, WASH and the DRR component.
8
v|Page


What mechanisms have been developed to maintain the infrastructures created or rehabilitated
by the project after the project ends? How has the system been working? What could be done
to improve the system?
Are there additional direct or indirect benefits derived from infrastructure construction or
rehabilitation that are not currently being captured? Do food/cash for work activities compete
with or complement the demands for household labor in other productive activities?
f) Example of Education Questions
Key Questions on What has the Project Done?
o
What kinds of activities has the project undertaken? What has the project done, what
the community has done in these activities?
Key Questions on Who Has Benefitted and How?


Who has benefited directly from the activities of the project? How have they benefitted? From the
following:
o Enrollment
o Formal performance of students on exams
o Student attentiveness
o Other dimensions of early childhood development
o Food security in the household of the student (more food for others, changes in diet)
o Changes in health and nutrition behavior of children and parents
o Changes in livelihoods strategies of students’ household (transhumance patterns
o Improved learning environment (performance of teachers and other staff)
o Performance of school committees or PTAs
o Other impacts?
What criteria were used to identify direct beneficiaries? Who else has benefitted indirectly from the
project and how have they benefitted?
o Other family members, parents and siblings
o Others in the community
o Others in other communities
Key Questions on How Long the Benefits Will Last?

The project will end in a few months, how will these benefits change after the project ends and
why?
o Which will continue as they are now?
o Which will increase or expand?
o Which will reduce or disappear?
Key Questions on Best Practices and Lessons Learned?
vi | P a g e


Of all of the things the project has done, which have had the greatest impact during the life of the
project and why?
If we were to do this project over again from the start, what should be done differently and why?
General Program Quality and Cross Cutting Areas
a) Example of Partnership and Consortium Management Questions












9
Did the partners/ consortium members develop a common vision for the partnership that is
agreed by appropriate levels of each agency9?
Did the partners/ consortium members develop a set of common criteria for excellence in
internal partnership/ consortium management: programmatic, financial, and
managerial accountability for service delivery to communities and/or project participants,
compliance to donors, and to each other?
Did the partnership/ consortium members agree to a set of appropriate financial,
administrative, and managerial systems that are based on consortium needs and each agency’s
strengths?
Did the consortium develop a contingency plan to address unforeseen shocks to the project or
to the partnership?
Whether the partners/ consortium members defined their roles based on capacities of each
agencies and the need of the project?
Has the partnership/consortium developed an operations manual documenting administrative,
financial, and human resource processes and procedures to remain in compliance with host
nation law and donor requirements?
Does the operations manual include an accountability-based performance evaluation process
that balances performance with resources?
Does the consortium or partnership have a guideline for conflict resolution, communication, and
decision-making protocols that reinforce transparency and accountability at all levels of the
consortium?
Whether the consortium/ partnership developed policies and procedures to respect the human
dignity of each person (consortium members, project participants, stakeholders, and other)
without regard for organization, job responsibility, or personal identity;
Did the consortium members conduct business in a transparent, timely, and respectful fashion;
and work to build a consortium based on trust and mutual respect, consistently supporting
positive interpersonal behavior;
Did the consortium members put the needs and identity of the consortium ahead of individual
organizational needs when representing the consortium?
Whether the consortium/ partnership allocates adequate10 resources to monitoring, evaluation,
learning, and knowledge management systems; supports staff in learning, change, and
Catholic Relief Services (CRS). 2008
vii | P a g e







innovation; and creates and sustains a culture that continually improves its management
practice from lesson learned, both failures and successes
How effective11 the partnerships in planning, developing implementation strategies, reviewing
monitoring reports, documenting and sharing lessons learned and taking programmatic policy
decisions. Provide specific recommendations to improve the quality of partnership.
Did the partners or consortium members periodically meet to review project progress, and
implementation challenges?
Which strategies have worked in encouraging partner participation and involvement? Which
strategies did not work?
Is the program effectively developing the capacity of counterparts and/or partners? If so, how?
Have there been unexpected developments relating to the external environment that called for
a change in strategy or plans? Was the process used to manage this change effective at all
levels of implementation? How could this process be strengthened?
Does the consortium have an open and transparent process for decision making and
arbitration? Can it be improved?
Have there been unexpected developments relating to the external environment that called for
a change in strategy or plans? Was the process used to manage this change effective at all
levels of implementation? How could this process be strengthened?
b) Example of Targeting Questions




Who from among the programs intended beneficiaries is benefiting from these activities? How
are they benefiting?
How would these beneficiaries be described on the poverty/food insecurity/vulnerability
continuum? For example, are they from the "poorest" end of the continuum, from somewhere
in the middle, or from the "top" end of the continuum?
How does the achievement on target numbers of people compare to proposed targets? Is the
program on-track to achieve its target beneficiary numbers?
Who else benefits from program activities who may not have been an intended beneficiary?
How are they benefiting?
10
Human and financial resources required to carry out monitoring activities agreed by the consortium maintaining
the agreed upon standard.
11
Attributes of effective partnership includes:
a) Open and transparent communication and decision making processes among the partners;
b) Common understanding of both individual and joint responsibilities of partners;
c) Involvement of key stakeholders in all stages of partnership and program development;
d) Strong leadership role taken by the prime or the awardee;
e) Flexibility in terms of scheduling meetings, adjusting roles and responsibilities, being willing to adapt to
changes in planning and implementation, and accepting differences in philosophy, management style, and ability
to contribute to the project; and
f) Mutual respect in the areas of organizational expertise and accepting others’ judgment.
viii | P a g e

Who has been adversely affected by program activities? How have they been affected?
c) Example of Integration Questions
With the Govt. Poverty Reduction Strategy




How well does the strategy of the project align with government strategies as reflected in the
PRSP?
Which activities in the project are making the largest contributions to the PRSP?
Which activities in the project may be creating or strengthening obstacles to achieving progress
described in the PRSP?
How do project activities interface with government systems? How effective is this? How could
this be changed to make these links more effective?
With USAID Country Strategy
 How well does the strategy of the project align with the development strategy of the USG?
 Which activities are making the largest contributions to the USG strategy?
 Which activities in the project may be creating or strengthening obstacles to achieving progress
described in the USG strategy?
 What other program activities funded by the USG are being implemented in the project areas?
How does the project interface with these other activities? How should this be changed?
Within each project partner/ consortium member
 How does the project fit within the overall development strategy of the partner?
 What are the biggest contributions that the project makes to the development strategy of the
partner?
 What other programs are being implemented by the partner in the same geographic areas as
this development food fid project? How does the project interface with these?
 What other programs are being implemented by the partner in the same sectors as the LIAP but
in different geographic areas? How does the LIAP interface with these? How do experiences
get exchanged across different projects in the same sector?
Within the project


What proportion of the project’s beneficiaries are common beneficiaries who have received
support to address their supply, access and utilization issues?
What potential synergies within the project are not being capitalized upon in terms of
complementary program activities that could be extended to the same beneficiaries?
d) Example of Sustainability Questions
ix | P a g e





What behavioral change has been induced by the project that needs to be sustained after the
project ends? What threats exist that could cause the behavioral change to revert to previous
practice? What external organizations, services, structures or relationships are needed to
support the behavioral change?
Is there an exit/ sustainability strategy developed for this project? Does it clearly define the
approaches, criteria for exiting, measurable benchmarks, timeline, actions steps, responsibilities
of different stakeholders and staff, indicators to measure progress, and a mechanism to assess
progress?
What services are being established or strengthened by the project that are intended to be
continued beyond the life of the project? What are the main threats to enabling these services
to survive? What is likely to happen to these services after the project ends? Strategies to
address these threats.
What physical structures are being constructed or rehabilitated by the project that are intended
to be continued beyond the life of the project? What are the main threats to maintaining these
structures? What is likely to happen to these structures after the project ends? Strategies to
address these threats.
What relationships are being established or strengthened by the project that are intended to be
continued beyond the life of the project? What are the main threats to enabling these
relationships to survive? What is likely to happen to these relationships after the project ends?
How is the program addressing these threats? Strategies to address these threats.
f)
Example of Gender Questions

Has there been a gender strategy either developed or adapted by the project? How well the
strategy has been implemented? Did the program offer some type of gender training at program
start-up? If so, have annual refresher courses as well as training for staff arriving to the program
after the initial training been offered?
What is the mix of male and female staff in the organization, and in the project by level. Why
proportion of female staff is low (if it is)? What has been done to improve the sex ratio of male
and female staff? Please provide realistic suggestions in improving the situation.
Is the working environment supportive of working professionals? Why are more female not
coming to work for the program? What type of challenges do female professionals face in
performing their work? What could be done to improve female participation?
Are the intended targeted beneficiaries (men, women, boys, girls) for specific program
interventions being reached? What are some of the benefits of the interventions targeting
specific gender groups? Have negative impacts of program interventions on gender been
observed? If so, what are they and what is being done to mitigate these impacts?”



x|Page



What types of benefits do female beneficiaries receive in light of additions to workload program
interventions may bring? Is there any evidence of women empowerment12 because of the
program?
Whether or not the program has accurate records around beneficiaries. Is the project
demanding more time from women? Has the work burden of women increased because of the
program? How does the community see women’s participation? What do the men and other
household members (for example mother-in-law) think about women’s participation in the
project?
Does the M&E system collect and process gender-sensitive, sex and age disaggregated data to
track outputs and outcome level changes?
g) Example of Social and Behavior Change Questions

Formative research on behaviors:
What sort of formative research on determinants of key behaviors has been conducted as part
of the project? How have the results of this formative research been used to guide program
interventions, activities and messaging?

Questions Coverage:
What is currently being done in the project to reach a high proportion of beneficiaries with highquality, persuasive messages and activities for behavior change? What appears to be working
and what is not? Are there ways in which the approach could be modified to bring about
improved behavior change?

Question on influencers:
Through what means have the project staff members or volunteers targeted husbands,
grandmothers, and other "influencers" (in addition to targeting mothers) with behavior change
messages and activities?

Question on quality of SBC key processes:
12
Female empowerment is achieved when women and girls acquire the power to act freely, exercise their rights,
and fulfill their potential as full and equal members of society. While empowerment often comes from within, and
individuals empower themselves, cultures, societies, and institutions create conditions that facilitate or undermine
the possibilities for empowerment. Dimensions of empowerment may include decision making power over
agricultural production activities including crop, livestock, poultry and fisheries; decision making power over microenterprises; ownership, access to, and decision making power over productive resources such as land, livestock,
agricultural equipment, consumer durables, and credit; decision making power over heath, nutrition and family
planning choices; control over income and expenditures; allocation of time to productive and domestic tasks;
satisfaction with the available time for leisure activities and increased participation in decisions around disaster
preparedness.
xi | P a g e
What tools are being used by project staff to measure and improve quality of key SBC process
(e.g., group facilitation, individual counseling)? What tools could be added to assure highquality SBC processes?
Implementation Processes
a) Example of Monitoring and Evaluation Questions
M&E Design
 Does the IPTT have robust indictors to measure a specific result stated in the Results Framework
(RF)? Does the IPTT have all of the FFP required standard indicators? What is the balance of
indicators? Does the project have too many indicators for each results stated in the Results
Framework?
 Does the IPTT include gender-sensitive indicators appropriate to the RF?
• Whether the targets are reasonable? Are they ambitious enough to make a difference to the
project beneficiaries? Are the targets justifiable -- based on or agency’s past experience with a
Title II project or another similar project? Comparable to the targets for similar interventions in
the country? Are targets clearly defined as cumulative or not? Have the targets been revised
using the baseline survey data?
 Does the project have a clearly defined comprehensive M&E plan that detailed out all of the
monitoring activities, tools, methodologies, frequency, analysis plan, data quality assurance
plan, communication strategy, data usage strategies, and feedback mechanism?
M&E Implementation
 How functional the routine output monitoring system is? Does it have well developed data
collection forms, data flow diagram, defined responsibilities for each partner and/or sector,
aggregation levels, and a clear link showing how the information will be aggregated and
reported onto the Annual Results Report and SAPQ? What percent of staff time is devoted to
routine monitoring?
 To what extent is the monitoring system operational? How this could be improved?
 Does the system produce reports ontime? Do the users find the reports helpful? Does the report
present data in an easy to understand format? Does it help the technicians, managers, front line
staff and partners to track the output level progress against targets? Does the report offer
interpretation of the data presented? Perception of the users about the quality of the
effectiveness of the system? How this can be improved?
 Does the monitoring system measures outcome level changes? Does the system use appropriate
methods and tools? How often it measures outcome? Is the system too burdensome to the
M&E team?
 How often the system produces outcome monitoring reports? How useful the reports are to the
users? Does the report present data in an easy to understand format? Does it help the
technicians, managers, front line staff and partners to measure adoption of project promoted
xii | P a g e

techniques? Does the report offer interpretation of the data presented? Perception of the users
about the quality of the effectiveness of the system? How this can be improved?
Does the outcome measurement system also investigates and present reasons behind the
successes and/ or under achievements? How useful these findings are to the users? How this
can be improved?
Data Quality
 What is the quality of data collected and reported through the M&E system? How does the
project manage data quality?
 Does the project have a data quality insurance mechanism? How effective the system is? How
this can be improved?
 Does the system insure all five dimensions of data quality – validity, reliability, timeliness,
integrity and precision? How this system has been working? How this can be improved?
 Does the project have an internal data quality assessment system? What is the quality of the
tools used for DQA? How has the system been working?
Use of Data in Project Management
 How does the programmatic decisions being made by the management? Are these based on
evidence? What types of evidences were used in making decisions?
 Is there any evidence suggesting that the project management and technicians used M&E data
to make programmatic decisions? How effective the communication strategy of the monitoring
results? Which communications strategies are useful and what needs to be improved? How it
could be improved?
 Is there a systematic approach to learn from successes and also acknowledge and learn from
problems? How the lessons are being captured, documented and communicated to the
stakeholders? Is the system working? What could be done to improve the learning process?
 What are the challenges faced by the M&E team in influencing project decisions through M&E
data? How they could be overcome?
M&E Capacity
 What is the M&E capacity (staff, budget, core competencies of staff) of the project vis-à-vis the
M&E roles and responsibilities? What can be done to improve the capacity?
 Does the organizational structure supportive of M&E? What are the barriers and how could this
be eliminated?
Environmental Indicators
 How does the project track environmental indicators? Does the project have capacity to
measure environmental indicators? What are the challenges in monitoring environmental
indicators? Recommendations to address the challenges.
Exit/ Sustainability Strategy
xiii | P a g e

Has there been an exit/ sustainability strategy developed. Are there clearly defined indicators to
measure the progress in implementing the exit/ sustainability strategies?
b) Example of Knowledge Management Questions
Internal Learning





Do internal reporting systems, including staff meetings and discussion, allow examination of
what did not go as planned, where adaptation is needed or where new idea could be applied?
Do internal reporting systems have feedback systems reaching to the beneficiary level?
Does the project have any systems in place where lessons can be shared across consortium
partners, or across technical areas?
How often do the project managers hear a new idea from field staff? What system is in place to
encourage staff to bring new ideas? How effective is the system?
How does the project identify, capture and document lessons learned, how do they get shared?
How often? What evidences does the project have in successful use of lessons learned in
improving program quality or avoiding mistakes?
Cross-organizational / Cross-project Learning



How best practices and lessons learned that are generated by the project disseminated out to
other organizations and how best practices and lessons learned from other organizations
channeled in to the project.
Does the management structure/culture support staff spending time creating and sharing
knowledge (i.e., documentation, report outs in meetings, brown bags, mentoring). Do job
descriptions identify clear knowledge sharing activities that should take place?
What strategies does the project use to apply best practices and lessons learned from other
agencies, other projects, and from other countries? Is there any evidence suggesting that the
project learned from others or used or adapted techniques or tools from other agencies, other
projects, or other countries?
Incentives

Does the management structure/culture support staff spending time creating and sharing
knowledge (i.e., documentation, report outs in meetings, brownbag, mentoring). Do job
descriptions identify clear knowledge sharing activities that should take place? Does the culture
support sharing of new or challenging ideas? How do the project managers provide feedback to
staff on their information or ideas?
Processes and Mechanisms for Sharing

Are there processes in place to facilitate the sharing of lessons: face-to-face (i.e., through
meeting topics, brownbag, mentoring programs, task teams), remotely with field locations
(through phone or skype or remote meeting software).
xiv | P a g e


Is there an easily accessible and widely publicized repository of lessons learned, case studies and
other information? If so, are there quality guidelines and an individual with a role in managing
and publicizing the information added?
Are there technical leaders and other ‘experts’ identified and made accessible for consultation
when needed?
c) Example of Commodity Management Questions
Commodity Pipeline
 How does the food pipeline work? (Please elaborate processes involved from ordering to
receiving at the port, inland transport, warehousing and distribution)
 Who all are responsible for call forward/sales order, port off-loading, transport to the transit
shed or port warehouse, inland transport to primary or secondary warehouse, warehouse
management, commodity accounting, beneficiary selection, distribution to end-users and enduse monitoring?
 What processes worked well in the entire food pipeline and which ones need adjustments?
 Are you using WBSCM to submit sales order/call forwards?
 Did you have any issue with sales order/call forward and how did you manage those?
 What problems did you face while receiving commodities at the port and how were those
managed?
 What problems did you face with inland transport of commodities to primary or secondary
warehouses or distribution sites and how were those managed?
 What problems did you face with warehouse management and how were those addressed?
 Were there any issues with beneficiary selection process and how were those managed?
 What problems did you face with final distribution of commodities to the beneficiaries and how
were those managed?
 What problems did you face with end-use monitoring and how were those managed?
 Did you face any problem with accounting of commodities and how were those managed?
 What are the types of losses that have occurred and how did those occur? What was the effect
of the losses on the program as well as subsequent ordering? What changes were made in
commodity management systems in response to the losses?
 What lessons have you learned with your commodity management system so far in this project?
Monetization
 What has been the history of monetization in the project?
 How is monetization currently being done?
 What problems have occurred with monetization and how were those managed?
 What has been the cost recovery rate on monetization over the life of the project? What is the
cost recovery rate on the most recent monetization?
 How is monetization being used to support market development?
Reporting
xv | P a g e



How are commodity reports prepared? Are you using QWICR to submit your reports?
What problems have occurred in report preparation and how have those been managed?
Has there been any comment from the donor on commodity reports and how did you respond
to those comments?
d) Example of General Management Questions






Who is responsible for the vision for the program? How well has the vision for the program
been articulated? How effectively has this vision been imparted to program staff within the
different agencies?
How are strategic and operational plans developed for the project? What has worked well in
this process? What has not worked well?
How have the implementation related problems identified, analyzed and solved? What has
worked well in terms of problem solving? What has not worked well?
How well does information get communicated throughout the project, including within the
project, with implementing partners at different levels, with external contacts and donors?
What has worked well? What has not worked well?
How well has the project communicated with stakeholders outside of the project? What has
worked well? What has not worked well?
What are the most importance lessons learned from the development food aid project relative
to management of the program?
e) Example of Financial Management Questions
Budget Questions
 What is the current LOA approved budget? How has this been revised since project start-up?
 What are expenditures through the last reporting period?
 What percentage of the approved budget has been spent by the end of fiscal year?
 What percentage of the approved budget is expected to be spent by the end of the project?
Cash Flow


How is cash for project implementation being provided to partners?
How effective has cash flow been managed in the project? Have there been any significant
delays in cash flow either from the donor to the prime awardee, from the prime awardee to the
major partners or from the prime awardee to its implementing partners? What was the cause
of the delays? What changes were made in managing cash flow?
Reporting
 How do financial reports for the project get prepared?
 What problems have occurred with financial reporting and how have these been resolved?
Cost Share
xvi | P a g e




How is the cost-share commitment being met?
What percentage of the cost-share commitment has been achieved by the end of fiscal year?
What percentage of the cost-share commitment will be achieved by the end of the project?
What other forms of cost-share have arisen since the program was initiated? How are these
being reported?
Audits



What audits have been completed on project funding since the prject was initiated?
What have been the audit findings?
How have these been addressed?
f)
Example of Human Resource Management Questions




What positions in the project are not still filled up? Reasons?
Is it difficult to get the necessary staff locally?
How many unfilled positions are still vacant? Number of positions vacant by sector.
Is it the issue of compensation package or expectations of qualifications from candidate, been a
problem?
Can any of the above be compromised?
What efforts have been made to fill up these positions?
Is staff retention be a problem? Are there frequent staff turnovers? Are there any glaring
reasons why they do so? How can saff turn over be minimized? Any concrete ideas?
What is the proportion of women to men among the staff? Is it difficult to get women staff and
if so why?
Is there any staff development plan? Does the project provide any coaching to new staff?
How effective the inputs of the staff in performance appraisals, and his expected needs are
met?
Are there opportunities for job rotation?
Do staff have a clear JD and roles and responsibilities are clearly laid out.
In general, how do the staff perceive the work environment?









xvii | P a g e