Evaluation Terms of Reference
Korca Area Development Program
Program Number 03446
FY10-FY13, Phase 1 of Implementation
World Vision Albania
Published February, 2013 © World Vision Albania
Tirana ADP FY11 Evaluation TOR
1
i.
Table of Contents
I.
TABLE OF CONTENTS ......................................................................................................................... 2
II.
ACKNOWLEDGEMENTS ..................................................................................................................... 3
III.
AFFIRMATION .................................................................................................................................... 3
IV.
GLOSSARY ........................................................................................................................................... 3
V.
INTRODUCTION .................................................................................................................................... 3
1.
DESCRIPTION OF PROGRAM BEING EVALUATED ................................................................... 5
2.
EVALUATION TARGET AUDIENCES .............................................................................................. 7
3.
EVALUATION TYPE .............................................................................................................................. 7
4.
EVALUATION PURPOSE AND OBJECTIVES................................................................................. 7
5.
EVALUATION METHODOLOGY ....................................................................................................... 9
6.
LIMITATIONS ........................................................................................................................................ 10
7.
AUTHORITY AND RESPONSIBILITY ............................................................................................. 10
8.
LOGISTICS .............................................................................................................................................. 11
9
PRODUCTS ............................................................................................................................................. 12
10 EMBEDDED DOCUMENTS .................................................................................................................... 12
11 LESSONS LEARNED ................................................................................................................................. 13
Tirana ADP FY11 Evaluation TOR
2
ii.
Acknowledgements
This Evaluation Terms of Reference (ToR) document was created jointly by the Area Development
Program (ADP) Manager in cooperation with the Ministry Quality (MQ) Team in National Office
(NO) with support from the Program Officer in Operations in the National Office.
iii.
Affirmation
Except as acknowledged by the references in this paper to other authors and publications, the
Evaluation TOR described herein consists of our own work, undertaken to secure funding,
implement the activities, describe and advance learning, as part of the requirements of World
Vision’s Design, Monitoring and Evaluation Learning System.
Primary quantitative and qualitative data collected throughout the evaluation process remains the
property of the communities and families described in this document. Information and data must be
used only with their consent.
Oltjan Drabo – Korca ADP Managewr
January 2013
iv.
ADP
DME
ITT
LEAP
RC
WV
CPU
CBO
IPM
DPA
CP
M&E
MEER
NO
SO
TDI
TOR
CWBO
v.
Glossary
Area Development Programme
Design, Monitoring and Evaluation
Indicator Tracking Table
Learning through Evaluation with Accountability and Planning
Registered Children
World Vision
Child Protection Unit
Community Based Organization
Integrated Programming Model
Development Program Approach
Child Protection
Monitoring and Evaluation
Middle East – Eastern Europe Region
National Office
Support Office
Transformational Development Indicator
Terms of Reference
Child Well Being Outcome
Introduction
Korca ADP is one of the youngest ADPs in World Vision Albania. FY10 was the first year for the
first phase of implementation whose goal is “To build the capacities of the partnership and partners
who work together to generate a community movement that intends to contribute to improvement
in education attainment of children while tackling issues that help build a child friendly community
that is inclusive of all despite gender, ethnicity and strata of the society one belongs to.”
Tirana ADP FY11 Evaluation TOR
3
Korca program was among three programs in MEER region that piloted IPM, thus assessment and
design processes used the former IPM (DPA model). The suggested IPM process of design combined
elements of needs based approach; rights based approach and appreciative approach. The
Framework of the Korca Programme aims at effectively engaging communities and specifically
actively engaged organizations and institutions which are already working on child well-being in their
own development affairs. These are aimed to be achieved mainly through the Education with
components of Early Childhood Stimulation. The last two years ADP has implemented Economic
Development and Child Protection/Youth Advocacy Project the last two as part of non Sponsorship
funded grants.
The Program has a strong element of community participation. Communities are targeted for
sensitization messages in regard to education and health issues.
The Program also aims at enabling partners to involve the community on education issues through
participation in education improvement efforts and on issues pertaining to Child Rights. The
Communities also participate in the upgrading the education structures especially primary school and
the kindergartens. This is central to the realization of interdependent and empowered communities.
The enhancement of education even to the extent of affecting Policy especially in the aspects of
education legislation and its implementation at the local level is a boost towards the domain of
transformed systems and structures. Additionally the engagement of the communities in aspects of
participatory budgeting, which LG suggested in their initiatives is another element of value in
ensuring accountability and transparency at local level budget allocation and expenditure for
education and specifically child development.
This document is a reference for the evaluation of the first phase of the implementation for Korca
ADP. It aims to identify progress towards impact. Below is a summary of the Korca ADP Evaluation:
Program
Program Phase
Evaluation Type
Evaluation Purpose
Primary Methodologies
Evaluation Start/End Dates
Anticipated Evaluation
Report Release Date
Tirana ADP FY11 Evaluation TOR
Korca Area Development Program
1 October 2010- 30 September 2013 (1st implementation phase)
Programme Evaluation (1st implementation phase)
To generate learning from the ADP first cycle design and operation to
inform the redesign process and product with intentional focus on lessons
learned and best practices regarding IPM technical approach.
Detailed evaluation methodology will be based on participatory
approaches and will be developed by the Lead Evaluator
February – May 2013
31 May 2013
4
1. Description of Program Being Evaluated
Operational period for
program
Program total budget
FY10 – FY13
Sponsorship funding HK $ 2,088,839
Non Sponsorship HK $ 159,673
Program target population
The ADP area has 50,100 inhabitants:
Pojan Commune: 17,246 inhabitants total (2,531 children aged 0-18 years &
14,715 adults)
Vreshtas Commune: 10,425 inhabitants total (2,104 children aged 0-18
years & 8,321 adults)
Libonik Commune: 14,337 inhabitants total (2,141 children aged 0-18 years
& 12,196 adults)
Maliq Municipality: 8,092 inhabitants total (1,656 children aged 0-18 years &
6,436 adults)
Program/Projects
Korca Area Development Program (# 03446)
• Education Project (A192638)
• Sponsorship Project (A188116)
Program location
Vreshtas, Libonik, Pojan Communes and Maliq Municipality
Number of Staff
14 full-time staff members including ED coordinator (NSF) (One part time)
Funding Source
WVHK
Project Manager and Report Oltjan Drabo – ADP Manager ArIola Kallciu – MQ Manager
contact information
Oltjan_drabo@wvi.org
ariola_kallciu@wvi.org
Tel: +355 68 90 10044
Tel: +355 68 20 58566
Geographical setting: Korca ADP consists of three communes (Pojan, Vreshtas, and Libonik) and the
municipality of Maliq. Maliq includes one town and three villages, Pojan has twelve villages, Vreshtas
four villages and Libonik has thirteen villages. To the north, the ADP shares borders with Pogradec
Region and Pirg Commune, to the south the ADP shares borders with Bulgarec and Voskop
Communes, to the west with Gore and Voskopoje Communes, and to the east with Liqenas
Commune and Devoll Region. The terrain in some of the villages is mountainous and can have harsh
weather with a lot of snow-fall in winter with the lowest temperature minus 20 Celsius degrees and
warm, dry weather in summer with the highest temperature 35 Celsius degrees.
Cultural Dynamics and Economic Development The communities of the communes and the municipality
are well known for their hard work, commitment to education and hospitality. The general tradition
and culture stresses a lot the value of the family institution. The patriarchal culture of the community
creates male domination of the society and women’s and children’s voices are not valued in decision
making. Albanians make-up the majority of the population; however, there are also some Roma,
Egyptian, Macedonian and Arumun minority ethnic groups.
The main sources of income are from remittances from emigration, agriculture and livestock. The
low economic level is a factor that affects the social life of the community. The area is well known
for the cultivation of fruit trees such as apples, plums and cherries, which contribute to the income
of numerous families.
Infrastructure: The infrastructure conditions have been improved the last year, yet the lack of
infrastructure is visible especially in the villages. The weak infrastructure includes not only the
local/national roads, but also the sewage, water and other systems.
The program outcomes (project goals & outcomes) with indicators are presented below:
Tirana ADP FY11 Evaluation TOR
5
Hierarchy
Objective
Indicator1
PROGRAM
Children are happy in an inclusive community
GOAL
CHILD SPONSORSHIP MANAGEMENT PROJECT
% of children (RC/Non RC) reporting to feel an increased
care by the community members towards them
% of community members reporting to have experienced
PROJECT
Children cared for and participate
an increased care for children in their village or
GOAL
neighborhoods.
% of children (RC/Non RC) articulating to have participated
in activities designed by children
ADP and NO technical staff practices
# of evidences indicating two ways influences between
OUTCOME enhance integration of child
sponsorship management project in
sector project (education) and CSMP
1
overall programming
OUTCOME
2
Partners, including Student
Governments (SGs) and ADP staff set
up and run the Child Sponsorship
Management Project
OUTCOME
3
Local partnership monitors, analyzes
data and takes appropriate actions on
meeting standards (participation,
benefits, presence of children)
OUTCOME
4
Sponsors continue to remain
committed in supporting communities
and children in KADP
Partners assessment indicates increased ability of partners
to run sponsorship activities as planned
Proportion of partnership decisions followed up and
reported on
# and value (in $) of community GNs increased in the life
of the project
Level of Sponsors Satisfaction remains the same or
increases
Sponsor initiated drops decrease over time
EDUCATION PROJECT
% of children that feel are exercising the right to education
PROJECT
GOAL
OUTCOME
1
OUTCOME
2
OUTCOME
3
Children enjoy the right to education
in an inclusive community
Children enjoy quality education
Children continuously improve their
life skills
Children at early age have equal
opportunities to be increasingly
stimulated in their growth
Tirana ADP FY11 Evaluation TOR
% of children and % of adults that identify at least 3 changes
that have taken place making the community an inclusive
one
% of children improving academic results
% of schools in community that have an increased Grade
Point Average at the end of the year
% of passing rate per school
% of children that articulate to feel appreciated, welcomed
and participate in education process
% of children (RC/non RC) who articulate they feel
supported in their education from parents.
% of children (RC/Non RC) continuously improving their
Development Assets in the Development Assets Profile
measurement
% of children who report that have been supported by
their parents/teachers in practicing life skills
The test results of 6 years old children in the next three
years indicate a positive trend (increased percentage) with
regards to children's mental and physical development.
% of parents indicating to use the knowledge gained in
6
OUTCOME
4
Partners in education coordinate
action for quality education
parenting practices.
System of coordination improved over time
# of joint decisions and actions undertaken by two or more
partners in the life of the project
2. Evaluation Target Audiences
The primary audience for the evaluation include:
- WV Hong Kong as the Support Office
- WV A&K
- ADP team
- The network for CWB in Korca ADP
- Major national and local stakeholders
Apart from the above, Korca ADP evaluation exercise is intended to include and influence the
following groups:
♦
♦
♦
♦
♦
♦
♦
♦
Registered and non-registered children and their families in ADP communities
Children and youth groups
Participants from the communities
School structures representatives
Health Centre Staff
Local government, such as Communes/municipality and Health and Education Directories
Local and International NGOs
Faith Based Organizations (FBO-s) and churches
3. Evaluation Type
This evaluation is of a summative type, and it aims to tack progress towards impact.
4. Evaluation Purpose and Objectives
The purpose of this evaluation is to (a) provide the ADP staff, partners and communities, with
recommendations to consider during the second the phase of implementation exploring
lessons learned, best practices regarding the IPM technical approach; (b) determine
effectiveness in how has the Program effectively contributed to the achievement of its objectives
and goals, in addressing the needs adjusted per outcome/objectives of the project; (c) explore
sustainability, how has the IPM approach contributed to the sustainability of initiatives in the
region.
As stated above, Korca ADP’s goal is “Children are happy in an inclusive community”. Korca ADP
consists of three sector interventions child sponsorship, education and health (through ECCD and
nutrition). These program objectives were determined with significant participation from local
partners (local authorities, organizations, institutions, etc.) and communities themselves.
The key objectives for review in this evaluation period are the appropriateness of the current
program design, the degree of impact the program has had, and the potential for sustainability
moving forward. In addition:
The key questions in each of the evaluation dimensions are the following:
Evaluation
objectives
Sub-Theme
Tirana ADP FY11 Evaluation TOR
Key Questions to Be Asked
7
Relevance and
appropriateness
Alignment with issues
prioritized
To evaluate the
relevance and
appropriateness of
programme design
and programme
contribution towards
the community
needs.
How relevant was the programme theory of change to the
prioritized community issues?
Have ADP/partnership interventions stayed focused on high
priority community issues?
To what extent have the Most Vulnerable been included in
the ADP/ partnership interventions?
Strengths and
Weaknesses
Compliance
How has the Development Approach influenced the focus
on the community-prioritized issues?
What are the strengths and weaknesses of programme
design with partners?
What are the strengths and weaknesses of the program/
projects M&E plan/s with partners?
How has the IPM approach aligned or been integrated with
the rest of WVA&K programming, processes and systems?
To what extent did the outputs and outcomes designed
contribute to achieving projects goals?
External Factors
Effectiveness
Logic of intervention
Assess the progress
made towards
achieving project
goals and outcomes
Participation & RC
inclusion
Compliance
Quality sponsorship
understanding and
integration
How were the cross-cutting themes incorporated in design?
How did assumptions defined in the design affect the
programme?
To what extent programme interventions addressed the
CWBOs based on the priorities of children, their families
and communities set during the design?
To what extent local community participated and
contributed to the program intervention?
To what extent RCs are among the participants of the
program?
To what extent the ADP implemented all planned projects
and met its targets?
To what extent the models of intervention as contained in
the PDD were actually applied?
Assess the degree to which sponsorship complements
contribution to CWBOs by the programme?
What is the perception of Child sponsorship by children and
community groups and what positive and negative effects
does it create?
What is the quality of communication to the sponsors?
Funding
External Factors
To what extent have sponsorship practices been integrated
with other development activities?
Was the type of funding sufficient for addressing high
priority needs?
How and which unforeseen external factors influenced the
programme?
What was done to manage the actual or potential impact of
external factors?
Sustainability
Participation
Tirana ADP FY11 Evaluation TOR
To what extent the Education and CSMP Partners and
8
To explore and learn
about the extent to
which the program
build towards
sustainability.
communities participated in the first phase of programme
implementation.
Capacity
Development
How the education and CSMP partners did contribute to
achieving project/programme goals?
Which capacities of partners were planned to be developed
from the design phase?
Which capacities of education partnership and communities
were built?
To what extent those capacities contributed to their
organisational sustainability and to their ability to sustain
achieved results?
What other capacities should be built?
To what level did the education partnership, communities
use the inputs and knowledge obtained during the program
design and implementation?
Recommendations
To what extent the current civil society, particularly the
education and CSMP partnership is ready for taking over the
program achievements?
What are the recommendations to be addressed during the
next phase?
What do recommendations say in regard to future
directions?
What are the specific recommendations to be addressed
during the next phase to better align the program with DPA
principles
5. Evaluation Methodology
Korca ADP is one of the first IPM pilots in WV that now is going in redesign; there is considerable
interest in exploring the lessons learned, best practices and recommendations regarding the IPM
technical approach.
Taking into consideration lessons learned during recent program evaluations in WVA&K IPM/DPA
approach and Korce design process, Korca ADP will adopt an integrated Evaluation and Redesign
process.
Program partners will be involved throughout the evaluation process in a meaningful way. All
partners involved in the implementation will be oriented regarding evaluation process. A
Committee group of partners will be elected from the Network of Partners through a transparent
and democratic process.
This Committee will be the key body who will be involved in the data analysis and results with the
same role as ADP staff. Further, they will be also responsible to communicate the evaluation findings
and results to the wider Network for CWB in Korca ADP.
The evaluation process will specifically explore the questions regarding program accomplishments
and impact in Korca ADP region during the three/four years of program implementation. Then the
evaluation will be linked into redesign processes in Steps 2 & 3 that explore the question of what has
been done by others in the region related to child well being and the social mapping to confirm
and/or enrich the partnership.
Tirana ADP FY11 Evaluation TOR
9
The data collection will follow a traditional evaluation process. Primary data will be collected
through Surveys, Focus group discussions and Key Informant Interviews. In addition, Secondary data
will be collected through review of documents and available secondary data sources. Relevant
documents may include: Program and project design documents (which include the logframe, M&E
Plans, etc.); Baseline Reports and Semi/annual program management reports.
These methods will be used in accordance with the specific evaluation objectives, according to the
key questions to be asked and depending on the target group. For each indicator an appropriate
method will be applied. Targeted population will be: children, parents, teachers, nurses, local NGOs, local government authorities on education and health.
One key difference will be the timing of the evaluation process. The inclusion of partnership
representatives in the evaluation process itself means that the pace of the evaluation might be
somewhat slower.
6. Limitations
1. This process is experimental, so there is, as yet, no learning on how integrating local
partners into the evaluation process will succeed in Albania.
2. The process might require a longer period of time as local stakeholders may not be able to
allocate full time for a short period.
3. ADP staff will also be engaged in redesign trainings, orientations, and processes as well as
ongoing sponsorship implications – so there may be issues with stretching the ADP staff
considerably.
7. Authority and Responsibility
The evaluation exercise will be led by xxxxxx, the Lead Evaluator who will work with the direct
support of MQ Manager and the ADP Manager. The evaluation team will be composed of appointed
NO MQ, TD, IP, Sponsorship staff and other ADP’s appointed staff.
The ADP Manager and the Partners’ Committee will be consulted throughout all phases of the
evaluation process in order to provide the evaluation team with their knowledge of program
interventions. The Evaluation Team Leader will have overall responsibility for coordinating the stages
of evaluation implementation with ongoing consultation with the ADP Manager and MQ manager.
All of the roles in the evaluation process (secondary data review, survey questionnaires creation,
field work, data collection, interviews, focus groups, data processing, writing of evaluation report,
etc.) will be coordinated and appointed by the Lead Evaluator in accordance with the Evaluation Plan
and Design.
Eval. Phase
Planning
Role
ADP Manager
ADP Operations
Manager
Ministry Quality
Officer
Tirana ADP FY11 Evaluation TOR
Primary task
• Coordinate collection of information for terms of reference.
• Advise the Evaluation Team Leader on evaluation priorities from a
program perspective.
• Identify appropriate community stakeholders for integration into
the team
• Introduce team leader to evaluation team and to partners.
• Provide feedback on evaluation ToR
• Advice and assist ADP Manager in preparing the evaluation ToR.
10
Ministry Quality
Manager
TD Manager
Evaluation Team
Leader
Data
Collection and
Analysis
Reporting and
Follow-Up
Evaluation Team
Leader
ADP staff
and
Local partners
Ministry Quality
Officer
Evaluation team
members
(community
stakeholders)
Evaluation Team
Leader
Evaluation team
members
ADP staff
and
Local partners
SO WV Hong Kong
• Provide strategic guidance on process and input on the quality of
products generated
• Give guidance and advice related to the process of evaluation with
partners
• Develop evaluation design
• Coordinate the evaluation process and the collection and analysis
of data.
• Participate actively as team members in the evaluation process
• Lead focus groups and interviews; record responses and qualifying
observations; prepare summaries of responses, etc.
• Assist with analysis and interpretation of data
• Logistical arrangements (organise FGD, interview, etc.)
• Give LEAP guidance to all teams in regard to evaluation
requirements (also member of Evaluation Team)
• Participate actively as team members in the evaluation process
• Lead focus groups and interviews; record responses and qualifying
observations; prepare summaries of responses, etc.
• Assist with analysis and interpretation of data
• Supervise the drafting of the report and circulate it for feedback.
• Incorporate feedback into the report.
• Provide critical feedback on draft.
• Provide critical feedback on draft.
• Provide critical feedback on draft.
8. Logistics
Evaluation phase
Planning (February 2013)
Role
Korca ADP
•
•
NO PO
•
NO MQ
•
Lead Evaluator
•
•
Preparation (March 2013)
Lead Evaluator
•
Field work (April 2013)
Lead Evaluator
•
NO MQ
•
•
Korca ADP
Tirana ADP FY11 Evaluation TOR
•
Primary task
Draft the Evaluation ToR
Circulate the ToR to the
partners for feedback and
amendment
Communicate the ToR to the SO
for feedback
Communicate the ToR the Lead
Evaluator
Finalize the ToR
Request clarifications with
regards to the ToR, if necessary
Develop, discuss and finalize the
evaluation design,
methodology/tools
Lead the field work of collection
and primary analysis of data
Coordinate and participate in the
field work
Train/orient interviewers, note
takers, as necessary
Coordinate meetings with key
stakeholders and FG participants
as per the timing agreed in the
11
Report writing (April-May 2013)
Lead Evaluator
•
•
Evaluation team
NO MQ
NO OPS
Korca ADP
•
•
•
•
•
•
WV Hong Kong
•
Evaluation design
Draft the evaluation report and
circulate it for feedback
Incorporate feedback into the
report
Finalize the report
Provide feedback on report
Provide feedback on report
Provide feedback on report
Provide feedback on the report
Circulate the report to the
partners for feedback
Provide feedback on the 2nd draft
of the report
9 Products
As the final product of the evaluation process, the Evaluation Report will be written by the
Evaluation Team Leader with the final approval of WV HONG KONG.
As it is an end term evaluation, the report findings and recommendations will be shared with
partners and reflected in the re-design process up to come in the second implementation phase.
The total estimated budget to be covered by Korca ADP for the evaluation is: 4040$
Note: For more details please refer to the budget embedded here
Korca Evaluation
budget.xls
10 Embedded Documents
Please note that the documents to be reviewed are embedded in this document
•
Korca ADP FY09-11 Program Design Document :
KorcaADP_Programm
eDesignDocument_FY09.doc
•
Korca ADP Baseline Report;
KorcaADP_Baseline_
FY10.doc
•
FY 12, Korca ADP Annual Program Management report
Tirana ADP FY11 Evaluation TOR
12
Korca ADP FY12
Management Report.doc
11 Lessons Learned
Reflection meetings will be organized throughout the evaluation process and lessons learned will be
included in the final evaluation report to ensure transfer of knowledge in similar exercises in WV
Albania and beyond.
Tirana ADP FY11 Evaluation TOR
13