UNDP-GEF Terminal Evaluations Review Form
I.
Background Data
Project Information
Project Title (per project appraisal document
submitted to GEF)
UNDP Project ID:
GEF Project ID:
Region:
Countries:
GEF Operational Program / Strategic Program:
Executing Agency:
Project Partners :
Project and Evaluation Milestones
(Milestones)
CEO endorsement:
Agency approval date:
Implementation start:
Midterm Evaluation:
Project Completion:
Terminal Evaluation Completion:
TE & QA Review particulars
Terminal Evaluation author(s):
TE QA Reviewer:
TE QA Review date:
Draft TE QA Submission Date:
Final TE QA Submission Date:
UNDP EO task manager approval date & sign:
II.
a.
Transformation of the Rural Photovoltaics (PV) Market
in Tanzania
1894
1196
Africa
Tanzania
OP 6
CC 1
Ministry of Energy and Minerals, Renewable Energy
Section
Tanzania Solar Energy Society (TASEA),
VETA, Financial Institutions (Banks),
SIDA PV Project,
Tanzania Bureau of Standards (TBS).
(Expected Dates)
Not Available
Not Available
Not Available
Not Available
December 2008
Not Available
(Actual dates)
February 2004
Not Available
April 2004
November 2006
March 2009
July 2009
Mohamed Ali Hamid and Finias Magessa
Elsa Da Costa
November 24, 2010
November 26, 2010
20, December 2010
Alan Fox, 22, December 2010
Project Design & Objectives
List the overall environmental objectives of the project:
The global objective of the proposed project is twofold: i) to reduce Tanzania’s energy related CO2
emission by substituting PV for fossil fuel (kerosene) utilised to provide basic electricity services to rural
homes and community users and ii) To improve people’s livelihoods by improving their access and
1
affordability of modern energy services.
b.
List the development objectives of the project:
To remove barriers with the aim of promoting the utilisation of PV to provide basic electricity services to
improve people’s livelihoods and reduce the dependency on imported fossil fuel.
c.
Identify the key project components
Component 1: Policy support & institutional strengthening;
Component 2: Awareness raising;
Component 3: Private sector strengthening;
Component 4: Financial engineering; and
Component 5: Learning and replication.
d.
Briefly assess the quality of project design, noting in particular the logical connections of
objectives to outcomes and outputs taking into account risks and assumptions, and the proposed
budget.
Overall the project components and outputs are generally logically connected, however the project design
is a little weak. The project components are described like objectives and no outcomes are presented. The
language of the outputs is not based in results-based management. The outputs are also described as
objectives, or very briefly described (i.e. “Business Development Services strengthened”). The
descriptions of the activities lack some details. They are quite broadly described and do not refer to any
specifics.
The ProDoc does present risks but not any assumptions. The risks are well described and do take into
account the many challenges of opening a new PV market introducing this new technology in quite a rural
area of Tanzania.
The budget is presented by component by GEF financing and by co-financing. Overall, the budget seems
sound for this project.
e.
Have indicators been established for the achievement of project outputs and outcomes? If so do
they build from GEF guidance, are they SMART?1
The project has set out indicators for each of its ‘immediate objectives’ and ouputs in a `project planning
matrix`. The quantitative indicators are good and well connected to the objectives and outputs, however
indicators of a more qualitative nature are quite weak, especially in regards to any output or objective
that aims to ‘strengthen’. These are often not as well linked to the output or objective and mostly
resemble targets rather than indicators.
f.
Were there any delays in the project preparation phase, prior to project Inception? If so, why? Did
the delays result in changes to objectives and planned outcomes?
There have not been any delays in the project preparation phase.
III.
Project Implementation
Project management
a.
Indicate whether there were changes in the environmental and development objectives of the
project during implementation, why these changes were made and what was the approval
process 2
1
The GEF Secretariat has issued guidance on the use of “SMART” indicators (Specific, Measureable, Achievable and
Attributable, Relevant and Realistic, and Time-bound, Timely, Trackable and Targeted). GEF climate change
indicators were developed in 2000, IW in 2002 & 2003, and biodiversity in 2003. For a description, see
www.gefweb.org/MonitoringandEvaluation/MEPoliciesProcedures/MEPIndicators/mepindicators.html
2
There were no changes in the environmental and development objectives.
b.
Were there delays in project implementation and completion? If so, indicate reasons and
comment on whether the delays affected project outcomes and sustainability.
There has been a 3 month project delay at completion. No reasons were provided for this delay, however
there is mention that he project manager had to resign due to illness, and some time was needed to find a
replacement.
Co-financing3
Source of co-financing
Total4
Type (cash or in-kind)
host gov’t contribution
In kind
Expected
147 600 USD
Actual
147 600 USD
GEF Agency (ies)
Other agencies (bilateral / multilateral)
Small Grant Trac:
UNESCO Belgium
240 000 USD
540 000 USD
240 000 USD
540 000 USD
Dutch Govt
630 000 USD
630 000 USD
SIDA PV
3 176 471 USD
3 176 471 USD
private sector
NGOs
Other
Total co-financing
4 734 071 USD
4 734 071 USD
4 734 071 USD
a.
Is there sufficient clarity in the reported co-financing to substantiate in-kind and cash Yes
co-financing from all listed sources?
Comment
The PIRs provided clear reporting of co-financing for the listed sources.
b.
If there is a difference in the level of expected and actual co financing, what were the
reasons?
Comment (to what extent, and note whether justifications were given)
Yes
There has been an increase in co-financing. During the project design it was anticipated that the SGP
would expend US$ 0.24 m to support community renewable energy initiatives around the Lake Zone.
However due to awareness created by the project the demand for PV support increased dramatically
2
The GEF guidance indicates the following possible reasons for changes: a) original objectives were not sufficiently
articulated; b) exogenous conditions changed, due to which a change in objectives was needed; c) project was
restructured because original objectives were overambitious; d) project was restructured because of a lack of
progress; e) other (specify).
3
GEF focuses considerable attention on the extent of co-financing across the portfolio, and uses co-financing as a
proxy for country ownership as well as an important indicator for sustainability. Accordingly, this table should be
filled in based on the numbers set out in the ProDoc for expected co-financing and then in the TE and/or Project
Close out note, on actual or realized co-financing. Especially in GEF-2 & 3 projects there may be instances where
no actual co-financing numbers have been tabulated, and/or no cash & in-kind breakouts have been provided.
4
Includes both co-financing during project design stage (PIF/PPG) and Project Implementation (ProDoc)
3
resulting into more proposals submitted by communities to UNDP/SGP. Most of the requests received
support due to their credibility and justification. Therefore there is an increase of US$ 0.42 to the
anticipated original figure.
c.
Were components supported by co-financing well integrated into the overall
project?
Comment
Yes
For the majority of the co-financing the total amounts were split between all of the project components,
thus integrating the co-financing into the overall project.
d.
Did the extent of materialization of co-financing affect project outcomes and/or
sustainability
Comment
No
Additional comments on co-financing:
Monitoring and Evaluation
a.
Assess the quality of the M&E design at project start up
MS
Comment: The monitoring and evaluation design at start-up mainly describes the different monitoring
or evaluation steps for a UNDP-GEF project. It does provide details in regards to who will undertake the
M&E and what are the outputs.There is no performance monitoring framework presented at project
design.
b.
Assess the quality of M&E Plan Implementation
Comment:
S
The performance monitoring framework presented in the PIRs is very well done. The quantitative
indicators are still relevant, and the qualitative indicators have been reworked so they are SMART.
Targets have been set for all key milestones (i.e. beginning, mid-project, and project end). One thing that
was missing at the first PIR was a baseline. However this was added in the last PIR for the majority of the
indicators. In addition, the objectives have been rewritten as proper outcomes using RBM language.
Was sufficient funding provided for M&E in the project budget?
The budget allocated for M&E was 247 900 USD. This represents 2 % of the total project budget, which
could be considered sufficient.
Was a midterm evaluation carried out and were MTE recommendations taken into account in
subsequent project implementation?
A midterm evaluation was undertaken in 2006, however most of the recommendations were quite
broad. In addition given the weakness of the TE, the reviewer is unable to provide a sound and evidence
based assessment on whether the MTE recommendations were implemented.
Were PIR self-evaluation ratings consistent with the MTE and TE findings? If not, were these
discrepancies identified by the project steering committee and addressed?
Yes
Country Ownership
a.
Are there indications that government agencies in addition to the GEF Focal Point have
participated in the project? if so, how?
4
The project has maintained good stakeholder participation throughout its lifetime through, Project
Steering Committee (ST) and Project Technical Committee (TC). The Steering Committee represented
Ministry of Energy and Minerals (Renewable Energy Section), Private sector companies, Tanzania Solar
Energy Society (TASEA), VETA, Financial Institutions (Banks), SIDA PV Project, Tanzania Bureau of
Standards (TBS), MoFA, VPO, UNDP, and RAS Office. Further, a number of institutions were
recommended to take part in the sustainability of the project results including REA, VETA and TASEA.
b.
Has the level of country ownership affected project outcomes and sustainability? If so, in what
ways?
Not when it comes to outcomes. However, if the Rural Energy Agency does not get more involved in the
PV market, it may affect the sustainability of the project results.
IA & EA Execution
a.
Assessment of the quality of Implementing Agency Execution:
 Focus on results
 Adequacy of supervision
 Quality of risk management
 Responsiveness to significant implementation problems (if any)
 Quality and timeliness of technical support to the project team
 Candor and realism in supervision reporting
 Suitability of chosen executing agency for project execution
Comment:
S
There is no information in the MTE or the TE on the quality of the implementing agency work, however in
reviewing the PIRs, the reviewer sees the progression and improvement in the monitoring and reporting
of this project. In addition, given the IA involvement in the Steering Committee and the success of the
project, the reviewer can deduct the quality of the IA was good.
b.
Assessment of the quality of Executing Agency Execution:
 Focus on results
 Adequacy of supervision inputs and processes
 Quality and timeliness of financial management support to the Project Team
 Quality of risk management
 Candor and realism in supervision reporting
Comment:
S
There is little available information in the TE and the MTE regarding the executing agency. However,
based on the success of this project, one can deduct that the executing agency has done a satisfactory
job in implementing this project. The MTE does state that the project team was rather small, but that the
use of key local consultants was a good strategy to deliver key parts of the project components.
Overall Quality of Project Outcomes
Comments and justifications:
S
HS
HS
There is no rating in
the TE
The project has done very well in achieving its outcomes.
Overall Quality of Project Implementation / Execution
5
Comments and justifications:
Given its high level of achievement, the project implementation is viewed as highly satisfactory, especially in
regards to its implementation approach, stakeholder involvement, budget and time management.
IV.
Assessment of Outcomes & Sustainability5
Criteria6
Relevance
 Were the project’s outcomes consistent with
the focal areas / operational program strategies
and country priorities?
Comments and justifications:
UNDP EO
HS
Terminal Evaluation
No rate was provided
in the TE
Indeed the project responded to GEF climate change focal area and operational program 6. In addition the
project was aligned with the national energy agenda to provide greater energy access to households, most
notably in rural areas, as well as providing options for alternative energy.
Effectiveness
HS
 Are the actual project outcomes commensurate
with the original or modified project objectives?
Comments and justifications:
No rate was provided
in the TE
Indeed the project outcomes and components have been kept well in sight throughout the project.
Efficiency
 Was the project cost effective?
 Was project implementation delayed, and if it
was, did that affect cost effectiveness?
Comments and justifications:
HS
No rate was provided
in the TE
There has been no evidence that the project has been inefficient. There is mention that the project has been
delayed in the end by 3 months but no reason was provided besides the resignation of the project manager due
to illness and the time to find a replacement.
Sustainability
a.
Financial resources:
What is the likelihood that financial resources
will be available to continue activities resulting
UNDP EO
(likelihood)
ML
TE
(likelihood)
There is no rating in
the TE
5
Each of the TEs should use the provided rating scale (set out at the end of this template / guide). This review
should include its own ratings assessments of project relevance, effectiveness, sustainability, etc, with reference
provided on the same ratings provided in the Terminal Evaluation. The reviewer will need to justify a different
rating based on new information or indicating errors made in the TE assessment. In cases where there is
insufficient information in the TE to make a determination, the reviewer should indicated U/A = unable to assess.
6
See Guidelines for GEF Agencies… 7 – 14 for a discussion on the criteria definitions. The bulleted points are
taken from this GEF guidance. The ratings will all be based on the 6 point scale of satisfactory achievement except
for sustainability – which switches to likelihood of achievement (also considering the likelihood of avoidance of
risks to sustainability)
6
in continued benefits?
b.
Socio-political:
Is there sufficient public stakeholder awareness
and support for the continuation of activities
providing benefit?
c.
Institutional framework and governance:
Are required systems for accountability and
transparency plus technical know-how in place?
d.
Environmental :
Are there environmental risks that can
undermine the future flow of project
environmental benefits?
Comments and justifications:
ML
There is no rating in
the TE
L
There is no rating in
the TE
HU
There is no rating in
the TE
The project has done a great job in setting up the proper institutional framework for the PV market in rural
Tanzania, however the sustainability weaknesses lie in the stakeholders that will continue the work and the
available financing and incentives for PV dealers and end-users. In addition the TE states the following four
areas to ensure sustainability:
 Replication activities to be completed.
 Standards on equipment and code of practice to be enforced.
 Awareness on system design, equipment quality and code of practice need to be enforced.
 End user, small PV shops and PV productive use financing to be sustained.
Monitoring & Evaluation system
 Is there a sound M&E plan in place to monitor
results and track progress toward achieving
objectives?
 Was the M&E plan sufficiently budgeted and
then carried out?
Comments and justifications:
S
There is no rating
in the TE
The initial M&E framework presented was quite weak however it greatly improved with each PIR, most notable
the last PIR. There were proper outcomes established and indicators with baselines and targets which have been
used to establish the high level of success of this project. There is no evidence that the budget was insufficient
for the M&E plan.
Overall Project Rating
S
There is no rating in
the TE
Comments and justifications:
The project has been highly successful, however the sustainability of its results remain fragile and could be
easily compromised by the remaining stakeholders. Overall, this project is a good example of transforming the
market for the rural PV market, however, key rural stakeholder should have been involved from the beginning
to ensure the sustainability of this transformation.
7
Impact7
V.
a.
The project has demonstrated verifiable improvement in ecological status
Yes
Comment
The project has promoted a significant growth in the PV market and use which has reduced an estimated
2.24 metric tonnes of CO2 emissions.
b.
The project has demonstrated verifiable reductions in stress on ecological systems
Yes
Comment
The reduction in CO2 emissions reduces stress on the climate system.
c.
The project has demonstrated through specified process indicators that progress is
being made towards achievement of stress reduction and/or ecological
improvement.
Comment
Yes
The project has reduced an estimated 2.24 metric tonnes of CO2 emissions through the installation of PV
systems and the reduction of taxation barriers.
d.
As a result of the project, there have been regulatory and policy changes at regional,
national and/or local levels.
Comment
Yes
There have been a significant reduction in duties and VAT for PV systems, which render them more
affordable for the rural population of Tanzania.
e.
Summarize the achieved intended or unintended impacts of the project:
 The project has reduced an estimated 2.24 metric tonnes of CO2 emissions.
 The project has made tangible impact in PV business development in Mwanza and the Lake‐Zone



VI.
area. Considerable PV dealers/shops have emerged due to the project activities which have also
contributed to new jobs creation in terms of PV business, installers and maintenance services.
PV systems are used to replace kerosene lighting which is contributing to less GHG emissions and
better health (less respiratory diseases) in the long term.
Demonstration system has enabled the availability of health service during night time maternity
health being the major beneficiary.
PV productive uses demonstrated a new window for income generation in rural areas.
Catalytic Role8
7
All TEs discuss sustainability but this is not synonymous with impact. It is important to look for references to
verifiable pollution reduction and ecological status improvement, and/or whether there are process indictors that
suggest such impacts should occur in the future as a result of project achievements. If the project is a foundation
setting effort, it is not expected to achieve stress reduction and/or status change impacts in the short to medium
term.
8
The GEF EO Guidelines on TERs notes that at this point there is no specific guidance for assessing catalytic roles
and replication in GEF projects. The expectation is that this will be required in the future, hence a section in the
review effort to consider four levels of catalytic results. No ratings are expected, however the reviewer should
consider the extent to which the project has achieved: a) production of a public good, b) demonstration, c)
replication, and d)scaling up.
8
a.
Production of a public good
Lowest level of catalytic result, including for instance development of new
technologies and approaches. No significant actions were taken to build on this
achievement, so the catalytic effect is left to ‘market forces’
Comment
Yes
The project had provided greater access to PV in rural Tanzania for household use.
b.
Demonstration
Steps have been taken to catalyze the public good, for instance through the
development of demonstration sites, successful information dissemination and
training
Comment
Yes
The majority of the project has been to raise awareness from policy-makers to private industry to end
users on PV. The project has undertaken trainings, workshops, literature production and demonstrations
in order to raise awareness of PV in rural Tanzania.
c.
Replication
Activities, demonstrations, and/or techniques are repeated within or outside the
project, nationally or internationally.
Comment
Yes
At the time of the TE, there were 8823 PV systems sold per annum and PV dealers extended their
presence to the replication area. There are 10 PV dealers and 6 PV dealers qualified for the supply-chain
financing modality.
d.
Scaling up
Approaches developed through the project are taken up on a regional / national
scale, becoming widely accepted, and perhaps legally required.
Comment
No
The project has targeted one rural region of Tanzania. This has not yet been taken up by another region
or at a national level.
I.
a.
Mainstreaming9
Based from the ProDoc and Terminal Evaluation, has the project focused attention on the nexus
of issues connecting environmental protection with other socio-economic priorities of UNDP such
as poverty reduction, crisis prevention and recovery, and democratic governance?
Yes, the project targets the end-users which are rural households, in order to provide them with a more
cost-efficient and healthier energy alternative to kerosene.
9
In addition to UNDP/GEF objectives aimed at global environmental benefit, other UNDP objectives should be
considered in the Review. These include the extent to which mainstreaming of key UNDP objectives on poverty
alleviation, crisis prevention and recovery, and democratic governance have been taken into account.
Increasingly, UNDP is expecting that these objectives are acknowledged in UNDP-GEF projects; however this has
not been a steadfast requirement in the past. It will be useful to note instances where such mainstreaming may
have occurred, in order to emphasize these linkages. For instance, if environmental protection activities have
contributed to improvements in local economic well being, better preparations to cope with natural disasters, and
broader improvements in governance, this should be mentioned. Both UNDP and GEF are focusing greater
attention to ensure that gender issues are taken into account in project formulation and implementation, (see
UNDP Gender Equality Strategy 2008-2011).
9
b.
Is the focus of the project identified as a priority in the UNDP country programme and as a
national/regional priority in a corresponding development plan?
This is not explicitly stated in the project documents.
c.
Have gender issues been taken into account in project design and implementation, (i.e. project
team composition, gender-related aspects of pollution impacts, stakeholder outreach to women’s
groups, etc)? If so, indicate how.
There is no evidence of gender mainstreaming in the project documents.
Is it possible to identify and define positive or negative effects of the project on local populations
(e.g. income generation/job creation, improved natural resource management arrangements
with local groups, improvement in policy frameworks for resource allocation and distribution,
regeneration of natural resources for long term sustainability)?
The project has provided greater accessibility of PV technologies for the local population. However, it is
too soon to establish the health benefits for the reduced use of kerosene for the rural population. See
impact section for more details on potential impacts.
VII.
a.
Project Lessons & Recommendations
Summarize the key lessons, good practices and approaches mentioned in the TE that may be
applicable to other projects:






b.
Summarize the main recommendations set out in the TE:



c.
The main lesson learned from this project is that, the entry point to the success made by this project
is attributed to the strong government commitment which could be explained by the breakthrough
made in the pro‐PV policy and institutional frameworks.
There is a potential for PV use for productive uses which needs to be explored further.
Though the project has demonstrated to the financing sector that PV is a financially viable business,
this activity remains a challenging one which needs more efforts to create sustained demand and
financing. Especially when it comes to end‐user financing.
The project has made an excellent use of the local knowledge by hiring local consultants as mush as
possible which had a notable contribution in the success achieved.
The strong and multi‐level targeting awareness (policy maker, dealers, consumer and technicians)
program implemented by the project has played a key role in the achievements made by the
project.
There is a need to link‐up the project with relevant governmental rural energy institutions,
especially the Rural Energy Agency (REA), which is the potential body to takeover most of the
project activities on phasing out.
Further support, from the Government of Tanzania and UNDP is needed to sustain the achievement
made on awareness of PV systems design and quality of equipment, financing modalities (dealer
and end‐user) and the enforcement of PV standards and code of practice.
The role of REA will be very instrumental in sustaining awareness created on PV in the region.
Installation of demonstration PV systems, awareness raising and strengthening PV financing
experience including more PV system for productive uses needs to be planned for and implemented
before final project phasing out.
Are there lessons and recommendations stemming from the project outcomes and
impacts that were not mentioned in the TE?
Comment (suggest lessons and recommendations);
No
The recommendations provided are clear.
10
VIII.
a.
Quality of Terminal Evaluation Report
Assessment Criteria10
Does the terminal evaluation report (TE) present an assessment of all relevant
outcomes and achievement of project objectives in the context of the focal area
program indicators, if applicable?
 Project outputs and outcomes are assessed against ProDoc/LFA plans,
including verifiable indicators.
 Variances between planned and actual results have been identified and
assessed, including adaptive management strategies used.
 TE has addressed project contributions towards GEF objectives (global
environmental benefit) as well as national development goals and strategies
Comment:
Rating
S
The TE does present the results of each objective and output in a text format, then in a table format for a
clear understanding of the project achievements.
b.
Is the TE consistent, the evidence presented complete and convincing, and the
ratings well substantiated?
 Assessment of project results is well organized, clearly written and
unambiguous.
Comment:
MU
This has been done in the table format of the presentation of achievement of results, however, the TE
does not provide an assessment by OECD DAC criteria.
c.
Does the TE present a sound assessment of the sustainability of project outcomes?
 Project design measures and/ or strategies for sustaining project results are
assessed.
 Socio-political risks to sustainability are considered, including stakeholder
ownership
 Institutional frameworks and governance risks to sustainability are
considered, including requisite systems for accountability and transparency
and required technical know-how
 Financial and technical resources required for sustaining project results are
considered
Comment:
MS
The TE only presents key issues that pertain to the sustainability of project results, and not an overall
assessment of project sustainability.
d.
Are the lessons and recommendations listed in the TE supported by the evidence
presented and relevant to the GEF portfolio and future projects?
 The lessons learned draw upon specific observations or data compiled
S
10
The first 6 of these criteria are taken directly from the Guidelines for GEF Agencies in Conducting Terminal
Evaluations (pg 16); the rest are additional questions to get a better fix on TE quality. The bullets under each initial
statement have been added as additional points to aid in considering TE quality.
11
during the evaluation.
 The recommendations provide specific advice for the project exit strategy or
post-project sustainability.
 The recommendations provide specific advice for future projects or
programming, or similar projects.
 The recommendations are sufficiently practical and in the realm of
feasibility for potential implementation.
 The lessons learned and recommendations are concise, clearly written and
understandable.
Comment:
The lessons learned and the recommendations presented in the TE are valid and based on clear evidence.
e.
Does the TE provides the actual project costs (totals, per activity, and per source)
and actual co-financing used?
 Project cost and funding data are presented, including actual co-financing
from each source.
 Variances between planned and actual expenditures are assessed and
explained.
 Observations from financial audits completed for the project are considered.
 Issues related to financing and co-financing commitments and performance
are discussed and explained
Comment:
HU
This was not presented in the TE.
f.
Does the TE include an assessment of the quality of the M&E plan at entry, the
operation of the M&E system used during implementation, and the extent M&E was
sufficiently budgeted for during preparation and properly funded during
implementation?
 The existence and quality of the M&E plan are assessed, including baseline
conditions, methodology and roles and responsibilities.
 The extent to which M&E were sufficiently budgeted and funded during
project preparation and implementation is assessed.
 The effectiveness of monitoring indicators from the project document for
measuring progress and performance is assessed.
 Compliance with the progress and financial reporting requirements/
schedule is assessed, including quality and timeliness of reports.
 The value and effectiveness of the monitoring and evaluation reports and
process was discussed with participants and assessed.
 The follow-up actions, or adaptive management, taken to respond to
monitoring and evaluation reports is assessed.
Comment:
HU
This was not presented in the TE.
g.
Has the TE identified the processes and factors that affected the attainment of
results?
U
12

Including (inter alia): project design; country ownership; stakeholder
involvement; financial planning; project implementation; Executing Agency
supervision; co-financing)
Comment:
The TE was rather weak in this domain. It touches on some of the issues pertaining to sustainability, but
it does not present a clear assessment of project implementation, or co-financing.
h.
Does the TE provide an assessment of the project assumptions and risks as set out in
the Project Document and LFA?
 The TE provides an assessment of the stated assumptions and risks, whether
they are logical and robust, and have helped to determine activities and
planned outputs.
 Externalities, (i.e. effects of climate change, global economic crisi, etc), were
been considered
Comment:
S
The TE does provide a review of the project risks and the status of these at the project end.
i.
Were key stakeholders or stakeholder groups, or a representative sample thereof,
consulted during the TE fact finding process?
 The TE includes a listing of persons interviewed
 A questionnaire was distributed and an analysis of responses is provided
 Members of civil society, NGOs and private sector were given an opportunity
to provide their views
 As relevant, local ministry officials were interviewed
 Pertinent national ministry officials in addition to the environmental focal
point were interviewed
Comment:
S
The TE includes a list of participants with a variety of members from government, the private sector, PV
dealers, health units, however, the TE would have been strengthen, had the evaluators included some
end-users.
j.
Does the TE Terms of Reference comply with UNDP & GEF guidance, including
concerning conflicts of interest11?
Comment (gaps, conflicts, inconsistencies):
No
The Terms of Reference generally comply however there is a discrepancy between was is asked to review
and the Sample outline. The TE follows the sample outline but not the expected outputs outlined in the
ToRs which weakens the TE.
k.
Does the TE demonstrate good evaluation practice?
Yes
Comment (TE strengths and weaknesses):
The TE does show good evaluation practice, however it is weak in the presentation of its results, the TE
layout, as well as in the grammar.
l.
Are there any evaluation findings involving significant administrative, financial
and/or human resources issues that require UNDP follow up, (e.g. corruption,
No
11
UNEG, GEF & UNDP guidelines stipulate that except in rare and unusual circumstances, Evaluators engaged by a
UN agency shall not have had any responsibility for the design, implementation or supervision of any of the
projects, programs or policies that they are evaluating.
13
mismanagement, etc)?
(Note issues and UNDP management response)
Overall Rating for the Terminal Evaluation12
Justification:
MU
The TE does a good job of presenting the project results, however there are serious
weaknesses in the presentation of results. Primarily, the TE does not present the results
within the context of the OECD DAC criteria, as such the reader has knowledge of the
targets and results achieved but not if they were relevant, efficient, or sustainable. In
addition, the TE has omitted any financial information regarding the project, or any
information on the project implementation set up and performance. Furthermore, the TE
does not address any M&E issues or achievements.
IX.
Management responses, Subsequent Actions and Sustainability
a.
A Management response13 to the evaluation was submitted
b.
List key proposed follow-up actions14






Yes
Date: August 2009
Intensive awareness campaigns were conducted in Mwanza as well as in the replication regions
through media and demonstration activities. Training of technicians and suppliers was also
provided aiming at improvement of quality and services delivered
In addition to this and in response to the recommendation raised by the evaluation team, the
project is providing special training to the suppliers on sizing and after sales services in Mwanza
and in the replication regions to bridge the identified gaps.
Since the project is closing its activities in December 2009, it was agreed during the PSC that the
Government/MEM to make follow up with Sida/MEM Solar project and make sure that the
testing and monitoring equipment are in place to facilitate the implementation of Solar PV
Standards.
The project was due to close up its activities in June 2009, but after receiving US$200,000 from
UNDP core resources (TRAC) a new work plan was prepared to include among others things, the
replication of the Solar PV productive use activities in the other regions, and also to implement
some of the recommendations raised from the Terminal Evaluation.
One of the activities of the Policy component was to work with REA to support mainstreaming
Solar PV technology in Rural Energy Master Plan (REMP). The project tried to liaise with REA,
but the main setback was the delay in the establishment and operationalization of the Agency.
However, plans are underway and REA will participate in the planned Project Tripartite meeting
in November 2009, where they will have an opportunity to learn the best practice and
experiences from the project.
The Standards have not yet been gazetted. The project has carried out a series of workshops to
disseminate the standards to the stakeholders. However, during the last PSC in March 2009, the
12
Use GEF numerical ratings 1-6, whereby 6= highly satisfactory, etc. Only the first 6 of the criteria set out above
(a-f) are used in the GEF TE quality rating, UNDP additions have a yes/no and comment section. To enable
consistency with the GEF EO expectations, use the GEF calculation for assessing the overall quality of the TE report
= o.3*(a+b) + 0.1* (c+d+e+f). This weights the overall rating towards relevance and achievement plus report
consistency and substantiation.
13
The responsible UNDP country office / regional bureau should provide a response to the TE including actions
that will be taken as a result of the findings. This gets posted to the Evaluation Resource Center
14
Of particular interest are the activities planned to scale up and replicate project outputs.
14



Government/MEM agreed to make follow-up together with TBS to finalize the gazettement of
the standards. The government also will make follow-up with Sida/MEM project to make sure
that the testing standards are in place as soon as possible to facilitate timely implementation
and enforcement of the standards. The Standards are now available at TBS and the copy can be
accessed for a nominal fee. Awaiting AG to issues the GN number
During project implementation the project undertook short term strategy to train a limited
number of technicians to support installation and after sales maintenance during
demonstration programme.
With regards to the long term sustainability the project mainstreamed Solar PV technology in
the VETA training programme/Curriculum and provided technical training to the VETA
instructors of the lake zone regions. The project has also provided equipment support for
training purposes to the VETA centers in Mwanza, Kagera, Shinyanga and Mara. The project has
just completed supporting VETA in the developing and translation training manuals for class
room training.
The project received a limited amount of funds (US$200,000) this year from TRAC to support
implementation of some hanging activities especially in the replication regions. The focus has
been mainly on the PV productive use and also providing training to the PV suppliers to ensure
quality service delivery.
Indicate actions that have been taken subsequently 15
c.







Training is being conducted and will be completed by mid November.
The Solar PV standards were developed by the UNDP/GEF Solar project and was jointly agreed
that Sida/MEM Solar Project will support capacity development of the implementing
institutions including procurement of equipment and training. Currently procurement of
equipment is underway
Additional support in a cost sharing basis will be provided to a limited number of attractive
small businesses in Kagera, Shinyanga and Mara to demonstrate functionality and applicability
of Solar PV in economical activities in rural areas.
Solar PV technology is one of the technologies that has identified by REA for rural energy
development, and support has been already provided to some projects through a business
proposal competition.
The project is working closely with the Government to make sure that the standards are
operationalized before closure of the project December 2009.
VETA has agreed and is taking the lead in mainstreaming the technology in the other VETA
centers in Tanzania.
The funds will support only a limited number of small businesses in the replication regions.
X.
Additional Comments 16
XI.
Information Sources for TE QA preparation
Project Document
PIR 2005
PIR 2006
PIR 2007
15
Consideration of subsequent actions by the responsible office will require a voice or written contact.
TE QA Reviewers should note any additional issues of interest / concern including ‘lessons’ of relevance to
subsequent evaluations and TE QA efforts.
16
15
PIR 2008
Midterm Evaluation
Terminal Evaluation
Terminal Evaluation Management Response
XII.
Information Gaps
XIII.
Ratings Scale
General ratings17:
 Highly Satisfactory (HS): no shortcomings
 Satisfactory (S): minor
 Moderately Satisfactory (MS):moderate
 Moderately Unsatisfactory (MU):
significant
 Unsatisfactory (U): major
 Highly Unsatisfactory (HU): severe
 Not applicable (N/A)
 Unable to assess (U/A)
Project sustainability ratings:
 Likely (L): negligible risks to sustainability
 Moderately Likely (ML) : moderate risks
 Moderately Unlikely (MU): significant
risks
 Unlikely (U): severe risks
 Highly Unlikely (HU)
 Not Applicable (N/A)
 Unable to Assess (U/A)
17
The ratings are supposed to be included in each of the TEs, based from GEF guidance. Shortcomings relate to
achievement of objectives in terms of relevance, effectiveness or efficiency, as well as the quality of M&E systems.
This same scale should be used to rate shortcomings in the terminal evaluation report.
16