UNDP-GEF Terminal Evaluations Review Form
I.
Background Data
Project Information
Project Title (per project appraisal document
submitted to GEF)
UNDP Project ID:
GEF Project ID:
Region:
Countries:
GEF Operational Program / Strategic Program:
Executing Agency:
Project Partners :
Project and Evaluation Milestones
(Milestones)
CEO endorsement:
Agency approval date:
Implementation start:
Midterm Evaluation:
Project Completion:
Terminal Evaluation Completion:
Consolidation and Implementation of the Patagonia
Coastal Zone Management Programme for
Biodiversity Conservation
PIMS 604
ARG 02 G 31
Latin America and Caribbean
Argentina
Coastal, Marine, and Freshwater Ecosystems - OP 2
Activities under this program concentrate on the
conservation and sustainable use of biodiversity in
coastal, wetland, mangrove, estuarine, marine, and
freshwater ecosystems.
Ministry of Foreign Affairs (in original PRODOC), as of
2002: Fundacion Patagonia Natural
All four provincial governments of the region,
environmental NGO, representatives of key industries
including fishing, tourist and oil, civil society
(Expected Dates)
30 June 2009
May 30, 2009
(Actual dates)
September 29, 1999
December 06, 1999
06 December 1999
July 2005
December 31, 2009
September 2010
TE & QA Review particulars
Terminal Evaluation author(s):
TE QA Reviewer:
TE QA Review date:
Draft TE QA Submission Date:
Final TE QA Submission Date:
UNDP EO task manager approval date & sign:
II.
James R. Barborak, Sandra Cesilini,
Carolina Vergara Lamarre
November 2010
November 26, 2010
20, December 2010
Alan Fox, 22, December 2010
Project Design & Objectives
1
a.
List the overall environmental objectives of the project:
As noted in the PRODOC: to conserve globally important marine and coastal biodiversity in Patagonia’s
coastal ecosystem by integrating conservation and biodiversity friendly production practices into regional
coastal planning and management.
b.
List the development objectives of the project:
This project is aimed at the improvement in the quality of life of human communities who depend on
coastal resources while maintaining biodiversity and productivity of Patagonia’s ecosystems. More
specifically, the purpose is to ensure that national, provincial and local stakeholders are able to effectively
manage and plan resource use in the context of integrated coastal zone management.
c.
Identify the key project components
The key components consisted of
 The Incorporation of the Province of Tierra del Fuego into the Patagonia Coastal Zone
Management Programme for biodiversity conservation.
 The consolidation and strengthening of the provincial and inter-provincial coordination
framework for long-term coastal zone planning and management aimed at biodiversity
conservation and biodiversity-friendly production practices.
 The establishment and full functionality of the Patagonia coastal Protected Areas Network for
conservation and sustainable use of biodiversity
 Raise awareness of biodiversity conservation through coastal zone management via the
Awareness Raising Programme
d.
Briefly assess the quality of project design, noting in particular the logical connections of
objectives to outcomes and outputs taking into account risks and assumptions, and the proposed
budget.
The relationship between project outputs, outcomes and objectives appears to be sufficiently logical, with
a consequent budget. Additionally, the PRODOC was reviewed and took into consideration the
recommendations made by the technical reviewer. These include: being more specific regarding the
involvement at the local and regional levels as well including an analysis of the economic benefits of
conservation biodiversity. The main risk assessed was that pertaining to the lack of political commitment
due to turnover at the national government level. This was taken into consideration by ensuring that
certain skills such as financial planning and management were included in training at the local and
regional levels.
e.
Have indicators been established for the achievement of project outputs and outcomes? If so do
they build from GEF guidance, are they SMART?1
Not in the PRODOC. There is mention of the establishment of indicators in the PRODOC regarding the
M&E system. However, the TE reviewer could not locate these indicators within the document. There was
no Performance Matrix provided nor a Logical Framework. The overall objective was broken down into
what were named as Specific Outputs, and then Activities. However, even in paragraph form, indicators
for measurement were absent. In subsequent PIRs, the Logical Framework appeared, however as the MTE
and the TE noted, at times it was difficult to determine the official version in use.
f.
Were there any delays in the project preparation phase, prior to project Inception? If so, why? Did
the delays result in changes to objectives and planned outcomes?
There was mention in the Spanish Project Document that the start of project implementation was delayed
resulting in GEF funds from 1999 being transferred to 2000, however it is unclear to the TE reviewer what
the causes were. This did not affect the project objectives, not its planned outcomes
1
The GEF Secretariat has issued guidance on the use of “SMART” indicators (Specific, Measureable, Achievable and
Attributable, Relevant and Realistic, and Time-bound, Timely, Trackable and Targeted). GEF climate change
indicators were developed in 2000, IW in 2002 & 2003, and biodiversity in 2003. For a description, see
www.gefweb.org/MonitoringandEvaluation/MEPoliciesProcedures/MEPIndicators/mepindicators.html
2
III.
Project Implementation
Project management
a.
Indicate whether there were changes in the environmental and development objectives of the
project during implementation, why these changes were made and what was the approval
process 2
Yes. However, the objectives were not changed entirely, only shifted in terms of time and focus. By 2002,
it was decided that Result 2 “The consolidation and strengthening of the provincial and inter-provincial
coordination framework for long-term coastal zone planning and management aimed at biodiversity
conservation and biodiversity-friendly production practices” would not be the focus of the first 2 years of
the project due to implementation difficulties between the Executing Agency (the Ministry of Foreign
Affairs) and the Technical Implementation Unit (TIU) (the NGO Fundacion Patagonia Natural). This
occurred because the Evaluation mission sent in 2002 regarded this Result as not feasible under prevailing
political conditions in Argentina.
b.
Were there delays in project implementation and completion? If so, indicate reasons and
comment on whether the delays affected project outcomes and sustainability.
Within a year of implementation there were already reports of project delays due to difficulties between
the executing agency, External Affairs and FPN as Technical Unit which lead to administrative and
managerial difficulties. The difficulties resulted in a review of implementation arrangements in July 2001.
An evaluation mission in 2001 recommended suspending project activities until a new implementation
agreement was reached; creating a management consortium represented by major project stakeholders
and strengthening the capacity of the TIU and give it greater responsibility in the implementation of the
project on site. UNDP subsequently stopped project disbursements to the EA in January 2002. At this
time, Argentina was also undergoing difficult economic times. Six months later, in June 2002 another
mission was sent in order to assess whether any progress had been made and it was found that overall no
agreement had been reached, there were still discrepancies over management roles and a substantial
amount of momentum had been lost and there was a need to re-build momentum. Result 2 (to
strengthen and consolidate regional coordination framework for long-term planning and management for
BD conservation and sustainable use) was also deemed not feasible under prevailing political conditions.
On june 14, 2002, an agreement was reached between External Affairs, UNDP and FPN. In general terms it
was decided that only Results 1,3 & 4 would be pursued in the next 2 years during which hopefully the
political situation in Argentina would stabilize. Then, Result 2 would be pursued. Phase one would be
executed under NGO Execution modality, with FPN as the Executing Agency. A Management consortium
would be established to evaluate project progress and this would be formed of the Ministry of Foreign
Affairs, UNDP, as well as other relevant ministries and provincial governments. The project closure date
was extended to end 2007 and two clearly defined new phases were established. The first of these phases
was defined as July 2002 to September 2004.
The project was actually extended from 5 to 9 years, concluding in 2009 once again due to the financial
breakdown in Argentina.
2
The GEF guidance indicates the following possible reasons for changes: a) original objectives were not sufficiently
articulated; b) exogenous conditions changed, due to which a change in objectives was needed; c) project was
restructured because original objectives were overambitious; d) project was restructured because of a lack of
progress; e) other (specify).
3
Co-financing3
Source of co-financing
Total4
Type (cash or in-kind)
Expected
Actual
host gov’t contribution
GEF Agency (ies)
Other agencies (bilateral / multilateral)
private sector
NGOs
other
Total co-financing
a.
Government (inkind) :
$US 7,300,000
$US 7.300.000
IDB/GoA via a project
IDB/WB/GoA1
$US 1,000,000
$US 3,900,000
(under negotiation
at the time of the
PRODOC)
$US 1,000,000
Wildlife Conservation
Society (WCS)/FPN
$US 500,000
$US 500, 000
In kind
Total budget: $us
12,700,000 (English
PRODOC)
$8,800,000
Is there sufficient clarity in the reported co-financing to substantiate in-kind and cash
co-financing from all listed sources?
Comment
(Yes/no)
Yes
The TE reviewer notes however that in the English PRODOC, the only budget breakdown detailing
financing and co-financing was in Section 3, near the beginning of the document. The Budget in Annex 6
was not included in the document. The Spanish PRODOC, however, included a very detailed project
budget in Annex D which included a break down over the 5 year span of the project. This, however,
lacked information regarding the sources of each funded activity. The Presupuesto A included within the
Spanish PRODOC itself, displays only the contributions of the GoA. The Section 3 regarding co-financing at
the beginning of the Spanish PRODOC does not mention the financing being negotiated between the
IDB/WB/GoA for $US 3,900,000.
b.
If there is a difference in the level of expected and actual co financing, what were the
reasons?
Comment (to what extent, and note whether justifications were given)
(Yes/no)
YES
Then co-financing mentioned in the Project document that was under negotiation via international
projects dealing with coastal zone management, biodiversity conservation, pollution, and protected areas
funded by the IDB, World Bank and GEF does not appear to have materialized. The MTE provides limited
evidence that the FPN has been contracted by the GEF Protected Areas Project to carry out specific work
3
GEF focuses considerable attention on the extent of co-financing across the portfolio, and uses co-financing as a
proxy for country ownership as well as an important indicator for sustainability. Accordingly, this table should be
filled in based on the numbers set out in the ProDoc for expected co-financing and then in the TE and/or Project
Close out note, on actual or realized co-financing. Especially in GEF-2 & 3 projects there may be instances where
no actual co-financing numbers have been tabulated, and/or no cash & in-kind breakouts have been provided.
4
Includes both co-financing during project design stage (PIF/PPG) and Project Implementation (ProDoc)
4
related to the establishment of the Monte Leon National Park, and had signed a general agreement with
the GEF Project on Biodiversity Conservation and Pollution Prevention, however there was little
evidence to suggest any other sort of practical cooperation. The TE reviewer could not find other evidence
regarding the materialization of this potential co-financing.
c.
Were components supported by co-financing well integrated into the overall
project?
(Yes/no)
No
The MTE mentions that there was not much evidence that the contributions from the GoA and IDB have
contributed to the Project objectives in any clearly identifiable way. This therefore questions how
integrated the co-financing was,
d.
Did the extent of materialization of co-financing affect project outcomes and/or
sustainability
(Yes/no)
No
Comment
It does not appear that the lack of materialization of co-financing from the IDB, World Bank and GEF
projectas mentioned above (b) affected the achievement of outcomes as noted by the MTE. It could
however, affect the eventual sustainability of the project.
Additional comments on co-financing:
Monitoring and Evaluation
a.
Assess the quality of the M&E design at project start up
Comment:
MU
Within the project document a breakdown of the Objective, outcomes and activities was included in
prose form however there was no Logical Framework nor Performance Matrix included. Additionally,
section 12 of the PRODOC pertaining to project Review, Reporting and Evaluation was very brief. It did
mention that monitoring would adhere to the UNDP monitoring system as well as quarterly reports of
the Project Implementation Unit (PIU) to the Project Executive Committee (PEC) and the Provincial
Coastal Zone Management Committees. In addition an annual report as well as financial report would be
prepared by the PIU for submission to the PEC. Reference was made to the use of annual external
technical evaluations as well. Reference to performance indicators was made however the TE reviewer
could not locate them. Overall, the paragraph lists only activities
Within the Spanish PRODOC, it was indicated that the matrix was to be found in Annex C. Unfortunately
this appears to have been cut off. Only Result 4 was legible however the names of categories were not
apparent and only the activity list along with the assumptions were fully legible.
By the first APR (2002), a Performance Measurement Matrix was apparent and included in the report,
complete with indicators. The MTE also noted deficiencies in the wording of some of the indicators,
mainly focussing on inputs in some cases rather than outputs and measurement. Some outcomes did not
have enough clearly identified activities, such as Outcome #4 (Awareness Raising)
b.
Assess the quality of M&E Plan Implementation
Comment:
S
Overall, the Project consistently underwent monitoring via annual PIRs and was subject to external
evaluations. The PIRs make reference to the findings and recommendations in the evaluations and take
them into consideration. The MTE mentioned that no Tripartite Reviews or meetings of the Management
Consortium had been scheduled as of 2005. The TE reviewer agrees with the overall conclusion reached
in the TE, viewing that overall the PIRs were very detailed and timely. The modifications to project
strategy to incorporate the recommendations of the MTE was also evident. An external MTE was carried
out as well as an external TE thus complying with UNDP-GEF requirements.
5
Was sufficient funding provided for M&E in the project budget?
There was not enough information for the TE reviewer to assess the financial allocation to M&E.
Was a midterm evaluation carried out and were MTE recommendations taken into account in
subsequent project implementation?
Yes. The MTE was carried out in 2005. One important recommendation concerned the creation of
Provincial Coastal Commissions the MTE suggested that they could create resistance from bureaucrats
who perceive them as a threat to their financial resources and the autonomy of their department. The
opposite is true of the Regional Coastal Commission. It is indispensable for coordinating the actions of
the Patagonian Provinces with respect to issues that depend on an ecosystem approach. A Regional
Commission would not compete directly for finances or power from existing management agencies, and
could possibly attract fresh financial resources. It also recommended making more use of modern
information technologies to improve the accessibility of educational materials.
The subsequent PIRs make reference to the amendments they have made in light of the findings from
the MTE (ie. Incorporating strategies to enhance financial sustainability; revising the outcome pertaining
to the formation of the Provincial CZM Committees,). Most importantly, its recommendation to keep
FPN as the EA was maintained via a participatory process of local and national stakeholders and ratified
in the 2005 Steering Committee meeting. The role of FPN as executing agency was confirmed by majority
and under recognition of National and Provincial authorities.
Were PIR self-evaluation ratings consistent with the MTE and TE findings? If not, were these
discrepancies identified by the project steering committee and addressed?
Yes, overall the PIR ratings were consistent with the MTE TE. Generally, they varied from S to HS.
Country Ownership
a.
Are there indications that government agencies in addition to the GEF Focal Point have
participated in the project? If so, how?
Yes. From the outset the participation of government agencies at various levels was evident. At the
provincial level the governors and government ministers of the provinces of Santa Cruz, Chubut, Rio
Negro and Tierra del Fuego as well as members of each of the departments with jurisdiction over the
coastal zone were also involved. Additionally at the regional level, the Patagonian Parliament agreed to
assist with the establishment of regional legislation within the scope of the Project. At the national level,
the Natural Resources and Sustainable Development Secretariat (Secretaría de Recursos Naturales y
Desarrollo Sustentable – SRNyDS) was maintained actively involved to ensure the required coordination
with other related efforts at federal levels. Meetings were held with the Fisheries Sub-secretariat and the
National Fisheries Research and Development Institute (Instituto Nacional de Investigación y Desarrollo
Pesquero – INIDEP) to define the coordination mechanisms for the implementation of the Project
activities in that sector during project design.
Additionally, the Project Executive Committee (PEC) was originally formed by the Ministry of Foreign
Affairs of Argentina, UNDP and FPN to oversee the execution of the Project. As has been elaborated
upon, this subsequently changed to include but the FPN. The Provincial Coastal Management
Committees (Provincial CMCs): included representatives of key provincial government agencies, NGOs,
the private sector and universities.
b.
Has the level of country ownership affected project outcomes and sustainability? If so, in what
ways?
6
Yes. The establishment of Coastal Committees in the provinces was seen as an exclusively political
objective by the Project Executing Agency. All decision making with regard to this project objective was
to be left entirely within the realm of the Executing Agency, which initially had been the Ministry of
Foreign Affairs. NGO participation was restricted when the time came to address the implementation of
this objective. FPN felt that although the decision to establish management structures was indeed a
political one, NGOs have a significant role to play in the building, facilitating, and maintaining of these
management structures, and as supportive overseers for effective implementation. The Evaluation
Mission identified this particular problem and proposed viable recommendations and solutions for
resolving it via the establishment of a Consortium to guide the project The MTE also noted that the
private sector and local communities had been omitted from the list of actors in the PRODOC. These
actors are crucial for a functional IZCM. The TE made similar recommendations to include the private
sector in order to enhance country ownership.
IA & EA Execution
a.
Assessment of the quality of Implementing Agency Execution:
 Focus on results
 Adequacy of supervision
 Quality of risk management
 Responsiveness to significant implementation problems (if any)
 Quality and timeliness of technical support to the project team
 Candor and realism in supervision reporting
 Suitability of chosen executing agency for project execution
Comment:
S
Overall, UNDP appears to have provided timely and constructive support in its supervisory role.
Throughout the documentation it was commented that UNDP facilitated discussions between project
authorities by organizing some meetings with local authorities and a workshop with all national
environmental authorities. Links between the project and authorities were strengthened and a dialogue
environment was created. “The Patagonian Agreement”, an agreement signed in 2004 between the
Governors of Rio Negro, Chubut, Santa Cruz and Tierra del Fuego and the Federal Government to endorse
the project, with Fundacion Patagonia Natural as Executing Agency and UNDP for the establishment of
the Provincial Coastal Zone management Committees, is an example of this and other agreements
signed with Coastal Municipalities, research institutes, universities, National Park Administration, etc.
Additionally, workshops relevant to the project were also organized such as LAC UNDP GEF Regional
Coordination Unit and the UNDP GEF HQ Biodiversity team on Long-term Financing for National Systems
of Protected Areas. It allowed information exchange and training on the critical issue of Protected Areas
financing and enhanced information exchange and lesson learning between many of the biodiversity SP1
projects in LAC.
Lastly of course, as a supervisory agent, its responsibility and sound judgment to determine when to
disburse funds and when to halt them became evident within the first year of implementation, during
the stalemate between FPN and the Ministry of Foreign Affairs The TE came to a similar conclusion that
overall UNDP was an effective implementing agency, especially offering balance to the nongovernmental executing agency.
b.
Assessment of the quality of Executing Agency Execution:
 Focus on results
 Adequacy of supervision inputs and processes
HS
7
 Quality and timeliness of financial management support to the Project Team
 Quality of risk management
 Candor and realism in supervision reporting
Comment:
Over the first year of implementation of the project, starting from June 2000, difficulties were observed
as far as the relation between the Executing Agent (EA) – Ministry of Foreign Affairs- and the Technical
Implementation Unit (TIU) –Fundacion Patagonia Natural- These resulted in delays in disbursements and
slow progress towards achievement of objectives. After the initial difficulties with the management
structures between the Ministry of Foreign Affairs and the Fundacion Patagonia Natural, an Evaluation
Mission identified and proposed viable recommendations and solutions for resolving it via the
establishment of a Consortium to guide project implementation. The idea was that this would result in a
far more participatory approach to the establishment of coastal zone management structures, thereby
increasing this component’s (Result 2) chances of success.
Both the MTE and TE noted the successful execution and timeliness in completion of tasks by FPN that
surpassed original targets and planned activities in the Project document. However it also mentioned
that FPN has been a leading NGO in biodiversity conservation for many years and questions what will
become of its role as biodiversity advocate once the project moves into the second Phase, since many
stakeholders came to view FPN as synonymous with the project itself and its role as a facilitator between
stakeholders.
Overall Quality of Project Outcomes
(UNDP EO rate)
S
(TE rate)
S
Comments and justification:
The MTE of 2005 evaluated the progress of the project since the restart of activities in 2002. Generally, it noted
the success of the first half of the project and commended the implementing agency for its use of adaptive
management. Both the MTE and the TE noted the project was under budget, on time and more activities had
been undertaken than in the original project design. Both highlighted the positive note of such successful
implementation given that Argentina was suffering through difficult sociopolitical and economic times.
Generally all the outcomes were satisfactory.
Overall Quality of Project Implementation / Execution
S
S
Comments and justifications:
The project suffered difficulties in implementation between the original EA (the Ministry of Foreign Affairs) and
FPN, which resulted in UNDP halting disbursements in 2002 for six months. The MTE found the implementation
to have been flexible and constructive. While working to achieve the project objectives, the Project staff
explored different means for achieving the desired outcome through participatory approaches that allow them
to tap into the experience of counterparts in the region. One weakness however is that numerous adjustments
to the logical framework rendered it difficult to ascertain which version was the official version currently being
used.
The TE also noted that the fluid collaboration between all the different actors: civil society, government,
business was considered to be an innovative strategic alliance. Once again the TE found the project to have run
smoothly given the rocky start, and commended FPN.
8
IV.
Assessment of Outcomes & Sustainability5
Criteria6
Relevance
 Were the project’s outcomes consistent with
the focal areas / operational program strategies
and country priorities?
Comments and justifications:
UNDP EO
S
Terminal Evaluation
S
Generally speaking the project’s outcomes are consistent with the focal areas and country priorities especially
given the fact that the project itself was a follow-up project to the Pilot Phase project and is designed to
consolidate and implement the Patagonia Coastal Zone Management Programme for Biodiversity Conservation.
The documentation details coherence with the uniqueness and importance of the biodiversity found on the
Patagonian coastline. The MTE however noted that in some project documents it was clear that biodiversity
conservation was the primary concern, while in others the balance between the quality of life and biodiversity
conservation is primary. And while relevant, the design and strategies focused more on what they wanted to
achieve and not on how it would be achieved and given the fact that this mandate was very political by
attempting to develop integrated approaches to address the interests of all Patagonian stakeholders, this
became even more daunting given the breakdown of the Argentinean economy in 2001.
The MTE did not find substantial evidence that linkages had been made between other policy initiatives such as
General Environmental Law, a Biodiversity Law, a Coastal Management Law, and a National Biodiversity
Strategy and Action Plan, although mentioned in the project document. The TE came to a similar conclusion
given the particular biodiversity importance of the region (habitat of certain key species) and threats to the
region (oil extraction and fishing). It also rated the relevance as satisfactory.
Effectiveness
S
 Are the actual project outcomes commensurate
with the original or modified project objectives?
Comments and justifications:
S
The MTE regarded the effectiveness of the Phase 1( 2002-2007) to have been highly satisfactory, on time and
under budget. That phase focused on the incorporation of the Province of Tierra del Fuego into the Program, the
development of a Patagonian Coastal Protected areas network, and the implementation of a public-awareness
program.,. By the completion of the MTE, the project was generally on schedule and under budget.
The TE commended the overall achievement of the outcomes as well as the adaptive management used
throughout the project such as using public audiences to generate consensus instead of the regular institutional
methods which can sometimes create bottlenecks. The TE also noted that the project went above and beyond
what it had set out to do, even accomplishing the establishment of a post graduate program.
5
Each of the TEs should use the provided rating scale (set out at the end of this template / guide). This review
should include its own ratings assessments of project relevance, effectiveness, sustainability, etc, with reference
provided on the same ratings provided in the Terminal Evaluation. The reviewer will need to justify a different
rating based on new information or indicating errors made in the TE assessment. In cases where there is
insufficient information in the TE to make a determination, the reviewer should indicated U/A = unable to assess.
6
See Guidelines for GEF Agencies… 7 – 14 for a discussion on the criteria definitions. The bulleted points are
taken from this GEF guidance. The ratings will all be based on the 6 point scale of satisfactory achievement except
for sustainability – which switches to likelihood of achievement (also considering the likelihood of avoidance of
risks to sustainability)
9
Based on the review of all the documentation, the TE reviewer considers the effectiveness to be satisfactory.
Efficiency
S
(S)
 Was the project cost effective?
 Was project implementation delayed, and if it
was, did that affect cost effectiveness?
Comments and justifications: The project disbursement rate appears to have been low. By the 2002 PIR
project disbursement (percentage of planned vs. actual expenditures) was low, at 22%. By 2004, the
disbursement ratio was 52%. The MTE did not have access to copies of the Project budget or information
regarding the funds that remained for Phase 2 of the Project, however, based on the review of APRs and the
2003 audit, it concluded that the project had been highly cost-effective. It deemed this due to the purchasing
power of the American dollar during the financial crisis, the fiscal caution of FPN and low per diem rates for
travel. The TE noted overall sound cost effectiveness in regards to the funds provided by GEF and that the
investments the project made in the coastal zone such as institutional strengthening and environmental
awareness-raising have helped to leverage other funds and has resulted in the incorporation of certain activities
originally funded by the project into government funded programs. This has also apparently aided in generating
new investments and conservation activities from other donors, public and prívate entities in the region
targeted by the project. This being said, it was not clear to the TE reviewer what were examples of such
programs, nor what these other funds were or who the donors were. .
Sustainability
a.
Financial resources:
What is the likelihood that financial resources
will be available to continue activities resulting
in continued benefits?
b.
Socio-political:
Is there sufficient public stakeholder awareness
and support for the continuation of activities
providing benefit?
c.
Institutional framework and governance:
Are required systems for accountability and
transparency plus technical know-how in place?
d.
Environmental :
Are there environmental risks that can
undermine the future flow of project
environmental benefits?
Comments and justifications:
UNDP EO
(likelihood)
ML
TE
(likelihood)
Overall rating was
ML -L
No individual rating
L
No individual rating
ML
No individual rating
ML
No individual rating
The MTE noted that what it considered to be a major flaw consisted in underestimation of financial and time
resources and diverse capacities to achieve sustainability. FPN will need significant support especially
financially. The financial plan to maintain the protected areas was deemed weak and inconsistent as well. In
addition, the potential and importance of the Trust Fund mechanism for providing incentives and filling gaps for
developing an ICZM approach should not be underestimated or neglected. It also commented that little thought
10
had been given to moving from a small project to a fully sustainable programme, mainly due to the difficulties
and challenges in getting everything started successfully in the first phase. The TE noted that the project has
made some headway in diversifying its financial sources however work still needs to be done in areas such as
developing conservation trusts and developing cost per environmental service financial mechanisms so as not to
depend on government financial sources seeing as this is not always reliable. Socio-politically speaking efforts to
promote sustainability have been primarily focused on awareness-raising which has resulted in an increase in
voluntary actions from the Patagonian society as a whole. Although progress has been made regarding
cooperation between provincial governments it appears that this is one area that might pose a greater threat to
sustainability due to differences in provincial laws, turnover and the general heterogeneity of actors. Continuing
to include NGOS, companies and the local communities as direct active participants will be crucial. The most
significant risk environmentally speaking, are changes that could result from climate change impacts which have
thus far not been taken into consideration. These could have detrimental effects to the conservation efforts.
Monitoring & Evaluation system
 Is there a sound M&E plan in place to monitor
results and track progress toward achieving
objectives?
 Was the M&E plan sufficiently budgeted and
then carried out?
Comments and justifications:
UNDP EO rating
S
TE rating
S
Although initially there was a lack of a PMF, monitoring did occur sufficiently throughout the project and did
demonstrate adaptive management. The most obvious example would be the rearrangement of
implementation management when an impasse was reached in 2002. As previously mentioned, the MTE noted
no Tripartite meetings had been called by UNDP and the Logical Framework underwent various amendments,
demonstrating principles of adaptive management. There were regular PIR submissions and two external
evaluations.
Overall Project Rating
Comments and justifications:
S
S
Despite its shortcomings, the TE reviewer considers the overall project rating to be satisfactory based on the
findings described in previous sections.
V.
a.
Impact7
The project has demonstrated verifiable improvement in ecological status
(Yes/no)
Yes
Comment
The TE reviewer did not have access to the last PIR 2010, therefore the assessment here was based
primarily on details provided in PIR 2009 and the TE. However the reviewer found the TE lacked sufficient
details in this regard. Some examples leading the reviewer to come to a positive assessment regarding
impacts conclusion include the levels of conservation at June 30, 2009
 Indicator species (Magellanic penguin, South American sea lion, Southern elephant seal and
Southern right whale, see below) all remain stable or increase.
7
All TEs discuss sustainability but this is not synonymous with impact. It is important to look for references to
verifiable pollution reduction and ecological status improvement, and/or whether there are process indictors that
suggest such impacts should occur in the future as a result of project achievements. If the project is a foundation
setting effort, it is not expected to achieve stress reduction and/or status change impacts in the short to medium
term.
11


b.
Regarding the Magellanic penguin: populations remain stable at 1 million pairs.
Regarding the South American sea lion breeding rates Total population currently close to
100,000 animals including all age classes. Counts indicate that populations appear to also be
increasing in southern Patagonia. (an increase from the baseline)
The project has demonstrated verifiable reductions in stress on ecological systems
(Yes/no)
Yes.
Comment
The TE reviewer bases her judgment on the same examples cited above.
c.
The project has demonstrated through specified process indicators that progress is
being made towards achievement of stress reduction and/or ecological
improvement.
Comment
(Yes/no)
Yes
d.
As a result of the project, there have been regulatory and policy changes at regional,
national and/or local levels.
(Yes/no)
Yes
Comment
The TE reviewer has noted the citation numerous times of a significant achievement regarding the
creation of the Marine Park in Golfo San Jorge (IMPGSJ),Law # 26.446 approved by National Congress, in
December 3, 2008. The Chubut Provincial Legislature had approved the Park in 2007 (Provincial Law #
5.668) as a first step in the process. The new National Law expands the formal protection of colonies of
marine birds from 50% to 75%, plus 100% of all fur seal breeding colonies on the coast of Patagonia in
Argentina. Additionally, in June 2008 the Federal Fisheries Council of Argentina approved the
establishment of an 1800 sq km no-fishing zone over the Burdwood Bank off the tip of Tierra del Fuego,
to protect benthic communities in this unique ecosystem. The decision expands protection of valuable
marine areas by 1%. In 2008 the National Parks Service of Argentina (Administración de Parques
Nacionales, APN) signed an agreement with the Argentine Navy whereby the 10,000 hectare property on
Peninsula Valdes at Punta Buenos Aires owned by the Navy will be managed as a protected area.
These are but examples and are not exhaustive as other protected areas have been established as well.
e.
Summarize the achieved intended or unintended impacts of the project:
Overall the main impacts the project has produced, concern the stabilization and in some cases the
increase of the population of certain key species and the legal establishment and recognition of new
protected areas, with budgeting and personnel for their management.
VI.
a.
Catalytic Role8
Production of a public good
Lowest level of catalytic result, including for instance development of new
technologies and approaches. No significant actions were taken to build on this
(Yes/no)
Yes
8
The GEF EO Guidelines on TERs notes that at this point there is no specific guidance for assessing catalytic roles
and replication in GEF projects. The expectation is that this will be required in the future, hence a section in the
review effort to consider four levels of catalytic results. No ratings are expected, however the reviewer should
consider the extent to which the project has achieved: a) production of a public good, b) demonstration, c)
replication, and d)scaling up.
12
achievement, so the catalytic effect is left to ‘market forces’
Comment
The Project has mostly resulted in public goods including protected areas and an increase in key species
population via inter-provincial collaboration.
b.
Demonstration
Steps have been taken to catalyze the public good, for instance through the
development of demonstration sites, successful information dissemination and
training
Comment
-
(Yes/no)
No
c.
Replication
Activities, demonstrations, and/or techniques are repeated within or outside the
project, nationally or internationally.
Comment
(Yes/no)
No
d.
Scaling up
Approaches developed through the project are taken up on a regional / national
scale, becoming widely accepted, and perhaps legally required.
Comment
(Yes/no)
No
I.
a.
Mainstreaming9
Based from the ProDoc and Terminal Evaluation, has the project focused attention on the nexus
of issues connecting environmental protection with other socio-economic priorities of UNDP such
as poverty reduction, crisis prevention and recovery, and democratic governance?
Yes. Generally speaking, democratic governance and poverty reduction were embedded into some of the
outcomes. Notably the decentralized nature of the project execution, the bottom-up approach and
consultative process with the variety of stakeholders, stresses democratic governance. Additionally,
activities regarding sustainable fishing and ecotourism were related to poverty reduction although it is
not clear to the TE reviewer whether they were integrated as alternative livelihoods.
b.
Is the focus of the project identified as a priority in the UNDP country programme and as a
national/regional priority in a corresponding development plan?
9
In addition to UNDP/GEF objectives aimed at global environmental benefit, other UNDP objectives should be
considered in the Review. These include the extent to which mainstreaming of key UNDP objectives on poverty
alleviation, crisis prevention and recovery, and democratic governance have been taken into account.
Increasingly, UNDP is expecting that these objectives are acknowledged in UNDP-GEF projects; however this has
not been a steadfast requirement in the past. It will be useful to note instances where such mainstreaming may
have occurred, in order to emphasize these linkages. For instance, if environmental protection activities have
contributed to improvements in local economic well being, better preparations to cope with natural disasters, and
broader improvements in governance, this should be mentioned. Both UNDP and GEF are focusing greater
attention to ensure that gender issues are taken into account in project formulation and implementation, (see
UNDP Gender Equality Strategy 2008-2011).
13
Yes, generally speaking the conservation of biodiversity is a national and regional priority as emphasized
in General Environmental Law, a Biodiversity Law, a Coastal Management Law, and a National
Biodiversity Strategy and Action Plan. There is no direct mention of the UNDP country programme
however the PRODOC does mention a programmatic approach to be carried out by UNDP and the World
Bank through a forthcoming initiative which was to focus on oil spill contingency planning and
management, mitigation of threats related to coastal shipping and land-based sources of pollution
(urban, industrial, agricultural), and distant seas fisheries management. UNDP had mentioned that it
would address those pertaining to coastal wildlife management; biodiversity friendly fishing practices for
near-shore and artisan fisheries; and strengthening of protected areas, including ecotourism
development.
c.
Have gender issues been taken into account in project design and implementation, (i.e. project
team composition, gender-related aspects of pollution impacts, stakeholder outreach to women’s
groups, etc)? If so, indicate how.
No. The TE reviewer could only find limited reference to gender implications and none within the project
design. There was mention of gender within both the MTE and the TE in regard to female park rangers.
Is it possible to identify and define positive or negative effects of the project on local populations
(e.g. income generation/job creation, improved natural resource management arrangements
with local groups, improvement in policy frameworks for resource allocation and distribution,
regeneration of natural resources for long term sustainability)?
Positive effects: the creation of newly protected and recognized areas has increased the local capacity to
technically and financially manage these parks although capacities still need to be built in this regard; there
has been a general increase in social capital and cohesion among the Patagonians vis-à-vis conservation of
their common environment; generally development of tourism (job creation), protection and sustainable
use of fish and oil resources (job protection), decrease of pollution etc.; the population has been exposed
to different possible revenue sources such as ecotourism .
Negative effects: it is not clear how the considerations of the indigenous communities have been taken
into account; it is apparent that consideration for the effects of climate change as mentioned in the TE have
not been taken into account in project design nor in implementation. This is especially relevant when
proposing alternative sustainable livelihoods to local populations as climate change can have a detrimental
effect on the sustainability of such livelihoods.
VII.
a.
Project Lessons & Recommendations
Summarize the key lessons, good practices and approaches mentioned in the TE that may be
applicable to other projects:
The key lessons the TE reviewer finds applicable to other projects include the following:




The use of a recognized and experienced NGO as an executing agency can be an effective and
efficient management alternative
In such projects covering multiple states or jurisdictions, it is important to resist the temptation of
utilizing too much of the project budget to invest in multiple offices, or personnel
Investments in formal and non-formal education are usually not reaped during the project
lifetime itself, but in the years that follow, thus aiding the sustainability of the initiative.
The process of establishing new inter-governmental structures to coordinate programs and policy
management for integrated management in a region (in this case the coastal region) requires a lot
of time, patience and flexibility
14
b.
Summarize the main recommendations set out in the TE:






c.
It is important to strengthen the technical and financial capacities of the provincial government,
local NGOs and other civil society actors in terms of planning and management of coastal marine
resources.
Regarding management of financial resources: Explore and promote various models of
management and financing of conservation projects that are innovative, such as conservation
trusts; promote and search for a wide range of external sources of financial resources as well as
the internal generation of financial resources
Develop studies concerning different sustainable institutional arrangements that will take change
of the management of such a large coastal zone initiative after project completion (and the NGO
executing agency in this case will no longer be the one solely in charge).
Strengthen public use programs and public awareness-raising initiatives in the protected areas
(public and private) in order to diversify products and ecotourism sites and other activities.
Increase the number and participation of business partners and make better use of practices and
modalities already in use such as certification and locally made brands.
Promote permanent formal structures of policies and programs of management of the coastal
zone between the national government, provincial governments and civil society.
Are there lessons and recommendations stemming from the project outcomes and
impacts that were not mentioned in the TE?
(Yes/no)
Yes.
Comment (suggest lessons and recommendations);
The TE reviewer would like to add four recommendations:




VIII.
Climate change should be integrated into coastal zone management projects given the impacts
climate change can have on coastal communities, especially in vulnerable areas of the globe such
as the polar regions (Patagonia being quite near to the South pole). These include but are not
limited to sea level rise, changes in water currents, temperature etc all of which have direct
impacts on the biodiversity being conserved and local livelihoods
A gender perspective and strategy should be integrated directly from the beginning and it should
be clearly laid out how such outreach will be achieved. The TE reviewer only found limited
references to female park wardens.
Consideration to Indigenous communities should also be explicitly laid out.
Engaging the private sector from the beginning by awareness-raising concerning the economic
benefits of biodiversity conservation (ie. New markets for organic, native species) can help foment
financial sustainability in the long run. Since this may be a long process, engaging the private
sector from the outset is important.
Quality of Terminal Evaluation Report
Assessment Criteria10
Rating
10
The first 6 of these criteria are taken directly from the Guidelines for GEF Agencies in Conducting Terminal
Evaluations (pg 16); the rest are additional questions to get a better fix on TE quality. The bullets under each initial
statement have been added as additional points to aid in considering TE quality.
15
a.
Does the terminal evaluation report (TE) present an assessment of all relevant
outcomes and achievement of project objectives in the context of the focal area
program indicators, if applicable?
 Project outputs and outcomes are assessed against ProDoc/LFA plans,
including verifiable indicators.
 Variances between planned and actual results have been identified and
assessed, including adaptive management strategies used.
 TE has addressed project contributions towards GEF objectives (global
environmental benefit) as well as national development goals and strategies
Comment:
(rate)
S
Yes. Generally speaking the TE made an adequate assessment. The TE reviewer however found that at
times it lacked detail. Although it provided a summary of what was achieved in order to compare with
what was initially targeted, the reviewer preferred to refer to the PIRs.
b.
Is the TE consistent, the evidence presented complete and convincing, and the
ratings well substantiated?
 Assessment of project results is well organized, clearly written and
unambiguous.
Comment:
(rate)
S
Yes. The TE’s assessment was logically based on the findings from the MTE and the PIRs. It was clearly
written however lacked sufficient detail (ie. Table 4 Global Assessment of the Outcomes achieved).
c.
Does the TE presents a sound assessment of the sustainability of project outcomes?
 Project design measures and/ or strategies for sustaining project results are
assessed.
 Socio-political risks to sustainability are considered, including stakeholder
ownership
 Institutional frameworks and governance risks to sustainability are
considered, including requisite systems for accountability and transparency
and required technical know-how
 Financial and technical resources required for sustaining project results are
considered
Comment:
(rate)
S
Yes. Generally the TE did explain each component of sustainability clearly and logically. It did not
however give a rating to each component.
d.
Are the lessons and recommendations listed in the TE supported by the evidence
presented and relevant to the GEF portfolio and future projects?
 The lessons learned draw upon specific observations or data compiled
during the evaluation.
 The recommendations provide specific advice for the project exit strategy or
post-project sustainability.
 The recommendations provide specific advice for future projects or
programming, or similar projects.
 The recommendations are sufficiently practical and in the realm of
(rate)
HS
16
feasibility for potential implementation.
 The lessons learned and recommendations are concise, clearly written and
understandable.
Comment:
The TE reviewer found the list of conclusions, recommendations and lessons learned explicit, wellfounded relevant to the project and future similar initiatives.
e.
Does the TE provides the actual project costs (totals, per activity, and per source)
and actual co-financing used?
 Project cost and funding data are presented, including actual co-financing
from each source.
 Variances between planned and actual expenditures are assessed and
explained.
 Observations from financial audits completed for the project are considered.
 Issues related to financing and co-financing commitments and performance
are discussed and explained
Comment:
(rate)
U
Although there was a brief analysis, there was no such detail provided in the TE.
f.
Does the TE include an assessment of the quality of the M&E plan at entry, the
operation of the M&E system used during implementation, and the extent M&E was
sufficiently budgeted for during preparation and properly funded during
implementation?
 The existence and quality of the M&E plan are assessed, including baseline
conditions, methodology and roles and responsibilities.
 The extent to which M&E were sufficiently budgeted and funded during
project preparation and implementation is assessed.
 The effectiveness of monitoring indicators from the project document for
measuring progress and performance is assessed.
 Compliance with the progress and financial reporting requirements/
schedule is assessed, including quality and timeliness of reports.
 The value and effectiveness of the monitoring and evaluation reports and
process was discussed with participants and assessed.
 The follow-up actions, or adaptive management, taken to respond to
monitoring and evaluation reports is assessed.
Comment:
(rate)
S
The TE included an overall assessment of the M&E system as a whole however in terms of financial
budgeting, it only assessed the contributions made by GEF. The TE indicated it did not have enough
information/documentation to make an assessment on the co-financing. For the latter it presented a
table from the 2010 PIR displaying the co-financed amounts.
g.
Has the TE identified the processes and factors that affected the attainment of
results?
 Including (inter alia): project design; country ownership; stakeholder
involvement; financial planning; project implementation; Executing Agency
(Yes/no)
Yes
17
supervision; co-financing)
Comment:
Overall, the TE did address the main obstacles such as the initial management structure, the financial
crisis in Argentina.
h.
Does the TE provide an assessment of the project assumptions and risks as set out in
the Project Document and LFA?
 The TE provides an assessment of the stated assumptions and risks, whether
they are logical and robust, and have helped to determine activities and
planned outputs.
 Externalities, (i.e. effects of climate change, global economic crisi, etc), were
been considered
Comment:
(Yes/no)
No
The TE reviewer did not find sufficient evidence of such an analysis in the TE.
i.
Were key stakeholders or stakeholder groups, or a representative sample thereof,
consulted during the TE fact finding process?
 The TE includes a listing of persons interviewed
 A questionnaire was distributed and an analysis of responses is provided
 Members of civil society, NGOs and private sector were given an opportunity
to provide their views
 As relevant, local ministry officials were interviewed
 Pertinent national ministry officials in addition to the environmental focal
point were interviewed
Comment:
(Yes/no)
Yes
Yes, the TE provided a list of stakeholders met as well as sites visited.
j.
Does the TE Terms of Reference comply with UNDP & GEF guidance, including
concerning conflicts of interest11?
(Yes/no)
Yes
Comment (gaps, conflicts, inconsistencies):
There is no specific mention of conflict of interest in the ToRs.
k.
Does the TE demonstrate good evaluation practice?
(Yes/no)
Yes
Comment (TE strengths and weaknesses):
Strengths: consulted a wide variety of stakeholders; provided concrete and applicable lessons learned and
recommendations; clearly written
Weaknesses: it lacked detail and specific numeric examples (ie. Number of key species conserved; number
of new areas protected etc)
11
UNEG, GEF & UNDP guidelines stipulate that except in rare and unusual circumstances, Evaluators engaged by a
UN agency shall not have had any responsibility for the design, implementation or supervision of any of the
projects, programs or policies that they are evaluating.
18
l.
Are there any evaluation findings involving significant administrative, financial
and/or human resources issues that require UNDP follow up, (e.g. corruption,
mismanagement, etc)?
(Note issues and UNDP management response)
(Yes/no)
No
Overall Rating for the Terminal Evaluation12
Justification:
S
Overall, despite the lack of detail in certain cases, the TE substantiated its conclusions and
findings and offered concrete and relevant recommendations and lessons learned which
will be valuable for future similar initiatives.
IX.
Management responses, Subsequent Actions and Sustainability
A Management response13 to the evaluation was submitted
a.
b.

List key proposed follow-up actions14
c.

Indicate actions that have been taken subsequently 15
X.
(Yes/no) NO Date:
Additional Comments 16
XI.
Information Sources for TE QA preparation
PIRS (2002, 2004, 2005 , 2006, 2007, 2008, 2009)
2002 Mission Report
MTE 2005
Argentina Proposal English
Argentina PRODOC Spanish 1
Argentina PRODOC Spanish 2
TE 2010
12
Use GEF numerical ratings 1-6, whereby 6= highly satisfactory, etc. Only the first 6 of the criteria set out above
(a-f) are used in the GEF TE quality rating, UNDP additions have a yes/no and comment section. To enable
consistency with the GEF EO expectations, use the GEF calculation for assessing the overall quality of the TE report
= o.3*(a+b) + 0.1* (c+d+e+f). This weights the overall rating towards relevance and achievement plus report
consistency and substantiation.
13
The responsible UNDP country office / regional bureau should provide a response to the TE including actions
that will be taken as a result of the findings. This gets posted to the Evaluation Resource Center
14
Of particular interest are the activities planned to scale up and replicate project outputs.
15
Consideration of subsequent actions by the responsible office will require a voice or written contact.
16
TE QA Reviewers should note any additional issues of interest / concern including ‘lessons’ of relevance to
subsequent evaluations and TE QA efforts.
19
TE annexes 2010
XII.
Information Gaps
XIII.
Ratings Scale
General ratings17:
 Highly Satisfactory (HS): no shortcomings
 Satisfactory (S): minor
 Moderately Satisfactory (MS):moderate
 Moderately Unsatisfactory (MU):
significant
 Unsatisfactory (U): major
 Highly Unsatisfactory (HU): severe
 Not applicable (N/A)
 Unable to assess (U/A)
Project sustainability ratings:
 Likely (L): negligible risks to sustainability
 Moderately Likely (ML) : moderate risks
 Moderately Unlikely (MU): significant
risks
 Unlikely (U): severe risks
 Highly Unlikely (HU)
 Not Applicable (N/A)
 Unable to Assess (U/A)
17
The ratings are supposed to be included in each of the TEs, based from GEF guidance. Shortcomings relate to
achievement of objectives in terms of relevance, effectiveness or efficiency, as well as the quality of M&E systems.
This same scale should be used to rate shortcomings in the terminal evaluation report.
20