ANNEX L
Quality Assessment of the Evaluation Report
All UNEP evaluation reports are subject to a quality assessment by the Evaluation Office. The quality assessment is used as a tool for providing
structured feedback to the evaluation consultants. The quality of the draft evaluation report is assessed and rated against the following criteria:
Substantive report quality criteria
A. Strategic relevance: Does the report present a well-reasoned, complete
and evidence-based assessment of strategic relevance of the intervention?
B. Achievement of outputs: Does the report present a well-reasoned,
complete and evidence-based assessment of outputs delivered by the
intervention (including their quality)?
C. Presentation Theory of Change: Is the Theory of Change of the
intervention clearly presented? Are causal pathways logical and complete
(including drivers, assumptions and key actors)?
D. Effectiveness - Attainment of project objectives and results: Does the
report present a well-reasoned, complete and evidence-based assessment of
the achievement of the relevant outcomes and project objectives?
UNEP EO Comments
Draft report:
This section has been discussed in
sufficient detail to support the
rating provided. There is evidence
provided to support the
assessment of project Relevance
as HS
Final report:
Draft report:
The section is reported in detail
and gives evidence on the quality
of outputs based on interviews
and collected data
Final report:
Draft report:
The TOC is presented as a
schematic and explained in
detailed narrative. Causal
pathways in the TOC diagram are
presented but there are no
distinctions made between the
results chains – all outputs lead to
all outcomes which in turn lead to
all intermediate states and on to
the intended Impact. The text
however describes these outputs,
outcomes, intermediate states,
etc. in satisfactory detail
Final report:
Draft report:
The report indicates that quite a
lot of effort has been made
towards achieving intended results
– as evidenced by various reports
and interviews – the evaluator
attributes this largely to the
effectiveness of the Secretariat.
Draft
Report
Rating
6
6
5.5
5.5
Final report:
E. Sustainability and replication: Does the report present a well-reasoned
and evidence-based assessment of sustainability of outcomes and replication
/ catalytic effects?
F. Efficiency: Does the report present a well-reasoned, complete and
evidence-based assessment of efficiency?
Draft report:
The report gives sufficient
information and examples to
describe project performance
under the sustainability criteria
Final report:
Draft report: The report escapes
making any assessment on
cost/time saving by the project. It
appears the project was extended
significantly (from 3 to 10 yrs). No
valid explanation is given other
than that the project supports an
on-going process. The report
implies the project cannot be held
accountable for applying efficiency
measures
5
4
Final
Report
Rating
Final report:
G. Factors affecting project performance: Does the report present a wellreasoned, complete and evidence-based assessment of all factors affecting
project performance? In particular, does the report include the actual project
costs (total and per activity) and actual co-financing used; and an assessment
of the quality of the project M&E system and its use for project
management?
H. Quality and utility of the recommendations: Are recommendations based
on explicit evaluation findings? Do recommendations specify the actions
necessary to correct existing conditions or improve operations (‘who?’
‘what?’ ‘where?’ ‘when?)’. Can they be implemented?
I. Quality and utility of the lessons: Are lessons based on explicit evaluation
findings? Do they suggest prescriptive action? Do they specify in which
contexts they are applicable?
Draft report: This section is
covered adequately for the most
part but needs some improvement
to make complete. Under ‘project
implementation and management’
one gets a sense that the
evaluator avoids going into detail
on how the low staffing at ISCC
has
impacted
the
project
performance – it is alluded to, but
the details on how this has
affected project management are
not disclosed. Under ‘supervision
and technical back-stopping’ there
is insufficient evidence to support
the rating provided.
Final report:
Draft report: They are based on
findings and for the most part they
state who should undertake the
actions recommended. They are
however disjointed - having
several issues being mentioned
under one recommendation - that
it becomes difficult to consolidate
the recommendation into one
concise and actionable proposal.
Final report:
Draft report: The lessons are
definitely based on the findings
but because the context of the
lesson is missing, this linkage can
only be assumed. The prescriptive
actions are not well framed and
need further improvement. It is
also not quite clear how they are
applicable in other contexts.
4.5
4.5
3.5
Final report:
Other report quality criteria
J. Structure and clarity of the report: Does the report structure follow EO
guidelines? Are all requested Annexes included?
K. Evaluation methods and information sources: Are evaluation methods
and information sources clearly described? Are data collection methods, the
triangulation / verification approach, details of stakeholder consultations
provided? Are the limitations of evaluation methods and information
sources described?
Draft report: Yes, the consultant
has followed the guidelines
provided in the TOR and by the
Evaluation Manager
Final report:
Draft report: The information
sources are clearly described. The
consultant consulted widely. She
uses triangulation to support
several arguments and report is
sufficiently evidence-based. It
does however present a more
‘what has happened’ take on the
project’s performance and less of
the evaluative aspect of why
things happened the way they did.
Generally it is a detailed and
informative report and shows a
greater effort was made over and
above what was expected of the
consultant considering the limited
6
5.5
resources.
Final report:
L. Quality of writing: Was the report well written?
(clear English language and grammar)
M. Report formatting: Does the report follow EO guidelines using headings,
numbered paragraphs etc.
Draft report: The report is well
written, comprehensible, and
logical. There are some typos in
the draft which are easily
corrected.
Final report:
Draft report: the report is well
formatted and easy to navigate.
Cross referencing is used and
corresponds correctly to the
appropriate sections referred to.
5
6
Final report:
OVERALL REPORT QUALITY RATING
5.15
0.00
S
A number rating between 1 and 6 is used for each criterion:
Highly Satisfactory = 6, Satisfactory = 5, Moderately Satisfactory = 4, Moderately Unsatisfactory = 3, Unsatisfactory = 2, Highly
Unsatisfactory = 1.