RESULTS-BASED M&E AND
IMPACT EVALUATION CONCEPTS
by
Suganya Hutaserani
Lead Evaluation Specialist
Independent Evaluation Department, ADB
Presentation at SHIPDET Special Topic Courses
14-18 November 2011
1
Contents of Presentation
• Links between Results-Based Monitoring & Evaluation
(M&E) and Performance-Based Planning & Budgeting
(PBB), in relation to Managing for Dev. Results (MfDR)
• How to Carry Out Project “Evaluation”?: Project
Evaluation Criteria and Rating System
• What is Meant by “Impact” in “Project Design &
Monitoring Framework (DMF)/Logframe”?
• What is Meant by “Impact” in “Rigorous Impact
Evaluation (RIE or IE)” and How to Include It in Project
Evaluation Criteria?
• Why is RIE or IE Needed?
2
Links between Results-Based M&E and PBB
• Performance-based planning and budgeting (PBB) means
that the entire planning and budgeting framework of a country is
policy-oriented and results-based (with performance targets),
linking “resources” to “results.”
• In countries where M&E systems are well developed (resultsbased) and integrated into their PBB, the whole process is then
considered a driver of success for socioeconomic progresses.
• This is because performance targets (e.g., outcome and impact
indicators) at various phases in the planning cycle could be
monitored and evaluated, with evaluation findings and
lessons fed into the PPB process to re-allocate resources to
facilitate the achievement of performance targets in subsequent
phases.
• Thus, results-based M&E and PBB are important tools
in
3
support of the country’s MfDR
How to Carry Out Project Evaluation?:
Project Evaluation Criteria
Core Criteria:
• Relevance (20%) = consistencies of project objectives with the
country needs, gov.’t plans, ADB strategies; and design quality
(e.g., quality of DMF, M&E design, technical aspects)
• Efficiency (30%) = project portfolio/implementation
performance, resource utilization (including EIRR)
• Effectiveness (30%) = project achievement of outcomes
• Sustainability (20%) = prospects for sustaining project
outputs/outcomes or “risks” to sustaining outcomes
Non-Core Criteria:
• Impacts = country’s achievement of dev. results (incl. MDGs
related to the project and other socioeconomic effects)
• Borrower/EA Performance = gov.’t ownership, compliance
with loan covenants, and provision of counterpart funds/staff
• ADB Performance = role in aid coordination, extent 4of
supervisions, other value addition (e.g., catalytic role)
Project Evaluation Criteria and Weights
5
Project Evaluation Rating System
Evaluation Scoring
Criteria
Relevance
Efficiency
Effectiveness
Sustainability
HS
S
PS
US
3
3
3
3
2
2
2
2
1
1
1
1
0
0
0
0
How to Apply the 4 Criteria to Rate a Project
Criteria
Relevance
Efficiency
Effectiveness
Sustainability
Overall
%
20
30
30
20
100
Weight Scores
0.2
0.3
0.3
0.2
1.0
3
2
2
2
WAS
0.6
0.6
0.6
0.4
2.2
Overall rating is determined based on the value of WAS:
2.7 =< HS <= 3.0
1.6 =< S < 2.7
0.8 =< PS < 1.6
0.0 =< US < 0.8
Sample DMF/Logframe of a Project in the
Education Sector
Design Summary/
Statements
Impact (Goal)
- Improved overall
access and quality of
primary education
Performance Targets/Indicators
Data
Sources/Reporting
Mechanisms
Assumptions
………………...
……………......
Risks
………………...
………………...
- increased net enrollment rate (NER) from ………………………
90% to 95% (during 2005-2011)
……………………...
- increased literacy rate from 95% to 98%
(during 2005-2011)
Outcomes (Purposes)
- Improved enrollment
in primary schools and
student performance
- increased enrollment of primary school
age children in project areas from 80% to
99% (during 2005-2011)
- increased primary school completion rate
in project areas from 75% to 95% (during
2005-2011)
Outputs
- Better school facilities
and revised curriculum
-
1,000 primary schools built/rehabilitated
curriculum and textbooks revised
6,000 sets of textbooks distributed
10,000 primary school teachers trained
Assumptions
and Risks
7
What is Meant by “Impact” in “Project Design
& Monitoring Framework (DMF)/Logframe”?
Outputs = amount of work, goods and services produced by a project
in a period (e.g., # of schools built, # of teachers trained,
new course materials produced; length of roads upgraded)
Outcomes = results of outputs of the project (e.g., increased
student enrollment and completion rates in schools in
project areas; reduced travel time and cost in project
areas; increased household income in project areas),
compared with those “before” the project
Impact = higher-level outcomes “contributed to” by the project
(e.g. increased employment rate, NER and literacy rate, and
other MDGs), compared with those “before” the project
8
What Is Meant by “Impact” in “Rigorous Impact Evaluation
(RIE or IE)” and How to Include It in Evaluation Criteria”?
Outputs = amount of work, goods and services produced by a project
in a period (e.g., # of schools built, # of teachers trained,
new course materials produced; length of roads upgraded)
Outcomes = results of outputs of the project (e.g., increased student
enrollment and completion rates in schools in project
areas; reduced travel time and cost in project areas;
increased household income in project areas), compared
with those “before” the project
Impact = the “outcomes” of the project compared to what they’d
have been in the absence of the project (i.e., comparison of
outcome data b/w “project areas” and “control areas” or
b/w “with” and “without” project). This means that
“outcome” (rather than “impact”) indicators from DMF are
normally used in doing RIE or IE, and the “impact”
9 estimated
by IE will be assessed under the “effectiveness” criterion.
What Is Meant by “Impact” in
“Rigorous Impact Evaluation (RIE or IE)”?
Household monthly
income ($)
200
With project
180
Project Impact/Effect
160
C
140
120
Without project
A
B
100
80
60
40
20
0
2000
2004
2008
2012
2016
10
2020
Year
Why is Rigorous Impact Evaluation (RIE or IE) Needed?
• Generally, when evaluating “effectiveness” of a project, we
compare the outcome data b/w “before” and “after” project
(e.g., increased HH income from $2/day in 2006 to $8/day in 2011).
• However, such an increase might not be due solely to our project,
but also influenced by assistance of other donors/projects.
• To find out the extent to which such an increase is due to our
project, (“attribution”) as an evidence supporting value for money
of our investment, we need to use RIE or IE method by comparing
outcome data b/w “project/treatment area” and
“counterfactual/control area,” and use rigorous techniques (e.g.
randomized control trials, or quasi-experimental technique) to
arrive at an unbiased estimate of the “impact.”
11
THANK YOU
WWW.ADB.ORG/EVALUATION
12