The impact oriented M&E-system of the
German-costarican program on waste
management
Dr. Stefanie Krapp
Evaluation Unit, German Technical Cooperation (GTZ)
Annual Conference of the Canadian Evaluation Society
Victoria, Canada
May 4th, 2010
13.04.2015
Page 1
Contents
1. Evaluation system of GTZ
2. The waste management program in Costa Rica
3. The program M&E-System
4. Evaluation design
5. Some selected results
6. Conclusions
13.04.2015
Page 2
Managing for development results
Planning a development
measure
Implementation
Completion
Sustainability
Results
Results-based monitoring
Evaluations
e.g. ex-ante
evaluation
e.g. project
progress review
e.g. final
evaluation
e.g. ex-post
evaluation
13.04.2015
Page 3
GTZ Evaluation system
Evaluation system
GTZ instruments
Self-evaluation
Steering by
operative units
Project progress
review (PPR)
e-VAL
Decentralised
evaluation
Steering by
operative units
Mostly interim
evaluation
Independent
evaluation
Steering by
Evaluation Unit
End of project
evaluation
e-VAL
Ex-post evaluation
External evaluation
Evaluation by
the Ministry of
Economic
Cooperation
and
Development
(BMZ)
Evaluation by
auditors
on behalf
of BMZ
2-5 years after completion
of project
Continuous results-based monitoring in every development measure
is the prerequisite for all evaluations.
13.04.2015
Page 4
German-costarican program
“Competition and Environment”
Goal: Improvement of waste management in municipalities and the industry
in selected regions of Costa Rica (04/2005 – 03/2013)
Component 1:
Comunication and
dialogue
Component 2:
Strategies, plans
and legal frame
Public and private
sector and the civil
society work
constructively
together in the
waste
management.
Relevant
stakeholders
accomplish the
new legal frame
conditions and
political
development
strategies in
waste
management.
Component 3:
Waste
management in
municipalities
Component 4:
Competition and
environment
friendly behavior
of the industrie
Improved abilities
of selected
municipalities to
carry out an
integrated waste
management.
Industry, services
and trade
implement
production
methods and
measures for
resource
recuperation.
13.04.2015
Page 5
German-costarican program
“Competition and Environment”
Intended Impacts
 Diffusion: The implemented structures and processes in
selected municipalities are taken over by other municipalities
and companies which lead to the improvement of waste
management in other regions and sectors.
 Positive image of Costa Rica as an ecotourism country
 Improvement of health
 Costa Rica as knowledge carrier in waste management and
innovative environmental technologies
 Growing environmental awareness
 Growing resource efficiency
 ….
13.04.2015
Page 6
M&E System
The impact oriented M&E-System of the program CYMA
Performance
Framework:
Result chains for
each component
with indicators
on each level
(output, use of
output, outcomes,
impacts)
Internal Monitoring:
Baselines &
continuous
performance selfassessment with
data collection,
time series
analysis,
household survey,
foto documentation
(output, use of
output, outcome)
Internal and
external Eval.:
a) Project Progress
Reviews
b) Specific
component related
topics
c) Anual panel
according to
outcome and
impact indicators
¡Participatively developed and implemented!
¡Linked with the program management system!
13.04.2015
Page 7
Internal Monitoring
Time series analysis
Ex. 1: Development of the quota of correctly deposited, composted or
recycled waste in the municipalities
Before
t2
t1
70%
75%
After
t3
t4
t5
78%
85%
90%
Intervention:
Implementation of waste management plans
in the municipalities
Ex. 2: Development of the quota of households connected to the waste
management system in the municipalities
13.04.2015
Page 8
Internal Monitoring C3
 Program group (P) = 8 pilot municipalities
 Counterfactual: Comparison group by matching procedure (C) =
3 municipalities not related to the program
 Baseline and continuous collection of data according to relevant
indicators of P and C
 Standardised telephone survey of municipal households about
the waste situation in the municipalities (P and C)
 Randomwalk: Photographical documentation of the „visual“
waste situation in the municipalities (P and C)
13.04.2015
Page 9
External Evaluation
Panel evaluation design
Impact
model:
Capacity of
the
counterparts,
Outcome and
Impact
Evaluation
frame:
indicators of
the impact
model
Longitudinal
study, incl.
expert
assessment,
guided
interviews
13.04.2015
Page 10
Impact Model
Impacts
Improved waste management
in municipalities and industries
in selected regions of CR.
Counterparts
Partner
structure
Goal system
Outcome
Organisational
structure
CYMA
Staff
Component 1
Component 4
Communication
Component 2
Fin. Resources
Component 3
13.04.2015
Page 11
External Evaluation: Panel
Before
After
Panel groups
 The same variables
 with the same operationalisation (guided interviews)
 are measured with the same people
 at different points of time.
Baseline
2nd survey
3rd survey
4th survey
2006
2007
2008
2010
13.04.2015
Page 12
Panel groups
(2006, 2007, 2008, 2010)
3 Ministries (Planning,
Health, Environment),
Chamber of Industries,
IFAM, 4 pilot
municipalities, program
respresentatives
Experts
Stakeholders
Municipalities, Tourism,
university professors,
Journalists
Counterparts
Gtz-Program PRODELO, 4 municipalities, chambers, ministeries (Economy,
Agriculture), NGOs, Institutes (CNP+L, CEGESTI, CICAP), Associations
(ACEPESA, ACIPLAST, FEMETROM, FEDEMUR, CONADECO), local
respresentatives (PRU-GAM, UNGL), political respresentatives, companies
(Holcim, Dos Pinos, Prolusa, Sur Química, Gente Reciclando, Fundación
Escazú Recicla)
13.04.2015
Page 13
Sumary
Monitoring data:




Baseline
Time series
Standardised telephone household survey
Foto documentation of the waste situation in municipalities
 Double-Difference Analysis:
Comparison of data collected at t3 with data collected at t1
(baseline data) and t2 of the program and comparison group
Data of the Panel survey
 Comparison of data collected at t4 with data collected at t1
(baseline), t2 and t3
 comparison with monitoring results at different points of time
Determine the attribution of program impacts
13.04.2015
Page 14
Selected results
 Law on waste management almost ratified, National Plan of
Waste Management, Municipal Waste Management Plans
 Diffusion of resource gentle and environment friendly
production in the private industry
 Increased separated disposal of waste
 Intensified interaction in the waste sector and therefore
continuing presence of the topic in the public
 Appreciation of the political role of the chamber of industries
 Costa Rica as knowledge carrier
 BUT: still critical pollution (mainly oncerning water, waste
disposal and recycling)
13.04.2015
Page 15
Conclusions
Main functions and goals of the M&E-System
 Joint learning
 Information on how the program is doing
 Steering information for the program management
 Evidence of program results (outcomes and impacts)
 Basis for the development of a national waste management
monitoring system in Costa Rica
Main conclusion:
The M&E system is not perfect but tries to show how the
program is doing and gives valid information on outcomes and
hints on impacts under the given circumstances by applying
different methods as „rigorous“ as posible!
13.04.2015
Page 16
Conclusions: Prerequisites











logical result chains and useful indicators
flexible system that allows adaptations
baselines
continuous data collection
good data basis in the municipalities
participation of counterparts in the development and
implementation of the M&E-system (ownership)
willing program manager
M&E system connected with the management system
staff responsible for the monitoring system
combination of internal and external evaluations anchored in
the monitoring system
pretension of rigorous evaluation designs to attribute impacts
(determination of a comparison group right at the beginning)
13.04.2015
Page 17
Thank you! Merci! Gracias! Danke!
Dr. Stefanie Krapp
Evaluation Unit, German Technical Cooperation (GTZ)
stefanie.krapp@gtz.de
www.gtz.de
13.04.2015
Page 18