1
INTRODUCTION TO MONITORING
AND EVALUATION SYSTEMS
POVERTY REDUCTION AND EQUITY
GACEVEDO@WORLDBANK.ORG
Why M&E Systems?
2
1.
2.
3.
4.
Improve the effectiveness of public policy
Improve the allocation of public expenditure
for higher welfare
Increase accountability
Strengthen design, implementation, and results
of programs and policies
Outline of the Presentation
3
1.
What is M&E?
2.
How M&E influences Policy?
3.
Country M&E Systems
4
1.- What is M&E
Monitoring
5
Provides regular information on performance
 A continuing function
1) Indication of extent of progress towards
goals
- poverty rates
- coverage of basic services
2) systematic data collection
3) usually conducted in-house

What to Monitor?
6
1.
2.
Identify few indicators that can be measured and
are of interest to policy makers
Prioritize input, output, outcome and impact
indicators for monitoring
Inputs
3.
Outputs
Outcomes
Develop a data collection system to sustain
monitoring
Impact
How to Monitor?
7
•
•
•
Use different types of information: surveys and censuses,
administrative data, and participatory exercises
Draw on logical framework and arrangements for results
monitoring which identifies:
-Data sources for each indicator
-Frequency of measurements
-The organization responsible for collecting information.
Different types of Monitoring:
-Annual progress report
-Database for continuous monitoring
-Desk reviews
Evaluation
8

Evaluation helps to analyze deviations from targets
and goals



( e.g Why learning achievement increased? Why
poverty decreased?)
Usually conducted by independent evaluators
A systematic and objective measurement of the
results achieved by a project/program/policy
What to Evaluate?
9


Evaluation can be time and resource intensive
Plan ahead evaluation and identify questions


Strategy, operation, and learning
Reserve impact evaluation in cases such as
Strategic significance of a policy or program
 Contributing to close knowledge gap
 Innovative nature of the policy or program

Trade Offs in Evaluation
Experimental
with random
assignment
High
Quasiexperimental
with statistical
controls
Strength of
causal inference
Participant
judgment and
expert opinion
Low
Beforeand-after
Low
High
Complexity and sometimes cost
Summing up…
11
Monitoring assesses progress in
implementation of ongoing programs
 Evaluation provides a snapshot against some
benchmarks
 Monitoring looks at progress relative to
targets and assumes there is causality
 Evaluation seeks to establish and prove
causality

12
2.
How M&E influences Policy?
M&E and the Policy Cycle
13
M&E needs to accompany the policy cycle so decisions
can be based on evidence
 Define
what you want  set goals and targets
what might work to reach targets  design
programs and evaluate ex-ante
 Analyze
 set performance indicators, a monitoring
system , and implement evaluations
 Implement
But we are far from there…
14

4 years ago, more than half of programs in Mexico
had no logical frameworks


Countries seldom set targets using scientific methods:



No explicit objectives, no definition of target population
Reduce violent crimes from x to y in 5 yrs… is that a
meager performance or a great performance?
Countries rarely define the outcome (say, nutrition), and
then decide in which sector the correct intervention is
Countries seldom analyze ex ante who the winners and
losers of a policy are
The Results Chain
Definition of
Objectives
and Targets
Policy and
Program
Design
Inputs
Activities
Outputs
Outcomes
Impacts
Results
Implementation
Amount of
services
provided
Health
literacy
Consumption,
Life
expectancy,
poverty
The Results Chain
Planning
Stage
Definition of
Objectives
and Targets
Policy and
Program
Design
Inputs
Activities
Outputs
Implementation
•What are the extent and causes of
poverty in a given country?
•How is a proposed policy expected to
affect the poor?
•How to set strategic targets for a
given policy/program?
Outcomes
Results
Impacts
The Results Chain (cont)
Monitoring Stage
Definition of
Objectives
and Targets
Policy and
Program
Design
Inputs
Activities
Outputs
Implementation
•What is a given policy/program
trying to achieve?
•What are the causal links through
which the project is supposed to
achieve its goals?
•Is the project achieving its objectives?
How to measure outputs and outcomes?
Outcomes
Results
Impacts
The Results Chain
•Is the execution of the project activities conducive to the expected
results?
•Are the chosen indicators good indicators?
•What is the impact of the intervention?
•Are the changes in well-being indeed attributable to the intervention?
Definition of
Objectives
and Targets
Policy and
Program
Design
Inputs
Activities
Outputs
Implementation
Evaluation Stage
Outcomes
Results
Impacts
Several methods to support evidence policy
19
Performance indicators
The logical framework (logframe) approach
Theory-based evaluation
Formal surveys
Rapid appraisal methods
Participatory methods
Public expenditure tracking surveys
Impact evaluation
Cost-benefit and cost-effectiveness analysis
Example: Consistency and Results

Consistency and Results covers:
 Design,
Strategic Planning, Target Population and
Coverage, Operation, Perception of Beneficiaries and
Final Results.


Instrument applied by external institutions under terms
of reference prepared by a Central Unit
The evaluation includes a Summary

Strengths, Challenges and Recommendations for every
program
Consistency and Results: Questions



DESIGN
 Does de program identifies clearly the problem it’s trying to solve?
 Is it clear that the products or services produced by the program logically
contribute to reduce the problem the program is tacking?
STRATEGIC PLANNING
 Does the program have updated strategic plans for the short, middle and
long-term?
 In the S-plan, does the final results are clearly established?
TARGET POPULATION AND COVERAGE
 Does the program have a method to identify and quantify the potential and
target population?
 Does the program have a coverage strategy for the short, middle and longterm?
Consistency and Results Questions (cont)

OPERATION
 Are
there standard and adequate procedures for the
selection of projects, beneficiaries?
 Are there documents showing that the procedures are done
according to the rules of operation.

PERCEPTION OF BENEFICIARIES
 Does
the program have instruments to measure the
satisfaction of the beneficiaries with the program?

FINAL RESULTS
 Does
the program systematically collect objective
information about its Goal (Fin y Propósito)?
 Does the program have rigorous impact evaluations?
Findings CONEVAL-Mexico
Topic
Findings
have a Purpose and Goal that relate to the problem’s
67% solution.
DESIGN
of the programs defined necessary and enough outputs
50% to achieve the Purpose.
of the plans clearly establish the results they want to
21% accomplish.
STRATEGIC PLANNING
of the programs have strategic plans in the short, middle
23% and long-term.
of the programs quantify the target and potential
TARGET POPULATION
AND COVERAGE
24% population.
of the programs have a coverage strategy for the short,
23% middle and long-term.
Findings CONEVAL-Mexico
Topic
Findings
have standardized and adequate procedures for the
72% project/ beneficiaries selection.
OPERATION
of the programs have efficacy indicators in their
84% operation.
BENEFICIARIES
PERCEPTION
of the programs have instruments to measure the
50% satisfaction level of the target population.
of the programs collect objective information about
34% their final results.
FINAL RESULTS
have external evaluations that allow to measure their
26% impact.
25
3.
Country M&E Systems
Country M&E Systems: Keys for success


Systems which permit the generation of information
(basic data, indicators, evaluations, etc), reliable,
credible and high quality
Improve the supply
A high level of utilization of the information
generated by the M&E system
Incentives from the demand side

Sustainability for the future
The demand side: specific uses of an M&E system
(or..why it’s needed?)

To support government planning at the national, sub national,
local and sectoral level in the establishment of targets/goals
and permanently ask the question why are targets not being
met?  focus on results

To support management public programs through monitor and
evaluate performance and effectiveness

To informing the redesign and design of programs and
interventions

To inform budget allocation decisions

To facilitate accountability
Contradictoy?
Countries choose one or all of these roles for M&E systems
Increasing effective demand and use of M&E

Pressure from civil society for budget accountability and vigilance of
state´s performance

Need to maintain macro - equilibrium and simultaneous pressures to
extend coverage of public services  efficiency




Produce information (data , indicators, evaluations) which can
effectively be used. Worry about the trust on the quality of
information
Incentives for sectoral and budget officials, congress, etc. so that
they use the information
Leaders who persuade and motivate the government to use the
information generated  a champion (person or institution) of
results-orientation
All this is usually more important than legislation
Building blocks on the supply side
1.
Improve Statistics:

household surveys, sectoral surveys and censuses

Better administrative records and sectoral statistics

Integration of systems to generate input and outcome indicators

Regional and local level data
to feed the production of M&E
 Sectoral/program
monitoring systems
 Government
–wide monitoring systems (Systems of
presidential goals, PRS monitoring).
 Performance
 Ex-ante
evaluation systems for public services
and Ex-post Evaluations (Process
evaluations , Rapid Assessments, Impact E.)
Ensuring sustainability


Permanently redefine the system. M&E tools have to be effectively used to improve
performance and effectiveness of the state at all levels:
 Are we spending right?
 Are programs being well implemented?
 Are programs attaining its targets?
 Are we spending on the right things?
 Are programs being effective
Establishing strategic alliances: M&E a permanent function of different entities that
should work together
 Finance ministries
 Planning agencies,
 Supreme Audit Institutions
 National Statistical Offices
 Evaluation offices ( embedded in an high leverage institutions)
 Sector ministries and its statistical offices
 Local governments
32

Thanks you