New frontiers
Evaluation methods
Theory of change
Project cycle and risk management
Jesper Johnsøn, CMI, U4
Bergen, February 4, 2014
Frontiers…
Current plateau – (cross) country-level, bribery, fraud
• Evaluations of specific reforms
• Integration with risk management
• Proxy indicators
Imbalances and the “missing middle”
• Imbalances:
– So far, resources have been invested in standardised
diagnostic and advocacy tools, not learning and evaluation
• Diagnostics: follow trends – indicators, data, monitoring
• Evaluation: what works – question, design and methods,
indicators, data, judgement
• Missing middle
Level of
results
Output
Outcome
Impact
Indicator
targets
Individual
Sector
Country
When and how to evaluate corruption
costs/anti-corruption effects?
Diagnose corruption risk (does corruption matter,
are patterns changing?) If yes, prepare for
evaluation (how and how much does corruption
matter, how effective is anti-corruption?)
1. Logic model: Think through how the initiative is
intended to create results by contributing to
behavioral, organisational, political, or societal
change, expressed in a result chain or theory of
change.
2. Monitor and evaluate: Work on increasing the
evaluability of the reform by shaping design,
establishing indicators, systematic data
collection, baselines, comparison groups
PROBLEM/
DIAGNOSIS
EVALUATION
QUESTION, DESIGN,
METHODS
INDICATORS
DATA
ANALYSIS,
COMPARISON,
JUDGEMENT,
CHANGE
Controls
Randomisation
Comparisons
Baselines
Trends
objectives
Overview of types of evaluationsClear
Indicators
Impact evaluations
Focus: cause-and-effect,
attribution, impact
Programme evaluations
Focus: effectiveness,
contribution, outcomes
Mixed methods
Triangulation
Methods: counterfactual, Methods:
comparative
comparison groups,
statistical analysis,
randomisation
analysis, costeffectiveness/benefit
analysis, benchmarking
Can answer question:
What would have
happened in the absence
of the programme?
Can answer question:
Has the programme
been effective in
achieving its stated
objectives?
Process evaluations
Focus: efficiency,
implementation, internal
mechanisms, outputs
Methods: workflow
analysis, compare
performed vs planned
activities and outputs.
Can answer question:
Is implementation done
according to plan, and do
results look attainable?
Code of Conduct example, level of results
Code of Conduct example, building an impact story
Diagnostics, evaluation, risk management and
the project cycle
• Framework for holistic use of different tools
• Shows interlinkages between diagnostics, risk management and
evaluation tools
Diagnostics, evaluation, risk management and
the project cycle
• http://prezi.com/osddirkyzzpw/?utm_campaign=sha
re&utm_medium=copy&rc=ex0share
The Proxy Challenge
Challenges we confront
• Linked, but not inseparable:
– Measuring changes in corruption levels
– Evaluating whether anti-corruption efforts are successful
• Progress on measuring bribery and financial fraud, but
two overall indicator problems remaining:
– How to measure other types of corruption (patronage, conflict
of interest, abuse of power, etc.)
– How to present a measure of overall corruption levels in a
country, region, sector, or organisation not biased towards
measurable types of corruption, and can show trajectories of
change
Why proxies?
• Good, existing “direct” indicators for bribery and
“Proxy indicators are alternatives to direct
financial fraud (need better application of methods)
indicators that more directly measure the
• Can capture “missing middle” – provide needed
phenomenon under study but that may be hard
innovation for some types of corruption, to capture
to
operationalize
or require
overly costly
data
trajectory
of performance,
and measure
behavioural
collection”
change
Criteria
Standard criteria for indicator development (SMART)
• Reflect de facto changes and behavioral changes
• Prioritize sensitivity to context over standardization
• Measure specific types of corruption
• Ensure that changes in corruption trends are
attributable to reforms
“To assemble a body of promising ideas…”
• Clearly define the proxy indicator (or indicators) you suggest as a
good measure of anti-corruption results
• Identify the type of corruption measured by the proxy indicator
• Clarify how the proxy indicator reflects changes in corrupt
behaviours
• Present ideas for how the validity of the proxy indicator can be
tested
• Describe any relevant geographic, sectoral, or institutional context
• Explain how the indicator can be combined with other proxy and
nonproxy (direct) indicators to obtain a better measurement of
overall anti-corruption progress
• Comment on the strengths and weaknesses of the proxy indicator
and the feasibility of its use by aid agencies, governments, civil
society, and others for the purposes of monitoring and reporting