BRACED Interim Knowledge Management Webinar
Interim Monitoring & Evaluation
Guidance for the BRACED
Programme II
16 May 2014
Purpose of the second webinar
• To address key questions from Grantees submitted after the first webinar
• To update Grantees on status of emerging guidance
• To explore key methodological issues in more detail
• To present draft ‘Bronze, Silver, Gold’ standards for key aspects of projectlevel M&E (‘M&E Standards’)
Content of webinar
1.
Update on emerging guidance
2.
Measuring KPI1 – numbers supported
3.
Collecting and using climate and related data
4.
Construction and using resilience indicators (KPI4)
5.
Attributing changes in resilience to project activities (KPI4)
6.
Use of control groups in attribution
7.
Report from Concern on Crisis Resilience Indexing System
1. Update on emerging guidance
4
Guidance available in May 2014
1.
Methodological guidance for all 15 KPIs (note KPI4 being revised below)
–
2.
Additional guidance on definition of direct beneficiaries
–
3.
Next iteration to be published on website shortly
Programme monitoring plan
–
7.
Final version to be on BRACED website before end of May
Revised log-frame
–
6.
In development – available on BRACED website by end of May
Measuring Resilience report
–
5.
BRACED website – complements guidance to KPI1 (1 above)
Revised KPI4 guidance
–
4.
Full set to be shared with grantees ASAP
In development – to be circulated by end of May
Value for Money for Adaptation framework
–
Flyer will be on BRACED website next week, full version following week (end of May)
2. Measuring KPI 1 – numbers supported
Martin Whiteside
6
KPI 1 - Number of people supported by DFID
programmes to cope with the effects of climate change
Q – Is the “number of people supported” in the BRACED logframe (KPI 1) the
same as our direct or indirect beneficiaries?
• Measuring an reporting KPI 1 is mandatory for all projects;
– Need consistency
– Check definitions in KPI 1 guidance notes
• Different projects/organisations have different definitions of ‘Direct’ and
‘Indirect’ beneficiaries – for KPI 1 reporting need to use KPI 1 definitions
that are specific to BRACED (see the BRACED website);
• Report number of individuals disaggregated by gender (can be calculated
from HH numbers as long as clear whole family are ‘supported;
• Counted and reported annually
DEFINITIONS
KPI 1 - Number of people supported by DFID programmes to
cope with the effects of climate change
‘Support’ - direct assistance from the programme in question, with
the explicit intention of helping people deal with climate change
impacts. It could include for example financial resources, assets,
agricultural inputs, training, communications (e.g. early warning
systems) or information (e.g. weather forecasting).
‘People supported’ - identified by the programme in question with a
direct relationship to it.
‘Effects of climate change’ - existing and future, sudden or gradual,
arising from primary consequences of CC (changes to precipitation,
temperature and sea level rise) - can include floods, droughts, storms,
landslides, salination, coastal inundation, heat or cold waves and
biodiversity loss.
Categories for reporting
KPI 1 - Number of people supported by DFID programmes to cope
with the effects of climate change
BRACED DIRECT
A – High intensity
Both targeted and high intensity e.g. people receiving social protection cash transfers, houses
raised on plinths, agricultural extension services, training of individuals in communities to
develop emergency plans and use early warning systems.
B – Medium Intensity
i)
Targeted & Medium intensity: e.g. people receiving weather information and text
message early warnings.
ii)
Not targeted & Medium intensity: e.g. people within the coverage of an early warning
system, or catchment area of a large infrastructure project (e.g. flood defences), or living in
a discrete community in which others have been trained in emergency response
BRACED INDIRECT
This does not contribute to the KPI 1 headline figure, but can be reported separately
C – Low intensity: e.g. people falling within an administrative area of an institution receiving
capacity building support, or catchment area of a Water Resources Management plan (can be
captured through the programme’s own monitoring, or ‘institutional development’ scorecard).
Question & Answers
3. Collecting & using climate data
Martin Whiteside
11
Collecting Shock and Stress (S&S) Data
Q - How should we collect climate shocks/trend data, and
how technical and detailed should it be?
Key concepts:
• Concentrate on S&S being addressed by your project
• Base focus on S&S as experienced by your target
beneficiaries;
• Triangulate against S&S relevant ‘scientific’ data
S&S Data - STEPS
Planning Phase
1.
2.
Define the S&Ss your project is building resilience against;
Participatory enquiry for each S&S:
–
–
–
3.
Beneficiary prioritisation of different S&S (understand differences)
Severity category for main S&Ss – v. severe; severe, normal, good?
Ground truth against known years/events in past
Historic agri/met/hydro/well-being ‘scientific’ data
– e.g. crop prod., rainfall, flood levels, deaths from disaster
– Reliability, specificity, relevance
4.
Triangulation
– Relevance to community experience of S&S (e.g. days without rain in growing
season may be more important than monthly rainfall)
– Data gaps – can we fill in implementation?
– Correlation between community perception and ‘scientific’ data?
5.
Trend
–
–
What trends can be discerned? (Across perception and measured?)
What are implications?
S&S monitoring - Draft Standards
Bronze
Definition of S&S identified and
shocks and
described by programme
stresses
staff or key informants
(S&S)
‘Scientific’
data
collection
Community
shock and
stress
perception
monitoring
Triangulatio
n
Trend data
Learning
Silver
Gold
Limited secondary data
(e.g. mm rain/year)
and/or not very
congruent with project
area or sub-areas and/or
with doubts on reliability
Limited ranking of very
severe, severe, normal
and good for year.
Staff/key informant
identification of S&S
supplemented by beneficiary
information on priority and
characterisation
Data relevant to some of the key
shocks/stresses, reasonably
congruent with project area and
with fair reliability. Limited
supplementary primary data
collection.
Ranking disaggregated by key
S&S and beneficiary type and or
geographical area.
Each Shock or Stress prioritised and
defined through participatory
approaches across range of beneficiary
types and contexts, ground truthed
against specific events and years.
Relevant, reliable, geographically specific
data covering most of key S&S,
disaggregated if necessary across key
zones of project area. Supplementary
primary data collection organized as
needed.
Ranking disaggregated by S&S and
beneficiary type and or geographical
area. Clear participant explanation for
each ranking and comparison with
previous similar years/events.
Significant correlation.
Little possible
Some correlation identified.
Weak - scientific data of
limited reliability,
specificity and relevance
supplemented by
anecdotal views;
Little
Moderate – reasonably reliable,
specific and relevant scientific
data correlated in some cases
with community perceptions
Strong - reliable, specific and relevant
scientific data correlated cases with
community perceptions
Significant
Very significant
S&S Data - STEPS
Implementation Phase
6. Beneficiary S&S perception monitoring – participatory, clear criteria (can
include questions on how project outputs have affected these)
7. Agri/met/hydro/well-being ‘scientific’ data - collect
–
–
8.
Relevant to S&S
May need to supplement/re-format/analyse/interpret
Triangulation – annual/event
Use of information - learning
• Beneficiary S&S priorities and experiences during project,
• Contextualising project outputs and outcomes – did these address felt S&Ss?
• Deepening understanding of relationship between ‘scientifically measured’ and
experienced S&Ss
• Enabling explanatory correlation between resilience outcomes and well-being
impact
• Additional understanding of trends
• Feed into continual project improvement.
Question & Answers
4. Constructing & using resilience
indicators
Nick Brooks
17
What should resilience indicators measure?
• Factors/attributes we think will make people better able to anticipate,
avoid, plan for, cope with, recover from & adapt to stresses & shocks
• These will be context-specific & should be identified during development or
early in implementation phase through, e.g. participatory assessments
• Factors/attributes to be measured are those that will/may be influenced
directly or indirectly by the project (positively or negative – potential for
unintended consequences)
• Other factors/attributes identified as important for resilience but that are
not related to project activities should be noted – these might be tracked to
provide context, but will not contribute to calculation of numbers with
improved resilience for reporting against KPI4
Resilience outcomes & project outputs
• Resilience measured at outcome level in BRACED log-frame
• Changes in resilience usually measured as project outcomes
• Two broad types of resilience outcome indicator:
1. Indicators that measure numbers of people sustainably adopting project outputs
– essentially measures of uptake of project outputs, where there is good
evidence that this ill improve people’s resilience (uptake indicators)
2. Indicators that measure changes in factors/attributes/status that we believe
(based on evidence) will make people more resilient that are not tied to project
outputs & may be measured for non-beneficiaries (status indicators)
• Ideally, project will complement (1) with (2)
Types of indicators
• Qualitative, e.g. based on beneficiary perceptions of how easily they can
cope with particular stresses/shocks, access resources that make them more
resilient, etc.
• Quantitative – binary, e.g. value of 0 for ‘no’ & 1 for ‘yes’ in answer to
certain questions (e.g. does beneficiary have access to certain resource,
meet a certain criterion that is important for resilience?)
• Quantitative – categorical/score based, e.g. assign score based on category
of resilience as measured by a particular indicator (low, moderate, high, etc.)
– Qualitative indicators can be converted to quantitative score-based indicators
• Quantitative – continuous, e.g. based on measurement of a continuous
variable such as household income, time to recover from previous shocks.
• Projects likely to employ a mixture of the above
Composite & individual indicators
Projects might use:
1.
A number of individual indicators, each representing a different aspect of
resilience as relevant to the project
– Measure changes in each indicator separately for individuals sampled
2.
Several composite indices, each representing a different dimension of
resilience as relevant to the project:
– e.g. income & food access; safety nets, access to services, adaptive capacity, etc
– Guidance on dimensions under KPI4, but not prescriptive
3.
A single composite index, representing resilience as relevant to the project
– Judge whether such aggregation appropriate based on nature of indicators
– Project might measure only one dimension of resilience (cf (2))
Construction of composite indicators
How to aggregate different types of indicator?
• Need to convert to common format, e.g. using:
– Discreet scores:

Split range of continuous variables into divisions, with each division
represented by a score, e.g. 1-3, 1-5;

Define categorical indicators using same scoring system

Add binary indicators (e.g. 3 or 5 indicators for same range as above)
– Scaled values: e.g. from 0-1
– Add values together or take average across values
• Need to consider weights
– Assign to constituent indicators based on relative importance
– Usually done subjectively, not straightforward
• These issues are less pressing (although not necessarily irrelevant) for
individual, disaggregated indicators
Numbers with improved resilience
For panel data/ longitudinal studies (sampling same individuals)
• Single composite index
– No. showing increase in index value minus no. showing decreased resilience
• Multiple composite indices
– No. showing improvement in 1 or more indices (and no decreases) minus
no. showing decreased resilience in 1 or more indices (and no increases)
• Multiple disaggregated indices
a)
For ‘improved resilience’, require a minimum number (X) of indicators to
show improvement, and a maximum number (Y) to show deterioration.

X and Y to be set according to context, with X > Y

Might need to demonstrate improvement in a set of core related indicators
b) Vice versa for ‘decreased resilience’
c)
•
Number with improved resilience is number fulfilling (a) minus (b)
Scale up from sample to beneficiary population, ensuring sufficient sample
size and considering disaggregation (take statistical advice)
Numbers with improved resilience
Where sampling does not involve the same individuals each time
• Define a threshold for improved resilience, based on e.g.
– Movement out of ‘low resilience’ category, to moderate or high
 For disaggregated indicators need to decide how categories constructed
– Minimum value of a composite index
• Sample at different periods
– Baseline data gathering period t0
– Subsequent sampling periods t1 , t2 … tn-1 , tn
• Number with improved resilience is:
– No. in lowest resilience category at tn-1 minus number in this category at tn
– No. above resilience threshold at tn minus no. above threshold at tn-1
• Sampling methodologies (e.g. panel survey or samples using different
beneficiaries) will have implications for how indicators are constructed
Resilience indicators – Draft Standards
Bronze
Silver
Gold
Counting
method
(KPI4)
No. with improved resilience,
based on no. adopting project
outputs, no. moving out of
lower resilience category, or no.
crossing res. threshold
Type of
indicator
Indicators measuring uptake of
project outputs supported by
evidence that this improves
resilience (can include indirect
beneficiaries where uptake
beyond direct beneficiaries, e.g.
emulation)
As Bronze, plus additional data on
changes in numbers in multiple
categories (e.g. low, med., high
resilience) – panel data or other
sampling
As Bronze plus one or more of:
• beneficiary perception-based
indicators - better/worse than
before; low, med., high for
different aspects of res.
• More ‘objective’ criteria for
different levels of resilience or
position with respect to
resilience threshold
Timing
Baseline and end
During implementation; 1 expost measurement;
During implementation; ≥1
ex-post measurements
Evidence Qualitative evidence based on
beneficiary feedback & good
base for
indicators theory of change
As Bronze plus case study
evidence from past stresses and
shocks
As Silver plus correlation of
indicators with losses for past
stresses & shocks
DisaggregGender
ation
Gender + other pre-determined
classes
Gender + other predetermined classes
As Bronze, plus additional data
on how much resilience has
improved based on sampling
of same individuals
Mix of uptake, perceptionbased & objective indicators,
with objective indicators
based on more detailed
categories and/or continuous
variables. Assessment of
consistency across different
indicators
Question & Answers
5. Attributing changes in resilience to
project activities
Nick Brooks
27
Attribution vs. Contribution
Grantees (& DFID) want to determine whether (and to what extent) the intervention
is making a difference
• Attribution: Determining the degree to which the projects are ‘causing’ the
observed resilience outcomes (and measured well-being impacts);
• Contribution: Determining the degree to which the projects are ‘contributing
to or helping to achieve’ the observed resilience outcomes (and impacts);
• In practice this distinction not critical – projects will ask:
– Are resilience & well-being (as measured by indicators) changing?
– To what extent is the project responsible for these measured changes?
– To what extent can we say project contributed to or caused these changes?
– If evidence suggests changes would not have happened without the project, we can
attribute changes to project; if evidence suggests project played a role but was not
sole driver, we can say project contributed to changes
– In practice there will be a continuum from no contribution to full attribution
How much of improved resilience is due to project?
No. with improved resilience  no. with improved resilience due to project support
Assess using
• Beneficiary feedback (qualitative explanatory enquiry)
– Build questions on where/extent to which project helped build resilience into
surveys used to gather data for resilience indicators (panel & other survey)
– Narratives from beneficiaries (panel survey: compare narratives over time)
• Comparisons of resilience (as represented using indicators)
– Before & after intervention
– Between groups at different stages of a phased intervention
– Between beneficiaries and comparison/control groups
– Comparison/control groups must have similar characteristics (e.g. livelihoods,
economic/policy contexts) and be exposed to similar/same hazards
Characterising contribution
Qualitative description of contribution
• If X people have improved resilience, and there is good evidence* that project
played a role, can say project contributed to improved resilience of X people
*
e.g. demonstrable differences between project area and other similar areas coupled with
feedback from key informants indicating project played significant role in these differences;
resilience outcomes match theory of change and no other plausible explanations; etc.
– How much did project contribute, and how did this vary across different groups?
Quantification of contribution
• Beneficiary perceptions from surveys – what proportion (X) of sample
representing beneficiary population N, say project contributed?
– Number with improved resilience due to project = X*N
– Can also ask in what ways project contributed, and by how much (a little, a lot, etc.)
• “Difference in difference”1 - what proportion of people are more resilient in
sample of beneficiary population (X) than in sample of control population (Y)?
– Number with improved resilience due to project = (X-Y)*N
• Need large, representative samples – seek statistical advice
1As
described by Khan and Anderson (2014): http://www.iied.org/climate-adaptation-good-development-two-sides-same-coin
Attribution/contribution – Draft Standards
Bronze
Qualitative
explanatory
enquiry
Project level estimate of
contribution of project based
on combination of qualitative
and quantitative reasoning.
Counterfactual
Before/after intervention.
Assessment of
contribution
Project ‘contributed to’
improved resilience of X
number of people.
Silver
As Bronze, but informed by
questions related to role of
project built into surveys of
beneficiaries. Include those
‘excluded’ from benefit.
Use of phased intervention
approach to compare
beneficiaries at different stages
of intervention through panel
survey.
Qualitative description of
extent to which project
contributed, e.g. one of several
factors, significant or major
(but not sole) factor, resilience
would not have been improved
without project; describe for
different groups of
beneficiaries.
Gold
As Silver, but also informed by
beneficiary narratives derived
from panel studies. Include
those ‘excluded’ from benefit.
Use of control/comparison
groups exposed to
similar/same stresses &
shocks.
Quantitative characterisation
that indicates the % of the total
numbers with improved
resilience that can be
attributed to the project
and/or the degree of change
that can be attributed to the
project.
31
Question & Answers
6. Use of control groups in attribution or
contribution
Nidhi Mittal
33
Role of Control Groups
• Role of control groups in measuring attribution or contribution
• The evaluation framework, including data sources/methods depends on
whether the focus is on attribution or contribution.
• Attribution analysis will require more experimental type evaluations:
• Use of experiment (and control groups) tries to isolate one factor—the
receipt of an intervention or a project , everything else being constant.
• Individual households or communities are randomly assigned to the
control group (randomised control trials).
• Quasi‐experiments — similar groups of beneficiaries, or communities
are used to create comparison groups ( no randomisation involved)
• Contribution analysis can be more theory-based and supported by a broader
range of methods and sources of data to develop a consistent narrative.
• Quasi experiments, case studies, correlation studies, longitudinal studies,
and sample surveys
34
Effective use of Control Groups
• Beyond KPI4: using control groups to assess resilience impacts
•
•
Use of control groups can be extended beyond measuring KPI 4.
The approaches for assessing improved resilience outcomes are broadly
applicable to assessing resilience impact indicators (e.g. well-being
indicators , loss and damage).
• Stakeholder discussions:
•
•
Engaging with stakeholders to ensure that the methods used are feasible,
realistic, and responsive to needs, and appropriate to the intervention.
Seeking expert advice to identify the types of questions to be asked,
understanding fully the risks of leading questions.
• Ethical challenges and limitations of using control groups
•
•
•
•
There are ethical questions on how/when to use control groups
When an actor interacts with beneficiaries, they expect certain actions.
Managing expectations and helping beneficiary groups understand the value
and challenges of contribution and attribution is key.
DFID’s Ethics Principles for Research and Evaluation (2011) provides further
guidance in relation to ethical use of control groups.
35
Support Available from Interim and full KM
• Planning future KM support on control groups
•
Projects should be aiming to deliver their M&E plans as described in the
concept notes and whereby they have planned or are planning to use control
groups, they should flag support needs to the KM.
•
Although encouraged, control groups are optional as a (gold standard)
measure and not mandatory if projects did not originally plan this in their
concepts.
• Guidance from the interim KM
•
•
The KPI4 guidance from the interim KM will provide some over-arching
principles of using control groups for measuring ‘attribution’ of resilience
outcomes or the ‘contribution to resilience of beneficiaries.
Brief telephone or email consultations/review of approaches to M&E
project leads can be provided by the interim KM to the grantees.
36
Support Available from Interim and full KM
• Planning full KM support
•
The full scope of the guidance and support from the KM cannot be
finalised until the permanent KM is in place, the final projects are chosen
and resourcing needs overall are assessed.
•
It is likely to be a flexible and iterative process when the full KM pulls
together the BRACED monitoring and evaluation approach overall on the
basis of the final list of project’s M&E frameworks.
• Information needed from grantees to plan future KM support
•
Projects should indicate whether they are planning to use control groups
and which standard(bronze, silver, gold) best reflects their M&E approach.
•
This information should be submitted to the interim KM within the Bronze,
Silver, Gold grading standards template to be shared by end May.
•
The interim KM will use this scoping assessment to map projects on the
grading standards scale , assess their support needs and inform the tailored
support plan for the full KM to provide to the grantees.
37
Question & Answers
6. Concern’s Crisis Resilience Indexing
System
Aine Magee
39
Community Resilience
Indexing System (CRIS)
Aine Magee- Concern M & E Adviser
Measuring Resilience
Due to difficulties posed in measuring resilience, Concern has
been developing a Community Resilience Indexing System
(CRIS) which evaluates the assumed characteristics of a
resilient community and comes up with an overall CRIS score
per community.
The CRIS score indicates the level of achievement of each
community against specific indicators/characteristics. It is
assumed that a high score, meaning a community has a lot of
the necessary characteristics, is indicative of a resilient
community. Concern could test this assumption through
BRACED.
Objectives of CRIS
The CRIS is designed to provide a semi-qualitative baseline/ endline
comparison tool to assess the impact of work in Disaster Risk Reduction
and building resilience, and to facilitate on-going monitoring by providing a
comprehensive set of indicators that can be tracked over time.
The intention is to investigate the correlation between these assumed
determinant of resilience and actual resilience through BRACED, as well
as in other contexts.
We want to test whether the assumed characteristics of resilience
(detailed in the CRIS tool) are correlated to communities who
“bounce back better” after a shock.
This work will be carried out with input from a research partner.
CRIS indicators
The CRIS Score is calculated on the basis of proxy indicators across 6
livelihood asset classes:
• Political Assets
• Social Assets
• Human Assets
• Financial Assets
• Physical Assets
• Natural Assets
Each indicator is assigned a score between 1 and 5, where 1 is the lowest
level of attainment, and 5 is the highest. An overall weighted score per asset,
and per communities is calculated.
Indicators have been developed based on DRR/resilience literature and
Concern experiences.
Progress to date:
Two pilots using CRIS to measure characteristics of resilience have been
conducted in Zambia and Sierra Leone.
Based on these pilots:
• the CRIS indicator set is being finalised
• the methodology for administering the tool is being refined
• the possibility of developing an app form of the tool for use on laptops,
smart phones or tablets is being investigated
CRIS in BRACED
Concern hopes to use the revised CRIS tool to measure a related
outcome level indicator for BRACED
“Average Score across target communities on the Community Resilience
Indexing System”.
We would hope to see an increase in the average score throughout the
programme indicating in an increase in the resilience characteristics of
target communities.
The suggestion is that the population in communities who show an
increase in resilience characteristics over time, can be included in the
measurement of KPI 4:
“No. of people with improved resilience as a result of BRACED projects “
Question & Answers
Opportunities for further knowledge sharing
If you would like the opportunity to present your own learning regarding
M&E tools and methodologies, please let us know via your Monitoring
Officer.
We would be happy to facilitate another webinar (given sufficient
interest) to support the dissemination of your experience.
Thank you for your participation in today’s webinar.
Any further comments or questions should be sent to your Monitoring
Officer and we will respond as soon as we are able.