Phase 2 Evaluation of the PD
Evaluation
Methodology
Reference Group Workshop/Meeting
11 – 13 February 2009
Methodological Issues
The broad choice of evaluation methodology, as we
discussed yesterday, follows on from the evaluation
questions chosen. However there are more specific
methods-related issues that need to be thought
about at this stage.
Methodological Issues
One way of focussing on methodology is to consider
what methodological ‘standards’ or ‘qualities’ we
expect this evaluation to have. (You will notice a
certain overlap with the previous workshop on
‘quality standards’!)
Methodological Issues
For example, it can be argued that this evaluation
should end up with the following qualities:
• A balanced & sufficient sample of countries
• Sufficient coverage of sectors and themes
• Information of good quality
• Offering the possibility of explanation and
attribution
Methodological Issues
If an evaluation has these qualities, it will allow us to
generalise to some extent across different PD
settings (‘external validity’); be confident that our
measurements or descriptions are consistent
(‘reliability’) and have confidence in the strength or
‘power’ of findings – that they are sufficiently
supported by the evidence collected and analysed
Methodological Issues
Making the right decisions about methodology at the
beginning will make an evaluation defensible, able
to withstand criticism when it eventually reports
Lets take each of the ‘attributes’ in turn
Methodological Issues
‘A balanced & sufficient sample of countries’
When policy makers ask about the PD they want to
know whether we are able to say something about
countries in different geographical regions; those
that are more and less aid dependent; and both
those that have strong institutions and governance
and others that have some elements of ‘fragility’
perhaps because they are still recovering from wars
or conflicts. We also need enough countries to be
able to support conclusions for the PD as a whole
Methodological Issues
‘Sufficient coverage of sectors and themes’
Countries selected need to cover the main policy areas
and sectors recognised as important for development
– for example healthcare, encouraging small
businesses, education, progress towards the MDGs,
international trade support. They also need to include
important themes such as: capacity development, civil
society participation, donor harmonisation, improving
governance & reducing fragility. This will allow
sensible comparisons to be made across cases
Methodological Issues
‘Information of good quality’
This requires:
• Available information – one rationale for selecting
some sectors/themes
• Willingness to use innovative data sources
• Ensuring that all pre-existing sources are ‘synthesised’,
reviewed and exploited
• Cross-checking (‘triangulating’) across multiple
sources of information
• Expending more time relatively on data collection
Methodological Issues
‘Offers the possibility of explanation & attribution’
There are two ‘classic’ ways we can attempt to explain.
First through longitudinal analyses that follow a causal
chain over time – this is the basis of time-series data
and panel studies as well as causal modelling, trackerstudies & ‘theory-based evaluations’. Second we can
compare across places, settings or time periods –
including before & after studies; comparison groups;
quasi experiments and full controlled experiments
Methodological Issues
On this basis, we can put together a possible starting list
of methods that could be used in Phase 2.
The list would include:
Methodological Issues
• Synthesis reviews of existing evaluations, research and
indicator systems
• Comparative in-depth case studies (of country
partnerships) which are chosen to contain a good
cross-section of common themes/sectors
• Longitudinal studies – either forward looking (‘theorybased’ mapping of plausible directions of travel) or
backward looking tracking back to PD-like, longer
established policies
• Targeted comparative studies to ‘supplement’ country
based case comparisons
Methodological Issues
Methodologies have to be understood as more than
analytic tools. How they are resourced and ‘steered’
will determine their value as much as their technical
sophistication. To take two examples:
Methodological Issues
Putting together the best team of experts to undertake
the evaluation at country & central levels will be
challenging. It may require bringing together public
sector and civil society expertise; national and possibly
regional resources & skills.
Methodological Issues
National ‘reference groups’ will need to open up access
for information and cooperation institutionally and
across government; to safeguard the independence
and credibility of the evaluation; & build bridges so as
to make it more likely that evaluation outputs will be
used and useful
Methodological Issues
These are some of the issues that needs more discussion
before the Terms of Reference for this evaluation are
prepared – and will therefore be taken up in the group
discussion session that follows
………..after any points of clarification