Phase 2 Evaluation of the PD
Key Issues for
Evaluation Design
Reference Group Workshop/Meeting
11 – 13 February 2009
Evaluation Design
Many choices must be made when finalising an
evaluation design, for example, we have to choose:
• The main focus and purpose
• The Unit of Analysis – what are the investigations
located
• Methods and techniques
• Information sources
• Management and governance etc.
All of these are discussed at this workshop
Evaluation Design
In this session we want to concentrate on two
important design choices:
• The evaluation questions that should be prioritised
in the evaluation
• Some propositions (or hypotheses) that follow
from the characteristics of the PD itself
Evaluation Questions
Evaluation questions are at the centre of any
evaluation – what are they?
At simple level they are questions we want answers to
and we are likely to be able to use the results to
inform policy or practice
Evaluation questions also concern effects and imply
judgement – placing a value on these effects
Evaluation Questions
So an evaluation question :
• Concerns an effect or result external to the
interventions of the programme or policy: if a
question is just about the internal working of the
policy it’s better dealt with by monitoring or
auditing!
For example……………
Evaluation Questions
An evaluation question also implies a judgement
criterion:
Some evaluation questions at least should help us
understand what constitutes ‘success’,
‘achievement’ or ‘good’ performance
For example……………
Evaluation Questions
Evaluation questions lead to certain types of ‘enquiry’.
They can be:
• Descriptive: so that the evaluation describes what
happens in a situation or over time
• Explanatory: so that the evaluation investigates
causal relationships
• Normative: so that the evaluation focuses on how
to bring about desired change
For example…………
Evaluation Questions
In the Approach Paper & the ‘linkages’ study different
types of evaluation questions are identified. The
main types are:
Results questions. For example: How has the
adoption of PD principles and commitments
improved development results
And
Counterfactual questions. For example: Are there
other ways – not following PD principles – that
could lead to similar results more effectively &
efficiently
Evaluation Questions
The emphasis on results follows from the focus of the
Phase 2 evaluation.
Of course we can ask ‘process’ and implementation
questions as well if these processes help us
understand results
Evaluation Questions
But these are examples only – in the group discussions
after this presentation you will have the chance to
put forward what you think are the most important
questions for the evaluation to answer
Characteristics of the PD
Many evaluation design choices aren’t choices at all –
they follow from the character of the policy or
programme being evaluated! For example the linkages
study identifies some of the special qualities of the PD
that make this evaluation very challenging
Characteristics of the PD
For example we know that:
• The PD is closely connected to other aid and
development policies: how to separate out PD effects?
• Development actors (governments, civil society etc.)
will use PD arrangements differently – there won’t be
one standard model
• The PD is not a single once-only intervention it is a
development and learning process
• Not all results of the PD will be evident by 2010!
Characteristics of the PD
• Evaluation will be difficult but possible! The ‘linkages’
study identifies 21 ‘propositions’ (or hypothesis) that
research suggests could be tested in the Phase 2
evaluation. These are on Page 16 of the Approach
Paper.
• These ‘propositions’ come from research rather than
practice. This is why it would be useful for people
here who are familiar with aid & development policy
to consider and review these propositions
Discussion Tasks
The two tasks for the discussion groups will concentrate
on these two design choices:
• Evaluation Questions
• Evaluation Propositions
• But first are there any questions or points of
clarification……..?