Impact evaluation in the
absence of baseline surveys
By Fabrizio Felloni, Office of Evaluation, IFAD
International Workshop on Development Impact
Evaluation, Paris, November 15, 2006
1
The context of IFAD

Relatively small projects: 2005 median of IFAD
loans = US$ 15.5 m, project costs = US$ 26 m

Focus on rural poverty reduction

Traditionally: limited field presence of IFAD (15
countries on a pilot basis), IFAD not executing or
supervising projects, limited self- evaluation

This scenario is evolving with new Action Plan
2
Field-based evaluation at IFAD - OE

Necessary to make up for distance of
headquarters from the field and information gap

Several types: project, country programme and
corporate evaluations

All include field visit and some form of primary
data collection

Project evaluations conducted just before or
soon after project closure
3
Methodological requirements

Standardised methodology for project and country
programme evaluations requires assessing impact
(standardised categories)

No standardised data collection methods: to be
identified at approach paper phase

Impact is but one of the analytical domains (also
relevance, effectiveness, efficiency, sustainability
innovation, performance of partners)
So no “dedicated” instrument for impact assessment

4
Shoe string evaluation in
action
Considerations from personal experience
5
A case of shoe string impact evaluation

See Bamberger et alii AJE, 25 (1), 2004

A number of constraints
1. Time and budget (impact is one of the evaluation
domains)
2. Poor performance of M&E function at project level
3. Absence or limited usefulness of baseline data (now
changing: baseline survey with anthropometric and
hh asset indicators for all new projects)
6
Logical steps for impact assessment
1
Preliminary
quantitative
mini-survey
Formulate first
impact
hypotheses, collect
evidence on
selected “basic
indicators”
2
Multi-disciplinary
field visit (mainly
qualitative + direct
observations)
Validate
hypotheses, probe
on a set of
narrower questions
3
Impact
assessment
Triangulation of
mini-survey, focus
groups and
individual
interviews + key
informants
7
A pragmatic approach

Within this context, impact assessment based on
triangulation, still important qualitative component

Still place for theory-based approach

Quantitative survey used to test and generate new
hypotheses, better focus questions during main
mission

Small sample size: 200 – 350 respondents including
project and control. Size determined by practical
issues (represent project activities, time,
transportation, budget)
8
“Ideal” scenario for the survey
1. Best case scenario: quasi-experimental design
T0: programme group
T1: programme group
Baseline
C0: control group
Follow-up
C1: control group
Ti and Ci : measurable characteristic of the population, i = time
of observation (0,1).
Unfortunately, this scenario is almost never found
9
Typical scenarios
1. Programme group only
T0: programme group
T1: programme group
Baseline
Follow-up
2. No baseline at all: most frequent case
???
Evaluation
10
Other common issues




Classical problems with “control samples” (selection
bias, spill-over effects, non-compliance)
Ex ante: (i) visit “similar” communities or hh in
administrative areas outside project, (ii) select “new
entries”
Ex post: Mostly dealt with qualitatively at mission
phase (triangulation)
Main constraint to use of econometric techniques:
availability of trained specialists, time (impact is one
of the evaluation domains)
11
Dealing with lack of baseline data

Several options (not mutually exclusive)
1. Reconstructing baseline data ex post: recall
method (more later)
2. Use key informants and triangulate (mostly
qualitative)
3. Reconstruct a baseline “scenario” with
secondary data (not always practical given
absence and quality of baseline studies)
4. Single difference with econometric techniques:
some practical obstacles (workload, time
constraints, availability of trained specialists)
12
Recall methods
Ask about current situation (e.g. cropping practices) now and at
programme start-up
recall
T0
T1: programme
group
C0
C1: control group
recall
13
Typical problems with recall methods

Telescoping of major events / expenditures

Under-estimation of small and routine events / expenditures

Recall time line (events that are 3 -7 years old)

Unintended misidentification of project start-up

“Strategic” behaviour of respondents (to please the
interviewer or express complaints)

Some indicators are more complex to identify and remember
with precision (income)
14
Some techniques to control problems



Concentrate on few impact variables that are easier to
“visualise” and recall. Some examples:
household appliances, livestock size (depending on the
context),
cropping patterns, agricultural and grazing practices,
community initiatives)

Help identify baseline point by helping recollect key facts
and events

Do not simply ask “what”, ask “why”, i.e. respondents to
state causal linkages. E.g. the number of goats increased:
why? and how? Also useful for attribution.

Pre-test the instruments
15
Practical examples
16
Ex 1.The Gambia: Rural Finance Project (2004)

Preliminary survey
- Project and control group
- Recall: income and assets at hh
and kafo level

Data analysis
- Descriptive statistics and
significance tests + principal
component analysis
- Generated two hypotheses:
(i) limited overall impact on hh income;
(ii) biases against relatively poorer hh in
villages
17
Gambia, Rural Finance (cont’d)

Field mission: focus groups,
individual interviews + key
informants
- Confirmed limited effects on income
generation opportunities
- Credit collateral: discouraged
participation from poorer hh,
ineffective in establishing credit
discipline

Main observations:
- Some validity threats in recall data on
income and monetary assets
- Consistency with qualitative findings
- Help focus the scope of field mission
18
Ex. 2 Ghana Upper East Region

Similar to the Gambia case (project & control, recall)

Multi-component agricultural project: main intervention, small
dams

Recall on household productive and other durable assets

Main findings seemed to show larger effects for project group

Some methodological shortcomings
- difficult to find matched observation for control group (given
multi-component nature)
- small sample size of control group may have affected
significance tests
19
Example 3. Morocco, Southern Oasis

Again, project and control,
with recall method

Many interventions, very
heterogeneous, difficult to
standardise questionnaires

Focus on perceptions of
trends (e.g. income
generating opportunities,
irrigation / potable water
availability, feed for
livestock)

Hypothesis: the project was
effective as a buffer measure
during years of drought.
Supported by qualitative
analysis in field mission
20
Concluding remarks

Preliminary survey and recall methods never a stand – alone
measure but rather propaedeutic to (mainly) qualitative
mission

Triangulation to validate reliability of reconstructed baseline:
survey data, with field observations, focus group, individual
interviews and key informants

By and large, trends suggested by preliminary survey found to
be consistent with qualitative data

Some legitimate concerns on accuracy of estimated means for
certain indicators (income, monetary assets)
21
Concluding remarks (cont’d)

Evolution towards focus on perceived trends on a narrower set
of key indicators

Cost effective to conduct preliminary work with local
specialists and students as enumerators

Project teams consulted in planning and sampling phase.
Results and database made available

Valuable experience for local students – enumerators

In principle, replicable model for public authorities in charge
of programme implementation
22