IPDET Lunch Presentation Series
Equity-focused evaluation:
Opportunities and challenges
Michael Bamberger June 27, 2013
1
Why equity matters
• Many programs do not reach/benefit the poorest and most
vulnerable sectors
• Widely-used indicators [MDGs, HDIs, economic growth] are based
on averages and conceal gap between the poorest [bottom 20%]
and the rest of the population is not being reduced
• The poorest groups are invisible
• Poor and vulnerable groups have special needs that are not
addressed
• Major gaps continue to exist between men and women
• No society has achieved full gender equity
2
Defining equity
• Equity refers to fairness
• It examines unequal outcomes (e.g. income, health, access to
electricity) to determine if differences are due to:
• Unavoidable circumstances (e.g. floods, war, international economy,
limited public resources)
• Free choice (some people may prefer less working hours and more
leisure)
• Avoidable circumstances = inequity:
• Unequal opportunities (family background, education, poverty)
• Contextual factors (economic, political, social, environmental etc)
3
Linking opportunities and outcomes in equity analysis
Opportunities
•
•
•
•
Education
Health
Security
Housing
Outcomes
Contextual factors
•
•
•
•
•
•
•
+/effects on
equity
Economic
Labor markets
Transportation
Political
Socio-cultural
Environmental
Psychological
•
•
•
•
•
Education
Employment
Income
Health and life expectancy
Political participation and
voice
EQUAL
EQUAL
Unavoidable
•
•
•
•
•
Aging
Natural disasters?
War?
Economic cycles?
Geographic location?
UNEQUAL
But not inequitable
Free choice?
UNEQUAL
Avoidable
UNEQUAL
and inequitable
4
Determinants of inequity
•
•
•
•
•
•
Income
Location
Ethnicity, caste, religion
Gender
Disability
Legal systems
5
Why so many evaluations do not
address equity
• Political reasons
• Equity analysis challenges indicators of progress (MDGs,
HDI etc)
• Governments do not wish to give equal benefits
• Roma, refugees, sexual orientation, ethnic minorities
• Socio-cultural
• Society may support marginalizing “undesirable” groups
• Methodological
6
Methodological challenges for evaluating equity
• Lack of disaggregated data
• Many outcomes are sensitive or difficult to identify
and measure
•
•
•
•
Unanticipated outcomes
Complex processes of behavioral change
Equity evaluation requires new methodologies
Findings can be threatening
7
Equity focused evaluation
• Assessing the relevance, effectiveness, efficiency,
impact and sustainability of interventions on
equitable development
• Identifying which groups benefit most and least from
programs and policies
• Assessing effectiveness of policy frameworks and
identifying policy gaps
8
Some equity-focused evaluation frameworks
• Theory of Change
• Contribution analysis
• Realist evaluation
• Social exclusion analysis
• Gender analysis
• Systems analysis
• Bottleneck analysis
• Policy gap analysis
9
Benefits of systems analysis for equity evaluation
• Programs are embedded in existing social and political
systems with historical traditions and linkages among
stakeholders
• Different actors have different perspectives on the program
• Boundaries – open or closed
• New interventions cause contradictions and conflict
• Outcomes determined by how conflicts resolved
10
Bottleneck supply and demand framework: factors affecting use
of services by vulnerable populations
Contextual
factors
• Economic
Supply side
factors
• Coverage
•Budgets and
available resources
• Political
• Institutional
• Legal and administrative
• Environmental
• Socio-cultural
Use of services by
vulnerable
population
Demand side factors
•Knowledge, Attitudes
and Practice of vulnerable
groups
• Community ownership
•Culturally acceptable
services
• Culturally sensitive staff
• distance
• cost of travel and fees
• available transport
•time
•poverty
11
Country and project level tools for
equity analysis
•
•
•
•
•
•
•
Multi-dimensional poverty index (UNDP)
Quintile analysis
Public expenditure incidence analysis
Gender inequity analysis
Access to public services [e.g. Citizen Report Cards]
Participant observation
Participatory group techniques [e.g. Wealth Ranking]
12
Evaluators’ E quity Checklist
• Who are the poorest and most vulnerable groups in your
•
•
•
•
program?
• What constraints affect their participation?
• Does your evaluation address these constraints?
Does your sampling frame cover all of the target population?
Are you sure about this?
Who does and does not use project services? Why?
Are there any constraints on women’s participation?
Will your evaluation design detect unanticipated and negative
outcomes? Are you sure about this?
13
A final reality check
• Do you really understand who are the vulnerable and
marginal groups?
• Are you really sure their situation will be covered by
the evaluation?
• Will your data collection instruments really capture
the situation of invisible groups who do not want to
be interviewed, who are socially marginalized or in
fear of persecution?
14
15