Evaluating Humanitarian Aid:
November 2013
Harry Shannon
McMaster University
Working with
• David Sanderson
Director, Centre for Development and Emergency Practice
(CENDEP), Oxford Brookes University
• Charles Parrack, CENDEP
Outline of talk
•
•
•
•
•
What is (humanitarian) aid
Issues in evaluation of aid
Shelter after disaster
Proposed research agenda for evaluation
Discussion
Work in Progress!
Humanitarian imperative
• Helping fellow human beings in (desperate) need
• “How we think about each other, our relationships,
our place in the ecosystem … are the most basic
questions … We must recognise humility; trying and
doing; governance … given them, don’t wait for the
perfect answer.”
– James Orbinski, former President of MSF
Maimonides: hierarchy of charitable giving
1. The lowest: Giving begrudgingly and making the recipient feel disgraced
or embarrassed.
2. Giving cheerfully but giving too little.
3. Giving cheerfully and adequately but only after being asked.
4. Giving before being asked.
5. Giving when you do not know who is the individual benefiting, but the
recipient knows your identity.
6. Giving when you know who is the individual benefiting, but the recipient
does not know your identity.
7. Giving when neither the donor nor the recipient is aware of the other's
identity.
8. The Highest: Giving money, a loan, your time or whatever else it takes to
enable an individual to be self-reliant.
charitywatch.org
Definitions (though boundaries fuzzy)
• (Overseas Development) Aid: flows to developing
countries and institutions whose aim is to promote
development and welfare of the recipient countries.
• Humanitarian aid/action (HA): aid that follows a
disaster (complex emergency), typically on a more
urgent basis than regular ODA
Contrast with …
• Humanitarian intervention: the use of military force
by a state to end human rights violations by a state
(R2P)
Millennium Development Goals, 2000
By 2015:
1.
2.
3.
4.
5.
6.
7.
8.
Eradicating extreme poverty and hunger
Achieving universal primary education
Promoting gender equality and empowering women
Reducing child mortality rates
Improving maternal health
Combating HIV/AIDS, malaria, and other diseases
Ensuring environmental sustainability
Developing a global partnership for development
Millennium Development Goals
• Specific targets set, e.g., between 1990 and 2015,
halve the proportion of people living on less than
$1.25
• Ban Ki Moon: ‘The Millennium Development Goals
(MDGs) have been the most successful global antipoverty push in history.’
• However, progress has been uneven
• Being replaced in 2015 by Sustainable Development
Goals
Development aid
•
•
•
•
•
$140 billion / year
Dambisa Moyo, William Easterly - sceptics
Jeffrey Sachs – proponent
Paul Collier - ?intermediate
Esther Duflo, Abhijit Banerjee – RCTs
(Randomized Controlled Trials)
• All support humanitarian aid (HA)
Duncan Green is Senior Strategic
Adviser at Oxfam GB
Duncan Green, From Poverty to Power
Sceptical view of
Canada’s NGOs
Humanitarian aid ‘system’ - some numbers
• Estimated 4,400 NGOs worldwide doing some
HA
(Haiti sometimes called a ‘Republic of NGOs’)
• 2010: estimated 274,000 humanitarian
workers worldwide
• Approx $14 billion / year
Humanitarian aid ‘system’
• UN humanitarian agencies
• International Movement of the Red Cross and
Red Crescent
• Five largest International NGOs
Médecins Sans Frontières (MSF)
Catholic Relief Services (CRS)
Oxfam International
International Save the Children alliance
World Vision International
• Increasing attention to ‘accountability’
Cluster approach – 2005
Inter-Agency Standing Committee (IASC)
CODE OF CONDUCT
for the International Red Cross and Red Crescent Movement
and NGOs doing humanitarian aid activities
1) The humanitarian imperative comes first.
2) Aid is given regardless of the race, creed or nationality of the
recipients and without adverse distinction of any kind. Aid
priorities are calculated on the basis of need alone.
3) Aid will not be used to further a particular political or
religious standpoint. (Independence)
4) We shall endeavour not to act as instruments of government
foreign policy. (Neutrality)
5) We shall respect culture and custom.
6) We shall attempt to build disaster response on local
capacities.
CODE OF CONDUCT
for the International Red Cross and Red Crescent Movement
and NGOs doing humanitarian aid activities
7) Ways shall be found to involve programme beneficiaries in
the management of relief programmes.
8) Relief aid must strive to reduce future vulnerabilities to
disaster as well as meeting basic needs.
9) We hold ourselves accountable to both those we seek to
assist and those from whom we accept resources.
10) In our information, publicity and advertising activities, we
shall recognise disaster victims as dignified human beings,
not hopeless objects.
Emphases added
What should
humanitarian groups
aspire to?
Some ethical dilemmas
• Do we respond in Japan, where it’s expensive,
or in the Congo where we can do more for less?
• If we work with, e.g., the Syrian government, do
we risk being co-opted into their strategy?
• If we bring food, do we drive down market
prices and upset local agriculture?
• If we give out food at distribution points, do we
put people at risk of attack when they walk
home?
Ctd …
Some ethical dilemmas (ctd)
• What is the balance between paternalism and
autonomy?
• Do we speak out about terrible things (only)
we see, or stay quiet, do our work and save
lives, and let others speak out?
• Relations with military?
These decisions are made under (severe) time
pressure, and possibly armed force and
persuasion/coercion.
Source: Hugo Slim
The Guardian, 28 August 2013
On 14 August, Médecins Sans Frontières (MSF) announced that,
after 22 years, it was closing all its activities in Somalia as “the
situation in the country has created an untenable imbalance
between the risks and compromises our staff must make, and our
ability to provide assistance to the Somali people.”
…
Of the 160 aid workers killed, 82 wounded and 99 kidnapped in
Somalia since 1997, the majority have taken place since 2007 (106
killed, 53 wounded and 57 kidnapped), when the overall security
environment deteriorated for both humanitarians and Somali
civilians.
The Guardian, 28 August 2013
Report on aid worker security
October 2013
Some practical concerns with HA
•
•
•
•
•
•
•
Little argument about the principle
Questions about how well HA is provided
Speed of response
Coverage of those in need
Duplication
Unnecessary help
Doing more harm than good (cholera in Haiti)
Critique of humanitarian aid
• Use of HA for political purposes has been going on
for a long time
• British soldiers in Afghanistan – giving food in
exchange for information on Taliban hiding places
• Colin Powell to NGOs: You are our force multipliers
• In Darfur, aid workers have provided actionable
information for ICC
• Secret services infiltrate NGOs
• Embassies keep an eye on what ‘their’ NGOs are
doing
Source: Antonio Donini
Other possible problems with aid
•
•
•
•
•
•
•
•
NGOs attract talent from local governments
Effect of internationals on local prices
NGOs supplant government roles
Plethora of NGOs; potential competition
Government corruption
Qualifications to provide help
(Non) Involvement in politics
Promised money may not arrive
Port-au-Prince, October 2011
Humanitarian/development aid or
conscience money?
Amira Hass in Israeli newspaper Haaretz, 6 Feb 2013
7 Golden Rules of Recovery Management
1. Trust survivors and avoid paternalism
2. Trust survivors and enable survivors to assess
their own needs
3. Trust survivors and provide cash rather than kind
4. Think locally
5. Give priority attention to vulnerable groups
6. Think ‘process’ not product, ‘sheltering’ not
shelters, ‘housing’ not houses
7. Adopt a long-term perspective while addressing
short-term needs
Ian Davis + Anshu Sharma, 14 Nov 2013
Toby Ord: “Aid Works (on Average)”
Talk in Oxford, 13 February 2013
• Cited Moyo:
– Over 60 years, more than US$1 trillion in aid and not much
good to show for it
– $1 trillion is nearly $1,000 per person on the planet today
• Ord:
– Nonsense, actually 1 billion people in Africa
– $1 tn / 440 mn (average number of people in Sub-Saharan
Africa over period)
= $2,263 per person = $38 per person per year
Toby Ord: “Aid Works (on Average)” (ctd)
Successes include:
Smallpox eradication
Oral rehydration therapy for
diarrheal diseases
Immunization
Malaria
Lives saved since eradication:
60-120 million
Annual deaths down from
4.6 million to 1.6 million
Measles annual deaths down
from 5 million to 1.2 million
Annual deaths down from
3.8 million to 0.7 million
Why evaluating ‘aid’ is difficult
•
•
•
•
•
•
•
•
•
Reason aid is given
Reverse causation
Non-developmental aid
How/where aid is actually spent
Fungibility
Multiple donors
Remittances
Unpredictability of aid
Short- vs. long-term impact of projects
Aid or …?
The Guardian, 21 February 2013
Critique of methods in evaluating aid
Stuckler, McKee, Basu 2012
•
•
•
•
•
•
Reviews are narrative not systematic
Evaluations don’t consider bias, confounding
Multiple outcome measures
Time lags
Data limitations
Power, Type II errors
They advocate:
• Experimentation
• Evaluation
• Replication
Evaluating aid - Definitions
• Process evaluation: how the intervention was done
• Impact evaluation: longer-term, intended or
unintended
• Indicator – measure of the effect
Experiments
• Small-scale RCTs
• Examples:
– paying parents if child
attends school
regularly
– giving food when
children vaccinated
– microcredit
Increasing attention to accountability
• ALNAP - Active Learning Network for
Accountability and Performance in
humanitarian action
• Jim Kim, President of World Bank: “Science of
delivery [of aid]”
• Various documents produced by different
NGOs on how to evaluate
Evaluating the humanitarian system
ALNAP
• Collates evaluations and produces summary
reports
• State of the Humanitarian System:
‘assesses progress and performance in six areas:
coverage/sufficiency, relevance/appropriateness,
effectiveness, connectedness, efficiency and coherence.’
• Interactive online evaluation training materials
Difficulties in evaluating HA impact
•
•
•
•
No baseline / control (or poor data)
Need for rapid information
Insecure / volatile environments
Surveys in settings with little information on
the target population
• Impact of single intervention vs broader
impact of programme
• Lack of capacity
Qualitative and quantitative methods
• “… much of the guidance for the evaluation of
humanitarian action suggests that evaluations are
more likely to provide robust evidence where they
use ‘mixed methods’ approaches.”
• “However, in practice, humanitarian evaluation ‘uses
mainly qualitative methods’ (Buchanan-Smith and
Cosgrave 2012).”
• “… evaluations still tend to undervalue the
experience of affected populations as a source of
evidence …”
James Darcy and Paul Knox Clarke, 2013
Focus on Shelter
• Seen as human right
• One of Clusters – incorporates settlement and nonfood items
• Housing used for purposes other than purely shelter
• Relatively little on shelter evaluation
• ‘Providing adequate shelter is one of the most
intractable problems in international humanitarian
response’ ‘(Humanitarian Emergency Response
Review, 2011)
• ‘Building Back Better’
Housing and health
Shaw, Ann Rev Pub Health, 2004
Right to adequate housing
• Article 25 of Universal Declaration of Human
Rights
• Adequacy includes
–
–
–
–
–
–
–
legal security of tenure
availability of services
affordability
habitability
accessibility
location
cultural adequacy
UNHCR, 1991
Section in
Sphere Handbook
Proposed timing of shelter after disaster
Relief
Recovery
Reconstruction
Emergency shelter
Temporary / transitional
shelter
Permanent shelter /
reconstruction
Charles Parrack
0-2 weeks
2 weeks – 2 years
2 years +
Shelter damage in Port-au-Prince
after 2010 Haiti earthquake
Damage to housing
%
95% CI
Completely destroyed
24.4
22.1-26.9
Partly damaged
41.5
38.9-44.1
No visible damage
34.1
31.7-36.6
Kolbe, Hutson, Shannon et al., Medicine, Conflict and Survival, 2010
Shelter for displaced - refugees
Proposed agenda for evaluating
shelter after disaster
• Step 1a: Systematically identify indicators
• Step 1b: Improve list by consulting past and
potential beneficiaries
• Step 2: Develop (short) questionnaire
• Step 3: Conduct longitudinal study (RCT?)
after disaster
Housing trade-off: durability vs comfort
© David Sanderson; with permission
Step 3: Conduct longitudinal study/RCT
after disaster
• Timing:
– start 1-2 weeks after disaster
– follow up at 6 months, 1 year and 2 years
• Allocate areas/communities/villages to different
approaches to housing, i.e., cluster
allocation/randomization
• E.g., ‘T-shelters’ vs. housing in camps vs. building
materials and tools
• Mixed methods – quantitative and qualitative
Feasibility of RCTs?
Example of randomization
at the individual level
Technical problems with RCTs include the need to deliver the
treatment in a standardised way, the difficulties of maintaining
stable membership in the treatment and control groups, and the
expense and effort needed. These constraints led a review on
evaluations of health promotion by the World Health Organization
in Europe to find that the use of RCTs for this was ‘in most cases,
inappropriate, misleading and unnecessarily expensive’ (WHO
Regional Office for Europe, 1998, p. 5). RCTs also require that the
evaluation be designed from the start of the project with random
selection of control and treatment groups. For all of these reasons,
RCTs are rare in humanitarian evaluation.
(emphasis added)
ALNAP EHA Pilot Guide p.113
Example of randomization
at the community (cluster) level
Discussion