Accountability Without Impact?
Or What Impacts Should
NGOs be accountable for?
John Clark
Oxford, Nov. 2010
Context - Funding trends


1980s – steep increase in support for NGOs
1990s – steep increase in official grants
USG accounted for 70% of US NGOs’ aid by late 90s

2000s – steep increase in CSO delivery of ODA
NGO service agents in big donor programs to deliver MDGs,
Increase in Community-Driven Development via CBOs

Recently – steep rise in business philanthropy
Private giving $49bn in OECD; ODA=$104bn
USG funding for US-NGOs down to 30%
Trends (cont’d)

Steep increase in numbers of NGOs
UK now has 7270 int’l charities, sharing £2.1bn

Growth of bijou charities
Very specific concerns, e.g. IBP, WWB, PTF, Tiri

The “My Cause” age: NGOs that allow a donor
to pick their winner
KIVA, Ashoka, Global Giving etc
So – it may be that the same questions
surface now about M&E as in 1980s, but
different people are asking the questions
Who is assessing NGO
performance?
A. Professional NGO development evaluators: who
ask the sort of questions in the background paper
B. Bilateral agency evaluators: who ask about
Effectiveness, Efficiency, Relevance and Sustainability
C. Philanthropy guides: who ask about the bottom line
and business efficiency. What gets measured gets managed.
But B and C influence the world in which A work
Business analogy
What do investment analysis measure?
Productivity
 Price advantage compared with rivals
 Market share, and changes in this
 Profits
 Market trends (consumer patterns, world affairs…)
They DON’T measure quality of products because this is best
judged by the market – by choice of citizens as consumers
In our world, what is the market? Poor communities? They
rarely have choice. The donors choose – by what measures?

Efficiency indicators for DFID
DFID Business Plan 2011-2015, November 2010



cost of running DFID;
numbers of employees,
% of costs due to “corporate services” (overheads)
…and for “3rd party spend”:



property costs per sq. m and per employee
cost and number of computers per employee
Value of major areas of spending (office products,
travel etc.)
No./value of govt. projects; if delivered to
time/budget
DFID Input Indicators








Cost per child supported in primary education
Cost per bed net distributed
Cost per sanitation facility (e.g. toilet, sewer) built or
upgraded
Cost of building, upgrading, maintaining or
rehabilitating roads
DFID spend on fighting corruption
Cost per birth delivered by nurses, midwives or doctors
DFID spend on climate change
DFID spend on multilateral organisations
DFID-achieved impact indicators







No. children completing 5yrs primary education
No. bed nets distributed
No. sanitation facilities built/upgraded
Km. roads built, upgraded, maintained etc
Worldwide Governance Indicator (corruption level)
No. births with help of nurses/midwives/doctors
No. poor people better able to cope with climatic shocks
NGO ratings agencies
American Institute of Philanthropy
Service to help donors find “top-rated charities
based on our rigorous analysis”
RATING CRITERIA:
 % budget spent on charity’s declared purpose
 Cost of raising $100
 Years of available assets
Charity Navigator
More elaborate set of SEVEN criteria: % NGO expenditures spent on programs
 Administrative expenses
 Fundraising expenses
 Fundraising efficiency
 Primary revenue growth
 Program expense growth
 Working capital ratio
None relate to the QUALITY of the programs
CN is now changing – to focus on Social Impact
NGO governance





The contagious effect of scandals (Unicef-Germany)
Pressure for transparency (2-way)
Focus on boards’ independence/professionalism
Growing no. of codes of conduct (OneWorld database)
Self-policing mechanisms (by trade assocs. e.g. InterAction)
The Age of Measurement




NGOs are increasingly assessed quantitatively
What can be measured may not be most important
Advocacy, complex systems change, empowerment
etc are traditionally seen as qualitative domains
But this doesn’t mean success can’t be assessed
MAXIM: find ways to COUNT what counts!
The client is the poor family, not the donor
In Operational Programs

Scant literature on impact of NGO projects
Lacuna interpreted as grounds to doubt NGO roles

10 health progs, comparison with state services:
clear NGO benefits: immunization up 11-21%;
malnutrition rates down 18%; TB cure rate ↑ 14%, cost  15%
 PHC outsourcing in Uganda: better care, 28% cheaper,
more motivated staff, better in detecting malaria, etc.
 Meta-analysis, 7 countries in education: completion
rates 74%, c.f. 59% in govt. schools; pass rates 67%, c.f. 37%;
cost per completer was 2.8 x higher in govt. schools

Shouldn’t be either/or – CSOs as co-providers:
HIV/AIDS services in hard-to-reach groups; child/mother nutrition
In participatory development

Harder to assess, but not impossible (few do it!)




Social accountability impact




KDP in Indonesia – combine CSOs + audits
Social Audits in NREGA, India (AP, Rajasthan)
Constituency Voice: Satisfaction, Ownership, Trust (Keystone)
Reducing errors and budget leaks
Reducing bad procurement processes
Improving quality and responsiveness of public services
Advocacy impact


Combating discrimination against women, Bangladesh
Stemming forestry concessions, Cambodia
Assessing social impact
Robert Chambers view
“Intensifying paradigmatic tug-of-war”
A. Top-down, donor-driven and bureaucratic …
targets set from above,
donors with log
frames, objectively
verifiable indicators,
impact evaluation in
its cruder forms …
reporting upwards on
achievements
B. Local diversity, differing values and evolving
goals, and methodological pluralism and
inventiveness
Participatory, more
egalitarian, more flexible,
concerned with what is less
tangible and less measurable
like relationships, and often
with trust, empowerment
and people.
e.g. 1 – Government Watch, Philippines






Pinpointing corruption and inefficiency; Education
Highlighting the problem won official allies
“Textbook Walk” – revealed culprits
Reforms reduced costs by 55%, time by 63%
Boy Scouts empowered – Governance Camp
World Scouts gathering – 2000 joined governance camp
Every step could be documented, measured
and returned to for long-term impact
e.g. 2 – Use of devolved funds, Kenya






CRSP had supported village devel. for years
CBO structure, with AKF support services
With ↑ in devolved funds, strategy changed
CBOs linked in clusters, federated at District level
Negotiation achieved with LGUs, councilors
CBO Monitoring of selected projects
RESULTS: more poor-focused projects, more successful
implementation, less un-spent funds, better local govt.
But no-one is monitoring these issues; supported by
PPA, DFID more concerned with “VfM”
N and S NGOs are in same boat




Not true that there is “clear breakdown in trust”
Issues may be ones discussed 20 yrs ago, but
doesn’t mean they were solved or are insoluble
Participation never went out of fashion
Interest in randomized trials is an interest in
showing results – reasonable in a $50bn industry
Conclusions

We can sneer – but this is the age of “counters”
Can’t beat ‘em, so need to join ‘em
Randomized trials can be effective (e.g.MIT-PAL)
Other approaches can also document progress
Needn’t be cynical; one experience is a story

“Anecdote” is an irregular noun
CSOs need to be challenged, but not changed




Academics, M&E specialists need to work with NGO
planners; to help quantify their outcomes
MAXIM: find ways to COUNT what counts!

The client is the poor family, not the donor