Reviewing the Current Evidence
Brandon Brockmyer
bbrockmyer@gmail.com
Jonathan Fox
fox@american.edu
Goals of the Synthesis Study
 To assess the current state of the
evidence for MSI effectiveness and
impact
 To identify remaining research gaps
 Still to come: To identify relevant lessons, insights, and
trends and make recommendations to MSIs, participating
governments, donors, and CSO good governance
practitioners
Conceptual Issue 1:
Defining “public governance
multi-stakeholder initiatives”



Global reach: participants on most continents
Voluntary: not encoded in international law
Multi-stakeholder membership: governments,
CSOs, multilateral organizations, and multinational
corporations

Multi-stakeholder governance: formal power-sharing
arrangement

Public governance-oriented: addresses policy and
decision-making by national governments

T/A/P focus: develops/improves processes or standards
for transparency, accountability, and/or public participation
Transnational public governance MSIs
are a small subset of a much larger universe
GAVI Vaccine Alliance
TMSIs
Kimberly Process
International Union for the
Conservation of Nature
T/A/P
Sanitation scores
for CA restaurants
Public
Governance
Integrated Health Services
Initiative in Zambia
Participatory Budgeting in Brazil
Transnational Public Governance MSIs
Transnational public governance MSIs
have important differences in scope
EITI
GIFT
OCP
CoST
OGP
Conceptual Issue 2:
Unpacking MSI progress

 Outputs 
 Process-Oriented; Compliance

  Effectiveness 
 Medium-term strategic outcomes; Influenced
by a mix of internal processes and external
social and material factors

   Impact
 Long-term outcomes; Heavily influenced by
external material and social factors
This framework draws on: Gutner & Thompson’s “The Politics of IO Performance”
published in The Review of International Organizations (2010)
Conceptual Issue 2:
Unpacking MSI progress

Definitions will depend on each MSI’s
Theory of Change

Definitions will also depend on the Level of
Analysis
 MSIs have international and national goals
 One stakeholder’s “output” can be another
stakeholder’s “input”
What types of evidence are out there?
 Single-country case studies (collected but not reviewed)
○
OGP-IRM reports, EITI validation reports; CoST assurance team reports; OGPSupport “Inspiring Stories”; “EITI stories”; OGP-Hub Civil Society progress
reports
 MSI strategy documents
○
OGP research agenda; EITI Report from the Working Group on Theory of
Change; OCP Guide for Practitioners
 Multi-country studies
○
EITI Progress Report 2014; CoST report on information disclosure and
assurance team findings; OCP Winter 2014 Update; Scanteam evaluation of
EITI; “Eye on EITI”
 Large-N quantitative studies
○
OGP-IRM technical report; BIC survey of civil society participation in EITI
 Cross-initiative studies
○
Khagram Task Force; OKF “Joined-up data”; “The possible shape of a land
transparency initiative: Lessons from other transparency initiatives”
Documents collected split evenly between
single-country focus and broader MSI focus
350
300
250
171
200
MSI Focus
150
Single-Country
Focus
100
154
50
0
# Documents Collected
Half of all reviewed documents focus on EITI
0
10
20
30
40
# Documents Reviewed
36
13
9
8
8
EITI
OGP
CoST
GIFT
OCP
The number of documents identified is
imperfectly related to MSI age
EITI
2000
2002
2001
OGP
2004
2003
2006
2005
2008
2007
2009
CoST
Most MSIs have only been
active for a few years
2010
OCP
2012
2011
2014
2013
GIFT
2015
EITI documents vary by scope and content
45
40
35
Single-country case
study
Multi-country case
study
# Documents
30
25
20
Large-N quantitative
study
MSI Strategy
Documents
15
10
Theory and/or Practice
Literature Review
5
Cross-initiative studies
0
EITI documents reviewed come from a variety
of sources
16
MSI Reports
14
12
# Documents
10
8
6
Commissioned
Evaluations
External (NGO,
Government, Industry)
4
Academic Articles &
Books
2
Blog Posts
0
Document analysis has been augmented by
stakeholder interviews (with more to come)
5
4
3
2
1
0
F
M
EITI: What does the evidence tell us?

Only EITI has been operating long enough to
generate serious reviews of medium and longterm effectiveness

EITI has made progress in improving the
transparency of extractive industry payments to
national governments

These bounded reforms have not translated into
widespread accountability demands by citizens or
improvements in resource distribution
EITI has made progress on a number of
process-oriented goals:
 Drawing attention to problems in the extractives
industry
 Bringing together diverse international
stakeholders
 Increasing uptake by governments and private
interests
 Creating new opportunities for dialogue between
government, civil society, and the private sector
at the national level
EITI has made progress implementing its
agenda in participating countries:

32 countries are compliant with EITI requirements
 EI payment information is being released (often for the first time)
 Countries are innovating

EITI implementation has led to some government policy
changes
 Nigeria EITI Act 2007; Liberia EITI Act 2009; Ghana Petroleum &
Management Bill 2011

Revenue disclosure has uncovered discrepancies in
extractive revenue payments
 e.g., Nigeria, DRC

Beyond identifying payment discrepancies, there are
few examples of tangible benefits to participating
countries:
 NEITI audits have identified US $9.8 billion owed to the Federal
Government, of which $2 billion has been recovered
 DRC found that $88 million in tax collections were missing in
2010, but no funds have been recovered, despite a long
investigation by the auditor general’s office

Without a systematic review of EITI countries, there is
no way to tell how widespread national-level progress
has been
Findings regarding national-level MSI progress seem to
be based on a relatively small subset of country cases
Country
OGP
EITI
Azerbaijan
Brazil
Croatia
Ghana
Guatemala
Indonesia
Liberia
Mexico
Nigeria
Peru
Philippines
Tanzania
UK
5
3
5
1
1
4
4
CoST
GIFT
OCP
2
3
1
4
1
7
5
3
3
4
4
26
2
1
3
1
2
1
4
Total
9
5
5
5
5
5
7
5
28
5
10
5
6
Countries for which five or more documents were collected across all MSIs
EITI has shown mixed results with regard
to macro-level social impacts:

EITI participation has a statistically positive
relationship with FDI, GDP per capita, and
perceptions about business climate, rule of law,
and government capacity
 Correlation studies alone are not able to determine the
direction of the causal arrow

No relationship between EITI and perceptions of
democracy, political stability, or corruption
 “Laggards” are washing out the successes of “leaders”
Most assessments conclude that EITI’s
agenda was too narrow to have broad
social impacts

EITI’s transparency standard was too narrow
 Disclosure should include the whole contracting process
as well as expenditure data; Information should be
disaggregated by project

Transparency gains will not improve accountability
in a vacuum
 Requires building additional capacity within civil society

EITI needs an explicit theory of change
 To help inform overall strategy for medium and long-term
goals; Establish benchmarks for evaluation
2013 Changes to the EITI Standard







EITI multi-stakeholder groups (MSGs) in each country are required to set
their own implementation objectives in a work plan
EITI Reports must contain basic contextual information about the
extractive sector (disclosure of licenses, revenue allocation by region)
New disclosure requirements (including disaggregated reporting,
subnational transfers, and social expenditures by companies)
Annual activity reports for all countries (not just compliant countries)
Validation will now be procured and managed by the International
Secretariat rather than by implementing countries
Countries revalidated every three years as opposed to every five years
Machine-readable data (encouraged)
Recommendations that were not included:

Expenditures disclosure

Contracts disclosure (encouraged)

Beneficial ownership (encouraged)

EITI validation gradient (i.e., fail, meets the minimum requirement, high pass)
From EITI Report from the Working Group on Theory of Change (WGTOC) (2012)
CoST & OGP:
What does the evidence tell us?
Like EITI, both CoST and OGP have helped to
shape debates on public governance at both the
international and national level
 Like EITI, both CoST and OGP can point to some
national-level policy changes
 Unlike EITI, neither CoST, nor OGP have been
subjected to an assessment of medium or long
term effectiveness

 CoST’s secretariat host organization (EAP) is currently
being reviewed
 OGP’s first independent evaluation will be completed in
2016
CoST & OGP show some progress on processoriented goals:


Bringing together diverse international
stakeholders
Creating some new opportunities for dialogue
between government and civil society (and, in the
case of CoST, the private sector)
 OGP-IRM data suggests this is an area for improvement

Significant uptake by national governments
(particularly OGP: 65 countries)
CoST & OGP show some progress implementing
their agendas in participating countries:

OGP: 37 countries have completed a full action plan
cycle; some policy changes at the national level
○ e.g., FOI laws in Brazil, Sierra Leone, UK (gains can’t
necessarily be attributed to OGP)

CoST: 8 countries have disclosed public works data for
some projects; some projects redesigned/canceled
○ e.g., inefficient contractor suspended for 2 years by Ethiopian
Roads Authority; public works contract for the reconstruction
of the Belize Bridge in Guatemala City was annulled

Evidence of progress limited to single-country
compliance reports and “success story” briefs
It is difficult to compare results across countries
for CoST and OGP because each participating
country sets its own goals
Requirement for
Country
Participation
Minimum
international
standard for
scope and
content
National
discretion on
scope and/or
content
MSI
EITI
CoST; OGP*;
GIFT; OCP
*Do OGP “starred” commitments improve cross-country comparability?
GIFT & OCP:
What does the evidence tell us?

GIFT and OCP have only recently formalized
their international governance structures

Both initiatives have completed high-level
principles

Both initiatives are still designing a strategy for
action at the national level
GIFT and OCP show progress promoting
their agendas at the international level:

GIFT
 Harmonizing fiscal transparency standards across the IMF, IBP,
and others
 UN resolution adopted GIFT’s high-level principles of fiscal
transparency

OCP
 UN Global Compact encourages Open Contracting
commitments

Open Government Partnership promotes both OC and FT
 National action plans
 Working groups
GIFT and OCP principles are being
implemented by national-level actors:

GIFT
 GIFT consults on fiscal transparency in Brazil, Mexico,
Philippines, and South Africa and is arranging peerlearning events in Latin America and Asia

OCP
 Open contracting principles have been implemented in
some projects in over 15 countries via the World Bank
 Seven countries have expressed interest in piloting the
Open Contracting Data Standard (OCDS)
The state of the evidence

Sources of evidence for effectiveness and impact
are still few and far between
 MSIs are still fairly new
 Existing documents focus on governance structure,
incentives for participation and membership, and
procedures to ensure fairness and compliance

Stakeholders report that at the moment, “success”
is still a fluid, negotiated concept
 Politics (and funding) matter
Other MSIs have learned from EITI evaluations




EITI needs an explicit theory of change
 Stakeholders report GIFT and OCP are working on TOC
 OGP and CoST already have them
The EITI standard should cover more of the EI value chain
 CoST considers the whole construction project value chain
 OCP considers all steps in the contracting process
EITI demonstrated no macro-level social impacts, despite ambitious
claims to poverty reduction and anti-corruption
 Neither CoST nor OCP focus on macro-level metrics of success, they
focus on “value for the money”
EITI needs to encourage linkages between national MSGs and other
national and international reform movements
 Stakeholders report linkages to the broader enabling environment are a
major strategy for OCP, GIFT, and OGP
Evaluation of effectiveness remains a challenge

All MSIs struggle to fund a robust internal mechanism for
M&E
 Stakeholders report instances where plans for more comprehensive evaluations were
cut due to budget constraints

All MSIs struggle with how to measure medium and long
term effects
 Often, “leaders” and “laggards” cancel each other out in the aggregate, limiting the
usefulness of large-N probability modeling; Comparing across case studies is difficult,
time consuming, and expensive

There is not a shared understanding of the terminology
used to evaluate MSIs (i.e., “impact”, “outcome”, “output”)
 e.g., “Impact” of EITI = Increasing membership? Increasing validation? National
reforms? Increases in FDI?

All MSI’s struggle with the “but for” question
 The complexity of global challenges precludes attribution of any positive effects to a
single entity, so MSIs must make the case (to funders and national citizenry) for their
contribution to desired outcomes
Theory of Change: Tool for strategy or just
a reporting device for donors?

Helpful for strategic planning

Essential for identifying performance metrics

Articulates a hypothesized causal chain that can be
tested as part of single or comparative case analysis
 Showcases the contribution of various MSI activities to the
causal chain

Increasingly important to funders
GIFT has a broad global TOC, but is still
working on their national-level approach
From “Towards Stronger Incentives for Increased Fiscal Transparency, Participation and Accountability” Discussion Paper (2012)
CoST’s TOC distinguishes between
intermediate outcomes, outcomes, and impact
From the CoST Public Website (“Objectives”) (2013)
OCP’s TOC for implementing OC principles shows the
important role played by the enabling environment and
highlights the cyclical nature of reform
From Open Contracting: A Guide for Practitioners (2013)
OGP’s national-level TOC links the action plan cycle to
the work of key actors at three levels
From OGP Four Year Strategy 2015-2018 (2014)
OGP’s international TOC is used to drive strategic
planning in 4 key areas
From OGP Four Year Strategy 2015-2018 (2014)
OGP uses their TOC to identify outcome metrics for
each of the four key strategic areas….
…and for medium-term
national outcomes as well
From OGP Four Year Strategy 2015-2018 (2014)
OGP also divides the overall scope of assessment into
international and national-level effects and notes where
responsibilities for evaluation are internal or external
From OGP Technical Report 1 (2014)
EITI incorporates the consideration of assumptions and
evidence into their work on TOC
From EITI Report from the Working Group on Theory of Change (WGTOC) (2012)
…but this can get out of hand...
From EITI Report from the Working Group on Theory of Change (WGTOC) (2012)
Research Gaps

Civil society engagement and empowerment

Civil society capacity and interest

Government champions and the political cycle

Private sector participation
Research Gaps

Civil society engagement and empowerment
 Stakeholders report that CSOs participating in national MSGs are
representative of only some segments of civil society, nor do they
always consult with the citizens most affected by MSG decisions
○ Who defines “participation” at the national level?
○ Are there ways to determine if and when CSOs face cooption due to their commitment to
MSI processes (i.e. their goal becomes “success of the MSI” rather than social impact)?
○ Are there ways to make MSI outputs more “demand driven”?
○ What effect does civil society participation have on MSI outcomes?
○ Are there tangible benefits to broader civil society participation?
 Stakeholders report concerns that civic space is shrinking in some
participating countries
○ Are there better ways to measure the health of civic space?
○ Is there any evidence that a “big tent” approach is preferable to imposing stricter rules
and sanctions?
○ If MSIs rely on civic participation, should we expect poorer outcomes in these countries?
If we don’t see poorer performance, what does that mean for the legitimacy of the MSI?
Research Gaps

Civil society capacity and interest
 Stakeholders report that the information being released to the
public is too technical for civil society to use
○ Who is responsible for turning data into actionable information?
○ How is disclosed information being used?
○ Are strategies for building capacity (e.g., learning sessions) working?
 Stakeholders report concerns that a lack of information use by
citizens risks increasing apathy towards governance reform
○ Are MSIs really changing the relationship between government and civil
society?
○ How committed are people to the MSI process?
Research Gaps

Government champions and the political cycle
 Stakeholders in all MSIs report numerous setbacks
resulting from changing domestic political realities
○ Where do potential reformers “sit” within government? How do we
○
○
○
○
create more of them (e.g., peer to peer learning)?
Which ministries are most successful at implementing reforms and
why?
What are some useful strategies for preserving momentum across an
election cycle or other shift in governmental priorities?
What is the effect of MSIs on national government policy, relative to
other actors, advocacy efforts, and events?
Does MSI participation influence whether and how national
governments support or repress civil society?
Research Gaps

Private sector participation
 Stakeholders report that some MSIs have been far more
successful at private sector outreach than others
○ What are the advantages and disadvantages of private sector
involvement?
○ Is there evidence that private sector participation makes a difference
in MSI outcomes?
○ What are some strategies for successful outreach?
Addressing “MSI fatigue”

Stakeholders report significant opportunity costs,
especially for national-level participants
 MSIs require time, energy and resources that cannot be
invested elsewhere
○ Stakeholders want more “hard numbers” on national and international
MSI outcomes (e.g., dollars saved, quality of roads and schools,
citizen satisfaction, perceptions of success) but other MSI outcomes
may be more intangible (e.g., linking pro-reform actors, preserving
spaces for civil society and government interaction, building trust
among coalition actors)
○
Mid-sized comparative case studies can provide a sense for whether
and how MSIs work across countries in the aggregate (single country
case studies lack external validity; large-N probability modeling lacks
causation)
 How long should stakeholders realistically expect wait to
see impacts before deciding to try something else?
Propositions for Discussion
I. Public governance MSIs are still operating within
the early links of the results chain (i.e., inputs 
outputs  outcomes)
 The link between transparency and accountability is
increasing understood as a virtuous cycle rather than a
linear relationship and relies on assumptions that are
still being tested
 It is too soon to expect meaningful evaluations of
“impact”
II. Points of engagement with the results chain will
vary by stakeholder (i.e., one stakeholder’s input is
another stakeholder’s output)
III. Institutional learning is occurring both within and
between MSIs
Propositions for Discussion
IV. There are opportunities for both
transnational and national-level synergies
between MSIs
Ford
G20
Hewlett
IBP
Integrity
Action
NRGI
OECD
Omidyar
One
TAI
TI
World Bank
WRI
OGP^
X
EITI
CoST*
X
X
X
X
GIFT
X
X
X
X
X
X
X
X
X
X
X
X
OCP
X
X
X
X
X
X
X
X
X
X
X
X
X
X
^ OGP has working
relationships with
EITI, GIFT, and OCP
* CoST sits on the
OCP board
Propositions for Discussion
IV. There are opportunities for both
transnational and national-level synergies
between MSIs
Country
OGP
EITI
Colombia
*
*
Guatemala
X
X
X
*
*
Honduras
CoST
GIFT
OCP
X^
Philippines
X
*
X
Tanzania
X
X
X
Ukraine
X
*
*
UK
X
*
X
USA
X
*
*
X
X
*^
X
X = Compliant with MSI; * = Commitment to MSI; ^ = OCDS pilot
Propositions for Discussion
I.
Public governance MSIs are still operating
within the early links of the results chain
II.
Points of engagement with the results chain will
vary by stakeholder
III.
Institutional learning is occurring both within
and between MSIs
IV.
There are opportunities for both transnational
and national-level synergies between MSIs
This research is ongoing and we welcome your input!