GLOBAL
EDUCATION
POLICY, IMPACT
EVALUATIONS, AND
ALTERNATIVES
The Political Economy of
Knowledge Production
D. Brent Edwards Jr.
Global Education Policy, Impact Evaluations, and
Alternatives
“Edwards provides an invaluable case study into how the World Bank produces
‘policy-based evidence’—rather than ‘evidence-based policy’—to reinforce its neoliberal bias. His detailed work on education policy in El Salvador exposes how the
Word Bank’s purportedly scientific and neutral ‘impact evaluations’ are anything
but.”
—Robin Broad, Professor, American University and John Simon
Guggenheim Fellow
“Through an impressive blend of theoretical and empirical analyses, Edwards carefully and critically scrutinises what has typically been taken as a ‘technical’ and
‘neutral’ mechanism in the field of educational aid—i.e., impact evaluation—and
demonstrates the breadth and depth of its consequences as a tool of educational
governance at the national and international levels. This major, original and important book represents a significant contribution to knowledge about the intersection of impact evaluation and educational aid.”
—Roger Dale, Professor, University of Bristol
“Brent Edwards skillfully employs a multi-level, political economy approach to
critically analyzing the agenda-setting role of the World Bank. His study of
EDUCO (Education with Community Participation) in El Salvador convincingly
documents the importance of in-depth studies of the historical and sociocultural
contexts in which reforms arise and are then extended to other countries. By situating this global reform within a comprehensive financial-political-intellectual
complex, he deftly critiques the seemingly rigorous and objective econometric
studies that served as the basis for global policy promotion of EDUCO, while
offering more appropriate research approaches that provide insight as to who benefits from what educational interventions.”
—Robert F. Arnove, Chancellor’s Professor Emeritus of Educational
Leadership and Policy Studies Indiana University, Bloomington
D. Brent Edwards Jr.
Global Education
Policy, Impact
Evaluations, and
Alternatives
The Political Economy of Knowledge Production
D. Brent Edwards Jr.
University of Hawaii at Manoa
Honolulu, HI, USA
ISBN 978-3-319-75141-2 ISBN 978-3-319-75142-9 (eBook)
https://doi.org/10.1007/978-3-319-75142-9
Library of Congress Control Number: 2018935687
© The Editor(s) (if applicable) and The Author(s) 2018
This work is subject to copyright. All rights are solely and exclusively licensed by the
Publisher, whether the whole or part of the material is concerned, specifically the rights of
translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on
microfilms or in any other physical way, and transmission or information storage and retrieval,
electronic adaptation, computer software, or by similar or dissimilar methodology now
known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publication does not imply, even in the absence of a specific statement, that such names are
exempt from the relevant protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information
in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, express or implied, with respect to the
material contained herein or for any errors or omissions that may have been made. The
publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Cover illustration: Détail de la Tour Eiffel(c) nemesis2207/Fotolia.co.uk
Printed on acid-free paper
This Palgrave Macmillan imprint is published by the registered company Springer International
Publishing AG part of Springer Nature
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
Preface
This book was written to extend a previous work by the author and in
hopes of making a contribution to how we understand knowledge production—but particularly the production and role of impact evaluations—
in the field of global education policy. While the empirical case presented
here has been addressed in previous publications, for example, a journal
article in Education Policy Analysis Archives in 2016 with Claudia Loucel,
those publications did not sufficiently address theoretical or methodological issues. This book thus seeks to fill these gaps by spelling out more fully
(a) what it means to view knowledge production in the field of global
education policy from a political economy perspective, (b) what it means
to critically review impact evaluations, (c) what it means to place a critical
review of impact evaluations within the political economy of global education reform, and (d) what methods can be used for carrying out such a
study. The goal is to help shed light on knowledge production methods
that are often seen as objective and neutral but which in reality contribute
to serving certain interests while marginalizing others.
v
Contents
1A Political Economy Perspective on Knowledge Production 1
International Organizations, Knowledge Production, and Global
Education Policy 2
The World Bank and Knowledge Production 5
Impact Evaluations and Policy-Relevant Knowledge 8
Purpose, Argument, and Contribution 10
The Case of EDUCO 12
Introduction to EDUCO 12
EDUCO’s Design 13
EDUCO’s Fate 14
Relationship to Previous Work 15
References 17
2Critically Understanding Impact Evaluations: Technical,
Methodological, Organizational, and Political Issues 23
Impact Evaluation Through Regression Analysis 25
Impact Evaluation Through Randomized Control Trials 30
Differences in Control and Treatment Groups 32
Unbiasedness Does Not Equal True Impact 34
Mean Treatment Effects Mask Variance 35
Lack of Generalizability 36
Technical Considerations 42
Hypothesis Testing 42
P Values and the Normal Curve 46
vii
viii
Contents
Reporting and Interpretation of Effects 50
Impact Evaluations and Organizational and Political Incentives 52
Organizational and Political Incentives 52
Cost 54
Data Mining 55
Conclusion 57
References 62
3Situating a Critical Review of Impact Evaluations Within
the Political Economy of Global Education Reform:
Definition and Method 69
Definition 70
Methods: Putting Knowledge in Its Place 74
Data Collection 75
Data Analysis 83
References 85
4The Case of EDUCO: Political-Economic Constraints
and Organizational Dynamics 89
Structural Transformation During a Civil War 90
Organizational Dynamics and EDUCO’s Emergence 92
References 98
5Impact Evaluations of EDUCO: A Critical Review 101
Study 1. World Bank. (1994). El Salvador: Community
Education Strategy: Decentralized School Management.
Washington, DC: The World Bank 103
Study 2. Umanzor et al. (1997). El Salvador’s EDUCO
Program: A First Report on Parents’ Participation in SchoolBased Management. Working Paper Series on Impact
Evaluation of Education Reforms, No. 4. Washington, DC:
World Bank 108
Study 3. Jimenez, E., & Sawada, Y. (1999). Do CommunityManaged Schools Work? An Evaluation of El Salvador’s
EDUCO program. The World Bank Economic Review, 13
(3), 415–441 110
Study 4. Sawada, Y. (2000). Community Participation, Teacher
Effort, and Educational Outcome: The Case of El Salvador’s
Contents ix
EDUCO Program. Michigan, USA: The William Davidson
Institute, University of Michigan Business School 112
Study 5. Jimenez, E., & Sawada, Y. (2003). Does Community
Management Help Keep Kids in Schools? Evidence Using Panel
Data from El Salvador’s EDUCO Program. Discussion Paper,
Center for International Research on the Japanese Economy.
Tokyo, Japan: University of Tokyo 114
Study 6. Sawada, Y., & Ragatz, A. (2005). Decentralization
of Education, Teacher Behavior, and Outcomes: The Case of El
Salvador’s EDUCO Program. In E. Vegas (Ed.), Incentive
to Improve Teaching: Lessons from Latin America.
(pp. 255–306). Washington, DC: World Bank 115
References 120
6Reconsidering the EDUCO Program and the Influence
of Its Impact Evaluations 123
What Can We Say About EDUCO? 124
Implications: National and International 127
National Implications 128
International Implications 129
References 134
7Impact Evaluations: Persistent Limitations, Alternative
Approaches, Possible Responses 139
The Book in Retrospect 140
Persistent Limitation 1: The Nature of Econometric Methods 141
Alternative Approaches 142
Persistent Limitation 2: The Political-Financial-­Intellectual
Complex 143
Possible Responses 145
Conclusion 146
References 147
ppendix 1: Alphabetic list and characterization of evaluative
A
EDUCO studies 151
Index 155
List of Abbreviations
ACE
ARENA
EDUCO
FEPADE
FMLN
FUSADES
MINED
NGO
OLS
RCT
SDU
UNESCO
UNICEF
USAID
Community Education Association (Asociación Comunal
Educativa)
Republican National Alliance (Allianza Republicana Nacional)
Education with Community Participation (Educación con
Participación de la Comunidad)
Business Fund for Educational Development
Farabundo Martí National Liberation Front (Frente Farabundo
Martí de Liberación Nacional)
Salvadoran Foundation for Economic and Social Development
Ministry of Education
Non-governmental organization
Ordinary least squares
Randomized control trial
Standard deviation unit
United Nations Education, Science, and Culture Organization
United Nations Children’s Fund
United States Agency for International Development
xi
List of Tables
Table 3.1
Table 3.2
Table 3.3
Table 3.4
Table 3.5
Table 3.6
Table 5.1
Goals of a critical review of impact evaluations within the
political economy of global education reform
71
Tasks of critical review of impact evaluations within the
political economy of global education reform
74
Summary of data collection phases, critical review periods, and
corresponding analytic tasks
76
Characterization of overall sample of evaluative EDUCO
literature80
Chronological listing of impact evaluations of EDUCO
82
Summary of analytic methods
84
Summary of the findings and limitations of the World Bank’s
Evaluative Studies of the EDUCO Program
104
xiii
CHAPTER 1
A Political Economy Perspective
on Knowledge Production
Abstract This chapter delineates what it means to bring a political economy lens to the issue of knowledge production within the field of global
education policy. In addition to characterizing this perspective generally,
this chapter addresses knowledge production in relation to the World Bank,
one of the most influential knowledge-producing organizations in this field
and the organization at the center of the empirical case presented in this
book. Both the material and ideational dimensions of the World Bank’s
influence are discussed. Beyond these issues, this chapter also (a) defines
and characterizes impact evaluation; (b) discusses the purpose, argument,
and contribution of this book; (c) provides background information on the
policy case from El Salvador that serves as the book’s empirical basis; and
(d) situates this book in relation to previous work by the author.
Keywords International organizations • Political economy • Impact
evaluations • World Bank • Global education policy • Knowledge
production
Since World War II data collection, research, and
recommendations for policy in education have depended,
to a significant extent, on international organizations.
These agencies have had the resources, scope, and sometimes the vision
to collect data on a large scale and to set education policy.
(Altbach, 1988, p. 137)
© The Author(s) 2018
D. B. Edwards Jr., Global Education Policy, Impact Evaluations, and
Alternatives, https://doi.org/10.1007/978-3-319-75142-9_1
1
2
D. B. EDWARDS JR.
[There is] a new political economy of knowledge production and use in
educational coalitions. In this environment, there are advantages for
organizations that can more effectively package and promote research,
can orchestrate a concerted effort to convey a consistent message through
multiple media outlets, and can place it in front of key people in the
policy pipeline. [Intermediary organizations] have demonstrated a
notable ability to succeed in this climate.
(DeBray, Scott, Lubienski, & Jabbar, 2014, p. 179)
International Organizations, Knowledge
Production, and Global Education Policy
It has long been acknowledged that international organizations are influential when it comes to the reform of education policy around the world
(Altbach, 1988; Berman, 1979, 1992).1 More recently, scholarship has
highlighted that many of these organizations, beyond having the resources
to collect data on a large scale, also use those resources to produce attractive knowledge products, to widely disseminate them, and even to deliver
them directly to policymakers (DeBray et al., 2014; Verger, Edwards, &
Kosar-Altinyelken, 2014). Importantly, what these observations underscore is that international organizations—or any intermediary organization
that produces knowledge about policies and practice to inform decisionmaking—both derive and generate their influence in material and ideational
terms. That is, the power of international organizations comes, on one
hand, from access to significant financial and organizational resources and,
on the other hand, from the ability to strategically employ those resources
to promote certain ideas or policies and to shape the conversation around
what kinds of reform are seen as desirable within the global education policy field (Barnett & Finnemore, 2005; Jakobi, 2009).
What the above comments presuppose is the combination of a few key
assumptions that should be clarified because they are fundamental to the
way that a political economy perspective understands the intersection of
international organizations, knowledge production, and the field of global
education policy.2 That is, the opening comments and the remainder of
this book depart from a number of precepts that should be stated
­unambiguously because they serve as the conceptual point of departure
for the methodological approach and the particular findings offered here.
The first assumption is that each international organization is animated by
A POLITICAL ECONOMY PERSPECTIVE ON KNOWLEDGE PRODUCTION
3
certain ideas, policies, or values more so than by others, with these ideas,
related to the mission of the organization and to the preferences of those
in leadership positions (Allison & Zelikow, 1999; Haas, 1990). Second,
international organizations, like all organizations, scan their environments
and look for ways to ensure stability (Malen & Knapp, 1997). Third, for
organizations that rely to a greater or lesser extent on the ability to sell,
mobilize, or produce ideas or knowledge products (e.g., research or other
analytic work), stability results from the perceived relevance of those ideas
(as judged by others) and the need for those knowledge products within
the larger political context (Porter, 1995). This dual orientation toward
organizational survival and political salience makes sense when one considers that international organizations—which range from non-­
governmental organizations (NGOs), to philanthropic foundations, to
think tanks, to corporate entities, to bilateral agencies and to multilateral
institutions—are not simply reactive features of the global education policy field but are also political actors in their own right. They are actors that
operate on one or more levels from the global to the local to influence a
range of ideational issues, such as which problems, policies, programs, and
so on, are seen as warranting attention, with the goal being to preserve
organizational longevity into the future by demonstrating the relevance of
the organization to those problems, policies, programs, and so on (Altbach,
1988; Ball, 2012; Berman, 1992; Salas-Porras & Murray, 2017). These
three assumptions thus highlight the interdependence among the political, organizational, and ideational factors that international organizations
must manage, to the extent possible, to survive in the world of global
education reform.3
The issues raised above are at the heart of the political economy perspective on the connections among international organizations, knowledge production, and the politics of global education policy. However, in
operating from a political economy perspective, it is important to further
draw attention to a fourth factor, namely, the variegated nature of the field
of global education policy. This factor highlights the fact that the political
economy perspective analyzes the dynamics highlighted above in relation
to (a) the uneven topography of this field of activity as states respond differently to the pressures of globalization (Lingard & Rawolle, 2011) and
to (b) the competition among actors that occurs as organizations strive for
increased legitimacy and influence (Edwards, Okitsu, da Costa, &
Kitamura, 2018; Mundy & Ghali, 2009). Additionally, and fundamentally,
the political economy perspective is sensitive to (c) the way that the field
4
D. B. EDWARDS JR.
of global education policy is overlaid on, is intertwined with, and must
respond to other structural considerations that often mediate the relationship between global educational actors and opportunities at the national
level through which these actors can influence educational policies and
programs (Hay, 2002). These structural considerations can relate to supranational constraints (as with World Trade Organization regulations),
regional bodies and accords, free trade agreements among countries,
intercountry political relationships, or intra-country political dynamics.
When it comes to the ways that the global education policy field interacts
with larger structural considerations, the point is that, first, the relationship between international organizations and national actors is shaped by
a variety of circumstances and, second, that we cannot think of this relationship between global and national actors in isolation, separate from the
kinds of structural issues mentioned above.
Taking the aforementioned assumptions as a starting point, the task at
hand in this book is to contribute to how we understand and investigate
the role and influence of knowledge production by international organizations within the field of global education reform. This opening section has
taken a first step in that direction by spelling out what it means to bring a
political economy lens to this issue. However, as the title of this book
indicates, the interest here is to go beyond a general focus on knowledge
production to examine a particular kind of knowledge production, that is,
the production of impact evaluations (discussed further below). Moreover,
as will become clear, the end goal is not only to unpack the methodological, technical, political, and organizational challenges in the production of
impact evaluations (as discussed in Chap. 2) but also to detail an approach
to critically understanding and examining the role that impact evaluations,
once produced, play within the political economy of global education
reform (discussed in Chap. 3). The final two goals are to demonstrate the
application of this approach in relation to a global education policy from
El Salvador (Chaps. 5, 6) and to reflect on the implications of this case for
ways forward, methodologically and otherwise (see Chap. 7).
Before moving on to focus on these goals, the present chapter sets the
stage by addressing a number of key issues. Due to its centrality in the
policy case from El Salvador, the first section characterizes the role of the
World Bank within the field of global education in relation to knowledge
production. The second section below defines and characterizes impact
evaluations. Subsequent sections (a) discuss the purpose, argument, and
contribution of this book, (b) provide background information on the
A POLITICAL ECONOMY PERSPECTIVE ON KNOWLEDGE PRODUCTION
5
policy case from El Salvador, and (c) situate this book in relation to previous work by the author.
The World Bank and Knowledge Production
As noted above, there are many kinds of international organizations operating in the realm of education reform globally. However, in the post-­
WWII period, when it comes to impact on education reform around the
world, arguably the most influential organizations have been multilateral
institutions, including the United Nations agencies and multilateral development banks. Though the organizations in this group often compete
among themselves for influence (Edwards et al., 2018), the institution
with the most sway from the 1970s to the 2010s has been the World Bank
(or simply “the Bank”) (Mundy, 1998; Mundy & Verger, 2015).4
As the primary institution of interest in this book (due to its centrality
in the case of focus), it is helpful to briefly characterize the World Bank.
Consider, for example, that, by the 1970s, the World Bank was not only
the “largest single provider of finance for educational development”
(Mundy, 1998, pp. 466–467) but also that, by 1970, World Bank lending
to education, at $409 million, already outstripped the total regular budgets of both the United Nations Education, Science, and Culture
Organization (UNESCO, $355 million) and the United Nations
Children’s Fund (UNICEF, $51million) combined (Mundy, 1998,
p. 459). By 1995, education sector lending from the World Bank reached
$2 billion (Mundy, 1998, p. 467), a with lending figures still at this level
in 2012 (Mundy & Verger, 2015) (having peaked at approximately $4 billion in 2004, World Bank/IEG, 2006).5
Importantly, though, World Bank influence stems from more than just
its lending abilities and the ability to make loan disbursement conditional
on stipulated educational reforms. As “the largest development research
institution in the world” (Gilbert & Vines, 2006, p. 49), it also produces
hundreds of knowledge products each year, including reports, academic
articles and books, impact evaluations, working papers, seminar proceedings, and policy briefs. Indeed, between 1998 and 2005 alone, the Bank
“undertook 705 research projects and published 3635 research publications in English” (Dethier, 2007, p. 471). When it comes to research on
education specifically, the Bank has been similarly prolific. In a 2011 publication, the World Bank boasts of its contribution to the “global knowledge base” on education through the production of approximately 500
6
D. B. EDWARDS JR.
journal articles and 500 books, book chapters, and working papers (p. 52;
see also Zapp, 2017). It furthermore claims that “only Harvard University
comes close” to matching its contribution to the economics of education
(p. 53). Of course, it needs to be remembered that the material and ideational aspects of influence go hand in hand, for during 2001–2010 alone
the education sector at the World Bank spent $49 million dollars on
research to produce “about 280 pieces of research and other analytical
work” (p. 52).
Importantly, the above characterization does not touch on the many
other ways that the World Bank collects data on education around the
world, such as through its recent initiative known as SABER or the Systems
Approach for Better Education Results. Through this initiative, the World
Bank seeks “to collect and analyze information on policies and identify
actionable priorities for strengthening education systems” (World Bank,
2013, p. 4), with the goal being to “make it possible for stakeholders to
obtain simple, objective, up-to-date snapshots of how their system is functioning, how well it is performing, and what concretely the system can do
to achieve measurably better results” (World Bank, 2011, p. 61). By collecting information on numerous aspects of education systems around the
world, the World Bank endeavors to create a “global knowledge base on
education policies and institutions” in order to provide evidence-based
guidance to countries on education reform (World Bank, 2013, p. 4).
One implication of the fact that the World Bank acts as “knowledge
bank”—by collecting data and producing research on such a scale (Samoff
& Stromquist, 2001; Zapp, 2017)—is that it can influence which reforms
are seen as legitimate and desirable, through its ability to elevate and promote its research in strategic ways within the global governance of education. Verger (2012) has shown how Bank staff do this in reference to the
policy of public-private partnerships, while Stone (2002) has shown how
the World Bank has strategically contributed to the Global Development
Network (a transnational non-governmental organization that focuses on
creating, sharing, and applying international development knowledge) to
promote research on open economies and free markets. Interestingly,
recent research has also shown that, as a result of the ubiquity, credibility,
and perceived usefulness of World Bank research, actors in the field of
global education policy have come to rely on it, to the point where they
look for it and feel uncomfortable without it, even in spaces of consultation designed to be open to other perspectives on education governance
and reform (Verger, Edwards, & Kosar-Altinyelken, 2014).
A POLITICAL ECONOMY PERSPECTIVE ON KNOWLEDGE PRODUCTION
7
Of course, an important aspect of the production of knowledge for
development (like other forms of knowledge) is that it is not free from bias
(Cox, 2002) and that its valuation is not free from the prevailing norms or
standards of quality (Walters, Lareau, & Ranis, 2009). Indeed, the production of knowledge suffers from bias and political considerations in
many ways and on many levels—for example, in terms of how data are collected (including which categories and terms to use in labeling data) and
subsequently interpreted (since data never speak for themselves but are
always viewed through some lens; Porter, 1995). Within organizations,
although they are contested sites where multiple perspectives may vie for
supremacy, and as such may not have a consistent message or ideology
(though this is unlikely or uncommon, particularly for organizations that
successfully brand themselves as reflecting certain perspectives), the fact
remains that organizations which traffic in knowledge production must
take a stance on those issues about which they produce knowledge (even
if their stance is not to take a stance). The positions or approaches advocated by an organization will tend to reflect the dominant perspective in
that organization or will otherwise reflect a perspective that is acceptable
or non-threatening to that organization’s survival (Allison & Zelikow,
1999; Malen & Knapp, 1997; Malin & Lubienski, 2015).
Moreover, in terms of prevailing norms around research standards, to
the extent that an organization seeks to be taken seriously, it will conform
with the expectations related to quality. For decades, the standard for
quality, rigor, and relevancy in policy research has been quantitative forms
of evaluation (Klees & Edwards, 2014). The specific form of quantitative
analysis that passes as most rigorous has changed over time and has become
more sophisticated in its search for the ability to identify the true effect of
policies and programs (see Chap. 2 for more). Nevertheless, the issue
remains that there are prevailing expectations around methods for knowledge production, and organizations that desire to participate in policy
debates or to influence policy trends will employ those methods that will
help them gain or retain credibility (Lubienski, Scott, & DeBray, 2014).
Scholars have addressed the issue of bias in relation to the World Bank,
pointing out that this institution, at its core, is guided by a neoliberal
worldview and that the research it produces reflects that worldview by
promoting policies and reforms based in market or quasi-market principles
(Broad, 2006; Klees & Edwards, 2014; Lauglo, 1996; Rao & Woolcock,
2007). That is, while the World Bank frames its findings and the policies it
recommends as the product of scientific inquiry guided by the highest
8
D. B. EDWARDS JR.
standards of social science (see, e.g., Khandker, Kookwal, & Samad, 2010;
Verger, 2012), these claims are undercut by the fact that the recommendations offered consistently favor neoliberal reforms based in market logic,
not to mention the fact that research findings and alternative interpretations which do not accord with the dominant neoliberal perspective within
the Bank are suppressed (Broad, 2006; Wade, 1996, 2001). In the realm
of education reform, the World Bank has, over the years, repeatedly
emphasized and/or funded policies geared toward school fees, school
competition, parental choice, accountability, and, more recently, low-fee
private schools, to mention a few examples.6
In order to sustain its business model, the World Bank, like other international development banks, must be able to demonstrate that its policies
are successful. Consequently, the World Bank not only promotes reforms
geared toward quality and efficiency that are generally accepted as appropriate—or for which it seeks to gain acceptance—but this organization
also conducts its own research on promising pilot programs or on-going
reforms, even if it is also the World Bank that is financing those examples,
despite the conflict of interest. If it can be shown through evaluations that
a particular policy causes positive outcomes in terms of student achievement or student dropout, for example, then that policy will, on the basis
of those evaluations, be promoted globally through World Bank publications and will be included in the short list of acceptable reforms that the
World Bank is willing to support in other countries (Heyneman, 2003;
Steiner-Khamsi, 2012). This state of affairs leads to the observation that
the World Bank produces “policy-based evidence” as opposed to
“evidence-­based policy.” The way this happens is a primary focus of the
present manuscript. Moreover, the World Bank’s preferred short list of
policies is an important aspect of global education policy because the
World Bank not only exercises some degree of policy influence in its
engagement with borrowing countries but also influences peer institutions
and bilaterial aid agencies as well (Edwards et al., 2018; Verger, Edwards,
& Kosar-Altinyelken, 2014).
Impact Evaluations and Policy-Relevant Knowledge
The development of impact evaluations is a form of knowledge production which seeks to determine the effects of an activity on certain outcomes. While many international organizations produce research and
publications, many fewer produce impact evaluations. In part, this is
A POLITICAL ECONOMY PERSPECTIVE ON KNOWLEDGE PRODUCTION
9
because impact evaluations require specialized knowledge, are difficult to
implement, and can be very costly. At the same time, however, they have
come to be seen by many as the best way to inform policy and practice,
which makes them highly desirable. Indeed, as Gertler, Martinez,
Premand, Rawlings, and Vermeersch (2016) write in their guidebook produced for the World Bank on doing impact evaluations in practice:
The robust evidence generated by impact evaluations is increasingly serving
as a foundation for greater accountability, innovation, and learning. In a
context in which policy makers and civil society are demanding results and
accountability from public programs, impact evaluation can provide robust
and credible evidence on performance and, crucially, on whether a particular
program has achieved or is achieving its desired outcomes. Impact evaluations are also increasingly being used to test innovations in program design
or service delivery. At the global level, impact evaluations are central to building knowledge about the effectiveness of development programs by illuminating
what does and does not work to reduce poverty and improve welfare. (p. 4,
emphasis added)
Thus, the promises and expectations around impact evaluations are high,
and it is precisely because of their potential for influencing policy and
global reform trends that they warrant critical attention.
Further examples of the prominence of impact evaluations can be found
in the support that they receive from international actors. Here, Vivalt
(2015) notes, first, that the UK aid agency (the Department for
International Development) has given a major grant to the World Bank to
expand its “already large impact evaluation works,” second, that the
“Millennium Challenge Corporation has committed to conduct rigorous
impact evaluations for 50% of its activities,” and, third, that the US Agency
for International Development (USAID) “is also increasingly invested in
impact evaluations” (p. 3).7 More tangibly, USAID’s interest in evaluation
is demonstrated, generally, by a 2011 policy that directs 3% of its programmatic funding to evaluation and, specifically, by the fact that “country
mission leaders are supposed to identify at least one opportunity for impact
evaluation for each development objective in their 3–5 year plans” (p. 3).
Of course, this support from international organizations is in addition to
the rising tide of impact evaluations that are being produced by researchers. Whereas as a review of three separate impact evaluation databases
shows that, at most, a couple were produced each year up through the
10
D. B. EDWARDS JR.
early 1990s, this figure skyrocketed to well over 300 by 2010 (Vivalt,
2015).
More depth will be provided on impact evaluations in Chap. 2. For
now, however, it should simply be mentioned that impact evaluations can
be carried out via a number of quantitative methods, all of which seek to
understand the results of an intervention, such as a policy or program, in
terms of specific outcomes. Their distinguishing characteristic, that is, the
characteristic that sets them apart from other forms of research, is their
claim to attribution. Again, in the words of Gertler et al. (2016): “an
impact evaluation assesses the changes in the well-being of individuals that
can be attributed to a particular project, program, or policy. This focus on
attribution is the hallmark of impact evaluations” (p. 4). Put differently,
impact evaluations measure the effects of an intervention, and they do this
typically by comparing the results of the group that participated in the
intervention against the results of a group that did not. In the realm of
education, impact evaluations commonly examine effects on test scores or
student retention in relation to new policies or program changes.
Purpose, Argument, and Contribution
With the issues of the previous sections in mind, the question becomes
how to conceptualize, investigate, and interpret the role of knowledge
production and mobilization by international organizations in relation to
global education reform, with a focus on what many consider to be the
most influential form of development knowledge—impact evaluations. As
briefly mentioned in the introductory section of this chapter, this overarching focus can be broken down into multiple goals. A first goal is to
present an approach to knowledge production by international organizations that entails not only carrying out a critical review of the evidence
produced but also placing that evidence into the political economic context out of which it emerged and interpreting its significance in relation to
the dynamics of global education reform at multiple levels. The second
goal is to delineate methods that can be employed to realize this approach.
Third, this short book seeks to demonstrate this approach by applying it
to the knowledge that was produced around a policy that emerged from
El Salvador in the 1990s and went global (more on this policy below). The
final goal is to reflect on the limitations of impact evaluations and to consider alternatives as well as ways forward for addressing the political economy of knowledge production in global education reform.
A POLITICAL ECONOMY PERSPECTIVE ON KNOWLEDGE PRODUCTION
11
As for argument, while this book focuses on the World Bank, it is suggested that the approach, methods, and lessons offered here are applicable
more generally, that is, beyond the case of the World Bank alone. For
example, while the World Bank and other lending institutions literally
depend on their ability to sell policies, the same is true of other organizations operating in the world of public policy, international development,
and global education policy. Indeed, think tanks, research centers, and
many other NGOs depend on their ability to “sell”—that is, to have
accepted as legitimate—their policy reports, research products, and programmatic offerings (with the latter often legitimated by research findings
produced by one or another of the same kind of organization) (Rich,
2011). The point is that the dynamics which animate, enable, and constrain the World Bank are also common to many other international organizations, as described in the first section of this chapter.
In terms of contribution, it is uncommon to question the wisdom of
impact evaluations, let alone to take the additional step of reexamining
their meaning within the context of institutional and political-economic
dynamics. Indeed, within the field of comparative and international education, impact evaluations are rarely even published. Evidence of this is the
fact that, between 1964 and 2011, three of the leading journals in this
field (namely, Comparative Education Review, Comparative Education,
and the International Journal for Educational Development) did not publish any studies that employed randomized control trials (a form of impact
evaluation; see Burde, 2012; Rust, Soumare, Pescador, & Shibuya, 1999).
Of course, the fact that scholars of comparative and international education tend not to produce impact evaluations does not mean that such
evaluations are inconsequential. As the section above on impact evaluations indicates, impact evaluations are increasingly prevalent; the implication is simply that they are being produced by researchers from other
fields, such as economics and health. Yet it is crucial that education scholars grasp the uses and limitations of impact evaluations, else they risk being
hostage to researchers from other fields who can and do conduct impact
evaluations in the realm of education.
This book responds to this situation, in Chap. 2, by delineating how
and why impact evaluations should be critically understood, so that education scholars can be informed consumers of such evaluations. Second, this
book shares, in Chap. 3, methods that are geared to enable a critical
understanding of the role that impact evaluations, once produced, play in
the field of global education policy more generally. The idea is to bring a
12
D. B. EDWARDS JR.
critical understanding not only to the production of impact evaluations
themselves but also to the way they are used subsequently. Furthermore,
by applying these methods to a particular policy example, the present
manuscript shows the insights that can be generated through both a reappraisal of a policy’s knowledge base and additional reflection on the political role of that knowledge as well as the implications of the combination
of the former (critical review of knowledge) with the latter (political role
of knowledge). To the extent that the book is able to help unpack, respond
to, and reconsider some of the complex and problematic issues that afflict
the instrumentalization of knowledge production in the field of global
education policy, it is hoped that it has made a contribution which will be
seen as valuable.
The Case of EDUCO
The issues raised in this book are explored through the case of a policy
from El Salvador and the impact evaluations that were produced to examine its effectiveness. This section briefly introduces that policy and its
essential features. As will become clear over the course of this manuscript,
this policy was chosen for analysis because the politics of knowledge production have been central to its evolution from an unknown program to a
reform model that has been circulated around the world.
Introduction to EDUCO
During the 1990s and 2000s, a policy known as Education with
Community Participation (EDUCO) not only became the cornerstone of
education reform in El Salvador but also became a global education policy
(Edwards, 2015, 2018), in that it circulated around the world in the
spaces where the international agenda for education reform is created and
communicated (Verger, Novelli, & Kosar-Altinyelken, forthcoming).
Indeed, since the mid-1990s, this policy, which decentralized the responsibility for hiring and firing teachers to parents in rural communities, has
been studied by technical groups from at least 18 countries (Meza,
Guzmán, & de Varela, 2004),8 in addition to being promoted by numerous international organizations through high-profile publications, including some of the most influential reports in the realm of international
development (Edwards, 2018). A select few examples are:
A POLITICAL ECONOMY PERSPECTIVE ON KNOWLEDGE PRODUCTION
13
• The World Bank’s World Development Report 2004, which focused
on making public services more accountable (World Bank, 2003)
• The report in 2005 of the United Nations task force on gender
equity (Birdsall, Levine, & Ibrahim, 2005)
• A publication by the Brookings Institution (an influential think tank
in Washington DC) in 2008 on effective development practices
(Hartman & Linn, 2008)
• The Global Monitoring Reports of the United Nations Education,
Science, and Culture Organization (UNESCO) in both 2009 and
2011, which addressed education governance and education in
conflict-­affected contexts (UNESCO, 2008, 2011)
While many political and economic factors influenced its emergence
and subsequent trajectory (Edwards, 2015, 2018), one important issue
that previous studies of this program have tended not to examine is the
central role played by impact evaluations. The present manuscript fills this
gap in the literature. It does this, first, by presenting a critical review of the
evidence base that has been used to promote EDUCO and, then, by discussing the meaning of that evidence—in light of the reappraisal presented
here—within the political-economic context that both produced EDUCO
and produced the impact evaluations that legitimated its dissemination
and promotion globally.
EDUCO’s Design9
Later chapters of this book focus on EDUCO’s politics and knowledge base.
While commentary is provided throughout the book as necessary on the nuts
and bolts of this program, the present section offers a condensed account of
EDUCO’s programmatic details, with the intention being to clarify the program’s core features before shifting focus to its other aspects. In short, the
EDUCO program was theorized to function in the following way and to
have the following impact. First, the education system would be more efficient if the central MINED transferred to the community level the responsibility for hiring and firing teachers, the latter of whom would work on
one-year contracts that were renewable at the discretion of the school council
that hired them. These school councils, which were known as community
education associations (or ACEs, for their name in Spanish), were made up
of five elected, volunteer parents from the community. (Note that the schools
managed by ACEs tended to be small schools without principals, though one
14
D. B. EDWARDS JR.
of the teachers would by default assume some of the administrative duties
typically associated with the principalship.) Not only was this community
arrangement seen as inherently more efficient on a system-wide basis
(since parents worked on a volunteer basis and since ACEs would mobilize
parents to build and maintain the school without compensation), but, in
addition, this arrangement was seen as more efficient because the ACEs
would manage the school’s budget and would, as such, be responsible for
purchasing only the educational materials that the school needed, thereby
eliminating waste in purchasing. Moreover, it was thought that this
arrangement would promote effectiveness because it would lead to more
consistent teacher attendance and improved student test scores, a consequence of the fact that teachers were under the scrutiny of community
actors to perform well or else lose their job. The MINED, on the other
hand, was responsible for the following: facilitating the creation of the
ACEs, training the members of the ACEs in administrative and accounting procedures, setting the minimum criteria for teacher selection by the
ACEs, designing and providing curricula, and overall coordination, supervision, and monitoring of the program as it was scaled up (Reimers, 1997;
World Bank, 1994).
EDUCO’s Fate
While the focus of this book is on the production and politics of knowledge, it should be mentioned that the EDUCO program is no longer in
operation in El Salvador. At the national level, while commitment to
EDUCO waxed and waned somewhat over the life of the program,
there was no doubt that it was a core feature of the educational preferences of the right-wing party that held power from 1989 to 2009,
meaning that the program was never in jeopardy of being canceled during that time. However, in 2009 the first left-wing president to be
elected in the postwar period began the process of undoing EDUCO
and converting the schools and teachers to the traditional public school
system (see Edwards, 2018, or Edwards, Martin, & Flores, forthcoming, for more). Though EDUCO would be effectively undone by 2011,
the program had a significant period of stability (1991–2011) and,
indeed, not only continues to live on in the literature on CBM but also
continues to be cited and to be revered as a strong form of community
participation (Edwards, 2018).
A POLITICAL ECONOMY PERSPECTIVE ON KNOWLEDGE PRODUCTION
15
Relationship to Previous Work
A few comments on how this book relates to and is distinct from my previous work on EDUCO and global education policy might be helpful, particularly in light of the fact that a series of publications have come out in
recent years on the intersection of these two issues. A comparison with
previous work shows that this manuscript has a specific purpose in drawing
out issues that have been broached briefly or tangentially in other works.
However, it is first necessary to briefly characterize that previous work.
I have developed three strands of work on the intersection of EDUCO
and global education policy over the past five years. Respectively, these
strands have focused on:
1. The overall trajectory of EDUCO, from its origins to its status as a
global education policy (Edwards, 2015, forthcoming-b; Edwards,
Victoria, & Martin, 2014, 2015), culminating with a recent book
(Edwards, 2018)
2. A critical analysis of the impact evaluations that have been performed
by the World Bank on EDUCO (Edwards, 2016; Edwards &
Loucel, 2016; Edwards & Pagés, 2016)
3. The implementation and local level experience of EDUCO
(Edwards, forthcoming-a; Edwards & Klees, 2012, 2014), including resistance to it at the community level (Edwards & Ávalos,
2015)
The present manuscript complements the first strand while building on
the second.
While the publications in the first strand have focused in depth on the
structural conditions, processes, and mechanisms by which EDUCO
emerged and then went global as a policy promoted by the World Bank,
they have not honed in on the role of impact evaluations and knowledge
production in the way that this book seeks to do. That is, while strand one
studies have encompassed the role of impact evaluations and knowledge
production, the primary focus has been on explaining the overall
­political-­economic dynamics that enabled EDUCO’s genesis in El Salvador
as well as the transnational pathways of influence through which EDUCO
was disseminated and leveraged beyond El Salvador.
In contrast, strand two studies have been dedicated to critiquing
EDUCO’s impact evaluations and to understanding them in relation to
16
D. B. EDWARDS JR.
EDUCO’s trajectory. However, what these studies have failed to do is
provide a conceptual framework for understanding how impact evaluations and knowledge production fit into the political economy of global
education reform more generally. Strand two work has also lacked a full
discussion of how and why impact evaluations should themselves be critiqued and critically interpreted.
Thus, in the chapters that follow, this book builds on these two
strands—but particularly Edwards and Loucel (2016). It does so, in Chap.
2, by taking on impact evaluations, that is, by addressing directly their
conceptual and technical limitations, not to mention the ways that they
necessarily intersect with political economy issues. Subsequently, in Chap.
3, this book further develops the strands mentioned above, first, by explicitly defining what is meant by a critical review of impact evaluations within
the political economy of global education reform and, second, by conceptually depicting the various spaces or levels out of which impact evaluations emerge and through which they travel as they impact the global
education policy field. In carrying out these tasks, Chap. 3 departs from
the discussion at the outset of Chap. 1 that anchors this book, that is, the
discussion of what is meant by a political-economic perspective on the
intersection of international organizations, knowledge production, and
global education policy. Finally, unlike previous studies on EDUCO,
Chap. 3 extensively details the methodological considerations and processes that undergirded the analysis of EDUCO’s impact evaluations and
their reinterpretation in subsequent chapters.
Notes
1. Portions of this section draw on Edwards and Loucel (2016).
2. For further discussion and description of the global education policy field,
see Edwards (2018) and Jakobi (2009).
3. For a few examples of studies on individual institutions that address these
dimensions, see Lingard and Sellar (2016), Mundy and Menashy (2014),
Mundy and Murphy (2001), Menashy and Manion (2016), Mundy and
Verger (2015), and Rambla, Verger, Edwards, Bonal, and Fontdevila
(2017).
4. See Edwards and Storen (2017) for an overview of the history and research
on the World Bank’s work in educational assistance.
5. For the period 1990–2005, Jones (2006, p. 95) reports that the World Bank
issued $26 billion in education loans, with $14 billion coming from regular
A POLITICAL ECONOMY PERSPECTIVE ON KNOWLEDGE PRODUCTION
17
lending channels and $12 billion being issued through at lower interest rates
for low-income countries.
6. Importantly, though, recent research has highlighted how the World Bank
is a complex organization that has evolved over time, with the implication
being that there is not always a 100% correspondence between the rhetoric
and recommendations of the headquarters, on one hand, and the lending
practices of the country-level staff, on the other (see, e.g., Mundy &
Menashy, 2014; Fontdevila & Verger, 2015).
7. The Millennium Challenge Corporation is a bilateral agency of the US government that is separate from the US Agency for International Development
that selects countries for funding on the basis of performance on various
policy indicators, among other factors.
8. The list of countries includes at least Guatemala, Honduras, Colombia,
Chile, Nicaragua, Panama, Dominican Republic, Mexico, Brazil, Paraguay,
Bolivia, Ghana, Mauritius Islands, Madagascar, Chad, Morocco, Senegal,
and Mauritania (Meza, Guzmán, & de Varela, 2004).
9. This section draws on Edwards (2016).
References
Allison, G., & Zelikow, P. (1999). Essence of decision: Explaining the Cuban missile
crisis (2nd ed.). New York: Longman.
Altbach, P. (1988). International organizations, educational policy, and research:
A changing balance. Comparative Education Review, 32(2), 137–142.
Ball, S. J. (2012). Global education inc.: New policy networks and the neo-liberal
imaginary. New York: Routledge.
Barnett, M., & Finnemore, M. (2005). The power of liberal international organizations. In M. Barnett & R. Duval (Eds.), Power in global governance
(pp. 161–184). Cambridge: Cambridge University Press.
Berman, E. (1979). Foundations, United States foreign policy, and African education. Harvard Educational Review, 49(2), 145–179.
Berman, E. (1992). Donor agencies and third world educational development,
1945–1985. In R. Arnove, P. Altbach, & G. Kelly (Eds.), Emergent issues in
education: Comparative perspectives (pp. 57–74). Albany: SUNY.
Birdsall, N., Levine, R., & Ibrahim, A. (2005). Toward universal primary education: Investments, incentives, and institutions. UN Millennium Project, task force
on Education and Gender Equality. London: Earthscan.
Broad, R. (2006). Research, knowledge, and the art of “paradigm maintenance”:
The World Bank’s development economics Vice-Presidency (DEC). Review of
International Political Economy, 13, 387–419.
18
D. B. EDWARDS JR.
Burde, D. (2012). Assessing impact and bridging methodological divides:
Randomized trials in countries affected by conflict. Comparative Education
Review, 56(3), 448–473.
Cox, R. (2002). The political economy of a plural world: Critical reflections on
power, morals and civilization. New York: Routledge.
DeBray, E., Scott, J., Lubienski, C., & Jabbar, H. (2014). Intermediary organizations in charter school policy coalitions: Evidence from New Orleans.
Educational Policy, 28(2), 175–206.
Dethier, J. (2007). Producing knowledge for development: Research at the World
Bank. Global Governance, 13, 469–478.
Edwards, D. B., Jr. (2015). Rising from the ashes: How the global education policy of community-based management was born from El Salvador’s civil war.
Globalisation, Societies and Education, 13(3), 411–432.
Edwards, D. B., Jr. (2016). A perfect storm: The political economy of community-­
based management, teacher accountability, and impact evaluations in El
Salvador and the global reform agenda. In W. Smith (Ed.), Global testing culture: Shaping education policy, perceptions, and practice (pp. 25–42). Oxford
Studies in Comparative Education). Oxford: Symposium.
Edwards, D. B., Jr. (2018). The trajectory of global education policy: Community-­
based management in El Salvador and the global reform agenda. New York:
Palgrave Macmillan.
Edwards, D. B., Jr. (forthcoming-a). Accountability through community-based
management? Implications from the local level implementation in El Salvador
of a globally-popular model. In K. A. Heidemann & R. A. Clothey (Eds.),
Another way: Decentralization, democratization, and the global politics of
community-­based schooling. Rotterdam: Sense.
Edwards, D. B., Jr. (forthcoming-b). School- and community-based management
as global education policy: History, trajectory, geography. In A. Verger,
M. Novelli, & H. Kosar-Altinyelken (Eds.), Global education policy and international development: New agendas, issues and programmes (2nd ed.). New York:
Continuum.
Edwards Jr., D. B., & Ávalos, E. (2015). Santa Marta y el Programa EDUCO:
Una experiencia de Resistencia, adaptación e inversion de la lógica neoliberal.
[Resistance to, and adaptation of, the program for Education with Community
Participation in El Salvador: The experience of Santa Marta and its inversion of
neoliberal reform logic.] San Salvador, El Salvador: Universidad Centroamericana.
Edwards, D. B., Jr., & Klees, S. (2012). Participation in international development
and education governance. In A. Verger, M. Novelli, & H. Kosar-Altinyelken
(Eds.), Global education policy and international development: New agendas,
issues and programmes (pp. 55–77). New York: Continuum.
Edwards, D. B., Jr., & Klees, S. (2014). Participación en el desarrollo internacional y la gobernanza de educación: Tres perspectivas y tres casos de El Salvador.
A POLITICAL ECONOMY PERSPECTIVE ON KNOWLEDGE PRODUCTION
19
[Participation in international development and education governance: Three
perspectives and three cases from El Salvador.] Latin American Review of
Comparative Education, 5, 107–121. [In Spanish].
Edwards, D. B., Jr., & Loucel, C. (2016). The EDUCO Program, impact evaluations, and the political economy of global education reform. Education Policy
Analysis Archives, 24(49), 1–50.
Edwards, D. B., Jr., Martin, P., & Flores, I. (forthcoming). Education in El
Salvador: Past, present, and prospects. In C. M. Posner, C. Martin, & Y. Martin
(Eds.), Education in México, Central America and the Latin Caribbean.
New York: Continuum.
Edwards, D. B., Jr., Okitsu, T., da Costa, R., & Kitamura, Y. (2018). “Organizational
legitimacy in the global education policy field: Learning from UNESCO and
the Global Monitoring Report,” Comparative Education Review, 61(1), 31–63.
Edwards, D. B., Jr., & Pagés, M. (2016, May). Análisis crítico de las evaluaciones
de impacto dentro de la economía política de la reforma educativa global: El
caso del programa EDUCO. [A critical analysis of impact evaluations within
the political economy of global education reform: The case of EDUCO.]
Epítome Educativo, 1(mayo), 1–11.
Edwards, D. B., Jr., & Storen, I. (2017). The World Bank and educational assistance.
In G. Noblit (Ed.), Oxford research encyclopaedia of education. Oxford University
Press. Retrieved from http://education.oxfordre.com/view/10.1093/acrefore/9780190264093.001.0001/acrefore-9780190264093-e-43
Edwards, D. B., Jr., Victoria, J. A., & Martin, P. (2014). Corrientes internacionales, desarrollos estructurales, preferencias nacionales y la implementación de
políticas educativas: Hallazgos en El Salvador durante el period 1990–2005.
[International trends, structural developments, national preferences and the
implementation of education policies: Findings from El Salvador during
1990–2005.] Journal of Supranational Policies of Education, 2, 111–140.
Edwards, D. B., Jr., Victoria, J. A., & Martin, P. (2015). The geometry of policy
implementation: Lessons from the political economy of three education reforms
in El Salvador during 1990-2005. International Journal of Educational
Development, 44, 28–41.
Fontdevila, C., & Verger, A. (2015). The World Bank’s doublespeak on teachers: An
analysis of ten years of lending and advice. Brussels: Education International.
Gertler, P., Martinez, S., Premand, P., Rawlings, L., & Vermeersch, C. (2016).
Impact evaluation in practice (2nd ed.). Washginton, DC: World Bank.
Gilbert, C. L., & Vines, D. (2006). The World Bank: Structure and policies.
Cambridge: Cambridge University Press.
Haas, E. (1990). When knowledge is power: Three models of change in international
organizations. Los Angeles: University of California.
Hartmann, A., & Linn, J. (2008). Scaling up: A framework and lessons for development effectiveness from literature and practice. Brookings global economy and
20
D. B. EDWARDS JR.
development working paper series. Retrieved from http://www.brookings.
edu/research/papers/2008/10/scaling-up-aid-linn
Hay, C. (2002). Globalisation as a problem of political analysis: Restoring agents
to a ‘process without a subject’ and politics to a logic of economic compulsion.
Cambridge Review of International Affairs, 15(3), 379–392.
Heyneman, S. (2003). The history and problems in the making of education policy at the World Bank 1960–2000. International Journal of Educational
Development, 23, 315–337.
Jakobi, A. (2009). Global education policy in the making: International organisations and lifelong learning. Globalisation, Societies and Education, 7(4),
473–487.
Jones, P. (2006). Education, poverty and the World Bank. Rotterdam: Sense.
Khandker, S., Kookwal, G., & Samad, H. (Eds.). (2010). Handbook on impact
evaluation: Quantitative methods and practices. Washington, DC: World Bank.
Klees, S., & Edwards, D. B., Jr. (2014). Knowledge production and technologies
of governance. In T. Fenwick, E. Mangez, & J. Ozga (Eds.), World yearbook of
education 2014: Governing knowledge: Comparison, knowledge-based technologies
and expertise in the regulation of education (pp. 31–43). New York: Routledge.
Lauglo, J. (1996). Banking on education and the uses of research. A critique of:
World Bank Priorities and Strategies for Education. International Journal of
Educational Development, 16(3), 221–233.
Lingard, B., & Rawolle, S. (2011). New scalar politics: Implications for education
policy. Comparative Education, 47(4), 489–502.
Lingard, B., & Sellar, S. (2016). The changing organizational and global significance of the OECD’s education work. In K. Mundy, A. Green, B. Lingard, &
A. Verger (Eds.), The handbook of global education policy (pp. 357–373).
Oxford: Wiley.
Lubienski, C., Scott, J., & DeBray, E. (2014). The politics of research production,
promotion, and utilization in educational policy. Educational Policy, 28(2),
131–144.
Malen, B., & Knapp, M. (1997). Rethinking the multiple perspectives approach to
education policy analysis: Implications for policy-practice connections. Journal
of Education Policy, 12(5), 419–445.
Malin, J., & Lubienski, C. (2015). Educational expertise, advocacy, and media
influence. Education Policy Analysis Archives, 23(6), 1–28.
Menashy, F., & Manion, C. (2016). The historical evolution and current challenges of the United Nations and global education policy-making. In K. Mundy,
A. Green, B. Lingard, & A. Verger (Eds.), The handbook of global education
policy (pp. 319–334). Oxford: Wiley.
Meza, D., Guzmán, J., & de Varela, L. (2004, May 25–27). EDUCO: A
community-­managed education program in rural areas of El Salvador. Paper
presented at scaling up poverty reduction: A global learning process and conference, Shanghai.
A POLITICAL ECONOMY PERSPECTIVE ON KNOWLEDGE PRODUCTION
21
Mundy, K. (1998). Educational multilateralism and world (dis)order. Comparative
Education Review, 42(4), 448–478.
Mundy, K., & Ghali, M. (2009). International and transnational policy actors in
education: A review of the research. In G. Sykes, B. Schneider, & D. Plank
(Eds.), Handbook of education policy (pp. 717–734). New York: Routledge.
Mundy, K., & Menashy, F. (2014). The World Bank and private provision of
schooling: A look through the lens of sociological theories of organizational
hypocrisy. Comparative Education Review, 58(3), 401–427.
Mundy, K., & Murphy, L. (2001). Transnational advocacy, global civil society?
Emerging evidence from the field of education. Comparative Education Review,
45(1), 5–126.
Mundy, K., & Verger, A. (2015). The World Bank and the global governance of
education in a changing world order. International Journal of Educational
Development, 40, 9–18.
Porter, T. (1995). Trust in numbers: The pursuit of objectivity in science and public
life. Princeton, NJ: Princeton University Press.
Rambla, X., Verger, A., Edwards, D. B., Jr., Bonal, X., & Fontdevila, C. (2017).
Meeting development goals: Evidence from the Civil Society Education Fund.
Development in Practice, 27(6), 851–864.
Rao, V., & Woolcock, M. (2007). The disciplinary monopoly in development
research at the World Bank. Global Governance, 13, 479–484.
Reimers, F. (1997). The role of the community in expanding educational opportunities: The EDUCO schools in El Salvador. In J. Lynch, C. Modgil, &
S. Modgil (Eds.), Education and development: Tradition and innovation. Equity
and excellence in education for development (Vol. 2, pp. 146–162). London:
Cassell.
Rich, A. (2011). Ideas, expertise, and think tanks. In D. Béland & R. Cox (Eds.),
Ideas and politics in social science research (pp. 191–208). New York: Oxford
University Press.
Rust, V. D., Soumare, A., Pescador, O., & Shibuya, M. (1999). Research Strategies
in Comparative Education. Comparative Education Review, 43(1), 86–109.
Salas-Porras, A., & Murray, G. (2017). Think tanks and global politics: Key spaces
in the structure of power. New York: Palgrave Macmillan.
Samoff, J., & Stromquist, N. (2001). Managing knowledge and storing wisdom?
New forms of foreign aid? Development and Change, 32, 631–656.
Steiner-Khamsi, G. (2012). For all by all? The World Bank’s global framework for
education. In S. Klees, J. Samoff, & N. Stromquist (Eds.), The World Bank and
education: Critiques and alternatives (pp. 3–20). Rotterdam: Sense.
Stone, D. (2002). Using knowledge: The dilemmas of ‘bridging research and policy’. Compare, 32(3), 285–296.
UNESCO. (2008). EFA Global Monitoring Report 2009: Overcoming inequality:
Why governance matters. Oxford: Oxford University Press.
22
D. B. EDWARDS JR.
UNESCO. (2011). EFA global monitoring report: The hidden crisis: Armed conflict
and education. Paris: UNESCO.
Verger, A. (2012). Framing and selling global education policy: The promotion of
public–private partnerships for education in low-income contexts. Journal of
Education Policy, 27(1), 109–130.
Verger, A., Edwards, D. B., Jr., & Kosar-Altinyelken, H. (2014). Learning from
all? The World Bank, aid agencies and the construction of hegemony in education for development. Comparative Education, 50(4), 1–19.
Verger, A., Novelli, M., & Kosar-Altinyelken, H. (Eds.). (forthcoming). Global
education policy and international development: New agendas, issues and programmes (2nd ed.). New York: Continuum.
Vivalt, E. (2015). How much can we generalize from impact evaluations? New York
University. Retrieved from https://pdfs.semanticscholar.org/6545/a87feaec7d6d0ba462860b3d1bb721d9da39.pdf
Wade, R. (1996). Japan, the World Bank, and the art of paradigm maintenance:
The East Asian Miracle in political perspective. New Left Review, 217, 3–38.
Wade, R. (2001). Making the world development report 2000: Attacking poverty.
World Development, 29(8), 1435–1441.
Walters, P., Lareau, A., & Ranis, S. (Eds.). (2009). Education research on trial:
Policy reform and the call for scientific rigor. New York: Routledge.
World Bank. (1994). El Salvador: Community education strategy: Decentralized
school management. Washington, DC: The World Bank.
World Bank. (2003). World development report 2004: Making services work for poor
people. Washington, DC: World Bank.
World Bank. (2011). Learning for all: Investing in people’s knowledge and skills to
promote development. Washington, DC: World Bank.
World Bank. (2013). The what, why, and how of the Systems Approach for Better
Education Results (SABER). Washington, DC: World Bank.
World Bank/IEG. (2006). From schooling access to learning outcomes: An unfinished agenda. An evaluation of World Bank support to primary education.
Washington, DC: World Bank.
Zapp, M. (2017). The World Bank and education: Governing (through) knowledge. International Journal of Educational Development, 53, 1–11.
CHAPTER 2
Critically Understanding Impact
Evaluations: Technical, Methodological,
Organizational, and Political Issues
Abstract It is crucial that scholars, but also policymakers and practitioners, have a critical understanding of the conceptual and technical limitations of impact evaluations, as well as the ways that they are necessarily
affected by organizational and political dynamics. In order to achieve a
critical understanding of impact evaluations, this chapter directs its attention toward both their most common form—regression analysis—as well
as the form that is seen to be more robust, that is, randomized control
trials (RCTs). The methodological assumptions of each of these are discussed, first, in conceptual terms before moving on to a review of some
more technical issues. The final section of the chapter turns to a consideration of how the production of impact evaluations is affected by organizational and political dynamics. In all, this chapter advocates a critical
understanding of impact evaluations in five senses: conceptually, technically, contextually, organizationally, and politically.
Keywords Knowledge production • Impact evaluation • Regression
analysis • Randomized control trial • RCT • Political economy • Critical
review
I am grateful to Steve Klees, Adrián Zancajo, and Charles Blake for their helpful
feedback on an earlier version of this chapter.
© The Author(s) 2018
D. B. Edwards Jr., Global Education Policy, Impact Evaluations, and
Alternatives, https://doi.org/10.1007/978-3-319-75142-9_2
23
24
D. B. EDWARDS JR.
As noted in Chap. 1, impact evaluations have rarely been produced by
scholars working in the field of comparative and international education—
or at least they have not been published in some of the field’s top journals.
Indeed, from 1964 to 2011, three of the leading journals in this field
(Comparative Education Review, Comparative Education, and the
International Journal for Educational Development) did not publish any
studies that employed what is considered by many as the most credible
form of impact evaluation, namely, randomized control trials (RCTs)
(Burde, 2012; Rust, Soumare, Pescador, & Shibuya, 1999). While other
forms of impact evaluation have undoubtedly been more common—for
example, evaluations that employ regression analysis—the point here is
that there is reason to believe that many scholars in this field, as in the field
of education more broadly (Cook, 2001, 2004), may not be well-versed
in impact evaluation methods nor their attendant assumptions and
limitations.1
Yet it is crucial that scholars, but also policymakers and practitioners,
have a critical understanding of their conceptual and technical limitations,
as well as the ways that they are necessarily affected by organizational and
political dynamics, despite the efforts of impact evaluation proponents to
make their results as objective as possible. The need for this critical understanding stems from the outsized influence that impact evaluations can
have on education reform agendas at various levels, as Chap. 1 discussed.
The present chapter is thus intended to help us move from a situation
where impact evaluations are uncritically cited as evidence of what works
to a situation where impact evaluations are viewed with appropriate caution and even skepticism. For as Gorard and Taylor (2004) note with statistical methods more generally: “There is a misconception among social
scientists that statistical analysis is somehow a technical, essentially objective, process of decision-making, whereas other forms of data analysis are
judgement-based, subjective and far from technical” (p. 19).
In order to achieve a critical understanding of impact evaluations, this
chapter directs its attention toward both their most common form—
regression analysis—as well as the form that is seen to be more robust, that
is, RCTs. The methodological assumptions of each of these are discussed,
first, in conceptual terms before moving on to a review of some more
technical issues. The final section of the chapter turns to a consideration of
how the production of impact evaluations is affected by organizational
and political dynamics. In all, this chapter advocates a critical understanding of impact evaluations in five senses: conceptually, technically,
CRITICALLY UNDERSTANDING IMPACT EVALUATIONS: TECHNICAL…
25
c­ ontextually, organizationally, and politically. Put into the context of this
book, the current chapter ends with a focus on how organizational and
political influences affect impact evaluations themselves while subsequent
chapters consider this perspective and its opposite or how impact evaluations intersect with and help to shape the political economy of global
education reform more generally.
Thus, in the sections that follow, the focus is on unpacking assumptions
and limitations, in order to make clear what is often overlooked, poorly
understood, or unproblematically accepted by researchers, scholars, and
policymakers. While many aspects of impact evaluations are complex and
technical, their main assumptions and limitations can be addressed conceptually or in general terms. A grasp on these issues opens up the world
of impact evaluations to non-specialists and brings greater scrutiny to
methods that are widely seen as the most legitimate for policy-relevant
knowledge production. Ultimately, given that impact evaluations are
approached from the perspective of the non-specialist, it is hoped that the
discussion offered here is helpful to the consumers of impact evaluations,
that is, other scholars, practitioners, policymakers.
Impact Evaluation Through Regression Analysis
Regression analysis is a statistical approach that attempts to identify the
effect of various independent variables (on one side of the equation) on a
dependent variable (on the other side). Typically, the independent variable
of interest in an impact evaluation is some treatment, for example, some
policy, program, project, or pedagogy, whereas the dependent variable of
interest in a given study can take a number of forms, for example, test
scores, student retention, teacher attendance, and so on. The remaining
independent variables included in the equation are used to account for the
other factors that can influence the dependent variable. Added to the independent variables, there is always an error term that is also included. This
error term represents the portion of the dependent variable that is not
explained by the included independent variables.
As an example of a regression equation, consider a scenario where a
researcher is trying to explain the association between test scores and five
different independent variables. The equation would look like this:
Y = A + B1X1 + B2 X2 + B3X3 + B4 X 4 + B5X5 + E
26
D. B. EDWARDS JR.
Here, Y represents test scores, X1 could represent student age, X2 could
represent student ethnicity, X3 could represent student gender, X4 could
represent family income, and X5 could represent teacher participation in a
recent professional development program, where this program represents
the “intervention” for which researchers are trying to determine the
impact. The first term on the right-hand side, A, represents a constant
(i.e., the value of Y when all other independent variables are zero). As
noted above, the error term, E, accounts for that part of Y that is not
explained by other variables. While ideally, as explained below, the error
term is zero because all relevant variables are included, this is rarely, if ever,
the case. In the example above, it is easy to imagine that there are more
than five variables that influence test scores, such as neighborhood characteristics, peer effects, or school funding, and so on. In practice, the error
term worries researchers, with one reason being that they are concerned
about the extent to which the omitted variables that the error term represents are correlated with the included variables.2 This same preoccupation
with correlation applies to the included independent variables as well. In
both cases, where variables are correlated, it throws off the estimates for
what researchers are most interested in—namely, the regression coefficients of the independent variables. Correlation is a problem because it
means that a change in one variable is necessarily associated with a change
in another variable, and thus it becomes difficult to understand the relationships among them.
The regression coefficients are represented in the above equation by
B1, B2, and so on. Simply put, a coefficient estimates the amount of
change that one will observe in the dependent variable (e.g., test score)
after a change in the value of an independent variable (e.g., family income
or teacher participation in a professional development program) while
holding all other variables constant. One can easily imagine that researchers, policymakers, and others are eager to know what changes in test score
(or other dependent variables) are associated with a range of different
independent variables, with family income and teacher participation in a
particular professional development program being only two examples. It
is precisely because they can estimate the change in outcomes of interest
that are associated with changes in other variables (and interventions) that
regression coefficients are the object of most interest in regression
analysis.
However, in order for regression analysis to reveal the true effect of the
treatment being examined, a number of conditions must be met. Klees
CRITICALLY UNDERSTANDING IMPACT EVALUATIONS: TECHNICAL…
27
(2016) succinctly spells out these conditions. As he notes, in order for the
independent variables (or, more technically, their coefficients) to provide
insight into causal relationships (and thus speak to attribution), regression
equations (a) must include all relevant variables (in which case there is no
error term), (b) must ensure that all variables are measured properly, and
(c) must specify the correct functional interrelationships among the variables (Klees, 2016; see also Leamer, 1983). Together, these conditions
speak to the requirements of proper model specification.
There are unresolvable disagreements and differences among researchers about the second and third of these (I will return to the first condition
further below). With regard to the second condition, there are, for example, debates of how best to measure student learning, which is often the
outcome of interest in impact evaluations. Other times it not clear how to
measure the intervention itself. As one example, Klees and Edwards
(2014) discuss research on the impact of textbook usage on student test
scores; however, across studies, evaluators used different and difficult to
interpret measures for textbook utilization. The implication here is that
there is no consensus on the best way to capture the underlying phenomena that the included variables are supposed to represent (a problem
known as construct validity), and this goes for nearly all of the variables
that are included in regression analysis, not only the outcome and the
treatment variables. Think of the multiple possible ways for (and limitations in) measuring socioeconomic status, family characteristics, peer
effects, neighborhood influences, teacher preparation, school administration practices, school climate, or curriculum policies. The list could go on.
As for the third condition listed above, similar diversity and uncertainty
prevails in practice. This is because “the possible functional relationships
are innumerable” (Klees, 2016, p. 88). In other words, while it is common for researchers to assume a linear relationship in regression analysis,
it is often the case that the relationships among the variables included are
not linear in reality.3 Assuming a linear relationship means assuming that
an increase in one variable, for example, days of school in a school year, is
associated with a consistent change in the outcome variable, for example,
test scores. Put differently, if we imagine a study on the effects of school
year duration on student achievement, linear modeling in regression analysis would assume that the change in student achievement is the same
whether one is considering an increase from 100 to 101 school days or
from 180 to 181. (That is, the slope between the independent variable, X,
and the dependent variable, Y, is assumed to be the same for all values of
28
D. B. EDWARDS JR.
the independent variable.) As additional examples, research has shown
that the relationship between level of schooling and mortality is nonlinear
(Everett, Rehkopf, & Rogers, 2013), as is the relationship between hours
of tutoring and student mathematics test scores (Wang & Guo, 2018). In
this latter example, math scores increase considerably in the beginning but
then taper off as hours of math tutoring continues to increase. While statisticians focus on how to model such relationships, the relevant issue for
our purposes is that, even where researchers have modeled for more complex relationships, they do so in ways that are idiosyncratic in that different
studies reflect different choices for how to model nonlinearity—and in no
case can we be sure that all the nonlinear relationships have been modeled
or included in the regression equation. As Luecke and McGinn (1975)
state: “regression analysis fails to detect dynamic relationships” (p. 347).
Even if our variables were accurately measured and appropriately modeled, there would still be the issue raised in the first condition mentioned
above—that a researcher “is obligated to include in the regression equation all variables that might have an important effect” (Leamer, 1983,
p. 34). However, when it comes to education (and other social phenomena), there is a plethora of relevant variables. A short list was mentioned
above in relation to the requirement of accurate measurement; however,
even with this short list, it is unlikely that we are able to identify, measure,
and include all relevant aspects of such issues as family characteristics, peer
effects, teacher behavior, school climate, and so on, let alone more macro
considerations related to social services, public policy, and economic structure, for example (Weiss & Rein, 1970).4 This state of affairs leads Pogrow
(2017) to conclude that “it is simply impossible … to control for all the
confounding variables in the chaotic world of educational practice regardless of how much money or time is spent” (p. 14; see also Ginsburg &
Smith, 2016).
The inability in practice to meet the conditions of regression analysis is
very problematic when it comes to determining the impact of an intervention. This is because it is only when they are met that the variables in the
equation provide accurate estimates of the impact of a given policy, project, program, or pedagogy. Practically speaking, when the first condition
is not met, we cannot be sure that we are arriving at the true impact of the
treatment because there may be other relevant yet excluded variables
which, in their absence, means that we are not controlling for all the differences between treatment and control groups (this is known as omitted
variable bias). When we fail to include relevant variables, the estimates for
CRITICALLY UNDERSTANDING IMPACT EVALUATIONS: TECHNICAL…
29
all the included variables are inaccurate; this is true as well for the variable
that should reflect the impact of the intervention. As noted above, though,
even if we were able to identify and include all relevant variables, we would
still have to measure and model them correctly. And even then, there are
more practical concerns—relevant to RCTs as well—about (a) whether
the outcome variable of interest is a meaningful proxy for the kind of long-­
term impacts one hopes to see, (b) whether the timing of the evaluation
affects the observed outcomes (as in the case of charter schools, where
outcomes tend to be lower for newer schools), and (c) whether student
age may affect registered impacts, among others (Schanzenbach, 2012).
In any event, for two interesting examples of how debates around specification have played out, see Luecke and McGinn (1975), who take on the
famous findings of the Coleman study on the impact of school and family
variables on student achievement, and Komatsu and Rappleye (2017),
who challenge recent and influential work on the connection between
student achievement and national economic growth.
In all cases of regression analysis, it is important to remember each of
the three conditions stated above. This is because the conclusions of
regression analysis are subject to the weakest link principle. In the words
of Cartwright (2007), this principle states that “the probability of the
conclusion can be no higher than that of the weakest premise … Suppose
you have 10 premises, 9 of them almost certain, one dicey. Your conclusion is highly insecure, not 90% probable” (p. 11). Likewise for regression
conditions, if any one is not met, the results are faulty, and to an unknown
extent, since the data included in a study “in fact contain no information
about the size of the bias” that results from incorrect specification (Leamer,
1983, p. 33). Put differently, the data (or assumptions) we have cannot
tell us what data (or assumptions) are missing (or inaccurate). The importance of all three conditions also extends to more complex forms of statistical modeling and evaluation that have regression analysis as their basis,
including path analysis, structural equation modeling, hierarchical linear
modeling, propensity score matching, or the use of instrumental variables
(Klees, 2016).
Of course, regression analysis is only one approach to understanding
impact.5 While regression analysis has been used for decades, and though
it still continues in some measure to be a “respected and widely used scientific procedure” (Williams & Evans, 1969, p. 128), calls have increased
in recent years for improved forms of impact evaluation that can avoid or
30
D. B. EDWARDS JR.
mitigate the limitations of regression analysis.6 RCTs are seen as the most
promising avenue.
Impact Evaluation Through Randomized
Control Trials
Cook (2004) exemplifies the dominant perspective in evaluation when he
states that “the randomized experiment is widely known as the best tool”
for determining the effect of a treatment (p. 115). Just like regression
analysis, RCTs can be used to measure impact; however, the approach is
different. As the name implies, the essential characteristic of this approach
is its reliance on randomization. Baker (2000) describes the approach and
its virtues in a book produced for the World Bank titled, Evaluating the
Impact of Development Projects on Poverty: A Handbook for Practitioners:
By randomly allocating the intervention among eligible beneficiaries, the
assignment process itself creates comparable treatment and control groups
that are statistically equivalent to one another, given appropriate sample
sizes. This is a very powerful outcome because, in theory, the control groups
generated through random assignment serve as a perfect counterfactual,
free from the troublesome selection bias issues that exist in all evaluations.
(p. 2)
Thus, randomization, in theory, does two things: (a) it provides a counterfactual (i.e., evidence on what happens when the treatment is not received,
just as regression analysis does ideally but not in practice), and (b) it provides a way to deal with selection bias.7 While the value of a counterfactual
should be clear, because of the valuable information it provides for
decision-­making purposes, more needs to be said about selection bias at
this point, in order to understand the attraction to—and the expectations
placed upon—RCTs. After briefly discussing selection bias, this section
turns its focus to some of the major assumptions and issues associated with
RCTs.
Selection bias emerges when there are systematic differences in the
treatment and control groups. This form of bias is a frequent challenge in
evaluation because governments often pick the beneficiaries of a policy in
a purposive (i.e., non-random) way, that is, specifically because the program or policy in question is intended to benefit certain kinds of individuals or individuals living in certain kinds of conditions (e.g., in the poorest
CRITICALLY UNDERSTANDING IMPACT EVALUATIONS: TECHNICAL…
31
neighborhoods or in schools with lowest test scores).8 The result is that it
becomes difficult to interpret the effect of the intervention on the treatment group because the groups are different in some ways, and so one is
not comparing apples with apples. In ideal (i.e., impossible) circumstances,
selection bias is not a problem for regression analysis because one includes
all relevant independent variables in the regression equation, thereby controlling for all differences between the two groups (Cartwright, 2007).
However, in practice, selection bias is always an issue because we do not
have data on all possible group differences, and so the mathematical estimates for the regression coefficients are incorrect.
In RCTs, to get around the problems of selection bias, participants are
randomly assigned to either the treatment or control groups prospectively,
that is, before the program begins (Gertler, Martinez, Premand, Rawlings,
& Vermeersch, 2016). (This is the opposite of regression analysis, where
the control group is frequently selected after the treatment group has
already been selected or the treatment already implemented.) In RCTs, in
the most ideal circumstances, a randomly drawn, representative sample
would be selected from the larger population of interest, and then that
sample would be randomly assigned to one of the two groups (Deaton &
Cartwright, 2016b; Gertler et al., 2016). In this case, we could be certain
that the two groups are identical except for the treatment, and thus we
would be able to attribute differences in the treatment effects for the two
groups to the intervention.
In another book produced by the World Bank, Impact Evaluations in
Practice, Gertler et al. (2016) first reiterate that the “randomized assignment process in itself will produce two groups that have a high probability
of being statistically identical” (p. 67) and then provide a useful description of what this should mean in practice. They employ the hypothetical
example of randomly selecting pieces of paper from a bowl where the
names of 1000 individuals have been recorded, with these individuals
making up the entire population of potential participants. Because their
discussion is instructive, I quote it at length:
Assume that of the original 1000 people, 40 percent were women. Because
the names were selected at random, of the 500 names drawn from the bowl,
approximately 40 percent will also be women. If among the 1000 people,
20 percent had blue eyes, then approximately 20 percent of both the treatment and the comparison groups should have blue eyes, too. In general, if
the population of eligible units is large enough, then the randomized
32
D. B. EDWARDS JR.
a­ ssignment mechanism will ensure that any characteristic of the population
will transfer to both the treatment group and the comparison group. Just as
observed characteristics such as sex or the color of a person’s eyes transfer to
both the treatment group and the comparison group, then logically characteristics that are more difficult to observe (unobserved variables), such as
motivation, preferences, or other personality traits that are difficult to measure, would also apply equally to both the treatment and comparison groups.
Thus, treatment and comparison groups that are generated through randomized assignment will be similar not only in their observed characteristics
but also in their unobserved characteristics. (p. 68)
One thing that stands out from this account is the fact that both observed
and unobserved characteristics are distributed evenly in the treatment and
control groups. Another feature of randomization reflected in this description is the clean and simple nature of the approach. As long as participants
are randomly assigned from the relevant population of interest, the headaches typically associated with quantitative evaluation boil down only to
the differences in average outcome effects across the treatment and control groups (known as average treatment effect). However, as will be discussed below, things are not always as simple or as straightforward as they
seem.
Differences in Control and Treatment Groups
Even with randomization, differences in the treatment and control groups
can be found. As Klees (2016) writes, “if you look closely at all potentially
relevant factors, control groups almost always turn out systematically different from the experimental group, and the result is that we no longer
have the ability to make clear inferences” (p. 93). Deaton and Cartwright
(2016a) also make this point: “Any RCT can be challenged ex post by
examining the differences between treatments and controls as actually
allocated, and showing that arguably important factors were unevenly distributed” (p. 2).9 Of course, when the samples are not identical in terms
of background factors, or, really, any variables, then it is not clear how to
interpret the observed results, that is, the difference in average treatment
effects across the control and treatment groups. Once we can no longer
simply compare means between the two groups, we must include controls
to make the groups comparable, just as with regression analysis.10 Here
again, there can be some correlation between the variables included in the
CRITICALLY UNDERSTANDING IMPACT EVALUATIONS: TECHNICAL…
33
equation and those not included (which are all lumped together in the
“error” term), such that the coefficients on the variables included are
biased (Deaton, 2010).
Researchers often employ statistical controls to account for the differences between the control and experimental groups. However, as Klees
(2016) notes,
the application of statistical controls becomes an ad hoc exercise, even worse
than the causal modeling regression approach. In the latter, at least there is
a pretense of developing a complete model of potentially intervening variables whereas with the former a few covariates are selected rather arbitrarily
as controls. In the end, one does not know whether to attribute achievement differences to the [intervention] or other factors. (p. 93)
Deaton (2010), for his part, echoes the above sentiment:
Conducting good RCTs is exacting, and often expensive, so that problems
often arise that need to be dealt with by various econometric or statistical
fixes. There is nothing wrong with such fixes in principle—though they
often compromise the substance … Their application takes us out of the
world of ideal RCTs and back into the world of everyday econometrics and
statistics. So that RCTs, although frequently useful, carry no special exemption from the routine statistical and substantive scrutiny that should be routinely applied to any empirical investigation. (p. 447; see also Leamer, 2010)
In practice, whether one attempts to address this situation by following
applicable theory about which factors are most relevant (as Klees (2016)
suggests) or by relying on previous research about which prognostic variables are associated with the outcome (as de Boer et al. (2015) suggest),
the point is that the fundamental feature which should set RCTs apart—
that is, randomization and equivalency of the treatment/control groups—
has broken down.
Beyond faulty randomization, there are additional reasons for incomparability of treatment and control groups, reasons that are also acknowledged by advocates of RCTs (Duflo & Kremer, 2003). These reasons
relate to crossover, attrition, spillover effects, and contamination. For
example, parents may move their children from a school without a certain
program to a school with it (crossover). Or individuals allocated to a treatment group may not actually participate in the treatment (e.g., by not
showing up—a form of attrition). There can also be differential attrition
34
D. B. EDWARDS JR.
across the treatment and control groups since those who participate in the
treatment may be less likely to move or to dropout, particularly if they
know they are in the treatment group (a phenomenon known as the
Hawthorne effect) (Schanzenbach, 2012). Additionally, “spillover effects”
might occur, meaning that there could be an indirect impact of the program on individuals who did not participate in that program. One example is a de-worming program that not only reduces the incidence of worms
in children in the test program but also reduces the likelihood of transmission of worms to other children in nearby communities or schools (Miguel
& Kremer, 2004). In a school setting, “an intervention targeted to some
children … may also benefit others … who were in the control group—
perhaps through peer effects or through adjustments in teaching within
the school” (Banerjee et al., 2017).
In the case of contamination, control groups elect to implement aspects
of the treatment—which can happen even when there is significant separation of communities. Sanson-Fisher, Bonevski, Green, and D’Este (2007)
note that, “in an age of readily available, rapid, electronic information
sharing, geographic distance does not guarantee control conditions”
(p. 157). As an example, these researchers mention the case of contamination in an intervention from the US Midwest where 15 treatment and
control communities were spread across 500 miles. Even though “all the
communities were distinct, separated from other communities by large
areas of very low population-density agricultural land … evidence emerged
from control communities that they had also implemented components of
the intervention,” which was designed to address alcohol consumption
(Sanson-Fisher et al., 2007, p. 157). The implication for each of these
threats is that, despite randomization, there can still be conditions that
jeopardize the comparability of the treatment and control groups if not
also, as in the spillover effects’ example, jeopardizing the measurement of
the program impact.11
Unbiasedness Does Not Equal True Impact
If we assume that randomization has worked, then “the difference in
means between the treatments and controls is an estimate of the average
treatment effect among the treated” (Deaton, 2010, p. 439). However,
while this fact is often the basis for interest in RCTs, what this focus fails
to highlight is that randomization does not necessarily mean that
CRITICALLY UNDERSTANDING IMPACT EVALUATIONS: TECHNICAL…
35
c­ oefficients are unbiased in each experiment, only that, on average, across
replicated experiments, they are, as Deaton and Cartwright (2016a) note:
Unbiasedness says that, if we were to repeat the trial many times, we would
be right on average. Yet we are almost never in such a situation, and with
only one trial (as is virtually always the case) unbiasedness does nothing to
prevent our single estimate from being very far away from the truth. (p. 1)
Leamer (1983) further explains: “Randomization implies that the … estimator is ‘unbiased,’ but that definitely does not mean that for each sample
the estimate is correct. Sometimes the estimate is too high, sometimes too
low” (p. 31). Fluctuation of impact across replications stems from the fact
that “‘random’ does not mean adequately mixed in every sample. It only
means that on the average, the … treatments are adequately mixed”
(Leamer, 1983, p. 31). The practical implication of this is that we cannot
be sure of the accuracy of the impact registered in a single RCT.
Contrasting with the above statements are assertions made by RCT
proponents. For example, in the World Bank’s manual on impact evaluation, Gertler et al. (2016) write: “We can be confident that our estimated
impact constitutes the true impact of the program, since we have eliminated all observed and unobserved factors that might otherwise plausibly
explain the difference in outcomes” (p. 69). Responding to this assertion,
Deaton and Cartwright (2016b) point out that “this statement confuses
actual balance in any single trial with balance in expectation over many
entirely hypothetical trials” (p. 10). In any given study, the consequence
of “imbalance” of characteristics across the treatment and control groups
is that researchers must result to controlling for differences through
regression analysis, with all the limitations that accompany it.
Mean Treatment Effects Mask Variance
A key weakness of RCTs is that they only provide estimates of mean treatment effects, since the outcomes of the individuals in the intervention and
control groups are averaged. The difference in these means is then taken
to be the average effect (hopefully benefit) that results for those who
received the treatment. However, this average could be masking a situation where “nearly all of the population is hurt with a few receiving very
large benefits” (Deaton, 2010, p. 439). This situation clearly presents
problems for decision-makers, whether it be a doctor who must decide
36
D. B. EDWARDS JR.
what is best for an individual patient or a policymaker who must make
choices about public policy. While researchers may be tempted to eliminate outliers that show large effects, this strategy has a downside, particularly when considering the kinds of intervention evaluations that are typical
in international development and global education policy. This downside
can be summarized thus: “If the experiment is a project evaluation
designed to estimate the net benefits of a policy, the elimination of genuine outliers … will vitiate the analysis. It is precisely the outliers that make
or break the program” (Deaton & Cartwright, 2016a, 2016b, p. 19).12
The objectivity of RCTs thus gives way to circumstances where discretion
and subjectivity are inherent in the production and use of findings. (More
will be said about comparison of means below, in the section on technical
considerations.)
Lack of Generalizability
There are a few reasons for which RCT results can lack generalizability.
These reasons range from more specific to more broad in nature. As will
be further discussed, these reasons tend to reflect the inherent trade-off in
such studies between internal and external validity.13 The former refers to
the extent to which the methods employed credibly produce the results
given; the latter, for its part, refers to the degree to which the results are
applicable to other settings (i.e., generalizable). RCTs are designed to
identify the causal impact of interventions for particular samples; their
design does not ensure that the results are transferable. Before proceeding, it should also be noted that, with the except of the first subsection
below, the critiques of generalizability presented here also apply to impact
evaluations that rely on regression analysis.
Randomizing an unrepresentative sample. The fact of random
assignment to treatment and control groups as a feature of data collection
does not mean that it was done in a way that ensures generalizability. This
is different from the issue raised earlier, where even in ideal random assignment circumstances, the treatment and control groups will differ in some
respect. That situation relates to internal validity. Here, rather, the issue is
that you may be randomly assigning subjects who were not initially randomly selected from the target population. For example, it may be necessary to assign communities or schools for intervention or control that
were already selected (or designated as candidates) in a non-random way
by the government, as in the recent, high-profile case in Liberia, where an
CRITICALLY UNDERSTANDING IMPACT EVALUATIONS: TECHNICAL…
37
RCT was done on the effects of private management of schools (Romero,
Sandefur, & Sandholtz, 2017). Likewise, when you must select from
among individuals who volunteer to participate in a study, you are still not
able to choose freely from among the target population. Instead, you perform a random assignment on a set of people (or schools or communities)
who are willing to participate in the experiment. As Clay (2010) discusses,
there is reason to believe that folks who volunteer for experiments are
systematically different from the population of interest (i.e., the population to which one would like to generalize) (see also Deaton, 2010). In all
cases—that is, for students, classrooms, schools, or communities—if you
randomly assign candidates who were not initially chosen from the group
to which you want to extend the results, then randomization does not
ensure generalizability.14 For example, if the treatment schools in your
sample are different from those in other areas of the school system because
they are more poor, have more immigrants, have more highly qualified
teachers, and so on, then you cannot be sure that the results would be
similar if the program were to be expanded.
Pilot context different from policy context. RCTs are often performed on pilot programs, and there is no guarantee that a scaled up version of the pilot will produce the same results. One obvious reason for
difference in outcomes would be if the pilot dealt with a small, unrepresentative sample. However, even if random selection is done to generate a
sizable, representative sample, there can be “general equilibrium effects
that operate on a large scale that are absent in a pilot” (Deaton, 2010,
p. 448). Put differently, there can be aspects of the larger context that
affect the program once scaled up that did not affect the pilot program.
Additional reasons for differences include the likelihood that programs are
more apt to attract the attention of unscrupulous officials as they scale up
or because program implementers are more likely to be invested in the
success of a pilot than are bureaucrats in charge of broad implementation,
when fidelity to program design can also be an issue (Deaton, 2010;
Schanzenbach, 2012). In the words of Rodrik (2008),“The typical evaluation will have been carried out in a specific locale on a specific group and
under specific experimental conditions. Its generalizability to other settings is never assured … and it is certainly not established by the evaluation itself” (Rodrik, 2008, p. 4). Even RCT proponents have recognized
this fact, as when Pritchett (n.d.) stated: “the ‘rigorous evidence’ about
what works when implemented by a non-governmental organisation (or
researchers as a field experiment) did not prove to be rigorous evidence
38
D. B. EDWARDS JR.
about what would happen in the context of government implementation”
(p. 2). Thus, although RCTs are touted as a means for determining “what
works” regardless of context, this belief belies an overly simplistic view of
taking interventions to scale. Rather than finding the secrets of a program’s success in one study, it can take many years and many evaluations.
Ganimian (2017), for example, discusses the case of a personalized learning program in India that underwent 5 RCTs over 9 years before evaluators thought they arrived at a clear understanding of the key features of the
intervention and documented implementation lessons (see also Banerjee
et al., 2017). It stands out that RCT advocates have tended to overlook
the lessons from the 1970s and beyond about the variety of local and
political factors that can influence policy implementation (see, e.g., Braun,
Ball, Maguire, & Hoskins, 2011; Lareau, 2009; McLaughlin, 1987,
1990).
RCTs are not a basis for policy transfer. An extension of the above
point is that RCTs are not a basis for the direct transfer of a program from
one country to another, a common practice in global education policy.
Because RCTs are seen as indicating “what works,” it has been suggested
that institutions like the World Bank should rely on them more when it
comes to deciding which policies to promote (Banerjee & He, 2008).
Banerjee and He (2008), for example, point to the RCT evidence on
school vouchers that was produced by the World Bank on Colombia during the 1990s. Rodrik (2008), however, questions the generalizability of
this evidence and poses a number of questions that illuminate the weakness of Banerjee and He’s (2008) suggestion. He asks:
How would the results change if we were to alter, as we necessarily have to,
the target population (children of secondary-school age in Colombia’s low-­
income neighborhoods)? Or the environment in which the experiment was
conducted (e.g., the availability and quality of nearby private educational
facilities)? Or some details of the program (e.g., the share of private-school
costs covered by the voucher)? (Rodrik, 2008, 19)
Of course, we cannot know how the results would change. When it comes
to policy transfer, RCTs are no more helpful than other forms of research,
and may be less so, since RCTs are not generalizable and since the results
do not tell us anything about program implementation.
As an additional example, consider the case of Progresa, the conditional cash transfer program from Mexico. This program has been adapted
by no fewer than 40 countries, at least in part because of the RCT that
CRITICALLY UNDERSTANDING IMPACT EVALUATIONS: TECHNICAL…
39
was produced showing its beneficial outcomes on school attendance (see
Peck & Theodore, 2015). Here, however, even the architect of the program has commented that “this program cannot be simply borrowed by
other countries when there are key differences, such as perhaps other
antipoverty programs with which a [conditional cash transfer] program
might compete or clash, or if schools cannot accommodate the additional
demands from students, or if policymakers are not committed to the program in the same way” (Deaton, 2010, p. 449; see also Levy, 2006).
From a critical perspective, these comments lead one to conclude that
“The RCT, with its vaunted rigor, takes us only a very small part of the
way we need to go for practical knowledge” (Cartwright, 2007, p. 19).15
RCT reports do not address context. As opposed to assuming that
RCT findings are relevant to other contexts, authors need to demonstrate
their relevance. Deaton and Cartwright (2016a) have been critical of inadequate attention in this area, as when they write:
If we want to go from an RCT to policy, we need to build a bridge from the
RCT to the policy. No matter how rigorous or careful the RCT, if the bridge
is built by a hand-waiving simile that the policy context is somehow similar
to the experimental context, the rigor in the trial does nothing to support a
policy. (p. 2)
Yet in practice, studies reporting on RCTs tend not to be preoccupied
with the context in which they are conducted. Rather, as Rodrik (2008)
observes, “The typical [RCT] says very little about external validity. If
there are some speculations about the background conditions which may
have influenced the outcomes and which do or do not exist elsewhere,
they are offered in passing and are not central to the flow of the argument” (Rodrik, 2008, p. 20). This is because RCTs are designed primarily
to deal with threats to internal validity. As such, “the typical field experiment makes no claims about the generalizability of the results … [though]
the careless reader may well walk away from the study thinking that she
has learned more about the broader policy question of interest than she
actually has” (Rodrik, 2008, p. 21). Arguably, lack of focus on context in
RCTs reinforces the notion that the relevance of such studies should not
be questioned. That is, lack of focus on context benefits RCT authors
because an extensive discussion of context and a substantive defense of a
study’s generalizability may make (otherwise unaware) readers cognizant
of the fact that RCT findings are not universally applicable.
40
D. B. EDWARDS JR.
Double standard regarding generalization. It is ironic that those
who are preoccupied with objectivity would fail to see that using RCTs in
practice is bound up with subjectivity. As critical attention is brought to
RCTs, authors, such as those quoted above, point out this contradiction.
To the extent that future RCT evaluations change at all to reflect this concern, study authors will have to “resort to credibility-enhancing arguments that feel rather similar to those that practitioners of cross-section
econometrics and qualitative studies have to resort” (Rodrik, 2008 p. 15).
Put differently, RCT professionals will have to focus on contextual factors
as opposed to focusing exclusively on the procedures used for ensuring
internal validity. An additional irony is that this shift in practices leads back
to the kinds of qualitative methods (e.g., case study, ethnography, historical analysis, etc.) that RCT proponents sought to avoid in the first place.
The point is that the contextual considerations necessary to judge transferability of findings of RCTs are the same considerations that are necessary
for other, supposedly less rigorous, forms of evaluation. As Rodrik (2008)
notes, “the ‘hard evidence’ from the randomized evaluation has to be
supplemented with lots of soft evidence before it becomes useful” (p. 5).
Blind replication is not the answer. It is often claimed that study
replication leads to more credible results. However, RCTs are frequently
carried out without establishing a connection to theory—that is, in addition to ignoring context, they also fail to address the theory of the program or policy being examined. As a result, we know something about the
impact of the intervention but not why it worked. Replicating studies by
employing RCTs across various locales will not help us explain the success
or failure of a program and, thus, how to modify it in different contexts.
Rodrik (2008) extends this point: “Absent a full theory that is being put
to a test, it is somewhat arbitrary to determine under what different conditions the experiment ought to be repeated. If we do not have a theory of
which Xj’s matter, we cannot know how to vary the background conditions. Moreover, everyone is free to come up with an alternative theory
that would enlarge the set of conditioning variables” (pp. 21–22).
Problematically for RCTs, then, these comments emphasize not only the
importance of considering context but also of incorporating theory into
the design and analysis of evaluations. RCT proponents may not welcome
this perspective since it undermines the objective way that RCTs have
been portrayed.
Methodologically, some researchers respond to the issues raised above
by suggesting that RCTs should be combined with qualitative methods.
CRITICALLY UNDERSTANDING IMPACT EVALUATIONS: TECHNICAL…
41
The thinking is that the insights generated by such methods complement
the shortcomings of RCTs, for example, by helping to explain (or to generate hypotheses about) the relationships among the variables in the study
(e.g., Burde, 2012; Castillo & Wagner, 2013; Deaton, 2010; Garbarino &
Holland, 2009). Others suggest that regression studies and RCTs should
be eschewed in favor of qualitative approaches alone (Klees, 2016) or that
carefully interpreted observational studies provide information that is just
as good as RCTs, given the problems of the latter in practice (Concato,
Shah, & Horwitz, 2000).16 While it is important at this point to be aware
of this response to the critiques mentioned above, the final chapter of this
book returns to a discussion of qualitative methods that can be seen as
complements or substitutes for regression and RCTs.
Critical understanding of evaluation context. In order to argue for
generalizability, RCT authors should consider how the context of their
study enabled or constrained the results. RCT readers similarly need to
have an in-depth understanding of the RCT context. That is, those who
seriously seek to learn from RCTs should strive to attain a critical understanding of the study context. Often, RCTs—and other impact evaluations—may only report on the basic characteristics of the control and
treatment groups or the schools and communities where the study was
carried out. However, the kinds of information and insights necessary to
critically interpret an impact evaluation go beyond this surface-level context. We must consider the kinds of questions and concerns posed above
in relation Colombia’s vouchers program and Mexico’s conditional cash
transfer initiative—which get at the structural, peripheral, and political
aspects of the context that go beyond the immediate community but
which have influenced the observed results. We must go beyond the basic
contextual descriptions that accompany impact evaluations. The goal
should be to gain an understanding of the socio-cultural context and the
relevant historical dynamics such that one has insight into the relations
within and among communities as well as among schools, communities,
the government, and other organizations (e.g., non-governmental organizations) so that one can perceive the ways that these less obvious aspects
of context can bleed into a study and affect not only participant behavior
but also the results reported by impact evaluations on specific outcomes
(Biglan, Ary, & Wagenaar, 2000). A critical understanding of context is
important in the case of the Education with Community Participation
(EDUCO) program, as will be shown in this book.
42
D. B. EDWARDS JR.
Technical Considerations
Aside from the general points above, there are additional dimensions of a
more technical nature along which impact evaluations should be critically
understood. These technical aspects relate the computation and assumptions of certain statistics. Though these aspects are grouped under the
label of “technical considerations,” they are still relatively basic. For our
purposes, what is fascinating is that the procedures and practices discussed
below have serious weaknesses and limitations that call into question the
meaningfulness of the impact evaluation enterprise.
Hypothesis Testing
The foundation of RCTs is the idea that the means for the outcome of
interest of the control and treatment groups can be compared to see if the
intervention made a difference, for example, by producing higher average
test scores in the treatment group. To make this determination, a statistical test is set up that is intended to indicate “how often differences as large
or larger than the differences we found would occur as a result of chance
or sampling [error]” (Carver, 1978, p. 380). The point of departure for
this test is the assumption that there is no difference in means, which, if
true, supports the claim that the intervention had no effect. This point of
departure is known as the null hypothesis, the opposite of which is the
research hypothesis that states that there is a difference in means.
The test most often used to make a decision about the null hypothesis
is known as the t-test. The values produced by the t-test are associated
with different levels of probability (known as p values). If the t-test statistic
is sufficiently large, the null hypothesis is rejected in favor of the research
hypothesis.17 Typically, if the t-test suggests that the probability is 5% or
less that results would be found as large or larger than the ones in the
study as a result of chance or sampling error, then the differences are
labeled statistically significant. By extension, the results are also taken as
support that the intervention “worked,” since “sampling, or chance, is no
longer considered a good [i.e., likely] explanation for the cause of the difference between the means” (Carver, 1978, p. 380).
One problem with the t-test and other similar tests is that they are very
sensitive to sample size. This is easy to understand when one notes how
the t-test is constructed. To that end, the overall t-statistic is defined as:
Effect/standard error of the effect estimate.18
CRITICALLY UNDERSTANDING IMPACT EVALUATIONS: TECHNICAL…
43
The “effect” in the numerator is simply the difference in the treatment
and control means (i.e., treatment mean—control mean). Thus, one way
to get a large t-statistic (and thus a result that is deemed statistically significant) is if the program is effective in improving the outcome for the treatment group (and thereby increasing the size of the numerator). However,
even if the difference in means is very small (which should raise questions
about the practical significance of the results), one can still arrive at a
t-statistic that indicates statistical significance by affecting the denominator. This brings us to the denominator of the t-statistic (i.e., the standard
error).
The standard error in the denominator of the above fraction gives some
measure of the variation (i.e., standard deviation) around the mean of the
individual results (e.g., individual test scores) for the treatment and control groups. That is, it speaks to the distribution of the results in these
groups. With that in mind, it can be mentioned that the standard error is
itself a fraction (or, more technically, in the case of the two-sample t-test,
the addition of two fractions) that boils down to
Standard deviation of the means/square root of sample sizes.
The practical implication of the standard error definition is that one can
greatly affect the interpretation of the overall results of a study by increasing the sample sizes of the treatment and control groups. To put this statement in relation to the comments in the preceding paragraph, if one
increases the sample size, the value of the standard error gets smaller,
which then impacts the value that results from the first fraction above, that
is, from the definition of the t-statistic, by making it larger. And as stated
previously, larger t-statistics are associated with smaller probabilities that
results as large or larger than those obtained would occur due to chance or
sampling error. At the point when such a probability dips below the 5%
threshold, it is customary for researchers to interpret the results as statistically significant, as also noted above.
The sensitivity of hypothesis testing to sample size is well known and
has been criticized by many scholars (e.g., Boruch, Rindskopf, Anderson,
Amidjaya, & Jansson, 1979; Carver, 1978, 1993; Deaton & Cartwright,
2016a, 2016b; Klees, 2016; and Levine, Weber, Hullett, Park, & Lindsey,
2008). As Levine et al. (2008) put it, sample size sensitivity “can lead to
dismissing potentially important findings when a sample is small and
embracing trivial effects with large samples” (p. 176). Of course, as sample
44
D. B. EDWARDS JR.
size increases, the outcome means and the standard deviations of the samples will stabilize (i.e., will become less sensitive to the addition of new
participants), which leads to more confidence (or more reliability) in the
t-statistic, but this doesn’t change the fact that researchers can ensure statistical significance for minute differences in means if they have large
enough samples.19 Moreover, there are no clear rules on how large is a
large enough sample nor for how small is too small to have faith in a statistically significant result (Deaton & Cartwright, 2016a, 2016b). Thus,
while sample size is a crucial issue, there is no consensus on how to address
its risks. For our purposes, the point is to be aware of the fact that practically dubious results of RCTs can show up as statistically significant because
of large sample sizes and, vice versa, that potentially meaningful differences may not register as statistically significant because of small
samples.20
While the comments above focused on RCTs, parallel comments can be
made with regard to hypothesis testing when using regression analysis as a
form of impact evaluation. The same test statistic and the same logic apply
to the determination of statistical significance of regression coefficients;
the difference is that the focus is on the dummy variable that represents
program participation (as opposed to the difference in outcome means).
One looks to see if the independent variable of interest (participation in
the intervention) has an impact on the dependent variable (e.g., test
scores) that is greater than zero (i.e., one looks to see if the relationship
between this variable and the outcome variable is not zero) while holding
all other variables constant. In other words, in the hypothesis test for
determining statistical significance, one compares the mean of the (coefficient of) intervention participation variable with zero in order to see if
the intervention contributes to explaining changes in the value of the outcome variable (or at least is associated with these changes) (Uriel, 2013).
More technically speaking, researchers hope to reject the null hypothesis
that the (coefficient of the) intervention participation dummy variable is
equal to zero—and, as with the difference in means in RCTs, they want to
make such a decision based on a low probability that results as large or
larger than those obtained would occur due to chance or sampling error.
For our purposes, though, the problem remains that statistical significance
for the intervention variable depends on sample size.
In practice, impact evaluation advocates suggest making sample size
decisions through a particular procedure. For example, the handbook produced by the World Bank on impact evaluations doesn’t suggest specific
CRITICALLY UNDERSTANDING IMPACT EVALUATIONS: TECHNICAL…
45
sample sizes; instead the authors suggest the use of “power calculations”
(Gertler et al., 2016), a procedure which, based on certain assumptions
(related to projected effect size and p value), will tell you the sample size
needed to have “a specified probability of declaring as significant a particular difference or effect” (Johnson, 1999, p. 767). In other words, power
calculations can be used to tell a researcher the sample size needed to
ensure a high probability that a particular effect size registers as statistically
significant.21 This characteristic of power analysis can lead to further skepticism of impact evaluations since researchers can reverse engineer the
sample size needed to have a good likelihood of arriving at a statistically
significant result.
As a final point on hypothesis testing, consider a more insidious issue,
one that primarily affects impact evaluations done using regression analysis. Specifically, the combination of the above problems—related to sensitivity of sample size and issues of statistical power, among others, such as
correlation among variables and omitted variable bias (discussed earlier)—
creates a situation that is difficult to overcome. This is because the combination of these factors leads to a level of “crud” (Meehl, 1986, 1997) or
“ambient correlational noise” (Lykken, 1968) that undermines the interpretation of results from hypothesis testing. As Levine et al. (2008) write:
The crud factor refers to possible systematic differences or associations
between variables that exist and are observed due to systematic uncontrolled
spurious third variables and extraneous factors and that are independent of
the verisimilitude of the predicted relations. … The problem is, it is practically impossible to know what proportion of a significant effect might be
due to crud and what is not. So even if sufficient statistical power exists, and
even if effect sizes are taken into account, it is still difficult to interpret the
results of [hypothesis testing] with precision. (p. 179)
Additionally, problems of spurious statistical significance can be worse
when there are few participants relative to the number of variables, since
one is trying to extract a great deal of information from a small informational base and thus it is all the more difficult to ensure that the interpretation of variables is unencumbered by correlation (Anderson, Burnham,
Gould, & Cherry, 2001; Flack & Chang, 1987; Freedman, 1983).
Although the points raised here are certainly relevant to regression analysis, they emerge more generally whenever randomization fails in an RCT
and post hoc controls must be added to ensure comparability of groups
(which is done through regression analysis).
46
D. B. EDWARDS JR.
P Values and the Normal Curve
More needs to be said about p values. While it was noted above that the
t-statistic is associated with probabilities—i.e., p values—that indicate
whether or not the null hypothesis should be rejected, the relationship
between the t-statistic and the null hypothesis as well as the assumptions
that underlie the p values should be further clarified. There are fundamental issues here that go to the heart of the impact evaluation enterprise.
I first return to the meaning of the p value. While it is sometimes
assumed that a p value can be interpreted as the probability that the results
observed were due to chance, or the probability of getting the results
based on the data gathered, this is not the case. That is, a p value of 5% (or
0.05) does not mean that, based on the data collected, there is a one in 20
chances of getting a particular result; rather, it means the opposite. As
Carver (1978) writes, the p value “is a number which tells us the proportion of the time that we can expect to find mean differences as large or
larger” than what was found if we were to repeat the study procedures
(p. 380, emphasis added). Put differently, if X difference is found in the
means in two groups, a p value of 0.05 says that we will find a difference
as large or larger than X in 5% of cases “when sampling a pair of means
from the same population” (p. 381).
However, in any given study or impact evaluation, we typically only
have one set of means from which to make a decision, and we have no way
of knowing what the other cases would produce. This fact leads to the
observation that the interpretation of a p value amounts to making a decision regarding statistical significance based on “hypothetical alternative
scenarios” (Nuzzo, 2015, p. 1). As Nuzzo (2015) further, and quite
damningly, elaborates:
The p-value summarizes how often results at least as extreme as those
observed would show up if the study were repeated an infinite number of
times when in fact only pure random chance were at work. …. This means
that the p-value is a statement about imaginary data in hypothetical study
replications, not a statement about actual conclusions in any given study.
Instead of being a “scientific lie detector” that can get at the truth of a particular scientific finding, the p-value is more of an “alternative reality
machine” that lets researchers compare their results with what random
chance would hypothetically produce. (p. 1)
With the increased scrutiny of p values in recent years, at least one journal
(Basic and Applied Social Psychology) decided, as of 2015, that it would no
CRITICALLY UNDERSTANDING IMPACT EVALUATIONS: TECHNICAL…
47
longer accept studies that employ them. This is a radical move given that
p values have been commonly used in practically all areas of research over
approximately the past 100 years (Fendler & Muzzafar, 2008; Gigerenzer,
Swijtink, Porter, Datson, Beatty, & Kruger, 1989). The significance of this
break from precedent is further underscored when one considers that p
values have been used in at least three million scientific papers (Nuzzo,
2015). In a move with direct relevance for impact evaluations, even the
American Statistical Association now advises that “scientific conclusions
and business or policy decisions should not be based on only whether a
p-value passes a specific threshold” (Novella, 2016, p. 4).
The above recommendation makes sense when one recalls that p values
are simply used to reject (or not) the null hypothesis that there is no difference in the two values being compared. In the case of an RCT, even
when we reject the null hypothesis that the two means are equal, all we are
saying is that there is a high probability that the means are not the same;
we cannot say why this is the case. Further to the point, Greenland et al.
(2016) highlight an issue which is frequently overlooked: “the p value
tests all the assumptions about how the data were generated (the entire
model), not just the targeted hypothesis it is supposed to test (such as a
null hypothesis)” (p. 339). That is to say, “not only does a p value not tell
us whether the hypothesis targeted for testing is true or not; it says nothing specifically related to that hypothesis unless we can be completely
assured that every other assumption used for its computation is correct”
(p. 339). The range of assumptions includes not only those conditions
mentioned earlier in relation to regression analysis but also such issues as
correctly following data collection protocols. Thus, the fact of rejecting
the null does not necessarily support the theory or policy of interest
(Meehl, 1986). Moreover, just because a p value suggests that the means
are different, this does not immediately imply that the difference is “substantial, or beneficial, enough to be of practical importance”—a conclusion for which more information would be needed, as discussed in the
section above on generalizability (Pogrow, 2017, p. 7).
For the second issue, it is necessary to go beyond the interpretation of
p values as probabilities to critique as well the underlying basis for those
probabilities. Simply put, the distribution against which p values are
judged is assumed to have a normal (or bell-curve) shape. On the basis of
this assumed distribution, individual values (such as means in RCTs or
individual variable coefficients in regression analysis) are judged to be
more or less likely based on their distance from the hypothetical mean of
48
D. B. EDWARDS JR.
the parent population. Thus in impact evaluations, it is assumed that all
social variables have normal distributions and that it makes sense to judge
statistical significance on the probability that a given result would not
occur—except by chance or sampling error—in the majority of cases.
One problem with the practice of making decisions on the grounds of
the normal distribution is that it was not created to be applied in the social
sciences. Initially, the normal distribution emerged in the eighteenth century as a way to describe probabilities associated with coin tossing (Fendler
& Muzaffar, 2008). It also developed from the work of astronomers, who
used the normal distribution as a way of assessing measurement error in
determining the positions of heavenly bodies (Gigerenzer et al., 1989). It
was only adopted in the social sciences in the nineteenth century, with the
development of modern nations, the growth of governmental bureaucracies, and the desire to use statistics “as a technology by which government
offices could rationalize systems of population management, diagnosis,
and intervention” (Fendler & Muzaffar, 2008, p. 72). Although initially
seen as scandalous and although there was initially significant pushback,
the “transformation in the understanding of [the] normal distribution
from a statement about the regularities of arithmetical probability to an
insight into the workings of society” was gradually accepted in the second
half of the nineteenth century (Fendler & Muzaffar, 2008, p. 71).
The shortcoming of this wholesale adoption, however, is that it meant
not only the importation of technique but also the imposition of interpretation. As Gigerenzer et al. (1989) relate:
When Adolphe Quetelet applied the curve describing observational errors in
astronomy to social statistics, he took over the substance as well as the form;
the interpretation as well as the technique. If astronomers understood the
normal distribution as a scattering of observations around the mean, true
value for, say, the position of a comet, then social statisticians must understand the same distribution as a scattering of nature’s ‘errors’ around the
mean ‘true’ value for, say, the moral condition of a nation. (p. 272)
The appropriation of this technique by psychology in the late 1800s and
1900s meant that the normal distribution was also applied to such things
as cleverness, wit, and civility (Fendler & Muzaffar, 2008). In so doing,
the underlying assumption was that “nature aimed at a fixed [i.e., average]
point in forming human beings, but made a certain frequency of errors”
(Goertzel, n.d., p. 3).
CRITICALLY UNDERSTANDING IMPACT EVALUATIONS: TECHNICAL…
49
However, despite its widespread use, there has long been recognition
from researchers that data often do not reflect the normal distribution. As
Berkson stated in 1938: “It is practically certain that any series of real
observations does not actually follow a normal curve with absolute exactitude in all respects” (p. 526). Separately, Goertzel (n.d.) comments that
“social life … is not a lottery, and there is no reason to expect sociological
variables to be normally distributed” (p. 14). Even Karl Pearson, one of
the founders of modern statistics “sought to move statistics away from the
tendency to assume that data are distributed according to the normal
curve, and to this end he defined a whole family of curves, of which the
normal was only one special case” (Gigerenzer et al., 1989, p. 113).
Recent research by O’Boyle and Aguinis (2012) has convincingly demonstrated the need to take the aforementioned admonitions seriously.
Here, the authors conducted studies that looked across five professional
fields—including researchers, entertainers, politicians, and athletes—and
found, based on 198 samples with 633,263 participants, that performance
is not normally distributed, in particular because the samples regularly
included extreme values.22 In that this study looked at individual performance, the results, arguably, are particularly salient for education, where
teachers, students, and principals are often evaluated individually.
Furthermore, in that impact evaluations in education often look at outcomes on tests of student knowledge, it should be mentioned that such
tests, by design, will produce results that are approximately normal, even
though the underlying variable is not (Goertzel, n.d.).23 To that end, it is
debatable what the underlying variable of student knowledge tests is—
knowledge? Scholastic aptitude? Intelligence? The larger point, though, is
that the practice of assuming normality of outcomes, including those
assessed in impact evaluations, is highly questionable (Deaton &
Cartwright, 2016a, 2016b).
The implication of the above is that the body of knowledge that has
been developed through previous impact evaluations may not mean much
of anything if the underlying basis for judging statistical significance
doesn’t reflect the actual distribution of data or, in the case of achievement
tests, doesn’t reflect the distribution of the underlying variable of interest.
More generally, for any type of distribution, one can question the wisdom
of using probability tests for judging the likelihood of an outcome, since
doing so assumes that social phenomena occur with some form of mathematical regularity and that it makes sense to judge current and future
50
D. B. EDWARDS JR.
events against an assumed distribution regardless of context (Fenlder &
Muzaffar, 2008; Klees, 2016; Meehl, 1978).
Reporting and Interpretation of Effects
Ignoring, for a moment, the substantive critiques presented above, there
are three practices related to reporting and interpreting the effects of
impact evaluations that should be addressed—all of which have been further discussed by Pogrow (2017). The first is that, at times, studies only
report adjusted outcome scores. The idea here is to make the outcome
scores better reflect what they would have been if the treatment and control groups had started from a similar base score. For example, “if the
initial reading scores of the experimental group are lower than the scores
of the comparison group, then the post-test scores of the experimental
group are statistically adjusted relatively upwards” (Pogrow, 2017, p. 7).24
The problem, however, is that the reader do not know the actual performance of the two groups, which makes it hard to interpret.
Second, and relatedly, it is common for a study to report relative results
only. As with the previous example, it is difficult to make sense of relative
results, since they do not tell us about the absolute performance or practical meaning of a study’s effect. In comparison with the control group, a
treatment group might have performed very well, but both groups might
have performed poorly overall. Of course, having this information is
important for making a decision about how to use the results of the evaluation. It is thus problematic that “top research journals and the What
Works Clearinghouse rely solely on the relative difference between an
experimental and comparison group to determine the effectiveness of an
intervention” (Pogrow, 2017, p. 7, emphasis in original).
One common way of reporting relative effect sizes is based on standard
deviation units (SDUs). Here, the experimental and control means on the
outcome variable are divided by their respective standard deviations, and
then the difference between these two figures is computed (Durlak, 2009).
The rule of thumb for interpreting the relative difference of SDUs holds
that a difference of around 0.2 SDUs is small, 0.5 is medium, and 0.8 is
large (Cohen, 1988). While 0.2 is regarded as small, it is generally accepted
that this difference is still “large enough to have practical significance; that
is, the experimental program can be judged effective” (Pogrow, 2017,
p. 8). The problem, as Pogrow (2017) writes, is that Cohen (1988) “characterized such a small difference as ‘difficult to detect’” and that “it is only
CRITICALLY UNDERSTANDING IMPACT EVALUATIONS: TECHNICAL…
51
at 0.5 … that Cohen characterizes differences as becoming ‘visible to the
naked eye’; that is, apparent to practitioners and community members”
(Pogrow, 2017, p. 8). Pogrow (2017) continues by noting that, at 0.2
SDU, “there is substantial overlap between the performance of the experimental and comparison groups, and only a very small percentage of students are benefitting” (p. 8). Thus, although an effect size of 0.2 (or less)
draws attention, in practice, Pogrow (2017) suggests that it “makes no
sense” to expect school leaders (or, I would add, others, such as policymakers) to adopt reforms that may be more expensive or time-­consuming—
not to mention potentially disruptive to the educational experience of
students—“in order produce improvements that are at best ‘difficult to
detect’” (p. 8). Further to the point, the aversion of school leaders or policymakers is all the more understandable when one remembers that,
regardless of effect size, standard deviation units describe relative performance, not actual performance.25
The third practice to highlight is the extrapolation to hypothetical
impact based on effect sizes. Since effect size is an abstract concept, it is
common for researchers to report effect sizes in terms that make more
sense intuitively. That is, because effect sizes are “just abstract numbers
that have no recognizable real-world meaning to practitioners or policy
makers” (Pogrow, 2017, p. 10), researchers result to extrapolations
whereby they take effect sizes and further convert them to hypothetical
impacts on test scores or other outcomes (as measured, e.g., in terms of
equivalency in terms of extra days of learning). For a sense of how these
procedures work in practice, consider the following commentary by
Pogrow (2017):
The most common type of extrapolation is to equate the effect size to
changes in test scores with statements such as: ‘The size of the effect favoring the experimental students is the equivalent of moving students from the
50th to the 58th percentile on the Stanford Reading Test.’ However, this
seemingly impressive improvement is only a hypothetical relative extrapolation because we do not know whether students actually scored at the 58th
percentile or the 18th. In these types of hypothetical extrapolations in
research journals students may not have even actually taken the test mentioned or any nationally normed test. In addition, this type of extrapolation
assumes that the distribution of scores for students at risk is the same as
national norms, which is clearly not true. (pp. 10–11, emphasis in original)
52
D. B. EDWARDS JR.
While authors may state that their extrapolations are “only an estimate and
should be used as a general guide rather than as empirical transformations,” the fact remains that “no one knows precisely what effect sizes
generated from adjusted normalized relative data equate to in terms of
actual days of learning or test scores” (Pogrow, 2017, p. 11). And, as with
before, these extrapolations still do not tell us actual performance on the
underlying data collection instrument. In other words, we are not able to
make sense of the measurements in context not only because they have
been transformed to effect sizes but also because those effect sizes have
been further transformed to speculative statements of impact in terms that
are more familiar to practitioners or policymakers but which have no
grounding in reality. With each transformation, more doubt is cast on
claims as to the objectivity of impact evaluations.
Impact Evaluations and Organizational and Political
Incentives
Impact evaluations are also shaped by organizational and political constraints. While these dynamics should not be surprising, they are rarely
made explicit in relation to impact evaluations. The present section
endeavors to share insights in this area.
Organizational and Political Incentives
Scholars have been aware of organizational incentives for many years, since
at least the 1960s, the heyday of program and policy analysis of the US
federal government. At that time, the Bureau of the Budget required the
creation of research offices in each federal department and agency that
would be “interposed between the head of the agency and the operating
programs and were charged with undertaking analysis that would provide
a hard quantitative basis on which to make decisions” (Williams & Evans,
1969, p. 120). However, as Williams and Evans (1969) point out, the fact
of being separate from the program offices does not imply that evaluation
has been removed from organizational conflicts of interest since evaluators
will still be “serving the agency head, who, after all, is the program’s chief
manager” (p. 132). They go on to conclude that “evaluation, with its
potential for indicating that a program is not working, is a weapon of the
arsenal of analysis which is difficult to handle” (p. 132). We have no reason to believe that the World Bank or any other knowledge-producing
CRITICALLY UNDERSTANDING IMPACT EVALUATIONS: TECHNICAL…
53
organization would be completely immune to these same dynamics, as
discussed in Chap. 1.
Relatedly, there is reason to believe that organizational incentives affect
impact evaluations produced by non-governmental organizations (NGOs).
This is particularly the case when an NGO both implements an intervention and is charged with evaluating it or knows that an evaluation will be
conducted. Commonsensically, as Banerjee et al. (2017) state:
“Organizations who take the trouble to participate in a [RCT] would
rather demonstrate success” (p. 14). This desire can lead NGOs to select
areas for program implementation that are likely to yield better results.
NGO bias is further suggested by the findings of Vivalt (2015), who looks
at the results of over 400 RCTs and finds that the effects of NGO-­
implemented programs are consistently higher. It should not be surprising, then, that Rodrik (2008) concludes that the results of studies in which
NGOs participate “may be as problematic as those from clinical trials
undertaken by the pharmaceutical industry” (Rodrik, 2008, pp. 22–23).
Of course, the problem of bias does not only affect government evaluation
offices and NGOs but rather all organizations, as the more general discussion of the political economy of knowledge production in Chap. 1 makes
clear.
Professional and organizational incentives can also discourage knowledge production. This issue applies to the replication of studies by
researchers. Here, there is a tension between the attempt to produce useful knowledge and the incentives that drive researchers and academic journals, who benefit from the discovery and publication of unique (positive)
results (Meehl, 1997). As Rodrik (2008) describes:
It is hard to imagine that leading journals will be interested in publishing the
results of an identical experiment that differs along one or two dimensions:
perhaps it is a different locale, or perhaps the policy varies a bit, but in all
other ways, the experiment remains the same. The conditions under which
the repetition is most useful for purposes of external validity—repetition
under virtually identical conditions, save for one or two differences—are
precisely the conditions that will make it unappealing for purposes of professional advancement. (p. 22)
Thus, although impact evaluation advocates suggest that replication can
help with external validity, the incentives that most benefit researchers
(who are preoccupied with promotion and want to make a name for themselves) and journals (which seek to extend their prestige and readership)
54
D. B. EDWARDS JR.
are at the same time disincentives for the conduct of studies that could
shed light on the conditions under which a given intervention is likely to
work.26
Similarly, it has been pointed out that policymakers face disincentives
when it comes to approving or financing impact evaluations. Here,
Pritchett (2002) argues that policymakers may have motivation to underinvest in evaluations—particularly credible evaluations such as RCTs—
since negative results may affect their ability to fund their preferred
programs. Baker (2000) echoes this sentiment about the reason for reluctance on the part of government officials while also noting the impact of
turnover in officials, since new officials may decide to change course before
an evaluation has concluded, despite the investment of significant time
and expense in its design, preparation, and partial completion.
Cost
Cost is of course an important factor in whether impact evaluations are
undertaken. Despite this, there is very little information available about it.
What evidence exists suggests that, at times, costs can run into the millions, particularly in health research, even when random assignment is
done at the level of the community (as opposed to the level of the individual) in order to save money. Biglan, Ary, and Wagenaar (2000), for
example, estimate that an experiment on 11 pairs of communities in relation to an intervention to decrease smoking cost $45 million, while
another experiment focused on preventing adolescent alcohol use in 28
small rural US communities cost over $6 million.
In contrast with the above examples, Baker (2000) discusses eight
impact evaluations led by the World Bank on social and educational programs and reports that the average cost is much lower. However, even at
an average cost of $433,000, impact evaluations can represent a significant
expenditure, particularly when one recalls that the World Bank typically
works with middle- and low-income countries where budgets for social
services are already insufficient. Moreover, and importantly, it must be
noted that the cost estimates provided by Baker (2000)—which ranged
from $263,000 to $878,000—do not include the value of staff time contributed by the client country, though they do include the value of World
Bank staff time (with these latter costs built into the overall amount that a
country borrows from the World Bank to finance reform). The omission
of country staff time means that the real costs are actually much higher. In
CRITICALLY UNDERSTANDING IMPACT EVALUATIONS: TECHNICAL…
55
Baker’s (2000) words: “It was not unusual to have up to five staff assigned
to the evaluation effort for several years, a level of effort sufficient to substantially raise the cost of the evaluation in many of the cases” (p. 79). A
further implication is that the assignment of staff by country governments
to impact evaluations means that they were not available to work on other
issues.
Kremer (2003) suggests that one way to make impact evaluations (in
particular, RCTs) cheaper is to outsource them and to restrict their size.
More specifically, Kremer’s (2003) idea is to have NGOs implement the
intervention at a lower scale and then to conduct the RCT on the basis of
that restricted reach. However, this opens up the question, raised earlier,
of organizational incentives to ensure that positive results are found.
Feldman and Haskins (2016) likewise offer suggestions on how to reduce
the costs of RCTs, with these suggestions focusing on (a) the use of existing data and (b) the embedding of evaluations into regular program operations. Problematically, however, these recommendations assume that
(expensive and) well-developed systems of student-, teacher-, and/or
school-level data are already in operation. The second suggestion of
Feldman and Haskins (2016) also fails to address the political issues that
inevitably creep in when deciding which programs to evaluate, since it is
not feasible for practical reasons to evaluate all or most new programs, not
to mention the political difficulty of regularly selling the idea that new
programs be phased in so that certain students must wait (often a few
years) to participate in them, all in the name of accommodating the impact
evaluation design. Thus, while Feldman and Haskins’ (2016) proposals
may allow for savings in certain circumstances, it is not clear that their
recommendations are workable given the assumptions and political “costs”
associated with them.
Data Mining
The pressure felt by researchers to arrive at statistically significant results
can be great and can lead to the practice of data mining. Here, a researcher
may, in the case of RCTs, engage in subgroup analysis in order to find
some slice of the sample for which the intervention had an effect. Deaton
(2010) describes the tension that leads to this practice when he writes: “In
large-scale, expensive trials, a zero or very small result is unlikely to be
welcomed, and there is likely to be considerable pressure to search for
some subpopulation or some outcome that shows a more palatable result,
56
D. B. EDWARDS JR.
if only to help justify the cost” (p. 441). Moreover, in remarking on the
subsequent data mining that follows from such pressures, Deaton (2010)
further notes that “a sufficiently determined examination of any trial will
eventually reveal some subgroup for which the treatment yielded a significant effect of some sort” (p. 440).
Of course, the search for significant effects is not restricted to RCTs
and can affect impact evaluations that rely on regression analysis. Leamer
(1983) famously wrote that regression analysis is an art that “is practiced
at the computer … involves fitting many, perhaps thousands of statistical
models … [and concludes when] the researcher pulls from the bramble of
computer output the one thorn of a model he likes best, the one he
chooses to portray as a rose” (p. 36). In a more recent paper, Anderson
et al. (2001) provide additional insight into the kinds of practices that
characterize data mining (or, in their words “data dredging”):
An analysis is done and the results are studied by the investigator. Patterns
noticed in these analysis results are used to build more models for further
analysis (e.g., a non-significant variable is dropped; a plot reveals an unexpected, often nonlinear pattern between the response variable and a predictor variable; transformations are applied on either the response or predictor
variable to increase the statistical significance of a variable; a nearly significant interaction term is sometimes retained). This cycle is repeated until all
the significant variables are retained whereas others are removed. Inference
is then based on this final model. (p. 313)
Thus, as opposed to selecting variables and building models based on
guidance from theory or from previous research, and then restricting oneself to the analysis of those models (Flack & Chang, 1987), researchers
may instead engage in a range of steps that are not reported when the
results are presented. A major problem is that the reader usually has no
idea about the extent of the data mining that has occurred.
As a final example of data mining, consider the practice of “chasing low
p values.” Instead of adjusting the components of the overall model to
influence which variables appear as statistically significant, here, one
manipulates the underlying data in multiple ways to produce such an
effect. Again, Anderson et al. (2001) provide an instructive account:
An experienced researcher may conduct an appropriate, objective analysis
using a variety of statistical methods and subject matter knowledge of the
system under study. However, the same analysis methods can be used to
CRITICALLY UNDERSTANDING IMPACT EVALUATIONS: TECHNICAL…
57
chase small P-values (i.e., “significance”) under the guise of exploratory data
analysis. A hypothetical example is the use of a particular randomization test
where P=0.18. A parametric test was then applied, yielding P=0.09. The
deletion of 3 outliers brought the P-value down to 0.06. Then several transformations of the response variable were done in an attempt to reduce the
observed significance level. Finally, statistical significance was achieved at
P=0.04 and this was the result published. (p. 313)
Whether such practices are employed out of a desire to generate evidence
for one’s preferred theory, or in order to have a better chance at publication (since journals tend not to accept results that are not statistically significant), or in response to other organizational pressures, they are practices
that should be acknowledged. While the examples above may be caricatures, and while there is no way to know the extent to which such practices
are used, they are certainly used to some degree, if more mildly. Yet more
mild mining is still mining and still influences which programs are seen as
seen as successful. As such, although the above practices may be typically
utilized to a much lesser extent, they can have a dramatic impact on policy,
particularly when the practices are not reported and the results are presented as objective.
Conclusion
This chapter has brought together a range of critiques and concerns
related to impact evaluations that are usually discussed separately. That is,
this chapter has attempted to present for consideration a broader range of
issues than is typically addressed. This was done, first, by making explicit
the methodological assumptions, technical weaknesses, and practical
shortcomings of both regression analysis and RCTs. More specifically, the
chapter explicitly stated the conditions which must be met in order for
regression analysis and RCTs to yield accurate and credible estimates of
the impact of a given intervention. However, as detailed, the reality of
impact evaluations prevents these conditions from being met. For example, in regression analysis, it is rarely if ever possible to include of all the
relevant variables as statistical controls, let alone to measure them appropriately and to correctly model their interrelationships; as for RCTs, one is
not rarely able to achieve a balanced and representative sample from the
population of interest through random selection (in the case of RCTs).
Other problems highlighted were (a) that unbiasedness of estimates
58
D. B. EDWARDS JR.
(thanks to randomization) does not imply that a given study reveals the
true impact of an intervention and (b) that the focus on mean outcomes
in RCTs masks the variation of outcomes in the sample and may lead to
detrimental decisions.
Rounding out the first portion of the chapter was an extended discussion of generalizability in relation to RCTs. While RCTs strive to ensure
that their results are internally valid, we are not automatically able to say
anything about the transferability of findings to other locations.
Interestingly—and ironically—making the case for the applicability of
results to other locales requires that scholars resort to the kinds of comparisons and qualitative methods that RCT advocates sought to get away
from in the first place. To that end, it was suggested that both the producers and consumers of RCTs should strive to obtain a critical understanding
of the evaluation context, meaning an understanding which goes beyond
the surface level to include socio-cultural, structural, historical, and political aspects. An understanding of this nature is seen as necessary in order to
perceive the ways that these (often ignored) aspects of context can creep
into and can affect not only the behavior of study participants but also the
outcomes of focus.
Beyond practical issues, the chapter also sought to go deeper by unpacking some of the underlying procedures and concepts upon which impact
evaluations are built but which are not usually given a second thought,
such as the use of hypothesis testing, p values, and the normal curve. With
regard to the first of these, such issues as sample size sensitivity and the
“crud factor” were broached. Misconceptions about p values were then
clarified, including the fact that a small p value, for example, 0.05, does
not indicate that there is a 5% likelihood of one’s results given the data but
rather that one would expect to find results at least as large as those
observed in 5% of cases “if the study were repeated an infinite number of
times when in fact only pure random chance were at work” (Nuzzo, 2015,
p. 1). As explained, p values are related to the normal (aka bell) curve with
the problem here being that the distribution assumed in the normal curve
does not make sense as the basis for the calculation of outcome probabilities. The normal curve was developed to describe the chance of outcomes
when flipping a coin and to describe measurement error in data collection
for studies of astronomy. The implication is that this distribution was never
intended to describe social outcomes and, indeed, more recent research
has demonstrated that outcomes across a number of fields are not normally distributed. By extension, we must question the very foundation
CRITICALLY UNDERSTANDING IMPACT EVALUATIONS: TECHNICAL…
59
upon which statistical significance is calculated and against which the success of an intervention is judged.
In the final portion of the chapter, organizational-political dynamics
were addressed. These dynamics related, for example, to the incentives
and conflicts of interest which influence knowledge-producing organizations. Not only were evaluation findings shaded by personal and organizational interests, but it was also discussed how the separation of the
evaluation unit from the programmatic unit may not be sufficient to completely ensure objectivity since both units tend to respond to the head of
the organization. While cost is, as discussed, also a major factor in moving
forward with impact evaluations, the chapter concluded with what is a
particularly troublesome practice—namely, data mining. This practice
refers to a set of decisions and actions taken by researchers in the process
of data analysis that are taken to shape the reported outcomes. As the
examples shared indicate, the data manipulations of researchers can drastically change the results that an intervention registers. However, since the
strategies engaged by researchers in the search for significant results may
or may not be stated in final reports, it is impossible to know the extent
the results are credible.
Ultimately, although this chapter has covered a broad range of issues, it
is argued that we must still go farther if we want to understand the place
of impact evaluations within the context of global education reform more
generally. That is, while it is certainly necessary to grasp the methodological assumptions, technical limitations, and organizational-political dynamics that affect the production of impact evaluations, it is also important to
consider how impact evaluations operate to produce or enable subsequent
action in the field of global education policy. This latter issue was the focus
of the first few sections of Chap. 1, and it is an issue to which I return
again in the following chapter, which seeks to more explicitly define the
idea behind—and the methods that can be employed to carried out—a
critical review of impact evaluations within the political economy of global
education reform.
Notes
1. It should also be acknowledged that cost and the difficulty of executing such
studies are deterrents as well.
2. This problem is known as endogeneity.
60
D. B. EDWARDS JR.
3. Researchers do not always assume linear relationships among the variables,
as when they include squared terms or use logarithms. However, as discussed below, even when nonlinear relationships are modeled, there are
still inconsistencies across studies, and no case can we be sure that all of the
interrelationships have been modeled correctly to reflect their relationships
in the real world.
4. To be sure, accounting for all possible causes of social phenomena is a challenge faced by all methodologies. The point here is that a pretense of
regression analysis is that all variables are included if we are to get accurate
estimates of regression coefficients.
5. Although we do not deal here with variations of regression analysis, the
critiques outlined here apply to these variations and the methods that are
built upon them. These variations and other methods include regression
discontinuity, differences in differences, path analysis, hierarchical linear
modeling, and structural equation modeling, among others (Klees, 2016;
Schlotter, Schwerdt, & Woessmann, 2011)
6. Interestingly, Rodrik (2008) provides a personal anecdote about how standards have changed: “When I was a young assistant professor [in the
1980s], one could still publish econometric results in top journals with
nary a word on the endogeneity of regressors. If one went so far as to
instrument for patently endogenous variables, it was often enough to state
that you were doing [instrumental variables], with the list of instruments
tacked into a footnote at the bottom of a table. No more. A large chunk of
the typical empirical—but non-experimental—paper today is devoted to
discussing issues having to do with endogeneity, omitted variables, and
measurement error. The identification strategy is made explicit, and is
often at the core of the paper. Robustness issues take a whole separate section. Possible objections are anticipated, and counter-arguments are
advanced. In other words, considerable effort is devoted to convincing the
reader of the internal validity of the study” (p. 20).
7. This quote also mentions sample size, an issue that will be addressed later
in this chapter, in the section on more technical issues.
8. Selection bias can also occur when participants are allowed to self-select
into a treatment, as discussed further in the section on “randomizing an
unrepresentative sample.”
9. As evidence on this point, a meta-evaluation of 213 studies on school-­
based interventions (47% of which employed randomized designs) found
that less than 1% of treatment and control group were identical at the
outset (Durlak, Weissberg, Dymnicki, Taylor, & Schellinger, 2011).
10. Interestingly, Cartwright (2007, p. 20) suggests that expert judgment is
more useful for addressing threats to internal validity than is pursuit of
statistical controls.
CRITICALLY UNDERSTANDING IMPACT EVALUATIONS: TECHNICAL…
61
11. The four threats addressed here are all threats to internal validity. As noted
in this section, when these threats—and therefore internal validity—are an
issue, one cannot be sure that the effects observed are due to the intervention under study. While there are many more potential threats to internal
validity that affect both regression analysis and RCTs (see, e.g., Mertens,
2005, pp. 121–124), these four have been addressed here because they can
affect RCTs in spite of randomization.
12. Although it doesn’t solve the problem of outliers, researchers often look at
subpopulations within the sample (e.g., according to gender, ethnicity,
class, etc.) to see how outcomes vary across them.
13. For additional discussion of threats to internal and external validity see, for
example, Mertens (2005). The goal here is not to speak to all possible
threats to internal and external validity but rather to address some of the
problematic assumptions and practices that are associated with RCTs in
practice.
14. In a point that connects with the next subsection, Deaton and Cartwright
(2016) underscore that “without a clear idea of how to characterize the
population of individuals in the trial, whether we are looking for an [average treatment effect] or to identify causality, and for which groups enrolled
in the trial the results are supposed to hold, we have no basis for thinking
about how to use the trial results in other contexts” (p. 27).
15. See conclusion chapter for discussion of alternative methods, including
realist evaluation, which focuses on understanding the mechanisms and
contexts in which interventions are likely to work, as opposed to focusing
only on the outcomes of an intervention, as is the case with impact
evaluations.
16. To be clear, Klees (2016) argues that quantitative data are still important
but that we can only rely on simple correlations and cross tabulations.
17. The language of rejecting the null hypothesis might sound odd, but as
Steidl, Hayes, and Schauber (1997) remind us: “In the framework of the
hypothetico-deductive method, research hypotheses can never be proven;
rather, they can only be disproved (rejected) with the tools of statistical
inference” (p. 271).
18. Another way of stating this definition would be mean difference/standard
error of mean difference.
19. Carver (1978) make a further and particularly interesting point on
researcher discretion when it comes to sample size: “Since the researcher
often has complete control over the number of subjects sampled, one of
the most important variables affecting the results of the research, the subjective judgment of the experimenter in choosing sample size, is usually
not controlled. Controlling experimenter bias is a much discussed problem, but not enough is said about the experimenter’s ability to increase the
62
D. B. EDWARDS JR.
odds of getting statistically significant results simply by increasing the number of subjects in an experiment” (p. 385).
20. For additional discussion of the shortcomings of hypothesis testing, see
Levine et al. (2008).
21. See Steidl et al. (1997) for a good discussion of power analysis.
22. O’Boyle and Aguinis (2012) suggest instead the use of the Paretian (power
law) distribution.
23. On test results and normality, Goertzel (n.d.) explains: “If a large number
of people fill out a typical multiple choice test such as the Scholastic
Aptitude Test (or a typical sociological questionnaire with precoded
responses such as ‘strongly agree, agree’) at random using a perfect die, the
scores are very likely to be normally distributed. This is true because many
more combinations of response give a sum that is close to the theoretical
mean than give a score that is close to either extreme” (p. 6).
24. This is done using the analysis of covariance statistical procedure. Sometimes
additional transformations of the data are made by converting outcomes to
normalized Z scores in order to compare outcomes from different tests.
See Pogrow (2017) for more.
25. See Pogrow (2017) for a discussion of examples of where small effect sizes
have been highlighted by researchers and the media and in ways that contradict how effect sizes should be interpreted.
26. It is because journals are more willing to publish evaluations with positive
or negative results rather than null results that one must also have caution
when considering the findings of meta-analysis or literature reviews, a
point that has been made by Glewwe (2014).
References
Anderson, D., Burnham, K., Gould, W., & Cherry, S. (2001). Concerns about
finding effects that are actually spurious. Wildlife Society Bulletin, 29(1),
311–316.
Baker, J. (2000). Evaluating the impact of development projects on poverty: A handbook for practitioners. Washington, DC: World Bank.
Banerjee, A., Banerji, R., Berry, J., Duflo, E., Kannan, H., Mukerji, S., Shotland,
M., & Walton, M. (2017). From proof concept to scalable policies: Challenges and
solutions, with an application. NBER Working Paper No. 22931. Retrieved
from https://economics.mit.edu/files/12359
Banerjee, A., & He, R. (2008). Making aid work. In W. Easterly (Ed.), Reinventing
foreign aid (pp. 47–92). Cambridge, MA: MIT.
Berkson, J. (1938). Some difficulties of interpretation encountered in the application of the chi-square test. Journal of the American Statistical Association,
33(203), 526–536.
CRITICALLY UNDERSTANDING IMPACT EVALUATIONS: TECHNICAL…
63
Biglan, A., Ary, D., & Wagenaar, A. (2000). The value of interrupted time-series
experiments for community intervention research. Prevention Science, 1(1), 31–49.
Boruch, R., Rindskopf, D., Anderson, P., Amidjaya, I., & Jansson, D. (1979).
Randomized experiments for evaluating and planning local programs: A summary on appropriateness and feasibility. Public Administration Review, 39(1),
36–40.
Braun, A., Ball, S., Maguire, M., & Hoskins, K. (2011). Taking context seriously:
Towards explaining policy enactments in the secondary school. Discourse:
Studies in the Cultural Politics of Education, 32(4), 585–596.
Burde, D. (2012). Assessing impact and bridging methodological divides:
Randomized trials in countries affected by conflict. Comparative Education
Review, 56(3), 448–473.
Cartwright, N. (2007). Are RCTs the gold standard? Centre for Philosophy of
Natural and Social Science. Technical Report 01/07. London: London School
of Economics. Retrieved from http://www.lse.ac.uk/CPNSS/research/concludedResearchProjects/ContingencyDissentInScience/DP/Cartwright.pdf
Carver, R. (1978). The case against statistical significance testing. Harvard
Educational Review, 48(3), 378–399.
Carver, R. (1993). The case against statistical significance testing, revisited. The
Journal of Experimental Education, 61(4), 287–292.
Castillo, N., & Wagner, D. (2013). Gold standard? The use of randomized controlled trials for international educational policy. Comparative Education
Review, 58(1), 166–173.
Clay, R. (2010). More than one way to measure: Randomized clinical trials have
their place, but critics argue that researchers would get better results if they also
embraced other methodologies. Monitor on Psychology, 41 (8), 52. Retrieved
from http://www.apa.org/monitor/2010/09/trials.aspx
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale,
NJ: Lawrence Erlbaum.
Concato, J., Shah, N., & Horwitz, R. (2000). Randomized, controlled trials,
observational studies, and the hierarchy of research designs. The New England
Journal of Medicine, 342(25), 1887–1892.
Cook, T. (2001). Science phobia: Why education researchers reject randomized
experiments. Education Next, (Fall), 63–68. Retrieved from http://www.indiana.edu/~educy520/readings/cook01_ed_research.pdf
Cook, T. (2004). Why have educational evaluators chosen not to do randomized
experiments? Annals of the American Academy of Political and Social Science,
589(Sep.), 114–149.
Deaton, A. (2010). Instruments, randomization, and learning about development. Journal of Economic Literature, 48(2), 424–455.
Deaton, A., & Cartwright, N. (2016a). The limitations of ramdomised controlled
trials. Vox. Retrieved from http://voxeu.org/article/limitations-randomisedcontrolled-trials
64
D. B. EDWARDS JR.
Deaton, A., & Cartwright, N. (2016b). Understanding and misunderstanding
randomized controlled trials. NBER Working Paper No. 22595. Retrieved from
http://www.nber.org/papers/w22595
De Boer, M., Waterlander, W., Kujper, L., Steenhuis, I., & Twisk, J. (2015).
Testing for baseline differences in randomized controlled trials: An unhealthy
research behavior that is hard to eradicate. International Journal of Behavioral
Nutrition and Physical Activity, 12(4). Retrieved from https://ijbnpa.biomedcentral.com/articles/10.1186/s12966-015-0162-z
Duflo, E., & Kremer, M. (2003, July 15–16). Use of randomization in the evaluation of development effectiveness. Paper prepared for the World Bank Operations
Evaluation Department Conference on Evaluation and Development
Effectiveness, Washington, DC. Retrieved from https://economics.mit.edu/
files/765
Durlak, J. (2009). How to select, calculate, and interpret effect sizes. Journal of
Pediatric Psychology, 34(9), 917–928.
Durlak, J., Weissberg, R., Dymnicki, A., Taylor, R., & Schellinger, K. (2011). The
impact of enhancing students’ social and emotional learning: A meta-analysis of
school-based universal interventions. Child Development, 82(1), 405–432.
Everett, B., Rehkopf, D., & Rogers, R. (2013). The nonlinear relationship
between education and mortality: An examination of cohort, race/ethnic, and
gender differences. Population Research Policy Review, 32 (6). Retrieved from
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3839428/
Feldman, A. & Haskins, R. (2016). Low-cost randomized controlled trials.
Evidence-Based Policymaking Collaborative. Retrieved from http://www.evidencecollaborative.org/toolkits/low-cost-randomized-controlled-trials
Fendler, L., & Muzzafar, I. (2008). The history of the bell curve: Sorting and the
idea of normal. Educational Theory, 58(1), 63–82.
Flack, V., & Chang, P. (1987). Frequency of selecting noise variables in subset
regression analysis: A simulation study. The American Statistician, 41(1),
84–86.
Freedman, D. (1983). A note on screening regression equations. The American
Statistician, 37(2), 152–155.
Ganimian, A. (2017). Not drawn to scale? RCTs and education reform in developing countries. Research on improving systems of education. Retrieved from
http://www.riseprogramme.org/content/not-drawn-scale-rcts-and-educationreform-developing-countries
Garbarino, S., & Holland, J. (2009). Quantitative and qualitative methods in
impact evaluation and measuring results. Governance and Social Development
Resource Centre. UK Department for International Development. Retrieved
from http://www.gsdrc.org/docs/open/eirs4.pdf
Gertler, P., Martinez, S., Premand, P., Rawlings, L., & Vermeersch, C. (2016).
Impact evaluation in practice (2nd ed.). Washginton, DC: World Bank.
CRITICALLY UNDERSTANDING IMPACT EVALUATIONS: TECHNICAL…
65
Gigerenzer, G., Swijtink, Z., Porter, T., Datson, L., Beatty, J., & Kruger, L.
(1989). The empire of chance: How probability changed science and everyday life.
New York: Cambridge University Press.
Ginsburg, A., & Smith, M. (2016). Do randomized controlled trials meet the “gold
standard”? A study of the usefulness of RCTs in the What Works Clearinghouse.
American Enterprise Institute. Retrieved from https://www.carnegiefoundation.org/wp-content/uploads/2016/03/Do-randomized-controlled-trialsmeet-the-gold-standard.pdf
Glewwe, P. (Ed.). (2014). Education policy in developing countries. Chicago:
University of Chicago.
Goertzel, T. (n.d). The myth of the bell curve. Retrieved from http://crab.rutgers.edu/~goertzel/normalcurve.htm
Gorard, S., & Taylor, C. (2004). Combining methods in educational and social
research. New York: Open University.
Greenland, S., Senn, S., Rothman, K., Carlin, J., Poole, C., Goodman, S., &
Altman, D. (2016). Statistical tests, P values, confidence intervals, and power:
A guide to misinterpretations. European Journal of Epidemiology, 31(4),
337–350.
Johnson, D. (1999). The insignificance of statistical significance testing. Journal of
Wildlife Management, 63(3), 763–772.
Klees, S. (2016). Inferences from regression analysis: Are they valid? Real-World
Economics Review, 74, 85–97. Retrieved from http://www.paecon.net/
PAEReview/issue74/Klees74.pdf
Klees, S., & Edwards, D. B., Jr. (2014). Knowledge production and technologies
of governance. In T. Fenwick, E. Mangez, & J. Ozga (Eds.), World yearbook of
education 2014: Governing knowledge: Comparison, knowledge-based technologies
and expertise in the regulation of education (pp. 31–43). New York: Routledge.
Komatsu, H., & Rappleye, J. (2017). A new global policy regime founded on
invalid statistics? Hanushek, Woessmann, PISA, and economic growth.
Comparative Education, 53(2), 166–191.
Kremer, M. (2003). Randomized evaluations of educational programs in developing countries: Some lessons. The American Economic Review, 93(2), 102–106.
Lareau, A. (2009). Narrow questions, narrow answers: The limited value of randomized controlled trials for education research. In P. Walters, A. Lareau, &
S. Ranis (Eds.), Education research on trial: Policy reform and the call for scientific rigor (pp. 145–162). New York: Routledge.
Leamer, E. (1983). Let’s take the con out of econometrics. The American Economic
Review, 73(1), 31–43.
Leamer, E. (2010). Tantalus on the road to asymptopia. The Journal of Economic
Perspectives, 24(2), 31–46.
Levine, T., Weber, R., Hullett, C., Park, H., & Lindsey, L. (2008). A critical
assessment of null hypothesis significance testing in quantitative communication research. Human Communication Research, 34(2), 171–187.
66
D. B. EDWARDS JR.
Levy, S. (2006). Progress against poverty: Sustaining Mexico’s Progresa-­
Oportunidades program. Washington, DC: Brookings Institution Press.
Luecke, D., & McGinn, N. (1975). Regression analyses and education production
functions: Can they be trusted? Harvard Educational Review, 45(3), 325–350.
Lykken, D. (1968). Statistical significance in psychological research. Psychological
Bulletin, 70(3), 151–159.
McLaughlin, M. (1987). Learning from experience: Lessons from policy implementation. Educational Evaluation and Policy Analysis, 9(2), 171–178.
McLaughlin, M. (1990). The Rand change agent study revisited: Macro perspectives and micro realities. Educational Researcher, 19(9), 11–16.
Meehl, P. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and
slow progress of soft psychology. Journal of Consulting and Clinical Psychology,
46(4), 806–834.
Meehl, P. (1986). What social scientists don’t understand. In D. Fiske &
R. Shweder (Eds.), Metatheory in social science: Pluralisms and subjectivities
(pp. 315–338). Chicago: University of Chicago.
Meehl, P. (1997). The problem is epistemology, not statistics: Replace significance
tests by confidence intervals and quantify accuracy of risky numerical predictions. In L. Harlow, S. Mulaik, & J. Steiger (Eds.), What if there were no significance tests? (pp. 393–425). Mahwah, NJ: Erlbaum.
Mertens, D. (2005). Research and evaluation in education and psychology:
Integrating diversity with quantitative, qualitative, and mixed methods (2nd
ed.). London: Sage.
Miguel, E., & Kremer, M. (2004). Worms: Identifying impacts on education and
health in the presence of treatment externalities. Econometrica, 72(1), 159–217.
Novella, S. (2016). P value under fire. Science-Base Medicine. Retrieved from
https://sciencebasedmedicine.org/p-value-under-fire/
Nuzzo, R. (2015). Scientists perturbed by loss of stat tools to sift research fudge
from fact. Scientific American. Retrieved from https://www.scientificamerican.com/article/scientists-perturbed-by-loss-of-stat-tools-to-sift-researchfudge-from-fact/
O’Boyle, E., & Aguinis, H. (2012). The best and the rest: Revisiting the norm of
normality of individual performance. Personnel Psychology, 65(1), 79–119.
Peck, J., & Theodore, N. (2015). Fast policy: Experimental statecraft at the thresholds of neoliberalism. Minneapolis: University of Minnesota.
Pogrow, S. (2017). The failure of the U.S. education research establishment to
identify effective practices: Beware effective practices policies. Education Policy
Analysis Archives, 25(5), 1–19. Retrieved from https://epaa.asu.edu/ojs/article/view/2517
Pritchett, L. (n.d.). “The evidence” about “what works” in education: Graphs to illustrate external validity and construct validity. Research on Improving Systems of
Education. Retrieved from https://www.cgdev.org/publication/evidence-aboutwhat-works-education-graphs-illustrate-external-validity
CRITICALLY UNDERSTANDING IMPACT EVALUATIONS: TECHNICAL…
67
Pritchett, L. (2002). It pays to be ignorant: A simple political economy of rigorous
program evaluation. The Journal of Policy Reform, 5(4), 251–269.
Rodrik, D. (2008). The development economics: We shall experiment, but how shall
we learn? Faculty Research Working Papers Series. RWP08-055. John
F. Kennedy School of Government. Harvard University. Retrieved from
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1296115
Romero, M., Sandefur, J., & Sandholtz, W. 2017. Can outsourcing improve
Liberia’s schools? Preliminary results from year one of a three-year randomized
evaluation of partnership schools for Liberia. Washington, DC: Center for Global
Development.https://www.cgdev.org/sites/default/files/partnership-schools-forliberia.pdf
Rust, V., Soumare, A., Pescador, O., & Shibuya, M. (1999). Research Strategies in
Comparative Education. Comparative Education Review, 43(1), 86–109.
Sanson-Fisher, R., Bonevski, B., Green, L., & D’Este, C. (2007). Limitations of
the randomized controlled trial in evaluating population-based health interventions. American Journal of Preventative Medicine, 33(2), 155–161.
Schlotter, M., Schwerdt, G., & Woessmann, L. (2011). Econometric methods for
causal evaluation of education policies and practices: A non-technical guide.
Education Economics, 19(2), 109–137.
Schanzenbach, D. (2012). Limitations of experiments in education research.
Education Finance and Policy, 7(2), 219–232.
Steidl, R., Hayes, J., & Schauber, E. (1997). Statistical power analysis in wildlife
research. The Journal of Wildlife Management, 61(2), 270–279.
Uriel, E. (2013). Hypothesis testing in the multiple regression model. Retrieved from
https://www.uv.es/uriel/4%20Hypothesis%20testing%20in%20the%20multiple%20regression%20model.pdf
Vivalt, E. (2015). How much can we generalize from impact evaluations? New York
University. Retrieved from https://pdfs.semanticscholar.org/6545/a87feaec7d6d0ba462860b3d1bb721d9da39.pdf
Wang, L., & Guo, K. (2018). Shadow education of mathematics in China. In
Y. Cao & F. Leung (Eds.), The 21st century mathematics education in China
(pp. 93–106). Berlin: Springer.
Weiss, R., & Rein, M. (1970). The evaluation of broad-aim programs: Experimental
design, its difficulties, and an alternative. Administrative Science Quarterly,
15(1), 97–109.
Williams, W., & Evans, J. (1969). The politics of evaluation: The case of Head
Start. Annals of the American Academy of Political and Social Sciences, 385,
118–132.
CHAPTER 3
Situating a Critical Review of Impact
Evaluations Within the Political Economy
of Global Education Reform: Definition
and Method
Abstract This chapter addresses issues of definition and method. The first
section takes on the challenge of defining what it means to conduct a critical review of impact evaluations within the political economy of global
education reform. As discussed, one must go beyond a critical evaluation
of the findings of impact evaluations and beyond a critical appraisal of the
production of impact evaluations to also consider the role that impact
evaluations play, once produced, in the field of global education policy.
Considering this role means examining their influence at the level of policymaking processes, organizational agendas, personal careers, and discursive context, among others. The latter sections of this chapter suggest
methods that can be used to put this approach to critical review into practice. The specifics of how these methods were employed in the policy case
from El Salvador (the focus of Chaps. 4, 5, and 6) are also detailed.
Keywords Knowledge production • Impact evaluation • Political
economy • Global education policy • Critical review • Methods
As the title implies, the present chapter addresses issues of definition and
method. The first section takes on the challenge of defining what it means
to conduct a critical review of impact evaluations within the political economy of global education reform. The latter sections of this chapter then
© The Author(s) 2018
D. B. Edwards Jr., Global Education Policy, Impact Evaluations, and
Alternatives, https://doi.org/10.1007/978-3-319-75142-9_3
69
70
D. B. EDWARDS JR.
suggest methods that can be used to put this approach to critical review
into practice. The specifics of how these methods were employed in the
policy case from El Salvador (the focus of Chaps. 4, 5, and 6) are also
detailed.
Definition
The approach defined here builds on the insights from Chaps. 1 and 2.
Specifically, as elaborated at the beginning of Chap. 1, the political economy perspective on the production of knowledge in the field of global
education policy centers on the assumptions that international organizations (but particularly those that are involved in knowledge production)
are animated more by certain ideas or policies than others, that these organizations look for ways to ensure stability and longevity, and that the goals
of stability and longevity are more likely to be achieved when their knowledge products are perceived by others as credible and relevant. It was
furthermore stated in Chap. 1 that—although international organizations
are proactive actors that make strategic calculations about the nature of
their knowledge production activities in order to remain relevant and
influential—they also must respond to the variegated nature of the field of
global education policy and to the fact that this field is additionally subject
to larger political-economic constraints beyond its control. Chapter 2 then
turned its attention to a discussion of the limitations and challenges that
arise in the production of impact evaluations themselves. The present
chapter brings together these various issues and endeavors to delineate
more completely and more specifically what it means to engage in a critical
review of impact evaluations with the political economy of global education reform.
Stated straightforwardly, a critical review of impact evaluations within
the political economy of global education reform means assessing a set of
evaluative studies that have been carried out on a policy or program not
only in relation to the methodological limitations of the studies but also in
relation to the meaning or significance of those studies for policy, for practice, and for the vested interests that may be advanced (or not) on a global
scale from their production and the findings they contain. As summarized
in Table 3.1, realizing this kind of review clearly means going beyond
(a) a consideration of the stated results and methodological limitations of
a corpus of studies, though this is an important step (as, e.g., in Bryk,
1981); it also means (b) understanding those studies within the multi-level
SITUATING A CRITICAL REVIEW OF IMPACT EVALUATIONS…
71
Table 3.1 Goals of a critical review of impact evaluations within the political
economy of global education reform
Goal Description
(a)
(b)
(c)
(d)
Consider the stated results and methodological limitations of a corpus of studies,
though this is an important step
Understand studies within the multi-level (i.e., international, national, local)
political-financial-intellectual complex out of which they emerged
Appraise the implications of those studies (in terms of their stated findings but also
in terms of their political meaning) for the vested actors and institutions that
facilitated, utilized, and/or otherwise benefitted from the studies
Tend to the ways that the studies contribute to oppressing certain perspectives or
policy preferences while elevating others at each level from the local to the global
(i.e., international, national, local) political-financial-intellectual complex
out of which they emerged; (c) appraising the implications of those studies
(in terms of their stated findings but also in terms of their political meaning) for the vested actors and institutions that facilitated, utilized, and/or
otherwise benefitted from the studies; and (d) tending to the ways that the
studies contribute to oppressing certain perspectives or policy preferences
while elevating others at each level from the local to the global (Klees &
Edwards, 2014; Lubienski, Weitzel, & Lubienski, 2009; Robertson &
Dale, 2009; Samoff, 1993, 2012; Verger, 2012).1 The present manuscript
engages with these steps in relation to the series of impact evaluations that
were produced by the World Bank of the Education with Community
Participation (EDUCO) program in El Salvador.
It is suggested that, ultimately, the insights gained through such an
analysis are more significant than asking whether a program worked or
whether a study’s findings hold up to a critical review of their methodological shortcomings alone. The added benefit of taking a critical review
of empirical findings (based, e.g., on their methodological basis) and placing those findings within the political economy of education reform and
the political economy of research production is that it shifts our gaze away
from an exclusive focus on research findings as if they were the product of
a clinical and objective process to instead focus on how those findings may
reproduce existing power structures and/or advance certain policy preferences over others; or, put differently, we are encouraged to look anew at
how and why research findings, by virtue of how they are produced and
the power relations that govern their production, may disadvantage ­certain
72
D. B. EDWARDS JR.
groups and with what implications (Water, 1998). Arguably, carrying out
such reviews is becoming more imperative, particularly as the relationships
among policymakers, funders, evaluators, publishers, and promoters of
research and “policy-relevant” findings continue to evolve and to blur on
a global scale and in ways that can involve a conflict of interest wherein
those who fund and carry out research are often also the beneficiaries of
that research (Ball, 2009, 2010; Goldie, Linick, Jabbar, & Lubienski,
2014; Klees & Edwards, 2014; Lubienski, Scott, & DeBray, 2014;
Nambissan & Ball, 2010; Scott & Jabbar, 2014; Verger, 2012).
In practice, then, in order to follow through with the type of critical
review described above, one must move from the macro to the micro and
then from the micro back to the macro, as depicted in Fig. 3.1 and as summarized in Table 3.2. That is to say, in going from the macro to the micro,
one must understand (a) the historical-structural aspects of the context in
question and, in particular, how they constrain the actors in that context;
(b) the range of involved actors and the ways they are situated within the
larger context, as well as their policy preferences; (c) the relevant processes
or opportunities for policy change that emerge, in addition to the rules,
requirements, and expectations that govern those processes; (d) the production and utilization of “legitimate” and “policy-relevant” knowledge
(in this case, impact evaluations), including who generated and funded
that knowledge; and (e) the findings of the relevant studies not only as
stated by their authors but also in terms of what one can reasonably claim
to know after considering the methodological and practical limitations
that accompany the production of the studies in question.
At this point, one not only has an understanding of the empirical findings of a body of research but also has a grasp of the political significance
of that research and the ways that it has been employed. Subsequently, in
going from the micro to the macro, one begins by (f) clearly problematizing that research in technical terms (where appropriate) and then considering the implications of both the stated findings and their critical
reinterpretation (i.e., what one can reasonably claim to know in light of
their limitations) for a number of issues, including (g) processes of policymaking or educational reform, (h) the marginalization of certain political
or alternative perspectives, (i) the advancement of political and institutional interests, (j) the emergence or prevalence of global reform trends,
and, at the most broad level, (k) the maintenance or extension of the
prevailing ideology or common sense in a certain area (in this case, educational governance). In working from the macro to the micro and then
SITUATING A CRITICAL REVIEW OF IMPACT EVALUATIONS…
1
Competing
Ideologies
73
2
Discursive
Context
3
International Context:
Global Governace, Regional Structure, Transnational Social
Movements, International Trends, Development Goals,
International Political Economic Relations
4
Country Context:
Historical, Structural & Political-Economic Constraints:
Local and national Level
5
Organizational Context:
Local, National, International
Impact
Evaluations2
6
Strategic Selective Individuals3
7
Education Policy
Policymaking Processes
Fig. 3.1 Context and potential trajectory of impact evaluations within the political economy of global education reform.1 Notes: (1) In practice, these levels are
intertwined. They are distinguished here for conceptual clarity. (2) The influence
of impact evaluations depends on many factors, such as those in this figure. Based
on how they align or not, the impacts will have more or less influence at each level
depicted. (3) Strategically selective individuals are “agents [who] make structurally
oriented strategic calculations which they put into practice through recursively
selected strategies and tactics (Jessop, 2001, p. 1224). That is, these actors “orient
their strategies and tactics in the light of their understanding of the current conjuncture and their ‘feel for the game’” (p. 1224)
back out to the macro, one is concerned with a range of intersecting
dimensions. It is not only structure, agency, or ideational context that
matters but rather the insights that result from a focus on all three. While
these aspects can be distinguished conceptually and visually (as in Fig. 3.1),
they are also nested, with the structural and ideational dimensions affects
the others in various says. While other authors have arguably employed
this approach when analyzing research findings (e.g., Edwards, 2012b;
74
D. B. EDWARDS JR.
Table 3.2 Tasks of critical review of impact evaluations within the political economy of global education reform
Task Description—methods for review ideally enable one to understand and assess
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
(k)
The historical-structural aspects of the context in question and, in particular, how
they constrain the actors in that context
The range of involved actors and the ways they are situated within the larger
context, as well as their policy preferences
The relevant processes or opportunities for policy change that emerge, in addition
to the rules, requirements, and expectations that govern those processes
The production and utilization of “legitimate” and “policy-relevant” knowledge (in
this case, impact evaluations), including who generated and funded that knowledge
The findings of the relevant studies not only as stated by their authors but also in
terms of what one can reasonably claim to know after considering the
methodological and practical shortcomings that accompany the production of the
studies in question
The technical limitations of the stated findings
The impact of the reviewed studies for processes of policymaking or educational
reform
The impact of the reviewed studies for the marginalization of certain political or
alternative perspectives
The impact of the reviewed studies for the advancement of political and
institutional interests
The impact of the reviewed studies for the emergence or prevalence of global
reform trends
The impact of the reviewed studies for the maintenance or extension of the
prevailing ideology or common sense in a certain area
The data collected made it possible to understand and assess (a), (b), (c), (d), (e), and (f), in addition to
understanding and assessing (h), (i), (j) and, at the most broad level, (k) (in this case, educational
governance)
Klees & Wells, 1983; Lubienski et al., 2009; Samoff, 1993; Verger, 2012),
it is hoped that specifying the process of analysis and review will make it
more clear and accessible for additional scholars to utilize.
Methods: Putting Knowledge in Its Place
The above discussion thus indicates that the review presented in this manuscript is based on much more than a review of impact evaluations alone,
and necessarily so, since the purpose of this study is to understand and
view those studies in relation to the larger reform context. To that end, the
findings discussed here have resulted from research conducted during a
SITUATING A CRITICAL REVIEW OF IMPACT EVALUATIONS…
75
six-year period, January 2009–January 2015 (see, e.g., Edwards, 2012b,
2013a, 2013b, 2018; Edwards & DeMatthews, 2014; Edwards & Klees,
2012; Edwards, Victoria, & Martin, 2015). As summarized in Table 3.3,
this period contained various phases of data collection and analysis.
Throughout, literature review, document and archival review, interviews,
and bibliographic methods were used to complement one another.
Two points are in order with regard to the data collected. First, and
perhaps most importantly, on the basis of the data collected, it was possible to accomplish the full range of analytic tasks set out in previous section
as being essential to placing a critical review within the political economy
of global education reform. Second, as can be seen in Table 3.3, neither
the collection of contextual and background data nor the critical review of
studies took place during a single point in time. Rather, the foundation of
the present study was a process of alternating phases wherein each phase
was targeted to specific data collection needs or analytic steps. In the end,
the combination of methods employed has afforded the author an in-­
depth understanding of the constraints on both El Salvador and its education system from the late 1970s through to 2014. This understanding
includes intimate knowledge of the processes of education reform, the key
institutional actors that participated in them (as well as their policy preferences), and the ways these actors sought to influence education reform.
Moreover, and crucially for the present critical review, the methods
employed allowed the author not only to identify and locate all impact
evaluations that had been conducted on the EDUCO program but also to
unpack, for example, how these studies were perceived and leveraged by
various actors, in addition to how they have taken on a life of their own, as
will be explained later in this manuscript. Not surprisingly, it was necessary
to engage in multiple periods of data collection. Each round of data collection served to eliminate remaining blind spots, thereby allowing further
insights into the meaning and significance of the impact evaluations being
critically reviewed.
Data Collection
It is necessary to comment more specifically on the nature of the data collected and how they were scaffolded to meet the needs of this study.
Although the stages of data collection were iterative, the following discussion treats the collection of each kind of data collection in isolation for simplicity. Most basically, background literature was thoroughly consulted from
76
D. B. EDWARDS JR.
Table 3.3 Summary of data collection phases, critical review periods, and corresponding analytic tasks
Phase
Description
Corresponding
analytic task(s)
1. January
2009
Initial data collection for two case studies of school-­
community relationships around EDUCO within
history and context of education reform during
1980s–2000s and data collected in form of
ethnographic observations, interviews, policy
documents, and background literature
Literature review on education politics in El Salvador
and collection and review of documents from multiple
organizations related to education reform during
2003–2009, in addition to 17 interviews with senior
representatives of key organizations in the policy
process
Review of literature on education reform in El Salvador
and on relevant global trends (i.e., decentralization and
participation in education governance) in post-WWII
context
First round of critical review of World Bank impact
evaluations
Review of World Bank literature on the role of
participation in education governance during
1980–2010
Extensive interviews with 109 local, national, and
international actors involved education reform and
implementation between the late 1980s and early
2000s in El Salvador, including interviews with key
staff of international organizations related to the role
of EDUCO studies, document collection and review of
archives from organizations involved in education
reform during same period, bibliographic searching to
identify use of EDUCO studies, literature review on
political-economic context of El Salvador, and
follow-up data collection for two case studies of
school-community relationships around EDUCO
Second round of critical review of World Bank impact
evaluations
Review of World Bank archives related to engagement
in El Salvador
(a), (b)
2. September
2009–April
2010
3. November
2010–
January 2011
4. February
2011
5. March
2011
6. August
2011–
September
2012
7. January
2013
8. September
2013
(b), (c), (d)
(a), (b)
(d), (e), (f)
(b)
(a), (b), (c), (d),
(g), (h), (i), (j),
(k)
(e), (f)
(a), (b), (c)
(continued)
SITUATING A CRITICAL REVIEW OF IMPACT EVALUATIONS…
Table 3.3
77
(continued)
Phase
Description
Corresponding
analytic task(s)
9. August–
September
2014
10.
November
2014
11. January
2015
Interviews with authors of EDUCO impact evaluations (d), (e), (f), (i),
(j)
12. August
2015
13.
September
2015
(e), (g), (h), (i),
(j), (k)
(h), (i), (j), (k)
Review of World Bank archives related to engagement
in El Salvador
(a), (b), (c)
Interviews with 29 local, national, and international
actors related to education reform processes and
politics in El Salvador during 1990–2014
Third round of critical review of World Bank impact
evaluations
Bibliographic searching to identify use of EDUCO
studies
(a), (b), (c), (d)
Key for analytic tasks: see Table 3.2
political economists, political scientists, Latin-American studies’ specialists,
historians, international relations experts, sociologists, and education scholars. This data source provided rich insights—both broad and specific—into
the historical-structural and political-economic limitations around education reform in El Salvador, as well as the nature of the education system
itself, with an emphasis on the period from the 1970s through the 2000s,
which corresponds to the years of focus for this study.
More generally, literature was systematically collected and reviewed on
the decentralization of education governance in the post-WWII period in
order to understand the larger, international trends within which EDUCO,
as a form of decentralization, was situated (see Chap. 5 for additional
comments on these trends). Seven top journals from the field of comparative and international education were reviewed during phase two of the
present study in addition to the Harvard Educational Review, the Journal
of Education Policy, and the Journal of Educational Administration. These
journals were chosen because, together, the articles in them reflect trends
in thinking around education governance. In all, 126 studies were
reviewed; a full list can be found in Edwards and DeMatthews (2014).
Relatedly, institutional literature from the World Bank on the issue of
participation in educational governance was also carefully reviewed ­(during
78
D. B. EDWARDS JR.
phase five). This literature was salient for three reasons: first, because, as
will be explained later, “participation” by citizens in the governance of
social services was a key theme in international development in the 1980s
and 1990s, and one that was invoked to support community management
reforms like EDUCO; second, because the World Bank was a leading proponent of this idea; and, third, because, in El Salvador, the World Bank
was the institution at the forefront of public sector “modernization”
(through which governance reforms were introduced) during the period
of interest in this study (but particularly during the late 1980s and 1990s).
Three pools of institutional literature were reviewed, including all annual
World Development Reports (the flagship publication of the World Bank
that highlights and promotes themes of interest to this institution) from
1980 to 2010, all official education strategy documents and, lastly, literature produced by the World Bank and its staff related to participation in
education governance specifically. These three pools provided key insights
into the reform preferences and policy logic of this influential institution
and its education staff—insights which would be essential for understanding the dynamics out of which EDUCO emerged in the early 1990s (see
Edwards (2012b) for more).
Building on this foundation, interviews were conducted at multiple
points, with 157 being conducted overall across 140 different interviewees. Thirty-four of these interviews were conducted with functionaries
within the Ministry of Education (MINED), while 50 were performed
with representatives of a range of international organizations, and an additional 14 were done with other national-level actors in El Salvador. The
remaining interviews were conducted with actors from local organizations
and communities. Together, by interviewing key individuals at each level
from the local to the global, the author was able to achieve both a broad
and deep understanding of the dynamics of education reform, policy
implementation, knowledge production, and knowledge mobilization.
This is particularly so because all of the key staff from the Ministry of
Education and from other influential institutions were interviewed.
Examples include each of the ministers of education from 1990 to 2009,
officials from relevant departments within the MINED during 1990–2014,
including the EDUCO office; team leaders, technical staff, and education
economists from the World Bank; education specialists from not only the
United States Agency for International Development (USAID) but also
the United Nations Education, Science, and Culture Organization
(UNESCO) and the German Technical Cooperation Agency, as well as
SITUATING A CRITICAL REVIEW OF IMPACT EVALUATIONS…
79
education specialists from Salvadoran think tanks and from both international and national non-governmental organizations involved in educational reform in El Salvador.
The understanding achieved through interviews was complemented by
a review of documents and archives. Archives were most extensively
reviewed from the MINED, the World Bank, and USAID—the three
institutions at the heart of education reform dynamics during the period
in question. MINED archives were located and reviewed during phase six
of data collection (see Table 3.3), during which time the author was
located in El Salvador for ten months. During this same period USAID
studies and project documents for the period 1983–2008 were downloaded and reviewed from the USAID’s Development Experience
Clearinghouse (USAID, n.d.). World Bank loan documents and education
sector studies were likewise downloaded and reviewed for the period
1969–2011 from the database that is publically available through the
Internet (World Bank, 2016). This source was further complemented by
accessing the World Bank’s internal archives in Washington, DC, on multiple occasions. Because these archives include internal memos as well as
correspondence between World Bank staff and their counterparts in El
Salvador, they afforded insights into educational reform dynamics that
were not contained in the publically available loan documents.
Organizational documents and archives were also systematically collected
from other key sources and involved organizations, such as the Inter-­
American Development Bank, the University of Central America, the
Salvadoran Foundation for Economic and Social Development (Fundación
Salvadoreña para el Desarrollo Económico y Social, FUSADES), the
Business Foundation for Educational Development in El Salvador
(Fundación Empresarial para el Desarrollo Educativo, FEPADE), the
Academy for Educational Development (now known as FHI360), and the
two national newspapers (La Prensa Gráfica and El Diario de Hoy).
At its core, the critical review presented here focuses on six impact
evaluations. These impact evaluations were identified by systematically
searching the Web of Knowledge database and Google using the search
terms “EDUCO” and “El Salvador.” Through interviews and through
the archival searching, it was confirmed that the author had identified all
impact evaluations that had been conducted.
As Table 3.4 shows, a total of 34 distinct examples of evaluative publications were uncovered. The evaluative literature on EDUCO included in
Table 3.4 is restricted to publications that address EDUCO’s
80
D. B. EDWARDS JR.
i­mplementation and effectiveness; it does not include general literature on
EDUCO’s history or the politics of its development, for example. For
clarity, with regard to Table 3.4, institutional studies are defined as those
that are produced by institutions (e.g., the MINED, World Bank) but
which are not intended for public diffusion; academic literature is defined
as journal articles, book chapters, and books; working papers are defined
as studies that have been published by a research center, university, or
other institution and which are labeled as a “working paper” or an “occasional paper”; and, finally, the literature under “organizational publications” includes studies that are publically disseminated through a periodic
institutional series, such as, in this case, the World Bank’s “En Breve” (In
Brief) publication. Table 3.4 also includes the self-explanatory categories
of university theses and conference papers. Appendix 1 includes a complete list of evaluative studies on EDUCO.
Within the overall sample of evaluative studies, six impact evaluations
were identified, where impact evaluations are defined as those that attempt
to determine the effect of an intervention (e.g., the EDUCO program) on
specific outcomes of interest (e.g., student achievement, student retention, teacher attendance, etc.). It is of note that all such evaluations were
produced by, financed by, or carried out by consultants of the World Bank,
an issue that will be discussed further later. These impact evaluations were
first located and obtained during phase three of data collection and were
then initially critically reviewed shortly thereafter, during phase four.
Unlike other evaluative studies on EDUCO, these studies were identified
Table 3.4
Characterization of overall sample of evaluative EDUCO literature
Type of publication
Institutional study
Academic literature
Impact evaluationa
University thesis
Working paper
Organizational publication
Conference paper
Total distinct literature examples in sample
Number of publications identified
13
8
6
4
4
3
2
34
a
Impact evaluations are listed separately for instructive purposes. Note, however, that they are not counted
twice. For this reason, the rows sum to more than the total of studies found. The six impact evaluations
take the form of academic literature (n = 2), working papers (n = 2), and institutional studies (n = 2)
SITUATING A CRITICAL REVIEW OF IMPACT EVALUATIONS…
81
and obtained without difficulty because they appear in search results via
Google. Yet it is important to note that additional searching was systematically conducted during subsequent data collection rounds, since the
author could not be sure that he had obtained all of the impact evaluations
until each of the archives had been exhausted during later data collection
phases. Additionally, as with the search for background literature more
generally, both English-language and Spanish-language literatures were
explored in the search for impact evaluations. Importantly, the interviews
conducted and archives consulted not only confirmed that the author had
found all the impact evaluations produced, they also shed light on which
impact evaluations were more influential, at what points in time, and in
what ways. These complementary data collection sources were thus invaluable to the present study because they placed the identified evaluations in
context and spoke to their political impact both within and outside El
Salvador as well as in multiple organizational settings, aspects that are necessary to understand in order to successfully carry out a critical review of
evidence within the political economy of global education reform.
Multiple rounds of critical review were conducted on the six studies
included here. The first round (phase three in Table 3.3) focused on the
stated findings and the limitations of the methods employed, while rounds
two and three of critical review (i.e., phases seven and 12 in Table 3.3)
revisited and reinterpreted those findings in light of methodological limitations and in light of the subsequent understanding that had been
achieved on education reform generally and the trajectory of the EDUCO
program specifically (Edwards, 2018). Conversely, in accordance with the
definition of critical review established earlier, these latter two issues (i.e.,
processes of education reform and the trajectory of EDUCO internationally) were also themselves reassessed in light of the critical understanding
that had been achieved of the EDUCO studies. In this way, a tripartite
analysis was conducted wherein the studies in question were viewed, first,
within their own limitations; second, within the surrounding organizational and contextual circumstances; and, third, in terms of how they
influenced the institutional, political-economic, and ideological contexts
into which they were inserted (as depicted in Fig. 3.1). All six impact
evaluations are listed in Table 3.5 in chronological order.
Finally, data collection focused on the reach of EDUCO and the ways
that the impact evaluations have been used by academics and other organizations. Here, during phases six and 13, the methods of Goldie et al.
(2014) were followed because, as in their case, the final stage in the p
­ resent
82
D. B. EDWARDS JR.
Table 3.5
Chronological listing of impact evaluations of EDUCO
Number Complete citation
1
2
3
4
5
6
World Bank. (1994). El Salvador: Community education strategy: Decentralized
school management. Washington, DC: The World Bank
Umanzor, S., Soriano, I., Vega, M., Jimenez, E., Rawlings, L., & Steele, D.
(1997). El Salvador’s EDUCO program: A first report on parents’ participation
in school-based management. Working Paper Series on Impact Evaluation of
Education Reforms, No. 4. World Bank
Jimenez, E., & Sawada, Y. (1999). Do community-managed schools work? An
evaluation of El Salvador’s EDUCO program. The World Bank Economic
Review, 13(3), 415–441
Sawada, Y. (2000). Community participation, teacher effort, and educational
outcome: The case of El Salvador’s EDUCO program. Michigan, USA: The
William Davidson Institute, University of Michigan Business School
Jimenez, E., & Sawada, Y. (2003). Does community management help keep kids
in schools? Evidence using panel data from El Salvador’s EDUCO program.
Discussion Paper, Center for International Research on the Japanese Economy.
Tokyo, Japan: University of Tokyo
Sawada, Y., & Ragatz, A. (2005). Decentralization of education, teacher
behavior, and outcomes: The case of El Salvador’s EDUCO program. In
E. Vegas (Ed.), Incentive to improve teaching: Lessons from Latin America
(pp. 255–306). Washington, DC: The International Bank for Reconstruction
and Development/The World Bank
study requires that we understand how it may be that a policy itself in
addition to “a small but defined set of studies is repeatedly cited by policy
advocates, as well as by a small and relatively self-contained set of researchers, as proof that [certain policies] ‘work’, while neglecting the methodological concerns about the studies and mixed effects of these programs
on different populations” (p. 285). Bibliometric searches were thus performed using the Web of Knowledge database, Google Scholar, and
Google, with search terms including the EDUCO program name in
English and Spanish, the names of the authors of the impact studies,
and the titles of those studies. It was necessary to include Google, in
addition Google Scholar, since EDUCO and its impact evaluations have
been featured in nonacademic contexts and publications. Where relevant, ResearchGate was also consulted because this site traces the subsequent citations of the papers uploaded there. The last source to
mention for bibliographic searching is the physical archives discussed
above. These archives, for example, from the World Bank and from
SITUATING A CRITICAL REVIEW OF IMPACT EVALUATIONS…
83
major Salvadoran newspapers, revealed events where EDUCO was
highlighted that would not appear in digital searches. Respectively, on
the basis of these methods, detail the global promotion of EDUCO as
well as the invocation of EDUCO impact studies in academic literature
and institutional publications.
Data Analysis
The phases of data collection shown in Table 3.3 were alternated with
periods of data analysis wherein a range of strategies were employed to
organize, array, and interpret the information gathered. These strategies,
which are summarized in Table 3.6, and which were initially adapted by
Edwards (2012a) based on techniques elaborated in Miles and Huberman
(1994) and Yin (2013), have been further adapted here to meet the needs
of the critical review. To be clear, while the first ten strategies in Table 3.6
are based on Edwards (2012a), who was interested in understanding the
political economy of policy formation processes, the last two have been
incorporated here to reflect the nature of the present critical review. Given
the scope of this study and the quantity of data collected, it was necessary
to engage each of these strategies in an iterative fashion, as new information was gathered. Crucially, it is through these methods that the author
was able to understand the various ideological, structural, political-­
economic, processual, and individual aspects of the education reform context in which the impact evaluations were produced and back into which
they circulated over time, as depicted in Fig. 3.1.
Ultimately, by alternating review of the studies themselves with investigation of the larger context, the author was able to arrive at an in-depth
understanding of the role of the World Bank’s impact evaluations. As indicated previously, this understanding is inherently critical in nature, given
that the approach in this study asks not only what impact evaluations
found but also what new can be found by analyzing those evaluations
within the various layers of political interests, vested organizations, policymaking processes, and structural constraints. Such findings can only
emerge through iterative methods, like those discussed here, that allow
one to go beyond official narratives around how the research was produced and used, for what purposes, and with what limitations. With that
in mind, the next chapter addresses the multi-level context out of which
EDUCO emerged and into which the impact evaluations were inserted. It
should be noted, however, that the contextual description provided below
84
D. B. EDWARDS JR.
Table 3.6
Summary of analytic methods
No. Strategy
1
2
3
4
5
6
7
8
9
10
11
Strategy description
Repeatedly
analyze and
code data
Assess data for context, isolate facts about education reform and
EDUCO (dates, events, actors involved, etc.), and code data (e.g.,
interviewee statements re dynamics, influence, process
mechanisms, etc.)
Memoing
Consider data gathered and record emerging findings, reflections,
or insights into meaning of interrelationships among data
Perspectival
Aggregate actors’ characterizations of events, processes, and
flow chart
others’ actions by level (i.e., local, national, global) and reform
period, and then analyze cross sections for themes
Events listing
Create a chronological listing of all events during policy formation
process, as understood on the basis of interrogation of data
Critical events
Distill events timeline to only include those events, actions, and
timeline
actors around which data converge; include commentary on
significance of each event, based on repeated analysis and evolving
understanding of process dynamics
Policy content
Trace emergence of policy content across process-related
matrix
documents produced by various actors
Actor-influence Distill and summarize, by level and across periods of reform,
matrix
actions and forms of influence of actors, as well as shift structural
characteristics
Context charts Visually represent the linkages among the essential events, actors,
and context for each period of reform. The focus is on mapping
context rather than attributing causation in the process of policy
formation
Logic models
Elaborate the interplay of context, events, and actors, and explain
the ways in which each of them acted and interacted to influence,
constrain, or enable reform generally while facilitating, inhibiting,
or otherwise affecting the development of the EDUCO program
specifically
Causal diagrams Departs from the logic model and ‘[d]isplay[s] … the most
important independent and dependent variables in a field of study
(shown in boxes) and of the relationships among them (shown by
arrows). The plot of these relationships is directional, rather than
solely correlational’ (Miles & Huberman, 1994, p. 153)
Critical review
Analyze evaluations and their findings in terms of methodological
of impact
limitations and in relation to the significance of those studies for
evaluations
policy, for practice, and for the vested interests that may be
advanced (or not) on a global scale from their production and the
findings they contain
(continued)
SITUATING A CRITICAL REVIEW OF IMPACT EVALUATIONS…
Table 3.6
85
(continued)
No. Strategy
Strategy description
12
Search and analyze digital sources and databases and physical
archives to trace promotion of policy and the invocation of the
corresponding evidence base
Bibliometric
analysis
Source: Adapted from Edwards (2012a)
is limited to only that information which is necessary in order to situate
and subsequently make sense of the critical review of impact evaluations
that follows. For a longer account, see Edwards (2018).
Notes
1. Certainly, many critical scholars engage with various of the aspects delineated here; however, the full range of goals (and, later, tasks) as combined
and spelled out in this chapter is not usually defined or presented as a unified
approach.
References
Ball, S. (2009). Privatising education, privatising education policy, privatising educational research: Network governance and the “competition state”. Journal of
Education Policy, 24(1), 83–99.
Ball, S. (2010). New voices, new knowledge and the new politics of education
research: The gathering of a perfect storm? European Educational Research
Journal, 9(2), 124–137.
Bryk, A. (1981). Disciplined inquiry or policy argument? Harvard Educational
Review, 51(4), 497–509.
Edwards, D. B., Jr. (2012a). Researching international processes of education
policy formation: Conceptual and methodological consideration. Research in
Comparative International Education, 7(2), 127–145.
Edwards, D. B., Jr. (2012b). The approach of the World Bank to participation in
development and education governance: Trajectories, frameworks, results. In
C. Collins & A. Wiseman (Eds.), Education strategy in the developing world:
Revising the World Bank’s education policy (pp. 249–273). Bingley, UK:
Emerald.
Edwards, D. B., Jr. (2013a). International processes of education policy formation: An analytic framework and the case of Plan 2021 in El Salvador.
Comparative Education Review, 57(1), 22–53.
86
D. B. EDWARDS JR.
Edwards, D. B., Jr. (2013b). The development of global education policy: A case
study of the origins and evolution of El Salvador’s EDUCO program. Unpublished
Dissertation, University of Maryland, College Park, MD.
Edwards, D. B., Jr. (2018). The trajectory of global education policy: Community-­
based management in El Salvador and the global reform agenda. New York:
Palgrave Macmillan.
Edwards, D. B., Jr., & DeMatthews, D. (2014). Historical trends in educational
decentralization in the United States and developing countries: A periodization
and comparison in the post-WWII context, Education Policy Analysis Archives,
22(40), 1–36. https://doi.org/10.14507/epaa.v22n40.2014
Edwards, D. B., Jr., & Klees, S. (2012). Participation in international development
and education governance. In A. Verger, M. Novelli, & H. Kosar-Altinyelken
(Eds.), Global education policy and international development: New agendas,
issues and programmes (pp. 55–77). New York: Continuum.
Edwards, D. B., Jr., Victoria, J., & Martin, P. (2015). The geometry of policy
implementation: Lessons from the political economy of three education reforms
in El Salvador during 1990–2005. International Journal of Educational
Development, 44, 28–41.
Goldie, D., Linick, M., Jabbar, H., & Lubienski, C. (2014). Using bibliometric
and social media analyses to explore the “echo chamber” hypothesis.
Educational Policy, 28(2), 281–305.
Jessop, B. (2001). Institutional re(turns) and the strategic-relational approach.
Environment and Planning A, 33, 1213–1235.
Klees, S., & Edwards, D. B., Jr. (2014). Knowledge production and technologies
of governance. In T. Fenwick, E. Mangez, & J. Ozga (Eds.), World yearbook of
education 2014: Governing knowledge: Comparison, knowledge-based technologies
and expertise in the regulation of education (pp. 31–43). New York: Routledge.
Klees, S., & Wells, S. (1983). Economic evaluation of education: A critical analysis
in the context of applications to educational reform in El Salvador. Educational
Evaluation and Policy Analysis, 5(3), 327–345.
Lubienski, C., Scott, J., & DeBray, E. (2014). The politics of research, production, promotion, and utilization in education policy. Educational Policy, 28(2),
131–144.
Lubienski, C., Weitzel, P., & Lubienski, S. T. (2009). Is there a “consensus” on
school choice and achievement? Advocacy research and the emerging political
economy of knowledge production. Educational Policy, 23(1), 161–193.
Miles, M., & Huberman, M. (1994). Qualitative Data Analysis: An expanded
sourcebook (2nd ed.). Thousand Oaks, CA: Sage.
Nambissan, G., & Ball, S. (2010). Advocacy networks, choice and private schooling for the poor in India. Global Networks, 10(3), 1–20.
SITUATING A CRITICAL REVIEW OF IMPACT EVALUATIONS…
87
Robertson, S., & Dale, R. (2009). The World Bank, the IMF, and the possibilities
of critical education. In M. Apple, W. Au, & L. Gandin (Eds.), The Routledge
international handbook of critical education (pp. 23–35). New York: Routledge.
Samoff, J. (1993). The reconstruction of schooling in Africa. Comparative
Education Review, 37(2), 181–222.
Samoff, J. (2012). “Research shows that …”: Creating the knowledge environment for learning for all. In S. Klees, J. Samoff, & N. Stromquist (Eds.), The
World Bank and education: Critiques and alternatives (pp. 143–157).
Rotterdam, Netherlands: Sense.
Scott, J., & Jabbar, H. (2014). The hub and the spokes: Foundations, intermediary organizations, incentivist reforms, and the politics of research evidence.
Educational Policy, 28(2), 233–257.
USAID. (n.d.). Development experience clearinghouse. Retrieved from https://
dec.usaid.gov/dec/home/Default.aspx
Verger, A. (2012). Framing and selling global education policy: The promotion of
PPPs in education in low-income countries. Journal of Education Policy, 27(1),
109–130.
Water, G. (1998). Critical evaluation for education reform. Education Policy
Analysis Archives, 6(20), 1–38.
World Bank. (2016). Projects & operations. Retrieved from http://web.worldbank.org/WBSITE/EXTERNAL/PROJECTS/0,,menuPK:41389~pagePK:
95863~piPK:95983~targetDetMenuPK:228424~targetProjDetPK:73230~tar
getProjResPK:95917~targetResMenuPK:232168~theSitePK:40941,00.
html#ProjectsDB
Yin, R. (2013). Case study research: Design and methods (5th ed.). Thousand Oaks,
CA: Sage.
CHAPTER 4
The Case of EDUCO: Political-Economic
Constraints and Organizational Dynamics
Abstract This chapter transitions to the empirical of focus in this book,
that is, the case of the Education with Community Participation (EDUCO)
program. The first section addresses the structural context that characterized El Salvador prior to the emergence of this program, in order to
understand the larger enabling and constraining factors that affected this
policy’s origins and trajectory. The second section then turns to characterize the organizational and strategic relational dynamics that would not
only influence EDUCO’s evolution but which are also necessary to understand in order to grasp later in this book how and why EDUCO itself has
been so influential. In ways that are discussed, the United States Agency
for International Development (USAID), the World Bank, and the United
Nations Education, Science, and Culture Organization (UNESCO) were
integral to the case of EDUCO.
Keywords Education with Community Participation • EDUCO • El
Salvador • International organization • Political economy • Education
policy
This chapter transitions to the empirical of focus in this book, that is, the
case of the Education with Community Participation (EDUCO) program.
The first section addresses the structural context that characterized El
© The Author(s) 2018
D. B. Edwards Jr., Global Education Policy, Impact Evaluations, and
Alternatives, https://doi.org/10.1007/978-3-319-75142-9_4
89
90
D. B. EDWARDS JR.
Salvador prior to the emergence of this program, in order to understand
the larger enabling and constraining factors that affected this policy’s origins and trajectory. The second section then turns to characterize the
organizational and strategic relational dynamics that would not only influence EDUCO’s evolution but which are also necessary to understand in
order to grasp later in this book how and why EDUCO itself has been so
influential. In ways that will be discussed, the United States Agency for
International Development (USAID), the World Bank, and the United
Nations Education, Science, and Culture Organization (UNESCO) were
integral to the case of EDUCO.
Structural Transformation During a Civil War
To understand the emergence of the EDUCO program, it is necessary to
first grasp the political-economic context of the 1980s in El Salvador.
During this decade, El Salvador found itself mired in an on-going civil war
which had, on one side, the government, the landed oligarchy, and the
armed forces and, on the other, a coalition of five rebel groups fighting for
socialist reform of the land, economy, and social services (LeoGrande,
1998). Collectively, these rebel groups were known as the Farabundo
Martí National Liberation Front (Frente Farabundo Martí para Liberación
Nacional, FMLN).1 The impetus for the uprising of these five groups was
the continual repression and persecution by the government and its death
squads of the population, particularly in rural areas, which was increasingly
unaccepting of the country’s extreme inequality and its concentration of
land ownership in the hands of a few (Montgomery, 1995). During
1980–1984 alone an estimated 70,000 civilians were murdered at the
hands of the death squads (Meade, 2010; Orr, 2001).
Layered upon this state of affairs were three additional—and intertwined—aspects of the political-economic context. The first was the
involvement of the United States. With the Cold War as a backdrop, the
socialist uprising in El Salvador (just as in neighboring Nicaragua) was an
issue of particular concern. In response to FMLN offensives, the United
States provided significant infusions of military and social aid. To put into
perspective the magnitude of the aid provided, consider that, while the
United States provided approximately $150 million in aid to El Salvador
from 1963 to 1979, it jumped to approximately $6 billion between 1980
and 1992 (Robinson, 2003, p. 89). In 1985, military aid alone amounted
to $533 million (Booth, Wade, & Walker, 2006, p. 105).
THE CASE OF EDUCO: POLITICAL-ECONOMIC CONSTRAINTS…
91
The second aspect to note was the international promotion of neoliberal ideology and reform by the Reagan administration internationally and
by the United States Agency for International Development (USAID)
within El Salvador. On this latter point, USAID perceived that the context
of the civil war provided an opportunity to reconfigure the political landscape of the country in order to promote the adoption of its preferred
economic and social policies (Robinson, 2003). In particular, USAID was
interested to counter the policy orientation of President José Napoleón
Duarte (1984–1989), a populist who did not favor liberalizing the economy, but whom the US Congress felt forced to support to ensure stability
of the country during the civil war (LeoGrande, 1998).
In line with this strategy, USAID supported the creation of a think
tank, FUSADES. This organization served as a key space through which
to (a) host famous international economists, such as the “Chicago Boys”
(Klein, 2007), (b) lend credibility to neoliberal reform principles, and (c)
raise the profile of Salvadoran economists and businessmen who shared
USAID’s policy preferences (discussed further below). That USAID provided $150 million in contracts to the Salvadoran Foundation for
Economic and Social Development (FUSADES) during 1983–1993
shows the importance of the latter to the former.
In 1986, USAID also facilitated the creation of the Business Fund for
Educational Development (FEPADE). As a technocratic organization,
FEPADE diagnosed the needs of the education sector and worked on
USAID’s education projects. It also provided training and capacity building to the Ministry of Education (MINED). Given its origins and orientation, FEPADE was a key source of technical capacity for USAID and the
Salvadoran government, particularly during the 1990s, once the rightwing
party assumed power. It is of note that the minister of education during
1990–1998—Cecilia Gallardo de Cano—was an education specialist at
FEPADE before becoming the minister of education. Crucially, she also
came from the core of the rightwing party known as the National
Republican Alliance (Alianza Republicana Nacionalista, ARENA) and, as
such, was closely networked with members from its highest echelon,
including President Cristiani. Moreover, she was keen political player who
aspired to become president one day.
Before proceeding with the developments of the 1990s, it is first important to highlight the third aspect of the political-economic context. This is
that the eventual success (in 1989) of ARENA was due in part to the
failure of the Christian Democrat party under the leadership of Duarte.
92
D. B. EDWARDS JR.
Initially, after his election in 1984, Duarte was handicapped by the rightist
majority in the National Assembly and so could not move forward with
the agrarian reform that had been initiated by the civilian junta that was
briefly in power prior to 1984.2 After 1985, even with control of the
National Assembly, Duarte and his party were restricted by the military,
USAID, and the conservative and reform-minded portion of the
Salvadoran business elite. In the face of resistance by these actors, Duarte
could neither bring about peace nor institute social welfare programs, as
he had hoped to do. Duarte also failed to open the economy the way that
many Salvadoran businessmen and the Reagan administration would have
liked. Instead, he implemented economic reforms that ultimately accelerated inflation and imposed higher taxes on imports (LeoGrande, 1998).
Subsequently, and with support from USAID, the rightwing candidate
for president, Alfredo Cristiani, won the presidency in 1989. Not surprisingly, Cristiani and “at least 17 business leaders and persons linked with …
[FUSADES] became part of the new government,” including the ministers of planning and finance and the head of the Central Reserve Bank
(Segovia, 1996, p. 55; Eriksson, Kreimer, & Arnold, 2000). Thereafter,
they pursued the economic reform package that was being promoted by
FUSADES and USAID. Though the economic and public sector reforms
enacted were often grouped under the innocuous label of “modernization,” it was clear that they represented the implementation of a neoliberal
perspective on policy. To that end, as documents from the World Bank
(1991) indicate, in the month following the assumption of power, Cristiani
“initiated a major stabilization and structural adjustment program” in
order to “create a more liberalized, private-sector led economy” (p. 5).
The first structural adjustment loan from the World Bank would be
approved in 1991, followed by a second one in 1993.3 After his election,
Cristiani also began to call for peace talks with the FMLN. At least in part,
this was due to the fact Cristiani and other business elite realized that their
investments in banking and industry would continue to suffer in the context of on-going conflict. Peace Accords were finally signed in January
1992 with the help of the United Nations.
Organizational Dynamics and EDUCO’s Emergence
A first priority of the Cristiani administration was to address the state and
management of the education system. In this area, the immediate challenge for the new minister of education, Cecilia Gallardo de Cano, who
THE CASE OF EDUCO: POLITICAL-ECONOMIC CONSTRAINTS…
93
assumed her post in 1990, was to identify an acceptable policy to expand
educational access. Countrywide, by 1990, at least 37% of children ages
7–14 were out of school, and in the conflict-affected areas government-­
provided education services were nonexistent (MINED, 1990).
Importantly, however, in searching for reform options, Gallardo had to
take into account three constraints. First, the reform strategy had to fit
with the larger focus on “modernization” that was a core characteristic of
the Cristiani government, as already mentioned. For public services like
education, this meant reducing bureaucracy and introducing new service
provision arrangements that would ensure efficiency, effectiveness, and
accountability. Second, Gallardo had to circumvent the teachers’ unions,
who were not only anti-privatization but also allies of the FMLN. Third,
the government saw it as necessary to incorporate the communities and
schools linked with the FMLN into the official education system.4 This
third condition was particularly essential from the government’s perspective because the civil war had not yet concluded at the time that Gallardo
was searching for reform options, and thus the government was still looking for ways to subjugate the regions and schools controlled by the FMLN.
While the constraints specified in the previous paragraph and the preceding section flowed from the structural characteristics of the political-­
economic context of the 1980s, going forward, during 1990–1994, these
constraints would, first, combine with the strategic action of key actors
and, then, through that interaction, would lead to the entrenchment of
policy priorities. Importantly for our purposes, the way in which vested
interests crystallized around EDUCO in this period of transition ensured
that EDUCO became a political priority for multiple actors—and political
priority that would be furthered by the elaboration of impact studies during 1994–2005. The evolution of the period 1990–1994 can be distilled
to three “moments.”
In the first moment, in midyear 1990, a consultant from UNESCO was
tasked with conducting an education sector assessment in order to understand the state of education in each of the country’s 14 departments. This
was a necessary task given that representatives of the government had not
visited many rural parts of the country for ten years due to the on-going
conflict. By conducing this assessment, the consultant realized a reform
model based in community management of education could be feasible
because she observed firsthand that many communities (but especially in
conflict-affected areas and in areas controlled by the FMLN) were already
engaging in self-management of the education of their kids due to the lack
94
D. B. EDWARDS JR.
of government-provided services (MINED, 1990; Gillies, 2010). In these
areas, a community member offered to teach basic education in exchange
for minimal, voluntary contributions (e.g., nominal payment or food
donations). Moreover, in FMLN areas, the schooling provided was
grounded in the tradition of liberation theology and in the teachings of
Paulo Freire in that it taught students to identify and to mobilize against
the political-economic structures that ensured their oppressed position.
For short, this approach is known as “popular education” (Hammond,
1998). But the larger point is that the Salvadoran government was threatened by the educational approach of the FMLN, as well as by the enduring
low-level warfare.
The second moment spanned late 1990 to mid-1991. During this time,
the suggestion from the UNESCO consultant was at first received with
skepticism. The minister of education did not initially see this idea as viable in political terms because it was based on the management practices of
FMLN-affiliated communities. For its part, the World Bank doubted that
it would be possible to implement a community management model when
the majority of parents had low levels of literacy. However, after successfully testing this model in a pilot program—and thanks to the persistence
of the UNESCO consultant—both the minister of education and the
World Bank began to realize the opportunity that such a model represented. That is, it occurred to the minister of education that this model
met her constraints because it was innovative (i.e., it helped to “modernize” the MINED by dealing a blow to the central bureaucracy, which had
traditionally been responsible to teacher management) and because it
would allow her to incorporate and, thereby, to subjugate the FMLN
communities. In addition, both the minister of education and the World
Bank both saw a community model as a way to weaken the teachers’
unions because it would be possible to require that the teachers in this new
program work on one-year contracts that would be renewable at the discretion of the community. And since these contracts would only affect
those teachers working in this new program, the teachers’ unions did not
put up sufficient resistance to block this reform model, as they had done
with previously when the minister of education explored the possibility of
adopting the voucher model from Chile (Edwards, 2015).
Separately, it dawned on the World Bank representatives working in El
Salvador that this community-based model—if it were successful—would
be a bankable policy in that they could sell it to other countries, especially
since the international reform trends of the day heavily favored efforts
THE CASE OF EDUCO: POLITICAL-ECONOMIC CONSTRAINTS…
95
toward the decentralization of central government functions and the “participation” of stakeholders in service provision (Edwards, 2012b; Edwards
& DeMatthews, 2014). But before moving ahead, two points are necessary here to illuminate the pressures and trends of the time. First, during
the 1980s, and continuing into the 1990s, the World Bank and other
international organizations were preoccupied with the idea of decentralization (Edwards, 2012a; Montero & Samuels, 2004), which was a popular idea in international development during the 1980s (Conyers, 1983).
In the first of these decades, the World Bank had conceptual models that
predicted increased efficiency and enhanced effectiveness from the decentralization of education systems (Winkler, 1989), though they did not
have empirical evidence. Moreover, before the EDUCO program, it had
not occurred to the World Bank that it would be possible to decentralize
to the community level; they were instead focused on municipal or departmental levels. Second, internationally, in the late 1980s and early 1990s,
much attention was drawn by specialists in international development to
the idea that “participation” was a central component of “good” development strategies (Cornwall, 2006; Leal, 2007), especially when it came to
education policy (Edwards, 2012a). This was reflected in March 1990 by
the high-profile declaration of the World Conference on Education for All
(WCEFA), which asserted that “partnerships at the community level …
should be encouraged” because “they can help harmonize activities, utilize resources more effectively, and mobilize additional financial and
human resources” (WCEFA Secretariat, 1990, p. 58, as cited in Bray,
2003, p. 32). During this time, international organizations also highlighted the concept of participation. In particular, USAID raised the issue
as a means to promote democracy and the effectiveness of public institutions (Montero & Samuels, 2004). The World Bank, in addition to
emphasizing the issue of effectiveness, honed in as well on the potential of
decentralization to result in more efficient and accountable service provision arrangements (Edwards, 2012a).
For these reasons, by mid-1991, the minister of education and the
World Bank decided to place the full weight of their support behind this
model. This was a major turning point because it was impossible for any
education reform to be successful without the backing of the World
Bank—the institution that would finance the overhaul of the education
sector’s governance system. Thus, in May 1991, these actors agreed on a
first loan for the education sector that included $10.3 million to solidify
and extend the community management model, which, from then on, was
96
D. B. EDWARDS JR.
known as “Education with Community Participation,” or EDUCO, as
previously noted. Over time, this program received support from the
Inter-American Development Bank as well as from the World Bank.
Indeed, these two institutions issued a total of $69.3 million in loans to
ingrain, to scale up, and to improve the EDUCO program during
1991–2006. It is also notable that the decision to pursue this reform
occurred well before the Peace Accords of January 1992, with a key implication being that this education governance reform strategy was selected
before the war had ended and, concomitantly, before the official (i.e., public and democratic) post-conflict process of research and consultation had
a chance to begin (Edwards, Victoria, & Martin, 2015).
The third moment extended from mid-1991 to 1994. In this period,
through extensive and intensive World Bank technical assistance, the features of this program were refined. Not surprisingly, both the theories that
guided the program’s development and the practical arrangements that
gave life to the program were guided by a neoliberal perspective on education management. To be specific, the key features of the program were as
follows: a group of five parents from each community was elected to form
a Community Education Association (Asociación Comunal para la
Educación, ACE), an entity which was given legal status and which was
charged with contracting and paying (with funds transferred from the
MINED to the ACE via bank account) the teacher(s) that would work in
their schools.5 These ACEs were also thought to act as accountability
levers that reduced teacher absenteeism and resulted in increased teacher
effort in the classroom, especially since teachers were hired on one-year
contracts. With the funds provided by the MINED, each ACE additionally had the responsibility of purchasing necessary didactic materials. These
arrangements were assumed to reduce waste (thereby increasing the efficiency of provision) and to enhance student achievement (thereby increasing effectiveness). Given this arrangement, parents in disadvantaged rural
communities were being asked to assume responsibilities traditionally
borne by the State and without compensation. In addition, community
members were often expected to provide the time, labor, and materials
either to construct or maintain their local school, particularly at the beginning of the program—and this was in addition to the informal school fees
they already incurred to cover basic costs associated with schooling. The
MINED, on the other hand, was responsible for the following: facilitating
the creation of the ACEs, training the members of the ACEs in administrative and accounting procedures, setting the minimum criteria for
THE CASE OF EDUCO: POLITICAL-ECONOMIC CONSTRAINTS…
97
teacher selection by the ACEs, designing and providing curricula, and
overall coordination, supervision, and monitoring of the program as it was
scaled up (Reimers, 1997b; World Bank, 1994).6
Thus, from late 1991 onward, the Minister was committed to scaling
up and widely implementing the EDUCO program, which soon became
the only program through which the Minister would allow educational
access to be augmented. The extent of this commitment, and the blinding
effect that it had on key personnel within the MINED’s EDUCO office,
is important to note. As Fernando Reimers (1997a) would later recall,
even when he, a researcher from Harvard University, highlighted negative
results from a study he led, those results were ignored by MINED
leadership.
By 1993, the program was operating in all 14 departments of the country, and by 1995 the program had been institutionalized in the sense that
the EDUCO office had been incorporated into the larger administrative
structure of the MINED (Edwards, 2018). In statistical terms, while the
program began as a pilot with six schools in January 1991, it encompassed
114 schools by the end of that same year. By 2004, the program had 7831
teachers and 378,208 students (Cuéllar-Marchelli, 2003). In 2010,
approximately 55% of rural public schools—which make up two-thirds of
all schools in El Salvador—operated under the EDUCO program (Gillies,
Crouch, & Flórez, 2010). The continued popularity and expansion of the
program was due, among other reasons, to its perceived success and to the
fact that the ARENA party—with its preference for neoliberal reform—
retained the presidency until 2009.7
Notes
1. These five political-military organizations were known as the Popular Forces
of Liberation, National Resistance, Revolutionary Army of the People,
Communist Party of El Salvador, and Revolutionary Party of Central
American Workers (Montgomery, 1995).
2. See Montgomery (1995, pp. 137–138, 187) for more on the agrarian
reform. In short, this reform, which had first begun in 1980, sought to
nationalize large farms (over 100 ha) and to transfer ownership of land to
those who rented it. In early 1985, the budget for this program was eliminated by the National Assembly.
3. These were followed with a “public sector modernization” loan from the
World Bank in 1996. For more on the significance of these loans in the
context of total aid during the 1990s, see Rosa and Foley (2000).
98
D. B. EDWARDS JR.
4. There were approximately 1000 popular education teachers from the FMLN
(Alvear Galindo, 2002, pp. 189, 200).
5. While the EDUCO program was initially only intended as a strategy to provide education at the preschool level and in grades 1–3, it was subsequently
expanded in 1994 to cover through grade 6 and then again in 1997 to cover
through grade 9 (Meza, Guzmán, & Varela, 2004). After 2005, even some
high schools became EDUCO schools (Gilles, Crouch, & Flórez, 2010).
6. For additional details on the technical aspects of the EDUCO program, see
Gilles et al. (2010).
7. See Edwards (2018, pp. 54–55) or Edwards (2018) for an update on the
EDUCO program in the post-2009 time period.
References
Alvear Galindo, V. (2002). La educación en Morazán, El Salvador, durante la guerra
civil de 1981 a 1992: ¿parte de una estrategia de supervivencia? [Education in
Morazán, El Salvador, during the civil war from 1981 to 1992: Part of a strategy
for survival?] Unpublished doctoral dissertation, Berlin Free University, Berlin.
Booth, J., Wade, C., & Walker, T. (2006). Understanding Central America.
Boulder, CO: Westview Press.
Bray, M. (2003). Community initiatives in education: Goals, dimensions and linkages withi governments. Compare, 33(1), 31–45.
Conyers, D. (1983). Decentralization: The latest fashion in development administration? Public Administration and Development, 3(2), 97–109.
Cornwall, A. (2006). Historical perspectives on participation in development.
Commonwealth & Comparative Politics, 44(1), 49–65.
Cuéllar-Marchelli, H. (2003). Decentralization and privatization of education in
El Salvador: Assessing the experience. International Journal of Educational
Development, 23(2), 45–166.
Edwards, D. B., Jr. (2012a). Researching international processes of education
policy formation: conceptual and methodological consideration. Research in
Comparative International Education, 7(2), 127–145.
Edwards, D. B., Jr. (2012b). The approach of the World Bank to participation in
development and education governance: Trajectories, frameworks, results. In
C. Collins & A. Wiseman (Eds.), Education strategy in the developing world:
Revising the World Bank’s education policy (pp. 249–273). Bingley: Emerald.
Edwards, D. B., Jr. (2015). Rising from the ashes: How the global education policy of community-based management was born from El Salvador’s civil war.
Globalisation, Societies and Education, 13(3), 411–432.
Edwards, D. B., Jr. (2018). The trajectory of global education policy: Community-­
based management in El Salvador and the global reform agenda. New York:
Palgrave Macmillan.
THE CASE OF EDUCO: POLITICAL-ECONOMIC CONSTRAINTS…
99
Edwards, D. B., Jr., & DeMatthews, D. (2014). Historical trends in educational
decentralization in the United States and developing countries: A periodization
and comparison in the post-WWII context. Education Policy Analysis Archives,
22(40), 1–36.
Edwards, D. B., Jr., Victoria, J. A., & Martin, P. (2015). The geometry of policy
implementation: Lessons from the political economy of three education reforms
in El Salvador during 1990–2005. International Journal of Educational
Development, 44, 28–41.
Eriksson, J., Kreimer, A., & Arnold, M. (2000). El Salvador: Post-conflict reconstruction: Country case evaluation. Washington, DC: World Bank.
Gillies, J. (Ed.) (2010). Education system reform and aid effectiveness: The power of
persistence. Washington, DC: USAID/Equip 2. Retrieved from http://www.
equip123.net/docs/E2-Power_of_Persistence.pdf
Gillies, J., Crouch, L., & Flórez, A. (2010). Strategic review of the EDUCO program. USAID. Retrieved from http://www.equip123.net/docs/e2-EDUCO_
Strategic_Review.pdf
Hammond, J. (1998). Fighting to learn: Popular education and guerrilla war in
ElSalvador. New Brunswick, NJ: Rutgers University Press.
Klein, N. (2007). The shock doctrine: The rise of disaster capitalism. New York:
Picadur.
Leal, P. A. (2007). Participation: The Ascendancy of a Buzzword in the Neo-­
liberal Era. Development in Practice, 17(4–5), 539–548.
LeoGrande, W. (1998). Our own backyard: The United States and Central America,
1977–1992. Chapel Hill, NC: University of North Carolina Press.
Meade, T. (2010). A history of modern Latin America: 1800 to the present. Sussex:
Wiley-Blackwell.
Meza, D., Guzmán, J., & de Varela, L. (2004, May 25–27). EDUCO: A
community-­managed education program in rural areas of El Salvador. Paper
presented at Scaling Up Poverty Reduction: A Global Learning Process and
Conference, Shanghai.
MINED. (1990). Servicios educativos a niños pobres Salvadoreños de 0 a 14 años.
[Educational services to poor Salvadoran children from 0 to 14 years of age.]. San
Salvador: Author.
Montero, A., & Samuels, D. (Eds.). (2004). Decentralization and Democracy in
Latin America. Notre Dame, IN: University of Notre Dame Press.
Montgomery, T. (1995). Revolution in El Salvador: From civil strife to civil peace
(2nd ed.). Boulder, CO: Westview.
Orr, R. C. (2001). Building peace in El Salvador: From exception to rule. In E. M.
Cousens, C. Kumar, & K. Wermester (Eds.), Peacebuilding as politics:
Cultivating peace in fragile societies (pp. 153–182). Boulder, CO: Lynne
Rienner Publishers, Inc.
100
D. B. EDWARDS JR.
Reimers, F. (1997a). Changing schools through participatory knowledge management in El Salvador: Can education systems learn? In D. Chapman, L. Mahlck,
& A. Smulders (Eds.), From planning to action: Government initiatives for
improving school-level practice (pp. 147–172). Paris: IIEP, UNESCO.
Reimers, F. (1997b). The role of the community in expanding educational opportunities: The EDUCO schools in El Salvador. In J. Lynch, C. Modgil, &
S. Modgil (Eds.), Education and development: Tradition and innovation. Equity
and excellence in education for development (Vol. 2, pp. 146–162). London:
Cassell.
Robinson, W. (2003). Transnational conflicts: Central America, social change, and
globalization. New York: Verso.
Rosa, H., & Foley, M. (2000). El Salvador. Good intentions: Pledges of aid for
postconflict recovery. In S. Forman & S. Patrick (Eds.), Good intentions: Pledges
of aid for postconflict recovery (pp. 113–158). Boulder, CO: Lynne Rienner.
Segovia, A. (1996). Macroeconomic performance and policies since 1989. In J. K.
Boyce (Ed.), Economic policy for building peace: The lessons of El Salvador
(pp. 51–72). Boulder, CO: Lynne Rienner.
Winkler, D. (1989). Decentralization in education: An economic perspective.
Washington, DC: World Bank.
World Bank. (1991). Report and recommendation of the President of the International
Bank for Reconstruction and Development to the Executive Directors on a proposed
structural adjustment loan in an amount equivalent to US$75.0 million to the
Republic of El Salvador. Washington, DC: Author.
World Bank. (1994). El Salvador: Community education strategy: Decentralized
school management. Washington, DC: Author.
World Conference On Education For All [WCEFA] Secretariat. (1990). WCEFA
framework for action. New York: WCEFA Inter-Agency Commission.
CHAPTER 5
Impact Evaluations of EDUCO:
A Critical Review
Abstract Despite the excitement around the Education with Community
Participation (EDUCO) program during the early 1990s as it was being
scaled up, the Ministry of Education and the representatives of the World
Bank knew that they would need solid evidence which demonstrated that
the program produced beneficial outcomes in order (a) to continue to
promote the program as a central policy for education reform in El
Salvador and (b) to be able to credibly promote the program internationally as a best practice. It was in this context of excitement and determination that the World Bank began to carry out evaluations of EDUCO. With
this in mind, this chapter critically reviews six key studies that were carried
out between 1994 and 2005 by the World Bank on the EDUCO program. These six studies are included here for critical review because they
represent each of the studies that were produced as impact evaluations, all
of which were generated by the World Bank. They represent the entire
body of “legitimate” and “policy-relevant” knowledge that was created in
order to evaluate whether the program worked by identifying the effects
of the EDUCO intervention.
Keywords Impact evaluation • Critical review • Education with
Community Participation • EDUCO • World Bank • Critical review
© The Author(s) 2018
D. B. Edwards Jr., Global Education Policy, Impact Evaluations, and
Alternatives, https://doi.org/10.1007/978-3-319-75142-9_5
101
102
D. B. EDWARDS JR.
It should be clear that each of the involved actors discussed in Chap. 4 had
a vested interest in the success of the Education with Community
Participation (EDUCO) program. The minister of education had bet her
career on community-level decentralization and had made it the primary
strategy by which she sought to reform the governance of the education
system. For their part, the representatives of World Bank who were working on this program saw it as a major opportunity both to make a name for
themselves and to develop a policy that could be useful to their organization more broadly, that is, a policy that could be sold as a best practice to
other countries.1
Yet, despite the excitement around this program during the early 1990s
as it was being scaled up, the Ministry of Education (MINED) and the
representatives of the World Bank knew that they would need solid evidence which demonstrated that the program produced beneficial outcomes in order (a) to continue to promote the program as a central policy
for education reform in El Salvador and (b) to be able to credibly promote
the program internationally as a best practice (Edwards, 2018). It was in
this context of excitement and determination that the World Bank began
to carry out evaluations of EDUCO.
With the above in mind, the remainder of this chapter aims to critically
review six key studies that were carried out between 1994 and 2005 by the
World Bank on the EDUCO program. These six studies are included here
for critical review because they represent each of the studies that were
produced as impact evaluations, all of which were generated by the World
Bank. In other words, in accordance with the methodology of this study,
and given that no other entity or actor published any impact evaluations
on EDUCO, these six studies represent the entire body of “legitimate”
and “policy-relevant” knowledge that was created in order to evaluate
whether the program worked by identifying the effects of the EDUCO
intervention. While these studies were integral to the continued popularity of EDUCO during the latter 1990s and 2000s, neither a comprehensive nor a critical analysis has been carried out of their methods and
findings. In carrying out such an analysis, the goal, on one hand, is to
provide new insights into EDUCO’s knowledge base; that is, the purpose
is to further inform what is known about the program. This is useful insofar as EDUCO in particular and community-level decentralization in general continue to be strategies that are considered for implementation by
development specialists and policymakers around the world (Edwards &
Higa, not yet published). On the other hand, through the analysis below,
IMPACT EVALUATIONS OF EDUCO: A CRITICAL REVIEW
103
the goal is also to engage in a larger discussion of the role, implications,
and limitations of impact evaluations as a key element in the political economy of education reform.
In what follows, the results of the critical review are presented. First,
the context, methods, and findings of each study are presented. Then,
where appropriate, the limitations of each study are detailed. The subsequent chapter of this book will reconsider what can reasonably be asserted
with regard to the effects and lessons of EDUCO, in addition to reinterpreting the significance of these studies for education in El Salvador and
globally. The discussion below is summarized in Table 5.1.
Study 1. World Bank. (1994). El Salvador:
Community Education Strategy:
Decentralized School Management.
Washington, DC: The World Bank
Just before the end of 1994, the World Bank produced a report that contained the results of a small evaluation of the effects of the EDUCO program. Despite the limitations of the study, the results received significant
attention, given that they were the first quantitative data available which
could claim to approximate the program’s impact (World Bank, 1994).
With a sample of 78 communities, the World Bank applied an ex post
comparison2 between two groups (EDUCO vs. traditional rural public
schools) as well as ordinary least squares (OLS) regression, all using data
collected in 1993. According to this report, and with reference to the two-­
group comparison, the evaluation showed better math test results for students in grades 1 and 2, but it is important to note that no statistical
controls were applied. Background characteristics (and only very few) are
taken into account in just one instance—related to preschool academic
achievement, and in this case the EDUCO variable is not significant.3
What was significant, rather, was family income (with there being a positive relationship between this variable and student achievement). While
the point here is that the study lacked rigor, this did not prevent the minister of education and the World Bank from pointing to these results as
evidence of EDUCO’s success. Not surprisingly, the appearance and reliance on these results coincided with the timing of a major political decision by the minister of education to continue and to further expand the
EDUCO program through additional World Bank loans. Later years
Umanzor et al. Two-group, ex post comparison
(1997)
with 1996 data
2
Limitations/comment
(continued)
– Without statistical controls, it is not
possible to determine if the differences
observed in the two-group comparison
are due to the EDUCO program
– In terms of regression, which the
authors only used to predict preschool
achievement outcomes, the only
significant variable relates to family
income (larger income = higher
achievement); the EDUCO variable did
not show an impact
– ACE members feel that they
– In terms of academic achievement, the
have more influence in schools’
authors conclude that EDUCO schools
decisions (though they did not
are no worse than traditional public
make more decisions in practice) schools, even when their scores are
– ACE members visit schools
lower, because student learning scores
more often
not statistically significantly worse
– EDUCO teachers meet/contact – Among the studies reviewed here, this
parents more often
is the one with the most credibility
Two-group, ex post comparison – EDUCO students in grades 1
and regression analysis, both with and 2 show better results on math
1993 data from 78 communities exam in two-group comparison
but not in regression analysis
World Bank
(1994)
1
Findings offered
Methods, dataa
Summary of the findings and limitations of the World Bank’s Evaluative Studies of the EDUCO Program
No. Study
Table 5.1
104
D. B. EDWARDS JR.
3
Methods, dataa
(continued)
Findings offered
– EDUCO significantly reduces
student absences
Jimenez and
Regression (OLS) analysis with
– Effect of EDUCO on language
Sawada (1999) Heckman two-step correction for scores is positive and significant
sample selection bias with 1996
data
No. Study
Table 5.1
(continued)
– The more fully specified models are
not significant; the authors decide to
focus on the model with the fewest
controls but with significant results
– Endogeneity: EDUCO students
possess unobserved characteristics that
correlate negatively with achievement
outcomes and cannot parse out true
impact of intervention
– Reduction in absences could have been
the result of the timing of new school
construction/new educational services
offered (shortly before data collection in
1996) and not a result of the EDUCO
program itself (recall that many of these
communities initially lacked schools)
Limitations/comment
IMPACT EVALUATIONS OF EDUCO: A CRITICAL REVIEW
105
4
Methods, dataa
(continued)
Sawada (2000) Regression (OLS) analysis with
instrumental variables with 1996
data
No. Study
Table 5.1
– Community participation in
EDUCO associated with lower
student-teacher ratio
– EDUCO participation positively
and significantly effects student
achievement
– EDUCO resulta logros
académicos mejores (en términos
estadísticos)
– Community participation (i.e.,
ACE member school visits)
enhances teacher effort (i.e.,
meetings) with parents
– Community participation
positively and significantly related
to teacher wages
Findings offered
(continued)
– The participation variable with an
effect is not specific to EDUCO but
represents participation in by parents in
councils of traditional and/or EDUCO
schools—and, as such, we do not know
if the observed effect is due to the
EDUCO program
– Not the case in more fully specified
model
– Moreover, in models where variables
for (a) community participation and (b)
the presence of an EDUCO school are
significant, the latter is negatively related
to teacher salary. In other words, the
presence of an EDUCO school
corresponded with lower teacher salaries
– Simultaneity; not possible to separate
effects or directionality of influence
between variables. That is, both variables
(participation and teacher effort) are
interrelated and, as such, cannot indicate
a concrete and independent relationship
with the dependent variable
– Spurious relationship; EDUCO
schools in sample have fewer students
Limitations/comment
106
D. B. EDWARDS JR.
Findings offered
Limitations/comment
– Attending EDUCO increases
– Endogeneity: the unobserved
probability of remaining in school characteristics that affect the likelihood
during grades 4–6
of participating in EDUCO also
negatively affect the decision to stay in
school
Sawada and
Regression (OLS) with PSM and – EDUCO schools produce more – 76 models estimated, variables
Ragatz (2005) Heckman two-step correction
frequent community participation inconsistently significant
with 1998 data
(i.e., teacher meetings with
parents) and teacher effort (i.e.,
teaching hours) and fewer teacher
absences
– EDUCO leads to increased
– Additional controls were available
language scores
which were not used in the models
presented with statistically significant
variables; when applied, language scores
were no longer significant
Jimenez and
Regression (probit) with
Sawada (2003) Heckman two-step correction
with 1998 data
Methods, dataa
(continued)
a
Data from 1993 come from a sample of 78 schools. This sample included 33 sections of preschool and 45 sections of grades 1 and 2. In each section for
grades 1 and 2, 17 students took the language and math tests. Basic information was also collected about the family to use as controls. Data from 1996 are
based on random sample of 311 schools with a third-grade section of at least ten students. The surveys administered included language and math tests, in
addition to a household survey and questionnaires for the director, third-grade teacher, parents, and two ACE members. Data from 1998 added new student
learning tests and teacher and parent surveys to the information collected in 1996. The comparison group for all quantitative studies was traditional rural
public schools
Key: EDUCO, Education with Community Participation Program; OLS, ordinary least squares; PSM, propensity score matching
Source: Edwards (2018)
6
5
No. Study
Table 5.1
IMPACT EVALUATIONS OF EDUCO: A CRITICAL REVIEW
107
108
D. B. EDWARDS JR.
would see the production of five additional (and more rigorous) studies,
and these would constitute the international evidence used to promote the
program, as discussed below.
Study 2. Umanzor et al. (1997). El Salvador’s
EDUCO Program: A First Report on Parents’
Participation in School-Based Management. Working
Paper Series on Impact Evaluation of Education
Reforms, No. 4. Washington, DC: World Bank
The second study was conducted by the World Bank and a few staff from
the MINED. It was an impact evaluation, and it relied on data collected in
1996 (most future studies would return to these data as well) through a
random sample of 311 schools. The population of schools from which the
sample was drawn had to have a third-grade section with at least ten students. At each school, in addition to administering language and math
tests to third-grade pupils, the team interviewed and gave a questionnaire
to the director (or person in charge), the third-grade teacher, five randomly selected fifth-grade students, their parents, and two members from
the Community Education Association (ACE). The questionnaire collected background data. Although the sample initially included four types
of schools—that is, EDUCO, mixed,4 traditional public, and private—the
1997 impact study only looked at EDUCO schools and traditional public
schools located in rural areas.
Underlying these data is an important feature to note before continuing
with the study’s results. Though the EDUCO sample was identified in a
random way, this is negated by the fact that the MINED initially chose the
communities which would participate in a non-random way. That is, as
discussed previously, the MINED purposefully selected the poorest and
most disadvantaged communities for the program (economists refer to this
strategy as endogenous program selection), thus indicating that selection
bias would be a problem inherent to the quasi-experimental studies conducted later. It is important to remember that the targeted areas were also
in conflict-affected territory that was affiliated with the Farabundo Martí
National Liberation Front (FMLN) where communities had experience
with self-management of education in the absence of government services.
With these data, Umanzor et al. (1997) ran a series of ex post matched
comparisons between the EDUCO schools and the traditional public
IMPACT EVALUATIONS OF EDUCO: A CRITICAL REVIEW
109
schools on a number of characteristics of interest. They focused on associations between school and family characteristics and the outcomes
observed in the school. They found both similarities and differences in
general characteristics across groups. For example, on average, and to a
statistically significant degree, EDUCO teachers had a higher level of education but less experience. This is because, at the outset, ACEs were
required to hire teachers who had at least a university-level teaching certificate (a requirement which was possible for ACEs to meet because of
excess supply of university-trained teachers at the time, a consequence of
years of low teacher intake to the system due to the war). EDUCO schools
also had less access to electricity, but more of them had classroom libraries
(most likely because a separate USAID program funded classroom supplies). Despite better libraries, EDUCO schools did not have more textbooks. Additionally, EDUCO students were found to live in much more
rudimentary homes (as in, without electricity or piped water but with dirt
floors), yet parents were not found to help with homework any less than
parents in the comparison group. In terms of school-community interaction, ACE members felt that they had more influence in school decisions,
though they did not actually make more decisions in the school’s management (Umanzor et al., 1997). ACE members also visited their community’s school more often than did parents on the school councils of traditional
rural schools. Likewise, EDUCO teachers met with parents more often
and contacted parents more frequently when students were absent. Lastly,
with regard to the variables that have received the most attention, there
were differences in neither student absenteeism, dropout, grade repetition
nor student achievement on either the math or language tests (Umanzor
et al., 1997).
Some of these differences are cause for optimism—as with ACE parents
being more involved in the school and EDUCO teachers meeting more
with parents. Nevertheless, educational interventions are assessed primarily in terms of their ability to increase student achievement, especially, in
this case, since the theory of the EDUCO program is based on the notion
that community-level management would result in increased teacher
effort, which would lead to better student learning. Although the comparison of the treatment and control group in this study did not show that
EDUCO students had significantly better achievement than students in
rural traditional public schools, these students also did not score significantly worse than their counterparts in public schools. Umanzor et al.
(1997) took this as a positive sign, given the relatively worse economic
110
D. B. EDWARDS JR.
conditions of EDUCO students, and conclude that EDUCO students
thus exhibit “better than expected academic performance” (p. 26). The
conclusions by Umanzor et al. (1997) are the most justifiable of the
­studies reviewed here. That said, it may be that the positive results observed
were not the result of the EDUCO program, per se, but rather a result of
the pre-existing social capital that had accumulated during the civil war in
the EDUCO communities, when they were forced by their circumstances
to work together to provide basic services, such as education.
Study 3. Jimenez, E., & Sawada, Y. (1999). Do
Community-Managed Schools Work? An Evaluation
of El Salvador’s EDUCO program. The World Bank
Economic Review, 13 (3), 415–441
The third study—by Jimenez and Sawada (1999)—emerged two years
later.5 Here, Jimenez and Sawada (1999) employed more advanced econometric techniques to assess whether EDUCO had an effect on either student achievement or student absences. In order to answer these questions,
they used the same 1996 data as Umanzor et al. (1997) and applied
regression analyses along with the Heckman two-step correction
(Heckman, 1979) in order to attempt to correct for the sample selection
bias inherent in the EDUCO treatment group.6 In their study, they successively added controls for student, family, teacher, classroom, school,
and community participation characteristics. Throughout, the comparison
group is traditional rural schools.
In their discussion of student achievement, the authors state:
“EDUCO’s … effect on language test scores is positive and significant,
while its effect on math performance is positive and not significant”
(Jimenez & Sawada, 1999, p. 428). Yet, this statement is quickly followed by a concession: “Our measure of EDUCO’s advantage in language may be imprecise. The estimate of the EDUCO coefficient is
sensitive to the specification of the participation equation” (p. 428).
What they allude to is that their more completely specified models clearly
show that the results for both language and math achievement are not significant. Nevertheless, the authors make inferences in the concluding
section to their paper from models missing controls for school inputs or
community participation. They state: “[EDUCO] has improved language scores” (p. 440).
IMPACT EVALUATIONS OF EDUCO: A CRITICAL REVIEW
111
In addition to the issue of model specification, there is the matter of
endogeneity stemming from sample selection bias. That is, the results provided by Jimenez and Sawada (1999) are confounded by the fact that the
students who attended EDUCO schools also possessed unobserved
characteristics that correlated with student achievement outcomes.
­
Jimenez and Sawada (1999) recognize this when they write, “The error
terms of the participation and achievement equations are negatively correlated. This means that EDUCO students have unobserved characteristics that are negatively correlated with achievement test scores” (p. 428).
Under conditions of endogeneity, one cannot truly parse out the impact
of the intervention on student achievement because the participation of
the student in the intervention is correlated with other factors which are
also correlated with student achievement.
Additionally, alternative conclusions can be drawn from the student
achievement regression results, keeping in mind their limitations. In math,
for example, although the EDUCO variable is not significant, what does
contribute to improved test scores is the availability of sanitary services.
With regard to language test scores, it is having a classroom library that
contributes positively. For both math and language, more frequent visits
to the school by ACE parents are positively associated with higher student
achievement. The data thus support the conclusions that schools in which
parents are involved and which possess basic resources encourage better
student performance. Given that the focus of the evaluation was to determine EDUCO’s impact, these findings are not highlighted.
The second question posed by Jimenez and Sawada (1999) is whether
there is an EDUCO effect on student attendance. They use the same data
and controls as with their models for student achievement, changing only
the dependent variable to indicate the number of days the student was
absent in the month prior to the survey. Here, they find that EDUCO
significantly reduces the number of days a student was absent. They thus
conclude that the accountability relationships instituted between parents
and teachers under EDUCO are responsible for reducing student absences.
However, if we decompose the effect of the EDUCO variable according to when the EDUCO schools were built, the picture changes (Jimenez
& Sawada, 1999). If the EDUCO school was built between 1991 and
1995, there was no effect; however, if the school was built in 1996, it
significantly decreased student absences. As opposed to attributing the
effect to relationships of accountability, the result could simply be due to
the fact that students received a new school in the same year that the
112
D. B. EDWARDS JR.
survey was administered. The presence of a new school—where there had
not been one previously—could significantly increase student
attendance.
Nevertheless, the authors offer a particular conclusion, one that aligned
with the theory of community-level accountability relationships and their
ability to lead students and teachers to perform better (Jimenez & Sawada,
1999). These conclusions are powerful, particularly given the fact that
they are offered by the World Bank and the fact that they were derived
from methods that are widely seen as rigorous and objective. To that end,
the literature shows that other authors have repeated these conclusions, as
did Barrera-Osorio, Fasih, Patrinos, and Santibañez (2009) in their review
of the effectiveness of school-based management in developing countries.
(See Edwards, 2018, Chap. 9, for additional examples.) Over time, these
conclusions reinforce the idea that community-level decentralization can
lead to fewer student absences.
Study 4. Sawada, Y. (2000). Community Participation,
Teacher Effort, and Educational Outcome:
The Case of El Salvador’s EDUCO Program.
Michigan, USA: The William Davidson Institute,
University of Michigan Business School
The fourth study was conducted by Sawada (2000) and attempted to
determine if the EDUCO program had an effect on student achievement,
teacher pay, teacher effort, or classroom inputs. Numerous models were
estimated using largely the same data and controls as were employed by
Jimenez and Sawada (1999), the difference being that Sawada (2000) uses
instrumental variables in an attempt to remedy the endogeneity which
results from the nature of the sample.7 In the end, four conclusions were
drawn, each of which is reviewed here.
The first conclusion stated that community participation was positively
related to teacher wages. However, although Sawada (2000) runs eight different models to test this, under the most complete specification, neither
the community participation variable nor the EDUCO variable is significant. The author refers in his conclusion to an OLS regression which did
not control for teacher characteristics. Moreover, in this particular model,
while community participation (defined as the number of times parents on
the ACE visit the school monthly) is positively associated with increased
IMPACT EVALUATIONS OF EDUCO: A CRITICAL REVIEW
113
wages, what Sawada (2000) does not mention is that the EDUCO variable (indicating the presence of a school managed by an ACE) is significantly and negatively related to wages.
Second, Sawada (2000) highlights the relationship between teacher effort
and community participation in EDUCO schools. Here, the model does
indeed show a statistically significant relationship between school visits by
ACE members and the number of hours teachers spend each month meeting with parents. However, the latent constructs, as measured and specified here, experience simultaneity, and, as such, it is not possible to separate
the effect of community participation from teacher effort. This is not to
say that there is not a relationship between the two variables. To be sure,
parents spending more time at school would be intertwined with the number of hours teachers spend meeting with parents. What remains unclear is
the nature and directionality of the relationship. Nevertheless, Sawada
(2000) concludes: “Community participation seems to enhance the
teacher effort level” (p. 26).
For the third finding, Sawada (2000) reports that there is a connection
between community participation and teacher-student ratio. In particular,
the finding is that the former causally decreases the latter. This is a curious
finding because the relationship between community participation and
teacher-student ratio is not clear. That is, one wonders why teacher-­
student ratio would be a function of community participation. While
Sawada (2000) does not offer an explanation of why this would be, it is
suggested here that this may be the result of the nature of the sample:
EDUCO schools were located in particularly rural areas, where there
tended to be fewer children per teacher.
Fourth and finally, Sawada (2000) speaks to student achievement.
Although the EDUCO variable was not in any of the model specifications
significantly related to test scores, the study is bookended by statements
which, at first glance, indicate the contrary. At the beginning and at the
very end of the paper, Sawada (2000) writes, “We observe consistently
positive and significant EDUCO participation effects on standardized test
scores” (pp. 1, 25, emphasis added). What he refers to, however, is not a
variable that represents parental participation in EDUCO but rather a variable for parental participation generally. The problem with this wording is
that the parental participation variable which was shown to be associated
with student achievement is not specific to EDUCO. Instead, it is a variable that represents participation in schools generally—by either parents of
EDUCO students or parents of students in traditional rural public schools.8
114
D. B. EDWARDS JR.
Such statements can filter through to future reviews of empirical work on
EDUCO and decentralization more generally.
Study 5. Jimenez, E., & Sawada, Y. (2003). Does
Community Management Help Keep Kids in Schools?
Evidence Using Panel Data from El Salvador’s
EDUCO Program. Discussion Paper, Center
for International Research on the Japanese
Economy. Tokyo, Japan: University of Tokyo
In the fifth study, Jimenez and Sawada (2003) attempt once more to
uncover an EDUCO effect.9 This time, they return to the question of
whether participating in EDUCO is associated with staying in school over
the two-year period 1996–1998. To answer this question new data collected in 1998 (from student learning tests and teacher and parent surveys
[Lindo, 1998, p. 21]) are added to the 1996 data and then run through
regression analyses (specifically, they use probit models) which utilize the
Heckman two-step correction for sample selection bias and the same controls as their other studies (Jimenez & Sawada, 1999; Sawada, 2000).
Once again, the authors acknowledge the inherent limitations they face
due to sample selection bias and the effects of this on their results. They
reveal that the error terms for the selection and substantive equations are
negatively correlated. As a result, Jimenez and Sawada (2003) note that
“unobserved characteristics regarding the children, households, and communities might positively affect the likelihood of selecting an EDUCO
school and might negatively … affect a student’s decision to continue in
school” (pp. 21–22). The implication is that their findings may suffer from
endogeneity and, as such, may not be valid.
Nevertheless, the findings of Jimenez and Sawada (2003) can be found
cited in the literature on education decentralization in developing countries. (Again, see Appendix 4.) One example is Barrera-Osorio et al.
(2009), who cite results from model one in the study by Jimenez and
Sawada (2003). While the results cited indicate that attending an EDUCO
school increases the probability of remaining in school for two years by 64
percent, we cannot be sure due the issue of endogeneity. Furthermore, in
this study, the sample included “mixed” schools that contained classes
that were traditionally managed as well as classes with teachers that were
managed by ACEs. It is reasonable to think that there may have been a
IMPACT EVALUATIONS OF EDUCO: A CRITICAL REVIEW
115
peer effect in these schools, as a result of mixing the two models. This
mixing also could have affected the results observed by the authors.
Study 6. Sawada, Y., & Ragatz, A. (2005).
Decentralization of Education, Teacher Behavior,
and Outcomes: The Case of El Salvador’s EDUCO
Program. In E. Vegas (Ed.), Incentive to Improve
Teaching: Lessons from Latin America. (pp. 255–306).
Washington, DC: World Bank
The sixth and final study came about in 2005, this time by Sawada and
Ragatz (2005). It builds on Jimenez and Sawada (1999). In this incarnation, the study asks how decentralization affects administrative processes,
teacher behavior, and student achievement. In addition to using the
Heckman two-step correction procedure, it also makes use of propensity
score matching—a strategy through which the compared treatment and
control communities are matched along a series of characteristics to ensure
that they are as identical as possible, given the communities in the sample.
In their analysis, Sawada and Ragatz (2005) eliminated the mixed schools
included by Jimenez and Sawada (2003), thus reducing the sample to 37
EDUCO schools and 96 traditional schools. In some models, the same
controls as before were used. Those instances where the controls were different are discussed below.
The results are interesting. In terms of administrative processes, the
only activity to occur more often at the school level in EDUCO schools
concerns hiring and firing teachers (Sawada & Ragatz, 2005). Neither
determining teacher salaries, giving teachers incentives, evaluating teachers nor spending school money occurred more often at the school level in
EDUCO than in traditional public schools. Not surprisingly, the results
show that teachers’ association relations (i.e., union meetings) occurred
more often in traditional rural public schools. This was the case because
teachers hired by ACEs are necessarily non-union (due to their annual
contractual status, which precluded them from belonging to teachers’
unions), and thus there was no teachers’ association activity to observe.
For teacher behavior, interpreting the results is like reading an Ouija
board. There are models for 19 different dependent variables. Each dependent variable tells us something different about teacher behavior—for
example, how many hours per week the instructor teaches, how many
116
D. B. EDWARDS JR.
hours they meet with parents, or how often they are absent. Further complicating the picture is that each model is then run in four different ways—
using (a) standard (i.e., OLS) regression with 42 independent variables,
(b) the Heckman two-step procedure, (c) revised OLS regression with 32
independent variables, and (d) propensity score matching with 32 independent variables.10 The problem across these 76 different combinations
is that there is tremendous variability in terms of what is significant. Those
results that are significant under propensity score matching with 32 controls are then non-significant in the OLS regression with 42 controls, for
example. Elsewhere, the Heckman models return significant coefficients,
but, in these cases, none of the other models do. In the end, there is not
one outcome variable that is significant across all four forms of regression.
Sawada and Ragatz (2005) still report, however, that “many key measures
survived the bias tests, including meeting with the parents, teacher
absence, and hours dedicated to teaching” (p. 295).
In addition to the above finding, the authors offer two additional conclusions based on the regression results. The first is as follows:
The results indicate that community participation seems to enhance the
teacher effort level, possibly because of intensive monitoring of teacher
behaviors and the implicit threat that exists because hiring and firing of
teachers occurs at the community level. (Sawada & Ragatz, 2005, p. 295)
And, second, “Community participation not only uses relevant information that outside government agencies are not likely to have, but also
imposes commitment on teachers, which leads them to exert greater
effort” (p. 295).
It is suggested here, however, that these assertions are not borne out by
the regression results, given the variability and inconsistency across models
discussed above. It seems that the authors were either confident in the
regression results or had a predilection to explain the results in terms of
accountability relationships that resonate with the World Bank’s theory of
community-level management of education, or both. In the end, and in
contrast with the interpretation by Sawada and Ragatz (2005), it is suggested here that the true effect is not known for any of the significant
variables presented by the authors. As Klees and Edwards (2014) assert:
Regression analysis theory … offers no guidance as to how inaccurate the
resulting regression coefficients are under real-world misspecification conditions …. With so many alternative specifications from which to choose,
IMPACT EVALUATIONS OF EDUCO: A CRITICAL REVIEW
117
regression analysts can almost always find their favorite variable to be
­significant, and thus the empirical literature becomes a debate over who has
the best specification. (p. 34)
With regard to Sawada and Ragatz (2005), the point is that widely different conclusions can be drawn depending on the specification one prefers,
and there is no way to resolve which specification is best.
The final question of this study pertained to student achievement.
Output is presented for a handful of coefficients across four specifications.
Three of the four models show significant effects for language scores of
being in an EDUCO school compared with being in a traditional rural
public school. What the models do not show, however, are the authors’
controls, though the footnote to the table does note that child and household characteristics were taken into account. Concerningly, Sawada (2000)
and Jimenez and Sawada (1999) ran the same models with additional controls for teacher quality, classroom quality, and school quality, and under
those circumstances no significant effect of EDUCO on student achievement was found. One wonders if Sawada and Ragatz (2005)—either purposefully, subconciously or through what has been termed “motivated
reasoning” (Achenbach, 2015; Kahan, 2013)—specified their models to
generate significant effects.11 There is no way to know for certain.
Nevertheless, the fact that fewer controls were employed puts in question
the conclusion they offer that “EDUCO program governance leads to
better effort of teachers, which improves educational outcomes” (Sawada
& Ragatz, 2005, p. 299).
Having critically reviewed the methods and findings of the six impact
evaluations of EDUCO, the next chapter turn to reappraise and reinterpret both the stated findings of these evaluations as well as the role that
these evaluations played more generally in the political economy of education reform.
Notes
1. On this point, it is necessary to recall the tremendous popularity of decentralization as a theme within the international development industry during
the 1990s (Edwards & DeMatthews, 2014) as well the predominance of the
World Bank during this time vis-à-vis other multilateral organizations, especially in the education sector (Mundy, 1998).
2. Meaning that the EDUCO schools and the traditional schools were selected
for comparison after the EDUCO intervention began.
118
D. B. EDWARDS JR.
3. Controls were included for family income, gender, and father’s ability to
read (World Bank, 1994).
4. Until 1999, the MINED allowed EDUCO and traditional schools to coexist through what it called “mixed” schools (Lindo Fuentes, 1998). Here, a
traditional school would also receive funding from the EDUCO program,
and the community would establish an ACE. With the funding received, the
community would hire a teacher for lower grades (grades 1–3). The school
would then use its non-EDUCO budget to pay a teacher to provide grades
4–6. As mentioned, the EDUCO program was initially targeted to communities where educational services were not provided during the civil war.
However, as the MINED began to scale up the program, it did so by creating “mixed” schools, as well.
5. It should be noted that Emmanuel Jimenez was at that time a member of
the World Bank’s influential Development Economics Vice-Presidency and
was well respected among his peers. The fact that he took up the cause of
EDUCO by doing research on it attracted the attention of others within the
Bank.
6. The Heckman two-step correction is employed where there are issues with
sample selection bias. Sample selection bias is normally a problem for regression analysis because, when the treatment and comparison groups systematically differ on a range of characteristics, we cannot be sure that the effect
observed for the treatment (e.g., EDUCO) is unbiased. That is, the effect
(or lack of it) could be due to unobserved variables related to the treatment
(labeled by economists as endogeneity). Under endogeneity, we can also not
be sure of the extent to which the estimates for the other variables in equation are biased as well (Berk, 1983). The best way to address this issue is to
first draw a representative random sample of participants from the population of interest. In most cases this is not possible, especially in social science
research. Such was the case with EDUCO, where participating communities
were not chosen randomly and where participating communities differed
systematically from a nationally representative sample.
Thus, to deal with this common issue, James Heckman developed his now
famous correction technique which treats sample selection bias as specification error (Heckman, 1979). Per this technique, a separate estimator term
is added to the principal regression equation of interest (i.e., the “substantive equation”). This term is arrived at by estimating through a (probit)
selection equation the likelihood of a student, for example, being placed in
an EDUCO school, given a set of selected characteristics. This probability,
then, is fed back into the substantive equation as part of a ratio (the inverse
Mills ratio) that represents the probability that a child with given characteristics will be excluded from the sample from which results are generalized,
conditional on participation in EDUCO (Berk, 1983). Ideally, then, the
IMPACT EVALUATIONS OF EDUCO: A CRITICAL REVIEW
119
added estimator is a sample selection correction term that adjusts for the fact
that certain characteristics are overrepresented in the sample being examined. From the regression output, one can then interpret a statistically significant result for this term’s coefficient as indicating that there is sample
selection bias. That is, a statistically significant coefficient signals that certain
characteristics are over or underrepresented.
Fundamental issues surface in practice, however. For a thorough review of
the many issues that can and do arise in practice through the use of the
Heckman two-step correction, see Bushway, Johnson, and Slocum (2007).
For example, if the same variables appear in both the selection equation and
the substantive equation, the issue of multicollinearity arises (Bushway et al.,
2007). That is, in concrete terms, if the same variables used to predict a
student’s participation in EDUCO are also used to predict a student’s test
scores, then the correction term added to the equation for test scores may
well covary with the other independent variables. What’s more is that this
technique assumes that the error terms for both the selection and substantive equation are jointly normal, meaning that they are independent of each
other (Heckman, 1979). However, the error terms will be correlated if there
are unobserved characteristics (omitted variables) relevant to the dependent
variables of both the selection and substantive equations. It should be noted
that, in the case of EDUCO, this is almost certainly the case. The reason is
as follows: socioeconomic conditions determined who participated in
EDUCO; these same poverty-related characteristics always influence student achievement. Ultimately, then, although this fix works well in theory,
in practice it actually worsens issues of misspecification because there are
now two linked regression equations to specify correctly and which themselves must not be correlated.
7. Instrumental variables are employed in regression analysis under conditions
of endogeneity. The idea is that one seeks to “find a variable (or instrument)
that is highly correlated with program placement or participation but that is
not correlated with unobserved characteristics affecting outcomes”
(Khandker, Koolwal, & Samad, 2010, p. 87). Instrumental variables must
be selected thoughtfully, as a weak instrumental variable—one that is correlated with unobserved characteristics or omitted variables—can worsen
the bias of coefficient estimates. Ideally, the instrumental variable serves as a
proxy for program participation while eliminating issues of endogeneity. See
the previous footnote for more on endogeneity.
8. The parents’ associations for traditional schools were known as Sociedades
de Padres de Familia (SdPF) or Family Parent Societies (Sawada, 2000,
p. 3).
9. A revised version of this study was later published in the journal, Economic
Development and Cultural Change (Jimenez & Sawada, 2014).
120
D. B. EDWARDS JR.
10. Propensity score matching could not be run using all 42 variables because
the communities were too dissimilar, according to Sawada and Ragatz
(2005).
11. In the words of Kahan (2013), “motivated reasoning refers to the tendency of people to conform assessments of information to some goal or
end extrinsic to accuracy … The goal of protecting one’s identity or standing in an affinity group that shares fundamental values can generate motivated cognition relating to policy-relevant facts” (p. 408).
References
Achenbach, J. (2015, August 27). May scientific studies can’t be replicated. That’s
a problem. The Washington Post. Retrieved from https://www.washingtonpost.com/news/speaking-of-science/wp/2015/08/27/trouble-in-sciencemassive-effort-to-reproduce-100-experimental-results-succeeds-only-36-times/
Barrera-Osorio, F., Fasih, T., Patrinos, H., & Santibánez, L. (2009). Decentralized
decision-making in schools: The theory and evidence on school-based management.
Washington, DC: The World Bank.
Berk, R. (1983). An introduction to sample selection bias in sociological data.
American Sociological Review, 48(3), 368–398.
Bushway, S., Johnson, B., & Slocum, L. (2007). Is the magic still there? The use
of the Heckman two-step correction for selection bias in criminology. Journal
of Quantitative Criminology, 23(2), 151–178.
Edwards, D. B., Jr. (2018). The trajectory of global education policy: Community-­
based management in El Salvador and the global reform agenda. New York:
Palgrave Macmillan.
Edwards, D. B., Jr., & DeMatthews, D. (2014). Historical trends in educational
decentralization in the United States and developing countries: A periodization
and comparison in the post-WWII context. Education Policy Analysis Archives,
22(40), 1–36. https://doi.org/10.14507/epaa.v22n40.2014.
Edwards, D. B., Jr., & Higa, S. (not yet published). The global education policy
of school-based management in conflict-affected contexts: Current reach,
prominent rationales, future research. Policy Futures in Education.
Heckman, J. (1979). Sample selection bias as a specification error. Econometrica,
47(1), 153–161.
Jimenez, E., & Sawada, Y. (1999). Do community-managed schools work? An
evaluation of El Salvador’s EDUCO program. The World Bank Economic
Review, 13(3), 415–441.
Jimenez, E., & Sawada, Y. (2003). Does community management help keep kids in
schools? Evidence using panel data from El Salvador’s EDUCO program.
Discussion Paper, Center for International Research on the Japanese Economy.
Tokyo: University of Tokyo.
IMPACT EVALUATIONS OF EDUCO: A CRITICAL REVIEW
121
Jimenez, E., & Sawada, Y. (2014). Does community management help keep kids
in schools? Evidence using panel data from El Salvador’s EDUCO program.
Economic Development and Cultural Change, 62(2), 307–338.
Kahan, D. (2013). Ideology, motivated reasoning, and cognitive reflection.
Judgment and Decision Making, 8(4), 407–424.
Khandker, S., Kookwal, G., & Samad, H. (Eds.). (2010). Handbook on impact
evaluation Quantitative methods and practices. Washington, DC: World Bank.
Klees, S., & Edwards, D. B., Jr. (2014). Knowledge production and technologies
of governance. In T. Fenwick, E. Mangez, & J. Ozga (Eds.), World yearbook of
education 2014: Governing knowledge: Comparison, knowledge-based technologies
and expertise in the regulation of education (pp. 31–43). New York: Routledge.
Lindo Fuentes, H. (1998). Comunidad, participación y escuelas: EDUCO en El
Salvador. [Community, participation and schools: EDUCO in El Salvador.].
Washington, DC: World Bank.
Mundy, K. (1998). Educational multilateralism and world (dis)order. Comparative
Education Review, 42(4), 448–478.
Sawada, Y. (2000). Community participation, teacher effort, and educational outcome: The case of El Salvador’s EDUCO program. Michigan: The William
Davidson Institute, University of Michigan Business School.
Sawada, Y., & Ragatz, A. (2005). Decentralization of Education, Teacher Behavior,
and Outcomes: The Case of El Salvador’s EDUCO Program. In E. Vegas
(Ed.), Incentive to improve teaching: Lessons from Latin America (pp. 255–306).
Washington, DC: World Bank.
Umanzor et al. (1997). El Salvador’s EDUCO program: A first report on parents’
participation in school-based management. Working Paper Series on Impact
Evaluation of Education Reforms, No. 4. Washington, DC: World Bank.
World Bank. (1994). El Salvador: Community education strategy: Decentralized
school management. Washington, DC: The World Bank.
CHAPTER 6
Reconsidering the EDUCO Program
and the Influence of Its Impact Evaluations
Abstract This chapter accomplishes two tasks. First, it reevaluates what is
known about the Education with Community Participation (EDUCO)
program in light of the critical review provided in Chap. 5. Second, it
reconsiders the influence of the impact evaluations of EDUCO in terms of
national and international implications. Not only did these studies provide
justification for scaling up EDUCO within El Salvador, but, internationally, they changed how international organizations and development professionals thought about decentralization and provided a basis from which
these organizations and individuals could promote an extreme form of
community-based management (one where parents are responsible for
hiring and firing teachers, among other things). Crucially, it is also shown
that, due to the knowledge base that has been created by the World Bank
in the form of impact evaluations, EDUCO has taken on a life of its own
and continues to live on in the literature on decentralization as well as
school- and community-based management.
Keywords Impact evaluation • El Salvador • World Bank • EDUCO •
Political economy • Global education policy
This chapter accomplishes two tasks. First, it reevaluates what is known
about the Education with Community Participation (EDUCO) program
© The Author(s) 2018
D. B. Edwards Jr., Global Education Policy, Impact Evaluations, and
Alternatives, https://doi.org/10.1007/978-3-319-75142-9_6
123
124
D. B. EDWARDS JR.
in light of the critical review provided in Chap. 5. Second, it reconsiders
the influence of the impact evaluations of EDUCO in terms of national
and international implications. As will be discussed, not only did these
studies provide justification for scaling up EDUCO within El Salvador,
but, internationally, they changed how international organizations and
development professionals thought about decentralization and provided a
basis from which these organizations and individuals could promote an
extreme form of community-based management (one where parents are
responsible for hiring and firing teachers, among other things). Crucially,
it is also shown that, due to the knowledge base that has been created by
the World Bank in the form of impact evaluations, EDUCO has taken on
a life of its own and continues to live on in the literature on decentralization as well as school- and community-based management.
What Can We Say About EDUCO?
The previous chapter places doubt on the findings that have been underscored in the impact evaluations of the EDUCO program. In light of the
critical review presented in Chap. 5, these studies can be interpreted as
repeated attempts to find evidence that shows the beneficial effects of a
particular version of decentralization. It was determined that World Bank
researchers and publications, in many instances, offered interpretations
which were questionable (based on the limitations of their methods and
data and the possible alternative explanations). In view of the reappraisal
in Chap. 5, it is argued here that not one of the studies reviewed definitively shows that the EDUCO program produced positive and significant
results on the variables of most interest to the World Bank and the Ministry
of Education (MINED). To the contrary, it is concluded that the EDUCO
program does not lead to improved student achievement, student attendance, student retention, or teacher absenteeism when the available controls are applied.1
The one finding which stands out across the studies is that EDUCO
schools did no worse (in statistically significant terms) on student achievement than traditional rural public schools (Umanzor et al., 1997).
EDUCO proponents have consistently interpreted this as a positive finding: since EDUCO schools were located in the most disadvantaged areas
of the country, one might have expected them to perform much worse
than their public school counterparts. However, given that EDUCO
schools performed just as well (i.e., no worse), EDUCO advocates
RECONSIDERING THE EDUCO PROGRAM AND THE INFLUENCE OF ITS…
125
c­ onclude that, under the EDUCO model, we can introduce an arrangement that is more accountable without sacrificing quality.
Yet one cannot be sure that the effect is due to the EDUCO model.
First, it must be remembered that the data for these studies were collected
during 1993–1998; second, it must be remembered that EDUCO schools
were, at the outset, purposefully targeted to FMLN communities (Reimers,
1997) as a way to undermine the system of popular education that had
developed during the civil war (Edwards, 2018). It should be noted that
the popular education system was not only present in the areas controlled
by the Farabundo Martí National Liberation Front (FMLN), but it was
also substantial in its reach: by the early 1990s, this system included
approximately 1000 teachers and 13,500 students (Alvear Galindo, 2002,
pp. 189, 200), although according to Reimers (1997) this system served
as many as 17,000 students. The implication is that, during the time when
data were being collected for the reviewed studies, the communities
included in the sample were in fact FMLN communities that had previous
experience with self-organization through the system of popular education as well as through cooperative and democratic forms of agricultural
production, communication, housing, and community government
(Asociación de Desarrollo Económico Social [ADES], 2005). This realization is problematic for the conclusions drawn by EDUCO supporters
because it means that the findings which they have attributed to the
EDUCO model could rather be the result of the practice with self-­
organization that these communities had during the civil war as part of
their efforts to prepare for the reality of a socialist and cooperative society
after the war, were the FMLN to win (Montgomery, 1995). Put differently, as opposed to attributing the results to EDUCO the results could
just as likely be attributed to the pre-existing social capital possessed by the
communities which participated in EDUCO.
Given the plausibility of this alternative explanation, there are now no
results which suggest the superiority—or even the equivalence—of
EDUCO to traditional rural public schools, particularly in terms of such
outcomes as student achievement, student attendance, student retention,
or teacher absenteeism. With regard to parental participation, Umanzor
et al. (1997) did show that parents in EDUCO schools visit their child’s
classroom more times per month (5.7 times vs. 1.6 times) and spend more
time meeting with teachers (EDUCO teachers spent 4.9 h/week meeting
with parents, versus 2.9 h/week for non-EDUCO teachers). But again,
these findings may be spurious because the groups of parents being
126
D. B. EDWARDS JR.
c­ompared were likely not the same in terms of their social capital, as
explained above. The results may have been equivalent for EDUCO and
traditional rural schools if the parents from the latter group had the same
pre-existing social capital.
Interestingly, however, in terms of lessons learned, the reappraisal of
the evidence actually suggests a contrary conclusion to that offered by the
authors of the studies. To reiterate, if we recall that the “EDUCO communities” in the impact evaluations are FMLN communities that had
adopted the EDUCO program, then any positive benefits observed may
be a reflection not of the design of the EDUCO program but rather a
reflection of the experience of cooperative self-organization and the social
capital that consequently accumulated with the parents who, under
EDUCO, served on the school councils. Seen in this light, the impact
evaluations of EDUCO are, ironically, evidence in favor of the opposition,
that is, evidence in favor of those same communities that the government
sought to undermine, and which the government did in fact undermine
through the requirement that EDUCO schools could only hire certified
teachers, which necessarily prevented the Community Education
Associations (ACEs) from hiring popular education teachers, since they
had low levels of education without the credentials specified under
EDUCO’s regulations. But the fact that EDUCO successfully subjugated
FMLN communities to the official system—despite proposals from popular educators to allow for two parallel systems (Aguilar, Becera, Burgos,
Castro, & García, 1993)—does not erase the social capital that accrued to
parents nor their prior experiences with self-organization, both of which
informed and arguably improved their work under EDUCO when it came
to hiring and supervising teachers and managing the school budget among
other tasks. Thus, the findings by Umanzor et al. (1997) that EDUCO
parents visited their children’s classrooms more often and spent more time
meeting with teachers is likely the result of efforts at community organization that go back to the 1970s, when rural Salvadorans began to mobilize
in response to governmental repression and in accordance with the critical
teachings of liberation theology (Montgomery, 1995). These same communities later became affiliated with the FMLN and put into practice critical teachings through their self-managed community schools and the
methods of critical educators such as Paulo Friere (Edwards & Klees,
2012). Ultimately, then, to the extent that the impact evaluations justifiably show an effect (as in Umanzor et al., 1997), this critical review suggests that this effect could be interpreted as supporting forms of community
RECONSIDERING THE EDUCO PROGRAM AND THE INFLUENCE OF ITS…
127
education management that are rooted in broad, community-wide cooperative organization, solidarity, and popular education, as opposed to a
more narrow, neoliberal version based on accountability, student test
scores, the ability to fire teachers, and abstract notions of efficiency. These
two versions of community participation in education governance have
been referred to as progressive and neoliberal, respectively (Edwards &
Klees, 2015).
In sum, then, the same studies that emphasize the benefits of the
EDUCO program can be interpreted differently to offer alternative conclusions. These alternative conclusions clearly cast doubt on the advisability of EDUCO as a public policy. Furthermore, by placing this critical
review in the political-economic context of the 1990s, it has been shown
that EDUCO, on the whole, (a) was a program which did not improve
key indicators of quality, (b) imposed costs on the rural parents2 (who
volunteered their time on the ACEs and who helped to build and repair
the schools in their communities), (c) sought to weaken the teachers
unions (by instituting one-year contracts for EDUCO teachers), and (d)
was directed at incorporating FMLN communities into the official education system (by replacing their popular education teachers and schools
with the EDUCO program and with teachers from outside their communities). Of course, this alternative interpretation highlights the political
motivations behind EDUCO. But it needs to be remembered that
EDUCO was always a political decision. Moreover, and not surprisingly, it
has been in the realm of politics that the EDUCO program has continued
to have relevance, due in no small part to the studies reviewed here, as is
discussed in the next section.
Implications: National and International
Within the particular political-economic context of the 1990s and 2000s,
these studies have contributed to numerous developments, both inside
and outside El Salvador. In what follows, the implications are discussed in
national and then international terms. It should be remembered that the
implications discussed below are not presented as being the result of the
EDUCO studies alone; rather, it is suggested that the EDUCO studies are
one contributing factor among many within the historical context characterized earlier in this manuscript.
128
D. B. EDWARDS JR.
National Implications
First, within El Salvador, these studies initially influenced the decision in
1994 to scale up the program countrywide, and then later served to maintain the focus of the MINED on EDUCO, even after the transition to a
new political administration in 1998 and again in 2003. Gillies (2010) has
also noted this latter point. Of course, the other side of the coin is that
these studies, and the justifications they offered, constituted significant
obstacles to overcome for those who opposed the program, particularly
because of their econometric nature, and thus the perception that their
findings were definitive, not to mention the difficult-to-unpack nature of
the methodology itself. At once, then, these studies provided political
capital to those in power while also raising barriers to reform for those
who sought to modify or repeal the program.
A second (and ironic) implication is that these studies—which popularized EDUCO for enhancing teacher accountability and for reflecting a
neoliberal logic with regard to efficiency and effectiveness—actually
helped to extend a model with roots in the FMLN’s approach to development, which was based in solidarity, community empowerment, and popular education (ADES, 2005; Edwards & Ávalos, 2015). However, as
noted, the FMLN model was sanitized and reframed according to market-­
based principles. Amid the focus on this new management model, what
received less attention was the fact that, through EDUCO, a second-class
system of schools was created that continued to marginalize large portions
of the student population. This outcome resulted from the fact that, in
addition to hiring less-experienced teachers, EDUCO communities, which
already tended to be located in poor rural areas, were also expected to
voluntarily contribute to the construction, maintenance, and management
of their community school (note that none of the parents on the ACEs
received compensation for their time).
More generally, and perhaps more perniciously, these studies contributed to the development and general acceptance of myths around the
EDUCO program. In El Salvador, during the last 20 years, it has become
common to espouse the notion that EDUCO is a program which introduces teacher accountability and which produces better student achievement, among other things. By repeating these assertions, researchers,
political actors, and education commentators have helped to spread a
“common sense” around EDUCO.
RECONSIDERING THE EDUCO PROGRAM AND THE INFLUENCE OF ITS…
129
International Implications
First, internationally, the reviewed studies have been used to promote the
implementation of a model that not only transfers to the community part
of the responsibility for education governance but also transfers part of the
cost of educational provision. That is, while the EDUCO impact evaluations have provided the justification for countries to experiment with
community-­level decentralization, in practice, these studies have provided
political cover for the introduction of decentralization policies that require
(whether officially or unofficially) that community members (a) make
donations to the school, (b) help with building and repairing school buildings and other structures (such as latrines), (c) contribute to school management voluntarily (e.g., through monitoring teachers, managing the
budget, purchasing materials), and (d) participate in administrative or
supplemental tasks (e.g., the development of school improvement plans or
the provision of school snacks for students) (Edwards & DeMatthews,
2014).
The second international implication, which relates to the first, is that
these studies have promoted the belief among development practitioners
that EDUCO-style community decentralization is a preferred reform for
education systems around the world. A prominent example is the World
Bank. As the studies reviewed here began to be produced, they garnered
attention outside of the staff members who initially worked on the
EDUCO program, including prominent staff within this organization.
Indeed, as Gershberg, Meade, and Andersson (2009) note, based on personal observation, “Jimenez and Sawada (1999) was influential in convincing World Bank staff that EDUCO was a successful model” (p. 198).3
Moreover, the evidence base constituted by both Jimenez and Sawada
(1999) and Sawada (2000) grabbed the attention of the authors of the
World Development Report 2004 (WDR 2004), which is the flagship publication of the World Bank (World Bank, 2003). In this Report, which has
come be regarded as a guide for how to introduce relations of accountability in public sector reform in developing countries, the EDUCO program was highlighted as the key model to emulate in the education sector.
By being included in the WDR 2004, EDUCO’s reach among development professionals was thus amplified, thereby increasing the influence of
this model on the global education reform agenda. Were it not for the
findings offered in the EDUCO studies, this would not have been possible. For additional examples of the ways that EDUCO was invoked in the
130
D. B. EDWARDS JR.
1990s and 2000s, after these studies began to come out, see Edwards
(2018, Chap. 9).
Third, in raising the profile of the EDUCO program, the EDUCO
studies also helped to raise the profile of the personnel from the MINED
who were instrumental in institutionalizing and implementing the
EDUCO program during the 1990s. For example, following Cecilia
Gallardo de Cano, who was the minister of education (1990–1998) when
EDUCO began, three of the four ministers of education during 1998–2009
(after which point the presidency passed from ARENA to the FMLN)
previously either worked for the EDUCO program or otherwise contributed to its furtherance.4 Of course, without the positive evidence presented by the EDUCO studies reviewed here, these actors would have
lacked the ability to claim success, or to have success attributed to them,
thereby making it more difficult to generate the political capital necessary
to assume such high-profile positions.
Fourth, these studies helped to generate interest from other countries.
Governmental study groups came from, at least, Argentina, Bangladesh,
Brazil, Chile, Equatorial Guinea, Honduras, Niger, and Thailand (Meza,
Guzmán, & de Varela, 2004). MINED representatives themselves also
went to numerous other countries. MINED interviewees in Edwards
(2018) mentioned that, over time, they were sent by the World Bank to
such countries as Bolivia, Brazil, China, Colombia, Guatemala, Honduras,
Mexico, Spain, and the United States. Not surprisingly, the World Bank
played a key role by financing these study trips, which was not an unusual
practice during the 1990s. Separately, these impact evaluations—together
with the movement of World Bank staff—have influenced the design and
adaptation of very similar models at least in Afghanistan, Burkina Faso,
Guatemala, Honduras, Nepal, Niger, Senegal, and Uganda, although it
should be noted that there may be more cases of the direct influence of the
reviewed studies, only that it is not possible to identify them because the
individuals inspired or unknowingly influenced by the evidence on
EDUCO do not always cite it publically as a point of reference (Najjumba,
Habyarimana, & Bunjo, 2013; Sawada, Kozuka, Aida, Griffen, Noguchi,
& Todo, 2014; Shibuya, 2013). To that end, an education specialist from
the World Bank with over 20 years of experience in El Salvador and other
countries explained how the lessons from EDUCO have become part of
“international knowledge,” such that development practitioners and governmental functionaries in other countries are unaware that they are following the example of El Salvador:
RECONSIDERING THE EDUCO PROGRAM AND THE INFLUENCE OF ITS…
131
Most development agencies now have learned about those models that happened, you know, more than 20 years ago, but, 20 years ago, actually, to go
from something that was a natural process [in El Salvador, in the absence of
governmental support] and then saying, ok, first we need to set up an organization to create a community body and let’s give some formal role to
parents and then they need to plan and create a school improvement plan
and they need to have committees where students come together with
teachers and parents and they execute those plans, and we need a mechanism to transfer resources from the central level to the school …, and then
how do we provide accountability of those resources—all those things …
20 years later are well known processes, not only in the Bank but in the
United Nation’s Children’s Fund, [the United States Agency for
International Development], the Japan International Cooperation Agency—
we know about those processes. Twenty years ago, they evolved from El
Salvador, and they have become now part of international knowledge that
some of our younger staff, the newer countries that are developing community participation programs don’t know that that evolved from El
Salvador. My staff worked with Afghanistan, you know, they don’t know.
The ministry of education didn’t know that what they call community participation schools and all these operational processes, that, you know, in
organizational management, in financial transfer, in planning, that it evolved
from this tiny nation. (INTACT34)
Fifth, by imbuing EDUCO with an aura of success, these studies have
created a foundation of legitimacy that allows the World Bank to call on
the former champions of the program for support. According to Edwards
(2018), one key functionary who worked with the EDUCO program for
many years early on mentioned that she would eventually “know all of
Latin America … and a great part of the world because of EDUCO”—
about 25 to 30 countries in all (p. 278). This functionary goes on to say,
“people wanted to know how it was possible that illiterate rural people
could manage, could hire, could fire teachers, how it was that the poor
could be interested in school. That is was people had a hard time understanding” (Edwards, 2018, p. 278). More recently, in 2011, this same
interviewee was sent by the World Bank to Afghanistan to discuss the
EDUCO model. For the World Bank, such Salvadoran education specialists have played a key role in EDUCO’s promotion. In the words of the
abovementioned interviewee, “I helped a lot with the community theme.
When people [(i.e., governments)] were scared to give money to the parents, I helped to make the school boards and transfer the funds to the
132
D. B. EDWARDS JR.
schools … in many places” (Edwards, 2018, p. 278). This former
Salvadoran education specialist served as reinforcement when officials in
other countries were reluctant to experiment with the EDUCO model.
Here again, actions such as being called on to defend and spread the program would not have been possible—or, arguably, would not have
occurred to the same extent—without the track record of success that had
been established through the studies discussed in this manuscript.
The sixth, final, and perhaps most direct international implication is
one that still has relevance: the evidence base discussed here has become a
body of literature that exists independently of the now-defunct EDUCO
program.5 (Again, see Edwards, 2018, Chap. 9, for extensive examples).
In terms of the World Bank, staff there continue to make use of the evidence base produced during the height of attention to EDUCO. In more
recent publications, as they review and write on findings related to decentralization in practice, they draw on EDUCO studies, despite their limitations, and in so doing signal what is, from their perspective, the enduringly
positive and strong example of EDUCO (see, e.g. Barrera-Osorio, Fasih,
Patrinos, & Santibañez, 2009; Bruns, Filmer, & Patrinos, 2011; Di
Gropello, 2006, 2007; Umansky & Vegas, 2007). As with the World
Bank, so too do other institutions continue to draw on the EDUCO studies reviewed here. Although this is evident from the list that can be found
in Edwards (2018, Chap. 9), included here are a few key examples (apart
from the World Development Report 2004, discussed above) that demonstrate that institutions as diverse as the Brookings Center, the Global
Partnership for Education, and the United Nations Education, Science,
and Culture Organization (UNESCO) have recently found the lessons
highlighted in EDUCO evaluations to be relevant.
• Brookings Institution, 2008: A report on improving development
effectiveness spotlights EDUCO and the fact that the school council
“hires, monitors, retains or dismisses teachers” (Hartman & Linn,
2008, p. 47).
• World Bank, 2009: A World Bank review of school-based management highlighted and classified the EDUCO program as one of the
stronger models because community members are responsible for
“hiring, firing and monitoring teachers” (Barrera-Osorio, Fasih,
Patrinos, & Santibañez, 2009, p. 7).
RECONSIDERING THE EDUCO PROGRAM AND THE INFLUENCE OF ITS…
133
• Global Partnership for Education, 2009: The Global Partnership for
Education (2009) focused on EDUCO as a “way to ensure that the
community contributes to improved educational outcomes” (p. 96).
• UNESCO, 2009: In the Global Monitoring Report, UNESCO
underscored that EDUCO is a model that increases client “voice,”
meaning that it entails mechanisms through which community members have recourse if teachers do not meet their expectations
(UNESCO, 2008).
• World Bank, 2011: In a review of evidence on accountability mechanisms in education, the World Bank again focused on the EDUCO
program as one of the “stronger” examples (Bruns, Filmer, &
Patrinos, 2011).
• World Bank, 2012: A typical example is found in a book written by
World Bank specialists about education reforms for sub-Saharan
Africa. They state: “the authority of community-managed EDUCO
schools to hire and fire teachers in El Salvador had a beneficial impact
on student outcomes compared with traditional schools serving similar populations” (Majgaard & Mingat, 2012, p. 149).
• Save the Children, 2013: In a report on community participation,
EDUCO is cited in relation to what mothers can do to enhance student outcomes (Save the Children, 2013).
It should be noted that the abovementioned institutions have highlighted EDUCO for a variety of reasons that do not have anything to do
with the original goals of the program. In looking at the evidence on
EDUCO, they have found—or have chosen to see—lessons related not
only to instituting community involvement in education (Global
Partnership for Education, 2009) but also to extending system coverage
rapidly (UNESCO, n.d.), achieving educational and gender equality
(Birdsall, Levine, & Ibrahim, 2005), providing education in post-conflict
contexts (Beleli et al., 2007; UNESCO, 2011), and ensuring the provision of education as a human right (UNESCO, 2007). In these ways, one
sees, on one hand, how the evidence base created by the World Bank continues to echo throughout the global education reform landscape and, on
the other hand, how various institutions have adapted the lessons of
EDUCO to their own purposes. In other words, the evidence on EDUCO
has become symbolic; these studies have a reputation for being rigorous,
or are often otherwise perceived by their readers as such, and so development professionals, by invoking them, lend credibility to the points they
134
D. B. EDWARDS JR.
wish to advance. Going forward, this will likely be one of the primary legacies of the EDUCO program internationally, especially as development
trends change, and as community-level decentralization becomes less popular, replaced by such hot topics as public-private partnerships and low-fee
private schools for the poor (Patrinos, Barrera-Osorio, & Guáqueta,
2009).
Notes
1. For additional detail and critique of EDUCO in practice at the community
level, see Edwards (forthcoming).
2. Indeed, EDUCO is concerning because of the burden that it can impose,
considering that the labor contributed by parents was equivalent to work of
805 full-time staff (an amount that equals 28% of the work done by the
administrative staff of the Ministry of Education) (Cuéllar-Marchelli, 2003).
3. Alec Gershberg served as a senior education economist for the World Bank
during the late 1990s.
4. These three ministers were Abigail de Perez (minister during 1998–1999,
formerly vice-minister and director of Planning for the MINED), Evelyn
Jacir de Lovo (Minister during 1999–2003, formerly director of the
Modernization Unit within the MINED, later working for the Organization
of American States as the director of the Departamento de Desarrollo Social
y Empleo), and Darlyn Meza (minister during 2004–2009, formerly vice-­
minister, director of Education, and coordinator of the EDUCO office).
Only Rolando Marin, who briefly served served as minister during 2004, did
not have a previous association with the EDUCO reform.
5. See Chap. 1 for more on EDUCO’s background. For more on EDUCO’s
undoing, see Edwards (2018) or Edwards, Martin, and Flores
(forthcoming).
References
ADES. (Asociación de Desarrollo Económico Social, Santa Marta). (2005). Una
sistematización de la educación popular en el cantón Santa Marta, Cabañas, El
Salvador, 1978–2001. [A systematization of popular education in the canton of
Santa Marta, Cabañas, El Salvador, 1978–2001.]. San Salvador, El Salvador:
ADES.
Aguilar, R., Becera, L., Burgos, J., Castro, J., & García, A. (1993). La educación
popular en Chalatenango: Un diagnóstico. [Popular education in Chalatenango:
A diagnostic.]. San Salvador: CIDEP.
RECONSIDERING THE EDUCO PROGRAM AND THE INFLUENCE OF ITS…
135
Alvear Galindo, V. (2002). La educación en Morazán, El Salvador, durante la guerra
civil de 1981 a 1992: ¿parte de una estrategia de supervivencia? [Education in
Morazán, El Salvador, during the civil war from 1981 to 1992: Part of a strategy
for survival?] Unpublished doctoral dissertation. Berlin: Berlin Free University.
Barrera-Osorio, F., Fasih, T., Patrinos, H., & Santibánez, L. (2009). Decentralized
decision-making in schools: The theory and evidence on school-based management.
Washington, DC: The World Bank.
Beleli, O., Chang, V., Fegelson, M., Kopel-Bailey, J., Maak, S., Mnookin, J.,
Nguyen, T., Salazar, M., Sinderbrand, J., & Tafoya, S. (2007). Education in
emergencies and early reconstruction: UNICEF interventions in Colombia,
Liberia, and Southern Sudan. Princeton, NJ: Woodrow Wilson School of Public
and International Affairs.
Birdsall, N., Levine, R., & Ibrahim, A. (2005). Toward universal primary education: Investments, incentives, and institutions. UN Millennium Project, Task
Force on Education and Gender Equality. London: Earthscan.
Bruns, B., Filmer, D., & Patrinos, H. A. (2011). Making schools work: New evidence on accountability Reforms. Washington, DC: The World Bank.
Cuéllar-Marchelli, H. (2003). Decentralization and privatization of education in
El Salvador: Assessing the experience. International Journal of Educational
Development, 23(2), 45–166.
Di Gropello, E. (2006). A Comparative analysis of school-based management in
Central America. World Bank Working Paper 72. Washington, DC: World
Bank.
Di Gropello, E. (2007). Education decentralisation and accountability relationships in Latin American and the Caribbean region. In T. Townsend (Ed.),
International handbook of school effectiveness and improvement (pp. 503–522).
Dordrecht: Springer.
Edwards, D. B., Jr. (forthcoming). Accountability through community-based
management? Implications from the local level implementation in El Salvador
of a globally-popular model. In K. A. Heidemann & R. A. Clothey (Eds.),
Another Way: Decentralization, Democratization, and the Global Politics of
Community-Based Schooling. Rotterdam: Sense.
Edwards, D. B., Jr. (2018). The trajectory of global education policy: Community-­
based management in El Salvador and the global reform agenda. New York:
Palgrave Macmillan.
Edwards, D. B., Jr., & Ávalos, E. (2015). Santa Marta y el programa EDUCO:
Una experiencia de resistencia, adaptación e inversión de la lógica neoliberal.
[Santa Marta and the EDUCO Program: An experience of resistance, adaptation and inversion of neoliberal reform logic.]. San Salvador, El Salvador:
Universidad Centroamericana.
Edwards, D. B., Jr., & DeMatthews, D. (2014). Historical trends in educational
decentralization in the United States and developing countries: A periodization
136
D. B. EDWARDS JR.
and comparison in the post-WWII context. Education Policy Analysis Archives,
22(40), 1–36. https://doi.org/10.14507/epaa.v22n40.2014.
Edwards, D. B., Jr., & Klees, S. (2012). Participation in international development
and education governance. In A. Verger, M. Novelli, & H. Kosar-Altinyelken
(Eds.), Global education policy and international development: New agendas,
issues and programmes (pp. 55–77). New York: Continuum.
Edwards, D. B., Jr., & Klees, S. (2015). Unpacking “participation” in development and education governance: A framework of perspectives and practices.
Prospects, 45(4), 483–499.
Edwards, D. B., Jr., Martin, P., & Flores, I. (forthcoming). Education in El
Salvador: Past, present, and prospects. In C. M. Posner, C. Martin, & Y. Martin
(Eds.), Education in México, Central America and the Latin Caribbean.
New York: Continuum.
Gershberg, A. I., Meade, B., & Andersson, S. (2009). Providing better education
services to the poor: Accountability and context in the case of Guatemalan
decentralization. International Journal of Educational Development, 29,
187–200.
Gillies, J. (Ed.). (2010). Education system reform and aid effectiveness: The power of
persistence. Washington, DC: USAID/Equip 2. Retrieved from http://www.
equip123.net/docs/E2-Power_of_Persistence.pdf
Global Partnership for Education. (2009). Six steps for abolishing primary school
fees: Operational guide. Washington, DC: World Bank.
Hartmann, A., & Linn, J. (2008). Scaling up: A framework and lessons for development effectiveness from literature and practice. Brookings Global Economy and
Development Working Paper Series. Retrieved from http://www.brookings.
edu/~/media/research/files/papers/2008/10/scaling-up-aid-linn/10_scaling_up_aid:linn.pdf
Jimenez, E., & Sawada, Y. (1999). Do community-managed schools work? An
evaluation of El Salvador’s EDUCO program. The World Bank Economic
Review, 13(3), 415–441.
Majgaard, K., & Mingat, A. (2012). Education in sub-Saharan Africa: A comparative analysis. Washington, DC: World Bank.
Meza, D., Guzmán, J., & de Varela, L. (2004, May 25–27). EDUCO: A
community-­managed education program in rural areas of El Salvador. Paper
presented at Scaling Up Poverty Reduction: A Global Learning Process and
Conference, Shanghai.
Montgomery, T. (1995). Revolution in El Salvador: From civil strife to civil peace
(2nd ed.). Boulder, CO: Westview.
Najjumba, I., Habyarimana, J., & Bunjo, C. (2013). Improving learning in
Uganda Vol. III: School-based management: Policy and functionality.
Washington, DC: World Bank.
RECONSIDERING THE EDUCO PROGRAM AND THE INFLUENCE OF ITS…
137
Patrinos, H., Barrera-Osorio, F., & Guáqueta, J. (2009). The role and impact of
public-private partnerships in education. Washington, DC: World Bank.
Reimers, F. (1997). The role of the community in expanding educational opportunities: The EDUCO schools in El Salvador. In J. Lynch, C. Modgil, &
S. Modgil (Eds.), Education and development: Tradition and innovation. Equity
and excellence in education for development (Vol. 2, pp. 146–162). London:
Cassell.
Save the Children. (2013). The right to learn: Community participation improving
learning. Westport, CN: Save the Children.
Sawada, Y. (2000). Community participation, teacher effort, and educational outcome: The case of El Salvador’s EDUCO program. Michigan: The William
Davidson Institute, Report No. 307, University of Michigan Business School.
Retrieved from https://deepblue.lib.umich.edu/handle/2027.42/39691
Sawada, Y., Kozuka, E., Aida, T., Griffen, D., Noguchi, H., & Todo, Y. (2014).
Do community-managed schools facilitate social capital accumulation?
Evidence from COGES project in Burkina Faso. Retrieved from http://
www.3ieimpact.org/media/filer_public/2014/09/18/do_community_managed_schools_facilities_social_capital_accumulation_sawada.pdf
Shibuya, K. (2013). Challenges in educational development in Africa and JICA’s
approach. In JICA, For inclusive and dynamic development in sub-Saharan
Africa (pp. 267–298). Tokyo: Japan International Cooperation Agency.
Umansky, I., & Vegas, E. (2007). Inside decentralization: How three Central
American school-based management reforms affect student learning through
teacher incentives. The World Bank Research Observer, 22(2), 197–215.
Umanzor, S., Soriano, I., Vega, M., Jimenez, E., Rawlings, L., & Steele, D.
(1997). El Salvador’s EDUCO program: A first report on parents’ participation
in school-based management. Working Paper Series on Impact Evaluation of
Education Reforms, No. 4. World Bank.
UNESCO. (n.d.). EDUCO: Educación con participación de la comunidad.
[EDUCO: Education with community participation.] Innovemos: Education
Innovations Network for Latin America and the Caribbean. Retrieved from
http://www.redinnovemos.org/content/view/189/92/lang,en/
UNESCO. (2007). Educational governance at the local level. Paris: Author.
UNESCO. (2008). EFA Global Monitoring Report 2009: Overcoming inequality:
Why governance matters. Oxford: Oxford University Press.
UNESCO. (2011). EFA Global Monitoring Report 2011: The hidden crisis: Armed
conflict and education. Paris: Author.
World Bank. (2003). World development report 2004: Making services work for poor
people. Washington, DC: Author.
CHAPTER 7
Impact Evaluations: Persistent Limitations,
Alternative Approaches, Possible Responses
Abstract This chapter addresses how the studies reviewed in this manuscript are generally problematic for two reasons that go beyond their specific findings. The first problem is the econometric nature of the studies,
while the second is the political-financial-intellectual complex from which
these studies were borne and back into which they went as they furthered
the interests of that complex. In addition to discussing these issues, this
chapter focuses on how the combination of these issues contributes to
relations of dependence between international researchers with expertise
in these methods and their counterparts in low- and middle-income countries. In response to this situation, alternative methodological approaches
are advocated. A number of suggestions are also made for addressing the
political-economic challenges that confront knowledge production in the
field of global education policy.
Keywords Knowledge production • Impact evaluation • Alternatives •
World Bank • Political economy • Global education policy • International
organization
After briefly reviewing what this book has tried to accomplish, the present
chapter addresses how the studies reviewed in this manuscript are generally problematic for two reasons that go beyond their specific findings.
© The Author(s) 2018
D. B. Edwards Jr., Global Education Policy, Impact Evaluations, and
Alternatives, https://doi.org/10.1007/978-3-319-75142-9_7
139
140
D. B. EDWARDS JR.
The first problem is the econometric nature of the studies, while the second is the political-financial-intellectual complex from which these studies
were borne and back into which they went as they furthered the interests
of that complex. In addition to discussing these issues, this chapter focuses
on how the combination of these issues contributes to relations of dependence between international researchers with expertise in these methods
and their counterparts in low- and middle-income countries. In response
to this situation, alternative methodological approaches are advocated. A
number of suggestions are also made for addressing the political-economic
challenges that confront knowledge production in the field of global education policy.
The Book in Retrospect
The present manuscript began, in Chap. 1, by presenting a political economy perspective on knowledge production and by situating that perspective in relation to the field of global education policy. Chapter 2 then
entailed an extensive discussion and critique of what many consider to be
the most credible form of policy-relevant knowledge—namely, impact
evaluations. Importantly, as the discussion in Chap. 2 indicates, critically
reviewing impact evaluations means that one takes into consideration their
technical, methodological, organizational, and political dimensions.
Chapter 3 subsequently defined and explained the methods that the study
in this book employed in order to place a critical review of impact evaluations within the political economy of global education reform—that is, in
order to understand not only the limitations of impact evaluations themselves but also the role that impact evaluations play and the influence they
can have across multiple contexts once they are produced.
The remaining chapters of the book turned to the empirical case of
focus from El Salvador. Chapter 4 examined the historical context within
which this case—that is, the Education with Community Participation
(EDUCO) program—emerged in order to explain the dynamics that surrounded the program from the outset. This was the necessary starting
point for analyzing this case because the impact evaluations reviewed here,
in addition to being produced by and for institutions and individuals with
vested interests, were also borne into a very receptive political-economic
context, that is, a context in which notions of decentralization were already
popular and in which the idea of community management of education
was received with excitement because it represented the next step in
IMPACT EVALUATIONS: PERSISTENT LIMITATIONS, ALTERNATIVE…
141
e­ xperimentation around transferring responsibilities away from the central
ministries of the state. Chapters 5 and 6 then engaged in a detailed reevaluation of the widely known and cited studies on EDUCO, followed by
a consideration of what—in light of this re-evaluation—can reasonably be
claimed to be known about the EDUCO program. Chapter 6 also contained a discussion of the various implications of these EDUCO studies
and the ways that they intersected with the political-economic context of
the 1990s and 2000s and contributed to the advancement of certain perspectives, political causes, organizational prerogatives, and personal
careers.
In this review, it has not only been demonstrated that the evidence used
to promote EDUCO is fundamentally flawed, but also that one cannot be
sure of any of the outcomes claimed by the studies reviewed here.
Moreover, as opposed to attributing the observed outcomes to the
EDUCO program, it was shown that there is reason to believe that the
outcomes could be the result of pre-existing social capital. Recall that the
EDUCO program was initially targeted to communities which were affiliated with the Farabundo Martí National Liberation Front (FMLN) and
which, as such, had experience with various forms of self-organization,
including the system of popular education schools that had developed
during the civil war.1 Given these experiences, it may be that the EDUCO
communities were well prepared to engage in the kind of community-level
management that the EDUCO program entailed. Put differently, there is
reason to believe that the EDUCO communities included in the studies
reviewed here had relatively more social capital than the communities surrounding the traditional rural public schools. To the extent that this is the
case, the control group is dissimilar and cannot be compared to treatment
group of EDUCO communities for the purpose of determining the impact
of the EDUCO program.
Persistent Limitation 1: The Nature
of Econometric Methods
Returning to the studies reviewed in this manuscript, they are generally
problematic for two reasons that go beyond their specific findings. The
first is the econometric nature of the studies, while the second is the
political-­
financial-intellectual complex from which these studies were
borne and back into which they went as they furthered the interests of that
142
D. B. EDWARDS JR.
complex. On the first point, one must note that econometric methods
(think of what in Chap. 2 was called regression analysis), while useful for
indicating the general nature of relationships or correlations among variables, cannot by their nature offer causal explanations. On one hand, this
is because data do not speak for themselves, and so researchers and practitioners must interpret the relationships among variables based on their
guiding frameworks, whether they be implicit or explicit (Kvernbekk,
2013). On the other hand, this is because econometric studies look at
inputs and outputs without being able to unpack the “black box” of how
a policy works (Pawson, 2006). That is to say, econometric methods are
limited in their ability to explain how and why each sub-component of a
program—such as EDUCO, for example—works or does not work in
practice. These methods restrict their focus to what goes in and what
comes out, thereby failing to account for both the inner workings of a
program as well as the many layers of context that impinge on how the
program operates in implementation (Pawson & Tilly, 1997). In the
words of Dumas and Anderson (2014), “Achieving high levels of rigor in
quantitative studies involves limitations that include social and cultural
decontextualization, ahistoricity, [and] the creation of randomization that
is seldom generalizable to real life settings” (p. 5). A fundamental point is
that, while quantitative data are considered ideal for empirically justifying
the implementation of education policies (that ultimately have and always
had political motivations), these data are not sufficient.
Alternative Approaches
It is for the above reasons that in-depth, thorough, and analytic qualitative
studies are necessary for answering policy-relevant questions and for providing policy-relevant findings. While Dumas and Anderson (2014) discuss a range of possibilities, a few approaches are highlighted here that
hold promise for informing policy. They are mentioned with the hope
that, going forward, they may spark interest among researchers as useful
ways to investigate complex educational issues. These approaches are vertical/horizontal case studies (Vavrus & Bartlett, 2006), process tracing
(Bennett & Elman, 2006), realist case study (Pawson, 2006), systems
thinking (Gillies 2010), ethnographic evaluation (Whitehead, 2002), participatory evaluation methods (Upshur, 1995; World Bank, 2011), and
narrative research (sometimes also known as life history research) (Lieblich,
Tuval-Mashiach, & Zilber, 1998; Moen, 2006). The common c­ haracteristic
IMPACT EVALUATIONS: PERSISTENT LIMITATIONS, ALTERNATIVE…
143
that these approaches share is that they complement and expand the findings that econometric analyses can offer, for example, by illuminating the
lived experiences and emic perspectives of stakeholders (narrative research,
participatory project assessment), by revealing which policy mechanisms
work in which contexts (realist evaluation), by indicating how policies are
implemented and affect the daily lives of people in practice (ethnographic
evaluation, participatory action research), by providing insight into how
the same policy can be interpreted and implemented differently across
multiple levels and multiple locations (vertical/horizontal case study), by
showing various sub-systems and various aspects of the education system
impact one another (systems thinking), and by illustrating how multiple
actors and multiple events influence one another in the process of policymaking and policy implementation (process tracing). As can be seen,
through these methods, researchers are frequently able to arrive at more
nuanced and more explanatory findings than would be the case through
the exclusive use of econometric methods.
Persistent Limitation 2: The Political-Financial-­
Intellectual Complex
Going back to the second issue mentioned above—that is, the political-­
financial-­intellectual complex that tends to produce impact evaluations
and the consultants who carry them out—the work of Samoff (1993) is
relevant. As a scholar who focuses on African education reform in the
context of international development efforts, and often under external
pressure, Samoff (1993) has observed conditions similar to those which
engulfed El Salvador in the late 1980s and 1990s. He characterizes the
dynamics of reform in such contexts in this way:
Having concluded that local decision makers have fundamentally mis-­
managed their responsibilities, the external agencies offer general and rapidly disbursed support in exchange for broad control. This broader form of
external assistance impels a demand for broader expert knowledge. The
expertise required goes well beyond, say, curriculum development and
teacher training. The relevant experts are those who can understand and
manage production, finance, international exchanges, and national planning
as well as social services. This call for broader expertise comes at a moment
of severe economic crisis, precisely the time when African countries are least
able to supply expertise and experts at the level and scale and with the credibility and legitimacy the external agencies demand. (Samoff, 1993, p. 186)
144
D. B. EDWARDS JR.
This description is reminiscent of the context and sequence of events
that the Salvadoran president, Alfredo Cristiani, experienced after being
elected in 1989. Not only was El Salvador subject to external pressure
from the United States Agency for International Development (USAID)
and the World Bank to enact certain reforms for both the economic and
social sectors, but Salvadoran entities such as Salvadoran Foundation for
Economic and Social Development (FUSADES), the Ministry of
Education (MINED), and the Ministry of Health also had to work with
international consultants and researchers from the United Nations
Education, Science, and Culture Organization (UNESCO) and the World
Bank, among other foreign institutions, who possessed the necessary clout
and technical capacity to credibly guide the reform and to carry out the
research that would be used for decision-making around the reform of
social services, such as education.
But Samoff (1993) makes an additional observation about the relationship between national and international actors that is key for our purposes.
In his words: “what is most powerful and most insidious in this relationship is the internalization within Africa of worldviews, research approaches,
and procedures for creating and validating knowledge that effectively perpetuate Africa’s dependence and poverty” (Samoff, 1993, p. 187). The
point here is that, over time, certain approaches to research—typically the
econometric and advanced statistical approaches utilized by the consultants—are seen as the only or the most legitimate methods of investigation, particularly for the purpose of policymaking (Samoff, 1996). The
implication in practice is that local researchers strive to replicate technically complex methods, in order for their research to achieve the same
legitimacy, though often researchers in middle- and low-income countries
are hindered by a lack of training in these methods and by a lack of funding to carry out costly studies. One consequence of this situation is that
local researchers continue to rely on international consultants and international organizations for both financial and technical resources, in addition
to relying on them for their reputation and status as capable knowledge
producers. A second consequence is that qualitative methods, such as
those mentioned above, are seen as less desirable and less appropriate
when it comes to their ability to inform education sector reform, despite
the fact that econometric methods, as discussed in Chap. 2, suffer from
numerous shortcomings themselves (Erickson & Gutierrez, 2002; Klees
& Edwards, 2014; Leamer, 1983; Samoff, 1991, 1996; Schroeder,
Sjoquist, & Stephan, 1986; Verschuren, 2001), though they are often
IMPACT EVALUATIONS: PERSISTENT LIMITATIONS, ALTERNATIVE…
145
unrecognized or purposefully overlooked, and despite the fact that the
relevance of qualitative research to policymaking has been thoroughly
defended (Dumas & Anderson, 2014).
Possible Responses
Both of the above-noted consequences have certainly played out—and
continue to play out—in El Salvador. Going forward, it will be necessary
to work in a number of different areas to address this situation, especially
to the extent that the goal is to ensure that there are well-trained Salvadoran
researchers who can work within and outside the Salvadoran government
to produce policy-relevant research from numerous theoretical perspectives and through various research methods. Based on this case from El
Salvador, it is also suggested that an additional goal should be to incorporate and to prioritize research that is conducted through the lens of social
justice, since, as shown in this manuscript, the reform process in El
Salvador actually served to further disadvantage those who were already
marginalized (i.e., parents and students in rural areas most impacted by
the civil war). A few broad, interrelated strategies to work toward these
ends include, first, reducing the power imbalance that has allowed external
actors and external studies to hold such influence; second, seeking alliances with domestic and international organizations who themselves prioritize principles of social justice in their research studies and in their
policy recommendations; third, developing research centers and the
research capacity necessary to complement (if not counter) official studies
produced by those with a vested interest in policy decisions; and, fourth,
connecting these centers and local researchers with international think
tanks, international scholars, transnational social movements, and networks of activists in order to provide resources and in order to raise the
profile of alternative worldviews generally and their prescriptions for
“development” as well as alternative perspectives on educational issues
specifically (Chan, 2007; Edwards & Klees, 2015).2 These strategies
would also help to counter, where necessary, the policy messages that are
being promoted around the world and in El Salvador by knowledge institutions, institutions which often bring a neoliberal perspective to their
policy work and rhetoric (Spring, 2015). The importance of these
­suggestions cannot be understated. Making progress in these areas would
drastically transform how education policies are made (and how, or
whether, they go global), particularly (but not exclusively) in low- and
146
D. B. EDWARDS JR.
middle-­income countries, which tend to be more susceptible to outside
influence, and particularly given the entrenched nature of neoliberal
frameworks used for understanding and for reshaping the provision of
social services globally (Ball, 2012; La Londe, Brewer, & Lubienski, 2015;
Lubienski, Brewer, & La Londe, 2016).
Importantly, as has been highlighted in this manuscript with regard to
the World Bank, these frameworks underlie the methods used and shape
the findings and recommendations offered. The implication is that there is
an ideological agenda within the World Bank that is not only protected
from research findings but which itself informs the findings produced
through ostensibly objective research. In the worst-case scenario, World
Bank researchers use empirical data to justify those policies that fit with the
organization’s preferences (Broad, 2006; Verger, 2012), though it should
be noted that the World Bank is not the exception. Recent studies have
shed light on what Carol Weiss (1995) has previously explained—that is,
when institutional agendas intersect with information, the result is that a
range of intermediary organizations within and outside the United States
(e.g., charities, think tanks, philanthropies, research firms, international
organizations) engage in the selective promotion of evidence and findings
in an attempt to influence education policy (Goldie, Linick, Jabbar, &
Lubienski, 2014; La Londe, Brewer, & Lubienski, 2015).
Conclusion
In the end, it is hoped that the present manuscript will help to take a step
forward to address the issues highlighted in this concluding section. As
has been argued, when it comes to research on education policy, new
methods and new political-institutional-financial arrangements are needed,
a point which the present manuscript has made clear by placing a critical
review of the EDUCO impact evaluations within the political economy of
global education reform. It is suggested that more such reviews are necessary in order to reveal the often-hidden interests and dynamics in and
through which policy-relevant knowledge is produced and subsequently
leveraged in diverse ways and in disparate locations in order to advance the
interests of certain actors or ideologies while retarding the realization of
others’ interests and with what implications.
IMPACT EVALUATIONS: PERSISTENT LIMITATIONS, ALTERNATIVE…
147
Notes
1. For more on the history of EDUCO and the politics of its emergence see,
in addition to Chap. 4 in this volume, Edwards (2018, forthcoming) and
Edwards and Ávalos (2015).
2. At the global level, one indicator of the timeliness of the suggestion to
increase the research capacity of southern knowledge agents is the fact that
it was proposed by actors involved in the process of setting the post-2015
goals (Quint & Lucas, 2014).
References
Ball, S. (2012). Global education, inc.: New policy networks and the neo-liberal
imaginary. New York: Routledge.
Bennett, A., & Elman, C. (2006). Qualitative research: Recent developments in
case study methods. Annual Review of Political Science, 9, 455–476.
Broad, R. (2006). Research, knowledge, and the art of “paradigm maintenance”:
The World Bank’s development economics Vice-Presidency (DEC). Review of
International Political Economy, 13(3), 387–419.
Chan, J. (2007). Between efficiency, capability, and recognition: Competing epistemes in global governance reforms. Comparative Education, 43(3), 359–376.
Dumas, M., & Anderson, G. (2014). Qualitative research as policy knowledge:
Framing policy problems and transforming education from the ground up.
Education Policy Analysis Archives, 22(11), 1–24. https://doi.org/10.14507/
epaa.v22n11.2014.
Edwards, D. B., Jr. (forthcoming). Accountability through community-based
management? Implications from the local level implementation in El Salvador
of a globally-popular model. In K. A. Heidemann & R. A. Clothey (Eds.),
Another way: Decentralization, democratization, and the global politics of
community-­based schooling. Rotterdam: Sense.
Edwards, D. B., Jr. (2018). The trajectory of global education policy: Community-­
based management in El Salvador and the global reform agenda. New York:
Palgrave Macmillan.
Edwards, D. B., Jr., & Ávalos, E. (2015). Santa Marta y el programa EDUCO:
Una experiencia de resistencia, adaptación e inversión de la lógica neoliberal.
[Santa Marta and the EDUCO Program: An experience of resistance, adaptation and inversion of neoliberal reform logic.] San Salvador, El Salvador:
Universidad Centroamericana.
Edwards, D., Jr., & Klees, S. (2015). Unpacking “participation” in development
and education governance: A framework of perspectives and practices. Prospects,
45(4), 483–499.
148
D. B. EDWARDS JR.
Erickson, F., & Gutierrez, K. (2002). Culture, rigor, and science in educational
research. Educational Researcher, 31(8), 21–24. https://doi.org/10.3102/00
13189X031008021.
Gillies, J. (Ed.) (2010). Education system reform and aid effectiveness: The power of
persistence. Washington, DC: USAID/Equip 2. Retrieved from http://www.
equip123.net/docs/E2-Power_of_Persistence.pdf
Goldie, D., Linick, M., Jabbar, H., & Lubienski, C. (2014). Using bibliometric
and social media analyses to explore the “echo chamber” hypothesis.
Educational Policy, 28(2), 281–305.
Klees, S., & Edwards, D. B., Jr. (2014). Knowledge production and technologies
of governance. In T. Fenwick, E. Mangez, & J. Ozga (Eds.), World yearbook of
education 2014: Governing knowledge: Comparison, knowledge-based technologies
and expertise in the regulation of education (pp. 31–43). New York: Routledge.
Kvernbekk, T. (2013). Evidence-based practice: On the function of evidence in
practical reasoning. Studier i Pedagogisk Filosofi, 2(2), 19–33.
La Londe, P., Brewer, T., & Lubienski, C. (2015). Teach for America and teach for
all: Creating and intermediary organization network for global education
reform. Education Policy Analysis Archives, 23(7), 1–25. http://epaa.asu.edu/
ojs/article/view/1829.
Leamer, E. (1983). Let’s take the con out of econometrics. American Economic
Review, 73(1), 31–41.
Lieblich, A., Tuval-Mashiach, R., & Zilber, T. (1998). Narrative research: Reading,
analysis and interpretation. Thousand Oaks, CA: Sage Publications.
Lubienski, C., Brewer, T., & La Londer, P. (2016). Orchestrating policy ideas:
Philanthropies and think tanks in the US education policy advocacy networks.
Australian Educational Researcher, 43(1), 55–73.
Moen, T. (2006). Reflections on the narrative research approach. International
Journal of Qualitative Methodology, 5(4), Article 5. Retrieved from https://
www.ualberta.ca/~iiqm/backissues/5_4/HTML/moen.htm
Pawson, R. (2006). Evidence-based policy: A realist perspective. London: Sage.
Pawson, R., & Tilley, N. (1997). Realistic evaluation. London: Sage.
Quint, R., & Lucas, S. (2014). The power and potential of southern think tanks for
the post-2015 agenda. Post2015.org: What comes after the MDGs? Retrieved from
http://post2015.org/2014/11/05/the-power-and-potential-of-southernthink-tanks-for-the-post-2015-agenda/
Samoff, J. (1991). The façade of precision in education data and statistics: A troubling example from Tanzania. The Journal of Modern African Studies, 29(4),
669–689.
Samoff, J. (1993). The reconstruction of schooling in Africa. Comparative
Education Review, 37(2), 181–222.
Samoff, J. (1996). Chaos and certainty in development. World Development, 24(4),
611–633.
IMPACT EVALUATIONS: PERSISTENT LIMITATIONS, ALTERNATIVE…
149
Schroeder, L., Sjoquist, D., & Stephan, P. (1986). Understanding regression analysis. London: SAGE.
Spring, J. (2015). Economization of education: Human capital, global corporations,
skills-based schooling. New York: Routledge.
Upshur, C. (1995). What is participatory evaluation (PE)? What are its roots?, The
Evaluation Exchange, 1(3/4). Retrieved from http://www.hfrp.org/evaluation/the-evaluation-exchange/issue-archive/participator y-evaluation/
what-is-participatory-evaluation-pe-what-are-its-roots
Vavrus, F., & Bartlett, L. (2006). Comparatively knowing: Making a case for the
vertical case study. Current Issues in Comparative Education, 8(2), 95–103.
Retrieved from http://devweb.tc.columbia.edu/i/a/document/25697_8_2_
Vavrus_Bartlett.pdf
Verger, A. (2012). Framing and selling global education policy: The promotion of
PPPs in education in low-income countries. Journal of Education Policy, 27(1),
109–130.
Verschuren, P. (2001). Holism versus reductionism in modern social science
research. Quality & Quantity, 35, 389–405. https://doi.org/10.1023/
A:1012242620544.
Weiss, C. (1995). The four “I’s” of school reform: How interests, ideology, information, and institution affect teachers and principals. Harvard Education
Review, 65(4), 571–592.
Whitehead, T. (2002). Introduction to ethnographic assessment and evaluation systems. Ethnographic Assessment & Evaluation Systems Working Paper #2.
Departments of Anthropology, University of Maryland. Retrieved from http://
www.cusag.umd.edu/documents/WorkingPapers/IntroEAES.pdf
World Bank. (2011). Participatory methods. Retrieved from http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTPOVERTY/EXTISPMA/0
,,contentMDK:20190347~menuPK:412148~pagePK:148956~piPK:216618
~theSitePK:384329,00.html
Appendix 1: Alphabetic list
and characterization of evaluative
EDUCO studies
Full citation
Ayalam, R. (2005). Consultoría: Evaluación del Programa de
Educación con Participación de la Comunidad-EDUCO.
[Consultancy: Evaluation of the Program of Education with
Community Participation-EDUCO.] San Salvador: MINED
Ávila de Parada, A., & Landaverde de Romero, M. (2007). La gestión
educativa y el desempeño de centros escolares de la modalidad educación
con participación de la comunidad (EDUCO) en El Salvador.
[Education management and the performance of schools in the Education
with Community Participation Program (EDUCO).] Unpublished
Thesis, Universidad Centroamericana, José Simeón Cañas
Briones, C., Escoto, A., Márquez, S., Orellana, V., & Ramos, C.
(2006). Diagnóstico censal de las áreas administrativa, financiera y
legal de las ACE del Programa EDUCO. [Census diagnostic of the
administrative, financial and legal areas of the ACE of the EDUCO
Program.] San Salvador: FLASCO
Cuéllar-Marchelli, H. (2001). Decentralization and privatization of
education in El Salvador: Assessing the experience. Occasional Paper
19. National Center for the Study of Privatization in Education.
New York: Teachers College, Columbia University
Cuéllar-Marchelli, H. (2003a). Decentralization and privatization of
education in El Salvador: Assessing the experience. International
Journal of Educational Development, 23(2), 45–166
Cuéllar-Marchelli, H. (2003b). The cost-effectiveness of EDUCO and
traditional public schools in rural El Salvador. Unpublished
dissertation, Teachers College, Columbia University, New York
Characterization
Institutional study
Unpublished
university thesis
Institutional study
Working paper
Academic literature
Unpublished
university thesis
(continued )
© The Author(s) 2018
D. B. Edwards Jr., Global Education Policy, Impact Evaluations, and
Alternatives, https://doi.org/10.1007/978-3-319-75142-9
151
152
APPENDIX 1: ALPHABETIC LIST AND CHARACTERIZATION OF EVALUATIVE...
(continued)
Di Gropello, E. (2005). Análisis comparativo de la gestión centrada
en la escuela en América Central. [Comparative analysis of
school-based management in Central America.] En Breve [In Brief],
72(junio), 1–4
Di Gropello, E. (2006). A Comparative analysis of school-based
management in Central America. World Bank Working Paper 72.
Washington, DC: World Bank
Di Gropello, E. (2007). Education decentralisation and
accountability relationships in Latin American and the Caribbean
region. In T. Townsend (Ed.), International handbook of school
effectiveness and improvement (pp. 503–522). Dordrecht,
Netherlands: Springer
Díaz, H. (1992). Evaluación del programa EDUCO: Periodo julio
1991—septiembre 1992. [Evaluation of the EDUCO program: Period
July 1991—September 1992.] San Salvador, El Salvador: MINED
Gillies, J., Crouch, L., & Flórez, A. (2010). Strategic review of the
EDUCO program. USAID. Retrieved from http://www.equip123.
net/docs/e2-EDUCO_Strategic_Review.pdf
Jimenez, E., & Sawada, Y. (1999). Do community-managed
schools work? An evaluation of El Salvador’s EDUCO program. The
World Bank Economic Review, 13(3), 415–441
Jimenez, E., & Sawada, Y. (2003). Does community management
help keep kids in schools? Evidence using panel data from El Salvador's
EDUCO program. Discussion Paper, Center for International
Research on the Japanese Economy. Tokyo, Japan: University of
Tokyo
Lindo Fuentes, H. (1998). Comunidad, participación y escuelas:
EDUCO en El Salvador. [Community, participation and schools:
EDUCO in El Salvador.] Washington, DC: World Bank
Meza, D., Guzmán, J., & de Varela, L. (2004a, May 25–27).
EDUCO: A community-managed education program in rural areas
of El Salvador. Paper presented at Scaling Up Poverty Reduction: A
Global Learning Process and Conference, Shanghai
Meza, D., Guzmán, J., & De Varela, L. (2004b). EDUCO: Un
programa de educación administrado por la comunidad en las zonas
rurales de El Salvador (1991–2003). [EDUCO: A program of
educational administration for the community in rural zones of El
Salvador (1991–2003).] En Breve [In Brief], 51, 1–4
Mora, J. (1992). Análisis comparativo de secciones de primer grado
EDUCO y no EDUCO. [Comparative analysis of EDUCO and
non-EDUCO first grade sections.] San Salvador, El Salvador:
MINED
Organizational
publication
Working paper for
research center
Academic literature
Institutional study
Institutional study
Academic literature
(and impact
evaluation)
Working paper for
research center
(and impact
evaluation)
Institutional study
Conference paper
Organizational
publication
Institutional study
(continued )
APPENDIX 1: ALPHABETIC LIST AND CHARACTERIZATION OF EVALUATIVE...
153
(continued)
Moore, A., & Florez, A. (2015, March 8–13). To centralize or
decentralize? Does school-based management improve student
learning? Lessons from EDUCO. Paper presented at the conference
of the Comparative and International Education Society,
Washington, DC
Parandekar, S. (2002). Democratic decentralization and people’s
participation: An examination of the EDUCO programme in El
Salvador. In L. Haagh & C. Helgo (Eds.), Social policy reform and
market governance in Latin America (pp. 165–179). New York:
Palgrave Macmillan
Parra, C. (1992). Testimoniales de una experiencia en Marcha.
[Testimonies of an experience in progress.] San Salvador, El Salvador:
MINED
Ramo, R. (2011). El nuevo espacio rural en El Salvador. Análisis de
un programa educativo de participación comunitaria. [The new
rural space in El Salvador. Analysis of an education program for
community participation.] Profesorado: Revista de Currículum y
Formación del Profesorado [Professoriate: Journal of Curriculum and
Teacher Training], 15(2), 93–107
Reimers, F. (1997b). The role of the community in expanding
educational opportunities: The EDUCO schools in El Salvador. In
J. Lynch, C. Modgil, & S. Modgil (Eds.), Education and
development: Tradition and innovation. Equity and excellence in
education for development (Vol. 2, pp. 146–162). London: Cassell
Rodríguez, J. (2003). Incentivos a escuelas y maestros: La experiencia
del “plan de estímulos a la labor educativa institucional” en El
Salvador. [Incentives for schools and teachers: The experience of the
“stimulus plan for institutional education labor” in El Salvador.]
Magister en Gestión y Políticas Públicas, Estudio de Caso 74.
Universidad de Chile, Santiago, Chile
Sastry, N., Melamid, E., & Ross, R. (1995). A cost analysis of
decentralized basic education in El Salvador (No. DRU-1112-WB).
California: RAND Corporation
Sawada, Y. (2000). Community participation, teacher effort, and
educational outcome: The case of El Salvador’s EDUCO program.
Michigan, USA: The William Davidson Institute, University of
Michigan Business School
Sawada, Y., & Ragatz, A. (2005). Decentralization of education,
teacher behavior, and outcomes: The case of El Salvador’s EDUCO
program. In E. Vegas (Ed.), Incentive to improve teaching: Lessons
from Latin America (pp. 255–306). Washington, DC: The
International Bank for Reconstruction and Development/The
World Bank
Conference paper
Academic literature
Institutional study
Academic literature
Academic literature
University thesis
Institutional study
Institutional study
(and impact
evaluation)
Academic literature
(and impact
evaluation)
(continued )
154
APPENDIX 1: ALPHABETIC LIST AND CHARACTERIZATION OF EVALUATIVE...
(continued)
SIMEDUCO. (2011). Ya no estamos dormidos. [We are no longer
asleep.] San Salvador: Instituto de Derechos Humanos de la
Universidad “José Simeón Cañas”
Srygley, M. (2013). Education in rural El Salvador: A case study on
value, quality, and accountability. Unpublished thesis. University of
Maryland, College Park
Umansky, I., & Vegas, E. (2007). Inside decentralization: How
three Central American school-based management reforms affect
student learning through teacher incentives. The World Bank
Research Observer, 22(2), 197–215
Umanzor, S., Soriano, I., Vega, M., Jimenez, E., Rawlings, L., &
Steele, D. (1997). El Salvador’s EDUCO program: A first report on
parents’ participation in school-based management. Working Paper
Series on Impact Evaluation of Education Reforms, No. 4. World
Bank
World Bank. (1994). El Salvador: Community education strategy:
Decentralized school management. Washington, DC: The World
Bank
World Bank. (2006). El Salvador: Rural schools triple enrollment
and deliver higher quality, results in action. En Breve, Issue 1.
Washington, DC: World Bank
World Bank. (2009a). Análisis institucional: EDUCO, una historia
de éxitos y resultados. [Institutional analysis: EDUCO, a history of
success and results.] Washington, DC: World Bank
World Bank. (2009b). Strengthening accountability in social service
delivery in Central America: The EDUCO school-based management
model. Washington, DC: World Bank
Institutional study
Unpublished
university thesis
Academic literature
Working paper (and
impact evaluation)
Institutional study
(and impact
evaluation)
Organizational
publication
Institutional study
Institutional study
Index1
A
ARENA, see Republican National
Alliance
B
Business Fund for Educational
Development (FEPADE), 79, 91
C
Chicago Boys, 91
Christian Democrat Party, 91
Cold War, 90
Cristiani, Alfredo, 91–93, 144
Critical review, 10, 12, 13, 16, 59,
69–85, 102–117, 124, 126, 127,
140, 146
D
Data mining, 55–57, 59
Death squad, 90
1
Decentralization, 77, 95, 102, 112,
114–117, 117n1, 124, 129, 132,
134, 140
Democratization, 91, 95, 96, 125
Duarte, José Napoleón, 91, 92
E
Education For All, 95
Endogeneity, 59n2, 60n6, 111, 112,
114, 118n6, 119n7
Ethnographic evaluation, 142,
143
External validity, 36, 39, 53, 61n13
F
FEPADE, see Business Fund for
Educational Development
Freire, Paulo, 94, 126
FUSADES, see Salvadoran Foundation
for Economic and Social
Development
Note: Page numbers followed by ‘n’ refer to notes.
© The Author(s) 2018
D. B. Edwards Jr., Global Education Policy, Impact Evaluations, and
Alternatives, https://doi.org/10.1007/978-3-319-75142-9
155
156
INDEX
G
Generalizability, 36–41, 47, 58,
142
German Technical Cooperation,
78
Global Development Network, 6
Global education policy field, 2–4, 16,
16n2
H
Heckman, James, 110, 114–116,
118–119n6
Hypothesis testing, 42–45, 58
I
Instrumental variable, 29, 60n6, 112,
119n7
Internal validity, 36, 39, 40, 60n6,
60n10, 61n11
L
Life history, 142
M
Modernization, 78, 92, 93, 97n3,
134n4
N
Narrative research, 142, 143
Non-governmental organizations
(NGOs), 3, 6, 11, 41, 53, 55, 79
Normal curve, 46–50, 58
Normal distribution, 48, 49
P
Participatory evaluation (PE), 142
Peace Accords, 92, 96
Philanthropic foundations, 3
Policy implementation, 38, 78, 143
Political-financial-intellectual complex,
140, 141, 143–145
Popular education, 94, 125–128,
141
Power calculation, 45
Process tracing, 142, 143
P value, 42, 45–50, 56–58
Q
Qualitative methods, 40, 41, 58,
144
Qualitative studies, 40, 142
R
Reagan, 91, 92
Realist case study, 61n15, 142
Republican National Alliance
(ARENA), 91, 97, 130
S
Salvadoran Foundation for Economic
and Social Development
(FUSADES), 79, 91, 92,
144
Sample size, 30, 42–45, 58, 61n19
Sampling error, 42–44, 48
Save the Children, 133
Selection bias, 30, 31, 108, 110, 111,
114, 118–119n6
Standard deviation unit (SDU), 50, 51
INDEX
157
System Approach for Better Education
Results (SABER), 6
Systems thinking, 142, 143
V
Vertical/horizontal case study, 142,
143
T
Think tanks, 3, 11, 13, 79, 91, 145, 146
T-statistic, 42–44, 46
W
World Development Report (WDR),
13, 78, 129, 132