Democracy Surveys as a Tool
in Impact Evaluations
Promise and Challenges
Dr. Margaret Sarles
Margaretsarles@gmail.com
US Agency for International Development
Cairo, March 31, 2009
Presentation objectives:
• Review the challenges of measuring
impact of programs in democratic
development
• Prospects and cautions of using
national democracy surveys as an
impact evaluation tool
• Dialogue: How can we make them more
useful?
Why is this an important topic?
• A growing consensus that democracy is a component
of development, either as a core objective or to
further economic development
• Foreign policy priority for many countries: human
rights, ROL, free and fair elections, democratic
states, democratic cultures: policy imperatives and
concern with trends
• High investment levels but little scrutiny of
effectiveness
• Evaluations of democracy investments provide the
evidence of what works, what doesn’t, under what
circumstances: USAID commitment.
Measuring Impact on democratic
change is difficult
• The general concepts are contested and vague.
What is success? At what level? (overall,
components, programs)
• Extremely diverse programs are grouped as
“democracy”
– community policing, strengthening civil society, elections,
parties, justice systems, human rights, transparency, anticorruption, decentralization
– anything that could be said to be part of the political system
or that contributes to democratic change (in any program)
• “Politicized” definitions by stakeholders
• *Practice outruns theory and academic analysis on
processes, sequencing and causal links of
democratic change variables (relevance)*
But the time is right to begin impact
evaluations in democratic development
• Volume of academic work is exploding. Case
studies, quantitative work  hypotheses
• Renewed donor emphasis on evaluation and
evidence (including this conference)
• USAID: (a) 20+ years’ experience  testable
hypotheses, and (b) National Academy of
Sciences recommendations and other studies
• In addition to burgeoning hypotheses, more
and better data are available, particularly
democracy surveys.
Regional surveys are expanding
• World Values surveys: 1981; 4 waves; 80+
countries; broad sociocultural scope
• Afrobarometer: 1993; 4 rounds; 18 countries
• Arab Barometer: 2005 “to produce scientifically
reliable data on the politically relevant attributes of
ordinary citizens….to contribute to political reform….”
1 round (2007), 5 countries.
• Asian Barometer: 2001; 13 E. Asian and 5 S.
Asian countries; 2 rounds
• Latinobarometro: 1995, 18 countries, annual,
views on economics, trade, politics
• AmericasBarometer; 1992 (1970s) 23 countries;
up to 10 rounds
• Plus many independent country-level surveys
USAID has been the dominant donor
supporting AmericasBarometer first as a
policy tool (NOT evaluation)
• Began with policy: The Peace Accords in Guatemala:
The DIMS (now 6 rounds): what do citizens think
about and expect of democracy?
– Diagnostic tool for governments (legislatures, executive) and
donors
– Mobilizing tool/data for reform
– Increased interest in democracy reform in society (media,
NGOs; especially comparative)
– “Voice” of the people in imperfect democracies
– Reports are geared towards a policy audience
• This becomes a winning scenario for
increasing stakeholders and keeping support
But they did also provide excellent
evaluation measures-- due to
methodology
- Time series data
– Transparency and high standards (sample
frame, pre-tests, etc.)
– Inclusiveness (6 languages in Guatemala)
– Allows multivariate analysis, and easy use for
both qualitative and quantitative analysis
– Learning from other surveys: concepts,
definitions
– Training of USAID democracy officers
And are good evaluation tools
because of substance
• Survey questions include (1) behavior (e.g.,voting,
corruption, participation; (2) political culture
(e.g.,system support); (3) knowledge; (4) attitudes
(e.g.,trust); (5) personal attributes
• USAID Democracy officers participated in
questionnaire development, for relevance (now true
for other donors)-- “Black,” “gray,” and “white”
questions (ex:DLG).
• Result: Used in Bolivia in over half the performance
indicators of program success (ex.)
Improvements now make surveys
useful for impact evaluations
• Introduction of over-sampling in
geographically based programs: 2004
• Allowed study of counterfactual for the first time
• Richer comparisons (against national and other country
data)
• Therefore had before/during/after, and an approximation
of with/without
• Evidence that randomization may not
decrease chances of project success (rarely
implemented; now being investigated )
However, they require
adaptation to be used in impact
evaluation
• Biggest requirement, and least understood,
is over-sampling
• Randomization is widely resisted
• Questions (and indices) usually need to be
added to be relevant to a particular
intervention (often difficult)
• Counterfactual methodologies need to be (1)
developed and (2) costed out
• Long-term commitment should be as
guaranteed as possible, understanding that
Examples of Impact Evaluations
Using National Surveys
• Bolivia Democratic Development and Citizen Participation
Project (1996 - 2003)
– Over-sampled treated municipalities; compared to non-treated
municipalities, to national sample, and to other countries.
– Questions focused on participation and satisfaction
• Kenya Civic Education Project
– Innovative use of “non-treated” population through matching
– Used surveys to determine beliefs and levels/kinds of participation
Both evaluations used multiple methods, even though dominated by
survey results
Nonetheless, there are significant
challenges to using survey research
in impact evaluations(1)
• Conceptual: Linking questions to (1) theory;
(2) processes of democratic change
• Methodological: ensuring basic standards
are met for design and implementation.
– Need for transparency
– Post-evaluation analysis is possible but late (Exs:
Paraguay; LAPOP)
– New methods help get it right in the first place
(PDAs, GPS, institution-building).
national and trans-national design and methods
What are the challenges of using
survey research in impact
evaluations? (2)
• Administrative: funding over project life
(Guatemala); complex planning and
budgeting for multi-country surveys;
developing stakeholders in each country;
• Maximizing use of them to justify costs, while
recognizing surveys are usually not sufficient
for measuring impact on democratic change
What should be done to improve
surveys for impact evaluations?
• Learn from the health model: periodic,
institutionalized, health surveys with committed
funding, widespread acceptance
• Support research on democracy indicators derived
from survey questions and indices (gap analysis,
basic research)
• Educate developmentalists on over-sampling, and
other appropriate survey techniques (randomization,
matching, etc.)
• Institutionalize funding/administration of surveys,
against the present dispersed, decentralized, often
competing regional surveys
References
•
•
•
•
•
•
•
AmericasBarometer.com (All country and regional reports and data from Latin American surveys
undertaken by Mitchell Seligson, Vanderbilt University)
Bollen, Ken et. Al. “Assessing International Evaluations: An Example from USAID’s Democracy and
Governance Programs,” American Journal of Evaluation, 26: 189-203, 2005
Chemonics International Inc. “Bolivia Democratic Development and Citizen Participation Final Report, 1962003,” November 2003
Finkel, Steven E. “The Impact of the Kenya National Civic Education Programme on Democratic Atttitudes,
Knowledge, Values, and Behavior. “ Report prepared for US Agency for International Development, Nairobi,
Kenya (AEP-I-00-00-00018, Task Order No. 806). December 30, 2003.
_________. “Kenya National Civic Education Impact Study: Report on Pre-Test, Prepared for US Agency
for International Development, “Nairobi (Management Systems International and U. of Virginia) August 13,
2002.
National Research Council of the National Academies, Improving Democracy Assistance: Building
Knowledge Through Evaluations and Research. National Academies Press, Washington, DC 2008
(available download on www.nap.edu)
Sarles, Margaret J. “Evaluating the Impact and Effectiveness of USAID’s Democracy and Governance
Programmes,” in Peter Burnell, Evaluating Democracy Support: Methods and Experiences (IDEA and
SIDA), 2007.