Proposal Abstract
Measuring Democracy: A Multidimensional, Tiered, and Historical Approach
Principal Investigators
Michael Coppedge (University of Notre Dame), John Gerring (Boston University),
Staffan Lindberg (University of Florida), Jan Teorell (Lund University, Sweden)
Project Managers
Michael Bernhard (University of Florida), Steven Fish (University of California,
Berkeley), Allen Hicken (University of Michigan), Kelly McMann (Case Western
Reserve University), Pamela Paxton (Ohio State), Carsten Schneider (Central European
University, Budapest), Holli Semetko (Emory University), Svend-Erik Skaaning (Aarhus
University, Denmark), Jeffrey Staton (Emory University)
Academic researchers, policy practitioners, and the general public alike have
come to depend on indicators of democracy such as the Freedom House and Polity
indexes. Such indicators are used to trace waves of democracy, educate students about
world politics, test theories about the causes and consequences of democracy, allocate
foreign aid, assess efforts to promote democracy, and withhold or grant legitimacy to
regimes and governments around the world. Because democracy indicators guide such
momentous decisions, it is crucial that they measure democracy accurately.
It is increasingly clear that existing indicators are not adequate for these purposes.
Ideally, indicators of democracy would reflect the varied meanings of democracy in
different parts of the world, measure each aspect of democracy precisely and without
bias, and pass standard tests of validity and reliability. It is also essential that they cover
almost all countries over a long span of time because it is only in a broad, long-term
perspective that fundamental patterns and trends in democratization can be perceived and
complex causal relationships can be sorted out.
Although there are now dozens of indicators of democracy, only a handful cover
most countries for many years. They are a) the Polity index, which covers all sizable
independent countries from 1800 (or their date of independence) to the present; b) the
Przeworski-Alvarez-Cheibub-Limongi (PACL) democracy/dictatorship dichotomy,
which covers 135 countries from 1950 to 1990 (and now updated to 2002 by Cheibub and
Gandhi); and c) Freedom House’s ratings of Political Rights and Civil Liberties, which
cover all countries and some dependent territories from 1972 to the present.
All of the indicators with extensive coverage suffer from multiple weaknesses.
The PACL dichotomy, which shows only whether or not “governing parties lose
elections,” is insensitive to all but the most dramatic regime changes or differences of
degree. Although Freedom House’s detailed checklists seem to take a wide variety of
information into account, its opaque process of translating raw information into final
scores makes it difficult to determine exactly what it measures. Rigorous analysis
suggests that it produces two separate indicators of what turns out to a single concept—
contestation (Coppedge et al. 2008). Polity focuses on the formal institutional aspects of
democracy. Both Polity and Freedom House do a good job of distinguishing the most
democratic polities from the least democratic ones, but neither makes much more precise
distinctions reliably. Polity, for example, cannot reliably say that the United States is
2
more democratic than the next 50 countries ranked below it (Pemstein et al. 2008; Treier
and Jackman 2008). The fact that most of these indicators are strongly intercorrelated is
not necessarily reassuring because agreement at the most and least democratic extremes
produces high correlations even when there is little correlation among scores in the
intermediate range, and because a shared bias can also produce high correlations. All
three indicators – and in fact most of the indicators with less extensive coverage as well –
measure a primarily liberal electoral concept of democracy that ignores participatory and
social dimensions of democracy that are often considered important outside the United
States.
We propose to measure democracy better in three respects. First, we would
radically disaggregate, i.e., create separate indicators of democracy’s dozens of
components and subcomponents. Much of the measurement error found in existing
indicators comes from their attempt to score countries on variables that are complex and
probably multidimensional, such as “freedom of expression” (Freedom House) or
“constraints on the executive” (Polity). No matter how detailed the coding criteria are,
measuring such concepts with a single variable inevitably leads to imprecision or false
precision because the defining attributes of the concept do not align according to a single
simple rank ordering. Our approach is to measure each of the more specific attributes
separately. This would enable us to measure them more precisely and reliably because we
would be working with more concrete and less subjective phenomena. For example,
instead of creating a single score for “constraints on the executive,” we would score
separately whether the executive is selected by the legislature; whether the legislature has
the power of the purse and other powers; whether the executive’s party controls a
legislative majority; whether the executive by-passes the legislature with decrees and
referendums; whether the executive appoints the supreme court; whether the executive
accepts judicial review; the extent to which the executive manipulates the court through
budget constraints, impeachments, or “packing” appointments; and so on. Each one of
these attributes can be rated more reliably than vague “constraints on the executive.”
Second, we would measure democracy more comprehensively by creating
indicators of neglected aspects of liberal and electoral democracy and of the participatory
and egalitarian dimensions of democracy, which most existing indicators tend to ignore.
Thus, we would construct indicators in all of the following thirteen categories:
sovereignty, voting, elections, descriptive representation, the executive, the legislature,
the judiciary, political parties, the media, other civil society organizations, local
government, civil liberty, and social equality.
Third, we would measure each of these categories extensively, to cover all
countries (and some dependent territories) from 1900 to the present whenever possible.
Some indicators would cover this entire range while others would cover only more recent
years, but there would be full coverage within each category by at least one indicator.
Once data are collected, we would aggregate selected variables into indicators of various
models of democracy—electoral, liberal, participatory, and egalitarian—and would make
it possible for other scholars or agencies to select and combine our variables into
indicators of their own preferred versions of democracy or components of democracy.
Our indicators would include estimates of reliability.
The Principal Investigators (PIs) would work with a dozen Project Managers who
are full-time faculty at universities in the U.S., Sweden, Denmark, and Hungary. Each
3
Project Manager would be in charge of a category of indicators and would finalize the
operational procedures for the indicators in that category in consultation with the PIs.
Some of the indicators can be produced by the Project Managers themselves or by teams
of coders they recruit, but most would require coding by country experts, who would be
paid to gather the necessary information and submit it via the Internet. For many
indicators we plan to have teams of three expert coders per country. Some of the Project
Managers would also serve as area coordinators who help identify, recruit, and oversee
country experts. The Helen Kellogg Institute for International Studies at the University of
Notre Dame would be the central administrative office for the project. It would require a
full-time staff person to work with Michael Coppedge to develop the data collection
website, coordinate with the Project Managers and other PIs, standardize and distribute
the questionnaires, compile and clean the submitted data, publicize the findings, and plan
three international conferences. The first would be an initial organizational meeting of
Project Managers with an advisory board. The second would be a midterm follow-up to
head off any problems that arise after the project is in the field. The third would be a
public conference presenting analyses of the newly collected data.
We anticipate that a project of this magnitude would require funding from more
than one source. In addition to the usual applications to the National Science Foundation
and our own universities, we plan to propose this project to USAID and several European
development and democracy-promoting agencies, preferably through the OECD’s
Development Assistance Committee (DAC).
A successful project would bring several major public benefits. First, it would
give governments and NGOs more precise, valid, and reliable indicators of democracy,
which they can use to evaluate the impact of their own democracy promotion efforts and
decide where to allocate future funding. Second, it would enable researchers to do more
rigorous research on the causes and consequences of democracy. Third, it would open up
a new area of research on causal relationships among components of democracy, which
would also be useful for democracy promotion. Fourth, it would apprentice hundreds of
researchers all over the world in rigorous measurement technology. Finally, it would
increase U.S. scholarly recognition of the broader criteria for democracy that are
considered relevant in other parts of the world.
References
Coppedge, Michael, Angel Alvarez, and Claudia Maldonado. 2008. "Two Persistent
Dimensions of Democracy: Contestation and Inclusiveness." Journal of Politics
70 (3):632-47.
Pemstein, Daniel, Stephen Meserve, and James Melton. 2008. "Democratic Compromise:
A Latent Variable Analysis of Ten Measures of Regime Type." In Social Science
Research Network.
Treier, Shawn, and Simon Jackman. 2008. "Democracy as a Latent Variable." American
Journal of Political Science 52 (1):201-17.