COMPARATIVE POLITICS
THE COMPARATIVE MANIFESTO
PROJECT
Giorgio Sirtori
Prof. Marco Giuliani
CMP’S OBJECTIVE
To measure policy positions of all relevant parties
competing in any democratic election in post WWII
period for OECD and EU members, Central and
Eastern Europe and (in the future) Latin America
and South-East Asia
MANIFESTO (ELECTION PROGRAM)
→ the text published by a political party in order to
compete for votes in national election
Why manifesto?
 Parties’
only authoritative policy statements
 Indicators of the parties’ policy preferences at a
given point in time
TWO-STEP PROCESS
1. Unitising − cutting text in quasi-sentence
The coding unit is a quasi-sentence, that
contains exactly one statement
e.g.: ″We need to address our close ties with our
neighbours (107) / as well as the unique challenges
facing small business owner in this of economic
hardship (402)″
.
2. Coding − find the right code for a quasi- sentence
Attribute to each coding-unit one, and only one, category
CMP developed a category system composed of 56
categories, grouped in 7 policy areas, designed to be
comparable between parties, countries, elections and
across time
.
Manifesto data can be used, and have been used, to
provide valid and reliable measurements of party
policy position
e.g.: Left-right scale estimates from Mapping policy
preferences (Budge et al. 2001)
CMP LEFT-RIGHT SCALE
CMP LEFT-RIGHT MEASURE
Left-right sore = proportion(right - left) %
(R – L)/(R + L) %
Scale ranges from -100 to +100
CMP left-right scale positions for British parties
(1945-1997)
COMPARATIVE POLITICS
SHORTCOMINGS OF LOCATING POLITICAL
PARTIES USING PARTY MANIFESTO DATA
Irina Dámátár
Prof. Marco Giuliani
ESTIMATING LEFT-RIGHT POSITIONS OF POLITICAL
PARTIES

The aim of the Comparative Manifestos Project is to place
parties on the left-right political spectrum, parting from the
information provided within their electoral programmes

Each party’s position is measured as the difference (in
percentages) between the right-associated text mentions and
left-associated ones; the final scale ranges from -100 to 100

The identity of left and right manifesto coding categories was
determined using a series of within-country exploratory factor
analysis of a wide range of coding categories
THE COMPONENTS OF THE CMP LEFT-RIGHT
SCALE
SHORTCOMINGS

No indication of the uncertainty associated with any CMP
estimate
- when placing a policy in a certain point on the left-right dimension,
no associated error is indicated

CMP left-right measure consists of pre-defined and fixed
scale components
- the constituent elements of the left-right scale are defined in the
same way for all countries, at all time periods
but the meaning of left and right can vary in time and space
(e.g.: market regulation is a left issue in the CMP dataset, while in
Germany is a valance issue)
- CMP measure may not include relevant variables, that can explain
much of the parties’ variation, while considering unrelated ones

The CMP scale combines party positions with party specific
measures of the relative salience of the left-right scale,
producing a measure that is affected by non-right-left issues
mentioned in the party manifesto
- positions of the parties are in part determined by coding categories
that are not in the CMP scale

The scores which CMP interpret as positional data may
indicate in fact the willingness of a party to move in
accordance to changing political circumstances rather than
their actual position
- hence, the measurement made by CMP and the score attribute to a
certain party lacks precision
.

CMP approach identifies position on the basis of saliency
scores; but this can be seen more as a movement of position
towards the electorate rather than a party’s true position
- to eliminate this shortcoming, Franzmann & Kaiser stress
upon the distinction between
position issues
matters upon which the electorate’s
opinions are clearly divided
valence issues
matters evaluated either negatively
or positively by all voters
- only position issues can structure the policy space
-hence, they classify items based on their position or valance
character
.

Using factor analysis to compute party positions leaded
to highly difficult interpretations when the number of
variables out weighted the number of cases (low measure of
sampling adequacy)

Addressing this shortcoming by focusing on the economic
“super-issue” as equivalent to the left-right scale disregards
the fact that left and right are not perceived only in economic
terms

Nevertheless, in its second stage CMP followed a two step
approach: issues were coded as left or right, and the resulting
composition was examined through factor analysis -> to see
whether these issues did indeed fit in the assumed dimension
- factor analysis is also problematic because aims at a
reduction of dimensions
EXPLAINING DIFFERENCES BETWEEN CMP SCALE
POSITIONS AND EXPERT SURVEYS JUDGEMENTS

Computing the position of a party using an expert survey
estimate may lead to different results than the ones of the CMP

Why? Although there is no pattern to explain these differences,
there are a number of hypothesis that can be considered:

I-st possibility: party policies change between the time of the
election for which CMP scored a party manifesto, and the time
when the expert survey was conducted

II-nd possibility: one of the two measurements contains
significant error
III-rd possibility: measuring different quantities
- party manifestos may have policy preferences not
expressed in their election programmes and which hence cannot
be captured in the scores provides by CMP
- no single pre-defined scale will accurately characterize
the left right dimension of politics in all countries
- “new” dimensions of politics, such as immigration
and environment, whose omission from the measurement can
cause inaccuracies
(e.g.: until 1980s, in Germany, environmental protection was a
valance issue or at least was not ideologically structuring the
policy space)

PLEASE NOTE THAT NO SHEEP WERE HARMED IN
THE MAKING OF THIS PRESENTATION

COMPARATIVE POLITICS
EXPERT SURVEYS
PARTY POLICY IN MODERNDEMOCRACIES
Kenneth Benoit
Michael Laver
Silvia Merisio
Prof. Marco Giuliani
ANALYSING PARTY COMPETITION

Why voters choose to support specific political parties?

Why parties do what they do once the election is over?
Public needs
and wants
Policy adopted
by political
parties
NEED TO ESTIMATE PARTY POLICY POSITIONS
 Problem
of gathering reliable information
Election Manifestos
Not systematic
analysis
How parties behave
(e. g. Roll call votes)
Surveying party
politicians
or party supporters
Extremely
expensive and
difficult across
countries
Expert surveys
EXPERT SURVEYS

“Expert”
most comprehensive population of
country specialists

These experts are asked to locate party policy positions, in
the party systems of which they have expert knowledge

On a set of predefined policy dimensions
POLICY DIMENSIONS
Economic Policy
 Social Policy
 European integration
 General policy (environment, immigration, etc.)
 Specific dimensions for Post-communist countries (e.g.
privatization, religion, etc.)
 Subjective sympathies/closeness to party
 General Left-Right dimension

All dimensions are evaluated on a scale ranging from 1 to 20
EXPERT SURVEYS
They provide information on party positions
Common format
Wide range of
countries
Expert surveys are a systematic way to
summarize the judgments of the consensus of
experts on the matters at issue
RISKS
The same concept/
dimension can be
interpreted in different
ways
How to assess the
content of validity
Importance of broadly
equivalent dimensions
e.g. economic policy
Reasonableness of
numbers and comparison
of the result with other
method
THE SURVEY
Reference to a previous work by Laver and Hunt (1992)
Survey conducted in 1988-89
 24 countries
 Core dimensions on economic and social policies

Benoit and Laver (2006)
Survey conducted in 2003
 47 countries (all European countries from the former Soviet Bloc)
 38 policy dimensions (including specific dimensions for each country)

Real continuity between the two works
THE SURVEY (2)
Inclusion of countries classified as non-fully democracies
by Freedom House;
e.g. Belarus, Russia, Albania, Moldova, Ukraine
 Translation in 24 languages;
 Web or paper format.

SURVEY DETAILS FROM EASTERN EUROPE COUNTRIES
POLICY DIMENSIONS FROM THE EXPERT SURVEYS (NUMBERS PROVIDE
THE TOTAL EXPERT RESPONSES ON EACH DIMENSION)
SURVEY DETAILS FROM ITALY
SURVEY DETAILS FROM LITHUANIA
CONCLUSIONS

High response rate in all countries

Wide sample of countries which had never been
investigated before

Flexible model

Possibility to use expert surveys for different aims and
topics
COMPARATIVE POLITICS
SHORTCOMINGS OF EXPERT
SURVEYS
Adelaida Foitik
Prof. Marco Giuliani
EXPERT SURVEYS – ADVANTAGES ->
POPULARITY






economical way of measuring party positions
provide information on party policy positions in a common and
standardised format
can be administered at any time, unlike manifestoes-tied to electoral
calendars
as long as experts are willing to respond to surveys, the expert survey
methodology may probe topics that do not surface in manifestos or
other data sources e.g. internal dissent within a party
quick and easy compared to other methods(content analysis of party
electoral programs or legislative behavioural studies)
they reflect the judgements of experts=> weight and legitimacy
BUT how valid is the information?? ANALYZE: what, how &
when
THE VALIDITY OF EXPERT JUDGMENTS
Framing survey questions:
 What is the basis of the judgments that experts offer?
 Do experts answer questions in the way they were intended?
-> Budge (2000): 4types of problems
 what ‘party’ is being judged by the expert? - Is it the party in the
electorate, the party in government or the party organization?
 what criteria do experts bring to bear when they judge party positions?
- what do abstract labels like ‘left’ and ‘right’ mean to the expert?
 do experts judge the intentions of parties or their behavior?
 what is the time frame for the judgments that we ask experts to make?
->Fundamental question: how do experts interpret the questions in
expert surveys and how do they link substantive knowledge about
parties to those questions?
REMARK
Wildly varying considerations when judging party positions
 misleading expert judgments




Solution: expert survey design- good questionnaire will attempt to:
avoid ambiguous terms such as ‘party’ and ‘left-right’
give them a more circumscribed meaning indicating precisely what
judgment should concern
specify time frames explicitly to limit variation on this dimension
CONT’D


e.g.: experts may be asked to judge the position of the party leadership
on issue X during the past year
e.g. in evaluating the left-right position of parties, experts may be asked
to describe what ‘left-right’ means in a particular country or what
criterion they used to define this dimension
HOWEVER, better expert survey design might not alleviate all of
the validity concerns raised by Budge: problems like telescoping
(recalling facts for the wrong time period
before/after they happened or afterwards) are bound to plague such
judgments, even when we provide an explicit time frame
Even the most carefully crafted question may still leave an
interpretative space for experts that could distort their judgments.
I. THE 1999 EXPERT SURVEY OF NATIONAL
PARTY POSITIONS ON EU INTEGRATION
->evaluated by Budge, developed by Ray (1999), objective: to assess the positions
of national political parties vis-à-vis European integration, run 1996 - 1999 ,
15 EU Member States
NOTE: in order to minimize response variation due to differential scale
interpretations by experts: the question specified
 the object that was to be evaluated (the party leadership)
 the time frame for the evaluation (1999)
 response options were explicitly labeled
=> the question was designed to put the experts in a common frame of mind
so that they would be judging the same object, on the same dimension, at the
same point in time
CONCLUSION OF THIS SURVEY
=> encouraging findings- the correlations among experts are on
average very high, both within particular countries and across
the whole set
-> substantial convergence in the judgment criteria that experts
use: remarkable consistency in responses, agreement in expert
placements of political parties with other measures.
=> considerable confidence in the expert data
II. BENOIT AND LAVER (2006)
-
-
-
instead of using experts’ answers to locate parties within a given policy
space, use them to directly map the ideal points of the respondents
Focus: the Italian case-rely on two expert surveys conducted with the
same methodology in a short time span (2003 ,2006)=> possible to
check for the robustness of our results
locate each party on the L-R dimension- all aspects of party policy
policy preferences of parties- 4 substantive policy dimension
(economic policy, social policy, the decentralization of decision making
and environmental policy )
“sympathy scale” asked experts to place all parties on a scale indicating
their own closeness to each party’s-> to test for any possible respondent
bias by checking whether expert placements of parties on substantive
dimensions were correlated with their personal sympathy for a party’s
policies
EXPERT PREFERENCES AND BIAS: AN ANALYSIS
- party positions estimated on the basis of survey data are not always reliable respondents tend to place the parties they like closer to where they locate
themselves( = rationalization problem)
=>This is a point well recognized in BL: “the classic problem of a sample
bias is not a concern [in expert surveys], even if the experts we consult
hold strong political preferences, as long as these preferences do not
interfere with their expert knowledge”
RESULTS:
=> on a pool of 158 observations (i.e., the number of parties surveyed), 20
biased parties (12.7%) were found. NOTE: almost two-third (65%) of the
biased parties are extreme-right ones. HOWEVER: not every extreme-right
party is biased (just the 59%) AND not every biased-party is indeed an
extreme-right one (one-third of the total)
REMARKS

the consequent level of bias (1.2 out of 10parties), may have
nothing to do with the “true” characteristics of the different
communities of political experts because information drawn
from individuals is quite vulnerable to selection bias

the selection bias is often due to low response rates. Instead of
trying to increase the rate of response in the hope of decreasing
the level of bias (if a selection bias problem is indeed present),
we should attempt to give a more circumscribed policymeaning to the questions administered, avoiding as far as possible
ambiguous terms such as, “Left-Right”
CONCLUSION OF THIS SURVEY




a rationalization problem is indeed present: this bias is relevant because it can
jeopardize the usefulness of expert scores for empirical analysis
the ambiguity of a label as Left-Right locations => expert judgments become
unreliable
evidence of bias in the expert placements of parties especially against right-wing
parties (not necessarily extreme-right ones)
the Left-Right position of a party is less important than the policy preferences
of experts (and their variance) to explain the probability of a party being biased
HOWEVER, the bias is less pronounced when we pass from a label as “LeftRight” to less abstract policy dimensions
CONCLUDING REMARKS

Reliance on expert judgments is an attractive option for measuring complex
phenomena such as party positions about policies: they are comparatively
straightforward to conduct, but we need to make sure about their validity.

As suggested by Budge’s concerns, those who use expert surveys should be
cautious.

We have investigated the shortcomings of the expert survey with particular
attention to 2 specific cases: while in the first one the results appear to be
encouraging revealing the remarkable agreement among experts about the
placement of parties, in the second one there is evidence of bias in the expert
placements of parties along the Left-Right dimension

Validity assessment is an exercise that should be a central part of expert
survey
THANK YOU FOR
YOUR KIND
ATTENTION