EXPERT SURVEYS
and PARTY
MANIFESTOS
Limits and potentials
STRUCTURE




Expert survey by Benoit and Laver on
party policy
Some reviews:
Curini’s analysis
Steenbergen and Marks
Party Manifestos by CMP
Curini’s analysis
DESIGNING THE EXPERT
SURVEY
Benoit and Laver conducted a set of
systematic surveys in order to collect the
information required to locate all politically
relevant parties on a wide range of policy
dimensions
The surveys were conducted from 2002-03,
covered 47 countries, resulted in 1,491 valid
expert responses, locating 387 different
political parties on scales relating to a total
of 37 unique policy dimensions.
1.1 IDENTIFYING EXPERTS
The typical expert is an academic
specializing in political parties and electoral
politics of his or her country.
Method divided in steps:
Contact the national political science
association of the country, if it exists, with a
request for its membership lists.

1.1 IDENTIFYING EXPERTS
If
it does not exist, “snowball” strategy is
used: short list of well-known experts from
the country, and ask each to name as many
additional experts as possible
Then contact each of those experts,
asking them in turn to name as many
additional experts as possible

Additional lists from universities and NGO,
and exclude journalists and political actors
1.2 IDENTIFYING PARTIES
“Politically relevant” parties where chosen,
when they met any one of three criteria:
Every
existing national party that won
seats in the national legislature at the
country’s most recent election
Every
existing national party that had won
at least 1 percent of the vote nationally at
the country’s most recent election
1.2 IDENTIFYING PARTIES
Any other parties indicated by local
experts as politically relevant despite not
meeting the other two criteria

Even small parties are included since they’re
crucial for the analysis of political
competition (ex. in forming gov)
also to avoid to exclude parties that would
play significant roles in party competition
over a longer period.
1.3 POLICY DIMENSIONS
Core set of four substantive policy dimensions
in order to allow for direct comparison
between countries:
1.ECONOMIC
POLICY
2.SOCIAL POLICY
3.ENVIRONMENTAL POLICY
4.DECENTRALIZATION OF DECISION MAKING
For each dimension a scale from 1 to 20, with
the lower numbers indicating a ‘left-wing’
position and the higher a ‘right-wing’ position.
LIMITS of EXPERT SURVEYS
2.1 HOW TO EVALUATE BIAS
BL are aware that there could be a systematic
relationship between the experts’ own
ideologies and their judgments about party
policy positions
SYMPATHY SCALE:
Experts are asked to place all parties on a
scale indicating their own closeness to each
party’s policies.
To assess bias BL used the following
regression-based technique:
For
each country-party section, they
regress experts’ placements on their
sympathy scores for the party in question
They
compute the mean and the variance of
the predicted left–right score for an expert
with an ‘indifferent’ sympathy score of 10.5
(the midpoint of the 1-to-20 scale), taking
this to be the ‘corrected’ placement
they
compare the ‘corrected’ expert placements with
the ‘actual’ placements (i.e. the mean score of
parties resulting from the expert survey) to see
whether or not these differences are statistically
distinguishable
From this analysis only seven parties appeared to
involve sistematic bias that cuold be statistically
distinguished from zero.
2.2 SAMPLING/IDEOLOGICAL BIAS

Ideological bias known as rationalisation
or projection: respondents tend to place
the parties they like closer to where they
perceive themselves to be (assimilation
effect) and to place those parties they
dislike farther away (contrast effect).

Sampling bias: respondents are
systematically unrepresentative of all
possible experts (more left-wing or liberal)
2.3 SOLVING IDEOLOGICAL BIAS
Focus on the sympathy question alone: risk
of under/overstimate IB
Curini:
Use an unfolding model in order to
estimate the location of the ideal point
using respondent’s preferences
“corrected score”: focus on the relationship
between respondents’ ideal points and
parties’ ideological stance
Scoreij = aj + βj [s(Policyi – Scorei)] + εi
Policyi : respondents’ position (unfolding
method) on the left-right scale
Scorei : mean party position on the same scale
Bj : indicator of ideological bias if it is
statistically distinguishable from zero
(as it was the case for 53 parties out of 158)
a direct way of “correcting” biased
score, neutralising the ideological bias
2.4 EVALUATING EXPERT SURVEYS
Budge four type of problems:
what
‘party’ is being judged by the expert?
what criteria do experts bring to bear when
they judge party positions?
do experts judge the intentions of parties or
their behavior?
what is the time frame for the judgments
that we ask experts to make?
HOW DO EXPERTS INTERPRET THE QUESTIONS IN
EXPERT SURVEYS AND HOW DO THEY LINK
SUBSTANTIVE KNOWLEDGE ABOUT PARTIES TO
THOSE QUESTIONS?
Steenbergen and Marks: these concerns can
be alleviated in the expert survey design:
Avoid
ambiguous terms
Give them a more circumscribed meaning
limit interpretation
but there may still be an interpretative space
for experts that could distort their judgments
The 1999 expert survey of national party
positions on European integration
Ray: “What was the overall orientation of the
party leadership towards European integration
in 1999”
7 response options
Specification of the object that was to be
evaluated (party leadership)
Specification of the time frame (1999)
provide to the experts a common frame
of mind
Assess variance in the experts’ judgments:
 Compute the standard deviation of their
placements of parties
 Perform a variance components analysis:
limited statistically significant variation
across the experts
rather similar judgments about EU
stances of the political parties
 Attributes of a party or party system:
measure party differentiation within a party
system as the standard deviation of party
positions in that country
variation is a function of party
differentiation, salience and dissent
Steenbergen and Marks’ model
The
“congeneric test model”: the
correlation between the expert judgments
should be greater, perhaps much greater, when
they are judging the same trait than when they
are judging different traits
similarity
coefficients: compare the
correlational patterns across experts, we should
expect to see a high similarity coefficient
between two experts if those experts indeed
evaluate the same trait
THE ITALIAN EXCEPTION
Reliability of expert judgments on Italian
parties is worse than that for other countries.
This is primarily due to one expert (E4).
The correlations between the judgments of
this expert (E4) and the remaining five
experts are low, suggesting that he or she
may have used different criteria for placing
Italian parties.
good grounds for doubting the validity
of this expert responses
although it is worth emphasizing that even
after including this expert the final results
for Italy look quite good
ALTERNATIVE MEASURES
Manifesto
Research Group: extract party
positions from party manifestos. It has coded
both favorable and unfavorable mentions of
European integration
Voter perceptions of party positions. The
EES project asked respondents where parties
stood on the issue of European integration.
Subject to projection effects voters may project
their own stance on European integration onto
the party
Use members of parliament (MPs) and
members of the European Parliament (MEPs) to
gauge party support for European integration
ANALYSING PARTY
MANIFESTOS
In 1979 the Manifesto Research Group (MRG)
was formed to analyze documents in 90
democracies comparatively and within a
common framework.
Manifesto: written as a whole and as overall
balance of subjects carefully considered
IT IS OVERALL PRIORITIES THAT MATTER RATHER
THAN PARTICULAR STANCES TAKEN IN ISOLATION
The idea behind the MRG is that parties argued
with each other by emphasizing different policy
priorities rather than confronting each other on
the same issues
Since 1989 CMP (the Comparative Manifesto
Project) has coded and standardized the
statements of party election programs of 25
democracies since 1945 into 56 policy
categories, such that each quasi-sentence of
every election program is coded into one, and
only one, specific category.
SENTENCE WHICH IS THE
VERBAL EXPRESSION OF ONE
POLITICAL IDEA OR ISSUE
(GENERAL CODING UNIT)
Basis of the policy estimates: textual data,
examination of parties’ and governments’ own
statements of policy, in the shape of election
programmes (manifestos) and declarations
in the Parliamentary debate before a vote of
confidence or investiture
estimates on what actors themselves have
said rather then on people’s judgements
Major source of evidence: texts, autoritative
documents issued by parties and governments
Collect and organize it in statistical form
13 categories as indicators of policy positions
emphasised by the “left” and another 13 as
indicators of policy positions emphasised by the
“right”.
The left-right position is constructed for each
party in each election by subtracting the sum of
its left statements from the sum of its right
statements.
As a result the range of this scale is between –
100 (i.e., the score of a party devoting its entire
programme to Left-wing issues) and +100 (i.e.,
the score of a party devoting its entire
programme to Right-wing issues).
ADVANTAGES OF PARTY
MANIFESTOS
They allow three types of comparison:
Changes
in policy position over time within
specific parties
Differences in policy positions across parties
even in different countries
Differences across countries
RELIABILITY and VALIDITY
Two measurement concerns:
Reliability: ARE THE PROCEDURES DERIVED FROM THE
THEORY RELIABLE, GIVING THE SAME RESULTS EVERY
TIME THEY ARE APPLIED? (computers will always apply
some procedures to the same text in the same way: this is
not always true of the human coders who have produced
the Manifesto data)
Validity:
ARE PROCEDURES VALID IN THE SENSE OF
ACTUALLY TELLING US WHAT WE THINK THEY TELL US
ABOUT CONTENT (OF MANIFESTOS) AND SENDERS
(PARTIES)? (is the underling theory itself correct? Are the
codings correlated with other measures of the same
conceptual construct?)
THE ITALIAN CASE
CURINI
using Manifesto scale, a tradeoff can emerge: if on one hand this
methodology
guarantees
a
more
theoretically sound comparison between
countries and time, on the other hand it
does not allow for specific characteristics of
individual party systems (that could
therefore be buried under a uniform leftright dimension).
The Italian party system was
characterized for a long period by:
THE PRESENCE OF TWO ANTI-SYSTEM PARTIES, THE
COMMUNIST PARTY AND THE ITALIAN SOCIAL
MOVEMENT ON THE RIGHT

THE ABSENCE OF ANY CREDIBLE ALTERNATIVE TO A
GOVERNMENT FORMULA BUILT AROUND THE
CHRISTIAN DEMOCRATIC PARTY


A HIGH NUMBER OF RELEVANT PARTIES
POLARIZED PLURALISM
PLACEMENT OF THE 4 MAIN ITALIAN PARTIES DURING THE “FIRST
REPUBLIC” (1948-92), USING THE MANIFESTO SCORE
Some results seem quite odd: in two occasion the DC is located at
the right of the MSI, along the 70s the PSI is at the left of the PCI,
and the PCI in the 1953 election is the furthest right party.
CURINI’S METHOD
For scholars who want to apply models that rely
on the policy preferences of the Italian political
parties and that wish to use PMD, Curini proposes
a method that consists of the following steps:
 The first change needed is the application of
a reliability analysis to the original Manifesto
scale, in order to assess the scale’s internal
consistency: the Cronbach’s alpha coefficient
is a famous indicator of consistency. Ideally, it
should be above .7. The scalability of the
original Manifesto scale for the country
investigated should be assessed
If the Cronbach’s alpha is satisfactory,
parties’ score should be smoothed by
recognizing how the score of a party at time
t does not translate directly in its position

(If however the reliability analysis shows
some inconsistency, the possibility to drop
some categories included in the original
left-right scale should be considered
before smoothing the parties’ scores)
CRONBACH’S ALPHA RESULTS
POSITIVE (.54) ALTHOUGH LOW
BY THE STANDARDS FOR LIKERT
SCALE. TO TEST WHETHER ALL
THE INDICATORS BELONG TO
THE SAME SCALE, VALUES FOR
THE CRONBACH ALPHA ARE
COMPUTED ALSO WHEN
SPECIFIC ITEMS ARE REMOVED
FROM THE SCALE. AS A RESULT
THE CATEGORY “DEMOCRACY”
SEEMS NOT TO TAP THE SAME
CONCEPTUAL DIMENSION AS
THE OTHER INDICATORS.
REMOVING THIS CATEGORY
HELPS A BIT THE CRONBACH
ALPHA RESULT (.58). HOWEVER,
THE STILL UNSATISFACTORY
VALUE OF THE SCALE SUGGESTS
THAT THE EQUAL WEIGHTING
ASSUMPTION OF ALL THE ITEMS
WHICH CHARACTERISES THE
ADDITIVE MANIFESTO SCALE BE
DROPPED.

Second modification: the Manifesto score
for a given party need not be identical to
its actual positional value
THE GENERAL PUBLIC IS EXPECTED TO
WEIGHT INFORMATION FROM THE PAST
AS WELL AS PROMISES AND
DECLARATIONS OF THE LATEST
MANIFESTOS TO IDENTIFY THE ACTUAL
POSITION OF A GIVEN PARTY
Then the position of a party at time t
should reflect not only the score obtained in
its current electoral program but also the
party’s general image emerging from the
cumulation of all its previous manifestos
To produce measures that capture this
dynamics, the exponentially weighted
moving average of current and past
manifesto positions is computed:
POSITIONt = a MANIFESTOt + (1.0 –
a)POSITIONt-1
Curini analyzes the 1953 general election
and, by adding the scores obtained from
the Manifesto scale without the category
“Democracy”, and his final scores, he
obtains more plausible results: the PCI
moves from being the most rightist party to
the most leftist, the PSI moves to the left of
the DC, MSI becomes the most rightist one.
CURINI’S CONCLUSIONS
Considering the 2001 Italian general
election, an advantage of his method stands
out:
IN THAT ELECTION MOST PARTIES (WITH THE EXCEPTION
OF COMMUNIST REFOUNDATION PARTY) DID NOT ISSUE
THEIR OWN ELECTORAL MANIFESTOS, BUT FORMED PREELECTORAL COALITIONS AND SUBSCRIBED TO A JOINT
PLATFORM. THIS IMPLIED THAT THE CMP WAS ABLE TO
CODE JUST ONE MANIFESTO FOR EACH COALITION.
NONETHELESS, ESTIMATION OF EACH PARTY’S
INDIPENDENT POLICY POSITIONS IS ALSO REQUIRED AS
PARTIES ARE STILL THE PRE-EMINENT POLITICAL ACTORS
WITHIN ITALY
CURINI’S CONCLUSIONS
By contrast, the smoothing procedure
included in his method allows him to
estimate different scores for different parties
even if at time t parties agree on the same
manifesto, precisely because different
parties (usually) have different histories.
Policy estimates of
Manifesto data are ideal
also for operationalizing
and testing coalition
theories. As an additional
check for the validity of
his scores, Curini uses
them as dependent
variables to test a widely
accepted theory: the
central role of the median
party in policy bargaining.
CONCLUSIONS
McDonald/Mendes: expert surveys are
deficent and the manifesto measures are
useful on two concerns:
The
experts place the parties in such stable
locations that there is little hope of using
expert surveys to investigate party policy
dynamics. The CMP is the only viable data
source for observing such dynamics and
analysing the party movements
Expert
surveys may produce suspect results
about where the parties stand on different
dimensions of politics and policy. There is
doubt that expert respondents actually can
and do make clear distinctions between policy
dimensions.
Steenbergen & Marks: there are good reasons
to trust expert survey results on party
positions:
Consistency of experts’ responses
Expert placements of political parties
converge with other measures
As we have seen both
methods have confident
support. BL admit and
analyse some divergent
estimates between expert
survey and party
manifestos.
They claim that these
differences are due to the
understimation of some
issues in the CMP analysis,
and obviously that their
method is the “correct”
one…
OUR CONCLUSIONS
Expert surveys rely on a certain leeway
accorded to respondents and therefore are
subject to the risk of bias
But when using Manifestos, still the analysis
is based on certain categories that can be
ambiguously assigned both to the left and the
right side, or on a certain statements that
sometimes could be more linked to strategies
(due to the elections) then to the real nature
of that party.
How can one really find the “hidden” real party
position on certain issues?