Methodological and Analytical Issues
Gaia Dallera
6 June, 2012
1
1.
2.
3.
4.
5.
6.
7.
Introduction
Defining Governance
Governance Data Sources for the WGI
Constructing the Aggregate WGI Measures
Using and Interpreting the WGI Data
Analytical Issues
Conclusions
2

The WGI are a research project to develop cross-country
indicators of governance.

We will focus on the methodology and key analytical issues
relevant to the overall WGI project.
3

“the traditions and institutions by which authority in a
country is exercised….
..This includes three areas:
1.
2.
3.
The process by which governments are selected,
monitored and replaced.
The capacity of the government to effectively formulate
and implement sound policies.
The respect of citizens and the state for the institutions
that govern economic and social interaction among
them”.
4
No.
Dimension
Indicators
Code
1.
Selected, Monitored and
Replaced
Voice and Accountability
VA
2.
Selected, Monitored and
Replaced
Political Stability and Absence of
Violence/Terrorism
PV
3.
Formulate and implement sound
policies
Government Effectiveness
GE
4.
Formulate and implement sound
policies
Regulatory Quality
RQ
5.
Respect of citizens and state for
institutions
Rule of Law
RL
6.
Respect of citizens and state for
institutions
Control of Corruption
CC
5
These indicators are based on several hundred variables
optained from 31 different data sources capturing governance
perception
Availability of the underlying data from (with few exceptions)
the individual data sources (transparency and replicability)
The WGI data sources reflect the perceptions of a very
diverse group of respondent:
There are four categories:
1.
2.
3.
4.
“Surveys” of domestic firms and individuals with first hand
knowledge of the governance situation
“Public sector data providers”
Commercial business information providers
Non-governamental organizations
6


An important qualification is that the sources of
data provide different coverage across countries
(some cover a majority of countries and some
cover small groups of countries).
Each individual variable are not comparable across
countries
7
8

Combines the six aggregate governance indicators using a statistical
tool known as the “Unobserved Component Model” (UCM)

The underlying premise of this statistical approach is straightforward
– each of the individual data sources provides an imperfect signal of
some deeper underlying notion of governance that is difficult to
observe directly

This means that we face a signal extraction problem:



How can we isolate an informative signal about the unobserved governance
component common to each individual data source?
How can we combine the many data sources to get the best possible signal of
governance in a country?
Therefore we construct a linear regression model in order to:
•
•
•
Standardize the data from these diverse sources into comparable units;
Construct an aggregate indicator of governance as a weighted average;
Construct a margin of error that reflects the unavoidable imprecision in
measuring governance.
9

Regression model
ykj=ak +bk(gj+ejk)

Different thing to note about this regression
model is that the error term is considered as an
independent variable.
“a” and “b” are parameters that reflect the fact
that different sources use different units to
measure governance.
 The error term and precisely its variance
captures two sources of uncertainty.

10

We consider the estimate of governance as a weighted
average of the re-scaled scores for each country.
K
E[gj I yj1,…., yjk ]=
∑
k=1
wk
yjk - ak
bk

The re-scaling puts the observed data from each source
into the common units we have chosen for unobserved
governance.

The larger the weights, the smaller the variance of the
error term.

A crucial observation is that there is an unavoidable
uncertainty around this estimate of governance.
11
 We
report the aggregate WGI measures in
two ways:
•
Standard normal units of the governance
indicator ranging from -2.5 to +2.5.
•
Percentile rank terms ranging from 0 (lowest) to
100 (highest).
12
13
14
 Size
of the confidence intervals varies across
countries.
 Resulting
confidence intervals are substantial
relative to the units in which governance is
measured.
15

Many of the small differences in estimates of
governance across countries are not likely to be
statistically significant at reasonable confidence
intervals, since the confidence intervals are likely to
overlap.

Example of overalapping confidence intervals:
Indicator
Country
Coefficient
Scale
Control of Corruption
Jamaica
5.0
4.0 to 8.0
Control of Corruption
Peru
6.0
4.0 to 6.0
16

This is not to say that aggregate indicators cannot be used
to make cross country comparisons.

There are in fact many pair-wise country comparisons that
are statistically significant.

63% of the pairwaise comparisons the confidence intervals
do not overlap
17

For example:
• 2009 Control of Corruption indicator covers 211
countries;
• Total of 21,155 pair-wise comparisons.
% of
Comparisons
Confidence Intervals
Significance
63%
90% confidence
intervals do not
overlap
Signals
statistically
significant
differences in the
indicator
73%
75% confidence
intervals do not
overlap
Same as above
18
UCM has three main advantages:

•
It maintains some of the cardinal information in the
underlying data.
•
It provides a natural framework for weighting the
re-scaled indicators by their relative precision.
•
It naturally emphasizes the uncertainty associated
with aggregate indicators. In fact it formalizes the
signal extraction problem and by doing that it
provides a rationale for a more inclusive approach
to combining data from different types of sources.
19
Unbalanced vs balanced samples:

•
Refers to the fact that WGI use all available
data sources for all countries as opposed to
using only those data sources that cover all
countries and in all time periods (permitting
balanced comparison).
20

This choice is based on the view that perception
data has a particular value in the measurement
of governance because:
•
Agents (citizens and enterprise) base their action on their
perceptions;
•
In many areas of governance there are few alternatives to
relying on perception data e.g. corruption ;
•
The “jure” notion of laws often differs from the de-facto
reality.
21

1.
Potential problems:
Interpretation of subjective data
•
•
•
2.
Perception data is imprecise;
But all measures of governance are imprecise proxies for the broader
concept;
Therefore it underscores the importance of using empirical methods and
taking seriously the extent of imprecision.
Systematic biases in perception data on governance introduced
by:
•
•
•
•
Different types of respondents may differ systematically in their
perceptions of the same underlying reality;
The ideological orientation of the organization providing the subjective
assessments of governance.
Possibility that different providers of governance perceptions data rely
on each other’s assessments (correlated perception errors).
BUT THERE IS LITTLE EVIDENCE OF SUCH BIASES!!!!
22
 WGI
reports six dimensions of
governance covering 200 countries
since 1996.
 Updated
annually.
 Based
on hundreds of variables from
many different data sources.
23
 Due
to the inherently unobservable
nature of the true level of governance
in a country, any observed measure is
only a proxy.
 Consequence
is that our estimates are
subject to non-trivial margins of error.
24
 Don’t
over interpret small differences in
performance (across countries or over time).
 Presence
of errors does not mean that the
WGI cannot be used to make meaningful
comparisons.
 Estimation
of and emphasis on such margins
of error is intended to enable users to make
more sophisticated use of imperfect
information.
25
 In
fact it is possible to make meaningful
comparisons cross-country and over time.
 Almost
67% of all cross-country comparisons
in 2009 result in highly significant differences
at 90% confidence intervals.
 More
than one quarter of countries show a
significant change in at least one of the six
WGI measures during the 2000-2009 decade.
26
THANK YOU
27