Measurement in the Public Sector
Professor Imran Rasul
UCL and IFS
January 2015
iGovern IE Workshop, Istanbul
Measurement Matters
• No matter how good the IE design, what can be learned from an evaluation depends crucially on
the quality of measured variables
• Outcomes (dependent variable):
•
•
•
•
Which measures of public service delivery (quantity and quality)?
What level of aggregation (pyramid: individual to ministry)?
Comparability across units of observation (common forms of data collection)
Comparability within units through time (short run, long run)
• Controls (explanatory variables):
•
•
•
•
•
•
Individual
Project
Organization
Population/citizenry being served (bottom up monitoring)
Bureaucratic-political nexus (top down monitoring)
…
Measuring Output Example:
Public Sector Projects (Across Ministries, 2003-12)
Each project, at its outset, has a proscribed budget, and planned progression (by year).
Potential Outcome 1: Meeting Progression Targets
Potential Outcome 2: Time/Cost Overruns
Measurement Challenges in the Public Sector
• Systemic interlinkages of activities in the public sector
• non-independence of project outcomes
• not all projects leading to outputs for citizenry (e.g. procurement, training)
• Intermediate steps in service delivery
• drafting plans, task allocation, coordination, implementation
• Attribution of output to specific individuals
• teamwork
• imperfect indicators of inputs (hours worked ≠ effort)
• Output not always measured by market prices
• harder to quantify underlying demand for, and welfare implications of, PS projects
• Varying objectives of public sector organizations
• equity, social welfare, etc.
• Establish common data collection methods: freq./measurement error
Opportunities
• Collating secondary data that already exists
• Engaging in primary data collection
•
•
•
•
bureaucrat characteristics (time use, non-cognitive traits)
organizational characteristics (management practices)
user characteristics (satisfaction with public services)
engage with researchers to help build capacity to collect own data
• Use of new technologies provides new possibilities to monitor:
• bureaucratic activity (time inputs)
• citizen usage
• Use of new/combined methods to answer research questions:
• bottom up monitoring quality of public services (mystery shoppers)
• uncovering illicit behaviors (list randomization, elicitation etc.)
• lab and field experiments (use games to measure structural/preference parameters)
(Continued) Case Studies on Measurement:
Role of Non-Pecuniary Incentives
• Sorting/recruitment into the to public sector
• What types of non-monetary incentive are likely to be effective in
public sector jobs?
• wage compression in public sector relative to private sector
• although, growth in use of pay-for-performance
• management practices and public service delivery
1. Sorting/Recruitment
• Roy [1951] model: individuals sort into jobs according to their
preferences, skills and job attributes
• XS differences in personality/preferences between those in public and
private sectors:
• preferences aligned with mission of organization/intrinsic motivation
• preference aligned with work environment/culture [teamwork, less exposure to
competition etc.]
• Research problem: costly to screen applicants based on these kinds of
preferences…
Financial Incentives and Recruitment
• Research Question: Do we induce adverse selection of types if we just use
posted wages to recruit workers into the public sector?
• Evidence: Dal Bo et al. [2013] study the impacts of experimentally varying
financial incentives on applicant traits
• Context: Mexico; 3750 vs 5000 pesos/month wage offer (33% increase)
• Measurement:
• Ability and personality:
• Earnings in previous job; Raven’s IQ, Big 5
• Motivation:
• Public Service Motivation [Perry 1996]
Result 1: Higher wages attract more applicants
Result 2: Higher IQ applicants apply
Fewer low IQ applicants
More high
IQ applicants
Result 3: Personality Traits of Applicants Vary
with Wage Offers
Result 4: No adverse selection on public service motivation
Unanswered Measurement Related Question: do such recruits perform better on the job?
Ashraf et al. [2014]: Experimentally Vary
Recruitment Posters for Health Workers
• Contrast is over making career incentives (wage progression) more or
less prominent in recruitment adverts
• Job characteristics element of Roy [1951] model
• Context: community health workers, Zambia
• Measured outcomes:
• Applicant pool [ability, motivation]
• Performance on the job
Such information can affect the beliefs individuals hold…
Bayesian individuals:
[Crawford and Sobel 1982, Milgrom and Roberts 1986,
Gentzkow and Kamenica 2011],
Non-Bayesians:
think categorically [Fryer and Jackson 2008, Mullanaithan et al. 2008]
limited memory/attention [Mullanaithan 2002, Eliaz and Spiegler 2011]
Non-informative dimensions of communication can affect belief formation
through salience, attention and framing
[Mullanaithan et al. 2008]
…or what they think might be signalled to others [Benabou and Tirole 2003]
Measurement of beliefs critical to understand why
applicant pool might have changed
[Desserano 2014]
Measured: Behaviour on the Job
2. Management
• Rasul and Rogger [2014]: linking the managerial practices that
bureaucrats operate under, to public services delivered
• Context: Nigeria, 63 government departments/federal agencies
• Includes 13 ministries, and other regional development authorities
• Measurement: dimensions of management practice (org. level)
• autonomy
• provision of incentives/monitoring (collection and use of KPIs)
• other (e.g. training, culture…)
Measuring Management Practices in Civil
Service Organizations
• `double blind' face-to-face group interviews with senior managers,
consensus reported
• management practice topics: roles, flexibility, incentives, monitoring,
facilities, skills, staffing, targets and culture
• rank each answer 1 to 5, increasing in common notion of good
practice
• derive three measures of management practice: autonomy,
incentives/monitoring, other dimensions
Table A2: Defining Management Practices
Management Practice
Topic
Autonomy
Roles
Specific Questions Related to this Topic
Can most staff above SGL 7 in your organization make substantive contributions to the policy formulation and
implementation process?
Can most staff above SGL 15 in your organization make substantive contributions to the policy formulation and
implementation process?
To what extent do the employees in this organization have the ability to determine how they carry out the
assignments in their daily work?
Flexibility
Does your organization make efforts to redefine its standard procedures in response to the specific needs and
peculiarities of a community?
How flexible would you say your organization is in terms of responding to new practices, new techniques, and
regulations?
At your organization, how efficiently is best practice shared between departments?
Given past experience, how effectively would a conflict within your organization be dealt with?
• Measurement: Public Service Delivery
•
•
•
•
Audits of 4700 public infrastructure projects conducted by the 63 organizations
11 project types (road, borehole, training etc.) [construction and non-construction]
projects evaluated by independent teams of engineers
outcome: completion rate (0-100%)
• Checks and balances in audits:
• centralized team of technocrats monitored the evaluation teams
• evaluators were asked to provide material, photographic, or video evidence to support
their reports
• random checks on evaluated sites
• We then hand coded the material from all projects
• 4721 projects, 63 civil service organizations, aggregate budget of US$800 million
• one year projects
20
10
0
Percent
30
40
Figure A: Project Completion Rates (Nigeria Rural Infrastructure)
0
.2
.4
.6
.8
Proportion completed, as reported by evaluation team
1
Context: 38% of our sampled projects in Nigeria have completion rate of zero
For projects that start: average completion rate of 70%
Hypothesis: overworked bureaucrats?
1,000
1,500
Figure B: Major Project Types by Implementing Organization
Distinct lack of specialization
in many project types
C
an
al
Fi
na
nc
ia
l
R
es
ea
rc
h
Ad
vo
ca
cy
ai
ni
ng
Tr
R
oa
d
D
Pr
am
oc
ur
em
en
t
n
at
io
tri
fic
in
g
El
ec
Bu
ild
Bo
re
ho
le
0
500
Overstretched organizations?
Notes: Each histogram bar represents the total number of projects for the given project type on the x-axis. Each colored band represents the projects at a
particular organization (different bands imply different organizations irrespective of whether they are the same color). The sample used to construct the
histogram is those projects for which proportion completed evaluation variable and management scores are available. The projects are classified here by their
primary classification.
Does Management Matter?
Conclusion: Measurement
• Quality of IE design depends crucially on measurement of variables
• Cannot start thinking about such issues too early in IE design
• Getting academics involved:
• Fresh ideas less constrained by existing practices
• Collection of new/objective data
• Collation of existing secondary data
• Primary data collection
• Faster dissemination of best practice across organizations
Further Reading
• Amin et al (2008) ‘Are you being served? New tools for measuring
service delivery’
• Carrera and Dunleavy (2013) ‘Growing the productivity of
government services’
• Pidd (2012) ‘Measuring the performance of public services: Principles
and practice’
• Shah et al (2005) ‘Public services delivery’