Does Measuring Performance
Lead to Managing for
Performance?
Examining a Poorly Understood
Relationship
M. Bryna Sanger
Deputy Provost & Senior VP, Academic Affairs
June, 2011
Performance Measurement is de rigueur in
U.S. Cities
• Many cities devote considerable resources to measuring
performance
• Multiple organizations support, promote, fund, and reward
performance measurement
• Performance measurement efforts respond to citizens
pressure for increased accountability
• Performance measurement is increasingly viewed as a sign of
“good government” and best practice
Improving the Quality of Public Services
APPAM / Moscow Conference
June 27-29, 2011 2
Many Reasons to Measure
Performance
• To evaluate, control, budget, motivate, promote,
celebrate, learn and improve (Behn, 2003)
• To enable learning and improvement, key elements of
management are:
 Understand what drives performance
 Collect, analyze, and disaggregate (by district or precinct)
timely data frequently
 Take risks, experiment, and tolerate well-conceived failures
 Evaluate and provide feedback relentlessly (about what
works and doesn’t)
 Sustain leadership to build a culture supportive of
innovation and problem-solving
Improving the Quality of Public Services
APPAM / Moscow Conference
June 27-29, 2011 3
“Does developing a culture of
performance measurement lead to
building a system for performance
management?”
• We hypothesized that it would:
Performance
Measurement
Question how
to improve
Learn how to
improve
Improving the Quality of Public Services
APPAM / Moscow Conference
Performance
Management
June 27-29, 2011 4
Research Design – Data Collection
• Identify US cities that measure their performance
• Conduct a search of citywide and agency level public
documents for evidence of their performance measures.
 Evidence was found in documents of 190 cities, including:
•
•
•
•
•
Strategic plans
Budgets
Performance reports
Annual reports on service efforts and accomplishments
Citizen surveys
Improving the Quality of Public Services
APPAM / Moscow Conference
June 27-29, 2011 5
Research Design – Data Analysis
• Evaluate the quality and character of their performance
measurement systems based on:





Nature of their measures and the quality of their reporting
The frequency of measurement
Benchmarked against other cities, other time periods
Disaggregated on a sub-jurisdictional level (looking for outliers)
Demonstrated evidence of target setting
• Analyze difference between cities with mature systems, and
those without
 Assess whether predicted demographic and environmental conditions
are related to these differences
Improving the Quality of Public Services
APPAM / Moscow Conference
June 27-29, 2011 6
Research Design – Interviews
• Select most developed systems (27) for qualitative interviews
with public officials at city and agency levels
 To understand:
• The history of their systems
• The motivation for their efforts
• The way performance data was used, whether they used it to
manage, and how (management structures)
• The kind of leadership responsible for system elements and the
culture around measurement and learning
• The impact of the recession on their efforts
Improving the Quality of Public Services
APPAM / Moscow Conference
June 27-29, 2011 7
Robust Performance Measurement is the
Result of a Complex Set of Conditions
•
A city’s performance measurement system is not a simple
result of factors suggested by the literature:




Measures of professional management
High levels of social capital
Progressive environments with high levels of political engagement
Demographic and environmental characteristics
Improving the Quality of Public Services
APPAM / Moscow Conference
June 27-29, 2011 8
Few Cities Use Performance Data to
Manage (even the most mature)
• Only 5 of 24 cities demonstrated the use of data to manage
for performance
• Longevity is dependent on the leadership of a performance
champion
 Changing leadership and level of resources committed pose regular
threat to systems
• Few cities go beyond agency level to measure citywide
• Cities seldom audit data to insure its integrity
• Few cities have developed a true culture of learning and
measurement necessary to imbed practices and internalize
values that sustain systems
Improving the Quality of Public Services
APPAM / Moscow Conference
June 27-29, 2011 9
Performance Driven Change is Highly
Constrained
• works best when:
Managerial autonomy is allowed, usually at the
agency level
Leadership is strong and stable
Learning culture is built with imbedded routines
Managerial discretion with in the organization is
promoted
External support is available
Improving the Quality of Public Services
APPAM / Moscow Conference
June 27-29, 2011 10
Have we Oversold the Performance
Movement?
• Our findings do not support the view that having a culture of
measurement leads inevitably to a performance managed system
• Significant accomplishments can be identified – as in the concrete
accomplishments of police departments and cities using other CompStator balanced scorecard type systems
• However, even cities with exemplary measurement systems fail to move
naturally to developing a learning organization and building the
management structures and routines to improve performance
• Performance managed systems have not yet systematically demonstrated
their cost-effectiveness
Improving the Quality of Public Services
APPAM / Moscow Conference
June 27-29, 2011 11
Characteristics of Cities that Measure
Performance
Median Household
Income
$48,691
$51,578
Mean Population
All Cities
Exemplary Cities
276,484
271,468
Racial Characteristics
Political Environment
All Cities
Exemplary Cities
All Cities
Asian
5%
NonPartisan
54%
Democratic
23%
NonPartisan
54%
Republican
23%
Democratic
25%
Republican
21%
Hispanic/
Latino
14%
Indian
1%
Atlantic
States
Southwest 5%
Midwest
18%
19%
South
27%
Mountain
States
11%
Pacific
20%
Asian
6%
Hispanic/
Latino
17%
Indian 1%
Black
14%
White
66%
Region
All Cities
Exemplary Cities
Black
7%
White
69%
Form of Government
Exemplary Cities
Southwest Midwest
21%
21%
South
13%
Mountain
States
17%
Pacific
28%
All Cities
Commission
2%
MayorCouncil
31%
CouncilManager
67%
Exemplary Cities
MayorCouncil
21%
CouncilManager
79%
Performance Measurement Maturity
Measures
By Agency – All Cities
100%
By Agency – Exemplary Cities
Performance Report
Budget
Strategic Plan
Citizens Survey
80%
60%
80%
Performance
Report
Budget
60%
Strategic Plan
100%
40%
40%
20%
20%
0%
0%
Fire
Police
Public Works
Parks
Fire
All Cities
80%
Police
Public Works
Type of Measure
Citywide
100%
Citizens
Survey
Exemplary Cities
100%
All Cities
Exemplary Cities
80%
60%
60%
40%
40%
20%
20%
0%
Parks
0%
Performance
Report
Budget
Strategic Plan
Citizens
Survey
Has 311
Inputs
Outputs
Outcomes
Efficiency
Quality
Indicators
Performance Measurement Maturity
Measures
100%
Benchmarking
All Cities
Exemplary Cities
80%
60%
40%
40%
20%
20%
0%
0%
Against Other
Time Periods
Against Other
Jurisdictions
100%
Use
Subjurisdictions
All Cities
80%
60%
Benchmarks are
Used
Targets
100%
Exemplary Cities
Targets are
Used
Most
Service
Areas
Some
Service
Areas
Few Service No Service
Areas
Areas
Frequent Reporting
All Cities
Exemplary Cities
50%
0%
Performance Report
Strategic Plan
Improving the Quality of Public Services
APPAM / Moscow Conference
June 27-29, 2011 14