Discovery Communities TA Institute
April 1, 2008
I N N O VAT I O N
Results-Based Accountability
Results Accountability
Decision-making
and Strategic Planning
Fiscal Policy Studies Institute
Santa Fe, New Mexico
WEBSITES
www.resultsaccountability.com
www.raguide.org
www.charteroakgroup.com
BOOK ORDERS
www.trafford.com
www.amazon.com
2
2
Why Are We Here?
3
3
SIMPLE
COMMON SENSE
PLAIN LANGUAGE
MINIMUM PAPER
USEFUL
4
4
Concepts to Take Away Today

Population accountability v. performance
accountability

Ends v. means

How to choose indicators and measures

The importance of a data development agenda

How to move from talk to action
5
5
Results Accountability
is made up of two parts:
Population Accountability
about the well-being of
WHOLE POPULATIONS
For Communities – Cities – Counties – States - Nations
Performance Accountability
about the well-being of
CLIENT POPULATIONS
For Programs – Agencies – and Service Systems
6
6
Results and Performance
Accountability
COMMON LANGUAGE
COMMON SENSE
COMMON GROUND
7
7
THE LANGUAGE TRAP
Too many terms. Too few definitions. Too little discipline
Benchmark
Outcome
Result
Modifiers
Indicator
Measurable Core
Urgent
Qualitative
Priority
Programmatic
Targeted
Performance
Incremental Strategic
Systemic
Measure
Goal
Objective
Target
8
Lewis Carroll Center for Language Disorders
8
The Humpty Dumpty Approach to
Language
“When I use a word,” Humpty Dumpty said,
in rather a scornful tone, “it means just
what I choose it to mean—neither more nor
less.” “The question is,” said Alice,
“whether you can make words mean so
many different things.” “The question is,”
said Humpty Dumpty, “which is to be
master—that’s all.”
LEWIS CARROLL, Through the LookingGlass, chapter 6, p. 205 (1934). First
published in 1872.
9
9
DEFINITIONS
RESULT
Population
A condition of well-being for
children, adults, families or communities.
Children born healthy, Children succeeding in school,
Safe communities, Clean Environment, Prosperous Economy
INDICATOR
A measure which helps quantify the achievement
of a result.
Performance
Rate of low-birthweight babies, Rate of high school graduation,
crime rate, air quality index, unemployment rate
PERFORMANCE MEASURE
A measure of how well a program, agency or service
system is working.
1. How much did we do?
Three types:
2. How well did we do it?
3. Is anyone better off?
= Customer Outcomes
10
10
From Ends to Means
From Talk to Action
Population
RESULT
ENDS
Performance
INDICATOR
PERFORMANCE
MEASURE
Customer outcome = Ends
Service delivery = Means
MEANS
11
11
Connecticut Glossary of RBA Terms
The Appropriations Committee standardized the terms
we use in Connecticut. You have a copy of the glossary.

We could have chosen other terms consistent with
Friedman’s RBA approach so this list is somewhat
arbitrary.

Its virtue is that everyone in Connecticut doing this
work – both executive branch and legislative branch – is
using the same vocabulary and meaning the same thing
by it. At least we should be able to understand each
other.

12
12
Some Connecticut Early Childhood
Indicators
 % Infants born at low birth weight
 % Births to mothers without a high school degree


% of kindergartners with all or most pre-literacy and personal
skills
% 4th grade reading scores at mastery or above
13
13
IS IT A RESULT, INDICATOR OR
PERFORMANCE MEASURE?
1. Safe Community
2. Crime Rate
3. Average Police Dept response time
4. A community without graffiti
5. % of surveyed buildings without graffiti
6. People have living wage jobs and income
7. % of people with living wage jobs and income
8. % of participants in job training who get living
wage jobs
14
14
IS IT A RESULT, INDICATOR OR
PERFORMANCE MEASURE?
9.
% HS graduates enrolling in college
10. Traffic-related death rate
11. Clean environment
12. Air pollutants in parts per million
13. % participating cities fixing treatment plants
14. All children eat healthy
15. % students eligible for free lunch participating
in free lunch
15
15
POPULATION
ACCOUNTABILITY
For Whole Populations
in a Geographic Area
16
16
Maryland
Results for Child Well-Being
● Babies born healthy
● Healthy children
● Children enter school ready to learn
● Children successful in school
● Children safe in their families and communities
● Stable and economically independent families
● Communities that support family life
17
17
Connecticut Results Statements
•
A clean and healthy Long Island
Sound
• All children healthy and ready to
learn by age 5
• All children ready by five and
fine by nine
18
18
Leaking Roof
(Results thinking in everyday life)
Experience:
Inches of Water
Measure:
Not OK
? Fixed
Turning the Curve
Story behind the baseline (causes):
Partners:
What Works:
Action Plan:
19
19
Seven Population Accountability
Questions

What are the quality of life conditions we want for the
children, adults and families who live in our community?

What would these conditions look like if we could see them?

How can we measure these conditions?

How are we doing on the most important of these
measures?

Who are the partners that have a role to play in doing
better?

What works to do better, including no-cost and low-cost
ideas?

What do we propose to do?
20
20
“We haven’t
got the money,
so we’ve got
to think.”
Lord Rutherford
1871 - 1937
21
21
Connecticut Population Template
Quality of Life Result
All Connecticut children are healthy and ready for
school success at age 5, contributing to a
reduction over time in Connecticut’s “achievement
gap” at Grade 4.
Key Funding Information
Total Current Funding
Funding Distribution
Total Federal Funds
Total State Funds
Capital Projects Subtotal
Other Funding
Percent of Total Funding
Contracted to Third Parties
Indicator 1: Infants born at Low Birth Weight
(LBW)
Dis tr ib u tion o f Bir th s an d L BW In f an ts , 2003
533.3 million
262.9 million
266.3 million
4.1 million
100 %
90 %
Story Behind the Baselines
41,719 babies were born in CT in 2005. About
6,000 (14%) are at risk because their family
income is at or below the Federal Poverty Level.
About 28% of young children at each age
(~12,000 children) are at risk of school unreadiness because their family income is at or
below 185% of the Federal Poverty Level.
29 %
35%
80 %
70 %
19 %
60 %
22%
50 %
40 %
30 %
52 %
40%
20 %
10 %
0%
Tot al B i rt hs
LB W
P r i or it y D i st r ic t s
Ot her S R D i st r ic t s
A l l o t h er T ow n s
Indicator 2: Births to Mothers Without a High
School Degree
% Births to Mothers without a High
School Degree
Nearly eight in ten of these “at risk” children
(78%) live in just 19 towns, CT’s Priority School
Districts. Another 15% live in the balance of
School Readiness Towns (39 towns). These 58
towns are listed at the back of this template. The
rest of CT’s at risk children, defined (for now) as
living in poverty, (7%) live in the remaining 111
communities.
25
Other indicators point to developmental
challenges for many of these same children:
20
15
10

Seven in ten of all low birth weight babies
(71%) live in the 58 School Readiness towns,
and half of them (52%) live in the 19 Priority
School Districts. Of note, while the average
rate of low birth weight babies is stable (but
too high), it is rising among African American
families.

About a quarter (23%) of mothers with young
children who live in Priority School Districts
have not attained a high school degree, a
rate much higher than for other communities
in CT. The proportion of mothers without a
high school degree is increasing in these 19
towns as well.

Nearly two-thirds (64%) of children from
Priority School District towns enter
kindergarten without the pre-literacy skills
needed for early school success.
5
0
2000
2001
2002
Pr ior it y Dist irct s
2003
Ot her SRP
2004
2005
Balanc e of Towns
Indicator 3: HUSKY A Enrollment and
Participation Rates
HUSKY A Enrollment and Well-Care Participation,
Children Under Six Years
85%
Percent
80%
75%
70%
65%
2002
2003
2004
2005
Enrollm e nt of
Eligible Childre n
78%
80%
82%
83%
At Le ast 1 Wellcare visit
72%
74%
76%
80%
Enrollment of Eligible Children
At Least 1 Well-care visit
22
22
Criteria for
Choosing Indicators
as Primary vs. Secondary Measures
Communication Power
Does the indicator communicate to a broad range of audiences?
Proxy Power
Does the indicator say something of central importance about the result?
Does the indicator bring along the data HERD?
Data Power
Quality data available on a timely basis.
23
23
Choosing Indicators
Worksheet
Safe Community
Outcome or Result_______________________
Candidate Indicators
Communication
Power
Proxy
Power
Data
Power
Measure 1
H M L
H M L
H M L
H
H
H
H
H
L
Measure 2
Measure 3
Measure 4
Measure 5
Measure 6
Measure 7
Measure 8
Data
Development
Agenda
24
24
Three Part Indicator List for each Result
Part 1: Primary Indicators
● 2 or 3 or 4 “Headline” Indicators
● What this result “means” to the community
● Meets the Public Square Test
Part 2: Secondary Indicators
● Everything else that’s any good (Nothing is wasted.)
● Used later in the story behind the baseline
Part 3: Data Development Agenda
● New data
● Data in need of repair (quality,timeliness etc.)
25
25
The Matter of Baselines
H
M
OK?
L
Point to Point
History
Turning the Curve
Forecast
Baselines have two parts: history and forecast
26
26
Performance
Accountability
For Programs, Agencies and Service
Systems
27
27
Results Accountability
is made up of two parts:
Population Accountability
about the well-being of
WHOLE POPULATIONS
For Communities – Cities – Counties – States - Nations
Performance Accountability
about the well-being of
CLIENT POPULATIONS
For Programs – Agencies – and Service Systems
28
28
“All Performance Measures
that have ever existed
for any program
in the history of the universe
involve answering two sets of
interlocking questions.”
29
29
Program Performance Measures
Quantity
Quality
How
Much
How
Well
did we do?
did we do it?
(#)
(%)
30
30
Program Performance Measures
Effort
How hard did we try?
Effect
Is anyone better off?
31
31
Program Performance Measures
Effort
How
Much
How
Well
Effect
32
32
Quantity
Quality
Input
Effort
How much
service did
we deliver?
How well
did we
deliver it?
Output
Effect
Program Performance Measures
How much
change / effect
did we produce?
What quality of
change / effect
did we produce?
33
33
Effect
Effort
Program Performance Measures
Quantity
Quality
How much
did we do?
How well
did we do it?
Is anyone
better off?
#
%
34
34
Effort
Education
Quantity
Quality
How much did we do?
How well did we do it?
Number of
students
Student-teacher
ratio
Effect
Is anyone better off?
Number of
high school
graduates
Percent of
high school
graduates
35
35
Effort
Education
Quantity
Quality
How much did we do?
How well did we do it?
Number of
students
Student-teacher
ratio
Effect
Is anyone better off?
Number of
9th graders who
graduate on time
and enter college or
employment after
graduation
Percent of
9th graders who
graduate on time
and enter college or
employment after
graduation
36
36
Effort
Health Practice
Quantity
Quality
How much did we do?
How well did we do it?
Number of
patients
treated
Percent of
patients treated
in less than
1 hour
Effect
Is anyone better off?
Incidence of
preventable
disease
Rate of
preventable
disease
(in the practice)
(in the practice)
37
37
Effort
General Motors
Quantity
Quality
How much did we do?
How well did we do it?
# of production hrs
Employees per
vehicle
produced
# tons of steel
# of cars produced
Effect
Is anyone better off?
Source: USA Today 9/28/98
# of cars sold
% market share
$ amount of Profit
Profit per share
$ car value after
2 years
% car value after
2 years
38
38
What Kind of PERFORMANCE MEASURE?
Upper Left
● # of people served
Lower Right
● % participants who got jobs
Upper Right
● staff turnover rate
Lower Left
● # participants who got jobs
Lower Right
● % of children reading at grade level
Upper Right
● cost per unit of service
Upper Left
Lower Right
● # applications processed
● % patients who fully recover
39
39
RBA Categories Account for All Performance Measures
(in the history of the universe)
TQM
Cost
Process
Input
Quality
Effort
Quantity
Efficiency Admin overhead, Unit cost
Efficiency,
Staffing ratios, Staff turnover
Staff morale, Access, Waiting time,
Waiting lists, Worker safety
Customer Satisfaction
Product
Output
Impact
Effect
(quality service delivery
& customer benefit)
Benefit value
Cost / Benefit ratio
Return on investment
Client results or client outcomes
Effectiveness
Value-added
Productivity
40
40
41
41
Effort
The Matter of Control
Quantity
Quality
How much did we do?
How well did we do it?
Most
Control
Effect
Is anyone better off?
Least
Control
PARTNERSHIPS
42
42
The Matter of Use
1. Fundamental Purpose is to Improve
Performance
as a contribution to improving results
2. Avoid the Performance Measurement
Equals Punishment Trap
● Acknowledge the experience as real.
● Work to create a healthy organizational environment
● Start small.
● Build bottom-up and top-down simultaneously.
43
43
Effort
Program Performance Measures
Quantity
Quality
How much
did we do?
How well
did we do it?
Effect
Is anyone
better off?
#
%
44
44
Connecticut Program Template
Connecticut Appropriations Committee RBA Template
Part II, Program/Agency/System Accountability Summary
Special Supplemental Nutrition Program for Women, Infants, and Children (WIC)
Program/Agency/System Purpose
The purpose of the WIC Program is to provide supplemental
foods, nutrition education and referrals for health and support
services to eligible, low income pregnant, postpartum and
breastfeeding women, and children 0-5.
Program/Agency/System
The Special Supplemental Nutrition Program for Women, Infants
and Children (WIC Program) / Department of Public Health,
Public Health Initiatives Branch
Performance Measure 1
The incidence of low birth weight (LBW) among infants whose
mothers were on the WIC Program for at least 6 months during
pregnancy does not exceed 6%.
Performance Measure 2
At least 50% of infants whose mothers were enrolled in the WIC
Program during pregnancy breastfeed.
Key Budget Information
Total Current Program Year Funding
Funding as Percent of All Funding
for Population Result
Program Funding As Percent of
Total Agency Budget
Funding Distribution
Total Federal Funds
Total State Funds
Capital Projects Subtotal
Other Funding
Percent of Total Current Funding
Contracted to Third Parties
Admin $9.38 M
Food $27.68 M
Admin 4%
Food 11%
$37.06 M
$0
$0
Admin 77%
Story Behind the Baselines
Participation in the WIC Program is associated with
Performance Measure 1: a reduction in rates of low
birth weight (LBW) and Performance Measure 3:
childhood anemia. It also is associated with better
cognitive performance. Early and continuous
enrollment in the program may result in further
improvement in LBW and anemia rates and, hence,
school readiness. Performance Measure 2: Children
who were breastfed score higher on cognitive and IQ
tests at school age. Breastfed infants experience fewer
and less severe cases of infectious and non-infectious
illnesses. Therefore, children who were breastfed are
more likely to be healthy and ready to learn.
Turning The Curves: What do you propose to
do over the next two years and why?
Performance Measure 3
The prevalence of anemia among children enrolled in the WIC
Program for at least one year does not exceed 10%.
1. Prepare and implement the USDA Value Enhanced
Nutrition Assessment (VENA) state plan to improve
client services; enhance local agency program
monitoring to improve program operations, nutrition
counseling and breastfeeding promotion.*
2. Contingent upon the receipt of additional funding,
conduct formative research to identify why
 eligible women do not routinely enroll in WIC
during their first trimester of pregnancy and
 eligible children leave the program before age 5,
and develop a targeted outreach initiative to address.
3. Contingent upon the receipt of additional funding,
increase WIC local agency staffing levels to the
state-recommended staffing pattern to enhance
outreach activities and nutrition and breastfeeding
education.
4. Contingent upon the receipt of additional funding,
replicate the existing model breastfeeding peer
counseling program in a selected WIC local agency.
*Indicates, low-cost, no-cost action steps, including reallocation
of existing resources.
45
45
How
Population
&
Performance Accountability
FIT TOGETHER
46
46
THE LINKAGE Between POPULATION and PERFORMANCE
POPULATION ACCOUNTABILITY
Healthy Births
POPULATION
RESULTS
Rate of low birth-weight babies
Stable Families
Rate of child abuse and neglect
Children Succeeding in School
Percent graduating from high school on time
PERFORMANCE ACCOUNTABILITY
Child Welfare Program
# Foster
Children
Served
% with
Multiple
Placements
# Repeat
Abuse/Neglect
% Repeat
Abuse/Neglect
CUSTOMER
Outcomes
Contribution
relationship
Alignment
of measures
Appropriate
responsibility
47
47
THE LINKAGE Between POPULATION and PERFORMANCE
POPULATION ACCOUNTABILITY
Healthy Births
Rate of low birth-weight babies
POPULATION
RESULTS
Children Ready for School
Percent fully ready per K-entry assessment
Self-sufficient Families
Percent of parents earning a living wage
PERFORMANCE ACCOUNTABILITY
Job Training Program
# persons
receiving
training
Unit cost
per person
trained
# who get
living wage jobs
% who get
living wage jobs
CUSTOMER
Outcomes
Contribution
relationship
Alignment
of measures
Appropriate
responsibility
48
48
49
49
What is Happening in the Legislature?
Institutionalizing RBA in the legislature:
•
The Appropriations Committee has created a new RBA subcommittee

Early Childhood Cabinet must develop an accountability plan
and make other recommendations for changes necessary to
ensure coordination, service integration, and accountability

The 2007 budget requires new and expanded programs to
report to the General Assembly and OPM using an approved
RBA framework, an effort lead by the Office of Fiscal
Analysis (OFA) and OPM

The co-chairs and rankings members of the other subcommittees of Appropriations are identifying particular
public policy issues (result statements) for RBA
development

All this month, the early childhood agencies that were part
of the RBA Phase II initiative last session reported back to
the legislature on the progress they have made
50
50
What is Happening at State Agencies
and in Communities?
The Child Poverty and Prevention Council
and the Juvenile Justice Policy and Operating
Coordinating Committee are both using RBA to
guide their implementation efforts

Municipalities and non-profit groups, e.g.,
workforce boards, are embracing RBA in areas
beyond early childhood

State agencies will be developing program
quality and outcome measures in response to
the new Appropriations Committee initiatives

They will be passing the requirements along
to their vendors

51
51
What is Happening at State Agencies
and in Communities?
• Organizations that contract with the State
will have new data collection and reporting
requirements, including outcome measures
• Managing performance will be more
complicated
• Likely to be greater requirements for
integration of programs and collaboration
with other agencies
• Opportunity for contractors to have input on
measurement
• Opportunity for program improvement
52
52
“If you do what you
always did,
you will get what you
always got.”
Kenneth W. Jenkins
President, Yonkers NY NAACP
53
53
Thank You
The Charter Oak Group, LLC
www.charteroakgroup.com
Ron Schack
ron_schack@yahoo.com
(860) 478-7847
Bennett Pudlin
bpudlin@charteroakgroup.com
(860) 324-3555
54
54
Turn the Curve Exercise: Population Well-being
5 min:
Starting Points
- timekeeper and reporter
- two hats (yours plus partner’s)
5 min:
Baseline
- forecast: Where is the trend line going?
- turn the curve: Is forecast OK or not OK?
15 min: Story behind the baseline
- causes/forces at work
- information & research agenda part 1 - causes
15 min: What works? (What would it take?)
- what could work to do better
- each partner’s contribution
- no-cost / low-cost ideas
- information & research agenda part 2 – what works
10 min: Report: Convert notes to one page
Two
pointers
to action
55
55
ONE PAGE Turn the Curve Report
Result: _______________
Indicator
Baseline
Indicator
(Lay Definition)
Story behind the baseline
-----------------------------------------------------
(List as many as needed)
Partners
-----------------------------------------------------
(List as many as needed)
Three Best Ideas – What Works
1. --------------------------2. --------------------------3. ---------No-cost / low-cost
4. ---------Off the Wall
Sharp
Edges
56
56