2012
Desiree Nangle, Evaluation Project Manager
Desiree.nangle@anglicarewa.org.au
SIMPLE
COMMON SENSE
PLAIN LANGUAGE
MINIMUM PAPER
USEFUL
With thanks to
The Fiscal Policy Studies Institute
Santa Fe, New Mexico
What is Results Based
Accountability?
• Starts with ends and works backwards to
means
– Ends are how customers are better off when
the program works the way it should
• RBA is meant to get you from talk to action
quickly
• Used for continuous program improvement
• Inclusive process
August, 2012
Results Based Accountability
is made up of two parts:
POPULATION ACCOUNTABILITY
Is about the well-being of
WHOLE POPULATIONS
For Communities – Cities – Counties – States - Nations
PERFORMANCE ACCOUNTABILITY
Is about the well-being of
CUSTOMER POPULATIONS
For Programs – Agencies – Services
7 Performance Accountability Questions
1. Who are our customers?
2. How can we measure if our customers
are better off?
3. How can we measure if we are
delivering services well?
4. How are we doing on the most
important of these measures?
5. Who are the partners that have a role to
play in boing better?
6. What works to do better, including lowcost and no-cost ideas?
7. What do we propose to do?
7 Performance Accountability Questions
1. Who are our customers?
2. How can we measure if our customers
are better off?
3. How can we measure if we are
delivering services well?
4. How are we doing on the most
important of these measures?
5. Who are the partners that have a role to
play in boing better?
6. What works to do better, including lowcost and no-cost ideas?
7. What do we propose to do?
Quantity
Quality
Effort
How much
service did
we deliver?
How well
did we
deliver it?
Effect
Performance Measures
How much
change / effect
did we produce?
What quality of
change / effect
did we produce?
Effort
Types of Measures Found in Each Quadrant
•
•
Quantity
Quality
How much did
we do?
How well did
we do it?
Number Clients
Number Activities
•
•
•
•
•
•
Client Satisfaction
Standards
Unit Costs
Wait List
Staff Quality
Targets
Effect
Is anyone better off?
•
•
•
•
Skills and Knowledge
Attitude and Opinion
Behaviour
Circumstance
Mums and Dads Forever
2012
Quantity
Quality
How much did we do?
How well did we do it?

Number of groups
Number of clients

Effort




No shows/Drop out rates
Bookings for intake
Intake group
Group to completion
Level of satisfaction Family Court
has with MDF
Staff satisfaction
Reconnects
Is anyone better off?

% of parents with more of a child focus
Effect
oassessed
using client self-reported perceptions in questionnaire
o
% of parents with less parent to parent conflict
o
% of parents with decreased litigation
oAssessed
via Family Court feedback questionnaire
o
% of clients seeking other services
o
% of parents with improved ability to co-parent
o
% of parents with improved self-awareness
Effort
Not All Performance Measures Are Created Equal
Quantity
Quality
How much did we do?
How well did we do it?
Least
Also
Very Important
Important
Effect
Is anyone better off?
Most
Important
Effort
The Matter of Control
Quantity
Quality
How much did we do?
How well did we do it?
Most
Control
Effect
Is anyone better off?
Least
Control
ONE PAGE Turn the Curve Report: Performance
Program:
_______________
Performance
Measure
Baseline
Story behind the baseline
----------------------------------------------------Partners
----------------------------------------------------Three Best Ideas – What Works
1. --------------------------2. --------------------------3. ---------No-cost / low-cost
4. --------- Off the Wall
Example of a Turn the Curve Report
Results Based Accountability
Service Description: Anglicare WA has chosen to implement Results Based Accountability (RBA) framework across all its services. This choice is
centred on a strong desire to know and have evidence the impact our services have.
Story Behind the Baselines
Headline Performance Measures
While all four baseline indicators increased from April 2012 to August 2012, some
had greater increases than others. While the percentage of staff understanding
RBA increased significantly (from 54% to 74%) since April, the increase in staff
supporting RBA use in Anglicare WA had a much less significant increase of only
2.5%. Part of the reason for the smaller increase is most likely due to the smaller
response rate on this survey (~25% of staff). The percentage of programs in Phase
two jumped considerably and progress was also made in moving services from
Phase two to Phase three since April (an 8% increase). This increase in services
moving to Phase three is most likely due to changes that were made in the
process based on suggestions given by Mark Friedman while he was here in June.
% of staff who understand what RBA is
% of programs in Phase 2 of RBA
implementation
% of programs in Phase 3 of RBA
implementation
% of staff who support RBA
implementation within Anglicare WA
Defined
Service
Users
All Anglicare
WA staff
RBA HEADLINE PERFORMANCE MEASURES
Action Plan to Continue Turning the Curves
In an effort to continue improving performance on our four headline
measures, Anglicare WA will:
• Revise the length of the survey and possibly how often it is being
sent out to staff in an effort to increase staff response rates
• Continue focusing on data that services do have to begin the
process of RBA implementation and work around helping them
develop tools to address each program’s data development
agenda
• Highlight success stories and curves turned to encourage a culture
of RBA monitoring among staff
Partners Who Can Help Us Do
Better
All Anglicare WA Programs and
Staff
The Community Sector
Other Not-for-Profit Organisations
National and Federal Government
Trained RBA Trainers
Data Development
Agenda
1. Determine population
outcomes our services
contribute to
2. Determine whether or
not there are
questions we can ask
across services that
use the same wording
and same scale
RBA on RBA
(roll out with services)
2012
RBA on RBA
Effort
Updated:
March 2013
Quantity
Quality
How much did we do?
How well did we do it?
• # of active services
by phases 19 in Phase 3
• # of staff involved in
active services by
phase
• staff satisfaction with 75%
training and support
• # RBA facilitators trained 7
• time to self-sustainability
• time programs spend in
each stage
• validity of measures
• validity of data
Effect
Is anyone better off?
• % staff that understand what RBA is 81%
• % staff that support RBA within Anglicare WA 66%
• % services implementing RBA (48 out of 61 services) =78%
• % services using RBA for program improvement 33%
• % services with the capacity to use RBA unassisted (1 service) = 2%
• % of curves turned
Do staff understand what RBA is?
December 2011: 53.6% Agree
April 2012: 68.3% Agree
August 2012: 73.5% Agree
March 2013: 80.9% Agree
2013
Do staff think RBA is good for
Anglicare?
December 2011: 55.2% Agree
April 2012: 60.4% Agree
August 2012: 66.3% Agree
March 2013: 66.1% Agree
2013
Results Based Accountability March 2013 Headline Measures Report
Service Description: Anglicare WA has chosen to implement Results Based Accountability (RBA) framework across all its services. This choice is
centred on a strong desire to know and have evidence the impact our services have.
RBA HEADLINE PERFORMANCE MEASURES
Story Behind the Baselines
Headline Performance Measures
% of staff who understand what RBA is
% of services in Phase 3 of RBA
implementation
% of staff who think RBA will help
improve their service
% of staff who support RBA
implementation within Anglicare WA
Action Plan to Continue Turning the Curves
In an effort to continue improving performance on our four headline
measures, Anglicare WA will:
•
…………
•
…………..
•
…………..
Partners Who Can Help Us Do
Better
All Anglicare WA Programs and
Staff
The Community Sector
Other Not-for-Profit Organisations
National and Federal Government
Trained RBA Trainers
Defined
Service
Users
All Anglicare
WA staff
Data Development
Agenda
1. Determine population
outcomes our services
contribute to
2. Determine whether or
not there are
questions we can ask
across services that
use the same wording
and same scale
Open-ended response
• Challenges of RBA
2012
• Opportunities with
RBA
Case Study
YES! Housing Project
YES! Housing
TURNING THE CURVE
How Much Did We Do?








How Well Did We Do It?
# of housed clients
# of referrals made by DCP
# of young people on the waitlist
# of allocations made by DoH
# of young people who disengage after being housed
# of days on waitlist
# of young people who complete the series of
workshops
# of clients that receive a certificate for completion of
ILS workshops









% of young people with case plans
% of young people who failed tenancies/eviction rate
% of qualified staff who have completed core training
including:
o Attending and participating in regular supervision
o Proper ways training
o TAS Training: Tenancy Law for Community Workers &
Advocating for Public Housing Tenants
o Other relevant training as identified in supervision
% of young people satisfied with the service
% of HSO from DoH who are satisfied with the program
% of clients referred to other agencies
% of waitlist population, housed population, and
mainstreamed population who are:
o Male/female
o Indigenous
o Parents
Unit Cost (cost/#of clients)
Referrals from DCP are regular and timely
Is Anyone Better Off?






% of clients who access safe, secure, and stable housing at the completion of case management
% of clients that demonstrate improved knowledge of ILS skills (i.e. home, work, education, parenting, tenancy and
healthy relationships)
% of clients that ustilse strategies developed through case management to address identified barriers to maintain
their housing
% of clients successfully linked to appropriate specialist services
% of clients regularly engaging with their support worker to develop their case plans
% of clients who achieve set goals
YES! Housing July – December 2012: RBA Headline Program Measures Report
Service Description: Anglicare WA’s YES! Housing program provides permanent housing for young people who are supported to establish independent living
through long term housing, including public housing properties.
RBA Headline Measures
Story Behind the Baselines
Graph 1: There continues to be an increase of young people on the waitlist trying to access safe, secure affordable
accommodation. Unfortunately YES! Housing does not have the resources necessary to meet this ever-growing need.
Graph 2: Data from graph 2 is based on the average difference from pre score to post score for the combined four ILS workshops
(health, tenancy, parenting skills and home skills). The total average increase went down 1% in this past 6 months, mainly due to
the health and relationships survey (deceased from a 31% increase Jan-June to 12% increase July-Dec). The Home Skills
workshop increased along with Tenancy Support while the Parenting Skills results decreased in the past 6 months.
Graph 3: 33.3% of clients with a case management plan mostly achieved or fully achieved their most important goal.
Graph 4: 96.8% of clients engaged have a case management plan.
Graph 5: 90% of clients self referred to the program or were referred through word of mouth. The remaining 10% were referred
from DCP and Community agencies.
Action Plan to Continue Turning the Curves
In an effort to continue improving performance on our headline measures, YES! Housing will:
• Research other options for content, presentation methods and implementation ideas from
clients participating in the Health & Relationship and Parenting workshops.
• Continue to implement the case management process to assist clients in achieving their most
important goals through regular outreach visits.
• Continue to create case plans with housed and waitlisted clients through implementing the
case management process and developing strategies to engage clients who have
disengaged from the program.
• Develop stronger networks with DCP, DoH, Community Organisations, other housing
organisations to offer support for their clients through case management process more
accommodation options for clients in other housing situations to maintain a successful
tenancy.
• Implement strategies developed in consultation with DCP during our recent Program Review
to access alternative affordable, safe, secure housing for clients on the YES! Housing waitlist by
establishing and developing relationships with private real estate.
• Review our current waitlist and remove ineligible clients.
Partners Who Can
Help Us Do Better
Internal partners including
Y-SHAC, Foyer, YPSG,
Youth Services Manager,
Fundraising
External partners including
Department of Housing,
Department for Child
Protection, Foundation
Housing, Community
Housing
Real Estate,YACWA,,
young people,
Clients
Young people 15-17 years who are
homeless or at risk of being
homeless.
Data Development Agenda
Yes! Housing will work on developing
and implementing new survey tools:
• to measure Independent Living Skills
knowledge increase that is only
post-workshop (not pre and post)
• to measure the Department of
Housing’s satisfaction with the work
done by YES! Housing
Four Phases of Implementation
Phase 2
Service Planning
Phase 3
Service Implementation
Phase 4
Normalise
• Scoping and readiness
assessment
• Workshop
• Define clients
• Agree outcomes
• Agree measures
• Data strategy
• Plot baselines
• Data collection
• Plot curves
• Monitor and use
• Refine
• Use unassisted
Effort
Awareness raising
• Emails
• Presentations to groups
•
•
Quantity
Quality
How much did
we do?
How well did
we do it?
•
•
•
•
•
•
number clients
number activities
client satisfaction
standards
unit costs
wait list
staff quality
targets
Is anyone better off?
Effect
Phase 1
•
•
•
•
skills and knowledge
attitude and opinion
behaviour
circumstance
ONE PAGE Turn the Curve Report: Performance
Program: _______________
Performance
Measure
Baseline
Performance Measure
Story behind the baseline
-----------------------------------------------------
(List as many as needed)
Partners
-----------------------------------------------------
(List as many as needed)
Three Best Ideas – What Works
1. --------------------------2. --------------------------3. ---------No-cost / low-cost
4. --------- Off the Wall
What if my
program doesn’t
show success?
Check out these
cool metrics!
I know I should
be measuring but
I don’t know how
What we do is an
not a science. It
can’t be measured
What does
the data say?
The Task (1)
Quadrant exercise: Performance
measures
• How much did we do?
• How well did we do it?
• Is anyone better off?
Effort
Types of Measures Found in Each Quadrant
•
•
Quantity
Quality
How much did
we do?
How well did
we do it?
Number Clients
Number Activities
•
•
•
•
•
•
Client Satisfaction
Standards
Unit Costs
Wait List
Staff Quality
Targets
Effect
Is anyone better off?
•
•
•
•
Skills and Knowledge
Attitude and Opinion
Behaviour
Circumstance