Social Sector Trials Evaluation Findings: Establishment Phase
Social Sector Trials background
Trialling New Approaches to Social Sector Change (the Trials) aims to
change the way cross-agency and community resources are used to
improve service delivery and outcomes for young people. Two models for
organising service delivery are being tested:
•
•
About the establishment phase evaluation
2011
Inputs
Tasks
2012
Outputs
Immediate
outcomes
2013
Intermediate
outcomes
Ultimate
outcomes
Establishment period
Purpose
The evaluation is being led by the Centre for Social Research and
Evaluation within the Ministry of Social Development in partnership with the
Ministries of Justice, Education, Health and the New Zealand Police. The
overarching evaluation aims to identify and assess:
The purpose of the establishment phase evaluation was to understand
what worked well, what did not and to provide evidence to enhance
decision making by key stakeholders.
any similarities or differences between the CI and NGO models
any enablers, barriers or challenges encountered
whether the Trials will lead to better approaches to service delivery
whether the Trials will contribute to improved youth outcomes.
Summaries of evaluation findings will be delivered at the conclusion of key
phases of the Trials initiative.
Sources of evaluation information
•
•
•
•
Many community organisations are already engaged
While the Trials are still in their early stages, results indicated most
community organisations had heard about them. Many organisations had
also met with CIs or NGOs about the Trials.
Survey results indicated the proportion of organisations working together to
deliver youth outcomes before the Trials started was very high. However,
there are opportunities to improve the quality of how organisations work
together, for example improving the coordination of services and
programmes.
No major initial differences between CI and NGO models
Evaluating the Trials
•
•
•
•
PAGE 1
Key findings
This A3 presents findings from the establishment period of the Trials.
Future evaluation activities will focus on progress towards key outcomes.
a Non-Government Organisation (NGO) model where an NGO is given
authority via contractual arrangements to deliver the Trials.
a Committed Individual (CI) model where a well regarded community
figure is employed as a public servant and delegated authority to deliver
the Trials.
July 2011
Individual and focus group interviews with the Trials Project Team, CIs,
NGOs.
Review of key documents.
Analysis of key themes from CI and NGO log books.
A survey of organisations in the six Trial locations conducted from late
May to early June 2011.
No major differences were observed between how the CI and NGO models
were being implemented or any initial impacts they were achieving. Some
differences were found in the characteristics of the Trial locations that may
influence how the Trials are implemented and the results they achieve.
Many barriers to implementation can be addressed
Government agency level actions that could enhance the delivery of the
Trials include standardising all or part of the contracting templates and
boundaries within and between agencies, and streamlining reporting and
decision making structures.
Trials specific actions include broadening the criteria for including contracts
in the Trials, and providing ongoing opportunities for community
stakeholders to have input into the Trials.
Results
Results are reported on: baseline levels and quality of collaboration between community organisations; stakeholder
engagement in the Trials; and enablers, barriers or challenges faced by stakeholders during establishment.
•
•
•
Survey results for collaboration were similar for CI and NGO locations.
Before the Trials almost all organisations reported working with others (96%).
Quality of collaboration was good but there is room for improvement in areas such as coordination,
dependability, and goal alignment.
Engagement in the Trials
•
Awareness of the Trials was high (81%) among community stakeholder organisations and over half (54%) had
already met with CIs or NGOs about the Trials.
Government
Collaboration between community organisations
Government level factors:
• existing mechanisms within government that facilitated the transfer of funding across government agency
appropriations
• other complementary government initiatives.
Community level factors:
• a history of action around youth issues
• existing community networks that helped to link CIs and NGOs with community stakeholders.
Trials level factors:
• development of systematic and transparent processes to prioritise options for Trials locations
• the autonomy and authority given to CIs and NGOs to carry out their tasks
• management of contracting relationships at a community level was seen by some providers as an opportunity
to improve services.
Trials
Enablers for establishing the Trials
Barriers/challenges for establishing the Trials
Solutions
Different funding and contracting systems across government agencies:
Created difficulties in identifying eligible contracts to be moved to the Trials
appropriation (e.g. some contracts were too complex to ‘unbundle’).
While the evaluation studies did not identify any solutions, parts of or all
contracting templates and boundaries (e.g. service delivery regions) could
be standardised across social sector agencies. This would help to identify
funding that can be transferred to the Trials appropriation and potentially
improve the ability to coordinate services across government in the future.
Complex decision making structures: Seeking high level decisions within
and between government agencies was time consuming and resource
intensive leading to delays in the establishment of the Trials.
Documentation indicates the establishment of the Joint Venture Board and
Trials Director is expected to strengthen collective cross-agency ownership.
This may reduce resource demands and speed up decision making for the
Trials.
Other competing government initiatives: In some instances the Trials were
seen as competing with other initiatives. For example, some regional service
providers participating in High Trust Contracting initiatives were now facing
dual lines of reporting.
CIs and NGOs reported meeting with relevant government agency
stakeholders to coordinate how initiatives were being delivered in locations.
Government agencies and providers involved with delivering other
government initiatives in Trial locations should also actively seek to
coordinate their activities with CIs and NGOs.
Changes in funding and contracting relationships: Some providers felt
threatened by the change in contract management to CIs and NGOs. In
some cases they were initially unwilling to engage with CIs/NGOs.
In some instances providers may face significant changes to their contracts
and their feeling threatened may be unavoidable. CIs and NGOs reported
managing relationships with affected providers was very important.
Scope of criteria used for contract selection: Fewer contracts were identified
for inclusion within the Trials appropriation than anticipated. This may impact
on the ability of CIs or NGOs to bring about change.
Key stakeholders suggested eligibility criteria for including contracts from
the partner social sector agencies could be revised and additional contracts
identified to increase resources available to the CIs and NGOs.
Keeping a low profile for the Trials during scoping: A need to establish the
Trials without drawing widespread attention within communities may have
affected the level of community stakeholder ownership and engagement in
the Trials.
To improve engagement CIs and NGOs reported meeting with community
stakeholders as early as possible in the planning process to introduce the
Trials and seek input into planning processes.
Social Sector Trials: Evaluation framework
PROCESSES
OUTCOMES
Inputs*
Tasks
Outputs
(Trials Project Team activities)
(CI and NGO activities)
(first six months)
Social sector governance
Engage key stakeholders
Stakeholder participation
Status
• Joint Venture Board and Trials
Director positions established
Enablers
• Strong leadership from Trials
Director
Enablers
• Existing community networks and
fora facilitating engagement
• Ability to link directly with youth and
government agencies
Status
• Forums, meetings and workshops
have been held with providers
• Survey findings indicated most (81%)
stakeholder organisations were
aware of the Trials and half (54%)
have participated in Trials meetings
Barriers/challenges
• Complex decision making
processes were time consuming
Barriers/challenges
• Some stakeholders unaware of the
Trials, the role of CIs or NGOs or
how they were recruited – potentially
leading to mistrust
Contracts and funding
Status
• Relevant contracts identified and
moved to MYD appropriation
Enablers
• Existing mechanisms enabled easy
transfer of appropriations
Barriers/challenges
• Differences in contract templates
and boundaries across government
agencies created challenges in
identifying what was in scope
• Scope of contract selection criteria
limited the number of contracts
identified
Locations identified
Status
• Six locations selected
Enablers
• Use of criteria to aid transparent
location selection process
Barriers/challenges
• Short time frame between agreeing
on the number of Trials locations
and having to recruit CIs and NGOs
Establish governance groups
Governance groups
Barriers/challenges
• Ensuring membership is balanced
• Ensuring a broad range of needs are
represented by board members
Status
• Terms of Reference have been
developed by CIs and NGOs
• Governance groups have been
established in each location
• In some locations working groups
have also been established
Develop Trials plans
Enablers
• Other government initiatives that
provide opportunities for
collaboration (e.g. CRM)
• National and local support to
participate in plan development
Barriers/challenges
• Competing government initiatives
• Relatively short time frames to
engage stakeholders and develop
draft plans
Trials plans signed and
actioned
Status
• At the time of writing this report
Trials plans were works in progress
and still in draft form
• Draft Trials plans included the four
youth outcomes and references to
improved collaboration
Recruiting CIs and NGOs
Status
• All CIs and NGOs recruited. Some
NGOs experienced delays in
recruiting project managers
Barriers/challenges
• Having to initially keep a low profile
meant the Project team was unable
to fully engage with communities.
This was linked to some
stakeholders feeling they did not
have a fair opportunity to apply for
CI or NGO roles
• Limited time to recruit, orientate and
train CIs and NGOs
PAGE 2
July 2011
Manage provider contracts
Enablers
• Providers willing to participate and
keen to improve performance
Barriers/challenges
• Trials specific activities within some
regional contracts had to be moved
to a separate contract leading to
dual reporting lines for providers
• Contracts being held by providers
who are not based in Trials locations
(e.g. a Hamilton based provider
holding a contract for services in
Taumarunui)
• Balancing time spent developing
Trials plans and managing contracts
Contracts aligned to Trials
plans
Status
• Expected to be included as part of
the Trials plans
Immediate**
Intermediate**
Ultimate**
Engagement with youth
outcomes
Baseline figures:
• Of the community organisations
surveyed most were working
towards improved participation in
education, training or employment
(77%)
• Over half were working towards
reduced alcohol and other drug
abuse (62%), youth offending (57%)
and truancy (53%)
Improved service delivery
for youth outcomes
To be explored in future evaluation
activities
Fit between services and
youth outcomes
Perceived needs:
• Improved coordination of service
delivery
• Sustainable family interventions
• Refocusing services from treatment
to prevention
• Workforce development
• Better engagement of communities
in planning
• More transparent use of resources
• Less resource being used on
administration
Improved action on the
causes of truancy,
offending, alcohol and
other drug abuse and nonparticipation in education,
training and employment
• Reduced truancy
• Reduced offending
• Reduced alcohol and
other drug abuse
• Improved participation
in education, training
and employment
See high level baseline outcome
indicator report
To be further explored in future
evaluation activities
To be explored in future evaluation
activities
Collaboration between
organisations
Baseline figures:
• Almost all (96%) organisations
reported they were collaborating
with other organisations
• A third were meeting with other
organisations at least weekly (29%),
a third at least once a month (31%)
• Local schools, Police and CYF were
the most common organisations
worked with
• Quality of collaboration was
generally good
• There may be scope to improve
areas such as alignment of goals,
frequency of joint planning,
coordination, and dependability
Perceived needs:
• Common perception among CIs and
NGOs that services were
uncoordinated and organisations
were not working together well
Evaluation Framework
• The evaluation framework is based on key expectations and actions for the
Trials. These were identified in the document review and stakeholder interviews.
• Findings on enablers and barriers have been mapped onto the inputs and tasks
of the framework.
• Baseline survey data and relevant stakeholder interview findings have been
mapped onto the outcomes.
• This framework will continue to be refined as new information becomes available.
Notes
* Only those inputs that were identified during the course of the evaluation have
been identified and this report may not reflect all inputs.
** This report focused on the Trials establishment period (inputs and tasks). Only
baseline information for outcomes has been provided.