Social Sector Trials Evaluation Summary: Implementation Phase - April 2012
The Trials Evaluation
Findings: Service Delivery
Background
What has happened?
Issues for young people
Trialling New Approaches to Social Sector Change (the Trials) tests the
ability of a Committed Individual (CI) or lead Non-Government Organisation
(Lead) to use cross-agency and community resources to improve
outcomes for young people aged 1218 in six locations: Levin, Gore,
Kawerau, Te Kuiti, Taumarunui and Tokoroa.
Trials locations have put significant effort into planning and delivering new and
innovative youth services. Each location has engaged members of their community in
the Trials, raised the profile of young people and the issues they face, delivered
collaborative initiatives and created new roles to work directly with young people.
Interview participants consistently identified the following factors as compromising
young people’s ability to help themselves:
Evaluation of the Trials
Improved service development and more services for young people
Findings from the establishment evaluation indicated that there were opportunities to
improve the quality of collaboration between youth service providers. Participants from
the current evaluation believed the way organisations were working together had
improved in a number of key areas, including:
2011
Implementation Phase
Evaluation
Implementation
evaluation
phase
Inputs
Tasks
Outputs
2012
Immediate
outcomes
2013
Intermediate
outcomes
•
•
•
•
•
•
What has changed?
The Centre for Social Research and Evaluation, within the Ministry of Social
Development, leads the evaluation in partnership with the Ministries of Justice,
Education and Health, and the New Zealand Police.
The evaluation is divided into three phases:
• Establishment Phase Evaluation: reported in July 2011
• Implementation Phase Evaluation: this A3, covering July to December 2011
• Final Phase: reporting progress towards outcomes, to be reported in 2013
Findings: Young People
•
“It is hard for young people to maintain positive change because there may be only one high
school and they can’t easily change their circle of influence.”
(NGO interviewee)
planning and development of youth services so they are more integrated,
collaborative and innovative
better knowledge about what services were available for young people and the best
approach to achieve positive change
less duplication of services for young people
increased community buy-in and community participation (including young people) in
youth service planning.
•
•
•
Ultimate
outcomes
What has happened for young people?
All locations have established, or were in the process of establishing, new youth
coordinator roles. These roles were different in each location, and included:
•
•
•
•
“… This gives responsibility to us and is not dictated from on high so the enthusiasm generated is
ours, we are helping our own.” (Local Government interviewee)
Figure One: Evaluation Phase
Purpose of the Implementation Phase Evaluation
The purpose of this phase of the evaluation was to document progress to date
(including tasks and outputs), barriers and opportunities encountered, and identify
immediate outcomes of the Trials.
Many evaluation participants reported that there were already more services available
for young people now than before the Trials and this would continue to improve.
Other actions to address improved youth outcomes have been implemented in
the Trials locations. Examples of these include:
“There was little available before the Trials started that wasn’t sport focused but now we are
developing a wider approach and more opportunities are presenting.”
(NGO interviewee)
•
Confidence the Trials will make a difference
Evaluation participants were largely optimistic about the Trials and the anticipated
gains the Trials would bring to the young people in their community. They described
early successes that buoyed their confidence that the Trials will make a positive
difference. They also recognised that the Trials were aiming to address a lot of
deeply embedded difficulties for young people and change would take time.
Improved leadership and planning processes
Participants viewed the Trials governance groups they participated in as highly
motivated and productive. They considered the planning process for youth service
delivery to be better informed, more integrated across the youth sector and more
innovative now, than it was prior to the Trials.
Better understanding of young people
New relationships through the formation of the governance groups had increased
participants’ understanding of the young people in their community, including what
strategies were likely to be effective for sustaining positive changes for them.
No differences between CI and NGO model outcomes
No major differences were observed in this evaluation, between what the CI and
NGO models were achieving. Some differences in the enablers and barriers each
model faced were emerging and will be investigated more closely in the final phase
evaluation, for example the importance of collegial support between CIs and how
lead NGOs engage other NGOs in service delivery.
File Reference
A588415
Date
4 April 2012
•
The following barriers to integrated youth services in the Trials sites were identified :
•
•
High Level Findings
•
Barriers
•
Not all service deliverers were engaged in the Trials (most areas had one key NGO
inadequately engaged, or disengaged, from the Trials).
Regional representation from some government agencies was sporadic due to the
large geographic area some government representatives covered.
Government agency input was sometimes limited by their internal bureaucracy, for
example: they had siloed approaches, and agencies had different geographical
boundaries from each other.
Positive Enablers
intensive mentoring roles with at-risk young people
School and community-based youth workers
activity-focused youth worker roles
youth participation coordinator (focused on engaging young people in Trials).
“Young people told us there was no one to talk to, they lacked a sense of belonging... The
Youth Coordinators are going to make a remarkable difference.”
(NGO lead interviewee)
Method
Findings reported in this A3 were drawn from:
• individual interviews, in four of the six Trials sites (Te Kuiti, Tokoroa,
Taumarunui and Levin) with Trial leads and key stakeholders - mainly
governance group members
• a short pre-interview questionnaire administered to the interviewees
• a review of key documents.
dysfunctional family environments
drug and alcohol use and addiction
lack of education, training and employment (ETE)
lack of specialist mental health services
lack of transport to out-of-town ETE and specialist mental health services
lack of positive young role models.
•
•
a day programme established for 20 young people previously not attending
school or alternative education
youth hubs established and in development to provide appropriately
supervised venues and events for youth
part-time holiday employment in local businesses secured for 12 young
people
a youth mentor employed to provide intensive mentoring with 30 at-risk young
people
a system for using community service to enable young people to pay
outstanding fines.
What has changed?
Ratings from the pre-interview questionnaire with evaluation participants showed
they had considerably more confidence that the Trials would deliver improved
youth outcomes for at-risk young people now than they had before the Trials.
Trials leads were seen as skilled agents of change
Trials leads were described as integral to the success of the Trials. In particular they:
• effectively mobilised key people in the community to get involved
• actively planned and delivered services for young people
• formed extensive networks of people and helped build bridges between them.
Potential Barriers to Longer-term Impact
Issues raised as potentially reducing Trials success in future included:
• lead NGOs becoming overstretched and unable to deliver at the same intensity
for young people overtime
• other government initiatives superseding the Trials and taking the focus away
from young people.
Influential governance groups and resources
Having influential decision makers within governance groups was viewed as critical to
gaining traction for youth within the town. Specifically, influential local governance:
• gave the Trials status
• helped leverage additional resources to allow quick Trials-related actions.
Potential Enablers of Longer-term Impact
Government and NGO support
All levels of government and NGO support were described as enabling Trials success,
specifically:
• Local government helped elicit community engagement and local buy-in to the Trials.
• Regional government agency representatives were informative and could mobilise
significant resources, including people.
• The Trials Directorate and Ministerial championship helped maintain the impetus of the
Trials and resolve issues pertaining to central government input.
• NGOs contributed key knowledge about what they delivered for young people and how
to make a difference for them.
Interviewees identified the following elements as important for sustaining longerterm change:
•
•
•
ensuring an alignment in each Trials location between the vision of the Trials
and what local leaders want for their town
local leaders’ commitment to long-term change for their young people
governance groups made up of people from within the community who were
passionate about young people and able to influence improvements for them.