Lessons on replication and evidence-based interventions

advertisement

Lessons on replication and evidence-based interventions: Mid-programme report

In 2013 over 23,000 young people entered the criminal justice system in the UK for the first time. Each occurrence has a significant impact on the well-being of the young person, their family and the wider community as well as considerable financial implications.

It was in this context that the Big Lottery Fund launched a £25m programme in 2012 aimed at improving the evidence base of what works to prevent youth offending. The Realising Ambition programme is providing grant funding alongside specialist support to replicate evidence-based interventions proven to reduce the likelihood of a young person’s involvement in the criminal justice system. It is also supporting delivery organisations to refine and build the evidence base of promising interventions.

The programme aims to:

• improve the evidence base of what works for whom and why in avoiding pathways into offending;

• promote learning about what it takes to replicate evidence-based interventions;

• help commissioners to ask the right questions about evidence, practice and impact.

The Realising Ambition programme is running from 2012 to 2017. During this time 22 delivery organisations will serve over 130,000 young people in

132 replication areas. They will be replicating 25 interventions, 10 of which are defined as evidence-based and 15 as promising. The portfolio includes school-based, family, community and therapeutic interventions. Each targets risk factors associated with youth offending, such as anti-social behaviour,

03

What is an intervention?

An intervention is a service delivered by an organisation designed to improve outcomes for children and young people. This may be an intervention designed and implemented within the

UK or it may be an evidence-based intervention imported from overseas.

The intervention may be distinct from the organisation that delivers it (that is, the intervention may have been designed and delivered by a different organisation to the one seeking to replicate it).

What is a promising intervention?

An intervention is ‘promising’ when it is tightly defined, underpinned by a solid theory of change and has some indicative

– though not experimental – evidence of impact. It is their welldefined nature and committed journey to evidencing impact that makes promising interventions stand out from the crowd.

What is an evidence-based intervention?

An evidence-based intervention is a discrete organised package of practices or services – often accompanied by implementation manuals, training and technical assistance support – that has been tested and found effective. Tested means it has been put through its paces by a rigorous experimental evaluation, comparing the outcomes of those receiving the intervention with those who do not. Effective means that that it has a clear and positive effect on child outcomes.

What is replication?

Replication involves delivering a proven intervention into new geographical areas or to new/ different audiences of children and young people. It is distinct from scaling-up, which is simply increasing the number of children and young people currently being served.

drug and substance misuse, and difficulties at school. As well as receiving support to replicate, four of the organisations are undergoing an experimental evaluation through randomised controlled trials (RCTs) to show whether the interventions have a positive outcome significantly greater than normal services.

The programme is led by Catch22 with the Social Research Unit, Substance and the Young Foundation as consortium partners. More information is available online at catch-22.org.uk/realising-ambition .

05

Figure 1: Realising

Ambition replication areas

1

1

2

2

5

1

4

2

3

4

During the course of the programme 22 delivery organisations are working across the UK, from

Aberdeen to Penzance, to deliver their interventions to over 130,000 children and young people.

Each number on this map refers to the number of interventions being delivered in this area

Realising Ambition is a programme of national and international significance due to its UK-wide nature and the scale of its delivery, as well as its potential to transform the way in which future provision is commissioned, delivered and managed.

1

1

3

1

1

2 3

1

2

2

1

1

4

1

1

2

1

1

1

2

1

2

3

2

1

1

1

1

1

2

1

2

1

1

1

2

2

4

1

4

2

1

1

1

1

1

1

1

1

1

1

1

1

10

1

2

07

Realising Ambition delivery organisations and interventions

Evidence-based interventions

Action for Children: Functional Family Therapy

Intensive, short-term family intervention for young people with conduct disorders and who are experiencing family conflict, displaying violent behaviour, at risk of offending, or on the edge of care.

Action for Children - Gweithredu dros Blant : Roots of Empathy

An intervention aimed at decreasing aggressive behaviour and improving empathy through focusing on caring for a baby.

Ambition (Clubs for Young People): Lions Quest Skills for Adolescence

Personal and social development programme to help young people make informed decisions and resist peer pressure, including the pressure to use alcohol or drugs.

Barnardo’s Northern Ireland: Life Skills Training (Botvin)

Universal evidence-based model of intervention to prevent substance misuse and youth violence in a personal and social skills setting.

Barnardo’s Northern Ireland: Fast Track/PATHS Plus

In-school service delivered by teachers focusing on social skills and emotion recognition.

Barnardo’s Northern Ireland: All Stars

School-based prevention programme addressing substance misuse and other high-risk behaviours delivered by teachers as part of the school curriculum.

Extern: Multi-Systemic Therapy

Intensive intervention to reduce out-of-home placements, school exclusions, substance misuse and family conflict.

North Bristol NHS Trust – Be Safe Service: Be Safe Service

Intervention to address problematic sexual behaviour, supporting both the child and their parent/caregiver.

Oxford Brookes University: Strengthening Families Programme 10-14 (UK)

Intervention to prepare children for adolescence and prevent pathways to offending.

Success for All-UK: The Co-operative Primary School

Professional development and materials to support Key Stage 2 children with socialemotional learning.

Promising interventions

Anne Frank Trust UK: Anne Frank Schools – Ambassadors Programme

Developing social and communication skills and challenging prejudice and hatred of others.

Ariel Trust: It’s not OK (INOK)

Resources for teachers to deliver intensive, issue-based education tackling a range of social issues.

Bang Edutainment: Strength 2 Strength

Intervention for young people and their families focusing on minimising risk factors that can encourage offending or anti-social behaviour.

09

Bridge Foundation: Stepping Up (Youth in Transition)

Intervention for vulnerable children making the transition from primary to secondary school.

Chance UK: Early Intervention Mentoring

Mentoring for children with behavioural difficulties that encourages them to work towards behavioural, educational and skills goals.

Children's Parliament: Children's Parliament Community Initiative

Intervention to effect a cultural shift in communities and promote human rights and pro-social relationships among young people on the periphery of anti-social behaviour or gang involvement.

Kidscape: Positive Assertive Confidence Strategies (PACS)

Identifying and targeting young people at risk of becoming bullies or engaging in criminal behaviour to prevent escalation and harm to others.

Malachi Community Trust: Early Intervention and Family Support

Holistic intervention to reduce the risk of anti-social behaviour and improve life chances of children from dysfunctional families who may have acute needs and issues including neglect and low attainment.

Respect: The Respect Young People's Programme

Intervention for families experiencing or propagating aggressive behaviour to prevent progression to offending.

11

Shelter: Realising Ambition

Seeks to divert 8-14 year olds away from pathways into offending and prevent actual and risks of homelessness.

Remedi: Safer Schools Partnership (Restorative Schools)

Intervention to address conflict in school, reduce exclusions and mediate restorative justice.

Trelya: Friends of the Children

Professional mentoring for vulnerable children in areas of significant social and economic deprivation to develop intellectual and emotional well-being.

Winston's Wish: SWITCH

Bereavement support intervention to help vulnerable children manage difficult feelings.

Working With Men: Conflict resolution – Uncut

Intervention for boys to increase awareness of alternatives to resolving conflict through violent means and prevent pathways into offending.

YMCA Scotland: Plusone Mentoring

Mentoring service to divert high-risk young people away from the criminal justice system, change behaviour and increase aspirations.

13

Our learning so far

Since the publication of the year one report five features have consistently emerged as characteristics that support an organisation to replicate effectively. Under each feature is a number of key learning points.

These are of particular relevance to those interested in investing in replication as well as delivery organisations wishing to replicate an intervention.

A successful Realising Ambition organisation...

01

Has a tightly defined and replicable intervention: the organisation is replicating an intervention that is underpinned by a strong logic model, with a clearly specified target group and set of activities.

Key learning points

• Intended beneficiary groups, clear eligibility criteria and referral routes should be established prior to delivery. This can help to significantly reduce the level of early attrition.

• Organisations should have a clear understanding of the core components of their intervention – the things that cannot be changed.

• Processes should be in place to explore the need for adaptation to fit local context and to manage any potential adaptation in a controlled way.

15

organisation bases its intervention on evidence of what works to improve outcomes and generates new evidence by monitoring and evaluating its activities.

Key learning points

• High-quality impact reporting should become routine, with the quantity and quality of data agreed between delivery organisations, commissioners and funders before delivery commences.

• Sufficient capacity should be built into interventions to allow for monitoring and reporting on the impact the intervention is having on beneficiaries.

• The strengths of both quantitative and qualitative information should be valued, and both used to demonstrate the effectiveness of an intervention to commissioners and funders.

demonstrate its ability to deliver against expectations and achieve

quality using performance management tools.

Key learning points

• Performance management, quality assurance and knowledge management systems should be initiated at the start of an intervention programme to effectively monitor delivery, intervention quality and outcomes.

• Using formal processes such as service level agreements can support the development of strong relationships between delivery partners.

• Highly skilled staff are key to successful delivery; recruitment and selection processes can be included in intervention handbooks to guide their selection.

draws on evidence of impact and realistic business modelling to support its future sustainability and the replication of the intervention.

Key learning points

• Building sustainability into an intervention from the outset is key to successful replication and organisational health.

• Sustainability and succession planning should be evidenced and tested by commissioners, rather than simply acknowledged, at application stages.

05

Is committed to continuous learning and improvement: the organisation

is committed to reflecting and learning from evidence in order to

improve its organisational structures to support delivery.

Key learning points

• Continuous learning may mean some aspects of an intervention need adaptation during its delivery.

• The learning process is supported both by developing a culture that emphasises reflection on practice and by having formal processes for delivery partners to share information and resources.

17

Looking to the future

Realising Ambition provides a significant opportunity to consider the implications of replicating evidence-based interventions for those organisations delivering them, those who commission or fund them, and those in a position to evaluate and manage them. Programme learning so far has focused on exploring what the key features are of organisations that replicate successfully. However, there are a number of key questions and issues beginning to emerge that we do not yet have answers to. These include:

• how to best find the balance between fidelity and adaptation;

• what types of interventions require more adaptation than others;

• the receptivity of commissioners to replicating evidence based interventions;

• what types of replication model best enable organisations to replicate successfully;

• what impact the interventions will have on children, young people and families, and why;

• how transferable learning from Realising Ambition is within an organisation.

These questions will continue to be explored during the remaining half of the

Realising Ambition programme through future reports from the consortium, the data being reported by projects, the programme level evaluation by the

Social Research Unit and the process evaluation by the Tavistock Institute of

Human Relations.

Find out more

You can find out more about Realising Ambition by contacting Catch22 or any of the consortium partners listed below.

Catch22

Contact: Ruth Marriott

Ruth.Marriott@catch-22.org.uk

catch-22.org.uk/Realising-Ambition

Social Research Unit

Contact: Bethia McNeil bmcneil@dartington.org.uk

dartington.org.uk/projects/realising-ambition

Substance

Contact: Neil Watson neil@substance.coop

substance.coop/current_projects/Realising_Ambition

Young Foundation

Contact: Tricia Hackett tricia.hackett@youngfoundation.org

youngfoundation.org/projects/realising-ambition

19

Download