Demonstrating outcomes in
youth work: challenges and
possibilities
Dr John Bamber
© The Centre for Effective Services 2011
Aims for the input
1. To explain the concept of evidence in a youth work
context
2. To distinguish between the terms ‘evidence-based’ and
‘evidence-informed ‘
3. To discuss the possibilities afforded by an evidence
informed approach to youth work, involving a theory of
change, and a logic model
4. To set out the challenges involved for different actors
1
Why evidence?
The Age of RESULTS and ACCOUNTABILITY
Key question: What difference does youth work make?
2
http://www.campbellcollaboration.org/
Is there a hierarchy of evidence?
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
Systematic reviews
1-4 ‘evidence based’ =
proof of ‘what works’?
Randomised control trials
Quasi-experimental study
‘Before’ and ‘after’ designs
Retrospective designs (no baseline)
Independent large scale evaluations
Traditional literature review
Independent small scale evaluations
Independent single studies
Opinion pieces – peer reviewed journals
Expert panel/committee
Internal evaluation – large or small scale
Practice wisdom – policies, standards, manuals
Personal experience – reflective practice
Objective?
Subjective?
3
http://www.colorado.edu/cspv/index.html
Evidence based programmes
http://www.promisingpractices.net/
http://whatworks.uwex.edu/Pages/2evidenceregistries.html
4
An evidence informed approach
Professor Sandra Nutley, University of Edinburgh
CES Practice Seminar Series, October 2010
Keynote address
Evidence-informed practice: Using research
to improve services for children and young
people
http://www.effectiveservices.org/ces-projects/p012seminar-series-practice-issues
5
Knowledge required is more than ‘what works’
• Know-about (problems): e.g. the nature, formation, and
interrelations of social problems.
• Know-why (requirements of action): explaining the
relationship between values and policy and practice.
• Know-how (to put into practice): e.g. pragmatic
knowledge about service and programme
implementation.
• Know-who (to involve): e.g. care teams; building alliances
for action.
Sandra Nutley (2010)
6
Evidence informed is...
An approach that helps people and organisations make
well-informed decisions by putting the best available
evidence at the heart of practice development and
service delivery.
Sandra Nutley (2010)
7
Good outcomes depend on effective action
Local decision-making, and the
harnessing of local initiative,
imagination and adaptation can be
pre-eminent, but are much more
likely to lead to effective action
when combined with a broad
collection of accessible knowledge
that is drawn from practice and
theory as well as research (Schorr,
L., 2003: 21-21).
http://www.lisbethschorr.org/4.html
8
http://www.theoryofchange.org/about/what-is-theory-of-change/
Constructing a theory of change
Consultation
Research
and
Theory
Practice
Policy
9
http://www.indefenceofyouthwork.org.uk/wordpress/wpcontent/uploads/2010/11/20252-Youth-storiesreport-2011_4th-1.pdf
Evidence from practice
• Value led and educative
• Dialogical and empowering
• Deliberative and
responsive
• Critical and reflective
• Process and results
focussed
10
Evidence from theory – a selection
Psychology:
Erikson
Psychosocial development
Education:
Piaget
Schemas, stage theory
Experiential Learning:
Kolb
Learning cycle
Learning styles
Multiple intelligences
Gardner
Communication:
Berne
Transactional analysis
Constructivism
Bandura
Reciprocal determinism
Action learning
Revans
L=P+Q
Political theory:
Marx
Class struggle
Theory of knowledge:
Habermas
Communicative action
Social change:
Freire
Praxis
11
Institute of Education, London. (March, 2012) International review of research literature on youth
development . 2012. Commissioned by Centre for Effective Services on behalf of Department of
Children and Youth Affairs, Dublin.
Evidence from research…a selection
McKee, V., Oldfield, C. and Poultney, J. (2010) The Benefits of Youth Work. Lifelong Learning UK and Unite.
Wiggins, M., Bonell, C., Burchett, H., Sawtell, M., Austerberry, H., Allen,E. and Strange, V. (2008) Young
People’s Development Programme Evaluation: Final Report. Social Science Research Unit, Institute of
Education and University of London with the London School of Hygiene and Tropical Medicine.
Bielby, G., Purdon, S., Agur, M., Gardiner, C., George, N., Golden, S. and Taylor, E. (2009) Empowering
Young People Pilots (EYPP) Evaluation, Final Report. National Foundation for Educational Research.
Lewis-Charp, H. Hanh C.Y., Sengouvanh S. and Lacoe, J. (2003) Extending the Reach of Youth Development
Through Civic Activism: Outcomes of the Youth Leadership for Development Initiative. Innovation Center for
Community and Youth Development.
Eccles, J. and Appleton Gootman, J. Editors (2002) Community Programs to Promote Youth Development
Committee on Community-Level Programs for Youth, National Research Council and Institute of Medicine.
Washington, DC: National Academy Press.
Fullerton, D. And Burtney, E. (2010) Young People’s Health Community Based Approaches, A Rapid Review.
Report to the Sexual Health Centre Cork. Insights Health and Social Research.
12
Evidence from policy
13
A common problem is that
activities and strategies often
do not lead to the desired
outcomes.
“I think you should be more
explicit here in Step Two.”
This is where evidence from consultation
comes in…
14
Evidence from consultation
Goals
Values
Needs and issues
Consultation:
participants
peers
experts
What are the needs, issues and interests?
What changes would people like to see?
What difference have we made, and has
anybody benefitted?
Outcomes
15
Adapted from: Devlin, M. and Gunning, A. (2009) The Purpose and Outcomes of Youth Work
Report to the Youth Services Interagency Group. Dublin: Irish youth Work Press
Proximal outcomes
Outcome Area
Sample outcome statements
Attitudes, beliefs
Participants are more ready to take on new and more diverse experiences
Enhanced confidence, self-esteem, awareness (personal and social)
More open to people from diverse backgrounds
Knowledge
Increased knowledge of local area, of society
Increased knowledge of rights, social issues, politics
More informed about health, sexuality, the law, careers and formal education
Skills
Enhanced capacity regarding: public speaking , problem solving, self-efficacy, making
decisions, critical thinking
Enhanced interpersonal abilities in relation to: teamwork, group work, communications
In creased abilities in arts and creativity regarding: music, dance, drama, writing
Increase in physical competence in relation to: sports, games, outdoor activities
Behavior
More engagement in structured and constructive activities
Increased involvement in decision-making
Enhanced positive and pro-social behaviour and diminishing negative and anti-social
behaviour
16
Adapted from: Devlin, M. and Gunning, A. (2009) The Purpose and Outcomes of Youth Work
Report to the Youth Services Interagency Group. Dublin: Irish youth Work Press
Distal outcomes
Outcome Area
Sample outcome statements
Education
Deeper engagement with formal education and alternative routes to
qualifications
Relationships with
adults
Improved communications and relationships between young people and
adults within communities
Health
Safer and more enjoyable sex, reduction in alcohol and substance use
Social conditions
Increased community cohesion, improvement in local conditions and
amenities, developing local leadership
Economic conditions
Increased self-sufficiency
Safety
Reduction in violence, vandalism, offending rates
Service provision
Increased support for homeless young people,
Improved cooperation and coordination between local services
17
Chain of outcomes - example
Short
Medium
Long-Term
Young people recognise
the benefits of engaging
in informal learning, and
spend more time in
structured developmental
activities
Enhanced sense of
capabilities regarding
community involvement,
education, training
Learning skills acquired
and transferred to other
areas of life eg work,
family life, politics
Young men more able to
articulate their needs and
wants
Enhanced political skills
Young people friendly
through their involvement environment , and safer
in decision making in
community for residents
youth work settings and
the wider community
18
Well defined information which shows whether
something is happening.
Outcomes and indicators - examples
Outcome
Indicators
Young people in the diversion
programme less likely to offend.
#,% Youth who will not be charged with a
new crime 1-6 months after successfully
completing the program. #,% Youth who will
not be charged with a new crime 7-12
months after successfully completing the
program.
Participants better able to manage
conflict in relationships
#,% Participants demonstrate better
knowledge of anger management in pre
and post program surveys.
Improved relationships and closer
family ties
#,% Parents and young people indicate
improved communication at three month
follow-up.
19
Outcomes and indicators - sources
http://www.urban.org/center/cnp/Projects/outcomeindicators.cfm
http://www.ces-vol.org.uk/
http://www.ceni.org/publications/research.asp
20
Monitoring and evaluation
Goals
Values
Needs and issues
Strategies
Inputs
Process
Outputs
Outcomes
I
N
D
I
C
A
T
O
R
S
Evidence
21
Logical Chain of Connections
Showing What the Program is to Accomplish
Situation Analysis
Monitoring and Evaluation
Goals derive from
Aim, mission, vision, values,
ideology
Collecting and interpreting information about implementation and results, and reporting findings
These processes take external factors into account and inform analysis, planning and implementation
Targets and Indicators that can provide signs of progress or achievement, may derive from standards and
benchmarks
Goals inform analysis
Populations and target groups
Problems and causes,
Situations and issues
Current provision: strengths,
weaknesses, gaps
Implementation
Resources
Inputs (matched to
Funding
Buildings
Technology
Partners
Staff and volunteers
Process
Strategies distil and give
focus to goals
Outline of main ways to
achieve outcomes
Outcomes are cumulative
changes
strategies)
Socio-economic and
organisational context affect
outcomes
Policy, legislation, funding,
politics, economy, culture, local
conditions
Activities
Value led and educative
Dialogical and empowering
Deliberative and responsive
Critical and reflective
Process and results focussed
Outputs are key areas of
work that enable desired
outcomes
Who we will reach
What we will do
Where, when and how it
will happen
Targets and numbers to
be reached
Frequency of activities
Standards to be achieved
Benchmarks
Short term
Gains in knowledge, skills,
awareness
Changes in attitudes
Changes in individual and group
behaviour – social action
I
N
D
I
C
A
T
O
R
S
Medium Term
Organisational development
Change in local situation and
circumstances
Change in policy, decisionmaking, practice
Long term
Social change – health,
education, civic, environment...
Evidence informs all aspects of the logic model
Taking ideas from social scientific research, literature, practice wisdom, policy and consultation processes, to inform understandings of problems,
situations and issues, as well as ideas about work that can enable desired outcomes and ways of monitoring and evaluating the work
22
22
Sample logic model
Monitoring and Evaluation
Implementation
Intentions
Organisational purpose
To develop young people personally,
socially, educationally and
recreationally in a high-quality, safe
environment that respects its users.
Analysis (includes consultation)
Issues to be addressed:
1. Young people’s psycho-social
development
2. Social change
3. Sexual health
Strategies
1. Adventure activities
2. Social action project
3. Information, advice, training
Resources
Activities
Outputs
Inputs (matched
Short
Results
Medium
Long term
Outcomes
to strategies)
Adventure:
8 linked weekend challenge
courses over 18 months
(includes personal and group
learning inventory) for 20
participants
Staff
Volunteers
Funding
Buildings
Technology
Partners
Process
Value led and
educative
Dialogical and
empowering
Deliberative and
responsive
Critical and reflective
Process and results
focussed
Social action project:
Over 12 months on an issue of
importance to young people
Series of monthly films, talks and
discussions on social issues
Information, advice, training:
Trial (x2) Teen Talk programme
(2 evenings per week for 2-3
hours, over 6 weeks)
Plan for further work
Evidence
•Participants are more ready to take on
new and more diverse experiences (S)
•Enhanced interpersonal skills (M)
•Learning skills acquired and
transferred to other areas of life: work,
family life, politics (LT)
•Increased involvement in decisionmaking processes (S)
•More informed decision-making (M)
•Local services for young people are
more user friendly (LT)
•More aware of harmful behaviours to
self and others (S)
•Risk factors reduced (M)
•Safer sexual practices (LT)
23
Challenges of an evidence informed approach
•Youth workers
•Managers
•Policy makers
•Researchers
24
Challenge for youth workers
• Assumptions underlie much of what we do
• Faulty assumptions are often the reason for poor
results – e.g. ‘process is all there is’
• Evidence helps us make our assumptions explicit
so that we can check and test them: espoused
versus theory-in-use
• Reflective practice (Schön, 1983)
http://www.infed.org/thinkers/et-schon.htm
25
Challenge for line managers
• Trust versus command and control
• Monitoring and reporting versus support, training
and development
• Creating spaces for reflection
• Promoting a culture of evidence
• Underpinning own practice with evidence
26
Fixsen, D. et al (2011)
http://www.implementationconference.org/materials/plenary
The challenge for policy makers
Performance Assessment
Systems
Intervention
Coaching
Facilitative
Administration
Training
Selection
Core
Implementation
Components
Decision Support
Data System
Leadership
Technical
Adaptive
27
Challenge for researchers
• Providing policy makers with usable ideas and information
that addresses current imperatives
• Developing user friendly frameworks and tools to evaluate
the process and the outcomes
• Collaborating with practitioners in addressing pressing
issues and problems
• Offering a critical and objective view that addresses the
realities of practice
28
Routes to evidence in youth work?
ActKnowledge
AECF Guide
Information about
evidence informed
approaches
Information about
theories of change
SAMSHA resources
Learning for
sustainability
PROSPER
Partnerships
Centre for Effective
Services
Information about
logic models
Evidence
Network
CyferNet Search
University of Arizona
Blueprint model
programmes
Campbell
Collaboration
Wisconsin online
Wisconsin
templates
Information about
programmes in
Ireland and Northern
Ireland
Together 4 all
W.K. Kellog
Foundation
Lifestart
Foundation
AECF
Community
Change
EPPI Centre
Information about
evidence-based and
informed
programmes
youngballymun
Archways
CDI Tallaght
Preparing for Life
Centre for Evidence-Based
Intervention
29
John Bamber
Principal Fellow – Community Development
Centre for Effective Services
jbamber@effectiveservices.org
0035314160513
www.effectiveservices.org
30