Undertaking Local Evaluation
Getting Started
Dr Miranda Wolpert
Dr Jess Deighton
Dr Leda Kamenopoulou
Emma Trustam
CAMHS EBPU
CORC
London 25th June 2010
Who you are…
•
•
•
•
•
•
•
•
•
Researchers
Nurses
Psychiatrists
Psychologists
Family Therapists
Social Workers
Commissioners
Trainers
Psychotherapists
What you are hoping for…
•
•
•
•
•
•
Knowing where to start
Structuring the evaluation and being
able to ask relevant questions
How to determine the best evaluation
methodologies, tools and measures to
use *****
Working out how to choose the most
helpful project.
Being confidant about when "enough
is enough" in terms of the pursuit of
data
How to plan and structure effective
evaluation
•
•
•
•
•
•
•
•
Interpretation in order to aid
understanding of how services are
performing
Sensitivity to issues of which I may be
unaware
Statistical analysis or appropriate
interpretation of findings.
How to merge data from different
sources
Dissemination of results- how and to
whom
Risk management
Ethical aspects
Ideas about some of the challenges
What are your biggest
challenges…
•
•
•
•
•
•
Time constraints
Lack of capacity
What tools are most appropriate to use
Getting practitioners involvement
Getting reliable data
Lack of robust data and information
locally
•
•
•
•
•
Getting questionnaires filled in
Working out what to evaluate
Getting honest answers
Getting management support
Putting conclusions into practice
By the end of today
You should be able to….
• Feel ready to approach evaluation in your own
setting
• Be aware of likely challenges
• Know when a local evaluation is not feasible or
advisable
Five key steps to local
evaluation
Step 1 Answer these questions:
“Who? What? When? Why? (allow 1 month*)
Step 2: Develop a plan (allow 1 month*)
Step 3: Collect information (if retrospective allow 1-2
months, if pre and post measures allow 6 - 12
months*)
Step 4: Collate information (allow 2 months*)
Step 5: Make sense of information ( allow 1 month*)
*clearly these all heavily context dependent and figures here just a guide…but things
generally take longer than you expect
Plan for day
•
•
•
•
•
•
•
•
•
11am Welcome and introduction
11.15 Overview: what is local evaluation?
11.30 Introduction to Erinsborough case study
11.40 Step 1: answering “who, what, when and why”?
1pm Lunch
1.45 Step 2 : developing a plan
2.45 Step 3 & 4: collecting information and collating information
3.15 Step 5: making sense of information
4pm Close
Overview
What is local evaluation? How does it link with
research and clinical practice?
• The research-practice continuum
– Where does local evaluation sit?
– What sources of information might be used
• Sources of Information
• Local Evaluation and ethics
– What are the ethical implications of research versus audit?
– When do we have to ask for ethical approval?
Practice
Research
Activity
Individual
practitioner
reflection
Supervision, regular
monitoring / evaluation
of individual practice,
progress
Monitoring /
evaluation of team
practice and
progress
Local evaluation,
monitoring /
evaluating a
service’s outcomes
Local research
aimed at national
audience
National / international
research aimed for
national / international
audience
Practice
Research
Activity
Individual
practitioner
reflection
Supervision, regular
monitoring / evaluation
of individual practice,
progress
Monitoring /
evaluation of team
practice and
progress
Local evaluation,
monitoring /
evaluating a
service’s outcomes
Local research
aimed at national
audience
National / international
research aimed for
national / international
audience
Local Authority or
PCT, possibly other
areas nationally while
acknowledging caveats
Nationally /
internationally
Generalisability
Individual
case
Individual
practitioner
Individual team
(possibly other teams
matched on a number
of features)
Local Authority or PCT
(possibly other LAs /
PCTs matched on
a number of features)
Practice
Research
Activity
Individual
practitioner
reflection
Supervision, regular
monitoring / evaluation
of individual practice,
progress
Monitoring /
evaluation of team
practice and
progress
Local evaluation,
monitoring /
evaluating a
service’s outcomes
Local research
aimed at national
audience
National / international
research aimed for
national / international
audience
Local Authority or
PCT, possibly other
areas nationally while
acknowledging caveats
Nationally /
internationally
Generalisability
Individual
case
Individual
practitioner
Individual team
(possibly other teams
matched on a number
of features)
Local Authority or PCT
(possibly other LAs /
PCTs matched on
a number of features)
Timescales
Ongoing,
iterative process
Can be either
Discrete
research project
Practice
Research
Activity
Individual
practitioner
reflection
Supervision, regular
monitoring / evaluation
of individual practice,
progress
Monitoring /
evaluation of team
practice and
progress
Local evaluation,
monitoring /
evaluating a
service’s outcomes
Local research
aimed at national
audience
National / international
research aimed for
national / international
audience
Local Authority or
PCT, possibly other
areas nationally while
acknowledging caveats
Nationally /
internationally
Generalisability
Individual
case
Individual
practitioner
Individual team
(possibly other teams
matched on a number
of features)
Local Authority or PCT
(possibly other LAs /
PCTs matched on
a number of features)
Timescales
Ongoing,
iterative process
Can be either
Discrete
research project
Requires ethical approval?
Unlikely
Sometimes
Almost always
Practice
Research
Activity
Individual
practitioner
reflection
Supervision, regular
monitoring / evaluation
of individual practice,
progress
Monitoring /
evaluation of team
practice and
progress
Local evaluation,
monitoring /
evaluating a
service’s outcomes
Local research
aimed at national
audience
National / international
research aimed for
national / international
audience
Local Authority or
PCT, possibly other
areas nationally while
acknowledging caveats
Nationally /
internationally
Generalisability
Individual
case
Individual
practitioner
Individual team
(possibly other teams
matched on a number
of features)
Local Authority or PCT
(possibly other LAs /
PCTs matched on
a number of features)
Timescales
Ongoing,
iterative process
Discrete
research project
Can be either
Requires ethical approval?
Unlikely
Sometimes
Almost always
Uses for the data
Inform practice
Both
Add to the general
evidence base
e.g. Trying to
defend
service
against cuts
Single-minded
Open-minded
Impact
How do we demonstrate that
the programme has a positive
impact?
Does the programme have an
impact?
Cost effectiveness
How do we show that the
programme is cost effective?
Is the programme costs
effective?
Specific groups
How do we demonstrate that
the programme works for
certain groups of children?
Does the efficacy of the
programme depends on which
groups are being worked with?
e.g.,
exploratory
research for
a local /
national
audience
Erinsborough example
Erinsborough Case-study
Commissioner for this
service is Anne McCarthy
Helen Morgan-client’s
mother- attends
Service Manager
Ravi Sharma
Lexie Morgan- 11 Year old client
Primary Mental Health
Worker – Richard Smith
Research Assistant
Rosa Capello
Introducing CAMHS in schools in
Erinsborough
1.
2.
3.
4.
5.
Richard is leading a project to introduce CAMHS in
schools.
The project has been up and running for the last 6
months.
Ravi has asked Richard to evaluate this initiative
Richard has been asked to produce a preliminary
report for the board by December 2010 and a final
report by February 2010. This will help the board
decide if they want to continue with this project
Ravi has agreed that Richard can have 2 hours a week
to do this. He can draw upon Rosa’s time up to 1 day a
week
About Erinsborough::
1.
Erinsborough is part of CORC (CAMHS Outcomes Research Consortium). All
teams within Erinsborough CAMHS (including that in the schools) are already
routinely collecting outcome data about child mental health
1. Goals
2. CHI ESQ
3. SDQ
The information is collected from each child once in the first session and again 6
months later
2.
Other information is captured as part of the usual system in the CAMHS service:
–
Child’s age, gender and ethnicity
–
Who the child lives with (e.g., mother only, father and mother etc.)
–
Referral rates
–
No of sessions seen
–
DNA rates
3. Erinsborough CAMHS has a service users group that meets every 6 months
Step 1
Answer : who, what, when and why?
SMART
Specific
• Well defined
• Clear to anyone that has a basic knowledge of the project
Measurable
• Know if the goal is obtainable and how far away completion is
• Know when it has been achieved
Agreed Upon
• Agreement with all the stakeholders what the goals should be
Realistic
• Within the availability of resources, knowledge and time
Time Based
• Enough time to achieve the goal
• Not too much time, which can affect project performanc
Step 2
Develop a plan
Step 3
Collect the information
Step 4
Collate the information
Step 5
Making sense of the
information