Welcome!
• We will begin our webinar at 9:00 am PT
• As you log on, do not be surprised if you
don’t hear anyone else; participants are
placed on mute in order to limit
background noise
‹#›
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
National CASA Performance Measurement
Lori Delale-O’Connor
Sarah Catherine Williams
Child Trends
April/May 2014
www.childtrends.org  Twitter/childtrends  www.facebook.com/childtrends:
Presenter Information
Lori Delale-O’Connor
Scientist
Child Trends
‹#›
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Having difficulty with the sound?
Please send us a message in the chat box.
Roadmap
Part I: Introductions and Overview of the National CASA project
 In-depth interviews
 Survey development, testing, and fielding
Part 2: Survey Results
 Survey completion demographics
 Survey data
 Potential “priority areas” identified
Part 3: Process of Becoming Evidenced-Based
 What it means to be evidence-based
 “Arrow” and process
 National CASA’s work to becoming evidence-based
‹#›
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
What you will walk away with…….
• Knowledge about the current data collection activity of
CASA/GAL programs across the country
• Understanding of what it means to be “Evidence Based” –
and why it matters for your CASA/GAL program
• Insight into how National CASA is moving forward with
performance measurement and working to build an
evidence base
‹#›
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Part I: Introductions and Overview
‹#›
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Child Trends’ Mission
To improve outcomes for children by providing
research and analysis to the people and institutions
whose decisions and actions affect children.
http://www.childtrends.org/
7
Research Areas
EARLY
CHILDHOOD
CHILD WELFARE
‹#›
EDUCATION (K-12)
FATHERHOOD AND
PARENTING
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
YOUTH DEVELOPMENT FERTILITY AND FAMILY
STRUCTURE
INDICATORS
DATA DEVELOPMENT
8
Child Trends’ Work
 Develop and track data on children and youth
 Help to design, implement, and evaluate child
and youth-serving programs
 Analyze and synthesize data and evidence
 Provide training and technical assistance
A key focus of our work is identifying evidencebased programs, practices, and policies
‹#›
9
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
National CASA/Child Trends Performance Measurement
Goal: Develop key measures to help National CASA
better understand performance and move toward
developing an evidence base.
Phase 1: Scan of current data collection and use (Sept-March)
 In-depth interviews
 Survey development & testing
 Survey fielding
Phase 2: Presentation of findings (April-May)
Phase 3: Recommendation of performance measures (May-June)
‹#›
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Part II: Survey Findings
‹#›
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Survey Completion Overview
Category
% Category
Level
Urbanicity
local
89
rural
50
state
9
suburban/mixed
36
state – direct advocacy
2
urban
14
Age (in years)
Program Model
10 or fewer
20
friend of the Court
56
11 – 20
36
guardian ad litem
28
21 or more
44
friend of the Court/Attorney team
10
Size (# of children served)
‹#›
%
Type of Agency
50 or less
24
independent non-profit
62
51 – 300
59
part of a government agency
21
301 or more
18
part of a non-profit umbrella agency
16
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Topics Addressed in the Survey
• Data collection responsibility and burden
• Data collection systems
• Systems used
• Meeting needs
• Data use
• Priorities for future data collection
‹#›
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Data collection and level of burden
Type of staff
%
Hours per month
CASA supervisors/coordinators
70
20
CASA administrators
56
21
Volunteer
16
4
Assistant or support staff
9
31
Other
5
19
CASA supervisors/coordinators
2
15
CASA administrators
2
14
volunteers
assistant/support staff
Don't know
‹#›
54
48
14
7
21
14
12
Very burdensome
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
28
49
38
Somewhat burdensom
8
18
31
Not very burdensome
9
Not burdensome at all
6
12
Data collection systems and satisfaction
Current Data System
% Current Data System
%
CASA Manager
39 Other
4
Microsoft Excel
36 Optima
4
COMET
16 CASA Tracker
4
State system
12 Own system
2
Efforts to Outcomes (ETO)
CASA Manager 1
COMET
Excel
State System
10
55
2
2
35
27
59
20
60
24
56
Not at all
‹#›
6 Microsoft Access
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Not very well
Somewhat well
15
18
20
Very well
Data use
‹#›
Activity
%
Reporting requirements
90
Demonstrate effectiveness
73
Make program improvements
62
Fundraising
58
Service improvements
43
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Data support needs
‹#›
Area of desired support
%
Training on how to analyze data
53
Guidance on collecting data
46
Funding for data collection/analysis
software
44
Guidance in identifying outcomes
44
Funding for personnel
40
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Data priorities
‹#›
Domain
%
Permanency
60
Well-being
44
Placements
44
Education
39
Mental health
31
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Participant Feedback
‹#›
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Participation
What do you think about the priority areas identified
in terms of their alignment with CASA?
• I think they are an accurate reflection of CASA’s mission and
the reach of a volunteer.
• I think one or more of them are outside of CASA’s mission and
the reach of a volunteer.
• I think one or more of them are outside of CASA’s mission, but
a volunteer may impact these areas anyway.
‹#›
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Participation
Do you/local programs you work with already collect
data on measures in the priority areas?
• Yes, I/they collect data on ALL of these measures.
• Yes, I/they collect data on SOME of these measures.
• No, I/they do not collect data on ANY of these measures.
‹#›
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Participation
What potential challenges do you see to getting data on
measures in the priority areas?
• My program/the programs I work with does not have access to the
data needed for these measures.
• My program/the programs I work with would need to administer
additional assessments (i.e., mental health
assessments/screenings) in order to capture data on these
measures.
• I am not clear on what type of data are needed for these measures.
• Programs may interpret the meaning of these measures differently,
and therefore data quality would be inconsistent.
‹#›
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Participation
What potential challenges do you see to reporting on
measures in the identified priority areas?
• It is a challenge to get the data for these measures, and
therefore my program/the programs I work with would most
likely be unable to report on these measures.
• I/the programs I work with can get the data on these
measures, but I am not confident I/they would be able to
report on them.
• There is a danger that programs could interpret the measures
differently, and therefore the data would not be as useful.
‹#›
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Part III: Becoming Evidence Based
‹#›
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
What We Mean by “Evidence-Based”?
 Evidence-based programs:
– Interventions that have been found to have positive effects
or impacts on targeted outcomes
 Evidence-based practices:
– Intervention strategies or program practices that have
been found to improve outcomes for children and youth
 Evidence-based policies:
– Regulations and laws that have been found to have
positive effects on child and youth outcomes
‹#›
25
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Levels of Evidence
Replicated
Experimental
High
Experimental
Quasiexperimental
Low
Evidence-informed, non-experimental
‹#›
26
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Collect Data on Performance & Outcome Measures
Targeting
Conduct
Needs
Assessment
Identify Your
Population
Select Intervention,
Develop Logic Model &
Identify Indicators
Implement Program/Approach &
Conduct Ongoing Performance
Management
Conduct an Implementation Evaluation
Conduct a Quasi-Experimental
Outcomes Evaluation
Conduct a Randomized-Controlled
Impact Evaluation
‹#›
27
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Participant Feedback
‹#›
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Participation
Where is your CASA/GAL program/the programs that you
work with on the arrow?
• We/They are not on the arrow at all.
• We/They are in the preliminary stages of performing a needs
assessment and identifying target population.
• We/They are in the more secondary stage of developing a logic
model and identifying indicators.
• We/They are in the conducting ongoing performance management
phase.
• We/They are in the more advanced stage of conducting an
implementation evaluation, quasi-experimental evaluation, or a
randomized control trial (RCT).
‹#›
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Food for thought—post-webinar survey question
Where do you see your CASA/GAL program/the programs you
work with being able to move on the arrow?
• Not any further than we/they currently are.
• Performing a needs assessment and identifying target
population.
• Developing a logic model and identifying indicators.
• Conducting ongoing performance management.
• Conducting an implementation evaluation, quasiexperimental evaluation, or a randomized control trial (RCT).
‹#›
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Building the Evidence Base
‹#›
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Benefits of Logic Models
• Set up a clear path to success or a roadmap for achieving
program goals
• Provide a visual test of whether inputs, and activities align
with outputs and outcomes
• Help to get everyone “on the same page”
• Facilitate performance management and evaluation
‹#›
32
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Logic Model Example
Program Mission
Inputs
Activities
Outputs
Program Component
Program Component
Inputs
33
Activities
Outputs
Outcomes
Outcomes
Inputs
Activities
Outputs
Outcomes
Logic Model Example
Outcomes
Outcomes
Outputs
Outputs
Activities
Activities
Inputs
34
Outcomes
Outputs
Activities
Inputs
Outcomes and Outputs
Outcomes: The observed changes in attitudes,
knowledge, or behavior among program
participants that result from a program / service /
initiative.
Output: Measures of evidence of program
implementation or quality.
‹#›
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Outputs are sometimes confused with outcomes
Outputs
Numbers of volunteers
Number of children served
‹#›
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Child Outcomes
Changes in health
Changes in behavior
Changes in knowledge
Changes in placement
Changes in permanency
What exactly is impact?
Researcher definition of impact: Linking (through study/research)
the observed outcomes to a program / service / initiative.
 Does the program lead to outcomes?
 Can we control for as many other explanations as possible?
Impact could also mean the very long term changes attributable to a
program / service / initiative.
 Child welfare advocates – do children who return home to their
parents STAY with their parents? Or do they re-enter foster care
at a later date or repeatedly?
Impact could also mean the collective outcomes across multiple
programs / services / initiatives.
 Does the same program offered in multiple locations have
outcomes that can be aggregated across the locations?
‹#›
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
How does measuring impact differ?
Program
Court Appointed
Special Advocates
Outcome(s)
Decrease in
length of stay in
foster care
Decrease in
trauma
symptoms
Are other conditions or programs
causing these changes?
Alternative
programming
‹#›
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Initiative
at local
child
welfare
agency
Change in
social worker at
public child
welfare agency
National CASA’s next steps for consideration…
• Developing a logic model
• Identifying a small set of “key measures” for all CASA/GAL
programs
• Addressing the evaluation challenges of networks structured
like National CASA
• Conveying potential benefits and collaborating across the
network to build an evidence base
‹#›
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Contact Information
Lori Delale-O’Connor
E-mail: ldelale-oconnor@childtrends.org
Sarah Catherine Williams
E-mail: swilliams@childtrends.org
‹#›
National CASA/Child Trends Performance Measurement
Lori Delale-O’Connor and Sarah Catherine Williams
Research in the Service of Children
www.childtrends.org
56