The Community Schools Evaluation Toolkit:
Moving the Research Agenda Forward
Reuben Jacobson, University of Maryland
Shital C. Shah, Coalition for Community
Schools
Agenda
2
 Research summary
 Community Schools Evaluation Toolkit
 Discussion
 Work time: work on your evaluations and network
www.communityschools.org
Research Summary
3
Are community schools effective at improving
outcomes for students, families, and the
community?
• Collected 153 studies of community school and
community school-like initiatives
–
–
CS models: lead-agency, community agency/CBO, universityassisted, and staff-initiated
Scale: boutique, local, state, national
• No studies in peer-reviewed journals
www.communityschools.org
Characterizing the Literature
4
• Mix of internal/external evaluators
• Great variation in study design
– Process/implementation studies (e.g., number of students
served)
– Outcome studies (e.g., improved achievement)
•
Number of outcomes depends on the unique strategies of each
community school – found at least 30 outcomes
– Tied to theory of action
• Selected 22 quasi-experimental studies (comparison
group, interrupted time series, controls)
www.communityschools.org
Achievement
5
The most rigorous studies indicate that community schools are
improving student achievement for some students.
• High-implementing CIS schools scored +6% than comparison
non-CIS schools in percent proficient in grade 8 math (p<.01)
and +5.1% in grade 8 reading (p<.05)
• Mixed results out of Chicago for school-level data
–
Student-level demonstrates that OST participation matters
• More students who participated in Children’s Aid Society
(CAS) after-school programs demonstrated a steady increase
from 2004 to 2007 in their math performance levels as
measured by the state assessment compared to students who
did not attend CAS activities (p<.05)
www.communityschools.org
Attendance
6
The most rigorous studies indicate that community schools are
improving attendance.
• CIS: High-implementing elementary schools had higher
attendance levels than comparison group (+0.2%, p<.05). So
did high-implementing high schools (+0.3%, p<.01)
• CAS: students who participated in CAS after-school programs
for 3-4 years had better attendance than students with lower
or no participation (p<.05)
• SF Beacons: participants who attended 30+ days of Beacon
after-school had 3.9% less total days of unexcused absences
than those who participated less than 30 days (p<.001)
www.communityschools.org
Graduation Rate
7
While a focus of many community schools, there is little
evidence of improving the graduation rate.
• CIS: high-implementing community schools had a
significantly better graduation rate than comparison
schools (+4.8 percent, p<.01)
–
Compared to other large-scale dropout programs, highimplementing CIS schools had the highest effect size for graduation
rate (ES=.31) and second-highest for dropout rate (ES=.36)
• CPS: schools that have been in CSI longer have better
rates of ninth-grade students “on-track” to graduate than
comparison group (p<.05)
www.communityschools.org
Behavior
8
While a focus of many community schools, there is
little evidence of improving student behavior.
• CPS schools had significantly less disciplinary
incidents than their matched comparison group from
2002 to 2006
• CAS teachers reported that they saw greater
improvement in students getting along with others
for students with higher rates of participation in CAS
than comparison students during the 2006-2007
school year (significance not reported)
www.communityschools.org
Other Outcomes
9
 Overall, few studies measure, evidence is weak, for
the following outcomes:




Parental engagement
Relationships with adults
Student engagement
Impact on instruction
www.communityschools.org
Future Directions for Evaluating Community
Schools
10
•
Utilize high-quality study designs that establish causality

•
•
•
•
•
•
Why? Funding. Have to demonstrate effectiveness and grant-makers are
increasingly aware of study design (e.g., i3)
Evaluate the CS strategy, not just programs


Community schools are more than after-school programming
Is the strategy or particular programming causing changes in outcomes?



level of implementation: fidelity to design matters (e.g., CIS)
longevity
intensity of the intervention (after-school participation, receiving health
services, …
Use appropriate unit of analysis (school, student, or both)
Measure:
Evaluate outcomes for families and community; also student health
Develop standard measures: e.g., how do you measure student
engagement?
Assess effectiveness of particular CS components to modify and
strengthen the design (e.g., do CS coordinators matter?)
www.communityschools.org
Community Schools Evaluation Toolkit
11
 Rationale for Results Framework
 John W. Gardner Center
 Community School Evaluators
 Coalition for Community Schools
 JP Morgan Chase
www.communityschools.org
What is in the Toolkit?
12
• 35 Page step-by-step manual
• Coalition for Community Schools Logic Model
• Indicators for 5 short-term results
• Indicators for 4 long-term results
• Descriptions of school, city, county, and state data
that is available to sites
• 45 surveys –for sites that are ready to collect
additional data
• Examples from real sites
www.communityschools.org
Goals of the Toolkit
13
• Introduce the Coalition for Community Schools
(CCS) Results Based Logic Model
• Support sites to use existing data sources and collect
additional data (if needed)
• Support sites to identify areas of success and areas
in need of improvement
• Provide evaluation planning tools
www.communityschools.org
4 Parts to the Toolkit:
14
1. Before you Start: Begin with the End in Mind
2. Get Ready: Prepare to Evaluate
3. Get Set: Designing the Evaluation
4. Go!: The Evaluation Process
www.communityschools.org
15
www.communityschools.org
Discussion
16
 Describe your experience evaluating CS.
 What challenges have you experienced? Successes?
 What are your evaluation needs?
www.communityschools.org
Work Time
17
 Work on your evaluations and network
www.communityschools.org
How to Get the Toolkit
18
• Coalition for Community Schools Web
site: www.communityschools.org
• Questions? Comments?
• shahs@iel.org
www.communityschools.org
Contact information
19
 Reuben Jacobson
 reubenjacobson@gmail.com
 Shital C. Shah
 shahs@iel.org
www.communityschools.org