afterschooloutcomes.org
Deborah Lowe Vandell, PhD.
Principal Investigator
Professor and Founding Dean
School of Education
University of California, Irvine
Project Researchers:
Valerie Hall, Ph.D.
Pilar O’Cadiz, Ph.D.
Andrea Karsh, M.A.
afterschool@uci.edu
Background and Rationale for Online
T
o
o
l
b
o
x
 Research indicates high quality afterschool programs can have
beneficial effects on students both socially and academically
 The Afterschool Outcome Measures Project at UC Irvine developed
an Online Toolbox in response to the State’s legislative mandate for
alternative accountability measures to standardized test scores for
publically funded afterschool programs in California*
 Online Toolbox includes reliable and valid measures of Positive
Behavior Change and Skill Development for use by diverse
afterschool programs
 Field Test of online toolbox at over 200 afterschool and summer
learning program sites
 Online Toolbox Statewide Implementation in Fall 2012/ Spring 2013
Development and Testing of Outcome Measures
THREE PHASES
PHASE 1—INSTRUMENT DEVELOPMENT (2008-2009)
 Literature review
 Survey of the field (expected outcomes and current practices)
 Selection of measures with input from the CDE and Accountability and
Outcomes Subcommittee (representation from diverse programs)
PHASE 2—PILOT STUDY (2009-2010)
 Pre-piloting at 9 sites to test procedures and instructions & to determine
feasibility of online administration of surveys
 Piloting at 28 sites to test protocols under varying conditions:
•
•
Range of program size, geographic location, and level of operation
Sites with/without internet access
PHASE 3—FIELD STUDY (2010-2011)
 Test of measures & online administration across the State of California
•
196 field test sites representing diverse program sizes and populations
served by 29 programs in 9 regions statewide
NEXT PHASE:
Statewide Implementation of Online Toolbox
for California ASES & 21stCCLC Programs
 Opportunity to participate open to all publically funded
afterschool programs in California:


Afterschool Education and Safety Program (ASES)
21st Century Community Learning Center (21stCCLC)
 Up to 1000 sites statewide in 2012-2013
 Up to 2000 sites to participate by 2014
 Representation of diverse programs across 11 regions in
the state
 WHAT MEASURES ARE IN TOOLBOX ?
 WHO TAKES THE SURVEYS?
 HOW DO PROGRAMS ADMINISTER SURVEYS?
 WHEN DO ASSESSMENTS TAKE PLACE?
 WHY DO IT?

Web-based surveys of student performance collected from three
respondents:
1. Students (grades 3-12)
2. Program staff
3. Classroom teachers

PRE and POST Surveys collected at TWO time points, for example:
 Time 1 (early Fall/Winter) & Time 2 (Spring)
 Beginning of Summer & End of Summer

Technical Assistance provided through:
 Online Instructions and Resources
 Email and Telephone Communication

Individualized reports are provided to all participating programs.
 These reports enable programs to compare their students’
skills and behaviors over time, and to compare their program
and site-level outcome scores to aggregate scores across the
State.
Programs have option to administer one, two, or three surveys:
(1) Student (grades 3-12); (2) Program Staff; (3) Classroom Teachers
STUDENT SELF-REPORTS
Skill Development
•
•
•
•
Work Habits
Reading Efficacy
Math Efficacy
Science Efficacy
Positive Behavior Change
•
Social Competencies
•
Reductions in Misconduct
PROGRAM STAFF &
CLASSROOM TEACHER REPORTS
—of Student Performance
Skill Development
•Work Habits
•Task Persistence
•Academic Performance (classroom
teachers only)
Program Quality (post only)
Positive Behavior Change
•Reductions in Aggressive Behaviors
•
Affiliation with Peers
•Prosocial Behavior with Peers
•
Relations with Program Staff
•Social Skills
•
Interest in Program Activities
Field Test Key Findings
 The Online Toolbox is an easy-to-use, inexpensive method
of assessing skill development and positive behavior.
 Elementary and middle school students reported having
high quality experiences at their ASES programs.
 Quality of experiences at ASES programs is linked to
students’ skill development and positive behaviors.
 Significant improvements in student performance were
found between the fall and spring assessments.
SIGNIFICANT FINDINGS
2010-2011 Field Test
PRE-POST OUTCOMES in MATCHED SAMPLE
Staff Reports of Elementary Students
 Increase in Prosocial Behavior **
Staff Reports of Middle School Students
 Increase in Prosocial Behavior **
Teacher Reports of Elementary Students
 Increase in Academic Performance **
 Decrease in Aggressive Behavior**
Student Reports of Quality Program Experience
 Positive student reports of program experience linked to
positive results in ALL student outcomes
** p<.01
FINDINGS
Staff Reports on Student Prosocial Behavior
Increase in pre-post scores—matched sample
Middle School Students
Elementary School Students
1.4
1.43
1.39
1.42
1.42
1.38
1.41
1.36
1.4
1.34
1.39
1.32
1.38
1.3
1.37
1.28
1.37
1.36
1.26
1.35
1.24
1.34
1.28
1.22
Fall
Spring
Fall
Spring
Teacher Reports on Elementary Students
3.2
0.4
3.13
0.35
3.1
0.35
3
0.3
0.28
2.9
2.8
0.25
0.2
2.79
2.7
0.15
2.6
0.1
Fall
Spring
Fall
Spring
Program Quality Distribution
Staff & Activities
Peer Affiliation
45.0%
45.0%
39.3%
40.0%
34.3%
35.0%
39.7%
40.0%
34.3%
35.0%
32.6%
31.9%
30.0%
30.1%
30.0%
27.4%
25.9%
25.0%
18.9%
15.0%
10.0%
24.8%
25.0%
20.0%
32.8%
20.0%
15.0%
9.9%
10.0%
5.5%
5.0%
7.1%
5.4%
5.0%
0.0%
0.0%
Low
Fair
Elementary
Good
Middle
Excellent
Low
Fair
Elementary
Good
Middle
Excellent
Quality to Outcomes
Student reports of higher quality experiences
with staff & activities are linked to greater
Math Efficacy
Student reports of higher quality
experiences with peers are linked
to less Aggressive Behavior
0.70
3.8
0.64
3.63
3.6
3.49
3.47
0.60
0.60
0.53
3.4
3.2
0.50
3.20
3.13
0.45
3
2.8
0.43
0.40
0.34
2.73
0.30
2.6
2.4
0.20
2.2
2
0.10
Low Quality
Mid Quality
Elementary
Middle
High Quality
Low Quality
Mid Quality
Elementary
Middle
High Quality
EASY TO ADMINISTER
ONLINE SURVEYS
Step-by-Step Instructions
available to download from website
http://afterschooloutcomes.org/
Sign-up & Instructions on Website
To APPLY
to participate in
Statewide Implementation
CLICK HERE
To download
Instructions for Administering Surveys
CLICK HERE
Program Staff & Teacher Surveys
Reports on Students in Afterschool Program
 Program Staff and Classroom teachers are assigned
individual access codes
 Use both individual access code and the ID code for
each student for whom they complete a report
 Each student report takes 3-5 minutes to complete.
 Student reports may be completed in multiple
sessions.
For example: A staff completing reports for 20 students can do
four reports a day (a 20-minute session) on five different days.
 Provides a battery of valid, reliable, and easy to administer youth
outcome measures
 Expands accountability tools beyond standardized test scores
and other academic measures
 Value in tracking individual student performance over time with
matched pre-post surveys, providing more powerful &
interpretable data
•
MATCHED = Respondent data are matched for both pre and post survey
administrations (i.e. same students, staff and/or teachers in fall and spring)
 Information can be useful to guide program improvement based
on student reports of program experience AND on student
performance
For more information on measures:
Forum for Youth Investment
publication (September 2011)
• Review of measures in Toolbox
with a technical appendix
• Down Load at:
forumfyi.org
Or, link from our website:
http://afterschooloutcomes.org/
Contact Info:
EMAIL: afterschool@uci.edu