ONLINE TOOL BOX
UNIVERSITY OF CALIFORNIA, IRVINE
CAOMP Email: afterschool@uci.edu
Background and Rationale for Online Toolbox
 Research indicates high quality afterschool programs can have beneficial effects
on students both socially and academically
 The California Outcome Measures Project (CAOMP) developed an
Online Toolbox in response to the State’s legislative mandate for alternative
accountability measures to standardized test scores for publically funded
afterschool programs*
 Toolbox includes reliable and valid measures of Positive Behavior Change and
Skill Development to be used by State funded programs as part of their
accountability measures
 Field Test of measures administered using online toolbox at over 200
afterschool and summer learning program sites concluded in Fall 2011

Online Toolbox will be made available for general use in Fall 2012
*In partnership with UC Davis and with support David and Lucile Packard Foundation &
California Department of Education
OVERVIEW
Afterschool Outcome Measures Online Toolbox

Web-based surveys of student performance collected from three
respondents:
1. Students (grades 3-8)
2. Program staff
3. Classroom teachers

PRE and POST Surveys collected at TWO time points, for example:
 Early Fall & Late Spring
 Beginning of Summer & End of Summer

Technical Assistance provided through:
 Online Instructions and Resources
 Email and Telephone Communication

Programs receive reports of their survey results at program and site
level and aggregate results of California statewide data for comparison
DESCRIPTION OF MEASURES
Programs have option to administer one, two or all three surveys:
(1) Student (grades 3-8; (2) Program Staff; and/or (3) Classroom Teachers
STUDENT SELF-REPORTS
—Elementary & Middle (5-6 subscales)
Skill Development
 Work Habits
 Reading Efficacy
 Math Efficacy
 Task Persistence (middle school only)
Positive Behavior Change
 Social Competencies
 Misconduct
Program Quality (spring only)
 Staff & Activities
 Peer Affiliation
PROGRAM STAFF &
CLASSROOM TEACHER REPORTS
—of Elementary & Middle School Student
Performance (5-6 subscales)
Skill Development
 Work Habits
 Task Persistence
 Academic Performance (classroom teachers only)
Positive Behavior Change
 Aggressive Behavior with Peers
 Prosocial Behavior with Peers
 Social Skills
SIGNIFICANT
FINDINGS
From 2010-11 Field Test
PRE-POST OUTCOMES in MATCHED SAMPLE
Staff Reports of Elementary Students
 Increase in Prosocial Behavior **
Staff Reports of Middle School Students
 Increase in Prosocial Behavior **
Teacher Reports of Elementary Students
 Increase in Academic Performance **
 Decrease in Aggressive Behavior**
Student Reports of Quality Program Experience
 Positive student reports of program experience linked to positive
results in ALL student outcomes
** p<.01
* p<.05
SIGNIFICANT FINDINGS
Staff Reports on Student Prosocial Behavior
Increase in pre-post scores—matched sample
Elementary School Students
Middle School Students
1.4
1.43
1.39
1.42
1.42
1.38
1.41
1.36
1.4
1.34
1.39
1.32
1.38
1.3
1.37
1.28
1.37
1.36
1.26
1.35
1.24
1.34
1.28
1.22
Fall
Spring
Fall
Spring
SIGNIFICANT FINDINGS
Teacher Reports on Elementary Students
Academic Performance
Increase in pre-post scores—
matched sample
3.2
Aggressive Behavior
Decrease in pre-post scores—
matched sample
0.4
3.13
0.35
3.1
0.35
3
0.3
0.28
2.9
2.8
0.25
0.2
2.79
2.7
0.15
2.6
0.1
Fall
Spring
Fall
Spring
Program Quality Distribution
Peer Affiliation
Staff & Activities
45.0%
45.0%
39.3%
40.0%
34.3%
35.0%
39.7%
40.0%
34.3%
35.0%
32.6%
31.9%
30.0%
30.1%
30.0%
27.4%
25.9%
25.0%
18.9%
15.0%
10.0%
24.8%
25.0%
20.0%
32.8%
20.0%
15.0%
9.9%
10.0%
5.5%
5.0%
7.1%
5.4%
5.0%
0.0%
0.0%
Low
Fair
Elementary
Good
Middle
Excellent
Low
Fair
Elementary
Good
Middle
Excellent
Quality to Outcomes
Student reports of higher quality
experiences with staff & activities are
linked to greater Math Efficacy
3.8
Student reports of higher quality
experiences with peers are linked to
less Aggressive Behavior
0.70
3.63
3.6
3.49
0.64
3.47
0.60
0.60
3.4
3.2
0.53
3.20
3.13
0.50
0.45
0.43
3
0.40
2.8
2.73
0.34
2.6
0.30
2.4
0.20
2.2
2
0.10
Low Quality
Mid Quality
Elementary
Middle
High Quality
Low Quality
Mid Quality
Elementary
Middle
High Quality
How does it work?
EASY TO ADMINISTER
Step-by-Step Instructions Available to
download from
CAOMP Website
afterschooloutcomes .org
Updated Website
to Launch
JANUARY 2012
Online Toolkit
Available to public
FALL 2012
GO TO PROJECT WEBSITE:
afterschooloutcomes.org
1. Click on “Field Test Surveys” tab at top.
2. SELECT appropriate survey:
Once survey link is selected
you should see this first page
of the survey.
Example:
Elementary Student Survey
3. Student
a. Selects language [English or Spanish]
b. Enters assigned survey code:
Click down
For Spanish
4. When student finishes survey—in about
10 minutes—this final page should appear.
Program Staff & Teacher Surveys
Reports on Students in Afterschool Program
 Program Staff and Classroom teachers are assigned
individual access codes
 Use both individual access code and the ID code for each
student for whom they complete a report
 Each student report takes 3-5 minutes to complete.
 Student reports may be completed in multiple sessions.
For example: A staff completing reports for 20 students can do four
reports a day (a 20-minute session) on five different days.

Expands accountability tools beyond standardized test scores and
other academic measures

Provides a battery of valid, reliable and easy to administer youth
outcome measures

•
Currently include: Skill Development, Positive Behavior Change & Program Experience
•
Process of testing measures of STEM (Science, Technology, Engineering and Math)
learning in afterschool
•
Aim is to make available a menu of measures that match a range of student outcome
goals for diverse out-of-school time programs
Value in tracking individual student performance over time with
matched pre-post surveys—providing more powerful & interpretable
data
•

Matched = Respondent data is matched for both pre and post survey administrations (i.e.
same students, staff and/or teachers in fall and spring)
Information can be useful to to guide program improvement based on
student reports of program experience AND on student performance