What Does the Research Say?
October 2014
Agenda
•
•
•
•
•
•
Program Description
Research Overview
Key Findings
Conclusions and Continual Improvement
Lessons Learned: Mentoring for Academic Gains
Action Planning
Kelvin
Program Model
• Afterschool, Summer,
High School Placement
• 650 hours/year,
+ 1000 hours in school
• Weekly academic mentoring
• Accelerated lessons, college trips,
career exposure
• Positive culture and youth
development approach
2%
Only
of
nonprofits have this
“gold-standard”
randomized research
RCT Study Design
$4 million over 7 years
Over 3 years
(2006, 2007,
2008),
951 students &
families apply and
interview for 3
cohorts
50% accepted
through lottery
50% assigned to
control group
Researchers test
& survey:
Baseline and
three follow-ups
Two-year results
and Summer
Snapshot
published in 2011
by P/PV
Four-year results
published in 2013
By MDRC
Randomized Control Trial (RCT) Study: funded by Wallace, WT Grant, Smith Richardson, Spencer, Atlantic Philanthropies, Bank of America
Randomization – in Brief
1. Hear about HA at
school,
neighborhood,
etc.
2. Interview
99% - enter
lottery.
3. Lottery
50% enter
program:
“treatment”
“Treatment”
Access to full
program – may
never attend.
“Control”
Never able to
access program.
But given list of
other after-school &
summer options
Annual testing
($120
payment),
surveys,
Follow-up from
researchers
Researchers
• Dr. Jean Grossman
– Princeton University, P/PV, MDRC
• Dr. Carla Herrera
– P/PV
• Dr. Leigh Linden
– Columbia University, University of
Texas- Austin
Outcome Measures Collected
• Hypothesis:
- With high dosage and structure, Higher Achievement would have a
measurable impact on academic outcomes and high school placement,
by first impacting attitudes and behaviors.
• 2 key outcomes:
– Standardized Test Scores
– Application, Acceptance to, and Matriculation at Competitive High Schools
Other outcomes of interest:
–
–
–
–
–
Behavior
Academic Attitudes
Perceptions of Peer and Adult Support
Participation in HA and Other OST Programs
Engagement in Academic Activities and High School-Related Activities
KEY FINDINGS
Two-year follow-up study
“Higher Achievement’s intensive
year-round program had a
significant impact on youth’s
standardized reading and math test
scores.”
“The longstanding Higher
Achievement model is making a
difference in the academic lives of
motivated, at-risk students who
could easily fall through the
cracks.”
Summer 2010 Snapshot
High levels of attendance and
retention, during tricky middle
school years:
97% of summer participants also in
after-school
No summer learning loss.
Also - no impact on test scores,
compared to control group,
over the course of one summer:
2010
Four-year follow up study
Academic Impacts
First- Second- FourthYear FU Year FU Year FU
Math problem-solving 0.03
0.10*
0.11*
Reading
comprehension
* = Statistical significance.
0.02
0.08†
0.04
Academic and Enrichment Activities
Activity
Community service
First-Year FU
Treatmt T-C Diff
(%)
(%)
53
0
Second-Year FU Fourth-Year FU
Treatmt T-C Diff Treatmt T-C Diff
(%)
(%)
(%)
(%)
60
4
74
4
Presented ideas to a group
out of school
Visited a college
63
6†
64
9*
69
11*
72
28**
73
28**
78
25**
Read books out of school
Writing out of school
74
73
3
7*
79
74
4
3
85
82
8†
14**
Visited a business
56
4
61
8*
70
14**
Events with OST
77
10**
80
6†
84
9*
Academic contests at OST
68
13**
68
11**
71
16**
Conclusions
• Year-round, multi-year:
high dosage yields results
• No effect after 1 year, only after 2
years:
– Academic results take time
• Reading gains level with control
group in 4-year follow-up:
– Control and treatment made gains.
Reading can be more self-directed.
Math requires more instruction.
Continual Improvement
• Common Core Standards Alignment:
– Curricula and Technology
• Explicitly teach writing skills
• Staff training to improve inference
skills for reading comprehension
• “How Children Succeed” &
Social/Emotional Skills
– Piloted three assessments:
PEAR HSA, SAYO, Gallup Student Poll +
Grit Scale
Annual Operating
Plan and Individual
Work Plans,
tied to Strategic
Plan.
Annual Retreat:
Outcomes and
RCT Results
Continuous
Improvement
Cycle
Mid-Year Retreats:
Org & City course
corrections
Update individual
work plans
Monthly Dashboard and
Site Observation
Discussions:
National & Local Staff
School Partners
Quarterly
Dashboard, linked
to Financial Model
HOW MENTORING LEADS TO
ACADEMIC GAINS
Maximizing Mentors
• One full-time staff member per site devoted to mentor support
(80 mentors)
• Orientation (4 hours)
• Ongoing mini-trainings
– lesson pacing, differentiated instruction, behavior management
•
•
•
•
Scripted lessons
Nightly session feedback
Monthly in-person observations
Quarterly report card data sharing with mentors
"Whatever affects one
directly, affects all
indirectly.
I can never be what I
ought to be until you are
what you ought to be.”
- Dr. Martin Luther King