Why Do College Coaching Interventions Work?
Scott Carrell, UC Davis and NBER
Bruce Sacerdote, Dartmouth College and NBER
1
0
0.2
0.4
0.6
US College Completion Ranks 11-13thth In OECD
1970
1980
US
ITA
1990
Year
NZ
FIN
Cascio Gordon Clark JEP 2008
2000
UK
SWE
SWI
2
US Dept of Ed Trio Programs


$879 million in FY 2011

GEAR up, Upward Bound, Talent Search
Fundamental tenets:



Catch students early..by 8th grade to get them college ready
Avoid discussion of randomized evaluation
Bruce and Scott don’t know what they are talking about
3
US Economist/Ed School Programs





Send letters to top students (Hoxby and Turner)
Auto fill the FAFSA (Bettinger Long Oreopolous
Sabonmatsu)
Send text messages (FAFSA reminders or deadline
reminders) (Castleman and Page)
Offer remote coaching or application help (Phillips and
Reber)
Carrell and Sacerdote: in person help, financial
incentives, “likely letters”

Clearly different populations in all of these
4
2000
0
0
500
1000
1500
1500
1000
500
Frequency
Frequency
2000
2500
2500
2010 Cohort: Frequency Counts 10th Grade Math Scores for Non College Goers
1100
1100
1120
1120
1140
math_score
1160
1140
1180
1160
math_score
No College
College
5
The Mission



Ask whether simple interventions (bonuses, mentoring,
likely letters) very late in the game can have a
meaningful impact on long term outcomes


Attending, persisting, graduating, income
Use Clearinghouse data to track into College
Are these marginal college going students very
different than the average?
For whom does it work? Try to infer mechanisms
using baseline interests, preparation, personality
measures

SAT Questionnaire, survey data
6
The Answer
 Yes
 If you are female
 Or a guy who has not taken the SATs
 If you do not have strong parental and teacher support for
completing applications
 If you are not extraverted
 What doesn’t work
 Likely letters
 Letters of encouragement (Delaware, NH)
 Cash Bonuses alone
7
Hypotheses
 Behavioral
 Fear of process
 Lack of easily obtained information
 Procrastination/disorganization
 Largely rational: have already optimized
 Need expert/ adult help
8
What is the Intervention
 Guidance counselors identify high school seniors at risk
of not applying to college
 Ideally students who have expressed interest in applying
 But have taken few steps
 Typically identify in mid-December
 We randomly choose half for the treatment group
 Guidance staff then invites treatment group to participate
9
What is the Intervention (2)
 We visit once per week for 3-5 weeks
–
–
–
–
–
–
–
–
Pair up each HS student with a Dartmouth student
1.5-4 hours per session
Start and complete applications
Send transcripts as needed
Pay application fees
Complete common apps, two year schools
Start the FAFSA
Sign up for SATs & send scores
 Incentives
–
–
–
–
–
Free coaching
App fees paid
$100 cash bonus upon completion
Get out of class
Pizza
10
The Process
11
Mission Accomplished (?)
12

Challenges Faced By HS Students And Guidance
Counselors
Horrible internet access and computers
– In first year, many schools had BLOCKED the application
sites for all NH Community Colleges
– When that was fixed, we still had credit card problems
– In some cases we brought our own laptops and MiFi cards

Students and parents unfamiliar with Common App,
FAFSA, online forms

Students were often selected to be the procrastinators
13
0
.1
.2
.3
.4
.5
Treatment and Control Standardized Reading Scores
-4
-2
0
2
x
Treatment
Control
14
0
.1
.2
.3
.4
.5
Standardized Math Scores: Treatment Group Versus All Non Experimental
-4
-2
0
2
4
6
x
Treatment
All Other
15
Table 1: Summary Statistics for Treatment and Control Groups
Control Group
Mentoring Treatment
Variable
Obs
Mean Std. Dev
Obs
Mean
Std.
Dev
Accepted Treatment
10th Grade Math Score (Standardized)
10th Grade Reading Score (Standardized)
Math > 50th Percentile in State
Reading > 50th Percentile in State
Math >75th Percentile
Reading > 75th Percentile
Free and Reduced Lunch Eligible
Male
Non-white
Any College (Clearinghouse)
Four Year College (Clearinghouse)
Persist for First Two Years Post Grad
Persist in a Four Year College
Enrolled 3+ Semesters
902
798
799
798
799
798
799
902
902
902
902
902
902
902
902
0
-0.480
-0.436
0.312
0.350
0.164
0.213
0.277
0.548
0.173
0.438
0.169
0.195
0.094
0.237
871
778
772
778
772
778
772
871
870
871
871
871
871
871
871
0.454
-0.286
-0.278
0.335
0.398
0.185
0.224
0.286
0.575
0.201
0.592
0.276
0.240
0.115
0.292
0.498
0.943
0.966
0.472
0.490
0.389
0.417
0.455
0.495
0.401
0.492
0.447
0.427
0.319
0.455
0
0.937
0.928
0.464
0.477
0.371
0.410
0.448
0.498
0.378
0.496
0.375
0.397
0.292
0.426
Students are randomly assigned to treatment within high school. Data include 2009-2014 cohorts.
Table 2: Show Treatment Status Unrelated to Pre-Treatment Characteristics
(1)
Treatment Status
Men
(2)
Treatment Status
Women
Standardized 10th Grade Math Score
0.001
(0.012)
0.041
(0.025)
Standardized 10th Grade Reading Score
-0.025+
(0.014)
-0.006
(0.020)
Free Reduced Lunch Eligible
-0.043
(0.027)
0.073
(0.046)
Student is Nonwhite
0.019
(0.032)
-0.038
(0.057)
Observations
1216
866
R-squared
0.355
0.321
F Pre-Treat Variables
1.281
2.109
Students are randomly assigned to treatment within high school. Data include 2009-2014 cohorts. Regressions include
highp-value
school*cohort dummies which is the level at which randomization 0.294
occurred. Standard errors0.098
are clustered at17
the
high school*cohort level. Regressions include birthyear*cohort dummies to control for students' age within grade.
Effect of Treatment Status on Applying
Mentoring
Treatment
Observations
R-squared
(1)
Apply to
Any
College
(Survey)
(OLS)
(2)
Apply to
Any
College
(Survey)
IV
Estimate
(3)
(4)
Women Men Apply
Apply to
to Any
Any
College
College
(Survey)
(Survey)
OLS
OLS
0.274**
(0.050)
0.375**
(0.068)
0.294**
(0.049)
0.243**
(0.078)
859
0.234
859
0.309
391
0.293
468
0.231
Survey data with 50% response rate. Control mean is .72.
Standard errors in parentheses, includes high school* cohort f.e. , age controls, male
+ significant at 10%; * significant at 5%; ** significant at 1%
18
Baseline Treatment Effects on Enrollment in Any College
Effects on Enrollment Any College
Mentoring Treatment (OLS)
Transcript Only (OLS)
Mentoring Treatment (IV)
(1)
Whole
Sample
(2)
Women
(3)
Men
(4)
(5)
Did Not Took SAT
Take SAT
0.060**
(0.018)
0.146**
(0.042)
0.007
(0.025)
0.083**
(0.026)
0.035
(0.035)
-0.005
(0.019)
0.005
(0.034)
0.000
(0.021)
0.035
(0.034)
-0.049
(0.035)
0.133**
(0.041)
0.299**
(0.087)
0.017
(0.061)
0.160**
(0.047)
0.086
(0.085)
First Stage for IV
Mentoring Treatment
0.463** 0.500** 0.429** 0.511** 0.444**
(0.039)
(0.044)
(0.042)
(0.033)
(0.070)
Observations
1,114in college1,509
1,453
Outcome
variable is a dummy equal to 1 if the student2,623
has any enrollment
including 2 year
or four year1,170
colleges. Outcome variables are based on the Nation Student Clearinghouse data. Data include 20092014cohorts. Regressions include high school*cohort dummies which is the level at which randomization
occurred. Standard errors are clustered at the high school*cohort level. ** p<0.01, * p<0.05, + p<0.1
19
Baseline Treatment Effects on Enrollment in A Four Year College
Effects on Enrollment Four Year
College
Whole
Sample
Women
Men
Did Not
Take SAT
Took
SAT
Mentoring Treatment (OLS)
0.057**
(0.018)
0.001
(0.015)
0.125**
(0.037)
0.107**
(0.031)
0.007
(0.022)
0.222**
(0.062)
0.020
(0.028)
0.003
(0.028)
0.047
(0.068)
0.103**
(0.026)
0.002
(0.012)
0.202**
(0.048)
-0.005
(0.033)
-0.038
(0.030)
-0.018
(0.083)
0.463**
(0.039)
2,623
0.500**
(0.044)
1,114
0.429**
(0.042)
1,509
0.511**
(0.033)
1,453
0.444**
(0.070)
1,170
Transcript Only (OLS)
Mentoring Treatment (IV)
First Stage for IV
Mentoring Treatment
Observations
20
Treatment Effects on Persistence in College (Women)
(1)
Women
(4)
(5)
Men
Women
No SAT Data
Enrolled in Enrolled Any Enrolled
Enrolled
Enrolled
3+ Semesters College Both Four Year
Four Year Second Year
School Years College Both College Both Conditional
Post
School Years School Years on Enrolled
Graduation
Post
Post
First Year
Graduation Graduation
Mentoring
Treatment
0.129*
(0.053)
(2)
Women
0.105*
(0.042)
(3)
Women
0.097**
(0.030)
0.014
(0.041)
-0.040
(0.066)
Observations
535
535
535
445
263
R-squared
0.123ways to measure
0.105 persistence
0.220 into the 0.165
Outcome variables 0.172
are four different
second year of college. Sample is limited to women in the 2009-2012
cohorts.
21
Evidence on Mechanisms
Interaction of Mentoring Treatment with Sources of Assistance on Applications
(1)
Coefficients on
Treatment *SAT/
Survey Measure
(2)
Coeff on
Treatment
Indicator
(3)
Coeff on
SAT/
Measure
(4)
N
(5)
Mean
(6)
SAT/Surv
indicator
regressed
on Male
Dummy
Do Not Need Help
With Educ Planning
-0.116*
(0.059)
0.126**
(0.058)
0.049
(0.039)
1302
0.829
0.015***
(0.004)
Survey Measure
Parents Help With
College Applications
-0.131*
(0.067)
0.118**
(0.045)
0.133***
(0.041)
724
0.468
0.014
(0.037)
Teacher Helps With
College Applications
-0.165*
(0.091)
0.112***
(0.030)
0.089
(0.062)
646
0.172
-0.023
(0.030)
Guidance Counselor
Helps with College
Application
-0.009
(0.069)
0.0541
(0.037)
0.037
(0.057)
724
0.312
-0.0982**
(0.034)
SAT Questionnaire
Measure
22
Dependent Variable is Enrollment in Any College
Interaction of Mentoring Treatment with Beliefs About Wages/ Tuition
Survey Measure
(1)
(2)
(3)
Coeff on Coeff on Coeff on
Treat*
Treat
Survey
Survey Indicator Measure
Measure
(4)
N
(5)
Mean
(6)
Survey
indicator
regressed
on Male
Dummy
Log (Hourly wage at Age
30 w. only HS Diploma)
-0.159*
(0.078)
0.545**
(0.233)
-0.052
(0.099)
354
2.931 0.307***
(0.052)
Log (Tuition+Fees
Community College)
-0.019
(0.036)
0.246
(0.310)
-0.023
(0.031)
506
9.052
0.032
(0.091)
Log (Tuition Fees NH
Public University)
-0.028
(0.030)
0.343
(0.295)
0.015
(0.023)
502
10.076
-0.040
(0.085)
Need College Degree for
Stated Career Goal
-0.010
(0.112)
0.027 0.273***
(0.094)
(0.055)
663
Dependent Variable is Enrollment in Any College
0.777 -0.089***
23
(0.032)
Evidence on Mechanisms
Interaction of Mentoring Treatment with Sources of Assistance on Applications
Survey Measure
Individual Measures
Likes to meet new people
Enjoy Amusement Rides
Composite Measures
Meets Deadlines/
Organized
Adventuresome
Self-Esteem
(1)
(2)
(3)
Coefficien Coefficient Coefficien
ts on
on
t on
Treatment Treatment
Survey
*Survey Indicator Measure
Measure
(4)
N
(5)
Mean
(6)
Survey
indicator
regressed
on Male
Dummy
-0.096**
(0.039)
0.031
(0.040)
-0.305***
(0.086)
-0.287**
(0.136)
0.280***
(0.085)
0.259**
(0.103)
0.150**
(0.055)
0.097
(0.087)
530
0.723
530
0.696
0.083
(0.189)
-0.275
(0.179)
-0.097
(0.128)
0.030
(0.082)
0.239
(0.143)
0.136
(0.096)
0.096
(0.133)
0.144
(0.146)
0.143
(0.092)
530
0.343
530
0.657
552
0.672
0.011
(0.022)
0.017
(0.021)
0.007
(0.028)
24
Dependent Variable is Enrollment in Any College
Interaction of Mentoring Treatment with Personality Traits
Survey Measure
Self Esteem
Believes In Self
(1)
(2)
(3)
Coefficients Coefficie Coefficie
on
nt on
nt on
Treatment*S Treatmen Survey
urvey
t Measure
Measure Indicator
(4)
N
(5)
Mean
(6)
Survey
Measure
regressed
on Male
Dummy
0.003
(0.041)
-0.044
(0.042)
-0.025
(0.088)
-0.063
(0.092)
0.083
(0.076)
0.110
(0.075)
0.096*
(0.053)
0.101
(0.064)
552
0.663
552
0.601
Change Important Things
-0.057
(0.111)
0.108
(0.088)
0.080
(0.076)
552
0.672
0.027
(0.040)
Solves Problems
-0.151* 0.186***
(0.081)
(0.064)
0.022
0.055
(0.088)
(0.069)
0.097
(0.070)
-0.001
(0.052)
552
0.739
552
0.683
0.048
(0.038)
0.003
(0.040)
Deals Well With Problems
Not Easily Pushed Around
25
Dependent Variable is Enrollment in Any College
Interaction of Mentoring Treatment with Personality Traits
Survey Measure
Skips Homework
Lose Papers Easily
Not Organized
Wastes Time
Waits Until Last Minute
Surprised By Deadlines
(1)
(2)
(3)
Coeff on
Coeff
Coef
Treat*Survy
Treat Survey
Measure Indicatr Measre
-0.047
(0.088)
-0.049
(0.128)
0.119
(0.087)
0.055
(0.058)
-0.012
(0.093)
0.071
(0.078)
0.076
(0.060)
0.063
(0.046)
0.022
(0.057)
0.042
(0.056)
0.073
(0.065)
0.041
(0.060)
0.059
(0.056)
-0.080
(0.086)
0.022
(0.055)
-0.039
(0.046)
0.103
(0.080)
0.106*
(0.056)
(4)
N
(5)
Mean
(6)
Survey
Measure
on Male
Dummy
530 0.408
0.075*
(0.043)
0.012
(0.032)
0.048
(0.040)
-0.117***
(0.044)
0.057
(0.044)
-0.027
(0.044)
530 0.157
530 0.306
516 0.479
516
0.411
516 0.481
26
Dependent Variable is Enrollment in Any College
Some Takeaways
 We find a boots on the ground approach is effective
– Appears to compensate for lack of or non take up parental or
teacher help
– No evidence that its more effective for people who struggle
with deadlines, planning, organization
– Helps students less far along in process, ie non SAT takers
– Much more robust effects for the women..lack of effects for
men very much related to labor market opportunities
 Can’t find evidence for effects from…
– Likely letters/ letters of encouragement
– Cash bonuses along..Though cash bonuses may be very
important for takeup
– Texting of CCSNH students has not yielded effects
– Nor did letter of encouragement to DE graduated seniors
27
More Takeaways
 We find evidence in favor of optimization story and
need skilled help story
 We don’t find evidence for super naïve behavioral story
 Or that failure to attend is driven by forgetfulness or lack of
organization
 But parents/mentors could indeed be solving this
 We suspect that letter based interventions or simple
text reminders may only work with specific groups or
in specific contexts
28
Also Mention If Time
 Huge drop off in participation when we removed the
cash bonus
 Men very likely to say they have a job they prefer to
college
 And no evidence in ACS of OLS return to two years of college
 Still have low average persistence in both treat and
control
 Can’t make the Big Five measures work
29
A Very Few Words on Returns to College
Oreopoulos and Salvanes
30
Caveats on Earnings Returns to College
 Large number of papers but sources of ID very
different
– We haven’t actually shown that the marginal students in
current college going interventions have earnings gains
– Bettinger, Gurantz, Kawano and I having trouble finding
earnings effects from winning the Cal-Grant
– The one thing we have learned is just how ridiculously noisy
the outcome is
31
Evidence on Mechanisms
Interaction of Mentoring Treatment with Sources of Assistance on Applications
32
Dependent Variable is Enrollment in Any College
Do We Affect Type of College Attended?
Treatment
0
1
Nashua Community College
Manchester Community College
Nhti - Concord's Community College
Great Bay Community College
Plymouth State University
Rivier College
University Of New Hampshire
Southern New Hampshire
Hesser College - Manchester
Community College Of Vermont
Colby Sawyer College
Hesser College -Nashua
Northern Essex Community College
University Of Massachusetts At Lowell
Arkansas State University - Newport
Pennsylvania State University
Total Shown
Total in College
25
15
14
11
2
4
7
0
4
1
1
3
0
0
0
0
24
22
16
8
8
7
6
5
4
3
2
2
2
2
1
1
87
114
113
130
33
Do We Affect Type of College Attended? (2)
(1)
(2)
Institution Attended
Institution
is Masters Degree
Attended is
Granting: Both
Masters Degree
Genders At Any Granting: Women
College
At Any College
Treatment
Observations
R-squared
(3)
Acceptance
Rate of
College
Attended:
Both
Genders
0.002
(0.040)
0.038
(0.081)
-2.237
(2.811)
383
0.133
171
0.231
145
0.106
Outcome variables measured in IPEDS data. Sample only includes students
in college and for whom we have IPEDS data
34
Split Sample By Test Score
(1)
(2)
(3)
(4)
Enrollment Enrollment Enrollment Enrollment
Two Year Four Year Two Year Four Year
College
College
College
College
Women
Women
Men
Men
Mentoring Treatment
0.137+
(0.067)
0.064
(0.042)
0.014
(0.057)
0.060
(0.041)
Reading Score > 50th
Percentile in Treatment
Group
-0.208*
(0.090)
0.116*
(0.052)
-0.054
(0.073)
-0.025
(0.047)
Transcript Only Group
-0.044
(0.045)
0.003
(0.032)
0.024
(0.037)
-0.026
(0.023)
788
0.124
788
0.174
1,080
0.075
1,080
0.231
Observations
R-squared
35
Interaction of Treatment with Immigration Status
(1)
Women:
Enrolled in
Any College
(2)
Men:
Enrolled in
Any Year
College
(3)
Women:
Enrolled in
Four Year
College
(4)
Men:
Enrolled in
Four Year
College
Treatment
0.150
(0.074)
0.027
(0.018)
0.202
(0.091)
0.085
(0.032)
Immigrant/ Refugee
-0.039
(0.384)
0.057
(0.419)
-0.211
(0.077)
0.099
(0.226)
Immigrant*Treatment
0.002
(0.285)
0.089
(0.219)
-0.071
(0.066)
-0.260
(0.173)
136
0.235
144
0.182
136
0.134
144
0.118
VARIABLES
Observations
R-squared
Data are from Manchester West 2010,2011 Cohorts. Sample is roughly 9% immigrants.
36
Mechanisms
n
Bonus? Incentive?
n
Same or Cross Gender Mentoring?
n
What do participants say?
n
More effective in schools with fewer resources?
37
Evidence From 2012 Cohort (Coaching Plus $100 Bonus Versus Bonus Alone)
38
Post-Survey Evidence on $100 Bonus
$100 bonus affect Decision to Complete
Program?
Freq.
Aware, no effect
11
Initially motivating not long
run factor
2
Not expecting
4
Important
2
Total
19
39
Take Up Rates Within Mentoring Group
What Aspects Helped the Most?
n
19 of 19 mentioned in person help with applications
n
5 of 19 mentioned the $100 bonus
– 3 of 7 men versus 2 of 10 women
n
12 of 19 mentioned our paying for application fees
– Rational bc a lot more than $100 for some students
41
Treatment Effect by High School
Average College Going Versus Effect Size For Women
43
.4
Average College Going Versus Guidance Counselors Per Student
.2
.3
Kearsarge
Pinkerton
Dover
Manchester West
Londonderry
Lebanon
.1
Nashua South
0
Nashua North
Portsmouth
.003
.004
.005
guidance_per_student
.006
44
Mentoring Treatment Interaction with Sources of Disadvantage
Mentoring Treatment Interaction with Sources of Disadvantage
Mentoring Treatment Interaction with Sources of Disadvantage
Stories from the HS Students



> hey Cambell, its Daniel M from west, one of the
student you hepled apply for college. If i remember
correctly, you told me to let you know what college i get
into. Well, i was recently accepted to NHTI in
Concord.
There are still some things i need to take care of, but its
nothing serious.
> P.S. Thanks for you help. Sorry for all the flirting
48
Cross Tab of Student Male and Assigned Male Mentor
assigned_male_mentor
Student is
Male
Total
0
1
0
76
73
149
1
61
95
156
137
168
305
Total
Mentors were assigned on a first come first served basis, but
when multiple arrivals occurred at the same time, we had a
modest bias towards same gender pairings. Regressions include
a dummy for being assigned to treatment but not showing up to
be assigned a mentor.
49
Is Cross Gender Mentoring More Effective?
Four Theories As To Why It Only Works for
Women
n
Treatment is actually a complement to ability or
persistence
– Look for other ways in which treatment might interact with
sources of advantage, help or test scores
n
Guys are primed to either be near the top of an
endeavor or quit
– Don’t respond well to feedback that they are in the middle
n
Labor economics
– Short run returns to college in NH are lower for guys
n
Are guys negatively selected into the treatment group?
– In part bc of $100 bonus?
51
Survey Data on Male And Female Study Subjects
N
1077
Women
Mean
0.57
N
1463
Men
Mean
0.57
SD
0.49
SD
0.50
Sat Math
95
42.77
8.33
133
45.42
9.38
Sat Verbal
95
44.02
8.48
133
45.14
10.47
Role Model College
43
0.16
0.37
61
0.21
0.41
Self Worth High
45
0.51
0.51
67
0.60
0.49
Remember Dates
44
0.30
0.46
67
0.31
0.47
Forget Deadlines
44
0.32
0.47
67
0.30
0.46
Have Control In Life
45
0.56
0.50
67
0.54
0.50
No Sat Data
52
Survey Data on Male And Female Study Subjects (Aspirations and
Expectations)
Aspire College4
Role Model College
Sat Aspire College4
Teachers Expect
College
Parents Want College4
Talked Future Plans
Teacher
Talked Future Plans
Parent
N
Women
Mean
SD
46
0.80
43
N
Men
Mean
SD
0.40
68
0.81
0.40
0.16
0.37
61
0.21
0.41
95
0.65
0.48
133
0.65
0.48
43
0.81
0.39
66
0.71
0.46
43
0.67
0.47
67
0.52
0.50
46
0.46
0.50
68
0.43
0.50
46
0.85
0.36
68
0.85
0.36
53
How Do Returns to College Differ for Men Versus Women in NH (Ages3140)?
(1)
Log Total
Income Men
NH
(2)
Log Total
Income
Women NH
(3)
Log Total
Income Men
All Other
States
(4)
Log Total
Income
Women All
Other States
High School
0.399**
(0.055)
0.409**
(0.103)
0.429**
(0.003)
0.477**
(0.005)
One to Three Years of
College
0.686**
(0.058)
0.667**
(0.104)
0.726**
(0.003)
0.761**
(0.005)
Four Plus Years of
College
1.081**
(0.055)
0.951**
(0.102)
1.251**
(0.003)
1.252**
(0.005)
Constant
9.955**
(0.051)
9.325**
(0.099)
9.748**
(0.003)
9.146**
(0.004)
4,674
0.140
77.69
4,570
0.049
33.52
1,129,025
0.168
15889
1,050,794
0.106
9056
0
7.53e-09
0
0
Observations
R-squared
F Test HS=Some
College
p-value
54
How Do Returns to College Differ for 21-30 year old Men Versus Women
in NH?
(1)
(2)
Log Total Log Total
Income
Income
Men NH Women NH
(3)
(4)
Log Total Log Total
Income
Income
Men All Women All
Other
Other
States
States
High School
0.343**
(0.075)
0.403**
(0.100)
0.345**
(0.004)
0.484**
(0.005)
One to Three Years
of College
0.339**
(0.078)
0.593**
(0.101)
0.405**
(0.004)
0.673**
(0.005)
Four Plus Years of
College
0.663**
(0.077)
0.848**
(0.099)
0.839**
(0.004)
1.193**
(0.005)
Observations
R-squared
F Test HS=Some
College
p-value
2925
0.033
0.00493
2898
0.046
14.49
828,881
0.055
414.6
794,172
0.095
3331
0.944
0.000144
0
0
55
Other Interventions Find Effects Differ By Gender?
n
Moving to Opportunity only saw decreases in crime
and increases in school engagement for girls
n
Angrist and Lavy bonuses in Israel for obtaining
Bagrut
n
Study one of Oreopoulos and Angrist bonuses plus
mentoring for U of Toronto students
– Study two did not find any effects
n
n
Dynarksi Ichino find bigger effects of offereing finacial
aid for women
Job Training programs affect women more
56
Other Interventions Find Effects Differ By Gender?
But not at all clear this is true for charter school studies
– Boston area charter studies find bigger effects for boys
Are effects bigger for students who are weaker in that
domain
– Did boys already attend college up to zero marginal returns?
– Girls need a shot of confidence or more help
Or is the treatment a complement to skill?
– Is it ultimately about personality measures, follow through
– Our intent is to collect ex-post survey measures of personality
traits, patients, follow through
57
Additional Interventions
Send one group a letter offering admission to their
closest public college and also community college
– We’ve had fantastic cooperation from admissions officers at
NH institutions (UNH, UNH Manchester, SNHU)
– Not much evidence of an effect yet
Compare to Hoxby Turner which has large effects from
letters sent to students in top 10% of SAT / ACT
distribution
– Our population is very different and perhaps needs much more
in person help hand-holding
58
Additional Interventions (2)
Have students meet with Admissions Officers from
Community College System
Have Lets Get Ready run the program.
Adds a serious SAT component
Ask how much results change when program is scaled up/ PI
is no longer involved
And when population of students is broadened
59
Additional Interventions (3)
Use text messaging to Community College students to
prevent Summer Melt, encourage persistence, help
students make the transition to 4 year institutions
60
Bottom Lines
Big differences by gender..critical to better understand
that..Quite possibly its not a general boy phenomenom
These marginal students persist at same rate as average
Program is enormously successful for immigrants
– And for resource poor high schools
– Suggests that we are in some way compensating for missing
component rather than reinforcing advantage
Difficult or impossible to evaluate this sort of program
using endogenous take up
May be real downsides to offering cash bonuses
May be real upsides to “expensive” in person help
61
Could We Estimate the Effects for Women Without a Randomized Control Group?
(1)
(2)
(3)
(4)
Women
Women
Women
Women
Four Year
Any Four Year
Any
College:
College:
College:
College:
Intended
Intended
Intended
Intended
Treatment Treatment Treatment Treatment
Versus All Versus All Versus All Versus All
Nonexperi Nonexperi Nonexperi Nonexperi
mental
mental
mental
mental
Assigned to Treatment Group
-0.162**
(0.030)
-0.004
(0.034)
Treated
Observations
R-squared
21,042
0.273
21,044
0.170
-0.138**
(0.040)
-0.041
(0.040)
20,934
0.272
20,936
0.171
62
Compare to Existing Work
HR Block (FAFSA) study
–
The Role of Simplification and Information in College Decisions: Results from the H&R
Block FAFSA Experiment
• Eric P. Bettinger, Bridget Terry Long, Philip Oreopoulos, and Lisa Sanbonmatsu
– Moved enrollment rates from 26.8 to 34.5 percent
– 7 percentage point effect in same age group
– We have a more intensive intervention and (arguably/
debatably) twice the effect size
COACH program (Avery and Kane)
Kane college mentoring in CA
– 5 percentage point effect on enrollment rates
Berman Ortiz and Bos (ongoing in LA Unified)
Phillips and Reber remote coaching in LA and CA
63
Estimate the Effects By Comparing Those That Accept Treatment to Those That Don't
(1)
(2)
(3)
(4)
Women 4 Women Any
Men Four
Men Four
Year College:
College: Year College: Year College:
Accepted
Accepted
Accepted
Accepted
Treatment
Treatment
Treatment
Treatment
Versus Didn't Versus Didn't Versus Didn't Versus Didn't
Accepted Treatment
0.076
(0.091)
-0.035
(0.092)
0.070+
(0.035)
0.017
(0.067)
Constant
0.613*
(0.244)
0.228
(0.466)
-0.477**
(0.138)
-0.601**
(0.040)
259
0.167
259
0.213
341
0.176
341
0.231
Observations
R-squared
64
Modifications in Follow Up Work
Try W/o the $100 bonus
– May actually increase treatment on treated for men
– Eliminate noise of people signing up for wrong reason
Have Lets Get Ready run the intervention
– Include SAT prep. Include juniors
Try letters of encouragement from college admissions
departments
Extensive pre and post surveys to learn more about
students personalities, career plans, stick-toitiveness
Try emails and texts to spur students to apply
65
Partnered with NH DOE
 NH DOE has been very supportive and flexible
– Commissioners and staff have leant their support to the
project
 Key pre-treatment and outcome variables collected by
NH DOE
– Nat’l Student Clearinghouse subscription
– Test scores and demographics
 Part of a larger Data Warehouse project underway
 Future projects being designed jointly
66
What our High Schools Look Like
67