Student Quality and Reported Student Quality

advertisement
Voluntary Disclosure,
Inference and the
Strategic Behavior of
Colleges
Michael Conlin Michigan State University
Stacy Dickert-Conlin Michigan State University
Gabrielle Chapman Syracuse University
Oregon State University– May 2009
Optional SAT Policies
“I SOMETIMES think I should write a handbook for college
admission officials titled “How to Play the U.S. News & World
Report Ranking Game, and Win!” I would devote the first
chapter to a tactic called “SAT optional.”
The idea is simple: tell applicants that they can choose
whether or not to submit their SAT or ACT scores.
Predictably, those applicants with low scores or those who
know that they score poorly on standardized aptitude tests
will not submit. Those with high scores will submit. When
the college computes the mean SAT or ACT score of its
enrolled students, voilà! its average will have risen. And so
too, it can fondly hope, will its status in the annual U.S.
News & World Report’s college rankings.”
Colin Driver, President of Reed College, New York Times, 2006
Optional SAT Policies
The thesis, first stated last year by The New Republic,
is that colleges are being less than honest about why
they abolish requirements that applicants submit their
SAT scores. Behind the rhetoric about "enhancing
diversity" and creating a more "holistic approach" to
admissions, the theory goes, many colleges "go
optional" on the SAT to improve their rankings. The
logic is rather simple: At an SAT-optional college,
students with higher scores are far more likely to
submit them, raising the institution's mean SAT score
and hence the heavily test-influenced rankings.
Brownstein (2001) in The Chronicle of Higher Education
U.S. News & World Report
(Criteria and weights for rankings colleges)
Prevalence of Optional Policy

As of Spring 2007, more than 700
colleges have SAT- or ACT- optional
policies.

24 of the top 100 liberal arts colleges
ranked by U.S. News & World are
SAT- or ACT- optional.
Overview
Research Questions
 Data
 Voluntary Disclosure Literature
 Reduced Form Results
- Colleges’ Decisions to Accept

- Applicants’ Decisions to Submit SAT I
Structural Framework
 Future Work

Research Questions

Are Colleges’ Admission Decisions influenced
by their incentive to increase their ranking in
publications like U.S. News & World Report?

Do Applicants behave strategically when
deciding whether to submit their SAT I scores
and how does this inform the voluntary
disclosure literature?

What is the college’s inference for applicants
who choose not to submit their SAT I scores?
College Data

Application data for 2 liberal arts schools in north
east
Each with approximately 1800 students
enrolled.
Both report a typical SAT I score in the upper 1200s/1600.
College X: 2 years ≈ 5 years after the optional policy
was instituted.
College Y: the year after the optional policy was instituted.



Numerical Score from Admission Department
Acceptance and Enrollment Decisions.
Performance Measures for those who Enroll.
College Board Data
SAT scores for those who elected not
to submit them to the college.
 Student Descriptive Questionnaire
(SDQ)
SATII are Subject Exams –

SAT II Scores
20 of them
 Self Reported income
Also have High School GPA from
 High school GPA
colleges but not standardized
 High school activities

Optional SAT I policies

College X
 Whether or not applicant submits SAT I scores,
require applicants to choose between submitting the
ACT scores or three SAT II: Subject Tests.

College Y
 Along with their SAT I scores, applicants can submit
scores from their SAT II exams, ACT exam, and/or
Advanced Placement exams. College Y applicants
are required to submit at least one of these scores if
they choose not to submit their SAT I scores.
Summary Statistics

15.3 percent of the 7,023 applicants to
College X choose not to submit SAT I
scores.

24.1 percent of the 3,054 applicants to
College X choose not to submit SAT I
scores.
Table 1 Summary Statistics
College X
N=6,567
SAT I Combined
(math+verbal) Score
SAT I Verbal Score
SAT I Math Score
SAT II Score(s) available
(1=yes)
Average SAT II Score (when
available)
ACT Score(s) available
(1=yes)
Average ACT Score (when
available)
Chose to
Submit
SAT I
(N=5550)
1272
(124)
Chose Not
to Submit
SAT I
(N=1017)
1139
(116)
641
(74)
570
(67)
632
(70)
569
(67)
0.856
(0.351)
0.815
(0.388)
633
(68)
590
(68)
0.015
(0.122)
24.6
(3.7)
College Y
N=3,504
Chose to Chose Not
SS
Submit
to Submit SS
SAT I
SAT I
(N=2659) (N=845)
1267
1229
***
(144)
(120)
***
***
633
(84)
610
(68)
***
***
634
(78)
619
(72)
***
***
0.677
(0.468)
0.804
(0.398)
***
***
632
(76)
632
(61)
0.013
(0.112)
0.200
(0.400)
0.141
(0.348)
23.7
(2.4)
26.6
(3.8)
26.1
(3.3)
***
Table 1 Summary Stats (cont)
College X
(N=6,567)
Chose to
Submit
SAT I
College Y
(N=3,504)
0.477
(0.500)
Chose
Not to
Submit
SAT I
0.503
(0.500)
0.657
(0.475)
0.778
(0.416)
Legacy (1=yes)
0.024
(0.153)
0.022
(0.146)
0.062
(0.242)
0.053
(0.225)
Apply Early
0.059
(0.235)
0.120
(0.325)
0.108
(0.311)
0.097
(0.296)
Intend to Apply for
Financial Aid
0.499
(0.500)
0.515
(0.500)
0.594
(0.491)
0.505
(0.500)
Attended Private HS
Female Student
SS
***
Chose to
Submit
SAT I
Chose Not
to Submit
SAT I
0.353
(0.478)
0.431
(0.495)
***
0.487
(0.500)
0.548
(0.498)
***
SS
Table 1 Summary Stats (cont)
College X (N=6,567)
Chose to
Submit SAT I
Chose Not to
Submit SAT I
White
0.835
(0.371)
African American
College Y (N=3,504)
Chose to
Submit SAT I
Chose Not to
Submit SAT I
0.834
(0.372)
0.877
(0.328)
0.859
(0.348)
0.029
(0.168)
0.031
(0.175)
0.032
(0.175)
0.049
(0.215)
0.003
(0.052)
0.007
(0.083)
0.002
(0.043)
0.002
(0.049)
Asian American
0.043
(0.202)
0.041
(0.199)
0.054
(0.226)
0.041
(0.199)
Hispanic
0.037
(0.190)
0.046
(0.210)
0.035
(0.185)
0.047
(0.212)
Unknown Race
0.053
(0.224)
0.040
(0.197)
Native American
SS
**
*
SS
**
Voluntary Disclosure: Theory

Grossman & Hart (1980) – when disclosure is
costless, complete unraveling occurs.

Grossman (1981) and Milgrom (1981) generalizes Grossman & Hart (1980)

Jovanovic (1982) – when disclosure is costly,
unraveling is not complete and it may not be
socially optimal to mandate disclosure
Voluntary Disclosure Example


Student i has the following
probability distribution in term
of SAT I scores.
When disclosure is costless,
Bayesian Nash Equilibrium
results in every type except
the worst disclosing and the
worst being indifferent
between disclosing and not
disclosing.
SAT I Score
Probability
1300
0.2
1200
0.4
1100
0.3
1000
0.1
Expected SAT I Score
1300(.2)+1200(.4)+1100(.3)+1000(.1)
=1170
Voluntary Disclosure Models

Comments:
Distribution depends on student characteristics that
are observable to the school such as high school
GPA.
With positive disclosure costs, the “unraveling” is not
complete and only the types with the lower SAT I
scores do not disclose.

Assumptions:
Common Knowledge.
Colleges use Bayesian Updating to Infer SAT I Score
of those who do not Submit/Disclose
Colleges’ incentives to admit an applicant is only a
function of his/her actual SAT I score (not whether the
applicant submits the score)
Voluntary Disclosure: Theory
o Eyster and Rabin (Econometrica,
2005) propose a new equilibrium
concept which they call a Cursed
Equilibrium. College correctly
predicts the distribution of the other
players’ actions but underestimates
the degree these actions are
correlated with the other players’
private information.
SAT I Score
Probability
1300
0.2
1200
0.4
1100
0.3
1000
0.1
“Fully” Cursed Equilibrium (χ=1)– College infers if applicant doesn’t disclose
that his/her expected SAT I score is
1300(.2)+1200(.4)+1100(.3)+1000(.1)=1170
“Partially” Cursed Equilibrium (χ=.4 for example)– College infers if applicant
doesn’t disclose that his/her expected SAT I score is
(1-.4) [(1100(.3)+1000(.1))/.4]+ (.4)1170 = 1113
Voluntary Disclosure: Empirical

Mathios (2000) – fat content in salad
dressings.

Jin and Leslie (2003) – hygiene quality grade
cards for restaurants in Los Angeles.

Jin (2004) – HMO accreditation and summary
statistics.

Robinson and Monk (2005) – applicants
submitting SAT scores to Mount Holyoke
College.
Colleges’ Incentive to Institute Optional
SAT Policy : Table 2
SAT I Score (1600)
– all applicants
College X
Chose to
Chose Not
Submit
to Submit
SAT I
SAT I
1272
1139
(124)
(116)
[5550]
[1017]
Probability of
Acceptance
.418
(0.493)
[5550]
.395
(0.489)
[1017]
SAT I Score
conditional on
Acceptance
1323
(107)
[2320]
1172
(99)
[402]
SAT I Score
conditional on
Enrollment
1281
(107)
[647]
Predicted SAT I
Score* (based on
those that want
SAT I considered)
1272
(89)
[5547]
SS
***
College Y
Chose to
Chose Not
Submit
to Submit
SAT I
SAT I
1267
1229
(144)
(120)
[2659]
[845]
SS
***
.445
(0.497)
[2659]
.488
(0.500)
[845]
**
***
1344
(115)
[1182]
1260
(103)
[412]
***
1155
(100)
[185]
***
1299
(113)
[351]
1227
(97)
[135]
***
1219#
(82)
[1017]
***
1263
(78)
[2659]
1251#
(90)
[845]
***
SS, statistical significance ; *** statistically different at 1% level, ** statistically different at 5% level, *
# Statistically different than the actual SATI score at the 1% level.
*Regression Results are in Table A1 of the Appendix.
College’s Acceptance Decision
Table 3 Columns I and III: Probit Regression
(Dependent Variable =1 if accept)
ME =.11
College X
0.2939*
(0.0214)
College Y
0.6164*
ME =.24
(0.0282)
ME =-.16
-0.3989*
(0.0552)
-0.1833**
(0.0813) ME =-.07
Submitted SAT2 Score
-2.6954*
(0.2688)
-2.4031*
(0.4883)
Submitted SAT2 Score* SAT2 Score/100
0.4554*
(0.0422)
0.3688* ME =.14
(0.0759)
SAT1 Score/100 (16 max)
Submitted SAT1 Score
ME =.17
Submitted ACT Score
Submitted ACT Score*ACT Score
-0.2171**
(0.1079)
ME =.01 0.0129*
(0.0042)
2.3137**
(1.1182)
-0.0684 ME =-.03
(0.0422)
Possible Explanations for Negative
Coefficient Estimate Associated with
Submit SATI
1.
2.
3.
For those who don’t submit, school
might be “overestimating” their score
Not submitting may be correlated
with error term – applicants who do
not submit are “more mature” or are
athletes.
School is behaving strategically when
deciding who to accept.
College’s Acceptance Decision
Table 3 Columns I and III (cont.)
College X
0.1250*
(0.0400)
College Y
0.2016*
(0.0574)
ME =.08
-0.5383*
(0.0393)
0.3179*
(0.0520)
ME =.12
No High School GPA reported
0.4608*
(0.0896)
0.5356*
(0.1270)
High School GPA A+
0.9152*
(0.1179)
0.9240*
(0.1684)
High School GPA A
0.8472*
(0.0858)
0.8642*
(0.1314)
High School GPA A-
0.6509*
(0.0762)
0.7023*
(0.1207)
Note: High School High School GPA B+
GPA B is omitted
category
High School GPA B-
0.4121*
(0.0755)
0.5081*
(0.1182)
-0.3198**
(0.1578)
-0.2292
(0.2402)
-0.5827***
(0.3422)
0.1748
(0.5214)
Attended Private High School
ME =.05
Female
High School GPA C
ME =-.21
College’s Acceptance Decision
Table 3 Columns I and III (cont.)
College X
0.1916*
(0.0526)
College Y
0.0802
(0.0777)
0.3311*
(0.0728)
-0.1335
(0.1145)
0.1132***
(0.0580)
-0.2186**
(0.0910)
0.7501*
(0.1220)
0.3033*
(0.1051)
ME =.12
1.8209*
(0.0859)
1.2020*
(0.0846)
ME =.44
-0.0814**
(0.0409)
-0.2519*
(0.0576)
ME =-.10
1.3637*
(0.1079)
1.8549*
(0.1546)
ME =.55
Native American
0.1560
(0.3197)
1.0906***
(0.5791)
Asian
0.5073*
(0.0855)
1.1541*
(0.1185)
Hispanic
0.5243*
(0.0938)
1.5306*
(0.1387)
Missing Income
Income <50K
50K <Income <100K
Legacy (1=yes)
Applied Early Decision
ME =.29
ME =.58
Intend to Apply for Financial Aid
ME =-.03
African American
Note: White
is omitted
category
ME =.48
Is the college more likely to accept
Applicant A or Applicant B if influenced
by Ranking Organizations?

Applicant A
White, Female, HS GPA is A-, Class Rank in top 10%,
Private High School, Legacy, Submitted SATII of 600,
Submitted SAT I of 1400.

Applicant B
White, Female, HS GPA is A-, Class Rank in top 10%,
Private High School, Legacy, Submitted SATII of 600,
Did not Submitted SAT I but college infers an SAT I
score of 1400 (based on observables to college).
Is the college more likely to accept
Applicant C or Applicant D if influenced
by Ranking Organizations?

Applicant C
White, Female, HS GPA is B, Class Rank in top quintile,
Private High School, Legacy, Submitted SATII of 550,
Submitted SAT I of 1100.

Applicant D
White, Female, HS GPA is B, Class Rank is top quintile,
Private High School, Legacy, Submitted SATII of 550,
Did not Submitted SAT I but college infers an SAT I
score of 1100 (based on observables to college).
College’s Acceptance Decision
Table 3 Columns II and IV: Probit Regression
College X
0.2207*
(0.0456)
College Y
0.3978*
(0.0844)
Submitted SAT1 Score
-1.3908**
(0.5516)
-3.0446*
(1.0482)
Submitted SAT1 Score* SAT1 Score/100
0.0849***
(0.0470)
0.2306*
(0.0843)
Submitted SAT2 Score
-2.6729*
(0.2690)
-2.4895*
(0.4895)
Submitted SAT2 Score* SAT2 Score/100
0.4517*
(0.0422)
0.3826*
(0.0761)
-0.2254**
(0.1080)
1.0074
(1.2088)
0.0134*
(0.0042)
-0.0220
(0.0452)
SAT1 Score/100 (16 max)
ME =.03
Submitted ACT Score
Submitted ACT Score*ACT Score
ME =.09
College’s Acceptance Decision
Table 4 : Predicted rather than Actual SAT I Score
In the spirit of Eyster & Rabin’s
“fully” cursed equilibrium.
Predicted SAT1 Score/100 (16 max)
Requested school use SAT1 Score
College X
College Y
I
0.3095***
(0.0234)
II
0.2065***
(0.0652)
III
0.6260***
(0.0291)
IV
0.3066***
(0.0897)
-0.1631***
(0.0518)
-1.4463*
(0.7602)
-0.0340
(0.0806)
-4.1312***
(1.0941)
Requested school use SAT1 Score*
SAT1 Score/100
0.1039*
(0.0614)
0.3248***
(0.0864)
Requested school use SAT2 Score
-2.4979***
(0.2795)
-2.5905***
(0.2852)
-2.2793***
(0.4887)
-2.5132***
(0.4947)
Requested school use SAT2 Score*
SAT2 Score/100
0.4256***
(0.0439)
0.4401***
(0.0448)
0.3468***
(0.0760)
0.3852***
(0.0770)
-0.1884*
(0.1082)
-0.2157***
(0.1095)
0.4580
(1.0611)
-0.5234
(1.0994)
0.0130***
(0.0042)
0.0139***
(0.0042)
0.0051
(0.0401)
0.0360
(0.0412)
Requested school use ACT Score
Requested school use ACT Score*ACT
Score
Interpretation of Point Estimates

College X
An applicant who scores a 1,000 on the SAT I score decreases her
probability of being accepted by 9.7 percentage points if she
submits her score while an applicant who scores a 1,500 increases
her probability of being accepted by 3.8 percentage points if she
submits.

College Y
An applicant who scores a 1,000 on the SAT I score decreases her
probability of being accepted by 16.8 percentage points if she
submits her score while an applicant who scores a 1,500 increases
her probability of being accepted by 12.6 percentage points if she
submits.
Interpretation of Point Estimates

College X
Applicants who submit their SAT I score are less likely
to be accepted by College X if their SAT I score is less
than 1,392 and are more likely to be accepted if their
score is greater than 1,392.

College Y
Applicants who submit their SAT I score are less likely
to be accepted if their SAT I score is less than 1,272
and are more likely to be accepted if their score is
greater than 1,272.
Possible Explanations for Negative
Coefficient Estimate Associated with
Submit SATI
1.
2.
3.
For those who don’t submit, school
might be “overestimating” their score
Not submitting may be correlated
with error term – applicants who do
not submit are “more mature” or are
athletes.
School is behaving strategically when
deciding who to accept.
Submission on College X Performance
Table 5B
Applicants’ Disclosure Decisions:
Table 2
SAT I Score (1600)
– all applicants
College X
Chose to
Chose Not
Submit
to Submit
SAT I
SAT I
1272
1139
(124)
(116)
[5550]
[1017]
Probability of
Acceptance
.418
(0.493)
[5550]
.395
(0.489)
[1017]
SAT I Score
conditional on
Acceptance
1323
(107)
[2320]
1172
(99)
[402]
SAT I Score
conditional on
Enrollment
1281
(107)
[647]
Predicted SAT I
Score* (based on
those that want
SAT I considered)
1272
(89)
[5547]
SS
***
College Y
Chose to
Chose Not
Submit
to Submit
SAT I
SAT I
1267
1229
(144)
(120)
[2659]
[845]
SS
***
.445
(0.497)
[2659]
.488
(0.500)
[845]
**
***
1344
(115)
[1182]
1260
(103)
[412]
***
1155
(100)
[185]
***
1299
(113)
[351]
1227
(97)
[135]
***
1219#
(82)
[1017]
***
1263
(78)
[2659]
1251#
(90)
[845]
***
SS, statistical significance ; *** statistically different at 1% level, ** statistically different at 5% level, *
# Statistically different than the actual SATI score at the 1% level.
*Regression Results are in Table A1 of the Appendix.
Figure 2: Predicted versus Actual SAT I Score
for those who Chose not to Submit
Figure 2: Predicted versus Actual SAT I Score
for those who Chose not to Submit
Conclusions from Reduced Form
1.
College admission departments are behaving
strategically by more (less) likely accepting applicants
who do not submit their SAT I scores if submitting their
scores would decrease (increase) the average SAT I
score the colleges report to the ranking organizations.
2.
Applicants are behaving strategically by choosing not to
reveal their SAT I scores if they are below a value one
might predict based on their other observable
characteristics.
Note that the reduced form estimates do not address
directly the college’s inference for those applicants who
do not submit.
Model and Structural Estimation
Summary Statistics for College X
N=324
N=122
N=5216
N=895
Summary Statistics for College Y
N=294
N=83
N=2440
N=785
Notation

μ(Xi)+εap +εen ,expected utility from attending
the college for applicant i
 μ(Xi) is portion of the applicant specific preferences
for attending College X that depends on the
observables variable.
 εap is unobservable applicant specific preferences
for attending College X that is known to the
applicant at the time she submits her application.
 εen is unobservable applicant specific preferences
for attending College X that is known to the
applicant at the time she makes enrollment
decision but not at time she submits her application
Notation (cont)




UR ,expected utility if applicant does not
attend College X and does not apply early
decision at College X.
UR-C , expected utility if applicant does not
attend College X and does apply early
decision at College X.
K , fixed cost of applying
εen , unobserved cost of submitting SAT I
Applicant’s Decision to apply early
decision and/or submit SAT I

Expected Utility if applicant applies early and submits
Pa(X,ed,s) Pe(X,ed,s) [μ(X)+εap+εen]+
[1-Pa(X,ed,s) Pe(X,ed,s)][UR-C]-K- εs

Expected Utility if applicant doesn’t apply early or submit
Pa(X,ned,ns) Pe(X,ned,ns) [μ(X)+εap+εen]+
[1-Pa(X,ned,ns) Pe(X,ned,ns)][UR]-K
Apply early, don’t submit and Don’t apply early, submit are analogous
Applicant applies early decision and
submits [assuming E(εen)=0] if
εs +[Κ1]εap > [Κ1][-μ(X)+C] ,
[Κ2]εap > [Κ2][-μ(X)]+[1- Pa(X,ed,ns)Pe(X,ed,ns)][C]
and
εs+[Κ3]εap > [Κ3][-μ(X)]+[1Pa(X,ed,ns)Pe(X,ed,ns)][C]
Κ1=Pa(X,ed,ns)Pe(X,ed,ns)-Pa(X,ed,s)Pe(X,ed,s),
Κ2=Pa(X,ed,ns)Pe(X,ed,ns)-Pa(X,ned,ns)Pe(X, ned,ns)
Κ3=Pa(X,ed,ns)Pe(X,ed,ns)-Pa(X,ned,s)Pe(X,ned,s)
Whether to Apply Early Decision
and/or Submit SATI Score
Literature on College
Objective Function



Ehrenberg (1999) single well-defined objective function may
explain “fairly well the behavior of small liberal arts colleges…”
(page 101).
Epple, Romano, and Seig (2006) GE model
 assume a school maximizes quality (average quality of the
student body, school expenditure per student, and the mean
income of the student body)
 s.t. balanced budget constraint and a fixed student body size.
Our Model
 To account for the college’s concern for the quality of its current
and future students and the understanding that future student
quality depends on the college’s ranking, we allow the college’s
objective function to depend on the perceived ability of the
incoming students, the “reported” ability of these students, and
the demographic characteristics of the student body.
College’s Decision to Accept Applicant
dsoisdsdf
College accepts applicant i if:
Pe(Xi,k,l) [ΠP(X+iP)+εqi + ΠR(X+iR)+ ΠD(X+iD)]
+(1- Pe(Xi,k,l)) [ΠP(X-iP)+ ΠR(X-iR)+ ΠD(X-iD)]
> ΠP(XriP)+ ΠR(XriR)+ ΠD(XriD)
Download