Voluntary Disclosure, Inference and the Strategic Behavior of Colleges Michael Conlin Michigan State University Stacy Dickert-Conlin Michigan State University Gabrielle Chapman Syracuse University Oregon State University– May 2009 Optional SAT Policies “I SOMETIMES think I should write a handbook for college admission officials titled “How to Play the U.S. News & World Report Ranking Game, and Win!” I would devote the first chapter to a tactic called “SAT optional.” The idea is simple: tell applicants that they can choose whether or not to submit their SAT or ACT scores. Predictably, those applicants with low scores or those who know that they score poorly on standardized aptitude tests will not submit. Those with high scores will submit. When the college computes the mean SAT or ACT score of its enrolled students, voilà! its average will have risen. And so too, it can fondly hope, will its status in the annual U.S. News & World Report’s college rankings.” Colin Driver, President of Reed College, New York Times, 2006 Optional SAT Policies The thesis, first stated last year by The New Republic, is that colleges are being less than honest about why they abolish requirements that applicants submit their SAT scores. Behind the rhetoric about "enhancing diversity" and creating a more "holistic approach" to admissions, the theory goes, many colleges "go optional" on the SAT to improve their rankings. The logic is rather simple: At an SAT-optional college, students with higher scores are far more likely to submit them, raising the institution's mean SAT score and hence the heavily test-influenced rankings. Brownstein (2001) in The Chronicle of Higher Education U.S. News & World Report (Criteria and weights for rankings colleges) Prevalence of Optional Policy As of Spring 2007, more than 700 colleges have SAT- or ACT- optional policies. 24 of the top 100 liberal arts colleges ranked by U.S. News & World are SAT- or ACT- optional. Overview Research Questions Data Voluntary Disclosure Literature Reduced Form Results - Colleges’ Decisions to Accept - Applicants’ Decisions to Submit SAT I Structural Framework Future Work Research Questions Are Colleges’ Admission Decisions influenced by their incentive to increase their ranking in publications like U.S. News & World Report? Do Applicants behave strategically when deciding whether to submit their SAT I scores and how does this inform the voluntary disclosure literature? What is the college’s inference for applicants who choose not to submit their SAT I scores? College Data Application data for 2 liberal arts schools in north east Each with approximately 1800 students enrolled. Both report a typical SAT I score in the upper 1200s/1600. College X: 2 years ≈ 5 years after the optional policy was instituted. College Y: the year after the optional policy was instituted. Numerical Score from Admission Department Acceptance and Enrollment Decisions. Performance Measures for those who Enroll. College Board Data SAT scores for those who elected not to submit them to the college. Student Descriptive Questionnaire (SDQ) SATII are Subject Exams – SAT II Scores 20 of them Self Reported income Also have High School GPA from High school GPA colleges but not standardized High school activities Optional SAT I policies College X Whether or not applicant submits SAT I scores, require applicants to choose between submitting the ACT scores or three SAT II: Subject Tests. College Y Along with their SAT I scores, applicants can submit scores from their SAT II exams, ACT exam, and/or Advanced Placement exams. College Y applicants are required to submit at least one of these scores if they choose not to submit their SAT I scores. Summary Statistics 15.3 percent of the 7,023 applicants to College X choose not to submit SAT I scores. 24.1 percent of the 3,054 applicants to College X choose not to submit SAT I scores. Table 1 Summary Statistics College X N=6,567 SAT I Combined (math+verbal) Score SAT I Verbal Score SAT I Math Score SAT II Score(s) available (1=yes) Average SAT II Score (when available) ACT Score(s) available (1=yes) Average ACT Score (when available) Chose to Submit SAT I (N=5550) 1272 (124) Chose Not to Submit SAT I (N=1017) 1139 (116) 641 (74) 570 (67) 632 (70) 569 (67) 0.856 (0.351) 0.815 (0.388) 633 (68) 590 (68) 0.015 (0.122) 24.6 (3.7) College Y N=3,504 Chose to Chose Not SS Submit to Submit SS SAT I SAT I (N=2659) (N=845) 1267 1229 *** (144) (120) *** *** 633 (84) 610 (68) *** *** 634 (78) 619 (72) *** *** 0.677 (0.468) 0.804 (0.398) *** *** 632 (76) 632 (61) 0.013 (0.112) 0.200 (0.400) 0.141 (0.348) 23.7 (2.4) 26.6 (3.8) 26.1 (3.3) *** Table 1 Summary Stats (cont) College X (N=6,567) Chose to Submit SAT I College Y (N=3,504) 0.477 (0.500) Chose Not to Submit SAT I 0.503 (0.500) 0.657 (0.475) 0.778 (0.416) Legacy (1=yes) 0.024 (0.153) 0.022 (0.146) 0.062 (0.242) 0.053 (0.225) Apply Early 0.059 (0.235) 0.120 (0.325) 0.108 (0.311) 0.097 (0.296) Intend to Apply for Financial Aid 0.499 (0.500) 0.515 (0.500) 0.594 (0.491) 0.505 (0.500) Attended Private HS Female Student SS *** Chose to Submit SAT I Chose Not to Submit SAT I 0.353 (0.478) 0.431 (0.495) *** 0.487 (0.500) 0.548 (0.498) *** SS Table 1 Summary Stats (cont) College X (N=6,567) Chose to Submit SAT I Chose Not to Submit SAT I White 0.835 (0.371) African American College Y (N=3,504) Chose to Submit SAT I Chose Not to Submit SAT I 0.834 (0.372) 0.877 (0.328) 0.859 (0.348) 0.029 (0.168) 0.031 (0.175) 0.032 (0.175) 0.049 (0.215) 0.003 (0.052) 0.007 (0.083) 0.002 (0.043) 0.002 (0.049) Asian American 0.043 (0.202) 0.041 (0.199) 0.054 (0.226) 0.041 (0.199) Hispanic 0.037 (0.190) 0.046 (0.210) 0.035 (0.185) 0.047 (0.212) Unknown Race 0.053 (0.224) 0.040 (0.197) Native American SS ** * SS ** Voluntary Disclosure: Theory Grossman & Hart (1980) – when disclosure is costless, complete unraveling occurs. Grossman (1981) and Milgrom (1981) generalizes Grossman & Hart (1980) Jovanovic (1982) – when disclosure is costly, unraveling is not complete and it may not be socially optimal to mandate disclosure Voluntary Disclosure Example Student i has the following probability distribution in term of SAT I scores. When disclosure is costless, Bayesian Nash Equilibrium results in every type except the worst disclosing and the worst being indifferent between disclosing and not disclosing. SAT I Score Probability 1300 0.2 1200 0.4 1100 0.3 1000 0.1 Expected SAT I Score 1300(.2)+1200(.4)+1100(.3)+1000(.1) =1170 Voluntary Disclosure Models Comments: Distribution depends on student characteristics that are observable to the school such as high school GPA. With positive disclosure costs, the “unraveling” is not complete and only the types with the lower SAT I scores do not disclose. Assumptions: Common Knowledge. Colleges use Bayesian Updating to Infer SAT I Score of those who do not Submit/Disclose Colleges’ incentives to admit an applicant is only a function of his/her actual SAT I score (not whether the applicant submits the score) Voluntary Disclosure: Theory o Eyster and Rabin (Econometrica, 2005) propose a new equilibrium concept which they call a Cursed Equilibrium. College correctly predicts the distribution of the other players’ actions but underestimates the degree these actions are correlated with the other players’ private information. SAT I Score Probability 1300 0.2 1200 0.4 1100 0.3 1000 0.1 “Fully” Cursed Equilibrium (χ=1)– College infers if applicant doesn’t disclose that his/her expected SAT I score is 1300(.2)+1200(.4)+1100(.3)+1000(.1)=1170 “Partially” Cursed Equilibrium (χ=.4 for example)– College infers if applicant doesn’t disclose that his/her expected SAT I score is (1-.4) [(1100(.3)+1000(.1))/.4]+ (.4)1170 = 1113 Voluntary Disclosure: Empirical Mathios (2000) – fat content in salad dressings. Jin and Leslie (2003) – hygiene quality grade cards for restaurants in Los Angeles. Jin (2004) – HMO accreditation and summary statistics. Robinson and Monk (2005) – applicants submitting SAT scores to Mount Holyoke College. Colleges’ Incentive to Institute Optional SAT Policy : Table 2 SAT I Score (1600) – all applicants College X Chose to Chose Not Submit to Submit SAT I SAT I 1272 1139 (124) (116) [5550] [1017] Probability of Acceptance .418 (0.493) [5550] .395 (0.489) [1017] SAT I Score conditional on Acceptance 1323 (107) [2320] 1172 (99) [402] SAT I Score conditional on Enrollment 1281 (107) [647] Predicted SAT I Score* (based on those that want SAT I considered) 1272 (89) [5547] SS *** College Y Chose to Chose Not Submit to Submit SAT I SAT I 1267 1229 (144) (120) [2659] [845] SS *** .445 (0.497) [2659] .488 (0.500) [845] ** *** 1344 (115) [1182] 1260 (103) [412] *** 1155 (100) [185] *** 1299 (113) [351] 1227 (97) [135] *** 1219# (82) [1017] *** 1263 (78) [2659] 1251# (90) [845] *** SS, statistical significance ; *** statistically different at 1% level, ** statistically different at 5% level, * # Statistically different than the actual SATI score at the 1% level. *Regression Results are in Table A1 of the Appendix. College’s Acceptance Decision Table 3 Columns I and III: Probit Regression (Dependent Variable =1 if accept) ME =.11 College X 0.2939* (0.0214) College Y 0.6164* ME =.24 (0.0282) ME =-.16 -0.3989* (0.0552) -0.1833** (0.0813) ME =-.07 Submitted SAT2 Score -2.6954* (0.2688) -2.4031* (0.4883) Submitted SAT2 Score* SAT2 Score/100 0.4554* (0.0422) 0.3688* ME =.14 (0.0759) SAT1 Score/100 (16 max) Submitted SAT1 Score ME =.17 Submitted ACT Score Submitted ACT Score*ACT Score -0.2171** (0.1079) ME =.01 0.0129* (0.0042) 2.3137** (1.1182) -0.0684 ME =-.03 (0.0422) Possible Explanations for Negative Coefficient Estimate Associated with Submit SATI 1. 2. 3. For those who don’t submit, school might be “overestimating” their score Not submitting may be correlated with error term – applicants who do not submit are “more mature” or are athletes. School is behaving strategically when deciding who to accept. College’s Acceptance Decision Table 3 Columns I and III (cont.) College X 0.1250* (0.0400) College Y 0.2016* (0.0574) ME =.08 -0.5383* (0.0393) 0.3179* (0.0520) ME =.12 No High School GPA reported 0.4608* (0.0896) 0.5356* (0.1270) High School GPA A+ 0.9152* (0.1179) 0.9240* (0.1684) High School GPA A 0.8472* (0.0858) 0.8642* (0.1314) High School GPA A- 0.6509* (0.0762) 0.7023* (0.1207) Note: High School High School GPA B+ GPA B is omitted category High School GPA B- 0.4121* (0.0755) 0.5081* (0.1182) -0.3198** (0.1578) -0.2292 (0.2402) -0.5827*** (0.3422) 0.1748 (0.5214) Attended Private High School ME =.05 Female High School GPA C ME =-.21 College’s Acceptance Decision Table 3 Columns I and III (cont.) College X 0.1916* (0.0526) College Y 0.0802 (0.0777) 0.3311* (0.0728) -0.1335 (0.1145) 0.1132*** (0.0580) -0.2186** (0.0910) 0.7501* (0.1220) 0.3033* (0.1051) ME =.12 1.8209* (0.0859) 1.2020* (0.0846) ME =.44 -0.0814** (0.0409) -0.2519* (0.0576) ME =-.10 1.3637* (0.1079) 1.8549* (0.1546) ME =.55 Native American 0.1560 (0.3197) 1.0906*** (0.5791) Asian 0.5073* (0.0855) 1.1541* (0.1185) Hispanic 0.5243* (0.0938) 1.5306* (0.1387) Missing Income Income <50K 50K <Income <100K Legacy (1=yes) Applied Early Decision ME =.29 ME =.58 Intend to Apply for Financial Aid ME =-.03 African American Note: White is omitted category ME =.48 Is the college more likely to accept Applicant A or Applicant B if influenced by Ranking Organizations? Applicant A White, Female, HS GPA is A-, Class Rank in top 10%, Private High School, Legacy, Submitted SATII of 600, Submitted SAT I of 1400. Applicant B White, Female, HS GPA is A-, Class Rank in top 10%, Private High School, Legacy, Submitted SATII of 600, Did not Submitted SAT I but college infers an SAT I score of 1400 (based on observables to college). Is the college more likely to accept Applicant C or Applicant D if influenced by Ranking Organizations? Applicant C White, Female, HS GPA is B, Class Rank in top quintile, Private High School, Legacy, Submitted SATII of 550, Submitted SAT I of 1100. Applicant D White, Female, HS GPA is B, Class Rank is top quintile, Private High School, Legacy, Submitted SATII of 550, Did not Submitted SAT I but college infers an SAT I score of 1100 (based on observables to college). College’s Acceptance Decision Table 3 Columns II and IV: Probit Regression College X 0.2207* (0.0456) College Y 0.3978* (0.0844) Submitted SAT1 Score -1.3908** (0.5516) -3.0446* (1.0482) Submitted SAT1 Score* SAT1 Score/100 0.0849*** (0.0470) 0.2306* (0.0843) Submitted SAT2 Score -2.6729* (0.2690) -2.4895* (0.4895) Submitted SAT2 Score* SAT2 Score/100 0.4517* (0.0422) 0.3826* (0.0761) -0.2254** (0.1080) 1.0074 (1.2088) 0.0134* (0.0042) -0.0220 (0.0452) SAT1 Score/100 (16 max) ME =.03 Submitted ACT Score Submitted ACT Score*ACT Score ME =.09 College’s Acceptance Decision Table 4 : Predicted rather than Actual SAT I Score In the spirit of Eyster & Rabin’s “fully” cursed equilibrium. Predicted SAT1 Score/100 (16 max) Requested school use SAT1 Score College X College Y I 0.3095*** (0.0234) II 0.2065*** (0.0652) III 0.6260*** (0.0291) IV 0.3066*** (0.0897) -0.1631*** (0.0518) -1.4463* (0.7602) -0.0340 (0.0806) -4.1312*** (1.0941) Requested school use SAT1 Score* SAT1 Score/100 0.1039* (0.0614) 0.3248*** (0.0864) Requested school use SAT2 Score -2.4979*** (0.2795) -2.5905*** (0.2852) -2.2793*** (0.4887) -2.5132*** (0.4947) Requested school use SAT2 Score* SAT2 Score/100 0.4256*** (0.0439) 0.4401*** (0.0448) 0.3468*** (0.0760) 0.3852*** (0.0770) -0.1884* (0.1082) -0.2157*** (0.1095) 0.4580 (1.0611) -0.5234 (1.0994) 0.0130*** (0.0042) 0.0139*** (0.0042) 0.0051 (0.0401) 0.0360 (0.0412) Requested school use ACT Score Requested school use ACT Score*ACT Score Interpretation of Point Estimates College X An applicant who scores a 1,000 on the SAT I score decreases her probability of being accepted by 9.7 percentage points if she submits her score while an applicant who scores a 1,500 increases her probability of being accepted by 3.8 percentage points if she submits. College Y An applicant who scores a 1,000 on the SAT I score decreases her probability of being accepted by 16.8 percentage points if she submits her score while an applicant who scores a 1,500 increases her probability of being accepted by 12.6 percentage points if she submits. Interpretation of Point Estimates College X Applicants who submit their SAT I score are less likely to be accepted by College X if their SAT I score is less than 1,392 and are more likely to be accepted if their score is greater than 1,392. College Y Applicants who submit their SAT I score are less likely to be accepted if their SAT I score is less than 1,272 and are more likely to be accepted if their score is greater than 1,272. Possible Explanations for Negative Coefficient Estimate Associated with Submit SATI 1. 2. 3. For those who don’t submit, school might be “overestimating” their score Not submitting may be correlated with error term – applicants who do not submit are “more mature” or are athletes. School is behaving strategically when deciding who to accept. Submission on College X Performance Table 5B Applicants’ Disclosure Decisions: Table 2 SAT I Score (1600) – all applicants College X Chose to Chose Not Submit to Submit SAT I SAT I 1272 1139 (124) (116) [5550] [1017] Probability of Acceptance .418 (0.493) [5550] .395 (0.489) [1017] SAT I Score conditional on Acceptance 1323 (107) [2320] 1172 (99) [402] SAT I Score conditional on Enrollment 1281 (107) [647] Predicted SAT I Score* (based on those that want SAT I considered) 1272 (89) [5547] SS *** College Y Chose to Chose Not Submit to Submit SAT I SAT I 1267 1229 (144) (120) [2659] [845] SS *** .445 (0.497) [2659] .488 (0.500) [845] ** *** 1344 (115) [1182] 1260 (103) [412] *** 1155 (100) [185] *** 1299 (113) [351] 1227 (97) [135] *** 1219# (82) [1017] *** 1263 (78) [2659] 1251# (90) [845] *** SS, statistical significance ; *** statistically different at 1% level, ** statistically different at 5% level, * # Statistically different than the actual SATI score at the 1% level. *Regression Results are in Table A1 of the Appendix. Figure 2: Predicted versus Actual SAT I Score for those who Chose not to Submit Figure 2: Predicted versus Actual SAT I Score for those who Chose not to Submit Conclusions from Reduced Form 1. College admission departments are behaving strategically by more (less) likely accepting applicants who do not submit their SAT I scores if submitting their scores would decrease (increase) the average SAT I score the colleges report to the ranking organizations. 2. Applicants are behaving strategically by choosing not to reveal their SAT I scores if they are below a value one might predict based on their other observable characteristics. Note that the reduced form estimates do not address directly the college’s inference for those applicants who do not submit. Model and Structural Estimation Summary Statistics for College X N=324 N=122 N=5216 N=895 Summary Statistics for College Y N=294 N=83 N=2440 N=785 Notation μ(Xi)+εap +εen ,expected utility from attending the college for applicant i μ(Xi) is portion of the applicant specific preferences for attending College X that depends on the observables variable. εap is unobservable applicant specific preferences for attending College X that is known to the applicant at the time she submits her application. εen is unobservable applicant specific preferences for attending College X that is known to the applicant at the time she makes enrollment decision but not at time she submits her application Notation (cont) UR ,expected utility if applicant does not attend College X and does not apply early decision at College X. UR-C , expected utility if applicant does not attend College X and does apply early decision at College X. K , fixed cost of applying εen , unobserved cost of submitting SAT I Applicant’s Decision to apply early decision and/or submit SAT I Expected Utility if applicant applies early and submits Pa(X,ed,s) Pe(X,ed,s) [μ(X)+εap+εen]+ [1-Pa(X,ed,s) Pe(X,ed,s)][UR-C]-K- εs Expected Utility if applicant doesn’t apply early or submit Pa(X,ned,ns) Pe(X,ned,ns) [μ(X)+εap+εen]+ [1-Pa(X,ned,ns) Pe(X,ned,ns)][UR]-K Apply early, don’t submit and Don’t apply early, submit are analogous Applicant applies early decision and submits [assuming E(εen)=0] if εs +[Κ1]εap > [Κ1][-μ(X)+C] , [Κ2]εap > [Κ2][-μ(X)]+[1- Pa(X,ed,ns)Pe(X,ed,ns)][C] and εs+[Κ3]εap > [Κ3][-μ(X)]+[1Pa(X,ed,ns)Pe(X,ed,ns)][C] Κ1=Pa(X,ed,ns)Pe(X,ed,ns)-Pa(X,ed,s)Pe(X,ed,s), Κ2=Pa(X,ed,ns)Pe(X,ed,ns)-Pa(X,ned,ns)Pe(X, ned,ns) Κ3=Pa(X,ed,ns)Pe(X,ed,ns)-Pa(X,ned,s)Pe(X,ned,s) Whether to Apply Early Decision and/or Submit SATI Score Literature on College Objective Function Ehrenberg (1999) single well-defined objective function may explain “fairly well the behavior of small liberal arts colleges…” (page 101). Epple, Romano, and Seig (2006) GE model assume a school maximizes quality (average quality of the student body, school expenditure per student, and the mean income of the student body) s.t. balanced budget constraint and a fixed student body size. Our Model To account for the college’s concern for the quality of its current and future students and the understanding that future student quality depends on the college’s ranking, we allow the college’s objective function to depend on the perceived ability of the incoming students, the “reported” ability of these students, and the demographic characteristics of the student body. College’s Decision to Accept Applicant dsoisdsdf College accepts applicant i if: Pe(Xi,k,l) [ΠP(X+iP)+εqi + ΠR(X+iR)+ ΠD(X+iD)] +(1- Pe(Xi,k,l)) [ΠP(X-iP)+ ΠR(X-iR)+ ΠD(X-iD)] > ΠP(XriP)+ ΠR(XriR)+ ΠD(XriD)