1
“How Optional Testing Works: Defining Promise in American Admissions”
William C. Hiss, Principal Investigator
Valerie Franks, Co-Author and Lead Researcher
Annual Meeting
American Association of Colleges and Universities
Washington, DC
January 23, 2015
1. Title page.
Thank you for coming to hear about “Defining Promise“ a study on optional
standardized testing, which NACAC published electronically last February.
A 70-page scholarly study swimming with statistical charts might be
expected to fall still-born from the electronic press—with a glass of sherry,
the solution to your insomnia problems. But while about testing, this study
2
addresses the social ethics of who gets to go to college, and the same day
that NACAC published this report, NPR, CBS Evening News and PBS
NewsHour all covered the study.
A few weeks later, the College Board held their news conference to
announce proposed changes to the SATs. The two, coming back to back,
created something of a media typhoon, not yet fully died down. It has been
a rare week since February without a trip like this one to present the study,
a media contact, or more institutions going test-optional, five in January.
Lumina Foundation has calculated the immense pool of low-income
students who need to get through higher education to have a chance. This
study provides the research support for optional testing as at least one
route by which that can happen.
Some of our most thoughtful educational researchers are coming to similar
conclusions about the predictive value of testing. In their 2009 book,
Crossing the Finish Line: Completing College at America’s Public Universities,
Presidents Bowen and McPherson, with their colleague Matthew Chingos,
examined what predicted graduation at a large set of public universities. I
quote: “In all but one of these 50 public universities, high school GPA
remains a highly significant predictor of six-year graduation rates after
taking account of the effects of test scores… Test scores, on the other hand,
routinely fail to pass standard tests of statistical significance when included
with high school GPA in regressions predicting graduation rates—especially
when we leave the realm of the most highly selective universities…. …the
remaining predictive power of the SAT/ACT scores disappears entirely when
we add controls for the high school attended, whereas the predictive power
of the high school GPA increases.” (close quote, pp. 115-6)
We agree with another finding in Crossing the Finish Line: the controversy of
“overmatching” –where students allegedly attend a college or university for
which they are not qualified, versus “undermatching” –where students
attend an institution that does not provide enough challenge or opportunity.
Bowen and his colleagues wrote, “The findings reported in this study fail to
3
provide any evidence of overmatching, but demonstrate that
undermatching is a massive problem.”
In a book just out, The Tyranny of the Meritocracy: Democratizing Higher
Education in America, Professor Lani Guinier of Harvard Law School, citing
our study, argues that testing principally constrains rather than opens paths
to opportunity, creating what she calls a “testocracy.”
2. “Take Back the Conversation.” Ethical and Practical Issues.
A few years ago, the NACAC Commission on Standardized Testing chaired by
Bill Fitzsimmons at Harvard made a recommendation, much cited since, that
counselors and admissions officers “take back the conversation” about
testing from organizations for whom it was either a highly profitable
business or a cause. This study contributes to that conversation; it is the
first national, peer-reviewed study to examine optional testing across
institutional types. We make no claims to have gotten everything right; no
study of this magnitude and scope should pretend to have slam-dunk results
on every issue. We hope our study will serve as encouragement for other
research, and will suggest a few studies in the last slide as “a legacy lap.”
4
Partly trained as an ethicist, I have worked on optional testing for 35 years.
The faculty at Bates, where I led the Admissions office from 1978 to 2000,
voted for an optional testing policy in 1984, and we have published careful
statistical reports at 5-year intervals since. This current national research
follows up on a 25-year look-back study of optional testing at Bates.
This study poses some broad ethical questions:
• Can optional testing lower access barriers and expand college
populations? Fundamentally, does testing help find promising
students, or does it artificially truncate the pools of students who
would succeed if they could be convinced to apply and enroll? As you
will see, at least based on this study, it is far more the latter.
• Who is exploring the breadth of human intellect and promise in
imaginative ways? Institutions with optional testing policies are not all
doing it the same way. Public universities, with optional testing
policies quite different than at privates, are seeing success.
• Who is doing the heavy lifting, serving broad constituencies? In our
study are two categories of institutions not getting much attention in
the various top-25 lists, minority-serving, and arts institutions. Also
included are a public university with 50,000 students and a long
history of strength in science and technology, and two small colleges
with traditions of creative curricular thinking, the College of the
Atlantic in Maine and Pitzer in California. The study also includes a
Native American college, and a minority-serving public flagship in a US
Territory. So we tried to look across a broad landscape.
There are also practical questions: Very simply, how well have these policies
worked? Others may raise the complex issues of test bias, but we ask a
much simpler question: how well do non-submitters succeed? What are the
four year cumulative college GPAs and graduation rates of submitters and
non-submitters, as segmented by institutional types?
5
What are the practical enrollment outcomes for applicant pools, geographic
reach, diversity, intellectual achievement, and accurate financial modeling?
Unexamined assumptions about testing seem to be costing individual
institutions millions of dollars. Optional testing mostly serves students, but
it also has to work for institutions.
3. Research Sample Overview.
We focused on four-year institutions that are non-profit, report data to
IPEDS, and have national visibility. We considered 400 institutions, and
contacted 130 to choose the 33 in our study.
Here are the institutions: 20 private, 6 public, 5 minority-serving, and 2 arts
institutions, with the exact numbers of students in each category listed. We
requested four cohort years of data from each institution, two graduated,
and two currently enrolled, to get a spread of data over time.
4. Publically Announced Institutions in the Study
6
After seeing the published study last March, 13 of the 33 institutions
announced that they were included in our study, and here they are. All 33
institutions shared enormous amounts of data, 40 variables on each of the
123,000 student records, 5 million data items. We guaranteed everyone
confidentiality—after all, none of us knew in 2010 how this would turn out.
Each institution, as a gesture of thanks, received a proprietary institutional
analysis, including a blind comparison study of the other institutions in their
category, over 2000 pages of individual reports.
5. Summary of Institution Type
7
We focused tightly on the years of college enrollment: we did no analysis on
the Admissions funnel before enrollment, nor on career or graduate school
outcomes. But there are additional studies crying to be done, and here is
one. From 2003 to 2010, the percentage of non-submitters enrolled at the
20 private institutions rose from 26% to 35%. But why? More nonsubmitters applying? More confidence among admissions deans that these
students are doing fine, and therefore can be admitted at higher rates?
Higher yield on non-submitters in the April wars? Higher rates of ED apps
from non-submitters? Better retention of non-submitters, so that they
show up in these enrolled student counts? We interviewed each dean of
admissions in the study, and from their anecdotal information, the answer
to all five of these questions may be “yes.” But we don’t know, and a
national study of optional testing as it has shaped the admissions funnel lies
awaiting.
A second issue is important for independent colleges and universities,
competing with public institutions. Most people, if they know of optional
testing at all, think it is used by private liberal arts institutions. But
increasingly, large public institutions have test-optional admissions,
8
admitting students on the basis of HS rank or GPA, no matter what their
testing. The obvious example is the “top 10%” policy in Texas, but also the
entire CSU system in CA, and many state flagships such as Mississippi and
Arizona. Some universities told us in early conversations, “We are not test
optional, we collect tests from everyone.” Our answer was, “Yes, you collect
tests, but if you are admitting students solely on rank or GPA, you are not
using testing to make admissions decisions. Those are the students we want
to study as ‘non-submitters,’ to use parallel terminology with the private
institutions.”
Of course many of these public university students had the required rank or
GPA for guaranteed admission, but also had very solid testing, above the
average of the university where they enrolled. So while they are technically
non-submitters, common sense tells you that they likely would have been
admitted anyway, with or without the automatic admissions policy. So this
study focuses on students admitted who met the HS rank or GPA criteria,
but with test scores below the averages of their public university. They are
the beneficiaries of their state’s assured or guaranteed admissions policy.
You will see the phrase “without above-average testing students”, as we
focus on students admitted with below-average testing. As you will see, the
findings are dramatic; these public universities seem largely to be
succeeding with their versions of optional testing.
Also, a comment also on the findings from the minority-serving and arts
institutions. Their data largely lines up with the overall findings, and their
students are included in all of the data dealing with the entire study. But in
these institutions, we often found problems of missing data or small head
counts. The minority-serving institutions do some of the heaviest of the
heavy lifting, and the arts institutions meet a wide array of student
expression. But we would quickly say that at these two types of institutions,
we do not have slam-dunk results to present, based on much thinner data.
6. Principal Findings
9
What are the principal findings? There are no significant differences in
either Cumulative GPA or graduation rates between submitters and nonsubmitters. Across the study, non-submitters (not including the public
university non-submitters with above-average testing,) earned Cumulative
GPAs that were only .05 lower than submitters, 2.83 versus 2.88. The
difference in their graduation rates was .6%. By any standard, these are
trivial differences. On this we hang the national sluicegates about who can
go to college and where they go? This does not make any sense. Valerie
and I in the research often felt like the kid by the side of the road, taking a
deep breath and saying, “The emperor doesn’t have any clothes on.”
College and university Cumulative GPAs closely track high school GPAs,
despite wide variations in testing. Students with strong HSGPAs generally
perform well in college, despite modest or low testing. In contrast, students
with weak HSGPAs earn lower college Cum GPAs and graduate at lower
rates, even with markedly stronger testing. A clear message: hard work and
good grades in high school matter, and they matter a lot.
10
When compared to the submitter population, Non-submitters are more
likely to be first-generation-to-college enrollees, all categories of minority
students, Pell Grant recipients, and women. But across institutional types,
white students also use optional testing policies at rates within low single
digits of the averages, so the policies have broad appeal across ethnic
groups. The breadth of students being served by optional testing is positive
and promising.
College admissions decisions made without testing are apparently just as
reliable as those made with testing. Testing may serve to artificially truncate
the applicant pools of students who would succeed if they could be
convinced to apply. The numbers of students in these categories are not
small. In our study, about 30%, or roughly 37,000, of the students are nonsubmitters, not counting the above-average testers. Please remember that
30% figure, as it raises a key question at the end.
7. Additional Findings
11
In a surprise finding, non-submitters display a distinct two-tail or bimodal
curve of family financial capacity. First-generation, minority and Pellrecipient students will often need financial aid support, but large pools of
students not qualifying for or not requesting financial aid help balance
budgets. Across institutional types, both low income and high income
students will use an optional testing policy if offered.
LD students, from a modest sample of 1050 students at 8 institutions, are
much more likely to apply as non-submitters, apply ED and perform at levels
close to their classmates. The evidence from a long-term study at Bates
found that given the modest accommodations to which these students are
legally entitled, their GPAs and graduation rates come up to class averages,
helping to increase the institution’s overall graduation rates. Another topic
for future research.
Non-submitters are commonly missed in consideration for no-need merit
awards, despite slightly better Cum GPAs and markedly higher graduation
rates than the submitters who receive merit awards. Institutions should
examine criteria for merit awards, especially standardized testing to qualify
students for no-need merit funding. A chart in a moment makes this point
painfully clear.
Based on our interviews with deans, non-submitters often expand applicant
pools, apply Early Decision at higher rates, increase minority enrollments,
expand geographic appeal, and allow for success by LD students.
8.
College Cumulative GPA versus High School GPA.
12
Now the moment you’ve been waiting for: statistical charts.
This is the first: a scatterplot which simply graphs four-year high school GPA
against four year college GPA. You see the pattern: With nothing else but
HS GPA and College GPA being looked at, this is an extremely tight and
evenly sloped set of data.
Most deans who have seen this chart have said, “Well, two four-year longdistance measures of curiosity, ambition, self-discipline, ability to be part of
a team and listen to teachers—maybe not a surprise that these two criteria
have such a tight overlap pattern.”
9. College Cumulative GPA Versus SAT
13
In contrast, in this chart college GPA is graphed against SATs, including
translated ACT scores. The shape of this graph is more diffuse and scattered.
It still shows some correlation, but clearly not a tight relationship.
Note also that we are using college cumulative GPA as our academic marker,
rather than the more volatile first year GPA as the College Board does in its
reports. It is a far more reliable marker of academic performance.
10.
Summary of Key Statistics. All 33 Institutions.
14
Here is the other chart of summary information, and you will see four of
these charts for different groups of students, so a moment to explain the
layout. There are two parallel columns of findings, with non-submitters on
the left and submitters on the right, with the N’s at the top. Each horizontal
row contains data on a particular issue, such as HSGPA, SATs or graduation
rates. The color coding reflects the level of statistical significance, with the
coding explained in the color keys. Grey colors reflect trivial, small or
insignificant differences, while the green colors reflect moderate, large or
significant differences. As with the scatter charts, don’t try to follow each
data point, but watch the patterns in various groups of students.
In this chart of all 33 institutions, the only difference is that the nonsubmitters come with moderately better HSGPAs, and then perform better
at small levels in college. But now look at the second chart. Here, the
above-average-testing students at the public universities have been
removed. Now the statistically large difference is in SAT scores, with the
non-submitters averaging 113 points below the submitters. But the core
finding: their cumulative GPAs are only .05 below the submitters, 2.83
versus 2.88, and their graduation rates are .6% below the submitters.
15
11.
Summary of Key Statistics, Private Institutions
These data are a summary snapshot for the 20 private colleges and
universities, with about 37,000 aggregate records. Submitters and nonsubmitters enter college with very closely parallel GPA’s but large
differences in SAT scores of 149 points. In college, non-submitters start with
first-year GPA differences of .15 with submitters, but gradually close the
GPA gap to .1 by graduation. Their graduation rates are different by rates of
1%-2% either in favor of submitters or non-submitters, depending on how
they are calculated. Non-submitters at privates are somewhat less likely to
be STEM majors--given the importance of STEM fields to our economy, a
finding worth more exploration.
This pattern runs through much of the study. The only large difference in
either entering credentials or in college performance is SAT scores.
12.
Summary of Key Statistics, Public Institutions
16
Here is the same data for the public university students, with the aboveaverage-testing students removed.
These lower-testing students enroll with HSGPAs that are much stronger,
almost three-tenths of a GPA point, and average SATs that are 93 points
below the submitters. They earn equal to stronger GPA’s, and significantly
better graduation rates than the submitters.
Again we see the pattern: non-submitters are more likely to be minorities,
first-gen, women and Pell recipients, at rather dramatic levels of 9 to 15
percentage points above the submitters.
At the publics, a small difference in STEM majors, only by 2%.
13.
Application Comparisons, Merit Award Recipients
17
I promised an example of how policy on optional testing might allow millions
of dollars to be more wisely spent, and here is that example. We collected
data on no-need merit awards, and found that a bit over 27,000, or 22%,
received a merit award.
When all students are included, as in the left hand graph, about equal
numbers of submitters and non-submitters received a merit award, but the
non-submitters have much higher college GPA’s and graduation rates, and
are more likely to be STEM majors. In the right hand graph, we removed the
above-average-testing non-submitters, and the graph reflects major
changes. First, the number of non-submitters with a merit award drops
dramatically, from 13,700 to a little over 5000. These below-average-testing
non-submitters still have better HSGPAs than the submitters, but SATs that
are 143 points below the submitters. Yet these below-average testers have
college GPAs that are slightly better than the submitters, and graduation
rates that are much better, by 6%.
Who are these lower-testing non-submitters? Once again—I hope by now
you can chant these categories in unison--minorities, first-gen, women and
18
Pell recipients, and by quite significant differences. These lower-testing
non-submitters are also more likely to be STEM majors, by 5%.
Acknowledging that the lion’s share of merit money is institutionally funded,
let me ask a blunt question. Why are we funding 2.7 times as many
submitters with lower HSGPAs and better testing, when those submitters
graduate with slightly worse cum GPAs and much worse graduation rates?
The answer, in all likelihood, is that using SAT and ACT scores to determine
merit awards is an unexamined assumption. At many institutions, nobody
has looked at the non-submitters versus submitters to see how they
perform.
Perhaps these institutions believe that the higher average SAT scores will
earn them a better ranking in USNews or elsewhere. If that is the strategy,
a word of advice from a retired dean who founded and chaired for a decade
the advisory committee of deans at USNews; I helped design the formulas.
Currently, SAT and ACT scores account for 8% of the ranking formula, while
graduation rates, when added up from three different parts of the formula,
count for 30%. So if you think that the fancy SAT scores of kids with less
strong HS records will help you, you might have your IR office calculate for
your institution the almost 4-1 heavier weight in the USNews formula for
graduation rate over SATs. In all likelihood, that 6% lower graduation rate
for submitters is coming around to bite you in the backside. Unexamined
assumptions about testing to determine who gets merit awards are in all
likelihood the tip of a large fiscal iceberg. Perhaps another place to “take
back the conversation.”
14.
Not a Victory Lap but a Legacy Lap… The Next Steps?
19
The sponsoring foundation for our study gave us a second grant to cover
travel expenses to present the study at conferences and on campuses. The
foundation said that they hoped it would be “not a victory lap but a legacy
lap—what are the next steps?”
Each of you might have suggestions for a legacy lap. But here are a few:
--A research team should examine the twin issues of “false negatives” of low
testing with potential “false positives” of high testing created by coaching.
This study provides evidence of the “false negatives” of students with
modest testing and strong results on everything else in high school and
college. We also know that many upper-income students are being coached
to death for the tests, and there is not much argument any more that
sustained coaching can significantly increase test scores. But we don’t
know, at least on a national basis, whether the coaching that inflates test
scores also improves college GPAs. My personal guess is that it does not,
and we are perhaps seeing some evidence of that in the submitters in this
study, with their less strong HSGPAs, stronger testing, and lower college
20
performance and graduation rates. If that is true, we may be seeing
diminished predictive value of the tests on both ends, high and low.
--An ethicist’s question. If we had a medical test for a serious condition,
such as liver or heart failure, that had a 30% rate of false negatives--that is,
the test said you didn’t have the condition when in fact you did--would that
be OK? Of course it wouldn’t. The firm supplying that test would be sued
out of business or closed by the FDA. You will recall that roughly 30% of the
students in our study are non-submitters. We offer a hypothesis, that the
false negative results in testing are in fact quite close to a false negative in a
medical test, given the importance of access to the strongest level of higher
education for which a student is qualified.
--Further, if 30% is the non-submitter share of enrolling students, what is the
true share of false negatives created by testing, including those who are
refused, attending community colleges and for-profit colleges, or not
attending at all? By definition, the 30% in our study are those who made it
through, who by good guidance counseling or grit or family encouragement
got into these institutions. So what is the real percentage, including those
who do not make it to where they should be? Is it 35% or 40% or 50%?
America cannot afford these levels of lost talent.
--Examine college success using 4-year Cum GPAs and graduation rates
rather than first-year GPAs as the principal yardstick. For too long, we have
held on to 1st year GPA to measure success, in part because of a din of
information from the testing agencies that their tests could only predict 1st
year performance, and nothing predicted after that, with all the changes in
college. This study and others such as Crossing the Finish Line are providing
the evidence that 4-year college cum GPAs and graduation rates ought to be
our markers. We should also add alumni and grad school outcomes in
future studies.
--Evaluate a broader band of research tools. Add Cohen’s d, Chi Square, bar
charts and scatterplots to regression (R-square) analysis. There is a much
richer array of tools than has commonly been used.
21
--Share published research on optional testing. Models are available from
NACAC, Bates and Ithaca, and in Joseph Soares, ed: SAT Wars: The Case for
Test-Optional Admissions, as well as Crossing the Finish Line. (I would add a
profound bow of thanks to the Bates Presidents since 1980 who have each
allowed us to get this research done: Clayton Spencer, Nancy Cable, Elaine
Hansen, Don Harward and Hedley Reynolds.)
--NACAC will be hosting a pre-conference workshop on optional testing and
research on testing next fall, to explore policy, non-cognitive skills,
implementing an optional testing policy, and so on.
15.
Institutions Recently Adopting Test-Optional Policies.
We mentioned that scarcely a few weeks went by without another college
or university deciding to become test-optional. Here is a list. This seems an
encouraging, interesting group, questioning assumptions that optional
testing is for small privates. Wesleyan and Bryn Mawr are very high-end
small privates. But look at the others. Hostra, a New York private, has over
12,000 students and law and medical graduate programs. And 6 of these 11
22
are public: with widely differing missions—Old Dominion, a public with
25,000 students, is in Norfolk, so with a military flavor to its students and
culture. Just within the last week or so, VCU has become test optional:
32,000 students, 2200 faculty, and major medical and science graduate
programs. Temple, with traditions of access for urban students, has an
enrollment of 38,000, Montclair State has 18,000 and Plymouth State has
7300, with many rural NH students. Recently I said to an audience of high
school students in Houston, it will not be long before you will choose to
apply to 8-12 very strong colleges and universities, all of which are test
optional. In recent months people have asked whether the optional testing
movement is reaching a “tipping point.” Bob Schaeffer at Fairtest wisely
said that we probably won’t know the tipping point until some time after we
pass it. But if these institutions and the 33 in our study are any sign, we can
feel the ground moving.
16.
Video Clip of Megan Guynes and Cristal Martin.
In recent months Valerie and I have been working with a film company in
San Francisco on a documentary about testing. Here is a very short clip from
23
their work to date. It is worth noting that these two recent Bates alumnae
come from small towns a long way from Bates: Megan from Rancho
Cucamonga, CA, 40 miles east of LA in the San Bernardino Valley, and Cristal
from Wichita Falls, TX, 15 miles from the OK border where Texas turns to go
up into the Panhandle. Are any of us doing Admissions recruiting travel to
those areas? Of course not, but one of the persistent findings of the study is
that optional testing helps institutions spread out geographically. Students
who will be helped by an optional testing policy, if they learn about it, will
apply from territories that are far beyond your normal geographic drawing
zones.
Thank you for coming, and glad to take questions and comments.
17.
Q&A