Real and Imaginary
Concerns in Assessing
College Readiness
Nathan R. Kuncel
Department of Psychology
Favored Career Choices:
In Rank Order
1.
2.
3.
4.
5.
6.
7.
8.
Jedi Knight
Star Fleet Captain
Batman
Playboy Millionaire
Psychology Professor
Evolutionary Biologist
Economist
Brilliant but Misunderstood Novelist
Jedi Knight
Innate heritable ability, unrelated to social class, focused
through mentoring and intense training, high self control
Star Fleet Captain
Born but seasoned leaders who succeed through
managed team effectiveness, charismatic and bold
Batman
Single formative childhood experience that motivates
massive skill investment, augmented by tremendous family
wealth, intense and disagreeable
Professor of Psychology
Masochistic enthusiasm for studying nearly intractable
research questions, fairly friendly
Stories We Tell About Achieving
Success
•
•
•
•
•
•
Predictable versus Unpredictable
Book Smarts versus Street Smarts
General versus Specific Aptitudes
Social Class Dependent versus Not
Talent versus Effort
Linear versus Asymptotic
Stories about Tests
–
–
–
–
–
–
Tests don’t even predict grades well
Tests don’t add anything to prediction
Tests don’t predict anything other than grades
Cognitive tests only predict academic outcomes
Cognitive tests are only proxies for SES
Beyond a certain point, scores don’t matter
Stories about Traditional Alternatives
– Letters of Recommendation
– Personal Statements
– Interviews
Frightening Cautionary Tales about the Future
– Faking
– Bias
– Unreliability
Assertion:
Tests Don’t Even Predict
Grades Well
What does “well” mean and how do we quantify it?
Zombie Plague
• Plague sweeps the world
• Plague transforms 60% of infected people
into horrifying and mindless monsters.
A Partial Treatment Is Developed!
Zombie
Live
No Treatment
60%
40%
Treatment
40%
60%
What is the correlation?
Tests and Grades
• Berry and Sackett (2009) examined SAT-grade r’s in
a sample of 165,000 students from 41 schools
• Corrected for restriction of range using schoolspecific applicant pools
• Obtained individual course grades for each course,
and computed validity for each course (148,072
validity coefficients)
– This removes influence of student choice of courses
– Estimate r between SAT and common course portfolio
Berry and Sackett (2009) results
SAT-First Year College GPA
r
observed
.36
corrected for range
restriction
.46
corrected for course
difficulty
.55
Story:
Tests Don’t Add Anything
to Prediction
Percent Earning a 3.8 or Higher Graduate GPA
Bridgeman, Burton, & Cline (2009)
Story:
Tests only predict 1st year grades
• Paper submitted to American Psychologist:
“As is well known, the SAT predicts nothing but first
year grades” [with no citation]
Evidence Across All Admissions Tests:
Kuncel & Hezlett (2007) Science
• 8 Outcomes
• 7 Admissions Tests
–
–
–
–
–
–
–
–
1st Year GPA
Graduate GPA
Faculty Ratings
Degree Attainment
Citation Counts
Research Productivity
Quals./Comps. Exams
Licensing Exams
–
–
–
–
–
–
–
GRE-T
GRE-S
MCAT
LSAT
GMAT
MAT
PCAT
Results are based on over 600,000 students across
over 3,000 independent samples
Kuncel & Hezlett (2007). Science, 315, 1080-1081.
If the critics were right….
Book Smarts vs. Street Smarts
or
Academic vs. Practical Intelligence
or
Book Learning vs. Common Sense
Book Smarts vs. Street Smarts
• Fundamental argument is that different
types of intelligence are needed in
academic versus real world contexts
• It is argued that academic contexts require
an “academic intelligence” or book smarts
• It is my belief that these arguments are
based (in part) on an overly narrow
conceptualization of the tasks that
comprise academic performance
Academic versus Practical Tasks
•
•
•
•
•
•
Academic
Formulated by others
Well-defined
Presents complete
information
Single method for
obtaining answer
Not embedded in
ordinary experience
Little or no intrinsic
interest
•
•
•
•
•
•
•
Practical
Requires problem
recognition and
formulation
Ill-defined
Requires information
seeking
Multiple acceptable
solutions
Multiple paths to solution
Embedded in ordinary
experience
Requires motivation and
personal involvement
Things Students Do Before
Taking a Test
•
•
•
•
•
•
•
•
Determine study objectives and methods
Manage goal conflicts
Coordinate work with other classmates
Seek additional information or materials
Handle school related finances
Negotiate with peers and faculty
Avoid counterproductive school behaviors
Structure effective communications
A Model of Undergraduate Student
Performance Dimensions
• Traditional Classroom
Success
• Written and Oral
Communication
• Personal Discipline
• Resolving Goal Conflicts
• Studying and Learning
Proficiency
• Sustained Goal Directed
Effort
• Interactive Learning and
Team Performance
• Administration
• Interpersonal Proficiency
• Non-Classroom
Performance
• Development of Life Goals
and Values
Kuncel (2002); Kuncel, Campbell, Hezlett, & Ones (2001)
Cross-Situational Validity:
A Direct Test
• However, we still lack a direct test:
– Examine the validity of a single test developed
for academic settings but used in both
academic and work settings
– Ideally we would also establish the relationship
between this test and other cognitive ability
measures
• Unfortunately, a single ability measure is
rarely used for both personnel selection and
educational admissions decisions
• With one notable exception….
MAT – Academic Performance
Grad. GPA
1st Year Grad. GPA
Faculty Ratings
Comprehensive
Exams
Research
Productivity
Degree Attainment
Time to Finish
0
0.1
0.2
0.3
0.4
Kuncel, Hezlett, & Ones (2004) JPSP
0.5
0.6
0.7
MAT – Transitional Variables and Creativity
Conseling Potential
Ratings
Career Potential
Ratings
Creativity Ratings
Conseling Work
Samples
Student Teaching
Ratings
Internship
Practicum Ratings
0
0.1
0.2
0.3
Kuncel, Hezlett, & Ones (2004) JPSP
0.4
0.5
0.6
MAT – Work Criteria
Member Prof.
Org.
Ed. Admin.
Performance
Counseling
Performance
Job
Performance
0
0.1
0.2
0.3
0.4
Kuncel, Hezlett, & Ones (2004) JPSP
0.5
0.6
Story:
Predictive Power of Cognitive Ability
Tests is an artifact of SES
Its Storytime…
• In the interest of truth in advertising, the SAT
should simply be called a “wealth test” - Guiner
(Undated)
• “The SAT merely measures the size of
student’s houses” - Kohn (2001)
• “Only thing the SAT predicts well now is
socioeconomic status” – Colvin (1997)
Recent Claims in the
American Psychologist
• “…SAT I scores lose any ability to predict freshman
year grades if the regression analyses control for
socioeconomic status” (p. 100) – Crosby, Iyer,
Clayton, and Downing (2003)
• “…SAT scores used for college admissions do not
predict freshman year grades when socioeconomic
status is controlled” (p. 1023) – Biernat (2003)
• Atkinson and Geiser (2009) claim that SAT
coefficients are “decisively diminished” when SES
and HSGPA are controlled.
Consistent SES Findings
Source
Sample
rtest-grade
rtestgrade.SES
U California
77,000
.37
.34
Sackett, Kuncel, et al. 2009a
155,000
.35
.33
Sackett, Kuncel, et al. 2009b
17,000
.37
.36
Moderate
observed
relationship
Does not go to
zero when
controlling for SES
University of California System Data
HSGPA
.30
SAT I
.22
Family Income
.03
Parents Education .05
“Ok, but I know someone who had
really high test scores and they didn’t
do as well as another friend who had
lower test scores. So scores only
matters to a certain degree, right?”
More is also better at school
Adjusted Freshman GPA
4
3.5
3
2.5
2
1.5
600
700
800
900
1000 1100 1200 1300 1400 1500 1600
SAT Score
• Arneson and Sackett (2009)
Coward & Sackett (1990)
• 174 studies on the relationship between
intelligence and job performance
• Studies used the 9 scale GATB (General
Aptitude Test Battery)
• Total sample size across all studies was a
substantial 36,614
• Found overwhelming support for a linear
relationship between ability and job
performance
• The commonly held notion appears to be
incorrect
Traditional Supplementary Predictors
• Letters of Recommendation
• Personal Statements
• Interviews
Letters
GRE
0.5
0.4
0.3
0.2
0.1
0
Result suggest no incremental validity for letters,
save for degree completion. Note that all letters were
structured and quantitative.
Kuncel, Vanelli, & Ones (2009)
Research
Productivity
Degree
Completion
Faculty
Rating
GPA Grad.
School
Research
Productivity
Degree
Completion
Faculty
Rating
GPA Grad.
School
0.5
0.4
0.3
0.2
0.1
0
Personal Statements
Cooper-Murphy, Klieger, Borneman, & Kuncel (2007) College and University
Admissions Interview: Results
N
k
r
obs
SD
obs
SD
r
80% cred.
Grade Point Average in
Graduate/Medical School
3540 37 0.12 0.11 0.04 .06 to .17
Medical Clinical and
Internship Performance
Rating
2641 23 0.15
0.1
0.04 .10 to .21
Graduate School
Performance Rating (non
Medical)
262
5 0.24 0.25 0.21 -.03 to .51
Medical Board
Examinations
231
3 0.13 0.07
0
.13 to .13
Yet There is Good Stuff Out There
Study Habits, Skills, and Attitudes
0.5
N
k
r
Aggregate Measures
18,517
107
.33
Study Skills
24,547
87
.25
Study Habits
23,390
102
.23
7,211
37
.26
Study Attitudes
0.4
0.3
0.2
0.1
0
SHSA
SAT
HSGPA
Crede & Kuncel (2007) Perspectives in Psychological Science
Class Attendance: Being There
N
Attendance
21,164
Crede, Roch, & Kieszczynka (2010) Review of
Educational Research
k
r
68 .41
Personality Predictors of Graduate Student Performance
N
k
robs SDobs SDr
429
6
.24
.12
.03
1,645
7
.16
.08
.04
Autonomy
518
3
.16
.02
.00
Ach. Via Indep.
132
3
.27
.12
.00
156
2
.26
.00
.00
Emotional Stability
1,180
3
.09
.10
.08
Conscientiousness
1,559
5
.08
.09
.07
Graduate GPA
Emotional Stability
Achievement
Task Performance
Qualifying Exam
Ach. Via Indep.
Obtain Ph.D.
Rigdon & Kuncel, 2010
Cautionary Tales
• Faking and Coaching
– Most alternatives that have been proposed
would be highly susceptible to deliberate faking
or test preparation coaching
• Personality Assessments
– Other ratings
• Study habits, attitude, and skill inventories
– Placement and guidance
• Biodata
– Verifiable content
• Situational judgments tests
Fairness and Bias
• Extensive research on standardized test
scores for bias
• Social class has also been scrutinized.
• Data for letters of recommendation and
personal statements in academic settings is
thinner.
• A summary of current knowledge on the next
slide…
New Alternatives Should be
Subjugated to Extensive Study for
Bias Before Implementation