Combining Cognitive and NonCognitive Measures:
Expanding the Domain of
College Performance and its Prediction
Neal Schmitt
Michigan State University
Presented at College Board, ETS, AERA
Conference
December 10, 2010
1
Acknowledgements
Jessica Fandre
 Alyssa Friede
 Michael Gillespie
 Anna Imus
 Brian Kim
 Stephanie Merritt
 Fred Oswald
 Matt Reeder
 College Board (Wayne
Mattern)

Tim Pleskac
Abigail Quinn
Lauren Ramsay
Smriti Shivpuri
Ruchi Sinha
Tae-Young Yoo
Mark Zorzie
Juliya Golubovich
Camara and Krista
2
Outline









History and Background
“Job analysis” or conceptualization
Instrumentation (description of noncognitive
measures used and outcomes examined)
Validity data
Subgroup Differences and Implications
Faking issues
Acceptability
Research (profiles, fit, goal orientation, dif)
Limitations and future research
3
Developing Alternative Measures
of Student Potential
Motivation for our work

Broaden the scope of student outcomes and
capabilities considered in college admissions.

Reduce adverse impact.

Test the feasibility of developing “noncognitive”
measures that are



valid
practical in terms of time and effort required to assess
less susceptible to faking
4
Developing Alternative Measures
of Student Potential (Oswald et al, 2004, JAP)


Identify a broader domain of college student
performance:

Review university mission statements and
department objectives

Interview with university staff responsible
for student life at Michigan State University

Review of the education literature on student
outcomes
Our systematic search resulted in
12 dimensions of student performance…
5
Dimensions of
College Student Performance
intellectual
intrapersonal
1.
Knowledge and
mastery of general
principles
7.
Social
responsibility and
citizenship
2.
Continuous learning,
intellectual interest
and curiosity
8.
Physical and
psychological
health
3.
Artistic and cultural
appreciation
9.
Career orientation
10.
Adaptability and
life skills
11.
Perseverance
12.
Ethics and
integrity
interpersonal
4.
Appreciation for
diversity
5.
Leadership
6.
Interpersonal skills
6
Two “Noncognitive” Measures

1. Situational judgment inventory



A situation is presented along with several
alternative courses of action.
The respondent is asked to indicate what she/he
would be most likely and least likely to do.
2. Biodata

Short, multiple choice reports of
past experience/background and/or
interests/preferences.
7
Situational Judgment Inventory (SJI)
Developed situational judgment items for
each of the 12 performance dimensions

Student generated critical incidents (CIs) for each dimension

Translated CIs to stems for each item

Other students generated solutions to these questions

Researchers edited the options

Re-sorting back into 12 dimensions

3 Answer Keys: (see Motowidlo, Dunnette, & Carter, 1990)



Expert student (junior and senior students) scoring
Resident Advisor scoring
African American key
8
Sample SJI Item for
Leadership
You are assigned to a group to work on a
particular project. When you sit down
together as a group, no one says anything.
a) -1 Look at them until someone eventually says something
b) Start the conversation yourself by introducing yourself
c) +1 Get to know everyone first and see what they are
thinking about the project to make sure the project’s goals
are clear to everyone
d) Try to start working on the project by asking everyone’s
opinion about the nature of the project
e) You would take the leadership role by assigning people to
do things or ask questions to get things rolling
9
Sample SJI Item for
Interpersonal Skills
You and some other students in your dorm area feel
that a small group of students are highly disruptive
during times when you would like to study or sleep.
What would you do?
a)
b)
c)
d)
e)
f)
Talk to the resident assistant about it, as that is one of the
responsibilities of their job.
-1 That’s part of life in the dorms. Let it go.
Bring it up at the next floor meeting.
+1 Politely talk to the disruptive students and ask them to
be more considerate. If the problem persists, talk to the
resident assistant.
Wear earplugs or headphones when necessary.
Develop and implement appropriate rules to address the
problem.
10
Sample Item for Knowledge







You decided early in the term to do a paper on a topic very interesting to
you. However, you have found it difficult to find information on your topic,
your job has taken more time than you wanted, and you have had more
work in your other courses than you anticipated. Now it seems like you
may have to engage in several "all-nighters" to complete your paper on
time. What would you do?
a. Seek help from other students who may have had a similar experience.
b. (-1)Pick a topic that can be completed quicker. An “A” is an “A”.
c. (1)Set up a schedule on which you can complete all of the other work
you need to do, spend as much time on the paper as possible, and meet
with the instructor to discuss what you have so far and get suggestions.
d. Do whatever it takes to complete the paper, including “all-nighters”.
e. Talk to the instructor about the situation and ask for advice.
f. Make the paper a priority, but take into account how much the paper is
worth in the class.
11
Biodata Measure
Developed biodata items for each dimension




Reviewed biodata item pools and adapted items
related to each major performance dimension.
Resorted items into dimensions.
Asked a pilot sample to respond to open-ended
versions of quantitative response options to
determine appropriate scale anchors.
We used a rational scoring approach to these
items, but also developed empirical keys against
3 criteria
12
Sample Biodata Items for
Leadership
1. The number of high school clubs and organized
a)
b)
c)
d)
e)
activities (such as band, sports, newspapers,
etc.) in which I took a leadership role was:
4 or more
3
2
1
I did not take a leadership role
2. How often do you talk your friends into doing
a)
b)
c)
d)
e)
what you want to do during the evening?
most of the time
sometimes (about half the time)
occasionally (about as often as others in my group
seldom or infrequently
never
13
Sample Biodata Items for
Multicultural Appreciation
1. How often have you participated in social service or
a)
b)
c)
d)
e)
charity organizations?
Four or more times
Three times
Two times
Once
Never
2. If given a choice at a restaurant, would you order any
a)
b)
c)
d)
food with which you are unfamiliar?
Never, I would always order foods that I know and enjoy
Sometimes I might try a new food if someone else ordered it
Occasionally I will order something new provided I can also
order familiar food at the same time
If given a chance, I will always order a new food and try it
14
Sample Biodata Items for
Social Responsibility
1.
a)
b)
c)
d)
e)
2.
a)
b)
c)
d)
e)
In the past year how many times have you considered the
environment when purchasing a product (for example
hairspray, or a car?)
Never
Once
Twice
Three or four times
Five times or more
How often do you work with not for profit groups?
Never
Not very often
Sometimes
Often
Always
15
Outcomes Examined
Self Ratings on behaviorally anchored
rating scales built around the 12
dimensions
 Self rated class attendance
 University archives (grades)
 Organizational citizenship behavior
 Deviance
 Continuation in school and graduation

16
Interpersonal skills—Self Rated Performance Outcome
Definition: Communicating and dealing well with others, whether in informal social situations or more formal
school-related situations. Being aware of the social dynamics of a situation and responding appropriately.
Fulfills
Expectations
Unsatisfactory
1
|
2
|
3
|
4
|
Exceptional
5
|
6
|
7
|
Before you make your rating, please read these two examples:
Example 1
Your roommate, usually a tidy person, has recently experienced some personal difficulties. As a result, the roommate
has become quite distracted and has left much of the household responsibilities to you. You have talked to the
roommate about your concerns, and empathetically requested that the roommate resume his/her share of the
responsibilities as soon as possible. A month passes and you are still doing too much of the roommate’s work. What do
you expect you would do?
Unsatisfactory
Fulfills Expectations
Exceptional
You ask to change rooms.
You do his/her share of the work,
and put anything of the
roommate’s that affects you in the
roommate’s area of the room.
You talk with the roommate again and
explain that you are suffering as a result of
the roommate’s behavior. You attempt to
come up with a mutually acceptable plan
of action.
Example 2
You have been standing in line for the restroom for some time after a campus event, and someone cuts into the line ahead
of you. What do you expect you would do?
Unsatisfactory
Fulfills Expectations
Exceptional
You comment loudly to someone
nearby how rude it is that people
cut in line.
You tell the person that there is a
line.
You calmly and politely inform the person that there
is a line and ask that they move to the back.
17
Validity Data: College GPA















Knowledge
Learning
Art Appr.
Mltcult.Appr.
Lead
Int. Skl.
Soc. Resp.
Health
Cr. Ornt.
Adapt
Perser
Ethics
SJI
HSGPA
ACT/SAT
2001-02
(N=614)
2003-04
(N=568)
.22
.05
.01
.07
.14
.04
.08
.23
-.02
.21
.15
.14
.16
.22
-.02
-.03
-.04
-.01
.33
.07
.14
-.06
.13
.07
.22
.11
.39
.34
2004-8
(N>1900)
2009-10
(N>550)
.26
.13
.18
.11
.09
.22
.10
.16
.12
.07
.13
.11
-.14
.05
.07
.17
.22
.53
.53
.08
.04
-.11
.01
.10
.13
.09
.29
.44
18
Validity Data: Class Absences
2001-02
(N>630)















Knowledge
Learning
Art Appr.
Mltcult.Appr.
Lead
Int. Skl.
Soc. Resp.
Health
Cr. Ornt.
Adapt
Perser
Ethics
SJI
HSGPA
ACT/SAT
-.18
.00
.07
.02
-.03
-.09
-.09
-.23
-.06
-.15
-.20
-.31
-.27
.11
2005-06
(N>900)
2008
(N >556)
2009-10
(N>600)
-.15
-.07
-.03
-.04
-.06
-.14
-.06
-.08
-.10
-.02
-.03
.04
-.10
-.06
-.04
-.08
-.17
-.05
-.10
-.18
-.17
-.16
-.04
.17
-.08
-.12
-.08
-.10
-.16
-.24
-.17
-.01
.14
-.11
-.05
-.05
.07
-.01
-.11
-.14
-.02
.07
19
Validity Data: Self Report-BARS















2001-02
(N=614)
Knowledge
.33
Learning
.27
Art Appr.
.51
Mltcult.Appr.
.38
Lead
.43
Int. Skl.
.15
Soc. Resp.
.35
Health
.28
Cr. Ornt.
.22
Adapt
.24
Perser
.34
Ethics
.11
SJI
.53
HSGPA
ACT/SAT
-.01
2003-04
(N=568)
.46
.37
.33
.34
.34
2008
(N>547)
.21
.24
.20
.29
.29
.32
.16
.36
.26
.46
.39
.54
.08
-.01
.26
.26
.22
.28
.36
.23
.23
.08
.01
2009-10
(N>600)
.32
.39
.33
.30
.38
.30
.29
.31
.30
.37
.22
.25
.02
.08
20
Incremental Validity: College GPA
Step 1: Act/SAT, HSGPA ∆R2
Step 2: Biodata, SJI ∆R2
Adjusted R
N
Noncog. Vars. Significant
01-02
.103
.089
.438
610
Know
Lrng.
Health
Adapt
03-04
.179
.070
.499
331
08
.398
.029
.419
1155
09-10
.200
.069
.480
296
Know Lrng.
Cr. Ornt.
Lrng. Health Know
Ethics Cr.Ornt.
SJI
SJI
21
Incremental Validity: Absenteeism and
BARS
05-06
Absenteeism
Step 1:
Step 2:
Act/SAT,
HSGPA ∆R2
Biodata,
SJI ∆R2
Adjusted R
N
Noncog. Vars.
BARS
08 follow-up
Absenteeism
BARS
.028
.019
.033
.008
.060
.215
.116
.240
.297
800
Health
Ethics
SJI
.484
801
Health
Ethics
SJI
Adapt
Cr.Ornt.
Persev.
.386
556
Lead
Health
Ethics
SJT
.497
547
Multic
Health
Persv
Ethics
22
Subgroup Differences in
Standardized Units (04sample)
SJT
Know
Cont.Lrn.
Artistic
Multi.Appr.
Leader
Respons.
Health
Car.Ornt.
Adapt.
Persev.
Ethics
HSGPA
SAT/ACT
Male-Female
-.15
.00
.10
-.17
-.16
-.15
-.25
.52
-.17
.00
-.23
-.19
-.04
.43
Cauc-Afr.Am.
.13
.36
.05
.31
-.08
.14
.16
.44
-.55
.11
-.19
.22
-.61
-1.46
Cauc-Hisp.Am.
.10
.23
.00
.01
-.46
.10
.01
.30
-.08
.16
.00
.29
.42
1.01
23
Percent of subgroups admitted under
various strategies
Top 15%
Top 50%
Top 85%
All
Hispanic
Cog Cog+
4.3 6.4
4.0 4.6
4.5 4.7
3.7
Asian
African
Caucasian
Cog Cog+ Cog Cog+ Cog Cog+
17.8 14.9
.9 4.1
77.0 74.6
10.5 10.1 8.3 10.0
77.1 75.3
7.6 7.7 18.4 18.7
69.5 69.0
9.0
19.4
67.8
Cog=equally weighted composite of HSGPA and SAT/ACT
Cog+=equally weighted composite of HSGPA, SAT/ACT, and Non-cognitive
measures.
24
Average Cumulative GPA for Subgroup Members who
graduated at various levels of selectivity
Hispanic
Asian
African
Caucasian
N Cog N Cog+ N Cog N Cog+ N Cog N Cog+ N Cog N Cog+
Top15 4 3.67 6 3.57 23 3.88 18 3.84 0
1 3.66 208 3.88 197 3.87
Top50 17 3.51 17 3 .42
55 3.65 54
3.63 15 3.46 15
3.38 651 3.66 638 3.66
Top85 23 3.44 22
64 3.62 64
3.62 49 3.05 47
3.08 929 3.52
3.47
905 3.54
25
Proportion of Subgroups Graduating in Four Years under
Different Levels of Selectivity
Hispanic
Cog
Top15 93
Top50 79
Top85 77
Cog+
90
78
76
Asian
Cog Cog+
89
91
72
75
68
68
African
Cog Cog+
100 92
87 87
79 78
Caucasian
Cog
72
59
51
Cog+
70
60
52
26
Admits versus Applicants: Standardized
mean differences (d)












Knowledge
Continuous Learning
Artistic Appreciation
Multicultural Appreciation
Leadership
Social Responsibility
Health
Career Orientation
Adaptability
Perseverance
Ethics
Situational Judgment
2004
.55
.43
.35
.48
.43
.46
.26
.25
.22
.39
.62
.50
2006
.54
.49
.30
.40
.35
.46
.30
.21
.25
.32
.53
.52
27
Conclusions on Faking Research (represents
summary of a series of studies






Applicants score higher than current students
(d=.2 to .6)
Coaching has a significant impact on the degree
of score inflation
Elaboration can minimize score inflation,
but…feasibility is an issue and its effects do not
appear to generalize to nonelaborated items
Elaboration has no impact on validity
Warnings do not appear to have much effect,
but…generalizability to an applicant situation has
not been evaluated.
Are they any less fakable than essays or less
inflated than letters of recommendation???
28
Sample Bogus Items: Carelessness

How often in the past year have you programmed in AJMR?

never
once
twice
three or four times
five times or more





How often, in the past three years, have you operated a
rhetaguard?

never
once
twice
three or four times
five times or more




29
Profiling Subgroups of Students
based on HSGPA, SAT/ACT and
Noncognitive Variables
30
31
32
33
34
35
36
Outcomes and Implications





There were predicted outcome differences (GPA,
absenteeism, satisfaction) across profile groups
Highly motivated, career-oriented group is most
likely to respond to remedial efforts particularly if
they relate to their career objectives
Marginal group without career objectives is a high
risk group. Career counseling and intensive
remediation may be necessary
Efforts to broaden the scope of interests of the
high ability, culturally limited may be desirable in
some universities
Students in the high ability well rounded group
might be identified as potential student leaders
and peer mentors/tutors.
37
Overall Conclusions





We can develop valid noncognitive measures that
relate to GPA and other important student
outcomes
Faking of the biodata and SJI remains a problem.
Reactions to their use are not significantly
different than reactions to the ACT/SAT
Subgroup differences are minimal and certainly
much less than those we find for cognitive ability
measures
There may be useful other ways to employ these
instruments; that is, to identify subgroups for
whom interventions designed to retain them will
be needed.
38
Thank you for your attention!
39