Why a measure of noncognitive
abilities is essential to drive
student success
Innovations 2013
Dallas, Texas
March 2013
Victoria Monaghan and Steve Robbins
Educational Testing Service
We don’t need to remind you:
More than 1.7 million students begin community college each year.
But many are
unprepared
• 50 percent of community college
students need remedial – or below
college level – courses
• 20 percent of 4-year college students
also start in remediation
• 40 percent of these students – frustrated
by their low placement –
don’t even bother to enroll in classes
2
And the path to college-level work has detours
4 our of every 10 students in remedial classes never complete
their remedial sequence
Placement tests alone do not yield strong predictions of how students will perform in college
We believe there is a critical need to use a holistic, multiple measures
approach to college course placements
3
What are Noncognitive Skills?
Conscientiousness
Metacognition
Social Support
Teamwork
Self Efficacy
Worry
Study
Skills
Anything outside of
“academic ability” or
“academic intelligence”
that contributes to or is
part of student learning
Institutional
Commitment
Motivation
Goal
Setting
4
Test Taking
Strategies
“Genius is 1%
inspiration
and 99%
perspiration.”
~Thomas Edison
The appeal of these skills
• We like them
• We (and others) value them
• They do indeed work
• We can change them
5
Our mission statements agree,
identifying noncognitive skills frequently
Taxonomy of Valued Academic Skills:
Mission Statements from 35 schools
(College Board/Michigan State University)1,2
•
•
•
•
•
1
2
Knowledge, learning, mastery
of general principles
Continuous learning,
intellectual interest and
curiosity
Artistic cultural appreciation
and curiosity
Multicultural tolerance and
appreciation
Leadership
•
•
•
•
•
•
•
Social responsibility,
citizenship and involve
Physical and psychological
health
Career orientation
Adaptability and life skills
Perseverance
Interpersonal skills
Ethics and integrity
Schmitt, Oswald, Kim, Imus, Drzakowski, Friede, & Shivpuri (2007).
Oswald, Schmitt, Kim, Ramsay, & Gillespie (2004).
6
And employers care as well
Conference Board/SHRM survey of > 400 employers from across the United States
(Casner-Lotto & Barrington, 2006)
Skill
Cognitive
Noncognitive
% Important
% Prepared
English Language
88
26
Mathematics
64
18
Science
33
15
Teamwork/ Collaboration
94
25
Professionalism/ Work Ethic
94
17
Ethics/ Social Responsibility
86
18
7
Adding a noncognitive
assessment allows for a
holistic view of students
• Respects the whole student (examines cognitive,
non-cognitive, behavioral factors)
• Focuses on the unique individuality of students
• Focuses on factors educators can control
and empower students to modify: such as
motivation, social connectedness, as opposed to
socio-economic, situational factors
• A common language for early warning, advising,
and acceleration decisions
8
Domain
Academic Skills
Tools and strategies
for academic success
Commitment
Active pursuit toward
an academic goal.
Self-Management
Reactions to
academic
and daily stress
Facet
Definition
Organization
Strategies for organizing work and time.
Meeting Class Expectations
Doing what’s expected to meet the
requirements of your course including
assignments and in-class behaviors.
I attend almost all of my classes.
I complete the reading that is assigned to
me.
Commitment to College Goals
Perceived value and determination to
succeed in and complete college.
One of my life goals is to graduate college.
The benefit of a college education
outweighs the cost.
Institutional Commitment
Attachment to and positive evaluations of This is the right school for me.
the school.
I’m proud to say I attend this school.
Sensitivity to Stress
Tendency to feel frustrated, discouraged,
or upset when under pressure or
burdened by demands.
I get stressed out easily when things don't
go my way.
I am easily frustrated.
Academic Self-Efficacy
Belief in one’s ability to perform and
achieve in an academic setting.
I'm confident that I will succeed in my
courses this semester.
I can do well in college if I apply myself.
Test Anxiety
General reactions to test-taking
experiences, including negative thoughts
and feelings (e.g., worry, dread).
When taking a test, I think about what
happens if I don't do well.
Before a test, my stomach gets upset.
Connectedness
A general sense of belonging and
engagement.
Social Support
Institutional Support
Connecting with
people and students
resources for success
Barriers to Success
Example Items
I make a schedule for getting my school
work done.
I take due dates seriously.
I feel connected to my peers.
People understand me.
If I don't understand something in class,
Attitudes about and tendency to seek help I ask the instructor for help.
from established resources.
I know how to find out what's expected of
me in classes.
Family pressures make it hard for me to
Financial pressures, family responsibilities,
commit to school.
conflicting work schedules, and limited
People close to me support me going to
institutional knowledge.
college. 9
From the “so what” to the “now what”……..
Data Integration
& Use Across Silos
Holistic Assessment
Student
Commitment &
Classroom Success
How do we...
• Identify these skills in an operational setting?
• Best disseminate that information to students, faculty, staff, and
institutions?
• Develop and target sound and practical interventions?
• Consider individual differences in background, skills, abilities,
and experiences?
10
Early Intervention is key
Three basic implications:
•
Educators have a profound influence on students
– Faculty can motivate and inspire students
– Educators can also shape and foster their intentionality and academic
determination
• Observable behaviors are indicative of internal states
– E.g., Motivation, engagement
• Educators have the opportunity and capability to guide
students towards desirable classroom/academic
behaviors
11
Noncognitive Assessment Experience
at Wright College
semester
Note. Results from Wright College (Spring 2009).
n = 453; All paths are statistically significant.
12
Research shows that behavioral
monitoring and intervention are key to
student success
Math
Readiness
Effort Level
High
Medium
Low
High
.96
.91
--
Medium
.94
.77
.82
Low
.86
.61
.38
Note. N = 411. Success rates were not calculated for cells
with insufficient sample sizes (denoted by a dash).
13
ETS/CCC Collaboration Objectives
• Combine COMPASS and Psychosocial Measure
to accelerate student progress and college
completion
– Students placed within the upper-band with high
psychosocial scores can enroll in the next level course
• Allow more individualized, actionable plans for
students tied to curricular goals
• Non-cognitive survey and in-class behavioral
ratings will be used to trigger Early Alert and
intervention strategies
• Create supplementary services tied to Early Alert
and to behavioral monitoring
Comprehensive Early Interventions
Map support services to non-cognitive psychological constructs:
Academic Skills, Self Management, Motivation, Social Support
Non-Cognitive
Construct
Psycho-social Skills
Existing natural fit support
services at CCC
Academic Skills
Study skills
Study skills workshop
Time management skills
Study skills workshop
Emotional control
Wellness Center
Resilience
Meeting with advisors,
tutors and instructors
Degree intention
Transfer center, advising
Class attendance
Advisors, tutors, instructors
Engagement
Advisors, tutors, instructors
Help seeking
New student orientations,
advisors, service referrals
Peer support
Student clubs and
organizations
Self Management
Motivation
Social support
Gap analysis: Need
additional intervention
strategies at individual
college?
Student Name:
Student ID:
Score Date:
Juan Rodriguez
12-34567
Sep 12, 2013
Matthew’s Background
• Race: Latino
• Gender: Male
• Parental Education: Unknown (Mother), Unknown (Father)
Academic Skills – Tools to succeed in the classroom
Advisor Report
COURSE ACCELERATION
ACADEMIC SUCCESS
SCORE*
NOT RECOMMENDED
LOW
RETENTION SUCCESS
SCORE*
LOW
SKILL REPORT
Skills
• Sometimes uses organizational tools (e.g., checklist, planner)
• Occasionally mismanages his schedule regarding assignments and exams
Next Steps
Refer Juan to the Tutoring Center for goal setting strategies and organizational tools. You
can find additional ways of working with Matthew by clicking here.
MODERATE
Commitment – Active pursuit towards an academic goal
Skills
• Moderately committed to attaining an associate degree
• Consistently attends class
• Moderate interest in engaging with resources on campus
Next Steps
The Career Center can help Juan align his degree goals with his career goals. You can find
additional ways of working with Matthew by clicking here.
MODERATE
Self-Management – Managing the daily expectations
Skills
• Usually calm during important tests and assignments
• Occasionally gives up when frustrated with assignments
Next Steps
Refer Juan to the Wellness Center on campus to help him manage pressures and succeed.
For effective ways of helping Matthew manage every day academic stress, click here.
MODERATE
Social Support – Networks to support academic success
Skills
• Rarely engages resources on and off campus
• Parents show little support for his college attendance
Next Steps
The Office of Student Life can connect Juan with valuable student groups on campus. For
helpful tools and tips about student connections, click here.
LOW
*Course Placement: Recommended academic placement action for student.
* Academic Success Score: Weighted composite of student’s SuccessNavigator™ profile and other academic indicators of student preparedness, such as self or institution reported high school GPA, SAT/ACT, etc.
* Retention Success Score: Projected likelihood that student will return for a second year at the institution .
Juan
Background: Latino Male
Major: Business
Status: Part-time (10 credits)
Current GPA: 2.29
Age:18
Parental Education: Unknown (Mother), Unknown (Father)
Misc: Undocumented ESL student who is in the National Guard to obtain citizenship. He works 40
hours/week at UPS to cover his tuition and help his family financially.
Academic Preparation:
•2.3 HS grade point average
• Scored 900 on SAT
• Placed in bottom third of placement tests
• Taking remedial Algebra and English
Success Navigator profile indicates:
He has moderate Academic Skills, Commitment, and SelfManagement, but low Social Support
Notes:
Overall Juan is struggling both academically and socially and is at-risk for dropping out. His financial /
work responsibilities and full-time student status have limited his ability to focus on his academics
and interact with other students. Since he has little support for attending college he would benefit
from engaging with formal (e.g. tutoring) and informal resources on campus. Also, because he
remains dedicated to earning his degree and is confident in his ability to succeed, he might benefit
from reducing his course load even further which will help him balance all his responsibilities.
Student Withdrew
Student Name:
Student ID:
Score Date:
Sarah Smith
13-75680
Sep 2, 2013
Sarah’s Background
• Race: Caucasian
• Gender: Female
• Parental Education: High School (Mother),
Below High School (Father)
Academic Skills – Tools to succeed in the classroom
Advisor Report
COURSE ACCELERATION
ACADEMIC SUCCESS
SCORE*
RECOMMENDED
RETENTION SUCCESS
SCORE*
MODERATE
HIGH
SKILL REPORT
Skills
• Sometimes uses organizational tools (e.g., checklist, planner)
• Occasionally mismanages her schedule regarding assignments and exams
Next Steps
Refer Sarah to the Tutoring Center to provide goal setting strategies and organizational
tools. You can find additional ways of working with Sarah by clicking here.
MODERATE
Commitment – Active pursuit towards an academic goal
Skills
• Strongly committed to attaining an associate degree
• Consistently attends class
• Strong interest in engaging with resources on campus
Next Steps
The Career Center can help Sarah align her degree goals with her career goals. You
can find additional ways of working with Sarah by clicking here.
HIGH
Self-Management – Managing the daily expectations
Skills
• Usually calm during important tests and assignments
• Sometimes gives up when frustrated with assignments
Next Steps
Refer Sarah to the Wellness Center on campus to help her manage pressures and
succeed. For effective ways of helping Sarah manage every day academic stress,
click here.
MODERATE
Social Support – Networks to support academic success
Skills
• Sometimes engages resources on and off campus
• Parents show some support for her college attendance
Next Steps
The Office of Student Life can connect Sarah with valuable student groups on
campus. For helpful tools and tips about student connections, click here.
MODERATE
*Course Placement: Recommended academic placement action for student.
* Academic Success Score: Weighted composite of student’s SuccessNavigator™ profile and other academic indicators of student preparedness, such as self or institution reported high school GPA, SAT/ACT, etc.
* Retention Success Score: Projected likelihood that student will return for a second year at the institution .
Sarah
Background: Caucasian Female
Major: Liberal Arts
Status: Full-time
Current GPA: 3.0
Age:19
Parental Education: Below High School (Mother), High School (Father)
Misc: She is the first member of her family to attend college. She was very involved in high school
extra-curriculars. She was class secretary, earned a varsity letter in soccer, and was a member
of the chorus.
Academic Preparation:
• 2.8 HS grade point average
• Scored below 900 on SAT
• Placed in remedial Algebra
Noncognitive profile indicates:
She has strong Commitment and above average Academic Skills, Self Management, and Social
Support
Prime candidate for acceleration
Notes:
Although Sarah struggles academically, especially in math, she is a conscientious student who is in
regular contact with her advisor and has used study groups to help her academically. Since she has
very high commitment both to the institution and to the goal of earning a degree, she would benefit
from continual encouragement to make use of the academic resources on campus such as the
tutoring center. She should also be encouraged to maintain regular contact with her professors.
Student Name:
Student ID:
Score Date:
Janet Jerzyk
18-34287
Sep 25, 2013
Janet’s Background
•
•
•
•
Advisor Report
COURSE ACCELERATION
ACADEMIC SUCCESS
SCORE*
RECOMMENDED
HIGH
Major: Education
Race: Multiracial/multiethnic
Gender: Female
Parental Education: Some College (Mother), High School (Father)
Academic Skills – Tools to succeed in the classroom
RETENTION SUCCESS
SCORE*
MODERATE
SKILL REPORT
Skills
• Always uses organizational tools (e.g., checklist, planner)
• Successfully manages her schedule regarding assignments and exams
Next Steps
Refer Janet to the Tutoring Center for goal setting strategies and organizational
tools. You can find additional ways of working with Janet by clicking here.
HIGH
Commitment – Active pursuit towards an academic goal
Skills
• Moderately committed to attaining an associate degree
• Consistently attends class
• Moderate interest in engaging with resources on campus
Next Steps
The Career Center can help Janet align her degree goals with her career goals. You
can find additional ways of working with Janet by clicking here.
MODERATE
Self-Management – Managing the daily expectations
Skills
• Calm during important tests and assignments
• Persists when frustrated with assignments
Next Steps
Refer Janet to the Wellness Center on campus to help her manage pressures and
succeed. For effective ways of helping Janet manage every day academic stress,
click here.
HIGH
Social Support – Networks to support academic success
Skills
• Sometimes engages resources on and off campus
• Parents show some support for her college attendance
Next Steps
The Office of Student Life can connect Janet with valuable student groups on
campus. For helpful tools and tips about student connections, click here.
MODERATE
*Course Placement: Recommended academic placement action for student.
* Academic Success Score: Weighted composite of student’s SuccessNavigator™ profile and other academic indicators of student preparedness, such as self or institution reported high school GPA, SAT/ACT, etc.
* Retention Success Score: Projected likelihood that student will return for a second year at the institution .
Janet
Background: Multiracial/Multi-ethnic Female
Major: Education
Status: Full-time
Current GPA: 3.25
Age:19
Parental Education: Unknown (Mother), Unknown (Father)
Misc: Janet has significant financial hardship and has received considerable financial aid. She also
works 25 hours/week as a waitress. She picks up extra shifts whenever possible.
Academic Preparation:
• 3.3 HS grade point average (top 20% of class)
• Scored 1200 on SAT
• Placed into college level courses
Noncognitive profile indicates:
She has strong Academic Skills, and Self-Management, and average Commitment and Social
Support.
Notes:
Janet is a capable student with strong organizational skills with promising academic potential. She is
confident in her ability to succeed and does not easily get distressed by her school work. Janet is not
inclined to seek out help and should be encouraged to make use of campus resources. In particular,
Janet has financial constraints and should be encouraged to reach out to her financial aid counselor
to discuss the availability of scholarships
Also, although Janet is committed to earning a degree, she has expressed interest in transferring
soon and may not feel connected to institution. She may benefit from seeing a transfer counselor
and engaging in extra-curricular activities.
Student Name:
Student ID:
Score Date:
Julia Fuentes
13-34588
Sep 15, 2013
Julia’s Background
• Race: Latina
• Gender: Female
• Parental Education: Some College (Mother),
Some College (Father)
Academic Skills – Tools to succeed in the classroom
Advisor Report
COURSE ACCELERATION
ACADEMIC SUCCESS
SCORE*
NOT RECOMMENDED
HIGH
RETENTION SUCCESS
SCORE*
LOW
SKILL REPORT
Skills
• Sometimes uses organizational tools (e.g., checklist, planner)
• Occasionally mismanages her schedule regarding assignments and exams
Next Steps
Refer Julia to the Tutoring Center to provide goal setting strategies and organizational
tools. You can find additional ways of working with Julia by clicking here.
MODERATE
Commitment – Active pursuit towards an academic goal
Skills
• Strongly committed to attaining an associate degree
• Consistently attends class
• Strong interest in engaging with resources on campus
Next Steps
The Career Center can help Julia align her degree goals with her career goals. You can
find additional ways of working with Julia by clicking here.
HIGH
Self-Management – Managing the daily expectations
Skills
• Usually anxious during important tests and assignments
• Occasionally gives up when frustrated with assignments
Next Steps
Refer Julia to the Wellness Center on campus to help her manage pressures and
succeed. For effective ways of helping Julia manage every day academic stress, click
here.
LOW
Social Support – Networks to support academic success
Skills
• Rarely engages resources on and off campus
• Parents show little support for her college attendance
Next Steps
The Office of Student Life can connect Julia with valuable student groups on campus.
For helpful tools and tips about student connections, click here.
LOW
*Course Placement: Recommended academic placement action for student.
* Academic Success Score: Weighted composite of student’s SuccessNavigator™ profile and other academic indicators of student preparedness, such as self or institution reported high school GPA, SAT/ACT, etc.
* Retention Success Score: Projected likelihood that student will return for a second year at the institution .
Julia
Background: Latina Female
Major: Liberal Arts, transfer to Psychology
Status: Full-time
Current GPA: 4.0
Age:19
Parental Education: Some College (Mother), Some College (Father)
Misc: Is the first in her family to attend college. She has five younger siblings whom she is
responsible for while her parents work. She was a top student in high school and a member
of the National Honor Society.
Academic Preparation:
• 3.9 HS grade point average (top 5% of class)
• Scored 27 on ACT (over 1200 on SAT)
• Placed into college level courses
Noncognitive profile indicates:
She has significantly above average Commitment, moderate Academic Skills and Self-Management,
and low Social Support
Notes:
Julia is an academically able student who made the Dean’s List in her first semester despite having
taken 17 credits. She is very committed to earning a degree and her long term educational
aspirations of earning a Ph.D. Due to her desire to do well academically, she is easily stressed and
gets very nervous prior to tests. In addition, because of her heavy course load and family
responsibilities she has not made many friends on campus. Julia would benefit from seeing a
counselor to help her overcome her anxiety and should be encouraged to engage in extra-curricular
activities.
Questions?
Or ask us later …….
Steve Robbins, Director, Research Innovations, Academic and Workforce Readiness
and Success, ETS. srobbins@ets.org
Victoria Monaghan, Director of Strategic Initiatives in Higher Education, ETS.
vmonaghan@ets.org
24
References
Allen, J., & Robbins, S. (2010). Effects of interest-major congruence, motivation, and academic performance on
timely degree attainment. Journal of Counseling Psychology, 57, 23-35.
Allen, J., Robbins, S., Casillas, A., & Oh, I.-S. (2008). Third-year college retention and transfer: effects of academic
performance, motivation, and social connectedness. Research in Higher Education, 49(7), 647-664.
American Association of Community Colleges (AACC). (2009). Community college fast facts. Retrieved July 10,
2009 from http://www2.aacc.nche.edu/research/index.htm
Belfield, C. R., & Crosta, P. M. (2012). Predicting success in college: The importance of placement tests and High
school transcripts. CCRC Working Paper No. 42. New York: Community College Research Center, Teachers
College, Columbia University.
Berkner, L., & Choy, S. (2008). Descriptive summary of 2003-04 beginning postsecondary students: Three years
later. Washington, DC: U.S. Department of Education, Institute of Education Sciences.
Bragg, D. D. (2001). Community college access, mission and outcomes: Considering intriguing intersections and
challenges. Peabody Journal of Education. 76(1), 93-116.
Community College Research Center. (2002a). Accountability and learning outcomes in community colleges.
Retrieved July 7, 2008, from http://ccrc.tc.columbia.edu/Seminar.asp?uid=8
Community College Research Center. (2002b). Accountability and learning outcomes in community colleges.
Retrieved July 7, 2008, 2008, from http://ccrc.tc.columbia.edu/Seminar.asp?uid=8
Habley, W., Bloom, J, & Robbins, S. (2012). Increasing persistence: Research-based strategies for college student
success. San Francisco: Jossey-Bass
25
References (cont.)
Habley, W., & McClanahan, R. (2004). What works in student retention – All survey colleges. Iowa City, IA: ACT,
Inc.
Harris, S. (1991) Can’t you guys read? Cartoons on academia. New Brunswick, NJ: Rutgers University Press.
Herzog, K., & Miller, M. P. (Eds.). (1985). Scholarship: More great cartoons from the Kappan. Bloomington, IN: Phi
Beta Kappa International Inc.
Le, H., Casillas, A., Robbins, S., & Langley, R. (2005). Motivational and skills, social, and self-management
predictors of college outcomes: Constructing the Student Readiness Inventory. Educational and
Psychological Measurement, 65, 482-508.
Li, K., & Cordes, C. (2010). Individualizing the Delivery of Math Developmental Education. Talk presented at the
34th Annual National Association of Developmental Education Conference, Columbus, OH.
Lotkowski, V., Robbins, S., & Noeth, R. (2004). The role of academic and non-academic factors in improving
college retention. Retrieved from ACT, Inc. website: www.act.org/research/policy/index.html.
Marke, R., & Robbins, S., (2013). Breaking the developmental education logjam: A holistic approach to
acceleration. Princeton, NJ: Educational Testing Service.
Marke, R., & Robbins, S., (2013). A holistic view of course placement fecisions – Avoiding the HS GPA trap.
Princeton, NJ: Educational Testing Service.
McCabe, R. H. (2000). No one to waste. Washington, DC: Community College Press.
NCES (2004). Remedial education at degree-granting postsecondary institutions in fall 2000 (NCES 2004-010).
NISOD (2012, March). Assessing college-readiness assessments. Webinar delivered at the annual NISOD
Conference.
Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students: A third decade of research. San Francisco:
Jossey-Bass.
26
References (cont.)
Peterson, C. H., Casillas, A., & Robbins, S. B. (2006). The Student Readiness Inventory and the Big Five:
Examining social desirability and college academic performance. Personality and Individual Difference, 41,
663-673.
Porchea, S., Allen, J., Robbins, S., & Phelps, R. (in press). Predictors of long-term enrollment and degree
outcomes for community college students: Integrating academic, psychosocial, socio-demographic, and
situational factors. Journal of Higher Education.
President’s Commission on Higher Education. (1947). Organizing higher education. Washington, DC: Author.
Robbins, S. B., Allen, J. Casillas, A., Akamigbo, A., Saltonstall, M., Campbell, R., Mahoney, E., & Gore, P.A.
(2009). Associations of resource and service utilization, risk level, and college outcomes. Research in
Higher Education, 50(1), 101-118.
Robbins, S., Allen, J., Casillas, A., Peterson, C., & Le, H. (2006). Unraveling the differential effects of
motivational and skills, social, and self-management measures from traditional predictors of college
outcomes. Journal of Educational Psychology, 98, 598-616.
Robbins, S. B., Lauver, K., Le, H., David, D., Langley, R., & Carlstrom, A. (2004). Do psychosocial and study skill
factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130, 261-288.
Robbins, S., Oh, I., Le, H., & Button, C. (2009). Intervention effects on college performance and retention as
mediated by motivational, emotional, and social control factors: Integrated meta-analytic path analyses.
Journal of Applied Psychology, 94, 1163-1184.
Scott-Clayton, J. (2012). Do High-Stakes Placement Exams Predict College Success? Predicting Success in
College: The Importance of Placement Tests and High School Transcripts. CCRC Working Paper No. 41. New
York: Community College Research Center, Teachers College, Columbia University.
Swail, W. S. (2004, January 23). Legislation to improve graduation rates could have the opposite effect. The
Chronicle of Higher Education, B16.
27