What Works in Promoting
Student Success
Steve Robbins
ACT, Inc.
University of Missouri-KC Retention Retreat
1
Overview
•
•
•
•
Why College Students Stay
What Works in Promoting Student Success
The Importance of an Early Warning System
A Model for Institution-wide Assessment and
Intervention
• A Model for Effective Placement and
Intervention with Developmental Students
• Putting It All Together
2
Harris, S. (1991) Can’t you guys read? Cartoons on Academia. Rutgers
University Press. New Brunswick, NJ, p.74
3
Why College Students Stay
Four-Year College Students:
• First-year GPA has large effects on the
likelihood of retention and transfer.
• Motivation (Academic Discipline) and precollegiate academic preparation have
indirect effects on retention and transfer by
working through first-year GPA.
4
Why College Students Stay (cont)
• Social connection has a direct effect on
retention.
• SES predictive of transfer behavior:
– Higher SES students transfer while poor
students give up.
– African-American students have high
commitment but difficulty with classes
resulting in higher drop-out rates.
See Robbins et al. (2006); Allen et al. (2008)
5
Harvard Business Review. (July-August 2007)
6
Meta-Analysis and Validity
Generalization as Key Tools
• Summarizing the effect of something over
multiple data points
• Create confidence intervals of the true
effect size
• Interpretation of multiple studies is better
than any individual study
7
Testing Integrated Meta-Analytic Path
Analysis
Motivation/
Skills
Intervention
Self -Regulation
Performance
Persistence
Social
Engagement
The Effects of College Interventions on College Outcomes as Mediated by PSF’s
8
Robbins et al. (2009)
Categorizing College Interventions
• Orientation (21 hours) – summer, early fall, time-limited
• Freshman Year Experience (45 hours)
• Academic (8 hours)
– Study skills
– Learning strategies
– Note-taking
• Self-Management (6 hours)
– Stress management
– Self-control
– Anxiety management
• Hybrid of Academic & Self-Management (12 hours)
9
Categorizing Psychosocial Factors
(PSFs)
Three Categories
• Motivation:
– Academic Discipline
– Commitment to College
• Self-Regulation:
– Steadiness
– Academic Self-Confidence
• Social Engagement:
– Social Connection
– Social Activity
Robbins, S., Allen, J., Casillas, A., Peterson, C., & Le, H. (2006)
Robbins, et al. (2004)
10
A Summary of Findings
• Interventions with an academic focus are
key
• Boost academic interventions using self
management strategies, i.e., Hybrid
• Align specific interventions to narrowed
outcomes (PSF and/or success) to
increase treatment effect
11
A Summary of Findings (cont)
• Rethink the goals & the focus of Freshman
Year Experience
• Understand mediating role of motivation
and self-regulation factors to promote
student success
12
Giving Guidance
Herzog & Miller (1985)
13
Using the Student Readiness Inventory
(SRI) as an Early Warning System
Met our criterion of being grounded in research:
– Reviewed meta-analysis of 109 studies that examined
predictors of academic performance and retention
Met our criterion of considering the whole student:
– Motivation, self-regulation, and social engagement
coupled with academic preparation factors
– Factors are amenable to change
14
Using the Student Readiness Inventory
(SRI) as an Early Warning System
Able to:
– Administer at orientation and have for early in the
semester
– Create an early warning system for targeted
intervention
– Aid support of previously identified “at-risk” groups
– Create a cross walk of scales to our services and use
within our existing framework of services including
• Academic coursework for freshmen
• Student Affairs programs and services
• Academic advising
15
Development of the Student
Readiness Inventory
Motivation and Skills
–
–
–
–
–
–
Commitment to College
Goal Striving
Academic Discipline
General Determination
Study Skills
Communication Skills
Social Engagement
– Social Activity
– Social Connection
Self-Management
– Academic Self-Confidence
– Steadiness
Robbins et al. (2004)
16
SRI Scale Definitions & Sample Items
SRI Scale
Definition
Sample Item
Academic Discipline
The amount of effort a student puts into
schoolwork and the degree to which a
student is hardworking and conscientious.
I consistently do my school work well.
Academic Self-Confidence
The belief in one's ability to perform well in
school.
I achieve little for the amount of time I spend
studying.
Commitment to College
One's commitment to staying in college and
getting a degree.
A college education will help me achieve my
goals.
Communication Skills
Attentiveness to others' feelings and
flexibility in resolving conflicts with others.
I'm willing to compromise when resolving a
conflict.
General Determination
The extent to which one strives to follow
through on commitments and obligations.
It is important for me to finish what I start.
Goal Striving
The strength of one's efforts to achieve
objectives and end goals.
I bounce back after facing disappointment or
failure.
Social Activity
One's comfort in meeting and interacting
with other people.
I avoid activities that require meeting new
people.
Social Connection
One's feelings of connection and
involvement with the college community.
I feel part of this college.
Steadiness
One's responses to and management of
strong feelings.
I have a bad temper.
Study Skills
The extent to which students believe they
know how to assess an academic problem,
organize a solution, and successfully
complete academic assignments.
I summarize important information in
diagrams, tables, or lists.
Sample SRI Profiles
• Class of 2007 college students completed the SRI at the
beginning of their 1st semester (Fall 2003)
• Sample profiles selected to contrast academic
preparation and motivation
Motivation
ACT
score
18
Low
High
Low
Student A
Student B
High
Student C
Student D
Low ACT score (composite = 14)
Student A
LOW ACT
Scale
Student B
LOW MOTIVATION
HIGH MOTIVATION
Percentile
Percentile
0
10
20
30
40
50
60
70
80
90
100
0
6
95
General Determination
17
72
Goal Striving
13
80
9
51
Study Skills
47
59
Communication Skills
34
99
Social Connection
76
49
Social Activity
9
83
Academic Self-Confidence
2
46
Steadiness
6
71
Academic Discipline
Commitment to College
Low
SRI Indices
Outcome
19
Medium
High
10
20
Low
30
40
50
60
70
Medium
80
90
100
High
Academic = 1
Academic = 20
Retention = 3
Retention = 17
1st-year GPA = 1.5
1st-year GPA = 2.22
Fall 07 status = dropped out
Fall 07 status = still enrolled
High ACT score (composite = 27)
Student C
HIGH ACT
Scale
Student D
HIGH MOTIVATION
LOW MOTIVATION
Percentile
Percentile
0
10
20
30
40
50
60
70
80
90
0
100
3
92
55
50
4
26
Commitment to College
22
22
Study Skills
31
38
Communication Skills
46
12
7
45
Social Activity
15
57
Academic Self-Confidence
32
92
9
53
Academic Discipline
General Determination
Goal Striving
Social Connection
Steadiness
Low
SRI Indices
Outcome
20
Medium
High
10
Low
20
30
40
50
60
70
Medium
80
90
High
Academic = 60
Academic = 96
Retention = 50
Retention = 88
1st-year GPA = 3.7
1st-year GPA = 4.0
Fall 07 status = dropped out
Fall 07 status = graduated
100
Two Case Examples:
1. Targeting at-risk students and
coordinating interventions across all
segments of the university
2. Developmental classes in college
21
Case I: Northern Arizona
• Public university
• 4-year institution with 800 faculty serving 22,000
students
• 13,000 undergraduate students on the Flagstaff
campus
• ~30% dropout rate after freshman year
• Implemented a card swiping system to monitor
resource use
• Required the SRI and created a smorgasbord of
systematic interventions for at-risk students
Robbins et al. (2008)
22
Strategic Intervention Model:
What We Wanted
• Implement early in the first semester
• Include groups known to be at risk
• Collaborate and not duplicate
1. Academic advising and first year programs
2. Native American Student Services outreach
program
3. Summer bridge program and first year mentoring
4. Student Support Services, a TRIO program
5. Any key groups interfacing with new freshmen
23
Strategic Intervention Model:
What We Wanted (cont)
• Be systematic
• Address the whole student
• Rely on a framework grounded in research
24
SRI Work Group
Creating a Network of Academic and
Social Resources
• VP identified a group of staff to spearhead effort
• Group planned and administered instrument
over the summer
25
SRI Work Group (cont)
•
Group identified:
1. Target populations (risk categories)
2. Services linked to needs
3. Other campus collaborator
– Freshman year seminar/experience
– First year advising center
4. A process for:
–
–
–
–
26
Assigning students to most appropriate offices
Making appointments with students
Notification and outreach
Follow up expectations
Program Model
Assigning Students & Distributing
Profiles
•
•
•
•
27
Student Affairs Programs and Services
Academic Advising Center for Freshmen
Freshman Year Experience 101
Freshman Skills-Based Course, EPS 101
Prioritizing Student Groups in
Student Affairs
“Waterfall” Approach: Assessing Risk &
Targeting Students to Be Seen by Each Office
• Summer Bridge Program
(Ethnic Minority, First Gen, Low Income)
• Native American Students
(Native American Student Services)
• Other Ethnic Minorities
(Multicultural Student Center)
28
Prioritizing Student Groups in
Student Affairs (cont)
• Other First Generation/Low Income
(SSS/Learning Assistance Centers)
• Living Off Campus
(Learning Assistance Centers)
• Other Low Scoring
(Advising and FYE/EPS)
(less than 50th percentile on either index)
29
Student Groups & University
Departments
No SRI
11%
Ethnic
Minority
7%
STUDENT AFFAIRS
Summer
Bridge
5%
Ethnic Minority Students
Native
American
3%
Invalid / Incomplete
10%
1st Gen & Low Income
6%
1st Gen and Low Income
4%
Learning Community
4%
African
American
2%
General
Advising Center
21%
Skills Class
5%
Low Scores Advising Center
9%
30
FYE Class
15%
–Multicultural Student Center
–Native Amer. Student Services
–Student Life (African American)
First Generation/Low Income Students
–Student Support Services
–Learning Assistance Centers
Learning Communities/Res Life
FIRST YEAR ADVISING CENTER
FRESHMAN YEAR PROGRAMS
–FYE course
–Skills-based course
COORDINATING OFFICES
–Dean of Students
–Assessment Office
One-on-One Meetings
in Student Affairs
Reaching Out and Intervening: A
Systematic Outreach
• Assign students to programs/office
according to service populations
• Look up student schedules and set
appointment times
31
One-on-One Meetings
in Student Affairs (cont)
• Send postcard home with appointment time
• Send postcard to campus address with
appointment time
1.
2.
3.
4.
5.
32
Personal e-mail with appointment time
Appointment time/date placed on student’s web calendar
Reminder phone call 1-2 days before appointment
Facebook contacts and text reminders
If miss scheduled meeting, protocol for rescheduling at least
two more times.
Meeting Purpose
• Above all else, connect with student.
• Acknowledge the student’s strengths and
identify campus activities to reinforce
them.
33
Meeting Purpose (cont)
• Match the student’s needs to campus
resources.
1. Website: home.nau.edu/emsa/sri.asp
2. Tool Shop: act.org/sri/studentguide/toolshop.html
•
Gather some data on SRI experience.
1. How did you hear about your appointment?
2. How seriously did you take the SRI when you
completed it at orientation?
34
Evaluating the Effects of Meeting
and Other Resource Use
Academic and Persistence
Outcomes
35
2007 Retention in 2008
Group Total
All -- All NAU FYR
(w/o International and Unknown)
Retained
% Retained
2848
884
1964
69%
652
208
444
68%
Targeted and Not Met
427
163
264
62%
Non - Targeted Students
1357
349
1008
74%
Invalid or No SRI
412
164
248
60%
Targeted and Met
36
Not Retained
2007 Retention in 2008
Not
Retained
Group Total
% Retained
102
35
67
66%
Targeted and Met
54
14
40
74%
Targeted and Not Met
23
9
14
61%
Non -Targeted Students
2
1
1
Invalid or No SRI
23
11
12
African American
37
Retained
--All NAU FYR
--52%
2007 Retention in 2008
Group Total
Retained
% Retained
316
109
207
66%
Targeted and Met
175
50
125
71%
Targeted and Not Met
88
37
51
58%
Non - Targeted Students
9
2
7
Invalid or No SRI
44
20
24
Hispanic -- All NAU FYR
38
Not Retained
--55%
2007 Retention in 2008
Group Total
Retained
% Retained
2208
654
1554
70%
Targeted and Met
301
98
203
67%
Targeted and Not Met
258
92
166
64%
Non -Targeted Students
1340
345
995
74%
Invalid or No SRI
309
119
190
61%
White --All NAU FYR
39
Not Retained
First Semester Academic Probation
for fall 2007 & fall 2008
100%
Met
Not Met
80%
60%
32%
40%
20%
19%
25%
18%
0%
2007
40
2008
Conclusions Based on Assessment
•
•
Non-targeted students that were expected to
do well did (74% retained one year later).
Is this number acceptable to UM-KC?
Targeted students who met fared better than
similarly at-risk students who declined to meet.
1. GPA /Academic Probation
2. Retention
•
41
There is room for growth within all categories.
Additional Research Findings and
Projects
Service Utilization
– Academic
– Social
– Recreational
– Academic referrals (advising center)
– Advising/Career sessions (advising center)
2008 Robbins et al study
42
Resource & Services Utilization
Resource/
Service
Category
Ret.
GPA
Ret.
GPA
Ret.
GPA
Academic Services
.75
2.75
.79
2.92
.70
2.63
Recreational
Resources
.77
2.87
.77
2.76
.71
2.67
Social
Resources
.72
2.83
.85
2.91
.69
2.63
Academic
Referrals
--
--
.88
2.95
.60
2.53
.84
2.67
.79
2.83
.57
2.68
Advisory / Career
Sessions
Level of Utilization
High
Moderate
Low
Robbins et al. (2008)
43
Association of Risk Level
& Academic Service Use on Retention
& 1st-year GPA
Risk Level
Service Use
Moderate
Low
Ret.
GPA
Ret.
GPA
Ret.
GPA
0 use
.61
2.11
.72
2.58
.77
3.14
1 – 3 Sessions
.72
2.41
.77
2.81
.84
3.38
4+ Sessions
.62
2.35
.81
2.81
.85
3.22
Gains show
interaction on
GPA
44
High
Summary Points
• Intervene early and strategically
• Designate a visible individual to coordinate
• Collaborate and use natural fits with existing
resources
• Conduct systematic analysis
• Use your data/feedback and improve the
process
• Maximize utility of the instrument
• Go beyond the limits of the instrument
45
Summary Points Continued:
Ensuring Student Success
• Academic preparation & performance are
at the hub
• Be clear on goals
Satisfaction, learning, & persistence are not the same
• Be strategic in your use of resources
• Move the mountain to the students
Don’t be afraid of intrusive advising
46
A Model for Wilbur Wright
College
Illinois Innovations in Education
Awardee for 2009
47
Before course
registration
Beginning
of course
During a
course
College
Developmental
Course
Best Practices
1. New Student
a. Administer COMPASS & SRI1
b. Supportive review & re-test policy2
Non-developmental
courses
Developmental
2. Developmental Student
a. Administer diagnostics1
b. Assess non-cognitive factors for risk2
3. Developmental Instruction
a. Instruction tied to diagnostics2
b. SRI practice & use2
2
3
4
Student Data / Characteristics
College Responses / Services /
Instructional Resources
Student Effort / Involvement /
Investment
College Records
a. Effort of students in class3
b. Students’ non-class service use3
a. Academic support
b. Non-academic
5. Post-Test
End of
course
1
During a
course
4. Monitoring Status
Superscript Legend
a.
b.
c.
d.
Diagnostics1
Placement1
Grades4
Course completion4
6. Next Course
a. Grades4
b. Course completion4
Success in Developmental Courses Dependent on
Academic and Psychosocial Risk
Success Rates in English Composition, by Academic and
Psychosocial (SRI) Risk Levels
SRI Risk Level
Academic
Risk
Level
49
High
Medium
Low
High
23%
40%
44%
Medium
32%
47%
58%
Low
53%
68%
78%
Success in Developmental Courses Dependent on
Academic and Psychosocial Risk (cont.)
Success Rates in Elementary Algebra, by Academic and
Psychosocial (SRI) Risk Levels
SRI Risk Level
Academic
Risk
Level
50
High
Medium
Low
High
15%
28%
27%
Medium
32%
41%
52%
Low
50%
67%
69%
We Observe the Effects of In-class Student
Behavior on Success
Success Rates in Elementary (precredit) Math, by COMPASS
Pretest and Behavior Rating Levels
Behavior Rating Level
COMPASS
Pretest
Scores
51
High
Medium
Low
High
92%
77%
59%
Medium
91%
62%
17%
Low
69%
44%
6%
We Observe the Effects of In-class Student
Behavior on Success (cont.)
Mean COMPASS Posttest Scores for Elementary (precredit)
Math, by COMPASS Pretest and Behavior Rating Levels
Behavior Rating Level
COMPASS
Pretest
Scores
52
High
Medium
Low
High
57.4
48.8
45.9
Medium
43.5
38.3
34.1
Low
39.8
33.9
32.6
We Observe the Effects of In-class Student
Behavior on Success (cont.)
Mean COMPASS Gain Scores for Elementary (precredit) Math,
by COMPASS Pretest and Behavior Rating Levels
Behavior Rating Level
COMPASS
Pretest
Scores
53
High
Medium
Low
High
19.5
8.9
8.7
Medium
15.7
11.4
6.2
Low
19.7
14.0
12.0
The 3 Pillars of Success
Academic
Readiness
Motivation
Instructional
Effectiveness
• Appropriately place and diagnose
• Address motivational skill and engagement behaviors
within and outside classroom
• Connect instruction to the diagnostic and curriculum
targets essential to academic achievement domain
(e.g., Math, English)
54
Wrap-Up and Questions for You?
• Do you have a systematic way of assessing
student risk?
• How coordinated are your services?
• Have Faculty “bought-in”?
• Do you monitor the Return on Investment (ROI)
of your central and college based resources?
• Are you meeting your retention goals?
• Are you meeting your time to degree attainment
goals?
55
References
ACT, Inc. (2008). What We Know about College Success: Using ACT Data to Inform Educational
Issues. Iowa City, IA: Authors.
ACT, Inc. (2007). State of College Readiness for Latino Students. Iowa City, IA: Authors.
ACT, Inc. & The National Council for Community and Educational Partnerships. (2007). Using
EXPLORE® and PLAN® data to evaluate GEAR UP programs. Iowa City, IA: Authors.
ACT, Inc. (2004). Schools Involving Parents in Early Postsecondary Planning. Iowa City, IA: Authors.
ACT, Inc. (2002). Creating Seamless Educational Transitions for Urban African American and
Hispanic Students. Iowa City, IA: Authors.
Allen, J., Robbins, S., Casillas,A., & Oh, I.-S. (2008). Why college students stay: Using academic
performance, motivation, and social engagement constructs to predict third-year college retention
and transfer. Research in Higher Education
Braxton, J., Sullivan, A., & Johnson, R. (1997). Appraising Tinto’s theory of college student departure.
In J. C. Smart (Ed.) Higher Education: Handbook of Theory and Research, 12, 107-158. New York:
Agathon.
Bucheri, C., Hampton, T., & Voelker, V. (eds.) (1991). The Student Body: Great Cartoons from the
Kappan. Phi Beta Kappa. Bloomington, IN.
56
References (cont.)
DesJardins, S. L., Kim, D. O, & Rzonca, C. S. (2002-2003). A nested analysis of factors affecting bachelor’s
degree completion. Journal College Student Retention, 4, 407-435.
Habley, W. & McClanahan, R. (2004). What Works in Student Retention – All Survey Colleges. ACT, Inc.
Iowa City, IA.
Harris, S. (1991) Can’t you guys read? Cartoons on Academia. Rutgers University Press. New Brunswick, NJ
Herzog, K. & Miller, M. P. (eds.) (1985). Scholarship: More Great Cartoons from the Kappan. Phi Beta Kappa.
Bloomington, IN.
Horn, L. & Nevill, S (2006). Profile of undergraduates in U.S. postsecondary education institutions: 20032004: With a special analysis of community college students (NCES 2006-184). U.S. Dept. of Education.
Washington, DC: National Center for Education Statistics.
Le, H., Casillas, A., Robbins, S., & Langley, R. (2005). Motivational and skills, social, and self-management
predictors of college outcomes: Constructing the Student Readiness Inventory. Educational and
Psychological Measurement, 65, 482-508.
Lotkowski, V., Robbins, S., & Noeth, R. (2004). The role of academic and non-academic factors in improving
college retention. ACT Policy Report. Iowa City, IA: ACT, Inc.
Pascarella, E. T., & Terenzini, P. T. (2005). How College Affects Students: A Third Decade of Research. San
Francisco: Jossey-Bass.
57
References (cont.)
Peterson, C. H., Casillas, A., & Robbins, S. B. (2006). The Student Readiness Inventory and the Big Five:
Examining social desirability and college academic performance. Personality and Individual Difference,
41, 663-673.
Porter, S.R. (2003-2004). Understanding Retention Outcomes: Using Multiple Data Sources to Distinguish
Between Dropouts, Stopouts, and Transfer-Outs. Journal of College Student Retention: Research,
Theory & Practice, 5(1), 53-70.
Robbins, S. B., Allen, J. Casillas, A., Akamigbo, A., Saltonstall, M., Cole, R., Mahoney, E. & Gore, P.A. (2009).
Associations of Resource and Service Utilization, Risk Level, and College Outcomes. Research in
Higher Education.
Robbins, S. Allen, J., Casillas, A., Peterson, C., & Le, H. (2006). Unraveling the differential effects of
motivational and skills, social, and self-management measures from traditional predictors of college
outcomes. Journal of Educational Psychology, 98, 598-616.
Robbins, S. B., Lauver, K., Le, H., David, D., Langley, R., & Carlstrom, A. (2004). Do psychosocial and study
skill factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130, 261-288.
Robbins, S., Oh, I., Button, C., & Le, H. (2009). The effects of college interventions on psychosocial mediators
and academic and persistence outcomes: An integrated meta-analysis. Manuscript submitted for
publication.
Swail, W. S. (2004, January 23). Legislation to improve graduation rates could have the opposite effect. The
Chronicle of Higher Education, B16.
58
What Works in Promoting
Student Success
Correspondence regarding this presentation
should be addressed to:
Steve Robbins
Vice President
Research
ACT, Inc.
Phone: 319-337-1227
steve.robbins@act.org
59