Harnessing the Power of
Alternative Assessment to Advance
the Completion Agenda
Eloy Ortiz Oakley
Superintendent-President, Long Beach City College
Terrence Willett
Director of Planning and Research, Cabrillo College/
RP Group Researcher
Patrick Perry
Vice Chancellor, CCCCO
Strengthening Student Success Conference Ocotber
9, 2013
1
Transition to College:
Assessment and Placement
• CCCs are open enrollment
• Requires assessing and planning for
students’ educational needs.
• Most colleges rely on standardized
assessment
• Most students placed into basic skills
• Placement into basic skills is
significant barrier to completion
2
LBCC’s Research
•
•
•
Examined five cohorts of more than 7,000
LBUSD grads who attend LBCC directly
after high school
Determined strongest predictors of
success in college courses
Improved alignment of assessment and
placement with evidence based
predictors of success
3
Key Findings
1. Standardized tests predict standardized
tests very well but…
2. Classroom performance predicts
classroom performance
3. More information tells us more about our
students’ likelihood to succeed
Findings presented real opportunities to
significantly improve student placement and
achievement, and their college experience
4
Reimagining the Transition to
College: Fall 2012 Promise Pathways
• Predictive placement model using multimethod, evidence-based assessment
• Prescriptive, full-time course load via firstsemester success plan
– Emphasis on Foundational Skills
– English, Reading, Math
5
Alternative placement:
Transfer-level Placement Rates
70%
60%
60%
50%
40%
31%
30%
20%
11% 13%
7%
10%
9%
0%
Transfer Level English
F11 First time students
F11 LBUSD
Transfer Level Math
F12 Promise Pathways
6
F2012 Promise Pathways 1-year vs. F2006
6-year rates of achievement
45%
41.4%
40%
35.9%
35%
31.2%
30%
27.3%
25%
20%
15%
15.0%
9.9%
10%
5%
11.9%
12.9%
5.1%
0%
Successfully Completed Successfully Completed
Transfer Math
Transfer English
F2011 LBUSD (N=1660)
F2012 Promise Pathways (N=976)
Behavioral Intent to
Transfer
F2006 LBUSD 6 Year Rate
7
Success rates in transfer-level courses
Fall 2012
70%
64%
62%
60%
55%
51%
50%
40%
30%
20%
10%
0%
Transfer Level English
Transfer Level Math
Non-Pathways (n=2095 English, 1803 Math)
Promise Pathways (n=516 English, 156 Math)
Neither of these differences approach significance, p >.30
8
Equity Impact of Promise Pathways:
F2011 Baseline Equity Gaps
70%
60%
50%
40%
36%
29%31%
30%
22%
21%
22%
20%
11%
10%
1%
10%
4%
11% 11%
7%
18%
11%
5%
0%
Successfully
Successfully
Behavioral Intent to
Completed Transfer Completed Transfer
Transfer
Math
English
F2011 Black
F2011 Asian
F2011 Hispanic
Full-time, full year
F2011 White
Equity Impact: F2012 Students of Color
Compared to F2011 White Baseline
70%
65%
62%
60%
53%
50%
44%
40%
40%
28%
30%
20%
10%
22%
39%
33%
28%
22%
31%
15%13%
10%
6%
0%
Successfully
Successfully
Behavioral Intent to
Completed Transfer Completed Transfer
Transfer
Math
English
F2012 Black
F2012 Asian
F2012 Hispanic
Full-time, full year
**F2011 White
Equity gaps remain but significant gains for all
F2012 Students
70%
57%
60%
52%
50%
44%
40%
40%
30%
20%
10%
29%
28%
65%
62%60%
53%
39%
33%
28%
15%13%
6%
0%
Successfully
Successfully
Behavioral Intent to
Completed Transfer Completed Transfer
Transfer
Math
English
F2012 Black
F2012 Asian
F2012 Hispanic
Full-time, full year
**F2011 White
Cohort 1 and 2 Promise
Pathways Demographics
70%
66%
60%
53%
50%
40%
30%
20%
10%
15%
10%
16%
12%
15%
10%
2% 2%
0%
African American Asian/PI/Filipino
F2012 (N=976)
Hispanic
White
F2013 (N=1321)
Unknown
A BRIEF OVERVIEW
The Student TranscriptEnhanced Placement
Project (STEPS)
Harnessing the Power of Alternative Assessment to Advance the Completion Agenda
13
What Are We Seeking to Do?
•
•
•
•
Examine the value of using transcripts as part of
the assessment process
Create predictive models using intersegmental
data to study students who had already taken
community college courses
Use models to analyze how well transcript data
predicts the first English & math courses students
take and how well they do in them
Recruit colleges to do local analyses to generate
campus-specific insights and trigger CCC/K-12
conversations
Harnessing the Power of Alternative Assessment to Advance the Completion Agenda
14
In English
tests
predict tests,
grades and courses
matter but
vary by college
15
Count of Colleges
Showing Significance
Predicting College English Level
11
10
9
8
7
6
5
4
3
2
1
0
Weakest
Intermediate
Strongest
CST's
A-G
Courses
HS
Course
Level
HS
Course
Grade
HS GPA*
Predictor Variable Category
Cox & Snell pseudo R-square ~ 0.35
Harnessing the Power of Alternative Assessment to Advance the Completion Agenda
16
In math
tests
predict tests,
but high school
level is also
important
17
Count of Colleges
Showing Significance
Predicting College Math Level
11
10
9
8
7
6
5
4
3
2
1
0
Weakest
Intermediate
Strongest
CST's
A-G
Courses
HS
Course
Level
HS
Course
Grade
HS GPA*
Predictor Variable Category
Cox & Snell pseudo R-square ~ 0.50
Harnessing the Power of Alternative Assessment to Advance the Completion Agenda
18
In English,
grades
predict
grades
19
Count of Colleges
Showing Significance
Predicting College English Success
11
10
9
8
7
6
5
4
3
2
1
0
Weakest
Intermediate
Strongest
CST's
A-G
HS
HS
HS College
Courses Course Course GPA* Course
Level Grade
Level
Predictor Variable Category
Cox & Snell pseudo R-square ~ 0.20
Harnessing the Power of Alternative Assessment to Advance the Completion Agenda
20
In math,
success
predictors
vary by
college
21
Count of Colleges
Showing Significance
Predicting College Math Success
11
10
9
8
7
6
5
4
3
2
1
0
Weakest
Intermediate
Strongest
CST's
A-G
HS
HS
HS College
Courses Course Course GPA* Course
Level Grade
Level
Predictor Variable Category
Cox & Snell pseudo R-square ~ 0.20
Harnessing the Power of Alternative Assessment to Advance the Completion Agenda
22
What Does This Mean?
• There is a disconnect between test scores
and performance in the first course taken
at many institutions
• Using transcript elements would help to
refine assessment
• The relative weight of variables might be
influenced by local factors, due to
considerations such as articulation and
curriculum
Harnessing the Power of Alternative Assessment to Advance the Completion Agenda
23
What Happens Next?
•
•
•
•
Updating the statewide analysis
Encouraging more colleges to replicate the study
so they understand the value of transcript data for
their own students
Starting conversations about how the study relates
to other aspects of developmental education
reform—such as alignment, course offerings, and
curriculum
Helping to develop a tool that enables colleges to
access high school data to inform multiple
measures assessment
Harnessing the Power of Alternative Assessment to Advance the Completion Agenda
24
Chancellor’s Office Multiple
Measure Warehouse Project
Patrick Perry
Vice Chancellor of Technology,
Research & Information Systems
Common Assessment Project
• AB743 passed in 2011 (AB743, Block):
– BOG shall establish a common assessment system
to be used as one of multiple measures for
placement and advisement
– Ongoing funding provided in 2013-14 budget for
project
– RFA to move $$ to district currently underway
Common Assessment Project
• Project intends to:
– Identify a common, systemwide assessment for
math, English, ESL
– Provide the assessment at no cost to districts for
unlimited usage
– Provide for score portability statewide
– Create a multiple measures data warehouse to
enhance placement activities
Multiple Measures Warehouse
• Will include:
– Outcomes of the common assessments
– CCC statewide MIS Enrollment data
– EAP scores
– CAHSEE scores
– All HS math and English enrollment records, GPA’s
• (likely through 11th grade)
Multiple Measures Warehouse
• Application will be written allowing a student
“placement profile” to be viewed by student
services personnel
• “big data” predictive analytics applied to
multiple measures variables to assist in
placement effectiveness
Contact Information
• Eloy Ortiz Oakley
• eoakley@lbcc.edu
• Terrence Willett
• terrence@cabrillo.edu
• Patrick Perry
• pperry@cccco.edu
30
Additional Resources
• More information about Long Beach City College's research
– http://bit.ly/PathwaysResearch
• More information about the RP Group’s Student TranscriptEnhanced Placement (STEPS) Project
– http://bit.ly/RPSTEPS
• Step by Step process for replicated the research and
placement:
– http://bit.ly/RPSTEPS2
• Jobs for the Future Report: Where to Begin?: The Evolving Role
of Placement Exams for Students Starting College
– http://bit.ly/JFFReport
• CCRC research on Assessment, Placement, and Progression in
Developmental Education
– http://bit.ly/CCRCAssess
31