Multiple Measures Assessment
Project
Welcome Webinar
Spring 2014
Agenda
• Multiple Measures and Common Assessment
• Overview of analysis
• Student Transcript Enhanced Placement Study
(STEPS) findings leading to Multiple Measures
Project (MMAP)
• Working with K-12 partners
• Compass or Accuplacer data
• Role of the pilot colleges
• Projected timeline
• Questions and Answers
Multiple Measures and Common Assessment
Initiative
• Role of the Common Assessment Advisory
Group
Overview of Analysis
• Examine the value of using transcripts as part of
the assessment process
• Create predictive models using intersegmental
data to study students who had already taken
community college courses
• Use models to analyze how well transcript data
predicts the first English & math courses students
take and how well they do in them
• Recruit colleges to do local analyses to generate
campus-specific insights and trigger CCC/K-12
conversations
4
Harnessing the Power of Alternative Assessment
to Advance the Completion Agenda
Student Transcript Enhanced
Placement Study (STEPS) –
Statewide Findings
In English tests
predict tests,
grades and courses matter
but vary by college
6
Count of Colleges
Showing Significance
Predicting College English Level
11
10
9
8
7
6
5
4
3
2
1
0
Weakest
Intermediate
Strongest
CST's
A-G
Courses
HS
Course
Level
HS
Course
Grade
HS GPA*
Predictor Variable Category
Cox & Snell pseudo R-square ~ 0.35
7
Harnessing the Power of Alternative Assessment
to Advance the Completion Agenda
In math tests
predict tests,
but high school level is
also important
8
Count of Colleges
Showing Significance
Predicting College Math Level
11
10
9
8
7
6
5
4
3
2
1
0
Weakest
Intermediate
Strongest
CST's
A-G
Courses
HS
Course
Level
HS
Course
Grade
HS GPA*
Predictor Variable Category
Cox & Snell pseudo R-square ~ 0.50
9
Harnessing the Power of Alternative Assessment
to Advance the Completion Agenda
In English,
grades predict
grades
10
Count of Colleges
Showing Significance
Predicting College English Success
11
10
9
8
7
6
5
4
3
2
1
0
Weakest
Intermediate
Strongest
CST's
A-G
HS
HS
HS College
Courses Course Course GPA* Course
Level Grade
Level
Predictor Variable Category
Cox & Snell pseudo R-square ~ 0.20
11 to Advance the Completion Agenda
Harnessing the Power of Alternative Assessment
In math,
success
predictors vary
by college
12
Count of Colleges
Showing Significance
Predicting College Math Success
11
10
9
8
7
6
5
4
3
2
1
0
Weakest
Intermediate
Strongest
CST's
A-G
HS
HS
HS College
Courses Course Course GPA* Course
Level Grade
Level
Predictor Variable Category
Cox & Snell pseudo R-square ~ 0.20
13 to Advance the Completion Agenda
Harnessing the Power of Alternative Assessment
What Does This Mean?
• Using high school transcript data could help
inform alignment/articulation and refine
assessment policies
• The relative weight of variables might be
influenced by local factors, due to
considerations such as articulation and
curriculum
• Barriers to implementation must be addressed
such as processing transcripts
14Assessment to Advance the Completion Agenda
Harnessing the Power of Alternative
What Happens Next?
• Updating the statewide analysis
• Encouraging more colleges to replicate the study
so they understand the value of transcript data for
their own students
• Starting conversations about how the study
relates to other aspects of developmental
education reform—such as alignment, course
offerings, and curriculum
• Helping to develop a tool that enables colleges to
access high school data to inform multiple
measures assessment
15 to Advance the Completion Agenda
Harnessing the Power of Alternative Assessment
Working with K-12 partners to upload data
• Key aspect of analysis using high school transcript data is
getting local K-12 data into Cal-PASS Plus
• Cal-PASS Plus can assist in outreach, though initially coming
from a college helps
• Cal-PASS Plus and RP Group can help provide materials and
templates
• Entire process to submit data is streamlined – drag and
drop files they already create for the CA Dept. of Education
Compass or Accuplacer data
- Does your college use Accuplacer or Compass
for student placements?
- If not, we ask that you submit you college’s data
to Cal-PASS Plus to include in the analysis.
Projected timeline
Jan – Mar: Build data warehouse for application to multiple
measures analysis – collect various data sources
May – July: Complete research on system wide multiple
measures models and algorithms
July – Sept: Develop and deploy online analytic tools and
decision tree models
Oct – Dec: Provide PD and direct support to pilot colleges to test
multiple measures tools and application; develop and test local
user interface to re-identify data for placement decisions
Dec – Ongoing: Provide cohort tracking data to pilot colleges on
course outcomes; integrate multiple measures project with
Common Assessment Initiative
Role of the pilot colleges
• Test interactive online placement tools
• Provide feedback on user experience
• Compare student placements using MMAP analysis
to your college’s current system
– How are students placing when comparing the two?
– What new conversations have occurred about improving
the placement process?
• Work with Cal-PASS Plus to track cohort outcomes
after applying multiple measures through the project
Questions?
Contacts:
STEPS Report and Summary
Mallory Newell
http://www.rpgroup.org/projects/steps
The RP Group
newellmallory@fhda.edu MMAP Project Site
http://www.rpgroup.org/projects/multi
Ken Sorey,
ple-measures-assessment-project
Cal-PASS Plus
ken@calpass.org
Cal-PASS Plus
https://www.calpassplus.org/
Amanda Avallone
Cal-PASS Plus
amanda@calpass.org