The Common Assessment Initiative &
Multiple Measures Assessment Project
March 15, 2016
Impact OF
Assessment/ Placement/ Preparatio
n O.rLQ_mplectiqd
n th e sea1e o tthe
ons er
A ;;,:
1
PJII.OrusiO A.l 01:\'Tl.Opt,IT 'T
•
LEAR NING
( I Illi ""
;,.to P<o10'-"1'1'C:Oll.A801Vt.J IO'
ID11"-'P.UlU'TY
STUO["''T CP...ITRfD
lltA!'IOILII. - "_,
STUDENT SUCCESS=
Scorecard
Starting
Cohort
N = 1941050
underprepared population
&
Prepared
Students
Complete
N=49,560
N=34,791
Unprepared
Students
70.2%
c)
N = 144,490
Note: the State Chancellor's Office defines unprepared as any completion oriented stud
t whose first course in math or Eng
40.5%
Complete
N = 58,518
h was below transfer level.
uities and the Effective Practices to Address Them. Presentation to the 2015 Strengthening
What kinds of Impact Does
Assessment/ Placement/ Preparation Have?
=r··. iit:.;,
EOUTTY C OM P U TI O>I
H.Oilis.IO .U Dl\ U-OfMll"l
:.:.t:'im' =
..
-
LEARNING
rruoim s Jcci:::::
A big part of our equity challenge is lurking in the
p Iacement of st udents
4 Levels
Below
3 Levels
Below
2 Levels
Below
1Level
Below
70%
What percent place
3 or 4 levels below?
Asian &
White
25%
Af American
& Hispanic
35%
55%
8%0 -------------- :>
uities and the Effective Practices to Address Them. Presentation to the 2015 Strengthening
Current Assessment and Placement
practices are engines of Inequity
ACIION
r:aOENCi o .,PR;-c;ze·
PATHWAYS Ul
EQUITY
'' [lt(,Rfii(IRII IC Ao
""""'*"
CR [ATER f OCUS
COMPLETION
PR.OtmiONA.LOEV£lOJ'\.1L:vT
Quantifying the contribution to inequity
in completion (preliminary findings)
Preliminary findings from one large California District
Service
Area
Population
¢
Enroll at
Community
College
i
Not much
inequity is
observed
through the
lens of our
traditionaI
access
measure.
Assess as
College
Ready
Credential
Seeking
Course Taking
l
i
50%-60%
The biggest
driver of
inequity in
outcomes
emerges in
our
placement
process
I
I
15%-25%
We need to
place some
sustained
attention on
pathways and
milestone
completion
¢
Sufficient
Academic
Performance
¢
Obtain
Completion
Credential
i
1
15%-25%
Our
traditional
approaches
just might be
widening the
gap.
uities and the Effective Practices to Address Them. Presentation to the 2015 Strengthening
Stephen Fletcher
Director of Assessment
DeAnza College



Placement Tests
Multiple Measures
Combination of tests and multiple measures


Bias, insensitivity, offensiveness
Validity:
◦ Is the instrument related to the prerequisite skills fir the
course(s)?
◦ Is the assessment result related to students’ success in a
course?

Reliability:
◦ Consistency of results over a defined period, usually two
weeks
◦ Consistency across raters
◦ Consistency across methods and/or testing stimuli
(items, forms, writing prompts, performance tasks)

Disproportionate Impact

Content
◦ Is the instrument related to the prerequisite skills
for success in the associated course(s)?

Criterion
◦ Is the assessment result related to students’
success in a course?

Consequential Validity
◦ Do students perceive themselves to be in the
appropriate class for their knowledge and skills?
◦ Do instructors perceive students as being in the
appropriate class?

Does the assessment have an differential effect
on different populations?
◦ What populations?
 Gender, race, ethnicity, linguistic and cultural group, age,
disability. Groups to start looking at are veterans and foster
youth
◦ What is a differential effect?
 Each group should perform within 80% of the best
performing group in the college.
 Each group you are considering should have at least 20
people.
◦ What if there is a differential effect?
 Describe what your college is going to do to help groups to
succeed.




Full: approved for seven years
Provisional: Lacks sufficient or recent
information to be given full approval; 1 year
then Probationary
Probationary: Missing critical information; 2
years
Not Approved: Instrument cannot be used for
placement
Josh Roberts English
Chair Sacramento City
College

Common assessment, not common
placement

Does not generate a “score” but instead a
student competency profile

Competency maps for English/Reading, ESL,
and Math

CAI English/Reading Competencies Overview

Reading:
◦
◦
◦
◦
◦
◦
◦
◦
◦
◦
Acquire and use vocabulary
Recognize Vocabulary appropriate to content
Follow instructions
Analyze ideas and events in text
Determine main and supporting ideas
Comprehend and summarize
Determine author’s purpose and stance
Analyze author’s stance
Evaluate texts
Analyze the structure of texts

Research and Response to Reading:
◦
◦
◦
◦
Analyze texts
Write arguments
Gather evidence
Paraphrase and quote

Vocabulary and Grammar:
◦ Demonstrate command of English grammar and
usage
◦ Demonstrate command of mechanics and
conventions
◦ Choose words effectively
◦ Determine meaning of words and phrases using
context
◦ Determine meaning of words using reference tools

Writing:
◦
◦
◦
◦
◦
◦
◦
◦
◦
Write informative or explanatory texts
Respond to a timed writing prompt
State a main idea
Support a main idea with details
Support a main idea with evidence
Organize writing
Write for specific purposes and audiences
Vary sentence structure
Edit writing
Reading
Reading Materials
Competency Number
Authentic text (print materials that
occur naturally in the lives of
learners outside of their
classrooms)
1
Demonstrate understanding of
words and phrases acquired
through conversations, reading
authentic texts, and being read to.
Acquire and Use Vocabulary
More complex authentic text (print
materials that occur naturally in
the lives of learners outside of their
classrooms)
2
Demonstrate understanding of
general academic language from
authentic business and mass
media sources.
Expository or academic fiction
or non-fiction text
3
Demonstrate understanding of
general academic and
technical words and phrases;
gather vocabulary knowledge
when considering a word or
phrase important to
comprehension or expression.
More complex expository or
academic fiction or non-fiction text
4
Demonstrate a nuanced
understanding of general academic,
technical, and domain- specific words
and phrases, sufficient for reading
and writing at the college and career
readiness level; demonstrate
independence in gathering
vocabulary knowledge when
considering a word or phrase
important to comprehension or
expression
Area
Competency
a
b
Weight
Score: Levels Below
1
2
3
4
Area Total
Weighted Total
Total Score
Total Weighted Score
Total Possible Range
Cut Scores
Student Placement
Competency
Research and Response to Reading
Reading
c
d
-1
2
2
3
e
f
1
2
2
3
22
28
g
h
4
2
2
3
i
1
j
2
a
4
b
4
c
d
2
4
3
2
13
19
69
84
43 - 127
ENGWR 300 = 43 - 63; ENGWR 101 = 64 - 84; ENGWR 51 = 85 - 105; No placement 106 - 127
ENGWR 101
Addtl. Measure
Vocabulary and Grammar
a
b
c
d
e
Writing
a
b
c
-3
2
3
3
15
12
3
4
2
3
3
d
e
2
4
2
1
19
25
f
g
h
i
2
2
2
2
GPA
HS ELA
2.65
C,B
2.65
2
http://bit.ly/MMAPAssess
John Hetts
Sr. Director of Data Science
Educational Results Partnership
Multiple Measures Overview
The use of umultiple measures" in assessment means:
o
o
o
o
the use of a more holistic, comprehensive, and ultimately accurate
measures of student capacity
set of
the standard by which any good assessment should be conducted
the legally required method of assessment in the California Community
Colleges
something that many if not most colleges fail to achieve in spirit (and in
some cases at all)
Appropriate use of evidence-based, multiple measures is one of
the key cornerstones on which to rebuild the foundations of
community college education
o
o
Demonstrates fundamental capacity of students to succeed if given the
chance
Powerful completion, equity, and real world implications
Basic
assessment/measurement theory:
◦ When you measure something you get:
 True score (thing you care about)
 Systematic error (regular error or bias in measurement)
Systematic error
FOR A F'Aift SE'-E CT I 0 N
EVERY800Y HAS TO TAKE
THE SAME EXAM: PLEASE
CLIMB THAT TREE
-. ..
-
---...:;,.=:-
- .. ,
-·
Basic
assessment/measurement theory:
◦ When you measure something you get:
 True score (thing you care about)
 Systematic error (regular error or bias in measurement)
 Random error (temporary errors)
◦ Important not to confuse precision (repeatability) with
accuracy (relation to true score)
Precision vs. accuracy
Low accuracy
High precision
High accuracy
Low precision
Basic
assessment/measurement theory:
◦ When you measure something you get:
 True score (thing you care about)
 Systematic error (regular error or bias in measurement)
 Random error (temporary errors)
◦ Important not to confuse precision (repeatability) with
accuracy (relation to true score)
Methodological
gold standard of assessment
◦ Triangulation to true score through assessment across
different:




methods of assessment (how)
context of assessment (who/where)
content domains (what)
time (when)
Also, they're required by law
Know your matriculation handbook, esp. Chap 2:
http:I /bit.ly/SSSPHandbook
Assessment is aholistic process through which each college
o
collects information about students to facilitate their success by
ensuring their appropriate placement into math, English, and ESL
curricula. Student assessments should reflect a variety of
informational sources that create a profile of a student's
academic strengths and weaknesses." p. 2.3
Colleges must adhere to the following regulations and guidelines
when implementing and managing any assessment instrument used
for course placement:
0
o
Course placement recommendations must be based on multiple measures
(sections 55502(i) and 55522(a)). Additional indicators of student readiness for
math, English, and ESL course content must be used together with placement test
suits. p. 2.4
Reality of current practice
CCs rely nearly entirely on standardized
assessment
o
Most CCCs meet letter of the law but majority fail to meet spirit
(WestEd, 2011) http:// bit.ly/WestEd2011
Majority of students placed below transfer-level
o
o
68% take at least one deved course (Scott-Clayton & Belfield,
201 5) bit.ly/CCRCPiacementAccuracy
75% (Scorecard) to 85% (2011 CCCCO BSI Accountability Report:
bit.ly/2011 BSI) of students in CCC
From a growing set of
examples
80%
80%
70%
60%
60%
60%
68%
58%
50%
40%
31%
40%
24%
30%
20%
14%
10%
9%
28%
20%
0%
0%
English
Accuplacer
Math
Accuplacer +MM
LBCC F2012 (N=~1000)
http://www.lbcc.edu/PromisePathways
English
Accuplacer
Math
Accuplacer + MM
SDCCD MMAP F2015
Pilot (N = ~1000)
http://bit.ly/MMAPPilot
90%
80%
70%
79%
80%
67%
60%
49% 49%
50%
40%
73%
70%
F12
F13
73%
79%
60%
40%
30%
20%
20%
10%
0%
0%
English
Non-Pathways
Math
Promise Pathways
Fall 2014 LBCC
http://bit.ly/MultipleMM
F14 HS
F14
Accuplacer
Data
F2014 Sierra College:
English
For Canada and Rio Hondo, see also
http://bit.ly/MMAPPilots
80%
60%
76%
59%
80%
65%
48%
60%
40%
40%
20%
20%
64%
57%
68%
59%
64%
57%
English
Math
Reading
0%
0%
English
Comparison
Math
HS Data
Davidson College 20132015
Rules used for English and Math: HSGPA >=2.6 and
completion of four years of mathematics
including one year beyond Algebra 2 in HS
Accuplacer
HS Data
Ivy Tech 2014-2015
Rules used for English and Math: HSGPA
>=2.6
70%
70%
60%
60%
50%
50%
40%
40%
34%
30%
21%18%
20%
10%
25% 24%
12%
51%
36%
39%
26%
21%
30%
20%
13%
64%
58%
12%
10%
4%
0%
0%
Math
Black
Hispanic
English
Asian
Fall 2011 LBUSD
White
Math
Black
Hispanic
English
Asian
White
F2012 Promise Pathways
• Multiple Measures Assessment Project Support
• http://bit.ly/MMAP2015
• http://bit.ly/MMAPGetStarted
• MMAP Project Team
On campus presentations
Webinars
In person convenings/Connection to peers
Tools and support for research methodology and data
analysis
• Provision of statewide model placement recommendations
and/or data for local, evidence-based model
• Outreach support for K-12 partnership improvement
•
•
•
•
Questions?
Ken Sorey
Educational Results Partnership/
Cal-PASS Plus
ken@edresults.org
Stephen Fletcher DeAnza
College
fletcherstephen@fhda.edu
Josh Roberts Sacramento
City College
robertj@scc.losrios.edu
John Hetts Educational Results
Partnership
jhetts@edresults.org