Running head: ENGLISH CTEP CUT-SCORE VALIDATION
English CTEP Cut-Score Validation
Rick Fillman
Institutional Research Analyst
Planning and Research Office
Summer 2013
ENGLISH CTEP CUT-SCORE VALIDATION Introduction
Validation of placement cut-scores is periodically required. The first part of
this report will examine the responses from a recent survey of instructors
and students regarding the appropriateness of their placement into courses
in the English core sequence. It is intended to address the appropriateness
of the academic placement and validate the cut-scores currently used for
English course placement in the Cabrillo Assessment Center. The second
part is an analysis of how the placements impact various demographic
groups.
Consequential Validity
In order to assess the accuracy and validity of placement, students and
instructors from randomly selected sections of ENGL-255 Basic English,
ENGL-100 Elements of Writing, and ENGL-1A College Composition were
surveyed early in the Spring 2012 semester as to the appropriateness of
their placement. Students indicated whether their skills and knowledge
caused them to feel they were under-qualified, qualified, or over-qualified
for the course material. Likewise, instructors in these sections rated each
student as to the appropriateness of their placement.
Assessing the consequences of placement in this fashion is known as
Consequential Validity. Students and instructors must be surveyed near the
beginning of the semester, usually in the fourth or fifth week. It is expected
that at this point students have been sufficiently exposed to the curriculum
to gauge their own preparedness, but not so far into the course as to have
had a chance to master the material. Similarly, it is expected that, at this
point, instructors have a reasonable understanding of their students’
abilities.
English placement uses the sum of the scores from the CTEP Reading
Comprehension test and the CTEP Sentence Structure & Grammar test plus
multiple measures points. The placement cut-scores are outlined on the
Cabrillo Assessment office webpage “Math and English Assessment Scoring
Criteria” (see Appendix A).
The assessment process for English places a student into one of three levels:
ENGL-255 Basic English
ENGL-100 Elements of Writing
ENGL-1A College Composition (transfer level)
ENGLISH CTEP CUT-SCORE VALIDATION Student and Instructor Surveys
The chart that follows shows how many sections were targeted, the total
section enrollment, and the survey yield from each.
Secti ons Surveyed
ENGL‐255
ENGL‐100
ENGL‐1A
Enrol l ed
9
14
24
Excl uded
181
367
699
85
235
496
Ins tructor ra ti ngs i ncl uded
95
129
187
Student ra ti ngs i ncl uded
80
112
158
Enrollment records were matched with Assessment data, and only students
whose enrollment follows a recent Cabrillo placement were included in the
analysis. The “Excluded” column represents students who qualified for
enrollment in the course by way of some other means: self-assessment1,
promotion (successful completion of the prerequisite course), or via an
equivalency from an external exam or from a course taken at another
institution. Generally, instructors were able to rate all students on their
rosters. Student self-ratings were obtained from the students in attendance
on the day of the survey. The larger number of exclusions among students
in the higher level courses represents the fact that a smaller proportion of
students in these courses qualify via a placement; a much larger number
qualify by way of satisfactory completion of the prerequisite course.
Instructor and student surveys were conducted in electronic format for all
online classes. The remaining surveys were conducted in class using paper
surveys. Samples of each of the survey instruments are presented in
Appendix B.
Although students were questioned as to the method of their placement,
these responses were not used. (The prompt was included in case ID
matching with student system assessment results were to become
problematic.)
Agreement
A threshold of 75% agreement is required for validity2, otherwise, the
English department faculty should consider an adjustment in the cut-scores.
At least 75% of students surveyed need to perceive themselves as properly
1
Self‐assessment would apply to the ENGL‐255 (Basic English) level only, as students have the option to start with the lowest level course in the sequence without testing. 2
California Community Colleges Chancellor’s Office (CCCCO), Standards, Policies and Procedures for the Evaluation of Assessment Instruments Used in the California Community Colleges, 4th EDITION, REVISED MARCH 2001. http://extranet.cccco.edu/Portals/1/SSSP/Matriculation/Assessment/ApprovedGuidelinesMarch2001.pdf ENGLISH CTEP CUT-SCORE VALIDATION placed. Similarly, and independently, faculty must rate at least 75% of
students as appropriately placed. Their respective ratings as to the
appropriateness of student placement are shown in the following table:
Rated as appropriately placed
Ins tructor
ENGL‐255
ENGL‐100
ENGL‐1A
Student
86.3%
86.8%
87.7%
91.3%
90.2%
82.9%
Instructors find, after rating students individually, that 86.3% percent of
students who placed into ENGL-255 were placed appropriately. When
students in the same sections rated their own preparedness for the course,
91% of those who placed into ENGL-255 agree that the placement was
appropriate. The student and instructor agreement regarding the
appropriateness of their placement into English sections meets the
placement validity threshold for all three placement levels.
The standard error (the standard deviation of the sample means) is a gauge
of accuracy – the smaller the number, the more likely the sample is an
accurate representation of the larger population. The standard error for the
validation surveys is shown below:
Standard Error
Ins tructor
ENGL‐255
ENGL‐100
ENGL‐1A
Student
0.038
0.032
0.026
0.024
0.018
0.018
Those whose placement were not rated as appropriate may be perceived as
either over or under-qualified for the course material. The following table
completes the dataset, showing all of the student and instructor
preparedness ratings for students who placed into the course.
Instructor/Student Ratings
Student
Instructor
Student is struggling
ENGL‐255
ENGL‐100
ENGL‐1A
5.3%
7.0%
5.3%
Student should be I a m not I a m I a m ove r‐
Student is placed qua l i fi ed qua l i fi ed qua l i fi e d
fine
higher
86.3%
86.8%
87.7%
8.4%
6.2%
7.0%
5.0%
3.6%
4.4%
91.3%
90.2%
82.9%
3.8%
6.3%
12.7%
ENGLISH CTEP CUT-SCORE VALIDATION Finally, when pairs of responses are examined, it is found that students and
instructors are generally in agreement with each other. When a pairedsamples t-Test is performed, it reveals no statistically significant differences
between the pairs of instructor and student ratings applied to each student.
Disproportionate Impact
This section of the report examines English course placement through the
equity lens. If all groups were equal, then the proportion of group members
who place into college level English (ENGL-1A) will match that of the
population tested. The extent to which these proportions are not in
alignment is known as disproportionate impact. Specific attention is
warranted when disproportionality exceeds the “80/20 rule”. Per EEOC
guidelines, any group that falls below 80% of the majority group's
placement rate has been disproportionately impacted3. Disproportionate
impact exists when the proportion of some group falls below 80% of the
“dominant group’s” score.
Ethnicity example:
The dominant or majority group is Whites.
49% of White students place into ENGL-1A
80% of 49% = 39.3%
So the expectation is that the rate for each of the other groups should come
to at least this level; groups whose ENGL-1A placement fall below this rate
are impacted.
As demographic data is generally available for all students, this analysis will
be performed for the entire population taking the English placement test
during the testing period - about 2500 persons. Placement results by
ethnicity, age, gender, and disability status are examined.
Disproportionate impact by ethnicity is evident, and could be found among
students identifying as African American, as Asian, as Latino, as Native
American, and as Non-respondents. Clearly, the Latino group is the largest,
with 1167 students, placing into ENGL-1A at an average rate 19.2
percentage points behind that of Whites. African Americans, with only 28
students testing, placed into ENGL-1A at a rate 17.9 percentage points
behind the majority group. Native Americans, with only 15 students tested,
place into ENGL-1A at a rate 12.6 percentage points behind the majority.
3
The 80% rule is rooted in labor law regarding testing in hiring practices. It was codified in 1978 in the Uniform Guidelines on Employee Selection Procedures document issued by the EEOC ‐ the U.S. Equal Employment Opportunity Commission, Department of Labor, and Department of Justice. Cabrillo College adopted the criteria in the 2004 Student Equity Plan: http://www.cabrillo.edu/services/pro/reports/docs/CabrilloSEP2004v4.pdf ENGLISH CTEP CUT-SCORE VALIDATION The non-response group, numbering 61 students, placed into ENGL-1A at a
rate 18.0 percentage points behind the majority.
No age group was found to be disproportionately impacted. Each age
group’s placement rate was within 80% of the majority 18-20 year old
group’s placement rate. Similarly, for gender, although females slightly outplace males, the rate of placement into ENGL-1A by males lags by only a few
percentage points.
Students who have received services from the Disabled Students Program
and Services (DSPS) are categorized by their type of disability. While
impact can be identified for several of the disability groups, only the “Other
Disability” group has sufficient numbers to be able to draw conclusions4.
However, as the rate for this group is only 0.8 percentage points behind that
of the non-disabled dominant group, the impact is negligible. Appendix C
reveals detailed data regarding the impacts on the respective demographic
groups.
Summary
Cut-scores for English course placement currently in use in the Assessment
Center at Cabrillo College are placing students into courses in a manner such
that at least 75% of both student and instructors, early in the semester, will
agree as to the appropriateness of their placement. Placement into the
English curriculum disproportionately impacts a large number of Latino
students, who are placed into remedial sections out-of-proportion to their
share of the population. Students identifying as African-American, Asian,
and Native American are also similarly impacted, although in much smaller
numbers.
The scope of this study is limited to the validation of the current cut-scores
as used for English course placement. The question of the overall efficacy of
the Assessment Placement system is a larger question.
Acknowledgements
The data collection effort for the cut-score validation surveys is substantial.
Coordinating surveys of instructors and students in some 47 separate
sections of English involved countless emails and phone calls, not to mention
lots of leg work and careful handling and recording of thousands of individual
survey data. For all this work, Judy Cassada, Institutional Research
Specialist, deserves a big thank you.
4
A group must have a sufficient number of members – about 10 or so – in order to be considered. ENGLISH CTEP CUT-SCORE VALIDATION Appendix A
[Extracted from Cabrillo Assessment Office Web page:
http://www.cabrillo.edu/services/assessment/scoring_criteria.html]
______________________________________________________________________________
CTEP ENGLISH --- two-part assessment places you into an English class.
Part 1: Reading Comprehension


7 reading selections
35 questions/30 minutes
Part 2: Sentence Structure & Grammar

30 questions/20 minutes
Your English
Score
Places you into:
0-17
Recommends you take ESL assessment or Eng 255
18-36
Eng 255 - Basic English
37-49
Eng 100 - Elements of Writing
50-65
Eng 1A - College Composition, Transfer Level
Appendix B – Survey materials Cabrillo College Test Validation Survey Please complete this survey even if you have filled it out for another class. This is a State requirement for Cabrillo’s accreditation. Your responses go directly to Cabrillo’s Planning & Research Office where they are kept strictly confidential. Thank you! Name: ____________________________________________________ Student I.D. Number_____________________
1. How did you qualify for this class?
 Cabrillo assessment placement
 Assessment placement from another school
 Completed prerequisite course
 Successfully challenged the prerequisite
 Other (please specify): _______________________
2. To what degree do you have the appropriate skills and/or knowledge to take this class?
 I am overqualified
 I am qualified
 I am not completely qualified
3. To what extent are personal reasons (illness, job/family responsibilities, personal problems, etc.)
interfering with your performance in this class?
 Not at all
 Somewhat
 Very much
Printed on 100% post consumer waste recycled fiber. Cabrillo College Test Validation Survey, English and Reading 2012
1. Cabrillo College Assessment Test Survey 2012
This is a State requirement for Cabrillo’s accreditation. Your responses go directly to Cabrillo’s Planning & Research Office where they are kept strictly confidential. Thank you! 1. For which course are you taking this survey?
6 *2. Please write down your name, class and section numbers, and
your student ID number in the spaces provided. Please complete the
survey even if you have filled it out for another class. Thank you!
Name:
Section #:
Student ID:
3. How did you qualify for this class?
j Cabrillo assessment placement
k
l
m
n
j Assessment placement from another school
k
l
m
n
j Completed prerequisite course
k
l
m
n
j Successfully challenged the prerequisite
k
l
m
n
Other (please specify) 4. To what degree do you have the appropriate skills and/or knowledge to take this class?
j I am not completely qualified.
k
l
m
n
j I am qualified.
k
l
m
n
j I am overqualified.
k
l
m
n
5. To what extent are personal reasons (illness, job/family responsibilities, personal
problems, etc.) interfering with your performance in this class?
j Not at all
k
l
m
n
j Somewhat
k
l
m
n
j Very much
k
l
m
n
Friday, February 24, 2012
Dear English and Reading Instructors,
As you are aware, all placement assessments must be validated on a six year cycle. This validation study is of
the cut scores for the assessment/placement tests we currently use from the Chancellor’s list of approved
instruments.
To accomplish this current process, please distribute, then collect, the enclosed student survey in this
randomly selected section. Administering this short survey should take about 10 minutes of class time, and
asks students if they believe they have been placed in the correct level. This is consequential validation, and is
one of the quickest and most commonly used forms of cut score validation, required by the California
Community College’s Chancellor’s Office. The results will be tallied by Cabrillo’s Planning & Research Office
(PRO) and reported shortly after final grades are reported.
Before distributing the survey forms please read or paraphrase the following statement:
“I am about to give you a quick survey that will ask you how you feel about the difficulty level of
this class. You will need to fill in your name, student I.D. number and class section number on
the form [give them the class section number from the roster]. Then you will circle the statement
on the form that best describes how you feel about this class.
This survey will be used to help validate our English [or Reading] placement exams so that
when a student takes the placement test the placement score accurately reflects the skills needed
in different class levels. It is an important part of how students are placed in the correct class
and helps prevent students from enrolling in a class that is too easy or too difficult. Your
participation is appreciated.”
As faculty, your part in the survey is to complete the enclosed roster, indicating how you match each student’s
skills and the level of challenge presented by the course. To do this, place one of the following signs next to
each student’s name on the provided roster.
Mark
Meaning
+ (a plus sign)
Student is not adequately challenged and should be placed higher
= (an equal sign)
Student is fine
–
Student is struggling and should be placed lower
(a minus sign)
Please Return these Materials no later than Friday, March 9:
Please return the completed student surveys, and your completed roster to either the BELA Division
Office, or send via campus mail to: Craig Hayward’s mail box, or Planning & Research Office (PRO)
mail box. Please contact Judy Cassada at 477-5577, or email jucassad@cabrillo.edu with any questions.
Thank you very much for participating in the study.
Margery Regalado, Dean
Counseling & Educational Support Services
Jim Weckler, Dean
BELA
Renée M. Kilmer
Vice President, Instruction
Craig Hayward, Director
Planning, Research & Knowledge Systems
Appendix C
Disproportionate Impact by Ethnicity
Non‐
Placement African Responden
Hispanic/La Multiple/Ot Native ts
White
American
Asian
tino
her
American
Levels
ENGL‐255
8
24
379
12
3
24
103
ENGL‐100
14
27
553
61
8
24
425
ENGL‐1A
6
18
235
71
4
13
510
28
69
1167
144
15
61
1038
Non‐
Placement African Responden
Hispanic/La Multiple/Ot Native ts
White
American
Asian
tino
her
American
Levels
ENGL‐255
28.6% 34.8%
32.5%
8.3%
20.0%
39.3%
9.9%
ENGL‐100
50.0% 39.1%
47.4%
42.4%
53.3%
39.3%
40.9%
ENGL‐1A
21.4% 26.1%
20.1%
49.3%
26.7%
21.3%
49.1%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
ENGL‐1A
`
Majority = white
80% of majority =
Below 39.3%?
Disproportionate Impact?
EEOC Gap
Below
Below
49.1%
39.3%
Below
‐
Hispanic/La Multiple/Ot
tino
her
Below
Below
‐
Native American
Non‐
Responden
ts
White
African American
Asian
yes
yes
yes
‐
yes
yes
‐
‐17.9%
‐13.2%
‐19.2%
‐
‐12.6%
‐18.0%
‐
553
1112
857
2522
Disproportionate Impact by Age
Placement 17 & under 18 ‐ 20 21 ‐ 25
26 ‐ 30
31 ‐ 40
Levels
ENGL‐255
5
347
102
31
ENGL‐100
7
769
166
80
ENGL‐1A
14
584
119
50
26
1700
387
41 ‐ 50
40
48
53
141
161
60+
51 ‐ 60
20
16
18
54
5
25
18
48
3
1
1
5
Placement 17 & under 18 ‐ 20 21 ‐ 25
26 ‐ 30
31 ‐ 40
41 ‐ 50
51 ‐ 60
60+
Levels
ENGL‐255
19.2% 20.4%
26.4%
19.3%
28.4%
37.0%
10.4%
60.0%
ENGL‐100
26.9% 45.2%
42.9%
49.7%
34.0%
29.6%
52.1%
20.0%
ENGL‐1A
53.8% 34.4%
100.0%
100.0%
30.7%
31.1%
37.6%
33.3%
37.5%
20.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
‐
‐
‐
‐
‐
21 ‐ 25
26 ‐ 30
31 ‐ 40
41 ‐ 50
51 ‐ 60
‐
‐
‐
‐
‐
‐
‐
‐
‐
‐
ENGL‐1A
`
Majority = 18‐20
80% of majority =
Below 27.5%?
‐
‐
Age 17 & under 18 ‐ 20
Disproportionate ‐
‐
Impact?
EEOC Gap
‐
‐
34.4%
27.5%
Below
60+
*
‐
553
1112
857
2522
Disproportionate Impact by Gender
Placement F
Levels
M
257
541
448
1246
ENGL‐255
ENGL‐100
ENGL‐1A
Placement F
Levels
ENGL‐255
ENGL‐100
ENGL‐1A
`
291
569
406
1266
M
20.6%
43.4%
36.0%
23.0%
44.9%
32.1%
100.0%
100.0%
Majority = Female
80% of majority =
Below 28.8%?
548
1110
854
2512
‐
‐
Gender
Disproportionate Impact?
F
M
‐
‐
EEOC Gap
‐
‐
36.0%
28.8%
Disproportionate Impact by Disability
Developme
Acquired ntally Placement Brain Delayed Hearing Learner
Impaired
Not Disabled Injury
Levels
ENGL‐255
ENGL‐100
ENGL‐1A
629
1188
883
2700
Placement Not Disabled
Levels
ENGL‐255
23.3%
ENGL‐100
44.0%
ENGL‐1A
2
1
3
100.0%
Majority = NO Disability
80% of majority =
Below 26.2%?
‐
Disability
Not Disabled
2
‐
Mobility Impaired
11
13
9
33
1
3
5
Acquired Developmen
Brain tally Delayed Hearing Injury
Learner
Impaired
66.7%
0.0%
32.7% 33.3%
100.0%
4
1
Learning Disabled
Learning Disabled
Other Disability
2
6
2
10
Mobility Impaired
Psychological Disability
21
26
16
63
Other Disability
5
5
6
16
Psychological Disability
80.0%
20.0%
0.0%
66.7%
0.0%
33.3%
33.3%
39.4%
27.3%
20.0%
60.0%
20.0%
33.3%
41.3%
25.4%
31.3%
31.3%
37.5%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
‐
‐
Below
Below
‐
Hearing Impaired
Learning Disabled
Mobility Impaired
Other Disability
Psychological Disability
32.7%
26.2%
Below
Acquired Developmen
Brain tally Delayed Injury
Learner
Disproportionate Impact?
‐
‐
*
‐
‐
*
yes
EEOC Gap
‐
‐
‐
‐
‐
‐
‐0.8%
‐
‐
676
1239
918
2833