English Assessment Validity
February 2012
Michael Orkin, Jo Ann Phillips, Hui Zhang, Sheryl Queen
Office of Institutional Research
Peralta Community College District
1

English Assessment Test
• New Peralta students take assessment tests for placement in English classes.
• Course placement from test scores is recommended but not mandatory.
• A student can take courses without placement tests if prerequisites are met.
2

• Peralta currently uses the ACT Compass test,
CENG, for English placement.
• CENG tests English proficiency with separate writing and reading tests.
• This will soon change...
3

• From Student Success Task Force, January,
2012:
“… all students will have access to common online assessment tools for English, mathematics and ESL and to pre-testing programs that help improve assessment outcomes. They also will be able to take the results of their tests to any community college in the state.”
4

For now: ACT Compass Test
Placement in English Courses
A decision rule based on the two test scores selects an English course:
• transfer level (English 1A)
• degree applicable (English 201A)
• Pre-collegiate (Basic Skills).
5

• If the writing score is less than 22, the student is not assessed into a course and must see a counselor.
• 699 students in the sample data had low writing scores and were excluded.
6

• Students taking CENG-recommended English courses in Fall 2008, Spring 2009, Fall 2009,
Spring 2010.
• Took test immediately preceding semester when they took English course.
• Never took test prior to semester when they took English course.
7

Validity measured in three ways:
• Relationship between assessment scores and grades.
• Comparison of assessed and non-assessed.
• Comparison of students who took the “right” course (recommended by CENG) and
“wrong” course (not recommended by
CENG).
8

Courses Count %
Eng 1A 1571 57%
Eng 201A
Basic Skills*
TOTAL
593 21%
598 22%
2762 100%
* English 252AB, 267AB, 268AB, 269AB
9

A B C D F W
Eng 1A 28% 23% 12% 5% 8% 25%
Eng 201A 21% 19% 15% 9% 9% 27%
Basic Skills 23% 22% 16% 4% 10% 24%
ALL 25% 22% 13% 6% 9% 25%
10

• 57% were placed in English 1A.
• 47% overall got A’s or B’s.
• 25% overall withdrew from class.
11

Are assessment test scores positively correlated with grades?
We use ANOVA (Analysis of Variance).
12

Reading Test Scores by Grade
Courses A - Test ave B - Test ave C - Test ave D - Test ave F/W- Test ave
ANOVA
Sig.
Eng 1A 91.61
87.33
86.98
87.01
87.88
0.000
Eng 201A 84.17
81.35
80.36
83.75
81.12
0.000
Basic Skills 61.84
58.92
57.99
60.08
57.29
0.036
13

All Test Scores by Grade
Courses
Eng 1A
Eng 201A
ANOVA
Sig. Reading
ANOVA
Sig. Writing
ANOVA
Sig. Combined
0.000
0.000
0.000
0.000
0.727
0.727
Basic Skills 0.036
0.170
0.481
14

• Reading test scores most sensitive to grades.
• Students who get A’s have higher average test scores than those who have lower grades.
• Writing test and Combined Tests only significant for English 1A.
15

As test scores get higher, does predicted chance of success increase?
We use logistic regression.
16

Reading Test Predicted Chance of Success
17

Reading Test Predicted Chance of Success
18

Reading Test Predicted Chance of Success
19

As test scores get higher, predicted chance of success increases but …
• Only for reading test scores.
• Different courses are selected across range of scores so much is filled in by software.
• Results aren’t statistically significant.
20

Do assessed students do better in their assigned courses than students who don’t take the assessment test?
We use ANOVA and Chi Square tests.
21

Courses Assessed Not Assessed
Eng 1A
Eng 201A
Basic Skills*
TOTAL
1571
593
598
2762
* English 252AB, 267AB, 268AB, 269AB
1838
621
727
3186
22

Not Assessed - Grade %
(Similar to Assessed)
A B C D F W
Eng 1A 25% 25% 15% 5% 6% 24%
Eng 201A 18% 18% 19% 9% 11% 25%
Basic Skills 21% 18% 13% 6% 10% 31%
ALL 23% 22% 16% 6% 8% 25%
23

Assessed vs. Not Assessed GPA
Courses
Eng 1A
Eng 201A
Basic Skills
TOTAL
Assessed
Course GPA
Not Assessed
Course GPA
ANOVA
Sig.
2.76
2.74
0.730
2.47
2.58
2.61
2.32
2.51
2.66
0.099
0.426
0.212
24

No significant results:
• Students who are assessed have similar class GPA as those who are not assessed.
25

Assessed vs. Not Assessed Success
Courses
Assessed % Not Assessed %
Success Success
Chi-Square
Sig.
Eng 1A 62% 65% 0.112
Eng 201A 55% 56% 0.795
Basic Skills 61% 53% 0.004
TOTAL 60% 60% 0.984
26

Success - Assessed vs. Not Assessed
Significant only at basic skills level:
• Students who are assessed have greater success than those who are not assessed.
27

Assessed – “Right” vs. “Wrong” Course
Do assessed students who take the right course do better than assessed students who take the wrong course?
We use Chi Square and ANOVA.
28

Counts - Right vs. Wrong Course
Courses
Eng 1A
Eng 201A
Basic Skills
TOTAL
Right Course
Count
Wrong Course
Count
% Right
Course
1571 378 81%
593
598
2762
479
127
984
55%
82%
74%
29

Success - Right vs. Wrong
Courses
Eng 1A
Eng 201A
Basic Skills
Right Course
Success %
62%
55%
61%
Wrong Course
Success %
53%
Chi Square
Sig.
0.001
52%
61%
0.364
0.932
30

Success - Right vs. Wrong
Significant only for English 1A:
• Assessed students who take right course have greater success than those who take wrong course.
31

GPA - Right vs. Wrong
Courses
Eng 1A
Eng 201A
Basic Skills
Right
Course GPA
2.76
Wrong
Course GPA
ANOVA
Sig.
2.39
0.000
2.47
2.58
2.34
2.58
0.178
0.986
32

GPA - Right vs. Wrong
Significant only for English 1A:
• Assessed students who take right course have higher GPA than those who take wrong course.
33

• Validity confirmed.
• Useful for counselors.
• Assessed results aren’t much different from non-assessed, pre-requisite method.
• Most students take English 1A.
• Students assessed into English 1A do significantly better than “wrong” students.
34