Presentation at the Fall 2011 Meeting of the
Michigan Educational Research Association


Identifying MME Cut Scores
2
University
2-Year Institution
2-Year Institution
Central Michigan University
Alpena Community College
Mid Michigan Community College
Eastern Michigan University
Delta College
Monroe County Community College
Ferris State University
Glen Oaks Community College
Montcalm Community College
Grand Valley State University
Gogebic Community College
Mott Community College
Michigan Technological University
Grand Rapids Community College
Muskegon Community College
Michigan State University
Henry Ford Community College
North Central Michigan College
Oakland University
Jackson Community College
Northwestern Community College
Northern Michigan University
Kalamazoo Valley Community College
Oakland Community College
Saginaw Valley State University
Kellogg Community College
Schoolcraft College
The University of Michigan-Ann Arbor
Kirtland Community College
Southwestern Michigan College
University of Michigan-Dearborn
Lake Michigan College
St. Clair County Community College
University of Michigan-Flint
Lansing Community College
Washtenaw Community College
Wayne State University
Macomb Community College
West Shore Community College
Western Michigan University
3
MME content area College courses used
Mathematics
College Algebra.
Courses identified by 4-year universities.
Reading
Reading-heavy courses such as entry-level literature, history,
philosophy, or psychology for 2-year universities.
Courses identified by 4-year universities.
Science
Entry level biology, chemistry, physics, or geology for 2-year
universities.
Courses identified by 4-year universities.
Social Studies
Entry level history, geography, or economics for 2-year universities.
4







Grades were put on a numeric scale from 0-4
0=F
1=D
2=C
3=B
4=A
Not used
o AU, AWF, DR, R, RA, FR, T, TR, X

Coded as 3.0
o P, CR

Coded as 0.0
o IN, N, NC, NE, NS, W, WF, WP, WX, and U
5
Course Grade
MME Score
MME
Content
Area
Sample
Size
Percent B
or higher
Mean
SD
Mean
SD
Math
6,286
47.0
2.49
1.18
1112.2
13.2
Reading
37,952
54.9
2.64
1.23
1117.2
24.6
Social
Studies
39,721
54.4
2.63
1.22
1135.4
26.3
Science
15,608
50.0
2.54
1.19
1123.5
23.5
6
MME Content Area
Course Type
Number of Students
Mathematics
College Algebra
6567
Literature
456
American History
1731
Other History
3010
Psychology
16231
Sociology
8236
Political Science
6114
Philosophy
1869
Other
2517
Reading
7
MME Subject Area
Science
Social Studies
Course Type
Number of Students
Biology/Life Science
8355
General Chemistry
5807
Physics
535
Other
1483
American History
1734
Other History
3006
Psychology
16230
Sociology
8231
Geography
612
Political Science
6108
Economics
3498
Other
2361
8
 Students
receiving an A
 Students receiving a B or better
 Students receiving a C or better
 Students
receiving a B or better in 4-year
universities
 Students receiving a B or better in 2-year
institutions
9

Logistic Regression (LR)
o Identify score that gives a 50% probability of achieving an A
o Identify score that gives a 50% probability of achieving a B or better
o Identify score that gives a 50% probability of achieving a C or better

Signal Detection Theory (SDT)
o Identify scores that maximize the proportion receiving consistent
classifications from MME to college grades
• i.e., both proficient/advanced and receiving a A/B/C or better
• i.e., both not proficient/partially proficient and receiving a A-/B-/C- or worse
o Equivalent to LR under mild monotonicity assumptions

Selected SDT as the preferred method because of its purpose
(maximizing consistent classification)
10
Where
• success is obtaining an A/B/C or better
• e is the base of the natural logarithm
• β0 is the intercept of the logistic regression
• β1 is the slope of the logistic regression
• x is the MME score
11
Percent of Students Earning a B or Better
Logistic Regression of Test Scores on College Grades
(Using Simulated Data)
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
950
1000
1050
1100
Test Score
1150
1200
1250
12
Percent of Students Earning a B or Better
Logistic Regression of Test Scores on College Grades
(Using Simulated Data)
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
950
1000
1050
1100
Test Score
1150
1200
1250
13
Percent of Students Earning a B or Better
Logistic Regression of Test Scores on College Grades
(Using Simulated Data)
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
950
1000
1050
1100
Test Score
1150
1200
1250
14
Percent of Students Earning a B or Better
Logistic Regression of Test Scores on College Grades
(Using Simulated Data)
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
950
1000
1050
1100
Test Score
1150
1200
1250
15

Freshman Grade (known cut score)
MME
(unknown cut score)
B- or Lower
B or Higher
College Ready
Inconsistent
Consistent
Not College Ready
Consistent
Inconsistent
Basic Idea
 Set the MME cut score to…
 Maximize the number of students in the Consistent cells
 Minimize the number of students in the Inconsistent cells
 Maximize consistent classification from MME to first-year college grades
16
MME Content Area Score
0
Known Cut Score
B- or lower
0.2
0.4
0.6
B or higher1
0.8
1.2
Grade in First Related Freshman Credit Bearing Course
17
MME Content Area Score
82.8%
consistent classification
0
Consistently Classified
Inconsistently Classified
Known Cut Score
Unknown Cut Score
B- or lower
0.2
0.4
0.6
B or higher1
0.8
1.2
Grade in First Related Freshman Credit Bearing Course
18
82.8%
consistent classification
MME Content Area Score
Adjust the unknown cut
score to maximize
consistent classification
0
Consistently Classified
Inconsistently Classified
Known Cut Score
Unknown Cut Score
B- or lower
0.2
0.4
0.6
B or higher1
0.8
1.2
Grade in First Related Freshman Credit Bearing Course
19
MME Content Area Score
88.6%
consistent classification
0
Consistently Classified
Inconsistently Classified
Known Cut Score
Unknown Cut Score
B- or lower
0.2
0.4
0.6
B or higher1
0.8
1.2
Grade in First Related Freshman Credit Bearing Course
20
MME Content Area Score
90.45%
consistent classification
0
Consistently Classified
Inconsistently Classified
Known Cut Score
Unknown Cut Score
B- or lower
0.2
0.4
0.6
B or higher1
0.8
1.2
Grade in First Related Freshman Credit Bearing Course
21







Analyses treating grades of A as the success criterion produced
unusable results (i.e., the highest possible MME scale scores
Analyses treating grades of C as the success criterion produced
unusable results (i.e., MME scale scores below chance level)
Analyses treating 4-year and 2-year institutions did produce different
cut scores, but they were within measurement error of each other
Used analyses based on all institutions and grades of B or better to
produce MME cut scores
Used probability of success of 33% and 67% to set the “partially
proficient” and “advanced” cut scores
SDT and LR produced very similar results
Used SDT because it was the preferred methodology
22
Content Area
Classification
Consistency
Partially
Proficient Cut
Score
Proficient Cut
Score
Advanced Cut
Score
Mathematics
65%
1093
1116
1138
Reading
63%
1081
1108
1141
Science
67%
1106
1126
1144
Social Studies
63%
1097
1129
1158
23


Identifying MEAP Cut Scores
24
Grade
Cohort
3
4
5
6
7
8
9
10
11
12
13
1
-
-
-
-
-
05-06
06-07
07-08
08-09
09-10
10-11
2
-
-
-
-
05-06
06-07
07-08
08-09
09-10
10-11
-
3
-
-
-
05-06
06-07
07-08
08-09
09-10
10-11
-
-
4
-
-
05-06
06-07
07-08
08-09
09-10
10-11
-
-
-
5
-
05-06
06-07
07-08
08-09
09-10
10-11
-
-
-
-
6
05-06
06-07
07-08
08-09
09-10
10-11
-
-
-
-
-
7
06-07
07-08
08-09
09-10
10-11
-
-
-
-
-
-
8
07-08
08-09
09-10
10-11
-
-
-
-
-
-
-
9
08-09
09-10
10-11
-
-
-
-
-
-
-
-
10
09-10
10-11
-
-
-
-
-
-
-
-
-
25

Logistic Regression (LR)
o Identify score that gives a 50% probability of achieving proficiency on a later-
grade test (i.e., MME or MEAP)

Signal Detection Theory (SDT)
o Identify scores that maximize the proportion receiving consistent classifications
from one grade to a later grade
• i.e., proficient/advanced on both tests
• i.e., not proficient/partially proficient on both tests
o Equivalent to LR under mild monotonicity assumptions

Equipercentile Cohort Matching (ECM)
o Identify scores that give the same percentage of students proficient/advanced
on both tests


Selected SDT as the preferred method because of its purpose
(maximizing consistent classification)
However, SDT and LR are susceptible to regression away from the mean
26


Same as for identifying MME cut scores
Criterion for success is proficiency on a later grade test rather
than getting a B or better in a related college course
27
Signal Detection Method (Simulated Data)
Grade 8 Score
Each dot represents a plot of
test scores in grade 8 and grade
11 for a single student
Known Cut Score
Unknown Cut Score
Grade 11 Score
28
Signal Detection Method (Simulated Data)
Grade 11: Proficient
Grade 8 Score
Grade 11: Not proficient
Known Cut Score
Unknown Cut Score
Grade 11 Score
29
Signal Detection Method (Simulated Data)
Grade 8: Proficient
Grade 11: Proficient
Grade 8 Score
Grade 8: Proficient
Grade 11: Not proficient
Known Cut Score
Unknown Cut Score
Grade 8: Not proficient
Grade 11: Not proficient
Grade 8: Not Proficient
Grade 11: Proficient
Grade 11 Score
30
Signal Detection Method (Simulated Data)
Grade 8 Score
Known cut score
stays where it is
Known Cut Score
Unknown Cut Score
Grade 11 Score
31
Grade 8 Score
Signal Detection Method (Simulated Data)
Known Cut Score
Unknown Cut Score
Move the unknown cut score up
or down to maximize the same
classifications in both grades
Grade 11 Score
32


The more links in the chain, the greater the effects of regression
Original plan for Math and Reading
o Link grade 11 MME to college grades
o Link grade 8 MEAP to grade 11 MME
o Link grade 7 MEAP to grade 8 MEAP
o Link grade 6 MEAP to grade 7 MEAP
o Link grade 5 MEAP to grade 6 MEAP
o Link grade 4 MEAP to grade 5 MEAP
o Link grade 3 MEAP to grade 4 MEAP


Original plan results in 7 links by the time the grade 3 cut is set
Original plan results in inflated cut scores in lower grades
33


Revised plan for Math and Reading
For Grade 3, 4, 5, 6
o Link grade 11 MME to college grades
o Link grade 7 MEAP to grade 11 MME
o Link grade 3, 4, 5, or 6 MEAP to grade 7 MME

For Grade 7, 8
o Link grade 11 MME to college grades
o Link grade 7 or 8 MEAP to grade 11 MME

Results in a maximum of three links for any one grade
34

No evidence of regression away from the mean in identifying
MEAP “proficient” cut scores
o Looking for a consistently lower percentage of students proficient as one
goes down in grades
o Used SDT to identify MEAP “proficient” cut scores

Evidence of regression away from the mean in identifying MEAP
“partially proficient” and “advanced” cut scores
o Increasingly smaller percentages of students in the “Not proficient” and
“Advanced” categories as one goes down in grade
o Used ECM instead to identify MEAP “Not Proficient” and “Advanced” cut
scores
35

No evidence of regression away from the mean in identifying
MEAP “proficient” cut scores
o Looking for a consistently lower percentage of students proficient as one
goes down in grades
o Used SDT to identify MEAP “proficient” cut scores

Evidence of regression away from the mean in identifying MEAP
“partially proficient” and “advanced” cut scores
o Increasingly smaller percentages of students in the “Not proficient” and
“Advanced” categories as one goes down in grade
o Used ECM instead to identify MEAP “Not Proficient” and “Advanced” cut
scores
36

Classification Consistency Rates for MEAP Cut Scores in
Mathematics
Grade
Cut Score
Partially Proficient
Proficient
Advanced
8
83%
86%
95%
7
81%
84%
95%
6
82%
83%
96%
5
81%
84%
95%
4
80%
82%
94%
3
77%
80%
95%
37

Classification Consistency Rates for MEAP Cut Scores in Reading
Grade
Cut Score
Partially Proficient
Proficient
Advanced
8
83%
78%
87%
7
86%
76%
85%
6
85%
74%
83%
5
88%
75%
84%
4
80%
82%
94%
3
80%
72%
86%
38

Classification Consistency Rates for MEAP Cut Scores in Science
Grade
Cut Score
Partially Proficient
Proficient
Advanced
8
80%
84%
92%
5
76%
82%
92%
39

Classification Consistency Rates for MEAP Cut Scores in Science
Grade
Cut Score
Partially Proficient
Proficient
Advanced
9
85%
81%
91%
6
81%
77%
91%
40


Creating Mini-Cuts for PLC Calculations in Reading and Mathematics
41
100
90
Conditional Standard Error of Measurement
80
70
60
50
40
30
20
10
0
205
255
305
Grade 3 Mathematics Scale Score
355
405
42
Conditional Standard Error of Measurement
70
60
50
205
255
305
Grade 3 Mathematics Scale Score
355
Advanced (A)
Proficient (P)
Partially Proficient (PP)
Not Proficient (NP)
100
90
80
40
30
20
10
0
405
43
100
90
Conditional Standard Error of Measurement
80
70
60
50
40
30
20
10
0
205
255
305
Grade 3 Mathematics Scale Score
355
405
44
Conditional Standard Error of Measurement
70
60
205
255
305
Grade 3 Mathematics Scale Score
355
A-Mid
P-High
P-Mid
P-Low
PP-High
PP-Low
NP-High
NP-Mid
NP-Low
100
90
80
50
40
30
20
10
0
405
45
Year X Grade Y
MEAP
Performance
Level
Low
Not
Mid
Proficient
High
Partially Low
Proficient High
Low
Proficient Mid
High
Advanced Mid
Year X+1 Grade Y+1 MEAP Performance Level
Not
Partially
Proficient
Proficient
Proficient
Adv
Low Mid High Low High Low Mid High Mid
M
I
I
SI
SI
SI SI
SI
SI
D M
I
I
SI
SI SI
SI
SI
D
D
M
I
I
SI SI
SI
SI
SD D
D
M
I
I
SI
SI
SI
SD SD
D
D
M
I
I
SI
SI
SD SD SD D
D
M
I
I
SI
SD SD SD SD D
D M
I
I
SD SD SD SD SD D
D
M
I
SD SD SD SD SD SD D
D
M
46


New Versus Old Cut Scores
47
Percent Proficient
Mathematics, Grade 11
100
90
80
70
60
50
40
30
20
10
0
Old Cut Scores
New Cut Scores
07-08
08-09
09-10
10-11
School Year
48
Percent Proficient
Mathematics, Grade 8
100
90
80
70
60
50
40
30
20
10
0
Old Cut Scores
New Cut Scores
07-08
08-09
09-10
10-11
School Year
49
Percent Proficient
Mathematics, Grade 7
100
90
80
70
60
50
40
30
20
10
0
Old Cut Scores
New Cut Scores
07-08
08-09
09-10
10-11
School Year
50
Percent Proficient
Mathematics, Grade 6
100
90
80
70
60
50
40
30
20
10
0
Old Cut Scores
New Cut Scores
07-08
08-09
09-10
10-11
School Year
51
Percent Proficient
Mathematics, Grade 5
100
90
80
70
60
50
40
30
20
10
0
Old Cut Scores
New Cut Scores
07-08
08-09
09-10
10-11
School Year
52
Percent Proficient
Mathematics, Grade 4
100
90
80
70
60
50
40
30
20
10
0
Old Cut Scores
New Cut Scores
07-08
08-09
09-10
10-11
School Year
53
Percent Proficient
Mathematics, Grade 3
100
90
80
70
60
50
40
30
20
10
0
Old Cut Scores
New Cut Scores
07-08
08-09
09-10
10-11
School Year
54


New Versus Old Cut Scores
55
Percent Proficient
Reading, Grade 11
100
90
80
70
60
50
40
30
20
10
0
Old Cut Scores
New Cut Scores
07-08
08-09
09-10
10-11
School Year
56
Percent Proficient
Reading, Grade 8
100
90
80
70
60
50
40
30
20
10
0
Old Cut Scores
New Cut Scores
07-08
08-09
09-10
10-11
School Year
57
Percent Proficient
Reading, Grade 7
100
90
80
70
60
50
40
30
20
10
0
Old Cut Scores
New Cut Scores
07-08
08-09
09-10
10-11
School Year
58
Percent Proficient
Reading, Grade 6
100
90
80
70
60
50
40
30
20
10
0
Old Cut Scores
New Cut Scores
07-08
08-09
09-10
10-11
School Year
59
Percent Proficient
Reading, Grade 5
100
90
80
70
60
50
40
30
20
10
0
Old Cut Scores
New Cut Scores
07-08
08-09
09-10
10-11
School Year
60
Percent Proficient
Reading, Grade 4
100
90
80
70
60
50
40
30
20
10
0
Old Cut Scores
New Cut Scores
07-08
08-09
09-10
10-11
School Year
61
Percent Proficient
Reading, Grade 3
100
90
80
70
60
50
40
30
20
10
0
Old Cut Scores
New Cut Scores
07-08
08-09
09-10
10-11
School Year
62


New Versus Old Cut Scores
63
Percent Proficient
Science, Grade 11
100
90
80
70
60
50
40
30
20
10
0
Old Cut Scores
New Cut Scores
07-08
08-09
09-10
10-11
School Year
64
Percent Proficient
Science, Grade 8
100
90
80
70
60
50
40
30
20
10
0
Old Cut Scores
New Cut Scores
07-08
08-09
09-10
10-11
School Year
65
Percent Proficient
Science, Grade 5
100
90
80
70
60
50
40
30
20
10
0
Old Cut Scores
New Cut Scores
07-08
08-09
09-10
10-11
School Year
66


New Versus Old Cut Scores
67
Percent Proficient
Social Studies, Grade 11
100
90
80
70
60
50
40
30
20
10
0
Old Cut Scores
New Cut Scores
07-08
08-09
09-10
School Year
10-11
68
Percent Proficient
Social Studies, Grade 9
100
90
80
70
60
50
40
30
20
10
0
Old Cut Scores
New Cut Scores
07-08
08-09
09-10
10-11
School Year
69
Percent Proficient
Social Studies, Grade 6
100
90
80
70
60
50
40
30
20
10
0
Old Cut Scores
New Cut Scores
07-08
08-09
09-10
10-11
School Year
70
Joseph
A. Martineau
o Executive Director
o Bureau of Assessment & Accountability
o Michigan Department of Education
o martineauj@michigan.gov
o 517-241-4710
71