Venessa A. Keesler, Ph.D.
Bureau of Assessment and Accountability
Michigan Department of Education
Presentation to MASFPS Fall Directors’ Institute
October 4, 2012
Things I’ve learned about accountability
since taking this job
#1: NO ONE (and I do mean no one) likes
accountability.
#2: If you are the one who is holding people
accountable, you are not going to be the most
popular person in the room
#3: See point number 1
So why do we do accountability?
Three myths; one reality
• Myth #1: To drive reform
• Myth #2: To create education policy
• Myth #3: Because we are gluttons for punishment
Reality:
• Accountability metrics/systems are quantitative articulations of
the core policy beliefs of the education system
• They help us measure our progress in meeting those core
policy goals
• They are the measure, not the purpose or the goal
Accountability Landscape: 2012
A new era of accountability
• Switching from a purely criterion-based system to
a normative system
• Criterion-based systems: Set average proficiency
targets for schools.
• Normative system: identifies the “worst” or
“best” or “lowest” or “highest”
Why the change?
Policy imperative for NCLB: all students CAN and
SHOULD demonstrate proficiency  criterionsystem with proficiency targets for all schools and
subgroups
10 years later: our average achievement is
increasing, but we still have students and schools
lagging behind
New policy imperative (ESEA Flex): we must
target our lowest performing schools AND our
lowest performing students more specifically and
strategically
The problem with average proficiency
rates at a school or district level
“Never trust an average”
Averages mask low performance
Example:
• Proficiency target is 50%
• Sunshine School has 20 students, 10 of whom are
proficient and 10 of whom are not.
• Sunshine School meets it’s target; hooray!
• BUT  those 10 not proficient students are possibly left
behind
Michigan’s Accountability System
Top to Bottom Ranking
• Priority Schools (bottom 5%)
• Focus Schools (largest achievement gaps)
• Reward Schools (high performing, high progress, beating
the odds)
Accountability Scorecard
• Proficiency targets for all schools
Moving Beyond the Label
We believe:
• Accountability only measures the core policy
beliefs
• The data in the accountability metrics is useful
and necessary for schools and districts to
understand their performance
• Understanding where you are as a school/district
is the first step toward moving forward
• Working smarter, not just harder
Using the Top to Bottom Ranking
Three main components by subject:
• Achievement
• Improvement in achievement over time
• The largest achievement gap between two subgroups
calculated based on the top scoring 30% of students
versus the bottom scoring 30% of students
Each component tells schools something about their
overall performance and can be used for diagnostics
Who receives a ranking?
Schools with 30+ full academic year (FAY)
students over the last two years in at least
two state-tested content areas; school
must be OPEN at time of list generation
Application  Some schools do not
receive a ranking if they:
Have too few FAY students
Only have one year of data
Tested Grades and Subjects
Reading and Mathematics: Grades 3-8 and 11
• In grades 3-8, testing every year allows us to figure
out student performance level change (our current
“growth” metric) in reading and math
• Students can either significantly improve, improve,
maintain, decline or significantly decline
Writing: Grades 4 & 7
Science: Grades 5 & 8
Social Studies: Grades 6 & 9
What about Reconfigured schools?
A school must change by four or more grades in order
to get a new code
• Example: A K-2 building becoming a K-6 building.
• New codes NOT granted when a school is reopened as a
charter, for example
If not, the school retains the old code and continues
to have data “point” at it from all students for whom
that code is their feeder school
There is no “phase reset” like there was in AYP
• If school population changed by 51%, could request a phase
reset—still got AYP calculations, but sanctions delayed
• Under Priority/Focus interventions, would simply have a
customized intervention.
Quick
Reference
for Z-Scores
Why do We Use Z Scores?
Z-scores are a standardized measure that helps you compare
individual student (or school) data to the state average data
(average scores across populations).
Z-scores allow us to “level the playing field” across grade levels
and subjects
Each Z-score corresponds to a value in a normal distribution. A ZScore will describe how much a value deviates from the mean.
What do you need to know: Z-scores are used throughout the
ranking to compare a school’s value on a certain component to the
average value across all schools.
What is a Z-Score?
Z-scores are centered around zero
Positive numbers mean the student or school is
above the state average
Negative numbers mean the student or school is
below the state average
…Worse than state average
-3
-2
-1
State
Average
0
Better than state average….
1
2
3
Z-Score Examples
Your school has a z-score of 1.5. You are better than the
state average.
Z-score of 1.5
…Worse than state average
-3
-2
-1
State
Average
0
Better than state average….
1
2
3
Z-Score Examples
Your school has a z-score of .2. You are better than
the state average, but not by a lot.
Z-score of 0.2
…Worse than state average
-3
-2
-1
State
Average
0
Z-score of 1.5
Better than state average….
1
2
3
Z-Score Examples
Your school has a z-score of -2.0. You are very far
below state average.
Z-score of -2.0
Z-score of 0.2
…Worse than state average
-3
-2
-1
State
Average
0
Z-score of 1.5
Better than state average….
1
2
3
How Is the Top to Bottom Ranking
Calculated
Two-Year Average
Standardized Student
Scale (Z) Score
School Achievement
Z-Score
Improvement Metric
(Performance Level
Change OR Four Year
Improvement Slope)
School Improvement
Z-Score
1/4
Two-Year Average
Bottom 30% - Top
30%
Z-Score Gap
School Achievement
Gap Z-Score
1/4
1/2
School
Content
Area Index
Content
Index Zscore
How Is the Top to Bottom Ranking
IMPORTANT PART RIGHT
Calculated
HERE!!!
Two-Year Average
Standardized Student
Scale (Z) Score
School Achievement
Z-Score
Improvement Metric
(Performance Level
Change OR Four Year
Improvement Slope)
School Improvement
Z-Score
1/4
Two-Year Average
Bottom 30% - Top
30%
Z-Score Gap
School Achievement
Gap Z-Score
1/4
1/2
School
Content
Area Index
Content
Index Zscore
How do we get Standardized Scale
Scores for Each Student?
Step #1: Take each student’s score on the test they took
and compare that score to the statewide average for
students who took that same test in the same grade and
year
This creates a student-level z-score for each student in
each content area
• Compare
•
•
•
•
MEAP to MEAP
MEAP-Access to MEAP-Access
MME to MME
MI-Access
• Participation to Participation
• Supported Independence to Supported Independence
• Functional Independence to Functional Independence
What do we do with those standardized
scores?
Step #2: Once each student has a z-score for each
content area (based on the test they took), we take
all of the students in a each school, and rank order
the students within the school.
• Z-scores will have come from different tests, and compare
students to statewide average for that grade, test, and subject
• But they can now be combined for the school
Step #3: Add up all z-scores and take the average.
This is now the average standardized student scale
score.
Step #4: Define the top and bottom 30% subgroups,
based on that rank ordering.
Student
Test Taken
Z-score
Tommy
Mi-Access, Participation
2.5
Sally
MEAP
2.0
Maura
MI-Access, SI
1.9
Fred
MEAP
1.5
Ichabod
MEAP-Access
1.0
Freud
MEAP
0.8
Maybelle
MI-Access, FI
0.7
Destiny
MEAP
0.5
Harold
MEAP
-0.2
Bickford
MI-Access, FI
-0.5
Talledaga
MEAP-Access
-0.7
Francine
MEAP
-1.2
Joey
MEAP
-1.9
William
MEAP
-2.2
Student
Test Taken
Z-score
Tommy
Mi-Access, Participation
2.5
Sally
MEAP
2.0
Maura
MI-Access, SI
1.9
Fred
MEAP
1.5
Ichabod
MEAP-Access
1.0
Destiny
Average Z-score
(average
MEAP
standardized student scale
MI-Access, FI
score):
0.28
MEAP divide by 15)
(sum all z-scores,
Harold
MEAP
-0.2
Bickford
MI-Access, FI
-0.5
Talledaga
MEAP-Access
-0.7
Francine
MEAP
-1.2
Joey
MEAP
-1.9
William
MEAP
-2.2
Freud
Maybelle
0.8
0.7
0.5
Student
Test Taken
Z-score
Tommy
Mi-Access, Participation
2.5
Sally
MEAP
2.0
Maura
MI-Access, SI
1.9
Fred
MEAP
Ichabod
MEAP-Access
1.0
Freud
MEAP
0.8
Maybelle
MI-Access, FI
0.7
Destiny
MEAP
0.5
Harold
MEAP
-0.2
Bickford
-0.5
Talledaga
MI-Access, FI
Bottom 30%
MEAP-Access
Francine
MEAP
-1.2
Joey
MEAP
-1.9
William
MEAP
-2.2
Top 30%
1.5
-0.7
What is important to show Schools?
Two-Year Average
Standardized Student
Scale (Z) Score
School Achievement
Z-Score
Two-Year Average
Performance Level
Change Index
School Performance
Level Change
Z-Score
1/2
1/4
School
Content
Area Index
Step #1: Achievement
How well did the school do in that subject?
Two-Year Average
Positive number = better than average
Bottom 30% - Top
School Achievement
30%
Near
= average1/4
Gapzero
Z-Score
Z-Score Gap
Negative number = worse than average
Content
Index Zscore
Step #2: Improvement
Is the school improving
in that
subject?
What is important
to show
Schools?
Positive number = greater rate of improvement
thanmathematics
average
For grade 3-8 reading and
Near zero = average improvement
Negative = slower rate of improvement than
Two-Year Average
School Achievement
Standardized Student
1/2 are declining
average; canZ-Score
also mean they
Scale (Z) Score
Improvement Score
School Improvement
Z-Score
1/4
Two-Year Average
Bottom 30% - Top
30%
Z-Score Gap
School Achievement
Gap Z-Score
1/4
School
Content
Area Index
Content
Index Zscore
Step #3: Achievement Gap
gap in that subject
between
top 30%
What isIs the
important
to show
Schools?
and bottom 30%:
For grade (positive
3-8 reading
and mathematics
number)
= smaller gap than
average
Two-Year Average
(negative
number) = larger gap than
School Achievement
Standardized Student
1/2
Z-Score
average
Scale (Z) Score
(near zero) = average gap
Improvement Value
School Performance
Level Change
Z-Score
1/4
Two-Year Average
Bottom 30% - Top
30%
Z-Score Gap
School Achievement
Gap Z-Score
1/4
School
Content
Area Index
Content
Index Zscore
Once they have looked at each
component, Discuss:
What’s the overall pattern?
• Low achievement?
• Declining achievement?
• Large gaps?
Where are the actionable areas?
• Which subjects need the most attention?
• Is everyone doing poorly (small gap, low achievement) or
are some students doing well and others falling behind
(decent achievement, but large gap)
What is important to show Schools?
For grade 3-8 reading and mathematics
Two-Year Average
Standardized Student
Scale (Z) Score
School Achievement
Z-Score
Two-Year Average
Performance Level
Change Index
School Performance
Level Change
Z-Score
1/2
1/4
Reward Schools
(for improvement)
School
Content
Area Index
Content
Index Zscore
Focus Schools
Two-Year Average
Bottom 30% - Top
30%
Z-Score Gap
School Achievement
Gap Z-Score
1/4
An example from data:
My daughter’s elementary school
K-4 building
Go here to get this tool:
• www.mi.gov/ttb
Final Point
Accountability data does not tell ALL
“Correlation does not imply causation, but it does
waggle its eyebrows suggestively and gesture
furtively while mouthing ‘look over there’.”
Moving beyond the label only takes capacity at
every level of the field
Resources
www.mi.gov/ttb
• Individual school lookup tool
• Diagnostic Worksheet
www.mi.gov/priorityschools
www.mi.gov/focusschools
www.mi.gov/rewardschools
Contact Information
Venessa A. Keesler
keeslerv@michigan.gov (personal email)
mde-accountability@michigan.gov
877-560-8378, option 6