MNPS Academic Performance
Framework 2011 - 2013
Paul Changas, Alan Coverstone and Christine Stenson
Metropolitan Nashville Public Schools (MNPS)
October 29, 2013
The MNPS Framework
Presentation Overview
• Why the need for a Framework and how it will be used?
• Our process in building a Framework
• Components of our Framework – indicators and measures
• Next steps
• Lessons learned that might be helpful to other districts
• A framework template for other districts/schools
2
The MNPS Framework
Possible Framework Uses
• To support efforts to raise student achievement
• To support the district’s accountability status
• To offer standardized accountability metrics to
complement increased school-level autonomy
• To inform — but not determine — decisions regarding
rewards, supports, and resource allocation for schools
• To provide school communities with a transparent set of
indicators to understand school performance
3
Why the Need for a Framework?
(Why Not Rely on State Accountability Results?)
• State absolute accountability system is primarily focused on districts
rather than schools
• Key school relative accountability results reported every three years
rather than annually
• State accountability is determined entirely by test scores – there are no
school culture measures
• There is value in an overall performance index
– While a single number cannot tell the whole story, a “bottom line” based upon
multiple measures is needed for making decisions
– Transparency – outline exactly what measures will be included and how they will be
weighted
– Manageability – quantity of data can be overwhelming
3 M’s of data use: make it Manageable
make it Meaningful
make it Matter
4
State Accountability Flowchart
5
The MNPS Framework
Indicators and Weighting
Academic
Progress
50%
Attainment
& College
Readiness
30%
Achievement
Gap 5%
School
Culture
15%
6
What Factors Should be Included in School Evaluation?
Audience Participation
• Participants pair up
• Identify 3-5 key factors or indicators that you would
include in a school performance system (academic, nonacademic, school culture, etc.)
• How would you prioritize these factors (or should they be
equally weighted)?
• Discuss for 5 minutes and then we will report out
7
The MNPS Framework
K-8 Measures and Weighting
Performance Scale
Minimum & Maximum
Key
Indicators
Academic
Progress
Measure
Weight
TVAAS mean NCE gain
-5
12
4%
25%
TCAP Mean achievement level increase as % of target
-40
100
3%
25%
10
90
73%
15%
0
75
78%
15%
0
20
53%
5%
50%
100%
12%
5%
NA
NA
NA
5%
Student perceptions - Tripod Favorability (Elementary)
55%
85%
Student perceptions - Tripod Favorability (Middle)
45%
75%
0%
5%
0
100
36%
100%
Attainment & TCAP% of students scoring proficient
College
% of students projected from TVAAS to score 21 or above on
Readiness
ACT Composite
Achievement Index based upon gap in TCAP % proficient between
Gap
subgroups
Educator perceptions - TELL TN Survey Favorability
School
Culture
% of
Variation due
to % Free or
Minimum Maximum
Reduced
Parent perceptions
Total Points (weighted composite)
50%
Progress
30%
Attainment
5%
Gap
15%
Culture
8
The MNPS Framework
High School Measures and Weighting
Performance Scale
Minimum & Maximum
Key
Indicators
Academic
Progress
Measure
Weight
TVAAS mean scale score gain
-10
12
3%
25%
Mean achievement level increase as % of target
-30
100
0%
25%
10
95
78%
10%
0
95
89%
10%
60
100
34%
10%
0
20
58%
5%
50%
100%
0%
5%
NA
NA
NA
5%
40%
75%
35%
5%
0
100
40%
100%
Attainment & % of students scoring proficient
College
% of students scoring 21 or above on ACT Composite
Readiness
% of students graduating on-time with a regular diploma
Achievement Index based upon gap in EOC % proficient between
Gap
subgroups
School
Culture
% of
Variation due
to % Free or
Minimum Maximum
Reduced
Educator perceptions - TELL TN Survey Favorability
Parent perceptions
Student perceptions - Tripod Favorability
Total Points (weighted composite)
50%
Progress
30%
Attainment
5%
Gap
15%
Culture
9
The MNPS Framework
The Academic Performance Scale
• Identify key performance measures
• Determine the weight of each measure
• Determine the performance scale of each measure
• Assign performance points to each school based upon
position on performance scale
10
The MNPS Framework
The Academic Performance Scale
Determine the performance scale of each measure
Performance Scale
Original NCE scale
-20
-10
0
-5
Performance Scale Minimum
10
20
12
Scale Maximum
11
The MNPS Framework
School Ratings for Growth
• Two measures (50% of Total Framework):
Value Added and Mean Achievement Level Increase
• K-8 subjects: Math, Reading/LA, & Science
• HS subjects: Algebra I and II, English I, II, II, & Biology I
• Not highly correlated to socioeconomic status
12
The MNPS Framework
Mean Achievement Level Increase
2013
2012
School A
90
30%
80
Advanced
10%
Proficient
Percent of Students
50
Advanced
+2%
Basic
30%
30
20
30%
32%
80
Advanced
10%
60
50
-2%
Advanced
8%
Proficient
Proficient
20%
24%
32%
+2%
70
+2%
40
90
20%
Basic
30%
School B
12%
Proficient
20%
70
60
+2%
2013
2012
100
Percent of Students
100
Basic
Basic
24%
30%
-4%
40
30
Below
Basic
40%
Below
Basic
20
40%
38%
10
Below
Basic
Below
Basic
44%
10
0
0
Overall 6% Improvement
Overall 4% Decline (despite increase in Profic/Adv)
13
The MNPS Framework
Mean Achievement Level Increase Goal Example
Previous
Year
Establish
Proficiency
Increase
Goals Based
Upon PriorYear Results
100%
90%
80%
Advanced (A)
10%
Proficient (P)
20%
Proficiency
Increase Goal
Results if
Goals Met
= 0% x A = 0%
Advanced
= 20% x P = 4%
Percent of Students
50%
Proficient
25%
70%
60%
14%
Basic (B)
30%
= 30% x B = 9%
Basic
37%
40%
30%
20%
Below Basic
(BB)
40%
= 40% x BB = 16%
10%
Below
Basic
24%
0%
TOTAL = 0%+4%+9%+16% = 29%
14
The MNPS Framework
K-8 Ratings for Achievement & College Readiness
• Two measures (30% of Total): Percent Proficient/Advanced and percent
of 4th/8th grade students projected to score 21 or higher on the ACT
• Percent Proficient/Advanced includes Math (or Algebra I), Reading/LA, &
Science
• These measures are correlated with socioeconomic status (SES), but
schools often break the pattern
15
The MNPS Framework
High School Ratings for Achievement & College Readiness
• Percent Proficient/Advanced includes Algebra I and II, English I, II, and
III, and Biology I
• ACT Composite score of 21 or higher is required for the Hope
Scholarship and is the average of the subject area college readiness
benchmarks.
• These measures are correlated with Socioeconomic status, but schools
often break the pattern.
16
The MNPS Framework
Achievement Gap and Survey Data
•
•
•
•
•
Gap Closure (5% of framework)
K-8 subjects: Math, Reading/LA, & Science
HS subjects: Algebra I and II, English I, II, II, & Biology I
TELL and TRIPOD surveys each count as 5% of framework.
Parent survey to be added
17
The MNPS Framework
School Rankings Across Measures
18
The MNPS Framework
Report Format
19
Charter School Accountability
MNPS Commitments
1. Set and hold charter schools accountable to clear,
measurable, and attainable academic, financial, and
operational performance standards and targets;
2. Close schools that fail to meet performance standards and
targets; and
3. Work proactively to identify and establish new, high
quality charter schools to serve students who attend
schools identified for closure.
20
Outcome-based Performance Management
Performance
Contracts
Accountabilitybased
Interventions
Consistent
Communication
(Predictable,
enforceable)
(Transparent, balanced,
comprehensive)
(Face validity,
engagement)
District-Charter Collaboration Compact
Focus Groups
Scorecard
Performance Management, Replication, and Closure (PMRC) Grant
Performance Frameworks
Policies and Contracts
MNPS Research and Assessment
Technical Development
Balanced Measures
Broad Applicability
Outcome-based Performance Management
How will we use the APF?
• Publish Annual School Report Cards
• Shape Renewal Process
• Shape Recommendations each October
22
Academic Performance
Mean Achievement Increase
TVAAS
TCAP (%PA)
Achievement Gap
School Culture Measures
ACT (21+)
Overall Performance (APF)
Year by year
Renewal Information
Projection and Review Level
Renewal Application Deadline
5-year Review Year
Outcome-based Performance Management
How will we use the APF?
• Publish Annual School Report Cards
• Shape Renewal Process
• Shape Recommendations each October
25
3-year
status
Watch
Action
Timeline
Renewal Review Level
NA
Simple Renewal
2-years
(Updated budget, plans, targets)
Full Renewal Review
(Full Renewal Application: Renewal Based on Best Interests
Standard)
Conditional Renewal Review
1-year
(Renewal possible but unlikely; significant changes required;
may recommend revocation depending on end of year results)
Current
Cycle
No Renewal
(May recommend revocation to take affect end of year)
Outcome-based Performance Management
How will we use the APF?
• Publish Annual School Report Cards
• Shape Renewal Process
• Shape Recommendations each October
27
2013 Status
3-Year Status
Recommended Action
Excelling
Excelling
Simple Renewal Review
Lead Academy MS
Satisfactory
Satisfactory
None
LEAD Academy HS
Satisfactory
Satisfactory
None
New Vision Academy
Satisfactory
Satisfactory
None
Liberty Collegiate Acad
Excelling
2-years
None
STEM Prep Academy
Excelling
2-years
None
Target
2-years
Notice: Revocation in 2014 Likely
Excelling
2-years
None
Insuffic Data
2-years
None
Cameron College Prep
Review
2-years
None
Knowledge Academy
Achieving
1-year
None
Boy's Prep
Target
1-year
Notice: Revocation in 2014 Possible
Smithson-Craigh Acad
Target
Target
Notice: Revocation in 2014 Likely
Smithson-Craigh MS
Target
Target
Closed May 2013
School
KIPP Academy
Drexel Prep School
Nashville Prep School
East End Prep
What Decisions Would You Be Comfortable With?
Audience Participation
• Participants pair up
• What types of decisions would you be comfortable making
with a performance framework for your school(s)?
• How many years of data would you need to make these
decisions?
• What additional information would you want in making
decisions about school performance?
• Discuss for 5 minutes and then we will report out
29
The MNPS Framework
Next Steps
• Show expanded data by individual year
• Break out mean achievement level increase, TVAAS, and
gap calculations by subject
• Break out key results by subgroup
• Provide professional development on the Framework
• Expand documentation
• Utilize results in evaluation of initiatives and in identifying
schools needing support and resources
30
The MNPS Framework
Lessons Learned
• Include stakeholders in development
• Transparency is critical
• Flexibility is essential as standards, assessments and policies
change, but the basic principles we value and include in the
Framework should hold up over time
• Construct the Framework in such a way that the top
performance category is within reach of any school,
regardless of socioeconomic factors
• Performance measures can vary significantly from year to
year, so multiple years of data should be utilized in critical
decisions
• Presentation format is important
31
The MNPS Framework
Performance Framework Template
Academic Performance Framework Template
Measure
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
TVAAS Mean Scale Score Gain
TCAP % Proficient/Advanced
Achievement Gap Index
Tripod Survey Favorability
TELL TN Survey Favorability
TOTAL
Scale
Scale
Reverse
Minimum Maximum Scale?
-5
10
20
0.55
0.5
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
12
90
0
0.85
1
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
No
No
Yes
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
No
Weight
(0-100)
School
Value
25
15
5
5
5
2.5
48
7.2
0.68
0.77
55
% of
Possible Unadjusted
Points
# of Points
44.1%
47.5%
64.0%
43.3%
54.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
0.0%
Adjusted#
of Points
11.03
7.13
3.20
2.17
2.70
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
20.05
12.95
5.82
3.94
4.91
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
26.22
47.67
32
The MNPS Framework
Decision-making Process for Each Indicator
Each Category of Data Requires the Following Decisions:
Unit (e.g.
School)
Subject
Areas
Which
Measure
Years of
Data
Weighting/
Points
Performance
Thresholds
For Example: Value Added
• Should Math, Reading/LA, Science be included?
• Should Math, Reading/LA, Science be weighted equally and be combined into a single score?
• Should there be discrete score categories (e.g. 1-5), or a linear transformation of the scores such that schools earn from 0 to 100
percent of possible points, or should we use a method that takes into account the fact that most schools’ scores are going to be
clustered around the average?
• Should the categories be discrete or continuous, do we base them on average growth from 2012 to 2013, or take other years of
growth into consideration? If we use two years of data, we get a bigger range of possible scores.
• Do we set our evaluation based on what has been average in the school district and the state or based on the growth standard (i.e.
zero growth is average)? If we use the growth standard, we may fall behind the state. If we use actual average NCE gains, we will be
rating schools on a very different basis than the state does in its report card and on the TVAAS site (for grades 3-8 only).
33
The MNPS Framework
Questions?
34