An Overview of the Types of Performance Data Used to Describe Missouri Schools A Close Look at the Indices that Governing Board Members Should Use to Evaluate Charter School Quality Sharon Ford Schattgen, Ph.D. November 7, 2012 Presentation Prepared for the Missouri Charter School Association’s Seminar for Governing Board Members Types of Performance Data Used to Evaluate School Quality Missouri Assessment Program Data MAP Grade-Level Assessment & End-of-Course Assessment Proficiency Rates TerraNova Median National Percentile Ranks MAP Scale Scores Averages, Arrayed by Cohort MAP Index Scores MAP Growth Estimates Other Summative Standardized Assessment Data Additional Standardized Achievement Test Scores (e.g., Stanford, ITBS, etc.) College Readiness/Placement Data (e.g., ACT/SAT Scores, Advanced Placement Scores) Local Interim Assessment Data (e.g., NWEA Assessment Data) Additional Performance Measures Attendance, Dropout, & Graduation Rates Post-Secondary Success Data Accountability Data Missouri School Improvement Program Annual Performance Report (MSIP APR) Information Associated with Special Designations 2 School X Demographic Data Enrollment/S chool Year 2007-08 2008-09 2009-10 Total 285 350 434 504 575 Black 270 (94.7%) 332 (94.9%) 407 (93.8%) 472 (93.7%) 540 (93.9%) 2 (.7%) 2 (.6%) 2 (.5%) 3 (.6%) Not reported White 13 (4.6%) 16 (4.6%) 25 (5.8%) 28 (5.6%) 31 (5.4%) Free/Reduce d Price Lunch 227 (85%) 321 (96.1%) 393 (92%) 467 (92.3%) 514 (89.4%) Hispanic 2010-11 2011-12 Free/Reduced Price Lunch Percentages for Corresponding Years: Comparison District = 72.3%, 68.7%, 83.8%, 85.7%, 87.4%; MO = 42.1%, 43.7%, 46.9%, 47.8%, 3 Note 1: Disaggregation of data by “subgroup” is required! One of the goals of the No Child Left Behind Act is to eliminate discrepancies in achievement across “subgroups” of students, which means Governing Board members should examine data that are disaggregated to allow you to see any “gaps” in performance. In other words, you need to review data that are disaggregated by racial or ethnic group, free/reduced-price lunch status, special-education status, and limited English proficiency status, and, when appropriate, gender. 4 Note 2: Most of the data Governing Board members will receive will not be presented in its original format. Rather, it will likely be summarized and reformatted for the purposes of creating an annual report. School personnel access their data from the DESE MCDS Portal, and test publishers also provide electronic and print reports. School Report Cards are posted at www.dese.mo.gov 5 Note 3: We assess achievement for specific purposes; here are 5 such purposes. 1. Provide summary information about what students know and can do relative to course objectives or content/process standards 2. Yield information on an interim basis to inform policymaker and/or educator decisions at classroom, school, or district level 3. Collect student-centered information to guide minute-byminute and/or day-to-day instruction 4. Place students in specific programs/services 5. Identify students who need further evaluation Purposes 3, 4, and 5 (listed in green, italic type) yield data that are not appropriate for inclusion in a report to a Governing Board. Data collected for 6 these purposes have utility only for educators. Perie, Marion, & Gong, 2009 Interim (instructional, evaluative, predictive) aka “common” or “benchmark” Formative (minute by minute and/or day-to-day integrated into lesson) aka “classroom” Frequency of administration Increasing Summative Scope (curriculum focus) & duration of cycle (time frame) Tiers of Achievement Assessment A Reminder: Types of Performance Data Used to Evaluate School Quality Missouri Assessment Program Data MAP Grade-Level Assessment & End-of-Course Assessment Proficiency Rates TerraNova Median National Percentile Ranks MAP Scale Scores Averages, Arrayed by Cohort MAP Index Scores MAP Growth Estimates Other Summative Standardized Assessment Data Additional Standardized Achievement Test Scores (e.g., Stanford, ITBS, etc.) College Readiness/Placement Data (e.g., ACT/SAT Scores, Advanced Placement Scores) Local Interim Assessment Data (e.g., NWEA Assessment Data) Additional Performance Measures Attendance, Dropout, & Graduation Rates Post-Secondary Success Data Accountability Data Missouri School Improvement Program Annual Performance Report (MSIP APR) Information Associated with Special Designations 8 Missouri Assessment Program (MAP) Grade-Level & End-of-Course (EOC) Assessment Proficiency Rates “MAP Proficiency Rate” is the percentage of students scoring in the top two MAP achievement levels—the Proficient Level and the Advanced Level. Following a brief introduction to the MAP, there is a sample line graph showing trends in MAP Grade-Level Proficiency Rates, a sample bar graph showing disaggregated GradeLevel Proficiency Rates for one year, and a sample table showing trends in End-of-Course Proficiency Rates. ASSESSMENT DATA 9 Missouri Assessment Program (MAP) • MAP includes required consecutive grade-level (3-8) assessments in Communication Arts and Mathematics and grade-span (5 and 8) assessments in Science. • MAP also includes end-of-course (EOC) exams, including English II, Algebra I, and Biology I. See DESE website for testing requirements. • MAP results are analyzed and reported in a variety of ways in order to inform policy and practice and to satisfy federal and state accountability requirements. • • • • Performance on each MAP assessment is reported in terms of four achievement levels: Below Basic, Basic, Proficient, and Advanced. The TerraNova tests, which yield National Percentile Ranks, are components of the MAP grade-level assessments. MAP results are also reported using Scale Scores (calculated by test publisher). DESE also calculates MAP Index Scores to use in the MO School Improvement Program (state accountability system). 10 MAP Mathematics Proficiency Rates Grade 7 School X Comparison Missouri 70 60.1 60 55.2 56.4 52.5 49.8 50 37.9 40 34.6 32.1 27.8 30 29.3 27.7 20 16.5 10 24.3 25.1 2011 2012 14.5 0 2008 2009 2010 11 2009 Proficiency Rates for Black Students and White Students: MAP Communication Arts Grade-Level and End-of-Course Assessments Columbia Public Schools and Missouri District Black District White State Black State White 100.0 90.0 85.6 Percent Proficient/Advanced 80.0 77.2 70.0 66.2 60.6 60.0 57.6 60.0 56.5 54.9 50.0 56.0 53.4 51.9 50.1 64.7 53.4 51.2 46.1 40.0 28.0 30.0 21.8 20.0 28.0 26.3 25.2 21.7 28.6 26.9 25.0 21.9 19.9 16.4 10.0 0.0 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Eng II EOC 12 MAP End-of-Course (EOC) Exam Proficiency Rates Algebra I 2009 2010 2011 2012 22.2% 32.8% 62.3% 59.9% N=72 N=49 N=50 N=50 Comparison School 22.7% 26.3% 41.7% 44.8% Missouri 52.6% 57.3% 59.7% 56.7% School X 13 TerraNova Median National Percentile Ranks TerraNova Tests are embedded within the MAP Grade-Level Assessments. Each TerraNova yields a National Percentile Rank, which allows us to compare the performance of our students to that of their national grade-level peers. To report group performance, we calculate the Median National Percentile Rank. TerraNova Median National Percentile Ranks for multiple years are shown on the following slide. ASSESSMENT DATA 14 TerraNova Median National Percentile Ranks School X Communication Arts School X Mathematics 2008 2009 2010 2011 2012 2008 2009 2010 2011 2012 Grade 6 41 36 33 33 43 Grade 6 28 21 27 22 32 Grade 7 43 39 44 43 41 Grade 7 29 33 29 31 31 Grade 8 34 36 48 44 37 Grade 8 49 45 56 29 36 School X Science (Grade 8) 2008: 39 2009: 30 2010: 37 2011: 36 2012: 37 NATIONAL Terra Nova Median Percentile Rank = 50; Range = 1 to 99 15 MAP Scale Score Averages, Arrayed by Cohort When arrayed by cohort, MAP Scale Score Averages show the change over time for a given group of students as they move through the grade span (e.g., sixth grade to seventh grade; seventh grade to eighth grade). This type of analysis, shown in the following table, indicates, in a very general way, whether there is student growth in achievement. ASSESSMENT DATA 16 MAP Scale Score Averages for Grade Cohorts School X Communication Arts Grade 6 Grade 7 Grade 8 2008 650 651 674 2009 644 653 675 2010 652 655 680 2011 646 654 676 2012 653 656 671 MAP Communication Arts Scale Scores range from 455 to 875 (grades 3-8). 17 MAP Index Scores Each year, DESE calculates a MAP Index Score for each content area in each grade span. The Index is calculated by multiplying the percentage of reportable students scoring within a given achievement level by a constant: 6 x % in Below Basic; 7 x % in Basic; 8 x % in Proficient; and 9 x % in Advanced. The resultant products are then summed to produce the Index, which ranges from 600-900. MAP Index Scores, arrayed over time, are used in the Missouri School Improvement Program as an indicator of whether there is longitudinal improvement in performance. The following table presents an example of MAP Index Scores over time. ASSESSMENT DATA 18 School Y MAP Index Scores Grade Level 2008 2009 2010 2011 2012 Grades 35 Math 711.5 725.0 735.0 739.4 742.6 Status 730.7 Source: Understanding Your Annual Performance Report, 2012-13. Missouri Department of Elementary & Secondary Education 19 MAP Growth Estimates MAP Growth Estimates are calculated using complex statistical formulae, which utilize each student’s MAP Scale Scores over time—for example, from sixth grade to seventh grade, and from seventh grade to eighth grade. The Department of Elementary and Secondary Education first reported MAP growth data in 2011 The next slide presents an example of MAP growth data. ASSESSMENT DATA 20 MAP Value-Added Growth Estimates Mathematics 2011 School X 2012 Between 0.095 and 0.16 School X Significant Between 0.082 and 0.25 Significant State Range: -3 to +3 State Average = 0 *Growth estimates were calculated using longitudinal MAP grade-level assessment data. 21 Other Summative Standardized Assessment Data Students may take other summative standardized assessments, in addition to the MAP— assessments such as the Stanford Achievement Test or the Iowa Test of Basic Skills or ACT’s EXPLORE or PLAN. If so, these tests will likely be administered in the fall or the spring of the year. Results are used to inform instructional decisions. A subsequent slide presents an example of data from PLAN, a standardized test that may be administered to 10th-grade students. ASSESSMENT DATA 22 Data from Other Standardized Achievement Tests • In addition to the MAP, students may take another summative assessment, such as Stanford Achievement Test, Iowa Test of Basic Skills, EXPLORE, or PLAN. • Scores from these tests could be reported in terms of: – Median Percentile Rank – Percent of Students Scoring At or Above Grade – Scale Score Average 23 PLAN Scale Score Averages Columbia Public Schools and Nation 2005-06 through 2008-09 Scale: 1-32 In 2008-09, 67% of CPS students earned Composite Scores at or above the national average. Scale Score Average 31 26 21 16 11 6 1 English Mathematics Reading Science Composite District 2005-06 18.6 18.7 18.2 19.7 18.9 Nation 2005-06 16.1 16.3 15.8 17.4 16.5 District 2006-07 18.6 19.2 18.5 19.7 19.1 District 2007-08 17.8 19.0 18.0 19.9 18.8 District 2008-09 18.2 19.2 18.0 19.1 18.8 Nation 2007-09 16.9 17.4 16.9 18.2 17.5 24 College Readiness/Placement Data The ACT is a set of curriculum-referenced tests that measure students’ readiness for college. ACT Scale Scores range from 1 to 36, and the 2012 national average is 21.1. A student whose Scale Score in a particular content area meets or exceeds the corresponding ACT College Readiness Benchmark has a high probability of being successful in an entry-level, credit-bearing college course in that subject. Scores from Advanced Placement Exams are used to award college credit or advanced placement in college coursework. Advanced Placement Exam cores range from 1 to 5. The following tables present example ACT data and Advanced Placement data. ASSESSMENT DATA 25 ACT Scale Score Averages Class of 2012 School X English Math Reading Science Composite 14 15.9 16.3 16 15.8 Comparison District Data not available 16.5 Missouri 21.4 21.1 21.9 21.5 21.6 National 20.5 21.1 21.3 20.9 21.1 26 ACT Data: Class of 2012 Percentage of Test-Takers Scoring at or Above National Average Composite Scale Score of 21 School X 20.0% Comparison District 18.2% Percentage of 2012 graduates taking ACT: School X = 85% Comparison District = 70% MO = 67-75% US = 52% Percentage of Test-Takers Scoring at or Above College Readiness Benchmark School X English: Math: Reading: Science: All 4: 18% 13% 15% 12% 12% Missouri English: Math: Reading: Science: All 4: 73% 46% 56% 33% 27% 27 Average Scores on Selected Advanced Placement Examinations, 2009 (Exams Most Frequently Taken by District's Students) District State Nation & Beyond 3. 74 3. 14 64 2. 84 2. 83 2. 2. 67 72 2. 2. 88 3. 21 99 2. 2. 92 88 2. 3. 31 3. 54 22 3. 2. 76 3.0 3. 48 3.5 2. 92 Average Score 4.0 3. 74 3. 93 4. 07 4.5 4. 02 4. 38 5.0 71 2. 2.5 2.0 1.5 1.0 Lang & Comp Psychology US History Calculus AB CPS n=189 CPS n=181 CPS n=143 CPS n=133 World History Lit & Comp Statistics Biology CPS n=120 CPS n=96 CSP n=74 CPS n=54 28 NWEA Assessment Indices Some charter schools periodically administer classroom-based assessments (such as those published by NWEA, the Northwest Evaluation Association) in order to monitor student progress and to adjust instruction based on student needs. Pre- and post-test NWEA assessment indices for the 2011-12 school year are presented on the following table; however, these data must be interpreted with care because we have no evidence of their technical characteristics (e.g., reliability and validity). However, such data could be appropriate for inclusion in a report to a board because they have the potential to provide information about student learning. ASSESSMENT DATA 29 Example of Local Interim Assessment: NWEA Indices Reading Math Grade 6 Percent At/Above Grade Level, Fall 2011 42% 18% Percent At/Above Grade Level, Spring 2012 44% 46% Percent At/Above Grade Level, Fall 2011 15% 20% Percent At/Above Grade Level, Spring 2012 40% 54% Percent At/Above Grade Level, Fall 2011 16% 25% Percent At/Above Grade Level, Spring 2012 53% 42% Percent At/Above Grade Level, Fall 2011 25% 36% Percent At/Above Grade Level, Spring 2012 66% 54% Grade 7 Grade 8 Grade 9 30 Attendance, Dropout, & Graduation Rates; Post-Secondary Success Data Attendance rates, dropout rates, and graduation rates, along with data about postsecondary success, are shown on the following four slides. These measures of performance complement achievement data. ADDITIONAL PERFORMANCE MEASURES 31 Attendance Rate 2008 2009 2010 2011 2012 Example School 91.2% 92.6% 93.4% 94.7% 93.9% Comparison District 88.6% 91.5% 92.0% 92.5% 93.1% Missouri 94.0% 94.4% 94.2% 94.4% 94.7% 32 Dropout Rate 2008 Example School Comparison District Missouri 2009 2010 2011 2012 0% 1.5% 1.6% 3.4% 7.8% 13.9% 13.8% 17.5% 23.6% 19.5% 3.5% 3.5% 3.2% 3.4% 3.4% 33 Graduation Rate Graduation Rate Academic Year: 2011-2012 Example School 88.7% Comparison District 74.5% Missouri 87.7% Provide trend data when possible. 34 Graduation Rate Post-Secondary Placement Rate • Over 85% of School Y’s 2010 graduates enrolled in post-secondary institutions, entered the job market in a field related to their career-education coursework, or joined the military. 35 Missouri School Improvement Program: Annual Performance Report The MSIP APR presents evaluative data about districts and charter schools in relation to 14 accountability standards. Schools are held accountable for only those standards pertaining to its grade span. In accordance with established criteria, the Missouri Department of Elementary & Secondary Education can issue designations based on a school’s or a district’s performance. The following two tables summarize School X’s and a comparison district’s 2012 MSIP APRs and provide data supporting a specific designation—in this case as a “focus school.” ACCOUNTABILITY DATA 36 MSIP Standard/Indicator School X Comparison District 9.1.3. MAP Grades 6-8 Math Not Met Not Met 9.1.4 MAP Grades 6-8 Comm Arts Not Met Not Met 9.1.5 EOC Algebra I Math Not Met Met 9.1.6 EOC English II Comm Arts Not Met Not Met [Bonus MAP Achievement] Not Met Not Met [MAP Grade 8 Science] Not Met Not Met 9.3 ACT Not Met Not Met 9.4.1 Advanced Courses Not Met Met 9.4.2 Career Education Courses Not Met Met 9.4.3 College Placement Not Met Met 9.4.4 Career Education Placement Not Met Met 9.5 Graduation Rate Met Met 9.6 Attendance Rate Met Met 9.7 Subgroup Achievement Not Met Not Met 37 School X “Focus School” Indices Percentage of Students Scoring Proficient or Advanced on MAP Assessments (i.e., MAP Proficiency Rate) 2010 2011 2012 Three-Year Cumulative Student Gap Group, English/Langua ge Arts 27.5% 22.8% 20.9% 24.0% Student Gap Group, Mathematics 29.0% 26.4% 29.7% 28.3% Combined English/Language Arts & Mathematics Proficiency Rate = 26.2% Targets: Communication Arts = 44.21%; Math = 44.78% 38 And, by the way . . . The tests, they are a changin’ • In 2014-15, new state assessments will become operational in Missouri—assessments that are referenced to the Common Core State Standards and are designed to measure students’ readiness for college and careers. • Two consortia are developing such assessments: * Smarter Balanced * PARCC 39 In closing . . . . Questions & Comments . . . . For further information, go to these websites. MAP Assessments & MSIP: www.dese.mo.gov Assessment Consortia: www.smarterbalanced.org www.parcconline.org 40