Teacher Evaluation for Teachers in Non-Tested Grades and Subjects A Pennsylvania Perspective Created by O David Deitz for the PMEA Leadership Conference 7/16/12 (there are still a lot of questions to be asked and answered!) • • • • • • • • • • • • • • • • (B) FOR PROFESSIONAL EMPLOYES AND TEMPORARY PROFESSIONAL EMPLOYES WHO SERVE AS CLASSROOM TEACHERS, THE FOLLOWING SHALL APPLY: (1) BEGINNING IN THE 2013-201 4 SCHOOL YEAR, THE EVALUATIO N OF THE EFFECTIVENESS OF PROFESSIONAL EMPLOYES AND TEMPORARY PROFESSIONAL EMPLOYES SERVING AS CLASSROOM TEACHERS SHALL GIVE DUE CONSIDERATION TO THE FOLLOWING: (I) CLASSROOM OBSERVATION AND PRACTICE MODELS THAT ARE RELATED TO STUDENT ACHIEVEMENT IN EACH OF THE FOLLOWING AREAS: (A) PLANNING AND PREPARATION. (B) CLASSROOM ENVIRONMENT. (C) INSTRUCTION. (D) PROFESSIONAL RESPONSIBILITIES. (II) STUDENT PERFORMANCE, WHICH SHALL COMPRISE FIFTY PER CENTUM (50%) OF THE OVERALL RATING OF THE PROFESSIONAL EMPLOYE OR TEMPORARY PROFESSIONAL EMPLOYE SERVING AS A CLASSROOM TEACHER AND SHALL BE BASED UPON MULTIPLE MEASURES OF • STUDENT ACHIEVEMENT. Goal: An effective teacher in every classroom. Teacher Effectiveness System in Act 82 (HB 1901) Observation/Evidence Danielson Framework Domains 1. Planning and Preparation 2. Classroom Environment 3. Instruction 4. Professional Responsibilities Building Level Data, 15% Building Level Data PSSA Achievement PVAAS Growth Graduation Rate Promotion Rate Attendance Rate AP Course Participation SAT/PSAT Teacher Specific Data PSSA Achievement Teacher Specific PVAAS Growth Data, 15% IEP Growth Locally Developed District Rubrics Observation/ Evidence, 50% Elective Data, 20% Elective Data/SLOs District Designed Measures and Examinations Nationally Recognized Standardized Tests Industry Certification Examinations Student Projects Pursuant to Local Requirements Student Portfolios Pursuant to Local Requirements 5 Charlotte Danielson’s Framework for Teaching Observation/Evidence Four Domains Danielson Framework Domains 22 Components 1. Planning and Preparation 50% 2. Classroom Environment 3. Instruction 4. Professional Responsibilities Building Level Data •Building Level Data • PSSA Achievement Building Level Data Achievement • PVAAS GrowthPSSA PVAAS Growth Graduation Rate • Graduation Rate Promotion Rate Attendance Rate AP Course Participation • Promotion Rate SAT/PSAT • Attendance Rate • AP Course Participation • SAT/PSAT 15 % Teacher Specific/Elective Data Teacher Specific Data PSSA Achievement PVAAS Growth IEP Growth Locally Developed District Rubrics Elective Data/SLOs District Designed Measures and Examinations Nationally Recognized Standardized Tests Industry Certification Examinations Student Projects Pursuant to Local Requirements Student Portfolios Pursuant to Local Requirements For NTGS: Elective 15% 20% 35% What are some ways to develop evidence of student achievement in non-tested grades and subjects? Three Approaches to the Challenge Measures of Collective Performance Other Assessments Student Learning Objectives Measures of Collective Performance School-wide growth measures (like the PA Report Card) Standardized assessments used to measure the performance of groups of teachers (like program evaluation) Other Assessments Standardized end of course assessments, formative-benchmark-unit assessments Teacher-developed assessments of student learning or growth Student Learning Objectives Content- and grade/course-specific measurable learning objectives that can be used to document student learning over a defined period of time Can be written to measure growth or mastery What are other states/schools using for non-tested grades and subjects? Student Learning Objectives • NY, IN, MA, GA, UT, RI, NH, New Haven CT, Austin TX Other Assessments • FL, DE, Pittsburgh PA Measures of Collective Performance • KY, Pittsburgh PA What model might be most appropriate for PA to develop? Student Learning Objective Models Models of Student Assessment Measurement of Student Performance & Teacher Effectiveness What is PA planning to do ? Student Learning Objectives with Model Assessments Why this approach? The “other assessment” approach causes narrowing of curriculum and is not financially sustainable. The “measures of collective performance” are already included in the formula (required by Race to the Top) and do not offer NTGS teachers an opportunity to be evaluated on what they actually teach. SLOs that don’t describe the assessment protocol tend to lack a true learning focus. The Approach • Provide a method for teachers to write SLOs. • Provide a method for teachers to design assessments appropriate to the SLO. • Provide piloted and peer-reviewed models on the PDE SAS website. • Provide professional development appropriate to SLO development. The Methodology: AN SLO TEMPLATE Looking at the components of PA’s SLO Template….. • • • • • • • • • Teacher Information Course Information Learning Objective Data and Targets Used to Establish the SLO Assessment/Performance Task Administration of the Assessment/Performance Task Data Collection Implementation Timeline Teacher Effectiveness Measure …one piece at a time. Teacher Information Teacher Name School Name District Name Course Information Course Title Content Area Grade Level(s) Brief Course Description Number of students per class/session Number of minutes per class/session Frequency of classes/sessions Total number of classes/sessions LO (Learning Objective) (SLO) Student Learning Objective SLO Rationale Why was this objective chosen? How do you know it is a worthy area of focus? What are the transfer goals? Based upon this objective, how will students be able to independently demonstrate their learning? Curriculum Framework Connection What are the big ideas, essential questions, concepts and competencies (as identified by PA’s Curriculum Framework) to be measured? Big Ideas Essential Questions Concepts Competencies Standards Alignment Based upon the big ideas, essential questions, concepts and competencies selected, identify the content standards that are aligned to this objective. Data and Targets Used to Establish the SLO Awareness of student level of preparedness What, if any, are the course or grade prerequisites? What is shown by the current (baseline) data? Which students may need targeted objectives (e.g. students with IEPs, diverse learners, etc.)? Targets What is the expected outcome (goal) by the end of the designated instructional period (i.e., year, semester, course length, etc.)? Using student baseline data, how many differentiated targets/goals are needed? Are the targets ambitious yet realistic? Assessment/Performance Task Assessment/Performance Task Name Description of the Assessment/Performance Task Describe the assessment/performance task that will be used to measure the learning objective (LO). Consider any products or artifacts that will be gathered as part of the data collection process. Identify the developer of the assessments/performance task used (e.g., teacher-made, district-developed, commercial, etc.). Assessment/Performance Task Objectives Rationale Describe how the assessment/performance task authentically reflects the learning objective (LO). How is this assessment/performance task a reliable and valid measure of student mastery or growth aligned with expected outcomes and goals? How does the assessment/performance task measure the ability of students to independently apply what they have learned? How is the assessment/performance task free of bias? How do the assessment/performance task outcomes inform instruction? How does the assessment/performance task align with grade, school and district level curricular goals? Growth or Mastery Does this assessment/performance task align with the student achievement goals for the LO? Growth (change in student achievement across two or more points in time) Mastery (attainment of a defined level of achievement) Growth and Mastery Administration of the Assessment/Performance Task Frequency of Assessment/Performance Task Administration How often and when is this assessment/performance task administered? If measuring growth, are multiple assessment windows in place? Resources Required What unique or specific equipment, technologies, or resources are needed to complete this assessment/performance task? Adaptations for Diverse Learners and/or Students with Disabilities What adaptations built into the assessment/performance task are needed assist diverse learners and/or students with disabilities? Personnel Can this assessment/performance task be administered by an equivalent peer? (If not, please explain.) Does a district policy exist with regard to assessment/performance task administration? Data Collection Measuring Student Progress/Preparation How was baseline data collected? (If baseline data was not collected, please explain.) Can baseline data be compared with the results of this assessment/performance task? Data Collection In what format will data be collected (e.g., database, graphed, portfolio, etc.)? Is a pre-post test being used? (If so, please describe.) How frequently will data be collected? Rubrics and Scoring Scales How will individual student growth or mastery be determined (defined and scored) using this assessment/performance task? Include the specific rubric/scoring scale that will be used. Does the rubric and/or scoring scale correlate with the assessment/performance task? Data Interpretation How can the assessment/performance task results be interpreted in the same way across equivalent peers? Is there a reliable and valid scoring and interpretive process (i.e., state developed, district-based, commercial, standardized, etc.) that is associated with the assessment/performance task? If so, please describe. Implementation Timeline Date SLO is due to principal Date(s) for Assessment and Data Collection Dates to complete Data Interpretation Date to present Teacher Effectiveness Measure Teacher Effectiveness Measure Classroom Objective How will the aggregated scores of individual student assessment results be used to define teacher effectiveness? Failing: few students Needs Improvement: Proficient: A significant Distinguished: An achieve content mastery less than a significant number of students exceptional number of or growth number of students achieve content students achieve achieve content mastery or growth content mastery or mastery or growth growth Targeted Objective How will the mastery or growth of targeted student populations be described and used to define teacher effectiveness? Failing: Did not meet goal, Needs Improvement: little to no student Did not fully meet goal mastery or growth but showed some student mastery or growth Targeted Population: Mastery and/or growth goal: Proficient: Met goal or otherwise demonstrated significant student mastery or growth Distinguished: Surpassed goal or otherwise demonstrated significant student mastery or growth Teacher Effectiveness Ratings What were the results of the assessments/tasks and how do they relate to the classroom and targeted objectives? Classroom Objective Notes/Explanation Failing Needs Improvement Proficient Distinguished Targeted Objective Failing Needs Improvement Proficient Distinguished Notes/Explanation SLO Process Dimension SLO –Assessment Developers-Expertise Who was involved in establishing the SLO and what is their expertise? Exemplars of Student Work Let’s see how we can apply this template to a music teaching setting. (Since choir is what I know the best, I’ll do a choir SLO. With a little reworking, this SLO would be applicable to other performance situations.) Teacher Information Teacher Name School Name District Name O David Deitz Teacher Effectiveness Demonstration PA Dept of Education Course Information Course Title Choir Content Area Music Grade Level(s) 6-8, 9-12 Brief Course Description Secondary level music course that provides opportunity to develop and apply the skills, techniques and knowledge appropriate to choral singing. An integral part of the course is the on-going preparation for public concert performances; in and out of school. Number of students per class/session 30-70 Number of minutes per class/session 40 Frequency of classes/sessions 3 times per week Total number of classes/sessions 108 LO (Learning Objective) (SLO) Student Learning Objective Students will develop independent performance skills appropriate to positive musical contribution to a choral performing ensemble. SLO Rationale Why was this objective chosen? How do you know it is a worthy area of focus? What are the transfer goals? Based upon this objective, how will students be able to independently demonstrate their learning? The quality and musicality of a choir is dependent on each individual as a contributor to the collective whole. Individual assessment informs each students of his/her progress and contribution as an independent musician and (hopefully) as a life-long participant in choral singing. Curriculum Framework Connection What are the big ideas, essential questions, concepts and competencies (as identified by PA’s Curriculum Framework) to be measured? The skills, techniques, elements and principles of the Big Ideas arts can be learned, studied, refined and practiced. Essential Why is it important to be able to create, recreate and Questions perform music independently? While much of the school-based musical experience happens within a group, it is also important for people Concepts to be able to create, recreate, rehearse and perform music independently. Demonstrate the ability to independently create, Competencies recreate, rehearse and perform musical works and explain why this is important. Standards Alignment Based upon the big ideas, essential questions, concepts and competencies selected, identify the content standards that are aligned to this objective. Academic Standards for the Arts and Humanities 9.1 Production, Performance and Exhibition of Dance, Music, Theatre and Visual Arts A. Know and use the elements and principles of each art form to create works in the arts and humanities Music elements: • duration • intensity • pitch • timbre Music principles: • composition • form • genre • harmony • rhythm • texture B. Recognize, know, use and demonstrate a variety of appropriate arts elements and principles to produce, review and revise original works in the arts. Music: • sing • play an instrument • read and notate music • compose and arrange •improvise C. Integrate and apply advanced vocabulary to the arts forms Data and Targets Used to Establish the SLO Awareness of student level of preparedness What, if any, are the course or grade prerequisites? What is shown by the current (baseline) data? Which students may need targeted objectives (e.g. students with IEPs, diverse learners, etc.)? There are no prerequisites for this class. No audition is required, but early screening to assign students to a choral singing part can provide the teacher with some baseline information. Students may have previously been in a choral ensemble class, providing a previous year summative assessment. In addition to information provided by IEPs, students with singing difficulties (pitch matching, speech problems, rhythm problems, and vocal production problems) may need objectives tailored to the nature of the difficulty. Four students with pitch match/tonality problems are identified for this SLO. Targets What is the expected outcome (goal) by the end of the designated instructional period (i.e., year, semester, course length, etc.)? Using student baseline data, how many differentiated targets/goals are needed? Are the targets ambitious yet realistic? Each student will demonstrate an ability to sing his/her choral part independently in an unaccompanied setting, exhibiting mastery and/or growth in such components as tone quality, expression and style, phrasing, pitch, intonation, rhythm, diction and articulation. Assessment/Performance Task Assessment/Performance Task Name Choral Music-Individual Student Performance Description of the Assessment/Performance Task Describe the assessment/performance task that will be used to measure the learning objective (LO). Consider any products or artifacts that will be gathered as part of the data collection process. Identify the developer of the assessments/performance task used (e.g., teachermade, district-developed, commercial, etc.). Each member of the high school choir will individually perform a teacherchosen unaccompanied excerpt from the concert music (repertoire). These will be recorded by having students sing either into a microphone or within the visual and audio range of a video camera. Music selected for the second assessment will be of an equivalent grade level, as adjudicated by professional music organizations. Assessment/Performance Task Objectives Rationale Describe how the assessment/performance task authentically reflects the learning objective (LO). How is this assessment/performance task a reliable and valid measure of student mastery or growth aligned with expected outcomes and goals? How does the assessment/performance task measure the ability of students to independently apply what they have learned? How is the assessment/performance task free of bias? How do the assessment/performance task outcomes inform instruction? How does the assessment/performance task align with grade, school and district level curricular goals? Individual assessments are the only way of ascertaining student progress within the context of the larger performance ensemble. Assessments of this type can both be used as an on-going process through formative assessments (making the necessary pedagogical adjustments in instruction and learning, for both the teacher and the student), as well as summative assessments in the final stages of concert preparation. Growth or Mastery Does this assessment/ performance task align with the student achievement goals for the LO? Growth (change in student achievement across two or more points in time) Mastery (attainment of a defined level of achievement) Growth and Mastery Administration of the Assessment/Performance Task Frequency of Assessment/Performance Task Administration How often and when is this assessment/performance task administered? If measuring growth, are multiple assessment windows in place? Students will be assessed at the end of the second quarter and again at the end of the fourth quarter. Resources Required What unique or specific equipment, technologies, or resources are needed to complete this assessment/performance task? Current choir repertoire, consisting of an eclectic representation of music from various styles, levels of difficulty, and written in various harmonic approaches (homophonic, polyphonic, syllabic, etc.). Each student will be given an assessment rubric included in their folder, in order to be familiar with the standards and various levels of proficiency. Basic recording device in order to record student assessments. Adaptations for Diverse Learners and/or Students with Disabilities What adaptations built into the assessment/performance task are needed assist diverse learners and/or students with disabilities? Students may be diagnosed/identified as possessing emotional or mental disorders such as ADHD, ADD, & MR; requiring instructional and assessment adaptations, according to the requirements of IDEA and the student’s IEP. Adaptations may include factors such as: the student’s comfort level in singing alone; student ability to learn, comprehend or retain subject matter; or to perform certain passages with rhythmic or melodic accuracy when singing alone. Personnel Can this assessment/performance task be administered by an equivalent peer? (If not, please explain.) Does a district policy exist with regard to assessment/performance task administration? This assessment could be administered by an equivalent peer; it can also be administered by the student. Data Collection Measuring Student Progress/Preparation How was baseline data collected? (If baseline data was not collected, please explain.) Can baseline data be compared with the results of this assessment/performance task? Assessment from the 2nd quarter will establish baseline data for the 4th quarter. Data Collection In what format will data be collected (e.g., database, graphed, portfolio, etc.)? Is a pre-post test being used? (If so, please describe.) How frequently will data be collected? Data will be collected from listening to the recordings and notating the level of student performance on the attached rubric. This process will be administered twice; the first will be considered a pre-test. Rubrics and Scoring Scales How will individual student growth or mastery be determined (defined and scored) using this assessment/performance task? Include the specific rubric/scoring scale that will be used. Does the rubric and/or scoring scale correlate with the assessment/performance task? Students will be assessed using the following rubric: COMPONENT A. Tone Quality 4-Advanced Tone is consistently focused, clear, and centered throughout the range of the voice. 3-Proficient Tone is focused, clear and centered through the normal singing range. Extremes in range sometimes cause tone to be less controlled. Tone quality typically does not distract from the performance. 2-Basic Tone is often focused, clear and centered, but sometimes the tone is uncontrolled in the normal singing range. Extremes in range are usually uncontrolled. Occasionally the tone quality distracts from overall performance. 1-Below Basic The tone is often not focused, clear or centered regardless of the range, significantly detracting from the overall performance. B. Expression and Style Performs with a creative nuance and style in response to the score and limited coaching. Typically performs with nuance and style that is indicated in the score or which is suggested by instructor or peer. Sometimes performs Rarely demonstrates with nuance and style expression and style. Just that is indicated in the plays the notes. score or which is suggested by instructor or peer. C. Phrasing Phrasing is always consistent and sensitive to the style of music being sung. Phrasing is usually consistent and sensitive to the style of music being sung. Phrasing is usually consistent and occasionally sensitive to the style of music being sung . D. Pitch Virtually no errors. Pitch is very accurate. An occasional isolated error, but most of the time pitch is accurate and secure. Some accurate pitches, Very few accurate or but there are frequent secure pitches. and/or repeated errors. E. Intonation Exemplary intonation. Practically of pitches are sung in tune. Slight imperfections, but most pitches are sung in tune, maintaining tonal center (key) throughout. Regular intonation Significant intonation difficulties, lacking difficulties throughout, consistent tonal center. resulting in poor tonal center or recognizable melody. Phrasing is rarely consistent and/or rarely sensitive to musical style. F. Rhythm The beat is secure and the rhythms are accurate for the pitches being sung. The beat is secure and the rhythms are mostly accurate. There are a few duration errors, but these do not detract from the overall performance. The beat is somewhat erratic. Some rhythms are accurate. Frequent or repeated duration errors. Rhythm problems occasionally distract from the overall performance. The beat is usually erratic and rhythms are seldom accurate distracting significantly from the overall performance. G. Diction Student articulates clearly, and the text of the music is understandable. Student articulates the words somewhat clearly, and the text can be understood most of the time. Student sometimes Student rarely articulates articulates the words, the words and the text is but the text is often not not discernible. discernible. H. Articulation Manner of articulation is stylistically clear, agile and unified, according to the written musical excerpt, adding a significant degree of clarity and contrast to the music. Manner of articulation is mostly clear, according to the written musical excerpt. Minor and infrequent inconsistencies do not significantly distract form the musical clarity and contrast. Manner of articulation is inconsistent and/or awkward and/or incorrect, according to the written musical excerpt. Adding some clarity and contrast. Manner of articulation is not very evident, according to the written excerpt, resulting in very little to no clarity and contrast. Data Interpretation How will the assessment/performance task results be interpreted in the same way across equivalent peers? Is there a reliable and valid scoring and interpretive process (i.e., state developed, district-based, commercial, standardized, etc.) that is associated with the assessment/performance task? If so, please describe. An equivalent peer could adjudicate student performance using this rubric by listening to the recordings. This process has a history of reliability, as found through the CT state common assessments. (ctcurriculum.org) Implementation Timeline Date SLO is due to principal Prior to the start of the school year. Date(s) for Assessment and Data 1st assessment: second week in January Collection 2nd assessment: 3rd week in May Dates to complete Data Interpretation Date to present Teacher Effectiveness Measure 1st week in June June 5, 2013 Teacher Effectiveness Measure Classroom Objective How will the aggregated scores of individual student assessment results be used to define teacher effectiveness? Failing: few students Needs Improvement: Proficient: A significant Distinguished: An achieve content mastery less than a significant number of students exceptional number of or growth number of students achieve content students achieve achieve content mastery or growth content mastery or mastery or growth growth 50% -74% of students will achieve Less than 50% of proficient or students will achieve proficient or advanced advanced levels in 6 out of 8 components levels in 6 out of 8 components (mastery) (mastery) or will achieve improved or will achieve improved scoring in 3 scoring in 3 out of 8 out of 8 components, components, with no with no decrease in the decrease in the other 5 components other 5 components (growth). (growth). 70 % -84% of students will achieve proficient or advanced levels in 6 out of 8 components (mastery) or will achieve improved scoring in 3 out of 8 components, with no decrease in the other 5 components (growth). 85% -100% of students will achieve proficient or advanced levels in 6 out of 8 components (mastery) or will achieve improved scoring in 3 out of 8 components, with no decrease in the other 5 components (growth). Targeted Objective How will the mastery or growth of targeted student populations be described and used to define teacher effectiveness? Failing: Did not meet goal, little to no student mastery or growth Needs Proficient: Met Distinguished: Improvement: Did goal or otherwise Surpassed goal not fully meet goal demonstrated otherwise but showed some significant student demonstrated student mastery or mastery or growth significant student growth mastery or growth Targeted Population: Four students are identified as having pitch match/tonality problems, as determined by history of participation and/or initial screening via audition process. Mastery and/or growth goal: Students with pitch match problems/tonality problems will be assessed with vocal or piano support, on the same voice part, with an expectation that pitch match occur a minimum of 90% of the time. Teacher Effectiveness Ratings What were the results of the assessments/tasks and how do they relate to the classroom and targeted objectives? Classroom Objective Failing Needs Improvement Proficient Distinguished Notes/Explanation Targeted Objective Notes/Explanation Failing Needs Improvement Proficient Distinguished Literature selected was not comparable between the two assessments. SLO Process Dimension SLO –Assessment Developers-Expertise Who was involved in establishing the SLO and what is their expertise? SLO/Student Assessment Development Team June 2012 Sheryl Monkelien, Chair, Music Education - Mansfield University Margaret Bauer, Executive Director – PMEA Dennis Emert, Instrumental Music – Fox Chapel SD – Pittsburgh Gerald Savage, Chair, Vocal Department, CAPA – Pittsburgh Ross Whiting, Fine Arts & Humanities Department – Bok Tech HS – Philadelphia Jamie Kasper, Associate Director – Arts Education Collaboration – Pittsburgh. Exemplars of Student Work Exemplars can be found at ctcurriculum.org. What is it that the PA SLO process hopes to accomplish? To meet federal Race to the Top Requirements To meet state legislated teacher evaluation requirements To utilize some processes that teachers already have in place To inform principals as to appropriate processes in content areas with which they may not be familiar To provide teachers in non-tested grades and subjects an opportunity to be evaluated in their content area To inform and instruct teachers in non-tested grades and subjects about standards-based education design: 1) 2) 3) 4) 5) Curriculum Frameworks Standards Backward Design Transfer of Learning PA Standards Aligned System Website To have an effective teacher in every classroom. Thanks for all that you do to bring the arts to the students and communities in Pennsylvania! OK, I guess this won’t affect my golf game too much! Phase III Teacher Effectiveness Pilot • • • • • • Want to develop or pilot an SLO? Name/email School/District Principal Name/email Specific Discipline in Music Develop-Pilot-Both Questions? oddeitz@comcast.net