Measuring Educator Effectiveness for Music Teachers A Pennsylvania Perspective Created by O David Deitz for 2012-2013 PMEA Mentoring Program Webinar Series (there are still a lot of questions to be asked and answered!) Big Idea: • • • • • • • • • • • • • • • • (B) FOR PROFESSIONAL EMPLOYES AND TEMPORARY PROFESSIONAL EMPLOYES WHO SERVE AS CLASSROOM TEACHERS, THE FOLLOWING SHALL APPLY: (1) BEGINNING IN THE 2013-201 4 SCHOOL YEAR, THE EVALUATIO N OF THE EFFECTIVENESS OF PROFESSIONAL EMPLOYES AND TEMPORARY PROFESSIONAL EMPLOYES SERVING AS CLASSROOM TEACHERS SHALL GIVE DUE CONSIDERATION TO THE FOLLOWING: (I) CLASSROOM OBSERVATION AND PRACTICE MODELS THAT ARE RELATED TO STUDENT ACHIEVEMENT IN EACH OF THE FOLLOWING AREAS: (A) PLANNING AND PREPARATION. (B) CLASSROOM ENVIRONMENT. (C) INSTRUCTION. (D) PROFESSIONAL RESPONSIBILITIES. (II) STUDENT PERFORMANCE, WHICH SHALL COMPRISE FIFTY PER CENTUM (50%) OF THE OVERALL RATING OF THE PROFESSIONAL EMPLOYE OR TEMPORARY PROFESSIONAL EMPLOYE SERVING AS A CLASSROOM TEACHER AND SHALL BE BASED UPON MULTIPLE MEASURES OF • STUDENT ACHIEVEMENT. Teacher Effectiveness System in Act 82 of 2012 Observation/Evidence Danielson Framework Domains 1. Planning and Preparation 2. Classroom Environment 3. Instruction 4. Professional Responsibilities Building Level Data Indicators of Academic Achievement Indicators of Closing the Achievement Gap, All Students Indicators of Closing the Achievement Gap, Subgroups Academic Growth PVAAS Other Academic Indicators Credit for Advanced Achievement Building Level Data, 15% Teacher Specific Data PVAAS / Growth Teacher Specific Data, 15% Observation/ Evidence, 50% Elective Data, 20% Elective Data/SLOs District Designed Measures and Examinations Nationally Recognized Standardized Tests Industry Certification Examinations Student Projects Pursuant to Local Requirements Student Portfolios Pursuant to Local Requirements 5 Observation/Evidence Danielson Framework Domains 1. Planning and Preparation 2. Classroom Environment 3. Instruction 4. Professional Responsibilities Building Level Data Indicators of Academic Achievement Indicators of Closing the Achievement Gap, All Students Indicators of Closing the Achievement Gap, Subgroups Academic Growth PVAAS Other Academic Indicators Credit for Advanced Achievement Building Level Data, 15% Observation/ Evidence, 50% Elective Data, 35% Elective Data/SLOs District Designed Measures and Examinations Nationally Recognized Standardized Tests Industry Certification Examinations Student Projects Pursuant to Local Requirements Student Portfolios Pursuant to Local 6 Charlotte Danielson’s Framework for Teaching Observation/Evidence Four Domains Danielson Framework Domains 22 Components 1. Planning and Preparation 50% 2. Classroom Environment 3. Instruction 4. Professional Responsibilities Domain Focus— Adapted from Danielson’s Framework for Teaching Planning and Preparation What a teacher knows and does in preparation for teaching. Classroom Environment All aspects of teaching that lead to a culture for learning in the classroom. Professional Responsibilities Instruction Professional responsibilities and behavior in and out of the What a teacher does classroom. to engage students in learning. 8 8 Multiple Measures of Teacher Effectiveness – Observation/Evidence and the Danielson Framework • Classroom observations by Principal/supervisor, including evidence that demonstrates behaviors associated with improving student achievement: – Planning and preparation, including selecting standards-based lesson goals and designing effective instruction and assessment; – Classroom environment, including establishing a culture for learning and appropriate classroom management techniques that maximize instructional time; – Instruction, including the use of research-based strategies which engage students in meaningful learning and utilize assessment results to make decisions abut student needs; and – Professional responsibilities, including using systems for managing student data and communicating with student families 9 The Framework for Teaching Domain 1: Planning and Preparation a. Demonstrating Knowledge of Content and Pedagogy b. Demonstrating Knowledge of Students c. Setting Instructional Outcomes d. Demonstrating Knowledge of Resources e. Designing Coherent Instruction f. Designing Student Assessments Domain 4: Professional Responsibilities a. Reflecting on Teaching b. Maintaining Accurate Records c. Communicating with Families d. Participating in a Professional Community e. Growing and Developing Professionally f. Showing Professionalism 10 The Framework for Teaching Charlotte Danielson Domain 2: The Classroom Environment a. Creating an Environment of Respect and Rapport b. Establishing a Culture for Learning c. Managing Classroom Procedures d. Managing Student Behavior e. Organizing Physical Space Domain 3: Instruction a. Communicating With Students b. Using Questioning and Discussion Techniques c. Engaging Students in Learning d. Using Assessment in Instruction e. Demonstrating Flexibility and Responsiveness 10 Danielson Framework, Domain 3: Instruction Component 1. Failing 3a: Communicating with students Expectations for learning, directions and procedures, and explanations of content are unclear or confusing to students. Teacher’s use of language contains errors or is inappropriate to students’ cultures or levels of development Expectations for learning, directions and procedures, and explanations of content are Teacher’s questions are Some of the teacher’s questions elicit a thoughtful response, but 3b: Using questioning and discussion techniques low-level or inappropriate, eliciting limited student participation, and recitation rather than discussion. 2. Needs Improvement clarified after initial confusion; teacher’s use of language is correct but may not be completely appropriate to students’ cultures or levels of development. most are low-level, posed in rapid succession. Teacher’s attempts to engage all students in the discussions are only partially successful. 3. Proficient Expectations for learning, directions and procedures, and explanations of content are clear to students. Communications are appropriate to students’ cultures and levels of development. Most of the teacher’s questions elicit a thoughtful response, and the teacher allows sufficient time for students to answer. The students are engaged and participate in the discussion, with the teacher stepping aside when appropriate. 4. Distinguished Expectations for learning, directions and procedures, and explanations of content are clear to students. Teacher’s oral and written communication is clear and expressive, appropriate to students’ cultures and levels of development, and anticipates possible student misconceptions. Questions reflect high expectations and are culturally and developmentally appropriate. Students formulate many of the highlevel questions and ensure that all voices are heard. 11 Building Level Data •Building Level Data • PSSA Achievement Building Level Data Achievement • PVAAS GrowthPSSA PVAAS Growth Graduation Rate • Graduation Rate Promotion Rate Attendance Rate AP Course Participation • Promotion Rate SAT/PSAT • Attendance Rate • AP Course Participation • SAT/PSAT 15 % Teacher Specific/Elective Data Teacher Specific Data PSSA Achievement PVAAS Growth IEP Growth Locally Developed District Rubrics Elective Data/SLOs District Designed Measures and Examinations Nationally Recognized Standardized Tests Industry Certification Examinations Student Projects Pursuant to Local Requirements Student Portfolios Pursuant to Local Requirements For MUSIC: Elective 15% 20% 35% What are some ways to develop evidence of student achievement in non-tested grades and subjects? Three Approaches to the Challenge Measures of Collective Performance Other Assessments Student Learning Objectives Measures of Collective Performance School-wide growth measures (like the PA Report Card) Standardized assessments used to measure the performance of groups of teachers (like program evaluation) Other Assessments Standardized end of course assessments, formative-benchmark-unit assessments Teacher-developed assessments of student learning or growth Student Learning Objectives Content- and grade/course-specific measurable learning objectives that can be used to document student learning over a defined period of time Can be written to measure growth or mastery What are other states/schools using for non-tested grades and subjects? Student Learning Objectives • NY, IN, MA, GA, UT, RI, NH, New Haven CT, Austin TX Other Assessments • FL, DE, Pittsburgh PA Measures of Collective Performance • KY, Pittsburgh PA What is PA planning to do ? Student Learning Objectives with Model Assessments Why this approach? The “other assessment” approach causes narrowing of curriculum and is not financially sustainable. The “measures of collective performance” are already included in the formula (required by Race to the Top) and do not offer NTGS teachers an opportunity to be evaluated on what they actually teach. SLOs that don’t describe the assessment protocol tend to lack a true learning focus. SLO Concepts • Student achievement can be measured in ways that reflect authentic learning of content standards. • Educator effectiveness can be measured through use of student achievement measures. Educator Effectiveness: SLO Measurement of Student Achievement Measurement of Educator Effectiveness www.education.state.pa.us > 1. Know and understand the Standards • PA Standards • Common Core Standards • Professional/Technical Standards 2. Assessment Tasks • Authentic to the grade or course • Aligned to Standards 3. Assessment Scoring • Can describe levels of student achievement toward standards based learning objectives THE PA SLO TEMPLATE & PROCESS What it is supposed to be: • A format to inform strong instructional practice and strong student achievement • A way to measure teacher effectiveness based on student achievement • An opportunity for teachers to define, describe and present data on student achievement in the content area that they teach What it is not supposed to be: • More paperwork for • More paperwork teachers that for has no teachers that no meaning or has purpose meaning or purpose • More testing for students • More testing for students • A weak substitute for PVAAS or other • A weak substitute for standardized testing PVAAS or other data standardized testing data Educator Effectiveness: SLO PA Educator Challenge To develop and implement an appropriately rigorous measure of teacher effectiveness based on student achievement in your content area through the use of the PA SLO Template. www.education.state.pa.us > The Approach • Provide a method for teachers to write SLOs. • Provide a method for teachers to design assessments appropriate to the SLO. • Provide piloted and peer-reviewed models on the PDE SAS website. • Provide professional development appropriate to SLO development. The Methodology: AN SLO TEMPLATE Educator Effectiveness: SLO STUDENT LEARNING OBJECTIVE TEMPLATE A format to provide a measure of teacher effectiveness based on student achievement of content standards, as applicable to the “Elective Data” portion of the Teacher Effectiveness System in Act 82 (HB 1901). www.education.state.pa.us > Looking at the Components of PA’s SLO Template • • • • • • • • Teacher Information Content Area Student Learning Objective Data and Targets Used to Establish the SLO Assessment/Performance Task Administration of the Assessment/Performance Task Evidence of Individual Student Achievement Strategies/Actions to Achieve the SLO Teacher Effectiveness Measure (Rating) • Implementation Timeline 30 …one piece at a time. 1. Teacher Information Teacher Name School Name District Name 2. Content Area Course Title or Content Area PA Standards Subject Area Grade Level(s) Brief Description of the course/ content area Number of students per class/session Number of minutes per class/session Frequency of classes/sessions Total number of classes/sessions 9. Teacher Effectiveness Measure Classroom Objective How will the aggregated scores of the “Evidence of Individual Student Achievement” results be used to define teacher effectiveness? Failing: few students achieve content mastery or growth Needs Improvement: less than a significant number of students achieve content mastery or growth Proficient: A significant number of students achieve content mastery or growth Distinguished: An exceptional number of students achieve content mastery or growth Grade 8 General Music Less than 60% of students will score 42-50 on a minimum of 2 assessments, with scores lower than 37 on the other two assessments. 60%-70% of students will score 42-50 on a minimum of 2 assessments, with scores lower than 37 on the other two assessments. 70%-84% of students will score 42-50 on a minimum on 2 out of the four assessments, and score no lower than 37 on the other two assessments. 85% of students will score 42-50 on a minimum of 3 out of the four assessments, and score no lower than 37 on a fourth assessment. HS Instrumental Music Mastery: Less than 50% of students achieve proficient or advanced levels in 5 out of 6 components on the second assessment. Growth: Less than 50% of students achieve improved scores in at least 2 out of 6 components, with no decrease in scores in the other components. Mastery: 50% -74% of students achieve proficient or advanced levels in 5 out of 6 components. Growth: 50% -74% of students achieve improved scores in at least 2 out of 6 components, with no decrease in scores in the other components.. Mastery: 70% -84% of students achieve proficient or advanced levels in 5 out of 6 components Growth: 70% -84% of students achieve improved scores in 2 out of 6 components, with no decrease in scores in the other components Mastery: 85% -100% of students achieve proficient or advanced levels in 5 out of 6 components Growth: 85% -100% of students achieve improved scores in 2 out of 6 components, with no decrease in the scores in other components 3. SLO (Student Learning Objective) Learning objectives that can be validly measured to document student learning over a defined period of time. Guiding Questions: Why is this objective important and meaningful to your students? Student Learning Objective Sample Statements: Grade 8 General Music: Students will utilize musical notation as both a performer and composer. HS Instrumental Music: Students develop independent performance skills appropriate to positive musical contribution in an instrumental performing ensemble. 3. SLO (Student Learning Objective) Learning objectives that can be validly measured to document student learning over a defined period of time. Guiding Questions: How is your student learning objective measurable and clearly understood by students in this grade/course? Based upon this objective, how will students independently demonstrate their learning? Sample Statements: SLO Rationale Grade 8 General Music: Students will independently demonstrate their learning by performing individually on the piano and guitar and by creating an individual musical composition based on the structural designs found in the music of the classic period (1750-1825). HS Instrumental Music: The quality and musicality of an instrumental ensemble is dependent on each individual as a contributor to the collective whole. Individual assessments inform each student of his/her progress and contribution as an independent musician. Curriculum Framework Connection What are the big ideas, essential questions, concepts and competencies (as identified by PA’s Curriculum Framework) to be measured? www.pdesas.org Big Ideas Essential Questions Statements from this section should be cut and pasted from: Concepts Competencies PA Standards Common Core Standards Professional/Industry Standards 4. Data and Targets Used to Establish the SLO Guiding Questions: What, if any, are the course or grade prerequisites? What do you know about the students in this class at the beginning of the learning process? What may typical and not-so-typical student progress look like on the way to achieving the learning objective? Sample Statements: Student Preparedness/ Baseline Data Grade 8 General Music: There are no specific prerequisites. They will have had some instruction in music history, composition, guitar, and keyboard in grades six and seven. Large lapses of time between instruction periods (up to a year) will cause a need for extensive review of concepts and competencies previously experienced. Most students will be successful at manipulating the instruments and writing music notation. Some students will struggle with conceptualizing ideas regarding music composition. A pretest of retained knowledge of music notation is administered on the first day of the course to inform instruction. 4. Data and Targets Used to Establish the SLO Guiding Questions: What is the expected outcome or goal by the end of the designated instructional period (i.e., year, semester, course length, etc.)? Using student baseline data, are differentiated targets/goals needed? If yes, what are those targets/goals? Are the targets ambitious yet realistic, and how will you monitor progress along the way? Targets Sample Statements: HS Instrumental Music: Each student will demonstrate an ability to perform his/her instrumental part independently in an unaccompanied setting, exhibiting mastery and/or growth in such components as tone quality, intonation, rhythm, musicality, and technique appropriate to the student’s specific instrument. 5. Assessment/Performance Task Name of the Assessment/ Performance Task Sample Statements: Grade 8 General Music: Classical Time Period Assessment Composition Assessment Keyboard Performance Assessment Guitar Performance Assessment HS Instrumental Music: HS Instrumental Music-Individual Student Performance 5. Assessment/Performance Task Description of the Assessment/ Performance Task Guiding Questions: Who is the developer of the assessments/performance task used (e.g., teacher-made, district-developed, commercial, etc.)? What is a description of the assessment/performance task that will be used to measure the student learning objective (SLO)? Are there any products or artifacts that will be gathered as part of the data collection process? Sample Statements: HS Instrumental Music: Individual Performance Assessments (IPA): The IPA will occur two times during the year, near the mid-point of the first and fourth marking periods. Students will usually submit the assessment electronically, a variety of submission formats will be available.. Material for this assessment will be drawn from both scales and ensemble music currently being rehearsed in class. Specific technical exercises taught and rehearsed in class may be included as well. IPA Requirements, First Marking Period 2012 Wind Symphony Educator Effectiveness: SLO Growth or Mastery Does this assessment/ performance task align with the student achievement goals for the SLO? Check one: Growth (change in student achievement across two or more points in time) Mastery (attainment of a defined level of achievement) Growth and Mastery www.education.state.pa.us > 6. Administration of the Assessment/Performance Task Frequency of Assessment/ Performance Task Administration Guiding Questions: How often and when is this assessment/performance task administered? If measuring growth, are multiple assessment windows in place? Sample Statements: Grade 8 General Music: Each of the four project assessments is administered once throughout the course. Formative assessments are inherent in the instructional design. A pretest is administered on the first day of class that assesses prior knowledge applicable to the composition and keyboard assessments. HS Instrumental Music: Students will be assessed at the midpoint of the first and fourth quarters. 6. Administration of the Assessment/Performance Task Guiding Questions: What unique or specific equipment, technologies, or resources are needed to complete this assessment/performance task? Resources Required Sample Statements: Grade 8 General Music: Keyboards, guitars, audio-playback equipment. HS Instrumental Music: Video/audio recording technology and/or methodologies for students to create online submissions that will require computers and appropriate software. 6. Administration of the Assessment/Performance Task All Classes: IEP and 504 accommodations will be implemented. Adaptations for Diverse Learners and/or Students with Disabilities Guiding Questions: What assessment/performance task adaptations are needed to assist diverse learners and/or students with disabilities? Sample Statements: Grade 8 General Music: Students struggling with keyboard or guitar playing skills can perform simple chord structures. Students can use a computer notation program to create the composition. HS Instrumental Music: IEP adaptations are made (additional time allotted, alternative print formats, teacher-directed instruction to help students feel more comfortable about being assessed individually, etc.) 6. Administration of the Assessment/Performance Task Most Classes: The assessments can be administered by an equivalent peer. Some Classes: Student use of equipment needed to complete the authentic assessment could be monitored by an equivalent peer, but should probably be monitored by the class instructor (i.e. athletic, career tech, chemistry, biology equipment). Personnel Guiding Questions: Can this assessment/performance task be administered by an equivalent peer (educator in a similar content area)? If not, please explain. Does a district policy exist with regard to assessment/performance task administration? Sample Statements: An equivalent peer can administer this assessment. Many students will selfadminister this assessment. 7. Evidence of Student Achievement Rubrics/ Scoring Scales Guiding Questions: How will individual student growth or mastery be determined (defined and scored) using this assessment/performance task? Include the specific rubric/scoring scale that will be used. Does the rubric and/or scoring scale correlate with the assessment/performance task? Sample Statements: HS Instrumental Music: See next slide…………. OR LOOK AT SCOTT SHEEHAN’S PMEA WEBINAR PRESENTATION 7. Evidence of Student Achievement Category Advanced 10-9 Tone quality is controlled, full, rich, and characteristic in all registers Proficient 8-7 Tone quality is characteristic in the normal playing range of the instrument, but distorts in a few passages or extreme registers (occasional lapses of control), these lapses rarely hinder the performance. Basic 6-5 Tone quality exhibits some flaws in production in the normal range of the instrument, extreme registers are usually uncontrolled, tone quality occasionally detracts from performance. Below Basic 4-0 Tone quality has significant flaws in basic production in all registers of the instrument and consistently detracts from the performance. Intonation Intonation is accurate in all ranges and registers Intonation is mostly accurate; the few problem pitches are adjusted to an acceptable standard during the performance. Intonation is somewhat accurate but consistently includes out-of-tune notes; these inaccuracies occasionally detract from the performance. Intonation is consistently inaccurate and hinders the quality of performance. Rhythmic Accuracy Rhythms are accurate and precise throughout the performance. Rhythms are nearly accurate; occasionally, rhythms lack precise interpretation. Most rhythm patterns are accurate, but errors in precision are present and occasionally detract from the performance. Many rhythms performed incorrectly or inconsistently, major errors are present and detract from the performance. Pitch Accuracy (melodic accuracy) Pitches are consistently correct. Occasional incorrect pitches are played, but do not detract from the performance Most pitches are correct, but errors are present and occasionally detract from the performance. Many pitches are performed incorrectly; this inaccuracy consistently seriously hinders the performance. Tone Quality 7. Evidence of Student Achievement Guiding Questions: In what format will data be collected (e.g., database, graphed, portfolio, etc.) Is a pre-post test being used? (If so, please describe.) How frequently will data be collected? Data Collection Scores from the rubric will be collected on a database. Data will be collected at the mid-point of the first and fourth marking periods. 7. Evidence of Student Achievement Scoring Student Progress/ Preparation Guiding Questions: How was baseline data collected? (If baseline data was not collected, please explain.) Can baseline data be compared with the results of this assessment/performance task? Sample Statements: Grade 8 General Music: Baseline data is collected through a pretest of retained knowledge; there is no assessment of authentic skills prior to instruction. HS Instrumental Music: Scoring from the first marking period assessment will be used as baseline data for describing growth. Scoring from the fourth marking period assessment will be used to describe growth and/or mastery. 7. Evidence of Student Achievement Guiding Questions: What evidence will be presented to principal/evaluator to support the teacher effectiveness measure? How will data be presented to the principal/evaluator (e.g., database, graphed, portfolio, individual student artifacts, etc.)? Data Presentation Sample Statements: Grade 8 General Music: Evidence will be presented through aggregated scores from the database. HS Instrumental Music: An ipad database program created by the teacher will be used to present scoring information. 7. Evidence of Student Achievement All Grades and Courses: The assessment can be scored by an equivalent peer. All Grades and Courses: The assessment is a commercially produced assessment that has a history of validity and reliability. Data Analysis and Interpretation Guiding Questions: How can the assessment/performance task results be interpreted in the same way across equivalent peers? Is there a reliable and valid scoring and interpretive process (i.e., state developed, district-based, commercial, standardized, etc.) that is associated with the assessment/performance task? If so, please describe. Sample Statements: An equivalent peer can use the rubrics to assess the projects. 8. Strategies/Actions to Achieve the SLO Guiding Questions: What formative assessment information lets you know if your instructional practices will lead to successful completion of the SLO? Assessment for Learning Sample Statements: Grade 8 General Music: Rubric will be provided to the students as a part of progress monitoring throughout the instructional period. HS Instrumental Music: Interim formative assessments during group lessons will be used to focus instruction toward successful completion of the SLO. 8. Strategies/Actions to Achieve the SLO Guiding Questions: Based upon reflection, what instructional practices would you like to change or strengthen? What professional learning and/or other type of support will help you to achieve this SLO? Alignment Sample Statements: with the Danielson Grade 8 General Music: Based on the formative assessment information provided by student composition and performance tasks, Framework students will be provided with supplemental materials to either for address music reading difficulties or enhance current abilities to go Teaching beyond the targets. (Danielson 3E) HS Instrumental Music: Based on weekly formative performance assessments, student seating within the band will be uniquely rearranged periodically to provide students the opportunity to develop higher levels of performance independence. (Danielson 2 E) Educator Effectiveness: SLO SLO Resources http://nassauboces.org/Page/1667 http://www.riseindiana.org/sites/default/files/files/RISE%201.0/Student%20Learning% 20Objectives%20Handbook%201%200%20FINAL.pdf http://www.ride.ri.gov/EducatorQuality/EducatorEvaluation/SLO.aspx http://www.gadoe.org/School-Improvement/Teacher-and-LeaderEffectiveness/Documents/SLO%20Manual.pdf Assessment Development Resource: beta.ctcurriculum.org The Elective Portion of PA’s Educator Effectiveness Design Student Learning Objectives: Understanding and Completing the SLO Template An Instructional Webinar Link: http://vclass.cciu.org/play_recording.html?recordingId=1 249208167245_1348758027178 Educator Effectiveness: SLO Contact: O. David Deitz SLO Project Lead, Educator Effectiveness oddeitz@comcast.net THANK YOU!