Data Wise Data-Driven Decision Making Tulsa Public Schools September 18, 2012 Objectives 2 Develop a common language for and understanding of data analysis Examine the 8-step Data Wise Decision Making Process Learn how to use multiple data sources to build a data-driven culture where evidence is used to make all decisions and action is taken for improvement. Agenda The Data Wise Decision-Making Cycle Case Study – Mount Baker Middle School Working with Data Data Analysis Break Data Analysis Continued Alignment to WISE Process Next Steps 3 Implementation Audit District Data and Recommendations for Future Actions - Fall 2011-Lead and Learn CenterDistrict Initiative Professional Learning Communities (District Average) Learning Context Instructional and Assessment Strategies/Practices Professional Development Leadership Practices Average Total Score 1.4 1.1 0.8 1.6 1.2 Implementation Scale Level of Implementation From To Deep Implementation 3.5 4.0 Full Implementation 2.5 3.49 Partial Implementation 1.5 2.49 Emerging Implementation .25 1.49 No Implementation 0 .24 Description/Recommendations: • Use data through an inquiry process to promote an action orientation and focus on results. • Staff members work collaboratively in processes that foster continuous improvement in all indicators of student achievement. • Leadership of school improvement processes is widely dispersed and helps sustain a culture of continuous improvement. Pair-Share Introduce yourself to a person near you. Share one way that you are currently using data at your school. Then Discuss: What conditions are most important for your staff to successfully engage in data analysis? 6 Three Supports to Using Data • Three factors must be in place to use data to improve instruction: 1. Use of a well-defined school improvement process (Armstrong & Anthes, 2001; Boudett, City, & Murnane, 2005; Holcomb, 2001; Love, Terc, & Regional Alliance for Mathematics and Science Education Reform, 2002) 2. Cultural shift to a learning organization, in which instruction is continually examined and improved (Boudett, 2007) 3. Reform must take place close to the classroom (Holcomb, 2001; Garvin, Edmondson, & Gino, March 2008). 7 THE DATA WISE DECISION-MAKING CYCLE Dr. Kathryn Boudett Dr. Elizabeth City Dr. Richard Murnane Stoplight Protocol 3 Full Implementation 2 Partial Implementation 1 Little or No Implementation (Really happening) (Sort of happening) (Not happening) 1. Organize for Collaborative Work Adopt an improvement process (WISE plan) Form leadership team Make time for collaborative work Create meeting structures Set norms, acknowledge work styles Review data and initiatives Boudett, City, & Murnane (2005) 2. Build Assessment Literacy Review skills tested Study how results are reported Learn principles of responsible data use Boudett, City, & Murnane (2005) 3. Create a Data Overview Choose a focus Display the data Allow teachers to make sense of the data Analyze data, find the story, and paint a picture Boudett, City, & Murnane (2005) 4. Dig into Data Assess school indicators Examine a wide range of student data Use protocols to stick to the evidence Identify a “Learner-Centered Problem” Develop a shared understanding of the knowledge and skills students need Boudett, City, & Murnane (2005) 5. Examine Instruction Examine a wide range of instructional data Get clear about the purpose of observation Use protocols to stick to evidence Identify a “Problem of Practice” Boudett, City, & Murnane (2005) 6. Develop an Action Plan Brainstorm solutions to the learner-centered problem Research and select instructional strategies Develop a common vision for implementation Agree on what the plan will look like Write tasks for implementation Boudett, City, & Murnane (2005) 7. Plan to Assess Progress Set student learning goals Identify implementation indicators Identify student achievement data Boudett, City, & Murnane (2005) 8. Act and Assess Communicate the action plan clearly Implement the action plan Visit classrooms frequently Assess student progress Plan PD to meet ongoing needs that emerge Promote consistency rather than conformity Adjust the action plan Celebrate success! Boudett, City, & Murnane (2005) DATA WISE IN ACTION Case: Mount Baker Middle School 18 Step 1: Organize for Collaborative Work Leadership team formed Data carousel protocol Data selected Whole staff meeting Step 2: Build Assessment Literacy Data displayed on charts Staff writes statements telling what data says Identify strengths and concerns Ask additional questions about data Step 3: Create a Data Overview Started with global picture Narrowed it down Focused on flat Math scores Step 4: Dig into Student Data Why are math scores low? Area of Statistics is weak. Step 5: Examine Instruction Incorporating statistics across the curriculum would support student learning. Step 6: Develop Action Plan Math team collaborated with content teams Math mentors served as a resource Coaching Science, reading, and language arts integration with statistics Step 7: Plan to Assess Progress End-of-Year Summative Assessment Anecdotal stories of student learning Step 8: Act and Assess -10 to -5 first year Let’s try it again! WORKING WITH DATA 27 The Data Pyramid 28 Phases of Data Acceptance When Adopting a Data-Driven Decision Making Model Challenging the Test “Question #3 is poorly worded.” “Answer ‘b’ is a trick answer.” “The students made silly mistakes.” Feeling inadequate or distrustful “How can two questions establish mastery?” “We don’t teach it in this format.” Confusion, overload “This is too much!” “How can I really use all of this?” TIP: Less is more. If teachers“Students take one action onproblems, one so do poorly on word Analytical but surface do more word problems.” standard from the data – theywe’ll begin to see the impact it “We need more reading.” can make and become bought-in. “The wrong answers show that students Looking for causes Changing teaching practice and improving student learning 29 can’t tell the difference between a summary and a theme.” “I need to write lesson plans for reteaching that differentiate between the different needs of my student groups.” Source: “Using Test Score Data to Focus Instruction” by Susan Trimble, Anne Gay, and Jan Matthews in Middle School Journal, March 2005 STEP 3- CREATE DATA OVERVIEW 30 Using Protocols with Data • The Comfort Zone is a place where we feel at ease, with no tension, have a good grip on the topic, and know how to navigate occasional rough spots with ease. • The Risk Zone is the most fertile place for learning. It is where people do not know everything, or sometimes not know anything at all, but are willing to take some risks. • Generally it is not a good idea to work from either your own Danger Zone or anyone else’s. That area is so full of defenses, fears, red-lights, desire for escape, etc., that it requires too much energy and time to accomplish anything. National School Reform Faculty The ORID Protocol The O questions identify objective facts. The key question is: What do we know about this? What we want are statements starting with terms like ‘I see… ‘, ‘There is evidence for… ‘,. These are documented but not analyzed. The I questions have to do with meaning. The key question is: What does it mean for me/you/the organization, etc? Interpretive questions might include ‘What if…?’, ‘What would it mean… ?’, ‘What would that do… ?’ and so on. This is the analytical phase 32 The R questions are about how people feel about the topic. They are about subjective perceptions. The key question is: How do we feel about this? The phase is one of identifying feelings and not of analyzing them. This is the stage at which a decision is produced. The key question is: What are we going to do? Focusing on the future, questions might include, ‘What would be the best course of action? “, ‘What would be achievable, positive outcomes? ‘ Whole Group Debrief •What did you learn by answering these questions collaboratively? •Would you use these questions with a grade level team or a school improvement team? Why or why not? •Based on your understanding of the data, what would you identify as your focus area for improvement? BREAK TIME STEP 4- DIG INTO STUDENT DATA Writing the Learner-Centered Problem Select a problem to focus on through which we can make a meaningful difference in student learning for many students. Rebecca A. Thessin, Ed.D., The George Washington University Sample Learner-Centered Problems Elementary •Students have difficulty drawing text-based inferences when reading. •Students are unable to read increasingly more sophisticated levels of text. Middle School •Students struggle when asked to solve multi-step math problems. •Students have difficulty isolating key scientific concepts when reading non-fiction text. Rebecca A. Thessin, Ed.D., The George Washington University 37 High School •Students have difficulty taking a critical stance, explaining the reasoning behind their stance, and supporting their stance with evidence. •Students have trouble applying algebraic reasoning skills, particularly on openended questions. Writing a Learner-Centered Problem On your template, please draft a learner-centered problem for the focus area you selected. Then be ready to share what you wrote. Asking Good Questions Helps to Triangulate the Problem Your goal is to “triangulate” the problem to: • Define the problem with greater accuracy. • Move beyond generalizations. • Avoid making inappropriate inferences from only one source of data. Trent Kaufman, 2006. What Else Do We Need To Know? Once we have drafted a learner-centered problem, we need to gather more information to help us to write an instructional goal. With your table, record 5-8 more questions that you have about students’ performance in this area. Rebecca A. Thessin, Ed.D., The George Washington University STEP 5- EXAMINE INSTRUCTION 41 Writing an Instructional Goal Instructional goal = a change in teacher practice, designed to improve student learning in a specific way. Or another way to look at it . . . An instructional goal is like a “testable hypothesis” –a change in teacher practice that, if made, then it is likely to lead to improved student outcomes. Rebecca A. Thessin, Ed.D., The George Washington University Defining the Problem and Goal • • • Learner-Centered Problem: With what are students struggling? This statement might begin with “Students are having difficulty . . .” Issue of Teacher Practice: What do teachers want to do better? This statement might begin with “Teachers want to . . .” Instructional Goal: What is the instructional practice that teachers will implement to change their instruction? This statement might begin with “Teachers will . . .” Rebecca A. Thessin, Ed.D., The George Washington University 43 Sample Problems and Goals • To write an instructional goal, let’s go back to start from a learner-centered problem we can work with for practice: • Level: Middle • Scope: Language Arts and Social Studies • • Learner-Centered Problem: Students are having difficulty providing evidence from texts that they read to support a particular point of view. Rebecca A. Thessin, Ed.D., The George Washington University Sample Problems and Goals cont. • • • • Issue of Teacher Practice: Teachers want to adopt common teaching strategies and assessment tools to provide students with more opportunities to learn how to pull evidence from texts to support an argument in language arts and social studies classes. Instructional Goal: Teachers will engage students in guided persuasive writing activities/tasks that involve selecting evidence and providing a platform for debate in order to develop the habit of substantiating opinion with evidence. Rebecca A. Thessin, Ed.D., The George Washington University 45 Draft an Instructional Goal On your template, please draft: • • 46 an Issue of Teacher Practice an Instructional Goal for your learner-centered problem. Evaluate Your Instructional Goal • The goal your team and your school selects to focus on should meet the following criteria: -Is common to many students -If solved, would meet your larger goals for students -Is supported by multiple sources of data and/or research-based practices -Is specifically focused on knowledge, skills, and learning dispositions that you want students to have -You have an understanding of the reasons behind students’ low performance in this particular area • Boudett, K., City, E., Murnane, R. (2006. Write a SMART Goal Specific • Well defined • Clear to anyone that has a basic knowledge of the project Measurable • Know if the goal is obtainable and how far away completion is • Know when it has been achieved Attainable • Agreement with all the stakeholders what the goals should be Results Oriented • Within the availability of resources, knowledge and time Time Bound • Enough time to achieve the goal • Not too much time, which can affect project performance Examples of SMART Goals Students will meet or exceed the district writing expectations as measured by the six-traits writing sample scoring. SMART GOAL: During the 2006-07 school year, the number of first through fifth grade regular education students at Sample School improving their writing skills in targeted traits will increase 5% at each grade level (see chart below) as measured by the Six-Traits scoring rubric monthly grade level assessments. 95% of all 11th graders receiving f/r lunch will score above the 40th NPR on ITEDs. (increase from 80%) During the 2006-07 school year, proficient 11th grade students in the f/r subgroup group (as indicated by the ITED math computation) at Sample School will increase by 15% as measured by the ITED math computation subtest. STEP 6- DEVELOP ACTION PLAN How Do We Plan the Change? Once you have examined the data and drafted an instructional goal, you will select strategies to use in the classroom to help you to reach your goal and determine the supports needed for full implementation. Discuss: How would you determine what new strategies to use to reach the goal your school or team establishes? What types of support might teachers need? 51 STEP 7- PLAN TO ASSESS PROGRESS 52 Plan to Assess Progress We won’t know if this change will lead to improved student outcomes until we actually implement the change! How will the school leader, coach, or teacher leader know which improvement efforts are yielding positive outcomes? Gather More Data! Cause and Effect Data Cause Data: Information based on actions of the adults in the system Effect Data: Student achievement results from various measurements Reeves, D. Standards Alignment: Tools and Processes. Leadership and Learning Matrix STEP 8- ACT AND ASSESS Act and Assess Using Cause and Effect Data Assessing Cause Data Walkthroughs Observations Focus Walks Instructional Rounds Peer Observations Assessing Effect Data Examining student work Common formative assessments Benchmark tests Universal screener Return to the Beginning of the Process ALIGNMENT TO WISE TOOL 59 Data Wise = The WISE Plan 2 4 2 4 1 5 5 7 8 5 6 3 NEXT STEPS PLCs are formed to All principals attend Data Wise training and receive training Staff conducts support implementation materials. research to efforts of identified determine exemplary practices. exemplary practices for identified Principals select appropriate Data deficiency. Wise training materials to familiarize staffs with 8-step Each PLC completes the model. YES PDSA PLC planning form and begins implementation efforts. Staffs engage in analysis of data to identify root cause(s) of deficiencies. Can root cause be addressed by staff? PLCs study implementation results and measure progress NO toward goals. Standardize or refine practices. Planning PLCs 63 EVALUATIONS