Iowa Support System for Schools and Districts in Need of Assistance Phase II: Diagnosis AEA Month Day, Year Today, you will . . . 2 Increase your understanding of the Diagnosis Phase. Access several tools for diagnosis. Experience working with a gap analysis. Work through a format for writing “if…then” statements and a theory of change that set the stage for action plan goals. Iowa Support System for Schools and Districts in Need of Assistance ©2009 The Phases of the Support for Schools and Districts Identified in Need of Assistance (SINA & DINA) Phase I – The Audit Phase 3 II – The Diagnosis Phase III – The Design Phase IV – The Implementation (and Monitoring) Phase V – Monitoring and Assessment Iowa Support System for Schools and Districts in Need of Assistance ©2009 Guiding Principles for Iowa Support Teams 4 Build on school/building strengths Encourage shared school leadership Analyze data to make decisions Focus on teaching and learning through a systems approach Integrate quality professional development practices Promote alignment of all components of the system Integrate existing state initiatives Reflect collaborative efforts throughout the structure Build capacity and accountability at all levels Implement evidence-based strategies Iowa Support System for Schools and Districts in Need of Assistance ©2009 Our Norms for Today Share experiences to enrich others. Ask questions. Learn by doing. Set aside any preconceived notions about gap analysis and diagnosis. Apply to your own work. Think S-A-L-S-A! Adapted from Training Manual for Assessing Impact: Evaluating Staff Development by Joellen Killion, ©2003 5 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Guiding Questions for Facilitators of Diagnosis 6 What is the severity of the problem? Does the audit profile accurately reflect all of your building and/or district data? What level of involvement have staff members had in the analysis of building and/or district data? Who is involved in/leads the school/district improvement process? What role will central office personnel play in this process? How is professional development organized in this building/district? Is the focus on factors that we can control? Iowa Support System for Schools and Districts in Need of Assistance ©2009 Guiding Questions for Facilitators of Diagnosis 7 How well are curriculum, instruction, and assessment aligned? What opportunities exist for the staff to implement the Iowa Professional Development Model (IPDM)? What are current initiatives at the building and/or district level? What is the history of the staff’s level of support in the school/district improvement process? What are the potential barriers for initiating and implementing needed changes in this building/district? Iowa Support System for Schools and Districts in Need of Assistance ©2009 Selection of Building/District Leadership Team Be sure to include: 8 Building principal(s) as involved participants in buildings and central office personnel in districts Content specialist(s) Special need areas (e.g., ELL, Special Education) A cross representation by grade level, experience, and expertise in buildings and by building, experience, and expertise in the district Those with credibility with colleagues Those with a balance of skills including but not limited to the following: leadership, organization, facilitation, data analysis, presentation Iowa Support System for Schools and Districts in Need of Assistance ©2009 Building/District Leadership Team Consider the following: Establish a common understanding of purpose/function of the team Clarify role, function, and responsibility of team members Define ground rules/norms for working together Develop team operations (e.g., agenda, minutes, time commitment, logistics) Identify supports available (e.g., central office, AEA staff, other resources) 9 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Definition of Diagnosis As part of the support framework for Schools and Districts in Need of Assistance (SINA & DINA), the Diagnosis Phase reviews prioritized areas from the audit summary. Through a comparison of the current reality with the desired state, a gap analysis is completed. The root causes that are contributing to the area(s) for further study are identified. If/then statements based on possible solutions and a theory of change are created as a final step to set the stage for the goals or action plan steps in the design phase. All of this is summarized on the Diagnosis Worksheet. 10 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Six Steps of Diagnosis 1. Review and refine current reality 2. Determine the desired state 3. Complete a gap analysis 4. Determine the root cause(s) 5. Identify and prioritize solutions (Instructional/Professional Development solutions must be research-based.) 6. Create “if…then”* statement(s) and a theory of change *Some districts/schools call this “determining opportunities for improvement” or “hypothesis/prediction statements.” 11 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Step 1: Review and Refine Current Reality 12 Review the audit. Review student achievement data. Request additional data as needed. Organize and analyze multiple sources/points of data for deeper understanding. Request additional data as needed. Document prioritized areas from the audit and the refined current reality on the Diagnosis Worksheet. Iowa Support System for Schools and Districts in Need of Assistance ©2009 Data To Consider For Step 1: Review and Refine Current Reality 13 Student Data – achievement, perception, demographic, participation, and enrollment. Professional Practices Data – strategies, certification, professional development, teacher perception, leadership. Programs and Structures Data – academic programs, support, curriculum, school / districtenvironment. Family and Community Partnership Data – parent involvement, parent perception, community involvement, community perception. Iowa Support System for Schools and Districts in Need of Assistance ©2009 Tools to Consider for Step 1: Review and Refine Current Reality 14 Diagnosis Worksheet Questions for Areas of Concern/Red Flags Analyze and Report Data Response Sheet - Iowa Professional Development Model (adapted) Item Analysis Summary - Iowa Professional Development Model (adapted) Performance Profile Analysis – Iowa Testing Services (adapted) Questioning Your Data Questioning Your Data – Genny Eric High School Pareto Diagram Questioning Your Data – Hilltop Central Iowa Support System for Schools and Districts in Need of Assistance ©2009 Activity for Step 1: Review and Refine Current Reality Review of the Audit Profile – – – 15 Use the audit profile and the guidance for reviewing the building/district profile to determine strengths and challenges revealed in the profile. Focus your discussion on those areas identified as areas for further study. Create/Revise “the current reality” based on information provided. Iowa Support System for Schools and Districts in Need of Assistance ©2009 Activity for Step 1: Review and Refine Current Reality Data Analysis – – – – 16 Review the data using the tool marked “Analyze & Report Data Response Sheet” (Iowa Professional Development Model) Discuss in small groups the findings Share out a summary of the discussion and the findings to the whole group Summarize the “current reality” on the Diagnosis Worksheet for Westlake Middle School Iowa Support System for Schools and Districts in Need of Assistance ©2009 Activity for Step 1: Review and Refine Current Reality Data Analysis – Another Activity – Review the data using the tool marked “Questioning Your Data – Genny Eric High School” 17 What do YOU SEE in these data? (Individual) What do WE SEE in these data? (Small Group) What are the “big six” that your group sees in these data? (Small Group) Share with large group Iowa Support System for Schools and Districts in Need of Assistance ©2009 Activity for Step 1: Review and Refine Current Reality Data Analysis – Optional Activity – Review the data using the tool marked “Questioning Your Data – Hilltop Central” What do I SEE in these data? What questions do I have about what I see? 18 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Using Data as a Lever for Change 19 Data can uncover problems that might otherwise remain invisible. Data can convince people of the need for change Data can confirm or discredit assumptions about students and school practices Data can get to the root cause of problems, pinpoint areas where change is most needed, and guide resource allocations. Data can help schools evaluate program effectiveness and keep the focus on student learning results. Iowa Support System for Schools and Districts in Need of Assistance ©2009 Using Data as a Lever for Change 20 Data can provide the feedback teachers and administrators need to keep going and stay on course. Data can prevent over-reliance on standardized test scores. Data can prevent one-size-fits-all and quick-fix solutions. Data can give schools the ability to respond to accountability questions. Data can help build a culture of inquiry and continuous improvement. Iowa Support System for Schools and Districts in Need of Assistance ©2009 Questions about Step 1 of the Diagnosis Phase… 21 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Six Steps of Diagnosis 1. Review and refine current reality 2. Determine the desired 3. Complete a gap analysis state 4. Determine the root cause(s) 5. Identify and prioritize solutions (Instructional/Professional Development solutions must be research-based.) 6. Create “if…then”* statement(s) and/or theory of change 22 *Some districts call this “determining opportunities for improvement” or “hypothesis/prediction statements” Iowa Support System for Schools and Districts in Need of Assistance ©2009 Definition of Desired State 23 Desired state is the picture of your school/district that you wish to reach after the completion of your school/district improvement process (your action plan). The state we have all agreed to reach and be held accountable to is . . . Determined collaboratively Based on research, best practices, and promising practices Iowa Support System for Schools “Doable” in the near future and Districts in Need of Assistance ©2009 Step 2: Determine the Desired State 24 Consult expert content specialist(s) Review research Develop the picture of your desired state Document the desired state on the Diagnosis Worksheet Iowa Support System for Schools and Districts in Need of Assistance ©2009 Step 2: Determine the Desired State Examples of Research to Review Iowa Content Networks Expert Content Specialists United States Department of Education Clearing House http://www.whatworks.ed.gov/ 25 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Tools to Consider for Step 2: Determine the Desired State 26 Structured Response Sheet – Every Child Reads (adapted) Content Network Tool for Reviewing Research Diagnosis Worksheet Iowa Support System for Schools and Districts in Need of Assistance ©2009 Activity for Step 2: Determine the Desired State 27 Provide the research/evidence to identify the expectations of the identified areas of concern. Develop common expectations based on research for all areas of concern. Document the desired state. Iowa Support System for Schools and Districts in Need of Assistance ©2009 Questions about Step 2 of the Diagnosis Phase… 28 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Six Steps of Diagnosis 1. Review and refine current reality 2. Determine the desired state 3. Complete a gap analysis 4. Determine the root cause(s) 5. Identify and prioritize solutions (Instructional/Professional Development solutions must be research-based.) 6. Create “if…then”* statement(s) and/or theory of change. *Some districts call this “determining opportunities for improvement” or “hypothesis/prediction statements” 29 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Step 3: Complete a Gap Analysis 30 Summarize Current Reality Summarize Desired State Identify deficits or the gap between desired state and current reality Identify themes and patterns across deficit areas/gaps, using guiding questions Document the gap analysis on the Diagnosis Worksheet Iowa Support System for Schools and Districts in Need of Assistance ©2009 Guiding Questions for Step 3: Complete a Gap Analysis - Identify themes and patterns across deficit areas/gaps, using guiding questions What are the relative strengths? What are the relative weaknesses? What are the trends, themes, and patterns? Consider the triangulation of data to verify the gap analysis. ITBS/ITED Teacher Observation 31 Diagnostic Measure Iowa Support System for Schools and Districts in Need of Assistance ©2009 Tools to Consider for Step 3: Complete a Gap Analysis 32 Gap Identification Worksheet Forced Field Analysis Planning Template Fishbone Graphic Organizer Diagnosis Worksheet Iowa Support System for Schools and Districts in Need of Assistance ©2009 Activity for Step 3: Complete a Gap Analysis For this activity we will use the Force Field Analysis to help us process the determination of the gap. 33 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Questions about Step 3 of the Diagnosis Phase… 34 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Six Steps of Diagnosis 1. Review and refine current reality 2. Determine the desired state 3. Complete a gap analysis 4. Determine the root cause(s) 5. Identify and prioritize solutions (Instructional/Professional Development solutions must be research-based.) 6. Create “if…then”* statement(s) and/or theory of change. *Some districts call this “determining opportunities for improvement” or “hypothesis/prediction statements” 35 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Definition of Diagnosis 36 As part of the support framework for Schools and Districts In Need of Assistance (SINA & DINA), the diagnosis phase reviews prioritized areas from the audit summary. Through a comparison of the current reality with the desired state, a gap analysis is completed. The root causes that are contributing to the area(s) for further study are identified. If/then statements and or a theory of change based on possible solutions are created as a final step to set the stage for the goals or action plan steps in the design phase. Iowa Support System for Schools and Districts in Need of Assistance ©2009 Definition of Root Cause The deepest underlying cause, or causes, of positive or negative symptoms within the area of concern that, if dissolved, would result in elimination, or substantial reduction, of the symptom. Source: Root Cause Analysis – School Leader’s Guide to Using Data to Dissolve Problems – Paul G. Preuss 37 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Root Cause Patterns 38 Chained or Linear Combined Contributing Iowa Support System for Schools and Districts in Need of Assistance ©2009 Root Cause Pattern: Chained A B C = Result X Chained Causes: Interrupt the chain and eliminate the result 39 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Root Cause Patterns: Combined A B Result Y C Combined Causes: The common presence of multiple factors brings about the result. Eliminate any one factor and the result disappears. 40 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Root Cause Patterns: Contributing A B Result Z C Contributing Causes: Multiple factors each contribute to a certain percentage of the result. To totally eliminate the result, all contributing causes must be removed. 41 Iowa Support System for Schools and Districts in Need of Assistance ©2009 When Is a Cause a Root Cause? 1. Would the problem have occurred if the cause had not been present? 2. Will the problem reoccur as the result of the same cause if the cause is corrected or dissolved? 42 If no, then it is a root cause! If yes, then it is a contributing cause. If no, then it is a root cause! If yes, then it is a contributing cause. Iowa Support System for Schools and Districts in Need of Assistance ©2009 When Is a Cause a Root Cause? 3. Will correction or dissolution of the cause lead to similar events? 43 If no, then it is a root cause! If yes, then it is a contributing cause. Iowa Support System for Schools and Districts in Need of Assistance ©2009 When Is a Cause a Root Cause? Other indicators that you have found the root cause: You run into a dead end asking what caused the proposed root cause. Everyone agrees this is a root cause. The cause is logical, makes sense, and provides clarity to the problem. The cause is something that you can influence and control. If the cause is dissolved, there is realistic hope that the problem can be reduced or prevented in the future. 44 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Insights on Root Causes Look for triangulation of data ITBS/ITED Teacher Observation 45 Diagnostic Data Aim to dissolve a problem, not just the symptom Think proactively, as well as reactively Realize some causes are contributing, not root Iowa Support System for Schools and Districts in Need of Assistance ©2009 Step 4: Determine the Root Cause(s) 46 Given what you identified in your gap analysis, determine the critical cause/effect relationships from your prioritized areas Document the root cause(s) on the Diagnosis Worksheet Iowa Support System for Schools and Districts in Need of Assistance ©2009 Tools for Step 4: Determine the Root Cause(s) 47 Diagnostic Tree Pareto Diagram Fishbone - Cause and Effect Diagram Relations Diagram Five Whys – Iowa Evaluator Approval Training Program (adapted) Diagnosis Worksheet Iowa Support System for Schools and Districts in Need of Assistance ©2009 Activity for Step 4 : Determine the Root Cause(s) 48 Process the cause/ effect relationships within the prioritized areas to identify the root causes. Using the Diagnostic Tree, identify possible causes in each of the domains. Review the evidence for supporting/not supporting these causes. Determine those that are evidence-based and/or “doable.” Iowa Support System for Schools and Districts in Need of Assistance ©2009 Questions about Step 4 of the Diagnosis Phase… 49 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Six Steps of Diagnosis 1. Review and refine current reality 2. Determine the desired state 3. Complete a gap analysis 4. Determine the root cause(s) 5. Identify and prioritize solutions (Instructional/Professional Development solutions must be research-based.) 6. Create “if…then”* statement(s) and/or theory of change. *Some districts call this “determining opportunities for improvement” or “hypothesis/prediction statements” 50 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Step 5: Identify and Prioritize Solutions Sources to Consider: •Consult expert content specialist(s) •Review research •Request information from the Iowa Support Team facilitator associated with successful SINA schools •Utilize Iowa Content Networks •Access USDE Clearinghouse 51 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Step 5: Identify and Prioritize Solutions 52 Identify possible solutions correlated to root causes if the solution is related to instruction or professional development, it should be research based Prioritize based on research (e.g., greatest impact on student achievement, response to intervention) Document the solution(s) on the Diagnosis Worksheet Iowa Support System for Schools and Districts in Need of Assistance ©2009 Step 5: Identify and Prioritize Solutions Possible Solution Categories Academic – – Quality Educator – – – – Professional Development Leadership Teaming and Collaboration Opportunities Culture & Climate Building/District System – – – – 53 Alignment of Curriculum, Instruction & Assessment with Standards Content & Strategy Selection Instructional and Operational Policies & Practices Resource Allocation (e.g., time for the content area, dollars for materials) Family & Community Communication Iowa Support System for Schools and Districts in Need of Assistance ©2009 Step 5: Identify and Prioritize Solutions Guidelines for Prioritizing Solutions: Will the implementation of this solution address our school’s/district’s area(s) of concern and underlying reasons for them? Will the proposed solution have a great impact on students’ learning? Are resources (i.e., people, time, money) available to implement this solution? 54 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Step 5: Identify and Prioritize Solutions Guidelines for Prioritizing Solutions: Do the advantages outweigh the disadvantages if we implement this solution? Could this solution be modified without impacting the effectiveness to best meet the needs/interests of our students, their families, our teachers, our classrooms, our school, and our district? Is everyone who is part of the challenge also a part of the solution? 55 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Tools for Step 5: Identify and Prioritize Solutions 56 Nominal Group Technique Diagnosis Worksheet Iowa Support System for Schools and Districts in Need of Assistance ©2009 Activity for Step 5: Identify and Prioritize Solutions For this activity the Nominal Group Technique will be used to prioritize the solutions. Questions to ask yourself about each solution: – – – – 57 Do we have evidence to support this solution? Is there research that shows this solution will have an impact? Is this an area over which we can have control and can influence change (e.g., proximal vs distal)? Do we have the resources to implement this solution? Is this solution sustainable? Iowa Support System for Schools and Districts in Need of Assistance ©2009 Activity for Step 5: Identify and Prioritize Solutions Other questions you might consider for each solution: 58 Will the implementation of this solution address our school’s/district’s areas of concern and underlying reasons for them? Will the proposed solution have a great impact on students’ learning? Are resources (i.e., people, time, money) available to implement this solution? Do the advantages outweigh the disadvantages if we implement this solution? Could this solution be modified without impacting the effectiveness to best meet the needs/interests of our students, their families, our teachers, our classrooms, our school, and our district? Is everyone who is part of the challenge also a part of the solution? Iowa Support System for Schools and Districts in Need of Assistance ©2009 Questions about Step 5 of the Diagnosis Phase… 59 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Six Steps of Diagnosis 1. Review and refine current reality 2. Determine the desired state 3. Complete a gap analysis 4. Determine the root cause(s) 5. Identify and prioritize solutions (Instructional/Professional Development solutions must be research-based.) 6. Create “if…then”* statement(s) and/or theory of change *Some districts call this “determining opportunities for improvement” or “hypothesis/prediction statements.” Iowa Support System for Schools 60 and Districts in Need of Assistance ©2009 Step 6: Create “if…then” statement(s) Based on prioritized solution(s) write “if…then” statement(s) “If…then” statements may either result in student achievement goal statements or action plan steps toward a student achievement goal. Document “if…then” statements on the Diagnosis Worksheet 61 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Step 6: Create “if…then” statement(s) Example: Possible Root Cause: Students have inadequate instructional time to master identified math skills. Possible solution: Classroom schedules are adjusted to accommodate increased instructional time in math. This may be accompanied by a district/building expectation for a specific amount of instructional time in math. If…then statement: If classroom schedules are adjusted to increase the amount of instructional time in math, then student achievement in math should improve. 62 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Step 6: Create “if…then” statement(s) Example: Root cause: Student achievement in reading comprehension is low because professional development lacks specific reading strategies targeted to student needs. Solution: Provide professional development on specific reading strategies targeted to student needs. If…then statement: If all teachers are provided professional development based on research-based reading strategies targeted to student needs, then student achievement in reading will improve. 63 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Step 6: Create “if…then” statement(s) Example: Root Cause: The articulated building/district assessment plan lacks formative assessments to guide math instruction. Solution: Identify/develop required formative assessments that align to the math curriculum and regularly guide instruction. If…then statement: If the required formative assessments in math align with curriculum and resulting information is used to change instructional practices, then student achievement in math will improve. 64 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Tools for Step 6: Create “if…then” statement(s) 65 If . . . Then Worksheet ORID Questioning PowerPoint – Iowa Evaluator Approval Training Program (adapted) Diagnosis Worksheet Iowa Support System for Schools and Districts in Need of Assistance ©2009 Activity for Step 6: Create “if…then” statement(s) For this activity, create “If…Then” statements to help the group process the relationships between the root causes and the prioritized solutions. 66 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Questions about Step 6 of the Diagnosis Phase… “If . . . Then Statements” 67 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Step 6: Complete a Theory of Change A Theory of Change . . . Identifies the components of a program (what the program does) Specifies the relationship among the components to explain how the change occurs (sequence of actions) Delineates the underlying assumptions upon which the program is based 68 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Tools for Step 6: Create a Theory of Change 69 Creating a Theory of Change – Blue and Green Card Activity Diagnosis Worksheet Iowa Support System for Schools and Districts in Need of Assistance ©2009 Activity for Step 6: Theory of Change Activity To further your understanding of the theory of change, sequence the blue and green cards to show a possible causal link among the components of the Youth Development Program. (Hint: Identify the starting and ending points first, then sequence all the other cards between those two.) 70 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Theory of Change Activity How do the blue and green cards differ? – – 71 The blue cards represent _____________. The green cards represent ____________. Iowa Support System for Schools and Districts in Need of Assistance ©2009 What process do you use? In collaboration with stakeholders: – Affinity process – Flow chart – Brainstorm 72 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Activity: Creating Your Own Theory of Change 73 Create a theory of change for Westlake Middle School. When you are satisfied, map the actions on the large chart paper. Post your chart. Iowa Support System for Schools and Districts in Need of Assistance ©2009 Assessing Evaluability Inputs/ Resources •Who? •What? •When? •Where? Activities / Processes Initial Experiences •Coaching •Follow Up •Monitoring Initial Outcomes Teacher Intermediate Outcomes Admin. Teacher Admin. •Knowledge •Attitudes •Skills •Aspirations •Behaviors Intended Results SMART Goals: Increased Student Achievement •Standards •Indicators •Data Points Backmapping Model Theory Of Change 74 Assumptions Iowa Support System for Schools and Districts in Need of Assistance ©2009 Activity: Assumptions Card Stack & Shuffle Write down assumptions you have about your theory of change. – – 75 One assumption per card Assumptions can be written for other people’s theory of change Cards are stacked by separate theory of change and elements. As a table group, cards are read aloud and placed “on the table.” Discuss your findings. Iowa Support System for Schools and Districts in Need of Assistance ©2009 Talk It Over What is the value of having a theory of change for a program? 76 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Questions about theory of change … 77 Iowa Support System for Schools and Districts in Need of Assistance ©2009 Six Steps of Diagnosis 1. Review and refine current reality 2. Determine the desired state 3. Complete a gap analysis 4. Determine the root cause(s) 5. Identify and prioritize solutions (Instructional/Professional Development solutions must be research-based.) 6. Create “if…then”* statement(s) 78 *Some districts call this “determining opportunities for improvement” or “hypothesis/prediction statements” Iowa Support System for Schools and Districts in Need of Assistance ©2009 Questions about the Diagnosis Phase… 79 Iowa Support System for Schools and Districts in Need of Assistance ©2009 End of Diagnosis Training Slides Proceed to Design Phase