Leadership for Effective Implementation and Sustainability of PBIS Kent McIntosh University of Oregon Goals for this Evening 1. 2. 3. 4. Describe the concepts of implementation science and sustainability Share the factors related to promoting sustainability of PBIS in schools Provide activities to use in your schools for sustaining effective practices…as soon as tomorrow Provide time to build an action plan Handouts: http://www.pbis.org Support for these projects: IES: NCSER (R324A120278) OSEP: TA Center on PBIS (H326S03002) Social Sciences and Humanities Council of Canada (SRG F09-05052) Hampton Endowment Fund (J07-0038) Investigating your Systems of Support CSI Maps RESPONSE TO INTERVENTION AND THE THREE TIER MODEL 1-5% 5-15% Tier 3 - Supports for a Few Tier 2 - Supports for Some 80-90% Tier 1 - Supports for All Three Tiered Model of Student Supports (from Susan Barrett) get these tiers of support These students + in order to increase student success = The goal of the tiers is student success, not labeling CSI Maps C = Core (Universal) S = Strategic (Targeted) I = Intensive (Intensive) Write down the support provided at each tier (strategies, programs, interventions) Write down how your school identifies what level of support students need handout RESPONSE TO INTERVENTION AND THE THREE TIER MODEL CSI Maps C = Core (Universal) S = Strategic (Targeted) I = Intensive (Intensive) Write down the support provided at each tier (strategies, programs, interventions) Write down how your school identifies what level of support students need CSI Maps: Interpretation 1. Identify GAPS in: Support 2. Assessment 1. 2. Consider priorities for filling gaps Discussion What did you learn? What surprised you? Going Deeper: Outcomes Next, put a check mark next to the practices/interventions/strategies for which: You collect student outcomes data Student outcomes data show improvement Discussion What did you learn? What surprised you? What is… fidelity of implementation? The extent to which the critical features of the practice (e.g., PBS) are implemented as intended Why assess it? Helps us improve outcomes for students Helps team target next steps and areas for improvement Use of Fidelity Data “Fidelity of implementation of academic and behaviour interventions is documented before measuring effects and assigning worth to them or blaming children (and teachers) for continuing failure.” (Algozzine, Putnam, and Horner, 2012) Good fidelity of implementation tools… 1. 2. 3. Identify the critical features of effective practices Show the degree to which our agreements and values are visible in our daily practice Should be used to celebrate successes and continual growth as opposed to magnifying shortcomings (adapted from Miller, 2010) Fidelity of Implementation Considerations How do you track that you have implemented well? Now, circle practices/interventions for which you collect fidelity data Discussion What did you learn? What surprised you? Report Out What happens next? Name 2 action plan steps Action Plan handout The WHAT and the HOW We just took inventory of the WHAT and organized it by tiers WHAT = the practices, programs or strategies themselves Now let’s talk about the HOW HOW = putting them in place An Introduction to Implementation Science Definitions What do we mean by implementation? “A specified set of purposeful activities at the practice, program, and system level designed to put into place a program or intervention of known dimensions with fidelity.” What is implementation science? “The study of how to promote implementation.” Common perception Implementation Equation Effective Interventions The “WHAT” Effective Implementation The “HOW” Positive Outcomes for Students (and adults) “Students cannot benefit from interventions they do not experience.” Dean Fixsen The National Implementation Research Network Monograph The State of the Art Synthesis of implementation research from 1970-2004 Program development and replication data Qualitative study of programs ctndisseminationlibrary.org/PDF/nirnmonograph.pdf Lessons from Failed Efforts (Fixsen, 2013) Longitudinal Studies of a Variety of Comprehensive School Reforms Evidencebased Interventions Actual Supports Years 1-3 Outcomes Years 4-5 Every Teacher Trained Fewer than 50% of the teachers received some training Fewer than 10% of the schools used the CSR as intended Every Teacher Continually Supported Fewer than 25% of those teachers received support Vast majority of students did not benefit Aladjem & Borman, 2006; Vernez, Karam, Mariano, & DeMartini, 2006 Quality Implementation LETTING IT HAPPEN Teachers are accountable vs. MAKING IT HAPPEN Implementation teams are accountable Based on Greenhalgh, Robert, MacFarlane, Bate, & Kyriakidou, 2004 How do we MAKE IT HAPPEN? 1. We pick what’s most likely to work Evidence-based 2. practices We organize our resources for success Multi-tiered systems of support Follow stages of implementation Exploration Installation Initial Implementation Full Implementation Sustainability Stages of Implementation (Fixsen et al., 2005) Exploration Installation Initial Implementation Full Implementation Sustainability Stages of Implementation Stage Exploration Primary Goal To determine whether the practice is a good match for the community, based on the needs and resources of the community and the demands and benefits of the practice Key Implementation Activities -Investigating the practice -Assessing needs and resources of the community and the demands and benefits of the practice -Developing an implementation plan for the chosen practice, including action items and a clear timeline Stages of Implementation Stage Installation Primary Goal To prepare and structure the resources and supports to build a solid framework for implementation Key Implementation Activities -Ensuring adequate funding and staffing -Developing mechanisms related to initial implementation and the day to day routines of the practice (e.g., referral forms, data management systems) -Training the school personnel to implement the practice Stages of Implementation Stage Initial Implementation Primary Goal To maintain the fidelity and principles of the practice throughout the initial resistance to change Key Implementation Activities -Implementing the practice with students -Structuring a support system for troubleshooting and professional development to maintain the fidelity of the practice and keep morale high Stages of Implementation Stage Full Implementation Primary Goal To continue to carry out the practice with high fidelity and monitor effectiveness as the practice shifts from a new initiative to the routine practice (i.e., the status quo) Key Implementation Activities -Monitoring fidelity with formal measures and comparing to fidelity criterion goals -Assessing practice effectiveness once fidelity criterion goals are met Stages of Implementation Stage Sustainability Primary Goal To respond to shifts in the community and in original support for the practice (e.g., staff turnover, loss of funding) and maintain the practice, with high fidelity, through these changes Key Implementation Activities -Continuing to monitor fidelity and effectiveness -Maintaining an awareness of the needs of the community as they change -Ensuring proper support and training in the practice as new staff enter the community What do we usually do well? What do we usually not do well? Exploration Installation Initial Implementation Full Implementation Sustainability What are the odds that a given school initiative will sustain? (Latham, 1988) Implementation Autopsies What specific experiences have you had with initiative abandonment? Get off the Train & Hope Train 5. Wait for new problem 1. React to Problem Behavior 4. Expect (Hope) for Implementation 2. Select and Add Practice 3. Hire Expert to Train Practice Back to your CSI Maps 1. 2. If you continued on the same or similar path, what will be the likely result? What would it look like in your “dream work world”? How do we MAKE IT HAPPEN? 1. We pick what’s most likely to work Evidence-based 2. practices We organize our resources for success Multi-tiered systems of support Follow stages of implementation 3. We provide support to the adults who are supporting the students Ongoing coaching District systems Exploration Installation Initial Implementation Full Implementation Sustainability Definition Sustainability Durable implementation of a practice at a level of fidelity that continues to produce valued outcomes (McIntosh et al., 2009) Barriers to Sustainability: The Three Cs Antecedent Behavior Consequence Ongoing Challenge Select Practices & Implement with Fidelity Student Outcomes Changes in Context Changes in Capacity Changes in Consequences - Lack of contextual fit - Loss of funding - New challenges exist - Attrition of key opersonnel - Diminished effectiveness odue to poor fidelity - Competing initiatives - Outcomes no longer operceived as important Why “admire the problem?” Identifying barriers (and plans to address them) is a critical step in systems-level change (Curtis, Castillo, & Cohen, 2008) Sometimes you need to avoid those obstacles! Sustainability Activity Barriers to Sustainability Worksheet Barriers to Sustainability Worksheet Risk Factors/ Barriers Changes in Context Changes in Capacity Changes in Consequences Plan to Address Barriers Barriers to Sustainability: The Three Cs Antecedent Behavior Consequence Ongoing Challenge Select Practices & Implement with Fidelity Student Outcomes Changes in Context Changes in Capacity Changes in Consequences - Lack of contextual fit - Loss of funding - New challenges exist - Attrition of key opersonnel - Diminished effectiveness odue to poor fidelity - Competing initiatives - Outcomes no longer operceived as important A Sustainable Community of Practice Please share: A barrier to sustainability (past or present) Your plan for addressing it (if you have one) Perceived Importance of Contextual Features for Sustainability of PBIS McIntosh, K., Predy, L., Upreti, G., Hume, A. E. & Mathews, S. (2014). Perceptions of contextual features related to implementation and sustainability of School-wide Positive Behavior Support. Journal of Positive behavior Interventions, 16, 29-41. 70 60 Number of Responses 50 40 30 20 10 0 Enablers of Sustainability Most important single perceived factor in sustainability? School Administrator Support Ok…what do we do when… A committed administrator moves on? 2. An administrator is opposed to PBIS? 1. Sustaining PBIS through Administrator Turnover (Strickland-Cohen, McIntosh, & Horner, 2014) School Team Maintain the PBIS handbook Document support among staff and stakeholders Collect and share outcomes data Meet with the new administrator District Team Build PBIS into written policy Build PBIS competencies into hiring criteria Develop district coaching capacity Research on Enhancing Principal Support (McIntosh, Kelm, & Canizal Delabra, in press) Interviews with 10 principals initially opposed or lukewarm to PBIS but are now champions Interview questions: What helped your active support for PBIS? 2. What hindered your active support for PBIS? 3. What would have made you support PBIS from the onset? 1. Helpful Strategies for Enhancing Principal Support (McIntosh, Kelm, & Canizal Delabra, in press) District Actions “Principal Academies” for new administrators Provide Basics of PBIS Role of administrators Provide coaching to schools Networking Arrange informal conversations with other principals supportive of PBIS Arrange site visits at nearby PBIS schools At the school Help school staff demonstrate support Feeling depressed about barriers? For schools already implementing PBIS, barriers are perceived as less important than facilitators (McIntosh et al., 2014) …and the presence of common barriers is only weakly related to sustained fidelity of implementation (Turri et al., under review) U.S. Schools using PBIS 21,278 August, 2015 25000 22500 20000 17500 15000 12500 10000 7500 5000 2500 0 00 01 02 03 04 05 06 07 08 09 10' 11' 12' 13' 14' 15' 16' 17' 18' Predicting Abandonment of PBIS (Nese et al., under review) Of 915 schools, what proportion of schools were still sustaining PBIS after 5 years? 93% In what year did schools abandon PBIS? % of Abandoning Schools 30% 25% 20% 15% 10% 5% 0% 1 2 3 4 5 Implementation Year 6 7 Four Principles for Sustaining PBIS Promote PRIORITY Ensure EFFECTIVENESS Increase EFFICIENCY Use data for CONTINUOUS REGENERATION What is PRIORITY? Importance in comparison to other practices Incorporation into core system components Connection to other initiatives Promoting PRIORITY Maximize visibility Present data to people with resources Describe effects of abandoning support for the practice Get into written policy Braid project with other initiatives Show how practice can lead to outcomes of new initiatives What is braiding? (Bohanon, Goodman, & McIntosh, 2009) Summary Statement Desired Alternative Typical Consequence Maintaining Consequence Setting Events Triggering Antecedents Behavior Presence of PBIS Coach Prevalence of Problem Behavior Implement PBIS Acceptable Alternative Reductions In Problem Behavior Social & Emotional Learning Restorative Practices Anti-Bullying Initiatives Academic Achievement School-based Mental Health Family Engagement Reducing Racial Inequities Steps in Braiding Initiatives 1. Identify shared, valued outcomes 2. Defend against activities that don’t help us meet those goals 3. What are our overall goals? No free lunches Find common structures (and language) that can be integrated Teams, data, professional development Sustainability Activity Braiding worksheet handout MTSS Initiative Alignment Worksheet 1. 2. 3. Identify a new initiative you want to implement (e.g., PBS, MTSS) and write it at the top Write all existing initiatives in the first column Complete the rows to find the value added by adopting the new initiative Make PBIS Efforts Public! Newsletters To Monthly/quarterly reports To parents school staff Formal presentations To school board To district administrators To PTA To community agencies and businesses Local news Newsletters What is EFFECTIVENESS? Extent to which the practice results in desired outcomes Effects must be attributed to the practice Positive Referrals vs. ODRs: FG Leary Fine Arts School, Chilliwack, BC 35000 400 350 30000 250 20000 200 15000 150 10000 100 5000 50 0 0 2006-07 2007-08 2008-09 2009-10 Office Discipline Referrals Positive Referral Slips 300 25000 ODR's Positives Ensuring EFFECTIVENESS Focus on FIDELITY OF IMPLEMENTATION Assess it regularly Use it to enhance what you already do Share data showing how fidelity is related to effects Measures to assess FIDELITY Team Implementation Checklist (TIC) PBIS Self-Assessment Survey (SAS) School-wide Evaluation Tool (SET) School-wide Benchmarks of Quality (BoQ) Benchmark of Advanced Tiers (BAT) Monitoring Advanced Tiers Tool (MATT) PBIS Tiered Fidelity Inventory (TFI) Available at: http://pbisapps.org What is EFFICIENCY? Relationship between continued effort and continued effectiveness Weighed against other potential practices Increasing EFFICIENCY Get it down on paper Lesson plans Schedules Agendas Focus on efficient team meetings Working Smarter Team Matrix Initiative, Committee Purpose Outcome Target Group Staff Involved SIP/SID/ etc Attendance Committee Character Education 1. Eliminate all initiatives that do NOT have a defined purpose and outcome measure Safety Committee 2. Combine initiatives that have the same outcome measure and same target group School Spirit Committee 3. Combine initiatives that have 75% of the same staff Discipline Committee 4. Eliminate initiatives that are not tied to School Improvement Goals DARE Committee PBIS Team handout Sample Team Matrix Initiative, Committee Purpose Outcome Target Group Staff Involved SIP/SID Attendance Committee Increase attendance Increase % of students attending daily All students Eric, Ellen, Marlee Goal #2 Character Education Improve character Improve character All students Marlee, J.S., Ellen Goal #3 Safety Committee Improve safety Predictable response to threat/crisis All students Has not met Goal #3 School Spirit Committee Enhance school spirit Improve morale All students Has not met Discipline Committee Improve behavior Decrease office referrals All students Ellen, Eric, Marlee, Otis DARE Committee Prevent drug use All students Don PBIS Team Implement 3tier model All students Eric, Ellen, Marlee, Otis, Emma Decrease ODRs, increase attendance, enhance academic engagement, improve grades Goal #3 Goal #2 Goal #3 What is CONTINUOUS REGENERATION? Collection of data to monitor fidelity, outcomes and context Adaptation over time while keeping critical features intact Ongoing investment in building local capacity Using data for CONTINUOUS REGENERATION Adjust practices for a changing environment Priority Effectiveness Efficiency Connect with a community of practice Create Communities of Practice Share fairs, networking sessions, district mini-conferences, web-based sharing Opportunities for school teams to: Celebrate successes Learn from peers Steal ideas Continue momentum Invite important stakeholders Legal Downloads http://www.pbismissouri.org http://www.pbisillinois.org http://bcpbs.wordpress.com http://pbismaryland.org http://www.cenmi.org/miblsi http://www.modelprogram.com/ http://www.PBISmn.org/ http://www.PBISvideos.org/ Using Data for Decision Making Sifton Elementary, Vancouver, WA 60 50 Total ODRs 40 Playground 30 Classroom 20 10 0 October November Sifton Playground Challenge Using Data for Decision Making Sifton Elementary, Vancouver, WA 60 50 Total ODRs 40 Playground 30 Classroom 20 10 0 October November Using Data for Decision Making Sifton Elementary, Vancouver, WA 60 50 Total ODRs 40 Playground 30 Classroom 20 10 0 October November Cautions for Continuous Regeneration When you keep it fresh… …avoid lethal mutations Consider the critical features of what makes PBIS effective Reward systems – recognition of their success Not a scrap of paper without recognition Not insincere praise Not the same for everyone! Sustainability Activity PBIS Systems Checkup handout Research on PBIS Sustainability What is the strongest predictor of PBIS sustainability? McIntosh, K., Mercer, S. H., Hume, A. E., Frank, J. L., Turri, M. G., & Mathews, S. (2013). Factors related to sustained implementation of School-wide Positive Behavior Support. Exceptional Children, 79, 293-311. What is more important to sustainability? Having a supportive administrator? Having a strong PBIS team? Results: Predictive Model Model fit indices acceptable (except χ2) χ2 (731) = 881.55, p < .001, CFI = .96, TLI = .96, RMSEA = .03 R 2 = .45 Factors Priority (B = .14, SE = .39, p > .05) Team Use of Data (B = .61, SE = .24, p < .05) District Priority (B = -1.14, SE = .66, p > .05) Capacity Building (B = .98, SE = .43, p < .05) 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 School 5.38 Priority ** .07 Sustained PBIS Fidelity Team Use of Data District Priority .47 -.34 Sustained PBIS Fidelity Capacity Building .41 What is more important to sustainability? Having a supportive administrator Having a strong PBIS team Takeaways PBIS is most sustainable with a supportive administrator and a strong team If administrators take too much of a lead, sustainability can suffer School teams can benefit from training in running meetings and using data District systems (coaching, training, CoPs) enable both admins and teams Your Turn! Find someone next to you: 1. Do you have a champion in your school? 2. How would your systems fare if that person moved on? 3. What steps do you need to take to ensure that you’ll continue the work? What critical features predict sustained PBIS implementation? Mathews, S., McIntosh, K., Frank, J. L., & May, S. (2014). Critical features predicting sustained implementation of school-wide positive behavior support. Journal of Positive Behavior Interventions, 16, 168-178. Research Questions 1. 2. To what extent do school personnel ratings of implementation of PBIS systems significantly predict sustained implementation and levels of problem behavior? Within any statistically significantly predictive PBIS systems, which critical features of these systems significantly predict sustained implementation? PBIS Self-Assessment Survey (Sugai, Horner, & Todd, 2000) Four Systems School-wide Non-classroom Classroom Individual Which system best predicts sustained implementation (BoQ) 3 years later? School-wide Non-classroom Classroom Individual Which system best predicts student outcomes (ODRs) 3 years later? School-wide Non-classroom Classroom Individual Which features best predict sustained implementation? Expected behaviors defined clearly Problem behaviors defined clearly Expected behaviors taught Expected behaviors acknowledged regularly Consistent consequences CW procedures consistent with SW systems Options exist for instruction Instruction/materials match student ability High rates of academic success Access to assistance and coaching Transitions are efficient Lessons learned for sustaining School-wide PBIS Focus on bringing PBIS into the classroom Consistency with SW systems High rates of acknowledgment for prosocial behavior Focus on quality differentiated instruction across academic domains Student instruction at their level SETTING Expectations Matrix All Settings Hallways Playground Cafeteria Library/ Computer Lab Study, read, compute. Sit in one spot. Assembly Respect Ourselves Be on task. Give your best effort. Be prepared. Walk. Have a plan. Eat all your food. Select healthy foods. Respect Others Be kind. Hands/feet to self. Help/share with others. Use normal voice volume. Walk to right. Play safe. Include others. Share equipment. Practice good table manners Whisper. Return books. Listen/watch. Use appropriate applause. Recycle. Clean up after self. Pick up litter. Maintain physical space. Use equipment properly. Put litter in garbage can. Replace trays & utensils. Clean up eating area. Push in chairs. Treat books carefully. Pick up. Treat chairs appropriately. Respect Property Classroom Classroom Procedures/Routines EXPECTATIONS Class-Wide Arrival Cooperative Learning Groups Independent Seat Work Whole Group Identify Attention Signal…….Teach, Practice, Reinforce Be Respectful Be Responsible Be Safe Classroom Procedures/Routines Class-Wide EXPECTATIONS Cooperative Learning Groups Arrival Independent Seat Work Whole Group Identify Attention Signal…….Teach, Practice, Reinforce Be Respectful • • • • Be Responsible • • • • • Be Safe • • Listen to others • Use inside voice Use kind words • Ask permission Enter/exit classroom prepared Use inside voice Be prepared • Follow directions • Be a problem solver Make choices that support your goals Keep hands, • feet, and objects to self Organize your self Walk Place materials • in correct area Begin warm-up • promptly • Use Time • Wisely Contribute • Complete your part Be a TASK master Use your neighbour • Use Materials • Carefully Keep hands, • feet, and • objects to self Walk • Listen to others • • Accept differences • • Use kind words • Encourage others Use quiet voice Follow directions • • • • • • Eyes/ears on speaker Raise hand to speak Contribute to learning Follow directions Take notes Meet your goals Stay at seat Keep hands, feet, and objects to self What do long-term sustainers tell us about sustainability? Andreou, T. E., McIntosh, K., Ross, S. W., & Kahn, J. D. (2015). Critical incidents in sustaining school-wide positive behavioral interventions and supports. Journal of Special Education, 49, 157-167. Settings and Participants Three schools and their school district in rural British Columbia Sustained PBIS systems for over a decade with minimal provincial support 17 educators 4 District administrators (current or former) 4 Principals (current or former) 6 General education teachers 3 Special education teachers Method Qualitative interview questions: What helps sustainability of PBIS? 2. What hinders sustainability of PBIS? 1. Findings: 13 Categories Continuous Teaching Positive Reinforcement SWPBIS Team Effectiveness Staff Ownership Adaptation Community of Practice Involving New Personnel Use of Data Access to External Expertise Maintaining Priority School Administrator Involvement Staff Turnover Conflict of Personal Beliefs/Mistaken Beliefs Positive Reinforcement “They believe that what we’re doing is actually making a difference because we see change in behavior.” “If people find a very positive environment with children…they’re very respectful and so on…then you want to...It reinforces that what we’re doing is right and we don’t want to lose that, we want to keep going.” Staff Ownership “If it is given to you from the principal, then there is an automatic reaction, the question is whether I have to do it or I don't have to do it...Coming from a teacher…there is a reason for why you’re doing it, because it makes my job easier, because I can tell you as a colleague, hey I'm doing this and it's…staff driven. Not being told to do it.” Involving New Personnel “First year teachers, they’re overwhelmed, but we kept saying, you know, we’d love to see you at a meeting, you know, it would really help out. And so when they go there it just automatically includes, “you need to start understanding how it really works.” A District Planning Tool for Sustainability PBIS Leadership Team Self-Assessment A self-assessment and action planning tool for district, regional, or state leadership teams An integrated action plan for sustainability Available for free at: http://pbis.org A School Team Planning Tool for Sustainability The SUBSIST Checklist A research validated self-assessment and action planning tool for school teams and coaches An integrated action plan for sustainability Available for free at: http://kentmcintosh.wordpress.com Sustainability Activity SUBSIST Checklist SUBSIST Checklist: Self-Assessment for Sustainability 1. Complete Self-Assessment What do we have in place? What would be the one addition that would make the biggest difference? 2. Create an Action Plan to enhance sustainability Identify WHO will do WHAT by WHEN handout A Sustainable Community of Practice Please share: One of your school’s sustainability strengths What’s working, strategies, ongoing activities, etc. How you overcame an obstacle Your top action plan task Something for Tomorrow “Positive Parent Postcards” Teachers are given a stamped, preaddressed postcard for each student in their classrooms at the start of the year GOAL: send a quick, positive note home for each student in the school “The Blank Matrix Activity” Provide students with a school-wide matrix (with blank expectation by setting cells) Have students write (or draw) expectations for each area Use results to: Revise matrix to include more “student-friendly” examples Identify areas or expectations that need reteaching Your Turn Again (time permitting) In one word, what have you found to be the single greatest FACILITATOR of sustainability? Your Turn Again (time permitting) In one word, what have you found to be the single greatest BARRIER to sustainability? Your Turn Again (time permitting) Complete the following steps with your team or on your own: 1. What are the three things that you think would make the most impact on your school’s systems? 2. Create an action plan: Consider WHO will do WHAT by WHEN Contact Information Kent McIntosh Special Education Program University of Oregon kentm@uoregon.edu @_kentmc Cannon Beach, Oregon © GoPictures, 2010 Handouts: http://kentmcintosh.wordpress.com Selected References Coffey, J., & Horner, R. H. (2012). The sustainability of school-wide positive behavioural interventions and supports. Exceptional Children, 78, 407-422. Curtis, M. J., Castillo, J. M., & Cohen, R. (2008). Best practices in system-level change. In A. Thomas & J. P. Grimes (Eds.), Best practices in school psychology V (pp. 887-901). Bethesda, MD: National Association of School Psychologists. Gietz, C. & McIntosh, K. (2014). Relations between student perceptions of their school environment and academic achievement. Canadian Journal of School Psychology, 29, 161-176. Hume, A. E., & McIntosh, K. (2013). Construct validation of a measure to assess sustainability of school-wide behavior interventions. Psychology in the Schools, 50, 1003-1014. Mathews, S., McIntosh, K., Frank, J. L., & May, S. (2014). Critical features predicting sustained implementation of school-wide positive behaviour support. Journal of Positive Behavior Interventions, 16, 168-178. Selected References McIntosh, K., Horner, R. H., & Sugai, G. (2009). Sustainability of systems-level evidence-based practices in schools: Current knowledge and future directions. In W. Sailor, G. Sugai, R. H. Horner, G. Dunlap (Eds), Handbook of positive behavior support (pp. 327-352). New York: Springer. McIntosh, K., MacKay, L. D., Hume, A. E., Doolittle, J., Vincent, C. G., Horner, R. H., & Ervin, R. A. (2011). Development and initial validation of a measure to assess factors related to sustainability of school-wide positive behaviour support. Journal of Positive Behavior Interventions, 13, 208-218. doi: 10.1177/1098300710385348 McIntosh, K., Mercer, S. H., Hume, A. E., Frank, J. L., Turri, M. G., & Mathews, S. (2013). Factors related to sustained implementation of school-wide positive behaviour support. Exceptional Children, 79, 293-311. McIntosh, K., Predy, L. K., Upreti, G., Hume, A. E., Turri, M. G., & Mathews, S. (2014). Perceptions of contextual features related to implementation and sustainability of school-wide positive behaviour support. Journal of Positive Behavior Interventions, 16, 29-41. Vaughn, S., Klingner, J., & Hughes, M. (2000). Sustainability of research-based practices. Exceptional Children, 66, 163-171.