System and Practice Change Through an Implementation Lens Ohio Alternative Response Summit Presentation on Implementation Science May 14, 2010 Allison Metz, PhD Karen Blase, PhD Melissa Van Dyke, LCSW Frank Porter Graham Child Development Institute University of North Carolina at Chapel Hill Common Goal To build capacity in child welfare agencies and to support Counties in achieving the sustainable implementation infrastructure and systemic change necessary for an effective Alternative Response model…. . . . To improve outcomes for families and children Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Overall Challenge Science to Service Gap What is known often is not what is adopted to help children, families, and caregivers Implementation Gap There are not clear pathways to implementation What is adopted often is not used with fidelity and good effect What is implemented often disappears with time and staff turnover Copyright © Dean L. Fixsen and Karen A. Blase, 2008 The National Implementation Research Network (NIRN) Craft knowledge EBP purveyors (program developers) EBP implementation site managers Implementation researchers Scientific information Program development and replication data Qualitative study of program developers Synthesis of the implementation evaluation and research literature Copyright © Dean L. Fixsen and Karen A. Blase, 2008 From the Synthesis of the Literature We Know That… Implementation issues are common across widely diverse domains Human service prevention and intervention (e.g. child welfare, substance abuse, mental health, violence prevention, education) Advanced manufacturing technologies Research-based clinical guidelines Engineering (e.g. bridge maintenance) Hotel management National franchise operations Cancer prevention and treatment Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Positive Intervention Outcomes ≠ Implementation Implementation has not been achieved by doing more or better research on interventions or on curricula The usability of a program or practice has nothing to do with the weight of the evidence regarding that program –“Evidence” on effectiveness helps you select what to implement for whom –“Evidence” on these outcomes does not help you implement the program or practice Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Problem Children and families cannot benefit from interventions they do not experience “In a study of real world practice in addiction treatment settings…found that many clinicians reported that they were doing CBT. However, an independent analysis of treatment sessions found that CBT strategies showed up in just 3 percent of them.” Journal of Substance Abuse Treatment (Vol. 35, No. 4) Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Insufficient Methods Diffusion/dissemination of information by itself does not lead to successful implementation Training alone, no matter how well done, does not lead to successful implementation Policies and funding alone do not lead to successful implementation Fixsen, Naoom, Blase, Friedman, Wallace, 2005 Copyright © Dean L. Fixsen and Karen A. Blase, 2008 What Works?... Fidelity Matters IMPLEMENTATION – The HOW INTERVENTION The WHAT Effective Effective NOT Effective BENEFITS Paper Implementation (High Fidelity) (Low Fidelity) NOT Effective Getting Here and Staying Here is the Challenge! Copyright © Dean L. Fixsen and Karen A. Blase, 2008 What Works?...Purveyor Successful implementation on a useful scale requires a purveyor An individual or group of individuals representing a program or practice who actively work to implement that practice or program with fidelity and good effect (Affiliated Private, Funded TA Centers, Intermediaries) Purveyors accumulate data & experiential knowledge & become more effective and efficient over time Purveyors differ in what they offer and how they help Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Purveyor AND Intermediary Structures and Strategies Simultaneous, Multi-Level Interventions Intermediary Purveyor and Practitioners Agency/Provider Management (leadership, policy) Administration (HR, structure) Supervision (nature, content) Local, County, & State Context Federal Context Copyright © Dean L. Fixsen and Karen A. Blase, 2008 What Works? …Implementation Best Practices There are intervention and prevention “best practices” and EBPs Practices Motivational Interviewing, Appropriate Risk Assessment, Social Skills Training Programs and Programs Nurse Family Partnership, MST, PCIT, TF-CBT, Strengthening Families, Incredible Years There are implementation best practices to make the principles and programs come to life in communities and agencies. Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Change What?….Practice? Program? Organization? Systems? From more limited and focused changes (e.g. targeted new policy, new program implemented by a provider agency) to more sweeping practice change (e.g., systems of care) all will require… Organizational and systems change Organizational and systems change are “in service” to the new or improved “future state” and “desired outcomes” The organization changes in order to…. The system needs to change in order to… Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Cascading Logic Model For systems to change, we need individuals to change their activities, patterns, behavior How do we identify, create change in, and measure changes in people’s behavior, knowledge, and attitudes that relate to better outcomes for children and families? How do we measure the products and processes that are created to support the service or system change? Clarity regarding leverage points and focus of the work for foundations and their beneficiaries at different points in the change process Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Cascading Logic Model Population Children at Risk of Child Abuse and Neglect Children and Families with substantiated issues Intervention Strategies and Measures (WHAT) Alternative Response Model Intervention Outcomes Increased family engagement in services Improved safety at home Reduced incidents of recurrence Cascading Logic Model Population Young Children at Risk of Child Abuse Children and Families with substantiated issues Populations Front-line Practitioners and Case Managers Intervention Strategies and Measures (WHAT) Alternative Response Model Intervention Outcomes Increased family engagement in services Improved safety at home Reduced incidents of recurrence Practice, Organizational and Systems Change Strategies (HOW) Implementation Related Outcomes Carefully selected staff and the provision of skillful, timely training, coaching, performance assessments , use of data in supportive administrative and state /county environments Competent use of selected Interventions & approaches Alternative Response (fidelity measures or staff performance assessments) Cascading Logic Model Population Young Children at Risk of Child Abuse Children and Families with substantiated issues Intervention Strategies and Measures (WHAT) Alternative Response Model Intervention Outcomes Increased family engagement in services Improved safety at home Reduced incidents of recurrence Practice, Organizational and Systems Change Strategies (HOW) Implementation Related Outcomes Front-line Practitioners and Case Managers Carefully selected staff and the provision of skillful, timely training, coaching, performance assessments , use of data in supportive administrative and state /tribal environments Competent use of selected Interventions & approaches Alternative Response (fidelity measures or staff performance assessments) Personnel in Child Welfare Training Systems, Supervisors, Data System Managers, Agency Administrators, State/Tribal Administrators Agreements with Program Developers and Consultants Training and coaching for Trainers – capacity building Release time for Supervisors to learn to coach Installation of fidelity monitoring systems Skillful, timely training, coaching, performance assessments , data systems in supportive administrative and regulatory environments Populations Cascading Logic Model Population Young Children at Risk of Child Abuse Children and Families with substantiated issues Populations Intervention Strategies and Measures (WHAT) Alternative Response Model Practice, Organizational and Systems Change Strategies (HOW) Intervention Outcomes Increased family engagement in services Improved safety at home Reduced incidents of recurrence Implementation Related Outcomes Front-line Practitioners and Case Managers Carefully selected staff and the provision of skillful, timely training, coaching, performance assessments , use of data in supportive administrative and state /county environments Competent use of selected Interventions & approaches Alternative Response (fidelity measures or staff performance assessments) Personnel in Child Welfare Training Systems, Supervisors, Data System Managers, Agency Administrators Agreements with Program Developers and Consultants Training and coaching for Trainers – capacity building Release time for Supervisors to learn to coach Installation of fidelity monitoring systems & outcome data Skillful, timely training, coaching, performance assessments , data use in supportive administrative environments Program and Agency Administrators, Child Welfare Agency Personnel at the Tribal/ State/County/Local level/External Partners/Courts Common mission, vision developed Stages of Implementation utilized Linked Implementation Teams created Formal structures created to develop practice informed policy loops Changes in Funding Streams support new functions and relationships Fidelity and outcome data systems developed and maintained Collaboration among providers to pool funding for infrastructure Sustainable infrastructure developed Relationships with consultants & Program Developers formalized Training and coaching for Trainers (pre/post) Release time for Supervisors to learn to coach Fidelity monitoring systems exist and are used Outcome measures exist and are used Funding and regulatory changes occur to support infrastructure Cascading Logic Model Population Young Children at Risk of Child Abuse Children and Families with substantiated issues Populations Intervention Strategies and Measures (WHAT) Alternative Response Model Practice, Organizational and Systems Change Strategies (HOW) Intervention Outcomes Increased family engagement in services Improved safety at home Reduced incidents of recurrence Implementation Related Outcomes Front-line Practitioners and Case Managers Carefully selected staff and the provision of skillful, timely training, coaching, performance assessments , use of data in supportive administrative and state /tribal environments Competent use of selected Interventions & approaches Alternative Response (fidelity measures or staff performance assessments) Personnel in Child Welfare Training Systems, Supervisors, Data System Managers, Agency Administrators Agreements with Program Developers and T/TA Centers Training and coaching for Trainers – capacity building Release time for Supervisors to learn to coach Installation of fidelity monitoring systems & outcome data Skillful, timely training, coaching, performance assessments , data use in supportive administrative environments Program and Agency Administrators, Child Welfare Agency Personnel at the Tribal/ State/County/Local level/External Partners/Courts Common mission, vision developed Stages of Implementation utilized Linked Implementation Teams created Formal structures created to develop practice informed policy loops Changes in Funding Streams support new functions and relationships Fidelity and outcome data systems developed and maintained Collaboration among providers to pool funding for infrastructure Sustainable infrastructure developed Relationships with consultants & Program Developers formalized Training and coaching for Trainers (pre/post) Release time for Supervisors to learn to coach Fidelity monitoring systems exist and are used Outcome measures exist and are used Funding and regulatory changes occur to support infrastructure Stages of Implementation Major Implementation Initiatives occur in stages: Exploration Installation Two to Four Years Initial Implementation Full Implementation Innovation Sustainability Fixsen, Naoom, Blase, Friedman, & Wallace, 2005 Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Exploration “Many implementation efforts fail because someone underestimated the scope or importance of preparation. Indeed, the organizational hills are full of managers who believe that an innovation’s technical superiority and strategic importance will guarantee acceptance.” Leonard-Barton & Kraus, Harvard Business Review, 1985 Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Exploration Exploration Goals: Examine degree to which the EBP, best practice, systems change meets the needs in the settings identified Determine whether moving ahead with the initiative and implementation is desirable and feasible Create readiness for change at many levels “Pay now or pay later.” Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Assess Feasibility Needs Fit Resource availability Evidence Readiness for replication or degree to which it is operationalized Capacity Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Need in Tribal/ State, Agency, Assessing Fit and Feasibility of Initiatives Setting Socially Significant Issues Parent & Community Perceptions of Need Data indicating Need Need Fit with current - Capacity Staff meet minimum qualifications Able to sustain Imp Drivers • Financially • Structurally Buy-in process operationalized • Practitioners • Families • Agency and Departments Fit •Initiatives • Tribal//State, County, Agency Priorities • Organizational structures • Community Values Capacity to Implement Readiness Qualified purveyor Expert TA available Mature sites to observe # of replications How well is it operationalized? Are Imp Drivers operationalized? Resource Availability Intervention Readiness for Replication Initiative : 5 Point Rating Scale: High = 5; Medium = 3; Low = 1. Midpoints can be used and scored as a 2 or 4. High Medium Low Need Evidence Fit Resources Availability Resources Staffing Training Data Systems Coaching & Supervision Administrative & system supports needed Time Evidence – is there any? Outcomes – Is it worth it? Fidelity or process data Cost – effectiveness data Number of studies Population similarities Diverse cultural groups Efficacy or Effectiveness Evidence Readiness for Replication © National Implementation Research Network 2009 Capacity to Implement Total Score: Adapted from work by Laurel J. Kiser, Michelle Zabel, Albert A. Zachik, and Joan Smith at the University of Maryland Assess Buy-in and Create Readiness Determine “buy-in” process and measures Prepare decision makers, agency leads, and partners for implementation and…. Operationalize the policy, evidence-based innovation, or systems change Describe benefits and risks and help interested parties gain confidence in managing risk Assess agencies’ and/or collaborating partners’ comfort with ability to manage risk Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Resistance to Change There is no such thing – only inadequate preparation It is not “their” problem, it is ours. Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Creating Readiness for Change Individual readiness for change Transtheoretical Model or Stages of Change Precontemplation Contemplation Preparation Action Maintenance Prochaska and DiClemente Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Stages of Change Stage of Change for Pre-Action Individuals: Precontemplation – 40% Contemplation – 40% Preparation – 20% “If only 20% of employees in organizations are prepared to take action. . . .” Janice M. Prochaska, James O. Prochaska, and Deborah A. Levesque (2001) Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Working with the “Not Quite Ready” Provide Information What is the change, why now, and what might it mean for me? “Reflect” Concerns In a neutral way, acknowledge and validate the person’s concerns (e.g. “It sounds like you need more information…”, “It makes sense to me that you are wondering about whether this is a passing fad…”) Roll with Resistance Resistance is a signal to change strategies New perspectives are invited but not imposed (e.g. “Would you like the opportunity to learn more about this?”) The person’s perspective is a valuable resource in learning more about what is concerning or challenging about the change initiative Miller and Rollnick, 1991 Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Engaging Stakeholders: Rationales A Rationale is a statement of Naturally or logically occurring outcomes Associated with current or future goals and/or behavior (“if…..then”). Benefits Data indicate that providing rationales: Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Is respectful and supportive of honest exploration of issues Promotes choice and “buy-in” Increases confidence in decision-making Promotes understanding of “functions” of the decisions being made rather than fostering compliance Developing Rationales Brief Aligned with Stage of Change Relevant to the Individual Pre-Contemplation and Contemplation Promote or provide assurance about the “upside” of taking risks Points out the “downside” of not moving forward – a different view of ‘risk’ Requires a leader to be “on purpose” Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Installation Installation Stage Goal To make the structural and instrumental changes necessary to initiate services “If you build it, they will come”. . . but you actually have to build it! Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Installation Activities Mutual Selection of Provider Organization (Community, Organization, Program Experts) Initial training provided for practitioners and provider organization staff Negotiate and enact structural and instrumental changes needed to initiate the program or practice (write new job descriptions, recruit, hire/redeploy staff, make special purchase of equipment, arrange for space, organize billing procedures) Establish referral flow, develop ancillary contacts Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Initial Implementation Initial Implementation Goal Survive the awkward stage! Learn from mistakes Continue “buy-in” efforts Manage expectations “Anything worth doing…is worth doing poorly.” Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Initial Implementation Activities Provide new services Manage staff turnover Manage change process Change practice Change organizational and community structure and culture Put infrastructure components in place Overcome fear and inertia Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Full Implementation Full Implementation Goals Maintaining and improving skills, activities, linkages throughout the system Components integrated, fully functioning Skillful practices by front line staff, supervisors, administrators Changes in policy that are reflected in practice at all levels Ready to be evaluated for expected outcomes “The only thing worse than failing and not knowing why you failed, is succeeding and not knowing why you succeeded.” ~ Jane Timmons-Mitchell Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Innovation Innovation First do it right (high fidelity) or “as intended” Then do it differently and better Consult with content experts Evaluate impact Ability to retain function while changing form given turnover, changing needs and context Ability to adhere to the underlying theory of change “Innovation needs to equal improvement.” Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Sustainability Goals of Sustainability Financial: Ensure funding streams for service and infrastructure Programmatic: Ensure high fidelity and positive outcomes through infrastructure improvement and maintenance Plan for turnover “The only thing harder than getting there is staying there.” Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Reflection: Stages What might this information tell us about how Alternative Response is implemented in the remaining counties? Should we build in time for “exploration” and planning for these change initiatives? Does this inform our work as we attempt to promote the scaling up of Alternative Response statewide in Ohio? What lessons have pilot counties learned regarding exploration work? installation work? initial implementation? What might the State or TA provider do to facilitate the new counties’ work during the stages of implementation? Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Putting “It” Into Action Who will do what differently? Who will help change occur? AND How will change happen? At all levels…. Practice Program Organization System Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Improved outcomes for children and families Implementation Drivers Performance Assessment Systems Intervention Coaching Training Selection Adaptive Integrated & Compensatory Technical Facilitative Administration Decision Support Data System Leadership Graphics by Steve Goodman,2009 © Fixsen & Blase, 2008 Implementation Drivers Competency Drivers are mechanisms that help to develop, improve, and sustain one’s ability to implement an intervention with fidelity and benefits to consumers. Competency Drivers include: Selection, Training, Coaching, and Performance Assessment Organization Drivers are mechanisms to create and sustain hospitable organizational and systems environments for effective services. Organization Drivers include: Decision Support Data System, Facilitative Administration, and Systems Intervention Leadership Drivers are methods to manage Technical problems where there is high levels of agreement about problems and high levels of certainty about solutions and to constructively deal with Adaptive challenges where problems are not clear and solutions are elusive Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Improved outcomes for children and families Implementation Drivers Performance Assessment Coaching Training Implementation Lens Selection Graphics by Steve Goodman,2009 © Fixsen & Blase, 2008 Sobering Observations "All organizations [and systems] are designed, intentionally or unwittingly, to achieve precisely the results they get." R. Spencer Darling Business Expert Copyright © Dean L. Fixsen©and Karen Fixsen, A. Blase, 2008 Dean Karen Blase, Robert Horner, George Sugai, 2008 Improved outcomes for children and families Implementation Drivers Performance Assessment Coaching Training Selection Graphics by Steve Goodman,2009 Systems Intervention Facilitative Administration Decision Support Data System © Fixsen & Blase, 2008 Decision-Support Data Systems Measure Fidelity AND Outcomes BECAUSE you need to know: Are we having an implementation problem? Low fidelity & Poor outcomes = Implementation problem Are we having an effectiveness problem? High fidelity and Poor outcomes = Effectiveness Problem Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Improved outcomes for children and families Implementation Drivers Performance Assessment Coaching Training Selection Graphics by Steve Goodman,2009 Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Systems Intervention Facilitative Administration Decision Support Data System © Fixsen & Blase, 2008 System Stability EXISTING SYSTEM Effective Innovations are Changed to Fit the System Or Operate in the Shadows (The Ghost System) Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Effective System Change EXISTING SYSTEM EXISTING SYSTEM IS CHANGED TO SUPPORT THE EFFECTIVENESS OF THE INNOVATION Effective Innovations are Changed to Fit the System Or Operate in the Shadows (Ghost System) Copyright © Dean L. Fixsen and Karen A. Blase, 2008 (Host System) EFFECTIVE INNOVATION Improved outcomes for children and families Implementation Drivers Performance Assessment Systems Intervention Coaching Training Selection Adaptive Integrated & Compensatory Technical Facilitative Administration Decision Support Data System Leadership Graphics by Steve Goodman,2009 © Fixsen & Blase, 2008 Reflection: Implementation Drivers How is the information related to Implementation Drivers relevant to scaling up Alternative Response? How do you see using the information regarding the Drivers in your work to support the continued implementation of Alternative Response in pilot counties? How do you see using the information regarding the Drivers in your work to support implementation of Alternative Response in new counties? Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Initiating and Managing Change Implementation Team(s) General Definition: Core group of individuals, who are representative of the stakeholders and “systems” and who are charged with guiding the overall implementation from exploration through to full implementation Benefits: Provides a focused and accountable structure to increase the likelihood that this effort will not be abandoned or derailed Scope of the initiative determines the scope of authority and the need for linked Implementation Teams Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Core Competency for Implementation Teams A team that: ■Knows the innovation very well (formal and practice knowledge) ■Knows implementation very well (formal and practice knowledge) ■Knows improvement cycles to make intervention and implementation methods more effective and efficient over time ■ Promotes systems change at multiple levels to create hospitable cultures, policies, and funding streams Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Implementation Teams Integrated and Interlocking at Multiple Levels Agency or “setting-based” teams Community or collaborative teams Tribal or state team Represents the stakeholders and the ‘system’ Focus is on Ongoing “buy-in” and readiness Installing and sustaining the drivers Fidelity & outcomes Alignment (funding and policy) Building the new system – linkages Problem-solving and sustainability Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Catawba County Child Well-Being Project Consumers Key DSS Leaders Design Team Sector Leads Stakeholders Funders Providers Meta- Team Leads + Design Team Rep’s EBP Area, Agency, or Unit EBP Area, Agency, or Unit EBP Area, Agency, or Unit IMPLEMENTATION TEAMS Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Senior Mgmt Team EBP Area, Agency or Unit Reflection – Implementation Teams What implications does the information regarding implementation teams have for work with pilot and new counties? How can the State and counties ensure that implementation teams are in place, clear on their roles, and linked to support the implementation of Alternative Response? Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Improvement Cycles Policy to Practice to Policy Cycles Transformation Zones Rapid Cycle Teams Problem-solving Practice Improvement Usability Testing Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Practice Policy (Plan) Policy Policy Enabled Practices (PEP) “External” System Change Support Policy Practice (Do) Structure Procedure Practice Practice Feedback Loops Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Practice (Do) Policy Policy Enabled Practices (PEP) Feedback Practice Informed Policy (PIP) Policy (Plan) Study - Act “External” System Change Support Policy Structure Procedure Practice Reflection – Improvement Cycles Are policy-practice feedback loops currently in place in pilot counties? What form do they take? How can policy-practice feedback loops be established in each county and statewide? What processes are in place to ensure “lessons learned” are shared? Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Summary Stage related activities prepare the system for a successful change process Competence needs to be developed and sustained Selection, training, coaching, fidelity measures help d… new practitioner behavior and skills change and support Organizations and systems need to change Data systems need to be used to make decisions Facilitative administrative practices & systems interventions create hospitable environments Policy enables new practice but practice needs to inform policy Copyright © Dean L. Fixsen and Karen A. Blase, 2008 Summary Purveyors and implementation teams help with system and service change Scaling up requires establishing implementation capacity and tending to necessary “scale-up shifts” You are never done – The environment is in motion Improvement cycles are critical The ‘right’ leadership strategies are needed for the issues at hand Copyright © Dean L. Fixsen and Karen A. Blase, 2008 For More Information Allison Metz, PhD 919-218-7540 allison.metz@unc.edu At the Frank Porter Graham Child Development Institute University of North Carolina Chapel Hill, NC www.scalingup.org http://nirn.fpg.unc.edu/ http://www.fpg.unc.edu/~nirn/resources/publications/Monograph/ Copyright © Dean L. Fixsen and Karen A. Blase, 2008 For More Information Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231). Download all or part of the monograph at: http://www.fpg.unc.edu/~nirn/resources/publications/ Monograph/ To order the monograph go to: https://fmhi.pro-copy.com/ Copyright © Dean L. Fixsen and Karen A. Blase, 2008