Bringing Evidence to Bear:
What Facilitates Adoption of Research-based
Evidence in Practice and Policy-making in Schools
Large Group Panel
OSEP Project Director’s Conference 2014
Washington, DC
Session Goals
• Discuss organizational and other factors that
facilitate take-up of new knowledge and facilitate
implementation of innovative programs and
policies
• Discuss program features that lends itself to early
adoption and uptake of programs in the school
environment
• Adopt a systems perspective in intervention
development for improved adoption
Panel
Presenters
• Dr. Vivian Tseng, Vice President, William T. Grant
Foundation, New York, NY
• Dr. David Test, Professor, Department of Special Education
and Child Development, Co-Director, National Secondary
Transition Technical Assistance Center, UNC Charlotte, NC
• Dr. Arun Karpur, Research Director, New York State
PROMISE, Research Faculty, Cornell University, New York, NY
Moderator:
• Dr. David Guardino, Research to Practice Division, U.S.
Department of Education, Office of Special Education
Programs
Format of LGP
• Presentation by individual presenters
• Q&A session moderated by Dr. Guardino
• Discussion with the attendees and the speakers
Promoting the Use of
Research Evidence
Vivian Tseng, Ph.D.
Supporting Research to inform
policy and practice to Improve the
Lives of Young People
Evidence-based Programs
Preventing Mental, Emotional, and Behavioral Disorders Among Young People: Progress and Possibilities (2006)
Research to Practice
 Improve Quality of Research
 Improve Communication, Dissemination, Marketing
 Demand Use of Research Evidence
 Incentivize Evidence-based Programs
Research and Practice
 Improve Research and Its Use
 Develop Shared Commitments
 Build Relationships and Trust
Organizations & Systems
Intermediaries
What’s an NSTTAC?
National Secondary Transition
Technical Assistance Center
The Transition to College and Career TA&D
Center for secondary transition funded by
USDOE, OSEP (1/1/06-12/31/14)
Co-Directors:
David W. Test (UNC Charlotte)
Paula Kohler (Western Michigan University)
Project Coordinator:
Cather Fowler (UNC Charlotte)
NSTTAC’s Purpose
• Disseminate information to State Education Agencies, Local
Education Agencies, schools, and other stakeholders to
improve the:
• implementation and scaling up of evidence-based practices to
develop appropriate measurable postsecondary goals and implement
transition services
• implementation of policies, procedures, and practices that facilitate
and increase the participation of students with disabilities in
programs and initiatives that are designed to ensure college- and
career-readiness
• achievement of compliance with IDEA’s transition requirements
• Support efforts to ensure that all students with disabilities are
prepared for college (or other postsecondary education and
training) and the workforce.
Model for Extending Transition Research
Effective Transition
Practices
Increase Capacity to
Implement Effective
Transition Practices
Data-Based
Decision Making
Professional
Development
Facilitate Implementation of
Effective Transition
Practices
Policy Analysis
and Change
Technical
Assistance
Broad Definitions
(Helsel, Hitchcock, Miller, Malinow, & Murray, 2006; Twyman, 2008)
EvidenceBased
Practices
• Are based on rigorous research designs
• Have demonstrated a record of success for improving
student outcomes
• Have undergone systematic review process using
quality indicators to evaluate level of evidence
ResearchBased
Practices
• Are based on rigorous research designs
• Have demonstrated a record of success for improving
student outcomes
Promising
Practices
• Are based on research
• Have demonstrated limited success
• Have used a ‘weak’ research design
Unestablished
Practices
• Are not based on research
• Have no data to support effectiveness
• Based on anecdotal evidence and/or professional
judgment
What We Have Done
• Reviewed experimental research to identify
evidence-based practices in secondary transition
• Identified 63 evidence-based practices
Taxonomy
Category
Evidence-Based
Practices
Research to Practice
Lesson Plan Starters
6
9
Student
Development
56
98
Family
Involvement
1
0
Program
Structure
9
9
Student
Focused
Planning
Student Development
Teaching Grocery Shopping
Skills
Teaching Laundry Tasks Using
Response Prompting
• Using Computer Assisted
Instruction
Teaching Leisure Skills
• Using Community Based
Instruction
• Using Constant Time Delay
• Using Response Prompting
• Using Response Prompting
• Using a System of Least to Most
Prompts
Teaching Safety Skills
Teaching Home Maintenance
Skills
• Using Progressive Time Delay
• Using Response Prompting
• Using Video Modeling
• Using Community Based Instruction
• Using a System of Least to Most
Prompts
Based on High-Quality Research, the
Field of Secondary Transition now has
Evidence-based:
Practices
Micro Level
 Specific
interventions
Predictors!
Macro Level
• Systems
• Programs
• “Generic” practices
In-School Predictors by Post-School Outcome Area
Education
Employment
•Career Awareness
X
X
•Occupational Courses
X
X
•Paid Employment/Work Experience
X
X
•Vocational Education
X
X
Predictors/Outcomes
•Work Study
•Exit Exam Requirements/High School
Diploma Status
•Program of Study
X
X
X
•Community Experiences
•Inclusion in General Education
Independent Living
X
X
X
X
X
Continuation of In-School Predictors by Post-School Outcome Area
Education
Employment
•Self-Advocacy/Self-Determination
X
X
•Self-Care/Independent Living
X
X
•Social Skills
X
X
•Interagency Collaboration
X
X
•Parent Expectations
X
X
Predictors/Outcomes
•Parental Involvement
Independent
Living
X
X
•Student Support
X
X
•Transition Program
X
X
X
Levels of TA
• Universal/general
– Website
– Annual Capacity Building Institute
• Targeted/specialized
– State specific topics/issues
• Intensive/sustained
– State & LEA specific topic/issues
Data-Driven Decision Making Tools
(Some Samples)
• Predictor Implementation Self-Assessment
• Data Use Toolkit (NPSO – Indicator 14)
• State Toolkit for Examining Post-School
Success (STEPSS; Indicators 1, 2,13, & 14)
• NDPC-SD Data Tools lite (Indicators 1 & 2)
• NSTTAC Planning Tool (multiple data sources,
including from other tools)
• Short Data Probe (Indicator focused)
• District Initiative Inventory (qualitative, broad
focus)
Assessing Evidence-Based
Programs
and Practices
Need in school, district, state
•
•
•
Need
Capacity
• Staff meet minimum qualifications
• Able to sustain Imp Drivers
• Financially
• Structurally
• Buy-in process operationalized
• Practitioners
• Families
• Agency
Academic & socially significant Issues
Parent & community perceptions of need
Data indicating need
Fit with current Initiatives
Fit
•School, district , state priorities
• Organizational structures
• Community values
Capacity to Implement
Readiness
• Qualified purveyor
• Expert or TA available
• Mature sites to observe
• Several replications
• How well is it operationalized?
• Are Imp Drivers operationalized?
Resource
Availability
Intervention Readiness
for Replication
EBP:
5 Point Rating Scale:
High = 5; Medium = 3; Low = 1.
Midpoints can be used and scored as a 2 or 4.
High
Medium
Low
Need
Fit
Resources Availability
Evidence
Evidence
Readiness for Replication
Capacity to Implement
Total Score:
Resources and supports for;
• Curricula & Classroom
• Technology supports (IT dept.)
• Staffing
• Training
• Data Systems
• Coaching & Supervision
• Administration & system
Evidence
• Outcomes – Is it worth it?
• Fidelity data
• Cost – effectiveness data
• Number of studies
• Population similarities
• Diverse cultural groups
• Efficacy or effectiveness
© National Implementation Research Network
2009
Adapted from work by Laurel J. Kiser, Michelle
Zabel,
Albert A. Zachik, and Joan Smith at the University
of Maryland
Organizational & Program Factors
• Commitment to Adopting EBPs
• Readiness
• Flexibility
• Communication
• Time
Utilizing Continuous Quality
Improvement for Promoting
Adoption and Uptake of Knowledge
NYS PROMISE Approach
Arun Karpur, MD, MPH
Research Director
NYS PROMISE
Employment and Disability Institute,
Cornell University
NYS Promise
NYS PROMISE will utilize an indigenous
model that naturally equips and engages
schools, local disability service providers,
independent living centers, one stop
centers, literacy zones, regional parent
training centers, work incentive planners,
regional transition specialists, and other
community transition stakeholders to
achieve higher postsecondary
employment, education and economic
outcomes for SSI youth by...
Services under
outcomes-based
payment model
Improved access to
key high quality
services leading to
intended outcomes
PROMISE-sponsored
case management &
service coordination
PROMISE-sponsored
parent training,
information & family
coaching
Improved
collaboration
between LEAs and
local service providers
Improved selfdetermination,
expectations,
engagement, attitudes
toward work and
financial literacy
Higher post-secondary employment,
education and economic outcomes for
youth on SSI
Improved quality of transition to
adulthood programs in LEAs for
youth in treatment group
CQI Feedback Loops : Project MIS , Fidelity Assessment, and TA
Increased Access to State and National TA on Transition Best Practices
NYS PROMISE Intervention
Continuous Quality Improvement
Continuous
Quality Improvement
Quality Improvement
Quality Planning Tools &
Processes
•
•
•
•
•
•
•
•
•
Outreach and Recruitment Guide
PROMISE Recruitment Tracking System
Intervention Guide for Practitioners and Evaluators
Training for using project MIS
Training for outcomes-based payment
Organizational capacity assessment for implementation
Brokering relationships between project partners
Formation of regional teams of stakeholders
NYS PROMISE State Steering Committee and
Coordinating Council
Quality Control Tools & Processes
• Project MIS tool [NYESS]– tracking program
participation and services received
• Online training and technical assistance tracking
(OTAT) system
• Program fidelity assessment – qualitative and
quantitative indicators
• Youth and Parent Voices – survey and qualitative
study
• Stakeholder Voices – collaboration survey, biannual learning communities, and steering
committee meetings
Quality Improvement Tools and
Processes
• Quarterly automated NYESS reports for RDS, Providers
and Parent Centers
• TA translating data into action plans at local, regional
and state-level
• Bi-annual comprehensive project reports at Learning
Community identifying (a) what was done well, (b)what
worked and (c) what needs to be done
• Sharing data with steering committee at state-level
• Engaging stakeholders in concept mapping activity to
synthesize information
• Annual progress report
Access to National and State TA
• Access to national and state TA on transition through
partnerships with NYS PDSC, NSTTAC training events,
and other research and TA entities
• Annual DCDT conference attendance
• Webinars on lessons learned from NYS PROMISE
implementation
• Reports from the National Evaluator on NYS PROMISE