A12: Roundtable/Networking
Evaluation Tools
Presented by
Peg Mazeika & Kao Moua Her
WI PBIS & RtI Technical Assistance Coordinators
Amy Tranel External Coach MMSD
The Wisconsin RtI Center/Wisconsin PBIS Network (CFDA #84.027) acknowledges the support of the Wisconsin Department of Public Instruction in the development of this
presentation and for the continued support of this federally-funded grant program. There are no copyright restrictions on this document; however, please credit the
Wisconsin DPI and support of federal funds when copying all or part of this material.
Today’s Agenda
•
Overview of PBIS and Academic Surveys and Tools on the PBIS Apps website
and Wisconsinrticenter.org website
PBIS
•
Academic
Benchmarks of
Quality (BOQ)
School-wide
Implementation
Review (SIR)
Self-Assessment
Survey (SAS)
All Staff Perception
Survey
Evaluation Action Plan
– Generate discussions, decisions, and selections on best fit tools
– Collaborate and develop district and/or school timelines
• Yearly Action Plan
• 3-5 Year Plan
•
Evaluation Schedule Examples
•
Questions/Discussion
Why Evaluate?
Synthesize Upon
• What is the current status of our
implementation?
• To what extent was Academic
RtI/SWPBS implemented as designed?
• What do we need to attain/maintain
fidelity? Move up to next tier? What
Training, Coaching, Staff Training?
Family/Community support? What
funding will be needed?
• Address the evaluation question: “Are
we doing what we said we would do?”
Stages of Implementation
Levels of Implementation
*Exploration
Sustainability
Not in Place
*Installation
District Leadership Team
Self-Assessment
• Use by district leadership team (for
example, district administrators,
principals, external coaches, etc)
• Use to self-assess RtI Academic &
Behavior overall for annual and long
term goals and needs
• Develop action plan for training,
funding, Coaching, professional
development, technical assistance for
team and staff
What is Evaluation?
“Evaluation is the process of collecting and
using information for decision-making.
A hallmark of School-wide Positive
Behavior Support (SWPBS) is a
commitment to formal evaluation.”
District Leadership Team Self Assessment
Wisconsinpbisnetwork.org
Appendix A Blueprint For Success
Current Tools for School PBIS
Implementation Evaluation
Research
Tool
Annual Assessment
Tool
Progress
Monitoring
Tool
Universal
System
System-wide
Evaluation Tool
(SET)
Benchmarks of Quality
(BoQ)
Early Childhood BoQ
(EC BoQ)
PBIS Self Assessment
Survey (SAS)
Tiered Fidelity Inventory
(TFI)
Team
Implementation
Checklist (TIC)
Secondary
&
Tertiary
Systems
(ISSET)
Benchmarks of Advanced
Tiers (BAT)
PBIS Self Assessment
Survey (SAS)
Tiered Fidelity Inventory
(TFI)
Monitoring
Advanced Tiers
Tool (MATT)
Outcome Measures:
School Safety Survey
Current Tools for Academic RtI
Implementation Evaluation
Research
Tool
Annual Assessment
Tool
Progress
Monitoring
Tool
Universal
Level (Tier 1)
School-wide Implementation Review
&
Selected Level
RtI All-Staff Perception Survey
(Tier II)
&
Intensive Level
(Tier III)
Outcome Measures:
Intensive
Selected
Universal
School-wide Implementation Review
&
RtI All-Staff Perception Survey
Current Tools for Academic RtI
Implementation Evaluation
Current Tools for Academic
Implementation Evaluation: School-wide
Implementation Review
What
An online self
assessment
tool for
Wisconsin
schools to help
them measure
School-wide
their RtI
Implementation implementatio
Review
n as it relates
to Wisconsin’s
RtI framework.
Why
To answer the
following
questions:
What does RtI
look like in
action?
What does RtI
look like in our
school?
What are our RtI
strengths?
Where have we
improved? What
do we still need
to work on?
Who
The SIR should
be completed
by school
leadership
teams that
include a
building
administrator.
When
It should be
taken annually
as part of the
school district’s
action planning
process.
How
The SIR is
available
through the
Wisconsin RtI
Center
website at no
cost.
Username
and password
for access to
the SIR and
reports will
be emailed to
the building
administrator.
Current Tools for Academic
Implementation Evaluation: RtI All-Staff
Perception Survey
What
It is used by school
staff for initial and
annual assessment
of perceptions of
implementation of
reading and
RtI All-Staff mathematics multiPerception level support
Survey
systems in their
school.
Why
The survey
results are
summarized and
used for a variety
of purposes
including:
decision making;
annual action
planning for each
content area; and
identifying staff
awareness of
specific RtI
practices, etc.
Who
All
instructional
staff in the
school should
be asked to
complete the
survey.
When
How
Initially, staff
complete the RtI
All-Staff
Perceptions
Survey after the
school leadership
team has
completed the
SIR. In subsequent
years, and as an
on-going
assessment and
planning tool, it
should be
completed before
the SIR.
It is required
that one
school staff
member be
responsible
for setting up
and
administering
the survey to
the
instructional
staff. The
survey must
be
administered
electronically
in order for a
report to be
generated.
www.pbisapps.org
wisconsinrticenter.org
Decisions….Decisions!
• Decisions about Evaluation Tools
– Which tools answer your district/schools’ evaluation
questions/needs?
– How will you use data to action plan district/team
training and support needs based on evaluation
results?
• Decisions about Evaluation Timeline
– When/how often do you administer the tools?
• Decisions about Evaluation Reporting
– Who & How Often will you report outcomes to school
staff and to the District Leadership?
Evaluation & Action Planning
Decisions about Evaluation Tools
•
Does your District/School need a Research tool?
• PBIS: (SET/ISSET)
• Academic: School-wide Implementation Review (Reading/Math) & RtI All-Staff Perception Survey
(Reading/Math)
•
Which Annual Assessment Tool(s) will you use?
• PBIS: BoQ, ECBoQ SAS, Safety Survey, BAT,
• Academic: School-wide Implementation Review (Reading/Math) & RtI All-Staff Perception Survey
(Reading/Math)
•
Which Screening/Progressing Monitoring Tool(s) will/do you use?
• PBIS: TIC, MATT
• Academic: School-wide Implementation Review (Reading/Math) & RtI All-Staff Perception Survey
(Reading/Math)
•
Does your District/School plan include evaluation of all levels of implementation? Tier I (Universal Level),
Tier II (Selected Level) -Tier III (Intensive Level)?
Example:
The District wants to bring RtI Behavior and Academics together
and create one evaluation plan.
• In what specific PBIS & Academic RtI areas are we strong,
and in which areas do we need improvement or training?
• Does our current evaluation plan meet the needs of our
school(s) in both Academic and Behavior?
• How can we schedule our evaluations so that we are not
overwhelming our schools?
• What Funding/Coaching /Training /Technical needs will we
have in the next few years to scale up our implementation
in both areas?
Use your ABCs: Existing Data
Attendance
• Attendance/tardies
• Chronic absenteeism
• Mobility
• Engagement
• Participation
Behavior
•
•
•
•
Office referrals
Suspensions
Behavioral screening
Internalizing behaviors
•
•
•
•
Developmental assets
At-risk support
Behavioral plan
Family stressors
•
•
•
•
Academic screening
Common assessments
Standardized testing
Grades
•
•
•
•
Retention
Accelerated learning
Interventions, Title I
Head Start, Summer School
Social-Emotional
Content
Assessment
Instrument
Team
Implementation
Checklist (TIC)
Self-Assessment
Survey (SAS)
RtI All-Staff
Perception Survey
School-wide
Implementation
Review
BoQ
Fall
(August/Sept/Oct)
X
Winter
(Dec/Jan)
Spring
(April/May)
X
Purpose
X Annually
X Annually (Before or
After the SIR)
OR
X Annually (Before or After
the SIR)
X Annually
Fidelity of
implementation
Staff perception T1,2,3
Assess needs
(Progress Monitor?)
Progress monitor
Behavior & XAcademic
Assessment Timeline
X Annually
MATT
X
X (until 80%)
BATT
X Annually
SSBD
ABC’s (Existing Data)
or Academic
Screening Data
Blue Print Self
Assessment Dist
Leadership Team
Progress monitor
Implementation Tier 1
X until 80%
X
X
X
X
X
X
X
Fidelity of
Implementation Tier 1
Progress monitor
implementation Tier2/3
Fidelity of
Implementation Tier2/3
Universal Screener
Early ID
Internalizer/Externalizer
Screening
Progress Monitor
Training and
implementation action
planning and support.
Adapted from Nieves Flores, 2012
Example
District/School Evaluation Questions
Newly implementing/not yet at fidelity;
Questions to be answered:
• What is our baseline, and what is our progress through the year in
our Systems development and implementation?
• Are we improving on our implementation compared to baseline?
Moving toward attaining Fidelity? Are student outcome data
improving?
• How do our individual staff members rate our implementation in
each area?
• Are our self-assessments consistent with data from outside
evaluators? (SET, Walk Through)
• Are we meeting our annual Action Plan Goals?
Example
District/School Evaluation Questions
Implementing PBIS for three years: We have been using the SAS,
BoQ, SIR and have started Tier 2.
• What specific areas in RtI are we strong in, and in which
areas do we need training or support to improve?
• Are we maintaining our implementation of RtI
Academic/PBIS at Tier 1 as we move up in the Tiers ?
• Do our staff members continue to ‘buy in’ to our
Academic/PBIS implementation?
• Have we assessed our implementation of in the areas of
Culturally Responsive Practices and Family Engagement?
Task: Developing your Plan
Questions to Consider:
• Which surveys are your schools/districts currently using?
• Do they answer your evaluation questions:
– Are we implementing RtI/SWPBIS with fidelity? (SAS, BoQ, BAT, SIR)
– Is there more information that we need? (TIC,MATT, SSS, CRP
Interview, Family Engagement Interview)
– How does our staff rate our implementation (SAS, RtI All-Staff
Perception Survey)
• Do your surveys assess ALL the Tiers you are implementing?
(TIC, ECBoQ, BoQ, MATT, BATT, TFI)
• Do you need a research tool? (SET, ISSET)
• Are you using a Universal Screener for Behavior/Social Skills?
• Are you Action Planning from your Assessments?
Task: Developing your Plan
Other questions to consider
• Are you “topping out” of the surveys you are currently
taking?
• Do you need to customize your evaluation plan for your
school, or will one evaluation plan fit for all schools in your
district?
• Are you assessing both Academic and Behavior RtI
implementation?
• Do you use Universal Screening?
• Is your Evaluation Plan manageable for your school(s)?
• Do you have a District Leadership Team that plans for
PBIS/RtI Implementation and support?
• Does your District Leadership Team Self Assess annually?
Task:
Measure/Tool
Complete this Plan for your District/School
Year One
Fall
Winter
Year Two
Spring
Fall
Year Three
Winter
Spring
Fall
Winter
Spring
Action Plan from your Evaluations!
Action Plan Resources:
Templates on PBISassessment.org for TIC,
SAS & BoQ etc. action plans
Use Action Plans available in resources area
Wisconsinpbisnetwork.org
TIPS format meeting minutes action plan
State specifically when you will review your
goals in your action plan!
Identify components you are
Working on Now!
What RtI components are already underway?
AREA:
Strengths Working
________
on now!
Universal
Selected
Intensive
Leadership
&
Organization
al Structures
BoQ/SIR Work on
Indicators next!
of
Success
School name_________________________________ Date______________
Implementation Action Plan:
Item/Section
Evidence/Data
Roadblocks
Strategies and
resources
Action:
Next Steps
Who?
When?
Goal:
Universal Screeners
For info on Selection of Universal Screeners
Go to Wisconsinpbisnetwork.org
enter Universal Screeners in Search box
Resources:
• pbisapps.org: website to connect to pbis assessment
materials , videos, resources
• Evaluation Blueprint for School-Wide Positive Behavior
Interventions & Support: Appendix A District Leadership
Assessment Tool (wisconsinpbisnetwork.org)
•
Bob Algozzine, Robert H. Horner, George Sugai, et.al
The Role of Fidelity Measures in Implementing Evidence-based
Practices: Lessons Learned from School-wide PBIS
•
Tary J. Tobin , Claudia G. Vincent, et.al