Evaluation Tools, On-Line
Systems, and Data-Based
Decision Making
Adapted from the Illinois PBIS Network
Version 3.0, Rev. 12.2.11
 This is a presentation of the Illinois PBIS Network. All rights reserved.
Agenda
Evaluation Tools:
 Team Implementation Checklist (TIC)
 Benchmarks of Quality (BoQ)
 Self Assessment Survey (SAS)
PBIS Assessment @ www.pbisapps.org
Outcomes of Session
 Introduction of Tier 1 Evaluation Tools
 Understand purpose of Tools
 Understand process of Tool completion
 Practice using the Tools
 Know where to complete surveys and
generate reports on www.pbisapps.org
Tier 1 Evaluation Tools…
 Team Implementation Checklist (TIC)
 Benchmarks of Quality (BoQ)
 Self-Assessment Survey (SAS)
Effort data (Are we working the plan?)
 Team Implementation Checklist (TIC)
 Self-Assessment Survey (SAS)
Outcome data (Is it having an effect?)
 “Big 5” Graphs
 Triangle %
Fidelity data (Are we following the plan?)
 School Evaluation Tool (SET)
 Benchmarks of Quality (BoQ)
Purpose of PBIS Evaluation Tools
 Getting Started
• Self Assessment Survey (SAS)
• Team Implementation Checklist (TIC)
 Getting to Implementation Fidelity
• Team Implementation Checklist (TIC)
• Benchmarks of Quality (BoQ)
 Sustaining Improvement
• SAS, BoQ, SET
PBIS Assessment Account
 Team Implementation Checklist 3.1
 Benchmarks of Quality
 Self Assessment Survey
Your PBIS Assessment Account
Your PBIS Technical Assistance Coordinator or
External Coach will:






Give each school a unique account number
Provide written directions on how to complete the
surveys
Set open/close time periods convenient for school(s)
to complete the survey
When a survey period has been closed the survey
results and graphs are generated automatically
Closed surveys can be reopened
You can take a survey more than once
A Brief Overview of PBIS Assessment:
www.pbisassessment.org
School Account and Login
Select “Surveys”
Access your open survey by clicking on “Launch”
PBIS Assessment Reports
Three reports are available for the Tools:
Total Score
Subscale Score
Individual Item
Team Implementation Checklist
(TIC)
Team Implementation Checklist
(TIC)


Self-assessment tool for monitoring the
implementation of school-wide PBS
Team self-assessment in six core areas
• Establish commitment
• Establish and maintain team
• Self-assessment
• Establish school-wide expectations
• Establish information system
• Build capacity for function-based support
Team Implementation Checklist
(TIC)


WHO: Completed by the Universal PBIS Team

WHERE: At your school during a Universal PBIS
Team meeting


WHEN: 3x per year
WHAT: Guides the development, implementation,
monitoring and revision process for building a
positive school culture
HOW: One person, usually the Internal PBIS Coach,
enters the data online at www.pbisapps.org
Online View of Team Checklist
Enter date the survey was completed
Enter the TIC Data
TIC Total Score Report
TIC Subscale Report
TIC Individual Items Report
TIC Activity In a Small Group…
 Review the results of the TIC Total Score,
Subscale & Individual Item Reports
 What do the survey data results say
about this Universal team’s functioning?
 What next steps would you recommend
this team include in their Action Plan?
 Why?
School-Wide Evaluation Tool
SET
School-Wide Evaluation Tool
(SET)
The SET is a research-validated instrument that
is designed to assess and evaluate the critical
features of school-wide effective behavior
support across an academic school year.
The Set was Designed To:
 To determine the extent to which schools are
already using School-wide Positive Behavior
Support (PBIS)
 To determine if training and technical
assistance efforts result in change when using
PBIS
 To determine if use of PBIS procedures is
related to valued change in the safety, social
culture, and violent behavior in schools
Information Gathered Can Be
Used To:
 Assess features that are in place
 Determine annual goals
 Evaluate on-going efforts
 Design and revise procedures
 Compare year to year efforts in the area of PBIS
SET Evaluates Questions Across
Seven Featured Areas:
 Expectations defined
 Correction procedures
 Behavioral expectations
taught
 Monitoring and
evaluation
 Acknowledgement
procedures
 Management
 District-level support
SWIS
“Big 5” Graphs
Triangle %
What are our “major” and
“minor” behavior referral forms
indicating?
What data options are available when a student
needs additional social interventions by a teacher
in the classroom (minor) or when other school
personnel need to me informed (major), how data
gets collected.
SWIS – Major & Minor
Other district based system
Report Options using SWIS:
 Average Referrals Per Day Per Month
 Referrals By Problem Behavior
 Referrals By Location
 By Student
 Referrals By Time
The Big 5
How are we doing?
Look for trends
Plan time to re-teach
By Problem Behavior
Are there target behaviors?
Behavior of the week or
Staff training/development
By Location
Look for hot spots
Review the rules for the locations
Review the lessons for teaching the
rules
By Time
By Student
How are we doing?
Look for trends
Plan time to re-teach
Do we have a
solid green
zone?
80%?
Self Assessment Survey
(SAS)
Self Assessment Survey (SAS)
 SAS assesses the extent to which PBIS
practices and systems (essential
components) are in place within a school
• School-wide
• Non-classroom
• Classroom
• Individual Student
 SAS provides entire staff (certified and
non-certified) perspective
Self-Assessment Survey (SAS)
 WHO: Completed by all teachers, staff and
administrators in your school building
 WHAT: Examines the status and need for improvement
of four behavior support systems (school-wide,
classroom, non classroom, individual students)
 WHERE: At your school
 WHEN: Annually, usually at the beginning or towards
the end of each school year
 HOW: All school staff take the survey online at PBIS
Assessment: www.pbisapps.org
Online View of
Self-Assessment Survey (SAS)
Online View of
Self-Assessment Survey (SAS)
SAS Total Score Report
• School-wide
• Non-classroom
• Classroom
• Individual
SAS Subscale Report
SAS Individual Items Report
Self-Assessment Survey Activity
with a Small Group…
 Use the SAS Individual Items Report as a
reference.
 Discuss how you might approach problem
solving with your team if the staff identified
an item or items as HIGH for Not In Place
but LOW for Improvement Priority.
Data-Based Decision Making
Data-Based Decision-Making
SWIS or SWIS Like Systems
 Developing Precise Statements
 Solution Development
• Prevention, Teaching, Reward, Extinction,
Corrective Consequence, and Data Collection
 Action Plan for Results
Using Data for On-Going
Problem-Solving
 Start with the perceptions not the data
 Use data in “decision layers”
• Is there a problem? (overall rate of ODR)
• Localize the problem
(location, problem behavior, students, time of day)
• Get specific
 Don’t drown in the data
 It’s “OK” to be doing well
 Be efficient
Six Things to Avoid






Define a solution before defining the problem
Build solutions from broadly defined, or fuzzy
problem statements
Failure to use data to confirm/define problem
Agree on a solution without building a plan for
how to implement or evaluate the solution
Agree on a solution but never assess if the
solution was implemented
Serial problem solving without decisions
Uses of Data

Review current status and identify problems early.
• Use data on a regular basis (every two weeks) to
monitor key indicators, and identify problems before
they become difficult.

Refine a problem statement to a level of precision that will
allow functional solutions.

Use data to test possible solutions.

Use data to assess if solutions are working
• If many students are making the same mistake it
typically is the system that needs to change not the
students.
Benchmarks of Quality
(BoQ)
Benchmarks of Quality (BoQ)
 The Benchmarks of Quality measures the
degree to which a school is implementing
the universal level of school-wide
positive behavior support.
 The results provide a mechanism for
school teams to identify strengths and
weakness based on fidelity criteria for
use in establishing future action plans.
Benchmarks of Quality (BoQ)
 WHO:
The PBIS Universal Team
 WHAT: Lists 10 critical elements of quality for Universal
School-wide PBIS implementation. This tool identifies
strengths and opportunities for change and growth
 WHERE: At your school
 WHEN:
The spring of each school year
Mar/Apr/May
 HOW: Online directions at www.pbisillinois.org
• PoI Narrated Instructions (PPT 2003 & 2007)
Technical Assistance Coordinator
Enters BoQ Online
BoQ Total Score Report
BoQ Subscale Report
BoQ Individual Items Report
BoQ Activity In a Small Group…
 Review the results of the BoQ Subscale
and BoQ Individual Item Reports.
 What action items would you recommend
this team consider for improving their
Faculty Commitment and Discipline
Procedures?
Review
Status and
Identify
Problems
Team Initiated
Problem Solving
(TIPS) Model
Evaluate and
Revise
Action Plan
Develop and
Refine
Hypotheses
Collect
and Use
Data
Develop and
Implement
Action Plan
Discuss and
Select
Solutions
Problem Solving
Foundations
Identifying Problems/Issues




What data to monitor
• ODR per day per month
• OSS, ISS, Attendance, Teacher report
• Team Checklist/ SET (are we doing what we planned to do?)
What question to answer
• Do we have a problem?
What questions to ask of Level, Trend, Peaks
• How do our data compare with last year?
• How do our data compare with national/regional norms?
• How do our data compare with our preferred/expected
status?
If a problem is identified, then ask
• What are the data we need to make a good decision?
Using Data to Refine Problem
Statement

The statement of a problem is important for
team-based problem solving.
•

Everyone must be working on the same problem with
the same assumptions.
Problems often are framed in a “Primary”
form, that creates concern, but that is not
useful for problem-solving.
•
•
Frame primary problems based on initial review of data
Use more detailed review of data to build “Solvable
Problem Statements.”
Precise Problem Statements
(What are the data we need for a decision?)
 Precise problem statements include
information about the Big Five questions:
•
•
•
•
•
What is problem, and how often is it
happening
Where is it happening
Who is engaged in the behavior
When the problem is most likely
Why the problem is sustaining
Primary vs. Precision Statements

Primary Statements
•
•
•
•
•
Too many referrals
September has more
suspensions than last
year
Gang behavior is
increasing
The cafeteria is out of
control
Student disrespect is
out of control

Precision Statements
•
There are more ODRs
for aggression on the
playground than last
year. These are most
likely to occur during
first recess, with a large
number of students,
and the aggression is
related to getting
access to the new
playground equipment.
Primary vs. Precision Statements

Primary Statements
•
•
•
•
•
Too many referrals
September has more
suspensions than last
year
Gang behavior is
increasing
The cafeteria is out of
control
Student disrespect is
out of control

Precision Statements
•
There are more ODRs
for aggression on the
playground than last
year. These are most
likely to occur during
first recess, with a
large number of
students, and the
aggression is related to
getting access to the
new playground
equipment.
Precise or Primary Statement?
 Children are using inappropriate
language with a high frequency in the
presence of both adults and other
children. This is creating a sense of
disrespect and incivility in the school.
 James D. is hitting others in the cafeteria
during lunch, and his hitting is
maintained by peer attention.
Precise or Primary Statement?

ODRs during December are higher than in any
other month.

Minor disrespect and disruption are increasing
over time, and are most likely during the last
15 minutes of our block periods when
students are engaged in independent seat
work. This pattern is most common in 7th and
8th grades, involves many students, and
appears to be maintained by escape from
work (but may also be maintained by peer
attention… we are not sure).
Precise or Primary Statement?
 Three 5th grade boys are name calling
and touching girls inappropriately during
recess in an apparent attempt to obtain
attention.
 Boys are engaging in sexual harassment.