DO WE SAY “ALL” BUT
MEAN “SOME”?
Data-driven Conversations
on Equity & Disproportionality
Bert Eliason & Katie Conley
PBIS Applications
NWPBIS Network Conference 2016
Portland – Red Lion Jantzen Beach
Educational and Community Supports
Educational and Community Supports (ECS) is a research unit within the
College of Education at the University of Oregon.
• ECS focuses on the development and implementation of practices that
result in positive, durable, and scientifically substantiated change in the
lives of individuals.
• Federal and state funded projects support research, teaching,
dissemination, and technical assistance.
PBIS Applications (PBISApps) is a series of educational tools created
within ECS and related to the implementation of multi-tiered systems of
support (MTSS).
•
The PBIS Application tools have been utilized in 25,000+ schools both domestically
and internationally.
www.pbis.org
School  Equity & PBIS
Data Sources: What is Necessary?
Required features:
• Consistent data collection
• Discipline referrals (ODRs)
• Student race/ethnicity
• School enrollment by race/ethnicity
• Instantaneous access for school teams—not just district
teams
• Capability to disaggregate ODRs and patterns by
race/ethnicity
• Capability to calculate risk indices and risk ratios by
race/ethnicity
Data Sources: What is Recommended?
Standardized forms with the 5W’s
• Who, what, when, where, why
Clear definitions of problem behaviors
Clear guidance in discipline procedures
• Office vs. staff-managed
Report generation
• On demand
• Disaggregated by
race/ethnicity
• Automatic calculation of
disproportionality
Problem Solving Model
Why is it
happening?
Is there a
problem?
Is the plan
working?
1. Problem
Identification
2. Problem
Analysis
4. Plan
Evaluation
3. Plan
Implementation
What should
be done?
Step 1: Problem Identification
Problem-solving approach
Is there a
problem?
1. Problem
Identification
• Use valid & reliable metrics
• Quantify the difference between
current outcomes and goals
• 62% of students have 0-1 ODR
(at least 80% recommended)
• 38% of students have 2+ ODRs
For disproportionality
• Quantify outcomes across
racial/ethnic groups
• Compare differences
• Racial Subgroups vs. White
Multiple metrics are recommended
Step 1: Problem Identification
Common Metrics
Risk Index
• Percent of a group at risk for an outcome
(e.g., receiving an ODR)
Number of Black Students with 1 or more ODR 41
= 70 = .59
Number of Black Students Enrolled
SWIS Ethnicity Reports
Referral Risk Index
Step 1: Problem Identification
Common Metrics
Risk Ratio
Risk index for one group divided by the risk index for
comparison group
• Comparison group is usually White students
• 1.0 is equal risk
• > 1.0 is overrepresentation
• < 1.0 is underrepresentation
Risk Index of Target Group
Risk Index of Latino Students
.82
Risk Index of
Comparison Group
Risk Index of White Students
.65
Risk Index of Black Students
.59
Risk Index of White Students
.65
= 1.27
= 0.91
Step 1: Problem Identification
Common Metrics
Risk Ratio
• Free Risk Ratio Calculator from Wisconsin RtI Center
www.wisconsinrticenter.org
Step 1: Problem Identification
Common Metrics
Composition – Percent of Students with Referrals
• Compares subgroup’s percentage of school population to the
subgroup’s percentage of just the students with ODRs
• Is the number students with ODRs for one subgroup
proportionate to that subgroup’s size within the school?
SWIS Ethnicity Reports
Students with Referrals
by Ethnicity
Step 1: Problem Identification
Common Metrics
Composition – Percent of Total Referrals
• Compares subgroup’s percentage of school population to
the subgroup’s percentage of all ODRs written
• Is the number of ODRs for one subgroup proportionate to
that subgroup’s size within the school?
SWIS Ethnicity Reports
Referrals by Ethnicity
Step 1: Problem Identification
Procedure
1. Select metrics to use
2. Calculate metrics and compare to goals
• Previous years from same school
• Local or national norms
• 2011-2012 U.S. public schools using SWIS with at least 10
African American and 10 White students
• Median risk ratio (African American to White) = 1.84
•
25th percentile = 1.38
• Logical criteria
• U.S. Equal Employment Opportunity Commission (EEOC)
•
Disparate impact criterion – “4/5s Rule”
•
Goal risk ratio range between .80 and 1.25
School Example: Rainie Middle School
School-wide Information System (SWIS)
• Calculate risk indices
• Calculate risk ratios
• African American = 3.2 (significant)
• Use US EEOC risk ratio goal range
• Latino/a = 1.1
60
50
40
30
20
10
0
48 Percent of School Enrollment
Socio-economic
31
Paid
28%
20
1
White
African Latino/a
American
Asian
FRL
72%
Step 2: Problem Analysis
Why is it
happening?
Is there a
problem?
1. Problem
Identification
2. Problem
Analysis
Step 2: Problem Analysis
Purpose: Identify underlying causes of the problem
Focus: Systems & practices that can be changed
Evaluate: Is the disproportionality identified
consistent across all situations or more pronounced
in some situations?
• Explicit bias vs. Implicit bias
• Disproportionality in all settings vs. Specific settings
Implicit Bias
Unconscious, automatic
Based on stereotypes
We all have it (even those affected by it)
Generally not an indication of our beliefs and values
More likely to influence:
• Snap decisions
• Decisions that are ambiguous
A Unidimensional View of Bias
Racial
Bias
Disproportionate
Discipline
A Multidimensional View of Bias
Racial
Bias
Vulnerable
Decision Point
Disproportionate
Discipline
Situation
What is a Vulnerable Decision Point
A specific decision that is more vulnerable to
effects of implicit bias
Two parts:
• Elements of the situation
• The person’s decision state (internal state)
Two Systems for Decision Making
 System 1: Fast Decisions
• Automatic, snap judgments
• Intuitive, unconscious
 System 2: Slow Decisions
• Deliberate decisions
• Allows for conscious attention
Kahneman, 2011
VDPs from national ODR data
Subjective problem behavior
• Defiance, Disrespect, Disruption
• Major vs. Minor
Non-classroom areas
• Hallways
• Playgrounds
Afternoons
Step 2: Problem Analysis
Purpose: Identify underlying causes of the
problem
Focus: Systems and practices that can be
changed
Step 2: Problem Analysis
Defining Disproportionate Discipline with Precision
• Who is involved?
• What are the problem behaviors?
• Where is it happening?
• When is it happening?
• Why are these things happening?
• Perceived function of problem behavior
Step 2: Problem Analysis
Vulnerable Decision Point (VDP)
Identified Subgroup
Location
Time of Day
Problem Behavior
Motivation
Assess
• PBIS implementation fidelity
• Achievement gap
Many sixth grade Latino/a students in the 7th grade are more likely to receive referrals from
the classroom, cafeteria, & commons for inappropriate language and physical aggression.
Referrals are perceived to be task avoidance and getting adult attention.
SWIS Ethnicity Reports & Drill Down
Total enrollment is approx. 25% Latino,
but of total number of students with
referrals 30% of the them are Latino/a.
Total enrollment is approx. 25%
Latino, but almost 29% of the total
referrals are given to Latino/a
students.
There are 123 Latino/a students enrolled and
101 of them have been given a referral. With a
risk index of 0.82, 82% of Latino/a students are
at risk for or have already received a referral.
SWIS Drill Down
Precise Problem
Statement
SWIS Drill Down
Subgroup: Latino/a Students
Who?
When?
What?
M-Disruption,
Inappropriate
Language,
Defiance
3rd grade
After 12:00 PM
4th grade
8:00 AM– 11:30 AM M-Defiance
7th grade
After 12:00 PM
Inappropriate Lang.,
Physical Aggression,
M-Inappropriate Lang.
Where?
Why?
All settings
Avoid task
Classroom
Avoid task
Classroom
Cafeteria
Commons
Avoid task
Obtain Adult
Attention
Precise Problem Statements
 Latino/a students in the 3rd grade are more likely to receive
referrals in the afternoon across all settings for disrespect,
inappropriate language, and disruption. Referrals are related
to task avoidance.
 Latino/a students in the 4th grade are more likely to receive
referrals from the classroom during the morning instructional
block for defiance. Referrals are related to task avoidance.
 Latino/a students in the 7th grade are more likely to receive
referrals from the classroom, cafeteria, & commons for
inappropriate language and physical aggression. Referrals are
related to task avoidance and getting adult attention.
School Example: Rainie Middle School
RMS PBIS Team Meeting Decisions
1. Use SWIS Drill Down to build precise problem statements
2. Assess PBIS implementation
3. Assess academic gaps
School Example: Rainie Middle School
Vulnerable Decision Point (VDP)
Latino/a 6th grade students
Identified Subgroup
Location
Time of Day
Problem Behavior
Motivation
are receiving referrals
throughout the day
in the classroom, cafeteria, &
commons
for inappropriate language
and dress code violations.
Referrals are perceived to
be related to getting peer
and adult attention.
School Example: Rainie Middle School
Assess PBIS implementation
TFI shows Tiers I, II, and III meet benchmark
Team still wants to work on improving ODR process
Major vs Minor
Office vs Staff
Tiered Fidelity Inventory (TFI)
School Example: Rainie Middle School
Assess academic gaps
African American and Latino/a students
lagging in reading proficiency.
100
80
60
40
20
0
29
71
10
56
48
Reading
90
44
52
Not Met
Met
Step 3: Plan Implementation
Why is it
happening?
Is there a
problem?
1. Problem
Identification
2. Problem
Analysis
3. Plan
Implementation
What should
be done?
Step 3: Plan Implementation
Information from Step 2 is used to select
strategies.
An action plan is created to ensure adequate
implementation of the strategies.
Action plans show everyone -
WHO will do WHAT by WHEN.
Step 3: Plan Implementation
Options
• All issues
• Calculate and share disproportionality data regularly
• Inadequate PBIS implementation
• Implement core features of PBIS to establish a foundation of support
• Misunderstandings regarding school-wide expectations
• Enhance culturally-responsive PBIS with input from the students/families
• Academic achievement gap
• Implement effective academic instruction
• Disproportionality across all settings (indicating explicit bias)
• Enact strong anti-discrimination policies that include accountability
• Disproportionality in specific settings (indicating implicit bias)
• Neutralize vulnerable decision points
School Example: Rainie Middle School
Step 4: Plan Evaluation
Why is it
happening?
Is there a
problem?
Is the plan
working?
1. Problem
Identification
2. Problem
Analysis
4. Plan
Evaluation
3. Plan
Implementation
What should
be done?
Step 4: Plan Evaluation
Evaluation Time Frame:
Identify time periods for evaluating
disproportionality data
Caution: Disproportionality metrics may not be
sensitive to rapid change
• Consider monthly assessment of implementation &
quarterly assessment of disproportionality metrics
• Avoid using risk indices as they will increase
throughout the year
Step 4: Plan Evaluation
1. Assess progress and fidelity of plan
implementation
2. Calculate metrics from Step 1
3. Compare to the goal determined in Step 1
4. Share results with relevant stakeholders
School Example: Rainie Middle School
• 6th grade team may need a
refresher on office vs. staff-managed
behaviors
African American ODR Risk Ratio
4
3.5
3
2.5
2
1.5
1
0.5
0
Risk Ratio
October
January
March
June
• Revise action plan for
next year
• Continue evaluation cycle
Problem Solving Model
Why is it
happening?
Is there a
problem?
Is the plan
working?
1. Problem
Identification
2. Problem
Analysis
4. Plan
Evaluation
3. Plan
Implementation
What should
be done?
Data-driven Conversations on
Equity & Disproportionality
Use your available tools and resources
to make sure you are offering all
students their best chance for success.
Bert Eliason & Katie Conley
PBIS Applications
training@pbisapps.org
NWPBIS Network Conference 2016