8/4/2015
Systems and Data for Monitoring the
Progress of Tier 2 Interventions
and Supports
A PROJECT OF SERESC
2015 Conference on School Culture, Climate
and Positive Behavior Supports
August 13, 2015
Howard Muscott, Ed.D.
New Hampshire Center for Effective Behavioral
Interventions and Supports at SERESC
Jim Fauth, Ph.D., Director, Antioch University,
Center for Research on Psychological Practice
Presenters
Names
JimFauth,Ph.D
HowardS.
Muscott,Ed.D.
Titles
EmailContact
Director,Antioch jfauth@antioch.edu
University,Center
forResearchon
Psychological
Practice
Director,NH
hmuscott@seresc.net
CEBISatSERESC
3
Outcomes
A PROJECT OF SERESC
1. To learn strategies for extending your
array of targeted group interventions.
2. To learn strategies for evaluating the
success of targeted group interventions
3. To learn about evaluation tools suitable
for real-time monitoring and feedback
about group process and progress
4. To learn strategies for assessing
implementation of targeted group
interventions with fidelity
1
8/4/2015
TIER 1: Effective Academic and Social Instruction Practices
School-wide and Classroom Behavioral Systems for Prevention and Early Response
Nomination and Activation Procedures including Screening
High Rate Positive Teacher: Student Contacts
Effective 2-Way Home-School Communication
TIER 2: Efficient Systematic Interventions (e.g., TCCE; Simple
Behavior Plans) for Students Non-Responsive to Tier 1 Supports
Array of Evidence-Based Group Interventions
Addressing Prevalent Functions of Behavior; Available for Students
Non-Responsive to Tier 1 and Early Tier 2 Supports
TIER 3: Individualized Behavior Support Planning
Mann & Muscott
(Functional Assessment and Intervention Planning)
(2007; 2010)
For Students Non-Responsive to Tier 1 and Tier 2 Supports
School-based
Intensive Supports
Intensive Behavior
Coordinator
Links to Wraparound-NH
Support Plans and
Tier 3: School &
Facilitation
Crisis Intervention
CommunityBased
Intensive
Links to
Links to
Supports
Regional Multi-Disciplinary
Community-based
Teams
Supports
PBIS Support Systems
Supporting
Decision
Making
Outcomes
Supporting
Staff Behavior
DATA
SYSTEMS
PRACTICES
Supporting
Student Behavior
1. Tier 2 Problem
Solving Team
and Processes
6. Data-Based
Decision Making
2. Aligning
Tier 2 Team
with System
Tier 2
Secondary Prevention
Targeted Approaches 3. Communication
A Function-Based
Perspective
with Staff and
Families
Muscott & Mann (2009)
5. Targeted Group
Interventions
Universal Primary
Prevention
4. Nomination &
Activation
Processes
District-wide
Administrative Team
2
8/4/2015
Tier 2
Secondary Prevention
Targeted Approaches
6. Data-Based
Decision Making
A Function-Based
Perspective
Muscott & Mann (2009)
5. Targeted Group
Interventions
District-wide
Administrative Team
Universal Primary
Prevention
System Development is Key!
Dean Fixsen, Karen Blase, Robert Horner,
George Sugai, (2008)
•
To scale up interventions we must first scale
up implementation capacity
•
Building implementation capacity is
essential to maximizing the use of Positive
Behavior Support and other innovations
Adapted from the Illinois PBIS Network
Levels of Monitoring
Progress at Tier 2
1. Individual Student Level: How is the child
or adolescent getting the intervention
progressing?
2. Intervention Level: How is the
intervention working for the students
receiving it?
3. School/System Level: How are the array
of interventions working for the students
receiving them?
3
8/4/2015
3-Tiered System of Support
Necessary Conversations (Teams)
School-wide
Team
Plans SW &
Class-wide
supports
Universal
Support
through SW
Program
Tier 3 Team
Tier 2 Team
System
Conversations
Uses process data;
evaluates overall
effectiveness; does not
involve discussion of
individual students
Problem Solving
Conversations
Matches students to
interventions and
monitors progress,
making adjustments as
needed
Uses Process data;
determines overall
intervention
effectiveness
FBA/BSP
PTR
CICO
Group interventions
Person
Centered
Planning
Group w. individual
features
Brief
FBA/BSP
Adapted from the Illinois PBIS Network DE-PBS Project March 2015
Skill Deficits
Targeted Group Interventions
1. Check In Programs
2. Mentoring Programs
3. Social Skills Instruction
4. Emotional Regulation
5. Stress Management
6. Executive Functioning
7. Academic Support
8. Service-Learning Programs
4
8/4/2015
‘Data-Based’ in an RtI model
Means That We:
1. Know WHAT outcomes we want to
achieve with the intervention.
2. Know WHAT success looks like?
3. Know HOW progress will be assessed.
4. Know WHEN progress will be
assessed.
5. Know CRITERIA for when students
are responsive to interventions (it’s
working) and non-responsive to
interventions (it’s not working).
6. Know what to try next along a
continuum of support.
Turn & Talk: Group
Interventions
 Turn Left:
 Talk: Discuss
the group
interventions
you have in
place, if any, at
your school
 Timeframe: 5
minutes
Teacher Check, Connect & Expect
Muscott (2010; 2015)
Intervention
and Provider
Teacher
Check,
Connect
Expect
Classroom
Teachers
and
Specialists
Goals
Improve
behaviors
crucial to
school success;
relationship
with classroom
and other
teachers; homeschool
communication
Student How to
Access
Need
and #s Support
Tier II
Adult
attention, Team
structure,
regular
feedback
2 per
class
Entry
Criteria
Dosage
(Time,
Days,
Weeks)
Exit
Criteria
* Exceed
cut scores
on
benchmarks
(e.g., 3
ODR, 5
absences,
etc.)
* Elevated
screening
score on
BESS
* Teacher
and/or Tier
II Team
Arrival,
Dismissal
and small,
standardized
and agreed
upon
number of
periods per
day
18 of 20
days at
75% points
and no
more than
one major
office
referral;
1-2 minutes
times
number of
periods
18 of 20
days self
monitoring
with similar
results
5
8/4/2015
Enhancing Your Array of
Targeted Group Interventions
1. Identify group
and goal(s) to
be addressed
by the
intervention
based on
student needs.
Early and Efficient Tier II Behavior
Interventions: Where to Look
to Determine Skills
 Begin with the skills embedded in
behavioral expectations from school-wide
behavior matrix
 Which
are most problematic for at-risk
students who have not responded to
schoolwide approaches?
 Review skills being delivered in groups
 Review social skills programs
 Consider academic interventions
Types of Social Skills
1. Basic Social Skills or Classroom Survival
Skills

Listening, Asking for Help, Asking a Question
2. Friendship-Making Skills


Introducing Yourself, Beginning a Conversation
Offering Help, Sharing, Apologizing
3. Skills for Dealing with Feelings

Expressing Feelings, Showing Understanding of
Another’s Feelings, Dealing with Anger
6
8/4/2015
Types of Social Skills
4. Skill Alternatives to Aggression
 Dealing with an Accusation, Responding
to Teasing
5. Skills for Dealing with Stress
 Dealing with Losing, Saying No,
Responding to Peer Pressure
6. Social Problem Solving and Planning Skills
 Setting a Goal, Making a Decision
Addressing Functions with
Targeted Group Interventions
 Big Idea with TGIs is ensuring the availability
of efficient /effective supports targeted to
prevalent ‘functions’ of behavior, usually:
Gain Adult Attention
Gain Peer Attention
 Escape Motivated (Avoid Academic Tasks / Avoid
Social Contexts)


 At Targeted Group level, student need (i.e.,
skill strengthening) or function should
influence referral to particular support.
Targeted Group Interventions and
Functions of Behavior
 Access Adult Attention Examples:
Check-In/ Check-Out
Mentoring Programs
 Service Learning
 Access Peer Attention Examples:
 Social Skills Instruction
 Peer Mentoring
 Self-monitoring system with Peer Share Component
 Interest groups with peers with like interests
 Address academic task avoidance or social avoidance using
Academic or Social Skills teaching
 Organization Skills
 Homework Planning or Completion Club
 Tutoring
 Specific Social Skills groups


7
8/4/2015
Enhancing Your Array of
Targeted Group Interventions
2. Find an
evidencebased
intervention or
develop
lessons to
address needs.
Responding to Problem Behavior in Schools:
The Behavior Education Program
A comprehensive book by Deanne A. Crone, Robert
H. Horner, and Leanne S. Hawken.
Guilford Publishing, Inc. ISBN 1-57320-940-7; Cat. #0940
List Price: $25.00
www.guilford.com
FL PBS Project: Targeted Interventions 2003-2004
Safe & Sound Programs
Safe and Sound: An Educational Leader’s Guide to
Evidence-Based Social and Emotional Learning
(SEL) Programs.
Collaborative for Academic, Social, and Emotional Learning.
(2003). Chicago, IL: Author.
http://www.casel.org/projects_products/safeandsound.php
FL PBS Project: Targeted Interventions 2003-2004
8
8/4/2015
Skillstreaming the Adolescent
Ellen McGinnis & Arnold Goldstein
Research Press
50 Prosocial Skills in 6 Categories






Beginning Social Skills (Listening)
Advanced Social Skills (Convincing
Others)
Friendship-Making Skills (Reading
Others)
Skills for Dealing with Feelings (Dealing
with Someone Else’s Anger)
Skill Alternatives to Aggression
(Negotiating)
Skills for Dealing with Stress (Standing
Up for a Friend)
Mentoring Program Evaluation
Newcomer (2005)
Decreases in:
 Meetings with counselor
 Office referrals
 Time outs
 Suspensions
 Detentions
Increases in:
 Student attendance
 Work completion
 Academic performance
 Homework completion
 Parental involvement
 Positive student-teacher
interaction
Enhancing Your Array of
Targeted Group Interventions
3. Determine the
qualified
person(s) with
time for
implementation.
9
8/4/2015
Turn & Talk:
Qualified Personnel
 Turn Right:
 Talk: Think about a
group intervention.
Who might the
qualified people be
who could
implement with
fidelity?
 Timeframe: 5
minutes
Mentor’s Role
Tim Lewis (2014)
 To provide guidance, support, and
encouragement for the student while
modeling such skills as effective
communication, empathy and concern for
others, and openness and honesty
 Commitment for entire academic year
 Include staff that are invested in students
Enhancing Your Array of
Targeted Group Interventions
4. Determine
the frequency
of the
intervention
(minutes,
days per
week,
weeks).
10
8/4/2015
Enhancing Your Array of
Targeted Group Interventions
5. Determine
the ideal
group size,
age/grade
range and
number of
groups.
Enhancing Your Array of
Targeted Group Interventions
6. Determine the
entry criteria for
identifying the
right students who
could benefit from
the intervention
and how they will
access support.
BEP Data-Based Decision
• The BEP is indicated when…
– High percentage of students with multiple
referrals (>15%)
– Problem behaviors are not dangerous or
violent
– Problem behaviors are found in multiple
locations throughout school, from multiple
staff
– Adult attention is reinforcing
FL PBS Project: Targeted Interventions 2003-2004
11
8/4/2015
BEP Example Behaviors
–
–
–
–
–
–
–
–
–
–
–
Disruptive
Talks out
Unprepared
Talks back to teacher
Uses inappropriate language
Tardy
Defiant
Refuses to do work
Difficulty taking turns
Refuses to share
Out of seat
– Difficulty following
directions
– Frequent peer conflict
– Low-grade aggression
(pushing, tripping, etc…)
FL PBS Project: Targeted Interventions 2003-2004
BEP Do Not Include:
Dangerous/violent students
Students who bring a weapon to school
Students who injure/may injure themselves
Students with a high number of referrals
Students with referrals from only one setting,
teacher, or time
• Students who find adult attention aversive
•
•
•
•
•
FL PBS Project: Targeted Interventions 2003-2004
Data-Based Decision:
Entrance Criteria
 Anger management training may be
indicated when:
 High
rates of referrals for
Fighting/Physical Aggression,
Disruption, Abusive Language,
Harassment, Defiance, Vandalism,
Weapons
 Student interviews indicate repeated
instances where students reacted in anger
without thinking
12
8/4/2015
Enhancing Your Array of
Targeted Group Interventions
7. Determine the
specific student
outcomes for
successful
participation in the
group (e.g., What
knowledge or
skills will they
have learned?).
Assessing Student Progress in
Group Interventions: What
Knowledge or
Information
Skills in Group
Skills in Other
Contexts:
Generalization
 Prosocial Behaviors
 Coping Strategies
 Stress Reduction
 Problem Solving
 Study Skills
 Organization Skills
 Homework
Completion
Generalization and Transfer
of Training
1. Teaching skills in multiple
settings.
2. Having different adults teach
the skills.
3. Having the students practice
the skills under different
conditions, with different
people and in different places.
4. Providing specific homework
activities.
13
8/4/2015
Zones of Regulation
 Group Outcome(s): To help students gain skills in the
area of self-regulation.
 Group Skills Taught:
The child can identify their feelings' intensity
The child can recognize personal triggers
 The child can judge the size of a problem
 The child can judge how their behavior might affect
others
 The child can develop a repertoire of "tools" for selfregulation.


Conversation Group
 Group Outcome(s): Improvement in initiating and
sustaining conversations
 Group Skills Taught:
The child can initiate a conversation with another
child.
 The child can sustain a conversation with another
child.
 The child can initiate a conversation with an adult.
 The child can sustain a conversation with an adult.

Whitcomb School
Positive Academic Work Skills (PAWS)
 Group Outcome(s): Improvement in organization and
study skills.
 Group Skills Taught:
The student uses one or more specific strategies for
studying for a quiz or test.
 The student keeps their binder organized.
 The student uses a daily planning system.
 The student completes homework assignments on
time.

14
8/4/2015
Turn & Talk:
Determining Student Outcomes
 Turn Right:
 Talk: Pick a group
intervention and
brainstorm student
outcomes.
Timeframe: 5
minutes
Enhancing Your Array of
Targeted Group Interventions
8. Determine
the
assessments
you will use
to measure
whether
students met
the outcomes.
Assessing Student Progress in
Group Interventions
What?
Knowledge or
Information
Skills in Group
Skills in Other
Contexts:
Generalization
How?
 Quiz; Test
 Criterion-referenced
Checklist
 Surveys:
Interventionist,
Teacher, Student,
Family
 Direct Observations
 Incident Reports
15
8/4/2015
Name:____________________________________
2 =No documentable majors or minors
PERIOD
Caring
Date:___________________
1 = No more than 1 documentable minor
Honesty
Respect
0 = Major or 2 documentable minors
Responsibility
Academic
Achievement
Total points
1 Arrival
__________/10
2
__________/10
3
__________/10
4
__________/10
5
__________/10
6
__________/10
Total
__________/60
__________/%
At each check in remember to consider each HIGH 5 expectation separately. For example, a student who receives a
minor for being unsafe may still receive full points for meeting the other 4 expectations.
Total Checks:______/60 = _____%
_______________________________________________________________________________________
Name:_________________________________________________
Date:______________________
Goal Met: ____Yes ____No
Goal: 45 points
Sandown North Elementary School
Teacher Check, Connect & Expect
Sandown North Elementary School
Teacher Check, Connect & Expect
16
8/4/2015
Sandown North Elementary School
Teacher Check, Connect & Expect
20 students given the intervention
14 of 20 (70%) success (averaged 80% or better)
4 partial (20%) success (averaged 70-79% and variable)
2 (10%) non-responders (averaged less than 50%)
Conversation Group
 Group Outcome(s): Improvement in initiating and sustaining
conversations
 Group Skills Taught:
The child can initiate a conversation with another child.
The child can sustain a conversation with another child.
 The child can initiate a conversation with an adult.
 The child can sustain a conversation with an adult.
 Develop a Criterion Referenced Assessment for Interventionist and
Teacher(s)
 All of the Time
 Most of the Time
 Some of the Time
 Never


17
8/4/2015
Zones of Regulation
 Group Skills Taught:





The child can identify their feelings' intensity
The child can recognize personal triggers
The child can judge the size of a problem
The child can judge how their behavior might affect others
The child can develop a repertoire of "tools" for selfregulation.
 Assessments:
 Quiz’s on knowledge of each skill
 Criterion referenced assessment of use of skill
 Reduction in major incidents
PAWS
Decrease in Referrals - Mentor Intervention
Fall 2000
Fall 2001
20
AVERAGE PERCENT DECLINE IN REFERRALS
18
62%
16
14
12
10
8
6
4
2
0
BT
BB
BK
BD
Cca
CR
CC
DD
DJ
FA
FD
HK
LD
PS
RJ
RJM
SC
SB
ST
TT
Student
18
8/4/2015
Student-focused monitoring and
Feedback
MEASURE(S)
FEEDBACK
MECHANISM
SUPPORT TOOLS
(OPTIONAL)
Evidence for Feedback Systems
Clinical Feedback works
Clinical Feedback works better than
Training in an EBP (yes, including for children)
Training in general (of any persuasion)
Supervision
CEU’s
Clinical Feedback possesses superior cost-utility, reach,
and sustainability characteristics
CLEARLY THE BEST “QI” STRATEGY
How/why does it work?
No Feedback
Feedback
57
19
8/4/2015
Evidence-based System
Outcome
Outcome Rating Scale
4 items, visual analogue, modeled after OQ-45
Other
Session Rating Scale
4 items, visual analogue; modeled after WAI
Transmission
In session; hardcopy or electronic
Age/Context
All
Research
Anker, M., Duncan, B., & Sparks, J. (2009). Using Client Feedback to
Improve Couple Therapy Outcomes: A Randomized Clinical Trial in a
Naturalistic Setting. Journal of Consulting and Clinical Psychology, 77,
693-704.
Costs
Cheap: free for paper, $100 for “Asist,”
“My Outcomes?”
Bottom Line
Simple and cheap but powerful
Best bet for the solo practitioner
Adapt to your context,
population, intervention
 Design around your intervention goals and
criteria
 Simple/crude measures often good enough
 Administer frequently
 Feedback progress information in as close to real
time as possible, to those in a position to do
something about it in the near term
 Investigate and adjust accordingly (see decision
support slide)
Decision support
Monitor Outcome + Deliver Feedback to Clinician
OT
NOT
Goals met
or
plateau?
Alliance
?
Readiness?
ADDRESS
AS
NEEDED
CONTINUE
COSOLIDATE
MAINTAIN
TERMINATE
Life?
Support?
Meds?
20
8/4/2015
Turn & Talk:
Assessing Student Outcomes
 Turn Right:
 Talk: Pick a group
intervention and
brainstorm potential
assessments.
 Timeframe: 5
minutes
Enhancing Your Array of
Targeted Group Interventions
9.
Identify exit
criteria for
determining
whether
intervention
was
successful
Teacher Check, Connect and Expect:
Assessing Progress
A review meeting should occur 4 weeks
(20 school days) after start of the program.
During the meeting, data is shared
regarding goals and determination of next
steps is made.
Success for Basic TCCE is 75% for 16 of
20 days
Success for Generalization is 75% for 18 of
20 days
21
8/4/2015
Teacher Check, Connect and Expect:
Assessing Progress Based on Data
A. Success for 4 weeks – Self monitoring 4 weeks
B. Partial Success -- Adapt TCCE to Basic Plus:
Add behavioral specificity (i.e., target a specific behavior
such as ‘completed classwork’ under ‘Responsible’)
 Add reinforcement
 Add teaching component
 Change aim line (70%)
 Continue TCCE, but add an additional support (i.e., group
intervention)
C. Discontinue TCCE and refer to secondary support team
(according to school process) to assess ‘function of behavior’
and access group or individualized supports
D. Discontinue TCCE with no additional support – monitor
progress

Conversation Group
 Group Outcome(s): Improvement in initiating and sustaining
conversations
 Group Skills Taught:
The child can initiate a conversation with another child.
The child can sustain a conversation with another child.
 The child can initiate a conversation with an adult.
 The child can sustain a conversation with an adult.
 Develop a Criterion Referenced Assessment for Interventionist and
Teacher(s)
EXIT CRITERIA
 All of the Time
Student will be able to demonstrate
 Most of the Time
all 4 skills most of the time for a two
week period both in and outside the
 Some of the Time
group.
 Never


Whitcomb School
Positive Academic Work Skills (PAWS)
 Group Skills Taught:
The student uses one or more specific strategies for
studying for a quiz or test.
 The student keeps their binder organized.
 The student uses a daily planning system.
 The student completes homework assignments on
time.
EXIT CRITERIA

Student will be able to: (1)
demonstrate the first three skills most
of the time for a two week period; and
(2) complete 80% or more of
homework assignments.
22
8/4/2015
Enhancing Your Array of
Targeted Group Interventions
10. Determine how
you will assess
fidelity of
implementation.
Assessing Fidelity of
Implementation
Key Questions
Was the program implemented as designed?
How effectively was the program implemented?
1.
2.
3.
4.
Did the number of sessions occur?
Were the lessons taught?
Were the critical features of instruction used?
Were the students present?
Strategies
1.
2.
3.
Logs: Attendance, Critical Features
Staff/interventionist self-assessments
Walkthroughs by peers, administrators
Assessing Fidelity of
Implementation: Self Assessments
 Use a 5 point rating scale from low implementation to high
implementation
 Determine questions that address the critical features of the
intervention.
 Check In Check Out




I checked in with each student every period I was supposed
to.
I asked the student to provide examples of how he showed
each trait.
I stressed the positive even if my student did not earn all
his/her points for that period?
I started off each rating period with a clean slate and was
optimistic my student could be successful.
23
8/4/2015
Fidelity Assessment Development Tool
Turn & Talk:
Assessing Fidelity of Implementation
 Turn Right:
 Talk: Pick a group
intervention and
brainstorm ways
you might assess
fidelity of
implementation.
 Timeframe: 5
minutes
Acknowledgements






Lucille Eber
Tim Lewis
George Sugai
Rob Horner
Hill Walker
Doug Cheney
 Kathleen Lane
 Delaware PBIS




Project
Florida PBIS Network
Eric Mann
Julie Prescott
Valarie Dumont
24