http://miblsi.cenmi.org
Evaluation in Michigan’s Model
Steve Goodman
sgoodman@oaisd.org
National PBIS Leadership Forum
October, 2011
Mission Statement
To develop support systems and
sustained implementation of a
data-driven, problem solving
model in schools to help students
become better readers with
social skills necessary for
success.
MiBLSi Project
Evaluation Team
Jennifer Rollenhagen
Measurement and
Evaluation Specialist
Anna Harms
Evaluation Coordinator
Ed Huth
Data Analyst
Nicole Matthews
Data Entry
Evaluation contributes to Project by:
•Developing and providing resources to enhance local capacity related
to measurement and evaluation, consistent with the implementation
research. Evaluation supports the competencies and capacity
necessary for implementation specialists and local districts to engage
in effective data-based decision making as part of an integrated
behavior and reading RtI model.
•Reporting on program activities and project outcomes to evaluate and
improve effectiveness and efficiencies of the project to ensure value
added to consumers and stakeholders. This is accomplished by
implementing Plan-Do-Study-Act cycles.
Several Purposes of MiBLSi Assessments
• Audit
– for “taking stock” of current
strengths/weaknesses and action
planning
• Formative evaluation
– for improving program while it is in the
process of being implemented
• Summative evaluation
– for improvement of future reiterations
Internal Evaluation
(within the project)
MiBLSi Value-Added
Work System
Feedback
Management/Coordination
Resources
Capital
People
Materials
Information
Work Systems:
Providing the RtI practices and the supports for
the these practices to take place successfully
within schools and districts
Technical
Assistance
Professional
Evaluation Financial
Learning
Stakeholders
/Funders
Investments:
• Funding
• Visibility
• Political support
Returns:
• Addressing critical issues
(Discipline/Ethnicity)
• Addressing program
directives (State
Performance Plan)
Valued RtI
Products/
Services
Consumers
(schools,
districts,
ISDs)
Organizational Level
Stake Holders
Process Level
Finance
Evaluation
Tech. Assistance
Worker
Consumers
Worker
Prof. Learning
Worker
Worker
Evaluation At Organizational, Process and Performer (worker) level:
At each , measurement takes place at determined interval. This
information is compared to established standards and provided as
feedback for the system.
Levels of Internal Evaluation
Level
Organizational
Process
Performer
(worker)
Description
Tools
Evaluating impact of the RtI project
• Stakeholders
• Addressing critical issues (e.g.,
Discipline/Ethnicity)
• Addressing program directives
(e.g., State Performance Plan)
• Consumers
• Valued results (e.g., Improved
quality program, fidelity of
implementation, successful
student outcomes)
• Annual reports
Evaluating operational productivity and
efficiency of each unit within
organization
Project management tools
• Fasttrack, MS Project
Evaluating worker productivity and
efficiency
• Job models
• Measures of program
quality
• Measures of fidelity of
implementation
• Measures of student
outcomes
• Survey of consumer
satisfaction
• Basecamp
Job Model: Measurement and Evaluation Specialist
File Maker Pro Data Base
External Evaluation
(outside the project)
Collecting information to evaluate implementation
effects and using this information for continuous
improvement
MiBLSi Project
ISD
Leadership
Team
LEA District
Leadership
Team
Building
Leadership
Team
Building Staff
•Fidelity of implementation (state)
•Systems integrity (project)
•Student success (project-wide)
•Fidelity of implementation (across districts)
•Systems integrity (district-ISD)
•Student success
•Fidelity of implementation (across schools)
•Systems integrity (district-LEA)
•Student success (district-wide)
•Fidelity of implementation (across grades)
•Systems integrity (school)
•Student success (school-wide)
•Student success/Intervention effectiveness
Assessments
Elementary Schools
• Major Discipline Referrals
• PBIS Self-Assessment Survey
• PBIS Team Implementation
Checklist
• Benchmarks of Quality (BOQ)
• Schoolwide Evaluation Tool (SET)
• Benchmarks for Advanced Tiers
(BAT)
• Dynamic Indicators of Basic Early
Literacy Skills (DIBELS)
• Planning and Evaluation Tool (PET)
for Effective Schoolwide Reading
Programs
• Effective Reading Support Team
Implementation Checklist
• Special Education Data Collection
Form
• Schoolwide Reading Analysis
Support Page
Middle/Junior High Schools
• Major Discipline Referrals
• PBIS Self-Assessment Survey
• PBIS Team Implementation
Checklist
• Benchmarks of Quality (BOQ)
• Schoolwide Evaluation Tool (SET)
• ORF/MAZE through AIMSWeb
• School-Wide Evaluation and
Planning Tool for Middle School
Literacy (SWEPT)
• Middle School Reading Team
Implementation Checklist
• Special Education Data Collection
Form
Building Level
Assist Teams in Using Data for
Decision-making
• First Year
– Winter systems review
– Spring Data Review
• Second Year
– Fall data review
– Winter data review
– Spring data review
• Third Year
– Fall data review
– Winter data review
– Spring data review
Assessment Booklet
• Description of
assessments
• Data collection
schedule
• Data summary
• Data forms and
assessment forms
Team Evaluation of Outcome, Process and
Systems Data
Assessment Schedule
(for Cohort 7 from MiBLSi website)
Video examples for
completing and
submitting PBIS
assessments
Improving the accuracy and
Consistency of Recording
Office Discipline Referrals
Example Situation
Developing Fluency
with Discipline Referral
Categories
Example Exercise 2:
Match the example
situation below to the
correct problem
behavior on the
discipline categories
answer sheet. Write
the letter in the column
for Exercise 2.
A.
Issac picked up his lunch and notice that another student was sitting in his
usual spot at the table. Issac squeezes into his spot forcing the student to
abruptly slide off the seat and onto the floor.
B.
Ian comes to school wearing
C.
During the test, Timothy is looking at Jennifer’s paper and copying her
answers.
D.
Taneshia walks to school each morning. She plays in the yard across from
the school until the bell rings. After she sees al
l the student run into the
building, she walks towards the school, late again.
E.
During lunch, Franz collects the empty milk cartons from his classmate. He
sets them on the floor in a line and then stamps on the cartons making a loud
noise with each stomp .
F.
Alexandra has been following Ben around the playground. She sometimes
makesExercise
various noises
(that seems
to bother
is near.
1
Exercise
2 him) when she
Exercise
3 Sometime
she tells him how she and her brother will stop him on the way home from
school and beat him up.
Problem Behavior
Inappropriate language
G.
Fighting physical injury
H.
Defiance/Disrespect
Lying/Cheating
I.
Harassment/Tease/Taunt
Disruption
J.
Excessive Tardiness
a T -shirt with a sexually suggestive picture.
Susan and Jill are best friends. Jill lives several houses down from the
school. During lunch Jill talks Susan into coming over to see her new puppy.
The school has a rule about staying on school property at lunch.
Billy’s teacher asks the class to take out
their math book. Billy groans and
says that math is stupid while not taking out his book. His teacher then tells
him he needs to get his book from his desk. Billy says he hates math, he
hates school and he is not going to do it.
Mr. Walter happens to walk into the boy restroom to find Matt and Chris
wadding up paper towels and stuffing them into the toilet.
Rebecca has been having difficulty with her assignments at school. Her
teacher, Ms. Swanson, sent a note home with Rebecca to be signed by her
parents and returned the next day. However, when the note was returned the
signature was Rebecca’s mother’s name but written in Rebecca’s
handwriting.
Truancy
K.
Property Damage
Forgery/ Theft
Other
The first grade class writes in a journal. The topic focuses on a particular
letter each day. During th e letter “F” day, Sarah writes a word she has
learned from some tough kids on the playground. She shows her journal to
all of the students around her.
District Level
Focus on Implementing with Fidelity
using Benchmarks of Quality (BoQ)/ODR ’06-’07 and ’07-’08
Average Change in Major Discipline Referrals:
One District Example (13 elementary schools)
10%
5%
Decrease 14.6%
0%
Increase 8%
-5%
-10%
-15%
-20%
Not Met Criteria (< 70) n=5,
range: 41-65
Met Criteria (> 70) n=8, range:
72-94
District Implementation Tracking Form
Leadership-Implementation Support Team SelfAssessment
Lesson Learned
• Teams need to be taught how to
analyze and use data
• Emphasis on directing resources to
need and removing competing activities
• As we grow, it is even more important to
systematic gather data that is accurate
and then act on the data for continuous
improvement
• More work is needed in developing
feedback cycles
“Even if you’re on the right
track, you’ll get run over if you
just sit there”
- Will Rogers