Digging a Little Deeper
How to Use State Accountability and PBMAS
to Strengthen Program Effectiveness
Texas Education Agency
Heather Christie and Lizette Ramos
Copyright © Texas Education Agency 2014. All rights reserved.
Purpose
A review of identification and interventions for
both state accountability and the Performance
Based Monitoring Analysis System (PBMAS).
How to use information from both systems to
conduct meaningful data analysis and needs
assessment.
How engaging in the TAIS continuous
improvement process improves student
performance and program effectiveness.
Copyright © Texas Education Agency 2014. All rights reserved.
State Accountability Rating System 2014
Performance
Indexes
Non-AEA District
and Campus
Targets
AEA District and
Campus Targets
Index 1-Student
Achievement
55
30
Index 2-Student*
Growth
D – 16; Elem – 33; MS - 28
N/A
Index 3-Closing
Performance Gaps*
D-28, Elem-28, MS-27,
HS-31
11
Index 4Postsecondary
Readiness
All
Components
All
Components
Graduation
/Drop-out
33
45
57
STAAR
Only
District 13
Elem 12
Middle 13
High
School 21
* Set at or about 5th percentile
What is Performance-Based Monitoring?
A district-level, data-driven monitoring system
developed and implemented annually since 2004
Established to shift from process to results-based
monitoring—program effectiveness and student
performance
Copyright © Texas Education Agency 2014. All rights reserved.
What is Performance-Based Monitoring?
Bilingual
Education/English
as a Second
Language (BE/ESL)
Career and
Technical
Education
(CTE)
No Child Left
Behind
(NCLB)
Copyright © Texas Education Agency 2014. All rights reserved.
Special Education
(SPED)
Guiding Principles of the PBMAS
 School District Effectiveness
 Statutory Requirements
 Indicator Design
 Maximum Inclusion
 Individual Program Accountability
 High Standards
 Annual Statewide Evaluation
 Public Input and Accessibility
 System Evolution
Copyright © Texas Education Agency 2014. All rights reserved.
Accountability and PBMAS Correlations
State Accountability System
PBMAS
Index 1-Student Achievement
System Safeguards
STAAR 3-8 Passing Rates
EOC Passing Rates
Index 3-Closing Performance Gaps
STAAR 3-8 Passing Rates
EOC Passing Rates
Index 4-Postsecondary Readiness
Annual Dropout Rate
RHSP/DAP Diploma Rate
Graduation Rate
Copyright © Texas Education Agency 2013. All rights reserved.
Data
Process
Reporting
State Accountability
• Districts
• Campuses
Performance-Based
Monitoring Analysis
System (PBMAS)
• Districts
Federal Requirements
(ESEA Waiver)
• Priority campuses
Data
Analysis
Needs
Assessment
Implement &
Monitor
Improvement
Plan
Targeted
Improvement Plan
with Quarterly
Progress Reports
Example
District PBMAS report has a
performance level 3 on indicator 10
TELPAS composite rate over
multiple years
TAIS Continuous Improvement
Process
Data Analysis
Needs
Assessment
Implement &
Monitor
Improvement
Plan
District Data Analysis
General Questions
INDEX 1
INDEX 2
INDEX 3
PBM Questions
INDEX 4
RF Questions
CSF Data Sources
Identify Problem Statements
Data
Analysis
Closer Look – PBM Questions
• Trends across programs
• Indicators with performance level (PL 2 or 3)
• Longitudinal reflection
• Campus contributions
Data
Analysis
How do I know
which
campuses are
contributing to
the PBMAS
report?
Data
Analysis
Copyright © Texas Education Agency 2014. All rights reserved.
Data Analysis – Campus Contribution
 Review TELPAS composite scores for students who have
5 or more years in US schools
Data
Analysis
Copyright © Texas Education Agency 2014. All rights reserved.
Are ELLs with 5 or more years in US schools scoring
beginning or intermediate?
Page 32 of 2014 PBMAS Manual
Calculation
number of LEP students in Grades 5-12 in U.S. schools five or more years
who received a TELPAS Composite Rating of Beginning or Intermediate
number of LEP students in Grades 5-12 in U.S. schools five or more years
who received a TELPAS Composite Rating
Data
Analysis
Copyright © Texas Education Agency 2014. All rights reserved.
Data Analysis – Campus Contribution
 Review TELPAS composite scores for students who have 5
or more years in US schools
 Compare campus data to PBMAS standards for that
indicator
 Rank order campus performance
 What does the data tell you?
 Now, we have to dig a little deeper…
Data
Analysis
Copyright © Texas Education Agency 2014. All rights reserved.
District Data Analysis
General Questions
INDEX 1
INDEX 2
INDEX 3
PBM Questions
INDEX 4
RF Questions
CSF Data Sources
Identify Problem Statements
Data
Analysis
Closer Look –
Support
Systems and
CSFs
Data
Analysis
Closer Look
Support Systems
Critical Success Factors
Capacity and Resources
Improve academic performance
Quality data to drive instruction
Communications
Leadership effectiveness
Increased learning time
Processes and Procedures
Family/community engagement
School climate
Organizational Structures
Teacher quality
Data Analysis - Next Steps
Conversations
Observations
Reviews
Surveys
Copyright © Texas Education Agency 2014. All rights reserved.
Data
Analysis
CSF Data Sources
Data
Analysis
District Data Analysis
General Questions
INDEX 1
INDEX 2
INDEX 3
PBM Questions
INDEX 4
RF Questions
CSF Data Sources
Identify Problem Statements
Data
Analysis
Data Analysis –
Develop Problem Statements
What does the review of data tell me
Not all campuses are implementing the
designated program model.
We have to digger a little deeper…
Data
Analysis
Copyright © Texas Education Agency 2014. All rights reserved.
Closer Look –
Problem Statements
Problem Statement
Missed Index(es)
PBM Indicators
PL 2 or 3
Data
Analysis
TAIS Continuous Improvement
Process
Data Analysis
Needs
Assessment
Implement &
Monitor
Improvement
Plan
Needs Assessment Tab Format
Data Tab
Problem Statements
Root Cause 1
Root Cause 2
Root Cause 3
Root Cause 4
Root Cause 5
And so on…..
Answers the
question WHY?
Needs
Assessment
Language Shift for Clarity
Need
Root Cause
Needs
Assessment
Needs Assessment
Determine Root Causes
All campuses are not implementing the designated
program model
You will have to dig a little deeper
To determine true root causes
Needs
Assessment
Copyright © Texas Education Agency 2014. All rights reserved.
Data Tab
Campuses not implementing
program model
Needs
Assessment
WHY?
• Inconsistent understanding of program
model among teachers
WHY?
• Training take aways not being implemented
• No follow-up or coaching WHY?
WHY?
• Not a focus in classroom walk-throughs
• Principals not trained and no monitoring
expectations set
ROOT CAUSE
Needs Assessment – Narrowing the Focus
District systems are not evident, but
Some campuses are meeting the needs
of ELLs better than others
Digging Deeper…..
Needs
Assessment
Copyright © Texas Education Agency 2014. All rights reserved.
Needs Assessment – Narrowing the Focus
What are those campuses doing?
How can that be shared with other
campuses? May become part of
strategy/intervention
section
ofmove
targeted
Does it make
sense to
those
improvement
plan
campuses practices
into district
systems/expectations?
Copyright © Texas Education Agency 2014. All rights reserved.
Needs
Assessment
Needs Assessment – Narrowing the Focus
 District systems in place
 Campus staff understand systems
 Campus staff are implementing the
systems as designed
 Now what???
 Are our systems effective?
 You may have to dig a little deeper.
Copyright © Texas Education Agency 2014. All rights reserved.
Needs
Assessment
TAIS Continuous Improvement
Process
Data Analysis
Needs
Assessment
Implement &
Monitor
Improvement
Plan
Questions to Consider
Annual Goals
 Do the annual goals address the areas of low
performance? Are they linked to the problem
statement?
Strategies
 Do the strategies directly address the root causes?
Quarterly
Goals
 Do the quarterly goals ensure the implementation
of the strategies and accomplish the annual goals?
Interventions
 Do the interventions reflect specific actions that
will be taken to achieve the quarterly goals?
Copyright © Texas Education Agency 2014. All rights reserved.
Improvement
Plan
Not all campuses are implementing the
designated program model
100% of campuses are fully implementing
district program model by end of 2014-2015
school year
Improvement
Plan
Principals not trained and no expectations
for walk-through monitoring
Establish written district expectations for classroom
walk-throughs to be incorporated into principal
trainings
Quarterly Planning
Goals and Interventions
100% Campuses
Implementing
Program Model
Improvement
Plan
25%
25%
25%
25%
Improvement
Plan
Copyright © Texas Education Agency 2014. All rights reserved.
25%
25%
25%
25%
Improvement
Plan
Copyright © Texas Education Agency 2014. All rights reserved.
Interventions
Quarterly “To Do” List
Quarterly Goal Examples
Q1 - 100% of administrators with bilingual programs
will conduct 1 walk-through with district support
staff by October 15, 2014 to begin inter-rater
reliability.
Q2 - Each campus with a BE program will conduct
10 walk-throughs each month with a 70% correct
implementation rate by January 20, 2014.
Quarterly Goal Examples
Q3 - District program support specialist will complete
observation and provide coaching for each classroom
below a 60% correct implementation rate by March 15,
2014, in an effort to increase implementation.
Q4 - District program support specialists will meet with
each campus principal of a bilingual program to share
staff implementation survey results and TELPAS scores
and will create a campus support plan for 15-16 by June
15, 2014.
Q1 (Aug, Sept, Oct)
100% of administrators with bilingual
programs will conduct 1 walk-through Q1 Goal:
with district support staff by October
15, 2014 to begin inter-rater reliability. Interventions:
1) Survey principals and staff
district protocols for BE
2) Create
program walk-throughs
3) Train principals
completes at least 1 walk4) Principal
through to promote inter-rater reliability
What data will be collected to monitor interventions?
Survey results indicating
1)
principal needs
District expectations for BE program
2)
implementation walk-throughs
Sign-in sheet, agenda, training
3)
materials
Walkthrough results to review
4)
inter-rater reliability
Q2 (No
Inter
What
collecte
inter
TAIS Continuous Improvement
Process
Data Analysis
Needs
Assessment
Implement &
Monitor
Improvement
Plan
Q1 (Aug, Sept, Oct)
100% of administrators with bilingual
programs will conduct 1 walk-through Q1 Goal:
with district support staff by October 15,
Interventions:
2014 to begin inter-rater reliability.
1) Survey principals and staff
district protocols for BE program walk2) Create
throughs
3) Train principals
4) Principal completes at least 1 walkthrough to promote inter-rater reliability
What data will be collected to monitor interventions?
1) Survey results indicating principal needs
District expectations for implementation of
2) BE program walk-throughs
3) Sign-in sheet, agenda, training materials
4) Walkthrough results to review inter-rater
reliability
Q2 (Nov,
Interve
What da
collected
interve
Today’s Key Take-Aways
Use of multiple data sources
Collaboration and coordination of effort
Be proactive
Commitment to continuous improvement
Copyright © Texas Education Agency 2014. All rights reserved.
Resources
TEA website (www.tea.state.tx.us)
A-Z Index
2014 Accountability
Performance-Based Monitoring
Texas Center for District and School Support
(www.tcdss.net)
Contact Information
Division of Program Monitoring and
Interventions
Phone: (512) 463-5226
Email: PMIdivision@tea.state.tx.us
Lizette Ramos lizette.ramos@tea.state.tx.us
Heather Christie heather.christie@tea.state.tx.us
Copyright © Texas Education Agency 2014. All rights reserved.
•
•
•
Copyright © Notice The materials are copyrighted © and trademarked ™ as the
property of the Texas Education Agency (TEA) and may not be reproduced without
the express written permission of TEA, except under the following conditions:
1. Texas public school districts, charter schools, and Education Service Centers
may reproduce and use copies of the Materials and Related Materials for the
districts’ and schools’ educational use without obtaining permission from TEA.
2. Residents of the state of Texas may reproduce and use copies of the Materials
and Related Materials for individual personal use only without obtaining written
permission of TEA.
3. Any portion reproduced must be reproduced in its entirety and remain
unedited, unaltered and unchanged in any way.
4. No monetary charge can be made for the reproduced materials or any
document containing them; however, a reasonable charge to cover only the
cost of reproduction and distribution may be charged.
Private entities or persons located in Texas that are not Texas public school districts,
Texas Education Service Centers, or Texas charter schools or any entity, whether
public or private, educational or non-educational, located outside the state of
Texas MUST obtain written approval from TEA and will be required to enter into a
license agreement that may involve the payment of a licensing fee or a royalty.
For information contact: Office of Copyrights, Trademarks, License Agreements, and
Royalties, Texas Education Agency, 1701 N. Congress Ave., Austin, TX 78701-1494;
phone 512-463-9270 or 512-463-9713; email: copyrights@tea.state.tx.us
Copyright © Texas Education Agency 2014. All rights reserved.