Response to Intervention Monitoring Student Progress at the Secondary Level Jim Wright www.interventioncentral.org www.interventioncentral.org Response to Intervention “Few agree on an appropriate curriculum for secondary students…; thus it is difficult to determine in what areas student [academic] progress should be measured.” -- Espin & Tindal (1998) Source: Espin, C. A., & Tindal, G. (1998). Curriculum-based measurement for secondary students. In M. R. Shinn (Ed.) Advanced applications of curriculum-based measurement. New York: Guilford Press. www.interventioncentral.org 2 Response to Intervention RTI Literacy: Assessment & Progress-Monitoring • • • To measure student ‘response to instruction/intervention’ effectively, the RTI model measures students’ academic performance and progress on schedules matched to each student’s risk profile and intervention Tier membership. Benchmarking/Universal Screening. All children in a grade level are assessed at least 3 times per year on a common collection of academic assessments. Strategic Monitoring. Students placed in Tier 2 (supplemental) reading groups are assessed 1-2 times per month to gauge their progress with this intervention. Intensive Monitoring. Students who participate in an intensive, individualized Tier 3 intervention are assessed at least once per week. Source: Burns, M. K., & Gibbons, K. A. (2008). Implementing response-to-intervention in elementary and secondary schools: Procedures to assure scientific-based practices. New York: Routledge. www.interventioncentral.org 3 Response to Intervention Measuring General vs. Specific Academic Outcomes • General Outcome Measures: Track the student’s increasing proficiency on general curriculum goals such as reading fluency. An example is CBM-Oral Reading Fluency (Hintz et al., 2006). • Specific Sub-Skill Mastery Measures: Track short-term student academic progress with clear criteria for mastery. An example is CBA-Math Computation Fluency (Burns & Gibbons, 2008). Sources: Burns, M. K., & Gibbons, K. A. (2008). Implementing response-to-intervention in elementary and secondary schools: Procedures to assure scientific-based practices. New York: Routledge. Hintz, J. M., Christ, T. J., & Methe, S. A. (2006). Curriculum-based assessment. Psychology in the Schools, 43, 45-56. www.interventioncentral.org 4 Response to Intervention Local Norms: Screening All Students (Stewart & Silberglit, 2008) Local norm data in basic academic skills are collected at least 3 times per year (fall, winter, spring). • Schools should consider using ‘curriculum-linked’ measures such as Curriculum-Based Measurement that will show generalized student growth in response to learning. • If possible, schools should consider avoiding ‘curriculumlocked’ measures that are tied to a single commercial instructional program. Source: Stewart, L. H. & Silberglit, B. (2008). Best practices in developing academic local norms. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 225-242). Bethesda, MD: National Association of School Psychologists. www.interventioncentral.org 5 Response to Intervention Local Norms: Using a Wide Variety of Data (Stewart & Silberglit, 2008) Local norms can be compiled using: • Fluency measures such as Curriculum-Based Measurement. • Existing data, such as office disciplinary referrals. • Computer-delivered assessments, e.g., Measures of Academic Progress (MAP) from www.nwea.org Source: Stewart, L. H. & Silberglit, B. (2008). Best practices in developing academic local norms. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 225-242). Bethesda, MD: National Association of School Psychologists. www.interventioncentral.org 6 Response to Intervention Measures of Academic Progress (MAP) www.nwea.org www.interventioncentral.org 7 Response to Intervention Applications of Local Norm Data (Stewart & Silberglit, 2008) Local norm data can be used to: • Evaluate and improve the current core instructional program. • Allocate resources to classrooms, grades, and buildings where student academic needs are greatest. • Guide the creation of targeted Tier 2 (supplemental intervention) groups • Set academic goals for improvement for students on Tier 2 and Tier 3 interventions. • Move students across levels of intervention, based on performance relative to that of peers (local norms). Source: Stewart, L. H. & Silberglit, B. (2008). Best practices in developing academic local norms. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 225-242). Bethesda, MD: National Association of School Psychologists. www.interventioncentral.org 8 Response to Intervention Apply the ‘80-15-5’ Rule to Determine if the Focus of the Intervention Should Be the Core Curriculum, Subgroups of Underperforming Learners, or Individual Struggling Students (T. Christ, 2008) – – – If less than 80% of students are successfully meeting academic or behavioral goals, the intervention focus is on the core curriculum and general student population. If no more than 15% of students are not successful in meeting academic or behavioral goals, the intervention focus is on smallgroup ‘treatments’ or interventions. If no more than 5% of students are not successful in meeting academic or behavioral goals, the intervention focus is on the individual student. Source: Christ, T. (2008). Best practices in problem analysis. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 159-176). www.interventioncentral.org 9 Response to Intervention Local Norms: Supplement With Additional Academic Testing as Needed (Stewart & Silberglit, 2008) “At the individual student level, local norm data are just the first step toward determining why a student may be experiencing academic difficulty. Because local norms are collected on brief indicators of core academic skills, other sources of information and additional testing using the local norm measures or other tests are needed to validate the problem and determine why the student is having difficulty. … Percentage correct and rate information provide clues regarding automaticity and accuracy of skills. Error types, error patterns, and qualitative data provide clues about how a student approached the task. Patterns of strengths and weaknesses on subtests of an assessment can provide information about the concepts in which a student or group of students may need greater instructional support, provided these subtests are equated and reliable for these purposes.” p. 237 Source: Stewart, L. H. & Silberglit, B. (2008). Best practices in developing academic local norms. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 225-242). Bethesda, MD: National Association of School Psychologists. www.interventioncentral.org 10 Response to Intervention How Does a Secondary School Determine a Student’s Math Competencies? “Tests [to assess secondary students’ math knowledge] should be used or if necessary developed that measure students’ procedural fluency as well as their conceptual understanding. Items should range in difficulty from simple applications of the algorithm to more complex. A variety of problem types can be used across assessments to tap students’ conceptual knowledge.” p. 469 Source: Ketterlin-Geller, L. R., Baker, S. K., & Chard, D. J. (2008). Best practices in mathematics instruction and assessment in secondary settings. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp.465-475). www.interventioncentral.org 11 Response to Intervention Making Use of Existing (‘Extant’) Data www.interventioncentral.org Response to Intervention Extant (Existing) Data (Chafouleas et al., 2007) • Definition: Information that is collected by schools as a matter of course. • Extant data comes in two forms: – Performance summaries (e.g., class grades, teacher summary comments on report cards, state test scores). – Student work products (e.g., research papers, math homework, PowerPoint presentation). Source: Chafouleas, S., Riley-Tillman, T.C., & Sugai, G. (2007). School-based behavioral assessment: Informing intervention and instruction. New York: Guilford Press. www.interventioncentral.org 13 Response to Intervention Advantages of Using Extant Data (Chafouleas et al., 2007) • Information is already existing and easy to access. • Students are less likely to show ‘reactive’ effects when data is collected, as the information collected is part of the normal routine of schools. • Extant data is ‘relevant’ to school data consumers (such as classroom teachers, administrators, and members of problem-solving teams). Source: Chafouleas, S., Riley-Tillman, T.C., & Sugai, G. (2007). School-based behavioral assessment: Informing intervention and instruction. New York: Guilford Press. www.interventioncentral.org 16 Response to Intervention Drawbacks of Using Extant Data (Chafouleas et al., 2007) • Time is required to collate and summarize the data (e.g., summarizing a week’s worth of disciplinary office referrals). • The data may be limited and not reveal the full dimension of the student’s presenting problem(s). • There is no guarantee that school staff are consistent and accurate in how they collect the data (e.g., grading policies can vary across classrooms; instructors may have differing expectations regarding what types of assignments are given a formal grade; standards may fluctuate across teachers for filling out disciplinary referrals). • Little research has been done on the ‘psychometric adequacy’ of extant data sources. Source: Chafouleas, S., Riley-Tillman, T.C., & Sugai, G. (2007). School-based behavioral assessment: Informing intervention and instruction. New York: Guilford Press. www.interventioncentral.org 17 Response to Intervention Universal Screening at Secondary Schools: Using Existing Data Proactively to Flag ‘Signs of Disengagement’ “Across interventions…, a key component to promoting school completion is the systematic monitoring of all students for signs of disengagement, such as attendance and behavior problems, failing courses, off track in terms of credits earned toward graduation, problematic or few close relationships with peers and/or teachers, and then following up with those who are at risk.” Source: Jimerson, S. R., Reschly, A. L., & Hess, R. S. (2008). Best practices in developing academic local norms. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 1085-1097). Bethesda, MD: National Association of School Psychologists. p.1090 www.interventioncentral.org 18 Response to Intervention Mining Archival Data: What Are the ‘Early Warning Flags’ of Student Drop-Out? • • • • A sample of 13,000 students in Philadelphia were tracked for 8 years. These early warning indicators were found to predict student drop-out in the sixth-grade year: Failure in English Failure in math Missing at least 20% of school days Receiving an ‘unsatisfactory’ behavior rating from at least one teacher Source: Balfanz, R., Herzog, L., MacIver, D. J. (2007). Preventing student disengagement and keeping students on the graduation path in urban middle grades schools: Early identification and effective interventions. Educational Psychologist,42, 223–235. . www.interventioncentral.org 19 Response to Intervention What is the Predictive Power of These Early Warning Flags? Number of ‘Early Warning Flags’ in Student Record Probability That Student Would Graduate None 56% 1 36% 2 21% 3 13% 4 7% Source: Balfanz, R., Herzog, L., MacIver, D. J. (2007). Preventing student disengagement and keeping students on the graduation path in urban middle grades schools: Early identification and effective interventions. Educational Psychologist,42, 223–235. . www.interventioncentral.org 20 Response to Intervention ‘Elbow Group’ Activity: What Extant/Archival Data Should Your RTI Team Review Regularly? • Discuss the essential extant/archival data that your RTI Team should review as ‘early warning indicators’ of students who are struggling. • What process should your school adopt to ensure that these data are reviewed regularly (e.g., every five weeks) to guarantee timely identification of students who need intervention assistance? www.interventioncentral.org 21 Response to Intervention Grades as a Classroom-Based ‘Pulse’ Measure of Academic Performance www.interventioncentral.org Response to Intervention Grades & Other Teacher Performance Summary Data (Chafouleas et al., 2007) • Teacher test and quiz grades can be useful as a supplemental method for monitoring the impact of student behavioral interventions. • Other data about student academic performance (e.g., homework completion, homework grades, etc.) can also be tracked and graphed to judge intervention effectiveness. Source: Chafouleas, S., Riley-Tillman, T.C., & Sugai, G. (2007). School-based behavioral assessment: Informing intervention and instruction. New York: Guilford Press. www.interventioncentral.org 23 Response to Intervention Marc Ripley 2-Wk 9/23/07 4-Wk 10/07/07 6-Wk 10/21/07 (From Chafouleas et al., 2007) 8-Wk 11/03/07 10-Wk 11/20/07 12-Wk 12/05/07 Source: Chafouleas, S., Riley-Tillman, T.C., & Sugai, G. (2007). School-based behavioral assessment: Informing intervention and instruction. New York: Guilford Press. www.interventioncentral.org 24 Response to Intervention Online Grading Systems www.interventioncentral.org 25 Response to Intervention Assessing Basic Academic Skills: CurriculumBased Measurement www.interventioncentral.org Response to Intervention Curriculum-Based Measurement: Advantages as a Set of Tools to Monitor RTI/Academic Cases • • • • • • • • Aligns with curriculum-goals and materials Is reliable and valid (has ‘technical adequacy’) Is criterion-referenced: sets specific performance levels for specific tasks Uses standard procedures to prepare materials, administer, and score Samples student performance to give objective, observable ‘low-inference’ information about student performance Has decision rules to help educators to interpret student data and make appropriate instructional decisions Is efficient to implement in schools (e.g., training can be done quickly; the measures are brief and feasible for classrooms, etc.) Provides data that can be converted into visual displays for ease of communication Source: Hosp, M.K., Hosp, J. L., & Howell, K. W. (2007). The ABCs of CBM. New York: Guilford. www.interventioncentral.org 27 Response to Intervention Assessing Basic Academic Skills: Curriculum-Based Measurement Reading: These 3 measures all proved ‘adequate predictors’ of student performance on reading content tasks: – Reading aloud (Oral Reading Fluency): Passages from content-area tests: 1 minute. – Maze task (every 7th item replaced with multiple choice/answer plus 2 distracters): Passages from content-area texts: 2 minutes. – Vocabulary matching: 10 vocabulary items and 12 definitions (including 2 distracters): 10 minutes. Source: Espin, C. A., & Tindal, G. (1998). Curriculum-based measurement for secondary students. In M. R. Shinn (Ed.) Advanced applications of curriculum-based measurement. New York: Guilford Press. www.interventioncentral.org 28 Response to Intervention Oral Reading Fluency Probe Sample http://www.rti2.org/rti2/oralReadings www.interventioncentral.org 29 Response to Intervention Maze Probe Sample http://www.rti2.org/rti2/mazes www.interventioncentral.org 30 Response to Intervention Curriculum-Based Evaluation: Math Vocabulary Format Option 1 • 20 vocabulary terms appear alphabetically in the right column. Items are drawn randomly from a ‘vocabulary pool’ • Randomly arranged definitions appear in the left column. • The student writes the letter of the correct term next to each matching definition. • The student receives 1 point for each correct response. • Each probe lasts 5 minutes. • 2-3 probes are given in a session. Source: Howell, K. W. (2008). Best practices in curriculum-based evaluation and advanced reading. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 397-418). www.interventioncentral.org 31 Response to Intervention Curriculum-Based Evaluation: Math Vocabulary Format Option 2 • 20 randomly arranged vocabulary definitions appear in the right column. Items are drawn randomly from a ‘vocabulary pool’ • The student writes the name of the correct term next to each matching definition. • The student is given 0.5 point for each correct term and another 0.5 point if the term is spelled correctly. • Each probe lasts 5 minutes. • 2-3 probes are given in a session. Source: Howell, K. W. (2008). Best practices in curriculum-based evaluation and advanced reading. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 397-418). www.interventioncentral.org 32 Response to Intervention Assessing Basic Academic Skills: Curriculum-Based Measurement Mathematics: Single-skill basic arithmetic combinations an ‘adequate measure of performance’ for low-achieving middle school students. Websites to create CBM math computation probes: •www.interventioncentral.org •www.superkids.com Source: Espin, C. A., & Tindal, G. (1998). Curriculum-based measurement for secondary students. In M. R. Shinn (Ed.) Advanced applications of curriculum-based measurement. New York: Guilford Press. www.interventioncentral.org 33 Response to Intervention Assessing Basic Academic Skills: Curriculum-Based Measurement Writing: CBM/ Word Sequence is a ‘valid indicator of general writing proficiency’. It evaluates units of writing and their relation to one another. Successive pairs of ‘writing units’ make up each word sequence. The mechanics and conventions of each word sequence must be correct for the student to receive credit for that sequence. CBM/ Word Sequence is the most comprehensive CBM writing measure. Source: Espin, C. A., & Tindal, G. (1998). Curriculum-based measurement for secondary students. In M. R. Shinn (Ed.) Advanced applications of curriculum-based measurement. New York: Guilford Press. www.interventioncentral.org 34 Response to Intervention Breaking Down Complex Academic Goals into Simpler Sub-Tasks: Discrete Categorization www.interventioncentral.org Response to Intervention Identifying and Measuring Complex Academic Problems at the Middle and High School Level • Students at the secondary level can present with a range of concerns that interfere with academic success. • One frequent challenge for these students is the need to reduce complex global academic goals into discrete sub-skills that can be individually measured and tracked over time. www.interventioncentral.org 36 Response to Intervention Discrete Categorization: A Strategy for Assessing Complex, Multi-Step Student Academic Tasks Definition of Discrete Categorization: ‘Listing a number of behaviors and checking off whether they were performed.’ (Kazdin, 1989, p. 59). • Approach allows educators to define a larger ‘behavioral’ goal for a student and to break that goal down into sub-tasks. (Each subtask should be defined in such a way that it can be scored as ‘successfully accomplished’ or ‘not accomplished’.) • The constituent behaviors that make up the larger behavioral goal need not be directly related to each other. For example, ‘completed homework’ may include as sub-tasks ‘wrote down homework assignment correctly’ and ‘created a work plan before starting homework’ Source: Kazdin, A. E. (1989). Behavior modification in applied settings (4th ed.). Pacific Gove, CA: Brooks/Cole.. www.interventioncentral.org 37 Response to Intervention Discrete Categorization Example: Math Study Skills General Academic Goal: Improve Tina’s Math Study Skills Tina was struggling in her mathematics course because of poor study skills. The RTI Team and math teacher analyzed Tina’s math study skills and decided that, to study effectively, she needed to: Check her math notes daily for completeness. Review her math notes daily. Start her math homework in a structured school setting. Use a highlighter and ‘margin notes’ to mark questions or areas of confusion in her notes or on the daily assignment. Spend sufficient ‘seat time’ at home each day completing homework. Regularly ask math questions of her teacher. www.interventioncentral.org 38 Response to Intervention Discrete Categorization Example: Math Study Skills General Academic Goal: Improve Tina’s Math Study Skills The RTI Team—with teacher and student input—created the following intervention plan. The student Tina will: Approach the teacher at the end of class for a copy of class note. Check her daily math notes for completeness against a set of teacher notes in 5th period study hall. Review her math notes in 5th period study hall. Start her math homework in 5th period study hall. Use a highlighter and ‘margin notes’ to mark questions or areas of confusion in her notes or on the daily assignment. Enter into her ‘homework log’ the amount of time spent that evening doing homework and noted any questions or areas of confusion. Stop by the math teacher’s classroom during help periods (T & Th only) to ask highlighted questions (or to verify that Tina understood that week’s instructional content) and to review the homework log. www.interventioncentral.org 39 Response to Intervention Discrete Categorization Example: Math Study Skills Academic Goal: Improve Tina’s Math Study Skills General measures of the success of this intervention include (1) rate of homework completion and (2) quiz & test grades. To measure treatment fidelity (Tina’s follow-through with sub-tasks of the checklist), the following strategies are used : Approached the teacher for copy of class notes. Teacher observation. Checked her daily math notes for completeness; reviewed math notes, started math homework in 5th period study hall. Student work products; random spot check by study hall supervisor. Used a highlighter and ‘margin notes’ to mark questions or areas of confusion in her notes or on the daily assignment. Review of notes by teacher during T/Th drop-in period. Entered into her ‘homework log’ the amount of time spent that evening doing homework and noted any questions or areas of confusion. Log reviewed by teacher during T/Th drop-in period. Stopped by the math teacher’s classroom during help periods (T & Th only) to ask highlighted questions (or to verify that Tina understood that week’s instructional content). Teacher observation; student sign-in. www.interventioncentral.org 40 Response to Intervention RTI Teams: Recommendations for Data Collection www.interventioncentral.org Response to Intervention RTI Teams: Recommendations for Data Collection • Collect a standard set of background information on each student referred to the RTI Team. RTI Teams should develop a standard package of background (archival) information to be collected prior to the initial problem-solving meeting. For each referred student, a Team might elect to gather attendance data, office disciplinary referrals for the current year, and the most recent state assessment results. www.interventioncentral.org 42 Response to Intervention RTI Teams: Recommendations for Data Collection • For each area of concern, select at least two progress-monitoring measures. RTI Teams can place greater confidence in their progress-monitoring data when they select at least two measures to track any area of student concern (Gresham, 1983)-ideally from at least two different sources (e.g., Campbell & Fiske, 1959). With a minimum of two methods in place to monitor a student concern, each measure serves as a check on the other. If the results are in agreement, the Team has greater assurance that it can trust the data. If the measures do not agree with one another, however, the Team can investigate further to determine the reason(s) for the apparent discrepancy. www.interventioncentral.org 43 Response to Intervention RTI Teams: Recommendations for Data Collection • Monitor student progress frequently. Progress-monitoring data should reveal in weeks--not months--whether an intervention is working because no teacher wants to waste time implementing an intervention that is not successful. When progress monitoring is done frequently (e.g., weekly), the data can be charted to reveal more quickly whether the student’s current intervention plan is effective. Curriculum-based measurement, Daily Behavior Report Cards, and classroom observations of student behavior are several assessment methods that can be carried out frequently. www.interventioncentral.org 44 Response to Intervention Monitoring Student Academic Behaviors: Daily Behavior Report Cards www.interventioncentral.org Response to Intervention Daily Behavior Report Cards (DBRCs) Are… brief forms containing student behavior-rating items. The teacher typically rates the student daily (or even more frequently) on the DBRC. The results can be graphed to document student response to an intervention. www.interventioncentral.org 46 Response to Intervention Daily Behavior Report Cards Can Monitor… • • • • • • Hyperactivity On-Task Behavior (Attention) Work Completion Organization Skills Compliance With Adult Requests Ability to Interact Appropriately With Peers www.interventioncentral.org 47 Response to Intervention Daily Behavior Report Card: Daily Version Jim Blalock Mrs. Williams www.interventioncentral.org May 5 Rm 108 Response to Intervention Daily Behavior Report Card: Weekly Version Jim Blalock Mrs. Williams Rm 108 05 05 07 05 06 07 05 07 07 05 08 07 05 09 07 40 www.interventioncentral.org 0 60 60 50 Response to Intervention Daily Behavior Report Card: Chart www.interventioncentral.org Response to Intervention Student Case Scenario: Jim • • • • Jim is a 10th-grade student who is failing his math course and in danger of failing English and science courses. Jim has been identified with ADHD. His instructional team meets with the RTI Team and list the following academic and behavioral concerns for Jim. Does not bring work materials to class Fails to write down homework assignments Sometimes does not turn in homework, even when completed Can be non-compliant with teacher requests at times. www.interventioncentral.org 51 Response to Intervention www.interventioncentral.org www.interventioncentral.org 52 Response to Intervention Closing Thoughts About Local Norms & RTI Assessment at the Secondary Level… www.interventioncentral.org Response to Intervention Local Norms: Set a Realistic Timeline for ‘Phase-In’ (Stewart & Silberglit, 2008) “If local norms are not already being collected, it may be helpful to develop a 3-5 year planned rollout of local norm data collection, reporting, and use in line with other professional development and assessment goals for the school. This phased-in process of developing local norms could start with certain grade levels and expand to others.” p. 229 Source: Stewart, L. H. & Silberglit, B. (2008). Best practices in developing academic local norms. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (pp. 225-242). Bethesda, MD: National Association of School Psychologists. www.interventioncentral.org 54 Response to Intervention Activity: How ‘RTI Ready’ is Your Middle or High School in Measuring Academic Skills? In your ‘elbow groups’: • Discuss the range of assessments that your school can access to measure the current performance and progress of students in basic academic skill areas. • How ‘RTI ready’ is your school in measuring these academic skills? • Be prepared to share your discussion points with the larger group. www.interventioncentral.org 55 Response to Intervention End www.interventioncentral.org 56 Response to Intervention RTI Team Initial Meeting Form: Secondary Student ProgressMonitoring Page 15 www.interventioncentral.org 57 Response to Intervention RTI Team Initial Meeting Form: Secondary Student ProgressMonitoring Page 16 www.interventioncentral.org 58 Response to Intervention RTI Team Initial Meeting Form: Secondary Student ProgressMonitoring Page 17 www.interventioncentral.org 59 Response to Intervention www.interventioncentral.org 60 Response to Intervention www.interventioncentral.org 61