Multiple Sources of Data RtI Documentation: Understanding, Interpreting and Connecting Data for Educational Decision Making Andrea Ogonosky, Ph.D., LSSP firstname.lastname@example.org 1 A complementary relationship between the RTI process and psychoeducational testing exists in evaluating for a potential learning disability. RtI: Problem Solving Assessment Progress Monitoring Diagnostics Progress Monitoring Diagnostics Universal Screening Progress Monitoring Interventions Student Instructional Level Supplemental Interventions 120 min per week additional Student Instructional Level Supplemental Interventions 90 min per week additional Grade Level Instruction/ Support Fidelity Intervention Well Checks Observe in Tiers I and II/III (ICEL) Consult with Teacher Review data weekly in PLC/ Planning meetings Check data collection Talk to parent From the Student Perspective The Team Goal is to create… Academic Learning, Mastery, and Achievement Social, Emotional, and Behavioral Development Independent Learner Self-Manager Activity How are you defining Tier 1 instruction and strategies? Is there consistency with this definition throughout your district? What information are you using to determine appropriate access to instruction? Successful Data Collection • • • • • Use of naturally occurring data Led by General Education, Supported by Special Education Problem-Solving Model Implemented with Integrity Systematic decision rules consistently implemented Access to Technology to Manage and Document DataBased Decision Making • Evidence of Improved Academic and Behavior Outcomes for All Students It is vitally important that there is an understanding that there is continued documented discussion and consultation between the teacher, the team, and the interventionist(s). Components Addressed When Using Multiple Data Sources • The interrelationship between classroom achievement and cognitive processing criteria – Classroom achievement – Academic Deficit (RtI) – Cognitive Processing – Behavior 10 Balancing Assessments -- Assessment systems -- Multiple measures -- Varied types -- Varied purposes -- Varied data sets -- Balanced with needs Multiple Data Sources Criterion Referenced Assessment Formative Summative Screen Progress Monitor Norm Referenced Assessment Diagnostic Comparative Progress Monitor Curriculum Based Measurement Rate of Improvement Universal Screen Progress Monitor Criterion Referenced Tests • Most common type of test used by teachers. • Criterion Referenced Tests measure mastery of a subject based on specific preset standards. • The questions used in the test are meant to show how much a student knows and how that student’s performance compares to expectations. Problems with Mastery Measurement • Hierarchy of skills is logical, not empirical. • Performance can be misleading: assessment does not reflect maintenance or generalization. • Assessment is designed by teachers or sold with textbooks, with unknown reliability and validity. • Number of objectives mastered does not relate well to performance on high-stakes tests. 13 Norm Referenced Tests • Norm-referenced tests compare a student's score against the scores of a group of student’s who have already taken the same exam, called the "norm group." • Score are often interpreted using percentiles. • Comparative in nature, aligned to researched based developmental and cognitive levels. Curriculum Based Measurement • • • Describes academic competence at a single point in time Quantifies the rate at which students develop academic competence over time Used to align and analyze effective instruction and intervention to increase student achievement 15 Types of Curriculum Based Measurement • Universal Screening data on all students provides an indication of individual student performance and progress compared to the peer group’s performance and progress. • Universal Screening data form the basis for examining trends (or patterns)on specific academic or behavior skills. Types of Curriculum Based Measurement Progress monitoring documents student growth over time to determine whether the student is progressing as expected in the designated level of instructional intervention. Generally this is often presented in graph form. District decision Rules Selection of goal-level material Collection baseline data and setting of realistic or ambitious goals Administration of timed, alternate measures weekly Application of decision-making rules to graphed data every 69 weeks Activity • List Formative Assessments your district/campus are currently using. • List Progress Monitors your district/campus are currently using. What norms? • List Summative Assessments your district/campus are currently using. • How do you use this data to aid in determining patterns of strengths and weaknesses? Dr. O’s Suggestion Begin with the End in Mind: STAAR, EOC RtI Data Analysis Achievement Data (Formative, Diagnostic) Academic Deficit Verification Hypothesis Generation Cognitive Processing Data Collection Component Data Review: Consider the impact of each domain relative to the “problem” 21 INSTRUCTION Instructional decision making regarding selection and use of materials Clarity of instructions Communication with expectations and cues Sequencing of lesson designs Pace of instruction Variety of practice activities CURRICULUM Long range direction for instruction Instructional philosophy Instructional materials Intent Stated outcomes of the content/instruction Pace of the steps leading to the outcomes General learner criteria as identified in the school improvement plan, LEA curriculum, and benchmarks ENVIRONMENT Physical arrangement of the room Furniture/equipment Rules Management plans Routines Expectations Peer context Task pressure Peer and family expectations LEARNER This is the last domain to consider and is addressed when: The curriculum and instruction are appropriate The environment is positive This domain includes student performance data: Academic Social/emotional 22 Existing Data Review – Determine the student’s current grade level/class status: academic progress (grades, unit tests, district benchmark, and work samples) – Teacher description/quantifiable concern – Parent Contact(s) – RtI data – Medical Information – Classroom Observations (ICEL) Analyze Multiple Data Sources Begin With Universal Screening Data Student score relative to cut-score in identified area of concern. If below cut-score possible need for increased intensity of supports (Tier 2) If above cut-score instruction is supported through Tier 1 supports Analyze Multiple Data Sources Review Teacher Progress Monitoring Data Formative and Summative Assessments Are they differentiated to identified area of concern? Evidence of alignment to student variables (learning styles) How does student compare to class peers in the identified area of concern Student Data Review student strengths in identified area of concern Review Student skill deficit/weaknesses in identified area of concern Determine Student Functional Levels • Identified assets and weaknesses • Identified critical life events, milestone • Identified academic variables such as “speed of acquisition” or retention of information • Identified issues of attendance, transitions, motivation, access to instruction Classroom Achievement Note: analysis of data assumes that the student has received adequate opportunities to learn and reasonable instructional options • Gather relevant and specific information about the student's performance in the general education curriculum (relative to peers) • Determine if a pattern of severe delay in classroom achievement in one or more areas has existed over time • Provide information about the curriculum and corresponding instructional demands Classroom Achievement Guiding question: What has been taught to the student and how? 1. Does the student meet the educational standards that apply to all students? 2. What are the academic task requirements for students in the referred student's classroom? 3. Has the student received instruction on the same content as the majority of students in the same grade? 4. What are the student's skills in relation to what has been taught? Classroom Achievement 5. What is the range of student skill levels? 6. Are there any other student’s performing at the same level as the referred student? Classroom Achievement / Academic Deficit : Sources of Data – – – – – – Observation: at least 1 within each Tier of support work samples: should be more than worksheets error analysis: is it aligned with focused instruction? criterion-referenced measures task analysis curriculum-based measures such as inventories. quizzes, rubrics, running records, informal reading inventories – record review: be careful not to dismiss the importance of historical data Analyze Multiple Data Sources Analyze Diagnostic Data Student strengths and weaknesses relative to identified area of concern What specific skills are lacking that present a barrier for student progress? Need to know developmental acquisition of skills relative to the identified area of concern. Is the identified skill deficit significantly below grade expectations? What effective practices are needed to address this skill deficit? Analyze Multiple Data Sources Review Progress Monitoring Data What curriculum based measure is your district using? What norms are being used to set goals. What ROI is expected? Progress Monitoring Ongoing and frequent monitoring of progress quantifies rates of improvement and informs instructional practice and the development of individualized programs. Integrity of Progress Monitoring Selected progress monitoring tools meet all of the following criteria: (1) Has alternate forms of equal and controlled difficulty; (2) specifies minimum acceptable growth; (3) provides benchmarks for minimum acceptable endof-year performance; (4) reliability and validity information for the performance level score are available. What does your district use? Rate of Improvement • Typical Rate of Improvement - the rate of improvement or the rate of change of a typical student at the assessment. • Targeted Rate of Improvement - the rate of improvement that a targeted student would need to make by the end of the instruction/intervention. • Attained Rate of Improvement - the actual rate of improvement the student ends up achieving as a function of the intervention. Integrity of Progress Monitoring Both conditions are met: (1) Frequency is at least weekly for Tier 1 and 2 students ; (2) Procedures are in place to ensure implementation accuracy (i.e., appropriate students are tested; scores are accurate; decision making rules are applied consistently). Additional Assessments (Tiers 2&3) Analogy: If there is a medical concern, you…. …. go to the doctor If the doctor is not sure what is wrong….. ….. more tests are given Question • What are the tests your campuses are giving in Tiers 2 &3? Should be diagnostic in nature. • What about progress monitoring? How are they measuring growth? Analyze Multiple Data Sources Review Outcome Data Student score relative to peers on district common or benchmark assessments Student score relative to peers on end of unit or other identified summative assessments Student score relative to peers on state assessments Align Data Sources Universal Screening Progress Monitoring Diagnostic Assessments Outcome Assessments Does the data tell a clear and concise story of the student’s learning needs? If there is inconsistency team must investigate why. Review integrity of instruction Alignment to student needs (Tier 1) Student variables Scientific inquiry • Triangulation: verification of a hypothesis or identification of a pattern by comparing and contrasting data • The ARD committee must be confident that the data collectively reveals a pattern consistent with the definition of SLD. • Data collected aligns with characteristics of other disability conditions. Scientific Inquiry • Explore potential alternate explanations when data is inconsistent • Ask (and answer additional) questions if the answer does not lie in the data • Assessment data not only provides answer to questions regarding eligibility, but it serves to design instructional and behavioral intervention strategies that are likely to improve student learning Improving SLD Identification Process • In order to improve the diagnostic process, you need to approach diagnosis and the FIE as a compilation of formal and informal data. • Problem Identification & Analysis–process of assessment and evaluation for identifying and understanding the causal and maintaining variables of the problem Cognitive Strengths/Assets and Weaknesses/Deficits • Measure abilities associated with the area of academic skill deficit and abilities not associated with the academic deficit – Broad abilities, narrow abilities, and core cognitive processes – Usually select a global measure for this, but can do this with focused assessment of processes if the referral is specific enough • How efficient is the student’s learning process – if deficiencies present they will interfere with/affect learning Cognitive Patterns • Some issues to consider: – The cognitive deficiency must be normatively based not person based; a weakness derived from ipsative analysis (intra-individual) is not relevant regardless of statistical significance – Low achievement in the absence of any cognitive deficiencies is not consistent with any processing model of LD Relationship of cognitive deficits to academic deficits • Determine the presence of a processingachievement consistency/concordance – There is a need to have measured the narrow abilities related to academic areas explained to describe pattern. • To do this step, must be familiar with the cognitive processes that are associated with achievement areas. Diagnostic Conclusions • The determination of a disability category, (especially MR and LD) must be made through the use of professional judgment, including consideration of multiple information/data sources to support the eligibility determination.(TEA, §89.1040 Eligibility Criteria, Frequently Asked Questions) • Information/data sources = statewide assessment results, formal evaluation test scores, RtI progress monitoring data, informal data (e.g., work samples, interviews, rating scales), anecdotal reports Why is a Tiered Model Important to SLD Identification? “Need to Knows” • Individual Differences in the Processes in the Learner’s Mind or Brain • Curriculum and Instructional Materials • Teachers’ Delivery of Instruction / Classroom management Processing – Disability & Intervention • Processing strengths and weaknesses associated with diagnosing LD, but also important and associated with other disabilities • If learning depends on processing, then having information about a student’s strengths and weaknesses can assist in design of interventions, selection of accommodations, and determination of modifications Main Processing Components to Assess (Dehn, 2005) Attention processing Auditory processing Executive processing Fluid reasoning Long-term retrieval Phonemic awareness Planning Processing speed Short-term memory Simultaneous processing Successive processing Visual Processing Working Memory Integrity of Diagnostic Data Cognitive Processing Model: (Fletcher, 2003): a multi-test discrepancy approach…and carries with it the problems involved with estimation of discrepancies and cut points. What is your district’s philosophy for defining processing strengths and weaknesses? Is your staff consistent with interpretation? Lack of clarity around what is to be accomplished is the biggest barrier to assessment for eligibility that you will encounter. Flawed logic….school staff understand the role assessment and diagnostic data play in decision making for struggling learners. Underscoring the Problem “Most teachers just don’t possess the skills to collect data, draw meaningful conclusions, focus instruction, and appropriately follow up to assess results. That is not the set of skills we were hired to do.” Convergence of Data: What to Look For • Increasing intensity of data collection aligned with student need/response to interventions • Diagnostic data that describes student strengths, weaknesses and response to intervention • Documentation of supplemental instruction aligned with diagnostic data Putting It All Together • Use achievement and RtI data to form hypothesis • Based upon hypothesis design normative achievement and cognitive processing assessment battery When Assessing Consider: • Rate of learning • Level of academic performance in one of the 8 areas for SLD • Memory • Processing speed • Attention/concentration • Visual motor integration • Behavior • • • • Executive functioning Speech/language Primary language Phonological processing • Academic fluency • Others? All the data have been collected…What constitutes a convergence of evidence? Essential Question How can assessment team and parents feel confident in making an appropriate SLD eligibility decision? Validating Underachievement Students who demonstrate reasonable progress in response to research-based strategies and interventions should not be determined eligible under the SLD category even though they may have academic weaknesses. What is reasonable progress? Identifying a SLD is a complex task and requires personalized, student-centered problem-solving. It is predicated on a belief that scaffolded, differentiated, high-quality general education instruction can be effective for most children and that only a few students demonstrate the severe and persistent underachievement associated with a SLD. Good decisions are based on A strong RTI framework Sufficient data collection and documentation Clearly articulated decision-making processes References • Essentials of Psychological Assessment Series. Hoboken, New Jersey: John Wiley & Sons. – Dehn, M.J. (2006). Essentials of Processing Assessment. – Flanagan, D.P. & Alfonso, V.C. (2011) Essentials of Specific Learning Disability Identification • SLD Identification Using Multiple Sources of Data. Cheramie, G. & Ogonosky, A. (2012). Presentation to Weatherford ISD.