Rate of Improvement Calculation and Decision Making Caitlin S. Flinn, EdS, NCSP Andrew E. McCrea, MS, NCSP Why we’re here… While there exists a wealth of convincing research supporting the implementation of a response-to-intervention (RtI) framework, there are many questions yet to be empirically answered. Within multi-tiered model of assessment and instruction/intervention, how do we know whether a student is learning? Measuring Learning Class tests Quizzes Assignment/homework completion and accuracy Ask students questions in class Grades/report cards State/local assessments Universal screening, benchmark assessments Progress monitoring With Progress Monitoring Data… How do we know if a student is learning? Look at the data points Where are they on the graph? Are the data points getting closer to the goal or benchmark? Is there a way to measure growth? Make an aimline toward goal Look to see where data points are compared to aimline Calculate Rate of Improvement (RoI) Today’s Objectives Explain what RoI is, why it is important, and how to compute it. Establish that Simple Linear Regression should be the standardized procedure for calculating RoI. Discuss how to use RoI within a problem solving/school improvement model. RoI Definition Rate of Improvement can be described algebraically as the slope of a line Slope is defined as: the vertical change over the horizontal change on a Cartesian plane. (x-axis and y-axis graph) Also called: Rise over run Formula: m = (y2 - y1) / (x2 - x1) Describes the steepness of a line (Gall & Gall, 2007) RoI Definition Finding a student’s RoI is determining the student’s learning Creating a line that fits the data points, a trendline To find that line, we use: Linear regression Ordinary Least Squares How does Rate of Improvement Fit into the Larger Context? School Improvement/Comprehensive School Reform Response to Intervention Dual Discrepancy: Level & Growth Rate of Improvement School Improvement/ Comprehensive School Reform Grade level content expectations (ELA, math, science, social studies, etc.). Work toward these expectations through classroom instruction. Understand impact of instruction through assessment. Assessment Formative Assessments/High Stakes Tests Does student have command of content expectation (standard)? Universal Screening using CBM Does student have basic skills appropriate for age/grade? Assessment Q: For students who are not proficient on grade level content standards, do they have the basic reading/writing/math skills necessary? A: Look at Universal Screening; if above criteria, intervention geared toward content standard, if below criteria, intervention geared toward basic skill. Progress Monitoring Frequent measurement of knowledge to inform our understanding of the impact of instruction/intervention. Measures of basic skills (CBM) have demonstrated reliability & validity (see table at www.rti4success.org). Classroom Instruction (Content Expectations) Measure Impact (Test) Proficient! Use Diagnostic Test to Differentiate Non Proficient Content Need? Basic Skill Need? Intervention Progress Monitor Intervention Progress Monitor With CBM If CBM is Appropriate Measure Rate of Improvement So… Rate of Improvement (RoI) is how we understand student growth (learning). RoI is reliable and valid (psychometrically speaking) for use with CBM data. RoI is best used when we have CBM data, most often when dealing with basic skills in reading/writing/math. RoI can be applied to other data (like behavior) with confidence too! RoI is not yet tested on typical Tier I formative classroom data. RoI is usually applied to… Tier One students in the early grades at risk for academic failure (low green kids). Tier Two & Three Intervention Groups. Special Education Students (and IEP goals) Students with Behavior Plans RoI Foundations Deno, 1985 Curriculum-based General measurement outcome measures Technically adequate Short Standardized Repeatable Sensitive to change RoI Foundations Fuchs & Fuchs, 1998 Hallmark components of Response to Intervention Ongoing formative assessment Identifying non-responding students Treatment fidelity of instruction Dual discrepancy model One standard deviation from typically performing peers in level and rate RoI Foundations Ardoin Slope & Christ, 2008 for benchmarks (3x per year) More growth from fall to winter than winter to spring Might be helpful to use RoI for fall to winter And a separate RoI for winter to spring RoI Foundations Fuchs, Fuchs, Walz, & Germann, 1993 Typical weekly growth rates in oral reading fluency and digits correct Needed growth to remediate skills Students who had 1.5 to 2.0 times the slope of typically performing peers were able to close the achievement gap in a reasonable amount of time RoI Foundations Deno, Fuchs, Marston, & Shin, 2001 Slope of frequently non-responsive children approximated slope of children already identified as having a specific learning disability How many data points? 10 data points are a minimum requirement for a reliable trendline (Gall & Gall, 2007) Is that reasonable and realistic? How does that affect the frequency of administering progress monitoring probes? How does that affect our ability to make instructional decisions for students? How can we show RoI? Speeches that included visuals, especially in color, improved recall of information (Vogel, Dickson, & Lehman, 1990) “Seeing is believing.” Useful for communicating large amounts of information quickly “A picture is worth a thousand words.” Transcends language barriers (Karwowski, 2006) Responsibility for accurate graphical representations of data Skills for Which We Compute RoI Reading Oral Reading Fluency Word Use Fluency Reading Comprehension Early Literacy Skills MAZE Retell Initial Sound Letter Naming Letter Sound Phoneme Segmentation Nonsense Word Spelling Written Expression Behavior Math Math Computation Math Facts Early Numeracy Oral Counting Missing Number Number Identification Quantity Discrimination Guidelines? Visual inspection of slope Multiple interpretations Instructional Need services for explicit guidelines Ongoing Research RoI for instructional decisions is not a perfect process Research is currently addressing sources of error: Christ, 2006: standard error of measurement for slope Ardoin & Christ, 2009: passage difficulty and variability Jenkin, Graff, & Miglioretti, 2009: frequency of progress monitoring Future Considerations Questions yet to be empirically answered What parameters of RoI indicate a lack of RtI? How does standard error of measurement play into using RoI for instructional decision making? How does RoI vary between standard protocol interventions? How does this apply to non-English speaking populations? How is RoI Calculated? Which way is best? Multiple Methods for Calculating Growth Visual Inspection Approaches “Eye Ball” Approach Split Middle Approach Tukey Method Quantitative Approaches Last point minus First point Approach Split Middle & Tukey “plus” Linear Regression Approach The Visual Inspection Approaches Eye Ball Approach 20 19 18 17 16 14 14 14 12 11 10 10 8 8 7 6 4 2 0 1 2 3 4 5 6 7 8 Split Middle Approach Drawing “through the two points obtained from the median data values and the median days when the data are divided into two sections” (Shinn, Good, & Stein, 1989). Split Middle 20 19 18 17 16 14 X(14) 14 14 12 10 10 8 11 X (9) X(9) 8 7 6 4 2 0 1 2 3 4 5 6 7 8 Tukey Method Divide scores into 3 equal groups Divide groups with vertical lines In 1st and 3rd groups, find median data point and median week and mark with an “X” Draw line between two “Xs” (Fuchs, et. al., 2005. Summer Institute Student progress monitoring for math. http://www.studentprogress.org/library/training.asp) Tukey Method 20 19 18 17 16 14 X(14) 14 14 12 11 10 10 8 8 X(8) 7 6 4 2 0 1 2 3 4 5 6 7 8 The Quantitative Approaches Last minus First Iris Center: last probe score minus first probe score over last administration period minus first administration period. Y2-Y1/X2-X1= RoI http://iris.peabody.vanderbilt.edu/resources.html Last minus First 20 19 18 17 16 14 14 14 12 11 10 10 8 8 7 (14-8)/(8-0)=0.75 6 4 2 0 1 2 3 4 5 6 7 8 Split Middle “Plus” 20 19 18 17 16 14 X(14) 14 14 12 11 10 10 8 X(9) 8 7 6 (14-9)/8=0.63 4 2 0 1 2 3 4 5 6 7 8 Tukey Method “Plus” 20 19 18 17 16 14 X(14) 14 14 12 11 10 10 8 8 X(8) 7 6 (14-8)/8=0.75 4 2 0 1 2 3 4 5 6 7 8 Linear Regression 20 19 18 17 16 14 14 14 12 11 10 10 8 8 7 6 y = 1.1429x + 7.3571 4 2 0 1 2 3 4 5 6 7 8 RoI Consistency? Any Method of Visual Inspection ??? Last minus First 0.75 Split Middle “Plus” Tukey “Plus” 0.63 Linear Regression 1.10 0.75 RoI Consistency? If we are not all using the same model to compute RoI, we continue to have the same problems as past models, where under one approach a student meets SLD criteria, but under a different approach, the student does not. Hypothetically, if the RoI cut-off was 0.65 or 0.95, different approaches would come to different conclusions on the same student. RoI Consistency? Last minus First (Iris Center) and Linear Regression (Shinn, etc.) only quantitative methods discussed in CBM literature. Study of 37 at risk 2nd graders: Difference in RoI b/w LmF & LR Methods Whole Year 0.26 WCPM Fall Spring 0.31 WCPM 0.24 WCPM McCrea (2010) Unpublished data Technical Adequacy Without a consensus on how to compute RoI, we risk falling short of having technical adequacy within our model. So, Which RoI Method is Best? Literature shows that Linear Regression is Best Practice Student’s daily test scores…were entered into a computer program…The data analysis program generated slopes of improvement for each level using an Ordinary-Least Squares procedure (Hayes, 1973) and the line of best fit. This procedure has been demonstrated to represent CBM achievement data validly within individual treatment phases (Marston, 1988; Shinn, Good, & Stein, in press; Stein, 1987). Shinn, Gleason, & Tindal, 1989 Growth (RoI) Research using Linear Regression Christ, T. J. (2006). Short-term estimates of growth using curriculum based measurement of oral reading fluency: Estimating standard error of the slope to construct confidence intervals. School Psychology Review, 35, 128-133. Deno, S. L., Fuchs, L. S., Marston, D., & Shin, J. (2001). Using curriculum based measurement to establish growth standards for students with learning disabilities. School Psychology Review, 30, 507-524. Good, R. H. (1990). Forecasting accuracy of slope estimates for reading curriculum based measurement: Empirical evidence. Behavioral Assessment, 12, 179-193. Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L. & Germann, G. (1993). Formative evaluation of academic progress: How much growth can we expect? School Psychology Review, 22, 27-48. Growth (RoI) Research using Linear Regression Jenkins, J. R., Graff, J. J., & Miglioretti, D.L. (2009). Estimating reading growth using intermittent CBM progress monitoring. Exceptional Children, 75, 151-163. Shinn, M. R., Gleason, M. M., & Tindal, G. (1989). Varying the difficulty of testing materials: Implications for curriculum-based measurement. The Journal of Special Education, 23, 223-233. Shinn, M. R., Good, R. H., & Stein, S. (1989). Summarizing trend in student achievement: A comparison of methods. School Psychology Review, 18, 356-370. So, Why Are There So Many Other RoI Models? Ease of application Focus on Yes/No to goal acquisition, not degree of growth How many of us want to calculate OLS Linear Regression formulas (or even remember how)? Pros and Cons of Each Approach Pros Eye Ball Split Middle & Tukey Easy Understandable No software needed Compare to Aim/Goal line Yes/No to goal acquisition Cons Subjective No statistic provided, no idea of the degree of growth Pros and Cons of Each Approach Pros Last minus First Provides a growth statistic Easy to compute Cons Does not consider all data points, only two Split Middle & Considers all data Tukey “Plus” points. Easy to compute No support for “plus” part of methodology Linear Regression Calculating the statistic All data points Best Practice An Easy and Applicable Solution Get Out Your Laptops! Open Microsoft Excel I love ROI Graphing RoI For Individual Students Programming Microsoft Excel to Graph Rate of Improvement: Fall to Winter Setting Up Your Spreadsheet In cell A1, type 3rd Grade ORF In cell A2, type First Semester In cell A3, type School Week In cell A4, type Benchmark In cell A5, type the Student’s Name (Swiper Example) Labeling School Weeks Starting with cell B3, type numbers 1 through 18 going across row 3 (horizontal). Numbers 1 through 18 represent the number of the school week. You will end with week 18 in cell S3. Labeling Dates Note: You may choose to enter the date of that school week across row 2 to easily identify the school week. Entering Benchmarks (3rd Grade ORF) In cell B4, type 77. This is your fall benchmark. In cell S4, type 92. This is your winter benchmark. Entering Student Data (Sample) Enter the following numbers, going across row 5, under corresponding week numbers. Week 1 – 41 Week 8 – 62 Week 9 – 63 Week 10 – 75 Week 11 – 64 Week 12 – 80 Week 13 – 83 Week 14 – 83 Week 15 – 56 Week 17 – 104 Week 18 – 74 *CAUTION* If a student was not assessed during a certain week, leave that cell blank Do not enter a score of Zero (0) it will be calculated into the trendline and interpreted as the student having read zero words correct per minute during that week. Graphing the Data Highlight cells A4 and A5 through S4 and S5 Follow Excel 2003 or Excel 2007 directions from here Graphing the Data Excel 2003 Across the top of your worksheet, click on “Insert” In that drop-down menu, click on “Chart” Excel 2007 Click Insert Find the icon for Line Click the arrow below Line Graphing the Data Excel 2003 A Chart Wizard window will appear Excel 2007 6 graphics appear Graphing the Data Excel 2003 Choose “Line” Choose “Line with markers…” Excel 2007 Choose “Line with markers” Graphing the Data Excel 2003 “Data Range” tab “Columns” Excel 2007 Your graph appears Graphing the Data Excel 2003 “Chart Title” “School Week” X Axis “WPM’ Y Axis Excel 2007 Change your labels by right clicking on the graph Graphing the Data Excel 2003 Choose where you want your graph Excel 2007 Your graph was automatically put into your data spreadsheet Graphing the Trendline Excel 2003 Excel 2007 Right click on any of the student data points Graphing the Trendline Excel 2003 Choose “Linear” Excel 2007 Graphing the Trendline Excel 2003 Excel 2007 Choose “Custom” and check box next to “Display equation on chart” Graphing the Trendline Clicking on the equation highlights a box around it Clicking on the box allows you to move it to a place where you can see it better Graphing the Trendline You can repeat the same procedure to have a trendline for the benchmark data points Suggestion: label the trendline Expected ROI Move this equation under the first Individual Student Graph: Fall to Winter 120 y = 2.5138x + 42.113 y = 0.8824x + 76.118 100 80 Benchmark Student: Sw iper 60 RoI Linear (Benchmark) 40 20 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Individual Student Graph The equation indicates the slope, or rate of improvement. The number, or coefficient, before "x" is the average improvement, which in this case is the average number of words per minute per week gained by the student. Individual Student Graph The rate of improvement, or trendline, is calculated using a linear regression, a simple equation of least squares. To add additional progress monitoring/benchmark scores once you’ve already created a graph, enter additional scores in Row 5 in the corresponding school week. Individual Student Graph The slope can change depending on which week (where) you put the benchmark scores on your chart. Enter benchmark scores based on when your school administers their benchmark assessments for the most accurate depiction of expected student progress. Programming Excel First Semester Calculating Needed RoI Calculating Benchmark RoI Calculating Student’s Actual RoI Quick Definitions Needed RoI Benchmark RoI The rate of improvement needed to “catch” up to the next benchmark. The rate of improvement of typically performing peers according to the norms Student’s Actual RoI Based on the available data points, this is the student’s actual rate of improvement per week Calculating Needed RoI In cell T3, type Needed RoI Click on cell T5 In the fx line (at top of sheet) type this formula =((S4-B5)/18) Then hit enter Your result should read: 2.83333... This formula simply subtracts the student’s actual beginning of year (BOY) benchmark from the expected middle of year (MOY) benchmark, then dividing by 18 for the first 18 weeks (1st semester). Calculating Benchmark RoI In cell U3, type Benchmark RoI Click on cell U4 In the fx line (at top of sheet) type this formula =SLOPE(B4:S4,B3:S3) Then hit enter Your result should read: 0.8825... This formula considers 18 weeks of benchmark data and provides an average growth or change per week. Calculating Student Actual RoI Click on cell U5 In the fx line (at top of sheet) type this formula =SLOPE(B5:S5,B3:S3) Then hit enter Your result should read: 2.5137... This formula considers 18 weeks of student data and provides an average growth or change per week. Graphing RoI For Individual Students Programming Microsoft Excel to Graph Rate of Improvement: Winter to Spring Setting Up Your Spreadsheet In cell A1, type 3rd Grade ORF In cell A2, type Second Semester In cell A3, type School Week In cell A4, type Benchmark In cell A5, type the Student’s Name (Swiper Example) Labeling School Weeks Starting with cell B3, type numbers 1 through 18 going across row 3 (horizontal). Numbers 1 through 18 represent the number of the school week. You will end with week 18 in cell S3. Labeling Dates Note: You may choose to enter the date of that school week across row 2 to easily identify the school week. Entering Benchmarks (3rd Grade ORF) In cell B4, type 92. This is your fall benchmark. In cell S4, type 110. This is your winter benchmark. Entering Student Data (Sample) Enter the following numbers, going across row 5, under corresponding week numbers. Week 1 – 74 Week 3 – 85 Week 4 – 89 Week 5 – 69 Week 6 – 85 Week 7 – 96 Week 8 – 90 Week 9 – 84 Week 10 – 106 Week 11 – 94 Week 15 – 100 *CAUTION* If a student was not assessed during a certain week, what do you put in that cell? Why? Graphing the Data Highlight cells A4 and A5 through S4 and S5 Follow Excel 2003 or Excel 2007 directions from here Graphing the Data Excel 2003 Across the top of your worksheet, click on “Insert” In that drop-down menu, click on “Chart” Excel 2007 Click Insert Find the icon for Line Click the arrow below Line Graphing the Data Excel 2003 A Chart Wizard window will appear Excel 2007 6 graphics appear Graphing the Data Excel 2003 Choose “Line” Choose “Line with markers…” Excel 2007 Choose “Line with markers” Graphing the Data Excel 2003 “Data Range” tab “Columns” Excel 2007 Your graph appears Graphing the Data Excel 2003 “Chart Title” “School Week” X Axis “WPM’ Y Axis Excel 2007 Change your labels by right clicking on the graph Graphing the Data Excel 2003 Choose where you want your graph Excel 2007 Your graph was automatically put into your data spreadsheet Graphing the Trendline Excel 2003 Excel 2007 Right click on any of the student data points Graphing the Trendline Excel 2003 Choose “Linear” Excel 2007 Graphing the Trendline Excel 2003 Excel 2007 Choose “Custom” and check box next to “Display equation on chart” Graphing the Trendline Clicking on the equation highlights a box around it Clicking on the box allows you to move it to a place where you can see it better Graphing the Trendline You can repeat the same procedure to have a trendline for the benchmark data points Suggestion: label the trendline Expected ROI Move this equation under the first Individual Student Graph 120 y = 1.0588x + 90.941 y = 1.8872x + 74.81 100 80 Benchmark Student: Sw iper 60 Rate of Improvement Expected RoI 40 20 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Challenge! What was the first equation? What is the slope of that equation? What was the second equation? What is the slope of that equation? Describe the achievement gap at the end of the school year. Programming Excel Second Semester Calculating Needed RoI Calculating Benchmark RoI Calculating Student’s Actual RoI Calculating Needed RoI In cell T3, type Needed RoI Click on cell T5 In the fx line (at top of sheet) type this formula =((S4-B5)/18) Then hit enter Your result is _____ ? This formula simply subtracts the student’s actual middle of year (MOY) benchmark from the expected end of year (EOY) benchmark, then dividing by 18 for the first 18 weeks (1st semester). Calculating Benchmark RoI In cell U3, type Benchmark RoI Click on cell U4 In the fx line (at top of sheet) type this formula =SLOPE(B4:S4,B3:S3) Then hit enter Your result should read: ____? This formula considers 18 weeks of benchmark data and provides an average growth or change per week. Calculating Student Actual RoI Click on cell U5 In the fx line (at top of sheet) type this formula =SLOPE(B5:S5,B3:S3) Then hit enter Your result should read: 1.89 This formula considers 18 weeks of student data and provides an average growth or change per week. Why Graph only 18 Weeks at a Time? Assuming Linear Growth… …Finding Curve-linear Growth Non-Educational Example of Curve-linear Growth Weight Loss Chart 205 200 10 Week RoI = -2.5 First 5 Weeks RoI = -3.6 Second 5 Weeks RoI = -1.5 200 197.5 195 193 Weight 190 189.5 186 185 184 182.5 181 180 179.5 178 175 170 165 1 2 3 4 5 6 Weeks 7 8 9 10 70 Academic Example of Curvilinear Growth 60 50 40 WCPM MOY to EOY = 1.19 30 20 BOY to MOY = 1.60 BOY to EOY = 1.35 10 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 Weeks McCrea, 2010 Looked at Rate of Improvement in small 2nd grade sample Found differences in RoI when computed for fall and spring: Ave RoI for fall: 1.47 WCPM Ave RoI for spring: 1.21 WCPM Ardoin & Christ, 2008 Slope for benchmarks (3x per year) More growth from fall to winter than winter to spring Christ, Yeo, & Silberglitt, in press Growth across benchmarks (3X per year) More growth from fall to winter than winter to spring Disaggregated special education population Graney, Missall, & Martinez, 2009 Growth across benchmarks (3X per year) More growth from winter to spring than fall to winter with R-CBM. Fien, Park, Smith, & Baker, 2010 Investigated relationship b/w NWF gains and ORF/Comprehension Found greater NWF gains in fall than in spring. DIBELS th (6 ) ORF Change in Criteria 2nd Fall to Winter 24 Winter to Spring 22 3rd 15 18 4th 13 13 5th 11 9 6th 11 5 AIMSweb Norms Based on 50th Percentile Fall to Winter Winter to Spring 1st 18 31 2nd 25 17 3rd 22 15 4th 16 13 5th 17 15 6th 13 12 Speculation as to why Differences in RoI within the Year Relax instruction after high stakes testing in March/April; a PSSA effect. Depressed BOY benchmark scores due to summer break; a rebound effect (Clemens). Instructional variables could explain differences in Graney (2009) and Ardoin (2008) & Christ (in press) results (Silberglitt). Variability within progress monitoring probes (Ardoin & Christ, 2008) (Lent). ROI as a Decision Tool within a Problem-Solving Model Steps 1. 2. 3. 4. Gather the data Ground the data & set goals Interpret the data Figure out how to fit Best Practice into Public Education Step 1: Gather Data Universal Screening Progress Monitoring Common Screenings in PA DIBELS AIMSweb MBSP 4Sight PSSA Validated Progress Monitoring Tools DIBELS AIMSweb MBSP www.studentprogress.org Step 2: Ground the Data 1) To what will we compare our student growth data? 2) How will we set goals? Multiple Ways to Look at Growth Needed Growth Expected Growth & Percent of Expected Growth Fuchs et. al. (1993) Table of Realistic and Ambitious Growth Growth Toward Individual Goal* *Best Practices in Setting Progress Monitoring Goals for Academic Skill Improvement (Shapiro, 2008) Needed Growth Difference between student’s BOY (or MOY) score and benchmark score at MOY (or EOY). Example: MOY ORF = 10, EOY benchmark is 40, 18 weeks of instruction (40-10/18=1.67). Student must gain 1.67 wcpm per week to make EOY benchmark. Expected Growth Difference between two benchmarks. Example: MOY benchmark is 20, EOY benchmark is 40, expected growth (4020)/18 weeks of instruction = 1.11 wcpm per week. Looking at Percent of Expected Growth Tier I Tier II Tier III Greater than 150% Between 110% & 150% Possible LD Between 95% & 110% Likely LD Between 80% & 95% Below 80% May Need More May Need More Needs More Needs More Likely LD Likely LD Tigard-Tualatin School District (www.ttsd.k12.or.us) Oral Reading Fluency Adequate Response Table Realistic Ambitiou Growth s Growth 1st 2.0 3.0 2nd 1.5 2.0 3rd 1.0 1.5 4th 0.9 1.1 5th 0.5 0.8 Fuchs, Fuchs, Hamlett, Walz, & Germann (1993) Digit Fluency Adequate Response Table Realistic Growth Ambitious Growth 1st 0.3 0.5 2nd 0.3 0.5 3rd 0.3 0.5 4th 0.75 1.2 5th 0.75 1.2 Fuchs, Fuchs, Hamlett, Walz, & Germann (1993) If Local Criteria are Not an Option Use norms that accompany the measure (DIBELS, AIMSweb, etc.). Use national norms. Making Decisions: Best Practice Research has yet to establish a blue print for ‘grounding’ student RoI data. At this point, teams should consider multiple comparisons when planning and making decisions. Making Decisions: Lessons From the Field When tracking on grade level, consider an RoI that is 100% of expected growth as a minimum requirement, consider an RoI that is at or above the needed as optimal. So, 100% of expected and on par with needed become the limits of the range within a student should be achieving. Is there an easy way to do all of this? Oral Reading Fluency 01/15/09 01/22/09 01/29/09 02/05/09 02/12/09 02/19/09 02/26/09 03/05/09 03/12/09 03/19/09 03/26/09 04/02/09 04/09/09 04/16/09 04/23/09 04/30/09 05/07/09 05/14/09 1 Benchmark Aiden Ava Noah Olivia Liam Hannah Gavin Grace Oliver Peyton Josh Riley Mason Zoe Ian Faith David Alexa Hunter Caroline 2 3 4 5 6 7 8 9 10 11 12 13 14 68 40 49 43 49 48 65 17 18 Needed RoI* Actual RoI** % of Expected RoI 49 45 60 71 95 1.61 2.17 167% 77 57 54 87 92 2.28 2.76 213% 69 61 54 84 2.28 2.01 156% 57 70 79 83 1.39 1.50 116% 36 54 70 83 1.94 1.58 122% 52 60 82 1.72 1.20 93% 67 68 84 79 1.44 1.66 129% 46 60 74 79 2.06 1.76 136% 51 51 57 78 2.22 1.45 112% 53 54 64 64 69 40 53 48 44 63 46 68 50 49 38 42 49 53 1.29 52 49 55 50 16 90 61 59 15 47 58 75 77 1.50 1.12 87% 55 48 36 67 77 2.28 1.62 125% 54 69 67 50 76 2.67 1.76 136% 49 50 64 74 2.06 1.17 91% 34 38 42 68 55 51 58 3.11 1.44 111% 41 31 45 49 47 30 46 2.72 0.24 19% 29 36 35 36 36 29 45 44 3.39 0.75 58% 30 23 44 52 43 19 63 38 3.33 0.79 61% 18 19 25 33 33 23 28 37 4.00 0.94 73% 23 23 48 38 32 34 3.72 0.75 58% 28 20 40 37 19 30 3.44 0.02 2% * Needed RoI based on difference betw een w eek 1 score and Benchmark score for w eek 18 divided by 18 w eeks 53 24 28 Expected RoI at Benchmark Level 25 Oral Reading Fluency Adequate Response Table ** Actual RoI based on linear regression of all data points Benchmarks based on DIBELS Goals 60 Realistic Grow thAmbitious Grow th 1st Grade 2.0 3.0 2nd Grade 1.5 2.0 3rd Grade 1.0 1.5 4th Grade 0.9 1.1 5th Grade 0.5 0.8 (Fuchs, Fuchs, Hamlett, Walz, & Germann 1993) 1/14/2011 1/121/2011 1/28/2011 5/14/2011 % of Expected Needed RoI Actual RoI RoI 1 2 3 18 Benchmark 68 90 1.29 Student 22 27 56 3.78 1.89 147% Access to Spreadsheet Templates http://sites.google.com/site/rateofimprove ment/home Click on Charts and Graphs. Update dates and benchmarks. Enter names and benchmark/progress monitoring data. What about Students not on Grade Level? Determining Instructional Level Independent/Instructional/Frustrational Instructional often b/w 40th or 50th percentile and 25th percentile. Frustrational level below the 25th percentile. AIMSweb: Survey Level Assessment (SLA). Setting Goals off of Grade Level 100% of expected growth not enough. Needed growth only gets to instructional level benchmark, not grade level. Risk of not being ambitious enough. Plenty of ideas, but limited research regarding Best Practice in goal setting off of grade level. Possible Solution (A) Weekly probe at instructional level and compare to expected and needed growth rates at instructional level. Ambitious goal: 200% of expected RoI Oral Reading Fluency 01/15/10 01/22/10 01/29/10 02/05/10 02/12/10 02/19/10 02/26/10 03/05/10 03/12/10 03/19/10 03/26/10 04/02/10 04/09/10 04/16/10 04/23/10 04/30/10 05/07/10 05/14/10 1 5th Grade 2 3 4 5 6 7 8 9 10 11 12 16 17 18 124 0.53 134 128 143 1.11 2.15 407% 92 101 115 116 129 108 121 1.11 1.43 271% 94 108 95 121 135 126 109 113 1.67 1.32 248% 99 66 100 83 92 107 109 93 1.39 0.89 168% 110 97 124 113 131 132 142 0.78 2.33 440% 96 74 108 62 107 92 94 95 1.56 0.39 74% 73 93 79 114 111 112 87 116 2.83 2.03 383% 102 112 122 103 118 135 105 112 1.22 0.51 97% 111 AB 121 121 134 131 99 120 0.72 0.11 21% 110 90 118 103 119 121 122 119 0.78 1.21 228% 105 90 79 89 120 123 119 0.94 1.22 231% 91 90 105 118 60 72 80 81 43 59 76 92 120 57 0.76 47 65 4.14 -0.06 -8% 137 110 2.71 4.94 646% 115 124 85 79 2nd Grade 15 % of Expected RoI 104 107 3rd Grade 14 115 104 4th Grade 13 Needed RoI* Actual RoI** 0.53 55 57 66 76 66 47 66 6.27 0.49 93% 72 84 92 94 82 76 82 3.25 -0.08 -15% 81 91 70 65 73 3.21 -1.41 -267% 70 104 79 79 68 4.50 -1.45 -274% 68 90 74 63 83 91 70 104 84 71 74 86 82 77 91 1.29 0.94 1.15 89% -0.06 2.21 171% Possible Solution (B) Weekly probe at instructional level for sensitive indicator of growth. Monthly probes (give 3, not just 1) at grade level to compute RoI. Goal based on grade level growth (more than 100% of expected). Step 3: Interpreting Growth What do we do when we do not get the growth we want? When to make a change in instruction and intervention? When to consider SLD? When to make a change in instruction and intervention? Enough data points (6 to 10)? Less than 100% of expected growth. Not on track to make benchmark (needed growth). Not on track to reach individual goal. When to consider SLD? Continued inadequate response despite: Fidelity with Tier I instruction and Tier II/III intervention. Multiple attempts at intervention. Individualized Problem-Solving approach. Evidence of dual discrepancy… 05/14/09 Needed Ro I* A c tual Ro I** 18 90 % o f Expec ted Ro I Dual Disc repanc y? Keep On Truckin Keep On Truckin 1.29 95 1.61 2.17 167% 92 2.28 2.76 213% 84 2.28 2.01 156% 83 1.39 1.50 116% 83 1.94 1.58 122% 82 1.72 1.20 93% 79 1.44 1.66 129% 79 2.06 1.76 136% 78 2.22 1.45 112% 77 1.50 1.12 87% 77 2.28 1.62 125% 76 2.67 1.76 136% 74 2.06 1.17 91% 58 3.11 1.44 111% 46 2.72 0.24 19% 44 3.39 0.75 58% 38 3.33 0.79 61% 37 4.00 0.94 73% 34 3.72 0.75 58% 30 3.44 0.02 2% BIG BIG BIG BIG BIG BIG PROBLEMS PROBLEMS PROBLEMS PROBLEMS PROBLEMS PROBLEMS Growth Criteria >125% 85% - 125% <85% Three Levels of Examples Whole Class Small Group Individual Student - Academic Data - Behavior Data Whole Class Example Computation 01/15/10 01/22/10 01/29/10 02/05/10 02/12/10 02/19/10 02/26/10 03/05/10 03/12/10 03/19/10 03/26/10 04/02/10 04/09/10 04/16/10 04/23/10 04/30/10 05/07/10 05/14/10 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Needed RoI* Actual RoI** % of Expected RoI 0.35 50th Percentile 25 31 25th Percentile 19 23 Student 6.5 9 8 Student 6 7.5 8.5 Student 4.5 Student 13 Student 8.5 0.24 5.5 11 13 1.72 0.61 5 11 11.5 1.72 0.57 161% 5.5 6.5 9.5 10.5 1.72 1.06 300% 173% 8 9.3 8 5.6 9.6 9.6 1.72 -0.23 -66% 8 10.5 10.5 5.6 9.3 9 1.72 -0.03 -7% 9 8 4 8 9 1.72 0.07 21% 6 10.5 9 1.72 0.43 122% 6 8 1.72 0.07 20% 7 1.72 -0.25 -71% -119% Student 8.5 5.5 Student 6.5 5.5 Student 6.5 9 4.5 Student 8 10.5 4.5 6.5 4 Student 9 10 5.6 6.6 5 4.6 6.6 1.72 -0.42 8 8 8.5 4 8 6.6 1.72 -0.18 -51% 3.5 6.5 1.72 -0.24 -67% 26% Student Student 9 4.5 4.5 4 3.5 Student 6.5 5 6.5 9 7.5 6.5 1.72 0.09 Student 5.5 3 8 4 6.5 6.3 1.72 0.19 55% Student 7.5 10 6.6 3.3 3 6.3 1.72 -0.46 -130% Student 5 5.5 6.5 6 5 6 1.72 0.04 11% Student 5 4 8 8.5 10 8 6 1.72 0.25 71% Student 4.5 3.5 5.5 1.72 -0.03 -8% 5 5.3 1.72 -0.14 -40% Student 6 5 2.5 5.5 4.5 10.5 * Needed RoI based on difference betw een w eek 1 score and Benchmark score for w eek 18 divided by 18 w eeks 11 Digit Fluency Adequate Response Table ** Actual RoI based on linear regression of all data points Percentiles based on AIMSw eb Grow th Tables Expected RoI at 50th Percentile Expected RoI at 25th Percentile Realistic Grow thAmbitious Grow th 1st Grade 0.3 0.5 2nd Grade 0.3 0.5 3rd Grade 0.3 0.5 4th Grade 0.75 1.2 5th Grade 0.75 1.2 (Fuchs, Fuchs, Hamlett, Walz, & Germann 1993) 3rd Grade Math Whole Class Who’s responding? Effective math instruction? Who needs more? N=19 4 > 100% growth 15 < 100% growth 9 w/ negative growth Small Group Example Oral Reading Fluency 09/11/09 09/18/09 09/25/09 10/02/09 10/09/09 10/16/09 10/23/09 10/30/09 11/06/09 11/13/09 11/20/09 11/27/09 12/04/09 12/11/09 12/18/09 01/01/10 01/08/10 01/15/10 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 Needed RoI* Actual RoI** % of Expected RoI 68 1.41 Benchmark 44 Student 35 39 41 45 42 45 52 57 62 1.83 1.49 106% Student 28 38 42 40 50 55 64 72 74 2.22 2.77 196% Student 26 28 32 31 27 29 35 34 38 2.33 0.57 41% Student 31 35 39 45 42 47 53 58 65 2.06 1.90 135% Student 40 44 38 48 52 64 72 74 78 1.56 2.62 186% * Needed RoI based on dif ference between week 1 score and Benchmark score for week 18 divided by 18 weeks Oral Reading Fluency Adequte Response Table ** Actual RoI based on linear regression of all data points Benchmarks based on DIBELS Goals Expected RoI at Benchmark Level Realistic GrowthAmbitious Growth 1st Grade 2.0 3.0 2nd Grade 1.5 2.0 3rd Grade 1.0 1.5 4th Grade 0.9 1.1 5th Grade 0.5 0.8 (Fuchs, Fuchs, Hamlett, Walz, & Germann 1993) Intervention Group Intervention working for how many? Can we assume fidelity of intervention based on results? Who needs more? Individual Kid Example 2nd Grade Reading Progress 100 y = 1.5333x + 42.8 90 90 80 79 Words Read Correct Per Minute 74 70 68 60 60 56 53 y = 0.9903x + 36.873 53 50 48 46 45 44 40 31 30 20 10 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 09/12/08 09/19/0809/26/0810/03/08 10/10/08 10/17/08 10/24/08 10/31/08 11/07/08 11/14/08 11/21/08 11/28/08 12/05/08 12/12/08 12/19/08 01/16/09 01/23/09 01/30/09 02/06/0902/13/09 02/20/0902/27/0903/06/09 03/13/0903/20/0903/27/0904/03/09 04/10/09 04/17/0904/24/09 05/01/09 Benchmark Linear (Benchmark) Linear Individual Kid Making growth? How much (65% of expected growth). Atypical growth across the year (last 3 data points). Continue? Make a change? Need more data? RoI and Behavior? Percent of Time Engaged in Appropriate Behavior 100 90 y = 7.2143x - 1.5 80 70 y = 3.9x + 19.8 Percent 60 50 40 y = 2x + 22 30 20 10 0 1 2 Baseline 3 4 Condition 1 5 6 Condition 2 7 8 9 Linear (Baseline) 10 11 12 Linear (Condition 1) 13 14 Linear (Condition 2) 15 16 17 Linear (Condition 2) 18 Step 4: Figure out how to fit Best Practice into Public Education Things to Consider Who is At-Risk and needs progress monitoring? Who will collect, score, enter the data? Who will monitor student growth, when, and how often? What changes should be made to instruction & intervention? What about monitoring off of grade level? Who is At-Risk and needs progress monitoring? Below level on universal screening Entering 4th Grade Example DORF (110) ISIP TRWM (55) 4Sight (1235) PSSA (1235) Student A 115 58 1255 1232 Student B 85 48 1216 1126 Student C 72 35 1056 1048 Who will collect, score, and enter the data? Using MBSP for math, teachers can administer probes to whole class. DORF probes must be administered oneon-one, and creativity pays off (train and use art, music, library, etc. specialists). Schedule for progress monitoring math and reading every-other week. Week 1 Reading 1st Reading X X X X X Math X X 4th 5th Math X 2nd 3rd Week 2 X X Who will monitor student growth, when, and how often? Best Practices in Data-Analysis Teaming (Kovaleski & Pedersen, 2008) Chambersburg Area School District Elementary Response to Intervention Manual (McCrea et. al., 2008) Derry Township School District Response to Intervention Model (http://www.hershey.k12.pa.us/56039310111408/lib/56039310111408/_files/Microsoft_Word__Response_to_Intervention_Overview_of_Hershey_Elementary_Model.pdf) What changes should be made to instruction & intervention? Ensure treatment fidelity!!!!!!!! Increase instructional time (active and engaged) Decrease group size Gather additional, diagnostic, information Change the intervention Final Exam… Student Data: 27, 29, 26, 34, 27, 32, 39, 45, 43, 49, 51, --, --, 56, 51, 52, --, 57. Benchmark Data: BOY = 40, MOY = 68. What is student’s RoI? How does RoI compare to expected and needed RoIs? What steps would your team take next? What if Benchmarks were 68 and 90 instead? The RoI Web Site http://sites.google.com/site/rateofimprovement/ Caitlin Flinn Bennyhoff CaitlinFlinn@hotmail.com Andy McCrea Download powerpoints, handouts, Excel graphs, charts, articles, etc. andymccrea70@gmail.com Matt Ferchalk mferchalk@norleb.k12.pa.us