Two Multi-Site Randomized Control Trials: Bottom-Up vs. Top-Down Approaches to Scaling Up PALS Is Response to Word-Problem Intervention among Students with MD Moderated by Concurrent RD? Doug Fuchs and Lynn Fuchs Vanderbilt University 1 Bottom-Up vs. Top-Down Approaches to Scaling Up PALS Doug Fuchs, Kristen McMaster, Laura Saenz, Devin Kearns, Lynn Fuchs, Loulee Yen, Don Compton, and Chris Lemons Vanderbilt University Chris Schatschneider Florida State University R305G04104 Institute of Education Sciences 2 Purpose of PALS Supplements the general education core program Implemented 3 times per week in reading; 2 times per week in math Creates a “routine” for teachers to differentiate instruction by creating many simultaneous peermediated lessons rather than one teacher-directed lesson PALS Reading: kindergarten, first grade, grades 2-6, high school PALS math: kindergarten, first grade, grades 2-6 3 PALS Research • • • • • • • Based on Juniper Gardens Classwide Peer Tutoring Over 15 years of experimental research Title I and Non-Title I schools Urban and suburban schools High, average, and low achievers Students with learning disabilities “Validated Practice” status (USDE, WWC, BEE) 4 Grades 2-6 PALS Partner Reading Paragraph Shrinking Prediction Relay 5 Partner Reading • • • • • • Conducted for 11-12 minutes Stronger reader reads aloud for 5 minutes Weaker reader reads same text aloud for 5 minutes Weaker reader retells story for 1-2 minutes Readers read quickly, correctly, and with expression Coaches listen, correct mistakes, and mark points (1 point for each correctly read sentence and 10 points for story retell) 6 Paragraph Shrinking • Conducted for 10 minutes • For 5 minutes: • Stronger reader reads new text aloud, summarizing paragraph by paragraph – Name the most important who or what (1 point) – Name the most important thing about the who or what (1 point) – Shrink it to 10 or fewer words (1 point) • For next 5 minutes: • Weaker reader reads new text aloud, summarizing paragraph by paragraph (as above) • Coach listens, corrects mistakes, and marks points 7 Prediction Relay 5 minutes, stronger reader read new text − − − − − − Makes prediction (1) Reads half page (1) Checks prediction (1) States main idea (3) Makes new prediction Continues to read 5 minutes, weaker reader continues on in new text, with the same activities Coach listens, corrects mistakes, and marks points 8 Two Kinds of PALS Research Randomized Controlled Trials Study schools include Title I and no Title I. Classrooms randomly assigned within schools to PALS and control groups. HA, AA, LA (including LD) students targeted in each classroom. Fidelity of treatment implementation. Individually administered pre-/posttests by trained examiners. 9 Improvement in Reading Improvement Over 16 Weeks 80 Experimental Control 60 40 20 0 Learning Disabled Low Achieving Average Achieving High Achieving 10 Two Kinds of PALS Research School-Improvement Projects Title I schools implement PALS school-wide. Our TA funded by Nashville’s Title I office No fidelity of treatment assessed. Group administered high-stakes tests. 11 Report Card Scores Based on Students’ Performance on the TCAP (CTB/McGraw-Hill) GOWER Subject ‘93 ‘94 MATH 76.0 (61) 107.8 (12) READING 74.0 (60) 112.5 (25) Language Arts 61.7 (60) 91.8 (33) Science 74.4 (58) 95.6 (24) Social Studies 60.1 (61) 81.4 (53) Note 1: A score of 100 means that students of a school are progressing at a rate equivalent to that of the national rate. Note 2: The numbers in parentheses represent Gower’s standing in relation to the Metro Schools’ other 66 elementary schools. 12 Typical Teacher Support in PALS Research Support was ongoing and on-site: RAs gave in-class assistance 1x or 2x per wk during training and implementation. RA support is costly, unlikely in wider implementations, an obstacle to scaling up. Absent such support, quality of implementations suffer. How to separate an intervention from its support system? How to scale-up (“export” the intervention from A to B) without researchers’ nurturance? There’s also a matter of transcending time…. 13 Purpose of the Present Study With Grades 2-6 PALS as a “prop,” and students’ reading achievement as the criterion, does a “bottom-up” approach beat a “top-down” approach to scaling up? Do these approaches affect teachers’ sustainability of PALS? 14 METHOD 15 Participating Sites Vanderbilt University (Nashville) University of Minnesota (Twin Cities Area) The University of Texas Pan American (Hidalgo County) Where K-PALS was developed and researched Large urban area, diverse student population Southern-most region of TX along the U.S. Mexico Border Has K-PALS “expertise”(Project Director, Coordinators) Schools had some prior experience with PALS Schools had no prior experience with PALS “PALS” has brandname recognition Schools regularly participate in University research projects Schools had no prior experience participating in research 16 Participating Teachers Two cohorts of 3rd-, 4th-, and 5th-grade teachers : − Cohort 1 Entered study in 2006-07 − Cohort 2 Entered study in 2007-08 Two years of study participation: − Year 1 Assigned randomly to PALS or Control − Year 2 PALS Teachers selected Top Down or Bottom Up PALS Control Teachers remained in Control group 17 Teachers by Study Group in Year 2 Cohort 1 Tennessee Minnesota Texas Total Top Down Bottom Up Control 5 5 6 16 5 5 7 17 7 4 7 18 Cohort 2 12 8 9 Tennessee 6 9 12* Minnesota 8 4 7 Texas 26 21 28* Total TOTAL 42 38 46* Teachers *This number is based on all MN controls; 10 were eliminated, right? 18 Students by Study Group in Year 2 Cohort 1 Tennessee Minnesota Texas Total Cohort 2 Tennessee Minnesota Texas Total TOTAL Students Top Down Bottom Up Control 60 61 61 182 60 65 90 215 68 49 88 205 136 72 95 303 86 109 46 241 102 151* 73 326* 485 456 531* 19 Study Conditions: Year 1 Control – Teachers implemented core language arts curriculum PALS – Teachers implemented with fidelity: − 3 times/week for 35-40 min (about 54 sessions) − Coaches and Readers: higher-performing readers paired with lower-performing readers − Four PALS Activities: Partner Reading (10 min) Retell (2 min) Paragraph Shrinking (10 min) Prediction Relay (10 min) 20 Study Conditions: Year 2 Control – Teachers implemented core language arts curriculum “Top Down” (TD) PALS − Teachers did PALS “by the book” − Fidelity of PALS implementation was emphasized “Bottom Up” (BU) PALS − Teachers implemented core components of PALS − Customization was strongly encouraged and supported 21 BU PALS: Core Elements 48 sessions minimum 35 minutes per session minimum 10 minutes of Partner Reading 10 minutes of Paragraph Shrinking A motivational peer reinforcement system 22 22 BU PALS: Requirements 23 Teachers asked to: − Conduct core elements of PALS as designed − Develop changes Match to curriculum, students’ needs, teaching style Create a type of PALS for the long term 23 Student Measures Academic Measures Test of Word Reading Efficiency (TOWRE) Woodcock Reading Mastery TestRevised (WRMT-R) − Word Identification Subtest Wide Range Achievement Test (WRAT) − Letter and Word Identification Subtests Comprehensive Reading Assessment Battery (CRAB; 2 passages) − Oral reading (1 min & 3 min) − Comprehension (10 open-ended questions) CBM Maze Task (2 passages) − Correct maze choices made in 2.5 min Iowa Test of Basic Skills (ITBS) − Reading Comprehension − Vocabulary Student Characteristics Demographics SWAN − Teachers rated each student’s abilities to focus attention, control activity, and inhibit impulses Teacher ratings − Teachers rated each student’s effort in reading and behavior in the classroom 24 Teacher Measures Classroom Measures PALS Calendars PALS Fidelity Language Arts Observation Classroom Atmosphere Rating Scale (Wehby) Survey of Enacted Curriculum (SEC): English and Language Arts Teacher Characteristics Demographics Berends teacher survey (assesses school climate, teacher professional development, teacher efficacy, etc.) 25 Procedures Pretesting (September-October) PALS Workshops (September-October) − Year 1: All teachers attend same workshop − Year 2: Separate TD and BU workshops PALS Implementation (~18 weeks) − − − − Teachers implemented 3 times per week for 35-40 min Weekly classroom visits from project staff Three “booster” sessions for TD and BU PALS teachers Two fidelity observations Language arts observations in PALS and Control classrooms − 45-60 min − Momentary time sampling of a variety of reading instructional components − Supplementary field notes Posttesting (March-May) 26 Scaling-Up PALS for Grades 2-6 Results 27 Organization of Study Year 1 (2006-07) COHORT 1 (1st YEAR) Top-Down PALS (W + Year 2 (2007-08) COHORT 1 (2nd YEAR) B + Top-Down PALS B Bottom-Up PALS Year 3 (2008-09) COHORT 2 (1st YEAR) COHORT 2 (2nd YEAR) Top-Down PALS Top-Down PALS Tutor) Top-Down PALS (W + Control Bottom-Up PALS + Helpers) Control Control Control 28 Analysis Procedures Create latent pretest and posttest variables combining 5 reading measures into 1 Create a latent change score − Produces an “error-free” change value Run 2-level HLM analyses − Outcome: Latent change score − Variables: Treatment condition (TD, BU, Control); Site (TN, MN, TX); latent pretest score − Random effects: Level 2 teacher effects; ICC = .10 Test comparability of groups on variables plausibly related to selection of TD or BU 29 Descriptive Statistics Table 1. Pretest-posttest raw score gains Reading Measure Top-Down Bottom-Up Control M SD M SD M SD 5.08 11.51 5.41 12.29 4.47 11.55 5.45 12.67 6.08 12.26 4.95 12.02 3.15 18.68 3.39 8.79 2.88 8.75 22.18 44.91 23.78 42.30 20.19 43.80 4.72 8.25 4.78 8.30 4.52 8.31 -0.03 0.97 0.18 0.95 -0.17 1.05 WRMT-R WID words correct TOWRE SWE words correct ITBS RC items correct CRAB Fluency words correct CBM Maze correct completions Latent change score (z-scored) Note: WRMT-R WID = Woodcock Reading Mastery Test, Revised, Word Identification subtest; TOWRE SWE = Test of Word Reading Efficiency, Sight Word Efficiency subtest; ITBS RC = Iowa Test of Basic Skills, Reading Comprehension subtest; CRAB = Comprehensive Reading Assessment Battery; CBM = Curriculum-Based Measurement 30 Regression Analysis Table 2. Multilevel Regression Effects Comparison Estimate SE PALS vs. Control TD vs. Control BU vs. Control TD vs. BU 0.247 0.150 0.343 -0.193 Random Effects Variance Teacher Residual 0.102 0.872 0.087 0.097 0.100 0.095 SE 0.024 0.035 t 2.85 ** 1.54 3.42 *** 2.03 * z 4.20 24.79 Note: Degrees of freedom for fixed effects = 109. * = p <.05; ** = p <.01; *** = p <.001 31 Effects for Study Groups by LA, AA, and HA Students Table 3. Child-level effect sizes for PALS (Hedges' g ) Student type Overall Low Achieving Average Achieving High Achieving PALS v. Control TD v. Control BU v. Control BU v. TD .25 .26 .23 .28 .15 .08 .19 .20 .34 .43 .27 .37 .19 .35 .08 .18 Note: Hedges' g effect sizes calculated using recommendations for What Works Clearinghouse reporting (IES, 2007) BU=Bottom-Up; TD=Top-Down; PALS = Peer-Assisted Learning Strategies. 32 Effects for Study Groups by LA, AA, and HA Students Effect Size (Hedges' g) Effect Sizes for PALS .50 .45 .40 .35 .30 * .25 Overall * .20 .15 * Low Average High .10 .05 .00 B U P A LS TD P A LS vs.C ontrol vs.C ontrol B U P A LS vs.TD P A LS P A LS vs.C ontrol 33 Is Response to Word-Problem Intervention among Students with MD Moderated by Concurrent RD? Lynn Fuchs, Sarah Powell, Pamela Seethaler, Paul Cirino, Jack Fletcher, Doug Fuchs, Carol Hamlett, and Rebecca Zumeta Vanderbilt University and University of Houston Journal of Educational Psychology, 2009 Grant #P01046261 National Institute of Child Health and Human Development 34 35 Study Purposes Examine the efficacy of tutoring protocols for remediating − Math fact deficits − Word problem deficits Assess whether treatment efficacy is different for − Students with MD alone versus − Students with MDRD Determine whether effects are comparable as a function of site − Nashville, where the tutoring protocols were developed − Houston, a site distal to the developers 36 Participants 924 students screened in 63 classrooms in 18 schools in Nashville and Houston (similar sample size at each site) Inclusion criteria: − WRAT-A: < 26th percentile − 5-item word-problem measure: score < 2 − At least 1 (of 2) WASI subtest T score: > 36 162 students eligible for the study; 133 students remained at posttesting Blocking on site (Nashville and Houston) and MD status (MD vs. MDRD), students randomly assigned to tutoring conditions: − Math Facts Tutoring (“Math Flash”) − Word Problem Tutoring (“Pirate Math”) − Control 37 Participants Treatment groups comparable on all variables MD vs. MDRD differences (across treatment groups) as expected Age Female Sub. Lunch Spec. Ed. WASI IQ WRAT-A WRAT-R MD 9 40% 68% 8% 92 88 105 MDRD 9 48% 90% 28% 85 81 78 38 Examined Efficacy of Two Tutoring Protocols Both Tutoring Protocols Delivered individually 48 sessions: 3 per week for 16 weeks 20-30 minutes per session Scripted lessons, which tutors studied (not read) Motivational system to ensure on-task behavior and hard, accurate work Each session audiotaped; tapes sampled and coded for fidelity, which was high for both tutoring conditions 39 Examined Efficacy of Two Tutoring Protocols The exclusive focus of Math Flash was math facts The primary focus of Pirate Math was word problems − but it also addressed foundational skills (math facts, procedural calculations, and algebra skills) 40 Pirate Math Tutoring 48 sessions: 3 per week for 16 weeks 20-30 minutes per session Scripted lessons, which tutors study (not read) Four units Foundational Skills for Word Problems Total Word Problems Difference Word Problems Change Word Problems 41 Pirate Math: Introductory Unit Teach students: − Efficient counting strategies to answer math facts − 2-digit procedural calculations − How to solve for X in addition and subtraction equations (a+b=c; x-y=z) − How to check work 42 Introductory Unit: Counting Up COUNTING UP Addition COUNTING UP Subtraction 1. Put the bigger number in your head and say it. 1. Put the minus number in your head and say it. 2. Count up the smaller number on your fingers. 2. Count up on your fingers to the number you started with. 3. Your answer is the last number you say. To ADD, you CAN reverse the numbers! 3. Your answer is the number of fingers you have up. To SUBTRACT, do NOT reverse the numbers. The minus number always goes first. Example: 5 - 2 = ? Example: 5 + 2 = ? 1. 2. 3. “5.” (Hold up 1 finger.) “6.” (Hold up 2 fingers.) “7.” “The answer is 7.” Example: 2 + 5 = ? 1. 2. 3. “5.” (Hold up 1 finger.) “6.” (Hold up 2 fingers.) “7.” “The answer is 7.” 1. 2. 3. “2.” (Hold up 1 finger.) “3.” (Hold up 2 fingers.) “4.” (Hold up 3 fingers.) “5.” “The answer is 3.” 43 Introductory Unit: Finding X in All 3 Positions of Equation If X is at the end of a number sentence, do what the problem tells you to do (e.g., 3 + 2 = X; 6 – 2 = X) If X is not at the end, and it’s an “X minus” problem, add (e.g., X – 2 = 4). If X is not at the end, and it’s not a X minus problem, subtract (e.g., X + 2 = 8; 5 – X = 2; 7 + X = 12). 44 Introductory Unit: Checking Work CHECKING YOUR WORK 1. Sense Does the work make sense? 2. Lining Up Did I line up the numbers correctly? 3. Math Did I add or subtract correctly? 4. Labels Did I use a label? 5. Signs Did I use signs correctly? 45 Remaining Units: Word-Problem Lessons Following Unit 1, four activities per session. 1. Flash-card warm up 2. Conceptual/strategic lesson using schema-broadening instruction 3. Sorting practice on identifying problem types 4. Paper/pencil review 46 1. Math Fact Flash Card Warm Up Math Fact flash cards comprise 200 addition and 11 subtraction facts - 6 − Sums 0-18 − Subtrahends 0-18 Tutor shows flash card to student: Know it or Count Up! − If student answers correctly, flash card placed in correct pile. − If student answers incorrectly, tutor asks student to 4 “Count Up”; once correct, goes in correct pile. + 5 − Student graphs score on graph. 47 2. Lesson Pirate Math RUN Students use “RUN” strategy for every word problem. RUN! 1. Read the problem. 2. Underline the question. 3. Name the problem type. Students learn to circle relevant information directly in the text or picture/graph/chart. 48 2. Lesson Pirate Math Setting Up Work Write the equation that goes with the problem type. Figure out what’s missing. Write X in your equation in the appropriate slot. Figure out what numbers are known. Write those numbers in the appropriate slots. Write the math signs. Find X. Make sure your answer has a number and a label. 49 2. Lesson Problem Types with Transfer Problem types at grade 2: Total, Difference, and Change Transfer features: − Irrelevant information − Money − Double-digit calculations − Finding relevant information in graphs and pictures. − Combining problem types. 50 2. Lesson Pirate Math Change Change problems with a starting amount that increases or decreases (a change) to make it a new amount. “Sarah had 10 pencils. Then she gave 4 pencils to Pamela. How many pencils does Sarah have now?” St +/- C = E CHANGE 1. How many do you start with? (St) 2. How many do you change? (C) Is there an increase? + Is there a decrease? 3. How many do you end with? (E) St + or - C = E 4. Write the number sentence. 5. Find X! 51 “Sarah had 10 pencils. Then, she gave 4 pencils to Pamela. How many pencils does Sarah have now?” 52 “Sarah had 10 pencils. Then, she gave 4 pencils to Pamela. How many pencils does Sarah have now?” Recognize problem type: Change problem. Write equation for Change problems: St +/- C = E. Identify missing information (E). Write that in the appropriate slot St +/- C = E X Identify the important given numbers (St and C). Write those in the appropriate slots. St +/- C = E 10 4 X Write math signs. St +/- C = E 10 - 4 = X Find X: X is at end so do what problem tells me to do: 10 – 4 = 6; X=6. Label answer: 6 pencils. 53 Lexie had some comic books in her desk. Then she bought 8 more. Now, she has 12 comic books. How many comic books did Lexie have in her desk to begin with? 54 Lexie had some comic books in her desk. Then she bought 8 more. Now, she has 12 comic books. How many comic books did Lexie have in her desk to begin with? Recognize problem type: Change problem. Write equation for Change problems: St +/- C = E. Identify missing information (St). Write that in the appropriate slot St +/- C = E X Identify the important given numbers (St and C). Write those in the appropriate slots. St +/- C = E X 8 12 Write math signs. St +/- C = E X + 8 = 12 Find X: X is not at end and it’s not an X minus problem, so subtract: 12 – 8 = 4; X=4. Label answer: 4 comic books. 55 Alicia has 3 friends in her math class. The chart shows how many stars Alicia and her friends earned on Monday. On Tuesday, Alicia lost 3 stars for talking. How many stars does she have now? Monday’s Star Chart Milo Trish David Alicia 0 2 4 6 8 10 Number of Gold Stars 56 Recognize problem type: Change problem. Identify transfer features: Irrelevant information (cross it out) and relevant information in a graph (number the graph). Write equation for Change problems: St +/- C = E. Identify missing information (St). Write that in the appropriate slot St +/- C = E X Identify the important given numbers (St and C). Write those in the appropriate slots. St +/- C = E 8 3 X Write math signs. St +/- C = E 8 - 3=X Find X: X is at end, so do what the problem says: 8 – 3 = 5; X=5. Label answer: 5 stars. 57 2. Lesson Pirate Math Total Total problems have two parts that are combined for a total. Total amount is the entire or combined amount. “Sarah has 5 pencils. Pamela has 3 pencils. How many pencils do the girls have in all?” P1 + P2 = T 58 2. Lesson Pirate Math Difference Difference problems compare two amounts to find the difference between them. “Sarah has 7 pencils. Pamela has 12 pencils. How many more pencils does Pamela have than Sarah?” B–s=D 59 3. Sorting Student sorts word problems by problem type for 2 minutes. Tutor reads cards to student. Student places cards on Sorting Mat. At end of 2 minutes, tutor counts number of correctly sorted cards and uses correction procedure for up to 3 incorrectly sorted cards. 60 3. Sorting Sorting Mat Total Difference Change ? 61 3. Sorting Maria and Jackie picked 16 flowers. Jackie picked 7 flowers. How many flowers did Maria pick? Maria picked 8 more flowers than Jackie. Jackie picked 4 flowers. How many flowers did Maria pick? Maria picked 11 flowers. Then Jackie took 4 of them for her Mom. How many flowers does Maria have now? 62 4. Paper/pencil review * 10 math facts * 4 double-digit calculations * 1 word problem 63 Motivation during Pirate Math Students earn coins throughout lesson for listening well, working hard, following directions, and correct work. At end of lesson, students color footsteps on treasure map equaling amount of coins earned. When students color 16 footsteps, they pick a prize from treasure box and receive a new map. 64 Examined Efficacy of Two Tutoring Programs The exclusive focus of math facts tutoring was math facts The primary focus of Pirate Math tutoring was word problems − but it also addressed foundational skills (math facts, procedural calculations, and algebra skills) 65 Efficacy: Fluency with Math Facts and Procedural Calculations On math facts, Pirate Math effects superior improvement compared to control group. No difference between Pirate Math and math facts tutoring. Notable, because Pirate Math only allocates an initial lesson and then 4-6 minutes per session on number combinations. On procedural calculations, Pirate Math effects superior improvement compared to control group and compared to math facts tutoring. Again, little time spent on procedural calculations. 66 Efficacy: Algebra On algebra, Pirate Math effects superior outcomes compared to control group and compared to math facts tutoring. Algebraic cognition improved even though students were severely deficient in math and young. Given strong focus on algebra in high schools, given graduation requirements for algebra, and given emphasis in NMAP, introducing algebra earlier in the curriculum may represent a productive innovation. 67 Correct Representation 68 Correct Representation 69 Incorrect Representation 70 Over Efficacy Results: Word Problems Work on these foundation skills (MFs, procedural calculations, algebra), combined with schema-broadening instruction, also produced differential growth on WP outcomes compared to MF tutoring group and compared to control group. MF tutoring did not result in improvement on WPs. Lack of transfer suggests that source of difficulty is not diverting attention from the complex mathematics to the MFs embedded in those problems, but rather failing to comprehend the relations among the numbers embedded in the narratives or to process the language in those stories adequately. Suggests that MFs may not be the bottleneck for WP performance. Instead, mathematics disability represents a more complicated pattern of difficulty, implicating language (as has been suggested elsewhere). 71 Is Tutoring Differentially Efficacious Depending on MD Status (MD vs. MDRD)? Why We Hypothesized MD students would be more responsive to tutoring than MDRD students For MFs − A key deficit among students with reading difficulty is phonological processing and − Phonological processing deficits are linked with difficulty in automatic retrieval of MFs. For WPs − Using text to construct a WP model involves language − Language profiles of students with MDRD are depressed compared to students with MD. 72 Is Tutoring Differentially Efficacious Depending on MD Status (MD vs. MDRD)? No evidence of differential responsiveness to intervention as a function of difficulty status on any outcome. Raises questions about the tenability of the MD/MDRD subtyping scheme and suggests the need to pursue other avenues for subtyping mathematics disability. Even so, across tutoring conditions and sites, students with MD outperformed students with MDRD at pre- and posttest. Additional work to examine the tenability of the MD/MDRD subtyping scheme is warranted, even as research pursuing alternative frameworks proceeds. 73 These Tutoring Protocols Are Transportable No MD/MDRD by treatment by site effects. No treatment by site effects. Tutoring protocols were comparably effective in Nashville and Houston, for MD and for MDRD students. 74 Overall Conclusions MF tutoring enhances fluency with MFs with transfer to procedural calculations but without transfer to algebra or WPs. For a comparable amount of tutoring time, WP tutoring (with work on foundational skills) enhances WP skill, fluency with MFs, procedural calculations, and algebra. 75 For Materials, Contact: Flora Murray flora.murray@vanderbilt.edu Vanderbilt University 228 Peabody College Department of Special Education Nashville, TN 37203 (615) 343-4782 76