The Impact of a Character Intervention on Student Outcomes: Results From Three Years of the Positive Action Program in the Chicago Randomized Trial Brian R. Flay, D.Phil. Professor Oregon State University Corvallis, OR David L. DuBois, Ph.D., Professor, Public Health, University of Illinois at Chicago Carol Gerber Allred, PhD Positive Action, Inc. Twin Falls, ID Peter Ji, Ph.D., Research Associate, Psychology University of Illinois at Chicago Presented at the Annual Meeting of the Society for Prevention Research, San Francisco, May 28 2008 Outline I. The Positive Action program philosophy/theory and components II. Prior evaluations and results III. Design of the Chicago study IV. Results from the Chicago study V. Conclusions 2 I. Overview of PA Program • Positive Action (PA) is a comprehensive school-based program designed to – promote student character and positive behavior, – prevent an array of student problem behaviors, and – improve student achievement. • PA is grounded in a broad theory of self-concept that posits – (a) students’ self-concepts and characters are determined by how they behave and how they feel about themselves when they do various behaviors and – (b) making positive and healthy behavioral choices results in feelings of self-worth. • Major features of the program include: – Teacher delivered scripted PA curriculum lessons in classroom – Teacher and school staff modeling/reinforcement of “PA behaviors” inside and outside of the classroom – School-wide activities (e.g., PA assemblies) led by principal and PA Committee 3 Positive Action Overview 1. The core of Positive Action is contained in an underlying philosophy. 2. The philosophy is described in the Thoughts-ActionsFeelings Circle. 3. Positive Action teaches the positive actions for the whole self through six units that are contained in five program components. 4. The five completely prepared components are: 1. K–12 curriculum, 2. Climate Development, 3. Counselors program, 4. Family program, and 5. Community program. 4 Basic Philosophy (Theory of Action) of the Positive Action Program & Circle You feel good about yourself when you do positive actions. Cognitive Behavior Therapy and Positive Psychology 5 Basic Philosophy (Theory of Action) of the Positive Action Program & Circle ...and You feel bad about yourself when you do negative actions. C.F. Depression 6 The Positive Action Program Targets Multiple Behaviors • By teaching that: – When you do good, you feel good – And there’s always a positive way of doing things AND • By teaching that doing positive actions helps: – Individuals, families, schools and communities develop positive self identities. 7 Limitations of Behavior-Specific Programs • Problem-specific –Usually only one behavior or one skill • Start too late –Upper elementary or middle school • Limited intensity and dose –Often only once a week for 10–20 sessions • Ecologically limited –Usually only in the classroom • Limited effect sizes –Average effect sizes in the 0.2 to 0.4 range • Effects not sustained –Few effects beyond one year, let alone into high school 8 The POSITIVE ACTION Program Components K–12 classroom curriculum over 1,200 lessons - using Teacher’s Kits (manuals and materials for each grade), classroom teachers present 15–20-minute lessons Principal’s Kits (Elementary and Secondary) a school-climate program to promote the practice and reinforcement of positive actions in the whole school population (students and staff) Counselor’s Kit used with selected individual students, small groups and families Family Kit contains prepared weekly home lessons paralleling the school program along with school parent-involvement activities Community Kit manuals and materials that align and encourage collaboration of all the environments (schools, families and community) involved in the program 9 Positive Action Focus Units (Learning Goals) • In the classroom curriculum and all other materials, the Positive Action content is taught through six focus units. Unit 1: Self-Concept: What It Is, How It’s Formed, and Why It’s Important (Philosophy & Circle) Unit 2: Physical and Intellectual Positive Actions for a Healthy Body and Mind (includes motivation to learn) Unit 3: Social/Emotional Positive Actions for Managing Yourself Responsibly Unit 4: Social/Emotional Positive Actions for Getting Along with Others by Treating Them the Way You Like to Be Treated (Social-Emotional Skills & Character) Unit 5: Social/Emotional Positive Actions for Being Honest with Yourself and Others (Mental Health) Unit 6: Social/Emotional Positive Actions for Improving Yourself Continually (Setting & Achieving Goals) 10 Logic/Theoretic Model of the Expected Effects of the Positive Action Program Program Components Climate Development, Family Kit, Teacher/Staff Training, K–12 Instruction Curriculum, Drug Education Supplements, Community Kit, Counseling Kit Immediate Outcomes * Improved relationships among school administrators, teachers, parents & community. * Improved classroom management. * Increased involvement of school with parents & community. PA Unit 1. 2. 3. 4. 5. 6. Improved character/self-concept Learning/Study skills Self-Management Interpersonal/social skills Self-honesty, responsibility Goal setting, future orientation Expected Effects Attitudes Toward Behaviors, Social Normative Beliefs, SelfEfficacy Improved Social and Character Development Improved Learning Environment Expected Impact Fewer Disciplinary Problems; Reduced Substance Use; Less Violence Improved School Attendance, Grades and Test Scores 11 Mapping of Positive Action Components Onto The Theory of Triadic Influence CLASSROOM SCHOOL FAMILY COMMUNITY Positive Action Program Components and Lessons (Exist in each Component) IntraPersonal (Individual) Social Context/ Situations Genetics Biology Personality Self Concept Self Management Time, Energy, Talents, Money, Anger Social & Emotional Health Sense of Self Social Competence SelfSocial Determination Skills Environment Family School Neighborhood Peers Feelings Empathy How to Treat Others Communica -tion Skills Conflict Resolution Self-Efficacy Socio-Cultural Social Others’ Bonding/ Behavior & Attachment Approval Others’ Expectations Desire to Please Social Normative Beliefs Mass Media Regulations Religion Economy Positive Role Models How You Like To Be Treated General Values Information Environment Evaluation of Expected Outcomes Consequences Attitudes Toward the Behavior Health & Drug Info Thinking Skills Creativity DecisionMaking ProblemSolving INTENTIONS/DECISIONS BEHAVIOR Experiences from Behavior 12 II. Prior Evaluations of PA • Early Studies of the PA Program’s Effects on Self-Concept • Effects of the Positive Action Program on Achievement and Discipline: Two Matched-Control Comparisons in Hawai’i and Nevada Elementary Schools (Flay, Allred & Ordway, 2001) • Long-term Effects of the Positive Action Program (Flay & Allred, 2003) • Utah State Annual Report of PA Family Classes Outcomes • Randomized Trial of PA in Hawaii Elementary School • Randomized Trial in Chicago Public Schools 13 In a prior quasi-experimental study, relative to elementary schools without PA, matched schools with PA reported: • Major reductions in problem behaviors • Up to 85% reductions in violence • Up to 71% reductions in substance use • Up to 90% fewer general disciplinary actions (with effects sometimes being larger in schools with higher levels of student poverty) • Up to 80% fewer suspensions • Up to 94% reductions in criminal bookings • Major improvements in school performance • • • • Up to 60% reductions of absenteeism, Up to 13% lower rates of chronic absenteeism. Up to 100% improvements on standardized achievement scores, Many schools report moving from one of the lowest scoring to one of the highest scoring in their district or state • Publication: Flay, Allred & Ordway, Am J. Health Behavior, 2001 • The matched-control study was replicated by Flay & Allred (Prevention Science, 2003) with the addition of pretest matching data. 14 Effects of 3 or 4 years of PA on 5th grade student self-reports of substance use: Hawaii randomized trial 2005 & 2006 All differences significant at p < .05 20.0 Control 18.0 PA 16.0 14.0 Percent 12.0 10.0 8.0 6.0 4.0 2.0 0.0 Smoked a Drank alcohol cigarette (or used (beer, wine or liquor)* some other form of tobacco)* Gotten drunk on alcohol* Used an illegal drug like marijuana or cocaine* Gotten high on drugs * 15 Effects of 3 or 4 years of PA on student self-reports of violence and sexual behaviors: Hawaii randomized trial 2005 & 2006 -- All differences significant at p < .05 12.0 10.0 Control PA Percent 8.0 6.0 4.0 2.0 0.0 Carried a Threatened Cut or knife or razor to to cut or stab stabbed use to hurt someone* someone on someone* purpose to hurt them* Carried a gun* Shot at someone* Voluntary sex with someone of the opposite sex* 16 Effects of 3 or 4 years of PA on teacher reports of 5th grade student behaviors: Hawaii randomized trial 2005 & 2006 Differences significant at p < .05** or P < .10* 35.0 30.0 Control PA Percent 25.0 20.0 15.0 10.0 5.0 0.0 Smokes (or may smoke) cigarettes (or other forms of tobacco) Drinks or Uses may drink drugs like alcohol marijuana or cocaine** Gets into Physically a lot of hurts others* Threatens fights** others** Destroys things belonging to others** 17 Effects of PA on Student Absenteeism Hawaii Randomized Trial (4 yrs of PA) (Multiple baseline - 1997-2001) 12.5 12 Average Days Absent 11.5 11 10.5 CONTROL Means PA Means 10 STATE Standard 9.5 9 Positive Action Program 8.5 1997 1998 1999 2000 2001 2002 2003 2004 2005* 2006* 2007* Year (* indicates significant differences between conditions) 18 79 Effect of PA on SAT (Stanford 9) Reading Scores Hawaii Randomized Trial 2000-2006 (4 yrs of PA) (No testing in 2001 due to a teacher strike) % Scoring Average or Better CONTROL Means 78 PA Means 77 STATE 76 75 74 73 72 71 70 Positive Action Program 69 2000 2002 2003 2004 2005 2006* 2007* Year (* indicate w here conditions significantly different) 19 III. The Chicago Randomized Trial of Positive Action • UIC/OSU school-based randomized trial • One site of a 7-site study funded by the IES of the US DoED • Schools randomly assigned from 7 matched pairs – Matched on school-level demographic variables, achievement and disciplinary referrals • Data collected from students and their teachers and parents, and school leadership (principal and PA Coordinator) – Beginning and end of grades 3 & 4 and end of grade 5 • Program Implementation and Outcomes – – – – – – Teacher amount and integrity of program delivery Parent involvement with program and schools Teacher classroom management Student character and behavior Student school attendance and achievement test scores Also school records data on achievement, attendance and disciplinary referrals • Last wave of data collection Spring 2007 20 Research Design • 7 matched pairs of schools randomly assigned to 2 conditions: – Early Starters: Start the program in the 2004-05 school year OR – Late Starters: Start the program 3 years later (2007-08 school year) • Schools eligible for inclusion were: – Community-based (No Magnet or Charter schools), Have not already used Positive Action, Not already participating in related projects (Project Northland, All Stars), – Enrollment > 50 and < 140 students per grade – Mobility rate no greater than 40% – More than 50% of students receive free or reduced price lunch – Less than 50% of students met achievement criteria on the ISAT • Schools matched into pairs before randomization: – Achievement scores, School size – enrollment, Ethnic Distribution, % Mobility, % Free/Reduced Lunch, Attendance and Truancy Rates, % Parent Involvement, % Teachers not meeting minimal requirements, CPS Region, Community crime statistics 21 Research Design 2 • Follow one cohort of students from the beginning of grade 3 to the end of grade 5 • Signed parental consent – 98.3% returned, 79.7% Yes • Surveys of students, teachers and parents at beginning and end of grades 3 and 4, and end of grade 5. • Teachers complete behavior rating scales on students in their class at beginning and end of grades 3 and 4, and at the end of grade 5. • Some of the measures are multi-site – collected by a national contractor (Mathematica Policy Research – MPR) at each of the 7 sites – and not available to PIs until about a year later • Some of the measures are site-specific – specific to our evaluation of the Positive Action program – collected by UIC research staff 22 Assessment of Program Impact • Multi-site – data collected with student, parent, and teacher surveys used by all SACD sites • Local site – data collected with student survey used by Chicago SACD site only • Data collected only on cohort of students that was in 3rd grade at start of study (Fall ’04) • Outcomes assessed at 5 time points – Year 1: Fall ’04 (T1), Spring ’05 (T2) – Year 2: Fall ’05 (T3), Spring ’06 (T4) – Year 3: Spring ’07 (T5) 23 Outcome Measures - Multi-site • Student (S), Parent (P), and Teacher (T) Surveys – Behavioral – BASC Aggression and Conduct Problem (P/T); Frequency of Delinquent Behavior (S); Social Competence— Prosocial Behavior (T) – Emotional – Empathy (S); Emotional Regulation (P, T) – Academic – Engagement vs. Disaffection for Learning (S); Academic Competence (T) – Not utilized for impact analyses in this presentation pending release of report of findings from the multi-site study • School Records – student grades, standardized test scores, attendance, discipline—some of these data are not yet available 24 Outcome Measures – Local • • • • Behavioral – Social Problem Solving (.44) – Aggressive (.50), Competence (.58) – Positive Health Behavior – Hygiene (.48), Food and Exercise (.43) Emotional – Affect - Positive (.68), Negative (.74) – Self-Esteem – Peer (.71), School (.68), Family (.75), Appearance (.66), Sports (.72), Global (.61) – Self-Esteem Formation – Positive (.74), Negative (.75) – Self-Esteem Process – Motivation (.73), Control (.34) – Life Satisfaction (.70) Academic – Student report of grades Mediators – Character and Social Development Scales (CASD) – Belief in the Moral Order – Positive values (.59), Negative values (.59) – Attachment – School (.74), Teacher (.68), Parents (.54), Friends (.71) – Rewards for Prosocial Behavior – Parents (.72), Teachers (.80) – Conventional Friends - Good friends (.57), Bad friends (.76) – Neighborhood Youth Inventory (.69) Note: Coefficient alphas in parentheses 25 Implementation Measures Ratings of Implementation Fidelity Consultation Notes Weekly Implementation Reports Teacher Interview PA Committee Roster Form PA Budget Unit Implementation Reports End-of-Year Survey PA Budget and Expenditures Report End-of-Year Implementation Survey PA Mid Year and End-ofYear Student Survey PA Research Team Classroom Teachers Principal Students PA Expenditures Report PA School Coordinators PA Committee Meeting Report Form PA School Activity Form End-of-Year Implementation Survey PA Committee Members Any School Staff Person PA Committee Meeting Attendance Log and Report Form PA Program Observation Form 26 Results 1: Program Implementation by Year Program Benchmarks: 2004-05 2005-06 2006-07 Teaching at least 4 lessons per week 61% 68% 66% Distribute 5 + Word of the Week Cards/week 22% 35% 35% Distribute 5 + PA Stickers per week Read 5 + notes from ICU Box Play PA Music 2 + days per week Spoke with 2 + Parents about PA per week 30% 42% 19% 25% 39% 48% 35% 45% 41% 49% 33% 43% Identified Academic Learning Standards in PA 81% 93% 91% Attended a PA assembly each unit 17% 47% 47% Teacher believes s/he delivered program quite well or very well 64% 71% 71% Teacher believes continued use of PA is very or extremely likely to improve student character Teacher believes continued use of PA is very or extremely likely to improve student academics Teacher Response Rates 61% 63% 68% 49% 53% 58% 53% 75% 76% 27 Lots of SACD activity in control schools Teacher-reported SACD activities in treatment and control schools by year Outcome % of teachers engaging in any SACD activity Teacher report of N of hours/week engaging in any SACD activity Spring 3rd grade Spring 4th grade Spring 5th grade Tx C ES Tx C Tx C ES 91.8 82.6 0.14 91.1 94.8 -0.09 96.4 88.8 0.13 76.3 35.5 0.49 84.1 76.6 80.3 76.4 0.05 ES 0.1 Data from IES/MPR Draft report, Appendix C, Table 5-13 28 Results 2: Impact Analyses • SAS PROC Mixed - Standard errors adjusted for clustering of students within schools – Model included school-level random intercept – Classroom level error term was excluded – Model did not include the pair-wise matching of schools – Covariates – • Student gender, student age, parent ethnicity, parent education level, # of people in household • T1 score on the criterion measure • T1 Negative school orientation – Student Report • T1 Normative beliefs about aggression – Student Report • T1 Victimization at school – Student Report • T1 Confusion, Hubbub and Order – Parent Report • T1 ADHD Symptomology – Teacher Report 29 Impact Analyses Results • At baseline, ES and LS demonstrated equivalence on all school-level variables (next slide) and all but four outcomes (positive values, parent and teacher rewards for prosocial behavior, bad friends: ES < LS). • Results indicate multiple emerging (.05 < p < .10) and significant (p < .05, two-tailed) program effects 30 Comparability of Matched Sets of Schools Chicago Study (No significant differences) 100 ES (Program) 90 LS (Control) 80 60 50 40 30 20 10 es C rim s he r ip ity Q ua l Pa r % Te ac at ic l tP ee n En ro y ilit M ob % er ty Po v y % an c Tr u % At tt en d e % hi ev Ac an % As i p H is % ck Bl a % W hi te s % ne s ea di Pa ir 0 R Percent 70 31 Figure A: Effects of 3 years of PA (Grade 3 through grade 5) on Local Character Measures: % Change by Outcome BASC Anxiety subscale 6.4 Character - Honesty 4.5 Character - Rules 2.9 Character - Self-Control 7.5 Character - Peer 3.8 Competent Problem Solving Style 11.5 Social Problem Solving Full Scale 9.6 Peer Affiliation – Bad Friends 15.2 Peer Affiliation – Good Friends 4.8 0 2 4 6 8 10 % Improvement 12 14 16 Latent Growth Curve Analysis and Effect Sizes for Local-site Student Level Outcome Scales: Summary of Noteworthy Findings Scale T vs. C Wave 1-4 % change Cohen ES T vs. C Wave 1-5 % change Cohen ES Social Problem Solving Scale Full Scale .04 11.3 .21* .04 9.6 .19*** Aggressive Problem Solving -.02 -8.4 -.15* -.01 1.0 -.10*** Competence Problem Solving Style .02 8.8 .14*** .02 11.5 .20*** Peer Group Affiliation–Good Friends .05 5.5 .23** .04 4.8 .21** Peer Group Affiliation–Bad Friends -.10 -14.1 -.28*** -.07 -15.2 CASD–Pro-social Behavior (Peer) .05 4.8 .25*** .04 3.8 .21** CASD–Honesty .06 4.8 .25*** .05 4.5 .24*** CASD–Self–Control .05 3.8 .17** .06 7.5 .30*** CASD–PA–Rules .05 3.4 .22*** .03 2.9 .18** -.31*** Notes: All significance levels one-tailed; *** p < .05, ** p < .10. Condition was coded as 0 = Control; 1 = PA. T vs. C denotes estimated difference (unstandardized) in slope between Treatment and Control schools. % change = (a – b) x 100, where by a = the treatment wave 4 or 5 mean divided by the control wave 4 or 5 mean and b = the treatment wave 1 mean divided by the control wave 1 mean. Cohen ES = a – b, where a = (treatment wave 4 or 5 mean – control wave 4 or 5 mean) / pooled standard deviation and b = (treatment wave 1 mean – control wave 1 mean) / pooled standard deviation. For all scales, higher scores indicate more of the construct. 33 Figure B: Effects of 3 years of PA on Behavior % Improvement Serious violence 12.5 Alcohol use 21.7 Smoking 28.2 0 5 10 15 20 25 30 % Improvement P.S. Similar reductions in the multi-site measure of Problem Behavior 34 % Ever Used Sustances and % Reduction by Condition: Grade 5 Chicago Randomized Trial 60.0% C PA % reduction 52.2% 50.0% 39.9% Percent 40.0% 36.2% 32.1% 28.2% 30.0% 28.3% 21.7% 19.5% 20.0% 13.2% 10.0% 9.4% 10.7% 5.1% 0.0% Ever smoked/usd Ever drank alcohol Ever gotten drunk tobacco Ever used tobacco, alcohol or drugs 35 % Ever Engage in Violence or Gang Activties and % Reduction By Condition: Grade 5 Chicago Randomized Trial 45.0% 39.4% 40.0% C PA % reduction 37.3% 35.0% 32.7% Percent 30.0% 27.7% 25.0% 22.2% 10.0% 21.3% 20.5% 18.2% 18.2% 20.0% 15.0% 26.7% 12.5% 12.4% 7.5% 4.9% 3.6% 5.0% 0.0% Ever threateed to cur ot stab someone Ever cut or stabbed someone Ever been Ever hung out Ever engaged asked to be in with gang in a gang members violence/gang activities 36 Effects on School-Level reports of misconducts and suspensions Average N of misconducts per 100 students by year and condition (80% reduction at 2006-07) 45.00 40.00 PA Average N of suspensions per 100 students by year and condition (77% reduction at 2006-07) 35.00 C PA C 30.00 35.00 25.00 30.00 25.00 20.00 20.00 15.00 15.00 10.00 10.00 5.00 5.00 0.00 0.00 2003-04 2004-05 2005-06 Year 2006-07 2003-04 2004-05 2005-06 2006-07 Year In ANCOVA models predicting year 4 differences from year 1 levels and condition, differences at year 4 are marginally significant for misconducts (p = .054) and significant for suspensions (p = .037) using one-tailed tests. 37 Program Effects on Standardized Test Scores CPS Standardized Achievement (% meeting or exceeding expectations on ISAT) by Condition: ES (PA) group shows 9% improvement [(60-55)/55] for reading, 4% for math, 6.5% for composite. Difference is statistically significant for reading. 66 ES(PA) LS(Co) 64 62 60 58 56 54 52 50 School Wide 2007 Reading School Wide 2007 Math School Wide 2007 Composite Program Effects on Standardized Test Scores Standardized Achievement Reading Scores by Condition ES(PA) group shows a 49% improvement compared to 30% for the ES(Co) group, or a 16.6% relative improvement [(%change in PA - %change in Co)/%Post Co] 65 % Meeting or Exceeding Expectations ES(PA) LS(Co) 60 55 50 45 40 35 2005 2006 2007 Year 39 V. Conclusions • Implementing school-wide character education programs to address a wide range of outcomes is challenging – Limited resources of urban school systems – NCLB • Evaluation of school-based character/social development programs is complicated by control schools implementing similar programs under “business as usual” conditions • Multiple program effects were obtained after 3 years of programming • School-level reports of misconducts and suspensions strengthen the robustness of the findings • Time trends in outcomes suggest increasing effects over time • School-wide social and character development education can be effective at: – decreasing multiple negative behaviors and – increasing multiple positive behaviors 40 Future Research • Investigate potential differential impacts of PA based on student gender, child risk level, etc. • Investigate whether schools with different levels in the quality of implementation yield different “impacts” • Validate SACD scales with observed student behavior • Examine impact of PA as student cohort progresses into upper elementary grades (grades 6-8) – Critical transitional period within emotional, behavioral, and academic domains 41 Future Work/Needs – The bigger picture • Larger scale trials – ICCs for attitudes (.03-.1) and behavior (.01-.05) are generally smaller than for achievement (.15-.2) – Still need Ns of 10-20 per condition rather than 7 • Improved measures of integrity and dosage delivered and received – Teacher, student and observer reports – Contractual reporting systems? • Longer term follow-ups – Effects take several years to even start emerging – Prior work suggests important long-term effects are possible • Methods of analysis to accommodate differential implementation – Propensity scoring, CACE, instrumental variable 42 Acknowledgment The findings reported here are based on research conducted by the authors as part of the Social and Character Development (SACD) research program funded by the Institute of Education Sciences (IES), U.S. Department of Education through Grant # R215S020218 to the University of Illinois at Chicago (2003-2005) and Oregon State University (2005-2008). The SACD Consortium consists of representatives from IES, the Centers for Disease Control and Prevention (CDC), and the national evaluation contractor, Mathematica Policy Research, Inc. (MPR), and each grantee site participating in the evaluation. The content of this presentation does not necessarily reflect the views or policies of the SACD Consortium members including IES, CDC, and MPR, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government. Correspondence concerning this presentation should be addressed to Brian R. Flay, D.Phil., Principle Investigator, Department of Public Health, 254 Waldo Hall, Oregon State University, Corvallis, OR 97330, brian.flay@oregonstate.edu. 43 SACD disclaimer statement: The Social and Character Development (SACD) research program funded by the Institute of Education Sciences (IES), U.S. Department of Education includes a national evaluation study conducted by Mathematica Policy Research (MPR), and complementary research studies conducted by each grantee. The findings reported here are based on the complementary research activities carried out by Brian Flay, Oregon State University, and David L. DuBois, University of Illinois at Chicago, under the SACD program. These findings may differ from the results reported for the SACD national evaluation study. The findings presented in this conference presentation are based on a smaller sample size of children, classrooms, and teachers, utilized a different set of outcome measures, and sought to answer complementary research questions. The content of this presentation does not necessarily reflect the views or policies of the SACD Consortium including IES, CDC, and MPR, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Department of Education. 44 Have a Positive Action Day! 45