Summary of Packard-CSU Ed.D. Dissertation Fellowship Report Title: The Impact of Collaboration in After-School Programs on Student Achievement and School Attendance Ed.D. Candidate: Matilda T. Soria, CSU Fresno Research Question(s): The current study investigated three research questions: 1. Is there a significant difference in student outcomes between school-community collaborative after-school programs versus after-school programs that are operated primarily by the school site? 2. Does degree of collaboration between schools and community-based organizations predict student outcomes? 3. How do after-school program personnel perceive collaboration with community-based organizations? Conceptual Framework and/or Guiding Purpose of the Study: Schools continue to struggle with ways to address the achievement gap for low-income students, and some researchers estimate that by 2030 language minority students are expected to comprise 40% of the school-aged population in the United States (Thomas & Collier, 2002). Extended learning programs, or programs that are provided in addition to instruction during the school day, are viewed as a promising strategy to close this achievement gap between poor and affluent students and between White students and students of color. Typically these extended learning programs operate during the after-school hours and are entitled after-school programs. Over the past 10 years, the level of federal and state funding for after-school programming has increased significantly; thus, there has been a steady increase of after-school programs across the country. As of 2010, the current funding level for California Department of Education’s (CDE) After School Education and Safety (ASES) Program is $550 million (CDE, 2010). Although each ASES funded after-school program is unique, there is a common model of which all programs subscribe. The ASES program must be aligned with, and not be a repeat of, the content of regular school day and other extended learning opportunities. Further, the program must operate for three hours per day, five days per week, and must provide two hours of academic support and one hour of enrichment. The variable that oftentimes differs among ASES after-school programs is the level of collaboration with partners outside the school site, including community-based organizations. Although states give priority to ASES grant applications that are jointly submitted by a local educational agency and a community-based organization or other public or private entity, the level of collaboration has not been specifically defined. Although ASES grantees are encouraged to establish collaborative partnerships to deliver after school programming, the effectiveness of this collaborative model has not been investigated. Even though theoretical research, in general, exemplifies the benefit of collaboration on student success, school reform, and neighborhood revitalization, there is very little empirical evidence. Relevant Theoretical and Empirical Literature: Theoretical Framework From birth, the home, school, and community concurrently affect children’s growth and development 1 Summary of Packard-CSU Ed.D. Dissertation Fellowship Report (Wasik & Karweit, 1994). The sustained importance of these contexts and value of collaboration between these cross-sector entities through each stage of a child’s development is described by a number of related theoretical frameworks. These include the social network paradigm, the theory of social capital, theory of overlapping spheres of influence, and community field theory. The social network paradigm emphasizes the importance of significant networks that provide support and resources such as information and money (Blau, 1964; Barnes, 1972; Leinhart, 1977). Social networks promote greater psychological and personal well-being in that they provide a buffer against negative stresses (D’Abbs, 1982). In addition, positive educational outcomes have been found among children with well-developed social networks versus children without them (Coates, 1987). Social network theorists recommend that schools should strengthen support systems for youth by developing tutoring, mentoring, and coaching programs for poor and minority youth led by adults in the school, family, and community (Clark, 1991). Coleman (1987) refers to social networks as an essential component of social capital. Social capital, gained via social networks, is the informational, attitudinal, and behavioral norms and skills that individuals can spend or invest to improve their chances for success in l institutions, such as schools. Coleman, Hoffer, and Kilgore (1982) report that students in Catholic and other private schools achieve a higher level in mathematics and verbal skills than students of public schools mainly because of the alignment of messages, expectations, and norms from the family, church, and school; in other words, the collaboration between these entities provides students in private and/or Catholic schools with the social capital necessary for school success. Community field theory provides an important framework for understanding the role of social capital in educational achievement (Wilkinson, 1991). From this perspective, community social capital develops from community members’ action to improve the local economy, provide human and social services, and express local cohesion and solidarity. Thus, the level of collaboration in a neighborhood or community in solving community issues increases likelihood of resolution. Epstein’s (1987) theory of overlapping spheres of influence also emphasizes the importance of collaboration between schools and communities. A central principle of this theory is that student academic success can be best achieved through cooperative or collaborative action (Epstein, 1992). Empirical Literature Although there are presently no published, empirical studies on the impact of collaboration within afterschool programs on student outcomes, several studies showcase the need for collaborative efforts within educational endeavors, including those offered during the after-school hours, particularly for those targeting low-income to poverty-level children (Anderson-Butcher & Ashton, 2004; AndersonButcher, Lawson, Bean, Flaspohler, Boone, & Kwiatkowski, 2008; Bronfenbrenner, 1979; Harvard Family Research Project, 2001; Shonkoff & Phillips, 2000; Taylor & Adelman, 1998). Although emerging evidence suggests that such partnerships are critical to the shared goal of supporting positive learning and development throughout the school-age years (Harvard Family Research Project, 2006), there are very few empirical studies that demonstrate the impact of collaborative partnerships between schools and community-based organizations on student outcomes in afterschool programs. Interprofessional collaborations have been associated with increased academic achievement and attendance (Family Care Healthy Kids Collaborative, 1994; Oppenheim, 1999; Smith, Armijo, & Stowitschek, 1997) and decreased levels of misconduct (Smith et al., 1997). Furthermore, Hatch (1998) claimed that community collaborations connected with schools have fostered improvements in attendance, overall experiences of parents, teachers, and students, and the quality of learning experiences for students. In the Massachusetts Afterschool Research Study, researchers found that after-school programs with stronger connections with school teachers and principals were found to be more successful at 2 Summary of Packard-CSU Ed.D. Dissertation Fellowship Report improving students’ homework completion, homework effort, positive behavior, and initiative. (Intercultural Center for Research in Education et al., 2005). Similarly, an evaluation of Supplemental Educational Services (SES) found that program quality suffered when there were not effective partnerships between schools and providers of SES services. Without the partnerships, the SES providers experienced challenges in aligning their academic services with in-school learning needs (U.S. Dept. of Education, 2004). Finally, a 2008 evaluation of Communities in Schools (CIS),a national dropout prevention program that connects an array of community resources to schools, reports that relative to comparison schools, CIS was found to have a positive impact on dropout, graduation, and attendance rates, as well as academic achievement (Caliber, 2008). As noted in From Neurons to Neighborhoods (Shonkoff & Phillips, 2000), “Bronfenbrenner's (1979) ecological model of child development portrays nested layers of influence on children emanating from the family out to the more amorphous realms of neighborhoods, policies, and social values” (p. 329330). Complementary Learning, a theoretical model developed by Harvard Family Research Project aligns with Bronfenbrenner’s ecological model and contends that a systemic approach that connects school and nonschool supports (including, family and community) ensure that children have essential skills to succeed in school and life (Harvard Family Research Project, 2010). Methods of Data Collection and Analysis: Research Question 1. Is there a significant difference in student outcomes between school-community collaborative Afterschool programs versus after-school programs that are operated primarily by the school site? 2. Does schoolcommunity collaboration predict student outcomes? 3. How do After-school program personnel perceive collaboration with community-based organizations? Independent Variables After-school types: Schoolcommunity collaborative; Other types of After-school programs Dependent Variables CST Math Performance Levels, CST ELA Performance Levels, and School attendance Degree of schoolcommunity collaboration CST Math Performance Levels, CST ELA Performance Levels, and School attendance N/A N/A Analysis Series of Two-way Repeated Measures ANOVAs and ChiSquare Analyses Data Collection Method Archival from California Department of Education and After-school providers Canonical Correlation Archival and Survey, assessing degree of collaboration Qualitative Semi-structured interviews and Site observations Study Findings: Research Question 1: Is there a significant difference in student outcomes between schoolcommunity collaborative after-school programs versus after-school programs that are operated primarily by the school site? 3 Summary of Packard-CSU Ed.D. Dissertation Fellowship Report A series of three two-way repeated measures ANOVAs revealed statistically significant differences in Math and English Language Arts (ELA) CST Performance Levels and school attendance for both the collaborative and school programs from the 2008-2009 school year to the 2009-2010 school year. These significant effects indicate that Math and ELA CST and school attendance were higher in school year 2009-2010 than in school year 2008-2009; in other words, students of both types of programs on average, improved from the baseline to the following year. The main effects for after-school program type were not found to be statistically significant; however, the interactions between after-school program type and school year were found to be significant for ELA CST and school attendance. These significant interactions reveal that ELA CST and school attendance across school years were different for collaborative versus school programs. Thus, the rate of improvements for students of collaborative programs was significantly higher on ELA CST over the two school years when compared with students of school programs. Moreover, the same pattern was found for school attendance; students of collaborative programs attended more school days than students of school programs over the two school years. Although the chi-square analysis is limited in that it cannot test for interaction effects, the series of 2 x 5 chi-square analyses revealed similar findings. The chi square tests for after-school program type and ELA CST Performance Levels for school year 2008-2009 and 2009-2010 were significant. Thus, performing well on the ELA CST was significantly dependent on after-school program type. During the baseline year, there were a greater percentage of students scoring at Basic, Proficient, and Advanced among school programs versus collaborative programs. Conversely, there were a greater percentage of students scoring at Far Below Basic and Below Basic among collaborative programs versus school programs during the baseline year. During the following school year, however, the reverse trend was found; there was a greater percentage of students scoring at Proficient and Advanced among the collaborative programs versus the school programs, while a greater percentage of students scoring at Far Below Basic, Below Basic, and Basic among the school programs. These results indicate that although students of school programs begin at an advantage in terms of Performance Level on the ELA CST, students of collaborative programs performed at a higher Performance Level when evaluating the results over the two year period. The same trend was found for math CST. The chi square test for after-school program type and math CST Performance Levels for school year 2008-2009 was significant; thus, performing well on the math CST in the baseline year was significantly dependent on after-school program type. The chi square test for 2009-2010, however, was not significant. These findings could mean that the addition of a collaborative partner provided an added benefit to the after-school program. The findings support previous research on the impact of collaboration on student outcomes (Caliber, 2008; Intercultural Center for Research in Education et al., 2005; and U.S. Department of Education, 2004). Research Question 2: Does degree of collaboration between schools and community-based organizations predict student outcomes? Correlation analyses were conducted among the dimensions of degree of collaboration, school attendance, and CST Performance Levels for Math and ELA. Results of the correlation analyses revealed significant and positive correlations between ELA CST and the governance, administration, autonomy, mutuality, and norms dimensions of degree of collaboration. The correlations between the dimensions of degree of collaboration and Math CST and school attendance were significant and negative; however, the correlations were weak (less than .50). A correlation greater than 0.8 is generally described as strong, whereas a correlation less than 0.5 is generally described as weak. The significant and negative correlations may have resulted due to perceptions about the collaborative partnership shared between the school site and the community-based organization. Survey participants included both after-school leaders who directly managed an after-school program, in addition to leaders who managed several after-school programs. Those leaders who are on-site may have a more realistic perception of the collaboration than those that are not off-site. A trend found within the survey data and substantiated with the qualitative interviews and site observations, was that site after-school leaders reported a lower degree of collaboration than off-site after-school leaders. 4 Summary of Packard-CSU Ed.D. Dissertation Fellowship Report In addition, a series of two canonical correlation analyses was run with student performance variables (school attendance and Math and ELA CST Performance Levels) on one side of the equation and the five dimensions of degree of collaboration on the other side of the equation. The full model across the three functions for 2008-2009 was found to be statistically significant with Wilk’s Lambda = .395 criterion, (F=3.47, df=15, 130.15, p<.01). Since Wilk’s Lambda represents the variance unexplained by the model, 1- Wilk’s Lambda yields the full model effect size in an r2 metric. Thus, for the set of four canonical functions, the r2 type effect size was .605 (1 – Wilk’s or 1 - .395), which indicates that the full model explained about 61% of the variance shared between the variable sets. The full model across the three functions for 2009-2010 was found to be statistically significant using the Wilk’s Lambda = .356 criterion, (F=3.94, df=15, 130.15, p<.01). For the set of three canonical functions, the r2 type effect size was .644, which indicates that the full model explained about 64% of the variance shared between the variable sets. The results of these findings indicate that degree of collaboration between school sites and community-based organizations in after-school programs may assist in predicting student performance, specifically Math and ELA CST Performance Levels and school attendance. Research Question 3: How do after-school program personnel perceive collaboration with community-based organizations? Interviews and site observations were conducted among six lead staff of after-school programs in Northern and Central California. Data were analyzed using a grounded theory approach; thus, initial and more in-depth inductive coding processes transpired resulting in six overarching themes: accountability, proximity, engagement, relationships, value of collaboration at leadership level, and outsider. Based upon interview and observation data, barriers of building and sustaining collaborative partnerships between schools and community-based programs included negative past experiences (value of collaboration at leadership level) and lack of community-based organizations nearby the school site (proximity). Benefits cited by interview participants included the ability to leverage resources and having a similar mission and values (accountability); and the ability of community-based organizations to engage students and offer programming that complements, but is different from the school day (engagement). Interview participants reported that consistency (accountability), communication, and time (relationships) were essential in building and sustaining an effective collaboration. The theme “outsider” continually surfaced across interviews with all of the communitybased partners (N=3), in addition to the district administrator participant who leads collaborative afterschool programs. These interview participants reported that because they are outside of the school system, they are heavily scrutinized, regardless of the degree of collaboration between their organization and the schools site. Thus, community-based organizations felt the pressure to perform above and beyond expectations. In addition, interview participants felt that the motivation to partner with school sites is to bring about positive changes in students lives and regardless of funding would continue to serve the school partner and students; thus, the mission and values of the organization supersede degree of collaboration and/or perceived barriers associated with collaboration. The qualitative results perhaps explain the negative relationships found when conducting correlational analyses, in addition to the canonical correlation results. Conclusions and Recommendations: Implications for Practice The results of this current study can be used to impact policy and practice in the field of after-school programming. Specifically, results of this study support prior studies on the impact of after-school programs on student outcomes, particularly student achievement and attendance. After-school programs have been found to impact math and reading test scores among elementary school students 5 Summary of Packard-CSU Ed.D. Dissertation Fellowship Report (Klein & Bolus, 2002); school grades (Welsh, Russell, Williams, Reisner, & White, 2002; Vandell, Reisner, & Pierce, 2007); and school attendance (Espino, Fabiano, & Pearson, 2004). Moreover, results support the California Department of Education’s conditional funding requirement of encouraging collaboration between schools and community-based organizations in the operation of after-school programs; results of this current study show that collaborative partnerships with community-based organizations provide an added benefit and value for after-school programs. In addition, results support theoretical evidence of the impact of collaboration. According to the Office of Juvenile Justice and Delinquency Program (1996), quality youth programming is dependent upon cross-sector collaborative planning and implementation. Although quantitative results show the positive impact of collaboration, qualitative results reveal barriers in developing and sustaining effective collaborative partnerships. These results support Bryson, Crosby, and Stone’s (2006) framework for understanding cross-sector collaboration model, which demonstrates external and internal pressures on cross-sector collaborative partnerships. Although internal barriers can be overcome, external barriers may be outside of the control of collaborative partners. These barriers may impact perception of the strength of the collaboration between partners. After-school programs in the study sample receive government funding and as a result may experience external pressures outside of their control, including changes in program requirements, evaluation and reporting, and other funding regulations. Thus, regardless of the degree of collaboration, these external pressures may impact perception. This may explain the negative correlational findings between the dimensions of degree of collaboration and CST Performance Levels and school attendance. Results from this study may also serve to assist after-school practitioners in developing and sustaining effective collaborative partnerships. Specifically the researcher will collaborate with regional and statewide networks to generate a written publication coupled with a training protocol for after-school providers. The manual will highlight barriers and best-practices for developing and sustaining evidence-based collaborative partnerships. Implications for Future Research The purpose of this current study was exploratory; thus, future research is necessary to substantiate findings. Due to limitations with the type of data received from after-school programs, specifically CST Performance Levels, the incorporation of CST raw scores is highly recommended. Although the data across grade levels for raw scores of CSTs cannot be combined to show overall effects, higher level, or parametric tests can be utilized with greater confidence. Random selection of ASES funded after-school programs and the inclusion of a comparison control group of students that are not enrolled in after-school programs would assist in the generalization of results to a broader population of after-school providers. Also recommended is the further exploration of barriers to develop and sustain collaborations between schools and community-based organizations in after-school programs and methods of overcoming these barriers. Exploration of barriers can be investigated via additional interviews with after-school leaders across programs in different regions of California. Finally, the researcher recommends the analysis of additional variables that were not included in this current study. Additional variables of interest may include: performance level of the partner school sites, such as the Academic Performance Index; quality of the organizational partner, including the assessments of such dimensions as annual operating budget and demonstrated effectiveness; and types of programs provided during the after-school program, such as a strong English Language Arts or Mathematics focus. Additional Description: 6 Summary of Packard-CSU Ed.D. Dissertation Fellowship Report Difficult economic times oftentimes serve as a catalyst for innovative solutions to community and educational issues. A present, significant issue is the increasing number of low-income to poverty-level families, in addition to the continual widening of the achievement gap for children of this economic bracket. As budget cuts continue to plague our local educational institutions, the ability to effectively collaborate and leverage resources with cross-sector partners lessons the burden of budget deficit. In addition, effective collaboration creates opportunities for alignment of services and reduction of duplication. Most importantly, effective collaboration provides for greater accountability (Agranoff & McGuire, 2003; Goldsmith & Eggers, 2004; Kickert, Klijn, & Koppenjan, 1997; Mandell, 2001; Rethemeyer, 2005). The results of this current study serve as empirical evidence of the impact of collaboration in afterschool programs on student achievement and school attendance. Further, results may be used to support collaborative after-school programs as an effective strategy for reducing the achievement gap for low-income to poverty-level students in policy and practice. Practical implications include the development of a guide for after-school practitioners to assist in developing effective collaborative partnerships between schools and community-based organizations. Selected References: Agranoff, R., & McGuire, M. (1998). Multinetwork Management: Collaboration and the Hollow State in Local Economic Development. Journal of Public Administration Research and Theory, 8 (1), 67-91. Anderson-Butcher, D., & Ashton, D. (2004). Innovative models of collaboration to serve children, youths, families, and communities. National Association of Social Workers, Inc., 39-53. Anderson-Butcher, D., Lawson, H. A., Bean, J., Flaspohler, P., Boone, B., & Kwiatkowski, A. (2008). Community collaboration to improve schools: Introducing a new model from Ohio. Children and Schools, 30(3), 161-172. Barnes, J. A. (1972). Social networks. Reading, Mass: Addison-Wesley Pub. Co. Blau, P. M. (1964). Exchange and Power in Social Life. New York: Wiley. Brofenbrenner, U. (1979). Contexts of child rearing: Problems and prospects. American Psychologist, 34(10), 844-850. Bryson, J. M., Crosby, B. C., & Stone, M. M. (2006). The design and implementation of cross-sector collaborations: Propositions from the literature. Public Administration Review, 44-55. Caliber (2008). Communities in Schools national evaluation school-level report: Summary findings. Fairfax, VA: Author. California Department of Education (2010). After School Education and Safety Program. Retrieved from http://www.cde.ca.gov/ls/ba/as/. Clark, M. L. (1991). Social identity, peer relations, and academic competence of African American adolescents. Education and Urban Society, 24(1), 41-52. Coleman, J. S. (1987). Families and schools. Educational Researcher, 16(6), 32-38. Coleman, J. S., Hoffer, T., & Kilgore, S. (1982). High School Achievement: Public, Catholic, and Private Schools Compared. New York: Basic Books. D’Abbs, P. (1982). Social Support Networks: A Critical Review of Models and Findings. Melbourne, Australia: Institute of Family Studies. Epstein, J. L. (1987). Toward a theory of family-school connections: Teacher practices and parent involvement. In K. Hurrelmann, F Kaufmann, & F. Losel (Eds). Social Intervention: Potential and constraints. pp. 121-136. New York: DeGruyster. Epstein, J. L. (1992). School and family partnerships. In M. Alkin (Ed.). Encyclopedia of educational 7 Summary of Packard-CSU Ed.D. Dissertation Fellowship Report research, 6th edition. pp. 1139-1151. New York: MacMillan. Espino, J., Fabiano, L., & Pearson, L. M. (2004). Citizen schools: Evidence from two Student cohorts on the use of community resources to promote youth development. Washington, D.C.: Policy Studies Associates. Family Care Healthy Kids Collaborative (1994). Interviews with key participants. San Fernando, CA: Author. Goldsmith, S., & Eggers, W. D (2004). Governing by Network: The New Shape of the Public Sector. Washington, DC: Brookings Institution. Harvard Family Research Project (2001). Retrieved from http://www.gse.harvard.edu/~hfrp/projects/afterschool. Harvard Family Research Project (2006). Building and evaluating out-of-school time connections. The Evaluation Exchange, Vol. XII, No. 1 and 2. Cambridge, MA: Author. Harvard Family Research Project (2010). Partnerships for learning: Promise practices in integrating school and out-of-school time program supports. Retrieved from Harvard Family Research Project website: http://www.hfrp.org/publications-resources/browse-our-publications/partnershipsfor-learning-promising-practices-in-integrating-school-and-out-of-school-time-program-supports. Hatch, T. (1998). How community action contributes to achievement. Educational Leadership, 55, 16-19. Intercultural Center for Research in Education, & National Institute on Out-of-School Time (2005). Pathways to success for youth: What works in afterschool: A report of the Massachusetts Afterschool Research Study (MARS). Boston, MA: United Way of Massachusetts Bay. Kickert, W., Klijn, E., & Koppenjan, J. (1997). Managing Complex Networks: Strategies for the Public Sector. London: Sage Publications. Klein, S. P., & Bolus, R. (2002). Improvements of math and reading scores of students who did and did not participation in the Foundations After School Enrichment Program during the 2001-2002 school year. Santa Monica, CA: Gansk. Leinhart, S. (Ed.). Social networks: A developing paradigm. New York: Academic Press, 1977. Mandell, M.P (2001). Getting Results through Collaboration: Networks and Network Structures for Public Policy and Management. Westport, CT: Quorum Books. Office of Juvenile Justice and Delinquency Prevention (1996). Combating violence and delinquency: The national juvenile justice action plan. Retrieved from http://www.juvenilecouncil.gov/resource/jjplansm.pdf_summary.pdf. Oppenheim, M. (1999). The critical place of community development in school transformation. The story of the Vaughn Family Center and Pacoima Urban Village. Teachers Education Quarterly, 26, 135-158. Rethemeyer , R. K. (2005). Conceptualizing and Measuring Collaborative Networks. Public Administration Review 65(1), 117 – 21. Shonkoff, J. P. & Phillips, D. A. (2000). From Neurons to Neighborhoods: The Science of Early Childhood Development. Retrieved from http://www.uwbec.org/documents/ school_readiness/Neurons%20to%20Neighborhoods.pdf. Smith, A. J., Armijo, E. J., & Stowitschek, J. (1997). Current approaches of case management in schools to improve children’s readiness to learn. Journal of case management, 6, 107-115. Taylor, L. & Adelman, H. S. (1998). A policy and practice framework to guide school-community connections. Rural Special Education Quarterly, 17, 62-70. Thomas, W. P. & Collier, V. P. (2002). A national study of school effectiveness for language Minority students’ long-term academic achievement. Retrieved from 8 Summary of Packard-CSU Ed.D. Dissertation Fellowship Report http://crede.berkeley.edu/research/llaa/1.1_final.html. U.S. Department of Education (2004). Early implementation of supplemental educational services under the No Child Left Behind Act: Year one report. Washington, DC. Vandell, D. L., Reisner, E. R., & Pierce, K. M. (2007). Outcomes Linked to High-Quality Afterschool Programs: Longitudinal Findings from the Study of Promising Afterschool Programs. Unpublished report. Wasik, B.A. & Karweit, N.L. (1994). Off to a good start: Effects of birth to three interventions on early school success. In R.E. Slavin, N.L. Karweit, and B.A. Wasik (Eds.) Preventing Early School Failure: Research, Policy, and Practice. Boston, MA: Allyn and Bacon. Welsh, M. E., Russell, C. A., Williams, I., Reisner, E. R., & White, R. N. (2002). Promoting learning and school attendance through after-school programs: Student-level changes in educational performance across TASC’s first three years. Washington, DC: Policy Studies Associates. Wilkinson, K. P. 1991. The community in rural America. New York: Greenwood Press. 9