[Type text] Concept Note: Impact Evaluation of Nadie es Perfecto, a Program to Improve Parenting Skills in Chile Emanuela Galasso (DECPI), Pedro Carneiro (UCL), Miguel Cordero (MOH-Chile) Background Parenting practices are key determinants of the physical and socio-emotional health of children, and parental education interventions are potentially an important strategy employed to improve parenting practices. Rapid social and lifestyle changes, increased female labor force participation, increased hours at work for both parents, and the gradual disappearance of the extended family as a traditional source of informal support, are increasing the pressures on parenting practices of families living in the so-called emerging economies. Parents play a critical role in fostering the development of children, and this particularly important in the early years. Healthy development in a child’s earliest years sets the stage for a strong development throughout the life span (not only in terms of health outcomes, but also cognitive and noncognitive skills, as suggested in Currie, 2009). There is evidence that parenting and the home environment accounts for a significant fraction of the preschool cognitive achievements (Schady, Paxson 2007) and school test-score gaps among different socioeconomic groups (Todd, Wolpin 2007). Parents, however, do not always have access to adequate help and information to support them in making decisions that can assist in fostering healthy child development. This problem is especially important among the poor. A recent LANCET comprehensive review of strategies to promote child development (Engle et al 2007) lists a few parenting and parent-child interventions to conclude that parenting programs combined with home visits and/or that involve direct opportunities for skill building and practices with the children are more effective than providing information to parents in enhancing child development outcomes. Evidence from Bangladesh (Aboud 2007, Hamadani et al 2006) suggests that both types of programs improve maternal knowledge, but only those combined with direct demonstrations and interaction with children showed significant gains child development scores. Extending our evidence on the relative effectiveness of such programs in other settings and with varying modalities of service delivery (community based vs individualized interaction) is however of critical policy importance, as they are associated with substantially different costs and potential for scalability in low and middle income settings. Furthermore, parenting interventions are often proposed as a tool for improving child outcomes, and an example is Nobody's Perfect, the Canadian program on which Nadie es Perfecto (NEP hereafter) is based on. There is very little evidence based on rigorous impact evaluations on this topic, and it is important to document how and under what circumstances maternal knowledge translates into improved behavioral change in parenting practices and how these benefits are in turn passed on to their child. “Nobody’s Perfect” the most widely disseminated parenting program in Canada, is one example of this research-practice gap. Canada has championed the development and provision of this type of intervention. An existing combined qualitative and quantitative evaluation (Chislett and Kennet 2007; Skrypnek and Charchun 2009) found that Nobody’s Perfect contributes to improvement in a number of parental outcomes that are potentially associated with superior child outcomes. The evaluation shows a decrease in negative or punitive practices, and improved parental ability to cope with parenting stressors, problem solving ability and perceptions of social support.1 Yet, the evaluation fails to rigorously examine whether these changes in parenting practices translate into gains in terms child development outcomes. There is now a significant opportunity to address this problem as the Chilean government is committed to launch a major evaluation of Nobody’s Perfect (or Nadie Es Perfecto). The government of Chile has selected this Canadian parenting intervention within their new nationwide early childhood policy, Chile Crece Contigo and it is planning to gradually roll it out within the country. The aim of this rigorous evaluation is to understand whether parenting programs are successful in enhancing child development. We plan to allow variation in the intensity of the intervention to test how the relative cost-effectiveness of different delivery mechanisms and feed back into the design of the program during its scaling up phase. In doing so, we aim to go be beyond the results of the impact evaluation and uncover the mechanisms by which a parenting program may or may not affect child development. Description of the Intervention Nobody’s Perfect is a parenting education program that has been developed Public Health Agency of Canada. It was introduced in the early 1980s in a few locations, and since 1987 it has been offered in the whole country. It targets mainly parents with children aged 0 to 5 who live in poor and isolated conditions. It is expected that the program leads to: 1. an increase in participants’ knowledge and understanding of their children’s health, safety and behavior; 2. a positive change in the parenting skill of participants in relation to their children’s health, safety and behavior; 3. an improvement in participants’ confidence and self-image as parents; 4. an improvement in participants’ coping skills as parents; and , 5. and increase in self-help and mutual support among parents The innovative approach builds on parents’ existing knowledge and capacities through group discussion and problem-solving learning activities. Facilitators create opportunities for change through building trusting relationships with parents and creating groups characterized by mutual support. Approaches to parenting that use teaching and reasoning in disciplining children rather than punitive approaches are associated with to superior child outcomes (Dooley & Stewart, 2007; Fletcher, Walls, Cook, Madison, & Bridges, 2008; Gershoff, 2002; Lansford et al., 2005). The basic Nobody’s Perfect parenting program is composed of 6 to 8 group-based parent education sessions. The group sessions are lead by a trained facilitator. The program employs adult education strategies to enhance participation and learning. This entailed encouraging participation, building on strengths, and limited use of didactics. Participating parents identify 1 These results are in line with a meta analysis of Triple P-Positive programs (Nowak and Heinrichs, 2008), which suggest positive effects of such programs on parenting skills, child behavior problems and parental well being, with stronger effects in more intensive formats of the intervention. Yet, little is known about the role of parenting programs in enhancing child development outcomes. 2 preferred parenting topics which are then emphasized in the sessions by the facilitator. The basic topics will include: (i) positive attending to your child, (ii) use of non-physical disciplinary strategies, (iii) importance of monitoring for safety, (iv) child nutrition, (v) approaches to enhance attachment, (vi) management of basic health concerns, (vii) strategies of self care (for the parent), and (viii) creating a stimulating/learning environment. Each of the set themes has (a) clear learning objectives (key messages), (b) in-session hands-on activities, (c) set learning materials for the facilitators, and (d) home-work activities. Group discussions will be supplemented by parent education booklets that are provided to participants. Five of the booklets are a component piece of the Nobody’s Perfect parenting program that provide basic points about parenting of young children with an easy reading level and brightly illustrated. The booklets are also available on the Chile Crece Contigo website. In health centers that will not receive the program, the Child Health National Program considers non-structured parent education sessions in primary care around different topics: respiratory disease prevention, oral health, home accidents prevention, basic care for new born, and others. These topics are covered by a nurse or other primary care professionals. These sessions are of universal coverage (100% of health centers and nearly 60% of the children in the target group (DEIS data, 2009). Nobody's Perfect will be added to a selected set of centers over and above this unstructured parental education. Therefore the comparison we will do is between clinics where the Nobody's Perfect (or its intensive version) is offered on top of already existing activities (not only unstructured parental education, but also some direct stimulation of children offered already in half the clinics), and those where Nobody's Perfect is not offered and only the already existing set of activities is available. I. Primary Research Questions The main research questions are the following: Does a structured parenting program promote improvement in knowledge and behavioral change in parenting practices? Does the intervention improve the mental health of the mother? Does it translate in reduced parental stress? Does the improvement in parenting practices translate into detectable changes in child development outcomes, ranging from health, to cognitive and socio-emotional development? What is the value added of increased intensity to the basic parenting program on knowledge, parenting and child development outcomes? (The exact nature of ‘intensity’ is discussed at more length in the description of the intervention). II. Outcome indicators We will focus our study on children ages 0 to 5. Messages regarding health, early stimulation and enhancing attachment are likely to be critical for the younger age group while sessions relating to behavioral issues, discipline will be more likely to apply to children above one/one and half years, once children are walking and start exploring the world. We extended the focus on the 0-2 to be able to measure the impact of the intervention on a wider range of behaviors and outcomes. The direct testing of children will conform to the guidelines on protocols and informed consent by the national IRB committee (Comite de Etica de Investicaciones con Humanos, Servicios de Salud). 3 Child Development outcomes o Young children (0-3.5 years): We plan to use the Bayley Scales of Infant Development administered by a trained rater. The test identifies the child development competencies and deficits across five major developmental domains: cognitive, language, socio-emotional, and fine and gross motor development and adaptive behavior. The Bayley scale also includes an examiner assessment aimed at measuring executive function. An adaptation of this scale has been widely used in Chile. A team member of this evaluation from the Psychiatric Department of the University of Chile has recently participated in the training and field work supervision of a large clinical trial using the Bayley’s (Rojas et al 2007). The evaluation team will also draw from the administrative data on the history of scores on the Escala de Evaluacion de Desarrollo Psicomotor (EEDP) that has been normed for Chile and used within as a screening device within the health visits (more on this below); o Older children (3.5-6 years): We will focus on a subset of domains of child development that have been linked to school readiness and subsequent performance in school: Vocabulary will be measured using the Test de Vocabulario en Imagines Peabody (TVIP), the Spanish version of the Peabody Picture Vocabulary Test (PPVT). Attention and speed of processing: the proposed instrument is the Leiter-R. Socio-emotional development: the proposed instrument is the Achenbach Child Behavior Checklist Executive function: the choice of the instrument is currently under discussion, allowing for experimentation during the pretest (Backward Digit Span and Stroop test). Parenting and home inputs: To assess family milieu and well-being as well as the quality of the mother-infant relationship we will use an abbreviated version of the Home Observation for Measurement of the Environment (HOME), infant version, containing 6 sub-scales and 45 items (Caldwell and Bradley 1984). This is an instrument designed to assess the quality and quantity of support for cognitive, social and emotional development that is available to a child through the home environment. It has been widely used in studies in the UK, US. It has been translated and used in Chile (Bulnes et al 1979, Tippie 2003). A Parent-Child Interaction Scale included in the Canadian National Longitudinal Survey of Children and Youth (NLSCY) will also be pre-tested (Johnson and Mash 1989). Maternal mental health and psychosocial well-being: The team will also rely on the expertise of the local psychologists at the U. of Chile to adapt measures of maternal selfefficacy and Parenting Sense of Competence Scale (Ohan et al 2000). Maternal mental health will be measured using: o the Center for Epidemiologic Studies Depression Scale (CESD) which captures symptoms of clinical depression, o a measure of psychosocial distress used in Mexico and Chile in the context of the evaluation of large scale conditional cash transfer programs. 4 Another key dimension arising from qualitative reports in Canada and preliminary structured interviews in Chile relates to the parental perceived sense of social support arising from the group interaction. Parental social support, problem-solving, and ability to cope with stress have all been found to impact quality of parenting (e.g., Cochran & Niego, 1995; Crnic & Low, 2002). Administrative data: The survey data will also be merged to detailed administrative data on each child-mother pair. Crece Contigo has a unique feature of a longitudinal follow-up of each child (apoyo al desarrollo biopsicosocial), with key administrative data on maternal pregnancy, birth outcomes and health visits linked for each child in the Integrated Social Information System (on line platform information system used for social protection programs in Chile). We will be therefore able to access retrospective history on risk factors and health outcomes for each child by collecting information on the personal national identification number. An informed consent will be administered to parents to be able access the health records of their children in the health clinics. In particular, the following key retrospective outcomes will be obtained from the clinical folder for each child: o Maternal outcomes: EPsA: Evaluacion psicosocial abreviada – screening test applied during prenatal checkups to detect multiple psychosocial risks2 EPDS: Edinburgh Postnatal Depression Scale– early screening for postnatal depression at 2 and 6 months of age of the child. o Child outcomes: Anthropometrics data: birthweight, head circumference, weight, and length of the infant examined using standard measures at birth, 8 weeks, and at the final follow-up assessment. These indices are measured regularly as part of a national program monitoring the growth and development of infants. Protocolo de Neurodesarrollo: screening for neurodevelopment between 0 to 2 months. Massie/Campbell Scale of Mother-Infant Attachment applied at 4 and 12 months. EEDP: Escala de Evaluación de Desarrollo Psicomotor (0-24 months old) applied at 8 and 18 months 3 and TEPSI: Test de Desarrollo Psicomotor (2-5 years old) administered at 36 months of age. Both are screening measures of language, social, coordination and gross motor delays developed in Chile and normed for the Chilean population. There is also a reduced version of the EEDP (pauta breve) administered at 1, 12, 21 and 24 months. In addition, basic socio-economic characteristics of the family can be obtained from the Ficha de Proteccion Social, the main targeting instrument of social programs in Chile, which captures socio-demographic information, employment, income and incidence disability and health shocks. 2 The tests detects the following risk factors: presence of depressive symptoms or history of depression, reported or suspected physical abuse, single parent, substance abuse. 3 The EEDP is re-administered at 10 and 21 months if there are signs of delays during the previous test results. The TEPSI is re-administered at 42 months if there are delays are detected in its previous tests results. 5 More in-depth socio-economic characteristics will be measured with a light household survey instrument, with special attention to time use and expenditures in children. Finally, we will attempt to collect data on the characteristics of health facilities and of program facilitators, using a combination of administrative and survey data. We plan to add a facilitator questionnaire, collecting demographic and socio-economic background information and proxies for their experience and ability for engage groups. Detailed cost information about the program will be collected in conjunction with the Ministry of Health. A critical component of the evaluation will be a cost-benefit analysis of the intervention. III. Methodology This program is to be implemented nationally in Chile but at present it does not exist yet anywhere. The target population of clinics includes basic family health care, rural health clinics, urban health clinics and health establishments with minimal service complexity: this corresponds to about 600 clinics in Chile, covering 342 municipalities. The evaluation will follow an experimental design, using the health clinic as the unit of randomization. The gradual roll-out of the program is necessary because of implementation constraints4, and it is not an imposition of the evaluation (rather, it is an opportunity we take advantage of). The fact that the set of clinics chosen for the early and late stages of the roll-out is done using a lottery is an imposition of the evaluation. Therefore, in an initial stage, a set of health clinics (nationally representative) will be randomly chosen from the universe of health clinics where the program will be first implemented. The first group covers about 2/3 of the health clinics in Chile. Within the first set of clinics, two equal size randomized arms will be implemented: Group A: Basic NEP – parenting only, as currently designed Group B: NEP with increased intensity: two extensions are under discussion with the program team and are of interest in light of its learning potential for scaling up: a. NEP basic program with two added group sessions that include with play and direct interaction with children; b. Basic NEP that holds constant the number of sessions but includes observation of video-materials. Group C: The remaining clinics (about 1/3) in group two will not be able to receive the program for a year and will form the control group. In larger health centers which have more than one facilitator, we will also experiment with variation within centers, by introducing encouragement protocols for certain age groups, such as those under the age of two. Children (and their respective household) attending these clinics will be assessed at baseline (before the program is implemented) and one year after implementation. Follow-up funding for a medium term assessment three years after implementation will be sought at a later stage. 4 Mainly driven by the time it needs to train the facilitators. 6 Power size calculations With 100 health clinics in treatment A, 100 clinics in B, and 100 health clinics in the control group, if we sample 5 mother/children pairs per clinic, assuming an intra-cluster correlation of 0.2, it is possible to detect an effect across pairs of alternatives of 0.2376 standard deviations in the Bayley's test using a 5% level of significance (and power of 80%). Under these power calculations suggest that we should survey at least 500 households in the each of the treatment groups and 500 households in the control group, which accounts for 1,500 households in total. One potential problem with this calculation is that participation in the program is voluntary, and therefore, it is likely that even though a program is offered in a clinic, less than 100% of the mothers take it up. Based on the take-up of other education programs offered in Chilean clinics on a voluntary basis, we believe we can predict a take-up rate of 50% to 80%, which would imply increasing this sample size by 25% to 100% (1 to 5 mother child pairs per center). IV. Evaluation Team Emanuela Galasso is Senior Economist at DECRG. She has extensive experience in evaluation of social programs. Especially relevant for this proposal is her evaluation of a nutrition and child development program in Madagascar. Pedro Carneiro is Associate Professor at the Department of Economics, University College London. He is an expert in the economics of child development and the evaluation of social programs. Miguel Cordero is responsible for managing the health components of Chile Crece Contigo at the Chilean Ministry of Health. He also has experience in the evaluation of health programs, having conducted a randomized control trial to measure the impact of an intervention to ameliorate maternal post-natal depression in Chile, measuring impacts both on mothers and children alike. Other members of the evaluation team include: - Cecilia Moraga: National Coordinator of Nadie es Perfecto, Ministry of Health - Paula Bedregal, Associate Professor, Universidad Católica de Chile, Departamento de Salud Pública - Alejandra Guerrero: Clinical psychologist at the Universidad Católica de Chile - Rodrigo Herrera and Paula Castro, Evaluation Unit within the Social Division in the Ministry of Planning V. Detailed Description of Funding Request Justification and Timeline The primary funding source for the intervention is the Chilean Government. The World Bank is involved with a technical assistance loan (P114774) that supports the strengthening the scaling up of the social protection system within the Ministry of Planning (MIDEPLAN). The universal early childhood policy, Chile Crece Contigo, is an integral part of the social protection strategy in Chile. One of the objectives of the loan is to strengthen the system of monitoring and evaluation of social programs. The adaptation of Nobody's Perfect to Chile is already done and the program is to be rolled out in the near future, as the implementation of different components of Chile Crece Contigo evolves over time. The implementation of Nobody's Perfect will be, as much as possible, coordinated with the evaluation efforts, for which these funds will be essential. Therefore, as much as 7 possible, we will try to delay the roll out of the program until funding for the evaluation is secured, if funding can be secured in the short run. Timeline: June 2010: baseline, with the immediate roll out of the program afterwards December 2010: baseline survey June 2011: follow-up survey December 2011: evaluation report Budget: A) low case scenario (1,500 households) Unit Weeks A. Staff salaries B. Consultant fees International Consultant (for the intensive component) Research Assistant/Field Coordinator (Local) Local consultant 1 (local psychologist Bayley's) Local consultant 2 (local psychologist) C. Travel & Subsistence Staff - International airfare Staff - Hotel & Per Diem International airfare - Field Coordinator Hotel & Per Diem - Field Coordinator International airfare - International Consultant D. Data Collection Baseline Data type 1: Household/Individual Data type 2: Direct cost of the Tests Pre-test and adaptation of the tests V. Other Report preparation Total Evaluation Costs Funds from other sources Funds requested from SIEF Rate/unit days/units Days Days days 500 200 200 200 7 80 60 20 Trips Days Trips Days Trips 1,500 100 1,500 100 1500 1 20 1 50 1 Indiv. Total Total 70 1,500 17,500 15,000 6000 USD 0 35,500 3,500 16,000 12,000 4,000 11,500 1,500 2,000 1,500 5,000 1,500 137,500 105,000 17,500 15,000 6000 190,500 190,500 B) high case scenario (3,000 households) Unit Weeks A. Staff salaries B. Consultant fees International Consultant (for the intensive component) Research Assistant/Field Coordinator (Local) Local consultant 1 (local psychologist Bayley's) Local consultant 2 (local psychologist) C. Travel & Subsistence Staff - International airfare Staff - Hotel & Per Diem International airfare - Field Coordinator Hotel & Per Diem - Field Coordinator International airfare - International Consultant D. Data Collection Baseline Data type 1: Household/Individual 8 Rate/unit days/units Days Days days 500 200 200 200 7 80 60 20 Trips Days Trips Days Trips 1,500 100 1,500 100 1500 1 20 1 50 1 Indiv. 70 3,000 USD 0 35,500 3,500 16,000 12,000 4,000 11,500 1,500 2,000 1,500 5,000 1,500 260,000 210,000 Data type 2: Direct cost of the Tests Pre-test and adaptation of the tests V. Other Report preparation Total Evaluation Costs Funds from other sources (tbd) Funds requested from SIEF Total Total 9 35,000 15,000 35,000 15,000 6,000 6,000 313,000 113,000 200,000 References: Aboud F.E. (2007) Evaluation of an early childhood parenting program in rural Bangladesh. Journal of Health, Population and Nutrition, 25, 3-13. Bronfenbrenner, U. (1986). The ecology of the family as a context for human development: Research perspectives. Developmental Psychology, 22, 723-742. Bronfenbrenner, U. (Ed). (2005). Making human beings human: Bioecological perspectives on human development. Thousand Oaks, CA: Sage. Bulnes B, Cajdler B, Edwards M, Lira M (1979) Inventario para evaluar el ambiente familiar: (HOME). Estudio exporatorio en una muestra de nivel socioeconómico bajo. Unpublished manuscript, Santiago, Chile. Caldwell, B.M., & Bradley, R.H. (1984). Home observation for measurement of the environment. Little Rock: University of Arkansas at Little Rock/Centre for Child. Campbell, D., & Palm, G.F. (2004). Group parent education: Promoting parent learning and support. Thousand Oaks, CA: Sage. Chislett G, Kennett D (2007). The effects of the Nobody’s Perfect Program on Parenting Resourcefulness and Competency. J Child Fam Stud 16:473-482. Crnic, K., & Low, C. (2002). Everyday stresses and parenting. In M.H. Bornstein (Ed.), Handbook of parenting: Vol. 5: Practical issues in parenting (2nd ed.) (pp.243-267). Mahwah, NJ: Lawrence Erlbaum. Cochran, M., & Niego, S. (1995). Parenting and social networks. In M.H. Bornstein (Ed.), Handbook of parenting: Volume 3: Status and social conditions of parenting (pp. 393418). Mahwah, NJ: Lawrence Erlbaum. Currie, J. (2009). Healthy, Wealthy, and Wise? Socioeconomic Status, Poor Health in Childhood, and Human Capital Development, Journal of Economic Literature, 47 #1, March 2009, 87-122. Dooley, M., & Stewart, J. (2007). Family income, parenting styles and child behaviouralemotional outcomes. Health Economics, 16, 145-162. Engle, PL., Maureen M. Black.et al. and the Lancet Child Development Series Steering Committee (2007). Strategies to avoid the loss of developmental potential in more than 200 million children in the developing world. The Lancet, 369, 229-242. Gershoff, E.T. (2002). Corporal punishment by parents and associated child behaviors and experiences: A meta-analytic and theoretical review. Psychological Bulletin, 128, 539579. Hamadani, J.D., Huda,S.N., Khatun, F., & Grantham-McGregor, S.M. (2006). Psychosocial stimulation improves the development of undernourished children in Rural Bangladesh. Journal of Nutrition, 136; 2645-2652. Johnson, C. and Mash, E.J. (1989). A measure of parenting satisfaction and self-efficacy, Journal of Clinical and Child Psychology, 18, 167-75. McLennan, J.D., & Lavis, J. (2006). What is the evidence for parenting interventions offered in a Canadian community? Canadian Journal of Public Health, 97,454-458. 10 McLennan, J.D., Leon, T., Haffey, S., & Barker, L.A.S. (2009). Exporting a Canadian parenting education program to the Dominican Republic. Public Health Nursing, 26, 183-191. Nowak, Chrisoph and Nina Heinrichs (2008). A Comprehensive Meta-Analysis of Triple PPositive Parenting Program Using Hierarchical Linear Modeling: Effectiveness and Moderating Variables. Clinical Child Family Psychological Review. 11: 114-44. Ohan J, Johnston C, Leung D (2000). The parenting sense of competence scale: Evidence of a stable factor structure and validity. Canadian Journal of Behavioural Science 32: 251264. Oldershaw L (2002). A national survey of parents of young children. Toronto, ON: Invest in Kids. Rojas, G. R. Fritsch, J. Solis, E. Jadresic, C. Castillo, M. González, V. Guajardo, G. Lewis, T. Peters, R. Araya (2007). Treatment of postnatal depression in low-income mothers in primary-care clinics in Santiago, Chile: a randomised controlled trial, The Lancet, Volume 370, Issue 9599, Pages 1629-1637 Schady, Norbert and Christina Paxson (2007). Cognitive Development among Young Children in Ecuador: The Roles of Wealth, Health, and Parenting. Journal of Human Resources, 42(1): 49-84. Skrypnek, Berna J. and Julianna Charchun (2009) An Evaluation of the Nobody’s Perfect Parenting Program: Final Report, Department of Human Ecology, University of Alberta, Canada. Tippie J (2003) Psychometrics and cultural modifications of HOME in Chile. Paper presented at Assessing home environment for children from diverse backgrounds. Workshop: Center for human growth and development. University of Michigan. Todd, Petra and Kenneth Wolpin (2007). The Production of Cognitive Achievement in Children: Home, School and Racial Test Score Gaps, Journal of Human Capital, 1:91-136. 11