Evaluation of
Chicago Public
Schools’ Virtual Pre-K
Ready For Math
Program That
Integrates PBS KIDS
Lab Digital Math
Content
A Report to the CPB-PBS Ready To Learn Initiative
Betsy McCarthy, PhD
Donna Winston, PhD
Linlin Li, PhD
Lisa Michel
Katie Salguero
Sara Atienza
October 2012
WestEd — a national nonpartisan, nonprofit research, development, and
service agency — works with education and other communities to promote
excellence, achieve equity, and improve learning for children, youth, and
adults. WestEd has 16 offices nationwide, from Washington and Boston to
Arizona and California, with its headquarters in San Francisco. For more
information about WestEd, visit WestEd.org; call 415.565.3000 or, toll-free, (877)
4-WestEd; or write: WestEd / 730 Harrison Street / San Francisco, CA 94107-1242.
© 2012 WestEd. All rights reserved.
Evaluation of the VPK Math Program page 3
Executive Summary
This report describes a research study of Chicago Public Schools’ (CPS) Virtual Pre-­‐K Ready for Math Program (VPK Math Program) and its integration of PBS KIDS transmedia resources. Funded by the U.S. Department of Education, the Ready To Learn grant supports the development of educational television programs and digital media targeted at preschool and early elementary school children and their families. Ready To Learn’s goal is to promote early learning and school readiness, with a particular interest in reaching children from low-­‐income families. WestEd assessed the VPK Math Program’s effectiveness with regard to preschool teachers’ and parents’ comfort with mathematics, usage of digital media for education, and home-­‐school engagement. This research study, which is one component of the Year 2 Ready To Learn program grant with the Corporation for Public Broadcasting (CPB) and the Public Broadcasting Service (PBS), also provides program administrators and PBS with useful feedback to facilitate further development, refinement, and replication of the VPK Math Program. Virtual Pre-­‐K, first launched in 2001, is a national parent involvement initiative based out of the Chicago Public Schools’ Office of Early Childhood Education. The program, which is bilingual in English and Spanish, is an interactive resource that bridges the experiences in the classroom and students’ homes and community to create a connection for learning. Participating teachers attend workshops to support their students’ learning with simple, fun activities. Accompanying activities are offered to parents through parent workshops and a backpack of resources that goes home with each family. The new elements of the program for the 2011-­‐2012 school year included a focus on math and the integration of PBS digital media resources using iPads. During the 2011-­‐2012 school year, the Chicago Public Schools’ Office of Early Childhood Education recruited 15 preschool teachers and 15 teacher assistants responsible for 31 preschool classrooms to participate in the VPK study. Participants came from 12 low-­‐income school sites with an average of 91% of the student body eligible for free or reduced-­‐price lunch. Each preschool classroom consisted of one teacher and one teacher assistant. All parents and guardians with children in these classes were invited to participate. The program included five teacher professional development workshops and five parent workshops throughout the school year. These workshops familiarized teachers and parents with the available resources; corresponding activities were sent home with the children each month. The three workshop facilitators also participated in the study. The research study utilized a mixed-­‐method design to examine the program’s effectiveness and included data from preschool teachers, parents, and workshop facilitators. Research Method
This section addresses the research questions, participants, measures, and data analysis utilized for the study. Research Questions
The following five research questions served as the focus of the study. •
How d o p arents and p reschool teachers u se the V PK M ath k its? •
Does p articipation in the V PK M ath P rogram change p arents’ actions and p erceptions A Report to the CPB-­‐PBS Ready To Learn Initiative Evaluation of the VPK Math Program page 4
with respect to m ath, p articipation in their child’s learning, and u se of d igital m edia? •
Are there changes in p reschool teachers’ actions and p erceptions w ith regard to m ath instruction and the u se of d igital m edia to support student learning? •
Does p articipating in the V PK M ath P rogram increase students’ interest and p erformance in m ath? •
What are the incentives and b arriers for the replication of the V PK M ath P rogram? Participants
All preschool teachers and parents in the Chicago Public Schools district who were actively participating in the VPK Math Program were invited to participate in the research study. The workshop and professional development session facilitators were also invited to participate. There was no inclusion/exclusion criteria based on gender, race, or ethnicity. The Office of Early Childhood Education in CPS invited 30 preschool teachers and teacher assistants, as well as three workshop facilitators, to participate in the research study of the VPK Math Program. All three of the workshop facilitators conducted the teacher professional development sessions, while two of the facilitators ran the parent workshops that were an integral part of the VPK Math Program. The VPK Math Program staff informed the recruited teachers and facilitators about WestEd’s research study of the program. Teachers and facilitators then had the opportunity to participate in WestEd’s study by signing consent forms that allowed WestEd to both analyze data previously collected from them by the VPK Math Program and also to collect new data. All preschool teachers, teacher assistants, and facilitators agreed to participate in the research study. The Office of Early Childhood Education invited approximately 560 parents associated with children in these 31 preschool classes to participate in the research study as well. Many of the participating parents came from low-­‐income backgrounds and a significant portion did not speak English as a first language. Consent forms and all other relevant documentation for parents were provided in both English and Spanish. WestEd provided a script for facilitators to read to help explain the study and consent forms to parents. A total of 484 parents participated in the research study. Measures
The research team collected data through surveys, interviews, focus groups, and a review of relevant archival data. Teachers completed pre-­‐surveys and post-­‐surveys in addition to reflection and technology review mini-­‐
surveys during their professional development sessions. About half of the participating teachers were interviewed. Parents completed a pre-­‐survey, post-­‐survey, a fidelity survey (backpack evaluation), and technology reviews. All surveys, fidelity surveys, and technology review forms were offered to parents in both English and Spanish. A subset of parents was also interviewed. Beginning in January 2012, facilitators participated in bimonthly interviews. Relevant archival data primarily consisted of documents and materials that had been provided to program participants, and also included survey data previously collected by program staff. WestEd researchers only reviewed archival survey data from participants who consented to participate in this research study. WestEd staff reviewed the materials provided to parents at each parent workshop. Additionally, agendas from the teacher professional development sessions and parent workshops were examined along with other documents supplied to participants. Data Analysis
The research study used a mixed-­‐methods, single-­‐group, pre-­‐test/post-­‐test design to examine the effectiveness A Report to the CPB-­‐PBS Ready To Learn Initiative Evaluation of the VPK Math Program page 5
of the VPK Math Program and the integrated PBS KIDS transmedia resources. WestEd conducted quantitative and qualitative analyses to examine the data. Standard and non-­‐parametric paired samples t tests were used to compare closed-­‐ended pre-­‐ and post-­‐survey responses for preschool teachers, teacher assistants, and parents. All qualitative data (interview transcripts, written responses to survey data) were loaded into qualitative data analysis software (i.e., HyperRESEARCH). Researchers used a combination of grounded theory (Strauss & Corbin, 1998) and established methods for coding qualitative data (Miles & Huberman, 1994) to identify and categorize open-­‐ended responses to the study’s research questions. Throughout the process, researchers used peer debriefing and auditing to check codes and concepts. Identified codes and concepts were further sorted to generate categories. All data were triangulated to generate findings. Findings
The following findings are organized by relevant research question with each of four outcome categories: teacher outcomes, parent outcomes, student outcomes, and program feedback. Teacher Outcomes
How do preschool teachers use the VPK Math kits? Teachers were given a VPK teacher’s guidebook that included five math-­‐related, activity-­‐based classroom lesson plans, At Home activities, and Out & About activities. The “kits” that accompanied the teacher’s guidebook included at least one children’s book to read to the students, as well as resource materials for each particular activity, including: rulers, tape, shower curtain, play dough, etc. Eighty percent of the teachers used all five lesson plans. Teachers felt the lessons were easily integrated into and supported their existing curriculum. Additionally, the activities gave teachers an opportunity to help their students make real-­‐life connections to math concepts by emphasizing math in a fun and engaging way. Preschool teachers accessed the digital media component of the VPK Math Program primarily using desktop computers in the classroom or the school computer lab and by using iPads. The majority of the schools used basic Internet and wireless connections to access websites. The most popular website visited was PBS KIDS Lab. PBS provided each of the 15 classroom sites with two iPads. Thirteen classroom sites used the iPads regularly as a focal learning point in the classroom. Teachers commented that the iPads were generally much easier for their preschool students to use than regular computers, as they do not require the same fine motor skills and coordination as a computer mouse. The teachers also liked the portability of the technology, as it allowed them to access the iPad any time they wanted to check for information pertaining to VPK class activities. The most popular PBS KIDS iPad apps were: Super WHY!, Sid the Science Kid, and Dinosaur Train. Were there changes in preschool teachers’ actions and perceptions with regard to math instruction and the use of digital media to support student learning? Teachers integrated technology into the classroom on a daily basis. A high percentage of them reported daily usage of and access to desktop computers and iPads. Their students also used these technology tools on a daily basis. Over the course of the VPK Math Program, preschool teachers developed more positive attitudes towards integrating technology in the classroom based on a 5-­‐point Likert-­‐type scale (Pre M = 3.44, Post M = 3.72). Specifically, there was an increase in the enjoyment of using technology (Pre M = 3.46, Post M = 3.77) and A Report to the CPB-­‐PBS Ready To Learn Initiative Evaluation of the VPK Math Program page 6
confidence in using technology (Pre M = 3.23, Post M = 3.62). Additionally, fewer participants felt nervous using computers (Pre M = 3.04, Post M = 3.35) and many felt technology helped them teach (Pre M = 3.62, Post M = 3.85). Preschool teachers reported that they intend to continue to use the resources they were exposed to during the VPK Math Program, including the iPads, websites, and apps. Preschool teachers indicated an increase in positive attitudes towards teaching math. Despite already high, positive attitudes towards teaching math as demonstrated by the pre-­‐survey based on a 5-­‐point Likert-­‐type scale (Pre M = 3.27), the post-­‐survey shows that this attitude increased as a result of their participation in the VPK Math Program (Post M = 3.41). Specifically, they felt more confident in their abilities to teach math (Pre M = 3.46, Post M = 3.75) and reported less difficulty teaching math concepts to their students (Pre M = 3.14, Post M = 3.36). Overall, participants reported that they enjoyed the math activities and particularly appreciated using math tools such as rulers and measuring tape with their preschool students. Preschool teachers reported many benefits from participating in the VPK Math Program including: student growth as a result of parental engagement in their classrooms; exposure to new activities related to mathematics; reinforcement of classroom activities at home through parental engagement; and participation in a professional learning community of participating teachers with whom ideas and activities are shared. Teachers reported the primary benefit of using PBS KIDS digital media in the VPK Math Program to be preschool student access and exposure to new technology, websites, and apps in order to reinforce and extend learning. With regard to how the digital component contributed to their own experience, teachers reported that they gained new skills and understanding of new concepts by learning how to integrate technology into their daily teaching. As a result of this integration, some teachers reported that they placed a greater emphasis on mathematics in their classroom. Parent Outcomes
How do parents use the VPK Math kits? The majority of the parents (89%) reported that they used VPK Math backpack materials. Parents, on average, used the materials one to five times per month as indicated by the workshop evaluations. Parents who attended the monthly workshops were more likely to use the VPK kits than parents who did not. The response to the parent workshops was overwhelmingly positive, as was the general response to the activities presented during the workshops. According to parents who filled out workshop evaluation forms, their favorite parts of the backpack materials were the books and the At Home activities. The most popular device through which parents accessed the PBS KIDS digital media was the home computer. Some parents reported using school and/or library computers when they did not have this technology at home. Does participation in the VPK Math Program change parents’ actions and perceptions with regard to math, participation in their children’s learning, and use of digital media? In general, participating parents reported a moderate to high level of involvement in their child’s school based on a 4-­‐point Likert-­‐type scale (Pre M = 3.16) at the beginning of the study. After participating in the VPK Math Program, this level of school involvement improved slightly (Post M = 3.21). In particular, parents reported an increase in talking to their child’s teacher about their child’s learning (Pre M = 3.09, Post M = 3.31). Teachers also felt parents’ moderate rates of involvement increased slightly during the year based on a 5-­‐point Likert-­‐type scale (Pre M = 2.69, Post M = 3.03). Teachers felt parents became more engaged in their child’s learning at home by using digital media to play educational games, working on math with their child, and A Report to the CPB-­‐PBS Ready To Learn Initiative Evaluation of the VPK Math Program page 7
asking teachers about their child’s math learning at home. In general, participating parents reported a moderate level of engagement in their child’s educational activities based on a 4-­‐point Likert-­‐type scale (Pre M = 2.88, Post M = 2.93). Parents exhibited a slight improvement in the frequency of working with their children after participating in the VPK Math Program, especially with respect to going online to do educational activities with their children, such as playing PBS KIDS learning games (Pre M = 2.35, Post M = 2.50). Parents’ awareness of their children’s math learning and of ways they can support this learning increased. On the survey (98% of respondents) and in the workshop evaluation forms and focus groups, parents reported that the VPK Math packets and workshops helped them better understand the math their children were learning in preschool and that they were much more aware of their children’s math learning as a result of participating in the program. Around 97% of parents reported on the surveys that the VPK Math packets and workshops helped them gain more confidence about what they can do to help their children learn math. In general, the pre-­‐survey indicated that participating parents had a moderate to high level of math interest and confidence. After participating in the VPK Math Program, parents significantly increased their levels of interest and confidence in math based on a 4-­‐point Likert-­‐type scale (Pre M = 3.08, Post M = 3.15). In particular, parents felt less worried about their ability to solve math problems (Pre M = 2.86, Post M = 3.00), and were more likely to view math as their favorite subject (Pre M = 2.89, Post M = 3.03). Most parents provided positive feedback about the digital component of the program. Many reported having positive experiences when asked to “explore PBSKIDS.org/lab with your child.” They felt the site was user friendly, fun, and engaging for their children. Parents also felt that the different PBS games like Curious George or Sid the Science Kid helped their children learn. Student Outcomes
Does participating in the VPK Math Program increase students’ interest and performance in math? Preschool teachers reported an increase in student interest and performance in the areas of math and technology use. Prior to the VPK Math Program, teachers rated their students as having moderately low levels of interest, comfort, and skill in these areas based on a 3-­‐point Likert-­‐type scale (Pre M = 2.07). Teacher post-­‐
survey responses indicated that teachers felt that their students improved in these areas (Post M = 2.40). Teachers reported that students felt more comfortable with math (Pre M = 1.92, Post M = 2.40) and technology (Pre M = 1.88, Post M = 2.40). According to teachers, there was also a noticeable increase in students’ technology skills (Pre M = 1.80, Post M = 2.28). Additionally, several teachers commented on the growth in students’ math vocabulary as a result of the program, as well as students’ ability to make connections between math vocabulary and the process of doing math. Program Feedback
What are the incentives and barriers for the replication of the VPK Math Program? The digital media aspect of the VPK Math Program provides two main incentives for parents to participate. One incentive is that it provides children an opportunity to be exposed to technology that might not otherwise have been available. The other incentive is that the PBS KIDS digital media resources provide an alternate way to engage children in learning. A Report to the CPB-­‐PBS Ready To Learn Initiative Evaluation of the VPK Math Program page 8
Teachers greatly enjoyed using the iPad with their students in the classroom. It provided a way for them to integrate technology into their curriculum and helped them reinforce certain concepts. Parents and teachers encountered several technology barriers. For parents, home access to technology was an issue because some parents did not have computers at home. Those that did have home computers sometimes had very slow Internet connections, which caused frustration while trying to play games. It was difficult to provide parents access to computers during workshops, as students were typically using them at that time. A few teachers had broken computers and/or unreliable Internet connections. A non-­‐technology barrier that some VPK Math Program participants encountered was the availability of resources in Spanish, as some of the books and online games were only available in English. Conclusion
This CPB-­‐PBS Ready To Learn Initiative research study examined the impact of the VPK Math Program on preschool teachers, parents, and students. Findings indicate that the VPK Math Program increases parents’ engagement in their children’s math learning, provides resources to supplement preschool teachers’ math instruction, and boosts students’ math and technology skills. Results of the study also show multiple benefits of the VPK Math Program for all three groups associated with teachers’ and parents’ participation. The iPads, PBS KIDS digital media resources, and lesson plans provided preschool teachers with additional tools to supplement their math instruction. Workshops supported teachers’ efforts to integrate these elements into their classroom. Parents acquired skills and resources to help them become more actively involved in their children’s math learning. Both parents and teachers remarked on the growth they observed in students’ math and technology skills as a result of their participation in the VPK Math Program. This study underscores the promise of integrating digital media resources into the proven Virtual Pre-­‐K model and focusing on math as a means to increase low-­‐income students’ mathematical learning. It also demonstrates that professional development, carefully designed lesson plans, and access to technology resources can help preschool teachers successfully augment their established curriculum and deepen their students’ conceptual understanding in mathematics while building technology skills. Additionally, the VPK Program promotes parent engagement in their children’s education and helps build a collaborative relationship between home and school. Results of the research study suggest that further study of the VPK Math Program is warranted. Topics for future study include: gauging how much students’ mathematical skill sets increase as a result of their teachers’ and parents’ involvement in the program, how parents use the digital and backpack materials together to support their children’s math learning, and the added value of each component of the VPK Math Program. As one teacher commented: “I think that Virtual Pre-­‐K Math should not go away. It should be something that other teachers have an opportunity to find out how to use, because I think it’s very, very beneficial. Not only for us, but for the students and the parents.” A Report to the CPB-­‐PBS Ready To Learn Initiative