1

2

3
Definition of Quality







4
EEC funded Research since 2009

Teacher Quality



MA Professional Development System Study
Project of the CAYL Institute
Literacy/Social Emotional/Numeracy/Digital Strategies

Screening and Assessment



Massachusetts Kindergarten Readiness Assessment System
Massachusetts Common Metric
Child Assessment in Universal Pre-Kindergarten

Program Quality

Evaluation of Summer 2010 Out-of-School Time Literacy &
Learning Promotion Grant





Institutions of Higher Education Mapping
QRIS Participation Study
QRIS Pilot Evaluation Phase I
QRIS Pilot Evaluation Phase II
QRIS Validation Study

5
EEC funded Research con’t

Community and Family Engagement


Early Childhood Information Systems
State Advisory Needs Assessment




Home Visiting Grant Evaluation
Limited English Proficiency
MA Child Care Resource and Referral Agencies (CCR&R) System
A Study Circle
Family Survey on Waitlist




Massachusetts Head Start Collaboration Needs Assessment
Survey Report
Early Education and Care Policies and Guidelines for Children
Whose Home Languages are other than or in addition to
English
Massachusetts Market Price Survey
Waitlist Analysis, Program Access Analysis, & Continuity of Care
Study

6
Examples of Key
Findings/Recommendations

Teacher Quality- MA Professional Development Study











Invest in supporting regional partnerships to implement the professional development system
Continue to define the scope, limits, and purpose of regional authority and responsibility particularly as it relates to innovation.
Assure a timely, accurate and high quality information flow from EEC to regional partnerships
Assess outreach efforts to Priority A, dual language and family educators.
Continue progress made to foster reciprocal communication, or communication loops, between EEC and regional partnerships to promote sharing of information and perspectives and to mutually inform policy and practice.
Share efforts by EEC to address compensation issues and engage regional partnerships in initiatives and efforts focused on compensation and incentives.
Implement utilization of MOU’s between educators and program directors, and collect/share data regarding MOU use to inform policy and practice development.
Advance the regions’ understanding about specific strategies for aligning professional development with QRIS.
Increase regional capacity around effective practices for assessing educator competencies and individual pathways, and use educator/provider assessment in a continuous cycle to informal professional development service planning and delivery.
Continue efforts to provide lead agents with access to relevant and useful EEC data on educators and providers.
Assess options for increasing the use of evidence-based professional development approaches, and support regions in evaluating the effectiveness of professional development services.

7
Examples of Key
Findings/Recommendations

Screening and Assessment- Child Assessment
Universal Pre- Kindergarten

Using web-based (or on-line) submissions as a way of collecting data should be a requirement of UPK grantees









EEC web-based licenses should indicate if they are a UPK grantee and specify the program’s UPK children.
While UPK grantees should be allowed to assess all of the children in their programs through the on-line license, they must delineate the children that are enrolled in UPK.
Increased and efficient communication with family child care grantees that belong to a family child care system needs to be systematically planned in order to raise submission rates.
Domain completion rates could be increased with professional development.
Professional development on how best to utilize the web-based system of assessing children should be offered annually
To determine specific professional development needs for teachers as related to early childhood assessment, EEC will need to depend on director interviews, teacher interviews and/or focus groups or observation.
Professional development coupled with technical assistance should be a systematic part of offering UPK grants and incentives.
Provide professional development and technical assistance to UPK grantees on how they can use their aggregated child assessment data at the local level.
Massachusetts should continue its practice of encouraging UPK programs to assess children using evidence-based assessment tools through grants and incentives to track child outcomes.

8

Examples of Key
Findings/Recommendations
Screening and Assessment- Child Assessment
Universal Pre-Kindergarten con’t


Massachusetts should consider whether it wants to add additional tools to it approved list, or whether it would in fact be better to narrow the choices to one or two tools that are most often used.
Examine whether High Scope Child Observation Record should remain one of the four EEC- approved child assessment tools due to low use.


Investigate new versions of the ASQ and the Creative Curriculum Developmental
Continuum to determine what, if anything, grantees need to incorporate them into their data collection systems.
Consider whether ASQ should remain an approved tool.

9
Examples of Key
Findings/Recommendations

Program Quality- QRIS Participation Study

Non-Participating programs
• Verify and update program contact information
• Increase communication with programs
• Ensure communication is language-accessible
• Consider ways to streamline the process for hard-to-reach programs
• Provide outreach to programs not currently participating to gauge interest and need

Participating programs
• Engage in additional public education regarding the QRIS
• Financial support and professional coverage
• More training
• Clarify how to access a QRIS contact person/unit/consultant at EEC
• Help programs acquire rating tools
• Glossary of terms and FAQ
• List of resources
• Consider changes to the QRIS rating system
• Provide follow-up
• Consider ways to provide QRIS support to programs that need it

10
Examples of Key
Findings/Recommendations

Community and Family Engagement- State Advisory
Needs Assessment

EEC may want to continue or expand initiatives to assist educators in understanding the importance of key features in the state’s QRIS standards.








EEC may want to consider these findings as it formulates future policies that promote degree attainment and increased competencies among educators.
EEC may want consider these findings as it formulates policies to assist programs and educators in supporting the needs of English language learners.
EEC may want to consider this finding as it formulates policies to promote and support inclusive opportunities for children with special needs or disabilities.
EEC may want to continue initiatives designed to help educators manage challenging classroom behaviors.
EEC may want to consider options for replicating the educator survey on an ongoing basis so that survey results can be linked to quality data from the QRIS and data from periodic family surveys to determine if programs and educators are meeting child and family needs, and overtime, to help evaluate the effectiveness of policy initiatives.
EEC may want to consider using data collected from both the representative sample and the public version of the survey to model the level of effort that would be required to make improvements to the early education workforce in different regions and to estimate the cost of related quality supports and incentives for programs and educators.
EEC may want to continue or expand community engagement and outreach initiatives that will help families understand important quality considerations and their connection to school achievement.
EEC may want to consider these differences in tailoring outreach strategies to promote the QRIS and other quality-related initiatives.

11
Examples of Key Findings

Community and Family Engagement- State Advisory
Needs Assessment con’t

EEC may want to expand strategies to support programs and educators in the inclusion of children with special needs.







EEC may want to keep in mind these findings and the need that some parents may have for more flexible scheduling as it considers changes to eligibility policies for families and as it considers options for rate reform.
EEC may want to consider tailoring certain policy strategies to target families that do not use formal early education and care programs, especially in promoting the expansion of QRIS.
EEC also may want to keep this finding in mind as it considers strategies to ensure that all families have access to translated materials that provide information on early education programs, literacy initiatives and other programs important to children and families.
EEC may want to consider financial assistance strategies and incentives that help make high-quality programs more accessible to low-income families and encourage them to make selections based on program quality.
EEC may want to keep this finding in mind when considering future support for family engagement initiatives, especially those focused on helping hard-to-reach families gain access to information on and connections to comprehensive supports and services.
EEC may want to continue or expand strategies to strengthen early literacy and language development and further expand efforts to engage hard-to-reach families that may not be connected to formal early education.
EEC may want to keep this finding in mind as it considers strategies to help connect children and families to additional supports and services that promote healthy child development.

12
Examples of Key Findings

Community and Family Engagement- State Advisory
Needs Assessment con’t


EEC may want to keep this finding in mind as it considers strategies to help connect children and families to additional supports and services that promote healthy child development.
EEC may want to consider options for replicating the family survey on an on-going basis that would allow survey results to be linked to quality and educator data from the QRIS to determine if programs are meeting child and family needs, and over time, to help evaluate the effectiveness of policy initiatives.

EEC may also want to consider qualitative methods – e.g., focus groups or family interviews - for evaluating the needs of families to augment the quantitative assessment.

13

14
Current Studies








15

Teacher Quality


Research Questions
• What are the the characteristics of individuals who have enrolled in the Professional
Qualifications Registry.
• What are the characteristics of professional development courses offered to early educators.
• What are the professional development experiences of individual educators attended.
• What are the core competencies that are targeted in these professional development experiences.
• Are the three content areas (social-emotional development, literacy and numeracy) included in the professional development experiences of educators.
• Determine how are professional development experiences, with respect to taking college versus CEU coursework and in relation to the core competencies and content areas covered, being distributed across early childhood programs.
Methodology
• A database was developed by merging data from the following databases. The data was analyzed to answer the above research questions.
• PQR
• EPS Grantee PD Attendance Records from FY12
• CSEFEL Training Attendance List
• EEC Course Catalog FY12
• EEC Program Priority List

Funded through ARRA funds.

16

Teacher Quality

Research Questions- Social Emotional
• Which classroom practices best support the healthy social/emotional development of children birth to age 5?
• How does the rate of expulsion relate to educators’ effectiveness across all learning domains with emphasis on competency in meeting the social/emotional needs of children?
• How does the rate of expulsion relate to the extent of parental engagement with the program?
• Which children remain most at risk for expulsion?
• What program characteristics are most associated with a healthy social/emotional climate and fewer expulsions?
• How do educators in the program create a classroom climate that is conducive to healthy social emotional development?
• What is the relationship between educators’ access to supports, such as intentional consultation on a regular basis and related professional development, and the success of children with challenging behaviors in programs?
• How do programs that have supports compare to those that do not in terms of expulsion rates, behavior problems, parental involvement, and observable classroom social-emotional climate?

17
Teacher Quality
• Research Questions- Literacy
• Which classroom practices best support the mastery of literacy in children birth to age 5 including those who are bi-lingual or whose primary language is not English?
• How does educator competency in curriculum areas addressing literacy affect the outcomes for children?
• What educator competencies help high needs children achieve desired literacy outcomes?
• What program characteristics are most associated with children achieving the desired literacy outcomes?
• What is the relationship between educators’ access to instructional supports, such as coaching and intentional professional development, and children achieving the desired literacy outcomes?
• What external factors affect programs’ and educators’ ability to achieve desired outcomes for the children in their care?
• Research Questions- Numeracy
• Which classroom practices best support the mastery of numeracy concepts by infant, toddler, and preschool children?
• How does educator competency in curriculum areas addressing numeracy affect the outcomes for children?
• What educator competencies help high needs children achieve desired numeracy outcomes?
• What program characteristics are most associated with children achieving the desired numeracy outcomes?
• What is the relationship between educators’ access to instructional supports, such as coaching and intentional professional development, and children achieving the desired numeracy outcomes?
• What external factors affect programs’ and educators’ ability to achieve desired outcomes for the children in their care?

18
Teacher Quality
• Research Questions- Digital Strategies
• Of the digital strategies examined, which are most effective in increasing teacher competency?
• Of the digital strategies examined, which are most effective in enabling parents to more effectively support their children’s development?
• How are digital resources best integrated into, and aligned with, the program’s curriculum?
• Which areas of development do these digital strategies support?
• How can the use of digital resources best reinforce and advance instruction?
• How do the characteristics of the early education setting interact with the digital strategies’ effectiveness?
• Are certain strategies more effective than others in supporting children’s healthy social, emotional development or the development of literacy and numeracy skills?
• What is the relationship between educators’ access to digital resources and accompanying instructional supports and children achieving the desired outcomes?
• How could the effectiveness of the digital strategies be increased?
• What external factors affect programs’ and educators’ ability to use digital strategies to help achieve desired outcomes?

Methodology
• AIR will use extant data collection from the state, regional offices, and local providers, Classroom observations (CLASS. ELLCO, COEMET) using validated, evidence-based tools in a stratified random sample of early childhood sites,
Teacher and administrator surveys, Teacher assessments of children’s socialemotional development and literacy and numeracy skills to collect data to answer the research questions.

Funded through ARRA funds.

19

20
Assessment and Screening




• What are the commonalities across each of the domains within each of the different assessment tools (Teaching Strategies GOLD,
Pearson Work Sampling and High Scope COR) including a separate analysis for preschool and kindergarten data.
• What is the feasibility of developing a statistical methodology to answer the question of what are the baseline skills, knowledge and abilities children enter with or bring to Massachusetts preschool and kindergarten children.
• Through analysis of the assessment data, identify common items with the domains.
• Using data analysis explore the items and the factor structure to determine reliability, inter-item consistency.
• Conduct confirmatory factor analyses and structural equation modeling.

21

22
Program Quality


Research Questions
• Validating the Five Key Components of Quality and the Associated
Standards of the MA QRIS
• How do the participating early education and care providers vary along the five key components of quality and their associated Standards?
• What does available evidence show regarding the link between the
Standards in the five components of quality and effects on child outcomes? Of the evidence-based Standards linked to child development outcomes which methods are used to measure these
Standards and is there a difference between self-assessment, as opposed to external verification, in relation to child outcomes?

23
Program Quality


• Assessing Differences in Quality Across Quality Levels and
Changes Over Time
• Do the QRIS levels or groups of levels (e.g., 1–2 vs. 3–4) represent true distinctions in program quality?
• Do the two groups of quality levels differ as expected in their underlying components of quality and their associated
Standards? Are some quality components and associated
Standards more important than others in distinguishing quality differences among the two groups of QRIS levels?
• Do the differences in quality between the QRIS levels or groups of levels (e.g., 1–2 vs. 3–4) vary according to the method in which the Standards are measured (self-assessment vs. external verification)? Is self-assessment a valid method for distinguishing between the QRIS levels or groups of levels?
• Is there an improvement in provider’s quality levels across time?
If so, what factors contributed to this improvement? What is the contribution of state quality improvement efforts to improvement in providers’ quality levels across time?

24
Program Quality


• Relating Quality Levels to Children’s Developmental Outcomes
• Do children enrolled in providers with QRIS levels 3–4 exhibit developmental gains over time and more optimal growth trajectories when compared with children enrolled with providers at quality levels 1–2 across the five domains of development
(language, cognitive, social, physical, approaches to learning)?
• Which components and Standards of the QRIS are most strongly associated with developmental outcomes and growth trajectories? What provider characteristics are most strongly associated with these child outcomes?
• The MA QRIS comprises five key components of quality including
Curriculum and Learning; Safe, Healthy Indoor and Outdoor
Environments; Workforce Development and Professional
Qualifications; Family and Community Engagement; Leadership,
Administration, and Management, as described in Exhibit 1.
Within each component, the Standards are specified for each quality level that is appropriate to the provider setting (e.g., center-based, school-based, family child care, and afterschool/out of school).

25
Program Quality

• The study will employ a mixed method crosssectional and longitudinal study design.
• Provider data will be collected at 2 time periods (Fall 2013 and Spring 2015) and child outcome data will be collected 3 times (Fall
2013, Spring 2014 and Spring 2015).
• A Pilot Study will be conducted in Spring 2013.

26

27
Community and Family Engagement


• How can we understand the community capacity sphere of the ecological model of child development?
•
• What is community capacity?
• What constitutes community capacity? What components, practices, and resources are markers of community capacity?
• How do capable communities mobilize to support child development?
• How do families and children connect with resources in their communities? By what pathways do such connections occur?
What are the key components of community capacity, as described in relevant literature and from the perspective of state- and community-based stakeholders?
• In what areas do specific communities demonstrate particular strength or weakness in relation to components of community capacity?
• In a subset of closely studied communities, what systems are in place to support child development?
• How accessible and well-utilized are systems/services?
• To what extent are services available well-aligned with the community’s demonstrated needs?
Are services available in the correct proportions with regard to need?
• What are the implications for policy and practice?

28
Community and Family Engagement


• Develop a definition of community capacity through engaging with community stakeholders as well as existing research
• Engage in community-specific data collection and assessment
• Develop data collection instruments and implement community-level data collection plans
• Work with community partners to deploy data collection efforts in relevant communities
• Monitor data collection efforts
• Gather, organize, and analyze community data; possibly engage in smallerscale data collection
• Identify, organize, and analyze secondary data
• On an ongoing basis, develop summaries of emergent findings, such that they can contribute to system planning and mid-course adjustments in response to community-level data. Sequence phases of data collection in such a fashion as to be complementary to other interdependent work pursued through the overall study.
