Cross-national measurement of
school learning environments
Creating indicators for evaluating
UNICEF’s Child Friendly Schools
Erin Godfrey1
David Osher2
Leslie Williams1
Sharon Wolf1
Juliette Berg1
Catalina Torrente1
Elizabeth Spier2
Larry Aber1
1New
International Society for Child
Indicators
3rd International Conference
28th July, 2011
The University of York
York University
Institutes for Research
2American
Overview of presentation
• Introduce importance of schooling and
•
•
UNICEF’s “Child Friendly Schools”
Initiative
Describe methods for creating multi-item
indicators to evaluate Child Friendly
Schools
Highlight implications for the
development of cross-national indicators
for use in comparative policy analysis
and advocacy efforts
Schooling important aspect of
child well-being
• Schooling is one experience that most
children worldwide have in common and
the most common means by which
societies prepare their young for the
future.” (UNICEF, 2009)

More than a billion children worldwide are in
school
• Primary mechanism for economic and
social development of nations
Schooling important aspect of
child well-being
• In the past decade, significant progress
•
•
has been made towards increasing
school enrollment
However, 101 million primary school age
children do not attend school regularly
(UNICEF, 2009)
Poor educational quality, combined with
other family and community factors


constrain enrollment and attendance
thwart learning and development
“Never let schooling interfere with your education.” - Mark Twain? Grant Allen?
UNICEF’s Child Friendly
Schools Initiative
•
•
•
Rights-based perspective rooted in CRC
Practical guide in pursuit of MDG 2
Embraces a multidimensional concept of
school quality addressing the total needs of
“the child as learner”
•
•
•
Based in WHO’s mental health initiatives on
connectedness, caring and support
UNICEF’s interest in child-, family- and communitycentered approaches to school improvement
School effectiveness research (e.g. Battistich & Horn, 1997;
Christenson & Thurlow, 2004; Osher et al, 2007; Osher & Kendziora, 2010)
“Quality education is education that works for every child and enables all
children to achieve their full potential.” (UNICEF, 2009)
UNICEF’s Child Friendly
Schools Initiative
•
Based on three key interrelated principles:
• child-centeredness
• democratic participation
• inclusiveness
•
•
Seeks to promote change and set quality
standards across entire sector
Recognizes local realities and promotes
pragmatic “pathways towards quality” rather
than a rigid blueprint
Global evaluation of CFS
•
•
•
UNICEF supports CFS initiative in 95 countries
worldwide
First global evaluation of CFS conducted in
2008 by American Institutes for Research (AIR,
2009)
6 countries in 4 regions




the Philippines and Thailand (East Asia and the
Pacific Region)
Guyana and Nicaragua (Latin America and the
Caribbean Region)
South Africa (East and Southern Africa)
Nigeria (West and Central Africa Region)
Global evaluation of CFS
• Mixed-methods evaluation of 20-27 child
friendly schools in each country



Surveys from students, teachers and school
heads
School and classroom observations
Interviews/focus groups with school heads,
parents, teachers
Global evaluation of CFS
• Results provided important information
•
about how key principles of CFS are
understood and implemented in multiple
contexts
Lack of resources limited the ability to
reach other goals


Determine the extent to which CFS
objectives are being realized within and
across countries
Create tools to monitor future progress
Global evaluation of CFS
• Relied heavily on multi-item scales
• Initial analyses established internal
•
reliability of scales (Cronbach’s )
Advantages to factor analytic techniques
in structural equation modeling
framework (Carle, Blumberg, Moore & Mbwana, 2011)
NYU Collaboration
•
Deeper psychometric exploration of the validity
and reliability of multi-item scales used as
indicators of school quality


•
•
Can a common core structure of perceptions of
school quality be identified across diverse
countries,
or are country-specific measures needed to
characterize these perceptions?
First step to create cross-national tools to
monitor CFS efforts
Provide roadmap to others creating crossnational indicators
Initial focus
•
Student perceptions of quality of learning
environment



•
Students’ experiences of connectedness, safety,
respect and engagement in school related to
attendance, learning and drop out
Consistent with child-centered approach to indicator
development (Ben-Arieh, 2008)
Not as tied to implementation differences
Three countries: South Africa, Nicaragua, the
Philippines

Diverse in local context and CFS implementation
and evaluation
CFS Conditions for Learning
Survey: three dimensions
•
Emotionally supportive climate (15 items,  = .74)

•
Challenging, student-centered learning
environment (14 items,  = .79)

•
How much students feel listened to, cared about, and helped
by teachers and other adults in the school
How strongly students perceive that teachers and other adults
in the school encourage the active engagement of students in
the learning process and the academic success of all students,
and feel that what they are learning is interesting
Safe, inclusive and respectful climate (27 items,  =
.83)

Two sub-dimensions: (1) the extent to which students perceive
the school to be inclusive and respectful of all types of
students and (2) how emotionally and physically safe students
feel
Sample
• 3,072 children



807 in South Africa
1431 in the Philippines
835 in Nicaragua
• Grades 5 and 6 (roughly 9-12 years old)
• 53% female
Analytic approach
• Exploratory and confirmatory factor
analyses, two steps
1. Determine best set of items for measuring
each domain within countries
•
•
More comprehensive information about
psychometric properties of items
Relative importance of each item for measuring
theoretical construct and substantive
interpretation
Analytic approach
• Exploratory and confirmatory factor
analyses, two steps
1. Determine best set of items for measuring
each domain within countries
2. Formally test measurement invariance to
determine whether items and scales
operate the same way across countries
• Particularly important in creating comparative
indicators out of scales
• Otherwise, differences in mean levels of scales
or relationships between scales could be
methodological artifacts, not true differences
Analytic approach
• Mplus used to handle complexities in
data (Muthen & Muthen, 1998-2007)


Can treat indicators as categorical
Can account for nesting of students within
schools
• Items are reverse-coded to reflect
positive association with the construct
Step 1 results: Measurement
models within countries
•
Ran exploratory and confirmatory factor
analyses separately for each dimension



•
Emotionally-supportive climate
Challenging, student-centered learning environment
Safe, inclusive and respectful climate
Each dimension had sound psychometric
properties with at least a subset of items


Items loaded significantly onto factor
Fit statistics indicated adequate model fit (CFI and TLI ≥
.90; RMSEA ≤ .06; WRMR ≤ 1.0 (Hu & Bentler, 1999; Kline, 2005))
•
Each dimension characterized by a core set of
items, as well as some differences
Step 1 example:
Philippines
Nicaragua
South Africa
Teachers at this school really care about students like me.
0.92
0.84
0.95
My teachers give me feedback on my assignments that help me to
improve my work.
0.95
0.71
0.90
Adults in this school are usually willing to give students extra help.
0.94
1.00
0.98
Teachers notice if I am having difficulty with my lessons.
0.71
0.97
0.99
0.83
0.84
1.00
0.84
0.92
0.97
0.68
0.45
0.69
0.59
0.54
0.56
0.70
0.56
0.78
1.00
0.41
0.98
0.95
---
0.96
0.56
0.91
---
---
0.92
0.31
Families like mine are involved in making decisions that affect this
school.
0.53
---
0.80
It is difficult for students like me to get extra help from teachers. (r)
0.35
---
---
Emotionally-Supportive Climate
17
48
54
57
58
62
11
22
60
37
50
47
61
66
69
Teachers give students opportunities to improve their work if they
do poorly on an assignment.
I can talk to teachers or other adults at school if I am having
problems in class.
I can talk with at least one adult at school about things that are
bothering me.
My family knows what goes on inside this school.
Students at this school have the materials they need to support
their learning.
This school does a good job teaching students what they really
need to know in life.
This school does a good job in preparing students to continue on
for more education after they graduate.
This school does not try to help students who are behind in their
work to catch up. (r)
Sometimes I am too hungry to pay attention in school. (r)
Step 2 results: Measurement
invariance across countries
•
Perform formal tests of measurement
invariance across countries (Gergorich, 2006; Temme,
2006)



•
Estimate a model in which item intercepts and
loadings are allowed to vary across countries
Estimate a model in which item intercepts and
loadings are constrained to be equal across
countries
Evaluate change in model fit using the 2 difference
statistic
Tests whether each item measures the
underlying dimension in the same way across
countries
Step 2 results: Measurement
invariance across countries
• For each dimension, we were able to
•
identify a subset of items that
demonstrated measurement invariance
across countries
These items measure each dimension in
the same way across countries

free from differential response bias due to
cultural, linguistic or other factors
Step 2 example:
Philippines
Nicaragua
South Africa
17 Teachers at this school really care about students
like me.
22 My family knows what goes on inside this school.
1.02
1.02
1.02
0.62
0.62
0.62
37 This school does a good job teaching students what
they really need to know in life.
54 Adults in this school are usually willing to give
students extra help.
57 Teachers notice if I am having difficulty with my
lessons.
58 Teachers give students opportunities to improve their
work if they do poorly on an assignment.
11 I can talk with at least one adult at school about
things that are bothering me.
48 My teachers give me feedback on my assignments
that help me to improve my work.
60 Students at this school have the materials they need
to support their learning.
62 I can talk to teachers or other adults at school if I am
having problems in class.
1.01
1.01
1.01
1.00
1.00
1.00
1.08
1.08
1.08
1.09
1.09
1.09
0.94
0.90
0.74
1.28
1.78
0.97
1.01
0.57
0.84
1.23
1.45
1.02
Emotionally-Supportive Climate
Summary of results
• A common core set of items does
•
characterize student perceptions of
school quality across three diverse
countries/regions
A subset of these items demonstrated
measurement invariance

These can be used to create indicators of
school quality that can be compared across
countries
Insight into substantive
interpretation of each dimension
•
Challenging, student-centered learning
environment


How much students perceive that teachers and
other adults in the school encourage the active
engagement of students in the learning process and
the academic success of all students, and feel that
what they are learning is interesting
Core set of items does capture all of these features
Insight into substantive
interpretation of each dimension
•
•
Challenging, student-centered learning
environment
Emotionally supportive climate


how much students feel listened to, cared about,
and helped by teachers and other adults in the
school
Core set of items assess how much students feel
cared about and helped, but not how much they feel
listened to
Insight into substantive
interpretation of each dimension
•
•
•
Challenging, student-centered learning
environment
Emotionally supportive climate
Safe, inclusive and respectful climate


Two sub-dimensions: (1) the extent to which
students perceive the school to be inclusive and
respectful of all types of students and (2) how
emotionally and physically safe students feel
Analyses revealed two different dimensions (1)
general feelings of safety, inclusion, fairness and
respect and (2) harsh treatment in school
Insight into substantive
interpretation of each dimension
•
•
•
Challenging, student-centered learning
environment
Emotionally supportive climate
Safe, inclusive and respectful climate
• Limit interpretation of scales to these
aspects, consider adding additional items
Important cross-country
differences
• Able to identify core similarities in the
•
•
measurement of each dimension
Also found notable differences in the
structure and substantive interpretation of
each dimension within countries
Reflected each country’s specific cultural,
linguistic, and sociopolitical context, as
well as differences in CFS
implementation and evaluation
Implications for CFS
•
Create two sets of monitoring tools


•
Analyses based on only three countries

•
The core set of invariant items can be used to
create tools for cross-national comparisons
This set can be augmented with additional countryspecific items that measure unique aspects of
school quality
Further work will extend analyses to all six countries
in global evaluation, others where these items are
used
Focus on reliability of scales

Further work to explore concurrent and predictive
validity and sensitivity to change
Implications for indicator
development
• Process holds promise for creating
•
•
reliable and meaningful monitoring tools
for international policies and programs
with varying conditions for
implementation and evaluation
Provide a roadmap for future evaluation
efforts
Foster a new more stringent concern with
psychometric properties of multi-item
indicators