Cautions About Inferences
From International
Assessments
Kadriye Ercikan
University of British Columbia
This discussion is based on a paper co-authored with WolffMichael Roth and Mustafa Asil,
in press in the Teachers College Record
Key Inferences From International Assessments
❖
Compare country performances for identifying the
countries with the best education systems
❖
Generate insights about effective policy and practice
strategies that are associated with higher learning
outcomes.
Some Examples
❖
TIMMS video study data (e.g., Shimizu, 1999)
❖
Germany’s efforts to make many system level changes
as a reaction to the “PISA Shock” (Ertl, 2006)
❖
England’s importing of mathematics teachers from
China as a reaction to the country’s low rank in the most
recent PISA mathematics assessment (FlorCruz, 2014)
Validity of Inferences
❖
Do country rankings really reflect the quality of education
in different countries?
❖
What are the fallacies of simply looking to higher
performing countries to identify strategies for improving
learning in our own countries?
Focus
❖
Describe limitations of using country rankings as
indicators of quality of education
❖
Discuss limitations of using correlates of higher
performance in high ranking countries as a way of
identifying strategies for improving education in our
home countries
Sequence of of Inferences
Inference 1: Identifying countries with the most effective
education
❖
Implied Propositions
Inference 2: Identifying effective policies or practices
responsible for high ranks
Implied propositions
Inference 3: Emulating policies and practices will lead to
improvement in education
Implied Propositions
Limitations in Identifying Countries with Most Effective
Education Systems
❖
There are several sources of incomparability in international assessments
that create inaccuracies in comparing and ranking countries
❖
Overall country performance is one of many indicators of education
systems
❖
Multiple indicators such as school drop out rates, school climate, student
and teacher behavior, and students’ perceptions of benefits of schooling
additional criteria to consider
❖
Focusing on performance at the level of entire countries may obscure
important within-country differences in rankings for subgroups, such as
gender or ethnic groups
Graduation Rate Rank versus Reading Score Rank
Comparison of gender groups within the five countries on
PISA 2009
Conclusions
❖
❖
Do higher ranks indicate better education systems?
❖
Comparability
❖
Multi-dimensionality of quality of education
Should we look toward other jurisdictions to copy strategies and practices
for the purpose of changing education in another jurisdiction?
❖
Variations in achievement are likely due to much more complex
interrelations of cultural, societal, and educational factors.
❖
Taking this or that student variable and education practice in one
context and trying to replicate it somewhere else may not result in the
aspired to learning outcomes from the original jurisdiction.