PISA truth effects: the construction of low performance
Author: Margareta Serder, Malmoe University, Sweden; margareta.serder@mah.se
Key words: PISA, performed realities, lack of competence, scientific literacy
Network: 23 Politics of Education
This paper is engaged in questions of the realities “performed into being” (Gorur, 2010; Latour,
1999) through knowledge assessment, with a specific interest in PISA and its assessment of
scientific literacy. The concern about PISA results accelerated in Sweden, where this study is
located, after the latest PISA survey 2012, due to what has been asserted as a historical decrease
in performance. During the broadcasted press conference at the Swedish National Agency for
Education held to present the national results, the director-general stated her confidence in PISA:
“Here at the National Agency for Education we trust PISA” (Ekström, 13-12-03). In line with
Gorur (2010), Popkewitz (2011) and many others, I believe there is reason to stress the
problematic with this type of unreserved faith in PISA as evidence about educational failure or
success. In this paper, I will suggest that low performance is an attribute of the assessment, more
than an attribute of school systems or single students. Perhaps surprisingly, I will begin my
argumentation in a discussion about microbes:
In The Historicity of Things, Bruno Latour (1999) asks: Where were the microbes before Pasteur? As
writes Latour, many would find it commonsensical to answer that the microbes were in the same
place before and after Pasteur; that Pasteur did not change the course of events – he only
‘discovered’ a pre-existing phenomenon, and named it microbes. According to Latour, existence
presumes experience, and for that, to be given attributes: ‘the mind’ for example has to get the
attributes ‘individual’, ‘dis-embodied’ or ‘thinking’ before it can be experienced as such1.
Therefore, ‘obviously’ thing-like objects for thought, such as microbes – e.g. the ferments in
Pasteur’s nineteenth century laboratory in Paris – first needed to be given circumstances,
conditions, specific nutrition demands, and not least ways of being detected. Thereafter, they
became microbes. And therefore, Latour insists, microbes did not exist before Pasteur “happened”
to them (p. 146).
I argue that microbes are not much unlike knowledge, skills and competencies: A thing-like
object for thought such as lack of competence (as in a 21st century, OECD-produced assessment
about young citizens’ skills for future life) did not exist before a detector had been
discovered/invented/constructed (Latour, 1999), making it possible to discern it as something (on
detectors, see Knorr Cetina, 1999 and Pickering, 1995). The detector here is the test-instrument2;
adjusted to detecting defined competencies by the means of specific test-items within the
assessment’s literacy framework.
The ontological argument for Latour lies in what we have learnt to hold for true and not:
1
2
Which is exactly why the mind also can be experienced as collective, embodied and acting, for instance
In the case of microbes, the detector was the nose of Louis Pasteur.
1
In the correspondence theory of truth, the ferments are either out there or not, and if they are out
there they have always been out there, and if they are not they have never been there. (Latour
1999, p. 149).
I propose that measured competencies are not ‘caused’ by knowledge and learnt skills, but are
coming into existence through the means construed to detect them and that lack of competence
is linked to the historical event of starting to try to detect a specific competence. In this paper, I aim
to 1) trace the production of truth about low performance by looking into how the competencies
related to scientific literacy are described and written out in PISA documents and test-items and
2) study how ‘lack of competence’ comes into existence when students are reading, making
meaning of and responding to test-items from PISA science (OECD, 2006).
Methodology
The research is based on two different data materials; 1) written PISA documents (frameworks,
analyses, test-items) and 2) a 16 hour video-taped material of research-produced situations in
which 15-year old students collaboratively engaged in answering PISA test-questions in scientific
literacy during a science lesson in the fall of 2010.
In the study, I am interesting myself in how science and performance is discursively and
materially produced in the PISA science assessment. I base my analysis on the discourse theory of
Foucault (1971), in which knowledge is linked to “being in the truth” (p. 36), that is, to represent
what is believed to be true in a specific time, culture or social context. I use the notion of
discourse as operating by the repetitive inclusion and exclusion of elements into the discourse; a
mechanism through which discursive actions travel and become, or not, part of the discourse.
Further, I am trying to adopt the non-realist approach to the contingency of matter from Latour
as described earlier.
Documents
The analyzed documents are the PISA assessment frameworks from three of the five surveys
(OECD 1999, 2006, 2012) with focus on 2006 in which the scientific literacy framework was
defined, and the first and the most recent international report of results and analyses of scientific
literacy (OECD 2007, 2013), one OECD report on “strong performers” (2010) and finally the
released test-items in scientific literacy from https://mypisa.acer.edu.au/(8 units/24 items).
Student setting
Four 9th grade classes from a Swedish compulsory school participated. They worked in teacherconstructed groups (à 2-4 students) with the instruction to discuss and agree on one common
written answer. The written material was also collected. Eleven test-items, (from Acid Rain [S485],
Greenhouse [S114] and Sunscreens [S447]), were selected using criteria of relevance to the age
group and situation and for long-term use in PISA (see Serder & Jakobsson, 2014).
Analytic procedure
The PISA documents were read through several times in order to conduct a discourse analysis
(Kendall & Wickham, 1999) concerned with how science, science knowers, scientific literacy and
performance was put forward in the documents.
2
The transcribed video-material of student discussions was analyzed with a focus on how students
interacted with the testing material in situations that were categorized as “lack of knowledge”. It
centered specific attributes of the test that could be seen to ‘prompt’ low performance (no
answers/answers that would rend no score) or experience of low performance.
Expected results from analysis
The documents stress life-long learning, including competencies such as “willingness to engage”,
“flexibility”, “problem solving” and “adaptation” (e.g. OECD, 2006, p. 10-11).
Science seems capable of solving all problems, creating a truth of ‘science and technology’ (ST) as
making people take rational decisions, based on scientific arguments. The competent student
shares incontestable, scientific values (OECD, 2006, p. 20).
Further, ST contributes to all levels of humanity. Implicitly, there are decisions that are ‘right’
(scientifically informed) while others are ‘wrong’ (e.g. socially, ethically or economically
informed). The future citizen is incessantly facing challenges that demand scientific competence,
who is willing to engage, and who is rational and always acting within this discourse of truth (see
e.g. OECD, 2006, s. 24). The student “at risk” (OECD, 2010, p. 29) has no interest in science in
this particular manner and does not privilege ST; and would actually, within this discourse, have a
lack.
Explicitly, low performers are described with statements such as “lack the skills needed to
function effectively (…) who fail to reach the baseline level of performance (…) therefore risk
facing difficulties (…) throughout their lives”. (OECD, 2013, p. 254).
Concerning the attributes of the detector (Knorr Cetina, 1999; Latour, 1999) single words
become decisive for the possibility or not to construe answers at the test. Words such as
“reference”, “pattern”, “step”, “factor”, “be found” etc. (see Serder & Jakobsson, 2014) are
discussed with different meanings in different groups. Therefore, attributes of the test-items can
be discussed in relation to students’ non-participation, blaming themselves, questioning of the
emerging image of science and answering ‘nonsensical’.
Rickard: the words are so strange…
Jana: what?
Rickard: ”reference substance”…
Jana: yeah…like, what is that?
Cornelia: (looking out through the window)
(silent reading)
Rickard: if only one knew what reference means…?
(Excerpt, Group L, Nov 2010)
Intent of publication
This work is part of a doctoral thesis in progress, intended to be defended in January 2015.
Hopefully this text in an elaborated version will be published within the thesis.
3
References
Ekström, A. (2013). Broadcast from press conference of the Swedish PISA results. 2013-12-03
Foucault, M. (1971). L’ordre du discours. Editions Gallimard.
Gorur, R. (2010). ANT on the PISA Trail: Following the statistical pursuit of certainty.
Educational Philosophy and Theory, 43(1), 76-93.
Kendhall, G. & Wickham, G. (1999). Using Foucault Methods. London: SAGE.
Knorr-Cetina, K. (1999). Epistemic Cultures. How the sciences make knowledge. Cambridge,
Massachussetts: Harvard University Press.
Latour, B. (1999). Pandora’s Hope: essays on the reality of science studies. Cambridge, Mass.: Harvard
University Press.
OECD (2013). PISA 2012 Results: What Students Know and Can Do – Student Performance in
Mathematics, Reading and Science (Volume I). OECD Publications.
OECD (2010). Strong performers and successful reformers in education. OECD Publications.
OECD (2007). Competencies for tomorrow’s world. Volume 1. Analysis. OECD Publications.
OECD (2006). Assessing Scientific, Reading and Mathematical Literacy. A Framework for PISA
2006. OECD Publications.
OECD, 1999. Measuring knowledge and skills. A new framework for assessment. OECD Publications.
Pickering, A. (1995). The Mangle of Practice: Time, Agency and Science. University of Chicago Press.
Popkewitz, T. (2011). PISA. In M. A. Pereyra, H.G. Kotthoff & R. Cowen (Eds.), PISA Under
Examination: Changing Knowledge, Changing Tests, and Changing Schools (pp. 31-46). Sense
Publishers.
Serder, M. & Jakobsson, A. (2014), “why bother so incredibly much?” Student perspectives on
PISA science assignments. Cultural Studies of Science Education. DOI: 10.1007/s11422-0139550-3
4