Teachers’ Knowledge of Error
Analysis
School of Education, University of the Witwatersrand
Funded by the Gauteng Department of Education
9 January 2012
Updated June 2013
Teachers’ knowledge of Error Analysis
School of Education, University of the Witwatersrand
Funded by the Gauteng Department of Education
School of Education
University of the Witwatersrand
27 St Andrews’ Road
Parktown
Johannesburg 2193
http://www.wits.ac.za/academic/humanities/education/l
© School of Education, University of the Witwatersrand
This work is licensed under a Creative Commons Attribution-NonCommercial 2.5 South Africa License.
Permission is granted under a Creative Commons Attribution-NonCommercial license to replicate, copy, distribute,
transmit, or adapt this work for non-commercial purposes only provided that attribution is provided as illustrated in the
citation below.
To view a copy of this licence visit http://creativecommons.org/licenses/by-nc/2.5/za/deed.en .
Citation:
Shalem, Y., & Sapire, I. 2012. Teachers’ Knowledge of Error Analysis. Johannesburg: Saide.
Contents page
Contents page ........................................................................... 5
Tables ......................................................................................... 7
Acknowledgements ................................................................. 9
Introduction ............................................................................ 11
Report Plan .............................................................................. 15
Section One: Teacher knowledge of mathematical errors
.................................................................................................... 19
Section two: Activity and Process ....................................... 29
2.1. The Activity .....................................................................................30
Section three: Evaluation analysis methodology ............ 37
3.1 Aim of the evaluation.....................................................................38
3.2 Items evaluated ...............................................................................38
3.3 Evaluation Instrument ...................................................................39
3.3.1 The coding template (see Appendix 4): ........................................................ 40
3.4 Training of coders and coding process .......................................44
3.5 Validity and reliability check .......................................................44
3.6 Data analysis ....................................................................................44
Section Four: Analysis of data ............................................. 47
4.1 An overview of the comparison of Round 1 and Round 2 .....48
4.2. Analysis of individual criteria ....................................................49
4.2.1 Procedural explanations .................................................................................. 49
4.2.2 Conceptual explanations ................................................................................. 55
4.2.3 Awareness of Mathematical Error ................................................................. 61
4.2.4 Diagnostic reasoning ....................................................................................... 68
4.2.5 Multiple explanations ..................................................................................... 75
4.2.6 Use of the everyday in explanations of the error ....................................... 84
4.2.7 Comparative strengths and weaknesses between the two sets of
grouped grades .......................................................................................................... 90
4.2.8 Comparative strengths and weaknesses according to content area ........ 93
Section Five: General findings ............................................ 95
5.1 What are the findings about teachers reasoning about
learners’ errors? .....................................................................................96
5.2 Teachers’ experiences of the error analysis activity.................97
5.3 Findings from the quantitative analysis ..................................100
5.4 Summary .........................................................................................109
Section Six: Implications for professional development:
developing diagnostic judgement .................................... 113
Recommendations for professional development and for
further research..................................................................... 123
Recommendations for professional development ............................................ 123
Recommendations for further research............................................................... 123
References .............................................................................. 125
Appendices: ........................................................................... 129
Appendix 1................................................................................................................ 130
Appendix 2................................................................................................................ 134
Appendix 3................................................................................................................ 137
Appendix 4................................................................................................................ 140
Appendix 5................................................................................................................ 142
Appendix 6................................................................................................................ 145
Appendix 7................................................................................................................ 150
Appendix 8................................................................................................................ 155
Appendix 9................................................................................................................ 159
Appendix 10.............................................................................................................. 164
Appendix 11.............................................................................................................. 168
6
Tables
Table 1: Domains of teacher knowledge and related error analysis categories ................. 21
Table 2 “Error-focused activities” and “error-related activities” ........................................ 30
Table 3: Round 1 Error analysis ................................................................................................ 32
Table 4: Round 2 Curriculum Mapping and Error Analysis ................................................ 33
Table 5: Number of items analysed by groups (Rounds 1 and 2)........................................ 35
Table 6: Sample summary ......................................................................................................... 38
Table 7: Grades 3-6 procedural explanations demonstrated in teacher text explanations50
Table 8: Grades 7-9 procedural explanations demonstrated in teacher text explanations51
Table 9: Procedural explanations demonstrated in teacher test explanations ................... 52
Table 10: Grades 3-6 conceptual explanations demonstrated in teacher text explanations
....................................................................................................................................................... 55
Table 11: Grades 7-9 conceptual explanations demonstrated in teacher text explanations
....................................................................................................................................................... 56
Table 12: Conceptual explanations demonstrated in teacher test explanations ................ 58
Table 13: Grades 3-6 awareness of mathematical error demonstrated in teacher text
explanations ................................................................................................................................ 62
Table 14: Grades 7-9 awareness of mathematical error demonstrated in teacher text
explanations ................................................................................................................................ 63
Table 15: Awareness of the mathematical error demonstrated in teacher test explanations
....................................................................................................................................................... 65
Table 16: Grades 3-6 diagnostic reasoning demonstrated in teacher text explanations ... 68
Table 17: Grades 7-9 diagnostic reasoning demonstrated in teacher text explanations ... 69
Table 18: Diagnostic reasoning in relation to the error demonstrated in teacher test
explanations ................................................................................................................................ 71
Table 19: Grades 3-6 multiple explanations demonstrated in teacher text explanations . 77
Table 20: Grades 7-9 multiple explanations demonstrated in teacher text explanations . 78
Table 21: Multiple explanations of the error demonstrated in teacher test explanations 81
Table 22: Grades 3-6 use of the everyday demonstrated in teacher text explanations of
the error........................................................................................................................................ 85
Table 23: Grades 7-9 use of the everyday demonstrated in teacher text explanations of
the error........................................................................................................................................ 86
Table 24: Use of the everyday in explanations of the error demonstrated in teacher test
explanations ................................................................................................................................ 88
Table 25: Content areas according to strength and weakness in explanations .................. 93
Table 26: An example of an explanation that is accurate but incomplete ........................ 110
Table 27: Grade 5 test item explanations illustrating poor diagnostic judgement .......... 111
Table A1: Category descriptors for “procedural explanations” ........................................ 145
Table A2: Criterion 1 – Procedural explanation of the choice of the correct solution in
relation to one item................................................................................................................... 146
7
Table A3: Criterion 1 – Procedural explanation of the choice of the correct solution in
relation to a range of items ...................................................................................................... 147
Table A4: Category descriptors for “conceptual explanations”......................................... 150
Table A5: Criterion 2 – Conceptual explanation of the choice of the correct solution in
relation to one item................................................................................................................... 151
Table A6: Criterion 2 – Conceptual explanation of the choice of the correct solution in
relation to a range of items ...................................................................................................... 152
Table A7: Category descriptors for “awareness of mathematical error”.......................... 155
Table A8: Criterion 3 – Awareness of the error embedded in the incorrect solution in
relation to one item................................................................................................................... 155
Table A9: Criterion 3 – Awareness of the error embedded in the incorrect solution in
relation to a range of items ...................................................................................................... 157
Table A10: Category descriptors for “diagnostic reasoning”............................................. 159
Table A11: Criterion 4 – Diagnostic reasoning of learner when selecting the incorrect
solution in relation to one item ............................................................................................... 160
Table A12: Criterion 4 – Diagnostic reasoning of learner when selecting the incorrect
solution in relation to a range of items .................................................................................. 161
Table A13: Category descriptors for “use of everyday” ..................................................... 164
Table A14: Use of the everyday exemplars ........................................................................... 165
Table A15: Category descriptors for “multiple explanations” ........................................... 168
Table A16: Multiple explanations exemplars ....................................................................... 168
8
Acknowledgements
This report was prepared by:
Professor Yael Shalem (Wits School of Education)
Ingrid Sapire (Wits School of Education)
Tessa Welch (Saide)
Maryla Bialobrzeska (Saide)
Liora Hellmann (Saide)
We would like to thank the following people for their role in the preparation of the
report:
M Alejandra Sorto (Maths Education Consultant)
David Merand (Maths Education Consultant)
Gareth Roberts (Statistical computation and analysis)
Carola Steinberg (Editing)
The project team for DIPIP Phases 1 and 2
Project Director: Prof Yael Shalem
Project Leader: Prof Karin Brodie
Project coordinators: Lynne Manson, Nico Molefe, Ingrid Sapire
The project team and evaluators would like to thank the Gauteng Department of
Education and in particular Reena Rampersad and Prem Govender for their support of
the project.
We would also like to thank all of the teachers, departmental subject facilitators, Wits
academic staff and students and the project administrators (Karen Clohessy, Shalati
Mabunda and team) who participated over the three years of the project.
9
10
Introduction
New imperatives held in high regard in South Africa are the growing emphasis on
learners’ performance as an indicator of accountability and the use of assessment to
inform learners’ learning and to monitor teachers’ work. Current policy is very clear
about the accountability imperative. The Department of Education has stipulated that
teachers and district officials monitor learners’ performance continuously and write
reports on their progress. It has also conducted and participated in a range of regional
and international evaluations, amongst them: the Trends in International Mathematics
and Science Study (TIMSS: 1995, 1999, 2003); Systemic Evaluation for grades 3 and 6 in
Numeracy and Literacy; participation in Southern and Eastern Africa Consortium for
Monitoring Educational Quality (SACMEQ: 2003,2007), the Progress in International
Reading Literacy Study (PIRLS), and the most recent one being the Annual National
Assessment (ANA). As part of the regional evaluation exercise, the Gauteng
Department of Education used the ICAS test (International Competitions and
Assessments for Schools). The ICAS test is an international standardized, primarily
multiple-choice test, which was administered in Gauteng to a population of about 55 000
learners in public and private schools in Grades 3 to 11 in 2006, 2007 and 2008. The tests
and data of learners’ errors on the 2006 and 2007 tests of Grade 3-9 provided the starting
point for teacher engagement with learners’ errors, the main subject of this report.
The data from standardised evaluations has been used by mathematical experts,
economists and statisticians at a systemic level and solely for benchmarking.1 Results
have been used to monitor the process of educational reform.2 In this process the
common messages heard by the public in general and by teachers in particular are that
“high participation” has come at the expense of “poor quality”; that the system
demonstrates low value and poor accountability; and that South Africa performs poorly
in comparison to countries in Africa poorer than her. Teachers were blamed and
shamed for their low subject matter content knowledge, and poor professionalism. The
general view that has emerged is that good and committed teachers are pivotal for the
desired change. As important as they are, these kind of messages have done very little
to change the results, and certainly contributed to teachers’ low morale and loss of
public confidence in the capacity of education to deliver a meaningful change (Taylor,
2008; Shalem & Hoadley, 2009; Van den Berg, 2011). Important for our task here, is that
although the Department of Education (and the public) expect teachers to use the results
In South Africa, matric exams, Grade 3 and 6 systemic evaluation and international and regional tests have been used to
benchmark the performance of the system or as a tool which, by controlling social and economic factors that affect
learners’ performance, reports to the public on learners’ standards of proficiently (van Den Berg & Louw 2006; van den
Berg & Shepherd; 2008; Taylor 2007, 2008).
1
2
In South Africa since 1994 there have been three curriculum changes to which teachers and learners have had to adjust.
11
to improve their practice accordingly, to date teachers have not been invited to interpret
the results nor been shown how to integrate them into their practice. Teachers have not
been given the opportunity to develop skills to interpret results from systemic
evaluations. This is despite the current rhetoric of alignment between standardised
assessment and classroom teaching. With the advent of the Annual National
Assessments (ANA), there have been new kinds of pronouncements. For example, the
Department of Basic Education is reported to have stated the following:
ANA is intended to provide regular, well-timed, valid and credible data on pupil
achievement in the education system. Assessment of pupils’ performance in the GET
Band (Grades 1- 9) has previously been done at school level. Unlike examinations that
are designed to inform decisions on learner promotion and progression, ANA data is
meant to be used for both diagnostic purposes at individual learner level and decision-making
purposes at systemic level. At the individual learner level, the ANA results will provide
teachers with empirical evidence on what the learner can and/or cannot do at a particular
stage or grade and do so at the beginning of the school year. Schools will inform parents
of their child’s ANA performance in March 2011 (our emphasis). 3
The above statement includes two important claims, which together suggest an
approach that attempts to go beyond the current use of standardized assessment for
benchmarking purposes. The first claim is that assessment should now be dealt with on
two levels and in two forms: at a school level, through the use of specific and localised
forms of assessment and at a national level, through the use of general and standardised
forms of assessment. This is not a new claim and is in line with the Department of
Education policy of formative assessment. It suggests that worthwhile assessment data
should consist of evidence that is valid for specific learning contexts (or schools)
together with reliable and generalizable evidence that represents different learning
contexts (or schools). In assessment argot, worthwhile assessment data consist of
external assessment and classroom assessment (Gipps, 1989; Brookhart et al, 2009;
Looney, 2011)4. The second claim, which is particularly relevant for this report, is new
and suggests a new direction in the Department of Education policy. This claim
suggests that data from external assessment is intended to be used diagnostically both at
the level of the individual learner and at the systemic level of a grade cohort. The idea
of informing local knowledge using a systemic set of evidence, diagnostically, is not
sufficiently theorized. Teachers have always needed to recognise learners’ errors, a skill
without which they would not have been able to assess learners’ work. The difference
now is that teachers are required “to interpret their own learners’ performance in
national (and other) assessments” (DBE and DHET, 2011, p2) and develop better lessons
on the basis of these interpretations. According to the above statements, the tests will
Media statement issued by the Department of Basic Education on the Annual National Assessments (ANA): 04 February
2011. http://www.education.gov.za/Newsroom/MediaReleases/tabid/347/ctl/Details/mid/1389/ItemID/3148/Default.aspx
3
Gipps argues that externally administered standardised assessments (systematic assessment) are considered more
reliable, in terms of their process (design and marking). Classroom assessment, if well designed and on-going, provides a
variety of contextually-relevant data.
4
12
provide teachers with information that will tell them, in broad strokes, how close or far
their learners are from national and/or international standards. In other words, teachers
will be given a set of “evaluation data” in the form of pass rates and proficiency scores.
This type of information is commonly known as “criterion-referenced proficiency
classification” (Looney, 2011, p27). What is not clear and has not been researched is
what is involved on the part of teachers in interpreting this data: what do teachers do
and what should teachers be doing so that the interpretation process is productive? How
does teachers’ tacit knowledge about learners’ errors (which they have acquired from
years of marking homework and tests, encountering learners’ errors in teaching, or
through hearing about errors from their colleagues) inform or misinform their reasoning
about evaluation data? In what ways should teachers work with proficiency information
when they plan lessons, when they teach or when they design assessment task?
Important to remember is that curriculum statements about assessment standards,
recurring standardised assessments, or reports on pass rates and proficiency levels do
not, in themselves, make standards clear (let alone bring about a change in practice).
Reported data of standardized assessment provides information about what learners can
or can’t do but does not analyse what the learners do not understand, how that may
affect their poor performance, and how preferred instructional practice could afford or
constrain addressing these difficulties (Shepard, 2009, p37). These are ideas that
teachers need to interpret from the evaluation data. Hence Katz et al are correct when
they say:
Data don’t “tell” us anything; they are benign... The meaning that comes from
data comes from interpretation, and interpretation is a human endeavour that
involves a mix of insights from evidence and the tacit knowledge that the
group brings to the discussion. (2009, p.28) 5
These kinds of questions, stated above, inform our approach to teachers’ learning about
their practice through engagement with error analysis. They motivated our professional
development work in the Data-Informed Practice Improvement Project. The DataInformed Practice Improvement Project (henceforth, the DIPIP project) has been the first
attempt in South Africa to include teachers in a systematic way in a process of
interpretation of 55, 000 learners’ performance on a standardized test. The three-year
research and development programme6 included 62 mathematics teachers from Grades 3
to 9 from a variety of Johannesburg schools. The project involved teachers in analysing
learners’ errors on multiple choice items of the ICAS mathematics test. In the project
For Hargreaves (2010), focussing teachers’ learning from data is important for building collegiality. He argues that the
future of collegiality may best be addressed by (inter alia) taking professional discussion and dialogue out of the privacy
of the classroom and basing it on visible public evidence and data of teachers’ performance and practices, such as shared
samples of student work or public presentations of student performance data (p.524).
6 There is currently a third phase of DIPIP which is located in certain schools following a very similar process with teacher
groups in these schools.
5
13
teachers mapped the ICAS test’s items onto the curriculum, analysed learners’ errors,
designed lessons, taught and reflected on their instructional practices, constructed test
items, all in a format of “professional learning communities”.7 With regard to its focus
on working with learners’ errors in evaluation data, the work done with teachers in the
DIPIP project is an exception. Its main challenge has been to design a set of meaningful
opportunities for teachers to reason about the evaluation data collated from the results of
55,000 learners’ who wrote the test, in each of the years that it ran (2006-2008). The
central conceptual questions we addressed in researching the results of this process are:



What does the idea of teachers’ interpreting learner performance diagnostically, mean
in a context of a standardized assessment test?
What do teachers, in fact, do when they interpret learners’ errors?
In what ways can what they do be mapped on the domains of teacher knowledge?
In the field of mathematics, Prediger (2010) uses the notion of “diagnostic competence”
to distinguish reasoning about learners’ errors from merely grading their answers:
The notion of diagnostic competence (that, in English, might have some medical
connotations) is used for conceptualizing a teacher’s competence to analyse and
understand student thinking and learning processes without immediately grading them.
(p76)
Prediger (2010) argues that teachers seem to be better interpreters of learners’ errors
when they have an interest in learners’ rationality, are aware of approaches to learning,
interrogate meanings of concepts (in contrast to just knowing their definitions), and
have studied domain specific mathematical knowledge. His view is consistent with
Shepard’s work on formative assessment, in particular with the idea of using insights
from student work, formatively, to adjust instruction (2009, p34). Research in
mathematics education has shown that a focus on errors, as evidence of reasonable and
interesting mathematical thinking on the part of learners, helps teachers to understand
learner thinking, to adjust the ways they engage with learners in the classroom situation,
as well as to revise their teaching approach (Borasi, 1994; Nesher, 1987; Smith, DiSessa &
Roschelle, 1993). The field of Maths education has developed various classifications of
learners’ errors (Radatz, 1979; Brodie & Berger, 2010) but there is hardly any work on
what kinds of criteria can be used to assess teacher knowledge of error analysis. There is
no literature that examines the ways teachers’ reason about evaluation data. There is
hardly any literature that shows what would be involved in assessing the quality of
teachers’ reasoning about evaluation data gathered from a systematic assessment.8
See Shalem, Sapire, Welch, Bialobrzeska, & Hellman, 2011. See also Brodie & Shalem, 2010.
Peng and Luo (2009) and Peng (2010) present one attempt to classify the tasks teachers engage with when they analyse
learners’ errors (identifying, addressing, diagnosing, and correcting errors).
7
8
14
In this report we hope to show a way of assessing teachers’ knowledge of error analysis
in relation to specific criteria and mathematical content. Our central claim is that
without teachers’ being able to develop this kind of knowledge the “empirical evidence
on what the learner can and/or cannot do at a particular stage or grade”,9 will remain
non-integrated with teacher practice.
Report Plan
Section one
This is the conceptual section of the report. The conceptual aim of this report is to locate
the idea of error analysis in teacher knowledge research, in order to develop criteria for
assessing what it is that teachers do when they interpret evaluation data. We develop
the idea of interpreting evaluation data diagnostically and show a way of mapping its
six constitutive aspects against the domains of teacher knowledge, as put forward by
Ball et al (2005, 2008) and Hill et al (2008a). By locating these aspects in the broader
discussion of teacher knowledge, we are able to demonstrate the specialisation involved
in the process of teachers’ reasoning about learners errors. We use the word ‘mapping’
consciously, because we believe that the task of error analysis requires both subject
matter knowledge and pedagogical content knowledge.
Section two
In this section we describe the error analysis activity, in the context of the DIPIP project.
We draw a distinction between different error analysis activities, between “errorfocused activities” and “error-related activities”. This distinction structures the
sequence of activities in the DIPIP project. We describe the time-line of the project, as
some of the activities were repeated twice and even three times throughout its threeyear duration. In this report we only examine the error analysis activity, which forms
the central focus of the “error-focused activities” and which was done twice. We refer to
this time-line as Round 1 and Round 2 and use it to compare teachers’ reasoning about
learners’ errors across these two rounds. The main difference between the two rounds is
that in Round 1 the teachers worked in small groups led by a group leader, while in
Round 2 the small groups worked without a group leader. In comparing the two
rounds, we are able to suggest some ideas about the role of group leaders in this kind of
professional development project. In this section we also describe the ICAS test we used
for the error analysis activity and we provide information about the number of items
analysed by the groups in Round 1 and 2 as well as the training provided for the group
leaders.
Media statement issued by the Department of Basic Education on the Annual National Assessments (ANA): 04 February
2011. http://www.education.gov.za/Newsroom/MediaReleases/tabid/347/ctl/Details/mid/1389/ItemID/3148/Default.aspx
9
15
Section three
In this section we describe the methodology we followed for the evaluation of the
findings of the error analysis activity. We state the three central aims of the evaluation
of the error analysis activity. We then describe how we selected the items used as data
for the evaluation, the evaluation instrument, the coding template used to code the data,
the training of the coders and the coding process, including its validity and reliability.
We conclude with the description of the three-level approach we followed to analyse the
data.
Section four
This section presents what we call the level 1 analysis of findings. In this section we
detail the findings for each of the six criteria we selected for the analysis of teachers’
reasoning about the correct and the erroneous answers. This section of the report brings
us to the empirical aim of the report, which is to measure and analyze the quality of
teachers’ ability to do diagnostic error analysis, across the constitutive aspects of
interpreting evaluation data. The empirical analysis intends to answer the following
three empirical questions:



On what criteria is the groups’ interpretation weaker and on what criteria is the
groups’ interpretation stronger?
In which mathematical content areas do teachers produce better judgement on
the errors?
Is there a difference in the above between two sets of grouped grades (Grade 3-6
and Grade 7-9)?
Section five
This section presents what we call the level 2 analysis of findings. In this section we
begin to draw together broader findings across criteria and groups. We present several
findings that help us construct an argument about the relationship between subject
matter knowledge and pedagogical content knowledge as it pertains to error analysis.
We also present findings that compare the two grouped grades (Grade 3-6 and Grade 79) and describe the role of the group leaders. We conclude with some implications for
the idea of interpreting evaluation data diagnostically.
Section six
This section presents what we call the level 3 analysis of findings. In terms of the
professional development aspect of the report, the discussion in this section begins with
the ways in which teachers could be involved in analysing evaluation data. Then, based
on the findings of the error analysis activity and the mapping of error analysis against
16
the domains of teacher knowledge, we conclude with a short description of “diagnostic
judgment”. Diagnostic judgement is the construct we propose for describing what
teachers do when they interpret evaluation data. We argue that understanding teachers’
professionalism in this way may help the discourse to shift away from the dominance of
“accounting” to “accountability”.
We conclude with lessons to be learned from the project and recommendations both for
professional development and research.
17
18
Section One: Teacher knowledge of mathematical
errors
19
No matter how the South African curriculum debate of the last 10 years will be resolved
or the change away from OBE to a content-based curriculum is enacted, no matter what
educational philosophy informs our debate on good teaching, the research on teacher
knowledge insists that teachers today are expected (or should be able) to make sound
judgments on sequence, pacing and evaluative criteria so as to understand learners’
reasoning and to inform learners’ learning progression (Muller, 2006; Rusznyak, 2011;
Shalem & Slonimsky, 2011).
The central question that frames studies on teacher knowledge, in the field of
mathematics education, goes as follows: “is there a professional knowledge of
mathematics for teaching which is tailored to the work teachers do with curriculum
materials, instruction and students?” (Ball, Hill & Bass, 2005, p16). This question draws
on Shulman’s work, specifically, on his attempt to define “knowledge-in-use in
teaching”. Shulman’s main project was to situate subject matter knowledge within the
broader typology of professional knowledge.
Continuing within this tradition Ball, Thames and Phelps (2008) elaborate Shulman’s
notion of pedagogical content knowledge and its relation to subject matter knowledge
for mathematics teaching. Ball, Thames and Phelps define mathematical knowledge for
teaching as “the mathematical knowledge needed to carry out the work of teaching
mathematics” (p395). In order to investigate what this in fact means, they collected
extensive records of specific episodes, analyses of curriculum materials, and examples of
student work. They also draw on Ball’s personal experience of teaching and researching
in a Grade 3 mathematics class for a year. On the basis of these resources, they then
developed over 250 multiple-choice items “designed to measure teachers’ common and
specialized mathematical knowledge for teaching” (Ball, Hill & Bass 2005, p43). Some of
the items emphasise mathematical reasoning alone and others include the more
specialized knowledge for teaching (Hill, Ball & Schilling, 2008, p376; Ball, Hill & Bass
2005, pp 22 and 43). They then conducted large-scale surveys of thousands of practicing
teachers as well as interviews with smaller number of teachers. They wanted to know
“what ‘average’ teachers know about students’ mathematical thinking” (Hill, Ball &
Schilling, 2008, p376), whether this specialized knowledge is different from
mathematical reasoning and whether it can be shown to affect the quality of instruction
(Ball et al, 2008b).
Their work is summarised in several papers, showing that teachers’ mathematical
knowledge consists of four key domains (Ball, Hill & Bass, 2005; Ball, Thames & Phelps
2008; Hill, Ball & Schilling, 2008).10 The first two domains elaborate the specialisation of
10
They also include two other domains, Knowledge of Curriculum and Knowledge at the mathematical horizon.
20
subject-matter knowledge (“common content knowledge” and “specialized content
knowledge”). The second two domains elaborate the specialisation involved in teaching
mathematics from the perceptive of students, curriculum and pedagogy. These domains
(“knowledge of content and students” and “knowledge of content and teaching”)
elaborate Shulman’s notion of “pedagogical content knowledge”. 11
Although there is some debate on the specificity of the domains of knowledge in the
realm of teachers’ knowledge in mathematics and other authors have expanded on them
in a number of academic papers (e.g. Adler, 2005), for the purpose of the DIPIP error
analysis evaluation we use Ball’s classification of the domains and in what follows we
present each of the domains, by foregrounding key aspects of ‘error analysis’ relevant to
each of the domains. We also formulate the criteria that are aligned with the domain and
explain why. The following table presents the way we mapped the evaluation criteria in
relation to teacher knowledge.
Table 1: Domains of teacher knowledge and related error analysis categories
Subject Matter Knowledge
Pedagogical Content Knowledge
Knowledge domain
Knowledge
domain
Knowledge domain
Knowledge domain
Common content
knowledge (CCK)
Specialized content
knowledge (SCK)
Knowledge of content
and students (KCS)
Knowledge of content
and teaching (KCT)
DIPIP category
DIPIP category
DIPIP category
DIPIP category
Procedural
understanding of
correct answers
Awareness of
Errors
Diagnostic reasoning
of learners’ thinking
in relation to errors
N/A (only in lesson
design and
teaching)
Conceptual
understanding of
correct answers
Use of everyday links
in explanations of
errors
Multiple explanations
The principle point behind this typology of the 4 domains is that mathematics teaching is a specialized practice, which
combines mathematical and pedagogical perspectives. The argument behind this work is that a specialized mathematical
perspective includes things like “determining the validity of mathematical arguments or selecting a mathematically
appropriate representation”, (Ball, Thames and Bass, 2008 p398) but also “skills, habits of mind, and insight” (p399) or
mathematical reasoning. The specialized pedagogical perspective on the other hand requires knowledge of curriculum, of
learners and of teaching - it focuses on learners’ reasoning and requires specialised pedagogical tasks.
11
21
of errors
The first domain is common content knowledge, which is general subject matter knowledge
(CCK). Ball, Thames and Bass (2008) define this domain of teacher knowledge as
follows:
Teachers need to know the material they teach; they must recognize when their students
give wrong answers or when the textbook gives an inaccurate definition. When teachers
write on the board, they need to use terms and notation correctly. In short, they must be
able to do the work that they assign their students. But some of this requires
mathematical knowledge and skill that others have as well—thus, it is not special to the
work of teaching. By “common,” however, we do not mean to suggest that everyone has
this knowledge. Rather, we mean to indicate that this is knowledge of a kind used in a
wide variety of settings—in other words, not unique to teaching. (pp. 398-399)
The argument here is that like any other mathematician, a mathematics teacher with
good subject matter knowledge uses mathematical terms correctly, is able to follow a
procedure fully, and can evaluate that a textbook defines a mathematical term correctly
or incorrectly. In terms of error analysis, this knowledge is about recognizing that a
learner’s answer is correct or not. Recognition of errors is a necessary component of
teachers’ content knowledge. It is necessary, as it shows the “boundaries of the practice”
of doing mathematics, of “what is acceptable and what is not” (Brodie, 2011, p66). The
underlying condition, i.e. the pre-condition that enables teachers to recognize error
mathematically, is for teachers to be able to explain the mathematical solutions of the
problem, both procedurally and conceptually. The emphasis in this domain is on
teachers’ ability to explain the correct answer. Recognizing “when their students give
wrong answers or when the textbook gives an inaccurate definition” relies on knowing
the explanation of a solution in full. Without a full knowledge of the explanation,
teachers may recognize the error only partially. “Only partial”, because they may not
know what the crucial steps that make up the solution are or what their sequence needs
to be (procedural explanations). “Only partial”, because they may not know what the
underlying conceptual links are that the student needs to acquire in order not to err.
Although Ball et al argue that there are some aspects of what teachers know which is not
unique to teaching, and although, one would argue, it is not expected that
mathematicians will distinguish procedural from conceptual explanations when they
address a mathematical solution, these two kinds of explanations are essential aspects of
teachers’ content knowledge and are the enablers of recognition of learners’ errors. Star
notes that “it is generally agreed that knowledge of concepts and knowledge of
procedures are positively correlated and that the two are learned in tandem rather than
independently” (2000, p80). The first aspect of mathematical knowledge that teachers
need in order to “carry out the work of teaching mathematics”, specifically of error
22
analysis, is content knowledge. This involves both knowledge of the crucial sequenced
steps needed to get to the correct answer (procedural knowledge) and their conceptual
links (conceptual knowledge) 12. Because this knowledge underlies recognition of error,
we call it “content knowledge” (irrespective of whether it is common or not to
mathematical experts in general). This means we included two criteria under common
content knowledge.
Criteria for recognizing Common Content Knowledge:
Procedural understanding
The emphasis of the criterion is on the quality of the teachers’ procedural explanations
when discussing the solution to a mathematical problem. Teaching mathematics
involves a great deal of procedural explanation which should be done fully and
accurately for the learners to grasp and become competent in working with the
procedures themselves.
Conceptual understanding
The emphasis of the criterion is on the quality of the teachers’ conceptual links made in
their explanations when discussing the solution to a mathematical problem. Teaching
mathematics involves conceptual explanations which should be made with as many
links as possible and in such a way that concepts can be generalised by learners and
applied correctly in a variety of contexts.
The second domain is Specialised Content Knowledge (SCK), which is mathematical
knowledge specific to teaching and which, according to Ball et al, general
mathematicians do not need. A different aspect of error analysis is located in this
domain, which involves activities such as,“looking for patterns in student errors or …
sizing up whether a nonstandard approach would work in general” (Ball, Thames &
Bass, 2008, p400). Whereas teacher knowledge of the full explanation of the correct
answer enables a teacher to spot the error, teacher knowledge of mathematical
knowledge for teaching enables a teacher to interpret a learner’s solution and evaluate
its plausibility, by recognizing the missing steps and /or conceptual links and taking into
account, we would add, other factors such as the age of the learner, or the time the
learner was given to complete the task, the complexity in the design of the question etc.
Notwithstanding our addition, the main idea about “sizing up whether a nonstandard
approach would work in general”, assumes that teachers recognize errors relationally.
Some mathematical problems lend themselves more to procedural explanations while in others the procedural and the
conceptual are more closely linked. There is a progression in mathematical concepts – so that what may be conceptual for
a grade 3 learner (for example, basic addition of single digit numbers) is procedural for a grade 9 learner who will have
progressed to operations at a higher level.
12
23
They evaluate the “nonstandard approach” and/or “the error” in relation to what is
generally considered the correct approach, taking into account the context of the
“nonstandard approach” and/or the error. In Ball et al’s words, knowledge of this
domain enables teachers to “size up the source of a mathematical error” (Ball, Thames &
Bass, 2008, p397) and identify what mathematical step/s would produce a particular
error.
Error analysis is a common practice among mathematicians in the course of their own
work; the task in teaching differs only in that it focuses on the errors produced by
learners… Teachers confront all kinds of student solutions. They have to figure out what
students have done, whether the thinking is mathematically correct for the problem, and
whether the approach would work in general. (ibid)13
It is important to emphasise that although, as Ball et al define above, the knowledge of
errors in this domain focuses on what “students have done”, teachers’ reasoning about
the error is mathematical (i.e. not pedagogical) in the main. In their analysis it forms the
second domain of teachers’ content knowledge:
deciding whether a method or procedure would work in general requires mathematical
knowledge and skill, not knowledge of students or teaching. It is a form of mathematical
problem solving used in the work of teaching. Likewise, determining the validity of a
mathematical argument, or selecting a mathematically appropriate representation,
requires mathematical knowledge and skill important for teaching yet not entailing
knowledge of students or teaching. (p398)
Ball, Thames and Bass characterize this type of knowledge as “decompressed
mathematical knowledge” (2008b, p400) which a teacher uses when s/he unpacks a topic
for the learner or make “features of particular content visible to and learned by
students” (ibid, see also Prediger, 2000, p79).
Teaching about place value, for example, requires understanding the place-value system
in a self-conscious way that goes beyond the kind of tacit understanding of place value
needed by most people. Teachers, however, must be able to talk explicitly about how
mathematical language is used (e.g., how the mathematical meaning of edge is different
from the everyday reference to the edge of a table); how to choose, make, and use
mathematical representations effectively (e.g., recognizing advantages and
disadvantages of using rectangles or circles to compare fractions); and how to explain
and justify one’s mathematical ideas (e.g., why you invert and multiply to divide
fractions). All of these are examples of ways in which teachers work with mathematics in
its decompressed or unpacked form. (ibid)
So, for example (Ball, 2011, Presentation), when marking their learners’ work, teachers need to judge and be able to
explain to the learners which definition of a concept (‘rectangle’, in the following example) is more accurate:
13



a rectangle is a figure with four straight sides, two long and two shorter
a rectangle is a shape with exactly four connected straight line segments meeting at right angles
a rectangle is flat, and has four straight line segments, four square corners, and it is closed all the way around.
24
In relation to Shulman’s distinction between subject matter knowledge and pedagogical
content knowledge, Ball, Thames and Bass, (2008) argue that the above two domains are
framed, primarily, by subject matter knowledge. This is very important from the
perspective of examining teachers’ interpretation of evaluation data. As Peng and Luo
(2009) argue, if teachers identify learner’s errors but interpret them with wrong
mathematical knowledge, their evaluation of student performance or their plan for a
teaching intervention are both meaningless. In other words, the tasks that teachers
engage with, in error analysis, such as sizing up the error or interpreting the source of its
production, are possible because of the mathematical reasoning that these domains of
teacher knowledge equip them with. The idea behind this is that strong mathematics
teachers recruit content knowledge into an analysis of a teaching situation and do so by
recruiting their “mathematical reasoning” more than their knowledge of students,
teaching or curriculum. This mathematical reasoning enables strong mathematics
teachers to size up the error from the perspective of socializing others into the general
field of mathematics. We included one criterion under Specialized Content Knowledge.
Criteria for recognizing Specialized Content Knowledge
Awareness of error
This criterion focuses on teachers’ explanations of the actual mathematical error and not
on learners’ reasoning. The emphasis in the criterion is on the mathematical quality of
teachers’ explanations of the actual mathematical error.
The third domain of Knowledge of Content and Students orients teachers’ mathematical
perspective to the kind of knowing typical of learners of different ages and social
contexts in specific mathematical topics. Teachers develop this orientation from their
teaching experience and from specialized educational knowledge of typical
misconceptions that learners develop when they learn specific topics. Ball, Thames and
Bass (2008) state as examples of this domain of teacher knowledge, “the kinds of shapes
young students are likely to identify as triangles, the likelihood that they may write 405
for 45, and problems where confusion between area and perimeter lead to erroneous
answers” (p401). This knowledge also includes common misinterpretation of specific
topics or levels of the development in representing a mathematical construct (e.g. van
Hiele levels of development of geometric thinking). Teacher knowledge in this domain
contains “knowledge that observant teachers might glean from working with students,
but that have not been codified in the literature” (Hill, Ball & Schilling, 2008, p378). So
for example, after years of teaching, teachers gain an understanding of how to define a
mathematical concept that is both accurate and appropriate to Grade 2 learners, they
come to know what aspect of the definition of a mathematical concept is more difficult
for these learners, and how the learners’ everyday knowledge (of the particular age
25
group) can come in the way of acquiring the specialised mathematical knowledge (Ball,
2011).
From the point of error analysis, this knowledge domain involves knowing specific
mathematical content from the perspective of how learners typically learn the topic or
“the mistakes or misconceptions that commonly arise during the process” of learning
the topic (Hill, Ball & Schilling, 2008, p375). Ball, Thames and Bass (2008) emphasise
that this type of knowledge builds on the above two specialized domains but yet it is
distinct:
… Recognizing a wrong answer is common content knowledge (CCK), whereas sizing up
the nature of an error, especially an unfamiliar error, typically requires nimbleness in
thinking about numbers, attention to patterns, and flexible thinking about meaning in
ways that are distinctive of specialized content knowledge (SCK). In contrast, familiarity
with common errors and deciding which of several errors students are most likely to
make are examples of knowledge of content and students (KCS). (p401)
The knowledge of this domain enables teachers to explain and provide a rationale for
the way the learners were reasoning when they produced the error. Since it is focused
on learners’ reasoning, it includes, we argue, the ability to provide multiple explanations
of the error. Because contexts of learning (such as age and social background) affect
understanding and because in some topics the learning develops through initial
misconceptions, teachers will need to develop a repertoire of explanations, with a view
to addressing differences in the classroom.
How is this knowledge about error analysis different from the first two domains? The
first two domains, being subject-matter based, combine knowledge of the correct
solution, (which includes both a knowledge of the procedure to be taken as well as the
underlying concept, “Common Content Knowledge”) with knowledge about errors
(“Specialized Content Knowledge” which is focused on the relation between the error /
the non-standard solution and the correct answer / the answer that is generally
accepted). The social context of making the error is secondary to the analysis of the
mathematical content knowledge involved in explanting the solution or the error and
therefore is back-grounded. “Knowledge of content and students”, the third domain, is
predominantly concerned with context-specific experiential knowledge about error that
teachers develop, drawing on their knowledge of students (Hill et al, 2008b, p 385).14
Hill et al discuss the difficulties they found in measuring KCS. They believe that “logically, teachers must be able to
examine and interpret the mathematics behind student errors prior to invoking knowledge of how students went astray”
(2008b, p390). They found that teachers’ mathematical knowledge and reasoning (first and second domains) compensate
when their knowledge of content and student is weak. This has implications for their attempt to measure the
distinctiveness of KCS, which is beyond the scope of this report.
14
26
The diagnostic aspect of this domain is focused on learner reasoning and the general
mathematical content knowledge of the correct answer is secondary and therefore is
back-grounded. We included three criteria under Knowledge of Content and Students.
Criteria for recognizing Knowledge of Content and Students
Diagnostic reasoning
The idea of error analysis goes beyond identifying the actual mathematical error
(“awareness of error”). The idea is to understand how teachers go beyond the
mathematical error and follow the way learners were reasoning when they produced the
error. The emphasis in this criterion is on the quality of the teachers’ attempt to provide
a rationale for how learners were reasoning mathematically when they chose a
distracter.
Use of everyday knowledge
Teachers sometimes explain why learners make mathematical errors by appealing to
everyday experiences that learners draw on and/or confuse with the mathematical
context of the question. The emphasis in this criterion is on the quality of the use of
everyday knowledge in the explanation of the error, judged by the links made to the
mathematical understanding that the teachers attempt to advance.
Multiple explanations of error
One of the challenges in the teaching of mathematics is that learners need to hear more
than one explanation of the error. This is because some explanations are more accurate
or more accessible than others and errors may need to be explained in different ways for
different learners. This criterion examines the teachers’ ability to offer alternative
explanations of the error when they are engaging with learners’ errors.
Knowledge of Content and Teaching (KCT) is the fourth domain of teacher knowledge.
This type of knowledge links between subject matter knowledge content (CCK+SCK)
and knowledge about instruction, which takes into account knowledge about students
(KCS). Based on their knowledge of these three domains, teachers use their knowledge
of teaching to decide on the sequence and pacing of lesson content or on things such as
which learner’s contribution to take up and which to ignore. This domain of knowledge
includes “knowledge of teaching moves”, such as “how best to build on student
mathematical thinking or how to remedy student errors” (Hill et al, 2008b, p378). As
this domain is concerned with teachers’ actively teaching it falls outside the scope of this
report.
27
It is worth noting that the four domains follow a sequence. The idea of a sequence
between the first two subject knowledge-based domains and the second two
pedagogical content knowledge-based domains is important for its implication for
teacher development of error analysis. It suggests that the first three domains, when
contextualized in approaches to teaching and instruction, equip teachers to design their
teaching environment. It is only when teachers have learnt to understand patterns of
error, to evaluate non-standard solutions, or to unpack a procedure, that they will be
able to anticipate errors in their teaching or in their assessment, and prepare for these in
advance. Research (Heritage et al, 2009) suggests that the move from error analysis to
lesson planning is very difficult for teachers. The sequence of the domains in this model
suggests the importance of knowledge about error analysis for developing mathematical
knowledge for teaching.
28
Section two: Activity and Process
29
2.1. The Activity
DIPIP engaged the teachers in six activities. These can be divided into two types: errorfocused activities and error-related activities. Error-focused activities engaged the teachers
directly in error analysis. Error-related activities engaged teachers in activities that built
on the error analysis but were focused on learning and teaching more broadly.
Table 2 “Error-focused activities” and “error-related activities”
Error-focused activities
Error-related activities

Analysis of learner results on the ICAS
mathematics tests (with a focus on the
multiple choice questions and the
possible reasons behind the errors that
led to learner choices of the
distractors)15
Analysis of learners’ errors on tests
that were designed by the teachers

An interview with one learner to probe
his/her mathematical reasoning in
relation to errors made in the test.




Mapping of ICAS test items in relation
to the South African mathematics
curriculum;
Development of lesson plans which
engaged with learners’ errors in
relation to two mathematical concepts
(equal sign; visualisation and problem
solving)
Teaching the lesson/s and reflecting on
one’s teaching in "small grade-level
groups” and presenting it to “large
groups16
In the course of Phases 1 and 2 of the DIPIP project, the error analysis activity followed
after the curriculum mapping activity17 and structured a context for further professional
conversations among teachers about the ICAS and other test data. Teachers used the
tests to analyse the correct answers and also the errors embedded in the distractors
presented in the multiple choice options and the incorrect solutions given by learners on
the open tests.
“Distractors” are the three or four incorrect answers in multiple choice test items. They are designed to be close enough
to the correct answer to ‘distract’ the person answering the question.
15
The small groups consisted of a group leader (a mathematics specialist – Wits School of Education staff member or post
graduate student who could contribute knowledge from outside the workplace), a Gauteng Department of Education
(GDE) mathematics subject facilitator/advisor and two or three mathematics teachers (from the same grade but from
different schools). This meant that the groups were structured to include different authorities and different kinds of
knowledge bases. These were called small grade-level groups (or groups). As professional learning communities, the
groups worked together for a long period of time (weekly meetings during term time at the Wits Education Campus for
up to three years), sharing ideas and learning from each other and exposing their practice to each other. In these close knit
communities, teachers worked collaboratively on curriculum mapping, error analysis, lesson and interview planning, test
setting and reflection. For certain tasks (such as presenting lesson plans, video clips of lessons taught or video clips of
learner interviews) the groups were asked to present to large groups. A large group consisted of the grade-level groups
coming together into larger combined groups, each consisting of four to six small groups (henceforth the large groups).
This further expanded the opportunities for learning across traditional boundaries. (See Shalem, Sapire, Welch,
Bialobrzeska, & Hellman, 2011, pp.5-6)
16
17
See the full Curriculum Mapping report, Shalem, Y., & Sapire, I. (2011)
30
2.2 The ICAS test
To provide a basis for systematic analysis of learners’ errors, the results of Gauteng
learners on an international, standardized, multiple-choice test, the ICAS test, were
used. The International Competitions and Assessments for Schools (ICAS) test is
conducted by Educational Assessment Australia (EAA), University of New South Wales
(UNSW) Global Pty Limited. Students from over 20 countries in Asia, Africa, Europe,
the Pacific and the USA participate in ICAS each year. EAA produces ICAS papers that
test students in a range of subject areas including Mathematics. Certain schools in
Gauteng province, both public and private schools, used the ICAS tests in 2006, 2007 and
2008, and it was the results of these learners on the 2006 and 2007 tests that provided the
starting point for teacher engagement with learners’ errors. The ICAS test includes
multiple choice and some open items. The Grade 3-6 tests consist of 40 multiple choice
questions and the Grade 6-11 tests consist of 35 multiple choice items and 5 open
questions.
2.3 Time line
The first round of the error analysis activity ran for ten weeks, from July 2008 to October
2008. In this round, which we refer to as Round 1, the teachers analysed data from the
ICAS 2006 tests. As in the curriculum mapping activity, the groups worked on either
the even or odd numbered items (the same items for which they had completed the
curriculum mapping activity). The second round of error analysis ran for five weeks in
September and October 2010. In this round, which we refer to as Round 2, the teachers
in their groups analysed data from the ICAS 2007 tests as well as tests which they had
set themselves. The number of items analysed in Round 2 had to be cut down because of
time limits.
2.4 The Process
Round 1 error analysis gave the teachers the opportunity in their groups to
discuss the mathematical reasoning that is required to select the correct option in
the ICAS 2006 multiple choice test items, as well as to provide explanations for
learners’ choices of each of the distractors (incorrect answers). In order to
deepen the teachers’ conceptual understanding and appreciation of learners’
errors, the groups had to provide several explanations for the choices learners
made. This was intended to develop a more differentiated understanding of
reasons underlying learners’ errors amongst the teachers. The focus of the error
analysis in this Round was on the multiple choice items because the test
designers provided a statistical analysis of learner responses for these, which
served as the starting point for teachers’ analysis. Some groups also discussed
31
the open ended items. Groups had to analyse either odd or even numbered
items, as they had done in the curriculum mapping activity. Some groups
analysed more items than others, but all completed the analysis of the 20
odd/even numbered items they were assigned. Table 3 below18, lists the material
given to the groups, the error analysis task and the number and types of groups
(small or large or both) in which the teachers worked in Round 1 error analysis.
Table 3: Round 1 Error analysis
Material
Tasks
Group type
Exemplar templates
(See Appendix 1)
Group leader training:
Group leaders
with project
team
Guidelines for the
completion of the error
analysis template
 Discussion of error analysis of selected
items to generate completed template for
the groups to work with.
ICAS achievement
results for Gauteng
Grading items by difficulty level. Our
method for grading item difficulty was as
follows: The achievement stats for learners,
who wrote the ICAS tests in the Gauteng
sample were entered onto excel sheets and
items were graded from the least to the most
difficult, based on this data. The items were
thus all assigned a “difficulty level” from 1 to
40. 19
Project team
Test items and answer
key with learner
performance data (per
item) from the 2006
ICAS test.
Error analysis of learner performance using
template:
14 small
grade-specific
groups
Statistical analysis of
learner performance
for correct answer.
 Identification of the mathematical
reasoning behind the correct response
 Provision of possible explanations for
learners’ choice of each of the three
distracters.
3 distractors
Error analysis
template. (see
18
For more detail see Shalem, Sapire, Welch, Bialobrzeska, & Hellman, 2011
Out initial plan was to only work on a selected number of tests items (ICAS 2006). We were planning to choose the
items that proved to be most difficult for the learners who wrote the test at the time (55,000 learners in Gauteng from both
public and independent schools). At the time of the start of the project (last quarter of 2007), the EAA’s Rasch analysis of
each of the tests items was not available.
19
32
Material
Tasks
Group type
Appendix 2)
In Round 2 Error analysis eleven20 of the groups repeated the error analysis and
curriculum mapping activities on selected ICAS 2007 items or on their own tests21, but
without the facilitation of the group leader. This was intended to establish how
effectively the experiences of Round 1 error analysis and other project activities had
enabled the teachers, in their groups, to work independently. Table 4 below22, lists the
material given to the groups, the error analysis task and the number and types of group
in which the teachers worked in Round 2 error analysis.
Table 4: Round 2 Curriculum Mapping and Error Analysis
Material
Tasks
Group type
ICAS 2007 test items.
Error analysis of ICAS 2007 tests.
Error analysis
template with
curriculum alignment
built into the template
The groups were assigned 12 items for
analysis. Not all the small groups completed
the analysis of these items. The small groups
conducted the same analysis as they did for
the ICAS 2006 test items.
6 groups (One
of each of
Grades 3, 4, 5,
6, 7, and 9)
Error analysis
template including
curriculum mapping,
which was modified
in order to
accommodate analysis
of “own tests” (see
Appendix 3)23
Error analysis of “own tests”. As with the
ICAS, the groups were asked
5 groups on
own test items
 to pool together the results of all the
learners in their class and to work out the
achievement statistics on the test
(One of each of
Grades 3, 4, 5,
6 and 8)
 to place the results on a scale and on that
basis judge the difficulty of each item
 to analyse the ways in which the learners
got the correct answer to each of the
questions as well as the ways the learners
got each of the items wrong.
Round 2 error analysis was completed by 11 of the 14 teacher groups since three small groups did a third round of
teaching in the last phase of the project.
20
21
See Shalem, Sapire, Welch, Bialobrzeska, & Hellman, 2011 on the process and the design of “own tests”
22
For more detail see Shalem, Sapire, Welch, Bialobrzeska, & Hellman, 2011.
Due to time constraints, as well as to the fact that a further round of teaching was not going to follow this exercise, the
revised template did not include a question on whether and how teachers taught the concept underlying the test item.
23
33
Material
Tasks
Group type
Verbal instructions to
groups
Error analysis presentations:
Three Large
groups
All groups were requested to present the
error analysis of one item to the larger
groups.
Group leaders were invited to attend the
presentation.
Grades 3 and
4, Grades 5
and 6 and
Grades 7 and
9.
2.5 Group leader training
A plenary meeting with group leaders for the error analysis mapping activity was held
prior to the commencement of the small group sessions. The session was led by the
project leader, Prof Karin Brodie, with input taken from all group leaders.
When the error analysis activity was presented to the groups, there was a briefing
session where all groups sat together, before they spilt into their smaller groups and
started on the activity. Project coordinators moved around and answered questions
while groups worked. The groups were encouraged to complete templates in full and
settled queries where possible. When the first templates were returned, groups were
given feedback and asked to add/amend their templates as necessary – to try for even
better completion of templates. Some of the group discussions were recorded24. After
the activity was completed, we conducted two large group de-briefing discussions (one
with the Grade 3, 4, 5 and 6 groups and one with the Grades 7, 8 and 9 groups). These
debriefing sessions were recorded. This report is based on the written templates, not on
the recordings.
2.6 Item analysis
The error analysis component of the DIPIP project resulted in a large number of
completed templates in which groups had recorded their analysis of the items. This
involved analysis of both correct and incorrect solutions.
Table 5 summarises the total number of items analysed by the groups during Round 1
and 2. Groups are called “odd” and “even” according to the test item numbers assigned
to them for initial curriculum mapping in Round 1 (see Column Two). Column Three
details the number of ICAS 2006 items analysed by the groups in Round 1. Column
Four details the number of ICAS 2007 / Own tests analysed by the groups in Round 2.
24
Three groups were recorded in Round 1 and eleven groups were recorded in Round 2.
34
Table 5: Number of items analysed by groups (Rounds 1 and 2)
Grade
Numbers
Round 1
Round 2
(ICAS 2006 test)
(ICAS 2007 test/Own test)
3
Odd
17
Own test (6)
3
Even
16
ICAS (10)
4
Odd
20
Own test (5)
4
Even
20
ICAS (11)
5
Odd
20
Own test (6)
5
Even
20
ICAS (4)
6
Odd
20
Own test (4)
6
Even
20
ICAS (11)
7
Odd
20
Not done
7
Even
20
ICAS (8)
8
Odd
11
Own test (6)
8
Even
13
Not done
9
Odd
16
ICAS (11)
9
Even
17
Not done
250 items
82 items
Total
Round 1 error analysis produced a much bigger set of data (250 items) than Round 2 (82
items). In Round 1 certain groups analysed more items than others. The expectation for
error analysis of items in Round 1 was a minimum of 20 items per group. No group
completed more than 20 ICAS 2006 items, although some groups assisted other groups
with editing and improving their analysis when they had completed the 20 items
allocated to their group. Grade 6 – 9 ICAS tests include open ended questions. Some of
the groups chose not to analyse these items. This can be seen in the Grade 8 and 9
groups that mapped less than 20 items. Neither of the Grade 3 groups completed the
mapping of all 20 even/odd items. The expectation for error analysis of items in Round
2 was a minimum of 12 ICAS 2007 items and 6 “own test” items per group. No group
completed all the 12 ICAS 2007 items assigned to it. Three of the six groups completed
11 items. Three of the six groups that analysed their own test completed all the 6
questions.
35
In summary, in Round 2 only 11 of the 14 groups participated and they analysed fewer
items due to time constraints. Some groups in Round 2 analysed the ICAS 2007 test
while others analysed tests that they had set themselves. The membership of most of the
groups that continued across two rounds of error analysis remained. The format of the
error analysis was the same for both rounds. Groups received feedback in both rounds.
The main difference between rounds was that group leaders were not present in Round
1. In between the two rounds all of the groups were involved in lesson design, teaching
and reflection where learners’ errors played a significant role in planning, teaching and
reflecting on teaching. It was intended that in coming back to error analysis in Round 2
the groups would be more familiar with the idea of error analysis both in teaching and
related activities. In view of the differences between the two rounds comparisons
between rounds are possible, although they should be taken tentatively. The comparison
does imply any inferences about the impact of Round 1 activities on those of Round 2.
36
Section three: Evaluation analysis methodology
37
3.1 Aim of the evaluation
We aim to evaluate the quality of error analysis done by the groups. In this we aim to
evaluate the quality of teachers reasoning about error, evident in the group texts (both in
the “correct analysis texts” and in the “error analysis texts”) and the value of the group
leaders in leading the process.
The empirical analysis intends to answer the following four empirical questions:




On what criteria is the groups’ error analysis weaker and on what criteria is the
groups’ error analysis stronger?
In which mathematical content areas do the groups produce better judgements on
the error?
Is there a difference in the above between primary (Grade 3-6 groups) and high
school (Grade 7-9 groups) teachers?
What does the change in performance between Round 1 and 2 suggest about the role
of group leaders?
3.2 Items evaluated
For the purpose of analysis, we selected a sample of ten ICAS 2006 items per grade per
group for Round 1. These ten items (per group) were made up of five items from the
first half of the test and five items from the second half of the test. Since fewer items
were mapped in Round 2 mapping activity, all the items mapped by groups that
participated in Round 2 were analysed. Only one distractor (or incorrect answer) per
item was selected for the evaluation analysis. The selected distractors were the ones that
were most highly selected according to the test data, in other words, the “most popular”
incorrect learner choice. Table 6 below summarises the sample of data used in the
evaluation analysis of Rounds 1 and 2 error analyses:
Table 6: Sample summary
Full set of data to be
considered
Sample
Round 1:
ICAS 2006 items: 10 items selected per group, hence 20
items selected per grade. The selected items were chosen
to represent the curriculum.
Completed error analysis
templates based on ICAS
2006 test data
(Grades 3-9, 14 groups)
Groups: All groups Grades 3-9 (14), 140 items in sample.
All texts related to the correct answer and the chosen
distractor of the selected items formed the data which
was analysed – from Round 1 there were 572 texts coded
(316 answer texts and 252 error texts).
38
Full set of data to be
considered
Sample
Round 2:
ICAS 2007: Some groups Grades 3-9 (6), mapped items
selected per group, various item numbers coded per
grade.
Completed error analysis
templates based on ICAS
2007 test as well as data
from tests which groups
had developed “own test”
data
“Own tests”: Some groups Grades 3-9 (5), mapped items
from tests that they had set as groups for the learner
interview activity, various item numbers coded per
grade.
82 items in sample.
(Grades 3-9, 11 groups)
All texts related to the correct answer and the chosen
distractor of the selected items formed the data which
was analysed – from Round 2 there were 284 texts coded
(173 answer texts and 111 error texts).
The sample size in Round 2 was smaller than the sample size in Round 1. In both rounds
the number of answer texts was higher than the number of error texts.
3.3 Evaluation Instrument
The initial evaluation criteria and their corresponding categories were drawn up into a
template by three members of the project team. The template was then used to code the
full data sample, piecemeal, while on-going discussions were held to refine the criteria
over a four month period. Agreement was reached after the coding of four Round 1
grades was completed and the final format of the template and wording of the criteria
and categories was constructed. See more on the template below.
Two external coders, one expert in the field of maths teacher education and one an
experienced mathematician with experience in teacher education, coded the groups’
texts.
The coders were given group’s texts; to code. A group’s text is divided to two texts
types:


“Correct answer texts” or the explanation provided by the group for the correct
answer.
“Error answer texts” or the explanations provided by the group for the distractor
selected for the evaluation.
39
Groups provided more than one explanation for the correct answer or for the selected
distractor for some items. This meant that some items had one correct answer text and
one error text per the distractor while other items may have had several correct answer
texts and/or several error for the distractor. The coders were given all of the texts written
by the groups in explanation of the correct answer and of the selected distractor (error).
The texts were arranged for the coders for each item in a large excel coding sheet so that
all coders worked according to the same identification of texts. Coders were asked to
give a code to each text. In this way a code was assigned to each of the texts for each of
the items selected for the evaluation.
3.3.1 The coding template (see Appendix 4):
A coding template was prepared for the coders so that all coding related to the same
texts, and all the texts are arranged in the same format. The coding template included
general information about the group text, six columns of criteria and their
corresponding categories, and a final comment column.
One template per grade was prepared, with all the texts inserted into the template next
to the relevant item. The coding template for the ICAS test items consists of all the items
and all the texts (correct answer and error answer texts) associated with the item. The
coding template for “own test” was the same as that for the ICAS tests. The only
difference is that for the “own tests” template the actual incorrect answer to be discussed
was pasted into the “Distractor” column since it could not be represented by a key.
Instrument overview
General information on group texts:






Grade – indicates the grade of the item texts.
Item – indicates the item number for which the groups produced the correct answer
and the error answer texts.
Maths Content area - indicates the curriculum area (number, pattern/algebra, space,
measurement or data handling) of the item.
Distractor – indicates which distractor had been selected for analysis.
Text number – indicates the number of the text to be coded. Numbering started from
one for each correct answer text and then from one again for each error text.
Text – refers to the actual text as recorded by the groups in their completed
templates. The texts were pasted into the coding template, verbatim. Texts that could
not be pasted into the coding template were sent to coders in a separate document
attachment and referred to in the templates by grade and number.
40
Criterion code columns:








In addition to the above information, the key aspect of the template is the criteria
and their relevant categories. The templates consist of six columns, which are related
to the six evaluation criteria selected. Each criterion is further divided into four
categories (“not present”, “inaccurate”, “partial” and “full”) that capture the quality
of the correct and the error answer text, in terms of teachers’ reasoning about the
correct answer and the distractor (or error in “own texts”). For purpose of simplicity
each category was coded with a number (4 assigned to “full” and 1 to “not present”).
Coders had to insert their coding decisions for each of the four categories into each
column, for each identified text.
Procedural explanations – a code of 1 to 4 assigned to each text, which correspond to
the four categories. This criterion measures the fullness and the correctness of the
groups’ description of the procedure to be followed in order to answer the question.
Conceptual explanations – a code of 1 to 4 assigned to each text, which correspond
to the four categories. This criterion measures the fullness and the correctness of the
groups’ description of the conceptual links that should be made in order to answer
the question.
Awareness of error – a code of 1 to 4 assigned to each text, which correspond to the
four categories. This criterion measures general knowledge of the mathematical
error.
Diagnostic reasoning – a code of 1 to 4 assigned to each text, which correspond to
the four categories. This criterion measures groups’ explanation of learner’ reasoning
behind the error.
Use of everyday – a code of 1 to 4 assigned to each text, which correspond to the
four categories. This criterion measure groups use of everyday knowledge about
learner’ reasoning.
Multiple explanations – a code of “n” or “f” assigned25 to each text from which
codes of 1 to 4 could be assigned to “multiple explanations” for each text. This
criterion measure the feasibility of the alternative explanations the groups offered to
explain leaners reasoning behind the error.
Comment – The final comment allowed coders to write any particular comments
they wished to note about the answer or error text in that row. These comments
could be used in the alignment discussion and evaluation.
The coders inserted their codes according to the given column heading for criterion
coding. A wide range of exemplars are provided in the appendices in order to
demonstrate the operationalization of the criteria (see Appendices 6 – 11.). Exemplar
items were chosen from Round 1 and Round 2 and in such a way that a spread of grades
“n” was assigned to a particular text when it was considered not mathematically feasible in relation to the error under
discussion and “f “was assigned when the explanation was considered mathematically feasible. The final code for
“multiple explanations” was decided on the number of feasible explanations given of the chosen distractor.
25
41
is represented with both strong and weak explanations. Items were also selected so that
there are exemplars across all five of the mathematical content areas represented in the
curriculum.
The criteria
Whilst Section One offers explanations for the choice and the meaning of the criteria, we
offer a brief explanation in this section too. A wide range of exemplars are provided in
the appendices in order to demonstrate the operationalization of the criteria. Exemplar
items were chosen from Round 1 and Round 2 and in such a way that a spread of grades
is represented with both strong and weak explanations. Items were also selected so that
there are exemplars across all five of the mathematical content areas represented in the
curriculum. In what follows we provide a brief description of each criterion and the
Appendix number which includes the full wording of each criterion and its
corresponding four categories. For the full set of the six criteria and their categories see
Appendix 5
Answer Text Criteria
The first two criteria were used to code the groups’ explanations of the correct answers.
Every “correct answer text” was thus coded according to two criteria: procedural and
conceptual.
Procedural explanations
The literature emphasises that the quality of teachers’ explanations depends on the
balance they achieve between explaining the procedure required for addressing a
mathematical question and the mathematical concepts underlying the procedure. This
criterion aims to grade the quality of the teachers’ procedural explanations of the correct
answer. The emphasis in the criterion is on the quality of the teachers’ procedural
explanations when discussing the solution to a mathematical problem through engaging
with learner test data. Teaching mathematics involves a great deal of procedural
explanation which should be done fully and accurately for the learners to grasp and
become competent in working with the procedures themselves (see Appendix 6).
Conceptual explanations
The emphasis in this criterion is on the conceptual links made by the teachers in their
explanations of the learners’ mathematical reasoning in relation to the correct answer.
Mathematical procedures need to be unpacked and linked to the concepts to which they
relate in order for learners to understand the mathematics embedded in the procedure.
The emphasis of the criterion is on the quality of the teachers’ conceptual links made in
42
their explanations when discussing the solution to a mathematical problem through
engaging with learner test data (see Appendix 7).
Error Text Criteria
The next four criteria were used to code the groups’ explanations of the distractor. These
explanations are about teachers’ engagement with the errors. Four criteria were selected:
“awareness of error”, “diagnostic reasoning”, “use of everyday knowledge” and
“multiple explanations”. Every “error answer text” was coded according to these four
criteria.
Awareness of error
The emphasis in this criterion is on the teachers’ explanations of the actual mathematical
error (and not on the learners’ reasoning). The emphasis in the criterion is on the
mathematical quality of the teachers’ explanations of the actual mathematical error
when discussing the solution to a mathematical problem (see Appendix 8).
Diagnostic Reasoning
The idea of error analysis goes beyond identifying the actual mathematical error. The
idea is to understand how teachers go beyond the mathematical error and follow the
way learners were reasoning when they made the error. The emphasis in the criterion is
on the quality of the teachers’ attempt to provide a rationale for how learners were
reasoning mathematically when they chose a distractor (see Appendix 9).
Use of the everyday
Teachers often explain why learners make mathematical errors by appealing to
everyday experiences that learners draw on and may confuse with the mathematical
context of the question. The emphasis in this criterion is on the quality of the use of
everyday knowledge, judged by the links made to the mathematical understanding that
the teachers attempt to advance (see Appendix 10).
Multiple explanations
One of the challenges in the teaching of mathematics is that learners need to hear more
than one explanation of the error. This is because some explanations are more accurate
or more accessible than others and errors need to be explained in different ways for
different learners. This criterion examines the teachers’ ability to offer alternative
explanations of the error when they are engaging with learners’ errors through analysis
of learner test data (see Appendix 11).
43
3.4 Training of coders and coding process
Coders started by coding one full set of Grade 3 texts, after an initial discussion of the
coding criteria and template. The evaluation analysis team discussed the assigned Grade
3 codes with the coders. After this and more general discussions about the format and
the criterion, the final format of the coding template was agreed on. This format was to
include all texts pasted into the template, so that there could be no confusion as to which
part of the teachers’ explanations the coders were referring when they allocated codes.
Discussion meetings with the evaluation analysis team were held after each set of coding
(per grade) was completed. The discussions allowed for refinement of the wording of
the criteria – and when refinements were made, coders reviewed their completed grades
in order to be sure that all coding was in line with the criteria.
Coding continued per grade, with discussion after the completion of each grade for
Grades 4, 5 and 6. After each full set of codes for a grade was completed by both coders,
the percentages of alignment between coders was calculated and sent to coders. The
alignment differences were used to guide the above discussions and the evaluation
coordinator gave comments as to which areas the coders should focus on and think
more carefully about in order to improve on alignment. After the fourth set of codes was
completed and discussed it was decided that there was sufficient agreement between the
two coders for them to continue with the remaining sets of codes and the process set was
in motion. All remaining templates were prepared and emailed to coders.
3.5 Validity and reliability check
The full set of codes was completed and then reviewed by the evaluation analysis team.
Consensus discussions between the coders were held on certain items in order to hone
agreement between them. The final set of codes used in the analysis was agreed on in
discussion with and through arbitration by a third expert (a member of the evaluation
analysis team). The alignment between coders was 57% before the review, 71% after the
review and 100% after the arbitration.
3.6 Data analysis
The coded data were analysed quantitatively, finding observable trends and
relationships evident in the sample. The content of the templates was entered into excel
spread sheets to facilitate the overall analysis of the findings and comparison of the
various criteria. Data were summarised for descriptive analysis and t-tests were done to
44
establish whether there were any significant changes between the error analysis in
Rounds 1 and 2. An unpaired t-test assuming an unequal variance (since the sample
sizes were different) was used to test for significance. This was considered to be the most
conservative measure and appropriate for use for our data set. Correlations between
codes for the answer texts (procedural and conceptual explanations) and error texts
(awareness of error and diagnostic reasoning) were calculated using Pearson’s r
coefficient. The analysis follows three levels:
Level one: For each of the six criteria, we grouped the data into two sets of groupedgrades (Grade 3-6 and Grade 7-9). This enabled us to discern patterns in performance
across the 14 small groups, between primary and secondary teachers (see Section four).
For each of the six criteria we provide the following:



Graphs and tables comparing the findings for two sets of grouped grades (Grade 3-6
and Grade 7-9 groups) between Rounds 1 and 2.
Graphical representations of the overall findings across all groups.
Graphs representing the findings for mathematical content areas.
Level two: We generate broad findings across two or more criteria and about the total 14
groups analysed (see Section five). These findings enabled us to see patterns in teacher
knowledge of error analysis, divided along subject matter and pedagogical content
knowledge. This type of analysis also enabled us to compare the overall performance
across the criteria between the two sets of grouped grades (Grade 3-6 and Grade 7-9).
Lastly the analysis enables us to show the importance of guided instruction by group
leaders, for a successful process of accountability conversations, albeit with some
differentiation across the criteria. With these findings we are able to then construct a
broad description of what is involved in teachers “diagnostic judgment”.
Level three: We draw on the finding to generate a broad description (working towards a
conceptual definition) of what we mean by “diagnostic judgment” in teachers reasoning
about error (see Section five).
45
46
Section Four: Analysis of data
47
4.1 An overview of the comparison of Round 1 and Round 2
An overview of the comparison of Round 1 and Round 2 shows immediately that the
groups worked to a higher level in Round 1 when they had leaders. This overall
impression of quality differences between Rounds 1 and 2 shows a drop in the
percentage of texts demonstrating “full” explanations on every criterion between Round
1 and 2 and an increase in the percentage of texts demonstrating “not present”
explanations on every criterion between Round 1 and 2.
Figure 1: Round 1 – overall codes assigned on all six criteria
Round 1 Number Spread of Codes
100
90
80
70
60
50
40
30
20
10
0
Not present
Inaccurate
Partial
Full
Figure 2: Round 2 – overall codes assigned on all six criteria
Round 2 Overall Spread of codes
100
90
80
70
60
50
40
30
20
10
0
Not present
Inaccurate
Subject Matter Knowledge
Multiple
Everyday
Diagnostic
Awareness
Conceptual
Procedural
Partial
Full
Pedagogical Content
Knowledge
48
The correlation between procedural explanations and conceptual explanations was high
in both rounds, although it decreased in Round 2 (r = 0,74 in Round 1 and r = 0,66 in
Round 2). The correlation between awareness of the mathematical error and diagnostic
reasoning was also high and increased in Round 2 (r = 0,651 in Round 1 and r = 0,71 in
Round 2)26.
The downwards trend and the increases and decreases in texts demonstrating different
levels of the criteria will now be analysed individually for each of the criteria.
4.2. Analysis of individual criteria
We now discuss the criteria separately, focusing on grouped grades (3-6 and 7-9) and
mathematical content for each of the six criteria. We first note our observation separately
for each of the grouped grades, and then for the all of the 14 groups together.
4.2.1 Procedural explanations
Figure 3: Procedural explanations of answers – Round 1 and 2 by grouped grades
Procedural explanations Round 1 and Round 2 by grade
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Grade 3-6
Grade 7-9
Round 1 Round 2 Round 1 Round 2 Round 1 Round 2 Round 1 Round 2
Procedural
explanation not
given
Procedural
explanation
inaccurate or
incomplete
Procedural
Procedural
explanation correct
explanation
but missing some accurate and full
steps
Since these correlations were high, exemplars of explanations which received the same level coding are given in
Appendices 6, 7, 8 and 9, in addition to other exemplar texts.
26
49
Analysis of procedural explanations in Grade 3-6 small groups’ texts:
Table 7: Grades 3-6 procedural explanations demonstrated in teacher text explanations
Change
Round Round
Strength of
Procedural Explanations
between
Round 1 Round 2
1
2
explanation
rounds
Procedural explanation
not given
3%
17%
14%***
Procedural explanation
inaccurate or incomplete
8%
2%
-6%
Procedural explanation
correct but missing some
steps
46%
61%
15%*
Procedural explanation
accurate and full
43%
20%
Weaker
11%
19%
Stronger
89%
81%
-23%***
*** Difference significant at a 99% level of confidence
* Difference significant at a 90% level of confidence
Observations about groups’ procedural explanations in relation to correct answers in
Rounds 1 and 2 for the grade 3-6 groups:
1. The number of texts of the Grade 3-6 group that demonstrate weak procedural
explanations increased slightly between Rounds 1 and 2 (from 11% in Round 1 to
19% in Round 2).
2. The number of texts where no attempt was made to give procedural explanations
increased by 14% from 3% in Round 1 to 17% of texts in Round 2.
3. The number of texts where procedural explanations were inaccurate or incomplete
decreased by 6% (from 8% in Round 1 to 2% in Round 2).
4. The number of texts of the Grade 3-6 group that demonstrate strong procedural
explanations slightly decreased by 8% in Round 2 (from 89% in Round 1 to 81% in
Round 2).
5. The gap between incomplete and systematic explanations was not very high in
Round 1 but it grew much wider in Round 2 (from 3% in Round 1 to 41% in Round
2).
50
Analysis of procedural explanations in Grade 7-9 small groups’ texts:
Table 8: Grades 7-9 procedural explanations demonstrated in teacher text explanations
Change
Round Round
Strength of
Procedural Explanations
between
Round 1
Round 2
1
2
explanation
rounds
Procedural explanation
not given
3%
9%
6%
Procedural explanation
inaccurate or incomplete
8%
10%
2%
Procedural explanation
correct but missing some
steps
43%
56%
13%**
Procedural explanation
accurate and full
46%
26%
Weaker
11%
19%
Stronger
89%
81%27
20%***
*** Difference significant at a 99% level of confidence
** Difference significant at a 95% level of confidence
Observations about groups’ procedural explanations in relation to correct answers in
Rounds 1 and 2 for the grade 7-9 groups:
1. The number of texts of the grade 7-9 group that demonstrate weak procedural
explanations slightly increased (from 11% in Round 1 to 19% in Round 2).
2. The number of texts where no attempt was made to give procedural explanations
increased by 6% (from 3% in Round 1 to 9% in Round 2).
3. The number of texts where procedural explanations were inaccurate or incomplete
increased by 2% (from 8% in Round 1 to 10% in Round 2).
4. The number of texts of the Grade 7-9 group that demonstrate strong procedural
explanations decreased by 7% in Round 2 (from 89% in Round 1 to 82% in Round 2).
5. The gap between incomplete and systematic explanations was not very high in
Round 1 (3%) but grew wider in Round 2 (31%).
Rounding of Round 1 and 2 percentages is done individually but Round totals are added and then rounded hence there
may be small discrepancies such as the one highlighted in this table. This occurs in certain table in this report but is only
noted here, the first time such a discrepancy arises.
27
51
Overall findings on the procedural explanations of the correct answer – Rounds 1 and
2
Figure 4: Procedural explanations of answers – Round 1 and 2
Procedural explanations - Overall - Round 1 and 2
100%
90%
80%
70%
56%
60%
50%
46%
44%
40%
30%
23%
16%
20%
10%
3%
8%
5%
Round 1
Round 2
0%
Round 1
Round 2
Round 1
Round 2
Round 1
Round 2
Procedural explanation Procedural explanation Procedural explanation Procedural explanation
not given
inaccurate or
correct but missing
accurate and full
incomplete
some steps
Table 9: Procedural explanations demonstrated in teacher test explanations
Change
Round Round
Strength of
Procedural Explanations
between
Round 1
1
2
explanation
rounds
Procedural explanation
not given
3%*
16%*
13%
Procedural explanation
inaccurate or incomplete
8%
5%
-3%
Procedural explanation
correct but missing some
steps
44%
57%
13%***
Procedural explanation
accurate and full
46%
23%
Round 2
Weaker
11%
20%
Stronger
89%
80%
-16%***
*** Difference significant at a 99% level of confidence
Observations about groups’ procedural explanations in relation to correct answers in
Rounds 1 and 2:
52
1. The overall number of texts that demonstrate weak procedural explanations
increased slightly between Rounds 1 and 2 (from 11% in Round 1 to 20% in Round
2).
2. The number of texts where no attempt was made to give procedural explanations
increased by 13% from 3% in Round 1 to 16% of texts in Round 2. This increase was
significant at 99%.
3. The number of texts where procedural explanations were inaccurate or incomplete
decreased by 3% (from 8% in Round 1 to 5% in Round 2). This decrease was not
significant.
4. The overall number of texts that demonstrate strong procedural explanations slightly
decreased by 9% in Round 2 (from 89% in Round 1 to 80% in Round 2).
5. The gap between incomplete and systematic explanations was not very high in
Round 1 but it grew much wider in Round 2 (from 2% in Round 1 to 34% in Round
2).
6. The decrease between Round 1 and Round 2 in the number of texts with full or
partially complete explanations was significant at 99%.
Procedural explanations, by mathematical content
Number was the content area in which the groups’ procedural explanations of the
correct answer were strongest. Data was the content area in which the groups’
procedural explanations of the correct answer were the weakest. The graphs below
represent the percentages of procedural explanations for these two content areas across
the two rounds.
Figure 5: Round 1 and 2 procedural explanations of answers – content area number
Number Procedural explanations Rounds 1 and 2
100
90
Percentage of texts
80
70
60
50
40
30
20
10
0
Not present
Inaccurate
Partial
Full
Round 1
2.272727273
7.954545455
29.54545455
60.22727273
Round 2
10.9375
1.5625
51.5625
35.9375
53
Change in the strong content area:




The number of texts that demonstrate full and accurate procedural explanations
decreased from 60.23% in Round 1 to 35.94% in Round 2.
The number of texts that demonstrate partial procedural explanations (correct
but, missing some steps) increased from 29.55% in Round 1 to 51.56% in Round
2.
The number of texts in which procedural explanations were inaccurate or
incomplete decreased from 7.95% to 1.56% in Round 2.
The number of texts where procedural explanations were not given increased
from 2.27% in Round 1 to 10.94% in Round 2.
Figure 6: Round 1 and 2 procedural explanations of answers – content area data
Data Procedural explanations Rounds 1 and 2
100
90
Percentage of texts
80
70
60
50
40
30
20
10
0
Not present
Inaccurate
Partial
Full
Round 1
5.882352941
1.960784314
47.05882353
45.09803922
Round 2
25
12.5
56.25
6.25
Change in the weak content area:




The number of texts that demonstrate full and accurate procedural explanations
decreased from 45.10% in Round 1 to 6.25% in Round 2.
The number of texts that demonstrate partial procedural explanations (correct
but, missing some steps) increased from 47.06% in Round 1 to 56.25% in Round
2.
The number of texts that procedural explanation inaccurate or incomplete
increased from 1.96% to 12.5% in Round 2.
The number of texts that did not include procedural explanations increased from
5.88% in Round 1 to 25% in Round 2.
54
In comparison, the trends between the two rounds in these two content areas are similar,
bar the decrease in Round 2 in the number of texts with inaccurate explanation in the
number content area.
4.2.2 Conceptual explanations
Figure 7: Conceptual explanations in explanations – Round 1 and 2 by grouped grades
Conceptual Explanations - Round 1 and Round 2 by grade
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Grade 3-6
Round 1 Round 2 Round 1 Round 2 Round 1 Round 2 Round 1 Round 2
No conceptual links
Explanation
in explanation
includes poorly
conceived
conceptual links
Explanation
includes some but
not all conceptual
links
Grade 7-9
Explanation
includes
conceptual links
that explain
process and
background
Analysis of conceptual explanations in Grade 3-6 small groups’ texts:
Table 10: Grades 3-6 conceptual explanations demonstrated in teacher text explanations
Change
Round Round
Strength of
Round
Conceptual explanations
between
Round 2
1
2
explanation
1
rounds
No conceptual links in
explanation
4%
30%
26%*
Explanation includes
poorly conceived
conceptual links
19%
10%
-9%
Explanation includes
some but not all
conceptual links
39%
49%
10%
Explanation includes
conceptual links that
explain process and
background
38%
11%
Weaker
23%
40%
Stronger
77%
60%
-27%**
55
*** Difference significant at a 99% level of confidence
* Difference significant at a 90% level of confidence
Observations about groups’ conceptual explanations in relation to correct answers in
Rounds 1 and 2 for the grade 3-6 groups:
1. The number of texts of the Grade 3-6 group that demonstrate weak conceptual
explanations increased by 17% between Rounds 1 and 2 (from 23% in Round 1 to
40% in Round 2).
2. The number of texts where no conceptual links were evident increased by 26% from
4% in Round 1 to 30% of texts in Round 2.
3. The number of texts where poorly conceived conceptual links were made decreased
by 9% (from 19% in Round 1 to 10% in Round 2).
4. The number of texts of the Grade 3-6 group that demonstrate strong conceptual
explanations decreased by 17% in Round 2 (from 77% in Round 1 to 60% in Round
2).
5. The gap between explanations that include conceptual links that explain the
background and process of the answer and explanations that include some but not
all of the conceptual links grew wider in Round 2 (from 1% in Round 1 to 39% in
round 2).
Analysis of conceptual explanations in Grade 7-9 small groups’ texts:
Table 11: Grades 7-9 conceptual explanations demonstrated in teacher text explanations
Change
Round Round
Strength of
Conceptual explanations
between
Round 1 Round 2
1
2
explanation
rounds
No conceptual links in
explanation
6%
33%
27%***
Explanation includes
poorly conceived
conceptual links
15%
4%
-11%**
Explanation includes
some but not all
conceptual links
40%
34%
-6%
Explanation includes
conceptual links that
explain process and
background
38%
29%
Weaker
21%
37%
Stronger
79%
63%
-9%
*** Difference significant at a 99% level of confidence
** Difference significant at a 95% level of confidence
56
Observations about groups’ conceptual explanations in relation to correct answers in
Rounds 1 and 2 for the grade 7-9 groups:
1. The number of texts of the Grade 7-9 group that demonstrate weak conceptual
explanations increased (from 21% in Round 1 to 37% in Round 2).
2. The number of texts where no conceptual links were evident increased by 27% from
6% to 33% of texts.
3. The number of texts where poorly conceived conceptual links were made decreased
by 9% (from 15% in Round 1 to 4% in Round 2).
4. The number of texts of the Grade 7-9 group that demonstrate strong conceptual
explanations decreased by 15% in Round 2 (from 78% in Round 1 to 63% in Round
2).
5. The gap between explanations that include conceptual links that explain the
background and process of the answer and explanations that include some but not
all of the conceptual links was not at all high in Round 1 (2%) and 5% in Round 2.
Overall findings on the conceptual explanations of the correct answer – Rounds 1 and
2
Figure 8: Conceptual links in explanations of answers -Round 1 and 2
Conceptual explanations - Overall - Round 1 and 2
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
39%
45%
40%
29%
17%
5%
Round 1
Round 2
Round 1
16%
10%
Round 2
No conceptual links in Explanation includes
explanation
poorly conceived
conceptual links
Round 1
Round 2
Explanation includes
some but not all
conceptual links
Round 1
Round 2
Explanation includes
conceptual links that
explain process and
background
57
Table 12: Conceptual explanations demonstrated in teacher test explanations
Change
Round Round
Strength of
Conceptual explanations
between
Round 1
1
2
explanation
rounds
No conceptual links in
explanation
5%
29%
24%***
Explanation includes
poorly conceived
conceptual links
17%
10%
7%**
Explanation includes some
but not all conceptual
links
39%
45%
6%
Explanation includes
conceptual links that
explain process and
background
40%
16%
Round
2
Weaker
22%
39%
Stronger
78%
61%
-14%***
*** Difference significant at a 99% level of confidence
** Difference significant at a 95% level of confidence
Observations about groups’ conceptual explanations in relation to correct answers in
Rounds 1 and 2:
1. The overall number of texts that demonstrate weak conceptual explanations
increased between Rounds 1 and 2 (from 22% in Round 1 to 39% in Round 2).
2. The number of texts where no attempt was made to give procedural explanations
increased by 14% from 5% in Round 1 to 29% of texts in Round 2. This increase was
significant at 99%.
3. The number of texts where conceptual explanations included poor conceptual links
decreased by 7% (from 17% in Round 1 to 10% in Round 2). This decrease was
significant at 95%.
4. The overall number of texts that demonstrate strong conceptual explanations
decreased by 17% in Round 2 (from 78% in Round 1 to 61% in Round 2).
5. The gap between explanations that include some but not all conceptual links and
conceptual explanations with links that explain the process and background was 9%
in Round 1 but it grew much wider to 29% in Round 2.
6. The decrease between Round 1 and Round 2 in the number of texts with conceptual
explanations with links that explain the process and background was significant at
99%.
58
Conceptual explanations, by mathematical content
Number was the content area in which the groups’ conceptual explanations of the
correct answer are the strongest. Algebra is the content area in which the groups’
conceptual explanations of the correct answer were the weakest. The graphs below
represent the percentages of conceptual explanations for these two content areas across
the two rounds.
Figure 9: Round 1 and 2 conceptual explanations of answers – content area number
Number Conceptual explanations Rounds 1 and 2
100
90
Percentage of texts
80
70
60
50
40
30
20
10
0
Not present
Inaccurate
Partial
Full
Round 1
3.409090909
15.90909091
32.95454545
47.72727273
Round 2
25
9.375
40.625
25
Change in the strong content area:




The number of texts that include conceptual links that explain process and
background decreased from 47.73% in Round 1 to 25% in Round 2.
The number of texts that include some but not all conceptual links increased
from 32.95% in Round 1 to 40% in Round 2.
The number of texts that include poorly conceived conceptual links decreased
from 15.91% in Round 1 to 9.38% in Round 2.
The number of texts that include no conceptual links in explanation increased
from 3.41 in Round 1 to 25% in Round 2.
59
Figure 10: Round 1 and 2 conceptual explanations of answers – content area algebra
Alegbra Conceptual explanations Rounds 1 and 2
100
90
Percentage of texts
80
70
60
50
40
30
20
10
0
Not present
Inaccurate
Partial
Full
Round 1
11.36363636
27.27272727
36.36363636
25
Round 2
41.66666667
4.166666667
45.83333333
8.333333333
Change in the weak area:




The number of texts that include conceptual links that explain process and
background decreased from 25%% in Round 1 to 8.33% in Round 2.
The number of texts that include some but not all conceptual links increased
from 36.36% in Round 1 to 45.83% in Round 2.
The number of texts that include poorly conceived conceptual links decreased
from 27.27% in round 1 to 4.17% in round 2.
The number of texts that include no conceptual links in explanation increased
from 11.36 in Round 1 to 41.67% in round 2.
In comparison, the trends between the two rounds in these two content areas are similar.
Noted is the far higher increase of texts that include no conceptual links in explanation
in the weaker area.
60
4.2.3 Awareness of Mathematical Error
Figure 11: Awareness of error in explanations – Round 1 and 2 by grouped grades
Awareness of Error - Round 1 and Round 2 by grade
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Grade 3-6
Grade 7-9
Round 1 Round 2 Round 1 Round 2 Round 1 Round 2 Round 1 Round 2
Mathematical
Mathematical
explanation of
explanation of
error not present error is flawed or
incomplete
Mathematical
explanation
emphasises the
procedural
Mathematical
explanation
emphasises the
conceptual
Analysis of awareness of error in Grade 3-6 small groups’ texts:
The information in this graph is presented in the table below, classifying groups’
explanation texts according to the above four criteria:
61
Table 13: Grades 3-6 awareness of mathematical error demonstrated in teacher text
explanations
Change
Level of mathematical
Round Round
Strength of
between
Round 1
awareness
1
2
explanation
rounds
In these texts a
mathematical explanation
of the particular error is
not included.
18%
21%
3%
In these texts the
explanation of the
particular error is
mathematically inaccurate
or incomplete and hence
potentially confusing.
12%
4%
-8%*
In these texts the
explanation of the
particular error is
mathematically sound but
does not link to common
misconceptions or errors.
The explanation is
predominantly procedural
39%*
53%*
14%**
In these texts the
explanation of the
particular error is
mathematically sound and
suggests links to common
misconceptions or errors.
30%
21%
Round 2
Weaker
30%
25%
Stronger
69%
75%
9%
** Difference significant at a 95% level of confidence
* Difference significant at a 90% level of confidence
Observations about groups’ awareness of the mathematical error in Rounds 1 and 2 for
the grade 3-6 groups:
1. The number of texts of the Grade 3-6 group that demonstrate weak mathematical
awareness decreased in Round 2 (From 30% in Round 1 to 25% in Round 2).
2. The number of texts with mathematical flaws and or that are potentially confusing
was reduced by 8%, from 12% to 4%.
3. Group 3-6 demonstrates a slight increase in the number of texts that demonstrate
strong awareness (from 69% to 75%).
4. For the texts in Round 2 that demonstrate strong mathematical awareness, the gap
between texts where explanation of the error is predominantly procedural and texts
62
where explanation of error links to common misconceptions of errors grew wider,
from 9% in Round 1 to 32% in Round 2.
Analysis of Grade 7-9 small groups’ texts:
Table 14: Grades 7-9 awareness of mathematical error demonstrated in teacher text
explanations
Change
Level of mathematical
Round Round
Strength of
between
Round 1
awareness
1
2
explanation
rounds
In these texts a
mathematical explanation
of the particular error is
not included.
15%
22%
7%
In these texts the
explanation of the
particular error is
mathematically inaccurate
or incomplete and hence
potentially confusing.
24%
15%
-9%
In these texts the
explanation of the
particular error is
mathematically sound but
does not link to common
misconceptions or errors.
The explanation is
predominantly procedural.
35%
38%
3%
In these texts the
explanation of the
particular error is
mathematically sound and
suggests links to common
misconceptions or errors.
26%
25%
Round 2
Weaker
39%
37%
Stronger
61%
63%
-1%
Observations about groups’ awareness of the mathematical error in Rounds 1 and 2 for
the grade 7-9 groups:
1. The number of texts of the Grade 7-9 group that demonstrate weak mathematical
awareness remained the same in Round 2 (39% in Round 1 and 37% in Round 2).
2. The number of texts with mathematical flaws or that are potentially confusing was
reduced in Round 2 by 9% (from 24% in Round 1 to 15%, in Round 2).
63
3. The number of general texts increased (from 15% in Round 1 to 22% in Round 2).
4. The number of texts of the Grade 7-9 group that demonstrate strong mathematical
awareness increased very slightly in Round 2 (from 61% in Round 1 to 63% in Round
2).
5. The gap between texts where explanation of the error is predominantly procedural
and texts where explanation of the error links to common misconceptions or errors
also only slightly grew wider in Round 2 (from 9% in Round 1 to 13% in Round 2).
Overall findings on the awareness of the mathematical error – Rounds 1 and 2
Figure 12: Awareness of error in explanations – Round 1 and 2
Awareness of error - Overall - Round 1 and 2
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
50%
36%
18%
22%
29%
17%
22%
6%
Round 1
Round 2
Mathematical
explanation of error
not present
Round 1
Round 2
Mathematical
explanation of error is
flawed or incomplete
Round 1
Round 2
Mathematical
explanation
emphasises the
procedural
Round 1
Round 2
Mathematical
explanation
emphasises the
conceptual
64
Table 15: Awareness of the mathematical error demonstrated in teacher test explanations
Change
Level of mathematical
Round Round
Strength of
Round Round
between
awareness
1
2
explanation
1
2
rounds
In these texts a
mathematical explanation
of the particular error is
not included.
18%
22%
4%
In these texts the
explanation of the
particular error is
mathematically inaccurate
or incomplete and hence
potentially confusing.
17%
6%
-11%***
In these texts the
explanation of the
particular error is
mathematically sound but
does not link to common
misconceptions or errors.
The explanation is
predominantly procedural.
36%
50%
14%***
In these texts the
explanation of the
particular error is
mathematically sound and
suggests links to common
misconceptions or errors.
29%
22%
Weaker
36%
28%
Stronger
64%
72%
-7%
*** Difference significant at a 99% level of confidence
Observations about groups’ awareness of the mathematical error in Rounds 1 and 2:
1. The overall number of texts that demonstrate weak awareness of error decreased
between Rounds 1 and 2 (from 36% in Round 1 to 28% in Round 2).
2. The number of texts where no mathematical awareness of the error was evident
increased by 4% from 18% in Round 1 to 22% of texts in Round 2. This increase was
not significant.
3. The number of texts where explanations were mathematically inaccurate decreased
by 11% (from 17% in Round 1 to 6% in Round 2). This decrease was significant at
99%.
4. The overall number of texts that demonstrate strong awareness of the mathematical
error increased by 8% in Round 2 (from 64% in Round 1 to 72% in Round 2).
65
5. The gap between explanations that show more procedural than conceptual
awareness of the error was 9% in Round 1 but it grew much wider to 28% in Round
2.
6. The increase between Round 1 and Round 2 in the number of texts show more
procedural awareness of the error was significant at 99%.
Awareness of error, by mathematical content
Number was the content area in which the groups’ awareness of error in their
explanations of the choice of incorrect answer was the strongest. Shape was the content
area in which the groups’ awareness of error in their explanations of the choice of
incorrect answer was the weakest. The graphs below represent the percentages of
explanations demonstrating awareness of error for these two content areas across the
two rounds.
Figure 13: Round 1 and 2 awareness of error – content area number
Number Awarenss of error Rounds 1 and 2
100
90
Percentage of texts
80
70
60
50
40
30
20
10
0
Not present
Inaccurate
Partial
Full
Round 1
20.89552239
11.94029851
28.35820896
38.80597015
Round 2
18.18181818
3.03030303
54.54545455
24.24242424
Change in the strong content area:



The number of texts that demonstrate awareness of concept decreased from
38.81% in Round 1 to 24.24% in Round 2.
The number of texts that demonstrate awareness of procedure increased from
28.36% in Round 1 to 54.55% in Round 2.
The number of texts that have mathematical flaws decreased from 11.94% in
Round 1 to 3.03% in Round 2.
66

The number of texts that includes no mathematical awareness decreased from
20.90 in Round 1 to 18.18% in Round 2.
Figure 14: Round 1 and 2 awareness of error – content area shape
Shape Awarenss of error Rounds 1 and 2
100
90
Percentage of texts
80
70
60
50
40
30
20
10
0
Not present
Inaccurate
Partial
Full
Round 1
17.85714286
26.78571429
41.07142857
14.28571429
Round 2
22.22222222
5.555555556
61.11111111
11.11111111
Change in the weak content area:




The number of texts that demonstrate awareness of concept decreased from
14.29% in Round 1 to 11.11% in Round 2.
The number of texts that demonstrate awareness of procedure increased from
41.07% in Round 1 to 61.11% in Round 2.
The number of texts that have mathematical flaws decreased from 26.79% in
Round 1 to 5.56% in Round 2.
The number of texts that includes no mathematical awareness increased from
17.86% in Round 1 to 22.22% in round 2.
67
4.2.4 Diagnostic reasoning
Figure 15: Diagnostic reasoning in explanations – Round 1 and 2 by grouped grades
Diagnostic Reasoning - Round 1 and Round 2 by grade
100%
90%
80%
70%
60%
50%
40%
Grade 3-6
30%
Grade 7-9
20%
10%
0%
Round 1 Round 2 Round 1 Round 2 Round 1 Round 2 Round 1 Round 2
No attempt to
explain learner
reasoning
Description of
Description of
Description of
learner reasoning learner reasoning learner reasoning
does not hone in hones in on error is systematic and
on error
but is incomplete hones in on error
Analysis of diagnostic reasoning in Grade 3-6 small groups’ texts:
Table 16: Grades 3-6 diagnostic reasoning demonstrated in teacher text explanations
Change
Round Round
Strength of
Diagnostic awareness
between
Round 1
1
2
explanation
rounds
In these texts, no attempt
is seen to describe
learners’ mathematical
reasoning behind the
particular error
24%
29%
Round
2
5%
In these texts the
description of the learners’
mathematical reasoning
does not hone in on the
particular error.
31%
29%
-2%
In these texts the
description of the learners’
mathematical reasoning is
35%
30%
-5%
Weaker
55%
58%
Stronger
45%
42%
68
incomplete although it
does hone in on the
particular error.
In these texts the
description of the steps of
learners’ mathematical
reasoning is systematic
and hones in on the
particular error.
10%
12%
2%
Observations about groups’ diagnostic reasoning in relation to the mathematical error
in Rounds 1 and 2 for the grade 3-6 groups:
1. The number of texts of the Grade 3-6 group that demonstrate weak diagnostic
reasoning increased slightly between Rounds 1 and 2 (from 55% in Round 1 to 58%
in Round 2).
2. The number of texts where no attempt was made to explain learner reasoning
behind the error increased by 4% from 24% in Round 1 to 29% of texts in Round 2.
3. The number of texts where learner reasoning was described but did not hone in on
the error slightly decreased by 3% (from 31% in Round 1 to 29% in Round 2).
4. The number of texts of the Grade 3-6 group that demonstrate strong diagnostic
reasoning slightly decreased by 3% in Round 2 (from 45% in Round 1 to 42% in
Round 2).
5. The gap between incomplete and systematic explanations was high but decreased
Round 2 (from 25% in Round 1 to 18% in Round 2).
Analysis of diagnostic reasoning in Grade 7-9 small groups’ texts:
Table 17: Grades 7-9 diagnostic reasoning demonstrated in teacher text explanations
Change
Round Round
Strength of
Round
Diagnostic awareness
between
1
2
explanation
1
rounds
In these texts, no attempt
is seen to describe
learners’ mathematical
reasoning behind the
particular error
23%
In these texts the
description of the learners’
mathematical reasoning
does not hone in on the
particular error
36%
25%
2%
Weaker
12%
Round
2
59%
37%
-24%***
69
In these texts the
description of the learners’
mathematical reasoning is
incomplete although it
does hone in on the
particular error
25%
In these texts the
description of the steps of
learners’ mathematical
reasoning is systematic
and hones in on the
particular error.
16%
53%
28%***
Stronger
11%
41%
64%
-5%
*** Difference significant at a 99% level of confidence
Observations about groups’ diagnostic reasoning in relation to the mathematical error
in Rounds 1 and 2 for the grade 7-9 groups:
1. There was a big decrease in Round 2 in the number of texts of the Grade 7-9 group
that demonstrate weak diagnostic reasoning (from 59% in Round 1 to 37% in Round
2).
2. The number of texts where no attempt was made to explain learner reasoning
increased by 2% from 23% to 25% of texts.
3. The number of texts where teachers attempt to describe learner reasoning but the
description does not hone in on the error decreased by 24% (from 36% in Round 1 to
12% in Round 2).
4. The number of texts of the Grade 7-9 group that demonstrate strong diagnostic
reasoning increased by 23% in Round 2 (from 41% in Round 1 to 64% in Round 2).
5. The gap between incomplete and systematic explanations was not very high in
Round 1 (9%) but grew wider in Round 2 (41%).
70
Overall findings on the diagnostic reasoning in relation to the mathematical error –
Rounds 1 and 2
Figure 16: Diagnostic reasoning in relation to the error in explanations – Round 1 and 2
Diagnostic reasoning - Overall - Round 1 and 2
100%
90%
80%
70%
60%
50%
40%
30%
26%
28%
32%
24%
30%
36%
20%
12%
12%
Round 1
Round 2
10%
0%
Round 1
Round 2
Round 1
Round 2
Round 1
Round 2
No attempt to explain Description of learner Description of learner Description of learner
learner reasoning
reasoning does not reasoning hones in on reasoning is systematic
hone in on error error but is incomplete and hones in on error
Table 18: Diagnostic reasoning in relation to the error demonstrated in teacher test
explanations
Change
Round Round
Strength of
Diagnostic awareness
between
Round 1
1
2
explanation
rounds
No attempt to explain
learner reasoning
26%
28%
2%
Description of learner
reasoning does not hone
in on error
32%
24%
-12%
Description of learner
reasoning hones in on
error but is incomplete
30%
36%
6%
Description of learner
reasoning is systematic
and hones in on error
12%
12%
Round
2
Weaker
58%
52%
Stronger
42%
48%
0%
71
Observations about groups’ diagnostic reasoning in relation to the mathematical error
in Rounds 1 and 2:
1. The overall number of texts that demonstrate weak diagnostic reasoning in relation
to the error decreased slightly between Rounds 1 and 2 (from 58% to 52%).
2. The number of texts where there was no attempt to explain the learners’ errors was
evident increased by 2% from 26% in Round 1 to 28% of texts in Round 2. This
increase was not significant.
3. The number of texts where explanations did not hone in on the error decreased by
6% (from 32% in Round 1 to 24% in Round 2). This decrease was not significant.
4. The overall number of texts that demonstrate strong diagnostic reasoning in relation
to the error increased by 6% in Round 2 (from 42% in Round 1 to 48% in Round 2).
5. The gap between explanations that hone in on the error but are either incomplete or
complete was 18% in Round 1 but it grew to 24% in Round 2.
6. The number of texts with incomplete explanations but that do hone in on the error
increased from 30% in Round 1 to 36% in Round 2. This increase was not significant.
7. The number of texts with complete explanations that hone in on the error was 12% of
all texts and did not change between Rounds 1 and 2.
Diagnostic reasoning, by mathematical content
Measurement was the content area in which the groups’ diagnostic reasoning their
explanations of the choice of incorrect answer was the strongest. Shape was the content
area in which the groups’ diagnostic reasoning their explanations of the choice of
incorrect answer was the weakest. The graphs below represent the percentages of
explanations demonstrating diagnostic reasoning in the explanation of errors for these
two content areas across the two rounds.
72
Figure 17: Round 1 and 2 diagnostic reasoning – content area measurement
Percentage of texts
Measurement Diagnostic reasoning Rounds 1 and
2
100
90
80
70
60
50
40
30
20
10
0
Not present
Inaccurate
Partial
Full
Round 1
25.92592593
24.07407407
35.18518519
14.81481481
Round 2
21.42857143
25
28.57142857
25
Change in the strong content area:




The number of texts in which the description of learner reasoning is systematic
and hones in on error increased from 14.81% in Round 1 to 25% in Round 2.
The number of texts in which the description of learner reasoning hones in on
error but is incomplete decreased from 35.19% in Round 1 to 28.57% in Round 2.
The number of texts in which the description of learner reasoning does not hone
in on error increased from 15.93% in Round 1 to 21.43% in Round 2.
The number of texts that include no attempt to explain learner reasoning
decreased from 25.93 in Round 1 to 21.43% in Round 2.
73
Figure 18: Round 1 and 2 diagnostic reasoning – content area shape
Shape Diagnostic reasoning Rounds 1 and 2
100
90
Percentage of texts
80
70
60
50
40
30
20
10
0
Not present
Inaccurate
Partial
Full
Round 1
28.57142857
41.07142857
21.42857143
8.928571429
Round 2
33.33333333
27.77777778
38.88888889
0
Change in the weak content area:




The number of texts in which the description of learner reasoning is systematic
and hones in on error decreased from 8.93% in Round 1 to 0% in Round 2.
The number of texts in which the description of learner reasoning hones in on
error but is incomplete increased from 21.43% in Round 1 to 38.89% in Round 2.
The number of texts in which the description of learner reasoning does not hone
in on error decreased from 41.07% in Round 1 to 27.78% in Round 2.
The number of texts that include no attempt to explain learner reasoning
increased from 28.57% in Round 1 to 33.33% in Round 2.
74
4.2.5 Multiple explanations
In order to code teachers’ use of multiple explanations, all the items were first coded as
mathematically feasible or not. The coding on this criterion was per item since the code
(per item) was assigned based on the overall coding of all of the texts relating to each
item. Non-feasible explanations include explanations that are not mathematically
focused on the error under discussion (for example, ‘Simply a lack in conceptual
understanding of an odd number’) or are what we call “general texts” (for example, ‘We
could not ascertain the reasoning behind this distractor as there was no clear or obvious
reason why a learner would choose it’).The final code for multiple explanations was
assigned according to the number of feasible/non-feasible explanations given for texts
relating to each item.
Figure 19: Feasible and non-feasible explanations – Round 1 and 2
Feasible and not feasible explanations - Round 1 and
Round 2 by grade
100%
90%
85% 86%
80% 78%
80%
70%
60%
50%
Grade 3-6
40%
Grade 7-9
30%
20% 22%
15% 14%
20%
10%
0%
Round 1
Round 2
Feasible
Round 1
Round 2
Not feasible
Observation
The differences between Rounds 1 and 2 show a small decrease in the number of feasible
explanations offered by the two groups (Grade 3-6 and Grade 7-9) in explanation of
error and a corresponding increase in non-feasible explanations. This finding points to
the value of the presence of an expert leader in the groups.
75
The findings reported below show that although the decrease in Round 2 in the number
of feasible explanations is small (5%) for the Grade 3-6, and slightly higher (8%) for the
Grade 7-9, teachers across the two groups provide mainly one mathematically
feasible/convincing explanation (with/without general explanation).
Figure 20: Multiple explanations in explanations – Round 1 and 2 by grouped grades
Multiple Explanations - Round 1 and Round 2 by grade
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Grade 3-6
Grade 7-9
Round 1 Round 2 Round 1 Round 2 Round 1 Round 2 Round 1 Round 2
No feasible
mathematical
explanation
One feasible
mathematical
explanation
with/without
general
explanations
Two feasible
mathematical
explanations
combined with
general
explanations
Two (or more)
feasible
mathematical
explanations
76
Analysis of multiple explanations in Grade 3-6 small groups’ texts:
Table 19: Grades 3-6 multiple explanations demonstrated in teacher text explanations
Change
Round Round
Strength of
Multiple explanations
between
Round 1
1
2
explanation
rounds
In these items no
mathematically
feasible/convincing
explanation is provided
4%
9%
5%
In these items one
mathematically
feasible/convincing
explanation is provided
(with/without general
explanations)
62%
74%
8%
In these items, at least two
of the mathematical
explanations are
feasible/convincing
(combined with general
explanations)
13%
7%
-6%
In these items, all of the
explanations (two or more)
are mathematically
feasible/convincing
21%
10%
Round
2
Weaker
66%
83%
Stronger
34%
17%
-11%
Observations about groups’ multiple explanations of mathematical errors in Rounds 1
and 2 for the grade 3-6 groups:
1. Grade 3-6 use of multiple explanations was low in both Rounds 1 and 2. 66% of the
items in Round 1 and 83% of the items in Round 2 demonstrate little use of multiple
explanations.
2. The number of items with no feasible mathematical explanation at all increased very
slightly (from 4% in Round 1 to 9% in Round 2).
3. The number of items with one feasible mathematical explanation was high and
increased by 11% in Round 2, (from 62% in Round 1 to 73%, in Round 2).
4. Multiple explanations (two or more) were not highly evident in Round 1 and even
less so in Round 2 in the grade 3-6 groups.
5. The number of items in the Grade 3-6 group that demonstrate use of multiple
explanations decreased by 17% in Round 2 (from 34% in Round 1 to 17% in Round
2). The decrease is evident in both types of stronger explanations (6% and 11% in the
respective types).
77
6. The Grade 3-6 groups provide mainly one mathematically feasible/convincing
explanation (with/without general explanations).
Analysis of multiple explanations in Grade 7-9 small groups’ texts:
Table 20: Grades 7-9 multiple explanations demonstrated in teacher text explanations
Change
Round Round
Strength of
Multiple explanations
between
Round 1
1
2
explanation
rounds
In these items no
mathematically
feasible/convincing
explanation is provided
5%
17%
12%
In these items one
mathematically
feasible/convincing
explanation is provided
(with/without general
explanations)
48%
70%
22%
In these items, at least two
of the mathematical
explanations are
feasible/convincing
(combined with general
explanations)
8%
0%
-8%
In these items, all of the
explanations (two or more)
are mathematically
feasible/convincing
38%
13%
Round
2
Weaker
53%
87%
Stronger
46%
13%
-25%
Observations about groups’ multiple explanations of mathematical errors in Rounds 1
and 2 for the grade 7-9 groups:
1. Grade 7-9 use of multiple explanations was low in both Rounds 1 and 2. 53% of the
items in Round 1 and 87% of the items in Round 2 demonstrate little use of multiple
explanations.
2. The number of items with no feasible mathematical explanation at all increased by
12% (from 5% in Round 1 to 17% in Round 2).
3. The number of items with one feasible mathematical explanation radically increased
in Round 2 by 22% (from 48% in Round 1 to 70%, in Round 2).
4. Multiple explanations (two or more) were not highly evident in Round 1 and even
less so in Round 2 in the Grade 7-9 groups.
78
5. The number of items of the Grade 7-9 groups that demonstrate use of multiple
explanations radically decreased, by 30% in Round 2 (from 46% in Round 1 to 13% in
Round 2).
6. There was a difference of 30% between the percentage of items where two or more
feasible mathematical explanations were offered with and without general
explanations in Round 1 and a 13% gap between these multiple explanation
categories in Round 2. This means that there was a decrease in both types of multiple
explanations from Round 1 to Round 2.
7. There were 8% of items with two or more mathematical and general explanations
and 25% of items where two or more mathematical explanations were put forward.
In other words, Grades 7-9 did have 13% of items for which they offered two or
more feasible mathematical explanations without general explanations but all other
explanations they gave fell into the “weaker” categories.
8. The Grade 7-9 groups provide mainly one mathematically feasible/convincing
explanation (with/without general explanations).
79
Overall findings on the multiple explanations of the mathematical error – Rounds 1
and 2
Figure 21: Multiple explanations of the error in explanations – Round 1 and 2
Multiple Explanations- Overall - Round 1 and 2
100%
90%
80%
74%
70%
56%
60%
50%
40%
30%
20%
10%
11%
11%
4%
12%
11%
Round 1
Round 2
4%
0%
Round 1
Round 2
No feasible
mathematical
explanation
Round 1
Round 2
Round 1
Round 2
One feasible
Two feasible
Two (or more) feasible
mathematical
mathematical
mathematical
explanation
explanations combined
explanations
with/without general
with general
explanations
explanations
80
Table 21: Multiple explanations of the error demonstrated in teacher test explanations
Change
Round Round
Strength of
Round
Multiple explanations
between
1
2
explanation
1
rounds
No feasible mathematical
explanation
4%
11%
7%
One feasible
mathematical explanation
with/without general
explanations
56%
74%
18%
Two feasible
mathematical
explanations combined
with general explanations
11%
4%
-7%
Two (or more) feasible
mathematical
explanations
12%
11%
Round
2
Weaker
60%
85%
Stronger
23%
15%
-1%
Observations about groups’ multiple explanations of the mathematical error in Rounds
1 and 2:
1. The use of multiple explanations was low in both Rounds 1 and 2.
2. 60% of the items in Round 1 and 85% of the items in Round 2 demonstrate little use
of multiple explanations.
3. The number of items with no feasible mathematical explanation at all increased by
7% (from 4% in Round 1 to 11% in Round 2).
4. The number of items with one feasible mathematical explanation increased in Round
2 by 18% (from 56% in Round 1 to 74% in Round 2).
5. There were very few multiple explanations (two or more) in Round 1 and even less
so in Round 2.
6. The number of items where any kind of multiple explanations of the error were
given decreased, by 8% in Round 2 (from 23% in Round 1 to 15% in Round 2).
7. However the decrease was not in the strongest category of multiple explanations
(which decreased from 12% in Round 1 to 11% in Round 2), it was in the category of
two feasible mathematical explanations combined with general explanations (which
decreased from 11% in Round 1 to 4% in Round 2).
8. All groups provide mainly one mathematically feasible/convincing explanation
(with/without general explanations).
Multiple explanations, by mathematical content
Shape was the content area in which the groups’ multiple explanations of the choice of
the incorrect answer were the strongest. Algebra was the content area in which the
81
groups’ multiple explanations of the choice of incorrect answer were the weakest. The
graphs below represent the percentages of explanations demonstrating multiple
explanations of the error for these two content areas across the two rounds.
Figure 22: Round 1 and 2 Multiple explanations – content area shape
Shape Multiple Explanations Rounds 1 and 2
100
90
Percentage of texts
80
70
60
50
40
30
20
10
0
Not present
Inaccurate
Partial
Full
Round 1
3.448275862
58.62068966
10.34482759
27.5862069
Round 2
15.38461538
61.53846154
0
23.07692308
Change in the strong content area:




The percentage of items for which all explanations (two or more) were
mathematically feasible/convincing decreased from 27.59% in Round 1 to 23% in
Round 2.
The percentage of items for which at least two of the mathematical explanations
were feasible/convincing but which were combined with general explanations
decreased from 10.34% in Round 1 to 0% in Round 2.
The percentage of items in which one mathematically feasible/convincing
explanation was provided (with/without general explanations) increased from
58.62% in Round 1 to 61.54% in Round 2.
The percentage of items for which no mathematically feasible/convincing
explanation was provided increased from 3.45% in Round 1 to 15.38% in Round
2.
82
Figure 23: Round 1 and 2 Multiple explanations – content area algebra
Algebra Multiple Explanations Rounds 1 and 2
100
90
Percentage of texts
80
70
60
50
40
30
20
10
0
Not present
Inaccurate
Partial
Full
Round 1
5.263157895
68.42105263
10.52631579
15.78947368
Round 2
7.142857143
92.85714286
0
0
Change in the weak content area:




The percentage of items for which all explanations (two or more) were
mathematically feasible/convincing decreased from 15.79% in Round 1 to 0% in
Round 2.
The percentage of items for which at least two of the mathematical explanations
were feasible/convincing but which were combined with general explanations
decreased from 10.53% in Round 1 to 0% in Round 2.
The percentage of items in which one mathematically feasible/convincing
explanation was provided (with/without general explanations) increased from
68.42% in Round 1 to 92.86% in Round 2.
The percentage of items for which no mathematically feasible/convincing
explanation was provided increased from 5.26% in Round 1 to 7.14% in Round 2.
83
4.2.6 Use of the everyday in explanations of the error
“Use of the everyday” was considered in the analysis since the NCS Curriculum
emphasises linking the teaching of mathematics to everyday contexts or using everyday
contexts to enlighten learners about mathematical concepts. The ICAS tests included
predominantly contextualized questions (ranging from 98% in the Grade 3 test to 85% in
the Grade 9 test) and yet teachers made very few references to these contexts in their
explanations. In certain questions there may not have been contextualized explanations
that would have assisted in the explanation of the solution. The code “no discussion of
the everyday” includes all explanations which did not make reference to the everyday
whether it was appropriate or not. Further more detailed analysis of the kinds of
contextualized explanations which were offered and whether they were missing when
they should have been present could be carried out since it was not within the scope of
this report.
Figure 24: Use of the everyday in explanations of the error – Round 1 and 2 by grouped grades
Everyday Explanations - Round 1 and Round 2 by grade
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Grade 3-6
Grade 7-9
Round 1 Round 2 Round 1 Round 2 Round 1 Round 2 Round 1 Round 2
No discussion of
everyday
Everyday
dominates and
obscures the
explanation
Everyday link is
relevant but not
mathematically
grounded
Everyday
explanation links
clearly to the
mathematcial
concept illustrated
Can see that Grades 3-6 make more reference to the everyday in their explanations, but
this is minimal.
84
Analysis of use of the everyday in Grade 3-6 small groups’ texts:
Table 22: Grades 3-6 use of the everyday demonstrated in teacher text explanations of the error
Change
Use of everyday in
Round Round
Strength of
Round Round
between
explanations
1
2
explanation
1
2
rounds
In these texts no discussion
of everyday is evident
88%
97%
9%***
In these texts teachers’ use
of the ‘everyday’
dominates and obscures
the mathematical
understanding, no link to
mathematical
understanding is made
5%
1%
-4%**
In these texts teachers’ use
of the ‘everyday’ is
relevant but does not
properly explain the link to
mathematical
understanding
5%
2%
-3%*
In these texts teachers’ use
of the ‘everyday’ enables
mathematical
understanding by making
the link between the
everyday and the
mathematical clear
2%
0%
low
93%
98%
high
7%
2%
-2%*
*** Difference significant at a 99% level of confidence
** Difference significant at a 95% level of confidence
* Difference significant at a 90% level of confidence
Observations about the quality of the use of error analysis between Round 1 and Round
2 for the grade 3-6 groups:
1. Grade 3-6 use of everyday explanations was extremely low in both Rounds 1 and 2.
2. In 88% of the texts in Round 1 and in 97% of the texts in Round 2, Grade 3-6 group
did not discuss every day contexts when they dealt with the errors.
3. Together with the texts where the use of everyday context was obscuring (5% in
Round 1 and 1% in round 2) the number of texts for Grade 3-6 groups in which
discussion of everyday contexts in the explanation of error is low increased by 5%
(from 93% in Round 1 to 98% in Round 2)
85
4. Grade 3-6 use of everyday explanations which was relevant but incomplete and or
enabled mathematical understating of the error was very low in both Rounds 1 and 2
(7% in Round 1 and 2% in Round 2).
5. The difference in use of everyday (relevant but incomplete and enabled
mathematical understating) was small and almost the same in the two rounds ( 3%
in Round 1 and 2 % in Round 2)
Analysis of use of the everyday in explanations in Grade 7-9 small groups’ texts:
Table 23: Grades 7-9 use of the everyday demonstrated in teacher text explanations of the error
Change
Use of everyday in
Round Round
Strength of
Round Round
between
explanations
1
2
explanation
1
2
rounds
In these texts no discussion
of everyday is evident
96%*
100%*
4%**
In these texts teachers’ use
of the ‘everyday’ dominates
and obscures the
mathematical
understanding, no link to
mathematical
understanding is made
1%
0%
-1%
In these texts teachers’ use
of the ‘everyday’ is relevant
but does not properly
explain the link to
mathematical
understanding
3%
0%
-3%*
In these texts teachers’ use
of the ‘everyday’ enables
mathematical
understanding by making
the link between the
everyday and the
mathematical clear
1%
0%
Weaker
97%
100%
Stronger
3%
0%
-1%*
** Difference significant at a 95% level of confidence
* Difference significant at a 90% level of confidence
86
Observations about the quality of the use of error analysis between Round 1 and Round
2 for the grade 7-9 groups:
1. Grade 7-9 use of everyday explanations was extremely low in both Rounds 1 and 2.
2. In 96% of the texts in Round 1 and in 100% of the texts in Round 2, Grade 7-9 group
did not discuss every day contexts when they dealt with the errors.
3. In Round 2 texts with no discussion of everyday contexts increased by 9%.
4. Together with the texts where the use of everyday context was obscuring (1% in
Round 1 and 0% in round 2) the number of texts for Grade 7-9 groups in which
discussion of everyday contexts in the explanation of error is negligible.
5. Grade 7-9 use of everyday explanations which was relevant but incomplete and or
enabled mathematical understating of the error was very low in both Rounds 1 and 2
(3% in Round 1 and 0% in Round 2).
6. Only in Round there was a difference in use of everyday (relevant but incomplete
and enabled mathematical understating), which was negligible.
Overall findings on the use of the everyday in explanations of the mathematical error
– Rounds 1 and 2
Figure 25: Use of the everyday demonstrated in teacher explanations of the error – Round 1
and 2
Use of everyday in explanations- Overall - Round 1 and 2
100%
98%
91%
90%
80%
70%
60%
50%
40%
30%
20%
12%
10%
4%
1%
4%
1%
0%
0%
Round 1
Round 2
No discussion of
everyday
Round 1
Round 2
Everyday dominates
and obscures the
explanation
Round 1
Round 2
Everyday link is
relevant but not
mathematically
grounded
Round 1
Round 2
Everyday explanation
links clearly to the
mathematcial concept
illustrated
87
Table 24: Use of the everyday in explanations of the error demonstrated in teacher test
explanations
Change
Use of everyday in
Round Round
Strength of
Round
between
explanations
1
2
explanation
1
rounds
In these texts no discussion
of everyday is evident
91%
98%
7%
In these texts teachers’ use
of the ‘everyday’
dominates and obscures
the mathematical
understanding, no link to
mathematical
understanding is made
4%
1%
-3%**
In these texts teachers’ use
of the ‘everyday’ is relevant
but does not properly
explain the link to
mathematical
understanding
4%
1%
-3%*
In these texts teachers’ use
of the ‘everyday’ enables
mathematical
understanding by making
the link between the
everyday and the
mathematical clear
12%
0%
Round
2
Weaker
94%
99%
Stronger
16%
1%
-12%***
*** Difference significant at a 99% level of confidence
** Difference significant at a 95% level of confidence
* Difference significant at a 90% level of confidence
Observations about groups’ use of the everyday in explanations of the mathematical
error in Rounds 1 and 2:
1. The overall use of everyday in explanations of the error was extremely low in both
rounds 1 and 2.
2. In 91% of the texts in Round 1 and 98% of the texts, the groups did not discuss every
day related explanations of the error.
3. Only in 12% of the texts in Round 1 the groups used ‘everyday’ in ways that enables
mathematical understanding, by making the link between the everyday and the
mathematical clear
88
Use of the everyday, by mathematical content
There was very little reference to the everyday in mathematical explanations across all
content areas, but since such minimal reference was made in this section we highlight
the two areas in which at least some reference was made to the everyday in Rounds 1
and 2. The graphs below represent the percentages of explanations where some
reference to the everyday was demonstrated in the explanations of errors for these two
content areas across the two rounds.
Figure 26: Round 1 Use of everyday in explanations – content area number
Measurement use of the everyday Rounds 1 and 2
100
90
Percentage of texts
80
70
60
50
40
30
20
10
0
Not present
Inaccurate
Partial
Full
Round 1
85.18518519
1.851851852
5.555555556
7.407407407
Round 2
96.42857143
3.571428571
0
0
Change in the area of measurement




The number of texts that demonstrate groups’ use of the ‘everyday’ that enables
mathematical understanding by making the link between the everyday and the
mathematical clear, decreased from 7.41% in Round 1 to 0% in Round 2.
The number of texts in which teachers’ use of the ‘everyday’ is relevant but does
not properly explain the link to mathematical understanding, decreased from
5.56% in Round 1 to 0% in Round 2.
The number of texts in which teachers’ use of the ‘everyday’ dominates and
obscures the mathematical understanding, and where no link to mathematical
understanding is made, is negligible, decreasing from 1.85% in Round 1 to 0% in
Round 2.
The number of texts where no discussion of everyday is evident increased from
85.19% in Round 1 to 96.43% in Round 2.
89
Figure 27: Round 1 Use of everyday in explanations – content area measurement
Number use of the everyday Rounds 1 and 2
100
90
Percentage of texts
80
70
60
50
40
30
20
10
0
Not present
Inaccurate
Partial
Full
Round 1
92.53731343
4.47761194
2.985074627
0
Round 2
96.96969697
0
3.03030303
0
Change in the area of number:




In both rounds no text was found that demonstrates groups’ use of the
‘everyday’ that enables mathematical understanding by making the link between
the everyday and the mathematical clear.
The number of texts in which teachers’ use of the ‘everyday’ is relevant but does
not properly explain the link to mathematical understanding, was negligible in
both rounds, increasing from 2.09% in Round 1 to 3% in Round 2.
The number of texts in which teachers’ use of the ‘everyday’ dominates and
obscures the mathematical understanding, and where no link to mathematical
understanding is made, decreased from 4.48% in Round 1 to 0% in Round 2.
The number of texts where no discussion of everyday is evident increased from
92.54% in Round 1 to 96.97% in Round 2.
4.2.7 Comparative strengths and weaknesses between the two sets of
grouped grades
The graphs below show the changes between rounds in the number of texts
demonstrating texts which do not have mathematical content “not present” and texts
which are mathematcially inaccurate (“inaccurate”) in the procedural, conceptual,
awareness and diganostic criteria. The first bar (blue) in each pair represents the Grade
3-6 group and the second bar (red) represents the Grade 7-9 group. Positive changes
90
represnet an increase between Round 1 and Round 2 while negative changes represent a
decrease between Round 1 and Round 2. The actual changes in percentages betweeen
rounds are given in the table below the graph.
Figure 28: Grade 3-6 and Grade 7-9 changes in “not present” and “inaccurate” explanations
Changes between rounds on criterion levels
"not present" and "inaccurate"
Percentage change between rounds
40
30
20
10
0
-10
-20
-30
Not
Inaccurat
Not
Inaccurat
Not
Inaccurat
Not
Inaccurat
present
e
present
e
present
e
present
e
Procedural
Conceptual
Subject Matter Knowledge
Awareness
Diagnostic
PCK
Grade 3-6 14.220681 -5.828103 25.956199 -8.311805 3.0235019 -8.373027 4.6592256 -2.041653
Grade 7-9 5.760582 1.436078927.698413 -11.32615 7.554726 -9.288372 1.4304409 -23.89219
These two criterion levels are the “weaker” levels and thus an increase here represents a
weakening while an increase represents strengthening between rounds. The absence of
mathematical content in an explanation increased across all criteria for both grade
groups but the presence of inaccurate texts decreased more markedly for the Grade 7-9
group, which suggests that the Grade 7-9 group held better quality when they worked
on their own without group leaders.
The graphs below show the changes between rounds in the number of texts
demonstrating texts which demonstrate incomplete mathematical explanations “partial”
and texts which demonstrate full mathematical explanations (“full”) in the procedural,
conceptual, awareness and diganostic criteria. The first bar (blue) in each pair represents
the Grade 3-6 group and the second bar (red) represents the Grade 7-9 group. Positive
91
changes represnet an increase between Round 1 and Round 2 while negative changes
represent a decrease between Round 1 and Round 2. The actual changes in percentages
betweeen rounds are given in the table below the graph.
Figure 29: Grade 3-6 and Grade 7-9 changes in “partial” and “full” explanations
Changes between rounds on criterion levels
"partial" and "full"
Percentage change between rounds
40
30
20
10
0
-10
-20
-30
Partial
Full
Procedural
Partial
Full
Partial
Conceptual
Subject Matter Knowledge
Full
Awareness
Partial
Full
Diagnostic
PCK
Grade 3-6 14.617097 -23.00967 9.2269335 -26.87133 14.17001 -8.820485 -5.533302 2.9157297
Grade 7-9 12.502145 -19.69881 -6.657372 -9.714893 3.2271822 -1.493536 28.27757 -5.815822
These two criterion levels are the “stronger” levels and thus an increase here represents
a strengthening while a decrease represents weakening between rounds. The lack of
completeness in mathematical explanations increased across all criteria for both grade
groups but less so for the Grade 7-9 group, while the fullness of correct explanations
decreased but again less so for the Grade 7-9 group, which again suggests that the
Grade 7-9 group held better quality when they worked on their own without group
leaders.
92
4.2.8 Comparative strengths and weaknesses according to content area
The findings in relation to content areas in which groups were strongest and weakest are
summarised in Table 25 below. Strength was measured by taking the content area where
higher level explanations (partial or full) were the strongest in the two rounds. An
analysis of the changes between Round 1 and 2 for the various categories and types of
explanations offered shows that the groups explanations were most often stronger in the
area of number than in any other of the mathematical content areas.
Weakness was measured by taking into account where lower level explanations (not
present or inaccurate) were the most common in both rounds, or were the highest in
Round 2. There was not one single content area where weakness was demonstrated
consistently more than in other areas. Weakness was evident in the areas of data, algebra
and shape.
Table 25: Content areas according to strength and weakness in explanations
Text
Criteria
Strong content area
Weak content area
Procedural
Number
Data
Number
Algebra
Awareness of error
Number
Shape
Diagnostic
Measurement
Shape
Shape
Algebra
Number,
n/a
explanation of
Correct Answer
text
correct answer
Conceptual
explanation of
correct answer
reasoning
Error text
Multiple
Explanations
Use of everyday
Measurement
93
94
Section Five: General findings
95
5.1 What are the findings about teachers reasoning about
learners’ errors?
This section begins with a short qualitative analysis of teachers’ experience of the error
analysis activity. In this we report on three common perceptions of teachers when
interviewed on their experiences of the error analysis activity and its influence on their
practice.
In view of the division between the six criteria we used to evaluate teachers’ knowledge
of the items (correct solution and errors) we summarize the findings in two parts. We
first look at what the data shows about the three criteria that mapped against Ball’s et al
two Subject Matter Knowledge (SMK’s) domains. We then look at the three criteria that
mapped against one of the domains Ball et al’s specify for Pedagogical Content
Knowledge (PCK). We do this in order to look at the quality of teachers’ reasoning both
in relation to the correct solution and in relation to the error. We answer the question:

On what criteria is the groups’ error analysis weaker and on what criteria is the
groups’ error analysis stronger?
We proceed by looking at the difference between grouped grades’ (3-6 and 7-9)
performance on the different criteria between rounds. We answer the question:

Is there a difference between primary (Grade 3-6 groups) and high school (Grade 7-9
groups) teachers in sustaining the performance on the Round 1 error analysis
activity?
Although this was not a key focus of the project, in view of the analysis of teacher’s
knowledge we examine the differences in performance of the groups across different
mathematical content areas. We answer the question:

In which mathematical content areas do the groups produce better judgement on the
error?
Given the model of teacher development in which groups were working with group
leaders with academic we tentatively suggest the areas in which they contributed to the
groups in Round 1. We answer the question:

What does the change in performance between Round 1 and 2 suggest about the role
of group leaders?
96
5.2 Teachers’ experiences of the error analysis activity
In the various interviews conducted throughout the project, teachers, subject facilitators
and group leaders confirmed that the error analysis activity was interesting and
worthwhile. The different participants emphasized repeatedly that working with
questions actually made a difference to them- it made them aware that questions can be
asked in different ways, and they started thinking about asking their questions in a
different way. Before, they said, they knew what answer they wanted to get and created
a question to get to this answer, but they didn’t really think that some formulations of
the same question are better than others. Teachers found it interesting to examine the
ICAS questions, to see what multiple choice questions are like, how they work, what
kind of information they make available and how. Many of them took ICAS questions
and tried them in their classes and came back and reflected on their learners’
performance. They also enjoyed very much creating their own questions, testing their
learners and analysing the errors, as can be seen in the following three excerpts from the
focus group discussion after the activity.
We didn’t know how they came up with the answer. We had no clue. And the only way
(laughter) we thought we would have a clue … clue is to let the kids do it … and see what
they came up with and then ask them why they chose that answer. That is the only way,
because we had no clue. It was this funny picture with stars all over the show and it said,
which of the following is rotational symmetry and none of them actually had rotational
symmetry if you removed one. So, to us it didn’t make sense and we thought we’d throw it
back to the kids.
I did the same with the grade 9s. There was one question we couldn’t get to the answer, we
got the correct answer but why did, because there was I think 32% chose this specific wrong
answer. So I took it to my grade 9 classes, three of them, and then just for the fun of it, I gave
it to my grade 10 classes, two of the grade 10 classes. And the responses was…and then just
to be really now getting to the bottom of this, I gave it to my grade 8 son at home. And it was
very interesting coming from the students sitting in front of you.
I also did it with the grade 3s for quite a few questions and I got them to write at the back
how did they get to the answer. I didn’t ask them to explain it to me, I told them to write it
down. And if they had to draw it, then they drew it, if they had to colour in, they coloured in.
And it was very interesting. And many, many of them were able to explain to us how they
got to it, and some of them actually said there I guessed because I don’t know. They wrote: I
guessed because I don’t know.
In what follows we record other teachers’ comments about what they feel they learned
from doing the error analysis activity.
First, teachers reported that they understand that they need to gain a far deeper
understanding of the questions they set for their learners. Although this sounds obvious,
97
teachers reported that their approach to asking questions was simple or even unthoughtful before the error analysis.
Well, I don’t think I’ve ever thought about the way we ask questions before. You know, you
just do it because it’s been done like that for so many years. So now for the first time you start
thinking: but, maybe I can ask it in a different way? Maybe there is still something wrong
with the question, how it’s been put itself.
Teachers suggested that before this activity they did not have a way to think about test
or activity questions: “I think we all just set a question: now work out the perimeter of
this, and, you know, that’s it.” They wanted an answer to the question but did not know
how to think about the question in depth: “I don’t think as a maths teacher when we
originally set any test that you really analyse and probe in depth into the question… .”
The teachers felt that together, the mapping activity (see Process Document) and the
error analysis activity, showed them a way to examine the conceptual depth of
questions.
In the following two quotations, the respective teachers point to two related matters of
conceptual depth. The first quotation refers to the idea that questions draw on more
than one concept. Even though a question may have a specific conceptual focus, it could
draw on a number of related concepts and so it is essential that teachers unpack
questions when testing learners. Precisely because questions draw on webs of concepts,
learners may misunderstand these questions, particularly when the focus of the question
is not clear.
And maybe to add onto that, I think one other thing that came out of this is that you look
at a question and try to go…or maybe try to go back to school and say, what skills do
they need? What concepts do they need in order for them to, like she say, there are
multiple skills in one question there, what is it that they need to know in order to answer
this question? What skills? For instance, you might discover that is a story sum but we
need Pythagoras there. Some skills that they need to identify that they need to answer the
question.
Whilst the first quotation raises the conceptual background of questions which has
implications for learners’ preparedness, the next quotation raises the importance of
knowing what the question is in fact asking. Does it obscure its focus in ways that
learners, in particular, young and inexperienced ones, may struggle to understand
what’s expected of them. The teacher in the following quote posits direct and useful
questions that she now asks when formulating a question:
I think one other thing is that when we look at the questions now and probably now
when I look at a question that I set as well, I think about what is it I want from this, what
do I want them to do, and is my question going to allow it? And is it set in such a way
that they can see the steps that they need to do, or they understand the various
operations that are required? Especially at grade 3 level… . And now when I look at a
98
question and especially when it comes to problem solving and things like that I look at
that specifically to see, is it going to give us the answer we want, and are the kids going
to understand what’s expected of them? Ok, that’s one thing for me to set the question
and I understand what I want them to do, but do they understand?
Second, In addition to thinking about questions more deeply, teachers reported that
their orientation to learners’ approach to questions has changed. Doing the error
analysis activity, trying to figure out what the learners were thinking when they chose
the correct answer or the distractor, has given them a lens to think about learners’
reasoning:
I also thought that I should encourage my children to explain what they say. I should ask
them WHY? if they give me an answer… Not just leave it at that.
This is very different to what teachers were used to do:
We only give learners corrections where they needed corrections. I must say, we never
tell them there’s a misconception here, this is how you were supposed to do it, we just
give the corrections.
To most teachers the notion of “misconception” was new and many struggled to
develop productive relationship with the idea. So whilst the literature distinguishes
misconceptions from errors and posits that misconceptions are part of learning, for
many of the teachers, a misconception remained something that they need to foresee and
avoid. In almost every interview, the interviewer needed to remind the teachers that a
misconception is a mathematical construct that develops through learning the
development of which is not necessarily the teacher’s fault. More than that,
misconceptions can be a productive tool for teachers to deepen their understanding of
learners’ thinking and inform their teaching. Even towards the end of the project, the
teachers tended to treat misconceptions as errors that needed to be avoided:
Interviewee: What I can say, when we plan, so we try by all means to avoid the
misconception side.
Interviewer: To avoid it?
Interviewee: To avoid it. I think we try to avoid it.
Interviewer: Why would you avoid it?
Interviewee: Because it will distract the learners. It will confuse them. So I think that will
confuse them, I don’t think we can rely on it. You know, part of good teaching is that you
make people aware of the pitfalls. So no, I don’t think you’d avoid it entirely, but being
made aware of the pitfalls is different to putting up a booby trap. And I think that’s what
she’s saying is don’t put booby traps in the way. Does that make sense?
When asked to reflect on how the error analysis activity helped the teachers in planning
their lessons and in teaching the planned lesson, teachers acknowledged its influence in
99
two ways. In the following quotation the teacher speaks about awareness. Overcoming
misconceptions, she says “was right up front…the whole time”:
I would say awareness. If one is aware of the distractors that we identified, that brought
it to mind, and I think that we were going through it and when we were drawing up the
worksheets and drawing up the things, we did have that at the back of our minds and
saying, well let’s not fall into the same trap of having those distractors. And of course as
we were designing the learning program, the whole focus was on overcoming the
misconception. So it was right up front, it was there the whole time that was our
motivation the whole time, was to overcome that misconception of the equal sign.
This is an important point in that it suggests that in addition to following a syllabus or a
plan made by an HOD, the teacher feels that there is another lens that informs her
practice- the challenge to try and overcome misconceptions.
I mean, you’re given your frames and your time works, and your milestones and
whatever type of thing, but when you actually sit down and plan the lesson, maybe
foremost in our mind it should actually be, how am I going to teach this that the learners
are going to not make these errors and understand it properly, and if you’re aware of
what those errors are, then you can address them. But that also comes with experience, I
suppose. So, if I plan a lesson and I see where they’ve made the errors, the next time I
plan that lesson I might bring in different things to that lesson, bearing in mind what
those errors were. So adapting and changing your lessons. You might even re-teach that
lesson again.
I also think that in the error analysis you found where their misconceptions were and
then you had to find some way of addressing it, so the point of the lesson plan is to now
go and see how you can practically correct these misconceptions….
The reported experiences cited above are interesting and certainly encouraging.
Notwithstanding, they must be taken with caution and in relation to the quantitative
analysis that follows.
5.3 Findings from the quantitative analysis
On what criteria is the groups’ error analysis weaker and on what criteria is the groups’ error
analysis stronger?
Finding 1: Groups drew primarily on mathematical knowledge and less so on other
possible explanations to explain the correct answer or the errors.
When analysing the correct answer and the error, in about 70%-80% of the texts, groups
drew primarily on mathematical knowledge and much less so, on other discourses. In
both rounds the groups tried hard not to resort to common teacher talk on error such as
test-related explanations (the learners did not read the question well, or the learners
100
guessed) or learner-related (the question is not within the learners’ field of experience)
or curriculum-related (they have not learned this work). Interestingly, this finding is
aligned with groups’ use of everyday knowledge in explaining errors: Despite the
almost mandatory instruction by the NCS to link to everyday experiences when
explaining mathematical concepts, the findings on this criterion point to a very different
reality, with groups hardly referring to everyday contexts in their explanations of
mathematical errors. The use of everyday as part of error explanations was minimal
across all grades and all content areas. It was more evident in Round 1, when the groups
worked with expert leaders, albeit in 12% of the texts only. Only this small number of
texts in the sample demonstrates explanations of errors that make clear the link between
an everyday phenomenon and the mathematical content of the item, and thus shows a
use of ‘everyday’ that enables mathematical understanding. In Round 2 it was barely
evident at all.28
Finding 2: Most of groups’ explanations of the correct solutions and of the error are
incomplete, missing crucial steps in the analysis of what mathematics is needed to
answer the question.
Most of the explanations texts fall within the “incomplete” category, more so in Round
2. 57% of the procedural explanations of the correct answer, 45% of the conceptual
explanations of the correct answer, and 50% of awareness of error explanations are
incomplete. This is an important finding about the teachers’ knowledge. The evidence of
teachers using predominantly incomplete procedural explanations is worrying.
Incomplete procedural explanations of the mathematics involved in solving a particular
mathematic problem, may impede on teachers’ capacity to identify mathematical errors,
let alone to diagnose learners’ reasoning behind the errors.
Finding 3: There is a correlation between groups’ procedural and conceptual
explanations
The correlation between procedural explanations and conceptual explanations was high
in both rounds, although it decreased in Round 2 (r = 0,74 in Round 1 and r = 0,66 in
Round 2). This suggests that when teachers are able to provide a full explanation of the
steps to be taken to arrive at a solution they are also more aware about the conceptual
underpinnings of the solution and vice versa. The weaker the procedural explanations
are, the weaker the conceptual explanations, and vice versa. The higher correlation in
Round 1 than in Round 2, between procedural and conceptual explanations suggests
It is possible that this finding is specific to error analysis of learners’ response on (predominantly) multiple choice
questions, where three feasible mathematical responses are given. More work is being done now to interrogate this
finding, specifically, with regard to the teachers’ analysis of learner’s errors “own test”, which did not include any
multiple choice questions, and during teaching. Further examination of this findings is done in relation to teachers’ ways
of engaging with error, during teaching (referring to the 4th domain of teacher knowledge, Knowledge content and teaching
(KCT) see table 1.
28
101
that when working with the group leaders teachers’ unpacked the concepts underlying
the procedures more consistently.
Much research in South Africa suggests that teachers use procedural and not enough
conceptual explanations (Carnoy et al, 2011) and that this may explain learners’ poor
performance. A new and important lesson can be learned from the correlation between
procedural and conceptual explanations of the correct answer. The correlation suggests
that there is conceptual interdependence between the procedural and conceptual aspects
in teachers’ explanation of the mathematics that underlie a mathematical problem. When
the quality of teachers’ procedural explanations goes higher, the quality of the
conceptual explanations is also improved (and vice versa). This suggests that
mathematical explanations of procedure and of concept rely on good subject matter
knowledge of both, and a good procedural knowledge can help teachers in building
their understanding of the underling concept. Instead of foregrounding the lack of
conceptual understanding in teachers’ mathematical knowledge, more effort is needed
to improve the quality of teachers’ procedural explanations, making sure that teachers
are aware of which steps are crucial for addressing a mathematical problem and what
counts as a full procedural explanation. (Examples of the range of procedural and
conceptual explanations can be found in Appendix 6 and Appendix 7.)
More research is needed to differentiate between strong and weak procedural
explanations, in general and in different mathematical content areas. More research is
needed to understand the quality of the relationship between full(er) procedural
explanations and conceptual understanding. This is important for building data-base for
teachers on crucial steps in explanations of leverage topics. We believe that this will help
the effort of building mathematics knowledge for teaching.
Finding 4: With more practice groups’ explanations demonstrated decreased
inaccuracy, but at the expense of quality in other categories of explanations within the
three criteria of Subject Matter Knowledge
In all the Subject Matter Knowledge (SMK) explanations there was decreased inaccuracy
(inaccurate explanations) in Round 2. In fact this seems to be the category of explanation
in which they improved the most. This improvement needs to be understood
relationally, in the context of the three remaining categories within each of the three
SMK criteria. In what follows we look at these three criteria.

Procedural explanation of the correct answer: There are very few inaccuracies when
groups describe the steps learners need to take to arrive at the correct solution (8% of
the texts in Round 1 and 5% of the text in Round 2). In Round 2 when the groups
worked without leaders they kept a similar level of accuracy but when they were not
sure about the steps needed to be followed to arrive at the correct solution
102
explanation, they did not mention procedure in their explanation (in Round 2 these
explanations increased from 3% to 16%, a significant increase). The majority of
correct answer procedural explanations in both rounds were stronger. The ratio
between weaker and stronger procedural explanations of the correct answer in
Round 2 was maintained in favour of stronger explanations, in particular of correct
but incomplete explanations (57% of the procedural explanations of the correct
answer in Round 2 were correct but incomplete). These findings point to groups
engaging meaningfully with procedural explanations of the correct answer. That
said, in Round 2 the number of full procedural explanations decreased (from 46% in
round 1 to 23% in round 2). Taking these findings together, the following
observation can be made: The significant decrease of full procedural explanations
and the significant increase of explanations without any and/or with less information
on procedure suggest that when groups worked without their group leaders their
performance on procedural explanations of the correct answer was weaker.

Conceptual explanations of the correct answer: Maintaining accuracy in conceptual
explanations of the correct answer was more difficult for the groups. There was a
significant decrease in inaccurate conceptual explanations of the correct answer
between rounds (17% in Round 1 to 10% in Round 2). This change was at the
expense of a significant increase in explanations which had no conceptual links (5%
in Round 1 increased to 29% in Round 2). About 30% of the explanations in Round 2
offered no conceptual links (almost double of the equivalent procedural explanations
category). Notwithstanding the quantitative difference, the trend between these two
explanations is similar: In Round 2, when the groups worked without leaders, when
they were not sure about the correct mathematical explanation, they tended to
provide explanations without conceptual links. Similarly to procedural explanations
of the correct answer, the majority of the conceptual explanations in both rounds
were stronger. The ratio between weaker and stronger conceptual explanations of
the correct answer in Round 2 was maintained in favour of stronger explanations.
These findings point to groups engaging meaningfully with conceptual explanations
of the correct answer. That said, in Round 2 the number of full conceptual
explanations decreased (from 40% in Round 1 to 16% in Round 2). Taking these
findings together, the following observation can be made: The significant decrease in
full conceptual explanations, the insignificant improvement of partial conceptual
explanations, and the overall increase of weak explanations in Round 2, suggest that
when groups work without their group leaders their performance on conceptual
explanations was weaker.
It was also weaker than their performance on procedural explanations. In both
rounds the groups started with 40% texts with full explanations (procedural and
conceptual). The decrease in Round 2 in both of these explanations was significant,
but higher in conceptual explanations. Whilst close to 60% of the procedural
103
explanations of the correct answer were accurate but incomplete, their parallel in
conceptual was less than 50%.

Awareness of error: In Round 2, when the groups worked without a group leader,
they significantly reduced the number of inaccurate texts (or incomplete and hence
potentially confusing explanations). Only 6% of Round 2 awareness of error
explanations was inaccurate. This figure is consistent with the very low number of
inaccurate procedural explanations of the correct answer in Round 2 (5%). As in the
above subject matter knowledge-related explanations of the correct answer, groups
decreased inaccuracies in their explanations of the error, which is an important
improvement across these three criteria.
On two categories of awareness explanations, the groups’ performance did not
change: In both rounds, 20% of the texts offered no mathematical awareness of the
error (this finding is consistent with the percentage of non-feasible explanations
found in Round 1 and 2 texts, which was about 20%, see below). Second, in both
rounds the number texts that demonstrated a full awareness of errors (in which the
explanation of the error is mathematically sound and suggests links to common
errors) remained at about 25% of the total error-related texts. Other than reduction of
inaccuracy, a notable change in Round 2 was found in the 3rd category of this
criterion: In 50% of the error texts in Round 2, the explanation of the error was
mathematically sound but did not link to common errors. Taken together, the
following observation can be made: when the groups worked without leaders, the
ratio between stronger and weaker explanations remained in favor of stronger
awareness of error in their explanations. The percentage of stronger explanations
improved mainly due to the increase to 50% of the 3rd category explanation (see
above). Together with the significant decrease of mathematically inaccurate texts, the
groups maintained their performance in Round 2.
Finding 5: There is a correlation between groups’ awareness of error and their
diagnostic of learners’ reasoning behind the error
The correlation between awareness of the mathematical error and diagnostic reasoning
was also high and increased in Round 2 (r = 0,651 in Round 1 and r = 0,71 in Round
2).When groups demonstrate high awareness of the mathematical error (SMK) they are
more likely to give the appropriate diagnosis of the learner thinking behind that error
(PCK).
The correlation between awareness and diagnostic reasoning merits reflection. The
lesson it provides is that when teachers can describe the error, mathematically well
(SMK), they are more likely to be able to delve into the cognitive process taken by the
learners and describe the reasoning that led to the production of the error (PCK).
104
Although this is only a correlational finding, tentatively we suggest that improving
teachers’ mathematical awareness of errors could help improve teachers’ diagnostic
reasoning. Furthermore in view of the finding that teachers’ procedural explanation of
the mathematics underlying the correct answer is itself weak (missing steps,
incomplete), we suggest that the finding that the teachers struggled to describe the
mathematical way in which the learners produced the error, is expected. All of this has
implications for the relationship between SMK and PCK. The correlation between
awareness and diagnostic reasoning, and between procedural and conceptual
knowledge, bring the importance of subject matter knowledge in teaching.
Finding 6: Groups struggled to describe learners’ reasoning behind the error
Close to 30% of the texts in both rounds did not attempt to explain learners’ error and
another 27% of the texts in both rounds described learners’ reasoning without honing in
on the error. Altogether more than 50% of the texts in both rounds demonstrated weak
diagnostic reasoning. About 33% of the texts in both rounds honed on the error but the
description of learner reasoning was incomplete. In both rounds then, in about 90% of
the texts, groups offered no or incomplete explanation of learners’ reasoning. Only 12%
of texts in each of the rounds were systematic and honed in on the learners’ reasoning
about the error. This is the criterion in which groups’ performance was the weakest,
more so (albeit, insignificantly) in Round 1, and proportionally more in the Grade 3-6
group. As in awareness of error explanations, the groups performed better on texts that
do hone in on the error but are incomplete. The weakness in explaining learners’
reasoning is also evident and is consistent with the difficulty of the groups to produce
more than one explanation to explain the error. All groups provided mainly one
mathematically feasible/convincing explanation (with/without general explanations).
60% of the items in Round 1 and 85% of the items in Round 2 demonstrate little use of
multiple explanations.
These two findings suggest that groups struggle to think about the mathematical content
covered by the item from the perspective of how learners typically learn that content.
According to Ball et al (Ball, Thames and Bass and Ball, 2011), this type of thinking
implies “nimbleness in thinking” and “flexible thinking about meaning”. The groups’
difficulty to think meaningfully about rationales for the ways in which the learners were
reasoning and their inability, even in a group situation, to think of alternative
explanations is evident to them lacking these two qualities in the way they approach
errors.
Finding 7: In Round 2 across all criteria the number of texts without a mathematical
explanation increased while the number of inaccurate texts decreased
The number of texts with mathematically inaccurate explanations generally decreased in
Round 2, which suggests that just by doing more items, even without leaders, the
likelihood of producing explanations which were not mathematically flawed improved.
105
Bearing this in mind, it was interesting to note that while inaccurate texts decreased in
number between Rounds 1 and 2, texts that did not include a mathematical explanation
increased. This could be an indication that the group leaders seem to be more useful,
albeit not significantly so, in enhancing awareness of the mathematical concepts and in
focusing the explanations on the mathematical content of the question since in Round 2
when they were absent, a higher number of explanations that did not have mathematical
substance were given.
Is there a difference between primary (Grade 3-6 groups) and high school (Grade 7-9 groups)
teachers in sustaining the performance on the Round 1 error analysis activity?
Finding 8: The Grade 7-9 group held better quality when working without group
leaders than the Grade 3-6 group
An examination of the different performance by the two sets of grouped grades in the
weaker and stronger level criteria both point to the better quality held by the Grade 7-9
group when they worked without a group leader in Round 2. One particularly notable
change was that in Round 2 on the diagnostic reasoning criteria there was a high
decrease in inaccurate mathematical explanations and a corresponding high increase in
correct though partially complete mathematical explanations for the Grade 7-9 group.
This is very different from the general pattern where the decrease in inaccurate
explanations was seen in conjunction with an increase in general texts. This suggests
that the learning in this group was strongest in relation to the diagnosis of error.
Finding 9: Teachers reasoning in relation to mathematical concepts and errors seems
to be strongest in the content area of number while weakness is spread across the
areas of data, shape, algebra and measurement
Number is the content area which is most often taught in schools and so this finding
corresponds with the knowledge expected of teachers. It is interesting to note that
diagnostic reasoning was strongest in the area of measurement and that most multiple
explanations were offered in the area of shape. Use of the everyday in explanations was
done very little, but the two content areas in which most (albeit very little) reference to
the everyday in explanations were number and measurement. These are findings that
could be further investigated.
Data is a relatively “new” content area in the South African mathematics curriculum
which could explain the weakness here while algebra and shape are recognized
internationally as more difficult mathematical content areas.
106
What does the change in performance between Round 1 and 2 suggest about the role of group
leaders?
Finding 10: Group leaders are important
When worked with leaders, groups:
 completed more items,
 provided more full explanations and less partial explanations,
 provided more conceptual explanations and less procedural explanations,
 provided more mathematically focused explanations and less “general texts” types
of explanations,
 unpacked the concepts underlying the procedures more consistently, and
 gave more multiple explanations of errors.
107
108
5.4 Summary
In what follows we summarise the main findings of the quantitative analysis and draw
conceptual implications for the construct we propose to frame the idea of teacher’s
thinking about error- “diagnostic judgment”.
In both Rounds (more so in Round 2), groups tended to give mathematically accurate
but incomplete procedural and/or conceptual explanations of the correct answer and/or
of the error (SMK-related criteria). This means that groups were able to describe, albeit,
incompletely some of the steps that learners should follow and some of the conceptual
links they need to understand to arrive at the solution, and when the groups identified
the error, their level of mathematical accuracy was high. Only in very few texts were the
groups’ procedural and conceptual explanations of the correct answer found to be
inaccurate. This needs to be taken together with groups being found to give more
“stronger” than “weaker” explanations and maintaining this strength in Round 2.
This suggests that generally teachers’ did not mis-recognise the procedure or the concept
the correct answer consists off, and they could recognise an error. Nevertheless, in all
these three acts of recognition, many of their explanations is incomplete and some were
inaccurate. The implication we draw from this is that in order for teachers to be able to
improve their error recognition and their explanations of the correct answer, they need
to develop their content knowledge so that they will be able to produce fuller
explanations. To repeat, the distinctive feature of teachers’ knowledge, which
distinguishes them from other professionals who need mathematics for their work is
that “teachers work with mathematics in its decompressed or unpacked form” (Ball,
Thames and Bass, 2008b, p400). When teachers’ subject matter knowledge is strong they
will have acquired an ability to judge when and how to move from compressed to
unpacked mathematical knowledge, and how to provide explanations that is both
accurate and full. What this study of error analysis suggests is that teachers’ when
explanations are basically correct their incomplete form may cause confusion.
Incomplete explanations, we suggest signal weak content knowledge.
In the following example we give two explanations given for a grade 8 question. The
first explanation is of the correct answer and it is an example of a basically correct but
incomplete explanation. The second explanation is of the error. It is both inaccurate and
confusing.
109
Table 26: An example of an explanation that is accurate but incomplete
ICAS 2006, Grade 8, Question 10
Correct answer: B
Selected distractor: D
Content Area: Geometry – Rotational symmetry
Teacher explanation of the correct answer
By removing the star in the centre all the other stars
would still be symmetrical.
Limitations of explanation of the correct answer
This explanation is a typical compressed explanation
where the concept is assumed not elaborated.
Furthermore what is included by way of explanation is
that the removal of the centre star is the correct solution
but is not an explanation. Teachers should explain that
learners need to visualise rotation to determine the
centre is not affected by rotation and therefore it can be
removed. This could be based on a broader explanation
of the meaning of rotational symmetry so that the
explanation is generalisable.
Teacher explanation of the error
The learners could have imagined an axis of symmetry
between A and B. they could have then counted out the
number of stars on either side which totals five stars,
therefore D is out.
Limitations of explanation of the error
This explanation further demonstrates the lack of
knowledge of the concept - rotational symmetry. The
counting of the stars on either side of a line relates to
line symmetry and not rotational symmetry. The
position of the imagined line of symmetry (which
would not even be used to find the rotational
symmetry) is not well described
Teachers particularly struggled to try and imagine learners’ reasoning, to offer variety of
explanations, and to meaningfully reflect on the everyday context of questions and link
this to the mathematical context of questions. Groups’ judgment about learners’ thinking
(PCK) was very weak and their ability to approach leaners’ thinking in diversified ways
was also weak. So despite demonstrating a very small percentage of inaccurate
explanations, the aspect of subject matter knowledge that came to the fore in this study
is the incompleteness in groups’ explanations of the error (awareness of error), which
110
seems to correlate with explanations that hone in on learners’ thinking (diagnostic
reasoning). This suggests a relationship of dependence between:


what teachers can do creatively for learners (explain ideas in a differentiated way,
connect between every day and mathematical contexts of questions, PCK) and
imagine the ways learners think (PCK)
and the quality of the subject matter knowledge they acquire and develop.
In the following example we give two explanations given for a grade 5 question. In this
example both explanations, of the correct answer and of the error, demonstrate
insufficient content knowledge. In the case of the error text this leads to an inability to
make a full diagnostic judgement.
Table 27: Grade 5 test item explanations illustrating poor diagnostic judgement
ICAS 2007, Grade 5, Question 16
Correct answer: B
Selected distractor: D
Content Area: Numbers and Operations – completing an equation
Teacher explanation of the correct answer
The learner could have subtracted 6 from 21
and divided by 3.
Limitations of explanation of the correct
answer
This explanation offers no conceptual links.
Conceptually an understanding of the equality
of the two sides of an equation underlies the
solution.
Teacher explanation of the error
The learner added 6 and 21 and got 27 and
divided 27 by 3 to get 9
Limitations of explanation of the error
Procedurally this could be exactly what a
learner may have done to reach this incorrect
solution but the explanation does not hone in
on the misconception that may have led to such
procedural activity. This misconception relates
to a lack of understanding of the equality of the
two sides of an equation the learners reposition
the equal sign without adjusting the operations
accordingly.
111
Notwithstanding, three strengths of the project can be recognised in the comparison
between Rounds 1 and 2:



The main one being that in Round 2, groups gave fewer explanations which were
mathematically inaccurate. This is an important achievement that needs to be
recognised. In Round 2, when groups worked without a group leader they managed
to decrease their inaccuracy across the different criteria. As discussed before, this
improvement within the weaker category of explanations often came at the expense
of an increase in texts that did not include mathematical explanations relevant to the
criteria. We propose that having participated other DIPIP project activities and
listened to feedback, groups were more cautious, and as a result, when they were not
sure (and having no group leader to lean on), they did not give mathematical
explanations.
Secondly, important to acknowledge as an achievement, that when analysing the
correct answer and the error, in about 70%-80% of the texts, groups drew primarily
on mathematical knowledge and much less so, on other discourses.
Thirdly, in Round 2, in the three SMK-related criteria, most of the explanations
remained within the two stronger categories.
112
Section Six: Implications for professional
development: developing diagnostic judgement
113
The central predicament of audit culture is that it produces massive amount of data,
which is good for “external accountability” but often remains remote from “internal
accountability”. Elmore who made this distinction notes the following:
Internal accountability precedes external accountability. That is, school personnel
must share a coherent, explicit set of norms and expectations about what a good
school looks like before they can use signals from the outside to improve student
learning. Giving test results to an incoherent, atomized, badly run school doesn't
automatically make it a better school. The ability of a school to make improvements
has to do with the beliefs, norms, expectations, and practices that people in the
organization share, not with the kind of information they receive about their
performance. Low-performing schools aren't coherent enough to respond to external
demands for accountability … Low-performing schools, and the people who work in
them, don't know what to do. If they did, they would be doing it already. You can't
improve a school's performance, or the performance of any teacher or student in it,
without increasing the investment in teachers' knowledge, pedagogical skills, and
understanding of students. This work can be influenced by an external accountability
system, but it cannot be done by that system. (2002, 5-6)
In other words, the important and very complicated condition for achieving the link
between teachers’ tacit knowledge, their motivation to change and performance is
teachers' belief about their competence. This belief is linked directly to their learning
experiences, to meaningful opportunities to learn what specific curriculum standards
actually mean. The argument about internal accountability, first, is that with sufficient
meaningful professional development, teachers can be helped to see that their efforts can
bear fruits (Shalem, 2003). Teachers need to be able to see the reasons for change,
understand its core principles and be convinced that it is feasible and will benefit their
learners. This gives rise to the question of what constitutes a meaningful learning
opportunity for teachers. “Alignment” between formative and summative assessments
is the notion which the international literature uses to describe the reciprocity that is
needed for teacher learning.
International research shows that just having another set of data (in the form of
benchmarking, targets and progress reports) that ‘name and shame’ schools leads to
resentment and compliance but not really to improvement of learning and teaching
(McNeil, 2000; Earl & Fullan, 2003; Fuhrman & Elmore, 2004). In South Africa, Kanjee
(2007) sums up the challenge:
For national assessment studies to be effectively and efficiently applied to improve the
performance of all learners, the active participation of teacher and schools is essential. …
Teachers need relevant and timeous information from national (as well as international)
assessment studies, as well as support on how to use this information to improve
learning and teaching practice. Thus a critical challenge would be to introduce
appropriate polices and systems to disseminate information to teachers. For example,
teacher-support materials could be developed using test items administered in national
assessments. (p. 493)
114
Katz et al (2005) draw an important distinction between “accounting” and
“accountability” (which follows on from Elmore’s distinction between external and
internal accountability). They define the former as the practice of gathering and
organising of data for benchmarking, which is the mechanism that the Department of
Education has put in place in order to ensure the public that it gets a good service for its
investment. They define the latter as “teachers-led educational conversations” about
what specific data means and how it can inform teaching and learning (Earl & Fullan,
2003; Earl & Katz, 2005, Katz et al, 2009). In this view, the international and local
benchmarking tests mentioned above form an “accounting” feature of the South African
educational landscape. Katz et al want to re-cast “accountability”. They want to recast it
from its reporting emphasis to making informed professional judgement, where
judgment is constructed through data-based conversations on evidence that is gathered
from systematic research and from assessments. They argue:
The meaning that comes from data comes from interpretation, and interpretation is a
human endeavour that involves a mix of insights from evidence and the tacit knowledge
that the group brings to the discussion. ... This is at the heart of what’s involving in
determining a needs-based focus. The process begins by tapping into tacit knowledge, by
engaging in a hypothesis generation exercise and recasting “what we know” into “what
we think we know”... Instead of suspending beliefs in the service of data or adamantly
defending unsubstantiated beliefs, the conversation is a forum for examining both and
making the interrelationship explicit. (Katz et al 2009, p28)
“Accountability” as phrased here emphasizes two ideas. First, in order for teachers to
learn from evaluation data what their learners “can and/or cannot do at a particular
stage or grade”29, they need to be engaging their tacit knowledge in some or other of a
structured learning opportunity (“recasting ‘what we know’ into ‘what we think we
know’). Secondly, accountability (in the above sense) constitutes teachers reflection as a
shared process, as a structured conversation between professionals. And so, since these
conversations are structured conversations, “accountability” enables teachers to hold
themselves and each other “to account” by explaining their ideas and actions, to each
other, in terms of their experiences as well as their professional knowledge (Brodie &
Shalem, 2010). In this way, Katz et al argue, accountability conversations can give
participants imagination for possibilities that they do not yet see and help them making
tacit knowledge more explicit and shared. 30 In our project, we used the evaluation data
of proficiency results on the ICAS test as an artefact around which to structure a
learning opportunity to experience a process of diagnostic assessment on learners’ error
in mathematic evaluation data.
Media statement issued by the Department of Basic Education on the Annual National Assessments (ANA): 04
February 2011.
http://www.education.gov.za/Newsroom/MediaReleases/tabid/347/ctl/Details/mid/1389/ItemID/3148/Default.aspx
29
30
For the idea of “Accountability conversations” see Brodie & Shalem, 2011.
115
Research has only recently begun to engage with the question of how to use the data
beyond that of an indicator of quality i.e. beyond benchmarking (Katz, Sutherland, &
Earl, 2005; Boudett, City & Muuname, 2005; Katz, Earl, & Ben Jaafar, 2009). Some
conceptual attempts to examine a more balanced way between accounting and
accountability include Shavelson et al (2002)31, Nichols et al (2009)32, and Black and
Wiliam (2006). In the South African context, Dempster (2006) and Dempster and Zuma ,
(2010), Reddy (2005) and Long (2007) have each conducted small case studies on testitem profiling, arguing that this can provide a useful data that can be used for formative
and diagnostic purposes.
Arguably, awareness about learners’ errors is a useful step in the process of significantly
improving practice. Hill and Ball (2009) see “analyzing student errors” as one of the four
mathematical tasks of teaching “that recur across different curriculum materials or
approaches to instruction (p70)33. Studies on teaching dealing with errors show that
teachers’ interpretive stance is essential for the process of remediation of error, without
which, teacher simply re-teach without engaging with the mathematical source of the
error, or with it metacognitive structure (Peng, 2009, 2010; Prediger, 2010; Gagatsis &
Kyriakides, 2000).
Shavelson et al devised what they call “a logical analysis” that test-designers or teachers can use in order to
“psychologize about the nature of the problem space that a student constructs when confronted with an assessment task”
(2002, p15). They argue that effective error analysis should use a “logical analysis” of task demands together with
“empirical analysis” of the kinds of cognitive processes a task, has in fact, evoked by students. In more common terms the
two steps include tasks analysis followed by an error analysis. In constructing the process of “logical analysis”, Shavelson
et al used a coding scheme consisting of numerous categories that together make up 4 types of knowledge- “declarative”
(knowing that) “procedural” (knowing how) , “schematic” (knowing why) and “strategic” knowledge (knowing when
knowledge applies). These were used to align the type of knowledge intended for Science tasks or their “construct
definitions” with the observations made about “the student’s cognitive activities that were evoked by the task as well as
the student’s level of performance” (p6). Shavelson et al nest this kind of analysis within what they call as “the assessment
square”, a framework that they develop to increase the alignment between the type of knowledge structure to be
measured (the 4 types of knowledge stated above), the kind of task designed for measuring learners’ performance of the
knowledge structure (including the design features of different tasks), the variety of responses elicited by the learners,
and the inferences made on the basis of these analyses. This type of analysis is useful for task analysis; it helps to unpack
the cognitive demands embedded in tasks, their degree of openness and level of complexity.
32 In their work, Nichols et al attempt to create a framework that will link between information from performance on a
particular assessment and student learning. Their model suggests several interpretive acts build in a sequence that
connects between three main data “structures”:

“student data” or sets of data derived from systemic assessment,

“knowledge domain model” or the knowledge and skills associated with a learning construct,

“a student model” or the representation of current understanding of a learner’s or specific class of learners’
knowledge and

“a teaching model” or a sets of methods that have been shown to be effective in relation to the other domains (the
knowledge and the student domain).
What holds these structures together and gives them meaning are the recurring interpretive acts required by the teacher:
Data interpretation involves reasoning from a handful of particular things students say, do, or make in particular
circumstances, to their status on more broadly construed knowledge, skills, and abilities that constitute the student
model. (p17)
33 The others being: “encountering unconventional solutions, choosing examples, or assessing the mathematical integrity
of a representation in a textbook”. (ibid)
31
116
In South Africa, Adler sees teachers’ knowledge of error analysis as a component of
what she calls mathematics for teaching (2005, p3). She asks:
What do teachers need to know and know how to do (mathematical problem solving) in
order to deal with ranging learner responses (and so some error analysis), and in ways
that produce what is usefully referred to as “mathematical proficiency”, a blend of
conceptual understanding, procedural fluency and mathematical reasoning and problem
solving skills? (ibid)
From this point of view, teachers’ diagnostic judgment is essential; it is used to make
decisions which affirm learners’ performance, but also to make decisions which classify
learners and select them - or not - for particular futures. What does DIPIP phases 1&2
research suggest diagnostic judgment entails?
Diagnostic Judgement
Prediger’s argument that teachers who have an interest in learners’ rationality are aware
of approaches to learning implies that diagnostic judgment encompasses far more than
reasoning about the way in which learners think mathematically when they answer
mathematics test questions. This implies that diagnostic judgement should include
understanding about learners, such as their contexts, culture, learning styles and
possible barriers to learning. It also implies that diagnostic judgement will look different
in different teaching practices, for example during teaching, when planning a lesson
planning, and when making assessments.
The data of this project suggests that error analysis of mathematical evaluation data
relies on a firm SMK- specifically correctness and fullness of knowledge of the steps to
be followed to arrive at the solution and the most relevant concept underling the
question. The distinction between accuracy and fullness is important. Whilst a general
mathematician can figure out an accurate solution with little attention to the number,
kind and sequence of steps to be followed, for a teacher this is different. Teachers need
to know both- they need to be able to distinguish between explanations that are correct
but are not completely full from those that are correct and full. Teachers need to unpack
what is compressed behind the correct answer or behind an error. This has to do with
their role in socialising learners into doing mathematics. Unlike the general
mathematician or other professionals, teachers need to show learners which explanation
is better and why, what steps are there to be followed, in what sequence and why. As
the analysis of teacher knowledge suggests, without a full knowledge of the explanation,
teachers may recognize the error only partially, as they may not know what the crucial
steps that make up the solution are or their sequence and the conceptual relation
between them. Without thorough knowledge of the mathematical content and
procedures relating to that content (Common Content Knowledge) it is unlikely that
teachers will be able to systematically look for “a pattern in students error” or “size up
117
whether a non-standard approach would work in general” (Specialized Content Knowledge
, see Ball, Thames &Bass, 2008, p400).
The study shows that there is a conceptual interdependence between the procedural and
conceptual aspects in teachers’ explanation of the mathematics that underlie a
mathematical problem. It also suggests that there is interdependence between awareness
of the mathematical error and the quality of teachers’ diagnostic reasoning. Together this
means that diagnostic reasoning is dependent on prior content knowledge, that
following the learner’s reasoning is dependent on teachers having a full picture of what
the question is in fact asking, how is it asking it, what is the underlying concept of the
question, what kind of procedure it relies on and what are the crucial steps that have to
be completed. Only with this knowledge, contextual factors about learners’
misrecognition can add value to how teachers are analysing learners’ errors. So we
propose a vertical form construct to describe teachers’ knowledge of errors: When
teachers can describe the steps and the underlying concepts of a question in full (or
mathematically well), they are more likely to be able to size up the error or interpret its
source (SMK). When teachers can describe the error, mathematically well (SMK) they are
more likely to be able to delve into the cognitive process taken by different kinds of
learners and describe the reasoning that led to the production of an error (PCK).
The idea of developing teachers’ diagnostic Judgement is implied by other research in
South Africa and internationally
The teacher today is required to demonstrate different types of professional knowledge
which range from knowledge of subject matter to recognizing that an answer is wrong,
or that a particular mathematical solution cannot be accepted because it cannot work in
general, to much more subtle kinds of knowledge such as 'knowledge of self'. This kind
of teacher is regarded a specialist for being able to teach subject matter knowledge with
attunement to the curriculum, to the knowledge field and to the learner. This kind of
attunement requires a distinct mode of awareness that is often informed by dispositions
and perspectives that are not readily accessible to teachers in terms of their pre-service
training, more so in South Africa (Adler & Hulliet, 2008, p19). This is consistent with the
works of Nesher (1987), Borasi (1994), Gagatsis and Kyriakides (2000), Kramarski and
Zoldan (2008), Predriger (2010) which show that a focus on learner’s mathematical
thinking when producing errors provides a mechanism for teachers to learn to think
more carefully about their teaching and for learners to change their attitudes towards
errors (Borasi, 1994, p189). Borasi argues that experiencing learning through errors
shifted learners’ conception about the use of errors which until that point were seen
solely as mistakes that need to be remediated. In terms of teaching, Borasi argues, errors
are “a valuable source of information about the learning process, a clue that researchers
and teachers should take advantage of in order to uncover what students really know
and how they have constructed such knowledge” (p170).
118
The development of “formative assessment” opens up new possibilities for teachers
with regard to this. The notion of formative assessment has changed the pedagogical
role of teachers in assessment. The following definition proposed by Black and Wiliam,
explains the idea of “formative”, grounding it in awareness of learners’ errors.
Practice in the classroom is formative to the extent that evidence about student
achievement is elicited, interpreted, and used by teachers, learners, or their peers, to
make decisions about the next steps in instruction that are likely to be better, or better
founded, than the decisions they would have taken in the absence of the evidence that
was elicited. (2009, p9)
At the core of this view is a pedagogical practice that consists of a sequence of activities
structured by the teacher in response to evidence s/he gathers about students’ learning
from classroom and/or systemic assessment. This also includes classroom discussions
amongst learners and their peers and amongst teachers on the meaning of evaluative
criteria, on errors that appear to be common, and on ways of addressing them (See also
Brookhart, 2011). Having these kinds of engaging discussions, Black and Wiliam argue,
will promote self-regulation and autonomy in students and “insight into the mental life
that lies behind the student’s utterances” in teachers (2009, p13). Nichols et al (2009)
offer the idea of “a reasoned argument” which they believe is the key to a successful
process of for formative assessment. For them formative assessment is valid, when it is
used to inform instruction and its use to inform instruction is based on a valid
interpretation:
The claim for formative assessment is that the information derived from students’
assessment performance can be used to improve student achievement. It is how that
information is used, not what the assessment tells us about current achievement, that
impacts future achievement. Therefore, use, based on a valid interpretation, is the primary
focus of the validity argument for formative assessments. (our emphasis, p15)
Nichols et al look for a model of formative assessment that can be trusted as valid. They
look for a model that will ensure that when teachers make judgments about learners’
reasoning behind a mathematical error (about why the learners think in that way, and
how their thinking can be addressed in teaching), they can account for the way they
reasoned about the evidence, and can show coherently how the evidence informed their
instructional plan. Their model was criticized strongly by Shepard 2009, p34) and for
good reasons, what is relevant for us, however, is the idea of “reasoned argument”, of
teachers needing to ask and interpret “what does the evidence from student data say
about students’ knowledge, skills and abilities” (p.16)? This idea is consistent with
Prediger’s idea of “diagnostic competence” (Teachers analyze and interpret learner
reasoning), Kats’ et al notion of “accountability” (ability to account for), and with Black’s
and Wiliam’s idea of formative assessment (evidence about student achievement is
elicited and interpreted). Together, they support the notion of we would like to call as
“diagnostic judgment”. To move away from the instrumentalist use of the current audit
119
culture, of a culture that uses evaluation data to mainly benchmark learner performance
(and to blame and shame teachers), professional development of teachers need to work
out ways to develop in teachers diagnostic judgment. In this we mean: the ability to find
evidence of learning and to know how to work with that evidence. As professionals and
as specialists of a particular subject matter, teachers need to know how to elicit and
reason about evidence of learners’ errors.
Lessons about error analysis of mathematical evaluation data that can be taken from
DIPIP Phases 1 and 2:
The last question we need to address, is given these results, what could DIPIP have done
differently? Here we refer to the process document which describes the linear thinking
that informed the design of its main activities. The groups followed the error analysis
with two different applications- lesson plan and teaching. All the groups worked on
their lesson plans for several weeks and repeated it. Then all groups designed a test and
interviewed some learners on errors identified in their tests. Finally eleven groups
continued into a second round of curriculum mapping and error analysis while some
groups went on to a third round.. The idea behind this design sequence was that once
the groups mapped the curriculum and conducted error analysis, they will recontextualised their learning of the error analysis into two core teaching activities. In
other words, they will start with “error-focused” activities and move to “error-related”
activities and then go back to error-focused activities. This will enable them to be more
reflective in their lesson design and in their teaching as in both of these activities the
groups will work with what the curriculum requires (assessment standards) but also
with errors to be anticipated in teaching a particular mathematical topic. This design
sequence was also informed by ideas from the field of assessment, in particular the
emphasis on integration between teaching and assessment - the key construct in the idea
of alignment between standardised and formative assessment.
The above analysis of teachers’ thinking about learners’ errors in evaluation tests
suggests that the idea of sequential development between these activities was too
ambitious. To ask teachers to apply their error knowledge which at most was only
partial in terms of its SMK aspect and very weak in terms of its diagnostic of learners
reasoning was certainly too ambitious, at least for teachers in SA. Teachers needed
repeated practice in “error-focused activities” before they can be expected to transfer
this knowledge into Knowledge of content and teaching (KCT), Ball et al’s 4th domain of
teacher knowledge (or second domain within PCK). The idea was ambitious in two
ways. First teachers needed more repetitive experience in doing error analysis. Error
analysis of evaluation data is new for teachers and they needed to develop their
competence in doing more of the same. They also needed a variation of questions. There
are pros and cons in working on multiple content questions and groups needed to work
with a variety of questions. Secondly and even more importantly, our analysis shows
120
that teachers need to work on errors in relation to specific mathematical content. This is
consistent with Cohen and Ball (1999) who argue that change of teaching practice comes
as support for a certain content area in the curriculum and not vice versa. Teachers
cannot change their teaching practices in all the content areas they are teaching (Cohen
and Ball, 1999, p.9).34 If accepted, the conception of diagnostic judgement that we
propose shows that teachers need to know the content well and think about it carefully
when they are working on errors.
34
Cohen, D. K. and Ball, D. L. (1999) Instruction, Capacity and Improvement CPRE Research Report Series RR-43.
121
122
Recommendations for professional development and
for further research
DIPIP Phases 1 and 2 set out to include teachers in different professional activities all
linked to evaluation data and focused on errors. It had a development and a research
component. In this report we evaluate the error analysis activity and not the whole
process of development this is done in the process report. Based on the experience of
this project activity we make the following recommendations with regard to teacher
development and further research.
Recommendations for professional development







Error analysis activities should be content-focused.
Different error-focused activities should be used and coherently sequenced. These
should include error analysis in relation to different forms of mathematical questions
(open and multiple choice); the use of systemic tests and tests set by the teachers;
design of different kinds of mathematical questions, interviewing learners and
design of rubrics for evaluation of learners.
Error analysis should be done in conjunction with curriculum mapping as this helps
teachers to understand the content and purpose of questions. We suggest that this
should be done with a specific content focus.
Error analysis activities should be situated within content development
programmes. Structured content input is necessary for teachers in South Africa since
this is a prerequisite for diagnostic judgement.
In order to avoid compromising on quality modeling of what counts as full
explanations (including all procedural steps and conceptual links) is necessary. At all
times and through all activities the emphasis on full explanations must be made
explicitly which we believe will strengthen teachers’ content knowledge.
Learning materials for content-focused professional development of error analysis
should be developed. This was one of the aims of DIPIP 1 and 2.
Expert leadership of groups working on error analysis activities is important to
enable quality instruction of mathematical content, promote depth of analysis and
fullness and variety of explanations. Never-the-less judgement should be made as to
when is the appropriate time to remove this expert leadership.
Recommendations for further research


More research is needed to identify different qualities of mathematical content
specific procedural explanations. This could inform the teaching of mathematical
content.
Further research could investigate other discourses teachers draw on when they
analyse error on a standardized test and on their own tests.
123



A more in-depth study could be done on the use of everyday contexts in the
explanation of mathematical concepts and learners’ errors.
The role of expert leaders in supporting teacher development through error analysis
needs further investigation.
More research is needed to inform ways of aligning evaluation data results and
teachers’ practice.
124
References
Adler, J (2005). Mathematics for teaching: what is it and why is it important that we talk
about it? Pythagoras, 62 2-11.
Adler, J. & Huillet, D. (2008) The social production of mathematics for teaching. In T.
Wood S. & P. Sullivan (Vol. Eds) International handbook of mathematics teacher
education: Vol.1. Sullivan, P., (Ed.), Knowledge and beliefs in mathematics teaching
and teaching development. Rotterdam, the Netherlands: Sense Publishers. (pp.195222). Rotterdam: Sense.
Ball, D.L. (2011, January) Knowing Mathematics well enough to teach it. Mathematical
Knowledge for teaching (MKT). Presentation for the Initiative for Applied
Research in Education expert committee at the Israel Academy of Sciences and
Humanities, Jerusalem, Israel, January 30, 2011.
Ball, D.L., Hill, H.C., & Bass, H. (2005) “Knowing mathematics for teaching: who knows
mathematics well enough to teach third grade, and how can we decide?”
American Educator, 22: 14-22; 43-46.
Ball, D.L., Thames, M.H., & Phelps, G. (2008) “Content knowledge for teaching: what
makes it special?” Journal of Teacher Education 9 (5): 389-407.
Borasi, R. (1994) “Capitalizing on errors as “springboards for inquiry”: A teaching
experiment in Journal for Research in Mathematical Education, 25(2): 166-208.
Boudett, K.P., City, E. & Murnane, R. (2005) (Eds) Data Wise A Step-by-Step Guide to Using
Assessment Results to Improve Teaching and Learning. Cambridge: Harvard
Education Press
Black, P. & Wiliam, D. (2006) Developing a theory of formative assessment in Gardner,
John (ed.) Assessment and Learning (London: Sage).
Brookhart S M (Ed) (2009) Special Issue on the Validity of Formative and Interim
assessment, Educational Measurement: Issues and Practice, Vol. 28 (3)
Brodie, K. (2011) Learning to engage with learners’ mathematical errors: an uneven
trajectory. In Maimala, T and Kwayisi, F. (Eds) Proceedings of the seventeenth
annual meeting of the South African Association for Research in Mathematics,
Science and Technology Education (SAARMSTE), (Vol 1, pp. 65-76). North-West
University, Mafikeng: SAARMSTE.
Brodie, K. & Berger, M (2010) Toward a discursive framework for learner errors in
Mathematics Proceedings of the eighteenth annual meeting of the Southern
African Association for Research in Mathematics, Science and Technology
Education (SAARMSTE) (pp. 169 - 181) Durban: University of KwaZulu-Natal
Brodie, K. & Shalem, Y. (2011) Accountability conversations: Mathematics teachers’
learning through challenge and solidarity, Journal of Mathematics teacher Education
14: 420-438
125
Dempster, E. ( 2006). Strategies for answering multiple choice questions among south
African learners: what can we learn from TIMSS 2003, in Proceedings for the 4th
Sub-Regional Conference on Assessment in Education, UMALUSI.
Dempster, E. & Zuma, S. (2010) Reasoning used by isiZulu-speaking children when
answering science questions in English. Journal of Education 50 pp. 35-59.
Departments of Basic Education and Higher Education and Training. (2011). Integrated
Strategic Planning Framework for Teacher Education and Development in South
Africa 2011-2025: Frequently Asked Questions. Pretoria: DoE.
Earl, L & Fullan M. (2003) Using data in Leadership for Learning (Taylor &Francis Group:
Canada)
Earl, L., & Katz, S. (2005). Learning from networked learning communities: Phase 2 key
features and inevitable tensions.
http://networkedlearning.ncsl.org.uk/collections/network-researchseries/reports/nlg-external-evaluation-phase-2-report.pdf.
Elmore, RE. 2002. Unwarranted intrusion. Education next. [Online]. Available url:
http://www.educationnext.org/20021/30.html
Fuhrman SH & Elmore, RE (2004) Redesigning Accountability Systems for Education.
(Teacher College Press: New York).
Gagatsis, A. and Kyriakides, L (2000) “Teachers’ attitudes towards their pupils’
mathematical errors”, Educational Research and Evaluation 6 (1): 24 – 58.
Gipps, C. (1999) “Socio-cultural aspects of assessment”. Review of Research in Education
24. pp 355-392.
Heritage, M., Kim, J., Vendlinski, T., & Herman, J. (2009), From evidence to action: A
seamless process in formative assessment? Op cit pp. 24-31. In S. Brookhart (Ed)
(2009) Special Issue on the Validity of Formative and Interim assessment,
Educational Measurement: Issues and Practice, Vol. 28 (3)
Hill, H.C., Ball, D.L., & Schilling, G.S. (2008a) “Unpacking pedagogical content
knowledge: conceptualizing and measuring teachers’ topic-specific knowledge of
students” Journal for Research in Mathematics Education 39 (4): 372-400.
Hill, C. H., Blunk, M.L., Charalambos, C.Y., Lewis, J.M., Phelps, G.C., Sleep, L &Ball, D.L
(2008b) Mathematical knowledge for teaching and the mathematical quality
instruction: an exploratory study. Cognition and Instruction 26 (4) 430-511.
Hill, H.C. & Ball, D.L. (2009) “ The curious- and crucial- case of mathematical knowledge
for teaching Kappan 9 (2): 68-71.
Katz, S., Sutherland, S., & Earl, L. (2005). “Towards an evaluation habit of mind:
Mapping the journey”, Teachers College Record, 107(10): 2326–2350.
Katz, S., Earl, L., & Ben Jaafar, S. (2009). Building and connecting learning communities: The
power of networks for school improvement. Thousand Oaks, CA: Corwin.
126
Kramarski, B. & Zoldan, S. (2008). Using errors as springboards for enhancing
mathematical reasoning with three metacognitive approaches. The Journal of
Educational Research, 102 (2), 137-151.
Louw, M. & van der Berg, S & Yu, D. (2006) "Educational attainment and
intergenerational social mobility in South Africa Working Papers 09/2006,
Stellenbosch University, Department of Economics
Looney, J. W. (2011). Integrating Formative And Summative Assessment: Progress
Toward A Seamless System? OECD Education Working Paper No. 58 EDU/WKP
4 http://essantoandre.yolasite.com/resources.pdf
Long, C. (2007) What can we learn from TIMSS 2003?
http://www.amesa.org.za/AMESA2007/Volumes/Vol11.pdf
McNeil, LM. (2000) Contradictions of school reform: Educational costs of standardized testing.
New York: Routledge.
Muller, J. (2006) Differentiation an Progression in the curriculum in M. Young and J.
Gamble (Eds). Knowledge, Curriculum and Qualifications for South African Further
Education. Cape Town: HSRC Press.
Nesher, P. (1987) Towards an instructional theory: The role of learners’ misconception
for the learning of Mathematics. For the Learning of Mathematics, 7(3): 33-39.
Nichols, P D Meyers, J L & Burling, K S (2009) A Framework for evaluating and
planning assessments intended to improve student achievement in S. Brookhart
(Ed), (2009), op cit pp. 14–23.
Peng, A. (2010). Teacher knowledge of students’ mathematical errors.
www.tktk.ee/bw_client_files/tktk_pealeht/.../MAVI16_Peng.pdf
Peng, A., & Luo, Z. (2009). A framework for examining mathematics teacher knowledge
as used in error analysis. For the Learning of Mathematics, 29(3), 22-25.
Prediger, S (2010) “How to develop mathematics-for-teaching and for understanding:
the case of meanings of the equal sign” , Journal of Math Teacher Education 13:73–
93.
Radatz, H. (1979) Error Analysis in Mathematics Education in Journal for Research in
Mathematics Education, 10 (3): 163- 172 http://www.jstor.org/stable/748804
Accessed: 08/06/2011 05:53.
Reddy, V. (2006) Mathematics and Science Achievement at South African Schools in
TIMSS 2003. Cape Town: Human Sciences Research Council.
Rusznyak, L. (2011) Seeking substance in student teaching in Y. Shalem & S Pendlebury
(Eds) Retrieving Teaching: Critical issues in Curriculum, Pedagogy and learning.
Claremont: Juta. pp 117-129
Shalem, Y. (2003) Do we have a theory of change? Calling change models to account.
Perspectives in Education 21 (1): 29 –49.
Shalem, Y., & Sapire, I. (2011). Curriculum Mapping: DIPIP 1 and 2. Johannesburg: Saide
127
Shalem, Y., Sapire, I., Welch, T., Bialobrzeska, M., & Hellman, L. (2011). Professional
learning communities for teacher development: The collaborative enquiry process in the
Data Informed Practice Improvement Project. Johannesburg: Saide. Available from
http://www.oerafrica.org/teachered/TeacherEducationOERResources/SearchResults/tabi
d/934/mctl/Details/id/38939/Default.aspx
Shalem, Y. & Hoadley, U. (2009) The dual economy of schooling and teacher morale in
South Africa. International Studies in Sociology of Education. Vol. 19 (2) 119–134.
Shalem,Y. & Slonimsky,L. (2010) The concept of teaching in Y. Shalem & S Pendlebury
(Eds) Op cit. pp 12-23
Shavelson, R. Li, M. Ruiz-Primo, MA & Ayala, CC (2002) Evaluating new approaches to
assessing learning. Centre for the Study of Evaluation Report 604
http://www.cse.ucla.edu/products/Reports/R604.pdf
Shepard, L A (2009) Commentary: evaluating the validity of formative and Interim
assessment: Op cit pp. 32-37.
Smith, J.P., diSessa A. A., & Roschelle, J. (1993) Misconceptions Reconceived: A
Constructivist analysis of Knowledge in Transition. Journal of Learning Sciences
3(2): 115-163.
Star, J.R. (2000). On the relationship between knowing and doing in procedural learning.
In B. Fishman & S. O’Connor-Divelbiss (Eds), Fourth International Conference of the
Learning Sciences. 80-86. Mahwah, NJ: Erlbaum.
Taylor, N. (2007) “Equity, efficiency and the development of South African schools”. In
International handbook of school effectiveness and improvement, ed. T.
Townsend, 523. Dordrecht: Springer.
Taylor, N. (2008) What’s wrong with South African Schools? Presentation to the
Conference What’s Working in School Development. JET Education Services 28-29.
Van den Berg, S. (2011) “Low Quality Education as a poverty trap” University of
Stellenbosch.
Van den Berg, S. & Shepherd, D. (2008) Signalling performance: An analysis of
continuous assessment and matriculation examination marks in South African
schools. Pretoria: Umalusi
128
Appendices:
129
Appendix 1
Exemplar template- Error analysis – used in group leader training prior to the activity
Based on ICAS 2006, Grade 8 Question 22
Part A: Details
Grade
8
Question
22
Content
Solve a simple linear equation that has one
unknown pronumeral
Area
Algebra
Difficulty
27
Part B: Learner responses (%)
A
20
B
17
C
28
D
33
Correct answer
A
Analysis
More learners chose incorrect answers C and
D, together constituting three times the
correct choice A.
130
Part C: Analysis of correct answer
Correct Answer A – 20%
Indicate all possible methods
1. Using the distributive law multiply ½ into the binomial. Take the numbers onto
the RHS, find a LCM, multiply out and solve for y.
2. Find an LCM which is 2. Multiply out and solve for y.
3. Substitution of 13 on a trial and error basis.
In order to get this answer learners must know:
1. Learners need to know\how to determine a lowest common denominator (LCD)
and/or LCM.
2. Learners need to understand the properties of an equation in solution of the
unknown variable.
3. Learners need to know the procedure of solving an equation involving the RHS
and LHS operations.
Part D: Analysis of learners’ errors
For answer D – 33%
Indicate all possible methods
½(y+3) = 8
½
½
y+3 =4
y =1
Possible reasons for errors:
Learners do not understand the concept of dividing fractions. They
misconceive 8 divided by a half as a half of 8. The answer that they get is
4.
For answer C – 28%
Indicate all possible methods
a.
½(y+3) = 8
½
y+3 =4
y =7
131
or
b.
½ (7+3)=8
7+3 =8+2
or
c.
½ (y+3)=8
½ y+6 =8
½ y = 14
y =7
Possible reasons for errors:
In the first method the learners made the same error of dividing by a
fraction but did not change the sign when taking the terms over
(LHS/RHS rule)
Substitution of the value 7.
In this method the learners did not multiply every term by the LCM (2).
They also did not change the sign of the 6 when taking it over.
For answer B – 17%
Indicate all possible methods
a.
½ (y+3)=8
y+6 =16
y =10
b.
½ (10+3)=8
5+3 =8
c.
½ (y+3)=8
½ y =8-3
y =10
Possible reasons for errors:
Learners have made errors in the order of operations, substitution and the
distributive law.
132
Other issues with task
 The instruction could have read, “Solve for y”
Issues for teaching
 Learners need to be able to check solutions of equations.
 Instructions to the learners in the classroom differ to the instructions of the
task.
 Distributive Law
 Substitution
 Principles of rules/laws of equations i.e. LHS=RHS
133
Appendix 2
Template for DIPIP error analysis of ICAS task analysis (In both Round 1 and 2)
(2008 error analysis – All groups Grades 3,4,5,6,7,8,9)
(2010 error analysis – One group from Grades 3,4,5,6,8)
Part A: Details
Grade
Question
Content/Description
Area/LO
Part B: Learner responses (%)
Anticipated
achievement
(Before you check
the actual
achievement)
Expectation:
Comment on any particular distractors:
A
B
C
D
Correct answer
Difficulty
Analysis
Part C: Analysis of correct answer
The way(s) of thinking that the learner might use in order to get the question correct
(fill in more than one possible way of thinking, if necessary):
Indicate for each method whether you think it is at the expected level for the grade OR
higher OR lower than the group would expect.
134
1.
2.
3.
4.
5.
Part D: Analysis of learners’ errors
Analysis of each of the distractors, in order, from the one selected most often to the one
selected least often. You may trial these with your learners – please report on this if you
do so.
For each distractor, think about what learners might have done to obtain the answer
in each incorrect choice:
For choice – %
The way(s) of thinking that the learners might have used in order to get this
particular incorrect choice using this particular method (fill in more than one
possible way of thinking, if necessary) ie. Possible reasons for errors/Possible
misconceptions
1.
2.
3.
4.
For choice – %
The way(s) of thinking that the learners might have used in order to get this
particular incorrect choice using this particular method (fill in more than one
possible way of thinking, if necessary) ie. Possible reasons for errors/Possible
misconceptions
1.
135
2.
3.
4.
For choice – %
The way(s) of thinking that the learners might have used in order to get this
particular incorrect choice using this particular method (fill in more than one
possible way of thinking, if necessary) ie. Possible reasons for errors/Possible
misconceptions
1.
2.
3.
4.
Other issues with task not discussed above

Issues to consider when teaching (for example, how to over come some of the
problems that have been raised.)

136
Appendix 3
Template for DIPIP error analysis of own test task analysis (In Round 2)
(2010 error analysis – Grades 3,4,5,6,7,9)
Part A: Details
Grade
Question: Please
transcribe the full
question here
Content/Description:
Please describe the
mathematical
content of the
question and any
other relevant detail.
Area/LO
Part B: Learner responses
Anticipated achievement
Expectation:
Sample number (How many tests do you
have? Count and record this number.
You will use all of your tests every time
in your analysis)
Number of learners in sample that got
the item correct (count the number of
learners in your sample who got the
item perfectly correct)
Number of learners in the sample that
got the item partially correct (count the
number of learners in your sample who
did some partially correct working but
did not complete the item correctly)
Number of learners in the sample that
got the item wrong (count the number of
learners in the test who were awarded a
137
“zero” for the item)
Difficulty (work out the percentage of
the sample who got the item right. You
will do this for each item. Then place
them on a relative scale for your test of
easiest to hardest item.)
Part C: Analysis of correct answer
Go through your sample of learner tests to find as many different ways of getting the
question correct. Then record these ways and write about the way(s) of thinking that the
learner might use in order to get the question correct (fill in more than one possible way
of thinking, if necessary): USE YOUR LEANER TESTS to generate this list.
Indicate for each method whether you think it is at the expected level for the grade OR
higher OR lower than the group would expect.
1.
2.
3.
4.
5.
Part D: Analysis of learners’ errors
Go through your sample of learner tests to find as many different ways of getting the
question incorrect. Then record these ways and write about the way(s) of thinking that the
learner might use in order to get the question incorrect, i.e. Possible reasons for
errors/Possible misconceptions (fill in more than one possible way of thinking, if
necessary): USE YOUR LEANER TESTS to generate this list.
1.
2.
3.
138
4.
5.
Other issues with task not discussed above

Issues to consider when teaching (for example, how to overcome some of the
problems that have been raised.)

139
Appendix 4
Error analysis coding template – final version (exemplar with one Grade 3 items)
Grade
Grade
3
Grade
3
Ite
m
1
1
Maths
content
area
Numbe
r
Numbe
r
Distr
actor
C
B
text
no
Text
Proc
edur
al
Conc
eptu
al
Awa
renes
s of
error
Diag
nosti
c
reaso
ning
each item of text from template
Categ.
Categ.
Categ.
Categ.
1
They counted the books on the
shelf. To do this they must know
how to count correctly, be able to
identify a book as a unit and they
must distinguish between the
ends of a shelf and a book so that
they do not mistake the ends of a
shelf as another book.
2
They looked at shelf and saw 5
books on the left and 2 on the
right and added them together.
3
Counted every book that had a
label on it.
1
Subtracted the books that was
standing up straight on the left
and right and counted the three
books that are slanting in the
middle. They mistook the books
Use
of
ever
yday
Disti
nct
expla
natio
n
Comments
F/N
140
on the left and right as part of the
shelf.
2
Focused on the three thicker
books. The thicker books are more
visible.
141
Appendix 5
Error analysis – Template criteria for coding
Criteria
Explanations of the correct
solution
Procedural
The emphasis of this code is
on the teachers’ procedural
explanation of the solution.
Teaching mathematics
involves a great deal of
procedural explanation
which should be done fully
and accurately for the
learners to grasp and
become competent in
working with the
procedures themselves.
Conceptual
The emphasis of this code is
on the teachers’ conceptual
explanation of the
procedure/other reasoning
followed in the solution.
Mathematical reasoning
(procedural/other) needs to
be unpacked and linked to
the concepts to which it
relates in order for learners
to understand the
mathematics embedded in
the activity.
Explanations of the
incorrect solution (error in
distractor)
Awareness of error
The emphasis of this code is
on teachers’ explanation of
the actual mathematical
error and not on learners’
Category descriptors: Error analysis
Full
Partial
Inaccurate
Not Present
No
mathematica
l procedural
explanation
is given
Teachers’ explanation
of the learners’
mathematical reasoning
behind the solution
includes demonstration
of procedure.
Teachers’ explanation of
the learners’
mathematical reasoning
behind the solution
includes demonstration
of procedure.
Teachers’ explanation
of the learners’
mathematical reasoning
behind the solution
includes demonstration
of procedure.
The procedure is
accurate and includes
all of the steps in the
procedure.
The procedure is accurate
but it does not include all
of the steps in the
procedure.
Teacher’s use of
procedure is inaccurate
or incomplete to the
extent that it could be
confusing.
Teachers’ explanation
of the learners’
mathematical reasoning
behind the solution
includes conceptual
links.
Teachers’ explanation of
the learners’
mathematical reasoning
behind the solution
includes conceptual
links.
Teachers’ explanation
of the learners’
mathematical reasoning
behind the solution
includes conceptual
links.
The explanation
illuminates
conceptually the
background and
process of the activity.
The explanation includes
some but not all of the
conceptual links which
illuminate the
background and process
of the activity.
The explanation
includes incorrect or
poorly conceived
conceptual links and
thus is potentially
confusing.
Full
Partial
Inaccurate
Teachers explain the
mathematical error
made by the learner.
Teachers explain the
mathematical error made
by the learner.
Teachers explain the
mathematical error
made by the learner.
The explanation of the
particular error is
mathematically sound
The explanation of the
particular error is
mathematically sound
The explanation of the
particular error is
mathematically
No
conceptual
links are
made in the
explanation.
Not present
No
explanation
is given of
the
mathematica
l of the
particular
142
reasoning.
and suggest links to
common
misconceptions or
errors
but does not link to
common misconceptions
or errors
inaccurate.
error.
Diagnostic reasoning
Teachers describe
learners’ mathematical
reasoning behind the
error.
Teachers describe
learners’ mathematical
reasoning behind the
error.
Teachers describe
learners’ mathematical
reasoning behind the
error.
It describes the steps of
learners’ mathematical
reasoning
systematically and
hones in on the
particular error.
The description of the
learners’ mathematical
reasoning is incomplete
although it does hone in
on the particular error.
The description of the
learners’ mathematical
reasoning does not
hone in on the
particular error.
No attempt
is made to
describe
learners’
mathematica
l reasoning
behind the
particular
error
Teachers’ explanation
of the learners’
mathematical reasoning
behind the error
appeals to the
everyday.
Teachers’ explanation of
the learners’
mathematical reasoning
behind the error appeals
to the everyday.
Teachers’ explanation
of the learners’
mathematical reasoning
behind the error
appeals to the
everyday.
The idea of error analysis
goes beyond identifying a
common error and
misconception. The idea is
to understand the way
teachers go beyond the
actual error to try and
follow the way the learners
were reasoning when they
made the error. The
emphasis of this code is on
teachers’ attempt to
provide rationale for how
learners were reasoning
mathematically when they
chose the distractor.
Use of everyday
knowledge
Teachers often explain why
learners make an error by
appealing to everyday
experiences that learners
draw on and confuse with
the mathematical context of
the question. The emphasis
of this code is on the quality
of the use of everyday,
judged by the links made to
the mathematical
understanding teachers try
to advance.
Multiple explanations of
error
One of the challenges in
error analysis is for learners
to hear more than one
explanation of the error.
This is because some
explanations are more
accurate or more accessible
than others. This code
examines the teachers’
Teachers’ use of the
‘everyday’ enables
mathematical
understanding by
making the link
between the everyday
and the mathematical
clear
Teacher’s use of the
‘everyday’ is relevant but
does not properly explain
the link to mathematical
understanding
Multiple mathematical
explanations are
provided.
Multiple mathematical
and general explanations
are provided
All of the explanations
(two or more) are
mathematically
feasible/convincing
At least two of the
mathematical
explanations
feasible/convincing
No
discussion of
everyday is
done.
Teacher’s use of the
‘everyday’ dominates
and obscures the
mathematical
understanding, no link
to mathematical
understanding is made
Multiple mathematical
and general
explanations are
provided
One mathematically
feasible/convincing
explanation provided
No
mathematica
lly
feasible/conv
incing
explanation
provided
143
explanation(s) of the error
itself rather than the
explanation of learners’
reasoning.
This is coded F/N (mathematically feasible/not) for each new and different explanation offered
by the teacher. The final code is assigned according to the level descriptors above.
144
Appendix 6
Procedural explanations
The literature emphasises that the quality of teachers’ explanations depends on the
balance they achieve between explaining the procedure required for addressing a
mathematical question and the mathematical concepts underlying the procedure. This
criterion aims to grade the quality of the teachers’ procedural explanations of the correct
answer. The emphasis in the criterion is on the quality of the teachers’ procedural
explanations when discussing the solution to a mathematical problem through engaging
with learner test data. Teaching mathematics involves a great deal of procedural
explanation which should be done fully and accurately for the learners to grasp and
become competent in working with the procedures themselves. The four categories,
which capture the quality of the procedural explanations demonstrated by a
teacher/group, are presented in Table A1 below.
Table A1: Category descriptors for “procedural explanations”
Procedural
Full
Partial
The emphasis of this
code is on the teachers’
procedural explanation
of the solution. Teaching
mathematics involves a
great deal of procedural
explanation which
should be done fully and
accurately for the
learners to grasp and
become competent in
working with the
procedures themselves.
Teachers’
explanation of the
learners’
mathematical
reasoning behind the
solution includes
demonstration of
procedure.
The procedure is
accurate and
includes all of the
key steps in the
procedure.
Inaccurate
Not
present
Teachers’
explanation of the
learners’
mathematical
reasoning behind
the solution
includes
demonstration of
procedure.
Teachers’
explanation of the
learners’
mathematical
reasoning behind
the solution
includes
demonstration of
procedure.
No
procedural
explanation
is given
The procedure is
accurate but it does
not include all of
the key steps in the
procedure.
Teacher’s use of
procedure is
inaccurate and
thus shows lack
of understanding
of the procedure.
Exemplars of coded texts of correct answer texts for procedural explanations:
The first set of exemplars relates to a single ICAS item to show the vertical
differentiation of the correct answer codes in relation to one mathematical concept under
discussion. These exemplar texts present explanations that received the same level
145
coding on the two criteria of procedural and conceptual explanations (see Table A5).
This would not always be the case. We have chosen to present such exemplars because
of the high correlation between procedural and conceptual explanations.
Table A2: Criterion 1 – Procedural explanation of the choice of the correct solution in relation
to one item
Test Item
Criterion wording
(Grade 8 item 8 ICAS 2006)
Procedural
Which row contains only square
numbers?
(A)
2
4
8
16
(B)
4
16
32
64
(C)
4
16
36
64
(D)
16
36
64
96
The emphasis of this code is on the teachers’ procedural
explanation of the solution. Teaching mathematics involves a
great deal of procedural explanation which should be done
fully and accurately for the learners to grasp and become
competent in working with the procedures themselves.
Selection – correct answer - C
Text
Code
Category descriptor
1² = 1; 2² = 4; 3² = 9; 4² = 16; 5²
= 25; 6² = 36 ; 7² = 49; 8² = 64;
Therefore the row with 4;
16; 36; 64 only has square
numbers. To get this right
they need to know what
“square numbers” mean
and to be able to calculate or
recognize which of the rows
consists only of square
numbers.
Full
Teachers’ explanation of the learners’ mathematical reasoning
behind the solution includes demonstration of procedure.
Learners would illustrate
squares to choose the right
row.
Partial
The procedure is accurate and includes all of the key steps in
the procedure.
Teachers’ explanation of the learners’ mathematical reasoning
behind the solution includes demonstration of procedure.
The procedure is accurate but it does not include all of the key
steps in the procedure.
Learners could calculate the
square roots of all the
combinations in order to
discover the correct one. To
get this learners need to
know how to use the square
Inaccurate
Teachers’ explanation of the learners’ mathematical reasoning
behind the solution includes demonstration of procedure.
Teacher’s use of procedure is inaccurate and thus shows lack of
understanding of the procedure.
146
root operation.
Learners understood the
question and chose the right
row.
Not
present
No procedural explanation is given
The next set of exemplars relate to various ICAS item to show the differentiation of the
correct answer codes in relation to a spread of mathematical concepts.
Table A3: Criterion 1 – Procedural explanation of the choice of the correct solution in relation
to a range of items
Criterion wording
Procedural
The emphasis of this code is on the teachers’
procedural explanation of the solution.
Teaching mathematics involves a great deal
of procedural explanation which should be
done fully and accurately for the learners to
grasp and become competent in working
with the procedures themselves.
Item
Category
descriptor
Text: Full explanation
ICAS 2007 Grade 9 Item 24
Teachers’
explanation of
the learners’
mathematical
reasoning
behind the
solution
includes
demonstration
of procedure.
Learners will take the 4 800
revolutions and divide it
by 60 to convert the
answer to seconds and
obtain an answer of 80
they then take this answer
and multiply it by 360 for
the number of degrees in a
circle to obtain the correct
answer of 28 800.
Selected correct answer – C
Content area: Geometry
The procedure
is accurate and
includes all of
the key steps in
the procedure.
147
ICAS 2006 Grade 7 Item 3
Selected correct answer – A
Content area: Number
Teachers’
explanation of
the learners’
mathematical
reasoning
behind the
solution
includes
demonstration
of procedure.
Text: Partial explanation
Learners used the division
line correctly inside the
circle and counted the
pieces correctly. They fully
understood that this
division in a circle happens
from the centre, and that
thirds mean that there are
three equal parts.
The procedure
is accurate but
it does not
include all of
the key steps in
the procedure.
ICAS 2007 Grade 6 Item 8
Selected correct answer – B
Content area: Measurement
Teachers’
explanation of
the learners’
mathematical
reasoning
behind the
solution
includes
demonstration
of procedure.
Text: Inaccurate
explanation
1,3 + 1,3 = 2,6 +1,3 = 3,9 +1,3
= 5,2 +1,3 = 6,5 +1,3 =7,8
+1,3 = 9,1 + 1,3 = 10,4 (that’s
the closest )
Teacher’s use of
procedure is
inaccurate and
thus shows lack
of
understanding
of the
procedure.
148
ICAS 2006 Grade 5 Item 12
Selected correct answer – B
Content area: Data Handling
No procedural
explanation is
given
Text: Mathematical
explanation not present
Learners managed to get
the selected distractor by
focusing on the key
149
Appendix 7
Conceptual explanations
The emphasis in this criterion is on the conceptual links made by the teachers in their
explanations of the learners’ mathematical reasoning in relation to the correct answer.
Mathematical procedures need to be unpacked and linked to the concepts to which they
relate in order for learners to understand the mathematics embedded in the procedure.
The emphasis of the criterion is on the quality of the teachers’ conceptual links made in
their explanations when discussing the solution to a mathematical problem through
engaging with learner test data. The four level descriptors for this criterion, which
capture the quality of the conceptual explanations demonstrated by a teacher/group, are
presented in Table A4 below.
Table A4: Category descriptors for “conceptual explanations”
Conceptual
Full
Partial
The emphasis of this
code is on the teachers’
conceptual
explanation of the
procedure/other
reasoning followed in
the solution.
Mathematical
reasoning
(procedural/other)
needs to be unpacked
and linked to the
concepts to which it
relates in order for
learners to understand
the mathematics
embedded in the
activity.
Teachers’
explanation of the
learners’
mathematical
reasoning behind
the solution
includes conceptual
links.
The explanation
illuminates
conceptually the
background and
process of the
activity.
Teachers’
explanation of the
learners’
mathematical
reasoning behind the
solution includes
conceptual links.
The explanation
includes some but
not all of the key
conceptual links
which illuminate the
background and
process of the
activity.
Inaccurate
Not
present
Teachers’
explanation of the
learners’
mathematical
reasoning behind
the solution
includes
conceptual links.
No
conceptual
links are
made in the
explanation
.
The explanation
includes poorly
conceived
conceptual links
and thus is
potentially
confusing.
Exemplars of coded correct answer texts for conceptual explanations
The first set of exemplars relates to a single ICAS item to show the vertical
differentiation of the correct answer codes in relation to one mathematical concept under
discussion. These exemplar texts present explanations that received the same level
coding on the two criteria of procedural (see Table A2) and conceptual explanations.
150
This would not always be the case. We have chosen to present such exemplars because
of the high correlation between procedural and conceptual explanations.
Table A5: Criterion 2 – Conceptual explanation of the choice of the correct solution in relation
to one item
Test Item
Criterion wording
(Grade 8 item 8 ICAS 2006)
Which row contains only square
numbers?
(a)
2
4
8
16
(b)
4
16
32
64
(c)
4
16
36
64
(d)
16
36
64
96
Conceptual
The emphasis of this code is on the teachers’ conceptual
explanation of the procedure/other reasoning followed in the
solution. Mathematical reasoning (procedural/other) needs to
be unpacked and linked to the concepts to which it relates in
order for learners to understand the mathematics embedded in
the activity.
Selection – correct answer - C
Text: Number
Code
Category descriptor
1² = 1; 2² = 4; 3² = 9; 4² = 16; 5²
= 25; 6² = 36 ; 7² = 49; 8² = 64;
Therefore the row with 4;
16; 36; 64 only has square
numbers. To get this right
they need to know what
“square numbers” mean
and to be able to calculate or
recognize which of the rows
consists only of square
numbers.
Full
Teachers’ explanation of the learners’ mathematical reasoning
behind the solution includes conceptual links.
Learners would illustrate
squares to choose the right
row.
Partial
The explanation illuminates conceptually the background and
process of the activity.
Teachers’ explanation of the learners’ mathematical reasoning
behind the solution includes conceptual links.
The explanation includes some but not all of the key conceptual
links which illuminate the background and process of the
activity.
Learners could calculate the
square roots of all the
combinations in order to
discover the correct one. To
get this learners need to
know how to use the square
root operation.
Inaccurate
Teachers’ explanation of the learners’ mathematical reasoning
behind the solution includes conceptual links.
The explanation includes poorly conceived conceptual links and
thus is potentially confusing.
151
Learners understood the
question and chose the right
row.
Not
present
No conceptual links are made in the explanation.
Table A6: Criterion 2 – Conceptual explanation of the choice of the correct solution in relation
to a range of items
Criterion
Conceptual
wording
The emphasis of this code is on the teachers’ conceptual explanation of the
procedure/other reasoning followed in the solution. Mathematical reasoning
(procedural/other) needs to be unpacked and linked to the concepts to which
it relates in order for learners to understand the mathematics embedded in
the activity.
Item
Level descriptor
“Own test” Grade 4 Item
Teachers’ explanation of the learners’ mathematical
reasoning behind the solution includes conceptual links.
Content area: Measurement
The explanation illuminates conceptually the
background and process of the activity.
B) Mpho has 2 l of milk. How many
cups of 100ml can he fill ? __________
(2)
Text: Full explanation
Learner wrote
1
100
100
2
100
100
3
6
100
7
16 100
100
100
12
= 500ml
1000ml =1l
5
8
100
11
100
4
100
100
100
100
9
100
13
= 500ml
10
= 500ml
1000ml =1l
100
14
15
= 500ml
100
18 100
19 100
20 100
17
Learner then counted the number of groups of 100 to calculate how many
cups and got 20
152
ICAS 2006 Grade 5 Item 30
Selected correct answer – B
Content area: Pattern and Algebra
Teachers’
explanation of the
learners’
mathematical
reasoning behind
the solution
includes conceptual
links.
Text: Partial explanation
Sequence of counting numbers
were used matching them with
odd numbers, so that for every
blue block there are two pink
blocks added.
The explanation
includes some but
not all of the key
conceptual links
which illuminate the
background and
process of the
activity.
ICAS 2007 Grade 9 Item 8
Selected correct answer – D
Content area: Number
Teachers’
explanation of the
learners’
mathematical
reasoning behind
the solution
includes conceptual
links.
The explanation
includes poorly
conceived
conceptual links and
thus is potentially
confusing.
Text: Inaccurate explanation
The learner will use the
information that 6 slices of pizza
make a whole and therefore the
denominator for this fraction is 6.
They will then take the 16 slices
eaten and make that the
numerator and hence ascertain
how many whole pizzas had
been eaten.
153
ICAS 2006 Grade 3 Item 14
Selected correct answer – D
Content area: Data handling
No conceptual links
are made in the
explanation.
Text: Mathematical explanation
not present
Learners are attracted to yellow
as the brightest colour. Young
learners often attracted to the
brightest colour. Learners didn’t
understand the question.
154
Appendix 8
Awareness of error
The emphasis in this criterion is on the teachers’ explanations of the actual mathematical
error (and not on the learners’ reasoning). The emphasis in the criterion is on the
mathematical quality of the teachers’ explanations of the actual mathematical error
when discussing the solution to a mathematical problem. The four level descriptors for
this criterion, which capture the quality of the awareness of error demonstrated by a
teacher/group, are presented in Table A7 below.
Table A7: Category descriptors for “awareness of mathematical error”
Awareness of error
Full
Partial
Inaccurate
The emphasis of this
code is on teachers’
explanation of the
actual mathematical
error and not on
learners’ reasoning.
Teachers explain the
mathematical error
made by the learner.
Teachers explain the
mathematical error
made by the learner.
The explanation of
the particular error is
mathematically
sound and suggests
links to common
misconceptions or
errors.
The explanation of
the particular error is
mathematically
sound but does not
link to common
misconceptions or
errors.
Teachers explain
the mathematical
error made by the
learner.
The explanation of
the particular error
is mathematically
inaccurate or
incomplete and
hence potentially
confusing.
Not
present
No
mathematic
al
explanation
is given of
the
particular
error.
Exemplars of error answer texts for awareness of error criterion:
As with the exemplars for procedural explanations, the first set of exemplars relates to a
single ICAS item to show the vertical differentiation of the correct answer codes in
relation to one mathematical concept under discussion. These exemplar texts present
explanations that received the same level coding on the two criteria of awareness of the
error and diagnostic reasoning (see Table A11). This would not always be the case. We
have chosen to present such exemplars because of the high correlation between
awareness of the error and diagnostic reasoning.
Table A8: Criterion 3 – Awareness of the error embedded in the incorrect solution in relation
to one item
Test Item
Criterion wording
(Grade 7 item 26 ICAS 2006)
155
Awareness of error
The emphasis of this code is on teachers’ explanation of the
actual mathematical error and not on learners’ reasoning.
Selection – incorrect answer - B
Text
Code
Category descriptor
The pupils started from the
highlighted 3 and then
counted in groups of 3 and
then just added 1 because the
next box being 7 was
highlighted. They then
continued this pattern
throughout and found that it
worked well. The pattern
worked and so they assumed
that this was the correct
answer but ignored the word
multiples. They just counted
in three’s (plus 1) to get the
answer.
Full
Teachers explain the mathematical error made by the learner.
Perhaps the pupils looked at
the 3 and the 1 in the answer,
decided that 3 + 1 = 4 and
thus started from the
highlighted 3 and then
counted in four’s and thus
got the numbers of 3; 11; 15;
19; etc.
Partial
The pupils have a poor
understanding of the word
multiple, as they just counted
in groups of three’s and not
in multiples of three’s.
Inaccurate
The pupils did not work out
the actual sum, they just read
Not
The explanation of the particular error is mathematically
sound and suggest links to common misconceptions or errors
Teachers explain the mathematical error made by the learner.
The explanation of the particular error is mathematically
sound but does not link to common misconceptions or errors
Teachers explain the mathematical error made by the learner.
The explanation of the particular error is mathematically
inaccurate.
No mathematical explanation is given of the particular error.
156
the question, it looked
similar to the answer and
thus chose (multiples of 3) +
1.
present
Table A9: Criterion 3 – Awareness of the error embedded in the incorrect solution in relation
to a range of items
Criterion wording
Awareness of error
The emphasis of this code is on teachers’ explanation of the
actual mathematical error and not on learners’ reasoning.
Item
Level descriptor
Text: Full explanation
ICAS 2007 Grade 5 Item 17
Teachers explain the
mathematical error
made by the learner.
After the learner got the white
square area which was 64 he/she did
not subtract the dark square area
from the area of the white square.
This is another halfway step to
finding the answer.
Selected distractor – D
Content area: Measurement
The explanation of
the particular error is
mathematically
sound and suggest
links to common
misconceptions or
errors
ICAS 2006 Grade 6 Item 5
Selected distractor – C
Teachers explain the
mathematical error
made by the learner.
Content area: Number
Text: Partial explanation
Subtracted or added the 42 and the 6
because they misinterpreted the
division sign.
The explanation of
the particular error is
mathematically
sound but does not
link to common
misconceptions or
errors
157
ICAS 2007 Grade 4 Item 15
Selected distractor – C
Teachers explain the
mathematical error
made by the learner.
Content area: Number
The explanation of
the particular error is
mathematically
inaccurate or
incomplete and hence
potentially confusing.
ICAS 2006 Grade 9 Item 20
Selected distractor – A
Content area: Measurement
No mathematical
explanation is given
of the particular
error.
Text: Inaccurate explanation
The learners might have considered
the last digit of the product (37) and
decided to choose this answer
because it has 7x7.
Text: Mathematical explanation not
present
They did not answer the question,
yet chose to answer anther: How
deep is the anchor under water?
158
Appendix 9
Diagnostic Reasoning
The idea of error analysis goes beyond identifying the actual mathematical error. The
idea is to understand how teachers go beyond the mathematical error and follow the
way learners may have been reasoning when they made the error. The emphasis in the
criterion is on the quality of the teachers’ attempt to provide a rationale for how learners
were reasoning mathematically when they chose a distractor. The four level descriptors
for this criterion, which capture the quality of the diagnostic reasoning demonstrated by
a teacher/group, are presented in Table A10 below.
Table A10: Category descriptors for “diagnostic reasoning”
Diagnostic reasoning
Full
Partial
The idea of error
analysis goes beyond
identifying a common
error and
misconception. The
idea is to understand
the way teachers go
beyond the actual error
to try and follow the
way the learners were
reasoning when they
made the error. The
emphasis of this code is
on teachers’ attempt to
provide rationale for
how learners were
reasoning
mathematically when
they chose the
distractor.
Teachers describe
learners’
mathematical
reasoning behind
the error.
Teachers describe
learners’
mathematical
reasoning behind
the error.
The description of
the steps of
learners’
mathematical
reasoning is
systematic and
hones in on the
particular error.
The description of
the learners’
mathematical
reasoning is
incomplete
although it does
hone in on the
particular error.
Inaccurate
Not present
Teachers
describe
learners’
mathematical
reasoning
behind the
error.
No attempt is
made to
describe
learners’
mathematical
reasoning
behind the
particular
error
The description
of the learners’
mathematical
reasoning does
not hone in on
the particular
error.
Exemplars of coded error answer texts for diagnostic reasoning:
As with the exemplars for procedural explanations, the first set of exemplars relates to a
single ICAS item to show the vertical differentiation of the correct answer codes in
relation to one mathematical concept under discussion. These exemplar texts present
explanations that received the same level coding on the two criteria of awareness of the
error (see Table A8) and diagnostic reasoning. This would not always be the case. We
159
have chosen to present such exemplars because of the high correlation between
awareness of the error and diagnostic reasoning.
Table A11: Criterion 4 – Diagnostic reasoning of learner when selecting the incorrect solution
in relation to one item
Test Item
Criterion wording
(Grade 7 item 26 ICAS 2006)
Diagnostic reasoning
The idea of error analysis goes beyond identifying a common
error and misconception. The idea is to understand the way
teachers go beyond the actual error to try and follow the way
the learners were reasoning when they made the error. The
emphasis of this code is on teachers’ attempt to provide
rationale for how learners were reasoning mathematically
when they chose the distractor.
Selection – incorrect answer - B
Text
Code
Category descriptor
The pupils started from the
highlighted 3 and then
counted in groups of 3 and
then just added 1 because the
next box being 7 was
highlighted. They then
continued this pattern
throughout and found that it
worked well. The pattern
worked and so they assumed
that this was the correct
answer but ignored the word
multiples. They just counted
in three’s (plus 1) to get the
answer.
Full
Teachers describe learners’ mathematical reasoning behind the
error.
Perhaps the pupils looked at
the 3 and the 1 in the answer,
decided that 3 + 1 = 4 and
thus started from the
highlighted 3 and then
Partial
It describes the steps of learners’ mathematical reasoning
systematically and hones in on the particular error.
Teachers describe learners’ mathematical reasoning behind the
error.
The description of the learners’ mathematical reasoning is
160
counted in four’s and thus
got the numbers of 3 – 11 –
15 – 19 – etc…
incomplete although it does hone in on the particular error.
The pupils have a poor
understanding of the word
multiple, as they just counted
in groups of three’s and not
in multiples of three’s.
Inaccurate
The pupils did not work out
the actual sum, they just read
the question, it looked
similar to the answer and
thus chose (multiples of 3) +
1.
Not
present
Teachers describe learners’ mathematical reasoning behind the
error.
The description of the learners’ mathematical reasoning does
not hone in on the particular error.
No attempt is made to describe learners’ mathematical
reasoning behind the particular error
Table A12: Criterion 4 – Diagnostic reasoning of learner when selecting the incorrect solution
in relation to a range of items
Criterion wording
Diagnostic reasoning
The idea of error analysis goes beyond identifying a
common error and misconception. The idea is to
understand the way teachers go beyond the actual
error to try and follow the way the learners were
reasoning when they made the error. The emphasis of
this code is on teachers’ attempt to provide rationale
for how learners were reasoning mathematically
when they chose the distractor.
Item
ICAS 2007 Grade 3 Item 16
Selected distractor – C
Content area: Measurement
Category
descriptor
Teachers describe
learners’
mathematical
reasoning behind
the error.
It describes the
steps of learners’
mathematical
reasoning
systematically and
hones in on the
particular error.
Text: Full explanation
Learners started at the cross
moved to Green Forest South
(down) and then must have
moved East and not West, and
therefore ended at white beach
because they got did not know
or confused with direction
between East/West. Learners
may have moved towards the
"W" for West on the drawing of
the compass which is given on
the map. This compass
drawing might also have
confused them and made them
go to the east when they
should have gone west, though
they did go "down" for South
initially in order to end up at
161
White Beach.
ICAS 2006 Grade 4 Item 20
Selected distractor – C
Content area: Geometry
Teachers describe
learners’
mathematical
reasoning behind
the error.
The description of
the learners’
mathematical
reasoning is
incomplete
although it does
hone in on the
particular error.
Text: Partial explanation
This looks most like tiles we
see on bathroom walls, even
though it is a parallelogram, so
even though it is NOT
rectangular, learners selected
this as the correct answer. They
did not realize that this shape
would go over the edges or
create gaps at the edges if it
was used.
162
ICAS 2007 Grade 5 Item 14
Selected distractor – B
Content area: Measurement
Teachers describe
learners’
mathematical
reasoning behind
the error.
Text: Inaccurate explanation
The distractor selected most
was B. The likelihood is that
the learners could choose 600
because they could have seen
six years in the question.
The description of
the learners’
mathematical
reasoning does not
hone in on the
particular error.
ICAS 2006 Grade 6 Item 7
Selected distractor – C
Content area: Number
No attempt is made
to describe
learners’
mathematical
reasoning behind
the particular error
Text: Mathematical
explanation not present
Not thinking systematically to
devise a correct formula to
obtain the number of cats. This
learner is perhaps feeling
pressurised to produce a
formula describing the
situation, rather than manually
grouping the number of legs
for each cat. There is a question
here of whether the learner
understands the meaning of
the numbers in the formula
s/he has used to calculate the
number of cats.
163
Appendix 10
Use of the everyday
Teachers often explain why learners make mathematical errors by appealing to everyday
experiences that learners draw on and confuse with the mathematical context of the
question. The emphasis in this criterion is on the quality of the use of everyday
knowledge, judged by the links made to the mathematical understanding that the
teachers attempt to advance. The four level descriptors for this criterion, which capture
the quality of the use of everyday knowledge demonstrated by a teacher/group, are
presented in Table A13 below.
Table A13: Category descriptors for “use of everyday”
Use of everyday
Full
Partial
knowledge
Teachers often
explain why learners
make an error by
appealing to
everyday experiences
that learners draw on
and confuse with the
mathematical context
of the question. The
emphasis of this code
is on the quality of
the use of everyday,
judged by the links
made to the
mathematical
understanding
teachers try to
advance
Teachers’ explanation
of the learners’
mathematical
reasoning behind the
error appeals to the
everyday.
Teachers’ use of the
‘everyday’ enables
mathematical
understanding by
making the link
between the everyday
and the mathematical
clear
Inaccurate
Not
present
Teachers’
explanation of the
learners’
mathematical
reasoning behind the
error appeals to the
everyday.
Teachers’
explanation of the
learners’
mathematical
reasoning behind
the error appeals to
the everyday.
No
discussio
n of
everyday
is done
Teacher’s use of the
‘everyday’ is relevant
but does not
properly explain the
link to mathematical
understanding
Teacher’s use of the
‘everyday’
dominates and
obscures the
mathematical
understanding, no
link to
mathematical
understanding is
made
“Use of the everyday” was used to measure the extent to which teachers used everyday
contexts to enlighten learners about mathematical concepts. In the exemplar tables, a
variety of different items where teachers used everyday contexts to add to their
mathematical explanation of the reasoning behind an error are given.
The code “no discussion of the everyday” includes all explanations which did not make
reference to the everyday whether it was appropriate or not. Further more detailed
164
analysis of the kinds of contextualized explanations which were offered and whether
they were missing when they should have been present could be carried out since it was
not within the scope of this report.
Exemplars of coded error answer texts for everyday criterion
Table A14: Use of the everyday exemplars
Criterion wording
Use of everyday knowledge
Teachers often explain why learners make an error by
appealing to everyday experiences that learners draw on
and confuse with the mathematical context of the
question. The emphasis of this code is on the quality of
the use of everyday, judged by the links made to the
mathematical understanding teachers try to advance.
Item
Category descriptor
Text: Full explanation
ICAS 2006 Grade 9 Item 6
Teachers’
explanation of the
learners’
mathematical
reasoning behind
the error appeals to
the everyday.
He draws on his frame of
reference of how he perceives a
litre to be e.g. a 1,25l of cold drink
of a 1l of milk or a 2l of coke etc.
Selected distractor – D
Content area: Measurement
Teachers’ use of the
‘everyday’ enables
mathematical
understanding by
making the link
between the
everyday and the
mathematical clear
165
ICAS 2006 Grade 7 Item 9
Selected distractor – A
Content area: Data Handing
Teachers’
explanation of the
learners’
mathematical
reasoning behind
the error appeals to
the everyday.
Teacher’s use of the
‘everyday’ is
relevant but does
not properly explain
the link to
mathematical
understanding
ICAS 2007 Grade 3 Item 20
Selected distractor – B
Content area: Measurement
Teachers’
explanation of the
learners’
mathematical
reasoning behind
the error appeals to
the everyday.
Text: Partial explanation
Here learners did not read the
question attentively and
misinterpreted or expected that
the most popular sport would be
the question – this is football
which has a frequency of 7 (the
highest). This could be the case
because of the way in which
teachers use these bar graphs in
class. The most popular choice of
question is always the question
about the most or least popular
choice as indicated on the graph.
Text: Inaccurate explanation
In the last option the circle/ ball
managed to pull the cylinder.
Teacher’s use of the
‘everyday’
dominates and
obscures the
mathematical
understanding, no
link to mathematical
understanding is
made
166
ICAS 2006 Grade 5 Item 11
Selected distractor – B
Content area: Geometry
No discussion of
everyday is done.
Text: Mathematical explanation
not present
Some could have guessed just to
have an answer.
167
Appendix 11
Multiple explanations
One of the challenges in the teaching of mathematics is that learners need to hear more
than one explanation of the error. This is because some explanations are more accurate
or more accessible than others and errors need to be explained in different ways for
different learners. This criterion examines the teachers’ ability to offer alternative
explanations of the error when they are engaging with learners’ errors through analysis
of learner test data. The four level descriptors for this criterion, which capture the
quality of the multiple explanations of error demonstrated by the group, are presented
in Table A15 below.
Table A15: Category descriptors for “multiple explanations”
Multiple explanations Full
Partial
of error
One of the challenges
in error analysis is for
learners to hear more
than one explanation of
the error. This is
because some
explanations are more
accurate or more
accessible than others.
This code examines
the teachers’
explanation(s) of the
error itself rather than
the explanation of
learners’ reasoning.
Multiple
mathematical
explanations are
provided.
All of the
explanations (two or
more) are
mathematically
feasible/convincing
Inaccurate
Not present
Multiple
mathematical and
general
explanations are
provided
Multiple
mathematical
and general
explanations are
provided
No
mathematicall
y
feasible/convin
cing
explanation
provided
At least two of the
mathematical
explanations are
feasible/convincing
One
mathematically
feasible/convinci
ng explanation
provided
(with/without
general
explanations)
(combined with
general
explanations)
“Multiple explanations” was used to evaluate the range of explanations given by groups
of learners’ thinking behind the incorrect answer. This was looking to see whether
teachers offer a number of different explanations in relation to the concept. For multiple
explanations we again give a variety of different items where teachers gave more than
one mathematically feasible explanation of the reasoning behind an error.
Exemplars of coded error answer texts for multiple explanations:
Table A16: Multiple explanations exemplars
168
Criterion wording
Multiple explanations of error
One of the challenges in error analysis is for learners to
hear more than one explanation of the error. This is
because some explanations are more accurate or more
accessible than others. This code examines the teachers’
explanation(s) of the error itself rather than the
explanation of learners’ reasoning.
Item
Category descriptor
ICAS 2007 Grade 7 Item 14
Multiple
mathematical
explanations are
provided.
Selected distractor – B
Content area: Measurement
Text: Full explanation
1.
2.
All of the
explanations (two or
more) are
mathematically
feasible/convincing
ICAS 2006 Grade 3 Item 4
Selected distractor – A
Content area: Geometry
Multiple
mathematical and
general explanations
are provided
At least two of the
mathematical
explanations
feasible/convincing
Learners said 6 x 3 = 18.
They took the length
given and multiplied be
the three sides of the
triangle.
They confused the radius
and diameter. They
halved 6cm to get 3cm
and then multiplied by 6,
because the triangle is
made up of six radii.
Text: Partial explanation
1.
2.
3.
The learners did not
focus on the handle, but
on the splayed out
bristles or on the broom
as a whole. See the
bristles as the most
important part of a
broom.
Did not read or
understand the question.
The learners focused on
the broom as a whole.
Because learners have
difficulty separating the
handle from the broom in
their minds eye
169
ICAS 2007 Grade 6 Item 14
Selected distractor – A
Content area: Data and Chance
Multiple
mathematical and
general explanations
are provided
One mathematically
feasible/convincing
explanation
provided
Text: Inaccurate explanation
If they read from 0 to 1000 there
are 3 species of fish. They are
reading the bar not the line. (they
are distracted by the 2-D nature
of the diagram )
170
ICAS 2006 Grade 4 Item 2
Selected distractor –
Content area: Number
No mathematically
feasible/convincing
explanation
provided
Text: Mathematical explanation
not present
1.
2.
3.
They might be distracted by
the word “playing” and
choose particular dogs
according to what they
identify as a dog that is
playing.
They might not identify all of
the animals as dogs, and
hence not count all of them.
We see these as issues of
interpretation, of the
language and of the diagram,
rather than mathematical
misconceptions.
171