Mathematical competence assessment: comparison of
student answers facing different styles of formulating the
examination questions
Genoveva Leví
Didáctica, Organización Escolar y Didácticas Especiales.
Facultad de Educación, UNED (Spain).
genovevalevi@edu.uned.es
Eduardo Ramos
Estadística, Investigación Operativa y Cálculo Numérico,
Facultad de Ciencias, UNED (Spain).
eramos@ccia.uned.es
José Antonio Carrillo
Estadística, Investigación Operativa y Cálculo Numérico,
Facultad de Ciencias, UNED (Spain).
jacarrillo@ccia.uned.es
Abstract
This paper presents a comparative analysis of correct answers given by
students at various math problems depending on the style which the exam
questions are written. The results show that a higher percentage of correct
answers is obtained when the formulation reflects a traditional style, without
reference to situations of everyday life as the genuine evaluation of
mathematical competence demands.
Introduction
The assessment of competences is one of the most important challenges that
entails the widespread introduction of the teaching-learning models aimed at
developing competences. According to the most prevalent notion of
competence in the literature (Gonzalez and Wanegaar, 2006; Beneitone et al,
2007) students reach competences not only based on the acquisition of
theoretical knowledge but also, and especially, showing practical knowledge in
a certain context, particularly in situations taken from real life. Consequently,
teachers need to consider new tools of evaluation and to redesign traditional
tests, by incorporating innovative elements to assess the level of competence
of students (Johnson, Penny and Gordon, 2009; Levi and Ramos, 2012; Levi
Ramos and Carrillo, 2014). An important reference is PISA project; its
approach to the assessment of reading, mathematical and scientific literacy is
now a benchmark with great international influence (OECD, 2014).
The evaluation of students through innovative tests facing real-life situations
brings important consequences for the planning and development of teaching.
The aim of this paper is to present some issues related to the evaluation of
mathematical competence. Specifically, we analyzed the responses given by
the students when the teacher evaluates them posing questions in the
traditional style of mathematics, compared to another questions which may be
resolved using exactly the same knowledge but are formulated in a language
close to real situation in which the mathematical problem is presented.
The following sections describe the experience realized and present some
results with its discussion.
The didactic experience
The framework of the experience is the Foundations Course for people over
25 years taught at the UNED of Spain. The aim of this course is to achieve
accreditation in order to pursue a university degree for people who have not
completed secondary school. The course includes two phases: the general
phase, which contains subjects of Spanish Language, Foreign Language and
Text Commentary; and the specific phase, whose contents depend on the
branch of knowledge of the degree that the student intends to pursue. One of
the subjects of the specific phase for students of Social Sciences is
Mathematics Applied to Social Sciences. The contents of this course pursue a
general education in the main fields of mathematical knowledge organized in
five teaching units:
1) Foundations, including predicate logic, sets and applications.
2) Arithmetic and Algebra, which include the different sets of numbers and
operations.
3) Geometry, which introduces the Cartesian plane, distances, lines and
some plane figures.
4) Analysis, which examines the concept of function, graph, derivatives,
maxima and minima.
5) Probability and Statistics, which studies the basic probabilistic model,
along with a brief introduction to descriptive statistics.
The study materials of the course primarily consist of a basic text prepared by
the teaching staff, written in a style appropriate for independent study
developing the five teaching units listed above.
The text includes a number of self-assessment questions as objective tests,
consisting of a statement and three alternatives, one and only one of which
represents the right solution. The style with which they are written is divided
into two large groups. Some, which we call "theoretical" are written in
traditional style characteristic of mathematics, raising the problem, calculus,
theoretical concept, etc., aseptically without any reference to a real context.
However, other questions, which we call "applied", are based on a brief
introductory text, which presents a situation taken from the reality that
describes the context of the test and includes the information required to
answer the question. Consideration of such questions is intended to face the
student with a problem of mathematical nature, similar to those that can be
found in everyday life, which can be solved with the knowledge acquired in the
course; that is, these questions are intended to assess the level of
mathematical competence of the student.
Classroom assessment, leading to the accreditation of the course, is divided
into two parts. The first exam is held in February and is not compulsory; it
includes the first two teaching units and liberates this part to those who get five
points or more out of ten. The second part, which takes place in June and is
mandatory, it includes only teaching units 3, 4 and 5 for those who passed the
test in February; however, the other students are faced with a test that
includes all the contents of the course.
The results we discuss in the next section refers to the school year 2014-15. In
the exams in February and June, the question listed in Table 1 were proposed.
As can be seen, the questions are quite similar, differing only that in February
we use a style "applied", including the context of the question, while a
"theoretical" style was used in June without reference to any context.
Table 1. Evaluation questions proposed in Mathematics Applied to Social
Sciences (UNED, Spain) in the year 2014-15.
February
Context: The following table is a copy of the payroll of a worker with printing
errors.
Pay items
Euros Discounts
Euros
Salary
▓▓▓
Common contingencies (4.70%) ▓▓▓
Trienniums
▓▓▓
Unemployment (1.60%)
▓▓▓
General complement
▓▓▓
Training (0.10%)
▓▓▓
Personal complement
14.08 Income Tax (14%)
207.93
Gross amount
▓▓▓
Total discounts ▓▓▓
Total liquid amount to be received ▓▓▓
The first two columns show the different remuneration concepts and payroll
amounts: salary (s), trienniums (t), general complement (g) and personal
complement (p). It is clear the data that refers to the personal complement
(14.08 euros). We call R to the set of all remuneration concepts, i.e. R = {s, t,
g, p}. The last two columns show the concepts of discounts and the resulting
amount: common contingencies (c), unemployment (d), training (f), which are
corresponding to the Social Security (SS) discounts, and the Income Tax (i). It
is clear the income tax amount (207.93 euros).We call D the set of discounts,
i.e. D = {c, d, f, i}. We call U the universal set that includes all the elements of
the payroll, i.e., U = {s, t, g, p, c, d, f, i}. The income tax discount is calculated
on the total income of the payroll, that is: Total income = s + t + g + p.
However, discounts corresponding to the SS are calculated as a percentage of
the monthly tax base, which is obtained as the twelfth of the annual gross
income. Such gross income is calculated by adding twelve ordinary pay, as
that of the Table, plus two extra payments. The worker knows that the extra
payments are slightly lower than the ordinary payments, because they differ on
the amount of personal complement, 14.08 euros, which is not perceived in
any of the two extras.
Question F1. Let C = {c, d, f} the Question F2. The gross amount of
set of the discounts that integrate payroll
social discounts. Which of the a) is 1,471.13 euros.
following
notations
accurately b) is 1,485.21 euros.
describes the relationship between c) can not be calculated from the
C and D?
available data.
a) C ≤ D.
b) C ∈ D.
c) C ⊂ D.
June
Context: Nothing.
Question J1. Let U = {a, b, c, d, e, Question J2. A company retained in the
f, g, h} a universal set. Consider the ordinary payroll worker 285.00 euros for
set A = {a, b, c} and B = {a, b, c, d}. various items. This amount represents
Which of the following notations 19% of their gross salary. Then such
accurately
describes
the gross salary
relationship between A and B?
a) is 1,215.00 euros.
a) A ≤ B.
b) is 1,500.00 euros.
c) can not be calculated from the
b) A ∈ B.
available data.
c) A ⊂ B.
Results
The number of students enrolled in course in 2014-15 year was 6148. In
February, 3348 students took the exam with questions F1 and F2; 2036 out of
them did not pass, so they had to repeat in June. In this new exam 835
students faced the J1 and J2 questions. The results included below refer to
this group, i.e., students who had to answer questions F1 and F2 in February
and J1 and J2 in June.
Table 2 shows the distribution of answers to each of the questions posed.
Table 2: Responses to the questions: A-B-C and X (non-response, double or triple
mark)
F1
J1
F2
J2
Num.
%
Num.
%
Num.
%
Num.
%
A
87
10,4
66
7,9
93
11,1
65
7,8
B
211 25,3
144
17,2
426
51,0
646
77,4
C
425 50,9
540
64,7
103
12,3
40
4,8
X
112 13,4
85
10,2
213
25,5
84
10,1
Total 835 100,0
835
100,0
835
100,0
835 100,0
The following figure graphically compares the percentages of correct answer,
incorrect answer and not answer to the questions F1-J1 and F2-J2.
Table 3 shows the results of the questions F1-J1 and F2-J2 in the two
proposals calls.
J1
A
B
C
X
Total
A
B
13 18
19 42
43 143
12 8
87 211
Table 3: Answers of the questions F1-J1 y F2-J2
F1
F2
C
X Total
J2 A
B
C
23 12
66
A
9 29
8
72 11
144
B 61 357 79
288 66
540
C
6 16
7
42 23
85
X 17 24
9
425 112 835
Total 93 426 103
X
19
149
11
34
213
Total
65
646
40
84
835
If a correct response is scored with +1, an incorrect with -0.5, and no response
with 0, and we consider the appropriate variables PF1, PF2, PJ1 and PJ2 then
the Wilcoxon signed-rank test for related samples, performed with SPSS 22,
leads to reject the hypothesis of equal medians PF1 versus PJ1 (p <0.0000)
and PF2 versus PJ2 (p <= 0.0000). We can establish that they are statistically
significant differences in both cases, that is, the percentage of correct answers
is less when questions are presented within a given context extracted from
everyday life.
Conclusions
The results we have obtained show that student responses to certain types of
mathematical questions depend on the context in which the question is
framed. The percentage of success is greater when the questions are written
in the traditional manner, without reference to a specific context, similar to real
life. Instead, the questions raised in a format that presents it in a situation
closer to reality get a lower percentage of correct answers and also a greater
number of non-responses. These results show that the real evaluation of
mathematical competence poses a double challenge to teachers, since not
only requires a review of the assessment model, but we must also investigate
new teaching approaches to allow students to acquire and develop real
competence.
References
Beneitone, P.; C. Esquetini; J. González; M. M. Maletá; G. Siufi and R.
Wanegaar (Eds.) (2007): Reflexiones y perspectivas de la educación
superior en América Latina. Informe Final -- Proyecto Tuning -- América
Latina 2004--2007, Universidad de Deusto, Universidad de Groningen,
http://tuning.unideusto.org/tuningal.
González, J. and R. Wagenaar (2006): Tuning Educational Structures in
Europe. La contribución de las universidades al proceso de Bolonia,
Universidad de Deusto, http://www.unideusto.org/tuning.
Johnson, R.L.; J.A. Penny and B. Gordon (2009). Assessing performance:
designing, scoring and validating performance tasks. New York: The
Guilford Press.
Leví, G. and E. Ramos (2012). Evaluación de la competencia matemática, En
A. Medina (ed.): Formación y desarrollo de las competencias básicas
(2ª Edición). Madrid: Universitas.
Leví, G.; E. Ramos and J.A. Carrillo (2014): Latent factors in the formation and
development of mathematical competence. In A. Rogerson (Ed.): The
Future of Mathematics Education in a Connected World. Proceedings of
the 12th International Conference: The Mathematics Education for the
Future. Montenegro.
OECD (2014). PISA 2012 Results: What Students Know and Can Do –
Student Performance in Mathematics, Reading and Science (Volume I,
Revised edition, February 2014), PISA, OECD Publishing.
http://dx.doi.org/10.1787/9789264201118-en.