-1-
Contents
Part I Rationale ........................................................................................................................... 2
Part II Conceptual Framework ................................................................................................... 3
1. Algebraic skills ................................................................................................................... 5
2. ICT tool use ........................................................................................................................ 8
3. Assessment ....................................................................................................................... 14
4. Integrating theory ............................................................................................................. 22
5. Choosing content that makes symbol sense ..................................................................... 23
6. ICT Tools for assessment ................................................................................................. 25
7. Designing the prototype and instruction .......................................................................... 29
Part III Methodology ................................................................................................................ 30
Appendix A .......................................................................................................................... 32
Appendix B .......................................................................................................................... 36
Appendix C .......................................................................................................................... 38
DITwis .............................................................................................................................. 38
Algebra Tutor ................................................................................................................... 39
Calmaeth........................................................................................................................... 40
Math Xpert: ...................................................................................................................... 41
Aplusix ............................................................................................................................. 42
L’Algebrista ..................................................................................................................... 43
webMathematica .............................................................................................................. 44
Wiris ................................................................................................................................. 45
AiM: Assessment in Mathematics.................................................................................... 46
CABLE ............................................................................................................................. 47
Hot potatoes...................................................................................................................... 48
Question Mark Perception ................................................................................................ 49
Wintoets ........................................................................................................................... 50
Moodle quiz module with extensions............................................................................... 51
Wallis ............................................................................................................................... 52
WebWork ......................................................................................................................... 53
TI interactive .................................................................................................................... 54
Cognitive Tutor ................................................................................................................ 55
Algebra Buster.................................................................................................................. 56
Appendix D .......................................................................................................................... 57
Maple TA ......................................................................................................................... 57
Digital Mathematical Environment .................................................................................. 58
Activemath ....................................................................................................................... 59
STACK ............................................................................................................................. 60
Wims ................................................................................................................................ 61
Appendix E ........................................................................................................................... 62
Appendix F ........................................................................................................................... 65
References ................................................................................................................................ 69
-2-
Part I Rationale
For several years now the skill level of students leaving secondary education in the
Netherlands has been questioned. Lecturers in higher education often complain of an apparent
lack of algebraic skills, for example. I was personally confronted with this challenge when I
redesigned the entry exam of the Free University from 2001-2004. This problem has only
grown larger in the last few years. In 2006 a national project was started to address and
scrutinize this gap in mathematical skills, called NKBW. In the same period use of ICT in
mathematics education has also increased. It is our conviction that ICT can be used to aid
bridging this gap.
Therefore this research will focus on two relevant issues in mathematics education in
secondary schools in the Netherlands: on the one hand signals from higher education that
freshmen students have a lack of algebraic skills, on the other hand the use of ICT in
mathematics education.
Relation to current curricular developments in math education
These developments have to be seen in a larger context. In 2007 the cTwo commission
(commission on the future of mathematics education) published a vision document (2007)
which has all the ingredients for this research.
First of all the importance of numbers, formulas, functions, change, space and chance are
stressed (viewpoint 4). On an algebraic level this corresponds with the sources of meaning
(Radford, 2004) for algebra. Activities are: modeling, manipulating formulas.
Also, the role of ICT in this process is described (viewpoint 10). ICT should be "use to
learn" and not "learn to use". This strict dichotomy will be difficult to accomplish, as they go
hand in hand. This will be elaborated on in the chapter on tool use.
In viewpoint 14 a specific case is made for the transition of students from secondary
education towards higher education. Again, it is stressed that this transition needs more
attention.
Viewpoint 15 stresses the importance of assessment of algebraic skills.
Finally, viewpoint 16 mentions the pen-and-paper aspect of mathematics.
Why with a computer tool?
But why should we use a computer in learning algebra? We contend that computers can aid
understanding of algebra by providing a learning environment that enables you to practice
algebra anytime, anyplace, anywhere, because:
-
Randomization of exercises means there are many more questions.
It is possible to use several representations
The applets can be used anyplace, anytime, anywhere.
Automated feedback can help in this process
Students tend to be more motivated
We will elaborate on this in our conceptual framework.
-3-
Part II Conceptual Framework
This research focuses on the question:
In what way can the use of ICT in secondary education support learning, testing and
assessing relevant mathematical skills?
First it is useful to analyze our question word-for-word.
In what way. To us it is not a question whether ICT can be used to support learning,
testing and assessing mathematical skills, but how this should take place.
Secondary education. In this research we focus on upper secondary education and in
particular students preparing to go on to higher education.
Learning, testing and assessing. Not only grades and scores are important, but also the
way in which mathematical concepts are learned and tested diagnostically. We
specifically aim to find out more about all three aspects.
Relevant mathematical skills. When students leave secondary education they are
expected to have learned certain skills. Here we focus on algebraic skills, with
particular attention given to “real understanding of concepts”, symbol sense.
So following a pragmatic approach three key issues are part of this research question: skills,
assessment and ICT tool use.
The structure of part II is as follows:
First we discuss the three key concepts algebraic skills, assessment and tool use in chapters
1, 2 and 3. Every section starts with a problem statement, then gives an overview of relevant
literature and ends with some words on the implications for my research.
In chapter 4 we integrate these concepts into one framework for my research.
Based on this conceptual framework two major decisions have to be made:
- Which ICT tool to use for assessment. For this we will formulate criteria based on the
conceptual framework and give an overview of available ICT tools for assessment.
- What content to use for learning, testing and assessing algebraic skills. Per question
we will motivate why the question is relevant for this research.
In chapters 5 and 6 these two decisions are explicated.
Together they will make up the design principles for our first prototype, which will be
summarized in chapter 7. In part III we then discuss the methodology we use
-4-
Rationale
Part I
Research question(s)
Algebraic skills
Tool use
Assessment
(chapter 1)
(chapter 2)
(chapter 3)
Conceptual framework
(chapter 4)
Content choice
Tool choice
(chapter 5)
(chapter 6)
Prototypical design
Part II
(chapter 7)
Methodology
-5-
1. Algebraic skills
In this chapter we focus on algebraic skills and symbol sense. For this it is important to sketch
a general outline of the subject at hand. In recent years
A. Problem statement
Algebraic skills of students are decreasing. We want to make sure that students really
understand algebraic concepts, so just testing basic skills is insufficient. What defines real
algebraic understanding?
B. Theoretical overview
In a historical context al-Khwarizmi, Vieta and Euler considered algebra to be a "tool for
manipulating symbols and for solving problems." In the 80s Fey and Good (1985) argued that
the "function concept is at the heart of the curriculum". More recently Laughbaum (2007) sees
ground for this statement in neuroscience.
To get a clear picture of algebraic skills and the purpose of algebra we have to look into the
theoretical foundations.
Meaning of algebra
Radford (2004) sees several sources of meaning in algebra:
1. Meaning from within mathematics, which can be divided into:
(a) Meaning from the algebraic structure itself, involving the letter-symbol form.
This is also referred to as "structure of expressions" or “structure sense” (Hoch &
Dreyfus, 2005). I would like to use the term " symbol sense" here, in line with Arcavi
(1994) and Drijvers (2003).
(b) Meaning from other mathematical representations, including multiple representations.
This corresponds with the "multirepresentational" views of Janvier (1987), Kaput
(1989) and van Streun (2000)
2. Meaning from the problem context.
3. Meaning derived from that which is exterior to the mathematics/problem context
(gestures, bodily movements, words, metaphors, artifacts use)
Ideally, all these sources would be addressed in an instructional sequence.
To focus more on the actual concepts that are learned Kieran's (1996) GTG model combines
several theories into one framework. In this model three activities are distinguished:
Generational, Transformational and Global/Meta-level activities.
In upper secondary and college level these activities apply:
 Generational activity with a Primary focus on the letter-symbolic form: form and
structure (Hoch & Dreyfus, 2005) and parameters.
 Generational activity with multiple representations: functions and their meaning,
symbolic and graphical representations hand in hand.
 Transformational activity related to notions of equivalence.
 Transformational activity related to equations and inequalities.
-6 Transformational activity related to factoring expressions.
 Transformational activity involving the integration of graphical and symbolic work.
Global/Meta-level activity involving problem solving
 Global/Meta-level activity involving modelling.
Algebraic activities in school
It is essential to have a clear view on what activities in secondary education have to with
algebra. A non-limitative list of activities include:









implicit or explicit generalization
investigation of patterns and numerical relations
problem solving though applying general or specific rules
reasoning with unknown or undetermined quantities
arithmetic operations with literal variables
symbolizing numerical operations and relations
tables and graphs represent formulas and are used to investigate them
formulas and expressions are compared and transformed
formulas and expressions are used to describe situations in which measures and
quantities play a role
 solution processes contain steps based on rules, but without meaning in the context
Grouping these activities one can distinguish two dimensions of algebraic skills: basic skills,
including algebraic calculations (procedural) and symbol sense (conceptual). The latter is
“actual understanding” of algebraic concepts.
Algebraic Skills
Basic skills:
algebraic calculation,
procedural routine
Symbol sense: algebraic reasoning,
strategic skills (Arcavi, 1994)
One can not do without the other. Both should be trained, making use of several influential
models on learning mathematics.
Or as Zorn (2002) puts it: "By symbol sense I mean a very general ability to extract
mathematical meaning and structure from symbols, to encode meaning efficiently in symbols,
and to manipulate symbols effectively to discover new mathematical meaning and structure."
Symbol Sense
The notion of “actual understanding” of mathematical concepts has been given different
names. Hoch called this "structure sense" at the beginning of 2003. Arcavi (1994) used the
term "symbol sense" , analogue to the term "number sense". It is an intuitive feel for when to
call on symbols in the process of solving a problem, and conversely, when to abandon a
symbolic treatment for better tools.
Drijvers (2006) sees an important role for both basic skills and symbol sense. The declining
algebraic skills of students is concerning. As Tall and Thomas (Tall & Thomas, 1991) put it:
-7"There is a stage in the curriculum when the introduction of algebra may make things hard,
but not teaching algebra will soon render it impossible to make hard things simple."
Several problems with symbol sense are:





process-object duality: a student thinks in terms of activity rather than objects.
visual properties of expressions
lack of flexibility
lack of meaning of algebraic expressions
lack of exercises
Building on this last observation Kop and Drijvers (Kop & Drijvers, in press) have suggested
a categorization of “symbol sense” type questions. This source will –together with other
sources- provide a starting point for designing a prototype.
Impact of technology
Technology has an impact on mathematics education. Research with calculators (Ellington,
2003) has shown that the pedagogical role of tool use should not be underestimated. The use
of tools seems to strengthen a positive attitude towards education, showing that there is more
to learning than just practicing and testing. van Streun (2000), Lagrange, Artigue, Laborde
and Trouche (2001) all determined enriched solution repertoires and a better understanding of
functions, especially through the use of multiple representations. However, use should not be
haphazard, but for prolonged use.
The next step in using tools for algebra was in the use of Computer Algebra Systems (CAS).
The first large-scale study on the use of CAS was by (1997)
It is also important to stress the changing roles of students and teachers. Guin and Trouche
(1999) noticed that students have different "styles" of coping with problems: random,
mechanical, rational, resourceful and theoretical.
The modes of graphing calculator used by Doerr and Zangor (2000) could also be applied to
the use of applets: computational, transformational, visualizing, verification and data
collection and analysis tool.
Finally, the advent of computing technology has also strengthened believe that multiple
representations of mathematical objects could be fully integrated in mathematics curriculum.
This could provide a valuable source of implicit feedback, making sure that the added value
of (formative) assessment could be greatly enhanced.
According to Lester (2007) three factors are important in technology-related studies
concerning algebra: time, the nature of the task and the role played by the teacher in
orchestrating the development of algebraic thought by means of appropriate classroom
discussion. One extra factor has to do with the instrumental genesis of the tool used. Transfer
of what has been learned has to take place. Therefore the relation between tool use and penand-paper has to be taken into account. More on this in the second chapter.
C. Algebraic skills in this research
We want to study whether algebraic skills, and in particular symbol sense, can be improved
by using an ICT tool.
-8-
2. ICT tool use
In this chapter I focus on the use of ICT1 tools in (mathematics) education.
A. Problem statement
It is important to study the way in which tools can be used to facilitate learning. How are tools
used and what characteristics do they have to have.
B. Theoretical overview
First I will sketch a general overview, ending in a description of the instrumental approach of
tool use. Here I will use the construct of figure 1, looking at how tools affect aspects of the
teaching and learning of mathematics.
Tool
From the Second Handbook of Research on
Mathematics Teaching and Learning (Lester
jr., 2007)
Student
Mathematical
Activity
Teacher
Curriculum
Content
Not only mathematical activity, students, teachers and curriculum are affected, also the
relationship between these aspects.
Technology and Mathematical activity
Many people use tools all the time. The Vygotskian notions on tool use (Vygotsky, 1978) sees
a tool as a mediator, a " new intermediary element between the object and the psychic
operation, directed at it" . In mathematics, tools have to have certain characteristics to be
beneficial. The Handbook on Research mentions three important issues:
Externalization of representations
Heid (1997) also mentions this. The important question remains: how is mathematical activity
influenced or changed by tool use? Feedback is mentioned. Otherwise time-consuming
"production work" as well. Unlike the physical tool a cognitive tool provides a "constraintsupport system" (Kaput, 1992) for mathematical activity.
Mathematical fidelity
A representation must be faithful to the underlying mathematical properties; this is
mathematical fidelity (Dick, 2007). In essence this means that a tool can represent maths
incorrectly. This also has to do, in my opinion, with the difference between "use to learn" and
"learn to use" (cTwo, 2007), as the latter means one has to know the shortcomings of a tool,
1
Information and Communication Technology. When use “tools” I mean “ICT tools”.
-9but this knowledge could also lead to a better understanding of a concept, thus a tool is used
to learn.
Another aspect is the underlying machine code for a certain tool. It often is the case that
certain extreme values yield strange results. So an important question is: "is the mathematical
fidelity of a math system good enough to support maths at secondary school?". It will almost
surely be a trade-off between this and the amount of time needed to improve the system.
Cognitive fidelity
This is the "degree to which the computer's method of solution resembles a person's method
of solution.". To make sure that transfer of knowledge or skill takes place
Technology and students
An important distinction in type of activity is between exploratory and expressive tools and
activities (Bliss & J., 1989). They reside on a continuum. So when a procedure is described
it´s exploratory but choosing one’s own procedure is expressive (albeit somewhat limited).
Initial play with a technological tool is often beneficial: it stimulates expression but also
builds a purposeful relationship with the tool , and thus instrumental genesis (Guin &
Trouche, 1999) can take place. However, structured guidance is often necessary, as to avoid
the "play paradox" (Hoyles & Noss, 1992). This means that " playing" with a tool sometimes
enables students to accomplish an activity without learning the intended concepts. To solve
the paradox "reflection" on the task at hand is advised.
When studying student use of a tool the construct of a 'work method' could work: Guin and
Trouche (1999) see five work methods: random work method, mechanical work method, a
resourceful work method, a rational work method and a theoretical work method.
The combination of the type of activity (exploratory, expressive) and work method should
enable us to deduct what students are thinking.
Technology and practice
In it important that there is pedagogical fidelity in tool use. This means that students actually
learn what the teacher has intended. So here we consider the match between technology and
practice..
An interesting choice is whether sometimes "privileging" is appropriate: using tools when
basics are known and rules are "internalized" (in a sense trivialized). Before that, tool use is
prohibited. The concept of privileging can also apply to certain mental activities, like proofs
etc. This coincides with the white box versus black box discussion (Buchberger, 1989), which
states that “privileging” with tools –meaning that tools may only be used when a concept is
understood- is necessary. This prevents students from just “executing an algorithm" without
knowing what they are doing.
Using technology in practice also means that the teacher role could change. This is also an
aspect that can be studied throught the construct of teacher role, e.g. Counselor and Technical
Assistant (Zbiek & Hollebrands, 2007).
- 10 The use of technology changes this role. But: this could very well clash with teacher's
expectations. Beaudin and Bowes (1997), and later Zbiek and Hollebrands introduced the
PURIA model for CAS implementation:
personal Play,
personal Use,
Recommendation,
Implementation,
and Assessment.
This could also be applied to the implementation of DME2 use in the Netherlands.
Technology and curriculum
There are several reasons why technology is adapted:
Representational fluency: technology makes it possible to move easily between several
representations. This also belongs to good design principles for technological environments
(Underwood et al., 2005). For example, the applet Algebra Arrows has a
multirepresentational aspect. One could think of the distinction: context-table-graph-formula.
Mathematical concordance is another construct looking at the level intended and real
knowledge building are the same or not. In analyzing the way that teachers and students
interact with cognitive tools, it is helpful to consider the mathematics of the tool, the
mathematics of the teacher, the mathematics the teacher intended through particular
technology-based activities, the mathematics that the student engaged in as a result of the
technology-based activity arranged by the teacher, and the mathematics that is learned.
Amplifiers and Reorganizers can be used to describe curricular roles of technology. (Pea
1985) Amplifiers accept the goals of the current curriculum and work to achieve goals better.
Reorganizers change the goal of the curriculum or the way the goals are obtained by
replacing, adding and reordering parts. I would say the WELP3 project would be an amplifier.
“The French school”
In the early 90s the use of Computer Algebra Systems was seen as a possible means to get rid
of manipulations (routine skills) and focus more on concepts and complex problem solving.
Artigue, in “the French school”, noticed that actual tool use should be scrutinized, as to
discover obstacles and difficulties in the classroom. Her thoughts in the 90s on this become
clear in this quotation:
“... we needed other frameworks in order to overcome some research traps that we were
more and more sensitive to, the first one being what we called the “technical-conceptual
cut”. Indeed, theoretical approaches used at that time ... tended to use this reference to
constructivism in order to caution in some sense the technical-conceptual cut, and we
felt the need to take some distance from these. Artigue (2002, p.247)”
2
3
Digital Mathematical Environment, http://www.fi.uu.nl/dwo/en/
Wiskunde En Les Praktijk, dutch project that was aimed at using applets in an algebra curriculum
- 11 Two approaches were combined to overcome these traps. On the one hand Vérillon and
Rabardels work on instrumentation ((1995), the ergonomic approach) and on the other hand
Chevallards anthropological approach (Task, Techniques used to solve Tasks, Technology or
Talk used to explain and justify Techniques, and Theory).
Instrumental approach
The instrumental approach of tool use is easily summed up In this research the instrumental
approach of tool use is more important:
Instrument = artefact + instrumentation scheme.
Verillon and Rabardel (1995) distinguish an artifact and an instrument. An artefact is only the
tool. An instrument is the psychological notion: the relationship between a person and an
artifact. Only when this relationship is established one can call this a "user agent". The mental
processes that come with this are called schemes.
Instrumental genesis is the process of an artifact becoming an instrument. In this process both
conceptual and technical knowledge play a role (again, "use to learn" and "learn to use"). In
instrumental genesis three aspects come together: task, theory, technology. They are closely
connected.
Trouche (2003) distinguishes a tool component with instrumentation (how the tool shapes the
tool-use) and instrumentalisation (the way the user shapes a tool), and also a psychological
component with schemes (Piaget & Inhelder, 1969) . According to several studies (Artigue,
(2002); (Guin, Ruthven, & Trouche, 2005) genesis for computer algebra systems is a
timeconsuming and lengthy process.
When focusing on particular aspects of instrumental genesis, for example instrumentation,
instrumentalization and technique (Guin & Trouche, 1999), it becomes more clear how
students can use tools more effectively and what obstacles hinder conceptual and technical
understanding of a tool. Trouche sees three functions: a pragmatic one (it allows an agent to
do something), heuristic (it allows the agent to anticipate and plan actions) and epistemic (it
allows the agent to understand something).
Also instrumental orchestration concerns the external steering of students’ instrumental
genesis (Guin & Trouche, 1999)
So there are also conceptual aspects within the cognitive instrumentation schemes. The
instrumental approach provides a good framework for looking at the relation between tool use
and learning from an individual perspective. Yackel and Cobb (1996) argued that
coordinating both perspectives is expected to explain a lot on the advent an use of computer
tools. Tool use and learning is especially apparent in mathematics education. Integration in
the classroom is essential, and to understand this we need to observe instrumentisation.
Anthropological approach
As tools are used in practice, a context, in activity one’s view on practice becomes important.
In the anthropological approach we discern task, techniques, technology and theory:
- 12 Technology and theory can be defined as knowledge per se
Task and technique: know-how relevant to a particular theory and technology
So, Artigue and Lagrange focus primarily on Task and Technique. Also, the distinction
between, for example, pragmatic (efficient) techniques for doing tasks, and epistemic
techniques (more focused, as I see it, on concepts on real symbol sense)
As Artigue says:
“Professional worlds as well as society at large have a pragmatic relationship with
computational tools: their legitimacy is mainly linked to their efficiency. But what
schools aim for ... is much more than developing an effective instrumented
mathematical practice. The educational legitimacy of tools for mathematical work has
thus both epistemic and pragmatic sources: tools must be helpful for producing results
but their use must also support and promote mathematical learning and
understanding. (Artigue, 2005)”
So use to task, theory and technique go hand in hand. Again citing Lagrange (1999):
“The argument for this is essentially in five parts:
1. Technical work does not disappear when doing mathematics with CAS, it is
transformed.
2. Within a theory, every topic has an accompanying set of tasks and techniques.
Novices progressively become skilled in techniques by doing, talking about,
and seeing the limits of techniques. This eventually leads to a theoretical
understanding of the topic.
3. Although rote repetition of a specific technique for a specific task is a
mathematically impoverishing experience, this is not a reason to jettison
techniques per se.
4. Techniques and schemes are linked. Students need time to develop rich
schemes by using techniques.
5. The empirical observation that diminishing the role of techniques encourages
teachers to avoid talking about them (Chevallard’s “technology”).”
Monaghan (Monaghan & Ozmantar, 2006) points out several issues in “the French school”,
for example the tension in the “technique”. He wonders whether the two approaches could be
integrated.
Two missing aspects in French theory are teachers and affect. Concerning the first: one could
say that orchestration has to do with teachers. This however seems more concerned with
“what can be done” and not so much a description of teachers’ practices. Perhaps it would be
good if studies would focus more on developing accounts of teachers coming to instrumental
genesis. Concerning affect: as it surely has a large impact on tool use (beliefs, attitudes,
motivations, emotions) it should have more attention.
Monaghan (2005) also has argued that de notion of schemes could be cut by looking towards
“activity theory”:
“Activity theory considers the actions of a person towards an objective (affective motives are
thus essential). Activity is mediated through artifacts, social procedures, and language, and
Trouche may find the ideas of mediation relevant to his considerations of orchestration.”
So here we see common ground between two frameworks: mediation can take place through
artifacts, procedures and language, corresponding well with the notion of orchestration.
- 13 -
C. ICT tool use in this research
We claim that tools can facilitate in the learning of algebraic skills. Therefore we want to
study how instrumental genesis takes place when using a prototype.
- 14 -
3. Assessment
In this chapter we look at the role of assessment in learning..
A. Problem statement
How can we efficiently make use of assessment for learning, and in particular for learning
algebra?.
B. Theoretical overview
Introduction
Black and William (1998) have argued for many years now that assessment should be a part
of learning and apart from learning. They describe how attention goes out to formal methods
of assessment, but that there are informal ways of looking at assessment. To make sure that
the whole learning process is served well more attention should be given to this aspect of
learning. Just measuring a student's existing state of knowledge then, is not enough.
The Second Handbook of Research on Mathematics Teaching and Learning (Lester jr., 2007)
specifically deals with assessment as a " bridge" between teaching and learning. In my
research -apart from this aspect- I would also like to stress the use of assessment activities to
promote learning based assessment.
Purpose of assessment
Black and Williams (2004) broadly see three functions:
 supporting learning (formative)
 certifying the achievements or potential of individuals (summative)
 evaluating the quality of educational programs or institutions (evaluative)
Summative assessment is also characterised as assessment of learning and is contrasted with
formative assessment, which is assessment for learning.
Summative assessment
In most curricula summative assessment is used. Summative assessment is mostly aimed at
grading and scoring. Some researchers argue that, instead of providing a certain grade which
seems to say what "level of knowledge" a student has, formative tests give the student an
insight into the nature of -for example- their misconceptions.
Formative assessment
Black and William have made a case for more formative assessment. In their article from
1998 they state that "improving formative assessment raises standards". Actively involving
the student, implementing formative assessment as an essential part of the curriculum and
motivating students through self-assessment are key benefits of formative assessment. Means
to do this are feedback, self-assessment, reflection and interaction. This makes a good case for
tools that aid these factors.
Formative assessment is a process in which self-reflection should result in more insight
(Crooks, 2001). Cowie and Bell (1999) define it as the process between teacher and student to
“enhance, recognise and respond to the learning”. Black and Wiliam (1998) only define
assessment as being ‘formative’ when the feedback from learning activities is actually used
to modify teaching to meet the learner's needs.
- 15 Formative versus summative
Recently a shift from more summative towards formative assessment has taken place. This
shift has also stressed the apparent disadvantages of formative assessment.
Firstly the fact that less teachers available means more work as formative assessment is more
time-consuming. Secondly modularization means that students learn about their mistakes
when finished with a subject. This promotes a grade-culture, for assessment -albeit formativeis used as a means to test whether a student has learned enough.
Here we clearly see tension between summative and formative assessment. As Black &
Wiliam (2004) put it:
“teachers seemed to be trapped between their new commitment to formative assessment and
the different, often contradictory demands of the external test system” (p. 45).
A positive connection between formative and summative assessment is sorely needed
(Broadfoot & Black, 2004) In this sense one could argue that summative and formative
assessment are potentially complementary (e.g.(Biggs, 1998); (Harlen, 2005)) and should be
integrated more (Shavelson, Black, Wiliam, & Coffey, 2002). This opposed to the viewpoint
that they should generally be kept apart (e.g. (Knight & Yorke, 2003))
Pre-emptive formative assessment
In reality education seems to tend to a balance of formative and summative assessment. They
have one element in common: they both are mostly conducted after learning. To be able to
actually learn during the process of learning means assessing throughout the course of a
module. Black and William call this pre-emptive formative assessment. It builds on
constructivist learning principles (Black & Wiliam, 2003) learning starts from the learner's
existing knowledge and learning entails actively incorporating new knowledge into an
existing knowledge framework. Using pre-emptive formative assessment means using
feedback as a central element in the learning process (Hattie, Biggs, & Purdie, 1996). Instead
of serving up feedback too little too late, feedback is used "pre-emptively", to make sure
whether a student is on the right track or not. The role of feedback in formative evaluation
was already stated in the 60s by Bloom (1969):
"Quite in contrast in the use of 'formative evaluation' to provide feedback and correctives at
each stage in the teaching-learning process. By formative evaluation we mean evaluation by
brief tests used by teachers and students as aids in the learning process. While such tests may
be graded and used as part of the judging and classificatory function of evaluation, we see
much more effective use of formative evaluation if it is separated from the grading process
and used primarily as an aid to teaching".
I contend that both a score-based approach, combined with the strength of formative
assessment is a good approach to learn more. Scoring has an inherent motivational aspect.
Also, adding several " modes" from practice to " exam" and in-between will facilitate
different uses of assessment, both formative and summative.
Carless (2007) suggests that a suitable timing for pre-emptive formative assessment is the
class, classes or a longer period preceding high stakes assessment. (see (Lester jr., 2007))
Framework for Classroom Assessment in Mathematics
- 16 In De Langes “Framework for Classroom Assessment in Mathematics” several principles for
good assessment in mathematics classrooms. This framework connects well to the OECD
framework that was designed for the PISA program.
“Principles for Classroom Assessment
1. The main purpose of classroom assessment is to improve learning
(Gronlund, 1968; de Lange, 1987; Black & Wiliam, 1998; and many others).
2. The mathematics is embedded in worthwhile (engaging, educative, authentic)
problems that are part of the students’ real world.
3. Methods of assessment should be such that they enable students to reveal
what they know, rather than what they do not know (Cockroft, 1982).
4. A balanced assessment plan should include multiple and varied opportunities
(formats) for students to display and document their achievements (Wiggins,
1992).
5. Tasks should operationalize all the goals of the curricula (not just the “lower”
ones). Helpful tools to achieve this are performance standards, including
indications of the different levels of mathematical thinking (de Lange, 1987).
6. Grading criteria should be public and consistently applied; and should include
examples of earlier grading showing exemplary work and work that is less
than exemplary.
7. The assessment process, including scoring and and grading, should be open
to students.
8. Students should have opportunities to receive genuine feedback on their
work.
9. The quality of a task is not defined by its accessibility to objective scoring,
reliability, or validity in the traditional sense but by its authenticity, fairness,
and the extent to which it meets the above principles (de Lange, 1987).”
In operationalizing these principles, especially principle 5, it is important to distinguish
different mathematical levels of thinking. In chapter 6 –when we choose our content- we will
see that De Langes “Assesment pyramid” has many similarities with other categorizations of
mathematical skill level.
The role of feedback
Using feedback after learning does not provide an incentive to actually use it to adjust
knowledge and/or beliefs. This means that feedback is often very ineffective (D. Carless,
2006), caused by the lack of iterative cycles of feedback and revision. In line with this
criticism Gibbs and Simpson (2004) have stated that timeliness and potential for student
action are key components of good feedback. This means that timing is important and that it
should be possible to act on the basis of the feedback provided, for example by enabling the
possibility to correct an answer. As some research suggests written feedback is less effective
than oral feedback (Boulet, 1990), as students pay little attention to teachers' written
comments (Zellermayer, 1989) or find them difficult to understand and act upon (Clarke,
2000). These two criteria form a basis for a pre-emptive approach of assessment. On the
teacher side a teacher should have insight into the misconceptions a student has.
Feedback is routinely applied to any information that a student is given about their
performance. Historically the term was used in systems engineering by Norbert Wiener in
1940 working on guns. In education feedback tends to be used differently. (Ramaprasad,
1983) says: " Feedback is information about the gap between the actual level and the
reference level of a system parameter which is used to alter the gap in some way" .
- 17 -
An interesting aspect, as Sadler (1989) noted, is that feedback is used to influence a certain
gap. So if we don't use recorded information for this purpose, it is not considered feedback.
This means that formative assessments -Ramaprasads feedback- are closely linked to
instructional consequences. The function of assessment becomes formative if student's
learning is served, through feeding feedback back into the system.
An example clarifies this: telling someone to work harder is not formative, as it doesn't
involve any feedback as how to work harder. Telling someone to use more steps when solving
an equation is formative.
Perhaps the distinction becomes even clearer if we see monitoring assessments provide
information of whether a student is learning or not, diagnostic assessments provide
information on what is going wrong and formative assessment provides information on what
to do about it.
Principles of feedback
In the learning process adapting instruction to meet students learning needs showed
substantial benefits, for example in studies by (Carpenter, Fennema, Peterson, Chiang, &
Loef, 1989) and (Black, Harrison, Lee, Marshall, & Wiliam, 2003). As the role of feedback
had to be taken into account, drill-and-practice use of the computer made formative
assessment difficult. An interesting question is whether the use of "more intelligent" new
technology makes a difference in this respect. (Bangert-Drowns, Kulik, Kulik, & Morgan,
1991) found that not being able to see the answer before trying a question is better. Also
giving details of the right answer, instead of just wrong or right, seemed more effective, as
other research has also confirmed ((Elshout-Mohr, 1994), (Dempster, 1991, 1992).
How effective is feedback?
Reviews conducted by (Natriello, 1987), (Crooks, 1988), (Bangert-Drowns et al., 1991) and
(P. Black & D. Wiliam, 1998) showed that that not all kinds of feedback to students about
their work are equally effective.
Mason and Bruning distinguish eight types of feedback based on available research {Mason,
2001 #157. Nyquist (2003) reviewed 185 studies in higher education, developing a typology
of different kinds of formative assessment:
Weaker feedback only: students are given only the knowledge of their own score or
grade, often described as " knowledge of results";
Feedback only: students are given their own score or grade, together with either clear
goals to work towards or feedback on the correct answers to the questions they
attempt, often described as "knowledge of correct results";
Weak formative assessment: students are given information about the correct results,
together with some explanation.
Moderate formative assessment: students are given information about the correct
results, some explanation, and some specific suggestions for improvement.
- 18 Strong formative assessment: students are given information about correct results,
some explanation, and specific activities to undertake in order to improve.
In these levels typology I do miss a category concerning feedback on the process of problem
solving. This also involves feedback on sub-parts of the solution. Also allowing for several
representations can provide insight in a given problem. In my research I will take into account
this process and representational feedback.
Several studies -including Nyquists- seem to show more effectiveness in assessment when
using feedback. The stronger the feedback the larger the effect seems to be. ((Elawar &
Corno, 1985); see (Greer, 2000)) for many more references).
Another issue is the use of scoring: scoring seems to have a negative impact (Butler & Nisan,
1986). Butler (1988) even concluded that the effects of diagnostic remarks completely
disappeared when grades were added. There also are indication that games (reference)
enhance motivation. So here we have a dilemma: do we use grades/marks and feedback
together, hoping motivation will overcome the disadvantages or not. We have indications that
games do motivate students, but how does this correspond with less motivation through
grading.
Felix (2003) asserts humanizing feedback is important and proposes this can be done by
providing structural hints, personalized hints, structural graphics, personalized graphics and
games.
It should also be noted that some research indicated that (Simmons & Cope, 1993) that the
possibility of using ICT tools, by providing room for the strategy "trial and improvement",
requires less mental effort than the "harder" way with paper-and-pencil. More is not always
better, scaffolding an exercise seems to be more effective than providing a complete solution
with a new assignment. It remains unclear how effective feedback actually is, because
the effectiveness seems to be determined by variables not yet well understood. It is also clear
that learning intentions play a significant role.
Of course, students themselves also play a large role: metacognition (more metacognition
from students enhance learning), motivation and learning. This is important to acknowledge
as we defined feedback as "influencing learning". But then we have to take into account those
factors that influence learning as well.
Nicol and MacFarlane (Nicol & MacFarlane-Dick, 2006) reinterpreted existing literature on
formative assessment and the use of feedback to their respective roles in self-regulated
learning. These “seven principles of good feedback practice” are provided in table 1.
Good feedback:
1. helps clarify what good performance is (goals, criteria, standards)
2. facilitates the development of self-assessment and reflection in
learning
3. delivers high quality information to students about their learning
4. encourages teacher and peer dialogue around learning
5. encourages positive motivational beliefs and self esteem
6. provides opportunities to close the gap between current and desired
performance
7. provides information to teachers that can be used to help shape
- 19 teaching.
Table 1: Seven principles of good feedback practice
(Nicol & MacFarlane-Dick, 2006)
Gibbs and Simpson (Gibbs & Simpson, 2004) were able to extract 11 conditions under which
assessment might support student learning and improve chances of success. Table 1 gives
these 11 conditions.
Assessment tasks [conditions 1-4]
· Capture sufficient study time and effort (in and out of class)
· Are spread evenly across topics and weeks
· Lead to productive learning activity (deep rather than surface learning)
· Communicate clear and high expectations.
Feedback [conditions 5-11]
· Is sufficient (in frequency, detail)
· Is provided quickly enough to be useful
· Focuses on learning rather than marks
· Is linked to assessment criteria/expected learning outcomes
· Makes sense to students
· Is received by students and attended to
· Is acted upon to improve work and learning
Table 2: Gibbs and Simpson’s (2004) 11 conditions
- 20 -
Computer Aided Assessment
In computer aided assessment (CAA) there has also been a shift from more summative use
towards formative. This of course corresponds with the qualities and limitations of computers.
As computers become more powerful and "smarter" more innovative use of feedback is called
for.
Harlen and Deakon Crick {Harlen, 2003 #155} conducted a review of studies to find out
whether there was evidence of the impact of the use of ICT for assessment of creative and
critical thinking skills on students and teachers. Evidence from two high-weight studies
showed that:
-
Teachers are helped in understanding students in their understanding of new material,
when information is stored an recorded.
Taking over some roles of assessing and providing feedback frees up time for the
teacher to focus on other ways to support learning.
Computer feedback during a test improves student performance when the same test
material is used again later on.
Sangwin's (2006) research on Computer Algebra Systems (CAS) provides an indication of
these new possibilities. For formative evaluation more open questions would be helpful,
contrary to multiple choice questions with distracters. Also, multi-step exercises, and
feedback on these steps, could provide formative evaluation for the student and teacher alike.
Types of questions
For CAA in mathematics Provided Response Questions (PRQ) are most common. This
category includes Multiple choice and matching questions. There are many problems with this
type of question: question distortion, only lower order thinking is stimulated and strategic
thinking. As Sangwin states this especially holds true for reversible mathematical skills where
one direction is significantly more difficult than the other, e.g. factoring and expanding.
Providing PRQ questions then probably means testing something the teacher doesn't intend to
test.
These types of questions correspond a lot with the well-known behavioural paradigm or “drill
instruction”. Also, because the questions and the answers are simple to construct and simple
to evaluate large-scale assessment, providing many numbers and figures (“16% of students
scored correctly on questions concerning integrals”) For management purposes this perhaps
would suffice, but for math educators a more qualitative approach would be better: what type
of mistakes are made, so I can address this issue in the next lesson. The learned lessons from
this, in fact a grouped list of misconceptions, could not only provide input for classroom
practice, but in the future also for improvements on software. This also holds for feedback on
the process of solving equations.
In another paper (Sangwin & Grove, 2006) also point out that ease of authoring tests for
teachers should not be neglected.
Implementing feedback in tools
Implementing more elaborate feedback in mathematics software is still is at an amateur level.
As Jeuring (2007) points out: much can be improved, especially on the level of providing
- 21 strategies. Approaches that can be used can be Sangwin's approach, but also based on rewrite
rules and state transitions (Goguadze, Gonzalez Palomo, & Melis, 2005). Unfortunately this
also means that interoperability issues (Goguadze, Mavrikis, & Palomo, 2006) exist.
Pyramid
C. Assessment in this research
We argue that a shift from summative assessment towards pre-emptive formative assessment
could be beneficial for learning. Feedback plays a large role in this type of assessment. We
will focus on using a tool that can pre-emptively perform formative assessment for learning
algebraic skills. Here we will use the eleven conditions under which assessment might support
student learning and improve chances of success (Gibbs & Simpson, 2004), as well as the
seven principles of good feedback practice (Nicol & MacFarlane-Dick, 2006). The ability to
cater for these conditions will be used –along with other criteria- to choose a tool for
assessment.
- 22 -
4. Integrating theory
We have seen that three “ingredients” come together in learning, testing and assessing
relevant mathematical skills.
I propose the following conceptual framework:
Assessment
ICT tools for assessment
Assessing algebraic skills
Learning
ICT tool use
ICT tools that facilitate
algebraic skills
Algebraic skills
Learning, testing and assessing relevant mathematical skills with ICT can be scrutinized in
three different ways:
 As tool use . Through instrumental genesis learning is facilitated.
 As ‘learning of algebra’. We look at symbol sense for ‘real learning’, and basic skills
for algorithms.
 As assessment. Whereby pre-emptive formative assessment not only shows the
learning result but also aids learning. Feedback is an important factor in this type of
assessment.
We aim to integrate the assessment of algebraic skills into one prototypical design, thus
giving us insight into these three aspects and their dependencies. This ‘triangulation’ helps in
creating a ‘body of knowledge’ on acquiring algebraic skills with the help of computer tools.
Hopefully this will lead to common ground within the research fields of algebra, assessment
and tool use.
We will now sum up the implications of this conceptual framework, and underlying
theoretical frameworks for the three topics, for our prototype.
- 23 -
5. Choosing content that makes symbol sense
In this chapter we will discuss how we choose questions for acquiring, testing and assessing
relevant algebraic skills. Together these questions will make up the content for our first
prototype. First we describe what we mean by relevant algebraic skills, then we use several
sources to choose relevant questions for our prototype and describe why they are relevant for
our research.
To choose our content we first have to determine what our algebraic focus will be. As we are
interested in ‘symbol sense’:
- What skill level does a question have?
- What type of ‘symbol sense’ does the question address?
Classification of skill level
In the NKBW project (2007) a classification for skill level of a question was used, involving
the letters A,B and C. This classification was used earlier in the Webspijkeren project (Kaper,
Heck, & Tempelaar, 2005), and drew on work by Pointon and Sangwin (2003).
Building on the general schema of Bloom's Taxonomy, Smith (Smith, Wood, M., &
Stephenson, 1996) came to eight categories.
Group A
1. Recall factual knowledge
2. Comprehension
3. Routine use of procedures
Group B
4. Information
transfer
5. Application in new
situations
Group C
6. Justifying and interpreting
7. Implications, conjectures
and comparisons
8. Evaluation
Using this classification scheme several exams and questions were classified . Some of them
were already analyzed in the NKBW project. From all these questions a selection was made,
based on the skill level of the question. Level “C” is almost impossible to cater for with ICT
tools. We decided to choose level B questions or level A questions with an adapted level B
approach.
It is interesting to point out the similarities of this approach to, for example, the assessment
pyramid by de Lange (1999) used for PISA, but also the TIMMS framework uses equivalent
skill levels:
These three levels are:
1. Reproduction, definitions, computations. (lower level)
2. Connections and integration for problem solving. (middle level)
3. Mathematization, mathematical thinking, generalization, and insight. (higher level)
We then analyzed, also with the help from work from NKBW project, documents that are
closely related with the expected algebraic skills going from secondary to higher education:
- 24 -
Several entry exams from university and HBO;
An exit exam for secondary education;
The book ‘basiswiskunde’ by Rob Bosch and v/d Craats;
Articles on symbol sense
Keeping in mind the skill levels we listed several suitable ‘symbol sense’ type questions. It
was now time to determine what to focus on.
Algebraic focus
Now the collection of questions had to be narrowed down. Final questions were selected by
focusing on Arcavi’s ‘flexible manipulation skills’ and the ‘choice of symbols’
In appendix Athe chosen questions are categorized into Arcavi behaviors and a rationale is
provided what every question is all about. Per question we answer:
1.
2.
3.
4.
5.
6.
Why is this an interesting question?
What skill or behavior is assessed here?
What answers do we expect?
What could be obstacles in answering this question?
What feedback could be given for this question?
What tools could be used to model this question?
As we use Arcavi’s categorization it is also a good idea to acknowledge his instructional
implications
-
symbolic manipulations should be taught in rich contexts which provide opportunities
to learn when and how to use these manipulations.
give a complex function (and graphing tool) and ask the function.
informal sense-making makes sense.
use algebraic symbolism early to empower symbols.
make use of post-mortem analysis of problem solutions
classroom dialogues and what if questions.
These implications will be used when we finalize the prototypical design and formulate a
didactical scenario
- 25 -
6. ICT Tools for assessment
In this chapter we will give criteria for ICT tools for assessment. Based on these criteria an
ICT tool will be chosen for the first prototype. This is our method:
1. First we compiled a large list of criteria, based on our theoretical framework and own
experiences;
2. From this list we extracted some minimal requirements and categories of requirements.The
categories were determined by collecting all possible criteria from the theoretical background,
grouping them and reformulating a category; The raw list of criteria can be found in
appendix B
3. A large list if tools was compiled, based on experiences from earlier projects, the Special
Interest Group Mathematics ‘toetsstandaarden’ working group {Bedaux, 2007 #160}, KLOO
research {Jonker, #22}, the FI math wiki on digital assessment and math software
(http://www.fi.uu.nl/wiki/index.php/Categorie:Ict), and google searches. As there are
hundreds of math tools an initial selection was based on the tool having at least some
characteristics of tools for assessment. Based on these requirements we:
4. First selected tools that met the minimal requirements. Tools that didn’t, were described but
not considered any further. For this we used a template.
5. Then the remaining tools were graded on the other categories by:
a. Browsing the web on more information and usage on the tool;
b. Installing the tool locally;
c. Using the tool with already existing content. We aim at using quadratic equations as
this tends to be subject that is catered for almost always;
d. Authoring our own content from chapter 5. Here it is possible that not all the finesses
of a tool become apparent. A minimal requirement is that authoring can be used. This
means for most tools that they have to be installed. For every installed tool I keep a
log of screenshots.
6. This resulted in a separate descriptions of the tools and a matrix, giving an overview of strong
and weak points of several tools.
Minimal requirements
-
Webbased: we find it essential that using the tool can be anytime, anyplace, anywhere, using
just a web browser.
Ease of authoring, configurable: it should be possible to add own content.
Actively developed: it is important that the tool is supported and has some sort of continuity.
Minimal math support: formulas should be displayed correctly and support basic mathematical
operations.
Storage of progress: it should be possible to store results, so students can come back later, if
necessary.
Categories
For every item we answer these questions



What does this item mean?
Why is it important?
How is it scored/weighted?
- 26 It is also important to note that we are scoring these tools in the light of assessment and algebra.
Therefore other aspects could give plus-points, but can never be the deciding factor.
Math specific
In this category we mention the variables that are -in our eyes- specific for mathematics. Here we
asked ourselves the question "what are -comparing with other subjects- aspects that distinguishes
math from other subjects.
1. Representational (sound representations, mathematical and cognitive fidelity)
It is important that the way a student can work resembles the 'paper-and-pencil' way. This
means being able to use graphs, tables and a certain resemblance to this way. We assert that
using multiple representations (Van Streun) and a connection to today's practices, increase
transfer.
2. Randomization
Assignments that are provided should not always be the same, but have varying values.
Randomization caters for this.
3. Multiple steps (within one question, from one to other question)
Here we distinguish two typesof 'multiple step' exercises.
a. Within a question: an open environment is provided and a student can choose what
path to follow. This can be just one step, but also twenty steps.
b. Between questions: some complex questions consist of several subquestions. Often
one questions builds on an earlier one.
4. Integrated CAS
For complex operations a CAS is necessary. We distinguish the possibility of using a CAS and
the availability of a CAS, and also the type of CAS.
Ease of use
5. Teacher, authoring (questions, text, links, graphs, multimedia)
Teachers should be able to make their own qustions, with text, links, graphs etc., in a userfriendly way. This aspect scores -based on my own usage- how well authoring can take place.
6. Student, usage (e.g. input editor, learning curve)
Students should be able to work with tool. This means that the user intrafce and structure of
the tool should be intuitive. The same holds for the input of mathematical formulae.
Registration
7. Answers
How much of the answers is stored? Are all the answers -also the wrong ones- stored, or only
the last answer.
8. Process
And how much of the process. How did a student come to a certain answer?
Assessment
9. Possibility of modes (pratice, exam)
Asessing formatively means that asessment is for learning. Therefore asessment takes place
during a course, not only at the end in an exam setting. Providing several modes is a good
facility.
10. Feedback (process, answer, global)
a. Process: giving functional feedback on the strategy used.
b. Answer: providing feedback on the answer given. The more specific this is, the more a
student can learn from it.
- 27 -
11.
12.
13.
14.
c. Global: overall mastery of a subject. Giving insight in a students progress on amore
global scale.
Hints
The possibility of providing hints or that hints are provided.
Review mode (what has he/she done wrong or right)
The possibility of scrutinizing ones wrong or correct answers, as to learn from them, including
process. The finer the granularity the more information -when needed- can be used.
Question types
Sometims only open questions are not enought. Other question types could provide more
flexibility.
Scoring
Being able to use game-like scoring could help motivate students 'getting the highest score'
Content management
15. Question management
How well can questions be managed? Can they be copied easily? Can modules be recycled
easily?
16. Use of standards
This also implies that a certain compliance to standards like QTI or SCORM is a plus.
17. Available content
If teachers are reluctant to make their own questions, a large user base and questions could
help.
Other
18. Cost
Using a tool in secondary and higher education . We mean the bare licenses for using the tool.
Other aspects of costs (support, training) are scored in aspects like 'ease of use' and
'continuity'.
19. License and modifiability
What type of license does this tool have? Does this license make it possible to modify the
software to ones own wishes (open source).
20. Technical requirements (also own installation)
How easy is it to install locally? How high are the technical requirements?
21. Continuity
Is there a community, firm or organisation that can provide help with a tool? Our experience is
that tools with small user bases and little support, provide less continuity.
22. Languages
How multilingual is the tool?
23. Stability and performance
Of course this item also depends on the somputer it is installed on. However, using and
installing a tool does give an impression on the overall performance and stability.
24. Looks
In education a tool should look good.
25. Integration for VLE
We propose to use the weights provided in the matrix in appendix E.
We would have liked to ask experts on the scores in the matrix, but this would imply that experts
should have detailed knowledge of all the tools. Although one could always argue that the scores
presented are arbitrary, we would contend that it provides a cgood picture of weak and strong points of
the tools.
- 28 Conclusion:






Wiris is an attaractive tool for standalone use within for example Moodle. However, the lack
of assessment functions means it is not suitable enough for our research.
WIMS probably is the quickest and most 'complete' of the tools with room for geometry,
algebra, etc. There also is a fair amount of content available, also in dutch. It is let down by
the feedback and the fact only one answer can be entered. Of course this can be programmed,
as it is a very powerful package, but here we see a steep learning curve.
Digital Mathematical Environment. Strong points are the performance, multisteps within
exercises plus feedback, authoring capability, SCORM export. Disadvantages: emphasis on
algebra (not extendable, dependent on the programmer), source code not avaialble.
STACK has a good philosophy with 'potential responses' and multistep questions. Also, the
integration with a VLE -unfortunately only moodle- is a plus. Installation, stability and
performance is a negative (slow and cumbersome), as well as its looks. It is also very
experimental, providing almost no continuity.
Maple TA has many points that STACK has, but with better looks and no real support for
'potential responses' and ' multistep questions'. They can be programmed, but this means -like
WIMS- coping with a steep learning curve. As it has its roots in assessment software question
types are well provided for. One could say that Maple TA started as assessment software and
his moving towards software for learning, while STACK started with learning and is moving
towards assessment software.
Activemath is more of an Intelligent Tutoring System than a tool for assessment. The question
module (ecstasy) is powerful, providing transition diagrams. However, authoring and technical
aspects make it less suitable for the key aspects we want to observe.
For summative assessment Maple TA is the best contender. For both summative (scores) and
formative assessment DME is best used, with STACK a runner up. For algebra DME is best used
because of the fact the 'process' is central. For all other mathematical topics WIMS is most versatile.
For an interactive book without assessment features Activemath could be used best. The best
standalone mathematical environment (CAS) is provided by Wiris
We conclude that DME is best suited for answering our research question, as it:
-
mixes formative and summative aspects of asessment into one tool;
provides an open algebra environment, opposed to the more closed questions (one answer) that
other tools provide;
is a stable and attractive tool (motivation);
The descriptions of the tools that were considered but did not meet the minimal requirements can be
found in appendix C. Descriptions of the tools that did meet the minimal requirements can be found in
appendix D. A comparative matrix assessing the strong and weak points of these tools can be found in
appendix E. In the long term we aim to publish these assessments in a wiki type environment so they
can be kept up to date.
Now we will use our tool to model the questions from chapter 5.
- 29 -
7. Designing the prototype and instruction
So based on a rationale and a problem statement, a conceptual framework was formulated,
leading to a motivated choice in ICT tool and content, plus design principles. In this chapter
we describe what prototype resulted from all of this. This prototype will be the prototype used
in the actual research cycles.
I want to make sure that transfer from the tool towards pen and pencil takes place (Kieran &
Drijvers, 2006). This is why I design an instructional sequence with both tool use and
pen/paper tests. This sequence has some similarities with the Hypothetical Assessment
Trajectory in the CATCH project.
For our prototype this means
 Formulate an orchestration for tool use
 A visual approach that facilitates transfer from computer to pen-and-paper
A prototype of the test can be found on
http://www.fi.uu.nl/dwo/voho
Login name: vohodemo
Password:
omedohov
In appendix F some screenshots can be found of the authoring process.
The first implementation has no randomization. The second implementation will have more
questions making use of randomization.
Authoring capabilities are available on request.
- 30 -
Part III Methodology
(This part is under construction for the research plan. Deadline: 1/4/08)
See research plan: hypothetical assessment trajectory
Self-assessment
It is tempting to quote here a postulate by Wiggins (1993): “An authentic education makes
self-assessment central.”
In 1998, Black and Wiliam were surprised to see how little attention in the research literature
had been given to task characteristics and the effectiveness of feedback. They concluded that
feedback appears to be less successful in “heavily-cued” situations (e.g., those found in
computer-based instruction and programmed learning sequences) and relatively more
successful in situations that involve “higher-order” thinking (e.g., unstructured test
comprehension exercises).
“Let us start with the Professional Standards for School Mathematics (NCTM, 1991). These
standards envision teachers’ responsibilities in four key areas:
 Setting goals and selecting or creating mathematical tasks to help students achieve
these goals.

Stimulating and managing classroom discourse so that both the students and the
teacher are clearer about what is being learned.

Creating a classroom environment to support teaching and learning mathematics.

Analyzing student learning, the mathematical tasks, and the environment in order to
make ongoing instructional decisions.”
The consideration of (a) the learning goals, (b) the learning activities, and (c) the thinking and
learning in which the students might engage is called the hypothetical learning trajectory
(Simon, 1995).
Our basic assumptions will be the following: there is a clearly defined curriculum for the
whole year—including bypasses and scenic roads—and the time unit of coherent teaching
within a cluster of related concepts is about a month. So that means that a teacher has learning
trajectories with at least three “zoom” levels. The global level is the curriculum, the middle
level is the next four weeks, and the micro level is the next hour(s). These levels will also
have consequences for assessment: end-of-the-year assessment, end-of-the-unit assessment,
and ongoing formative assessment.
Hypothetical Assessment Trajectory.
Some of the ideas we describe have been suggested by Dekker and Querelle (1998).
Before
Entry test. A short, written entry test consisting of open-ended questions.
During
During. While in the trajectory, there are several issues that are of importance to teachers and
students alike. One is the occurrence of misconceptions of core ideas and concepts
- 31 -

Short quizzes, sometimes consisting in part of one or more problems taken directly
from student materials.

Homework as an assessment format (if handled as described in our earlier section on
homework).

Self-assessment—preferable when working in small groups. Potential important
difficulties will be dealt with in whole-class discussion.
This ongoing and continuous process of formative assessment, coupled with the teachers’ socalled intuitive feel for students’ progress, completes the picture of the learning trajectory that
the teacher builds.
After. At the end of a unit, a longer chapter, or the treatment of a cluster of connected
concepts, the teacher wants to assess whether the students have reached the goals of the
learning trajectory. This test has both formative and summative aspects depending of the
place of this part of the curriculum in the whole curriculum.
- 32 -
Appendix A
This appendix describes the chosen questions in more detail, answering the following
questions.
1.
2.
3.
4.
5.
6.
Why is this an interesting question?
What skill or behavior is assessed here?
What answers do we expect?
What could be obstacles in answering this question?
What feedback could be given for this question?
What tools could be used to model this question?
1
1. This question addresses whether a student has ‘gestalt’ quality: does her or she
recognize similar parts of an equation.
2. Behavior #6: flexible manipulation skills.
3. When a student recognizes similar parts he or she would:
or
(Note: students could easily forget that
becoming 0 yields two answers.)
or
or
or
Students could also be attracted by the brackets and tempted to lose them. Perhaps
the fact that this is a lot of work will keep them from doing so. Perhaps an easier
but similar question like
would be dealt with
this way.
4. Kop and Drijvers (Kop & Drijvers, in press) mention some causes: firstly the fact
that students see formulas as recipes (process) instead of coherent objects.
Secondly visual characteristics of questions play a role (‘visually salient’). In
question one it is tempting to lose the brackets. In question 2 it is tempting to
square the roots, as to lose them. In both cases this gives a lot of work, but brings
us no closer to the solution. Thirdly a lack of flexibility in manipulating
expressions. Fourthly a lack of meaning. Perhaps using a model helps
understanding what is asked. Lastly there often is a lack of practice in solving such
questions. With this last point we explicitly contend that insight also comes from
practicing. Or to quote Freudenthal:
“Advocates of insightful learning are often accused of being soft on training.
- 33 Rather than against training, my objection to drill is that it endangers retention of
insight. There is, however a way of training — including memorisation — where
every little step adds something to the treasure of insight: training integrated with
insightful learning.”
(Freudenthal, 1991)
5. Perhaps giving the possibility of substitution, or at least making visual or symbolic
groupings could help a student. Pointing out that a student should look for
corresponding terms.
6. Here giving the possibility to plot functions and thus to see what solutions an
equation has, could be helpful. A tool should also enable students to algebraically
solve an equation, preferably in ‘What you See Is What You Get’ form, to help
transfer from the tool to ‘pen and paper’.
2
solved for v
1. This question addresses whether a student has ‘gestalt’ quality: does her or she
recognize similar parts of an equation, what characteristics are ‘visually salient’
2. Behavior #6: flexible manipulation skills.
3. This question has to do with –as Arcavi puts it- ‘gestalt’ and ‘circularity’. Wenger
(Wenger, 1987) :
“If you can see your way past the morass of symbols and observe the equation #1 (
which is required to be solved for v) is linear in v, the
problem is essentially solved: an equation of the form av=b+cv, has a solution of
the form v=b/(a-c), if a≠c, no matter how complicated the expressions a, b and c
may be. Yet students consistently have great difficulty with such problems. They
will often perform legal transformations of the equations, but with the result that
the equations become harder to deal with; they may go “round in circles” and after
three of four manipulations recreate an equation that they had already
derived…Note that in these examples the students sometimes perform the
manipulations correctly…”
4. Here the roots are important: they attract attention, but actually trying to take
squares on both sides would be a mistake. The chance that students are not able to
continue or come ‘back where they were’ (circularity) is large. Also, the fact that
there are two variables, contrary to question 1, poses a problem.
5. Perhaps giving the possibility of substitution, or at least making visual or symbolic
groupings could help a student. Pointing out that a student should look for
corresponding terms, but also what ‘solved for v’ means.
6. Here giving the possibility to plot functions and thus to see what solutions an
- 34 equation has, could be helpful. A tool should also enable students to algebraically
solve an equation, preferably in ‘What you See Is What You Get’ form, to help
transfer from the tool to ‘pen and paper’. Two variables should be supported.
3
1. This question addresses whether a student has ‘gestalt’ quality: does her or she
recognize similar parts of an equation, what characteristics are ‘visually salient’
2. Behavior #6: flexible manipulation skills.
3. This question again has to do with –as Arcavi puts it- ‘gestalt’ and ‘circularity’.
This equation can be solved ‘the easy way’ of ‘the hard way’. Noticing that
4. Here the square and the brackets are important: they attract attention, but actually
rewriting the left side of the equation would be a mistake. The chance that students
are not able to continue or come ‘back where they were’ (circularity) is large. This
mistake could be even more tempting to make when
is used.
5. Perhaps giving the possibility of substitution, or at least making visual or symbolic
groupings could help a student.
6. Here giving the possibility to plot functions and thus to see what solutions an
equation has, could be helpful. A tool should also enable students to algebraically
solve an equation, preferably in ‘What you See Is What You Get’ form, to help
transfer from the tool to ‘pen and paper’.
4
Find the pattern and proof the result always is 2
(source: Martin Kindt)
1. This question addresses whether a student understands that using certain symbols
could help solving a problem.
2. Behavior #5: the choice of symbols. In this question Arcavi would call some
premonitory feeling for an optimal choice of symbols an important part of symbol
sense.
3. Here we expect students to make a choice of symbols and then rewrite the
expression. So there are two steps. It could also be interesting to see whether the
choice of symbols differs, but of course both giving 2 as answer. For example:
- 35 -
versus
Choosing symbols wisely also could make the next step easier or more difficult.
4. Of course, not choosing ones symbols wisely here –or not even having a clue as to
how to choose symbols- poses a problem in this question.
5. A suggestion for choosing a variable good be given, for example “remember that
subsequent numbers could be modeled by n, n+1, n+2”.
6. A tool should be able to ask for an expression, determine whether it is correct, and
enable the student to rewrite the expressions. So the solution process is very
important.
5
Of a function f(x) we know:
How much is
?
(Note: the original question is a multiple choice question)
1. This question addresses whether a student has ‘gestalt’ quality: does her or she
recognize similar parts of an equation.
2. Behavior #6: flexible manipulation skills.
3. Students would have to recognize that a function is not given, and hopefully be
triggered that something else plays a role, in this case transformations. If this is
recognized it should not be too hard to see that the function was shifted two to the
right and the integral as well.
4. Here the difficulty of the question lies in the fact that the function f is unknown.
Students will probably wonder how the integral can be calculated without actually
knowing the function. A more specific question arises when we choose a certain
function, or first start with two question with given functions, and then the general
statement.
5. Hinting on the fact that this is a general statement - it holds for all function- is
important. Therefore any chosen function could provide insight. Also the process
of transformation of a function is an important concept.
6. Here giving the possibility to plot functions could be helpful.
- 36 -
Appendix B
Categories
Content: algebra
 Multistep exercises, process feedback
 Sound representations and operations (!!!): mathematical and cognitive fidelity
 Possibility of using more than one representation;
Multiple representations can help better understanding of concepts.
 Randomization
 Wysiwyg formulas and input editor
 Integrated CAS
 Level of competencies.
 Gradual difficulty curve. Coherence and balance.
Technical: tool
 Results and answers should be stored and accessible for both student and teacher.
 Sharability and use of standards (questions and units)
 Easy authoring for teachers (because teachers are often “neglected learners” (Sangwin
& Grove, 2006))
 Scoring as a game, for motivation.
 Other tools available
 Performance
 Possibility of using multimedia
 Ease of use
 Customizability by teacher
 Contexts.
 Prerequisites server and client-side
 Stability
 Languages
Assessment
 It should provide feedback at the right moment
 Several "modes" ranging from diagnostics to exam. “Zoom” level. Several training
modes (from practice to exam).
 Enables a qualitative analysis of student work to reap the benefits of formative
evaluation;
 on a meta-level scoring should also be implemented to allow for quantitative data;
 A student should be able to correct an answer. Possibility to correct an answer
 The combination of elements from both summative and formative assessment should
enable a teacher to adapt better to student's learning.
 Go from feedback only to weak formative assessment or even moderate. This has a lot
to do with the state of the art. (level of feedback)
- 37  The 11 conditions under which assessment might support student learning and
improve chances of success.
 Seven principles of good feedback practice.
 Open questions providing an expressive environment
 Halfopen/closed questions. Open/closed questions
 Functional feedback on the answers
Formative assessment
 Feedback on the process`
 Global feedback: mastery level within curriculum
 Possibility of providing hints
 One answer versus multiple steps
 Storage of answers
 Storage of the process
 Question management
 Adaptivity
General
 Licenses
 User base for content availability
 Continuity
- 38 -
Appendix C
Not considered: WebLearn from RMIT Melbourne, mathxl from Addison-Wesley, Algebra
Interactive, Core-Plus Mathematics
Minimal requirements
-
Webbased: we find it essential that using the tool can be anytime, anyplace, anywhere, using
just a web browser.
Ease of authoring, configurable: it should be possible to add own content.
Actively developed: it is important that the tool is supported and has some sort of continuity.
Minimal math support: formulas should be displayed correctly and support basic mathematical
operations.
Storage of progress: it should be possible to store results, so students can come back later, if
necessary.
DITwis
Name
Date
Version
Key words
DITwis
20080217
DIT5
Too difficult too author. No multiple step solutions. No CAS in the
background. Web-based javascript. SCORM possible.
http://wiskunde.stmichaelcollege.nl/DITwis/
Screenshots
Doesn’t
too difficult to author needed functionality
adher to
minimal
requirement
- 39 -
Algebra Tutor
Name
Date
Version
Key words
Algebra Tutor
20080217
Unknown
Interesting Eliza-type tutor. There seems to be no authoring option. Only
algebra, no graphics.
http//www.algebratutor.org
Screenshots
Doesn’t
Not possible to add own questions
adher to
minimal
requirement
- 40 -
Calmaeth
Name
Date
Version
Key words
Screenshots
Calmaeth
20080229
Unknown
https://calmaeth.maths.uwa.edu.au/
Doesn’t
Not possible to add own questions
adher to
minimal
requirement
- 41 -
Math Xpert:
Name
Date
Version
Key words
Math Xpert
20080217
Unknown
Standalone windows. Not web-based. But with steps. 89.95 dollar per license
for the three topics covered. This looks like a suitable program, but it’s not
web-based. This would hinder the ‘anytime-anyplace’ criterium.
http://mathware.stores.yahoo.net/mathpert.html
It Shows the Steps!
MathXpert focuses on learning mathematics by helping the student to work through
problems successfully. Students often encounter difficulty in problem solving
because they've made a trivial mistake that compounds in later stages, or because
they've forgotten an important step of the strategy. Either of these problems result in
the student becoming stuck, blocking momentum and halting the learning process.
MathXpert is designed to eliminate these roadblocks by actively helping students
solve any problem correctly. The easy-to-use interface allows the student to focus
on the correct strategy of problem-solving.
Screenshots
Doesn’t
adher to
minimal
requiremen
t
Not web-based.
- 42 -
Aplusix
Name
Date
Version
Key words
Aplusix
20080217
II
There are four modes: training, test, self-correction, observation. Two
dimensional editor. Scores. 400 patterns of exercises. A disadvantage: it’s not
web-based, so every students needs a copy of this software. Replay mode for
looking at the process.
Aplusix is a new sort of software for arithmetic and algebra which lets
students solve exercises and provides feedback: it verifies the
correctness of the calculations and of the end of the exercises. Aplusix
has been designed to be integrated into the regular work of the class: it
is close to the paper-pencil environment, it uses a very intuitive editor of
algebraic expressions (in two dimensions); it contains 400 patterns of
exercises. Experiments in several countries and in several situations,
from 2002, have had very positive results, measured with pre-test and
post-test.
http://aplusix.imag.fr/en/index.html
Distributor is http://www.rhombus.be/index1.html . For 600 euro there is a
sitewide license. As far as I can see then you need to pay 10 euro per student.
Screenshots
Doesn’t
Not web-based
adher to
minimal
requirement
- 43 -
L’Algebrista
Name
Date
Version
Key
words
L’Algebrista
20080217
Unknown
http://www-studenti.dm.unipi.it/~cerulli/LAlgebrista/index.php?lang=ita doesn’t
work. http://telearn.noe-kaleidoscope.org/warehouse/Cerulli-M-Mariotti-M2000-bis.pdf and
http://www.dm.unipi.it/~didattica/CERME3/proceedings/Groups/TG8/TG8_Ceru
lli_cerme3.pdf are papers about this program.
Screensho
ts
Doesn’t
URL doesn’t work
adher to
minimal
requireme
nt
- 44 -
webMathematica
Name
Date
Version
Key words
webMathematica
20080217
2
Only seems to provide a frontend for the mathematica program. There are
some quizzes on algebra. I’m not impressed. There also is a license fee.
http://www.wolfram.com/products/webmathematica/index.html
Screenshots
Doesn’t
No assessment software, storage of results.
adher to
minimal
requirement
- 45 -
Wiris
Name
Date
Version
Key words
Wiris
20080218
2
Because of the integration with moodle we included this in the matrix, even
though one minimal requirement hadn’t been met.
http://www.mathsformore.com/
http://www.wirisonline.net/
Screenshots
Doesn’t
adher to
minimal
requirement
No assessment software, storage of results.
- 46 -
AiM: Assessment in Mathematics
Name
Date
Version
Key words
AiM: Assessment in Mathematics
20080217
3.0
This open source program is a . Maple is running in the background. This is
probably –Maplesoft has it’s Maple TA- why it has been discontinued. The
developers wanted a similar program, based on open source CAS. See STACK
and CABLE
"AiM is an open-source system for computeraided assessment in mathematics and related
disciplines, with emphasis on formative
assessment."
http://maths.york.ac.uk/aiminfo/
Screenshots
Doesn’t
Software discontinued
adher to
minimal
requirement
- 47 -
CABLE
Name
Date
Version
Key words
CABLE
20080217
1.0
Similar to a very first version of STACK and following up on the AiM
program. In the background the open source CAS axiom runs.
Overview
Computer algebra systems (CAS) are well-established research tools for
performing symbolic manipulation of mathematical expressions, such as
algebra and calculus. CABLE is designed to be a cost-effective, online
infrastructure for writing, testing and databasing lightweight and flexible
mathematical learning objects.
This infrastructure allows both staff and students to interact with computer
algebra for the following functions:
Instantiation and delivery of objects with randomised parameters.

Evaluation and feedback of answers to questions contained in these
objects. Typically each object has one question associated with it.

Generation of worked solutions from templates with reference to the
instantiated parameters.

Simple analysis of answers from cohorts of students.
Design has preferred formative learning rather than assessment.
http://www.cable.bham.ac.uk/
Screenshots
Doesn’t
adher to
minimal
requirement
Software not maintained
- 48 -
Hot potatoes
Name
Date
Version
Key words
Hot Potatoes
20080217
6.2
Quiz software
The Hot Potatoes suite includes six applications, enabling you to
create interactive multiple-choice, short-answer, jumbledsentence, crossword, matching/ordering and gap-fill exercises
for the World Wide Web. Hot Potatoes is not freeware, but it is
free of charge for those working for publicly-funded non-profitmaking educational institutions, who make their pages available
on the web. Other users must pay for a licence. Check out the
Hot Potatoes licencing terms and pricing on the Half-Baked
Software Website.
Unfortunately the mathematics support is quite bad. By using SCORM
tracking scores within a VLE is possible. No support for a CAS.
http://web.uvic.ca/hrd/halfbaked/
Screenshots
Doesn’t
No good math support
adher to
minimal
requirement
- 49 -
Question Mark Perception
Name
Date
Version
Key words
Question Mark Perception
20080217
Unknown
Is assessment software with no particular support for CAS related questions.
Use of MathML is possible.
http://www.questionmark.com/. Example at
http://www.questionmark.com/us/tryitout_k12.aspx
Screenshots
Doesn’t
No good math support
adher to
minimal
requirement
- 50 -
Wintoets
Name
Date
Version
Key words
Wintoets
20080217
4.0 WEB
Is assessment software with no particular support for CAS related questions.
Dutch.
http://www.drp.nl/Producten/WinToets-40-WEB.html
Screenshots
Doesn’t
No good math support
adher to
minimal
requirement
- 51 -
Moodle quiz module with extensions
Name
Date
Version
Key words
Moodle quiz module
20080217
All moodle versions
Limited question types but can be extended with STACK (see appendix C)
http://www.moodle.org . Communication through the OPAQUE protocol:
http://docs.moodle.org/en/Development:Open_protocol_for_accessing_question
_engines As this technology actually uses STACK in the default installation, we
will look at the possibilities. Moreover: Moodle is required for STACK.
Screenshot
s
Doesn’t
adher to
minimal
requireme
nt
No math support (CAS)
- 52 -
Wallis
Name
Date
Version
Key words
Screenshots
Wallis
20080217
Unknown
http://www.maths.ed.ac.uk/~wallis/
Doesn’t
Software not maintained
adher to
minimal
requirement
- 53 -
WebWork
Name
Date
Version
Key words
WebWork
20080217
2.4.1
WeBWorK is a web-based interactive system designed to make homework in
mathematics and the sciences more effective and efficient. It can be used with
moodle.
http://webwork.maa.org/moodle/
Screenshots
Doesn’t
Difficult to install and use
adher to
minimal
requirement
- 54 -
TI interactive
Name
Date
Version
Key words
TI interactive
20080229
http://www.t3ww.org/pdf/TII.pdf
http://education.ti.com/educationportal/sites/US/productDetail/us_ti_interactive
.html
Screenshots
Doesn’t
No registration for assessment
adher to
minimal
requirement
- 55 -
Cognitive Tutor
Name
Date
Version
Key words
Cognitive Tutor
20080229
Unknown
http://www.carnegielearning.com/products.cfm
Screenshot: http://www.carnegielearning.com/products_algebraI.cfm
Screenshots
Doesn’t
No authoring function.
adher to
minimal
requirement
- 56 -
Algebra Buster
Name
Date
Version
Key words
Algebra Buster / Algebrator
20080229
Unknown
Looks nice.
http://www.algebra-online.com/Solving%20systems%20of%20equations.htm
http://www.softmath.com/algebra-help/free-printable-linear-equation.html
Screenshots
Doesn’t
Not web-based
adher to
minimal
requirement
- 57 -
Appendix D
Maple TA
Name
Date
Version
Key words
Screenshots
Maple TA
20080229
3.0
http://www.maplesoft.com/
Dutch distributor: http://www.candiensten.nl/software/details.php?id=26
Mathmatch: http://www.mathmatch.nl/onderwerpen.diag.php
More content: http://mapleta.can.nl/classes/kamminga/
Tested via: experiences from earlier workshops, {Heck, 2004 #2}
- 58 -
Digital Mathematical Environment
Name
Date
Version
Key words
Digital Mathematical Environment
20080229
Unknown
This evaluation is based on (past) experiences of the Galois and Sage projects.
Several tests were made in a special environment for secondary and higher
education topics: http://www.fi.uu.nl/dwo/voho
After registering the test becomes available.
schoollogin: vo-ho
student password: passw_leerling_vo
Screenshots
- 59 -
Activemath
Name
Date
Version
Key words
Screenshot
s
Activemath
20080229
1.0
http://www.activemath.org/
Activemath is a comprehensive tool with much more than is our focus in our
research (tutorial component, mathematical learner model).
- 60 -
STACK
Name
Date
Version
Key words
Screenshots
STACK
20080229
2.0
In the new version 2.0 of STACK Moodle is required. Installing STACK is
quite difficult. http://www.stack.bham.ac.uk/ The possibilities of Stack are
also evaluated using our knowledge of the earlier version 1.0 and the moodle
quiz module. There is a lot of higher education material.
- 61 -
Wims
Name
Date
Version
Key words
WIMS
20080217
3.57 (3.62 released)
http://wims.unice.fr/
Exercises can be made by using Createxo. Power modules are also possible,
but require more administrator rights. For dutch content:
http://wims.math.leidenuniv.nl/wims -> studentenbereik -> Gestructureerde
aanpak algebraische vaardigheden
(administrator: Relinde Jurrius)
username test, password gaav
Screenshots
- 62 -
Appendix E
A more detailed rationale for every score can be found in the worksheet matrix_tools_290208.xls
Wm=WIMS, S=STACK, M=Maple TA, D=DME, Wr=Wiris, A=Activemath
Tool
Wm S
M
D
Wr
A
Wgt Notes
Key aspect of our research is: formatively assessing symbol sense by using an ICT tool.
Therefore, even if an item is valuable, weights were determined by keeping this clearly in
mind.
Scr Item
1 Representational
3
3
3
5
4
3
Here we mean both the fact that the mathematics behind the tool should be sound, but
5 also that content can be displayed in several representations. (Van Streun, Janvier)
2 Randomization
5
5
5
5
1
3
4 One strong point of computer tools is that they can randomly create exercises.
Multisteps: within
3a question
4
2
2
5
1
3
Asking for one answer is one thing, being able to find your 'own way' towards an answer
5 is considered a strong point.
Multisteps: between
3b questions
2
5
2
2
1
2
The ideal tool should be able to connect several exercises into one exercise, making it
3 possible to make more complex (type C) questions.
4 Integrated CAS
4
5
5
3
5
5
An integrated CAS is a plus, but we should keep in mind that our research focuses on
3 relatively simple mathematics.
5 Teacher, authoring
2
3
3
4
1
2
As Sangwin stated teachers are 'neglected learners'. Ease of authoring should be an
5 important variable.
6 Student, usage
4
2
3
5
5
3
5 Of course, ease of use for students should also be taken into account.
Registration:
7 answers
3
5
4
4
1
2
We want a tool to store all answers a student gives in. Also storing wrong answers is a
5 plus.
- 63 -
Registration:
8 process
1
2
2
5
1
2
To get insight in WHAT a student has done right/wrong one has to know the 'way to an
5 answer'.
9 Possibility of modes
5
5
5
5
1
1
In a didactical scenario, both practicing and testing plays a role. Providing these modes is
5 a part of formative assessment
2
Providing feedback on the way a student answers a problem, could be a part of formative
assessment. Here we can distinguish general remarks (we see these as Hints) and
5 actually saying something about the process.
10a Feedback: process
1
1
1
3
1
10b Feedback: answer
3
5
3
4
3
5
The possibility of giving more feedback than right/wrong is deemed valuable. We score
whether it is possible, and how easy it is to author. Feedback on 'in-between' answers is
5 seen as a plus. Ease of authoring these questions is also taken into account.
10c Feedback: global
2
2
2
2
1
4
An overall view of one's mastery of a subject is seen as global feedback. It is valued
5 slightly lower than the other feedback types.
11 Hints
4
4
4
2
1
5
3 Feedback can also be provided in the form of hints.
12 Review mode
1
3
4
5
1
1
Essential for formative assessment, as a student can examine his/her answers and
5 mistakes, and subsequently learn from them.
13 Question types
3
2
5
1
1
3
Providing a variety of question types makes a tool more versatile. As an item it isn't as
2 important as the other items.
14 Scoring
3
4
3
3
1
1
We see scoring as a way to motivate students. This is contrary to literature on formative
2 assessment.
Question
15 management
3
4
5
3
1
2
Being able to copy, delete and recycle questions is an important aspect for practical tool
3 use.
- 64 -
16 Use of standards
2
3
4
4
3
4
Sharing content -both formulas, questions and whole packages- is easier when standards
3 are being used.
17 Available content
4
2
5
4
3
2
3 The availability of existing content.
18 Cost
5
5
1
3
2
5
2 The pricing of a tool.
License and
19 modifiability
5
5
1
3
2
4
2 We value open source higher than closed source, because we can easier adapt code.
Technical
20 requirements
1
1
3
4
4
2
We score the technical requirements of a tool. Here we make use of our experiences
2 installing the tool, but also previous experiences with the used technologies.
21 Continuity
2
1
4
3
4
3
It is very important that teachers can rely on a tool to be supported for a while. This score
4 reflects this.
22 Languages
5
2
2
3
4
4
1 Is the tool available in more than one language?
Stability and
23 performance
4
2
3
4
4
2
Being able to use the tool in classroom practice, with a fair amount of students, is
2 important. This score is also based on earlier experiments with tools.
24 Looks
2
1
3
4
4
3
1 Here we also mean the structure a tool provides.
25 Integration for VLE
3
5
5
5
3
3
2 3=it exists but very experimental, 5=exists and works
Tot. SCORE
285 309 318 374 204 271
- 65 -
Appendix F
Note: these
screenshots are
from the DWO, the
dutch equivalent of
the DME.
Here we see the
difficulty of ‘open’
questions. Of
course just
providing the
answer is quite
easy, but how can
we make sure that
partially correct
answers are
graded that way?
The solution in this
case was to add
this possibility to
the answer space.
However, this
limits the ‘open’
character of the
question
somewhat, as the
student has to
provide at least
that answer to get
all the points.
Rewriting both
sides without
brackets is
possible.
The question
remains whether
one should give a
hint.
- 66 This
implementation of
the second
question relies on
the expression to
be seen as an
equation. It
remains to be seen
whether other
expressions with
v= are graded as
correct or not.
In question 3
substitution is
possible.
Circularity –in the
form of rewriting
the left side of the
equation- is
possible, and can
also be awarded
points
- 67 Here we encounter
some difficulties
modeling the open
character of the
question. We could
force an
expression with n
to be used, but the
whole point is that
that a student
could choose
his/her own
variable(s).
Using hints and a
more closed
instruction could
also help.
Integrals can be
computed using
mathematica in the
background.
However, there is
no function here,
as the
characteristic
holds for all
functions.
- 68 Just using
DME/DWO in a
more limited way,
not asking steps
but only one
answer, is also an
option.
- 69 -
References
Arcavi, A. (1994). Symbol Sense: Informal Sense-Making in Formal Mathematics. For the
Learning of Mathematics, 14(3), 24-35.
Artigue, M. (2002). Learning mathematics in a CAS environment: The genesis of a reflection
about instrumentation and the dialectics between technical and conceptual work.
International Journal of Computers for Mathematical Learning, 7, 245-274.
Artigue, M. (2005). The integration of symbolic calculators into secondary education: some
lessons from didactical engineering. In K. R. D. Guin, & L. Trouche (Eds.) (Ed.), The
didactical challenge of symbolic calculators: Turning a computational device into a
mathematical instrument (pp. 231-294). New York: Springer.
Bangert-Drowns, R. L., Kulik, C.-L. C., Kulik, J. A., & Morgan, M. (1991). The instructional
effect of feedback in test-like events. Review of Educational Research, 61(2), 213238.
Beaudin, M., & Bowers, D. (1997). Logistics for facilitating CAS instruction. In J. M. J.
Berry, M. Kronfellner, & B. Kutzler (Eds.) (Ed.), The state of computer algebra in
mathematics education (pp. 126-135). Lancashire, England: Chartwell-York.
Bedaux, J., & Boldy, M. (2007). Ontwerpversie Aanbevelingen Toetsstandaarden Wiskunde.
Biggs, J. (1998). Assessment and classroom learning: a role for summative assessment?
Assessment in education, 5(1), 103-110.
Black, P., Harrison, C., Lee, C., Marshall, B., & Wiliam, D. (2003). Assessment for learning:
Putting it into practice. Buckingham, UK: Open University Press.
Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in
Education: Principles, Policy & Practice, 5(1), 7-73.
Black, P., & Wiliam, D. (1998). Inside the Black Box: Raising Standards Through Classroom
Assessment. Phi Delta Kappan, 80(2), 139-144.
Black, P., & Wiliam, D. (2003). In praise of educational research: formative assessment.
British Educational Research Journal, 29(5), 751-765.
Black, P., & Wiliam, D. (2004). The formative purpose: assessment must first promote
learning. M. Wilson (Ed.), Towards coherence between classroom assessment and
accountability.
Bliss, J., & J., O. (1989). Tools for exploratory learning. Journal of Computer Assisted
Learning, 5, 37-50.
Bloom, B. S. (1969). Some theoretical issues relating to educational evaluation. In R. W. T.
(Ed.) (Ed.), Educational evaluation: New roles, new means: The 68th yearbook of the
National Society for the Study of Education (pp. 26-50). Chicago: University of
Chicago Press.
Boulet, M., Simard, G. & Demelo, D. (1990). Formative evaluation effects on learning music.
Journal of Educational Research, 84, 119-125.
Broadfoot, P., & Black, P. (2004). Redefining assessment? The first 10 years of Assessment
in Education. Assessment in Education: Principles, Policy & Practice, 11(1), 7-27.
Buchberger, B. (1989). Should students learn integration rules? SIGSAM Bulletin, 24(1), 1017.
Butler, R. (1988). Enhancing and undermining intrinsic motivation: The effects of taskinvolving evaluation on interest and performance. British Journal of Educational
Psychology, 58, 1-14.
Butler, R., & Nisan, M. (1986). Effects of no feedback, task-related comments, and grades on
intrinsic motivation and performance. Journal of Educational Psychology, 78(3), 210216.
- 70 Carless, D. (2006). Differing perceptions in the feedback process. Studies in Higher
Education, 31(2), 219-233.
Carless, D. (2007). Conceptualizing Pre-emptive Formative Assessment. Assessment in
Education: Principles, Policy & Practice, 14(2), 171-184, 114.
Carpenter, T. P., Fennema, E., Peterson, P. L., Chiang, C. P., & Loef, M. (1989). Using
knowledge of children's mathematics thinking in classroom teaching: an experimental
study. American Educational Research Journal, 26(4), 499-531.
Clarke, S. (2000). Getting it right – distance marking as accessible and effective feedback in
the primary classroom. In S. A. (Ed.) (Ed.), Feedback for learning. London:
RoutledgeFarmer.
Cowie, B., & Bell, B. (1999). A model of formative assessment in science education.
Assessment in Education: Principles, Policy & Practice, 6(101-116).
Crooks, T. (1988). The impact of classroom evaluation practices on students. Review of
Educational Research, 58(4), 438-481.
Crooks, T. (2001). The Validity of Formative Assessments. Paper presented at the British
Educational Research Association Annual Conference.
cTwo. (2007). Rijk aan betekenis, visie op vernieuwend wiskundeonderwijs.
Dempster, F. N. (1991). Synthesis of research on reviews and tests. Educational Leadership,
48(7), 71-76.
Dempster, F. N. (1992). Using tests to promote learning: A neglected classroom resource.
Journal for Research and Development in Education, 25(4), 213-217.
Dick, T. (2007). Keeping the faith. Fidelity in technological tools for mathematics education.
In J. F. C. K. (Eds.) (Ed.), Research on technology and the teaching and learning of
mathetics: Syntheses, cases, and perspectives. (Vol. 2, pp. 333-339). Greenwich, CT.
Doerr, H. M., & Zangor, R. (2000). Creating Meaning for and with the Graphing Calculator.
Educational Studies in Mathematics, 41, 143-163.
Drijvers, P. (2003). Learning algebra in a computer algebra environment. Design research on
the understanding of the concept of parameter. University of Utrecht, Utrecht, the
Netherlands.
Drijvers, P. (2006). Context, abstractie en vaardigheid in schoolalgebra. Nieuw Archief voor
Wiskunde, 5, 198-203.
Elawar, M. C., & Corno, L. (1985). A Factorial experiment in teachers' written feedback on
student homework: Changing teacher behavior a little rather than a lot. Journal of
Educational Psychology, 77(2), 162-173.
Elshout-Mohr, M. (1994). Feedback in self-instruction. European Education, 26(2), 58-73.
Felix, U. (2003). Humanising automated online learning through intelligent feedback. from
http://nl.start2.mozilla.com/firefox?client=firefox-a&rls=org.mozilla:nl:official
Fey, J. T., & Good, R. A. (1985). Rethinking the sequence and priorities of high school
mathematics curricula. C. R. Hirsch & J. Zweng (Eds.), The secondary school
mathematics curriculum, 1985 yearbook 43-52.
Freudenthal, H. (1991). Revisiting Mathematcs Education, China Lectures. Dordrecht:
Kluwer Academic Publishers.
Gibbs, G., & Simpson, C. (2004). Conditions under which assessment supports students’
learning. Learning and teaching in higher education, 1, 3-31.
Goguadze, G., Gonzalez Palomo, A., & Melis, E. (2005). Interactivity of Exercises in
ActiveMath. Paper presented at the 13th International Conference on Computers in
Education, Singapore.
Goguadze, G., Mavrikis, M., & Palomo, A. G. (2006). Interoperability Issues between
Markup formats for Mathematical Exercises. from
- 71 file:///c:/Documents%20and%20Settings/chbo366/Mijn%20documenten/promotie/we
balt3.pdf
Greer, T. J. (2000). An Examination of the Effects of Feedback Type and Regulation Style on
Test Achievement and Review Strategy Using Computer Web-based Anatomy Labs
with First Year Kinesiology Students. University of Calgary, Calgary, Alberta.
Guin, D., Ruthven, K., & Trouche, L. (Eds.). (2005). The didactical challenge of symbolic
calculators: Turning a computational device into a mathematical instrument. New
York: Springer.
Guin, D., & Trouche, L. (1999). The complex process of converting tools into mathematical
instruments: The case of calculators. International Journal of Computers for
Mathematical Learning, 3, 195-227.
Harlen, W. (2005). Teachers’ summative practices and assessment for learning – tensions and
synergies. The Curriculum Journal, 16(2), 207-223.
Hattie, J., Biggs, J., & Purdie, N. (1996). Effects of learning skills intervention on student
learning: a meta-analysis. International Journal of Educational Research, 11, 187212.
Heid, M. K. (1997). The technological revolution and the reform of school mathematics.
American Journal of Education, 106(1), 5-61.
Hoch, M., & Dreyfus, T. (2005). Structure Sense in High School Algebra: The Effect of
Brackets: International Group for the Psychology of Mathematics Education, 35
Aandwind Street, Kirstenhof, Cape Town, 7945, South Africa. Web site:
http://igpme.org.
Hoyles, C., & Noss, R. (1992). A pedagogy for mathematical microworlds. Educational
Studies in Mathematics, 23, 31-57.
Janvier, C. (1987). Problems of Representation in the Teaching and Learning of Mathematics.
27-32.
Jeuring, J. (2007). Strategie feedback.
Jonker, V. WisWeb - digitaal toetsen.
Kaper, W., Heck, A., & Tempelaar, D. (2005). Beschrijving van kennis in twee dimensies
geïllustreerd aan twee cursussen aan de UM en UvA.
Kaput, J. J. (1989). Linking representations in the symbolic systems of algebra. S. Wagner &
C. Kieran (Eds.)(Research agenda for mathematics education: Research issues in the
learning and teaching of algebra), 167-194.
Kaput, J. J. (1992). Technology and mathematics education. In D. G. (Ed.) (Ed.), Handbook
of research om mathematics teaching (pp. 515-556). New York: MacMillan.
Kieran, C. (1996). The Changing Face of School Algebra. Paper presented at the 25th Annual
Conference of the International Group for the Psychology of Mathematics Education,.
Kieran, C., & Drijvers, P. (2006). The Co-Emergence of Machine Techniques, Paper-andPencil Techniques, and Theoretical Reflection: A Study of CAS Use in Secondary
School Algebra. International Journal of Computers for Mathematical Learning,
11(2), 205-263, 259.
Knight, P., & Yorke, M. (2003). Assessment, learning and employability: (SRHE / Open
University, Maidenhead).
Kop, P., & Drijvers, P. (in press).
Lagrange, J.-B. (1999). Complex calculators in the classroom: Theoretical and practical
reflections on teaching pre-calculus. International Journal of Computers for
Mathematical Learning, 4, 51-81.
Lagrange, J. B., Artigue, M., Laborde, C., & Trouche, L. (2001). A meta-study on IC
technologies in education: Towards a multidimensional framework to tackle their
- 72 integration. Paper presented at the 25th Conference of the International Group for the
Psychology of Mathematics Education.
Lange, J. d. (1999). Framework for Classroom Assessment in Mathematics.
Laughbaum, E. D. (2007). An Argument for Using a Function Approach When Teaching
Remedial Algebra/Algebra.
Lester jr., F. K. (Ed.). (2007). Second Handbook of Research on Mathematics Teaching and
Learning.
Monaghan, J. (2005). Computer algebra, instrumentation and the anthropological approach.
Paper presented at the CAME4. from
http://www.lonklab.ac.uk/came/events/CAME4/CAME4-topic1-Monaghan-paper.pdf
Monaghan, J., & Ozmantar, M. F. (2006). Comments on the Co-Emergence of Machine
Techniques, Paper-and-Pencil Techniques, and Theoretical Reflection. International
Journal of Computers for Mathematical Learning, 11(3), 351-360, 310.
Natriello, G. (1987). The impact of evaluation processes on students. Educational
Psychologist, 22(2), 155-175.
Nicol, D. J., & MacFarlane-Dick, D. (2006). Formative assessment and self-regulated
learning: A model and seven principles of good feedback practice. Studies in Higher
Education, 31(2), 199-218.
NKBW, C. (2007). Eindrapport Nationale Kennisbank Basisvaardigheden Wiskunde.
Nyquist, J. B. (2003). The benefits of reconstructing feedback as a larger system of formative
assessment: a meta-analysis. Vanderbilt University.
Piaget, J., & Inhelder, B. (1969). The psychology of the child. London: Routledge & Kegan
Paul.
Pointon, A., & Sangwin, C. J. (2003). An analysis of undergraduate core material in the light
of hand held computer algebra systems.
Radford, L. (2004). Syntax and Meaning. M. J. Høines and A. B. Fuglestad (eds.), Vol.
1(Proceedings of the 28 Conference of the international group for the psychology of
mathematics education (PME 28)), 161-166.
Ramaprasad, A. (1983). On the definition of feedback. Behavioural Science, 28(1), 4-13.
Sadler, D. R. (1989). Formative assessment and the design of instructional systems.
Instructional Science, 18(119-144).
Sangwin, C. (2006). Assessing Elementary Algebra with STACK.
Sangwin, C., & Grove, M. (2006). STACK: addressing the needs of the "neglected learners".
Paper presented at the WEBALT.
Shavelson, R. J., Black, P. J., Wiliam, D., & Coffey, J. E. (2002). On Aligning Formative and
Summative Assessment. Paper presented at the National Research Council’s
Assessment In Support of Instruction and Learning: Bridging the Gap Between LargeScale and Classroom Assessment Workshop.
Shaw, N., Jean, B., & Peck, R. (1997). A statistical analysis on the effectiveness of using a
computer algebra system in a developmental algebra course. The Journal of
Mathematical Behavior 16(2), 175-180.
Simmons, M., & Cope, P. (1993). Angle and rotation: Effects of differing types of feedback
on the quality of response. Educational Studies in Mathematics, 24(2), 163-176.
Smith, G., Wood, L., M., C., & Stephenson, B. (1996). Constructing mathematical
examinations to assess a range of knowledge and skills. International Journal of
Mathematics Education in Science and Technology, 27(1), 65-77.
Tall, D., & Thomas, M. (1991). Encouraging versatile thinking in algebra using the computer.
Educational Studies in Mathematics, 22, 125-147.
- 73 Trouche, L. (2003). Managing the complexity of human/machine interaction in a computer
based learning environment: Guiding students’ process command through
instrumental orchestrations. Paper presented at the CAME3.
Underwood, J. S., Hoadley, C., Lee, H. S., Hollebrands, K., DiGiano, C., & Renninger, K. A.
(2005). IDEA: Identifying Design Principles in Educational Applets. Educational
Technology Research and Development, 53(2), 99-112, 114.
Van Streun, A. (2000). Representations in applying functions. International Journal of
Mathematical Education in Science and Technology, 31(5), 703-725.
Verillon, P., & Rabardel, P. (1995). Cognition and artifacts: A contribution to the study of
though[t] in relation to instrumented activity. European Journal of Psychology in
Education, 9, 77-101.
Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes.
Cambridge, MA: Harvard University Press.
Wenger, R. H. (1987). Cognitive science and algebra learning. Cognitive science and
mathematics education, 217-251.
Yackel, E., & Cobb, P. (1996). Sociomathematical Norms, Argumentation, and Autonomy in
Mathematics. Journal for Research in Mathematics Education, 27(4), 458-477.
Zbiek, R. M., & Hollebrands, K. (2007). A research-informed view of the process of
incorporating mathematics technology into classroom practice by inservice and
prospective teachers. In M. K. H. G. B. (Eds.) (Ed.), Research on Technology in the
Learning and Teaching of Mathematics: Syntheses and Perspectives.
Zellermayer, M. (1989). The study of teachers’ written feedback to students’ writing: changes
in theoretical considerations and the expansion of research contexts. Instructional
Science, 18, 145-165.