Webinar Series
This session is part of e-Assessment Scotland 2013
28 Aug 2013:
Assessing algebra concepts - the potentials and
limitations of online delivery
Robyn Pierce, Caroline Bardini and Kaye Stacey (University of
Melbourne, Australia)
Your hosts
Professor Geoff Crisp,
Dean Learning and Teaching, RMIT University
geoffrey.crisp[@]rmit.edu.au
Dr Mathew Hillier,
Teaching and Educational Development Institute,
University of Queensland
mathew.hillier[@]uq.edu.au
Assessing algebra concepts.
The potentials and limitations of online
delivery
Robyn Pierce (Presenter)
Caroline Bardini
Kaye Stacey
University of Melbourne
Australia, Victoria, Melbourne
Presentation authors
Robyn Pierce
Presenter
Caroline Bardini
Kaye Stacey
Presentation Outline
Pilot project funded by the Australian government’s
Office of Learning and Teaching.
• Preliminary online mathematics survey –issues of
framing items for use online.
• Preliminary findings about first year university
students' understanding of functions
• Examples of Specific Mathematical Assessments
that Reveal Thinking (SMART tests) proposed to
identify student thinking and provide feedback
and advice to teaching staff.
Project rationale
Recent report from Australia’s Chief Scientist
(Mathematics, Engineering & Science in the National Interest (Chubb et al, 2012))
The proportion of
students at school
studying
mathematics and
science is still
declining, especially
at the highest level,
and in universities is
virtually flat.
Figure 1.4 Student participation in mathematics, 19952010
Project rationale
• The need to increase the success rate in the
mathematical sciences at first year and
improve their sense of mastery and
confidence to continue is vital.
• “This is not about ‘dumbing down’. It is not in
Australia’s interest to produce a raft of people
with second rate skills and knowledge.”
(Chubb et al, p20).
• Previous research has shown that many
people who are apparently ‘successful’
secondary school students harbour incorrect
or incomplete conceptions of fundamental
mathematical ideas.
• Through extensive targeted practice in Year
12, students may ‘paper-over’ their
fundamental difficulties and become adept at
answering examination questions and so
proceed to university without robust
understanding.
Project rationale
• Strategies already developed to support
students include Mathematics and Statistics
Learning Centres where individual assistance
is available to students, especially those in
first year.
• Diagnostic tests are not conducted at
Melbourne because they are currently
– too expensive
– too time consuming
– there is limited opportunity for remediation.
Project rationale
Assessment for Learning
10
Assessment for learning
Diagnosing
Interpreting
& planning
Teaching
Diagnostic
assessment
Tutors Content
and
Curriculum
Knowledge
Tutor’s Mathematics
Pedagogical Content
Knowledge
Expect university mathematics tutors have
11 strong CK but limited MPCK:
Project details -Stage 1
Aim
Develop a current evidence base for incorrect or
incomplete fundamental algebraic conceptions
prevalent amongst commencing first year
undergraduate mathematics and statistics
students.
Project details – Stage 1
Evidence
(i) data from all first year undergraduate
mathematics and statistics students at the University
of Melbourne via a brief survey and timed screening
task through the university’s Learning Management
System
(ii) ‘think aloud’ interviews with volunteer students
(iii) interviewing experienced teaching staff
Project details -Stage 2
Make innovative use of technology to produce
sample Specific Mathematical Assessments that
Reveal Thinking (SMART tests) to add a different
dimension to the assessment for learning that
has been offered in the past to first year
students.
Project details -Stage 3
Produce
online
feedback,
illustrating
appropriate teaching strategies and examples
for use with students who are identified as
having specific misconceptions.
Project details: focus
Pilot project limited to algebra.
Undergraduates’ misconceptions in two key areas
of algebra to be identified: “variables, unknowns
and parameters” and “functions and variation”.
Resources will target the most common first year
“hot spots”.
Focus on functions
• Verbalise: define or describe
• Recognise functions in multiple representations:
– Algebraic
– Graphic
– Numeric
• Manipulate functions
On-line quiz
Maths quiz via LMS –Advantages
• Platform widely used by uni teachers across faculties
and subjects (repository of lecture notes,
announcements –not exclusive to assessment)
• Students are familiar with it (no extra burden on
technology skills)
• LMS provides direct link to students’ information
(subject, etc.)
• Variety of types of assessments (anonymous surveys,
tests, etc – each with different features)
• Export of students’ responses directly to Excel
(however in a rather undigestible format…)
Features of our quiz
• One attempt only
• Limited on time (35 minutes) – most
questions to be answered «quickly»
(encouraging students’ first ‘reactions’), no
calculations should be needed
• Track record on number of accesses (with or
without attempts), date, time, duration of
attempt (‘unaswered’ responses at the end of quiz by a
student who spent 35min might indicate a lack of time
rather than ‘I don’t know how to do it’)
Disadvantages – item design
Maths editor is limited e.g. no simultaneous
equations. Need to use Word editor, print
screen, crop, save as image and then « attach »
(see next slide). Lengthy process!
Maths editor limitations
Disadvantages –item design
Quite rudimentary tools. Less friendly
environment for students and little flexibility in
using multiple features at once (e.g. cannot
have multiple choice for different subquestions – see next slide)
Rudimentary look/tools
Originally:
(taken from SMART
test)
Multiple
choice for
each response
Rudimentary look/tools
On-line:
(LMS)
Pen-and-paper vs online
Originally:
We hoped to have
multiple choices for
each a,b,c,d (as in
e.g. a menu). But
limitations forced
‘blanks’.
Pen-and-paper vs online
Online:
To avoid constructed response (i.e. type into blanks as in pen-and-paper), we chose
multiple choice. But this forced to separate each sub-question
Pen-and-paper vs online
• What was one question in pen-and-paper has become 4!
• Might discourage students as it adds length to the test
• Burden on unnecessary reading (by repeating same stimulus 4 times)
Disadvantages – item responses
• Although exported to Excel, a couple of hours had to
be spent on re-arranging data into a more readable
format (yet still hard to interpret)
• Maths editor not available for students’ responses.
Various consequences:
– On design of the item (e.g. Originally : “f(x) = x2-x+2. Find
f(f(x))”. Online: multiple choice for the coefficient of x2 in
f(f(x))). Relieve students from typing a complex expression
and avoid typos.
– On the format of the answers collected (see next slide)
Disadvantages – format of answers
Need to have a human to
look at each answer to
decide if right or wrong (e.g.
assuming typos, accepting
whole equation –and not
only answer, accepting non
expanded form, etc.)
Looking back…
Even with its limitations, good support for
our purposes
Results to date
• Misconceptions reported in the literature are
still prevalent
Reading, classifying, key word search
Percentage of students
including this concept
n=383
Rule
Unique y-value
Vertical line test
Mapping
One-to-one or many-to-one
Graph
At least one valid statement
51
19
4
4
5
6
60
Dichotomy – auto-marked
Rule
Percentage who
responded correctly
n=383
Quadratic function
89
Linear function – general form
85
Hybrid - continuous
6
Hybrid- not continuous
44
Rule in words 2 variables
50
Rule in words 3 variables
61
Diagnostic test results and feedback
For automatically marked items:
– Number correct
– Items correct – incorrect
• No diagnosis of misconceptions
• Advice based on score would be vague
• We can do better – but this requires a more
sophisticated system
• The LMS quiz provided a quick screening tool
providing us with evidence of the ‘hot spots’ to
target
• Develop tests targeting prevalent, incomplete or
incorrect conceptions and feedback for teachers and
students.
SMART tests: what are they?
SMART tests: example
SMART test: feedback for teachers
SMART test: feedback for teachers
Teaching
suggestions
References
Project – to be done
• Development and trialling of SMART tests
targeting important ‘hotspots’ for starting
University mathematics
• still to be completed…
Thank you
Further questions?
Robyn Pierce
Caroline Bardini
Kaye Stacey
r.pierce@unimelb.edu.au
c.bardini@unimelb.edu.au
k.stacey@unimelb.edu.au
Try yourself at http://www.smartvic.com