Effective strategies for online testing in large classes

advertisement
Eective strategies for online testing in large classes
Jonathan Kress
School of Mathematics and Statistics
University of New South Wales, Australia
Maple TA User Summit 2014
Jonathan Kress
Eective strategies for online testing in large classes
UNSW Australia and me
University of New South Wales
Sydney Australia
54000 students
Large Science and Engineering Faculties
25% international students
UNSW in top 8 of 35 Australian Universities
School of Mathematics and Statistics teaches approximately 3000
rst year students
Jonathan Kress
Senior lecturer in the School of Mathematics and Statistics
At UNSW since 2001
Creating and administering online tests since 2004
Jonathan Kress
Eective strategies for online testing in large classes
Courses using Maple TA at UNSW
School of Mathematics and Statistics
MATH1131/1141 Science and Engineering (1700)
assumed knowledge / class test preparation / online Maple /
invigilated Maple lab
MATH1151 Actuarial Stuides (200)
class test preparation / online Matlab
MATH1011 Lower level rst year course (100)
weekly tutorial preparation, online Maple
MATH1031 Maths for Life Sciences (400)
class test preparation, Maple
MATH1041 Stats for Life Sciences (400)
online quizzes, invigilated R or Excel lab
Higher Year Stats/Numerical Methods/Engineering Maths (2000)
intro to matlab, various online tests
Faculty of Engineering
Higher year undergraduate and Masters Engineering (1000)
mostly longer assignments
Jonathan Kress
Eective strategies for online testing in large classes
Why use online testing and Maple TA?
Regular testing encourages students to keep up to date
Provides immediate feedback, anytime, anywhere
Reduces labour on routine marking
(What should be done with that saving?)
Allows repeated attempts according to student need
Personalisation of learning
encourages collaboration that isn't just copying
Analysis of student responses can be used to improve teaching
Very exible algorithmic question generation and grading
It lacks features we don't want during invigilated tests
eg, messaging or access to course materials
Plan to use adaptive questions in the future
Jonathan Kress
Eective strategies for online testing in large classes
Dierent scenarios for use of Maple TA
Formative assessment for learning
Assumed knowledge quiz (placement test)
for students in MATH1131 who should be in MATH1011
Class test preparation
in the week preceding a written class test
Weekly tutorial preparation
a short online test preceding each week's tutorial
Progressive (mastery) tests
must pass each test to access the next test
Practice followed by actual test
unlimited practice but limited number of attempts count
Practice tests
no marks awarded, feedback only
Summative assessment of learning
Invigilated lab tests
traditional exam conditions but Maple TA graded
Jonathan Kress
Eective strategies for online testing in large classes
How students view Maple TA
Students expect to be rewarded with marks.
Students expect things to work without having to read instructions.
Student confusion about the naming of Maple TA and Maple
lead to an increase in student use of Maple!
From a MATH1131 survey in 2011:
Jonathan Kress
Eective strategies for online testing in large classes
Student comments (2013 and 2014)
Feedback from quizzes was very helpful.
Constant tests are good.
Online tests are mostly just annoying and take up valuable time.
(Course could be improved by) more online questions. . .
The online tests and Maple tests instead of assignments were the best
features for this course.
The online tests helps in getting a better grip over the subject.
(Best features of the course) on-line tests
I found the online tests provided a good revision tool as they provided
immediate feedback which I could use to improve.
Should have more smaller online tests on maple ta so that it forces
students to learn and be up to date with the course as it moves quickly.
Regular online and class tests prompt us to be continually up to date. . .
The online quizzes kind of force you to learn how to do each question, and
since you have 5 tries you can work on a question until you get it right.
More weighting on online tests. The time used to complete the online
tasks was not appropriate in comparison to the percentage it makes up.
Online quizzes were well set out. It was useful to have several attempts.
Jonathan Kress
Eective strategies for online testing in large classes
Engagement
Do students engage with tests?
In MATH1131 online tests are worth approximately 1% each.
Approximately 90% of students attempt each test.
In MATH2019 online tests do not count towards nal grade.
Between 1% and 10% of students attempt the tests.
Incentives
Marks count towards nal grade
0.5% per 30 minute online test seems sucient!
Immediate feedback
Flexibility
allow some tests to be missed or failed
eg only count best 8 of 12
allow repeated attempts
only count best mark
allow plenty of time
More and smaller tests to avoid unnecessary repetition
Jonathan Kress
Eective strategies for online testing in large classes
Set and forget
Our main current use of Maple TA is to provide practice questions and
feedback to students with minimal intervention.
Provide written and screen capture video instructions.
Link to Maple TA provided next to link to instructions.
Maple TA class homepage tells students to read the instructions.
Compulsory Using Maple TA test must be completed rst.
Jonathan Kress
Eective strategies for online testing in large classes
Using Maple TA
Jonathan Kress
Eective strategies for online testing in large classes
Using Maple TA
Jonathan Kress
Eective strategies for online testing in large classes
Set and forget
Allow multiple attempts.
Revision only versions of tests available after deadline.
Course outline states no extenstions or additional attempts allowed.
Only best attempt counts.
Allow some tests to be missed without penalty.
Questions contain embedded instructions.
Jonathan Kress
Eective strategies for online testing in large classes
Tutorial preparation tests in MATH1011
Encourage regular preparation
for tutorials.
MATH1011 has separate
algebra and calculus
components.
Weekly tutorial preparation
tests for the calculus introduced
in 2004.
Amnesty for missed tests in
week 5 of semester.
Course survey comments
requested similar tests for
algebra.
Evidence of increase in average
score for nal exam questions
related to online tutorial prep
questions.
Jonathan Kress
Eective strategies for online testing in large classes
Tutorial preparation tests in MATH1011
Jonathan Kress
Eective strategies for online testing in large classes
Class test preparation
Provide feedback to help students prepare for class tests.
Initially allowed 3 attempts but spent too much time dealing with
special pleadings for extra attempts so increased to 5 attempts.
Jonathan Kress
Eective strategies for online testing in large classes
Invigilated tests
Maple component of MATH1131 is assessed with invigilated lab test
students book time for test on Moodle
student ID cards are checked
computers (112) have exam mode with no internet access
Before 2010
paper based test
student Maple worksheets collected, printed then marked by hand
and returned in tutorial class
very labour intensive
students had no feedback prior to test
Since 2010
Maple TA test
test is available for unlimited practice on Maple TA before lab test
test restricted to lab IP range
easy immediate grading once all tests have nished
Since 2014
MATH1041 Stats for Life Sciences has an invigilated lab test partly
graded by Maple TA.
Jonathan Kress
Eective strategies for online testing in large classes
Maple Lab Test
Jonathan Kress
Eective strategies for online testing in large classes
Allowing for special cases
Although we say no extenstions or additional attempts will be
granted, in reality there will be special cases.
We use revision only versions for special cases.
Revision only tests also depend on Declaration and Using Maple
TA tests.
Revision only test become available as soon as `for marks' deadline
has passed.
Jonathan Kress
Eective strategies for online testing in large classes
Summary
Expect bugs in your questions.
Expect students to do uncontrolled tests collaboratively.
Students won't attempt optional tests.
Assume students won't read the instructions.
Assume students will start the test when they are not prepared.
But if you just want students to learn, the solution to all of this is. . .
Jonathan Kress
Eective strategies for online testing in large classes
Summary
Expect bugs in your questions.
Expect students to do uncontrolled tests collaboratively.
Students won't attempt optional tests.
Assume students won't read the instructions.
Assume students will start the test when they are not prepared.
But if you just want students to learn, the solution to all of this is. . .
don't worry!
Oer enough marks that the students care but you don't.
Provide lots of ways for lost marks to be recovered.
Have a strict non-intervention policy.
Jonathan Kress
Eective strategies for online testing in large classes
Download