Improving Pedagogical Efficiency and
Effectiveness using an Automated
Testing System
Ramon Lawrence, Alyosha Pushak,
Teresa Wrzesniewski
University of British Columbia Okanagan
Introduction
Technology can improve teaching efficiency and delivery, but
the most repetitive and time-consuming tasks of assignment
and test preparation and marking are still mostly manual.
Automated testing systems have the potential to dramatically
reduce the amount of time spent on these repetitive tasks,
freeing up instructor time for valuable student contact, and
allowing for more uniformity on student and instructor
achievement and evaluation.
We have developed the AutoEd system that is an automated
question generation and marking system and evaluated it in a
first year Physics course.
Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski
Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System
UBCO Research Week Presentation 2011
University of British Columbia
Copyright© 2011
Page 2
Background – Testing Systems
There is a variety of technology for student testing:
clickers, web-based learning, course management systems, etc.
It is challenging to determine which technologies are appropriate
for certain teaching styles and courses, and what are the correct
products and systems that provide the appropriate payback for
student learning and institutional deployment.
Most of the question systems are limited to static questions.
There is no learning value in repetitive practice.
The questions do not encourage or improve learning; they simply
test if previous learning has occurred.
Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski
Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System
UBCO Research Week Presentation 2011
University of British Columbia
Copyright© 2011
Page 3
Background – Question Templates
A question template is a question that tests a specific concept
that can be re-used by varying its contents.
Example: Calculate the distance travelled by a car in 2 hours
that is travelling 60 km/h.
Template: Calculate the distance travelled by a $x$ in $y$ that
is travelling $z$.
Randomization: For x, y, and z.
Answer: y*z (with correct units)
Incorrect answers can also be encoded with feedback.
Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski
Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System
UBCO Research Week Presentation 2011
University of British Columbia
Copyright© 2011
Page 4
Previous Systems
Previous question template systems:
WWWAssign built by Dr. Larry Martin for physics courses.
Currently a commercial product called WebAssign.
Aplia - social sciences
Mastering Physics
Gradiance - computer science
Carnege Learning – mathematics
These products have textbook questions and provide web
hosting. The cost is $15-$100 per student per course.
Course management systems such as WebCT/Blackboard have
question systems and some support for randomized questions.
Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski
Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System
UBCO Research Week Presentation 2011
University of British Columbia
Copyright© 2011
Page 5
Previous Systems – Open Source
There have been several university led efforts on building these
systems and providing access.
Quest Service at The University of Texas at Austin
many science/math questions
outside users can get access by requesting UT Austin accounts
LON-CAPA is an open source system built primarily at
Michigan State University.
It has hundreds of thousands of resources and questions that
are shared by multiple institutions. Institutions are part of a
sharing network for distributing questions and resources.
Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski
Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System
UBCO Research Week Presentation 2011
University of British Columbia
Copyright© 2011
Page 6
Selecting a System
Selecting an appropriate system depends on one key decision:
Who pays?
1) Student pays
another cost in addition to textbooks paid by students
students will be reluctant to pay especially if not used extensively
commercial systems are great from an instructor point of view
2) Institution pays
Highly expensive for the institution so likely limited to open
source systems. Both of which have their own restrictions.
An in-house system is the only other choice.
Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski
Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System
UBCO Research Week Presentation 2011
University of British Columbia
Copyright© 2011
Page 7
AutoEd System Overview
The AutoEd system was designed and built by Alyosha Pushak
(undergrad honours) and Dr. Lawrence in computer science.
It is written completely in Java and uses open source systems for
the database and web server.
The database stores all user question and answers.
Users interact with the system using a web browser.
Students can:
Practice questions and do assignments as given by the
instructor. Students receive immediate feedback.
Instructors can:
Create practice questions and assignments.
Examine student assignment progress and marks at any time.
Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski
Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System
UBCO Research Week Presentation 2011
University of British Columbia
Copyright© 2011
Page 8
AutoEd – Student Test
Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski
Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System
UBCO Research Week Presentation 2011
University of British Columbia
Copyright© 2011
Page 9
AutoEd – Question Authoring
Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski
Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System
UBCO Research Week Presentation 2011
University of British Columbia
Copyright© 2011
Page 10
AutoEd – Student Completion Rate
Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski
Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System
UBCO Research Week Presentation 2011
University of British Columbia
Copyright© 2011
Page 11
Physics Evaluation
The AutoEd system was evaluated in Phys 112 in Fall 2010:
2 paper assignments, 2 electronic assignments, 1 choice
Electronic assignments were converted from paper assignments
by adding randomization.
Surveys conducted at the first and last classes.
First survey: 105 out of 111 surveys completed (95.5% return)
Second survey: 76 out of 89 surveys completed (85.4% return)
Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski
Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System
UBCO Research Week Presentation 2011
University of British Columbia
Copyright© 2011
Page 12
Physics Evaluation Results
Format
choice: 90% of the students used AutoEd for all three
assignments and 92% used it for the last assignment.
Paper or electronic preference: 70% chose electronic while an
additional 15% chose both as equal.
Experience: Over 72% of the students chose satisfactory or
very satisfactory when asked about their experience using the
system while only 3% rated it as very unsatisfactory.
Help learning: Almost half of the students (45%) reported that
AutoEd helped their learning.
Completion: Students reported that AutoEd helped them
complete their assignments (81%) and hand their assignments in
on time (84%).
Other uses: Over 92% of the students thought AutoEd should
be used again in Physics 112, and 87% felt that it could be used
in other classes (chemistry, biology, mathematics, English).
Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski
Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System
UBCO Research Week Presentation 2011
University of British Columbia
Copyright© 2011
Page 13
Insights and Improvements
1) Watch technical issues with the system and questions.
Issues with incorrect marking and significant figures.
Question preparation takes time and all answers must be
checked before releasing the assignment.
2) Users will use the Internet to find answers.
5% of users wanted the system to block Internet searches.
Copy-and-paste electronic question versus typing it in!
Effectiveness as an evaluation tool as more answers are online?
3) Students really do wait until the last minute.
By removing the overhead of assignment generation and
collection/marking, more frequent, smaller assignments is better.
Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski
Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System
UBCO Research Week Presentation 2011
University of British Columbia
Copyright© 2011
Page 14
PHYS 122: Paper vs. Electronic
A1
A2
A3
A4
2010 - hand in
89%
94%
89%
84%
2011 - online
93% 90% 93%
98%
Average grade [%] 2010 - hand in 60.3 82.9 67.3
2011 - online 84.4 71.3 77.5
62.1
% handed in
Median grade [%]
86.3
2010 - hand in
67.1
92.2
77.9
75
2011 - online
100
100
100
100
Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski
Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System
UBCO Research Week Presentation 2011
University of British Columbia
Copyright© 2011
Page 15
PHYS 122: Question Writing
Q3: Using Kirchhoff’s rules for the circuit below find:
a)The current in each resistor. [8 marks]
b) The potential difference between points c and f [2
marks]
Randomize R3 from 4.00 kΩ to 10.0 kΩ, increments 1.0 kΩ.
Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski
Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System
UBCO Research Week Presentation 2011
University of British Columbia
Copyright© 2011
Page 16
PHYS 122: Question Writing (2)
Q2. A conductor suspended by two flexible wires (as shown
below) has a mass per unit length of 0.040 kg/m. What
current must exist in the conductor for the tension in the
supporting wires to be zero when the magnetic field, B =
3.60 T points into the page? Give your answer with 3 sig.
figs. [5 marks]
Randomize B from 2.60 T to 4.60 T, intervals 0.10 T.
Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski
Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System
UBCO Research Week Presentation 2011
University of British Columbia
Copyright© 2011
Page 17
Future Work and Conclusions
It is possible to build a customized question answer system
with minimal cost. Although the upfront costs and time is
higher, the payback in terms of student costs and instructor
flexibility is significant.
We are currently evaluating the system in several other
courses and will release it as open source to the community.
The system will be maintained and extended for many years.
Limited time offer:
We will create the questions for you over the summer for any
courses that you may be interested in using the system for!
Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski
Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System
UBCO Research Week Presentation 2011
University of British Columbia
Copyright© 2011
Page 18
AutoEd – Demonstation
Let's have a quick tour of the AutoEd system.
Ramon Lawrence, Alyosha Pushak, Teresa Wrzesniewski
Improving Pedagogical Efficiency and Effectiveness using an Automated Testing System
UBCO Research Week Presentation 2011
University of British Columbia
Copyright© 2011
Page 19