Assessment Using
Online Assignment Management
Systems
Glenn Ledder
Department of Mathematics
University of Nebraska-Lincoln
gledder@math.unl.edu
Assessment Using
Online Assignment Management
Systems
1. Defining the terms
2. OAM advantages and disadvantages
3. Low-maintenance administration
4. Accurate and reproducible results
5. Getting meaningful results
Defining the Terms
“Assessment”
• Conflicting meanings
“Assessment”
• Conflicting meanings
– General meaning: measuring academic
success in any context
“Assessment”
• Conflicting meanings
– General meaning: measuring academic
success in any context
– Specific meaning: measuring academic
success in a broad context
“Assessment”
• Conflicting meanings
– General meaning: measuring academic
success in any context
– Specific meaning: measuring academic
success in a broad context
(larger than a single course)
Assessment vs Evaluation
• Dictionary definition:
– “assess”: to determine the importance, size,
or value of
– “evaluate”: to determine or fix the value of
Assessment vs Evaluation
• Dictionary definition:
– “assess”: to determine the importance, size,
or value of
– “evaluate”: to determine or fix the value of
• Connotation:
– “assessment” usually emphasizes the making
of summary judgments
– “evaluation” usually emphasizes the
measurement of quantitative value
Assessment vs Evaluation
• Assessment:
– measuring knowledge and understanding
• Evaluation:
– measuring achievement
Assessment vs Evaluation
• Assessment:
– measuring knowledge and understanding
– used for placement
• Evaluation:
– measuring achievement
– used to determine course grades
Assessment vs Evaluation
• Assessment:
– measuring knowledge and understanding
– used for placement
• Evaluation:
– measuring achievement
– used to determine course grades
• Test: an instrument used for assessment,
evaluation, or both
OAM Advantages
and
Disadvantages
OAM Advantages/Disadvantages
+ Immediate grading
(no wait for grading or machine scoring)
OAM Advantages/Disadvantages
+ Immediate grading
+ General purpose testing center
(no need for direct faculty involvement)
OAM Advantages/Disadvantages
+ Immediate grading
+ General purpose testing center
– Administrative difficulties
(How do you maintain the database?)
OAM Advantages/Disadvantages
+ Immediate grading
+ General purpose testing center
– Administrative difficulties
+ Algorithmic Q’s and symbolic A’s
(individual answers, no prompts)
OAM Advantages/Disadvantages
+ Immediate grading
+ General purpose testing center
– Administrative difficulties
+ Algorithmic Q’s and symbolic A’s
– Difficulties in writing questions
(How to avoid tricky, picky, sticky?)
Low-Maintenance Administration
Low-Maintenance Administration
• Lots of instructors and proctors
– Should require minimal knowledge
Low-Maintenance Administration
• Lots of instructors and proctors
– Should require minimal knowledge
• One or two faculty manage the system
– Should require minimal time
Low-Maintenance Administration
• Lots of instructors and proctors
– Should require minimal knowledge
• One or two faculty manage the system
– Should require minimal time
• Questions are “permanent.”
• Student records are “transient.”
The Math 106 EDU folder structure
106 COURSE
•Question banks
•Gateway exam
•Practice assignments
The 106 COURSE data is
“permanent.” The only
regular changes are to the
Gateway exam dates.
Maintained by faculty
manager.
The Math 106 EDU folder structure
106 COURSE
•Question banks
•Gateway exam
•Practice assignments
The 106 COURSE data is
“permanent.” The only
regular changes are to the
Gateway exam dates.
Maintained by faculty
manager.
106 CLASS
•Question banks
•Gateway exam
•Practice assignments
•Student records
Students register for their
own 106 CLASS. Tests
are inherited. Student
records are local.
Maintained by individual
instructors/proctors.
Accurate and Reproducible Results
Reproducibility
• Tests should measure something.
– The test has to yield consistent results in
multiple administrations for a single student.
Reproducibility
• Tests should measure something.
– The test has to yield consistent results in
multiple administrations for a single student.
• But questions must change when a test is
given at different times.
– This is especially important for evaluations.
Reproducibility
• Use template problems to get a great
variety of answers.
• Use template problems to get uniformity
of content and difficulty.
• Group problems into categories that are
consistent in content and difficulty.
The Math 106 Gateway Exam
10 questions, 8 correct to pass
1.
Elementary functions:
xn, sin(ax), cos(ax), tan(ax), eax, ln x, nx
2. Products
3. Quotients
4. Compositions
5. Compositions of compositions
6. Products with a composite factor
7. Compositions of products
8. Quotients with an embedded composition
9. Quotients with an embedded product
10. Functions defined by equations
Category 4 - Compositions
A, C, N > 0; B ≠ 0; K ≠ 0, 1;
The algorithm chooses 5 random integers
satisfying certain requirements.
Category 4 - Compositions
A, C, N > 0; B ≠ 0; K ≠ 0, 1;
X = t, u, v, w, x, y, z;
The algorithm randomly chooses an
independent variable.
Category 4 - Compositions
X = t, u, v, w, x, y, z; A,C,N>0; B≠0; K≠0,1
P = XN+B, XN+BX
Q = AXN+B, AXN+BX, sqrt(X)+B
S = sin AX, cos AX, tan AX
T = e -CX+B, eKX+BX
U = Ae -CX+B, AeKX+BX, A ln X, ANX
The algorithm randomly creates functions
P, Q, S, T, U, using some of the chosen
integers.
Category 4 - Compositions
X = t, u, v, w, x, y, z; A,C,N>0; B≠0; K≠0,1
P = XN+B, XN+BX
Q = AXN+B, AXN+BX, sqrt(X)+B
S = sin AX, cos AX, tan AX
T = e -CX+B, eKX+BX
U = Ae -CX+B, AeKX+BX, A ln X, ANX
F1 = sqrt(P), sqrt(S), sqrt(T)
F2 = SN, TN
F3 = ln Q, ln CS
F4 = eQ, eCS
F5 = sin Q, cos Q, sin U, cos U
The algorithm
chooses one of 5
functions based on
prior elements.
There are 38
templates, each
with 7 independent
variables
and at least one
parameter
Reproducibility
• Tests should measure something.
– The test has to yield consistent results in
multiple administrations for a single student.
Accuracy
• Tests should measure what
they
were intended to measure.
Accuracy
• Tests should measure what they were
intended to measure.
– Questions should have to be done right to be
counted right.
Accuracy
• Tests should measure what they were
intended to measure.
– Questions should have to be done right to be
counted right.
– Students who can do what the test is supposed
to measure should get a good score.
Counted right → Done right
• Savvy test takers can do well on multiple
choice tests even when they don’t
understand the material.
Counted right → Done right
• Savvy test takers can do well on multiple
choice tests even when they don’t
understand the material.
• For calculations, students should have to
type in the answer.
Counted right → Done right
• Savvy test takers can do well on multiple
choice tests even when they don’t
understand the material.
• For calculations, students should have to
type in the answer.
• For conceptual questions, use multiple
selection. (Which of the following…)
Can do → Good score
• Tricky: Misinterpretation or small mistakes
account for many wrong answers.
Can do → Good score
• Tricky: Misinterpretation or small mistakes
account for many wrong answers.
Find the derivative of cose2x
and
Find the derivative of e-2 cos x
are tricky
Can do → Good score
• Tricky: Misinterpretation or small mistakes
account for many wrong answers.
Find the derivative of cose2x
and
Find the derivative of e-2 cos x
are tricky, compared to
Find the derivative of e2 cos x
Can do → Good score
• Tricky: Misinterpretation or small mistakes
account for many wrong answers.
Find the derivative of cose2x
and
Find the derivative of e-2 cos x
are tricky, compared to
Find the derivative of e2 cos x
The problem is
meant to test the
chain rule, not
elementary
algebra.
Can do → Good score
• Picky: Too many details connect minor
errors with wrong answers.
Find the derivative of cos2(2x+3)+ 4 sin x.
Find the derivative of
5-2x
x
4
cos
(
2
x
e
).
Can do → Good score
• Picky: Too many details connect minor
errors with wrong answers.
Find the derivative of cos2(2x+3)+ 4 sin x.
Find the derivative of cos2(2x+3).
Find the derivative of
5-2x
x
4
cos
Find the derivative of
5
x
e cos
(
(
2
x
e
2
x
e
).
).
Can do → Good score
• Picky: Too many details connect minor
errors with wrong answers.
Find the derivative of cos2(2x+3)+ 4 sin x.
Find the derivative of cos2(2x+3).
Find the derivative of
5-2x
x
4
cos
Find the derivative of
5
x
e cos
(
(
2
x
e
2
x
e
).
).
The simpler
problems
are hard
enough.
Can do → Good score
• Sticky: Difficulties in transferring answer
from paper to computer.
2
2x
Find the derivative of —–.
x+3
2
4x(x+3)-2x
————
(x+3)2
Can do → Good score
• Sticky: Difficulties in transferring answer
from paper to computer.
2
2x
Find the derivative of —–.
x+3
2x2
Find the derivative of —– at x = 2.
x+3
2
4x(x+3)-2x
————
(x+3)2
32
—
25
Can do → Good score
• Sticky: Difficulties in transferring answer
from paper to computer.
2
2x
Find the derivative of —–.
x+3
2x2
Find the derivative of —– at x = 2.
x+3
2
4x(x+3)-2x
————
(x+3)2
32
—
25
2
2x
Type 4x(x+3)-2x2 / (x+3)2, get 4x(x+3)- ———2
(x+3)
Getting Meaningful Results
Accuracy
• Tests should measure what
they
were intended to measure.
Getting Meaningful Results
• Tests should measure what they were
intended to measure.
• What should OAM evaluations and
assessments be intended to measure?
Don’t give up paper evaluations
Don’t give up paper evaluations
• Use paper exams for questions that
demand partial credit and questions
where the answer is an integral, a graph,
or an explanation.
Don’t give up paper evaluations
• Use paper exams for questions that
demand partial credit and questions
where the answer is an integral, a graph,
or an explanation.
• Use OAM evaluation for routine
computations and basic concepts.
Don’t give up paper evaluations
• Use paper exams for questions that demand
partial credit and questions where the answer
is an integral, a graph, or an explanation.
• Use OAM evaluation for routine computations
and basic concepts.
• Testing takes time and effort. Decide what is
important and then create the test to match.
• Set high standards and allow retakes.
Do give up paper assessments
Do give up paper assessments
• Assess routine computations and basic
concepts.
Do give up paper assessments
• Assess routine computations and basic
concepts.
• Expect very few retakes. Qualified students
should pass on the first try.
• Provide necessary formulas online or as a
printed supplement. Test conceptual
understanding and knowledge, not memory for
trivia.