Using QuestionMark Perception
for summative on-line assessment
Marija Cubric
May 2006
Context


Level: UG1
Module: Introduction to programming



Number of students: ~100



Practical “hands-on” subject that requires
some amount of thinking 
Cannot be learnt from books only
Mix of BISF, AMIS, BS, MB, ..
Last year failure rate: 56%
Main Issues:



Low attendance (>50% never showed up!)
Low motivation (does not count towards the
final grade)
Not enough contact time (1hpw tutorial – not
sufficient)
Solution (first iteration )

Attendance


Motivation


Weekly on-line test (20% of final
grade)
Publishing “league tables” (top 10, 10
most improved etc)
Contact time (tutorials)

Increased from 1hpw to 2hpw
Theory

Objective questions (MCQ/MRQ) can be designed to test
all levels of Bloom’s educational competencies:






Objective questions can be used in summative/ not only
formative assessment, but need to be mixed up with other
forms of assessment


Creativity, design and development, non-cognitive skills,
writing style etc – cannot be tested with objective questions.
Receiving immediate feedback is important part of any
constructivism*-based learning cycle such as Kolb’s
learning cycle


Knowledge
Comprehension
Application
Analysis
Evaluation
(*Constructivism - knowledge construction is social process
that occurs through collaboration with others)
Amanda Jeffreys & Andy Oliver (BLUE team) run w/shops
on how to design objective tests
Tool

QuestionMark Perception




3rd party sw, licensed to UH (trial)
Not part of StudyNet
http://www.questionmark.com/uk/hom
e.htm
Tool administrator: Angus Shaw
<r.a.shaw@herts.ac.uk>


To setup account for you
To create participant groups (module
code)
Features


Roles: administrator or participant
Administrator functions:
Question management (C/E/D)
 Assessment management
 Schedule management
 Group management
 Report management


Participants functions:
Start & submit test
 Review test results

Login

http://www.perception.herts.ac.uk/
em/login.asp
Question database
Types of questions
Configuring assessment
Configuring schedule
Report example (full score list report)



















Result index
Assessment ID
Assessment name
Assessment author
Assessment last modified
Participant
Participant group
Participant details
Hostname
IP address
Status
Date/time started
Date/time finished
Total score
Maximum score
Percentage score
Time taken
Topic percentage score
Topic result
Types of reports
Test example

Login to:

http://www.perception.herts.ac.uk/q/p
erception.dll
Feedback


Not immediate (except the score)
Run through the answers
immediatelly after the test (extra 510 minutes)
Results (tutor’s perception)

Increased attendance:


Increased motivation:


Average tutorial attendance: 75%
“League tables” very popular!
Better results?


Yet to be seen
Expecting raise of 10% in pass rate

“Liminality” (Glynis Cousin)

“mimicking” rather than “thinking”
(“cut&paste” strategy)
Results (students’ perception)

Data: anonymous questionnaire (74
respondents)
Using in-class test technology has helped my
learning of the subject
Weekly in-class tests were the main reason for
attending tutorials
I was looking forward to see the results of the
weekly tests published on the StudyNet
I would recommend use of the in-class test
technology in other BS modules
Using technology for weekly in-class tests was fun
69%
70%
62%
73%
59%
Results (usability)
Login procedure was easy and always worked
The URL was easy to remember
The application never crashed
I had no problems saving (i.e. submitting) my work
The time to load the in-class test pages was good
The in-class test technology was easy to use
The in-class test technology was easy to learn
82%
39%
35%
74%
58%
78%
66%
Workload




Creating questions (10min/q)
Creating assessments & schedules
(30min/w)
Importing reports to Excell &
generating league tables (30min/w)
Beware: extra time >4hpw (incl. 20
new questions)
Questions?

Can questions be parameterized &
parameters within question randomized


Can questions database be exported


NO
Questions-yes, not sure how to expeort
answers???
Can it be used outside of the campus

YES (with usual proxy settings for remote
access)
URLs

Admin login:


Participant login:


http://www.perception.herts.ac.uk/em/
login.asp
http://www.perception.herts.ac.uk/q/p
erception.dll
More on QuestionMark Perception:

http://www.questionmark.com/uk/hom
e.htm
Top 10 – see StudyNet for full
score table
hp8aa
fc6bc
lk6aa
ym6ah
el6ah
gr5ab
tw6ah
sy2ac
ar8aa
ls6ax
score
90%
90%
80%
80%
80%
80%
80%
80%
80%
80%
time
00:06:10
00:06:34
00:05:20
00:05:45
00:05:51
00:06:10
00:06:19
00:06:22
00:06:52
00:07:46
trend
53%
23%
21%
-6%
32%
10%
68%
158%
40%
13%
UP
UP
UP
DOWN
UP
UP
UP
UP
UP
UP
Top 10 (most improved) – see
StudyNet for full score table
ka8do
he6aj
sy2ac
rm6ca
od6ak
jl6an
kj8aw
az8aa
qm6ac
bs8cm
test3
50%
60%
50%
60%
50%
30%
70%
60%
40%
40%
new tot
29%
37%
31%
40%
35%
22%
52%
44%
31%
32%
prev tot
18%
25%
22%
30%
27%
18%
43%
37%
27%
28%
trend
59%
47%
44%
33%
28%
22%
22%
21%
17%
15%
UP
UP
UP
UP
UP
UP
UP
UP
UP
UP