QA guidelines for use of Computer Assisted Assessment. November 2013
Last modified by Towers, Paul A.
Appendix A
Operational Guidelines for the use of summative CAA 2013/2014 – Semesters
1 and 2
The provision exists, University-wide, for all staff to use computer assisted
assessment as part of their summative assessment of learning, through the
QuestionMark Perception software.
This document outlines where the responsibilities lie for each of the different
tasks needed for rendering such an assessment. This Appendix needs to be
read in conjunction with the University Quality Guidelines for CAA.
These steps do not include preparing your students for a CAA, which should
follow the usual steps in briefing students for any summative assessments.
Preparation for using Summative CAA
Task
Who does this?
1. Agreeing the changes (if needed) to a module’s
Module team
assessment regime with FQECs
2. Training in CAA assessment design and the
CTEL – eLearning
writing of objective questions
Advisors with the
Module team
3. Question writing and peer review of the questions Module team –
to be used.
external examiner
Deciding on the marking scheme for each
question and the pass rate etc.
4. Question entry into the QuestionMark Live. This
produces a file with a .qpack extension and is
what CTEL need to create the test on the server.
Training on the different ways to enter questions
is available through CTEL and the eLearning
advisors.
5. Booking of the room for the CAA:


1
If this is in the University exam weeks then
Registry will book the room.
If it is, for example in week 6 or 7, then the
module team needs to book the computer lab.
Module team /
School
responsibility.
Training available
from eLearning
advisors.
Module team /
School
responsibility
(Registry if in
University exam
weeks)
QA guidelines for use of Computer Assisted Assessment. November 2013
Last modified by Towers, Paul A.
a. For CAA exams which fall within the exam period
ITS will ensure:


ITS responsible for
checking PCs and
network
Planned network or system outages do not take
place
ITS would respond to any help call, regarding
hardware issues, from the exam room as per the
AV hotline response – which will be placed on the
highest priority – and a member of the ITS team
will appear as soon as possible.
b. If the CAA falls outside of the exam period:
If the module leader informs ITS (with two
weeks’ notice) ITS will respond as in 6(a). IT
service number is 01902 322000 (Internal
extension 2000)
Module team /
School
responsibility for
telling ITS.
– Support will be on a best endeavours basis
CTEL will inform IT services of all QMARK’s that
are taking place.
6. Passing the finished set of questions to CTEL
(in .qpack format – see 4 above) – 4 to 6 weeks
before examination date.
Module team /
School
responsibility to
pass over .qpack
file to CTEL
7. Put the question sets into the QMARK server
software.
CTEL for mounting
questions into the
QMARK server
software.
This stage also generates the test’s unique
assessment ID and Group name. Students use
these two, together with their University login
details to access the assessment.
8. Assessment ID and Group name is to be passed
to the module team to test, 4-6 weeks before the
examination.
Disclaimer email to be sent to the module team,
that they must proof read and test the
examination.
2
CTEL
Module team to
test the
examination.
QA guidelines for use of Computer Assisted Assessment. November 2013
Last modified by Towers, Paul A.
9. Checking that the test performs as expected
when on the server – well in advance of the test
day to allow for corrections to be made to the
test questions, wording, typos, or to the scoring
schema.
Module team to
test the test and
inform CTEL if any
changes are
needed.
The day of the examination
Task
10. Arriving at the test room 30 minutes to 1 hour
before the test is due to start – switching on all
PCs and entering the QMARK login which is
available from CTEL. This locks down the PC and
only gives access to the QMARK assessment.
Who does this?
Module team /
invigilators /
School
responsibility
Additional support to lock down the computers
should be obtained within the faculty/department.
On a best endeavours basis a member of the CTEL
team will be present for the start of the test, given
advance notice of the date, time, and place of the
CAA.
CTEL DO NOT invigilate the examination.
11. Being on the end of the phone to handle / try and ITS
resolve any last minute issues or problems with
any hardware issues in the room. The ITS
response is as per the AV hotline response – which
will be placed as the highest priority (ex 2000) –
and a member of the ITS team will appear as soon
as possible. Opening hours for IT services 8am 7pm term time, 8am till 5pm vacation time.
3
12. Checking the IDs of all students entering exam
room will be done by Registry invigilators.
Module team /
invigilators /
Registry
13. Giving out test the Assessment ID and Group
name, and invigilation of the assessment is as per
University guidelines, and for invigilators.
After the examination
14. Extracting the results of the test, including the
analysis of the test questions and forwarding these
on to the module leader.
Module team /
School
responsibility
CTEL
QA guidelines for use of Computer Assisted Assessment. November 2013
Last modified by Towers, Paul A.
15. Changing the test questions in line with the
supplied test analysis data and statistics for future
use. It is recommended that 15% of questions be
refreshed annually.
Module team /
School
responsibility
1. Preparation for using summative CAA.
i. The use of CAA as a summative assessment method needs to be approved
and quality assured in the same ways as all summative assessments, through
validation or through the minor modifications procedure. Staff considering its
use should consult the Centre for Technology Enhanced Learning web pages
on CAA - http://www.wlv.ac.uk/default.aspx?page=25019
ii. The University only authorises the use of QuestionMark Perception for
computer-assisted summative assessments. Thus WOLF, PebblePad or other
software must not be used.
iii. ITS and CTEL support the Q-Mark Perception software and its application.
Where alternative, in-house software is used e.g., STECH, Schools need to
have local procedures to assure its fitness for purpose.
iv. Module leaders implementing CAAs need to ensure that the assessment
type is validated for use in the usual manner and obtain endorsement from
the School Quality Enhancement Committee for minor modifications, if a
module changes its assessment type.
v. Training in best practise in the use of summative CAA is available from
CTEL. UQEC recommends that all module teams using a CAA first attend a 2
hour workshop before scoping and writing their CAA assessment. Training in
the production of high quality tests and question writing is available from
CTEL. Training on the use of the software to enter questions is also available
from CTEL.
vi. There needs to be significant and clear communication with the students
about the tests. In advance of the test, candidates need to know all of the
following:a) What part of the syllabus is covered in the test
b) How many questions there are?
c) The ratio of this test to their final module grade
d) Where they find the practice test
e) How to use and navigate the software – a hand-out is available from
BLU to give out to, and email students,
f) The time allowed to take the test,
g) If they can bring anything else into the exam e.g., dictionaries,
calculators etc.
4
QA guidelines for use of Computer Assisted Assessment. November 2013
Last modified by Towers, Paul A.
h) The marking scheme e.g., if negative marking is being used, or some
questions carry more marks than others.
vii. Candidates need to have seen the QMARK test interface. This can be done
through distribution of an open URL via e-mail or by a mock examination.
viii. ITS provide a method of securely locking down the PCs to be used in a
summative CAA. This disables access from the PCs to other computer
resources such as the internet, chat systems, personal files, etc. This secure
lockdown must be used. The password and username is available from CTEL
and must not be given to any student.
ix. The usual procedures for internal moderation of an assessment need to
be followed, as well as those for moderation by the external examiner.
x. ITS need to be notified of the time and duration of any summative CAAs
two weeks in advance, so any network issues can be accounted for.
xi. The usual procedures for students who obtain support for their
assessments is available via the Student Enabling Centre. Special Needs
Tutors and Module leaders have direct access to the students
examination/assessment arrangements and this is available on e:Vision.
2. Writing the test
i. Tests should usually be no longer than 60 minutes.
ii. The questions should ideally appear on-screen in randomised mode – to foil
collusion. This may involve both a random sequence of presentation of the
questions and use of randomised answer options for each question.
iii. The decision on how to allocate the percentage mark for a test’s marks is
left to the module team. Module teams need to remember to mitigate against
guessing. Ways of doing this will be covered in the CAA training sessions, and
a simple way is to write questions with five choices..
iv. Questions should be cross-checked by at least one other member of the
module team before being entered into the software.
v. Where free text entry questions are used, question setters need to be
aware of all alternatives, misspellings, punctuation, capitals etc.
vi. Questions and the marking parameters also need to be tested once
having been entered into the server software by CTEL. This checking of the
test’s questions is a module team responsibility.
5
QA guidelines for use of Computer Assisted Assessment. November 2013
Last modified by Towers, Paul A.
vii. Best practice in the use of summative CAA advocates the use of banks of
questions. Module teams may wish to work towards the use of such banks.
CTEL can advise on how these might be built up.
3. Test day
i. The University guidelines for invigilation apply
http://www.wlv.ac.uk/pdf/Invigilators%20instructions%20May%202013.p
df
ii. The candidates’ identity needs to be checked as they enter the exam room.
iii. The software generates a unique test ID number and test password. These
two, together with the student’s own University password and username, are
used to access the test.
iv. The unique test ID number and test password need to be printed out in
advance and given to students in the exam room.
v. Staff responsible for invigilation need to be in the room around 30 / 45
minutes before the start of the exam.
vi. Staff responsible for invigilation need to switch on all PCs in advance of
students entering the room and log on to the Q-Mark server using the
ITS/CTEL supplied secure lockdown name and password. The Q-Mark
software will then be ready for the student to log on using their own ID and
password together with unique test ID number and test password.
vii. Staff responsible for invigilation need to assure themselves that they can
deal with minor problems such as monitor not being on, mouse cable coming
loose etc.
viii. Whenever possible staff responsible for invigilation need to assure
themselves that they can access the heating / cooling system for the rooms
which are likely to be warmer than usual exam rooms due to the PCs.
ix. Staff responsible for invigilation need to inform the candidates prior to the
exam starting:
 The time allowed for the exam.
 How many questions there are.
 How to get technical help.
 Remind them of the outline marking scheme, e.g., if negative marking
is being used.
 Rules regarding early finishes and leaving the exam room.
6
QA guidelines for use of Computer Assisted Assessment. November 2013
Last modified by Towers, Paul A.
x. In the event of network failure, a record needs to be kept by invigilators of
such system failures. The University exam procedures, as were there to be a
fire alarm, then need to be followed.
xi. In the event of an individual PC system failure, a record needs to be kept
by invigilators. The student should be moved to a free PC and asked to
continue where they left off.
xii. When rooming the exam it is best practice to have a number of free PCs
(1 per 20 students) to cater for breakdowns.
xiii. Module teams are responsible for identifying any students with enabling
needs which may impinge on their success with a computer based summative
assessment and make any necessary alternative arrangements for these
students. Student Enabling Centre needs to be consulted.
xiv. Candidates should be allocated PCs in advance. This should be done
randomly.
xv. The ratio of invigilators to students should follow the normal University
guidelines.
xvi. Depending on the layout of the PC lab and the number of candidates,
invigilators may prefer to use 1 candidate per 2 PCs.
xvii. Candidates need to print their name and sign their paper containing
their allocated user ID and password. These should be collected and kept by
the invigilator.
xviii. Training for invigilators who are externally appointed is available from
CTEL and recommended for those non-conversant in CAA.
4. Post-test implementation
i. The results of students’ performance on exams are available immediately on
completion through CTEL.
ii. The Q-Mark software also provides an analysis of performance on each
question and each item in each question. It is recommended that this be used
by module teams in their review of the validity and reliability. This report will
be supplied by CTEL.
iii. It is anticipated that staff will need to freshen up their questions year on
year. A replacement of around 15% is suggested. The choice of which
questions to replace needs to be guided by the detailed feedback of students’
performance on each question, and the test analysis generated by the QMARK
software.
7
QA guidelines for use of Computer Assisted Assessment. November 2013
Last modified by Towers, Paul A.
iv. Feedback on the test needs to be given to those students who request it.
However if staff wish to use the test again, then it is recommended that
feedback be given in general terms e.g., you did not perform well on the
questions about ox-bow lakes’ formation.
v. Candidates who are known to be dyslexic must have their responses
checked manually, if any of the questions involve free text entry.
For any further information on the use of Computer Assisted Assessment or
any of these guidelines please contact Paul Towers, eLearning advisor x 2628.
8