a
A University
Strategy for
Delivering Online
Assessment
Dr Richard Parsons
Director of the Learning Centre
University of Dundee
r.parsons@dundee.ac.uk
Learning Centre
University of Dundee
Richard Parsons 23 March, 2016
University of Dundee
 More than 10 years substantial




Learning Centre
University of Dundee
experience of online assessments
Blackboard clients – Academic
Suite (virtually 100% modules)
Delivering up to 3500 formative
assessment sessions a day with
QM Perception
Delivering 50+ summative exams a
year with QM Perception
Range of online assessment
activities recognised
Richard Parsons 23 March, 2016
Blackboard Server Configuration
Summer 2006 –
Bb upgrade 6.3 to 7.1 on Unix,
QM application from 3.4 to 4.1,
Bb - QM bridge software (twice),
QM repositories for assessments,
Blackboard building block deployments
and upgrades,
Solaris and Oracle database upgrades,
Annual module rollovers,
Enhanced PDP programmes,
Medical School curriculum map
integration,
University restructuring configuration
changes
Online Assessment System
Assessment
software
design &
deployment
Central
assessment
support
Learning
systems
integration
Learning Centre
University of Dundee
Assessment
hardware
design &
deployment
Clear policy
and
procedures
Staff and
student
education
Richard Parsons 23 March, 2016
The Hardest Task
Remains writing quality questions
 Staff time is precious
 Students are more critical than ever before
 Assessments to be aligned
 Questions need to be prepared ahead of
time
 Quality runs to feedback, distracters, variety,
depth, currency
 Authoring Q within the software is secondary
Learning Centre
University of Dundee
Richard Parsons 23 March, 2016
Staff and Student Education
 For 4 years at Dundee we





have run an Online
Assessment course, online
Covers technical and
pedagogical aspects
5 weeks long, 3 to 6 hours a
week.
Formally accredited by our
Education Faculty (10 SC)
Building a bank of staff skills
Most students need little
guidance
Learning Centre
University of Dundee
Richard Parsons 23 March, 2016
Examples of Assessment Strategies
 15 Q from pool of 100, max mark taken (Genetics)
 Take assessment first, then repeat in an open fashion.
Mark taken from first run. (Medicine)
 10 numerical questions, each from a pool of 50
randomly generated Q (Life Sciences)
 Student revision – students invited to compose 2 Q
each with FB, all compiled automatically, checked &
made available for open assessment (Env Sci)
 Open assessment, one submission only (Law)
 Summative (range of departments)
Learning Centre
University of Dundee
Richard Parsons 23 March, 2016
Quality Hardware and Systems
 Central servers built & maintained by ITS
 LC involved in system design
 Three servers, one formative, one summative,
one shared repository
 HP BL20p Blades Dual Pentium 3400 MHz 6GB
RAM, dual NIC backplane.
 Databases
 Formative on Oracle Unix server
 Summative on Oracle Unix server
 Shared on Oracle Unix server
Learning Centre
University of Dundee
Richard Parsons 23 March, 2016
University of Dundee
Questionmark Perception V4 Systems
Model
Participant
Oracle 10G
Oracle 10G
Participant
Participant
QMP V4.2
CAA-01
QMP V4.2
CAA-02
Production Exam
Repository
Production
Formative
Repository
LC Administrator
Participant
QM / Bb
integration
QMP V4
Authoring
Manager
Oracle 10G
Oracle 10G
Participant
QMP V4.2
CAA-03
Local Author (Staff)
Development
Repository
Blackboard
7.1
My Dundee
Participant
QMP V4
Authoring
Manager
Local Author (Staff)
QMP V4
Authoring
Manager
Local Author (Staff)
David Walker, University of Dundee
Participant
Clear Policy and Procedures
 Central online assessment system required clear







guidance
Prepared draft Policy and Procedures
Generally follows BS7988:2002
Developed for about a year, inc. consultation
Passed at University Senate
Excellent for ensuring quality
Runs to 17 pages
Additional documentation for Online Assessment
system administrators
Learning Centre
University of Dundee
Richard Parsons 23 March, 2016
Summative Procedures
 Assessment must be tested
 Assessment loaded to both servers,







scheduled on one
Central staff attend examinations
Use special ”exam” accounts
Use Perception Secure Browser
Candidates complete Examination
Attendance form
Immediately after the exam a backup of
results is taken
Use two way radios or mobiles
No system is fullproof, 99% success
target ?
Learning Centre
University of Dundee
Richard Parsons 23 March, 2016
Self & Peer Assessment Bb Building Block
Assessments are added and edited by the instructors in any course
One or more questions, each with one or more marking criteria
Marking criteria as discrete values or range
Feedback provided as score and text if required
Students automatically allocated to mark peers (1 to many)
Peer assessments are assigned automatically
Self assessment is an option
Marks and reports are visible to the Instructor
All marks can be downloaded for analysis in excel
Moderated results automatically passed to Bb Gradebook
Exercise text is available for download and online plagiarism
checks
 Questions and criteria are stored for reuse and exchange











Learning Centre
University of Dundee
Richard Parsons 23 March, 2016
The Dundee My PDP resources
Online resource to support PDP; timetabled induction sessions; scope
for faculty tailoring. Available for free under license.
Each student can prepare an online portfolio based on templates,
culminating in prioritised action points and an enhanced CV. Portfolios
can be shared online.
Learning Centre
University of Dundee
Richard Parsons 23 March, 2016
Mobile Campus Map
 Java game-based map
 Rich links with key campus
information
 Single download
 Popular with students
www.dundee.ac.uk/mobilemap
Learning Centre
University of Dundee
Richard Parsons 23 March, 2016
Additional Online Assessment
 Self & Peer
 Framework for plagiarism advice and






management
Procedures for online submission
Discussion board assessments
ePortfolio assessment rubrics and tools
Classroom/personal response systems
Business intelligence activities
Overall framework of OA support at Dundee
Learning Centre
University of Dundee
Richard Parsons 23 March, 2016
Individual concerns within an online assessment world
eLearning Manager
Are the staffing levels correct?
Is our software choice
appropriate?
Are the server specifications
correct to cope with the
expected load?
Are staff education procedures
in place?
IT Server Administrator
Is the server fully patched,
reliable and secure?
Is the OA software functioning
correctly and reliably
Do we have monitoring and is it
working?
Are the logging and backup
procedures working correctly?
College Officer
Who can electronically author
my questions?
Can we use textbook questions?
Is this assessment driving
learning?
Can we book an IT suite for 240
students on Wednesday the
25th?
Two students have additional
requirements.
Will the results transfer directly
to the SRS?
Lecturer
Are my questions too simple?
Is this technology reliable?
Do the students find it tedious?
Where can we obtain challenging
questions?
Are these assessments aligned with
our learning outcomes?
IT Desktop Administrator
Does the desktop software
operate correctly?
Are the “exam accounts” secure
and available?
Are the PCs well maintained and
reliable?
Traditional Lecturer
Student
This is nice and private
I am getting real feedback here
Do I know the answers?
Can I learn from this?
I am at home in this online
environment
Distance Student
Timetabling Officer
Can the proposed exams be
fitted into the best rooms?
Are assessments receiving
appropriate prioritisation?
Is my technology compatible?
Is my connection reliable?
Is this a replacement for F2F?
Online Assessment Officer
Are all assessments up-to-date
and correctly available?
Is the server-client functionality
all compatible?
Will the next examination run
smoothly?
Are the staff education
programmes ready?
Examinations Officer
Are these results 100% reliable?
secure?
Do our procedures represent the
best possible practice?
Richard Parsons, University of Dundee
Procedural
Elements
Actors
OA policy and procedures
ZZ + AA
Student and staff training
Examination practices
XX + WW
College staff
Web guidance + VV
University student desktop
Question creation
Revisit annually?
Academic staff + XX + WW
Room booking procedures
Student personal PCs
BB + CC
Academic staff + XX + WW
XX + WW
DD + EE
OA application software & support
Technical database considerations
Hardware & infrastructure
XX + WW
13 specialist
staff +
academics &
college staff
FF + GG
Operating system & web server
Technical
The entire operation has
limited inertia, limited
history, sustainable
staffing, space, budget.
Represents a framework
Question/assessment repositories
Network
Each addressed by a
tasked individual(s) within
a supported team, and
each element able to cope
with growth and changes
XX + HH
HH + II
Richard Parsons, University of Dundee
The Annual Cycle of Online Assessment
Sept.
Testing &
tuning
Modules
begin
Online resit
exams
Annual
framework
meeting
Summer
upgrade
window
Online exams
June
Dec.
Mid
semester
window
Online
exams
•Growth may occur each year
OA staff
education
module
Richard Parsons, University of Dundee
March
•Changes are made to the
systems
•A framework allows the
process to be sustained
Summary
 Dundee has an institutional online assessment &




some elements of a strategic framework
Software, hardware, P&P, staffing, integration
and education. 13 specialist staff.
Secure and reliable, but straightforward to use
for students and staff
Let the academics create the Qs to stimulate
learning
Costs are neither insignificant or astronomical
Learning Centre
University of Dundee
Richard Parsons 23 March, 2016
Learning Centre
University of Dundee
Richard Parsons 23 March, 2016