a A University Strategy for Delivering Online Assessment Dr Richard Parsons Director of the Learning Centre University of Dundee r.parsons@dundee.ac.uk Learning Centre University of Dundee Richard Parsons 23 March, 2016 University of Dundee More than 10 years substantial Learning Centre University of Dundee experience of online assessments Blackboard clients – Academic Suite (virtually 100% modules) Delivering up to 3500 formative assessment sessions a day with QM Perception Delivering 50+ summative exams a year with QM Perception Range of online assessment activities recognised Richard Parsons 23 March, 2016 Blackboard Server Configuration Summer 2006 – Bb upgrade 6.3 to 7.1 on Unix, QM application from 3.4 to 4.1, Bb - QM bridge software (twice), QM repositories for assessments, Blackboard building block deployments and upgrades, Solaris and Oracle database upgrades, Annual module rollovers, Enhanced PDP programmes, Medical School curriculum map integration, University restructuring configuration changes Online Assessment System Assessment software design & deployment Central assessment support Learning systems integration Learning Centre University of Dundee Assessment hardware design & deployment Clear policy and procedures Staff and student education Richard Parsons 23 March, 2016 The Hardest Task Remains writing quality questions Staff time is precious Students are more critical than ever before Assessments to be aligned Questions need to be prepared ahead of time Quality runs to feedback, distracters, variety, depth, currency Authoring Q within the software is secondary Learning Centre University of Dundee Richard Parsons 23 March, 2016 Staff and Student Education For 4 years at Dundee we have run an Online Assessment course, online Covers technical and pedagogical aspects 5 weeks long, 3 to 6 hours a week. Formally accredited by our Education Faculty (10 SC) Building a bank of staff skills Most students need little guidance Learning Centre University of Dundee Richard Parsons 23 March, 2016 Examples of Assessment Strategies 15 Q from pool of 100, max mark taken (Genetics) Take assessment first, then repeat in an open fashion. Mark taken from first run. (Medicine) 10 numerical questions, each from a pool of 50 randomly generated Q (Life Sciences) Student revision – students invited to compose 2 Q each with FB, all compiled automatically, checked & made available for open assessment (Env Sci) Open assessment, one submission only (Law) Summative (range of departments) Learning Centre University of Dundee Richard Parsons 23 March, 2016 Quality Hardware and Systems Central servers built & maintained by ITS LC involved in system design Three servers, one formative, one summative, one shared repository HP BL20p Blades Dual Pentium 3400 MHz 6GB RAM, dual NIC backplane. Databases Formative on Oracle Unix server Summative on Oracle Unix server Shared on Oracle Unix server Learning Centre University of Dundee Richard Parsons 23 March, 2016 University of Dundee Questionmark Perception V4 Systems Model Participant Oracle 10G Oracle 10G Participant Participant QMP V4.2 CAA-01 QMP V4.2 CAA-02 Production Exam Repository Production Formative Repository LC Administrator Participant QM / Bb integration QMP V4 Authoring Manager Oracle 10G Oracle 10G Participant QMP V4.2 CAA-03 Local Author (Staff) Development Repository Blackboard 7.1 My Dundee Participant QMP V4 Authoring Manager Local Author (Staff) QMP V4 Authoring Manager Local Author (Staff) David Walker, University of Dundee Participant Clear Policy and Procedures Central online assessment system required clear guidance Prepared draft Policy and Procedures Generally follows BS7988:2002 Developed for about a year, inc. consultation Passed at University Senate Excellent for ensuring quality Runs to 17 pages Additional documentation for Online Assessment system administrators Learning Centre University of Dundee Richard Parsons 23 March, 2016 Summative Procedures Assessment must be tested Assessment loaded to both servers, scheduled on one Central staff attend examinations Use special ”exam” accounts Use Perception Secure Browser Candidates complete Examination Attendance form Immediately after the exam a backup of results is taken Use two way radios or mobiles No system is fullproof, 99% success target ? Learning Centre University of Dundee Richard Parsons 23 March, 2016 Self & Peer Assessment Bb Building Block Assessments are added and edited by the instructors in any course One or more questions, each with one or more marking criteria Marking criteria as discrete values or range Feedback provided as score and text if required Students automatically allocated to mark peers (1 to many) Peer assessments are assigned automatically Self assessment is an option Marks and reports are visible to the Instructor All marks can be downloaded for analysis in excel Moderated results automatically passed to Bb Gradebook Exercise text is available for download and online plagiarism checks Questions and criteria are stored for reuse and exchange Learning Centre University of Dundee Richard Parsons 23 March, 2016 The Dundee My PDP resources Online resource to support PDP; timetabled induction sessions; scope for faculty tailoring. Available for free under license. Each student can prepare an online portfolio based on templates, culminating in prioritised action points and an enhanced CV. Portfolios can be shared online. Learning Centre University of Dundee Richard Parsons 23 March, 2016 Mobile Campus Map Java game-based map Rich links with key campus information Single download Popular with students www.dundee.ac.uk/mobilemap Learning Centre University of Dundee Richard Parsons 23 March, 2016 Additional Online Assessment Self & Peer Framework for plagiarism advice and management Procedures for online submission Discussion board assessments ePortfolio assessment rubrics and tools Classroom/personal response systems Business intelligence activities Overall framework of OA support at Dundee Learning Centre University of Dundee Richard Parsons 23 March, 2016 Individual concerns within an online assessment world eLearning Manager Are the staffing levels correct? Is our software choice appropriate? Are the server specifications correct to cope with the expected load? Are staff education procedures in place? IT Server Administrator Is the server fully patched, reliable and secure? Is the OA software functioning correctly and reliably Do we have monitoring and is it working? Are the logging and backup procedures working correctly? College Officer Who can electronically author my questions? Can we use textbook questions? Is this assessment driving learning? Can we book an IT suite for 240 students on Wednesday the 25th? Two students have additional requirements. Will the results transfer directly to the SRS? Lecturer Are my questions too simple? Is this technology reliable? Do the students find it tedious? Where can we obtain challenging questions? Are these assessments aligned with our learning outcomes? IT Desktop Administrator Does the desktop software operate correctly? Are the “exam accounts” secure and available? Are the PCs well maintained and reliable? Traditional Lecturer Student This is nice and private I am getting real feedback here Do I know the answers? Can I learn from this? I am at home in this online environment Distance Student Timetabling Officer Can the proposed exams be fitted into the best rooms? Are assessments receiving appropriate prioritisation? Is my technology compatible? Is my connection reliable? Is this a replacement for F2F? Online Assessment Officer Are all assessments up-to-date and correctly available? Is the server-client functionality all compatible? Will the next examination run smoothly? Are the staff education programmes ready? Examinations Officer Are these results 100% reliable? secure? Do our procedures represent the best possible practice? Richard Parsons, University of Dundee Procedural Elements Actors OA policy and procedures ZZ + AA Student and staff training Examination practices XX + WW College staff Web guidance + VV University student desktop Question creation Revisit annually? Academic staff + XX + WW Room booking procedures Student personal PCs BB + CC Academic staff + XX + WW XX + WW DD + EE OA application software & support Technical database considerations Hardware & infrastructure XX + WW 13 specialist staff + academics & college staff FF + GG Operating system & web server Technical The entire operation has limited inertia, limited history, sustainable staffing, space, budget. Represents a framework Question/assessment repositories Network Each addressed by a tasked individual(s) within a supported team, and each element able to cope with growth and changes XX + HH HH + II Richard Parsons, University of Dundee The Annual Cycle of Online Assessment Sept. Testing & tuning Modules begin Online resit exams Annual framework meeting Summer upgrade window Online exams June Dec. Mid semester window Online exams •Growth may occur each year OA staff education module Richard Parsons, University of Dundee March •Changes are made to the systems •A framework allows the process to be sustained Summary Dundee has an institutional online assessment & some elements of a strategic framework Software, hardware, P&P, staffing, integration and education. 13 specialist staff. Secure and reliable, but straightforward to use for students and staff Let the academics create the Qs to stimulate learning Costs are neither insignificant or astronomical Learning Centre University of Dundee Richard Parsons 23 March, 2016 Learning Centre University of Dundee Richard Parsons 23 March, 2016