How to organise computer based
examinations in mass education?
Experiences and
guidelines
Dr. Jens Bücking
Nasjonal konferanse for automatiserte tester
29.-30.mai i Trondheim
1
Background at the University of Bremen
 12 faculties with 23.000 students and appr. 70.000
exams/semester, actually 5000 exams/semester in
business science only
 Increasing number of exams due to the shift to
BA/MA-studies
 Introduction of aptitude and placement tests
 But: Decreasing manpower within the faculties
Pressure to rationalize examination procedures
2
The eExamination Project
 Aim:
Creation of a campus-wide eLearning service for
computer based assessment
 Actions:
Investigation and evaluation of technical,
organisational and didactical implications of
computer based examinations
 Current status:
Pilot phase finished, now transfer of experiences
from business science to other faculties
3
eExaminations at the University of Bremen
(December 2004 – April 2006)
 14 examinations
(11 in business science, 3 in educational science)
 5669 registered students
(Ø 405 candidates per exam, range 47 – 962)
 Mean capacity was only 20 PCs / room (range 10-54)
Ø 24 repetitions/exam (range 5-64), up to:
7 rooms in parallel and 6 time slots/day
very high investment in organisation
4
21.02.05 08:00-10:00
10:30-12:30
13:00-15:00
15:30-17:30
18:00-20:00
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
23.02.05 08:00-10:00
10:30-12:30
13:00-15:00
15:30-17:30
●
●
●
●
24.02.05 15:00-16:30
●
GW2
●
●
●
●
●
●
●
●
WIWI
●
●
●
●
●
●
●
●
MZH3
●
●
●
●
●
●
●
●
●
MZH2
●
●
●
●
●
22.02.05 08:00-10:00
10:30-12:30
13:00-15:00
15:30-17:30
MZH1
●
●
●
●
●
BIBA2
18.02.05 09:00-10:30
11:00-12:30
13:00-14:30
15:00-16:30
17:00-18:30
18:00-19:30
BIBA1
●
●
●
●
Slot
MZH3
●
●
●
●
day
MZH2
●
●
●
●
GW2
MZH1
●
●
●
●
WIWI
BIBA2
●
●
●
●
Slot
16.02.05 08:00-10:00
10:30-12:30
13:00-15:00
15:30-17:30
day
BIBA1
Example of an e-examination schedule
●
●
●
●
●
●
●
●
●
●
●
●
2 courses
1500 candidates
100 repetitions
5
Why e-Examinations?
Advantages for students
 enhanced management of studies due to instant
feedback on examination result
 higher grade of objectivity
 elaborated feedback
 transparency and effective preparation for exams
…
6
Why e-Examinations?
Advantages for the university
 Rationalization of selection of students and of
placement tests
 Reduced costs
Support of
by by
an an
effective
andand
 Support
ofthe
theadministration
administration
effective
integrated management
of of
examinations
integrated
management
examinations
7
Pro argument: More efficient administration
Integration of e-examinations
eExamination
• Meyer
• Müller
• Lemke
• Schulze
• ---
Management
software
Students
candidates
approved yes/no
questions
candidates
results
room / time
exam results
Examination server
8
Why e-Examinations?
Advantages for teachers
 reduced
reducedwork
workload
load

 elaborated feedback on teaching success
 e-assessments (self-assessments, mock exams, final
exams) as part of the didactical concept in eLearning
 enhanced audit quality management
 re-usability of question catalogues
 collaborative production of catalogues
 ...
9
Pro argument: Reduced expenditure of human labour
Comparison of work load (1)
Exam parameters
1
2
3
4
Number of questions / catalogue
Production time/question (h)
Development of catalogue (h)
Reuses of catalogue
Mean Development time/season (h)
Preparation of exam (h)
Registrations
Places/room
Duration of single exam (h)
Repetitions needed
Observers/exam
Working hours of observers (h)
IT-staff needed during exam
Working hours of IT-staff (h)
Working hours for exam
Time need for scoring (min/question)
Time need for scoring (h/exam)
Mean time consumption (h)
Mean time consumption in Days
Written
eExam 1 eExam 2
60
0,5
30
1
30
8
400
200
2,0
2
2
8
0
0
8
0,5
200
240
1
240
3
80
32
400
20
2,0
20
2
80
1
40
120
0
0
240
1
240
3
80
32
400
120
2,0
4
2
16
1
8
24
0
0
246
232
136
31
29
17
days
35
31
30
29
25
20
17
15
10
5
0
Written
eExam 1
eExam 2
Excel-Experiment
10
Pro argument: Reduced expenditure of human labour
Comparison of work load (2)
40
35 factor: questions/exam
factor: number of students
50
Working days
Working days
60
40
30
20
10
scoring time = 0,5 min | 60 questions | duration 2 h
0
200
400
600
800
written test
eExam 1: 20 places/room
eExam 2: 120 places
working days per season =
catalogue dev. + prep. + exam operation + scoring
Reuse of catologue =
1 for written tests , 3 for eExams
20
15
10
5
0
scoring time = 0,5 min | 400 students | duration 2 h
0
1000
60
Working days
0
30
25
50
20
40
60
80
factor: scoring time/question
40
30
20
10
60 questions | 400 students | duration 2 h
0
0,0
0,2
0,4
0,6
0,8
1,0 min
11
Organisation of e-exams: Stakeholders and
tasks
•eLearn.Project management controlling of deadlines
Support
• Developing
blueprints of questions
Administration
of exam server
• Administration
of•examination
•
Preparation
of
PC-pools
(incl. syllabus
and scoring)
• Briefing
with
supervisors,
authors
(regulations,
bookings,
Supervisors • Quality control and •final
Handing
out
and
collection
of
inspection
and editors
schedules)exam
• Advice
and training of eExam
Author
papers
of catalogue
commissioners,
• Organising
authors,tests
editors
in PC• Organising
the subscription
of operation
• Instruction
of students
andPools
administration
students
• Authentification
• Organising
studentsofinspection
Technical
PC• Publication of •exam
data preparation
• Supervision
exam protocols
• Organisation
of pools
supervision
• Developing
templates
forofauthors
• Assistance
and trouble-shooting
•
Interface
to other IT-service units
• Sampling
of "exam blueprints
packages“
• Processing
the question
and to papers,
the software
for editor
supervisors
(exam
list company
with the
software
•checklists,
Task force
in case of emergency
of candidates,
• Organising
and performing
the local
during
the
operation
of examsEditor
protocols,quality
contingency
plan,
…)
and
webbased
check
Administration
• Archival
storagecatalogues
of exam data
• Updating
the
existing
eExam
(results,Commissioner
protocols, logfiles)
12
Support material provided by the ZMML
 Excel-sheet for costbenefit analysis
 Templates (question
types, excel-sheets, ... )
 Protocols and
troubleshooting-list
 Contingency-plans
 Checklists for all
stakeholders
Checklist for supervisors
1. Start
Login with user ....
Check function of mouse, keyboard and
screeen
Deactivate image icons, virus scan and
screen saver
Distribute exam paper and manual
2. Instruction
Exam regulation: .....
Explain login procedure
Technical hints
3. End
Log out
Activate alarm control
....
13
Organisation of e-exams: Work packages
WP1 Administration: rooms, scheduling, supervision
WP2 Organisation of registration
WP3 Catalogue development + quality check
WP4 Technical preparation + test of function
WP5 Examination
WP6 Postprocessing
14
Example WP3: Question catalogues
eLearning-service
eExam Commissioner
Editor
Author
Assignment of deadlines
Authoring
Training
Quality control
Development of templates
Upload to exam server
Production of blueprints
Final release
15
Example WP4: Preparation of exam
eLearning-service
Editor
eExam Commissioner
Author
Training
Administration of server
Briefing
Function tests
Supervisors
Administration
Prep. of candidates lists
Prep. of exam papers
Prep. of manual
Sampling of exam packages
16
Controlling of eExaminations
17
Take-home Messages
 Reduction of work load becomes true with
■
■
■
■
high number of candidates/exam (> 300)
sufficient number of PCs/room (> 50)
(otherwise) long duration for correction (≥ 30sec/question)
short exams (≤ 2h) with many questions (> 50)
 Go for a professional operated testcenter!
■ Dramatical reduction of working hours for supervision
■ Higher security and safety, autonomous LAN
■ Easier and faster workflows for subscription, scheduling and
technical preparation
■ Fascilitation of innovative examination procedures
18
Thanks for your attention!
www.zmml.uni-bremen.de/tagung
Congress: Computergestützte Prüfungen: Praxisbeispiele und Konzepte
(Computer based examinations: Examples and Strategies)
21.11.2005, ZMML - University of Bremen
19