Canadian Medical School Exam Survey Report
(Completion rate: 100.0%)
1. Please identify your school.
#
Response
1.
University of Manitoba
2.
UBC Faculty of Medicine
3.
UofA
4.
Université de Sherbrooke
5.
University of Calgary
6.
Queen's University
7.
University of Ottawa
8.
Dalhousie
9.
Memorial University, Faculty of Medicine
2. Please indicate the number of UGMY1 medical students at your school.
#
Response
1.
110
2.
289
3.
Yr1=162; 167 in each of year 2, 3, and yr 4=181
4.
215
5.
511
6.
400
7.
165
8.
110
9.
81
3. Please indicate how your medical school delivers examinations. (Select all that apply)
Response
Chart
Percentage
Count
Paper
78%
7
Desktop in a lab
44%
4
Laptop (student owned)
22%
2
Laptop (school owned)
0%
0
Tablet (student owned)
11%
1
Tablet (school owned)
0%
0
Total Responses
9
3a. Please explain
#
Response
1.
We are currently in the process of evaluating the delivery of high stakes exams to students utilizing the
QuestionMark Perception Evaluations System. Currently low to moderate stakes exams and tests/quizzes are
delivered via our Curriculum Management System (OPAL) which leverages QuestionMark.
2.
For paper based exam we generate a paper copy of the exam using our exam bank-IDEAL. The OMR bubbles are
created seperately for each exam. The buubles are scanned using optical scanning software- REMARK. Final
scoring and data analysis is done using SPSS. For online delivery, we set up the exam in IDEAL and deliver it in
our computer lab.
3.
Most exams (nearly all) in our program are computer based from years 1-4. We have a 90 station lab that we use
for clerkship, and a 240 station (divided by 120/60/60-public lab) for year one and two exams.
4.
All examinations are paper delivered except, evidently the practical form (OSCE) with standardized patient
5.
A couple years ago - we purchased laptops in hope to do on line exams. Are plan was to use these laptops in our
theatre using wireless communication. Even though we increased the wireless communication ports - we ran into
hugh problems with delay in exam transmission. In some cases a three hour normal exam ran to 5 hours. We also
had students complaining of image quality of the slides - which lead to an appeal. It was then decided since we do
not have a computer lab - that we revert back to paper.
6.
We currently still do our exams on paper however we do a formative online exams.
7.
we used to use electronic and stopped because of instability of software and lack of secure exam site.
8.
We have a computer classroom of 80+ computers that we run online secure high stakes exams on.
4. Do you use a computer lab for exams?
Response
Chart
Percentage
Count
Yes
67%
4
No
33%
2
Total Responses
6
4a. If yes, how many computer labs do you have access to for exams?
#
Response
1.
2
2.
6
3.
We primarily use two labs;
4.
80 plus extras
4b. If yes, please indicate the number of seats available in computer labs for exams.
#
Response
1.
30
2.
120 at the main campus
3.
90 stn; 240 stn
4.
80 plus extra seats if necessary
5. Which software do you use for exam delivery?
#
Response
1.
QuestionMark Perception, MCC proprietary software for MCCQE.
2.
IDEAL
3.
Internally developed software called Qwogo
4.
ExamSoft
5.
For formative online exams we use Questionmark.
6.
Questionmark
6. Please explain what your key drivers/priorities/requirements were in deciding which
software to use for exam delivery.
#
Response
1.
Integration with Curriculum Management System, Analytics, Reporting, Security.
2.
This is an outdated software which is no longer upgraded. We are in the process of finalizing a new software for
exam banking and delivery.
3.
Internally developed. However, it lacks item creation capability; item banking capability; and item blueprinting
capability. It only delivers a computer based exam.
4.
eliminate any human error and to minimized the time for marking following examinations
5.
Secure question bank system. Powerful analytics. Online secure exam delivery. Shuffled answers. Locally hosted
not a cloud-based solution.
7. Please indicate the format of electronically-delivered exams:
Response
Chart
Percentage
Count
Completely online
83%
5
Paper answer sheets
17%
1
Total Responses
6
7a. Please provide more details about the format of electronically-delievered exams
#
Response
1.
As noted above.
2.
please see the previous comments in 3a
3.
Mostly MCQ with some images. No audio or video (though did test this, which placed too much strain on the
network.)
4.
Questions are entered into the Questionmark system and tagged appropriately. Then exams are created from those
questions and delivered in the computer room.
8. How are electronic exams governed within your school?
Response
Chart
Percentage
Count
Centrally governed
100%
5
Ungoverned - course instructors use what
they wish
0%
0
Other
0%
0
Total Responses
5
8. How are electronic exams governed within your school? (Other)
#
Response
8a. Please provide more details about the governance of electronic exams at your school.
#
Response
1.
The electronc exams are set up and anlyzed centrally at the main campus (Educational Assessment Unit, UBC).
However it is delivered to various sites all over the province at the same time.
2.
managed by UME office, assessment person. Looking for an assistant dean, assessment and evaluation for the
future.
3.
All assessments are done through the Undergraduate office in conjunction with the elearning team at Medicine.
9. What additional measures or software do you use to set up an exam (e.g., to prevent
cheating, etc.)?
#
Response
1.
Secure Browser (Lockdown)
2.
restricted access to exam during a specified time period with lack down internet access.
3.
all examination questions and randomized and the distractors are scrambled.
10. Are there any restrictions to the operating systems that you require for exams?
Response
Chart
Percentage
Count
Yes
67%
2
No
33%
1
Not Sure
0%
0
Total Responses
3
10a. Please explain
#
Response
1.
We have adopted a device agnostic architecture in order to allow for user driven engagement and experience.
2.
The IDEAl software only runs on a WIndows XP platform and need activeX.
3.
must run Firefox or Google Chrome - does not work on Exploer
11. Which operating systems do you support?
Response
Chart
Percentage
Count
Windows only
33%
1
Macs only
0%
0
Mixed (PCs and Macs)
33%
1
Mixed with conditions
0%
0
Other
33%
1
Total Responses
11. Which operating systems do you support? (Mixed with conditions)
#
Response
11. Which operating systems do you support? (Other)
#
Response
1.
Mobile and Tablets
3
12. Please indicate the type of question banks used at your school.
Response
Chart
Percentage
Count
Local databases
67%
6
Purchased software
22%
2
Other
33%
3
Total Responses
9
12. Please indicate the type of question banks used at your school. (Purchased software)
#
Response
1.
IDEAL
2.
questionmark
questions are created by faculty
12. Please indicate the type of question banks used at your school. (Other)
#
Response
1.
Internally developed, dated bank that needs work
2.
Excell
3.
questionmark
questions are created by faculty
11a. Please provide any other information about the question banks used at your school.
#
Response
1.
In the process of significant upgrading to support renewed curriculum.
2.
Own developed
3.
We are currently looking to either purchase and build a curriculum module that would manage exam questions.
12. In exams, besides text, do you include? (Select all that apply)
Response
Chart
Percentage
Count
Images
100%
8
Virtual slides
12%
1
Cases
62%
5
Other
12%
1
Total Responses
12. In exams, besides text, do you include? (Select all that apply) (Other)
#
Response
1.
Power Point slides
8
13. Please indicate the types of questions used in your exams
Response
Chart
Percentage
Count
Multiple choice
100%
8
Short answer
75%
6
Labelling
25%
2
Other
25%
2
Total Responses
8
13. Please indicate the types of questions used in your exams (Other)
#
Response
1.
Matching
2.
fill in blank true/false
14. How many FTE positions do you have to handle electronic based exams?
#
Response
1.
2.0
2.
1 per academic year as well as exam coordination staff
3.
2
4.
approximately 3 FTE's
5.
2
15. Please provide any other information about the exams used at your Medical School.
#
Response
1.
Would be nice to use MCC MCQ and CDM software for our computer based exams - though adaptive testing
would add complexity and minimum data bank size.
2.
We are thinking of implementing online exams in the near future.
3.
We launched Questionmark just this past September. So far it has been successful. While not perfect (still
working on improving workflow) the system functions as we hoped.