DEVELOPMENT OF PEER COLLABORATION MODELS FOR WRITING FORMATIVE
EXAMINATION QUESTIONS TO AID ASSESSMENT AND FEEDBACK.
APPLICATION TO LEARNING AND TEACHING DEVELOPMENT FUND - APRIL 2012
Joanne Burke1, (joanne.burke@glasgow.ac.uk), Donald Cameron2, Vivian Binnie2, Lesley Nicolson3, Jennifer
Hammond3, Carol Ditchfield1, Margaret-Ann Flynn1, Max Field1, John Hamer4, Helen Purchase4 and
Amanda Sykes5 (Medicine1, Dentistry2, Veterinary3, Computing Science4, Student Learning Service5)
Summary:
This application builds on a previous LTDF project to support and develop a computer based platform to enable
students to write, review and practice two different types of examination questions.
The National Student Survey has highlighted the need to improve formative assessment and feedback in support
of student learning. In addition to teacher generated feedback, one way to enhance student feedback is to engage
students in this process – to generate feedback for themselves and their peers.
PeerWise is a web-based software programme that allows students to write their own multiple choice questions,
and answer and critique those set by others. Students are required to provide an explanation with their answer, and
hence they extend their knowledge base when writing a question and gain immediate feedback when their peers
answer these questions. This project aims to extend the use of PeerWise to year 1-3 students in Medicine,
Veterinary Medicine and Dentistry. In order to monitor the effect on student learning, the project will interview
students and look at the correlation between usage and exam performance.
The second aspect of this project is to develop software to accept questions called Situational Judgement Tests
(SJTs). These are specialised types of questions designed to assess professional attitudes in all three disciplines,
covering key graduate attributes developed during undergraduate training. SJTs are now also used as part of
selection tests for medical students when entering their first postgraduate training posts. This project aims to work
with the PeerWise developer to pilot the use of new SJTWise software. Students will use SJTWise to write, review
and practice SJT questions and to obtain feedback using similar principles to PeerWise. Student views will be
obtained and usage data will be correlated with clinical exam performance to analyse potential enhancement to
learning.
1. Aims and outcomes during the next two years will be to:
a) extend use of PeerWise to engage students in years 1-3 in all three disciplines in MCQ writing for formative
assessment. After the next two years there will be data from 1,500 medical, 540 dental and 840 veterinary students.
b) assess the effectiveness of PeerWise, by comparing written examination marks for students who write questions
and those who simply answer the MCQs. Further qualitative data will be obtained from focus groups from the
different year groups.
c) build on the experience of students generating SJT questions, and integrate SJT writing sessions into medical,
veterinary and dental courses to build question banks in each discipline for future use and to explore potential areas
of overlap between disciplines.
d) develop the software prototype for SJT questions to produce a more sophisticated programme (to be named
SJTWise) that will provide 500 medical 180 dental and 280 veterinary students with opportunities to write, answer
and evaluate SJT questions and use this process as a novel learning opportunity.
e) assess the effectiveness of SJTWise by comparing marks for those who write questions and those who just
answer questions in their OSCE examinations. This will involve reviewing questions which assess professionalism
issues as compared to questions assessing clinical skills. Qualitative evaluation will be obtained using focus groups
and questionnaires.
2. Background and previous work from LTDF bid 2011:
a) Need for the Project. This project addresses two University challenges. Firstly, National Students Surveys
highlighted limited formative assessment and a need for improved feedback in all Colleges including Medical,
Veterinary and Dental schools. Secondly, UK regulatory bodies (General Medical and Dental Councils and to the
Royal College of Veterinary Surgeons) insist that Universities teach and assess professionalism and other graduate
attributes during student training. This bid will build on and utilise students’ experience to address both challenges,
to develop new technology to help students with feedback and formative assessment, and continue to support staff
by sharing existing experience and expertise within the schools as these developments take place.
i. Both formative assessment and effective feedback are known to be powerful influences on student
learning (1), and engaging students in the process is a key component of effective feedback (2,3). By
developing their own questions, students can enhance their personal learning during course programmes, and,
by using additional IT support that provides formative assessment with feedback from their peers, students in
all three schools can build knowledge, critical thinking and personal confidence.
ii. The L&T strategy indicates a need to address Glasgow graduate attributes, including professionalism to
develop portfolios of generic skills that, over and above basic knowledge, can help students generate
successful job applications. Assessing graduate attributes is a complex area and often requires staff to assess a
student portfolio including presentations, essays, written reports and simulation exercises (4). These can be
difficult to assess objectively and time consuming to complete. New IT-based techniques can address the
issue of consistency in assessing graduate attributes, but students have limited experience in use of these
technologies, and this is an area that should be improved.
b) Multiple Choice Questions (MCQs) are integral to Dental, Medical and Veterinary assessment. During our
present LTDF project, students in each school wrote MCQ’s and shared these with their peers using the PeerWise
on line repository (5). This software (developed by Paul Denny and John Hamer) has been used in several
Universities to allow students to submit questions with explanations and evidence based answers on line (6).
Questions are peer-rated, subject to student-led discussion in a forum, and provide students completing the question
with immediate feedback about their performance in comparison to their peer group. In so doing, PeerWise allows
students to build up a valuable question bank for self-directed formative assessment. To our knowledge this is the
first time that PeerWise has been used in these disciplines and evaluated to this extent.
PeerWise: With funds from LTDF in 2011, PeerWise was introduced to year-1 students in all three subjects with
different models for each discipline. Participating students were given information about PeerWise and reminded by
staff to make use of PeerWise. Usage up to February 2012 is summarised in Table 1 and is as follows:1) Year 1 and 2 Veterinary students were expected to write eight and answer twenty questions (5), which
contributed to 5% of annual summative assessment. In year 1, compliance was >99%. PeerWise use peaked prior to
each summative assessment, and in feedback from students about all course elements (lectures etc.) students scored
PeerWise second highest. Several questions submitted to PeerWise have been used in summative assessment.
2) Year 1 dental students were required to write two and answer ten questions as part of a course assignment, but
received no reward for participating (7). Compliance was >99%. Usage peaked before examinations, and informal
feedback indicated no concerns about using PeerWise in this way. Year 2 dental students heard about PeerWise use
in Year 1 and expressed an interest to use PeerWise as well.
3) Year 1 Medical students engaged with PeerWise on a voluntary basis. About 80% used PeerWise, predominantly
by answering questions just prior to examinations. Focus group data showed that students perceived that practice in
using MCQs was helpful, that knowledge was required to write questions, and praised the variety of questions in a
single resource. Students were concerned about lack of quality assurance, that some question authors did not include
full explanations for the answers, and that each question could only be attempted only once.
Table 1. Details of PeerWise usage by students in each School
School / total number
Number of students
Number of students
of students in year 1
Writing/authoring questionsanswering questions
Veterinary Medicine (147)
Dentistry (95)
Medicine (273)
146
95
45
147
94
217
Total number of questions
answered by students in
year 1
19,766
13,342
16,632
c) Situational Judgement Tests: Learning about professionalism is integral to the University graduate attributes,
and assessing professionalism is required by the General Medical and Dental Councils and the Royal College of
Veterinary Surgeons (4, 8). SJTs can be set which use clinical scenarios to assess these graduate attributes and SJTs
are already used in postgraduate selection for General Practitioners. SJTs now make up 50% of marks in final year
assessments before medical students apply for their foundation year (FY) posts after graduation. Veterinary and
Dental curriculum developers are reported to be considering their use. SJTs consist of questions and answers where
students either rank five appropriate actions in response to a clinical scenario, or select three most appropriate
actions from eight possible correct answers (8). By assessing graduate attributes using clinical scenarios, SJTs
assess students’ reflective learning skills at the highest level of Bloom’s Taxonomy, and are readily adaptable for
marking by computer. In the recent pilot study using SJTs before their applications for FY posts in 2012, 50% of
University of Glasgow final year medical students fell into the bottom third of the national average, a response to be
improved upon in our student group. If a tool akin to PeerWise was available, Glasgow students could gain valuable
personal experience in writing SJTs and using these for practice purposes, while at the same time developing a bank
of practice questions for future use.
As part of the 2011 LTDF bid, a project was set up for two Year-4 medical students, aiming to assess whether
students could write effective SJTs. With guidance, they wrote twenty SJTs based on the ranking five-option
format (9), which were subject to revision following review by Faculty staff. Over 160 (out of 253) Year-4 students
volunteered to sit a formative assessment based on these questions. Marking the questions by hand took the students
28 hours, with the distribution of marks being similar to those published for UK medical students (9). The 163
students attended predominantly for practice purposes, with only 14% perceiving themselves as prepared for SJT
tests prior to formative assessment — a figure that rose to >60% after the test. More than half the students reported
seeing similar scenarios during their training, indicating potential experience that can be tapped for writing SJTs.
The majority said they would purchase a book on SJTs, indicating a perceived need for practice questions. Over
40% expressed an interest in writing SJTs, indicating that a substantial number of students could be involved in
building question banks for the future.
3. Student Engagement: We believe that students are an underused resource within the University, and have
shown that students can use Peer Assisted Learning to play a more active role in their own learning and assessment
(10,11). Student use of MCQs in PeerWise and in writing SJTs has demonstrated the need for, and the value of,
these novel educational tools.
PeerWise: In summary, preliminary studies have seen over 370 Year-1 students engage with PeerWise for over six
months. Usage shows that the majority have used the system extensively, and that students perceive several benefits
to use of PeerWise, Students have also identified ways to improve PeerWise for their benefit as outlined above.
Situational Judgement Tests: In the national pilot study of final-year medical students, Glasgow students
performed poorly in SJT tests in comparison to the national average, and there is a desire for additional practice in
SJT use. Evaluation of our student project show that 4th year students have had experience of the clinical scenarios
used in SJTs and can write these questions (possibly with some Faculty support) to the benefit of their peers. More
importantly, students have the desire to write SJTs to help develop a question bank
Student Inclusion in this application: We propose to continue to involve students in writing, answering and
evaluation of MCQs using PeerWise. In addition, students from Medicine, Dentistry and Veterinary schools will
write SJTs for their peer group, provide an evidence base for their answers and, once SJTWise is further developed
as an IT resource, will make use of this system as a question bank. In 2013 and 2014, we propose to offer two
summer student scholarships to assist with evaluation of PeerWise and SJTWise, and to establish areas for
improvement in the future.
4. Methods
This two year project consists of four phases outlined in Table 2:
a) Student Participation in and development of PeerWise. We will extend provision of PeerWise to the next two
consecutive Year-1 undergraduate groups so that the students in the first three years in each school will have access
to PeerWise. After the next two years, 1,500 medical, 360 dental and 840 veterinary students will be given the
opportunity to generate, answer and evaluate MCQs for formative assessment. Early experience has identified areas
in PeerWise that students perceive need development which will be put in place; e.g. allowing students access to
one question more than once to help revision. Use of PeerWise has been higher than expected, but by continuing to
adopt the different models in all three schools we will establish which is the most appropriate. PeerWise could be
compulsory with marks awarded for participation (veterinary students), compulsory as part of an assignment with
no marks for participation (dental students), or completely voluntary (medical students). Data from three year
groups will provide large student numbers to enable comparison of PeerWise use and student examination results.
b) Situational Judgement Tests and development of SJTWise. PeerWise cannot be applied to SJTs, but by the
end of the 2011 LTDF project an SJTWise prototype will be available, allowing students to submit SJT questions on
line in the ‘rank five’ option format, and summer students will have evaluated this prototype. In this application we
will focus on:
 supporting the creation of SJTWise for both formats of SJTs, allowing individuals and student groups to submit
questions and to write a justification of their answers
 generating a forum for every SJT submitted to allow students to focus on engaging in an on line debate that
resolves into a collective decision about the final version of each SJT
 providing a facility within SJTWise to enable Faculty to review relevant statistical data about administrative
issues (e.g. details of student use, SJT writing and answering questions, time of use etc)
 providing a facility in SJTWise for review of SJTs by Faculty staff for quality assurance.
This will enable students to write, evaluate and modify their questions (in both SJT formats) in light of student
centred discussion and for Faculty to administer the use of SJTWise as with PeerWise
c) SJTWise in curriculum development. By the end of the previous LTDF project (Table 2), the SJTWise
prototype for ‘rank five’ option questions will be in place, so that all students will have access to this SJT format.
from September 2012. As with PeerWise, each school is likely to adopt a different approach for course integration
during the project, but because writing SJTs is recognised as a complex process, students have requested a level of
Faculty quality assurance which will be an essential part of development of SJTWise in each school.
 Medical School. Discussions with the Faculty team coordinating the Medical school 4th year course recognise a
need to incorporate SJTs into the curriculum from 2012/13 onwards. Medical students will be timetabled a half
day workshop to brainstorm SJT scenarios in small groups, debate answer choices, write explanations and submit
SJTs through SJTWise. Questions will be subject to review by the student peer group using SJTWise in the first
instance, and student groups will submit ‘final’ revised versions of their SJT to Faculty for quality assurance.
Some questions generated can be included in formative assessment planned during their 5th year as students
approach the summative examination and feedback given through SJTWise.
 Veterinary School. Tutorials on “becoming a professional” are already incorporated early in the curriculum. It is
proposed that from 2012/13 each student will generate a scenario with answers for subsequent discussion in small
groups during these tutorials. After agreement on a final version by the group, each SJT will be submitted through
SJTWise to the rest of the year. Each student will have ownership of one question, and will review comments
from their peers. Selected members of Faculty staff will be approached to provide quality assurance.
 Dental School. Situational judgements are currently underutilized for teaching purposes in the Dental school.
There will be pilot development of questions by students working in small group settings in the 5th year in 2012.
These will be extended in 2013 for all 5th year students providing question banks for teaching purposes but also
for potential formative assessment in the future after submission to Faculty staff for quality assurance purposes.
d) Staff support through discussions. In the project’s first year, Faculty from all three disciplines have met
regularly and have established strong working relationships. Staff will continue to meet to share progress,
experience and discuss further project implementation and outcomes. Specific details about student recruitment,
training and developing project evaluation in each of the Schools will be addressed to ensure consistency in
approach and to ensure the maximum benefit for College students.
5. Application to Glasgow Graduate Attributes
In addition to building the student knowledge base, as well as competence and ability to deal with uncertainty in
vocational subjects, writing questions / formulating evidence based answers in MCQ and SJT formats will also:a) enhance problem solving skills and benefit written communication skills;
b) encourage engagement with peers in each discipline and the student community as a whole;
c) develop higher level critical thinking and discipline relevant professional skills;
d) provide an environment to encourage team working and review of ethical/professional issues.
6. Potential Applicable/ Transferability
Project results will be disseminated to;a) more students in each discipline – results from this project can be rolled out to each of the five year groups for
PeerWise. Subsequently, students in the three schools will be provided with access to SJTWise, and pedagogical
aspects will be decided by Faculty staff at a local level after discussion with the student body.
b) students of other disciplines – PeerWise is already available within the University. Results from this study on
PeerWise and SJTWise will be disseminated to students in other professional disciplines within the University of
Glasgow via Learning and Teaching Centre. Workshops and seminars will be developed to encourage uptake
among other disciplines in the University.
c) academics from other Universities – data from this project will be submitted for review at local, national and
international meetings, some of which offer opportunities to deliver workshops to enhance awareness amongst the
interested community. Results will be written up for publication in high impact peer reviewed journals.
7. Evaluation
A wide range of data will be collected about PeerWise and SJTWise to enable a full and thorough evaluation to be
performed as indicators of change in student learning. These will include:-
a) Student use of PeerWise and SJTWise. Student participation is recorded, allowing review of questions, breadth of
subject matter, usage, and peer assessment for quality control and potential use of questions in subsequent years.
b) Students will continue to be surveyed using Course Experience Questionnaires (CEQs) and focus groups (FGs)
giving information on use and value of PeerWise. Similar evaluation of student perceptions of SJTWise will be
undertaken, particularly in how well SJT questions address graduate attributes.
c) Questionnaires will be developed to facilitate quality assurance of SJT questions submitted by students covering
content and style. Experts will be asked to assess scenarios, answers, explanations and feedback statements. Staff
may select suitable questions for formative and summative exams in subsequent years. CEQs will be developed and
FGs will be used to inform on staff perceptions on use of PeerWise and SJTWise.
d) Additionally, we will compare results from PeerWise with results from written examination performance.
Comparison of SJTWise use will be more complex but could be applied between usage and results from specific
OSCE questions that address communication and professional skills.
8. Timetable
Table 2 summarises achievements to date and that remaining to be completed in the existing LTDF project. Table 3
describes the steps and the milestones (in bold) of the proposed project. This includes rolling out, further evaluation
of PeerWise as well as the development, use, and evaluation of the SJTWise software.
Table 2 Present LTDF funded project running from 2011-2012
PEERWISE (PW)
Activity
Evaluation
Still to complete
Instigated PW for 1st year dental, veterinary
and medical students
Usage, some focus groups
Complete focus groups
Complete course experience questionnaire
SITUATIONAL JUDGEMENT TEST
(SJT) DEVELOPMENT
4th Year medical Students (SSC) developed
a paper based SJT / sat by 4th year peers
Evaluation of paper based tests including
usage, questionnaire
Complete SJTWise prototype by July 2011
Summer students
-review data and evaluation from PW
- trial the SJTWise prototype
- finish written instructions for writing SJTs for
use in SJTWise
Table 3. Proposed timetable for new application from 2012-2014
Year 1 LTDF application for
2012-13 provisional dates
Aug 2012 – Dec 2012
Sep 2012 - Feb 2013
Feb - May2013
Jun 2013 - Aug 2013
Jun 2013 - Aug 2013
Sep 2012 - Feb 2014
Feb 2014 –May 2014
Jun –Jul 2014
Jun-Jul/Aug 2014
PEERWISE (PW) USE
AND DEVELOPMENT
Update existing Ethics Application
SITUATIONAL JUDGEMENT TEST
(SJT) USE AND DEVELOPMENT
Update existing Ethics Application
Evaluate SJTWise prototype
Develop SJTWise final version
Recruiting year 1 students in all schools
Develop and run small group SJT writing days
Roll out PW to year 2 students
for year 4 medical students
Initiate SJT writing for veterinary and dental
students in professionalism teaching session
Review usage of PW, by focus group
Review usage of SJT by focus group
and questionnaires
and questionnaires
Compare PW usage with exam results
Faculty quality assessment of SJTs and
communicate results with year 4 students
Summer students review data collected from all schools
Recruit new year 1 students
Set up SJT programme for next year 4
Roll out PW to year 2 & 3 students
Continue SJT writing for veterinary and dental
Students in professionalism teaching session
Review usage of PW, focus group
Review Glasgow SJTs in light of
Summative examination results in January
and questionnaires
Evaluate experience of student written MCQsEvaluate experience of student written SJTs
using questionnaires, focus groups and
using questionnaires, focus groups and
comparison with examination marks
comparison with examination marks
Summer students review evaluation
Staff write up project
9. Budget
This project requires funding to develop PeerWise and SJTWise software to enable by students in all three schools
to write MCQs and SJT so that knowledge, feedback, and graduate-attribute skills can be enhanced in as many
students as possible. Time is therefore requested for Dr. John Hamer, an educationalist and software developer with
extensive published experience on PeerWise and its evaluation (6) to develop the necessary software. Funding is
also requested for a clinical academic (SP2, one day/week) to work with students and clinical colleagues in all three
disciplines on scenario development, train students in SJT use and, set up SJT workshops. It is anticipated that the
applicant would work towards a medical education degree while in post. Time is also requested to release present
teaching staff and pay for student summer projects to evaluate PeerWise and SJTWise use, and to write up data for
dissemination.
Table 3 Budget Costs for two Year Project
COSTS YEAR 1
COSTS YEAR 2
JH IT Consultant
(32.5 sessions @ £200 each)
Student Summer Scholarships
(two students for three weeks)
Teaching staff time (JB CD)
£4,000 (20 sessions)
£2,500 (12.5 sessions)
£1,200
£1,200
£1,000
£1,000
Teaching staff time (VB DC)
£1,000
£1,000
Clinical Academic
(honorary speciality trainee)
(one day per week, UCEA spinal point 2)
TOTAL
£8435
£8435
(basic salary £6,812, superannuation £1,090, (basic salary £6,812, superannuation £1,090,
national insurance &£ 5320)
national insurance &£ 5320)
£15,635
£14,135
Total requested £29,770
8. References
1) Hattie J, and Timperley H (2007). The power of feedback. Review of Educational Research, 77, 181-112.
2) Topping K. (1998) Peer Assessment between students in colleges and universities. Review of Ed. Research, 68, 249-276.
3) Sadler, D. (2010). Beyond feedback: developing student capability in complex appraisal. Assessment & Evaluation in Higher
Education, 35(5), 535-550.
4) Learning outcomes for the medical undergraduate in Scotland: A foundation for competent and reflective practitioners
(2000) Scottish Doctor http://www.scottishdoctor.org
5) Sykes A et al (2010) PeerWise; the Marmite of Veterinary student learning 10th European Conference on e-Learning
6) Denny P, Hamer J et al (2008). PeerWise: students sharing their multiple choice questions. Proc. 4th International
Workshop on Computing Education Research. ACM, 51-58.
7) Cameron D, Binnie V et al (2010). Peer Assisted Learning – an innovative approach for Dentistry, AMEE Conference 2010
8) Improving selection into foundation programmes (2009) Medical Schools Council http://www.isfp.org.uk/Pages/SJT-andEPM.aspx
9) Hughes N, Hurley R et al (2012). Can students write effective Situational Judgement tests? Abstract submitted to AMEE
2011.
10) Burke J Fayed S. et al (2007). Peer-assisted learning in the acquisition of clinical skills: a supplementary approach to
musculoskeletal system training. Medical Teacher, 29, 577-82.
11) Cameron K, Grosset A et al (2010). Can students write Multiple Choice Questions for formative assessment. TALE
Conference. September 2010.
Should this application be successful and be grant funded, we give consent for this application to be published on the
Learning and Teaching Centre website.