SLMS CPD
Online assessment to enhance learning: Toolkit
Note: all objective tests (Moodle quizzes with MCQs, EMQs, MEQs) need to be piloted. Item
reliability needs to be high.
COURSE START
STEP 1 – break online ice
Create the right conditions for online distance assessment.
Set up a Forum for introductions. Build an online learning community by getting learners to write
about: their specialism, their context, their reason for doing the course, any concerns.
Get all lecturers on the course to model this forum post by introducing themselves first. Give
learners a deadline for this task.
STEP 2 – put learners into groups
Get a teaching administrator to look at the texts. Put learners in groups of three. Let them know
which groups they’re in via your News Forum – send out a list. Groups should be as diverse as
possible – different countries, different specialisms, different interests, mixed age and gender. The
groups will be used for peer feedback and group assessment tasks. You can sometimes combine two
groups.
STEP 3 – test existing knowledge
Set up extended matching questions (EMQs) or modified essay questions (MEQs ie questions with
vignettes) to test existing knowledge. Give learners the opportunity to access info to ensure they
know enough at the start of the course. Set the diagnostic tests so learners can repeat them until
they get them right.
Get your teaching administrator to check if there are any learners who are struggling. Mail them to
ensure they’ve got the resources they need.
STEP 4 – test writing and referencing ability
Are you asking for written assessment tasks? Ensure your global learners can do them or offer them
alternative forms of assessment if they can’t. Is that feasible? (Audio, video, mind maps, diagrams).
TEMPLATE FOR FORMATIVE AND SUMMATIVE ASSESSMENT OF A LEARNING OUTCOME
STEP 1 – write a visible learning outcome/s for a unit of study (perhaps two or three weeks)
By the end of this [unit of study] learners will be expected to be able to:
-
Select an appropriate treatment option for patients with […] OR
Select an appropriate surgical intervention for […]
Diagnose […] using […]
Rosalind Duhs, March 2013
1
Note the structure of the distance learning outcome. The learner is expected to be able to use (apply)
knowledge and this is made clear in the learning outcome. The outcome is about what the learner
will be able to do at the end of the unit of study, not the process of learning. The learning outcome
has to be visible so it can be assessed. ‘Know about, understand, be aware of’ are not visible.
‘Explain, describe, identify’ lead to an explanation, description, or identification, which can be
assessed.
STEP 2 – provide a source of knowledge and skills and plan learning activities with timelines
The most compelling way to learn is to find things out with peers, enquiry-based or problem-based
learning, linked to a case. Learners can bring their own cases to an online forum for discussion, draw
on course materials, and work out a treatment plan together in their groups or cross-groups. They
can watch a lecture and discuss questions in a forum online, always relating content to their own
practice and context. They can build up a set of cases together online and compare options and
outcomes.
Ensure that if your learning outcome involves diagnosis, learners get plenty of opportunities to
practise diagnosis. If it involves clinical reasoning, make sure there are plenty of scenarios to trigger
clinical reasoning. Link learning outcomes to online learning activities.
STEP 3 – plan formative assessment (feedback) interwoven with learning activities
Short quizzes, online forum comments with peer feedback, short formal texts, and short audio or
video clips can all be used to evaluate understanding of knowledge or ability to perform stage by
stage skills. Short, frequent formative assessment tasks support learning and confirm learners’
progress.
STEP 4 – plan summative assessment (contributes to final results)
The literature is full of questions about confirming who actually does assessment tasks when
assessment is online and not invigilated. This issue has to be considered for high stakes assessment.
You never know who actually wrote an essay yet we often assess by essay.
Security issues aside, devise a test for the learning outcome you have selected. You need learners to
do clinical reasoning if your learning outcome is linked to clinical reasoning around a particular
condition. MCQs will not test the ability to reason. They will test knowledge.
See the table overleaf for suggestions for online assessment tasks and what they test. It’s adapted
from Epstein (2007) ‘Assessment in Medical Education’ N Engl J Med;356:387-96.
STEP 5 – plan assessment criteria related to examples of the type of work you expect for openended assessment tasks
Online distance CPD will attract a wide range of learners from different backgrounds. They need to
know what is expected of them from the start of the course. Examples of the type of work which is
required and the way assessment criteria are applied to that work (upload a few different
approaches and levels of work) need to be supplied and learners need to practise giving feedback to
each other applying the assessment criteria to their own work and the work of others.
Rosalind Duhs, March 2013
2
Online Assessment Task
(consider group work)
What the task can assess
Written tasks
Formative or summative assessment
Multiple Choice – single best answer or extended Knowledge, problem-solving
matching
Key feature and script concordance
Clinical reasoning, problem-solving, application
of knowledge
Short answer questions
Interpretation of diagnostic tests, problemsolving, clinical reasoning
Structured essays
Synthesis of information, interpretation and use
and critique of literature
Log of case management
Clinical reasoning, problem-solving, application
of knowledge
Learning log related to changes in approaches to Course learning related to application of course
patient care
knowledge
Wiki (jointly edited text)
Group clinical reasoning, problem-solving,
application of knowledge
Web page (produced alone or with others)
Ability to present new approaches to clinical
reasoning, problem-solving, application of
knowledge
Online posters (individual or group)
Could test a range of knowledge and skills
Audio or video poster presentation
depending on the task
Audio and video
Formative or summative assessment
Unstructured evaluation of clinical work or skills
with open comments
With checklists for comment/rating
Modified online OSCE, communicate with
standardised patient via Skype, some diagnostic
info supplied
High-tech online simulation with team deciding
on action and reacting to events as they unfold
Recorded presentation of patient history
Clinical skills, communication, teamwork,
leadership
As above
Knowledge, skills, communication, applying
knowledge to practice
Self.- peer and patient assessment
Mainly formative, summative
possible
Could be written, audio or video or on apps such
as ‘Explain Everything’, drawings, diagrams
Teamwork, communication and interpersonal
skills, application of course learning
E-portfolios
Summative
Can contain any type of electronic file, written,
recorded, video-recorded, but clear guidelines
are needed to keep them manageable. Portfolios
can be compiled from draft texts produced
during the course.
Demonstrate attainment of specific course
learning outcomes and include an account of
how content provides evidence of achievement
so encourages reflection on learning.
Rosalind Duhs, March 2013
Procedural skills, teamwork, simulated clinical
dilemmas
Application of course learning, clinical reasoning,
problem-solving
3
Evaluation and revision
Distance online CPD is complex to plan and run. It is essential to monitor student engagement,
workload, and adjust courses according to learner needs. Piloting of assessment tasks, especially
objective e-tests (MCQs etc) is vital.
Resources
Good example of Key Feature Questions with patient vignette:
http://www.academia.edu/1749144/Writing_Key_Features_Problems
Script Concordance Tests: guidelines
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2427021/
Support
CALT SLMS staff (Rosalind Duhs and Teresa McConlogue) will help with course and assessment
design - r.duhs@ucl.ac.uk ; t.mcconlogue@ucl.ac.uk
E-learning Environments can provide expert support with Moodle - ele@ucl.ac.uk
Rosalind Duhs, March 2013
4