Generic Operative Supervised Learning Event – Briefing Document
The generic operative supervised learning event (GOSLE) has been developed in response
to the experiences of trainees and trainers with current PBAs. When asked in a recent survey
what changes could be made to PBAs to improve their use as a learning tool, a number of
suggestions arose, summarised below1, 2:
1.
2.
3.
4.
More procedures should be available as PBAs only exist for a subset of procedures.
PBAs should be more specific to the procedure.
Documentation should be more user-friendly and shorter, with fewer tick-boxes.
The global rating scale is very coarse. Most trainees get a rating of 2 or 3 for the
majority of their training, which makes it difficult to demonstrate progression,
assessed at ARCP.
5. Trainers should be trained in their use.
6. Trainers and trainees should share responsibility for their completion.
Further issues with the current PBA that have been addressed are:
1. No provision to record which parts of a procedure are undertaken.
2. Free-text fields are not used to great effect and do not encourage reflection.
3. No ability to recognise excellence.
The combination of these factors can make completing a PBA time-consuming. It may not
fully support the educational experience and completion can become a box-ticking exercise to
reach the requisite annual target for workplace-based assessments (WBAs).
The GOSLE takes these issues into account:
1. Possible application to any surgical procedure, in any specialty.
2. Fewer checkboxes, making it quicker to complete.
3. The consent component removed as this should generally be performed in a different
setting using a CEX or CBD.
4. Specification of the proportion and steps of the procedure undertaken, mirroring the
process of learning new operations.
5. Relative complexity of the procedure can be recorded.
6. Can be used in a simulated setting.
7. Expanded global rating scale to demonstrate progression and increase usefulness to
a wider range of trainees
8. The feedback text boxes provide clear guidance on what should be entered, inviting
targeted feedback about what was done well and specifics for how to improve.
9. An additional box has been added to describe complications or unexpected
complexity.
10. All relevant guidance notes are on the form itself, which may reduce the need for
specific training in the form’s use.
It would be useful in future to integrate the GOSLE with the logbook so assessments can link
directly to procedures and supplement the SAC guidance that trainees need to log a minimum
of 1800 cases prior to CCT, as a further layer of quality control.
A form that is quicker and more useful to complete may help shift us to a culture of
formatively assessing most procedures, rather than being forced to select only procedures
that are included in the PBA list. Trainers should be more likely to initiate an assessment
leading to regular structured feedback on performance and consequently improve the quality
of training. While the generic DOPS allows any procedure to be assessed, similar problems
arise including too many inappropriate fields and a coarse global rating.
GOSLE Pilot: AO Course Basingstoke November 2012
A pilot study of the GOSLE was undertaken at an AO Principles course. Its aims were to:
1. Obtain feedback about the ease-of-use and acceptability of the GOSLE from trainers
and trainees
2. Determine how the GOSLE could be improved
3. Compare the quality of feedback in the PBA and the GOSLE
Course delegates had their performance of various fixation techniques assessed using a mix
of PBAs and GOSLEs. Faculty and delegates completed a questionnaire relating to the
assessment methods. Responses were received from 25 delegates and 12 faculty members.
Overall, trainees were positive about the GOSLE (figure 1), and trainers even more so (figure
2).
Trainee Opinions about GOSLE
0%
20%
40%
60%
80%
100%
Easy to use
Strongly Agree
Good for Feedback
Agree
Neutral
Global Rating Easy
to Understand
Global Rating good
Discriminator
Figure 1. Trainee experiences of the GOSLE.
Disagree
Strongly Disagree
High Volume Trainee Opinions about GOSLE
0%
20%
40%
60%
80%
100%
Easy to use
Strongly Agree
Good for Feedback
Agree
Neutral
Global Rating Easy
to Understand
Disagree
Strongly Disagree
Global Rating good
Discriminator
Figure 2. Trainer experiences of the GOSLE.
Figures 3 and 4 indicate trainees’ and trainers’ preferences between the PBA and GOSLE.
The majority of trainees preferred the GOSLE or had no preference, with relatively few
preferring the PBA. Conversely, there is very strong support for the GOSLE by trainers.
Trainee Preferences
0%
10%
20%
30%
40%
50%
60%
70%
80%
90% 100%
Feedback Tool
Demonstrates
Progression
GOSLE
Neutral
Quicker to Complete
Preferred Overall
Figure 3. Trainee comparison between GOSLE and PBA.
PBA
Trainer Preferences
0%
20%
40%
60%
80%
100%
Feedback Tool
Demonstrates
Progression
GOSLE
Neutral
PBA
Quicker to Complete
Preferred Overall
Figure 4. Trainer comparison between GOSLE and PBA.
The quality of feedback in each of the available feedback boxes was categorised as either:




Descriptive: only descriptive comments, such as ‘simulated procedure’
General Feedback: comments such as ‘well done’
Specific Feedback: specific advice such as ‘aim for a better reduction’ or ‘good
guide wire positioning’
No feedback given
Approximately 50% of PBA feedback areas contained no feedback, and when given, it was
often non-specific (figure 5). Conversely, 90% of GOSLE feedback areas contained feedback
(figure 6) and it was usually specific.
ISCP PBA Feedback Boxes
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Comments
Descriptive
Conclusions
General
Specific
None
Actions
Figure 5. PBA Feedback Box Content
GOSLE Feedback Boxes
0%
10%
20%
30%
40%
50%
60%
70%
80%
90% 100%
Good Performance
Descriptive
General
Specific
None
Improvement
Figure 6. GOSLE Feedback Box Content
In summary, this pilot study demonstrated that trainees and trainers are positive about the
GOSLE. Higher-quality feedback was given for the GOSLE. We would hope that this
translates to better engagement with workplace-based assessment if it were to be adopted.
In almost all instances, the GOSLE is preferred to the PBA. Whilst the GOSLE is not
intended to replace the PBA, it appears to better support formative feedback for any
procedure performed. The GOSLE can therefore be used to complement the PBA, which is a
much more structured and prescriptive tool that lends itself to summative assessment of
performance. This can reduce trainee anxieties about the dichotomy of formative and
summative assessment that affects the PBA.
Validation
Following the pilot study, we believe that the GOSLE is ready for a larger-scale trial. Two
distinct features of the form require validation.
1. The global rating
2. User satisfaction and acceptability, which is crucial for a tool that is supposed to be
high-volume.
There are three proposed arms of the study:
1. Workplace-based study to determine feasibility, acceptability and performance of
the tool. A limited number of consultants and trainees have been using this
assessment and feedback has been universally positive. However, this is a selfselected sample and to avoid this sampling bias, we would like to undertake a
larger-scale trial of orthopaedic trainees.
2. Videotaped procedures performed by different levels of trainee which can then be
distributed and graded. This will assess the reliability of the global rating scale. It
could be combined with a parallel rating using the PBA.
3. Simulated procedures ‘scripted’ to different levels of performance to determine the
accuracy of the global rating scale and construct validity.
Several previous trials have reported on validation of the PBA3, 4. They have used
generalisability theory to assess reliability and similar methodology can be employed.
Recruitment has been difficult for the workplace-based trial as the GOSLE is regarded as
another assessment in addition to the 40 WBA target without any material benefit from
trainees. If SAC approval can be granted to allow GOSLEs to be counted toward the 40 WBA
target for a trial period, there will be no difficulty in finding volunteers. This will also improve
the validity of the results as it will emphasise equivalence with other forms of assessment and
reduce selection bias.
Note: The GOSLE can be tested online at
test@test.com and password is 123.
www.gosle.co.uk The username is
RM Davies
ST4 Trauma & Orthopaedics
North Western Deanery
PG Turner
Consultant Orthopaedic Surgeon & Head of School of Surgery
North Western Deanery
L Hadfield-Law
British Orthopaedic Association Education Advisor
References
1.
Hunter A. Unpublished Data. 2012.
2.
BOTA. Position Statement: Procedure Based Assessments in Orthopaedics and
Trauma: British Orthopaedic Trainees' Association; 2012.
3.
Beard JD, Marriott J, Purdie H, Crossley J. Assessing the surgical skills of trainees in
the operating theatre: a prospective observational study of the methodology. Health Technol
Assess. 2011; 15(1): i-xxi, 1-162.
4.
Marriott J, Purdie H, Crossley J, Beard JD. Evaluation of procedure-based
assessment for assessing trainees' skills in the operating theatre. The British Journal of
Surgery. 2011; 98(3): 450-7.