Surgical Endoscopy (2018) 32:4173–4182
https://doi.org/10.1007/s00464-018-6162-8
and Other Interventional Techniques
A novel assessment tool for evaluating competence in video-assisted
thoracoscopic surgery lobectomy
Katrine Jensen1,2
Lars Konge2
· René Horsleben Petersen1 · Henrik Jessen Hansen1 · William Walker3 · Jesper Holst Pedersen1 ·
Received: 6 September 2017 / Accepted: 21 March 2018 / Published online: 30 March 2018
© Springer Science+Business Media, LLC, part of Springer Nature 2018
Abstract
Background Specific assessment tools can accelerate trainees’ learning through structured feedback and ensure that trainees attain the knowledge and skills required to practice as competent, independent surgeons (competency-based surgical
education). The objective was to develop an assessment tool for video-assisted thoracoscopic surgery (VATS) lobectomy by
achieving consensus within an international group of VATS experts.
Method The Delphi method was used as a structured process for collecting and distilling knowledge from a group of internationally recognized VATS experts. Opinions were obtained in an iterative process involving answering repeated rounds
of questionnaires. Responses to one round were summarized and integrated into the next round of questionnaires until
consensus was reached.
Results Thirty-one VATS experts were included and four Delphi rounds were conducted. The response rate for each round
were 68.9% (31/45), 100% (31/31), 96.8% (30/31), and 93.3% (28/30) for the final round where consensus was reached. The
first Delphi round contained 44 items and the final VATS lobectomy Assessment Tool (VATSAT) comprised eight items with
rating anchors: (1) localization of tumor and other pathological tissue, (2) dissection of the hilum and veins, (3) dissection of
the arteries, (4) dissection of the bronchus, (5) dissection of lymph nodes, (6) retrieval of lobe in bag, (7) respect for tissue
and structures, and (8) technical skills in general.
Conclusion A novel and dedicated assessment tool for VATS lobectomy was developed based on VATS experts’ consensus.
The VATSAT can support the learning of VATS lobectomy by providing structured feedback and help supervisors make the
important decision of when trainees have acquired VATS lobectomy competencies for independent performance.
Keywords Assessment tool · VATS · video-assisted thoracoscopic surgery · Lobectomy · Thoracic surgical training ·
Competency-based education
Pulmonary lobectomy performed by video-assisted thoracoscopic surgery (VATS) has become the method of choice
over lobectomy by thoracotomy for early stage lung cancer,
because of its minimally invasive approach and the benefits
* Katrine Jensen
katrine.jensen@rh.regionh.dk
1
Department of Cardiothoracic Surgery 2152, University
Hospital of Copenhagen, Rigshospitalet, Copenhagen,
Denmark
2
Copenhagen Academy for Medical Education and Simulation
(CAMES) 5404, University of Copenhagen, Blegdamsvej 9,
2100 Copenhagen, Denmark
3
Department of Cardiothoracic Surgery, Royal Infirmary
of Edinburgh, Edinburgh, Scotland, UK
associated with it [1–4]. But the adoption rate of this technique is still low in some countries [5]. Thoracic surgical
trainees are learning to master this technique predominantly
by apprenticeship [6–8] where a training program typically
rely on senior surgical supervisors to provide formative feedback and obtain an impression of the trainees’ competences
throughout the training, without the use of predefined criteria. However, this type of non-criterion-based rating, as log
books and direct observation without specific assessment
criteria, is largely subjective and unreliable and therefore
cannot be considered an optimal method to assess competency [9–12].
To assess trainees’ surgical skills, surgical education
has therefore moved to base teaching and training in specific competencies, and there has been an effort to create
13
Vol.:(0123456789)
4174
evaluation tools that allow for objective, valid, and reliable
assessment of clinical performance [13–16] i.e., evaluation of real-life procedures through direct observation via
video-based technical skills evaluation in the operating room
[12, 17]. The integration of objective assessment tools into
training is essential to monitor skill acquisition, provide a
basis for structured evaluations and constructive feedback,
aid with talent selection, provide documentation for future
quality assurance, provide transparent and consistent standards, and increased safety for the patients [18–20].
Assessments of technical skills using expert-based ratings, such as a skill-specific checklist and global rating scale,
and computer-based assessments, such as hand motion efficiency analysis, are more objective and reliable, and are
crucial to the structured learning of skills [21]. However,
expert-based assessment tools are difficult to develop.
Experts often use variations in surgical techniques for the
same procedure, modify the order of procedural steps, and
prefer different instruments [22]. When developing an
expert-based assessment tool for technical performance, the
first step is to pool the opinions of experts in the same field
and to reach consensus as to which aspects of a procedure
that best assess performance, regardless of technique [4, 23].
Aim
This study aimed to obtain consensus within an international
group of recognized VATS experts regarding the development of an assessment tool for VATS lobectomy trainees.
The assessment tool would be designed to measure clinical VATS lobectomy performance and provide structured
assessment of selected competencies required to perform
VATS lobectomies safely and proficiently.
Materials, methods, and data analysis
The Delphi method process
The Delphi method is a structured process for collecting and
distilling knowledge from a group of experts by obtaining
opinions in an iterative process that involves mailing repetitive rounds of questionnaires. Comments and items from
one round are categorized and integrated into the next round
of questionnaires to seek agreement, disagreement, and
insights. The Delphi technique’s unique contribution is the
condensation of differing expert’s opinions into consensus
for decision making, without creating direct confrontation
or allowing strong individuals to dominate the process [23].
A Delphi process can be divided into the following three
stages [23]: (1) Exploration establishing criteria for selection
of participants, establishment of the expert panel, design of
13
Surgical Endoscopy (2018) 32:4173–4182
the data collection and analysis instruments, drawing up the
initial questionnaire and piloting it; (2) distillation iterative
rounds of questionnaires, incorporation of the expert’s comments into the questionnaire, monitoring of the response rate/
dropout rate, knowing when to stop the Delphi rounds; and (3)
utilization complementing qualitative data with quantitative
data, publishing study data.
Exploration
Inclusion of experts to the Delphi expert panel
The VATS experts invited to be part of the Delphi expert panel
were selected from the VATS consensus group [4], where they
were chosen based on their surgical VATS lobectomy experience, their publication record/knowledge of VATS lobectomy,
and by their role as committed members of organizations such
as the European Society of Thoracic Surgeons, American
Association of Thoracic Surgeons, and the Asian Society for
Cardiovascular and Thoracic Surgery. The experts were provided with a description of the purpose of the Delphi study and
how results were to be used, and the procedures were clearly
stated. The experts were explained that they were identified as
internationally recognized experts and were asked to nominate
others for inclusion in the expert panel.
Experts were included if they completed the first Delphi
round before a specified deadline of four weeks. The VATS
experts were from different regions of the world so global
diversities in opinions were duly represented.
Development of the first‑round Delphi questionnaire
and pre‑testing of the questionnaire
A list of potential global rating items was generated by
the authors based on the existing sparse literature [14, 16,
24–28] and the authors’ expert knowledge of VATS lobectomy and how to assess the competence of clinicians performing VATS lobectomy. A Global Rating Scale with 44
items were produced (Table 1), and pilot testing was done by
distributing the questionnaire via SurveyMonkey® (SurveyMonkey, California, USA) to three senior thoracic surgeons
at the Copenhagen University Hospital, Rigshospitalet,
Denmark (HJH, RHP, and JHP) to test the questionnaire for
content and wording. The items were then modified according to their comments.
Distillation
Sending the first modified questionnaire to participants
and collecting returns
During the first round, participants individually rated the
importance of each global rating item as an indicator of the
Surgical Endoscopy (2018) 32:4173–4182
4175
Table 1 Items in the first Delphi round
final global rating scale). Participants had the opportunity
to provide reasons for their choices, add additional items,
and suggest a different wording of items. Expert panelists
were asked to comment on the list of potential items and
identify any additional items. The response forms were
coded with the participants’ email address to keep track of
returns so individual responses remained confidential, but
were potentially not anonymous to the principal investigator (KJ) since she handled the system to collect responses.
The analysis of the results/comments was done after these
were exported from SurveyMonkey® and anonymized by
an IT-technician not otherwise involved in the study. Some
participants left signed comments, and these were answered
directly via email by KJ.
Original items (n = 44)
Dissection of the artery(ies)
Recognition of anatomical landmarks
Respect for the esophagus
Skills to control bleeding
Recognition of anatomical abnormalities/variations
Respect for n. Phrenicus
Dissection of the bronchus
Dissection of the hilum
Dissection of the vein(s)
Localization of pathological tissue
Strategy/technique to deal with unexpected findings
Hemostasis at the end of procedure
Ability to adapt to individual pathological circumstances
Dissection of lymph nodes
Use of stapler on structures
Precision of movements
Hemostasis perioperatively
Retrieval of the lobe in bag
Number of lymph node stations dissected
Respect for tissue in general
Respect for the lung parenchyma in adjacent lobe(s)
Technical skills in general
Strategies for camera placement
Handling of instruments
Item for the rater: difficulty of case
Stapling of the fissure
Retraction of the lobe to get good visualization
Respect for the lung parenchyma in affected lobe
Forward planning of the procedure
Use of unnecessary force
Flow of the procedure
Independent procedure completion
Management of findings
Confidence of movements
Placement of chest tube
Fluidity of hand motions
Depth perception
Economy of movements
Bimanually dexterity
Respect for n. Vagus
Dissection of the lower pulmonary ligament
Time for completion of procedure
Use of a fissure-less technique
Number of instrument switches
competence of trainees learning to perform VATS lobectomy on an ordinal scale from 1 Not important (item should
under no circumstance be on the final global rating scale)
to 5 absolutely mandatory (item should definitely be on the
Analyzing the first‑round responses and preparing
the second‑round questionnaire
The median rating for each item and the proportion of
panelists rating an item to four or above, was calculated.
A global rating item was removed if its median rating was
below four. Based on the experts’ comments, items and
anchors were combined, modified and/or expanded, and proposed additional items were added (Fig. 1). The views of all
participating experts were given equal weight. Consensus
on an item was reached when 80% or more of the experts
scored an item on the global rating scale to four or five for
any given round. The items were then grouped conceptually
into categories (Dissection, strategic and cognitive skills,
respect for tissue and structures, hemostasis, and technical
skills) to make it easier for panelists to comprehend each
item when returned in the second round.
Sending the second‑round questionnaire to participants
and collecting returns
The experts were given a written explanation along with the
second-round questionnaire and asked to verify that we had
correctly interpreted their responses and to verify and refine
the items. Each item was given appropriate rating anchors
and the experts were asked to rate the items on the new list
and suggest additional items that they might not have considered initially, and to comment on the wording of the anchors.
Analyzing the subsequent rounds of responses
In the subsequent rounds, the list of items was even more
narrowed down, and the experts were again given a written explanation of how their answers were interpreted, and
asked to rate the items, suggest additional items, and to comment on the wording of the anchors. The goal of the final
phase was to reach a consensus in the rating of the relevant
items, as defined above. The numbers of participants were
13
4176
Surgical Endoscopy (2018) 32:4173–4182
Fig. 1 Flowchart
monitored not to drop below a critical level, which in this
case was considered to be a minimum of fifteen panelists
[23]. The process continued until consensus was achieved
(until all remaining items were scored to a minimum of four
by a minimum of 80% of the experts).
Utilization
Preparing the final report and sharing with participants
The final version of the VATS lobectomy assessment tool
was prepared and sent to the expert panel for final approval
and refinement of the content and wording. Written acceptances to be named in the published paper were obtained.
Results
Thirty-one out of the 45 invited VATS experts completed
Delphi round one and were included to be a part of the expert
panel (Table 2). No additional VATS experts were appointed
by the expert panel. Four Delphi rounds were conducted
13
from June 2nd to November 6th 2015. The response rate
for round one was 68.9% (31/45), 100% (31/31) for round
two, 96.8% (30/31) for round three, and 93.3% (28/30) for
the final round. Consensus on final items was reached in
Delphi round four. The first Delphi round contained 44 items
(Table 1) and Fig. 1 shows how many items the expert rated
to four or above for each round, and if items were added
or combined. The final VATS lobectomy assessment tool
(VATSAT) contains 8 items (Fig. 2), where two can be categorized as cognitive competencies (items 1 and 7) and six
as technical competencies (items 2–6 and 8).Table 3 shows
the score and some of the comments for the final items for
the fourth round.
Discussion
We used the Delphi method to solicit opinions from geographically dispersed and busy experts, hereby aggregating
quantitative data of the expert’s opinions on VATS lobectomy assessment without a face-to-face meeting to avoid
direct confrontation of experts with opposing participants
Surgical Endoscopy (2018) 32:4173–4182
4177
Table 2 Overview of countries
Country (n = 16)
Participants
(n = 31)
Australia
Austria
Belgium
Denmark
France
Germany
Hong Kong (China)
Italy
Japan
Luxembourg
Netherlands
Poland
Spain
Switzerland
United Kingdom
United States of America
3
1
3
2
2
2
1
1
1
1
2
1
1
1
4
5
from a previous established VATS consensus group [4]
were chosen, since the quality and accuracy of responses
in a Delphi study are only as good as the expert quality of
the participants who are involved in the process. Thirty-one
experts were included in this study. Ten to 15 people may
be adequate for a focused Delphi where participants do not
vary a great deal, but most studies use between 15 and 35
experts [23]. The dropout rate of a Delphi study is often high
and thereby reduces the participant for subsequent statistical
analysis, but here it was only 9.7% (3/31) over the two last
rounds. The wider a time gap allowed between each round
the more change in the experts’ circumstance, knowledge,
and general situation, and we strived for as short as possible
study period. Psychological factors such as work pressure for
the researchers and the experts, the time of day and month
when the survey was completed, the mood of the participant
as well as the construction of the questionnaire itself could
have impacted the study and caused random and systematic errors, but these are almost impossible to detect [23].
The items were carefully constructed and the rounds clearly
organized with explicit instructions for the experts, but in
the first Delphi round, the experts commented that six items
were not clearly understood. These items were rephrased
and explained and sent to the second Delphi round even
if they did not score a median of four or above. We were
aware of the threat that our subjective interpretation could
have when reflecting upon changes like these and failure
to understand the context for the consensus may have led
to failure of capturing important information. However, the
experts were asked to comment on their reasoning, which
allowed for further construct validation by asking experts to
validate the researcher’s interpretation and categorization of
the variables.
The expert panel reached consensus on eight items which
compared to other global rating scales such as the Objective
Structured Assessment of Technical Skills (OSATS) can be
considered an appropriate number of items [9, 16, 19, 29].
We chose to develop a global rating scale without an accompanying checklist since the global rating scale has higher
inter-item and inter-station reliability and may better capture
nuanced elements of expertise compared with the checklist
[30]. It is found that trainees scoring high on a checklist were
deemed incompetent when compared with how they scored
on a global rating scale, and the practice of universal adoption of checklists as the preferred method of assessment of
procedural skills should be questioned since the global rating
scale outperforms the checklist in its discriminant ability and
inter-rater agreement [21, 30].
The experts in the panel are using different types of
VATS lobectomy (single port, two or three port technique,
retractors, anterior/posterior approach), but the VATSAT
is designed to rate a trainee regardless of technique and
lobes. The technique does not influence the rating items,
consequently, rather than having a useful scale specific to an
approach, only generic criteria emerged. Competence in performing a VATS lobectomy optimally requires proficiency
demonstrated in three domains: (1) cognitive competencies,
(2) technical competencies, and (3) integrative competencies (i.e., communication, judgment, clinical reasoning)
[28]. Integrative and to a part cognitive competencies were
not included in the final assessment tool, since these can be
difficult to rate from a video-based performance, unless the
video recording is with audio. Many VATS operating theaters already have recording equipment for the thoracoscope,
but this does not include audio, and the trainee would have
to be equipped with a microphone. If a trainee is not fluent
in other languages than their primary, only raters who speak
the trainees language can rate the performance, and therefore
integrative competencies were not included since the VATS
lobectomy assessment tool was thought to be developed
for full objectivity independent of nationality. Video-based
technical skills evaluation has been shown to be feasible,
valid, and reliable, to improve trainees’ self-assessment
skills by promoting reflective practice, and is effective in a
self-directed learning environment among novices [12, 17].
However, cognitive and integrative competencies are
important factors for the direct assessment of the trainee, and
assessment of the surgeons’ non-technical skills can enhance
the surgical training portfolio and should be an integral feature of the development and assessment of operative skills,
and assessment tools for non-technical skills have already
been developed for this purpose i.e., the NOTSS (NonTechnical Skills for Surgeons) assessment tool, NOTECHS
13
4178
Surgical Endoscopy (2018) 32:4173–4182
Fig. 2 The final VATS lobectomy Assessment Tool (VATSAT) comprised eight items with rating anchors
(Non-Technical Skills), and OTAS (Observational Teamwork Assessment for Surgery) [31]. The VATSAT assesses
mandatory technical skills necessary to perform a VATS
lobectomy, and should be used in a thoracic surgery competency-based curriculum that comprised assessment tools
for both technical and non-technical skills (cognitive skills/
decision making) and pre-op checklists.
13
Mastery learning is a new paradigm for medical education
and a stringent form of competency-based education where
the basic principles of mastery learning are that educational
excellence is expected and can be achieved by all learners
and that all learners achieve all educational objectives with
little or no variation in outcome [32, 33]. A mastery learning
curriculum for VATS lobectomy could begin with theory
Surgical Endoscopy (2018) 32:4173–4182
4179
Table 3 Comments from the VATS expert panel on the final items during the Delphi rounds
Items
Comments
1. Localization of tumor and other pathological tissue
Percent score ≥ 4 = 97%
“Important, must be sure that there is no pleuritis carcinomatosa.”
“This is very important point. If the pleural dissemination is looked over, the procedure
should not be proceeded.”
“Even if he or she is a trainee, any surgeon should look for the pathology and consider
the process of the procedure.”
“If a tumor is clearly visible, palpation of it might not be necessary.”
“Some small tumors are not palpable by VATS technique.”
“Single pulmonary vein must be checked to prevent unnecessary pneumonectomy.”
“For VATS lobectomy, hilar dissemination is a basic skill.”
“Dissection in the relevant layer to identify vessels and prevent vessel injury.”
“Pulmonary vein transection is a mandatory skill in VATS lobectomy.”
“Venous dissection for VATS lobectomy is a fundamental technique for a trainee.”
“Dissection AND correct identification.”
“Should the surgeon check the number of veins and origin of the middle lobe vein on CT
and confirm his impression at this stage.”
“Important. Failures with artery are most common cause for bleeding.”
“In most of the normal anatomy patients, even the trainee should be able to dissect and
staple arteries safely by themselves.”
“Pulmonary arterial dissection and separation is also a mandatory skill in VATS lobectomy.”
“Knows when to prepare main PA for control.”
“Probably most important.”
“Dissection AND correct identification.”
“What about awareness of and assessment for anatomical variations e.g. single pulmonary vein, abnormal arterial anatomy.”
“Excellent.”
“Bronchial dissection is also a mandatory skill in VATS lobectomy.”
“Safe and sufficient bronchial dissection is also very important skill to remove the hilar
lymph nodes in lung cancer surgery.”
“Important to also deal with the bronchial arteries.”
“Lymph node dissection technique and the extent of the lymph node dissection is not
standardized.”
“So far, lymph node dissection is only a diagnostic procedure. And proper lymph node
dissection technique is not universal yet.”
“The term radically suggests you are doing a lymph node clearance. Some people only
sample (which is an argument for another day). I therefore feel the word radically
should be removed.”
“May need to decide what the trainee is supposed to do : some units sample, others (few
in reality, I think) radically dissect.”
“Safe and sufficient lymph node dissection is an important technique in lung cancer
surgery. However, even radical lymph node dissection is proven to be important for the
staging of lung cancer.”
“I BELIEVE MENTION SHOULD BE MADE IF TRAINEE IS ABLE TO REMOVE
THE LYMPH NODES INTACT AND EN-BLOC OR IF THE NODES ARE
REMOVED PIECEMEAL.”
“To retrieve the lobe through the small incision without spreading of tumor cells, the
trainee should be able to perform this process.”
“There are many kinds of retrieval bags form easy to place a lobe to difficult.”
“This can be time consuming in the best of hands. Lack of knowledge of extraction
manouevres would be my concern.”
“The wording of Anchor 1 might suggest that the trainee has already spread tumour
cells. Perhaps it would be better to add “creating the risk of”, or similar, after “without”.”
2. Dissection of the hilum and veins
Percent score ≥ 4 = 97%
3. Dissection of the arteries
Percent score ≥ 4 = 97%
4. Dissection of the bronchus
Percent score ≥ 4 = 97%
5. Dissection of lymph nodes
(Sampling/radically resection and stations/number of
stations decided by the supervisor preoperatively)
Percent score ≥ 4 = 87%
6. Retrieval of lobe in bag
Percent score ≥ 4 = 81%
13
4180
Surgical Endoscopy (2018) 32:4173–4182
Table 3 (continued)
Items
Comments
7. Respect for tissue and structures
Percent score ≥ 4 = 83%
“Lung tissue is the most fragile tissue in the body. Appropriate handling of the lung is
mandatory.”
“Any surgeon should not injure the adjacent tissue.”
“For easier assessment, this could be summarised as to handling of any mediastinal
structure: esophagus, main bronchi; azygos vein, aorta...”
“I think there should be a general question I respect for surrounding structure not a
specific structure.”
“Lung surgeons should be able to perform procedure without damaging the lung.”
“Adjacent lobes should be well preserved.”
“If the technical skill is complete, the trainee is no longer a trainee.”
“I believe that any this is a pre-requiry assessment. Any trainee not scoring 5 on this is
not ready to do any part of dissection pulmonary veins, arteries or bronchi.”
“The trainee should be able to use instruments with somewhat familiarity.”
“The trainee should be able to use instruments correctly and without force.”
“This should definitely be trained elsewhere than during an operation on a real patient
(wet lab, animal model..).”
“Interesting. Good project.”
“I think this evolving into a nice training record. It should allow objective scoring and
graphs of each area to show progression - or not.”
“All very important issues.”
“Well done -congratulations!”
“The evaluation of the technique is very important for the training of VATS lobectomy.”
“I believe this should be a good tool for rating a trainee in VATS Lobectomy. I have
only one more suggestion; I would add an item about the ability to understand when
it needs to stop the procedure and call the supervisor because the case is too difficult
(hilar calcified lymph nodes, thick adhesions, vascular anomalies, etc.)”
“I agree with all the items but as I told before I will implement the pre-operative course
with perfect description of the CTscan and the previsible difficulties of the case. I also
will add installation of the patient on the table which for mandatory before starting one
case.”
“Love it.”
8. Technical skills in general
Percent score ≥ 4 = 83%
Comments for the final VATSAT
and passing a multiple choice test as described by Savran
et al. [26]. After training on simulators to reach competency
in basic VATS skills, more advanced VATS skills could be
assessed utilizing virtual reality simulators, lobectomy on
a porcine heart and lung block, or on an anesthetized pig
measured with the VATSAT and feedback given, which in
turn can lead to self-regulated learning [24, 25, 27]. Thereafter the trainee can operate supervised until proficient, where
objective assessments with the VATSAT can be used to
ensure that a proficiency level has been reached and potentially as a condition for independent practice [20]. Here the
VATSAT can also be used for giving meaningful feedback to
the trainee since assessment of operative skills in the operating theater using this assessment tool has the potential to
establish learning curves and allowing adequate monitoring
of the trainees’ progress in achieving technical competencies [19]. The ongoing development of surgical simulators
[24] and educational curricula will hopefully facilitate the
transition to a competency-based VATS lobectomy program
based on mastery learning where trainees must demonstrate
technical competence to progress to the next level of training and gain certification and re-certification and ultimately
ensuring better and faster technical skill acquisition as well
13
as improved quality of care and patient safety [12, 19]. An
assessment tool used in The National Training Programme
in Laparoscopic Colorectal Surgery in England was found by
Miskovic et al. [29] to reliably assess technical performance
in laparoscopic colorectal surgery and is used on a national
scale to judge specialist technical performance. Miskovic
et al. point out that other specialties can adapt the same
method, and this suggest that the VATSAT could be incorporated in a competency-based training program for thoracic
surgeons, since these assessment tools are very comparable.
An objective assessment tool for VATS lobectomy has
previously not been produced, and our instrument adds to the
literature in that it is developed in a comprehensive and systematic manner, and is designed to rate trainees from video
recordings of their actual performances for full objectivity.
The study has identified, by consensus, what the most important steps of the operation are, and this may help trainees
conceptualize the operation prior to learning, and it may
help them problem-solve more efficiently to optimize their
learning curve. Studies to provide validity evidence for the
VATSAT are ongoing and are being applied to both virtual reality-simulated procedures and real-life procedures.
The VATSAT provides supervisors and assessors with a
Surgical Endoscopy (2018) 32:4173–4182
procedure-specific assessment tool for evaluating VATS
lobectomy performance and is sought to aid both supervisor
and trainee with learning curves and feedback on technical
skills.
Conclusion
A novel and unique VATS lobectomy assessment tool was
developed in a comprehensive, systematic, and transparent
manner using the Delphi method in collaboration with a
large number of internationally recognized VATS experts.
Response rates were high and consensus on items and
anchors were reached in Delphi round four, and the final
VATS lobectomy assessment tool consists of eight items.
The assessment tool allows for rating of trainees VATS
lobectomy skills based on video recordings of their performance for full objectivity.
Acknowledgements VATS lobectomy experts were kindly asked to
participate as part of the expert panel, and could decline at any phase
in the study. At any time, participants could withdraw their consent
for participation and their data would be deleted. All data were kept
strictly confidential. The questionnaires from each expert were not
discussed with colleagues or other individuals. The experts were told
that the answers they gave in the questionnaires were anonymous for
all except the main author who was handling the data via email and
Survey Monkey, but that they would be acknowledged by name in the
published paper if they wished to.
VATS lobectomy assessment tool expert panel: Baste JM, Bodner J,
Cao C, Casali G, D’amico T, De Ryck F, Decaluwe HMA, Decker G,
Dunning J, Gossot D, Kohno T, Leschber G, Liptay MJ, Loscertales J,
McKenna RJ, Mitchell JD, Oosterhuis JW, Piwkowski CT, Schmid T,
Schneiter D, Shackcloth M, Siebenga J, Sihoe A, Sokolow Y, Solaini
L, Wright GM, Yan TD.
Funding The salary of Katrine Jensen was partly funded by a grant
from The Danish Cancer Society (Kræftens Bekæmpelse, Denmark).
Compliance with ethical standards
Disclosures Henrik Jessen Hansen and René Horsleben Petersen are
at the speaker’s bureau for Covidien. Jesper Holst Pedersen, William
Walker, and Lars Konge have no conflicts of interest or financial ties
to disclose.
Ethical approval No samples from humans were used in the study and
no drugs were administered; hence, this study needed no approval from
The Danish National Committee on Biomedical Research Ethics.
References
1. Falcoz PE, Puyraveau M, Thomas PA, Decaluwe H, Hurtgen M,
Petersen RH, Hansen H, Brunelli A (2016) Video-assisted thoracoscopic surgery versus open lobectomy for primary non-smallcell lung cancer: a propensity-matched analysis of outcome from
the European Society of Thoracic Surgeon database. Eur J Cardiothorac Surg 49:602–609. https​://doi.org/10.1093/ejcts​/ezv15​4
4181
2. Laursen L, Petersen RH, Hansen HJ, Jensen TK, Ravn J, Konge L
(2016) Video-assisted thoracoscopic surgery lobectomy for lung
cancer is associated with a lower 30-day morbidity compared with
lobectomy by thoracotomy. Eur J Cardiothorac Surg 49:870–875.
https​://doi.org/10.1093/ejcts​/ezv20​5
3. Whitson BA, Groth SS, Duval SJ, Swanson SJ, Maddaus MA
(2008) Surgery for early-stage non-small cell lung cancer: a systematic review of the video-assisted thoracoscopic surgery versus
thoracotomy approaches to lobectomy. Ann Thorac Surg 86:2008–
2018. https​://doi.org/10.1016/j.athor​acsur​.2008.07.009
4. Yan TD, Cao C, D’Amico TA, Demmy TL, He J, Hansen H, Swanson SJ, Walker WS (2014) Video-assisted thoracoscopic surgery
lobectomy at 20 years: a consensus statement. Eur J Cardiothorac
Surg 45:633–639. https​://doi.org/10.1093/ejcts​/ezt46​3
5. Cao C, Tian DH, Wolak K, Oparka J, He J, Dunning J, Walker
WS, Yan TD (2014) Cross-sectional survey on lobectomy
approach (X-SOLA). Chest 146:292–298. https:​ //doi.org/10.1378/
chest​.13-1075
6. Petersen RH, Hansen HJ (2010) Learning thoracoscopic
lobectomy. Eur J Cardiothorac Surg 37:516–520. https​://doi.
org/10.1016/j.ejcts​.2009.09.012
7. Ferguson J, Walker W (2006) Developing a VATS lobectomy programme—can VATS lobectomy be taught? Eur J Cardiothorac
Surg 29:806–809. https​://doi.org/10.1016/j.ejcts​.2006.02.012
8. Carrott PW, Jones DR (2013) Teaching video-assisted thoracic surgery (VATS) lobectomy. J Thorac Dis. https​: //doi.
org/10.3978/j.issn.2072-1439.2013.07.31
9. Moorthy K, Munz Y, Sarker SK, Darzi A (2003) Clinical
review objective assessment of technical skills in surgery. BMJ
327:1032–1037
10. Moon MR (2014) Technical skills assessment in thoracic surgery
education: we won’t get fooled again. J Thorac Cardiovasc Surg
148:2497–2498. https​://doi.org/10.1016/j.jtcvs​.2014.09.057
11. Grantcharov TP, Reznick RK (2008) Teaching procedural skills.
BMJ 336:1129–1131. https​://doi.org/10.1136/bmj.39517​.68695​
6.47
12. Lodge D, Grantcharov T (2011) Training and assessment of technical skills and competency in cardiac surgery. Eur J Cardiothorac
Surg 39:287–293. https​://doi.org/10.1016/j.ejcts​.2010.06.035
13. Gardner AK, Scott DJ, Choti M, Mansour JC (2015) Developing
a comprehensive resident education evaluation system in the era
of milestone assessment. J Surg Educ. https​://doi.org/10.1016/j.
jsurg​.2014.12.007
14. Meyerson SL, Tong BC, Balderson SS, D’Amico TA, Phillips JD,
Decamp MM, Darosa DA (2012) Needs assessment for an errorsbased curriculum on thoracoscopic lobectomy. Ann Thorac Surg
94:368–373. https​://doi.org/10.1016/j.athor​acsur​.2012.04.023
15. Öhrn A, Olai A, Rutberg H, Nilsen P, Tropp H (2011) Adverse
events in spine surgery in Sweden. Acta Orthop 82:727–731. https​
://doi.org/10.3109/17453​674.2011.63667​3
16. Konge L, Lehnert P, Hansen HJ, Petersen RH, Ringsted C (2012)
Reliable and valid assessment of performance in thoracoscopy.
Surg Endosc 26:1624–1628. https ​ : //doi.org/10.1007/s0046​
4-011-2081-7
17. Aggarwal R, Grantcharov T, Moorthy K, Milland T, Darzi A
(2008) Toward feasible, valid, and reliable video-based assessments of technical surgical skills in the operating room. Ann Surg
247:372–379. https​://doi.org/10.1097/SLA.0b013​e3181​60b37​1
18. ten Cate O, Scheele F (2007) Competency-based postgraduate
training: can we bridge the gap between theory and clinical practice? Acad Med 82:542–547. https:​ //doi.org/10.1097/ACM.0b013​
e3180​5559c​7
19. Hopmans CJ, den Hoed PT, van der Laan L, van der Harst E,
van der Elst M, Mannaerts GHH, Dawson I, Timman R, Wijnhoven BPL, IJzermans JNM (2014) Assessment of surgery residents’ operative skills in the operating theater using a modified
13
4182
20.
21.
22.
23.
24.
25.
26.
Surgical Endoscopy (2018) 32:4173–4182
objective structured assessment of technical skills (OSATS): a
prospective multicenter study. Surgery 156:1078–1088. https​://
doi.org/10.1016/j.surg.2014.04.052
Vassiliou MC, Feldman LS (2011) Objective assessment, selection, and certification in surgery. Surg Oncol 20:140–145. https​
://doi.org/10.1016/j.suron​c.2010.09.004
Szasz P, Louridas M, Harris KA, Aggarwal R, Grantcharov TP
(2015) Assessing technical competence in surgical trainees: a systematic review. Ann Surg 26:1046–1055. https​://doi.org/10.1097/
SLA.00000​00000​00086​6
Rocco G, Internullo E, Cassivi SD, Van Raemdonck D, Ferguson MK (2008) The variability of practice in minimally invasive
thoracic surgery for pulmonary resections. Thorac Surg Clin
18:235–247. https​://doi.org/10.1016/j.thors​urg.2008.06.002
Day J, Bobeva M (2005) A generic toolkit for the successful management of delphi studies. Electron J Bus Res Methods 3:103–116
Jensen K, Bjerrum F, Hansen HJ, Petersen RH, Pedersen JH,
Konge L (2015) A new possibility in thoracoscopic virtual reality simulation training: development and testing of a novel virtual
reality simulator for video-assisted thoracoscopic surgery lobectomy. Interact Cardiovasc Thorac Surg 21:420–426. https​://doi.
org/10.1093/icvts​/ivv18​3
Jensen K, Ringsted C, Hansen HJ, Petersen RH, Konge L (2014)
Simulation-based training for thoracoscopic lobectomy: a randomized controlled trial: virtual-reality versus black-box simulation. Surg Endosc 28:1821–1829. https​://doi.org/10.1007/s0046​
4-013-3392-7
Savran MM, Hansen HJ, Petersen RH, Walker W, Schmid T,
Bojsen SR, Konge L (2015) Development and validation of a
theoretical test of proficiency for video-assisted thoracoscopic
13
27.
28.
29.
30.
31.
32.
33.
surgery (VATS) lobectomy. Surg Endosc 29:2598–2604. https​://
doi.org/10.1007/s0046​4-014-3975-y
Meyerson SL, LoCascio F, Balderson SS, D’Amico TA (2010)
An inexpensive, reproducible tissue simulator for teaching thoracoscopic lobectomy. Ann Thorac Surg 89:594–597. https​://doi.
org/10.1016/j.athor​acsur​.2009.07.067
Tong BC, Gustafson MR, Balderson SS, D’Amico TA, Meyerson
SL (2012) Validation of a thoracoscopic lobectomy simulator.
Eur J Cardiothorac Surg 42:364–369. https:​ //doi.org/10.1093/ejcts​
/ezs01​2
Miskovic D, Ni M, Wyles SM, Kennedy RH, Francis NK, Parvaiz
A, Cunningham C, Rockall TA, Gudgeon AM, Coleman MG,
Hanna GB (2013) Is competency assessment at the specialist level
achievable? A study for the national training programme in laparoscopic colorectal surgery in England. Ann Surg 257:476–482.
https​://doi.org/10.1097/SLA.0b013​e3182​75b72​a
Ilgen JS, Ma IWY, Hatala R, Cook DA (2015) A systematic
review of validity evidence for checklists versus global rating
scales in simulation-based assessment. Med Educ 49:161–173.
https​://doi.org/10.1111/medu.12621​
Sharma B, Mishra A, Aggarwal R, Grantcharov TP (2011) Nontechnical skills assessment in surgery. Surg Oncol 20:169–177.
https​://doi.org/10.1016/j.suron​c.2010.10.001
Barsuk JH, Cohen ER, Wayne DB, Siddall VJ, McGaghie WC
(2016) Developing a simulation-based mastery learning curriculum. Simul Healthc J Soc Simul Healthc 11:52–59. https​://doi.
org/10.1097/SIH.00000​00000​00012​0
Mcgaghie WC (2015) Mastery learning: it is time for medical
education to join the 21st century. Acad Med 90:1–4. https​://doi.
org/10.1097/ACM.00000​00000​00091​1