Response to reviewer feedback for systematic review submitted to BMC HSR:
Title: Systematic review of safety checklists for use by medical care teams in acute hospital settings –
limited evidence pertaining to their effectiveness.
Authors: Henry Ko, Tari Turner, Monica Finnigan.
Dear BMC HSR Editorial Office,
Please find our response and changes to our systematic review submitted to your journal. All
changes have been tracked within the article. We have also make some discretionary changes within
sentences to help make sentences clearer in meaning. We explain our changes made below:
Reviewer: Christian Wulff
ABSTRACT:
- Major Compulsory Revisions: None
- Minor Essential Revisions
1. I recommend that you use the same checklist-term throughout the paper. – For example you use
the term ‘safety checklist’ in the Abstract’s Background, but ‘medical checklists’ in the Abstract’s
Results. Please be consistent to not confuse the readers. Else please state that the two terms are
synonymous.
AUTHORS CORRECTIONS: Amended as suggested throughout the document. All terms now say
“safety checklist”. Changes are highlighted in yellow.
2. Methods: I recommend that you first state the databases you searched and then list the limits (E.g.
“We searched the databases [X,Y,Z…] for [type of papers / studies?] published in English before
September 2009.”).
AUTHORS CORRECTIONS: Amended as suggested. See first sentence and HK5.
3. Methods: I do not know “All EBM”. I suppose that “All EBM” is an acronym for some evidence
based search-machine? Please write out.
AUTHORS CORRECTIONS: See HK4. We have corrected this to read “Cochrane Library”.
4. Methods: I recommend that you state the different possible levels of “quality” or “risk of bias”
and then use only one of these terms (i.e. either use ‘high/ moderate quality’ or ‘moderate / high
risk of bias’ in the Abstract’s Results section.
AUTHORS CORRECTIONS: We felt this was not appropriate to state in the abstract. We have not seen
different possible levels of quality of risk of bias stated in the abstract methods section of systematic
reviews.
5. Results: I recommend that you delete the first (negatively worded) sentence (“We were unable […]
of medical checklists.”) and add the study designs of the nine included studies to the following
sentence. – I think it is sufficient, that you end the Abstract’s Conclusion by calling for more studies,
especially high level studies as randomized controlled trials.
AUTHORS CORRECTIONS: We assume CW mean this correction for the Conclusions section. We
deleted as suggested. See HK11.
- Discretionary Revisions
6. Question to Background-section: Is it important to particularly list ‘junior medical staff’? - At least
in Denmark (where I come from), most front-line doctors are ‘junior doctors’. Likewise many frontline nurses are ‘junior nurses’. I believe it is more important to clarify who fill in the safety checklist
(doctors, nurses (or both), and maybe other health care personnel).
AUTHORS CORRECTIONS: We deleted “junior medical staff” from the document.
BACKGROUND:
- Major Compulsory Revisions: None
- Minor Essential Revisions
1. I propose that you insert a definition of ‘safety checklist’ and state synonymous terms (‘medical
checklist’?) if you want to use such. – Maybe mention a few word about whether most safety
checklists are paper-and-pencil or electronic or maybe?.
AUTHORS CORRECTIONS: Amended as suggested. Added a short definition. See HK14.
- Discretionary Revisions
2. “Checklists have seen improved safety outcomes…..”: Is the word “seen” correct? (My mothertongue is not English…)
AUTHORS CORRECTIONS: Amended as suggested. Inserted “might contribute to”. See HK12.
METHODS:
- Major Compulsory Revisions
1. I believe that the section on ‘Protocol registration’ has to be moved to the end of the paper.
AUTHORS CORRECTIONS: Left as is. PRISMA checklist requires protocol the registration section
within the methods section.
2. Search Strategy: I think it is important that you go from the general to the specific (see also
Abstract,2.). I propose starting with something like: “We searched the databases [X,Y,Z…] for [type of
papers?] published in English before September 2009. The search in MEDLINE used ……”.
AUTHORS CORRECTIONS: Amended as suggested. See HK16, HK 17, and HK18.
3. Page 4, line 1: I do not understand the purpose of “Validated search filters were used to separate
systematic reviews and clinical trials from other types of studies.”? - I suppose you used the
systematic reviews to find papers reporting clinical trials / comparative studies on your topic?
AUTHORS CORRECTIONS: Explained what the validated filters were. See HK19.
4. Please add a reference to the ‘critical appraisal questions’ sentence in the “Quality assessment”section and Table 1.
AUTHORS CORRECTIONS: Added link to CCE website. See HK26.
5. I think it is very important that you state which criteria (and cut-offs) you used to deem whether
studies were of low / moderate / high risk of bias. – Did you use all questions listed in Table 1 or only
those presented in Table 3? Moreover how did you select the Quality Assessment items to be
included in Table 3?
AUTHORS CORRECTIONS: Inserted a brief explanation of our assessment method. See HK28.
- Minor Essential Revisions
6. Inclusion and exclusion criteria: Similar to Abstract’s note 6.: Is it important to particularly list
‘junior medical staff’?
AUTHORS CORRECTIONS: Deleted “junior medical staff”.
7. Quality assessment: Grammar: I believe “…is intended…”, “…will be performed…” and “…will
attempt…” should be written in past tense, i.e. “…was intended…”, “…was performed…” etc.?
AUTHORS CORRECTIONS: Amended as suggested. All changed to past tense throughout the
document.
RESULTS:
- Major Compulsory Revisions
1. Search results: I don’t understand the partitioning of the different study types. I think this
presentation confuses the readers. – I.e. Initially I thought that N=5881+684+3297=9862 could be
found in Figure 1? I suppose this is not the case because of duplicates?
AUTHORS CORRECTIONS: Clarified the wording for this part. See HK31.
2. Table 3 (Study quality): (See also Methods 5.) I do not understand the selection of the quality
assessment items. I do not understand what you mean by “Attributable to intervention”? To
understand how you deem whether the studies have moderate or high risk of bias, I recommend you
to add a sort of score column and that you state ‘cut-off’ scores in the Table’s legend.
AUTHORS CORRECTIONS: Amended as suggested. Changed to "Outcome attributable to
intervention" to make it clearer what we mean in Table 3.
- Minor Essential Revisions
3. Study findings: The sentence: “The key results from individual studies are summarised by clinical
setting in Table 4.” – One could add: “And also below in the text”, because you also go through the
results in the text. I find the exposition of the findings in the text (page 5 from “ICU setting” to the
end of the Result section at page 8) much too comprehensive. Instead of going through each of the
studies’ outcomes, you should consider presenting only the summarised findings in the text (setting
by setting as now). Maybe you could present the precise outcomes in a separate Table, to be
uploaded as an additional file (meant for readers particularly interested in the field).
AUTHORS CORRECTIONS: We have shorted the results section slightly. However we feel we have
both summarised the results for each setting as well as briefly noted key outcomes for each study as
befitting a systematic review. See HK35 for one of the added sentences as suggested by CW.
4. Table 4: I think that it would help the readers’ interpretation if you add a column between Setting
and Results where you state the number of studies.
AUTHORS CORRECTIONS: Amended table as suggested.
DISCUSSION:
- Major Compulsory Revisions: None
- Minor Essential Revisions
1. I think you should delete the first (negatively loaded) sentence in the Discussion (page 8): “There
is a lack […] patient safety.”
AUTHORS CORRECTIONS: Amended as suggested.
2. I do not think that the sentence (page 10, line 9) is very explicit: “There were differences between
treatment groups for outcome measurement periods.” Regarding what?
AUTHORS CORRECTIONS: Amended as suggested. Reworded to be clearer in meaning. See HK51.
- Discretionary Revisions
2. The sentence (Page 8, line 17) “There is a lack…”: “Is” should be changed to ‘was’ (?)
AUTHORS CORRECTIONS: Amended as suggested.
3. Page 9, line 1: “…loads of staff, level of training, or other factors…” I believe “or” should be
changed to “and”.
AUTHORS CORRECTIONS: Amended as suggested.
4. Note to page 10, l 6: “In most studies the checklist itself was not validated prior to
implementation. For example, one study states that “it is not clear that each element of the checklist
needed to be there.”” I believe this point is very important and could be elaborated. The overriding
premise for implementing checklists is that all check items are based on best evidence of
effectiveness!
AUTHORS CORRECTIONS: Amended as suggested. Added sentence. See HK50 for added sentences
about validation of checklists.
CONCLUSIONS:
No comments.
Reviewer: Doug Elliott.
Major Compulsory Revisions
There are no revisions required in this category.
Minor Essential Revisions
Although alluded to in the discussion, I suggest that a specific sub-section on ‘recommendations for
practice’ (equivalent to ‘Implications for further research’) would be of additional benefit for readers
(despite the low level evidence)
AUTHORS CORRECTIONS: Amended as suggested. See HK59.
Discretionary Revisions
My suggestions are:
Page / paragraph / line
6 / 1 / 5: cite references where a reduction in ICU LOS was observed
AUTHORS CORRECTIONS: Inserted references as suggested.
6 / 1 / last 5 lines: these two sentences (studies about reduced LOS) can be re-located to before
‘Agarwal’ (6 / 1 / 6) to improve flow of argument, and address the above comment
AUTHORS CORRECTIONS: Amended as suggested.
Consider a re-format percentages from ‘97% to 100%’ to ’97-100%’ to improve readability; e.g. 6 / 1
/ 15-17
AUTHORS CORRECTIONS: Not changed. We feel that keeping the “to” avoids confusion with a minus
sign.
6 / 2: synthesis of study findings can be improved with further editing, and reducing the ‘strongauthor’ orientation to sentence structure in this paragraph
AUTHORS CORRECTIONS: Amended as suggested. Made various changes as highlighted in the text to
the Results section to make it flow better.
7 / 1 / 1-2: depending on the above comment, there may need to be some elaboration of the
methods and findings from Romangnuolo in this sentence
AUTHORS CORRECTIONS: Amended as suggested. See HK38.
8 / 1 / 5: assume this was ‘appropriate’ antibiotics
AUTHORS CORRECTIONS: Amended as suggested.
17-Table 2: narrative could be further edited to reduce superfluous terms and resulting white space
AUTHORS CORRECTIONS: Amended as suggested. Made various changes to sentence structure and
words to make section flow better.
Reviewer: Karena Hewson
• Major Compulsory Revisions:
Abstract:
1. At 433 words the abstract is too long (author guidelines state maximum is 350 words). Needs to
be more succinct – particularly the results (i.e. stick to key findings) and conclusion (should only be
1-2 sentences max.). Examples of where words could be cut down include:
- Consider deleting 5th sentence under ‘Results’ section i.e. ‘ Measured outcomes were diverse...
etc.’
- Consider deleting 7th sentence under ‘Results’ section i.e. ‘This means the results of the
studies...etc.’
- 2nd sentence under ‘Conclusions’ section i.e. ‘The included studies suggest...’
– don’t need to repeat the level of bias, keep in results. A better sentence may read something like –
‘The included studies suggest some benefits of using medical checklists to improve protocol
adherence and patient safety, but due to the risk of bias evidenced, they should be interpreted with
caution.’
AUTHORS CORRECTIONS: All amended as suggested. Abstract now 321 words.
Discussion:
2. Need to state the limitations of this systematic review.
AUTHORS CORRECTIONS: Amended as suggested. Section added on p10 and p11 entitled “Strengths
and limitations of this systematic review.”
• Minor Essential Revisions:
Title:
3. Include the key finding into the title of the paper.
AUTHORS CORRECTIONS: Amended as suggested. See HK1.
Method:
4. What are the exclusion criteria? It appears that multi-faceted interventions such as those used by
DuBose et al (2008) and Wall et al (2005) were excluded. The exclusion criteria need to be clearly
stated under the ‘Inclusion and exclusion criteria’ heading.
AUTHORS CORRECTIONS: Amended as suggested. See all highlight amendments in the “Inclusion and
exclusion criteria” section.
5. The process of selecting and appraising the studies is unclear. Did both reviewers independently
select and appraise each of the studies? Or did one select and both did the appraisals? How were
the two appraisals brought together? How were any conflicts dealt with?
AUTHORS CORRECTIONS: Clarified in the “Inclusion and exclusion criteria” section.
6. The last sentence under the ‘Quality assessment’ heading is long and a bit unclear – would suggest
that authors review for clarity.
AUTHORS CORRECTIONS: Shortened and clarified the meaning of the heading.
Results:
7. Re: sentence ‘From all the found articles, 224 full text articles were retrieved for review’ - Please
clarify whether articles that were available in full-text on the database were the only ones retrieved?
Or did you obtain articles that fit the criteria regardless of whether they were available online or not
e.g. through library services?
AUTHORS CORRECTIONS: Clarified. Reworded the “Search results” section to make this clearer.
8. I strongly recommend noting the difference in how LOS was calculated & evaluated across the
different studies – particularly in ICU settings where multiple studies all have a different way of
looking at LOS. For example, Agarwal showed LOS based on a calculation of the number of days a
patient had been in the ICU at midnight which would exclude those patients discharged during the
day (i.e. a very crude and far from accurate measure). This could be done in the results section (i.e.
text) and/or in Table 2.
AUTHORS CORRECTIONS: LOS issue addressed briefly in the Discussion. Our changed from KW’s
suggestions also partially address this.
9. It may also be worth mentioning the problem with calculating LOS as a mean e.g. Narasimhan
assumes LOS is normally distributed, though this is often not the case – particularly for ICUs (i.e.
positive skew due to short stays).
AUTHORS CORRECTIONS: LOS issue addressed briefly in the Discussion. Our changed from KW’s
suggestions also partially address this.
10. Pronovost only evaluated LOS from implementation of the daily goals sheet onwards, so didn’t
really even have a historical control for comparison. I think this could be noted somewhere.
AUTHORS CORRECTIONS: LOS issue addressed briefly in the Discussion. Our changed from KW’s
suggestions also partially address this.
11. In Table 2, under the ‘Intervention’ column for the Weingarten (2004) study, authors write that
‘it was unclear if the interventions were the same across hospitals’. I conversely think it was clear
that the interventions were NOT the same across hospitals. On page 161 of the Weingarten paper
there are two tables summarising the number of hospitals by number of interventions implemented
and the top 10 interventions and intervention combinations. The intervention pertaining to
checklists also included other types of forms and reminders. The authors acknowledge that they
were not prescriptive on content, method or implementation strategies, and that hospitals chose
the intervention strategy that was most appropriate for their institution. I suggest altering the
wording to more accurately reflect the nature of the interventions used in this study.
AUTHORS CORRECTIONS: Amended as suggested. Changed sentence in Table 2.
References:
12. Reference no. 4 is the incorrect reference – I believe it is supposed to be ref. 13 instead.
AUTHORS CORRECTIONS: Updated reference list.
13. Reference no. 12 appears only on p. 10 in the discussion section – it has not been included
elsewhere in the paper e.g. results of review, so not sure if it is meant to be there.
AUTHORS CORRECTIONS: Updated reference list.
14. Reference list and in-text referencing needs to be fixed in light of the above revisions.
AUTHORS CORRECTIONS: Updated referencing.
• Discretionary Revisions:
Method:
15. I think ‘All EBM’ should be ‘All EBM Reviews’
AUTHORS CORRECTIONS: Changed to “Cochrane Library”.
16. In the first sentence under ‘Inclusion and exclusion criteria’, suggest using ‘included all’ instead
of ‘was’.
AUTHORS CORRECTIONS: Amended as suggested.
17. Second sentence under ‘Inclusion and exclusion criteria’ needs improving. Suggest ‘The
intervention was care given with the use of safety checklists that addressed safety concerns, which
were applied to patients by medical care teams, including junior medical staff.’
AUTHORS CORRECTIONS: Changed as suggested by CW. No JMO term used.
18. Third sentence under ‘Inclusion and exclusion criteria’ – suggest use ‘provided’ instead of ‘given’.
AUTHORS CORRECTIONS: Amended as suggested.
19. I’m not sure if the statement in ‘Missing data’ is relevant, or whether it needs to be made clearer
as to what sort of data you sought from the authors of included studies.
AUTHORS CORRECTIONS: A short sentence is included to clarify what was done if there was “missing
data”.
Results:
20. Try to avoid repeating the phrase ‘study types’ twice within the same sentence (see first
sentence under ‘search results’.
AUTHORS CORRECTIONS: Deleted some words to make it flow better.
21. Suggest following changes to 7th sentence under ‘search results’ i.e. ‘Four clinical settings were
covered, including five studies in the intensive care unit (ICU)’.
AUTHORS CORRECTIONS: Amended to make sentences flow better.
Discussion:
22. I wonder whether the point around validation of checklists requires a little more discussion,
particularly on the weaknesses of the studies that had attempted some validation, and what is
required of further studies (and mention this more clearly under ‘implications for further research’)?
Could refer to a commentary written about checklist development – see Winters et al Crit Care 2009
(v13: 210)
AUTHORS CORRECTIONS: Amended as suggested. See changes that were made from DE’s
suggestions for implications for practice section.
23. As per my comments re: LOS in results section, I think LOS requires further discussion –
particularly the impact of checklists on patient outcomes and whether this is an adequate measure
of quality improvement. There is some mention of this on p.10, but I think it is pretty clear that
outcomes were NOT uniformly defined across the studies. There are arguably better markers of
patient safety than LOS.
AUTHORS CORRECTIONS: Amended as suggested. See HK 47 to HK49, plus other corrections to LOS
outcomes from other reviewer corrections. Only a short sentence was used to discuss LOS as this
topic probably deserves a whole article on its own to do it justice.
24. Re: sentence ‘More RCTs are needed in this area to increase the level of evidence’. Are the
authors suggesting that RCT study designs are the only acceptable way of studying this topic, or are
other robust study designs/methods also called for? See Jean Louis Vincent’s commentary in Crit
Care Med 2010, v.38(10) for example.
AUTHORS CORRECTIONS: Removed the reference to RCTs. We felt the call for more high quality
research was adequate and understandable.
25. Last complete sentence on p.8 ‘This may suggest that the setting...’ etc – is unclear, suggest that
authors revise.
AUTHORS CORRECTIONS: Changed to make clearer.
26. Re: 5th sentence in the second paragraph on p.9 ‘It is uncertain if other factors (e.g. new policy
directives) could have influenced the behaviour of staff in caring for patients.’ – perhaps also
consider the impact of the unit’s safety culture (see Bryan Sexton & Peter Pronovost’s work in ICUs).
AUTHORS CORRECTIONS: Changed to make clearer.
27. The 6th sentence in the second paragraph on p.9 is unclear, consider revising.
AUTHORS CORRECTIONS: Changed to make clearer.
28. Thought the ‘Implications for further research’ section could be stronger. Try to address the
limitations in the studies reviewed e.g. outcome measurement, link between process and outcomes
evaluated, ensuring adequate controls, patient selection, study designs, evaluation of checklist use,
implementation model used, studies that include adequate validation work and evaluate the impact
of extraneous variables such as safety culture of the unit and policy changes. Also consider
incorporating long term evaluation and sustainability under this section.
AUTHORS CORRECTIONS: Addressed throughout the discussion, plus addressed via other reviewers
suggested changes.
29. Not sure of the value add re: refs 16 and 17 to implications for further research. The Wieser
(2010) study appears to be a sub-study of the Haynes (2009) one. And the descriptive paper by Lyons
does little to strengthen the argument for better study designs (retrospective or prospective).
Consider deleting and replace with directions for further research (as per previous comment).
AUTHORS CORRECTIONS: Changed to make clearer.
30. Consider writing a lead-in statement pertaining to the use of technology in the clinical setting
prior to the future research directions of Southern Health, as it currently reads as an after-thought
rather than a part of the research plan or process.
AUTHORS CORRECTIONS: Changed to make clearer.
We would like to thank the reviewers for their time and effort in providing feedback on our
systematic review, and in improving the content of it. We thank the BMC HSR editorial office for
their timely coordination of the review process.
Kind Regards,
Dr Henry Ko.