See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/338645987 Double-blind peer review of manuscripts: opportunities, challenges, and way forward Article in Physical Therapy Reviews · January 2020 DOI: 10.1080/10833196.2019.1698161 CITATIONS READS 2 120 3 authors: Ashokan Arumugam Poonam Mehta University of Sharjah University of Otago 83 PUBLICATIONS 255 CITATIONS 20 PUBLICATIONS 73 CITATIONS SEE PROFILE G D Baxter University of Otago 407 PUBLICATIONS 12,768 CITATIONS SEE PROFILE Some of the authors of this publication are also working on these related projects: Neuropathic pain View project Systematic reviews on knee joint propriocetion View project All content following this page was uploaded by Ashokan Arumugam on 03 July 2020. The user has requested enhancement of the downloaded file. SEE PROFILE Physical Therapy Reviews ISSN: 1083-3196 (Print) 1743-288X (Online) Journal homepage: https://www.tandfonline.com/loi/yptr20 Double-blind peer review of manuscripts: opportunities, challenges, and way forward Ashokan Arumugam, Poonam Mehta & G. David Baxter To cite this article: Ashokan Arumugam, Poonam Mehta & G. David Baxter (2020) Double-blind peer review of manuscripts: opportunities, challenges, and way forward, Physical Therapy Reviews, 25:1, 1-6, DOI: 10.1080/10833196.2019.1698161 To link to this article: https://doi.org/10.1080/10833196.2019.1698161 Published online: 14 Feb 2020. Submit your article to this journal Article views: 24 View related articles View Crossmark data Full Terms & Conditions of access and use can be found at https://www.tandfonline.com/action/journalInformation?journalCode=yptr20 PHYSICAL THERAPY REVIEWS 2020, VOL. 25, NO. 1, 1–6 https://doi.org/10.1080/10833196.2019.1698161 Double-blind peer review of manuscripts: opportunities, challenges, and way forward Peer review is a widely used and accepted process by which a scientific, academic, or professional work is evaluated by a group of experts in the relevant fields. Peer review has a long history: the first use of the process dates to the early seventeenth Century [1]. Even so, peer review has only been institutionalized since the 1940s [2] and utilized by most biomedical journals by the late twentieth century [3]. For academic journals, a peer review helps the editor(s) to decide about the suitability of the work for publication. Once a piece of work (manuscript) is submitted to a journal, the editorial board members assign two or more experts to review the paper and provide feedback on the scientific merits and suitability for publication. The editorial team scrutinizes and adjudicates on the reviewers’ comments, along with or without their own review of the work, and submits a decision (revise, accept, reject or resubmit). If the submitted manuscript meets the best scientific standards, it may be accepted unconditionally, without any revisions. In those cases where a submitted manuscript does not initially meet acceptable scientific, language and/or technical standards (in terms of originality, concept, design, grammar, syntax, punctuation, etc.), certain amendments and clarification will be requested from the authors before peer review. If the manuscript does not fall within the scope of a journal or have a high similarity index (plagiarism), it might be rejected at the first instance without any peer review. Overall, peer review remains the accepted ‘quality-control’ process that involves blinded or unmasked (open) scrutiny of the authors’ scholarly (research) work by other experts in the relevant fields to check its scientific validity and merit. In this editorial, we consider the opportunities and challenges of a double-blind peer review process which has been commonly employed by many physiotherapy journals. Specifically, we discuss whether mitigating the challenges in maintaining the anonymity of the authors is of paramount importance compared to relying on the quality of the review, irrespective of the success or failure in concealing the authors’ identities. Further, we have CONTACT Ashokan Arumugam aarumugam@sharjah.ac.ae; Sciences, University of Sharjah, United Arab Emirates. ß 2019 Informa UK Limited, trading as Taylor & Francis Group added a few recommendations to combat the challenges in peer review. Closed and open peer reviews Blinding is an overarching term for closed peer review used by many journals. Based on the level of blinding, closed peer review has been classified into three categories: single-blind, double-blind, and triple-blind [4]. As the term indicates, in single-blind reviews, the reviewers are aware of the authors’ details but not vice versa. In double-blind reviews, both the reviewers and authors are not aware of each other’s identity. With triple-blind reviews, the editors, reviewers, and authors remain anonymous to each other; this is claimed to best prevent bias stemming from any persons involved in deciding the fate of a manuscript. Open peer review, on the other hand, reveals the identity of the reviewers and authors to each other. This system aims at a greater transparency, accountability, and fairness [5], while acknowledging credit for peer reviewers (via publication of reviewer’s names/reports alongside the article) [6]. In addition, other types of peer review based on variables such as timing (critical appraisal of the manuscript post-publication), transfer (manuscripts with peer-review reports exchanged between subject-related journals), and collaboration (a team of reviewers review a manuscript together) have recently evolved (https://authorservices.wiley.com/Reviewers/journal-reviewers/whatis-peer-review/types-of-peer-review.html). Double-blind peer review process – opportunities and challenges Opportunities Closed peer review, principally double-blind peer review, is commonly used as part of academic publishing, with the stated aim of preventing bias in reviewing, e.g. reviewer bias in terms of authors’ expertise (novice vs. expert; early-career vs. senior), gender (male vs. female), language proficiency (native [English] speakers vs. non-native speakers), institution (research-intensive vs. teaching- ashokanpt@gmail.com Department of Physiotherapy, College of Health 2 GUEST EDITORIAL Table 1. A list of Physiotherapy Journals with a double-blind peer review policy. Journal Scimago Rank Journal of Orthopaedic and Sports Physical Therapy Musculoskeletal Science & Practice Journal of Physiotherapy Physiotherapy Journal of Neurologic Physical Therapy Journal of Geriatric Physical Therapy Physical Therapy Reviews Brazilian Journal of Physical Therapy Pediatric Physical Therapy Physical Therapy in Sport Physiotherapy Theory and Practice Physiotherapy Canada European Journal of Physiotherapy The Indian Journal of Physiotherapy and Occupational Therapy Journal of Women’s Health Physical Therapy Physiotherapy Quarterly Archives of Physiotherapy Q1 Q1 Q1 Q1 Q1 Q1 Q1 Q1 Q2 Q2 Q2 Q2 Q3 Q4 Q4 Q4 NA H Index 104 78 61 47 44 32 30 28 42 39 39 22 6 2 1 1 NA Abbreviations: NA – Not applicable; Q1-Q4 – Quartiles 1-4. This list is not exhaustive, but it highlights that many physiotherapy journals prefer a double-blind peer review policy for their manuscripts. intensive), research group (well-established vs. unrecognized), country (developed vs. developing/ underdeveloped; high vs. middle/low income), nationality (author’s vs. reviewer’s), religion, and/or political identity [4, 7–9]. Challenges Despite the intention of refereed journals to improve the content and quality of the scientific papers published, the peer review process continues to be criticized for various forms of biases [4, 10], parochialism [11], conservatism (reluctance towards accepting groundbreaking or innovative research) [10], the Hawthorne effect [12], excessive costs [13], the Matthew effect [14, 15] (supporting papers based on authors’ reputation rather than relying on the quality of those papers), a long turnaround time [10], nonconstructive or over-positive noncritical reviews [16], lack of adequate number of reviewers [16], and the inability of reviewers to detect academic fraud (e.g. plagiarism) [17] and/or methodological, statistical, and interpretation shortcomings [18]. Moreover, we must note that this system is not a fully objective process because it is based on reviewers’ opinions, which might differ based on the factors such as their subject knowledge, level of expertise, experience, judgements, any conflicts of interest, etc. [4]. Although reviewers provide a recommendation concerning the decision on a manuscript, the final decision on a manuscript should be solely determined at the discretion of the editorial team [4]. A survey conducted by Ceci and Peters (1984) revealed that three quarters of reviewers of some psychology journals were ignorant of the authors’ identities; furthermore, only 10% of those who were able to identify the authors were ‘highly confident’ in their guesses [19]. In contrast, Yankauer (1991) found that the reviewers of the American Journal of Public Health seem to correctly identify author and/ or institution around 40% of the time [20]. Blinding seems to be successful at times but not always. Furthermore, since these studies were published there has been a lot of changes in academic research that might significantly challenge the ability to adequately blind authors’ identities (see below). Prevalence of (implicit) biases, amongst others, in double-blind peer review might be attributed to the challenges in maintaining the anonymity of the authors to the reviewers. Though many physiotherapy journals (see Table 1) adopt a double-blind peer review policy, in concordance with most health journals, whether double-blinding is fully achieved remains ambiguous owing to several reasons. Salient issues include: Prior registration of systematic review and clinical trial protocols; Authors’ information publicly available on the institutional website/repository and other (research) social media platforms (ResearchGate, Google Scholar, LinkedIn, Academia, Twitter, Facebook, etc.); Publishing of theses/dissertations online through an institutional repository; Secondary analysis of previously published data; Systematic review updates; Multiple publications (e.g. publishing the findings of a longitudinal trial periodically), including so called ‘salami-sliced’ publications; Secondary publications (translation of previously published paper in one language to the other); A series of manuscripts (identified as part 1, 2, etc.) from the same author(s) on a single topic/study; Full trials following previously published feasibility or pilot studies; PHYSICAL THERAPY REVIEWS Abstracts published previously in conference proceedings; Mechanical detection (editorial staff failing to remove title page or acknowledgements before sending the manuscript for peer review, authors including personal acknowledgments or referring to their earlier work within the main body of text [19], and conflict of interest and financial disclosure unmasked in the manuscript owing to editorial staff oversight); Use of personalized (self-invented) techniques, procedures, and analytical methods [19]; (Discriminative) self-citations [20, 21]; Authors with a prolific publication history [21]; Personal knowledge (reviewers being aware of the researchers (authors) conducting studies in their own areas of expertise) [20]; Adding pre-prints of their publication in ResearchGate, institutional repositories, and other platforms; Authors suggesting their own (known) peer reviewers; Funding bodies publishing a list of research proposals/studies funded by them. Anonymity versus quality of peer review Current good research practice guidelines encourage a priori registration of clinical trials or systematic reviews (protocols) [22], which would obviously preclude blinding of the authors’ identity when they are subsequently publishing the findings of their completed trials or reviews. [23]. Anonymity may not be a major concern for many journals nowadays. If the reviewers are aware of, or able to detect, the identity of the authors, then it would enable the reviewers to compare the authors’ previous works and allow identification of some forms of plagiarism (duplication, salami-slicing, text-recycling, etc.). Some reviewers claim that knowing authors’ whereabouts and track record would aid in adjudicating on the manuscript [20]. In fact, much of the bias in peer review depends on the ethics and integrity of the reviewers, and recognizing their own biases including confirmatory bias. Not-withstanding this, arguably a more significant issue for journals is ensuring the quality of the reviews performed by the reviewers, rather than maintaining the anonymity of authors. Many a researcher, whether novice or expert, might well accept a request to review manuscripts even if the subject is out of their expertise. Some reasons might be – an acknowledgement of the reviewers’ names in the journal for which they reviewed, and that some publishers (e.g. Elsevier) reward their reviewers with an outstanding reviewer 3 award or a certificate of appreciation. If reviewers are not able to critique the manuscript adequately because of lack of time and/or expertise, then there are risks that papers when published may not be of good quality. Irrespective of blinding or unmasking authors’ details, it is anecdotally reported that early career/novice researchers frequently provide more extensive and critical feedback on submissions than their experienced colleagues. Why a researcher should serve as a peer reviewer? The peer review process would promote written expression of the reasoning skills, views, and perceptions, critical thinking ability, and promote professional responsibility of (novice) reviewers [24]. In particular, indulging in double-blind peer review would preclude the Hawthorne effect (modifying their comments/decision because they are being observed) and facilitate the expression of opinion by the reviewers. Through engagement in peer review, students have been found to demonstrate higher levels of metacognition (critical thinking, monitoring, planning, and/or regulation) [25]. Similar outcomes could be expected for peer reviewers of journals. Active participation in peer review would equip reviewers with required career experience that would improve the quality of their feedback as well as writing their future manuscripts emanating from their work [26]. Moreover, peer reviewing allows the reviewers to update their previous knowledge with the latest research evidence. Being a frequent reviewer for a journal might augment the chances of becoming an editorial board member of that journal (https://editorresources.taylorandfrancis.com/ reviewer-guidelines/). Ways to mitigate the challenges in peer review The most important challenge with the blinded peer review process is not whether to abandon or adopt it, but how to improve the quality of it. Some of the potential strategies that can improve the quality and/or outcomes of a closed or open peer review process are discussed below. Training and supporting reviewers In order to improve the quality of the research papers, some sort of formal training and mentoring should be provided to early career researchers or novice reviewers for appropriately evaluating important components of a manuscript and, 4 GUEST EDITORIAL specifically, the factors contributing to risk of bias in the findings of the study [27, 28]. Early career reviewers learn the process of reviewing either by trial and error or with the guidance of an experienced reviewer; for instance, PhD students mentored by their supervisors who might be experienced and/or skillful reviewers. The editors of “International Journal of Exercise Science” have published certain guidelines for students and academic professionals for peer reviewing in the field of Exercise science [29, 30]. Following such guidelines, undertaking some form of online training and support by research organizations or journals would enhance the peer reviewing skills and may mitigate the risks associated with poor peer review [31]. In addition, such training courses could provide continuing education units or credits [32]. Moreover, providing a detailed feedback on the peer review reports to the reviewers would help in improving the outcomes of the peer review process. Rigorous selection and availability of reviewers The goal of every editor is to find skilled and knowledgeable reviewers who will do a credible job of providing a critical yet constructive review of a manuscript. At the same time, early career reviewers might avail every opportunity to be a reviewer as they get the chance to review someone else’s work and provide a thoughtful feedback on it. Further, employing junior researchers such as early career researchers, postdoctoral researchers, research fellows, and graduate students will increase the available number of reviewers [26]. Despite selecting a reviewer based on their experience (novices or experts), the editors should choose them based on their qualifications, expertise, and areas of research interests matching with those of the scope of the manuscripts. It should be a mutual responsibility for both editors and reviewers to uphold the integrity of peer review process. This is essential in healthcare research because the findings of (peer reviewed) manuscripts may directly influence people’s lives and, hence, such research must be carefully scrutinized before publication [33]. Standardizing procedures and reporting guidelines To ensure the quality of reporting in healthcare research, many journals require the authors to follow one of the reporting guidelines addressing key methodological information relevant to the conduct of their study [34]. Similarly, peer review documented using a standardized reporting guideline or checklist would address the objectives, requirements, and outcomes of the process. This may be an important step to minimize the disparity in reviewers’ feedback. Quality checks To ensure the quality of the peer review report, using a valid tool among already established scales or checklists (n ¼ 24) [27] or routine editor’s subjective quality rating [35] could be employed. Through these methods, the peer review report can be assessed for its quality or sources of bias. Currently, none of the available risk of bias assessment or methodological quality rating tools of studies of different designs seem to include the type of peer review process and/or the quality of peer review reports of the article while assessing the overall quality of the studies. However, presenting the review quality scores in addition to the risk of bias or methodological quality scores of a paper may ensure the scientific soundness of the study. Time spent on peer review Considering the time constraints of academics and early career researchers, sufficient time should be allowed to review a manuscript and/or its subsequent revisions. From an editor’s perspective, to keep a quality control of review process, reviewers could be asked to report on how long they spent carrying out the review. There seems no association between the editors’ assessment of review quality and the time taken by reviewers to return their reviews [36]. However, we recommend keeping a record of time spent on the reviews as a quick feedback for the reviewers to understand if they have spent enough quality time in reviewing the article. Rewarding reviewers The editors should acknowledge that peer reviewers are volunteers who offer their services despite their regular work commitments. Without receiving any reward in return, performance of peer reviewers tends to deteriorate gradually [37]. Some journals might reward their reviewers’ by providing an appreciation certificate, publishing their names in an issue of their journal, allowing access to certain electronic databases for a few months, subsidizing (open access) article processing fee waiver for their future submissions, and allowing free access to the journal’s articles. It would be prudent to introduce a new metric or index to indicate the number of manuscripts reviewed annually by each reviewer PHYSICAL THERAPY REVIEWS [32]. Perhaps, such a metric might be considered as an adjunct in evaluation for academic promotions. For example, Publons is an online platform where researchers can maintain a verified record of their work, including their publications, citation metrics, and peer reviews with a unique ResearcherID (https://publons.com/about/home/). It offers a free webinar for training peer reviewers. Further, this platform allows its members to track and exhibit their research impact as an author, editor and/or peer reviewer. In summary, given the increased demand on the number of publications and submitted manuscripts these days, we acknowledge and appreciate the responsibility that an editorial team carries on to yield an honest, constructive yet challenging, and unbiased peer review process. To mitigate the challenges of a (double-blind) peer review process, amongst other strategies discussed above, perhaps quality checks and feedbacks on peer review reports seem very essential. Even so, alternate peer review processes such as open reviews could be adopted to gain more transparent and less biased review reports whenever blinding authors’ identities is impossible because of the issues we have discussed in this article. Open peer reviews could be a potentially useful strategy to promote a shared responsibility between authors, reviewers, and editors to uphold the integrity of the peer review process. Having said that, the success of this process would ultimately depend on the number of reviewers consenting to reveal their identities. There is a wide scope for conducting research on the pros and cons of open and closed peer review systems in journals publishing studies of physiotherapy, health sciences, and complementary and alternative medicine research. There might be a need for conferences focusing on the opportunities, challenges, and solutions for different peer review systems in these scientific disciplines. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. Disclosure statement None declared. 19. Ethical approval 20. Not required. 21. Funding 22. No internal or external funding was received for preparing the manuscript. 23. References 1. ELSEVIER: What is peer review? https://www.elsevier.com/reviewers/what-is-peer-review. 24. 5 Rennie D. Editorial peer review: its development and rationale In: Jefferson T, Godlee F, editors. Peer review in health sciences. 2nd ed. BMJ Books; 2003. Retrieved from https://www.wiley.com/en-us/Peer+Review+inþ Health+Sciences%2C+2nd+Edition-p-9780727916853 Benos DJ, Bashari E, Chaves JM, et al. The ups and downs of peer review. Adv Physiol Educ. 2007;31: 145–152. Ali PA, Watson R. Peer review and the publication process. Nurs Open. 2016;3:193–202. Van Rooyen S, Godlee F, Evans S, et al. Effect of open peer review on quality of reviews and on reviewers’ recommendations: a randomised trial. BMJ. 1999;318:23–27. Ross-Hellauer T. What is open peer review? A systematic review. F1000Res. 2017;6:588. Pinholster G. Journals and funders confront implicit bias in peer review. Science. 2016;352:1067–1068. Fox CW, Burns CS, Meyer JA. Editor and reviewer gender influence the peer review process but not peer review outcomes at an ecology journal. Funct Ecol. 2016;30:140–153. Smith R. Peer review: a flawed process at the heart of science and journals. J R Soc Med. 2006;99: 178–182. Lee CJ, Sugimoto CR, Zhang G, et al. Bias in peer review. J Am Soc Inf Sci Tec. 2013;64:2–17. Goldbeck-Wood S. Evidence on peer review-scientific quality control or smokescreen? BMJ (Clin Res ed). 1999;318:44–45. PubMed PMID: 9872890; eng. Van Rooyen S, Godlee F, Evans S, et al. Effect of blinding and unmasking on the quality of peer review. J Gen Intern Med. 1999;14:622–624. Smith R. Classical peer review: an empty gun. Breast Cancer Res. 2010;12:S13. Merton RK. The Matthew effect in science: The reward and communication systems of science are considered. Science. 1968;159:56–63. Birkmaier D, Wohlrabe K. The Matthew effect in economics reconsidered. J Informetrics. 2014;8: 880–889. Rocco TS, Collins JC. The nature of peer-review: Confessions of a formerly-good reviewer and observations of an emerging scholar. Taylor & Francis; 2012. Walker R, Rocha da Silva P. Emerging trends in peer review—a survey. Front Neurosci. 2015;9:169. Godlee F, Gale CR, Martyn CN. Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial. JAMA. 1998;280:237–240. Ceci SJ, Peters D. How blind is blind review? Am Psychol. 1984;39:1491–1494. Yankauer A. How blind is blind review?. Am J Public Health. 1991;81:843–845. Hill S, Provost F. The myth of the double-blind review? Author identification using only citations. SIGKDD Explor Newsl. 2003;5:179–184. PLoS-Medicine-Editors. Best practice in systematic reviews: the importance of protocols and registration. PLoS Med. 2011;8:e1001009. De Angelis C, Drazen JM, Frizelle FA, et al. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Circulation. 2005;111:1337–1338. Kern VM, Saraiva LM, dos Santos Pacheco RC. Peer review in education: promoting collaboration, 6 25. 26. 27. 28. 29. 30. 31. 32. 33. GUEST EDITORIAL written expression, critical thinking, and professional responsibility. Educ Inform Technol. 2003;8:37–46. Liu E-F, Lin SS, Chiu C-H, et al. Web-based peer review: the learner as both adapter and reviewer. IEEE Trans Edu. 2001;44:246–251. Donaldson MR, Hanson KC, Hasler CT, et al. Injecting youth into peer-review to increase its sustainability: a case study of ecology journals. IEE. 2010;3. Superchi C, Gonzalez JA, Sola I, et al. Tools used to assess the quality of peer review reports: a methodological systematic review. BMC Med Res Methodol. 2019;19:48. Schroter S, Black N, Evans S, et al. Effects of training on quality of peer review: randomised controlled trial. BMJ. 2004;328:673. Simpson KJ. Reviewing an original research manuscript for the international journal of exercise science: a guide for students and professionals. Int J Exercise Sci. 2008;1(2):43–49. Stone WJ, Navalta JW, Lyons TS, et al. From the editors: a guide for peer review in the field of exercise science. Int J Exercise Sci. 2018;11:1112. Freda MC, Kearney MH, Baggs JG, et al. Peer reviewer training and editor support: results from an international survey of nursing peer reviewers. J Prof Nurs. 2009;25:101–108. Ruiz JG, Candler C, Teasdale TA. Peer reviewing elearning: opportunities, challenges, and solutions. Acad Med. 2007;82:503–507. Hill CJ. The value of serving as a peer reviewer: an editorial. IJAHSP. 2018;16:8. View publication stats 34. 35. 36. 37. Hirst A, Altman DG. Are peer reviewers encouraged to use reporting guidelines? A survey of 116 health research journals. PLoS One. 2012;7:e35621. Callaham ML, Baxt WG, Waeckerle JF, et al. Reliability of editors’ subjective quality ratings of peer reviews of manuscripts. JAMA. 1998;280: 229–231. Black N, Van Rooyen S, Godlee F, et al. What makes a good reviewer and a good review for a general medical journal? JAMA. 1998;280:231–233. Callaham M, McCulloch C. Longitudinal trends in the performance of scientific peer reviewers. Ann Emerg Med. 2011;57:141–148. Ashokan Arumugam Department of Physiotherapy, College of Health Sciences, University of Sharjah, Sharjah, United Arab Emirates http://orcid.org/0000-0001-5795-3812 Poonam Mehta Graduate School of Health, Discipline of Physiotherapy, University of Technology Sydney, Sydney, NSW, Australia http://orcid.org/0000-0002-9307-9648 G. David Baxter Centre for Health, Activity and Rehabilitation Research (CHARR), School of Physiotherapy, University of Otago, Dunedin, New Zealand http://orcid.org/0000-0003-2968-6950