Introduction

The peer-review process serves the dual purposes of assisting editors with publication decisions and providing authors with constructive feedback [1–3]. Journals vary in their approach to peer review, particularly in decisions about revealing the identity of reviewers to authors and authors to reviewers [1, 4, 5]. For many journals, masking reviewer identities became the norm as a way to encourage honest critique without fear of retaliation from authors. However, the soundness of this approach has come into question as such masking may inadvertently encourage reviewers to be less conscientious about providing constructive and respectful feedback to authors [6]. To mitigate this concern, many journals provide guidelines for reviewers that set expectations for quality feedback [7, 8]. In addition, to enhance transparency and accountability, some journals have an open peer-review process that either requires or gives reviewers the option to share their identity and/or make their review publicly available [4, 9]. While such efforts to increase transparency are thought to improve review quality by increasing reviewers’ sense of accountability, a series of studies conducted by the BMJ found no difference in the quality of masked versus unmasked peer reviews [10–12]. The only statistically significant finding was that reviewers were more likely to decline the invitation to review if they knew their identity would be revealed to authors [11, 12]. An important caveat in the transparency hypothesis is that most journals provide space for reviewers to provide confidential comments to the editor, which effectively allows authors to express concerns they feel uncomfortable sharing with authors. Reviewers’ decision recommendations are also often not shared with authors. Concerns can arise among authors when confidential comments and recommendations to the editor result in an editor’s decision to reject a manuscript and the rationale for the decision is not shared with the authors.

Reviewer guidelines provide variable guidance about how to use the confidential comments to the editor section. The Committee On Publication Ethics (COPE) ethical guidelines for peer reviewers [13] advises reviewers to “ensure your confidential comments and recommendations for the editor are consistent with your report for the authors; most feedback should be put in the report that the authors will see.” The guidelines further remind reviewers that these comments “should not be a place for denigration or false accusation, done in the knowledge that the authors will not see your comments” [13 p.4]. The journal Academic Medicine provides a comprehensive guide for reviewers of research manuscripts that describes how reviewers can use that section [14]. The guide suggests using the confidential comments to the editor to:

“recommend additional review by someone with specific expertise, make specific comments on the quality of the manuscript, provide opinions about the relevance or significance of the work, or raise potential ethical concerns. The reviewer can also use this space to give the editor more nuanced and detailed information and to explain the severity of any problems detected in the manuscript, along with the likelihood that the authors can address the problems through a revision. The reviewer should ensure that his or her confidential comments to the editor do not contradict the comments directed toward the authors and that they provide information that is relevant only to the editor.” [14 p.10]

By contrast, PLOS One explicitly prohibits confidential comments to the editor and instructs authors to only use this section to declare competing interests [15]. “Confidential concerns relating to publication or research ethics” are handled outside the peer review system, through email to the journal [15]. Despite these guidelines, we know little about how reviewers use this section, which leaves a crucial gap in understanding transparency and accountability in peer review.

From authors’ perspectives, submitting a manuscript for publication is, at minimum, an opportunity to receive feedback to improve their work. However, for reasons described above, this feedback can be confusing or unhelpful when the feedback received does not align with an editor’s decision to reject the manuscript or offers little insight into their decision. This lack of transparency and alignment warrants further consideration for two reasons. First, honest and constructive feedback from reviewers plays a key role in improvement of scholarship. Second, non-transparency (whether warranted or not) raises doubts about the legitimacy and fairness of the peer-review process and suggests an area for process improvement.

To better understand issues related to transparency and quality feedback in peer review, we examined how reviewers use the confidential comments section of an online peer-review system. Our specific research questions are:

1. How do reviewers use the confidential comments to the editor section during peer review?
2. To what extent does the content and tone of reviewers’ comments to the editor align with their comments to the authors when reviewers make a “reject” recommendation?

Materials and methods

Data analysis

We conducted a combination of conventional and directed content analysis on reviewer comments [16]. To reflect the purpose and content covered in confidential comments to the editor, we developed a coding scheme based on studies of peer review [6, 17, 18] and guidelines for peer review (e.g., COPE Guidelines [13], Academic Medicine review guide[8]). Our primary codes covered content, section of the manuscript, tone, and alignment between the content and tone of comments to the authors and editors when the reviewer recommended “reject.” We added sub-categories as needed via conventional content analysis (deriving categories from the data). Content codes included overall verdict, quality of argument, structure and language, design and methods, reference to comments to authors, summary of manuscript, and other. We also coded the section of the manuscript referenced in the comments. We coded the tone of comments as critical, constructive, supportive, or neutral and noted comments in which the reviewer addressed the editor by name or in a way that conveyed a familiar tone.

To evaluate the alignment between reviewers’ comments to the editor and comments to authors, we restricted our analysis to reviews in which the reviewer recommended “reject,” as we anticipated that these reviews had the greatest potential for discrepancies between what the reviewers wrote to the authors and to the editors. We coded these reviews for alignment of content and tone. Content was considered not to align if the reviewers mentioned concerns about the manuscript to the editor that they did not mention to the authors. We expected that comments to authors would cover more content than comments to editors and did not consider this a lack of alignment. Tone was based on overall tone. We coded all reviews that did not contain comments to the editor as aligned with comments to the author. This approach allowed us to calculate the overall incidence of alignment in reviews where reviewers recommended rejecting the manuscript.

To facilitate coding, BCO created an initial code book as informed by the above. BCO, LAM, ARA, ED then each independently coded 15 reviews and then met to discuss their coding, reconcile differences and adjust the code book. Using the revised code book, BCO and LAM independently each coded an additional 35 reviews then met to discuss and revise the code book. The remaining reviews were then assigned to BCO, LAM, ARA and JAC such that each review was coded by two authors. BCO and LAM met with all authors to reconcile codes by discussion.

We calculated descriptive statistics for a) key features of the articles in our review (Table 1) and b) key features of reviews with and without comments to the editor (Table 2), and c) characteristics of reviews with comments to the editor, including: content (Table 3), section of the manuscript, and tone (Table 4). Our study is descriptive, so we did not conduct statistical tests to compare reviews with and without comments to the editor. Rather, we provide this information to situate our findings in context.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Descriptive statistics for 168 articles submitted with completed reviews for Perspectives on Medical Education from January 1, 2019 to August 24, 2020.

https://doi.org/10.1371/journal.pone.0260558.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Descriptive statistics for 358 reviews of 168 articles submitted to Perspectives on Medical Education in 2019 or 2020, by reviews with comments to editor and reviews with no comments to editor.

https://doi.org/10.1371/journal.pone.0260558.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Content of comments: Categories with frequency, description, and example comments for each category.

https://doi.org/10.1371/journal.pone.0260558.t003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Tone of comments: Categories with frequency, description, and example comments for each category.

https://doi.org/10.1371/journal.pone.0260558.t004

Results

Our final dataset included 358 reviews of 168 articles submitted and peer reviewed from January 1, 2019 to August 24, 2020. Each article had 1 to 5 reviews. Articles with only one review reflect situations in which the article was still under review and awaiting an additional reviewer when we extracted the data or rare occasions when an associate editor provided a second review (not captured in the database) due to difficulty recruiting reviewers. The majority of articles were original research articles (65%) and most articles had 2 or more reviews (2.1 reviews, on average). Among the 144 articles with a final decision from the editor, more than half were rejected (88/144, 61%). Table 1 displays descriptive information on included articles.

Discussion

Our study sought to better understand issues related to transparency in peer review by examining how reviewers use the confidential comments to editors. Our analysis shows that roughly half of reviews contained no additional comments to the editor. Among those that did, most were brief and summarized the reviewers’ overall evaluation of the manuscript, often referencing their comments to the authors. Comments most often addressed the quality of the arguments and aspects of the research design, methods, or data. The vast majority of comments to the editor had a critical or constructive tone, which roughly aligns with the decision terms recommended by reviewers. In reviews that recommended rejecting the manuscript, the content aligned more often than not and, in many cases, the additional content was not related to the quality of the manuscript. Tone also aligned in most of these comments.

Overall our findings suggest that most reviewers are transparent in their reviews, sharing their feedback with authors even when they judge the manuscript unsuitable for publication and rarely making additional comments to the editor that raise concerns not shared with the authors. These findings add an important piece of information to the current literature on innovations in peer review. A number of journals have shifted to processes that unmask reviewers’ identity as either an optional or required part of peer review [19]. Studies examining reviewers’ preferences for sharing their identity with authors [20–22] and the effect of such signed reviews on the quality, tone, and time spent on reviews have shown mixed effects [10, 11, 22–24], perhaps reflecting different norms across different fields and reviewer pools. None of these studies indicate whether these unmasked reviewers can write confidential comments to editors. This option provides a mechanism for reviewers to raise concerns regarding ethical issues that warrant further investigation by editors (e.g., plagiarism, duplicate publication) via a confidential channel that likely feels more comfortable to reviewers. Yet this option means that a piece of the peer-review process remains invisible to authors, which could fuel suspicion about bias in the review process. Findings from our exploratory study suggest such suspicions may be unwarranted. Some journals aim to alleviate such suspicions by instructing reviewers to use the comments to editors for a limited set of purposes such as concerns about conflict of interest or unethical breeches [15]. We found very few instances of comments for these purposes, suggesting that such guidance would likely result in infrequent use of confidential comments to editors.

The journal we selected provided minimal guidance to reviewers for how to use the confidential comments to editors, which gives us insight into reviewers’ interpretations and potential norms in the field about how one might use this section. That said, the diversity of comments included in this section, and the fact that 51% of reviewers did not use the confidential comments area, suggests that journals might benefit from giving clearer guidance or training to their reviewer pool. Many journals provide guidance and checklists to reviewers, the contents of which were catalogued in a recent scoping review of biomedical journal peer review guides [7]. Unfortunately, this review did not report the number of guides that included content about how to use the confidential comments section, but may be worth exploring to understand the norms around use of this section and to help standardize language across peer review guides.

Our findings revealed some potential benefits of confidential comments to editors that warrant consideration. Lee and colleagues characterize peer review as an inherently social and partial process—one that cannot and arguably should not attempt to be objective [25]. Our findings align with this characterization and reveal ways that reviewers use these comments to signal engagement in the scholarly community and reflect on their positionality. We found reviewers using this section to express gratitude for the opportunity to review the manuscript and willingness to review a revision. Editors may find value in these expressions of goodwill and commitment to the community. Reviewers also showed signs of self-awareness and desire for improvement by noting concern about the tone of their review and indicating a desire for feedback. Such comments reveal reviewers’ thoughtfulness in crafting reviews and may suggest a way to reward and sustain their efforts by giving them ways to hone their skills and learn from editors. Reviewers also occasionally used the confidential comments section to alert editors to their perspective on a topic or limitations regarding evaluation of the content or methodology. Such information may provide helpful context to editors, particularly when needing to make a decision based on disparate reviews. Eliminating or restricting use of this section without providing alternative means and guidance about how best to communicate such information to editors might reduce opportunities for reviewers to communicate with editors in ways that enrich relationships in what can otherwise feel like an impersonal process.

Several articles have described various types of bias and partiality that may apply to peer review, including bias for or against the author based on personal characteristics (race, ethnicity, gender, status) or emotions toward the author (e.g. sympathy for their situation), the status or prestige of the author’s institution, and the ideology, philosophical orientation, or methodology put forth by the author [25–27]. We did not evaluate comments specifically for such biases and partiality, though our coding of “other” content could have captured some of this content. Since authors and institutions were deidentified in our dataset, our ability to evaluate this content was limited. That said, this type of analysis could be a fruitful direction for future research, particularly to understand when reviewer partiality is problematic and how comments to the editor may help or hinder awareness of such partiality.

Our findings should be considered in light of several study limitations. First, we analyzed comments from a single journal within the field of medical education. It is possible that in another medical education journal, or one outside of the field, comments to the editor differ from those analyzed here, which limits the generalizability of our findings. Second, we focused on determining alignment between comments to the author and editor for those manuscripts that reviewers recommended rejecting. Future research should consider exploring alignment for those manuscripts recommended to accept and revise. Despite the fact that the majority of comments were written by authors located in English-speaking countries, this study also included comments from reviewers for which English was likely not their first language. This point could have introduced subtle differences in the comments that our predominantly North American research team did not detect. Therefore, we encourage investigators to consider broadening their research team in future work, or engaging with stakeholders from the reviewers’ countries to obtain feedback on the findings.

Conclusion

Findings from our exploratory study indicate that nearly half of reviews contain no confidential comments to the editor and those that do use them in a variety of ways. Most comments included a summary or rationale for the reviewer’s overall evaluation of the manuscript and carried a critical or constructive tone. Reviewers’ commitment to transparency, based on lack of use of confidential comments and alignment between the content and tone of reviewers’ comments to editors and authors, appears to be strong and in many cases lack of alignment in content reflected concerns unrelated to the manuscript or circumstances the reviewer felt important call to the editors’ attention. These findings suggest opportunities for further investigation and discussion about the pros and cons of retaining or clarifying the purpose of confidential comments to the editors as part of peer review.

Acknowledgments

Disclaimer: The views expressed in this article are those of the authors and do not necessarily reflect the official policy or position of the Uniformed Services University of the Health Sciences, the Department of Defense, or the U.S. Government.