Editor survey

The editor survey was designed to identify journals that have a TOP Level II or TOP Level III policy in place and, for those particular journals, to collect data on how these policies are communicated to authors and reviewers; whether or not the policy is enforced and by what means; and challenges and strategies of successful policy implementation. The survey instrument included a combination of multiple choice, Likert scale, and open-ended questions, with programmed skip patterns enabled to allow for omission of question items not applicable to certain respondents (see S1 Appendix). The survey was administered using the Qualtrics web survey platform.

The editor survey sample was generated by selecting the top 250 ranked journals in the biological, health, and social sciences subject domain categories according to their Eigenfactor scores published in the 2016 InCites Journal Citation Report [17]. Inclusion of titles in the three specified subject domains required selection of InCites categories that corresponded with the three broader study-specified domains. In some cases, journal titles appeared in more than one category. For these journals, we made a determination of which domain the journal should be placed based on a closer examination of the disciplinary focus of the journal.

An invitation to participate in the survey was sent via email in December 2017 to 702 journal editors for whom we were able to locate email addresses from publicly available web resources. Delivery of 18 of the 702 emails failed (i.e., “bounced”). When also taking into account direct email replies from 4 individuals who indicated that they were no longer serving as journal editor and 2 cases in which the email address was no longer valid for the intended recipient, there was a total of 678 potential survey respondents. Approximately two weeks after delivery of the initial invitation, a reminder email was sent to individuals who had not completed the survey, with a second reminder sent to non-respondents approximately two weeks after that. 113 respondents participated in the survey to yield a 16.7% response rate.

Author survey

The author survey instrument included questions that helped to determine the degree of individual authors’ awareness and understanding of the data policy of the journal in which they published. Additional questions asked about authors’ challenges while performing specific tasks to fulfill data policy requirements. Authors who have also served as peer reviewers were presented with additional questions related to their experiences with data policy implementation from a peer reviewer’s perspective. These questions focused on their process and challenges of evaluating underlying data, code, and/or associated research materials.

The survey instrument included a combination of multiple choice, Likert scale, and open-ended questions, with programmed skip patterns to allow the survey to omit questions not applicable to individual respondents (see S2 Appendix). The survey was administered April to May 2018 using the Qualtrics web survey platform.

Based on the editors’ survey responses, we identified the journals from our list with a data policy that: 1) required authors to provide or describe access to data, code, and/or other research materials associated with the article and/or 2) included a policy enforcement mechanism. This yielded a list of 51 journals. We then examined the first 10 research articles in the current issue of each journal and compiled a list of the corresponding authors. If a journal did not designate a corresponding author, we used the first author. If an issue did not contain 10 research articles, we obtained authors from the preceding issue. Editorials, reviews, letters, and other types of articles not reporting empirical results were omitted. This exercise yielded 510 authors for our author survey. Using publicly available web resources, we identified email addresses for all 510.

Email invitations to participate in the survey were sent to the 510 authors in April 2018. Eight bounced and in one case the author was on sabbatical and unable to reply. Of the resulting 501 potential respondents, 80 participated in the survey, yielding a 16.0% response rate. The 80 respondents represented 34 of the 51 data policy-issuing journals we identified in the editor survey.

Policy content analysis

Data policies were analyzed for journals for which the editor responded “Yes” to the question, “Has [Journal] issued a policy that requires authors to provide access to data, code, and/or other research materials underlying research findings presented in their articles?” and confirming that the policy required the authors to do any one of the following:

• Submit data underlying article findings to a trusted repository
• Submit analytic methods (e.g., code, scripts, packages) to a trusted repository
• Submit research materials (e.g., codebook, readme files) to a trusted repository
• Explain access restrictions for data that cannot be shared due to legal or ethical reasons
• Describe the process for accessing data that cannot be shared due to legal or ethical reasons
• Other (Please specify)

Editors who selected the “Other” category most often specified policy details that echoed that of the predefined categories including policy exemptions for legal and ethical reasons, and provisions for repository deposition of data.

Journals that did not meet the above criteria, but for which the editor did specify a mechanism for data policy enforcement (“Which of the following procedures are included in data policy compliance checks to ensure authors have fulfilled policy requirements?”) were also included in the sample for policy analysis.

For each of the selected journals, we located language text on each journal’s website that referenced instructions to authors, editorial policies, data policies, and/or any other text that provided information to authors on specific requirements for manuscript submission and publication. Editors were also given the opportunity in the survey to provide a URL to the policy or upload policy documents. If relevant language was not located on a journal’s primary website or submitted by the editor, we attempted to locate policy information within the front-facing pages of the manuscript submission portal.

Available policy text was imported into the MAXQDA qualitative data analysis software platform to apply a coding scheme to indicate the presence of a data policy and, if present, any articulation of requirements for data, analytic methods (code), and/or research materials transparency in accordance with the strength of language and conditions as stipulated by TOP Guidelines. Codes were applied to text that indicated specific requirements (i.e., availability statement, repository deposit) as well as mechanisms for policy enforcement (i.e., verification of computational reproducibility).

The content analysis was performed independently by two project team members and tested for intercoder agreement. Each coder reviewed the policy text and identified terms that specified the policy conditions, strength of language, and relevant terms used to describe the type of transparency requirement (i.e., analytics methods (code), data, research materials). In cases for which there were differences in coding, team members reexamined the policy to reach consensus on the most appropriate code application in accordance with the established code definitions.

Data files for the editor survey, author survey, and policy content analysis are available in the UNC Dataverse Repository at https://doi.org/10.15139/S3/DKOUDY [18].

Data policies according to editors

Of the 113 editors who responded to the survey, 57.5% (n = 65) reported their journal having a data policy (see S1 Table). However, when these editors were asked specifically about policy content, only 45.1% (n = 51) specified the presence of transparency requirements. Of these 51 editors, 66.7% (n = 34) indicated a data transparency requirement, 54.9% (n = 28) indicated an analytic methods (or code) transparency requirement, and 37.3% (n = 19) indicated a research materials transparency requirement. S2 Table shows the breakdown by domain of journals and policy requirements as reported by editors.

TOP Level III policies that included provisions for verifying computational reproducibility of reported results were rare, with only 11.8% (n = 6) of editors reporting verification of findings using authors’ submitted code, data, and associated research materials as being part of the policy (S3 Table).

In their open-ended responses to the question, “What have been the greatest challenges of data policy implementation for [the journal]?”, more than one editor mentioned policy enforcement. One respondent provided details: “I'm quite sure that some of the data sets we've accepted are not in very good shape, and editors and reviewers don't always take the time to check. Fully implementing this would require additional staff and funding and cannot be done by volunteer editors and reviewers.”

When it comes to policy implementation success, respondents mentioned both social and technical aspects as contributors to success. More than one noted the advantages of community consensus and engagement in the implementation of data policies. In an open-ended survey question, “What strategies or mechanisms have contributed most to the success of data policy implementation for [the journal]?”, respondents cited “Developing policies with significant input from stakeholders and broad consensus,” “Strong community buy-in,” and “A general acceptance in the community that data should be made available” as exemplifying social contributors to policy success. In terms of technical contributors, some authors made note of their use of data repository services to make the process of submitting data easier for authors.

Data policies according to authors

Of the 80 authors who responded to the survey, there were 5 (6.3%) who indicated that they were not aware of the data policy at any time during the manuscript review and publication process. For the 73 authors who answered the question, “How easy or difficult was it for you to understand what was expected of you to fulfill the requirements of the data policy?”, 86.3% (n = 63) found it somewhat or very easy to understand what was expected of them, whereas 13.7% (n = 10) found it somewhat or very difficult to understand the policy (S4 Table).

Despite the ease with which the majority of authors reported understanding the data policies, a smaller percentage of authors were aware of the specific transparency requirements stated in written policies. For the 34 journals represented in the author survey, 67.6% (n = 23) were noted by at least one author to have a data transparency requirement, 47.1% (n = 16) to have an analytic transparency requirement, and 38.2% (n = 13) to have a research materials transparency requirement written in the policy (S5 Table).

When given the opportunity to share more details of their experience in regard to data policy compliance and/or implementation (In the space below, please share any other information regarding your experience as an author and/or a reviewer complying with or implementing journal data policies), respondents cited the effort required to prepare data for sharing. The sentiment articulated in the following was shared by more than one author: “Submitting data and code is not ‘challenging’ per se (in the same sense that other aspects of scientific research is challenging), but it does take effort. The level of labeling, commenting, and general tidiness that is acceptable is very different for what a researcher can tolerate in materials for their personal use vs. they want to show others and will be useful for others.”

Like comments from the editor survey, some authors noted the benefit of having a mechanism in place for submitting data to a repository to ease the process of policy compliance: “If data is to be stored elsewhere from the journal (e.g. Dryad) it is much easier when the journal creates the entry and provides the link for upload.”

Data policy text

The content analysis included the policies for the 51 journals for which editors indicated that a policy is in place. The location of policies on respective journals’ websites varied, with 49.0% (n = 25) embedded within the text of instructions to authors, editorial policies, or similar information describing the manuscript submission process. 33.3% (n = 17) were also embedded within other instructions, but were able to be located by using a link that navigated directly to the data policy section (i.e., indexed). We located other data policies on a dedicated webpage on the journal website (5.9%; n = 3) and in downloadable documents (3.9%; n = 2). We were unsuccessful in locating the data policy language for 7.8% (n = 4) of journals. S6 Table provides a breakdown of policy locations by domain.

Of the 47 policies we were able to locate, 100.0% included language that mentions data transparency, 73.7% (n = 38) included mention of analytic methods transparency, and 70.2% (n = 33) mentioned research materials transparency (S7 Table).

While the majority of journal policies include mentions of transparency requirements, it was not a straightforward process to determine the stringency, or strictness, of the policy. In many cases, policy language was vague or contradicted itself with expressions of both encouragement and requirement in different parts of the policy text. This experience is consistent with other data policy analysis studies [10, 19–20]. S8 Table lists examples of phrases used to express policy requirements.

We identified 51.1% (n = 24) policies that used terms to convey that transparency was encouraged, and 76.6% (n = 36) that declared transparency a requirement. In 29.8% (n = 14) of the cases, the policy included language indicating both encouragement and requirement of transparency. Just 2 policies from the social sciences included a verification of analysis results requirement (S9 Table).

Another confounding aspect of the policies we analyzed was the assortment of terminology used to refer to data, analytic methods, and research materials transparency. For the most part, policies specified “data” or “dataset”, but in several cases, policies referred to data by naming the domain-specific materials used as evidence to support research findings (e.g., “sequences,” “microarrays”). Identifying policies with analytic methods transparency requirements was also straightforward, with use of terms such as “command files,” “scripts,” and “algorithms.” On the other hand, where policies described research materials transparency, the language was vague, referring simply to “materials” or failing to define specific types of materials (e.g., “documentation,” “metadata,” “other artifacts).” S10 Table provides examples of terminology we interpreted to refer to data, analytic methods, and research materials.

Editor, author, and policy alignment

When we review the results of the surveys and policy content analysis in parallel, we can begin to answer the research questions we sought to address. Comparing editors’ survey responses, authors’ survey responses, and the policy content (where a journal is represented in both the editor and author surveys), we discovered discrepancies between what editors declare their policies to require, what authors understand the policy to require, and what the policy itself states as requirements (see Fig 1). For all categories of transparency requirements, editors underestimate their own journals’ policies. Likewise, authors underestimate policy requirements, but to a greater degree. Even though all 34 policies in the comparison mention data transparency, only 24 editors and 23 authors identified this requirement. For the 27 policies that included an analytic transparency requirement, this was recognized by only 16 authors, and 21 editors. For the research materials transparency requirement, 13 editors and 12 authors were closer in agreement with its presence but is about half the number of the 26 policies that did state this requirement.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Comparison of editors’ conception of the policy, authors’ understanding of the policy, and the policy as written.

https://doi.org/10.1371/journal.pone.0230281.g001

There was also a discrepancy between editors and authors as to their understanding of when the journal policy’s data guidance was presented to authors (Fig 2). When asked “At what point during the manuscript review and publication process did you become aware of the [journal’s] data policy requirements?”, 74% (39/53) of the authors responded, “At the time of initial manuscript submission”. The editors of the same journals as those 39 responses gave a congruent response only 23% (9/39) of the time. Almost half (46%, or 18/39) of the editors responded that the data policy was communicated after peer review, either before or upon final manuscript acceptance, but only one of the 53 authors responded that the data policy was communicated after peer review. While not all the responses can be mapped between the author and editor surveys, only 10 of the 53 (19%) responses strictly agree. It is important to emphasize that this is disagreement between the editor and authors of each individual journal. In the five cases where the author selected “Other”, all but one of the free text responses indicate that the author already knew the data policy prior to selecting the journal, and the remaining author could not recall the timing of when they learned about the data policy. Interestingly, four of the authors responded that “I was not aware of the data policy at any point during the manuscript submission and publication process”.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Comparison of editors’ and authors’ responses regarding the timing at which journal data policy guidance is communicated to authors.

https://doi.org/10.1371/journal.pone.0230281.g002

Do journal-issued data policies articulate clear requirements for data, analytic methods, and research materials sharing and/or verification of computational reproducibility as prescribed in TOP Guidelines?

All of the data policies we examined included statements describing data transparency, and the majority described analytic methods and research materials transparency. However, it was less clear whether the intention of the policy was to encourage or require transparency practices. In some cases, the policy contradicted itself with the appearance of phrasing that suggested both encouragement and requirement of a particular transparency practice. Because of this ambiguity, many policies did not fit neatly, if at all, into the TOP Guidelines matrix to differentiate among TOP levels of stringency. If policies intended to reflect standards of transparency and rigor, this intention was not manifested consistently in policy language.

Do editors understand the journal-issued data policies as they have been written?

A number of editors who responded to the survey understated the requirements as written in the policy documents. All correctly confirmed the presence of a data policy for their journals; however, some editors did not recognize the presence of additional transparency requirements for analytic methods or research materials. Others misstated whether transparency was encouraged or required. This discrepancy suggests that editors are not fully knowledgeable of how policies may be (mis)interpreted by authors.

Do journal-issued data policies use language that obligates authors to comply with the policy?

To an even greater degree than editors, authors are unaware of the specific transparency requirements stated in the data policy language. Nonetheless, authors are well aware of the effort involved in sharing data. For authors to feel obligated to comply with a journal-issued data policy, they need to know of the conditions set forth in the policies to compel them to invest the effort to prepare materials for sharing. Our findings suggest that data policies do not effectively communicate the journal’s expectations in a way that compels authors to fulfill those expectations.

Recommendations

For journals considering implementing a data policy—and those seeking greater policy compliance—it is important that policies are clearly articulated and accurately understood. Here, we offer four specific recommendations. While the extent to which following these recommendations will contribute to a shared understanding of the data policy among editors and authors cannot easily be predicted, they are relatively straightforward to implement and are under the journal’s control.