http://orcid.org/0000-0003-0128-2938
Figures
Abstract
Objectives
Prospective registration of animal studies has been suggested as a new measure to increase value and reduce waste in biomedical research. We sought to further explore and quantify animal researchers’ attitudes and preferences regarding animal study registries (ASRs).
Setting and participants
We conducted a survey with three different samples representing animal researchers: i) corresponding authors from journals with high Eigenfactor, ii) a random Pubmed sample and iii) members of the CAMARADES network.
Main outcome measures
Perceived level of importance of different aspects of publication bias, the effect of ASRs on different aspects of research as well as the importance of different research types for being registered.
Results
The survey yielded responses from 413 animal researchers (response rate 7%). The respondents indicated, that some aspects of ASRs can increase administrative burden but could be outweighed by other aspects decreasing this burden. Animal researchers found it more important to register studies that involved animal species with higher levels of cognitive capabilities. The time frame for making registry entries publicly available revealed a strong heterogeneity among respondents, with the largest proportion voting for “access only after consent by the principal investigator” and the second largest proportion voting for “access immediately after registration”.
Conclusions
The fact that the more senior and experienced animal researchers participating in this survey clearly indicated the practical importance of publication bias and the importance of ASRs underscores the problem awareness across animal researchers and the willingness to actively engage in study registration if effective safeguards for the potential weaknesses of ASRs are put into place. To overcome the first-mover dilemma international consensus statements on how to deal with prospective registration of animal studies might be necessary for all relevant stakeholder groups including animal researchers, academic institutions, private companies, funders, regulatory agencies, and journals.
Citation: Wieschowski S, Laser H, Sena ES, Bleich A, Tolba R, Strech D (2020) Attitudes towards animal study registries and their characteristics: An online survey of three cohorts of animal researchers. PLoS ONE 15(1): e0226443. https://doi.org/10.1371/journal.pone.0226443
Editor: Luciane Cruz Lopes, University of Sorocaba, BRAZIL
Received: August 1, 2019; Accepted: November 26, 2019; Published: January 6, 2020
Copyright: © 2020 Wieschowski et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: This study was funded by the German Research Foundation (DFG), http://gepris.dfg.de/gepris/projekt/260525081 (grant number STR 1070/4-1). DS applied for and received the grant. SW was paid by this funding. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Background
The implementation of prospective animal study registries (ASRs) has been suggested as one of several important measures that may increase value and reduce waste in biomedical research [1]. The first academically hosted ASR was launched in 2017 (www.preclinicaltrials.eu, [2]). This ASR mentions that the Dutch parliament passed a motion on July 3rd, 2018 stating that prospective registration of animal studies and sharing of data should become the norm [3]. Recently, the first governmental ASR was launched in Germany (www.animalstudyregistry.org [4]).
One of the main objectives of prospective study registration is to improve our knowledge of ongoing and completed but unpublished studies. For clinical trials, the pros and cons of prospective trial registration have been discussed over the past three decades [5]. The same challenges that led to the development of registries for clinical research have been increasingly discussed with regard to animal research over the past 5–10 years [1, 6–9]. Better knowledge about ongoing and completed studies may improve the planning and review of new studies, including replications. Furthermore, such knowledge helps us to better understand and ultimately reduce selective and biased reporting of results; it also assists the authors of systematic reviews in identifying the full set of relevant studies for a specific review question. Selective and biased reporting negatively affects research, health care, and efficient funding of research. The Declaration of Helsinki, the internationally acknowledged ethical framework for medical research with humans, added study registration and unbiased reporting as core ethical principles in the 2008 revision [10]. Currently, there are several national and international registries for clinical studies that allow the identification of ongoing clinical studies, their study design characteristics, and in some cases summary results after the study’s completion [11] [12].
A workshop on “Publication bias in animal research” organized by the UK NC3Rs (National Centre for the Replacement, Refinement & Reduction of Animals in Research) in 2015 also focused on the issue of ASRs. A panel debate in this workshop demonstrated the broad lines of, and strong contrasts in, argumentation for and against ASRs. All panel participants agreed, however, that future decision making on the issue of ASRs depends strongly on contexts such as registry characteristics and knowledge about conflicting stakeholder interests. In 2016, some of us published the results of stakeholder interviews that assessed the full spectrum of potential ASR-related strengths, weaknesses, facilitators, and barriers [6]. In addition to the abovementioned benefit of reducing publication bias, stakeholders also mentioned that ASRs might, for example, increase public support of animal research or could serve as a means to increase inter-researcher exchange [6]. Among the mentioned weaknesses of an ASR were the additional administrative burden as well as concerns regarding the protection of intellectual property [6].
The objective of this study was to add a quantitative dimension to these qualitative results. We aimed to study the attitudes and preferences of animal researchers regarding the potential strengths and weaknesses of ASRs as well as ASR characteristics that might facilitate implementation.
Methods
Ethics statement
The ethics committee of Hannover Medical School granted a waiver for ethics approval to this study. We did not involve patients or the public in our work.
Survey development
The design of our survey instrument was informed by previously published surveys of animal researchers on publication bias [9] and of clinical researchers on data sharing [13] as well as by our own data collected during interviews with different stakeholders on animal study registration [6]. The instrument was refined after discussions within the group of authors that included experts on animal research, publication bias, and survey methodology. A cognitive pretest of the survey was performed with four researchers who had between five and 10 years of experience in animal research.
We first asked for demographic data of the respondents using multiple responses as well as open questions. The next survey domains were “attitudes towards value, waste, and reproducibility in animal research”, “opinion on the extent of non-published animal studies”, “attitudes towards strengths and weaknesses of ASRs”, “types of animal studies that need registration” and “facilitators and barriers for implementing ASRs”. In these domains, we employed Likert-type scales, multiple responses and open questions. The full survey instrument is provided in S1 File.
Sampling
We used three different samples for our survey. The first (“journal sample”) was created using an approach similar to that of Rathi et al. [13], using the 2015 Thomson Reuters Journal Citation Report and a filter including all life science disciplines. The filter is available as in S2 File). From the identified journals, we selected those ten journals with the highest normalized Eigenfactor that include publications on in vivo animal research, and we used their names in a search string applied in PubMed (see Box 1). From all articles for each journal, we extracted the corresponding authors of the first (i.e., newest) 70 articles (totalling 700 articles), removed authors that occurred more than once, and ended up with a final list of 833 corresponding authors. Note that several papers had more than one corresponding author.
Box 1: Search string and included journals
((("Journal Name*"[Journal]) AND "disease models, animal"[MeSH Terms]) NOT (("comment"[Publication Type] OR "editorial"[Publication Type] OR "review"[Publication Type]))) AND ("2012/01/01"[Date—Create]: "2015/12/31"[Date—Create]).
*wildcard for the 10 journal names: PLoS One, Nature, Proceedings of the national academy of sciences of the United States of America, Science, Cell, Nature Communications, Journal of Neuroscience, Blood, Circulation, Journal of Immunology.
The second sample (“random sample”) used a random sample of in vivo animal research added in 2016 to the PubMed database and was created using the following search filter:
((((animal[Title/Abstract] AND "in vivo"[Title/Abstract]) OR "disease models, animal"[MeSH Terms]) NOT (("comment"[Publication Type] OR "editorial"[Publication Type] OR "review"[Publication Type]))) AND ("2016/01/01"[Date—Create]: "2016/12/31"[Date—Create]).
E-mail addresses of the corresponding authors were automatically extracted using PDFMiner implementation in Python and using a regular expression (RegEx). The program code is available at [14]. We performed an additional data-cleansing step by checking manually for invalid addresses, duplicates and journal emails. From the resulting 13,801 e-mail addresses, 6,000 were selected using a random sequence retrieved from www.random.org. This number was chosen because it was expected to yield approximately 300 responses when applying the response rate from our first sample (5%).
The third sample (“CAMARADES”) was contacted through CAMARADES (Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies; www.camarades.info). To maintain confidentiality, this sample was not contacted via the survey software but by e-mails from a CAMARADES member (ES), providing a generic web link to the survey. A total of 1,384 persons were contacted with this approach.
Conduct of survey
The survey was conducted between August 2017 and January 2018 according to Dillman’s total design method ([15]). Potential respondents were contacted via e-mail with a standardized invitation letter including a link to the survey. Non-responders were reminded via e-mail one week and three weeks after the initial invitation. As an incentive for survey participation, respondents could participate in a lottery to win one of ten Amazon vouchers (100€ each), and the link to this lottery was provided after completion of the survey.
The project received a waiver from the responsible institutional review board.
Analysis
The 31 respondents who only entered demographic data but did not answer any further items were excluded from the analysis. Their data were only used to check for differences between completers and drop-outs. The chi-square test was used to test for associations between responses and the survey samples as well as socio-demographic characteristics. All other statistical analyses were descriptive. In the 5-point Likert scales, we defined “substantial differences” in response patterns as at least a 1-point difference. Missing data for survey items are documented in each data table.
A Cronbach analysis for each of the multiple item questions showed a good reliability with values of 0.752 to 0.877.
Adherence to reporting STROBE statement checklist items is documented in S1 Checklist.
Results
Response rate
In total, we received 444 responses, 47 from the “preselected journal sample”, 270 from the “random Pubmed sample” and 127 from the “CAMARADES sample” (see Table 1). The overall response rate was highest in the CAMARADES sample (9.4%), and about half of this in the journal sample (5.7%) and the random sample (4.5%). The percentage of complete responses was between 100% (in the journal sample) and 84% (in the CAMARADES sample).
Demographic data
In the following section, we report demographic data for all respondents who completed the survey (n = 413). For better readability, we only present percentages in the text. See Table 2 and Tables A-H in S3 File for specific demographic data for each of the three samples. In total, 40% of survey responders were female and 60% were male. The mean age was 46 years. Most respondents described their current employer as an academic institution (88%), and only a few indicated government (6%), nonprofit organization (3%) or private industry (3%). The majority (60%) were associate or full professors, 29% were assistant professors or postdocs, 9% were PhD students and 2% were Master’s or Bachelor’s students. The majority had published 3–10 articles containing animal experiments over the past three years (45%), 30% had published more than 10 articles, 17% had published 1–2 articles, and 8% did not publish on animal experiments in this time period. Regarding the total funding volume over the last three years, 27% received more and 48% received less than 500,000€, and 25% who had not yet received third-party funding. 48% of respondents indicated that they primarily or only conduct basic animal research, 29% indicated that they primarily or only conduct preclinical animal research, and the remaining 23% indicated that they perform both basic and preclinical research to “an equal level”.
Survey results
Non publication / Publication bias.
Survey respondents indicated that 25% (median, IQR = 35) of their own experiments performed over the last three years were not published. They assumed that approximately 35% (median, IQR = 30) of experiments from other researchers in their field were not published.
The survey respondents consistently found that publication bias is very important (level 4 on a scale from 1 to 4, median, IQR = 1) for all surveyed aspects of animal research, namely, i) planning of future preclinical or clinical research, ii) duplication of research efforts, iii) public support of animal research, and iv) trust of the scientific community in animal research (Table 3).
Impact of animal study registries.
We asked the survey respondents about their agreement/disagreement with regard to nine potential strengths and four potential weaknesses of ASRs. As described in the Methods section, we identified this set of potential strengths and weaknesses based on prior qualitative expert interviews.
On average, the respondents from all three samples agreed (that is, a median value higher than 3.0 on a scale from 1–5) with all nine potential strengths and with three of the four potential weaknesses (see Table 4). Respondents, on average, disagreed (value less than 3.0) with the potential weakness that ASR might “damage the reputation/career of researchers that register studies with “negative/inconclusive” findings” (median = 2, IQR = 1, 53% chose strong or moderate disagreement). The highest agreements (all with a median of 4) were indicated for the potential strengths that ASRs might i) “improve dissemination of study findings” (IQR = 1, 67% chose strong or moderate agreement), ii) “improve refinement in animal research” (IQR = 1, 68% chose strong or moderate agreement) and iii) “help avoid unnecessary repetition of animal experiments” (IQR = 2, 68% chose strong or moderate agreement). The highest agreement for potential weakness was for “add administrative burden to animal research” (median = 4, IQR = 1, 84% chose strong or moderate agreement). See Tables A-F in S4 File for more detailed information on agreement/disagreement across the three surveyed subsamples.
Table A in S4 File allows the comparison of response patterns across the three samples as well as across potentially relevant socio-demographic characteristics such as age, academic rank, the extent of third-party funding, and the number of publications. In general, the response patterns do not differ substantially (>1 point of the 5-point scale) either across the three samples or across subgroups of respondents with different socio-demographic characteristics. For less substantial differences, we of course identified differences between the subgroups (see Table B in S4 File). However, those differences were generally smaller than those between the three survey samples (see Table A in S4 File), indicating either that the subgroup differences were rather small (i.e., smaller than the random differences between the three samples) or that our survey samples were substantially different, therefore generating larger differences than single sociodemographic factors.
Next, we asked for an estimation of how ASRs would influence overall efficiency in animal research. This question considered that registering a study protocol in an ASR is an additional administrative effort that might take approximately 15–60 minutes (if a study protocol is already available) while information given in ASRs might also save time (e.g., via support in identifying similar studies and designing non-duplicative studies). The average estimate for the impact on efficiency lay between “no impact” and “somewhat increase” (mean = 3.41, median = 4, IQR = 2, see Table 5).
For this response, minor differences were identified between the survey samples as well as for the socio-demographic factors age, funding and number of articles published (see Tables C-D in S4 File).
Design of animal study registries.
The last set of questions addressed the concrete design and characteristics of an ASR. We first asked whether the importance of registering animal studies differs according to the types of animal species used or the study objectives. The percentage of respondents indicating that registering animal studies is “not important at all” varied for study objectives from 3% (preclinical efficacy) to 15% (basic research) and for the animal species from 3% (non-human primates) to 21% (fish). For more detailed data, see Table 6 and Tables E-F in S4 File. The remaining respondents, on average, indicated that registering basic research is “moderately” important and that registering animal research for the other objectives (preclinical efficacy, preclinical safety, and environmental risks) is “very” or “extremely” important. With regard to animal species, the remaining respondents indicated that registering studies with non-human primates is “extremely” important, registering studies with other large animals and rodents is “very” important, and registering studies with fish and other types of animals is “moderately” important (Table 7).
Again, we did not find substantial differences in response patterns across samples or socio-demographic characteristics. However, using the Chi-square test, we did find an association between the responses and the survey sample as well as socio-demographic characteristics (see Table G in S4 File). The CAMARADES sample assessed the importance of each category as more important compared to the random and journal samples.
In addition, respondents had the opportunity to name other types of studies that should or should not be registered. We received 95 responses. After clustering by inductive category formation [16], we found that most responses referred to the distinction between confirmatory and exploratory studies (n = 18). The second most frequent responses named species already covered by our survey, additional species or stated that all animal research should be registered (each n = 9). Six respondents asked for registration of studies for regulatory approval. Among other suggestions were further study types and research areas, such as infection studies, in silico studies, ex vivo and in vitro research and systematic reviews.
Mitigation of potential weaknesses of animal study registries.
Certain ASR characteristics can also help to mitigate the potential weaknesses of ASRs. One safeguard to address “theft of ideas” as a potential weakness of prospective study registration is to delay public access to registered protocols (e.g., via embargos). Our survey asked about preferences regarding this safeguard and the most appropriate time point for full public access. We distinguished three alternative time points: i) immediate, ii) 1–3 years or more after registration, and iii) only after “consent” by the PI. We also allowed respondents to indicate “other options”. Overall, for each of the three samples, we found substantial percentages (20% or more) for all three alternative time points, and 14% of all respondents indicated other options (Table 8). The overall most preferred alternative in the random and journal sample was “only after consent by the principal investigator” (36% and 32%). The CAMARADES sample most often (37%) preferred “immediately after registration (as is currently the case for clinical trial registries)”. Chi-square tests showed dependence only on the survey sample, not on any of the tested socio-demographic factors (see Table H in S4 File). Among the suggestions for other options were “after publication” (n = 8), “never”, “immediately after study conclusion”, “depends on the amount of information in the registry”, “combined options, e.g., "consent or 1 year after completion, whichever comes first" (each n = 5), and “after patent filing” (n = 4).
Raw survey data are available in S1 Table.
Discussion
In this survey, we obtained responses from 413 animal researchers (response rate 7%) on their attitudes and preferences regarding the controversially discussed topic of animal study registries (ASR). We sampled survey respondents from three different sources (high Eigenfactor journals, random PubMed sample, CAMARADES sample). In the following, we discuss six core findings.
First, in comparison with the only other survey that we are aware of, our survey of an international sample of animal researchers found lower estimates for how often animal experiments do not get published (50% vs. 34%, respectively) [9]. The former survey was distributed among Dutch laboratory animal researchers in 2011 [9]. We expect to obtain a more valid measure of the extent of non-publication through our currently ongoing follow-up of a random and stratified sample of all completed animal studies at two large German university medical centers (https://osf.io/az7mt/).
Second, across all three samples and across all demographic particularities, we found a general agreement that ASRs bear nine potential strengths and three potential weaknesses for future animal research (see Table 3, Tables A-F in S4 File). The survey thus provided a quantitative confirmation of results gathered in prior qualitative interview research with different animal-research stakeholder groups [6]. For a more detailed discussion we refer to this results publication where we focussed on the different strengths, weaknesses, barriers and opportunities ofs ASRs.
Third, the respondents indicated, on average, that some aspects of ASRs can increase, but other aspects can also decrease the administrative burden of animal research. In summary, the respondents conclude that ASRs will likely not affect or even slightly decrease the administrative burden. Pilot and feasibility studies for ASRs should evaluate and minimize the time needed to register an animal study. As long as protocols for animal studies already exist for authorization purposes, it seems plausible to assume that the extra time required for uploading the protocol information into an ASR is low. ASRs, of course, must do their part and should facilitate the upload of protocol information as best as possible. The remaining administrative burden could be outweighed by the benefits of ASR, such as support in identifying studies similar to the ones that animal researchers are planning to do. This support could help to design studies that are innovative and do not waste time duplicating existing studies.
Fourth, in addition to administrative burden, the survey respondents indicated that the potential theft of ideas is another potential weakness of ASRs. However, on average, this threat was only judged to be somewhat increased. Future studies should evaluate how much embargo time appropriately protects against such risks, including risks to intellectual property.
Fifth, according to the respondents, the level of importance of registering different types of animal species is highly correlated with the animals’ level of cognitive capabilities. One explanation for this finding might be that animal researchers not only link the general legitimacy of animal research to the cognitive capabilities of animals but also to the efforts to guarantee or even increase the value of completed studies with the respective animals. In other words, increasing value and reducing waste of research through more transparency about animal studies might be acknowledged not only as an obligation towards science and society but also as a moral obligation towards the animals used in the respective studies.
Sixth, the time frame for making registry entries publicly available revealed strong heterogeneity among the researchers. While the largest proportion voted for access only after consent by the principal investigator, the second most frequent option was access immediately after registration. Respondents, however, also noted in the survey’s open-ended questions that more information on the registry or on the details to be published was needed before a time frame could be set.
Regarding the survey instrument used here, we can assume a good reliability as shown by the cronbach analysis for each of the multiple item questions with α values of 0.752 to 0.877. For content validity, we can assume a high face validity, since our survey was designed with the knowledge we gained in expert interviews with different stakeholders [6]. Regarding construct validity, there is not much research on the same topic. However, ter Riet et al.[9] also asked for publication rates of animal researchers and yielded similar responses, although our respondents gave slightly lower estimates for their own publication rates (75% vs. 80%) and higher estimates for their colleagues’ publication rates (65% vs. 50%). ter Riet et al. also surveyed animal researchers on importance of publication bias and how a registry would affect different aspects of research. As in our survey, respondents rated those items as “very important”.
Our study has several limitations. First, the response rate after two reminders was low (5%). We expected a low response rate for a survey of randomly selected international samples of animal researchers. An NC3R representative told us in a personal communication that they also receive approximately 5% responses to their national surveys of animal researchers. Despite the fact that we were able to collect responses from more than 400 animal researchers, our sample certainly does not represent the “average” animal researcher. The demographic data of our sample indicate the participation of more senior and experienced animal researchers, with 60% of the survey participants being full or associate professors and 29% being assistant professors or postdocs. Furthermore, the majority of respondents (75%) had published 3–10 articles or more over the past three years. Our sample therefore did not thoroughly capture the attitudes of younger generations of animal researchers. However, the fact that the more senior and experienced animal researchers clearly indicated the practical importance of publication bias and the importance of ASRs underscores the awareness of this problem among animal researchers and the willingness to actively engage in study registration if effective safeguards for the potential weaknesses of ASRs are put into place.
What are the next steps for implementing ASRs and what can we learn from the already established registries for clinical research? In clinical research, registries have been argued for and have already existed on a non-mandatory basis since the 1980s [17, 18]. However, prospective registration only became a widely established practice in clinical research after obligatory journal policies and national and international laws were introduced in the years 2007 and 2008. More recent experiences with non-mandatory registries for animal research suggest a similar tendency. The first international registry focusing on animal studies (www.preclinicaltrials.eu) only includes 20 protocols after more than one year of conduct (accessed 9 April 2019). In January 2019, the site www.animalstudyregistry.org was launched by the German Centre for the Protection of Laboratory Animals (Bf3R), affiliated with the Federal Institute for Risk Assessment (BfR). This Bf3R registry currently lists 6 studies (accessed 9 April 2019). Other sources for registering animal studies, such as the Open Science Framework (OSF, https://osf.io/) or www.researchregistry.com, do exist, but the search fields of these online sources do not allow the identification of the full set of prospectively registered animal studies. Some journals have developed new publication formats such as preregistered protocols [19] or have implemented open peer review [20]. The first evaluations of this new publication model demonstrated the need for better standards regarding the content of pre-registered protocols [21]. Further discussion on the potential strengths and weaknesses of ASRs within the scientific community of animal research is needed. The results of interview and survey research such as our study might help to facilitate a balanced debate that accounts for both potential strengths and potential weaknesses.
In addition to academic discourse on this topic of ASRs, research institutions will most likely play an important role in facilitating the uptake of registries. For example, institutions can reward and incentivize the efforts of individual researchers to increase transparency such as the prospective registration of protocols. One example is the recent activities at the Berlin Institute of Health (BIH) in cooperation with the Charité –Universitätsmedizin Berlin and the Max Delbrück Center for Molecular Medicine. Researchers publishing papers based on pre-registered protocols receive 1,000€ grants for their working groups. Academic institutions can further reward pre-registration if it plays a role in the selection and hiring processes for professorships. In addition to these indirect incentives for ASRs, funders, regulatory bodies or journals could also shift to more direct obligations. In clinical research, pre-registration has become mandatory for many funding and approval decisions.
To overcome the first-mover dilemma, international consensus statements on how to manage the prospective registration of animal studies might be necessary for all mentioned stakeholder groups: research institutions, funders, regulatory agencies, journals, and researchers (from academia and industry).
Acknowledgments
We thank Siegfried Geyer und Sabine Bossert for their help in developing the survey instrument.
References
- 1. Kimmelman J, Anderson JA. Should preclinical studies be registered? Nat Biotechnol. 2012;30(6):488–9. pmid:22678379.
- 2.
Utrecht RM. Online register for preclinical animal studies launched. 2017.
- 3.
preclinicaltrials.eu. https://www.preclinicaltrials.eu/documents/Motie_Tjeerd_de_Groot.pdf.
- 4.
BfR. Strengthen science, avoid unnecessary animal experiments 2019. https://www.bfr.bund.de/en/press_information/2019/01/strengthen_science__avoid_unnecessary_animal_experiments-239405.html.
- 5. Dickersin K, Rennie D. Registering clinical trials. JAMA. 2003;290(4):516–23. Epub 2003/07/24. pmid:12876095.
- 6. Wieschowski S, Silva DS, Strech D. Animal Study Registries: Results from a Stakeholder Analysis on Potential Strengths, Weaknesses, Facilitators, and Barriers. PLoS Biol. 2016;14(11):e2000391. pmid:27832101.
- 7. Prinz F, Schlange T, Asadullah K. Believe it or not: how much can we rely on published data on potential drug targets? Nat Rev Drug Discov. 2011;10(9):712. pmid:21892149.
- 8. Sena ES, van der Worp HB, Bath PM, Howells DW, Macleod MR. Publication bias in reports of animal stroke studies leads to major overstatement of efficacy. PLoS Biol. 2010;8(3):e1000344. pmid:20361022.
- 9. ter Riet G, Korevaar DA, Leenaars M, Sterk PJ, Van Noorden CJ, Bouter LM, et al. Publication bias in laboratory animal research: a survey on magnitude, drivers, consequences and potential solutions. PLoS One. 2012;7(9):e43404. pmid:22957028.
- 10. World Medical A. World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. Jama. 2013;310(20):2191–4. pmid:24141714.
- 11. De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, et al. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. N Engl J Med. 2004;351(12):1250–1. pmid:15356289.
- 12.
International Cf, (CIOMS) OoMS, the icw, (WHO) WHO. International Ethical Guidelines for Health-related Research Involving Humans 2016. https://cioms.ch/wp-content/uploads/2017/01/WEB-CIOMS-EthicalGuidelines.pdf.
- 13. Rathi V, Dzara K, Gross CP, Hrynaszkiewicz I, Joffe S, Krumholz HM, et al. Sharing of clinical trial data among trialists: a cross sectional survey. BMJ. 2012;345:e7570. pmid:23169870.
- 14.
Laser H. E-Mail_Extractor, https://github.com/HALaser/E-Mail_Extractor/blob/master/python_code.
- 15.
Dillman DA. Mail and Internet Surveys: The Tailored Design Method—2007 Update with New Internet, Visual, and Mixed-Mode Guide: John Wiley & Sons; 2011 31.01.2011.
- 16. Mayring P. Qualitative content analysis: theoretical foundation, basic procedures and software solution 2014.
- 17. Simes RJ. Publication bias: the case for an international registry of clinical trials. J Clin Oncol. 1986;4(10):1529–41. Epub 1986/10/01. pmid:3760920.
- 18. Dickersin K, Rennie D. The evolution of trial registries and their use to assess the clinical trial enterprise. JAMA. 2012;307(17):1861–4. Epub 2012/05/03. pmid:22550202.
- 19. Chambers CD, Forstmann B, Pruszynski JA. Science in flux: Registered reports and beyond at the European Journal of Neuroscience. Eur J Neurosci. 2019;49(1):4–5. pmid:30584679.
- 20. Carroll HA, Toumpakari Z, Johnson L, Betts JA. The perceived feasibility of methods to reduce publication bias. PloS one. 2017;12(10):e0186472. pmid:29065125.
- 21. Hardwicke TE, Ioannidis JPA. Mapping the universe of registered reports. Nature Human Behaviour. 2018;2(11):793–6. pmid:31558810