Introduction

The question of how the brain creates the mind has captivated humankind for thousands of years. With recent advances in human in vivo brain imaging, we now have effective tools to peek into biological underpinnings of mind and behavior. Even though we are no longer constrained just to philosophical thought experiments and behavioral observations (which undoubtedly are extremely useful), the question at hand has not gotten any easier. These powerful new tools have largely demonstrated just how complex the biological bases of behavior actually are. Neuroimaging allows us to give more biologically grounded answers to burning questions about everyday human behavior (“why do we crave things?”, “how do we control learned responses?”, “how do we regulate emotions?”, etc.), as well as influencing how we think about mental illnesses.

In addition to fantastic advances in terms of hardware we can use to study the human brain (function Magnetic Resonance Imaging, magnetoencephalography, electroencephalography, etc.), we have also witnessed many new developments in terms of data processing and modelling. Many bright minds have contributed to a growing library of methods that derive different features from brain signals. Those methods have widened our perspective on brain processes, but also resulted in methodological plurality [1]. Saying that there is no single best way to analyze a neuroimaging dataset is an understatement; we can confidently say that there are many thousands of ways to do that.

Having access to a plethora of denoising and modelling algorithms can be both good and bad. On one side, there are many aspects of brain anatomy and function that we can extract and use as dependent variables, which maximizes the chances of finding the most appropriate and powerful measure to ask a particular question. On the other side, the incentive structure of the current scientific enterprise combined with methodological plurality can be a dangerous mix. Scientists rarely approach a problem without a theory, hypothesis, or a set of assumptions, and the high number of “researcher degrees of freedom” [2] can implicitly drive researchers to choose analysis workflows that provide results that are most consistent with their hypotheses. As Richard Feynman said “The first principle is that you must not fool yourself—and you are the easiest person to fool.” Additionally, neuroimaging (like almost every other scientific field) suffers from publication bias, in which “null” results are rarely published, leading to overestimated effect sizes (for review of this and other biases see [3]).

Recent years have seen an increase in alarming signals about the lack of replicability in neuroscience, psychology, and other related fields [4]. Neuroimaging studies generally have low statistical power (estimated at 8%) due to the high cost of data collection, which results in an inflation of the number of positive results that are false [5]. To avoid a widespread crisis in our field and consequent loss of credibility in the public eye, we need to improve how we do science. This article aims to complement existing literature on the topic [6–8] by compiling a practical guide for researchers at any stage of their careers that will help them make their research more reproducible and transparent while minimizing the additional effort that this might require (Fig 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Three pillars of Open Science: data, code, and papers.

https://doi.org/10.1371/journal.pbio.1002506.g001

How to Deal with Data

Data are a central component of the scientific process. When data are made openly accessible, they not only allow the scientific community to validate the accuracy of published findings but also empower researchers to perform novel analyses or combine data from multiple sources. Papers accompanied by publicly available data are, on average, cited more often [9,10], while at the same time exposing fewer statistical errors [11]. Data sharing has been mandated by some grant funding agencies, as well as journals. Some also argue that sharing data is an ethical obligation toward study participants in order to maximize the benefits of their participation [12]. Neuroimaging has a substantial advantage in terms of ease of data capture, since the data generation process is completely digital. In principle, one could provide a digital record of the entire research process for the purpose of reproducibility. However, even though data sharing in neuroimaging has been extensively reviewed in [13] and [14], there is little practical advice on the topic.

How to Deal with Code

Neuroimaging data analysis has required computers since its inception. A combination of compiled or script code is involved in every Positron Emission Tomography (PET), MRI, or Electroencephalography (EEG) study, as in most other fields of science. The code we write to analyze data is a vital part of the scientific process and, similar to data, is not only necessary to interpret and validate results but can be also used to address new research questions. Therefore, the sharing of code is as important as the sharing of data for scientific transparency and reproducibility.

Because most researchers are not trained in software engineering, the code that is written to analyze neuroimaging data (as in other areas of science) is often undocumented and lacks the formal tests that professional programmers use to ensure accuracy. In addition to the lack of training, there are few incentives to spend the time necessary to generate high-quality and well-documented code. Changes in the incentive structure of science will take years, but in the meantime, perceived poor quality of code and lack of thorough documentation should not prevent scientists from publishing it [24]. Sharing undocumented code is much better than not sharing code at all and can still provide benefits to the author. Perhaps the most compelling motivation for sharing code comes from citation rates. Papers accompanied by usable code are, on average, cited more often than their counterparts without the code [25].

An additional concern that stops researchers from sharing code is fear that they will have to provide user support and answer a flood of emails from other researchers who may have problems understanding the codebase. However, sharing code does not oblige a researcher to provide user support. One useful solution to this problem is to set up a mailing list (for example, with Google) and point all users to ask questions through it; in this way, answers are searchable, so that future users with the same questions can find them via a web search. Alternatively, one can point users to a community-driven user support forum for neuroinformatics (such as NeuroStars.org) and ask them to tag their questions with a label uniquely identifying the software or script in question; we have found this to be a useful support solution for the OpenfMRI project. Both solutions foster a community that can lead to users helping each other with problems, thus relieving some of the burden from the author of the software. In addition, since the user support happens through a dedicated platform, there is less pressure on the author to immediately address issues than there would be with user requests sent directly by email.

Many of the issues with code quality and ease of sharing can be addressed by careful planning. One tool that all research programmers should incorporate into their toolbox is the use of a Version Control System (VCS) such as git. VCS provides a mechanism for taking snapshots of evolving codebase that allow tracking of changes and reverting them if there is a need (e.g., after making a change that ends up breaking things). Adopting a VCS leads a to cleaner code base that is not cluttered by manual copies of different versions of a particular script (e.g., “script_version3_good_Jan31_try3.py”). VCS also allows one to quickly switch between branches—alternative and parallel versions of the codebase—to test a new approach or method without having to alter a tried and tested codebase. For a useful introduction to git, we refer the reader to [26]. We encourage scientists to use git rather than other VCS’s due to a passionate and rapidly growing community of scientists who use the GitHub.com platform, which is a freely available implementation of the git VCS. In the simplest use case, GitHub is a platform for sharing code (which is extremely simple for those who already use git as their VCS), but it also includes other features that make contributing to collaborative projects, reviewing, and testing code simple and efficient. The OSF mentioned above can be used to link together data and code related to a single project. It can also be used to set an embargo period on the code so it could be submitted with the paper while minimizing the risk of “scooping.”

Striving for automation whenever possible is another strategy that will not only result in more reproducible research but can also save a lot of time. Some analysis steps seem to be easy to perform manually, but that remains true only when they need to be performed just once. Quite often in the course of a project, parameters are modified, the list of subjects is changed, and processing steps need to be rerun. This is a situation in which having a set of scripts that can perform all of the processing steps automatically instead of relying on manual interventions can really pay off. There are many frameworks that help design and efficiently run neuroimaging analyses in automated fashion. Those include, but are not limited to: Nipype [27], PSOM [28], aa [29], and make [30]. As an example, for our recent work on the MyConnectome project [31], we created a fully automated analysis pipeline, which we implemented using a virtual machine (https://github.com/poldrack/myconnectome-vm).

While automation can be very useful for reproducibility, the scientific process often involves interactive interrogation of data interleaved with notes and plots. Fortunately, there is a growing set of tools that facilitate this interactive style of work while preserving a trace of all the computational steps, which increases reproducibility. This philosophy is also known as “literate programming” [32] and combines analysis code, plots, and text narrative. The list of tools supporting this style of work includes, but is not limited to: Jupyter (for R, Python and Julia - http://jupyter.org), R Markdown (for R: http://rmarkdown.rstudio.com), and matlabweb (for MATLAB: https://www.ctan.org/pkg/matlabweb). Using one of those tools not only provides the ability to revisit an interactive analysis performed in the past but also to share an analysis accompanied by plots and narrative text with collaborators. Files created by one of such systems (in case of Jupyter they are called Notebooks) can be shared together with the rest of the code on GitHub, which will automatically render included plots so they can be viewed directly from the browser without requiring installation of any additional software.

As with data, it is important to accompany shared code with an appropriate license. Following [6], we recommend choosing a license that is compatible with the open source definition such as Apache 2.0, MIT, or GNU General Public License (GPL: https://opensource.org/licenses). The most important concept to understand when choosing a license is “copyleft.” A license with a “copyleft” property (such as GPL) allows derivatives of your software to be published, but only if done under the same license. This property limits the range of code your software can be combined with (due to license incompatibility) and thus can restrict the reusability of your code; for this reason, we generally employ minimally restrictive licenses such as the MIT license. Choosing an open source license and applying it to your code can be greatly simplified by using a service such as choosealicense.com.

How to Deal with Publications

Finally, the most important step in dissemination of results is publishing a paper. An essential key to increasing transparency and reproducibility of scientific outputs is an accurate description of methods and data. This not only means that the manuscript should include links to data and code mentioned before (which entails that both data and code should be deposited before submitting the manuscript) but also thorough and detailed description of methods used to come to a given conclusion. As an author, one often struggles with a fine balance between detailed description of different analyses performed during the project and the need to explain the scientific finding in the clearest way. It is not unheard of that for the sake of a better narrative, some results are omitted (http://sometimesimwrong.typepad.com/wrong/2015/11/guest-post-a-tale-of-two-papers.html). At the same time, there is a clear need to present results in a coherent narrative with a clear interpretation that binds the new results with an existing pool of knowledge (http://www.russpoldrack.org/2015/11/are-good-science-and-great-storytelling.html). We submit that one does not have to exclude the other. A clear narrative can be provided in the main body of the manuscript and the details of methods used together with null results, and other analyses performed on the dataset can be included in the supplementary materials, as well as in the documentation of the shared code. In this way, the main narrative of the paper is not obfuscated by too many details and auxiliary analyses, but all of the results (even null ones) are available for the interested parties. Such results from extra analyses could include, for example, all of the additional contrasts that were not significant and thus not reported in the main body of the manuscript (of which unthresholded statistical maps should be shared, for example, using a platform such as NeuroVault). Often, these extra analyses and null results may seem uninteresting from the author's point of view, but one cannot truly predict what other scientists can be interested in. In particular, the null results (which are difficult to publish independently) can contribute to the growing body of evidence that can be used in the future to perform meta-analyses. For a more extensive set of recommendation for reporting neuroimaging studies, see the recent report from the Organization for Human Brain Mapping’s Committee on Best Practices in Data Analysis and Sharing (COBIDAS) report (http://www.humanbrainmapping.org/cobidas/).

The last important topic to cover is accessibility of the manuscript. To maximize the impact of published research, one should consider making the manuscript publicly available. In fact, many funding bodies (NIH, Wellcome Trust) require this for all manuscripts describing research that they have funded. Many journals provide an option to make papers open access, albeit sometimes at a prohibitively high price (for example, the leading specialist neuroimaging journal—NeuroImage—requires a fee of US$3,000). Unfortunately, the most prestigious journals (Nature and Science) do not provide such an option despite many requests from the scientific community. Papers published in those journals remain “paywalled”—available only through institutions that pay subscription fees, or through public repositories (such as PubMed Central) after a sometimes lengthy embargo period. The scientific publishing landscape is changing [33,34], and we hope it will evolve in a way that will give everyone access to published work as well as to the means of publication. In the meantime, we recommend ensuring open access by publishing preprints at Biorxiv or arXiv before submitting the paper to a designated journal. In addition to making the manuscript publicly available without any cost, this solution has other advantages. Firstly, it allows the wider community to give feedback to the authors about the manuscript and potentially improve it, which is beneficial for both the authors as well as the journal the paper will be submitted to; for example, the present paper received useful comments from three individuals in addition to the appointed peer reviewers. Secondly, in case of hot topics, publishing a preprint establishes precedence on being the first one to describe a particular finding. Finally, since preprints have assigned DOIs, other researchers can reference them even before they will be published in a journal. Preprints are increasingly popular, and the vast majority of journals accept manuscripts that have been previously published as preprints. We are not aware of any neuroscience journals that do not allow authors to deposit preprints before submission, although some journals such as Neuron and Current Biology consider each submission independently, and thus one should contact the editor prior to submission.

To further improve accessibility and impact of research outputs, one can also consider sharing papers that have already been published in subscription-based journals. Unfortunately, this can be difficult due to copyright transfer agreements many journals require from authors. Such agreement gives the journal exclusive right to the content of the paper. However, each publisher uses a different set of rules, and some of them allow limited sharing of the work you have to surrender your rights to. For example, Elsevier (publisher of NeuroImage) allows authors to publish their accepted manuscripts (without the journal formatting) on a noncommercial website, a blog, or a preprint repository (https://www.elsevier.com/about/company-information/policies/sharing). Wiley (publisher of Human Brain Mapping) has a similar policy for submitted manuscripts (before the paper gets accepted), but requires an embargo of 12 months before authors can share the accepted manuscript (http://olabout.wiley.com/WileyCDA/Section/id-826716.html). Policies for other journals might vary. SherPa/ROMEO (http://www.sherpa.ac.uk/romeo) is a database that allows authors to quickly check what the journal they published with allows them to share and when.

There are multiple options when it comes to choosing a repository to share manuscripts published in subscription-based journals. Private websites, institutional repositories, and preprint servers seem to be well within the legal restrictions of most journals. Commercial websites such as researchgate.com and academia.edu remain a legal grey zone (with some reports of Elsevier taking legal actions to remove papers from one of them: http://svpow.com/2013/12/06/elsevier-is-taking-down-papers-from-academia-edu/). If the research has been at least partially funded by NIH, one can deposit the manuscript in PubMed Central (respecting appropriate embargos: https://nihms.nih.gov).

Discussion

In this guide, we have carefully selected a list of enhancements that every neuroimaging researcher can make to their scientific workflow that will improve the impact of their research, benefiting not only them individually but the community as a whole. We have limited the list to mechanisms that have been tested and discussed in the community for a number of years and which have clear benefits to the individual researcher. However, the way science is conducted is evolving constantly, and there are many more visions that could be implemented. In the following section, we discuss some of the emerging trends that may become commonplace in the future.

Summary

The scientific method is evolving towards a more transparent and collaborative endeavor. The age of digital communication allows us to go beyond printed summaries and dive deeper into underlying data and code. In this guide, we hope to have shown that there are many improvements in scientific practice everyone can implement with relatively little added effort that will improve transparency, replicability, and impact of their research (Box 1). Even though the added transparency might, in rare cases, expose errors, those are a natural part of the scientific process. As a community, researchers should acknowledge their existence and try to learn from them instead of hiding them and antagonizing those who make them.

Acknowledgments

We would also like to thank Cyril Pernet, Yaroslav Halchenko, and Cameron Craddock for helpful comments and corrections, as well as Michael Milham for a stimulating discussion.