Introduction

In 2005, the International Committee of Medical Journal editors (ICMJE; www.icmje.org) introduced a policy [1] to require registration of all clinical trials into a registry prior to enrolment of participants. This policy was designed to address the increasing concern that negative trials were not being published [2], leading to a biased literature. Prospective declaration of primary outcomes prior to analysing data is also vital to ensure that analyses cannot influence which measures are reported on in the final publication. These problems often resulted in erroneous conclusions about efficacy, such that clinical guidelines were based on an incomplete and in some cases fraudulent literature [3–8]. To address these problems, the ICMJE developed trial registries as a public platform where information about trial protocols and pre-specified outcome measures for treatment trials could be made publicly available [9].

Journal members of ICMJE began requiring, as a condition of publication, evidence of prospective registration of any clinical trial [1]. The ICMJE define an acceptable registry as one which includes at a minimum a unique identifying number, statements of the intervention and comparison studied, the study hypothesis, definitions of primary and secondary outcome measures, eligibility criteria, key trial dates, target number of subjects, funding source, and contact information for the principal investigator [1]. However, research has highlighted that there are ongoing challenges with adherence to ICMJE’s registration policy, with studies showing continued publication of unregistered studies [10], even in journals that require mandatory registration [11–13]. While it appears that timely registration of trials is improving [13], about half of published trials are not being prospectively registered, with many clinical trials not being registered at all [12, 14, 15]. Selective outcome reporting and biased publication of positive findings is also evident across medical disciplines [12, 16–19], even after the adoption of the ICMJE guidelines.

To date, no study has investigated to what extent psychiatry has adopted ICMJE guidelines as a condition of publication and when they do require registration, whether this requirement is influencing the types of trials being published. Our study had four objectives: 1) to determine the prevalence of articles published in psychiatry journals adhering to ICMJE guidelines that were correctly prospectively registered since the implementation of the ICMJE policy; 2) of those trials identified as prospectively registered with clear POMs, to examine the consistency between POMs in the trial registries and published articles including whether there were retrospective changes to the POMs in the registry and to establish whether any discrepancies between the registries and published articles seemed to favor statistical significance; 3) in prospectively registered trials with clear POMs, to examine consistencies between the participant numbers listed in the trial registries and the published articles; and 4) to determine for clinical trials identified, whether source of funding was related to better adherence with ICMJE guidelines.

Methods

Results

From 3,305 articles identified from all five journals spanning the time frame of January 1^st 2009 to July 31^st 2013, 520 articles were found to be potentially relevant to our study based on the title and abstract (Fig 1). Upon further inspection, 141 articles were excluded due to being outside of the scope. Of the 379 remaining articles, 198 reported having begun patient enrolment before 1 July 2005 (or 1 February 2007 for those articles published in JAACAP and BP). Out of the remaining 181 studies determined to require prospective trial registration, 21 (11.6%) were deemed to be unregistered. Of the 160 registered trials, 17 had an unclear primary outcome measure(s) in the article, 61 had been retrospectively registered, 20 had an unclear or imprecise POM and assessment times in the registry, and two had protocols in registries that were not freely available. Thirty (49.2%) of the 61 retrospectively registered trials took a year or more to be listed on the registry website (Table 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Flowchart of Articles Used to Assess Registration and Selective Reporting.

https://doi.org/10.1371/journal.pone.0133718.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Length of time taken after start date for trial registration to be created for trials that were retrospectively registered.

https://doi.org/10.1371/journal.pone.0133718.t001

Of the 181 trials identified as requiring prospective registration, 60 (33.1%) were correctly registered before the onset of patient enrolment, with clear descriptions of POMs and the times participants would be assessed.

Among the 60 correctly registered articles (S1 File), 17 (28.3%) showed discrepancies between the outcome measures in the trial registry and the published article (Table 2) with the most commonly identified discrepancy being an omission in the published article of a declared POM in the registry. For the 17 articles with discrepancies, we were unable to determine the reason for the discrepancy for five because the POMs had been omitted from the article. For the remaining twelve, seven studies were considered as having made changes to favour statistically significant results and five were not.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Differences in Primary Outcome Measures between Trial Registry and Published Article for Prospectively Registered Trials with Clear Descriptions of Outcome Measures and Differences in Primary Outcomes Favouring Statistically Significant Results.

https://doi.org/10.1371/journal.pone.0133718.t002

Forty-three (71.7%) studies were found to have no discrepancies between the article and the ‘current primary outcome measure’ section in the registry. However, as trial registries are able to be updated at any time by authors, and some clinical trial registries keep a record of updates, we also investigated whether changes had been made to the pre-specified POMs in the trial protocol. While three protocols were published in registries that do not track changes, we found that 17 of these 43 articles had retrospectively updated the POMs in their protocol after the start date of the trial by either adding, removing or altering an outcome measure or timeframe for a POM. One of these articles made these changes after the article had already been submitted for publication. In other words, of the 60 articles found to be correctly prospectively registered, only 26 of them had no evidence of discrepancies between the trial protocol and article and showed no retrospective changes to the POMs in the registry.

The differences in participant numbers between the article and the original entry in the trial registry ranged from no difference, to 67.9% change. Eight (13.3%) of the articles showed no change in participant numbers between the trial registry and the article. Of the articles displaying a difference in participant numbers between the original registry entry and the article, 30 (57.7%) of articles actually reported an increase in participants rather than a decrease. Thirty-two (53.3%) of the final 60 articles had a participant difference less than 20%, while a further 16 (26.7%) had a change in enrolment of between 20% and 50%. Changes of 50% or more occurred in four (6.7%) studies, with two studies having increased from their original target in the trial registry and two having decreased. Five of the articles’ trial protocols were published in the register ISRCTN which does not allow changes to be tracked. We were therefore unable to determine whether the participant number listed in the registry was updated retrospectively. In three cases, the participant number was only entered into the trial registry retrospectively.

Table 3 shows sources of funding as they pertain to the type of registration. Thirty-seven per cent of pharmaceutically-funded trials were properly registered with no changes to POMs versus 6% of private trials, 10.3% of publicly funded trials and 10% of those with a combination of funding. A chi square analysis of these percentages showed a significant group difference (χ2 (3) = 10.19, p < .05), indicating that there were more trials properly registered by the pharmaceutical companies than the private and publicly funded trials. This finding suggests that prospective registration of trials with no changes to POMs was more likely to occur with pharmaceutically-funded studies.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Funding source of research across different categories of registration.

https://doi.org/10.1371/journal.pone.0133718.t003

Discussion

Based on a 4.5 year review of all studies published in the top five impact factor journals in psychiatry that also require prospective registration of clinical trials as a condition of publication, we found that of the 181 articles that were deemed to require registration, 21 (11.6%) were determined to be unregistered and 61 (33.7%) were retrospectively registered. A further 20 (11%), although prospectively registered, POMs were not clearly delineated in the registry. Only 60 (33.1%) articles were correctly prospectively registered with clear descriptions of the POMs in the article and trial registries but, over half of these trials made changes to their originally declared POMs. Indeed, only 26 (14.4%) of the 181 articles published were prospectively registered with no retrospective changes to POMs.

Therefore, even though guidelines of all the journals we investigated clearly state that trials started since 1 July 2005 (or 2007 for two of the journals) must have been prospectively registered in order to be published, almost half of trials requiring registrations did not meet this basic requirement, confirming a continued problem of researchers (and journals) fully observing the ICMJE guidelines [12–14, 20]. It also suggests that journal editors are either not checking trial registries for appropriate registration or determined that there were valid reasons for why the trial was not properly registered and chose to publish it. Communication with the editor of one of the journals (AJP) suggested that this latter reason was indeed the case.

We also determined that selective outcome reporting continues to be a problem, consistent with other research [18, 21–23], even with prospectively registered trials. Seventeen (28.3%) of the 60 correctly registered articles displayed evidence of selective outcome reporting. In at least seven of these cases, changes favored statistically significant findings; in another five, there was no evidence of changes favouring statistically significant findings; and in the remaining five, we had insufficient information to make a decision because the outcomes were not included in the publications. We did not assume that dropping a POM meant that it had been dropped because it was non-significant, this is simply one possibility for why a change like this may have occurred. Despite the extensive publications highlighting this significant problem for over two decades [8, 20], some researchers continue to be selective in their reporting or change a priori decisions (possibly based on findings), despite the knowledge that this practice can have significant repercussions for health decisions and policies and clinical guidelines [21]. If pre-specified POMs are excluded from published trial reports because the findings do not support the hypotheses, there is the potential to overestimate the effectiveness of a treatment or intervention resulting in a bias in the literature. As demonstrated by case studies on paroxetine [9], burying primary outcome data because it is not statistically significant can have dangerous implications for the individuals who eventually may end up using the treatment. Changing time frames can result in a drug appearing more effective as compared with placebo, as evidenced by the reporting of Ballenger et al [24] on Alprazolam. In this study, the researchers focused on the 4 week data despite the fact that the trial was 8 weeks. While at 4 weeks there was clear advantage of the drug over placebo, this advantage disappeared at the 8 week time point. If there is compliance among editors and authors, mandatory trial registration would constitute one step of ensuring an unbiased and open psychiatric literature. Other steps to ensure full transparency would include release of raw data sets to independent investigators and uploading into public databases the trial protocols.

Even in publications that appeared to show consistency between the registry and the publication, evidence of potential post-hoc changing of POMs in the registry was revealed. This means that POMs listed in the registries are currently consistent with the POMs reported in the articles; however, when one tracks the changes being made to the registry, original POMs had been altered in 17 of the 43 correctly registered articles at some point either during the course of the study or after the study was completed. Except for one trial where changes to the POMs were made after submission of the article (based on submission date and dates when changes were made to the registry), we could not determine if these changes were being made before or after statistical analyses and why the changes had been made. Even though the trial registries are meant to be transparent such that changes can be traced, they can be challenging to navigate and no matter how transparent the trial registries become [25], researchers should make clear in the publication why changes were made to POMs retrospectively. There could be benign reasons for changing POMs such as a better POM was found during the course of the trial or an error had been made.

Fifty two (86.7%) of the studies in our final sample of 60 prospectively registered studies with clearly identified POMs showed differences in participant enrolment between their first trial registry entry and the published article. However, 30 (50%) of these studies actually increased participant numbers from their first estimate in the registry. Of the 22 (36.7%) studies that had sample sizes smaller than the expected numbers listed in the trial registry, 12 (20%) were found to have a difference between 20% and 50% while a further two (3.3%) displayed a difference of 50% or more. The remaining eight studies had a difference in participant numbers less than 20%, a number expected if one considers dropouts. This change in sample size is better than that observed in other studies whereby the original sample size declared in the registry was achieved in about 60% of published papers [23]. While some authors did state the reasons for these differences, citing difficulties with recruitment, lack of funding and strict inclusion/exclusion criteria among other reasons for not reaching their target number of participants, it is also possible that in some cases these high levels of change may reflect early analysis to detect group differences.

In order to determine the original sample size and POMs listed in the trial protocols from when they were first created, we tracked the changes in the registry itself. This is a feature that is possible in registries such as Clinicaltrials.gov and ANZCTR, but not ISRCTN. In order to achieve full transparency of research, it is important to have information such as the dates and changes made to POMs and participant numbers in trial protocols freely and readily available.

We identified that it was not always obvious, to us or to study authors, whether registration was required for particular studies. For example, for some unregistered trials, while the authors did report on health related outcomes, these outcomes were not the primary focus of the paper and therefore, it is possible that authors were unclear as to whether registration was required. For trials that are not RCTs, researchers may not consider or be aware that nonRCTs would be classified as a clinical trial based on the ICMJE guidelines as they prospectively assigns human subject to an intervention in order to study the cause-and-effect relationship between the intervention and a health outcome. It is also possible that authors are not registering their studies or doing so retrospectively because they were unaware of the need for trial registrations. It takes time for the guidelines as well as the rationale behind the guidelines to come to the attention of researchers and some may simply have been unaware of the inception of clinical trial registries. It is also possible that the trial registries can be slow to register trials and while a trial may have been submitted prospectively, delays may have resulted in patients being recruited before final registration was granted. This may explain a time lag of a month, but is harder to justify for those where a year or more has passed.

Our study aimed to investigate those studies that were reporting the primary outcome measures of a clinical trial. It is not always easy to determine if a manuscript is reporting secondary outcomes or other areas of further investigation beyond the primary measures. In order to achieve this, we read each article in full to identify that it was not a secondary analysis and where necessary, checking trial registries. While we attempted to ensure the studies we included were the first report on the dataset, we acknowledge that it is possible that we may not have been able to identify every instance of secondary publications of results through this method. We are also aware that in some cases, it is acceptable for multiple reports to be published on a single trial, each reporting on different primary outcome measures. In these instances, it is possible that we may have identified a discrepancy in POMs between the article and the registry, whereas in fact, those POMs were analysed in a subsequent publication on the same study. In these instances, we advocate that any publication reporting on some but not all POMs for a trial should specifically outline that remaining POMs will be discussed in following publications.

The inception of trial registries is a step in the right direction to address a long standing challenge in psychiatry as well as other medical disciplines, of negative trials not being published as well as outcome measures being changed, added or deleted, depending on the results, creating a biased literature that cannot be relied on to adequately guide clinical decisions. Most journals do not mandate trial registration as a condition of publication and therefore it is to their credit that the journals we investigated have done so. Indeed, based on the journals we found, European and Australasian journals have been less likely to adopt mandatory registration. It takes time for guidelines to filter down to the researchers and this may be why so many studies are not prospectively registering trials nor appreciating the significance of doing so. It appears that researchers conducting trials funded by the pharmaceutical industry are more aware of the importance of properly registering trials and not making retrospective changes to POMs. However, the journals that are members of the ICMJE also hold a responsibility to ensure those guidelines are met, and if not, and they choose to publish the work, that it is clear to readers that they have deviated from the expectations stated in their guidelines to authors. While the journals may have good reasons for doing so, readers assume that the trials published in these journals are prospectively registered and therefore the journals have an obligation to make decisions that err from this standard fully transparent.

When new guidelines come along, it can take time for behavior to change. Therefore, maybe it should not be surprising how few researchers are correctly registering their trials. Journals are in a powerful position to change the behaviors of researchers simply by mandating they meet specific requirements before a paper will be published. Those journals that aspired to meet the ICMJE guidelines have clearly felt justified to deviate from them. Journal editors clearly allow some discretion when submissions are made; however, these editorial decisions are not currently included in the published manuscript. There is also a long way to go in terms of increasing the number of journals that deem prospective registration of clinical trials as sufficiently important to make it a requirement of publication.

Looking forward, the situation could be improved if some of the following suggestions were to be adopted: 1) member journals of the ICMJE should have a dedicated person checking trial registries, trials should simply not be published if they haven’t been prospectively registered as determined by the ICMJE or the journals should state clearly and transparently reasons why studies might be published without adhering to ICMJE guidelines, 2) if authors do change POMs or participant numbers or retrospectively register their trials, the reasons should be clearly outlined in the methods, 3) to further improve transparency, authors could upload the full clinical trial protocol, including all amendments, to the registry website and provide the raw data from a clinical trial in a format accessible to the research community, 4) greater effort needs to be made to ensure authors are aware of the importance of prospectively registering trials, by improving guidelines for submission [26] and when applying for ethical approval, and 5) finally, reviewers should not make decisions about the acceptability of a study for publication based on whether the findings are positive or negative as this may be implicitly encouraging authors to be selective in reporting results. Indeed, a recent study showed that only one third of peer reviewers examined registered trial information and reported any discrepancies to the journal editors [27]. It is clear that while there are guidelines in place to decontaminate the medical literature, authors, reviewers and journal editors need to work together to ensure that these guidelines are upheld.

Supporting Information

Acknowledgments

Thanks to Ben Goldacre for inspiring the idea for this research from his book Bad Pharma and to Prof Chris Frampton for statistical consultation.