Background

Study publication bias and within-study outcome reporting bias have been recognised as two forms of reporting bias that can affect evidence based practice. There is strong evidence of an association between direction of results and publication status; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported (range of odds ratios: 2.2 to 4.7) [1,2]. As a result of these two forms of bias, systematic reviews summarising the available evidence are likely to overestimate the effect of treatment [3,4]. Reporting bias may therefore result in inappropriate health care decisions by policy makers, clinicians and patients, which potentially harm patients, waste resources, and misguide future research. While there is growing empirical evidence for the existence of these biases, little is known about the relative size of these two related problems affecting the available research evidence. Using data taken from protocols of clinical research projects submitted to the research ethics committee of the University of Freiburg (Germany) and associated fully published articles, we estimate the proportion of missing participant outcome data due to lack of publication of the study and the proportion due to missing outcome data within a published study.

Methods

Results

The study cohort contained 311 RCTs of which three were excluded as they presented no results for the comparator group. This is a high risk of bias form of selective reporting which has previously been documented [3], and our decision to exclude here is that our missing data computations would mask this serious reporting problem by simply reducing the sample sizes of these trials to what was either reported in the treatment arm or not reported in the comparator arm. Of the remaining 308 trials, 167 were published (54%) and 141 (46%) were unpublished. There was a tendency for published trials to specify more outcomes in their protocols than unpublished trials (Table 1). Further, the median number of recruited participants was higher than the number of planned participants in unpublished trials, and a higher proportion of published trials were commercially funded (Table 1). The majority of trials (published and unpublished) were multi-centre trials and of these multi-centre trials, most were international trials (Table 1). The primary centre for the multi-centre trials was Freiberg for 12% of published trials and 14% for unpublished trials.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Outcome, participant and funding characteristics of trials that were published and unpublished.

https://doi.org/10.1371/journal.pone.0157883.t001

The 308 published and unpublished trials contained data on 3407 distinct outcomes of 260,563 participants (Table 1). From the published trials 70% (1595/2253) of the outcomes were published in full, 7% (167/2253) of the outcomes went unreported while the remainder were partially reported. Clearly all 1154 outcomes from the unpublished studies went unreported.

At the individual participant level, the total amount of participant outcome data was 2,618,116 (Table 2). Under a half of this data (47%) was fully reported but at least 81% were partially reported. Four percent of the data was missing (within-study selective outcome reporting) from published trials and 15% was missing from entirely unpublished trials.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Proportion of reported and missing participant outcome data.

https://doi.org/10.1371/journal.pone.0157883.t002

Sixty percent (119/198) of commercially funded trials were published while 50% (20/40) of non-commercially funded trials were published. A commercially funded trial had a 20% increased probability of being published compared to a non-commercially funded trial although this difference did not reach statistical significance (relative risk 1.20, 95% confidence interval (0.86, 1.67; p = 0.27)). This wide confidence interval suggested that there was a high level of uncertainty in this finding.

Of the published trials, nearly a fifth (23/119; 19%) of the commercially funded trials fully reported all pre-specified outcomes compared to 30% (6/20) of the non-commercially funded trials. Commercially funded trials were less likely to fully report all outcomes than non-commercially funded trials although this difference again did not reach statistical significance with a wide confidence interval denoting a high level of uncertainty in this finding (relative risk 0.64, 95% confidence interval (0.30, 1.38; p = 0.26)).

Discussion

Our study has shown that (81%) of participant outcome data from trials is either reported in full or partially reported. Direct empirical evidence for the existence of study publication bias and outcome reporting bias has been summarized previously but [1], to our knowledge, this study is novel seeking to quantify how much outcome data is missing from trials and how much of this missingness is attributable to lack of publication of entire studies or of outcomes from studies for which other data has been published. Missing data from published studies which could be associated with within-study selective outcome reporting appeared to be less of a problem than missing data from entirely unpublished studies. Nevertheless we have shown that the amount of missing data from both published and unpublished trials is more than half (53%) if partially reported data is treated as unpublished. Considering partially reported data as unpublished reflects the fact that such data cannot be included in a meta-analysis right away. In a survey, most Cochrane review authors related that they requested additional unpublished data from trialists but only about half of them obtain them [9]. Furthermore, partial reporting has previously been associated with a high risk of outcome reporting bias [3,10]. Our study is also likely to underestimate the impact of partial reporting, since outcomes with multiple time-points were considered to be fully reported if just one time-point was reported in full.

Conclusions

The full publication of outcome data from clinical trials helps avoid unnecessary duplication of research while clinical trial registration allows the relevant stakeholders identify that a clinical trial exists. The systematic recording and monitoring of trial information within a registry goes some way to prevent and detect the selective publication and selective reporting of clinical research outcomes.

The large proportion of partially reported outcome data suggests that journal editors and reviewers need to more closely monitor and address this problem. Relevant reporting guidelines clearly advocate for complete reporting of outcome data and are endorsed by many journals [14]. However, routine checks of the essential items to be reported are often still not implemented in peer review and editorial procedures. As summarized in recent systematic reviews of non-publication and reporting biases [1] there is solid evidence confirming both areas of problem [2]. Future studies need to explore ways how the known underreporting of outcome data can be tackled effectively e.g. by creating incentives for complete and accurate reporting of outcome data from clinical trials and for making raw trial data more accessible.

Ethical approval

No ethics committee opinion was required for this study.

Supporting Information

Acknowledgments

We are grateful to the research ethics committee of the Albert-Ludwigs-Universität Freiburg/Germany, its administrative staff, and the local investigators who responded to our survey. We also thank Alexander Hellmer and Florian Volz for help with extraction of protocol data, and Patrick Oeller for updating the literature search.