Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Who Has Used Internal Company Documents for Biomedical and Public Health Research and Where Did They Find Them?

  • L. Susan Wieland ,

    lswieland@gmail.com

    Affiliation Center for Evidence-Based Medicine, Brown University, Providence, Rhode Island, United States of America

  • Lainie Rutkow,

    Affiliation Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, United States of America

  • S. Swaroop Vedula,

    Affiliation Johns Hopkins University, Baltimore, Maryland, United States of America

  • Christopher N. Kaufmann,

    Affiliation Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, United States of America

  • Lori M. Rosman,

    Affiliation William H. Welch Medical Library, Johns Hopkins University, Baltimore, Maryland, United States of America

  • Claire Twose,

    Affiliation William H. Welch Medical Library, Johns Hopkins University, Baltimore, Maryland, United States of America

  • Nirosha Mahendraratnam,

    Current address: Avalere Health LLC, Washington, DC, United States of America

    Affiliation Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, United States of America

  • Kay Dickersin

    Affiliation Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, United States of America

Abstract

Objective

To describe the sources of internal company documents used in public health and healthcare research.

Methods

We searched PubMed and Embase for articles using internal company documents to address a research question about a health-related topic. Our primary interest was where authors obtained internal company documents for their research. We also extracted information on type of company, type of research question, type of internal documents, and funding source.

Results

Our searches identified 9,305 citations of which 357 were eligible. Scanning of reference lists and consultation with colleagues identified 4 additional articles, resulting in 361 included articles. Most articles examined internal tobacco company documents (325/361; 90%). Articles using documents from pharmaceutical companies (20/361; 6%) were the next most common. Tobacco articles used documents from repositories; pharmaceutical documents were from a range of sources. Most included articles relied upon internal company documents obtained through litigation (350/361; 97%). The research questions posed were primarily about company strategies to promote or position the company and its products (326/361; 90%). Most articles (346/361; 96%) used information from miscellaneous documents such as memos or letters, or from unspecified types of documents. When explicit information about study funding was provided (290/361 articles), the most common source was the US-based National Cancer Institute. We developed an alternative and more sensitive search targeted at identifying additional research articles using internal pharmaceutical company documents, but the search retrieved an impractical number of citations for review.

Conclusions

Internal company documents provide an excellent source of information on health topics (e.g., corporate behavior, study data) exemplified by articles based on tobacco industry documents. Pharmaceutical and other industry documents appear to have been less used for research, indicating a need for funding for this type of research and well-indexed and curated repositories to provide researchers with ready access to the documents.

Introduction

Even though the scientific research enterprise and healthcare decisions rely on the biomedical literature being complete and accurate, it is neither [1], [2]. It is now well-established that strength and direction of findings is associated with selective reporting, and when this happens it is termed a “reporting bias” [3]. Reporting biases may manifest as omission of study findings from the literature, either completely or in part; for example, particular outcomes or analyses may be omitted [4]. Reporting biases originate mainly with the investigators, not journal editors, and occur when research is sponsored by for-profit and not-for-profit entities [5]. Funding by for-profit companies appears to be independently associated with selective reporting, however [6].

Research on selective reporting and other reporting biases is made possible when the published literature can be compared with other sources of information about the same research studies, for example from research ethics committees [7], [8], [9], funding agencies [10], [11], clinical trial registers [12], [13], documents and data released by regulatory authorities [2], [14], [15], and internal company documents released though litigation or other means [4], [16], [17], [18]. Internal company documents serve as a valuable source of information about industry-sponsored research for those producing evidence summaries [19], those concerned about an entire industry's global marketing behavior [20], and for those wishing to report a study's findings as a restorative author [21]. For example, the clinical study reports produced by pharmaceutical companies at the completion of a clinical trial typically include the protocol (what was planned) and a detailed description of study analyses and findings [17], [22]. In a study of industry trials where available internal documents were compared with publications, the primary outcome defined in the protocol (internal) disagreed with that in the publication for the majority of trials [4]. Frequently, when trials had findings that were not statistically significant (p≥0.05) for the protocol-defined primary outcome in the internal documents, they were either not published in full or were published with a changed primary outcome favoring the company's drug [17]. Examination of internal company memos and other documents indicated that alterations in what was presented in publications were part of a “publication strategy” to disseminate trial findings and influence publication content [18]. Internal documents may serve as unique sources for evidence about company activities in pursuit of strategic goals, e.g., a company's marketing or publication strategy [18], [23] and on how corporations affect health, more generally [24], [25]. Over the past 20 years, internal company documents have become increasingly available, first from tobacco companies and later from pharmaceutical companies. While the existence of tobacco company document repositories is well known, little has been published about the range of primary health research uses for internal company documents across all industries, and whether there is a need for a repository or repositories for documents from industries other than tobacco has not been addressed in the literature.

Our objective was to describe the characteristics of public health and healthcare research using internal company documents across industries. The ultimate goal of our research was to document for others the potential sources of accessible internal company data for public health and healthcare research, particularly in the area of pharmaceutical research, and, in doing so, to take the first steps toward exploring the current use and future potential for repositories of internal company information.

Methods

Study eligibility

Articles were eligible if they described a study that addressed a research question or objective related to public health or healthcare, and internal company documents were explicitly referred to as the source of data (i.e., information) examined in the study. We considered internal company documents to include emails, memoranda, reports (including those with study data), presentations, meeting minutes, and other documents not originally intended to be publicly available. If documents were prepared for outside entities that were not employees or subcontractors, then we did not consider them to be internal company documents (e.g., we did not consider clinical trial protocols which are shared with institutional review boards and investigators to be internal company documents, nor did we include published research performed by company staff or contractors). We defined public health and healthcare in a broad sense to include studies of incidence, prevalence, etiology, prevention, diagnosis, harm, or prognosis, as well as any other studies concerning products or materials with health effects. Eligible studies could be qualitative (including descriptive and exploratory studies) or quantitative primary studies. While we recognize that systematic reviews may include additional information from internal company documents, we did not include them because this would have necessitated first, identifying all systematic reviews, and then second, checking each of them to see whether they used internal company documents.

Search methods

Initial search efforts.

Initially, we used the authors' combined file of articles meeting our eligibility criteria (n = 35 articles using pharmaceutical, tobacco and other industry documents) as a “reference set” against which we tested various search strategies. Four of the authors (NM, SSV, LR, and CNK) developed a search strategy using relevant Medical Subject Heading (MeSH) terms and title and abstract text words from the reference set. This initial search retrieved about 2 million citations in PubMed and, considering this too many citations to review, we started over.

Approach finally used to search electronically for eligible studies.

Working with an informationist (LMR), the team revised the PubMed search strategy by identifying a more targeted combination of keywords and MeSH headings from the reference set and running separate searches. Examples of MeSH headings included industry[majr], disclosure[mesh], and access to information[mesh], and keywords included terms such as “industry documents” (see Appendix S1 for search strategy). PubMed was searched on December 21, 2010 and updated PubMed and Embase searches were run on April 7, 2011. These searches, taken together, we termed “the search finally used” and it yielded 9,305 citations after removal of duplicate records. We did not restrict the search finally used by language, year of publication, or type of publication.

We also report in the Results section an additional search we did after this one, in an attempt to find more articles that used internal pharmaceutical company documents. We based this additional search on articles found by the electronic search strategy finally used, described above.

Screening, data extraction, and data analysis

To determine whether articles met inclusion criteria for our study, two authors independently screened the title and abstract of each citation and then independently examined the full text of each article considered unclear or possibly eligible as a result of screening. Differences in opinion were resolved through discussion.

Two authors independently extracted data from eligible English-language articles using a standardized online data extraction form. Articles not in English were assigned to a single data extractor with expertise in the language. We extracted information on the following items: language and date of publication, type of company (e.g., tobacco, pharmaceutical); source(s) of internal documents (e.g., litigation, U.S. Freedom of Information Act (FOIA) request); type of research question (e.g., about strategic behavior on the part of a company or industry, about effects of a therapeutic intervention); type of internal documents (e.g., research studies, internal memos); and funding source (e.g., government, non-profit, for-profit) for the study (see Appendix S2 for data abstraction form). Discrepancies in data extraction were resolved through discussion. One author extracted additional details about funding source(s) after the initial data abstraction was completed.

When an included article focused on research methods, one author classified the article into one of the following categories: 1) criticism of industry research (e.g., suggestion of misconduct or problems with dissemination of research), 2) exploration of company research methods that was not focused on criticism of the company, or 3) exploration of methods for accessing or analyzing internal company documents for the purposes of non-company research. A second author verified the classification with disagreements resolved through discussion.

One author collected additional details via email correspondence with authors of pharmaceutical research articles when the articles contained insufficient or unclear information about the location of internal company documents. These details were abstracted into a table, and a second author read the emails to verify the abstraction.

One author compared the results of our searches to known reference standards of research articles using internal company documents. For tobacco articles, we used the online Tobacco Documents Bibliography at the Tobacco Control Archives held by the University of California, San Francisco (UCSF) [26] as the reference standard. We compared articles classified in the Bibliography as “journal articles” and dated March 2011 or earlier against articles in our search results.

For research articles using internal company documents for other types of companies, we were not aware of a source we could use as a true reference standard. We were particularly interested in identifying articles using pharmaceutical company documents and therefore applied three methods to identify additional articles we might have missed through our electronic searches.

First, we used Science Citation Index – Web of Science to retrieve citations to our included pharmaceutical company articles, limiting citations to those published in March 2011 or earlier. Two authors independently screened the citation titles and abstracts for eligibility and independently examined the full text if eligibility was unclear or probable. All differences were resolved through discussion or consultation with a third author.

Second, one author visited the website of the Drug Industry Document Archive (DIDA) [27] and checked the Resources page (dida.library.ucsf.edu/resources.jsp) for potentially eligible journal articles published in March 2011 or before. A second author confirmed the eligibility classification.

Finally, we retrieved a few potentially eligible articles through ad hoc means (e.g., through authors of included articles using pharmaceutical documents), and two authors agreed upon final eligibility classification. Articles that we identified though these three methods, plus the pharmaceutical company studies we had already identified, were considered a “reference standard,” understanding that they were a more complete set of articles using internal pharmaceutical company documents and not a true reference standard.

We performed descriptive statistical analyses, including counts of the number of studies with different characteristics, and cross tabulations of the joint distribution of study characteristics.

Results

Our search of PubMed and Embase retrieved 9,305 unique records. After screening, 357 articles were classified as eligible for the study. Our searches of other sources to identify additional articles using pharmaceutical documents identified two from Science Citation Index – Web of Science, one from DIDA, and one via ad hoc contact with colleagues. We therefore analyzed a total of 361 articles (see Figure 1 and Appendix S3). The great majority of articles were conducted using internal tobacco company documents (325/361; 90%). Others used documents from pharmaceutical companies (20/361; 6%), manufacturing companies (9/361; 2%), mining companies (2/361; <1%), transportation companies (2/361; <1%), alcohol companies (1/361; <1%), and other companies (5/361; 1%) (see Table 1). Three articles reported studies that used documents from more than one type of company, as described in table legends. Studies were published between 1982 and April 2011. All but six articles (355/361; 98%) were published in English: two were in Spanish and one each in French, German, Portuguese, and Swedish.

thumbnail
Figure 1. Flow chart of articles through the screening process.

https://doi.org/10.1371/journal.pone.0094709.g001

thumbnail
Table 1. Source of documents for articles using internal documents from different types of companies (n = 361 articles).

https://doi.org/10.1371/journal.pone.0094709.t001

Most included articles relied upon internal company documents obtained through litigation (350/361; 97%) (see Table 1). The single tobacco company article not using documents from litigation used documents provided by a whistleblower, and those documents were later also released through litigation. Litigation was also the primary source of internal company documents from the pharmaceutical industry (18/20; 90%). Cooperation with the company being studied was the next most common source of documents (9/361; 2%).

The research questions posed in included articles were primarily about company strategies to promote or position the company and its products (326/361; 90%) (see Table 2 and Appendix S2). Research questions about research methods were posed only by tobacco or pharmaceutical company articles; all pharmaceutical company articles in this category focused upon company misconduct (9/9; 100%), as did the majority of tobacco company articles (18/31; 58%).

thumbnail
Table 2. Research questions for articles using internal documents from different types of companies (n = 361 articles).

https://doi.org/10.1371/journal.pone.0094709.t002

It was often difficult to identify the exact type of internal company documents used in the articles. Our interest was not in the format of the document but rather in the type of document information that appeared to have been used in the article. We were interested in whether the document contained quantitative study data and if so whether the data was produced by the company itself or by another entity acting on behalf of the company. We also wished to capture whether the document was the result of routine company activities. We therefore classified the type of document information as belonging to one or more of four categories: 1) quantitative study data from internal company studies (e.g., analysis or re-analysis of quantitative data from studies conducted by the company), 2) quantitative data from non-company studies (e.g., quantitative data quoted from market research conducted on behalf of a company), 3) data from company records collected as part of routine company activities (e.g., employee records), and 4) ‘other’ types of data, information generally from miscellaneous documents such as memos or letters, or from unspecified types of documents. Most studies in the review (344/361; 95%) were classified as using ‘other’ types of company data (see Table 3), with only 20% overall using quantitative data. Use of quantitative data varied quite a bit across the types of companies examined, however, with almost half (9/20) of studies using pharmaceutical company documents accessing quantitative data.

thumbnail
Table 3. Types of internal company data/information in articles using internal documents from different types of companies (n = 361 articles).

https://doi.org/10.1371/journal.pone.0094709.t003

Articles describing studies using tobacco company documents consistently referred to physical or online tobacco company document repositories as the location of the documents used in the study. Articles describing studies using pharmaceutical company documents did not have a consistent source of documents, and we investigated the current location of those documents. Of the 20 articles using internal pharmaceutical company documents, two used documents made available directly to the researchers by the company, and the remainder used documents released as a result of litigation (n = 18, including one case in which documents were leaked from litigation) (see Table 4). There is substantial overlap in the litigation, authors, and/or documents used in these articles, although this is sometimes because of multiple instances of related litigation, not because the same documents were used (e.g., Vedula 2009 examined documents from multiple Neurontin litigations; and separate documents from one of these litigations were also examined in Steinman 2006, Steinman 2007, and Landefeld 2009). The internal pharmaceutical company documents released as a result of litigation are not necessarily publicly available. Documents used in articles can be found in DIDA (9 articles); court records only (2, including 1 article where actual documents are not accessible although the report on the documents is accessible); and court records plus website (4 articles with active website links and 3 articles citing non-working links).

thumbnail
Table 4. Sources and location of pharmaceutical internal company documents.

https://doi.org/10.1371/journal.pone.0094709.t004

Explicit information about study funding was found in 290/361 (80%) articles, of which a small number (10/290; 3%) specified that the study had not been funded and 280 listed funding. Among the 280 articles describing funding received specifically for the study, the most common source of funding was the U.S. government, followed by non-profit organizations (see Table 5). The majority of studies reporting U.S. government funding reported that some or all of the government funding was from the National Cancer Institute (NCI) at the US National Institutes of Health (225/248; 91%), and aside from one study using documents from the nuclear industry, the NCI-funded studies all examined documents from the tobacco industry. Overall, 224/361 (62%) studies in this review used tobacco documents from litigation and were funded by the NCI. Seven articles using tobacco company documents, and one article using pharmaceutical company documents, mentioned that the authors or their institutions received funding to develop archives of documents related to their investigation.

thumbnail
Table 5. Types of funding reported by research articles using internal documents from different types of companies (n = 361articles).

https://doi.org/10.1371/journal.pone.0094709.t005

The source of our reference standard for tobacco studies, the Tobacco Documents Bibliography, contained 579 journal articles published in March 2011 or earlier. Our searches identified 337/579 (58%) of these records, of which 307/337 (91%) were deemed eligible and included in our study. Of the 242 remaining journal articles contained in the reference standard, 173 were indexed in PubMed or Embase and were not captured by our searches. On the other hand, 18/325 (6%) of the tobacco company records in our study were not included in the Tobacco Documents Bibliography.

Given the low number of citations reporting research using internal pharmaceutical company documents that were captured by the search we finally used, an informationist designed an additional strategy tailored to be more sensitive and to identify research using internal pharmaceutical company documents, and a second informationist reviewed the strategy. Eighteen (two later determined to be not eligible) pharmaceutical company research articles (15 PubMed records and 3 Embase) retrieved by our original search formed the basis for this “drug industry” search strategy. One author reviewed the reference lists of the 18 articles and selected references on the topic of internal pharmaceutical company documents (n = 53), and a colleague provided a list of additional related articles (n = 8). Keywords and MeSH terms from the 18 originally included articles, the 53 selected references, and the 8 related articles were combined into a more targeted and potentially more sensitive search strategy, which was run in PubMed. This search strategy captured 17/18 of the PubMed citations to the pharmaceutical company articles finally included in our study. To achieve this level of sensitivity, however, the new, more sensitive PubMed search identified 26,399 “hits”, of which 25,605 had not been identified by the previous search, and we decided that this was an unmanageable number for continuing to search for eligible articles.

Discussion

Internal company documents serve as a valuable source of information about industry activities for those who wish to know about the impact of those activities upon the health of the public. Internal documents from pharmaceutical companies include not only information on marketing and policy activities but also contain quantitative and other data related to clinical trials carried out on company products. Data from all trials are critical for a complete and accurate assessment of interventions within systematic reviews [15], [19], [28]. In response to this imperative, systematic reviewers and other healthcare stakeholders have been working hard towards making trial data held by companies publically available [29], [30] so that patients, providers, and policy makers can have a full picture of all that is known. Because internal company documents are not located in a single place, and they are not published or indexed in a bibliographic database, they are difficult to identify and locate. We elected to identify research articles using internal company documents across all industries as a starting point, knowing that we would likely leave many documents that are available for research unidentified.

What we learned, first, is that thousands of internal tobacco company documents, mainly released through litigation, are located in repositories around the world [31], including searchable online repositories (e.g., the Legacy Tobacco Documents Library at legacy.library.ucsf.edu and Tobacco Documents Online at tobaccodocuments.org) [32]. Ninety percent (325/361) of the research articles meeting our criteria used internal tobacco company documents and all but one of these used documents made available through litigation. The reference standard we used for articles employing internal tobacco documents indicated that our searches failed to find hundreds of additional, potentially eligible, studies of tobacco documents and that our searches also identified tobacco articles not included in the Tobacco Documents Library.

The second thing we learned was that identification of non-tobacco studies using internal company documents was harder than we had anticipated. Only 36/361 articles that we identified used non-tobacco sources, and more than half of these (20/36) were concerned with pharmaceutical company documents. We made every effort to ensure a thorough search of PubMed and Embase databases to retrieve all relevant documents, and to be practical we designed a search strategy that elected precision over sensitivity. It is possible that there are additional relevant articles from other non-tobacco industries (e.g., the chemical industry, the food and agricultural industry) that our search failed to retrieve. We need to identify better search terms for retrieving articles that use internal company documents, and consider consistent indexing of such articles. New machine learning approaches to searching databases may be a way to improve the retrieval of difficult-to-find articles as well. We also found that in contrast to tobacco company documents, which are contained in well-indexed repositories developed to facilitate public access to information, the pharmaceutical company documents are available in a range of sites, not all of which are well-known or accessible to the public. In addition, bibliographies of articles using internal pharmaceutical company documents, similar to the Tobacco Documents Bibliography, would greatly ease the identification of research using internal pharmaceutical company data.

These findings point to the importance of having one or more indexed and searchable repositories in place to assure comprehensive identification of internal company documents. Litigation has been an important source of internal company documents for research, and some documents from pharmaceutical company litigation have now been placed in DIDA; indeed, DIDA was started with funds from litigation. Nevertheless, the majority of pharmaceutical company documents in the studies we found were made available through websites (some no longer accessible), were obtained through collaboration with the company, or are court documents that one must know exist to be able to find. Comprehensive well-indexed and searchable repositories of internal company documents from pharmaceutical and other industries, similar to the repositories that exist for documents from the tobacco industry, are critical for the development of a program of research using other types of internal documents, including restorative authorship [21]. One or more repositories of internal documents from pharmaceutical companies, including trial data, will become critically important should the European Medicines Agency adopt the policy of requiring pharmaceutical company release of clinical trial data [33]. Until such time as well-populated, well-indexed, and well-publicized repositories are developed, the best advice for those seeking to find and use internal documents and related unpublished documents (e.g., regulatory documents) from pharmaceutical companies is to follow the activities outlined by Chan in 2012 [34].

The third thing we learned is that funding for research using internal company documents is uneven. Where there has been funding available, notably for the tobacco-related research, many important research projects have been conducted. Three-quarters of the tobacco research was funded by the U.S. government, primarily the NCI. Indeed, the NCI established a program of research and actively solicited researchers to develop projects using internal tobacco documents (e.g., http://grants.nih.gov/grants/guide/pa-files/PAR-01-063.html). From 2000 to 2007, the NCI provided a total of approximately $23 million to fund tobacco industry document research studies [personal written communication, Tobacco Control Research Branch Behavioral Research Program, Division of Cancer Control and Population Sciences, NCI, July 12 2013].

In contrast, research studies making use of internal pharmaceutical company documents have typically not been federally funded. Most articles we identified (13/20) reported no funding or had no explicit information about funding for the research, and only 3/20 reported US state or federal government funding. If, to date, only a handful of research studies have used internal pharmaceutical company documents, then it may be because of lack of available funding. Given the importance of research using internal tobacco documents to our current knowledge and views about tobacco and its health effects, a similar investment in other areas, including the pharmaceutical area, could also yield potentially important findings.

We do not know whether our initial search would have found more, fewer, or the same number of studies using pharmaceutical documents in our reference standard, if we had followed through and screened the over 2 million citations retrieved. It is possible that the number of research articles using pharmaceutical company documents is actually small and that we found most of them. We know that of the articles we identified, there was considerable overlap in documents, authors, and drugs examined. If we have identified most of the relevant articles, it highlights all there is to be gained by making all publicly available source documents (especially clinical study reports and datasets) accessible in one or a few locations, assuming this will prompt new research [22].

The studies of internal pharmaceutical company documents we identified, and others, have provided important signals for evidence-based medicine, indicating that the published literature, generally, is not always reliable and that much of what is known remains unpublished [4], [35], [36], [37], [38], [39], [40]. Society expects scientific studies to be conducted and disseminated following generally accepted tenets of scientific integrity and to adhere to a code of research ethics. We found that 9/20 studies using internal pharmaceutical company documents examined research methods used by the company, and all of these studies (9/9) were critical of the scientific and ethical integrity of the companies' research. Most of the research articles we identified examined strategies used by pharmaceutical companies to achieve commercial goals, which runs counter to scientific research goals.

While our particular interest in this project was pharmaceutical company documents, other company documents released through litigation or other means and potentially useful for health-related research and for setting governmental standards (e.g., regarding environmental hazards) should also be made centrally available to researchers. These collections of corporate documents should ideally be linked or merged, as companies often collaborate across industries (e.g., large corporations control both tobacco and alcohol companies) to promote their interests, often at the expense of public health [41]. Studies of these activities would be facilitated if searching could be done across several industries. Current methods for identifying internal company documents from litigation and other sources, with or without study data and CSRs, include word of mouth, unstructured searches of the internet, and, in the US, searching the Public Access to Court Electronic Records (PACER) system (www.pacer.gov). All of these methods are of uncertain reliability, sensitivity, and precision. Our findings are similar to those from almost 25 years ago that retrospective searching for unpublished trials was not useful, and that a comprehensive register of all initiated trials was needed [42].

Our study is limited by our focus on relevant research indexed in PubMed or Embase, and by a search date that is now more than three years in the past. However, while our study is limited by the possibility that we overlooked relevant research using internal company documents, including documents used in systematic reviews, we are able to conclude that our findings highlight the great need for well-indexed and curated repositories so that researchers can have ready access to internal company documents. The existing DIDA repository is a good start but additional funds are required to make it maximally useful to researchers. Each document in the repository needs to have consistent indexing information (metadata) such as title, author, date, bates number, and document type. This would either need to be provided (e.g., by the plaintiffs' attorneys) or a vendor would have to be hired to create it. In addition, funding is needed for DIDA's ongoing curation to support, for example, information science and programming staff. Linking repositories and bibliographies (e.g., unpublished data in systematic reviews) should be explored, as well as linking these sources and registers of studies (e.g., ClinicalTrials.gov and the Cochrane Register of Studies). The research articles we identified relating to tobacco industry documents are a testament to how information in internal company documents can contribute to improving the community's understanding of enhancing transparency in communicating research findings.

Acknowledgments

Many thanks to colleagues who read articles in non-English languages to assess eligibility: Isabelle Boutron, Peter Doshi, Stephan Ehrhardt, Tom Jefferson, Karmela Krleza-Jeric, Tianjing Li, Joerg Meerpohl, Isabel Rodriguez-Barraquer, Rob Scholten, Gerard Urrutia-Cuchi.

Thanks also to authors of eligible studies who filled in details as necessary of where they obtained the internal company documents their articles referenced. Thanks to Tom Greene for advising on potential ways of accessing federal court papers.

Author Contributions

Conceived and designed the experiments: LR SSV KD. Analyzed the data: LSW. Wrote the paper: LSW KD. Developed and carried out search strategies: LR SSV CNK LMR CT NM. Screened articles for eligibility: LSW LR SSV CNK NM KD. Created and revised data abstraction form: LSW KD. Abstracted data from eligible articles: LSW LR SSV CNK LMR CT KD. Communicated with article authors and others to confirm information on location of internal documents and identify additional eligible articles: KD. Contributed to final edits on manuscript: LSW LR SSV CNK LMR CT NM KD.

References

  1. 1. Song F, Parekh S, Hooper L, Loke YK, Ryder J, et al.. (2010) Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess 14 : iii, ix-xi, 1–193.
  2. 2. Turner EH, Knoepflmacher D, Shapley L (2012) Publication bias in antipsychotic trials: an analysis of efficacy comparing the published literature to the US Food and Drug Administration database. PLoS Med 9: e1001189.
  3. 3. Dickersin K, Chalmers I (2011) Recognizing, investigating and dealing with incomplete and biased reporting of clinical research: from Francis Bacon to the WHO. J R Soc Med 104: 532–538.
  4. 4. Vedula SS, Bero L, Scherer RW, Dickersin K (2009) Outcome reporting in industry-sponsored trials of gabapentin for off-label use. N Engl J Med 361: 1963–1971.
  5. 5. Dickersin K (1997) How important is publication bias? A synthesis of available data. AIDS Educ Prev 9: 15–21.
  6. 6. Lundh A, Sismondo S, Lexchin J, Busuioc OA, Bero L (2012) Industry sponsorship and research outcome. Cochrane Database Syst Rev 12: MR000033.
  7. 7. Dickersin K, Min YI, Meinert CL (1992) Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. JAMA 267: 374–378.
  8. 8. Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG (2004) Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 291: 2457–2465.
  9. 9. Decullier E, Lheritier V, Chapuis F (2005) Fate of biomedical research protocols and publication bias in France: retrospective cohort study. BMJ 331: 19.
  10. 10. Dickersin K, Min YI (1993) Publication bias: the problem that won't go away. Ann N Y Acad Sci 703: 135–146 discussion 146–138.
  11. 11. Chan AW, Krleza-Jeric K, Schmid I, Altman DG (2004) Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ 171: 735–740.
  12. 12. Simes RJ (1986) Publication bias: the case for an international registry of clinical trials. J Clin Oncol 4: 1529–1541.
  13. 13. Dwan K, Altman DG, Cresswell L, Blundell M, Gamble CL, et al.. (2011) Comparison of protocols and registry entries to published reports for randomised controlled trials. Cochrane Database Syst Rev: MR000031.
  14. 14. Rising K, Bacchetti P, Bero L (2008) Reporting bias in drug trials submitted to the Food and Drug Administration: review of publication and presentation. PLoS Med 5: e217 discussion e217.
  15. 15. Doshi P, Jefferson T, Del Mar C (2012) The imperative to share clinical study reports: recommendations from the Tamiflu experience. PLoS Med 9: e1001201.
  16. 16. Krumholz HM, Ross JS, Gross CP, Emanuel EJ, Hodshon B, et al. (2013) A historic moment for open science: the Yale University Open Data Access Project and Medtronic. Ann Intern Med 158: 910–911.
  17. 17. Vedula SS, Li T, Dickersin K (2013) Differences in reporting of analyses in internal company documents versus published trial reports: comparisons in industry-sponsored trials in off-label uses of gabapentin. PLoS Med 10: e1001378.
  18. 18. Vedula SS, Goldman PS, Rona IJ, Greene TM, Dickersin K (2012) Implementation of a publication strategy in the context of reporting biases. A case study based on new documents from Neurontin litigation. Trials 13: 136.
  19. 19. Jefferson T, Jones MA, Doshi P, Del Mar CB, Heneghan CJ, et al. (2012) Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database Syst Rev 1: CD008965.
  20. 20. Hurt RD, Ebbert JO, Muggli ME, Lockhart NJ, Robertson CR (2009) Open doorway to truth: legacy of the Minnesota tobacco trial. Mayo Clin Proc 84: 446–456.
  21. 21. Doshi P, Dickersin K, Healy D, Vedula SS, Jefferson T (2013) Restoring invisible and abandoned trials: a call for people to publish the findings. BMJ 346: f2865.
  22. 22. Doshi P, Jefferson T (2013) Clinical study reports of randomised controlled trials: an exploratory review of previously confidential industry reports. BMJ Open 3..
  23. 23. Hill KP, Ross JS, Egilman DS, Krumholz HM (2008) The ADVANTAGE seeding trial: a review of internal documents. Ann Intern Med 149: 251–258.
  24. 24. Freudenberg N (2012) The manufacture of lifestyle: the role of corporations in unhealthy living. J Public Health Policy 33: 244–256.
  25. 25. McDaniel PA, Malone RE (2009) The role of corporate credibility in legitimizing disease promotion. Am J Public Health 99: 452–461.
  26. 26. The UCSF Tobacco Documents Bibliography. Available: www.library.ucsf.edu/tobacco/docsbiblio (Accessed 17 January 2013): UCSF Library and Center for Knowledge Management.
  27. 27. Drug Industry Document Archive. Available: http://dida.library.ucsf.edu/ (Accessed 11 February 2013): UCSF Library and Center for Knowledge Management.
  28. 28. Doshi P, Jones M, Jefferson T (2012) Rethinking credible evidence synthesis. BMJ 344: d7898.
  29. 29. The Cochrane Collaboration (2013) Cochrane Collaboration announces formal support for AllTrials initiative. News release 19 April 2013. Available: http://www.cochrane.org/node/4906 (Accessed 7 July 2013).
  30. 30. Food and Drug Administration (2013) Availability of Masked and De-identified Non-Summary Safety and Efficacy Data; Request for Comments. Federal Register. Available: https://www.federalregister.gov/articles/2013/06/04/2013-13083/availability-of-masked-and-de-identified-non-summary-safety-and-efficacy-data-request-for-comments (Accessed 7 July 2013).
  31. 31. Anderson SJ, McCandless PM, Klausner K, Taketa R, Yerger VB (2011) Tobacco documents research methodology. Tob Control 20 Suppl 2ii8–11.
  32. 32. WHO (2004) The tobacco industry documents. What they are, what they tell us, and how to search them. Available: http://www.who.int/tobacco/communications/TI_manual_content.pdf (Accessed 12 February, 2013).
  33. 33. Publication and access to clinical-trial data: an inclusive development process. Available: http://www.ema.europa.eu/ema/index.jsp?curl=pages/special_topics/general/general_content_000556.jsp. (Accessed December 15, 2013).
  34. 34. Chan AW (2012) Out of sight but not out of mind: how to search for unpublished clinical trial evidence. BMJ 344: d8013.
  35. 35. Psaty BM, Furberg CD, Ray WA, Weiss NS (2004) Potential for conflict of interest in the evaluation of suspected adverse drug reactions: use of cerivastatin and risk of rhabdomyolysis. JAMA 292: 2622–2631.
  36. 36. Steinman MA, Bero LA, Chren MM, Landefeld CS (2006) Narrative review: the promotion of gabapentin: an analysis of internal industry documents. Ann Intern Med 145: 284–293.
  37. 37. Jureidini JN, McHenry LB, Mansfield PR (2008) Clinical trials and drug promotion: Selective reporting of study 329. International Journal of Risk and Safety in Medicine 20: 73–81.
  38. 38. McHenry LB, Jureidini JN (2008) Industry-sponsored ghostwriting in clinical trial reporting: a case study. Account Res 15: 152–167.
  39. 39. Psaty BM, Kronmal RA (2008) Reporting mortality findings in trials of rofecoxib for Alzheimer disease or cognitive impairment: a case study based on documents from rofecoxib litigation. JAMA 299: 1813–1817.
  40. 40. Ross JS, Hill KP, Egilman DS, Krumholz HM (2008) Guest authorship and ghostwriting in publications related to rofecoxib: a case study of industry documents from rofecoxib litigation. JAMA 299: 1800–1812.
  41. 41. White J, Bandura A, Bero LA (2009) Moral disengagement in the corporate world. Account Res 16: 41–74.
  42. 42. Hetherington J, Dickersin K, Chalmers I, Meinert CL (1989) Retrospective and prospective identification of unpublished controlled trials: lessons from a survey of obstetricians and pediatricians. Pediatrics 84: 374–380.
  43. 43. Applbaum K (2009) Getting to yes: corporate power and the creation of a psychopharmaceutical blockbuster. Cult Med Psychiatry 33: 185–215.
  44. 44. Spielmans GI (2009) The promotion of olanzapine in primary care: an examination of internal industry documents. Soc Sci Med 69: 14–20.
  45. 45. Woods S, Tek C, Srihari VH (2011) Adverse drug effects not detected at licensing review: Regulatory autopsy of olanzapine. Schizophrenia Bulletin 37: 30.
  46. 46. Breggin PR (2006) Court filing makes public my previously suppressed analysis of Paxil's effects. Ethical Hum Psychol Psychiatry 8: 77–84.
  47. 47. Jureidini JN, McHenry LB (2011) Conflicted medical journals and the failure of trust. Account Res 18: 45–54.
  48. 48. Krumholz HM, Ross JS, Presler AH, Egilman DS (2007) What have we learnt from Vioxx? BMJ 334: 120–123.
  49. 49. Ross JS, Madigan D, Konstam MA, Egilman DS, Krumholz HM (2010) Persistence of cardiovascular risk after rofecoxib discontinuation. Arch Intern Med 170: 2035–2036.
  50. 50. Steinman MA, Harper GM, Chren MM, Landefeld CS, Bero LA (2007) Characteristics and impact of drug detailing for gabapentin. PLoS Med 4: e134.
  51. 51. Landefeld CS, Steinman MA (2009) The Neurontin legacy—marketing through misinformation and manipulation. N Engl J Med 360: 103–106.
  52. 52. Fugh-Berman AJ (2010) The haunting of medical journals: how ghostwriting sold “HRT”. PLoS Med 7: e1000335.
  53. 53. Bernschneider-Reif S, Oxler F, Freudenmann RW (2006) The origin of MDMA (“ecstasy”)—separating the facts from the myth. Pharmazie 61: 966–972.
  54. 54. Greene JA (2005) Releasing the flood waters: diuril and the reshaping of hypertension. Bull Hist Med 79: 749–794.