http://orcid.org/0000-0002-4861-826X
Figures
Abstract
Objective
To examine how the first randomised controlled trials (RCTs) evaluating the efficacy of cholesterol-lowering diets in the secondary prevention of coronary heart disease were interpreted in reviews of the literature prior to the National Institutes of Health consensus conference in 1984.
Design
Claim-specific citation network analysis was used to study the network of citations between reviews and RCTs over a defined period (1969–1984). RCTs were identified and classified according to whether their conclusions supported or opposed the use of dietary fat modification/restriction in the secondary prevention of coronary heart disease. Each review published in this period that cited any of the RCTs was classified as supportive, neutral, or unsupportive to the use of dietary fat modification based on a quotation analysis of its evaluation of the findings of these RCTs. Citation bias and underutilisation were detected by applying a comparative density measure, in-degree centrality, and out-degree in a series of sub-graph analyses.
Results
In total, 66 unique publications were identified (four RCTs—one supportive, three unsupportive; 62 reviews—28 supportive, 17 neutral, 17 unsupportive). On average, supportive reviews underutilised the available RCTs to a greater degree than other reviews. Amongst the supportive group, citation bias was common—23 (82%) reviews cited only the one RCT that was supportive.
Citation: Leng RI (2018) A network analysis of the propagation of evidence regarding the effectiveness of fat-controlled diets in the secondary prevention of coronary heart disease (CHD): Selective citation in reviews. PLoS ONE 13(5): e0197716. https://doi.org/10.1371/journal.pone.0197716
Editor: Iratxe Puebla, Public Library of Science, UNITED KINGDOM
Received: November 6, 2017; Accepted: April 30, 2018; Published: May 24, 2018
Copyright: © 2018 Rhodri Ivor Leng. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: This work was supported by the Economic and Social Research Council (ESRC) doctoral studentship and the ESRC’s Advanced Quantitative Methods (AQM) enhanced stipend (URL: http://www.esrc.ac.uk/skills-and-careers/studentships/esrc-students/advanced-quantitative-methods-aqm-enhanced-stipend/). The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The author has declared that no competing interests exist.
Introduction
In 1961, the American Heart Association published guidelines recommending that people who have had one or more atherosclerotic heart attacks should reduce their intake of saturated fatty acids and modestly increase that of polyunsaturated fatty acids [1]. The rationale was an early version of the diet–heart hypothesis, that dietary intake of saturated fatty acids, by increasing total serum cholesterol, contributes to coronary heart disease (CHD) via the development of atherosclerosis, while polyunsaturated fatty acids decrease serum cholesterol and lower the risk of CHD [2]. In the 1950s, this theory was supported by: (i) epidemiological evidence of an association between fat intake, particularly saturated fatty acids intake, and raised serum cholesterol levels; (ii) an association between serum cholesterol and coronary incidence and mortality; (iii) short-term metabolic ward trials that suggested that saturated fatty acids increased serum cholesterol while polyunsaturated fatty acids appeared to modestly lower serum cholesterol [3].
However, some scientists warned of the dangers of mistaking correlation for causation, particularly in population studies [4] and in a multifactorial disease such as CHD [5, 6]. By the time that the guidelines were issued, no randomised controlled trial (RCT) had tested the link between the consumption of different types of fat and coronary risk in secondary prevention (i.e., after a clinical manifestation of CHD), but, in the 1960s, five RCTs examined this. The Los Angeles Veterans Administration Diet Study [7] combined secondary prevention and primary prevention (i.e., before the development of CHD), but was generally regarded as relevant only to primary prevention due to the way the authors stratified their data and presented their findings. Four other trials specifically examined the effect of a modified diet in the secondary prevention of CHD in patients who had experienced a recent myocardial infarction: the Research Committee Low-Fat Diet Study [8] examined total fat restriction, while the Rose Corn Oil Study [9], the Oslo Diet-Heart Study [10], and the Medical Research Council’s Soya-bean Oil Study [11] examined the impact of restricting saturated fatty acids and increasing polyunsaturated fatty acid intake. These were the only published secondary prevention RCTs until the partial publication of the Sydney Diet–Heart Study [12] in 1978, which only published results concerning all-cause mortality. The full results of that trial on cardiovascular disease (CVD) and CHD mortality were recently recovered and re-analysed by Ramsden et al. [13], who found a significant harmful effect of replacing saturated fatty acids with polyunsaturated fatty acids on all-cause mortality, CVD mortality, and CHD mortality.
The four RCTs published in 1965 [8, 9], 1966 [10], and 1968 [11] had similar methodologies, and were published in major journals, but produced discrepant findings (S1–S4 Tables). The Oslo study, published in 1966, found significantly lower levels of fatal reinfarction, non-fatal reinfarction, and angina pectoris in the intervention group, but found no significant relationship between diet and sudden death or all-cause mortality [10]. The other three studies all returned equivocal or negative results regarding reinfarction, coronary mortality, cardiovascular events, and all-cause mortality. The author of the Oslo study regarded his results as supporting dietary fat modification in secondary prevention, while the authors of the other studies regarded their results as evidence against such treatments.
In December 1984, the National Institutes of Health (NIH) gathered a panel of experts to resolve whether the link between raised serum cholesterol and heart attack was causal, and under what circumstances dietary or drug therapy should be initiated [14]. After reviewing the available evidence, the panel concluded that an elevated serum cholesterol level is a cause of CHD and that lowering it is protective. To lower serum levels of cholesterol, the panel recommended that all men, women, and children over 2 years of age should eat a diet consisting of no more than 30% fat, should reduce their intake of saturated fatty acids to less than 10% of total calories, and should increase their intake of polyunsaturated fatty acids to 10% of total calories. Specifically, on secondary prevention, the panel concluded:
“Studies are available that indicate a beneficial effect of treating high cholesterol levels in individuals with preexisting clinical disease (secondary intervention) … It is encouraging that the progression of established lesions may be retarded by appropriate dietary and drug therapy” (p.2084)
The conference was widely regarded as bringing a (temporary) end to the controversy over both the relationship of cholesterol with CHD and the relationship between dietary fats and CHD [15].
Yet, by 1984, the four RCTs discussed previously remained the only available RCTs with data on the effect of dietary modification on coronary mortality and morbidity in secondary prevention. However, the results of these trials were conflicting–an evaluation supported both by the directors of each study and by recent meta-analyses [16–19]. According to Harcombe et al. [16], the NIH advice was not supported by the available RCTs, neither in primary or secondary prevention. As can be seen in the risk-ratio analysis in the supplement (S4 Table), the evidence from these RCTs available prior to 1984 leads to no clear conclusion as to whether dietary modification is an appropriate treatment in secondary prevention. Recent meta-analyses also agree that, when the findings of these four trials are considered together, no significant difference exists on the “hard” clinical end-points–the incidence of myocardial infarction, fatal CHD, and all-cause mortality–between the intervention and control [16–19].
The present study examines how these four trials were utilised in the literature and their impact on published scientific opinion regarding the case for dietary fat modification/restriction in the secondary prevention of CHD, prior to the NIH consensus conference. By using a modified version of claim-specific citation network analysis, a methodology developed by Greenberg [20], it maps and evaluates the network of review papers that cited these RCTs over a defined period (1969–1984); the date by which all relevant RCT data would have been available to reviewers to the start of the NIH conference. The rationale for examining how these studies were cited is the claim that the diet–heart controversy stems from selective citation [21, 22]. In 1992, Uffe Ravnskov, in one of the first published demonstrations of citation bias, examined 22 cholesterol-lowering trials and found that supportive trials were cited almost six times as often as others [21]. In three highly-cited reviews associated with dietary policy [23–25], he went on to document citation bias and quotation bias, which describes the erroneous or selective interpretation of experimental findings [22]. Finally, in a response to an article in the BMJ in 2003, he claimed that belief in the diet–heart hypothesis was kept alive by selective citation [26].
The present study tested these claims by examining how dietary RCT results were cited in reviews during the period leading up the publication of dietary guidelines and closure of controversy, following Greenberg’s [20] finding that reviews tend to be the major propagators of information regarding the findings of primary studies. To do so, it examines two outcomes of selective citation: citation bias–the selective citing of data supporting a particular outcome [20]; and research underutilisation–when authors fail to reference all available data relevant to a specific claim. These are outcomes of sub-optimal citation practices that distort the evidence base by inflating the influence of some studies while under-representing others [20, 27]. To detect these factors, network analysis was used to document citation bias and underutilisation by mapping the citation links between RCTs and reviews that were themselves classified by quotation analysis.
Materials and methods
Study selection
To be included, a trial had to be a unifactorial RCT that: (i) was published before 1984; (ii) examined the effect of dietary fat restriction (<30% of total calories derived from dietary fat) OR dietary fat modification (replacing saturated fatty acids with polyunsaturated fatty acids) on coronary benefit/risk in the secondary prevention of CHD; (iii) published data on sample numbers AND serum cholesterol changes AND mortality and morbidity figures regarding incidence of myocardial infarction, coronary mortality (including myocardial infarction and sudden death), and all-cause mortality; (iv) included only patients with established CHD (myocardial infarction, angina pectoris); (v) a minimum intervention and follow-up period of 12 months.
RCTs were identified via a literature search on Web of Science via the ‘all database’ function with the following Boolean string (“dietary fat” OR “fatty acids” OR “saturated fat” OR “low fat diet” OR “cholesterol lowering diet” OR “modified fat diet” OR “restricted fat diet” OR “corn oil” OR “soya bean oil” OR “unsaturated fat” OR “linoleic acid”) AND (“coronary heart disease” OR “ischaemic heart disease” OR “myocardial infarction” OR “cardiovascular disease” OR “atherosclerotic heart disease”) AND (“trial” OR “controlled trial” OR “randomised controlled trial” OR “intervention trial” OR “clinical trial”), limited to publications between 1950 and 1984. This returned 244 publications across four databases in March 2017 (MEDLINE, Web of Science Core Collection, BIOSIS Citation Index, FSTA® - the Food Science Resource). An additional examination of trials identified by recent systematic reviews and meta-analyses was also conducted to ensure all relevant studies were identified [16–19]. After screening publication titles and abstracts, 19 trials were identified as potentially relevant. After reading these, four RCTs were identified that met the inclusion criteria, all of which were published in the decade after the publication of the American Heart Association guidelines. The findings of these four RCTs were published in seven papers: the results of the Oslo Trial were published in four journals [10, 28–30]. These were merged, reducing the primary source studies to four. The results of these trials are included in the analysis in the supplement (S1–S4 Tables).
One trial, the Los Angeles Veterans Administration Diet Study [7], was omitted due to the inability to assess data on the effectiveness of dietary modification in secondary prevention due to the way the authors stratified their data and reported their findings. This was further justified by the prevailing opinion of the scientific community as expressed in reviews, which appeared to regard it relevant only to primary prevention, and its exclusion from meta-analyses covering secondary prevention specifically [18]. Another trial, the Sydney Diet–Heart Study [12], was omitted because only results on all-cause mortality were reported within the time frame and because it remained uncited in the literature by reviews until 1985 [30]. The Finnish Mental Hospital Study [31], a commonly-cited study in this period, was excluded on the basis that it was not an RCT; it was a “cross-over” trial in two mental hospitals and combined secondary and primary prevention. Further, it was not a standard cross-over trial: the patient groups were not the same in the diet arm and the control arm, no blinding was mentioned, and investigators re-coded deaths post-hoc. Bierenbaum et al.’s [32] St Vincent’s Hospital Study was excluded because of the lack of appropriate randomisation due to the assigning of a control group post-hoc that had “similar characteristics” to the intervention. Five trials were excluded on the basis of being primary prevention trials [33–35] and/or multifactorial [36–37], and six early secondary prevention trials were excluded due to a lack of appropriate randomisation and/or control [38–43].
Review papers, published in academic journals in English between 1969 and 1984 and which cited one or more of the studies, were identified by searching Scopus and Web of Science using indexed citation links associated with the relevant RCTs. To be included, a paper had to be a review or an extended editorial (with at least one citation) published in an academic journal within which the coronary mortality and morbidity results of the identified RCTs were discussed. During this period, only narrative reviews covered this topic, and none described a systematic search strategy. Excluded from analysis were letters, notes, book chapters, other primary articles, any paper not published in an academic journal, and any review that cited the RCTs without discussing mortality and morbidity. The search in Scopus returned 110 citing articles, of which 30 met the inclusion criteria. The search on Web of Science returned 215 citing articles, from which an additional 32 reviews were recovered that met the inclusion criteria. In total, 62 citing reviews were included in the analysis [44–105].
Classification of papers
The RCTs were classified using ‘quotation analysis’ [20, 22, 27] according to whether the findings were evidence for or against the use of dietary fat modification/restriction in the secondary prevention of CHD. Quotations from the papers were collected to enable papers to be classified by the authors’ interpretation of their findings and these are reproduced in the supplement (S5 Table). For example, the authors of the Rose Corn Oil Study, an unsupportive trial, concluded that “under the circumstances of this trial corn oil cannot be recommended in the treatment of ischaemic heart disease” (p. 1533). By contrast, the author of the Oslo Study, the only supportive trial, concluded that the observed “reduction of the serum cholesterol level associated with a reduced CHD relapse rate strongly suggests a cause and effect relationship” (p. 79).
Reviews were similarly classified by quotation analysis by the authors’ evaluation of the RCT evidence. Quotations were identified either in the passage in which RCTs were cited or were stated explicitly in a conclusion, and these are reproduced in the supplement (S5 Table). Reviews were then classified into one of three categories: supportive reviews contained statements implying that the available trial evidence supported dietary intervention; neutral reviews contained statements highlighting the conflicting nature of the trial results and contained no statement either supporting or opposing dietary intervention; unsupportive reviews contained statements implying that the available trial evidence did not support the use of dietary intervention.
Network data
Each paper was given an identifier code (Vn). All authors, title of paper, year of publication, paper type (RCT or review), and classification were recorded to identify each vertex in a ‘node list’ format (S6 Table). This attribute information was used to examine the relationship between classification variables and citations behaviours. The citations to (incoming edges) and from (outgoing edges) all papers were recorded. The bibliographies of each review were screened to ensure that all citations to RCTs were identified within this sample. Citation information on the relationship between papers was recorded in an ‘edge list’ format in an Excel spreadsheet (S7 Table). This describes a network as a set of vertices (papers) connected by ‘edges’ (citation links). From these data, a graph was constructed and analysed using R’s ‘statnet’ package [106, 107] and visualised in Gephi [108]. Citation information was converted into an adjacency matrix that records edges (citations) between vertices (papers):
A graph [G] was constructed of the vertex set [V] and edge set [E]:
The vertices were coloured by classification and sized by in-degree, labelled by number assigned in the bibliography of this paper, and visualised initially using Gephi’s ‘ForceAtlas 2’ algorithm [108, 109] and later manually realigned in sub-graphs.
Two outcome measures–citation bias and research underutilisation–were used. Following Greenberg [20], citation bias refers to the act of citing only evidence of a particular classification. For example, if paper A references only supportive studies and no contradictory studies then it displays citation bias. Citation bias only describes the most extreme cases of evidence selection, but any underutilisation may create distortions in the literature–i.e., if reviews fail to use all relevant evidence even if a clear preference for a particular outcome does not influence selection. To detect these, the unit of analysis is the citation pathway, and the occupied citation pathway (an edge linking two vertices) is compared to all possible citation pathways between reviews and RCTs. Three measures are used to measure citation bias and underutilisation–a novel modified density measure, in-degree centrality, and out-degree.
The network in this study is an example of a directed acyclic network [110] because papers can only cite previously published papers and can never cite future literature or cite themselves. To know the density, the proportion of potential edges present, a measure of network connectivity is needed. Generally, if P is a set of primary studies, S a set of reviews, and E the number of edges from S to P, then
captures the number of times that a set of reviews cites a set of RCTs as a proportion of the maximum possible number of citations, allowing comparison between different classifications of review. Thus, this equation can be used to understand divergent utilisation of primary data.
In-degree reflects how often a study has been cited. The in-degree of a vertex is the number of edges from other vertices to that vertex, and, for a vertex i, is given by:
Out-degree reflects the number of references from a particular review. The out-degree of a vertex is the number of edges from that vertex to other vertices, given by:
To establish citation bias, the network was divided into three sub-graphs of the RCTs with reviews of each classification (supportive, neutral, unsupportive) and only edges from reviews to RCTs. The vertices that cite only RCTs of a particular classification were then counted–providing an exact account of citation bias and research utilisation.
Finally, a Pearson’s chi square test was performed on the distribution of citations to these four RCTs between the three classifications of review to establish whether the difference in citation behaviour is significant or simply the product of chance.
Results
Of the four RCTs, one was supportive of dietary intervention [10] in the context of secondary prevention, while three were not [8, 9, 11]. Between 1969 and 1984, these were cited in 62 reviews, 28 of which were supportive of dietary intervention [78–105], 17 were unsupportive [61–77], and 17 ‘neutral’ reviews came to no clear conclusion on this issue [44–60]. Of these, 28 were published in the first 7 years (12 supportive; 12 neutral; four unsupportive), and 34 in the next 8 years (16 supportive; five neutral; 13 unsupportive).
Fig 1 shows the citation network. Of the four RCTs, the most cited was the Oslo study with 57 citations. This was the only one of the four RCTs that was supportive of dietary intervention, but the other three (unsupportive) RCTs together attracted a similar number of citations (32 to MRC Soya-bean, 15 to Rose Corn, 17 to Research Committee Low-fat). Here, citation bias, particularly by supportive reviews, is signified by the many studies that cite only the Oslo study. Clearly, the distribution of citations to these studies is uneven. A high proportion of the reviews (27 of 62; 43%) cited only one of the RCTs, 20 cited two of them, six cited three and only nine cited all four. Nevertheless, taking the three unsupportive trials together, supportive trials are cited 57 times and unsupportive trials 64 times. Overall the graph has a density D of 0.49 –indicating only 49% of the possible citations exist between reviews and these RCTs.
Citation network graph with four RCTs connected to 62 citing reviews by a total of 121 citations. Each vertex represents a paper and the integer label corresponds to a paper in this paper’s bibliography. The colour of a vertex corresponds to a study’s classification (blue is supportive, green is neutral, and red is unsupportive). Vertex size is proportional to in-degree (total number of incoming citations). Each edge corresponds to a specific citation link and the direction of the edge is established by the direction of the arrow head. Visualised with Gephi’s ‘Force Atlas 2’ algorithm [109].
To create an exact account of the utilisation of the primary RCT data by reviews, three sub-graphs were constructed including only reviews of a particular classification (supportive, neutral, unsupportive) and only edges from reviews to RCTs.
Neutral reviews
Of the neutral reviews (Fig 2), one cites only supportive data, which represent the only case of citation bias in this group. Overall the density of this sub-graph (D = 0.68) is higher than that of the network as a whole, indicating that reviews which cited more of the available primary literature were more likely to come to no clear conclusion than the average. While the Oslo Study received most citations (17), its influence is nearly rivalled by the MRC Soya-bean Study, which received 15; the Rose Corn Oil Study and the Research Committee Low-fat Study were each cited seven times. Six of the reviews cited all four RCTs and only one cited only one of them. Thus, compared to the whole network, neutral reviews were more likely to fully utilise the available RCT evidence and less likely to cite only one RCT.
This graph is composed of four RCTs connected to 17 citing reviews by a total of 46 citations. Each vertex represents a paper and the integer label corresponds to a paper in this paper’s bibliography. The colour of a vertex corresponds to a study’s classification (blue is supportive, green is neutral, and red is unsupportive). Vertex size is proportional to in-degree (total number of incoming citations). Each edge corresponds to a specific citation link, the direction of the edge is established by the direction of the arrow head, and the colour of an edge indicates the classification of the cited article. This was visualised manually in Gephi [108] by lining up reviews in the centre, placing supportive the RCT below and the unsupportive RCTs on top.
Unsupportive reviews
The sub-graph of unsupportive reviews (Fig 3) had a density of 0.56. The Oslo study and the MRC study were each cited 12 times by unsupportive reviews, the Rose study six times and the Low-fat study eight times. Five of the 17 unsupportive reviews cited only evidence that supported their position. Three reviews cited only the supportive trial, implying a difference of interpretation between the reviewers and the authors of that study. Therefore, in total eight reviews (47%) exhibit citation bias for trials of a particular outcome. Three reviews cited just one RCT and two cited all four; most (nine) cited two of the RCTs.
This graph is composed of four RCTs connected to 17 reviews by a total of 38 citations. Each vertex represents a paper and the integer label corresponds to a paper in the reference list. The colour of a vertex corresponds to a study’s classification (blue is supportive and red is unsupportive). Vertex size is proportional to in-degree (total number of incoming citations). Each edge corresponds to a specific citation link, the direction of the edge is established by the direction of the arrow head, and the colour of an edge indicates the classification of the cited article.
Supportive reviews
Compared with the neutral and unsupportive reviews, the 28 supportive reviews (Fig 4) used much less of the available primary data (D = 0.33 vs 0.56 in unsupportive reviews and 0.68 in neutral reviews). All 28 cited the single supportive RCT, the Oslo study, while the three other RCTs were cited only nine times. In all, 23 of the supportive reviews (82%) cited only the Oslo study, and only five of these reviews cited any of the other RCTs (two cited one other, two cited two others, and just one cited all four RCTs).
This graph is composed of four RCTs connected to 28 citing reviews by a total of 37 citations. Each vertex represents a paper and the integer label corresponds to a paper in the reference list. The colour of a vertex corresponds to a study’s classification (blue is supportive and red is unsupportive). Vertex size is proportional to in-degree (total number of incoming citations). Each edge corresponds to a specific citation link, the direction of the edge is established by the direction of the arrow head, and the colour of an edge indicates the classification of the cited article.
Table 1 gives the distribution of citations to RCTs from different classifications of review.
A Pearson’s chi square test was performed to test whether this distribution is significantly related to classification or the product of chance. The validity of a chi square test depends on the conditions that the expected count is greater than 5 for at least 80% of the cells, and that each cell has a count of at least 1: these conditions were both satisfied. The test result establishes that the difference observed in the citation behaviours between reviews is significant (Χ 2(6, N = 121) = 18.19, p = 0.006).
Discussion
This study examined the propagation of experimental evidence derived from RCTs in reviews of the literature during a period of scientific controversy. This was not a minor academic dispute; this was an issue of considerable public health importance at the time. It documented a major divide between reviews over what the available empirical evidence was, how it should be evaluated, and what the implications were for dietary intervention in the secondary prevention of CHD. Overwhelmingly, reviews that disseminated a supportive evaluation of the results of RCTs in the context of secondary prevention cited only data that supported this position. They did not critique the discrepant data, they simply ignored it, or were ignorant of its existence. Other reviews were overall much more balanced in their citation and this difference in citation behaviour appears to explain divergence of opinion on this matter.
While the Oslo trial, a supportive RCT, was by far the most influential, its influence was not uniform amongst different classifications of review. This was the only RCT to arrive at statistically significant findings. A number of studies have reported evidence of a bias in the literature, in that studies with statistically significant results tend to be cited more often than studies that report non-significant or equivocal results [21, 111–114]. However, such a bias for significant outcomes fails to explain the discrepancies between the citing behaviours of different groups.
The degree of utilisation of primary data by reviews appeared to be mediated by a review’s position, with supportive reviews likely to use less of the available RCT evidence than others. Again, this may be explained by a preference for statistically significant findings, but, as discussed, there are problems with this explanation. An alternative hypothesis may be that the shape of the literature itself determines the selection of RCT evidence–where reviewers find evidence from other papers. In this case, if a reviewer believes another paper to have identified all relevant evidence, they may replicate selective citation. This has been described by Greenberg’s information cascade hypothesis–where chains of citations from one paper to another lead researcher’s to only a segment of the available evidence, if those authors simply follow bibliographies [20]. This hypothesis may be supported by Simkin and Roychowdhury’s [115] finding that on average only 20% of the cited literature in bibliographies has been read by the authors. If scientists do tend to gain their information about primary studies by reading other papers rather than the originals, this is particularly worrying in this case due to the amount of bias in the selection of evidence by reviews and the degree of underutilisation. Collectively, the citing reviews studied in this paper have been cited 2,157 times (by August 2017) according to Web of Science (supportive reviews 651; neutral 489; unsupportive 1017); however, without further analysis, no statement can be made about what influence these reviews had on the literature.
Thirdly, citation bias was considerably more prevalent in supportive reviews. This seems to be an example of homophily—the tendency for a vertex to share an edge with a vertex with which it shares an attribute, and is consistent with Heider’s [116] theory that individuals choose relationships that reinforce existing preferences and beliefs. However, this fails to explain the unsupportive reviews, which we would expect to only cite evidence that supports their perspective. We did not observe this.
Fourthly, few reviews cited all relevant data–with underutilisation common. In the period concerned, conventions governing systematic search strategies were in their infancy. The reviews were mostly of the narrative form, a type that has been criticised for selection bias and the promotion of partial perspectives of the evidence. Indeed, one finding of this paper is that reviews in this field were poor at considering all the available empirical evidence. Bushman and Wells [117] offer experimental evidence of the role of availability bias (a heuristic that describes the tendency of humans to judge the prevalence/probability of an event by the ease in which they are able to recall certain information), which the authors argue influence the propensity for selection bias in narrative reviews compared with systematic reviews and meta-analysis, which impose a rigorous structure of literature searching and study evaluation on the researcher. Importantly, narrative reviews are not the only papers in which citation bias has been documented, Robinson and Goodman [118] documented significant citation bias amongst RCTs addressing the same question and may be explained by the lack of systematic search strategies amongst these papers.
Finally, did the selection of evidence itself explain the divergence of opinion amongst reviews? That is, were these authors only aware of the data they cited? If so, it would seem those underutilising the literature arrive at positions that are based on partial perspectives. Or did pre-existing opinions held by reviewers determine their selection of evidence? That is, were opinions formed first and evidence selected afterwards? If so, this would be a case of confirmation bias. Here, we see the complexities involved in attempting to determine which direction the arrow of causality points–is evidence formative in scientific opinion? Or do pre-existing opinions dictate the evidence we choose to examine?
The major findings offer an alternative to the prevailing notion of what a scientific controversy is, i.e. that scientific disagreement stems from different theoretical positions, where researchers looking at the same evidence can justifiably come to different conclusions [119]. Disagreement in this instance appears to stem from the selection of different sources of evidence, rather than the possession of different theoretical perspectives. Trinquart et al. [27] came to similar conclusions when examining the controversy about the suggested benefits of restricting dietary salt intake. They observed “two almost distinct and disparate lines of scholarship, one supporting and one contradicting the hypothesis” (p.57)–lines that relied on different sources of evidence and which differed in their citing behaviour.
Implications
Scientists, at present, are concerned at a wide range of factors that they understand to be distorting science. These include publication bias [120, 121], problems with replication [122], and citation biases [20, 21, 27, 111–114]. Their combined effect has undermined faith in the idea that our sciences are self-correcting, and increasingly commentators fear that our system of knowledge production is broken. De Solla Price [123] and later Ravetz [124] raised concerns that as science grew it may experience ‘senility’–with the evidence base and citing literature growing too large for effective quality control, but it is a view that is gaining increasing attention [125].
This study focused on citation bias, a structural bias that leads to distortions of authority, whereby some positions are held up by the inflated authority of supporting primary studies and where scientists do not interact with, and in some cases may be unaware of, rival perspectives and unsupportive data [20]. This can lead to polarisation, where different scientists draw from different sources of evidence and arrive at opposing conclusions to similar questions [27]. Unlike these two studies, which examined recent citation practices, the present study focused on a historic analysis of divergent citation practices amongst papers drawing different conclusions as to what the available empirical evidence supports. Citation bias and research underutilisation were exactly documented, as were the different citation behaviours of different classifications of review.
This case study was chosen because it represents a unique example where the relevant RCTs were published in quick succession (1965–1968), and these remained the only RCTs addressing this particular question prior to a major consensus building conference in 1984. This provided a 17-year citation window, from the time all results of the RCTs were available to researchers to the beginning of this conference, where the evidence-base, as derived from RCTs, remained constant (i.e. no additional secondary prevention RCTs were published that would have impacted the evidence-base and, possibly, citation behaviours). Further, as these RCTs were the first to test this particular treatment, the relevant evidence-base and citation behaviour could not have been influenced by prior studies, at-least not evidence derived from RCTs, which may have distorted the results. This provided a unique opportunity to study how scientists utilise a small evidence-base over a long and undisturbed period. One surprising findings is that, even with a small evidence-base, scientists did not generally cite these studies in a representative manner nor did their citation behaviours improve as this period unfolded, and this challenges our understanding that it may be the size of the literature responsible for this phenomenon.
Therefore, the major findings of this study demonstrate that citation bias is not a new phenomenon–challenging narratives that suggest our science has become increasingly less reliable in recent years because science has grown too large–and that scientific disagreement may revolve around something more simple than divergent theoretical positions–scientists may just be looking at different sources of data.
The results of this study should not be taken as making claims about the current state of knowledge regarding the efficacy of dietary intervention in secondary prevention. Nor should it be used to fuel the current controversy over scientific advice concerning dietary fat consumption. Indeed, the selection of a historical cases study was, in part, chosen to avoid this paper being mistaken for attempting to weigh in on what particular treatment is suitable in secondary prevention. Since the 1980s, much work has been conducted in this area and the results of this study do not impact the scientific case. Rather, the aim was to provide a deeper understanding of the ways in which empirical data is used by scientists, how it may fuel scientific disagreement, and to introduce a method capable of detecting this particularly efficiently. Its focus was on citation practices and evidence dissemination, not the validity of current scientific theories.
Supporting information
S1 Table. Study characteristics of four RCTs examining dietary fat restriction/modification in the secondary prevention of CHD.
A comparison of the Oslo Diet–Heart Study, Rose Corn Study, Research Committee Low-fat Study, and Medical Research Council’s MRC Soya-bean Oil Trial.
https://doi.org/10.1371/journal.pone.0197716.s001
(DOCX)
S2 Table. Intervention and diet characteristics of four RCTs examining dietary fat restriction/modification in the secondary prevention of CHD.
A comparison of the Oslo Diet–Heart Study, Rose Corn Study, Research Committee Low-fat Study, and Medical Research Council’s MRC Soya-bean Oil Trial.
https://doi.org/10.1371/journal.pone.0197716.s002
(DOCX)
S3 Table. A comparison of the incidence of myocardial infarction, CHD mortality (myocardial infarction + sudden death), combined fatal + non-fatal CHD events, all-cause mortality, and changes in total serum cholesterol in four secondary prevention RCTs.
A comparison of the Oslo Diet–Heart Study, Rose Corn Study, Research Committee Low-fat Study, and Medical Research Council’s MRC Soya-bean Oil Trial.
https://doi.org/10.1371/journal.pone.0197716.s003
(DOCX)
S4 Table. A risk-ratio analysis of the results of four secondary prevention RCTs.
A comparison of the Oslo Diet–Heart Study, Rose Corn Study, Research Committee Low-fat Study, and Medical Research Council’s MRC Soya-bean Oil Trial.
https://doi.org/10.1371/journal.pone.0197716.s004
(DOCX)
S5 Table. A quotation analysis of four RCTs and 62 citing reviews.
https://doi.org/10.1371/journal.pone.0197716.s005
(DOCX)
S6 Table. Attribute data on all studies (vertices) included in this study.
https://doi.org/10.1371/journal.pone.0197716.s006
(DOCX)
S7 Table. Edge data on all citations between reviews and RCTs included in this study.
https://doi.org/10.1371/journal.pone.0197716.s007
(DOCX)
Acknowledgments
I would like to thank Professor Steve Sturdy, Dr Valeria Skafida, Dr Gil Viry, Professor Steve Yearley, Professor Gareth Leng, and Dr Sharon Hollingsworth for their time and feedback on this manuscript.
References
- 1. Page IH, Allen EV, Chamberlain FL, Keys A, Stamler J, Stare F. Dietary fat and its relation to heart attacks and strokes. Circulation. 1961; 23: 133–136.
- 2. Keys A, Anderson JT, Grande F. Prediction of serum-cholesterol responses of man to changes in fats ill the diet. Lancet. 1957; 273(7003): 959–966. pmid:13482259
- 3. Jolliffe N. Fats, cholesterol, and coronary heart disease: a review of recent progress. Circulation. 1959; 20: 109–127 pmid:13663199
- 4. Yerushalmy J, Hilleboe HE. Fat in the diet and mortality from heart disease; a methodologic note. N Y State J Med. 1957; 57: 2343–2354. pmid:13441073
- 5. Mann GV. The epidemiology of coronary heart disease. Am J Med. 1957; 23: 463–480. pmid:13458210
- 6. Yudkin J. Diet and coronary thrombosis. Hypothesis and fact. Lancet. 1957; 273: 155–62. pmid:13450357
- 7. Dayton S, Pearce ML, Hashomoto S, Dixon WJ, Tomiyasu U. A controlled clinical trial of a diet high in unsaturated fat in preventing complications of atherosclerosis. Circulation. 1969; 40: II-1–II-63.
- 8. Rose GA, Thomson WB, Williams RT. Corn oil in treatment of ischaemic heart disease. BMJ. 1965; 1: 1531–1533. pmid:14288105
- 9. Committee Research. Low-fat diet in myocardial infarction: a controlled trial. Lancet. 1965; 2: 501–504. pmid:4158171
- 10. Leren P. The effect of plasma-cholesterol-lowering diet in male survivors of myocardial infarction. A controlled clinical trial. Acta Med Scand Suppl. 1966; 466: 1–92. pmid:5228820
- 11. Medical Research Council. Controlled trial of soya-bean oil in myocardial infarction: Report of a research committee to the Medical Research Council. Lancet. 1968; 292: 693–700.
- 12. Woodhill JM, Palmer AJ, Leelarthaepin B, McGilchrist C, Blacket RB. Low fat, low cholesterol diet in secondary prevention of coronary heart disease. Adv Exp Med Biol. 1978; 109: 317–330. pmid:727035
- 13. Ramsden CE, Zamora D, Leelarthaepin B, Majchrzak-Hong SF, Faurot KR, Suchindran CM et al. Use of dietary linoleic acid for secondary prevention of coronary heart disease and death: evaluation of recovered data from the Sydney Diet Heart Study and updated meta-analysis. BMJ. 2013; 346: e8707 pmid:23386268
- 14. National Institutes of Health Consensus Development Program Consensus conference. Lowering blood cholesterol to prevent heart disease. JAMA. 1985; 253: 2080–2086.
- 15. Garrety K. Social worlds, actor-networks and controversy: the case of cholesterol, dietary fat and heart disease. Soc Stud Sci. 1997; 21: 727–773.
- 16. Harcombe Z, Baker JS, Cooper SM, Davies B, Sculthorpe N, DiNicolantonio JJ et al. Evidence from randomised controlled trials did not support the introduction of dietary fat guidelines in 1977 and 1983: a systematic review and meta-analysis. Open Heart. 2015; 2(1): e000196.
- 17. Hooper L, Martin N, Abdelhamid A, Davey Smith G. Reduction in saturated fat intake for cardiovascular disease. Cochrane Database Syst Rev. 2015;(6): CD011737. pmid:26068959
- 18. Schwingshackl L, Hoffmann G. Dietary fatty acids in the secondary prevention of coronary heart disease: a systematic review, meta-analysis and meta-regression. BMJ Open 2014; 4: e004487. pmid:24747790
- 19. Hamley S. The effect of replacing saturated fat with mostly n-6 polyunsaturated fat on coronary heart disease: a meta-analysis of randomised controlled trials. Nutr J. 2017; 16: 30. pmid:28526025
- 20. Greenberg SA. How citation distortions create unfounded authority: analysis of a citation network. BMJ. 2009; 339:b2680 pmid:19622839
- 21. Ravnskov U. Cholesterol lowering trials in coronary heart disease: frequency of citation and outcome. BMJ. 1992; 305: 15–19. pmid:1638188
- 22. Ravnskov U. Quotation bias in reviews of the diet-heart idea. J Clin Epidemiol. 1995; 48: 713–719. pmid:7730926
- 23. Kannel WB, Doyle JT, Ostfeld AM, Jenkins CD, Kuller L, Podell RN et al. Optimal resources for primary prevention of atherosclerotic diseases. Atherosclerosis study group. Circulation. 1984; 70: 157A–205A.
- 24.
National Research Council Diet and Health. Implications for Reducing Chronic Disease Risk. Washington, DC: National Academy Press, 1989.
- 25. LaRosa JC, Hunninghake D, Bush D, Criqui MH, Getz GS, Gotto AM et al. The cholesterol facts. A summary of the evidence relating dietary fats, serum cholesterol, and coronary heart disease. A joint statement by the American Heart Association and the National Heart, Lung, and Blood Institute. Circulation. 1990; 81(5): 1721–1733. pmid:2184951
- 26. Ravnskov U. The diet-heart idea is kept alive by selective citation. BMJ. 2003; 327: 1348.
- 27. Trinquart L, Johns DM, Galea S. Why do we think we know what we know? A metaknowledge analysis of the salt controversy. Int J Epidemiol. 2016; 45: 251–260. pmid:26888870
- 28. Leren P. The effect of plasma-cholesterol-lowering diet in male survivors of myocardial infarction. A controlled clinical trial. Bull N Y Acad Med. 1968; 44: 1012–1020. pmid:4875746
- 29. Leren P. Plasma cholesterol lowering diet. The effect in male survivors of myocardial infarction. Minn Med. 1969; 52: 1231–1235. pmid:5804990
- 30. Leren P. The Oslo diet-heart study. Eleven-year report. Circulation. 1970; 42:935–942. pmid:5477261
- 31. Turpeinen O, Karvonen MJ, Pekkarinen M, Miettinin M, Elosuo R, Paavilainen E. Dietary prevention of coronary heart disease: the Finnish mental hospital study, Int J Epidemiol. 1979; 8: 99–118. pmid:393644
- 32. Bierenbaum ML, Fleischman AI, Green DP, Raichelson RI, Hayton T, Watson PB et al. The 5-year experience of modified fat diets on younger men with coronary heart disease. Circulation. 1970; 42: 943–952. pmid:5477262
- 33. National Diet-Heart Study Research Group. The National Diet-Heart Study Final Report. Circulation. 1968; 37(3 Suppl.): I1–428.
- 34. Christakis G, Rinzler SH, Archer M, Winslow G, Jampel S, Stephenson J et al. The anti-coronary club. A dietary approach to the prevention of coronary heart disease—a seven-year report. Am J Public Health Nations Health 1966; 56: 299–314. pmid:5948223
- 35. Frantz ID, Dawson EA, Kuba K, Brewer ER, Gatewood LC, Bartsch GE. The Minnesota coronary survey: Effect of diet on cardiovascular events and deaths. American Heart Association Scientific Proceedings. Circulation. 1975; 41 (Suppl. 2): 2–4.
- 36. Hjermann T, Velve-Byre K, Holme I, Leren P. The effect of diet and smoking intervention on the incidence of coronary heart disease. Lancet. 1981; 2(8259): 1303–1310. pmid:6118715
- 37. Multiple Risk Factor Intervention Trial Research Group. Multiple risk factor intervention trial. Risk factor changes and mortality results. JAMA. 1982; 248: 1465–1477. pmid:7050440
- 38. Morrison LM. Diet in coronary atherosclerosis. J Am Med Assoc. 1960; 173:884–888. pmid:14424373
- 39. Lyon TP, Yankley A, Gofman JW, Strisower B. Lipoproteins and diet in coronary heart disease. Calif Med. 1956; 84: 325–328. pmid:13316532
- 40. Pilkington TR, Stafford JL, Hankin VS, Simmonds FM, Koerselman HB. Practical diets for lower serum lipids. Brit Med J. 1960; 1(5165): 23–25. pmid:20788847
- 41. Hood B, Sanne H, Örndahl G, AhlstÖm M, Welin G. Long-term prognosis in essential hypercholesterolemia. Actu Med Scand. 1965; 178: 161–173.
- 42. Hansen PF, Geill T, Lund E. Dietary Fats and Thrombosis. Lancet 1962; 2:1193–1194. pmid:13952618
- 43. Nelson AM. Treatment of atherosclerosis by diet. I. Results in patients followed from 36 to 72 months. Northwest Med. 1965; 55: 643–649.
- 44. Cornfield J, Mitchell S. Selected risk factors in coronary disease. Possible intervention effects. Arch Environ Health. 1969; 19: 382–394. pmid:5807741
- 45. Brown DF. Blood lipids and lipoproteins in atherogenesis. Am J Med. 1969; 46: 691–704. pmid:4892338
- 46. Walker AR. Can expectation of life in Western populations be increased by changes in diet and manner of life? II. S Afr Med J. 1969; 43: 768–775. pmid:4895907
- 47. Simborg DW. The status of risk factors and coronary heart disease. J Chron Dis 1970; 22: 515–552. pmid:5437449
- 48. Zelis R, Mason DT, Spann JF. The hyperlipoproteinemias. A simplified classification and approach to therapy. Calif Med. 1970; 112: 32–37. pmid:4924875
- 49. Logue RB, Robinson PH. Medical management of angina pectoris Circulation. 1972; 46(6): 1132–1145. pmid:4404551
- 50. Stormorken H. Epidemiological studies. Relation between dietary fat and arterial and venous thrombosis. Haemostasis. 1973; 2:1–12. pmid:4597983
- 51. Jacobson NL. The controversy over the relationship of animal fats to heart disease. BioScience. 1974; 24(3): 141–148.
- 52. Howard AN. Hypolipaemic drugs and coronary heart disease. J Clin Pathol Suppl. (R Coll Pathol.) 1975; 9: 106–114.
- 53. Shaper AG. Primary and secondary prevention trials in coronary heart disease. Postgrad Med J. 1976; 52: 464–469. pmid:967765
- 54. Kritchevsky D. Diet and atherosclerosis. Am J Pathol 1976; 84: 615–632. pmid:786036
- 55. Yeshurun D, Gotto AM. Drug treatment of hyperlipidemia. Am J Med. 1976; 60: 379–396. pmid:1258886
- 56. Lewis B. Hypothesis into theory—the development of aetiological concepts of ischaemic heart disease: a review. J R Soc Med. 1978; 71: 809–818. pmid:215767
- 57. Hulley SB, Sherwin R, Nestle M, Lee PR. Epidemiology as a guide to clinical decisions II. Diet and coronary heart disease. West J Med. 1981; 135: 25–33. pmid:7257376
- 58. Hulley SB, Lo B. Choice and use of blood lipid tests. An epidemiologic perspective. Arch Intern Med. 1983; 143: 667–673. pmid:6340621
- 59. Goldman GL, Pichard AD. The natural history of coronary artery disease: does medical therapy improve the prognosis? Prog Cardiovasc Dis. 1983; 25: 513–552. pmid:6133314
- 60. Stallones RA. Ischemic heart disease and lipids in blood and diet. Annu Rev Nutr. 1983; 3: 155–185. pmid:6357234
- 61. Dayton S, Pearce ML. Prevention of coronary heart disease and other complications of arteriosclerosis by modified diet. Am J Med. 1969; 46: 751–762. pmid:4892340
- 62. Lees RS, Wilson DE. The treatment of hyperlipidemia. N Engl J Med. 1971; 284: 186–195 pmid:4923684
- 63. Carnie JA. Diet and heart disease. Sci Prog. 1972; 60: 231–243. pmid:4581384
- 64. Ahrens EH. The management of hyperlipidemia: whether, rather than how. Ann Intern Med. 1976; 85: 87–93. pmid:779574
- 65. Borhani NO. Primary prevention of coronary heart disease: a critique. Am J Cardiol. 1977; 40: 251–259. pmid:195453
- 66. Mann GV. Diet-heart: end of an era. N Engl J Med. 1977; 297: 644–650. pmid:197407
- 67. Mann GV. Coronary heart disease—the doctor’s dilemma. Am Heart J. 1978; 96: 569–571. pmid:263389
- 68. Mann JI. Fats and atheroma: a retrial. Br Med J. 1979; 1(6165): 732–734. pmid:435754
- 69. McMichael J. Fats and atheroma: an inquest. Br Med J. 1979; 1(6157): 173–175. pmid:105776
- 70. McMichael J. Fats and arterial disease. Am Heart J. 1979; 98: 409–412. pmid:484428
- 71. Walker AR. Dietary fat intake and serum cholesterol levels in coronary heart disease. S Afr Med J. 1980; 58(1): 7–12 pmid:6996145
- 72. Inkeles S, Eisenberg D. Hyperlipidemia and coronary atherosclerosis: a review. Medicine. 1981; 60(2): 110–123. pmid:7012540
- 73. Oliver MF. Serum cholesterol—the knave of hearts and the joker. Lancet. 1981; 2(8255): 1090–1095. pmid:6118533
- 74. May GS, Eberlein K, Furberg CD, Passamani ER, DeMets DL. Secondary prevention after myocardial infarction: a review of long-term trials. Prog Cardiovasc Dis. 1982; 24: 331–352. pmid:6119737
- 75. Buckwald H, Fitch L, Moore RB. Overview of randomized clinical trials of lipid intervention for atherosclerotic cardiovascular disease. Control Clin Trials. 1982; 3: 271–283. pmid:6759035
- 76. Coates ME. Reviews of the progress of dairy science: dietary lipids and ischaemic heart disease. J Dairy Res. 1983; 50: 541–557. pmid:6358290
- 77. Oliver MF. Why measure cholesterol after myocardial infarction, and when? Br Med J. (Clin Res Ed.) 1984; 289: 1641–1642.
- 78. Malmros H. Dietary prevention of atherosclerosis. Lancet. 1969; 2(7618): 479–484. pmid:4185482
- 79. Rathmann DM, Stockton JR, Melnick D, Stare FJ. Dynamic utilization of recent nutritional findings: Diet and cardiovascular disease. CRC Crit Rev Food Technol. 1970; 1: 331–338.
- 80. Little JA. Fats in Adult Nutrition. Can J Public Health. 1971; 62: 27–35. pmid:4925957
- 81. Vergroesen AJ. Dietary fat and cardiovascular disease: possible modes of action of linoleic acid. Proc Nutr Soc. 1972; 31: 323–329. pmid:4579353
- 82. Christakis G, Rathmann D. Position Statement on Diet and Coronary Heart Disease. American Health Foundation. Prev Med. 1972; 1: 255–286. pmid:4560604
- 83. Stamler J. Epidemiology of coronary heart disease. Med Clin North Am. 1973; 57(1): 5–46. pmid:4569832
- 84. Anderson JT, Grande F, Keys A. Cholesterol-lowering diets. Experimental trials and literature review. J Am Diet Assoc. 1973; 62: 133–142. pmid:4739366
- 85. Mcbean LD, Spechmann EW. An interpretive review: diet in early life and the prevention of atherosclerosis. Pediatr Res. 1974; 8: 837–842. pmid:4606213
- 86. Renaud S. Dietary fats and arterial thrombosis. Thromb Res. 1974; 4(Suppl. 1):25–35.
- 87. Grande F. Diet and Atherosclerosis. S Afr Med J. 1974; 48: 1660–1668. pmid:4604321
- 88. Glueck CJ, Mattson F, Bierman E. Diet and coronary heart disease: another view. N Engl J Med. 1978; 298: 1471–1473. pmid:206833
- 89. Abrahamson JH, Hopp C. The control of cardiovascular risk factors in the elderly. Prev Med. 1976; 5: 32–47. pmid:772670
- 90. Kannel WB. Prevention of cardiovascular disease. Curr Probl Cardiol. 1976; 1:1–68.
- 91. Grundy SM. Treatment of hypercholesterolemia. Am J Clin Nutr. 1977; 30: 985–992. pmid:326023
- 92. Hegsted DM. Dietary goals-a progressive view. Am J Clin Nutr. 1978; 31: 1504–1509. pmid:28662
- 93. Eaton RP. High density lipoprotein—key to anti-atherogenesis. J Chronic Dis. 1978; 31: 131–135. pmid:209070
- 94. Steinberg D. Research related to underlying mechanisms in atherosclerosis. Circulation. 1979; 60: 1559–1565. pmid:387283
- 95. Glueck CJ, Connor WE. Diet and atherosclerosis: past, present and future. West J Med. 1979; 130: 117–122. pmid:371145
- 96. Hegsted DM. Optimal nutrition. Cancer. 1979; 43(5 Suppl.): 1996–2003. pmid:445385
- 97. Nora JJ. Identifying the child at risk for coronary disease as an adult: A strategy for prevention. J Pediatr. 1980; 97: 706–714. pmid:7000993
- 98. Hymowitz N. Behavioural approaches to preventing heart disease: risk factor modification. Int J Mental Health. 1980; 9: 27–69.
- 99. Hornstra G. Dietary prevention of coronary heart disease. Effect of dietary fats on arterial thrombosis. Postgrad Med J. 1980; 56: 563–570. pmid:7008047
- 100. Stamler J. The established relationship among diet, serum cholesterol and coronary heart disease Acta Med Scand. 1980; 207: 433–446. pmid:7424562
- 101. Steinberg D. Underlying mechanisms in atherosclerosis. J Pathol. 1981; 133: 75–87. pmid:7009813
- 102. Hornstra G. The role of (n-6) fatty acids and their metabolites in arterial thrombus formation. Prog Lipid Res. 1981; 20: 407–413. pmid:6804970
- 103. Grundy SM, Bilheimer D, Blackburn H, Brown WV, Kwiterovjch PO, Mattson F et al. Rationale of the Diet-Heart Statement of the American Heart Association. Report of Nutrition Committee. Circulation. 1982; 65(4): 839A–854A pmid:7060268
- 104. Council of Scientific Affairs. Dietary and pharmacologic therapy for the lipid risk factors. JAMA. 1983; 250: 1873–1879. pmid:6620484
- 105. Furberg CD, May GS. Effect of long-term prophylactic treatment on survival after myocardial infarction. Am J Med. 1984; 76: 76–83. pmid:6146258
- 106.
R Development Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria, 2008 [cited May 2017]. Available from: http://www.R-project.org.
- 107. Handcock M, Hunter D, Butts C, Goodreau S, Krivitsky P, Bender-deMoll S et al. ‘statnet: Software Tools for the Statistical Analysis of Network Data’. The Statnet Project, 2003 [cited April 2017]. Available from: http://www.statnet.org. in R package version 2016.9, Stable URL: CRAN.R-project.org/package=statnet.
- 108.
Bastian M, Heymann S, Jacomy M. Gephi: an open source software for exploring and manipulating networks. International AAAI Conference on Weblogs and Social Media, 2009 [cited April 2017]. Available from: https://gephi.org/users/publications/.
- 109. Jacomy M, Venturini T, Heymann S, Bastian M. ForceAtlas2, a Continuous Graph Layout Algorithm for Handy Network Visualization Designed for the Gephi Software. PLoS One. 2014; 9(6): e98679. pmid:24914678
- 110.
Newman MEJ. Networks: An Introduction. Oxford, Oxford University Press, 2010.
- 111. Gøtzsche PC. Reference bias in reports of drug trials. Br Med J. (Clin Res ed.) 1987; 295: 654–656.
- 112. Nieminen P, Rucker G, Miettunen J, Carpenter J, Schumacher M. Statistically significant papers in psychiatry were cited more often than others. J Clin Epidemiol. 2007; 60: 939–946. pmid:17689810
- 113. Fanelli D. Positive results receive more citations, but only in some disciplines. Scientometrics. 2013; 94: 701–709.
- 114. Misemer BS, Platts-Mills TF, Jones CW. Citation bias favoring positive clinical trials of thrombolytics for acute ischemic stroke: a cross-sectional analysis. Trials. 2016; 17: 473. pmid:27677444
- 115. Simkin MV, Roychowdhury VP. Read before you cite! Complex Syst. 2003; 14: 269–274.
- 116.
Heider F. The Psychology of Interpersonal Relations. Hoboken, NJ, USA: John Wiley & Sons, 1958.
- 117. Bushman BJ, Wells GL. Narrative impressions of literature: The availability bias and the corrective properties of meta-analytic approaches. Pers Soc Psychol Bull. 2001; 27: 1123–1130.
- 118. Robinson KA, Goodman SN. A systematic examination of the citation of prior research in reports of randomized, controlled trials. Ann Intern Med. 2011; 154: 50–55. pmid:21200038
- 119.
Kuhn T. The Structure of Scientific Revolutions (2nd edn.). Chicago; University of Chicago Press, 1962 [1970].
- 120. Young NS, Ioannidis JPA, Al-Ubaydli O. Why current publication practices may distort science. PLoS Med 2008; 5(10): e201. pmid:18844432
- 121. Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ et al. Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess. 2010; 14(8): iii, ix-xi, 1–193. pmid:20181324
- 122. Ioannidis JP. Why most published research findings are false. PLoS Med. 2005; 2(8): e124. pmid:16060722
- 123.
de Solla Price DJ. Little Science, Big Science. New York: Columbia. University Press, 1963.
- 124.
Ravetz JR. Scientific Knowledge and Its Social Problems. Oxford: Clarendon Press, 1971.
- 125. Parolo DB, Pan K, Ghosh R, Huberman R, Kimmo BA, Kaski K et al. Attention decay in science. J Informetr. 2015; 9: 734–745.