Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Open Access

Peer-reviewed

Research Article

Patient-important outcomes in systematic reviews: Poor quality of evidence

  • Youri Yordanov ,

    Roles Conceptualization, Data curation, Formal analysis, Methodology, Resources, Visualization, Writing – original draft, Writing – review & editing

    youri.yordanov@aphp.fr

    Affiliations INSERM, U1153, Paris, France, Sorbonne Universités, UPMC Paris Univ-06, Paris, France, Service des Urgences - Hôpital Saint Antoine, Assistance Publique–Hôpitaux de Paris (APHP), Paris, France, Centre d’Épidémiologie Clinique, Hôpital Hôtel Dieu, Assistance Publique–Hôpitaux de Paris (APHP), Paris, France

    ⨯
  • Agnes Dechartres,

    Roles Conceptualization, Data curation, Formal analysis, Methodology, Supervision, Validation, Writing – original draft, Writing – review & editing

    Affiliations INSERM, U1153, Paris, France, Centre d’Épidémiologie Clinique, Hôpital Hôtel Dieu, Assistance Publique–Hôpitaux de Paris (APHP), Paris, France, Faculté de Médecine, Université Paris Descartes, Sorbonne Paris Cité, Paris, France, Cochrane France, Paris, France

    ⨯
  • Philippe Ravaud

    Roles Conceptualization, Formal analysis, Methodology, Supervision, Writing – review & editing

    Affiliations INSERM, U1153, Paris, France, Centre d’Épidémiologie Clinique, Hôpital Hôtel Dieu, Assistance Publique–Hôpitaux de Paris (APHP), Paris, France, Faculté de Médecine, Université Paris Descartes, Sorbonne Paris Cité, Paris, France, Cochrane France, Paris, France, Columbia University, Mailman School of Public Health, Department of Epidemiology, New York, United States of America

    ⨯

Abstract

Background

Cochrane reviewers are strongly encouraged to evaluate the quality of evidence for the most important outcomes by using the GRADE approach and to report these results in a Summary of Findings (SoF) table. We aimed to assess whether outcomes reported in the SoF table of Cochrane reviews could be considered patient-important outcomes (PIOs) and the quality of the available evidence for these outcomes.

Methods

We performed a methodological review of Cochrane reviews published between March 2011 and September 2014. For a random sample of Cochrane reviews reporting a SoF table, we extracted all outcomes reported in this table and evaluated whether they could be considered PIOs (i.e., mortality, other clinical events, adverse events, function, pain, quality of life and therapeutic decisions). Then, we collected the quality of evidence for every outcome in these SoF tables.

Results

We included 290 reviews issued by 47 of the 53 Cochrane Review Groups. Every SoF table included a median of 5 outcomes, for a total of 1414 outcomes; 1089 (77%) could be considered PIOs. Almost all reviews (n = 278, 96%) included at least one PIO in their SoF table. The quality of evidence for the outcomes was high for 12% (n = 168), moderate for 28% (n = 402) and low or very low for 45% (n = 640). Less than one quarter of reviews (n = 63) included at least one PIO with high-quality evidence that favoured a benefit of the experimental intervention evaluated in half of them (n = 34 reviews).

Conclusions

Many outcomes reported in the SoF table of recent Cochrane reviews can be considered PIOs. However, the quality of available evidence remains limited for these outcomes.

Citation: Yordanov Y, Dechartres A, Ravaud P (2018) Patient-important outcomes in systematic reviews: Poor quality of evidence. PLoS ONE 13(4): e0195460. https://doi.org/10.1371/journal.pone.0195460

Editor: Gianni Virgili, Universita degli Studi di Firenze, ITALY

Received: November 14, 2017; Accepted: March 22, 2018; Published: April 5, 2018

Copyright: © 2018 Yordanov et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: The data have been uploaded as Supporting Information files.

Funding: The authors received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

Introduction

In the last decade, clinicians and researchers have been encouraged to recommend healthcare interventions based on their effect on patient-important outcomes (PIOs) such as death, other clinical events, quality of life or functional outcomes[1–6]. However, several methodological reviews reported that the use of PIOs in randomized controlled trials (RCTs) of various medical specialities is still far from optimal [7–10]. In a recent report assessing trials of critically ill patients, less than one quarter of primary outcomes (24%) were PIOs[9]. This proportion was as low as 5% when exploring outcomes besides mortality after intensive care unit discharge, such as functional disability or quality of life[9]. Similar results were previously found for diabetes[8] or cardiovascular [7] trials, with 18% and 23% of these trials assessing PIOs. In contrast, some recent studies suggested that systematic reviews are more likely to evaluate PIOs than their individual trials[10, 11].

A key component of systematic reviews is the elaboration of a clear and focused clinical question by specifying the types of participants, interventions, comparisons, and outcomes that should be considered[12, 13]. When planning a Cochrane review, authors are strongly encouraged to consider all outcomes that are meaningful to patients, physicians, policy makers or any other health care stakeholders, regardless of their availability in individual trials [13, 14]. When reporting the review, they are asked to present a Summary of Findings (SoF) table summarizing the quality of evidence and treatment effect magnitude for the most essential outcomes for decision-making by using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach [13–18].

In this study, we aimed to evaluate: 1) whether outcomes reported in the SoF table could be considered PIOs and 2) the quality of evidence for these outcomes in a sample of recent Cochrane reviews.

Methods

This is a methodological review of recently published Cochrane reviews. For each Cochrane review reporting an SoF table, we extracted all outcomes reported in this table and evaluated whether they could be considered PIOs. Then, we evaluated the quality of evidence for these outcomes.

Data sources

We previously obtained data from all systematic reviews published between March 2011 and September 2014 from the Cochrane Collaboration [19]. Data were provided as XML files and contained all information reported by the review authors using RevMan[20], the review software developed by the Cochrane Collaboration to prepare and update systematic reviews.

Review selection

Using R 3.2.2 and the XML package, we identified all reviews of RCTs including at least one SoF table. Reviews including observational studies were excluded. We also excluded reviews including only trials published before 2007 to focus on recent topics. Then, we manually examined all review titles and abstracts when necessary to select those evaluating a healthcare intervention (pharmacological or non-pharmacological). Reviews evaluating diagnostic test accuracy or economic evaluations were excluded. Among the set of eligible reviews, by using a random number generator, we drew a random sample of 300 reviews for comprehensive evaluation.

Identification of the SoF for the main comparison

Most reviews have a single SoF table, but some have several, corresponding to different comparisons. In this case, we manually identified the main comparison as reported by the review authors. If no main comparison was identified as such, we selected the one with the largest number of available outcomes and included the largest number of trials. When the same SoF table reported various comparisons, the review was excluded.

Data collection

For every included systematic review, we collected the following characteristics by using a standardized data extraction sheet:

  • Review general characteristics: title, digital object identifier (DOI), protocol publication date, review first publication date, Cochrane Review Group (e.g., Gynaecological Cancer Group), type of intervention (i.e., pharmacological or non-pharmacological), number of excluded and included trials.
  • Characteristics of the main comparison: Patient or population description, setting, intervention and comparator
  • Every outcome reported in the SoF table including outcome description and follow-up period.

Classification of outcomes

We categorized each outcome as follows: mortality, other clinical events (e.g., myocardial infarction or stroke), therapeutic decision (e.g., transfusion), function (e.g., anxiety, depression, disability and dyspnoea), pain, quality of life, adverse events or side effects (identified as such by the review authors), physiological parameters (e.g., blood pressure, weight), biological parameters (e.g., cholesterol levels), radiological parameters (e.g., measure of joint space), compliance (e.g., discontinuation for any reason), process (e.g., duration of surgical procedure), resource use (hospitalisations), cost-effectiveness and satisfaction with care. A single reviewer classified all outcomes. As a quality measure, 10% of the outcomes were classified independently in duplicate (YY and AD).

Definition of PIOs

We considered mortality, other clinical events, adverse events, function, pain, quality of life and therapeutic decisions as PIOs consistent with previous studies [8, 11, 21, 22].

Quality of evidence

We evaluated the quality of evidence of each outcome reported in the SoF table based on the review authors assessment. The evidence was classified as high quality if further research is very unlikely to change the authors confidence in the estimate of effect; moderate quality if further research is likely to have an important impact on the authors’ confidence in the estimate of effect and may change the estimate; low quality if further research is very likely to have an important impact on the confidence in the estimate of effect and likely to change the estimate; and very low quality if the estimate of the effect is very uncertain[18]. We evaluated separately the quality of evidence for outcomes that could be considered PIOs. For those considered with high quality of evidence, we also evaluated whether this was evidence of benefit for the experimental intervention (results in favour of the experimental intervention: i.e., statistically significant results).

Statistical analysis

The analysis was mostly descriptive. Continuous data are presented as median (Q1–Q3) and qualitative data as frequency (percentage). All analyses involved use of R v3.1.1 (R Foundation for Statistical Computing, Vienna, Austria. http://www.R-project.org/).

Results

Selection of relevant Cochrane systematic reviews

The selection process was previously described[19]. Briefly, between March 2011 and September 2014, 2796 Cochrane systematic reviews were published, 1670 did not include an SoF table (60% of screened reviews) and 820 were eligible for inclusion. We identified a random sample of 300 reviews, but 10 were additionally excluded because their SoF presented different interventions within the same table, which left 290 reviews for further evaluation.

Characteristics of included reviews

The reviews were issued by 47 of the 53 Cochrane Review Groups[23]. Every review included a median of 11 trials overall (Q1-Q3: 5–21), with a median of 5 trials per main comparison (Q1-Q3: 3–12)[19]. The experimental intervention was non-pharmacological in 40% of the reviews (n = 115). Every SoF table included a median of 5 outcomes (Q1-Q3: 3–7), for a total of 1414 outcomes. The corresponding meta-analyses included a median of 3 trials (Q1-Q3: 2–7).

Outcome classification

Among the 1414 outcomes reported in the SoFs, the most common were functional outcomes (27%, n = 384), clinical events (14%, n = 198) and adverse events (12%, n = 174). Mortality represented 10% (n = 138) of the outcomes and quality of life 7% (n = 98). Among the outcomes, biological parameters represented 6% (n = 89), process and resource use 5% (n = 74), and physiological parameters and compliance 4% (n = 56) and 3% (n = 45). A total of 1089 outcomes (77%, 95% CI: 75–79) could be considered PIOs (Table 1). Almost all reviews included at least one PIO in the SoF table (n = 278, 96%; 95% CI: 93–98) (Fig 1). Per review, the median proportion of PIOs among the outcomes reported in the SoF was 86% (Q1-Q3: 67–100).

thumbnail
Download:
Table 1. Classification of outcomes reported in the Summary of Findings (SoF) table in 290 recent Cochrane reviews.

https://doi.org/10.1371/journal.pone.0195460.t001

thumbnail
Download:
Fig 1. Patient-important outcomes in a random sample of Cochrane reviews.

https://doi.org/10.1371/journal.pone.0195460.g001

Quality of available evidence

The quality of evidence provided by the outcomes in the SoF tables was high for 12% (n = 168, 95% CI: 10–14) and moderate for 28% (n = 402, 95% CI: 26–31) (Table 2). For 45% of the outcomes (n = 640, 95% CI: 43–48), the quality of the evidence was low or very low. No GRADE assessment was available for 12% of the outcomes (n = 164, 95% CI: 10–13) because according to the review authors, these outcomes were not reported or measured in any individual trials. The quality of the available evidence was very close when focusing on only PIOs, with 41% (n = 449) having high and moderate evidence and 44% (n = 475) low to very low evidence.

thumbnail
Download:
Table 2. Quality of available evidence for outcomes reported in the SoF tables for the main comparison in 290 recent Cochrane reviews.

https://doi.org/10.1371/journal.pone.0195460.t002

Less than one-quarter of the reviews, 22% (n = 63; 95% CI: 17–27), included at least one PIO with evidence considered of high quality by the review authors. In half (n = 34) of these reviews, the results were in favour of the experimental intervention (i.e., statistically significant results) (Fig 1).

Discussion

In this study, we evaluated whether outcomes reported in the SoF table of recent Cochrane reviews could be considered PIOs and the quality of evidence provided for these outcomes. The large variety of review groups represented allowed us to explore very different medical specialties. More than three-quarters of the outcomes reported in the SoF tables could be considered PIOs. However, for a large proportion of the available evidence (45%), the quality was considered low or very low. About one quarter of the reviews included at least one PIO with high-quality evidence and for half of them, this evidence showed a benefit of the experimental intervention.

The Cochrane Collaboration strongly encourages review authors to include SoF tables to present their main findings [13]. These tables, first introduced in Cochrane reviews in 2008, aim to synthesize in a simple, transparent and accessible format, key information on the assessed interventions’ magnitude of effect, sum of available data for the main outcomes and quality of evidence [13]. Therefore, SoF tables should include the most important outcomes whether they were measured in individual trials or not[13]. Previous studies reported that as compared with reviews without SoF tables, the inclusion of SoF tables seemed to improve readers’ general understanding of the reviews and allowed them to better identify the critical information and find results for important outcomes (93% vs 44%)[24]. However, not all Cochrane reviews present a SoF table. In our study, we excluded 60% of Cochrane reviews because they had no SoF table, which is consistent with a previous study showing a proportion of inclusion of SoF tables evolving from 31% in 2008 to 57% in 2013 [25]. An alternative version of the SoF table called the interactive SoF (iSoF) allowing review authors to choose alternative displays (absolute effects rather than relative effects) is being tested[26], which may help generalise their use.

We found that most of the outcomes reported in the SoF tables could be considered PIOs, with a median proportion of PIOs in the SoF tables of 86%. In a study that focused on recently published Cochrane and non-Cochrane reviews but also on registered systematic reviews protocols (PROSPERO), Ameur et al described that 95% of Cochrane reviews included at least one PIO among their primary outcomes. This proportion was higher than for non-Cochrane reviews[11]. Two previous studies also found consistent results, with 68% to 71% of the reviews considering PIOs as primary outcomes[21, 22]. A large survey of the completeness of main outcomes mapped the availability of information for a major clinical outcome across randomized trials and systematic reviews in an entire discipline: 75% of reviews reported the major clinical outcome as compared with only 20% of primary trial reports [27]. All these results highlight the gap in outcomes between systematic reviews and their individual trials, which may explain why the quality of evidence was frequently limited for the outcomes reported in the SoF table, as shown by our results. In our study, only 22% of the reviews had at least one PIO with a high level of evidence, with half showing a benefit of the experimental intervention. The failure to consider PIOs in clinical trials may lead to erroneous evaluation of benefits, with possible serious consequences for patients, and represents waste[28–31]. To improve this situation, core outcome sets (COSs) should be developed (ie, standardized sets of outcomes, that have been agreed upon, and that should be measured and reported in every trial for a specific healthcare condition) and their use promoted [32–34]. The number of COSs developed is still limited but is progressively increasing, such as their use in systematic reviews [11, 25, 33, 35–37].

Our study has limitations. Although we relied on detailed and previously published definitions for PIOs, considerable judgment remains when assessing the importance of the outcomes, and patients or experts in the assessed medical field were not involved in this evaluation. Our results cannot be generalized to all systematic reviews because Cochrane reviews have precise conduct and reporting guidelines[13–15] and were described as more transparently reported than all other types of systematic reviews[38]. Our study confirms that many outcomes reported in the SoF tables of recent Cochrane reviews can be considered PIOs. However, the quality of evidence for these outcomes remains limited.

Conclusions

Our study confirms that many outcomes reported in the SoF tables of recent Cochrane reviews can be considered PIOs. However, the quality of evidence for these outcomes remains limited.

Supporting information

S1 Table. Quality of available evidence for outcomes reported in the SoF tables for the main comparison in 290 recent Cochrane reviews: Comparison of all outcomes, to PIOs and surrogate outcomes.

https://doi.org/10.1371/journal.pone.0195460.s001

(DOCX)

S1 File. Supporting data 1: Complete list of outcomes, classified and categorized.

https://doi.org/10.1371/journal.pone.0195460.s002

(XLSX)

S2 File. Supporting data 2: All the study data.

https://doi.org/10.1371/journal.pone.0195460.s003

(XLSX)

References

  1. 1. Guyatt G, Montori V, Devereaux PJ, Schunemann H, Bhandari M. Patients at the center: in our practice, and in our use of language. ACP J Club. 2004;140(1):A11–2. pmid:14711297.
  2. 2. Richards T, Montori VM, Godlee F, Lapsley P, Paul D. Let the patient revolution begin. BMJ. 2013;346:f2614. Epub 2013/05/16. pmid:23674136.
  3. 3. Montori VM, Gandhi GY, Guyatt GH. Patient-important outcomes in diabetes—time for consensus. Lancet. 2007;370(9593):1104–6. pmid:17905147.
  4. 4. Frank L, Basch E, Selby JV, Patient-Centered Outcomes Research I. The PCORI perspective on patient-centered outcomes research. Jama. 2014;312(15):1513–4. pmid:25167382.
  5. 5. Methodology Committee of the Patient-Centered Outcomes Research I. Methodological standards and patient-centeredness in comparative effectiveness research: the PCORI perspective. Jama. 2012;307(15):1636–40. pmid:22511692.
  6. 6. Selby JV, Beal AC, Frank L. The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda. Jama. 2012;307(15):1583–4. pmid:22511682.
  7. 7. Rahimi K, Malhotra A, Banning AP, Jenkinson C. Outcome selection and role of patient reported outcomes in contemporary cardiovascular trials: systematic review. BMJ. 2010;341:c5707. pmid:21041324.
  8. 8. Gandhi GY, Murad MH, Fujiyoshi A, Mullan RJ, Flynn DN, Elamin MB, et al. Patient-important outcomes in registered diabetes trials. Jama. 2008;299(21):2543–9. pmid:18523223.
  9. 9. Gaudry S, Messika J, Ricard JD, Guillo S, Pasquet B, Dubief E, et al. Patient-important outcomes in randomized controlled trials in critically ill patients: a systematic review. Ann Intensive Care. 2017;7(1):28. pmid:28271450.
  10. 10. Saldanha IJ, Li T, Yang C, Owczarzak J, Williamson PR, Dickersin K. Clinical trials and systematic reviews addressing similar interventions for the same condition do not consider similar outcomes to be important: a case study in HIV/AIDS. J Clin Epidemiol. 2017;84:85–94. pmid:28249722.
  11. 11. Ameur H, Ravaud P, Fayard F, Riveros C, Dechartres A. Systematic reviews of therapeutic interventions frequently consider patient-important outcomes. J Clin Epidemiol. 2017. pmid:28147235.
  12. 12. Khan KS, Kunz R, Kleijnen J, Antes G. Five steps to conducting a systematic review. J R Soc Med. 2003;96(3):118–21. pmid:12612111.
  13. 13. Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration 2011. www.cochrane-handbook.org.
    • 14. O’Connor D, Green S, Higgins JPT. Chapter 5: Defining the review question and developing criteria for including studies. In: Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated September 2011]. ed. www.cochrane-handbook.org.: The Cochrane Collaboration; 2011.
      • 15. Schünemann HJ, Oxman AD, Higgins JPT, Vist GE, Glasziou P, Akl E, et al. Chapter 11: Completing ‘Summary of findings’ tables and grading the confidence in or quality of the evidence. In: Higgins JPT, Chandler J, (editors) CM , editors. Cochrane Handbook for Systematic Reviews of Interventions version 5.2.0 (updated June 2017). ed. Cochrane, 2017. http://training.cochrane.org/handbook: The Cochrane Collaboration; 2017.
        • 16. Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schunemann HJ. What is "quality of evidence" and why is it important to clinicians? Bmj. 2008;336(7651):995–8. Epub 2008/05/06. pmid:18456631.
        • 17. Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, et al. Grading quality of evidence and strength of recommendations. Bmj. 2004;328(7454):1490. Epub 2004/06/19. pmid:15205295.
        • 18. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650):924–6. pmid:18436948.
        • 19. Yordanov Y, Dechartres A, Atal I, Tran V, Boutron I, Crequit P, et al. Avoidable waste of research related to outcome planning and reporting in clinical trials. BMC Medicine—under review. 2017.
        • 20. Dechartres A, Ravaud P, Atal I, Riveros C, Boutron I. Association between trial registration and treatment effect estimates: a meta-epidemiological study. BMC medicine. 2016;14(1):100. pmid:27377062.
        • 21. Alonso-Coello P, Carrasco-Labra A, Brignardello-Petersen R, Neumann I, Akl EA, Vernooij RW, et al. Systematic reviews experience major limitations in reporting absolute effects. J Clin Epidemiol. 2016;72:16–26. pmid:26560992.
        • 22. Agarwal A, Johnston BC, Vernooij RW, Carrasco-Labra A, Brignardello-Petersen R, Neumann I, et al. Authors seldom report the most patient-important outcomes and absolute effect measures in systematic review abstracts. J Clin Epidemiol. 2016. pmid:27555080.
        • 23. Cochrane Collaboration. "Cochrane Review Groups"[Internet] 2017 [Date Accessed: 19/09/2017] [cited 2017 19/09/2017]. www.cochrane.org/contact/review-groups].
          • 24. Rosenbaum SE, Glenton C, Oxman AD. Summary-of-findings tables in Cochrane reviews improved understanding and rapid retrieval of key information. J Clin Epidemiol. 2010;63(6):620–6. pmid:20434024.
          • 25. Clarke M, Williamson PR. Core outcome sets and systematic reviews. Syst Rev. 2016;5:11. pmid:26792080.
          • 26. Alonso-Coello P, Vernooij RWM, Carrasco-Labra A, Vandvik P, Guyatt GH. RE: Which estimates comprise optimal reporting in systematic reviews? Journal of Clinical Epidemiology. 2017;81(Supplement C):140–1. https://doi.org/10.1016/j.jclinepi.2017.01.001.
          • 27. Ioannidis JP, Horbar JD, Ovelman CM, Brosseau Y, Thorlund K, Buus-Frank ME, et al. Completeness of main outcomes across randomized trials in entire discipline: survey of chronic lung disease outcomes in preterm infants. BMJ. 2015;350:h72. Epub 2015/01/28. pmid:25623087.
          • 28. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374(9683):86–9. pmid:19525005
          • 29. Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014;383(9912):166–75. pmid:24411645.
          • 30. Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis JP, et al. Biomedical research: increasing value, reducing waste. Lancet. 2014;383(9912):101–4. pmid:24411643.
          • 31. Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014;383(9913):267–76. Epub 2014/01/15. pmid:24411647.
          • 32. Williamson P, Altman D, Blazeby J, Clarke M, Gargon E. Driving up the quality and relevance of research through the use of agreed core outcomes. J Health Serv Res Policy. 2012;17(1):1–2. pmid:22294719
          • 33. Williamson P, Clarke M. The COMET (Core Outcome Measures in Effectiveness Trials) Initiative: Its Role in Improving Cochrane Reviews. Cochrane Database Syst Rev. 2012;5:ED000041. Epub 2012/05/18. pmid:22592744.
          • 34. Williamson PR, Altman DG, Blazeby JM, Clarke M, Devane D, Gargon E, et al. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13:132. Epub 2012/08/08. pmid:22867278
          • 35. Kirkham JJ, Gargon E, Clarke M, Williamson PR. Can a core outcome set improve the quality of systematic reviews?—a survey of the Co-ordinating Editors of Cochrane Review Groups. Trials. 2013;14. pmid:23339751
          • 36. Clarke M, Williamson PR. Core outcome sets and systematic reviews. Syst Rev. 2016;5(1):11. pmid:26792080.
          • 37. Gorst SL, Gargon E, Clarke M, Blazeby JM, Altman DG, Williamson PR. Choosing Important Health Outcomes for Comparative Effectiveness Research: An Updated Review and User Survey. PLoS ONE. 2016;11(1):e0146444. pmid:26785121.
          • 38. Page MJ, Shamseer L, Altman DG, Tetzlaff J, Sampson M, Tricco AC, et al. Epidemiology and Reporting Characteristics of Systematic Reviews of Biomedical Research: A Cross-Sectional Study. PLoS Med. 2016;13(5):e1002028. pmid:27218655.