Introduction

Systematic reviews (SRs) are “prepared using a systematic approach to minimizing biases and random errors which is documented in a materials and methods section” [1] and have become established as a linchpin of evidence-based practice, influencing clinical practice and informing health policy. It is, therefore, important that SRs adhere to rigorous methodology and are clear and unbiased. While interventional SRs are prominent in the assessment of the comparative effectiveness of healthcare procedures, other reviews including epidemiologic reviews, reviews of diagnostic tests, qualitative reviews and individual patient meta-analysis are commonplace and increasingly influential [2]. There is, therefore, a similar onus on clear and accurate reporting of these types of SR.

A plethora of reporting and methodological guidelines have been developed to assist in the reporting and conduct of all aspects of research, including SRs [3] and randomized controlled trials (CONSORT [4] and its extensions) and observational studies (e.g. STROBE [5]). These guidelines have had a positive impact on the reporting of research, although further work and awareness is required to elevate the standard of reporting to required levels [6]–[8]. Appropriate reporting of meta-analyses was initially outlined in the QUOROM statement [9]. This was subsequently updated to encompass systematic reviews generally. However, guidelines and adjuncts tailored specifically to each type of systematic review exist. For example, MOOSE guidelines [10] guide the reporting and conduct of epidemiologic reviews, AMSTAR is accepted as a valid tool for assessing the methodological quality of interventional SRs [11], while QUADAS [12] and latterly QUADAS-2 [13] have been developed to aid methodological assessment in diagnostic reviews.

Inadequate reporting risks shrouding the conduct of systematic reviews impairing potential judgements in relation to the emphasis that should be placed upon their results and conclusions. It is, therefore, important that these guidelines are considered and applied where necessary to optimise both the conduct and reporting of all types of SR. The aims of the present research were to assess the frequency of correct use of appropriate guidelines in SRs and to highlight factors associated with inappropriate use or omission of relevant guidelines.

Methods

Articles citing the PRISMA guidelines were searched from January 1^st to December 31^st 2012, using the Web of Knowledge database (http://apps.webofknowledge.com) by one of the authors (PSF). The first step consisted of a search for PRISMA publications on the PRISMA website (http://prisma-statement.org). The ‘Create citation report’ tool on the Web of Knowledge database was accessed to identify articles citing one of these six PRISMA guideline articles. SRs only were to be included in the research with other article types including narrative reviews, meta-epidemiologic studies, letters, editorials and other studies excluded from the analysis.

Two researchers (PSF and DK) screened the titles and abstracts of all retrieved references. Electronic versions of potentially eligible reviews were retrieved and analyzed by the authors to assess eligibility. Review articles without a specific methodology section were excluded. Two reviewers independently extracted data from eligible reviews using piloted forms. Calibration was initially undertaken on 10 articles and disagreements were resolved through consensus or by discussion with a third reviewer (NP).

Data were obtained on the type of systematic review, medical field of publication, number of authors, involvement of a methodologist or statistician in the paper, specific PRISMA article cited, reason for the citation, and whether meta-analysis was undertaken. The involvement of a methodologist was gauged from affiliations or qualifications reported and information in the review methodology section.

Descriptive statistics were performed to categorize the number of SRs identified within each medical field and to relate reporting to the remaining study characteristics. Pearson chi-squared test and Fisher’s exact test were used to determine the association between appropriate PRISMA citation and trial characteristics including medical specialty, number of authors, methodologist involvement, specific PRISMA statement article cited, reason for citation, type of SR, and implementation of meta-analysis or otherwise. Univariable and multivariable logistic regression modeling was used to determine the association between appropriate PRISMA statement citation and predictor variables including medical field of research, number of authors, methodologist involvement, cited article and whether meta-analysis was undertaken. The adequacy of the model was assessed using the Hosmer-Lemeshow test. The level of statistical significance for all tests was pre-specified at 0.05. All statistical analyses were conducted with statistical software (Stata 12.1, Stata Corp, College Station, TX, USA).

Results

Seven hundred and one SRs were identified pertaining to 10 medical fields (Figure 1, Table 1). The highest number of SRs was published in General Medical (n = 121), Public Health (n = 103) and Cardiovascular (n = 93) fields. The majority of studies were authored by more than 6 researchers without the involvement of a methodologist and included meta-analysis. Most were interventional SRs, citing PRISMA as a guideline for reporting reasons, while the most frequently cited article was that of Moher et al., (2009) published in PLoS Medicine. The guideline most frequently omitted within the subset of SRs citing only PRISMA [3] was MOOSE [9] (Table 2), with 236/284 (83%) of epidemiologic reviews citing the latter guidelines, in isolation. Similarly, in SRs of diagnostic tests 23/57 (40%) cited PRISMA without referring to either QUADAS or QUADAS-2, the remainder cited PRISMA³ in isolation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. Flow diagram of article retrieval.

https://doi.org/10.1371/journal.pone.0096407.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Characteristics of the included SRs (n = 701).

https://doi.org/10.1371/journal.pone.0096407.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Omitted guideline among the subset of SRs citing PRISMA, based on field of publication (n = 263).

https://doi.org/10.1371/journal.pone.0096407.t002

In the univariable analysis (Table 3), medical field of research was associated with appropriate PRISMA [3] citation (Other: OR = 2.23, 95% CI: 1.25, 4.0; Obstetrics: OR = 2.87, 95% CI: 1.36, 6.05; General Medicine: OR = 3.17, 95% CI: 1.82, 5.50; General Surgery: OR = 4.02, 95% CI: 1.78, 9.09; Cardiovascular: OR = 5.22, 95% CI: 2.76, 9.88; Orthopedics/Physiotherapy: OR = 3.75, 95% CI: 1.65, 8.53; Dentistry: OR = 3.80, 95% CI: 1.38, 10.42; compared to Public Health). Involvement of a methodologist and use of meta-analysis were also identified as significant predictors for the outcome. In the multivariable analysis (Table 3), medical field (Other: OR = 2.57, 95% CI: 1.41, 4.69; Obstetrics: 3.40, 95% CI: 1.58, 7.33; General Medicine: OR = 3.41, 95% CI: 1.93, 6.04; General Surgery: OR = 4.91, 95% CI: 2.12, 11.37; Cardiovascular: OR = 5.11, 95% CI: 2.63, 9.91; Orthopedics/Physiotherapy: OR = 5.40, 95% CI: 2.30, 12.64; Dentistry: OR = 5.68; 95% CI: 2.01, 16.02; compared to Public Health) and methodologist involvement (OR = 1.97, 95% CI: 1.37, 2.83; compared to non-involvement) remained significant predictors of appropriate PRISMA citation. There was also evidence that inclusion of a meta-analysis was associated with appropriate citation of SR guidelines (OR = 1.73, 95% CI: 1.22, 2.35). The predicted probabilities from the adjusted model for appropriate PRISMA citation are shown by medical field and inclusion of a meta-analysis or otherwise in Figure 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 2. Predicted probabilities of appropriate PRISMA citation with 95% confidence intervals (CIs) derived from the adjusted model based on the medical field and implementation of a meta-analysis within the SR.

https://doi.org/10.1371/journal.pone.0096407.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Univariable and multivariable logistic regression derived ORs and 95% confidence intervals (CI) for appropriate PRISMA citation in the identified SRs (n = 701).

https://doi.org/10.1371/journal.pone.0096407.t003

Discussion

The requirement for high quality systematic reviews of healthcare interventions, epidemiological data and diagnostic information is clear as the emphasis on evidence-based decision-making continues. A corollary of this situation is for more complete and transparent reporting of the design and conduct of a SR, allowing stakeholders to reach informed decisions based on the validity of the findings. The PRISMA guidelines [3] were developed with the expressed aim of standardizing reporting and enhancing the clarity of reporting of SRs. While the emphasis of the guidelines changed from a focus on interventional research to a more generic approach from its precursor [9], the PRISMA [3] guidelines do not consider nuances of certain aspects of other review types. It is, therefore, important that its use is supplemented by reference to additional tailored reporting and quality assessment guidelines in respect of these review types. The findings from the present study, however, indicate that a significant percentage of epidemiologic and diagnostic reviews are published without reference to these complementary resources.

While there is significant overlap between MOOSE [10] and PRISMA [3], there are items unique to each with MOOSE [10] guidelines comprised of 35 items. In particular, MOOSE [10] incorporates very detailed instructions in respect of search strategy, including delineation of the qualifications of the searchers, use of hand-searching and approaches to dealing with unpublished and non-English literature, emphasizing the centrality of this aspect of the review to meta-analysis. The importance of the assessment of the potential for bias in primary studies is reinforced in both guidelines. However, greater emphasis is placed on the interpretation of the results of the review, specifically regarding possible alternate explanations for the observed findings in the MOOSE guidelines [10]. This distinction reflects the elevated susceptibility of observational research to both bias and confounding limiting the potential inferences and the degree to which the results of the review can be trusted and used to inform healthcare decisions [14]. The finding from the present study of 83% of epidemiologic reviews citing PRISMA [3] without referring to MOOSE [10] suggests that these reviews may lack complete reporting of these methodological aspects and failure to explore the reasons for the observed findings in sufficient detail.

Meta-analyses of diagnostic tests have become increasingly prominent in medicine and will potentially have an increasing role in healthcare as decision makers consult the evidence before implementing new diagnostic technologies. It is important that such analyses provide reliable results; this is contingent upon both the quality of conduct and the clarity of reporting. As with over review types, diagnostic systematic reviews are susceptible to shortcomings pertaining to conduct and reporting [15]–[17]. To mitigate potential weaknesses, quality assessment of the primary studies has become integral to diagnostic reviews; the QUADAS [12] and more recently QUADAS-2 [13] guidelines have facilitated this process. The present study, however, revealed that 40% of diagnostic reviews did not refer to either of these guidelines risking inadequate methodological comparison of constituent primary studies in diagnostic reviews.

Disentanglement of the conduct of SRs from reporting is more complex than is the case with primary studies, as significant overlap exists between the two. Nevertheless, PRISMA guidelines [3] were developed with the intention of improving the reporting of reviews; the AMSTAR guidelines [11] have emerged as a valid tool to assess the conduct of SRs. While meta-epidemiologic studies were omitted from the analysis in the present cross-sectional study, it was noted that 14% of these also failed to refer to AMSTAR guidelines [11] during methodological assessment of SRs suggesting an over-reliance on PRISMA [3] in this further respect. It is, therefore, important that greater awareness of these guidelines arises.

The over-reliance on PRISMA guidelines [3] in respect of reporting of SRs reflects widespread awareness and endorsement of these guidelines. This is certainly a positive development, which has led to enhanced reporting of SRs [6], although further work is required [7], [8]. However, while a plethora of reporting guidelines are available in a prominent on-line resource [18], lack of awareness of alternatives and complementary guidelines is potentially problematic. This situation may relate to widespread espousal and endorsement of reporting guidelines including PRISMA [3] and CONSORT [4] by biomedical journals, while other guidelines, including MOOSE [10] and QUADAS [12], have not been treated similarly. In recent times efforts have been made to improve compliance with PRISMA guidelines using active editorial involvement [6]; these efforts have not been replicated in respect of other guidelines, including those pertaining to observational research. It is likely, therefore, that over-reliance on PRISMA will continue, unless similar efforts are made to improve awareness of complementary guidelines. It is, therefore, important that emphasis on the reporting of not just interventional reviews but also epidemiologic, diagnostic and other systematic reviews is enhanced. This development will be facilitated by raising awareness of other guidelines, such as MOOSE [10] and QUADAS-2 [13] and encouraging active editorial input to enhance reporting of all review types.

Potential limitations of this study included the relatively short time period assessed; however, a large number of eligible studies from diverse areas of medicine were included. Nevertheless, temporal changes in reporting characteristics will not be identified. In addition, given that the focus of the research was to assess the usage of PRISMA guidelines specifically, it is likely that epidemiologic and diagnostic reviews that cite more specific guidelines in isolation, without referring to PRISMA, exist.

Conclusions

There is evidence that significant awareness of PRISMA guidelines exist. However, usage of other accepted reporting guidelines and tools for assessment of methodological quality have less prominence and may be under-utilized with negative connotations on both reporting and methodological quality of systematic reviews.