Figures
Abstract
Background
The QUOROM and PRISMA statements were published in 1999 and 2009, respectively, to improve the consistency of reporting systematic reviews (SRs)/meta-analyses (MAs) of clinical trials. However, not all SRs/MAs adhere completely to these important standards. In particular, it is not clear how well SRs/MAs of acupuncture studies adhere to reporting standards and which reporting criteria are generally ignored in these analyses.
Methods
We performed a literature search for studies published prior to 2014 using the following public archives: PubMed, EMBASE, Web of Science, the Cochrane Database of Systematic Reviews (CDSR), the Chinese Biomedical Literature Database (CBM), the Traditional Chinese Medicine (TCM) database, the Chinese Journal Full-text Database (CJFD), the Chinese Scientific Journal Full-text Database (CSJD), and the Wanfang database. Data were extracted into pre-prepared Excel data-extraction forms. Reporting quality was assessed based on the PRISMA checklist (27 items).
Results
Of 476 appropriate SRs/MAs identified in our search, 203, 227, and 46 were published in Chinese journals, international journals, and the Cochrane Database, respectively. In 476 SRs/MAs, only 3 reported the information completely. By contrast, approximately 4.93% (1/203), 8.81% (2/227) and 0.00% (0/46) SRs/Mas reported less than 10 items in Chinese journals, international journals and CDSR, respectively. In general, the least frequently reported items (reported≤50%) in SRs/MAs were “protocol and registration”, “risk of bias across studies”, and “additional analyses” in both methods and results sections.
Conclusions
SRs/MAs of acupuncture studies have not comprehensively reported information recommended in the PRISMA statement. Our study underscores that, in addition to focusing on careful study design and performance, attention should be paid to comprehensive reporting standards in SRs/MAs on acupuncture studies.
Citation: Liu Y, Zhang R, Huang J, Zhao X, Liu D, Sun W, et al. (2014) Reporting Quality of Systematic Reviews/Meta-Analyses of Acupuncture. PLoS ONE 9(11): e113172. https://doi.org/10.1371/journal.pone.0113172
Editor: Brett Thombs, McGill University, Canada
Received: November 5, 2013; Accepted: October 23, 2014; Published: November 14, 2014
Copyright: © 2014 Liu et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: Project supported by the National Natural Science Foundation of China (Grant No. 81373882) http://www.nsfc.gov.cn/Portal0/default152.htm The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Systematic reviews (SRs) and meta-analyses (MAs) summarize large amounts of evidence and are a valuable tool for keeping clinicians up to date within their specialty [1], [2]. As with all research, however, the value of SRs/MAs depends on how the analyses are performed, the actual findings, and the clarity of reporting [3]. If key information is reported poorly, the potential usefulness of the SRs/MAs is diminished.
Since 1987, numerous researchers have recognized the need to evaluate the quality of these types of reviews. For example, in 1987 Sacks and colleagues [4] evaluated reporting in SRs/MAs and found that it was inadequate. The Consolidated Standards of Reporting Trials (CONSORT) Group subsequently developed the Quality of Reporting of Meta-Analyses (QUOROM) statement to address suboptimal MA reporting. Ten years later, an updated QUOROM statement—entitled Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement—was developed and published [3]. The PRISMA statement consists of a checklist of 27 study reporting items such as title, abstract, methods, results, discussion, and funding sources. The checklist is intended to guide authors of SRs/MAs to improve the consistency and quality of reporting.
Acupuncture, a traditional medicine technique, has been widely used in clinical practice for thousands of years in China and many western countries. The number of published SRs/MAs of acupuncture studies has increased considerably in recent years. As the transparency and completeness of SRs/MAs in many fields is still not optimal [5]–[8], we examined how well SR/MA reporting standards have been followed in the field of acupuncture and compared adherence to these standards in acupuncture SRs/MAs published in three different types of journals/databases.
Methods
The protocol for this study was written in Chinese and has not been published. The study was not a classical systematic review, but we tried to report it according to PRISMA Checklist [3] (Text S1).
Inclusion/exclusion criteria
We included all SRs or MAs of acupuncture published in Chinese journals, international journals, and the Cochrane Database of Systematic Reviews (CDSR) prior to 2014. The experimental group of SRs/MAs of acupuncture studies was also compared with a control group of SRs/MAs of studies of other interventions, such as herbal medicine, massage and western medicine. Participants: human in any conditions, not animal; Intervention: acupuncture; Comparisons: sham acupuncture or other interventions, such as herbal medicine, massage, western medicine, etc; Outcomes: no limitations; Study design: SRs/MAs. We excluded SRs/MAs that focused primarily on traditional Chinese medicine (TCM) other than acupuncture (e.g., herbal medicine, massage).
Search strategy
We comprehensively and systematically searched the following literature archives for SRs/MAs published prior to 2014: CDSR, PubMed, EMBASE, Web of Science, Chinese Biomedical Literature Database (CBM), the TCM database, Chinese Journal Full-text Database (CJFD), Chinese Scientific Journal Full-text Database (CSJD), and the Wanfang database. Databases were searched three times: on March 24, 2011 for all entries submitted prior to March 2011, on June 12, 2012 for all entries submitted prior to January 2012 and on January 11, 2014 for all entries submitted prior to 2014. The search terms “acupuncture”, “needling”, “ear acupuncture”, “electroacupuncture”, “electro-acupuncture”, “acupuncture points”, “acupressure”, “moxibustion”, and “acupoint” were used with the terms “systematic review” or “meta-analysis”. The search strategy is presented in Checklist S1.
Screening
The titles and abstracts of the studies were independently screened by at least two reviewers (Jiao Huang, Xu Zhao, or Rui Zhang) based on inclusion and exclusion criteria, and the full text of potentially suitable articles was retrieved for further assessment (Text S2).
Data extraction and analysis
Data were extracted independently by at least two reviewers (Rui Zhang, Jiao Huang, Xu Zhao, or Danlu Liu) in accordance with the PRISMA checklist and the assessment checklist for SRs/MAs of acupuncture studies developed for this study. Inconsistencies were subsequently resolved by discussion between the two reviewers or final decisions were made by the third principal investigator (Yali Liu). Data input utilized a standardized form and was done by trained data extractors (Wanting Sun, Pen Zhang, and Hua Cao). The form consisted of a general characteristics section (title, first author, funding source, study design, disease(s) examined, diagnostic criteria, intervention, and outcome) and a 27-item PRISMA information section (including title, abstract, introduction, method, results, discussion, and funding). Each item was assessed as “yes” if it was described in the paper or “no” if it was not (Text S3). Data were summarized with descriptive statistical analysis. For continuous data, the means ± SD was provided and one-way ANOVA was used. Data that followed a normal distribution were compared using the LSD- t test. Dichotomous data were summarized with descriptive statistical analysis (frequency, percentage). Pearson's χ2 test and/or Fisher's exact test were used to assess differences in reporting among groups. P values less than 0.05 were considered significant. All statistical analyses were performed using Microsoft Excel (version 2007) and SPSS (version 13.0) software.
Results
Search
Our initial literature search identified 3993 potential SRs/MAs of acupuncture-related studies. After closer examination, 476 were chosen for inclusion in our analysis (Text S4). Of these, 203, 227, and 46 were published in Chinese journals, international journals, and CDSR, respectively. A flow chart of the literature search is shown in Figure 1.
General characteristics
General characteristics of the SRs/MAs analyzed are summarized in Table 1. The earliest acupuncture SRs/MAs in Chinese journals and international journals were published in 2002 and 1989, respectively. The number of acupuncture SRs/MAs in Chinese and international journals increased sharply after 2005, whereas the majority of acupuncture SRs/MAs in CDSR were published between 2008 and 2013 (Figure 2).
Acupuncture SRs/MAs in Chinese journals were conducted entirely by Chinese authors, whereas those published in international journals tended to be multinational collaborations, with Chinese first authors being most prevalent (27.31%, 62/227). Chinese authors were also most prevalent first authors in CDSR studies (34.78%, 16/46). The percentage of published acupuncture SRs/MAs in Chinese journals, international journals, and the CDSR that reported at least one funding source was 54.19% (110/203), 50.66% (115/227), and 89.13% (41/46), respectively, and the maximum number of funding sources reported was 5, 3, and 7, respectively.
The majority of SRs/MAs (95.37%, 454/476) included at least one randomized controlled trial (RCT). Nervous system diseases, musculoskeletal system diseases, and mental illness were most frequently examined (23.11, 18.91 and 13.44%, respectively). Approximately 38% (180/476) of the SRs/MAs reported western diseases or TCM syndromes in their diagnostic criteria. All acupuncture SRs/MAs examined described the interventions in detail. 39.92% (190/476) and 19.54% (93/476) SRs/MAs included adverse events and quality-of-life in the outcome which were reported as primary and/or secondary outcomes.
PRISMA information reporting
Comparison of PRISMA reporting among the three types of journals/databases (Table 2)
Among 476 SRs/MAs, only 3 reported the information completely. By contrast, approximately 4.93% (1/203), 8.81% (2/227) and 0.00% (0/46) SRs/MAs reported less than 10 items on the checklist in Chinese journals, international journals, and CDSR, respectively. In general, the least frequently reported items (reported≤50%) in SRs/MAs were item 5 (“protocol and registration”), 15 and 22 (“risk of bias across studies”), and 16 and 23 (“additional analyses”). The remaining items on the checklist were adequately reported (i.e >90%), with the items listed in Table 2 being especially well reported.
Comparison of PRISMA reporting before and after release of the PRISMA statement (Table 3)
We found no statistical difference (P>0.05) for item 2 (“structured summary”), 5 (“protocol and registration”), 6 (“eligibility criteria”), 8 (“search”), 10 (“data collection process”), 11 (“data items”), 12 (“risk of bias in individual studies”), 16 (“additional analyses”), 17 (“study selection”), 18 (“study characteristics”), 23 (“additional analysis”) and 27 (“funding”) between SRs/MAs published prior to release of the PRISMA statement and those published after its release. Unfortunately, the rate of reporting of two items (“objective” and “information sources”) had decreased in 2010–2013 compared with before 2010 (P<0.001).
Comparison of PRISMA reporting in Science Citation Index (SCI) and non-SCI journals (Table 3)
We found that PRISMA reporting in SRs/MAs in SCI journals was more complete overall than in non-SCI journals, especially for items 4 (“objective”), 5 (“protocol and registration”), 7 (“information sources”), 8 (“search”), 9 (“study selection”), 11 (“data items”), 23 (“additional analysis”), and 27 (“funding”) (P<0.001).
Discussion
Over the last decade, numerous studies have assessed the quality of reporting in SRs/MAs by their compliance with assessment instruments such as the QUOROM and PRISMA statements [9]–[12]. These studies focused predominantly on SRs/MAs covering diagnostic research and critical care. Although some quality assessment studies have looked at acupuncture SRs/MAs [13]–[15], they have focused mainly on methodological diversity in database searching, risk of bias, and heterogeneity in search strategies among CDSR. Our study compared reporting quality and PRISMA compliance in acupuncture SRs/MAs between different journal types.
We found that the five PRISMA items, namely “Protocol and registration”, “Risk of bias across studies” (both in the methods and results), and “Additional analyses” (both in the methods and results) in the methods and results, are not frequently reported, indicating that the overall quality of reporting in acupuncture SRs/MAs is far from adequate. Compared with SRs/MAs published in CDSR, those in Chinese and international journals were of inferior reporting quality. One possible explanation for the limited compliance may be that journals have failed to incorporate the PRISMA statement into their instructions to authors about submitting SRs/MAs [16]. We also found that SR/MA reporting was more complete in SCI journals than in non-SCI journals but that both require improvement in adherence to PRISMA standards.
Several studies have focused on the reporting quality of SRs/MAs covering the fields of TCM [17], [18], physical therapy [19], orthopaedics [20], and oral implantology [21] field, which showed that the reporting quality was indeed poor. Although differences exist between these results and those we repot here, the reporting of major items in the PRISMA statement was similar to what we found in our present study. Additional, Fleming PS et al. [22] found that the quality of reporting was considerably better in reviews published in CDSR (P<0.001) than in non-CDSR.
Both the QUOROM and PRISMA statements encourage the use of specific terms in the titles of SRs/MAs, which help to identify these studies. Because of the special title format requirements of the CDSR, however, SRs/MAs published in this database cannot conform to the QUOROM/PRISMA recommendation.
Unequivocal descriptions of the scientific background and rationale for using acupuncture in the treatment of both western diseases and TCM syndromes provide the reader with a better understanding of the research context and rationale of SRs/MAs. In this respect, SRs/MAs in the CDSR were more explicit in their descriptions than those in international or Chinese journals.
The importance of protocol consistency and registration of SRs/MAs to the transparency of reporting is underscored by the fact that they are considered key aspects of the “reporting guidelines for systematic review protocols” in the international prospective register of systematic reviews (PROSPERO) [23], [24]. We found that only SRs/MAs published in the CDSR provided protocol and registration details.
The PRISMA standards suggest that methodological details such as eligibility criteria, information sources, search strategies, study selection criteria, and data collection processes are necessary to judge the quality and accuracy of SRs/MAs. The majority of the SRs/MAs published in the CDSR adequately reported these items, whereas those published in Chinese and international journals did not. Eligibility criteria are an aspect of the PICOS criteria (participants, interventions, comparisons, outcomes, and study design) central to the PRISMA approach. We propose that it is equally important that search strategies be uniformly reported. Many international journals require information about search strategies in at least one database, and the flexibility of the CDSR layout allows reporting of search strategies for multiple databases. Chinese journals, however, rarely request search strategy information. There is also considerable need for more consistency in the databases obtain acupuncture studies. We propose that, AcuBriefs (www.acubriefs.com), AcuBase (www.acubase.fr), Acudoc2 RCT (www.acubriefs.com/), and the TCM database are the most systematic and comprehensive sources for acupuncture information. Chinese RCTs make up the highest proportion of primary studies included in acupuncture SRs/MAs. If methods for sequence generation, allocation concealment, and study blindness are not adequately described, low-quality studies [25] may mislead reviewers.
We found that there is also considerable inconsistency in reporting of study selection criteria. For example, many primary studies on acupuncture report a random allocation design but are not specific enough for the reader to determine if they are actual RCTs. We propose that these uncertainties should be clarified by contacting the primary authors to determine the appropriateness of including the studies in the SRs/MAs. Because it has been suggested that only 6.8% of acupuncture efficacy studies published in Chinese journals are based on actual RCTs [25], we strongly propose that authors of SRs/MAs verify this information prior to inclusion of studies.
Acupuncture is considered an alternative or complementary treatment to western medical interventions such as drugs and surgery, and it can be considered a separate specialty. Thus, SRs/MAs on acupuncture require not only compliance with general PRISMA reporting standards but also accurate reporting of acupuncture information. As a result, it is necessary to develop an extension of the PRISMA statement for acupuncture.
There are several limitations to our study. First, our analyses were limited to acupuncture-specific SRs/MAs and therefore may not be applicable to SRs/MAs in other fields. Second, our assessment process was not blinded, and therefore the outcomes may be influenced by publication date and other factors. Third, our assessment criteria (yes or no) did not allow partial information to be used. Fourth, our study focused primarily on acupuncture rather than other TCM. We failed to distinguish acupuncture from herbal medicine massage, or western medicine because individual SRs/MAs we included in our analysis often contained several control groups rather than one group.
In summary, SRs/MAs of acupuncture studies have not comprehensively reported the information recommended in the PRISMA statement. Our study underscores that, in addition to focusing on careful study design and performance, attention should be paid to comprehensive reporting standards when publishing SRs/MAs of acupuncture studies.
Supporting Information
Text S1.
The English and Chinese databases search strategy.
https://doi.org/10.1371/journal.pone.0113172.s002
(DOC)
Acknowledgments
We thank Yongteng Xu, Xianxia Yan, Shengping Yang, Xin Tian, Yannan Zhou,Yiming Lu, Qingshan Guo (Lanzhou University) for previous work that contributed to the development in this study. We thank BiomEditor for providing assistance with final revision of the manuscript.
Author Contributions
Conceived and designed the experiments: YLL KHY. Performed the experiments: RZ JH XZ DLL WTS PZ HC. Analyzed the data: YFM JH. Contributed reagents/materials/analysis tools: YJW. Wrote the paper: YLL RZ.
References
- 1. Oxman AD, Cook DJ, Guyatt GH (1994) Users' guides to the medical literature. VI. How to use an overview. 238 Evidence-Based Medicine Working Group. JAMA 272: 1367–1371.
- 2. Swingler GH, Volmink J, Ioannidis JP (2003) Number of published systematic 239 reviews and global burden of disease: database analysis. BMJ 327: 1083–1084.
- 3. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 339: b2535.
- 4. Sacks HS, Berrier J, Reitman D, Ancona-Berk VA, Chalmers TC (1987) Meta-analysis of randomized controlled trials. New Engl J Med 316: 450–455.
- 5. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG (2007) Epidemiology and reporting characteristics of systematic reviews. PLoS Med 4: e78 [PMID: 17388659].
- 6. Kelly KD, Travers A, Dorgan M, Slater L, Rowe BH (2001) Evaluating the quality of systematic reviews in the emergency medicine literature. Ann Emerg Med 38: 518–526.
- 7. Richards D (2004) The quality of systematic reviews in dentistry. Evid Based Dent 5: 17.
- 8. Delaney A, Bagshaw SM, Ferland A, Manns B, Laupland KB, Doig CJ (2005) A systematic evaluation of the quality of meta-analyses in the critical care literature. Crit Care 9: R575–582.
- 9. Willis BH, Quigley M (2011) The assessment of the quality of reporting of meta-analyses in diagnostic research: a systematic review. BMC Med Res Methodol 11: 163.
- 10. Delaney A, Bagshaw SM, Ferland A, Manns B, Laupland KB, Doig CJ (2005) A systematic evaluation of the quality of meta-analyses in the critical care literature. Crit Care 5: R575–82 [PMID: 16277721].
- 11. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, et al. (1999) Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet 354: 1896–1900.
- 12. Delaney A, Bagshaw SM, Ferland A, Manns B, Laupland KB, et al. (2005) A systematic evaluation of the quality of meta-analyses in the critical care literature. Crit Care 9: 575–582.
- 13. Sood A, Sood R, Bauer BA, Ebbert JO (2005) Cochrane systematic reviews in acupuncture: methodological diversity in database searching. J Altern Complement Med 11(4): 719–722.
- 14. Liu Y, Yang S, Dai J, Xu Y, Zhang R, Jiang H, Yan X, Yang K (2011) Risk of Bias Tool in Systematic Reviews/Meta-Analyses of Acupuncture in Chinese Journals. PLoS One 6(12): e28130.
- 15. Lui S, Smith EJ, Terplan M (2010) Heterogeneity in search strategies among Cochrane acupuncture reviews: is there room for improvement? Acupunct Med 28(3): 149–153.
- 16. Tao KM, Li XQ, Zhou QH, Moher D, Ling CQ, Yu WF (2011) From QUOROM to PRISMA: A Survey of High-Impact Medical Journals' Instructions to Authors and a Review of Systematic Reviews in Anesthesia. PLoS One 6(11): e27611.
- 17. Ma B, Guo JW, Qi GQ, Li H, Peng J, Zhang Y, et al. (2011) Epidemiology, Quality and Reporting Characteristics of Systematic Reviews of Traditional Chinese Medicine Interventions Published in Chinese Journals. Plos One 6(5): e20185.
- 18. Ma B, Qi GQ, Lin XT, Wang T, Chen ZM, Yang KH (2012) Epidemiology, quality, and reporting characteristics of systematic reviews of acupuncture interventions published in Chinese journals[J]. J Altern Complement Med 18(9): 813–817.
- 19. Padula RS, Pires RS, Alouche SR, Chiavegato LD, Lopes AD, Costa LO (2012) Analysis of reporting of systematic reviews in physical therapy published in Portuguese. Rev Bras Fisioter 16(4): 381–8.
- 20. Gagnier JJ, Kellam PJ (2013) Reporting and methodological quality of systematic reviews in the orthopaedic literature. J Bone Joint Surg Am 5, 95(11): e771–7.
- 21. Kiriakou J, Pandis N, Fleming PS, Madianos P, Polychronopoulou A (2013) Reporting quality of systematic review abstracts in leading oral implantology journals. J Dent 41(12): 1181–7.
- 22. Fleming PS, Seehra J, Polychronopoulou A, Fedorowicz Z, Pandis N (2013) A PRISMA assessment of the reporting quality of systematic reviews in orthodontics. Angle Orthod 83(1): 158–163.
- 23.
PROSPERO. international prospective register of systematic reviews. Available. http://www.crd.york.ac.uk/prospero/.
- 24. Moher D, Shamseer L, Clarke M, et al. (2011) Reporting 270 Guidelines for Systematic Review Protocols. The19 Cochrane Colloquium
- 25. Wu T, Li Y, Bian Z, et al. (2009) Randomized trials published in some Chinese journals: how many are randomized?. Trials 10 (1): 46.