http://orcid.org/0000-0002-3202-2832
Figures
Abstract
Background
The STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) statement was published in October 2007 to improve quality of reporting of observational studies. The aim of this review was to assess the impact of the STROBE statement on observational study reporting and study design quality in the nephrology literature.
Setting & Population
European and North American, Pre-dialysis Chronic Kidney Disease (CKD) cohort studies.
Selection Criteria for Studies
Studies assessing the association between CKD and mortality in the elderly (>65 years) published from 1st January 2002 to 31st December 2013 were included, following systematic searching of MEDLINE & EMBASE.
Outcome
Quality of study reporting using the STROBE statement and quality of study design using the Newcastle Ottawa Scale (NOS), Scottish Intercollegiate Guidelines Network (SIGN) and Critical Appraisal Skills Programme (CASP) tools.
Results
37 papers (11 Pre & 26 Post STROBE) were identified from 3621 potential articles. Only four of the 22 STROBE items and their sub-criteria (objectives reporting, choice of quantitative groups and description of and carrying out sensitivity analysis) showed improvements, with the majority of items showing little change between the period before and after publication of the STROBE statement. Pre- and post-period analysis revealed a Manuscript STROBE score increase (median score 77.8% (Inter-quartile range [IQR], 64.7–82.0) vs 83% (IQR, 78.4–84.9, p = 0.05). There was no change in quality of study design with identical median scores in the two periods for NOS (Manuscript NOS score 88.9), SIGN (Manuscript SIGN score 83.3) and CASP (Manuscript CASP score 91.7) tools.
Limitations
Only 37 Studies from Europe and North America were included from one medical specialty. Assessment of study design largely reliant on good reporting.
Conclusions
This study highlights continuing deficiencies in the reporting of STROBE items and their sub-criteria in cohort studies in nephrology. There was weak evidence of improvement in the overall reporting quality, with no improvement in methodological quality of CKD cohort studies between the period before and after publication of the STROBE statement.
Citation: Rao A, Brück K, Methven S, Evans R, Stel VS, Jager KJ, et al. (2016) Quality of Reporting and Study Design of CKD Cohort Studies Assessing Mortality in the Elderly Before and After STROBE: A Systematic Review. PLoS ONE 11(5): e0155078. https://doi.org/10.1371/journal.pone.0155078
Editor: Xu-jie Zhou, Renal Division, Peking University First Hospital, CHINA
Received: January 20, 2016; Accepted: April 24, 2016; Published: May 11, 2016
Copyright: © 2016 Rao et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: This work was part-funded by the UK Renal Registry (https://www.renalreg.org/) and forms part of AR’s Doctorate of Philosophy (PhD) thesis. It was also part-funded by the European Community's Seventh Framework Programme under grant agreement number HEALTH-F2-2009-241544 (SysKID) (K.B., V.S.S., K.J.J.), and the European Renal Association-European Dialysis and Transplant Association (ERA-EDTA-www.era-edta.org/) under the Quality European Studies initiative (K.B.). Additionally, this article was co-written by V.S.S and K.J.J on behalf of the ERA-EDTA Registry which is an official body of the ERA-EDTA. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Chronic kidney disease (CKD) is a complex chronic condition, and in recent years has emerged as a major public health problem[1, 2]. CKD has been termed a “Geriatric Giant”, as this disproportionately affects the elderly and is assuming epidemic proportions. Also with increasing life expectancy, patients are surviving longer with chronic conditions including CKD [3]. With the increasing burden of CKD, research of treatments developed to improve morbidity and mortality is vital [4]. Randomised controlled trials (RCTs) indisputably hold many advantages over observational studies, but owing to ethical or other considerations, may be difficult or impossible to undertake[5–7]. In nephrology there has not only been a lack of RCTs, but a large proportion of these RCTs have had negative or null findings[6]. Observational studies can provide extremely valuable additional evidence, and when rigorously undertaken may yield similar results as RCTs at far lower expense[8–11].
Standardized reporting of cohort studies is crucial for the evaluation of the merits and flaws of observational research. Inadequate reporting is associated with potentially biased estimates of treatment effects and limits the assessment of a study’s strengths, weaknesses and generalizability[12]. In order to address this, the STrengthening the Reporting of OBservational Studies in Epidemiology (STROBE) initiative developed recommendations on what should be incorporated in a precise and thorough report of an observational study. The STROBE statement and checklist were published in October 2007[13, 14]. These reporting guidelines were envisioned to make issues such as confounding, bias, and generalizability more ostensible. In the long term, this would improve the methodology of studies by increased awareness of these issues for researchers designing a new study[15, 16].
The scientific value and reliability of the conclusions drawn from a study are determined to a major extent by the quality of the study design[17]. A variety of tools currently exist to assess the risk of bias (methodological quality) of observational studies, and are employed when undertaking a systematic review. These include quality scales, simple checklists, or checklists with a summary judgment for assessment of the risk of bias[18].
The objectives of this review were (a) to determine whether the publication of the STROBE statement is associated with an improvement in the reporting quality of cohort studies assessing mortality in elderly patients with CKD; and (b) to determine whether the publication of the STROBE statement is associated with a decrease in risk of bias (improvement in the methodological quality) of cohort studies assessing mortality in elderly patients with CKD.
Materials and Methods
Data selection
A systematic literature search was performed in Medline and Embase using the OvidSP interface to identify all papers describing pre-dialysis CKD cohort studies in the elderly (> 65 years) where mortality was reported as an outcome. This systematic review was conducted a part of the background preparation for the EQUAL study which is an international (European) multicentre prospective observational cohort study looking at the timing of the start of dialysis in elderly patients (≥65 years) with estimated glomerular filtration rate (eGFR) of ≤ 20mls/min and therefore the review is restricted to CKD cohort studies in the elderly[19]. The search query is presented in Item A in S1 File (available as online supplementary material).
Papers published between 1st January 2002 and 31st December 2013 were included, as the KDOQI Clinical Practice Guidelines for Chronic Kidney Disease: Evaluation, Classification, and Stratification were published in 2002[20]. Only articles published in English were considered for the purposes of the review. The initial search strategy yielded more than 10,000 hits, hence the number of studies were reduced by restricting the search to European and North American studies. Each article was double sifted at title, abstract and full text stage using predefined study inclusion and exclusion criteria. Any disagreements about inclusion were resolved by discussion.
The systematic review aimed to cover reporting and design of observational studies before and after the publication of the STROBE statement which was published in October 2007. We assessed reporting and methodological quality during two time periods: before STROBE between 1/1/2002-31/12/2007 and after STROBE 1/10/2008-31/12/2013, allowing a one-year run-in period. By excluding publications in the immediate twelve months post-STROBE we allowed a period of one year for submission, revision and publication of research adhering to the new guidelines.
Data extraction
The reporting of the selected studies was assessed using the STROBE checklist itself, and the methodological quality assessed using three tools. Thirteen of the 22 STROBE checklist items were assessed with 2 to 6 questions per item generating 55 questions. The STROBE checklist is presented in Table A in S1 File (available as online supplementary material). These could be answered as “yes,” “partly,” “no,” “unclear,” or “not applicable”. We used similar methodology to that reported in the publication by Langan et al[21].
To assess methodological quality, the articles were scored on the Newcastle Ottawa Scale (NOS). At the time this study was designed NOS was recommended by Cochrane for evaluating the risk of bias in observational studies for inclusion in systematic reviews [22, 23]. The articles were also scored using the Scottish Intercollegiate Guidelines Network (SIGN) checklist for cohort studies[24], and Critical Appraisal Skills Programme (CASP) cohort studies checklist[25] to estimate concurrent validity of NOS tool. These three checklists were chosen because they were simple checklists without an additional summary judgement[26].
The eligible papers that were identified by the sifting process were each scored using the STROBE, NOS, SIGN and CASP checklists by two reviewers. Where there was disagreement between reviewers, consensus was reached by discussion.
Outcome measure
Quality of study reporting was calculated by specific STROBE items and at a manuscript level. A STROBE question score (SQS) was calculated; the number of publications in a period that adequately reported a question divided by the number of publications in which this question was applicable, expressed as a percentage (item analysis). A Manuscript STROBE score (MSS) was calculated for every manuscript; the number of questions (maximum of 55 questions) adequately reported in the publication divided by the number of applicable questions, expressed as a percentage (manuscript analysis).
Similarly, to assess the quality of study design the manuscript NOS score (MNOS), manuscript SIGN score (MSiS) and manuscript CASP score (MCAS) were calculated; the number of questions adequately addressed (in each appraisal tool) divided by the number of applicable items, expressed as a percentage in order to facilitate comparison.
Data Analysis
Comparison between pre and post-period SQS was performed by calculating the risk (proportion) difference between the two groups using the Wald test and respective 95% confidence intervals, with Benjamini and Hochberg adjusted p values (False Discovery Rate) to control for multiple testing[27]. MSS, MNOS, MSiS and MCAS were reported as a median with respective interquartile range (IQR). Pre- and post-period median MSS, MNOS, MSiS and MCAS were compared using the Mann-Whitney (MW) test. Despite excluding articles published for a period of 1 year after introduction of STROBE, this could potentially have been insufficient for uptake and penetration of new information. Therefore a spline linear regression model was used to determine the impact of STROBE over time[28]. Sub-group analyses of MSS were carried out restricting articles to those published in nephrology journals, STROBE endorsing and non-endorsing journals and by journal impact factor in the year that the article was published. Sensitivity analyses were carried out by excluding the outlying MSS if any data points were less than 1.5 interquartile ranges (IQRs) below the first quartile or above the third quartile (< Q1–1.5×IQR or > Q3 + 1.5×IQR). Simple and weighted kappa statistics were used to compare agreement between reviewers for the NOS, SIGN and CASP checklists. All tests were two-tailed, and p values, < 0.05 were considered statistically significant. Data were analysed using STATA v13.1 (College Station, TX, USA) and SAS v9.3 (SAS Institute, Cary, NC, USA) software.
Results
Reporting Quality
Fig 1 shows the flow diagram of exclusions. Of the 3621 articles initially identified by the Medline and Embase search, 3584 (98.9%) were excluded after the sifting process (Fig 1). Only 37 articles met the pre-defined selection criteria for the scoring stage of the review during the inclusion period. Of these 37 articles, 11 were in the pre-STROBE era (1/1/2002-31/12/2007) & 26 in the post-STROBE period (1/10/2008-31/12/2013). Twenty-two of these articles were published in nephrology and 15 in other medical journals. The list of articles considered at the scoring stage of the study is provided in Item B in S1 File (available as online supplementary material).
Table 1 summarises the STROBE, NOS, SIGN and CASP scores for each of the articles in the pre and post-STROBE period. In most cases, reporting quality (STROBE) and methodological quality (NOS, SIGN and CASP) correlated well. However, in some articles methodological quality scored highly with a low score for reporting and vice versa.
The citations for the manuscripts are listed in online supplementary material (Item B in S1 File).
Some of the STROBE question scores showed a ceiling effect as they were already at a maximum level in the pre-STROBE period and could therefore only remain static or decline. Others saw improvements over the period such as “choice of quantitative groups” (30% vs 71%, p = 0.02), “addressing of losses to follow up” (0% vs 36%, p < 0.001), “description of and carrying out sensitivity analysis” (18% vs 58%, p = 0.01 & 18% vs 65%, p = 0.002) and “usage of flow diagram” (0% vs 19%, p = 0.01). However, after adjusting for multiple testing, the change in only two items’ scores remained unlikely to be due to chance; “addressing of losses to follow up” (p = 0.02) and “carrying out sensitivity analysis” (p = 0.04). The majority of STROBE questions showed little improvement between the two periods. Some critical questions, such as hypothesis specification and those important to interpretation of study validity such as sample size estimation, addressing missing data, addressing loss to follow up, reason for non-participation and usage of flow diagram continue to be under reported with less than 50% reporting these items in both periods. Details regarding the reporting of the 55 STROBE items in the 37 included cohort studies are shown in Table 2.
Pre- and post-period analyses revealed an increase in MSS (median score 77.8 (IQR, 64.7–82.0) vs 83 (IQR, 78.4–84.9), p = 0.04) (see Table 3). Any pre-STROBE period articles with MSS scores less than 47.4 and post-STROBE period less than 69 were considered to be outliers. Excluding outliers, the improvement in the MSS between the two periods showed a stronger statistical relationship (p = 0.01). The results were essentially unchanged when restricted to nephrology journals or stratified by STROBE endorsing or non-endorsing journals, though there was less statistical power to test for differences. Journals with impact factor < 5 saw greater change over the two periods when compared to journals with impact factor ≥ 5 but given the overlap in the confidence intervals this may have occurred by chance.
Time series analysis of MSS showed that there was a significant improvement in the quality of reporting in the latter three years (1/1/11 to 31/12/13) when compared to the first three years (1/10/2008 to 31/12/2010) after the introduction of the STROBE statement (Table 4). Longitudinal analysis of the MSS using a spline linear regression model (Fig 2), having excluded outliers, suggested a turning point in 2008 with a slight negative trend in the pre-STROBE period (coefficient—0.06, SE 0.11) and a positive slope in the post-STROBE period (coefficient 0.21 SE 0.05) but this may have occurred by chance (Slope change coefficient 0.27, SE 0.16; p value = 0.10).
Methodological quality (study design); comparison in the pre- and post-STROBE period
We found no evidence for any change in the methodological quality of studies in the pre and post-STROBE period using the Newcastle Ottawa Scale (NOS) (median MNOS 88.9% [IQR, 66.7–100] vs 88.9% [IQR, 88.9–100], p = 0.51), Scottish Intercollegiate Guidelines Network (SIGN) (median MSiS 83.3% [IQR, 61.5–100] vs 83.3% [IQR, 70–90.9], p = 0.93) and Critical Appraisal Skills Programme (CASP) (median MCAS 91.7% [IQR, 83.3–100] vs 91.7% [IQR, 83.3–100], p = 0.93) (Fig 3).
Inter-rater agreement
Agreement between raters for the NOS, SIGN and CASP tools was calculated using the simple or weighted Kappa coefficient. These were assessed at three levels: raters’ agreement on applicability, clarity (can’t say) and yes/no. The inter-rater agreement for each of the tools was overall inadequate, with the NOS tool having poor agreement between the three pairs of raters’. The CASP tool fared slightly better compared to the SIGN tool in raters’ assessment of clarity. A summary table of Kappa coefficients is included in Table B in S1 File (available as online supplementary material).
Discussion
This systematic review assessed the impact of the publication of the STROBE statement on quality of study design and reporting of methodology. It showed that, after publication of STROBE, a large proportion of the STROBE items and sub-criteria continue to be underreported in CKD cohort studies of mortality in elderly patients. Reporting rates were lowest for hypothesis specification, usage of flow diagrams and addressing missing data. There was evidence of improvement in the reporting quality of CKD cohort studies particularly in the latter three years of the post-STROBE period, which was also seen when looking at the temporal patterns but this may have occurred by chance. We found no evidence that the quality of study design as assessed by 3 different tools NOS, SIGN and CASP had improved. However, these quality assessment tools have poor to moderate inter-rater reliability and might not be suitable for use without consensus agreement between raters.
The publication of CKD guidelines in 2002 has potentially had an impact on the volume of CKD research with approximately 2.5 times the number of studies in the post-STROBE period compared to the pre-STROBE period[20].
Inadequate reporting not only hinders critical assessment by others of the strengths and weaknesses in study design, conduct, and analysis, it affects judgement of whether and how results can be included in systematic reviews and also impacts on the reader assessment of the studies generalizability [29]. Our results are consistent with other studies assessing deficiencies in reporting of individual STROBE items such sample size, use of flow diagram and reporting of missing data [21, 30–35].
A number of studies, including a Cochrane review, have demonstrated improvements in reporting quality of randomised control trials (RCTs) after the introduction of the Consolidated Standards of Reporting Trials (CONSORT) statement with a significant improvement in journals endorsing this guideline statement[36–40]. An RCT has also shown that using reporting guidelines in the peer review process improves the quality of manuscripts[41]. Our study showed weak evidence of improvement in the quality of reporting of CKD cohort studies over time following the introduction of the STROBE statement. The improvements unfortunately fell short of the intended expectations when compared to the impact the CONSORT statement had achieved upon the reporting quality of RCTs. These results were similar to the only other study looking at quality of reporting, published in the dermatology literature. Those authors attributed the lack of improvement to the short follow up period after STROBE introduction (2008–10)[30]. However, in our study the small improvement could be attributable to the fact that the reporting of nephrology literature in the pre-STROBE period was already of a higher standard (median MSS 77.8 IQR 64.7–82.0) in comparison to dermatology literature (median score 58 IQR 46–73).
Journal endorsement of reporting guidelines has been shown to improve reporting quality of manuscripts submitted to journals[41]. Given that only two medical journals (British Medical Journal & Ageing) and one renal journal (American Journal of Kidney Diseases) included in this review had endorsed the STROBE statement, any evidence of improvement in reporting quality of cohort studies in nephrology literature is probably attributable to the penetration of STROBE statement over time rather than to its endorsement by journals[42]. The lack of improvement of reporting standards seen in the STROBE endorsing journals is not an indictment of these journals but maybe attributable to the small sample size to accurately test for differences between the groups. An important observation that was made during the process of this review was that despite studies having similar reporting standards, reflected by their similar MSS, some studies had failed to adequately report essential criteria.
For most of the articles included in this study reporting and methodological quality were well correlated, however the assessment of the methodological quality of a study is largely dependent on adequate reporting of the research. Therefore, drawing any inferences about a study’s design quality is made harder if the reporting quality is inadequate.
One of the main goals of reporting guidelines was to improve reporting clarity and not necessarily improve the quality of research, but in due course achieve it as an indirect effect. Due to interchangeable usage of the terminology ‘reporting quality’ and ‘methodological quality’, the STROBE statement has often been used inappropriately for the assessment of methodological quality of observational research[16]. There are a number of assessment tools that have been developed to assess quality and susceptibility to bias in observational studies with only half of the identified tools have described their development or validity and reliability [26]. The review by Sanderson et al highlighted the lack of a single obvious tool for assessing quality of observational epidemiological studies[26]. The bias assessment tools used in this study (NOS, SIGN and CASP) were subjective, differed by content, format and validity. The bias assessment tools identified deficiencies in the articles relating to consideration of participant’s lost to follow up (attrition bias), exposure level or prognostic factor measured only once (detection bias), and inadequate methods of outcome assessment (detection bias). However, given that the assessment of methodological quality is largely reliant on the reporting of study design, one might therefore fail to detect differences in design quality if reporting is inadequate. Also given the latency period of designing a new study, undertaking it and then publishing it, might have been simply too soon for the STROBE statement to have influenced the methodological quality of studies. The NOS tool was previously recommended by Cochrane for evaluating the risk of bias but published literature has demonstrated poor inter-rater reliability between individual reviewers[43, 44]. The results of our study are consistent with these findings as all of the three tools (NOS, SIGN and CASP) showed poor agreement between individual reviewers. The usability of a tool depends on its clarity. Moreover, the tools contain items whose scoring is subjective and dependent on reviewers’ perceptions and domain knowledge. Cochrane now recommends the ACROBAT-NRSI bias assessment tool for non-randomized studies which has been developed by members of the Cochrane Bias Methods Group and the Cochrane Non-Randomised Studies Methods Group[45, 46]. However, at the time of drafting this manuscript, this tool remains yet to be tested for consistency between individual reviewers.
A strategy to improve inter-rater agreement would be tailoring and training of reviewers prior to implementation of the tools. Due to the poor reliability of the tools demonstrated here, it should be strongly considered that each study should be assessed by at least two reviewers prior to inclusion in a systematic review/meta-analysis.
One of the strengths of our study is that we studied the impact of STROBE upon both quality of reporting and study design. The study has good internal validity as the selection and evaluation processes were independently performed by two reviewers. However, as the articles were included from one field of medicine (CKD) we must be cautious in generalising our findings to other areas. The other limitation of the study was that it only covered articles from Europe and North America. There was also an imbalance in the number of studies assessed in the two periods probably due to the KDIGO CKD guidelines which were published in 2002. This imbalance could have potentially introduced a lack of power to detect difference in quality. It was also impossible to blind the reviewers to the publication date during the sifting stage of the review, and the journal name during the review of quality which could have biased the reviewers’ assessment of quality of the study. Finally, whilst we examined a five-year period post-STROBE, it is possible that we failed to find any benefit for methodological quality due to the long latency period between designing a new study, obtaining funding, undertaking data collection, analysis and publication.
Conclusion
This study highlights continuing deficiencies in the reporting of observational studies in the nephrology literature. However, the publication of the STROBE statement may have positively influenced the quality of some aspects of observational study reporting. There was no evidence, however, that methodological quality improved over this time period. With continued efforts from researchers and with particular focus on the domains identified as deficient by the STROBE statement and bias reporting tools, this presents an opportunity to improve the validity of observational research in nephrology. With increased awareness by authors and editors regarding compliance of manuscripts to the STROBE statement and journal endorsement of the STROBE statement, we hope that not only reporting but also the design of future studies will be improved.
Supporting Information
S1 File.
Item A: Search strategy for systematic review. Table A: STROBE scoring sheet. Item B: List of articles included for review by date of publication. Table B: Summary of simple and weighted Kappa coefficient, measuring agreement between reviewers for the NOS, SIGN and CASP tool.
https://doi.org/10.1371/journal.pone.0155078.s001
(DOCX)
Acknowledgments
We thank Ingeborg M Nagel, medical information specialist and clinical librarian at Academic Medical Center, University of Amsterdam, Netherlands, for valuable input into search strategy for the systematic review
Author Contributions
Conceived and designed the experiments: AR FC. Performed the experiments: AR FC KB SM KJ VS. Analyzed the data: AR RE. Contributed reagents/materials/analysis tools: AR FC KB SM VS KJ YB LH. Wrote the paper: AR FC KB SM RE VS KJ YB LH.
References
- 1. Mangione F, Dal Canton A. Chronic kidney disease epidemic: myth and reality. Intern Emerg Med. 2011;6(1):69–76.
- 2. Jha V, Garcia-Garcia G, Iseki K, Li Z, Naicker S, Plattner B, et al. Chronic kidney disease: global dimension and perspectives. The Lancet. 382(9888):260–72.
- 3. Ahmed Abdelhafiz SA, Flint K, El Nahas AM. Is Chronic Kidney Disease in Older People a New Geriatric Giant? Aging Health. 2011;7(5):749–62.
- 4. El Nahas AM, Bello AK. Chronic kidney disease: the global challenge. The Lancet. 365(9456):331–40.
- 5. Kovesdy CP, Kalantar-Zadeh K. Observational Studies Versus Randomized Controlled Trials: Avenues to Causal Inference in Nephrology. Advances in Chronic Kidney Disease. 19(1):11–8. pmid:22364796
- 6. Novak JE, Inrig JK, Patel UD, Califf RM, Szczech LA. Negative trials in nephrology: what can we learn? Kidney Int. 2008;74(9):1121–7. pmid:18563051
- 7. Jager KJ, Stel VS, Wanner C, Zoccali C, Dekker FW. The valuable contribution of observational studies to nephrology. Kidney Int. 2007;72(6):671–5. http://www.nature.com/ki/journal/v72/n6/suppinfo/5002397s1.html. pmid:17597701
- 8. Concato J, Shah N, Horwitz RI. Randomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs. New England Journal of Medicine. 2000;342(25):1887–92. pmid:10861325.
- 9. Thadhani R, Tonelli M. Cohort Studies: Marching Forward. Clinical Journal of the American Society of Nephrology. 2006;1(5):1117–23. pmid:17699334
- 10. Benson K, Hartz AJ. A Comparison of Observational Studies and Randomized, Controlled Trials. New England Journal of Medicine. 2000;342(25):1878–86. pmid:10861324.
- 11. Johnston SC, Rootenberg JD, Katrak S, Smith WS, Elkins JS. Effect of a US National Institutes of Health programme of clinical trials on public health and costs. The Lancet. 367(9519):1319–27.
- 12. Tooth L, Ware R, Bain C, Purdie DM, Dobson A. Quality of Reporting of Observational Longitudinal Research. American Journal of Epidemiology. 2005;161(3):280–8. pmid:15671260
- 13. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. The Lancet. 370(9596):1453–7.
- 14. Nijsten T, Spuls P, Stern RS. Strobe: A beacon for observational studies. Archives of Dermatology. 2008;144(9):1200–4. pmid:18794467
- 15. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for reporting observational studies. International Journal of Surgery. 12(12):1495–9. pmid:25046131
- 16. da Costa BR, Cevallos M, Altman DG, Rutjes AWS, Egger M. Uses and misuses of the STROBE statement: bibliographic study. BMJ Open. 2011;1(1).
- 17. Carlson MDA, Morrison RS. Study Design, Precision, and Validity in Observational Studies. Journal of Palliative Medicine. 2009;12(1):77–82. PMC2920077. pmid:19284267
- 18. Shamliyan T, Kane RL, Dickinson S. A systematic review of tools used to assess the quality of observational studies that examine incidence or prevalence and risk factors for diseases. Journal of Clinical Epidemiology. 2010;63(10):1061–70. pmid:20728045
- 19. Jager KJ, Ocak G, Drechsler C, Caskey FJ, Evans M, Postorino M, et al. The EQUAL study: a European study in chronic kidney disease stage 4 patients. Nephrology Dialysis Transplantation. 2012.
- 20. National Kidney Foundation. K/DOQI Clinical practice guidelines for chronic kidney disease: evaluation, classification and stratification. American Journal of Kidney Diseases. 2002;39(Suppl 2):S1–S246.
- 21. Langan S, Schmitt J, Coenraads P, Svensson A, von Elm E, Williams H, et al. The reporting of observational research studies in dermatology journals: A literature-based study. Archives of Dermatology. 2010;146(5):534–41. pmid:20479302
- 22.
Wells GA, Shea B, O'Connell D, Peterson J, Welch V, Losos M, et al. The Newcastle-Ottawa Scale (NOS) for assessing the quality if nonrandomized studies in meta-analyses. Available: http://www.ohri.ca/programs/clinical_epidemiology/oxford.htm. Accessed July 28, 2015.
- 23.
Higgins JPT, Green S (Eds): Cochrane handbook for systematic reviews of interventions version 5.1.0 [Accessed March 2011]. Available: http://handbook.cochrane.org/chapter_13/13_5_2_3_tools_for_assessing_methodological_quality_or_risk_of.htm. Accessed July 28, 2015.
- 24.
Scottish Intercollegiate Guidelines Network (SIGN) checklist for cohort studies. http://www.sign.ac.uk/methodology/checklists.html. Accessed July 28, 2015. Available: http://www.sign.ac.uk/methodology/checklists.html.
- 25.
CASP UK. Critical Appraisal Skills Programme (CASP) checklist for cohort studies. http://www.casp-uk.net/#!casp-tools-checklists/c18f8. Accessed July 28, 2015. Available: http://www.casp-uk.net/#!casp-tools-checklists/c18f8.
- 26. Sanderson S, Tatt ID, Higgins JP. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. International Journal of Epidemiology. 2007;36(3):666–76. pmid:17470488
- 27. Benjamini Y, Hochberg Y. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society Series B (Methodological). 1995;57(1):289–300.
- 28. Wagner AK, Soumerai SB, Zhang F, Ross-Degnan D. Segmented regression analysis of interrupted time series studies in medication use research. Journal of Clinical Pharmacy and Therapeutics. 2002;27(4):299–309. pmid:12174032
- 29. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. Strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ: British Medical Journal. 2007;335(7624):806–8. PMC2034723. pmid:17947786
- 30. Bastuji-Garin S, Sbidian E, Gaudy-Marqueste C, Ferrat E, Roujeau J-C, Richard M-A, et al. Impact of STROBE Statement Publication on Quality of Observational Study Reporting: Interrupted Time Series versus Before-After Analysis. PLoS ONE. 2013;8(8):e64733. pmid:23990867
- 31. Müller M, Egger M. Strengthening the reporting of observational epidemiology (STROBE) in sexual health. Sexually Transmitted Infections. 2009;85(3):162–4. pmid:19478105
- 32. Poorolajal J, Cheraghi Z, Irani AD, Rezaeian S. Quality of Cohort Studies Reporting Post the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement. Epidemiol Health. 2011;33(0):e2011005–0.
- 33. Fung AE, Palanki R, Bakri SJ, Depperschmidt E, Gibson A. Applying the CONSORT and STROBE Statements to Evaluate the Reporting Quality of Neovascular Age-related Macular Degeneration Studies. Ophthalmology. 116(2):286–96.e4. pmid:19091408
- 34. Sorensen AA, Wojahn RD, Manske MC, Calfee RP. Using the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement to Assess Reporting of Observational Trials in Hand Surgery. The Journal of Hand Surgery. 2013;38(8):1584–9.e2. pmid:23845586
- 35. Papathanasiou AA, Zintzaras E. Assessing the Quality of Reporting of Observational Studies in Cancer. Annals of Epidemiology. 2010;20(1):67–73. pmid:20006277
- 36. Plint Amy C M D, Morrison Andra, Schulz Kenneth, Altman Douglas G, Hill Catherine and Gaboury Isabelle. Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Med J Aust 2006;185(5):263–7. pmid:16948622
- 37. Moher D, Jones A, Lepage L, for the CG. Use of the consort statement and quality of reports of randomized trials: A comparative before-and-after evaluation. JAMA. 2001;285(15):1992–5. pmid:11308436
- 38. Anna Partsinevelou EZ. Quality of reporting of randomized controlled trials in polycystic ovary syndrome. Trials. 2009;10(106).
- 39. Lucy Turner LS, Altman Douglas G, Schulz Kenneth F, Moher David. Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review. Systematic Reviews. 2012;1(60).
- 40. Kane RL, Wang J, Garrard J. Reporting in randomized clinical trials improved after adoption of the CONSORT statement. Journal of Clinical Epidemiology. 2007;60(3):241–9. pmid:17292017
- 41. Cobo E, Cortés J, Ribera JM, Cardellach F, Selva-O’Callaghan A, Kostov B, et al. Effect of using reporting guidelines during peer review on quality of final manuscripts submitted to a biomedical journal: masked randomised trial2011 2011-11-22 12:16:35.
- 42.
STROBE Statement. STROBE Endorsement. Available: http://www.strobe-statement.org/index.php?id=strobe-endorsement. Accessed October 29, 2015. Available: http://www.strobe-statement.org/index.php?id=strobe-endorsement.
- 43. Hartling L, Milne A, Hamm MP, Vandermeer B, Ansari M, Tsertsvadze A, et al. Testing the Newcastle Ottawa Scale showed low reliability between individual reviewers. Journal of Clinical Epidemiology. 66(9):982–93. pmid:23683848
- 44. Stang A. Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol. 2010;25(9):603–5. pmid:20652370
- 45. Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343.
- 46.
Sterne JAC HJ, Reeves BC on behalf of the development group for ACROBAT-NRSI. A Cochrane Risk Of Bias Assessment Tool: for Non-Randomized Studies of Interventions (ACROBAT-NRSI), Version 1.0.0. http://www.riskofbias.info. Accessed Ocotber 30, 2015. 24 September 2014. Available from: http://www.riskofbias.info.