Study overview

We were guided by the Preferred Reporting Items for Systematic Reviews and Meta‑Analysis (PRISMA) statement [7], PRISMA 2009 Checklist (S1 Checklist), and the Cochrane Handbook for Systematic Reviews of Interventions[8] to undertake this systematic review and report the results. As part of this process, we identified research questions a priori and registered the review protocol with the international prospective register of systematic reviews (PROSPERO registration number: 42017073205).

Objectives

The objective of this systematic review was to critically appraise the overall quality of published guidelines for the prevention of SSI using the Appraisal of Guidelines for Research and Evaluation II (AGREE II) tool. Subsumed under this objective were the following questions:

1. Using the AGREE II tool, what is the quality of the CPGs and strength of their recommendations for the prevention of SSI?
2. Have the CPGs been revised, updated or improved over time?

As part of the appraisal process, we evaluated the similarities and differences of the main recommendations of the guidelines in parallel with the evaluation of their quality.

Eligibility criteria

We applied the following inclusion criteria:

• Published international and national guidelines on the management and/or prevention of SSI;
• Published as full-text between January 1990 to February 2018;
• Guidelines published in English, as these are the most accessible and widely available;
• Most recent complete guideline (from a single working group i.e. CDC) and any partial revisions for the guideline published thereafter;
• SSI prevention/management guidelines that make recommendations across the pre-operative, intra-operative and post-operative phases; and,
• Include an explicit statement identifying the document as a ‘guideline’.

We applied the following exclusion criteria:

• Guidelines under development;
• Guidelines specific to one institution or surgical specialty (i.e. local hospital guidelines or orthopaedic surgery, e.g., Smith & Dahlen[9]);
• Guidelines inclusive of only one phase of care, e.g., Ubbink et al[10] (i.e., postoperative phase focusing on wound care and pain management)
• Complete guidelines with publication dates that have been superseded by more recent complete guidelines (i.e. the 1999 CDC guidelines have superseded the 1992 guidelines);
• Guidelines that only cover one aspect of SSI prevention (i.e. antibiotic prophylaxis); and,
• Clinical practice standards, defined as a statement reached through consensus, which clearly identifies the desired outcome. Usually used in audit as a measure of success.[11]

Data sources and search strategy

With the assistance of a health librarian, we conducted systematic electronic searches using the Cochrane Library, Cumulative Index of Nursing and Allied Health Literature (CINAHL), EMBASE, MEDLINE and ProQuest databases. We tailored searches to the capabilities of each database, and where appropriate used either Medical Subject Headings (MeSH) or keywords, and Boolean connectors AND, OR and NOT to combine search terms. Since computerised searches identify only 50% of eligible studies,[12] ancestry searching and journal hand-searching were also conducted. We imported all database results into an Endnote (v X7, Clarivate Analytics) reference manager program prior to screening. We supplemented the database searches by searching guidelines repositories. These included The National Guideline Clearinghouse, New Zealand Guidelines Group, The National Institute for Health and Care Excellence (NICE), The National Health and Medical Research Council (NHMRC)–Australian Clinical Practice Guidelines, CPG Infobase: Clinical Practice Guidelines (Canadian Medical Association), Scottish Intercollegiate Guideline Network (SIGN), Clinical Key (Elsevier), and BMJ Best Practice. An example of the searches undertaken across database and guideline repositories is provided in S1 Table, MEDLINE (EBSCO) search strategy and guideline repository searches.

Data screening and extraction

Groups of titles and abstracts were assigned to three review authors who independently screened their allocated sample, and those deemed to meet the inclusion criteria were assessed further in full text. A fourth reviewer arbitrated where there was a lack of clarity around inclusion. We documented the reasons for exclusion. One review author performed data extraction using an extraction tool specifically developed for this review based on guideline characteristics relative to quality (AGREE II [13]).

Quality assessment of CPGs

To appraise the overall quality of each included CPG, we used the Appraisal of Guidelines, Research and Evaluation (AGREE) II statement [13] to guide our systematic assessment of all published CPGs eligible for inclusion. It is generic and can be applied to guidelines in any disease area targeting any step along the healthcare continuum, including screening, diagnosis, treatment or interventions.[13] The AGREE II tool has 23 items comprising six quality-related domains (Scope and purpose, Stakeholder involvement, Rigour of development, Clarity of presentation, Applicability and Editorial independence) and a 24^th overall assessment item.[13] A brief description of the criteria in each of these domains is included as an S2 Table. AGREE II domain definitions. The AGREE tool was updated to AGREE II to (i) encompass a greater consideration of health generally as opposed to a specific ‘clinical’ focus; (ii) move items to more appropriate domain categories; and (iii) the addition of items (i.e. strength and limitations of the body of evidence in Domain 3: Rigour of Development).[13]

The redesigned AGREE II tool uses a 7-point Likert scale, 1 = strongly disagree through to 7 = strongly agree.[14] Scores for each domain were calculated by summing all scores of the individualised items within each domain and then standardising as follows: (obtained score minus minimal possible score) divided by (maximum possible score minus minimum possible score).[14] The minimum standardised score for each domain was 0% and the maximum was 100%. Scores were converted to percentages values for %mean score determination. The scale was transformed to the following: 1 = 14.29%, 2 = 28.57%, 3 = 42.86%, 4 = 57.14%, 5 = 71.43%, 6 = 85.71%, and 7 = 100%. A guideline is ‘‘ recommended” if most of the domains (≥ 4) scored above 50%. A guideline is ‘‘recommended with modifications” if 3 domain items scored above 50%. A guideline is ‘‘not recommended” if ≥4 domains scoring less than 50%. Appraisers assigned scores depending on the completeness and quality of reporting and scores increased as more criteria were met.

CPGs deemed eligible for inclusion were appraised independently by members of the authorial team using the AGREE II tool.[13] Each CPG was assigned three appraisers. The AGREE II requires at least two appraisers to reach acceptable interrater agreement on the tool.[14] We classified the degree of agreement according to the scale by Landis and Koch;[15] poor (0.00), slight (0.00 and 0.20), fair (0.21 to 0.40), moderate (0.41 to 0.60), substantial (from 0.61 to 0.80), and very good or almost perfect (0.81 to 1.00). Prior to completing appraisals of each CPG, all review authors completed the AGREE II Online Training Tool [16] and participated in three rounds of calibration. Authors completed online appraisals using the My AGREE PLUS platform.[17] During the quality appraisal process, we met regularly to discuss results, clarify information and resolve differences by consensus. We measured the mean proportion of agreement relative to overall quality (AGREE II) of included CPGs among three assessors in each AGREE domain using the ICC with 95% confidence intervals (CI).

Quality appraisal

Three assessors appraised each CPG. The %mean scores for each AGREE II domain were calculated by summing the scores of the individual items in domains and then standardising minimum/maximum scores ranging from 0% to 100%. The results were as follows: 1) scope and purpose (%mean ± SD = 86.3±23.5); 2) stakeholder involvement (%mean ± SD = 64±31.0); 3) rigour of development (%mean ± SD = 68.7±30.6); 4) clarity and presentation (%mean ± SD = 88.5±16.7); 5) applicability (%mean ± SD = 44±30.2); and, 5) editorial independence (%mean ± SD = 61±37.6).

The overall quality scores of each guideline (including updates) across each domain of the AGREE II are presented in Table 2. Where CPGs included updates, we appraised the update as part of the original guideline. In the first AGREE II domain, Scope and Purpose, quality scores ranging from 39% (±14.3) to 100% (±0) with 5 of 6 CPGs scores over 50%. The second domain Stakeholder Involvement, mean scores across CPGs varied from 15% (±13.3) to 100% (±0), with 4 of 6 of CPGs with scores greater than 50%. In the third AGREE II domain, Rigour of Development, mean scores ranging from 21% (±20.5) to 97% (±6.9) with 4 of 6 CPGs scored over 50%. The fourth domain, Clarity of Presentation mean scores varied from 56% (±21.6) to 100% (±0) and all CPGs scored over 50%. In the fifth domain, Applicability, mean scores were much lower overall, ranging from 4% (±6.5) to 86% (±13.4), and only 3 of 6 CPGs had scores greater than 50%. In the sixth domain, Editorial Independence, 4 of 6 CPGs scored over 50%, with scores ranging from 11% (±11.7) to 100% (±0). Based on the appraisal of individual AGREE II domains and overall scores, 4 of 6 of the included CPGs were rated as “recommended”.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. AGREE II scaled domain scores of CPGs for SSI prevention.

https://doi.org/10.1371/journal.pone.0203354.t002

Across the AGREE II domains, ICC coefficients among appraisers ranged from 0.86 (95%CI 0.73–0.94) to 0.98 (95%CI 0.96–0.99; p<0.001), indicating an almost perfect level of agreement.[15]

Levels of evidence used in CPGs to inform recommendations

Table 3 shows the levels of evidence for recommendations across each of the three phases of surgical care, i.e., pre-operative, intra-operative and post-operative. Two of the recently developed complete guideline,[21, 22] and recent updates of two others [23, 30] used the GRADE system to rank recommendations. Only one CPG [25] was developed using a working group (based on expert opinion). Comparatively, there was consistent agreement in the ranking of recommendations across CPGs (including updates) relative to the following SSI prevention interventions; hair removal, antibiotic prophylaxis, and the wearing of surgical attire. However, across most other SSI prevention interventions/strategies in the preoperative and intraoperative period, agreement relative to level of evidence and the number of recommendations was inconsistent. S4 Table–Evidence level systems used across CPGs details the different evidence systems used for each recommendation identified in each CPG. S5 Table–Recommendations across all CPGs that informed Table 3 details specific recommendations across each CPG. The 2017 CDC guideline [23] included content specific to prosthetic joint arthroplasty. There were far fewer recommendations identified across included CPGs in relation to the post-operative phase, particularly pertaining to wound care.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Levels of evidence for recommendations for SSI prevention as reported in included CPGs.

https://doi.org/10.1371/journal.pone.0203354.t003

Strengths and limitations

As with all systematic reviews, we acknowledge some limitations. The inclusion of CPGs covering all phases of surgical care and across all specialties meant that we necessarily excluded high quality CPGs [10] that were more focussed and specific, and perhaps more user-friendly to busy clinicians, such as Ubbink et al.[10] Although our search methods were exhaustive and robust, we may have possibly missed other CPGs and updates. However, our extensive search strategy covered all indexed and grey literature, and used multiple appraisers who undertook training and calibration to assess the quality of the CPGs. We used an appraisal tool with established validity and reliability [14] and all reviewers independently appraised CPGs. However, there may be different levels of understanding of the AGREE II tool among appraisers. To address this, we held regular meetings to ensure consistency in the appraisal process across the included CPGs. These discussions offered appraisers the opportunity to present information overlooked by others in the team, therefore clarifying and increasing understanding of the criteria upon which to evaluate the CPGs. Finally, our research team comprised of healthcare professionals from varied professional disciplines, research expertise and experience, thus adding a deeper dimension to guideline interpretation and appraisal.

Implications for translation

Implementation of evidence-based information remains a challenge in many healthcare contexts and it is often difficult to assess application and performance of a CPG in clinical practice. It takes approximately 5 years for any given CPG to be adopted into routine clinical practice and even the broadly accepted guidelines are often not fully followed.[32, 42] Multiple factors influence guideline use including patient, provider, institutional context and systems issues; yet implementation is meant to overcome these barriers.[32, 46] Implementation tools that increase guideline accessibility using a variety of user-friendly formats.[31] For instance, presenting information found in the CPGs as recommendations with evidence summaries, repositories for tools for implementation, and implementation plans and toolkits make guidelines more accessible. Developers should also consider the feasibility and acceptability of implementing SSI prevention interventions across patient groups and different clinical settings, including those in developing countries.

Good information is essential for choosing wisely clinical interventions and treatments in SSI prevention, thus avoiding wasteful healthcare. Most of the reviewed CPGs lacked information about cost-effectiveness and risk-benefit analyses of the strategies used in SSI prevention. Many of the SSI prevention interventions used across the pre-, intra- and post-operative periods have never been subjected to rigorous economic evaluation,[48] but nevertheless continue to be used in clinical practice. Thus, it is important to conduct rigorous parallel economic evaluations alongside trials of clinical effectiveness,[49] as this will provide greater guidance to healthcare decision and policy makers. In terms of applicability and implementation, the inclusion of cost analyses studies in CPGs will assist clinicians in selecting the best available evidence-based options in healthcare organisations with limited resources.[49, 51]