Introduction

Accurate, timely, and reliable public health data are essential for the delivery of high-quality healthcare services. In voluntary medical male circumcision (MC) programs, a key indicator of program quality is the rate of adverse events (AEs)[1]. AEs in clinical MC settings are uncommon with few mortalities [2]. After clinical trials in sub-Saharan Africa demonstrated that MC was safe [3–5], an AE rate of 2% moderate or severe AEs became a commonly-used standard for safety [6, 7]. In general, severe AEs are those requiring surgical intervention or hospitalization, whereas any AE not classified as severe, but which requires intervention by a health care provider or medication, is considered moderate [8, 9]. AEs are also categorized by type, e.g. infection, bleeding, etc. [9]. AE rates may be higher in MC programs operating at scale without the clinical oversight and control of research studies. Although AE reporting definitions may differ slightly, field settings with active surveillance (proactive patient follow-up) often report higher AE rates, varying from 7% [10, 11] to nearly 18% [12]. In contrast, passive surveillance settings (typically routine care settings without added proactive patient tracing) rely predominantly on clients presenting at a health facility. AE rates from passive surveillance in sub-Saharan Africa report lower AE rates, beneath the 2.0% threshold [13, 14]. In countries and settings with severe healthcare shortages and fewer resources, including Zimbabwe [15], passive surveillance may be weak [16].

Lack of reliable, quality data jeopardizes program integrity and, potentially, patient safety. Between 2008 and 2016, Zimbabwe’s Ministry of Health and Child Care (MoHCC) reported 807,060 MCs [17] and a national AE rate of 0.35%, an estimated 2,825 AEs [18]. However, using the global standard of 2% AEs, Zimbabwe should have approximately 16,141 complications. AE identification, documentation and reporting among MC programs in sub-Saharan Africa may be affected by poor data quality [19–21]. In Zimbabwe, previous studies on data quality and AE identification within the MC program revealed challenges to correct and complete data [22, 23]. The low AE rate in Zimbabwe may indicate potential AE underreporting.

ZAZIC, named for the partners that formed a consortium in 2013 (International Training and Education Center for Health (I-TECH), Zimbabwe Association of Church related Hospitals (ZACH) and Zimbabwe Community Health Intervention Research Project (ZiCHIRe)), cooperates with Zimbabwe’s MoHCC to implement a large-scale, integrated MC program across 10 districts [24]. Ensuring quality service provision is a critical component of ZAZIC’s MC program. ZAZIC implements routine monitoring and evaluation (M&E) activities to identify and address weaknesses in data collection, data reporting, and data use. From October, 2014 to September, 2017, ZAZIC conducted 205,847 MCs [25]. Over this highly productive period, the reported moderate and severe AE rate from the routine, passive surveillance system was 0.3% [13, 24, 26], a rate far lower than the global standard. ZAZIC’s reported adherence to the standard indicator of “at least one follow-up visit within 14 days of MC,” another indicator of program quality [27], was high at 95% (P. Marongwe, personal communication, May 31, 2018). However, as with other passive surveillance programs, ZAZIC was concerned that the low rate of reported AEs could reflect potential underreporting.

Therefore, to review AE data and confirm the reported AE rate, ZAZIC conducted a quality assurance (QA) study in 2017. The study had two objectives: 1) to strengthen MC program AE documentation and 2) to increase confidence in the quality of reported AE data. The mixed-method study took place from May, 2017 to August, 2017 and included three components: 1) 100 prospective tandem MC reviews with a gold-standard (GS) ZAZIC clinician working alongside site-based, MoHCC MC clinicians; 2) retrospective review and comparison of three previous months of routine AE data from MC registers, client intake forms (CIF), monthly return forms (MRF) and District and Health Information Systems (DHIS2); and 3) brief qualitative interviews with MoHCC site staff about AE-related practices. GS clinicians provided in-person mentoring, support, and feedback on the results of both the prospective and retrospective activities before concluding site visits. It is hoped these results will improve confidence in ZAZIC’s AE data and spur similar QA efforts across other MC programs operating at scale.

Methods

Results

Prospective tandem reviews

The expected observed tandem reviews were 600; 585 (97.5%) reviews were performed (Table 1). Of the 585 clients, twenty-seven were identified with AEs (4.6%) across all sites: 6 were severe AEs (22%) and 21 (78%) moderate AEs. The observed AE rates ranged from 1.0–8.0%, with an average AE rate of 4.6%. Of all observed AEs, 17 (63.0%) occurred within 7 days post-operative. The ages of those observed with an AE ranged from 10–26 years, with an average age of 11.9 years. Of the 27 AEs observed, 22 (81%) were ages 14 and under.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Characteristics of the 27 AEs observed during tandem reviews.

https://doi.org/10.1371/journal.pone.0218137.t001

Retrospective record review

The numbers of reported AEs should be consistent across all reporting forms. AEs were not recorded nor reported across all forms consistently (Table 2). Only one site, Site 5, had consistent AE reporting across records: they reported zero AEs during the 3-month period reviewed.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Retrospective AEs found during AE form review.

https://doi.org/10.1371/journal.pone.0218137.t002

Estimating unidentified AEs

AE rates from the prospective, tandem reviews (observed) were higher than those documented from retrospective data (reported in DHIS2) (Table 3). Applying the observed AE rates to 95% of the previous MCs at the six sites as an estimation of unreported AEs, it is possible that 3,431 AEs (95% CI: 1,483–8,070) could have been missed in those 6 sites over three years of routine ZAZIC program implementation. Pooling the overall MC output from the 6 sites, and using the average AE rate of 4.62%, a slightly more conservative 3,032 AEs (95% CI: 2,044–4,419) may have been unidentified in those same sites between October, 2014 and September, 2017.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Potential missed adverse events in 6 study sites: October 2014- September 2017.

https://doi.org/10.1371/journal.pone.0218137.t003

Discussion

ZAZIC implemented a mixed-method, quality assurance activity in 6 purposively selected, high-volume MC sites. In contrast to reported moderate/severe AE rates ranging from 0.1%-0.6%, AE rates of 1.0–8.0% were observed through prospective, tandem, post-operative MC reviews. Retrospective record review and site interviews confirm AE data discrepancies and weaknesses in AE reporting and documentation. Although these findings are not generalizable nor definitive, they are highly suggestive that actual AE rates are higher than reported AE rates, decreasing confidence in the reliability and validity of routine AE identification and reporting. It is unlikely that this phenomenon is unique to the study sites or to ZAZIC. Significant underreporting of AEs is likely in other MC programs at scale. Despite raising concerns, there was no evidence of missed severe AEs resulting in permanent impairment or morbidity. We discuss several lessons learned and next steps to help ensure continuous quality improvement for both data quality and patient safety.

Primarily, AE underreporting appears attributable to gaps in AE documentation. Although ZAZIC data quality audits show evidence of improvements in correct and complete data collection in static sites [23], there are several likely pathways that reduce complete AE documentation. First, AEs may be treated correctly, but not reported through routine program channels. AEs treated at hospital outpatient departments or within networks of private healthcare providers are likely not reported to the MC teams. Second, although there is a standard MoHCC AE reporting form, this is not well decentralized or consistently used. Instead, AE information is reported on, and aggregated from, multiple, duplicative MoHCC and ZAZIC data collection tools that create a burden for providers. Lastly, there are weaknesses in data flow. There is only ad hoc flow of MC data from RHCs to static sites. Implementation of a single, MoHCC/ZAZIC AE reporting form for use at all points of client interaction and one channel of AE reporting could greatly reduce bottlenecks and delays.

Second, clients may not attend reviews, leading to missed AE identification. The indicator of “at least 1 follow-up visit within 14 days of MC” is reported to calculate the follow-up rate on the MRF. During site attachments, GS reported that clients’ spontaneous follow-up appeared lower than the reported 95%, echoing previous studies finding challenges in adherence to post-operative visits [14, 33, 34]. On the client side, in all MC settings, men were counselled on the healing process and where to seek care if needed. Therefore, men healing without complication may self-assess and not come for follow ups. On the healthcare worker side, shortages in human resources compound the long distances between clients and care settings especially in outreach settings. In this study, observed routine follow-up visits would largely not have occurred, or not occurred on time, in the absence of vehicle support. As a result, RHC nurses likely review only those who proactively return for follow-up at the clinic, prioritizing vehicle use for those identified by family or friends with suspected AEs. Also, it is often not feasible for MC teams, themselves, to return. Therefore, in ZAZIC’s integrated program, MC teams work jointly with RHC teams to provide follow-up coverage and care. It is possible that outreach clinicians pre-fill registers with expected follow-up visit dates, counting those aspirational dates as actual visits. In outreach settings, MC clinicians may also tell patients to seek follow-up at their local RHCs, and assume that they did, without verification. However, it is clear that all ZAZIC sites require closer supervision and more frequent QA activities to verify that all MC clients in their catchment area receive timely follow-up or document missed visits appropriately.

Lastly, ZAZIC needs to improve AE monitoring among adolescents. In our tandem reviews, 80% of observed AEs were among boys ages 14 and under. Among ZAZIC clients, overall, younger boys ages 10–14 represent the majority of ZAZIC program MCs. Although our previous research showed that these young clients are not more likely to have an AE, they are 3 times more likely to have infections as compared to their older peers [13]. Sites should distribute wound care instructions for clients and guardians, yet, most observed sites did not have wound care materials, potentially leaving parents unaware of proper wound care or the importance of routine post-operative reviews. Some guardians may also prefer to manage AEs at home. However, adolescent clients were observed with mild, moderate or severe AEs due to poor hygiene, including being unbathed, wearing dirty underwear, or using dirty cloth or string to elevate the penis. It appears difficult for younger clients to maintain clean wounds [35, 36]. Widespread misperceptions of how to speed healing through traditional herbal remedies or use of hypertonic saline also encourage infection [37, 38]. In response, ZAZIC should conduct additional community and school dialogues with parents and youth before MC campaigns and complement these activities with revised wound care instructions tailored for those with lower literacy [39].

Several reassuring practices emerged from this activity. First, the majority of clients with scheduled follow-up visits were located and MC verification completed as part of the review. Second, despite documentation weaknesses, AEs were properly identified and managed at all care settings, both static and outreach, suggesting that providers are providing quality care for MC clients. Also, few discrepancies in AE identification or severity grading were found during tandem reviews, demonstrating that MoHCC MC clinicians are well skilled. Lastly, all MoHCC clinicians identified, completed, and reported AEs according to MOHCC standard reporting guidelines when observed, confirming correct knowledge of AE protocols and policies.

This study had several limitations. First, it required additional financial, transport, and human resources over routine program monitoring. Second, only 6 of 21 sites were assessed on a limited number of days and all service providers were not observed. Therefore, findings may have differed if other clients, clinicians, or calendar days, were included. The GS vehicle enabled active follow up that otherwise may not have occurred; therefore, it is possible that findings may have differed from observations of clients who were able to attend visits without vehicle support. Moreover, post-operative follow-up visits were reported at approximately 95% across all sites; however, sub-optimal client attendance at routine post-operative visits was observed and noted in GS discussions with site teams. This suggests discrepancies between reported and actual follow-up visits. GS did not record the number of times they failed to locate a client; therefore, we cannot report the percent of clients reviewed as compared to those scheduled. In subsequent investigations, GS should conduct a more formal assessment of actual versus reported adherence to routine follow-up visits. Additionally, as GS did not observe 100 consecutive reviews, it is possible that more skilled clinicians were assigned to the GS on subsequent visits or that the observed clinicians improved as a result of the tandem review process. However, this would not alter the overall number of AEs identified, reducing the magnitude of this potential bias on results. Moreover, the observed AE rate from tandem reviews at each site was applied as a constant value to previous MCs in those same sites over a large time period to estimate potential missed AEs (Table 3, column (d)). Many changes likely took place between 2014 and 2017 that could have influenced AE rates. However, as both the MC programs and clinicians likely improved over time, resulting in lower AE rates by the 2017 observation period, we feel confident that there were likely a large number of unidentified or unreported AEs over those previous years. Lastly, AE definitions may vary over time, setting, or between countries, limiting quantitative comparison of AE rates between programs or over time. However, as there are few prospective studies of active surveillance for AEs in programs running at scale, these findings are highly relevant for others implementing MC programs in the region.

Conclusions

Reported AE rates collected through passive surveillance appear low. In contrast, the active surveillance employed through this QA study found that observed AE rates are considerably higher while record review found that AE data quality is sub-optimal. It is unlikely that this phenomenon is unique. Although the results may not be definitive nor generalizable, they require follow-up action. ZAZIC promotes quality assurance and patient safety as critical components of its MC implementation. Therefore, to meet the study objectives of strengthening AE documentation and increase confidence in AE data quality, there were several quality assurance activities implemented to address the study findings.

First, in early 2018, ZAZIC launched a Quality Improvement Task Force to conduct continuous quality improvement activities including spot audits and data quality reviews focused on AEs. Second, ZAZIC and the MoHCC reinforced clear expectations with regard to recognition and reporting of AEs by developing a single, standardized AE reporting tool for implementation at static, outreach, and outpatient care settings where clients may seek care. In combination with improved decentralization of, and training on, the MoHCC AE reporting form, this tool aims to help ensure that AE reports are integrated consistently into DHIS2. Third, ZAZIC and MoHCC continue to reassure service providers that reporting AEs is a sign of quality programing and does not result in punishment. Fourth, in rural areas, ZAZIC conducts intensive training in post-operative MC reviews, AE surveillance, and AE reporting for RHC nurses to further strengthen AE management, data collection and reporting. Fifth, as additional nurses and vehicles, especially in rural areas, are needed to ensure patient follow-up adherence and maintain program quality while expanding coverage, ZAZIC continuously trains healthcare workers in AE recognition and further expanded its vehicle fleet to support MoHCC operations. Lastly, enhanced risk reduction strategies are still needed for younger clients within home and school settings. In response, ZAZIC revised wound care instructions for clients and care givers, distributing them widely to sites beginning in fall, 2018. Overall, continuous monitoring of, and reporting on, client safety would strengthen MC program quality and facilitate the establishment of sustainable, safe MC programs at scale.

Supporting information

Acknowledgments

Disclaimer: The findings and conclusions in this article are those of the author(s) and do not necessarily represent the official position of the funding agencies.

The authors wish to thank the Zimbabwe Ministry of Health and Child Care and the MC teams from its 10 districts for their dedication to the MC program.