http://orcid.org/0000-0002-1057-9867
Figures
Abstract
Various patient safety interventions have been implemented since the late 1990s, but their evaluation has been lacking. To obtain basic information for prioritizing patient safety interventions, this study aimed to extract high-priority interventions in Japan and to identify the factors that influence the setting of priority. Six perspectives (contribution, dissemination, impact, cost, urgency, and priority) on 42 patient safety interventions classified into 3 levels (system, organizational, and clinical) were evaluated by Japanese experts using the Delphi technique. We examined the relationships of the levels and the perspectives on interventions with the transition of the consensus state in rounds 1 and 3. After extracting the high-priority interventions, a chi-squared test was used to examine the relationship of the levels and the impact/cost ratio with high priority. Regression models were used to examine the influence of each perspective on priority. There was a significant relationship between the level of interventions and the transition of the consensus state (p = 0.033). System-level interventions had a low probability of achieving consensus. “Human resources interventions,” “professional education and training,” “medication management/reconciliation protocols,” “pay-for performance (P4P) schemes and financing for safety,” “digital technology solutions to improve safety,” and “hand hygiene initiatives” were extracted as high-priority interventions. The level and the impact/cost ratio of interventions had no significant relationships with high priority. In the regression model, dissemination and impact had an influence on priority (β = -0.628 and 0.941, respectively; adjusted R-squared = 0.646). The influence of impact and dissemination on the priority of interventions suggests that it is important to examine the dissemination degree and impact of interventions in each country for prioritizing interventions.
Citation: Hatakeyama Y, Fujita S, Iida S, Nagai Y, Shimamori Y, Ayuzawa J, et al. (2020) Prioritization of patient safety health policies: Delphi survey using patient safety experts in Japan. PLoS ONE 15(9): e0239179. https://doi.org/10.1371/journal.pone.0239179
Editor: Qiang Zeng, South China University of Technology, CHINA
Received: October 3, 2019; Accepted: September 1, 2020; Published: September 17, 2020
Copyright: © 2020 Hatakeyama et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information file.
Funding: This research was supported by the Health and Labour Sciences Research Grants (grant number: H29-Iryo-Ippan-004, 19IA1002) of the Japanese Ministry of Health, Labour and Welfare. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Patient safety has been a global concern against the background of the occurrence of serious adverse events since the late 1990s [1]. The World Health Organization defined patient safety as “the reduction of risk of unnecessary harm associated with health care to an acceptable minimum” [2]. To improve patient safety, various activities have been introduced by governments, medical/specialty societies, accreditation bodies, and healthcare organizations in many countries, such as incident reporting, training and deployment of safety managers, standardization of care, and changes to payment schemes [3]. Recently, Hasegawa and Fujita reported on the current status of patient safety policies in the Organisation for Economic Co-operation and Development (OECD) member countries [4].
Japan experienced a series of serious patient safety accidents in the late 1990s. In 2002, the General Policy for Medical Safety was published by the Ministry of Health, Labour and Welfare. It stressed the importance of a systematic approach for securing safe workplace environments, and cultivating a patient safety culture. Hospitals were required to have safety standards, an in-hospital incident reporting system, a patient safety committee, and education for hospital staff about patient safety. Hospitals were encouraged to assign patient safety managers, and hospitals with patient safety managers could receive more money as of 2006. A nationwide adverse event reporting system was introduced (2004 for accidents, 2005 for close calls). It is one of the biggest databases of adverse events in the world. In 2009, a no-fault compensation system for cerebral palsy was introduced. Babies with medium to severe cerebral palsy can receive money from this insurance scheme. In 2015, a lethal adverse event investigation system was introduced. In this unique system, the hospital chief executive officer has to report any lethal case possibly due to an adverse event to a third-party organization (the Adverse Event Investigation and Support Center), conduct an in-hospital investigation, and submit the report along with preventive methods to the organization.
With increasing healthcare expenditure, prioritizing patient safety interventions based on evidence has become an important issue. However, previous studies have not found a consensus on whether patient safety interventions improve patient safety [5, 6]. A few individual patient safety projects have been largely successful, but it remains unclear if other efforts are effective in improving patient safety [7]. Given the cost of patient safety interventions, it is important to prioritize them in a limited resource environment; however, the published evidence on their cost and cost-effectiveness is insufficient [8].
The OECD published a report titled “The Economics of Patient Safety: Strengthening a Value-Based Approach to Reducing Patient Harm at National Level” in 2017 [9]. In this report, the OECD showed the results of a panel survey of expert academics and policy makers about prioritizing patient safety interventions in a resource-constrained environment. Based on experts’ ratings on the impact and implementation cost of patient safety interventions, they extracted valued interventions for improving patient safety from 42 interventions at the system, organizational, and clinical levels. Subsequently, they suggested that using a survey and nominal group technique or other approach (e.g., Delphi) for promoting value-based investment in patient safety could identify specific priority areas based on local contexts and expertise, and thus identify the optimal mix of components for a national patient safety strategy.
The OECD used the impact and cost of patient safety as factors to identify the priority of patient safety interventions. However, patient safety interventions to be promoted could be defined by not only the impact and cost but also other factors. It is important to identify factors that should be counted when deciding the priority of patient safety interventions in each country or region. It has not been made clear which factors should be considered in prioritizing patient safety interventions.
Based on patient safety experts’ consensus, this study aimed to extract high-priority patient safety interventions in Japan and to identify the factors influencing the priority of these interventions.
Materials and methods
Delphi process
The Delphi technique was used for obtaining a consensus on patient safety interventions from patient safety experts. This technique is a forecasting method that involves repeatedly asking experts to summarize their opinions [10, 11]. The Delphi technique has been used to solve an array of healthcare problems ranging from those of an individual hospital or department to those of a statewide agency or state [12].
We chose 24 experts including two representatives of nationwide organizations related to patient safety, five hospital administrators, seven patient safety managers at each hospital, eight researchers of patient safety and two other famous in this field. A Delphi survey was conducted over three rounds by mail (round 1) and e-mail (rounds 2 and 3) from July to October 2017. During the rounds, the results of the previous round were presented to the participants.
According to the OECD report [9], the questionnaire consisted of 10 patient safety interventions at the system level, 14 at the organizational level, and 18 at the clinical level (totaling 42 interventions) on a 5-point Likert scale. Table 1 shows the interventions included. We set 6 perspectives for assessing the importance of interventions in the past (contribution), present (dissemination), and future (impact, cost, urgency, and priority). In each round, participants were asked to rate all 42 interventions on a 5-point scale from 6 perspectives: past contribution to patient safety (contribution; 1: Small to 5: Large), present dissemination (dissemination; 1: Low to 5: High), expected effects in reducing harm if implemented (impact; 1: Low to 5: High), cost to implement (cost; 1: Low to 5: High), urgency (urgency; 1: Low to 5: High), and priority for future implementation (priority; 1: Low to 5: High). Ratings on past contributions were asked in round 1 only; the others were asked in all three rounds.
Ethical Approval was obtained from the Ethics Committee of Toho University School of Medicine (No. A17025).
Data analysis
For assessing the variance of the scores rated by experts, we calculated the means (with standard deviations [SD]) and medians (with interquartile range [IQR]) of the rating scores for each intervention. To assess the convergence through the rounds, we examined the relationship between the level (system, organizational, and clinical) of interventions and the transition of the consensus state in rounds 1 and 3. We defined the consensus as being reached if IQR = < 1 following the previous studies using 5-point Likert scale [13, 14]. The transitions of the consensus state were classified into four categories: 1) consistent consensus (consensus was achieved in both round 1 and round 3), 2) change to consensus (consensus was not achieved in round 1, but achieved in round 3), 3) change to dissensus (consensus was achieved in round 1, but not achieved in round 3), and 4) consistent dissensus (consensus was not achieved in round 1 or round 3). We used a chi-squared test for assessing the independence of the relationship between the levels of interventions and the consensus state from round 1 to 3. If relevance was found, we calculated the adjusted residuals of each cell in a cross table as a post hoc analysis. Consequently, the relationships between the five perspectives and the transitions of the consensus state were examined using the same methods.
A high-priority intervention was defined as an intervention with a median above the median of all interventions’ priority. To examine the relationships of the levels and the impact/cost ratio of interventions with the high-priority interventions, we calculated the mean and median scores of 42 interventions in each perspective using the results of the contributions in round 1 and the others in round 3. A chi-squared test for independence between the priority of interventions (high priority or not high priority) and the level of interventions (system, organizational, or clinical) was conducted. Subsequently, after defining high-impact/cost ratio interventions as interventions with a median above the median of all interventions in impact/cost ratio, we performed a chi-squared test for independence between priority (high priority or not high priority) and the impact/cost ratio (high impact/cost ratio or low impact/cost ratio).
Spearman’s correlation coefficients (ρ) were calculated from the mean values of each item in each perspective to assess the relationships within the six perspectives. These correlation coefficients represent the relationship of perspectives with the priority including the influence of the other perspectives. To identify the influences of each perspective on priority, we examined the impact on priority from the other perspectives with regression models adjusted for the influence of each perspective. We set four models. Model 1 was a model for examining the impact and cost of interventions on their priority based on the OECD report. Model 2 included contribution and dissemination as explanatory variables and was performed to assess the influence of past and present patient safety efforts on the priority of interventions. Including contribution, dissemination, and impact, we assessed the influence of previous efforts and the impact of interventions in Model 3. Model 4 was conducted to examine the influence of all perspectives on the priority of interventions with simultaneous forced entry.
All data were analyzed using IBM SPSS Statistics version 25, and a p value of <0.05 was considered statistically significant.
Results
Delphi process
All 24 experts (Table 2) responded in all 3 rounds. The results of rounds 1 and 3 are shown in S1 Table.
Table 3 shows the relationship between the levels of interventions and the transition of the consensus state from round 1 to 3 in the Delphi survey. Each cell shows the numbers and percentages of items (n, %), adjusted residuals (Adj. Res), and p values of adjusted residuals (p). There was a significant relationship between the levels of interventions and the transition of the consensus state (chi-squared = 13.723, degree of freedom [df] = 6, p = 0.033). In the residual analysis, we found a tendency for dissensus in priority of interventions at the system level (adjusted residual = 2.62, p = 0.009) and a tendency for consensus in priority of them at the clinical level (adjusted residual = 2.43, p = 0.015). There were no significant relationships between the six perspectives and the transition of the consensus state (chi-squared = 1.625, df = 4, p = 0.804).
Relationship of the level and the impact/cost ratio of interventions with high-priority interventions
The median score of the 42 interventions on priority was 3.61. Of the 42 interventions, 21 were extracted as high-priority interventions. Table 4 shows the high-priority interventions with their level and the ratio of impact/cost. There were 2 interventions from the system level, 8 from the organizational level, and 11 from the clinical level. No significant relationship between the priority of interventions (high priority or not) and the level of interventions was observed (chi-squared = 4.755, df = 2, p = 0.092).
Of the 21 high-priority interventions, 2 interventions—“human resources interventions” (ratio of impact/cost = 0.88) and “digital technology solutions to improve safety” (ratio of impact/cost = 0.86)—were low cost-effective. The other 19 interventions with high priority were evaluated as high cost-effective: “patient identification and procedure matching protocols” (ratio of impact/cost = 1.54), “procedural/surgical checklists” (ratio of impact/cost = 1.42), “peri-operative medication protocols” (ratio of impact/cost = 1.33), and “hand hygiene initiatives” (ratio of impact/cost = 1.30) were among them. There was no significant relationship between the impact/cost ratio and high priority (chi-squared = 2.381, df = 1, p = 0.123).
Influence of perspectives on priority
Table 5 shows the item scores by the level of intervention based on the results of the contribution in round 1 and the others in round 3. The mean scores of contribution, dissemination, impact, urgency, and priority were higher in the order of clinical, organizational, and system levels. As for cost, the mean scores were in the reverse order of the other perspectives. The standard deviations of each perspective were in the order of system, organizational, and clinical levels in all perspectives.
Table 6 shows the correlations between the six perspectives. There was a strong relationship between contribution and dissemination (ρ = 0.904), but neither of the two perspectives did not have significant correlations with priority. Impact had moderate correlations with priority (ρ = 0.719). There was a considerably strong correlation between urgency and priority (ρ = 0.960). These results demonstrated that experts considered the future importance (impact, urgency, and priority) of interventions to be different from the past and present importance (contribution and dissemination) of them.
Based on the results of the correlation analysis, we excluded urgency from the regression analysis because urgency and priority might be considered the same perspectives in the panels. The results of the regression model with priority as the objective variable and the other four perspectives as explanatory variables are shown in Table 7. In Model 1, which assessed the influence of impact and cost as used in the OECD report,9 impact had an influence on priority (standardized coefficient (β) = 0.710). In Model 2, with contribution and dissemination of previous interventions, there was no significant influence on priority. In Model 3, adding impact to the perspectives used in Model 2, dissemination had a negative correlation (β = -0.504) and impact a positive one (β = -0.893) with priority. In Model 4, using four perspectives, dissemination had a negative correlation (β = -0.628) and impact a positive one (β = 0.941) with priority. Cost had no significant influence on priority in any of these models. The highest adjusted R-squared was observed in Model 4 (0.646), followed by Model 3 (0.639) and Model 1 (0.509). The regression analysis indicated that dissemination as well as impact was a factor should be considered in prioritizing interventions.
Discussion
By using the Delphi technique to summarize the opinions of patient safety experts, this study extracted high-priority patient safety interventions and examined the influence of perspectives on the priority of patient safety interventions.
In the Delphi process, the probability of a transition state of consistent consensus was lower and the probability of a transition of consistent dissensus was higher than expected in system-level interventions. Similarly, there were large SDs and IQRs for scores at the system level in each perspective. These results might mean that the opinions of the experts on the system-level interventions showed diversity and difficulty in achieving a consensus compared to the organizational and clinical levels. In the OECD report, the largest SD of cost and impact was found at the system level: “pay-for performance (P4P) schemes for patient safety” for cost and “system-level public engagement and health literacy initiatives” for impact [9]. It might have been difficult for the experts to have the same images on system-level interventions since some experts were working on the clinical and organizational levels.
The OECD report extracted valued interventions based on the ratio of impact/cost [9]. However, the priority of interventions was not related to the impact/cost ratio in the present study. The ratio of impact/cost could be reworded as cost-effectiveness. Cost-effectiveness analysis, such as quality adjusted life year and the incremental cost-effectiveness ratio, has been introduced for approving new drugs in several countries, such as the United Kingdom, Canada, and Australia [15]. In Japan, cost-effectiveness analysis for the pricing of drugs and medical devices started in the 2017 fiscal year [16]. Japanese experts might not have been accustomed to associating the impact/cost ratio with the priority of intervention, as in this study, since cost-effectiveness analysis in Japan has only just begun and has not been widely disseminated.
In the regression models, it was revealed that impact had a significant positive influence and dissemination a significant negative influence on priority. While the OECD report identified the valued interventions from the impact/cost ratio [9], this study suggested that, adding to the impact of interventions, the degree of dissemination of the interventions was also important in deciding the priority of patient safety interventions. The dissemination of patient safety interventions could vary among countries or regions. Hasegawa and Fujita reported the variety in the adoption of patient safety policies in OECD member counties [4]. The difference in the degree of dissemination of patient safety interventions would be derived from the history and system of the healthcare in each country or region. For example, “clinical incident reporting and management system” had the highest score for contribution (4.08) and the lowest score for priority (3.29, lower than the first quartile of 42 interventions) in this study. As mentioned, since 2002, all hospitals have been required by law to have an in-hospital incident reporting system, and the nationwide incident reporting system began in 2004 in Japan. These already introduced health policies might lessen the priority of a “clinical incident reporting and management system.” The results of this study do not necessarily deny the recommendations of previous studies; rather, it is necessary to examine the dissemination degree and impact of the recommended interventions before applying them in a specified country or region.
When using the Delphi technique, we asked 24 experts, and used IQR as the indicator of consensus. There is no consensus on “an optimal number of subjects in a Delphi study” [17]. In a systematic review on Delphi studies showing the widely varied numbers of participants, 40.0% studies used 11–25 participants in final round [18]. In addition, “Delphi subjects should be highly trained and competent within the specialized area of knowledge related to the target issue” [17]. We selected 24 experts who were representatives of nationwide organizations related to patient safety, hospital administrators, patient safety managers at each hospital, researchers of patient safety and other famous in this field. Diamond et al. conducted a systematic review of 72 Delphi studies, and reported that the definition of consensus showed a wide range of definitions among studies. The most common definition for consensus was percent agreement (percentage of same rating, used in 25 studies), followed by the proportion of ratings within a range (percentage of rating within a certain range, used in 16 studies), the measure of central tendency (median ranking, in 8 studies), and decrease in variance (interquartile range, 6 studies) [18]. Percent agreement and proportion of agreement were likely to be used in dichotomous questions. IQR was used in this study since 5-point Likert rating scale was used for assessing importance of the patient safety interventions.
This study has certain limitations. The results might be influenced from other factors that we did not examined, such as the order of Delphi questionnaire and the characteristics of experts [19]. The priority of the interventions presented in this study was based on the results of a Delphi survey with Japanese experts in patient safety. Therefore, the results should be applied to other countries and regions with caution.
Conclusion
Using the Delphi technique, “human resources interventions,” “professional education and training,” “medication management/reconciliation protocols,” “pay-for performance (P4P) schemes and financing for safety,” “digital technology solutions to improve safety,” and “hand hygiene initiatives” were extracted as high-priority interventions. It was suggested that the experts in patient safety had difficulty assessing interventions at the system level and the cost-effectiveness of the interventions. The ratio of impact/cost, which was used in the OECD report, and level of interventions did not have a significant relationship with priority. Instead, a positive influence of impact and a negative influence of dissemination on priority were found. The influence of impact and dissemination on the priority of patient safety interventions might suggest that it is important to examine the dissemination degree and impact of interventions in each country when prioritizing interventions.
Supporting information
S1 Table. Item scores of each intervention in round 1 and 3.
https://doi.org/10.1371/journal.pone.0239179.s001
(DOCX)
References
- 1.
Institute of Medicine. To Err Is Human: Building a Safer Health System, Washington, D.C.: National Academy Press 1999.
- 2.
World Health Organization (WHO). WHO Patient Safety Curriculum Guide: Multi-Professional Edition. Geneva: World Health Organization 2011.
- 3. McCannon J, Berwick DM. A new frontier in patient safety. JAMA 2011;305:2221–2. pmid:21632485
- 4.
Hasegawa T, Fujita S. Patient Safety Policies: Experiences, Effects and Priorities; Lessons from OECD Member States. Tokyo: Ministry of Health, Labour and Welfare 2018. https://doi.org/10.1016/j.jcjq.2018.03.007 pmid:30008355
- 5. Baines R, Langelaan M, de Bruijne M, Spreeuwenberg P, Wagner C. How effective are patient safety initiatives? A retrospective patient record review study of changes to patient safety over time. BMJ Qual Saf. 2015;24:561–71. pmid:26150548
- 6. Landrigan CP, Parry GJ, Bones CB, Hackbarth AD, Goldmann DA, Sharek PJ. Temporal trends in rates of patient harm resulting from medical care. N Engl J Med. 2010;363:2124–34. pmid:21105794
- 7. Gandhi TK, Berwick DM, Shojania KG. Patient safety at the crossroads. JAMA. 2016;315:1829–30. pmid:27139052
- 8.
Zsifkovits J, Zuba M, Geißler W, Lepuschütz L, Pertl D, Kernstock E, et al. Costs of Unsafe Care and Cost-effectiveness of Patient Safety Programmes. Brussel: European Union 2016.
- 9.
Slawomirski L, Auraaen A, Klazinga N. The Economics of Patient Safety: Strengthening a Value-based Approach to Reducing Patient Harm at the National Level. Paris: OECD 2017.
- 10. Dalkey N, Helmer O. An experimental application of the Delphi method to the use of experts. Management Science 1963;9:458–67.
- 11.
Gordon T, Helmer O. Report on a Long-Range Forecasting Study. California: The RAND Corporation 1964.
- 12. Fink A, Kosecoff J, Chassin M, Brook RH. Consensus methods: characteristics and guidelines for use. Am J Public Health. 1984;74:979–83. pmid:6380323
- 13. Wiangkham T, Duda J, Haque MS, Rushton A. Development of an active behavioural physiotherapy intervention (ABPI) for acute whiplash-associated disorder (WAD) II management: a modified Delphi study. BMJ Open. 2016;6(9):e011764. pmid:27630069
- 14. Price J, Rushton A, Tyros V, Heneghan NR. Consensus on the exercise and dosage variables of an exercise training programme for chronic non-specific neck pain: protocol for an international e-Delphi study. BMJ Open. 2020;10(5):e037656.
- 15. Clement FM, Harris A, Li JJ, Yong K, Lee KM, Manns BJ. Using effectiveness and cost-effectiveness to make drug coverage decisions: a comparison of Britain, Australia, and Canada. JAMA. 2009;302:1437–43. pmid:19809025
- 16. Shiroiwa T, Fukuda T, Ikeda S, Takura T, Moriwaki K. Development of an Official Guideline for the Economic Evaluation of Drugs/Medical Devices in Japan. Value Health. 2017;20:372–8.15 pmid:28292481
- 17. Hsu CC, Sandford BA. The Delphi technique: making sense of consensus. Pract Assess Res Eval. 2007;12:1e7.
- 18. Diamond IR, Grant RC, Feldman BM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol. 2014;67(4):401‐409. pmid:24581294
- 19. Birko S, Dove ES, Özdemir V. Evaluation of Nine Consensus Indices in Delphi Foresight Research and Their Dependency on Delphi Survey Characteristics: A Simulation Study and Debate on Delphi Design and Interpretation. PLoS One. 2015;10(8):e0135162. pmid:26270647