http://orcid.org/0000-0002-0448-6349
Figures
Abstract
Background
Antimicrobial stewardship (AMS) programs promote appropriate use of antimicrobials and reduce antimicrobial resistance. Technological developments have resulted in smartphone applications (apps) facilitating AMS. Yet, their impact is unclear.
Objectives
Systematically review AMS apps and their impact on prescribing by physicians treating in-hospital patients.
Study eligibility criteria
Studies focusing on smartphone or tablet apps and antimicrobial therapy published from January 2008 until February 28th 2019 were included.
Results
Thirteen studies met the eligibility criteria. None was a randomized controlled trial. Methodological study quality was considered low to moderate in all but three qualitative studies. The primary outcomes were process indicators, adherence to guidelines and user experience. Guidelines were more frequently accessed by app (53.0% - 89.6%) than by desktop in three studies. Adherence to guidelines increased (6.5% - 74.0%) significantly for several indications after app implementation in four studies. Most users considered app use easy (77.4%—>90.0%) and useful (71.0%—>90%) in three studies and preferred it over guideline access by web viewer or booklet in two studies. However, some physicians regarded app use adjacent to colleagues or patients unprofessional in three qualitative studies. Susceptibility to several antimicrobials changed significantly post-intervention (from 5% decrease to 10% - 14% increase) in one study.
Conclusions
Use of AMS apps seems to promote access to and knowledge of antimicrobial prescribing policy, and increase adherence to guidelines in hospitals. However, this has been assessed in a limited number of studies and for specific indications. Good quality studies are necessary to properly assess the impact of AMS apps on antimicrobial prescribing. To improve adherence to antimicrobial guidelines, use of AMS apps could be considered.
Citation: Helou RI, Foudraine DE, Catho G, Peyravi Latif A, Verkaik NJ, Verbon A (2020) Use of stewardship smartphone applications by physicians and prescribing of antimicrobials in hospitals: A systematic review. PLoS ONE 15(9): e0239751. https://doi.org/10.1371/journal.pone.0239751
Editor: Dafna Yahav, Rabin Medical Center, Beilinson Hospital, ISRAEL
Received: June 17, 2020; Accepted: September 11, 2020; Published: September 29, 2020
Copyright: © 2020 Helou et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: The authors received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Appropriate prescribing of antimicrobials is crucial for individual patients to increase the chance of therapeutic success and to prevent spread of antimicrobial resistance (AMR) on a broader scale. For this reason, governments and healthcare institutions have developed and implemented antimicrobial stewardship (AMS) programs to improve appropriate prescribing [1–3].
Local antimicrobial guidelines help physicians to prescribe appropriate antimicrobial therapy. However, guidelines change and increasing complexity of care requires easily accessible and frequently updated guidelines. Printed booklets and digital documents may not be sufficient for this purpose. In the age of information technology (IT), many processes within the healthcare system have been digitized or automated and IT has become an intrinsic part of modern medicine [4–6]. IT interventions such as electronic health records (EHR), clinical decision support systems (CDSS), and antimicrobial drug approval systems increase guideline adherent prescribing. Such tools assist in a more timely intravenous to oral switch, and decrease overall antimicrobial consumption [4, 7, 8]. Nevertheless, appropriate prescribing of antimicrobials can still be improved [9, 10]. AMS is important for general practice and hospitals, but prevalence of antimicrobial resistant microorganisms is the highest in hospitals, even in countries with overall low resistance rates [11, 12]. Furthermore, reserve antimicrobials are mainly used in hospitals [11, 12].
With the introduction of smartphones, applications (apps) can be accessed without the necessity of a non-mobile desktop and can simultaneously provide a framework to integrate CDSSs. Besides accessibility, apps offer several other advantages such as the most up to date content, short start-up time and administrator privileges to inform users of specific updates [13].
In Europe and North America the number of unique mobile phone subscribers was respectively 85% and 84% at the end of 2017. In the same year over 318.000 mobile health (mHealth) apps were available in app stores [14, 15]. The majority of healthcare workers utilizes mHealth apps (77.2%) on a regular basis in the United States [16]. Although smartphone apps have high potential for becoming a key component of AMS programs, user experience, uptake and effect on prescription of antimicrobials have not been systematically reviewed to the best of our knowledge. The aim of this study was to systematically review antimicrobial stewardship apps and their impact on prescribing by physicians treating in-hospital patients
Methods
This systematic review was performed in accordance with the guidelines of Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement [17]. (S1 Text).
Eligibility criteria
Studies which focused on AMS app use by physicians treating in-hospital patients were assessed for eligibility. Studies focusing on smartphone or tablet apps and antimicrobial therapy published from January 2008 until February 28th 2019 were included. The year 2008 was chosen since the two most popular app stores, App Store (iOS) and the precursor of Google Play (Android) were launched that year [18, 19]. We included randomized controlled trials (RCT), non-RCTs, time series, before-after studies and qualitative studies. Excluded were studies solely considering antimicrobial prophylaxis, only including patients younger than 18 years of age, case reports, conference papers, editorials, letters to editor and reviews or meta-analysis. Language was no exclusion criterion. We excluded studies which only described app use in the general practice and outpatient setting. In these settings, prevalence and severity of infectious diseases, available antimicrobials and routes of administration differ significantly compared to an in-hospital setting.
Search strategy and review design
EMBASE, MEDLINE (Ovid), Cochrane Central, Web of Science and Google Scholar databases were searched for relevant quantitative and qualitative studies published before February 28th 2019. The search strategy was developed together with an information specialist and specified for each database. Search terms included “antimicrobial”, “anti-infective agent”, “prescription”, “application” and “mobile phone” (S2 Text). Search results were imported to Endnote (Clarivate Analytics, Philadelphia, PA, USA). After removal of duplicate studies, two investigators (RH and DF) independently screened all articles on title and abstract. Articles were included for full text review if selected by either investigator. In case of doubt, articles were included for full text review. Both investigators independently assessed full texts for eligibility and extracted data. Disagreements were resolved in discussion with a third investigator (AV).
Study outcome
Primary study outcomes are process indicators such as number of downloads, average monthly use and guidelines assessed, adherence to guidelines and user experience. Secondary study outcomes are drug consumption, susceptibility and costs.
Data analysis
Data was extracted using standardized forms (S3 Text). The quality of included studies was independently assessed by two investigators (RH and DF). Disagreements were resolved by discussion with a third investigator (AV). To assess the five different study designs we used five risk of bias assessment tools [20–24]. Due to large variations in study design and outcome parameters, study outcome could not be pooled and used for meta-analysis.
Results
Study characteristics
Thirteen studies met the eligibility criteria and were evaluated in this systematic review (Fig 1). Primary outcomes were process indicators such as downloads, average app use and time spent per guideline, evaluated in seven studies. Changes in adherence to guidelines and user experience were analysed in four and five studies, respectively. Antimicrobial consumption was evaluated in two studies (Table 1). In ten of the thirteen studies the app was custom built for the study (Table 2).
Study quality
Emphasis on app dissemination and use, impact of app use and user experience resulted in quantitative, qualitative and mixed methods study designs such as: uncontrolled before-after (five), controlled before-after (two), interrupted time series (one), cross-sectional (six) and qualitative studies (three). Quality was evaluated with the corresponding tools [20–24]. Study designs varied greatly due to different metrics studied, e.g. app dissemination and use, impact of app use and user experience. Overall, methodological study quality was considered low to moderate for the before-after, interrupted time series and cross-sectional studies [25–35] and moderate to high for qualitative studies [30, 35, 36] (S4 Text). In most studies participants and outcome assessors were not blinded. Furthermore, outcomes were usually measured at one time point. Finally, the amount of eligible physicians enrolled was generally unclear as well as the loss to follow-up because information on user retention was lacking.
Process indicators
Seven observational studies [26, 28, 30, 32–35] reported analytics of app use (Table 3). Five studies [26, 30, 32–34] evaluated total number of downloads. All registered an increase of downloads during their study periods (3–14 months). A study that assessed an app containing all hospital antimicrobial guidelines recorded an increase in average monthly app accessions during a 29-month period [32]. In contrast, the monthly app accessions decreased over 3 months in the study of Yoon et al. [34] which assessed an app containing only two guidelines, the treatment of community-acquired pneumonia and urinary tract infection. Additionally, clinicians accessed the guidelines more frequently by app than by desktop in all three studies evaluating number of accessions [26, 28, 30]. One study reported a decrease in time spent per individual guideline over the course of the study possibly demonstrating familiarization with the app [32]. The most frequently accessed guidelines were those outlining treatment for respiratory, skin & soft tissue and genitourinary infections [26, 28, 32, 34].
Adherence to guidelines
Four studies analysed whether empirical prescribing of antimicrobials was according to guidelines, such as choice of drug, dose, interval and route of administration [25, 31, 34, 37]. (Table 4) The study of Charani et al. in which all antimicrobial guidelines were implemented in the app reported a significant increase in adherence to guidelines in surgical wards and a non-significant increase in general medicine wards [37]. This increase persisted after six and twelve months. Two studies [25, 31] reported an significant increase in adherence to guidelines for pyelonephritis and uncomplicated diarrheal diseases respectively during the intervention period. One study [34] showed increased adherence to the community-acquired pneumonia guideline in one hospital, but not in the other. Also, no change in adherence to the UTI guideline was shown in any of the three participating hospitals. Documentation of stop/review dates and indication for starting antimicrobials in the medical charts was evaluated in one study [37]. No change in documentation of stop/review dates was reported during or after the intervention period. Remarkably, documentation on the reason to start antimicrobials decreased significantly during intervention and this sustained during follow up measurements over the next two years [37].
User experience
In five studies user experience was analysed by means of interviews, focus groups or surveys [27, 30, 32, 35, 36]. The app was considered easy to use by 77.4% [35], 88.9% [27] and >90.0% [32] and useful by 71.0% [35], 85.2% [27] and >90.0% [32] of the users in before-after surveys with 31, 27 and 112 respondents, respectively. In one survey, 59 respondents reported app use increased their knowledge base regarding antimicrobial prescribing, while 81% reported app use helped them adhere to the guidelines [30]. Another survey reported 68% of the 31 respondents found app use time saving [35] Interviewees [36] as well as >90% of 112 survey respondents [32] favoured the app guidelines over the web viewer or paper guidelines. Discomfort using the app in front of patients or colleagues due to a sense of unprofessionalism was mentioned by 20.0% of 59 survey respondents [30] and 35.7% of 14 interviewees [36] but this was not experienced by others [32].
Frequent app use was inversely associated (survey respondents (SR) 106; risk ratio (RR) 0.03; confidence interval (CI) 0.0018–0.5; p = 0.0002) with preferring senior physician advice over antimicrobial guidelines, while frequent app use encouraged users to discuss incorrect prescribing by colleagues (SR 92; RR 3.8; CI 1.5–9.7; p = 0.005) [32]. Furthermore, app use was associated with a 1.1 point (p = 0.04) higher change in knowledge score of antimicrobial prescribing in 62 medical students and junior physicians compared to the control group [27] and improved awareness of antimicrobial stewardship (SR 91; RR 6.8; CI 2.1–21.7; p = 0.001) [32].
Drug consumption, susceptibility and costs
Monthly average antimicrobial drug consumption was the focus of one study conducted in Brazil [33]. The app used in this study contained guidelines advising against use of some antimicrobials (e.g. carbapenems) while encouraging use of others (i.e. aminoglycosides) based on cost and susceptibility profile. After app introduction, the use of aminoglycosides and cefepime increased significantly while the use of piperacillin/tazobactam and meropenem decreased significantly. However, it should be noted that during the study period piperacillin/tazobactam was replaced by cefepime within the guideline for hospital-acquired infections. Furthermore, a significant increase in susceptibility to meropenem (73%–83%, p < 0.05) and polymyxin (69%–83%, p < 0.05) and a significant decrease in susceptibility to cefepime was described post-intervention (62%–57%, p < 0.05) [33]. In the year of implementation, a significant reduction of $296,485 USD (p<0.05) in antimicrobial drug costs was attained compared to the pre-implementation year. In a different study the optimal approach to promote safe use of beta-lactam antibiotics in inpatients with a history of penicillin allergy was evaluated [29]. Penicillin and cephalosporin use increased in the intervention period after introduction of a decision support app containing a decision tree for beta-lactam antibiotics (50% of 199 patients) compared to the standard of care period (38% of 148 patients). In the app intervention period odds of treatment with penicillin and cephalosporin were significantly increased (aOR 1.8% [95% CI 1.1, 2.9]).
Discussion
In this systematic review, 13 studies which assessed antimicrobial stewardship smartphone apps in the hospital setting were analysed. Several studies measured different outcomes, applied different designs and varied in quality. In the reviewed studies, AMS apps were increasingly used or downloaded after implementation in five studies, guideline adherent prescribing of antimicrobials increased overall significantly in four studies and in one study this resulted in significantly less resistance to some antimicrobials and to a significant decrease in total drug costs. In general, users favoured the app based guidelines over web or paper versions in two studies, but some reported that app use in front of patients or colleagues felt unprofessional in three studies. Overall, although of varying quality, the studies indicate that AMS apps might increase guideline accessibility and offer physicians a friendly and efficient way of using antimicrobial guidelines.
Content of all but one app in the reviewed studies focused solely on therapy to improve the prescribing of antimicrobials according to the guidelines. To evaluate appropriate prescribing of antimicrobials, different outcome parameters were selected: choice of antimicrobials, dose, dosing interval and route of administration. One study also included indication and stop/review date documentation [37]. This variation in outcome parameters reflects the difficulty in defining appropriate antimicrobial prescribing. Since Gyssens et al. first described quality indicators (QI) for appropriate antimicrobial prescribing in 1992, QI’s have been added and debated in the infectious diseases community without reaching consensus, although many different quality indicators have been proposed [38–40]. In the studies reviewed, a limited set of outcome parameters was evaluated, leaving out many insightful quality indicators of appropriate antimicrobial prescribing, such as switch from intravenous to oral therapy and timely initiation of antimicrobial therapy [39]. Clearly defining quality indicators is essential in order to prevent interpretive bias and to compare studies promoting prescribing interventions.
A factor that was not always taken into account but should be considered for each AMS app is how users experience them. Several studies show that physicians with different backgrounds are enthusiastic about apps, find them user-friendly and helpful in their work and would recommend them to colleagues [41, 42]. In the studies reviewed, some physicians regarded smartphone use unprofessional in front of patients. However, a study focussing on outpatients found that more than half of patients were “fine” when physicians used their smartphone to access information during consultations, and thirteen percent reported it was “not fine” [43]. Initially used for calling and messaging, the increasing number of features and applications helped evolve the mobile phone into a possible valuable multifunctional tool for personal and professional use.
AMS apps could help to educate students, but also junior and senior physicians, in antimicrobial use. Although for students the education on antimicrobials and AMS varies within and between countries, globally, the knowledge of appropriate antimicrobial prescribing of final-year medical students is limited [44–47]. Additionally, prescribing errors are prevalent among junior physicians who are usually responsible for the majority of drug prescriptions in hospitals [48, 49]. The reviewed studies showed that AMS apps have additional educational value by improving knowledge of medical students and junior physicians on the prescribing of antimicrobials. Furthermore, as some of the younger smartphone using doctors have become attending physicians, smartphones will be an increasingly used tool for the prescription of antimicrobials.
The effect of clinical decision support systems (CDSS) on antimicrobial prescribing in the healthcare setting such as hospitals have been evaluated in many studies including several randomized controlled trials and systematic reviews [50–52]. Many are designed to improve guideline adherent prescribing of antimicrobials and are integrated into the EHR. Alternatively, almost all studied apps in our systematic review are standalone facilitating easy implementation in hospitals, are low cost and pose no risk in regard to losing patient data. However, a standalone system lacks patient specific data such as allergies, lab results, microbiological test results and previous treatment with antimicrobials which are mandatory to assess before antimicrobials can appropriately be prescribed [52]. Overall, CDSS have a positive effect on adherence to guidelines for antimicrobial treatment and decreased antimicrobial consumption. Yet, similar to our findings, these outcomes were not unanimous [50, 51]. Furthermore, the studies we reviewed lacked important clinical outcomes reported in the studies evaluating CDSS such as mortality, length of stay and time to therapy. As for AMS apps user experience is an important stepping stone for successful CDSS implementation. In spite of many studies on CDSS and its impact on antimicrobial prescribing the need for good quality studies remains for this IT-intervention too [50–52].
Strengths & limitations
This systematic review has some strengths and limitations. One of the strengths is the clear start date of studies on this subject which coincides with the date app stores launched. A limitation is the exclusion of studies focusing on general practice or outpatient care. Therefore, we cannot draw conclusions on the advantage of AMS app use in these settings. Since the overall methodological study quality varied considerably and comparison between studies was limited due to large variations in study design and outcome parameters, only cautious conclusions could be drawn. Currently, the impact of a smartphone app on antimicrobial prescribing by physicians in hospitals is being evaluated in an international randomized trial [53].
Conclusions
In this systematic review, the crossroad of healthcare and smartphone technology was explored. Smartphones may be used to improve knowledge of antimicrobial stewardship, to access antimicrobial guidelines and thereby improve important aspects of healthcare. During implementation of AMS apps, physician opinions and app uptake should be considered to optimize its impact. The small number of studies on AMS apps illustrate the novelty of this research area. Additionally, the quality of the data was limited. High quality, randomized, multi-centre studies including robust clearly defined clinical, microbiological and process outcomes are needed to evaluate the impact of AMS apps on antimicrobial prescribing and its role within healthcare.
References
- 1. World Health Organization. WHO Global Strategy for Containment of Antimicrobial Resistance. 2001.
- 2. Davey P, Marwick CA, Scott CL, Charani E, McNeil K, Brown E, et al. Interventions to improve antibiotic prescribing practices for hospital inpatients. Cochrane Database Syst Rev. 2017;2:CD003543. pmid:28178770
- 3. Schuts EC, Hulscher M, Mouton JW, Verduin CM, Stuart J, Overdiek H, et al. Current evidence on hospital antimicrobial stewardship objectives: a systematic review and meta-analysis. Lancet Infect Dis. 2016;16(7):847–56. pmid:26947617
- 4. Baysari MT, Lehnbom EC, Li L, Hargreaves A, Day RO, Westbrook JI. The effectiveness of information technology to improve antimicrobial prescribing in hospitals: A systematic review and meta-analysis. Int J Med Inform. 2016;92:15–34. pmid:27318068
- 5. Bremmer DN, Trienski TL, Walsh TL, Moffa MA. Role of Technology in Antimicrobial Stewardship. Med Clin North Am. 2018;102(5):955–63. pmid:30126584
- 6. Cresswell K, Mozaffar H, Shah S, Sheikh A. Approaches to promoting the appropriate use of antibiotics through hospital electronic prescribing systems: a scoping review. Int J Pharm Pract. 2017;25(1):5–17. pmid:27198585
- 7. Filice G, Drekonja D, Greer N, Butler M, Wagner B, MacDonald R, et al. Antimicrobial Stewardship Programs in Inpatient Settings: A Systematic Review. 2013.
- 8. Akhloufi H, Hulscher M, van der Hoeven CP, Prins JM, van der Sijs H, Melles DC, et al. A clinical decision support system algorithm for intravenous to oral antibiotic switch therapy: validity, clinical relevance and usefulness in a three-step evaluation study. J Antimicrob Chemother. 2018;73(8):2201–6. pmid:29718336
- 9. Akhloufi H, Streefkerk RH, Melles DC, de Steenwinkel JE, Schurink CA, Verkooijen RP, et al. Point prevalence of appropriate antimicrobial therapy in a Dutch university hospital. Eur J Clin Microbiol Infect Dis. 2015;34(8):1631–7. pmid:26017664
- 10. Denny KJ, Gartside JG, Alcorn K, Cross JW, Maloney S, Keijzers G. Appropriateness of antibiotic prescribing in the Emergency Department. J Antimicrob Chemother. 2019;74(2):515–20. pmid:30445465
- 11.
National Institute for Public Health and the Environment MoH, Welfare and Sport,. NethMap 2019. 2019.
- 12. ECDC. Surveillance of antimicrobial consumption in Europe 2013–2014. 2018.
- 13. Oehler RL, Smith K, Toney JF. Infectious diseases resources for the iPhone. Clin Infect Dis. 2010;50(9):1268–74. pmid:20233061
- 14. GSMA. The Mobile Economy 2018 2018 [Available from: https://www.gsma.com/mobileeconomy/wp-content/uploads/2018/05/The-Mobile-Economy-2018.pdf.
- 15. IQVIA Institute. The Growing Value of Digital Health 2017 [Available from: https://www.iqvia.com/insights/the-iqvia-institute/reports/the-growing-value-of-digital-health.
- 16. Markley JD, Pakyz A, Bernard S, Lee K, Appelbaum N, Bearman G, et al. A survey to optimize the design of an antimicrobial stewardship smartphone app at an academic medical center. Am J Infect Control. 2017;45(3):317–20. pmid:27838166
- 17. Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535. pmid:19622551
- 18.
Apple. iPhone App Store Downloads Top 10 Million in First Weekend 2008 [Available from: https://www.apple.com/newsroom/2008/07/14iPhone-App-Store-Downloads-Top-10-Million-in-First-Weekend/.
- 19.
PCWorld. Google Launches Android Market 2008 [Available from: https://www.pcworld.com/article/152613/google_android_ships.html.
- 20. Cochrane consumers & communication review group. The Cochrane Collaboration’s tool for assessing risk of bias; adapted using EPOC’s criteria for studies other than RCTs. Guide for review authors on assessing study quality2013.
- 21. Cochrane consumers & communication review group. The Cochrane Collaboration’s tool for assessing risk of bias; adapted using EPOC’s criteria for ITS studies and CCRG input. Guide for review authors on assessing study quality2013.
- 22.
Critical Appraisal Skills Programme. CASP Qualitative checklist 2018 [Available from: https://casp-uk.net/wp-content/uploads/2018/01/CASP-Qualitative-Checklist-2018.pdf.
- 23.
National Heart LaBI. Quality Assessment Tool for Before-After (Pre-Post) Studies With No Control Group 2013 [Available from: https://www.nhlbi.nih.gov/sites/default/files/media/docs/risk-assessment.pdf.
- 24.
National Heart LaBI. Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies 2013 [Available from: https://www.nhlbi.nih.gov/sites/default/files/media/docs/risk-assessment.pdf.
- 25. Antonello VS, Cunegatto MC, Rosin ET, Castilhos JX, Beduschi FG, Bumaguin DB, et al. O uso de ferramentas educacionais móveis para o aperfeiçoamento da prescrição médica do tratamento de pielonefrite aguda na gestação: estudo transversal retrospectivo. Rev Bras Ginecol Obstet. 2019;41(2):97–101. pmid:30786306
- 26. Hoff BM, Ford DC, Ince D, Ernst EJ, Livorsi DJ, Heintz BH, et al. Implementation of a Mobile Clinical Decision Support Application to Augment Local Antimicrobial Stewardship. J Pathol Inform. 2018;9:10. pmid:29692947
- 27. Fralick M, Haj R, Hirpara D, Wong K, Muller M, Matukas L, et al. Can a smartphone app improve medical trainees' knowledge of antibiotics? Int J Med Educ. 2017;8:416–20. pmid:29200402
- 28. Young HL, Shihadeh KC, Skinner AA, Knepper BC, Sankoff J, Voros J, et al. Implementation of an institution-specific antimicrobial stewardship smartphone application. Infect Control Hosp Epidemiol. 2018;39(8):986–8. pmid:29925458
- 29. Blumenthal KG, Wickner PG, Hurwitz S, Pricco N, Nee AE, Laskowski K, et al. Improving antibiotic choice in hospitalized medical patients reporting penicillin allergy. J Allergy Clin Immunol. 2017;139(2):AB181.
- 30. Charani E, Kyratsis Y, Lawson W, Wickens H, Brannigan ET, Moore LSP, et al. An analysis of the development and implementation of a smartphone application for the delivery of antimicrobial prescribing policy: Lessons learnt. J Antimicrob Chemother. 2013;68(4):960–7. pmid:23258314
- 31. Haque F, Ball RL, Khatun S, Ahmed M, Kache S, Chisti MJ, et al. Evaluation of a Smartphone Decision-Support Tool for Diarrheal Disease Management in a Resource-Limited Setting. PLoS Negl Trop Dis. 2017;11(1).
- 32. Panesar P, Jones A, Aldous A, Kranzer K, Halpin E, Fifer H, et al. Attitudes and behaviours to antimicrobial prescribing following introduction of a smartphone App. PLoS ONE. 2016;11(4).
- 33. Tuon FF, Gasparetto J, Wollmann LC, Moraes TPD. Mobile health application to assist doctors in antibiotic prescription–an approach for antibiotic stewardship. Braz J Infect Dis. 2017;21(6):660–4. pmid:28941393
- 34. Yoon CH, Ritchie SR, Duffy EJ, Thomas MG, McBride S, Read K, et al. Impact of a smartphone app on prescriber adherence to antibiotic guidelines in adult patients with community acquired pneumonia or urinary tract infections. PLoS ONE. 2019;14(1).
- 35. Payne KF, Weeks L, Dunning P. A mixed methods pilot study to investigate the impact of a hospital-specific iPhone application (iTreat) within a British junior doctor cohort. Health Informatics J. 2014;20(1):59–73. pmid:24550565
- 36. Shenouda JEA, Davies BS, Haq I. The role of the smartphone in the transition from medical student to foundation trainee: a qualitative interview and focus group study. BMC Med Educ. 2018;18(1):175. pmid:30064424
- 37. Charani E, Gharbi M, Moore LSP, Castro-Sanchéz E, Lawson W, Gilchrist M, et al. Effect of adding a mobile health intervention to a multimodal antimicrobial stewardship programme across three teaching hospitals: An interrupted time series study. J Antimicrob Chemother. 2017;72(6):1825–31. pmid:28333297
- 38. Gyssens IC, van den Broek PJ, Kullberg BJ, Hekster Y, van der Meer JW. Optimizing antimicrobial therapy. A method for antimicrobial drug use evaluation. J Antimicrob Chemother. 1992;30(5):724–7. pmid:1493990
- 39. Kallen MC, Prins JM. A Systematic Review of Quality Indicators for Appropriate Antibiotic Use in Hospitalized Adult Patients. Infect Dis Rep. 2017;9(1):6821. pmid:28458795
- 40. Le Marechal M, Tebano G, Monnier AA, Adriaenssens N, Gyssens IC, Huttner B, et al. Quality indicators assessing antibiotic use in the outpatient setting: a systematic review followed by an international multidisciplinary consensus procedure. J Antimicrob Chemother. 2018;73(suppl_6):vi40–vi9. pmid:29878218
- 41. Schneider S, Salm F, Vincze S, Moeser A, Petruschke I, Schmucker K, et al. Perceptions and attitudes regarding antibiotic resistance in Germany: a cross-sectoral survey amongst physicians, veterinarians, farmers and the general public. J Antimicrob Chemother. 2018;73(7):1984–8. pmid:29590400
- 42. Paradis M, Stiell I, Atkinson KM, Guerinet J, Sequeira Y, Salter L, et al. Acceptability of a Mobile Clinical Decision Tool Among Emergency Department Clinicians: Development and Evaluation of The Ottawa Rules App. JMIR Mhealth Uhealth. 2018;6(6):e10263. pmid:29891469
- 43. Micallef C, McLeod M, Castro-Sanchez E, Gharbi M, Charani E, Moore LS, et al. An Evidence-Based Antimicrobial Stewardship Smartphone App for Hospital Outpatients: Survey-based Needs Assessment Among Patients. JMIR Mhealth Uhealth. 2016;4(3):e83. pmid:27383743
- 44. Abbo LM, Cosgrove SE, Pottinger PS, Pereyra M, Sinkowitz-Cochran R, Srinivasan A, et al. Medical students' perceptions and knowledge about antimicrobial stewardship: how are we educating our future prescribers? Clin Infect Dis. 2013;57(5):631–8. pmid:23728148
- 45. Wasserman S, Potgieter S, Shoul E, Constant D, Stewart A, Mendelson M, et al. South African medical students' perceptions and knowledge about antibiotic resistance and appropriate prescribing: Are we providing adequate training to future prescribers? S Afr Med J. 2017;107(5):405–10. pmid:28492121
- 46. Chuenchom N, Thamlikitkul V, Chaiwarith R, Deoisares R, Rattanaumpawan P. Perception, Attitude, and Knowledge Regarding Antimicrobial Resistance, Appropriate Antimicrobial Use, and Infection Control Among Future Medical Practitioners: A Multicenter Study. Infect Control Hosp Epidemiol. 2016;37(5):603–5. pmid:26809753
- 47. Dyar OJ, Nathwani D, Monnet DL, Gyssens IC, Stalsby Lundborg C, Pulcini C, et al. Do medical students feel prepared to prescribe antibiotics responsibly? Results from a cross-sectional survey in 29 European countries. J Antimicrob Chemother. 2018;73(8):2236–42. pmid:29746647
- 48. Ryan C, Ross S, Davey P, Duncan EM, Francis JJ, Fielding S, et al. Prevalence and causes of prescribing errors: the PRescribing Outcomes for Trainee Doctors Engaged in Clinical Training (PROTECT) study. PLoS One. 2014;9(1):e79802. pmid:24404122
- 49. Ashcroft DM, Lewis PJ, Tully MP, Farragher TM, Taylor D, Wass V, et al. Prevalence, Nature, Severity and Risk Factors for Prescribing Errors in Hospital Inpatients: Prospective Study in 20 UK Hospitals. Drug Saf. 2015;38(9):833–43. pmid:26115701
- 50. Laka M, Milazzo A, Merlin T. Can evidence-based decision support tools transform antibiotic management? A systematic review and meta-analyses. J Antimicrob Chemother. 2020;75(5):1099–111. pmid:31960021
- 51. Rittmann B, Stevens MP. Clinical Decision Support Systems and Their Role in Antibiotic Stewardship: a Systematic Review. Curr Infect Dis Rep. 2019;21(8):29. pmid:31342180
- 52. Rawson TM, Moore LSP, Hernandez B, Charani E, Castro-Sanchez E, Herrero P, et al. A systematic review of clinical decision support systems for antimicrobial management: are we failing to investigate these interventions appropriately? Clin Microbiol Infect. 2017;23(8):524–32. pmid:28268133
- 53. Helou RI, Catho G, Peyravi Latif A, Mouton J, Hulscher M, Teerenstra S, et al. Study protocol for an international, multicentre stepped-wedge cluster randomised trial to evaluate the impact of a digital antimicrobial stewardship smartphone application. BMJ Open. 2020;10(6):e033640. pmid:32503867