http://orcid.org/0000-0001-7231-9510
Figures
Abstract
Background
Vector-borne diseases are a public health problem in Colombia, where dengue virus infection is hyperendemic. The introduction of other arboviruses, such as chikungunya and Zika in the last three years, has aggravated the situation. Mobile health (mHealth) offers new strategies for strengthening health care and surveillance systems promoting the collection, delivery, and access of health information to professionals, researchers, and patients. Assessing mobile application performance has been a challenge in low- and middle-income countries due to the difficulty of implementing these technologies in different clinical settings. In this study, we evaluate the usability and acceptability of a mobile application, FeverDX, as a support tool in the management of patients with febrile syndrome and suspected arboviruses infection by general practitioners from Colombia.
Methods
A pilot implementation study was conducted to evaluate the usability and acceptability of FeverDX using the modified version of the Mobile Application Rating Scale (uMARS). The evaluation form included 25 questions regarding quantity and quality of information, engagement, functionality, aesthetics, impact, and acceptability by healthcare workers. Each item uses a 5-point scale (1-Inadequate, 2-Poor, 3-Acceptable, 4-Good, 5-Excellent). A global score was obtained for the evaluation form test by determining the median scores of each subsection. A descriptive statistical analysis of the data obtained was performed.
Results
Between December 2016 and January 2017, a total of 20 general practitioners from the Emergency room and hospitalization areas evaluated FeverDX. Less than half (9/20) of the evaluators had a comprehensive knowledge of the Colombian Ministry of Health’s guidelines for the diagnosis and management of arboviruses, and evaluators partially (4/9) or completely (5/9) agreed that the content of the application follows the management guidelines. On uMARS scale, FeverDX excelled regarding impact (median 5; IQR = 5–5), functionality (median 5; IQR = 4.8–5), and information and scientific basis (median 4; IQR = 4–4). FeverDX scored well regarding user feedback (median 4; IQR = 4–4.5), design and aesthetics (median 4; IQR = 4–4.3), and subjective assessment of quality (median 4.5; IQR = 4.3–4.8).
Conclusions
FeverDX, a mobile application, is a novel mHealth strategy to strengthen care processes and facilitate the detection and reporting of notifiable surveillance diseases. It could improve adherence to clinical practice guidelines for the management and prevention of prevalent diseases as arboviruses in healthcare settings.
Although this pilot study used a small sample size, FeverDx performed adequately in a simulated emergency consultation. Further implementation studies are needed to increase the reliability of mHealth technologies in different scenarios.
Citation: Rodríguez S, Sanz AM, Llano G, Navarro A, Parra-Lara LG, Krystosik AR, et al. (2020) Acceptability and usability of a mobile application for management and surveillance of vector-borne diseases in Colombia: An implementation study. PLoS ONE 15(5): e0233269. https://doi.org/10.1371/journal.pone.0233269
Editor: Abdallah M. Samy, Faculty of Science, Ain Shams University (ASU), EGYPT
Received: November 29, 2019; Accepted: May 1, 2020; Published: May 29, 2020
Copyright: © 2020 Rodríguez et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: This study was funded by Fundación Valle del Lili and Universidad Icesi. The funders don't play role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
In Latin America and the Caribbean, arboviruses are a major public health problem [1]. Before the arrival and subsequent spread of chikungunya and Zika viruses, Aedes spp. transmitted dengue virus with a record of 2,38 million cases in the Americas in 2016 [2]. During 2014, Colombia experienced the importation and rapid transmission of chikungunya, with approximately 96,687 cases reported. In September 2015, Colombia reported its first case of Zika, reaching a total of 11,712 the same year [3].
The limited availability of diagnostic tests in endemic areas in developing countries, represents a challenge for differential diagnosis, especially in the case of co-infection [4–6] because arboviruses infections share non-specific clinical manifestations [7]. This situation encourages the development and implementation of innovative initiatives in different scenarios, as health services and public health surveillance entities. An example is mobile health (mHealth), referring to the practice of medical and public health supported by mobile devices, such as mobile phones, patient monitoring devices, personal digital assistants, and other wireless devices [8]. For more than a decade, the World Health Organization (WHO) has promoted the development and application of mHealth as an essential resource in the provision of health services [9]. According to a survey conducted by WHO, only 12% (14/115) of the WHO Member States conducted formal evaluations of mHealth-related initiatives [8].
Supply and demand for mobile applications development in the healthcare area have been recently increasing [10]. Although mHealth performance evaluations are growing [11–13], regarding the majority of commercially mobile health applications in our research field, companies and researchers have not yet incorporated an extensive performance evaluation, including usability and app quality. Furthermore, additional tailored and rigorous evaluation studies on the mHealth clinical impact and reliability are still required [14,15].
FeverDx is a mHealth tool designed by a group of physicians for healthcare workers (HCW) in a local context where digital transformation is limited, making this tool tailored to the specific needs of the end-user in terms of time and ease of use, access to national health guidelines content, and is a critical step of the reporting process of mandatory notifiable diseases. Moreover, the mobile app provides rapid access without an internet connection to locally stored clinical data regarding the most frequent fever syndromes clinical guidelines and supports general practitioners (GPs) in the approach and management of patients with a fever syndrome and suspected arboviral infection.
This study evaluated the usability and acceptability of FeverDx mobile application, a mobile application to guide general practitioners (GPs) in the approach and management of patients with a fever syndrome and suspected arbovirus infections. GPs from different clinical areas at Fundación Valle del Lili (FVL) performed FeverDx.
Materials and methods
We conducted a cross-sectional, pilot implementation study of FeverDx in Cali, Colombia.
Setting
The study was conducted at Fundación Valle del Lili (FVL) University Hospital between December 2016 and January 2017. FVL is a non-profit university hospital that works as a referral health center for the southwestern region of Colombia, and it is associated with Universidad Icesi. In 2016, a total of 17,321 cases of dengue, 1,204 of chikungunya, and 14,638 of Zika were reported in the city by the Local Health Authority [16].
The study protocol was approved by the Comité de Ética en Investigación Biomédica from FVL (IRB/EC No. 584–201, followed the ethical principles for medical research outlined by the Declaration of Helsinki [17] and the declaration 8430 of 1993 from Colombia’s Ministry of Health [18].
Mobile application
FeverDx is a mobile application (app) developed by the FVL in association with Universidad Icesi. The app was created following the Methodology for the Development for Mobile Applications (MDAM) [19]. This guideline recommends five phases intending to guide the process of app design and implementation: analysis, design, development, performance testing, and delivery. The minimal requirements for the use of the app are a tablet or smartphone device supported by either Android and iOS operating systems with at least intermittent access to Wi-Fi or mobile network connectivity.
This app runs based on two modules: the clinical guideline and the evaluation modules. The guideline module allows GPs to obtain a rapid-offline reference to the CPGs for acute febrile syndromes such as Zika, dengue, chikungunya, and other common infections in tropical areas (leptospirosis). Each guideline includes general topics concerning clinical features, diagnostic, treatment, and prevention (Fig 1).
A. Main Menu; B. Clinical Guidelines; C. Guideline Display.
The patient evaluation module allows the GP to create a local record (mobile phone database) of the patient data, which can be consulted, modified, or deleted at any time. Fig 2 shows the algorithm integrated into the application to support the approach and management of patients with a febrile syndrome. This algorithm highlights the essential points of patient care, such as the evaluation of general symptoms, vital signs, and the alarm signs, severity signs, which include all dengue, Zika, and chikungunya warning signs.
Patient information obtained from medical history, physical examination, and laboratory tests are recorded using a graphical interface (checklists and alphanumeric fields) (Fig 3).
A. Risk Conditions; B. General Evaluation; C. Clinical Evaluation; D. Physical Exam.
The mobile app shows a variety of messages from an internal feedback “library” which is based on the Colombian Ministry of Health´s clinical practice guidelines (CPG) for the diagnosis and management of arboviruses (dengue, chikungunya, and Zika) and additional common infections in tropical countries [20–23]. An infectious disease specialist reviewed the included recommendations.
Obtained data from registration, evaluation, and reporting is stored on a local server, where it is available to be consulted by national epidemiological surveillance entities.
The FeverDx app is compliant with the Policy of treatment and protection of personal data (apps.co) from the Colombian Ministry of Information and Communications Technologies [24].
Study procedures
The pilot study was conducted in a controlled setting where GPs from the emergency room and hospitalization areas, that had no prior experience with the app, were recruited through an individual invitation. The usability evaluation was carried out in a single session according to the following stages:
- Delivery of materials (an Android or iOS device, medical records, evaluation forms) and instructions about the application use and the evaluation process.
- Simulated emergency consultation (Interaction with the application).
- Individually and confidentially usability evaluation by GPs.
The convenience sampling method was implemented to select a sample of 100 different clinical records of patients who consulted FVL for acute febrile illness from the Institutional Epidemiological Surveillance Committee and Statistics Department Databases. Demographic, clinical, and laboratory data were obtained from the FVL medical charts. Five clinical records were randomly assigned to each of the twenty GPs who used the app to perform a simulated emergency consultation interacting with the patient evaluation module.
During the clinician-application interaction, feedback messages related to patient risk- group, signs of alarm, severity, and recommended management, appear intended to guide the GP in the approach and management of the patient.
The simulated emergency consultation that included the entry of all patient-related data, lasted 6 minutes on average. Afterward, there was no time limit to experience FeverDx freely. The evaluation of usability and acceptability was performed anonymously, individually, and secretly by each GP. The assessment included the perception of usefulness and user experience satisfaction.
Evaluation tool and analysis
Implementation outcomes were extracted into a template using a Microsoft Excel® spreadsheet. We developed the evaluation form based on four different evaluation systems for mHealth applications: 1) mobile health (mHealth) evidence reporting and assessment (mERA) checklist [25]; 2) iSYScore index [26]; 3) Mobile App Rating Scale (MARS) [27]; 4) User Version of the Mobile Application Rating Scale (uMARS) [28]; and 5) additional evaluation methods [29] and implementation outcomes (acceptability [30] and usability [31]).
The evaluation form comprises 25 questions measuring quantity and quality of information (Section A), engagement (Section B), functionality (Section C), aesthetics (Section D), the impact of patient management from the practitioner perspective (Section E) and acceptability by HCW (Section F) (S1 File).
Knowledge evaluation regarding the CPG for vector-borne disease management was included in the first two questions of the evaluation form. Each item uses a 5-point scale (1-Inadequate, 2-Poor, 3-Acceptable, 4-Good, 5-Excellent) (S1 File). A global score was obtained for the evaluation form test by determining the median scores of each subsection.
A descriptive analysis was performed using STATA® statistical software (Stata Corp, 2011, Stata 12 Base Reference Manual, College Station, TX, USA). Continuous data were summarized as median and interquartile ranges, and categorical variables presented as frequencies and proportions.
Results
Between December 2016 and January 2017, a total of twenty GPs tested and evaluated FeverDx mobile application.
CPG knowledge
All data regarding CPG knowledge evaluation was gathered from GPs with no prior use of the FeverDx app. The proportion of GPs with comprehensive knowledge of the Colombian’s CPG for diagnosis and management of arboviruses (dengue, chikungunya, and Zika) was less than half of GPs evaluated (9/20), and less than one-third of them applied them in their practice (6/20). Of those GPs who reported having comprehensive knowledge of CPG, more than half (5/9) wholly agreed and, 4/9 partially agreed that the application information matched with CPG information.
Application evaluation
Table 1 presents an overview of the information quality, engagement, functionality, aesthetics, quality, and impact categories evaluation scores. The median of the global evaluation score was 4.0 (out of 5.0). Impact and functionality obtained the highest score with a median of 5 (IQR = 5–5) and 5 (IQR = 4.8–5), respectively. Among GPs, 19 out of 20 partially/wholly agreed that the application could increase their knowledge and understanding about dengue, chikungunya, and Zika infection management.
Regarding the user feedback, GPs scored a median of 4 (IQR = 4–4.5), and “interest” obtained the high scored of this category 5 (IQR = 4–5). Almost one-third of participants (6/20) reported the mobile application content as suitable for physicians (without content problems) and more than half (13/20), that the application information (content) was correctly oriented with minor issues. Only one of the participants considered the content as acceptable and it was not geared toward physicians.
Concerning impact, the intention to change domain was assessed with a median score of 5 (IQR = 4–5), and the majority (18/20) of GPs partially/wholly agreed that the app will probably change attitudes towards clinical management of dengue, chikungunya, and Zika. Moreover, all the GPs would recommend the app, and most of them (17/20) rated it with 4 out of 5 according to the overall star rating (Fig 4).
Discussion
To our knowledge, this is the first study evaluating the usability and acceptability of a mobile application for the clinical approach to acute febrile syndrome management in Colombia. In general, there is a lack of evidence regarding the assessment and performance of mobile apps in healthcare facilities. For this reason, more initiatives to evaluate mHealth developments are needed in different contexts to know the real impact of these technologies, and to generate evidence about its performance in the field [15].
In this study, we described an alternative approach to evaluate mHealth through validated scales such as uMARS, which had demonstrated a reliable degree of app quality evaluation and the ability to obtain valuable information from users about mobile apps [28]. Our findings showed that FeverDx evaluation obtained a high overall score as well as in most of its subscales. Hence, the use of FeverDx in regions where arboviruses are endemic or hyperendemic potentially could help GPs and stakeholders in the management of acute febrile syndromes such as arboviruses infections. Although this is a pilot study, our findings are a starting point for the development of new implementation studies that allow the evaluation of application performance in different contexts and scenarios.
One of the challenges in low and middle-income countries is the surveillance of vector-borne diseases and their epidemiology. The epidemiological surveillance process includes the notification, collection, and analysis of data, which allows for the generation of timely, valid and reliable information for stakeholders and policymakers, and thus to guide public health policy and create prevention and control strategies [32]. However, this process is often time-consuming. We believe that FeverDx could help in the solution of this problem and facilitate this process through innovative development that is easy to reach and manage from an app that can be used in devices such as smartphones.
Mobile applications designed for epidemiological surveillance of arboviruses are limited in their scope. A study conducted in 2018, which reviewed the available Google Play and App Store applications, showed that 26 applications relevant to surveillance are available. This review included 7 surveillance applications related explicitly to dengue and Zika, aimed at case reporting and outbreaks monitoring through the use of interactive maps. Some of these applications were limited by location, operating system (Android only), and although most of them were developed for global use, most of them were in the English language, which could narrow their applicability in different contexts [33,34]. Additional mHealth developments with a different approach are available such as surveillance (DengueWatch, Dengue map, ZikaTracke, Kidenga App, Break Zika, Predict and Beat dengue) and general information (Dengue–Manejo clínico, Arbo App, Dengue App, Dengue síntomas tratamiento y prevención, Dengue Treatment, WHO Zika App), however evidence regarding implementation in clinical scenarios is scarce [34,35].
There are relevant conditions to be considered for the mobile health applications development and adoption: the local demographic, geographic, connectivity, culture, and socio-sanitary aspects; and the particular needs of the potential end-users, general public, healthcare workers or surveillance authorities [34,36]. Among the commercially available mHealth developments, a small proportion combines different functionalities such as vector-borne disease management (evaluation, classification, and treatment) or rapid offline clinical information review.
In this sense, FeverDx is innovative in the field of arboviruses, it is in the local language, and has been evaluated for usability and acceptability. It offers a solution to local and national surveillance systems, based on these identified needs, to fulfill an early and timely case report. Furthermore, it could potentially support the decision-making process in real-world clinical environments.
Another relevant finding was that a not insignificant proportion (25%) of the GPs had not read the Colombian CPGs for the management of vector-borne diseases. Previous studies in this matter have shown limitations in the adoption of CPGs. For example, a study conducted in Zimbabwe in 2011 that evaluated the attitudes of GPs towards the use of CPGs showed that 57.9% of GPs felt that guidelines would not improve their diagnostic ability, even though that 65.9% were prepared to use guidelines in their practice [37]. Furthermore, qualitative studies conducted to assess GPs' experiences when applying multiple guidelines found that GPs’ responses clustered around two major topics: 1) Complications for the GPs of applying multiple guidelines; and, 2) Complications for their patients when GPs apply multiple guidelines [38]. Thus, further strategies for CPGs implementation and real-world use are needed. For this reason, we propose friendly mobile technologies as a feasible and reliable strategy for promoting the adoption of CPGs.
Finally, mobile technology offers unique opportunities to reach and engage HCW to enhance health literacy and public health authorities for real-time disease surveillance, and that the real-time case reporting feature could improve the current manual registration and report process [33].
FeverDx is a user-centered app intended to drive clinical information to GPs and facilitate epidemiological surveillance. Our findings could promote the adoption and engagement of this app in a clinical setting; thus, improving the care of patients with febrile syndromes in health centers in tropical areas resulting, eventually, in better clinical outcomes.
Limitations
Our study is not without limitations and results should be interpreted in the context of the study design. First, the study included twenty GPs from the emergency department and hospitalization areas of one center highly specialized in the treatment of severely ill patients, which limits the generalizability of our study and the applicability of our results in other environments. Furthermore, the GPs included in the study and the mobile application developers come from the same institution, which may have introduced a courtesy bias. To face this limitation, we assured an anonymous and independent evaluation process for the participants. Second, the fact that we tested the app in a tertiary care center makes our study prone to selection bias. Third, the implementation of FeverDx is currently restricted to Colombia because its design was mainly based on the Colombian Ministry of Health´s mandatory notification sheets and CPG. Application content was strengthened with the international recommendations given by the WHO and with the review by an infectious diseases specialist with experience in arboviral research to mitigate this issue.
Despite our limitations, we consider that the application, in general, has acceptable performance, and our study advances the knowledge of mHealth technology for the diagnosis of febrile syndromes (arboviruses and other common infections) in settings where innovative initiatives are needed to improve patient outcomes. Finally, further research with a larger sample size and with the participation of other healthcare centers need to be conducted to support our initial findings.
Acknowledgments
We thank the team of developers at Universidad Icesi and the editorial group at Fundación Valle del Lili for their support in this project.
References
- 1. Marcondes CB, Contigiani M, Gleiser RM, Reisen W. Emergent and reemergent arboviruses in South America and the Caribbean: Why so many and why now? J. Med. Entomol. 2017;54:509–32. pmid:28399216
- 2. World Health Organization. Dengue and severe dengue. Newsroom 2019.
- 3. Alvis-Guzmán N, Zakzuk-Sierra J, Vargas-Moranth R, Alcocer-Olaciregui A, Parra-Padilla D. Dengue, Chikunguña y Zika en Colombia 2015–2016. Rev. MVZ Córdoba 2017;22:5994.
- 4. Cardoso CW, Paploski IAD, Kikuti M, Rodrigues MS, Silva MMO, Campos GS, et al. Outbreak of Exanthematous Illness associated with Zika, Chikungunya, and Dengue viruses, Salvador, Brazil. Emerg. Infect. Dis. 2015;21:2274–6. pmid:26584464
- 5. Parreira R, Centeno-Lima S, Lopes a., Portugal-Calisto D, Constantino a., Nina J. Dengue virus serotype 4 and Chikungunya virus coinfection in a traveller returning from Luanda, Angola, January 2014. Eurosurveillance 2014;19:6–9. pmid:24650864
- 6. Furuya-Kanamori L, Liang S, Milinovich G, Soares Magalhaes RJ, Clements AC a, Hu W, et al. Co-distribution and co-infection of chikungunya and dengue viruses. BMC Infect. Dis. 2016;16:84. pmid:26936191
- 7. Mattar S, Tique V, Miranda J, Montes E, Garzon D. Undifferentiated tropical febrile illness in Cordoba, Colombia: Not everything is dengue. J. Infect. Public Health 2017:6–11. pmid:28162961
- 8. World Health Organization. mHealth: New horizons for health through mobile technologies. Observatory 2011;3:66–71.
- 9. World Health Organization. 58a Asamblea Mundial de la Salud. Resoluciones y Decisiones. Cibersalud. 58a Asam. Mund. La Salud 2005:128–30.
- 10. Jahns R-G, Houck P. Mobile Health Market Report 2013–2017. Research2Guidance 2013:116.
- 11. Boyce SP, Nyangara F, Kamunyori J. A mixed-methods quasi-experimental evaluation of a mobile health application and quality of care in the integrated community case management program in Malawi. J. Glob. Health 2019;9:010811. pmid:31263554
- 12. Haque F, Ball RL, Khatun S, Ahmed M, Kache S, Chisti MJ, et al. Evaluation of a Smartphone Decision-Support Tool for Diarrheal Disease Management in a Resource-Limited Setting. PLoS Negl. Trop. Dis. 2017;11:1–19. pmid:28103233
- 13. Hamrioui S, Torre-díez I De, López-coronado M, Herreros J. Implementation and Performance Evaluation of a Mobile Health Application for Auto-controlling of Cardiac Diseases. Int. J. Clin. Med. Informatics ISSN 2018;1:13–7.
- 14. Fougerouse P-A, Yasini M, Marchand G, Aalami OO. A Cross-Sectional Study of Prominent US Mobile Health Applications: Evaluating the Current Landscape. AMIA… Annu. Symp. Proceedings. AMIA Symp. 2018;2017:715–23.
- 15. Sim I. Mobile devices and health. N. Engl. J. Med. 2019;381:956–68. pmid:31483966
- 16. Rojas Palacios JH. Boletín epidemiológico No. 10. Cali: 2016.
- 17. World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. JAMA 2013;310:2191–4. pmid:24141714
- 18. Ministerio de Salud. Resolución 8430 (octubre 4): Por la cual se establecen las normas científicas, técnicas y administrativas para la investigación en salud 1993:1–19.
- 19. Gasca Mantilla MC, Camargo Ariza LL, Medina Delgado B. Methodology for mobile application development. Tecnura 2014;18:20–35.
- 20.
Ministerio de Salud y Protección Social. Guía para la atención clinica integral del paciente con dengue. Bogotá D.C.: Ministerio de Salud y Protección Social; 2010.
- 21.
Ministerio de Salud y Protección Social. Lineamientos para el manejo clínico de los pacientes con el virus chikungunya. Bogotá D.C.: 2013.
- 22.
Ministerio de Salud y Protección Social. Lineamientos para la atención clínica integral de pacientes con zika en Colombia. Bogotá D.C.: 2016.
- 23.
Instituto Nacional de Salud (INS). Protocolo de vigilancia en salud pública: leptospirosis. Bogotá D.C.: Ministerio de Salud y Protección Social; 2016.
- 24.
Ministerio de Tecnologías de la Información y las Comunicaciones. Política de tratamiento y protección de datos personales. Apps.Co available at.
- 25. Agarwal S, LeFevre AE, Lee J, L’Engle K, Mehl G, Sinha C, et al. Guidelines for reporting of health interventions using mobile phones: mobile health (mHealth) evidence reporting and assessment (mERA) checklist. BMJ 2016;352:i1174. pmid:26988021
- 26. Grau I, Kostov B, Gallego J a., Grajales F, Fern??ndez-Luque L, Sis??-Almirall A. Método de valoración de aplicaciones móviles de salud en español: el índice iSYScore. Semergen 2016;42:575–83. pmid:26879598
- 27. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile App Rating Scale: A New Tool for Assessing the Quality of Health Mobile Apps. JMIR MHealth UHealth 2015;3:e27. pmid:25760773
- 28. Stoyanov SR, Hides L, Kavanagh DJ, Wilson H. Development and Validation of the User Version of the Mobile Application Rating Scale (uMARS). JMIR MHealth UHealth 2016;4:e72–e72. pmid:27287964
- 29. Grundy QH, Wang Z, Bero L a. Challenges in Assessing Mobile Health App Quality: A Systematic Review of Prevalent and Innovative Methods. Am. J. Prev. Med. 2016;51:1051–9. pmid:27659122
- 30. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda 2011;38:65–76. pmid:20957426
- 31. Madan A, Dubey SK. Usability Evaluation Methods: a Literature Review. Int. J. Eng. Sci. Technol. 2012;4:590–9.
- 32. García Perez C, Aguilar PA. Vigilancia epidemiológica en salud. Arch. Médico Camagüey 2013;17:784–805.
- 33. Mohanty B, Chughtai A, Rabhi F. Use of Mobile Apps for epidemic surveillance and response–availability and gaps. Glob. Biosecurity 2019;1:37–49.
- 34. Early J, Gonzalez C, Gordon-Dseagu V, Robles-Calderon L. Use of Mobile Health (mHealth) Technologies and Interventions Among Community Health Workers Globally: A Scoping Review. Health Promot. Pract. 2019;20:805–17. pmid:31179777
- 35. Agarwal S, Perry HB, Long LA, Labrique AB. Evidence on feasibility and effective use of mHealth strategies by frontline health workers in developing countries: Systematic review. Trop. Med. Int. Heal. 2015;20:1003–14. pmid:25881735
- 36.
Barrera Valencia John Jairo. Análisis de la salud móvil en Colombia 2016–2017. Universidad CES, 2018.
- 37. Bhagat K, Nyazema N. General practitioners and clinical guidelines. East Afr. Med. J. 2001;78:30–4. pmid:11320762
- 38. Austad B, Hetlevik I, Mjølstad BP, Helvik AS. Applying clinical guidelines in general practice: A qualitative study of potential complications. BMC Fam. Pract. 2016;17:1–8. pmid:27449959