http://orcid.org/0000-0002-6167-6309
Figures
Abstract
There are two million people with chronic kidney disease (CKD) worldwide. In Mexico, it is estimated that by 2025, there will be 212 thousand CKD cases. Among the renal replacement treatments, peritoneal dialysis (PD) exists either in the continuous ambulatory (CAPD) or automated (APD) mode, which requires continuous monitoring and strict control. Thus, several software systems have been proposed to perform reliable remote monitoring of patients using PD but also to achieve the goal with effectiveness, efficiency and satisfaction; i.e., in software engineering, this is called usability. However, few studies have addressed usability issues using case studies with patients and medical staff in real domains. In this paper, we present a usability assessment of a telemonitoring system for patients with CKD on peritoneal dialysis treatment through a case study with patients and medical staff of the Mexican Institute of Social Security (IMSS). The usability evaluation was carried out through the application of two satisfaction instruments. These instruments evaluated multiple usability criteria, such as navigability, interactivity, motivation, satisfaction, and applicability. The results obtained from the usability evaluation show that, on average, the services offered by the system have 91.3% acceptance by users (patient-doctors), with the APD and CAPD exchange data registration services having the highest acceptance for patients, with a positive perception of 94.5% and 92.3%, respectively. Meanwhile, for the doctors and nurses, the alarm reception for patients in a risk situation was highest with 95% acceptance. Based on the obtained results, the evaluated telemonitoring system holds wide acceptance, satisfaction, and applicability from patients’ and doctors’ perspectives. It is also noted that the evaluated system considers and satisfies the requirements and suitable parameters that should be monitored in PD treatment according to studies presented in the literature.
Citation: Martínez García MA, Fernández Rosales MS, López Domínguez E, Hernández Velázquez Y, Domínguez Isidro S (2018) Telemonitoring system for patients with chronic kidney disease undergoing peritoneal dialysis: Usability assessment based on a case study. PLoS ONE 13(11): e0206600. https://doi.org/10.1371/journal.pone.0206600
Editor: Micah Chan, University of Wisconsin, UNITED STATES
Received: May 2, 2018; Accepted: October 16, 2018; Published: November 6, 2018
Copyright: © 2018 Martínez García et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper.
Funding: This project was supported by the Consejo Nacional de Ciencia y Tecnología (CONACYT) of Mexico with project number 1525.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Worldwide, there are two million people with chronic kidney disease (CKD) [1]. In Mexico, the problem of terminal CKD presents alarming dimensions since based on projections, it is estimated that by 2025, there will be 212 thousand cases [1]. This disease in the Mexican context is classified as an emerging disease because of the growing number of cases, including approximately 80,000 patients in the IMSS who receive, among the renal replacement therapy PD options, continuous ambulatory (CAPD) and automated (APD) [2–4] PD. Both PD modalities must be accompanied by a diet to control the levels of phosphorus, potassium, sodium, and calcium, as well as restrictions in fluid intake [5]. In these renal replacement treatments, approximately 70% of patients attend the IMSS [2]. CAPD and APD are home therapies, carried out throughout the day, and it is only necessary to go to the hospital to carry out tests and reviews, which usually occur every three or four months [6]. The main PD disadvantages consist of the fact that the patient or a caregiver should perform the treatment. In addition, it requires careful dedication when taking biomedical data and carrying out analytical work to determine dehydration or fluid retention in the body [5]. The patient also evaluates and records several types of data on a dialysis sheet, including the dialysis total time, infused PD fluid volume, infusion time per cycle, dwell time, drained fluid characteristics (clear, cloudy, other aspects), and assessment of pain presence and/or discomfort. These data, recorded on paper, are delivered for review every three months. During this period, there is no biomedical index monitoring or rapid risk situation control in cases of an imbalance or delay in the limit range provided by the physician. Additionally, illegibility problems in the patient's notes should be considered, as well as the notes being lost or forgotten. Some studies [7–12] have proposed systems to address this problem. However, the telemonitoring system presented by Cuevas JR et al. [13] is characterized by offering different services for monitoring, control and remote treatment of patients on CAPD and APD. This system is assembled by a native Android application, which provides services to the patient, and by a mobile Web application, which provides a set of services to the medical staff. In this context, usability is one of the central requirements in software quality terms that this type of system must provide [14]. The ISO 9241–11 standard defines usability as the capability of a product to be used by certain users to achieve specific objectives with effectiveness, efficiency, and satisfaction within a specific use context [14]. The evaluation of the systems' usability is a tool that allows the users to identify the most important characteristics that are qualified when using the applications and, in turn, provides information about aspects that should be improved in the system to increase the possibility of its use in the daily lives of people. There are different studies in the literature [15, 16] that identify the lack of research focused on evaluating the usability of telemonitoring systems through case study series with patients on PD treatment and physicians in a real environment [15–20].
This article presents a usability evaluation conducted on a telemonitoring system, proposed by Cuevas et al. [13], for patients with chronic kidney disease in peritoneal dialysis treatment through a case study with patients and medical staff of the IMSS. To achieve this, the usability evaluation was carried out on the main services offered in the mobile applications for patients and physicians belonging to the telemonitoring system proposed in [13]. In our study, we perform a usability evaluation method composed of two questionnaires or satisfaction instruments aimed at the end users of the system. This evaluation was due to the possibility of having patients and medical staff committed to interacting and using the system as a tool that complements the follow-up and control of PD treatment. The results obtained show that the APD and CAPD exchange data registration services had the highest acceptance in terms of usability among patients with 94.5%. On the other hand, the results obtained in the usability evaluation of the services offered to the physician reflect that the highest percentage of positive qualifications (95%) was obtained in the alarm reception service for patient risk situations. Finally, it is noted that the telemonitoring system [13] evaluated in this study considers and satisfies the ideal requirements and parameters that should be monitored in the PD treatment according to the study presented by Nayak et al. [21].
Materials and methods
Case study description
Within the format of a case study, a descriptive, prospective, longitudinal and observational analysis was conducted regarding a telemonitoring system for patients with CKD under dialysis support, as proposed by Cuevas et al. [13]. To achieve this goal, systems engineers from the National Laboratory on Advanced Informatics (LANIA) were trained in the management, monitoring, and registration of dialytic therapy of patients on PD at the General Hospital of Zone No. 11 of Xalapa, Veracruz, Mexico. The evaluated system allows, through a mobile web application and a native application in Android 4.0 or higher, patients to interact with doctors and nurses remotely regarding their symptoms, signs, and conditions of their dialysis therapy, as well as identifying risk situations through alert generation, which may avoid hospital admissions to emergency units. To prove the system usability, tests were carried out under fictitious cases, and then the case study was carried out by means of the selection for convenience of a sample obtained from the population of patients assisted by the PD program (CAPD and APD), with the approval of the National Research Commission Scientific (Ethical Committee) of the IMSS and the signing of informed consent by participants. Selection criteria included the availability of the patient and the relatives to receive training in the system management, facility of use of mobile devices with Android 4.0 or higher and an Internet connection. Twenty-four patients, 15 on CAPD and 9 on APD, participated in the case study together with four nephrologists, one internist and two specialist nurses. After training the physicians on the mobile web application, the patients were telemonitored using the system for a period of 9 months in order to reach an average monitoring time of five months/patient. Twenty-four subjects completed the monitoring, applying two usability evaluation instruments to patients, patients' relatives, doctors, and nurses.
Development of usability evaluation instruments
According to the Nielsen decalogue [22], the best practice guide of mobile web application development [23] and the related works analysis [15–20], diverse criteria for the usability evaluation of the telemonitoring system were identified and are detailed below (see Table 1).
Considering the described criteria, in our study, two instruments were constructed for evaluating the usability of the telemonitoring system introduced in [13]:
- The instrument for evaluating the patient's native application services is included later in this document as Instrument 1.
- The instrument for evaluating the medical personnel's mobile web application services is included later in this document as Instrument 2.
Moreover, these instruments allowed us to obtain feedback regarding the design and functionality of the telemonitoring system.
Description of instrument 1.
Instrument 1 was designed to evaluate the usability of the following services of the patient's native Android application [13]:
- APD and CAPD exchange data registration services. These services allows the patient to enter the necessary biomedical data that manage the monitoring and control of dialysis under the CADP and ADP modalities.
- Alerts. This service offered to the patient informs the doctor, by a text message and an email, concerning a patient's possible risk situation, such as the characteristic of drained fluid being different from transparent or ultrafiltration not being as expected.
- Notifications search. This service allows patients to examine the dates of events related to their PD treatment, such as medications, reminders and doctor's recommendations.
According to the services mentioned above, Instrument 1 is composed of three sections. Section 1 includes 39 questions that evaluate the usability of the APD and CAPD exchange data registration services. Section 2, a set of 14 questions, evaluates the usability of the alert service. Finally, section 3, composed of 35 questions, evaluates the notifications search service.
Description of instrument 2.
Instrument 2 was designed to evaluate the usability of the following services of the doctor's mobile Web application [13]:
- APD and CAPD exchange data registration search services. In these services, the medical staff can monitor different data of the patient's treatment, such as ultrafiltration, partial balance, total balance, type of fluid and abnormal symptoms (dizziness, vomiting, swelling, etc.) that the patient reports.
- Alerts search. In this option, the physician can consult the alert details generated by the patients
- Notifications. By this service, the doctor can generate a message to the patient regarding the following situations: a recommendation in response to a received alert, a reminder of an appointment, or a recommendation of adjusting the patient’s therapy and/or medication.
Therefore, Instrument 2 is composed of three sections. Section 1 includes 36 questions that evaluate the usability of the APD and CAPD exchange data registration search services. On the other hand, the 2nd and 3rd sections evaluate the usability of the alerts search service and notifications, respectively. Both sections are composed of 34 questions.
Results
Sociodemographic and clinical results
In this case, we analyzed sociodemographic and clinical information of the patients' sample, 12 men and 12 women with a mean age of 53.4 years. In addition, 46% of them have a basic educational level, 25% have a medium educational level, and 29% have a high educational level (see Table 2). These patients had DP treatments for 43.3 months on average before starting the case study, where 71.4% had an assisted process and 28.6% were self-sufficient. During the telemonitoring process and control of the patients on PD treatment through the system by health staff, several risk situations were immediately detected, such as fluid retention, dehydration, and peritonitis. Moreover, it is highlighted that the average effective use time of the application by patients was 4.9 months/patient (see Table 3).
Instrument results
In the developed instruments, the assessment scale selected for each criterion consists of four conceptual levels of satisfaction: a) Strongly agree, b) Moderately agree, c) Somewhat agree and d) Neutral, based on the Likert scale [24]. To simplify the results' presentation and provide a general idea of the bias that the perception of the patients and medical staff involved in the case study takes, a dichotomy is considered on the assessment scale. Therefore, the responses Strongly Agree and Moderately Agree were grouped as positive or favorable rating responses, whereas Somewhat Agree and Neutral were grouped as negative or unfavorable rating responses. The instrument results are presented below.
Instrument 1 results.
In general, the CAPD and APD exchange data registration services were the most accepted by patients with 94.5%, as shown in Fig 1.
The criteria that obtained 100% favorable opinion were (see Fig 1) control and freedom, consistency and standards, recognition instead of memory, minimalist and aesthetic design, aim and purpose clarity, interactivity, feedback and applicability. In a range of 91% to 94%, favorable evaluations are the criteria of congruence between the real world and the system, prevention of errors by an adequate design, efficiency in error handling, motivation and satisfaction (recommendation). Finally, the system visibility criteria, flexibility and efficiency of use, help documentation, navigability, and activities obtained a favorable evaluation between 78% and 89%.
Regarding the alert generation service, a general percentage of 88% favorable opinions was obtained from the patients. The criteria evaluated with a 100% favorable opinion were satisfaction and applicability (see Fig 2). Moreover, the system visibility criteria, congruence between the real world and the system, consistency and standards, flexibility and efficiency of use, efficiency in handling errors, aim and purpose clarity, and feedback obtained a positive evaluation between 79% and 95%. In this service, the motivation criterion was the one that obtained the lowest percentage of favorable opinion (63%). This was because some patients did not receive feedback from the doctor in response to the alert generated.
The notifications search service obtained 83% in the general percentage of positive opinion (see Fig 3). In this service, the satisfaction criterion (recommendation) was the best evaluated with a 100% favorable opinion, and the system visibility criteria, control and freedom, consistency and standards, prevention of errors by an adequate design, recognition instead of memory, minimalist and aesthetic design, efficiency in error handling, help documentation, aim and purpose clarity, navigability, feedback, motivation, and applicability obtained a positive opinion percentage between 71% and 94%. The criterion of congruence between the real world and the system was the lowest in this service, with 59% favorable opinion.
Instrument 2 results.
The results obtained from Instrument 2, through the medical staff answers in terms of favorable opinion, were 94% for the CAPD and APD exchange registrations search service, 95% for the alerts search service, and 93% for the notification generation service. Concerning the CAPD and APD exchange registration search service, the evaluation criteria with 100% positive opinion were system visibility, control and freedom, recognition instead of memory, minimalist and aesthetic design, help documentation, navigability, interactivity, and activities (see Fig 4). The rest of the evaluation criteria also obtained a satisfactory percentage of favorable opinion between 81% and 93%.
Regarding the alerts search service, the evaluation criteria with 100% favorable opinion were the prevention of errors by an appropriate design, recognition instead of memory, minimalist and aesthetic design, handling error efficiently, help documentation, aim and purpose clarity, navigability, interactivity, and satisfaction (recommendation) (see Fig 5). The other evaluation criteria are between 71% and 95% favorable opinion.
Finally, within the notification generation service, the evaluation criteria with 100% favorable opinion were consistency and standards, minimalist and esthetic design, aim and purpose clarity, navigability, interactivity, feedback, and satisfaction (recommendation) (see Fig 6). The other evaluation criteria obtained between 79% and 96% favorable opinion.
Overall, it was found that from the patients and doctors' viewpoint: 1) The text, the images, the order and the way in which the information is presented are adequate, allowing a natural relationship between the users (patient and doctor) and the application; 2) both mobile applications of the system follow a consistent standard in all its parts, namely, its design, operation, and terminology; 3) both the patient and the doctor have all the information available and are not forced to use their memory to follow the sequence of the interaction within the services used; 4) the interface is flexible, having the necessary resources to behave according to the needs of both users; 5) there are no distracting text or image elements that are important for operational purposes; 6) the objective pursued by each service is clear to the patient and doctors; 7) within the services, there are clear and consistent navigation options; 8) the patient and doctor feel motivated to use the services; 9) the patient and doctor perceive the services to be easy and pleasant to use; and therefore recommend them to other patients and doctors; and 10) patients and doctors perceive a high utility of the telemonitoring system.
Discussion
Some studies [15–20] have carried out a usability assessment of systems focused on remote monitoring and control of patients with CKD on PD treatment. In [17], a possible study protocol was introduced, which evaluates the utility and scalability of a telemedicine system for patients with CKD. However, the study did not include results considering a case study in a real scenario. Regarding the study presented in [18], only the satisfaction criterion was evaluated by comparing the system against face-to-face consultation. In [19], two types of evaluations were carried out: evaluative and formative, which focused on functionality and performance qualitative features with a group of 29 patients. Finally, the study presented in [20] describes a costs and savings evaluation about consultation time and hospital admissions of a telemedicine system integrated with video conference equipment that should be installed in the patients' homes.
Based on the review of studies presented by Wallace E.L, Rojas SV, and KS Nayak [15, 16, 21] and in this work, it is noted that there is limited research work focused on the assessment of the usability of telemonitoring systems through a case study with patients on CAPD/APD and medical staff in a real environment. Therefore, in our research, we included criteria that represent preponderant indicators that cannot be ignored, such as system visibility, congruence between the real world and the system, control and freedom, consistency and standards (organization), prevention of errors by an adequate design, recognition instead of memory, flexibility and efficiency of use, minimalist and aesthetic design, handling errors efficiently and help documentation, aim and purpose clarity (goals), navigability, interactivity, activities, and feedback. Likewise, criteria that are less frequent in the reviewed studies can be identified, and for that reason, it is interesting to explore them to make a contribution, such as motivation, satisfaction (recommendation) and applicability, which are criteria directly associated with the users' perspective. In this regard, this study considers intrapersonal aspects of the users subject to the stimulus generated by the system usage, giving an assessment oriented towards the system relevance for stakeholders, which allows for accurate feedback.
On the other hand, it is also important to establish a baseline for user satisfaction measures, with the aim of future system improvements that support adherence to the system and promote the recommendation of its usage to other stakeholders in PD therapies. Concerning the applicability criterion, it is necessary to know those system services that stakeholders may consider less relevant for the follow-up and control of their PD therapy, promoting systematic system updates.
Therefore, the usability analysis and assessment presented in this paper can serve as a reference point for system developers looking to improve or develop telemonitoring systems. It can also be seen that there is a need to standardize the usability assessment criteria for systems of this nature since, despite great efforts in the matter, system properties that motivate the creation of new evaluation criteria have emerged.
Based on the results obtained in the usability assessment, the system evaluated [13] in this study considers and satisfies the ideal requirements and parameters that should be monitored in PD treatment according to the study presented by KS Nayak [21].
Conclusions
In this work, a usability assessment of a telemonitoring system for patients with CKD on PD treatment was presented based on a case study involving patients and doctors of the IMSS in a real environment. The results obtained from the case study show that, on average, the services offered to the patient by the system have 89.5% acceptance, with the APD and CAPD exchange data registration services being the most acceptable for patients, with a positive perception of 94.5% and 92.3%, respectively. On the other hand, services offered to medical staff obtained an average of 93.74% acceptance; in this case, the alarm reception service for patients in risk situations with 95% was the most accepted among the medical staff. Finally, it is noted that the evaluated system in this study considers and satisfies the requirements and suitable parameters that should be monitored in PD treatment according to the specialized literature.
References
- 1. Cadena-Naranjos DA, Javier-Hernández AM. Detección de Enfermedad Renal Crónica Oculta Mediante Fórmula de Cockcroft-Gault en Pacientes con Enfermedades Crónicas Degenerativas en el Primer Nivel de Atención en Comalcalco, Tabasco, México. Salud en Tabasco 2016; 22:42–45.
- 2.
México: Secretaría de Salud. Tratamiento Sustitutivo de la función Renal. Diálisis y Hemodiálisis en la Insuficiencia Renal Crónica en el Segundo y Tercer Nivel de Atención, CDMX, México: Centro Nacional de Excelencia Tecnológica en Salud (CENETEC); 2014.
- 3.
Instituto Mexicano del Seguro Social, Coordinación de Administración de Riesgos Institucionales. Evaluación de los riesgos considerados en el programa de administración de riesgos institucionales. Dirección de Finanzas 2007; 39–44.
- 4. Cueto-Manzano AM, Rojas-Campos E. Status of renal replacement therapy and dialysis peritoneal in Mexico, International Society for Peritoneal Dialysis 2007; 27:142–148.
- 5. Contreras F, Espinosa JC, Esguerra GA. Quality of life, self-efficacy, coping styles and adherence to treatment in patients with chronic kidney disease undergoing haemodialysis treatment. Psicología y Salud 2008; 18:165–179.
- 6. Treviño BA. Insuficiencia renal crónica: enfermedad emergente, catastrófica y por ello prioritaria. Cirugía y Cirujanos 2004; 72:3–4. pmid:15087044
- 7. Nakamoto H. Telemedicine System for Patients on Continuous Ambulatory Peritoneal Dialysis. Peritoneal Dialysis International 2007; 27:21–26.
- 8. Kirsch C, Mattingley-Scott M, Muszynski C, Schaefer F, Weiss C. Monitoring chronically ill patients using mobile technologies. IBM systems journal 2007; 46:85–93.
- 9. Kaldoudi E, Passadakis P, Panagoutsos S, Vargemezis V. Homecare Telematics for Peritoneal Dialysis. The Journal on Information Technology in Healthcare 2007; 5:372–378.
- 10. Gómez-Martino JR, Suárez-Santisteban MA, Gallego Domínguez S, González Castillo PM, Covarsi Rojas A, Castellano Cerviño I, et al. Telemedicina aplicada a la nefrología. Otra forma de consulta. Nefrología 2008; 4:407–412.
- 11.
Sota Y, Yamamoto K, Hirakawa M. Support of Self-Management for Chronic Kidney Failure Patients. Proceedings of the 2011 Visual Information Communication-International Symposium; August 04–05; Hong Kong, China, ACM; 2011.
- 12. Nakamoto H. How Automated Peritoneal Dialysis Is Applied and Maintained in Japan. Contributions to nephrology 2012; 177:13–23. pmid:22613910
- 13. Cuevas JR, Dominguez EL, Velazquez YH. Telemonitoring System for Patients with Chronic Kidney Disease Undergoing Peritoneal Dialysis. IEEE Latin America Transactions 2016; 14:2000–2006.
- 14.
ISO—International Organization for Standardization. ISO 9241–11 Ergonomic requirements for office work with visual display terminals (VDTs). Part 11: Guiadance of usability, 1998, Available at: http://www.it.uu.se/edu/course/homepage/acsd/vt09/ISO9241part11.pdf
- 15. Wallace EL, Rosner MH, Alscher MD. Remote Patient Management for Home Dialysis Patients. Kidney International Reports 2017; 2:1009–1017. pmid:29634048
- 16. Rojas SV, Gagnon MP. A Systematic Review of the Key Indicators for Assessing Telehomecare Cost-Effectiveness. Telemedicine and e-Health 2008; 14:896–904. pmid:19035798
- 17. Jeffs L, Jain AK, Man RH, Onabajo N, Desveaux L, Shaw J, et al. Exploring the utility and scalability of a telehomecare intervention for patients with chronic kidney disease undergoing peritoneal dialysis—a study protocol. BMC Nephrology 2017; 18:155. pmid:28486991
- 18. Fernandes NM, Bastos MG, De Oliveira NA, Costa Ado V, Bernardino HS. Telemedicine: Development of a distance care system for pre-dialysis chronic kidney disease patients. Jornal Brasileiro de Nefrologia 2015; 37:349–358. pmid:26398645
- 19. Skiadas M, Agroyiannis B, Carson E, Cramp D, Fourtounas C, Darnige A, et al. Design, Implementation and Preliminary Evaluation of a Telemedicine System for Home Haemodialysis. Journal of Telemedicine and Telecare 2002; 8:157–164. pmid:12097177
- 20. Gallar P, Vigil A, Rodriguez I, Ortega O, Gutierrez M, Hurtado J, et al. Two-year experience with telemedicine in the follow-up of patients in home peritoneal dialysis. Journal of Telemedicine and Telecare 2007; 13: 288–292. pmid:17785025
- 21. Nayak KS, Ronco C, Karopadi AN, Rosner MH. Telemedicine and remote monitoring: supporting the patient on peritoneal dialysis. Peritoneal Dialysis International Journal 2016, 36: 362–366.
- 22.
J. Nielsen, “10 Usability Heuristics for User Interface Design”, 1995. [Online]. Available at: http://www.nngroup.com/articles/how-to-conduct-a-heuristic-evaluation/
- 23.
Asociación Mexicana de la Industria de Tecnologías de Información, Mejores prácticas para el desarrollo y uso de aplicaciones móviles, 2013. [Online]. Available at: http://amiti.org.mx/wp-content/uploads/2013/10/Estudio-Apps_Mejores-Pr%C3%A1cticas.pdf
- 24.
Bertram D. Likert scales. University of Calgary, Department of Computer Science Web Site [Online], 2007. Available at: http://poincare.matf.bg.ac.rs/~kristina//topic-dane-likert.pdf