Correction
30 Jan 2014: Little A, Medhanyie A, Yebyo H, Spigt M, Dinant GJ, et al. (2014) Correction: Meeting Community Health Worker Needs for Maternal Health Care Service Delivery Using Appropriate Mobile Technologies in Ethiopia . PLOS ONE 9(1): 10.1371/annotation/fedf94d2-cf4e-494c-8828-85861ce282a5. https://doi.org/10.1371/annotation/fedf94d2-cf4e-494c-8828-85861ce282a5 View correction
Figures
Abstract
Background
Mobile health applications are complex interventions that essentially require changes to the behavior of health care professionals who will use them and changes to systems or processes in delivery of care. Our aim has been to meet the technical needs of Health Extension Workers (HEWs) and midwives for maternal health using appropriate mobile technologies tools.
Methods
We have developed and evaluated a set of appropriate smartphone health applications using open source components, including a local language adapted data collection tool, health worker and manager user-friendly dashboard analytics and maternal-newborn protocols. This is an eighteen month follow-up of an ongoing observational research study in the northern of Ethiopia involving two districts, twenty HEWs, and twelve midwives.
Results
Most health workers rapidly learned how to use and became comfortable with the touch screen devices so only limited technical support was needed. Unrestricted use of smartphones generated a strong sense of ownership and empowerment among the health workers. Ownership of the phones was a strong motivator for the health workers, who recognised the value and usefulness of the devices, so took care to look after them. A low level of smartphones breakage (8.3%,3 from 36) and loss (2.7%) were reported. Each health worker made an average of 160 mins of voice calls and downloaded 27Mb of data per month, however, we found very low usage of short message service (less than 3 per month).
Conclusions
Although it is too early to show a direct link between mobile technologies and health outcomes, mobile technologies allow health managers to more quickly and reliably have access to data which can help identify where there issues in the service delivery. Achieving a strong sense of ownership and empowerment among health workers is a prerequisite for a successful introduction of any mobile health program.
Citation: Little A, Medhanyie A, Yebyo H, Spigt M, Dinant G-J, Blanco R (2013) Meeting Community Health Worker Needs for Maternal Health Care Service Delivery Using Appropriate Mobile Technologies in Ethiopia. PLoS ONE 8(10): e77563. https://doi.org/10.1371/journal.pone.0077563
Editor: Waldemar A Carlo, University of Alabama at Birmingham, United States of America
Received: May 10, 2013; Accepted: September 3, 2013; Published: October 29, 2013
Copyright: © 2013 Little et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This study was made possible through primary funding provided by Mekelle University (Ethiopia) (http://www.mu.edu.et). Venture Strategies Innovations (http://vsinnovations.org; AECID (http://www.aecid.es) and Alcala University (http://www.uah.es) also gave additional funding. Software development was supported by Digital Campus Ltd (http://www.digital-campus.org), a UK-based not for profit company. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript
Competing interests: The authors have declared that no competing interests exist.
Introduction
There is considerable enthusiasm for mobile health (mHealth) interventions and it has been argued that there is huge potential for mobile-health interventions to have beneficial effects on health and health service delivery processes, especially in resource-poor settings [1,2]. While a number of innovative mHealth projects have been launched in Ethiopia and other low-income countries in the past years, many have been short-term or have covered a limited geography [3,4]. Recently, the Ethiopian Federal Ministry of Health (FMoH) has identified the need to develop a scalable and comprehensive mHealth platform and strategy that could meet long-term needs and strengthen the primary health care system [5,6].
mHealth is a term used for interventions and programs designed to support medical and public health through the use of mobile technology [7,8]. The term commonly refers to mobile communication devices, such as mobile phones and smartphones, to deliver health services and transmit health-related information. mHealth ranges from simple mobile-based phone applications for the transfer of health information on basic handsets via short message service (SMS) to highly sophisticated diagnostic applications that rely on more advanced equipment (smartphones and tablets) and robust back-end data systems [9,10,11]. The 2011 mHealth in Ethiopia report identified five priority areas where mHealth could best help to strengthen the primary health care system: referrals, data exchange, supply chain management, training, education and consultation [5].
Ethiopia’s health needs are vast and reflect the high poverty levels, with more than thirty million people living in extreme poverty. In the last decade, real improvements have been seen in health services and outcomes, including a significant reduction of under five mortality, but this was from a very low baseline and huge challenges remain to guarantee Ethiopians have access to quality maternal and newborn health services [12]. Around 90% of all births take place at home and only 26% of women living in rural areas who give birth receive antenatal care from a skilled health provider and less than 3% of women receive postnatal care in the first week after delivery. Despite recent advances, neonatal mortality rate is still 37 deaths per 1000 live births and 1 in 17 Ethiopian children dies before their first birthday [13].
Ethiopia’s Health Extension Program (HEP) is a key component of the strategy to address these reproductive, maternal and newborn health program barriers. Strengthening and supporting the HEP is crucial for further acceleration of progress towards health-related Millennium Development Goals [14,15,16]. This is the primary channel through which health education, basic curative care and preventive components of primary health care reach Ethiopia’s population. HEP’s primary implementers are Health Extension Workers (HEWs) who as the frontline workers in the country’s health system interact with communities and families. HEWs have a variety of information and communication needs, and their ability to effectively communicate and exchange information directly impacts their ability to provide care to the communities they serve [17,18,19].
Whilst the Ethiopian FMoH is eager to explore the use of mobile technologies, without solid evidence of the health benefits and cost-effectiveness, they are reluctant to invest the scarce resources in widespread implementation [20,21,22,23,24].
Our aim in this study has been to meet the technical needs of Health Extension Workers (HEWs) and midwives for maternal health using appropriate mobile technologies tools. The health workers in our maternal health care project have now been using smartphones for more than twenty months, so we have been able to build up a good picture about what works and where there are issues. Most of the information in this paper is based on field reports our research team at Mekelle University (AM, HY) have been sending back following the training sessions they have been running and followed up discussions with the health workers.
Materials and Methods
Background: setting
This study was conducted in Tigray region, the northern most regional state of Ethiopia. Two districts,Kilte Awelalo and Hintalo Wajerat, were selected for the study, in consultation with the regional health bureau. In total, taking into account staff replacements, 20 HEWs, 12 midwives and 5 supervisors were involved. Equal proportions of health workers, health posts and health centers were selected from each district. Access to transportation and GPRS network coverage were the two main criteria considered for selection districts and health facilities. The study was approved by the health research and ethics review committee at the College of Health Sciences of Mekelle University. Training of the health workers and deployment of the platform began in August 2011. The data in this paper includes all the real patient encounters recorded between December 2011 and May 2013.
Ethics statement
The study was approved by the health research and ethics review committee of the College of Health Sciences of Mekelle University (no: ERC 0032/2011). Written consent for participation was obtained for each health worker. The health workers were informed about their right to withdraw from the study at any time.
Development: Case management tools and scorecard/analytics dashboard mobile applications
The technical components developed and deployed as part of this project cover: 1) case management tools; 2) Scorecard/analytics dashboard
These components have been built on systems already available, using open source components as far as it’s been possible. Any new code and content developed through this project has been released under the appropriate open source or creative commons license models. The technical development methodology was rapid, agile and iterative, allowing us to respond very quickly to changes in users needs/demands and in response to user feedback.
Case management tools.
We used the OpenDataKit platform (ODK) for the case management tools development [Appendices 1 and 2]. ODK is a generic data collection tool, designed for workers in the field, especially those with poor or no mobile internet connectivity, to collect data which can then be submitted to a central server when a data connection becomes available. The decision to use ODK was based on several factors: a) open source - so we were able to implement localisations and customisations and host our own back-end server to directly access the database, required for developing the scorecard and analytics dashboard; b) supports XForms standard - and so supports a wide variety of question types - text/numeric entry, multiple choice and multiple select and displays these to users on mobile devices running Android. In addition, it can handle Global Positioning System (GPS) location information, photos, videos, audio, and barcodes; c) flexible to expand for other areas, such as basic stock control, immunisation records.
Several customisations were required to make the phones and ODK suitable for use with the health workers in Ethiopia: a) local language support; b) supporting local calendar and c) ODK widgets.
The native language for all the HEWs, midwives and supervisors is Tigrinya, which uses a non-Latin script, Ge’ez. The version of Android we were using (version 2.3) does not natively support the Ge’ez script, although this script is supported in more recent versions of Android (version 4+). We installed an additional system font on all the phones to allow displaying text in Ge’ez, installation of this also required all the phones to be rooted. To allow data entry in Tigrinya or Amharic we developed a Ge’ez virtual keyboard, since at the time, no Ge’ez capable keyboards were available for Android, although now other Ge’ez keyboards are available. This allowed the health workers to decide for themselves which keyboard they preferred to use and whether they preferred to enter text data in either Ge’ez or Latin scripts (Figure 1a and 1b).
Ethiopia has its own calendar based on the Julian calendar, which is widely used in preference to the Gregorian calendar, especially in rural areas. We developed an Ethiopian date widget for ODK to allow health workers to enter dates in the calendar system they understood well. In addition to the Ethiopian date widget, we also developed three other ODK widgets to assist health workers in scheduling. One for calculating and displaying the expected delivery date (EDD), based on the last menstrual period (LMP) entered and two more widgets for giving suggested next antenatal care (ANC) and postnatal care (PNC) appointment dates, again based on the LMP date entered. For the appointment dates, only a suggested date range was given, the health workers and patients were free to enter whatever date may be most suitable for them. The pregnancy calculator was also transformed into a standalone Ethiopian Pregnancy Calculator application.
Scorecard and analytics dashboard.
We developed an analytics dashboard and a mobile scorecard to allow HEWs, midwives, their supervisors and the local health bureaus to track the progress of pregnant mothers, their medical and pregnancy risk factors, and a range of key performance indicators. Providing information back to health workers and their supervisors about their performance, was designed to help the health workers manage their workload and patients. Performance indicators included the number of ANC, Delivery and PNC visits made.
ODK is primarily a data collection tool and it has no built-in functionality for creating customized reports back to mobile users. We built both the analytics dashboard and the mobile scorecard to access the ODK database directly to provide customised reports and information back to health workers, supervisors, local health bureaus and the research team. Although the analytics dashboard and the mobile scorecard both derive the data from the ODK database, each were created for different use cases and targeted towards different user groups, although any registered user may access either should they wish.
We felt it was important that the data collected by health workers could be used directly by them to assist them in their work, for example scheduling appointments, risk factor assessment, and for them to feel ownership of the project and data, rather than a health management information system, used only by health bureaus and the FMoH.
Analytics dashboard for local health bureaus management and the research team.
These groups would be most interested in getting an overall picture of the level of activity in different health posts and centres, key performance indicators and the ability to compare performance between different health posts, centres or districts. Regular reports were available for supervisors and health bureaus, as well as full details of any protocol forms entered so they could be printed as a paper backup or reference at the health post (Figure 2).
The analytics dashboard also highlighted inconsistencies in the data, for example where a patient id may have been used twice for different patients. We made the assumption that these users would have access to a laptop or personal computer (PC) with a reasonable internet connection, so the analytics dashboard is web based and designed to run in a full laptop or PC web browser, with an active internet connection.
Mobile scorecard for the HEWs, midwives and supervisors.
These staff, based in more rural areas, require information such as which pregnant mothers are due for maternal care (ANC or PNC) visits or delivery in the coming days/weeks, and any associated risk factors. Midwives at local health centres need information about upcoming deliveries in their district, so any special preparations can be made for delivery, especially with high risk cases, even when they may not have seen these patients before (Figure 3a&b).
Showing deliveries due and associated risk factors (personal data has been pixelated).
Access to this data was provided by an Hypertext Markup Language (HTML5) web application, which could be accessed via the web browser of their smartphone. Using a local database for the web app, information would be cached and accessible even when they had no internet connection. The data would update whenever an active connection was available. Four key areas of information were provided in this application: a) performance indicators - number of different visits (ANC, PNC and delivery) made in the last month compared to the previous month; b) Deliveries due in the next month – with basic patient information (name, identification (ID), village and phone number if available), and risk factor analysis; c) Tasks and appointments due in the next month - basic patient information (name, ID, village and phone number if available) and type of appointment; d) Appointments overdue (missed) - basic patient information (name, id, village and phone number if available) and type of appointment.
Given the reliability and speed of the mobile internet connection available to these health workers, we kept the amount of information cached on the phone to the minimum. Neither the analytics dashboard, or the mobile scorecard gave users any access to edit or change any data.
Details to obtain all the code for the technical development, as well as access to the demonstration site, is available in appendices S1 and S2.
Implementation, Monitoring and Evaluation
Monitoring and evaluation: data analysis, follow up, user survey.
During data collection period, monthly visits were made to the participant health posts and centres. Telephone contact was used often, especially when any problems with the phone usage and protocols registration were detected in the analytics dashboard and required consultation with the health worker. A mixed method of both qualitative and quantitative approaches was used to explore the feasibility of implementing these mHealth tools. In-depth interviews took place in the middle of the actual implementation period. It was semi-structured with open ended and probing questions. All interviews were tape recorded and all data was transcribed verbatim in the local language and then translated into English. Respondents were asked to evaluate the reliability of the solution as well as their satisfaction. Detailed results of this survey are being explained in another submitted manuscript.
Training for Health Workers.
We pre-tested the customized ODK application and maternal health care protocols and trained health workers over a period of three months, with a mix of group and outreach training. The training included basic functions of the smartphone, configuring GPRS internet connection on the smartphone, purpose and content of the maternal health care protocols, installing and launching ODK software, switching from English language to local language and vice versa. During this pre-test phase, health workers practiced with the smartphone, and customized ODK, and maternal health protocols by submitting practice data to the server. During each training day, feedback was collected and improvements on the protocols and application were made accordingly. Actual implementation of the whole package of the application and assessment of its feasibility at the health posts and health centers was carried out from December 2011 to May 2013. Health workers were using the smartphones when giving services of ANC, delivery and PNC to women and were submitting real data to the server. During this phase, onsite supervision was made every month by the research team.
Protocol and form development.
Initial versions of the protocol forms were developed by the research team, these were then iteratively refined based on feedback from health workers during the pre-implementation phase (Table 1). Due to the complexity/length of these protocol forms, most of the editing was done in an XML editor, rather than one of the automatic tools for building the forms. Once the protocols were finalised, all the text was translated into Tigrinya, so the health workers had the option to view the questions in English or Tigrinya.
| Protocol | Comments |
|---|---|
| Registration | Initial registration of the patient |
| ANC First Visit | The first antenatal care encounter |
| ANC Transfer | For patients who receive care at more than one health post/centre |
| ANC Follow Up | Follow up antenatal care encounters |
| ANC | This was introduced in may 2012 and replaced the ANC First Visit, ANC Transfer and ANC Follow Up protocol forms |
| ANC Lab Test | Lab test results when patient visited the health centre |
| Delivery: | Delivery/labor |
| Termination | For recording premature termination of pregnancies (e.g. induced or spontaneous abortion) |
Table 1. Full list of maternal and neonatal protocol forms used.
Data security.
To maintain security of the medical information transmitted between the phones and the server, all connections between the phone and the server were made over secure http using an SSL certificate. Once data entered in ODK was submitted to the server, the data was no longer stored on the phones and users needed their username and password to access either the mobile scorecard or analytics dashboard. Once logged into either of these, users are only able to view information related to patients in their districts. Full patient records were not stored on the phone.
Patient identification.
Patient identification (patient ID) was an important issue, since there isn’t a standard regional/national identification number we could readily use. Each patient encounter is recorded in a physical log book and the patient ID was simply the number of the next row in their log book. To try to save confusion between patients having different patient IDs in the log book and the electronic protocols, we identified patients by a combination of a health post code number (selected as the health post/centre name when viewed by health workers) and the id from the log book. Patients were given a card with their details to present when they returned for another appointment, whether at the same health post/centre or a different one. In addition to the health post and patient id, every protocol asked for the patients year of birth, age and first name - so we could use this information to determine if a patient id number may have been incorrectly input. For reference, in the rural areas many people do not know their exact date of birth and hence age, making it difficult to use these as reliable aids for patient identification, however we could use these to highlight where there may be a discrepancy.
Technical support.
Day-to-day technical support was provided by the local research team, including installing the phone system, applications, protocol forms and dealing with any queries from the health workers. Any issues the local research team couldn’t resolve were passed to the Digital Campus technical team for investigation.
Phone battery recharging.
Since many of the health workers did not have reliable access to mains electricity supply, all were provided with a solar lamp and phone charger. We originally provided a d.light (San Francisco, USA) but later changed to supplying ST2 solar lamp/chargers from the Solar Energy Foundation (Addis Ababa, Ethiopia), since these were available for purchase and supported in country. During the training sessions health workers were shown how to turn off/on the Wifi / Bluetooth / GPRS / GPS to help improve battery life.
Phone usage.
We placed no restrictions on the phones regarding the applications which could be accessed or what the top-up balance could be used for. The health workers were free to use and install any application, including using the phone, text-messaging and internet browser. Each health worker was provided with a 100 birr (five point three USD) top up card approximately once a month, and they were free to purchase and use additional top-up cards. We were able to obtain phone usage information - amount of top-ups and how much spent on voice, SMS and data each month - directly from the local mobile operator, EthioTelecom.
Technical specifications
Smartphones.
HTC Hero smartphones were purchased second hand, in order to keep the project costs down. Phones were unlocked, rooted and Cyanogen, a custom operating system distribution was installed. Each health worker was provided with a charger/adapter, cables, solar lamp/charger, plus an SD card of at least 2Gb. Dual battery chargers and extra batteries were also provided for those who had very poor mains electricity access, or who often spent extended time out of their health posts. With every phone we also gave a rubber or plastic protective cover and a small bag with shoulder strap for protection.
Server/Software.
The server (Dell PowerEdge) is running as a Ubuntu 10.04 (LTS) virtual machine (using VirtualBox), with MySQL 5, PHP 5, Apache 2 and Tomcat 6. The ODK software, both ODK Aggregate and ODK Collect has been kept up to date with current stable release versions, currently running ODK Aggregate 1.0.4 and ODK Collect 1.1.7.
Results
The key results have been separated into four categories below. Overall we found fewer technical issues than initially expected and the health workers very quickly became comfortable in using the phones.
Technical - hardware/infrastructure
The mobile internet connection, although not fast, was found to be much better than originally expected, even in rural and quite remote areas. 23 health posts and centers from a total of 47 (48.9%) had GPRS connection available at the time we visited (April 2011). Only those who were in an area of connectivity during this survey were included in the project. We had very few instances of the mobile data network being unavailable for a substantial period of time (more than one day). Contact with the local telecoms office informed us of the nature of the problems and when they were likely to be resolved. In April 2013 GPRS connection was available in 35 health posts and centres (74.4%) of our study districts. We had very few instances of the mobile data network being unavailable for a substantial period of time (more than one day). Contact with the local telecoms office informed us of the nature of the problems and when they were likely to be resolved.
Our initial expectation was that we may need to replace 25% of the phones through loss or breakage, according to previous reports. Until May 2013, only two phones out of 36 had been stolen - but one was later recovered (2.7%). Three phones (8.1%) had issues with insensitive screens and were replaced. This low level of breakage/loss was very significant, especially since we were using second-hand phones.
The phone model (HTC Hero) chosen for this study had an initial cost at eBay.com of one hundred and sixty-nine USD for lightly used models while retail value was five hundred and thirteen USD for new ones (2010). The cost of the phones was reduced more than 50% in the in the following months (down to 75 USD by January 2013).
Figure 4 shows a graph of the average monthly expenditure of the HEWs, midwives and supervisors on mobile top-up cards broken down by money spent on voice, data and SMS. Table 2 shows how this money spent translates into minutes of voice calls, data downloaded and SMSs sent per month. The number of SMSs sent by HEWs and midwives is very low. One possible reason for this - and this also came from our baseline survey interviews - was that the health workers don’t use text messaging because they are not confident in using the Latin alphabet, or perhaps they know the recipient of the message cannot read the Latin alphabet, or does not have a Ge’ez capable phone. From the amount of data usage, we can see that both health workers and supervisors are using the data connection for more than just submitting patient encounter records and the mobile scorecard and in the case of supervisors, substantially more. The data shows that each health worker per month makes approximately 160 mins of voice calls, downloads 27Mb of data and sends 3 SMSs. The health workers were adding their own top-up balance too in addition to the five point three USD we were giving. What was interesting for us is that the health workers are clearly using the data connection for much more than simply submitting the protocol forms and accessing the mobile scorecard.
Graph of the average monthly expenditure of the health workers on mobile top-up cards broken down by money spent on voice, data and SMS. Interestingly, the number of SMSs sent by HEWs and midwives was very low.
Technical - software
Input from the health workers was critical in the development of the protocol forms, which questions should be asked and clarifying any misunderstanding and ambiguities in the questions and response options. After producing an initial draft of the protocols, we made several revisions of each protocol form and its questions, based on input from the health workers and analysis of how they were using the forms for practice data entry. These iterations allowed us to identify potential problems before starting to collect data on real patients.
In using ODK for gathering information over an extended period of time, with different forms completed at different times, we are essentially forcing ODK to do something it was not designed for. Although ODK assigns a unique identification number to each form submitted, there is not a built-in way to link different form instances, either of the same or different form types, submitted at different times, which is required for gathering longitudinal information, hence we used the patient id to match up forms. Since the patient records were not permanently on the phones, we were depending on the patient ID being entered correctly to match up different encounters with a single patient. Patient id issues are discussed below.
We found that occasionally health workers would adjust the phone system date to match the Ethiopian calendar. For example, setting the phone date to 2 September 2005, to represent the date 2 Meskerem 2005 in the Ethiopian calendar, but which actually corresponds to 12 September 2012 in the Gregorian calendar. This led to the health workers being unable to submit protocols to the server because the phone operating system assumed the security certificate was not within it's validity period and their connections to the server were rejected.
In the early phases of the project, when the protocol forms were being refined and updated, we needed a well coordinated approach to transfer data and users from using one version of a particular protocol form to the updated one. ODK currently does not have functionality to automatically push out updates to the users’ phones and transfer all users to the new protocols at the same time.
Although we did not restrict the phone usage in any way, health workers had access to to change any and all of the system settings, install/remove applications etc, we had a very low number of issues. Almost all the support issues we did experience (for example, the ODK application accidentally being uninstalled) were easily managed by the local research team.
Usage Statistics, Data Quality and Consistency
From figure 5 there appears to be no evidence of increased usage of protocol forms by health workers over time. However, given the only incentive to use the application was to assist them in their work, and there was no penalty if they did not use the application, it is encouraging to see that there is continued usage and no drop in activity.
From figure 6, we can see that the midwives were substantially more active in submitting all types of protocols than the HEWs, although we would not expect HEWs to enter the ANC Lab Test protocol because these tests are always done by midwives, nor a high rate of delivery by HEWs since current Regional Health Bureau advice is that HEWs should not routinely assist with delivery. The difference between the rate of ANC and PNC protocols, especially for HEWs, probably reflects the priority which has been given to antenatal care compared to postnatal care and shows there is a lot of room for improvement in postnatal care provision (Figure 7). The usage of each type of protocol and the number of ANC and PNC encounters per health worker is in line with data from the EDHS [13]. In addition to the difference in activity between different health worker roles we noticed highly variable usage between different health workers and between different health posts and centres. Discussion of these issues is beyond the scope of this technical paper and it is being developed in another manuscript.
The difference between the rate of ANC and PNC protocols, especially for HEWs, probably reflects the priority which has been given to antenatal care compared to postnatal care and shows there is a lot of room for improvement in postnatal care provision.
As with the number of protocol forms submitted, we see varying levels of activity between different health posts/centres and roles (Figure 8). Little input from supervisors as to exactly what data they needed to help improve their supervisory role was obtained. We were given various statistical reporting template forms from local health bureaus, and we could have used data from our database to help complete this information. However, the reports seemed to vary significantly from area to area and the indicators reported on regularly changed.
Although the health workers all seemed to well understand the system for patient identification, we had many issues where the same patient ID was used for multiple patients. We identified approximately 6% of all protocol forms entered had an issue regarding patient identification. The check data used in each protocol (name, year of birth and age) helped us to determine where a patient id may have been used more than once or duplicate protocol forms being submitted. It should be noted there is naturally significant overlap between duplicate patient ids and inconsistencies in the age/year of birth. Possible causes for duplication of patient ids were the following:
- The protocol is edited and saved as new protocol before submission, in which case 2 copies of the protocol form were submitted for the same patient;
- Network connection issues - breaks in the network connection may mean the protocol form being successfully submitted to the server, but the notification of successfully submission is not received back on the phone, so the phone believes the protocol form has not yet been submitted, and so is submitted again;
- Typo in the patient ID;
- Restarting numbering in the log books - there appear to have been cases where the sequential numbering in the log book has restarted, for example at the beginning of a new year, meaning that patient ID number are reused for new patients.
- Multiple registration systems. Some health facilities did not use a single registration system, patients were registered differently depending on what service they came to use, for example, one log book for terminations and another for deliveries, but both log books using the same sequence of patient ID numbers. This led to patients being given the same patient ID that had already been given to others.
Consequences of patient ids being entered incorrectly or duplicated include: a) Appointment reminders being allocated to the wrong patient; b) Risk factor analysis either highlighting more risk factors than the patient actually had or omitting risk factors for a given patient.
The analytics scorecard highlighted possible patient identification issues, and in theory the majority should have been straightforward to resolve, for example by contacting the health worker shortly after the error was made to establish what the correct data should be. However delays in looking into these issues meant that a large backlog of errors (going back several months) was time consuming to resolve.
There may be instances where the same patient has been registered twice, for example, when attending a health post for an antenatal care encounters, and then re-registered at the health centre when attending for delivery. Variations in the spelling of names, especially when transliterated into the Latin alphabet and lack of consistent usage/knowledge of an exact date of birth, makes it hard to analyse, without significant effort, how many instances of re-registration occurred.
The analytics dashboard allows us to identify where training issues may be the cause of incorrect data being entered. An example is the fundal height and newborn weight measurements. In figure 9 for the fundal height, the majority of the data entered is consistent with our expectations, with the data points along the very bottom of the scatter plot. The handful of data points on the top of the chart appear to show where typos have been made in data entry. However the second line data points around the fundal height of 150, appear to show a regular and consistent error being made.
The handful of data points on the top of the chart appear to show where typos have been made in data entry. However the second line data points around the fundal height of 150, appear to show a regular and consistent error being made.
Analysis of the patient ages and parity shows we may have two underrepresented groups in our data, those who are pregnant for the first time and those who are under 20 years old. Discussions for the reasons for this are beyond the scope of this paper, but these findings highlight how the data collected could be used by health managers to ensure that health services are reaching those most at risk of complications during pregnancy.
User Survey, Usability and User Acceptance
Most health workers in this project very rapidly learned how to use and became comfortable with the devices and the Android operating system, so after initial orientation to using the devices, only limited technical support was needed and most of this could be provided by the local support staff by phone or during periodic visits. The feedback we have received from the health workers has been overall very positive.
Ownership of the phones was a strong motivator for the health workers, who recognised the value and usefulness of the devices, so took care to look after them. For most health workers the smartphones quickly became their main phone. We believe that this is because we trusted them with the devices and gave them the flexibility to use the devices for personal use too. Regarding personal usage, all the health workers had customised the phone background image using photos, usually of their family, and many had transferred music onto their phones. In addition to the top-up balance we gave them as part of the project, they also added their own balance to compensate for their personal usage of voice calls, SMS and internet usage.
Many health workers initially complained about the short battery life of the phones - especially when compared to their own standard mobile phones which may stay charged for several days to a week. Although we had given all the health workers solar lamp chargers and, in some cases, dual batteries, is seems few used this as the primary method for keeping the phones charged. Although less than 10% of the health posts were connected to mains electricity, most health workers did have access to mains electricity at home.
Not surprisingly, health workers mentioned that using the protocols took a long time. We knew that it was always likely that the protocol forms would increase the time for a patient encounter. Not necessarily solely due to the technology, but also because we were asking them to ask quite a comprehensive set of questions and a physical examination. Previously, without the electronic protocols, the patient encounters may not have been as thorough. From the start/end times, automatically logged by the phones, we could identify how long health workers spent for each patient encounter. For an ante-natal care first encounter the average time for the patient encounter was 20 minutes. In our visits, the health workers and mothers seemed very comfortable with using the protocols, as it checked that all the right questions were being asked during the patient encounter and the results are being published in a forthcoming paper.
The health workers seemed most comfortable using the Tigrinya versions of the protocols. Health workers could switch between English and Tigrinya and were free to enter text data in either Latin or Ge’ez script, although very few questions required any text input. They appreciated the flexibility allowing them to view and enter data in either English or Tigrinya and the fact they could use the Ethiopian calendar for entering dates.
Health workers appreciated that the changes made were based on their suggestions for improvements. They seemed keen to see us using the same system for other aspects of their work, for example integrated management of childhood illnesses, tuberculosis, immunisations and stock control.
Discussion
Introducing new technologies in environments such as primary health care in low-income countries will always be challenging. mHealth applications are complex interventions that require changes to the behavior of health care professionals who will use them and changes to systems or processes in delivery of care [25,26,26,27]. One of the key reasons that this project had significant activity levels and low device loss rate is because of the sense of ownership and intimacy the health workers developed to their smartphones. Because we did not restrict the usage of the mobile devices only to their professional work, the health workers appreciated how useful the smartphones can also be in their personal lives. Health workers were confident in experimenting with the devices, for example using the camera, loading music, accessing the internet and some have even created Facebook accounts - despite no training or advice on how to do this. For almost all the participants this was the first opportunity they have had to access the internet or use a powerful computing device [27,28,28,29].
We agree with other authors that in the context of limited resources, implementing untested mHealth interventions at scale without a proven theory of behaviour change is likely to result in many failed scale up projects and significant levels of wasted resources [26,27]. Because our health workers were able to develop this sense of ownership and empowerment, they were willing to take good care of the devices, for example, ensuring that they were always charged up, both with power and top-up credit. With the devices powered up with top-up balance and always to hand, and so always available for their professional work, rather than being left in a drawer somewhere, then not being charged up or with credit when they actually come to use them. They clearly developed a connection with and a motivation to use the phones that we may not see with other types of devices, e.g. a PC, and have no evidence that the health workers were abusing the lack of restrictions on personal use [20,29,30].
Inclusion of the health workers in the development of the protocols, as well as localisation into their local language (Tigrinya) and allowing dates to be entered and displayed in the Ethiopian calendar, also contributed to the sense of ownership. There is already good evidence that the use of standardised protocols can improve the accuracy and effectiveness of routine diagnosis and treatment, although there remains the need for developing a better mechanism of delivery of protocols to enable widespread use, and the use of mobile devices could bridge this gap [30,31,31,32,11].
The final aspect we felt increased the sense of ownership was the implementation of the mobile scorecard and a complete cycle of data. We wanted to ensure the health workers had a reason to enter the data, and they could see that it could help improve their professional work and managing their time for appointments, rather than purely collecting data on their performance, or for health management statistical information [32,33].
Our approach to unrestricted use varies somewhat from previous mHealth projects [31,32]. Trying to restrict usage of the device to purely professional, for example by locking down or uninstalling applications, could be a mistake in terms of getting engagement from the health workers. Similarly, we started from the point of view that we would provide the mobile devices. Relying on the low-end phones already owned by the health workers, we feel, would have severely restricted what was achievable, as we would always be working with the lowest common denominator of technology available. Since the health workers are already poorly paid, expecting them to provide their own devices for their professional work is expecting too much. In addition, other projects [33,34] have experienced issues with providing technical support when using the phones already owned by health workers and so had to change to providing devices.
The analytics dashboard was an extremely valuable tool for monitoring, giving us real time information on the encounters being made by the health workers. This allowed us to very quickly identify when there may be problems or issues, either with the mobile internet connection or the actual data entered. From the analytics dashboard data, we are able to identify a number of training issues. This has led us to develop a mobile learning application to assist in resolving these training issues. The mobile learning application uses content from the Ethiopian FMoH approved HEW upgrade training programme [HEAT reference, 35], supplemented with video and assessment exercises and can all be run offline on the phones. This application is under active development and we are in the process of trialling the application with a larger group of health workers. Further information about the mobile learning application can be found in appendix S3.
Although the usage level by some of the health workers was low (see Figure 6), especially compared to the midwives, possibly as a result of better training, higher professional status, higher wages, we can see that all the health workers quickly learn how to use the devices and protocol forms with a minimal amount of training, plus the technical infrastructure is good enough to run these types of systems. We could see that the health workers are able to enter data with a manageable amount of errors, as described [36], most of which could be resolved with a little extra training, supervision and practice.
Regarding the software side of technical system, the ability to get a good overview in real time of the activity of the health workers was especially valuable. Given that ODK is not designed to support longitudinal information and the issues we have had regarding patient identification, some aspects of this technical implementation may not be suitable for scale up. Rather, a cut down version of an electronic medical records system (EMRS), a mobile lightweight EMRS, would perhaps be more appropriate, especially one which not only just stored a subset of patient records, but also some of the key performance indicators and assisted the health workers with managing appointments, risk factors, referrals and scheduling.
Currently, it does not appear that a full EMRS system would either be required or could be successfully implemented. Development of open standards for all these systems is going to be particularly critical for equity in e- and mHealth. There is a need for a governing body to certify open standards and enable countries’ access to standards that meet criteria [26,25,37]. However, a full EMRS system [36,38] would not fix some of the underlying issues of the health system organization, such as improving the supervision of health workers, professional career development, staff turnover and the coordination and referrals between health posts and health centres.
Improved patient identification would be key to a very successful light-EMRS or full-EMRS. There are already some efforts in Ethiopia to resolve this identification issue, for example the national Health Management Information System or Family Folder system, but these aren’t fully rolled out to all the health posts we are working with, so we weren’t able to take advantage of these. A patient identification system which included using some form of check digit would seem particularly advantageous. Alternative biometric approaches, such as those currently being deployed in India [39], although valuable, may introduce technology dependencies and costs which may not be currently feasible in Ethiopia.
Nowadays, smartphones are still more expensive than standard mobile phones, and this has often been cited as a concern regarding scalability [32]. We would anticipate that cheaper and better devices matching the requirements described earlier would be available shortly. Using the phones for multiple applications, such as case management, stock control, mobile learning and assessment, will make them much more cost effective [27]. We have already started looking at which phone models may be a good replacement for the smartphones used in this project. In this regard, Android phones are beginning to be assembled in Ethiopia, so these could be a good alternative option to importing phones. The key factors in the decision of which mobile device to be used would be the following: 1) value for money; 2) runs a recent version of Android platform; 3) the device can be rooted easily, if necessary, to enable the installation of local language fonts; 4) hard-wearing and robust; 5) screen size, especially for delivery of multimedia content.
With the number of health workers included in our project, it has been too early to measure any significant impact on health outcomes. The project and data collection is ongoing, so we will continue to monitor the data for any evidence of this. A key challenge for the mHealth field is the need for better and more evaluation and the next step is that high quality trials should be conducted to establish the effects of clinical diagnosis and management support such as protocols and decision support systems on clinical outcomes using mobile devices. The effects of such support on the management of different diseases and on objective disease outcomes should be evaluated.
Supporting Information
Appendix S1.
Source Code. Information and links for accessing all the source code for the applications and our customisations.
https://doi.org/10.1371/journal.pone.0077563.s001
(DOCX)
Appendix S2.
Demonstration Site. A demonstration site of the analytics dashboard and scorecard applications.
https://doi.org/10.1371/journal.pone.0077563.s002
(DOCX)
Appendix S3.
Mobile Learning Application How to access and install the mobile learning application (OppiaMobile).
https://doi.org/10.1371/journal.pone.0077563.s003
(DOCX)
Acknowledgments
We are grateful to the Tigray Regional Tigray Health Bureau and all health extension workers, midwives and supervisors who took part in the study. Cast-Info sl (Barcelona, Spain) provided us with unrestricted access to their secure server services.
Author Contributions
Conceived and designed the experiments: AL AM MS GD RB. Performed the experiments: AL AM HY RB. Analyzed the data: AL AM MS RB. Contributed reagents/materials/analysis tools: AL AM HY MS GD RB. Wrote the manuscript: AL AM RB.
References
- 1. Aker JC, Mbiti IM (2010) Mobile Phones and Economic Development in Africa. J Econ Perspect, 24(3): 207–232. doi:https://doi.org/10.1257/jep.24.3.207.
- 2. Lester RT, Ritvo P, Mills EJ, Kariri A, Karanja S, Chung MH (2010) Effects of a mobile phone short message service on antiretroviral treatment adherence in Kenya (WelTel Kenya1): a randomised trial. Lancet, 376(9755): 1838-1845. doi:https://doi.org/10.1016/S0140-6736(10)61997-6. PubMed: 21071074.
- 3. Free C, Phillips G, Watson L, Galli L, Felix L et al. (2013) The effectiveness of M-health technologies to improve health care service delivery processes: a systematic review and meta-analysis. PLOS Med 10 (1) : e1001363. doi:https://doi.org/10.1371/journal. pmed.1001363.
- 4. Gurman TA, Rubin SE, Roess AA (2012) Effectiveness of mHealth behavior change communication interventions in developing countries: a systematic review of the literature. J Health Commun 17: 82–104(sup1), 82-104.) doi:https://doi.org/10.1080/10810730.2011.649160. PubMed: 22548603.
- 5.
VitalWave Consulting (2011) mHealth in Ethiopia. Strategies for a new framework. Available: http://www.vitalwaveconsulting.com/pdf/2011/mHealth. Accessed: 3 August 2013.
- 6.
mHealth Alliance, Vital Wave Consulting (2013) Sustainable Financing for Mobile Health (mHealth): Options and Opportunities for mHealth financial models in low and middle-income countries. Available at http://www.mhealthalliance.org. Accessed: 3 August 2013.
- 7.
World Health Organization Global Observatory for eHealth (2011) New horizons for health through mobile technologies. Available: http://whqlibdoc.who.int/publications. Accessed 3 August 2013.
- 8. Collins F (2012) How to fulfill the true promise of “mHealth”: Mobile devices have the potential to become powerful medical tools. Sci Am 307 (1): 16. doi:https://doi.org/10.1038/scientificamerican0712-16. PubMed: 22779258.
- 9.
Mechael PN, Batavia H, Kaonga N, Searle S, Kwan A (2010) Barriers and Gaps Affecting mHealth in Low and Middle Income Countries: Policy White Paper. Center for Global Health and Economic Development, Earth Institute, Columbia University.
- 10. Zurovac D, Sudoi RK, Akhwale WS, Ndiritu M, Hamer DH et al. (2011) The effect of mobile phone text-message reminders on Kenyan health workers' adherence to malaria treatment guidelines: a cluster randomised trial. Lancet 378 (9793): 795–803. doi:https://doi.org/10.1016/S0140-6736(11)60783-6. PubMed: 21820166.
- 11. Tomlinson M, Solomon W, Singh Y, Doherty T, Chopra M et al. (2009) The use of mobile phones as a data collection tool: a report from a household survey in South Africa. BMC Med Inform Decis Mak 9: 51. doi:https://doi.org/10.1186/1472-6947-9-51. PubMed: 20030813.
- 12. Darmstadt GL, Bhutta ZA, Cousens S, Adam T, Walker N et al. (2005) Evidence-based, cost-effective interventions: how many newborn babies can we save? Lancet 365(9463): 977-988. doi:https://doi.org/10.1016/S0140-6736(05)71088-6. PubMed: 15767001.
- 13.
Central Statistical Agency (2011) [Ethiopia] and ICF International (2012) Ethiopia Demographic and Health Survey. Addis Ababa, Ethiopia and Calverton, MD, USA: Central Publishing House Statistical Agency and ICF International. Available: http://ethiopia.usaid.gov/sites/default/files/Demographic%20Health%20Survey%202011%20Ethiopia%20Final%20Report.pdf. Accessed 2 August 2013.
- 14.
Natons United. (2009) The Millennium Development Goals Report. Available: . http://www.un.org/millenniumgoals/pdf/MDG_Report. Accessed on the 3rd august.
- 15. Medhanyie A, Spigt M, Dinant G, Blanco R (2012) Knowledge and performance of the Ethiopian healh extension workers on antenatal and delivery care: a cross-sectional study. Hum Resour Health.10 (1): 44. doi:https://doi.org/10.1186/1478-4491-10-44. PubMed: 23171076.
- 16. Medhanyie A, Spigt M, Kifle Y, Schaay N, Sanders D et al. (2012) The role of health extension workers in improving utilization of maternal health services in rural areas in Ethiopia: a cross sectional study. BMC Health Serv Res12: 352. doi:https://doi.org/10.1186/1472-6963-12-352. PubMed: 23043288.
- 17. Lemay NV, Sullivan T, Jumbe B, Perry CP (2012) Reaching remote health workers in Malawi: baseline assessment of a pilot mHealth intervention. J Health Commun; 17 Suppl 1: 105-117. doi:https://doi.org/10.1080/10810730.2011.649106. PubMed: 22548604.
- 18. Swider SM (2002) Outcome effectiveness of community health workers: an integrative literature review. Public Health Nurs.19(1): 11–20. doi:https://doi.org/10.1046/j.1525-1446.2002.19003.x. PubMed: 11841678.
- 19.
Earth Institute (2010) Barriers and Gaps Affecting mHealth in Low and Middle Income Countries: A Policy White Paper. Washington, DC: mHealth Alliance. Available: http://cghed.ei.columbia.edu/sitefiles/file/mHealthBarriersWhitePaperFINAL.pdf.
- 20. Rowe AK, de Savigny D, Lanata CF, Victora CG (2005) How can we achieve and maintain high-quality performance of health workers in low-resource settings? Lancet 366(9490): 1026-1035. doi:https://doi.org/10.1016/S0140-6736(05)67028-6. PubMed: 16168785.
- 21.
MHealth Education consortium (2012). Available: http://www.mhealthed.org/iheed_report_updates.pdf. Accessed 3 August 2013.
- 22. Florez-Arango JF, Iyengar MS, Dunn K, Zhang J (2011) Performance factors of mobile rich media job aids for community health workers. J Am Med Inform Assoc 18(2): 131-137. doi:https://doi.org/10.1136/jamia.2010.010025. PubMed: 21292702.
- 23. Van Heerden A, Tomlinson M, Swartz L (2012) Point of care in your pocket: a research agenda for the field of m-health.Bull World Health Organ 90(5): 393-394. doi:https://doi.org/10.2471/BLT.11.099788. PubMed: 22589575.
- 24. Lund S, Hemed M, Nielsen BB, Said A, Said K et al. (2012) Mobile phones as a health communication tool to improve skilled attendance at delivery in Zanzibar: a cluster-randomised controlled trial. BJOG. 119(10): 1256-1264. doi:https://doi.org/10.1111/j.1471-0528.2012.03413.x. PubMed: 22805598.
- 25.
Health Education and Training. (HEAT) programme. Available at . http://www.open.ac.uk/africa/heat. Accessed: 3 August 2013.
- 26.
The PLOS Medicine (2013) A reality checkpoint for mobile health: three challenges to overcome. PLoS; Med (2):e1001395. 10. doi:https://doi.org/10.1371/journal.pmed.1001395.
- 27. Tomlinson M, Rotheram-Borus MJ, Swartz L, Tsai AC (2013) Scaling up mHealth: where is the evidence?PLOS Med 10(2):e1001382. doi:https://doi.org/10.1371/journal.pmed.1001382. PubMed: 23424286.
- 28. Aboud FE, Singla DR (2012) Challenges to changing health behaviours in developing countries: a critical overview. Soc Sci Med 75: 589–594. doi:https://doi.org/10.1016/j.socscimed.2012.04.009. PubMed: 22633158.
- 29.
Fishbein M, Bandura A, Triandis HC, Kanfer FH, Becker M et al. (2000) Factors influencing behavior and behavior change. In: AS BaumT. RevensonJ. Singer. Handbook of Health Psychology. New Jersey: Lawrence Earlbaum Associates.
- 30. Walshe K (2007) Understanding what works and why in quality improvement. The need for theory driven evaluation. Int J Qual Health Care 19: 57–59. doi:https://doi.org/10.1093/intqhc/mzm004. PubMed: 17337518.
- 31.
Rotheram-Borus MJ, Tomlinson M, Swendeman D, Lee A, Jones E (2012) Standardized functions for smart-phone applications: examples from maternal and child health. Int J Telemed App. 2012: 973237 doi:https://doi.org/10.1155/2012/973237.
- 32. Free C, Phillips G, Galli L, Watson L, Felix L et al. (2013) The effectiveness of mobile-health technology based health behaviour change or disease management interventions for health care consumers: a systematic review. PLOS Med 10 (1): e1001362. doi:https://doi.org/10.1371/journal. pmed.1001362.
- 33. Were MC, Shen C, Bwana M, Emenyonu N, Musinguzi N et al. (2010) Creation and evaluation of EMR-based paper clinical summaries to support HIV-care in Uganda, Africa.Int J Med Inform 79(2): 90-96. doi:https://doi.org/10.1016/j.ijmedinf.2009.11.006. PubMed: 20036193.
- 34.
Mechael P, Dodowa . Health Research Center for The Grameen Foundation (2009); MoTECHMHealth Ethnography Report. Washington, DC, USA: Grameen Foundation. Available:http://www.healthunbound.org/sites/default/files/Grameen_Foundation_FinalReport3.pdf. Accessed 3 August 2013.
- 35. Lane SJ, Heddle NM, Arnold E, Walker I (2006) A review of randomized controlled trials comparing the effectiveness of hand held computers with paper methods for data collection. BMC Med Inform Decis Mak 6: 23. doi:https://doi.org/10.1186/1472-6947-6-23. PubMed: 16737535.
- 36. Amoroso CL, Akimana B, Wise B, Fraser HS (2010) Using electronic medical records for HIV care in rural Rwanda. Stud Health Technol Inform;160(1): 337-341) PubMed: 20841704.
- 37. Estrin D, Sim I (2010) Health care delivery. Open mHealth architecture: an engine for health care innovation. Science 330 (6005): 759–760. doi:https://doi.org/10.1126/science.1196187. PubMed: 21051617.
- 38. Tierney WM, Achieng M, Baker E, Bell A, Biondich P et al. (2010) Experience implementing electronic health records in three East African countries. Stud Health Technol Inform;160 (1): 371-375. PubMed: 20841711.
- 39. Nilekani N (2011) Building a foundation for better health: the role of the Aadhaar number. Natl Med J India 24 (3): 133-135. PubMed: 21786839.