Overview of mobile health applications

Over the last few years, we have observed an excessive evolution of mobile health applications than any other innovation in health care [42]. The term ‘app’ is an abbreviation of the word ‘application’ that refers to a program that has been developed for a specific purpose and is generally configured to run on handheld devices like cell phones, tablets, computers, and certain wearable devices such as smartwatches [43]. In recent times, we have observed a proliferation of mobile apps that are mostly created for only two of the most important tech giants of smartphone operating systems- Apple IOS and Google Android. The World Health Organization (WHO) defines these applications as, ‘medical and public health practice supported by mobile devices such as mobile phones, patient monitoring devices, personal digital assistants (PDAs) and other wireless devices’. Similarly, according to Hamel et al. [44] mHealth ‘is the use of portable devices such as smartphones and tablets to improve health’. mHealth apps in practice run on mobile devices with remote network connectivity and dynamic execution of contexts [45]. Context is any information that can be used to characterize the situation of an entity, being either a person, place or object [46]. The context awareness system collects real-time data from patients in a comprehensive manner and presents them to health care professionals to manage their tasks in order to increase the quality of patient care [47, 48].

Extant literature on mHealth applications exhibits how mHealth apps are commonly used for health education, disease self-management, remote monitoring of patients, and collection of dietary data [49, 50]. The information collected through mHealth apps can provide useful support for encouraging health behaviour change, chronic disease self-care and effective management of many conditions while keeping healthcare providers informed of the patient’s condition [50]. This also strengthens the relationship between the patient and the provider [51]. According to Scoping review findings, mHealth apps provide the potential for general practitioners to take medical history and make diagnoses, perform some physical examinations, aid in clinical decision making, and manage long term disease-specific care and promote general health well-being [52]. Whilst traditionally, health care is delivered to individuals through face to face interaction with health care professionals, technology advancement has facilitated communication between patients and health care professionals [53]. Moreover, mobile phone apps are commonly incorporated into the design of health promotion programs both face-to-face and online [54]. Based on these, mHealth apps are mainly categorised into two categories—lifestyle management (such as fitness, lifestyle modification, diet and nutrition) and chronic disease management apps (such as mental health, diabetes, and cardiovascular diseases). The other categories include- self-diagnosis, meditational re-minders, and patient portal apps [43].

Studies document how mHealth apps provide information and health behaviour interventions at a meagre cost, that is, easy to access and personalized for user-specific needs [55]. A systematic review of mHealth apps for chronic diseases showed that there are improvements in patient lifestyle related to, healthier eating, weight loss, controlled blood pressure and glycemic levels, including treatment awareness options among patients [21]. With mHealth apps, users can receive important information through push notifications and can monitor their health status at a personal level. Litman and his colleagues [56] from their mobile exercise App research found that exercise app users are more likely to exercise during their leisure time as compared to those who do not use exercise apps. They also highlighted that exercise apps may potentially increase exercise levels and health outcomes such as BMI levels. However, it is also noted that mHealth apps also pose some limitations, and users may face a life-threatening situation if the mHealth apps are not switched to use and made available all the time. In addition, private and sensitive communications are exchanged between the user and the health care provider through these apps. Hence, usability studies should be sufficiently evaluated to determine the practical way in assessing mHealth apps usability [57]. This is critical as any app developer can develop and market a mHealth app through the app store without conducting any usability test [58]. Thus, studies emphasize the importance of a regulatory body to assess the accuracy, quality and performance of health apps, as some mHealth apps were hacked, leading to patient data breaches [43]. Moreover, without any appropriate security measures and processes in place, mHealth app users may become exposed to the severe repercussions of information security and privacy infringements [59]. Hence, we note that there are no general international standards that a data and privacy policy should offer users. This further suggests that, information on how privacy policies determine app star rating is relatively absent. Many privacy policies do not focus on the security features related to the app and therefore do not create confidence in usage for users. In most cases, users may be ignorant and are more likely to use apps which have ambiguous privacy policies [60].

Fig 1(A) shows number of mHealth apps uploaded in the app store each year; Smart phone penetration in the last five years is illustrated in Fig 1(B). The pie chart shows various categories of apps in terms of percentage (See Fig 1(C))

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. (A) Year-wise number of mHealth apps uploaded in the google play, and (B) Smart phone penetration in the last five years, and (C) Pie chart shows various categories of apps in percentage.

https://doi.org/10.1371/journal.pone.0258050.g001

We have found five scales, such as THESIS [42], MARS [61, 62], Brief DISCERN [63], uMARS [64] and ORCHA-24 [65], THESIS [42] is an open-source application rating tool that is performed using a Delphi process. The tool can calculate usability, risks, and health app benefits. Authors used a standardized method for selecting chronic disease applications with 4 stars and < 4 stars, then scored them with THESIS for their reliability and internal consistency of the tool. The THESIS testing indicates that applications serving chronic illness patients can perform better, particularly in terms of privacy/safety and interoperability. THESIS guarantees further testing and can lead software and policymakers to continue improving app efficiency, enabling applications to enhance patient results more reliably.

Mobile app rating scale (MARS) [66] is an app rating scale for mHealth that has been developed to measure the reliability and efficiency of mHealth applications. The MARS Scale uses four objective quality scales: “functionality”, “aesthetics”, “engagement” and “information quality”; and one subjective quality scale; which were refined into the “23-item MARS”. The MARS has shown excellent intra-class correlation accuracy and reliability.

The Brief DISCERN score [63] can assess quality of mHealth apps for panic disorder (PD) using factors such as transparency, interactivity, self-help score, and evidence-based content. The Brief DISCERN score showed a number of linear regressions that considerably predicted the content quality and self-help performance.

An end-user version of the MARS, called uMARS [64], proposed to assess the quality of mHealth apps. The uMARS had outstanding internal consistency for all subscales with strong individual alphas. The uMARS are used to accumulate app reviews at 1-, 3-, and 6-month intervals. The uMARS had strong individual alphas for all subscales and had outstanding internal stability.

The Organization for the Review of Care and Health Applications- 24 Question Assessment (ORCHA-24) [65] is National Health Service (NHS) approved scale which can evaluate the quality of applications for Chronic Insomnia Disorder on the Android Google Play store for high quality and low-risk health apps. The ORCHA-21 system performed evaluation using evaluation criterion such as efficiency; engagement; protection and privacy; user interface, and quality of clinical efficacy. Table 1 shows the various mHealth app rating scales found in the literature.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Various mHealth app rating scale.

https://doi.org/10.1371/journal.pone.0258050.t001

None of the rating scales considered, to the best of our knowledge, various characteristics of online user feedback and performed sentiment analysis for mHealth applications, which could then be assigned a new rating, substantially different from the original rating. Thus, the proposed mHealth app rating tool, called ACCU³RATE, uses evaluation criteria such as user star rating, user text review, UI design, functionality, security and privacy, and clinical approval.

Determinants of mobile health app ratings

At present, there is little known about which mHealth apps are most commonly used and whether they align with app quality [67]. Majority of the mHealth apps found in the app stores are available in trial versions to attract potential users or as a promotional strategy to increase the adoption rate [68]. Fiordelli et al. [9] in their systematic review brought to our attention that there are millions of mHealth apps (free and paid) publicly available on app stores, and to-date they have not been evaluated them. Byambasuren et al. [58] highlighted that low quality and lack of effectiveness hinders the usability of mHealth apps, as the majority are developed by non-health care organisations which raises questions about the accuracy and trustworthiness of these apps [69].

While evaluating behavioral mHealth apps, studies report that app ratings are broadly inconsistent and often contradictory while most popular behavioural health apps rating are not favorable [67]. Furthermore, other frequently downloaded behavioral mHealth apps have questionable support in the literature and offer no evidence-based behavioural support. For example, research on the usage of mHealth apps by cancer survivor patients revealed that some apps used theoretical models of behaviour change, but the majority of the apps in the market did not apply any of those theoretical elements in their apps [55].

Some scholars suggest that independent and reliable sources should evaluate mHealth apps and should recommend a collection of trustworthy apps to health care professionals to, be referred to patients [58]. According to Levine et al. [42] an app rating tool can be developed to identify mHealth apps that may cause potential harm and breach the privacy policy. When an appropriate mHealth app is prescribed by health care professionals, they must be confident about the functionality of the app, maintenance of data privacy and its usability [58]. A survey conducted among mHealth app users in the USA found that 82 percent users would change providers if they were aware of alternative apps which were more secure [70]. There is also no clear evidence available to suggest that an app developed by a health care organization will have more usability. Bivji et al. [69] identified that price, user ratings, in-app purchase options and in-app advertisements were the greatest predictors of app downloads from the app store.

Other studies present that the actual usability and quality rating scales are primarily developed only for professionals, and not for end-users who are patients or care providers. It is widely recognized that rating scales that are usable by all end users would make mHealth apps more accessible and meaningful to consumers [71]. Janatkhah et al. [72] found that a mHealth app’s usability is associated with the education level, employment and place of user’s residence. While there is evidence of a weak relationship between user app ratings and usability and clinical utility of the app [73] in the literature, generally we note that users make decisions related to app use, based on the title, price, star ratings, reviews and number of downloads [74]. Furthermore, mHealth users disproportionally favour apps that take advantage of unique features of smartphones with higher ratings and present the real benefits to users to keep track of their health records [45].

Generally, we observe that mHealth app users’ opinions or satisfaction levels, star rating system, app description of checklists are some common features used to measure the quality of the apps. However, scholars argue that none of these methods are scientifically adequate to measure the quality of the apps [75]. Singh et al. [73] identified that store rating alone is not enough to determine whether an app is of high quality but inclusion of clinicians and users’ reviews will ensure that the recommended apps are usable and clinically useful.

Further, we note that in the literature users’ reviews of mHealth apps in relation to app rating is generally scarce. There is researchs on the effect of online assessment (i.e. whether reviews in an examination collection are overwhelmingly positive or negative) on different dependent variables, in particular on the perceived utility of the reviews and attitudes towards a product [76–78]. This operationalization of review effectiveness warrants helpfulness and facilitates user’s decision-making [79] whilst inspiring trust and confidence among other potential users [80–83]. Nevertheless, the existence of a contradictory total rating reduces the credibility of the analysis due to its negative impact on consumer product attributes [84].

Some scholars are sceptical about the review ‘helpfulness’ level related to user votes, and suggest other diverse features, such as basic, stylistic, and semantic characteristics by applying data mining techniques from online user reviews [40]. Although this multi-process conceptualization of multiple determinants for review effectiveness is emerging in the literature, some scholars are of the view that participation in reviews and polarity of reviews may be driven by different consumer behaviors [79] and emotions [85] so should be studied separately. Although it is observed that positive review sentiments increase readership [86].

The basis for digital health research is often primarily based on cataloging the number and types of apps dealing with a specific health issue that assess the theoretical effect of such features on user experience or identifies the behavioral concepts in a subset of mHealth apps. Studies can also explore a variety of multi-dimensional app features that contribute to a better user experience and eventually alter long-term behaviours [1]. Based on these gaps from the mHealth app rating, our study aims to extend current knowledge on how users’ perceptions and app reviews can generate new app rating that may be different than the original/existing app rating. This paper also presents a new methodology, an AI-enabled mHealth app rating scale, which takes multidimensional measures to generate app rating. By adopting text mining as a novel methodology to extract various semantic characteristics from reviewers’ texts, this paper evaluates user review determinants on mHealth app ratings.

mHealth applications

The mHealth applications are an important part of the forthcoming next-generation Internet of Healthcare Things (IoHT) [87]. This paper categorises the mHealth apps into the following six types.

Behavioral change and medication.

Behavioral improvement and medication mHealth apps identify behavioral changes that improve patients health and facilitate personalized medicine (see Fig 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Factors affecting mobile app review.

https://doi.org/10.1371/journal.pone.0258050.g002

Factors affecting mHealth app rating

In determining mHealth app quality scores important factors related to the app such as the clinical approval, UI design, functionality, security and privacy are often ignored by the current practice. The complete list of factors which are considered essential in determining an appropriate rating of the mHealth app are provided below:

ACCU³RATE app rating model

The proposed app rating tool, called ACCU³RATE for mHealth Apps, provides a multidimensional application quality evaluation using criteria such as user star rating, user text review, UI design, functionality, security and privacy, and clinical approval. The app rating tool eliminates the aforementioned variables from the app developer and generates an improved rating based on the user experience. In the conventional rating, the quality of the app is judged only by user stars. However, our research suggests that the declaration of the developer as well as user text review must be incorporated. The relationship between input parameters and the model parameters is shown in Fig 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Relationship between input parameters and model parameters.

https://doi.org/10.1371/journal.pone.0258050.g003

Developers declaration

An app developer declares the mHealth app criteria such as clinical approval, standardization certification, functionality, UI design, security and privacy. The app developer’s response calculates the initial rating (initial ∈ [0, 5]). Algorithm 1 reflects the complete procedure from calculating the initial rating to changing the rating based on user feedback. At the time of the publication of Apps, the initial rating for an app is created. Based on the user experience (star rating and review), this rating changes. The feature “users recalculation” returns a positive or negative value depending on the polarity of user comments in the Algorithm 1.

Algorithm 1: Initial rating based on developer’s feedback

Result: R_Base

total_topic, R_Base = 0, R[total_topic], counter[total_topic]

if FDA_Approval_Req_Dev_Response() == True then

 total_topic = 5;

else

 total_topic = 4;

while r = 1 to total_topic do

 R[r] = developer_ans_on_topic_r();     // Initial Rating based on developer response, 0 or (5/total_topic)

 R_Base += R[r];

end

Publish(R_Base);

User rating and review

The key variables considered in this paper are discussed as follows.

Text mining and sentiment analysis.

The polarity of user reviews in the mHealth app can be extracted using the following steps (see Fig 4).

1. Step 1: Selection of mHealth apps: The mHealth applications are selected from the Google Play and App Store for a specific category (such as health-related apps).
2. Step 2: Collection of comments: Based on the assumption that the user star rating, user review, number of downloaded features, and app features can influence the app’s reputation, the aforementioned data for the selected apps is accumulated. NLTK (Natural Language Toolkit) [93], a versatile package of natural language processing algorithms, analyzes the user’s critique/comments. All comments from the app review segment are gathered using scrubbing software.
3. Step 3: Extraction of Keywords: Keywords are extracted from a customer’s review that contributes to the app’s credibility. We break down the analysis of comments into small bits like words and use tokenization techniques, where word tokenization splits the entire text into words. The frequency that relates to the number of each word in the method is then transmitted and counted. In text data, mining stopwords (i.e., am, are, this, a, an, etc.) is viewed as noise. To delete all of these, we created a stopword list and filtered the tokenized word list with it. After tokenization, the words go through stemming, i.e. a normalization process where the term is shortened to the root word or a derived affix is trimmed.
4. Step 4: Identification of polarity: The polarity of the comments is defined based on sentiment analysis [94]. We use the clinical ontology and lexicon-based approach to evaluate emotions from a written text to count positive and negative terms among the review comments. The classification of comments is one of the most important aspects in the text mining industry. We created a data dictionary to evaluate our dataset and, with the aid of the word2vec algorithm [95], and then we fit the data dictionary with the comment data. Using the NLTK text corpora and word2vec algorithm, these review comments are analyzed to find any constructive negative review comments.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Flow diagram for opinion mining.

https://doi.org/10.1371/journal.pone.0258050.g004

We then use the Naive Bayes classifier [96, 97] to evaluate the positive and negative polarities of the comments. We use two environments, NLTK [98] and Spider [99] for sentiment analysis. Word charts in Fig 5 shows the wordcloud for positive and negative polarities of words found in the comments of the app.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Word cloud for Polarity of words found in the comments of selected apps– (A) positive polarity words, and (B) negative polarity words.

https://doi.org/10.1371/journal.pone.0258050.g005

Approval and certification

mHealth technologies originated mainly from non-medical practice. As a consequence, such applications can lead to poor content, poor treatment and unfavorable outcomes. Inappropriate use of apps by patients and clinicians could lead to a worsening of health conditions if there is no evidence-based guidelines for app growth. Thus, clinical approval is necessary for mHealth applications especially if the app is used for monitoring of clinical data, provide diagnosis, or intervention for a medical condition.

Strict protocols and criteria are essential for ensuring the quality of an app. The mHealth software developer should have an understanding of the multidisciplinary standards such as ISO 14971:2007 [100] and ISO 14971:2019 [101]. Any mHealth app with clinical approval and ISO certification improves the credibility of the app.

Fig 6 shows calculation of “Clinical Approval” value using four questionnaires such as

• Q1: Is the app monitored or analyzed patient data or patient specific medical device data/function?
• Q2: Is the app connected to control the operation, function or energy source of a medical devices?
• Q3: Is the app connected to medical device that assists licensed practitioner to diagnose/treat a medical condition?
• Q4: Is the app followed any International Standard and received Certificate?

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Calculation of clinical approval value.

https://doi.org/10.1371/journal.pone.0258050.g006

Fuzzy logic for fusion

The proposed App rating tool ACCU³RATE uses a fuzzy logic controller (FLC) which generates a rating score using criteria such as user star rating, user text review, UI design, functionality, security and privacy, and clinical approval. The FLC is a rule-based system used extensively to design decision-support systems [102]. Fig 7 shows the FLC which combines the knowledge extracted from the user star rating, user text review, clinical approval, UI design, functionality, security and privacy, and thereby generates an App rating. The FLC has mainly three layers; these are fuzzification, rule base and inference system and defuzzification [103].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Fuzzy logic based fusion technique which combines the knowledge extracted from the users’ star rating, users’ text review, clinical approval, UI design, functionality and security & privacy and thereby generates a score.

https://doi.org/10.1371/journal.pone.0258050.g007

The fuzzification layer converts the input variables into fuzzy forms using linguistic terms and Gaussian membership functions that convert the crisp inputs into fuzzy inputs. We have chosen three linguistic terms (Low, Medium, and High) for Users’ Star Rating (USR); and two linguistic terms (Low and High) for users’ text review (UTR), clinical approval (CliA), UI design (UI), functionality (Func), and Security and Privacy (S&P).

The rule layer includes 62 “IF … THEN” rules for the system and a fast forward rule selection algorithm is used to select optimal rules for improving the FLC’s accuracy and computation. All the rules are represented by a heatmap in Fig 7

After the fuzzy inferencing, all the consequents are integrated by multiplying a weight value. The defuzzification layer converts the fuzzy value into an average value. In this case, the center of the area, the defuzzification method is employed for its simplicity.

Fig 7 also shows the input and output membership functions, and the surf plots refer to the relationship between the input and output variables. Fig 8 shows the input membership and the relationship between the input-output in the rule base system.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Input membership and relation between the input-output in the rule base system.

https://doi.org/10.1371/journal.pone.0258050.g008

Tool and rating correlation

Cronbach alpha is used to measure interrelated items to estimate quality score (i.e., scale reliability) and internal consistency of the scale. On the other hand, the interrater reliability index, called Cohen’s kappa, determines the extent to which two raters’s ratings agree or disagree with each other for qualitative objects.

Data collection

The data collection phase started with selecting appropriate apps to provide the proof-of-concept. The apps were searched using the keywords heart disease, heart related medical app, healthy heart, and heart care in the Google Play Store and Apple store to evaluate the proposed AI-based app rating scale. A total of 317 applications in different categories have been identified based on the search criteria. Each app was initially qualitatively assessed and screened with titles, descriptions, and associated screenshots provided in the Play Store. Out of the total 317 m-Heath apps, 278 are from the Google Play Store while, 39 are from the Apple Store. Then, we set some exclusion criteria for apps, such as duplicate apps, apps not related to heart disease, apps downloaded less than 5000 times, game and simulation apps, apps not updated in more than 12 months, and non-English apps. Consequently, after filtering based on all these criteria, we selected 43 apps (23 apps from Google Play Store and 20 apps from the Apple Store) for further analysis. Fig 9 shows the case study design and app selection flowchart.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Case study design and app selection flowchart.

https://doi.org/10.1371/journal.pone.0258050.g009

Correlation analysis

The proposed mHealth app rating tool provides a multidimensional assessment of the App interaction, functionality, aesthetics, subjective quality, security and privacy.

Fig 10 illustrates the correlation coefficient of selected parameters such as user experience (user star rating and text review), approval and certification, UI design, security and privacy. The correlation coefficient (r) between the variable is considered for the app scale assessment. User experience has found to have influence on functionality (r = 0.5), privacy and security (r = 0.2). The UI design also affects star rating (r = 0.1) and text review (r = 0.1). App functionality is correlated with user star rating (r = 0.5) and text review (r = 0.5). The fuzzy logic based mHealth app evaluation framework has been employed to produce app ratings based on user star rating, user text review, clinical approval, UI design, functionality, and security and privacy. From the initially selected 317 apps, our filtering categories (such as consistency, user context, aesthetics, ethical issues, privacy, security, connectivity, alertness, ease of use, record and clinical approval) informed the final selection of 43 apps. The input factors, proposed in this paper, include various model parameters such as users’ star rating, users’ text review, clinical approval, UI design, functionality, and security and privacy (See Fig 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Correlation matrix illustrating the coefficient among factors considered for the assessment of the app scale.

https://doi.org/10.1371/journal.pone.0258050.g010

Fig 11(A) shows the average score for the different parameters, while, Fig 11(B) presents the box plot for the average rated value for conventional app rating scores, fuzzy based scores and app scores based on expert opinion. It has been found that the fuzzy based rating has a high variance compared to the conventional app rating, whereas the fuzzy based rating shows a high relationship in contrast to scoring based on expert opinion (p = 0.84). Fig 11(C) demonstrates app scores for conventional and fuzzy-based ratings for all the apps chosen. It has been noted that fuzzy based rating decreases the rating value if the selected input parameters have not been found in these 43 apps.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. Comparison of traditional mobile app ratings and fuzzy based app ratings.

(A) Average scores with respect to each parameter for all the selected 43 apps, (B) Box plot for the average rated value for traditional, fuzzy based ratings, as well as ratings based on expert opinion, (C) app scores for traditional and fuzzy based ratings for all the selected apps, and (D) Comparison of various app scales in terms of ICC.

https://doi.org/10.1371/journal.pone.0258050.g011

It has been found that the Cohen’s kappa value varied as: approval and certification (0.44), functionality (0.41), UI design (0.50), security and privacy (0.55). On the other hand, the Cronbach alpha value is found to be 0.86 which shows the ACCU³RATE has an acceptable degree of reliability of scale and internal consistency.

When the proposed app rating scale ACCU³RATE is compared with the THESIS, ORCHA-24, uMARS, and MARS in terms of the ICC on 43 apps (23 apps from the Google Play Store and 20 apps from the Apple Store), it is found that the factors in the same domain have 2-way mixed ICC = .86 (95% CI 0.81–0.92) in contrast to the app scales THESIS: 0.85; ORCHA-24: 0.79; uMARS: 0.90; MARS: 0.88 (See Fig 11(D))

The high value of ICC (ICC = .86) shows components under the same criteria in the proposed app scale resemble one another very closely, implying that the ACCU³RATE app scale is very reliable.