Background

The United States has one of the lowest tuberculosis (TB) case rates in the world, but there is still considerable progress to be made before TB can be eliminated from the US, with over 9,000 cases reported in 2017 [1]. One of the challenges continues to be considerable geographical heterogeneity; for instance, the TB case rate in Los Angeles County (LAC) (5.7 cases per 100,000 population) is significantly higher than the U.S. national average (2.9 cases per 100,000 population). In 2016, there were 550 confirmed TB cases in LAC [2]. LAC had the highest number of TB cases and the 10^th highest TB incidence rate among California’s 61 health jurisdictions. We therefore focus on LAC in the present study on technology to improve TB treatment.

Directly observed therapy (DOT) is the preferred strategy for active tuberculosis treatment. DOT is defined as an in person, direct observation by a health care worker of the patient ingesting each dose of medication. DOT directs partial responsibility of treatment to the provider and helps ensure that patients complete an adequate course of TB treatment [3]. While this is an effective method for ensuring TB treatment adherence– 92.6% of TB patients in LAC completed treatment within a year in 2013 –DOT is resource-intensive and requires a high number of nurse-hours per TB patient [4]. Although some DOT sessions are conducted during patients’ clinic visits, in LAC, DOT is mostly carried out in the field, whereby nurses visit TB patients at home to directly observe ingestion of TB drugs which in turn incurs travel expenses and additional time to coordinate home visits [5].

A common alternative to DOT is video directly observed therapy (VDOT), where healthcare professionals observe patients take their medications through a video conferencing software. Many recent studies find success and/or adherence rates with VDOT to be comparable to in-person DOT [6–14]. Some studies have also investigated VDOT costs to the healthcare system in addition to adherence outcomes. Buchman and Cabello (2017) studied traditional DOT vs Skype observed therapy (SOT) in 24 patients from Nassau County, New York [15]. They found cost savings under SOT of $1,008 per-month and $42 per-case per-month. Holzman, Zenilman, and Shah (2018) conducted a prospective pilot study of a VDOT intervention called miDOT (emocha Mobile Health Inc.) in three tuberculosis clinics in Maryland [7]. They followed 28 adult patients receiving active or latent TB treatment. For a standard 6-month treatment, VDOT saved $1,391 per-patient compared to DOT and adherence rates of 98% and 94% for DOT and VDOT, respectively (P = 0.17). Garfein et al. (2018) studied asynchronous VDOT—videos of medication ingestion are recorded by the patient and watched by a health care worker at a later time—versus traditional DOT in five health districts in California (San Diego, San Francisco, Santa Clara, San Joaquin, and Imperial) [8]. They enrolled a total of 174 adults to VDOT and compared outcomes to 159 historical controls treated with DOT. The 6-month treatment cost with VDOT (range $3,031-$3,911) was 32% (range 6%-46%) less than DOT ($3,212-$5,788) across districts.

Recent technological advances have introduced further innovations in this area–artificial intelligence (AI)-based software that uses computer vision and machine learning to confirm the patient, the drugs, and ingestion in real time, further reducing health worker burden. However, there is still limited data on the efficacy of AI-based monitoring methods for TB treatment adherence, and no prior studies to our knowledge have evaluated its cost-effectiveness. In this study, we examine both for a cohort of pulmonary TB patients tracked by the Los Angeles County Department of Public Health (LACDPH) using AiCure (New York, NY), an AI platform that uses computer vision and machine learning to automatically confirm the patient, the drugs, and ingestion in real time. Under AiCure, failure in any one of these criteria leads to an email or SMS text message alert sent to a nurse (see Fig 1 for illustration) who then follows up with the patient in real time. Patients can access AiCure using a smart phone application which uses the device camera to record medication consumption; the platform includes the option to have the video sent to nurses to review later. This platform is currently used in drug development trials and population health settings, across multiple therapeutic areas including neuroscience, cardiovascular, and infectious disease [16–18]. Additional details on AiCure are provided in the online S1 File. By focusing on LAC, a large metropolitan area with high TB morbidity and substantial patient heterogeneity, we hope to provide insight into the cost-effectiveness of AI-based treatment monitoring platforms for a critical TB population in the U.S.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. AiCure artificial intelligence platform.

Software automatically observes and confirms patient, medication, and ingestion. HIPAA Health Insurance Portability and Accountability Act. Source: AiCure, LLC (New York, NY) with written permission to use and adapt.

https://doi.org/10.1371/journal.pone.0254950.g001

Methods

Data

Patients and nurses were trained to use the AiCure software in a pilot study of 43 active TB patients in Pacoima, Los Angeles, CA. In this pilot study, eligible patients were provided with an internet and camera-enabled smart phone and trained to record their doses using the software. This data has not been published elsewhere, and we present it here for the first time. Our final analysis sample for the AiCure arm of our model included 43 patients confirmed during 2015 to 2017 who either began and completed or began and moved/defaulted from active TB treatment. Mean age at diagnosis in this sample was 48.4 years, with 21% of patients having multiple comorbidities, 21% with diabetes, and 76% with pulmonary TB disease (Table 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Summary statistics by treatment arm.

https://doi.org/10.1371/journal.pone.0254950.t001

We compared these patients with a cohort of patients with active TB treated with DOT in the same clinic, confirmed between 2013 and 2014 (n = 172), the years immediately preceding the AiCure pilot. To best approximate the population that would have been eligible for AiCure, we included only patients who clinicians determined to be eligible for AiCure treatment. Our control final sample included 71 patients previously treated for active TB under DOT. Average age at diagnosis in this sample was 50.9, with 35% having multiple comorbidities, including 28% diabetic and 79% with pulmonary disease (Table 1). We calculated P-values using two-sided, two-sample t-tests for proportions or means with unequal variances. No demographic factors including age, gender, or race were statistically significantly different across treatment arms. Although not selected for, the demographic characteristics and proportion of patients with comorbidities in each treatment arm were similar. In both arms most patients had no behavioral risk factors and no major morbidities other than pulmonary complications.

Costs

For both treatment arms, we calculated drug acquisition costs to be $174.16 per-patient per-month, assuming treatment with the standard regimen of isoniazid (INH), rifampin (RIF), pyrazinamide (PZF), and ethambutol (EMB) [3, 19]. AiCure-specific fees include licensing for $750 per clinic per month up to 50 patients, $1,500 per clinic per month for technical support, a fixed cost of $52 per patient for a phone, and $47.50 per patient per month for phone service. We estimated the remaining costs in the model using Licensed Vocational Nurse (LVN) or Registered Nurse (RN) hourly compensation rates provided by LACDPH (Table 2). Additional details on costs are available in the online S1 File.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Baseline model parameters.

https://doi.org/10.1371/journal.pone.0254950.t002

Health states and treatment progression

We modeled patient progression from treatment initiation to completion using a Markov model. In the model, individuals were on-treatment, had completed treatment, or had defaulted (Fig 2). Consistent with LACDPH active TB treatment guidelines, we assumed a minimum successful treatment length of five months for all patients [3]. At each 30-day increment from 4–16 months of treatment, patients had some probability of completing or defaulting from treatment on each treatment arm (values found using the LACDPH dataset). Consistent with clinical practice, non-adherent patients on AiCure would be moved to DOT for the remainder of treatment. See S1 Table in S1 File for detailed transition probabilities by arm.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Markov state transition model for AiCure and DOT arms.

AiCure patients (left) could default to DOT treatment (right). DOT directly observed therapy.

https://doi.org/10.1371/journal.pone.0254950.g002

Results

Among AiCure patients, average total days on treatment was 252.5 (SD = 69.4, range 94 to 401), slightly higher but not statistically significantly different from the DOT average of 247.1 (SD = 89.1, range 8 to 456) (p = 0.729) (Table 3). Among those who successfully completed treatment, mean days were 258.9 (SD = 64.4) and 270.5 (SD = 67.4) for AiCure and DOT, respectively (p = 0.395).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Result statistics by treatment arm.

https://doi.org/10.1371/journal.pone.0254950.t003

Using the model, we found that AiCure results in greater benefits at a lower cost over DOT. In our base case analysis, AiCure cost an average of $2,668 per-patient, while DOT cost 83% more, at $4,894 per-patient over the 16-month horizon (Table 4). Our model results indicated AiCure produced 1.05 QALYs per patient over the 16-month time horizon and DOT produced 1.03 QALYs.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Model results under base case and scenario analyses.

https://doi.org/10.1371/journal.pone.0254950.t004

Incrementally, AiCure patients gain an additional 0.02 QALYs at cost savings of $2,226 over DOT. At WTP of $50K, $100K, and $150K per QALY, resulting in incremental NMB per patient under AiCure versus DOT that were all above zero ($3,142, $4,057, and $4,973 respectively).

In univariate DSA, no variation in the model inputs within reasonable ranges resulted in a negative incremental NMB for AiCure. The model was most sensitive to the number of doses taken per-member per-month (11 in the base case). Varying this parameter between 5.5 and 16.5 doses per-month resulted in an NMB range of $3,633 to $6,313 WTP of $150K/QALY. The model was also highly sensitive to hourly wage for licensed vocational nurses (LVN) of $35.45; varying this value (+/- 50%) resulted in an NMB range of $3,885 to $6,061 (see tornado diagram in S1 Fig in S1 File). These results suggest much of the cost savings from AiCure can be attributed to the significant decrease in personnel costs required per dose. As DOT treatment requires LVNs or other skilled personnel to engage in significant travel, the trip frequency/distance and hourly rate of these individuals drives large cost differences.

In PSA, AiCure produced more QALYs per patient at a lower cost than DOT in 93.5% of 10,000 simulations (Fig 3). At WTP/QALY of $50K, $100K, and $150K, the probability of AiCure being cost-effective is 95.3%, 95.9%, and 96.4%, respectively. These results suggest the base scenario is robust to univariate and probabilistic variations in the input parameters. However, we find there is some probability that DOT is the more cost-effective choice as the WTP for a QALY decreases (see cost-effectiveness acceptability curve [CEAC] in Fig 3B).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Probabilistic sensitivity analysis results.

Panel a: Incremental values per-patient of AiCure relative to DOT. Panel b: Cost-effectiveness acceptability curve (CEAC) by treatment arm. DOT directly observed therapy, QALY quality-adjusted life year.

https://doi.org/10.1371/journal.pone.0254950.g003

Our scenario analyses also suggest that the most a clinic could lose by adopting AiCure would be relatively small (Table 4). In the “worst case” scenario, we find that AiCure would not be cost-effective, generating 0.01 more QALYs per patient for $3,672 more dollars than DOT (AiCure ICER of $433,646). On the other hand, clinics could stand to gain much more by adopting AiCure: in the “best case” scenario, we find that AiCure could generate 0.03 more QALYs per patient for $7,603 less costs than DOT. The difference in health outcomes are quite small in both the “best” and “worst” cases, indicating that patient health would not be substantially affected with or without AiCure, although the financial impact could be large. See the online S1 File for additional details on our primary and secondary scenario analyses.

In the third scenario analysis, we compared AiCure to VDOT, where nurses monitor patient video records to ensure the AI system is operating correctly. We find that even with this additional monitoring the system would still be cost saving. If 7 minutes were spent reviewing each dose (as observed in LA County’s system after AiCure’s initial adoption during implementation of the parent research study), AiCure still saved $1,517 per patient relative to DOT. This 7-minute figure per-dose is included in the “worst case” scenario analysis previously described and is comparable to figures found for VDOT in previous studies. AI monitoring systems could potentially save a few hundred dollars per patient compared to VDOT systems. Consistent with prior studies, we estimate that there would be a cost savings of roughly $1,500 per patient if nurses reviewed video of each dose, which is about $700 less savings than using the AI monitoring system in our base case [7, 8, 15]. This could amount to substantial savings for large health systems if AI monitoring was used at scale.

Discussion

The data from LAC patients suggest that AiCure would be an effective method of reducing nurse and patient burden and achieving similar levels of DOT success. Patients on AiCure completed treatment in a statistically indistinguishable amount of time as DOT (p = 0.729) while costing $2,226 less, on average. Sensitivity analysis results showed that despite the uncertainty around completion times due to the limited number of patients in the LAC data, AiCure would be the cost-effective strategy over 95% of the time with willingness-to-pay levels over $50,000 per QALY. Even in cases where DOT was cost-effective over AiCure, the largest amount DOT could save over AiCure over all possible input values was modest ($3,672 saved per-patient) relative to the largest savings for AiCure over all possible input values ($7,603 saved per-patient).

However, these results may not be generalizable to all TB clinics and all patients. The cohorts used for comparison in this study were not perfectly statistically similar across all characteristics, constitute a small sample of all patients, and indeed may not be fully representative of the LAC TB patient population. We limited our analyses to non-drug resistant pulmonary TB cases only, as most TB cases in LAC are not complicated by extra-pulmonary TB or drug resistance–in 2015, 71.5% of TB patients had exclusively pulmonary disease. These would also be the most likely patients to be first shifted to an AI Platform, as their treatment regimens are relatively simpler. The cost savings and treatment outcomes in this analysis may not apply to a patient population with more complicated case histories or comorbidities, nor to health programs less geographically sprawling than those in LAC–the costs included those of LVN travel (an average of $35.45 per hour and 54 cents per mile in the base analysis). However, while cost-savings may be lower in smaller counties, variation in travel costs or nurse wages in the sensitivity analyses did not lead to DOT becoming cost-effective over AiCure in any scenario.

We also wish to acknowledge the assumptions in our model. We do not consider in the model possible side effects or delays in appropriate care due to a reduction in frequency of patient-nurse contacts, as we assume that patients’ monthly clinic visits are sufficient to address any treatment complications. We note that our base case may underestimate the true cost savings of AI treatment monitoring platforms for patients with more than 2–3 DOT field or clinic appointments per week; an increased frequency of clinic or field visits would only widen cost differences through necessary LVN and RN personnel hours. Recent ATS/CDC TB treatment guidelines recommend the use of daily rather than intermittent (such as twice or thrice weekly) TB therapy [26, 27]. This recommendation has been difficult to implement fully due to staff capacity. As LACDPH increasingly meets this standard, we would expect the true cost savings of AiCure to increase. We do not consider transmission in this analysis, with the assumption that both intervention arms effectively limit spread of TB soon after the patient starts treatment. We also assume that all patients who complete care would have roughly equivalent quality of life post-treatment whether on DOT or AiCure, and we therefore only consider the 16-month treatment for the time horizon.

Despite these limitations, our analysis suggests that AiCure would be a cost-saving alternative to DOT with no sacrifice in quality of care, provided it is offered to uncomplicated non-drug resistant pulmonary TB cases. Increased access to and implementation of AI treatment monitoring systems could potentially reduce the costs of already resource-constrained public health systems and free nurses to perform skilled tasks instead of traveling to patients or reviewing videos of patient to ensure adherence. These results suggest that the AiCure platform may be a reasonable option to consider for patients beyond those in the pilot study.

Conclusions

The evidence from this pilot study suggests that AI treatment monitoring platforms for TB may be worth investing in as cost-savings under typical operation should be substantial compared to DOT without degradation of patient outcomes. Our findings contribute to a growing literature on the value and application of artificial intelligence in medicine to improve clinical decision-making, implement efficiencies, and drive behavioral modification across large patient populations, especially with regards to remote treatment options for conditions which disproportionately affect disadvantaged populations in the US. Future work may look at the heterogeneity in outcomes by patient population and the value of conducting additional, larger-scale multi-arm studies to reduce uncertainty around efficacy and cost estimates.

Supporting information

Acknowledgments

We thank Stuart McMullen, of the Centers for Disease Control and Prevention and Ana Delia Hernandez, of the Los Angeles County Tuberculosis Control Program for managing all aspects of the clinical project using AiCure. We thank Wendy Gilman, Angela Lozano, Youghaper Bornazyan, Joselito Florendo, of the Pacoima Health Center for their work in the clinical study using AiCure. We also thank Adam Hanina and Laura Shafner of AiCure (New York, NY) for helpful comments on an earlier version of our cost-effectiveness study.