Study design

Stage I of the CADLAD trial entailed two distinct steps. First, a prospective, non-randomized development step was conducted to generate machine-learned algorithms for assessment of the presence of significant coronary artery disease (CAD) using paired phase signals with clinical outcomes data. Second, a blinded, paired comparison was performed to verify the performance of the machine-learned algorithm from step 1 with regard to the assessment of significant CAD as determined by coronary angiography, which represents the “gold standard” for evaluation of CAD. A patient was considered CAD positive for development purposes if at least one of the two following criteria were fulfilled: 1) one lesion had a stenosis of ≥ 70% or 2) if at least one lesion has a reduced fractional flow reserve (FFR) of ≤ 0.80. This is consistent with the accepted definition of clinically significant diameter narrowing of a coronary artery [7–8]. Based on the American College of Cardiology (ACC) guidelines [7]. FFR evaluation was the final determinate of CAD in vessels undergoing flow wire assessment.

In order to accurately machine learn, the results of a known standard must be paired with a signal to develop a solution that correctly reflects actual physiologic status. In this trial, all subjects (N = 606) had been referred for coronary angiography, and all subjects underwent the planned catheterization procedure with dye injections of both right and left coronary arteries. The CADLAD trial was designed to utilize the results of coronary angiograms as the “standard” and paired these catheterization findings with phase signals for the purpose of machine learning. Thereby, the machine-learning platform developed solutions to assess the presence of CAD. This method required enrollment of only subjects already referred by their physician for coronary angiography.

An objective of the protocol was that the results should reflect a broad cross section of clinical practice, and this goal prompted selection of hospitals and clinics that mirrored the diverse array of facilities providing care to patients with heart disease. The aim was attainment of machine-learned solutions that could be generalized. The physicians participating in the protocol followed angiogram interpretation consistent with the ACC guidelines for determining CAD including the use of flow wire when recommended [7]. Employing this well-established categorization of lesion significance replicates the current general cardiology practice with reference to interpretation.

Written informed consent was obtained from all participants in this trial. The individuals in this manuscript have given written informed consent (as outlined in PLOS consent form) to publish these case details. The non-significant risk protocol for the CADLAD trial was approved by a central institutional review board (Western IRB) and performed at 12 enrolling centers in the United States. All study sites were community-based hospitals with expertise in interventional cardiology.

Study participants

All participants had suspected but not known CAD and had been referred by their physicians for nonurgent angiography. Additional inclusion criteria included age > 21 years old and the ability to understand protocol requirements as well as provide informed consent. Exclusion criteria were prior documented myocardial infarction (MI) or previous percutaneous coronary interventions (PCI), prior coronary artery bypass grafting (CABG), indication for invasive coronary angiography other than to assess for obstructive CAD (e.g. arrhythmia, cardiomyopathy, valvular abnormality), previous heart valve replacement, previous sustained or paroxysmal atrial or ventricular arrhythmia, infiltrative myocardial disease (amyloid, sarcoid, right ventricular dysplasia), presence of cardiac implantable electronic device, including implantable cardioverter defibrillator, pacemaker, implantable loop recorders and other monitors, implantable neuro-stimulators, congenital heart disease, breast implants, pregnancy, breast feeding, currently taking any Type IA, IC or III antiarrhythmic, any history of amiodarone therapy, clinically significant chest deformity (e.g., pectus excavatum or pectus carinatum), and neuromuscular disease if the condition resulted in tremor or muscle fasciculations.

System and device description

The cPSTA System is a medical device system that uses tomography to analyze phase signals and assess the presence of significant coronary artery disease in the major coronary arteries (Fig 1). The first component is the Phase Signal Acquisition (PSAQ) System which includes: 1) the phase signal recorder (PSR), a hand-held instrument that acquires and transmits resting phase signals along with ancillary patient-specific information (gender, age, etc.); and 2) a cloud-based phase signal data repository (PSDR) that accepts, stores, and allows retrieval of the signals as well as ancillary patient-specific information. The second component is a coronary artery disease analytical engine (CAD AE) that processes and evaluates the phase signals to assess the presence and significance of coronary artery disease using the machine-learned algorithms. The final component is Cloud Services (CS), including the health care provider (HCP) Web Portal that the clinician utilizes to interpret images, review results, and generate a report. The report can be saved as a record for inclusion in the patient’s medical record.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Utilization of the Cardiac Phase Space Tomography Analysis (cPSTA) System.

Phase signal data are collected and transferred to cloud. The generated models and analysis are available for physician assessment. cPSTA System = Cardiac Phase Space Tomography Analysis System, CAD = coronary artery disease. Reprinted from presentation materials of A4L under a license, with permission from A4L and W20, original production 2016.

https://doi.org/10.1371/journal.pone.0198603.g001

Study procedures

After providing informed consent, participants underwent phase signal acquisition within seven days prior to angiography. Signals were captured utilizing the hand-held (PSR) device via seven sensors positioned on the chest and back. Phase signal data was collected for approximately 3 minutes and the data package was then transmitted wirelessly to the cloud based PSDR. An analytic engine, consisting of software based on the machine-learned algorithms, analyzed the acquired data and generated tomography images. The results can be made available through a secure web portal (Fig 2); however, in this trial, the outcomes from cPSTA were not provided to the physician caring for the participant, and no clinical decision-making was based on the cPSTA findings. All phase signals were procured by trained staff at each enrolling center. Angiography was performed by local physicians who made all clinical decisions regarding appropriate medical therapy. If the interventional cardiologist observed a lesion of questionable significance, use of a fractional flow wire was at his/her discretion. The participant’s involvement in the study was complete following acquisition of the phase signals and performance of the angiography. There was no required follow-up.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Development and verification of machine-learned predictor.

The learning phase pairs “gold standard” results with phase signals for machine learning to develop algorithms. The verification phase tests the performance of the final algorithms on naïve signal data. cPSTA System = Cardiac Phase Space Tomography Analysis System. Reprinted from presentation materials of A4L under a license, with permission from A4L and W20, original production 2016.

https://doi.org/10.1371/journal.pone.0198603.g002

Statistical analysis

Statistical analysis of the machine learned algorithm performance was done by direct calculation of the sensitivity, specificity, negative predictive value (NPV), and positive predictive value (PPV) using standard formulas. Bootstrap confidence intervals were used to assess the significance of these scores. Each subject was categorized into true negative (TN), true positive (TP), false negative (FN) and false positive (FP) by using a fixed, experimentally-determined threshold on the continuous output of the predictor, with an output below this threshold indicating a prediction of CAD negative; otherwise, the predictor is indicating CAD positive.

The 95% confidence intervals on these statistics were computed with the use of bootci, a MatLab R2016b (MathWorks; Natick, MA) function that returns the intervals employing bootstrap evaluation [9]. The receiver-operator characteristic curve (ROC) and its corresponding area under the curve (AUC) were computed with the use of the MatLab R2016b function perfcurve [10].

Machine learning and algorithm development

The ML method employed was elastic net, which is a linear regression method with L1 and L2 regularization penalties [11]. The development set of signals (N = 512) was divided into separate training (N = 339) and validation (N = 173) sets. Elastic net was chosen because the combination of L1 and L2 penalties helps the approach overcome issues surrounding high-dimensional data with small training sets. In addition, it has been shown that linear regression techniques only need two independent observations per variable to avoid overfitting [12]. The elastic net received as input a set of features that represent tomographic and dynamical properties of the signal, along with the subject’s age, gender and heart rate. The features used in the machine-learned algorithms were extracted from each signal, meaning the signal is a sample and the features are components of the sample. A total of 405 features were used, with correlation-based feature reduction being applied before the linear model was learned (see below). When a high number of features are used for development of the model, dimensionality problems may arise with machine learning. High regularization methods can be used effectively to avoid overfitting and ameliorate the difficulties related to this issue [13]. Elastic net employed in our study automatically selects and regularizes the features to overcome the dimensionality issues. Elastic net has been shown to effectively train predictive models in high dimensional datasets with small sample sizes [14].

Additionally, the quality of subject’s signal was evaluated with a noise score (“noise volume”) designed to evaluate the impact of noise in the signal through changes in three-dimensional phase space by environmental factors such as complex powerline harmonics, medical imaging devices, and other electrical equipment in the proximity of the acquisition.

The elastic net algorithm learns a weight (w_j) for each extracted feature (f_j) with the final prediction for a given participant being calculated as where b is the bias term. The L1 and L2 regularization methods penalized large weight values during learning, which helps prevent overfitting on the training set. Elastic net optimizes the above summation using the following loss function, incorporating both the fit of the output from the model as well as the regularization terms.

The parameters α and λ control the L1 and L2 penalties. Training was performed on a modified (continuous) Gensini score as the target [15], which applied severity and location scores for each lesion in the coronary tree and summed over the lesions to produce the score. This model only took into account the subject’s worst-case lesion (that with the maximal product of severity and lesion scores) and applied a logarithm such that it was more tractable for a linear model. A dampening factor of 0.25 was applied to the worst-case Gensini score when collaterals were present in order to increase the value of that modified Gensini score in predicting the subject’s true (binary) CAD label.

As with all machine learning algorithms, elastic net has several hyper-parameters, which modify the optimization process (e.g., α and λ). The implementation employed supported the upweighting of data points in the training set based on a threshold on the modified Gensini score, allowing for the adjustment of the importance of various subject categories during learning. An additional supported parameter is the reduction of the feature set by removing features that are correlated over a specific threshold. For the model reported here, a correlation threshold of 0.907 was used, reducing the feature set from 405 features to 315.

To automatically tune these various hyper-parameters, composed of weight, Gensini threshold for weighting, correlation reduction threshold, and two elastic net-specific hyper-parameters α and λ, a genetic algorithm (GA) was utilized [16,17]. At each generation of the GA, the performance of every genome (set of parameter values) in the current iteration was evaluated by inputting its values into elastic net, running elastic net, and measuring the final model’s performance on both the training and validation sets. The model’s continuous output of the modified Gensini score was treated as a predictor of the subjects’ binary CAD status. The model’s performance on these datasets was measured using the area under the receiver operator characteristic curve, so that all possible thresholds on the continuous output were considered. Since CAD is a continuous disease, with a large variety of lesion locations and sizes within the disease-positive and disease-negative cohorts, the modified Gensini score allowed the linear model to predict the disease along the natural continuum, and binning was only considered when assessing the fitness of the hyperparameters. Additionally, the fitness function allowed higher quality data, as measured by noise volume, to receive a higher weighting in the performance assessment. The fitness function is summarized in equation form below where T and V denote training and validation sets respectively and i denotes the noise subset as identified by thresholds on the noise volume. Five overlapping noise subsets are used as corresponding to four noise thresholds, where all data is contained in the largest noise set, but data above the first threshold was excluded from the second noise subset, and so on. Defining the fitness function to optimize the AUC on 10 different overlapping noise subsets from the development set has the advantage of avoiding overfitting any single set as well as optimizing the performance on all the sets.

Genomes whose parameters enabled elastic net to find good models relative to other genomes in the population were selected for reproduction to create new genomes. During reproduction, each parameter in a genome underwent “mutation” with a probability of 0.33, with mutations either perturbing the parameter by a random value or replacing it with a new random value. Half of all new genomes also underwent single-point crossover, where a random position on the genome was selected, with the parameters before this point coming from one high-performing genome from the previous generation, and the second set of parameters coming from a different, high-performing genome.

These new genomes become the population for the next generation, and the whole process repeated until no further improvements in fitness scores were observed for 10 consecutive generations of the GA. The resultant linear model that adequately assessed significant CAD was embedded in the analytic software and verified against the naïve cohort.

Study population

Enrollment began in May 2016 and was completed in June 2017. There were 606 participants in the study, with 159 (31%) having protocol defined obstructive coronary lesions. The algorithm development cohort (training [N = 339] and validation [N = 173] sets) consisted of signals from 512 participants with an additional 94 participants’ signals serving as a naïve verification cohort (not utilized for machine learning purposes). The verification cohort was used for the blind testing of the machine-learned predictor. Demographics for the studied population are shown in Table 1. There was no significant difference between the development and verification cohorts except with regard to age of the subjects (p = 0.04). The entire population, both development and verification cohorts, included nearly 40% women. However, in this population presenting with chest pain and referred for angiography, only one third of participants had significant obstructive lesions as defined by the protocol at coronary angiography.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Demographics of population.

https://doi.org/10.1371/journal.pone.0198603.t001

Performance of the machine-learned predictor

The final output of ML, the fixed algorithms incorporating features of the signal, were embedded within the CAD AE. Testing of the analytic engine of the cPSTA System blindly in the naïve verification cohort showed the final predictors had a sensitivity of 92% (95% CI: 74%-100%) and specificity of 62% (95% CI: 51%-74%) for the assessment of obstructive coronary artery disease [18]. There was an automatic selection of a clinically-relevant threshold from the ROC curve. The negative predictive value (NPV) was 96% (95% CI: 85%-100%) and the PPV = 46% (95% CI: 33%-62%).

Safety of acquiring phase signals

The protocol anticipated the possibility of skin irritation or allergic reaction to the sensors used for collection of the phase signals. In 606 participants, no adverse events were reported.