Study design

This is a multicenter, cross-sectional analysis of prospectively collected subject data that was collected from spatially and temporally distinct cohorts. Cohort 1 included subjects from three geographically separate tertiary-care retina referral centers—the USC Roski Eye Institute (Los Angeles, CA), Retinal Consultants of Arizona (Phoenix, AZ) and Associated Retinal Consultants (Royal Oak, MI). Cohort 2 included subjects from a separate DR teleretinal screening center located at Los Angeles County + University of Southern California (LAC+USC) Medical Center (Los Angeles, CA). Approval was obtained from the University of Southern California Institutional Review Board (IRB) and Western IRB. The study adhered to the tenets of the Declaration of Helsinki. Subjects consented to have their data available for research purposes and this manuscript is based on secondary analyses of prospectively collected data.

For Cohort 1, inclusion criteria for non-diabetic controls was availability of ophthalmological exam results and OCTA images at the time of clinical exam and no self-reported history of diabetes mellitus (DM). These were non-diabetic subjects where the examining ophthalmologist ruled out any significant ocular pathology. OCTA imaging is standard-of-care at these retina specialist clinics. Diabetic subjects were included if they had a known diagnosis of any DM, ophthalmological exam results which showed at least mild DR, and OCTA images at the time of clinical examination. Subject data was collected from February 2016 and June 2019.

For Cohort 2, non-diabetic controls were not available because only diabetic subjects were referred for DR screening in the LAC+USC Teleretinal DR Screening clinic [21]. Subjects with diabetes were included in Cohort 2 if they had a conclusive DR diagnosis (with an official result in the teleretinal medical record) and simultaneous OCTA imaging. Data was collected between April and December 2019.

Exclusion criteria for all subjects were use of any intraocular pressure lowering drops, any significant media opacity, and any other ophthalmic diagnosis except dry eye, vitreous syneresis, posterior vitreous detachment, asymptomatic epiretinal membrane without retinal distortion, peripheral retinal tear not requiring incisional surgery, and trauma without any notable sequelae.

DR in subjects from Cohort 1 was classified based on a comprehensive ophthalmologic evaluation including dilated fundus examination performed by a board-certified and fellowship-trained retina specialist. DR categories were mild NPDR, moderate NPDR, severe NPDR and PDR based on the criteria set by the International Clinical DR and DME severity scales [5]. For subjects in Cohort 2, classification of DR was made using 3 nonmydriatic 45-degree digital fundus photographs which has significant agreement with 7-field ETDRS DR staging [22, 23]. DR staging was performed by an optometrist specifically trained and certified for teleretinal DR screening at the Los Angeles County screening clinic. This expert’s evaluation was then independently reviewed and adjudicated by one of our board certified and fellowship trained retina specialists. For all subjects, DME was assessed by manual review of all OCT B-scans. Demographic and clinical history including age, sex, and diagnoses of hyperlipidemia, hypertension, and diabetes were obtained from medical records in both cohorts.

OCTA at all sites was performed using a commercially available CIRRUS AngioPlex 5000 (Carl Zeiss Meditec, Dublin, CA, USA). 3mm x 3mm images consisting of 245 horizontal B-scans and 245 individual A-scans per B-scan were obtained. OCTA images were included if they had a signal strength > 7, were centered on the fovea, lacked significant hypo- or hyperreflective artifacts on en face structural images, and had fewer than 10 motion artifacts. If both eyes of a subject met the inclusion criteria, a highest quality image (fewest number of imaging artifacts) was determined as well.

The superficial retinal layer (SRL) was the layer analyzed in all eyes to minimize the confounding impact of projection artifacts, projection artifact removal software, or layer segmentation. The commercially available Cirrus AngioPlex automated segmentation algorithm defined the SRL from the internal limiting membrane to the outer boundary of the inner plexiform layer. All OCT B-scan segmentation lines were reviewed for SRL segmentation errors and none of the images included in the final analysis required manual resegmentation.

A previously validated, semi-automated, custom MATLAB (MathWorks Inc., Natlick, MA) algorithm was applied to a grayscale bitmap image of the SRL exported from the AngioPlex machine to calculate the VSD and VDI for each image [6–8, 24–26]. VSD is a measure of vascular length and is computed by skeletonizing all vessel segments to a width of one pixel, counting the number of pixels that represents all skeletonized vessels and dividing that by the total number of pixels in the image [7, 15]. VDI is derived by taking the number of pixels showing blood flow in a binarized OCTA image and dividing that by the number of pixels showing blood flow in a skeletonized OCTA image [7, 15].

Statistical analysis

All statistical analysis and regression models were performed using SAS Version 9.4 (Cary, North Carolina). A post-hoc analysis of study power was performed. We found that our sample size provides more than 80% power for detecting significant differences between the control and any DR groups and the mild DR and referable DR groups for VSD or VDI.

A mixed effects multinomial logistic regression model was developed using subjects from Cohort 1 as the training cohort and modeling disease severity as the dependent nominal variable with 3 levels including control, mild, and referable DR (a previously defined group [27] which included moderate/severe NPDR, and PDR). Routinely collected clinical and demographic data shown to be potential predictors of DR were considered for model inclusion with VSD and VDI: these included age, sex, hypertension, and hyperlipidemia. Variables significantly associated with the DR severity outcome in a univariable model were selected for a multivariable model using a backward and stepwise elimination method until a parsimonious model with all variables statistically significant at p < 0.05 via the Satterthwaite denominator degree-of-freedom method was obtained. During this process, age was removed from the model, so that the complete model included VSD, VDI, sex, hypertension, and hyperlipidemia. In other words, age was not a significant confounder for our models and thus age was not retained as a covariate. Other work has previously failed to find an association between age and severity of retinopathy [28]. All models were adjusted for the random effects of eye-nesting within subject and ophthalmic center. The Newton-Raphson with ridging optimization technique was utilized to help with the convergence of the procedures. All models we subsequently describe show acceptable goodness-of-fit with the Hosmer-Lemeshow test p > 0.05, indicating our models are well-calibrated.

The discriminatory performance of the final multivariable model in determining any DR versus controls and referable DR versus mild DR was assessed by generating receiver operating characteristic (ROC) curves and calculating the area under the curve (AUC).

For each classification of interest, the AUC from models including VSD, VDI, or both VSD and VDI was compared nonparametrically to a general covariate model called “Model 1” that included only sex, hypertension, and hyperlipidemia in order to determine the incremental discriminatory contribution of OCTA metrics [29]. The p-value for the null hypothesis that the difference in the AUCs = 0 was generated by estimating a covariance matrix using the method of structural components and computing the test statistic having an asymptotically chi-square distribution [29].

Each constrained comparison of interest was further internally validated for the complete prediction model using ten-fold cross validation. This involved splitting the data into ten equally sized groups, estimating the model coefficients on 9 of the 10 groups, and then evaluating its accuracy in the group held out. The AUC was averaged over the 10 validation sets and reported as the evaluation metric. The prediction model derived from Cohort 1 was then applied to data from Cohort 2 which served as a validation data set. Model performance for the referable DR versus mild DR comparison in Cohort 2 was similarly quantified by calculating the AUC; incremental contributions of VSD and VDI were assessed as described for Cohort 1.

Using one highest quality single image for each subject in both cohorts, ROC-based thresholds were used to estimate the optimal values for both VSD and VDI in discriminating any DR versus control and referable DR versus mild DR. The threshold giving the highest correct classification rate was chosen as the most appropriate summary measure.

Multivariate model performance in discriminating between any DR severity versus control and referable DR versus mild NPDR

Fig 2 shows the ROC curves for classifications of any DR versus control and referable DR versus mild DR in Cohort 1. The AUCs for Model 1 classifying any DR versus control and referable DR versus mild DR as seen in Table 3 were 0.80 (95% CI: 0.76–0.84) and 0.69 (95% CI: 0.64–0.75), respectively. The addition of VSD significantly (p < 0.001) increased Model 1’s AUC by 0.10 to 0.90 (95% CI: 0.87–0.93) and by 0.19 to 0.88 (95% CI: 0.84–0.91) for any DR versus control and referable DR versus mild DR, respectively. The addition of VDI to Model 1 increased the AUC by 0.05 to 0.85 (95% CI: 0.82–0.88) and by 0.12 to 0.81 (95% CI: 0.76–0.86) respectively, for any DR versus control and referable DR versus mild DR. A comprehensive model with VSD, VDI, sex, hypertension, and hyperlipidemia was significantly (p ≤ 0.02 for every case) superior in its discriminatory ability compared to any other combination of comorbid factors with VSD or VDI, with an AUC of 0.91 (95% CI: 0.88–0.93) and 0.90 (95% CI: 0.86–0.93) for any DR versus control and referable DR versus mild DR, respectively. Model AUCs along with sensitivity and specificity are shown in Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Receiver operator characteristic (ROC) curves.

ROC curves for the constrained comparisons of any DR severity versus control and referable DR versus mild non-proliferative DR in Cohort 1. Additional lines are drawn for the sequential addition of VSD and VDI as parameters to the baseline model (labeled Model 1 and drawn in blue) containing sex, hypertension, and hyperlipidemia. A red line is drawn for a model with the parameters for Model 1 with VSD (red), a green line is drawn for a model with the parameters of Model 1 with VDI (green), and a brown line is drawn for the complete model containing the parameters of Model 1 with VSD and VDI (brown). Area under the curve for each ROC curve is listed in parenthesis next to each model in the legend.

https://doi.org/10.1371/journal.pone.0262996.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Model evaluation.

https://doi.org/10.1371/journal.pone.0262996.t003

Limitations

One possible limitation of this study is that we did not analyze the deep retinal layer (DRL) though this decision was made for several reasons [17, 35–38]. Durbin et al. have reported that the AUC of retinal capillary density in the SRL was the best for detecting DR [11]. SRL vessel density changes may also be more reliably detectable than those in the DRL [39–41]. SRL analysis also does not require projection artifact removal which is available but not clearly validated against a meaningful ground truth [14, 36, 42, 43]. Reproducibility and repeatability has been shown in the SRL [44] as well. Though axial length calibration may also affect the VSD and VDI measurements [45], this information is not readily available at a diabetic retinopathy screening center which comprised our validation cohort.

The nature of this study as a secondary analysis created additional limitations. This prevents us from claiming that this is a predictive model as patients were not prospectively categorized before classification by the ophthalmologist. Furthermore, the model does not incorporate certain patient demographics including those with diabetes without DR, and thus the any DR versus control comparison could not be validated in our external cohort. All of these concerns could be further addressed by a prospective study.