http://orcid.org/0000-0002-3292-9581
Figures
Abstract
Purpose
To compare macular and peripapillary vessel density values calculated on optical coherence tomography angiography (OCT-A) images with different algorithms, elaborate conversion formula, and compare the ability to discriminate healthy from affected eyes.
Methods
Cross-sectional study of healthy subjects, patients with diabetic retinopathy, and glaucoma patients (44 eyes in each group). Vessel density in the macular superficial capillary plexus (SCP), deep capillary plexus (DCP), and the peripapillary radial capillary plexus (RCP) were calculated with seven previously published algorithms. Systemic differences, diagnostic properties, reliability, and agreement of the methods were investigated.
Results
Healthy eyes exhibited higher vessel density values in all plexuses compared to diseased eyes regardless of the algorithm used (p<0.01). The estimated vessel densities were significantly different at all the plexuses (p<0.0001) as a function of method used. Inter-method reliability and agreement was mostly poor to moderate. A conversion formula was available for every method, except for the conversion between multilevel and fixed at the DCP. Substantial systemic, non-constant biases were evident between many algorithms. No algorithm outperformed the others for discrimination of patients from healthy subjects in all the retinal plexuses, but the best performing algorithm varied with the selected plexus.
Conclusions
Absolute vessel density values calculated with different algorithms are not directly interchangeable. Differences between healthy and affected eyes could be appreciated with all methods with different discriminatory abilities as a function of the plexus analyzed. Longitudinal monitoring of vessel density should be performed with the same algorithm. Studies adopting vessel density as an outcome measure should not rely on external normative databases.
Citation: Rabiolo A, Gelormini F, Sacconi R, Cicinelli MV, Triolo G, Bettin P, et al. (2018) Comparison of methods to quantify macular and peripapillary vessel density in optical coherence tomography angiography. PLoS ONE 13(10): e0205773. https://doi.org/10.1371/journal.pone.0205773
Editor: Gemmy Cheung, Singapore National Eye Centre, SINGAPORE
Received: June 28, 2018; Accepted: September 27, 2018; Published: October 18, 2018
Copyright: © 2018 Rabiolo et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information file. We have now included 4 supplementary databases in our revised manuscript. File S1 is a general database with demographic and clinical data. Files S2, S3, and S4 show the processed OCT-A data for the macular SCP, macular DCP, and peripapillary RCP.
Funding: The authors received no specific funding for this work.
Competing interests: Kouros Nouri-Mahdavi: Heidelberg Engineering (F,R). Giuseppe Querques is consultant for: Alimera Sciences, Allergan Inc, Bayer Shering-Pharma, Fidia-Sooft, Heidelberg, KHB, Lumithera, Novartis, Roche, Sandoz (Berlin, Germany), Topcon, Zeiss. Francesco Bandello has the following disclosures: Allergan (S), Alimera (S), Bayer (S), Farmila-Thea (S), Schering Pharma (S), Sanofi-Aventis (S), Novagali (S), Pharma (S), Hoffmann-La Roche (S), Genentech (S), Novartis (S). This does not alter our adherence to PLOS ONE policies on sharing data and material.
Introduction
Optical coherence tomography angiography (OCT-A) is a recent imaging modality that allows non-invasive, rapid, depth-resolved visualization of all the chorioretinal vascular layers. [1, 2] OCT-A devices use different algorithms, all of which are based on the assumption that erythrocytes in the blood vessels are the only moving structures within sequentially acquired B-scans and that they act as a natural motion contrast. Since its recent introduction, OCT-A has gained increased popularity and has been applied to a broad spectrum of disease. [3] En face OCT angiograms can be subjectively evaluated for presence of abnormalities, or can be further post-processed to obtain quantitative, objective metrics. Vessel density, defined as the percentage of the angiocube occupied by retinal vessels, has gained increasing popularity, and represents a promising imaging endpoint for future clinical trials. Its reliability, however, needs to be fully validated. [4] Previous studies demonstrated good intra- and inter-operator repeatability of vessel density for images acquired in the same location, with the same angiocube size, machine, and quantification algorithm. [5, 6] Vessel density results, however, can significantly differ among various devices and depend on angiocube size, scan location, signal strength. [5, 7–9] The reported thresholding algorithms employed to binarize OCT-A angiograms and calculate vessel density, are highly heterogeneous. Some instruments use their own proprietary software, these include Cirrus (AngioPlex software, Carl Zeiss Meditec, Inc., Dublin, CA, USA), [5, 10] AngioVue software (Optovue, Inc., Fremont, CA, USA), [11, 12] and RS -3000 Advance (Nidek, Gamagori, Japan). [6] In the majority of the studies, however, images are exported and post-processed with a variety of different thresholding methods, including fixed cutoffs, [13–15] dynamic cutoff (e.g., mean, [9, 16, 17] ImageJ [National Institutes of Health, Bethesda, MD] default algorithm, [18] Otsu’s algorithm), [19, 20] prototype software, [21, 22] and more complex methods combining preprocessing filters and multilevel thresholds strategies. [23–26] It is still uncertain whether different algorithms lead to the same or, at least, similar results and findings from many studies could have been influenced by the algorithm utilized.
The aims of this study were to compare macular and peripapillary vessel density values calculated with seven different binarization strategies, to calculate conversion formulas between the algorithms, and to compare their diagnostic performance in differentiating healthy subjects from patients affected by diabetic retinopathy (DR) and glaucoma.
Material and methods
Study design and patients
One hundred thirty-two eyes of 84 subjects that had OCTA at the Glaucoma Unit, and Medical Retina and Imaging Unit of the Department of Ophthalmology, University Vita-Salute, San Raffaele Hospital, Milan, Italy, were retrospectively evaluated. The study was approved by the San Raffaele Hospital scientific committee, and it adheres to the recommendations of the Declaration of Helsinki. Informed written consent was obtained from all the subjects included. Electronic clinical records, SD-OCT (Cirrus HD-OCT 5000; Carl Zeiss Meditec, Inc.), and swept source (SS)-OCT-A (PLEX Elite 9000, Carl Zeiss Meditec, Inc., Dublin, CA, USA) images from healthy subjects, patients with DR, and patients with glaucoma were reviewed. General inclusion criteria were age ≥18 years old, refractive error between -6 and +3 diopters, and availability of structural OCT and 6x6 mm OCT-A scans with a signal strength ≥ 7. General exclusion criteria included previous ocular surgery other than uncomplicated cataract extraction and intraocular lens implantation performed >6 months before enrollment, and artifacts on OCT-A images. Inclusion criteria specific to the DR group were: type 1 or 2 diabetes, and presence of diabetic retinopathy. Inclusion criteria specific to the glaucoma group were: (i) history of primary open-angle glaucoma; (ii) documented glaucoma damage at the optic disc; (iii) repeatable glaucomatous perimetric damage, defined as a glaucoma hemifield test (GHT) result outside normal limits, and a pattern standard deviation (PSD) with p value <5% normal limits; or (iv) presence of a cluster of ≥3 adjacent points on the pattern deviation plot with a probability of <5%, including at least 1 point having a probability <1% on at least two consecutive standard achromatic visual fields; (v) peripapillary retinal nerve fiber layer (pRNFL) thickness with p <5% in at least one quadrant. Presence of diseases other than DR and glaucoma, respectively, in the DR and glaucoma groups was also an exclusion criterion. Healthy subjects met the following inclusion criteria: (i) no history or evidence of any posterior segment disease; (ii) normal-appearing optic disc and retina on dilated fundus examination; (iii) normal foveal profile on the structural macular SD-OCT scan; (iv) average and quadrant pRNFL thickness within 99% confidence limits; and (v) at least one reliable normal visual field, defined as PSD within 95% confidence limits of the normative database, and a GHT results within normal limits.
Structural SD-OCT measurements
The structural SD-OCT images consisted of the Optic Disc Cube 200x200, and Macular Cube 512x128 patterns. Control subjects and glaucoma patients had both scans, whereas diabetic patients had only the macular scan. The manufacturer’s software was used to calculate the average pRNFL thickness and central macular thickness (CMT) values on the peripapillary and macular scans, respectively.
SS-OCTA device and scanning protocol
The SS-OCTA device (PLEX Elite 9000, Carl Zeiss Meditec, Inc., Dublin, CA, USA) uses a swept laser source with a central wavelength of 1040–1060 nm (980–1120 nm full bandwidth) and operates at 100,000 A-scans per second. The axial and transverse resolutions of the system are ∼6 μm and ∼20 μm in tissue, respectively. The OMAG algorithm, which is based on variations in both the intensity and phase information between sequential, co-registered B-scans, was used to generate an OCT-A image. [27, 28] The 6x6 mm angiocube consisted of 500x500 A-scans. En face images consist of a 1024x1024-pixel array with 5.9 μm spacing between pixels. For DR eyes, all scans were centered on the fovea and automated segmentation of the layers carried out to define superficial (SCP) and deep (DCP) capillary plexuses were reviewed. The device’s projections removal algorithm was applied to DCP images. For the glaucoma group, scans were centered on the optic nerve; the segmentation algorithm defined the area between the inner limiting membrane and the outer boundary of the RNFL to isolate the peripapillary radial capillary plexus (RCP). Healthy subjects had both macular and peripapillary OCT-A angiograms. Anonymized raw files were downloaded from the Zeiss PLEX Elite 9000 instruments and uploaded in the Advanced Retina Imaging (ARI) network portal. En face angiograms were then exported in PNG (Portable Network Graphics) format.
Quantitative analysis of OCT-A images
Seven different threshold strategies were used to binarize en face angiograms and calculate vessel density. The algorithms included the Macular Density algorithm v 0.6.1 developed by Zeiss, a manual thresholding technique, three ImageJ autothresholding algorithm (i.e., mean, default, Otsu), a semiautomatic method using a fixed threshold, and a method combining preprocessing filters with multilevel threshold strategies. Besides the analysis performed with the Zeiss macular density algorithm, all the other ones were carried out using the ImageJ software v 1.51 (National Institutes of Health, Bethesda, MD). At the end of the binarization process, the ratio between the number of white pixels (i.e., vessels) and the number of total pixels was calculated to obtain the vessel density. Fig 1 illustrates an example of binarization outcomes with the employed methods.
Columns and rows show methods and segmented plexuses, respectively.
ARI Zeiss macular density algorithm v 0.6.1.
This is a prototype, proprietary algorithm that allows quantification of vascular density both at the SCP and DCP in the macular area. Two indices are generated: perfusion density and vessel density. The former is generated through image binarization, while the latter relies on a skeletonization process. In this manuscript, we included only the first index in our analysis since it is the one based on thresholding, as the other algorithms tested. Since the algorithm was developed for quantification of macular perfusion parameters, it was not used for measuring the peripapillary vasculature. Macular density algorithm was run directly in the ARI portal.
Manual thresholding.
Three independent operators (AR, RS, FG) independently binarized all the images using a manual, semiautomatic method. Each image was opened twice in the ImageJ software: one was not processed and used as reference, the other was binarized using a manual threshold. Each operator arbitrary chose a cutoff for each image to obtain the best correspondence with the one used as reference.
Mean, default, Otsu autothreshold algorithms.
Each image was loaded in the ImageJ software and processed using three different autothreshold algorithms: mean, default, and Otsu. Specification of these algorithms can be found online (https://imagej.net/Auto_Threshold). Briefly, the mean algorithm sets the mean level of grey in the image as cutoff. The default is a special method used by ImageJ software, which is a modification of the IsoData algorithm. The Otsu algorithm performs a two cluster-based binarization applying a threshold, which minimizes and maximizes the intra-class and inter-class variances, respectively.
Fixed threshold.
In this scenario, the same cutoff was utilized for every image of the same plexus. The median level of gray of macular SCP, macular DCP, and peripapillary RCP of the control group was chosen as threshold value. For the macula centered images, we used “59” and “32” for the SCP and DCP, respectively. For the optic nerve centered images, a value of “74” was used as cutoff.
Preprocessing filtering with multilevel thresholds strategies.
The method used was previously described. [23] Briefly, images were pre-processed using a top-white filter with a window size of 12 pixels. Then, the image was duplicated. One copy underwent binarization through the median auto local threshold; the other one was processed with a Hessian filter and binarized using a Huang autothreshold. The two binarized images were compared and only pixels positive with both methods were counted as vessels.
Statistical analysis
Data were tested for normality with the Pearson-D’Agostino test. Differences for demographic and main clinical data across groups were evaluated with analysis of variance (ANOVA) and Kruskal-Wallis test, respectively for parametric and non-parametric variables, and Dunnet and Dunn tests were used as post-hoc tests to compare patients with glaucoma and DR groups to healthy subjects for parametric and non-parametric variables, respectively. Differences for categorical variables were assessed with the chi-square test. Differences between glaucoma patients and healthy subjects for VCDR and pRNFL were evaluated with the t-test and the Mann-Whitney test, respectively.
Differences among the algorithms for vessel density values were evaluated with the repeated measures ANOVA or Friedman test for parametric and nonparametric variables, respectively. Pairwise comparisons were investigated with the Tukey test or Dunn’s multiple comparisons as post-hoc tests for parametric and nonparametric data, respectively. Differences in vessel density between healthy and affected eyes at each plexus and with each method were investigated with a linear mixed effect model with presence of disease and age as fixed factors, and fellow eye as random factor.
Inter-algorithm reliability and inter-operator reliability for manual threshold selection were evaluated with the intraclass correlation coefficient (ICC) and the 95% confidence interval (CI) based on a single rating, concordance, 2-way mixed effect model. Values of ICC between 0 and 0.5 indicate poor reliability, moderate reliability between 0.5 and 0.75, good reliability between 0.75 and 0.9, and excellent reliability above 0.9. [29] We used Bland-Altman analysis to evaluate inter-algorithm agreement and limits of agreement (LOA) were set at 1.96 standard deviations (SDs) as this gives the 95% CI.
A structural equation model was used to generate calibration equations to quantify systemic bias between each pair of algorithms, and to elaborate conversion formulas from one method to another.
Ability to discriminate affected eyes from healthy eyes at each plexus was evaluated with receiver operating characteristic (ROC) curves. ROC curves were estimated on a generalized linear model in order to adjust for age, and eyes of the same patients were considered as clustered to account for correlations between eyes. [30] Area under the curve (AUROC) values and 95% confidence intervals were calculated. AUROCs of 0.5 and 1 represent lack of and perfect discrimination, respectively. Pairs of ROC curves were compared with the DeLong’s test.
All statistical analyses were performed with R software (R Foundation for Statistical Computing, Vienna, Austria, https://www.r-project.org), GraphPad Prism software 6.0 (GraphPad Software, Inc., San Diego, CA, USA), and SPSS software 21 (SPSS Inc., Chicago, IL, USA). A p-value <0.05, after adjustments with the Benjamini-Hochberg test, was considered significant.
Results
Table 1 summarizes the demographic and main clinical data of the cohort of patients. Patients in the glaucoma groups were significantly older than those in control group (p = 0.008), and they had increased VCDR (p <0.0001) and reduced pRNFL thickness (<0.0001). Patients with DR had significantly greater CMT compared with controls (p = 0.001), and 18 eyes (40.9%) featured diabetic macular edema. In the DR group, 28 and 3 patients suffered from type 1 and 2 diabetes, respectively, with a mean hemoglobin A1c of 6.9 ± 1.1%. Severity of DR was mild in 1 eye, moderate in 6 eyes, severe in 8 eyes, proliferative in 28 eyes, and unknown in 1 eye.
Fig 2 and Table 2 show the macular SCP, DCP and peripapillary RCP vessel density values estimated with the different methods. Healthy eyes exhibited higher vessel density values at all plexuses compared with eyes with disease regardless the algorithm used (p < 0.01). No significant difference between diabetic eyes with and without DME estimated with every method at both SCP and DCP was found (p-value > 0.31). Vessel density values were significantly different among the different methods at all the plexuses (p < 0.0001). All the pairwise comparisons for the macular SCP (Table 3) were significant (p < 0.05), except for the pairs ARI and mean (p = 0.99), ARI and fixed (p = 0.70), mean and fixed (p = 0.10), and Otsu and multilevel (p = 0.20). At the macular DCP level (Table 4), all the pairwise comparisons were significant (p< 0.01), except for the pairs ARI and multilevel (p = 0.99), manual and default (0.99), manual and Otsu (p = 0.13), manual and fixed (p = 0.49), Otsu and fixed (0.99). At the peripapillary RCP level (Table 5), all pairs had significantly different vessel density values (p < 0.001), except for manual and mean (p = 0.08), mean and default (p = 0.99), and Otsu and multilevel (p = 0.99).
Vessel density values calculated with the tested methods for the entire cohort, diabetic retinopathy (DR) and glaucoma patients for the macular superficial capillary plexus (SCP) (A,B,C), deep capillary plexus (DCP) (D,E,F), and peripapillary radial capillary plexus (RCP) (G,H,I). Processing methods differences were significant at p < 0.0001 at all levels. SCP: superficial capillary plexus; DCP: deep capillary plexus; RCP: radial capillary plexus; DR: diabetic retinopathy.
Inter-algorithm ICC values of the SCP, DCP, and RCP are illustrated in Tables 3, 4 and 5, respectively. Default and Otsu algorithms had an excellent reliability at every plexus. Reliability of ARI algorithm was good when compared with default and Otsu methods. At the SCP, a good reliability was observed also for the pairs mean and default, mean and Otsu, and mean and multilevel. At the DCP, the manual algorithm had a good reliability in comparison with both default and Otsu methods. The pairs mean and multilevel, default and fixed, and Otsu and fixed had a good reliability at the peripapillary RCP. All the other pairs exhibited a moderate to poor reliability. Notably, fixed and multilevel had a negative ICC at the DCP since their vessel density values were negatively correlated.
Results of Bland-Altman analysis for the inter-method agreement of the SCP, DCP, and RCP are also displayed in Tables 3, 4 and 5, respectively. Limits of agreement and mean differences were wider in the DCP, indicating a lower level of agreement between methods for this plexus. Otsu and default algorithms exhibited an excellent agreement. On the contrary, the fixed algorithm had a poor agreement with all the other methods.
As shown in Fig 3, the inter-operator reliability for the manual method was moderate for the DCP overall and in the subset of patients with DR, and poor for all the other groups. Vessel density values were significantly different between each pair of operator, except the comparison between operator 1 and 2 at the DCP for the cohort of all patients, and between operator 3 and both operators 1 and 2 for the DCP of the patients with DR.
**** significant at p < 0.0001; *** significant at p < 0.001; ** significant at p < 0.01; NS: not significant; DR: diabetic retinopathy; ICC: intraclass correlation coefficient; op: operator; SCP: macular superficial capillary plexus; DCP: macular deep capillary plexus; RCP: peripapillary radial capillary plexus.
Table 6 displays the calibration equations for the methods. The conversion formula from one instrument to another was available for all the tested algorithms, except for the conversion from multilevel to fixed (and vice versa) at the DCP since observed values measured by these two algorithms were negatively correlated. Notably, the default and Otsu algorithms at the peripapillary RCP had almost no bias, and a small bias at the other plexuses, meaning that the systematic error between these two algorithms is extremely low. A substantial bias was evident between many algorithms. The comparative plots for the SCP, DCP, and peripapillary RCP are graphically shown in Figs 4, 5 and 6, respectively.
Green lines represent the no-bias line, while black circles demonstrate the true corresponding measurement among coupled devices.
Green lines represent the no-bias line, while black circles show the true corresponding measurement among coupled devices.
Green lines represent the no-bias line, while black circles demonstrate the true corresponding measurement among coupled devices.
ROC curves for the identification of patients with DR (macular SCP, DCP) and glaucoma (peripapillary RCP) are illustrated in Fig 7. At the SCP level, ARI and manual algorithms performed better (p < 0.05) than all the other methods, but they did not significantly differ each other (p = 0.6). At the DCP level, the manual, Otsu, mean, and default algorithms had higher AUROCs than other methods, and the difference was significant (p <0.05) except for the pair mean and ARI (p = 0.1). At the ONH level, the fixed algorithm had the best performance to distinguish glaucomatous patients from healthy subjects, although the difference was significant only in comparison with multilevel (p = 0.029).
Legends show area under the ROC curve and 95% confidence interval. SCP: macular superficial capillary plexus; DCP: macular deep capillary plexus; RCP: peripapillary radial capillary plexus.
Discussion
In the present study, we compared seven different methods to calculate vessel density on OCT-A angiograms in the macular and peripapillary areas in healthy subjects, and in patients with either DR or glaucoma. Methods tested included the ARI Zeiss proprietary algorithm v 0.6.1, a manual method, a static algorithm using the same fixed threshold for all eyes, three dynamic autothresholds (i.e., mean, default, Otsu) adapting their values in relationship to image properties, and a more complex algorithm employing a preprocessing filter followed by a multilevel thresholding strategy. We evaluated the inter-algorithm differences, reliability, and agreement. Moreover, we provided a calibration between algorithms to generate conversion formulas from one method to another. Finally, we investigated the ability of the algorithms to identify differences in vessel densities between healthy subject and patients with DR or glaucoma, and we compared their diagnostic performance.
Since its commercialization, OCT-A has gained increasing popularity in the ophthalmic community. The capability to image the retinal and choroidal vasculatures in non-invasive, fast, three-dimensional, depth-resolved fashion represents a considerable advantage of this technique over the traditional, dye-based diagnostic tests. Inspection of OCT-A angiograms may be of great help for the clinician in the diagnostic process. Identification of choroidal neovascularization in an asymptomatic patient with “dry” AMD or in a patient suffering from central serous chorioretinopathy are two examples where this new technology has a significant, clinical impact. OCT-A angiograms can be post-processed to obtain quantitative, objective measurements. Two established indices are the vessel density and, specific to the macular region, the fovea avascular zone (FAZ) area. The latter is a capillary-free zone corresponding to the foveola, while the former is the percentage of angiocube occupied by retinal vessels. A number of studies investigated changes in such indices in various ocular pathologies and their relationship with disease severity, activity, and response to treatments in several chorioretinal diseases, including diabetic retinopathy, [31, 32] retinal vein occlusion, [33] age-related macular degeneration, [11, 34] and retinal dystrophies. [13–15, 17] Vessel density applications are not limited to the retinal field, and are also useful in optic nerve diseases such as glaucoma [10, 22] and anterior ischemic optic neuropathy. [12] According to Cole et al., [4] vessel density could become a future surrogate endpoint for clinical trials. In a recent commentary, however, Garrity and Sarraf [35] claimed the need of additional technical and clinical research to fully elucidate properties and reliability of quantitative indices before their application in research trials and clinical practice. There is substantial evidence that the intra- and inter-operator repeatability of vessel density is good for images acquired in the same location, with the same angiocube size, machine, and quantification algorithm. [5, 6, 36] Nevertheless, some other factors both in the acquisition and in the post-processing phase can negatively affect its reliability. Some retinal diseases, such as cystoid macular edema, can profoundly disorganize the retinal architecture leading to inaccurate and unreliable results. [35] Low signal strength is correlated with lower vessel density, and this should be considered in patients with cataract or ocular surface disease. [5] Corvi et al [7] evaluated the reliability of quantitative indices, including vessel density, measured with seven different instruments in a cohort of healthy patients, and they found a poor reliability among the tested devices. These results are not surprisingly since different instruments have varying properties, including wavelength of laser beam, number of A-scans, algorithm used to detect flow, segmentation boundaries, and image resolution. In a previous study, we demonstrated that vessel density significantly differs across different angiocubes sizes since wider images are characterized by lower resolution compared to smaller, denser, scans. [9] Uji and colleagues [23, 37] evaluated the impact of multiple en face angiograms averaging, and found that it increases the image quality and impacts on the quantitative measurements reducing the variability. Only one study has provided some information about the impact of different post-processing imaging. Pedinielli et al [38] observed that the same image may lead to different macular vessel density values if quantified by means of skeletonization, mean thresholding, or proprietary AngioVue software (Optovue, Inc., Fremont, CA, USA). Differences between vessel density obtained by means of skeletonization and thresholding were predictable since they measure different units. Skeletonization does not take into account the vessel dimension and treats all the vessels in the same way irrespective to their size, so it minimizes the impact of large retinal vessels over the capillary network. On the contrary, image thresholding reveals the real percentage of retinal vasculature. Our results partially corroborate and expand the previous findings. Differences among methods were highly significant, and reliability and agreement values were mostly poor to moderate, with few exceptions. Differences were highest at the macular DCP with Manual, Default, Otsu, and fixed showing less differences, while ARI and Mean appear to report notably lower and higher values, respectively. The calibration equations revealed that systematic differences between algorithms were pretty consistent, and most of them exhibited a non-constant bias. A conversion formula was present for all the algorithms except for multilevel and fixed thresholds at the DCP. These two algorithms displayed very poor reliability and agreement, and, in the DCP, they surprisingly exhibited a negative correlation meaning that studies based on these algorithms may potentially lead to opposite results. On the other hand, we appreciated an excellent reliability and agreement between Otsu and default algorithms. These two methods demonstrated a small systemic bias for the macular area and, notably, no bias in the peripapillary area, suggesting that values derived from these algorithms are almost interchangeable. Conversion formulas provide a method to convert values from one algorithm to another, and, theoretically, longitudinal monitoring could be performed with images processed with different algorithms. However, most of the most methods exhibited a substantial, non-costant bias. Performance of these formulas should be validated in a different larger dataset before one may confidently switch from one algorithm to another during the follow-up. In the light of these considerations, we suggest to use the same algorithm in the longitudinal follow-up since processing methods are not directly interchangeable.
Several authors have conducted studies on large cohorts of healthy subjects to build normative databases for quantitative metrics, including vessel density. Coscas et al [39] were the first group to provide macular vessel density values in a Caucasian population. Iafe et al [40] also reported vessel density values in 70 healthy subjects, and Garrity et al [41] reported results of repeated analyses on the same cohort of patients updating the previous results based on more sophisticated software enabling projection removal and improvement of the segmentation algorithm with isolation of the intermediate capillary plexus. Bazvand et al [42] published a normative quantitative database for the papillary and peripapillary area. Other studies provided macular vessel density values in different populations (i.e., in Asia, Middle-East), and in pediatric subjects. [43–45] The results of these studies should be interpreted with extreme caution, and are not generalizable since vessel density values are dependent on the device, angiocube size, image averaging, and, as shown here, post-processing algorithm employed. Although not formally demonstrated in the present study, the updates of the proprietary software released by manufacturers could also cause changes in vessel density affecting longitudinal follow-up of patients.
Several studies unequivocally demonstrated that patients with DR and glaucoma have lower macular and peripapillary vessel density values, respectively. [2, 3, 21, 22, 24, 26, 31] In this study, we tried to replicate these well-established findings analyzing the same pool of images with seven different methods. All algorithms found a significant reduction in vessel density in the patients with glaucoma and DR compared to healthy subjects. We believe this is an important finding since it indicates that previous studies were not biased by the algorithm used at least for those tested in this study. We investigated which algorithm had the best diagnostic performance to discriminate patients from healthy subjects. No method outperformed the others in all the retinal plexuses, but the performance depended on the selected plexus. In the macular area, manual and ARI methods had the best AUROCs with regard to the SCP. At the DCP level, manual and autothresholds (i.e., mean, default, Otsu) algorithms had the best discriminating ability. As to the peripapillary RCP, all algorithms had a similar performance except for the multilevel, which had the smallest AUROC. These differences may be related to differences in the levels of grey in images segmented at different plexuses. Manual algorithms had a good discrimination ability in all the three analyzed plexuses, and this could indicate the flexibility of this method that allows the rater to manually adjust the threshold value based on a visual feedback. Unfortunately, manual algorithm is highly subjective and has a poor inter-rater reliability. Moreover, it is highly time-consuming and it requires trained raters.
Limitations of this study should be kept in mind. The retrospective nature dictated the fact that some variables (e.g., axial length) potentially affecting vessel density were not available. [46] Many methods to quantify vessel density have been published, and their relationship with those assessed in the present study remain unknown. Nevertheless, we tested a large number of methods using different strategies (including manual method, semiautomatic method with fixed threshold, semiautomatic methods with dynamic thresholds, and multilevel methods preceded by image filtering). All the images were acquired with the same device, and results might not be generalizable to other instruments. The calibration equations have not been tested on an external dataset, so they need to be fully validated. Also, they were based on the vessel density values of the study sample, and the relationship between two methods can differ for observations outside the range. Finally, our study was limited at the retinal vascular plexuses, and does not provide information about choroidal circulation.
In conclusion, we provide an extensive comparison of methods to quantify vessel density on OCT-A angiograms. Absolute values calculated with different algorithms are not directly interchangeable since methods have systemic differences, poor reliability, and poor agreement. Nevertheless, all the tested algorithms revealed significant differences between healthy and affected eyes, although they had different discriminatory abilities, which varied according to the plexus analyzed. This study indicates that longitudinal monitoring of the vessel density should be carried out with the same instrument, same scan pattern and location, and same algorithm. Studies adopting vessel density as an outcome should not rely on external normative database but include their own control groups. Knowledge of the properties for each algorithm could help researchers to select the best algorithm according to the plexus studied.
Supporting information
S1 Dataset. General dataset for demographic and clinical data.
Readable table containing general data of the study population, including age, sex, eye laterality, presence of glaucoma, presence and stage of diabetic retinopathy, presence and type of diabetes, levels of hemoglobin A1c, presence of diabetic macular edema (DME), vertical cup-to-disc ratio (VCDR), retinal nerve fiber layer thickness (RNFL), and macular thickness. Missing or not applicable data are indicated as “NA”.
https://doi.org/10.1371/journal.pone.0205773.s001
(CSV)
S2 Dataset. Dataset for superficial capillary plexus.
Optical coherence tomography angiography processed data for the macular superficial capillary plexus. DME: diabetic macular; OP1: operator 1; OP2: operator 2; OP3: operator 3. Missing or not applicable data are indicated as “NA”.
https://doi.org/10.1371/journal.pone.0205773.s002
(CSV)
S3 Dataset. Dataset for deep capillary plexus.
Optical coherence tomography angiography processed data for the macular deep capillary plexus. DME: diabetic macular; OP1: operator 1; OP2: operator 2; OP3: operator 3. Missing or not applicable data are indicated as “NA”.
https://doi.org/10.1371/journal.pone.0205773.s003
(CSV)
S4 Dataset. Dataset for peripapillary radial capillary plexus.
Optical coherence tomography angiography processed data for the peripapillary radial capillary plexus. DME: diabetic macular; OP1: operator 1; OP2: operator 2; OP3: operator 3. Missing or not applicable data are indicated as “NA”.
https://doi.org/10.1371/journal.pone.0205773.s004
(CSV)
References
- 1. Rabiolo A, Carnevali A, Bandello F, Querques G. Optical coherence tomography angiography: evolution or revolution? Expert Review of Ophthalmology. 2016;11(4):243–5.
- 2. Spaide RF, Fujimoto JG, Waheed NK, Sadda SR, Staurenghi G. Optical coherence tomography angiography. Prog Retin Eye Res. 2017. pmid:29229445.
- 3. Cohen SY, Miere A, Nghiem-Buffet S, Fajnkuchen F, Souied EH, Mrejen S. Clinical applications of optical coherence tomography angiography: What we have learnt in the first 3 years. Eur J Ophthalmol. 2018:1120672117753704. pmid:29554812.
- 4. Cole ED, Ferrara D, Novais EA, Louzada RN, Waheed NK. Clinical Trial Endpoints for Optical Coherence Tomography Angiography in Neovascular Age-Related Macular Degeneration. Retina. 2016;36 Suppl 1:S83–S92. pmid:28005666.
- 5. Lei J, Durbin MK, Shi Y, Uji A, Balasubramanian S, Baghdasaryan E, et al. Repeatability and Reproducibility of Superficial Macular Retinal Vessel Density Measurements Using Optical Coherence Tomography Angiography En Face Images. JAMA Ophthalmol. 2017;135(10):1092–8. pmid:28910435.
- 6. Al-Sheikh M, Tepelus TC, Nazikyan T, Sadda SR. Repeatability of automated vessel density measurements using optical coherence tomography angiography. Br J Ophthalmol. 2017;101(4):449–52. pmid:27450146.
- 7. Corvi F, Pellegrini M, Erba S, Cozzi M, Staurenghi G, Giani A. Reproducibility of Vessel Density, Fractal Dimension, and Foveal Avascular Zone Using 7 Different Optical Coherence Tomography Angiography Devices. Am J Ophthalmol. 2018;186:25–31. pmid:29169882.
- 8. Dong J, Jia YD, Wu Q, Zhang S, Jia Y, Huang D, et al. Interchangeability and reliability of macular perfusion parameter measurements using optical coherence tomography angiography. Br J Ophthalmol. 2017. pmid:28336674.
- 9. Rabiolo A, Gelormini F, Marchese A, Cicinelli MV, Triolo G, Sacconi R, et al. Macular Perfusion Parameters in Different Angiocube Sizes: Does The Size Matter in Quantitative Optical Coherence Tomography Angiography? Invest Ophthalmol Vis Sci. 2018;59(1):231–7. pmid:29340651.
- 10. Rao HL, Dasari S, Riyazuddin M, Puttaiah NK, Pradhan ZS, Weinreb RN, et al. Diagnostic Ability and Structure-function Relationship of Peripapillary Optical Microangiography Measurements in Glaucoma. J Glaucoma. 2018;27(3):219–26. pmid:29329139.
- 11. Toto L, Borrelli E, Mastropasqua R, Di Antonio L, Doronzo E, Carpineto P, et al. Association between outer retinal alterations and microvascular changes in intermediate stage age-related macular degeneration: an optical coherence tomography angiography study. Br J Ophthalmol. 2017;101(6):774–9. pmid:27625163.
- 12. Augstburger E, Zeboulon P, Keilani C, Baudouin C, Labbe A. Retinal and Choroidal Microvasculature in Nonarteritic Anterior Ischemic Optic Neuropathy: An Optical Coherence Tomography Angiography Study. Invest Ophthalmol Vis Sci. 2018;59(2):870–7. pmid:29490340.
- 13. Battaglia Parodi M, Cicinelli MV, Rabiolo A, Pierro L, Bolognesi G, Bandello F. Vascular abnormalities in patients with Stargardt disease assessed with optical coherence tomography angiography. Br J Ophthalmol. 2017;101(6):780–5. pmid:27628426.
- 14. Battaglia Parodi M, Cicinelli MV, Rabiolo A, Pierro L, Gagliardi M, Bolognesi G, et al. Vessel density analysis in patients with retinitis pigmentosa by means of optical coherence tomography angiography. Br J Ophthalmol. 2017;101(4):428–32. pmid:27343210.
- 15. Battaglia Parodi M, Rabiolo A, Cicinelli MV, Iacono P, Romano F, Bandello F. Quantitative Analysis of Optical Coherence Tomography Angiography in Adult-Onset Foveomacular Vitelliform Dystrophy. Retina. 2018;38(2):237–44. pmid:28145974.
- 16. Rabiolo A, Zucchiatti I, Marchese A, Baldin G, Sacconi R, Montorio D, et al. Multimodal retinal imaging in central serous chorioretinopathy treated with oral eplerenone or photodynamic therapy. Eye (Lond). 2018;32(1):55–66. pmid:29265111.
- 17. Battaglia Parodi M, Romano F, Cicinelli MV, Rabiolo A, Arrigo A, Pierro L, et al. Retinal Vascular Impairment in Best Vitelliform Macular Dystrophy Assessed by Means of Optical Coherence Tomography Angiography. Am J Ophthalmol. 2018;187:61–70. pmid:29288639.
- 18. Cicinelli MV, Rabiolo A, Marchese A, de Vitis L, Carnevali A, Querques L, et al. Choroid morphometric analysis in non-neovascular age-related macular degeneration by means of optical coherence tomography angiography. Br J Ophthalmol. 2017;101(9):1193–200. pmid:28057649.
- 19. Chidambara L, Gadde SG, Yadav NK, Jayadev C, Bhanushali D, Appaji AM, et al. Characteristics and quantification of vascular changes in macular telangiectasia type 2 on optical coherence tomography angiography. Br J Ophthalmol. 2016;100(11):1482–8. pmid:26823394.
- 20. Sacconi R, Corbelli E, Carnevali A, Mercuri S, Rabiolo A, Querques L, et al. Optical coherence tomography angiography in pseudophakic cystoid macular oedema compared to diabetic macular oedema: qualitative and quantitative evaluation of retinal vasculature. Br J Ophthalmol. 2018. pmid:29463502.
- 21. Zhang S, Wu C, Liu L, Jia Y, Zhang Y, Zhang Y, et al. Optical Coherence Tomography Angiography of the Peripapillary Retina in Primary Angle-Closure Glaucoma. Am J Ophthalmol. 2017;182:194–200. pmid:28797550.
- 22. Triolo G, Rabiolo A, Shemonski ND, Fard A, Di Matteo F, Sacconi R, et al. Optical Coherence Tomography Angiography Macular and Peripapillary Vessel Perfusion Density in Healthy Subjects, Glaucoma Suspects, and Glaucoma Patients. Invest Ophthalmol Vis Sci. 2017;58(13):5713–22. pmid:29114838.
- 23. Uji A, Balasubramanian S, Lei J, Baghdasaryan E, Al-Sheikh M, Sadda SR. Impact of Multiple En Face Image Averaging on Quantitative Assessment from Optical Coherence Tomography Angiography Images. Ophthalmology. 2017;124(7):944–52. pmid:28318637.
- 24. Sandhu HS, Eladawi N, Elmogy M, Keynton R, Helmy O, Schaal S, et al. Automated diabetic retinopathy detection using optical coherence tomography angiography: a pilot study. Br J Ophthalmol. 2018. pmid:29363532.
- 25. Eladawi N, Elmogy M, Helmy O, Aboelfetouh A, Riad A, Sandhu H, et al. Automatic blood vessels segmentation based on different retinal maps from OCTA scans. Comput Biol Med. 2017;89:150–61. pmid:28806613.
- 26. Kim AY, Chu Z, Shahidzadeh A, Wang RK, Puliafito CA, Kashani AH. Quantifying Microvascular Density and Morphology in Diabetic Retinopathy Using Spectral-Domain Optical Coherence Tomography Angiography. Invest Ophthalmol Vis Sci. 2016;57(9):OCT362–70. pmid:27409494.
- 27. Huang Y, Zhang Q, Thorell MR, An L, Durbin MK, Laron M, et al. Swept-source OCT angiography of the retinal vasculature using intensity differentiation-based optical microangiography algorithms. Ophthalmic Surg Lasers Imaging Retina. 2014;45(5):382–9. pmid:25230403.
- 28. Roisman L, Zhang Q, Wang RK, Gregori G, Zhang A, Chen CL, et al. Optical Coherence Tomography Angiography of Asymptomatic Neovascularization in Intermediate Age-Related Macular Degeneration. Ophthalmology. 2016;123(6):1309–19. pmid:26876696.
- 29. Koo TK, Li MY. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med. 2016;15(2):155–63. pmid:27330520.
- 30. Medeiros FA, Lisboa R, Weinreb RN, Girkin CA, Liebmann JM, Zangwill LM. A combined index of structure and function for staging glaucomatous damage. Arch Ophthalmol. 2012;130(5):E1–10. pmid:22826832.
- 31. Durbin MK, An L, Shemonski ND, Soares M, Santos T, Lopes M, et al. Quantification of Retinal Microvascular Density in Optical Coherence Tomographic Angiography Images in Diabetic Retinopathy. JAMA Ophthalmol. 2017;135(4):370–6. pmid:28301651.
- 32. Samara WA, Shahlaee A, Adam MK, Khan MA, Chiang A, Maguire JI, et al. Quantification of Diabetic Macular Ischemia Using Optical Coherence Tomography Angiography and Its Relationship with Visual Acuity. Ophthalmology. 2017;124(2):235–44. pmid:27887743.
- 33. Samara WA, Shahlaee A, Sridhar J, Khan MA, Ho AC, Hsu J. Quantitative Optical Coherence Tomography Angiography Features and Visual Function in Eyes With Branch Retinal Vein Occlusion. Am J Ophthalmol. 2016;166:76–83. pmid:27038893.
- 34. Cicinelli MV, Rabiolo A, Sacconi R, Lamanna F, Querques L, Bandello F, et al. Retinal vascular alterations in reticular pseudodrusen with and without outer retinal atrophy assessed by optical coherence tomography angiography. Br J Ophthalmol. 2018. pmid:29363531.
- 35. Garrity ST, Sarraf D. The Arc of Change in Optical Coherence Tomographic Angiography Technology: Progression Toward Greater Reliability. JAMA Ophthalmol. 2017;135(10):1098–9. pmid:28910424.
- 36. Manalastas PIC, Zangwill LM, Saunders LJ, Mansouri K, Belghith A, Suh MH, et al. Reproducibility of Optical Coherence Tomography Angiography Macular and Optic Nerve Head Vascular Density in Glaucoma and Healthy Eyes. J Glaucoma. 2017;26(10):851–9. pmid:28858159.
- 37. Uji A, Balasubramanian S, Lei J, Baghdasaryan E, Al-Sheikh M, Borrelli E, et al. Multiple enface image averaging for enhanced optical coherence tomography angiography imaging. Acta Ophthalmol. 2018. pmid:29855147.
- 38. Pedinielli A, Bonnin S, Sanharawi ME, Mane V, Erginay A, Couturier A, et al. Three Different Optical Coherence Tomography Angiography Measurement Methods for Assessing Capillary Density Changes in Diabetic Retinopathy. Ophthalmic Surg Lasers Imaging Retina. 2017;48(5):378–84. pmid:28499048.
- 39. Coscas F, Sellam A, Glacet-Bernard A, Jung C, Goudot M, Miere A, et al. Normative Data for Vascular Density in Superficial and Deep Capillary Plexuses of Healthy Adults Assessed by Optical Coherence Tomography Angiography. Invest Ophthalmol Vis Sci. 2016;57(9):OCT211–23. pmid:27409475.
- 40. Iafe NA, Phasukkijwatana N, Chen X, Sarraf D. Retinal Capillary Density and Foveal Avascular Zone Area Are Age-Dependent: Quantitative Analysis Using Optical Coherence Tomography Angiography. Invest Ophthalmol Vis Sci. 2016;57(13):5780–7. pmid:27792812.
- 41. Garrity ST, Iafe NA, Phasukkijwatana N, Chen X, Sarraf D. Quantitative Analysis of Three Distinct Retinal Capillary Plexuses in Healthy Eyes Using Optical Coherence Tomography Angiography. Invest Ophthalmol Vis Sci. 2017;58(12):5548–55. pmid:29075766.
- 42. Bazvand F, Mirshahi R, Fadakar K, Faghihi H, Sabour S, Ghassemi F. The Quantitative Measurements of Vascular Density and Flow Area of Optic Nerve Head Using Optical Coherence Tomography Angiography. J Glaucoma. 2017;26(8):735–41. pmid:28692595.
- 43. Wang Q, Chan S, Yang JY, You B, Wang YX, Jonas JB, et al. Vascular Density in Retina and Choriocapillaris as Measured by Optical Coherence Tomography Angiography. Am J Ophthalmol. 2016;168:95–109. pmid:27183862.
- 44. Borrelli E, Lonngi M, Balasubramanian S, Tepelus TC, Baghdasaryan E, Iafe NA, et al. Macular Microvascular Networks in Healthy Pediatric Subjects. Retina. 2018. pmid:29474304.
- 45. Ghassemi F, Fadakar K, Bazvand F, Mirshahi R, Mohebbi M, Sabour S. The Quantitative Measurements of Vascular Density and Flow Areas of Macula Using Optical Coherence Tomography Angiography in Normal Volunteers. Ophthalmic Surg Lasers Imaging Retina. 2017;48(6):478–86. pmid:28613354.
- 46. Sampson DM, Gong P, An D, Menghini M, Hansen A, Mackey DA, et al. Axial Length Variation Impacts on Superficial Retinal Vessel Density and Foveal Avascular Zone Area Measurements Using Optical Coherence Tomography Angiography. Invest Ophthalmol Vis Sci. 2017;58(7):3065–72. pmid:28622398.