https://orcid.org/0000-0002-2057-5261
Figures
Abstract
The recent medical applications of deep-learning (DL) algorithms have demonstrated their clinical efficacy in improving speed and accuracy of image interpretation. If the DL algorithm achieves a performance equivalent to that achieved by physicians in chest radiography (CR) diagnoses with Coronavirus disease 2019 (COVID-19) pneumonia, the automatic interpretation of the CR with DL algorithms can significantly reduce the burden on clinicians and radiologists in sudden surges of suspected COVID-19 patients. The aim of this study was to evaluate the efficacy of the DL algorithm for detecting COVID-19 pneumonia on CR compared with formal radiology reports. This is a retrospective study of adult patients that were diagnosed as positive COVID-19 cases based on the reverse transcription polymerase chain reaction among all the patients who were admitted to five emergency departments and one community treatment center in Korea from February 18, 2020 to May 1, 2020. The CR images were evaluated with a publicly available DL algorithm. For reference, CR images without chest computed tomography (CT) scans classified as positive for COVID-19 pneumonia were used given that the radiologist identified ground-glass opacity, consolidation, or other infiltration in retrospectively reviewed CR images. Patients with evidence of pneumonia on chest CT scans were also classified as COVID-19 pneumonia positive outcomes. The overall sensitivity and specificity of the DL algorithm for detecting COVID-19 pneumonia on CR were 95.6%, and 88.7%, respectively. The area under the curve value of the DL algorithm for the detection of COVID-19 with pneumonia was 0.921. The DL algorithm demonstrated a satisfactory diagnostic performance comparable with that of formal radiology reports in the CR-based diagnosis of pneumonia in COVID-19 patients. The DL algorithm may offer fast and reliable examinations that can facilitate patient screening and isolation decisions, which can reduce the medical staff workload during COVID-19 pandemic situations.
Citation: Jang SB, Lee SH, Lee DE, Park S-Y, Kim JK, Cho JW, et al. (2020) Deep-learning algorithms for the interpretation of chest radiographs to aid in the triage of COVID-19 patients: A multicenter retrospective study. PLoS ONE 15(11): e0242759. https://doi.org/10.1371/journal.pone.0242759
Editor: Raffaele Serra, University Magna Graecia of Catanzaro, ITALY
Received: September 24, 2020; Accepted: November 10, 2020; Published: November 24, 2020
Copyright: © 2020 Jang et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: De-identified participant data are within the manuscript and its Supporting Information files.
Funding: This work was supported by the Research Program of Medicity Daegu Council funded by Daegu Metropolitan City (grant number COVID19_DM22) The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
During the outbreak of rapidly transmitted infectious diseases, such as the novel Coronavirus disease 2019 (COVID-19) [1, 2], early isolation through the early detection of suspected patients is the most basic and important response strategy in emergency departments (EDs). Chest radiography (CR) constitutes a fast and relatively inexpensive imaging modality for finding lesions in the lungs. The portability of CR equipment can protect medical personnel from viruses and minimize the risk of spreading the virus [3, 4]. In this regard, Chung et al. [5], reported radiological abnormalities in 93.5% of the patients in EDs during the lockdowns through the pandemic that demonstrates the effectiveness of the clinical symptoms and CR-based revised triage and ED surveillance protocol. COVID-19 lung infection commonly produces ground-glass and consolidative opacities with a bilateral, peripheral, and lower lung distribution. Ground-glass opacity (GGO) lesions exhibit unclear boundaries, and the corresponding images are often not clear; this makes their detection in CRs difficult for non-expert clinicians. Real-time analysis by radiologists may improve the detection rate of COVID-19 pneumonia in CR; however, such analysis is impractical because of time and budgetary constraints.
The recent medical applications of deep learning (DL) algorithms have been attracting increasing attention. In particular, the performances of DL algorithms have attracted attention for the detection of pulmonary malignancy, active tuberculosis, pneumothorax, and pneumonia in CR images [6–9]. Previous research has demonstrated the clinical efficacy of the DL algorithm in terms of its ability to improve speed and accuracy in image reading [7, 10]. If the DL algorithm achieves a performance that is equivalent to that achieved by physicians in the detection of CR with COVID-19 pneumonia, the automatic interpretation of the CR with DL algorithms can significantly reduce the burden on clinicians and radiologists in a sudden surge of suspected COVID-19 patients. The aim of this study was to evaluate the efficacy of DL algorithms for detecting COVID-19 pneumonia on CR compared with radiological reports.
Methods
Study setting and population
By May 1, 2020, there were 6,852 confirmed COVID-19 patients in Daegu, Korea (the first case was detected on February 18, 2020) [11]. Daegu Metropolitan City has 15 emergency medical centers that provide health services to a population of 2.48 million and an area of 883.54 square kilometers. Of these, two level 1 EDs and three level 2 EDs participated in this study. These EDs provide the highest level of emergency care services in the region and annually care approximately 200,000 ED patients, accounting for 42.0% of patients visiting EDs in Daegu Metropolitan City [12]. Herein, we note that this study is a retrospective study of adult patients determined to be positive based on the COVID-19 reverse transcription polymerase chain reaction (RT-PCR) among all the patients who were admitted to five emergency medical centers and one community treatment center [13] in Daegu, Korea from February 18, 2020 to May 1, 2020. Of these, 12 patients who did not undergo CR were excluded from the study. The institutional review board (IRB) of Yeungnam University Medical Center reviewed and approved our study (IRB registration number: 2020-06-019). Subsequently, the IRBs of Daegu Catholic University Medical Center (CR-20-152), Daegu Fatima Hospital approved the study (DFE20ORIO073), Kyungpook National University Hospital (KNUH 2020-05-072), and Kyungpook National University Chilgok Hospital (KNUCH 2020-05-012). The requirement for the acquisition of informed consent was waived owing the retrospective nature of this study.
Reference standards
The diagnosis of pneumonia in the present study was based on radiological findings. CR images without chest computed tomography (CT) scans were classified as positive for COVID-19 pneumonia if the radiologist identified GGO, consolidation, or other infiltration. Patients with evidence of pneumonia on chest CT scans were also classified as positive for COVID-19 pneumonia. In the case of CR images without CT scans, the results were re-evaluated by three thoracic radiologists for expert consensus [14]. The three radiologists independently determined whether patients had radiological evidence of pneumonia or not based on a retrospective review of the CRs. The final judgment was formulated based on the majority agreement of the three radiologists.
Acquisition of CR images
Along with the relevant patient clinical data, the attending physician at each participating hospital used the Digital Imaging and Communications in Medicine standard to retrieve the anonymized initial CR images of COVID-19 patients who visited their ED. Each CR was matched with the formal reading report by a radiologist, and all the images were uploaded to a database after being anonymized. CR images exhibiting typical pneumonia symptoms, including GGO patterns, consolidation, or other infiltration, were classified as positive for pneumonia according to the formal reading report of hospital radiologists; otherwise, they were classified as negative.
Analysis of CR images with DL algorithm
The CR images were evaluated using a commercialized DL algorithm capable of analyzing CR images (Lunit INSIGHT for CR 2, Lunit; accessible at https://insight.lunit.io) that was approved by the Ministry of Food and Drug Safety of Korea [7, 15]. The algorithm was developed to detect major thoracic diseases, including pulmonary malignancy, active pulmonary tuberculosis, pneumonia, and pneumothorax [7]. In our approach, upon the input of a CR image, classification predictions of the image are generated, and the neural network features of each slice of the CR image are combined by means of a max pooling operation. The resulting feature map is fed to a connected layer. The resulting output is a probability score for the class. The abnormality score of the DL algorithm reflects the likelihood of the presence of lesions. In the study, an abnormality score of >15% in which the indicated lesion location was consistent with the location of the actual lesion based on heatmap images was considered positive; whereas, abnormality scores >15% in which the indicated lesion location was not related to the actual lesion and abnormality scores <15% were defined as negative. In the study, the location of the lesion was confirmed by overlaying with the input radiograph. The evaluation of the localization accuracy was performed by a board-certified radiologist who reviewed all the heatmap images and determined if the DL algorithm was accurate in its classification. The classifications of the DL algorithm were considered correct when the lesion locations were accurate.
Statistical analysis
The accuracy of the DL algorithm for the reference standard was calculated with a 95% confidence interval based on the DeLong method for the receiver operating characteristic (ROC) curve and area under the ROC curve (AUROC). Accordingly, the sensitivity, specificity, positive predictive values (PPV), and negative predictive values (NPV), were subsequently obtained. The sensitivities and specificities were compared using McNemar’s test, and the PPVs and NPVs were compared with the generalized score statistics. We also performed a subgroup analysis of COVID-19 patients who underwent chest CT scans. Statistical tests were performed with the use of SAS (version 9.4, SAS Institute Inc., Cary, NC, USA) and MedCalc (version 19.4.1, MedCalc Software, Ostend, Belgium). The results were considered to be statistically significant when the p-value was less than 0.05.
Results
Clinical characteristics of patients with COVID-19
In total, 279 patients were divided into two groups according to the interpretation of the CR images. The “COVID-19-with-pneumonia” group included 182 patients and the “COVID-19-without-pneumonia” group included 97 patients (Table 1). Among the 279 COVID-19 RT-PCR confirmed patients, 92 patients were diagnosed with COVID-19 with pneumonia, and 16 patients were diagnosed with COVID-19 without pneumonia as confirmed by chest CT scans (Fig 1). Of the 170 patients without CT scans, 91 patients were classified into the COVID-19-with-pneumonia group and 20 patients were classified into the COVID-19-without-pneumonia group based on the review of the CR images by a thoracic radiologist (Fig 1).
CR, chest radiography; CT, computed tomography; RT-PCR, reverse transcription polymerase chain reaction.
The proportion of patients aged 65 years and older was 73.1% in the COVID-19-with-pneumonia group and 17.5% in the COVID-19-without-pneumonia group. In the COVID-19-with-pneumonia group, fever was the most common presenting symptom, followed by dyspnea, cough, and sputum. The numbers of asymptomatic patients were 4 (2.2%) and 54 (55.7%) in the COVID-19-with- and COVID-19-without-pneumonia groups, respectively.
Performance of DL algorithm on chest radiograph images
The sensitivity and specificity of the DL algorithm were 95.6%, and 88.7%, respectively. No significant difference was observed in the AUROC value of the DL algorithm for the detection of COVID-19 with pneumonia compared with the radiology report (P = 0.322) (Table 2). Fig 2 shows the representative cases of CR and CT scans of the COVID-19-with-pneumonia group. Two of the illustrated cases show diffuse bilateral consolidation compatible with pneumonia. Another illustrated case shows right lower lung consolidation that was subsequently confirmed by CT scans. We note that each case is suitably localized and detected by the DL algorithm.
(A, B, C) Chest radiography (D, E, F) DL algorithm heatmap overlaid on the image feature related to pneumonia. (G, H) Computed tomography scan, showing lung consolidation and ground-glass opacities that are suitably localized and detected by the DL algorithm.
In subgroup analysis of patients on whom chest CT scans were performed, the sensitivity and specificity of the DL algorithm for detection of COVID-19 with and without pneumonia on CR were 74.9%, and 84.6%, respectively. The AUROC value of the DL algorithm and radiology report were 0.749 and 0.849, respectively (P = 0.258) (Table 3).
False-positive interpretations by the DL algorithm
There were 11 false-positive results obtained with the DL algorithm: normal vascular marking (n = 6), increased vascular marking (n = 2), emphysematous lung (n = 1), interstitial thickening (n = 1), and subsegmental atelectasis (n = 1). These results are shown in Fig 3.
DL algorithm heatmap overlaid on image feature related to (A) interstitial thickening and (B, C) normal vascular marking.
False-negative interpretations by the DL algorithm
There were eight false-negative results obtained with the DL algorithm. One of the illustrated cases was interpreted as positive for pneumonia in the radiology report. A chest CT scan also showed small focal GGOs in the right middle lobe and multifocal consolidations in the right lower lobe. This case was reported as negative for pneumonia in both the radiology report and the DL algorithm. Another illustrated case shows a small amount of multifocal GGOs in chest CT scans (Fig 4).
(A, B) DL algorithm classified chest radiography as negative for pneumonia. Chest computed tomography scans show (C) focal GGOs in the right middle lobe, (D) multifocal consolidations in the right lower lobe, and (E, F) small amounts of GGOs.
Discussion
The purpose of this study was to examine the performance of a DL algorithm that detects COVID-19 pneumonia from CR images compared with that of radiology reports. This study also validated the diagnostic performance and efficacy of a DL algorithm in analyzing the CR images of COVID-19 patients based on a multicenter study. The DL algorithm used in the study demonstrated a satisfactory diagnostic performance (AUROC = 0.921) in diagnosing pneumonia from the CR images of COVID-19 patients.
Regarding the detection of COVID-19 with pneumonia based on chest CT, both the DL algorithm and radiology reports yielded respective sensitivities of 93.5% and 88%. In this regard, Hurt et al. [16] reported that DL algorithms can have a higher applicable value for clinical diagnostic workflow based on their DL-algorithm-based CR analysis of five COVID-19 patients. According to previous studies performed in China [17, 18], chest CT yielded a high sensitivity of 97% and a low rate of 3.9% for missed diagnosis cases of COVID-19. Therefore, the sensitivity of the DL algorithm may be acceptable for use in clinical practice in COVID-19 pandemic situations.
February 18, 2020, demarcated the onset of the COVID-19 pandemic in Daegu, Korea. Since then, the median time of ED admissions to diagnosis of COVID-19 through RT-PCR was approximately 7 h in cases that resulted in ED temporary closure [5]. Additionally, the guidelines of the World Health Organization suggest the use of chest imaging for the diagnostic workup in symptomatic patients with suspected COVID-19 when RT-PCR testing is available but results are delayed [19]. Given that the timely diagnosis by RT-PCR can be limiting in a pandemic situation, and the fact that radiologic abnormality may precede the conclusive RT-PCR positive outcome, CR imaging may help the early diagnosis or triage of patients in pandemic situations. Moreover, considering the comparable diagnostic ability of the DL algorithm to that of CR for the detection of COVID-19 pneumonia, the observed clinical symptoms in conjunction with DL-based CR analysis may be useful in facilitating rapid decisions regarding in-hospital isolation, treatment facilities, or self-quarantine orders in EDs or screening clinics, particularly during pandemics.
In this study, there was a high proportion of relatively young asymptomatic patients in the COVID-19-without-pneumonia group. Approximately 7% of the asymptomatic patients were found to exhibit pneumonia as per CR, which clearly indicates that CR has limited value for diagnosing COVID-19 in asymptomatic patients. These results support the hypothesis that the efficacy of the image modality in the diagnosis can change depending on the outbreak and epidemic-phase status of nations [20].
There are several limitations to this study. First, this study was not a randomized, controlled study; thus, a potential for bias, which is characteristic of retrospective studies, may be present. Second, there was a difference in the period of time between the onset of symptoms and CR imaging [21]. Accordingly, the DL-algorithm-based CR results, clinical symptoms, and hemodynamic status should be considered in the classification of patients and their disposition. Third, there were 6,852 confirmed cases of COVID-19 in Daegu, but only 279 patients were included in the analysis. It is likely that a higher proportion of relatively severely ill patients who visited the EDs were included. Despite the fact that patients from the community treatment center who had mild symptoms or were asymptomatic were also included in the analysis, it is possible that there may have been selection bias. Fourth, the performance of a diagnostic test may vary depending on the characteristics of the target population. Although the algorithm yielded an acceptable performance in the present study, it may exhibit a lower performance in populations with a higher proportion of mild cases, or a lower prevalence. Fifth, this study comprised only confirmed COVID-19 patients and evaluated performance for the detection of pneumonia, not for the detection of COVID-19. Considering that a proportion of the patients did not exhibit pneumonia on CR, the performance of the DL algorithm for the identification of COVID-19 may be limited. Sixth, the successful use of CR in diagnosis depends on the different phases of the epidemic outbreak and the environments with varying critical resource availability [19, 20, 22]. Therefore, there may be limits to the generalization of the classification system based on the DL algorithm regarding its potential use in other countries.
Conclusions
In this study, the DL algorithm demonstrated a satisfactory diagnostic performance comparable with that of radiology reports in the CR-based diagnosis of pneumonia in COVID-19 patients. In pandemic situations, such as the COVID-19, wherein medical resources and personnel are limited, the emergency medical system can be burdened considerably. In this context, the DL algorithm offers fast and reliable examinations that can facilitate decisions regarding patient screening and isolation, which can reduce the workload on medical staff.
Supporting information
S1 Dataset. De-identified participant dataset.
https://doi.org/10.1371/journal.pone.0242759.s001
(SAS7BDAT)
References
- 1.
Organization WH. WHO Director-General’s remarks at the media briefing on 2019-nCoV on 11 February 2020. World Health Organization, Geneva Available via https://wwwwhoint/dg/speeches/detail/who-director-general-s-remarks-at-the-media-briefing-on-2019-ncov-on-11-february-2020 Accessed. 2020;10.
- 2. Zhu N, Zhang D, Wang W, Li X, Yang B, Song J, et al. A Novel Coronavirus from Patients with Pneumonia in China, 2019. N Engl J Med. 2020;382(8):727–33. Epub 2020/01/25. pmid:31978945.
- 3. Dennie C, Hague C, Lim RS, Manos D, Memauri BF, Nguyen ET, et al. The Canadian Society of Thoracic Radiology (CSTR) and Canadian Association of Radiologists (CAR) Consensus Statement Regarding Chest Imaging in Suspected and Confirmed COVID-19. Can Assoc Radiol J. 2020.
- 4.
Radiology ACo. ACR recommendations for the use of chest radiography and computed tomography (CT) for suspected COVID-19 infection. ACR website Advocacy-and Economics/ACR-Position-Statements/Recommendations-for-Chest-Radiography-and-CTfor-Suspected-COVID19-Infection Updated March. 2020;22.
- 5. Chung HS, Lee DE, Kim JK, Yeo IH, Kim C, Park J, et al. Revised Triage and Surveillance Protocols for Temporary Emergency Department Closures in Tertiary Hospitals as a Response to COVID-19 Crisis in Daegu Metropolitan City. J Korean Med Sci. 2020;35(19):e189. Epub 2020/05/19. pmid:32419401; PubMed Central PMCID: PMC7234857.
- 6. Hwang EJ, Nam JG, Lim WH, Park SJ, Jeong YS, Kang JH, et al. Deep Learning for Chest Radiograph Diagnosis in the Emergency Department. Radiology. 2019;293(3):573–80. Epub 2019/10/23. pmid:31638490.
- 7. Hwang EJ, Park S, Jin KN, Kim JI, Choi SY, Lee JH, et al. Development and Validation of a Deep Learning-Based Automated Detection Algorithm for Major Thoracic Diseases on Chest Radiographs. JAMA Netw Open. 2019;2(3):e191095. Epub 2019/03/23. pmid:30901052; PubMed Central PMCID: PMC6583308.
- 8. Lakhani P, Sundaram B. Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology. 2017;284(2):574–82. pmid:28436741
- 9. Nam JG, Park S, Hwang EJ, Lee JH, Jin K-N, Lim KY, et al. Development and validation of deep learning–based automatic detection algorithm for malignant pulmonary nodules on chest radiographs. Radiology. 2019;290(1):218–28. pmid:30251934
- 10. Kim JH, Kim JY, Kim GH, Kang D, Kim IJ, Seo J, et al. Clinical Validation of a Deep Learning Algorithm for Detection of Pneumonia on Chest Radiographs in Emergency Department Patients with Acute Febrile Respiratory Illness. Journal of Clinical Medicine. 2020;9(6). pmid:32599874
- 11.
Current status of COVID-19 in Daegu: The official wbsite of Dagu Metropolitan City; [cited 2020 July 24]. Available from: http://covid19.daegu.go.kr/00936642View1.html.
- 12.
Emergency Medical Statistics 2018 Yearbook: National Emergency Medical Center of Korea; [cited 2020 October 28]. Available from: https://www.e-gen.or.kr/nemc/statistics_annual_report.do.
- 13. Park PG, Kim CH, Heo Y, Kim TS, Park CW, Kim CH. Out-of-Hospital Cohort Treatment of Coronavirus Disease 2019 Patients with Mild Symptoms in Korea: an Experience from a Single Community Treatment Center. J Korean Med Sci. 2020;35(13):e140. Epub 2020/04/04. pmid:32242347; PubMed Central PMCID: PMC7131899.
- 14. England JR, Cheng PM. Artificial Intelligence for Medical Image Analysis: A Guide for Authors and Reviewers. AJR Am J Roentgenol. 2019;212(3):513–9. Epub 2018/12/18. pmid:30557049.
- 15. Hwang EJ, Kim H, Yoon SH, Goo JM, Park CM. Implementation of a Deep Learning-Based Computer-Aided Detection System for the Interpretation of Chest Radiographs in Patients Suspected for COVID-19. Korean J Radiol. 2020;21(10):1150–60. Epub 2020/07/31. pmid:32729263; PubMed Central PMCID: PMC7458860.
- 16. Hurt B, Kligerman S, Hsiao A. Deep Learning Localization of Pneumonia: 2019 Coronavirus (COVID-19) Outbreak. J Thorac Imaging. 2020;35(3):W87–W9. Epub 2020/03/25. pmid:32205822; PubMed Central PMCID: PMC7180130.
- 17. Ai T, Yang Z, Hou H, Zhan C, Chen C, Lv W, et al. Correlation of Chest CT and RT-PCR Testing for Coronavirus Disease 2019 (COVID-19) in China: A Report of 1014 Cases. Radiology. 2020;296(2):E32–E40. Epub 2020/02/27. pmid:32101510; PubMed Central PMCID: PMC7233399.
- 18. Li Y, Xia L. Coronavirus Disease 2019 (COVID-19): Role of Chest CT in Diagnosis and Management. AJR Am J Roentgenol. 2020;214(6):1280–6. Epub 2020/03/05. pmid:32130038.
- 19. Akl EA, Blazic I, Yaacoub S, Frija G, Chou R, Appiah JA, et al. Use of Chest Imaging in the Diagnosis and Management of COVID-19: A WHO Rapid Advice Guide. Radiology. 2020:203173. Epub 2020/07/31. pmid:32729811; PubMed Central PMCID: PMC7393953.
- 20. Rubin GD, Ryerson CJ, Haramati LB, Sverzellati N, Kanne JP, Raoof S, et al. The Role of Chest Imaging in Patient Management during the COVID-19 Pandemic: A Multinational Consensus Statement from the Fleischner Society. Radiology. 2020;296(1):172–80. Epub 2020/04/08. pmid:32255413; PubMed Central PMCID: PMC7233395.
- 21. Wong HYF, Lam HYS, Fong AH, Leung ST, Chin TW, Lo CSY, et al. Frequency and Distribution of Chest Radiographic Findings in Patients Positive for COVID-19. Radiology. 2020;296(2):E72–E8. Epub 2020/03/29. pmid:32216717; PubMed Central PMCID: PMC7233401.
- 22. Yang W, Sirajuddin A, Zhang X, Liu G, Teng Z, Zhao S, et al. The role of imaging in 2019 novel coronavirus pneumonia (COVID-19). Eur Radiol. 2020. Epub 2020/04/17. pmid:32296940; PubMed Central PMCID: PMC7156903.