Deep Learning for Accurate Diagnosis of Liver Tumor Based on Magnetic Resonance Imaging and Clinical Data - PubMed (original) (raw)
doi: 10.3389/fonc.2020.00680. eCollection 2020.
Ming Cheng 2, Yu-Bo Tao 2, Yi-Fan Wang 1, Sarun Juengpanich 1, Zhi-Yu Jiang 1, Yan-Kai Jiang 1 2, Yu-Yu Yan 2, Wei Lu 3 4, Jie-Min Lue 1, Jia-Hong Qian 2, Zhong-Yu Wu 5, Ji-Hong Sun 3, Hai Lin 2, Xiu-Jun Cai 1
Affiliations
- PMID: 32547939
- PMCID: PMC7271965
- DOI: 10.3389/fonc.2020.00680
Deep Learning for Accurate Diagnosis of Liver Tumor Based on Magnetic Resonance Imaging and Clinical Data
Shi-Hui Zhen et al. Front Oncol. 2020.
Abstract
Background: Early-stage diagnosis and treatment can improve survival rates of liver cancer patients. Dynamic contrast-enhanced MRI provides the most comprehensive information for differential diagnosis of liver tumors. However, MRI diagnosis is affected by subjective experience, so deep learning may supply a new diagnostic strategy. We used convolutional neural networks (CNNs) to develop a deep learning system (DLS) to classify liver tumors based on enhanced MR images, unenhanced MR images, and clinical data including text and laboratory test results. Methods: Using data from 1,210 patients with liver tumors (N = 31,608 images), we trained CNNs to get seven-way classifiers, binary classifiers, and three-way malignancy-classifiers (Model A-Model G). Models were validated in an external independent extended cohort of 201 patients (N = 6,816 images). The area under receiver operating characteristic (ROC) curve (AUC) were compared across different models. We also compared the sensitivity and specificity of models with the performance of three experienced radiologists. Results: Deep learning achieves a performance on par with three experienced radiologists on classifying liver tumors in seven categories. Using only unenhanced images, CNN performs well in distinguishing malignant from benign liver tumors (AUC, 0.946; 95% CI 0.914-0.979 vs. 0.951; 0.919-0.982, P = 0.664). New CNN combining unenhanced images with clinical data greatly improved the performance of classifying malignancies as hepatocellular carcinoma (AUC, 0.985; 95% CI 0.960-1.000), metastatic tumors (0.998; 0.989-1.000), and other primary malignancies (0.963; 0.896-1.000), and the agreement with pathology was 91.9%.These models mined diagnostic information in unenhanced images and clinical data by deep-neural-network, which were different to previous methods that utilized enhanced images. The sensitivity and specificity of almost every category in these models reached the same high level compared to three experienced radiologists. Conclusion: Trained with data in various acquisition conditions, DLS that integrated these models could be used as an accurate and time-saving assisted-diagnostic strategy for liver tumors in clinical settings, even in the absence of contrast agents. DLS therefore has the potential to avoid contrast-related side effects and reduce economic costs associated with current standard MRI inspection practices for liver tumor patients.
Keywords: MRI; artificial intelligence; deep learning; diagnosis; liver cancer; liver mass.
Copyright © 2020 Zhen, Cheng, Tao, Wang, Juengpanich, Jiang, Jiang, Yan, Lu, Lue, Qian, Wu, Sun, Lin and Cai.
Figures
Figure 1
Data and strategy. (A), Number of patients and images per category. (B), Strategy for development and validation. (B, i), Magnetic resonance images of patients in training set were first downloaded from the PACS; Liver tumors was outlined in related images of T2-weighted sequences as Regions of interest (ROI) by ITK-SNAP software; pre-processed images and obtained six different scan sequences pictures for each cross section of the lesion (T2, diffusion,T1 pre-contrast, late arterial phase, portal venous phase, equilibrium phase); (B, ii), six sequences of each cross section input to CNN as a whole-image from six channels, encoded clinical data was input to CNN; (B, iii), the Inception-ResNetV2 architecture was used and fully trained using the training set or partly retrained using the new-training set with clinical data; (B, iv), classifications were performed on images from an independent validation set, and the values were finally aggregated per patient to extract the T-SNE and the statistics; (B, v), Clinical data was encoded using one-hot encoding as preparation for three-way malignancy classifier. HH, Hepatic hemangioma; Nodules, other benign nodules; Metastatic, Metastatic malignancy from other sites; Primary, Primary malignancy except HCC.
Figure 2
Performance of CNN models and radiologists in external validation set. (A–C) Receiver operating characteristic (ROC) curves in the validation set (n = 201 patients). (A) Model A: seven-way classifier with six sequences. (B) Model B: seven-way classifier with three unenhanced sequences. (C) Model C,D: binary classifier for benign and malignancy with six sequences and three unenhanced sequences. (D–F) ROC curves in the new validation set of malignant tumors (n = 99 patients). (D–F), (D) Model E: three-way classifier with six sequences. (E) Model F: three-way classifier with six sequences and clinical data. (F) Model G: three-way classifier with three sequences and clinical data. The crosses indicate the performance of average radiologists for each category, the length of the cross represents the confidence Interval (CI).
Figure 3
Illustration of classifiers learned by deep-learning projected to 2 dimensions for visualization via the t-SNE algorithm using values of the last fully connected layer in the CNNs of the validation set. (A–C), Scatterplots where each point represents an image of lesions and the color represents the true category, show how the algorithm clusters. (A), Model A: seven-way classifier with six sequences images, shows that seven clusters of the same clinical classes, and we can see benign tumor clusters are better than that of three malignant tumors. The purple point clouds(benign nodules) are effectively divided from red point clouds (HCC). (B), Model E: three-way classifier with six sequences images for malignant tumors. (C), Model G: three-way classifier with three sequences images and clinical data for malignant tumors, shows that three different color point clouds are more effectively clustered than Model E. (D) Insets of T2 images show some categories. (i), Hepatocellular carcinoma (ii), Metastatic malignant tumors from pancreas (iii), Intrahepatic cholangiocarcinoma (iv) DN that are difficult to identify with HCC. (v) malignant fibrous histiocytoma represented by outlier point clouds of c(d,v).
Figure 4
Saliency map for example images from seven categories of the validation set and a special case which not appeared in training set. These maps reveal the pixels that most influence a CNN's prediction. Saliency maps show the pixel gradients with relative to the CNN loss function. Darker pixels represent pixels with greater influence. Clear correlation between lesions and saliency maps are revealed. We selected T2 image as the original control, the middle is reconstructed image of three sequences (the left column is from three plain scan sequences, the right column is from three enhanced sequences), the right is a corresponding saliency map. (A) cyst, (B) FNH, (C) hemangioma, (D) benign nodule, (E) HCC, (F) primary adenocarcinoma, (G) metastatic malignancy originating from pancreas, (H) malignant fibrous histiocytoma, which still gains a good display although this rare type did not appear in the training set.
References
LinkOut - more resources
Full Text Sources