Multimodal skin lesion classification using deep learning - PubMed (original) (raw)
. 2018 Nov;27(11):1261-1267.
doi: 10.1111/exd.13777. Epub 2018 Sep 27.
Affiliations
- PMID: 30187575
- DOI: 10.1111/exd.13777
Multimodal skin lesion classification using deep learning
Jordan Yap et al. Exp Dermatol. 2018 Nov.
Abstract
While convolutional neural networks (CNNs) have successfully been applied for skin lesion classification, previous studies have generally considered only a single clinical/macroscopic image and output a binary decision. In this work, we have presented a method which combines multiple imaging modalities together with patient metadata to improve the performance of automated skin lesion diagnosis. We evaluated our method on a binary classification task for comparison with previous studies as well as a five class classification task representative of a real-world clinical scenario. We showed that our multimodal classifier outperforms a baseline classifier that only uses a single macroscopic image in both binary melanoma detection (AUC 0.866 vs 0.784) and in multiclass classification (mAP 0.729 vs 0.598). In addition, we have quantitatively showed the automated diagnosis of skin lesions using dermatoscopic images obtains a higher performance when compared to using macroscopic images. We performed experiments on a new data set of 2917 cases where each case contains a dermatoscopic image, macroscopic image and patient metadata.
Keywords: deep learning; dermatology; dermatoscopy; feature fusion; multimodal.
© 2018 The Authors. Experimental Dermatology Published by John Wiley & Sons Ltd.
Similar articles
- Diagnostic accuracy of content-based dermatoscopic image retrieval with deep classification features.
Tschandl P, Argenziano G, Razmara M, Yap J. Tschandl P, et al. Br J Dermatol. 2019 Jul;181(1):155-165. doi: 10.1111/bjd.17189. Epub 2018 Oct 17. Br J Dermatol. 2019. PMID: 30207594 Free PMC article. - The effects of skin lesion segmentation on the performance of dermatoscopic image classification.
Mahbod A, Tschandl P, Langs G, Ecker R, Ellinger I. Mahbod A, et al. Comput Methods Programs Biomed. 2020 Dec;197:105725. doi: 10.1016/j.cmpb.2020.105725. Epub 2020 Aug 26. Comput Methods Programs Biomed. 2020. PMID: 32882594 - Skin lesion classification with ensembles of deep convolutional neural networks.
Harangi B. Harangi B. J Biomed Inform. 2018 Oct;86:25-32. doi: 10.1016/j.jbi.2018.08.006. Epub 2018 Aug 10. J Biomed Inform. 2018. PMID: 30103029 - Integrating Patient Data Into Skin Cancer Classification Using Convolutional Neural Networks: Systematic Review.
Höhn J, Hekler A, Krieghoff-Henning E, Kather JN, Utikal JS, Meier F, Gellrich FF, Hauschild A, French L, Schlager JG, Ghoreschi K, Wilhelm T, Kutzner H, Heppt M, Haferkamp S, Sondermann W, Schadendorf D, Schilling B, Maron RC, Schmitt M, Jutzi T, Fröhling S, Lipka DB, Brinker TJ. Höhn J, et al. J Med Internet Res. 2021 Jul 2;23(7):e20708. doi: 10.2196/20708. J Med Internet Res. 2021. PMID: 34255646 Free PMC article. Review. - Role of In Vivo Reflectance Confocal Microscopy in the Analysis of Melanocytic Lesions.
Serban ED, Farnetani F, Pellacani G, Constantin MM. Serban ED, et al. Acta Dermatovenerol Croat. 2018 Apr;26(1):64-67. Acta Dermatovenerol Croat. 2018. PMID: 29782304 Review.
Cited by
- Leveraging AI and patient metadata to develop a novel risk score for skin cancer detection.
Islam S, Wishart GC, Walls J, Hall P, Seco de Herrera AG, Gan JQ, Raza H. Islam S, et al. Sci Rep. 2024 Sep 6;14(1):20842. doi: 10.1038/s41598-024-71244-2. Sci Rep. 2024. PMID: 39242690 Free PMC article. - Multimodal data integration for oncology in the era of deep neural networks: a review.
Waqas A, Tripathi A, Ramachandran RP, Stewart PA, Rasool G. Waqas A, et al. Front Artif Intell. 2024 Jul 25;7:1408843. doi: 10.3389/frai.2024.1408843. eCollection 2024. Front Artif Intell. 2024. PMID: 39118787 Free PMC article. Review. - Diagnosing Solid Lesions in the Pancreas With Multimodal Artificial Intelligence: A Randomized Crossover Trial.
Cui H, Zhao Y, Xiong S, Feng Y, Li P, Lv Y, Chen Q, Wang R, Xie P, Luo Z, Cheng S, Wang W, Li X, Xiong D, Cao X, Bai S, Yang A, Cheng B. Cui H, et al. JAMA Netw Open. 2024 Jul 1;7(7):e2422454. doi: 10.1001/jamanetworkopen.2024.22454. JAMA Netw Open. 2024. PMID: 39028670 Free PMC article. Clinical Trial. - Pre-trained multimodal large language model enhances dermatological diagnosis using SkinGPT-4.
Zhou J, He X, Sun L, Xu J, Chen X, Chu Y, Zhou L, Liao X, Zhang B, Afvari S, Gao X. Zhou J, et al. Nat Commun. 2024 Jul 5;15(1):5649. doi: 10.1038/s41467-024-50043-3. Nat Commun. 2024. PMID: 38969632 Free PMC article. - Exploring the influence of transformer-based multimodal modeling on clinicians' diagnosis of skin diseases: A quantitative analysis.
Zhang Y, Hu Y, Li K, Pan X, Mo X, Zhang H. Zhang Y, et al. Digit Health. 2024 May 23;10:20552076241257087. doi: 10.1177/20552076241257087. eCollection 2024 Jan-Dec. Digit Health. 2024. PMID: 38784049 Free PMC article.
Publication types
MeSH terms
LinkOut - more resources
Full Text Sources
Other Literature Sources
Medical