Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines - PubMed (original) (raw)
Review
Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines
Shih-Cheng Huang et al. NPJ Digit Med. 2020.
Abstract
Advancements in deep learning techniques carry the potential to make significant contributions to healthcare, particularly in fields that utilize medical imaging for diagnosis, prognosis, and treatment decisions. The current state-of-the-art deep learning models for radiology applications consider only pixel-value information without data informing clinical context. Yet in practice, pertinent and accurate non-imaging data based on the clinical history and laboratory data enable physicians to interpret imaging findings in the appropriate clinical context, leading to a higher diagnostic accuracy, informative clinical decision making, and improved patient outcomes. To achieve a similar goal using deep learning, medical imaging pixel-based models must also achieve the capability to process contextual data from electronic health records (EHR) in addition to pixel data. In this paper, we describe different data fusion techniques that can be applied to combine medical imaging with EHR, and systematically review medical data fusion literature published between 2012 and 2020. We conducted a systematic search on PubMed and Scopus for original research articles leveraging deep learning for fusion of multimodality data. In total, we screened 985 studies and extracted data from 17 papers. By means of this systematic review, we present current knowledge, summarize important results and provide implementation guidelines to serve as a reference for researchers interested in the application of multimodal fusion in medical imaging.
Keywords: Data integration; Machine learning; Medical imaging.
© The Author(s) 2020.
Conflict of interest statement
Competing interestsThe authors declare no competing interests.
Figures
Fig. 1. Timeline of publications in deep learning for medical imaging.
Timeline showing growth in publications on deep learning for medical imaging, found by using the same search criteria on PubMed and Scopus. The figure shows that fusion has only constituted a small, but growing, subset of medical deep learning literature.
Fig. 2. Fusion strategies using deep learning.
Model architecture for different fusion strategies. Early fusion (left figure) concatenates original or extracted features at the input level. Joint fusion (middle figure) also joins features at the input level, but the loss is propagated back to the feature extracting model. Late fusion (right figure) aggregates predictions at the decision level.
Fig. 3. PRISMA flowchart of the study selection process.
Two authors independently screened all records for eligibility. Seventeen studies were included in the systematic review.
Similar articles
- Artificial intelligence-based methods for fusion of electronic health records and imaging data.
Mohsen F, Ali H, El Hajj N, Shah Z. Mohsen F, et al. Sci Rep. 2022 Oct 26;12(1):17981. doi: 10.1038/s41598-022-22514-4. Sci Rep. 2022. PMID: 36289266 Free PMC article. - Data-driven modeling and prediction of blood glucose dynamics: Machine learning applications in type 1 diabetes.
Woldaregay AZ, Årsand E, Walderhaug S, Albers D, Mamykina L, Botsis T, Hartvigsen G. Woldaregay AZ, et al. Artif Intell Med. 2019 Jul;98:109-134. doi: 10.1016/j.artmed.2019.07.007. Epub 2019 Jul 26. Artif Intell Med. 2019. PMID: 31383477 Review. - Multimodal deep learning for liver cancer applications: a scoping review.
Siam A, Alsaify AR, Mohammad B, Biswas MR, Ali H, Shah Z. Siam A, et al. Front Artif Intell. 2023 Oct 27;6:1247195. doi: 10.3389/frai.2023.1247195. eCollection 2023. Front Artif Intell. 2023. PMID: 37965284 Free PMC article. - Multimodal fusion models for pulmonary embolism mortality prediction.
Cahan N, Klang E, Marom EM, Soffer S, Barash Y, Burshtein E, Konen E, Greenspan H. Cahan N, et al. Sci Rep. 2023 May 9;13(1):7544. doi: 10.1038/s41598-023-34303-8. Sci Rep. 2023. PMID: 37160926 Free PMC article. - Self-supervised learning for medical image classification: a systematic review and implementation guidelines.
Huang SC, Pareek A, Jensen M, Lungren MP, Yeung S, Chaudhari AS. Huang SC, et al. NPJ Digit Med. 2023 Apr 26;6(1):74. doi: 10.1038/s41746-023-00811-0. NPJ Digit Med. 2023. PMID: 37100953 Free PMC article. Review.
Cited by
- Effects of artificial intelligence implementation on efficiency in medical imaging-a systematic literature review and meta-analysis.
Wenderott K, Krups J, Zaruchas F, Weigl M. Wenderott K, et al. NPJ Digit Med. 2024 Sep 30;7(1):265. doi: 10.1038/s41746-024-01248-9. NPJ Digit Med. 2024. PMID: 39349815 Free PMC article. Review. - Multimodal masked siamese network improves chest X-ray representation learning.
Shurrab S, Manzanares AG, E Shamout F. Shurrab S, et al. Sci Rep. 2024 Sep 28;14(1):22516. doi: 10.1038/s41598-024-74043-x. Sci Rep. 2024. PMID: 39341871 Free PMC article. - Parametric seasonal-trend autoregressive neural network for long-term crop price forecasting.
Hong W, Choi SC, Oh S. Hong W, et al. PLoS One. 2024 Sep 26;19(9):e0311199. doi: 10.1371/journal.pone.0311199. eCollection 2024. PLoS One. 2024. PMID: 39325794 Free PMC article. - Applications of Multimodal Artificial Intelligence in Non-Hodgkin Lymphoma B Cells.
Isavand P, Aghamiri SS, Amin R. Isavand P, et al. Biomedicines. 2024 Aug 5;12(8):1753. doi: 10.3390/biomedicines12081753. Biomedicines. 2024. PMID: 39200217 Free PMC article. Review. - Integrated noninvasive diagnostics for prediction of survival in immunotherapy.
Yeghaian M, Bodalal Z, Tareco Bucho TM, Kurilova I, Blank CU, Smit EF, van der Heijden MS, Nguyen-Kim TDL, van den Broek D, Beets-Tan RGH, Trebeschi S. Yeghaian M, et al. Immunooncol Technol. 2024 Jul 9;24:100723. doi: 10.1016/j.iotech.2024.100723. eCollection 2024 Dec. Immunooncol Technol. 2024. PMID: 39185322 Free PMC article.
References
Publication types
LinkOut - more resources
Full Text Sources
Other Literature Sources