3D reconstructions of brain from MRI scans using neural radiance fields (original) (raw)
Related papers
ArXiv, 2022
Computed tomography (CT) is an effective medical imaging modality, widely used in the field of clinical medicine for the diagnosis of various pathologies. Advances in Multidetector CT imaging technology have enabled additional functionalities, including generation of thin slice multiplanar cross-sectional body imaging and 3D reconstructions. However, this involves patients being exposed to a considerable dose of ionising radiation. Excessive ionising radiation can lead to deterministic and harmful effects on the body. This paper proposes a Deep Learning model that learns to reconstruct CT projections from a few or even a single-view X-ray. This is based on a novel architecture that builds from neural radiance fields, which learns a continuous representation of CT scans by disentangling the shape and volumetric depth of surface and internal anatomical structures from 2D images. Our model is trained on chest and knee datasets, and we demonstrate qualitative and quantitative high-fidel...
Ultra-NeRF: Neural Radiance Fields for Ultrasound Imaging
arXiv (Cornell University), 2023
We present a physics-enhanced implicit neural representation (INR) for ultrasound (US) imaging that learns tissue properties from overlapping US sweeps. Our proposed method leverages a ray-tracing-based neural rendering for novel view US synthesis. Recent publications demonstrated that INR models could encode a representation of a three-dimensional scene from a set of two-dimensional US frames. However, these models fail to consider the view-dependent changes in appearance and geometry intrinsic to US imaging. In our work, we discuss direction-dependent changes in the scene and show that a physics-inspired rendering improves the fidelity of US image synthesis. In particular, we demonstrate experimentally that our proposed method generates geometrically accurate B-mode images for regions with ambiguous representation owing to view-dependent differences of the US images. We conduct our experiments using simulated B-mode US sweeps of the liver and acquired US sweeps of a spine phantom tracked with a robotic arm. The experiments corroborate that our method generates US frames that enable consistent volume compounding from previously unseen views. To the best of our knowledge, the presented work is the first to address view-dependent US image synthesis using INR.
Deep learning-based reconstruction of highly accelerated 3D MRI
arXiv (Cornell University), 2022
To accelerate brain 3D MRI scans by using a deep learning method for reconstructing images from highlyundersampled multi-coil k-space data Methods: DL-Speed, an unrolled optimization architecture with dense skip-layer connections, was trained on 3D T1-weighted brain scan data to reconstruct complex-valued images from highly-undersampled k-space data. The trained model was evaluated on 3D MPRAGE brain scan data retrospectivelyundersampled with a 10-fold acceleration, compared to a conventional parallel imaging method with a 2-fold acceleration. Scores of SNR, artifacts, gray/white matter contrast, resolution/sharpness, deep gray-matter, cerebellar vermis, anterior commissure, and overall quality, on a 5-point Likert scale, were assessed by experienced radiologists. In addition, the trained model was tested on retrospectively-undersampled 3D T1weighted LAVA (Liver Acquisition with Volume Acceleration) abdominal scan data, and prospectively-undersampled 3D MPRAGE and LAVA scans in three healthy volunteers and one, respectively. Results: The qualitative scores for DL-Speed with a 10-fold acceleration were higher than or equal to those for the parallel imaging with 2-fold acceleration. DL-Speed outperformed a compressed sensing method in quantitative metrics on retrospectively-undersampled LAVA data. DL-Speed was demonstrated to perform reasonably well on prospectivelyundersampled scan data, realizing a 2-5 times reduction in scan time. Conclusion: DL-Speed was shown to accelerate 3D MPRAGE and LAVA with up to a net 10-fold acceleration, achieving 2-5 times faster scans compared to conventional parallel imaging and acceleration, while maintaining diagnostic image quality and realtime reconstruction. The brain scan data-trained DL-Speed also performed well when reconstructing abdominal LAVA scan data, demonstrating versatility of the network.
3D Image Reconstruction in Medicine and Beyond
In the medical field, 3D volumes are reconstructed mainly by tomographic techniques. Transmission or emission projection data sets are acquired and processed to reconstruct slices across the volume of interest, e.g., the patient's body. Although initially based on 2D acquisitions, current reconstruction techniques for emission modalities (e.g. PET) use more sensitive 3D acquisition data sets processed through modified or entirely original algorithms. Beyond the simple 3D imaging that single tomographies permit, multimodality approaches and equipment actually help reconstruction algorithms to perform better. Mixing these technical developments with complex clinical imaging protocols provides the foundation for a refined and open-ended multidimensional and multisensor approach to either diagnosis or therapy planning and follow-up. 1.
Medical Transformer: Universal Brain Encoder for 3D MRI Analysis
ArXiv, 2021
Transfer learning has gained attention in medical image analysis due to limited annotated 3D medical datasets for training data-driven deep learning models in the real world. Existing 3D-based methods have transferred the pre-trained models to downstream tasks, which achieved promising results with only a small number of training samples. However, they demand a massive amount of parameters to train the model for 3D medical imaging. In this work, we propose a novel transfer learning framework, called Medical Transformer, that effectively models 3D volumetric images in the form of a sequence of 2D image slices. To make a high-level representation in 3D-form empowering spatial relations better, we take a multi-view approach that leverages plenty of information from the three planes of 3D volume, while providing parameter-efficient training. For building a source model generally applicable to various tasks, we pre-train the model in a self-supervised learning manner for masked encoding ...
3D RECONSTRUCTION AND VOLUME COMPUTING IN MEDICAL IMAGING
Medical imaging domain is a field of interest which concentrates great efforts to offer software tools for the assisted diagnosis. Segmentation, 3D reconstruction and visualization are techniques that allow physicians to observe possible pathological structure inside of the human body. Computing the volume of such a structure offers critical information in the evaluation of the disease gravity. In this paper we present a method for 3D reconstruction of the pathological zone and propose a method for volume computing. We consider this as a start for developing a complex software tool for Deep Brain Stimulation procedure in the neurosurgery domain.
ArXiv, 2021
In the medical field, landmark detection in MRI plays an important role in reducing medical technician efforts in tasks like scan planning, image registration, etc. First, 88 landmarks spread across the brain anatomy in the three respective viewssagittal, coronal, and axial are manually annotated, later guidelines from the expert clinical technicians are taken subanatomy-wise, for better localization of the existing landmarks, in order to identify and locate the important atlas landmarks even in oblique scans. To overcome limited data availability, we implement realistic data augmentation to generate synthetic 3D volumetric data. We use a modified HighRes3DNet model for solving brain MRI volumetric landmark detection problem. In order to visually explain our trained model on unseen data, and discern a stronger model from a weaker model, we implement Gradient-weighted Class Activation Mapping (GradCAM) which produces a coarse localization map highlighting the regions the model is foc...
LITERATURE SURVEY FOR 3D RECONSTRUCTION OF BRAIN MRI IMAGES
Since Doctors had only the 2D Image Data to visualize the tumors in the MRI images, which never gave the actual feel of how the tumor would exactly look like. The doctors were deprived from the exact visualization of the tumor the amount of the tumor to be removed by operation was not known, which caused a lot of deformation in the faces and structure of the patients face or skull. The diversity and complexity of tumor cells makes it very challenging to visualize tumor present in magnetic resonance image (MRI) data. Hence to visualize the tumor properly 2D MRI image has to be converted to 3D image. With the development of computer image processing technology, three-dimensional (3D) visualization has become an important method of the medical diagnose, it offers abundant and accurate information for medical experts. Three-dimensional (3-D) reconstruction of medical images is widely applied to tumor localization; surgical planning and brain electromagnetic field computation etc. The brain MR images have unique characteristics, i.e., very complicated changes of the gray-scales and highly irregular boundaries. Traditional 3-D reconstruction algorithms are challenged in solving this problem. Many reconstruction algorithms, such as marching cubes and dividing cubes, need to establish the topological relationship between the slices of images. The results of these traditional approaches vary depending on the number of input sections, their positions, the shape of the original body and the applied interpolation technique. These make the task tedious and time-consuming. Moreover, satisfied reconstruction result may not even be obtained when the highly irregular objects such as the encephalic tissues are considered. Due to complexity and irregularity of each encephalic tissue boundary, three-dimensional (3D) reconstruction for MRI image is necessary. A Literature survey is done to study different methods of 3D reconstruction of brain images from MRI images.
Deep Convolutional Encoder-Decoder algorithm for MRI brain reconstruction
Medical & Biological Engineering & Computing, 2020
Compressed Sensing Magnetic Resonance Imaging (CS-MRI) could be considered a challenged task since it could be designed as an efficient technique for fast MRI acquisition which could be highly beneficial for several clinical routines. In fact, it could grant better scan quality by reducing motion artifacts amount as well as the contrast washout effect. It offers also the possibility to reduce the exploration cost and the patient's anxiety. Recently, Deep Learning Neuronal Network (DL) has been suggested in order to reconstruct MRI scans with conserving the structural details and improving parallel imaging-based fast MRI. In this paper, we propose Deep Convolutional Encoder-Decoder architecture for CS-MRI reconstruction. Such architecture bridges the gap between the non-learning techniques, using data from only one image, and approaches using large training data. The proposed approach is based on autoencoder architecture divided into two parts: an encoder and a decoder. The encoder as well as the decoder has essentially three convolutional blocks. The proposed architecture has been evaluated through two databases: Hammersmith dataset (for the normal scans) and MICCAI 2018 (for pathological MRI). Moreover, we extend our model to cope with noisy pathological MRI scans. The normalized mean square error (NMSE), the peak-to-noise ratio (PSNR), and the structural similarity index (SSIM) have been adopted as evaluation metrics in order to evaluate the proposed architecture performance and to make a comparative study with the state-of-the-art reconstruction algorithms. The higher PSNR and SSIM values as well as the lowest NMSE values could attest that the proposed architecture offers better reconstruction and preserves textural image details. Furthermore, the running time is about 0.8 s, which is suitable for real-time processing. Such results could encourage the neurologist to adopt it in their clinical routines.
arXiv (Cornell University), 2023
The proliferation of technologies, such as extended reality (XR), has increased the demand for high-quality threedimensional (3D) graphical representations. Industrial 3D applications encompass computer-aided design (CAD), finite element analysis (FEA), scanning, and robotics. However, current methods employed for industrial 3D representations suffer from high implementation costs and reliance on manual human input for accurate 3D modeling. To address these challenges, neural radiance fields (NeRFs) have emerged as a promising approach for learning 3D scene representations based on provided training 2D images. Despite a growing interest in NeRFs, their potential applications in various industrial subdomains are still unexplored. In this paper, we deliver a comprehensive examination of NeRF industrial applications while also providing direction for future research endeavors. We also present a series of proof-ofconcept experiments that demonstrate the potential of NeRFs in the industrial domain. These experiments include NeRFbased video compression techniques and using NeRFs for 3D motion estimation in the context of collision avoidance. In the video compression experiment, our results show compression savings up to 48% and 74% for resolutions of 1920x1080 and 300x168, respectively. The motion estimation experiment used a 3D animation of a robotic arm to train Dynamic-NeRF (D-NeRF) and achieved an average peak signal-to-noise ratio (PSNR) of disparity map with the value of 23 dB and an structural similarity index measure (SSIM) 0.97.