Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey) (original) (raw)

Visio-Linguistic Brain Encoding

ArXiv, 2022

Enabling effective brain-computer interfaces requires understanding how the human brain encodes stimuli across modalities such as visual, language (or text), etc. Brain encoding aims at constructing fMRI brain activity given a stimulus. There exists a plethora of neural encoding models which study brain encoding for single mode stimuli: visual (pretrained CNNs) or text (pretrained language models). Few recent papers have also obtained separate visual and text representation models and performed late-fusion using simple heuristics. However, previous work has failed to explore: (a) the effectiveness of image Transformer models for encoding visual stimuli, and (b) co-attentive multi-modal modeling for visual and text reasoning. In this paper, we systematically explore the efficacy of image Transformers (ViT, DEiT, and BEiT) and multi-modal Transformers (VisualBERT, LXMERT, and CLIP) for brain encoding. Extensive experiments on two popular datasets, BOLD5000 and Pereira, provide the fol...

A Deep Autoencoder for Near-Perfect fMRI Encoding

2018

Encoding models of functional magnetic resonance imaging (fMRI) data attempt to learn a forward mapping that relates stimuli to the corresponding brain activation. Computational tractability usually forces current encoding as well as decoding solutions to typically consider only a small subset of voxels from the actual 3D volume of activation. Further, while brain decoding has received wider attention, there have been only a few attempts at constructing encoding solutions in the extant neuroimaging literature. In this paper, we present a deep autoencoder consisting of convolutional neural networks in tandem with long short-term memory (CNNLSTM) model. The model is trained on fMRI slice sequences and predicts the entire brain volume rather than a small subset of voxels from the information in stimuli (text and image). We argue that the resulting solution avoids the problem of devising encoding models based on a rule-based selection of informative voxels and the concomitant issue of w...

Construction of Subject-independent Brain Decoders for Human FMRI with Deep Learning

Brain decoding, to decode a stimulus given to or a mental state of human participants from measurable brain activities by means of machine learning techniques, has made a great success in recent years. Due to large variation of brain activities between individuals, however, previous brain decoding studies mostly put focus on developing an individual-specific decoder. For making brain decoding more applicable for practical use, in this study, we explored to build an individualindependent decoder with a large-scale functional magnetic resonance imaging (fMRI) database. We constructed the decoder by deep neural network learning, which is the most successful technique recently developed in the field of data mining. Our decoder achieved the higher decoding accuracy than other baseline methods like support vector machine (SVM). Furthermore, increasing the number of subjects for training led to higher decoding accuracy, as expected. These results show that the deep neural networks trained by large-scale fMRI databases are useful for construction of individual-independent decoders and for their applications for practical use.

Evaluation of Task fMRI Decoding With Deep Learning on a Small Sample Dataset

Frontiers in Neuroinformatics, 2021

Recently, several deep learning methods have been applied to decoding in task-related fMRI, and their advantages have been exploited in a variety of ways. However, this paradigm is sometimes problematic, due to the difficulty of applying deep learning to high-dimensional data and small sample size conditions. The difficulties in gathering a large amount of data to develop predictive machine learning models with multiple layers from fMRI experiments with complicated designs and tasks are well-recognized. Group-level, multi-voxel pattern analysis with small sample sizes results in low statistical power and large accuracy evaluation errors; failure in such instances is ascribed to the individual variability that risks information leakage, a particular issue when dealing with a limited number of subjects. In this study, using a small-size fMRI dataset evaluating bilingual language switch in a property generation task, we evaluated the relative fit of different deep learning models, inco...

Toward a universal decoder of linguistic meaning from brain activation

Nature communications, 2018

Prior work decoding linguistic meaning from imaging data has been largely limited to concrete nouns, using similar stimuli for training and testing, from a relatively small number of semantic categories. Here we present a new approach for building a brain decoding system in which words and sentences are represented as vectors in a semantic space constructed from massive text corpora. By efficiently sampling this space to select training stimuli shown to subjects, we maximize the ability to generalize to new meanings from limited imaging data. To validate this approach, we train the system on imaging data of individual concepts, and show it can decode semantic vector representations from imaging data of sentences about a wide variety of both concrete and abstract topics from two separate datasets. These decoded representations are sufficiently detailed to distinguish even semantically similar sentences, and to capture the similarity structure of meaning relationships between sentences.