Decoding the visual and subjective contents of the human brain - PubMed (original) (raw)
Decoding the visual and subjective contents of the human brain
Yukiyasu Kamitani et al. Nat Neurosci. 2005 May.
Abstract
The potential for human neuroimaging to read out the detailed contents of a person's mental state has yet to be fully explored. We investigated whether the perception of edge orientation, a fundamental visual feature, can be decoded from human brain activity measured with functional magnetic resonance imaging (fMRI). Using statistical algorithms to classify brain states, we found that ensemble fMRI signals in early visual areas could reliably predict on individual trials which of eight stimulus orientations the subject was seeing. Moreover, when subjects had to attend to one of two overlapping orthogonal gratings, feature-based attention strongly biased ensemble activity toward the attended orientation. These results demonstrate that fMRI activity patterns in early visual areas, including primary visual cortex (V1), contain detailed orientation information that can reliably predict subjective perception. Our approach provides a framework for the readout of fine-tuned representations in the human brain and their subjective contents.
Conflict of interest statement
Competing interests statement The authors declare that they have no competing financial interests.
Figures
Fig. 1
Orientation decoder and ensemble orientation selectivity. (a) The orientation decoder predicts stimulus orientation based on fMRI activity patterns. The cubes depict an input fMRI activity pattern obtained while the subject viewed gratings of a given orientation (left). The circles are “linear ensemble orientation detectors,” each of which linearly combines the fMRI voxel inputs (weighted sum plus bias; the bias component not shown). The weights (W) are determined by a statistical learning algorithm (linear support vector machine) applied to a training data set, such that the output of each detector becomes largest for its “preferred orientation” (θ_i_). The final unit (rectangle with “Max”) decides the prediction to be the preferred orientation of the detector with the highest value. (b) Orientation selectivity of individual voxels and linear ensemble orientation detectors. The decoder was trained using actual fMRI responses to eight orientations (S1, 400 voxels from V1/V2). Average responses are plotted as a function of orientation for two representative voxels, and for 45° and 135° detectors (error bar, standard deviation).
Fig. 2
Decoding stimulus orientation from ensemble fMRI activity in the visual cortex. (a) Decoded orientation responses for eight orientations. Polar plots indicate the distribution of predicted orientations for each of eight orientations (S2, 400 voxels from V1/V2, 22 samples per orientation). The same values are plotted at symmetrical directions as stimulus orientation repeats every 180°. Solid black lines show the true stimulus orientations. (b) Decoded orientation responses for all four subjects (400 voxels from V1/V2, total 160–192 samples for each subject). Results for individual orientations are pooled relative to the correct orientations, and aligned to the vertical line. (c) Across-session generalization. Decoded orientation responses were obtained by training a decoder with Day 1’s data and testing with Day 2’s data (31 days and 40 days apart for S1 and S2, respectively).
Fig. 3
Orientation selectivity across the human visual pathway. Decoded orientation responses are shown for individual visual areas from V1 through V4 and MT+ (S3, 100 voxels per area). The color map indicates _t_-values associated with the responses to the visual field localizer for V1 through V4, and to the MT+ localizer for MT+ (see Methods). The voxels from both hemispheres were combined to obtain the results, though only the right hemisphere is shown. All other subjects showed similar results of progressively diminishing orientation selectivity in higher areas.
Fig. 4
Pairwise decoding performance as a function of orientation difference (all pairs from eight orientations), for (I) grating images (pixel intensities), (II) fMRI images (voxel intensities), and (III) transformed grating images. The gratings (I) were 20×20 pixel black/white images with 2–3 stripes. The fMRI images (II) were those obtained in the present study (responses to gratings of eight orientations; 400 voxels from V1/V2). The transformed images (III) were created by linear orientation filtering (Gabor-like filters for four orientations) of the grating images (I) followed by thresholding (non-linearity) and addition of noise. The orientations of these images were decoded for each pair of orientations (chance level, 50%). For (I) and (III), the average performance with five sets of phase randomized images is plotted (error bar, standard deviation). For (II), the average performance of four subjects is shown. The grating images (I) resulted in poor performance regardless of orientation difference. In contrast, the fMRI images (II) and the transformed grating images (III) both showed performance that improved with orientation difference, reaching near perfect levels at 90°.
Fig. 5
Orientation preference map on flattened cortical surface. The color maps depict the orientation preference of individual voxels on the flattened surface of left ventral V1 and V2 for subjects S2 and S3 (scale bar, 1 cm). Each cell delineated by thick lines is the cross section of a single voxel (3×3×3 mm) at the gray-white matter boundary. Voxel colors depict the orientation detector for which each voxel provides the largest weight. The overall color map indicates a template pattern that activates each detector most effectively. The weights were calculated using 400 voxels from V1/V2 including all the quadrants. Other subjects also showed scattered but different patterns of orientation preference. Note that the color map indicates only weak preference for one orientation over others. Simple averaging of the voxels with the same orientation preference does not lead to sharp orientation tuning, as found in Fig. 1b (see also Supplementary Fig. 2 online).
Fig. 6
Simulation of one-dimensional array of columns and voxels. Each column was assumed to respond to orientation input according to a Gaussian-tuning function peaking at its preferred orientation (SD, 45°; noise was added to the output). The preferred orientation shifted by a constant degree (a) or by a constant degree plus noise (b). In each trial, a single orientation was given as input, and the outputs of 100,000 columns (color band) were sampled by 100 voxels (gray boxes). The actual location of voxel sampling was randomly jittered on each trial (Gaussian distribution with an SD of a 1/4 voxel size) to take into account residual head motion. The number of stimulation trials was chosen to match the fMRI experiment. The sampled voxel data were analyzed using the same decoding procedure. As can be seen in the polar plots on the right, orientation can be readily decoded from the irregular array of columns (b), but not from the regular array (a). Similar results were obtained with a wide range of simulation parameters. Note that if voxel sampling is no longer jittered to mimic minor brain motion, orientation can be decoded even from the regular column array. This is because the high spatial frequency component can still persist after the sampling by large voxels. However, given that pulsatile brain motion and minor head motion cannot be fully eliminated or corrected with 3D alignment procedures, it seems unlikely that such high-frequency information contributes much to the orientation content in our fMRI data.
Fig. 7
Mind-reading of attended orientation. (a) Mind-reading procedure. First, a decoder was trained using fMRI activity evoked by single gratings to discriminate 45° vs. 135°. Black and white gratings (equal contrast) were counterbalanced across trials. In the attention experiment, a plaid pattern composed of two gratings (black/white counterbalanced) was presented. The color of the central fixation spot was changed to indicate which orientation (45° or 135°) the subject should pay attention to in each trial. The fMRI activity patterns obtained in the attention experiment were classified by the decoder trained with single gratings. (b) Mind-reading performance. Gray lines plot decoded orientation responses for the “attend to 45°” and “attend to 135°” conditions (S3, 800 voxels from V1–V4). Thick lines indicate attended orientations. (c) Mind-reading performance across the human visual pathway. The percentage of correct decoding is plotted by visual area (chance level, 50%; 800 voxels for V1–V4 combined, 200 voxels for V1, V2, V3, and V3a/V4v, and 100 voxels for MT+). Colored lines show the performance of four individual subjects. Black points and lines depict the mean cross-validation performance obtained with single gratings (training session).
Comment in
- Imaging orientation selectivity: decoding conscious perception in V1.
Boynton GM. Boynton GM. Nat Neurosci. 2005 May;8(5):541-2. doi: 10.1038/nn0505-541. Nat Neurosci. 2005. PMID: 15856054 No abstract available.
Similar articles
- Transient attention enhances perceptual performance and FMRI response in human visual cortex.
Liu T, Pestilli F, Carrasco M. Liu T, et al. Neuron. 2005 Feb 3;45(3):469-77. doi: 10.1016/j.neuron.2004.12.039. Neuron. 2005. PMID: 15694332 Free PMC article. - Spatial scale and distribution of neurovascular signals underlying decoding of orientation and eye of origin from fMRI data.
Larsson J, Harrison C, Jackson J, Oh SM, Zeringyte V. Larsson J, et al. J Neurophysiol. 2017 Feb 1;117(2):818-835. doi: 10.1152/jn.00590.2016. Epub 2016 Nov 30. J Neurophysiol. 2017. PMID: 27903637 Free PMC article. - Imaging orientation selectivity: decoding conscious perception in V1.
Boynton GM. Boynton GM. Nat Neurosci. 2005 May;8(5):541-2. doi: 10.1038/nn0505-541. Nat Neurosci. 2005. PMID: 15856054 No abstract available. - The human visual cortex.
Grill-Spector K, Malach R. Grill-Spector K, et al. Annu Rev Neurosci. 2004;27:649-77. doi: 10.1146/annurev.neuro.27.070203.144220. Annu Rev Neurosci. 2004. PMID: 15217346 Review. - Human visual processing as revealed by magnetoencephalography.
Kaneoke Y, Watanabe S, Kakigi R. Kaneoke Y, et al. Int Rev Neurobiol. 2005;68:197-222. doi: 10.1016/S0074-7742(05)68008-7. Int Rev Neurobiol. 2005. PMID: 16443015 Review. No abstract available.
Cited by
- Effects of connectivity hyperalignment (CHA) on estimated brain network properties: from coarse-scale to fine-scale.
Farahani FV, Nebel MB, Wager TD, Lindquist MA. Farahani FV, et al. bioRxiv [Preprint]. 2024 Aug 28:2024.08.27.609817. doi: 10.1101/2024.08.27.609817. bioRxiv. 2024. PMID: 39253413 Free PMC article. Preprint. - Hue and orientation pinwheels in macaque area V4.
Parajuli A, Felleman DJ. Parajuli A, et al. J Neurophysiol. 2024 Aug 1;132(2):589-615. doi: 10.1152/jn.00366.2023. Epub 2024 Jul 11. J Neurophysiol. 2024. PMID: 38988289 - Visual working memories are abstractions of percepts.
Duan Z, Curtis CE. Duan Z, et al. Elife. 2024 May 31;13:RP94191. doi: 10.7554/eLife.94191. Elife. 2024. PMID: 38819426 Free PMC article. - Neuronal tuning and population representations of shape and category in human visual cortex.
Bougou V, Vanhoyland M, Bertrand A, Van Paesschen W, Op De Beeck H, Janssen P, Theys T. Bougou V, et al. Nat Commun. 2024 May 30;15(1):4608. doi: 10.1038/s41467-024-49078-3. Nat Commun. 2024. PMID: 38816391 Free PMC article. - Functional coordinates: Modeling interactions between brain regions as points in a function space.
Poskanzer C, Anzellotti S. Poskanzer C, et al. Netw Neurosci. 2022 Oct 1;6(4):1296-1315. doi: 10.1162/netn_a_00264. eCollection 2022. Netw Neurosci. 2022. PMID: 38800459 Free PMC article.
References
- Paradiso MA. A theory for the use of visual orientation information which exploits the columnar structure of striate cortex. Biol Cybern. 1988;58:35–49. - PubMed
Publication types
MeSH terms
LinkOut - more resources
Full Text Sources
Other Literature Sources
Medical