Deep learning for FTIR histology: leveraging spatial and spectral features with convolutional neural networks - PubMed (original) (raw)

Deep learning for FTIR histology: leveraging spatial and spectral features with convolutional neural networks

Sebastian Berisha et al. Analyst. 2019.

Abstract

Current methods for cancer detection rely on tissue biopsy, chemical labeling/staining, and examination of the tissue by a pathologist. Though these methods continue to remain the gold standard, they are non-quantitative and susceptible to human error. Fourier transform infrared (FTIR) spectroscopic imaging has shown potential as a quantitative alternative to traditional histology. However, identification of histological components requires reliable classification based on molecular spectra, which are susceptible to artifacts introduced by noise and scattering. Several tissue types, particularly in heterogeneous tissue regions, tend to confound traditional classification methods. Convolutional neural networks (CNNs) are the current state-of-the-art in image classification, providing the ability to learn spatial characteristics of images. In this paper, we demonstrate that CNNs with architectures designed to process both spectral and spatial information can significantly improve classifier performance over per-pixel spectral classification. We report classification results after applying CNNs to data from tissue microarrays (TMAs) to identify six major cellular and acellular constituents of tissue, namely adipocytes, blood, collagen, epithelium, necrosis, and myofibroblasts. Experimental results show that the use of spatial information in addition to the spectral information brings significant improvements in the classifier performance and allows classification of cellular subtypes, such as adipocytes, that exhibit minimal chemical information but have distinct spatial characteristics. This work demonstrates the application and efficiency of deep learning algorithms in improving the diagnostic techniques in clinical and research activities related to cancer.

PubMed Disclaimer

Conflict of interest statement

Conflicts of interest

There are no conflicts to declare.

Figures

Fig. 1

Fig. 1

Chemically stained (a–d) and mid-infrared (e) images of two breast biopsy cores. Individual cores from two separate patients are shown with tissue stained with Massson’s trichrome (a) and H & E (b), as well as immunohistochemical labels for cytokeratin (c) and vimentin (d). Colormapped mid-infrared images of the corresponding two cores are shown (e), where color indicates the magnitude of the absorbance spectrum in arbitrary units at 1650 cm−1.

Fig. 2

Fig. 2

Spatial visual differences between different cell types. Cropped regions around pixels from HD cores (top row – band 1650 cm−1, bottom row – band 3800 cm−1) consisting of (a) adipocytes, (b) blood, (c) epithelium, (d) collagen, and (e) necrosis.

Fig. 3

Fig. 3

Schematic presentation of the CNN architecture used for classification of HD data. A spatial region of size 33 × 33 is cropped around each pixel. Data cubes of size 33 × 33 × 16 are fed into one convolution layer. Each input is convolved with filters of size 3 × 3 outputing 32 feature maps. The following layer is a max pooling layer, which reduces the spatial dimensions by half. Feature extraction continues with two more convolution layers consisting of 64 feature maps each. After another max pooling layer, the extracted features are vectorized and fed to a fully connected layer with 128 units. The last layer, softmax, consisting of 6 units (number of classes) outputs a vector of class probabilities. At the end, maximum probability is used to map each input pixel to its corresponding class labels. Legend: BN – batch normalization, LRN – local response normalization, MP – maximum pooling layer, FC – fully connected layer.

Fig. 4

Fig. 4

Three-dimensional plots of the confusion matrices for SVM (a) and the proposed CNN (b) on SD data. We see particularly large improvements in adipocyte classification as well as increased differentiation between collagen and myofibroblasts. Both of these results are likely due to the inclusion of spatial features.

Fig. 5

Fig. 5

ROC curves and AUC values for each individual class obtained using both SVM and CNN to classify SD data. For the CNN, ROC curves are computed by training the classifiers for each class, where elements of that class have the target value 1 and elements outside of that class have the target value 0. The ROC value for the SVM is obtained by calculating the posterior probability based on the percentage of individual votes. While the CNN provides a significant improvement for most classes, the increased differentiation between adipocytes, myofibroblasts, and collagen stands out due to the prevalence of both in breast biopsies.

Fig. 6

Fig. 6

3D plot of confusion matrices obtained for all independent test pixels in the HD (1.1 μm) microarray image data using SVM (a) and CNN (b) classifiers.

Fig. 7

Fig. 7

ROC curves and AUC values for each individual class for SVM and CNN classifiers applied to HD data.

Fig. 8

Fig. 8

HD classification using a convolutional neural network with both spectral and spatial features. (a) FTIR validation microarray (11 557 × 17 000 pixels) showing the Amide I (1650 cm−1) absorption band. (b) Classified cores labeled using a false-color overlay on the Amide I absorbance band. Individual classified cores are shown in false-color (c and e) with corresponding images H&E stained adjacent sections (d and f ). H&E images are labeled with cell types of interest, annotated using additional immunohistochemical stains of adjacent sections.

Similar articles

Cited by

References

    1. Mittal S, Yeh K, Leslie LS, Kenkel S, Kajdacsy-Balla A and Bhargava R, Proc. Natl. Acad. Sci. U. S. A, 2018, 115, E5651–E5660. - PMC - PubMed
    1. Fernandez DC, Bhargava R, Hewitt SM and Levin IW, Nat. Biotechnol, 2005, 23, 469–474. - PubMed
    1. Benard A, Desmedt C, Smolina M, Szternfeld P, Verdonck M, Rouas G, Kheddoumi N, Rothé F, Larsimont D, Sotiriou C, et al., Analyst, 2014, 139, 1044–1056. - PubMed
    1. Yeh K, Kenkel S, Liu J-N and Bhargava R, Anal. Chem, 2014, 87, 485–493. - PMC - PubMed
    1. Bhargava R and Levin IW, Spectrochemical analysis using infrared multichannel detectors, John Wiley & Sons, 2008.

LinkOut - more resources