Self-supervised learning in medicine and healthcare (original) (raw)

References

  1. Rajpurkar, P., Chen, E., Banerjee, O. & Topol, E. J. AI in health and medicine. Nat. Med. 28, 31–38 (2022).
    Article CAS Google Scholar
  2. Sambasivan, N. et al. “Everyone wants to do the model work, not the data work”: data cascades in high-stakes AI. In Proc. 2021 CHI Conference on Human Factors in Computing Systems (Association for Computing Machinery, 2021); https://doi.org/10.1145/3411764.3445518
  3. Russakovsky, O. et al. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115, 211–252 (2015).
    Article Google Scholar
  4. Irvin, J. et al. CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison. In 33rd AAAI Conference on Artificial Intelligence, AAAI 2019, 31st Innovative Applications of Artificial Intelligence Conference, IAAI 2019 and the 9th AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019 590–597 (AAAI Press, 2019).
  5. Huh, M., Agrawal, P. & Efros, A. What makes ImageNet good for transfer learning? Preprint at https://doi.org/10.48550/arXiv.1608.08614 (2016).
  6. Chen, T., Kornblith, S., Norouzi, M. & Hinton, G. A simple framework for contrastive learning of visual representations. In Proc. 37th International Conference on Machine Learning (eds. Daumé, H. III & Singh, A.) 1597–1607 (PMLR, 2020).
  7. Chen, X., Fan, H., Girshick, R. & He, K. Improved baselines with momentum contrastive learning. Preprint at https://doi.org/10.48550/arXiv.2003.04297 (2020).
  8. Zbontar, J., Jing, L., Misra, I., LeCun, Y. & Deny, S. Barlow Twins: self-supervised learning via redundancy reduction. In Proc. 38th International Conference on Machine Learning (eds. Meila, M. & Zhang, T.) 12310–12320 (PMLR, 2021).
  9. Sowrirajan, H., Yang, J., Ng, A. Y. & Rajpurkar, P. MoCo-CXR: MoCo pretraining improves representation and transferability of chest X-ray models. In Medical Imaging with Deep Learning 2021 727–743 (PMLR, 2021).
  10. Soni, P. N., Shi, S., Sriram, P. R., Ng, A. Y. & Rajpurkar, P. Contrastive learning of heart and lung sounds for label-efficient diagnosis. Patterns 3, 100400 (2022).
    Article Google Scholar
  11. Zhang, Y., Jiang, H., Miura, Y., Manning, C. D. & Langlotz, C. P. Contrastive learning of medical visual representations from paired images and text. Preprint at https://doi.org/10.48550/arXiv.2010.00747 (2020).
  12. Sriram, A. et al. COVID-19 prognosis via self-supervised representation learning and multi-image prediction. Preprint at https://doi.org/10.48550/arXiv.2101.04909 (2021).
  13. Han, Y., Chen, C., Tewfik, A. H., Ding, Y. & Peng, Y. Pneumonia detection on chest X-ray using radiomic features and contrastive learning. In 2021 IEEE 18th International Symposium on Biomedical Imaging ISBI 247–251 (IEEE Computer Society, 2021).
  14. Azizi, S. et al. Big self-supervised models advance medical image classification. In 2021 IEEECVF International Conference on Computer Vision ICCV 3458–3468 (IEEE Computer Society, 2021).
  15. Vu, Y. N. T. et al. MedAug: contrastive learning leveraging patient metadata improves representations for chest X-ray interpretation. In Proc. 6th Machine Learning for Healthcare Conference (eds Jung, K. et al.) 755–769 (PMLR, 2021).
  16. Lu, M. Y., Chen, R. J. & Mahmood, F. Semi-supervised breast cancer histology classification using deep multiple instance learning and contrast predictive coding. In Medical Imaging 2020: Digital Pathology (eds. Tomaszewski, J. E. & Ward, A. D.) 11320J (SPIE, 2020).
  17. Yang, P., Hong, Z., Yin, X., Zhu, C. & Jiang, R. Self-supervised visual representation learning for histopathological images. in Medical Image Computing and Computer Assisted Intervention – MICCAI 2021 (eds. de Bruijne, M. et al.) 47–57 (Springer International Publishing, 2021).
  18. Srinidhi, C. L., Kim, S. W., Chen, F.-D. & Martel, A. L. Self-supervised driven consistency training for annotation efficient histopathology image analysis. Med. Image Anal. 75, 102256 (2022).
    Article Google Scholar
  19. DiPalma, J., Suriawinata, A. A., Tafe, L. J., Torresani, L. & Hassanpour, S. Resolution-based distillation for efficient histology image classification. Artif. Intell. Med. 119, 102136 (2021).
    Article Google Scholar
  20. Kiyasseh, D., Zhu, T. & Clifton, D. A. CLOCS: Contrastive Learning of Cardiac Signals across space, time, and patients. In Proc. 38th International Conference on Machine Learning (eds. Meila, M. & Zhang, T.) 5606–5615 (PMLR, 2021).
  21. Banville, H. J. et al. Self-supervised representation learning from electroencephalography signals. In 2019 IEEE 29th International Workshop on Machine Learning Signal Process MLSP (IEEE Computer Society, 2019); https://doi.org/10.1109/MLSP.2019.8918693
  22. Gopal, B. et al. 3KG: contrastive learning of 12-lead electrocardiograms using physiologically-inspired augmentations. in Proc. Machine Learning for Health (eds. Roy, S. et al.) 156–167 (PMLR, 2021).
  23. Jiao, J. et al. Self-supervised contrastive video-speech representation learning for ultrasound. Med. Image Comput. Comput. Assist. Interv. 12263, 534–543 (2020).
    Google Scholar
  24. Wang, Y., Wang, J., Cao, Z. & Barati Farimani, A. Molecular contrastive learning of representations via graph neural networks. Nat. Mach. Intell. 4, 279–287 (2022).
    Article Google Scholar
  25. Xie, Y., Xu, Z., Zhang, J., Wang, Z. & Ji, S. Self-supervised learning of graph neural networks: a unified review. IEEE Trans. Pattern Anal. Mach. Intell. (2022); https://doi.org/10.1109/TPAMI.2022.3170559
  26. Meng, X., Ganoe, C. H., Sieberg, R. T., Cheung, Y. Y. & Hassanpour, S. Self-supervised contextual language representation of radiology reports to improve the identification of communication urgency. AMIA Jt. Summits Transl. Sci. Proc. 2020, 413–421 (2020).
    Google Scholar
  27. Girgis, H. Z., James, B. T. & Luczak, B. B. Identity: rapid alignment-free prediction of sequence alignment identity scores using self-supervised general linear models. NAR Genom. Bioinform. 3, lqab001 (2021).
  28. Li, Y. et al. BEHRT: transformer for electronic health records. Sci. Rep. 10, 7155 (2020).
    Article CAS Google Scholar
  29. Wang, X., Xu, Z., Tam, L., Yang, D. & Xu, D. Self-supervised image-text pre-training with mixed data in chest X-rays. Preprint at https://doi.org/10.48550/arXiv.2103.16022 (2021).
  30. Rasmy, L., Xiang, Y., Xie, Z., Tao, C. & Zhi, D. Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. npj Digit. Med. 4, 86 (2021).
    Article Google Scholar
  31. Li, F. et al. Fine-tuning Bidirectional Encoder Representations From Transformers (BERT)-based models on large-scale electronic health record notes: an empirical study. JMIR Med. Inform. 7, e14830 (2019).
    Article Google Scholar
  32. Kraljevic, Z. et al. Multi-domain clinical natural language processing with MedCAT: the Medical Concept Annotation Toolkit. Artif. Intell. Med. 117, 102083 (2021).
    Article Google Scholar
  33. Kostas, D., Aroca-Ouellette, S. & Rudzicz, F. BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data. Front. Hum. Neurosci. 15, 653659 (2021).
    Article Google Scholar
  34. Baevski, A., Zhou, Y., Mohamed, A. & Auli, M. wav2vec 2.0: a framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems (eds Larochelle, H. et al.) 12449–12460 (Curran Associates, 2020).
  35. Boyd, J. et al. Self-supervised representation learning using visual field expansion on digital pathology. In 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) 639–647 (IEEE Computer Society, 2021).
  36. Vaswani, A. et al. Attention is all you need. In Proc. 31st International Conference on Neural Information Processing Systems 6000–6010 (Curran Associates, 2017).
  37. Jaegle, A. et al. Perceiver IO: a general architecture for structured inputs and outputs. In International Conference on Learning Representations 4039 (ICLR, 2022).
  38. Akbari, H. et al. VATT: transformers for multimodal self-supervised learning from raw video, audio and text. In Advances in Neural Information Processing Systems (eds. Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P. S. & Vaughan, J. W.) 24206–24221 (Curran Associates, 2021).
  39. Nagrani, A. et al. Attention bottlenecks for multimodal fusion. In Advances in Neural Information Processing Systems (eds Ranzato, M. et al.) 14200–14213 (Curran Associates, 2021).
  40. Choromanski, K. et al. Masked language modeling for proteins via linearly scalable long-context transformers. Preprint at https://doi.org/10.48550/arXiv.2006.03555 (2020).
  41. Jumper, J. et al. Highly accurate protein structure prediction with AlphaFold. Nature 596, 583–589 (2021).
    Article CAS Google Scholar
  42. Rao, R. M. et al. MSA Transformer. In Proc. 38th International Conference on Machine Learning (eds Meila, M. & Zhang, T.) 8844–8856 (PMLR, 2021).
  43. Lu, A. X., Zhang, H., Ghassemi, M. & Moses, A. Self-supervised contrastive learning of protein representations by mutual information maximization. Preprint at bioRxiv https://doi.org/10.1101/2020.09.04.283929 (2020).
  44. Yang, C., Wu, Z., Zhou, B. & Lin, S. Instance localization for self-supervised detection pretraining. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 3986–3995 (IEEE Computer Society, 2021).
  45. Jana, A. et al. Deep learning based NAS score and fibrosis stage prediction from CT and pathology data. In 2020 IEEE 20th International Conference on Bioinformatics and Bioengineering BIBE 981–986 (IEEE Computer Society, 2020).
  46. Ohri, K. & Kumar, M. Review on self-supervised image recognition using deep neural networks. Knowl. Based Syst. 224, 107090 (2021).
    Article Google Scholar
  47. Holmberg, O. G. et al. Self-supervised retinal thickness prediction enables deep learning from unlabelled data to boost classification of diabetic retinopathy. Nat. Mach. Intell. 2, 719–726 (2020).
    Article Google Scholar
  48. Spahr, A., Bozorgtabar, B. & Thiran, J.-P. Self-taught semi-supervised anomaly detection on upper limb X-rays. In 2021 IEEE 18th International Symposium on Biomedical Imaging ISBI 1632–1636 (IEEE Computer Society, 2021).
  49. Radford, A. et al. Learning transferable visual models from natural language supervision. In Proc. 38th International Conference on Machine Learning (eds Meila, M. & Zhang, T.) 8748–8763 (PMLR, 2021).
  50. Geirhos, R. et al. Shortcut learning in deep neural networks. Nat. Mach. Intell. 2, 665–673 (2020).
    Article Google Scholar
  51. Sagawa, S., Koh, P. W., Hashimoto, T. B. & Liang, P. Distributionally robust neural networks. In International Conference on Learning Representations 1796 (ICLR, 2020).
  52. Fedorov, A. et al. Tasting the cake: evaluating self-supervised generalization on out-of-distribution multimodal MRI data. Preprint at https://doi.org/10.48550/arXiv.2103.15914 (2021).
  53. Li, Z. et al. Domain generalization for mammography detection via multi-style and multi-view contrastive learning. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2021 (eds de Bruijne, M. et al.) 98–108 (Springer, 2021).
  54. Endo, M., Krishnan, R., Krishna, V., Ng, A. Y. & Rajpurkar, P. Retrieval-based chest X-ray report generation using a pre-trained contrastive language-image model. in Proc. Machine Learning for Health (eds. Roy, S. et al.) 209–219 (PMLR, 2021).
  55. Sriram, A. et al. COVID-19 prognosis via self-supervised representation learning and multi-image prediction. Preprint at https://doi.org/10.48550/arXiv.2101.04909 (2021).
  56. Chen, R. J. & Krishnan, R. G. Self-supervised vision transformers learn visual concepts in histopathology. in LMLR at Neural Information Processing Systems (NeurIPS, 2021).
  57. Brown, T. et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (eds Larochelle, H. et al.) 1877–1901 (Curran Associates, 2020).
  58. Logé, C. et al. Q-Pain: a question answering dataset to measure social bias in pain management. In Proc. Neural Information Processing Systems Track on Datasets and Benchmarks (eds. Vanschoren, J. & Yeung, S.) 105 (NeurIPS, 2021).
  59. Larrazabal, A. J., Nieto, N., Peterson, V., Milone, D. H. & Ferrante, E. Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis. Proc. Natl Acad. Sci. USA 117, 12592 (2020).
    Article CAS Google Scholar
  60. Gamble, P. et al. Determining breast cancer biomarker status and associated morphological features using deep learning. Commun. Med. 1, 14 (2021).
    Article Google Scholar

Download references