Machine Learning-Based Automatic Classification of Video Recorded Neonatal Manipulations and Associated Physiological Parameters: A Feasibility Study - PubMed (original) (raw)
doi: 10.3390/children8010001.
Satoshi Kusuda 2, Ryan M McAdams 3, Shubham Gupta 1, Jayant Kalra 1, Ravneet Kaur 1, Ritu Das 1, Saket Anand 4, Ashish Kumar Pandey 5, Su Jin Cho 6, Satish Saluja 7, Justin J Boutilier 8, Suchi Saria 9, Jonathan Palma 10, Avneet Kaur 11, Gautam Yadav 12, Yao Sun 13
Affiliations
- PMID: 33375101
- PMCID: PMC7822162
- DOI: 10.3390/children8010001
Machine Learning-Based Automatic Classification of Video Recorded Neonatal Manipulations and Associated Physiological Parameters: A Feasibility Study
Harpreet Singh et al. Children (Basel). 2020.
Abstract
Our objective in this study was to determine if machine learning (ML) can automatically recognize neonatal manipulations, along with associated changes in physiological parameters. A retrospective observational study was carried out in two Neonatal Intensive Care Units (NICUs) between December 2019 to April 2020. Both the video and physiological data (heart rate (HR) and oxygen saturation (SpO2)) were captured during NICU hospitalization. The proposed classification of neonatal manipulations was achieved by a deep learning system consisting of an Inception-v3 convolutional neural network (CNN), followed by transfer learning layers of Long Short-Term Memory (LSTM). Physiological signals prior to manipulations (baseline) were compared to during and after manipulations. The validation of the system was done using the leave-one-out strategy with input of 8 s of video exhibiting manipulation activity. Ten neonates were video recorded during an average length of stay of 24.5 days. Each neonate had an average of 528 manipulations during their NICU hospitalization, with the average duration of performing these manipulations varying from 28.9 s for patting, 45.5 s for a diaper change, and 108.9 s for tube feeding. The accuracy of the system was 95% for training and 85% for the validation dataset. In neonates <32 weeks' gestation, diaper changes were associated with significant changes in HR and SpO2, and, for neonates ≥32 weeks' gestation, patting and tube feeding were associated with significant changes in HR. The presented system can classify and document the manipulations with high accuracy. Moreover, the study suggests that manipulations impact physiological parameters.
Keywords: CNN; IoT; LSTM; electronic medical records; machine learning; neonatal intensive care units; physiological deviations; physiological parameters; streaming server; video monitoring.
Conflict of interest statement
The Child Health Imprints (CHI) as an organization is focused on using technology to improve outcomes in NICU. It is disclosed that all the associated members are employees of CHI. The team has created iNICU, NEO, and analytics modules focused on the early prediction of disease and optimizing outcomes. Harpreet Singh and Ravneet Kaur are co-founders and own stock in the CHI. The informatics and clinical advisory team are responsible for providing academic inputs.
Figures
Figure A1
(a) Camera installed on custom-made wall mount to monitor the neonate in a Neonatal Intensive Care Unit (NICU). (b) Wall mount for the camera showing different sections. The wall mount weighs 1 kg.
Figure A2
Size comparison of a 3 × 3 Rubik’s cube and NEO TINY system client.
Figure A3
Physical image of NTS client.
Figure A4
Schematic diagram of the NEO TINY system client depicting: (a) Main Board, (b) Power Supply, (c) Communication Ports.
Figure A5
Synchronization of physiological data and video data with the server clock.
Figure A6
Bedside interface to display manipulation video and physiological parameters.
Figure A7
Vital tracking data monitoring screen (‘Total Data Points’ refer to the data points expected by the time of the last update, and ‘Data Points received’ refers to data points received in actual.).
Figure A8
Baby placard notifying the disconnection of sensor capturing physiological data (the red icon is flashed continuously until the physiological data resumes).
Figure 1
NEO TINY system (NTS) client module in typical Neonatal Intensive Care Unit (NICU) settings (box with yellow-colored border highlight the NTS client, and red-colored boxes highlight other devices).
Figure 2
The overall architecture of machine learning (ML)-based video classification system in the NICU.
Figure 3
Images of manipulation: (i) patting, (ii) diaper change, and (iii) tube feeding, the region of interest marked with a yellow border.
Figure 4
Deep learning architecture for neonatal video classification utilizing Convolutional Neural Network (CNN) and Long Short Term Memory (LSTM) network.
Figure 5
t-Distributed Stochastic Neighbor Embedding (t-SNE) visualization for the manipulations (patting, diaper change, and tube feeding) (a) without transfer learning and (b) with transfer learning. Perplexity is 35, and the number of iterations is 20,000.
Figure 6
CNN-based model accuracy for classifying manipulation images.
Figure 7
Automatic tagging of manipulation videos: The first frame identified as manipulation is marked on the top left, and dotted lines show manipulation.
Figure 8
Variability in physiological signals captured every minute (average values) during the manipulations in the clinical setting. (a) patting, (b) diaper change, and (c) tube feeding. * number of manipulations/number of patients.
References
LinkOut - more resources
Full Text Sources
Miscellaneous