Problem Behaviors Recognition in Videos using Language-Assisted Deep Learning Model for Children with Autism (original) (raw)
Related papers
Computational Intelligence and Neuroscience
Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder characterized by persistent difficulties including repetitive patterns of behavior known as stereotypical motor movements (SMM). So far, several techniques have been implemented to track and identify SMMs. In this context, we propose a deep learning approach for SMM recognition, namely, convolutional neural networks (CNN) in time and frequency-domains. To solve the intrasubject SMM variability, we propose a robust CNN model for SMM detection within subjects, whose parameters are set according to a proper analysis of SMM signals, thereby outperforming state-of-the-art SMM classification works. And, to solve the intersubject variability, we propose a global, fast, and light-weight framework for SMM detection across subjects which combines a knowledge transfer technique with an SVM classifier, therefore resolving the “real-life” medical issue associated with the lack of supervised SMMs per testing subject in particular. We...
A Novel Automated Approach for Deep Learning on Stereotypical Autistic Motor Movements
Advances in Medical Diagnosis, Treatment, and Care, 2021
Autism spectrum disorder (ASD) is an ongoing neurodevelopmental disorder, with repeated behavior called stereotypical movement autism (SMM). Some recent experiments with accelerometer features as feedback to computer classifiers demonstrate positive findings in persons with autistic motor disorders for the automobile detection of stereotypical motor motions (SMM). To date, several methods for detecting and recognizing SMMs have been introduced. In this context, the authors suggest an approach of deep learning for recognition of SMM, namely deep convolution neural networks (DCNN). They also implemented a robust DCNN model for the identification of SMM in order to solve stereotypical motor movements (SMM), which thus outperform state-of-the-art SMM classification work.
Deep Learning Based Multimedia Data Mining for Autism Spectrum Disorder (ASD) Diagnosis
2019 International Conference on Data Mining Workshops (ICDMW), 2019
Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder characterized by deficits in social communication and restricted and repetitive patterns of behavior. Autism is estimated to affect 1 in 59 children in the United States and costs roughly $35B to the society. Early diagnosis of ASD is vital for promoting early intervention and positive developmental outcomes. Traditional diagnostic procedures for ASD include structured behavioral observation by a trained clinician. Diagnosticians typically rely on the Autism Diagnostic Observation Schedule (ADOS-2) to quantify ASD symptoms. In this paper, we take a parallel approach and investigate language modalities and discover associations between objective measurements of social communication and ASD symptoms. We analyze 33 children with autism and extract their linguistic patterns from their conversations with diagnosticians in a clinical setting. Our methods use Long-Short Term Memory (LSTM) networks to learn Speech Activity Detection (SAD) and speaker diarization patterns to generate the vocal turn-taking metrics. We then use our novel proposed pipeline to predict the ADOS-2 Calibrated Severity Scores (CSS) of Social Affect (SA). The proposed framework achieve state-of-the-art predictive diagnostic estimates of ASD severity compared to industry's leading algorithms. Results compared with the language acquisition system Language ENvironment Analysis (LENA) and other algorithms indicate a significant improvement in the R 2 measure.
2022 IEEE International Conference on Image Processing (ICIP)
Autism Spectrum Disoder (ASD) is a neurodevelopmental disorder characterized by (a) persistent deficits in social communication and interaction, and (b) presence of restrictive, repetitive patterns of behaviours, interests or activities. The stereotyped repetitive behaviours are also referred to as stimming behaviours. We propose a deep learning based approach to automatically predict a child's stimming behaviours from videos recorded in unconstrained conditions. The child's region in the video is tracked and its skeletal joints are derived using the pose estimator. The heatmap representation of skeletal joints and the raw video signals are used as inputs to the two pathways of the RGBPose-SlowFast deep network to model stimming behaviours. The proposed model is evaluated using the publicly available Self-Stimulatory Behaviour Dataset (SSBD) of stimming behaviours. The generalization ability of the model is validated using the Autism dataset containing child's motor actions. Our experiments demonstrate state-of-the-art results on both datasets.
Signal Processing
Autism Spectrum Disorders are associated with atypical movements, of which stereotypical motor movements (SMMs) interfere with learning and social interaction. The automatic SMM detection using inertial measurement units (IMU) remains complex due to the strong intra and inter-subject variability, especially when handcrafted features are extracted from the signal. We propose a new application of the deep learning to facilitate automatic SMM detection using multiaxis IMUs. We use a convolutional neural network (CNN) to learn a discriminative feature space from raw data. We show how the CNN can be used for parameter transfer learning to enhance the detection rate on longitudinal data. We also combine the long short-term memory (LSTM) with CNN to model the temporal patterns in a sequence of multi-axis signals. Further, we employ ensemble learning to combine multiple LSTM learners into a more robust SMM detector. Our results show that: 1) feature learning outperforms handcrafted features; 2) parameter transfer learning is beneficial in longitudinal settings; 3) using LSTM to learn the temporal dynamic of signals enhances the detection rate especially for skewed training data; 4) an ensemble of LSTMs provides more accurate and stable detectors. These findings provide a significant step toward accurate SMM detection in real-time scenarios.
Detection of Autism Spectrum Disorder Using Deep Learning
INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT
Social interaction, conduct, and cognitive ability are all impacted by the neuro developmental illness known as autism spectrum disorder (ASD). Even though ASD diagnosis can be difficult and time-consuming, early detection and intervention can improve long-term results. Early childhood is when autism spectrum disorder first manifests, and it eventually leads to issues with social, academic, and occupational functioning in society. Within the first year, autism signs are frequently visible in children. Some infants display autistic spectrum disorder symptoms as early as infancy, including decreased eye contact, a lack of responsiveness to their name, or a lack of interest in carers. To identify the presence of disorder at an early stage, use a deep learning system like LSTM. Self-Stimulatory Behaviours Dataset (SSBD) was used to collect the datasets, and video dataset was used to construct the system. The feature extraction algorithm is the Blaze Pose algorithm. The model file has be...
Convolutional Neural Network for Stereotypical Motor Movement Detection in Autism
ArXiv, 2015
Autism Spectrum Disorders (ASDs) are often associated with specific atypical postural or motor behaviors, of which Stereotypical Motor Movements (SMMs) have a specific visibility. While the identification and the quantification of SMM patterns remain complex, its automation would provide support to accurate tuning of the intervention in the therapy of autism. Therefore, it is essential to develop automatic SMM detection systems in a real world setting, taking care of strong inter-subject and intra-subject variability. Wireless accelerometer sensing technology can provide a valid infrastructure for real-time SMM detection, however such variability remains a problem also for machine learning methods, in particular whenever handcrafted features extracted from accelerometer signal are considered. Here, we propose to employ the deep learning paradigm in order to learn discriminating features from multi-sensor accelerometer signals. Our results provide preliminary evidence that feature le...
2018 26th International Conference on Software, Telecommunications and Computer Networks (SoftCOM), 2018
In this paper we demonstrate the effectiveness of a deep learning approach for object detection and classification using a mono-vision feedback of a NAO humanoid robot for assessing the child’s behavior during a free play with standardized toys. The free play is one of the tasks contained in the standard ADOS-2 autism spectrum disorder diagnostic protocol used by clinicians. In order to make an accurate, robust and fast object detector, a new data set for learning and testing has been created to enable a reliable assessment of the child’s behavior while playing with the toys. This has also led to the development of algorithms and mechanism to assess child’s attention based on the toys that the child is playing with. This paper concludes with the discussion about the challenges encountered and their solutions, as well as about the prospective development goals focused on achieving more robust and accurate child attention analyzer.
MMASD: A Multimodal Dataset for Autism Intervention Analysis
arXiv (Cornell University), 2023
Original ASD Videos Figure 1: MMASD provides multiple multimodal privacy-preserving features derived from original videos via ROMP [40], OpenPose [5], and Lucas-Kanade [29], including optical flow, 2D / 3D skeleton, and clinical evaluation results. The dataset is publicly accessible for addressing research questions centered on social and behavioral interactions of children with ASD in playful group activities.
PloS one, 2024
In recent years, research has been demonstrating that movement analysis, utilizing machine learning methods, can be a promising aid for clinicians in supporting autism diagnostic process. Within this field of research, we aim to explore new models and delve into the detailed observation of certain features that previous literature has identified as prominent in the classification process. Our study employs a game-based tablet application to collect motor data. We use artificial neural networks to analyze raw trajectories in a "drag and drop" task. We compare a two-features model (utilizing only raw coordinates) with a fourfeatures model (including velocities and accelerations). The aim is to assess the effectiveness of raw data analysis and determine the impact of acceleration on autism classification. Our results revealed that both models demonstrate promising accuracy in classifying motor trajectories. The four-features model consistently outperforms the two-features model, as evidenced by accuracy values (0.90 vs. 0.76). However, our findings support the potential of raw data analysis in objectively assessing motor behaviors related to autism. While the fourfeatures model excels, the two-features model still achieves reasonable accuracy. Addressing limitations related to sample size and noise is essential for future research. Our study emphasizes the importance of integrating intelligent solutions to enhance and assist autism traditional diagnostic process and intervention, paving the way for more effective tools in assessing motor skills.