American Sign Language Recognition and Generation : A CNN-based Approach (original) (raw)
Related papers
Statistics Optimization & Information Computing, 2024
Sign language is commonly used by people with hearing and speech impairments, making it difficult for those without such disabilities to understand. However, sign language is not limited to communication within the deaf community alone. It has been officially recognized in numerous countries and is increasingly being offered as a second language option in educational institutions. In addition, sign language has shown its usefulness in various professional sectors, including interpreting, education, and healthcare, by facilitating communication between people with and without hearing impairments. Advanced technologies, such as computer vision and machine learning algorithms, are used to interpret and translate sign language into spoken or written forms. These technologies aim to promote inclusivity and provide equal opportunities for people with hearing impairments in different domains, such as education, employment, and social interactions. In this paper, we implement a DeafTech Vision (DTV-CNN) architecture based on the convolutional neural network to recognize American Sign Language (ASL) gestures using deep learning techniques. Our main objective is to develop a robust ASL sign classification model to enhance human-computer interaction and assist individuals with hearing impairments. Through extensive evaluation, our model consistently outperformed baseline methods in terms of precision. It achieved an outstanding accuracy rate of 99.87% on the ASL alphabet test dataset and 99.94% on the ASL digit dataset, significantly exceeding previous research, which reported an accuracy of 90.00%. We also illustrated the model's learning trends and convergence points using loss and error graphs. These results highlight the DTV-CNN's effectiveness and capability in distinguishing complex ASL gestures.
Sign Language Recognition using Deep Learning
International Journal for Research in Applied Science & Engineering Technology (IJRASET), 2022
Millions of people with speech and hearing impairments communicate with sign languages every day. For hearingimpaired people, gesture recognition is a natural way of communicating, much like voice recognition is for most people. In this study, we look at the issue of translating/converting sign language to text and propose a better solution based on machine learning techniques. We want to establish a system that hearing-impaired people may utilise in their everyday lives to promote communication and collaboration between hearing-impaired people and people who aren't trained in American Sign Language (ASL). To develop a deep learning model for the ASL dataset, we'll use a technique called Transfer Learning in combination with Data Augmentation.
IJERT-Sign Language to Text and Speech Translation in Real Time Using Convolutional Neural Network
International Journal of Engineering Research and Technology (IJERT), 2020
https://www.ijert.org/sign-language-to-text-and-speech-translation-in-real-time-using-convolutional-neural-network https://www.ijert.org/research/sign-language-to-text-and-speech-translation-in-real-time-using-convolutional-neural-network-IJERTCONV8IS15042.pdf Creating a desktop application that uses a computer's webcam to capture a person signing gestures for American sign language (ASL), and translate it into corresponding text and speech in real time. The translated sign language gesture will be acquired in text which is farther converted into audio. In this manner we are implementing a finger spelling sign language translator. To enable the detection of gestures, we are making use of a Convolutional neural network (CNN). A CNN is highly efficient in tackling computer vision problems and is capable of detecting the desired features with a high degree of accuracy upon sufficient training.
Annals of Emerging Technologies in Computing
Sign language (SL) recognition is intended to connect deaf people with the general population via a variety of perspectives, experiences, and skills that serve as a basis for the development of human-computer interaction. Hand gesture-based SL recognition encompasses a wide range of human capabilities and perspectives. The efficiency of hand gesture performance is still challenging due to the complexity of varying levels of illumination, diversity, multiple aspects, self-identifying parts, different shapes, sizes, and complex backgrounds. In this context, we present an American Sign Language alphabet recognition system that translates sign gestures into text and creates a meaningful sentence from continuously performed gestures. We propose a segmentation technique for hand gestures and present a convolutional neural network (CNN) based on the fusion of features. The input image is captured directly from a video via a low-cost device such as a webcam and is pre-processed by a filteri...
An Adam based CNN and LSTM approach for sign language recognition in real time for deaf people
Bulletin of Electrical Engineering and Informatics, 2024
Hand gestures and sign language are crucial modes of communication for deaf individuals. Since most people can't understand sign language, it's hard for a mute and an average person to talk to each other. Because of technological progress, computer vision and deep learning can now be used to count. This paper shows two ways to use deep knowledge to recognize sign language. These methods help regular people understand sign language and improve their communication. Based on American sign language (ASL), two separate datasets have been constructed; the first has 26 signs, and the other contains three significant symbols with the crucial sequence of frames or videos for regular communication. This study looks at three different models: the improved ResNet-based convolutional neural network (CNN), the long short-term memory (LSTM), and the gated recurrent unit (GRU). The first dataset is used to fit and assess the CNN model. With the adaptive moment estimation (Adam) optimizer, CNN obtains an accuracy of 89.07%. In contrast, the second dataset is given to LSTM and GRU and a comparison has been conducted. LSTM does better than GRU in all classes. LSTM has a 94.3% accuracy, while GRU only manages 79.3%. Our preliminary models' real-time performance is also highlighted.
Hand Gesture Alphabet Recognition for American Sign Language using Deep Learning
International Journal of Scientific Research in Science, Engineering and Technology, 2021
Speech impairment limits a person's capacity to speak and communicate with others, forcing them to adopt other communication methods such as sign language. Sign language is not that widely used technique by the deaf. To solve this problem, we developed a powerful hand gesture detection tool that can easily monitor both dynamic and static hand motions with ease. Gesture recognition aims to translate sign language into voice or text for individuals who have a rudimentary comprehension of that, which will be a tremendous help in communication between deaf-mute and hearing people. We describe the design and implementation of an American Sign Language (ASL) fingerspelling translator based on spatial feature identification using a convolutional neural network.
Sign Language Translation Using Deep Convolutional Neural Networks
KSII Transactions on Internet and Information Systems, 2020
Sign language is a natural, visually oriented and non-verbal communication channel between people that facilitates communication through facial/bodily expressions, postures and a set of gestures. It is basically used for communication with people who are deaf or hard of hearing. In order to understand such communication quickly and accurately, the design of a successful sign language translation system is considered in this paper. The proposed system includes object detection and classification stages. Firstly, Single Shot Multi Box Detection (SSD) architecture is utilized for hand detection, then a deep learning structure based on the Inception v3 plus Support Vector Machine (SVM) that combines feature extraction and classification stages is proposed to constructively translate the detected hand gestures. A sign language fingerspelling dataset is used for the design of the proposed model. The obtained results and comparative analysis demonstrate the efficiency of using the proposed hybrid structure in sign language translation.
Deep Learning Approach For Sign Language Recognition
Jurnal Ilmiah Teknik Elektro Komputer dan Informatika (JITEKI), 2023
Sign language is a method of communication that uses hand gestures between people with hearing loss. Each hand sign represents one meaning, but several terms don't have sign language, so they have to be spelled alphabetically. Problems occur when communicating between normal people with hearing loss, because not everyone understands sign language, so a model is needed to recognize sign language as well as a learning tool for beginners who want to learn sign language, especially alphabetic sign language. This study aims to create a hand sign language recognition model for alphabetic letters using a deep learning approach. The main contribution of this research is to produce a real-time hand sign language image acquisition, and hand sign language recognition model for Alphabet. The model used is a seven-layer Convolutional Neural Network (CNN). This model is trained using the ASL alphabet database which consists of 27 categories, where each category consists of 3000 images or a total of 87,000 hand gesture images measuring 200×200 pixels. First, the background correction process is carried out and the input image size is changed to 32×32 pixels using the bicubic interpolation method. Next, separate the dataset for training and validation respectively 75% and 25%. Finally the process of testing the model using data input of hand sign language images from a web camera. The test results show that the proposed model has good performance with an accuracy value of 99%. The experimental results show that image preprocessing using background correction can improve model performance.
Real Time Video Recognition of Signs for Deaf and Dump Using Deep Learning
International Journal for Research in Applied Science & Engineering Technology (IJRASET), 2022
To pass the message is one of the essential prerequisites for endurance in the general public. Gesture based communication is a typical specialized technique for hard of hearing stupid local area. It makes out of an assortment scope of motions, activities and, surprisingly, facial feelings.Gesture based communication is utilized by 70 million individuals all over the planet. Understanding communication via gestures is one of the essential empowering influences in assisting clients of gesture-based communication with speaking with the remainder of the general public. The hard of hearing and dump local area moves back with regards to the intelligent part with ordinary individuals. This makes a tremendous hole among hard of hearing and dump individuals and ordinary individuals. Since our local area have no clue about communication through signing. In this project an application is created which will fill in as a learning instrument first of all in communication via gestures that includes hand recognition. Application is made to change gesture-based communication over to message. An application which makes an interpretation of Sign language to message, which utilizes the portable camera to catch the picture of the hand motion. Then, at that point, the caught picture goes through the series of activity. The CNN model is utilized to extricate the elements of the caught picture and makes an interpretation of it into text.
Sign Language Detection with CNN
International Journal for Research in Applied Science & Engineering Technology (IJRASET), 2022
Sign language is one of the oldest and most natural forms of language for communication , but since most people do not know sign language and interpreters are very difficult to come by, we have come up with a real time method using neural networks for fingerspelling based American sign language. In our method, the hand is first passed through a filter and after the filter is applied the hand is passed through a classifier which predicts the class of the hand gestures. Our method provides 95.7% accuracy for the 26 letters of the alphabet.