Towards Performance Improvement in Indian Sign Language Recognition (original) (raw)
Related papers
International Journal of Advanced Computer Science and Applications
Sign language, a language used by Deaf community, is a fully visual language with its own grammar. The Deaf people find it very difficult to express their feelings to the other people, since the other people lack the knowledge of the sign language used by the Deaf community. Due to the differences in vocabulary and grammar of the sign languages, complete adoption of methods used for other international sign languages is not possible for Indian Sign Language (ISL) recognition. It is difficult to handle continuous sign language sentence recognition and translation into text as no large video dataset for ISL sentences is available. INSIGNVID-the first Indian Sign Language video dataset has been proposed and with this dataset as input, a novel approach is presented that converts video of ISL sentence in appropriate English sentence using transfer learning. The proposed approach gives promising results on our dataset with MobilNetV2 as pretrained model.
Gesture Based Real-time Indian Sign Language Interpreter
International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 2021
Hand gesture is one of the methods used in sign language for non-verbal communication. It is most commonly used by hearing & speech impaired people who have hearing or speech problems to communicate among themselves or with normal people. Developing sign language applications for hearing impaired people can be very important, as hearing & speech impaired people will be able to communicate easily with even those who don’t understand sign language. This project aims at taking the basic step in bridging the communication gap between normal people, deaf and dumb people using sign language. The main focus of this work is to create a vision based system to identify sign language gestures from the video sequences. The reason for choosing a system based on vision relates to the fact that it provides a simpler and more intuitive way of communication between a human and a computer. Video sequences contain both temporal as well as spatial features. In this project, two different models are used to train the temporal as well as spatial features. To train the model on the spatial features of the video sequences a deep Convolutional Neural Network. Convolutional Neural Network was trained on the frames obtained from the video sequences of train data. To train the model on the temporal features Recurrent Neural Network is used. The Trained Convolutional Neural Network model was used to make predictions for individual frames to obtain a sequence of predictions. Now this sequence of prediction outputs was given to the Recurrent Neural Network to train on the temporal features. Collectively both the trained models i.e. Convolutional Neural Network and Recurrent Neural Network will produce the text output of the respective gesture.
A Translator for Indian Sign Language to Text and Speech
International Journal for Research in Applied Science and Engineering Technology IJRASET, 2020
Verbal Communication is the only way using which people have interacted with each other over the years but the case stands different for the disabled. The barrier created between the impaired and the normal people is one of the setbacks of the society. For the impaired people (deaf & mute), sign language is the only way to communicate. In order to help the deaf and mute communicate efficiently with the normal people, an effective solution has been devised. Our aim is to design a system which analyses and recognizes various alphabets from a database of sign images. In order to accomplish this, the application uses various techniques of Image Processing such as segmentation & feature extraction. We use the machine learning technique, Convolutional Neural Network for detection of sign language. We convert the image by cropping the background and keeping only gesture, after that we convert the gesture into black & white scale in png format into 55*60 resolution. This system will help to eradicate the barrier between the deaf-mute & normal people. This system will standardize the Indian Sign Language in India. It will also improve the quality of teaching and learning in deaf and mute institutes. Just as Hindi is recognized as the standard language for conversation throughout India, ISL will be recognized as the standard sign language throughout India. The main aim of this work is serving the mankind that is achieved by providing better teaching and better learning.
Real-Time Detection and Translation for Indian Sign Language using Motion and Speech Recognition
International Journal for Research in Applied Science & Engineering Technology (IJRASET), 2022
Being able to communicate effectively is perhaps one of the most important life skills of all. Speaking is the primitive form of communication and it is what enables us to express ourselves. But life can get really difficult if one lacks the gift of auditory capability. These people communicate with something called sign language. Sign language is a distinctive yet exclusive language, which has been developed for deaf community to be a part of the common culture. In India, there is a large population who are dependent on this form of communication. However, due to the lack of awareness of sign language in our day-today lives, they feel isolated and disconnected with the world. Therefore, we have created a platform which can bridge the gap of this isolation and misunderstandings, using the concepts of Deep Learning. Sign-L is a sign language translator, which can translate actions to text and voice to actions through animation. Not just translations, but it also provides tutorials to learn sign language and increase the much-needed awareness among others as well.
Conversion of Sign Language Video to Text and Speech
International Journal for Research in Applied Science & Engineering Technology (IJRASET), 2022
Sign Language recognition (SLR) is a significant and promising technique to facilitate communication for hearingimpaired people. Here, we are dedicated to finding an efficient solution to the gesture recognition problem. This work develops a sign language (SL) recognition framework with deep neural networks, which directly transcribes videos of SL sign to word. We propose a novel approach, by using Video sequences that contain both the temporal as well as the spatial features. So, we have used two different models to train both the temporal as well as spatial features. To train the model on the spatial features of the video sequences we use the (Convolutional Neural Networks) CNN model. CNN was trained on the frames obtained from the video sequences of train data. We have used RNN(recurrent neural network) to train the model on the temporal features. A trained CNN model was used to make predictions for individual frames to obtain a sequence of predictions or pool layer outputs for each video. Now this sequence of prediction or pool layer outputs was given to RNN to train on the temporal features. Thus, we perform sign language translation where input video will be given, and by using CNN and RNN, the sign shown in the video is recognized and converted to text and speech.
Computer Vision-Based Bengali Sign Language To Text Generation
2022 IEEE 5th International Conference on Image Processing Applications and Systems (IPAS), 2022
In the whole world, around 7% of people have hearing and speech impairment problems. They use sign language as their communication method. As for our country, there are lots of people born with hearing and speech impairment problems. Therefore, our primary focus is to work for those people by converting Bangla sign language into text. There are already various projects on Bangla sign language done by other people. However, they focused more on the separate alphabets and numerical numbers. That is why, we want to concentrate on Bangla word signs since communication is done using words or phrases rather than alphabets. There is no proper database for Bangla word sign language, so we want to make a database for our work using BDSL. In recognition of sign language (SLR), there usually are two types of scenarios: isolated SLR, which takes words by word and completes recognize action, and the other one is continuous SLR, which completes action by translating the whole sentence at once. We are working on isolated SLR. We introduce a method where we are going to use PyTorch and YOLOv5 for a video classification model to convert Bangla sign language into the text from the video where each video has only one sign language word. Here,we have achieved an accuracy rate of 76.29% on the training dataset and 51.44% on the testing dataset. We are working to build a system that will make it easier for hearing and speech-disabled people to interact with the general public.
Automatic Indian Sign Language Recognition for Continuous Video Sequence
ADBU Journal of Engineering Technology (AJET), 2015
Sign Language Recognition has become the active area of research nowadays. This paper describes a novel approach towards a system to recognize the different alphabets of Indian Sign Language in video sequence automatically. The proposed system comprises of four major modules: Data Acquisition, Pre-processing, Feature Extraction and Classification. Pre-processing stage involves Skin Filtering and histogram matching after which Eigen vector based Feature Extraction and Eigen value weighted Euclidean distance based Classification Technique was used. 24 different alphabets were considered in this paper where 96% recognition rate was obtained. Keywords: Eigen value, Eigen vector, Euclidean Distance (ED),Human Computer Interaction, Indian Sign Language (ISL), Skin Filtering. Cite as:Joyeeta Singh, Karen Das "Automatic Indian Sign Language Recognition for Continuous Video Sequence", ADBU J.Engg.Tech., 2(1)(2015) 0021105(5pp)
INDIAN SIGN LANGUAGE TRANSLATION FOR HARD-OF-HEARING AND HARD-OF-SPEAKING COMMUNITY
IRJET, 2022
Sign language is an integral part of human communication as it has allowed people to communicate with the hard of speaking and hearing community and understand them better. However, not everyone is capable of using sign language which causes a barrier between. One finds it hard to communicate without an interpreter. With the help of deep learning and machine learning systems, we can eliminate said barriers. The purpose of our machine learning project is to create a web/phone camera based sign language recognition and translation system that would convert sign language gestures to text and vice versa in real time. It is possible to implement them via two ways : vision-based or glove-based systems. Capturing and translating the signs from the real life world will be the core objective of this project. Convolutional Neural Network (CNN) algorithm is used to implement our project. OpenCV video stream will be used to capture the real time gestures through the web camera or the phone camera. The preprocessed images are then fed to the Keras CNN model. We get the output in the form of text predicting the sign. Not only does each country have its own sign language but there are also many other regional sign languages too. Due to the Covid-19 pandemic, the alternative to normal communication is Video-calling, Facetime, etc. Hardspeaking and hearing people are not able to use such facilities effectively causing a hindrance in communication. Our paper aims to find a solution to such a problem and proposes a system for the translation of sign language using a webcam, mic, smart mobile phones, etc.
Interactive Learning Application for Indian Sign Language
Intelligent Sustainable Systems
Sign language is a language used by deaf and dumb people to communicate with one another as well as rest of the world. Since this language is not known to everyone, it poses a barrier for their personal development and also limits their opportunities. We proposed to develop an interactive sign language learning application with embedded sign recognition LSTM model which recognizes the video signs uploaded by the user. Users can either upload pre-recorded videos with the given constraints or capture videos through webcam. We have restricted our domain to emergency-related words only. The proposed model has achieved 88.94% training accuracy and 86.4% testing accuracy using CNN for feature extraction and LSTM for classification.
Real-time Telugu Sign Language Translator with Computer Vision
International Journal for Research in Applied Science and Engineering Technology
Sign language is the basic communication method among hearing disabled and speech disabled people. To express themselves, they require an interpreter or motion sensing devices who/which converts sign language in a few of the standard languages. However, there is no system for those who speak in the Telugu language and hence they are forced to speak in the national language over the regional language of their culture along with the same issues of cumbersome hardware or need for an interpreter. This paper proposes a system that detects hand gestures and signs from a real-time video stream that is processed with the help of computer vision and classified with object detection YOLOv3 algorithm. Additionally, the labels are mapped to corresponding Telugu text. The style of learning is transfer learning, unlike conventional CNNs, RNNs or traditional Machine Learning models. It involves applying a pre-trained model onto a completely new problem to solve the related problem statement and adapts to the new problem's requirements efficiently. This requires lesser training effort in terms of dataset size and greater accuracy. It is the first system developed as a sign language translator for Telugu script. It has given the best results as compared to the existing systems. The system is trained on 52 Telugu letters, 10 numbers and 8 frequently used Telugu words.