Using NLG and sensors to support personal narrative for children with complex communication needs (original) (raw)
Related papers
International Journal of Human-Computer Studies, 2016
This paper describes a Natural Language Generation system (NLG), How was School Today? that automatically creates a personal narrative from sensor data and other media (photos and audio). It can be used by children with complex communication needs in schools to support interactive narrative about personal experiences. The robustness of story generation to missing data was identified as a key area for improvement in a feasibility study of the system at a first special needs school. This paper therefore suggests three possible methods for generating stories from unstructured data: clustering by voice recording, by location, or by time. Clustering based on voice recordings resulted in stories that were perceived as most easy to read, and to make most sense, by parents in a quantitative evaluation. This method was implemented in the live system, which was developed and evaluated iteratively at a second special needs school with children with different usage profiles. Open challenges and possibilities for nlg in augmented and alternative communication are also discussed.
A new tool to support interactive narrative experience for children with communication disorders
2010
An interactive narrative tool for children with communication disorders has been developed that aims to provide children with the experience of interactive narrative based conversation. The tool, which runs on a tablet PC, automatically generates an interactive narrative based on information collected by the tool using environmental sensors and other data. Staff can also record voice messages which are automatically integrated into the narrative text. The child can personalise the generated story by adding comments or deleting unwanted events before using it for interactive communication. A user-centred design involving children, staff and parents led to the development of the prototype, which was evaluated with two children in a special school. The results of the initial evaluation showed that both children could use the system for interactive conversation and that it provided parents with a focus when discussing the school day. A follow-on project is now underway to address issues about everyday use in a school environment and its use with a wider spectrum of intellectual/learning disabilities.
Using NLG to Help Language-Impaired Users Tell Stories and Participate in Social Dialogues
2009
Augmentative and Alternative Communication (AAC) systems are communication aids for people who cannot speak because of motor or cognitive impairments. We are developing AAC systems where users select information they wish to communicate, and this is expressed using an NLG system. We believe this model will work well in contexts where AAC users wish to go beyond simply making requests or answering questions, and have more complex communicative goals such as story-telling and social interaction.
A flexible approach to natural language generation for disabled children
Proceedings of the 21st International Conference on computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop on - COLING ACL '06, 2006
Natural Language Generation (NLG) is a way to automatically realize a correct expression in response to a communicative goal. This technology is mainly explored in the fields of machine translation, report generation, dialog system etc. In this paper we have explored the NLG technique for another novel applicationassisting disabled children to take part in conversation. The limited physical ability and mental maturity of our intended users made the NLG approach different from others. We have taken a flexible approach where main emphasis is given on flexibility and usability of the system. The evaluation results show this technique can increase the communication rate of users during a conversation.
ArXiv, 2021
Augmentative and alternative communication (AAC) is a practical means of communication for people with language disabilities. In this study, we propose PICTALKY, which is an AI-based AAC system that helps children with language developmental disabilities to improve their communication skills and language comprehension abilities. PICTALKY can process both text and pictograms more accurately by connecting a series of neural-based NLP modules. Moreover, we perform quantitative and qualitative analyses on the essential features of PICTALKY. It is expected that those suffering from language problems will be able to express their intentions or desires more easily and improve their quality of life by using this service. We have made the models freely available alongside a demonstration of the Web interface 1. Furthermore, we implemented robotics AAC for the first time by applying PICTALKY to the NAO robot.
Storytelling Utilizing Generative AI to Foster Inclusion of Individuals with Disabilities
This article presents the comprehensive design and evaluation of a digital storytelling system tailored for Latin American children aged 4 to 6, leveraging generative artificial intelligence. Tests were conducted to assess the system’s functionality, content diversity, generation times, and voice quality, including intonation, speed, and pronunciation. The results substantiate the system’s operational efficacy and user-friendly interface. The stories generated demonstrate substantial diversity, as indicated by Jaccard indices calculations, which reveal a maximum value of 0.2 derived from evaluating 30 distinct stories. As expected, there was a proportional increase in story generation times relative to their length. ’Onyx’ from OpenAI’s text-to-speech (TTS) was identified as the most appropriate voice for storytelling. Nonetheless, pronunciation inaccuracies were observed across all tested TTS model voices. The analysis demonstrated that the system generates a variety of stories that foster value formation in Spanish-speaking children, thereby promoting the importance of including individuals with disabilities. Notably, all content within the stories was found to be suitable for children, with no inappropriate material detected in any of the narratives.
Proceedings of the 16th International Technology, Education and Development Conference, 2022
Many approaches to the definition of animated story exist, but a good "one size fits all" solution is difficult to adopt and implement. Software for the creation of simple animated stories is often more complex than necessary. Moreover, usually, it is not possible to personalize the functionalities provided by the User Interface to the specific users' needs. This, in turn, affects the effectiveness of such software. To address these problems, we defined, implemented, and tested a methodology to simplify the process of designing software for animated stories. We defined an Intermediate Language (IL) that abstracts basic concepts useful for the description of a story. The components and concepts of the IL were inspired by the Business Process Model and Notation (BPMN) and the Scratch language. To validate our approach, we applied it to the design and implementation of an app for storytelling. The app we created permits to define the story by means of a simple graphical language; then, a first translation module generates an IL representation; finally, a second translation module produces a Scratch executable file. We tested our app with subjects with intellectual disability. Psychologists supervised the experiments and observed improvements in subjects' vocal expression, attention level and narration quality.
Storytelling App for Children with Hearing Impairment Using Natural Language Processing (NLP
During childhood, children's teachers, parents or grandparents read them a lot of fantastic stories. They did that for the time we couldn't read. Unfortunately, not everyone is blessed with an ability to hear. The children with hearing impairments might not have had a chance to know such stories at least in their childhood. This project is based on an app which narrates children's stories to hearing impaired children by taking in stories in form of text as input and giving images of sign language gestures and speech as output. In this paper, a platform Kahani which translates written English into Indian Sign Language is presented.
A Text to Image Story Teller Specially Challenged Children - Natural Language Processing Approach
2015
Every human relishes happiness when he or she becomes a creator. This happiness does not have any metrics. A person who plants a tree, to a person who works on the most complicated flight engines feels happy on his creation. The work attempted in this paper is to make children the creators. Children have wonderful capabilities to create and innovate new things. Specially challenged children are equally talented in their own way. Challenged children identified with capability outperform everybody in their competitive line. This research work is an application which can be used by such children which kindles their creativity and helps them showcase their abilities. These children can create their own pictorial story board in a fashion as it comes to their mind. The text story is handled as a natural language input and translated to a scrolling layer of images, which are picked by the user.
Multi-user Narrative Sharing System for Children with Chronic Illness
… 2009. ICALT 2009. Ninth IEEE International …, 2009
In our project, we are developing a system that facilitates the hospitalized children with chronic illness to narrate and share their stories. In this system, children aged K-6 th grade can create their own and coconstruct narratives with peers and bedside teachers in a multi-user environment.