Technological troubleshooting based on sentence embedding with deep transformers (original) (raw)
Related papers
Generating Semantic Matches Between Maintenance Work Orders for Diagnostic Decision Support
Annual Conference of the PHM Society
Improving maintenance knowledge intelligence using text data is challenging since the maintenance information is mainly recorded as text. To unlock the knowledge from the maintenance text, a decision-making solution based on retrieving similar cases to help solve new maintenance problems is proposed. In this work, an unsupervised domain fine-tuning technique, Transformer-based Sequential Denoising Auto-Encoder (TSDAE) is used to fine-tune the BERT (Bidirectional Encoder Representations from Transformers) model on domain-specific corpora composed of the Maintenance Work Orders (MWOs). Unsupervised fine-tuning helped the BERT model to adapt MWOs text. Results indicate fine-tuned BERT model can generate semantic matches between MWOs regardless of the complex nature of maintenance text.
Algorithmically Exploiting the Knowledge Accumulated in Textual Domains for Technical Support
2021
Processing natural language and extract relevant information in deep technical engineering domain remains an open challenge. On the other side, manufacturers of high-value assets which often deliver product services through the equipment life, supporting maintenance, spare parts management and remote monitoring and diagnostics for issues resolution, have availability of a good amount of textual data containing technical cases with a certain engineering depth. This paper presents a case study in which various Artificial Intelligence algorithms were applied to historical technical cases to extract know-how useful to help technicians in approaching new cases. Initially the work process and available data are presented; the focus is on the outbound communication delivered from the technical team to the site operators, that is structured in 3 main paragraphs: event description, technical assessment, recommended actions. The work proceeded in two parallel streams: the first concerned the ...
Transformer induced enhanced feature engineering for contextual similarity detection in text
Bulletin of Electrical Engineering and Informatics , 2022
Availability of large data storage systems has resulted in digitization of information. Question and answering communities like Quora and stack overflow take advantage of such systems to provide information to users. However, as the amount of information stored gets larger, it becomes difficult to keep track of the existing information, especially information duplication. This work presents a similarity detection technique that can be used to identify similarity levels in textual data based on the context in which the information was provided. This work presents the transformer based contextual similarity detection (TCSD), which uses a combination of bidirectional encoder representations from transformers (BERT) and similarity metrics to derive features from the data. The derived features are used to train the ensemble model for similarity detection. Experiments were performed using the Quora question similarity data set. Results and comparisons indicate that the proposed model exhibits similarity detection with an accuracy of 92.5%, representing high efficiency.
Building a Question Answering System for the Manufacturing Domain
ArXiv, 2021
The design or simulation analysis of special equipment products must follow the national standards, and hence it may be necessary to repeatedly consult the contents of the standards in the design process. However, it is difficult for the traditional question answering system based on keyword retrieval to give accurate answers to technical questions. Therefore, we use natural language processing techniques to design a question answering system for the decision-making process in pressure vessel design. To solve the problem of insufficient training data for the technology question answering system, we propose a method to generate questions according to a declarative sentence from several different dimensions so that multiple question-answer pairs can be obtained from a declarative sentence. In addition, we designed an interactive attention model based on a bidirectional long short-term memory (BiLSTM) network to improve the performance of the similarity comparison of two question sentences. Finally, the performance of the question answering system was tested on public and technical domain datasets.
Adapting natural language processing for technical text
Applied AI Letters, 2021
Despite recent dramatic successes, natural language processing (NLP) is not ready to address a variety of real-world problems. Its reliance on large standard corpora, a training and evaluation paradigm that favors the learning of shallow heuristics, and large computational resource requirements, makes domain-specific application of even the most successful NLP techniques difficult. This paper proposes technical language processing (TLP) which brings engineering principles and practices to NLP specifically for the purpose of extracting actionable information from language generated by experts in their technical tasks, systems, and processes. TLP envisages NLP as a socio-technical system rather than as an algorithmic pipeline. We describe how the TLP approach to meaning and generalization differs from that of NLP, how data quantity and quality can be addressed in engineering technical domains, and the potential risks of not adapting NLP for technical use cases. Engineering problems can benefit immensely from the inclusion of knowledge from unstructured data, currently unavailable due to issues with out of the box NLP packages. We illustrate the TLP approach by focusing on maintenance in industrial organizations as a case-study. K E Y W O R D S domain adaptation, maintenance records, natural language processing, technical data, technical language processing 1 | INTRODUCTION Natural language processing (NLP) has recently made rapid and significant advances across a wide variety of tasks. These were enabled by improvements in language models that predict characters, words, or sentences from surrounding context, which have become a central theme in NLP research. 1-3 The foremost example, Generative Pre-trained Transformer 3 (GPT-3), has been dubbed the "most powerful language model ever" 4 and recently demonstrated strong performance on many existing data sets for a variety of NLP tasks such as translation, question answering, unscrambling words, and news article generation. 5 Early users have shown its ability to generate text ranging from guitar tablature, to website layouts, to computer code. 4 This material is declared a work of the U.S. Government and is not subject to copyright protection in the United States. Approved for public release; distribution is unlimited.
Text Similarity Using Siamese Networks and Transformers
International Journal for Research in Applied Science & Engineering Technology (IJRASET), 2022
In result-oriented conversational models like message renders and chatbots, finding the similarity between the input and output text result is a big task. In general, the conversational model developers lean to provide a minimal number of utterances per instance, and this makes the classification a difficult task. This problem becomes more difficult when the length of the processed text per action is short and length of the user input is long. Identical sentence pair detection reduces manual effort for users with high reputation. Siamese networks have been one of the best innovative architectures designed in the field of natural language processing. A Siamese network was initially designed for computer vision applications. Later the core concept of this algorithm was designed for NLP ,to identify similarity for two given sentences. Siamese networks are used in this algorithm. It's an artificial neural network also known as a twin neural network that works in tandem on two independent input vectors to calculate equivalent output vectors using the same weights. Also there are few commonly addressed drawbacks like word sense disambiguation and memory intolerance of initial inputs for sentences having more than 15-20 words. To tackle these issues, we propose a modified algorithm that integrates the transformer model implicitly with the core part of the siamese network. Transformer model helps to generate each output position based on the semantic analysis of overall sentence and can also deal with homonyms, by extracting its meaning based, which is syntactic based and semantic based on the overall sentence or paragraph or text.
Sentence encoders for semantic textual similarity: A survey
The last decade has witnessed many accomplishments in the field of Natural Language Processing, especially in understanding the language semantics. Well-established machine learning models for generating word representation are available and has been proven useful. However, the existing techniques proposed for learning sentence level representations do not adequately capture the complexity of compositional semantics. Finding semantic similarity between sentences is a fundamental language understanding problem. In this project, we compare various machine learning models on their ability to capture the semantics of a sentence using Semantic Textual Similarity (STS) Task. We focus on models that exhibit state-of-the-art performance in Sem-Eval(2017) STS shared task. Also, we analyse the impact of models' internal architectures on STS task performance. Out of all the models the we compared, Bi-LSTM RNN with max-pooling layer achieves the best performance in extracting a generic semantic representation and aids in better transfer learning when compared to hierarchical CNN.
Scientific Reports, 2022
We propose a deep graph learning approach for computing semantic textual similarity (STS) by using semantic role labels generated by a Semantic Role Labeling (SRL) system. SRL system output has significant challenges in dealing with graph-neural networks because it doesn't have a graph structure. To address these challenges, we propose a novel SRL graph by using semantic role labels and dependency grammar. For processing the SRL graph, we proposed a Deep Graph Neural Network (DGNN) based graph-U-Net model that is placed on top of the transformers to use a variety of transformers to process representations obtained from them. We investigate the effect of using the proposed DGNN and SRL graph on the performance of some transformers in computing STS. For the evaluation of our approach, we use STS2017 and SICK datasets. Experimental evaluations show that using the SRL graph accompanied by applying the proposed DGNN increases the performance of the transformers used in the DGNN. The problem of similarity learning is a significant issue in pattern recognition. The goal of similarity learning is to learn a measure to reflect the semantic distance according to a specific task 1. Similarity learning includes looking for similarity patterns to find complicated and implicit semantic patterns. Similarity learning in the text area is studied in the STS computation field. STS measures the degree of semantic overlap between two texts 2. The ability to determine the semantic relationship between two texts is an integral part of machines that understand and infer natural language 3 hence STS is a directly or indirectly significant component of many applications such as information retrieval 4 , recognition of paraphrases 5 , textual entailment 6 , question answering 7 , text summarization 8 , measuring the degree of equivalence between a machine translation output and a reference translation 9 and also text summarization evaluation, text classification, document clustering, topic tracking, essay scoring, short answer scoring, etc. STS is also closely related to paraphrase identification and textual entailment recognition. Numerous research studies have been carried out on computing semantic similarity score between two sentences. The goal of the research studies in these fields is to construct a system that is able to predict the results having maximum adequateness with those assigned by human annotators. Due to the limited amount of available annotated data, variable length of sentences, and complex structure of natural language, computing semantic similarity remains a hard problem 10. An effective step was taken by computing word embeddings 11. Using word embeddings has led to valuable results in various Natural Language Processing (NLP) tasks. In recent years, in deep learning models, a variety of approaches have been proposed. These models have different architectures; therefore, their powers to detect implicit patterns for recognizing similarity are different. Some models utilized linear structure based on Recurrent Neural Network (RNN) architecture including Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) models 6,10,12-14 ; some of them use a grammar tree accompanied by input text. However, it was not clear how to effectively capture the relationships among multiple words of a sentence in such a way that yields the meaning of the sentence. Efforts to obtain embeddings for larger chunks of text had not been so successful 15. The NLP community had not found the best supervised approach for embedding that captures the semantics of a whole sentence 15. With the introduction of BERT 16 , the design of a new generation of powerful models has begun. These models are collected under the name of Transformers such as BERT, RoBERTa 17 , etc. Transformers provide general-purpose architectures for natural language understanding and natural language generation 18. Transformers are trained on a large corpus while handling long-range dependencies between input sequences and output sequences and they can capture the meaning of the
International Journal of Recent Technology and Engineering (IJRTE) , 2019
Natural Language Processing (NLP) is a subfield of Artificial Intelligence and getting lot of focus on research and development due to emergence of its applications. The research areas in focus are conversation systems, Language processing, Machine Translation, Deep learning. The researches in these areas lead to development of many tools to build industrial applications. Combining Deep Learning techniques with Natural Language Processing is finding lot of applications in domains such as Healthcare, Finance, Manufacturing, Education, Retail and customer service. This paper provides bird's view of advancement in research, development and application areas of Natural Language Processing. This paper captures21research focus areas, 22 development tools and 6 domains where Natural Language Processing are making rapid advancements.
RECKONition: a NLP-based system for Industrial Accidents at Work Prevention
2021
Extracting patterns and useful information from Natural Language datasets is a challenging task, especially when dealing with data written in a language different from English, like Italian. Machine and Deep Learning, together with Natural Language Processing (NLP) techniques have widely spread and improved lately, providing a plethora of useful methods to address both Supervised and Unsupervised problems on textual information. We propose RECKONition, a NLP-based system for Industrial Accidents at Work Prevention. RECKONition, which is meant to provide Natural Language Understanding, Clustering and Inference, is the result of a joint partnership with the Italian National Institute for Insurance against Accidents at Work (INAIL). The obtained results showed the ability to process textual data written in Italian describing industrial accidents dynamics and consequences.