Performance Evaluation of Keyword Extraction Techniques and Stop Word Lists on Speech-To-Text Corpus (original) (raw)
Related papers
Automatic keyword extraction for the meeting corpus using supervised approach and bigram expansion
Spoken Language Technology Workshop, …, 2008
In this paper, we tackle the problem of automatic keyword extraction in the meeting domain, a genre significantly different from written text. For the supervised framework, we proposed a rich set of features beyond the typical TFIDF measures, such as sentence salience weight, lexical features, summary sentences, and speaker information. We also evaluate different candidate sampling approaches for better model training and testing. In addition, we introduced a bigram expansion module which aims at extracting "entity bigrams" using Web resources. Using the ICSI meeting corpus, we demonstrate the effectiveness of the features and show that the supervised method and the bigram expansion module outperform the unsupervised TFIDF selection with POS (part-of-speech) filtering. Finally, we show the approaches introduced in this paper perform well on the speech recognition output.
Unsupervised approaches for automatic keyword extraction using meeting transcripts
… : The 2009 Annual Conference of the …, 2009
This paper explores several unsupervised approaches to automatic keyword extraction using meeting transcripts. In the TFIDF (term frequency, inverse document frequency) weighting framework, we incorporated partof-speech (POS) information, word clustering, and sentence salience score. We also evaluated a graph-based approach that measures the importance of a word based on its connection with other sentences or words. The system performance is evaluated in different ways, including comparison to human annotated keywords using F-measure and a weighted score relative to the oracle system performance, as well as a novel alternative human evaluation. Our results have shown that the simple unsupervised TFIDF approach performs reasonably well, and the additional information from POS and sentence score helps keyword extraction. However, the graph method is less effective for this domain. Experiments were also performed using speech recognition output and we observed degradation and different patterns compared to human transcripts.
Keyword Extraction Performance Analysis
2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR)
This paper presents a survey-cum-evaluation of methods for the comprehensive comparison of the task of keyword extraction using datasets of various sizes, forms, and genre. We use four different datasets which includes Amazon product data-Automotive, SemEval 2010, TMDB and Stack Exchange. Moreover, a subset of 100 Amazon product reviews is annotated and utilized for evaluation in this paper, to our knowledge, for the first time. Datasets are evaluated by five Natural Language Processing approaches (3 unsupervised and 2 supervised), which include TF-IDF, RAKE, TextRank, LDA and Shallow Neural Network. We use a tenfold cross-validation scheme and evaluate the performance of the aforementioned approaches using recall, precision and F-score. Our analysis and results provide guidelines on the proper approaches to use for different types of datasets. Furthermore, our results indicate that certain approaches achieve improved performance with certain datasets due to inherent characteristics of the data.
Improved Keyword and Keyphrase Extraction from Meeting Transcripts
Keywords play a vital role in extracting the correct information as per user requirements. Keywords are like index terms that contain the most important information about the content of the document. Keyword Extraction is the task of identifying a keyword or keyphrase from a document that can help users easily to understand the documents. Meeting transcripts is significantly different from document or other speech domains. This paper aims to extract keywords and keyphrases from meeting transcripts and also to add some additional features for improving the keyword and keyphrase extraction method. Here, this method is performed by both human transcripts and ASR transcripts and the keywords are extracted through MaxEnt and SVM classifier and Extraction of bigram and trigram keywords retrieval using N-gram based approach efficiently and also to identify the low frequency keywords using LDA (Latent Dirichlet Approach).Finally, the quality of the Extracted keywords is improved using pattern features through sequential pattern mining.
Computing Research Repository, 2004
Lexical resources such as WordNet and the EDR electronic dictionary (EDR) have been used in several NLP tasks. Probably partly due to the fact that the EDR is not freely available, WordNet has been used far more often than the EDR. We have used both resources on the same task in order to make a comparison possible. The task is automatic assignment of keywords to multi-party dialogue episodes (i.e. thematically coherent stretches of spoken text). We show that the use of lexical resources in such a task results in slightly higher performances than the use of a purely statistically based method.
Application of Extractive Text Summarization Algorithms to Speech-to-Text Media
Lecture Notes in Computer Science
This paper presents how speech-to-text summarization can be performed using extractive text summarization algorithms. Our objective is to make a recommendation about which of the six text summary algorithms evaluated in the study is the most suitable for the task of audio summarization. First, we have selected six text summarization algorithms: Luhn, TextRank, LexRank, LSA, SumBasic, and KLSum. Then, we have evaluated them on two datasets, DUC2001 and OWIDSum, with six ROUGE metrics. After that, we have selected five speech documents from ISCI Corpus dataset, and we have transcribed using the Automatic Speech Recognition (ASR) from Google Cloud Speech API. Finally, we applied the studied extractive summarization algorithms to these five text samples to obtain a text summary from the original audio file. Experimental results showed that Luhn and TextRank obtained the best performance for the task of extractive speech-to-text summarization on the samples evaluated.
Using web text to improve keyword spotting in speech
2013
For low resource languages, collecting sufficient training data to build acoustic and language models is time consuming and often expensive. But large amounts of text data, such as online newspapers, web forums or online encyclopedias, usually exist for languages that have a large population of native speakers. This text data can be easily collected from the web and then used to both expand the recognizer's vocabulary and improve the language model. One challenge, however, is normalizing and filtering the web data for a specific task. In this paper, we investigate the use of online text resources to improve the performance of speech recognition specifically for the task of keyword spotting. For the five languages provided in the base period of the IARPA BABEL project, we automatically collected text data from the web using only Limit-edLP resources. We then compared two methods for filtering the web data, one based on perplexity ranking and the other based on out-of-vocabulary (OOV) word detection. By integrating the web text into our systems, we observed significant improvements in keyword spotting accuracy for four out of the five languages. The best approach obtained an improvement in actual term weighted value (ATWV) of 0.0424 compared to a baseline system trained only on LimitedLP resources. On average, ATWV was improved by 0.0243 across five languages.
Keyword Spotting: An Audio Mining Technique in Speech Processing – A Survey
Audio mining is a branch of data mining that is used to search and analyze the contents of audio signal automatically. Keyword spotting (KWS) is an important audio mining technique which searches audio signals for finding the occurrences of given keyword within the input spoken utterance. KWS provides a satisfactory audio mining solution for various tasks like spoken document indexing and retrieval. The research in audio mining has received increasing attention due to the increase in amount of audio content in the Internet, telephone call conversation and other sources. KWS is classified according to the type of input speech content and the method used for spotting. A number of approaches have been used in keyword spotting like DTW, HMM, Neural Network, Vector quantization and other approaches. KWS has been utilized in a broad variety of applications. A majority of such applications relate to audio indexing and phone call routing. In this paper, various audio mining methods and keyword spotting techniques are discussed.
Spoken Keyword Based Speech Recognizer for Extracting Information
This paper investigates multilayer feed forward back-propagation artificial neural network hybridized with ontology, a knowledge engineering tool for developing speech recognizer that will assist to retrieve Indian tourism information in hands free mode. Spoken key words are collected from 30 speakers of both genders in Indian tourism domain. This paper also used ontology technique a knowledge engineering approach in order to reduce the search space during information extraction. Thirteen Mel frequency cepstral coefficients have been applied to represent speech signal. Features from the selected frames are used to train multilayer FFBPNN. The same routine is applied to the speech signal during the recognition stage in such a way that unknown test patterns are classified to the nearest patterns and retrieved. The system can be said to be robust as average accuracy for clean data is 88.6% % while that for noisy data is 67.7 %.
Sentence extraction-based automatic speech summarization and evaluation techniques
This paper presents automatic speech summarization tech-niques and its evaluation metrics, focusing on sentence extraction-based summarization methods for making abstracts from spontaneous presentations. Since humans tend to sum-marize presentations by extracting important sentences from introduction and conclusion parts, this paper proposes a method using sentence location. Experimental results show that the proposed method significantly improves automatic speech summarization performance for the condition of 10% sum-marization ratio. Results of correlation analysis between subjective and objective evaluation scores confirm that ob-jective evaluation metrics, including summarization accu-racy, sentence F-measure and ROUGE-N, are effective for evaluating summarization techniques.