Clustering-Based Summarization of Transactional Chatbot Logs (original) (raw)

Asymob: a platform for measuring and clustering chatbots

2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)

Chatbots have become a popular way to access all sorts of services via natural language. Many platforms and tools have been proposed for their construction, like Google's Dialogflow, Amazon's Lex or Rasa. However, most of them still miss integrated quality assurance methods like metrics. Moreover, there is currently a lack of mechanisms to compare and classify chatbots possibly developed with heterogeneous technologies. To tackle these issues, we present Asymob, a web platform that enables the measurement of chatbots using a suite of 20 metrics. The tool features a repository supporting chatbots built with different technologies, like Dialogflow and Rasa. Asymob's metrics help in detecting quality issues and serve to compare chatbots across and within technologies. The tool also helps in classifying chatbots along conversation topics or design features by means of two clustering methods: based on the chatbot metrics or on the phrases expected and produced by the chatbot. A video showcasing the tool is available at https://www.youtube.com/watch?v=8lpETkILpv8.

Clustering Vietnamese Conversations from Facebook Page to Build Training Dataset for Chatbot

Jordanian Journal of Computers and Information Technology, 2022

The biggest challenge of building chatbots is training data. The required data must be realistic and large enough to train chatbots. We create a tool to get actual training data from Facebook messenger of a Facebook page. After text preprocessing steps, the newly obtained dataset generates FVnC and Sample dataset. We use the Retraining of BERT for Vietnamese (PhoBERT) to extract features of our text data. K-Means and DBSCAN clustering algorithms are used for clustering tasks based on output embeddings from PhoBERT base. We apply V-measure score and Silhouette score to evaluate the performance of clustering algorithms. We also demonstrate the efficiency of PhoBERT compared to other models in feature extraction on the Sample dataset and wiki dataset. A GridSearch algorithm that combines both clustering evaluations is also proposed to find optimal parameters. Thanks to clustering such a number of conversations, we save a lot of time and effort to build data and storylines for training chatbot.

Topic-Focused Summarization of Chat Conversations

Lecture Notes in Computer Science, 2013

In this paper, we propose a novel approach to address the problem of chat summarization. We summarize real-time chat conversations which contain multiple users with frequent shifts in topic. Our approach consists of two phases. In the first phase, we leverage topic modeling using web documents to find the primary topic of discussion in the chat. Then, in the summary generation phase, we build a semantic word space to score sentences based on their association with the primary topic. Experimental results show that our method significantly outperforms the baseline systems on ROUGE F-scores.

Combining Textual Content and Structure to Improve Dialog Similarity

ArXiv, 2018

Chatbots, taking advantage of the success of the messaging apps and recent advances in Artificial Intelligence, have become very popular, from helping business to improve customer services to chatting to users for the sake of conversation and engagement (celebrity or personal bots). However, developing and improving a chatbot requires understanding their data generated by its users. Dialog data has a different nature of a simple question and answering interaction, in which context and temporal properties (turn order) creates a different understanding of such data. In this paper, we propose a novelty metric to compute dialogs' similarity based not only on the text content but also on the information related to the dialog structure. Our experimental results performed over the Switchboard dataset show that using evidence from both textual content and the dialog structure leads to more accurate results than using each measure in isolation.

Streamer-A Tool for Clustering Conversations in Social Networks

2017

In this paper, we present Streamer, a search application running over streams of Twitter messages. As opposed to most services that only do simple text search over conversations, Streamer aims to cluster messages together in order to simplify analyzing a large number of messages from similar topics. The novelty of Streamer is that, unlike most applications that use fixed corpus or categories when clustering, it works with streaming data that may debate about any number of topics: Twitter messages are continuously retrieved and the clusters are updated as more data comes in. The running time and clustering quality of the application were evaluated using purity and Silhouette coefficient.

We've had this conversation before: A Novel Approach to Measuring Dialog Similarity

arXiv (Cornell University), 2021

Dialog is a core building block of human natural language interactions. It contains multiparty utterances used to convey information from one party to another in a dynamic and evolving manner. The ability to compare dialogs is beneficial in many real world use cases, such as conversation analytics for contact center calls and virtual agent design. We propose a novel adaptation of the edit distance metric to the scenario of dialog similarity. Our approach takes into account various conversation aspects such as utterance semantics, conversation flow, and the participants. We evaluate this new approach and compare it to existing document similarity measures on two publicly available datasets. The results demonstrate that our method outperforms the other approaches in capturing dialog flow, and is better aligned with the human perception of conversation similarity.

Summarization and Visualization of Digital Conversations

2010

Search engines have become an essential tool for the majority of users for finding information in the huge amount of documents contained in the Web. Even though, for most ad-hoc search tasks, they already provide a satisfying performance, certain fundamental properties still leave room for improvement. For example, if users perform general questions, they get frequently lost in navigating the huge amount of documents returned and typically stop their search after scanning a couple of result pages. Basically, results are ranked based on word frequencies and link structures, but other factors, such as sponsored links and ranking algorithms, are also taken into account. Standard search engines do not consider semantic information that can help in recognizing the relevance of a document with respect to the meaning of a query, so that users have to analyze every document and decide which documents are relevant with respect to the meaning implied in their search. Therefore, they also struggle for matching the individualized information needs of a user. Since users are different, and want to access information according to their experience and knowledge, different techniques for constructing user models, analyzing user profiles and deriving information about a user for the adaptation of content have been proposed. An emerging approach is to use Semantic Web and Web 2.0 technologies to model information about users.

ASAP An Advanced System for Assessing Chat Participants

2008

The paper presents a method and an implemented system for the assessment of the participants’ competences in a collaborative environment based on an instant messenger conversation (chat). For each utterance in the chat, a score is computed that takes into account several features, specific to text mining (like the presence and the density of keywords, via synonymy), natural language pragmatics and to social networks. The total rating of the competence of a participant is computed considering the scores of utterances and inter-utterance factors. Within the frame of the developed system, special attention was given to multiple ways of visualizing the analysis’ results. An annotation editor was also implemented and used in order to construct a “golden standard”, which was further employed for the evaluation of the developed assessment tools.

Summarizing Online Conversations: A Machine Learning Approach

Summarization has emerged as an increasingly useful approach to tackle the problem of information overload. Extracting information from online conversations can be of very good commercial and educational value. But majority of this information is present as noisy unstructured text making traditional document summarization techniques difficult to apply. In this paper, we propose a novel approach to address the problem of conversation summarization. We develop an automatic text summarizer which extracts sentences from the conversation to form a summary. Our approach consists of three phases. In the first phase, we prepare the dataset for usage by correcting spellings and segmenting the text. In the second phase, we represent each sentence by a set of predefined features. These features capture the statistical, linguistic and sentimental aspects along with the dialogue structure of the conversation. Finally, in the third phase we use a machine learning algorithm to train the summarizer on the set of feature vectors. Experiments performed on conversations taken from the technical domain show that our system significantly outperforms the baselines on ROUGE F-scores.