Automatically Generating Annotator Rationales to Improve Sentiment Classification (original) (raw)

Using ``Annotator Rationales'' to Improve Machine Learning for Text Categorization

2007

We propose a new framework for supervised machine learning. Our goal is to learn from smaller amounts of supervised training data, by collecting a richer kind of training data: annotations with "rationales." When annotating an example, the human teacher will also highlight evidence supporting this annotation-thereby teaching the machine learner why the example belongs to the category. We provide some rationale-annotated data and present a learning method that exploits the rationales during training to boost performance significantly on a sample task, namely sentiment classification of movie reviews. We hypothesize that in some situations, providing rationales is a more fruitful use of an annotator's time than annotating more examples.

Machine Learning with Annotator Rationales to Reduce Annotation Cost

We review two novel methods for text categorization, based on a new framework that utilizes richer annotations that we call annotator rationales. A human annotator provides hints to a machine learner by highlighting contextual "rationales" in support of each of his or her annotations. We have collected such rationales, in the form of substrings, for an existing document sentiment classification dataset [1]. We have developed two methods, one discriminative [2] and one generative , that use these rationales during training to obtain significant accuracy improvements over two strong baselines. Our generative model in particular could be adapted to help learn other kinds of probabilistic classifiers for quite different tasks. Based on a small study of annotation speed, we posit that for some tasks, providing rationales can be a more fruitful use of an annotator's time than annotating more examples.

Guest Editorial: Explainable artificial intelligence for sentiment analysis

Knowledge Based Systems, 2022

Social media analytics have proven valuable in numerous research areas as a pragmatic tool for public opinion mining and analysis [1]. Sentiment analysis addresses the dynamics of complex socio-affective applications that permeate intelligence and decision making in the sentient and solution-savvy Social Web [2]. Having started as simple polarity detection, contemporary sentiment analysis has advanced to a more nuanced analysis of affect and emotion sensing [3]. Detecting fine-grained sentiment in natural language, however, is tricky even for humans, making its automated detection very complicated. Moreover, online opinions can be put forth in the form of text reviews or ratings, for a product as a whole, or each of its individual aspects [4]. Multiple and lengthy reviews, usage of casual dialect with microtext (wordplay, neologism and slang), use of figurative language (sarcasm, irony), multilingual content (code-mixed and code-switched) and opinion spamming add challenges to the task of extracting opinions. Recently memes, GIFs, typographic (artistic way of text representation), info-graphic (text embedded along with an image) visual content and edited videos also dominate social feeds. Consequently, the intra-modal modeling and intermodal interactions between the textual, visual and acoustic components add to the linguistic challenges [5]. Therefore, conceptualization and development of multi-faceted sentiment analysis models to adequately capture

MARTA: Leveraging Human Rationales for Explainable Text Classification

2021

Explainability is a key requirement for text classification in many application domains ranging from sentiment analysis to medical diagnosis or legal reviews. Existing methods often rely on “attention” mechanisms for explaining classification results by estimating the relative importance of input units. However, recent studies have shown that such mechanisms tend to mis-identify irrelevant input units in their explanation. In this work, we propose a hybrid human-AI approach that incorporates human rationales into attention-based text classification models to improve the explainability of classification results. Specifically, we ask workers to provide rationales for their annotation by selecting relevant pieces of text. We introduce MARTA, a Bayesian framework that jointly learns an attention-based model and the reliability of workers while injecting human rationales into model training. We derive a principled optimization algorithm based on variational inference with efficient updat...

Knowledge-Guided Sentiment Analysis Via Learning From Natural Language Explanations

IEEE Access, 2021

Sentiment analysis is crucial for studying public opinion since it can provide us with valuable information. Existing sentiment analysis methods rely on finding the sentiment element from the content of user-generated. However, the question of why a message produces certain emotions has not been well explored or utilized in previous works. To address this challenge, we propose a natural language explanation framework for sentiment analysis that provides sufficient domain knowledge for generating additional labelled data for each new labelling decision. A rule-based semantic parser transforms these explanations into programmatic labelling functions that generate noisy labels for an arbitrary amount of unlabelled sentiment information to train a sentiment analysis classifier. Experiments on two sentiment analysis datasets demonstrate the superiority it achieves over baseline methods by leveraging explanations as external knowledge to joint training a sentiment analysis model rather th...

Learning with rationales for document classification

Machine Learning

We present a simple and yet effective approach for document classification to incorporate rationales elicited from annotators into the training of any off-the-shelf classifier. We empirically show on several document classification datasets that our classifier-agnostic approach, which makes no assumptions about the underlying classifier, can effectively incorporate rationales into the training of multinomial naïve Bayes, logistic regression, and support vector machines. In addition to being classifier-agnostic, we show that our method has comparable performance to previous classifier-specific approaches developed for incorporating rationales and feature annotations. Additionally, we propose and evaluate an active learning method tailored specifically for the learning with rationales framework.

Explaining Sentiment Classification

Interspeech 2019

This paper presents a novel 1-D sentiment classifier trained on the benchmark IMDB dataset. The classifier is a 1-D convolutional neural network with repeated convolution and max pooling layers. The main contribution of this work is the demonstration of a deconvolution technique for 1-D convolutional neural networks that is agnostic to specific architecture types. This deconvolution technique enables text classification to be explained, a feature that is important for NLP-based decision support systems, as well as being an invaluable diagnostic tool.

On the Evaluation of the Plausibility and Faithfulness of Sentiment Analysis Explanations

IFIP advances in information and communication technology, 2022

With the pervasive use of Sentiment Analysis (SA) models in financial and social settings, performance is no longer the sole concern for reliable and accountable deployment. SA models are expected to explain their behavior and highlight textual evidence of their predictions. Recently, Explainable AI (ExAI) is enabling the “third AI wave” by providing explanations for the highly non-linear black-box deep AI models. Nonetheless, current ExAI methods, especially in the NLP field, are conducted on various datasets by employing different metrics to evaluate several aspects. The lack of a common evaluation framework is hindering the progress tracking of such methods and their wider adoption. In this work, inspired by offline information retrieval, we propose different metrics and techniques to evaluate the explainability of SA models from two angles. First, we evaluate the strength of the extracted “rationales” in faithfully explaining the predicted outcome. Second, we measure the agreement between ExAI methods and human judgment on a homegrown dataset1 to reflect on the rationales plausibility. Our conducted experiments comprise four dimensions: (1) the underlying architectures of SA models, (2) the approach followed by the ExAI method, (3) the reasoning difficulty, and (4) the homogeneity of the ground-truth rationales. We empirically demonstrate that anchors explanations are more aligned with the human judgment and can be more confident in extracting supporting rationales. As can be foreseen, the reasoning complexity of sentiment is shown to thwart ExAI methods from extracting supporting evidence. Moreover, a remarkable discrepancy is discerned between the results of different explainability methods on the various architectures suggesting the need for consolidation to observe enhanced performance. Predominantly, transformers are shown to exhibit better explainability than convolutional and recurrent architectures. Our work paves the way towards designing more interpretable NLP models and enabling a common evaluation ground for their relative strengths and robustness.

A Survey on Opinion Reason Mining and Interpreting Sentiment Variations

IEEE Access, 2021

Tracking social media sentiment on a desired target is certainly an important query for many decision-makers in fields like services, politics, entertainment, manufacturing, etc. As a result, there has been a lot of focus on Sentiment Analysis. Moreover, some studies took one step ahead by analyzing subjective texts further to understand possible motives behind extracted sentiments. Few other studies took several steps ahead by attempting to automatically interpret sentiment variations. Learning reasons from sentiment variations is indeed valuable, to either take necessary actions in a timely manner or learn lessons from archived data. However, machines are still immature to carry out the full Sentiment Variations' Reasoning task perfectly due to various technical hurdles. This paper attempts to explore main approaches to Opinion Reason Mining, with focus on Interpreting Sentiment Variations. Our objectives are investigating various methods for solving the Sentiment Variations' Reasoning problem and identifying some empirical research gaps. To identify these gaps, a real-life Twitter dataset is analyzed, and key hypothesis for interpreting public sentiment variations are examined.

Using Aspect-Based Analysis for Explainable Sentiment Predictions

Natural Language Processing and Chinese Computing, 2019

Sentiment Analysis is the study of opinions produced from human written textual sources and it has become popular in recent years. The area is commonly divided into two main tasks: Documentlevel Sentiment Analysis and Aspect-based Sentiment Analysis. Recent advancements in Deep Learning have led to a breakthrough, reaching state-of-the-art accuracy scores for both tasks, however, little is known about their internal processing of these neural models when making predictions. Aiming for the development of more explanatory systems, we argue that Aspect-based Analysis can help deriving deep interpretation of the sentiment predicted by a Document-level Analysis, working as a proxy method. We propose a framework to verify if predictions produced by a trained Aspect-based model can be used to explain Document-level Sentiment classifications, by calculating an agreement metric between the two models. In our case study with two benchmark datasets, we achieve 90% of agreement between the models, thus showing the an Aspect-based Analysis should be favoured for the sake of explainability.