Are You Sure That This Happened? Assessing the Factuality Degree of Events in Text (original) (raw)

FactBank: A corpus annotated with event factuality

Recent work in computational linguistics points out the need for systems to be sensitive to the veracity or factuality of events as mentioned in text; that is, to recognize whether events are presented as corresponding to actual situations in the world, situations that have not happened, or situations of uncertain interpretation. Event factuality is an important aspect of the representation of events in discourse, but the annotation of such information poses a representational challenge, largely because factuality is expressed through the interaction of numerous linguistic markers and constructions. Many of these markers are already encoded in existing corpora, albeit in a somewhat fragmented way. In this article, we present FACTBANK, a corpus annotated with information concerning the factuality of events. Its annotation has been carried out from a descriptive framework of factuality grounded on both theoretical findings and data analysis. FactBank is built on top of TimeBank, adding to it an additional level of semantic information.

A Factuality Profiler for Eventualities in Text

2008

parents Mercè and Toni, who have always believed in me; my siblings and allies, Arnau, Mariona, and Bernat; and my unconditional aunt and uncles, Josep, Anna, and Toni. I want to thank all of them very specially for being my base camp, my resource for rest, food, blankets, and light every time I've needed some.

The EVALITA 2016 Event Factuality Annotation Task (FactA)

2016

English. This report describes the FactA (Event Factuality Annotation) Task presented at the EVALITA 2016 evaluation campaign. The task aimed at evaluating systems for the identification of the factuality profiling of events. Motivations, datasets, evaluation metrics, and postevaluation results are presented and discussed. Italiano. Questo report descrive il task di valutazione FactA (Event Factaulity Annotation) presentato nell’ambito della campagna di valutazione EVALITA 2016. Il task si prefigge lo scopo di valutare sistemi automatici per il riconoscimento della fattualitá associata agli eventi in un testo. Le motivazioni, i dati usati, le metriche di valutazione, e risultati post-valutazione sono presentati e discussi.

Overview of FACT at IberLEF 2019: Factuality Analysis and Classification Task

2019

In this paper we describe the FACT shared task (Factuality Annotation and Classification Task), included in the First Iberian Languages Evaluation Forum (IberLEF). Factuality is understood, following [6], as the category that determines the factual status of events, that is, whether events are presented or not as certain. In order to analyze event references in texts, it is crucial to determine whether they are presented as having taken place or as potential or not accomplished events. This information can be used for different applications like Question Answering, Information Extraction, or Incremental Timeline Construction. Despite its centrality for Natural Language Understanding, this task has been underresearched, with the work by [7] as a reference for English and [8] for Spanish. For Italian, a task similar to FACT has been proposed in the past [4]. The bottleneck to advance on this task has usually been the lack of annotated resources, together with its inherent difficulty. ...

From structure to interpretation: A double-layered annotation for event factuality

Current work from different areas in the field points out the need for systems to be sensitive to the factuality nature of events mentioned in text; that is, to recognize whether events are presented as corresponding to real situations in the world, situations that have not happened, or situations of uncertain status. Event factuality is a necessary component for representing events in discourse, but for annotation purposes it poses a representational challenge because it is expressed through the interaction of a varied set of structural markers. Part of these factuality markers is already encoded in some of the existing corpora but always in a partial way; that is, missing an underlying model that is capable of representing the factuality value resulting from their interaction. In this paper, we present FactBank, a corpus of events annotated with factuality information which has been built on top of TimeBank. Together, TimeBank and FactBank offer a double-layered annotation of event factuality: where TimeBank encodes most of the basic structural elements expressing factuality information, FactBank adds a representation of the resulting factuality interpretation.

FacTA: Evaluation of Event Factuality and Temporal Anchoring

Proceedings of the Second Italian Conference on Computational Linguistics CLiC-it 2015

English. In this paper we describe FacTA, a new task connecting the evaluation of factuality profiling and temporal anchoring, two strictly related aspects in event processing. The proposed task aims at providing a complete evaluation framework for factuality profiling, at taking the first steps in the direction of narrative container evaluation for Italian, and at making available benchmark data for high-level semantic tasks. Italiano. Questo articolo descrive FacTA, un nuovo esercizio di valutazione su fattualità ed ancoraggio temporale, due aspetti dell'analisi degli eventi strettamente connessi tra loro. Il compito proposto mira a fornire una cornice completa di valutazione per la fattualità, a muovere i primi passi nella direzione della valutazione dei contenitori narrativi per l'italiano e a rendere disponibili dati di riferimento per compiti semantici di alto livello.

Determining modality and factuality for text entailment

2007

Recognizing textual entailment (TE) is a complex task involving knowledge from many different sources. One major source of information in this task is event factuality, since the inferences derivable from factual eventualities are different from those judged as possible or as non-existent. Some TE systems already factor in factuality features at the local level, but determining the factuality of events more generally involves dealing with information that is nonlocal to a particular textual event. In this paper, we present a tool providing events with their factuality values, characterized as pairs of modality and polarity features. In previous work, we identified polarity and modality at the local context with a performance of 92% precision and 56% recall. The research presented here extends and enhances our algorithm to incorporate the influence of non-local context as well as the identification of sources.

A New Dataset and Evaluation for Belief/Factuality

Proceedings of the Fourth Joint Conference on Lexical and Computational Semantics, 2015

The terms "belief" and "factuality" both refer to the intention of the writer to present the propositional content of an utterance as firmly believed by the writer, not firmly believed, or having some other status. This paper presents an ongoing annotation effort and an associated evaluation.

Epistemic Modality Certainty in Texts

This article introduces a type of uncertainty that resides in textual information and requires epistemic interpretation on the information seeker's part. Epistemic modality, as defined in linguistics and natural language processing, is a writer's estimation of the validity of propositional content in texts. It is an evaluation of chances that a certain hypothetical state of affairs is true, e.g., definitely true or possibly true. This research shifts attention from the uncertainty-certainty dichotomy to a gradient epistemic continuum of absolute, high, moderate, low certainty, and uncertainty. An analysis of a New York Times dataset showed that epistemically modalized statements are pervasive in news discourse and they occur at a significantly higher rate in editorials than in news reports. Four independent annotators were able to recognize a gradation on the continuum but individual perceptions of the boundaries between levels were highly subjective. Stricter annotation instructions and longer coder training improved intercoder agreement results. This paper offers an interdisciplinary bridge between research in linguistics, natural language processing, and information seeking with potential benefits to design and implementation of information systems for situations where large amounts of textual information are screened manually on a regular basis, for instance, by professional intelligence or business analysts.

Overview of FACT at IberLEF 2020: Events Detection and Classification

2020

In this paper we present the second edition of the FACT shared task (Factuality Annotation and Classification Task), included in IberLEF2020. The main objective of this task is to advance in the study of the factuality of the events mentioned in texts. This year, the FACT task includes a subtask on event identification in addition to the factuality classification subtask. We describe the submitted systems as well as the corpus used, which is the same used in FACT 2019 but extended by adding annotations for nominal events.