Complementarity, F-score, and NLP Evaluation (original) (raw)

Abstract

This paper addresses the problem of quantifying the differences between entity extraction systems, where in general only a small proportion a document should be selected. Comparing overall accuracy is not very useful in these cases, as small differences in accuracy may correspond to huge differences in selections over the target minority class. Conventionally, one may use per-token complementarity to describe these differences, but it is not very useful when the set is heavily skewed. In such situations, which are common in information retrieval and entity recognition, metrics like precision and recall are typically used to describe performance. However, precision and recall fail to describe the differences between sets of objects selected by different decision strategies, instead just describing the proportional amount of correct and incorrect objects selected. This paper presents a method for measuring complementarity for precision, recall and F-score, quantifying the difference between entity extraction approaches.

Anthology ID:

L16-1040

Volume:

Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)

Month:

May

Year:

2016

Address:

Portorož, Slovenia

Editors:

Nicoletta Calzolari,Khalid Choukri,Thierry Declerck,Sara Goggi,Marko Grobelnik,Bente Maegaard,Joseph Mariani,Helene Mazo,Asuncion Moreno,Jan Odijk,Stelios Piperidis

Venue:

LREC

SIG:

Publisher:

European Language Resources Association (ELRA)

Note:

Pages:

261–266

Language:

URL:

https://aclanthology.org/L16-1040

DOI:

Bibkey:

Cite (ACL):

Leon Derczynski. 2016. Complementarity, F-score, and NLP Evaluation. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 261–266, Portorož, Slovenia. European Language Resources Association (ELRA).