Natural Questions: A Benchmark for Question Answering Research (original) (raw)

Tom Kwiatkowski,Jennimaria Palomaki,Olivia Redfield,Michael Collins,Ankur Parikh,Chris Alberti,Danielle Epstein,Illia Polosukhin,Jacob Devlin,Kenton Lee,Kristina Toutanova,Llion Jones,Matthew Kelcey,Ming-Wei Chang,Andrew M. Dai,Jakob Uszkoreit,Quoc Le,Slav Petrov

Abstract

We present the Natural Questions corpus, a question answering data set. Questions consist of real anonymized, aggregated queries issued to the Google search engine. An annotator is presented with a question along with a Wikipedia page from the top 5 search results, and annotates a long answer (typically a paragraph) and a short answer (one or more entities) if present on the page, or marks null if no long/short answer is present. The public release consists of 307,373 training examples with single annotations; 7,830 examples with 5-way annotations for development data; and a further 7,842 examples with 5-way annotated sequestered as test data. We present experiments validating quality of the data. We also describe analysis of 25-way annotations on 302 examples, giving insights into human variability on the annotation task. We introduce robust metrics for the purposes of evaluating question answering systems; demonstrate high human upper bounds on these metrics; and establish baseline results using competitive methods drawn from related literature.

Anthology ID:

Q19-1026

Volume:

Transactions of the Association for Computational Linguistics, Volume 7

Month:

Year:

2019

Address:

Cambridge, MA

Editors:

Lillian Lee,Mark Johnson,Brian Roark,Ani Nenkova

Venue:

TACL

SIG:

Publisher:

MIT Press

Note:

Pages:

452–466

Language:

URL:

https://aclanthology.org/Q19-1026/

DOI:

10.1162/tacl_a_00276

Bibkey:

Cite (ACL):

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural Questions: A Benchmark for Question Answering Research. Transactions of the Association for Computational Linguistics, 7:452–466.

Cite (Informal):

Natural Questions: A Benchmark for Question Answering Research (Kwiatkowski et al., TACL 2019)

Copy Citation:

PDF:

https://aclanthology.org/Q19-1026.pdf

Video:

https://aclanthology.org/Q19-1026.mp4