COMET: A Neural Framework for MT Evaluation (original) (raw)
Abstract
We present COMET, a neural framework for training multilingual machine translation evaluation models which obtains new state-of-the-art levels of correlation with human judgements. Our framework leverages recent breakthroughs in cross-lingual pretrained language modeling resulting in highly multilingual and adaptable MT evaluation models that exploit information from both the source input and a target-language reference translation in order to more accurately predict MT quality. To showcase our framework, we train three models with different types of human judgements: Direct Assessments, Human-mediated Translation Edit Rate and Multidimensional Quality Metric. Our models achieve new state-of-the-art performance on the WMT 2019 Metrics shared task and demonstrate robustness to high-performing systems.
Anthology ID:
2020.emnlp-main.213
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Editors:
Bonnie Webber,Trevor Cohn,Yulan He,Yang Liu
Venue:
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2685–2702
Language:
URL:
https://aclanthology.org/2020.emnlp-main.213
DOI:
10.18653/v1/2020.emnlp-main.213
Bibkey:
Cite (ACL):
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A Neural Framework for MT Evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.
Cite (Informal):
COMET: A Neural Framework for MT Evaluation (Rei et al., EMNLP 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.emnlp-main.213.pdf
Video:
https://slideslive.com/38938781
Code