A Call for Clarity in Reporting BLEU Scores

What it is about

The paper argues that BLEU is not one metric but a family of parameterised methods, and that published scores cannot be compared because papers differ in hidden choices. The main one is preprocessing of the reference: papers tokenise the reference themselves before scoring, so the set of n-grams changes with the tokeniser. Across the twelve WMT 2017 language arcs, the same system output scored 0.5 to 1.8 BLEU points apart depending on tokenisation, compound splitting, case handling and unknown-word masking, with about a point on average. The author checks a handful of well-cited papers and often cannot tell which configuration they used. The recommendation is to let the metric do all reference processing, as WMT does, and to publish a version string that records the settings. The paper also releases SacreBLEU, a tool that downloads the standard test-set references and reports the signature.

Ideas

• I1. Print a signature with every score in tools/translate/src/measure.rs: scorer name, n-gram order, case mode, number of references, and the reference set's name. It is the cheapest way to make a Mova score comparable with a later one.
• I2. Keep the scorer away from the references. Both sides are tokenised by the scorer, never by the caller. Add a negative control: changing the spacing of a reference must not change the score. That test would fail on a scorer that trusts caller-side preprocessing.
• I3. Pin the test set by verse id, not by line count. The paper's example of two versions of one WMT test set, with different sentence counts, is the same risk as a Bible edition with a different verse split.
• I4. Report case-sensitive and case-insensitive scores in the same table. The paper finds case handling the largest single effect.

For Mova

The code is tools/translate/src/measure.rs and the scorer's test directory. The paper's main value for the translator is a discipline, not a formula: every number carries its settings, and the references are fixed inputs. The translator's references differ in style (JPS, KJV, WEB, ASV), so the signature must also name the reference set, or two scores will look comparable when they are not.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-6319 · arXiv 1804.08771

@inproceedings{post2018call,
  title = {{A Call for Clarity in Reporting BLEU Scores}},
  author = {Post, Matt},
  booktitle = {Proceedings of the Third Conference on Machine Translation: Research Papers},
  year = {2018},
  doi = {10.18653/v1/w18-6319},
  eprint = {1804.08771},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/w18-6319}
}

References

20 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. (The restored version that was released after the evaluation (§2.4) can be selected by using -t wmt14/full.) It prints out a version string recording all the parameters as ’+’ delimited KEY.VALUE pairs (here shortened with --short): Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. ArXiv e-prints, abs/1409.0473. ↗
2. E. Black, S. Abney, D. Flickenger, C. Gdaniec, R. Grishman, P. Harrison, D. Hindle, R. Ingria, F. Jelinek, J. Klavans, M. Liberman, M. Marcus, S. Roukos, B. Santorini, and T. Strzalkowski. 1991. A procedure for quantitatively comparing the syntactic coverage of english grammars. In Speech and Natural Language: Proceedings of a Workshop Held at Pacific Grove, California, February 19-22, 1991. BLEU+c.mixed+l.en-de+#.1+s.exp +t.wmt14+tok.13a+v.1.2.10 recording: • mixed case evaluation Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference on machine translation (wmt17). In Proceedings of the Second Conference on Machine Translation, pages 169–214. Association for Computational Linguistics. • on EN-DE • with one reference • and exponential smoothing • on the WMT14 dataset • using the WMT standard ’13a’ tokenization Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluation the role of bleu in machine translation research. In 11th Conference of the European Chapter of the Association for Computational Linguistics. • with S ACRE BLEU 1.2.10. S ACRE BLEU is open source software released under the Apache 2.0 license. find
3. Summary Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian Stker, Katsuitho Sudoh, Koichiro Yoshino, and Christian Federmann. 2017. The CHRF metric is also available via the -m flag. word problem in neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 11–19. Association for Computational Linguistics. find
4. Overview of the iwslt 2017 evaluation campaign. In 14th International Workshop on Spoken Language Translation, pages 2–14, Tokyo, Japan. find
5. David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 263–270. Association for Computational Linguistics. find our card
6. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank. Computational Linguistics, Volume 19, Number 2, June 1993, Special Issue on Using Large Corpora: II. find our card
7. Jonathan H. Clark, Chris Dyer, Alon Lavie, and Noah A. Smith. 2011. Better hypothesis testing for statistical machine translation: Controlling for optimizer instability. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 176–181. Association for Computational Linguistics. find
8. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. find our card
9. Ines Rehbein and Josef van Genabith. 2007. Treebank annotation schemes and parser evaluation for german. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL). find
10. Jonas Gehring, Michael Auli, David Grangier, and Yann Dauphin. 2017. A convolutional encoder model for neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 123–135. Association for Computational Linguistics. find
11. Ehud Reiter. 2018. A structured review of the validity of bleu. Computational Linguistics, 0(0):393–401. find our card
12. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2014. Is machine translation getting better over time? In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 443–451. Association for Computational Linguistics. find
13. Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nadejde. 2017. Nematus: a toolkit for neural machine translation. In Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 65–68. Association for Computational Linguistics. find
14. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1–10. Association for Computational Linguistics. find
15. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725. Association for Computational Linguistics. find
16. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177–180. Association for Computational Linguistics. find
17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. ArXiv e-prints, abs/1706.03762. ↗
18. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, ukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. ArXiv eprints, abs/1609.08144. ↗
19. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015a. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421. Association for Computational Linguistics. find
20. Thang Luong, Ilya Sutskever, Quoc Le, Oriol Vinyals, and Wojciech Zaremba. 2015b. Addressing the rare find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/post-2018-call-clarity-reporting-bleu-scores/