Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics

What it is about

The paper checks how metrics are judged. The usual test is the Pearson correlation between a metric's system scores and human direct-assessment scores, computed on the WMT 2019 systems. The authors show that this correlation is fragile: a single outlier system, far worse than the rest, can lift the correlation of all metrics above 0.97 when it is in a sample, and removing outliers lowers the correlations for BLEU, chrF, YiSi and ESIM in several language pairs. Outliers are found with the median and the median absolute deviation, with a cutoff of 2.5. When only the top four systems are kept, correlations in some pairs fall to near zero or below zero, and rolling-window tests show that Pearson's r is unstable for small samples of close systems. The second part turns to pairwise ranking. It sets a threshold on the metric's gain that trades type I errors (accepting differences humans do not see) against type II errors (rejecting differences humans do see), and reports how often each occurs at a given threshold.

Ideas

• I1. Before any claim that a Mova measure agrees with a gold, rerun the check with and without outlier systems, using the MAD rule with cutoff 2.5, and report both r values. Apply it to the translator's own comparisons of readers, such as meta-world against word-by-word.
• I2. Decide author deltas with a paired test over verses instead of a count rule. The current rule (at least three more tokens fixed, in at least half the context) has no error rate attached; a bootstrap over the same verses gives one.
• I3. Report the spread of a score over random subsets of verses, drawn with replacement as the paper does for rankings. Two systems whose difference lies inside that spread should be called equal.
• I4. Test the metrics on close variants separately. The paper shows that metrics lose validity among near-equal systems; the meta-world ties close Bible authors for the same reason, so the same check applies there.

For Mova

The code is the evaluation harness in tools/translate and the measures in tools/translate/src/measure.rs. The paper matters for the translator's release gate, which runs once per release on held-out data. A gate on a single number is fragile in the way the paper describes, so the release report should carry the subset spread and the outlier check. The threshold analysis is the right tool for choosing between a delta and no delta.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.448 · arXiv 2006.06264

@inproceedings{mathur2020tangled,
  title = {{Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics}},
  author = {Mathur, Nitika and Baldwin, Timothy and Cohn, Trevor},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.448},
  eprint = {2006.06264},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.acl-main.448}
}

References

30 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Francis J Anscombe. 1973. Graphs in statistical analysis. The American Statistician, 27(1):17–21. find
2. Vic Barnett and Toby Lewis. 1974. Outliers in Statistical Data. Wiley. find
3. Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. find
4. Findings of the 2019 conference on machine translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1–61, Florence, Italy. Association for Computational Linguistics. find
5. Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12–58, Baltimore, Maryland, USA. Association for Computational Linguistics. find
6. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 131–198, Berlin, Germany. Association for Computational Linguistics. find
7. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the role of Bleu in machine translation research. In 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy. Association for Computational Linguistics. find
8. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1657–1668, Vancouver, Canada. Association for Computational Linguistics. find
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, USA. find
10. Marina Fomicheva and Lucia Specia. 2019. Taking MT evaluation metrics to extremes: Beyond correlation with human judgments. Computational Linguistics, 45(3):515–558. find
11. Yvette Graham and Timothy Baldwin. 2014. Testing for significance of increased correlation with human judgment. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 172–176, Doha, Qatar. find
12. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2017. Can machine translation systems be evaluated by the crowd alone? Natural Language Engineering, 23(1):3–30. find
13. Yvette Graham, Nitika Mathur, and Timothy Baldwin. 2014. Randomized significance tests in machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 266–274, Baltimore, Maryland, USA. Association for Computational Linguistics. find
14. Boris Iglewicz and David Caster Hoaglin. 1993. How to detect and handle outliers, volume 16. Asq Press. find
15. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388–395, Barcelona, Spain. find
16. Christophe Leys, Christophe Ley, Olivier Klein, Philippe Bernard, and Laurent Licata. 2013. Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median. Journal of Experimental Social Psychology, 49(4):764–766. find
17. Chi-kiu Lo. 2019. YiSi — a unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 507–513, Florence, Italy. find
18. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62–90, Florence, Italy. find
19. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2019. Putting evaluation in context: Contextual embeddings improve machine translation evaluation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2799–2808, Florence, Italy. find
20. Jason W Osborne and Amy Overbay. 2004. The power of outliers (and why researchers should always check for them). Practical Assessment, Research & Evaluation, 9(6):1–12. find
21. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002a. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, USA. find our card
22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002b. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, USA. find our card
23. Maja Popović. 2015. chrF: character n-gram f-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. find our card
24. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Belgium, Brussels. Association for Computational Linguistics. find our card
25. Ehud Reiter. 2018. A structured review of the validity of BLEU. Computational Linguistics, 44(3):393– 401. find our card
26. Peter J Rousseeuw and Mia Hubert. 2011. Robust statistics for outlier detection. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 1(1):73–79. find
27. Aaron Smith, Christian Hardmeier, and Joerg Tiedemann. 2016. Climbing mont BLEU: The strange world of reachable high-BLEU translations. In Proceedings of the 19th Annual Conference of the European Association for Machine Translation, pages 269–281. find
28. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the Association for Machine Transaltion in the Americas, pages 223–231. find our card
29. Amanda Stent, Matthew Marge, and Mohit Singhai. 2005. Evaluating evaluation methods for generation in the presence of variation. In Computational Linguistics and Intelligent Text Processing, pages 341– 351, Berlin, Heidelberg. Springer Berlin Heidelberg. find
30. Rand Wilcox. 2004. Inferences based on a skipped correlation coefficient. Journal of Applied Statistics, 31(2):131–143. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/mathur-2020-tangled-up-bleu-reevaluating-evaluation/