Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation

What it is about

The paper replaces the usual one-score human judgment with explicit error annotation. Professional translators, working with the full document, marked errors in the outputs of the ten top systems of WMT 2020 (human references included) for English-to-German and Chinese-to-English, using the MQM framework. The errors were given categories and severities, with at most five errors per segment and a special category for garbled segments. Scores were computed from weighted errors, and the weights were chosen by resampling: a major-to-minor ratio of 5 to 1, with punctuation errors weighted 0.1, gave the most stable system ranking. The MQM ranking sharply revised the WMT crowd ranking, clearly preferred human translation over machine output, and showed low correlation between crowd-worker scores and MQM. Among the automatic metrics, several based on pre-trained embeddings correlated better with MQM than the crowd scores did. The corpus, with over 100,000 annotated segments, is released.

Ideas

• I1. Run a stability check on the translator's residual weights: resample verses with replacement about ten thousand times and count how often the ranking of translator variants holds, as the paper does for MQM weights. Choose the weights per residual class from that, before any claim.
• I2. Cap the residual per verse, as the paper caps errors at five per segment, so that one garbled verse cannot dominate a total. Give such a verse a single "non-translation" flag, as the paper does.
• I3. Test the value of context. The paper says document context changes judgments. A reader that uses the previous verse's frames can be compared with one that does not, on the same residual.
• I4. Do not treat a single crowd-like score as gold. Check any chrF- or BLEU-based gold against a small set of expert-labelled verses before using it for decisions.

For Mova

The code is the residual and measure code in tools/translate. The paper gives the translator a stability test for its weights and a per-verse cap, both of which can be run on the Bible dev pairs without any new annotation. The corpus of expert labels is the kind of gold the translator lacks for the residual classes, and a small version of the study would be the right next step.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00437 · arXiv 2104.14478

@article{freitag2021experts,
  title = {{Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation}},
  author = {Freitag, Markus and Foster, George and Grangier, David and Ratnakar, Viresh and Tan, Qijun and Macherey, Wolfgang},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00437},
  eprint = {2104.14478},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1162/tacl_a_00437}
}

References

34 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. ALPAC. 1966. Language and Machines: Computers in Translation and Linguistics; a Report, volume 1416, National Academies. find
2. Eleftherios Avramidis, Aljoscha Burchardt, Christian Federmann, Maja Popović, Cindy Tscherwinka, and David Vilar. 2012. Involving Language Professionals in the Evaluation of Machine Translation. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 1127–1130, Istanbul, Turkey. European Language Resources Association (ELRA). find
3. Loı̈c Barrault, Magdalena Biesialska, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubešić, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. 2020. Findings of the 2020 Conference on Machine Translation (WMT20). In Proceedings of the Fifth Conference on Machine Translation, pages 1–55, Online, Association for Computational Linguistics. find
4. Luisa Bentivogli, Mauro Cettolo, Marcello Federico, and Christian Federmann. 2018. Machine Translation Human Evaluation: An investigation of evaluation based on Post-Editing and its relation with Direct Assessment. In International Workshop on Spoken Language Translation. find
5. Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 Conference on Machine Translation (WMT17). In Second Conference on Machine Translation, pages 169–214. The Association for Computational Linguistics. find
6. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 Conference on Machine Translation. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 131–198, Berlin, Germany. Association for Computational Linguistics. https://doi.org/10.18653/v1/W16-2301 ↗
7. Chris Callison-Burch, Philipp Koehn, Christof Monz, Josh Schroeder, and Cameron Shaw Fordyce. 2008. Proceedings of the Third Workshop on Statistical Machine Translation. In Proceedings of the Third Workshop on Statistical Machine Translation. find
8. Sheila Castilho, Joss Moorkens, Federico Gaspari, Rico Sennrich, Vilelmini Sosoni, Panayota Georgakopoulou, Pintu Lohar, Andy Way, Antonio Valerio Miceli Barone, and Maria Gialama. 2017. A Comparative Quality Evaluation of PBSMT and NMT using Professional Translators. AAMT. find
9. Lukas Fischer and Samuel Läubli. 2020. What’s the difference between professional human and machine translation? A blind multi-language study on domain-specific MT. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 215–224, online. European Association for Machine Translation. find
10. Marina Fomicheva. 2017. The Role of Human Reference Translation in Machine Translation Evaluation. Ph.D. thesis, Universitat Pompeu Fabra. find
11. Mikel L. Forcada, Carolina Scarton, Lucia Specia, Barry Haddow, and Alexandra Birch. 2018. Exploring gap filling as a cheaper alternative to reading comprehension questionnaires when evaluating machine translation for gisting. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 192–203. find
12. Markus Freitag, Isaac Caswell, and Scott Roy. 2019. APE at scale and its implications on MT evaluation biases. In Proceedings of the Fourth Conference on Machine Translation, pages 34–44, Florence, Italy. Association for Computational Linguistics. find
13. Markus Freitag, David Grangier, and Isaac Caswell. 2020. BLEU might be guilty but references are not innocent. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 61–71. find
14. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33–41. find
15. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2017. Can machine translation systems be evaluated by the crowd alone? Natural Language Engineering, 23(1):3–30. find
16. Yvette Graham, Barry Haddow, and Philipp Koehn. 2020. Translationese in machine translation evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 72–81. find
17. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. 2018. Achieving Human Parity on Automatic Chinese to English News Translation. arXiv preprint arXiv:1803.05567. ↗
18. Filip Klubička, Antonio Toral, and Vı́ctor M. Sánchez-Cartagena. 2018. Quantitative fine-grained human evaluation of machine translation systems: A case study on english to croatian. Machine Translation, 32(3):195–215. find
19. Philipp Koehn and Christof Monz. 2006. Manual and automatic evaluation of machine translation between european languages. In Proceedings on the Workshop on Statistical Machine Translation, pages 102–121. find
20. Moshe Koppel and Noam Ordan. 2011. Translationese and its dialects. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, pages 1318–1326. find
21. Samuel Läubli, Sheila Castilho, Graham Neubig, Rico Sennrich, Qinlan Shen, and Antonio Toral. 2020. A set of recommendations for assessing human–machine parity in language translation. Journal of Artificial Intelligence Research, 67:653–672. find
22. Samuel Läubli, Rico Sennrich, and Martin Volk. 2018. Has machine translation achieved human parity? A case for document-level evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4791–4796. find
23. Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014. Multidimensional quality metrics (MQM): A framework for declaring and describing translation quality metrics. Tradumàtica, pages 455–463. find
24. Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020. Results of the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 688–725, Online. Association for Computational Linguistics. find
25. Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. 2018. Comparing Bayesian models of annotation. Transactions of the Association for Computational Linguistics, 6:571–585. https:// doi.org/10.1162/tacl a 00040 ↗
26. Maja Popović. 2015. chrF: Character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. https://doi.org/10.18653/v1/W15-3049 pages 2685–2702, Online. Association for Computational Linguistics. ↗ our card
27. Carolina Scarton and Lucia Specia. 2016. A reading comprehension corpus for machine translation evaluation. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3652–3658. find
28. Craig Thomson and Ehud Reiter. 2020. A gold standard methodology for evaluating accuracy in data-to-text systems. In Proceedings of the 13th International Conference on Natural Language Generation, pages 158–168. find
29. Antonio Toral. 2020. Reassessing claims of human parity and super-human performance in machine translation at WMT 2019. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 185–194. find
30. Antonio Toral, Sheila Castilho, Ke Hu, and Andy Way. 2018. Attaining the unattainable? Reassessing claims of human parity in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 113–123, Belgium, Brussels. Association for Computational Linguistics. find
31. David Vilar, Gregor Leusch, Hermann Ney, and Rafael E. Banchs. 2007. Human evaluation of machine translation through binary system comparisons. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 96–103. find
32. Maja Popović. 2020. Informative manual evaluation of machine translation output. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5059–5069. find
33. John S. White, Theresa A. O’onnell, and Francis E. O’Mara. 1994. The ARPA MT evaluation methodologies: Evolution, lessons, and future approaches. In Proceedings of the First Conference of the Association for Machine Translation in the Americas. find
34. Ricardo Rei, Craig Stewart, Ana C. Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Mike Zhang and Antonio Toral. 2019. The effect of translationese in machine translation test sets. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 73–81. find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/freitag-2021-experts-errors-context-large-scale/