Mova0.5 pre-release⌕
☰
home / docs / papers / Machine translation / Linguistically Motivated Evaluation of the 2022 State-of-the-art Machine Translation Systems for Three Language Directions

Linguistically Motivated Evaluation of the 2022 State-of-the-art Machine Translation Systems for Three Language Directions

What it is about

The paper runs a linguistically motivated test suite on 29 systems from the WMT 2022 shared task, in German–English, English–German and, for the first time, English–Russian. The suite has about 7,000 sentences, with more than a hundred phenomena per direction, and each output is judged pass or fail. Average accuracy for German–English and English–German is close to 95%, with about 5 points left; for English–Russian it is about 80%. The weakest phenomena include idioms, pseudogapping, resultative predicates and middle voice, and intransitive verbs of motion, whose many Russian equivalents confuse the systems. The authors also track accuracy per phenomenon across years, and note that some categories have become too easy and need harder items.

Ideas

For Mova

The places are tools/translate (meta.rs, profile.rs) and en/src/lin.rs. The suite is a held-out instrument: keep its items outside the release data and read the scores once per release, as the held-out rule says.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wmt-1.40

@inproceedings{macketanz2022linguistically,
  title = {{Linguistically Motivated Evaluation of the 2022 State-of-the-art Machine Translation Systems for Three Language Directions}},
  author = {Macketanz, Vivien and Manakhimova, Shushen and Avramidis, Eleftherios and Lapshinova-koltunski, Ekaterina and Bagdasarov, Sergei and Möller, Sebastian},
  booktitle = {Proceedings of the Seventh Conference on Machine Translation (WMT)},
  year = {2022},
  doi = {10.18653/v1/2022.wmt-1.40},
  url = {https://doi.org/10.18653/v1/2022.wmt-1.40}
}

References

27 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eleftherios Avramidis and Vivien Macketanz. 2022. Linguistically motivated evaluation of machine translation metrics based on a challenge set. In Proceedings of the Seventh Conference on Machine Translation, Abu Dhabi. Association for Computational Linguistics. find
  2. Eleftherios Avramidis, Vivien Macketanz, Arle Lommel, and Hans Uszkoreit. 2018. Fine-grained evaluation of Quality Estimation for Machine translation based on a linguistically motivated Test Suite. In Proceedings of the AMTA 2018 Workshop on Translation Quality Estimation and Automatic Post-Editing, pages 243–248, Boston, MA. Association for Machine Translation in the Americas. find
  3. Eleftherios Avramidis, Vivien Macketanz, Ursula Strohriegel, Aljoscha Burchardt, and Sebastian Möller. 2020. Fine-grained linguistic evaluation for state-of-the-art Machine Translation. In Proceedings of the Fifth Conference on Machine Translation, pages 346–356, Online. Association for Computational Linguistics. find
  4. Eleftherios Avramidis, Vivien Macketanz, Ursula Strohriegel, and Hans Uszkoreit. 2019. Linguistic Evaluation of German-English Machine Translation Using a Test Suite. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 445–454, Florence, Italy. Association for Computational Linguistics. find
  5. Ondřej Bojar, Jiří Mírovský, Kateřina Rysová, and Magdaléna Rysová. 2018. EvalD Reference-Less Discourse Evaluation for WMT18. In Proceedings of the Third Conference on Machine Translation, pages 545–549, Belgium, Brussels. Association for Computational Linguistics. find
  6. Aljoscha Burchardt, Vivien Macketanz, Jon Dehdari, Georg Heigold, Jan-Thorsten Peter, and Philip Williams. 2017. A Linguistic Evaluation of Rule-Based, Phrase-Based, and Neural MT Engines. The Prague Bulletin of Mathematical Linguistics, 108:159–170. find
  7. Franck Burlot, Yves Scherrer, Vinit Ravishankar, Ondřej Bojar, Stig-Arne Grönroos, Maarit Koponen, Tommi Nieminen, and François Yvon. 2018. The WMT’18 Morpheval test suites for English-Czech, English-German, English-Finnish and Turkish-English. In Proceedings of the Third Conference on Machine Translation, pages 550–564, Belgium, Brussels. Association for Computational Linguistics. find
  8. Silvie Cinkova and Ondřej Bojar. 2018. Testsuite on Czech–English Grammatical Contrasts. In Proceedings of the Third Conference on Machine Translation, pages 565–575, Belgium, Brussels. Association for Computational Linguistics. find
  9. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondrej Bojar. 2021. Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain. In Proceedings of the Sixth Conference on Machine Translation, pages 733–774, Online. find
  10. Liane Guillou and Christian Hardmeier. 2016. PROTEST: A Test Suite for Evaluating Pronouns in Machine Translation. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 636–643, Portorož, Slovenia. European Language Resources Association (ELRA). find
  11. Liane Guillou, Christian Hardmeier, Ekaterina Lapshinova-Koltunski, and Sharid Loáiciga. 2018. A Pronoun Test Suite Evaluation of the English– German MT Systems at WMT 2018. In Proceedings of the Third Conference on Machine Translation, pages 576–583, Belgium, Brussels. Association for Computational Linguistics. find
  12. Maja Popović. 2019. Evaluating Conjunction Disambiguation on English-to-German and French-to-German WMT 2019 Translation Hypotheses. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 464–469, Florence, Italy. Association for Computational Linguistics. find
  13. Christian Hardmeier, Christine Basta, Marta R. Costa-jussà, Gabriel Stanovsky, and Hila Gonen, editors. 2022. Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP). Association for Computational Linguistics, United States. find
  14. Alessandro Raganato, Yves Scherrer, and Jörg Tiedemann. 2019. The MuCoW Test Suite at WMT 2019: Automatically Harvested Multilingual Contrastive Word Sense Disambiguation Test Sets for Machine Translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 470–480, Florence, Italy. Association for Computational Linguistics. find
  15. Ulrich Heid and Elke Hildenbrand. 1991. Some practical experience with the use of test suites for the evaluation of SYSTRAN. In the Proceedings of the Evaluators’ Forum, Les Rasses. Citeseer. find
  16. Pierre Isabelle, Colin Cherry, and George Foster. 2017. A Challenge Set Approach to Evaluating Machine Translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2486–2496, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  17. Margaret King and Kirsten Falkedal. 1990. Using test suites in evaluation of machine translation systems. In Proceedings of the 13th conference on Computational Linguistics, volume 2, pages 211–216, Morristown, NJ, USA. Association for Computational Linguistics. find our card
  18. Tom Kocmi, Tomasz Limisiewicz, and Gabriel Stanovsky. 2020. Gender coreference and bias evaluation at wmt 2020. In Proceedings of the Fifth Conference on Machine Translation, pages 357–364, Online. Association for Computational Linguistics. find
  19. Vivien Macketanz, Eleftherios Avramidis, Aljoscha Burchardt, and Hans Uszkoreit. 2018. Fine-grained evaluation of German-English Machine Translation based on a Test Suite. In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, pages 584–593, Belgium, Brussels. Association for Computational Linguistics. find
  20. Vivien Macketanz, Eleftherios Avramidis, Aljoscha Burchardt, He Wang, Renlong Ai, Shushen Manakhimova, Ursula Strohriegel, Sebastian Möller, and Hans Uszkoreit. 2022. A Linguistically Motivated Test Suite to Semi-Automatically Evaluate German– English Machine Translation Output. In Proceedings of the Language Resources and Evaluation Conference, pages 936–947, Marseille, France. European Language Resources Association. find
  21. Annette Rios, Mathias Müller, and Rico Sennrich. 2018. The Word Sense Disambiguation Test Suite at WMT18. In Proceedings of the Third Conference on Machine Translation, pages 594–602, Belgium, Brussels. Association for Computational Linguistics. find
  22. Kateřina Rysová, Magdaléna Rysová, Tomáš Musil, Lucie Poláková, and Ondřej Bojar. 2019. A Test Suite and Manual Evaluation of Document-Level NMT at WMT19. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 455–463, Florence, Italy. Association for Computational Linguistics. find
  23. Yves Scherrer, Alessandro Raganato, and Jörg Tiedemann. 2020. The MUCOW word sense disambiguation test suite at WMT 2020. In Proceedings of the Fifth Conference on Machine Translation, pages 365– 370, Online. Association for Computational Linguistics. find
  24. Tereza Vojtěchová, Michal Novák, Miloš Klouček, and Ondřej Bojar. 2019. SAO WMT19 Test Suite: Machine Translation of Audit Reports. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 481– 493, Florence, Italy. Association for Computational Linguistics. find
  25. Andrew Way. 1991. Developer-Oriented Evaluation of MT Systems. In Proceedings of the Evaluators’ Forum, pages 237–244, Les Rasses, Vaud, Switzerland. ISSCO. find
  26. Vilém Zouhar, Tereza Vojtěchová, and Ondřej Bojar. 2020. WMT20 Document-Level Markable Error Exploration. In Proceedings of the Fifth Conference on Machine Translation, pages 371–380, Online. Association for Computational Linguistics. find
  27. Vivien Macketanz, Eleftherios Avramidis, Shushen Manakhimova, and Sebastian Möller. 2021. Linguistic Evaluation for the 2021 State-of-the-art Machine Translation Systems for German to English and English to German. In Proceedings of the Sixth Conference on Machine Translation, pages 1059–1073, Online. Association for Computational Linguistics. 81 49 56 36 69 149 76 87 19 60 59 167 4058 83 5049 5049 Ambiguity Composition Coordination & ellipsis False friends Function word LDD & interrogatives MWE Named entitiy & terminology Negation Non-verbal agreement Punctuation Subordination Verb tense/aspect/mood Verb valency micro-average macro-average 83.2 89.7 84.6 90.0 83.1 90.1 90.1 93.9 91.1 83.3 89.9 89.3 78.9 95.4 100.0 98.3 89.8 92.8 81.6 81.9 84.0 95.9 94.6 80.6 89.9 86.6 89.5 95.4 94.7 96.7 98.3 87.4 83.3 83.1 93.8 93.9 92.9 77.8 91.3 89.3 81.6 94.3 100.0 96.7 91.5 93.4 81.3 84.3 LanBr Onl-W Onl-B 85.7 89.6 84.0 95.9 94.6 83.3 88.4 88.6 82.9 90.8 100.0 91.7 98.3 88.0 84.9 83.1 Onl-A 83.3 89.4 92.6 98.0 94.6 66.7 92.8 91.9 81.6 90.8 94.7 96.7 89.8 92.2 81.5 88.0 JDExp 84.1 88.9 84.0 100.0 94.6 77.8 84.1 77.9 82.9 94.3 100.0 93.3 98.3 92.8 83.1 81.9 Onl-Y 81.8 87.7 77.8 98.0 92.9 80.6 89.9 88.6 78.9 88.5 100.0 88.3 91.5 90.4 80.4 81.9 PROMT 84.2 87.6 87.7 98.0 94.6 83.3 91.3 90.6 80.3 94.3 100.0 88.3 66.1 91.6 83.3 77.1 Onl-G 58.2 61.3 43.2 65.3 67.9 61.1 52.2 61.1 48.7 78.2 68.4 61.7 67.8 74.3 57.3 50.6 LT22 avg 80.9 86.0 81.9 93.2 90.9 77.2 85.5 84.9 78.4 91.3 95.3 90.2 87.9 89.2 79.6 79.1 77.6 78.9 81.7 76.4 84.6 88.5 76.3 85.6 81.6 86.0 86.0 82.1 83.6 86.2 90.1 90.4 87.2 85.3 91.8 91.1 find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/macketanz-2022-linguistically-motivated-evaluation-2022-state/