HUME: Human UCCA-Based Evaluation of Machine Translation

What it is about

HUME is a human semantic evaluation measure. Only the source sentence is annotated, with UCCA (a cross-lingual scheme with scenes, participants, centres, elaborators and linkers). Bilingual annotators then judge whether each unit is preserved in the translation. Because the garbled output is never parsed and no reference is needed, the measure avoids two weak points of earlier semantic measures. The study covers English to Czech, German, Polish and Romanian, reports good inter-annotator agreement, shows that annotation quality is robust to sentence length, and finds correlation with direct-assessment adequacy scores. The paper also compares HUME with HMEANT: in its UCCA Wikipedia sample, copula clauses (about 22% of sentences), nominal argument structures (about 49%) and linkers (about 56%) are ones that a verb-only frame measure ignores.

Ideas

• I1. A source-side coverage score without a reference. For each meta token read from the source verse, check whether the output chunk for its concept is present. The score per verse is the share preserved, weighted by unit type (centres above elaborators, as the paper weights by type).
• I2. Count omitted function words and linkers per verse as their own residual class (open problem 4). The paper shows they are common, and frame-based measures miss them.
• I3. Keep the source-only design. The reader already reads one source, so a HUME-style check measures what the standard encoder drops, without the noise of a second parse of the output.

For Mova

The places are tools/translate (reader and encoder) and prag (units). The human judgments are not for Mova. The automatic source-side check is, and it can run on every verse.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/d16-1134

@inproceedings{birch2016hume,
  title = {{HUME: Human UCCA-Based Evaluation of Machine Translation}},
  author = {Birch, Alexandra and Abend, Omri and Bojar, Ondřej and Haddow, Barry},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1134},
  url = {https://doi.org/10.18653/v1/d16-1134}
}

References

33 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Omri Abend and Ari Rappoport. 2013. Universal conceptual cognitive annotation (ucca). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 228–238, Sofia, Bulgaria, August. Association for Computational Linguistics. find our card
2. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of Linguistic Annotation Workshop and Interoperability with Discourse, pages 178–186. find our card
3. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, MI, USA. Association for Computational Linguistics. find our card
4. Alexandra Birch, Barry Haddow, Ulrich Germann, Maria Nadejde, Christian Buck, and Philipp Koehn. 2013. The feasibility of HMEANT as a human MT evaluation metric. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 52–61, Sofia, Bulgaria, August. Association for Computational Linguistics. find
5. Ondřej Bojar and Dekai Wu. 2012. Towards a Predicate-Argument Evaluation for MT. In Proceedings of the Sixth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 30–38, Jeju, Republic of Korea, July. Association for Computational Linguistics. find
6. Ondřej Bojar, Miloš Ercegovčević, Martin Popel, and Omar F. Zaidan. 2011. A grain of salt for the WMT manual evaluation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 1–11, Edinburgh, Scotland. find
7. Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12–58, Baltimore, Maryland, USA, June. Association for Computational Linguistics. find
8. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015. Findings of the 2015 workshop on statistical machine translation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 1–46, Lisbon, Portugal, September. Association for Computational Linguistics. find
9. Alexander Chuchunkov, Alexander Tarelkin, and Irina Galinskaya. 2014. Applying HMEANT to English-Russian Translations. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 43–50, Doha, Qatar, October. Association for Computational Linguistics. find
10. Robert M.W. Dixon. 2010a. Basic Linguistic Theory: Grammatical Topics, volume 2. Oxford University Press. find
11. Robert M.W. Dixon. 2010b. Basic Linguistic Theory: Methodology, volume 1. Oxford University Press. find
12. Robert M.W. Dixon. 2012. Basic Linguistic Theory: Further Grammatical Topics, volume 3. Oxford University Press. find
13. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the second international conference on Human Language Technology Research, pages 138–145, San Diego, CA, USA. Morgan Kaufmann Publishers Inc. find
14. Marina Fomicheva and Lucia Specia. 2016. Reference bias in monolingual machine translation evaluation. In 54th Annual Meeting of the Association for Computational Linguistics, ACL, Berlin, Germany. find
15. Jesús Giménez and Lluís Màrquez. 2007. Linguistic features for automatic evaluation of heterogenous mt systems. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 256–264. find
16. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2015a. Can machine translation systems be evaluated by the crowd alone? Natural Language Engineering, pages 1–28. ↗
17. Yvette Graham, Nitika Mathur, and Timothy Baldwin. 2015b. Accurate evaluation of segment-level machine translation metrics. In Proc. of NAACL-HLT, pages 1183–1191. find
18. Yvette Graham. 2015. Improving evaluation of machine translation quality estimation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1804–1813, Beijing, China, July. Association for Computational Linguistics. find
19. Jan Hajič, Eva Hajičová, Jarmila Panevová, Petr Sgall, Ondřej Bojar, Silvie Cinková, Eva Fučíková, Marie Mikulová, Petr Pajas, Jan Popelka, Jiří Semecký, Jana Šindlerová, Jan Štěpánek, Josef Toman, Zdeňka Urešová, and Zdeněk Žabokrtský. 2012. Announcing Prague Czech-English Dependency Treebank 2.0. In Proceedings of the Eighth International Language Resources and Evaluation Conference (LREC’12), pages 3153–3160, Istanbul, Turkey, May. ELRA, European Language Resources Association. find
20. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, Companion Volume Proceedings of the Demo and Poster Sessions, pages 177–180, Prague, Czech Republic. Association for Computational Linguistics. find
21. Ding Liu and Daniel Gildea. 2005. Syntactic features for evaluation of machine translation. In ACL 2005 Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 25–32. find our card
22. Chi-kiu Lo and Dekai Wu. 2011. Structured vs. flat semantic role representations for machine translation evaluation. In Proceedings of the Fifth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 10–20. Association for Computational Linguistics. find
23. Chi-Kiu Lo and Dekai Wu. 2014. On the Reliability and Inter-Annotator Agreement of Human Semantic MT Evaluation via HMEANT. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykjavik, Iceland. European Language Resources Association (ELRA). find
24. Arle Richard Lommel, Maja Popovic, and Aljoscha Burchardt. 2014. Assessing Inter-Annotator Agreement for Translation Error Annotation. In MTE: Workshop on Automatic and Manual Metrics for Operational Translation Evaluation. LREC. find
25. Matouš Macháček and Ondřej Bojar. 2015. Evaluating Machine Translation Quality Using Short Segments Annotations. The Prague Bulletin of Mathematical Linguistics, 103:85–110, April. ↗
26. Pedro Marinotti. 2014. Measuring semantic preservation in machine translation with HCOMET: human cognitive metric for evaluating translation. Master’s thesis, University of Edinburgh. ↗
27. Stephan Oepen and Jan Tore Lønning. 2006. Discriminant-based MRS banking. In Proceedings of LREC, pages 1250–1255. find our card
28. Karolina Owczarzak, Josef van Genabith, and Andy Way. 2007. Evaluating machine translation with LFG dependencies. Machine Translation, 21(2):95–119. ↗
29. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, PA, USA. Association for Computational Linguistics. find our card
30. Petr Sgall, Eva Hajičová, and Jarmila Panevová. 1986. The Meaning of the Sentence and Its Semantic and Pragmatic Aspects. Academia/Reidel Publishing Company, Prague, Czech Republic/Dordrecht, Netherlands. find
31. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of association for machine translation in the Americas, pages 223–231. find our card
32. Elior Sulem, Omri Abend, and Ari Rappoport. 2015. Conceptual annotations preserve structure across translations: A French-English case study. In ACL 2015 Workshop on Semantics-Driven Statistical Machine Translation (S2MT), pages 11–22. find
33. Jörg Tiedemann. 2009. News from OPUS – a collection of multilingual parallel corpora with tools and interfaces. In Recent Advances in Natural Language Processing, volume 5, pages 237–248, Borovets, Bulgaria. John Benjamins. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/birch-2016-hume-human-ucca-based-evaluation/