MEANT: An inexpensive, high-accuracy, semi-automatic metric for evaluating translation utility based on semantic roles

What it is about

MEANT judges translation utility, meaning whether a reader can recover who did what to whom. For each predicate in the reference and in the output it aligns predicates by lexical similarity, aligns the role fillers of each aligned pair by weighted bipartite matching, and computes a weighted F-score over the matched role-filler pairs. Frames are labelled either by untrained monolingual annotators (the human version, HMEANT) or by an automatic shallow semantic parser (the automatic version). On sentence-level adequacy judgments, HMEANT reaches a correlation of 0.43, the same as HTER and far above BLEU at 0.20. The automatic version reaches 0.34, about 80% of HTER, at a much lower labour cost. The study is a meta-evaluation over WMT-style data.

Ideas

• I1. A frame-level score for translator output in normal mode. For each verse, the frames of the output (predicate, roles from prag) are matched against the frames read from the reference by the same reader, and a weighted F-score per verse is reported. This gives an adequacy number without a parser on the output.
• I2. Implement the bipartite matching for role fillers deterministically: Hungarian assignment with ties broken by index, and candidates held in ordered maps, so two runs agree.
• I3. Use the predicate alignment as a check on the lexicon (open problem 2). When a concept merge makes two predicates align that the frames treat differently, the merge is counted as an error.

For Mova

The places are prag (frames), tools/roundtrip, and tools/translate/src/measure.rs. The paper's frames come from a learned SRL parser. Mova has deterministic frames, so the transferable part is the matching and the weighted score, not the parser. Measure first on the dev verses, then run the same check on the held-out verses once.

Topics: Semantic role labeling · Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{lo2011meant,
  title = {{MEANT: An inexpensive, high-accuracy, semi-automatic metric for evaluating translation utility based on semantic roles}},
  author = {Lo, Chi-kiu and Wu, Dekai},
  booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
  year = {2011},
  url = {https://aclanthology.org/P11-1023/}
}

References

20 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Satanjeev Banerjee and Alon Lavie. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the 43th Annual Meeting of the Association of Computational Linguistics (ACL-05), pages 65–72, 2005. find our card
2. Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation. In Proceedings of the Joint 5th Workshop on Statistical Machine Translation and MetricsMATR, pages 17–53, Uppsala, Sweden, 15-16 July 2010. find
3. G. Doddington. Automatic Evaluation of Machine Translation Quality using N-gram Co-occurrence Statistics. In Proceedings of the 2nd International Conference on Human Language Technology Research (HLT-02), pages 138–145, San Francisco, CA, USA, 2002. Morgan Kaufmann Publishers Inc. find
4. Jesús Giménez and Lluís Màrquez. Linguistic Features for Automatic Evaluation of Heterogenous MT Systems. In Proceedings of the 2nd Workshop on Statistical Machine Translation, pages 256–264, Prague, Czech Republic, June 2007. Association for Computational Linguistics. find
5. Jesús Giménez and Lluís Màrquez. A Smorgasbord of Features for Automatic MT Evaluation. In Proceedings of the 3rd Workshop on Statistical Machine Translation, pages 195–198, Columbus, OH, June 2008. Association for Computational Linguistics. find
6. Philipp Koehn and Christof Monz. Manual and Automatic Evaluation of Machine Translation between European Languages. In Proceedings of the Workshop on Statistical Machine Translation, pages 102–121, 2006. find
7. Gregor Leusch, Nicola Ueffing, and Hermann Ney. CDer: Efficient MT Evaluation Using Block Movements. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics (EACL-06), 2006. find
8. Ding Liu and Daniel Gildea. Syntactic Features for Evaluation of Machine Translation. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, page 25, 2005. find our card
9. Ding Liu and Daniel Gildea. Source-Language Features and Maximum Correlation Training for Machine Translation Evaluation. In Proceedings of the 2007 Conference of the North American Chapter of the Association of Computational Linguistics (NAACL-07), 2007. find
10. Chi-kiu Lo and Dekai Wu. Evaluating machine translation utility via semantic role labels. In Seventh International Conference on Language Resources and Evaluation (LREC-2010), pages 2873–2877, Malta, May 2010. find
11. Chi-kiu Lo and Dekai Wu. Semantic vs. syntactic vs. n-gram structure for machine translation evaluation. In Dekai Wu, editor, Proceedings of SSST-4, Fourth 229 Workshop on Syntax and Structure in Statistical Translation (at COLING 2010), pages 52–60, Beijing, Aug 2010. find
12. Chi-kiu Lo and Dekai Wu. SMT vs. AI redux: How semantic frames evaluate MT more accurately. In 22nd International Joint Conference on Artificial Intelligence (IJCAI-11), Barcelona, Jul 2011. To appear. find
13. Sonja Nießen, Franz Josef Och, Gregor Leusch, and Hermann Ney. A Evaluation Tool for Machine Translation: Fast Evaluation for MT Research. In Proceedings of the 2nd International Conference on Language Resources and Evaluation (LREC-2000), 2000. find
14. Karolina Owczarzak, Josef van Genabith, and Andy Way. Evaluating machine translation with LFG dependencies. Machine Translation, 21:95–119, 2008. find
15. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL-02), pages 311–318, 2002. find our card
16. Sameer Pradhan, Wayne Ward, Kadri Hacioglu, James H. Martin, and Dan Jurafsky. Shallow Semantic Parsing Using Support Vector Machines. In Proceedings of the 2004 Conference on Human Language Technology and the North American Chapter of the Association for Computational Linguistics (HLT-NAACL-04), 2004. find
17. Mark Przybocki, Kay Peterson, Sébastien Bronsart, and Gregory Sanders. The NIST 2008 Metrics for Machine Translation Challenge - Overview, Methodology, Metrics, and Results. Machine Tr, 23:71–103, 2010. find
18. Matthew Snover, Bonnie J. Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas (AMTA-06), pages 223–231, 2006. find our card
19. Christoph Tillmann, Stephan Vogel, Hermann Ney, Arkaitz Zubiaga, and Hassan Sawaf. Accelerated DP Based Search For Statistical Translation. In Proceedings of the 5th European Conference on Speech Communication and Technology (EUROSPEECH-97), 1997. find
20. Clare R. Voss and Calandra R. Tate. Task-based Evaluation of Machine Translation (MT) Engines: Measuring How Well People Extract Who, When, Where-Type Elements in MT Output. In Proceedings of the 11th Annual Conference of the European Association for Machine Translation (EAMT-2006), pages 203–212, Oslo, Norway, June 2006. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lo-2011-meant-inexpensive-high-accuracy-semi/