Towards Heterogeneous Automatic MT Error Analysis

What it is about

The paper argues that automatic MT metrics should be used for error analysis, not only for a single score. It compiles a set of metric variants at five linguistic levels: lexical (WER, PER, BLEU, NIST, GTM, ROUGE, METEOR), shallow syntax, dependency and constituency overlap, shallow semantics (named entities and semantic roles), and discourse representations. The variants are computed in the IQMT framework. The case study uses the NIST 2005 Arabic–English test set, with one human-aided system and the best statistical system. Surface metrics prefer the statistical system, while deeper metrics give rankings closer to human judgments. Reports are produced at document and sentence level, showing at which level a system loses. There is no formal study of how reliable the reports are.

Ideas

• I1. A level profile for each translator output: score it per level (concept, tag, head and relation, frame role, word order) and print the level where the score falls. The profile replaces one residual number with a short diagnosis.
• I2. Keep the paper's warning as a rule. When a translator variant improves the exact-match score but not the frame-level score, the surface gain is flagged. A delta is kept only if the deeper-level score also improves.
• I3. A dependency-overlap score for subject and object pairs, computed on the meta trees of two readings of one verse. It measures open problem 1 directly: do the two readings agree on the heads?

For Mova

The places are tools/translate/src/measure.rs, tools/roundtrip (frames), prag (roles) and tools/macula (Hebrew trees). The metric variants themselves are not needed, since Mova's frames are deterministic. What carries over is the level-by-level report. Nothing here has been tried in Mova.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.63317/3fq8iorodjww

@inproceedings{gimenez2008heterogeneous,
  title = {{Towards Heterogeneous Automatic MT Error Analysis}},
  author = {Giménez, Jesús and Màrquez, Lluís},
  booktitle = {Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)},
  year = {2008},
  doi = {10.63317/3fq8iorodjww},
  url = {https://doi.org/10.63317/3fq8iorodjww}
}

References

27 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Enrique Amigó, Julio Gonzalo, Anselmo Peñas, and Felisa Verdejo. 2005. QARLA: a Framework for the Evaluation of Automatic Sumarization. In Proceedings of the 43th Annual Meeting of the Association for Computational Linguistics. find
2. Enrique Amigó, Jesús Giménez, Julio Gonzalo, and Lluı́s Màrquez. 2006. MT Evaluation: Human-Like vs. Human Acceptable. In Proceedings of the Joint 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics (COLING-ACL). find
3. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization. Currently, most metrics are only available for English. Some are also available for Spanish and Catalan. find our card
4. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the Role of BLEU in Machine Translation Research. In Proceedings of EACL. find
5. Simon Corston-Oliver, Michael Gamon, and Chris Brockett. 2001. A Machine Learning Approach to the Automatic Evaluation of Machine Translation. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics (ACL), pages 140–147. find
6. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the 2nd Internation Conference on Human Language Technology, pages 138–145. find
7. Michael Gamon, Anthony Aue, and Martine Smets. 2005. Sentence-Level MT evaluation without reference translations: beyond language modeling. In Proceedings of EAMT, pages 103–111. find
8. Jesús Giménez and Lluís Màrquez. 2008. On the Robustness of Linguistic Features for Automatic MT Evaluation. (Under submission). find
9. Jesús Giménez and Lluı́s Màrquez. 2007. Linguistic Features for Automatic Evaluation of Heterogeneous MT Systems. In Proceedings of the ACL Workshop on Statistical Machine Translation, pages 256–264. find
10. Jesús Giménez. 2007. IQMT v2.1. Technical Manual (LSI-07-29-R). Technical report, TALP Research Center. LSI Department. http://www.lsi.upc.edu/˜nlp/-IQMT/IQMT.v2.1.pdf. ↗
11. Hans Kamp. 1981. A Theory of Truth and Semantic Representation. In J.A.G. Groenendijk, T.M.V. Janssen, and M.B.J. Stokhof, editors, Formal Methods in the Study of Language, pages 277–322. Mathematisch Centrum, address = Amsterdam. find
12. David Kauchak and Regina Barzilay. 2006. Paraphrasing for Automatic Evaluation. In Proceedings of the Joint Conference on Human Language Technology and the North American Chapter of the Association for Computational Linguistics (HLT-NAACL), pages 455–462. find
13. Alex Kulesza and Stuart M. Shieber. 2004. A learning approach to improving sentence-level MT evaluation. In Proceedings of the 10th International Conference on Theoretical and Methodological Issues in Machine Translation. find
14. Audrey Le and Mark Przybocki. 2005. NIST 2005 machine translation evaluation official results. In Official release of automatic evaluation scores for all submissions, August. find
15. Chin-Yew Lin and Franz Josef Och. 2004a. Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL). find
16. Chin-Yew Lin and Franz Josef Och. 2004b. ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation. In Proceedings of the 20th International Conference on Computational Linguistics (COLING). find
17. Ding Liu and Daniel Gildea. 2005. Syntactic Features for Evaluation of Machine Translation. In Proceedings of ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization. find our card
18. Dennis Mehay and Chris Brew. 2007. BLEUATRE: Flattening Syntactic Dependencies for MT Evaluation. In Proceedings of the 11th Conference on Theoretical and Methodological Issues in Machine Translation (TMI). find
19. I. Dan Melamed, Ryan Green, and Joseph P. Turian. 2003. Precision and Recall of Machine Translation. In Proceedings of the Joint Conference on Human Language Technology and the North American Chapter of the Association for Computational Linguistics (HLT-NAACL). find
20. Sonja Nießen, Franz Josef Och, Gregor Leusch, and Hermann Ney. 2000. An Evaluation Tool for Machine Translation: Fast Evaluation for MT Research. In Proceedings of the 2nd International Conference on Language Resources and Evaluation. find
21. Karolina Owczarzak, Declan Groves, Josef Van Genabith, and Andy Way. 2006. Contextual Bitext-Derived Paraphrases in Automatic MT Evaluation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas (AMTA), pages 148–155. find
22. Karolina Owczarzak, Josef van Genabith, and Andy Way. 2007. Dependency-Based Automatic Evaluation for Machine Translation. In Proceedings of SSST, NAACL-HLT/AMTA Workshop on Syntax and Structure in Statistical Translation, pages 80–87. find
23. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. Bleu: a method for automatic evaluation of machine translation, rc22176. Technical report, IBM T.J. Watson Research Center. find our card
24. Grazia Russo-Lassner, Jimmy Lin, and Philip Resnik. 2005. A Paraphrase-Based Approach to Machine Translation Evaluation. Technical report, University of Maryland, College Park. find
25. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, , and John Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation. In Proceedings of AMTA, pages 223–231. find our card
26. C. Tillmann, S. Vogel, H. Ney, A. Zubiaga, and H. Sawaf. 1997. Accelerated DP based Search for Statistical Translation. In Proceedings of European Conference on Speech Communication and Technology. find
27. Liang Zhou, Chin-Yew Lin, and Eduard Hovy. 2006. Re-evaluating Machine Translation Results with Paraphrase Support. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 77–84. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/gimenez-2008-heterogeneous-automatic-mt-error-analysis/