RED: A Reference Dependency Based MT Evaluation Metric

What it is about

RED parses only the reference, never the hypothesis, so that parsing noise from the MT output does not enter the score. It uses two kinds of reference dependency n-grams. Headword chains capture long-distance dependencies: their words must occur in the output in the same order, and the score falls with the difference between the relative distances in the reference and in the output. Fixed and floating structures capture local, continuous n-grams. The metric is tuned on WMT 2010 and tested on WMT 2012 and 2013. It correlates better with human judgments than BLEU, TER and HWCM, and after tuning and extra resources it beats METEOR and SEMPOS at system level and is comparable with METEOR at sentence level.

Ideas

• I1. Score the output against the reference structure without parsing the output. The output's word positions are compared with the reference chain's positions through the difference of relative distances. The translator's output is a linear string, so this is a direct test of word order.
• I2. Use the distance-based matching score as a residual term for en/src/lin.rs: a head and a modifier are penalised by how far apart they land, not only whether they are adjacent.
• I3. Fixed and floating structures as a check on function words and boundaries (open problem 4): a construction present in the reference must appear as a continuous chunk in the output.

For Mova

The places are tools/translate/src/measure.rs, en/src/lin.rs and tools/roundtrip. The tuned weights must be fixed on dev and checked once on test; do not tune them on the held-out verses.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{yu2014red,
  title = {{RED: A Reference Dependency Based MT Evaluation Metric}},
  author = {Yu, Hui and Wu, Xiaofeng and Xie, Jun and Jiang, Wenbin and Liu, Qun and Lin, Shouxun},
  booktitle = {Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers},
  year = {2014},
  url = {https://aclanthology.org/C14-1193/}
}

References

19 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Boxing Chen and Roland Kuhn. 2011. Amber: A modified bleu, enhanced ranking metric. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 71–77, Edinburgh, Scotland, July. Association for Computational Linguistics. find
2. Boxing Chen, Roland Kuhn, and George Foster. 2012. Improving amber, an mt evaluation metric. In Proceedings of the Seventh Workshop on Statistical Machine Translation, WMT ’12, pages 59–63, Stroudsburg, PA, USA. Association for Computational Linguistics. find
3. Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 85–91. Association for Computational Linguistics. find
4. Jesús Giménez and Lluı́s Màrquez. 2007. Linguistic features for automatic evaluation of heterogenous mt systems. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 256–264. Association for Computational Linguistics. find
5. Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81–93. find
6. Alon Lavie and Abhaya Agarwal. 2007. Meteor: an automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, StatMT ’07, pages 228–231, Stroudsburg, PA, USA. Association for Computational Linguistics. find
7. Ding Liu and Daniel Gildea. 2005. Syntactic features for evaluation of machine translation. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 25–32. find our card
8. Chi-kiu Lo and Dekai Wu. 2013. MEANT at WMT 2013: A tunable, accurate yet inexpensive semantic frame based MT evaluation metric. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 422–428, Sofia, Bulgaria, August. Association for Computational Linguistics. find
9. Chi-kiu Lo, Anand Karthik Tumuluru, and Dekai Wu. 2012. Fully automatic semantic mt evaluation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 243–252, Montréal, Canada, June. Association for Computational Linguistics. find
10. Matouš Macháček and Ondřej Bojar. 2011. Approximating a deep-syntactic metric for mt evaluation and tuning. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 92–98. Association for Computational Linguistics. find
11. Dennis Mehay and Chris Brew. 2007. BLEUTRE: Flattening Syntactic Dependencies for MT Evaluation. In Proceedings of the 11th Conference on Theoretical and Methodological Issues in Machine Translation (TMI). find
12. F.J. Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 1, pages 160–167. Association for Computational Linguistics. find
13. Karolina Owczarzak, Josef van Genabith, and Andy Way. 2007. Dependency-based automatic evaluation for machine translation. In Proceedings of the NAACL-HLT 2007/AMTA Workshop on Syntax and Structure in Statistical Translation, SSST ’07, pages 80–87, Stroudsburg, PA, USA. Association for Computational Linguistics. find
14. K. Papineni, S. Roukos, T. Ward, and W.J. Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics. find our card
15. W Pirie. 1988. Spearman rank correlation coefficient. Encyclopedia of statistical sciences. find
16. Martin F Porter. 2001. Snowball: A language for stemming algorithms. find
17. Libin Shen, Jinxi Xu, and Ralph Weischedel. 2010. String-to-dependency statistical machine translation. Computational Linguistics, 36(4):649–671. find
18. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of Association for Machine Translation in the Americas, pages 223–231. find our card
19. Matthew Snover, Nitin Madnani, Bonnie J Dorr, and Richard Schwartz. 2009. Fluency, adequacy, or hter?: exploring different human judgments with a tunable mt metric. In Proceedings of the Fourth Workshop on Statistical Machine Translation, pages 259–268. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/yu-2014-red-reference-dependency-based-mt/