A Structured Review of the Validity of BLEU

What it is about

The paper asks whether BLEU is a valid evaluation technique, meaning whether its scores correlate with the real-world usefulness or user satisfaction of NLP systems. It is a structured review in the clinical sense: the author fixed the search and inclusion criteria before reading. The ACL Anthology was searched by title for terms such as "evaluating evaluation", "metric validity" and "BLEU", and all papers reporting WMT metric results were added. Papers were screened for English, standard BLEU, a correlation with human judgment, at least five systems or texts, and no double counting. From 34 papers the review collected 284 correlations, each with its granularity (system or text), reference count, gold-standard type and correlation kind. The main finding is that the evidence supports BLEU for diagnostic evaluation of MT systems, which is what it was made for, but not for evaluating single texts, not outside MT, and not for scientific hypothesis testing. The paper grades each correlation as high, medium or low, with 0.85 and 0.70 as the cut-offs.

Ideas

• I1. State the validity scope in the output of tools/translate/src/measure.rs: system-level diagnostic only, not single-verse judgment. Verse-level scores then stay out of any decision, and the report says so.
• I2. Keep a structured table of every Mova measure against the gold it was checked on, using the review's fields: granularity, number of references, gold type, correlation kind, and the value. It is a small file in docs/ that grows with each release.
• I3. Apply the paper's inclusion rule to the translator's own validity claims: no correlation claim with fewer than five systems or texts. Below that, say that the measure is unvalidated.
• I4. Record possible bias in each check, as the review does. When the measure and the system are both the translator's own work, the table should name that.

For Mova

The code is the report code of tools/translate and the measure notes in docs/. The review gives the translator a rule for what it may claim: its scores are diagnostics for system design (author deltas, reader choices), not a judgment of one verse. It also gives a template for keeping the validity record in one place, which the translator needs once more than one measure is in use.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00322

@article{reiter2018structured,
  title = {{A Structured Review of the Validity of BLEU}},
  author = {Reiter, Ehud},
  journal = {Computational Linguistics},
  year = {2018},
  doi = {10.1162/coli\_a\_00322},
  url = {https://doi.org/10.1162/coli_a_00322}
}

References

16 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Banerjee, Satanjeev and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL-2005 Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72. find our card
2. Belz, Anja and Albert Gatt. 2008. Intrinsic vs. extrinsic evaluation measures for referring expression generation. In Proceedings of ACL-2008, pages 197–200. find
3. Belz, Anja and Ehud Reiter. 2006. Comparing automatic and human evaluation of NLG systems. In Proceedings of EACL 2006, pages 313–320. find
4. Bojar, Ondřej, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016a. Results of the WMT16 metrics shared task. In Proceedings of WMT-2016, pages 199–231. find
5. Bojar, Ondřej, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016b. Findings of the 2016 conference on machine translation. In Proceedings of WMT-2016, pages 131–198. find
6. Bouamor, Houda, Hanan Alshikhabobakr, Behrang Mohit, and Kemal Oflazer. 2014. A human judgment corpus and a metric for Arabic MT evaluation. In Proceedings of EMNLP-2014, pages 207–213. find
7. Callison-Burch, Chris, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the role of BLEU in machine translation research. In Proceedings of EACL-2006, pages 249–256. find
8. Doddington, George. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of HLT 2002, pages 138–145. find
9. Espinosa, Dominic, Rajakrishnan Rajkumar, Michael White, and Shoshana Berleant. 2010. Further meta-evaluation of broad-coverage surface realization. In Proceedings of EMNLP-2010, pages 564–574. find
10. Graham, Yvette. 2015. Re-evaluating automatic summarization with BLEU and 192 shades of ROUGE. In Proceedings of EMNLP-2015, pages 128–137. find
11. Kilickaya, Mert, Aykut Erdem, Nazli Ikizler-Cinbis, and Erkut Erdem. 2017. Re-evaluating automatic metrics for image captioning. In Proceedings of EACL-2017, pages 199–209. find
12. Moher, David, Alessandro Liberati, Jennifer Tetzlaff, and Douglas G. Altman. 2009. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. BMJ, 339:b2535. find
13. Novikova, Jekaterina, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Proceedings of EMNLP-2017, pages 2241–2252. find
14. Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of ACL 2002, pages 311–318. find our card
15. Prasad, Vinay, Chul Kim, Mauricio Burotto, and Andrae Vandross. 2015. The strength of association between surrogate end points and survival in oncology: A systematic review of trial-level meta-analyses. JAMA Internal Medicine, 175(8):1389–1398. find
16. Reiter, Ehud and Anja Belz. 2009. An investigation into the validity of some metrics for automatically evaluating natural language generation systems. Computational Linguistics, 35(4):529–558. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/reiter-2018-structured-review-validity-bleu/