Bleu: a Method for Automatic Evaluation of Machine Translation
What it is about
The paper proposes BLEU, a cheap automatic score for MT that is meant to stand in for human judges during development. A candidate is compared with one or more human references. For each n-gram order from 1 to 4, the candidate counts are clipped at the largest count that the same n-gram has in any single reference, and the clipped counts are divided by the candidate's total. The four precisions are combined by a uniform geometric mean, and a brevity penalty punishes outputs shorter than the closest reference length, computed over the whole corpus. The authors show that the precision alone already separates a human translator from a weak machine system, and that the combined score ranks five "systems" (two human translators, three machine systems) in the same order as human judges on a 500-sentence test set. The score rises with the number of references, so only scores with the same reference set can be compared.
Ideas
- I1. Add a corpus-level BLEU to
tools/translate/src/measure.rsnext to exact match and chrF: clipped n-gram counts kept inBTreeMap, max over references, brevity penalty, and the reference set named in the output. It is deterministic and needs no external tool. - I2. Use the clipping rule as a test for author deltas. A delta such as WEB «thou» → you should raise clipped precision against WEB and leave KJV-scored output unchanged except where the reference actually has «thou». Check both on dev.
- I3. Report the brevity penalty and the precision separately. A reader that drops function words gives short but precise output, and a single BLEU number hides which of the two went wrong (open problem 4).
- I4. Do not use per-verse BLEU for decisions. The geometric mean is zero whenever one order has no match, which is common in a single verse. Use corpus-level scores, or state the smoothing used.
For Mova
The code is tools/translate/src/measure.rs. The multi-reference clipping is the useful part: the Bible pairs (JPS, KJV, WEB, ASV) are exactly the case the paper built the score for, and they show the reference effect the paper warns about. The first measure to run is BLEU for the standard against the standard with one author delta, twice, with the same reference set named in every line. The archaic-reference effect (problem 6) can be measured as how much of KJV-style output is clipped away when it is scored against WEB.
Cite
@inproceedings{papineni2002bleu,
title = {{Bleu: a Method for Automatic Evaluation of Machine Translation}},
author = {Papineni, Kishore and Roukos, Salim and Ward, Todd and Zhu, Wei-Jing},
booktitle = {Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics},
year = {2002},
doi = {10.3115/1073083.1073135},
url = {https://doi.org/10.3115/1073083.1073135}
}References
- E.H. Hovy. 1999. Toward finely differentiated evaluation metrics for machine translation. In Proceedings of the Eagles Workshop on Standards and Evaluation, Pisa, Italy. find
- Kishore Papineni, Salim Roukos, Todd Ward, John Henderson, and Florence Reeder. 2002. Corpus-based comprehensive and diagnostic MT evaluation: Initial Arabic, Chinese, French, and Spanish results. In Proceedings of Human Language Technology 2002, San Diego, CA. To appear. find
- Florence Reeder. 2001. Additional mt-eval references. Technical report, International Standards for Language Engineering, Evaluation Working Group. http://issco-www.unige.ch/projects/isle/taxonomy2/ ↗
- J.S. White and T. O’Connell. 1994. The ARPA MT evaluation methodologies: evolution, lessons, and future approaches. In Proceedings of the First Conference of the Association for Machine Translation in the Americas, pages 193–205, Columbia, Maryland. find
This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/papineni-2002-bleu-method-automatic-evaluation-machine/