chrF: character n-gram F-score for automatic MT evaluation

What it is about

The paper proposes chrF, an F-score over character n-grams. Precision is the share of hypothesis character n-grams found in the reference, recall is the share of reference n-grams found in the hypothesis, both averaged over n-gram lengths up to 6, and a parameter β gives recall β times the weight of precision. The metric needs no tokeniser, no extra resources and no language-specific rules. On the WMT 2012 to 2014 outputs for seven target languages, the system-level correlations with human rankings were comparable to or better than BLEU, TER and METEOR. At segment level on WMT 2014, the variant with β = 3 (chrF3) reached the highest Kendall correlations for translation from English and was close to the best shared-task metrics for translation into English. The choice of β = 3 is described as a preliminary value.

Ideas

• I1. Implement chrF in tools/translate/src/measure.rs as the normal-mode second score, with character 6-gram counts in BTreeMap. It needs no tokeniser, which matters for inflected forms and for any later non-English target, where word matching breaks on inflection (open problem 5).
• I2. Use chrF as a secondary check on author deltas. An override that raises exact match but lowers chrF on the author's dev set is suspicious, because chrF sees spelling-level and form-level gains that exact match cannot.
• I3. Do not adopt the paper's β = 3 without a test. Run β in {1, 2, 3} on dev, report each value, and choose by the two-run check. The paper itself leaves β open.
• I4. Compare the English output of the stack with its own gold at the segment level and at the system level, and report both, since the paper's two levels disagree for some languages.

For Mova

The code is tools/translate/src/measure.rs, the normal-mode scorer. chrF fits normal mode because it does not depend on the target tokeniser, which the translator does not yet control well for Hebrew-to-English and for the linearizer output in en/src/lin.rs. It does not replace exact mode, where the residual in bits is the gate. The first measure is chrF on the Bible dev split, run twice, with the β value stated.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/w15-3049

@inproceedings{popovic2015chrf,
  title = {{chrF: character n-gram F-score for automatic MT evaluation}},
  author = {Popović, Maja},
  booktitle = {Proceedings of the Tenth Workshop on Statistical Machine Translation},
  year = {2015},
  doi = {10.18653/v1/w15-3049},
  url = {https://doi.org/10.18653/v1/w15-3049}
}

References

11 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgements. In Proceedings of the ACL 05 Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization, pages 65–72, Ann Arbor, MI, June. find
2. Ondřej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2013. Findings of the 2013 Workshop on Statistical Machine Translation. In Proceedings of the Eighth Workshop on Statistical Machine Translation (WMT-13), pages 1–44, Sofia, Bulgaria, August. find
3. Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. Findings of the 2014 Workshop on Statistical Machine Translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation (WMT-14), page 1258, Baltimore, Maryland, USA, June. find
4. Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2012. find
5. Findings of the 2012 Workshop on Statistical Machine Translation. In Proceedings of the Seventh Workshop on Statistical Machine Translation (WMT-12), page 1051, Montral, Canada, June. Association for Computational Linguistics. find
6. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL 02), pages 311– 318, Philadelphia, PA, July. find our card
7. Kristen Parton, Joel Tetreault, Nitin Madnani, and Martin Chodorow. 2011. E-Rating Machine Translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation (WMT-11), pages 108– 115, Edinburgh, Scotland. find
8. Maja Popović. 2011. Morphemes and POS tags for n-gram based evaluation metrics. In Proceedings of the Sixth Workshop on Statistical Machine Translation (WMT-11), pages 104–107, Edinburgh, Scotland, July. find
9. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A Study of Translation Error Rate with Targeted Human Annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas (AMTA 06), pages 223–231, Boston, MA, August. find
10. Miloš Stanojević and Khalil Sima’an. 2014a. BEER: BEtter Evaluation as Ranking. In Proceedings of the Ninth Workshop on Statistical Machine Translation (WMT-14), pages 414–419, Baltimore, Maryland, June. find
11. Miloš Stanojević and Khalil Sima’an. 2014b. Fitting Sentence Level Translation Evaluation with Many Dense Features. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP-14), pages 202–206, Doha, Qatar, October. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/popovic-2015-chrf-character-n-gram-f/