chrF deconstructed: beta parameters and n-gram weights

What it is about

The paper examines the two free settings of chrF: the β parameter and the weights of the n-gram orders. It sweeps β from 1/6 (precision-heavy) to 6 (recall-heavy) and computes segment-level Kendall's tau against human rankings on the WMT 2014 and WMT 2015 outputs, for English and several other target languages. It compares the character scores with the word-level F-score at the same β values. The best value is β = 2, with the word F-score peaking at the same point, and recall-heavy settings are better than precision-heavy ones. Among single character n-gram orders, 2 and 3 correlate best, and the character scores beat the word scores. Three non-uniform n-gram weight schemes were tried; they helped on some test sets and hurt on others, so the uniform weights were kept. On a small set with direct human scores, the character score varied less among good translations than the word score did, which the paper reads as tolerance of acceptable variants.

Ideas

I1. Make β a parameter in tools/translate/src/measure.rs, with a default of 2. Sweep a fixed grid on dev, report the mean and the rank columns the paper uses (share of cases where a setting beats the others), and only then choose.
I2. Keep the n-gram weights uniform. The paper finds that non-uniform weights rarely win across languages, so do not add a weight table until a dev gain holds in two runs on separate subsets.
I3. Use the spread of scores among close variants as a diagnostic. For each verse with several accepted renderings, measure how much chrF varies; a measure that is tight on good output and loose on bad output is the property the paper wants, and it can be checked on the Bible authors.
I4. Report the sweep as a table in every release note, with the test set used, so that no β value is picked after the test is seen (held-out rule).

For Mova

The code is tools/translate/src/measure.rs. The paper's main lesson for the translator is that recall weighs more than precision for judging a rendering, which fits the open question of whether the meta-world loses content (function words, open problem 4). A β sweep on the Bible dev pairs is cheap and deterministic. The uniform-weight result also argues against tuning per-order weights on a corpus with a single style.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/w16-2341

@inproceedings{popovic2016chrf,
  title = {{chrF deconstructed: beta parameters and n-gram weights}},
  author = {Popović, Maja},
  booktitle = {Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers},
  year = {2016},
  doi = {10.18653/v1/w16-2341},
  url = {https://doi.org/10.18653/v1/w16-2341}
}

References

1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the 9th Workshop on Statistical Machine Translation (WMT-14), pages 12–58, Baltimore, Maryland, USA, June. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/popovic-2016-chrf-deconstructed-beta-parameters-n/