chrF++: words helping character n-grams

What it is about

The paper asks whether word n-grams add anything to the character chrF score, and whether chrF tracks direct human assessment (absolute scores per segment) as well as it tracks the relative rankings used earlier. It uses the WMT 2015 and 2016 outputs with direct assessments for translation into English and into Russian, and Pearson correlation at segment level. First, β = 2 is confirmed as the best recall weight for both character and word F-scores, with β = 2 and β = 3 close. Then single n-gram orders are compared: word 1-grams and 2-grams correlate well, while word 3- and 4-grams make the word score too pessimistic, and character 3- to 6-grams are the strongest character orders. Adding word unigrams and bigrams to the six character n-grams raises the mean correlation over the test sets (about 0.58 against 0.57 for plain chrF and 0.53 for the word score). The paper names the combination chrF++ and notes that unigrams alone (chrF+) are best for Russian, and that the test sets are too small to decide between the two.

Ideas

• I1. Add word unigrams and bigrams to the character counts in tools/translate/src/measure.rs to get the chrF++ form. Drop word 3- and 4-grams, since the paper shows they hurt. Split words on whitespace and keep the counts in BTreeMap.
• I2. Word bigrams test word order, which the character score mostly misses. For the translator this is a direct probe of the linearizer in en/src/lin.rs: a change in word order should move the bigram part of the score and leave the character part alone.
• I3. Report chrF, chrF+ and chrF++ per target language. The gain in the paper was clearest for Russian, a morphologically rich target, so Hebrew-to-English and English-to-Hebrew results may differ and should not be averaged together.
• I4. Build a small direct-assessment set of about one hundred verses scored by a person, as the paper's preliminary evidence suggests. Without one, the claim that chrF++ tracks quality is untested for Mova.

For Mova

The code is tools/translate/src/measure.rs, with the word part reading the output of the linearizer. The paper gives a cheap normal-mode measure that is sensitive to word order, which the translator currently lacks outside exact mode. The first run is chrF++ against chrF on the dev split, run twice, with the word orders named in the output.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-4770

@inproceedings{popovic2017chrf,
  title = {{chrF++: words helping character n-grams}},
  author = {Popović, Maja},
  booktitle = {Proceedings of the Second Conference on Machine Translation},
  year = {2017},
  doi = {10.18653/v1/w17-4770},
  url = {https://doi.org/10.18653/v1/w17-4770}
}

References

2 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgements. In Proceedings of the ACL-05 Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization. Ann Arbor, MI, pages 65–72. find
2. Ondřej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016. Results of the WMT16 Metrics Shared Task. In Proceedings of the First Conference on Machine Translation (WMT-16). Berlin, Germany, pages 199–231. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/popovic-2017-chrf-words-helping-character-n/