CharacTer: Translation Edit Rate on Character Level

What it is about

CharacTER is a translation edit rate computed on characters. It keeps the word-level phrase shift of TER, so a moved phrase costs something, but the edit distance after the shifts is a character-level Levenshtein distance, with spaces counted as characters. Shifting characters directly was about 44 times slower, so the shift is done on words. Three choices make it work better than TER on morphologically rich languages: the cost of a shift is the average word length of the moved phrase, two words match when identical or within one edit (for stem variants such as code and codes), and the error count is normalised by the hypothesis length instead of the reference length. The last change removes the bias of TER towards short outputs and was the largest single gain, up to about nine points in correlation. On WMT13 and WMT14 system-level correlations, the variant beats chrF and BEER on several target languages, by up to about seven points, while costing roughly ten per cent more time than TER.

Ideas

• I1. A word-shift and character-edit score for the translator's normal mode, so that hath against has or thee against thou costs a small edit and not a full miss. Compare with exact match on held-out verses, and run twice.
• I2. Test both normalisers on the Bible pairs. The hypothesis length rewards longer outputs, so adopt it only if it still wins once that bias is checked.
• I3. Charge a moved phrase by its average word length, so that displaced content words cost more than displaced function words. This is a count-based way to score the word-order residual class.
• I4. Use the one-edit stem rule as a near-match flag in the lexicon report, so that a plural or suffix variant is shown as a near merge and not as a failure.

For Mova

The places are tools/translate (measure.rs) for the normal-mode score and en/src/morph.rs for suffix tolerance. It touches open problem 5 (number and person forms that a merge such as thee/thou/ye loses). The paper is about MT scoring, not about morphology, so only the shift cost, the normaliser and the stem tolerance are taken over.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/w16-2342

@inproceedings{wang2016character,
  title = {{CharacTer: Translation Edit Rate on Character Level}},
  author = {Wang, Weiyue and Peter, Jan-Thorsten and Rosendahl, Hendrik and Ney, Hermann},
  booktitle = {Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers},
  year = {2016},
  doi = {10.18653/v1/w16-2342},
  url = {https://doi.org/10.18653/v1/w16-2342}
}

References

11 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Miloš Stanojević and Khalil Sima’an. 2014. BEER: BEtter Evaluation as Ranking. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 414–419, Baltimore, MA, USA, June. find
2. Ondřej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016. Results of the WMT16 Metrics Shared Task. In Proceedings of the First Conference on Statistical Machine Translation, Berlin, Germany, August. find
3. Miloš Stanojević, Amir Kamran, Philipp Koehn, and Ondřej Bojar, 2015. Results of the WMT15 Metrics Shared Task. Association for Computational Linguistics. find
4. George Doddington. 2002. Automatic Evaluation of Machine Translation Quality Using N-gram Co-occurrence Statistics. In Proceedings of the Second International Conference on Human Language Technology Research, pages 128–132, San Diego, CA, USA, March. find
5. Alon Lavie and Abhaya Agarwal. 2007. METEOR: An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceedings of the ACL 2007 Workshop on Statistical Machine Translation, pages 228–231, Prague, Czech Republic, June. find
6. Matouš Macháček and Ondřej Bojar. 2013. Results of the WMT13 metrics shared task. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 45–51, Sofia, Bulgaria, August. find
7. Matouš Macháček and Ondřej Bojar. 2014. Results of the WMT14 metrics shared task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 293–301, Baltimore, MA, USA, June. find
8. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 311–318, Philadelphia, PA, USA, July. find our card
9. Karl Pearson. 1895. Notes on Regression and Inheritance in the Case of Two Parents. In Proceedings of the Royal Society of London, volume 58, pages 240–242, London, UK, June. find
10. Maja Popović. 2015. CHRF: Character n-gram F-Score for Automatic MT Evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisboa, Portugal, September. find our card
11. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/wang-2016-character-translation-edit-rate-character/