Ranking Translations using Error Analysis and Quality Estimation

What it is about

The paper presents TerrorCat, a learned metric trained on human pairwise rankings from WMT 2008 to 2012. Its input features are error-category frequency differences between two hypotheses. Two automatic error analysers (Addicter and Hjerson) compare each hypothesis with its reference, and every error count is normalised by hypothesis length and split by part of speech. The main new step maps each language's tag set onto 15 general part-of-speech classes with no morphology, so one classifier works for every target language; the language-independent model is roughly as good as the language-specific ones. Adding black-box quality-estimation features (sentence length, language-model probability and similar) to the English–Spanish model raised system-level correlation from 0.66 to 0.72. Segment-level correlations stay around 0.2 to 0.3.

Ideas

• I1. Use a coarse, language-free tag map (about 15 classes, no morphology) for the coarse tag of each meta token. Error counts per class then mean the same thing for every translator target.
• I2. Split the residual by coarse class as well as by layer. Lexical and order residual bits counted per class become features for ranking translator variants, instead of one total.
• I3. Reference-free features (length of the output, a count-based language-model score of the output) as an extra input when ranking author deltas in normal mode, measured on dev only.

For Mova

The places are tools/translate/src/measure.rs, tools/roundtrip, and the coarse tags of en/src/morph.rs. A learned ranker is not what Mova wants; the count-based form is. The paper learns from human rankings, which Mova does not have for verses, so the target of any ranking has to be the residual itself. It touches open problem 2 (merges that are noise) and open problem 5 (number and person as features).

Topics: Part-of-speech tagging · Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.5167/uzh-86867

@inproceedings{fishel2013ranking,
  title = {{Ranking Translations using Error Analysis and Quality Estimation}},
  author = {Fishel, Mark},
  booktitle = {Proceedings of the Eighth Workshop on Statistical Machine Translation},
  year = {2013},
  doi = {10.5167/uzh-86867},
  url = {https://doi.org/10.5167/uzh-86867}
}

References

5 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2012. Findings of the 2012 workshop on statistical machine translation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 10–51, Montréal, Canada. find
2. Mark Fishel, Rico Sennrich, Maja Popović, and Ondřej Bojar. 2012. Terrorcat: a translation error categorization-based mt quality metric. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 64–70, Montréal, Canada. find
3. Maja Popović and Hermann Ney. 2011. Towards automatic error analysis of machine translation output. Computational Linguistics, 37(4):657–688. find
4. Helmut Schmid. 1995. Improvements in part-of-speech tagging with an application to german. In Proceedings of the ACL SIGDAT-Workshop, Dublin, Ireland. find
5. Daniel Zeman, Mark Fishel, Jan Berka, and Ondřej Bojar. 2011. Addicter: What is wrong with my translations? The Prague Bulletin of Mathematical Linguistics, 96:79–88. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/fishel-2013-ranking-translations-error-analysis-quality/