Error Analysis of Statistical Machine Translation Output

What it is about

The paper gives a hand-built error typology for statistical MT output and applies it to the RWTH system in the first TC-STAR evaluation, for Spanish-to-English, English-to-Spanish and Chinese-to-English, on speech-derived data (clean text, verbatim transcripts and recogniser output). The top level has five classes: missing words, word order, incorrect words, unknown words and punctuation. Missing words are split into content and filler words, word order into word and phrase level, local and long range, and incorrect words into wrong lexical choice, wrong disambiguation, wrong form, extra words, style and idioms. The authors observe that automatic metrics such as WER, PER, BLEU and NIST cannot say what the main problems are. For English-to-Spanish, verb tense errors made up about 15 percent of the errors, and noun-adjective reordering was prominent. For Spanish-to-English, lexical choice and disambiguation were about 28 percent, and most missing words were filler words that did not change the meaning. The typology is the basis for a later automatic classifier.

Ideas

• I1. Use this typology as the label set for the translator's residual: missing versus extra words, local and long-range order, word and phrase order, wrong lexical choice, wrong form, idiom. Most residual classes already match, so the change is in the names.
• I2. Count missing filler words apart from missing content words in tools/translate/src/measure.rs. This is open problem 4 in another form: function words and boundaries should be features of meta tokens, and their loss needs its own number.
• I3. Treat the be-type disambiguation as a sense class. The paper's example, two Spanish verbs for one English verb, is the analogue of merging two concepts that differ in use, such as be and have in the current lexicon (open problem 2).
• I4. Report verb tense and person as a separate error class for every target language with a verbal system. This is a morphology measure that en/src/morph.rs could supply.

For Mova

The code is the residual labels in tools/translate and the morphology side in en/src/morph.rs. The typology gives the translator a human-readable error list that matches its own classes, so that residual numbers can be read in the same terms as earlier error studies. The paper's finding that filler words account for much of the missing material supports the open problem on function words.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{vilar2006error,
  title = {{Error Analysis of Statistical Machine Translation Output}},
  author = {Vilar, David and Xu, Jia and D'Haro, Luis Fernando and Ney, Hermann},
  booktitle = {Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06)},
  year = {2006},
  url = {https://aclanthology.org/L06-1244/}
}

References

5 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proc. ARPA Workshop on Human Language Technology. find
2. LDC. 2005. Linguistic data consortium chinese training data resources. http://www.ldc.upenn.edu/Projects/TIDES/mt2005cn.htm. ↗
3. Ariadna Font Llitjós, Jaime G. Carbonell, and Alon Lavie. 2005. A framework for interactive and automatic refinement of transfer-based machine translation. In Proc. of the 10th Annual Conf. of the European Association for Machine Translation (EAMT), Budapest, Hungary, May. find
4. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proc. of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 311–318, Philadelphia, PA, July. find our card
5. David Vilar, Evgeny Matusov, Saša Hasan, Richard Zens, and Hermann Ney. 2005. Statistical Machine Translation of European Parliamentary Speeches. In Proceedings of MT Summit X, pages 259–266, Phuket, Thailand, September. Asia-Pacific Association for Machine Translation (AAMT). find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/vilar-2006-error-analysis-statistical-machine-translation/