Morpho-syntactic Information for Automatic Error Analysis of Statistical Machine Translation Output

What it is about

The paper uses word classes to find out where an SMT system goes wrong, for Spanish and English in the TC-STAR parliament data. It computes WER and PER separately for nouns, adjectives and verbs, and for full forms and base forms. The gap between PER and one minus WER, measured on nouns and adjectives, indicates reordering: words are translated correctly but in the wrong order. Spanish puts most adjectives after the noun, so the gap is an indicator of the noun-adjective problem in both directions. A large difference between full-form and base-form PER for Spanish indicates inflection errors. The authors also test a pre-reordering step that moves each noun past its adjective group on the source side. On a small 13,000-sentence subset this raised Spanish-to-English BLEU from 43 to about 49, but on the full corpus the gain was about 2 points, from 55 to 56, so the benefit depends on data size. Their error analysis of the full system showed the reordering gap was above 20 percent for both directions.

Ideas

I1. Compute WER and PER per word class in tools/translate/src/measure.rs, with classes taken from the UD part-of-speech tags of the stack output (tools/roundtrip trees). The PER-minus-WER gap per class is an order signal.
I2. Use full-form against base-form PER as an inflection signal. en/src/morph.rs supplies base forms, and a large gap for a target language flags a morphology problem, which open problem 5 (number and person from UD features) would then address.
I3. Test a pre-reordering rule, such as moving the noun past the adjective, on both a small and the full dev split, and run each twice. The paper's result shows the same rule can help on little data and do nothing on more, so the claim must be made on the size the translator will use.
I4. Use the per-class rates to locate the Hebrew reader's weakest part (open problem 7). The verb and subject classes are where a verb-subject order normaliser would show up.

For Mova

The code is tools/translate/src/measure.rs for the per-class rates and en/src/lin.rs for any order rule. The paper gives a diagnostic that separates reordering from inflection without a reference parser, which is useful while the translator's reader is still the weakest part. The data-size effect in the paper is a warning for any pre-reordering rule the translator considers.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{popovic2006morpho,
  title = {{Morpho-syntactic Information for Automatic Error Analysis of Statistical Machine Translation Output}},
  author = {Popović, Maja and Gispert, Adrià de and Gupta, Deepa and Lambert, Patrik and Ney, Hermann and Mariño, José B. and Federico, Marcello and Banchs, Rafael},
  booktitle = {Proceedings on the Workshop on Statistical Machine Translation},
  year = {2006},
  url = {https://aclanthology.org/W06-3101/}
}

References

18 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Bogdan Babych and Anthony Hartley. 2004. Extending bleu mt evaluation method with frequency weighting. In Proc. of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL), Barcelona, Spain, July. find
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgements. In 43rd Annual Meeting of the Assoc. for Computational Linguistics: Proc. Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization, pages 65–72, Ann Arbor, MI, June. find
Adrià de Gispert, José B. Mariño, and Josep M. Crego. 2005. Improving statistical machine translation by classifying and generalizing inflected verb forms. In Proc. of the 9th European Conf. on Speech Communication and Technology (Interspeech), pages 3185– 3188, Lisbon, Portugal, September. find
George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proc. ARPA Workshop on Human Language Technology, pages 128–132, San Diego. find
Sharon Goldwater and David McClosky. 2005. Improving stastistical machine translation through morphological analysis. In Proc. of the Conf. on Empirical Methods for Natural Language Processing (EMNLP), Vancouver, Canada, October. find
Young-suk Lee. 2004. Morphological analysis for statistical machine translation. In Proc. 2004 Meeting of the North American chapter of the Association for Computational Linguistics (HLT-NAACL), Boston, MA, May. find
Evgeny Matusov, Gregor Leusch, Oliver Bender, and Hermann Ney. 2005. Evaluating machine translation output with automatic sentence segmentation. In Proceedings of the International Workshop on Spoken Language Translation (IWSLT), pages 148–154, Pittsburgh, PA, October. find
Sonja Nießen and Hermann Ney. 2000. Improving SMT quality with morpho-syntactic analysis. In COLING ’00: The 18th Int. Conf. on Computational Linguistics, pages 1081–1085, Saarbrücken, Germany, July. find
Sonja Nießen and Hermann Ney. 2001a. Morphosyntactic analysis for reordering in statistical machine translation. In Proc. MT Summit VIII, pages 247–252, Santiago de Compostela, Galicia, Spain, September. find
Sonja Nießen and Hermann Ney. 2001b. Toward hierarchical models for statistical machine translation of inflected languages. In Data-Driven Machine Translation Workshop, pages 47–54, Toulouse, France, July. find
Sonja Nießen, Franz J. Och, Gregor Leusch, and Hermann Ney. 2000. An evaluation tool for machine translation: Fast evaluation for mt research. In Proc. Second Int. Conf. on Language Resources and Evaluation (LREC), pages 39–45, Athens, Greece, May. find
Kishore Papineni, Salim Roukos, Todd Ward, and Wie-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proc. of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pages 311–318, Philadelphia, PA, July. find our card
Maja Popović and Hermann Ney. 2004. Towards the use of word stems & suffixes for statistical machine translation. In Proc. 4th Int. Conf. on Language Resources and Evaluation (LREC), pages 1585–1588, Lissabon, Portugal, May. find
Maja Popović and Hermann Ney. 2006. POS-based word reorderings for statistical machine translation. In Proc. of the Fifth Int. Conf. on Language Resources and Evaluation (LREC), Genova, Italy, May. find
Maja Popović, David Vilar, Hermann Ney, Slobodan Jovičić, and Zoran Šarić. 2005. Augmenting a small parallel text with morpho-syntactic language resources for Serbian–English statistical machine translation. In 43rd Annual Meeting of the Assoc. for Computational Linguistics: Proc. Workshop on Building and Using Parallel Texts: Data-Driven Machine Translation and Beyond, pages 41–48, Ann Arbor, MI, June. find
David Vilar, Evgeny Matusov, Saša Hasan, Richard Zens, and Hermann Ney. 2005. Statistical machine translation of european parliamentary speeches. In Proc. MT Summit X, pages 259–266, Phuket, Thailand, September. find
David Vilar, Jia Xu, Luis Fernando D’Haro, and Hermann Ney. 2006. Error analysis of statistical machine translation output. In Proc. of the Fifth Int. Conf. on Language Resources and Evaluation (LREC), page to appear, Genova, Italy, May. find our card
Richard Zens, Oliver Bender, Saša Hasan, Shahram Khadivi, Evgeny Matusov, Jia Xu, Yuqi Zhang, and Hermann Ney. 2005. The RWTH phrase-based statistical machine translation system. In Proceedings of the International Workshop on Spoken Language Translation (IWSLT), pages 155–162, Pittsburgh, PA, October. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/popovic-2006-morpho-syntactic-information-automatic-error/