A Survey of Qualitative Error Analysis for Neural Machine Translation Systems

What it is about

The paper is a qualitative review of errors made by neural MT systems in production, across 33 languages from 10 families. Native speakers reviewed samples and grouped the errors into named-entity issues, morphology, syntax, omission or addition of text, punctuation, capitalisation, and pathological output such as repetition or hallucination. Problematic translations are singled out as the riskiest: those that could carry health, safety, legal or financial consequences, or that add toxic language absent from the source. The authors report average shares for three of the groups: named-entity issues about 4%, morphology about 2% and syntax about 3%. They argue that averages such as BLEU hide these classes, and that a categorised, challenge-style test set is the first step toward fixing them. The text available to this card was extracted from slides, so the layout is lost and the figures are as stated there.

Ideas

• I1. Use the category list as the residual class set for the translator's normal mode: named entities (divine names, places, people), morphology (UD FEATS), order, omission and addition, punctuation and capitalisation, and a pathological class for looping output.
• I2. A deterministic omission and addition count per verse: meta tokens with no chunk in the output, and output words with no meta source. Both are counts and go straight into the residual.
• I3. A repeat guard in the encoder: the same chunk three times in a row fails the verse outright, not just a score drop.

For Mova

The places are tools/translate/src/measure.rs for the classes and en/src/lin.rs for order errors. The neural error rates are not Mova's concern, but the categories are useful as a checklist: they cover what the translator's residual classes (open problems 4 and 5) do not yet name, such as omission and pathological output.

Topics: Named entity recognition · Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{diaz2020survey,
  title = {{A Survey of Qualitative Error Analysis for Neural Machine Translation Systems}},
  author = {Díaz, Denise and Cross, James and Chaudhary, Vishrav and Kishky, Ahmed and Koehn, Philipp},
  booktitle = {Proceedings of the 14th Conference of the Association for Machine Translation in the Americas (Volume 2: User Track)},
  year = {2020},
  url = {https://aclanthology.org/2020.amta-user.3/}
}

References

5 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Koehn, P. and Knowles, R. (2017). Six challenges for neural machine translation. find
2. Lommel, A. (2018). Metrics for translation quality assessment: a case for standardising error typologies. In Translation Quality Assessment, pages 109–127. Springer. find
3. Popovic, M. (2018). Error classification and analysis for machine translation quality assessment. In Translation Quality Assessment, pages 129–158. Springer. find
4. Stahlberg, F. (2020). The Roles of Language Models and Hierarchical Models in Neural Sequence-to-Sequence Prediction. PhD thesis, University of Cambridge. find
5. Ugawa, A., Tamura, A., Ninomiya, T., Takamura, H., and Okumura, M. (2018). Neural MT incorporating named entity. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3240–3250, S Proceedings of the 14th Conference of the Association for Machine Translation in the Americas October 6 - 9, 2020, Volume 2: MT User Track find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/diaz-2020-survey-qualitative-error-analysis-neural/