From Human to Automatic Error Classification for Machine Translation Output

What it is about

The paper turns a manual error typology into an automatic classifier. The classifier uses the word-level alignment behind WER and two position-independent error rates, one computed on the reference (words of the reference missing from the output) and one on the hypothesis (extra words in the output). Each error is then placed in one of five classes. Inflectional errors are those where the base form is right but the full form is wrong, judged with base forms from TreeTagger. Reordering errors are words present on both sides that WER counts as errors. Missing and extra words are errors that the position-independent rates count and that share no base form with an error elsewhere. Everything else is a lexical error. The method is language-independent apart from needing base forms. It was compared with human classification on three GALE outputs (strict) and three WMT 2009 outputs (flexible). Correlations across error classes and across systems were high, with the weakest result for extra words. Precision was lower for reordering, because frequent words such as articles, "and" and "or" are wrongly counted as moved.

Ideas

• I1. Port the rule set to the translator's readers. Words in the output that the reference does not match are missing or extra; words on both sides that the alignment marks as moved are reordering. Base forms can come from en/src/morph.rs. Implement it as a count-based classifier in tools/translate/src/measure.rs.
• I2. Handle frequent function words as their own count. The paper shows they produce false reordering counts. This connects to open problem 4, where function words become features of meta tokens.
• I3. Use the classifier per author delta. Each delta should move errors out of one class. A before-and-after class count table for each override is a direct check that the override does what it claims.
• I4. Build the paper's confusion table between lexical errors and missing or extra words for the meta-world's merge decisions. It shows which concept merges (open problem 2) generate the wrong class.

For Mova

The code is tools/translate/src/measure.rs with base forms from en/src/morph.rs. The classifier is a rule-based, count-based measure of error classes, which is what the translator needs for its residual. The paper's own caveat, that the classes depend on the alignment, applies directly to the translator's Hebrew reader.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{popovic2011human,
  title = {{From Human to Automatic Error Classification for Machine Translation Output}},
  author = {Popović, Maja and Burchardt, Aljoscha},
  booktitle = {Proceedings of the 15th Annual Conference of the European Association for Machine Translation},
  year = {2011},
  url = {https://aclanthology.org/2011.eamt-1.36/}
}

References

13 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Avramidis, Eleftherios and Philipp Koehn. 2008. Enriching morphologically poor languages for statistical machine translation. In Proceedings of the 46rd Annual Meeting of the Association for Computational Linguistics (ACL 08), pages 763–770, Columbus, Ohio, June. find
2. Callison-Burch, Chris, Philipp Koehn, Christof Monz, and Josh Schroeder. 2009. Findings of the 2009 Workshop on Statistical Machine Translation. In Proceedings of the Fourth Workshop on Statistical Machine Translation, pages 1–28, Athens, Greece, March. find
3. Callison-Burch, Chris, Philipp Koehn, Christof Monz, Kay Peterson, Mark Przybocki, and Omar Zaidan. 2010. Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR (WMT 10), pages 17–53, Uppsala, Sweden, July. find
4. Khalilov, Maxim and José A. R. Fonollosa. 2009. N-gram-based statistical machine translation versus syntax augmented machine translation: comparison and system combination. In Proceedings of the 12th Conference of the European Chapter of the Association for Computation Linguistic (EACL 09), pages 424–432, Athens, Greece, March. find
5. Levenshtein, Vladimir Iosifovich. 1966. Binary Codes Capable of Correcting Deletions, Insertions and Reversals. Soviet Physics Doklady, 10(8):707–710, February. find
6. Li, Jin-Ji, Jungi Kim, Dong-Il Kim, and Jong-Hyeok Lee. 2009. Chinese syntactic reordering for adequate generation of korean verbal phrases in chinese-to-korean smt. In Proceedings of the Fourth Workshop on Statistical Machine Translation (WMT 09), pages 190–196, Athens, Greece, March. find
7. Llitjós, Ariadna Font, Jaime G. Carbonell, and Alon Lavie. 2005. A framework for interactive and automatic refinement of transfer-based machine translation. In Proceedings of the 10th Annual Conference of the European Association for Machine Translation (EAMT 05), pages 87–96, Budapest, Hungary, May. find
8. Lopez, Adam and Philip Resnik. 2005. Pattern visualization for machine translation output. In Proceedings of HLT/EMNLP on Interactive Demonstrations, pages 12–13, Vancouver, Canada, October. find
9. Max, Aurélien, Rafik Makhloufi, and Philippe Langlais. 2008. Explorations in using grammatical dependencies for contextual phrase translation disambiguation. In Proceedings of the 12th Annual Conference of the European Association for Machine Translation (EAMT 08), pages 114–119, Hamburg, Germany, September. find
10. Popović, Maja and Hermann Ney. 2007. Word Error Rates: Decomposition over POS classes and Applications for Error Analysis. In Proceedings of the 2nd ACL 07 Workshop on Statistical Machine Translation (WMT 07), pages 48–55, Prague, Czech Republic, June. find
11. Popović, Maja, Adrià de Gispert, Deepa Gupta, Patrik Lambert, Hermann Ney, José B. Mariño, Marcello Federico, and Rafael Banchs. 2006. Morphosyntactic Information for Automatic Error Analysis of Statistical Machine Translation Output. In Proceedings of the 1st NAACL 06 Workshop on Statistical Machine Translation (WMT 06), pages 1–6, New York, NY, June. find our card
12. Vilar, David, Jia Xu, Luis Fernando D’Haro, and Hermann Ney. 2006. Error Analysis of Statistical Machine Translation Output. In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC 06), pages 697–702, Genoa, Italy, May. find our card
13. Zhou, Ming, Bo Wang, Shujie Liu, Mu Li, Dongdong Zhang, and Tiejun Zhao. 2008. Diagnostic evaluation of machine translation systems using automatically constructed linguistic check-points. Proceedings of the 22nd International Conference on Computational Linguistics (CoLing 2008), pages 1121– 1128, August. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/popovic-2011-human-automatic-error-classification-machine/