Automatic MT Error Analysis: Hjerson Helping Addicter

What it is about

The paper presents an open-source tool for detailed MT error analysis. It joins two systems: Hjerson, the automatic classifier from Popović (2011), and Addicter, which brings alignment, a web interface and a corpus browser. Errors are found by comparing the output with the reference through a chosen word alignment, and then classed as morphological, reordering, missing, extra or lexical. Several alignments are built in (WER, longest common subsequence, a greedy injective alignment, and an HMM alignment), and the user can supply others, for example GIZA++ output or a "via-source" alignment that chains reference-to-source and source-to-hypothesis links. The interface shows an error summary table, a test-data browser where errors are coloured sentence by sentence, and a word explorer. The authors stress that alignment quality decides the classes: a bad alignment turns one lexical error into a pair of missing and extra words. Hjerson cannot separate wrong lexical choice from untranslated words without the source, which Addicter shows as a separate "other mismatch" class.

Ideas

• I1. The alignment choice is part of the result. For the translator's Hebrew reader, which aligns words with IBM Model 1, run the error counts with two alignments (for example LCS and an HMM-style one) and report both, as Addicter does, so that the error counts show their dependence on the alignment.
• I2. Compose alignments instead of building a new aligner. Chain the reader's Hebrew-to-meta alignment with the meta-to-English links of the lexicon in tools/translate/src/meta.rs, which gives a Hebrew-to-English alignment for error counts, in the spirit of the via-source option.
• I3. Build a static HTML report from the Rust side with a per-class count table that links to verses, so that the residual can be checked by eye. The paper's summary-to-browser link is the model.
• I4. The source is available to the translator, so it can separate wrong lexical choice from untranslated words, which the paper's tool could not. Report that split as a feature of the Mova measure.

For Mova

The code is tools/translate/src/measure.rs and a report generator in tools/translate. The alignment dependence is the translator's own problem, since the meta-world is built on alignments, so the paper's advice to measure with more than one alignment applies directly. The browser idea suits the residual review, where a number is only useful if the verse behind it can be opened.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.63317/4vvqg5vs7xyw

@inproceedings{berka2012automatic,
  title = {{Automatic MT Error Analysis: Hjerson Helping Addicter}},
  author = {Berka, Jan and Bojar, Ondřej and Fishel, Mark and Popović, Maja and Zeman, Daniel},
  booktitle = {Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)},
  year = {2012},
  doi = {10.63317/4vvqg5vs7xyw},
  url = {https://doi.org/10.63317/4vvqg5vs7xyw}
}

References

15 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ondřej Bojar. 2011. Analyzing Error Types in English-Czech Machine Translation. Prague Bulletin of Mathematical Linguistics, 95:63–76, March. find
2. Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems. In Proceedings of the EMNLP 2011 Workshop on Statistical Machine Translation. find
3. Mark Fishel, Ondřej Bojar, Daniel Zeman, and Jan Berka. 2011. Automatic Translation Error Analysis. In Text, Speech and Dialogue: 14th International Conference, TSD 2011, volume LNAI 3658. Springer Verlag, September. http://www-i6.informatik.rwth-aachen.de/web/Software/EvalTrans/index.html ↗
4. Mark Fishel, Ondřej Bojar, and Maja Popović. 2012. Terra: a Collection of Translation Error-Annotated Corpora. In International Conference on Language Resources and Evaluation, Istanbul, Turkey, May. find
5. Jesús Giménez and Lluı́s Màrquez. 2010. Asiya: An Open Toolkit for Automatic Machine Translation (Meta-)Evaluation. The Prague Bulletin of Mathematical Linguistics, (94):77–86. find
6. James W. Hunt and M. Douglas McIlroy. 1976. An Algorithm for Differential File Comparison. Computing Science Technical Report 41, Bell Laboratories, June. find
7. Phillip Koehn. 2010. An Experimental Management System. The Prague Bulletin of Mathematical Linguistics, 94. find
8. Vladimir Iosifovich Levenshtein. 1966. Binary Codes Capable of Correcting Deletions, Insertions and Reversals. Soviet Physics Doklady, 10(8):707–710, February. find
9. Nitin Madnani. 2011. iBLEU: Interactively Debugging & Scoring Statistical Machine Translation Systems. In Proceedings of the Fifth IEEE International Conference on Semantic Computing. find
10. Sonja Nießen, Franz Josef Och, Gregor Leusch, and Hermann Ney. 2000. An evaluation tool for machine translation: Fast evaluation for mt research. In Proceedings of the 2nd International Conference on Language Resources and Evaluation, pages 39–45, Athens, Greece, May-June. find
11. Franz Josef Och and Hermann Ney. 2003. A Systematic Comparison of Various Statistical Alignment Models. Computational Linguistics, 29(1):19–51. find
12. Maja Popović. 2011. Hjerson: An open source tool for automatic error classification of machine translation output. The Prague Bulletin of Mathematical Linguistics, pages 59–68. find
13. David Vilar, Jia Xu, Luis Fernando D’Haro, and Hermann Ney. 2006. Error Analysis of Machine Translation Output. In International Conference on Language Resources and Evaluation, pages 697–702, Genoa, Italy, May. find
14. Daniel Zeman, Mark Fishel, Jan Berka, and Ondřej Bojar. 2011. Addicter: What is wrong with my translations? The Prague Bulletin of Mathematical Linguistics, 96:79– 88. find
15. Ming Zhou, Bo Wang, Shujie Liu, Mu Li, Dongdong Zhang, and Tiejun Zhao. 2008. Diagnostic evaluation of machine translation systems using automatically constructed linguistic check-points. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 1121–1128, Manchester, UK. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/berka-2012-automatic-mt-error-analysis-hjerson/