Semi-automatic error analysis for large-scale statistical machine translation

What it is about

The paper relates properties of the input documents to the translation quality of a large SMT system, scored per document by HTER. The properties are the source show or newspaper, genre, dialect, style, out-of-vocabulary rate, source and target language-model perplexity, the rate of correctly translated names, and for speech input the ASR error types. Each factor is ranked by mutual information with the score, using Freedman–Diaconis binning for continuous values. The data are the 2006 GALE evaluation sets for Arabic and Chinese, text and audio, in four genres. Across most conditions the source and the OOV rate rank highest. For broadcast news, ASR deletions are the strongest factor, traced to very quiet speakers and to acoustic segmentation. Dialect ranks lower than expected. No inter-annotator agreement was measured.

Ideas

• I1. Before writing any new author delta, rank per-verse features (source book, translator, OOV rate of the Hebrew concept, verse length, reader confidence) by mutual information with the residual bits. Features at the top point to where a delta or a lexicon merge will pay.
• I2. Use the same ranking to choose the next style deltas (open problem 8). Add a control: the mutual information of a shuffled copy of each feature, so that small verse counts cannot produce a top factor by chance.
• I3. Treat the reader's OOV rate per verse as a feature. The paper's OOV effect suggests that unseen concepts are where the standard encoder fails, and this is visible without any reference.

For Mova

The place is tools/translate/src/measure.rs, with a small report in the same crate. Bins and tables must be ordered (BTreeMap), and the report must print the same output on two runs. Verses are short, so mutual-information estimates are noisy. The paper works at document level; Mova works per verse, so check every top factor against the shuffled-feature control before it is believed.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{kirchhoff2007semi,
  title = {{Semi-automatic error analysis for large-scale statistical machine translation}},
  author = {Kirchhoff, Katrin and Rambow, Owen and Habash, Nizar and Diab, Mona},
  booktitle = {Proceedings of Machine Translation Summit XI: Papers},
  year = {2007},
  url = {https://aclanthology.org/2007.mtsummit-papers.39/}
}

References

11 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. S. Banerjee and A. Lavie (2005) METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization, Ann Arbor, MI. find our card
2. G. Doddington (2002) Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In: Proceedings of the ARPA Workshop on Human Language Technology (HLT), San Diego, CA, USA, pp. 128-132. find
3. D. Freedman and P. Diaconis (1981) On the histogram as a density estimator: L 2 theory. Probability Theory and Related Fields 57(4), 453-476. find
4. Philip Koehn (2003) Noun phrase translation. PhD Thesis, University of Southern California. find
5. E. Matusov N. Ueffing and H. Ney (2006) Computing Consensus Translation from Multiple Machine Translation Systems Using Enhanced Hypotheses Alignment. In Proceedings of EACL 2006, pp. 33-40, Trento, Italy, April 2006. find
6. S. Niessen, F.J. Och, G. Leusch, and H. Ney (2000) An evaluation tool for machine translation: fast evaluation for MT research. In: Proceedings of the Second International Conference on Language Resources and Evaluation (LREC), pp. 39-45. find
7. F.J. Och, D. Gildea, S. Khudanpur, A. Sarkar, K. Yamada, A. Fraser, S. Kumar, L. Shen, D. Smith, K. Eng, V. Jain, Z. Jin and D. Radev (2003) Syntax for Statistical Machine Translation. Final Report of the Johns Hopkins 2003 Summer Workshop. K. Papineni, S. Roukos, T. Ward and W-J. find
8. Zhu (2002) BLEU: A method for automatic evaluation of machine translation. In: Proceedings of ACL 2002. find our card
9. M. Popovic and H. Ney (2006) Error Analysis of Verb Inflections in Spanish Translation Output. In: Proceedings of the TC-STAR Workshop on Speech-to-Speech Translation, Barcelona, Spain, pp. 99-103. find
10. M. Popovic, A. de Gispert, D. Gupta, P. Lambert, H. Ney, J.B. Mariño, M. Federico, and R. Banchs (2006) Morphosyntactic Information for Automatic Error Analysis of Statistical Machine Translation Output, Proceedings of the ACL Workshop on Statistical Machine Translation, pp. 1-6. find our card
11. M. Snover, B. Dorr, R. Schwartz, J. Makhoul, L. Micciulla and R. Weischedel (2005) A study of translation error rate with targeted human annotation, University of Maryland Technical Report, UMIACS-TR-2005-58. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/kirchhoff-2007-semi-automatic-error-analysis-large/