A Study of Translation Edit Rate with Targeted Human Annotation

What it is about

The paper proposes Translation Edit Rate (TER): the minimum number of edits (insertion, deletion, substitution, and shift of a word sequence, each costing one) that turns a system output into one of the references, divided by the average reference length. Finding the minimum with shifts is NP-complete, so the authors use a greedy search for shifts and an exact edit distance for the rest. A variant, HTER, first has a fluent native speaker edit the system output into a new reference with the same meaning, and then measures TER against that single reference. On 100 Arabic sentences with two systems, one targeted reference gave an edit rate of about 33.5 against about 49.6 for TER with four untargeted references, a reduction of about a third. HTER also correlated better with human fluency and adequacy judgments than BLEU did. Annotators spent three to seven minutes per sentence, and the authors note that a second check pass changed 0.63 words per sentence on average.

Ideas

• I1. Add a TER-style edit count with greedy shifts to tools/translate/src/measure.rs. It gives a word-level distance between the stack output and the reference, and its edit script is more informative than a single rate.
• I2. Treat the edit script as an error source. Insertions, deletions, substitutions and shifts map onto the translator's residual classes (missing and extra words, lexical choice, word order). A count of shifts tests the order class directly.
• I3. For a few hundred dev verses, have a native speaker edit the output into a fluent reference, as HTER does. Use it only on dev, never on the test split, which is read once per release.
• I4. Write the annotation rule down before editing starts. The paper warns that editors who only copy a reference make the metric look worse; the rule (fix what the metric counts, keep meaning) should be a fixed file.

For Mova

The code is tools/translate/src/measure.rs, with the edit script read out for the residual report. The HTER idea gives the translator a reference that is fluent and close to the output, which the Bible references are not, since their English is archaic (open problem 6). It is a costly human step, so it is a dev-only check for a small set of verses.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{snover2006study,
  title = {{A Study of Translation Edit Rate with Targeted Human Annotation}},
  author = {Snover, Matthew and Dorr, Bonnie and Schwartz, Rich and Micciulla, Linnea and Makhoul, John},
  booktitle = {Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers},
  year = {2006},
  url = {https://aclanthology.org/2006.amta-papers.25/}
}

References

13 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACL 2005 Workshop on Intrinsic and Extrinsic Evaulation Measures for MT and/or Summarization. find our card
2. George Doddington. 2002. Automatic Evaluation of Machine Translation Quality Using N-gram Co-Occurence Statistics. In Human Language Technology: Notebook Proceedings, pages 128–132. find
3. Robert Frederking and Sergei Nirenburg. 1994. Three Heads are Better than One. In Proceedings of the Fourth Conference on Applied Natural Language Processing, ANLP-94. find
4. Margaret King. 1996. Evaluating Natural Language Processing Systems. Communication of the ACM, 29(1):73–79, January. find
5. Kevin Knight and Ishwar Chander. 1994. Automated Postediting of Documents. In Proceedings of National Conference on Artificial Intelligence (AAAI). find
6. Inderjeet Mani, Gary Klein, David House, and Lynette Hirschman. 2002. SUMMAC: A Text Summarization Evaluation. Natural Language Engineering, 8(1):43– 68. find
7. S. Niessen, F.J. Och, G. Leusch, and H. Ney. 2000. An evaluation tool for machine translation: Fast evaluation for MT research. In Proceedings of the 2nd International Conference on Language Resources and Evaluation (LREC-2000), pages 39–45. find
8. Joseph Olive. 2005. Global Autonomous Language Exploitation (GALE). DARPA/IPTO Proposer Information Pamphlet. find
9. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Traslation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. find
10. Grazia Russo-Lassner, Jimmy Lin, and Philip Resnik. 2005. A Paraphrase-Based Approach to Machine Translation Evaluation. Technical Report LAMP-TR-125/CS-TR-4754/UMIACS-TR-2005-57, University of Maryland, College Park. find
11. Dana Shapira and James A. Storer. 2002. Edit Distance with Move Operations. In Proceedings of the 13th Annual Symposium on Computational Pattern Matching, pages 85–98. find
12. Somayajulu Sripada, Ehud Reiter, and Lezan Hawizy. 2004. Evaluating an NLG System Using Post-Editing. Technical Report AUCS/TR0402, Department of Computing Science, University of Aberdeen. find
13. Joseph P. Turian, Luke Shen, and I. Dan Melamed. 2003. Evaluation of Machine Translation and its Evaluation. In Proceedings of MT Summit IX. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/snover-2006-study-translation-edit-rate-targeted/