Blast: A Tool for Error Analysis of Machine Translation Output

What it is about

Blast is an open-source Java tool with a graphical interface for human error analysis of MT output. The annotator sees a source, a reference and a hypothesis segment, and tags the wrong words with error types taken from a hierarchical typology that is loaded from a file. Annotations are stored as word indices with a label, so one word span can carry an error. A preprocessing step highlights similarities between the hypothesis and the reference at several levels (exact, lower-cased, stemmed, synonym, paraphrase), adapted from the METEOR-NEXT aligner. The tool has annotate, edit and search modes and prints counts per typology node. The paper is a tool description; it reports no quality measurement.

Ideas

• I1. Store each residual as a span of token indices with a class from a typology file, the way Blast does. The classes already in use (lexical, construction, order, function words, punctuation and case, morphology) become a data file, and a tools/translate report lists the spans per class.
• I2. A search over translated verses for all spans of one class (for example every order error, whatever its subclass), used to check that a fix to one rule did not create new errors of the same class elsewhere.
• I3. A similarity overlay for a translator output against each author's reference, at exact, lower-cased, stem and concept levels, shown as support marks. A human reviewer then sees which meta tokens already agree before deciding what a residual means.

For Mova

The useful part is the design, not the code: error labels attached to token indices, the typology as data, and counts by typology node. The place is tools/translate (measure.rs) and its per-class residual report. Blast's typology is hand-made; Mova's classes come from the residual, so the typology file must be generated from counts, not written by hand. Nothing here has been tried in Mova.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{stymne2011blast,
  title = {{Blast: A Tool for Error Analysis of Machine Translation Output}},
  author = {Stymne, Sara},
  booktitle = {Proceedings of the ACL-HLT 2011 System Demonstrations},
  year = {2011},
  url = {https://aclanthology.org/P11-4010/}
}

References

15 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Lars Ahrenberg, Magnus Merkel, and Michael Petterst-edt. 2003. Interactive word alignment for language engineering. In Proceedings of EACL, pages 49–52, Budapest, Hungary. find
2. Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Proceedings of ACL, pages 597–604, Ann Arbor, Michigan, USA. find our card
3. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (Meta-) evaluation of machine translation. In Proceedings of WMT, pages 136–158, Prague, Czech Republic, June. find
4. Michael Denkowski and Alon Lavie. 2010. METEOR-NEXT and the METEOR paraphrase tables: Improved evaluation support for five target languages. In Proceedings of WMT and MetricsMATR, pages 339–342, Uppsala, Sweden. find
5. Mireia Farrús, Marta R. Costa-jussà, José B. Mariño, and José A. R. Fonollosa. 2010. Linguistic-based evaluation criteria to identify statistical machine translation errors. In Proceedings of EAMT, pages 52–57, Saint Raphaël, France. find
6. Mary Flanagan. 1994. Error classification for MT evaluation. In Proceedings of AMTA, pages 65–72, Columbia, Maryland, USA. find
7. Ariadna Font Llitjós and Jaime Carbonell. 2004. The translation correction tool: English-Spanish user studies. In Proceedings of LREC, pages 347–350, Lisbon, Portugal. find
8. Meghan Lammie Glenn, Stephanie Strassel, Lauren Friedman, and Haejoong Lee. 2008. Management of large annotation projects involving multiple human judges: a case study of GALE machine translation post-editing. In Proceedings of LREC, pages 2957– 2960, Marrakech, Morocco. find
9. Eduard Hovy, Margaret King, and Andrei Popescu-Belis. 2002. Principles of context-based machine translation evaluation. Machine Translation, 17(1):43–75. find
10. Masaki Murata, Kiyotaka Uchimoto, Qing Ma, Toshiyuki Kanamaru, and Hitoshi Isahara. 2005. Analysis of machine translation systems’ errors in tense, aspect, and modality. In Proceedings of PACLIC 19, pages 155–166, Taipei, Taiwan. find
11. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of ACL, pages 311–318, Philadelphia, Pennsylvania, USA. find our card
12. Maja Popović, Adrià de Gisper, Deepa Gupta, Patrik Lambert, Hermann Ney, José Mariño, and Rafael Banchs. 2006. Morpho-syntactic information for automatic error analysis of statistical machine translation output. In Proceedings of WMT, pages 1–6, New York City, New York, USA. find our card
13. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human notation. In Proceedings of AMTA, pages 223–231, Cambridge, Massachusetts, USA. find
14. Sara Stymne and Lars Ahrenberg. 2010. Using a grammar checker for evaluation and postprocessing of statistical machine translation. In Proceedings of LREC, pages 2175–2181, Valetta, Malta. find
15. David Vilar, Jia Xu, Luis Fernando D’Haro, and Hermann Ney. 2006. Error analysis of machine translation output. In Proceedings of LREC, pages 697–702, Genoa, Italy. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/stymne-2011-blast-tool-error-analysis-machine/