Multidimensional quality metrics: a flexible system for assessing translation quality

What it is about

The paper describes Multidimensional Quality Metrics (MQM), a framework from the QTLaunchPad project for building translation quality metrics out of a shared catalogue of issue types. The catalogue had about 108 issue types in 2013, arranged in a hierarchy and drawn from more than twenty existing metrics and checking tools, with BLEU and error-analysis tools included as sources. The issue types sit under five branches: fluency, accuracy, verity (conformity with the real world), design and internationalisation. A concrete metric chooses a small subset, usually no more than about twelve types, guided by twelve dimensions from the translation-specification standard, such as audience, purpose and register. The scoring rule starts from 100 and subtracts weighted penalties per word, with severity weights of 1 for minor, 5 for major and 10 for critical, taken from the LISA QA model. Penalties for source-side errors are added back, so a translator who repairs the source is not punished. The paper argues that one fixed metric cannot serve all translation tasks, and that the framework is free to implement.

Ideas

I1. Map the translator's residual classes (lexical, construction, order, function words, punctuation and case, morphology) to MQM issue types in a table under docs/. The labels then match other metrics, and the translator's residual becomes portable.
I2. Give each residual token a severity: minor for punctuation and case, major for lexical choice and missing content, critical for a changed meaning. Report a weighted penalty per verse next to the bit count, and keep the bit count as the gate.
I3. Credit source repairs. A reader that corrects a source-side problem, for example a wrong tree in the Hebrew reader, should be counted apart from output loss, as the paper's source credit intends.
I4. Select the measure per author or style class, using a small set of dimensions, as the paper recommends. This suits the author deltas and the style-class deltas of open problem 8.

For Mova

The code is the residual report in tools/translate and the label table in docs/. The paper gives the translator a shared vocabulary for its errors and a way to weight them, so that a verse with one wrong word and a verse with a changed meaning are not counted alike. It does not give a better gate than the residual in bits, and should not replace it.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{lommel2013multidimensional,
  title = {{Multidimensional quality metrics: a flexible system for assessing translation quality}},
  author = {Lommel, Arle Richard and Burchardt, Aljoscha and Uszkoreit, Hans},
  booktitle = {Proceedings of Translating and the Computer 35},
  year = {2013},
  url = {https://aclanthology.org/2013.tc-1.6/}
}

References

6 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Garvin, David. 1984. “What does ‘product quality’ really mean?” Sloan Management Review, fall, pp. 25–45. find
ISO. 2012. ISO/TS 11669:2012: Translation projects -- General guidance. Geneva: ISO. find
LISA. 2006. “LISA QA Model 3.1: Assisting the localization development, production and quality control processes for global product distribution” (press release). Romainmôtier: LISA. find
Lommel, Arle R. 2013. “Measuring translation quality in today’s automated lifecycle”. Presentation at tcworld 2013, Wiesbaden, Germany, November 8. Available from http://tagungen.tekom.de/fileadmin/tx_doccon/slides/453_A_Unified_Model_for_Document_and_Translation_Quality_Assurance.pdf (accessed November 14, 2013). ↗
QTLaunchPad. 2013a. Multidimensional Quality Metric quality issue types, version 2.5.5. https://docs.google.com/document/d/1E8IR1-8bR_M7VouHQhogUPpP2htpprwMMx1Mk9KPBTI/pub (accessed November 14, 2013). —. 2013b. Error analysis in near miss translations. http://www.qt21.eu/launchpad/deliverable/error-analysis-near-miss-translations (accessed November 14, 2013). ↗
SAE International. 2001. SAE J2450: Translation Quality Metric. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lommel-2013-multidimensional-quality-metrics-flexible-system/