The Multidimensional Quality Metrics (MQM) Framework: a new framework for translation quality assessment

What it is about

The paper tests whether MQM can be used reliably by novice raters. Twenty-nine student translations of one short French news story were rated by nine novice raters and two experts, using an MQM metric shaped after the error categories of the American Translators Association certification exam. Each rater marked errors sentence by sentence and assigned them to categories. Viability was judged on three things: practicality, measured as average time on task; validity, measured as correlation between novice and expert scores; and reliability, analysed with Many-Facet Rasch Measurement, which separates rater severity, item difficulty and the scale. The authors conclude that the framework can be viable with novice raters under the right conditions, which means clear category definitions, a fixed severity scale and a short training. The paper does not set pass thresholds, which it leaves out of scope. The study is small, a single text and nine novice raters, so the result is about the framework's use, not about a finished scale.

Ideas

• I1. Use a Rasch-style or plain agreement check to test the translator's residual labels when two people label the same residual tokens. The paper's three tests (time, correlation, facets) can be the template for a Mova gate on labels.
• I2. Use the expert-novice split as a control. Compare the translator's automatic classes with a small hand-labelled set from a careful labeller, and report agreement per class, which is where the lexical and order labels need checking.
• I3. Record the time per verse for hand labelling. It is the practicality figure the paper uses, and it decides how large a hand-checked dev set can be.
• I4. Restrict wide error lists to the failure set, the verses where the reader and encoder disagree, rather than running them on every verse.

For Mova

The code is the residual labelling in tools/translate. The paper is a method for checking whether human labels can be trusted, which the translator needs before it uses hand labels as a dev reference for the residual classes. The small size of the study means it gives a procedure, not a result to copy.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.26034/cm.jostrans.2015.343

@article{mariana2015multidimensional,
  title = {{The Multidimensional Quality Metrics (MQM) Framework: a new framework for translation quality assessment}},
  author = {Mariana, Valerie and Cox, Troy and Melby, Alan},
  journal = {The Journal of Specialised Translation (JoSTrans)},
  year = {2015},
  doi = {10.26034/cm.jostrans.2015.343},
  url = {https://doi.org/10.26034/cm.jostrans.2015.343}
}

References

32 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1.  Bachman, Lyle F. and Adrian S. Palmer (1996). Language Testing in Practice: Designing and Developing Useful Language Tests. Oxford: Oxford University Press. find
2.  Conde, Tomas (2011). "Translation Evaluation on the Surface of Texts: A Preliminary Analysis." The Journal of Specialised Translation 15, 69-86. http://jostrans.org/issue15/art_conde.php (consulted 16.11.2014) ↗
3.  Colina, Sonia (2009). "Further Evidence for a Functionalist Approach to Translation Quality Evaluation." Target: International Journal on Translation Studies 21 (2), 235-64. find
4. Doherty, Stephen (2013). “Translation Quality Models and Tools – Is There Room for Improvement?”. Globalization and Localization Association blog, July 18, http://www.gala-global.org/blog/2013/translation-quality-models-and-tools-is-there-room-for-improvement/ (consulted 16.11.2014) ↗
5.  Eckes, Thomas (2011). Introduction to Many-Facet Rasch Measurement: Analyzing and Evaluating Rater-Mediated Assessments. Frankfurt Am Main: Peter Lang. find
6.  Evans, Norman W., K. James Hartshorn, Troy L. Cox, and Teresa Martin De Jel (2014). "Measuring Written Linguistic Accuracy with Weighted Clause Ratios: A Question of Validity." Journal of Second Language Writing (24), 33-50. find
7.  Eyckmans, June, Philippe Anckaert, and Winibert Segers (2009). "The Perks of Norm-referenced Translation Evaluation." Claudia V. Angelelli and Holly E. Jacobson (eds). Testing and Assessment in Translation and Interpreting Studies: A Call for Dialogue Between Research and Practice. Amsterdam: John Benjamins, 73-93. find
8.  Kane, M. T. (2006). “Validation.” R. L. Brennan (ed.) Educational Measurement (4th edition). Westport, CT: American Council on Education and Praeger. find
9.  Koby, Geoffrey S. and Alan K. Melby (2013). "Certification and Job Task Analysis (JTA): Establishing Validity of Translator Certification Examinations." The International Journal of Translation and Interpreting Research (5)1, 174-210. find
10.  Koby, Geoffrey S., Alan K. Melby, and Arle Lommel (2013). “Mapping of MQM to ATA.” (unpublished, developed through a conference call). find
11.  Koby, Geoffrey, and Gertrud Champe (2013). "Welcome to the Real World: Professional-Level Translator Certification." The International Journal of Translation and Interpreting Research (5)1, 156-173. find
12.  Kockaert, Hendrik J. and Winibert Segers (forthcoming). “Evaluation de la traduction : la méthode PIE (Preselected Items Evaluation).” Turjuman, Journal of Translation Studies.  Le Nouvel Observateur (2009). “Le Prof “désobéisseur,” sanctionné, perd 7.000 Euros.” July 24, http://tempsreel.nouvelobs.com/societe/20090724.OBS5350/le-prof-desobeisseur-sanctionne-perd-7-000-euros.html (consulted 14.11.2014). ↗
13.  Linacre, John M. (2013) Facets computer program for many-facet Rasch measurement, version 3.71.2. www.winsteps.com (consulted 16.11.2014). find
14.  ─ (n.d.). "Misfit Diagnosis: Infit Outfit Mean-square Standardized." Measurement Software (consulted 16.11.2014) find
15. Winsteps Rasch  Linacre, John M. (2011) Personal communication (consulted 16.11.2014). find
16.  Lunz, Mary E., and John A. Stahl. (1990) “Judge Consistency and Severity Across Grading Periods.” www.sagepublications.com. (consulted 15.11.2014). find
17.  Lommel, Arle (ed.) (2014). “Multidimensional Quality Metrics (MQM) Definition.” http://www.qt21.eu/mqm-definition/definition-2014-08-19.html (consulted 16.11.2014) ↗
18.  Nord, Christiane (1997). Translating as a Purposeful Activity: Functionalist Approaches Explained. Manchester: St. Jerome. find
19.  O'Brien, Sharon (2012). "Towards a Dynamic Quality Evaluation Model for Translation." The Journal of Specialised Translation 17, 55-77. http://jostrans.org/issue17/art_obrien.php (consulted 16.11.2014). ↗
20.  Shrock, Sharon A. and William C. C. Coscarelli (2007). Criterion-referenced Test Development: Technical and Legal Guidelines for Corporate Training. San Francisco: Pfeiffer. find
21.  Snow, Tyler (forthcoming) "Establishing the Viability of the Multidimensional Quality Metrics Framework." Thesis. Brigham Young University.  The Corpus of Contemporary American English: 450 million words, 1990-present. http://corpus.byu.edu/coca/ (consulted 05.12.2014). ↗
22.  The Length 05.12.2014). of a Logit. http://www.rasch.org/rmt/rmt32b.htm ↗
23. (consulted  Vermeer, Hans J. (1987). “What does it mean to translate?”, Indian Journal of Applied Linguistics 13(2), 25-33. find
24.  Waddington, Christopher (2001). "Different Methods of Evaluating Student Translations: The Question of Validity." Meta: Translators' Journal 46(2), 311-25. find
25.  Zhu, W., C.D. Ennis and A. Chen (1998). “Many-faceted Rasch modeling expert judgment in test development.” Measurement in Physical Education and Exercise Science 2(1), 21-39. find
26. Websites  ATA, American Translators Association (2014). Framework for Standardized Error Marking http://www.atanet.org/certification/aboutexams_error.php (consulted 16.11.2014) ↗
27.  ATA, American Translators Association (2011). ATA Certification Program Rubric for Grading. http://www.atanet.org/certification/aboutexams_rubic.pdf (consulted 16.11.2014) ↗
28.  Centre for the Study of Higher Education (2002). A Comparison of Norm-referencing and Criterion-referencing Methods for Determining Student Grades in Higher Education. http://www.cshe.unimelb.edu.au/assessinglearning/05/normvcrit.html (consulted 16.11.2014) ↗
29.  Institute for Objective Measurement, The length http://www.rasch.org/rmt/rmt32b.htm (consulted 05.12.2014). of a logit. ↗
30.  MQM (2014). Multidimensional Quality Metrics. http://www.qt21.eu/mqm-definition (consulted 05.12.2014). ↗
31.  National Council of State Boards of Nursing (NCSBN). “What Logit?”https://www.ncsbn.org/What_is_a_Logit.pdf (consulted 16.11.2014) is a  QTLP (2014). Quality Translation Launch Pad. http://www.qt21.eu/launchpad/content/new-goal-quality-translation (consulted 05.12.2014). ─, definition. http://www.qt21.eu/mqm-definition/ (consulted 05.12.2014). ─, section 8. http://www.qt21.eu/mqm-definition/definition-2014-08-19.html#scoring ─, section 10. http://www.qt21.eu/mqm-definition/definition-2014-08-19.html#mappings (consulted 05.12.2014). ↗
32. ─, metric builder, http://scorecard2.gevterm.net/ (consulted 05.12.2014). ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/mariana-2015-multidimensional-quality-metrics-mqm-framework/