Mova0.5 pre-release⌕
☰
home / docs / papers / Machine translation / Evaluation Metrics for Knowledge-Based Machine Translation

Evaluation Metrics for Knowledge-Based Machine Translation

What it is about

The paper argues that a machine translation system should be judged both by global black-box scores and by a component-wise causal analysis, so that each error can be traced to the module that made it. For a knowledge-based system with an interlingua, it defines coverage (did the analyser or generator produce an output at all) and correctness (was that output right) for each side, and the translation correctness is the product of the four. Human graders place each output in one of three bins (correct, acceptable, incorrect) and attach error codes that name the module and the error type, so that the list of failures can be sorted to the people who maintain each part. A weighted variant gives partial credit by the share of correctly translated words. The authors apply it to an English-to-French heavy-equipment application and report translation correctness for 17 randomly chosen texts in a range that is still short of the final system.

Ideas

For Mova

The places are tools/translate (measure.rs) for the per-layer report and tools/roundtrip for the error classes it already uses. The paper describes a rule-based knowledge system and its human grading is not for Mova; the decomposition of the score by layer and the sorted error codes are the parts that carry over, and they match the residual measure Mova already keeps in bits.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{mitamura1994evaluation,
  title = {{Evaluation Metrics for Knowledge-Based Machine Translation}},
  author = {Mitamura, Teruko and Carbonell, Jaime G.},
  booktitle = {COLING 1994 Volume 1: The 15th International Conference on Computational Linguistics},
  year = {1994},
  url = {https://aclanthology.org/C94-1013/}
}

References

12 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. [11 Carbonell, J., Mitamura, T., and E. Nyberl; (1993). "Evahmting KBMT in tht I,arge," Japan-US Workshop on Machine-Aided Translaliov, Nov. 22-24, Washington, D.C. {2] Carbonell, J. and Y. Wilks (1991). "Machint Transhl-tion: An In-Depth Tntorial," 29th Annual Meeting of the Association for Compntational Linguistics, Univero sity of CaliR)rnia, Be,'keley, CA, June 18-21, [371 Ooo{Imml and Nirtnburg, eds. (1991). A Case Study in Knowledge-Based Machine Translation, San Mateo, CA: Morgan Kaufmann. find
  2. [4] Isalmra, Sin-nou, Yamabana, Moriguchi and Nonmra, (1993). "JEIDA's l'roposed Method for l'valuating Machine "['ranslalion (Translation Quality)," l',oceedmgs o/ SIGNLP 93-NL-96, July. find
  3. Dlcctlonic h~dustry 1)evclolmlent Association, A Japrmese View of Machine "l'ran.','httion tin l,ight of tile Considerations and Recommendations Reported by AL . PAC, U.S.A., JEIDA Machine Translation System P,t-search Commitlec, Tokyo. [6] King, M. (1993). "Panel on Evaluation: MT Summit IV. Introduction." l'roceedit~gs of MT Summit IV, July 20-22, Kobe, Japan. 151 J[tp{ll/ 17] Mitamura, "i,., E. Nyberg "and J. Cmbonell (1991). "An Efficient lntcrlinl.,,ua Translation System for Multil ivlglml Docunlent Production," Proceedings o/'Machine 7?anslatim) Summit III, Washinglon, DC, July 2-d. find
  4. [81 Nagao, M. (1984) . "A Ftamework of a Mechanical Transl:ltion l}ctween Japanese and Enp, lish I)y Analogy Princil}lC," Artificial and Iluman Intelligence, Elithorn, A. and Bauerii, P,. (eds.), Elsevier Science Publishers, B. V. 1984. find
  5. [91 Nagao, M. (1985) . "Evaluation of tilt Quality of Machint-Transhtttd Sentences and the Control of Language," .lournal of h!formatioa Proeessi,]g Society of .lapan, 26(1 (}): 1197-1202. find
  6. [l{}l Nakaiwa, Morimoto, Matsndaira, Na,'ita anti Nomura, (1993). "J F.I DA's Pml}osed Metho{I for Evahmtinl; Machine Traushltion (1)evClOl}tr's Guideliiles)," I'roceedings o/ SIGNI.I' 9.}'-NL-96, J lily. find
  7. [111 Nomura, If. (1093). "l.~v.'lhmtion Method of Ma{:hitm '['r[llIsltttiOll: t't'Olll tile Viewl}oit,t of Natural hanguape Processing," I'roceedings of MT Summit IV, Jtlly 20-22, Kobt, J al}an. [12] Nyberg, 12. and T. Mitamura (1992). "Tim KANT Sys-- tom: I:tlS[, Accurtlte, lligh-Quality TratlsIatioll in Ptac-Ileal Domains," Proceedings of COEING 1992, Nante.,;, France, July. find
  8. [131 Rinscht, Adriane (1993). "Towards a MT Evaluation Methodolop, y," I'roceedings of the t,',fth International Con/?rem:e o,,* Theoretical and Methodological lssttes iH Machine 7)'anslatiott, July 14-16, Kyolo, J.2tl}tlll. find
  9. (ld] Rolling, I... (1993). "P:mel Contribution on MT Evahmtion," I'roceedMgs o/MT Summil IV, July 20-22, Kobt, Japan. find
  10. [15] Takzly,:una, ltoh, Yagisawa, Mogi and Nomura (1993), "JHDA's Proposed Method for l:.vahmtiug Machine Transhltio, (End User System Selection)," l'r~)ceedings of SIGNI,I' 93-NL-96, July. find
  11. [16J VanSlype, G. (1979). "l:.valuation of the 1978 Version of tile SYSTRAN F.uglish-French Automatic System or the Coml]tissioll of the I!tu{}l}tan COmmuvlitics," 7'he Incorporated Linguist 18:86-89. 117 ] Vasconcellos, M. (1993). "P:mtl Discussion: Eval u,'ltion Method of Machine Translati{m," Proceedings of MT Summit IV, July 2{}-22, Kobe, Japan. find
  12. [18l Wilks, Y. (1991). "SYSTP, AN: It Obviously Works, but Ilow Much Can it be hnprovtd?," Teclnfical Rtt}ott MCCS-.91-215, (2}reputing Research 1 ,aborat{n'y, New Mexico Statt University, l.as Cruces. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/nyberg-1994-evaluation-metrics-knowledge-based-machine/