Using Test Suites in Evaluation of Machine Translation Systems

What it is about

An early proposal to evaluate MT by test suites rather than by ratings or error counts. The paper argues that no single evaluation fits all buyers, so it proposes two suites: one of sentences drawn from a bilingual corpus of the intended use, and one of translational mismatches and ambiguities. Each structure gets two parallel lists, A and B. The developer sees the A list and corrects dictionary coding; the B list is never used for fixes. Its results after each round show the side effects of the fixes. Cases that remain unsolved are classified by the time the maker estimates a change would need. The paper gives a procedure and no measured results.

Ideas

• I1. Use the A and B split for the lexicon work (open problem 2). Fit the merges on the A verses only, and report the B-verse residual after each change as the side-effect check. A merge that raises the B residual is noise.
• I2. Keep the paper's rule that every test item has a pre-agreed acceptable output. For Mova, the verse's reference translation by another author can serve as that target, so no new translations are written.
• I3. Sort the remaining failures by estimated fix cost (a lexicon entry, an encoder rule, a reader change). That gives the order of work across the eight open problems.

For Mova

The places are tools/translate (meta.rs for the lexicon, profile.rs for the encoder). The A and B lists must be fixed before any merge is tried, and the held-out rule still applies.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{king1990using,
  title = {{Using Test Suites in Evaluation of Machine Translation Systems}},
  author = {King, Margaret and Falkedal, Kirsten},
  booktitle = {COLING 1990 Volume 2: Papers presented to the 13th International Conference on Computational Linguistics},
  year = {1990},
  url = {https://aclanthology.org/C90-2037/}
}

References

7 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

2. Flickinger, D. et al. HP-NL Test Suite, (as of June 30, 1987). find
3. King, Margaret. A Practical Guide to the Evaluation o/ Machine Translation Systems. ISSCO. Intermediate Report to Suissetra, Feb., 1989. Revised version in preparation. find
4. Lehrberger, J. and Bourbeau, L. Machine Translation: Linguistic Characteristics o / M T Systems and General Methodology of Evaluation. John Benjamin. 1988. find
5. Slocum, J. and Bennet, W. An Evaluation o/ METAL: the LRC Machine Translation System. Second Conference of the European Chapter of the ACL, 1985. find
6. Van Slype, G. Critical Study o/ Methods for Evaluating the Quality o/ Machine Translation Systems. Bureau Marcel Van Dijk, Bruxelles and CCE, 1979. find
7. Vasconcellos, M. (ed.) Technology Translation Strategy. SUNY, 1988. find
8. Wilks, Y. and LATSEC Inc. Comparative Translation Quality Analysis, Final Report, Latsec Inc., 1979. as find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/king-1990-test-suites-evaluation-machine-translation/