Holistic Regression Testing for High-Quality MT: Some Methodological and Technological Reflections

What it is about

The LOGON project, a Norwegian-English MT system for tourism texts, uses regression testing as its main quality measure. It combines hand-built test suites, a development corpus of real text and tools that profile each run. The closed-class suite ('base') has close to 300 items over about 15 top-level phenomena, 66 in total. A 110-item sample of tourism text (average 13 words, up to 30) is the development set. The profiles report coverage and ambiguity for analysis, transfer and generation per phenomenon. The authors argue that regression testing needs annotated inputs and a database of results, not log files, and that stable results across releases keep the gold data cheap to update.

Ideas

• I1. Build a regression suite under treebanks/en/: a hand-written closed-class set of about 300 items, each tagged with its phenomenon. Run en parse-eval on it at each release and store the results as a table keyed by item and release.
• I2. Per-phenomenon profile: group EWT test errors by the phenomenon tags of the suite and report LAS per group.
• I3. Stability gate: count the analyses that change between two en builds on the suite and on the 110-item sample. Every change must be listed. Negative control: a deliberately broken rule must turn the gate red.

For Mova

The release gate is en/src/udreg.rs, with en ud-check and en folds for decisions. The chapter for the closed-class items is 19. The lexicon: vocabulary, multiword expressions and register. The LOGON data are not in our repo and are MT-specific; the suite has to be written by us.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{oepen2005holistic,
  title = {{Holistic Regression Testing for High-Quality MT: Some Methodological and Technological Reflections}},
  author = {Oepen, Stephan and Dyvik, Helge and Flickinger, Dan and Lønning, Jan Tore and Meurer, Paul and Rosén, Victoria},
  booktitle = {Proceedings of the 10th EAMT Conference: Practical applications of machine translation},
  year = {2005},
  url = {https://aclanthology.org/2005.eamt-1.27/}
}

References

15 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Carroll, J., Copestake, A., Flickinger, D., & Poznanski, V. (1999). An efficient chart generator for (semi-)lexicalist grammars. In Proceedings of the 7th European Workshop on Natural Language Generation (pp. 86 – 95). Toulouse, France. find
2. Copestake, A. (2002). Implementing typed feature structure grammars. Stanford, CA: CSLI Publications. find
3. Copestake, A., Flickinger, D., Sag, I. A., & Pollard, C. (2003). Minimal Recursion Semantics. An introduction. In preparation, CSLI Stanford, Stanford, CA. find
4. Dyvik, H. (1999). The universality of f-structure. Discovery or stipulation? The case of modals. In Proceedings of the 4th International Lexical Functional Grammar Conference. Manchester, UK. find
5. Flickinger, D. (2000). On building a more efficient grammar by exploiting types. Natural Language Engineering, 6 (1) (Special Issue on Efficient Processing with HPSG), 15 – 28. find our card
6. Flickinger, D., Nerbonne, J., Sag, I. A., & Wasow, T. (1987). Toward evaluation of NLP systems (Technical Report). Hewlett-Packard Laboratories. (Distributed at the 24th Annual Meeting of the Association for Computational Linguistics) find
7. Geist, A., Bequelin, A., Dongarra, J., Manchek, W. J. R., & Sunderam, V. (Eds.). (1994). PVM. Parallel Virtual Machine. A users’ guide and tutorial for networked parallel computing. Cambridge, MA: The MIT Press. find
8. Oepen, S., & Callmeier, U. (2004). Measure for measure. Towards increased component comparability and exchange. In H. Bunt, J. Carroll, & G. Satta (Eds.), New developments in parsing technology. Dordrecht, The Netherlands: Kluwer Academic Publishers. find
9. Oepen, S., Dyvik, H., Lønning, J. T., Velldal, E., Beermann, D., Carroll, J., Flickinger, D., Hellan, L., Johannessen, J. B., Meurer, P., Nordgård, T., & Rosén, V. (2004). Som å kapp-ete med trollet? Towards MRS-based Norwegian – English Machine Translation. In Proceedings of the 10th Internation… find
10. Oepen, S., & Flickinger, D. P. (1998). Towards systematic grammar profiling. Test suite technology ten years after. Journal of Computer Speech and Language, 12 (4) (Special Issue on Evaluation), 411 – 436. find
11. Oepen, S., Netter, K., & Klein, J. (1997). tsnlp — Test Suites for Natural Language Processing. In J. Nerbonne (Ed.), Linguistic Databases (pp. 13 – 36). Stanford, CA: CSLI Publications. find
12. Oepen, S., Toutanova, K., Shieber, S., Manning, C., Flickinger, D., & Brants, T. (2002). The LinGO Redwoods treebank. Motivation and preliminary applications. In Proceedings of the 19th International Conference on Computational Linguistics. Taipei, Taiwan. find our card
13. Sågvall Hein, A., Forsbom, E., Weijnitz, P., Gustavii, E., & Tiedemann, J. (2003). MATS. A glass box machine translation system. In Proceedings of the 9th Machine Translation Summit (pp. 24 – ). New Orleans, LO. find
14. Wahlster, W. (Ed.). (2000). Verbmobil. Foundations of speech-to-speech translation. Berlin, Germany: Springer. find
15. Wilson, P. R., Johnstone, M. S., Neely, M., & Boles, D. (1995). Dynamic storage allocation: A survey and critical review. In Proceedings of the International Workshop on Memory Management. Kinross, Scotland: Springer. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2005-oepen-holistic-regression-testing-high-quality/