Dimensions of Deep Grammar Validation

What it is about

The paper sets out a method for validating a large deep grammar at each release, using the English Resource Grammar as the example. It names ten dimensions. Four are the main tensions: robustness, precision, efficiency and stability of the mappings on test suites. The others cover several processing systems, bi-directional use, grammar configurations, disambiguation, applications, and the external interface specifications, such as the list of semantic predicates. The tools are growing test suites, both hand-built and corpus-based, and a profiler that stores competence and performance data for each item. Gold treebanks from the suites measure each new revision. The paper gives no numbers; it describes one release cycle.

Ideas

• I1. en/src/udreg.rs: extend the release gate with one line per dimension: coverage on EWT test, LAS, seconds, the number of changed analyses since the last release, and the version of the tag inventory. The gate must turn red on a seeded regression.
• I2. Stability metric: count how many EWT dev trees change between two en builds, and print it beside LAS in en parse-eval.
• I3. Configuration check: run en ud-check with and without en/src/ctx.rs and report the difference in LAS on EWT dev, as a configuration dimension.

For Mova

The code is en/src/udreg.rs and the commands are en ud-check and en parse-eval; the chapter is 37. Register and variation. The paper has no numbers and describes an HPSG grammar. Borrow the checklist only, and measure on EWT.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{flickinger2005dimensions,
  title = {{Dimensions of Deep Grammar Validation}},
  author = {Flickinger, Dan},
  booktitle = {Proceedings of the Australasian Language Technology Workshop 2005},
  year = {2005},
  url = {https://aclanthology.org/U05-1001/}
}

References

15 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Valerie Barr and Judith Klavans. 2001. Verification and validation of language processing systems: Is it evaluation? Proceedings of the Workshop on Evaluation Methodologies for Language and Dialogue Systems, ACL2001. find
2. Valerie Barr and Ellen Siefring. 2004. Verification of lexicalized tree adjoining grammars. Online Proceedings of the Seventh International Workshop on TAG and Related Formalisms. find
3. Norbert Broeker. 2000. The use of instrumentation in grammar engineering. Proceedings of COLING 2000. find
4. Ulrich Callmeier. 2000. PET — A platform for experimentation with efficient HPSG processing techniques. Natural Language Engineering, 6 (1) (Special Issue on Efficient Processing with HPSG):99 – 108. find
5. Ann Copestake, Dan Flickinger, Ivan A. Sag, and Carl Pollard. 2006. Minimal Recursion Semantics. An introduction. Research on Language and Computation. (to appear). find
6. Ann Copestake. 2002. Implementing Typed Feature Structure Grammars. CSLI Publications, Stanford, CA. find
7. Marc Dymetman and Pierre Isabelle. 1988. Reversible logic grammars for machine translation. Proceedings of the 2nd International Conference on Theoretical and Methodological Issues in the Machine Translation of Natural Languages. find
8. EAGLES. 1996. Evaluation of Natural Language Processing Systems. Final Report EAG-EWG-PR.2. EU Report. find
9. Dan Flickinger, Jan Tore Lønning, Helge Dyvik, Stephan Oepen, and Francis Bond. 2005. SEM-I rational MT: Enriching deep grammars with a semantic interface for scalable machine translation. In Machine Translation Summit X, Phuket. (to appear). find our card
10. Dan Flickinger. 2000. On building a more efficient grammar by exploiting types. Natural Language Engineering, 6 (1):15 – 28. find our card
11. Lynette Hirschman and Henry Thompson. 1998. Overview of evaluation in speech and natural language processing. In G. Varile and A. Zampolli, editors, Survey of the State of the Art in Human Language Technology. Cambridge University Press, New York. find
12. Jan Tore Lønning, Stephan Oepen, Dorothee Beermann, Lars Hellan, John Carroll, Helge Dyvik, Dan Flickinger, Janne Bondi Johannessen, Paul Meurer, Torbjørn Nordgård, Victoria Rosén, and Erik Velldal. 2004. LO-GON. A Norwegian MT effort. In Proceedings of the Workshop in Recent Advances in Scandinav… find
13. Stephan Oepen and Dan Flickinger. 1998. Towards systematic grammar profiling. Test suite technology ten years after. Journal of Computer Speech and Language, 12 (4):411 – 436. find
14. Stephan Oepen, Daniel Flickinger, Kristina Toutanova, and Christopher D. Manning. 2004. LinGO Redwoods. A rich and dynamic treebank for HPSG. Journal of Language and Computation. find our card
15. Karen Sparck Jones and Julia Galliers. 1998. Evaluating Natural Language Processing Systems. Springer-Verlag, Berlin. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2005-flickinger-dimensions-deep-grammar-validation/