Parsing Ungrammatical Input: An Evaluation Procedure
What it is about
The question is how well a parser handles an ungrammatical sentence. The method needs two parallel corpora: ungrammatical sentences and their corrected versions. Each ungrammatical sentence is parsed, and its parse is scored against the parse of each correction produced by the same parser; the best score counts. The source is a 20,000-word corpus of ungrammatical English from newspapers, email, academic papers and websites, where 20% of sentences have more than one correction. Charniak's 2000 parser reaches 91% precision and 91% recall, with 32% complete matches against a correction and 16% of sentences below 75%. Incorrect word forms make up 44% of the corpus, missing words 28% and extra words 15%.
Ideas
- I1.
en parse-eval: for a sentence with several corrections, score it against the best one and print the chosen correction. Measure on a small set of sentences that we write and correct ourselves, with two or more corrections for some of them. - I2. Error-type breakdown: tag each test sentence as word form, missing word, extra word, composite or multiple errors, then report LAS and complete-match rate per type. Measure the complete-match rate per type on our own set.
- I3. Extra-word rule in
en/src/expert.rs: flag atothat follows a verb and does not head an infinitive, with the reason written byen explain. Measure the precision of the flag and the LAS change on the own set. - I4. Negative control: score the same parser on a clean set and on its corrupted copy. If the drop is zero, the gate is broken and the test must be fixed before any claim.
For Mova
The chapters are 22. Phrases: noun, adjective, adverb and prepositional phrases and 37-register-and-variation.md. The code is en/src/parse.rs, en/src/expert.rs, en/src/explain.rs and the en parse-eval command. The paper's corpus is not in Mova's data, and its licence is NC, so it cannot be built into anything. Any test set must be written by us and used for measurement only, and the paper's figures belong to a different parser.
Cite
@inproceedings{foster2004parsing,
title = {{Parsing Ungrammatical Input: An Evaluation Procedure}},
author = {Foster, Jennifer},
booktitle = {Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04)},
year = {2004},
doi = {10.63317/3vjm8zsqfw8p},
url = {https://doi.org/10.63317/3vjm8zsqfw8p}
}References
- Black, E., S. Abney, D. Flickinger, C. Gdaniec, R. Grishman, P. Harrison, D. Hindle, R. Ingria, F. Jelinek, J.Klavans, M. Liberman, M. Marcus, S. Roukos, B. Santorini, and T. Strzalkowski, 1991. A procedure for quantitatively comparing the syntactic coverage of english grammars. In Proceedings of th… find
- Charniak, 2000. A maximum-entropy-inspired parser. In Proceedings of the NAACL-2000. find
- Collins, Michael, 1997. Three generative lexicalized models for statistical parsing. In Proceedings of the 35th ACL. Madrid. find
- Copestake, Ann, 2002. Implementing Typed Feature Structure Grammars. CSLI Lecture Notes. Cambridge: Cambridge University Press. find
- Foster, Jennifer and Carl Vogel, 2004. Good reasons for noting bad grammar: Constructing a corpus of ungrammatical language. In Stephan Kepser and Marga Reis (eds.), Pre-Proceedings of the International Conference on Linguistic Evidence: Empirical, Theoretical and Computational Perspectives. Tübing… find
- Marcus, Mitchell P., Beatrice Santorini, and Mary Ann Marcinkiewicz, 1993. Building a large annotated corpus of english: the penn treebank. Computational Linguistics, 19(2):313–330. find our card
- Musillo, Gabriele and Khalil Sima’an, 2002. Towards comparing parsers from different linguistic frameworks: An information theoretic approach. In Proceedings of the ”Beyond Parseval - Towards Improved Evaluation Measures for Parsing Systems” Workshop, 3rd LREC. Las Palmas, Gran Canaria. find
This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2004-foster-parsing-ungrammatical-input-evaluation-procedure/