Adapting a WSJ-Trained Parser to Grammatically Noisy Text

What it is about

The paper makes a treebank of ungrammatical English by injecting one or two errors into Penn Treebank sentences and changing the tree only at the yield. The error types are missing word, extra word, real-word spelling, agreement and verb form. A Charniak-based parser trained on clean text loses 4.3 points of F-score on one-error sentences (Noisy00) and 8.8 points on two-error sentences (Noisiest00), on 1,921 WSJ00 sentences. Retraining on a mix of clean and noisy trees improves the noisy sets without hurting clean text. A simple classifier that compares parse probabilities between the clean and the noisy-trained parser keeps the clean results stable. The classifier picks the wrong parser for about 45% of Noisy00 sentences, but those are sentences the clean parser already handles well.

Ideas

• I1. A noise generator in Rust, as a test tool: apply the five error types to EWT test sentences with one or two edits, and measure LAS of en parse on clean and noisy versions. The drop should be close to the paper's 4.3 and 8.8 points, which tests whether the setup is sane. Use the noisy copies for measurement only, not for training.
• I2. Two-parser switch: keep the current perceptron and a second one trained on EWT train plus noisy copies, and pick the output with the higher beam score. Measure LAS on clean EWT test (it must not drop) and on the noisy copies (it must rise). A negative control with a random switch must show no gain, or the gate is not doing its job.
• I3. Error-type table: report LAS by error type in en parse-eval. The paper finds that agreement errors are already ignored by the parser, and that real-word spelling errors gain most from retraining. Measure the same breakdown on Mova's noisy copies before changing the model.

For Mova

The chapter is 37. Register and variation, for learner and noisy English, with 21-clause-structure.md for the parse side. The code is en/src/parse.rs (beam 8, early update) and the en parse-eval command. The paper's data are WSJ-derived and its noise rules come from Foster's empirical study, so the noisy copies are test only and the rules are described, not copied. Every Mova figure must be on EWT test, with decisions on the two-fold check.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/1557690.1557753

@inproceedings{foster2008adapting,
  title = {{Adapting a WSJ-Trained Parser to Grammatically Noisy Text}},
  author = {Foster, Jennifer and Wagner, Joachim and Genabith, Josef van},
  booktitle = {Proceedings of ACL-08: HLT, Short Papers},
  year = {2008},
  doi = {10.3115/1557690.1557753},
  url = {https://doi.org/10.3115/1557690.1557753}
}

References

10 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eugene Charniak and Mark Johnson. 2005. Course-to-fine nbest-parsing and maxent discriminative reranking. In Proceedings of ACL-2005. find
2. Jennifer Foster. 2005. Good Reasons for Noting Bad Grammar: Empirical Investigations into the Parsing of Ungrammatical Written English. Ph.D. thesis, University of Dublin, Trinity College. find
3. Jennifer Foster. 2007. Treebanks gone bad: Parser evaluation and retraining using a treebank of ungrammatical sentences. IJDAR, 10(3-4), December. find
4. Frederik Fouvry. 2003. Robust Processing for Constraint-based Grammar Formalisms. Ph.D. thesis, University of Essex. find
5. Kong Joo Lee, Cheol Jung Kweon, Jungyun Seo, and Gil Chang Kim. 1995. A robust parser based on syntactic information. In Proceedings of EACL-1995. find
6. Daisuke Okanohara and Jun’ichi Tsujii. 2007. A discriminative language model with pseudo-negative examples. In Proceedings of ACL-2007. find
7. Carolyn Penstein Rosé and Alon Lavie. 1997. An efficient distribution of labor in a two stage robust interpretation process. In Proceedings of EMNLP-1997. find
8. David Schneider and Kathleen McCoy. 1998. Recognizing syntactic errors in the writing of second language learners. In Proceedings of ACL/COLING-1998. find
9. Noah A. Smith and Jason Eisner. 2005. Contrastive Estimation: Training Log-Linear Models on Unlabeled Data. In Proceedings of ACL-2005. find
10. Joachim Wagner, Jennifer Foster, and Josef van Genabith. 2007. A comparative evaluation of deep and shallow approaches to the automatic detection of common grammatical errors. In Proceedings of EMNLP-CoNLL-2007. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2008-foster-adapting-wsj-trained-parser-grammatically/