Reranking and Self-Training for Parser Adaptation

What it is about

Parsers trained on the Wall Street Journal lose much accuracy on other genres: the standard parser scores 89.7% on WSJ but only 82.9% on the Brown corpus. The authors show two techniques that raise Brown accuracy without any labeled Brown data: a reranker over the 50-best parses, which lifts Brown to 85.2%, and self-training on unlabeled out-of-domain text (NANC), where the parser's own reranked parses become new training data, which lifts it to 87.8%. That is a 28% error reduction, and it beats a parser and reranker trained on labeled Brown data (88.4%). The gains come from using unlabeled text from a different domain together with the reranker.

Ideas

• I1. Self-training on the school corpus: parse the 1.17M-word school corpus with the en beam-8 parser, keep the reranked 1-best trees, retrain the perceptron and measure dev LAS on EWT and ESLSpok against the current model (run twice for determinism).
• I2. Reranker for domain shift: add a 50-best reranker (the Charniak and Johnson idea) to the en parser and measure LAS on the ATIS queries (current 63.0) and on GENTLE (current 67.9); the paper's gain from 82.9 to 85.2 on Brown is the size to expect.
• I3. Adaptation check for annot silver: before a silver corpus is used for training, parse it with the current model and keep only the trees that a reranker prefers, and measure how many silver trees change under the filter.

For Mova

The self-training loop belongs to the training commands in en and the reranker would sit beside en/src/parse.rs. The silver corpus is the school corpus under corpus/. The chapter that the ideas touch is 08. National varieties and 37. Register and variation. Measure first: EWT dev LAS after one self-training round on the school corpus, compared with the current model.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/1220175.1220218

@inproceedings{mcclosky2006reranking,
  title = {{Reranking and Self-Training for Parser Adaptation}},
  author = {McClosky, David and Charniak, Eugene and Johnson, Mark},
  booktitle = {Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics},
  year = {2006},
  doi = {10.3115/1220175.1220218},
  url = {https://doi.org/10.3115/1220175.1220218}
}

References

16 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Michiel Bacchiani, Michael Riley, Brian Roark, and Richard Sproat. 2006. MAP adaptation of stochastic grammars. Computer Speech and Language, 20(1):41–68. find
2. Eugene Charniak and Mark Johnson. 2005. Coarse-to-fine n-best parsing and MaxEnt discriminative reranking. In Proc. of the 2005 Meeting of the Assoc. for Computational Linguistics (ACL), pages 173–180. find our card
3. Andrew B. Clegg and Adrian Shepherd. 2005. Evaluating and integrating treebank parsers on a biomedical corpus. In Proceedings of the ACL Workshop on Software. find
4. Paul R. Cohen. 1995. Empirical Methods for Artificial Intelligence. The MIT Press, Cambridge, Massachusetts. find
5. B ROWN / WSJ 0.030 (0) 0.004 (0.1) B ROWN /B ROWN 0.031 (0) 0.006 (0.025) 0.002 (0.27) find
6. Michael Collins. 2000. Discriminative reranking for natural language parsing. In Machine Learning: Proceedings of the Seventeenth International Conference (ICML 2000), pages 175–182, Stanford, California. find our card
7. W. Nelson Francis and Henry Kučera. 1979. Manual of Information to accompany a Standard Corpus of Present-day Edited American English, for use with Digital Computers. Brown University, Providence, Rhode Island. find
8. Daniel Gildea. 2001. Corpus variation and parser performance. In Empirical Methods in Natural Language Processing (EMNLP), pages 167–202. find our card
9. David Graff. 1995. North American News Text Corpus. Linguistic Data Consortium. LDC95T21. find
10. Rebecca Hwa. 1999. Supervised grammar induction using training data with limited constituent information. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 72–80, University of Maryland. find
11. Matthew Lease and Eugene Charniak. 2005. Parsing biomedical literature. In Second International Joint Conference on Natural Language Processing (IJC-NLP’05). find
12. Michell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Comp. Linguistics, 19(2):313–330. find our card
13. David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Proceedings of HLT-NAACL 2006. find our card
14. Adwait Ratnaparkhi. 1999. Learning to parse natural language with maximum entropy models. Machine Learning, 34(1-3):151–175. find
15. Satoshi Sekine. 1997. The domain dependence of parsing. In Proc. Applied Natural Language Processing (ANLP), pages 96–102. find
16. Mark Steedman, Miles Osborne, Anoop Sarkar, Stephen Clark, Rebecca Hwa, Julia Hockenmaier, Paul Ruhlen, Steven Baker, and Jeremiah Crim. 2003. Bootstrapping statistical parsers from small datasets. In Proc. of European ACL (EACL), pages 331–338. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2006-mcclosky-reranking-self-training-parser-adaptation/