Corpus Variation and Parser Performance

What it is about

The paper asks how much a statistical parser depends on the corpus it is trained on. It uses Collins' Model 1, a lexicalized generative parser, trained and tested on the Wall Street Journal (39,832 training sentences) and on the Brown corpus (about 21,800 training sentences, with every tenth sentence held out). A model trained on WSJ loses several points on Brown (recall 80.3 against 86.1 on WSJ), while a Brown-trained model does better on Brown than the WSJ model does. Removing the lexical-bigram parameters costs under 0.5 points on WSJ and leaves the cross-corpus results unchanged, so the bigrams are corpus-specific. Pruning the bigrams by a relative-entropy criterion removes 43% of the 736,000 parameters with a small loss.

Ideas

• I1. Lexical-feature ablation across banks: train en on EWT only, then on EWT plus ESLSpok, and test on ATIS (63.0 now) and GUM (76.85, test only) with and without lexical-identity features in en/src/parse.rs; the paper predicts that lexical pairs add little across corpora.
• I2. Pruning curve as a size measure: prune lexical features by a count-times-divergence score to a fixed budget and report EWT test LAS against the number of kept features, for the shipped model size.
• I3. Corpus-specific feature listing: after training, list the ten most weighted lexical feature pairs per bank, sorted deterministically with BTreeMap, to see which features are domain-specific before the model is shipped.

For Mova

The code is en/src/parse.rs (features) and en/src/ptag.rs (its lexicon). The chapters are 19. The lexicon: vocabulary, multiword expressions and register and 37-register-and-variation.md. Measure first: the LAS drop on ATIS and GUM when lexical features are removed, since the paper shows that matched data beats more unmatched data.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{gildea2001corpus,
  title = {{Corpus Variation and Parser Performance}},
  author = {Gildea, Daniel},
  booktitle = {Proceedings of the 2001 Conference on Empirical Methods in Natural Language Processing},
  year = {2001},
  url = {https://aclanthology.org/W01-0521/}
}

References

18 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Douglas Biber. 1993. Using register-diversi ed corpora for general language studies. Computational Linguistics , 19(2):219{241, June. find
2. Eugene Charniak. 1997. Statistical parsing with a context-free grammar and word statistics. In AAAI97 , Brown University, Providence, Rhode Island, August. find
3. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proceedings of the 1st Annual Meeting of the North American Chapter of the ACL (NAACL) , Seattle, Washington. find
4. Michael Collins, Jan Hajic, Lance Ramshaw, and Christoph Tillmann. 1999. A statistical parser for czech. In Proceedings of the 37th Annual Meeting of the ACL , College Park, Maryland. find
5. Michael Collins. 1996. A new statistical parser based on bigram lexical dependencies. In Proceedings of the 34th Annual Meeting of the ACL . find
6. Michael Collins. 1997. Three generative, lexicalised models for statistical parsing. In Proceedings of the 35th Annual Meeting of the ACL . find
7. Michael Collins. 1999. Head-Driven Statistical Models for Natural Language Parsing . Ph.D. thesis, University of Pennsylvania, Philadelphia. find our card
8. Michael Collins. 2000. Discriminative reranking for natural language parsing. In Proceedings of the ICML . find our card
9. Rebecca Hwa. 1999. Supervised grammar induction using training data with limited constituent information. In Proceedings of the 37th Annual Meeting of the ACL , College Park, Maryland. find
10. David Magerman. 1995. Statistical decision-tree models for parsing. In Proceedings of the 33rd Annual Meeting of the ACL . find
11. Ivan A. Melcuk. 1988. Dependency Syntax: Theory and Practice . State University of New York Press. find
12. Adwait Ratnaparkhi. 1997. A linear observed time statistical parser based on maximum entropy models. In Proceedings of the Second Conference on Empirical Methods in Natural Language Processing . find
13. Douglas Roland and Daniel Jurafsky. 1998. How verb subcategorization frequencies are a ected by corpus choice. In Proceedings of COLING/ACL , pages 1122{1128. find
14. Douglas Roland, Daniel Jurafsky, Lise Menn, Susanne Gahl, Elizabeth Elder, and Chris Riddoch. 2000. Verb subcategorization frequency di erences between business-news and balanced corpora: the role of verb sense. In Proceedings of the Association for Computational Linguistics (ACL-2000) Workshop on Comparing Corpora . find
15. Kristie Seymore and Roni Rosenfeld. 1996. Scalable backo language models. In ICSLP-96 , volume 1, pages 232{235, Philadelphia. find
16. Daniel Sleator and Davy Temperley. 1993. Parsing english with a link grammar. In Third International Workshop on Parsing Technologies , August. find our card
17. A. Stolcke, C. Chelba, D. Engle, V. Jimenez, L. Mangu, H. Printz, E. Ristad, R. Rosenfeld, D. Wu, F. Jelinek, and S. Khudanpur. 1996. Dependency language modeling. Summer Workshop Final Report 24, Center for Language and Speech Processing, Johns Hopkins University, Baltimore, April. find
18. Andreas Stolcke. 1998. Entropy-based pruning of backo language models. In Proc. DARPA Broadcast News Transcription and Understanding Workshop , pages 270{274, Lansdowne, Va. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2001-gildea-corpus-variation-parser-performance/