Evaluating a Statistical CCG Parser on Wikipedia

What it is about

The authors ask how well a statistical CCG parser (C&C) works on Wikipedia text. They hand-label 200 random sentences, which took about 20 hours. Accuracy is 4.3% lower than on newswire, and the parser is 54% slower. Self-training on Simple English Wikipedia raises accuracy on Wikipedia by 0.8% and narrows the gap to 3.8%. Speed improves by 20% on newswire and by 34% on Wikipedia. At the measured speed a full Wikipedia dump would take about 52 days of CPU time. The main lesson is that domain adaptation for a lexicalised parser is mostly adaptation of the supertagger.

Ideas

• I1. Domain drift probe: run en parse-eval on EWT test, on PUD, and on a 200-sentence sample of our own encyclopedic-style text, hand-labelled. Report the LAS drop, as the paper's 4.3% comparison does.
• I2. Self-training with silver trees: add en parses of plain school text to the perceptron, and measure LAS on EWT test with two identical runs. The paper's +0.8% is the reference size.
• I3. Speed table: report sentences per second for EWT and for the hand-labelled sample, and check whether any tag-set pruning gives a speed gain of the kind the paper reports.

For Mova

The code is en/src/parse.rs and en/src/ptag.rs; use en folds for the decision. No Wikipedia text enters the build: the self-training source must be our own plain text. The paper's numbers are C&C numbers on CCG output and are not comparable with LAS.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/1699765.1699771

@inproceedings{honnibal2009evaluating,
  title = {{Evaluating a Statistical CCG Parser on Wikipedia}},
  author = {Honnibal, Matthew and Nothman, Joel and Curran, James R.},
  booktitle = {Proceedings of the 2009 Workshop on The People's Web Meets NLP: Collaboratively Constructed Semantic Resources (People's Web)},
  year = {2009},
  doi = {10.3115/1699765.1699771},
  url = {https://doi.org/10.3115/1699765.1699771}
}

References

16 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Srinivas Bangalore and Aravind Joshi. 1999. Supertagging: An approach to almost parsing. Computational Linguistics, 25(2):237–265. find
2. Stephen Clark and James R. Curran. 2007. Wide-coverage efficient statistical parsing with CCG and log-linear models. Computational Linguistics, 33(4):493–552. find our card
3. Mark Dredze, John Blitzer, Partha Pratim Talukdar, Kuzman Ganchev, João Graca, and Fernando Pereira. 2007. Frustratingly hard domain adaptation for dependency parsing. In Proceedings of the CoNLL Shared Task Session of EMNLP-CoNLL 2007, pages 1051–1055. ACL, Prague, Czech Republic. find our card
4. Jason Eisner. 1996. Efficient normal-form parsing for Combinatory Categorial Grammar. In Proceedings of the Association for Computational Linguistics, pages 79–86. Santa Cruz, CA, USA. find
5. Daniel Gildea. 2001. Corpus variation and parser performance. In Proceedings of the EMNLP Conference, pages 167–202. Pittsburgh, PA. find our card
6. Julia Hockenmaier and Mark Steedman. 2005. CCGbank manual. Technical Report MS-CIS-05-09, Department of Computer Science, University of Pennsylvania. find
7. Julia Hockenmaier and Mark Steedman. 2007. CCGbank: a corpus of CCG derivations and dependency structures extracted from the Penn Treebank. Computational Linguistics, 33(3):355–396. find our card
8. Tibor Kiss and Jan Strunk. 2006. Unsupervised multilingual sentence boundary detection. Computational Linguistics, 32(4):485–525. find our card
9. Edward Loper and Steven Bird. 2002. NLTK: The natural language toolkit. find
10. Joakim Nivre, Johan Hall, Sandra Kübler, Ryan McDonald, Jens Nilsson, Sebastian Riedel, and Deniz Yuret. 2007. find
11. The CoNLL 2007 shared task on dependency parsing. In Proceedings of the CoNLL Shared Task Session, pages 915–932. Prague, Czech Republic. find
12. PediaPress. 2007. mwlib MediaWiki parsing library. http://code.pediapress.com. ↗
13. Laura Rimell and Stephen Clark. 2009. Porting a lexicalized-grammar parser to the biomedical domain. Journal of Biomedical Informatics. (in press). find
14. Mark Steedman. 2000. The Syntactic Process. The MIT Press, Cambridge, MA. find
15. Mark Steedman, Rebecca Hwa, Stephen Clark, Miles Osborne, Anoop Sarkar, Julia Hockenmaier, Paul Ruhlen, Steven Baker, and Jeremiah Crim. 2003. Example selection for bootstrapping statistical parsers. In Proceedings of HLT-NAACL 2003. Edmonton, Alberta. find
16. Gisle Ytrestøl, Stephan Oepen, and Daniel Flickinger. 2009. Extracting and annotating Wikipedia sub-domains. In Proceedings of the 7th International Workshop on Treebanks and Linguistic Theories, pages 185–197. Groningen, Netherlands. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2009-honnibal-evaluating-statistical-ccg-parser-wikipedia/