Parsing Any Domain English text to CoNLL dependencies

What it is about

The paper benchmarks English parsers on text outside the Wall Street Journal. Constituency parsers (Charniak-Johnson with its reranker, Berkeley, Stanford) are converted to CoNLL dependencies with pennconverter, so that they can be compared with two dependency parsers (MateTools, a graph parser, and idparser, a transition parser). The test sets are WSJ sections 22 to 24, Brown genres, QuestionBank questions, a BNC sample and two biomedical sets. The authors also train rerankers for the Berkeley and Stanford parsers. The findings are: reranking helps the lexicalised parsers (Stanford) on every test set but can hurt the Berkeley parser on QuestionBank; self-training of the Charniak-Johnson parser is the strongest single gain; and the dependency parsers fall off more sharply out of domain. The pre-trained Stanford model scores unusually high on QuestionBank, because part of that corpus was in its training data. The paper gives per-corpus LAS and UAS tables for every model, and they show that a benchmark can be misleading when the training split is not clean.

Ideas

• I1. A per-domain table like the paper's Table 2 for en parse-eval: one row per model and one column per genre (school text, narrative, questions, dialogue), with LAS and UAS. Store the table as a BTreeMap-ordered file so that it is identical on every run.
• I2. A deterministic reranker in en/src/parse.rs: keep the k best arc sets from the perceptron parser and rerank them with a fixed set of rule features (projectivity, valid root, one subject per verb). Measure the gain per domain, and keep a negative control in which the rerank features are shuffled.
• I3. Check the split before any claim, as the paper's QuestionBank result shows. Keep the Mova evaluation sets disjoint from the training lessons at the sentence and document level, and report the overlap count.
• I4. Use the cross-formalism comparison: convert Mova's UD output to a second scheme (the paper uses CoNLL and constituency-converted trees) in tools/parsecmp, to see whether an error is in the parser or in the scheme.
• I5. Do not adopt self-training from a parser's own output. Where the paper's gain comes from a self-trained model, Mova should instead take more examples from the teacher, labelled once, and keep the held-out set out of that loop.

For Mova

Domains and genre for school texts: 37. Register and variation and 08. National varieties. Questions: 26. Questions. Measurement design: en/src/parse.rs (reranking), tools/parsecmp (per-domain comparison and conversion), en parse-eval. Measure: LAS and UAS per domain, run twice, with the held-out split recorded. The paper's data are not ours to ship; use the ideas and our own UD sets.

Topics: Dependency parsing

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{kolachina2012parsing,
  title = {{Parsing Any Domain English text to CoNLL dependencies}},
  author = {Kolachina, Sudheer and Kolachina, Prasanth},
  booktitle = {Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)},
  year = {2012},
  url = {https://aclanthology.org/L12-1652/}
}

References

30 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. B. Bohnet. 2010. Very High Accuracy and Fast Dependency Parsing is not a Contradiction. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 89–97, Beijing, China, August. Coling 2010 Organizing Committee. find
2. Ö. Çetinoǧlu, J. Foster, J. Nivre, D. Hogan, A. Cahill, and J. van Genabith. 2010. LFG without C-structures. In Proceedings of the 9th International Workshop on Treebanks and Linguistic Theories. find
3. D. Cer, M.C. De Marneffe, D. Jurafsky, and C.D. Manning. 2010. Parsing to Stanford Dependencies: Trade-offs between speed and accuracy. In Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC’10). European Language Resources Association (ELRA), May. find our card
4. E. Charniak and M. Johnson. 2005. Coarse-to-Fine n-best Parsing and MaxEnt Discriminative Reranking. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 173–180, Ann Arbor, Michigan, June. Association for Computational Linguistics. find our card
5. E. Charniak. 2000. A Maximum-Entropy-Inspired Parser. In Proceedings of the 1st conference on North American Chapter of the Association for Computational Linguistics, pages 132–139. Association for Computational Linguistics, June. find
6. G. Chrupala. 2007. Better Training for Function Labeling. In Proceedings of the International Conference RANLP-2007, Borovets, Bulgaria. find
7. M.C. De Marneffe, B. MacCartney, and C.D. Manning. 2006. Generating Typed Dependency Parses from Phrase Structure Parses. In Proceedings of LREC, volume 6, pages 449–454. find our card
8. J. Foster and J. van Genabith. 2008. Parser Evaluation and the BNC: Evaluating 4 constituency parsers with 3 metrics. In Proceedings of the Sixth conference on International Language Resources and Evaluation (LREC’08). find our card
9. J. Gao, G. Andrew, M. Johnson, and K. Toutanova. 2007. A Comparative Study of Parameter Estimation Methods for Statistical Natural Language Processing. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 824–831, Prague, Czech Republic, June. Association for Computational Linguistics. find
10. D. Gildea. 2001. Corpus Variation and Parser Performance. In Proceedings of the 2001 Conference on Empirical Methods in Natural Language Processing, pages 167–202. Association for Computational Linguistics. find our card
11. R. Johansson and P. Nugues. 2007. Extended Constituent-to-Dependency Conversion for English. In Proceedings of the 16th Nordic Conference on Computational Linguistics (NODALIDA). Citeseer. find our card
12. M. Johnson and A.E. Ural. 2010. Reranking the Berkeley and Brown Parsers. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 665–668, Los Angeles, California, June. Association for Computational Linguistics. find
13. J. Judge, A. Cahill, and J. van Genabith. 2006. QuestionBank: Creating a Corpus of Parse-Annotated Questions. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 497–504, Sydney, Australia, July. Association for Computational Linguistics. find
14. J.D. Kim, T. Ohta, Y. Tateisi, and J. Tsujii. 2003. Genia corpus- A Semantically Annotated Corpus for Bio-Textmining. Bioinformatics, 19(suppl 1):i180. find
15. D. Klein and C.D. Manning. 2003. Accurate Unlexicalized Parsing. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 423–430, Sapporo, Japan, July. Association for Computational Linguistics. find our card
16. M. Lease and E. Charniak. 2005. Parsing Biomedical Literature. In Proceedings of 2nd International Joint Conference on Natural Language Processing, pages 58–69. Asian Federation of Natural Language Processing. find
17. M. Marcus, G. Kim, M.A. Marcinkiewicz, R. MacIntyr, A. Bies, M. Ferguson, K. Katz, and B. Schasberger. 1994. The Penn Treebank: Annotating Predicate Argument Structure. In Proceedings of the workshop on Human Language Technology, HLT ’94, pages 114–119, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
18. D. McClosky, E. Charniak, and M. Johnson. 2006. Effective Self-training for Parsing. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 152–159, New York City, USA, June. Association for Computational Linguistics. find our card
19. F. W. Nelson and H. Kučera. 1979. Manual of Information to accompany a Standard Corpus of present-day edited American English, for use with digital computers. Brown University, Department of Lingustics. find
20. J. Nivre, J. Hall, S. Kübler, R. McDonald, J. Nilsson, S. Riedel, and D. Yuret. 2007. find
21. The CoNLL 2007 Shared Task on Dependency Parsing. In Proceedings of the CoNLL Shared Task Session of EMNLP-CoNLL 2007. Association for Computational Linguistics. find
22. S. Petrov and D. Klein. 2007. Improved Inference for Unlexicalized Parsing. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 404–411, Rochester, New York, April. Association for Computational Linguistics. find
23. S. Petrov, L. Barrett, R. Thibaux, and D. Klein. 2006. Learning Accurate, Compact, and Interpretable Tree Annotation. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 433–440, Sydney, Australia, July. Association for Computational Linguistics. find our card
24. S. Petrov, P. Chang, M. Ringgaard, and H. Alshawi. 2010. Uptraining for Accurate Deterministic Question Parsing. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 705–713, Cambridge, MA, October. Association for Computational Linguistics. S. Sekine and MJ. find
25. Collins. 1997. Evalb bracket scoring program. web page. find
26. M. Surdeanu, R. Johansson, A. Meyers, L. Màrquez, and J. Nivre. 2008. find
27. The CoNLL 2008 Shared Task on Joint Parsing of Syntactic and Semantic Dependencies. In CoNLL 2008: Proceedings of the Twelfth Conference on Computational Natural Language Learning, pages 159–177, Manchester, England, August. Coling 2008 Organizing Committee. find our card
28. Y. Tateisi, A. Yakushiji, T. Ohta, and J. Tsujii. 2005. Syntax Annotation for the GENIA corpus. In Proceedings of 2nd International Joint Conference on Natural Language Processing, pages 222–227. Asian Federation of Natural Language Processing. find
29. I. Titov and J. Henderson. 2007. A Latent Variable Model for Generative Dependency Parsing. In Proceedings of the Tenth International Conference on Parsing Technologies, pages 144–155, Prague, Czech Republic, June. Association for Computational Linguistics. find
30. H. Yamada and Y. Matsumoto. 2003. Statistical Dependency Analysis with Support Vector Machines. In Proceedings of the Eighth International Conference on Parsing Technologies. Association for Computational Linguistics. find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/kolachina-2012-parsing-any-domain-english-text/