Extending a Parser to Distant Domains Using a Few Dozen Partially Annotated Examples

What it is about

The paper revisits domain adaptation for parsers. First, a constituency parser (the Reconciled Span Parser, a span-classification model with contextual word representations and no POS input) trained on the Wall Street Journal reaches 94.3 F1 on the WSJ test set and over 90 F1 on the Brown corpus without any Brown training. For syntactically distant domains, the model is trained directly on partial annotations: a few dozen sentences where annotators mark only some constituents. On geometry text, the share of error-free parses rises from 45.2% to 72.6% with about five dozen annotated sentences, with no drop on newswire. The key design choice is that the model's probability unit matches the annotation unit (is this span a constituent), so partial annotation needs no change to the training procedure.

Ideas

• I1. gparse training: use partial arc annotations (only the arcs a human or teacher has confirmed, with the rest masked out of the loss) and measure LAS on EWT dev with 50, 100 and 200 partially labelled sentences against full training; the paper's claim is that partial labels recover most of the domain gain.
• I2. en parse-eval domain curve: train the en perceptron on EWT, test on a distant domain (ATIS, lowercase queries, LAS 63.0 now) and add a few dozen corrected sentences from that domain; report LAS per corrected sentence, as the paper reports error-free share.
• I3. Report the share of error-free test sentences (sentence exact match) next to LAS in en parse-eval; the paper uses it as the practical measure, and it is more sensitive to the few errors a user sees.

For Mova

The chapters are 20. Syntax: the Universal Dependencies frame and 21-clause-structure.md. The code is gparse (the arc scorer and its training masks) and en/src/parse.rs for the perceptron. The idea fits Mova's rule that examples are not truth: a partially checked sentence is a safer training unit than a full silver tree. Measure on a held-out domain, not on the training data.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1110

@inproceedings{joshi2018extending,
  title = {{Extending a Parser to Distant Domains Using a Few Dozen Partially Annotated Examples}},
  author = {Joshi, Vidur and Peters, Matthew and Hopkins, Mark},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1110},
  url = {https://doi.org/10.18653/v1/p18-1110}
}

References

47 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eric Baucom, Levi King, and Sandra Kübler. 2013. Domain adaptation for parsing. In RANLP. find
2. Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. 2006. Analysis of representations for domain adaptation. In NIPS. find
3. John Blitzer, Ryan T. McDonald, and Fernando Pereira. 2006. Domain adaptation with structural correspondence learning. In EMNLP. find
4. Avrim Blum and Tom Mitchell. 1998. Combining labeled and unlabeled data with co-training. In Proceedings of the eleventh annual conference on Computational learning theory. ACM, pages 92–100. find
5. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In ANLP. find
6. Eugene Charniak and Mark Johnson. 2005. Coarse-to-fine n-best parsing and maxent discriminative reranking. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, pages 173–180. find our card
7. Dan Garrette and Jason Baldridge. 2013. Learning a part-of-speech tagger from two hours of annotation. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 138–147. find
8. Rebecca Hwa. 1999. Supervised grammar induction using training data with limited constituent information. CoRR cs.CL/9905001. find
9. John Judge, Aoife Cahill, and Josef Van Genabith. 2006. Questionbank: Creating a corpus of parse-annotated questions. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pages 497–504. find
10. Daisuke Kawahara and Kiyotaka Uchimoto. 2008. Learning reliability of parses for domain adaptation of dependency parsing. In IJCNLP. find
11. Jonathan K. Kummerfeld, David Leo Wright Hall, James R. Curran, and Dan Klein. 2012. Parser showdown at the wall street corral: An empirical investigation of error types in parser output. In EMNLP-CoNLL. find our card
12. Zhenghua Li, Min Zhang, and Wenliang Chen. 2014. Soft cross-lingual syntax projection for dependency parsing. In COLING. find
13. Zhenghua Li, Yue Zhang, Jiayuan Chao, and Min Zhang. 2016. Training dependency parsers with partial annotation. CoRR abs/1609.09247. ↗
14. Xuezhe Ma and Fei Xia. 2013. Dependency parser adaptation with subtrees from auto-parsed target domain data. In ACL. find
15. James Cross and Liang Huang. 2016. Span-based constituency parsing with a structure-label system and provably optimal dynamic oracles. In EMNLP. find
16. Saeed Majidi and Gregory R. Crane. 2013. Committee-based active learning for dependency parsing. In TPDL. find
17. Hal Daumé. 2007. Frustratingly easy domain adaptation. CoRR abs/0907.1815. ↗
18. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. Computational linguistics 19(2):313–330. find our card
19. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. CoRR abs/1602.07776. http://arxiv.org/abs/1602.07776. ↗
20. Jenny Rose Finkel and Christopher D. Manning. 2009. Hierarchical bayesian domain adaptation. In HLT-NAACL. find
21. Daniel Flannery, Yusuke Miyao, Graham Neubig, and Shinsuke Mori. 2011. Training dependency parsers from partially annotated corpora. In IJCNLP. find
22. Daniel Flannery and Shinsuke Mori. 2015. Combining active learning and partial annotation for domain adaptation of a japanese dependency parser. In IWPT. find
23. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In NIPS. find
24. David McClosky. 2010. Any domain parsing: automatic domain adaptation for natural language parsing . find
25. David McClosky, Eugene Charniak, and Mark Johnson. 2006a. Effective self-training for parsing. In HLT-NAACL. find our card
26. David McClosky, Eugene Charniak, and Mark Johnson. 2006b. Reranking and self-training for parser adaptation. In ACL. find our card
27. David McClosky, Eugene Charniak, and Mark Johnson. 2010. Automatic domain adaptation for parsing. In HLT-NAACL. find
28. Jason Mielens, Liang Sun, and Jason Baldridge. 2015. Parse imputation for dependency annotations. In ACL. find
29. Seyed Abolghasem Mirroshandel and Alexis Nasr. 2011. Active learning for dependency parsing using partially annotated sentences. In IWPT. find
30. Joakim Nivre, Yoav Goldberg, and Ryan T. McDonald. 2014. Constrained arc-eager dependency parsing. Computational Linguistics 40:249–527. find
31. Joakim Nivre, Johan Hall, Sandra Kübler, Ryan Mc-Donald, Jens Nilsson, Sebastian Riedel, and Deniz Yuret. 2007. The conll 2007 shared task on dependency parsing. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL). find
32. Fernando Pereira and Yves Schabes. 1992. Inside-outside reestimation from partially bracketed corpora. In ACL. find
33. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. 2018. Deep contextualized word representations. ArXiv e-prints . find
34. Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In ACL. find
35. Mitchell Stern, Jacob Andreas, and Dan Klein. 2017a. A minimal span-based neural constituency parser. CoRR abs/1705.03919. http://arxiv.org/abs/1705.03919. ↗
36. Mitchell Stern, Daniel Fried, and Dan Klein. 2017b. Effective inference for generative neural parsing. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017. pages 1695–1700. https://aclanthology.info/papers/D17-1178/d17-1178. ↗
37. Yuka Tateisi, Akane Yakushiji, Tomoko Ohta, and Jun’ichi Tsujii. 2005. Syntax annotation for the genia corpus. In Companion Volume to the Proceedings of Conference including Posters/Demos and tutorial abstracts. find
38. Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, HLT-NAACL 2003, Edmonton, Canada, May 27 - June 1, 2003. http://aclweb.org/anthology/N/N03/N03-1033.pdf. ↗
39. Wenhui Wang and Baobao Chang. 2016. Graph-based dependency parsing with bidirectional lstm. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). volume 1, pages 2306–2315. find
40. Juntao Yu, Mohab Elkaref, and Bernd Bohnet. 2015. Domain adaptation for dependency parsing via self-training. In IWPT. find
41. Barbara Plank. 2011. Domain adaptation for parsing. Citeseer. find
42. Roi Reichart and Ari Rappoport. 2007. Self-training for enhancement and domain adaptation of statistical parsers trained on small datasets. In ACL. find
43. Kenji Sagae. 2010. Self-training without reranking for parser domain adaptation and its impact on semantic role labeling. find
44. Kenji Sagae and Jun’ichi Tsujii. 2007. Dependency parsing and domain adaptation with lr models and parser ensembles. In EMNLP-CoNLL. find
45. Manabu Sassano and Sadao Kurohashi. 2010. Using smaller constituents rather than sentences in active learning for japanese dependency parsing. In ACL. find
46. Min Joon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. 2015. Solving geometry problems: Combining text and diagram interpretation. In EMNLP. find
47. Kathrin Spreyer, Lilja Ovrelid, and Jonas Kuhn. 2010. Training parsers on partial trees: A cross-language comparison. In LREC. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/acl-2018-joshi-extending-parser-distant-domains-few/