Stanford's Graph-based Neural Dependency Parser at the CoNLL 2017 Shared Task

What it is about

The Stanford system is a biaffine graph parser: a BiLSTM encoder, separate ReLU layers for the head and dependent roles of arcs and labels, and biaffine scorers for arcs and labels. Each word vector adds a character-level representation, built by a character LSTM whose hidden states are attended over and whose final cell state is kept, to a pretrained vector and a frequent-word vector. A separate tagger with the same architecture supplies UPOS and XPOS, and each language gets one tagger and one parser trained only on words and tags. The system was ranked first on all five metrics, with a macro average of 93.09 UPOS and 76.30 LAS. The analysis shows that the graph parser's advantage over the transition-based UDPipe 1.1 grows with the non-projectivity of the test set (p < 0.001), that the parser overfits small treebanks, and that the character model helps more as morphological complexity rises.

Ideas

• I1. gparse: add a character-level word path (char embeddings, a character LSTM, attention over its hidden states plus the final cell state) beside the word embedding, and measure LAS on the EWT dev folds (cv) against the current word-only input.
• I2. gparse: train the parser on its own tagger's predicted tags rather than gold tags, which the paper reports beats UDPipe tags (p < 0.001); measure LAS on EWT dev with gold-tag and predicted-tag training.
• I3. en parse-eval: split EWT dev and test by projectivity and report LAS on the non-projective sentences separately, to see whether the projective arc-standard parser loses accuracy where the paper's graph parser gains.
• I4. gparse/src/eisner.rs: for non-projective output, add a cycle-repair step that picks the best root and fixes cycles as the paper does, and measure LAS on GUM test (test only, NC licence).

For Mova

The ideas touch the character path and the tag feed of gparse, plus the decoder in gparse/src/eisner.rs, and the projectivity split in en parse-eval. The chapters are languages/en/21-clause-structure, languages/en/25-relative-clauses (where non-projective attachments live) and languages/en/29-ellipsis-word-order-information-structure. Measure first I3, since it is free on the existing parser and shows whether the projective assumption costs anything on English.

Topics: Dependency parsing · Part-of-speech tagging

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/k17-3002

@inproceedings{dozat2017stanford,
  title = {{Stanford's Graph-based Neural Dependency Parser at the CoNLL 2017 Shared Task}},
  author = {Dozat, Timothy and Qi, Peng and Manning, Christopher D.},
  booktitle = {Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies},
  year = {2017},
  doi = {10.18653/v1/k17-3002},
  url = {https://doi.org/10.18653/v1/k17-3002}
}

References

18 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Miguel Ballesteros, Chris Dyer, and Noah A Smith. 2015. Improved transition-based parsing by modeling characters instead of words with lstms. EMNLP . find
2. Wang Ling, Tiago Luı́s, Luı́s Marujo, Ramón Fernandez Astudillo, Silvio Amir, Chris Dyer, Alan W Black, and Isabel Trancoso. 2015. Finding function in form: Compositional character models for open vocabulary word representation. NAACL . find
3. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. EMNLP . editors, Information Access Evaluation meets Multilinguality, Multimodality, and Visualization. 5th International Conference of the CLEF Initiative (CLEF 14). Springer, B… find
4. Ryan McDonald, Fernando Pereira, Kiril Ribarov, and Jan Hajič. 2005. Non-projective dependency parsing using spanning tree algorithms. In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages… find our card
5. Scott E. Reed and Nando de Freitas. 2016. Neural programmer-interpreters. find
6. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. International Conference on Learning Representations . find
7. Milan Straka, Jan Hajic, Jana Straková, and Jan Hajic jr. 2015. Parsing universal dependency treebanks using neural networks and search-based oracle. In International Workshop on Treebanks and Linguistic Theories (TLT14). page 208. find
8. Alexander H. Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. In ACL 2016. pages 1400– 1409. find
9. Milan Straka, Jan Hajič, and Jana Straková. 2016. UD-Pipe: trainable pipeline for processing CoNLL-U files performing tokenization, morphological analysis, POS tagging and parsing. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016). European Langua… find
10. Joakim Nivre. 2009. Non-projective dependency parsing in expected linear time. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1-Volume 1. Association for Computational Linguis… find
11. Daniel Zeman, Martin Popel, Milan Straka, Jan Hajič, Joakim Nivre, Filip Ginter, Juhani Luotolahti, Sampo Pyysalo, Slav Petrov, Martin Potthast, Francis Tyers, Elena Badmaeva, Memduh Gökırmak, Anna Nedoluzhko, Silvie Cinková, Jan Hajič jr., Jaroslava Hlaváčová, Václava Kettnerová, Zdeňka U… find
12. CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational Linguistics. find our card
13. Joakim Nivre, Željko Agić, Lars Ahrenberg, et al. 2017a. Universal dependencies 2.0 CoNLL 2017 shared task development and test data. LIN-DAT/CLARIN digital library at the Institute of Formal and Applied Linguistics, Charles University. http://hdl.handle.net/11234/1-2184. ↗
14. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajič, Christopher Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: A multilingual treebank collection. In Proceedings of the 10th Inter… find our card
15. Joakim Nivre et al. 2017b. Universal Dependencies 2.0. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics, Charles University, Prague, http://hdl.handle.net/ 11234/1-1983. http://hdl.handle.net/11234/1-1983. ↗
16. Xingxing Zhang, Jianpeng Cheng, and Mirella Lapata. 2016. Dependency parsing as head selection. find
17. Barbara Plank, Anders Søgaard, and Yoav Goldberg. 2016. Multilingual part-of-speech tagging with bidirectional long short-term memory models and auxiliary loss. ACL . find
18. Martin Potthast, Tim Gollub, Francisco Rangel, Paolo Rosso, Efstathios Stamatatos, and Benno Stein. 2014. Improving the reproducibility of PAN’s shared tasks: Plagiarism detection, author identification, and author profiling. In Evangelos Kanoulas, Mihai Lupu, Paul Clough, Mark Sander-son, Mark Hall… find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/ud-2017-dozat-stanford-s-graph-based-neural/