Recursive LSTM Tree Representation for Arc-Standard Transition-Based Dependency Parsing

What it is about

The paper encodes a partial dependency tree as one dense vector, built by applying LSTMs recursively to each head and its dependents. A greedy arc-standard parser then needs only the vectors of the top two stack items. The authors say this is the smallest feature set published for arc-standard parsers, and that its accuracy is competitive. On the CoNLL 2018 English EWT test set the scores are UAS 82.86 and LAS 80.18, with punctuation included. The recursive vectors encode structure well, and the authors note that treating dependents as unordered costs about 1% accuracy.

Ideas

• I1. en/src/parse.rs: compare the current feature set with a two-item stack feature set, using the same perceptron, on EWT dev. Report LAS and training seconds, and run twice before any claim.
• I2. Dependent order: the paper finds that losing the order of dependents costs about 1%. Add the left-to-right order of a head's dependents to its feature in en/src/parse.rs, and measure LAS on EWT dev.
• I3. Scoring convention: report LAS with and without punctuation on EWT test, because the paper's scores include punctuation. en ud-eval should print both, so that comparisons are like with like.

For Mova

The code is en/src/parse.rs; the commands are en parse-eval and en ud-eval; the chapter is 21. The simple clause. Mova's current EWT LAS of 79.8 excludes punctuation, so compare only like with like. EWT is CC BY-SA and may be used for training. The model here is neural, so only the feature-set and order ideas are ported.

Topics: Dependency parsing

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-8012

@inproceedings{elkaref2019recursive,
  title = {{Recursive LSTM Tree Representation for Arc-Standard Transition-Based Dependency Parsing}},
  author = {Elkaref, Mohab and Bohnet, Bernd},
  booktitle = {Proceedings of the Third Workshop on Universal Dependencies (UDW, SyntaxFest 2019)},
  year = {2019},
  doi = {10.18653/v1/w19-8012},
  url = {https://doi.org/10.18653/v1/w19-8012}
}

References

24 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Globally normalized transition-based neural networks. arXiv preprint arXiv:1603.06042. find
2. Miguel Ballesteros, Yoav Goldberg, Chris Dyer, and Noah A. Smith. 2016. Training with exploration improves a greedy stack lstm parser. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2005–2010. Association for Computational Linguistics. find
3. Danqi Chen and Christopher D Manning. 2014. A fast and accurate dependency parser using neural networks. In EMNLP, pages 740–750. find our card
4. James Cross and Liang Huang. 2016. Incremental parsing with minimal features using bi-directional LSTM. CoRR, abs/1606.06406. find
5. Marie-Catherine De Marneffe, Bill MacCartney, Christopher D Manning, et al. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of LREC, volume 6, pages 449–454. find our card
6. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A Smith. 2015. Transition-based dependency parsing with stack long short-term memory. arXiv preprint arXiv:1505.08075. find our card
7. Yarin Gal. 2015. A theoretically grounded application of dropout in recurrent neural networks. arXiv:1512.05287. find
8. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learn… find
9. Christoph Goller and Andreas Kuchler. 1996. Learning task-dependent distributed representations by backpropagation through structure. In Neural Networks, 1996., IEEE International Conference on, volume 1, pages 347–352. IEEE. find
10. Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580. find
11. Diederik Kingma and Jimmy Ba. arXiv:1412.6980. 2014. find
12. Adam: A method for stochastic optimization. arXiv preprint Eliyahu Kiperwasser and Yoav Goldberg. 2016a. Easy-first dependency parsing with hierarchical tree lstms. arXiv preprint arXiv:1603.00375. find
13. Eliyahu Kiperwasser and Yoav Goldberg. 2016b. Simple and accurate dependency parsing using bidirectional lstm feature representations. arXiv preprint arXiv:1603.04351. find our card
14. Phong Le and Willem Zuidema. 2014. The inside-outside recursive neural network model for dependency parsing. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 729–739, Doha, Qatar, October. Association for Computational Linguistics. find
15. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313–330. find our card
16. Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pages 807–814. find
17. Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, Kevin Duh, Manaal Faruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji, Lingpeng Kong, Adhiguna Kuncoro, Gaurav Kumar, Chaitanya Malaviya, Pau… find
18. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. find
19. Tianze Shi, Liang Huang, and Lillian Lee. 2017. Fast(er) exact decoding and global training for transition-based dependency parsing via a minimal feature set. CoRR, abs/1708.09403. find
20. Richard Socher, Christopher D Manning, and Andrew Y Ng. 2010. Learning continuous phrase representations and syntactic parsing with recursive neural networks. In Proceedings of the NIPS-2010 Deep Learning and Unsupervised Feature Learning Workshop, pages 1–9. find
21. Pontus Stenetorp. 2013. Transition-based dependency parsing using recursive neural networks. In NIPS Workshop on Deep Learning. Citeseer. find
22. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural La… find
23. David Weiss, Chris Alberti, Michael Collins, and Slav Petrov. 2015. Structured training for neural network transition-based parsing. arXiv preprint arXiv:1506.06158. find
24. Hao Zhou, Yue Zhang, Shujian Huang, and Jiajun Chen. 2015. A neural probabilistic structured-prediction model for transition-based dependency parsing. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, pages 1213–1222. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/ud-2019-elkaref-recursive-lstm-tree-representation-arc/