Deep Biaffine Attention for Neural Dependency Parsing

What it is about

The paper builds on the neural graph-based parser of Kiperwasser and Goldberg. It uses a larger and more heavily regularised BiLSTM encoder, passes the top recurrent states through dimension-reducing MLPs, and replaces the MLP attention and the affine label classifier with biaffine ones: one for arc scores over all head-dependent pairs and one for labels. Each arc and label score is a biaffine function of a head-side and a dependent-side vector, which keeps the classifier close in form to an affine one. On the English Penn Treebank the parser reaches 95.7 UAS and 94.1 LAS, which is 1.8 and 2.2 points above Kiperwasser and Goldberg and close to the best transition-based parser of the time (95.8 UAS). The paper also reports which hyperparameter choices change accuracy most.

Ideas

• I1. gparse: check that the arc-head and arc-dependent representations, and the label-head and label-dependent representations, each pass through their own dimension-reducing MLP before the biaffine scorer, and add them if not; measure LAS on EWT dev (cv).
• I2. gparse: report UAS and LAS in the same run with separate loss sums for the arc and label heads, and check whether the label loss hurts the arc score on EWT dev, as the paper's joint training assumes it does not.
• I3. gparse: run the paper's hyperparameter sweep on dropout and the regularisers only (a small grid on EWT dev) before any architectural change, since its results say which settings move accuracy most; start only after the operator's go-ahead, because each GPU run is expensive.

For Mova

The ideas touch the biaffine heads in gparse and its training loop, and they use EWT dev through the cv folds. The chapter that matters most is languages/en/20-syntax-overview, with languages/en/23-complementation for the label set. Measure first I2, which is cheap and tells whether the joint loss costs arc accuracy before any change to the architecture.

Topics: Dependency parsing

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 1611.01734

@misc{dozat2016deep,
  title = {{Deep Biaffine Attention for Neural Dependency Parsing}},
  author = {Dozat, Timothy and Manning, Christopher D.},
  year = {2016},
  eprint = {1611.01734},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1611.01734}
}

References

20 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. Globally normalized transition-based neural networks. In Association for Computational Linguistics, 2016. URL https://arxiv.org/abs/1603.06042. ↗
2. Gabor Angeli, Melvin Johnson Premkumar, and Christopher D Manning. Leveraging linguistic structure for open domain information extraction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL 2015), 2015. find
3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. International Conference on Learning Representations, 2014. find
4. Miguel Ballesteros, Yoav Goldberg, Chris Dyer, and Noah A Smith. Training with exploration improves a greedy stack-LSTM parser. Proceedings of the conference on empirical methods in natural language processing, 2016. find
5. Samuel R Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D Manning, and Christopher Potts. A fast unified model for parsing and sentence understanding. ACL 2016, 2016. find
6. Danqi Chen and Christopher D Manning. A fast and accurate dependency parser using neural networks. In Proceedings of the conference on empirical methods in natural language processing, pp. 740–750, 2014. find our card
7. Hao Cheng, Hao Fang, Xiaodong He, Jianfeng Gao, and Li Deng. Bi-directional attention with agreement for dependency parsing. arXiv preprint arXiv:1608.02076, 2016. find
8. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A Smith. Transition-based dependency parsing with stack long short-term memory. Proceedings of the conference on empirical methods in natural language processing, 2015. find our card
9. Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. International Conference on Machine Learning, 2015. find
10. Klaus Greff, Rupesh Kumar Srivastava, Jan Koutnı́k, Bas R Steunebrink, and Jürgen Schmidhuber. LSTM: A search space odyssey. IEEE Transactions on Neural Networks and Learning Systems, 2015. find
11. Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. A joint many-task model: Growing a neural network for multiple nlp tasks. arXiv preprint arXiv:1611.01587, 2016. find
12. Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2014. find
13. Eliyahu Kiperwasser and Yoav Goldberg. Simple and accurate dependency parsing using bidirectional LSTM feature representations. Transactions of the Association for Computational Linguistics, 4:313–327, 2016. find our card
14. Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, and Noah A. Smith. What do recurrent neural network grammars learn about syntax? CoRR, abs/1611.05774, 2016. URL http://arxiv.org/abs/1611.05774. ↗
15. Omer Levy and Yoav Goldberg. Dependency-based word embeddings. In ACL 2014, pp. 302–308, 2014. find
16. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention-based neural machine translation. Empirical Methods in Natural Language Processing, 2015. find
17. Ankur P Parikh, Hoifung Poon, and Kristina Toutanova. Grounded semantic parsing for complex knowledge extraction. In Proceedings of North American Chapter of the Association for Computational Linguistics, pp. 756–766, 2015. find
18. Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pp… find
19. Kristina Toutanova, Xi Victoria Lin, and Wen-tau Yih. Compositional learning of embeddings for relation paths in knowledge bases and text. In ACL, 2016. find
20. David Weiss, Chris Alberti, Michael Collins, and Slav Petrov. Structured training for neural network transition-based parsing. Annual Meeting of the Association for Computational Linguistics, 2015. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/missing-2016-dozat-deep-biaffine-attention-neural-dependency/