Ontology-Aware Token Embeddings for Prepositional Phrase Attachment

What it is about

A type-level embedding gives one vector per word form, so "pool" as a place and "pool" as a game share a vector. The paper builds a vector for each token as a weighted sum of WordNet concept vectors. The weights come from a context-dependent distribution over the word's senses and their hypernyms, with a sense prior that decays with sense rank. The model is a bi-LSTM that picks the head of a prepositional phrase, trained and tested on the English data of Belinkov et al. (test set of 1951 examples). It reaches 89.7% accuracy, against 84.3% for plain GloVe vectors and 88.7% for the earlier HPCD system; the abstract reports a gain of 5.4 absolute points, or 34.4% fewer errors. Removing the sense priors gives 88.4% and removing attention gives 87.5%.

Ideas

• I1. onto/src/wsd.rs: the sense prior is a rank-based weight, as in the paper. Use the same weight for the PP head candidates: score each candidate by the hypernym overlap between the head and the PP object, using the first senses and at most 5 hypernym levels (the paper's caps). Measure accuracy on EWT PP cases with and without the feature, in two identical runs.
• I2. en/src/parse.rs: add a deterministic feature for shared hypernyms of the head and the object of the preposition, drawn from OEWN, to the perceptron that chooses the PP head. Measure LAS on EWT test with en parse-eval and the change on PP arcs alone.
• I3. Ablation first: remove the sense-rank prior and the context weight in turn, as the paper does, and measure each drop on EWT before adding more.
• I4. Training-size curve: train the scorer on 2000, 5000 and all EWT sentences and report the gap with and without the hypernym feature. The paper finds the gap is largest on small training sets.

For Mova

The chapters are 22. Phrases: noun, adjective, adverb and prepositional phrases and 23-complementation.md. The code is onto/src/wsd.rs for the sense weights and en/src/parse.rs for the PP choice. The paper uses WordNet 3.1 and an LSTM, so no number carries over: any figure for Mova must come from EWT test, with OEWN and the perceptron. The Belinkov test set is WSJ text from the LDC-licensed Penn Treebank, so it is for tests only.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-1191 · arXiv 1705.02925

@inproceedings{dasigi2017ontology,
  title = {{Ontology-Aware Token Embeddings for Prepositional Phrase Attachment}},
  author = {Dasigi, Pradeep and Ammar, Waleed and Dyer, Chris and Hovy, Eduard},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-1191},
  eprint = {1705.02925},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p17-1191}
}

References

27 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eneko Agirre. 2008. Improving parsing and pp attachment performance with sense information. In ACL. Citeseer. find
2. Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2016. Linear algebraic structure of word senses, with applications to polysemy. arXiv preprint arXiv:1601.03764 . ↗
3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR abs/1409.0473. ↗
4. David Belanger and Sham M. Kakade. 2015. A linear dynamical system model for text. In ICML. find
5. Yonatan Belinkov, Tao Lei, Regina Barzilay, and Amir Globerson. 2014. Exploring compositional architectures and word vector representations for prepositional phrase attachment. Transactions of the Association for Computational Linguistics 2:561–572. find our card
6. Eric Brill and Philip Resnik. 1994. A rule-based approach to prepositional phrase attachment disambiguation. In Proceedings of the 15th conference on Computational linguistics-Volume 2. Association for Computational Linguistics, pages 1198–1204. find
7. Danqi Chen and Christopher D Manning. 2014. A fast and accurate dependency parser using neural networks. In EMNLP. pages 740–750. find our card
8. Xinxiong Chen, Zhiyuan Liu, and Maosong Sun. 2014. A unified model for word sense representation and disambiguation. In EMNLP. pages 1025–1035. find
9. François Chollet. 2015. Keras. https://github.com/fchollet/keras. ↗
10. Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard H. Hovy, and Noah A. Smith. 2015. Retrofitting word vectors to semantic lexicons. In NAACL. find
11. Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1. Association for Computational Linguistics, pages 873–882. find
12. Sujay Kumar Jauhar, Chris Dyer, and Eduard H. Hovy. 2015. Ontologically grounded multi-sense representation learning for semantic vector space models. In NAACL. find
13. Richard Johansson and Luis Nieto Pina. 2015. Embedding a semantic network in a word space. In In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics–Human Language Technologies. Citeseer. find
14. Karin Kipper, Anna Korhonen, Neville Ryant, and Martha Palmer. 2008. A large-scale classification of english verbs. Language Resources and Evaluation 42(1):21–40. find
15. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In NAACL. find
16. Tao Lei, Yuan Zhang, Regina Barzilay, and Tommi Jaakkola. 2014. Low-rank tensors for scoring dependency structures. In ACL. Association for Computational Linguistics. find
17. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. CoRR abs/1310.4546. ↗
18. George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM 38(11):39– 41. ↗ our card
19. Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, and Andrew McCallum. 2015. Efficient non-parametric estimation of multiple embeddings per word in vector space. arXiv preprint arXiv:1504.06654 . ↗
20. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In EMNLP. find
21. Adwait Ratnaparkhi, Jeff Reynar, and Salim Roukos. 1994. A maximum entropy model for prepositional phrase attachment. In Proceedings of the workshop on Human Language Technology. find our card
22. Joseph Reisinger and Raymond J Mooney. 2010. Multi-prototype vector-space models of word meaning. In HLT-ACL. find
23. Philip Resnik. 1993. Semantic classes and syntactic ambiguity. In Proceedings of the workshop on Human Language Technology. Association for Computational Linguistics. find
24. Sascha Rothe and Hinrich Schütze. 2015. Autoextend: Extending word embeddings to embeddings for synsets and lexemes. In ACL. find
25. Luke Vilnis and Andrew McCallum. 2015. Word representations via gaussian embedding. In ICLR. find
26. Mo Yu and Mark Dredze. 2014. Improving lexical embeddings with semantic knowledge. In ACL. find
27. Benat Zapirain, Eneko Agirre, Lluis Marquez, and Mihai Surdeanu. 2013. Selectional preferences for semantic role classification. Computational Linguistics . find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/onto-2017-dasigi-ontology-aware-token-embeddings-prepositional/