Prepositional Phrase Attachment over Word Embedding Products

What it is about

The paper scores a PP attachment with a tensor product of the word vectors of the verb, the noun, the preposition and the object, with a low-rank constraint on the tensor. Each candidate head is scored with the same projected PP vector, so the PP is projected once per sentence. The vectors are Skip-gram and Skip-dep embeddings trained on BLLIP, Wikipedia and the NYT corpus. On the RRR test set of 3,097 samples, the model outperforms recent systems except one that uses WordNet features. It is the best on the Wikipedia test (381 samples) and competitive on the NYT test (293 samples). In the multiple-candidate setting of Belinkov et al., it reaches 88.4% on English and 81.1% on Arabic. Rank 50 keeps 88% accuracy with about one sixth of the parameters.

Ideas

• I1. en/src/parse.rs: add conjunction features (preposition x head x object) to the averaged perceptron. This is the transparent, sparse analogue of the tensor product. Measure PP attachment accuracy and LAS on UD EWT test, with two identical runs.
• I2. Count-based low-rank analogue: build a PPMI matrix of (noun, preposition) co-occurrence from en parses of a raw corpus, reduce it to rank 50 by SVD with a fixed seed, and score candidates with the reduced vectors. Measure against the count baseline on EWT test. The SVD must be bit-exact across runs, or it is an approximant and is reported as one.
• I3. Check the paper's claim that the product beats sum and concatenation: score all three on the same EWT dev split under en folds.

For Mova

The chapters are 22. Phrases: noun, adjective, adverb and prepositional phrases and 23-complementation.md. The code is en/src/parse.rs. The model is neural, so Mova builds only the sparse analogue, and the paper's numbers are not Mova's. The RRR, Wikipedia, NYT and Belinkov sets are test only. Embeddings trained on BLLIP (licensed data) may not be carried into the build.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{madhyastha2017prepositional,
  title = {{Prepositional Phrase Attachment over Word Embedding Products}},
  author = {Madhyastha, Pranava Swaroop and Carreras, Xavier and Quattoni, Ariadna},
  booktitle = {Proceedings of the 15th International Conference on Parsing Technologies},
  year = {2017},
  url = {https://aclanthology.org/W17-6305/}
}

References

33 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1. Association for Computational… find
2. Eneko Agirre, Timothy Baldwin, and David Martinez. 2008. Improving parsing and pp attachment performance with sense information. In ACL. pages 317– 325. find
3. Borja Balle and Mehryar Mohri. 2012. Spectral learning of general weighted automata via constrained matrix completion. In Proceedings of Advances in Neural Information Processing Systems (NIPS) . find
4. Jonathan K Kummerfeld, David Hall, James R Curran, and Dan Klein. 2012. Parser showdown at the wall street corral: An empirical investigation of error types in parser output. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Lan… find our card
5. Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2014. Tailoring continuous word representations for dependency parsing. In Proceedings of Association of Computational Linguistics(ACL) Short Papers. find
6. Yonatan Belinkov, Tao Lei, Regina Barzilay, and Amir Globerson. 2014. Exploring compositional architectures and word vector representations for prepositional phrase attachment. Transactions of the Association for Computational Linguistics 2:561–572. find our card
7. Tao Lei, Yu Xin, Yuan Zhang, Regina Barzilay, and Tommi Jaakkola. 2014. Low-rank tensors for scoring dependency structures. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Baltimore, Maryla… find
8. Eugene Charniak, Don Blaheta, Niyu Ge, Keith Hall, John Hale, and Mark Johnson. 2000. BLLIP 1987-89 WSJ Corpus Release 1, LDC No. LDC2000T43. Linguistic Data Consortium. find
9. Pranava Swaroop Madhyastha, Xavier Carreras, and Ariadna Quattoni. 2014. Learning task-specific bilexical embeddings. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers. Dublin City University and Association for Computational Linguistics,… find
10. Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, pages 740–750. http://www.aclweb.org/antho… ↗ our card
11. Pranava Swaroop Madhyastha, Xavier Carreras, and Ariadna Quattoni. 2015. Tailoring word embeddings for bilexical predictions: An experimental comparison. In International Conference on Learning Representations (Workshop Contribution) . find
12. Michael Collins and James Brooks. 1999. Prepositional phrase attachment through a backed-off model. In Natural Language Processing Using Very Large Corpora, Springer, pages 177–189. find
13. John Duchi and Yoram Singer. 2009. Efficient online and batch learning using forward backward splitting. Journal of Machine Learning Research 10:2899– 2934. find
14. Andre Martins, Miguel Almeida, and A. Noah Smith. 2013. Turning on the turbo: Fast third-order non-projective turbo parsers. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, pages 617–622.… find our card
15. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural… find our card
16. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In Proceedings of International Conference on Learning Representations . find
17. Jeff Mitchell and Mirella Lapata. 2008. Vector-based models of semantic composition. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pages 236–244. find
18. Daniel Fried, Tamara Polajnar, and Stephen Clark. 2015. Low-rank tensors for verbs in compositional distributional semantics. In Short Papers of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL 2015). pages 731–736. find
19. Ndapandula Nakashole and Tom M Mitchell. 2015. A knowledge-intensive model for prepositional phrase attachment. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics,(ACL). pages 365–375. find
20. Donald Hindle and Mats Rooth. 1993. Structural ambiguity and lexical relations. Computational linguistics 19(1):103–120. find
21. Jiri Stetina and Makoto Nagao. 1997. Corpus based pp attachment ambiguity resolution with a semantic dictionary. In Proceedings of the fifth workshop on very large corpora. find
22. Marian Olteanu and Dan Moldovan. 2005. Pp-attachment disambiguation using large context. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 273–280. find
23. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Stroudsburg, PA, USA, ACL ’10, p… find
24. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP). pages 1532– 1543. http://www.aclweb.org/anthology/D14-1162. ↗
25. Slav Petrov and Ryan McDonald. 2012. Overview of the 2012 shared task on parsing the web. Notes of the First Workshop on Syntactic Analysis of Non-Canonical Language (SANCL). find
26. Mo Yu, Mark Dredze, Raman Arora, and Matthew R. Gormley. 2016. Embedding lexical features via low-rank tensors. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics… find
27. Audi Primadhanty, Xavier Carreras, and Ariadna Quattoni. 2015. Low-rank regularization for sparse conjunctive feature spaces: An application to named entity classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conf… find
28. Shaojun Zhao and Dekang Lin. 2004. A nearest-neighbor method for resolving pp-attachment ambiguity. In Natural Language Processing–IJCNLP 2004, Springer, pages 545–554. find
29. Ariadna Quattoni, Borja Balle, Xavier Carreras, and Amir Globerson. 2014. Spectral regularization for max-margin sequence tagging. In Tony Jebara and Eric P. Xing, editors, Proceedings of the 31st International Conference on Machine Learning (ICML-14). JMLR Workshop and Conference Proceedings, pages… find
30. Adwait Ratnaparkhi, Jeff Reynar, and Salim Roukos. 1994. A maximum entropy model for prepositional phrase attachment. In Proceedings of the workshop on Human Language Technology. Association for Computational Linguistics, pages 250–255. find our card
31. Djamé Seddah, Sandra Kübler, and Reut Tsarfaty. 2014. Introducing the spmrl 2014 shared task on parsing morphologically-rich languages. In Proceedings of the First Joint Workshop on Statistical Parsing of Morphologically Rich Languages and Syntactic Analysis of Non-Canonical Languages. Dublin City… find
32. Richard Socher, Eric H Huang, Jeffrey Pennin, Christopher D Manning, and Andrew Y Ng. 2011. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In In Proceedings of Advances in Neural Information Processing Systems. pages 801–809. find
33. Nathan Srebro, Jason Rennie, and Tommi S Jaakkola. 2004. Maximum-margin matrix factorization. In In proceedings of Advances in Neural Information Processing Systems (NIPS). pages 1329–1336. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2017-madhyastha-prepositional-phrase-attachment-over-word/