Exploring Compositional Architectures and Word Vector Representations for Prepositional Phrase Attachment

What it is about

PP attachment chooses which candidate head a prepositional phrase depends on. The paper scores each candidate by composing the word vectors of the head, the preposition and the child in a small neural network, trained on attachment decisions. The English data are the Penn Treebank converted to dependencies (sections 2-21 for training, section 23 for testing, 1,951 test instances); the Arabic data are CATiB. The best model, with syntactic vectors and relearning, reaches 88.7% on English and 82.6% on Arabic, above the RBG parser and the self-trained Charniak parser (80.8% on Arabic). Dependency-based vectors beat linear window vectors on English (87.1% against 85.4%), and in the parser the PP predictions lift UAS from 93.96 to 94.05.

Ideas

• I1. en/src/parse.rs: a count-based analogue of the syntactic vectors. Count head-preposition-child triples from the en parses of raw text (dependency context, not a linear window), turn them into PMI scores, and use them as a feature in the PP choice. Measure on EWT PP attachments: linear-window counts against dependency counts, as in the paper's 85.4 against 87.1.
• I2. Distance buckets: the paper uses separate weights for heads 1 to 5 words from the preposition, clipped at 5. Add a weight table per distance bucket to the PP features and measure accuracy on EWT with and without it.
• I3. Candidate window check: the paper limits candidate heads to 10 words before the preposition, which covers about 94% of Arabic and 99% of English attachments. Count on EWT how many attachments fall outside such a window before dropping any candidate.
• I4. Use the 1,951-instance WSJ test set only as a test for Mova, never as training data, and compare with the paper's 88.7% only as a reference.

For Mova

The chapter is 22. Phrases: noun, adjective, adverb and prepositional phrases, and the code is en/src/parse.rs, where the candidate set and the arc choice live. Measure first the candidate window (I3), because it is cheap and decides the search space. The paper's English data is WSJ text under the Penn Treebank terms, so it is for tests only. The neural composition is not kept; only the transparent count-based analogue is a Mova idea.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00203

@article{belinkov2014exploring,
  title = {{Exploring Compositional Architectures and Word Vector Representations for Prepositional Phrase Attachment}},
  author = {Belinkov, Yonatan and Lei, Tao and Barzilay, Regina and Globerson, Amir},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2014},
  doi = {10.1162/tacl\_a\_00203},
  url = {https://doi.org/10.1162/tacl_a_00203}
}

References

49 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eneko Agirre, Timothy Baldwin, and David Martinez. 2008. Improving Parsing and PP Attachment Performance with Sense Information. In Proceedings of ACL-HLT. find
2. Tressy Arts, Yonatan Belinkov, Nizar Habash, Adam Kilgarriff, and Vit Suchomel. 2014. arTenTen: Arabic Corpus and Word Sketches. Journal of King Saud University - Computer and Information Sciences. find
3. Michaela Atterer and Hinrich Schütze. 2007. Prepositional Phrase Attachment Without Oracles. Computational Linguistics, 33(4). find
4. Mohit Bansal, Keving Gimpel, and Karen Livescu. 2014. Tailoring Continuous Word Representations for Dependency Parsing. In Proceedings of ACL. find
5. Yonatan Belinkov, Nizar Habash, Adam Kilgarriff, Noam Ordan, Ryan Roth, and Vı́t Suchomel. 2013. arTen-Ten: a new, vast corpus for Arabic. In Proceedings of WACL. find
6. Yoshua Bengio and Xavier Glorot. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of AISTATS, volume 9, May. find
7. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of ICML. find our card
8. Eric Brill and Philip Resnik. 1994. A Rule-Based Approach to Prepositional Phrase Attachment Disambiguation. In Proceedings of COLING, volume 2. find
9. Volkan Cirik and Hüsnü Şensoy. 2013. The AI-KU System at the SPMRL 2013 Shared Task : Unsupervised Features for Dependency Parsing. In Proceedings of SPMRL. find
10. Michael Collins and James Brooks. 1995. Prepositional Phrase Attachment through a Backed-Off Model. CoRR. find our card
11. Michael Collins. 1997. Three Generative, Lexicalised Models for Statistical Parsing. In Proceedings of ACL. find
12. Ronan Collobert and Jason Weston. 2008. A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning. In Proceedings of ICML. find
13. Fabrizio Costa, Paolo Frasconi, Vincenzo Lombardo, and Giovanni Soda. 2003. Towards Incremental Parsing of Natural Language Using Recursive Neural Networks. Applied Intelligence, 19(1-2). find
14. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. JMLR, 12. find
15. Chris Dyer. n.d. Notes on AdaGrad. Unpublished manuscript, available at http://www.ark.cs.cmu.edu/cdyer/adagrad.pdf. Nuria Gala and Mathieu Lafourcade. 2007. PP attachment ambiguity resolution with corpus-based pattern distributions and lexical signatures. ECTI-CIT Transactions on Computer and Information Technology, 2. ↗
16. Pablo Gamallo, Alexandre Agustini, and Gabriel P. Lopes. 2003. Acquiring Semantic Classes to Elaborate Attachment Heuristics. In Progress in Artificial Intelligence, volume 2902 of LNCS. Springer Berlin Heidelberg. find
17. Spence Green. 2009. Improving Parsing Performance for Arabic PP Attachment Ambiguity. Unpublished manuscript, available at http://www-nlp.stanford.edu/courses/ cs224n/2009/fp/30-tempremove.pdf. ↗
18. Nizar Habash and Owen Rambow. 2005. Arabic Tokenization, Part-of-Speech Tagging and Morphological Disambiguation in One Fell Swoop. In Proceedings of ACL. find our card
19. Nizar Habash and Ryan Roth. 2009. CATiB: The Columbia Arabic Treebank. In Proceedings of the ACL-IJCNLP. find our card
20. Nizar Habash, Owen Rambow, and Ryan Roth. 2005. MADA+TOKAN: A Toolkit for Arabic Tokenization, Diacritization, Morphological Disambiguation, POS Tagging, Stemming and Lemmatization. In Proceedings of the Second International Conference on Arabic Language Resources and Tools. find
21. Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2012. Improving neural networks by preventing co-adaptation of feature detectors. CoRR. find
22. Karin Kipper, Anna Korhonen, Neville Ryant, and Martha Palmer. 2008. A large-scale classification of English verbs. Language Resources and Evaluation, 42(1). find
23. Terry Koo, Xavier Carreras, and Michael Collins. 2008. Simple Semi-supervised Dependency Parsing. In Proceedings of ACL-HLT. find
24. Jonathan K. Kummerfeld, David Hall, James R. Curran, and Dan Klein. 2012. Parser Showdown at the Wall Street Corral: An Empirical Investigation of Error Types in Parser Output. In Proceedings of EMNLP-CoNLL. find our card
25. Tao Lei, Yu Xin, Yuan Zhang, Regina Barzilay, and Tommi Jaakkola. 2014. Low-Rank Tensors for Scoring Dependency Structures. In Proceedings of ACL. find
26. Omer Levy and Yoav Goldberg. 2014. Dependency-Based Word Embeddings. In Proceedings of ACL. find
27. Percy Liang. 2005. Semi-Supervised Learning for Natural Language. Master’s thesis, Massachusetts Institute of Technology. find
28. Andre Martins, Noah Smith, Eric Xing, Pedro Aguiar, and Mario Figueiredo. 2010. Turbo Parsers: Dependency Parsing by Approximate Variational Inference. In Proceedings of EMNLP. find
29. Andre Martins, Miguel Almeida, and Noah A. Smith. 2013. Turning on the Turbo: Fast Third-Order Non-Projective Turbo Parsers. In Proceedings of ACL. find our card
30. David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective Self-Training for Parsing. In Proceedings of HLT-NAACL. find our card
31. Ryan McDonald, Koby Crammer, and Fernando Pereira. 2005. Online Large-Margin Training of Dependency Parsers. In Proceedings of ACL. find our card
32. Srinivas Medimi and Pushpak Bhattacharyya. 2007. A Flexible Unsupervised PP-attachment Method Using Semantic Information. In Proceedings of IJCAI. find
33. Sauro Menchetti, Fabrizio Costa, Paolo Frasconi, and Massimiliano Pontil. 2005. Wide coverage natural language processing using kernel methods and neural networks for structured data. Pattern Recognition Letters, 26(12). find
34. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient Estimation of Word Representations in Vector Space. In Proceedings of Workshop at ICLR. find
35. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013b. Distributed Representations of Words and Phrases and their Compositionality. In Proceedings of NIPS. find
36. Jaouad Mousser. 2010. A Large Coverage Verb Taxonomy for Arabic. In Proceedings of LREC. find
37. J. Nivre, J. Hall, and J. Nilsson. 2006. MaltParser: A Data-Driven Parser-Generator for Dependency Parsing. In Proceedings of LREC. find
38. Marian Olteanu and Dan Moldovan. 2005. PP-attachment Disambiguation using Large Context. In Proceedings of HLT-EMNLP. find
39. Princeton University. 2010. WordNet. http:// wordnet.princeton.edu. ↗
40. Adwait Ratnaparkhi, Jeff Reynar, and Salim Roukos. 1994. A Maximum Entropy Model for Prepositional Phrase Attachment. In Proceedings of HLT. find our card
41. Horacio Rodrı́quez, David Farwell, Javi Ferreres, Manuel Bertran, Musa Alkhalifa, and M. Antonia Martı́. 2008. Arabic WordNet: Semi-automatic Extensions using Bayesian Inference. In Proceedings of LREC. find
42. Djamé Seddah, Reut Tsarfaty, Sandra Kübler, Marie Candito, Jinho D. Choi, Richárd Farkas, Jennifer Foster, et al. 2013. Overview of the SPMRL 2013 Shared Task: A Cross-Framework Evaluation of Parsing Morphologically Rich Languages. In Proceedings of SPMRL. find
43. Richard Socher, Christopher D. Manning, and Andrew Y. Ng. 2010. Learning Continuous Phrase Representations and Syntactic Parsing with Recursive Neural Networks. In Proceedings of NIPS Deep Learning and Unsupervised Feature Learning Workshop. find
44. Richard Socher, John Bauer, Christopher D. Manning, and Ng Andrew Y. 2013. Parsing with Compositional Vector Grammars. In Proceedings of ACL. find
45. Jiri Stetina and Makoto Nagao. 1997. Corpus Based PP Attachment Ambiguity Resolution with a Semantic Dictionary. In Fifth Workshop on Very Large Corpora. find
46. Simon Šuster. 2012. Resolving PP-attachment ambiguity in French with distributional methods. Master’s thesis, Université de Lorraine & Rijksuniversiteit Groningen. find
47. Joseph Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010. Word Representations: A Simple and General Method for Semi-Supervised Learning. In Proceedings of ACL. find
48. Martin Volk. 2002. Combining Unsupervised and Supervised Methods for PP Attachment Disambiguation. In Proceedings of COLING. find our card
49. Stefan Wager, Sida Wang, and Percy Liang. 2013. Dropout Training as Adaptive Regularization. In Proceedings of NIPS. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/tacl-2014-belinkov-exploring-compositional-architectures-word-vector/