Resolving Prepositional Phrase Attachment Ambiguities with Contextualized Word Embeddings

What it is about

The paper treats PP attachment as a choice among candidate heads (a verb and a noun, or several heads) for a preposition, with the child noun as context. It fine-tunes a contextual embedding model (RoBERTa or BERT) on two English data sets, RRR (binary verb-or-noun) and BLBG (a list of candidate heads). The test sets have 3097 and 1951 items. On RRR the best model reaches 88.8% against 88.1% for the earlier best result, a gain of 0.7 points. On BLBG the head-words-only model is 1.7 points above the best earlier result, and adding the full sentence as context gives a further gain: the paper reports a total of 4.1 points over that result. Full sentence context adds about 2% to 3% accuracy, without any syntactic annotation. The model has at most 126M parameters.

Ideas

• I1. en/src/ctx.rs and en/src/parse.rs: add the lemmas of the whole sentence as a bag of features for the PP attachment choice, with no syntax. Measure accuracy on EWT test PP attachments with and without the bag, on the two-fold check, and run twice to confirm the numbers are identical.
• I2. A transparent head-triple table: counts of (candidate head, preposition, child noun) with back-off to (head, preposition) and then to the preposition alone, printed by en explain with the counts that decided. This is the cheap analogue of the contextual model. Measure on EWT test against the perceptron.
• I3. Silver labels from the teacher: a large model proposes attachment labels for unlabelled English PP sentences in annot offline, and a Rust gate keeps only the ones that agree with the parser's tree evidence. The count table is trained on that silver set. The teacher never answers at test time. Measure the table's accuracy on gold EWT test.

For Mova

The code is en/src/parse.rs and en/src/ctx.rs. The chapter is 22. Phrases: noun, adjective, adverb and prepositional phrases. The neural model itself is not a Mova component; only the context idea and the evaluation habit carry over. The RRR and BLBG sets are derived from the Penn Treebank, so they are test only and must not enter any training file. Any figure for Mova must come from EWT test.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{ratnaparkhi2021resolving,
  title = {{Resolving Prepositional Phrase Attachment Ambiguities with Contextualized Word Embeddings}},
  author = {Ratnaparkhi, Adwait and Kumar, Atul},
  booktitle = {Proceedings of the 18th International Conference on Natural Language Processing (ICON)},
  year = {2021},
  url = {https://aclanthology.org/2021.icon-main.40/}
}

References

19 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. C. Baker, C. Fillmore, and J. Lowe. 1998. The Berkeley FrameNet Project. In COLING-ACL. find our card
2. Yonatan Belinkov, Tao Lei, Regina Barzilay, and Amir Globerson. 2014. Exploring compositional architectures and word vector representations for prepositional phrase attachment. Transactions of the Association for Computational Linguistics, 2:561–572. find our card
3. Eric Brill and Philip Resnik. 1994. A rule-based approach to prepositional phrase attachment disambiguation. In COLING 1994. find
4. Michael Collins and James Brooks. 1995. Prepositional phrase attachment through a backed-off model. In Proceedings of the Third Workshop on Very Large Corpora. find our card
5. Pradeep Dasigi, Waleed Ammar, Chris Dyer, and Eduard Hovy. 2017. Ontology-aware token embeddings for prepositional phrase attachment. find our card
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. find
7. Bich-Ngoc Do and Ines Rehbein. 2020. Parsers know best: German PP attachment revisited. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2049–2061. find
8. Donald Hindle and Mats Rooth. 1993. Structural ambiguity and lexical relations. Computational linguistics, 19(1):103–120. find
9. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. find
10. Pranava Swaroop Madhyastha, Xavier Carreras, and Ariadna Quattoni. 2017. Prepositional phrase attachment over word embedding products. In Proceedings of the 15th International Conference on Parsing Technologies, pages 32–43. find our card
11. M. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Comput. Linguistics, 19:313–330. find our card
12. G. Miller. 1995. Wordnet: a lexical database for english. Commun. ACM, 38:39–41. find our card
13. Marian Olteanu and Dan Moldovan. 2005. PP-attachment disambiguation using large context. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 273–280. find
14. Adwait Ratnaparkhi, Jeff Reynar, and Salim Roukos. 1994. A maximum entropy model for prepositional phrase attachment. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994. find our card
15. Jiri Stetina and Makoto Nagao. 1997. Corpus based PP attachment ambiguity resolution with a semantic dictionary. In Fifth Workshop on Very Large Corpora. find
16. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
17. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… find
18. Mo Yu, Mark Dredze, Raman Arora, and Matthew R Gormley. 2016. Embedding lexical features via low-rank tensors. In Proceedings of NAACL-HLT, pages 1019–1029. find
19. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. CoRR, abs/1808.05326. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2021-ratnaparkhi-resolving-prepositional-phrase-attachment-ambiguities/