Mova0.5 pre-release⌕
☰
home / docs / papers / English / We Understand Elliptical Sentences, and Language Models should Too: A New Dataset for Studying Ellipsis and its Interaction with Thematic Fit

We Understand Elliptical Sentences, and Language Models should Too: A New Dataset for Studying Ellipsis and its Interaction with Thematic Fit

What it is about

The authors build ELLie, a dataset of English elliptical sentences (sluicing, gapping and other constructions) organised by thematic role: agent, patient, instrument, location and time. Each item pairs a typical, atypical or impossible filler with the elided clause. They test two Transformer language models on three tasks: whether the sentence probability is higher for typical events, whether the probability of candidate fillers follows thematic fit, and whether the models can retrieve the elided verb. The models prefer typical events, but they fail to separate atypical from impossible ones and retrieve the elided verb poorly. The authors conclude that the models have limited knowledge of ellipsis.

Ideas

For Mova

Ellipsis is a case where the tree alone does not give the meaning, which is the point of prag. The language models are not allowed in Mova, so only the count-based fit and the frame-copy rule are options. Relevant chapters are languages/en/29-ellipsis-word-order-information-structure and 23-complementation. Measure first the share of ELLie elided verbs recovered by the frame copy, with no training on the dataset.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-long.188

@inproceedings{testa2023we,
  title = {{We Understand Elliptical Sentences, and Language Models should Too: A New Dataset for Studying Ellipsis and its Interaction with Thematic Fit}},
  author = {Testa, Davide and Chersoni, Emmanuele and Lenci, Alessandro},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.188},
  url = {https://doi.org/10.18653/v1/2023.acl-long.188}
}

References

51 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Peter W. Culicover and Ray Jackendoff. 2006. The Simpler Syntax Hypothesis. TRENDS in Cognitive Sciences, 10:413–418. find
  2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL. find
  3. Rahul Aralikatte, Matthew Lamm, Daniel Hardt, and Anders Søgaard. 2021. Ellipsis Resolution as Question Answering: An Evaluation. In Proceedings of EACL. find our card
  4. Myroslava O Dzikovska, Charles B Callaway, Elaine Farrow, Johanna D Moore, Natalie Steinhauser, and Gwendolyn Campbell. 2009. Dealing with Interpretation Errors in Tutorial Dialogue. In Proceedings of SIGDIAL. find
  5. Marco Baroni and Alessandro Lenci. 2010. Distributional Memory: A General Framework for Corpus-Based Semantics. Computational Linguistics, 36(4):673–721. find
  6. Katrin Erk, Sebastian Padó, and Ulrike Padó. 2010. A Flexible, Corpus-Driven Model of Regular and Inverse Selectional Preferences. Computational Linguistics, 36(4):723–763. find
  7. Klinton Bicknell, Jeffrey L Elman, Mary Hare, Ken McRae, and Marta Kutas. 2010. Effects of Event Knowledge in Processing Verbal Arguments. Journal of Memory and Language, 63(4):489–505. find
  8. Todd R Ferretti, Marta Kutas, and Ken McRae. 2007. Verb Aspect and the Activation of Event Knowledge. Journal of Experimental Psychology: Learning, Memory, and Cognition, 33(1):182. find
  9. Emmanuele Chersoni, Philippe Blache, and Alessandro Lenci. 2016. Towards a Distributional Model of Semantic Complexity. In Proceedings of the COLING Workshop on Computational Linguistics for Linguistic Complexity (CL4LC). find
  10. Emmanuele Chersoni, Ludovica Pannitto, Enrico Santus, Alessandro Lenci, and Chu-Ren Huang. 2020. Are Word Embeddings Really a Bad Fit for the Estimation of Thematic Fit? In Proceedings of LREC. find
  11. Emmanuele Chersoni, Enrico Santus, Philippe Blache, and Alessandro Lenci. 2017. Is Structure Necessary for Modeling Argument Expectations in Distributional Semantics? In Proceedings of IWCS. find
  12. Emmanuele Chersoni, Enrico Santus, Alessandro Lenci, Philippe Blache, and Chu-Ren Huang. 2021. Not All Arguments Are Processed Equally: A Distributional Model of Argument Complexity. Language Resources and Evaluation, pages 1–28. find
  13. Emmanuele Chersoni, Enrico Santus, Ludovica Pannitto, Alessandro Lenci, Philippe Blache, and C-R Huang. 2019. A Structured Distributional Model of Sentence Meaning and Processing. Natural Language Engineering, 25(4):483–502. find
  14. Todd R Ferretti, Ken McRae, and Andrea Hatherell. 2001. Integrating Verbs, Situation Schemas, and Thematic Role Concepts. Journal of Memory and Language, 44(4):516–547. find
  15. Jonathan Ginzburg and Ivan Sag. 2000. Interrogative Investigations. Stanford: CSLI Publications. find
  16. Clayton Greenberg, Vera Demberg, and Asad Sayeed. 2015a. Verb Polysemy and Frequency Effects in Thematic Fit Modeling. In Proceedings of the NAACL Workshop on Cognitive Modeling and Computational Linguistics. find
  17. Clayton Greenberg, Asad B Sayeed, and Vera Demberg. 2015b. Improving Unsupervised Vector-Space Thematic Fit Evaluation via Role-Filler Prototype Clustering. In Proceedings of NAACL-HLT. find
  18. Victor Petrén Bach Hansen and Anders Søgaard. 2020. What Do You Mean ‘Why?’: Resolving Sluices in Conversations. In Proceedings of AAAI. find
  19. Mary Hare, Michael Jones, Caroline Thomson, Sarah Kelly, and Ken McRae. 2009. Activating Event Knowledge. Cognition, 111(2):151–167. find
  20. Xudong Hong, Asad Sayeed, and Vera Demberg. 2018. Learning Distributed Event Representations with a Multi-task Approach. In Proceedings of *SEM. find
  21. Won Ik Cho, Emmanuele Chersoni, Yu-Yin Hsu, and Chu-Ren Huang. 2021. Modeling the Influence of Verb Aspect on the Activation of Typical Event Locations with BERT. In Findings of ACL-IJCNLP 2021. find
  22. Pauline Jacobson. 2012. Direct Compositionality. In Markus Werning, Wolfram Hinzen, and Edouard Machery, editors, The Oxford Handbook of Compositionality, pages 109–128. Oxford University Press, Oxford. find
  23. Tagyoung Chung and Daniel Gildea. 2010. Effects of Empty Categories on Machine Translation. In Proceedings of EMNLP, pages 636–645. find
  24. Carina Kauf, Anna A Ivanova, Giulia Rambelli, Emmanuele Chersoni, Jingyuan S She, Zawad Chowdhury, Evelina Fedorenko, and Alessandro Lenci. 2022. Event Knowledge in Large Language Models: The Gap between the Impossible and the Unlikely. arXiv preprint arXiv:2212.01488. ↗
  25. Peter W Culicover and Ray Jackendoff. 2005. Simpler Syntax. Oxford University Press. find
  26. Alessandro Lenci. 2011. Composing and Updating Verb Argument Expectations: A Distributional Semantic Model. In Proceedings of the ACL Workshop on Cognitive Modeling and Computational Linguistics. find
  27. Carol Madden-Lombardi, Peter Ford Dominey, and Jocelyne Ventre-Dominey. 2017. Grammatical Verb Aspect and Event Roles in Sentence Processing. PLOS One, 12(12). find
  28. Yuval Marton and Asad Sayeed. 2022. Thematic Fit Bits: Annotation Quality and Quantity Interplay for Event Participant Representation. In Proceedings of LREC. find
  29. Kazunaga Matsuki, Tracy Chow, Mary Hare, Jeffrey L Elman, Christoph Scheepers, and Ken McRae. 2011. Event-Based Plausibility Immediately Influences On-Line Language Comprehension. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37(4):913. find
  30. Ken McRae, Mary Hare, Jeffrey L Elman, and Todd Ferretti. 2005. A Basis for Generating Expectancies for Verbs from Nouns. Memory & Cognition, 33(7):1174–1184. find
  31. Ken McRae and Kazunaga Matsuki. 2009. People Use their Knowledge of Common Events to Understand Language, and Do So as Quickly as Possible. Language and Linguistics Compass, 3(6):1417–1429. find
  32. Ken McRae, Michael J Spivey-Knowlton, and Michael K Tanenhaus. 1998. Modeling the Influence of Thematic Fit (and Other Constraints) in On-line Sentence Comprehension. Journal of Memory and Language, 38(3):283–312. find
  33. Marjorie J. McShane. 2005. A Theory of Ellipsis. Oxford University Press. find
  34. Jason Merchant. 2013. Voice and Ellipsis. Linguistic Inquiry, 44(1):77–108. find
  35. Jason Merchant. 2018. Ellipsis: A Survey of Analytical Approaches. In Jeroen van Craenenbroeck and Tanja Temmerman, editors, A Handbook of Ellipsis. Oxford University Press. find
  36. Kanishka Misra. 2022. minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models. arXiv preprint arXiv:2203.13112. ↗
  37. Paolo Pedinotti, Giulia Rambelli, Emmanuele Chersoni, Enrico Santus, Alessandro Lenci, and Philippe Blache. 2021. Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge. In Proceedings of *SEM. find
  38. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models Are Unsupervised Multitask Learners. OpenAI Blog, 1(8):9. find
  39. Giulia Rambelli, Emmanuele Chersoni, Alessandro Lenci, Philippe Blache, and Chu-Ren Huang. 2020. Comparing Probabilistic, Distributional and Transformer-based Models on Logical Metonymy Interpretation. In Proceedings of AACL-IJCNLP. find
  40. Ola Rønning, Daniel Hardt, and Anders Søgaard. 2018. Linguistic Representations in Multi-task Neural Networks for Ellipsis Resolution. In Proceedings of the EMNLP Workshop on Analyzing and Interpreting Neural Networks (BlackboxNLP). find
  41. Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff. 2020. Masked Language Model Scoring. In Proceedings of ACL. find
  42. Enrico Santus, Emmanuele Chersoni, Alessandro Lenci, and Philippe Blache. 2017. Measuring Thematic Fit with Distributional Feature Overlap. In Proceedings of EMNLP. find
  43. Asad Sayeed, Clayton Greenberg, and Vera Demberg. 2016. Thematic Fit Evaluation: An Aspect of Selectional Preferences. In Proceedings of the ACL Workshop on Evaluating Vector Space Representations for NLP. find
  44. Kerstin Schwabe and Susanne Winkler. 2003. The Interfaces: Deriving and Interpreting Omitted Structures. John Benjamins Publishing. find
  45. Mark Steedman and Jason Baldridge. 2011. Combinatory Categorial Grammar. Non-Transformational Syntax: Formal and Explicit Models of Grammar, pages 181–224. find
  46. Ottokar Tilk, Vera Demberg, Asad Sayeed, Dietrich Klakow, and Stefan Thater. 2016. Event Participant Modelling with Neural Networks. In Proceedings of EMNLP. find
  47. Jeroen van Craenenbroeck and Tanja Temmerman, editors. 2018. The Oxford Handbook of Ellipsis. Oxford University Press, Oxford. find
  48. Paolo Vassallo, Emmanuele Chersoni, Enrico Santus, Alessandro Lenci, and Philippe Blache. 2018. Event Knowledge in Sentence Processing: A New Dataset for the Evaluation of Argument Typicality. In Proceedings of the LREC Workshop on Linguistic and Neuro-Cognitive Resources (LiNCR). find
  49. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020. BLiMP: The Benchmark of Linguistic Minimal Pairs for English. Transactions of the Association for Computational Linguistics, 8:377– 392. find our card
  50. Hongming Zhang, Jiaxin Bai, Yan Song, Kun Xu, Changlong Yu, Yangqiu Song, Wilfred Ng, and Dong Yu. 2019a. Multiplex Word Embeddings for Selectional Preference Acquisition. In Proceedings of EMNLP-IJCNLP. find
  51. Hongming Zhang, Hantian Ding, and Yangqiu Song. 2019b. SP-10K: A Large-scale Evaluation Set for Selectional Preference Acquisition. In Proceedings of ACL. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/acl-2023-testa-we-understand-elliptical-sentences-language/