Mova0.5 pre-release⌕
☰
home / docs / papers / Assessing the influence of attractor-verb distance on grammatical agreement in humans and language models

Assessing the influence of attractor-verb distance on grammatical agreement in humans and language models

Summary

The study asks how the distance between an attractor noun and the verb affects subject–verb number agreement in people and in neural language models. An attractor is an intervening noun whose number may differ from the subject's. The authors construct English sentences of fixed length in which the attractor is either right before the verb or further away, plus a baseline with no intervening noun. Humans judged grammaticality of sentences presented word by word, and the same stimuli were given to two transformer models. Both humans and models err more when the attractor is close to the verb, but in the hardest conditions models fall to near chance while humans mostly overcome the interference. The authors attribute the proximity effect to local transition probabilities between neighbouring words, while acknowledging that cue-based memory retrieval could also explain it. Superficially similar behaviour thus hides different processing of syntax.

How Mova uses it

Effectiveness in Mova

Not measured separately. Background reading for two rules. Against the reference treebanks: attraction through of in EWT 2.18 — verbs 7 cases / 1 violation (exactly "pattern … indicate"), copula or auxiliary 13 / 5; GUM 16 / 3 and 27 / 8.

👨‍🔬💥

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.emnlp-main.998 · arXiv 2311.16978

@inproceedings{zacharopoulos2023assessing,
  title = {{Assessing the influence of attractor-verb distance on grammatical agreement in humans and language models}},
  author = {Zacharopoulos, Christos and Desbordes, Théo and Sablé-Meyer, Mathias},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.998},
  eprint = {2311.16978},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.998}
}

References

37 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sophie L Arana, Jan-Mathijs Schoffelen, Tom Mitchell, and Peter Hagoort. 2021. Mvpa does not reveal neural representations of hierarchical linguistic structure in meg. bioRxiv. find
  2. Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. If you use this software, please cite it using these metadata. find
  3. Kathryn Bock and Carol A Miller. 1991. Broken agreement. Cognitive psychology, 23(1):45–93. find
  4. Jonathan R. Brennan, Edward P. Stabler, Sarah E. Van Wagenen, Wen-Ming Luh, and John T. Hale. Abstract linguistic structure correlates with temporal activity during naturalistic comprehension. 157-158:81–94. find
  5. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. find
  6. Noam Chomsky. 1957. Syntactic structures. Mouton publishers. Accepted: 2019-04-10T06:25:21Z Journal Abbreviation: Janua linguarum. find
  7. Cas W Coopmans, Helen De Hoop, Karthikeya Kaushik, Peter Hagoort, and Andrea E Martin. 2021. Structure-(in) dependent interpretation of phrases in humans and lstms. In The Society for Computation in Linguistics (SCiL 2021), pages 459–463. find
  8. Stanislas Dehaene, Florent Meyniel, Catherine Wacon-gne, Liping Wang, and Christophe Pallier. 2015. The neural representation of sequences: from transition probabilities to algebraic patterns and linguistic trees. Neuron, 88(1):2–19. find
  9. Théo Desbordes, Yair Lakretz, Valérie Chanoine, Maxime Oquab, Jean-Michel Badier, Agnès Trébu-chon, Romain Carron, Christian-G. Bénar, Stanislas Dehaene, and Jean-Rémi King. 2023. Dimensionality and ramping: Signatures of sentence integration in the dynamics of brains and deep language models. Journ… find
  10. Nai Ding, Lucia Melloni, Hang Zhang, Xing Tian, and David Poeppel. 2015. Cortical tracking of hierarchical linguistic structures in connected speech. Nature Neuroscience, 19(1):158–164. find
  11. Victoria Fossum and Roger Levy. 2012. Sequential vs. hierarchical syntactic models of human incremental sentence processing. In Proceedings of the 3rd Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2012), pages 61–69, Montréal, Canada. Association for Computational Linguistics. find
  12. Julie Franck, Gabriella Vigliocco, and Janet Nicol. 2002. Subject-verb agreement errors in french and english: The role of syntactic hierarchy. Language and cognitive processes, 17(4):371–404. find
  13. Stefan L Frank and Rens Bod. 2011. Insensitivity of the human sentence-processing system to hierarchical structure. Psychological science, 22(6):829–834. find
  14. Karl Friston, Rosalyn J. Moran, Yukie Nagai, Tadahiro Taniguchi, Hiroaki Gomi, and Josh Tenenbaum. 2021. World model learning and inference. 144:573–590. find
  15. A. Hahne and J. D. Jescheniak. 2001. What’s left if the Jabberwock gets the semantics? An ERP investigation into semantic and syntactic processes during auditory sentence comprehension. Brain Research. Cognitive Brain Research, 11(2):199–212. find
  16. Christopher Hammerly, Adrian Staub, and Brian Dillon. 2019. The grammaticality asymmetry in agreement attraction reflects response bias: Experimental and modeling evidence. Cognitive psychology, 110:70– 104. find
  17. Todd R Haskell and Maryellen C MacDonald. 2005. Constituent structure and linear order in language production: evidence from subject-verb agreement. Journal of Experimental Psychology: Learning, Memory, and Cognition, 31(5):891. find
  18. Marc D Hauser, Noam Chomsky, and W Tecumseh Fitch. 2002. The faculty of language: what is it, who has it, and how did it evolve? science, 298(5598):1569–1579. find
  19. Jaap Jumelet, Willem Zuidema, and Dieuwke Hupkes. 2019. Analysing neural language models: Contextual decomposition reveals default reasoning in number and gender assignment. arXiv preprint arXiv:1909.08975. find
  20. Yair Lakretz, Théo Desbordes, Jean-Rémi King, Benoît Crabbé, Maxime Oquab, and Stanislas Dehaene. 2021. Can rnns learn recursive nested subject-verb agreements? find
  21. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of lstms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521–535. find
  22. R Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. 2021. How much do language models copy from their training data? evaluating linguistic novelty in text generation using raven. arXiv preprint arXiv:2111.09509. find
  23. Brian McElree, Stephani Foraker, and Lisbeth Dyer. 2003. Memory structures that subserve sentence comprehension. Journal of memory and language, 48(1):67–91. find
  24. Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Nar-jes Nikzad, Meysam Chenaghlu, and Jianfeng Gao. 2021. Deep learning–based text classification: a comprehensive review. ACM Computing Surveys (CSUR), 54(3):1–40. find
  25. Nicola Molinaro, Horacio A Barber, and Manuel Carreiras. 2011. Grammatical agreement processing in reading: Erp findings and future directions. cortex, 47(8):908–930. find
  26. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… find
  27. Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2017. Jfleg: A fluency corpus and benchmark for grammatical error correction. find
  28. Matthew J Nelson, Imen El Karoui, Kristof Giber, Xiaofang Yang, Laurent Cohen, Hilda Koopman, Sydney S Cash, Lionel Naccache, John T Hale, Christophe Pallier, et al. 2017. Neurophysiologi-cal dynamics of phrase-structure building during sentence processing. Proceedings of the National Academy of Sci… find
  29. Christophe Pallier, Anne-Dominique Devauchelle, and Stanislas Dehaene. 2011. Cortical representation of the constituent structure of sentences. Proceedings of the National Academy of Sciences, 108(6):2522– 2527. find
  30. Neal J Pearlmutter, Susan M Garnsey, and Kathryn Bock. 1999. Agreement processes in sentence comprehension. Journal of Memory and language, 41(3):427–456. find
  31. Colin Phillips, Matthew W Wagers, and Ellen F Lau. 2011. Grammatical illusions and selective fallibility in real-time language comprehension. Experiments at the Interfaces, 37:147–180. find
  32. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. find
  33. Luigi Rizzi. 2004. On the cartography of syntactic structures. The Structure of CP and IP, pages 3–15. find
  34. Rushen Shi, Camille Legrand, and Anna Brandenberger. 2020. Toddlers track hierarchical structure dependence. Language Acquisition, 27(4):397–409. find
  35. Gabriella Vigliocco and Janet Nicol. 1998. Separating hierarchical relations and word order in language production: Is proximity concord syntactic or linear? Cognition, 68(1):B13–B29. find
  36. Matthew W Wagers, Ellen F Lau, and Colin Phillips. 2009. Agreement attraction in comprehension: Representations and processes. Journal of memory and language, 61(2):206–237. find
  37. Jana Willer Gold, Boban Arsenijević, Mia Batinić, Michael Becker, Nermina Čordalija, Marijana Kre-sić, Nedžad Leko, Franc Lanko Marušič, Tanja Mil-ićev, Nataša Milićević, et al. 2017. When linearity prevails over hierarchy in syntax. pnas. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/zacharopoulos-2023-assessing-influence-attractor-verb-distance/