Mova0.5 pre-release⌕
☰
home / docs / papers / Subject-verb agreement with Seq2Seq transformers: Bigger is better, but still not best

Subject-verb agreement with Seq2Seq transformers: Bigger is better, but still not best

Summary

The paper studies what determines whether T5-style sequence-to-sequence transformers get English subject–verb agreement right, where the correct verb form depends on structure rather than on the nearest noun. Rather than comparing token probabilities, models are fine-tuned on a tense-change task: given a past-tense sentence, where number is not visible on the verb, they rewrite it in the present tense. Fine-tuning uses about a thousand sentence pairs from Wikipedia, while test sentences are generated by a PCFG and contain distractor nouns inside prepositional phrases and relative clauses; teacher forcing makes the model choose between the two verb forms. Models of different size and shape, and models pre-trained on corpora of different type and size, are compared. Larger models and larger corpora generally make fewer errors, but no model reproduces the human error pattern: models are affected more by linear than by structural proximity of the distractor. Scale alone therefore does not guarantee hierarchical generalization.

How Mova uses it

Effectiveness in Mova

Not measured separately. Background reading for two rules. Checked against the EWT 2.18 reference treebank: plural subject with VBZ — verbs 1166 cases / 7 violations, copula or auxiliary 2528 / 8; singular subject with VBP — verbs 54 / 22, copula or auxiliary 129 / 37 (most violations are genuine errors in the source text).

👨‍🔬💥

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{wilson2023subject,
  title = {{Subject-verb agreement with Seq2Seq transformers: Bigger is better, but still not best}},
  author = {Wilson, Michael and Zhou, Zhenghao and Frank, Robert},
  booktitle = {Proceedings of the Society for Computation in Linguistics 2023},
  year = {2023},
  url = {https://aclanthology.org/2023.scil-1.24/}
}

References

26 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, and Adriane Boyd. 2020. spaCy: Industrial-strength natural language processing in Python. find
  2. Suhas Arehalli and Tal Linzen. 2020. Neural language models capture some, but not all, agreement attraction effects. In Proceedings of the 42nd Annual Meeting of the Cognitive Science Society, pages 370–376. find
  3. Alexandra Kuznetsova, Per B. Brockhoff, and Rune H. B. Christensen. 2017. lmerTest package: Tests in linear mixed effects models. Journal of Statistical Software, 82(13):1–26. find
  4. Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1):1– 48. find
  5. Kathryn Bock and J. Cooper Cutting. 1992. Regulating mental energy: Performance units in language production. Journal of Memory and Language, 31:99– 127. find
  6. Jonathan R. Brennan, Chris Dyer, Adhiguna Kuncoro, and John T. Hale. 2020. Localizing syntactic predictions using recurrent neural network grammars. Neuropsychologia, 146. find
  7. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. find
  9. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language P… find
  10. Julie Franck, Gabriella Vigliocco, and Janet Nicol. 2002. Subject-verb agreement errors in French and English: The role of syntactic hierarchy. Language and Cognitive Processes, 17(4):371–404. find
  11. Karim Lasri, Alessandro Lenci, and Thierry Poibeau. 2022. Does BERT really agree ? fine-grained analysis of lexical dependence on a syntactic task. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2309–2315, Dublin, Ireland. Association for Computational Linguistics. find
  12. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521–535. find
  13. Brian MacWhinney. 2000. The CHILDES Project: Tools for Analyzing Talk, third edition, volume II: The Database. Lawrence Erlbaum Associates, Mahwah, NJ. find
  14. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. CoRR, abs/1808.09031. find
  15. R. Thomas McCoy, Robert Frank, and Tal Linzen. 2020. Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networks. Transactions of the Association for Computational Linguistics, 8:125–140. find
  16. Aaron Mueller, Robert Frank, Tal Linzen, Luheng Wang, and Sebastian Schuster. 2022. Coloring the blank slate: Pre-training imparts a hierarchical inductive bias to sequence-to-sequence models. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1352–1368, Dublin, Ireland. A… find
  17. Karl Mulligan, Robert Frank, and Tal Linzen. 2021. Structure here, bias there: Hierarchical generalization by jointly learning syntactic transformations. In Proceedings of the Society for Computation in Linguistics 2021, pages 125–135, Online. Association for Computational Linguistics. find
  18. Benjamin Newman, Kai-Siang Ang, Julia Gong, and John Hewitt. 2021. Refining targeted syntactic evaluation of language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3710–3723, Online. As… find
  19. OpenAI. 2023. Gpt-4 technical report. find
  20. Jackson Petty and Robert Frank. 2021. Transformers generalize linearly. CoRR, abs/2109.12036. — [arXiv:2109.12036](https://arxiv.org/abs/2109.12036) find
  21. R Core Team. 2022. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. find
  22. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683. find
  23. Tom De Smedt and Walter Daelemans. 2012. Pattern for python. Journal of Machine Learning Research, 13(66):2063–2067. find
  24. Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. 2021. Scale efficiently: Insights from pre-training and fine-tuning transformers. CoRR, abs/2109.10686. — [arXiv:2109.10686](https://arxiv.org/abs/2109.10686) find
  25. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. find
  26. Ethan Wilcox, Pranali Vani, and Roger Levy. 2021. A targeted assessment of incremental processing in neural language models and humans. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processin… find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/wilson-2023-subject-verb-agreement-seq2seq-transformers/