Causal Interventions Reveal Shared Structure Across English Filler–Gap Constructions

What it is about

The paper asks whether a language model uses one shared mechanism for several English filler–gap constructions (embedded wh-questions, matrix wh-questions, relative clauses, clefts, pseudo-clefts, topicalization). It uses Distributed Alignment Search on Pythia models (1.4B, 2.8B, 6.9B) to find a feature that marks the gap, trains it on one or more source constructions, and measures how much causal effect it has on other constructions with the ODDS metric. Transfer is strong across the filler–gap constructions, and it is stronger when the animacy of the extracted item matches. Transfer across clause boundaries is weak. A source-and-sink analysis shows that frequent constructions (relative clauses 504 and embedded wh-questions 308 in English-EWT) transfer well to rarer ones, and that similarity in filler type, inversion, and parent category predicts transfer. The study uses templated minimal pairs, 200 sentences per training set, and 400 test sentences.

Ideas

• I1. Count each construction's frequency in Mova's own parse of UD English EWT, and use the counts as a priority list for the grammar: the rule set for the frequent constructions gets tested first, the rare ones inherit from them. Measure the gap-resolution accuracy per construction in en parse-eval output against those counts.
• I2. Build a deterministic rule-sharing matrix. For each gap-filling rule in prag, record which constructions it fires on, as a BTreeMap of counts. Rules that fire on many constructions are the sources; those that fire on none are the sinks. The matrix is the rule-level analogue of the paper's transfer graph, and it needs no neural model.
• I3. Add the paper's controls to the gap test: a subject-verb agreement sentence with the same distance between subject and verb, and a transitive/intransitive continuation with the same predicted label. A gap rule that only counts distance or predicts a label will fire on both controls; a negative control that must score worse is a direct check.
• I4. Test cross-clause gap filling: a wh-filler in a matrix clause and a gap in an embedded clause, with animacy and filler type varied. Mova should find the gap across a finite clause only when the rule allows it. Measure the same four-way split as the paper's Table 2 (filler, inversion, embed, front).

For Mova

The chapters are 25. Relative clauses, 26. Questions (matrix and embedded wh-questions), 23. Complementation (wonder/know classes) and 29. Ellipsis, word order and information structure (clefts, topicalization). The code is prag/src/roles.rs and prag/src/frames.rs for the gap and the fronted filler, en/src/parse.rs for the trees, and world/src/qa.rs for the answer to the question. The measure is per-construction gap accuracy with the control sentences, run twice to check it is identical.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.1271 · arXiv 2505.16002

@inproceedings{boguraev2025causal,
  title = {{Causal Interventions Reveal Shared Structure Across English Filler–Gap Constructions}},
  author = {Boguraev, Sasha and Potts, Christopher and Mahowald, Kyle},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1271},
  eprint = {2505.16002},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.1271}
}

References

44 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aryaman Arora, Dan Jurafsky, and Christopher Potts. 2024. CausalGym: Benchmarking causal interpretability methods on linguistic tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14638–14663, Bangkok, Thailand. Association for Computational Linguistics. ↗
2. Dale J. Barr, Roger Levy, Christoph Scheepers, and Harry J. Tily. 2013. Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3):255–278. ↗
3. Debasmita Bhattacharya and Marten van Schijndel. 2020. Filler-gaps that neural networks fail to generalize. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 486–495, Online. Association for Computational Linguistics. ↗
4. Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling. Preprint, arXiv:2304.01373. ↗
5. Cameron J Buckner. 2024. From deep learning to rational machines: What the history of philosophy can teach us about the future of artificial intelligence. Oxford University Press. ↗
6. Noam Chomsky. 1957. Syntactic Structures. Walter de Gruyter. find
7. Peter W. Culicover. 1999. Syntactic Nuts: Hard Cases, Syntactic Theory, and Language Acquisition. Oxford University Press, Oxford. ↗
8. Marie-Catherine De Marneffe, Christopher D Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal dependencies. Computational Linguistics, 47(2):255– 308. find our card
9. Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. 2021. Causal analysis of syntactic agreement mechanisms in neural language models. ↗
10. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1828–1843, Online. Association for Computational Linguistics. find
11. Janet Dean Fodor. 1989. Empty categories in sentence processing. Language and Cognitive Processes, 4(3-4):SI155–SI209. ↗
12. Richard Futrell and Kyle Mahowald. 2025. How linguistics learned to stop worrying and love the language models. Behavioral and Brain Sciences, page 1–98. ↗
13. Richard Futrell, Peng Qian, Edward Gibson, Evelina Fedorenko, and Idan Blank. 2019. Syntactic dependencies correspond to word pairs with high mutual information. In Proceedings of the Fifth International Conference on Dependency Linguistics (Depling, SyntaxFest 2019), pages 3–13, Paris, France. Association for Computational Linguistics. ↗
14. Atticus Geiger, Duligur Ibeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, et al. 2023. Causal abstraction: A theoretical foundation for mechanistic interpretability. arXiv preprint arXiv:2301.04709. ↗
15. Alexandra Kuznetsova, Per B Brockhoff, and Rune HB Christensen. 2017. lmertest package: tests in linear mixed effects models. Journal of statistical software, 82:1–26. ↗
16. Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. 2019. The emergence of number and syntax units in LSTM language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 11–20, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
17. Nur Lan, Emmanuel Chemla, and Roni Katzir. 2024. Large language models and the argument from the poverty of the stimulus. Linguistic Inquiry, pages 1–56. ↗
18. Karim Lasri, Tiago Pimentel, Alessandro Lenci, Thierry Poibeau, and Ryan Cotterell. 2022. Probing for the usage of grammatical number. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8818–8831, Dublin, Ireland. Association for Computational Linguistics. ↗
19. Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. 2021. Causal abstractions of neural networks. In Advances in Neural Information Processing Systems, volume 34, pages 9574–9586. find
20. Christopher D Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. 2020. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, 117(48):30046–30054. ↗
21. Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah Goodman. 2024. Finding alignments between interpretable causal variables and distributed neural representations. In Causal Learning and Reasoning, pages 160–187. PMLR. find
22. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. In Advances in Neural Information Processing Systems, volume 35, pages 17359–17372. Curran Associates, Inc. ↗
23. Jonathan Ginzburg and Ivan A. Sag. 2001. Interrogative Investigations: The Form, Meaning, and Use of English Interrogatives. CSLI, Stanford, CA. find
24. Aaron Mueller, Yu Xia, and Tal Linzen. 2022. Causal analysis of syntactic agreement neurons in multilingual language models. In Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pages 95–109, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. ↗
25. Katherine Howitt, Sathvik Nair, Allison Dods, and Robert Melvin Hopkins. 2024. Generalizations across filler-gap dependencies in neural language models. In Proceedings of the 28th Conference on Computational Natural Language Learning, pages 269–279, Miami, FL, USA. Association for Computational Linguistics. ↗
26. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744, Online. Association for Computational Linguistics. ↗
27. Anastasia Kobzeva, Suhas Arehalli, Tal Linzen, and Dave Kush. 2023. Neural networks can learn patterns of island-insensitivity in Norwegian. In Proceedings of the Society for Computation in Linguistics 2023, pages 175–185, Amherst, MA. Association for Computational Linguistics. ↗
28. Ryan M Nefdt. 2023. Language, Science, and Structure: A Journey into the Philosophy of Linguistics. Oxford University Press. ↗
29. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4034–4043, Marseille, France. European Language Resources Association. find our card
30. Satoru Ozaki, Dan Yurovsky, and Lori Levin. 2022. How well do lstm language models learn filler-gap dependencies? In Proceedings of the Society for Computation in Linguistics 2022, pages 76–88. find
31. Steven T. Piantadosi. 2024. Modern language models refute chomsky’s approach to language. In Edward Gibson and Moshe Poliak, editors, From fieldwork to linguistic theory: A tribute to Dan Everett (Empirically Oriented Theoretical Morphology and Syntax 15), pages 353—414. Berlin: Language Science Press. find
32. Martin J Pickering and Holly P Branigan. 1998. The representation of verbs: Evidence from syntactic priming in language production. Journal of Memory and Language, 39(4):633–651. ↗
33. Christopher Potts. 2023. Characterizing english preposing in pp constructions. Journal of Linguistics, pages 1–39. ↗
34. Grusha Prasad, Marten van Schijndel, and Tal Linzen. 2019. Using priming to uncover the organization of syntactic representations in neural language models. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 66–76, Hong Kong, China. Association for Computational Linguistics. ↗
35. Juan Diego Rodriguez, Aaron Mueller, and Kanishka Misra. 2025. Characterizing the role of similarity in the property inferences of language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 11515–11533, Albuquerque, New Mexico. Association for Computational Linguistics. ↗
36. John Robert Ross. 1967. Constraints on Variables in Syntax. Ph.D. thesis, MIT, Cambridge, MA. find
37. Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, and Chris Manning. 2014. A gold standard dependency corpus for English. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 2897– 2904, Reykjavik, Iceland. European Language Resources Association (ELRA). ↗
38. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Causal mediation analysis for interpreting neural nlp: The case of gender bias. Preprint, arXiv:2004.12265. ↗
39. Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2023. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations. find
40. Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018. What do RNN language models learn about filler–gap dependencies? In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 211–221, Brussels, Belgium. Association for Computational Linguistics. ↗
41. Ethan Gotlieb Wilcox, Richard Futrell, and Roger Levy. 2024. Using computational models to test syntactic learnability. Linguistic Inquiry, 55(4):805–848. ↗
42. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Huggingface’s transformers: State-of-the-art natural language processing. Preprint, arXiv:1910.03771. ↗
43. Zhengxuan Wu, Atticus Geiger, Aryaman Arora, Jing Huang, Zheng Wang, Noah Goodman, Christopher Manning, and Christopher Potts. 2024. pyvene: A library for understanding and improving PyTorch models via interventions. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: System Demonstrations), pages 158–165, Mexico City, Mexico. Association for Computational Linguistics. ↗
44. Zhengxuan Wu, Atticus Geiger, Thomas Icard, Christopher Potts, and Noah Goodman. 2023. Interpretability at scale: Identifying causal mechanisms in Alpaca. In Advances in Neural Information Processing Systems, volume 36, pages 78205–78226. Curran Associates, Inc. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/boguraev-2025-causal-interventions-reveal-shared-structure/