Can language models capture syntactic associations without surface cues? A case study of reflexive anaphor licensing in English control constructions

What it is about

In "The lawyer promised the artist to make fun of himself", the reflexive himself must agree with the subject (the controller), while in "The lawyer persuaded the artist to make fun of himself" it must agree with the object. The infinitive carries no agreement, so only the reflexive reveals the controller, and such sentences are almost absent from corpora. The authors test GPT-2 (small, 117M parameters, and XL, 1.5B) on minimal pairs built from 5 noun phrases, 3 subject-control verbs (promise, offer, guarantee), 3 object-control verbs (persuade, tell, force) and 5 embedded clauses, with 100 pairs per verb. Object control is near perfect (GPT-2 small 100%, XL 97%). Subject control with an intervening object is poor: 4% for small and 23% for XL, below the 33% chance level. Without the object, both models reach 100%. The authors read this as the model linking the reflexive to the nearest noun phrase. COCA, over 1 billion words, has one example with promise and a controller-agreeing reflexive.

Ideas

• I1. Controller rule in the lexicon: store for each control verb a flag, subject-to-subject or object-to-subject (the valency types of the Osborne 2017 account), in en/src/dict.rs as an enum. Resolve the reflexive in a non-finite clause to that controller. Measure accuracy on minimal pairs generated from the paper's design, and use a negative control with the flags swapped, which must drop to zero.
• I2. Linear-proximity baseline as a required control: in every reflexive test, also report the accuracy of the "nearest noun phrase" rule. The paper shows the nearest-noun rule is what fails, so any Mova coreference rule must beat it on the subject-control items.
• I3. Template generator for the test set: the paper's 5 nouns, 6 verbs and 5 clauses, singular and plural, with a distractor noun, can be generated in tools/ as data sorted in a BTreeMap. This gives a held-out set for coref/src/sieve.rs that needs no corpus.
• I4. Number agreement as a separate measure: the reflexive takes the number of its controller, so check the plural and singular combinations as a separate column, as the paper does with its distractor condition.
• I5. Keep the lexical fact in the lexicon, not in a learned model: the paper's finding that promise and persuade differ in meaning, not in surface form, is an argument for explicit lexical flags over a statistical learner.

For Mova

Control and reflexives: 23. Complementation (control verbs), 12. Pronouns, determiners and numerals (reflexive forms and number), 35. Reference and coreference (antecedent choice). Code: en/src/dict.rs (control flags), coref/src/sieve.rs and coref/src/mention.rs (controller resolution), coref/src/eval.rs (accuracy and F1 on the generated pairs). Measure: accuracy on the generated minimal pairs by control type, with the nearest-noun baseline and the flag-swap negative control, run twice to check determinism.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{lee2022can,
  title = {{Can language models capture syntactic associations without surface cues? A case study of reflexive anaphor licensing in English control constructions}},
  author = {Lee, Soo-Hwan and Schuster, Sebastian},
  booktitle = {Proceedings of the Society for Computation in Linguistics 2022},
  year = {2022},
  url = {https://aclanthology.org/2022.scil-1.18/}
}

References

15 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5185–5198, Online. Association for Computational Linguistics. find
2. Karen Courtenay. 1998. Summary: Subject control verb PROMISE in English. https://linguistlist.org/issues/9/9-651/. ↗
3. Mark Davies. 2008. The corpus of contemporary American English: 450 million words, 1990-present. find
4. Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 32–42, Minneapolis, Minnesota. Association for Computational Linguistics. find
5. Yoav Goldberg. 2019. Assessing BERT’s syntactic abilities. arXiv preprint, abs/1901.05287. ↗
6. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1195–1205, New Orleans, Louisiana. Association for Computational Linguistics. find
7. Jennifer Hu, Sherry Yong Chen, and Roger Levy. 2020a. A closer look at the performance of neural language models on reflexive anaphor licensing. In Proceedings of the Society for Computation in Linguistics 2020, pages 323–333, New York, New York. Association for Computational Linguistics. find
8. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020b. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744, Online. Association for Computational Linguistics. find
9. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of lstms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521–535. find
10. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics. find
11. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog. find
12. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computational Linguistics. find
13. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377–392. find our card
14. Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018. What do RNN language models learn about filler–gap dependencies? In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 211–221, Brussels, Belgium. Association for Computational Linguistics. ↗
15. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lee-2022-can-language-models-capture-syntactic/