How do we get there? Evaluating transformer neural networks as cognitive models for English past tense inflection

What it is about

The paper asks whether a transformer learns the English past tense as symbolic categories (regular or irregular, and a suffix rule) or as pattern association. It uses 4,039 CELEX verbs (3,857 regular, 150 irregular, 32 ambiguous) and a set of training sets that vary type frequency (each verb once) and token frequency (each verb as often as in CELEX). It tests on 80 real verbs held out from training and on the 58 nonce verbs of Albright and Hayes (2003). Transformers reach over 95% accuracy on unseen regulars and 0% to 22% on unseen irregulars. Regular performance depends on type frequency and not token frequency, and irregular performance shows the reverse pattern, which the authors read as partial symbolic learning. Correlation with human nonce-word data is weak, so the authors conclude the model is not a good cognitive model.

Ideas

• I1. Turn the three hypotheses into predictions a rule learner can be checked against. The paper's H1 (symbolic learner: type frequency only), H2 (pattern associator: type and token) and H3 (hybrid) each predict a different split of errors. Measure the same split for en/src/morph.rs past-tense rules: misclassified regular versus wrong suffix.
• I2. Run the frequency manipulation on the rule learner, not on a network. Build training sets with type-only and token-weighted counts, and check that the learned regular rule changes only with type counts. This is a cheap, deterministic test of the paper's claim.
• I3. Balanced resampling per epoch (the paper's BALANCE set, 129 regulars and 129 irregulars) is a count-based idea: use it as the sampling rule in a decision list, so irregulars are not drowned by regulars. Report irregular accuracy with and without it.
• I4. Negative control: give the held-out real verbs a random suffix label. Any rule-based score must drop to chance on that split, or the test is not reading the verb form.

For Mova

The chapters are 13. Verbs (regular past tense) and 14. Irregular verbs (classes of irregulars). The code is en/src/morph.rs, and the test is the same 80-verb held-out split and 58 nonce verbs, used once per release only (the held-out rule of the project). The measure: accuracy split by regular and irregular, and the type-versus-token effect, run twice to confirm identical output. The transformer itself is not for Mova.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.aacl-main.81 · arXiv 2210.09167

@inproceedings{ma2022how,
  title = {{How do we get there? Evaluating transformer neural networks as cognitive models for English past tense inflection}},
  author = {Ma, Xiaomeng and Gao, Lingyu},
  booktitle = {Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.aacl-main.81},
  eprint = {2210.09167},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.aacl-main.81}
}

References

25 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Adam Albright and Bruce Hayes. 2003. Rules vs. analogy in english past tenses: A computational/experimental study. Cognition, 90(2):119– 161. find
2. R Harald Baayen, Richard Piepenbrock, and Leon Gulikers. 1995. The celex lexical database (release 2). Distributed by the Linguistic Data Consortium, University of Pennsylvania. find
3. Deniz Beser. 2021. Falling through the gaps: Neural architectures as models of morphological rule learning. arXiv preprint arXiv:2105.03710. ↗
4. Lois Bloom. 1973. One word at a time: The use of single word utterances before syntax, volume 154. Walter de Gruyter. find
5. Lois Bloom, Lois Hood, and Patsy Lightbown. 1974. Imitation in language development: If, when, and why. Cognitive psychology, 6(3):380–420. find
6. Joan Bybee. 2003. Phonology and language use, volume 94. Cambridge University Press. find
7. Maria Corkery, Yevgen Matusevych, and Sharon Goldwater. 2019. Are we there yet? encoder-decoder neural networks as cognitive models of english past tense inflection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3868–3877. find our card
8. Christine Cuskley, Francesca Colaiori, Claudio Castellano, Vittorio Loreto, Martina Pugliese, and Francesca Tria. 2015. The adoption of linguistic rules in native and non-native speakers: Evidence from a wug task. Journal of Memory and Language, 84:205–223. find
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
10. Roy Patrick Higginson. 1985. Fixing: Assimilation in language acquisition. Ph.D. thesis, Washington State University. find
11. Brian MacWhinney. 2000. The CHILDES project: The database, volume 2. Psychology Press. find
12. Gary F Marcus, Ursula Brinkmann, Harald Clahsen, Richard Wiese, and Steven Pinker. 1995. German inflection: The exception that proves the rule. Cognitive psychology, 29(3):189–256. find
13. Gary F Marcus, Steven Pinker, Michael Ullman, Michelle Hollander, T John Rosen, Fei Xu, and Harald Clahsen. 1992. Overregularization in language acquisition. Monographs of the society for research in child development, pages i–178. find
14. Kate McCurdy, Sharon Goldwater, and Adam Lopez. 2020. Inflecting when there’s no majority: limitations of encoder-decoder neural networks as cognitive models for german plurals. arXiv preprint arXiv:2005.08826. ↗
15. Steven Pinker and Alan Prince. 1988. On language and connectionism: Analysis of a parallel distributed processing model of language acquisition. Cognition, 28(1-2):73–193. David. find
16. E. Rumelhart and James L. McClelland. 1986. On Learning the Past Tenses of English Verbs, page 216–271. Cambridge, MA, USA. find
17. Jacqueline Sachs. 1983. Talking about the there and then: The emergence of displaced reference in parent-child discourse. Children’s language, 4:1–28. find
18. Assaf Singer and Katharina Kann. 2020. The nyu-cuboulder systems for sigmorphon 2020 task 0 and task 2. In Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 90–98. find
19. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30. find
20. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. Advances in neural information processing systems, 28. find
21. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
22. Jason Wei, Dan Garrette, Tal Linzen, and Ellie Pavlick. 2021. Frequency effects on syntactic rule learning in transformers. arXiv preprint arXiv:2109.07020. ↗
23. Christo Kirov and Ryan Cotterell. 2018. Recurrent neural networks in linguistic theory: Revisiting pinker and prince (1988) and the past tense debate. Transactions of the Association for Computational Linguistics, 6:651–665. find
24. Hao Zhang, Richard Sproat, Axel H Ng, Felix Stahlberg, Xiaochang Peng, Kyle Gorman, and Brian Roark. 2019. Neural models of text normalization for speech applications. Computational Linguistics, 45(2):293– 337. find
25. Wei Zhao, Liang Wang, Kewei Shen, Ruoyu Jia, and Jingming Liu. 2019. Improving grammatical error correction via pre-training a copy-augmented architecture with unlabeled data. In Proceedings of NAACL-HLT, pages 156–165. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/ma-2022-how-do-we-get-there/