Are we there yet? Encoder-decoder neural networks as cognitive models of English past tense inflection

What it is about

Kirov and Cotterell (2018) reported that an encoder-decoder network with attention fits human nonce-word past tenses better than an older rule model. The authors re-run that model on the Albright and Hayes (2003) nonce-verb production data, with 20 random seeds and two training sets (A&H's 4,253 CELEX verbs and K&C's 4,039). Training accuracy is near perfect. Correlation with human production probabilities, however, ranges from 0.15 to 0.56 for regular forms across seeds, and complete recall@5 is 39% to 47% on the A&H data, so single runs are unstable. Averaging 50 simulations as if each were one participant helps regulars (0.45) but not irregulars (0.19). Over eight measures, the A&H rule model fits the human data best on six. Lower training epochs raise correlation, but drop accuracy on real irregular verbs to 6.5%.

Ideas

• I1. Multi-seed reporting in morph.rs-style experiments: any Mova rule-set that is tuned or sampled must report the spread over runs, not one number. This is the paper's main lesson, and it applies to the rule learner as much as to a network.
• I2. Implement an A&H-style rule learner in Rust: each suffix rule gets a confidence from its precision and its number of applicable cases, and a verb takes the rule with the highest confidence. This is a count-based model that can say which rule fired and why, which suits the transparency aim.
• I3. Test the "islands of reliability" idea as a measure: group nonce or unseen verbs by whether a high-confidence rule applies, and report accuracy per group, as the paper does with its four categories.
• I4. Negative control: train on verbs with their past forms shuffled across stems. The rule learner must fall to chance on the regular pattern, or the test is not measuring generalisation.

For Mova

The chapter is 14. Irregular verbs, with 13. Verbs for the regular rule. The code is en/src/morph.rs (inflection and morph::archaic). Measure: exact-match accuracy of the past tense on held-out verbs, with the spread over repeated runs. The human nonce-word data are used only for evaluation, and only if their licence allows it; check before use. The neural model itself is not for Mova.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1376 · arXiv 1906.01280

@inproceedings{corkery2019are,
  title = {{Are we there yet? Encoder-decoder neural networks as cognitive models of English past tense inflection}},
  author = {Corkery, Maria and Matusevych, Yevgen and Goldwater, Sharon},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1376},
  eprint = {1906.01280},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p19-1376}
}

References

30 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Adam Albright and Bruce Hayes. 2003. Rules vs. analogy in English past tenses: a computational/experimental study. Cognition, 90:119–161. find
2. Ben Ambridge. 2010. Children’s judgments of regular and irregular novel past-tense forms: New data on the English past-tense debate. Developmental Psychology, 46:1497–1504. find
3. R. Harald Baayen, Richard Piepenbrock, and Leon Gulikers. 1995. CELEX2 LDC96L14. Web Download. Linguistic Data Consortium, Philadelphia, PA. find
4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA. find
5. William Bechtel and Adele Abrahamsen. 1991. Connectionism and the mind: An introduction to parallel processing in networks. Basil Blackwell, Oxford, England. find
6. Toms Bergmanis and Sharon Goldwater. 2018. Context sensitive neural lemmatization with Lematus. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1391–1400. Association for Computational Linguistics. find
7. Ryan P. Blything, Ben Ambridge, and Elena V.M. Lieven. 2018. Children’s acquisition of the english past-tense: Evidence for a single-route account from novel verb production data. Cognitive Science, 42:621–639. find
8. Joan Bybee and Sandra Thompson. 1997. Three frequency effects in syntax. In Proceedings of the 23rd Annual Meeting of the Berkeley Linguistics Society, pages 378–388. Berkeley Linguistics Society, Berkeley, CA. find
9. Harald Clahsen, Monika Rothweiler, Andreas Woest, and Gary F. Marcus. 1992. Regular and irregular inflection in the acquisition of German noun plurals. Cognition, 45:225–255. find
10. Jeffrey Elman, Elizabeth Bates, Mark H. Johnson, Anette Karmiloff-Smith, Domenico Parisi, and Kim Plunkett. 1996. Rethinking innateness: A connectionist perspective on development. MIT Press, Cambridge, MA. find
11. Mirjam Ernestus and R. Harald Baayen. 2004. Analogical effects in regular past tense production in Dutch. Linguistics, 42:873–903. find
12. Sharon Goldwater, Thomas L. Griffiths, and Mark Johnson. 2006. Interpolating between types and tokens by estimating power-law generators. In Advances in NIPS-18, pages 459–466. Curran Associates, Inc., Red Hook, NY. find
13. Katharina Kann and Hinrich Schütze. 2016. MED: The LMU system for the SIGMORPHON 2016 shared task on morphological reinflection. In Proceedings of the 14th Annual SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 62–70. Association for Computational Linguistics, Stroudsburg, PA. find
14. Christo Kirov and Ryan Cotterell. 2018. Recurrent neural networks in linguistic theory: Revisiting Pinker and Prince (1988) and the past tense debate. Transactions of the Association for Computational Linguistics, 6:651–665. find
15. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. OpenNMT: Open-source toolkit for neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, System Demonstrations, pages 67–72. Association for Computational Linguistics. find
16. Tal Linzen. 2019. What can linguistics and deep learning contribute to each other? Response to Pater. Language, 95:99–108. find
17. Mark S. Seidenberg and David C. Plaut. 2014. Quasiregularity and its discontents: The legacy of the past tense debate. Cognitive Science, 38:1190– 1228. find
18. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, pages 3104–3112. Curran Associates, Inc., Red Hook, NY. find
19. Matthew D. Zeiler. 2012. ADADELTA: An adaptive learning rate method. Computing Research Repository, arXiv:1212.5701. ↗
20. Tal Linzen and Brian Leonard. 2018. Distinct patterns of syntactic agreement errors in recurrent networks and humans. In Proceedings of the 40th Annual Conference of the Cognitive Science Society, pages 692– 697. Cognitive Science Society, Austin, TX. find
21. Gary F. Marcus. 1995. The acquisition of the English past tense in children and multilayered connectionist networks. Cognition, 56:271–279. find
22. Gary F. Marcus. 1998. Can connectionism save constructivism? Cognition, 66:153–182. find
23. Michael McCloskey. 1991. Networks and theories: The place of connectionism in cognitive science. Psychological Science, 2:387–395. find
24. Timothy J. O’Donnell. 2015. Productivity and reuse in language: A theory of linguistic computation and storage. MIT Press, Cambridge, MA. find
25. Janet Pierrehumbert. 2001. Stochastic phonology. Glot International, 5:195–207. find
26. Steven Pinker and Alan Prince. 1988. On language and connectionism: Analysis of a parallel distributed processing model of language acquisition. Cognition, 28:73–193. find
27. Steven Pinker and Michael T. Ullman. 2002. The past and future of the past tense. Trends in Cognitive Sciences, 6:456–463. find
28. Kim Plunkett and Patrick Juola. 1999. A connectionist model of English past tense and plural morphology. Cognitive Science, 23:463–490. find
29. Sandeep Prasada and Steven Pinker. 1993. Generalisation of regular and irregular morphological patterns. Language and Cognitive Processes, 8:1–56. find
30. David E. Rumelhart and James L. McClelland. 1986. On learning the past tenses of English verbs. In James L. McClelland, David E. Rumelhart, and the PDP Research Group, editors, Parallel distributed processing: Explorations in the microstructure of cognition, chapter 18, pages 216–271. MIT Press, Cambridge, MA. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/corkery-2019-are-we-there-yet-encoder/