Mova0.5 pre-release⌕
☰
home / docs / papers / English / Morphological Inflection: A Reality Check

Morphological Inflection: A Reality Check

What it is about

The paper examines how morphological inflection benchmarks are built and scored. It names three problems: uniform sampling by type creates unnatural train-test splits, a single split gives unstable model rankings, and train and test items overlap in uncontrolled ways. The overlap is split into four types of (lemma, feature set) pairs, and the test sets are grouped as featsAttested or featsNovel. Weighted sampling gives the highest mean accuracy for Arabic with large training (77.76%), and overlap-aware sampling the lowest (61.06%). For Arabic, the mean accuracy is 77.70% on featsAttested against 41.92% on featsNovel. Models generalise poorly to new feature sets. The authors argue that in fusional languages such as English, a feature set does not map directly onto the form, so generalising to a novel set is often hard.

Ideas

For Mova

The code is en/src/morph.rs, with en folds for the two-fold check and en morph-eval for scores. The chapters are 13. Verbs and 11-nouns.md. The paper's English data is UniMorph and CHILDES material, under NC or other restrictions, so use it for tests and measurements only and never in what is built. Measure first: the attested-versus-novel gap on the English split, with the cv spread stated next to it.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-long.335 · arXiv 2305.15637

@inproceedings{kodner2023morphological,
  title = {{Morphological Inflection: A Reality Check}},
  author = {Kodner, Jordan and Payne, Sarah and Khalifa, Salam and Liu, Zoey},
  booktitle = {ACL 2023 (Toronto)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.335},
  eprint = {2305.15637},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.acl-long.335}
}

References

93 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Javier Aguado-Orea and Julian M Pine. 2015. Comparing different models of the development of verb inflection in early child Spanish. PloS One, 10(3):e0119613. find
  2. Janet Bang and Aparna Nadig. 2015. Learning language in autism: Maternal linguistic input contributes to later vocabulary. Autism Research, 8(2):214–223. find
  3. Elizabeth Bates, Inge Bretherton, and Lynn Sebestyen Snyder. 1991. From first words to grammar: Individual differences and dissociable mechanisms, volume 20. Cambridge University Press. find
  4. Khuyagbaatar Batsuren, Omer Goldman, Salam Khalifa, Nizar Habash, Witold Kieraś, Gábor Bella, Brian Leonard, Garrett Nicolai, Kyle Gorman, Yustinus Ghanggo Ate, Maria Ryskina, Sabrina Mielke, Elena Budianskaya, Charbel El-Khaissi, Tiago Pimentel, Michael Gasser, William Abbott Lane, Mohit Raj, Matt… find our card
  5. Heike Behrens. 2006. The input–output relationship in first language acquisition. Language and cognitive processes, 21(1-3):2–24. find
  6. Emily M. Bender. 2009. Linguistically naïve != language independent: Why NLP needs linguistic typology. In Proceedings of the EACL 2009 Workshop on the Interaction between Linguistics and Computational Linguistics: Virtuous, Vicious or Vacuous?, pages 26–32, Athens, Greece. Association for Computati… find
  7. MM Berl, LM Balsamo, B Xu, EN Moore, SL Wein-stein, JA Conry, PL Pearl, BC Sachs, CB Grandin, C Frattali, et al. 2005. Seizure focus affects regional language networks assessed by fMRI. Neurology, 65(10):1604–1611. find
  8. Johannes Bjerva and Isabelle Augenstein. 2018. From phonology to syntax: Unsupervised linguistic typology at different levels with language embeddings. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V… find
  9. Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. 2022. Systematic inequalities in language technology performance across the world’s languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5486–5505, Dublin, Irelan… find
  10. Lynn Bliss. 1988. The development of modals. The journal of applied developmental psychology, 9:253– 261. find
  11. Lois Bloom, Lois Hood, and Patsy Lightbown. 1974. Imitation in language development: If, when, and why. Cognitive psychology, 6(3):380–420. find
  12. Lois Masket Bloom. 1970. Language development: Form and function in emerging grammars. Ph.D. thesis, Columbia University. find
  13. John Neil Bohannon III and Angela Lynn Marquis. 1977. Children’s control of adult speech. Child Development, pages 1002–1008. find
  14. Susan R Braunwald. 1971. Mother-child communication: the function of maternal-language input. Word, 27(1-3):28–50. find
  15. Michael R Brent and Jeffrey Mark Siskind. 2001. The role of exposure to isolated words in early vocabulary development. Cognition, 81(2):B33–B44. find
  16. Giuseppe Capelli, Victoria Marrero, and María José Al-bala. 1994. Aplicación del sistema morfo a una mues-tra de lenguaje infantil. Procesamiento del Lenguaje Natural, 14. find
  17. Erwin Chan. 2008. Structures and distributions in morphological learning. Ph.D. thesis, University of Pennsylvania, Philadelphia, PA. find
  18. Jane Chandlee. 2017. Computational locality in morphological maps. Morphology, 27(4):599–641. find
  19. Eve V Clark. 1978. Awareness of language: Some evidence from what children say and do. In The child’s conception of language, pages 17–43. Springer. find
  20. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Arya D. Mc-Carthy, Katharina Kann, Sabrina J. Mielke, Garrett Nicolai, Miikka Silfverberg, David Yarowsky, Jason Eisner, and Mans Hulden. 2018. The CoNLL– SIGMORPHON 2018 shared task: Universal morphological r… find
  21. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2017. find
  22. CoNLL-SIGMORPHON 2017 shared task: Universal morphological reinflection in 52 languages. In Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection, pages 1–30, Vancouver. Association for Computational Linguistics. find
  23. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016. find
  24. The SIGMORPHON 2016 shared Task— Morphological reinflection. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 10–22, Berlin, Germany. Association for Computational Linguistics. find
  25. Barbara L Davis and Peter F MacNeilage. 1995. The articulatory basis of babbling. Journal of Speech, Language, and Hearing Research, 38(6):1199–1211. find
  26. Martha Jo-Ann Demetras. 1986. Working Parents’ Conversational Responses to their two-year-old sons. The University of Arizona. find
  27. Roger Brown. 1973. A first language: The early stages. Harvard University Press, Cambridge, MA. find
  28. Marty Demetras. 1989. Changes in parents’ conversational responses: A function of grammatical development. ASHA, St. Louis, MO. find
  29. Joan L Bybee. 1991. Natural morphology: The organization of paradigms and language acquisition. Crosscurrents in second language acquisition and linguistic theories, 2:67–92. find
  30. Marty J Demetras, Kathryn Nolan Post, and Catherine E Snow. 1986. Feedback to first language learners: The role of repetitions and clarification questions. Journal of child language, 13(2):275–292. find
  31. Katherine Demuth, Jennifer Culbertson, and Jennifer Alter. 2006. Word-minimality, epenthesis and coda licensing in the early acquisition of English. Language and speech, 49(2):137–173. find
  32. MacArthur—Bates Communicative Development Inventory in identifying language abilities of late-talking and typically developing toddlers. American Journal of Speech-Language Patholog, 14:40–51. find
  33. David K Dickinson and Patton O Tabors. 2001. Beginning literacy with language: Young children learning at home and school. Paul H Brookes Publishing. find
  34. Roy Patrick Higginson. 1985. Fixing: Assimilation in language acquisition. Ph.D. thesis, Washington State University. find
  35. Micha Elsner, Andrea D Sims, Alexander Erdmann, Antonio Hernandez, Evan Jaffe, Lifeng Jin, Martha Booker Johnson, Shuan Karim, David L King, Luana Lamberti Nunes, et al. 2019. Modeling morphological learning, typology, and change: What can the neural sequence-to-sequence framework contribute? Journa… find
  36. Christo Kirov, Ryan Cotterell, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sabrina J. Mielke, Arya McCarthy, Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2018. UniMorph 2.0: Universal Morphology. In Proceedings of the Eleventh Internation… find
  37. Andrea Feldman. 1998. Constructing grammar: fillers, formulas, and function. Ph.D. thesis, University of Colorado at Boulder. find
  38. Viviana Fratini, Joana Acha, and Itziar Laka. 2014. Frequency and morphological irregularity are independent variables. Evidence from a corpus study of Spanish verbs. Corpus Linguistics and Linguistic Theory, 10(2):289–314. find
  39. Catherine Garvey and Robert Hogan. 1973. Social speech and social interaction: Egocentrism revisited. Child Development, pages 562–568. find
  40. Virginia C Gathercole. 1986. The acquisition of the present perfect: Explaining differences in the speech of Scottish and American children. Journal of Child Language, 13(3):537–560. find
  41. Susan A Gelman, John D Coley, Karl S Rosengren, Erin Hartman, Athina Pappas, and Frank C Keil. 1998. Beyond labeling: The role of maternal input in the acquisition of richly structured categories. Monographs of the Society for Research in Child development, pages i–157. find
  42. Ronald Bradley Gillam and Nils A Pearson. 2004. TNL: test of narrative language. Pro-ed Austin, TX. find
  43. Jean Berko Gleason. 1980. The acquisition of social speech routines and politeness formulas. In Language, pages 21–27. Elsevier. find
  44. Omer Goldman, David Guriel, and Reut Tsarfaty. 2022. (un)solving morphological inflection: Lemma overlap artificially inflates models’ performance. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 864–870, Dublin, Ireland. Ass… find
  45. William S Hall and William C Tirre. 1979. The communicative environment of young children: Social class, ethnic, and situational differences. Center for the Study of Reading Technical Report; no. 125. find
  46. John Heilmann, Susan Ellis Weismer, Julia Evans, and Christine Hollar. 2005. Utility of the Jordan Kodner. 2022. Computational Models of Morphological Learning. Oxford University Press. find
  47. Jordan Kodner, Salam Khalifa, Khuyagbaatar Batsuren, Hossep Dolatian, Ryan Cotterell, Faruk Akkus, Antonios Anastasopoulos, Taras Andrushko, Aryaman Arora, Nona Atanalov, Gábor Bella, Elena Budianskaya, Yustinus Ghanggo Ate, Omer Goldman, David Guriel, Simon Guriel, Silvia Guriel-Agiashvili, Witold… find
  48. Stan A Kuczaj II. 1977. The acquisition of regular and irregular past tense forms. Journal of verbal learning and verbal behavior, 16(5):589–600. find
  49. Constantine Lignos and Charles Yang. 2018. Morphology and language acquisition. Cambridge handbook of morphology, pages 765–791. find
  50. Josetxu Linaza, María Eugenia Sebastián, and Cristina del Barrio. 1981. Lenguaje, comunicación y com-prensión: Conferencia a nual de la sección de psi-cología del desarrollo de la british psychological society. Infancia y Aprendizaje, 4(sup1):195–197. find
  51. Zoey Liu and Emily Prud’hommeaux. 2022. Data-driven model generalizability in crosslinguistic low-resource morphological segmentation. Transactions of the Association for Computational Linguistics, 10:393–413. find
  52. Zoey Liu, Crystal Richardson, Richard Hatcher, and Emily Prud’hommeaux. 2022. Not always about you: Prioritizing community needs when developing endangered language technology. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3… find
  53. Colleen E Morisset, Kathryn E Barnard, and Cathryn L Booth. 1995. Toddlers’ language development: Sex differences within social risk. Developmental Psychology, 31(5):851. find
  54. Susana López Ornat. 1997. What lies in between a pre-grammatical and a grammatical representation? Evidence on nominal and verbal form-function mappings in Spanish from 1; 7 to 2; 1. Contemporary perspectives on the acquisition of Spanish, 1:3–20. find
  55. Katherine Nelson. 2006. Narratives from the crib. Harvard University Press. find
  56. Mohamed Maamouri, Ann Bies, Tim Buckwalter, and Wigdan Mekki. 2004. The Penn Arabic Treebank: Building a large-scale annotated Arabic corpus. In NEMLAR conference on Arabic language resources and tools, volume 27, pages 466–467. Cairo. find
  57. Brian MacWhinney. 1991. The CHILDES language project: Tools for analyzing talk. Journal of Speech, Language and Hearing Research, 40:62–74. find
  58. Rochelle S Newman, Meredith L Rowe, and Nan Bernstein Ratner. 2016. Input and uptake at 7 months predicts toddler vocabulary: The role of child-directed speech and infant processing skills in language development. Journal of child language, 43(5):1158– 1173. find
  59. Johanna G Nicholas and Ann E Geers. 1997. Communication of oral deaf and normally hearing children at 36 months of age. Journal of Speech, Language, and Hearing Research, 40(6):1314–1327. find
  60. Brian MacWhinney. 2000. The CHILDES Project: The Database, volume 2. Psychology Press, Abingdon-on-Thames. find
  61. Anat Ninio, Catherine E Snow, Barbara A Pan, and Pamela R Rollins. 1994. Classifying communicative acts in children’s interactions. Journal of communication disorders, 27(2):157–187. find
  62. Brian MacWhinney and Catherine Snow. 1985. The child language data exchange system. Journal of Child Language, 12(2):271–295. find
  63. Kemal Oflazer and Murat Saraçlar. 2018. Turkish natural language processing. Springer. find
  64. María del Carmen Aguirre Martínez and Sonia Mariscal Altares. 2005. Cómo adquieren los niños la gramática de su lengua: perspectivas teóricas. Editorial UNED. find
  65. Arya D. McCarthy, Christo Kirov, Matteo Grella, Amrit Nidhi, Patrick Xia, Kyle Gorman, Ekaterina Vylomova, Sabrina J. Mielke, Garrett Nicolai, Miikka Silfverberg, Timofey Arkhangelskiy, Nataly Krizhanovsky, Andrew Krizhanovsky, Elena Klyachko, Alexey Sorokin, John Mansfield, Valts Ernštreits, Yuval… find
  66. Arya D. McCarthy, Ekaterina Vylomova, Shijie Wu, Chaitanya Malaviya, Lawrence Wolf-Sonkin, Garrett Nicolai, Christo Kirov, Miikka Silfverberg, Sabrina J. Mielke, Jeffrey Heinz, Ryan Cotterell, and Mans Hulden. 2019. find
  67. The SIGMORPHON 2019 shared task: Morphological analysis in context and cross-lingual transfer for inflection. In Proceedings of the 16th Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 229–244, Florence, Italy. Association for Computational Linguistics. find
  68. Lorraine McCune. 1995. A normative study of representational play in the transition to language. Developmental psychology, 31(2):198. find
  69. Rosa Montes. 1987. Secuencias de clarificación en conversaciones con niños (morphe 3-4). Universidad Autónoma de Puebla. find
  70. Ann M Peters. 1987. The role of imitation in the developing syntax of a blind child. Text-Interdisciplinary Journal for the Study of Discourse, 7(3):289–309. find
  71. Tiago Pimentel, Maria Ryskina, Sabrina J. Mielke, Shijie Wu, Eleanor Chodroff, Brian Leonard, Garrett Nicolai, Yustinus Ghanggo Ate, Salam Khalifa, Nizar Habash, Charbel El-Khaissi, Omer Goldman, Michael Gasser, William Lane, Matt Coler, Arturo Oncevay, Jaime Rafael Montoya Samame, Gema Celeste Silv… find
  72. SIGMORPHON 2021 shared task on morphological reinflection: Generalization across languages. In Proceedings of the 18th SIGMOR-PHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 229–259, Online. Association for Computational Linguistics. find
  73. Steven Pinker and Michael T Ullman. 2002. The past and future of the past tense. Trends in Cognitive Sciences, 6(11):456–463. find
  74. V Remedi. 2014. Creación de corpus de datos sobre estudio longitudinal de adquisición de lenguaje de una niña de la región central de Argentina. Ph.D. thesis, National University of Córdoba. find
  75. Brian Roark and Richard Sproat. 2007. Computational approaches to morphology and syntax, volume 4. Oxford University Press. find
  76. Amye Warren-Leubecker. 1982. Sex differences in speech to children. Ph.D. thesis, Georgia Institute of Technology. find
  77. Pamela Rosenthal Rollins. 2003. Caregivers’ contingent comments to 9-month-old infants: Relationships with later language. Applied Psycholinguistics, 24(2):221– 234. find
  78. Silvan Wehrli, Simon Clematide, and Peter Makarov. 2022. CLUZH at SIGMORPHON 2022 shared tasks on morpheme segmentation and inflection generation. In Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 212–219, Seattle, Washington. Ass… find
  79. Jacqueline Sachs and KE Nelson. 1983. Talking about the there and then: The emergence of displaced reference in parent-child discourse. Children’s Language, 4:1–28. find
  80. Richard M Weist, Aleksandra Pawlak, and Karen Hoffman. 2009. Finiteness systems and lexical aspect in child Polish and English. Linguistics, 47(6):1321– 1350. find
  81. R Keith Sawyer. 2013. Pretend play as improvisation: Conversation in the preschool classroom. Psychology Press. find
  82. Wikimedia. 2022. Wikimedia Downloads. find
  83. Mark S. Seidenberg and D. Plaut. 2014. Quasiregular-ity and its discontents: The legacy of the past tense debate. Cognitive Science, 38 6:1190–228. find
  84. Melanie Soderstrom, Megan Blossom, Rina Foygel, and James L Morgan. 2008. Acoustical cues and grammatical units in speech to two preverbal infants. Journal of Child Language, 35(4):869–902. find
  85. Richard A Sprott. 1992. Children’s use of discourse markers in disputes: Form-function relations and discourse in child language. Discourse Processes, 15(4):423–439. find
  86. Patrick Suppes. 1974. The semantics of children’s language. American Psychologist, 29(2):103. find
  87. Virginia Valian. 1991. Syntactic subjects in the early speech of American and Italian children. Cognition, 40(1-2):21–81. find
  88. Lori J. van Houton. 1986. The role of maternal input in the acquisition process: The communicative strategies of adolescent and older mothers with the language learning children. In The Proceedings of the Boston University Conference on Language Development. find
  89. Ekaterina Vylomova, Jennifer White, Elizabeth Salesky, Sabrina J. Mielke, Shijie Wu, Edoardo Maria Ponti, Rowan Hall Maudslay, Ran Zmigrod, Josef Valvoda, Svetlana Toldova, Francis Tyers, Elena Klyachko, Ilya Yegorov, Natalia Krizhanovsky, Paula Czarnowska, Irene Nikkarinen, Andrew Krizhanovsky, Tia… find
  90. SIGMORPHON 2020 shared task 0: Typologically diverse morphological inflection. In Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 1–39, Online. Association for Computational Linguistics. find
  91. Shijie Wu, Ryan Cotterell, and Mans Hulden. 2021. Applying the transformer to character-level transduction. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1901–1907, Online. Association for Computational Linguistics. find
  92. Shijie Wu, Ryan Cotterell, and Timothy O’Donnell. 2019. Morphological irregularity correlates with frequency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5117–5126, Florence, Italy. Association for Computational Linguistics. find
  93. Karina Hess Zimmermann. 2003. El desarrollo linguístico en los años escolares: análisis de narraciones infantiles. Ph.D. thesis, El Colegio de México. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lex-2023-kodner-inflection-reality-check/