Mova0.5 pre-release⌕
☰
home / docs / papers / English / Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words

Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words

What it is about

The paper asks how the input segmentation of a pretrained model, BERT, affects its reading of English derivatives. A WordPiece split of "superbizarre" mixes the stem "bizarre" with the prefix "super", so the model links the word to "superb". The authors add DelBERT, which segments the input at derivational boundaries, and test it on semantic probes for sentiment and topicality with three data sets: Amazon (sentiment), and ArXiv and Reddit (topicality). DelBERT beats BERT by a large margin on all three, and the gain is larger on ArXiv and Reddit than on Amazon. Each model is trained with 20 random seeds on a 60/20/20 split. Human accuracy on 100 sampled words per set is 71% (Amazon) to 78% (ArXiv), which the authors give as the upper bound.

Ideas

For Mova

The code is en/src/morph.rs for the splitter, en/src/tok.rs for the boundaries, and prag for polarity, with en morph-eval and en tok-eval as checks. The chapters are 16. Derivation I: prefixes and 15-adjectives-adverbs.md. The paper's model is neural and its data sets are not for the build, so only the segmentation idea and the probe design transfer, as deterministic rules. Measure first: the boundary accuracy for prefixed words on EWT test, with two identical runs.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-long.279 · arXiv 2101.00403

@inproceedings{valentin2021superbizarre,
  title = {{Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words}},
  author = {Valentin and Hofmann},
  booktitle = {ACL-IJCNLP 2021 (online)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.279},
  eprint = {2101.00403},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.acl-long.279}
}

References

101 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Heike Adel, Ehsaneddin Asgari, and Hinrich Schütze. 2017. Overview of character-based models for natural language processing. In International Conference on Computational Linguistics and Intelligent Text Processing (CICLing) 18. find
  2. Maria Alegre and Peter Gordon. 1999. Frequency effects and the representational status of regular inflections. Journal of Memory and Language, 40:41–61. find
  3. Ben Athiwaratkun, Andrew Wilson, and Anima Anand-kumar. 2018. Probabilistic fasttext for multi-sense word embeddings. In Annual Meeting of the Association for Computational Linguistics (ACL) 56. find
  4. R. Harald Baayen, Ton Dijkstra, and Robert Schreuder. 1997. Singulars and plurals in Dutch: Evidence for a parallel dual-route model. Journal of Memory and Language, 37:94–117. find
  5. R. Harald Baayen and Rochelle Lieber. 1991. Productivity and English derivation: A corpus-based study. Linguistics, 29(5). find
  6. R. Harald Baayen, Robert Schreuder, and Richard Sproat. 2000. Morphology in the mental lexicon: A computational model for visual word recognition. In Frank van Eynde and Dafydd Gibbon, editors, Lexicon development for speech and language processing, pages 267–293. Springer, Dordrecht. find
  7. Raymond Bertram, Matti Laine, R. Harald Baayen, Robert Schreuder, and Jukka Hyönä. 2000a. Affixal homonymy triggers full-form storage, even with inflected words, even in a morphologically rich language. Cognition, 74:B13–B25. find
  8. Raymond Bertram, Robert Schreuder, and R. Harald Baayen. 2000b. The balance of storage and computation in morphological processing: The role of word formation type, affixal homonymy, and productivity. Journal of Experimental Psychology: Learning, Memory, and Cognition, 26(2):489–511. find
  9. Elisabeth Beyersmann, Johannes C. Ziegler, and Jonathan Grainger. 2015. Differences in the processing of prefixes and suffixes revealed by a letter-search task. Scientific Studies of Reading, 19(5):360–373. find
  10. Parminder Bhatia, Robert Guthrie, and Jacob Eisenstein. 2016. Morphological priors for probabilistic neural word embeddings. In Conference on Empirical Methods in Natural Language Processing (EMNLP) 2016. find
  11. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. find
  12. Kaj Bostrom and Greg Durrett. 2020. Byte pair encoding is suboptimal for language model pretraining. In Findings of Empirical Methods in Natural Language Processing (EMNLP) 2020. find
  13. Jan A. Botha and Phil Blunsom. 2014. Compositional morphology for word representations and language modelling. In International Conference on Machine Learning (ICML) 31. find
  14. Cristina Burani and Alfonso Caramazza. 1987. Representation and processing of derived words. Language and Cognitive Processes, 2(3-4):217–227. find
  15. Cristina Burani and Alessandro Laudanna. 1992. Units of representation for derived words in the lexicon. In Ram Frost and Leonard Katz, editors, Orthography, phonology, morphology, and meaning, pages 361– 376. North-Holland, Amsterdam. find
  16. Brian Butterworth. 1983. Lexical representation. In Brian Butterworth, editor, Language production: Development, writing and other language processes, pages 257–294. Academic Press, London. find
  17. Joan Bybee. 1988. Morphology as lexical organization. In Michael Hammond and Michael Noonan, editors, Theoretical approaches to morphology: Approaches in modern linguistics, pages 119–141. Academic Press, San Diego, CA. find
  18. Joan Bybee. 1995. Regular morphology and the lexicon. Language and Cognitive Processes, 10(425-455). find
  19. Alfonso Caramazza, Alessandro Laudanna, and Cristina Romani. 1988. Lexical access and inflectional morphology. Cognition, 28(297-332). find
  20. Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. In Workshop on Natural Language Processing for Conversational AI 2. find
  21. Kenneth Church. 2020. Emerging trends: Subwords, seriously? Natural Language Engineering, 26(3):375–382. find
  22. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pretraining text encoders as discriminators rather than generators. In International Conference on Learning Representations (ICLR) 8. find
  23. Ryan Cotterell and Hinrich Schütze. 2018. Joint semantic synthesis and morphological analysis of the derived word. Transactions of the Association for Computational Linguistics, 6:33–48. find
  24. Ryan Cotterell, Ekaterina Vylomova, Huda Khayrallah, Christo Kirov, and David Yarowsky. 2017. Paradigm completion for derivational morphology. In Conference on Empirical Methods in Natural Language Processing (EMNLP) 2017. find
  25. Scott Deerwester, Susan T. Dumais, George Furnas, Thomas Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391–407. find
  26. Daniel Deutsch, John Hewitt, and Dan Roth. 2018. A distributional and orthographic aggregation model for English derivational morphology. In Annual Meeting of the Association for Computational Linguistics (ACL) 56. find our card
  27. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HTL) 2019. find
  28. Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019. Show your work: Improved reporting of experimental results. In Conference on Empirical Methods in Natural Language Processing (EMNLP) 2019. find
  29. Daniel Edmiston. 2020. A systematic analysis of morphological content in BERT models for multiple languages. In arXiv 2004.03032. — [arXiv:2004.03032](https://arxiv.org/abs/2004.03032) find
  30. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Conference on Empirical Methods in Natural Language Processing (EMNLP) 2019. find
  31. Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34–48. find
  32. Hao Fang, Mari Ostendorf, Peter Baumann, and Janet B. Pierrehumbert. 2015. Exponential language modeling using morphological features and multi-task learning. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(12):2410– 2421. find
  33. Laurie B. Feldman and Carol A. Fowler. 1987. The inflected noun system in Serbo-Croatian: Lexical representation of morphological structure. Memory and Cognition, 15(1):1–12. find
  34. Pius ten Hacken. 2014. Delineating derivation and inflection. In Rochelle Lieber and Pavol Štekauer, editors, The Oxford handbook of derivational morphology, pages 10–25. Oxford University Press, Oxford. find
  35. Bo Han and Timothy Baldwin. 2011. Lexical normalisation of short text messages: Makn sens a \#twitter. In Annual Meeting of the Association for Computational Linguistics (ACL) 49. find
  36. Martin Haspelmath and Andrea D. Sims. 2010. Understanding morphology. Routledge, New York, NY. find
  37. Benjamin Heinzerling and Michael Strube. 2019. Sequence tagging with contextual and non-contextual subword representations: A multilingual evaluation. In Annual Meeting of the Association for Computational Linguistics (ACL) 57. find
  38. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HTL) 2019. find
  39. Valentin Hofmann, Janet B. Pierrehumbert, and Hinrich Schütze. 2020a. DagoBERT: Generating derivational morphology with a pretrained language model. In Conference on Empirical Methods in Natural Language Processing (EMNLP) 2020. find
  40. Valentin Hofmann, Janet B. Pierrehumbert, and Hinrich Schütze. 2020b. Predicting the growth of morphological families from social and linguistic factors. In Annual Meeting of the Association for Computational Linguistics (ACL) 58. find
  41. Valentin Hofmann, Hinrich Schütze, and Janet B. Pierrehumbert. 2020c. A graph auto-encoder model of derivational morphology. In Annual Meeting of the Association for Computational Linguistics (ACL) 58. find
  42. Abhilash Jain, Aku Rouhe, Stig-Arne Grönroos, and Mikko Kurimo. 2020. Finnish ASR with deep transformer models. In Conference of the International Speech Communication Association (INTER-SPEECH) 21. find
  43. Uli H. Frauenfelder and Robert Schreuder. 1992. Constraining psycholinguistic models of morphological processing and representation: The role of productivity. In Geert Booij and Jaap van Marle, editors, Yearbook of morphology 1991, volume 26, pages 165– 183. Kluwer, Dordrecht. find
  44. Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Annual Meeting of the Association for Computational Linguistics (ACL) 57. find
  45. Philip Gage. 1994. A new algorithm for data compression. The C Users Journal, 12(2):23–38. find
  46. Emilia Kacprzak, Laura M. Koesten, Luis-Daniel Ibáñez, Elena Simperl, and Jeni Tennison. 2017. A query log analysis of dataset search. In International Conference on Web Engineering (ICWE) 17. find
  47. Hélène Giraudo and Jonathan Grainger. 2003. On the role of derivational affixes in recognizing complex words: Evidence from masked priming. In R. Harald Baayen and Robert Schreuder, editors, Morphological structure in language processing, pages 209– 232. De Gruyter, Berlin. find
  48. Nora Kassner and Hinrich Schütze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Annual Meeting of the Association for Computational Linguistics (ACL) 58. find
  49. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. 2016. Character-aware neural language models. In Conference on Artificial Intelligence (AAAI) 30. find
  50. Diederik P. Kingma and Jimmy L. Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR) 3. find
  51. Max Kisselew, Sebastian Padó, Alexis Palmer, and Jan Šnajder. 2015. Obtaining a better understanding of distributional models of german derivational morphology. In International Conference on Computational Semantics (IWCS) 11. find
  52. Matej Klemen, Luka Krsnik, and Marko Robnik-Šikonja. 2020. Enhancing deep neural networks with morphological information. In arXiv 2011.12432. — [arXiv:2011.12432](https://arxiv.org/abs/2011.12432) find
  53. Victor Kuperman, Raymond Bertram, and R. Harald Baayen. 2008. Morphological dynamics in compound processing. Language and Cognitive Processes, 23(7-8):1089–1132. find
  54. Victor Kuperman, Robert Schreuder, Raymond Bertram, and R. Harald Baayen. 2009. Reading of polymorphemic Dutch compounds: Towards a multiple route model of lexical processing. Journal of Experimental Psychology: Human Perception and Performance, 35(3):876–895. find
  55. Alessandro Laudanna and Cristina Burani. 1985. Address mechanisms to decomposed lexical entries. Linguistics, 23(5). find
  56. Alessandro Laudanna and Cristina Burani. 1995. Distributional properties of derivational affixes: Implications for processing. In Laurie B. Feldman, editor, Morphological aspects of language processing, pages 345–364. Lawrence Erlbaum, Hillsdale, NJ. find
  57. Angeliki Lazaridou, Marco Marelli, Roberto Zamparelli, and Marco Baroni. 2013. Compositional-ly derived representations of morphologically complex words in distributional semantics. In Annual Meeting of the Association for Computational Linguistics (ACL) 51. find
  58. Alina Leminen, Eva Smolka, Jon Duñabeitia, and Christos Pliatsikas. 2019. Morphological processing in the brain: The good (inflection), the bad (derivation) and the ugly (compounding). Cortex, 116:4–44. find
  59. Minh-Thang Luong, Richard Socher, and Christopher D. Manning. 2013. Better word representations with recursive neural networks for morphology. In Conference on Computational Natural Language Learning (CoNLL) 17. find
  60. Wentao Ma, Yiming Cui, Chenglei Si, Ting Liu, Shi-jin Wang, and Guoping Hu. 2020. CharBERT: Character-aware pre-trained language model. In International Conference on Computational Linguistics (COLING) 28. find
  61. Leon Manelis and David A. Tharp. 1977. The processing of affixed words. Memory and Cognition, 5(6):690–695. find
  62. Louis Martin, Benjamin Muller, Pedro J. Suárez, Yoann Dupont, Laurent Romary, de la Clergerie, Éric V., Djamé Seddah, and Benoit Sagot. 2020. CamemBERT: A tasty French language model. In Annual Meeting of the Association for Computational Linguistics (ACL) 58. find
  63. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In arXiv 1301.3781. find
  64. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems (NIPS) 26. find
  65. Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Annual Meeting of the Association for Computational Linguistics (ACL) 47. find
  66. Jeremy M. Needle and Janet B. Pierrehumbert. 2018. Gendered associations of english morphology. Journal of the Association for Laboratory Phonology, 9(1):119. find
  67. Boris New, Marc Brysbaert, Juan Segui, Ludovic Ferrand, and Kathleen Rastle. 2004. The processing of singular and plural nouns in french and english. Journal of Memory and Language, 51(4):568–585. find
  68. Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019. Justifying recommendations using distantly-labeled reviews and fined-grained aspects. In Conference on Empirical Methods in Natural Language Processing (EMNLP) 2019. find
  69. Sebastian Padó, Aurélie Herbelot, Max Kisselew, and Jan Šnajder. 2016. Predictability of distributional semantics in derivational word formation. In International Conference on Computational Linguistics (COLING) 26. find
  70. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Conference on Empirical Methods in Natural Language Processing (EMNLP) 2014. find
  71. Yuval Pinter, Cassandra L. Jacobs, and Jacob Eisenstein. 2020. Will it unblend? In Findings of Empirical Methods in Natural Language Processing (EMNLP) 2020. find
  72. Ingo Plag. 2003. Word-formation in English. Cambridge University Press, Cambridge, UK. find
  73. Siyu Qiu, Qing Cui, Jiang Bian, Bin Gao, and Tie-Yan Liu. 2014. Co-learning of word representations and morpheme representations. In International Conference on Computational Linguistics (COLING) 25. find
  74. Péter Rácz, Clay Beckner, Jennifer Hay, and Janet B. Pierrehumbert. 2020. Morphological convergence as on-line lexical analogy. Language, 96(4):735– 770. find
  75. Péter Rácz, Janet B. Pierrehumbert, Jennifer Hay, and Viktória Papp. 2015. Morphological emergence. In Brian MacWhinney and William O’Grady, editors, The handbook of language emergence, pages 123– 146. Wiley, Hoboken, NJ. find
  76. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  77. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1–67. find
  78. Kathleen Rastle and Matthew H. Davis. 2008. Morphological decomposition based on the analysis of orthography. Language and Cognitive Processes, 23(7-8):942–971. find
  79. Kathleen Rastle, Matthew H. Davis, and Boris New. 2004. The broth in my brother’s brothel: Morpho-orthographic segmentation in visual word recognition. Psychonomic Bulletin and Review, 11(6):1090–1098. find
  80. Alexandre Salle and Aline Villavicencio. 2018. Incorporating subword information into matrix factorization word embeddings. In Workshop on Subword/Character LEvel Models 2. find
  81. Robert Schreuder and R. Harald Baayen. 1995. Modeling morphological processing. In Laurie B. Feldman, editor, Morphological aspects of language processing, pages 131–154. Lawrence Erlbaum, Hillsdale, NJ. find
  82. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and Korean voice search. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP) 37. find
  83. Hinrich Schütze. 1992. Word space. In Advances in Neural Information Processing Systems (NIPS) 5, pages 895–902. find
  84. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Annual Meeting of the Association for Computational Linguistics (ACL) 54. find
  85. Suzanna Sia, Ayush Dalmia, and Sabrina J. Mielke. 2020. Tired of topic models? Clusters of pretrained word embeddings make for fast and good topics too! In Conference on Empirical Methods in Natural Language Processing (EMNLP) 2020. find
  86. Joseph P. Stemberger. 1994. Rule-less morphology at the phonology-lexicon interface. In Susan D. Lima, Roberta Corrigan, and Gregory Iverson, editors, The reality of linguistic rules, pages 147–169. John Benjamins, Amsterdam. find
  87. Gregory Stump. 2017. Rule conflation in an inferential-realizational theory of morphotactics. Acta Linguistica Academica, 64(1):79–124. find
  88. Gregory Stump. 2019. Some sources of apparent gaps in derivational paradigms. Morphology, 29(2):271– 292. find
  89. Marcus Taft. 1979. Recognition of affixed words and the word frequency effect. Memory and Cognition, 7(4):263–272. find
  90. Marcus Taft. 1981. Prefix stripping revisited. Journal of Verbal Learning and Verbal Behavior, 20:289– 297. find
  91. Marcus Taft. 1988. A morphological-decomposition model of lexical representation. Linguistics, 26:657– 667. find
  92. Marcus Taft. 1991. Reading and the mental lexicon. Lawrence Erlbaum, Hove, UK. find
  93. Marcus Taft. 1994. Interactive-activation as a framework for understanding morphological processing. Language and Cognitive Processes, 9(3):271–294. find
  94. Marcus Taft. 2004. Morphological decomposition and the reverse base frequency effect. The Quarterly Journal of Experimental Psychology, 57(4):745– 765. find
  95. Marcus Taft and Kenneth I. Forster. 1975. Lexical storage and retrieval of prefixed words. Journal of Verbal Learning and Verbal Behavior, 14:638–647. find
  96. Samson Tan, Shafiq Joty, Lav R. Varshney, and Min-Yen Kan. 2020. Mind your inflections! Improving NLP for non-standard Englishes with base-inflection encoding. In Conference on Empirical Methods in Natural Language Processing (EMNLP) 2020. find
  97. Ivan Vulić, Edoardo M. Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. 2020. Probing pretrained language models for lexical semantics. In Conference on Empirical Methods in Natural Language Processing (EMNLP) 2020. find
  98. Ekaterina Vylomova, Ryan Cotterell, Timothy Baldwin, and Trevor Cohn. 2017. Context-aware prediction of derivational word-forms. In Conference of the European Chapter of the Association for Computational Linguistics (EACL) 15. find
  99. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc Le V, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Ni… find
  100. Yi Yang and Jacob Eisenstein. 2017. Overcoming language variation in sentiment analysis with social attention. Transactions of the Association for Computational Linguistics, 5:295–307. find
  101. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems (NeurIPS) 33. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lex-2021-hofmann-superbizarre-derivational/