Mova0.5 pre-release⌕
☰
home / docs / papers / English / Interpretable Word Sense Representations via Definition Generation: The Case of Semantic Change Analysis

Interpretable Word Sense Representations via Definition Generation: The Case of Semantic Change Analysis

What it is about

The authors generate a natural-language definition for each usage of a target word with a fine-tuned Flan-T5 model. Usages are first grouped into clusters, which stand for senses. For each cluster the most prototypical generated definition becomes the sense label. These labels are then used in semantic change analysis, so that a change in a word's usage can be read as a change in its definitions. On word-in-context similarity judgements, the definition-based representations beat token embeddings. For the sense labels, annotators found neither label satisfactory for 15% of clusters. A definition-based label was judged better than a usage-based one in 31% of cases, the reverse in 7%, and definition-based labels were of sufficient quality in 80% of instances, against 68% for usage-based labels.

Ideas

For Mova

The touch points are onto/src/wsd.rs (sense choice and explanations), tools/gutenberg (public-domain books) and the en tokenizer for context windows. The chapter is 37. Register and variation, with 19. The lexicon: vocabulary, multiword expressions and register. The paper's Flan-T5 model is a neural teacher idea; Mova takes only the prototype-label step, and the DWUG data is for tests only. Measure first the share of acceptable WordNet labels on a small hand-judged sample of clusters.

Topics: Word sense disambiguation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-long.176 · arXiv 2305.11993

@inproceedings{giulianelli2023interpretable,
  title = {{Interpretable Word Sense Representations via Definition Generation: The Case of Semantic Change Analysis}},
  author = {Giulianelli, Mario and Luden, Iris and Fernandez, Raquel and Kutuzov, Andrey},
  booktitle = {ACL 2023 (Toronto)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.176},
  eprint = {2305.11993},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.acl-long.176}
}

References

35 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Reem Alatrash, Dominik Schlechtweg, Jonas Kuhn, and Sabine Schulte im Walde. 2020. CCOHA: Clean corpus of historical American English. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6958–6966, Marseille, France. European Language Resources Association. find our card
  2. Fatemah Almeman and Luis Espinosa Anke. 2022. Putting WordNet’s dictionary examples in the context of definition modelling: An empirical analysis. In Proceedings of the Workshop on Cognitive Aspects of the Lexicon, pages 42–48, Taipei, Taiwan. Association for Computational Linguistics. find
  3. Tal August, Katharina Reinecke, and Noah A. Smith. 2022. Generating scientific definitions with controllable complexity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8298–8317, Dublin, Ireland. Association for Computational… find
  4. Nikhil Bansal, Avrim Blum, and Shuchi Chawla. 2004. Correlation clustering. Machine Learning, 56(1):89– 113. find
  5. Michele Bevilacqua, Marco Maru, and Roberto Navigli. 2020. Generationary or “how we went beyond word sense inventories and learned to gloss”. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7207–7221, Online. Association for Computational Ling… find our card
  6. Leonard Bloomfield. 1933. Language. Allen & Unwin. find
  7. Tadeusz Caliński and Jerzy Harabasz. 1974. A dendrite method for cluster analysis. Communications in Statistics - Theory and Methods, 3(1):1–27. find
  8. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. find
  9. Mark Davies. 2012. Expanding horizons in historical linguistics with the 400-million word Corpus of Historical American English. Corpora, 7(2):121–157. find
  10. Artyom Gadetsky, Ilya Yakubovskiy, and Dmitry Vetrov. 2018. Conditional generators of words definitions. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 266–271, Melbourne, Australia. Association for Computational Linguistics… find
  11. Noah Gardner, Hafiz Khan, and Chih-Cheng Hung. 2022. Definition modeling: Literature review and dataset analysis. Applied Computing and Intelligence, 2(1):83–98. find
  12. Mario Giulianelli, Marco Del Tredici, and Raquel Fernández. 2020. Analysing lexical semantic change with contextualised word representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3960– 3973, Online. Association for Computational Linguistic… find
  13. Daniil Homskiy and Nikolay Arefyev. 2022. DeepMis-take at LSCDiscovery: Can a multilingual word-in-context model replace human annotators? In Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change, pages 173– 179, Dublin, Ireland. Association for Computational Ling… find
  14. Han Huang, Tomoyuki Kajiwara, and Yuki Arase. 2021. Definition modelling for appropriate specificity. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2499–2509, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  15. Jie Huang, Hanyin Shao, Kevin Chen-Chuan Chang, Jinjun Xiong, and Wen-mei Hwu. 2022. Understanding jargon: Combining extraction and generation for definition modeling. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3994–4004, Abu Dhabi, United Arab E… find
  16. Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, et al. 2022. State-of-the-art generalisation research in NLP: A taxonomy and review. arXiv preprint arXiv:2210.03050. find
  17. Shonosuke Ishiwatari, Hiroaki Hayashi, Naoki Yoshi-naga, Graham Neubig, Shoetsu Sato, Masashi Toy-oda, and Masaru Kitsuregawa. 2019. Learning to describe unknown phrases with local and global contexts. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa… find
  18. Adam Kilgarriff. 1997. I don’t believe in word senses. Computers and the Humanities, 31(2):91–113. find our card
  19. Cunliang Kong, Yun Chen, Hengyuan Zhang, Liner Yang, and Erhong Yang. 2022. Multitasking framework for unsupervised simple definition generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5934–5943, Dublin, Ireland. Asso… find
  20. Sinan Kurtyigit, Maike Park, Dominik Schlechtweg, Jonas Kuhn, and Sabine Schulte im Walde. 2021. Lexical semantic change discovery. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (V… find
  21. Andrey Kutuzov, Lilja Øvrelid, Terrence Szymanski, and Erik Velldal. 2018. Diachronic word embeddings and semantic shifts: a survey. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1384–1397, Santa Fe, New Mexico, USA. Association for Computational Linguistics… find
  22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. find
  23. Timothee Mickus, Denis Paperno, and Matthieu Constant. 2019. Mark my word: A sequence-to-sequence approach to definition modeling. In Proceedings of the First NLPL Workshop on Deep Learning for Natural Language Processing, pages 1–11, Turku, Finland. Linköping University Electronic Press. find
  24. Timothee Mickus, Kees Van Deemter, Mathieu Constant, and Denis Paperno. 2022. Semeval-2022 task 1: CODWOE – comparing dictionaries and word embeddings. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1–14, Seattle, United States. Association for Computa… find
  25. George A Miller. 1995. WordNet: A lexical database for English. Communications of the ACM, 38(11):39– 41. find our card
  26. George A. Miller, Claudia Leacock, Randee Tengi, and Ross T. Bunker. 1993. A semantic concordance. In Human Language Technology: Proceedings of a Workshop Held at Plainsboro, New Jersey, March 21-24, 1993. find
  27. Sunny Mitra, Ritwik Mitra, Suman Kalyan Maity, Martin Riedl, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. 2015. An automatic approach to identify word sense changes in text media across timescales. Natural Language Engineering, 21(5):773–798. find
  28. Ke Ni and William Yang Wang. 2017. Learning to explain non-standard English words and phrases. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 413–417, Taipei, Taiwan. Asian Federation of Natural Language Processing. find
  29. Thanapon Noraset, Chen Liang, Larry Birnbaum, and Doug Downey. 2017. Definition modeling: Learning to define word embeddings in natural language. In Thirty-First AAAI Conference on Artificial Intelligence. find
  30. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1– 67. find
  31. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,… find
  32. Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi. 2020. SemEval-2020 task 1: Unsupervised lexical semantic change detection. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1–23, Barcelona (online). International Committee for Com… find
  33. Dominik Schlechtweg, Nina Tahmasebi, Simon Hengchen, Haim Dubossarsky, and Barbara McGillivray. 2021. DWUG: A large resource of diachronic word usage graphs in four languages. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7079–7091, Online and Punta… find
  34. Nina Tahmasebi, Lars Borin, and Adam Jatowt. 2021. Survey of computational approaches to lexical semantic change detection. Computational approaches to semantic change, 6:1. find
  35. Length Relative Position Absolute Position BertScore Bleu Nist Length Relative Position Absolute Position BertScore Bleu Nist 1.000000 -0.121793 0.575304 0.067180 0.076133 0.044873 -0.121793 1.000000 0.626032 0.052725 0.074697 0.062041 0.575304 0.626032 1.000000 0.128785 0.159078 0.110559 0.067180 0… find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lex-2023-giulianelli-definition-generation-senses/