In the LLM era, Word Sense Induction remains unsolved

What it is about

The paper argues that word sense induction (WSI), which groups the occurrences of a word into senses without a sense list, is badly evaluated. The usual SemEval sets over-represent polysemous words and do not keep the natural polysemy and frequency of a corpus. The authors build a SemCor-derived set that keeps them, and compare pre-trained embeddings and clustering methods across parts of speech, with an LLM-based method. The strong baseline is "one cluster per lemma" (1cpl), which scores 73.6 on the F-B3 measure, and no unsupervised method beats it. The best previous system, PolyLM, scores 73.0, and adding enough unlabelled examples to the clustering is reported to beat it. The authors conclude that WSI is not solved, and that LLMs struggle with the task.

Ideas

• I1. onto/src/wsd.rs: report the one-cluster-per-lemma baseline next to every WSI or WSD score. Any new Mova method must beat it on the same SemCor-derived set. The negative control is the baseline itself, which should reproduce its own score exactly.
• I2. Deterministic clustering of contexts: for each lemma in the school silver corpus, represent each occurrence by its head-dependent pairs from the en parse, and cluster with a fixed threshold in a BTreeMap-ordered loop. Measure F-B3 against a held-out annotated set, with the 1cpl baseline beside it.
• I3. Instances per lemma: measure the F-B3 of the clustering as the number of corpus instances per lemma grows (for example 10, 50, 100), to find the point where it passes 1cpl. The paper's threshold of 50 is the first value to test.

For Mova

The code is onto/src/wsd.rs for the baseline and en/src/parse.rs for the context features, with en ctx-report as a check. The chapter is 19. The lexicon: vocabulary, multiword expressions and register. The paper's data are SemCor-derived, so use them for tests and measurement only. The LLM parts are not for Mova: the large model may only label training data offline. Measure first: F-B3 for the 1cpl baseline and for the clustering, run twice and compared.

Topics: Word sense disambiguation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.findings-acl.882 · arXiv 2603.11686

@inproceedings{mosolova2025llm,
  title = {{In the LLM era, Word Sense Induction remains unsolved}},
  author = {Mosolova, Anna and Candito, Marie and Ramisch, Carlos},
  booktitle = {Findings of ACL 2025 (Vienna)},
  year = {2025},
  doi = {10.18653/v1/2025.findings-acl.882},
  eprint = {2603.11686},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2025.findings-acl.882}
}

References

40 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
2. Hadi Abdine, Moussa Kamal Eddine, Michalis Vazirgiannis, and Davide Buscaldi. 2022. Word sense induction with hierarchical clustering and mutual information maximization. find
3. Rotem Dror, Lotem Peled-Cohen, Segev Shlomov, and Roi Reichart. 2020. Statistical Significance Testing for Natural Language Processing. Springer International Publishing. find
4. Eneko Agirre and Aitor Soroa. 2007. SemEval-2007 task 02: Evaluating word sense induction and discrimination systems. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 7–12, Prague, Czech Republic. Association for Computational Linguistics. find
5. Katrin Erk and Diana McCarthy. 2009. Graded word sense assignment. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 440–449, Singapore. Association for Computational Linguistics. find
6. Enrique Amigó, Julio Gonzalo, Javier Artiles, and Fe-lisa Verdejo. 2008. A comparison of extrinsic clus-Katrin Erk, Diana McCarthy, and Nicholas Gaylord. 2009. Investigations on word senses and word usages. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th Inte… find
7. The COLING 2016 Organizing Committee. find
8. Matan Eyal, Shoval Sadde, Hillel Taub-Tabib, and Yoav Goldberg. 2022. Large scale substitution-based word sense induction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4738–4752, Dublin, Ireland. Association for Computation… find our card
9. Linlin Li, Ivan Titov, and Caroline Sporleder. 2014. Improved estimation of entropy for evaluation of word sense induction. Computational Linguistics, 40(3):671–685. find
10. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783. find
11. Amir Ahmad Habibi, Bradley Hauer, and Grzegorz Kondrak. 2021. Homonymy and polysemy detection with multilingual information. In Proceedings of the 11th Global Wordnet Conference, pages 26–35, University of South Africa (UNISA). Global Wordnet Association. find
12. Yoshihiko Hayashi. 2025. Evaluating LLMs’ capability to identify lexical semantic equivalence: Probing with the word-in-context task. In Proceedings of the 31st International Conference on Computational Linguistics, pages 6985–6998, Abu Dhabi, UAE. Association for Computational Linguistics. find
13. Ondřej Herman and Miloš Jakubíček. 2024. Shad-owSense: A multi-annotated dataset for evaluating word sense induction. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 14763–14769, Torino, Italia. ELR… find
14. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: The 90% solution. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pages 57–60, New York City, USA. Association for Computational Linguistics. find
15. Nancy Ide and Keith Suderman. 2004. The American national corpus first release. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC‘04), Lisbon, Portugal. European Language Resources Association (ELRA). find
16. David Jurgens and Ioannis Klapaftis. 2013. SemEval-2013 task 13: Word sense induction for graded and non-graded senses. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 2… find
17. Alexandros Komninos and Suresh Manandhar. 2016. Structured generative models of continuous features for word sense induction. In Proceedings of COLING Bastien Liétard, Pascal Denis, and Mikaela Keller. 2024. To word senses and beyond: Inducing concepts with contextualized language models. In Proceed… find
18. Qianchu Liu, Fangyu Liu, Nigel Collier, Anna Korhonen, and Ivan Vulić. 2021. MirrorWiC: On eliciting word-in-context representations from pretrained language models. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 562–574, Online. Association for Computationa… find
19. Juan Antonio Lossio-Ventura, Clement Jonquet, Mathieu Roche, and Maguelonne Teisseire. 2016. Automatic biomedical term polysemy detection. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC‘16), pages 1684–1688, Portorož, Slovenia. European Language Resou… find
20. Suresh Manandhar, Ioannis Klapaftis, Dmitriy Dligach, and Sameer Pradhan. 2010. SemEval-2010 task 14: Word sense induction &disambiguation. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 63–68, Uppsala, Sweden. Association for Computational Linguistics. find
21. Rada Mihalcea, Timothy Chklovski, and Adam Kilgarriff. 2004. The senseval-3 English lexical sample task. In Proceedings of SENSEVAL-3, the Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text, pages 25–28, Barcelona, Spain. Association for Computational Linguis… find
22. George A. Miller, Claudia Leacock, Randee Tengi, and Ross T. Bunker. 1993. A semantic concordance. In Human Language Technology: Proceedings of a Workshop Held at Plainsboro, New Jersey, March 21-24, 1993. find
23. Anna Mosolova, Marie Candito, and Carlos Ramisch. 2024. Injecting Wiktionary to improve token-level contextual representations using contrastive learning. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 34… find
24. Roberto Navigli and Daniele Vannella. 2013. SemEval-2013 task 11: Word sense induction and disambiguation within an end-user application. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemE… find
25. Andrei Novikov. 2019. Pyclustering: Data mining library. Journal of Open Source Software, 4(36):1230. find
26. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, et al. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774. find
27. Miguel Ortega-Martín, Óscar García-Sierra, Alfonso Ardoiz, Jorge Álvarez, Juan Carlos Armenteros, and Adrián Alonso. 2023. Linguistic ambiguity analysis in chatgpt. Preprint, arXiv:2302.06426. find
28. Alexander Panchenko, Eugen Ruppert, Stefano Faralli, Simone Paolo Ponzetto, and Chris Biemann. 2017. Unsupervised does not mean uninterpretable: The case for word sense induction and disambiguation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Li… find
29. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:282… find
30. Francesco Periti, David Alfter, and Nina Tahmasebi. 2024. Automatically generated definitions and their utility for modeling word meaning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14008–14026, Miami, Florida, USA. Association for Computational… find
31. Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations. arXiv preprint. ArXiv:1808.09121 [cs]. find
32. Oscar Sainz, Oier Lopez de Lacalle, Eneko Agirre, and German Rigau. 2023. What do language models know about word senses? zero-shot WSD with language models and domain inventories. In Proceedings of the 12th Global Wordnet Conference, pages 331–342, University of the Basque Country, Donostia - San S… find
33. Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi. 2020. SemEval-2020 task 1: Unsupervised lexical semantic change detection. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1–23, Barcelona (online). International Committee for Com… find
34. Sylvia Springorum, Sabine Schulte im Walde, and Jason Utt. 2013. Detecting polysemy in hard and soft cluster analyses of German preposition vector spaces. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 632– 640, Nagoya, Japan. Asian Federation of Nat… find
35. T. G. D. K. Sumanathilaka, Nicholas Micallef, and Julian Hough. 2024. Can llms assist with ambiguity? a quantitative evaluation of various large language models on word sense disambiguation. Preprint, arXiv:2411.18337. find
36. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lh… find
37. Christos Xypolopoulos, Antoine Tixier, and Michalis Vazirgiannis. 2021. Unsupervised word polysemy quantification with multiresolution grids of contextual embeddings. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 33… find
38. Kosuke Yamada, Ryohei Sasano, and Koichi Takeda. 2021. Semantic frame induction using masked word embeddings and two-step clustering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing… find
39. Deniz Ekin Yavas. 2024. Assessing the significance of encoded information in contextualized representations to word sense disambiguation. In Proceedings of the Third Workshop on Understanding Implicit and Underspecified Language, pages 42–53, Malta. Association for Computational Linguistics. find
40. Deniz Ekin Yavas, Timothée Bernard, Laura Kallmeyer, and Benoît Crabbé. 2024. Improving word sense induction through adversarial forgetting of morphosyntactic information. In Proceedings of the 13th Joint Conference on Lexical and Computational Semantics (*SEM 2024), pages 238–251, Mexico City, Mexi… find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lex-2025-mosolova-wsi-unsolved/