Large Scale Substitution-based Word Sense Induction

What it is about

The paper induces word senses for every frequent word of a large corpus. For each corpus position a masked language model proposes the top five substitutes, and the substitute vectors are clustered per word type into senses. The result is a sense-tagged corpus in which each sense comes with indicative words. Storing the top five substitutes needs under 20 GB for English Wikipedia, against more than 8 TB for full vectors. The approach covers 16k vocabulary items that account for 92% of word occurrences. On 2,000 sampled instances of 20 ambiguous words, the induced senses are scored against manual annotation, and they beat Babelfy's F1 in that test. The same method applied to 31 million PubMed abstracts finds scientific senses that standard inventories lack.

Ideas

• I1. en/src/lexgen.rs and onto/src/wsd.rs: replace the masked-model substitutes with the English synonyms and hypernyms from onto (Open English WordNet). For each token, take the candidate set, and cluster the occurrences of one lemma by the overlap of their candidate sets with a fixed rule. Measure F1 on 100 hand-annotated sentences per word for 20 ambiguous words.
• I2. New-sense probe: list the school-corpus tokens whose best onto synset overlaps poorly with their context, and count the groups that a human confirms as a new sense. The paper's "corona" example is the model case.
• I3. Compact store: record the top-5 candidate ids per token as 2-byte values in a table, and measure the bytes per token and the time per token on the school corpus against the current format.

For Mova

The code is onto/src/wsd.rs for the senses and en/src/lexgen.rs for the candidate lists, with en ctx-report for the context check. The chapter is 19. The lexicon: vocabulary, multiword expressions and register. The substitutes in the paper come from a neural model, which Mova does not use at test time, so only the clustering and storage ideas transfer. The 2,000-instance annotated set is for tests only. Measure first: F1 of the rule-based clusters against the hand-annotated sample, two runs identical.

Topics: Word sense disambiguation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.acl-long.325 · arXiv 2110.07681

@inproceedings{eyal2022large,
  title = {{Large Scale Substitution-based Word Sense Induction}},
  author = {Eyal, Matan and Sadde, Shoval and Taub-Tabib, Hillel and Goldberg, Yoav},
  booktitle = {ACL 2022 (Dublin)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.325},
  eprint = {2110.07681},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.acl-long.325}
}

References

47 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Reinald Kim Amplayo, Seung-won Hwang, and Min Song. 2019. Autosense model for word sense induction. In The Thirty-Third AAAI Conference on Artifi-a reproduction of outlier identification using non-contextual word representations. In Proceedings of the First Workshop on Evaluation and Comparison of N… find
2. Asaf Amrami and Yoav Goldberg. 2018. Word sense induction with neural biLM and symmetric patterns. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4860–4867, Brussels, Belgium. Association for Computational Linguistics. find
3. Asaf Amrami and Yoav Goldberg. 2019. Towards better substitution-based word sense induction. arXiv preprint arXiv:1905.12598. — [arXiv:1905.12598](https://arxiv.org/abs/1905.12598) find
4. Giusepppe Attardi. 2015. Wikiextractor. https:// github.com/attardi/wikiextractor. find
5. Osman Başkaya, Enis Sert, Volkan Cirik, and Deniz Yuret. 2013. AI-KU: Using substitute vectors and co-occurrence modeling for word sense induction and disambiguation. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Worksho… find
6. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: Pretrained language model for scientific text. In EMNLP. find
7. Philip Blair, Yuval Merhav, and Joel Barry. 2016. Automated generation of multilingual clusters for the evaluation of distributed representations. arXiv preprint arXiv:1611.01547. — [arXiv:1611.01547](https://arxiv.org/abs/1611.01547) find
8. Rexhina Blloshmi, Tommaso Pasini, Niccolò Campolungo, Somnath Banerjee, Roberto Navigli, and Gabriella Pasi. 2021. Ir like a sir: Sense-enhanced information retrieval for multiple languages. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1030– 1041. find
9. Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. 2008. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment, 2008(10):P10008. find
10. Anna Breit, Artem Revenko, Kiamehr Rezaee, Mohammad Taher Pilehvar, and Jose Camacho-Collados. 2021. WiC-TSV: An evaluation benchmark for target sense verification of words in context. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Mai… find
11. Jesper Brink Andersen, Mikkel Bak Bertelsen, Mikkel Hørby Schou, Manuel R. Ciosici, and Ira Assent. 2020. One of these words is not like the other: José Camacho-Collados and Roberto Navigli. 2016. Find the word that does not belong: A framework for an intrinsic evaluation of word vector representati… find
12. Jose Camacho-Collados and Mohammad Taher Pilehvar. 2018. From word to sense embeddings: A survey on vector representations of meaning. Journal of Artificial Intelligence Research, 63:743–788. find
13. Xinxiong Chen, Zhiyuan Liu, and Maosong Sun. 2014. A unified model for word sense representation and disambiguation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1025–1035, Doha, Qatar. Association for Computational Linguistics. find
14. Davide Colla, Enrico Mensa, and Daniele P. Radicioni. 2020. LessLex: Linking multilingual embeddings to SenSe representations of LEXical items. Computational Linguistics, 46(2):289–333. find
15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
16. Gertraud Fenk-Oczlon, August Fenk, and Pamela Faber. 2010. Frequency effects on the emergence of polysemy and homophony. International Journal of Information Technologies and Knowledge, 4(2):103– 109. find
17. Antoni Hernández-Fernández, Bernardino Casas, Ramon Ferrer-i Cancho, and Jaume Baixeries. 2016. Testing the robustness of laws of polysemy and brevity versus frequency. In International Conference on Statistical Language and Speech Processing, pages 19–29. Springer. find
18. Matthew Honnibal, Ines Montani, Sofie Van Lan-deghem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python. find
19. Eric Huang, Richard Socher, Christopher Manning, and Andrew Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 873–882, Jeju Island, Korea.… find
20. Ignacio Iacobacci and Roberto Navigli. 2019. Lst-membed: Learning word and sense representations from a large semantically annotated corpus with long short-term memories. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1685–1695. find
21. Ignacio Iacobacci, Mohammad Taher Pilehvar, and Roberto Navigli. 2015. SensEmbed: Learning sense embeddings for word and relational similarity. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Pr… find
22. David Jurgens. 2011. Word sense induction by community detection. In Proceedings of TextGraphs-6: Graph-based Methods for Natural Language Processing, pages 24–28, Portland, Oregon. Association for Computational Linguistics. find
23. David Jurgens and Ioannis Klapaftis. 2013. SemEval-2013 task 13: Word sense induction for graded and non-graded senses. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 2… find
24. Ioannis Klapaftis and Suresh Manandhar. 2010. Word sense induction & disambiguation using hierarchical random graphs. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 745–755, Cambridge, MA. Association for Computational Linguistics. find
25. Ioannis P Klapaftis and Suresh Manandhar. 2008. Word sense induction using graphs of collocations. In ECAI, pages 298–302. find
26. Alexandros Komninos and Suresh Manandhar. 2016. Structured generative models of continuous features for word sense induction. In Proceedings of COL-ING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3577–3587. find
27. Daniel Loureiro, Kiamehr Rezaee, Mohammad Taher Pilehvar, and Jose Camacho-Collados. 2021. Analysis and evaluation of language models for word sense disambiguation. Computational Linguistics, pages 1–55. find
28. Suresh Manandhar, Ioannis Klapaftis, Dmitriy Dligach, and Sameer Pradhan. 2010. SemEval-2010 task 14: Word sense induction &disambiguation. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 63–68, Uppsala, Sweden. Association for Computational Linguistics. find
29. Massimiliano Mancini, Jose Camacho-Collados, Ignacio Iacobacci, and Roberto Navigli. 2017. Embedding words and senses together via joint knowledge-enhanced training. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 100–111, Vancouver, Canada. Assoc… find
30. Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. context2vec: Learning generic context embedding with bidirectional LSTM. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 51–61, Berlin, Germany. Association for Computational Linguistics. find
31. Tomás Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings… find
32. George A. Miller. 1992. WordNet: A lexical database for English. In Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992. find our card
33. Andrea Moro, Alessandro Raganato, and Roberto Navigli. 2014. Entity linking meets word sense disambiguation: a unified approach. Transactions of the Association for Computational Linguistics, 2:231– 244. find
34. Roberto Navigli and Simone Paolo Ponzetto. 2010. BabelNet: Building a very large multilingual semantic network. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 216–225, Uppsala, Sweden. Association for Computational Linguistics. find
35. Tommaso Pasini and Roberto Navigli. 2017. Train-O-Matic: Large-scale supervised word sense disambiguation in multiple languages without manual training data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 78–88, Copenhagen, Denmark. Association for C… find
36. Maria Pelevina, Nikolay Arefiev, Chris Biemann, and Alexander Panchenko. 2016. Making sense of word embeddings. In Proceedings of the 1st Workshop on Representation Learning for NLP, pages 174– 183, Berlin, Germany. Association for Computational Linguistics. find
37. Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum… find
38. Mohammad Taher Pilehvar and Nigel Collier. 2016. De-conflated semantic representations. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1680–1690, Austin, Texas. Association for Computational Linguistics. find
39. Radim Řehůřek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pages 45–50, Valletta, Malta. ELRA. find
40. Joseph Reisinger and Raymond J. Mooney. 2010. Multi-prototype vector-space models of word meaning. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 109–117, Los Angeles, California. Association for Comput… find
41. Brian Roark and Eugene Charniak. 2000. Noun-phrase co-occurrence statistics for semi-automatic semantic lexicon construction. arXiv preprint cs/0008026. find
42. Sascha Rothe and Hinrich Schütze. 2015. AutoEx-tend: Extending word embeddings to embeddings for synsets and lexemes. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long P… find
43. Micah Shlain, Hillel Taub-Tabib, Shoval Sadde, and Yoav Goldberg. 2020. Syntactic search by example. In ACL. find
44. Dayu Yuan, Julian Richardson, Ryan Doherty, Colin Evans, and Eric Altendorf. 2016. Semi-supervised word sense disambiguation with neural models. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1374–1385, Osaka, Japan. find
45. The COLING 2016 Organizing Committee. find
46. Zhi Zhong and Hwee Tou Ng. 2010. It makes sense: A wide-coverage word sense disambiguation system for free text. In Proceedings of the ACL 2010 System Demonstrations, pages 78–83, Uppsala, Sweden. Association for Computational Linguistics. find
47. George Kingsley Zipf. 1945. The meaning-frequency relationship of words. The Journal of general psychology, 33(2):251–256. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lex-2022-eyal-substitution-wsi/