Word Sense Induction with Attentive Context Clustering

What it is about

ACCWSI induces the senses of an ambiguous lemma without a large pretrained model. A small word2vec model is trained on the corpus; for each sentence with the query lemma, cosine similarity between the lemma vector and each context word gives softmax weights, and the weighted sum is a context-aware vector. These vectors are clustered with DBSCAN, and eps is chosen by shrinking it until more than half the points are noise. On SemEval-2010 Task 14 the method reaches a supervised recall of 63.7% (all words), and 62.7% when trained on 2.6% of the data, against 62.4% for the best listed competitor. On Biblical Hebrew it separates two senses of khalal (dead body, desecrate) and five of zakar (male, memory). The word2vec window is 5 for English and 2 for Classical Hebrew.

Ideas

• I1. Set the word2vec window per language from a small gold check, not a constant: the authors found 2 for Classical Hebrew against 5 for English, which fits the short verse structure.
• I2. Keep the attention highlight as the explanation: the context words with the largest weights for each cluster say why a use was put in that sense. This gives a transparent sense label that world can store with each frame.
• I3. Build a small hand-checked gold set from the paper's own examples (khalal, zakar, and the "bank" test) and use it as a regression test for any sense-splitting rule, with a negative control (a shuffled-context run that must fail).
• I4. Use their eps rule (tighten until noise passes 50%) as a deterministic splitter, with BTreeMap ordering so the cut is reproducible.

For Mova

I1 and I2 apply to the lemma senses in The lexicon, and to world for sense frames. I3 is a test for the same. I4 is a candidate rule for macula dict lemma senses. The method is embedding-based, so it is a teacher-side or diagnostic tool, not part of the parser.

Topics: Word sense disambiguation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.46298/jdmdh.9175

@inproceedings{stekel2021word,
  title = {{Word Sense Induction with Attentive Context Clustering}},
  author = {Stekel, Moshe and Azaria, Amos and Gordin, Shai},
  booktitle = {Proceedings of the Workshop on Natural Language Processing for Digital Humanities},
  year = {2021},
  doi = {10.46298/jdmdh.9175},
  url = {https://doi.org/10.46298/jdmdh.9175}
}

References

19 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eneko Agirre and German Rigau. 1996. Word sense disambiguation using conceptual density. arXiv preprint cmp-lg/9606007. find
2. Asaf Amrami and Yoav Goldberg. 2019. Towards better substitution-based word sense induction. arXiv preprint arXiv:1905.12598. ↗
3. Stefan Bordag. 2006. Word sense induction: Triplet-based clustering and automatic evaluation. In 11th Conference of the European Chapter of the Association for Computational Linguistics. find
4. Baobao Chang, Wenzhe Pei, and Miaohong Chen. 2014. Inducing word sense with automatically learned hidden concepts. In Proceedings of COL-ING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 355–364. find
5. Jim Cowie, Joe Guthrie, and Louise Guthrie. 1992. Lexical disambiguation using simulated annealing. In COLING 1992 Volume 1: The 14th International Conference on Computational Linguistics. find
6. Matan Eyal, Shoval Sadde, Hillel Taub-Tabib, and Yoav Goldberg. 2021. Large scale substitution-based word sense induction. arXiv preprint arXiv:2110.07681. ↗ our card
7. Aysu Ezen-Can. 2020. A comparison of lstm and bert for small corpus. arXiv preprint arXiv:2009.05451. ↗
8. Evelina Fedorenko and Rosemary Varley. 2016. Language and thought are not the same thing: evidence from neuroimaging and neurological patients. Annals of the New York Academy of Sciences, 1369(1):132. find
9. Andrea Galassi, Marco Lippi, and Paolo Torroni. 2020. Attention in natural language processing. IEEE Transactions on Neural Networks and Learning Systems. find
10. Yoav Goldberg and Omer Levy. 2014. word2vec explained: deriving mikolov et al.’s negative-sampling word-embedding method. arXiv preprint arXiv:1402.3722. ↗
11. Greg Hamerly and Charles Elkan. 2004. Learning the k in k-means. Advances in neural information processing systems, 16:281–288. find
12. Dekang Lin. 1998. Automatic retrieval and clustering of similar words. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 2, pages 768–774. find
13. Suresh Manandhar, Ioannis Klapaftis, Dmitriy Dligach, and Sameer Pradhan. 2010. Semeval-2010 task 14: Word sense induction &disambiguation. In Proceedings of the 5th international workshop on semantic evaluation, pages 63–68. find
14. Patrick Pantel and Dekang Lin. 2002. Discovering word senses from text. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 613–619. find
15. Douglas A Reynolds. 2009. Gaussian mixture models. Encyclopedia of biometrics, 741:659–663. find
16. Erich Schubert, Jörg Sander, Martin Ester, Hans Peter Kriegel, and Xiaowei Xu. 2017. Dbscan revisited, revisited: why and how you should (still) use dbscan. ACM Transactions on Database Systems (TODS), 42(3):1–21. find
17. Hinrich Schütze. 1998. Automatic word sense discrimination. Computational linguistics, 24(1):97–123. find
18. Goldee Udani, Shachi Dave, Anthony Davis, and Tim Sibley. 2005. Noun sense induction using web search results. In Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pages 657– 658. find
19. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2021-stekel-word-sense-induction-attentive-context/