Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison

What it is about

Results for all-words word sense disambiguation (WSD) are hard to compare, because the datasets, sense-inventory versions and preprocessing differ. The paper converts five Senseval and SemEval all-words sets (7,253 instances across nouns, verbs, adjectives and adverbs) to one XML format, maps their sense labels to WordNet 3.0 by a semi-automatic mapping checked by hand, and preprocesses everything with the same Stanford CoreNLP pipeline. The training corpora are SemCor (manually tagged, about 226K annotations) and SemCor plus OMSTI (an automatically labelled corpus of about 0.9M annotations). Six systems are compared under one protocol: three supervised (IMS with its default features, IMS with word embeddings, Context2Vec), three knowledge-based (extended Lesk, UKB, Babelfy), and the most-frequent-sense and first-WordNet-sense baselines, scored by F1. Supervised systems beat knowledge-based ones; IMS with embeddings in place of surrounding-word features is best (69.7 F1 on the pooled set, trained on SemCor plus OMSTI); the WordNet first sense (65.2) is hard to beat, and only Babelfy is above it (65.5). Verbs stay below 58% for every system, and the supervised systems choose the most frequent sense for more than 75% of instances.

Ideas

• I1. Run onto/src/wsd.rs under the paper's protocol. Convert a test set to the SemEval-13 all-words XML, map the senses to the inventory Mova uses, and report the most-frequent-sense and first-sense baselines on the same file in the same run. No WSD number is reported without these two lines. Measure F1 per part of speech; run twice and compare the output files.
• I2. Report verbs apart from nouns, with the ambiguity beside each score (the paper gives 4.8 for nouns and 10.4 for verbs). A verb with more than ten senses is too fine for a sense list (the paper's example, keep, has 22 WordNet senses). Test whether a frame choice from prag/src/frames.rs beats the first sense on those verbs; if it does not, the explanation says that the first sense was used.
• I3. A transparent local-context scorer, as the paper's strong baseline. The linear classifier uses words at distance one to three, collocations and the parts of speech of the neighbours. In Rust, write it as a table of feature weights, where each feature prints its vote in the explanation. Train on SemCor only and compare with the most-frequent-sense baseline on the held-out sets. Negative control: train again with shuffled sense labels; the score must fall to the baseline or below, or the gate is broken.
• I4. A prior-bias audit. Count how often the chosen sense is the first sense on each test set; the paper finds over 75% for the IMS-based systems. If onto shows the same share, the result is mostly a prior and the explanation must say so. Report the share next to every accuracy.

For Mova

The code is onto/src/wsd.rs (WSD v1: Lesk, prior, core and domain terms, as described in Smart annotator — layers, context, memory, the WSD v1 entry) and en/src/ctxeval.rs for the context features. The chapter is 19. The lexicon: vocabulary, multiword expressions and register. Measures: the baselines on the same file (I1), the per-part-of-speech split (I2), the shuffled-label control (I3) and the first-sense share (I4). The frozen 60 sentences stay frozen: they are not used for tuning. Licence: the paper is CC BY 4.0, but it does not give the licences of the datasets it uses (Senseval, SemEval, SemCor, OMSTI, which is built from a parallel corpus). Check each one in Data and model licences before use. Until then they are for tests only, and nothing from them enters a release.

Topics: Word sense disambiguation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{raganato2017word,
  title = {{Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison}},
  author = {Raganato, Alessandro and Camacho-Collados, Jose and Navigli, Roberto},
  booktitle = {Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers},
  year = {2017},
  url = {https://aclanthology.org/E17-1010/}
}

References

54 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eneko Agirre and Aitor Soroa. 2009. Personalizing PageRank for Word Sense Disambiguation. In Proceedings of EACL, pages 33–41. find
2. Eneko Agirre, Oier Lopez De Lacalle, Christiane Fellbaum, Andrea Marchetti, Antonio Toral, and Piek Vossen. 2010a. Semeval-2010 task 17: All-words word sense disambiguation on a specific domain. In Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Directions, pages 123–128. find
3. Eneko Agirre, Oier Lopez De Lacalle, Christiane Fellbaum, Andrea Marchetti, Antonio Toral, and Piek Vossen. 2010b. Semeval-2010 task 17: All-words word sense disambiguation on a specific domain. In Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Directions, pages 123–128. 20 The poor performance of Lesk ext +emb on adjective instances is particularly noticeable. 21 We have additionally set up a CodaLab competition based on this evaluation framework. find
4. Eneko Agirre, Oier Lopez de Lacalle, and Aitor Soroa. 2014. Random walks for knowledge-based word Philip Edmonds and Scott Cotton. 2001. Senseval-2: Overview. In Proceedings of The Second International Workshop on Evaluating Word Sense Disambiguation Systems, pages 1–6, Toulouse, France. sense disambiguation. Computational Linguistics, 40(1):57–84. find
5. Satanjeev Banerjee and Ted Pedersen. 2003. Extended gloss overlap as a measure of semantic relatedness. In Proceedings of the 18th International Joint Conference on Artificial Intelligence, pages 805–810, Acapulco, Mexico. find
6. Steffen Eger, Rüdiger Gleim, and Alexander Mehler. 2016. Lemmatization and morphological tagging in german and latin: A comparison and a survey of the state-of-the-art. In Proceedings of LREC 2016. find
7. Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The wacky wide web: a collection of very large linguistically processed web-crawled corpora. Language resources and evaluation, 43(3):209–226. find
8. Andreas Eisele and Yu Chen. 2010. MultiUN: A Multilingual Corpus from United Nation Documents. In Proceedings of the Seventh conference on International Language Resources and Evaluation, pages 2868–2872. find
9. Pierpaolo Basile, Annalina Caputo, and Giovanni Semeraro. 2014. An Enhanced Lesk Word Sense Disambiguation Algorithm through a Distributional Semantic Model. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1591–1600, Dublin, Ireland. find
10. Weiwei Guo and Mona T. Diab. 2010. Combining orthogonal monolingual and multilingual sources of evidence for all words WSD. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1542–1551, Uppsala, Sweden. find
11. Osman Başkaya and David Jurgens. 2016. Semi-supervised learning with induced word senses for state of the art word sense disambiguation. Journal of Artificial Intelligence Research, 55:1025–1058. find
12. Taher H. Haveliwala. 2002. Topic-sensitive PageRank. In Proceedings of the 11th International Conference on World Wide Web, pages 517–526, Hawaii, USA. find
13. Hiram Calvo and Alexander Gelbukh. 2015. Is the most frequent sense of a word better connected in a semantic network? In International Conference on Intelligent Computing, pages 491–499. Springer. find
14. Ignacio Iacobacci, Mohammad Taher Pilehvar, and Roberto Navigli. 2016. Embeddings for word sense disambiguation: An evaluation study. In Proceedings of ACL, pages 897–907, Berlin, Germany. find
15. José Camacho-Collados, Mohammad Taher Pilehvar, and Roberto Navigli. 2015. A Unified Multilingual Semantic Representation of Concepts. In Proceedings of ACL, pages 741–751. find
16. Karen Spärck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28:11–21. find
17. Mikael Kågebäck and Hans Salomonsson. 2016. Word sense disambiguation using a bidirectional lstm. arXiv preprint arXiv:1606.03568. ↗
18. José Camacho-Collados, Claudio Delli Bovi, Alessandro Raganato, and Roberto Navigli. 2016a. A Large-Scale Multilingual Disambiguation of Glosses. In Proceedings of LREC, pages 1701– 1708, Portoroz, Slovenia. find
19. L. Y. Keok and H. T. Ng. 2002. An empirical evaluation of knowledge sources and learning algorithms for word sense disambiguation. In Proceedings of the 7 th Conference on Empirical Methods in Natural Language Processing, pages 41–48, Philadelphia, USA. find
20. José Camacho-Collados, Mohammad Taher Pilehvar, and Roberto Navigli. 2016b. Nasari: Integrating explicit knowledge and corpus statistics for a multilingual representation of concepts and entities. Artificial Intelligence, 240:36–64. find
21. Michael Lesk. 1986. Automatic sense disambiguation using machine readable dictionaries: How to tell a pine cone from an ice cream cone. In Proceedings of the 5th Annual Conference on Systems Documentation, Toronto, Ontario, Canada, pages 24–26. find
22. Yee Seng Chan and Hwee Tou Ng. 2005. Scaling up word sense disambiguation via parallel texts. In AAAI, volume 5, pages 1037–1042. find
23. Xinxiong Chen, Zhiyuan Liu, and Maosong Sun. 2014. A unified model for word sense representation and disambiguation. In Proceedings of EMNLP, pages 1025–1035, Doha, Qatar. find
24. Jordi Daude, Lluis Padro, and German Rigau. 2003. Validation and tuning of wordnet mapping techniques. In Proceedings of RANLP. find
25. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60. find
26. Oier Lopez de Lacalle and Eneko Agirre. 2015. A methodology for word sense disambiguation at 90% based on large-scale crowdsourcing. Lexical and Computational Semantics (* SEM 2015), page 61. find
27. Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. context2vec: Learning generic context embedding with bidirectional lstm. In Proceedings of CONLL. find
28. Simone Paolo Ponzetto and Roberto Navigli. 2010. Knowledge-rich Word Sense Disambiguation rivaling supervised system. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1522–1531, Uppsala, Sweden. find
29. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781. ↗
30. George A Miller, Martin Chodorow, Shari Landes, Claudia Leacock, and Robert G Thomas. 1994. Using a semantic concordance for sense identification. In Proceedings of the workshop on Human Language Technology, pages 240–243. Association for Computational Linguistics. find
31. Marten Postma, Ruben Izquierdo, Eneko Agirre, German Rigau, and Piek Vossen. 2016. Addressing the MFS Bias in WSD systems. In Proceedings of LREC, Portoroz, Slovenia. find
32. Tristan Miller, Chris Biemann, Torsten Zesch, and Iryna Gurevych. 2012. Using distributional similarity for lexical expansion in knowledge-based word sense disambiguation. In COLING, pages 1781– 1796. find
33. Sameer Pradhan, Edward Loper, Dmitriy Dligach, and Martha Palmer. 2007. SemEval-2007 task-17: English lexical sample, SRL and all words. In Proceedings of SemEval, pages 87–92. find
34. George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41. ↗ our card
35. Alessandro Raganato, Claudio Delli Bovi, and Roberto Navigli. 2016. Automatic Construction and Evaluation of a Large Semantically Enriched Wikipedia. In Proceedings of IJCAI, pages 2894–2900, New York City, NY, USA, July. find
36. Andrea Moro and Roberto Navigli. 2015. Semeval-2015 task 13: Multilingual all-words sense disambiguation and entity linking. Proceedings of SemEval-2015. find
37. Sascha Rothe and Hinrich Schütze. 2015. Autoextend: Extending word embeddings to embeddings for synsets and lexemes. In Proceedings of ACL, pages 1793–1803, Beijing, China. find
38. Andrea Moro, Alessandro Raganato, and Roberto Navigli. 2014. Entity Linking meets Word Sense Disambiguation: a Unified Approach. Transactions of the Association for Computational Linguistics (TACL), 2:231–244. find
39. Hui Shen, Razvan Bunescu, and Rada Mihalcea. 2013. Coarse to fine grained sense disambiguation in wikipedia. Proc. of *SEM, pages 22–31. find
40. Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial Intelligence, 193:217– 250. find
41. Benjamin Snyder and Martha Palmer. 2004. The English all-words task. In Proceedings of the 3rd International Workshop on the Evaluation of Systems for the Semantic Analysis of Text (SENSEVAL-3), Barcelona, Spain, pages 41–43, Barcelona, Spain. find
42. Roberto Navigli, Kenneth C. Litkowski, and Orin Hargraves. 2007. SemEval-2007 task 07: Coarse-grained English all-words task. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), Prague, Czech Republic, pages 30–35. find
43. Kaveh Taghipour and Hwee Tou Ng. 2015a. One million sense-tagged instances for word sense disambiguation and induction. CoNLL 2015, page 338. find
44. Kaveh Taghipour and Hwee Tou Ng. 2015b. Semi-supervised word sense disambiguation using word embeddings in general and specific domains. Proceedings of NAACL HLT 2015, pages 314–323. find
45. Roberto Navigli, David Jurgens, and Daniele Vannella. 2013. SemEval-2013 Task 12: Multilingual Word Sense Disambiguation. In Proceedings of SemEval 2013, pages 222–231. find
46. Hanghang Tong, Christos Faloutsos, and Jia-Yu Pan. 2006. Fast random walk with restart and its applications. In ICDM, pages 613–622. find
47. Roberto Navigli. 2009. Word Sense Disambiguation: A survey. ACM Computing Surveys, 41(2):1–69. find
48. Rocco Tripodi and Marcello Pelillo. 2016. A game-theoretic approach to word sense disambiguation. arXiv preprint arXiv:1606.07711. ↗
49. Roberto Navigli. 2012. A quick tour of word sense disambiguation, induction and related approaches. In SOFSEM 2012: Theory and practice of computer science, pages 115–129. Springer. find
50. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2011. A universal part-of-speech tagset. arXiv preprint arXiv:1104.2086. ↗
51. Dirk Weissenborn, Leonhard Hennig, Feiyu Xu, and Hans Uszkoreit. 2015. Multi-Objective Optimization for the Joint Disambiguation of Nouns and Named Entities. In Proceedings of ACL, pages 596– 605, Beijing, China. find
52. Mohammad Taher Pilehvar and Roberto Navigli. 2014. A large-scale pseudoword-based evaluation framework for state-of-the-art Word Sense Disambiguation. Computational Linguistics, 40(4). find
53. Dayu Yuan, Julian Richardson, Ryan Doherty, Colin Evans, and Eric Altendorf. 2016. Semi-supervised word sense disambiguation with neural models. In Proceedings of COLING, pages 1374–1385. find
54. Zhi Zhong and Hwee Tou Ng. 2010. It Makes Sense: A wide-coverage Word Sense Disambiguation system for free text. In Proceedings of the ACL System Demonstrations, pages 78–83. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/onto-2017-raganato-word-sense-disambiguation-unified-evaluation/