Mova0.5 pre-release⌕
☰
home / docs / papers / English / WordNet under Scrutiny: Dictionary Examples in the Era of Large Language Models

WordNet under Scrutiny: Dictionary Examples in the Era of Large Language Models

What it is about

The paper asks how good the usage examples in WordNet are, compared with examples from other dictionaries and with examples generated by ChatGPT and GPT-3. Two annotators rated 87 words, giving 522 examples and 3,132 ratings on the GDEX criteria (naturalness, informativeness and similar). WordNet examples are rated highly natural but weak on informativeness, especially for the 45 words whose definitions were judged unclear. ChatGPT examples, including those filtered by GDEX-style criteria, were the most informative. On a word-similarity test with contextual embeddings built from examples, ChatGPT examples gave the best results on all datasets. Automatic GDEX-style scores were also computed for a larger set of examples.

Ideas

For Mova

The touch points are onto (WordNet data and onto/src/wsd.rs) and the annot teacher side for offline generation only. The paper's ratings and examples are CC BY-NC, so use them for tests and measurements, never in what is built. Measure first the effect of adding WordNet examples to the Lesk gloss bag on the deterministic WSD, with two identical runs.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.63317/4skerayorcxh

@inproceedings{almeman2024wordnet,
  title = {{WordNet under Scrutiny: Dictionary Examples in the Era of Large Language Models}},
  author = {Almeman, Fatemah Yousef and Schockaert, Steven and Anke, Luis Espinosa},
  booktitle = {LREC-COLING 2024 (Torino)},
  year = {2024},
  doi = {10.63317/4skerayorcxh},
  url = {https://doi.org/10.63317/4skerayorcxh}
}

References

90 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Asma Ben Abacha and Dina Demner-Fushman. 2019. A question-entailment approach to question answering. BMC Bioinformatics, 20(1). find
  2. Eneko Agirre and Philip Edmonds. 2007. Word sense disambiguation: Algorithms and applications, volume 33. Springer Science & Business Media. find
  3. Fatemah Almeman and Luis Espinosa-Anke. 2022. Putting wordnet’s dictionary examples in the context of definition modelling: An empirical analysis. In Proceedings of the Workshop on Cognitive Aspects of the Lexicon, pages 42–48. find
  4. Fatemah Almeman, Hadi Sheikhi, and Luis Espinosa Anke. 2023. 3D-EX: A unified dataset of definitions and dictionary examples. find
  5. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 451–462. find
  6. Tal August, Katharina Reinecke, and Noah A. Smith. 2022. Generating scientific definitions with controllable complexity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ana-pen ambiguity m-clause ↑ FKRGL DCR CLI 0.08 0.09 0.02 0.08… find
  7. Lenka Bajčetić and Thierry Declerck. 2022. Using Wiktionary to create specialized lexical resources and datasets. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3457–3460, Marseille, France. European Language Resources Association. find
  8. Satanjeev Banerjee and Ted Pedersen. 2002. An adapted lesk algorithm for word sense disambiguation using wordnet. volume 2276, pages 136–145. find
  9. Ahmet Basal. 2019. Learning collocations: Effects of online tools on teaching english adjective-noun collocations. British Journal of Educational Technology, 50(1):342–356. find
  10. Henri Bejoint. 2014. The bloomsbury companion to lexicography edited by howard jackson. Dictionaries: Journal of the Dictionary Society of North America, 35:374–381. find
  11. Michele Bevilacqua, Marco Maru, and Roberto Navigli. 2020. Generationary or “how we went beyond word sense inventories and learned to gloss”. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7207–7221, Online. Association for Computational Ling… find our card
  12. Thorsten Brants and Alex Franz. 2006. Web 1t 5-gram version 1 (2006). Linguistic Data Consortium, Philadelphia. find
  13. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. find
  14. Elia Bruni, Nam Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. J. Artif. Intell. Res., 49:1–47. find
  15. Meru Brunn, Yllias Chali, and Christopher Pinchak. 2001. Text summarization using lexical chains. find
  16. BSI. 1973a. Natural Fibre Twines, 3rd edition. British Standards Institution, London. BS 2570. find
  17. BSI. 1973b. Natural fibre twines. BS 2570, British Standards Institution, London. 3rd. edn. find
  18. Alexander Budanitsky and Graeme Hirst. 2001. Semantic distance in wordnet: An experimental, application-oriented evaluation of five measures. Workshop on WordNet and Other Lexical Resources. find
  19. Jose Camacho-Collados, Luis Espinosa Anke, and Mohammad Taher Pilehvar. 2018. The interplay between lexical resources and natural language processing. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorial Abstracts, pages 17–23… find
  20. A. Castor and L. E. Pollux. 1992. The use of user modelling to guide inference and learning. Applied Intelligence, 2(1):37–53. find
  21. Ting-Yun Chang and Yun-Nung Chen. 2019. What does this word mean? explaining contextualized embeddings with natural language definition. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing… find
  22. Qianglong Chen, Feng-Lin Li, Guohai Xu, Ming Yan, Ji Zhang, and Yin Zhang. 2022. Dictbert: Dictionary description knowledge enhanced language model pre-training via contrastive learning. In International Joint Conference on Artificial Intelligence. find
  23. J.L. Chercheur. 1994. Case-Based Reasoning, 2nd edition. Morgan Kaufman Publishers, San Mateo, CA. find
  24. N. Chomsky. 1973. Conditions on transformations. In A festschrift for Morris Halle, New York. Holt, Rinehart & Winston. find
  25. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, A… find
  26. Meri Coleman and Ta Lin Liau. 1975. A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60(2):283. find
  27. Edgar Dale and Jeanne S Chall. 1948. A formula for predicting readability: Instructions. Educational research bulletin, pages 37–54. find
  28. Claudio Delli Bovi, Luca Telesca, and Roberto Navigli. 2015. Large-scale information extraction from textual definitions through deep syntactic and semantic analysis. Transactions of the Association for Computational Linguistics, 3:529–543. find
  29. Umut Ufuk Demirhan. 2016. A frequency dictionary of turkish: Core vocabulary for learners. find
  30. Jörg Didakowski, Lothar Lemnitzer, and Alexander Geyken. 2012. Automatic example sentence extraction for a contemporary german dictionary. In Proceedings EURALEX, pages 343–349. find
  31. Umberto Eco. 1990. The Limits of Interpretation. Indian University Press. find
  32. Liana Ermakova, Tristan Miller, Anne-Gwenn Bosser, Victor Manuel Palma Preciado, Grigori Sidorov, and Adam Jatowt. 2023. Overview of joker–clef-2023 track on automatic wordplay analysis. In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 397–415. Springe… find
  33. Luis Espinosa-Anke, Jose Camacho-Collados, Sara Rodríguez Fernández, Horacio Saggion, and Leo Wanner. 2016a. Extending wordnet with fine-grained collocational information via supervised distributional learning. In Proceedings of COLING 2016: Technical Papers. The 26th International Conference on Com… find
  34. Luis Espinosa-Anke, Horacio Saggion, Francesco Ronzano, and Roberto Navigli. 2016b. Extasem! extending, taxonomizing and semantifying domain terminologies. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30. find
  35. Manaal Faruqui, Jesse Dodge, Sujay Jauhar, Chris Dyer, Eduard Hovy, and Noah Smith. 2014. Retrofitting word vectors to semantic lexicons. find
  36. Christiane Fellbaum. 2001. Manual and automatic semantic annotation with wordnet. In Proceedings of the NAACL WordNet and Other Lexical Resources workshop, page 3 – 10, Pittsburgh. find
  37. Christiane Fellbaum. 2013. Wordnet. In Carol Chapelle, editor, The encyclopedia of applied linguistics, pages 6739–6746. Blackwell Publishing Ltd. find
  38. Rudolph Flesch. 1948. A new readability yardstick. Journal of applied psychology, 32(3):221. find
  39. Ana Frankenberg-Garcia, Robert Lew, Jonathan C Roberts, Geraint Paul Rees, and Nirwan Sharma. 2019. Developing a writing assistant to help eap writers with collocations in real time. ReCALL, 31(1):23–39. find
  40. Ana Frankenberg-Garcia, Geraint Paul Rees, and Robert Lew. 2020. Slipping Through the Cracks in e-Lexicography. International Journal of Lexicography, 34(2):206–234. find
  41. Artyom Gadetsky, Ilya Yakubovskiy, and Dmitry Vetrov. 2018. Conditional generators of words definitions. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 266– 271. find
  42. Daniela Gerz, Ivan Vulić, Felix Hill, Roi Reichart, and Anna Korhonen. 2016. SimVerb-3500: A large-scale evaluation set of verb similarity. find
  43. Mario Giulianelli, Iris Luden, Raquel Fernandez, and Andrey Kutuzov. 2023. Interpretable word sense representations via definition generation: The case of semantic change analysis. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pag… find our card
  44. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. find
  45. Paul Gerhard Hoel. 1971a. Elementary Statistics, 3rd edition. Wiley series in probability and mathematical statistics. Wiley, New York, Chichester. ISBN 0 471 40300. find
  46. Paul Gerhard Hoel. 1971b. Elementary Statistics, 3rd edition, Wiley series in probability and mathematical statistics, pages 19–33. Wiley, New York, Chichester. ISBN 0 471 40300. find
  47. Eduard Hovy, Roberto Navigli, and Simone Paolo Ponzetto. 2013. Collaboratively built semi-structured content and artificial intelligence: The story so far. Artificial Intelligence, 194:2–27. find
  48. Eric H. Huang, Richard Socher, Christopher D. Manning, and Andrew Y. Ng. 2012. Improving word representations via global context and multiple word prototypes. In Annual Meeting of the Association for Computational Linguistics (ACL). find
  49. Mandar Joshi, Kenton Lee, Yi Luan, and Kristina Toutanova. 2020. Contextualized representations using textual encyclopedic knowledge. find
  50. Adam Kilgarriff, Milos Husák, Katy McAdam, Michael Rundell, and Pavel Rychlỳ. 2008. Gdex: Automatically finding good dictionary examples in a corpus. In Proceedings of the XIII EURALEX international congress, volume 1, pages 425–432. Universitat Pompeu Fabra Barcelona. find our card
  51. Adam Kilgarriff, Pavel Rychlý, Pavel Smrz, and David Tugwell. 2004. The sketch engine. In Proceedings of the 11th EURALEX International Congress, pages 105–115, Lorient, France. find
  52. J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report, Naval Technical Training Command Millington TN Research B… find
  53. Iztok Kosem, Milos Husák, and Diana McCarthy. 2011. Gdex for slovene. Proceedings of eLex, pages 151–159. find
  54. Iztok Kosem, Kristina Koppel, Tanara Zingano Kuhn, Jan Michelfeit, and Carole Tiberius. 2019. Identification and automatic extraction of good dictionary examples: the case (s) of gdex. International Journal of Lexicography, 32(2):119–137. find
  55. Oi Yee Kwong. 2001. Word sense disambiguation with an integrated lexical resource. In Proceedings of the NAACL WordNet and Other Lexical Resources workshop, page 11 – 16, Pittsburgh. find
  56. Lothar Lemnitzer, Christian Pölitz, Jörg Didakowski, and Alexander Geyken. 2015. Combining a rule-based approach and machine learning in a good-example extraction task for the purpose of lexicographic work on contemporary standard german. In Proceedings of the eLex 2015 conference, pages 11–13. find
  57. Jiayi Liao, Xu Chen, and Lun Du. 2023. Concept understanding in large language models: An empirical study. find
  58. Qianchu Liu, Fangyu Liu, Nigel Collier, Anna Korhonen, and Ivan Vulić. 2021. Mirrorwic: On eliciting word-in-context representations from pretrained language models. In Proceedings of the 25rd Conference on Computational Natural Language Learning (CoNLL). find
  59. Nikola Ljubešić and Mario Peronja. 2015. Predicting corpus example quality via supervised machine learning. In Proc. Electronic Lexicography in the 21st Century Conference (eLex), pages 477–485. find
  60. Margaret Masterman. 1957. The thesaurus in syntax and semantics. Mech. Transl. Comput. Linguistics, 4:35–43. find
  61. Timothee Mickus, Kees Van Deemter, Mathieu Constant, and Denis Paperno. 2022. Semeval-2022 task 1: CODWOE – comparing dictionaries and word embeddings. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1–14, Seattle, United States. Association for Computa… find
  62. D.I. Moldovan and Rada Mihalcea. 2000. Using wordnet and lexical operators to improve internet searches. Internet Computing, IEEE, 4:34 – 43. find
  63. Jorge Morato, Miguel Marzal, Juan Llorens, and Jos Moreiro. 2004. Wordnet applications. Proceedings of the 2nd Global Wordnet Conference, 2004. find
  64. Nikola Mrkšić, Ivan Vulić, Diarmuid O Séaghdha, Ira Leviant, Roi Reichart, Milica Gašić, Anna Korhonen, and Steve Young. 2017. Semantic specialization of distributional word vector spaces using monolingual and cross-lingual constraints. Transactions of the association for Computational Linguistics,… find
  65. Thanapon Noraset, Chen Liang, Larry Birnbaum, and Doug Downey. 2017. Definition modeling: Learning to define word embeddings in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31. find
  66. M. Pasca and S. Harabagiu. 2001. The informative role of wordnet in open-domain question answering. Workshop on WordNet and Other Lexical Resources at NAACL. find
  67. M Phoodai, Richárd Rikk, M Medveď, M Měchura, I Kosem, J Kallas, C Tiberius, and M Jakubíček. 2023. Exploring the capabilities of chatgpt for lexicographical purposes: A comparison with oxford advanced learner’s dictionary within the microstructural framework. In Proceedings of the eLex 2023 Confere… find
  68. Ildikó Pilán, Elena Volodina, and Lars Borin. 2016. Candidate sentence selection for language learning exercises: from a comprehensive framework to an empirical evaluation. Traitement Automatique des Langues, 57(3):67–91. find
  69. Ildikó Pilán, Elena Volodina, and Richard Johansson. 2013. Automatic selection of suitable sentences for language learning exercises. In 20 Years of EUROCALL: Learning from the Past, Looking to the Future: 2013 EUROCALL Conference Proceedings, pages 218–225. Research-publishing. net Dublin. find
  70. Ildikó Pilán, Elena Volodina, and Richard Johansson. 2014. Rule-based and machine learning approaches for second language sentence-level readability. In Proceedings of the ninth workshop on innovative use of NLP for building educational applications, pages 174–184. find
  71. Xiao Pu, Lin Yuan, Jiaxu Leng, Tao Wu, and Xinbo Gao. 2023. Lexical knowledge enhanced text matching via distilled word sense disambiguation. Knowledge-Based Systems, page 110282. find
  72. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. find
  73. Geraint Paul Rees and Robert Lew. 2023. The effectiveness of openai gpt-generated definitions versus definitions from an english learners’ dictionary in a lexically orientated reading task. International Journal of Lexicography, page ecad030. find
  74. Sreelekha S and Pushpak Bhattacharyya. 2016. Lexical resources to enrich english malayalam machine translation. find
  75. H. Gregory Silber and Kathleen F. McCoy. 2002. Efficiently computed lexical chains as an intermediate representation for automatic text summarization. Computational Linguistics, 28(4):487– 496. find
  76. Charles Joseph Singer, E. J. Holmyard, and A. R. Hall, editors. 1954–58. A history of technology. Oxford University Press, London. 5 vol. find
  77. Edgar A Smith and J Peter Kincaid. 1970. Derivation and validation of the automated readability index for use with technical materials. Human factors, 12(5):457–564. find
  78. Karen Sparck Jones. 1986. Synonymy and semantic classification. Edinburgh University Press. find
  79. Jannik Strötgen and Michael Gertz. 2012. Temporal tagging on different domains: Challenges, strategies, and gold standards. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), pages 3746– 3753, Istanbul, Turkey. European Language Resource Association… find
  80. S. Superman, B. Batman, C. Catwoman, and S. Spiderman. 2000. Superheroes experiences with books, 20th edition. The Phantom Editors Associates, Gotham City. find
  81. Ivan Vulić and Nikola Mrkšić. 2018. Specialising word vectors for lexical entailment. pages 1134– 1145. find
  82. Noah Webster. 1900. Webster’s unabridged dictionary of the English language. Kikwansha. find
  83. Steven R. Wilson, Walid Magdy, Barbara McGillivray, Venkata Rama Kiran Garimella, and Gareth Tyson. 2020. Urban dictionary embeddings for slang nlp applications. In International Conference on Language Resources and Evaluation. find
  84. Hongyuan Xu, Yunong Chen, Zichen Liu, Yanlong Wen, and Xiaojie Yuan. 2022. Taxoprompt: A prompt-based generation method with taxonomic context for self-supervised taxonomy expansion. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 4432–44… find
  85. Tae Yano and Moonyoung Kang. 2016. Taking advantage of wikipedia in natural language processing. find
  86. Guobiao Zhang, Wenpeng Lu, Xueping Peng, Shoujin Wang, Baoshuo Kan, and Rui Yu. 2022. Word sense disambiguation with knowledge-enhanced and local self-attention-based extractive sense comprehension. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4061–4070. find
  87. Lei Zhang, Fanchao Qi, Zhiyuan Liu, Yasheng Wang, Qun Liu, and Maosong Sun. 2020. Multi-channel reverse dictionary model. find
  88. Ruimin Zhu, Thanapon Noraset, Alisa Liu, Wenxin Jiang, and Doug Downey. 2019. Multi-sense definition modeling using word sense decompositions. 9. find
  89. Language Resource References Miller, George. 1995. WordNet: A Lexical Database for English. [link]. find our card
  90. Oxford Dictionary, English. 1989. Oxford english dictionary. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lex-2024-almeman-wordnet-examples-llm/