Do Pretrained Contextual Language Models Distinguish between Hebrew Homograph Analyses?

What it is about

Unpointed Hebrew is full of homographs whose analyses differ in segmentation (the prefix or suffix is or is not a word), in morphosyntax, or only in sense. The authors ask whether contextual embeddings (mBERT, HeBERT, AlephBERT) separate these analyses. They build 75 challenge sets from the 3,600 most frequent words, about 150K tagged sentences from news, Wikipedia, literature and social media, with 2–5 analyses per homograph. Each homograph gets its own MLP on the embedding of the target token (word-expert setup, 10-fold CV, macro-F1). AlephBERT is best and clearly beats the word2vec BiLSTM of their earlier study. It is strongest on segmentation and morphosyntax, weaker on pure sense, weaker with 4- or 5-way ambiguity and with 3-piece word-piece splits, and not helped by masking the target. With only 5 examples per analysis it stays close to full CV.

Ideas

• I1. Report each homograph score split into three buckets (segmentation, morphosyntax, sense), as the paper does. Do the same for tools/hbo unpointed mode, so a segmentation gain cannot hide a sense loss.
• I2. Few-shot check as a test: for each homograph, train the tools/hbo disambiguation rules on 5 examples per analysis and report accuracy against the full-data rule set. It shows how far a rule table can go with little data.
• I3. Stratify the evaluation by the frequency ratio of the two analyses (balanced vs skewed, 1:1 to 50:1), the way the paper plots it, so that skewed homographs are scored separately.

For Mova

This is a Modern Hebrew paper. I1 and I3 are measurement designs for tools/hbo (unpointed mode) and go into its test notes. I2 is a test for the unpointed segmenter tables (lex, pre) and the rules in Nouns, adjectives, pronouns and numerals and Prepositions, prefixes, conjunctions and particles once Mova reads Modern Hebrew (plan in 12. Mova and Modern Hebrew: where it stands, what it would need). Nothing here changes languages/hbo/.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.eacl-main.59

@inproceedings{shmidman2023do,
  title = {{Do Pretrained Contextual Language Models Distinguish between Hebrew Homograph Analyses?}},
  author = {Shmidman, Avi and Shmidman, Cheyn Shmuel and Bareket, Dan and Koppel, Moshe and Tsarfaty, Reut},
  booktitle = {Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.eacl-main.59},
  url = {https://doi.org/10.18653/v1/2023.eacl-main.59}
}

References

27 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mohammed Alaeddine Abderrahim and Mohammed El-Amine Abderrahim. 2022. Arabic word sense disambiguation for information retrieval. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 21(4). find
2. Meni Adler and Michael Elhadad. 2006. An unsupervised morpheme-based hmm for hebrew morphological disambiguation. In COLING/ACL 2006 - 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, COLING/ACL 2006 - 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, pages 665–672, United States. Association for Computational Linguistics (ACL). 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, COLING/ACL 2006 ; Conference date: 17-07-2006 Through 21-07-2006. find our card
3. Eneko Agirre and Philip Edmonds, editors. 2006. Word Sense Disambiguation: Algorithms and Applications, volume 33 of Text, Speech and Language Technology. Springer. find
4. Yonatan Belinkov. 2021. Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics, pages 1–13. find
5. Avihay Chriqui and Inbal Yahav. 2021. Hebert hebemo: a hebrew bert model and a tool for polarity analysis and emotion recognition. find our card
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. find
7. Ali Elkahky, Kellie Webster, Daniel Andor, and Emily Pitler. 2018. A challenge set and methods for noun-verb ambiguity. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2562–2572, Brussels, Belgium. Association for Computational Linguistics. find
8. Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34–48. find
9. Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating models’ local decision boundaries via contrast sets. find our card
10. Yoav Goldberg. 2019. Assessing bert’s syntactic abilities. CoRR, abs/1901.05287. ↗
11. Nizar Habash and Owen Rambow. 2005. Arabic tokenization, part-of-speech tagging and morphological disambiguation in one fell swoop. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 573–580, Ann Arbor, Michigan. Association for Computational Linguistics. find our card
12. Josef Klafka and Allyson Ettinger. 2020. Spying on your neighbors: Fine-grained probing of contextual embeddings for information about surrounding words. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4801–4811, Online. Association for Computational Linguistics. find
13. Stav Klein and Reut Tsarfaty. 2020. Getting the find
14. Laroussi Merhben, Zouaghi Anis, and Mounir Zrigui. 2010. Ambiguous arabic words disambiguation. pages 157–164. find
15. Laroussi Merhbene, Anis Zouaghi, and Mounir Zrigui. 2013. A semi-supervised method for Arabic word sense disambiguation using a weighted directed graph. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 1027–1031, Nagoya, Japan. Asian Federation of Natural Language Processing. find
16. Roberto Navigli. 2009. Word sense disambiguation: A survey. ACM Comput. Surv., 41. find
17. Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Shaked Greenfeld, and Reut Tsarfaty. 2021. Alephbert:a hebrew large pre-trained language model to start-off your hebrew nlp application with. find our card
18. Danny Shacham and Shuly Wintner. 2007. Morphological disambiguation of Hebrew: A case study in classifier combination. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 439– 447, Prague, Czech Republic. Association for Computational Linguistics. find our card
19. Rushin Shah, Paramveer S. Dhillon, Mark Liberman, Dean Foster, Mohamed Maamouri, and Lyle Ungar. 2010. A new approach to lexical disambiguation of Arabic text. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 725–735, Cambridge, MA. Association for Computational Linguistics. find
20. Avi Shmidman, Joshua Guedalia, Shaltiel Shmidman, Moshe Koppel, and Reut Tsarfaty. 2020. A novel challenge set for Hebrew morphological disambiguation and diacritics restoration. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3316–3326, Online. Association for Computational Linguistics. find our card
21. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. find
22. Reut Tsarfaty, Dan Bareket, Stav Klein, and Amit Seker. 2020. From SPMRL to NMRL: What did we learn (and unlearn) in a decade of parsing morphologically-rich languages (MRLs)? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7396–7408, Online. Association for Computational Linguistics. find
23. Reut Tsarfaty, Amit Seker, Shoval Sadde, and Stav Klein. 2019. What’s wrong with hebrew nlp? and how to make it right. find our card
24. Shuly Wintner. 2014. Morphological processing of semitic languages. In NLP of Semitic Languages. find
25. Yadollah Yaghoobzadeh, Katharina Kann, T. J. Hazen, Eneko Agirre, and Hinrich Schütze. 2019. Probing for semantic classes: Diagnosing the meaning content of word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5740–5753, Florence, Italy. Association for Computational Linguistics. find
26. Shlomo Yona and Shuly Wintner. 2005. A finite-state morphological grammar of Hebrew. In Proceedings of the ACL Workshop on Computational Approaches to Semitic Languages, pages 9–16, Ann Arbor, Michigan. Association for Computational Linguistics. find our card
27. Mengjie Zhao, Philipp Dufter, Yadollah Yaghoobzadeh, and Hinrich Schütze. 2020. Quantifying the contextualization of word representations with semantic class probing. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1219–1234, Online. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/eacl-2023-shmidman-do-pretrained-contextual-language-models/