Mova0.5 pre-release⌕
☰
home / docs / papers / Ancient Hebrew / A Novel Challenge Set for Hebrew Morphological Disambiguation and Diacritics Restoration

A Novel Challenge Set for Hebrew Morphological Disambiguation and Diacritics Restoration

What it is about

Hebrew homographs are often unbalanced: one analysis is 30 to 129 times more frequent than the other. A tagger can then score well on the majority reading and fail on the minority one, which is the reading that matters in rare cases. The authors build a challenge set of 21 homographs (12 unbalanced, 9 balanced) with more than 56K sentences from news, books and Wikipedia. Each analysis has at least 400 sentences; the minority analysis was found by an automatic diacritizer and then checked by student taggers and a Hebrew expert. YAP, the best Hebrew morphological tagger of the time, gives F1 below 0.8 on most unbalanced cases and as low as 0.1–0.4 on two pronoun/preposition pairs. Specialised classifiers (word2vec and morphological-lattice context, BiLSTM or concatenation) raise the average F1 from 0.67 to 0.95. A few collocations dominate the majority sentences, so each word-neighbour pair is capped at 20 sentences.

Ideas

For Mova

I1 and I2 are the test set for tools/hbo (unpointed mode): a contrast set for each minority reading of pairs such as את (object marker vs "you" f.) and the others in Pronouns, particles and prepositions. I3 is the scoring rule for the same test. The paper's per-homograph figures are a Modern Hebrew baseline for the same kind of pair (2. Script and spelling).

Topics: Part-of-speech tagging

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.findings-emnlp.297

@inproceedings{shmidman2020novel,
  title = {{A Novel Challenge Set for Hebrew Morphological Disambiguation and Diacritics Restoration}},
  author = {Shmidman, Avi and Guedalia, Joshua and Shmidman, Shaltiel and Koppel, Moshe and Tsarfaty, Reut},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.297},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.297}
}

References

30 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ray Fabri, Michael Gasser, Nizar Habash, George Kiraz, and Shuly Wintner. 2014. Linguistic introduction: The orthography, morphology and syntax of semitic languages. In Imed Zitouni, editor, Natural Language Processing of Semitic Languages, Theory and Applications of Natural Language Processing, pages 3–41. Springer. find
  2. Meni Adler and Michael Elhadad. 2006. An unsupervised morpheme-based HMM for Hebrew morphological disambiguation. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 665–672, Sydney, Australia. Association for Computational Linguistics. find our card
  3. Ali Fadel, Ibraheem Tuffaha, Bara’ Al-Jawarneh, and Mahmoud Al-Ayyoub. 2019a. Arabic text diacritization using deep neural networks. CoRR, abs/1905.01965. ↗
  4. Srinivas Bangalore and Aravind K. Joshi. 1999. Supertagging: An approach to almost parsing. Computational Linguistics, 25(2):237–265. find
  5. Ali Fadel, Ibraheem Tuffaha, Bara’ Al-Jawarneh, and Mahmoud Al-Ayyoub. 2019b. Neural arabic text diacritization: State of the art results and a novel approach for machine translation. Proceedings of the 6th Workshop on Asian Translation. find our card
  6. Mohamed Bebah, Amine Chennoufi, Azzeddine Mazroui, and Abdelhak Lakhouaja. 2014. Hybrid approaches for automatic vowelization of arabic texts. CoRR, abs/1410.2646. ↗
  7. Aviezri S. Fraenkel, David Raab, and Eliezer Spitz. 1979. Semi-automatic construction of semantic concordances. Computers and the Humanities, 13:283– 288. find
  8. Yonatan Belinkov and James R. Glass. 2015. Arabic diacritization with recurrent neural networks. In EMNLP. find our card
  9. Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating nlp models via contrast sets. find
  10. Steven Bird, Robert Dale, Bonnie Dorr, Bryan Gibson, Mark Joseph, Min-Yen Kan, Dongwon Lee, Brett Powley, Dragomir Radev, and Yee Tan. 2008. The acl anthology reference corpus: A reference dataset for bibliographic research in computational linguistics. find
  11. Ping Chen, Wei Ding, Chris Bowes, and David Brown. 2009. A fully unsupervised word sense disambiguation method using dependency knowledge. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 28–36, Boulder, Colorado. Association for Computational Linguistics. find
  12. Nizar Habash, Owen Rambow, and Ryan Roth. 2009. Mada+tokan: A toolkit for arabic tokenization, diacritization, morphological disambiguation, pos tagging, stemming and lemmatization. Proceedings of the 2nd International Conference on Arabic Language Resources and Tools (MEDAR). find
  13. Yaacov Choueka and Serge Lusignan. 1985. Disambiguation by short contexts. Computers and the Humanities, 19:147–157. find
  14. Marti A. Hearst. 1991. Noun homograph disambiguation using local context in large text corpora. In University of Waterloo, pages 1–22. find
  15. Paula Czarnowska, Sebastian Ruder, Edouard Grave, Ryan Cotterell, and Ann Copestake. 2019. Don’t forget the long tail! a comprehensive analysis of morphological generalization in bilingual lexicon induction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 974–983, Hong Kong, China. Association for Computational Linguistics. find
  16. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735–1780. find
  17. Alon Itai and Shuly Wintner. 2008. Language resources for hebrew. Language Resources and Evaluation, 42:75–98. find
  18. Kareem Darwish, Ahmed Abdelali, Hamdy Mubarak, and Mohamed Eldesouki. 2020. Arabic diacritic recovery using a feature-rich bilstm model. find
  19. Stav Klein and Reut Tsarfaty. 2020. Getting the find
  20. Ali Elkahky, Kellie Webster, Daniel Andor, and Emily Pitler. 2018. A challenge set and methods for noun-verb ambiguity. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2562–2572, Brussels, Belgium. Association for Computational Linguistics. find
  21. Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 302–308, Baltimore, Maryland. Association for Computational Linguistics. find
  22. Frederick Liu, Han Lu, and Graham Neubig. 2018. Handling homographs in neural machine translation. find
  23. Alexis Amid Neme and Sébastien Paumier. 2019. Restoring arabic vowels through omission-tolerant dictionary lookup. Language Resources and Evaluation, 54(2):487–551. find our card
  24. Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, Kevin Duh, Manaal Faruqui, Cynthia Gan, Dan Garrette, Yangfeng Ji, Lingpeng Kong, Adhiguna Kuncoro, Gaurav Kumar, Chaitanya Malaviya, Paul Michel, Yusuke Oda, Matthew Richardson, Naomi Saphra, Swabha Swayamdipta, and Pengcheng Yin. 2017. Dynet: The dynamic neural network toolkit. arXiv preprint arXiv:1701.03980. ↗
  25. Avi Shmidman, Shaltiel Shmidman, Moshe Koppel, and Yoav Goldberg. 2020. Nakdan: Professional Hebrew diacritizer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 197– 203, Online. Association for Computational Linguistics. find our card
  26. Reut Tsarfaty, Dan Bareket, Stav Klein, and Amit Seker. 2020. From SPMRL to NMRL: What did we learn (and unlearn) in a decade of parsing morphologically-rich languages (MRLs)? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7396– 7408, Online. Association for Computational Linguistics. find
  27. Reut Tsarfaty, Shoval Sadde, Stav Klein, and Amit Seker. 2019. What’s wrong with Hebrew NLP? and how to make it right. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pages 259–264, Hong Kong, China. Association for Computational Linguistics. find our card
  28. Yorick Wilks and Mark Stevenson. 1997. The grammar of sense: Using part-of-speech tags as a first step in semantic disambiguation. find
  29. David Yarowsky. 1994. Decision lists for lexical ambiguity resolution: Application to accent restoration in Spanish and French. In 32nd Annual Meeting of the Association for Computational Linguistics, pages 88–95, Las Cruces, New Mexico, USA. Association for Computational Linguistics. find
  30. George K. Zipf. 1949. Human Behaviour and the Principle of Least Effort. Addison-Wesley. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/findings-2020-shmidman-novel-challenge-set-hebrew-morphological/