Nakdan: Professional Hebrew Diacritizer

What it is about

Nakdan adds vowel points to undiacritized Hebrew, for modern, Rabbinic and poetic texts, through a web interface that lets an editor choose among the alternatives. The pipeline has three stages: a POS tagger and a morphological tagger (bi-LSTMs with a character model, trained on about 200K hand-tagged tokens, 92% coarse-POS and 79% full-morphology accuracy); a filter that intersects each word's candidate points with a hand-built lexicon of about 50K lexemes and about 5.5M inflected forms plus proper-name lists; and a bi-LSTM that ranks the surviving candidates in context, with beam search over the points of each letter. On a 6,000-word modern test, word-level accuracy is 88.2%, against 80.9% for Morfix and 66.4% for Snopi. The authors also report that knowing the points cuts the morphological ambiguity from 9.1 to 2.4 analyses per form, and knowing the morphology cuts the point ambiguity from 6.2 to 1.4. Rabbinic and poetic genres get their own models, and the Rabbinic test (Bet Yosef) is harder for all systems.

Ideas

• I1. Filter, then rank, for the unpointed mode of tools/hbo: take the segmenter's candidate analyses for a word, keep only those the lightest tagger allows, and rank the survivors. Report the average number of candidates before and after each filter, as the paper does. A random choice from the filtered set gives 87.1% on the modern test, which is the baseline a rule-based ranking must beat.
• I2. A table of collocation overrides: the paper keeps about 10K hand-written pairs that fix the state of a word in a fixed phrase (the paper's example: the noun bayit is forced into its construct form inside one fixed two-word collocation, and into the plural construct inside its plural). Mova can keep such a table in The Noun, the Adjective and the Numeral with the reason for each entry.
• I3. Genre split: the paper states that the modern morphology model does not transfer to Rabbinic Hebrew and must be retrained on tagged historical text. Add a note to 11. Mishnaic Hebrew (hbo-x-mishnaic) that Mishnaic rules are tested on their own gold set, and do not reuse the Biblical tables unchanged.

For Mova

I1 is design for tools/hbo (unpointed mode) and its measure; I2 belongs in The Noun, the Adjective and the Numeral and Syntax: the clause and the sentence; I3 is a rule of testing for 11. Mishnaic Hebrew (hbo-x-mishnaic). The 6,000-word modern test corpus is released by the authors and can be used as a measurement of 2. Script and spelling.

Topics: Part-of-speech tagging

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-demos.23

@inproceedings{shmidman2020nakdan,
  title = {{Nakdan: Professional Hebrew Diacritizer}},
  author = {Shmidman, Avi and Shmidman, Shaltiel and Koppel, Moshe and Goldberg, Yoav},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations},
  year = {2020},
  doi = {10.18653/v1/2020.acl-demos.23},
  url = {https://doi.org/10.18653/v1/2020.acl-demos.23}
}

References

10 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gheith Abandah, Alex Graves, Balkees Al-Shagoor, Alaa Arabiyat, Fuad Jamour, and Majid Al-Taee. 2015. Automatic diacritization of arabic text using recurrent neural networks. International Journal on Document Analysis and Recognition (IJDAR), 18:183–197. find
2. Meni Adler and Michael Elhadad. 2006. An unsupervised morpheme-based HMM for hebrew morphological disambiguation. In ACL 2006, 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, Sydney, Australia, 17-21 July 2006. find our card
3. Yonatan Belinkov and James R. Glass. 2015. Arabic diacritization with recurrent neural networks. In EMNLP. find our card
4. Yaacov Choueka and Yoni Neeman. 1995. Nakdan-text,(an in-context text-vocalizer for modern hebrew). In BISFAI-95, The Fifth Bar Ilan Symposium for Artificial Intelligence. find
5. Michael Elhadad, Yael Netzer, David Gabay, and Meni Adler. 2005. Hebrew morphological tagging guidelines. Technical report, Technical report, Ben-Gurion University, Dept. of Computer Science. find
6. Ya’akov Gal. 2002. An hmm approach to vowel restoration in arabic and hebrew. In Proceedings of the ACL-02 workshop on Computational approaches to semitic languages, pages 1–7. Association for Computational Linguistics. find our card
7. Yoav Goldberg. 2017. Neural network methods for natural language processing. Synthesis Lectures on Human Language Technologies, 10(1):1–309. find
8. Amir More and Reut Tsarfaty. 2016. Data-driven morphological analysis and disambiguation for morphologically rich languages and universal dependencies. In Proceedings of COLING 2016. find
9. Hamdy Mubarak, Ahmed Abdelali, Hassan Sajjad, Younes Samih, and Kareem Darwish. 2019. Highly effective Arabic diacritization using sequence to sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2390–2395, Minneapolis, Minnesota. Association for Computational Linguistics. find
10. Mohsen Rashwan, Ahmad Sallab, Hazem Raafat, and Ahmed Rafea. 2015. Deep learning framework with confused sub-set resolution architecture for automatic arabic diacritization. IEEE Transactions on Audio Speech and Language Processing, 23. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2020-shmidman-nakdan-professional-hebrew-diacritizer/