Hebrew Diacritics Restoration using Visual Representation

What it is about

The system DiVRIT restores Hebrew niqqud at word level, as a candidate-selection task. A candidate generator (k-nearest neighbours over undiacritized forms of the same length, using the frequency-sorted pointing patterns of each neighbour) produces two or three pointed candidates per word. A visual encoder, a PIXEL-style vision transformer pretrained on rendered Hebrew, embeds each candidate as an image, and a text encoder (AlephBERT GIMMEL-small) embeds the context. The best candidate is picked by an inner product. Evaluated on a 20K-token Modern Hebrew test set, the model reaches 92.7% word accuracy in the oracle setting (the right form is always among the candidates) and 87.9% with its own generator. Nakdimon scores 89.8% and MenakBERT 94.1%, and the hybrid Dicta system 95.8%. Majority-class and one-neighbour baselines reach 84.9% and 87.1%. Auxiliary losses, balanced sampling and more candidates did not help; mirroring the candidate images helped at three candidates.

Ideas

• I1. A rule-based candidate generator: take all pointed forms of each consonantal word from a lexicon with frequencies (for example from macula dict), filter them by the shape of the word, and rank them by frequency. Measure coverage (how often the right form is in the set) against candidate-set size, as the paper does; its curve suggests three to four candidates.
• I2. Report the no-learning floor on the same test set: the most-frequent-pointing baseline and a one-neighbour baseline. Any Mova ranking must beat both, with word accuracy and the separate decision accuracy for dagesh and shin dots.
• I3. The oracle-versus-generator gap (about 5 points here) is the cost of candidate generation, not of scoring. For Mova, the first target is coverage of the candidate set, and the ranker comes second.

For Mova

I1 and I2 belong in the unpointed mode of tools/hbo and in 1. Script and pointing (what the niqqud encode, and which distinctions Modern and Biblical readings share). I3 is a plan for the order of work in tools/hbo. The paper is on Modern Hebrew, so its numbers are not a claim about Biblical or Rabbinic text.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.loreslm-1.44

@inproceedings{elboher2026hebrew,
  title = {{Hebrew Diacritics Restoration using Visual Representation}},
  author = {Elboher, Yair and Pinter, Yuval},
  booktitle = {Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.loreslm-1.44},
  url = {https://doi.org/10.18653/v1/2026.loreslm-1.44}
}

References

39 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yonatan Belinkov and James Glass. 2015. Arabic diacritization with recurrent neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2281– 2285, Lisbon, Portugal. Association for Computational Linguistics. find our card
2. Nadav Borenstein, Phillip Rust, Desmond Elliott, and Isabelle Augenstein. 2023. PHD: Pixel-based language modeling of historical documents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 87–107, Singapore. Association for Computational Linguistics. find
3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc. find
4. Wei-Rui Chen, Ife Adebara, and Muhammad Abdul-Mageed. 2024. Interplay of machine translation, diacritics, and diacritization. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7559–7601, Mexico City, Mexico. Association for Computational Linguistics. find
5. Ido Cohen, Jacob Gidron, and Idan Pinto. 2024. Menakbert – hebrew diacriticizer. arXiv preprint arXiv:2410.02417. ↗ our card
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
7. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. Preprint, arXiv:2010.11929. ↗
8. Salman Elgamal, Ossama Obeid, Mhd Kabbani, Go Inoue, and Nizar Habash. 2024. Arabic diacritics in the Bar Gazit, Shaltiel Shmidman, Avi Shmidman, and Yuval Pinter. 2025. Splintering nonconcatenative languages for better tokenization. In Findings of the Association for Computational Linguistics: ACL 2025, pages 22405–22417, Vienna, Austria. Association for Computational Linguistics. find
9. Elazar Gershuni and Yuval Pinter. 2022. Restoring Hebrew diacritics without a dictionary. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1010–1018, Seattle, United States. Association for Computational Linguistics. find our card
10. Kyle Gorman and Yuval Pinter. 2025. Don‘t touch my diacritics. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 285–291, Albuquerque, New Mexico. Association for Computational Linguistics. find
11. Eylon Gueta, Omer Goldman, and Reut Tsarfaty. 2023a. Explicit morphological knowledge improves pre-training of language models for hebrew. Preprint, arXiv:2311.00658. ↗ our card
12. Eylon Gueta, Avi Shmidman, Shaltiel Shmidman, Cheyn Shmuel Shmidman, Joshua Guedalia, Moshe Koppel, Dan Bareket, Amit Seker, and Reut Tsarfaty. 2023b. Large pre-trained models with extra-large vocabularies: A contrastive analysis of hebrew bert models and a new one to outperform them all. Preprint, arXiv:2211.15199. ↗ our card
13. Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16000–16009. find
14. Edward Klyshinsky, Olga Karpik, and Alexander Bondarenko. 2021. A comparison of neural networks architectures for diacritics restoration. In Recent Trends in Analysis of Images, Social Networks and Texts, pages 235–245. Springer. find
15. Vladimir I. Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10(8):707–710. find
16. Omer Levy, Yoav Goldberg, and Ido Dagan. 2015. Improving distributional similarity with lessons learned from word embeddings. Transactions of the Association for Computational Linguistics, 3:211–225. find
17. Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. In Advances in Neural Information Processing Systems, volume 34, pages 9694–9705. Curran Associates, Inc. find
18. Jonas Lotz, Elizabeth Salesky, Phillip Rust, and Desmond Elliott. 2023. Text rendering strategies for pixel language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10155–10172, Singapore. Association for Computational Linguistics. find
19. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc. find
20. Alberto Muñoz-Ortiz, Verena Blaschke, and Barbara Plank. 2025. Evaluating pixel language models on non-standardized languages. In Proceedings of the 31st International Conference on Computational Linguistics, pages 6412–6419, Abu Dhabi, UAE. Association for Computational Linguistics. find
21. Jakub Náplava, Milan Straka, and Jana Straková. 2021. Diacritics restoration using bert with analysis on czech language. Prague Bulletin of Mathematical Linguistics, 116(1):27–42. find
22. Pedro Javier Ortiz Suárez, Benoit Sagot, and Laurent Romary. 2019. Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures. In Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7), pages 9–16, Cardiff, UK. Leibniz-Institut für Deutsche Sprache. find
23. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR. find
24. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics. find
25. Qiang Ren and Junli Wang. 2025. Irrelevant patch-masked autoencoders for enhancing vision transformers under limited data. Knowledge-Based Systems, 310:112936. find
26. Phillip Rust, Jonas F. Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lhoneux, and Desmond Elliott. 2023. Language modelling with pixels. In The Eleventh International Conference on Learning Representations. find
27. Elizabeth Salesky, David Etter, and Matt Post. 2021. Robust open-vocabulary translation from visual text representations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7235–7252, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
28. Sara Shatnawi, Sawsan Alqahtani, and Hanan Aldarmaki. 2024. Automatic restoration of diacritics for speech data sets. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4166–4176, Mexico City, Mexico. Association for Computational Linguistics. find
29. Avi Shmidman, Shaltiel Shmidman, Moshe Koppel, and Yoav Goldberg. 2020. Nakdan: Professional Hebrew diacritizer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 197–203, Online. Association for Computational Linguistics. find our card
30. Leon Sick, Dominik Engel, Pedro Hermosilla, and Timo Ropinski. 2025. Attention-guided masked autoencoders for learning image representations. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 836–846. find
31. Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. 2013. Zero-shot learning through cross-modal transfer. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc. find
32. Yintao Tai, Xiyang Liao, Alessandro Suglia, and Antonio Vergari. 2024. PIXAR: Auto-regressive language modeling in pixel space. In Findings of the Association for Computational Linguistics: ACL 2024, pages 14673–14695, Bangkok, Thailand. Association for Computational Linguistics. find
33. Reut Tsarfaty, Shoval Sadde, Stav Klein, and Amit Seker. 2019. What‘s wrong with Hebrew NLP? and how to make it right. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pages 259–264, Hong Kong, China. Association for Computational Linguistics. find our card
34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Us-zoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. find
35. Yongqin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata. 2019. Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(9):2251–2265. find
36. Qinyuan Ye, Belinda Z. Li, Sinong Wang, Benjamin Bolte, Hao Ma, Wen-tau Yih, Xiang Ren, and Madian Khabsa. 2021. On the influence of masking policies in intermediate pre-training. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7190–7202, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
37. Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3914–3923, Hong Kong, China. Association for Computational Linguistics. find
38. Li Yujian and Liu Bo. 2007. A normalized levenshtein distance metric. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(6):1091–1095. find
39. Lei Zhou, Huidong Liu, Joseph Bae, Junjun He, Dimitris Samaras, and Prateek Prasanna. 2023. Self pre-training with masked autoencoders for medical image classification and segmentation. In 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), pages 1–6. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2026-elboher-hebrew-diacritics-restoration-visual-representation/