NeoDictaBERT: Pushing the Frontier of BERT models for Hebrew

What it is about

The authors move the NeoBERT encoder design (pre-norm RMSNorm, RoPE, SwiGLU, 28 layers at width 768, 4,096-token context) to Hebrew. Two masked-language models are trained: NeoDictaBERT on Hebrew only (about 232B tokens in a first phase and 46B in a second), and NeoDictaBERT-bilingual on a 60–40 English–Hebrew mix with a new 128k WordPiece vocabulary. Every selected token is masked, at a 20% rate. On the Hebrew UD treebank (IAHLT revision) NeoDictaBERT reaches 91.77 UAS and 87.80 LAS, against 90.52 and 86.54 for DictaBERT. The largest gain is in question answering (exact match 76.40 against 70.35 for DictaBERT). The bilingual model is best on most rows, and its English retrieval scores on MTEB v2 are close to other leading encoders.

Ideas

• I1. Report Mova's Modern Hebrew parsing numbers on the same treebank and split as these baselines (UAS 91.77, LAS 87.80), so that any comparison is like for like.
• I2. Keep the paper's NER set (nine categories, about 6,200 sentences) as a held-out check for a future rule-based Modern Hebrew entity tagger. The paper's F1 values (DictaBERT 87.01, NeoDictaBERT 86.97) are a reference line, not a target to copy.
• I3. Test the prefix handling: check how the tokenizer splits prefixed words (the letters for "and", "the", "to", "from", "in", "like") against the prefix tables learned in tools/hbo. Count the words where the split differs. This tests segmentation coverage with no model in the loop.

For Mova

I1 goes into treebanks/he/ and 11. Modern Hebrew in Universal Dependencies. I2 is a measurement for a future Modern Hebrew entity check, not a model. I3 is a test for Prepositions, prefixes, conjunctions and particles, and the prefix tables are compared with the segmenter in tools/hbo. The encoder itself is a teacher only and never runs at test time.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2510.20386

@misc{shmidman2025neodictabert,
  title = {{NeoDictaBERT: Pushing the Frontier of BERT models for Hebrew}},
  author = {Shmidman, Shaltiel and Shmidman, Avi and Koppel, Moshe},
  year = {2025},
  eprint = {2510.20386},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.20386}
}

References

28 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eylon Gueta, Avi Shmidman, Shaltiel Shmidman, Cheyn Shmuel Shmidman, Joshua Guedalia, Moshe Koppel, Dan Bareket, Amit Seker, and Reut Tsarfaty. 2023. Large pre-trained models with extra-large vocabularies: A contrastive analysis of hebrew bert models and a new one to outperform them all. Eric Harper, Somshubra Majumdar, Oleksii Kuchaiev, Li Jason, Yang Zhang, Evelina Bakhturina, Vahid Noroozi, Sandeep Subramanian, Koluguri Nithin, Huang Jocelyn, Fei Jia, Jagadeesh Balam, Xuesong Yang, Micha Livne, Yi Dong, Sean Naren, and Boris Ginsburg. NeMo: a toolkit for Conversational AI and Large Language Models. find our card
2. Adam Amram, Anat Ben David, and Reut Tsarfaty. 2018. Representations and architectures in neural sentiment analysis for morphologically rich languages: A case study from Modern Hebrew. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2242–2252, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find our card
3. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations (ICLR 2021). Also arXiv preprint arXiv:2006.03654. ↗
4. Dan Bareket and Reut Tsarfaty. 2021. Neural Modeling for Named Entities and Morphology (NEMO2). Transactions of the Association for Computational Linguistics, 9:909–928. find our card
5. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations (ICLR). find
6. Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, and Junichi Tsujii. 2020. Characterbert: Reconciling elmo and bert for word-level open-vocabulary representations from characters. find
7. Lola Le Breton, Quentin Fournier, Mariam El Mezouar, John X. Morris, and Sarath Chandar. 2025. Neobert: A next-generation bert. find
8. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. find
9. Amir DN Cohen, Hilla Merhav Fine, Yoav Goldberg, and Reut Tsarfaty. 2023. Heq: a large and diverse hebrew reading comprehension benchmark. find our card
10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep NeoDictaBERT is available at https://huggingface.co/dicta-il/neodictabert NeoDictaBERT-bilingual is available at https://huggingface.co/dicta-il/ neodictabert-bilingual Dan Hendrycks and Kevin Gimpel. 2023. Gaussian error linear units (gelus). ↗
11. Yoav Levine, Noam Wies, Or Sharir, Hofit Bata, and Amnon Shashua. 2020. Limits to depth efficiencies of self-attention. In Advances in Neural Information Processing Systems, volume 33, pages 22640–22651. Curran Associates, Inc. find
12. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. find
13. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. find
14. Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. 2024. Fineweb-edu: the finest collection of educational content. find
15. Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie, and Benjamin Van Durme. 2025. mmbert: A modern multilingual encoder with annealed language learning. find
16. Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037, Dubrovnik, Croatia. Association for Computational Linguistics. find
17. Noam Shazeer. 2020. Glu variants improve transformer. find
18. Shaltiel Shmidman, Avi Shmidman, Amir DN Cohen, and Moshe Koppel. 2024a. Adapting llms to hebrew: Unveiling dictalm 2.0 with enhanced vocabulary and instruction capabilities. find our card
19. Shaltiel Shmidman, Avi Shmidman, and Moshe Koppel. 2023. Dictabert: A state-of-the-art bert suite for modern hebrew. find our card
20. Shaltiel Shmidman, Avi Shmidman, Moshe Koppel, and Reut Tsarfaty. 2024b. MRL parsing without tears: The case of Hebrew. In Findings of the Association for Computational Linguistics: ACL 2024, pages 4537–4550, Bangkok, Thailand. Association for Computational Linguistics. find our card
21. Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2021. Fast wordpiece tokenization. find
22. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023. Roformer: Enhanced transformer with rotary position embedding. find
23. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. find
24. Reut Tsarfaty. 2013. A unified morpho-syntactic scheme of stanford dependencies. In Proc. of ACL. find
25. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672. ↗
26. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, and Iacopo Poli. 2025. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2526–2547, Vienna, Austria. Association for Computational Linguistics. find
27. Amir Zeldes, Nick Howell, Noam Ordan, and Yifat Ben Moshe. 2022. A second wave of UD Hebrew treebanking and cross-domain parsing. In Proceedings of EMNLP 2022, pages 4331–4344, Abu Dhabi, UAE. find our card
28. Biao Zhang and Rico Sennrich. 2019. Root mean square layer normalization. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/langstream-2025-shmidman-neodictabert-pushing-frontier-bert-models/