Mova0.5 pre-release⌕
☰
home / docs / papers / Modern Hebrew / HEBATRON: A Hebrew-Specialized Open-Weight Mixture-of-Experts Language Model

HEBATRON: A Hebrew-Specialized Open-Weight Mixture-of-Experts Language Model

What it is about

The authors adapt NVIDIA's Nemotron-3 Nano sparse mixture-of-experts model (30B parameters, about 3B active per token) to Hebrew. Continued pretraining runs in three phases, ordered from easy to hard: formal Hebrew and English first (legal, parliamentary, literary and academic text), then colloquial and social-media Hebrew for a short phase, then long documents up to 65,536 tokens. Supervised fine-tuning uses 2M bilingual samples, including 200,147 procedurally generated Hebrew instruction-following items that test morphology and prefix handling. The ordering matters: the easy-to-hard run scores 68.00 on their benchmark aggregate, against 64.99 for the reversed order with the same compute. The model reports a Hebrew reasoning average of 73.8%, against 68.9% for DictaLM-3.0-24B-Thinking, and about 9 times the inference throughput of the 24–27B dense baselines. The English average stays at 86.0%.

Ideas

For Mova

I1 goes into the plan in 12. Mova and Modern Hebrew: where it stands, what it would need. I2 goes into 05b. Verb paradigms: the strong verb in seven binyanim and the weak roots as a generator and checker. I3 is a test for tools/hbo. The model weights are a teacher only, not something Mova depends on.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2605.11255

@misc{kayzer2026hebatron,
  title = {{HEBATRON: A Hebrew-Specialized Open-Weight Mixture-of-Experts Language Model}},
  author = {Kayzer, Noam and Revital, Dan and Joseph, Ori Bar and Arvatz, Smadar and Levi, Or and Geva, Tal and Shmidman, Shaltiel and Cohen, Amir DN and Ordan, Noam and Baruch, Omer and Zinkovskaia, Kate and Apini, Zevi and Weinberger, Sarel},
  year = {2026},
  eprint = {2605.11255},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.11255}
}

References

65 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. The claude 3 model family: Opus, sonnet, haiku. Technical report, Anthropic, 2024. find
  2. Wissam Antoun, Fady Baly, and Hazem Hajj. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the LREC Workshop on Language Models and Resources for Arabic NLP, 2020. find
  3. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th International Conference on Machine Learning (ICML), pages 41–48, 2009. find our card
  4. Mayee F. Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. Skill-it! a data-driven skills framework for understanding and training language models. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL https://arxiv.org/abs/2307.14430. ↗
  5. Shouyuan Chen et al. LongLoRA: Efficient fine-tuning of long-context large language models. In Proceedings of ICLR 2024, 2024. find
  6. Avihay Chriqui and Inbal Yahav. HeBERT & HebEMO: pre-trained Hebrew BERT and Hebrew sentiment analysis. arXiv preprint arXiv:2102.01909, 2022. ↗ our card
  7. Peter Clark et al. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. ↗
  8. Karl Cobbe et al. Training verifiers to solve math word problems. arXiv:2110.14168, 2021. ↗
  9. Cohere. Command R7B Arabic. Technical report, Cohere, 2025. URL https://cohere.com/blog/command-r7b-arabic. ↗
  10. Alexis Conneau et al. Unsupervised cross-lingual representation learning at scale. In Proceedings of ACL 2020, pages 8440–8451, 2020. find
  11. Yiming Cui et al. Efficient and effective text encoding for Chinese LLaMA and Alpaca. arXiv preprint arXiv:2304.08177, 2023. ↗
  12. Sumanth Doddapaneni et al. Towards leaving no Indic language behind: Building monolingual corpora, benchmark and models for Indic languages. In Proceedings of ACL 2023, 2023. find
  13. Mohamed Elgaar and Hadi Amiri. Curriculum learning for LLM pretraining: An analysis of learning dynamics. arXiv preprint arXiv:2601.21698, 2026. URL https://arxiv.org/abs/2601.21698. ↗
  14. William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23 (1):1–39, 2021. find
  15. Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. Continual pre-training for cross-lingual LLM adaptation: Enhancing Japanese language capabilities. In arXiv preprint arXiv:2404.17790, 2024. ↗
  16. Gemini Team et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2024. arXiv preprint Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211, 2013. ↗
  17. Google DeepMind. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. ↗
  18. Suchin Gururangan et al. Don’t stop pretraining: Adapt language models to domains and tasks. In Proceedings of ACL 2020, pages 8342–8360, 2020. find
  19. Hebatron Team. Hebatron: A hebrew-specialized open-weight mixture-of-experts language model. https://huggingface.co/HebArabNlpProject/Hebatron, 2026. ↗ our card
  20. Dan Hendrycks et al. Measuring massive multitask language understanding. In Proceedings of ICLR 2021, 2021. find
  21. Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of ACL 2018, pages 328–339, 2018. find
  22. Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Juncai He, Ziche Liu, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. AceGPT, localizing large language models in Arabic. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Mexico City, Mexico, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.naacl-long.450. URL https://aclanthology.org/2024. naacl-long.450. ↗
  23. Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats L. Richter, Quentin Anthony, Timothée Lesort, Eugene Belilovsky, and Irina Rish. Simple and scalable strategies to continually pre-train large language models. Transactions on Machine Learning Research, 2024. URL https://arxiv.org/abs/2403.08763. ↗
  24. Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. The interplay of variant, size, and task type in Arabic pre-trained language models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop, 2021. find
  25. Albert Q Jiang et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. ↗
  26. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 6282–6293. Association for Computational Linguistics, 2020. find
  27. Divyanshu Kakwani et al. IndicNLPSuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages. In Findings of EMNLP 2020, 2020. find
  28. James Kirkpatrick et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521–3526, 2017. find
  29. Mario Michael Krell, Matej Kosec, Sergio P Perez, and Andrew Fitzgibbon. Efficient sequence packing without cross-contamination: Accelerating large language models without impacting performance. arXiv preprint arXiv:2107.02027, 2021. ↗
  30. Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Christopher A Choquette-Choo, Katherine Lee, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. MADLAD-400: A multilingual and document-level large audited dataset. In Advances in Neural Information Processing Systems (NeurIPS), 2023. find
  31. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, et al. BLOOM: A 176B-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. ↗
  32. Katherine Lee et al. Deduplicating training data makes language models better. In Proceedings of ACL 2022, pages 8076–8092, 2022. find
  33. Percy Liang et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022. ↗
  34. Xuhong Luo et al. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747, 2023. ↗
  35. Sam McCandlish, Jared Kaplan, Georgios Vitkovskiy, and Team OpenAI. An empirical model of large-batch training. arXiv preprint arXiv:1812.06162, 2018. ↗
  36. Paulius Micikevicius et al. FP8 formats for deep learning. arXiv preprint arXiv:2209.05433, 2022. ↗
  37. Deepak Narayanan et al. Efficient large-scale language model training on GPU clusters using Megatron-LM. In Proceedings of SC’21, 2021. find
  38. NITE. The Israeli psychometric entrance test: Structure and properties. Technical report, National Institute for Testing and Evaluation, 2023. find
  39. NVIDIA. Nemotron-3-Nano-30B-Base technical report. Technical report, NVIDIA Corporation, 2025. URL https://huggingface.co/nvidia. ↗
  40. OpenAI et al. GPT-4o system card. Technical report, OpenAI, 2024. URL https://openai.com/index/gpt-4o-system-card/. ↗
  41. Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In Proceedings of the 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7), pages 9– 16, Cardiff, United Kingdom, 2019. Leibniz-Institut für Deutsche Sprache. doi: 10.14618/IDS-PUB-9021. URL https://inria.hal.science/hal-02148693. ↗
  42. Long Ouyang et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022. find
  43. Guilherme Penedo et al. The FineWeb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557, 2024. ↗
  44. Bowen Peng et al. YaRN: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023. ↗
  45. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. MAD-X: An adapter-based framework for multi-task cross-lingual transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654–7673, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.617. URL https://aclanthology.org/2020.emnlp-main.617. ↗
  46. Jack W Rae et al. Scaling language models: Methods, analysis & insights from training Gopher. arXiv preprint arXiv:2112.11446, 2021. ↗
  47. Colin Raffel et al. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020. find
  48. Samyam Rajbhandari et al. ZeRO: Memory optimizations toward training trillion parameter models. In Proceedings of SC’20, 2020. find
  49. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, 2011. find
  50. Holger Schwenk et al. CCMatrix: Mining billions of high-quality parallel sentences on the Web. In Proceedings of ACL 2021, pages 6490–6500, 2021. find
  51. Shaltiel Shmidman, Avi Shmidman, Amir David Nissan Cohen, and Moshe Koppel. Adapting LLMs to Hebrew: Unveiling DictaLM 2.0 with enhanced vocabulary and instruction capabilities. arXiv preprint arXiv:2407.07080, 2024. ↗ our card
  52. Shaltiel Shmidman, Avi Shmidman, Amir David Nissan Cohen, and Moshe Koppel. Dicta-LM 3.0: Advancing the frontier of Hebrew sovereign LLMs. Technical report, DICTA / Bar-Ilan University, 2026. find our card
  53. Mohammad Shoeybi et al. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. ↗
  54. Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V Le. Don’t decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489, 2017. ↗
  55. Teknium et al. Hermes 3 technical report. Technical report, NousResearch, 2024. URL https://huggingface.co/NousResearch/Hermes-3-Llama-3.1-70B. ↗
  56. Kushal Tirumala et al. D4: Improving LLM pretraining via document de-duplication and diversification. Advances in Neural Information Processing Systems, 36, 2023. find
  57. Hugo Touvron et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. ↗
  58. Ahmet Üstün et al. Multilingual unsupervised neural machine translation with denoising adapters. In Proceedings of EMNLP 2022, 2022. find
  59. Ahmet Üstün et al. Aya model: An instruction finetuned open-access multilingual language model. arXiv preprint arXiv:2402.07827, 2024. ↗
  60. Yizhong Wang et al. Self-Instruct: Aligning language models with self-generated instructions. In Proceedings of ACL 2023, pages 13484–13508, 2022. find
  61. Jason Wei et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022. find
  62. Peng Xu et al. ChatQA 2: Bridging the gap to proprietary LLMs in long context and RAG capabilities. arXiv preprint arXiv:2407.14518, 2024. ↗
  63. Linting Xue et al. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of NAACL 2021, pages 483–498, 2021. find
  64. Rowan Zellers et al. HellaSwag: Can a machine really finish your sentence? In Proceedings of ACL 2019, pages 4791–4800, 2019. find
  65. Xuemiao Zhang, Liangyu Xu, Feiyu Duan, Yongwei Zhou, Sirui Wang, Rongxiang Weng, Jingang Wang, and Xunliang Cai. Preference curriculum: LLMs should always be pretrained on their preferred data. In Findings of the Association for Computational Linguistics: ACL 2025, pages 21181–21198, Vienna, Austria, 2025. Association for Computational Linguistics. URL https://aclanthology.org/2025.findings-acl.1091. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/langstream-2026-kayzer-hebatron-hebrew-specialized-open-weight/