HalleluBERT: Let Every Token That Has Meaning Bear Its Weight

What it is about

The paper presents HalleluBERT, a RoBERTa-style encoder for Modern Hebrew in base (126M parameters) and large (357M) sizes, pre-trained from scratch on about 49.1 GB of deduplicated Hebrew web text (HeDC4) and Hebrew Wikipedia, with a 52k byte-level BPE vocabulary. Pre-training used 100k steps with an 8k global batch on a TPU pod (about 30 hours for base, about 6 days for large). It is evaluated on named entity recognition (BMC and token-level NEMO, micro-F1) and on sentiment classification (SMCD, macro-F1). Base reaches 93.33 on BMC and 87.06 on NEMO. Large has the best overall average (88.95 against 87.49 for XLM-RoBERTa large). The results come from a single seed, without confidence intervals, and the paper lists its own limits: no error analysis, no whole-word masking, and no control for register (Modern against Biblical Hebrew).

Ideas

• I1. A measurement only (the data is NC): compare the token-level and morpheme-level scores of Mova's he segmentation on the paper's benchmark split, to see how many errors are segmentation errors and how many are tag errors.
• I2. The paper's vocabulary-size question (32k, 52k or 128k) carried over to Mova's segmenter: count the segments per word under each of Mova's split tables, and report the number, so tokenisation cost is measured before any change.
• I3. A register check for any Hebrew model used as a teacher: the paper's own gap (no separation of Modern and Biblical Hebrew text) becomes a test that scores the same model on modern and Biblical samples before any claim about it.

For Mova

I1 is a measure for 12. Mova and Modern Hebrew: where it stands, what it would need, not built into anything. I2 goes into 2. Script and spelling and the segmenter notes of tools/hbo. I3 goes into 10. Historical layers: Biblical, Mishnaic, Medieval and Modern Hebrew and tools/langseg.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.63317/3qdqexx4e9i2

@inproceedings{schmitt2026hallelubert,
  title = {{HalleluBERT: Let Every Token That Has Meaning Bear Its Weight}},
  author = {Schmitt, Raphael},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3qdqexx4e9i2},
  url = {https://doi.org/10.63317/3qdqexx4e9i2}
}

References

26 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. 2022. Towards a Cleaner Document-Oriented Multilingual Crawled Corpus. arXiv e-prints, page arXiv:2201.06642. ↗
2. Adam Amram, Anat Ben David, and Reut Tsarfaty. 2018. Representations and architectures in neural sentiment analysis for morphologically rich languages: A case study from Modern Hebrew. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2242– 2252, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find our card
3. Dan Bareket and Reut Tsarfaty. 2021. Neural Modeling for Named Entities and Morphology (NEMO2). Transactions of the Association for Computational Linguistics, 9:909–928. find our card
4. Naama Ben-Mordecai. 2005. Hebrew named entity recognition. Master’s thesis, Department of Computer Science, Ben-Gurion University. find
5. Branden Chan. 2020. XLM-RoBERTa: The multilingual alternative for non-english NLP. Library Catalog: towardsdatascience.com. find
6. Branden Chan, Stefan Schweter, and Timo Möller. 2020. German’s next language model. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6788–6796, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
7. Avihay Chriqui and Inbal Yahav. 2022. Hebert and hebemo: A hebrew bert model and a tool for polarity analysis and emotion recognition. IN-FORMS Journal on Data Science, 1(1):81–95. find
8. Pieter Delobelle, Thomas Winters, and Bettina Berendt. 2020. RobBERT: a Dutch RoBERTa-based Language Model. arXiv:2001.06286 [cs]. ArXiv: 2001.06286. ↗
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
10. Eylon Gueta, Avi Shmidman, Shaltiel Shmidman, Cheyn Shmuel Shmidman, Joshua Guedalia, Moshe Koppel, Dan Bareket, Amit Seker, and Reut Tsarfaty. 2022. Large pre-trained models with extra-large vocabularies: A contrastive analysis of hebrew bert models and a new one to outperform them all. find our card
11. Henry He, Johann Frei, and Raphael Scheible-Schmitt. 2025. The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLP. ISSN: 2693-5015. find
12. Norman P. Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Cliff Young, Xiang Zhou, Zongwei Zhou, and David Patterson. 2023. TPU v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings. find
13. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177–180, Prague, Czech Republic. Association for Computational Linguistics. find
14. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692 [cs]. ArXiv: 1907.11692. ↗
15. Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie, and Benjamin Van Durme. 2025. mmbert: A modern multilingual encoder with annealed language learning. find
16. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020. CamemBERT: a Tasty French Language Model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203–7219, Online. Association for Computational Linguistics. find
17. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. arXiv:1904.01038 [cs]. ArXiv: 1904.01038. ↗
18. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. find
19. Raphael Scheible, Johann Frei, Fabian Thomczyk, Henry He, Patric Tippmann, Jochen Knaus, Victor Jaravine, Frank Kramer, and Martin Boeker. 2024. GottBERT: a pure German language model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 21237–21250, Miami, Florida, USA. Association for Computational Linguistics. find
20. Raphael Scheible-Schmitt and Johann Frei. 2025. GeistBERT: Breathing life into German NLP. In Proceedings of the Workshop on Beyond English: Natural Language Processing for all Languages in an Era of Large Language Models, pages 42– 50, Varna, Bulgaria. INCOMA Ltd., Shoumen, BULGARIA. find
21. Raphael Scheible-Schmitt, Henry He, and Armando B. Mendes. 2025. PortBERT: Navigating the depths of Portuguese language models. In Proceedings of the Workshop on Beyond English: Natural Language Processing for all Languages in an Era of Large Language Models, pages 59– 71, Varna, Bulgaria. INCOMA Ltd., Shoumen, BULGARIA. find
22. Raphael Scheible-Schmitt and Stefan Schweter. 2025. Sindbert, the sailor: Charting the seas of turkish nlp. find
23. Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Shaked Greenfeld, and Reut Tsarfaty. 2021. Alephbert:a hebrew large pre-trained language model to start-off your hebrew nlp application with. find our card
24. Vitaly Shalumov and Harel Haskey. 2023. Hero: Roberta and longformer hebrew language models. find our card
25. Cagri Toraman, Eyup Halit Yilmaz, Furkan Şahinu¸c, and Oguzhan Ozcelik. 2023. Impact of tokenization on language models: An analysis for turkish. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(4):1–21. find
26. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2026-schmitt-hallelubert-let-every-token-that/