BEReshiT: an Ancient Hebrew Model based on DictaBERT

What it is about

The authors fine-tune DictaBERT, a BERT model for Modern Hebrew, on the Hebrew Bible (about 373,000 words) and on clean Dead Sea Scrolls lines (about 68,000 words), to make an Ancient Hebrew model, BEReshiT, and a morphology submodel, BEReshiT-morph. They train twelve versions that differ in how many layers are frozen; the best freezes nine and trains three. Evaluation is a masked-word cloze on a 200-verse set: half is spread over genres, half is built around five phenomena absent or rare in Modern Hebrew (wayyiqtol chains, infinitive absolute with a finite verb, verb-subject-object order, suffixed possessives, particles). BEReshiT beats DictaBERT by 0.02 to 0.09 in accuracy once BEREL and GPT-4o, which may have seen the biblical text, are set aside. On the DSS-only items their accuracy drops to 0.18 (BEREL) and 0.09 (GPT). The morphology submodel, with prefixes split off first, reaches mean macro-F1 0.76 and 0.97 for POS.

Ideas

• I1. A phenomenon-stratified test for tools/hbo: use the paper's design (verses grouped by wayyiqtol chains, VSO, suffixed possessives, particles, infinitive absolute with finite verb) as a held-out set, and report the parser's and segmenter's scores per phenomenon as well as overall.
• I2. A contamination rule: the authors drop BEREL and GPT from the headline comparison because they have seen the text. For Mova, the segmenter tables learned from MACULA must not be scored on the verses they were learned from; hold verses out before any tools/hbo number is written down.
• I3. Prefix-first morphology: they split prefixes before tagging, and report POS macro-F1 on the same split. Mova can report per-feature macro-F1 for POS and prefix tags on one split, so the numbers can be compared with theirs without a neural model.

For Mova

I1 and I3 are measures for tools/hbo (its test suite and its report); the particles and possessive items feed Pronouns, particles and prepositions, and the wayyiqtol item feeds The verb. I2 is a rule for how tools/hbo tables are built and tested, recorded in 13. Biblical Hebrew in Mova: the tools. The neural model itself is not a Mova target; only the data design and the measures are.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.63317/4j3oje5q7bcd

@inproceedings{nikolovastoupak2026bereshit,
  title = {{BEReshiT: an Ancient Hebrew Model based on DictaBERT}},
  author = {Nikolova-Stoupak, Iglika and Amblard, Maxime and Rey, Frédérique},
  booktitle = {Proceedings of the Fourth Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA 2026) @ LREC 2026},
  year = {2026},
  doi = {10.63317/4j3oje5q7bcd},
  url = {https://doi.org/10.63317/4j3oje5q7bcd}
}

References

26 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 1955–2009. Discoveries in the Judean Desert. Clarendon Press, Oxford. https://www.qumran-digital.org/ ↗
2. David Bamman and Patrick J. Burns. 2020. Latin bert: A contextual language model for classical philology. find
3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 4171–4186. find
4. Simon Gabay, Pedro Ortiz Suarez, Alexandre Bartz, Alix Chagué, Rachel Bawden, Philippe Gambette, and Benoît Sagot. 2022. From freem to d’alembert: a large corpus and a language model for early modern french. find
5. Ayaka Harigai, Oshitaka Toyama, Mitsutoshi Nagano, Mirei Abe, Masahiro Kawabata, Li Li, Jin Yamamura, and Kei Takase. 2025. Response accuracy of gpt-4 across languages: insights from diagnostic radiology questions. Japanese Journal of Radiology. find
6. Michael A. Hedderich, Lukas Lange, Heike Adel, Jannik Strötgen, and Dietrich Klakow. 2021. A survey on recent approaches for natural language processing in low-resource scenarios. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2545–2568. Association for Computational Linguistics. find
7. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138, Minneapolis, Minnesota. Association for Computational Linguistics. find
8. Aaron D. Hornkohl. 2014. Ancient Hebrew Periodization and the Language of the Book of Jeremiah: The Case for a Sixth-Century Date of Composition, volume 74 of Studies in Semitic Languages and Linguistics. Brill, Leiden. find
9. Avi Hurwitz. 2014. A Concise Lexicon of Late Biblical Hebrew: Linguistic Innovations in the Writings of the Second Temple Period. Vetus Testamentum Supplements. Brill, Leiden. find
10. Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651–3657, Florence, Italy. Association for Computational Linguistics. find
11. Pratik M. Joshi, Sebastin Santy, Ameya Godbole, Kalika Bali, and Chandra Sekhar Mukherjee. 2020. The state and fate of linguistic diversity and inclusion in the nlp world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293. Association for Computational Linguistics. find
12. Geoffrey Khan, Shmuel Bolozky, Steven E. Fassberg, Gary A. Rendsburg, Aaron D. Rubin, Ora R. Schwarzwald, and Tamar Zewi, editors. 2013. Encyclopedia of Hebrew Language and Linguistics, 1 edition, volume 1-4. Brill, Leiden. find
13. Piroska Lendvai, Uwe Reichel, Anna Jouravel, Achim Rabus, and Elena Renje. 2023. Domain-adapting BERT for attributing manuscript, century and region in pre-Modern Slavic texts. In Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change, pages 15–21, Singapore. Association for Computational Linguistics. find
14. Josh McGiff and Nikola S. Nikolov. 2025. Overcoming data scarcity in generative language modelling for low-resource languages: A systematic review. arXiv preprint. find
15. Ricardo Muñoz Sánchez, David Alfter, Simon Dobnik, Maria Irena Szawerna, and Elena Volodina. 2024. Jingle BERT, jingle BERT, frozen all the way: Freezing layers to identify CEFR levels of second language learners using BERT. In Proceedings of the 13th Workshop on Natural Language Processing for Computer Assisted Language Learning, pages 137–152, Rennes, France. LiU Electronic Press. find
16. Iglika Nikolova-Stoupak, Maxime Amblard, Sophie Robert-Hayek, and Frédérique Rey. 2025. A classifier of word-level variants in witnesses of biblical Hebrew manuscripts. In Findings of the Association for Computational Linguistics: ACL 2025, pages 21313–21329, Vienna, Austria. Association for Computational Linguistics. find our card
17. OpenAI. 2024. Gpt-4o system https://openai.com/index/gpt-4o-system-card/. ↗
18. Robert Rezetko and Ian Young. 2014. Historical Linguistics and Biblical Hebrew: Steps toward an Integrated Approach, volume 9 of Ancient Near East Monographs. SBL Press, Atlanta. find
19. Stefan Schweter, Luisa März, Katharina Schmid, and Erion Çano. 2022. hmbert: Historical multilingual language models for named entity recognition. find
20. Avi Shmidman, Joshua Guedalia, Shaltiel Shmidman, Cheyn Shmuel Shmidman, Eli Handel, and Moshe Koppel. 2022. Introducing berel: Bert embeddings for rabbinic-encoded language. find our card
21. Avi Shmidman, Ometz Shmidman, Hillel Gershuni, and Moshe Koppel. 2024a. MsBERT: A new model for the reconstruction of lacunae in Hebrew manuscripts. In Proceedings of the 1st Workshop on Machine Learning for Ancient Languages (ML4AL 2024), pages 13–18, Hybrid in Bangkok, Thailand and online. Association for Computational Linguistics. ↗ our card
22. Shaltiel Shmidman, Avi Shmidman, Amir DN Cohen, and Moshe Koppel. 2024b. Adapting llms to hebrew: Unveiling dictalm 2.0 with enhanced vocabulary and instruction capabilities. find our card
23. Shaltiel Shmidman, Avi Shmidman, and Moshe Koppel. 2023. Dictabert: A state-of-the-art BERT suite for modern Hebrew. find our card
24. Elad Shvartz, Leah Attal, Omri Zur, Zaki Nujeidat, Gilad Plopsky, and Daniel Bahir. 2025. Bilingual comparison of the performance of gpt-4o and gpt-4 on multilingual medical questions. European Journal of Radiology. find
25. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. Bert rediscovers the classical nlp pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601, Florence, Italy. Association for Computational Linguistics. find
26. Wenhao Zhu, Hongyu Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2024. Multilingual machine translation with large language models. In Findings of the Association for Computational Linguistics: NAACL 2024. card. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2026-stoupak-bereshit-ancient-hebrew-model-based/