DictaBERT: A State-of-the-Art BERT Suite for Modern Hebrew
What it is about
The paper presents DictaBERT, a BERT-architecture model for Modern Hebrew, pre-trained with masked language modelling only (next-sentence prediction removed) on about 3.8B tokens: 70% cleaned HeDC4 web text and 30% news, blogs, subtitles and novels. Three fine-tuned heads are released with it: DictaBERT-seg (prefix segmentation, 52K sentences), DictaBERT-morph (five classifiers for POS, proclitic functions, features, suffix presence and suffix features) and DictaBERT-heq (question answering on HeQ). Against mBERT, AlephBERT, AlephBERT-Gimmel, HeRo and mT5-Base it scores best on most tasks: segmentation 98.16 (AlephBERT-Gimmel 98.09), NER 87.01 (HeRo 86.79), sentiment 89.79 (HeRo 89.56), and QA F1 72.9, close to mT5-XL at 73.5 with about ten times the parameters. The paper is a model release; its evaluation tables are the reference numbers.
Ideas
- I1. Constrain prefix segmentation by a table of legal proclitic sets keyed on the word's initial letters. The model applies only the valid function sets for a word's first letters at inference. A deterministic
pretable in the segmenter can enforce the same rule; measure how many illegal splits it removes on a held-out set. - I2. Keep the morphology as separate outputs: POS, proclitic functions (several may stack), core features, suffix presence and suffix features. This is a usable layout for mapping Modern Hebrew tags to UD features, one slot per decision.
- I3. Reproduce the multi-task segmentation and POS setup of Seker et al. (2021) on the same data before any claim about a Mova segmenter, and report it against the paper's Table 1 figures.
For Mova
I1 goes into Prepositions, prefixes, conjunctions and particles (the legal prefix sets) and into the planned Modern Hebrew segmenter in 12. Mova and Modern Hebrew: where it stands, what it would need, following the table pattern of tools/hbo. I2 goes into 11. Modern Hebrew in Universal Dependencies. I3 is a test for 12. Mova and Modern Hebrew: where it stands, what it would need. The model itself is neural and is only a teacher or a reference; nothing of it is a dependency.
Cite
@misc{shmidman2023dictabert,
title = {{DictaBERT: A State-of-the-Art BERT Suite for Modern Hebrew}},
author = {Shmidman, Shaltiel and Shmidman, Avi and Koppel, Moshe},
year = {2023},
eprint = {2308.16687},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2308.16687}
}References
- Adam Amram, Anat Ben David, and Reut Tsarfaty. 2018. Representations and architectures in neural sentiment analysis for morphologically rich languages: A case study from Modern Hebrew. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2242–2252, Santa The base model DictaBERT is available at https://huggingface.co/dicta-il/dictabert The fine-tuned segmentation model DictaBERT-seg is available at https://huggingface.co/dicta-il/ dictabert-seg The fine-tuned morphology model DictaBERT-morph is available at https://huggingface.co/dicta-il/ dictabert-morph The fine-tuned QA model DictaBERT-heq is available at https://huggingface.co/dicta-il/ dictabert-heq. Fe, New Mexico, USA. Association for Computational Linguistics. ↗ our card
- Dan Bareket and Reut Tsarfaty. 2021. Neural Modeling for Named Entities and Morphology (NEMO2). Transactions of the Association for Computational Linguistics, 9:909–928. find our card
- Avihay Chriqui and Inbal Yahav. 2022. HeBERT and HebEMO: A hebrew BERT model and a tool for polarity analysis and emotion recognition. INFORMS Journal on Data Science, 1(1):81–95. ↗ our card
- Amir DN Cohen, Hilla Merhav Fine, Yoav Goldberg, and Reut Tsarfaty. 2023. Heq: a large and diverse hebrew reading comprehension benchmark. find our card
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. find
- Matan Eyal, Hila Noga, Roee Aharoni, Idan Szpektor, and Reut Tsarfaty. 2022. Multilingual sequence-to-sequence models for hebrew nlp. find our card
- Eylon Gueta, Avi Shmidman, Shaltiel Shmidman, Cheyn Shmuel Shmidman, Joshua Guedalia, Moshe Koppel, Dan Bareket, Amit Seker, and Reut Tsarfaty. 2023. Large pre-trained models with extra-large vocabularies: A contrastive analysis of hebrew bert models and a new one to outperform them all. find our card
- Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. find
- Shoval Sade, Amit Seker, and Reut Tsarfaty. 2018. The Hebrew Universal Dependency treebank: Past present and future. In Proceedings of the Second Workshop on Universal Dependencies (UDW 2018), pages 133–143, Brussels, Belgium. Association for Computational Linguistics. find our card
- Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Shaked Greenfeld, and Reut Tsarfaty. 2021. Alephbert:a hebrew large pre-trained language model to start-off your hebrew nlp application with. find our card
- Vitaly Shalumov and Harel Haskey. 2023. Hero: Roberta and longformer hebrew language models. arXiv:2304.11077. ↗ our card
- Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2021. Fast wordpiece tokenization. find
- Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics. find
- Amir Zeldes, Nick Howell, Noam Ordan, and Yifat Ben Moshe. 2022. A second wave of ud hebrew treebanking and cross-domain parsing. find our card
This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/arxivrecent-2023-shmidman-dictabert-state-art-bert-suite/