Introducing DictaLM -- A Large Generative Language Model for Modern Hebrew

What it is about

The authors release DictaLM, a 7B-parameter generative model for Modern Hebrew, trained on about 7.5 billion tokens. Most of the data is a cleaned HeDC4 (80%), the rest is news, blogs, subtitles and novels. The cleaning removes about 15% of HeDC4 with histograms, gibberish detectors and a perplexity filter from a Hebrew BERT. The corpus keeps only English and Hebrew words, and every other language becomes one <foreign> token. A byte-pair tokenizer with 56,000 entries gives about 1.3 tokens per word. The base model is then instruction-tuned on Hebrew QA data and translated MPT-Instruct. A second base model, DictaLM-Rab, is trained with half of each batch drawn from 1.2B tokens of Rabbinic Hebrew. The paper gives no benchmark scores; the authors call the release an alpha.

Ideas

• I1. Map every non-target-language span to one <foreign> marker before parsing. tools/langseg already cuts a mixed text into language segments, so its non-Hebrew, non-English segments could be replaced by a marker, and the parser would see a clean sentence with a known gap.
• I2. A register sampler for 11. Mishnaic Hebrew (hbo-x-mishnaic): draw a fixed share (for example 50%) of Mishnaic or Rabbinic sentences into a training or test mix, with oversampling, and measure the effect on a held-out check. Run it twice, as the determinism rule asks, before claiming any gain.
• I3. A cheap corpus filter: drop sentences a small statistical model finds very surprising (a character or word n-gram model in Rust, in place of the paper's BERT). Measure how much text goes, and check a sample of the removed sentences.

For Mova

I1 goes into tools/langseg and the prag input stage. I2 goes into 11. Mishnaic Hebrew (hbo-x-mishnaic) as a test-set design. I3 is a corpus step in the Hebrew data pipeline, not a parser change.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2309.14568

@misc{shmidman2023introducing,
  title = {{Introducing DictaLM -- A Large Generative Language Model for Modern Hebrew}},
  author = {Shmidman, Shaltiel and Shmidman, Avi and Cohen, Amir David Nissan and Koppel, Moshe},
  year = {2023},
  eprint = {2309.14568},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.14568}
}

References

11 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Adam Amram, Anat Ben David, and Reut Tsarfaty. 2018. Representations and architectures in neural sentiment analysis for morphologically rich languages: A case study from Modern Hebrew. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2242–2252, Santa Fe, New Mexico, USA. Association for Computational Linguistics. https://github.com/NVIDIA/NeMo https://github.com/NVIDIA/ TransformerEngine 8 https://www.sefaria.org.il/ 9 https://library.dicta.org.il/ ↗ our card
2. Dan Bareket and Reut Tsarfaty. 2021. Neural Modeling for Named Entities and Morphology (NEMO2). Transactions of the Association for Computational Linguistics, 9:909–928. find our card
3. Amir DN Cohen, Hilla Merhav Fine, Yoav Goldberg, and Reut Tsarfaty. 2023. Heq: a large and diverse hebrew reading comprehension benchmark. find our card
4. Dan Hendrycks and Kevin Gimpel. 2023. Gaussian error linear units (gelus). find
5. Omri Keren and Omer Levy. 2021. Parashoot: A hebrew question answering dataset. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 106–112. find our card
6. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. CoRR, abs/1508.07909. ↗
7. Vitaly Shalumov and Harel Haskey. 2023. Hero: Roberta and longformer hebrew language models. arXiv:2304.11077. ↗ our card
8. Noam Shazeer. 2020. Glu variants improve transformer. find
9. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2022. Roformer: Enhanced transformer with rotary position embedding. find
10. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.03762. ↗
11. Charles Welch, Rada Mihalcea, and Jonathan K. Kummerfeld. 2020. Improving low compute language modeling with in-domain embedding initialisation. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/langstream-2023-shmidman-introducing-dictalm-large-generative-language/