Knesset-DictaBERT: A Hebrew Language Model for Parliamentary Proceedings

What it is about

The authors continue masked-language-model training of DictaBERT on the Knesset Corpus, about 384 million tokens of plenary and committee protocols. They split the data 80/10/10 and use 256-token chunks with 15% masking. Training runs two epochs at learning rate 1e-4 in fp16. On a test set of about 3.2 million sentences the perplexity falls from 22.87 (base DictaBERT) to 6.60. Top-1 masked-token accuracy rises from 48.02% to 52.55%, top-2 from 58.60% to 63.07%, and top-5 from 68.98% to 73.59%. The authors note that the corpus carries the political and social biases of parliamentary debate.

Ideas

I1. A register cloze test for treebanks/he/iahltknesset: hide one word per sentence, ask a candidate model which word fits, and report top-1, top-2 and top-5 agreement. This measures how well the register's lexicon is covered, and it is a check on 8. Lexicon and register.
I2. Domain adaptation measured by perplexity, as a small statistical model: a word n-gram model trained on the Knesset text, compared against a general Hebrew model on the same held-out sentences. The gain should be reported with the held-out split, not the training split.
I3. Record the bias caveat in the data card: any lexicon or frequency list taken from parliamentary text should carry a label saying it reflects that register and its politics.

For Mova

I1 is a test in treebanks/he/iahltknesset, with its result written into 8. Lexicon and register. I2 is an n-gram experiment for the he lexicon. I3 is a note in the treebanks/he/ data notes.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2407.20581

@misc{goldin2024knesset,
  title = {{Knesset-DictaBERT: A Hebrew Language Model for Parliamentary Proceedings}},
  author = {Goldin, Gili and Science, Shuly Wintner   Department of Computer and Haifa, University of and Israel)},
  year = {2024},
  eprint = {2407.20581},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.20581}
}

References

4 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
Gili Goldin, Nick Howell, Noam Ordan, Ella Rabinovich, and Shuly Wintner. 2024. The Knesset Corpus: An annotated corpus of Hebrew parliamentary proceedings. ArXiv, abs/2405.18115. ↗ our card
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2017. Mixed precision training. CoRR, abs/1710.03740. ↗
Shaltiel Shmidman, Avi Shmidman, and Moshe Koppel. 2023. DictaBERT: A state-of-the-art BERT suite for Modern Hebrew. This figure "committee_ttr.png" is available in "png" format from: http://arxiv.org/ps/2407.20581v1 ↗ our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/langstream-2024-goldin-knesset-dictabert-hebrew-language-model/