Introducing BEREL: BERT Embeddings for Rabbinic-Encoded Language

What it is about

A short paper releasing a BERT-base model trained on about 220 million words of Rabbinic Hebrew (the Sefaria and Dicta libraries). The authors argue that Modern Hebrew models (HeBERT, AlephBERT) falter on Rabbinic texts because those texts mix in Talmudic Aramaic (even Aramaic affixes on Hebrew words), spell plene and defective freely, and abbreviate one word in every five to ten (apocopes with an apostrophe, acronyms with a gershayim). Their tokenizer change is to keep an abbreviation whole instead of splitting at the apostrophe or quotation mark. They test on a challenge set of homographs and release the model and the set for unrestricted use; the extracted text holds no detailed numbers.

Ideas

I1. Tokenisation: treat a word with ׳ or ״ inside as one token (abbreviation or acronym) before any prefix splitting; a rule worth having in the Mishnaic and later-Hebrew readers, and a list of common Rabbinic abbreviations with their expansions.
I2. Spelling normalisation: map plene and defective spellings to one consonantal key (drop medial ו/י as matres) when looking a word up in the Torah dictionary; measure how many Mishnah word types then find a Torah lemma.
I3. The homograph challenge set, if its licence is open as stated, is a ready test for any sense or lemma choice Mova makes on unvocalised Hebrew.

For Mova

I1 belongs to 11. Mishnaic Hebrew (hbo-x-mishnaic) (orthography, abbreviations) and to whatever tokeniser reads Mishnaic text; I2 to macula dict lookup and 12-biblical-mishnaic-modern.md (spelling across stages). I3 is a measure for a future unvocalised-Hebrew lemmatiser; the model itself is a neural teacher at most, never the answerer.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2208.01875

@misc{shmidman2022introducing,
  title = {{Introducing BEREL: BERT Embeddings for Rabbinic-Encoded Language}},
  author = {Shmidman, Avi and Guedalia, Joshua and Shmidman, Shaltiel and Shmidman, Cheyn Shmuel and Handel, Eli and Koppel, Moshe},
  year = {2022},
  eprint = {2208.01875},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.01875}
}

References

9 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Avihay Chriqui and Inbal Yahav. 2021. Hebert & hebemo: a hebrew bert model and a tool for polarity analysis and emotion recognition. find our card
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meet ing of the Association for Computational Linguistics, pages 7871 7880, Online. Association for Computational Linguistics. find
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. find
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. find
Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Shaked Greenfeld, and Reut Tsarfaty. 2021. Alephbert: A hebrew large pre-trained language model to start-off your hebrew NLP application with. CoRR, abs/2104.04052. ↗ our card
Reut Tsarfaty, Amit Seker, Shoval Sadde, and Stav Klein. 2019. What’s wrong with Hebrew NLP? and how to make it right. CoRR, abs/1908.05453. ↗ our card
Shuly Wintner. 2014. Morphological Processing of Semitic Languages, pages 43–66. Springer Berlin Heidelberg, Berlin, Heidelberg. find
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/hbo-2022-shmidman-introducing-berel-bert-embeddings-rabbinic/