MsBERT: A New Model for the Reconstruction of Lacunae in Hebrew Manuscripts

What it is about

The authors pretrain a BERT-base model, MsBERT, on Hebrew manuscript transcriptions. The paper argues that existing Hebrew BERTs, including BEREL trained on printed rabbinic texts, break on manuscript orthography: extra matres lectionis, acronyms written with apostrophes, truncated words, and the proclitic לש. The model has a new 128,000-token WordPiece vocabulary with [GAP] and [ONEGAP] tokens, and is trained in three phases on about 67 million words of manuscript transcriptions, 49 million words of pre-print rabbinic texts and a larger rabbinic corpus. On a masked-word test from two manuscripts (shir hashirim rabba and me'en sh'iltot), MsBERT beats BEREL on top-1 accuracy (59.99 against 47.33 and 63.99 against 56.24). On a citation-identification test of mekhilta de-rashbi and shir hashirim rabah, it reaches 89.93 recall against 81.94 for BEREL, with precision about the same. The authors expect it to fail on liturgical poetry, which is not in the training data.

Ideas

• I1. A deterministic normaliser for the four manuscript deviations the paper lists (extra matres lectionis, apostrophe acronyms, proclitic לש, truncated words). Map each variant to a canonical form with a table, and measure whether tools/ocrfix or the hbo segmenter gets more words right on the same fragments.
• I2. The gap convention: keep two separate tokens, one for a gap of unknown length and one for a single missing word. Mova's segmenter can mark these as different kinds of lacuna, so that a rule can fill one missing word from a table and leave longer gaps open.
• I3. The citation test (words in a midrash that belong to a Biblical quotation, with no quotation marks) is a good test for a rule-based quotation detector in 11. Mishnaic Hebrew (hbo-x-mishnaic). Build a small gold set from the paper's annotated fragments, and measure precision and recall of a rule set that uses verse-level lexical overlap.

For Mova

I1 belongs to tools/ocrfix (deterministic correction) and to the hbo segmenter. I2 is a convention for tools/hbo lacunae and for tools/ocrfix. I3 is a test for the Mishnaic quotation rules in 11. Mishnaic Hebrew (hbo-x-mishnaic). The model itself is neural and is not used at test time.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.ml4al-1.2

@inproceedings{shmidman2024msbert,
  title = {{MsBERT: A New Model for the Reconstruction of Lacunae in Hebrew Manuscripts}},
  author = {Shmidman, Avi and Shmidman, Ometz and Gershuni, Hillel and Koppel, Moshe},
  booktitle = {Proceedings of the 1st Workshop on Machine Learning for Ancient Languages (ML4AL 2024)},
  year = {2024},
  doi = {10.18653/v1/2024.ml4al-1.2},
  url = {https://doi.org/10.18653/v1/2024.ml4al-1.2}
}

References

12 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yannis Assael, Thea Sommerschield, and Jonathan Prag. 2019. Restoring ancient text using deep learning: a case study on Greek epigraphy. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6368–6375, Hong Kong, China. Association for Computational Linguistics. ↗
2. David Bamman and Patrick J. Burns. 2020. Latin BERT: A contextual language model for classical philology. CoRR, abs/2009.10053. ↗
3. Simcha Emanuel. 2019. Hidden Treasures from Europe [Hebrew], volume 2. Mekize Nirdamim, Jerusalem. find
4. Ethan Fetaya, Yonatan Lifshitz, Elad Aaron, and Shai Gordin. 2020. Restoration of fragmentary Babylonian texts using recurrent neural networks. Proceedings of the National Academy of Sciences (PNAS), 117 (37):22743—-22751. find our card
5. Niv Fono, Harel Moshayof, Eldar Karol, Itai Assraf, and Mark Last. 2024. Embible: Reconstruction of Ancient Hebrew and Aramaic texts using transformers. In Findings of the Association for Computational Linguistics: EACL 2024, pages 846–852, St. Julian’s, Malta. Association for Computational Linguistics. find our card
6. Eylon Gueta, Avi Shmidman, Shaltiel Shmidman, Cheyn Shmuel Shmidman, Joshua Guedalia, Moshe Koppel, Dan Bareket, Amit Seker, and Reut Tsarfaty. 2023. Large pre-trained models with extra-large vocabularies: A contrastive analysis of hebrew bert models and a new one to outperform them all. Preprint, arXiv:2211.15199. ↗ our card
7. Menachem I. Kahana. 2005. The Genizah Fragments of the Halakhic Midrashim [Hebrew]. Magnes Press, Jerusalem. find
8. Benjamin Richler. 2014. Guide to Hebrew Manuscript Collections, second, revised edition. The Israel Academy of Sciences and Humanities, Jerusalem. find
9. Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Shaked Greenfeld, and Reut Tsarfaty. 2021. Alephbert:a hebrew large pre-trained language model to start-off your hebrew nlp application with. Preprint, arXiv:2104.04052. ↗ our card
10. Avi Shmidman, Joshua Guedalia, Shaltiel Shmidman, Cheyn Shmuel Shmidman, Eli Handel, and Moshe Koppel. 2022. Introducing berel: Bert embeddings for rabbinic-encoded language. Preprint, arXiv:2208.01875. ↗ our card
11. Thea Sommerschield, Yannis Assael, John Pavlopoulos, Vanessa Stefanak, Andrew Senior, Chris Dyer, John Bodel, and Jonathan Prag. 2023. Machine learning for ancient languages: A survey. Computational Linguistics, 49(3):703–747. find
12. Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2021. Fast wordpiece tokenization. Preprint, arXiv:2012.15524. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2024-shmidman-msbert-new-model-reconstruction-lacunae/