Embible: Reconstruction of Ancient Hebrew and Aramaic Texts Using Transformers

What it is about

Damaged inscriptions are restored by masked-language-model prediction. The authors fine-tune TavBERT (character-level), mBERT, DistilBERT and AlephBERT-Gimmel on 22,144 Biblical verses, and test on 536 verses with 5%, 10% and 15% of characters or words masked. Their Ensemble combines TavBERT's character and whitespace predictions with the top word candidates of AlephBERT-Gimmel, and keeps only candidates that match both. With unknown whitespace, the Ensemble's WordHit@5 at 5% masking is 0.440, against 0.147 for AlephBERT-Gimmel alone (Table 2). Known whitespace raises all scores. The results are below the Akkadian reference, which the authors attribute to genre. The test uses masked Tanakh verses, not real inscriptions, and no Aramaic inscriptions were used in pretraining.

Ideas

• I1. Constrained word completion: when the damaged word's length and boundaries are known from the font size and gaps, drop every candidate of the wrong length. This is a deterministic filter that can sit on top of any candidate list in tools/ocrfix.
• I2. Keep only candidates that two independent sources agree on (a character-level guess and a word-level guess). The intersection is a rule Mova can implement with BTreeMap ordering, and it can be run with a dictionary such as macula dict in place of a neural word model.
• I3. Treat word separation as its own checked decision, since known whitespace raised accuracy a lot. This matters for the unpointed mode of tools/hbo.
• I4. Adopt their masking protocol (5/10/15%, two masking strategies, fixed seed) as a reproducible test harness for tools/ocrfix and for the robustness of tools/hbo.

For Mova

I1 and I2 apply to tools/ocrfix (correction of damaged printed editions). I3 applies to the unpointed mode of tools/hbo and to 1. Script and pointing. I4 is a test set for both. The inscriptions are for Biblical Aramaic texts, annotated, where the paper's limitation (masked Bible, not inscriptions) is noted.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.findings-eacl.56

@inproceedings{fono2024embible,
  title = {{Embible: Reconstruction of Ancient Hebrew and Aramaic Texts Using Transformers}},
  author = {Fono, Niv and Moshayof, Harel and Karol, Eldar and Assraf, Itai and Last, Mark},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-eacl.56},
  url = {https://doi.org/10.18653/v1/2024.findings-eacl.56}
}

References

17 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yannis Assael, Thea Sommerschield, and Jonathan Prag. 2019. Restoring ancient text using deep learning: a case study on Greek epigraphy. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6368–6375, Hong Kong, China. Association for Computational Linguistics. ↗
2. Avihay Chriqui and Inbal Yahav. 2022. Hebert and hebemo: A hebrew bert model and a tool for polarity analysis and emotion recognition. INFORMS Journal on Data Science, 1(1):81–95. ↗ our card
3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
4. Akkadian texts: A masked language modelling approach. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4682–4691, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
5. Jiaming Luo, Frederik Hartmann, Enrico Santus, Regina Barzilay, and Yuan Cao. 2021. Deciphering Undersegmented Ancient Scripts Using Phonetic Prior. Transactions of the Association for Computational Linguistics, 9:69–81. ↗ our card
6. Joana Ribeiro, Shashi Narayan, Shay Cohen, and Xavier Carreras. 2018. Local string transduction as sequence labeling. In 27th International Conference on Computational Linguistics, pages 1360–1371. Association for Computational Linguistics (ACL). find
7. Shruti Rijhwani, Antonios Anastasopoulos, and Graham Neubig. 2020. Ocr post correction for endangered language texts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5931–5942. ↗
8. Sascha Rothe, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. 2021. A simple recipe for multilingual grammatical error correction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 702–707. ↗
9. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. ↗
10. Samrat Dutta, Shreyansh Jain, Ayush Maheshwari, Souvik Pal, Ganesh Ramakrishnan, and Preethi Jyothi. 2022. Error correction in asr using sequence-to-sequence models. arXiv preprint arXiv:2202.01157. ↗
11. William M Schniedewind. 2006. Aramaic, the death of written hebrew, and language shift in the persian period. Margins of Writing, Origins of Cultures, pages 137–147. find
12. Ethan Fetaya, Yonatan Lifshitz, Elad Aaron, and Shai Gordin. 2020. Restoration of fragmentary babylonian texts using recurrent neural networks. Proceedings of the National Academy of Sciences, 117(37):22743–22751. ↗ our card
13. Judy Siegel-Itzkovich. 2023. Ancient tablet found on mount ebal predates known hebrew inscriptions. The Jerusalem Post. Avaialble at: https://www.jpost.com/archaeology/ article-743039. 14 May 2023. ↗
14. Eylon Guetta, Avi Shmidman, Shaltiel Shmidman, Cheyn Shmuel Shmidman, Joshua Guedalia, Moshe Koppel, Dan Bareket, Amit Seker, and Reut Tsarfaty. 2022. Large pre-trained models with extra-large vocabularies: A contrastive analysis of hebrew bert models and a new one to outperform them all. arXiv preprint arXiv:2211.15199. ↗ our card
15. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291–306. ↗
16. Omri Keren, Tal Avinari, Reut Tsarfaty, and Omer Levy. 2022. Breaking character: Are subwords good enough for mrls after all? arXiv preprint arXiv:2204.04748. ↗
17. Koren Lazar, Benny Saret, Asaf Yehudai, Wayne Horowitz, Nathan Wasserman, and Gabriel Stanovsky. 2021. Filling the gaps in Ancient find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2024-fono-embible-reconstruction-ancient-hebrew-aramaic/