Large Language Models and Biblical Hebrew: Limitations, pitfalls, opportunities

What it is about

A critical reply to Elrod (2023; card hbo-2023-elrod-nothing-new-under-sun-study). Staps argues that Elrod's GPT results should be read far less favourably, because of interference on two levels: from modern languages (English, Modern Hebrew) in any Hebrew the model writes, and from translations and commentaries in any answer about a biblical text. Counting tasks are already solved exactly by deterministic tools over tagged data (Text-Fabric/ETCBC), which can also be corrected, so an LLM adds nothing but an interface without accuracy guarantees. GPT's generated "Biblical Hebrew" shows Modern Hebrew spelling and syntax (plene דיבר, a subject-initial asyndetic relative, a missing כי, an odd שמעה). The way forward is task-dependent: small models trained on Biblical Hebrew alone, white-box where the "why" matters (dating, classification), black-box only where accuracy alone counts, measured by benchmarks rather than impressions; the Bible's under 500 000 words limit what any trained model can do.

Ideas

• I1. An interference check for any Hebrew a teacher model produces: flag Modern Hebrew markers — plene spelling of piel and other forms the Torah writes defective (compare with the Torah dictionary's spelling column), lemmas absent from the Torah lexicon, missing כי/אֲשֶׁר where Biblical syntax requires them. Rule-based, explainable, a gate before any generated Hebrew enters a course.
• I2. A teacher's answer about a verse may come from translations, not the Hebrew; when Mova checks a teacher's reading, compare it with JPS 1917 too, and treat agreement with the translation against the trees as suspect, not as confirmation.
• I3. The paper's division of labour is Mova's: exact counts from the trees, models only where no exact method exists, white-box first. Cite it in the tools chapter as outside support.

For Mova

I1 is a small checker beside macula dict (spelling and lexicon from the Torah) for the Torah course and any generated Hebrew; its rules belong in 12. Biblical, Mishnaic and Modern Hebrew compared (Modern vs Biblical differences). I2 is a rule for teacher checks in tools/macula/src/prag.rs's agreement measure with JPS. I3 is a paragraph in 13-mova-tools.md. The paper is CC BY-SA: paraphrased only.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.7146/hn.v9i1.144177

@inproceedings{staps2024large,
  title = {{Large Language Models and Biblical Hebrew: Limitations, pitfalls, opportunities}},
  author = {Staps, Camil},
  booktitle = {HIPHIL Novum 9},
  year = {2024},
  doi = {10.7146/hn.v9i1.144177},
  url = {https://doi.org/10.7146/hn.v9i1.144177}
}

References

18 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Amgoud, Leila. 2023. ‘Explaining black-box classifiers: Properties and functions’. International Journal of Approximate Reasoning 155:40–65. https://doi.org/10.1016/j.ijar.2023.01.004 ↗
2. Assael, Yannis, Thea Sommerschield, Brendan Shillingford, Mahyar Bordbar, John Pavlopoulos, Marita Chatzipanagiotou, Ion Androutsopoulos, Jonathan Prag & Nando de Freitas. 2022. ‘Restoring and attributing ancient texts using deep neural networks’. Nature 603(7900):280–283. https://doi.org/10.1038/s41586-022-04448-z ↗
3. Bender, Emily M., Timnit Gebru, Angelina McMillan-Major & Shmargaret Shmitchell. 2021. ‘On the dangers of stochastic parrots: Can language models be too big? ’. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610–623. ACM. https://doi.org/10.1145/3442188.3445922 ↗
4. Bird, Steven & Haejoong Lee. 2007. ‘Graphical query for linguistic treebanks’. In Proceedings, PACLING 2007 – 10 th Conference of the Pacific Association for Computational Linguistics. Melbourne. http://hdl.handle.net/11343/34835 ↗
5. Elrod, A. G. 2023. ‘Nothing new under the sun? The study of Biblical Hebrew in the era of generative pre-trained AI’. HIPHIL Novum 8(2):1–32. https://doi.org/10.7146/hn.v8i2.143114 ↗ our card
6. Elrod, A. G. 2024. ‘Uncovering theological and ethical biases in LLMs: An integrated hermeneutical approach employing texts from the Hebrew Bible’. HIPHIL Novum 9(1):2–45. https://doi.org/10.7146/hn.v9i1.143407 ↗
7. Naaijer, Martijn, Constantijn Sikkel, Mathias Coeckelbergs, Jisk Attema & Willem Th. van Peursen. 2023. ‘A Transformer-based parser for Syriac morphology’. In Proceedings of the Ancient Language Processing Workshop associated with RANLP-2023, 23–29. https://aclanthology.org/2023.alp-1.3 ↗ our card
8. OpenAI. 2023. ‘GPT-4 technical report’. https://doi.org/10.48550/arXiv.2303.08774 Rillig, Matthias C., Marlene Ågerstrand, Mohan Bi, Kenneth A. Gould & Uli Sauerland. ‘Risks and benefits of Large Language Models for the environment’. Environmental Science & Technology 57(9):3464– 3466. https://doi.org/10.1021/acs.est.3c01106 ↗
9. This issue is not specific to LLMs, of course. For example, the Tiberias stylistic classifier for the Hebrew Bible is likewise closed source (https://tiberias.dicta.org.il/, retrieved February 9, 2024). ↗
10. Roorda, Dirk. 2017–2022. Annotation/text-fabric. Zenodo. https://doi.org/10.5281/zenodo.592193 Roorda, Dirk, Christiaan Erwich, Cody Kingham & SeHoon Park. 2017–2023. ETCBC/bhsa. Zenodo. https://doi.org/10.5281/zenodo.10049740 ↗
11. Rudin, Cynthia. 2019. ‘Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead’. Nature Machine Intelligence 1(5):206–215. https://doi.org/10.1038/s42256-019-0048-x ↗
12. Strubell, Emma, Ananya Ganesh & Andrew McCallum. 2019. ‘Energy and policy considerations for deep learning in NLP’. In Proceedings of the 57 th Annual Meeting of the Association for Computational Linguistics, 3645–3650. Florence: Association for Computational Linguistics. https://doi.org/10.18653/v1/P19-1355 ↗
13. Tunstall, Lewis, Leandro von Werra & Thomas Wolf. 2022. Natural Language Processing with Transformers: Building language applications with Hugging Face. Beijing: O’Reilly. find
14. Van de Bijl, Etienne P., Cody Kingham, Wido van Peursen & Sandjai Bhulai. 2019. ‘A probabilistic approach to syntactic variation in Biblical Hebrew’. https://doi.org/10.5281/zenodo.2546802 ↗ our card
15. Van der Schans, Yanniek, David Ruhe, Wido van Peursen & Sandjai Bhulai. 2020. ‘Clustering biblical texts using recurrent neural networks’. https://doi.org/10.5281/zenodo.4003509 ↗ our card
16. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser & Illia Polosukhin. 2017. ‘Attention is all you need’. Advances in Neural Information Processing Systems 30 (NIPS 2017). https://doi.org/10.48550/arXiv.1706.03762 ↗
17. Wilson-Wright, Aren. 2023. ‘COHeN’. https://huggingface.co/gngpostalsrvc/COHeN, version 82ff154, retrieved March 18, 2024. ↗
18. Young, Ian & Robert Rezetko. 2008. Linguistic dating of biblical texts. London: Equinox. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/hbo-2024-staps-large-language-models-biblical-hebrew/