Nothing New Under the Sun? The Study of Biblical Hebrew in the Era of Generative Pre-trained AI

What it is about

The author tests GPT-3.5 and GPT-4 (May 2023) on Biblical Hebrew in three ways. Asked from memory to count things (the conjunction ו in Genesis 1, true answer 101; verbs in Jonah 1, true answer 77; forms of the root יטב in Jonah), every model fails; GPT-4 says 102 for ו, then drifts further when asked "are you certain?", and is worst when given the Hebrew text itself. As a text predictor it fills artificial lacunae in an unpointed Jonah 1 and real lacunae in a Dead Sea Scroll fragment (4QMMT) with plausible but uneven results. Asked to write Text-Fabric queries over the ETCBC database, it produces code that gives the exact counts, so the model is useful as a front end to a real database, not as an analyst.

Ideas

• I1. A ready set of count questions with gold answers (ו in Gen 1 = 101 by ETCBC; verbs in a chapter; forms of a root in a book) as a test of Mova's Torah trees and query tools: Mova must answer them exactly and say how it counted. Our segmentation may differ; record both counts and why.
• I2. The paper's result restates Mova's rule: the large model answers badly from memory and well only through a deterministic query. A plain-language → query path over the parse database (a teacher writes the query, Mova runs it and explains) is the right shape; the teacher never answers the count.
• I3. Lacuna filling as a measured task: delete 15% of words of a Torah chapter in 1–3-word gaps (by seed) and score a rule/frequency-based filler from the dictionary and trees; a baseline for later models.

For Mova

I1 is a test file for the parse database and the counts in 13. Biblical Hebrew in Mova: the tools (and a check for 10-lexicon.md's frequency figures). I2 belongs to the query front end of the parse database, with the teacher limited to writing queries. I3 is a measure on the Torah trees; any result goes to 13-mova-tools.md.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.7146/hn.v8i2.143114

@inproceedings{elrod2023nothing,
  title = {{Nothing New Under the Sun? The Study of Biblical Hebrew in the Era of Generative Pre-trained AI}},
  author = {Elrod, A. G.},
  booktitle = {HIPHIL Novum 8},
  year = {2023},
  doi = {10.7146/hn.v8i2.143114},
  url = {https://doi.org/10.7146/hn.v8i2.143114}
}

References

8 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bubeck, Sébastien, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, et al. 2023. "Sparks of Artificial General Intelligence: Early Experiments with GPT-4." Accessed May 28, 2023. http://arxiv.org/abs/2303.12712. ↗
2. ETCBC. n.d. "BHSA." Accessed May 28, 2023. https://etcbc.github.io/bhsa/. ↗
3. Future of Life Institute. 2023. "Pause Giant AI Experiments: An Open Letter." Accessed May 28, 2023. https://futureoflife.org/open-letter/pause-giant-ai-experiments/. ↗
4. "Generative AI Could Raise Global GDP by 7%." 2023. Accessed May 28, 2023. https://www.gold-mansachs.com/intelligence/pages/generative-ai-could-raise-global-gdp-by-7-percent.html. ↗
5. OpenAI. n.d. "API Reference: Create Completion." OpenAI API Reference. Accessed May 28, 2023. https://platform.openai.com/docs/api-reference/models. ↗
6. OpenAI. n.d. "GPT-4 Is OpenAI’s Most Advanced System, Producing Safer and More Useful Responses." Accessed May 28, 2023. find
7. Parry, Donald W., and Emanuel Tov, eds. 2004. The Dead Sea Scrolls Reader. Vol. Texts Concerned with Religious Law. Leiden ; Boston: Brill. find
8. Roorda, Dirk. n.d. "Text-Fabric." Accessed May 28, 2023. https://annotation.github.io/text-fabric/tf/. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/hbo-2023-elrod-nothing-new-under-sun-study/