Mova0.5 pre-release⌕
☰
home / docs / papers / Ancient Hebrew / An Information-Theoretic Approach to Identifying Formulaic Clusters in Textual Data

An Information-Theoretic Approach to Identifying Formulaic Clusters in Textual Data

What it is about

The paper asks whether the formulaic (repetitive, rigidly structured) parts of a text can be separated from the rest without labels. It proposes an unsupervised soft clustering method. Each text unit gets a soft membership in one of two clusters, and the score is the cluster-weighted negative log-likelihood, so that the self-information (−log p) of each unit under its cluster matters. Discrete versions use Bernoulli, Binomial or Multinomial models for presence or counts of features; a Gaussian version uses a regularised weighted covariance. On synthetic data the method is often better than GMM, k-means, DBSCAN and cross-entropy clustering in sparse, high-dimensional settings, but not uniformly. On the Hebrew Bible (STEPBible Leningrad morphology, cumulative n-gram counts over verse windows) it is run on three expert partitions: Genesis (genealogies versus narrative), Exodus (P versus non-P) and Leviticus (P versus H). Across a grid of n-gram size, window length and feature count, the share of configurations with normalised MCC of 85% or more is 40.0% against 14.6% for k-means in Genesis, 68.0% against 30.0% in Exodus, and 45.5% against 29.8% in Leviticus. The best P/H configuration (3-grams, 12-verse window, 500 features) reaches MCC 93.5%. The authors also show that the agreement depends on the scale of the representation.

Ideas

For Mova

I1 is a feature for world and for the discourse work in Discourse: narrative, speech and the shape of a text. I2 is a rule for every evaluation in tools/hbo. I3 is a test set for tools/macula (macula compare), used on the expert labels only, not on the paper's cluster output.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.1017/chr.2025.10011 · arXiv 2503.07303

@misc{yoffe2025information,
  title = {{An Information-Theoretic Approach to Identifying Formulaic Clusters in Textual Data}},
  author = {Yoffe, Gideon and Segev, Yair and Sober, Barak},
  year = {2025},
  doi = {10.1017/chr.2025.10011},
  eprint = {2503.07303},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1017/chr.2025.10011}
}

References

67 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Albertz, R. 2018. The recent discussion on the formation of the Pentateuch/Hexateuch. Hebrew Studies 59:65–92. find
  2. Altonji, Joseph G, and Lewis M Segal. 1996. Small-sample bias in GMM estimation of covariance structures. Journal of Business & Economic Statistics 14 (3): 353–366. find
  3. Antonia, A., H. Craig, and J. Elliott. 2014. Language chunking, data sparseness, and the value of a long marker list: explorations with word n-grams and authorial attribution. Literary and Linguistic Computing 29 (2): 147–163. find
  4. Ashurbekova, Karina, Antoine Usseglio-Carleve, Florence Forbes, and Sophie Achard. 2021. Optimal shrinkage for robust covariance matrix estimators in a small sample size setting. find
  5. Bickel, Peter J, and Elizaveta Levina. 2008. Regularized estimation of large covariance matrices. find
  6. Bühler, A., G. Yoffe, T. Römer, B. Sober, I. Finkelstein, E. Piasetzky, and N. Dershowitz. 2024. Exploring the stylistic uniqueness of the Priestly source in Genesis and Exodus through a statistical/computational lens. Zeitschrift für die alttestamentliche Wissenschaft 136 (2). find
  7. Byock, Jesse L. 1984. Saga form, oral prehistory, and the Icelandic social context. New Literary History 16 (1): 153–173. find
  8. Casale, Francesco Paolo, Adrian Dalca, Luca Saglietti, Jennifer Listgarten, and Nicolo Fusi. 2018. Gaussian process prior variational autoencoders. Advances in neural information processing systems 31. find
  9. Church, Kenneth, and Patrick Hanks. 1990. Word association norms, mutual information, and lexicography. Computational linguistics 16 (1): 22–29. find
  10. Clover, Carol J. 1982. The medieval saga. Cornell University Press. find
  11. Coleman, Stephen. 2019. The psalmic oral formula revisited: a cognitive-performative approach. Biblical Interpretation 27 (2): 186–207. find
  12. Davis, Jason, and Inderjit Dhillon. 2006. Differential entropic clustering of multivariate gaussians. Advances in Neural Information Processing Systems 19. find
  13. Dershowitz, I., N. Akiva, M. Koppel, and N. Dershowitz. 2015. Computerized source criticism of biblical texts. Journal of Biblical Literature 134 (2): 253–271. find
  14. Devlin, J., M.-W. Chang, K. Lee, and K. Toutanova. 2018. Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  15. Dilokthanakul, Nat, Pedro AM Mediano, Marta Garnelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. 2016. Deep unsupervised clustering with gaussian mixture variational autoencoders. arXiv preprint arXiv:1611.02648. ↗
  16. Ester, Martin, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. 1996. Density-based spatial clustering of applications with noise. In Int. conf. knowledge discovery and data mining, vol. 240. 6. find
  17. Faigenbaum-Golovin, Shira, Alon Kipnis, Axel Bühler, Eli Piasetzky, Thomas Römer, and Israel Finkelstein. 2024. Critical biblical studies via word frequency analysis: unveiling text authorship. arXiv preprint arXiv:2410.19883. ↗ our card
  18. Fraade, Steven D. 1991. From tradition to commentary: Torah and its interpretation in the Midrash Sifre to deuteronomy. Vol. 73. SUNY Press. find
  19. Gamallo, Pablo, José Ramom Pichel Campos, and Inaki Alegria. 2017. A perplexity-based method for similar languages discrimination. In Proceedings of the fourth workshop on nlp for similar languages, varieties and dialects (vardial), 109–114. find
  20. Giraldo, Luis Gonzalo Sanchez, Murali Rao, and Jose C Principe. 2014. Measures of entropy from data using infinitely divisible kernels. IEEE Transactions on Information Theory 61 (1): 535–548. find
  21. Gitay, Yehoshua. 1980. Tradition and interpretation: a study of the use and application of formulaic language in the so-called ebed YHWH-psalms. find
  22. Haran, M. 1981. Behind the scenes of history: determining the date of the Priestly source. Journal of Biblical Literature 100 (3): 321–333. find
  23. Hartigan, John A, Manchek A Wong, et al. 1979. A k-means clustering algorithm. Applied statistics 28 (1): 100–108. find
  24. Holzinger, H. 1893. Einleitung in den Hexateuch. Vol. 1. Mohr Siebeck. find
  25. Houdouin, Pierre, Esa Ollila, and Frédéric Pascal. 2023. Regularized em algorithm. In Icassp 2023-2023 ieee international conference on acoustics, speech and signal processing (icassp), 1–5. IEEE. find
  26. Huang, Rongqing, and John HL Hansen. 2007. Dialect classification on printed text using perplexity measure and conditional random fields. In 2007 ieee international conference on acoustics, speech and signal processing-icassp’07, 4:IV–993. IEEE. find
  27. Huertas-Tato, J., A. Martín, and D. Camacho. 2023. Understanding writing style in social media with a supervised contrastively pre-trained transformer. arXiv (October). https://doi.org/10.48550/arXiv.2310.11081. eprint: 2310.11081. ↗
  28. Hurvitz, A. 1968. The chronological significance of Aramaisms in Biblical Hebrew. Israel Exploration Journal, 234–240. find
  29. . 2014. A concise lexicon of Late Biblical Hebrew. Linguistic Innovations in the Writings of the Second Temple Period, 160. find
  30. Jensen, Minna Skafte. 1980. The Homeric question and the oral-formulaic theory. Vol. 20. Museum Tusculanum Press. find
  31. Klakow, Dietrich, and Jochen Peters. 2002. Testing the correlation of word error rate and perplexity. Speech Communication 38 (1-2): 19–28. find
  32. Klostermann, August. 1907. Der Pentateuch: beiträge zu seinem verständis und seiner entstehungsgeschicte. G. Böhme. find
  33. Knohl, I. 2007. The sanctuary of silence: the Priestly Torah and the holiness school. Eisenbrauns. find
  34. Kuenen, Abraham, and Philip Henry Wicksteed. 1886. An historico-critical inquiry into the origin and composition of the Hexateuch (Pentateuch and book of Joshua). Macmillan. find
  35. Kurzynski, Maciej. 2023. The stylometry of Maoism: quantifying the language of Mao Zedong. In Proceedings of the joint 3rd international conference on natural language processing for digital humanities and 8th international workshop on computational linguistics for uralic languages, 76–81. find
  36. Li, Zuchao, Rui Wang, Kehai Chen, Masso Utiyama, Eiichiro Sumita, Zhuosheng Zhang, and Hai Zhao. 2020. Data-dependent gaussian prior objective for language generation. In International conference on learning representations. find
  37. Long, Burke O, Sven Tengström, and Sven Tengstrom. 1984. Die toledotformel und die literarische struktur der priesterlichen erweiterungsschicht im Pentateuch. find
  38. Magoun Jr, Francis P. 1953. Oral-formulaic character of Anglo-Saxon narrative poetry. Speculum 28 (3): 446–467. find
  39. Neusner, Jacob. 1988. The Mishnah: an introduction. Bloomsbury Publishing PLC. find
  40. Paquot, Magali, and Sylviane Granger. 2012. Formulaic language in learner corpora. Annual Review of Applied Linguistics 32:130–149. find
  41. Peckham, Brian. 2019. The composition of the Deuteronomistic History. Vol. 35. Brill. find
  42. Plum, Karin Friis. 1989. Genealogy as theology. Scandinavian Journal of the Old Testament 3 (1): 66–92. find
  43. Polak, Frank H. 2006. Linguistic and stylistic aspects of epic formulae in ancient Semitic poetry and biblical narrative. Biblical Hebrew in Its Northwest Semitic Setting. Jerusalem: The Hebrew University Magnes Press and Winona Lake: Eisenbrauns, 285–304. find
  44. . 2017. Syntactic-stylistic aspects of the so-called “priestly” work in the torah. Le-Ma ‘an Ziony: Studies in Honor of Ziony Zevit, 345–382. find
  45. Principe, Jose C. 2010. Information theoretic learning: Renyi’s entropy and kernel perspectives. Springer Science & Business Media. find
  46. Radday, Y. T., and H. Shore. 1985. Genesis: an authorship study in computer-assisted statistical linguistics. Vol. 103. Analecta Biblica. Biblical Institution Press. find
  47. Radford, Alec, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748–8763. PmLR. find
  48. Read, John, and ISP Nation. 2008. Measurement of formulaic sequences. In Formulaic sequences: acquisition, processing and use, 23–35. John Benjamins Publishing Company. find
  49. Römer, Thomas. 2009. The Exodus narrative according to the Priestly document. The Strata of the Priestly Writings. Contemporary Debate and Future Directions, 157–174. find
  50. Ross, Jerome Clayton. 1997. The composition of the Holiness Code (Lev. 17-26). University of Pittsburgh. find
  51. Royer, Martin. 2017. Adaptive clustering through semidefinite programming. Advances in Neural Information Processing Systems 30. find
  52. Schmid, Konrad. 2015. Distinguishing the world of the Exodus narrative from the world of its narrators: the question of the priestly Exodus account in its historical setting. Israel’s Exodus in Transdisciplinary Perspective: Text, Archaeology, Culture, and Geoscience, 331–344. find
  53. Shannon, Claude Elwood. 1948. A mathematical theory of communication. The Bell system technical journal 27 (3): 379–423. find
  54. Shectman, Sarah, and Joel S Baden. 2009. The strata of the priestly writings: contemporary debate and future directions. Vol. 95. Theologischer Verlag Zürich. find
  55. Shmidman, A., J. Guedalia, S. Shmidman, C. S. Shmidman, E. Handel, and M. Koppel. 2022. Introducing berel: bert embeddings for Rabbinic-encoded language. arXiv preprint arXiv:2208.01875. ↗ our card
  56. Smith, M. S. 1996. The literary arrangement of the Priestly redaction of Exodus: a preliminary investigation. The Catholic Biblical Quarterly 58 (1): 25–50. find
  57. Stamatatos, E. 2009. A survey of modern authorship attribution methods. Journal of the American Society for Information Science and Technology 60 (3): 538–556. find
  58. Stipp, Hermann-Josef. 2017. 11 formulaic language and the formation of the Book of Jeremiah. In Jeremiah’s scriptures, 145–165. Brill. find
  59. Tabor, Jacek, and Przemyslaw Spurek. 2014. Cross-entropy clustering. Pattern Recognition 47 (9): 3046–3059. find
  60. Vink, J. G. 1969. The date and origin of the Priestly code in the Old Testament. In The Priestly code and seven other studies, 1–144. Brill. find
  61. Vlassis, Nikos, and Aristidis Likas. 2002. A greedy em algorithm for gaussian mixture learning. Neural processing letters 15:77–87. find
  62. Wärters, William R. 1976. Formula criticism and the poetry of the Old Testament. find
  63. Wellhausen, J. 1885. Prolegomena to the history of Israel: with a reprint of the article Israel from the Encyclopaedia Britannica. A. & C. Black. find
  64. Wood, David. 2019. Classifying and identifying formulaic language. In The routledge handbook of vocabulary studies, 30–45. Routledge. find
  65. Yang, Linxiao, Ngai-Man Cheung, Jiaying Li, and Jun Fang. 2019. Deep clustering by gaussian mixture variational autoencoders with graph embedding. In Proceedings of the ieee/cvf international conference on computer vision, 6440–6449. find
  66. Yao, Jianfeng, Shurong Zheng, and ZD Bai. 2015. Sample covariance matrices and high-dimensional data analysis. Cambridge UP, New York. find
  67. Yoffe, G., A. Bühler, N. Dershowitz, T. Römer, E. Piasetzky, I. Finkelstein, and B. Sober. 2023. A statistical exploration of text partition into constituents: the case of the Priestly source in the books of Genesis and Exodus. In Findings of the association for computational linguistics: acl 2023, 1918–1940. Toronto, Canada: Association for Computational Linguistics. https://aclanthology.org/2023.f indings-acl.121. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/langstream-2025-yoffe-information-theoretic-approach-identifying-formulaic/