Identification of Parallel Passages Across a Large Hebrew/Aramaic Corpus

What it is about

A fast method for finding all approximate parallel passages (20+ words) in a large Hebrew/Aramaic corpus, where spelling (plene vs defective, final letters, א/י), particles and synonyms vary. Each word is reduced to its two rarest letters (by corpus letter frequency), keeping their order; this strips most prefixes and matres lectionis, so דילמא and דלמא both become דל. From every five-word window four "skip-grams" of four words are taken (each leaving one word out), packed into collision-free integers and indexed. A parallel is a cluster of at least 3 matching skip-grams, gaps of at most 8 words, spanning at least 20 words, found by binning matches along the diagonal of a dot-plot. On the Babylonian Talmud (1.8 million words) it finds 4 602 parallel pairs in about 11 seconds on one thread; on a test pair of tractates it finds all 41 parallels that exhaustive Levenshtein finds, plus 5 more, with no false positives seen.

Ideas

• I1. The two-rarest-letters key is a cheap spelling-robust word signature for unvocalised Hebrew: use it to match Torah words against other witnesses (Samaritan Pentateuch, Qumran texts, Mishnah quotations of verses) where plene/defective spelling differs, before any morphological analysis.
• I2. Skip-gram clusters as the parallel finder for quotation detection: find every Torah verse quoted or paraphrased in the Mishnah corpus, giving aligned Biblical–Mishnaic pairs for the comparison chapter.
• I3. The whole method is integer-only and deterministic; a Rust implementation is small and fits Mova's rule of BTreeMap order and two identical runs. Measure recall on known inner-Torah parallels (Exod 20 / Deut 5) against a plain lemma-Jaccard method.

For Mova

I1 is a helper for 1. Script and pointing (matres lectionis, plene/defective) and for any OCR or witness-alignment work. I2 feeds 11-mishnaic.md and 12-biblical-mishnaic-modern.md with aligned quotations. I3 is a tool to write beside tools/macula; the paper is read-only, so the method is reimplemented from its description.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.46298/jdmdh.1388 · arXiv 1602.08715

@article{shmidman2016identification,
  title = {{Identification of Parallel Passages Across a Large Hebrew/Aramaic Corpus}},
  author = {Shmidman, Avi and Koppel, Moshe and Porat, Ely},
  journal = {Journal of Data Mining \& Digital Humanities, Special Issue on Computer-Aided Processing of Intertextuality in Ancient Languages, Towards a Digital Ecosystem: NLP. Corpus infrastructure. Methods for Retrieving Texts and Computing Text Similarities (March 11, 2018) jdmdh:1388},
  year = {2016},
  doi = {10.46298/jdmdh.1388},
  eprint = {1602.08715},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.46298/jdmdh.1388}
}

References

14 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Basile C., Benedetto D., Caglioti G., and Degli Esposti M. A Plagiarism Detection Procedure in Three Steps: Selection, Matches and Squares, PAN 2009:19-23. find
2. Barron-Cedeno A., Basile C., Degli Esposti M. and Rosso P. Word length n-grams for text re-use detection, in Proceedings of the 11th international conference on Computational Linguistics and Intelligent Text Processing, ser. CICLing’10. 2010:687-699. find
3. Büchler M. Informationstechnische Aspekte des Historischen Text Re-use (Engl. Computational Aspects of Historical Text Re-use Detection), Ph.D. dissertation, Universitat Leipzig, 2013. find
4. Büchler, M., Burns, P. R., Müller, M., Franzini, E., and Franzini, G. Towards a Historical Text Re-use Detection, in: Biemann, C. and Mehler, A. (eds.) Text Mining, Theory and Applications of Natural Language Processing. 2014. ↗
5. Skolnik F. (ed.) Encyclopedia Judaica, second edition. Keter Publishing House (Jerusalem), 2007. find
6. Franzini G., Büchler M., Franzini E., and Moritz M. Scaling Historical Text Re-use. The IEEE International Conference on Big Data 2014 (IEEE BigData 2014), 2014:27-30. find
7. Grozea C., Gehl C., and Popescu M. ENCOPLOT: Pairwise Sequence Matching in Linear Time Applied to Plagiarism Detection. 3rd PAN Workshop. Uncovering Plagiarism, Authorship and Social Software Misuse. 2009:10. find
8. Grozea C., and Popescu M. The Encoplot Similarity Measure for Automatic Detection of Plagiarism Notebook for PAN at CLEF 2011. find
9. Guthrie D., Allison B., Liu W., Guthrie L. and Wilks Y. A Closer Look at Skip-gram Modelling. Proceedings of the 5th International Conference on Language Resources and Evaluation. 2006:1-4. find
10. Harris Z. Distributional structure. Word. 1954;10(23):146-162. http1 https://www.dropbox.com/s/e3syo8vhncptruj/FindAllSimilarPassages.exe ↗
11. Karp R. and Rabin M. O. Efficient randomized pattern-matching algorithms. IBM Journal of Research and Development 1987; 31(2):249-260. find
12. Klein E., Dershowitz N., Wolf L., Almogi O., and Wangchuk D. Finding Inexact Quotations Within a Tibetan Buddhist Corpus. Digital Humanities. 2014:486-488. find
13. Lee, J. A Computational Model of Text Reuse in Ancient Literary Texts. Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. 2007:472-479 find
14. Lee L. Measures of distributional similarity. 37th Annual Meeting of the Association for Computational Linguistics. 1999:25-32. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/hbo-2016-shmidman-identification-parallel-passages-across-large/