Filling the Gaps in Ancient Akkadian Texts: A Masked Language Modelling Approach

What it is about

Editors of damaged cuneiform tablets fill broken parts from context, and the authors treat this as masked language modelling over Oracc transliterations (about 1M words, 2.3M signs). The number of missing signs is taken as known, since editors estimate it from the 'x' marks. A BERT model is trained from scratch, and a multilingual BERT is fine-tuned on Akkadian, and a greedy decoder extends single-token predictions to whole signs and words by counting sign delimiters. Multilingual pretraining with Akkadian fine-tuning gives the best results, with Hit@5 of about 89% on common genres; zero-shot multilingual BERT beats the Akkadian-only model by about 10%. Two Assyriologists judged the top predictions in a controlled test with real and random distractors, and found at least one of the top three useful in most cases, but accuracy falls for longer spans and for the Lexical and Decree genres.

Ideas

• I1. Gap filling for damaged Hebrew or Aramaic inscriptions and ostraca: take the number of missing letters as known, and rank candidate fillings from the letter n-grams of macula and the Biblical Aramaic texts, annotated corpus. This is a table, not a model, so it can be measured against the editions.
• I2. The delimiter-counting decoder: use word boundaries from the segmenter in tools/hbo to limit how many words a gap of a given length can span. This is a rule that the paper's decoder implements softly.
• I3. Report hit rates by genre and span length, and run a plausibility check with the original text and a random distractor as controls, as the paper does, so that annotator noise is measured.

For Mova

I1 goes into tools/ocrfix (damaged editions) and Biblical Aramaic texts, annotated. I2 is a rule for tools/hbo, with word-boundary tables from 1. Script and pointing. I3 is the evaluation design for any gap filler, kept with the held-out tests.

Topics: Akkadian and cuneiform NLP

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.emnlp-main.384

@inproceedings{lazar2021filling,
  title = {{Filling the Gaps in Ancient Akkadian Texts: A Masked Language Modelling Approach}},
  author = {Lazar, Koren and Saret, Benny and Yehudai, Asaf and Horowitz, Wayne and Wasserman, Nathan and Stanovsky, Gabriel},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.384},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.384}
}

References

17 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Michael Jursa. 2004. Accounting in neo-babylonian institutional archives: structure, usage, implications. Creating Economic Order: Record-keeping, Standardization, and the Development of Accounting in the Ancient Near East, Bethesda, pages 145–198. find
2. Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. 2020. Parsing with multilingual BERT, a small corpus, and a small treebank. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1324–1334, Online. Association for Computational Linguistics. find
3. Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Emerging cross-lingual structure in pretrained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6022– 6034, Online. Association for Computational Linguistics. find
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
5. Ethan Fetaya, Yonatan Lifshitz, Elad Aaron, and Shai Gordin. 2020. Restoration of fragmentary babylonian texts using recurrent neural networks. Proceedings of the National Academy of Sciences, 117(37):22743–22751. ↗ our card
6. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. find
7. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
8. Mikko Luukko, Aleksi Sahala, Sam Hardwick, and Krister Lindén. 2020. Akkadian treebank for early neo-assyrian royal inscriptions. In Proceedings of the 19th International Workshop on Treebanks and Linguistic Theories, pages 124–134, Düsseldorf, Germany. Association for Computational Linguistics. find our card
9. Hiroki Nakayama, Takahiro Kubo, Junya Kamura, Yasufumi Taniguchi, and Xu Liang. 2018. doccano: Text annotation tool for human. Software available from https://github.com/doccano/doccano. ↗
10. A Leo Oppenheim. 2013. Ancient Mesopotamia: portrait of a dead civilization. University of Chicago Press. find
11. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics. find
12. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996– 5001, Florence, Italy. Association for Computational Linguistics. find
13. Aleksi Sahala, Miikka Silfverberg, Antti Arppe, and Krister Lindén. 2020. BabyFST - towards a finite-state based computational model of ancient babylonian. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 3886– 3894, Marseille, France. European Language Resources Association. find our card
14. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149–5152. find
15. Christopher Bromhead Fleming Walker. 1987. Cuneiform, volume 3. Univ of California Press. find
16. Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833–844, Hong Kong, China. Association for Computational Linguistics. find
17. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, ukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08144. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/emnlp-2021-lazar-filling-gaps-ancient-akkadian-texts/