EvaCun 2025 Shared Task: Lemmatization and Token Prediction in Akkadian and Sumerian using LLMs

What it is about

The shared task asks for two things on cuneiform text: lemmatization (the base form of each word) and token prediction (filling a masked word from its context). The data are new for NLP: transliterated first-millennium literary texts from the Electronic Babylonian Library and Old Babylonian letters from Archibab, about 10,200 fragments for lemmatization and about 28,500 for token prediction, with an 80/20 split. Scores are plain accuracy, broken down into function and non-function words, and in-vocabulary and out-of-vocabulary (OOV) words. Three teams submitted lemmatizers and one a token predictor. The best lemmatizer reaches 0.94 overall (0.97 in-vocabulary, 0.72 OOV), a bit above the BabyLemmatizer baseline (0.93) and well above a dictionary lookup that picks the most common lemma (0.83, with 0.00 OOV). Token prediction is hard for every system: the best overall accuracy is 0.21, and OOV accuracy is 0.03 or below. The authors note that only four of sixteen teams that signed up finished.

Ideas

• I1. Report every score on the same four splits (all, function, non-function, in-vocabulary, OOV). The gap between in-vocabulary and OOV accuracy is the number to watch, and a rule system can be tested on it separately. The dictionary baseline's 0.00 OOV shows what a pure lookup cannot do.
• I2. Match predicted lemmas to gold through a fixed, documented normalisation (macron and circumflex, optional final mimation, optional initial waw), applied at evaluation time only. For Hebrew, the same kind of table can normalise spelling variants of lemmas in macula dict before scoring.
• I3. Use the two-model setup as a pattern: a context-blind lemmatizer first, then a context-based pass only on items with low confidence. The paper reports this as about 1% gain, which is small, but it shows the low-confidence items as the place to spend effort.

For Mova

I1 is a measurement for any lemma task in treebanks/akk and treebanks/hbo/ptnk, and the dictionary floor is a required baseline. I2 belongs in the evaluation code of macula compare and in the lemma notes of The lexicon. I3 is an idea for tools/hbo lemma handling, to test before it is built.

Topics: Akkadian and cuneiform NLP

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.alp-1.33

@inproceedings{gordin2025evacun,
  title = {{EvaCun 2025 Shared Task: Lemmatization and Token Prediction in Akkadian and Sumerian using LLMs}},
  author = {Gordin, Shai and Sahala, Aleksi and Spencer, Shahar and Klein, Stav},
  booktitle = {Proceedings of the Second Workshop on Ancient Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.alp-1.33},
  url = {https://doi.org/10.18653/v1/2025.alp-1.33}
}

References

37 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Michele Cammarosano. 2014. 3d-joins und schrift-metrologie: A quantitative approach to cuneiform palaeography. In Current Research in Cuneiform Paleography. Proc. of a Workshop held at the 60th Rencontre Assyriologique Internationale. University of Warsaw. find
2. Dominique Charpin. 2014. Ressources assyriologiques sur internet. Bibliotheca Orientalis, 71(3):331–357. find
3. Danlu Chen, Aditi Agarwal, Taylor Berg-Kirkpatrick, and Jacobo Myerston. 2023. CuneiML: A Cuneiform Dataset for Machine Learning. Journal of Open Humanities Data, 9(1):30. find
4. Y. Cobanoglu, J. Laasonen, F. Simonjetz, I. Khait, S. Cohen, Z. Földi, A. Hätinen, A. Heinrich, T. Mitto, G. Rozzi, L. Sáenz, and E. Jiménez. 2024. Transliterated cuneiform tablets of the electronic babylonian library platform. Journal of Open Humanities Data, 10(1):19. find
5. Yunus Cobanoglu. 2023. Transliterated Fragments of the Electronic Babylonian Literature Project (eBL). Zenodo. find
6. Yael Cohen-Sygal and Shuly Wintner. 2006. Finite-state registered automata for non-concatenative morphology. Computational Linguistics, 32(1):49–82. find
7. Tim Collins, Sandra I Woolley, Luis Hernandez Munoz, Andrew Lewis, Eugene Ch’ng, and Erlend Gehlken. 2014. Computer-assisted reconstruction of virtual fragmented cuneiform tablets. In 2014 International Conference on Virtual Systems & Multimedia (VSMM), pages 70–77. IEEE. find
8. Peter T. Daniels. 2023. “Look with thine ears”: Why Writing Is Syllable-based. WORD, 69(1):91–116. Publisher: Routledge. Liudmila L. Fedorova. On the Typology of Writing Systems. In Grapholinguistics in the 21st Century, pages 805–824, Online. Fluxus Editions. find
9. Ethan Fetaya, Yonatan Lifshitz, Elad Aaron, and Shai Gordin. 2020. Restoration of fragmentary babylonian texts using recurrent neural networks. Proceedings of the National Academy of Sciences, 117(37):22743–22751. ↗ our card
10. Denis Fisseler. 2019. Contributions to computer-aided analysis of cuneiform tablet fragments. Ph.D. thesis, Dissertation, Dortmund, Technische Universität, 2019. find
11. Gösta Gabriel, Karenleigh A. Overmann, and Annick Payne, editors. 2021. Signs - sounds - semantics: nature and transformation of writing systems in the Ancient Near East. Number 13 in Wiener offene Orientalistik. Ugarit Verlag, Münster. Sh. find
12. Gordin. 2014. Visualizing Knowledge and Creating Meaning in Ancient Writing Systems Proceedings of the International Workshop of the Research Group “Notational Iconicity”, 24-25 Sep. 2010. Berliner Beiträge zum Vorderen Orient. PeWe-Verlag, Glad-beck. Type: Edited Book. find
13. Shai Gordin, Gai Gutherz, Ariel Elazary, Avital Romach, Enrique Jiménez, Jonathan Berant, and Yoram Cohen. 2020. Reading Akkadian cuneiform using natural language processing. PLOS ONE, 15(10):e0240511. find
14. Gai Gutherz, Shai Gordin, Luis Sáenz, Omer Levy, and Jonathan Berant. 2023. Translating Akkadian to English with neural machine translation. PNAS Nexus, 2(5):pgad096. find
15. W. W. Hallo. 1991. The Concept of Canonicity in Cuneiform and Biblical Literature: A Comparative Apprasial. In K. L. Younger, W. W. Hallo, and B. F. Batto, editors, The Biblical Canon in Comparative Perspective: Scripture in Context IV, pages 1–19. Edwin Mellen Press, Lewiston. Type: Book Section. find
16. Jenna Kanerva, Filip Ginter, Niko Miekka, Akseli Leino, and Tapio Salakoski. 2018. Turku neural parser pipeline: An end-to-end system for the conll 2018 shared task. In Proceedings of the CoNLL 2018 Shared Task: Multilingual parsing from raw text to universal dependencies, pages 133–142. find our card
17. Laura Kataja and Kimmo Koskenniemi. 1988. Finite-state Description of Semitic Morphology: A Case Study of Ancient Accadian. In Coling Budapest 1988 Volume 1: International Conference on Computational Linguistics. find our card
18. Koren Lazar, Benny Saret, Asaf Yehudai, Wayne Horowitz, Nathan Wasserman, and Gabriel Stanovsky. 2021. Filling the gaps in Ancient Akkadian texts: A masked language modelling approach. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4682–4691, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find our card
19. Aaron Macks. 2002. Parsing akkadian verbs with prolog. In Proceedings of the ACL-02 workshop on Computational approaches to semitic languages. find our card
20. Matthew Ong and Shai Gordin. 2024. Linguistic annotation of cuneiform texts using treebanks and deep learning. Digital Scholarship in the Humanities, 39(1):296–307. find
21. Émilie Pagé-Perron, Maria Sukhareva, Ilya Khait, and Christian Chiarcos. 2017. Machine Translation and Automated Analysis of the Sumerian Language. In Proceedings of the Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 10–16, Vancouver, Canada. Association for Computational Linguistics. find
22. Aleksi Sahala. 2014. Babyparser: Muinaisbabylonian morfologian ohjelmallinen jäsentäminen. Helsinki: University of Helsinki, MA Thesis. find
23. Aleksi Sahala. 2021. Contributions to computational assyriology. Helsinki: University of Helsinki. find
24. Aleksi Sahala, Tero Alstola, Jonathan Valk, and Krister Linden. 2022. Babylemmatizer: A lemmatizer and pos-tagger for akkadian. In CLARIN Annual Conference, pages 14–18. CLARIN ERIC. find
25. Aleksi Sahala and Eliese-Sophia Lincke. 2024. Neural lemmatization and pos-tagging models for coptic, demotic and earlier egyptian. In Proceedings of the 1st Workshop on Machine Learning for Ancient Languages (ML4AL 2024), pages 87–97. find
26. Stefan Weninger, Geoffrey Kahn, Michael P. Streck, and Janet C. E. Watson, editors. 2011. The Semitic Languages: An International Handbook. Number 36 in Handbooks of Linguistics and Communication Science. De Gruyter, Berlin. find
27. Imed Zitouni, editor. 2014. Natural language processing of semitic languages. Theory and applications of natural language processing. Springer, Berlin Heidelberg. find
28. Aleksi Sahala and Krister Lindén. 2023. A Neural Pipeline for Lemmatizing and POS-tagging Cuneiform Languages. In Proceedings of the Ancient Language Processing Workshop at the 14th International Conference on Recent Advances in Natural Language Processing RANLP 2023, pages 203–212. find
29. Aleksi Sahala, Miikka Silfverberg, Antti Arppe, and Krister Lindén. 2020a. Automated Phonological Transcription of Akkadian Cuneiform Texts. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3528–3534. find our card
30. Aleksi Sahala, Miikka Silfverberg, Antti Arppe, and Krister Lindén. 2020b. BabyFST: Towards a Finite-State Based Computational Model of Ancient Babylonian. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3886– 3894. find our card
31. Fabian Simonjetz, Jussi Laasonen, Yunus Cobanoglu, Alexander Fraser, and Enrique Jiménez. 2024. Reconstruction of cuneiform literary texts as text matching. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 13712–13721. find
32. Gustav Ryberg Smidt, Katrien De Graef, and Els Lefever. 2024. Keep me PoS-ted: experimenting with Part-of-Speech prediction on Old Babylonian letters. it - Information Technology, 65(6):264–274. Publisher: De Gruyter Oldenbourg. find
33. Thea Sommerschield, Yannis Assael, John Pavlopoulos, Vanessa Stefanak, Andrew Senior, Chris Dyer, John Bodel, Jonathan Prag, Ion Androutsopoulos, and Nando De Freitas. 2023. Machine Learning for Ancient Languages: A Survey. Computational Linguistics, 49(3):703–747. find
34. Steve Tinney. 2019. L2: How it Works. find
35. Stephen Tyndall. 2012. Toward automatically assembling hittite-language cuneiform tablet fragments into larger texts. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 243–247. find
36. Juan-Pablo Vita, editor. 2021. find
37. Vita, J.-P. (Ed.). (2021). History of the Akkadian language. Brill. Number volume 152/1-2 in Handbook of Oriental studies = Handbuch der Orientalistik : Section one. The Near and Middle East. Brill, Leiden. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2025-gordin-evacun-2025-shared-task-lemmatization/