Beyond Base Predictors: Using LLMs to Resolve Ambiguities in Akkadian Lemmatization

What it is about

A system for the EvaCun 2025 Akkadian lemmatization task, built in three layers. Three base predictors (a dictionary lookup and two T5 models) propose a lemma. Where they disagree, which is 18.7% of the test tokens, an LLM module resolves the case: a Predictor proposes a lemma with reasoning from the eBL dictionary entries, and a Validator checks it. Before this, a rule-based domain detector decides whether the fragment follows the eBL or the Archibab lemma convention, and post-editing rules then adapt the output to that convention. The domain-aware dictionary raises validation accuracy from 77.6% to 82.6%, and the two T5 models reach 87.0% and 88.3%. On the 500 disagreement cases, the LLM stage reaches 77.4%. Where all base predictors agree, the error is about 2.8%. The error analysis says the system struggles most with small differences such as capitalisation (4% of all errors) and with a few ambiguous logograms: BI fails in 36% of its cases, and the divine name UTU is lemmatised as Šamaš where the gold has šamšu. Roman index errors make up 7.3% of all errors.

Ideas

• I1. Keep one lemma table per convention and fall back to a merged table when a token is missing from the domain table (the paper's domain-aware dictionary: about +5 points over one table). This is a deterministic, testable step for any multi-edition lexicon, such as treebanks/akk.
• I2. Detect the convention from surface cues before lemmatizing: bracket types for determinatives, subscript digits against accent marks for homograph indices. For Mova this is a rule over the input, and it can be checked by hand on a sample.
• I3. Use the disagreement set of cheap predictors as the place to spend effort. The LLM's answers on those cases can be used as training labels for a rule table, while the LLM itself stays out of test time. This matches Mova's teacher-only rule: the big model supplies labels, and the answer comes from Mova's own rules.
• I4. Write the per-domain post-editing as explicit string rules (drop the homograph index in one convention, change macrons to circumflexes in the other) with a test for each rule.

For Mova

I1, I2 and I4 apply to treebanks/akk, where lemma conventions are a data problem, and to tools/ocrfix for edition-specific conventions in printed Hebrew and Aramaic. I3 is a training procedure, to be run only with the operator's word and kept out of the test path. Any test of these ideas should be measured against the dictionary-only floor.

Topics: Akkadian and cuneiform NLP

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.alp-1.30

@inproceedings{riemenschneider2025beyond,
  title = {{Beyond Base Predictors: Using LLMs to Resolve Ambiguities in Akkadian Lemmatization}},
  author = {Riemenschneider, Frederick},
  booktitle = {Proceedings of the Second Workshop on Ancient Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.alp-1.30},
  url = {https://doi.org/10.18653/v1/2025.alp-1.30}
}

References

13 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Our system was designed specifically for the Eva-Cun 2025 Shared Task, utilizing only the data provided by the task organizers. In a real-world implementation scenario, several enhancements could substantially improve performance. A straightforward improvement involves implementing more powerful base predictors. For instance, adapting the BabyLemmatizer or pre-training a more comprehensive T5 model on the full corpus of available digitized Akkadian texts– rather than being limited to the task-provided data– would establish a stronger foundation for the entire system. The BabyLemmatizer has demonstrated accuracy rates of approximately 95% (albeit on different datasets), indicating significant potential improvements for base lemmatizers that could be seamlessly integrated into our architecture. Acknowledgements Another limitation of our current approach stems from working with an LLM with closed weights. We thank the anonymous reviewers for their valu-As it is not possible to fine-tune Claude, we in- able suggestions. vested considerable effort in designing prompts that would guide the model to produce appropriately References formatted output, creating computational overhead Dominique Charpin. 2014. The Assyriologist and the during inference. Parameter-efficient fine-tuning Computer. The “Archibab” Project. Hebrew Bible methods such as LoRA (Hu et al., 2022) could po-and Ancient Israel, 3(1):137–153. tentially streamline this process by teaching the model expected output formats directly. Unfortu- Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu nately, our initial experiments with Qwen (Yang Chen. 2022. LoRA: Low-rank adaptation of large et al., 2024) and Gemma (Riviere et al., 2024)– language models. In International Conference on models offering accessible weights–yielded subop-Learning Representations. 191 find
2. Laura Kataja and Kimmo Koskenniemi. 1988. Finite-state description of Semitic morphology: A case study of ancient Accadian. In Coling Budapest 1988 Volume 1: International Conference on Computational Linguistics. find our card
3. Aaron Macks. 2002. Parsing Akkadian verbs with Prolog. In Proceedings of the ACL-02 Workshop on Computational Approaches to Semitic Languages, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. find our card
4. Piotr Nawrot. 2023. nanoT5: Fast & simple pre-training and fine-tuning of t5 models with limited resources. In Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023), pages 95–101, Singapore. Association for Computational Linguistics. find
5. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. ↗
6. Roman Yangarber, Jakub Piskorski, Anna Dmitrieva, Michał Marcińczuk, Pavel Přibáň, Piotr Rybak, and Josef Steinberger. 2023. Slav-NER: the 4th cross-lingual challenge on recognition, normalization, classification, and linking of named entities across Slavic languages. In Proceedings of the 9th Workshop on Slavic Natural Language Processing 2023 (Slavic-NLP 2023), pages 179–189, Dubrovnik, Croatia. Association for Computational Linguistics. find
7. Frederick Riemenschneider and Kevin Krahn. 2024. Heidelberg-boston @ SIGTYP 2024 shared task: Enhancing low-resource language analysis with character-aware hierarchical transformers. In Proceedings of the 6th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, pages 131–141, St. Julian’s, Malta. Association for Computational Linguistics. find
8. Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. ↗
9. Aleksi Sahala, Tero Alstola, Jonathan Valk, and Krister Linden. 2022. Babylemmatizer: A lemmatizer and pos-tagger for akkadian. In CLARIN Annual Conference Proceedings, 2022, CLARIN Annual Conference Proceedings, page 14–18, Netherlands. CLARIN ERIC. CLARIN Annual Conference ; Conference date: 10-10-2022 Through 12-10-2022. find
10. Aleksi Sahala and Krister Lindén. 2023. A neural pipeline for POS-tagging and lemmatizing cuneiform languages. In Proceedings of the Ancient Language Processing Workshop, pages 203–212, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria. find
11. Aleksi Sahala, Miikka Silfverberg, Antti Arppe, and Krister Lindén. 2020. BabyFST - towards a finite-state based computational model of ancient Babylonian. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3886– 3894, Marseille, France. European Language Resources Association. find our card
12. Steve Tinney. 2019. L2: How it works. Oracc: The Open Richly Annotated Cuneiform Corpus. find
13. Krzysztof Wróbel and Krzysztof Nowak. 2022. Transformer-based part-of-speech tagging and lemmatization for Latin. In Proceedings of the Second Workshop on Language Technologies for Historical and Ancient Languages, pages 193–197, Marseille, France. European Language Resources Association. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2025-riemenschneider-beyond-base-predictors-llms-resolve/