Mova0.5 pre-release⌕
☰
home / docs / papers / Semitic languages / Expanding the Lexicon of Ge'ez Based African Languages: A Comparative Study of Amharic and Tigrinya

Expanding the Lexicon of Ge'ez Based African Languages: A Comparative Study of Amharic and Tigrinya

What it is about

XLM-R's Latin-centred subword vocabulary splits Amharic and Tigrinya words, written in the Ge'ez script, into many pieces: on the authors' held-out text, 2.07 tokens per word for Amharic and 3.13 for Tigrinya. The authors build VEXMLM by training language-specific SentencePiece tokenizers, adding 30,000 Ge'ez-script pieces to the XLM-R vocabulary (250,002 to 280,002 entries), and initialising the new embeddings with the mean of the old ones. It is then trained with continued masked-LM pretraining and fine-tuned on NER, QA and sentiment. Fertility falls by 28.0% for Amharic and 45.9% for Tigrinya, below Glot500. Downstream, NER improves over XLM-R (Tigrinya +0.0084 accuracy), extractive QA is clearly worse, and sentiment is comparable. An ablation on Tigrinya NER shows that adding vocabulary alone lowers accuracy on out-of-vocabulary words, and only continued pretraining makes the new pieces help. The authors state that the pretraining text has no documented licence and is not redistributed.

Ideas

For Mova

I1 is a measurement for tools/hbo, in both pointed and unpointed modes, on text from the Hebrew and Aramaic chapters. I2 applies to any lexicon change in The lexicon. I3 is a rule for the test sets of tools/hbo.

Topics: Named entity recognition

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2607.15209

@misc{teklehaymanot2026expanding,
  title = {{Expanding the Lexicon of Ge'ez Based African Languages: A Comparative Study of Amharic and Tigrinya}},
  author = {Teklehaymanot, Hailay Kidu and Yadeta, Debela Desalegn and Nejdl, Wolfgang},
  year = {2026},
  eprint = {2607.15209},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.15209}
}

References

40 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D’souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, et al. 2021. MasakhaNER: Named entity recognition for African languages. Transactions of the Association for Computational Linguistics, 9:1116–1131. find
  2. Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. 2023. Do all languages cost the same? tokenization in the era of commercial language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9904–9923, Singapore. Association for Computational Linguistics. find
  3. Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow. 2022. Adapting pre-trained language models to African languages via multilingual adaptive fine-tuning. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4336–4349, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. find
  4. Lisa Beinborn and Yuval Pinter. 2023. Analyzing cognitive plausibility of subword tokenization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4478–4486, Singapore. Association for Computational Linguistics. find
  5. Yekun Chai, Yewei Fang, Qiwei Peng, and Xuhong Li. 2024. Tokenization falling short: On subword robustness in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1582–1599, Miami, Florida, USA. Association for Computational Linguistics. find
  6. Ethan C. Chau and Noah A. Smith. 2021. Specializing multilingual language models: An empirical study. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 51–61, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  7. Iaroslav Chelombitko, Egor Safronov, and Aleksey Komissarov. 2024. QTok: A comprehensive framework for evaluating multilingual tokenizer quality in large language models. arXiv preprint arXiv:2410.12989. ↗
  8. Jonathan H Clark, Dan Garrette, Iulia Turc, and John Wieting. 2022. Canine: Pre-training an efficient tokenization-free encoder for language representation. Transactions of the Association for Computational Linguistics, 10:73–91. find
  9. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451. Association for Computational Linguistics. find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  11. Abteen Ebrahimi and Katharina Kann. 2021. How to adapt your pretrained multilingual model to 1600 languages. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4555–4567. Association for Computational Linguistics. find
  12. Gebremeskel Hagos Gebremedhin and Abera Asefa Mebrahtu. 2020. Linguistic evolution of ethiopic language: A comparative discussion. International Journal of Interdisciplinary Research and Innovations, 8(1):1–9. find
  13. Getatchew Haile. 1996. Ethiopic writing. In Peter T. Daniels and William Bright, editors, The World’s Writing Systems, pages 569–576. Oxford University Press, New York. find
  14. Omer Goldman, Avi Caciularu, Matan Eyal, Kris Cao, Idan Szpektor, and Reut Tsarfaty. 2024. Unpacking tokenization: Evaluating text compression and its correlation with model performance. arXiv preprint arXiv:2403.06265. ↗
  15. Jimin Hong, Gibbeum Lee, and Jaewoong Cho. 2024. Accelerating multilingual language model for excessively tokenized languages. arXiv preprint arXiv:2401.10660. ↗
  16. Ayyoob ImaniGooghari, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André Martins, François Yvon, and Hinrich Schütze. 2023. Glot500: Scaling multilingual corpora and language models to 500 languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1082–1117, Toronto, Canada. Association for Computational Linguistics. find
  17. Seungduk Kim, Seungtaek Choi, and Myeongho Jeong. 2024. Efficient and effective vocabulary expansion towards multilingual large language models. arXiv preprint arXiv:2402.14714. ↗
  18. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Association for Computational Linguistics. find
  19. Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, and Madian Khabsa. 2023. XLM-V: Overcoming the vocabulary bottleneck in multilingual masked language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13142–13152, Singapore. Association for Computational Linguistics. find
  20. Tomasz Limisiewicz, Jiří Balhar, and David Mareček. 2023. Tokenization impacts multilingual language modeling: Assessing vocabulary allocation and overlap across languages. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5661–5681, Toronto, Canada. Association for Computational Linguistics. find
  21. Yihong Liu, Peiqin Lin, Mingyang Wang, and Hinrich Schuetze. 2024. OFA: A framework of initializing unseen subword embeddings for efficient large-scale multilingual continued pretraining. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1067–1097, Mexico City, Mexico. Association for Computational Linguistics. find
  22. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  23. Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. 2022. WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3992–4006, Seattle, United States. Association for Computational Linguistics. find
  24. Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Abinew Ali Ayele, Nedjma Ousidhoum, David Ifeoluwa Adelani, Seid Muhie Yimam, Ibrahim Sa’id Ahmad, Meriem Beloucif, Saif M. Mohammad, Sebastian Ruder, Oumaima Hourrane, Pavel Brazdil, Alipio Jorge, Felermino Dário Mário António Ali, Davis David, Salomey Osei, Bello Shehu Bello, Falalu Ibrahim, Tajuddeen Gwadabe, and 8 others. 2023. AfriSenti: A Twitter sentiment analysis benchmark for African languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13968–13981, Singapore. Association for Computational Linguistics. find
  25. Kelechi Ogueji, Yuxin Zhu, and Jimmy Lin. 2021. Small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 116–126. find
  26. Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, et al. 2024. Byte latent transformer: Patches scale better than tokens. arXiv preprint arXiv:2412.09871. ↗
  27. Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. 2023. Language model tokenizers introduce unfairness between languages. Advances in Neural Information Processing Systems, 36. find
  28. David W. Phillipson. 2012. Foundations of an African Civilisation: Aksum and the Northern Horn, 1000 BC–AD 1300. Eastern Africa Series. James Currey, Woodbridge, UK. find
  29. François Remy, Pieter Delobelle, Hayastan Avetisyan, Alfiya Khabibullina, Miryam de Lhoneux, and Thomas Demeester. 2024. Trans-tokenization and cross-lingual vocabulary transfers: Language adaptation of LLMs for low-resource NLP. arXiv preprint arXiv:2408.04303. ↗
  30. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3118–3135. Association for Computational Linguistics. find
  31. Tilahun Abedissa Taffa, Ricardo Usbeck, and Yaregal Assabie. 2024. Low resource question answering: An Amharic benchmarking dataset. In Proceedings of the Fifth Workshop on Resources for African Indigenous Languages @ LREC-COLING 2024, pages 124–132, Torino, Italia. ELRA and ICCL. find
  32. Hailay Kidu Teklehaymanot, Dren Fazlija, Niloy Ganguly, Gourab Kumar Patro, and Wolfgang Nejdl. 2024. TIGQA: An expert-annotated question-answering dataset in Tigrinya. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16142–16161, Torino, Italia. ELRA and ICCL. find
  33. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems, 30. find
  34. Chunliu Wang, Huiyuan Lai, Malvina Nissim, and Johan Bos. 2023. Pre-trained language-meaning models for multilingual parsing and generation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5586–5600, Toronto, Canada. Association for Computational Linguistics. find
  35. Hai Wang, Dian Yu, Kai Sun, Jianshu Chen, and Dong Yu. 2019. Improving pre-trained multilingual model with vocabulary expansion. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 316–327, Hong Kong, China. Association for Computational Linguistics. find
  36. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291–306. ↗
  37. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498. find
  38. Atsuki Yamaguchi, Aline Villavicencio, and Nikolaos Aletras. 2024. How can we effectively expand the vocabulary of LLMs with 0.01GB of target language text? arXiv preprint arXiv:2406.11477. ↗
  39. Hailemariam Mehari Yohannes and Toshiyuki Amagasa. 2022. A method of named entity recognition for Tigrinya. ACM SIGAPP Applied Computing Review, 22(3):56–68. find
  40. Shiyue Zhang, Vishrav Chaudhary, Naman Goyal, James Cross, Guillaume Wenzek, Mohit Bansal, and Francisco Guzman. 2022. How robust is neural machine translation to language imbalance in multilingual tokenizer training? In Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 97–116, Orlando, USA. Association for Machine Translation in the Americas. Additional Downstream Metrics Table 6 reports macro-F1 for NER and SA and entity-level F1 for NER, complementing the accuracy scores in Table 4. Dataset XLM-R ∗ VEXMLM SA (Macro-F1) AfriSenti (Amharic) NER (Macro-F1) MasakhaNER (Amharic) Tigrinya NER (Tigrinya) NER (Entity-F1) MasakhaNER (Amharic) Tigrinya NER (Tigrinya) 0.5036 0.7109 0.7875 0.6248 0.6817 0.4971 ± 0.0193 0.7423 ± 0.0122 0.8219 ± 0.0069 0.6347 ± 0.0148 0.7282 ± 0.0079 Task (Metric) *Table 6: Macro-F1 and entity-level F1 (fractions) on the test splits. VEXMLM: mean ± standard deviation over five seeds. ∗ XLM-R: single run (seed 42). Bold marks the higher score where the difference exceeds one standard deviation of VEXMLM.* Reproducibility find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/langstream-2026-teklehaymanot-expanding-lexicon-ge-ez-based/