Exploring Tokenization Strategies and Vocabulary Sizes for Enhanced Arabic Language Models

What it is about

The authors pre-train a medium BERT-style Arabic model (based on a Safaya model on OSCAR) with four tokenizers (BPE, WordPiece, BPE after Farasa morphological segmentation, and word level) at three vocabulary sizes (16k, 28k, 44k), giving twelve models. Each is fine-tuned on four tasks: news classification (SANAD), hate speech (Levantine tweets), sentiment (ASAD, several dialects) and natural language inference. Farasa with BPE gives the best F1 on news, hate speech and sentiment in the paper's tables; the exception is sentiment, where Egyptian dialect forms are segmented badly. Morphological segmentation keeps the token-to-word ratio stable as the vocabulary shrinks, while plain BPE and WordPiece lose it. The vocabulary size has a smaller effect on the outcome than the authors expected, and the paper names out-of-vocabulary words as the main weakness of the morphological tokenizer.

Ideas

• I1. A token-to-word ratio report for the Hebrew segmenter in tools/hbo, per vocabulary size of its table of prefix and suffix splits, so that a segmenter change shows as a measured ratio before any downstream test.
• I2. A dialect and register test set for 8. Lexicon and register: colloquial Modern Hebrew clitic forms that the Hebrew segmenter cannot split, kept as a fixed list of cases, as the paper's Egyptian example shows.
• I3. Report out-of-vocabulary rate on held-out text for the segmenter, since the paper finds OOV words to be the main failure of the morphological method. Use it as a gate for new segmenter tables.

For Mova

I1 is a measurement for tools/hbo (segmenter) and macula dict, recorded with the segmenter tables. I2 applies to 8. Lexicon and register and 05. The verb: system, binyanim, tense and use. I3 is a test for tools/hbo on unseen MACULA text. No neural tokenizer is adopted; the paper is used only for the measures.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2403.11130

@misc{alrefaie2024exploring,
  title = {{Exploring Tokenization Strategies and Vocabulary Sizes for Enhanced Arabic Language Models}},
  author = {Alrefaie, Mohamed Taher and Morsy, Nour Eldin and Samir, Nada},
  year = {2024},
  eprint = {2403.11130},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.11130}
}

References

51 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abadji, J., Suarez, P.O., Romary, L., Sagot, B., 2022. Towards a Cleaner Document-Oriented Multilingual Crawled Corpus, in: 2022 Language Resources and Evaluation Conference, LREC 2022. European Language Resources Association (ELRA), pp. 4344–4355. find
2. Abdelali, A., Darwish, K., Durrani, N., Mubarak, H., 2016. Farasa: A Fast and Furious Segmenter for Arabic. NAACL-HLT 2016 - 2016 Conf. North Am. Chapter Assoc. Comput. Linguist. Hum. Lang. Technol. Proc. Demonstr. Sess. 11–16. https://doi.org/10.18653/V1/N16-3003 ↗ our card
3. Abdelali, A., Mubarak, H., Samih, Y., Hassan, S., Darwish, K., 2021. QADI: Arabic Dialect Identification in the Wild. WANLP 2021 - 6th Arab. Nat. Lang. Process. Work. Proc. Work. 1–10. find
4. Alfaidi, A., Alwadei, H., Alshutayri, A., Alahdal, S., 2023. Exploring the Performance of Farasa and CAMeL Taggers for Arabic Dialect Tweets. Int. Arab J. Inf. Technol. 20, 349–356. https://doi.org/10.34028/iajit/20/3/7 ↗
5. Alharbi, B., Alamro, H., Alshehri, M., Khayyat, Z., Kalkatawi, M., Jaber, I.I., Zhang, X., 2020. ASAD: A Twitter-based Benchmark Arabic Sentiment Analysis Dataset. find
6. Ali, M., Fromm, M., Thellmann, K., Rutmann, R., Lübbering, M., Leveling, J., Klug, K., Ebert, J., Doll, N., Schulze Buschhoff, J., Jain, C., Arno Weber, A., Jurkschat, L., Abdelwahab, H., John, C., Ortiz Suarez, P., Ostendorff, M., Weinbach, S., Sifa, R., Kesselheim, S., Flores-Herr, N., Iais, F., Institute, L., Jülich, F., Alpha, A., 2023. Tokenizer Choice For LLM Training: Negligible or Crucial? ArXiv abs/2310.0, null. find
7. Alkaoud, M., Syed, M., 2020. On the Importance of Tokenization in {A}rabic Embedding Models. Proc. Fifth Arab. Nat. Lang. Process. Work. 119–129. find
8. Alyafeai, Z., Ahmad, I., 2021. Arabic Compact Language Modelling for Resource Limited Devices. WANLP 2021 - 6th Arab. Nat. Lang. Process. Work. Proc. Work. 53–59. find
9. Alyafeai, Z., Al-shaibani, M.S., Ghaleb, M., Ahmad, I., 2023. Evaluating Various Tokenizers for Arabic Text Classification. Neural Process. Lett. 55, 2911–2933. https://doi.org/10.1007/s11063-022-10990-8 ↗
10. Antoun, W., Baly, F., Hajj, H., 2020. AraBERT: Transformer-based Model for Arabic Language Understanding, in: Al-Khalifa, H., Magdy, W., Darwish, K., Elsayed, T., Mubarak, H. (Eds.), Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection. European Language Resource Association, Marseille, France, pp. 9–15. find
11. Aribandi, V., Tay, Y., Schuster, T., Rao, J., Zheng, H.S., Mehta, S.V., Zhuang, H., Tran, V.Q., Bahri, D., Ni, J., Gupta, J., Hui, K., Ruder, S., Metzler, D., 2022. EXT5: TOWARDS EXTREME MULTI-TASK SCALING FOR TRANSFER LEARNING, in: ICLR 2022 -10th International Conference on Learning Representations. International Conference on Learning Representations, ICLR. find
12. Bostrom, K., Durrett, G., 2020. Byte Pair Encoding is Suboptimal for Language Model Pretraining, in: Cohn, T., He, Y., Liu, Y. (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online, pp. 4617–4624. https://doi.org/10.18653/v1/2020.findings-emnlp.414 ↗
13. Choe, D., Al-Rfou, R., Guo, M., Lee, H., Constant, N., 2019. Bridging the Gap for Tokenizer-Free Language Models. find
14. Devlin, J., Chang, M.W., Lee, K., Toutanova, K., 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. find
15. NAACL HLT 2019 - 2019 Conf. North Am. Chapter Assoc. Comput. Linguist. Hum. Lang. Technol. - Proc. Conf. 1, 4171–4186. find
16. Dhyani, B., 2021. Transfer Learning in Natural Language Processing: A Survey. Math. Stat. Eng. Appl. 70, 303–311. https://doi.org/10.17762/msea.v70i1.2312 ↗
17. Ding, S., Renduchintala, A., Duh, K., 2019. A Call for Prudent Choice of Subword Merge Operations in Neural Machine Translation, in: Forcada, M., Way, A., Haddow, B., Sennrich, R. (Eds.), Proceedings of Machine Translation Summit XVII: Research Track. European Association for Machine Translation, Dublin, Ireland, pp. 204–213. find
18. Einea, O., Elnagar, A., Al Debsi, R., 2019. SANAD: Single-label Arabic News Articles Dataset for automatic text categorization. Data Br. 25, 104076. https://doi.org/10.1016/j.dib.2019.104076 ↗
19. ElJundi, O., Antoun, W., El Droubi, N., Hajj, H., El-Hajj, W., Shaban, K., 2019. Hulmona (a-jòêg): The universal language model in arabic. ACL 2019 - 4th Arab. Nat. Lang. Process. Work. WANLP 2019 - Proc. Work. 68–77. https://doi.org/10.18653/V1/W19-4608 ↗
20. Gong, C., He, D., Tan, X., Qin, T., Wang, L., Liu, T.-Y., 2018. FRAGE: Frequency-Agnostic Word Representation, in: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (Eds.), Advances in Neural Information Processing Systems. Curran Associates, Inc. find
21. Hiraoka, T., Takase, S., Uchiumi, K., Keyaki, A., Okazaki, N., 2021. Joint Optimization of Tokenization and Downstream Model, in: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. Association for Computational Linguistics (ACL), pp. 244– 255. https://doi.org/10.18653/v1/2021.findings-acl.21 ↗
22. Jimenez, M., Maxime, C., Traon, Y. Le, Papadakis, M., 2018. On the impact of tokenizer and parameters on n-gram based code analysis, in: Proceedings - 2018 IEEE International Conference on Software Maintenance and Evolution, ICSME 2018. Institute of Electrical and Electronics Engineers Inc., pp. 437–448. https://doi.org/10.1109/ICSME.2018.00053 ↗
23. Kudo, T., 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. ACL 2018 - 56th Annu. Meet. Assoc. Comput. Linguist. Proc. Conf. (Long Pap. 1, 66–75. https://doi.org/10.18653/v1/p18-1007 ↗
24. Laurer, M., Van Atteveldt, W., Casas, A., Welbers, K., 2023. Less Annotating, More Classifying: Addressing the Data Scarcity Issue of Supervised Machine Learning with Deep Transfer Learning and BERT-NLI. Polit. Anal. 32, 84–100. https://doi.org/10.1017/pan.2023.20 ↗
25. Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., Wu, Z., Zhao, L., Zhu, D., Li, X., Qiang, N., Shen, D., Liu, T., Ge, B., 2023. Summary of ChatGPT-Related research and perspective towards the future of large language models. Meta-Radiology 1, 100017. https://doi.org/10.1016/j.metrad.2023.100017 ↗
26. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V., 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. find
27. Loshchilov, I., Hutter, F., 2019. Decoupled weight decay regularization. 7th Int. Conf. Learn. Represent. ICLR 2019. find
28. Lowphansirikul, L., Polpanumas, C., Jantrakulchai, N., Nutanong, S., 2021. WangchanBERTa: Pretraining transformer-based Thai Language Models. find
29. Metke-Jimenez, A., Raymond, K., Maccoll, I., 2011. Information extraction from web services: A comparison of tokenisation algorithms, in: Proceedings of the 2nd International Workshop on Software Knowledge, SKY 2011, in Conjunction with IC3K 2011. pp. 12–23. https://doi.org/10.5220/0003698000120023 ↗
30. Mikolov, T., Chen, K., Corrado, G., Dean, J., 2013. Efficient estimation of word representations in vector space. 1st Int. Conf. Learn. Represent. ICLR 2013 - Work. Track Proc. find
31. Mulki, H., Haddad, H., Bechikh Ali, C., Alshabani, H., 2019. L-HSAB: A Levantine Twitter Dataset for Hate Speech and Abusive Language. Association for Computational Linguistics (ACL), pp. 111–118. https://doi.org/10.18653/v1/w19-3512 ↗
32. Nayak, A., Timmapathini, H., Ponnalagu, K., Gopalan Venkoparao, V., 2020. Domain adaptation challenges of {BERT} in tokenization and sub-word representations of Out-of-Vocabulary words, in: Rogers, A., Sedoc, J., Rumshisky, A. (Eds.), Proceedings of the First Workshop on Insights from Negative Results in NLP. Association for Computational Linguistics, Online, pp. 1–5. https://doi.org/10.18653/v1/2020.insights-1.1 ↗
33. Oudah, M., Almahairi, A., Habash, N., 2019. The Impact of Preprocessing on Arabic-English Statistical and Neural Machine Translation, in: Forcada, M., Way, A., Haddow, B., Sennrich, R. (Eds.), Proceedings of Machine Translation Summit XVII: Research Track. European Association for Machine Translation, Dublin, Ireland, pp. 214–221. find
34. Park, C., Eo, S., Moon, H., Lim, H., 2021. Should we find another model?: Improving Neural Machine Translation Performance with ONE-Piece Tokenization Method without Model Modification, in: NAACL-HLT 2021 - 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Industry Papers. pp. 97–104. https://doi.org/10.18653/v1/2021.naacl-industry.13 ↗
35. Park, K., Lee, J., Jang, S., Jung, D., 2020. An Empirical Study of Tokenization Strategies for Various {K}orean {NLP} Tasks, in: Wong, K.-F., Knight, K., Wu, H. (Eds.), Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing. Association for Computational Linguistics, Suzhou, China, pp. 133–142. find
36. Pennington, J., Socher, R., Manning, C.D., 2014. GloVe: Global vectors for word representation. EMNLP 2014 - 2014 Conf. Empir. Methods Nat. Lang. Process. Proc. Conf. 1532–1543. https://doi.org/10.3115/v1/d14-1162 ↗
37. Rai, A., Borah, S., 2021. Study of various methods for tokenization, in: Lecture Notes in Networks and Systems. Springer Science and Business Media Deutschland GmbH, pp. 193–200. https://doi.org/10.1007/978-981-15-6198-6_18 ↗
38. Rust, P., Pfeiffer, J., Vulic, I., Ruder, S., Gurevych, I., 2021. How good is your tokenizer? On the monolingual performance of multilingual language models, in: ACL-IJCNLP 2021 - 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Proceedings of the Conference. Association for Computational Linguistics (ACL), pp. 3118–3135. https://doi.org/10.18653/v1/2021.acl-long.243 ↗
39. Safaya, A., Abdullatif, M., Yuret, D., 2020. KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media, in: 14th International Workshops on Semantic Evaluation, SemEval 2020 - Co-Located 28th International Conference on Computational Linguistics, COLING 2020, Proceedings. International Committee for Computational Linguistics, pp. 2054–2059. https://doi.org/10.18653/v1/2020.semeval-1.271 ↗
40. Schuster, M., Nakajima, K., 2012. Japanese and Korean voice search, in: ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings. pp. 5149–5152. https://doi.org/10.1109/ICASSP.2012.6289079 ↗
41. Sennrich, R., Haddow, B., Birch, A., 2016a. Neural machine translation of rare words with subword units. 54th Annu. Meet. Assoc. Comput. Linguist. ACL 2016 - Long Pap. 3, 1715– 1725. https://doi.org/10.18653/v1/p16-1162 ↗
42. Sennrich, R., Haddow, B., Birch, A., 2016b. Improving neural machine translation models with monolingual data. 54th Annu. Meet. Assoc. Comput. Linguist. ACL 2016 - Long Pap. 1, 86–96. https://doi.org/10.18653/v1/p16-1009 ↗
43. Tay, Y., Tran, V.Q., Ruder, S., Gupta, J., Chung, H.W., Bahri, D., Qin, Z., Baumgartner, S., Yu, C., Metzler, D., 2022. CHARFORMER: FAST CHARACTER TRANSFORMERS VIA GRADIENT-BASED SUBWORD TOKENIZATION, in: ICLR 2022 - 10th International Conference on Learning Representations. International Conference on Learning Representations, ICLR. find
44. Toraman, C., Yilmaz, E.H., Şahinuç, F., Ozcelik, O., 2022. Impact of Tokenization on Language Models: An Analysis for Turkish. ACM Trans. Asian Low-Resource Lang. Inf. Process. 22. https://doi.org/10.1145/3578707 ↗
45. Tran, K., 2020. From English To Foreign Languages: Transferring Pre-trained Language Models. find
46. Vasiu, M.A., Potolea, R., 2020. Enhancing Tokenization by Embedding Romanian Language Specific Morphology. Proc. - 2020 IEEE 16th Int. Conf. Intell. Comput. Commun. Process. ICCP 2020 243–250. https://doi.org/10.1109/ICCP51029.2020.9266140 ↗
47. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I., 2017. Attention is all you need. Adv. Neural Inf. Process. Syst. 2017-Decem, 5999–6009. find
48. Wang, G., Ge, Y., Ding, X., Kankanhalli, M., Shan, Y., 2023. What Makes for Good Visual Tokenizers for Large Language Models? find
49. Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser, Ł., Gouws, S., Kato, Y., Kudo, T., Kazawa, H., Stevens, K., Kurian, G., Patil, N., Wang, W., Young, C., Smith, J., Riesa, J., Rudnick, A., Vinyals, O., Corrado, G., Hughes, M., Dean, J., 2016. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. find
50. Xue, L., Barua, A., Constant, N., Al-Rfou, R., Narang, S., Kale, M., Roberts, A., Raffel, C., 2022. ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models. Trans. Assoc. find
51. Zhang, W., Li, X., Yang, Y., Dong, R., 2021. Pre-training on mixed data for low-resource neural machine translation. Inf. 12, 133. https://doi.org/10.3390/info12030133 ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/langstream-2024-alrefaie-exploring-tokenization-strategies-vocabulary-sizes/