On the Importance of Tokenization in Arabic Embedding Models

What it is about

The paper changes only the tokenization step of two embedding models for Arabic. For Word2Vec, Arabic Wikipedia is segmented with Farasa into prefixes, stem and suffixes before training; a word vector is then built from its longest subword vector and the average of the rest, weighted by alpha = 0.3. The result is about 60% smaller in vocabulary size (155K against 383K) and gives a top-1 word analogy accuracy of 10.10% against 8.88% for the baseline (CBOW, dimension 200). For out-of-vocabulary words it reaches 13.32% on the analogy set, where fastText falls from 8.05% to 6.44%. For multilingual BERT, the same kind of segmentation is placed in front of the tokenizer and needs no pretraining; on LABR and APMD this beats the baseline and AraBERT in some columns, and character-level segmentation (CharBERT) does best on APMD. The authors' example is a word whose BPE split gives the wrong meaning (a place noun read as a liquid measure).

Ideas

• I1. Build a vector or a feature for an unseen Hebrew word from its parts: the longest stem-like part plus the averaged rest, with a fixed weight, is a rule that needs only the segmenter. For tools/hbo, use it to give an unknown lemma the features of its prefix and suffix pieces when the stem is missing from the lexicon.
• I2. Measure segmentation against a morpheme list, not only against the end task: the paper shows that a BPE split can give a wrong meaning (the mal\ab case). Build a short list of Ancient Hebrew words where the tokenizer's split and the MACULA morpheme split differ, and count how often each splitter gets them right.
• I3. Keep the window size in proportion to the average number of parts per word (the paper uses 10 for an average of 1.97 parts against the baseline's 5). A Hebrew context window for any neural component of tools/hbo or gparse should be set in morphemes, not in words.

For Mova

I1 is a fallback rule for tools/hbo (unknown lemmas) and The lexicon. I2 is a test set for tools/hbo (segmenter against MACULA) and tools/macula. I3 is a note for gparse configuration when it is run on segmented Hebrew. The embedding models themselves are not part of Mova's plan, so only the segmentation and OOV rules carry over.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{alkaoud2020importance,
  title = {{On the Importance of Tokenization in Arabic Embedding Models}},
  author = {Alkaoud, Mohamed and Syed, Mairaj},
  booktitle = {Proceedings of the Fifth Arabic Natural Language Processing Workshop},
  year = {2020},
  url = {https://aclanthology.org/2020.wanlp-1.11/}
}

References

36 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for Arabic. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11–16, San Diego, California, June. Association for Computational Linguistics. find our card
2. Mahmoud Al-Ayyoub, Aya Nuseir, Kholoud Alsmearat, Yaser Jararweh, and Brij Gupta. 2018. Deep learning for arabic nlp: A survey. Journal of computational science, 26:522–531. ↗
3. Mohamed Aly and Amir Atiya. 2013. LABR: A large scale Arabic book reviews dataset. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 494–498, Sofia, Bulgaria, August. Association for Computational Linguistics. ↗
4. Zaid Alyafeai. 2020. ARBML. https://github.com/zaidalyafeai/ARBML, January. ↗
5. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France, May. European Language Resource Association. Iz Beltagy, Matthew E. Peters, and Arman Cohan. arXiv:2004.05150. 2020. Longformer: The long-document transformer. ↗
6. Yassine Benajiba and Paolo Rosso. 2007. Anersys 2.0: Conquering the ner task for the arabic language by combining the maximum entropy with pos-tag information. In IICAI, pages 1814–1823. find
7. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606. ↗
8. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder–decoder approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103–111, Doha, Qatar, October. Association for Computational Linguistics. find
9. Abdelghani Dahou, Shengwu Xiong, Junwei Zhou, and Mohamed Abd Elaziz. 2019. Multi-channel embedding convolutional neural network model for arabic sentiment classification. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 18(4), May. ↗
10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
11. Ahmed El-Kishky, Xingyu Fu, Aseel Addawood, Nahil Sobh, Clare Voss, and Jiawei Han. 2019. Constrained sequence-to-sequence Semitic root extraction for enriching word embeddings. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 88–96, Florence, Italy, August. Association for Computational Linguistics. find our card
12. Ashraf Elnagar, Yasmin S. Khalifa, and Anas Einea, 2018. Hotel Arabic-Reviews Dataset Construction for Sentiment Analysis Applications, pages 35–52. Springer International Publishing, Cham. find
13. Mohammed Elrazzaz, Shady Elbassuoni, Khaled Shaban, and Chadi Helwe. 2017. Methodical evaluation of Arabic word embeddings. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 454–458, Vancouver, Canada, July. Association for Computational Linguistics. find
14. Ali Farghaly and Khaled Shaalan. 2009. Arabic natural language processing: Challenges and solutions. ACM Transactions on Asian Language Information Processing (TALIP), 8(4):1–22. ↗
15. Philip Gage. 1994. A new algorithm for data compression. C Users Journal, 12(2):23–38. find
16. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018). find
17. Zellig S Harris. 1954. Distributional structure. Word, 10(2-3):146–162. find
18. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759. ↗
19. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
20. Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la Clergerie, Djamé Seddah, and Benoı̂t Sagot. 2020. Camembert: a tasty french language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. find
21. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. ↗
22. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. ↗
23. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. find
24. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana, June. Association for Computational Linguistics. find
25. Radim Řehůřek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pages 45–50, Valletta, Malta, May. ELRA. http://is.muni.cz/publication/884893/en. ↗
26. Rana Aref Salama, Abdou Youssef, and Aly Fahmy. 2018. Morphological word embedding for arabic. Procedia computer science, 142:83–93. find
27. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. ↗
28. Pamela Shapiro and Kevin Duh. 2018a. Bpe and charcnns for translation of morphology: A cross-lingual comparison and analysis. arXiv preprint arXiv:1809.01301. ↗
29. Pamela Shapiro and Kevin Duh. 2018b. Morphological word embeddings for Arabic neural machine translation in low-resource settings. In Proceedings of the Second Workshop on Subword/Character LEvel Models, pages 1–11, New Orleans, June. Association for Computational Linguistics. find our card
30. Abu Bakr Soliman, Kareem Eissa, and Samhaa R El-Beltagy. 2017. Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science, 117:256–265. find
31. Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645–3650, Florence, Italy, July. Association for Computational Linguistics. find
32. Stephen Taylor and Tomáš Brychcı́n. 2018. The representation of some phrases in arabic word semantic vector spaces. Open Computer Science, 8(1):182–193. ↗
33. Andrew Trask, Phil Michalak, and John Liu. 2015. sense2vec-a fast and accurate method for word sense disambiguation in neural word embeddings. arXiv preprint arXiv:1511.06388. ↗
34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
35. Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: Bert for finnish. ↗
36. Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. BERTje: A Dutch BERT Model. arXiv:1912.09582 [cs], December. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2020-alkaoud-importance-tokenization-arabic-embedding-models/