An Arabic Morphological Analyzer and Generator with Copious Features

What it is about

The paper presents CALIMA Star, an Arabic analyzer and generator built on the BAMA/SAMA/ALMOR family of pre-compiled six-table databases (three lexicon tables for prefixes, stems and suffixes, three compatibility tables). The authors add features that the older databases lack: functional gender and number (annotated by hand for about 77,000 inflected forms), rationality (annotated for about 18,000 nominal lemmas), state and case with explicit ambiguous suffixes, roots and patterns, phonological form, four tokenization schemes precomputed per entry, out-of-context lemma and POS probabilities, and CATiB and UD POS tags. Evaluated on the PATB development set, tokenization matches MADAMIRA on 99.5% (diacritized) of D3 cases, and the top-one joint lemma-POS choice reaches 92% POS, 90% lemma and 88% joint accuracy. Speed is traded for database size: 40 features per word, and the large database runs second of three on speed.

Ideas

• I1. Precompile the tokenization into each lexicon entry, as CALIMA Star does, rather than regenerating it after analysis. For tools/hbo, store the split of each prefix-stem-suffix triplet (the lex, pre tables) with its clitic segmentation, so the segmenter reads the split from the table instead of recomputing it.
• I2. Round-trip test: for every lemma in the Ancient Hebrew tables, generate all inflected forms and analyze them back; a form that does not return to its own features is a bug in the table. The paper's reinflection check (source word to target word, same lemma and POS) is the template.
• I3. Joint lemma-POS score from the training counts (SRILM, no smoothing) as a context-free disambiguator for tools/hbo. Measure it on a held-out MACULA slice before using it, and report it next to the segmenter's own accuracy.

For Mova

I1 goes into tools/hbo (the segmenter tables, lex and pre). I2 is a test for tools/hbo and tools/macula dictionary output. I3 is a measure, not a rule: it is a baseline for the POS choice in The verb and The Noun, the Adjective and the Numeral, to run on the MACULA held-out split. The database-wide features (rationality, state) do not apply to Hebrew as such; the state idea (explicit definite/construct analysis when the article is not the only marker) may touch The Noun, the Adjective and the Numeral.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-5816

@inproceedings{taji2018arabic,
  title = {{An Arabic Morphological Analyzer and Generator with Copious Features}},
  author = {Taji, Dima and Khalifa, Salam and Obeid, Ossama and Eryani, Fadhl and Habash, Nizar},
  booktitle = {Proceedings of the Fifteenth Workshop on Computational Research in Phonetics, Phonology, and Morphology},
  year = {2018},
  doi = {10.18653/v1/w18-5816},
  url = {https://doi.org/10.18653/v1/w18-5816}
}

References

40 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. sortium, University of Pennsylvania, 2002. LDC Catalog No.: LDC2002L49. find
2. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, David Yarowsky, Jason Eisner, and Mans Hulden. 2016. find
3. The SIGMORPHON 2016 shared task on morphological reinflection. In Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 10–22. find
4. Mona Diab, Nizar Habash, Owen Rambow, and Ryan Roth. 2013. LDC Arabic treebanks and associated corpora: Data divisions manual. arXiv preprint arXiv:1309.5652. ↗
5. Mona Diab, Kadri Hacioglu, and Daniel Jurafsky. 2004. Automatic Tagging of Arabic Text: From Raw Text to Base Phrase Chunks. In Proceedings of the 5th Meeting of the North American Chapter of the Association for Computational Linguistics/Human Language Technologies Conference (HLT-NAACL04), pages 149–152, Boston, MA. find
6. David Graff, Mohamed Maamouri, Basma Bouziri, Sondos Krouna, Seth Kulick, and Tim Buckwalter. 2009. Standard Arabic Morphological Analyzer (SAMA) Version 3.1. Linguistic Data Consortium LDC2009E73. find
7. Nizar Habash. 2007. Arabic Morphological Representations for Machine Translation. In A. van den Bosch and A. Soudi, editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods. Springer. find
8. Nizar Habash. 2010. Introduction to Arabic Natural Language Processing. Morgan & Claypool Publishers. find
9. Nizar Habash, Fadhl Eryani, Salam Khalifa, Owen Rambow, Dana Abdulrahim, Alexander Erdmann, Reem Faraj, Wajdi Zaghouani, Houda Bouamor, Nasser Zalmout, et al. 2018. Unified guidelines and resources for Arabic dialect orthography. In Proceedings of the International Conference on Language Resources and Evaluation (LREC). find
10. Nizar Habash, Ramy Eskander, and Adbelati Hawwari. 2012. A Morphological Analyzer for Egyptian Arabic. In NAACL-HLT 2012 Workshop on Computational Morphology and Phonology (SIGMOR-PHON2012), pages 1–9. find our card
11. Nizar Habash and Owen Rambow. 2005. Arabic tokenization, part-of-speech tagging and morphological disambiguation in one fell swoop. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 573– 580, Ann Arbor, Michigan. Association for Computational Linguistics. find our card
12. Nizar Habash and Owen Rambow. 2006. MAGEAD: A Morphological Analyzer and Generator for the Arabic Dialects. In Proceedings of ACL, pages 681– 688, Sydney, Australia. Association for Computational Linguistics. find our card
13. Nizar Habash, Owen Rambow, and Ryan Roth. 2009. MADA+TOKAN: A toolkit for Arabic tokenization, diacritization, morphological disambiguation, POS Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for Arabic. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11–16, San Diego, California. find our card
14. Imad A. Al-Sughaiyer and Ibrahim A. Al-Kharashi. 2004. Arabic morphological analysis techniques: A comprehensive survey. Journal of the American Society for Information Science and Technology, 55(3):189–213. find
15. Sarah Alkuhlani and Nizar Habash. 2011. A Corpus for Modeling Morpho-Syntactic Agreement in Arabic: Gender, Number and Rationality. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL’11), Portland, Oregon, USA. find our card
16. Mohamed Altantawy, Nizar Habash, and Owen Rambow. 2011. Fast Yet Rich Morphological Analysis. In Proceedings of the 9th International Workshop on Finite-State Methods and Natural Language Processing (FSMNLP 2011), Blois, France. find
17. Mohamed Altantawy, Nizar Habash, Owen Rambow, and Ibrahim Saleh. 2010. Morphological Analysis and Generation of Arabic Nouns: A Morphemic Functional Approach. In Proceedings of the seventh International Conference on Language Resources and Evaluation (LREC), Valletta, Malta. find our card
18. Kenneth Beesley, Tim Buckwalter, and Stuart Newton. 1989. Two-Level Finite-State Analysis of Arabic Morphology. In Proceedings of the Seminar on Bilingual Computing in Arabic and English. find
19. Kenneth R. Beesley. 1996. Arabic finite-state morphological analysis and generation. In Proceedings of COLING-96, the 16th International Conference on Computational Linguistics, Copenhagen. find
20. Fadi Biadsy, Nizar Habash, and Julia Hirschberg. 2009. Improving the Arabic Pronunciation Dictionary for Phone and Word Recognition with Linguistically-Based Pronunciation Rules. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 397–405, Boulder, Colorado. find
21. Mohamed Boudchiche, Azzeddine Mazroui, Mohamed Ould Abdallahi Ould Bebah, Abdelhak Lakhouaja, and Abderrahim Boudlal. 2017. AlKhalil Morpho Sys 2: A robust Arabic morpho-syntactic analyzer. Journal of King Saud University-Computer and Information Sciences, 29(2):141–146. find
22. Abderrahim Boudlal, Abdelhak Lakhouaja, Azzeddine Mazroui, Abdelouafi Meziane, MOAO Bebah, and M Shoul. 2010. Alkhalil Morpho Sys1: A morphosyntactic analysis system for Arabic texts. In International Arab conference on information technology, pages 1–6. Benghazi Libya. find
23. Tim Buckwalter. 2002. Buckwalter Arabic Morphological Analyzer Version 1.0. Linguistic Data Con-Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), Reykjavik, Iceland. tagging, stemming and lemmatization. In Proceedings of the Second International Conference on Arabic Language Resources and Tools. The MEDAR Consortium. find
24. Nizar Habash and Ryan M Roth. 2009. CATiB: The Columbia Arabic Treebank. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 221–224. find our card
25. Nizar Habash and Fatiha Sadat. 2006. Arabic Preprocessing Schemes for Statistical Machine Translation. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pages 49–52, New York City, USA. find
26. Nizar Habash, Abdelhadi Soudi, and Tim Buckwalter. 2007. On Arabic Transliteration. In A. van den Bosch and A. Soudi, editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods. Springer. find
27. Salam Khalifa, Sara Hassan, and Nizar Habash. 2017. A morphological analyzer for Gulf Arabic verbs. In Proceedings of the Workshop for Arabic Natural Language Processing 2017 (co-located with EACL 2017), Valencia, Spain. find
28. Salam Khalifa, Nasser Zalmout, and Nizar Habash. 2016. YAMAMA: Yet Another Multi-Dialect Arabic Morphological Analyzer. In Proceedings of the International Conference on Computational Linguistics (COLING): System Demonstrations, pages 223–227. find
29. Seth Kulick, Ryan Gabbard, and Mitch Marcus. 2006. Parsing the Arabic Treebank: Analysis and Improvements. In Proceedings of the 5th Conference on Treebanks and Linguistics Theories, pages 31–32. find
30. Mohamed Maamouri, Ann Bies, Tim Buckwalter, and Wigdan Mekki. 2004. The Penn Arabic Treebank: Building a Large-Scale Annotated Arabic Corpus. In NEMLAR Conference on Arabic Language Resources and Tools, pages 102–109, Cairo, Egypt. find
31. Yuval Marton, Nizar Habash, and Owen Rambow. 2013. Dependency parsing of Modern Standard Arabic with lexical and inflectional features. Computational Linguistics, 39(1):161–194. find our card
32. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Portorož, Slovenia. find our card
33. Robert Parker, David Graff, Ke Chen, Junbo Kong, and Kazuaki Maeda. 2011. Arabic Gigaword Fifth Edition. LDC catalog number No. LDC2011T11, ISBN 1-58563-595-2. find
34. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan M Roth. 2014. MADAMIRA: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Anas Shahrour, Salam Khalifa, Dima Taji, and Nizar Habash. 2016. CamelParser: A system for Arabic syntactic analysis and morphological disambiguation. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations, pages 228–232. find our card
35. June E Shoup. 1980. Phonological aspects of speech recognition. Trends in speech recognition, pages 125–138. find
36. Otakar Smrž. 2007. ElixirFM — Implementation of Functional Arabic Morphology. In Proceedings of the 2007 Workshop on Computational Approaches to Semitic Languages: Common Issues and Resources, pages 1–8, Prague, Czech Republic. find our card
37. Andreas Stolcke. 2002. SRILM an Extensible Language Modeling Toolkit. In Proceedings of the International Conference on Spoken Language Processing. find
38. Michelle Vanni and Rémi Zajac. 1996. The Temple Translator’s Workstation Project. In Proceedings of a workshop on held at Vienna, Virginia: May 6-8, 1996, pages 101–106. find
39. Nasser Zalmout and Nizar Habash. 2017a. Don’t throw those morphological analyzers away just yet: Neural morphological disambiguation for Arabic. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 704– 713, Copenhagen, Denmark. find our card
40. Nasser Zalmout and Nizar Habash. 2017b. Optimizing tokenization choice for machine translation across multiple target languages. The Prague Bulletin of Mathematical Linguistics, 108(1):257–269. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2018-taji-arabic-morphological-analyzer-generator-copious/